Problem Summary:
The client failover service for PI adapter is not highly available. If the client failover service node is unavailable, then the PI adapter instances that it manages will lose failover functionality. This is further exacerbated if the primary PI adapter node goes down as well. In this case the secondary PI adapter will stay in its last current state of backup until human intervention (role override on the secondary PI adapter node).
Solution:
Client failover service does not need to be redundant if the PI adapter pairs that it manages are smart enough to communicate with each other in the case the failover service goes down. In the unique case of the primary PI adapter and client failover service becoming unavailable, then the secondary will automatically take over.
This potential solution allows for self-healing without human intervention at remote sites.
Below is a potential workaround (not tested). If it works, then we would like to see it incorporated into PI adapters.
Client failover service goes down.
primary PI adapter goes down.
secondary is in a backup state.
outside service/script hosted on secondary periodically checks the health of the primary, and client failover service (this can be a simple ping).
-
if both primary, and Client failover service are down, then the secondary assumes primary state with a role override command:
https://docs.aveva.com/bundle/client-failover-service/page/administration/perform-a-role-override.html
Part of me likes the idea of having redundant instances of the Client Failover Service, but another part of me thinks that the only time that this would be useful is in an edge case. The scenario that you described requires 2 of the 3 servers to go down in a specific order, and there is at least a manual solution to that problem that does not introduce new instances of any services.
It would be more likely for the 2 AVEVA Adapter servers to be down (with no restriction on which goes down first), and this would also break the flow of data with 2 of the 3 servers being down, but there is no solution to this that does not add another service instance (in this case, the AVEVA Adapter) to the system on the fly. In this scenario, would the solution be to add a 3rd AVEVA Adapter server (i.e. a 4th server to the system, whether ahead of time or on the fly), or would we just deal with this scenario as it happens and take steps to minimize the chances that it happens in the first place?
This is a great idea and will fill a gap in the Client Failover Server operation.