Offsite DAG member throwing Event ID 1135

[blin]

I have a three member DAG. Two of my DAGs are local, wtih the third being remote with VPN connectivity. According to the EMC, all of my DBs are healthy on all three DAG members (actually, healthy on two of them, and mounted on the third). So everything looks good.



However, when I go into Event Viewer, I see that there are quite a few Event ID 1135 (Microsoft-Windows-FailoverClustering) showing that my remote DAG member was "removed from the Active Failover Cluster Membership". Here is the error in full:

Event[0]:
Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 2012-01-11T07:49:15.912
Event ID: 1135
Task: Node Mgr
Level: Critical
Opcode: Info
Keyword: N/A
User: S-1-5-18
User Name: NT AUTHORITY\SYSTEM
Computer: xxxx.xxx..local
Description:
Cluster node 'XXXX' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

If EMC shows all Green and Healthy, should I be concerned wtih this apparent 'Critical' failure of the Failover Cluster?

Could this be caused by latency on our VPN (we average around 25-30 ms latency on that link).




I might should add that my remote DAG member has a single NIC that both replication and mapi traffic flow. On my two local DAG members, they each have two nics, one dedicated for repl and one for mapi.

[blin]

Database copies aren't supported between Mailbox servers with round trip network latency greater than 500 milliseconds (ms).
For your referance go through below link and follow the step.
http://technet.microsoft.com/en-us/libr ... 3973(WS.10).aspx

[blin]

Have you installed KB2550886 ? http://support.microsoft.com/kb/2550886

It's higly recommended to do so!!

Recommended Windows Hotfix for Database Availability Groups running Windows Server 2008 R2
http://blogs.technet.com/b/exchange/arc ... 08-r2.aspx

You might also benefit from configuring the heartbeatsetting (CrossSubnetDelay & CrossSubnetThreshold)
http://technet.microsoft.com/en-us/libr ... 7562(WS.10).aspx



--------------------------------------------------------------------------------
Martina Miskovic - http://www.nic2012.com/

[blin]

The issue might be also caused by the network related issue.

Please following the suggestion of Martina to install the necessary hotfixes. if the issue continues, keep on monitor on the VPN network connection.


--------------------------------------------------------------------------------

[blin]

I applied the hotfix, but to no success. However, I adjusted the Cluster prop CrossSubnetThreshold to 12 (by following the excellent link provided by Martina-Thanks!). This has eliminated the alerts. Is this value (12) too high. I know I need to balance the latency reality on my stretched DAG with the need to not wait too long before it recognizes there is a legitimate failure. Any feedback on this setting? The default was 5.

[blin]

We had seen some cases similar this one. In most cases. it is network or name resolution issue. Can you ping all servers by IP addresses?
--------------------------------------------------------------------------------