Home | Site Map | Cisco How ToNet How To | Wireless |Search | Forums | Services | Donations | Careers | About Us | Contact Us|

DFS Replication Issues - RPC, WMI, and Service

Active Directory, Domain, DNS, WINS, DHCP, SBS, New Releases.

DFS Replication Issues - RPC, WMI, and Service

Postby guest » Thu Jul 29, 2010 11:52 am

I have been fighting with an issue that plagues mostly one of three file servers in our DFS replication structure. The DFS Replication essentially stops replicating data between that member and the other two members. For clarity, I'll refer to the 3 servers as Server1, Server2, and Server3, with Server 3 being the one with the issue.

Usually only one of the two other members (Server1 or Server2) will start reporting communication failures to Server3, but no updates will occur on the other server to/from Server3. Replication between Server1 and Server2 seems very stable and has few issues, although I have experienced the same issue at one time or another in the past on them.

When this occurs on Server3, I can no longer query the DFS Replication WMI objects, and DFS Replication will not respond to RPC queries for the health reports. Additionally, if I try to stop the DFS Replication service, dfsrs.exe will not exit and stays running. I have even gone as far as using Process Explorer to close every open file and resource used by dfsrs.exe, and it still will not terminate.

I restarted Server3 last night three times, before DFS Replication finally started replicating files again instead of going into its zombie-like, unresponsive state. However, this morning it appears to be unresponsive again. The final reboot I performed last night (or technically this morning) at 1AM (EST), and by 5:30AM I started seeing DFS Replication errors on the other servers (Server1 and Server2), such as the following:

Error: 1727 (The remote procedure call failed and did not execute.)
Connection ID: C1F99E80-6777-419D-8797-1FE1E54F0603
Replication Group ID: F19350A6-9EE3-4584-BFFC-6DB13B248EC5

I do not see any errors or warnings on Server3 for DFS Replication.

Additionally, the Debug logs on Server3 do appear to continue logging USN journal updates, as once I start seeing RPC errors on the other servers, I only see entries in the debug log like the following:

20090205 09:04:00.582 2500 USNC 2615 UsnConsumer::CreateNewRecord ID record created from USN_RECORD:
+ USN_RECORD:
+ RecordLength: 136
+ MajorVersion: 2
+ MinorVersion: 0
+ FileRefNumber: 0x70000000278FF
+ ParentFileRefNumber: 0x33000000016E5D
+ USN: 0x2bc6fa668
+ TimeStamp: 20090205 09:04:00.582 Eastern Standard Time
+ Reason: Close File Create
+ SourceInfo: 0x0
+ SecurityId: 0x0
+ FileAttributes: 0x20
+ FileNameLength: 72
+ FileNameOffset: 60
+ FileName: amexspeakerseriesc02_05_09 MAHER.ppt
+
20090205 09:04:00.582 2500 USNC 2680 UsnConsumer::TombstoneOrDelete LDB Deleting ID Record. uid:{8EC2FB1D-7BD7-4930-A9A4-C641EE3B9DD7}-v414255
20090205 09:04:00.582 2500 USNC 2685 UsnConsumer::TombstoneOrDelete ID record deleted from USN_RECORD:
+ USN_RECORD:
+ RecordLength: 136
+ MajorVersion: 2
+ MinorVersion: 0
+ FileRefNumber: 0x70000000278FF
+ ParentFileRefNumber: 0x33000000016E5D
+ USN: 0x2bc6fa6f0
+ TimeStamp: 20090205 09:04:00.582 Eastern Standard Time
+ Reason: Close File Delete
+ SourceInfo: 0x0
+ SecurityId: 0x0
+ FileAttributes: 0x20
+ FileNameLength: 72
+ FileNameOffset: 60
+ FileName: amexspeakerseriesc02_05_09 MAHER.ppt
+
Tablet and Smartphone Setup Guide
http://www.quicksetupguide.com

Troubleshooting Vista Wireless
http://chicagotech.net/
guest
 
Posts: 10191
Joined: Mon Nov 27, 2006 1:10 pm

Re: DFS Replication Issues - RPC, WMI, and Service

Postby guest » Thu Jul 29, 2010 11:52 am

Hi Jason,


Thanks for the update.



Since you suspect the issue might be related to IPv6-IPv4 tunnels, is it possible for you to have a chance to disable the IPv6 on the domain controller on the sites and then test to if there is any RPC communication error in the “dcdiag /e /v" report?



I will be appreciated that if you can post the result of the above command here in the forum, so that I can help you on finding the root cause of this issue.



Thanks.
Tablet and Smartphone Setup Guide
http://www.quicksetupguide.com

Troubleshooting Vista Wireless
http://chicagotech.net/
guest
 
Posts: 10191
Joined: Mon Nov 27, 2006 1:10 pm

Re: DFS Replication Issues - RPC, WMI, and Service

Postby guest » Thu Jul 29, 2010 11:53 am

Thank you for the response David.

I do have some-what good news. I spent another night rebooting the problematic file server a few times, and after 3-4 reboots, DFS finally started synchronizing on it without locking up. It has been running smoothly since last Friday night (2/6/2009). I am not certain as to why, but I have an idea.

We have 2 Windows 2008 domain controllers in each of the three sites, and DFS is not having any troubles polling AD and pulling any new information. I ran DCDiag in one of the sites, and I was surprised to see that when it checked FRS it reported RPC problems.

I am beginning to suspect it has something to do with the IPv6-IPv4 tunnels I configured to route IPv6 traffic between the sites (our routers only support IPv4). The problematic file server was the first 2008 Server in that site, and so I originally configured the IPv6-IPv4 tunnel on it; where-as in the other sites, I already had a few 2008 Servers and decided to use the 2008 domain controllers for the tunnels.

I had decided Friday night, before performing all of the reboots to the problematic file server, to move the IPv6-IPv4 tunnel to the 2008 domain controller in that site. I am beginning to think it is not coincidence that I was having RPC communication problems on the file server when it had the IPv6-IPv4 tunnel, and now that I moved the tunnel to the domain controller and finally run DCDiag that I am seeing similar RPC communication errors on the domain controllers. I have not experienced any problems with Group Policies or other files in the SYSVOL not replicating between the servers, so I cannot be positive that it is anything more than a coincidence.

If the tunnel is the problem, I suspect the problem is related to the additional IPv6 address used for the tunnel. I have the dynamic DNS registration disabled for the tunnel interface, and have verified only the primary IPv6 and IPv4 addresses have been registered in DNS (along with the various AD subdomains for the service records).

I am going to try configuring different 2008 servers for the tunnels and check to see if that clears up the RPC communication errors in DCDiag.

Just to be thorough, in response to your questions regarding the firewalls, I have only the built-in Server 2008 firewall enabled on the servers and had already tried allowing all IP communications between the three servers. There are no other firewalls for communications between the sites on the WAN connection. The problem did not seem to be with establishing connections, but with the Dfsrs service locking up.
--------------------------------------------------------------------------------
Tablet and Smartphone Setup Guide
http://www.quicksetupguide.com

Troubleshooting Vista Wireless
http://chicagotech.net/
guest
 
Posts: 10191
Joined: Mon Nov 27, 2006 1:10 pm

Re: DFS Replication Issues - RPC, WMI, and Service

Postby guest » Thu Jul 29, 2010 11:54 am

I removed the IPv6-IPv4 tunnels from all three sites, and it cleared up the FRS RPC errors in the DCDiag logs.

Fortunately, over the past week I found updated firmware for our routers with the necessary features we need along with IPv6 support. I just completed the configuration on the routers, and the DCDiag is still clean with IPv6 routing re-enabled between the sites.

I suspect that RPC was having troubles with the additional IPv6 addresses assigned to the tunnel interfaces. Now that we have our actual routers handling the IPv6 through IPv4 tunneling, everything seems fine with IPv6 communications.
Tablet and Smartphone Setup Guide
http://www.quicksetupguide.com

Troubleshooting Vista Wireless
http://chicagotech.net/
guest
 
Posts: 10191
Joined: Mon Nov 27, 2006 1:10 pm


Return to Windows

Your Ad Here

Who is online

Users browsing this forum: Google Adsense [Bot] and 19 guests