OK, so I've been tasked with building two new DNS structures in my lab at work so that we can test our products with both Windows and Linux DNS servers. The Windows systems went in and worked with about a day of total work, including services such as DHCP, NTP, FTP, TFTP, WDS, WSUS, LDAP, and AD. Quick and easy.
The Linux config didn't go so easy.
I finally, after about 3 weeks of working on it, got the servers to run DHCP and DDNS (so the DHCP clients would appear in DNS lookups) and wrote down and fully documented my process to get it to work. Well, it worked, sort of...
Now, after working properly for 2 months and adding about 70 systems, it suddenly started exhibiting strange issues. When doing a DNS lookup, it will fail to respond to the system, then the system will ask the secondary DNS server, and the secondary DNS server asks the primary again, and it gives a response. So, like this:
system has 2 DNS servers configured:
10.10.20.10 (vt.test.com master server, anything beyond vt.test.com is forwarded to 10.10.10.34)
10.10.10.34 (test.com master DNS server, knows that 10.10.20.10 is authoritative for vt.test.com)
The system does a lookup for rhel1.vt.test.com, which is only listed in 10.10.20.10. It goes to 10.10.20.10 first, which doesn't reply. Then it asks 10.10.10.34, which asks 10.10.20.10. It gets a reply and forwards the reply on to the system asking for the lookup.
In a more simple scenario, it's like this:
guy 1 asks guy 2: how do you look up a phone number online?
guy 2: I don't know.
guy 1 asks guy 3: how do you look up a phone number online?
guy 3: I don't know, I'll ask guy 2. He always knows this stuff.
guy 3 asks guy 2: how do you look up a phone number online?
guy 2: oh, I just use yp.com.
guy 3 tells guy 1: Guy 2 says just use yp.com.