K3S, Ubiquity, and PiHole - What not to do.

I started the Ubiquity journey in early 2020 with the purchase of an Ultimate Dream Machine Pro along with various switches and access points.  I've been very happy with their products and highly recommend them for the prosumer.  

I've also been a fan of PiHole over the years mainly for it's ad blocking, not really using it's DNS features.

I've been using K3S over the last month or so but I've experienced alot of DNS related issues.

This weekend I decided to clear up all my K3S issues and get it working with PiHole to do name resolution along with setting up my Work network so I can resolve names.

The first thing I did was try and figure out why I was having issues resolving domain names in my K3S pods.  

I started out by running a container so I could play around with DNS resolution.

I ran this: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml and then this: kubectl exec -i -t dnsutils -- nslookup www.google.com

I was getting errors over and over again so I did some more digging leveraging these pages with no luck.

https://ranchermanager.docs.rancher.com/v2.5/troubleshooting/other-troubleshooting-tips/dns#check-if-domain-names-are-resolving

https://medium.com/@hsahu24/understanding-dns-resolution-and-resolv-conf-d17d1d64471c

https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

I finally started looking at my Ubiquity Network settings.  I had turned off pretty much everything including Content Filtering but I noticed that I had something set in the Domain Name field.

I removed the name so it looked like the image above and voila, I was able to run the nslookup command in my container and resolve www.google.com using my PiHole.  Long story short, having that name there would add a row to all the resolv.conf files of every computer in that network to search for that entry first when resolving domain names.  

The next issue I ran into was trying to get my Work network accessing the PiHole on my Lab network.  I had set the DNS server on my Work network to use the PiHole and the computers on my Work network said they were using it but alas there were no logs on the PiHole that said they were using it.  I added and removed firewall for nearly an hour thinking that was the issue when I noticed that the Content Filtering was set to Work.

I set this to None, did another nslookup from my Work network to my PiHole and voila, I saw logs.  

The last thing I did was create DNS names for all the computers and services on both networks so I could see things more clearly on the PiHole.  It's now been two days since I've set this all up and everything has been great.  Hopefully this will help someone running into the same situations I ran into.