Rendered at 19:07:20 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
mx7zysuj4xew 3 hours ago [-]
> "It's always DNS"
This is a stupid meme that became a truism that really needs to die.
Stop mystifying basic network services.
All you're doing is scaring people off from running their own independent services leading to further consolidation and monopolization of the web
icedchai 3 hours ago [-]
Yes. DNS is actually one of the simplest services to self host. I remember reading O'Reilly's "DNS and BIND" book as a teenager! I've been hosting my own public DNS since 1995, both personally and professionally.
quesera 2 hours ago [-]
Remember too, that DNS and BIND was like 800 pages long, and most of it was important to know back then.
Today, fire up Unbound and/or NSD, write a few simply-formatted files, maybe download the current root zonefile, if not included in the distribution, and go.
I used to know a lot about name resolution. I still remember some of it, maybe most! But I haven't thought about a DNS problem in decades, and I've been running nameservers of all kinds continuously. Moving away from BIND was the most important first step. djbdns/tinydns, and now the modern replacements, just work better in all ways.
icedchai 2 hours ago [-]
Is installing BIND any more difficult than NSD today? It's all simply formatted files. Other than inertia, BIND has features that I do use, like "nsupdate." Last I checked NSD didn't support that.
quesera 2 hours ago [-]
I've heard that BIND has improved since 1995, but I haven't touched it since then. See also: Sendmail. :)
icedchai 1 hours ago [-]
Sendmail's configuration was a bit of a mess, for sure! I moved on to qmail then eventually postfix.
quesera 1 hours ago [-]
Same! The migration from qmail to Postfix was a bit painful for my circumstances, because it also required moving from ezmlm to GNU Mailman.
But Postfix has been great to manage and I have only slight and passing nostalgia for the Internet Systems, Security, and Services of the 1990s. I remember it being fun at the time though!
InsidiousWombat 3 hours ago [-]
I've noticed that was usually in context of some windows DNS problems (as it is integrated in their arcane active directory stack) and then people started repeating it as it is some kind of common, universal problem, and not relatively easy to host and easy to make highly available service
ikidd 2 hours ago [-]
If you started networking in or before the noughts, this meme speaks to you.
Maybe (maybe) it's less true now, but that's because of lessons learned in sweat and lost hair over the previous quarter century.
So you can be dismissive of that experience, but that doesn't make it invalid. And it still happens today.
mx7zysuj4xew 1 hours ago [-]
I can imagine running chrooted daemons to serve Round-robin DNS with multiple master and slave servers was tricky, especially prior to RFC 3484. But overall I don't really see how a zonefile from 1998 is different from a zonefile in 2026
iJohnDoe 3 hours ago [-]
Agreed, the meme is quite worn out at this point. It's often used improperly, which makes its use even more tiring.
talkingtab 3 hours ago [-]
What came to me as I read this is the prevalence of Bread Making Machines. It seems to me We have becomes a society that thinks we cannot do things. Bread making is a skill that was pretty universal - and without a machine. Now, instead of learning the skill, we buy a machine or get a library or pay a service. In many cases these {machines, libraries, services} work in simple cases but have severe limitations. Plain Old People developed DNS and all the other networking things. (thank you Van Jacobson, et al.)
There is a lot of complexity out there, but I have found many times that the solutions that protect you from that complexity are often as complex or more so.
reticulates 7 hours ago [-]
I'm not familiar with your product, just the blog post + cell architecture documentation. Question: the documentation states that basins are assigned to a cell on creation and cannot move. Why choose to have explicit DNS records instead of wildcards with a subdomain per location? e.g: {basin}.us-east-1.s2.dev where *.us-east-1.s2.dev points to the cell? My best guess is that you prefer to have explicit architecture in case you need to move to a new cell for some reason, but maybe there was a different reason? The stuff around hitting the database on every query that necessitates a firewall seems like a huge risk vs. a wildcard CNAME which adds no additional burden.
reticulates 7 hours ago [-]
Also, unsolicited advice but having your website, integration endpoints, status page and email under the same domain is a bad idea, a single point of failure. Especially a domain that is due to expire in just a few months. You should have a separate domain for infrastructure and a separate status page domain, each renewed for 10 years in advance (the maximum). e.g: s2.dev for website, s2status.com for status page and s2stream(s).com for your Basin URLs.
misker1 6 hours ago [-]
Thank you for the advice, the status page move makes a lot of sense! I don't think we have the bandwidth to separate the basin subdomains just now, but very helpful in terms of SPoF.
edit: oh, and the 10 year renewal, that's the top of the list.
misker1 6 hours ago [-]
The main two reasons: we have multiple cells per location (we want to keep cells decent in size to reduce failures) and a move between cells is something we have on the roadmap. I tried to find a wildcard solution at the beginning of building this, I just couldn't find a good one.
bajard 6 hours ago [-]
My ISP kept intercepting, editing and blocking my DNS queries. I had to encrypt, obfuscate, self-host, re-route and randomize them to get rid of this harassment.
ada-h00k 3 hours ago [-]
What country is that in? Because that sounds extremely illegal. Here in Germany, ISPs do have their own DNS servers that don't resolve certain domains, but there's nothing stopping you from using a different DNS server. Manipulating traffic to prevent that wouldn't be legal either.
_verandaguy 3 hours ago [-]
Not OP, but this reminds me of what Canadian ISPs used to do in the late 2000s/early 2010s where reaching your usage limit would result in an HTML message being injected at the top of any non-HTTPS page's `<body>` telling you that you're approaching (or over) your monthly quota.
Dark times, those were.
roncesvalles 6 hours ago [-]
Just use DoH? It's generally understood that most ISPs fuck with port 53 DNS traffic now.
eddythompson80 6 hours ago [-]
Doh is not as cool to say encrypted, obfuscated, rerouted and randomized.
MiscIdeaMaker99 6 hours ago [-]
Back in the day, I remember running our own DNS servers because we had customers who wanted to white-label our product with their own domains, and _lots_ of marketing departments didn't want to deal with their IT people, so we offered to host DNS for them. Of course, that's assuming they had IT people at all.
About half of the domains were registered by us on behalf of the customer, while the other half delegated a sub-domain to us. There was also a small minority who hosted all of the DNS themselves.
getfacl 7 hours ago [-]
Out of curiosity, was using PowerDNS with CloudSQL (fed from a spanner stream) considered?
There are always trade offs of course but it would have the advantage of DNS spikes not impacting spanner.
InsidiousWombat 5 hours ago [-]
We run PowerDNS with MariaDB replication on the backend
Zero problems for last 15 years. Near-instant propagation.
Scaling options are great, as the amount of changes in records per second is generally small even if you have hundreds of domains
misker1 4 hours ago [-]
I'll admit that we haven't looked at using a Spanner stream with PowerDNS. We did try Spanner and their stale reads feature: https://docs.cloud.google.com/spanner/docs/samples/spanner-r..., however the records were not available instantly after our API said the basin was created. Will investigate.
whinvik 5 hours ago [-]
I am very curious how they decided to have most infra in GCP but only DNS on AWS Route 53. I would expect that one could have everything on just GCP no?
The other interesting thing that I am starting to think more about is that every part of the infra stack will get slowly rebuilt because of AI. More load, more granular permissions, more firewalls are some of the things I can think that will need to be built in.
shikhar 4 hours ago [-]
Most of our infra is in AWS, actually – but we do use GCP Spanner as a global control plane database, including basin metadata and its mapping from basin to cell. So for the DNS server which relies on these mappings, GCP made sense. In future we also want to run S2 cells in GCP.
UltraSane 4 hours ago [-]
I like AWS DNS because it has a 100% SLA and they said they actually designed it to stop billing if needed to free up resources. You can create VERY fine grained IAM Policies that are very useful for secure automation and delegation. And of course the API is excellent.
justinc8687 5 hours ago [-]
I for one, do the same with my company. I've thought of "worse case scenarios" and the one I think among the worst is my AWS account gets hacked and ransomed. Having my DNS hosted with another provider gives me a quick way to rebuild the stack from off-site backups, switch over DNS and tell the hackers to go f$%$ themselves. If my DNS is hosted with AWS or God forbid, my registrar, then I'm just boned.
saldoran 5 hours ago [-]
[flagged]
TacticalCoder 5 hours ago [-]
DNS is incredibly complex IMO. But I still run unbound at home: it s of course not the same thing as running a publicly facing DNS.
But being able to blacklist hundreds of thousands, heck probably millions, of domains that are homoglyph attacks on my usual banking and brokers website is just too good to pass upon.
I started doing that after my broker sent me a warning by email about a phishing site that was a 100% perfect copy of the real thing. Basically:
mycoolbroker.com
vs:
mycoo1broker.com
Of course I use 2FA etc. But I still wouldn't want my credentials phished and then using those an attacker could try to reset the 2FA.
So I wrote a little piece of code generating a shitload of potential homoglyphic attacks on my most important sites' names.
I also blacklist a great many TLDs entirely. So many countries get null routed / NX_DOMAINed.
And so many ads disappear. And sk many domains serving porn and malware.
All going down the drain.
I love it.
As a side note I also reject trafic from a huge number of IP blocks (entire countries / entire ISPs / entire companies like Meta).
It s amazing how much of the Web / Internet keeps working just fine after all that.
It s also pretty much something you set up once then forget about it.
And I did all that in the pre-LLM days. Now it s even easier.
Gareth321 5 hours ago [-]
I also run unbound on OPNsense but for almost exactly the opposite reason: evading censorship. Here in Europe so many websites are blocked by DNS at the ISP level due to local or EU laws and edicts. Unbound ensures I can browse a free and open internet free from government and corporate censorship.
misker1 10 hours ago [-]
I wanted to share our journey of running our own DNS server (it's not as scary in the age of AI, but still...). If anyone has questions I can answer them.
koito17 7 hours ago [-]
If you're running a k8s cluster, you're already running a DNS server. It shouldn't be too scary IMO, but I've also wasted a lot of time writing DNS-related libraries and application in the past, so I'm probably biased.
I find the following part of the article interesting.
These [DNS] records would need strong consistency too. When a basin is created, its address should be instantly resolvable, otherwise clients would be waiting around for a cached NXDOMAIN to expire.
DNS is, almost by design, an eventually consistent system. Strong consistency would mean authoritative servers for a zone have to coordinate updates and agree on the current state before answering queries. That would almost defeat the availability goals that date back to the very origin of DNS. To quote the very old RFC 1034:
Access to information is more critical than instantaneous updates or guarantees of consistency
I am guessing S2 has clients query a specific DNS server directly so that there is strong consistency in practice. Also, it's unclear to me whether it's intended for records to propagate to other name servers. It'd be nice if the article included an appendix for people who don't know what S2 is (or how clients typically access the service).
misker1 6 hours ago [-]
Ah, I wrote strong consistency but what I really meant was atomic, on the creation side. Before, our basin DNS entries were created async in Route53, and they could stay in that Creating status for a long time (DDoS, provider limits, etc.). Now, once the API says 200, the DNS entry is guaranteed to exist at our nameservers.
Clients use regular resolvers, no special server, accessed via http1/2. The funny thing is that we actually had a client who was relying on DNS being strongly consistent. Since that isn't really feasible, we asked them to add a small delay after creating a basin, and they did.
lbriner 7 hours ago [-]
> DNS is, almost by design, an eventually consistent system.
I agree with this. Offering instant updates creates a large performance benchmark, especially if you experience massive growth. Unless this is mitigated with TTL to some extent? I would personally be looking at how to reduce the database load for misbehaving clients e.g. 5 second cache on all queries or allowing a customer to choose a "tier" where they won't need fast instant updates. If they choose e.g. 5 minutes TTL, you can cache these extensively, even just as disk files rather than RAM.
misker1 6 hours ago [-]
I agree with this too and to be really honest we are still tweaking all the caches between DNS Firewall our S2 DNS server and the CNAMEs we hand out. The instant part was more for new basins being created.
xtiansimon 6 hours ago [-]
I manage my own DNS record at zoneedit (set it and forget it). What’s something else (tricky, instructive, whatever) about running your own DNS that I don’t experience?
InsidiousWombat 4 hours ago [-]
It's not tricky. The problems are entirely due to one or more of
* Not understanding what DNS is (eventual consistency etc.)
* Round peg square hole problem ("oh we rely on the record updating instantly")
* Using a Microsoft products to serve DNS
slyall 8 hours ago [-]
What is the scale on the graph?
misker1 6 hours ago [-]
Daily queries, over ~7 weeks.
InsidiousWombat 4 hours ago [-]
The scale is per day ? So, like 25 queries per second in peak ?
You could've just used PowerDNS + SQL backend. Near-instant updates, easily can handle far bigger traffic, easily scalable
misker1 4 hours ago [-]
Uh, the dashboard data is from CF and it kept changing on me, the data is for 4 weeks. The peak traffic is somewhere at about 120rps for DNS queries. I'll have to look at the PowerDNS + SQL backend combo. My biggest desire was to stay on Route53 and not run anything. Since we had to, I guess it made a little sense to build a custom thing.
InsidiousWombat 3 hours ago [-]
That's about what we run (I think absolute peak is at ~200rps, ~100 average) and we're seeing like... 20% of single core (DDR4 gen Epyc)).
DIYing it definitely have an upfront development cost (either gonna have to manage it via DB or use PowerDNS APIs), but running costs are pennies.
bigstrat2003 3 hours ago [-]
It was never scary. DNS is a very simple service.
icedchai 4 hours ago [-]
How frequently are new DNS records being added / modified?
misker1 3 hours ago [-]
New records are added frequently, as clients create unique basins. The problem is it's usually a steady stream... then a massive spike. Modifications are rare, they are usually deleted. These don't really impact the DNS server though, a client could create a million basins and thus new DNS entries, nothing needs to happen at the DNS layer since it's all queried from the Spanner backend when needed.
This is a stupid meme that became a truism that really needs to die.
Stop mystifying basic network services.
All you're doing is scaring people off from running their own independent services leading to further consolidation and monopolization of the web
Today, fire up Unbound and/or NSD, write a few simply-formatted files, maybe download the current root zonefile, if not included in the distribution, and go.
I used to know a lot about name resolution. I still remember some of it, maybe most! But I haven't thought about a DNS problem in decades, and I've been running nameservers of all kinds continuously. Moving away from BIND was the most important first step. djbdns/tinydns, and now the modern replacements, just work better in all ways.
But Postfix has been great to manage and I have only slight and passing nostalgia for the Internet Systems, Security, and Services of the 1990s. I remember it being fun at the time though!
Maybe (maybe) it's less true now, but that's because of lessons learned in sweat and lost hair over the previous quarter century.
So you can be dismissive of that experience, but that doesn't make it invalid. And it still happens today.
There is a lot of complexity out there, but I have found many times that the solutions that protect you from that complexity are often as complex or more so.
edit: oh, and the 10 year renewal, that's the top of the list.
Dark times, those were.
About half of the domains were registered by us on behalf of the customer, while the other half delegated a sub-domain to us. There was also a small minority who hosted all of the DNS themselves.
There are always trade offs of course but it would have the advantage of DNS spikes not impacting spanner.
Zero problems for last 15 years. Near-instant propagation.
Scaling options are great, as the amount of changes in records per second is generally small even if you have hundreds of domains
The other interesting thing that I am starting to think more about is that every part of the infra stack will get slowly rebuilt because of AI. More load, more granular permissions, more firewalls are some of the things I can think that will need to be built in.
But being able to blacklist hundreds of thousands, heck probably millions, of domains that are homoglyph attacks on my usual banking and brokers website is just too good to pass upon.
I started doing that after my broker sent me a warning by email about a phishing site that was a 100% perfect copy of the real thing. Basically:
vs: Of course I use 2FA etc. But I still wouldn't want my credentials phished and then using those an attacker could try to reset the 2FA.So I wrote a little piece of code generating a shitload of potential homoglyphic attacks on my most important sites' names.
I also blacklist a great many TLDs entirely. So many countries get null routed / NX_DOMAINed.
And so many ads disappear. And sk many domains serving porn and malware.
All going down the drain.
I love it.
As a side note I also reject trafic from a huge number of IP blocks (entire countries / entire ISPs / entire companies like Meta).
It s amazing how much of the Web / Internet keeps working just fine after all that.
It s also pretty much something you set up once then forget about it.
And I did all that in the pre-LLM days. Now it s even easier.
I find the following part of the article interesting.
DNS is, almost by design, an eventually consistent system. Strong consistency would mean authoritative servers for a zone have to coordinate updates and agree on the current state before answering queries. That would almost defeat the availability goals that date back to the very origin of DNS. To quote the very old RFC 1034: I am guessing S2 has clients query a specific DNS server directly so that there is strong consistency in practice. Also, it's unclear to me whether it's intended for records to propagate to other name servers. It'd be nice if the article included an appendix for people who don't know what S2 is (or how clients typically access the service).Clients use regular resolvers, no special server, accessed via http1/2. The funny thing is that we actually had a client who was relying on DNS being strongly consistent. Since that isn't really feasible, we asked them to add a small delay after creating a basin, and they did.
I agree with this. Offering instant updates creates a large performance benchmark, especially if you experience massive growth. Unless this is mitigated with TTL to some extent? I would personally be looking at how to reduce the database load for misbehaving clients e.g. 5 second cache on all queries or allowing a customer to choose a "tier" where they won't need fast instant updates. If they choose e.g. 5 minutes TTL, you can cache these extensively, even just as disk files rather than RAM.
* Not understanding what DNS is (eventual consistency etc.) * Round peg square hole problem ("oh we rely on the record updating instantly") * Using a Microsoft products to serve DNS
You could've just used PowerDNS + SQL backend. Near-instant updates, easily can handle far bigger traffic, easily scalable
DIYing it definitely have an upfront development cost (either gonna have to manage it via DB or use PowerDNS APIs), but running costs are pennies.