Granite Canyon processes & procedures (using tt.tc as an example)

[Steve Senator]

This discussion, while initiated in relation to tt.tc may shed some
light for others regarding the Public DNS processing, so I have cc'd
it to this discussion.

Daniel Kleinsinger writes:
 > Can you explain why I have DNS update COMPLETE emails from soa at 5:15 your
 > time (4:15 my time) and an email from soa at 3:30 your time (1:30 my time,
 > daylight savings time for you, not for me) for serials 154669967 and
 > 154667493, but ns1, which last reloaded at 9:00 your time (8:00 my time),
 > has serial 154540344 published?

The process is approximately as follows:
1 - Zone updates flow in to soa at granitecanyon.com via email.
2 - As long as there is one in the queue, the queue is processed
    about every 10 minutes
3 - Every 6 hours on soa the zones are snapshotted.
    This is what gets adjusted when we're under a very heavy load.
    It can be as infrequent as once per 24 hours. During this past
    week it actually took up to 48 hours a few times as I was debugging
    some scripts.
4 - This bundle is propagated over to ns1
    The total time for the above (steps #3 and #4) is ~3.5 hours currently.
5 - Then ns1's name server is reloaded.
    This takes ~15-20 minutes, currently. (~30K zones)

 > And what does the system do when there is a DOA (submitted earlier)
 > and a SOA (submitted later) on the queue for the same domain name?

DOA records are processed in step #1 and #2, above.
ns2 follows the procedures outlined in #1 and #2, above, and only
propagates zones through the normal secondary AXFR mechanisms.

Just like your OS - the last write wins. If a succession of entries
make it into the queue, the last one is the only one that matters.

I'm trying to re-adjust the process in #3 and #4 so only incremental
updates are propagated, analogous to the NOTIFY protocol which solves
a similar problem. The problems arise because
 1) These are live systems.
 2) The Public DNS does not run a profit.
 3) There's only a few hours when I can do uninterrupted work on this
    per week.

Note that others adminster the other name server(s). I run ns1, soa
and news.  I prioritize as 1) ns1, 2) soa and 3) news. Every platter
on ns1 and news is mirrored, either directly (via RAID1 controllers)
or where space doesn't permit, by propagation to another server. The
current news box will eventually replace soa which is adequate for an
http server, but wasn't built redundantly and wouldn't be
cost-effective to upgrade.