Multi Master/Primary Authoritative DNSSEC DNS Nameserver With Synced/Replicated COMMON Dir/Vol For BIND

[Terik Erik Ashfolk]

Hi Matthijs. THANK YOU.
This "MUSIC" tool is indeed appears to be most suitable assisting 
addon tool for BIND to support MULTI-SIGNER MODEL-2 (aka MULTI 
MASTER/PRIMAR)Y DNS NAME SERVER, at this moment.

I think i've seen another project Seen few other project also doing 
similar

I regret, i did not follow all the links in your ref in your ISC 
pdf, which appeared in ggl search result earlier, & i assumed 
by-now BIND has builtin Multi-Signer MODEL-2 aka 
Multi-Master/Primary DNSSEC mode support.

If the "MUSIC" was in C / C++ / perl, etc that would have been 
better for me.
I will prepare a similar debian VPS server, & then compile "music", 
& find the needed binary files + config files, and copy-over to 
nameservers.


By the way, any plan to share a pre-compiled standalone static 
binary & config files released as tar/etc file in github ?


 From readme page, my interpretation is its installed in each 
primary/master providers/nameservers, & can perform independently 
if/when 2 of the 3 provider/namerservers went down/failed for 
prolonged time.
If above assessment is correct, then it is fully HA 
(High-Availability) addon solution for BIND.
But if "MUSIC" needs to be indicated manually, that, 2 went down 
out of 3, then its not fully HA.


As BIND itself do not have builtin RFC 8901 MODEL-2 (Multi-Signer 
MODEL-2) support now in v9.18, (current version in Debian is 
v9.18-28), until v9.20 standard/stable arrives in future, I need to 
do multi-signer model-2 with other tools, And, using other-ways to 
update+sync is still necessary for MultiSigner / Multi-Master 
objectives.
So, i want to try other ways of updating/signalling, instead of 
using BIND's TSIG based connections, etc.
I want to try shared storage based solution, where files are 
indicating/performing as signal for making changes, etc to ensure 
only those (providers/nameserver) who are up/running is able to 
send/receive signal for changes, and BIND still running fine (and 
removing expired keys, etc), etc.


It is sad that https://launchpad.net/~isc/+archive/ubuntu/bind repo 
do not have Debian packages. It has BIND v9.20 for Ubuntu.
But, Ubuntu's base is Debian, so it should run on Debian too.
BIND9 v9.20 atleast in Debian "testing" repo, ... i will attempt to 
find out if BIND9 9.20.2-1-testing can work with dependencies from 
standard debian repo. Dont want to get anything other than BIND 
from "testing".

I think i need to use advanced option in "apt" to install the 
"testing" BIND in a different directory. Disable default BIND, & 
enable the "testing" BIND.

Thank you.

Erik.

Erik T Ashfolk.


On 9/29/24 11:36 PM, Matthijs Mekking wrote:
> Hi Erik,
> 
> There is no configuration option for enabling multi-signer in BIND.
> 
> BIND 9.20 is able to deal with multi-signer setups, but as Mark 
> mentioned earlier, all the coordination needs to be done outside 
> the name server.
> 
> You may consider MUSIC for this: https://github.com/DNSSEC- 
> Provisioning/music
> 
> Best regards,
> 
> Matthijs
> 
> On 9/28/24 03:50, Terik Erik Ashfolk wrote:
>> Does the BIND have command/parameter for configuring+running BIND 
>> in Multi-Signer MODEL-2 mode as specified in RFC 8901 ?
>> https://www.rfc-editor.org/rfc/rfc8901.html
>>
>> in another words, Can BIND itself handle multiple-provider's 
>> (aka: multiple-nameserver's) KSKs, ZSKs, DNSKEYs, etc RRsets and 
>> create/update RRSIGs accordingly with Multi-Signer MODEL-2 mode ?
>>
>> If it can what commands/parameters enable such mode ?
>> What "update-policy" it needs ?
>>
>> Erik.
>>
>> Erik T Ashfolk.
>>
>>
>> On 9/27/24 2:53 PM, Terik Erik Ashfolk wrote:
>>> According to the page
>>> https://blog.apnic.net/2021/08/25/multi-signer-dnssec-models/
>>> in MODEL 2.
>>> I added an improved image as attachment.
>>> <img alt="ZSK-signing" src="cid:ImportZSK-PublishDSofKSK.jpg" />
>>> MULTI-ZSK-SIGNING IS ONE OF THE SOLUTION, and appears to be 
>>> suitable for my case.
>>>
>>> So, multi-signing with ZSKs from multiple nameservers would have 
>>> worked,
>>> when nameservers were using separate "zones" & "keys" folder,
>>>
>>> I needed to sign n1's zone file with n2's ZSK & with n3's ZSK.
>>> I needed to sign n2's zone file with n1's ZSK & with n3's ZSK.
>>> I needed to sign n3's zone file with n1's ZSK & with n2's ZSK.
>>>
>>> Because 3 nameservers are using SYNCED/REPLICATED shared 
>>> directories & files,
>>> so each ZSK & KSK are available to other nameservers.
>>>
>>> for "key-directory"
>>> n1 using "/mnt/vol/v1/etc/bind/n1/keys"
>>> n2 using "/mnt/vol/v1/etc/bind/n2/keys"
>>> n3 using "/mnt/vol/v1/etc/bind/n3/keys"
>>>
>>> and shared common directory for BIND keys is
>>> "/mnt/vol/v1/etc/bind/keys"
>>>
>>> and shared directory is
>>> "/mnt/vol/v1"
>>>
>>> is there an option in BIND, that can monitor+enable additional 
>>> ZSK signing from new ZSK key from other namerservers for same 
>>> domain ?
>>> if not, please add this as new feature in BIND.
>>>
>>> if BIND itself cannot do the monitoring + multi-ZSK-signing now, 
>>> then, HOW can i monitor the ".../bind/n1/keys" (or ".../bind/n2/ 
>>> keys" or ".../bind//n3/keys" or ".../bind/keys" ) sub-dirs under 
>>> shared-directory and find that BIND has began to use a new ZSK 
>>> key ?
>>>
>>> or HOW can i get a signal from BIND in each nameserver ? that, 
>>> BIND has began to use a new ZSK key ?
>>>
>>> so-that, i can trigger/run another script in each nameserver 
>>> (which added new ZSK key) to begin signing my domain's zone file 
>>> in other 2 nameservers with the new ZSK.
>>>
>>> example : if n1 added a new ZSK for "example.com" domain, then a 
>>> "new-zsk-key-monitoring-script.sh" script will create 2 files
>>> "signal-n2-ExampleCom-MZS-zskNUMBER.txt"
>>> "signal-n3-ExampleCom-MZS-zskNUMBER.txt"
>>> in the shared-bind-directory : "/mnt/vol/v1/etc/bind/keys".
>>> Then "monitor-for-signal-file.sh" script running in n2 & n3, 
>>> will get that signal, & run "multi-ZSK-sign-script.sh" to mulit 
>>> ZSK signing.
>>>
>>>
>>> Thanks in advance.
>>>
>>> Erik.
>>>
>>> Erik T Ashfolk.
>>>
>>>
>>>
>>>
>>> On 9/26/24 7:26 PM, TErik Ashfolk wrote:
>>>> Hello BIND Community.
>>>>
>>>> Looking forward to your suggestions, advises on setup DNSSEC 
>>>> enabled zones on multiple master/primary authoritative DNS 
>>>> server (Nameserver) with synced/replicated common shared 
>>>> directories/ volume.
>>>>
>>>>
>>>> Please skip the section(s) that you dont need to read/scan,
>>>>
>>>> & goto the QUESTIONS , the last section.
>>>>
>>>>
>>>> OBJECTIVES (END-RESULT):
>>>>
>>>> Trying to achieve HA (High-Availability <https:// 
>>>> en.wikipedia.org/ wiki/High_availability>), so-that, as long as 
>>>> 1 master/primary is up/running, then my domains are still 
>>>> available to world, and allowing users to obtain DNSSEC 
>>>> verified domain- name to IP-address resolving, etc from BIND 
>>>> DNS server services.
>>>>
>>>>
>>>>
>>>> RESOURCES:
>>>>
>>>> • Servers : rented 3 servers on 3 locations from different 
>>>> server providers.
>>>>
>>>> • Domain : I have multiple domains from domain providers 
>>>> (registrar) . Here i will use "example.com"
>>>>
>>>> • Each server has 1 IPv4-address, 1 IPv6-address.
>>>>
>>>> • Domain provider's "Use your own Nameserver" is pointed to 3 
>>>> hostnames in 3 nameservers : n1.example.com ( 192.10.2.11 , 
>>>> 2001:db8:1::1 ) , n2.example.com ( 198.51.100.12 , 
>>>> 2001:db8:2::2 ) , n3.example.com ( 203.0.113.13 , 
>>>> 2001:db8:3::3 ) IP-addresses.
>>>>
>>>> • Each IP-adrs has it's RDNS setup done, to correspond & match 
>>>> with nameserver's hostname.
>>>>
>>>> • Using Debian GNU/Linux 12 (bookworm) OS in each server. 
>>>> ( Server operator can use any other OS, its their choice/ 
>>>> preference. ) ( By the way, Debian GNU/Linux is base of Ubuntu 
>>>> Linux, kind of similar to: RedHat Enterprise GNU/Linux is base 
>>>> of Fedora Linux. )
>>>>
>>>> • Using ISC BIND9 (9.18.0 , specifically now : 9.18.28) DNS 
>>>> server software, in each server. ( Server operator can use any 
>>>> other DNS server, its their choice/preference. )
>>>>
>>>>
>>>> ABOUT FILE/DIRECTORY REPLICATION:
>>>>
>>>> • For directories & files replication/sync purpose, using 
>>>> Gluster software (1 <https://en.wikipedia.org/wiki/Gluster>,2 
>>>> <https:// docs.gluster.org/en/latest/>,3 <https:// 
>>>> serverfault.com/ a/1165339/217110>,4 <https:// 
>>>> www.howtoforge.com/how-to-install- glusterfs-on-debian-12/>). 
>>>> ( There are many other choices for server operators:Multi- 
>>>> Master Replicaiton <https:// en.wikipedia.org/wiki/Multi- 
>>>> master_replication>,List <https:// en.wikipedia.org/wiki/ 
>>>> List_of_cluster_management_software>), its their own choice 
>>>> what suits best/works for their need/purpose.
>>>>
>>>> • When any file/directory changes ( i.e: in "n1" server ) , 
>>>> then the replication/sync software that is installed/ 
>>>> monitoring , will nearly immediately or within few seconds, 
>>>> begin to make same changes to same file/dir in server- 
>>>> operator's other servers ( n2 , n3 ), that are member of 
>>>> replication/sync volume/ directory. These replication software 
>>>> uses time-server to have accurate time. Whichever edit/ 
>>>> creation/deletion/modification is done last, that takes 
>>>> priority & duplicated/replicated/synced.
>>>>
>>>>
>>>>
>>>> SHARED/COMMON STORAGE/VOLUME/DIRECTORY:
>>>>
>>>> • I created a large file ("data-s1.img"), ~ 300 MB in size, 
>>>> inside root-partition , at "/storage/s1/data-s1.img" . 
>>>> Formatted with XFS filesystem creation/make tools . Attached 
>>>> large-file into a loop block device . Mounted it in "/data/s1" 
>>>> directory . Created a systemd service "mount-storage.service" 
>>>> in Debian to do previous steps one after another, so-that it 
>>>> can succeed in mounting during boot , (as "/etc/fstab" was not 
>>>> suitable for this purpose) . Others can create/use a 2nd 
>>>> partition in same storage drive (i.e: "/dev/ sda2") or add 
>>>> another storage drive (i.e: "/ dev/sdb") in server . Others can 
>>>> use a script (or "/etc/fstab") during boot to mount, etc.
>>>>
>>>> • after above steps, replication software (Gluster) was used to 
>>>> create replication volume "v1" inside the storage-mount-point 
>>>> ("/ data/s1") , so it became "/data/s1/v1" . i configured 
>>>> gluster to enable SSL/TLS based secure connection for 
>>>> replication process. Gluster also needs user to mount the 
>>>> volume as "glusterfs" type mount-point to monitor data r/w & 
>>>> replicate, & its done in : "/ mnt/ vol/v1" mount-point of 
>>>> volume "v1" . Followed stepshere <https:// serverfault.com/ 
>>>> a/1165339/217110>(& changed file/dir names).
>>>>
>>>> • the files+dirs under "/mnt/vol/v1/" is replicated/synced in 
>>>> each server, available/accessible in each server, in same 
>>>> location, has exact same contents.
>>>>
>>>> • Created "/mnt/vol/v1/etc/bind" directory for BIND aka named 
>>>> aka DNS server aka nameserver software usage. Applied : chgrp 
>>>> bind / mnt/vol/v1/etc/bind
>>>>
>>>> • Moved the "zones" dir+files from "/etc/bind", from "n1" 
>>>> server into the "/mnt/vol/v1/etc/bind/n1/" directory, & done 
>>>> similar for "n2" & "n3" servers . Moved the "keys" dir+files 
>>>> from "/etc/ bind", from "n1" into the "/mnt/vol/v1/etc/bind/n1" 
>>>> directory, & done similar for "n2" & "n3".
>>>>
>>>> • so, "/mnt/vol/v1/etc/bind/keys" & "/mnt/vol/v1/etc/bind/ 
>>>> zones" folders/directories are COMMON for all servers: "n1", 
>>>> "n2", "n3".
>>>>
>>>> • "n1" using "/mnt/vol/v1/etc/bind/n1/keys" dir & “/mnt/vol/v1/ 
>>>> etc/ bind/n1/zones” dir, so i created symlink inside to 
>>>> point+goto the replicated/synced mount-point, command : ln -s 
>>>> "/ mnt/vol/v1/etc/ bind/n1/keys" "/etc/bind/keys" ; ln -s "/ 
>>>> mnt/vol/ v1/etc/bind/n1/ zones" "/etc/bind/zones" ;
>>>>
>>>> • "n2" using "/mnt/vol/v1/etc/bind/n2/keys" dir & “/mnt/vol/v1/ 
>>>> etc/ bind/n2/zones” dir . & created symlinks as shown above.
>>>>
>>>> • "n3" using "/mnt/vol/v1/etc/bind/n3/keys" dir & "/mnt/vol/v1/ 
>>>> etc/ bind/n3/zones" dir. & created symlinks as shown above.
>>>>
>>>> • Added permissions in AppArmor "/etc/apparmor.d/local/ 
>>>> usr.sbin.named" file, for BIND/named, so that BIND/named can 
>>>> use "v1" replicated-volume "/mnt/vol/v1" BIND directories : / 
>>>> mnt/vol/ v1/etc/bind  , /mnt/vol/v1/etc/bind/zones , /mnt/vol/ 
>>>> v1/etc/bind/ keys , /mnt/vol/v1/etc/bind/n1/zones , /mnt/vol/ 
>>>> v1/etc/bind/n1/ keys in n1 server ( and i have done similar for 
>>>> n2 & n3 )  . Then applied changes with command : 
>>>> apparmor_parser -r /etc/ apparmor.d/ usr.sbin.named
>>>>
>>>> • Also applied or re-checked if the ownership-&-permission 
>>>> (O&P) convention used+recommended by BIND/named for directories 
>>>> & files, are applied/done on the dirs+files inside the "/mnt/ 
>>>> vol/ v1/etc/ bind, etc.
>>>>
>>>>
>>>>
>>>> DNSSEC & DNS:
>>>>
>>>> • Each nameserver has BIND DNS server named daemon software . 
>>>> Each BIND need to be Authoritative for my domains 
>>>> ("example.com" , "example2.com", etc) & response back to any 
>>>> DNS servers/clients query for my domains & for my subnet's 
>>>> reverse- zone . And each BIND DNS server also need to serve/ 
>>>> perform as a recursive DNS resolver for any queries made into 
>>>> "localhost" ( 127.0.0.1 , ::1 ).
>>>>
>>>> • Followed various related steps as-much-possible from "DNSSEC 
>>>> Howto for BIND 9.9+ <https://wiki.debian.org/ 
>>>> DNSSEC%20Howto%20for%20BIND%209%2E9%2B>” , ISCBIND docs for 
>>>> 9.18.28 <https://downloads.isc.org/isc/bind9/9.18.28/doc/arm/ 
>>>> html/> ( that i'm using now while writing this msg ) , etc . 
>>>> Debian OS will update BIND in distro’s repo, & then my/op's 
>>>> servers will be updated to that version . For next/latest 
>>>> version, goto ISC BIND download pagehere <https://www.isc.org/ 
>>>> download/>, search for "PDF" word, select/click on the HTML / 
>>>> PDF doc version that you want to read/follow.
>>>>
>>>> • As each nameserver has different IP-addresses, so i've kept 
>>>> the "named.conf", "named.conf.local", "named.conf.options" 
>>>> files in the /etc/bind of server itself, for faster loading .
>>>>
>>>> • The "named.conf" file has BIND ACLs, and include directives . 
>>>> This file has same content in each server . Has “ acl 
>>>> LocalHostR { 127.0.0.1; ::1; } ; LocalHostRv4 { 127.0.0.1; } ; 
>>>> LocalHostRv6 { ::1; } ; acl BlockedNets { 0.0.0.0/8 ; 
>>>> 192.0.2.0/24 ; 224.0.0.0/3 ; 10.0.0.0/8 ; 172.16.0.0/12 ; 
>>>> 192.168.0.0/16 ; } ; acl N1-IPv4 { 192.10.2.11; } ; acl N1-IPv6 
>>>> {2001:db8:1::11; }; ” , etc, (remove quote symbols) . ( i added 
>>>> more IPv4 & IPv6 in BlockedNetslater ).
>>>>
>>>> • The "named.conf.local" file has forward zones & reverse zones 
>>>> declarations : each zone has "type primary;" directive/option 
>>>> set , each zone has "file" directive with file located in 
>>>> replicated volume location . Each local zones & each local 
>>>> reverse-zone for IP-address) have "allow-query 
>>>> { LocalHostR; };" . My each domain's zone (i.e: "zone 
>>>> "example.com" { ... };") declarations, & reverse- zone for my 
>>>> own subnet, has "allow-query { any; };" .
>>>>
>>>> •My domain “example.com”zone declaration in “named.local.conf” 
>>>> file : “ zone "n1.example.com" { type master ; file "/mnt/vol/ 
>>>> v1/ etc/bind/zones/db.example.com" ; allow-query { any; } ; 
>>>> serial- update-method unixtime ; key-directory "/mnt/vol/v1/ 
>>>> etc/bind/n1/ keys" ; dnssec-policy opPolicy ; inline-signing 
>>>> yes ; notify no ; }; ” (remove quote symbols) . We allowed 
>>>> query from anyone . By the way, i also have a sub-domain zone 
>>>> declared in “named.local.conf” file as zone : “ zone 
>>>> "ns.example.com" { … }; ” , nearly same as “example.com”.
>>>>
>>>> • The "named.conf.options" file has “dnssec-policy 
>>>> "opPolicy" { ... };”  , "options { ... };" , "logging { ... };" 
>>>> sections/declarations . Logging uses the server's "/var/log/ 
>>>> named" dir ( into "Update_Debug.log" , "Security.log" , 
>>>> "BIND.log" files ).
>>>>
>>>> • The “options { … };” in “named.conf.options” file :  “ 
>>>> options { recursion yes ; allow-recursion { LocalHostR; } ; 
>>>> allow-query- cache { LocalHostR; } ;allow-query-cache-on 
>>>> { LocalHostR; } ; allow-query { LocalHostR; } ; allow- 
>>>> recursion-on { LocalHostR; } ; empty-zones-enable yes ; 
>>>> blackhole { BlockedNets; } ; allow- transfer { none; } ; auth- 
>>>> nxdomain no ; listen-on { N1-IPv4; LocalHostRv4; } ; listen-on- 
>>>> v6 { N1-IPv6; LocalHostRv6; }; rate- limit { ... }; }; 
>>>> ” (remove quote symbols). We restricted recursion by allowing 
>>>> only LocalHostR, not external, not BlockedNets.
>>>>
>>>> • DNS server, for non-dnssec part of DNS related queries & 
>>>> responses for domain(s), IPv4-adrs, IPv6-adrs, etc (forward 
>>>> lookup/ resolve , subnet IP-adrs reverse resolve/lookup ) 
>>>> WORKING FINE , from n1 & n2 & n3 . Authoritative mode is 
>>>> working for my domains . And "localhost" inside server can also 
>>>> provide website- name/domain- name To IP-address resolve 
>>>> response, to the the local software/ daemons/clients that are 
>>>> running inside server.
>>>>
>>>> • To ENABLE DNSSEC : i add "dnssec-validation auto;" inside 
>>>> "options" inside "named.conf.options" file , i add "key- 
>>>> directory "/mnt/vol/v1/etc/bind/n1/keys" ; inline-signing yes;" 
>>>> in "zone "example.com" { ... };" in "named.conf.local" file, 
>>>> etc , ( changed the "n1" into "n2" for "n2" server, & similarly 
>>>> in n3. )
>>>>
>>>> • For "dnssec-policy" directive about KSK & ZSK cert+key 
>>>> creation, usage period, signing, validity, verification, etc , 
>>>> i'm using shorter TTL period, etc , so-that dnssec/dns config 
>>>> lines can be changed+applied quickly during DNSSEC setup 
>>>> phase : dnssec-policy "opPolicy" {  ksk lifetime P88D algorithm 
>>>> RSASHA256 ;  zsk lifetime 22D algorithm RSASHA256 ; dnskey-ttl 
>>>> PT10M ;  publish-safety P2D ;  retire-safety P3D ;  purge-keys 
>>>> P3D ; signatures-refresh P5D ;  signatures-validity 
>>>> P10D ;  signatures-validity-dnskey P11D ; max-zone-ttl 
>>>> PT30M ;  zone-propagation-delay PT1H ;  parent- ds-ttl 
>>>> PT1H ;  parent-propagation-delay PT1H ; nsec3param iterations 0 
>>>> optout yes salt-length 0 ;  };
>>>>
>>>>
>>>> Now finally into the
>>>>
>>>> QUESTIONS:
>>>>
>>>> • How can i create 1 KSK key ( in "n1" server first ), for a 
>>>> (single) domain ("example.com") and get the DS code from KSK 
>>>> key and add that 1 DS in domain-provider ( to send to the 
>>>> TLD ), & configure other 2 nameservers ( n2 , n3 ) to use that 
>>>> 1 DS record from TLD & use that same/common 1 KSK file from the 
>>>> synced/ replicated directory, while "type master;" is set for 
>>>> my domain/ zone in each nameserver ?
>>>>
>>>> ( Using 3 KSK & their 3 DS in domain-provider did not work, 
>>>> created error indicators in DNSViz & in "DNSSEC- 
>>>> Annalyzer.VerisignLabs” test sites, when each nameserver used 
>>>> separate directories, files, etc.
>>>>
>>>> • if i specify same/COMMON (replicated) dir "/mnt/vol/v1/etc/ 
>>>> bind/ keys" & "/mnt/vol/v1/etc/bind/zones/zonename" inside 3 
>>>> nameserver's “named.conf.local” file domains/zones , Can BIND 
>>>> DNS server add their own RRSIG response/lines for DNS records 
>>>> (into same zone file) without removing earlier or other 
>>>> nameserver's RRSIG lines (unless related ZSK key/period expired) ?
>>>>
>>>> • How do i disable/clean/move/backup earlier DNSSEC keys/ 
>>>> usages, & setup DNSSEC completely as anew . ( Our zone TTLs are 
>>>> short 7m to 1h , (during setup/test phase) , So within 7m to an 
>>>> hour, all older-records should be discarded from caches. )
>>>>
>>>>
>>>> Thanks in advance for helpful responses.
>>>>
>>>> Erik.
>>>>
>>>> Erik T Ashfolk.
>>>>
>>>
>>