Sporadic Timeouts after upgrading to bind9.20

Klaus Darilion klaus.darilion at nic.at
Fri Sep 6 07:04:51 UTC 2024 

Hi Ondřej!

I play around with eu-stack. When I call  eu-stack -p 1605200 -v (during normal operations) the stacktrace looks meaningless to me (See below). Do I need a certain parameter or do I have to install debug symbols?

Thanks
Klaus

# eu-stack -p 1605200 -v
PID 1605200 - process
TID 1605200:
#0  0x00007b8ceb529ee0     epoll_pwait - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec52c9fa - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#2  0x00007b8cec513ce8 - 1 uv_run - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec6708d1 - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#4  0x000057c70ed697a6 - 1 main - /usr/sbin/named
#5  0x00007b8ceb42a1ca - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00007b8ceb42a28b - 1 __libc_start_main - /usr/lib/x86_64-linux-gnu/libc.so.6
#7  0x000057c70ed6a175 - 1 _start - /usr/sbin/named
TID 1605201:
#0  0x00007b8ceb52725d     syscall - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec0418ec - 1 - /usr/lib/x86_64-linux-gnu/liburcu-cds.so.8.1.0
#2  0x00007b8cec041da5 - 1 - /usr/lib/x86_64-linux-gnu/liburcu-cds.so.8.1.0
#3  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#4  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1605202:
#0  0x00007b8ceb529ee0     epoll_pwait - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec52c9fa - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#2  0x00007b8cec513ce8 - 1 uv_run - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec6708d1 - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#4  0x00007b8cec68502a - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#5  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1605207:
#0  0x00007b8ceb529ee0     epoll_pwait - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec52c9fa - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#2  0x00007b8cec513ce8 - 1 uv_run - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec6708d1 - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#4  0x00007b8cec68502a - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#5  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1605208:
#0  0x00007b8ceb529ee0     epoll_pwait - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec52c9fa - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#2  0x00007b8cec513ce8 - 1 uv_run - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec6708d1 - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#4  0x00007b8cec68502a - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#5  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1605209:
#0  0x00007b8ceb529ee0     epoll_pwait - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec52c9fa - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#2  0x00007b8cec513ce8 - 1 uv_run - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec6708d1 - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#4  0x00007b8cec68502a - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#5  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1605210:
#0  0x00007b8ceb529ee0     epoll_pwait - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec52c9fa - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#2  0x00007b8cec513ce8 - 1 uv_run - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec6708d1 - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#4  0x00007b8cec68502a - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#5  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1605211:
#0  0x00007b8ceb529ee0     epoll_pwait - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec52c9fa - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#2  0x00007b8cec513ce8 - 1 uv_run - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec6708d1 - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#4  0x00007b8cec68502a - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#5  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1605212:
#0  0x00007b8ceb529ee0     epoll_pwait - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec52c9fa - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#2  0x00007b8cec513ce8 - 1 uv_run - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec6708d1 - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#4  0x00007b8cec68502a - 1 - /usr/lib/x86_64-linux-gnu/libisc-9.20.1-1+ubuntu24.04.1+deb.sury.org+1-Ubuntu.so
#5  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1605213:
#0  0x00007b8ceb52725d     syscall - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8cec0a6a85 - 1 - /usr/lib/x86_64-linux-gnu/liburcu.so.8.1.0
#2  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#3  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1606172:
#0  0x00007b8ceb498d61     - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8ceb49b7dd - 1 pthread_cond_wait - /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007b8cec5253ad - 1 uv_cond_wait - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec5177fe - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#4  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1606173:
#0  0x00007b8ceb498d61     - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8ceb49b7dd - 1 pthread_cond_wait - /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007b8cec5253ad - 1 uv_cond_wait - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec5177fe - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#4  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1606174:
#0  0x00007b8ceb498d61     - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8ceb49b7dd - 1 pthread_cond_wait - /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007b8cec5253ad - 1 uv_cond_wait - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec5177fe - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#4  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1606175:
#0  0x00007b8ceb498d61     - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8ceb49b7dd - 1 pthread_cond_wait - /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007b8cec5253ad - 1 uv_cond_wait - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec5177fe - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#4  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1606176:
#0  0x00007b8ceb498d61     - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8ceb49b7dd - 1 pthread_cond_wait - /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007b8cec5253ad - 1 uv_cond_wait - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec5177fe - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#4  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1606177:
#0  0x00007b8ceb498d61     - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8ceb49b7dd - 1 pthread_cond_wait - /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007b8cec5253ad - 1 uv_cond_wait - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec5177fe - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#4  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1606178:
#0  0x00007b8ceb498d61     - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8ceb49b7dd - 1 pthread_cond_wait - /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007b8cec5253ad - 1 uv_cond_wait - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec5177fe - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#4  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
TID 1606179:
#0  0x00007b8ceb498d61     - /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007b8ceb49b7dd - 1 pthread_cond_wait - /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007b8cec5253ad - 1 uv_cond_wait - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#3  0x00007b8cec5177fe - 1 - /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
#4  0x00007b8ceb49ca94 - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007b8ceb529c3c - 1 - /usr/lib/x86_64-linux-gnu/libc.so.6



--
Klaus Darilion, Head of Operations
nic.at GmbH, Jakob-Haringer-Straße 8/V
5020 Salzburg, Austria

From: Ondřej Surý <ondrej at isc.org>
Sent: Wednesday, September 4, 2024 7:23 PM
To: Klaus Darilion <klaus.darilion at nic.at>
Cc: bind-users at lists.isc.org
Subject: Re: Sporadic Timeouts after upgrading to bind9.20

Klaus,

is that recursive or authoritative? Anything unusual like RPZ or catz?

Try snapshoting the call stack with eu-stack and save the one when the timeout happens.

Ondrej
--
Ondřej Surý — ISC (He/Him)

My working hours and your working hours may be different. Please do not feel obligated to reply outside your normal working hours.


On 4. 9. 2024, at 19:06, Klaus Darilion via bind-users <bind-users at lists.isc.org<mailto:bind-users at lists.isc.org>> wrote:

Hello!

On our production name servers we have check every 30s if bind is alive by sending a SOA query to bind. Today I upgraded a few nodes from 9.18.x (x between 17 and 27) to 9.20.1 (Ubuntu 24.04 with packages from ISC ppa).

Since that, we have sporadic timeouts (3s). On the nodes with more qps we see it more often.

Before I dig into the problem, are there any specific changes to 9.20 that I should look at? Maybe some default value changes for socket buffers, thread handling …?

Thanks
Klaus

--
Klaus Darilion, Head of Operations
nic.at GmbH, Jakob-Haringer-Straße 8/V
5020 Salzburg, Austria

--
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users at lists.isc.org<mailto:bind-users at lists.isc.org>
https://lists.isc.org/mailman/listinfo/bind-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20240906/61205ac1/attachment-0001.htm>

More information about the bind-users mailing list