[stork-users] stork-server choking on rpc message size from BIND

[Slawek Figiel]

Hi,

thanks for your report.

I agree the problem is the same as in #398. The named statistics are 
exchanged between the server and agent in uncompressed form. It causes 
the data size may exceed the GRPC limit if there are too many DNS zones. 
We should implement the approach from #398 for named-related traffic and 
compress it with gzip.

Regards,
Slawek

On 10/06/2023 13:05, Darren Ankney wrote:
> Hi,
>
> The two issues do sound similar.  I suggest you create an account on
> gitlab.isc.org and report this issue there in a new issue.  Feel free
> to mention the old closed issue as well, as you've done here.  It
> would be helpful if you attached your two named.conf files and
> statistics output as well.  Perhaps have a look through the files and
> anonymize as necessary.  The named.conf files, you can easily do this
> by retrieving with `named-checkconf -px` and then obfuscating any
> further info that you want hidden (zones etc...).
>
> Thank you,
>
> Darren Ankney
>
> On Fri, Jun 9, 2023 at 7:20 PM vom513 <vom513 at gmail.com> wrote:
>> Hello all,
>>
>> Running Stork 1.11.0.
>>
>> I have Stork happy monitoring Kea.  In my lab - that’s all I had (no BIND installed there).
>>
>> Now I have it on my prod boxes which also run BIND.  I fought through some config/rndc/etc issues and seem to have gotten that mostly sorted.
>>
>> However, of my two BIND boxes, one runs views and the other doesn’t.  The one that runs views seems to be overrunning the max. message size or something ?  The one that doesn’t have views populates it’s cache hit ratio and doesn’t log the below errors.
>>
>> Jun  9 18:58:37 ice stork-server[1280317]: time="2023-06-09 18:58:37" level="warning" msg="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5625179 vs. 4194304)" file="          manager.go:110  " agent="1
>> 27.0.0.1:8081"
>> Jun  9 18:58:37 ice stork-server[1280317]: time="2023-06-09 18:58:37" level="warning" msg="Failed to send the following named statistics command: " file="           grpcli.go:328  " agent="127.0.0.1:8081" stats URL="http://127.0.0.1:8053/
>> json/v1"
>> Jun  9 18:58:37 ice stork-server[1280317]: time="2023-06-09 18:58:37" level="warning" msg="Problem retrieving stats from named: failed to send named statistics command via the agent 127.0.0.1:8081, the agent is still not responding" file=
>> "         appbind9.go:59"
>>
>>
>> And sure enough:
>>
>> vom at ice:~$ curl -s http://127.0.0.1:8053/json/v1 | wc -c
>> 5622336
>>
>> vs.
>>
>> vom at ns2:~$ curl -s http://127.0.0.1:8053/json/v1 | wc -c
>> 526111
>>
>> So the ice machine has a json stats response that is 10 times bigger (?!)
>>
>> Perhaps I have an issue with BIND - but other than views - these two boxes are pretty much the same config.  Not sure what I could possibly be doing wrong there...
>>
>> Anyway, I found this:
>>
>> https://gitlab.isc.org/isc-projects/stork/-/issues/398
>>
>> Seems like the same issue, except I’m hitting it with BIND and not Kea.
>>
>> Any info or advice would be appreciated.  Thanks.
>>
>> --
>> Stork-users mailing list
>> Stork-users at lists.isc.org
>> https://lists.isc.org/mailman/listinfo/stork-users