[bind10-dev] bind10.isc.org web server lagginess

[David W. Hankins]

On Tue, Sep 14, 2010 at 09:40:29AM -0500, Jeremy C. Reed wrote:
> I have now setup a robots.txt file.
> 
> http://bind10.isc.org/robots.txt

That should help!  One of the things I noticed while I was looking is
that some URI's, like /timeline*, appear to have a lock associated
with them.  Although I had multiple /timeline 'wget' commands running
(so I was sure it was not HTTP 1.1 connection sharing), only one
apache2 process was pegged at 100% CPU (despite multiple processes
and in-process threading in this configuration of apache (mpm_worker)),
and the wget's appeared to start getting data after the last one
finishes.  Some of them erred out with an 'unable to get database lock'
error.  So if one user hits /timeline, everyone else needing its
database lock hangs.  This may explain our perception that the bind10
site is slow; it's actually just head-of-line blocking.


I extended my one-line awk a little bit, let's see how it stacks up
now, and we can see how it changes after the robots.txt change;

root at bind10:/var/log/apache2# awk 'BEGIN { botsecs=0; totalsecs = 0; }  /.*(([bB][Oo][Tt])|([Cc][Rr][Aa][Ww][Ll])|([Ss][Ll][Uu][Rr][Pp])).*[0-9]$/ { botsecs += $(NF); split($7, dirs, "(/|\?)"); botpages[dirs[2]] += $(NF); isbot[$1] = 1; } /[0-9]$/ { totalsecs += $(NF); addresses[$1] += $(NF); split($7, dirs, "(/|\?)"); pages[dirs[2]] += $(NF); } END { printf("Top 10 pages:\nSEC %%tot %%bot First Directory of page\n"); for (page in pages) { if (pages[page] > 0) { printf("%3d (%d%%) (%d%%) /%s\n", pages[page], (pages[page] * 100) / totalsecs, (botpages[page] * 100) / pages[page], page) | "sort -n | tail -10"; } } close("sort -n | tail -10"); printf("---\n"); printf("Top 5 client addresses:\nSEC %%tot address \"(bot)\"\n"); for (addr in addresses) { if (addresses[addr] > 0) { printf("%3d (%d%%) %s %s\n", addresses[addr], (addresses[addr] * 100) / totalsecs, addr, isbot[addr] ? "(bot)" : "") | "sort -n | tail -5"; } } close("sort -n | tail -5"); printf("---\nBots:  %d (%d%%)\nTotal: %d\n", botsecs, (botsecs*100) / totalsecs, totalsecs); }' access.log ssl_access.log


Current output is below.  The 92% of bots out of the 45% of wait times
from bot crawling /timeline should be noticeable to us in usability.

Top 10 pages:
SEC %tot %bot First Directory of page
1517 (1%) (37%) /roadmap
2431 (1%) (86%) /query
3128 (2%) (0%) /login
3467 (2%) (97%) /wiki
6197 (4%) (100%) /log
7866 (5%) (99%) /browser
8289 (5%) (50%) /changeset
11333 (7%) (48%) /ticket
31054 (21%) (88%) /report
66375 (45%) (92%) /timeline
---
Top 5 client addresses:
SEC %tot address "(bot)"
4642 (3%) 195.169.221.157 (bot)
4732 (3%) 217.155.47.50 
6156 (4%) 66.249.67.112 (bot)
16127 (11%) 66.249.68.103 (bot)
83607 (57%) 67.195.110.150 (bot)
---
Bots:  118856 (81%)
Total: 145671

-- 
David W. Hankins	BIND 10 needs more DHCP voices.
Software Engineer		There just aren't enough in our heads.
Internet Systems Consortium, Inc.		http://bind10.isc.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/bind10-dev/attachments/20100914/7e822c99/attachment.bin>