Tradindexed cache entries

[Julien ÉLIE]

Hi Russ,

> Yes, indeed, you're leaking open overview files out of the cache somehow.
> They're falling out of the hash table and hence never being found and
> closed until you run out of open file descriptors.
>
>> Is there something else to check?
>
> I think there's either a bug in the tradindexed cache code or in the hash
> table implementation, or possibly elsewhere in the server that's
> corrupting memory and happening to corrupt the hash table.

I think I have just found out what is going on.
tdx_cache_delete() does not decrement the known number of elements in the hash
table, even though they are removed from it.  So at some point, we end up
having 0 element in the hash, though the count is non zero.  Therefore, no
"oldest cache entry" is found.

This fix seems to solve the issue:

Index: storage/tradindexed/tdx-cache.c
===================================================================
--- storage/tradindexed/tdx-cache.c     (révision 8880)
+++ storage/tradindexed/tdx-cache.c     (copie de travail)
@@ -197,9 +197,12 @@
 void
 tdx_cache_delete(struct cache *cache, HASH hash)
 {
-    if (!hash_delete(cache->hashtable, &hash))
+    if (!hash_delete(cache->hashtable, &hash)) {
         warn("tradindexed: unable to remove cache entry for %s",
              HashToText(hash));
+    } else {
+        cache->count--;
+    }
 }



data_cache_reopen() calls:
 * tdx_cache_delete()
 * tdx_data_open()
 * tdx_cache_insert()

where tdx_cache_insert() clears out the oldest entry in the cache when
it is full.




To reproduce the problem:

* overcachesize: 4 in inn.conf

% lsof -c innd | grep overview
/home/news/spool/overview/group.index
/home/news/spool/overview/group.index
/home/news/spool/overview/f/s/z/fr.sci.zetetique.IDX
/home/news/spool/overview/f/s/z/fr.sci.zetetique.DAT
/home/news/spool/overview/n/l/f/news.lists.filters.IDX
/home/news/spool/overview/n/l/f/news.lists.filters.DAT
/home/news/spool/overview/f/t/fr.test.IDX
/home/news/spool/overview/f/s/p/fr.sci.philo.IDX
/home/news/spool/overview/f/s/p/fr.sci.philo.DAT
/home/news/spool/overview/f/t/fr.test.DAT

* post a supersedes in fr.test

% lsof -c innd | grep overview
/home/news/spool/overview/group.index
/home/news/spool/overview/group.index
/home/news/spool/overview/n/l/f/news.lists.filters.IDX
/home/news/spool/overview/n/l/f/news.lists.filters.DAT
/home/news/spool/overview/f/t/fr.test.IDX
/home/news/spool/overview/f/c/o/l/c/fr.comp.os.linux.configuration.IDX
/home/news/spool/overview/f/c/o/l/c/fr.comp.os.linux.configuration.DAT
/home/news/spool/overview/f/t/fr.test.DAT

We've lost one file.

* post two other supersedes in fr.test

% lsof -c innd | grep overview
/home/news/spool/overview/group.index
/home/news/spool/overview/group.index
/home/news/spool/overview/f/t/fr.test.IDX
/home/news/spool/overview/f/t/fr.test.DAT

And we have "tradindexed: unable to find oldest cache entry" for every
subsequent article received.



With the above patch, I cannot reproduce that scenario.




I have a remaining question:  is it normal that lsof returns deleted entries?
I for instance had a CAF file (I assume removed during expiry):
   /home/news/spool/articles/timecaf-02/9a/4b33.CF.20401 (deleted)
or also .IDX-NEW and .DAT-NEW files...  Are they supposed to remain visible
in lsof?

-- 
Julien ÉLIE

« As-tu jamais fait un rêve, Neo, dont tu avais l'impression
  qu'il était réel ? Qu'est-ce qui se passerait si tu ne te réveillais jamais
  de ce rêve, Neo ? Comment reconnaîtrais-tu le monde réel
  du monde imaginaire ? » (Morpheus, _Matrix_)