[bind10-dev] revised/refactored data source design proposal

[JINMEI Tatuya / 神明達哉]

At Sat, 4 Jun 2011 12:39:54 +0200,
Michal 'vorner' Vaner <michal.vaner at nic.cz> wrote:

> On Fri, Jun 03, 2011 at 11:36:00AM -0700, JINMEI Tatuya / 神明達哉 wrote:
> > BTW, are you saying this is what we decided at the last f2f?  If so,
> > maybe I really didn't understand the seeming consensus correctly (I'm
> > not necessarily opposing to the idea itself, although it seems more
> > ambitious and will have more technical challenges).
> 
> Well, AFAIK we agreed on that something in this direction would be a good way to
> solve many problems. So, while we didn't yet commited to writing it that way, I
> kind of expected this is the way we're considering right now. And we didn't
> specify full details, of course, and if there's update only through the DB or
> not is definitely a detail, but that's the easiest way I personally see. I might
> have misunderstood.

Okay, clarifying this (at least what we actually discussed at that
time) is probably a good agenda item for the next bi-weekly call.

> > If I remember correctly, SQLite3 doesn't even allow that (MySQL allows
> > it with the condition you mentioned), but as I will note below I think
> > we can find a way to work this around if it's really needed.
> 
> Really? How it can know it moved from one thread to another except
> by explicitly asking for the thread ID (and doing so explicitly
> looks like a stupid way to introduce restriction on use). But if it
> really is there, then the pool approach might be problematic.

Hmm, apparently my information was old.  I re-googled it and found
this: http://www.sqlite.org/faq.html which states:

"Prior to version 3.3.1, an sqlite3 structure could only be used in
the same thread that called sqlite3_open() to create it...The
restriction on moving database connections across threads was relaxed
somewhat in version 3.3.1. With that and subsequent versions, it is
safe to move a connection handle across threads as long as the
connection is not holding any fcntl() locks. "

Anyway, I suspect this point is not that critical as I think we
already agreed.

> > One of my concerns in this case is if we pass a connection state
> > holding an open transaction to an application, it will open up a wider
> > possibility of starvation due to a bug in the application side.
> > Another concern when we use the entire "client" for the transaction,
> > it may cause a confusing results such as the application tries to
> > start a writable transaction using the same client and blocks due to
> > its own read transaction.
> 
> Do DBs really distinguish between read transaction and write
> transaction on the user side? I never heard of such thing.

I don't understand what you mean or even which point of my concerns
you're referring to...basically my concern is that it will increase
the possibility of bug like this: http://bind10.isc.org/ticket/730

> > > I don't know, they are probably effective enough in the big DBs,
> > > they just start sending you sending row after row and you read them
> > > one by one. I can't imagine anything more effective in the DB world,
> > > but I'm not expert there at all.
> > 
> > I'm not sure we're talking about the same thing...is your comment
> > about the "another discussion"?  If so, yes, "they just start sending
> > you row after row and you read them one by one".  The problem is it
> > may take longer time, during which the database is read-locked and
> > cannot be updated.
> 
> Ah, well, while being unimportant, I believe they don't really do
> such thing.  They tried to teach us how DBs work inside and insisted
> on some keeping copies when rewriting it and something still reads
> the old state, and not really locking in most cases (but I have some
> doubts about how real the stuff they were telling was, some of the
> things didn't really make sense).

I admit I'm not a DB specialist, so there may be a way to work it
around.  But my simple experiments supported my concern.  If some
client does:

  mysql_query(conn, "SELECT name, rdtype, ttl, rdata "
                    "FROM records WHERE zone_id=1;");
  result = mysql_use_result(conn);

and tries to get RRs via mysql_fetch_row(result), other process or
thread cannot change (they should be able to start a write transaction
but cannot complete committing it) the database until this client
performs "mysql_free_result(result)".

A common practice in such a case is to use mysql_store_result()
instead of mysql_use_result(conn), but I suspect it's also problematic
for a huge zone because the resulting data can be very big and may
even bigger for the reasonably available memory for the server.

---
JINMEI, Tatuya
Internet Systems Consortium, Inc.