[kea-dev] update on this ....FRe: thoughts on 3601

[Tomek Mrugalski]

On 30/10/15 19:57, Thomas Markwalder wrote:
> So there is a minor twist to the VersionedCSVFile approach.
> 
> Currently, it allows one to receive current schema data from
> a file containing prior schema records. For example
> if the current schema is 3.0 and Kea is given a 1.0 lease file:
> 
> 1.0 file --> VersionedCSVFile --> 3.0 data ---> MemfileLeaseMgr
> 
> This will occur automatically whenever MemfileLeaseMgr loads the
> lease file(s).  Currently, this happens inside the MemfileLeaseMgr
> constructor.
> 
> At this point, however, the physical lease file(s) are still 1.0 form.
> The header row has the list of 1.0 columns, the data rows have 1.0 data.
> Any new records written to the file will be in 3.0 form.  Operationally,
> this is fine although not optimal. VersionedCSVFile can load files that
> have mixed generation data.  In fact, such a file could contain 1.0, 2.0,
> and 3.0 records and it will load correctly.  Whenever the file is loaded,
> older schema records are upgraded, current schema records pass thru. Kea
> always gets all records in current schema form.
> 
> The twist is how do we physically convert the files?
Do I understand it correctly that once you load 1.0 file into Kea
version that expects 3.0 data, it will append 3.0 data to a file that
has 1.0 header? If that is so, I think it's a very bad idea. It would
make the lease file inconsistent and difficult to process by any
software that happens to not be Kea. That was one of the reasons why we
decided to use CSV in the first place: to be able to process the file by
other software.

I think that once the version inconsistency is detected, the software
should rewrite the whole CSV file to a new format. This will introduce
an extended startup time after Kea upgrade, but that's perfectly ok.
Software upgrades are infrequent (and planned) events.

> The most obvious answer is to let LFC take care of it.  LFC will convert
> them automatically.  Old schema records go in, current scheme lease files
> are written out.  Of course this only happens if LFC is enabled or they
> run it manually.
> 
> Keeping in mind the VerionedCSVFile can expose the knowledge that the data
> required updating, what should we do if we detect at start up that
> LFC is disabled AND the files need upgrading?
> 
> 1. We log an error message, stating they should enable LFC or run it
> manually.
> 
> We make sure document by example how to do this for the purposes of
> upgrading.
> (We should probably do this anyway)
> 
> 2. We trigger a one-shot run of LFC when Memfile lease manager starts up
> IF we detect an upgrade is needed AND LFC is disabled.
I would go even one step further and refuse to start the server until
the LFC upgrade run is completed. It's much better to have couple
seconds downtime that is planned and avoidable if needed. Does 10
seconds pause bothers you? Then don't upgrade at this time. The
alternative, to have an inconsistent database, is much worse.

On a related note, I think having Kea the ability to upgrade on load is
fine, but we still should provide kea-admin upgrade capability, even if
it only calls kea-lfc. This is purely from operational perspective.
Without it, one could not upgrade the database without running the DHCP
service. Update and providing service are two different operations and
there may be valid scenarios, where you want to just do the upgrade, but
not run the service.

Tomek