[kea-dev] Lease File Cleanup in Kea - Design Document

[Shawn Routhier]

On Jan 6, 2015, at 2:41 AM, Chaigneau, Nicolas wrote:

> 
> 
> Hello Marcin,
> 
> 
> Thanks for sharing this design document.
> You'll find my comments hereafter.
> 
> Regards,
> Nicolas.
> 
> 
> 
> 
> 1) about the clean-up algorithm
> 
>> The cleanup of any file by the kea-lfc is achieved by removing lease entries which have expired, i.e. "expired" field holds the time later than the current time minus an arbitrary value to account for a possible positive time skew.
> 
> 
> From what I understand of this, the LFC will simply keep lines for which the "expired" field is not reached yet (and remove the others), regardless of the lease actual status.
> This implies that:
> - If for a given lease, a DHCP Release has been processed, only the related entry in the file is cleaned up. Other entries (related to the initial request, and renew - if any) associated to this given lease are maintained until they also expire, even though there is no such lease anymore.
> - If a client sends renews, they will all be kept until they individually expire. For a typical renew-timer set to half of lease duration, this means that for an active lease there will be two entries in the lease file. Also, faulty devices may send many renews, in which case we would have as many non expired entries for the same lease.
> 
> Is that correct ?

I think that would be correct in some readings of the text.  We had a short chat about releases this morning and have two different
options for handling them (and there may be others)
1) My preferred one is that the release entry will also contain the expiration time the same as other lease entires.  It will then get removed when the
expiration time comes. So if I have a client that gets a lease that expires at 10 and the client release it at 5 I would have an entry for 
(active, expires at 10 ….)
and
(released, expires at 10 ….)
Both of these would be removed then the LFC is run after 10.  Before then if the server is stopped and restarted it will be able to determine that
the lease was scheduled to expire at 10 but has already been released and therefore could be reused.  This does require the release entry to include
the expiration time, I'm not sure if that is already true or not.
2) Another option is that the release entries aren't removed until they have migrated to the beginning of the file.  That is if there are any normal entires
before we get to a release we don't remove the lease.  Eventually all of the normal entires would expire and the release entry would be first, then the
next time the LFC runs it would remove the release entry.

In both cases we end up keeping entries around that aren't strictly necessary but it will make the processing simpler.
> 
> From the requirements page, I understood that all redundant entries would be removed.
> To achieve this I believe you would need to keep in memory the list (maybe as a hash ?) of active leases along with their expiration time, which you would update while parsing <previous lease file> and <lease file copy>. Only when the parsing is complete could <LFC output file> be created.
> 
> This would also address the issue you noted in the case of a crash between renaming <LFC output file> and deleting <lease file copy>.
> 
> That said, I'm not sure this is really indispensable. The proposed algorithm ensures that all redundant information will eventually be cleaned up. This may take a while though, depending on the configured lease time. In my case I plan to have a relatively short lease time so that would probably not be too much of an issue.

I believe you are correct.  It would save some space and time at restart if we were to try and remove redundant information.  But that would complicate the
process and for now it doesn't seem useful or necessary.  The server is going to need to handle redundant information in any case (imagine that it is restarted
before the LFC runs).
> 
> If it's too complicated to handle, maybe this could be implemented in an ulterior version of LFC ?

I think the proper way to approach this is to write the simpler version first and see how it works.  If we get feedback indicating that the files have large amounts
of redundant information we can update the LFC (or provide another one) at that time.  
> 
> (would that be complicated ? I guess Kea will have to read the two files anyway so it can populate its lease backend. Maybe this is reusable for kea-lfc ?)
> 
> 
> 
> 
> 2) about kea-lfc usage
> 
>> kea-lfc [-4|-6] -p <previous-lease-file-name> -i <lfc-input-lease-file-name> -o <output-file-name> -c <kea-configuration-file>
> 
> The example allows to specify the three file names.
> However according to "how the name is constructed", I believe these are not necessary, since they are automatically constructed by Kea (from the actual lease file name).
> Moreso, it is required for "output" and "previous" files to be on the same filesystem (otherwise the rename would be a copy).
> 
> Consequently, only <kea-configuration-file> seems necessary to me. It allows to get the actual lease file name, and derive the name of the related lease files.

I agree that we could reduce it to a smaller number of file names as input.  I'm not sure what Marcin's thinking was on that. 

But writing the program to expect the multiple file names does mean that we can change the names more easily in the future 
and having only one place where the names are generated is likely to make for more consistent usage over time.
> 
> 
> 
> 
> 
>> Hi,
>> 
>> Following the thread regarding the requirements for Lease File Cleanup in Kea (https://lists.isc.org/pipermail/kea-dev/2014-December/000211.html), I have created a short design document for this feature:
>> 
>> http://kea.isc.org/wiki/LFCDesign
>> 
>> I have to admit that I was in rush preparing the document, so please forgive me if anything is underspecified or unclear. I will be happy to clarify if you point out the specific paragraphs that should be made clearer.
>> 
>> The last section of this document proposes the split of implementation into actual tickets. It goes along with the initial estimates for the implementation work (specified in days).
>> 
>> Please review as soon as possible because we need to incorporate this in our release schedule this week.
>> 
>> Thanks,
>> Marcin
>> 
> 
> 
> This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
> 
> _______________________________________________
> kea-dev mailing list
> kea-dev at lists.isc.org
> https://lists.isc.org/mailman/listinfo/kea-dev