dhcpd problem on virtual machines

[Alex Zeffertt]

Jeffrey Hutzelman wrote:
> --On Thursday, April 15, 2010 02:04:40 PM +0100 Alex Zeffertt 
> <alex.zeffertt at eu.citrix.com> wrote:
> 
>> There are several possible solutions to this, and I'm keen to find out
>> which of these - if any - seems the best to you:
>>
>>   i)   Always use AF_PACKET in dhclient
>>   ii)  Use AF_INET in dhcpd for receiving packets instead of AF_PACKET (*)
>>   iii) Don't verify the checksum in dhcpd if the hardware supports
>> checksum offload
> 
> There's always:
> 
> iv) fix the virtualization layer so that it doesn't pass legitimate UDP 
> traffic with incorrect but nonzero checksums.
> 
> -- Jeffrey T. Hutzelman (N3NHS) <jhutz+ at cmu.edu>
>    Carnegie Mellon University - Pittsburgh, PA
> 
> 


Unfortunately this is an intended feature of the virtualisation support in the 
linux kernel, as the following snippet from linux-2.6.32/include/linux/skbuff.h 
shows:


/* A. Checksumming of received packets by device.
  *
  *      NONE: device failed to checksum this packet.
  *              skb->csum is undefined.
  *
  *      UNNECESSARY: device parsed packet and wouldbe verified checksum.
  *              skb->csum is undefined.
  *            It is bad option, but, unfortunately, many of vendors do this.
  *            Apparently with secret goal to sell you new device, when you
  *            will add new protocol to your host. F.e. IPv6. 8)
  *
  *      COMPLETE: the most generic way. Device supplied checksum of _all_
  *          the packet as seen by netif_rx in skb->csum.
  *          NOTE: Even if device supports only some protocols, but
  *          is able to produce some skb->csum, it MUST use COMPLETE,
  *          not UNNECESSARY.
  *
  *      PARTIAL: identical to the case for output below.  This may occur
  *          on a packet received directly from another Linux OS, e.g.,
  *          a virtualised Linux kernel on the same host.  The packet can
  *          be treated in the same way as UNNECESSARY except that on
  *          output (i.e., forwarding) the checksum must be filled in
  *          by the OS or the hardware.
  *

I think what the above is saying is that if you have two virtualised Linux 
guests on the same host it would be wasteful to calculate UDP or TCP checksums 
on packets sent between them.  To avoid this the virtual network devices 
advertise that they support checksum offloading.  As a result the packets that 
are actually passed between the guests are partially checksummed - that is their 
checksum fields contain the checksum of the UDP or TCP pseudo header.

Now this is not a problem for AF_INET sockets since the receiving kernel knows 
that the frame is valid and therefore does not verify the checksum.  However, 
for AF_PACKET sockets the kernel does not even know that the packet is an IP 
packet and so it would be wrong for it to blank the UDP checksum (or in the case 
of TCP to calculate the correct checksum) before passing the frame to userspace.

To fix this in the "virtualisation layer" would mean

  i) removing the PARTIAL flag for received packets in the Linux networking stack
  ii) changing all virtual device drivers so that they calculate TCP checksums 
and blank or calculate UDP checksums on packets received from guests on the same 
host, and then apply a different flag.

This is not a great idea for a number of reasons.  Firstly it requires editing a 
number of drivers that are under different authors' control.  Secondly it 
creates  a lot of inefficiency (and avoiding this inefficiency was the goal of 
adding the flag in the first place).  And thirdly, you'd have to take on the 
Linux developers who made this design decision in the first place ;-)

However, fixing this in dhcpd/dhclient is much more straight forward.  We simply 
need to acknowledge that if the sender uses AF_INET then so should the receiver. 
  Or alternatively, if the receiver uses AF_PACKET then so should the sender.

Regards,

Alex