Problem with multiple interfaces / multiple IPs per interface in linux, with dhcp server.
dhcp1 at thehobsons.co.uk
Fri Jul 14 10:36:39 UTC 2006
> > I assume you have a reason for this, it's very wasteful of addresses
>> (you lose 3 of each block of 8) but doesn't gain you anything traffic
> > wise. It's not wrong, I'm just curious.
>I have an unrecommended ethernet network topology, with 8 level
>cascaded without-management 8-port 100Mbit/s switches. As you can see
>(if you can imagine or draw on a paper), that if i have a big /24
>network and some computer on an edge wants to talk to some computer on
>the other edge its MAC address would configure all the uplink ports of
>all the switches that the message passes through. Now, if there are
>another pair of computers sharing a part of the net with the other
>two, the configured uplink MAC addresses no longer apply. So, I
>devided the network into networks serving each switch, so the uplink
>is always the gateway's MAC address. I know that I manged to load the
>server for every transfer in the net and to reduce at half the speed
>if two pairs of computers communicate simulaniously, but the fact is
>that the computers more often access the internet through the gateway,
>so I managed to reduce the arp traffic inside every micro-network with
>this hack. My Internet link is 512 kbit/s up and max 1Mbit/s down.
>I'm opened to sugestions if someone has another better ideea (not
>buying managed swithes).
Your division of the network by IP address won't make any difference
to any of the switches. I'll explain how switches learn the network
topology and then you'll understand why you are putting a lot of work
into achieving nothing ! Obviously there are details left out, but
the basic process is there.
A switch does not learn THE mac address of a connected device, it
learns ALL the mac addresses reachable through that port. If you look
at the specs for a switch, it will typically say something like "8192
entry mac table" - meaning that it has a table able to hold 8192 mac
addresses and the port the mac is reachable through. This value is
important as it sets an upper limit on the number of devices you can
have on your network - any more than that and the switch will run out
of space. in the table.
When you power on the switch, this mac table is empty. When a packet
is received (I'll ignore broadcasts for a moment), the switch looks
in the mac table for both the source and destination mac addresses
found in the packet. If the source address is not found, or the port
doesn't match, then the table is updated - the switch now knows which
port it can send traffic for that mac address via. At this point, the
destination mac address is not known - so the only thing the switch
can do is send the packet out on all the ports except the port it
came in on*. This will happen on every switch, so all connected
devices will see this packet.
When a reply is sent, the 'route' back to the original device is now
known - so the packet will only be passed out of the port through
which the device can be reached. But while passing the packet, the
switch will again check the source mac address and add it to the
mac-port map - so now it knows which port it can use to reach that
device. Future traffic between these devices will now only be passed
through those links needed - other links will not see the traffic.
Very quickly, each switch will learn the mac address of any connected
devices - and which port they can be connected via. This is not "one
mac per port", but will be "as many macs as can be reached via a
I mentioned above that I was ignoring broadcasts, now I will return
to them. A broadcast will have an all-ones mac address which the
switch will detect as being a broadcast - it will not try and store
this in it's mapping table, but it will know to send the packet out
through all connected links except the port it came in through. This
means that a broadcast packet will travel right through the network
and reach every device. Note this last sentence, it is important.
Once a device sends a broadcast packet, then barring faults, EVERY
switch in the network will learn it's mac address and the port via
which it can be reached. it does not matter how you connect the
switches, they will all learn and store the devices mac address.
Since (almost) all devices will broadcast a packet sometime, it's
inevitable that all the switches will get to see all mac addresses.
You see, before a device can send a packet to another device it must
know it's mac address - and typically it gets to find this out by
broadcasting an ARP packet ("I'm looking for the device with address
184.108.40.206"). In doing so, it has announced to the whole network that it
exists ! Even if you can avoid such ARP packets (not easy), there is
the minor detail of .... the device broadcasting to find a dhcp
Notice two things I haven't talked about - subnets or management !
Management is irrelevant to the basic mac learning process - it
provides the administrator with some tools, sometimes rather usefully
allowing you to query the mac table to find out where a device is !
Another facility you typically get with management is Spanning Tree
Protocol (STP). This allows you to build a network with redundant
links which would normally result in loops and the network quickly
dying as packets go round and round and round. STP takes care of
finding these loops and 'turning off' enough links to remove them -
but if a link or switch fails, STP can turn links back on to repair
the network (as far as possible) and minimise the impact.
As for IP addresses, the switches (at this level of sophistication)
do not look at that - they go no further into the packet than source
and destination mac.
Now, having told you why you are wasting your time with your multiple
subnets, let me tell you why you are correct to consider the switch
topology. Depending on your traffic flows, there can be benefits in
arranging devices such that high throughput conversations remain
within one switch.
Lets assume A & B are on one switch, and C & D are on another. A & B
can exchange their full 100mbps each way with each other. C & D can
also exchange 100mpbs each way between them.
However, if your conversations were A-C and B-D, then in between the
conversations would be sharing a single link, so the total would be
100mbps between them (or 50mbps each if the conversations were
'equal'). Note that in a fully switched network each link is full
duplex so these are 'each way' figures - the total throughput is
double these figures (ie 200mbps intra switch and
Ie A-->C plus B-->D is limited to 100mbps and C-->A plus D-->B is
limited to 100mbps.
This happens irrespective of IP address - you only need to consider
the traffic flows.
But consider if A&B are in the 220.127.116.11/29 subnet, and C&D are in the
18.104.22.168/29 subnet. Your conversations between A&C and B&D must now go
via the router - so instead of having a link carrying A-->C plus
B-->D traffic, you have the router link carrying A-->C, B-->D, plus
C-->A and D-->B traffic on both it's inbound and outbound wires. Thus
you have now reduced your throughput potentially to 25mbps each way
per conversation !
In practical terms, on real networks, there are other factors that
tend to limit throughput. Certainly for "general office use" you are
unlikely to have any problems however you arrange the network. It
only becomes crucial if you have a small number of high bandwidth
users (such as a CAD or Design dept moving huge files around) - then
it can be useful to give them their own switch so their traffic is
kept largely separate from the rest of the network. If you have a
central file server, give it a gigabit NIC and connect it to a
gigabit port on the switch at the head of the tree - that way, there
are several 100mbps paths that can ultimately share that 1G pipe to
* It doesn't send the packet out via the port it came in on as that
would result in loops. If there is a single device connected, it
knows about the packet (it sent it !). If there is another switch
connected, it will already have taken care of sending the packet out
via the rest of it's own ports.
** There are some exceptions, some switches (rarely found these days)
were limited to one mac per port and were hardcoded that any traffic
for macs not in the limited list would be sent via a dedicated uplink
port. These switches can ONLY have an end device connected, you
cannot connect multiple devices (via another switch or hub) to the
downstream ports on them.
More information about the dhcp-users