Problem with multiple interfaces / multiple IPs per interface in linux, with dhcp server.

[Simon Hobson]

psihozefir wrote:

>  > I assume you have a reason for this, it's very wasteful of addresses
>>  (you lose 3 of each block of 8) but doesn't gain you anything traffic
>  > wise. It's not wrong, I'm just curious.


>I have an unrecommended ethernet network topology, with 8 level
>cascaded without-management 8-port 100Mbit/s switches. As you can see
>(if you can imagine or draw on a paper), that if i have a big /24
>network and some computer on an edge wants to talk to some computer on
>the other edge its MAC address would configure all the uplink ports of
>all the switches that the message passes through. Now, if there are
>another pair of computers sharing a part of the net with the other
>two, the configured uplink MAC addresses no longer apply. So, I
>devided the network into networks serving each switch, so the uplink
>is always the gateway's MAC address. I know that I manged to load the
>server for every transfer in the net and to reduce at half the speed
>if two pairs of computers communicate simulaniously, but the fact is
>that the computers more often access the internet through the gateway,
>so I managed to reduce the arp traffic inside every micro-network with
>this hack. My Internet link is 512 kbit/s up and max 1Mbit/s down.
>I'm opened to sugestions if someone has another better ideea (not
>buying managed swithes).

Your division of the network by IP address won't make any difference 
to any of the switches. I'll explain how switches learn the network 
topology and then you'll understand why you are putting a lot of work 
into achieving nothing ! Obviously there are details left out, but 
the basic process is there.



A switch does not learn THE mac address of a connected device, it 
learns ALL the mac addresses reachable through that port. If you look 
at the specs for a switch, it will typically say something like "8192 
entry mac table" - meaning that it has a table able to hold 8192 mac 
addresses and the port the mac is reachable through. This value is 
important as it sets an upper limit on the number of devices you can 
have on your network - any more than that and the switch will run out 
of space. in the table.


When you power on the switch, this mac table is empty. When a packet 
is received (I'll ignore broadcasts for a moment), the switch looks 
in the mac table for both the source and destination mac addresses 
found in the packet. If the source address is not found, or the port 
doesn't match, then the table is updated - the switch now knows which 
port it can send traffic for that mac address via. At this point, the 
destination mac address is not known - so the only thing the switch 
can do is send the packet out on all the ports except the port it 
came in on*. This will happen on every switch, so all connected 
devices will see this packet.

When a reply is sent, the 'route' back to the original device is now 
known - so the packet will only be passed out of the port through 
which the device can be reached. But while passing the packet, the 
switch will again check the source mac address and add it to the 
mac-port map - so now it knows which port it can use to reach that 
device. Future traffic between these devices will now only be passed 
through those links needed - other links will not see the traffic.

Very quickly, each switch will learn the mac address of any connected 
devices - and which port they can be connected via. This is not "one 
mac per port", but will be "as many macs as can be reached via a 
port"**.


I mentioned above that I was ignoring broadcasts, now I will return 
to them. A broadcast will have an all-ones mac address which the 
switch will detect as being a broadcast - it will not try and store 
this in it's mapping table, but it will know to send the packet out 
through all connected links except the port it came in through. This 
means that a broadcast packet will travel right through the network 
and reach every device. Note this last sentence, it is important.

Once a device sends a broadcast packet, then barring faults, EVERY 
switch in the network will learn it's mac address and the port via 
which it can be reached. it does not matter how you connect the 
switches, they will all learn and store the devices mac address.


Since (almost) all devices will broadcast a packet sometime, it's 
inevitable that all the switches will get to see all mac addresses. 
You see, before a device can send a packet to another device it must 
know it's mac address - and typically it gets to find this out by 
broadcasting an ARP packet ("I'm looking for the device with address 
1.2.3.7"). In doing so, it has announced to the whole network that it 
exists ! Even if you can avoid such ARP packets (not easy), there is 
the minor detail of .... the device broadcasting to find a dhcp 
server.


Notice two things I haven't talked about - subnets or management !

Management is irrelevant to the basic mac learning process - it 
provides the administrator with some tools, sometimes rather usefully 
allowing you to query the mac table to find out where a device is ! 
Another facility you typically get with management is Spanning Tree 
Protocol (STP). This allows you to build a network with redundant 
links which would normally result in loops and the network quickly 
dying as packets go round and round and round. STP takes care of 
finding these loops and 'turning off' enough links to remove them - 
but if a link or switch fails, STP can turn links back on to repair 
the network (as far as possible) and minimise the impact.


As for IP addresses, the switches (at this level of sophistication) 
do not look at that - they go no further into the packet than source 
and destination mac.



Now, having told you why you are wasting your time with your multiple 
subnets, let me tell you why you are correct to consider the switch 
topology. Depending on your traffic flows, there can be benefits in 
arranging devices such that high throughput conversations remain 
within one switch.

Lets assume A & B are on one switch, and C & D are on another. A & B 
can exchange their full 100mbps each way with each other. C & D can 
also exchange 100mpbs each way between them.

However, if your conversations were A-C and B-D, then in between the 
conversations would be sharing a single link, so the total would be 
100mbps between them (or 50mbps each if the conversations were 
'equal'). Note that in a fully switched network each link is full 
duplex so these are 'each way' figures - the total throughput is 
double these figures (ie 200mbps intra switch and 
100mbps/conversation inter-switch).
Ie A-->C plus B-->D is limited to 100mbps and C-->A plus D-->B is 
limited to 100mbps.

This happens irrespective of IP address - you only need to consider 
the traffic flows.


But consider if A&B are in the 1.2.3.0/29 subnet, and C&D are in the 
1.2.3.8/29 subnet. Your conversations between A&C and B&D must now go 
via the router - so instead of having a link carrying A-->C plus 
B-->D traffic, you have the router link carrying A-->C, B-->D, plus 
C-->A and D-->B traffic on both it's inbound and outbound wires. Thus 
you have now reduced your throughput potentially to 25mbps each way 
per conversation !


In practical terms, on real networks, there are other factors that 
tend to limit throughput. Certainly for "general office use" you are 
unlikely to have any problems however you arrange the network. It 
only becomes crucial if you have a small number of high bandwidth 
users (such as a CAD or Design dept moving huge files around) - then 
it can be useful to give them their own switch so their traffic is 
kept largely separate from the rest of the network. If you have a 
central file server, give it a gigabit NIC and connect it to a 
gigabit port on the switch at the head of the tree - that way, there 
are several 100mbps paths that can ultimately share that 1G pipe to 
the server.


* It doesn't send the packet out via the port it came in on as that 
would result in loops. If there is a single device connected, it 
knows about the packet (it sent it !). If there is another switch 
connected, it will already have taken care of sending the packet out 
via the rest of it's own ports.

** There are some exceptions, some switches (rarely found these days) 
were limited to one mac per port and were hardcoded that any traffic 
for macs not in the limited list would be sent via a dedicated uplink 
port. These switches can ONLY have an end device connected, you 
cannot connect multiple devices (via another switch or hub) to the 
downstream ports on them.


Simon