<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 15px; font-family: Calibri, sans-serif;">
<div>Hi Tomek, Francis,</div>
<div><br>
</div>
<div>Thanks a lot for your replies!</div>
<div><br>
</div>
<div>Yes, each namespace in Linux creates an isolated set of networking resources </div>
<div>like interfaces, IPv6/IPv6 protocol stack, routing tables, port numbers (sockets)</div>
<div>etc.</div>
<div><br>
</div>
<div><a href="http://man7.org/linux/man-pages/man7/namespaces.7.html">http://man7.org/linux/man-pages/man7/namespaces.7.html</a></div>
<div><br>
</div>
<div>"sudo ip l” in the default/global namespace does not show interfaces in other </div>
<div>namespaces. Below is the output in the default/global namespace.</div>
<div><br>
</div>
<div>
<div># sudo ip l</div>
<div>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</div>
<div>2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000</div>
<div> link/ether c0:67:af:af:72:8f brd ff:ff:ff:ff:ff:ff</div>
<div>3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000</div>
<div> link/ether c0:67:af:af:72:90 brd ff:ff:ff:ff:ff:ff</div>
<div>4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000</div>
<div> link/ether c0:67:af:70:d1:f4 brd ff:ff:ff:ff:ff:ff</div>
<div>5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000</div>
<div> link/ether c0:67:af:70:d1:f5 brd ff:ff:ff:ff:ff:ff</div>
<div>6: ovs-system: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/ether a2:c0:30:03:f9:78 brd ff:ff:ff:ff:ff:ff</div>
<div>7: br-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/ether c0:67:af:af:72:8f brd ff:ff:ff:ff:ff:ff</div>
<div>8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/ether c0:67:af:70:d1:f4 brd ff:ff:ff:ff:ff:ff</div>
<div>9: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/ether 82:9f:84:d9:af:46 brd ff:ff:ff:ff:ff:ff</div>
<div>127930: phy-br-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP mode DEFAULT qlen 1000</div>
<div> link/ether ba:92:1d:02:a6:d4 brd ff:ff:ff:ff:ff:ff</div>
<div>127931: int-br-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP mode DEFAULT qlen 1000</div>
<div> link/ether 1e:10:3b:da:a6:86 brd ff:ff:ff:ff:ff:ff</div>
<div># </div>
<div><br>
</div>
</div>
<div>
<div>"sudo ip l” in a namespace also does not show interfaces in other namespaces.</div>
</div>
<div>It shows just the interfaces in a particular namespace. Below is the output from</div>
<div>the first four namespaces (testns-DHCP-network-0….testns-DHCP-network-3)</div>
<div>and the last (2000th) namespace testns-DHCP-network-1999.</div>
<div><br>
</div>
<div>
<div># sudo ip netns exec testns-DHCP-network-0 ip l</div>
<div>1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT </div>
<div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</div>
<div>126573: tapa748acb8-a1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff</div>
<div><br>
</div>
<div># sudo ip netns exec testns-DHCP-network-1 ip l</div>
<div>1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT </div>
<div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</div>
<div>127893: tapfa1a4b40-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff</div>
<div><br>
</div>
<div># sudo ip netns exec testns-DHCP-network-2 ip l</div>
<div>1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT </div>
<div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</div>
<div>127274: tap109d0256-f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff</div>
<div><br>
</div>
<div># sudo ip netns exec testns-DHCP-network-3 ip l</div>
<div>1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT </div>
<div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</div>
<div>127362: tap4ddbb28d-02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff</div>
<div><br>
</div>
<div># sudo ip netns exec testns-DHCP-network-1999 ip l</div>
<div>1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT </div>
<div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</div>
<div>126459: tap9283904c-19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT </div>
<div> link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff</div>
<div><br>
</div>
</div>
<div>An interface in one namespace can ping/reach an interface in another </div>
<div>namespace only if there is a layer-2 bridge (like OpenSwitch I’m using) that </div>
<div>bridges both the interfaces so that inter-namespace communication is </div>
<div>possible.</div>
<div><br>
</div>
<div>I increased SNDBUF_SIZE and RCVBUF_SIZE in src/lib/dhcp/iface_mgr_linux.cc</div>
<div>to a large value (256K = 256 * 1024 = 262144), rebuilt perfdhcp, and still see the</div>
<div>same error. SNDBUF_SIZE and RCVBUF_SIZE were 32768 before.</div>
<div><br>
</div>
<div>
<div>const static size_t SNDBUF_SIZE = 262144;</div>
</div>
<div><br>
</div>
<div>
<div>const static size_t RCVBUF_SIZE = 262144;</div>
</div>
<div><br>
</div>
<div>Also, I see the same error when I run perfdhcp in the first few namespaces (</div>
<div>testns-DHCP-network-0….testns-DHCP-network-3) and the last few namespaces (</div>
<div>testns-DHCP-network-1990….testns-DHCP-network-1999).</div>
<div><br>
</div>
<div>Below is the same error when I run perfdhcp in the first namespace </div>
<div>testns-DHCP-network-0 after increasing SNDBUF_SIZE and RCVBUF_SIZE to</div>
<div>262144 and rebuilding perfdhcp.</div>
<div><br>
</div>
<div>
<div># sudo ip netns exec testns-DHCP-network-0 /usr/local/sbin/perfdhcp -x aiestT -4 -b mac=fa:16:00:00:00:00 -r 5 -R 5 -p 10 -l tapa748acb8-a1</div>
<div>Running: perfdhcp -x aiestT -4 -b mac=fa:16:00:00:00:00 -r 5 -R 5 -p 10 -l tapa748acb8-a1</div>
<div>IPv4</div>
<div>lease-type=address-only (IA_NA option added to the client's request)</div>
<div>rate[1/s]=5</div>
<div>clients=5</div>
<div>base[0]=mac=fa:16:00:00:00:00</div>
<div>test-period=10</div>
<div>drop-time[0]=1</div>
<div>drop-time[1]=1</div>
<div>aggressivity=1</div>
<div>broadcast</div>
<div>elp-offset=-1</div>
<div>sid-offset=-1</div>
<div>rip-offset=-1</div>
<div>diagnostic-selectors=aiestT</div>
<div>interface=tapa748acb8-a1</div>
<div>server=255.255.255.255</div>
<div>Set MAC to fa::16::00::00::00::00</div>
<div>Set DUID to 000100011d7a145a000c01020304</div>
<div>Error running perfdhcp: There is no address for interface: tapa748acb8-a1, port: 67, address family: AF_INET</div>
<div># </div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div>
<div>Vikram Hosakote</div>
<div>OpenStack Software Engineer</div>
<div><a href="mailto:vhosakot@cisco.com">vhosakot@cisco.com</a></div>
<div>Cloud and Virtualization Group</div>
<div>Cisco Systems</div>
<div>Boxborough, MA, USA</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Tomek Mrugalski <<a href="mailto:tomasz@isc.org">tomasz@isc.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, September 2, 2015 at 6:33 AM<br>
<span style="font-weight:bold">To: </span>Vikram Hosakote <<a href="mailto:vhosakot@cisco.com">vhosakot@cisco.com</a>>, Marcin Siodelski <<a href="mailto:marcin@isc.org">marcin@isc.org</a>>, "<a href="mailto:kea-users@lists.isc.org">kea-users@lists.isc.org</a>"
<<a href="mailto:kea-users@lists.isc.org">kea-users@lists.isc.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Kea-users] perfdhcp error<br>
</div>
<div><br>
</div>
<div>
<div>
<div>On 02/09/15 07:32, Vikram Hosakote (vhosakot) wrote:</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>I have been running perfdhcp using an interface inside a Linux namespace </div>
<div>successfully. I have also been able to successfully run multiple</div>
<div>instances of perfdhcp using interfaces in multiple namespaces simultaneously.</div>
</blockquote>
<div>Cool. Thanks for sharing this info.</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>But, when I have 2,000 namespaces and run perfdhcp inside each of these</div>
<div>namespaces, I see that perfdhcp throws the following error even though the </div>
<div>interface used by perfdhcp has an IP address. This error is NOT seen when I</div>
<div>run perfdhcp instances in a few (100-200) namespaces but is seen when I</div>
<div>run them in 2,000 namespaces.</div>
</blockquote>
<div>I'm not familiar with the namespaces you mention, but I presume each</div>
<div>namespace creates its own network interface that is visible through ip,</div>
<div>right?</div>
<div><br>
</div>
<div>So when creating 2000 namespaces, how many network interfaces are</div>
<div>reported by sudo ip l? I think you may have hit the boundary of our</div>
<div>netlink interface that assumes that the information about interfaces</div>
<div>will not be larger than 32KB. If my theory is right, the info provided</div>
<div>by the kernel is larger than the buffer we provide (32K), so it's</div>
<div>getting truncated and we parse only the first 32KB of data. That means</div>
<div>that if you have a lot (2000 you mentioned definitely falls into that</div>
<div>category)</div>
<div>of interfaces, only the initial couple hundred would be visible by</div>
<div>perfdhcp (or kea for that matter, as they share libdhcp++).</div>
<div><br>
</div>
<div>Can you do an experiment? Please configure 2000 namespaces and get a</div>
<div>list interfaces: ip l. Try to run perfdhcp on one of the first 50 or so</div>
<div>interfaces. If my theory is right, it should work. Now try to run it on</div>
<div>one of the last interfaces returned by ip l. It should fail.</div>
<div><br>
</div>
<div>If that is so, I have an easy fix for you. Open</div>
<div>src/lib/dhcp/iface_mgr_linux.cc and see lines 123, 125. There are</div>
<div>SNDBUF_SIZE and RCVBUF_SIZE. Please increase RCVBUF_SIZE and see if it</div>
<div>helps.</div>
<div><br>
</div>
<div>Tomek</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div></div>
<div>Error running perfdhcp: There is no address for interface:</div>
<div>tapa748acb8-a1, port: 67, address family: AF_INET</div>
<div></div>
<div>Below is the interface tapa748acb8-a1 (inside the namespace </div>
<div>testns-DHCP-network-0) that has the IP address 1.0.0.1. The interface is up.</div>
<div></div>
<div># sudo ip netns exec testns-DHCP-network-0 ifconfig</div>
<div>tapa748acb8-a1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500</div>
<div> inet 1.0.0.1 netmask 255.255.255.0 broadcast 1.0.0.255</div>
<div> ether ee:ff:ff:ff:ff:ef txqueuelen 0 (Ethernet)</div>
<div> RX packets 138 bytes 9280 (9.0 KiB)</div>
<div> RX errors 0 dropped 0 overruns 0 frame 0</div>
<div> TX packets 45 bytes 3590 (3.5 KiB)</div>
<div> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</div>
<div></div>
<div>Below is the perfdhcp output (with verbose diagnostics enabled). Could you </div>
<div>please let me know what the issue is. What am I missing ? Is this a </div>
<div>scale/concurrency issue ? The same exact perfdhcp command below works </div>
<div>perfectly fine when I have a few (100-200) namespaces and run perfdhcp</div>
<div>instances using interfaces in them.</div>
<div></div>
<div># sudo ip netns exec testns-DHCP-network-0 /usr/local/sbin/perfdhcp -x</div>
<div>aeistT -4 -b mac=fa:16:00:00:00:00 -r 5 -R 5 -p 10 -l tapa748acb8-a1</div>
<div>Running: perfdhcp -x aeistT -4 -b mac=fa:16:00:00:00:00 -r 5 -R 5 -p 10</div>
<div>-l tapa748acb8-a1</div>
<div>IPv4</div>
<div>lease-type=address-only (IA_NA option added to the client's request)</div>
<div>rate[1/s]=5</div>
<div>clients=5</div>
<div>base[0]=mac=fa:16:00:00:00:00</div>
<div>test-period=10</div>
<div>drop-time[0]=1</div>
<div>drop-time[1]=1</div>
<div>aggressivity=1</div>
<div>broadcast</div>
<div>elp-offset=-1</div>
<div>sid-offset=-1</div>
<div>rip-offset=-1</div>
<div>diagnostic-selectors=aeistT</div>
<div>interface=tapa748acb8-a1</div>
<div>server=255.255.255.255</div>
<div>Set MAC to fa::16::00::00::00::00</div>
<div>Set DUID to 000100011d793ce4000c01020304</div>
<div>Error running perfdhcp: There is no address for interface:</div>
<div>tapa748acb8-a1, port: 67, address family: AF_INET</div>
<div># </div>
<div></div>
<div></div>
<div>Regards,</div>
<div>Vikram Hosakote</div>
<div>OpenStack Software Engineer</div>
<div><a href="mailto:vhosakot@cisco.com">vhosakot@cisco.com</a> <<a href="mailto:vhosakot@cisco.com">mailto:vhosakot@cisco.com</a>></div>
<div>Cloud and Virtualization Group</div>
<div>Cisco Systems</div>
<div>Boxborough, MA, USA</div>
<div></div>
<div></div>
<div>_______________________________________________</div>
<div>Kea-users mailing list</div>
<div><a href="mailto:Kea-users@lists.isc.org">Kea-users@lists.isc.org</a></div>
<div><a href="https://lists.isc.org/mailman/listinfo/kea-users">https://lists.isc.org/mailman/listinfo/kea-users</a></div>
<div></div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</span>
</body>
</html>