Virtualising nx-osv (9k) with KVM - unable to connect it to other vms

casperionx · ‎05-17-2020

So, i've hit a bit of a stumbling point - Im currently doing some testing with the nx-osv 9k switch with KVM/qemu.

This is more so for lab work than anything else, however I've seemed to run into a stumbling block.

When using KVM/qemu as my virtualisation method of choice, I am unable to get any connectivity between guests connected through the nx-osv switch.

When I connect my virtual routers (CSR1000vs) directly via veth or linux bridges they can ping eachother no worries.

However when I fit a nx-osv in the middle and connect them via veth, bridge, tap interfaces, they cant communicate between eachother via the switch.

I've checked the mac tables, and I cant see the other side learning the mac address (or the switch learning any mac addresses either), so something is weird. And Im at a bit of a loss.

I have tried it in gns3 and it works there. So obviously I must be missing something.

Im using libvirt manager to create and run the machines, and manually creating the networks either using virsh xml or ip link statements for veth/tap.

The fact that when I have used the bridge and veth and tap interfaces directly between the two routers they can see eachtoher. But when the switch is in the middle, they cant. (Also they're on the same vlan (vlan 1) and they are on ports 2 and 3 of the nx-osv (the first being management).

Anyone have any suggestions? The config is basically straight out of the box, aside from adding the ip addresses to the routers and making them up.

casperionx · ‎05-18-2020

All right so I've gotten some where with this but still routers wont talk to switches and vice versa.

However, I've trialed 2 9500's and they communicate to each other just fine.

Interestingly:
I can see in both arp tables, when I removed one switch then all the interfaces bar one on the CSR1, mac addresses appear for both CSRs in the switch, and in both CSRs arp tables...

CSR2#show ip arp 
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.255.1 0 5254.0054.276a ARPA GigabitEthernet1
Internet 172.16.255.2 - 5254.00ba.562e ARPA GigabitEthernet1

 CSR1#sho ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.255.1 - 5254.0054.276a ARPA GigabitEthernet2
Internet 172.16.255.2 17 5254.00ba.562e ARPA GigabitEthernet2

9500 mac table:
9500# show mac address-table 
Legend: 
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 1 5254.00ba.562e dynamic 0 F F Eth1/1
* 1 fe54.000d.3043 dynamic 0 F F Eth1/1
* 1 fe54.0029.ac8e dynamic 0 F F Eth1/1
* 1 fe54.00af.4b85 dynamic 0 F F Eth1/2
G - 5211.870d.1b08 static - F F sup-eth1(R)

SO Im at a loss why cant I ping each router when the arp requests come though just fine and populate the arp table in each router??