Mgmt0 for VDCs on same chassis can't communicate

Sean Lair · ‎05-15-2015

We have a Nexus 7700 running two different contexts. One is the Admin context and then another standard context. They are configured with IP addresses within the same subnet (10.101.0.0/24), but they can't communicate with each - other ping doesn't even work between them.

However, they can both talk to all other devices on that subnet just not each other... I've included the output from some commands below along with the NX-OS version. Both mgmt0 interfaces in each VDC have different MAC addresses as they should.

Is this a bug or am I missing something here? The upstream switchport is just an access port.

Thanks for any help/insight anyone can provide!

Sean

version 6.2(8a)

Admin Context

show run int mgmt0

interface mgmt0
vrf member management
ip address 10.101.0.10/24

show int mgmt0
mgmt0 is up
admin state is up
Hardware: GigabitEthernet, address: 8480.2da5.f3c4 (bia 8480.2da5.f3c4)
Internet Address is 10.101.0.10/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is routed
full-duplex, 1000 Mb/s
Auto-Negotiation is turned on
Auto-mdix is turned off
EtherType is 0x0000
1 minute input rate 32 bits/sec, 0 packets/sec
1 minute output rate 24 bits/sec, 0 packets/sec
Rx
188445 input packets 6622 unicast packets 181375 multicast packets
448 broadcast packets 48204530 bytes
Tx
90779 output packets 20 unicast packets 90737 multicast packets
22 broadcast packets 21779795 bytes

Other Context

show run int mgmt0

interface mgmt0
vrf member management
ip address 10.101.0.11/24

show int mgmt0

mgmt0 is up
admin state is up
Hardware: GigabitEthernet, address: 8480.2da5.f3c5 (bia 8480.2da5.f3c5)
Internet Address is 10.101.0.11/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is routed
full-duplex, 1000 Mb/s
Auto-Negotiation is turned on
Auto-mdix is turned off
EtherType is 0x0000
1 minute input rate 696 bits/sec, 0 packets/sec
1 minute output rate 696 bits/sec, 0 packets/sec
Rx
5647969 input packets 5466145 unicast packets 181376 multicast packets
448 broadcast packets 495885390 bytes
Tx
5569760 output packets 5466115 unicast packets 92181 multicast packets
11464 broadcast packets 474893154 bytes

Reza Sharifi · ‎05-16-2015

Since each interface is in a different VDC, they will not talk to each other. Think of VDCs as 2 different switches. Unless you connect the VDCs together the interfaces will not talk to each other.

HTH

Sean Lair · ‎05-18-2015

Hi Reza, thanks for the reply. I understand they are two different switches, but they both have IPs in the same subnet and thus should be able to talk to each other, through the external switch that mgmt0 is connected. Here are the IPs assigned:

10.101.0.10/24

10.101.0.11/24

As seen above, they are in the same subnet and should be able to talk to each other across the network.

Thanks

Sean

Reza Sharifi · ‎05-18-2015

Hi Sean,

Are both ports in the same vlan on the external switch?

Sean Lair · ‎05-18-2015

Hi Reza,

Since these are VDC, those IPs are all on the physical mgmt0 on the switch. mgmt0 is physically shared across all VDC, so yes they are technically on the same VLAN.

Thanks

Sean

Reza Sharifi · ‎05-18-2015

Hi Sean,

Even though there is only one mgmt interface, each VDC gets its own management IP and can't talk to each other unless you connect the mgmt interface physically to an external switch.

from 7k documentation:

The out-of-band management interface (mgmt0) can be used to manage all VDCs. Each VDC has its own representation for mgmt0 with a unique IP address that can be used to send syslog, SNMP and other management information. The mgmt interfaces are not shared across multiple VDCs.

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/white_paper_c11-701112.html

HTH

Sean Lair · ‎05-18-2015

Hi Reza, the mgmt0 interface is physically connected to an external switch. I can talk to all the mgmt IPs from other hosts on the network. But from within a VDC I cannot even ping other mgmt0 IPs (of other VDCs)