N3K-1:inter vlan 11ip add 10

devinliu1991 · ‎10-31-2014

Hi, all

We had 3 N3K-3048 arrived recently with NX-OS version uk9.6.0.2.U1.2, and we built a simple network with topology below:

3 N3K join the same HSRP group with N3K-1 in active state, N3K-2 in standby state and N3K-3 in listen.

N3K-1 and N3K-2 have a default route to outside network above, but N3K-3 doesn't. So if N3K-3 deals the IP packets from PCs in layer 3, the network would be blocked.

Now our PC could not visit the outside network, and the HSRP states of the 3 deives have been checked and are OK(N3K-1 active,N3K-2 standby,N3K-3 listen). And also the MAC address PC uses to visit outside network is the HSRP virtual MAC.

When I designate the real VLAN 11 IP address of N3K-1 to be the default gateway instead of the virtual IP address in HSRP group, the outside network becomes available for our PC.

Is this a bug of the NX-OS version, or a feature, or limitation?

devinliu1991 · ‎10-31-2014

Added,

We had also built a similar network with 3 3560X, and it just worked properly. So what makes the listen state N3K-3 dealing with the virtual MAC address?

Reza Sharifi · ‎10-31-2014

Hi,

A couple of questions:

What happens if you make N3K-3 the active router?

What happens if you use N3K-3 as a layer-2 device (no HSRP)?

devinliu1991 · ‎10-31-2014

Since N3K-3 doesn't have a default route to the outside network, the packet that forwarded to N3K-3 would be dropped.

If N3K-3 becomes a layer-2 device, packet could travel through it to N3K-1 or N3K-2 and the network will work properly.

But in our network, N3K-3 should be the default gateway in some cases to reach the internal network.

What I really can not understand is that, since the N3K-1 is the active router owned the VMAC, why N3K-3 deals with the the frame with the VMAC as the destination MAC address.

THX

Reza Sharifi · ‎10-31-2014

Can you post relevant configs from all 3 switches?

devinliu1991 · ‎10-31-2014

N3K-1:

inter vlan 11

ip add 10.10.11.252 255.255.255.0

hsrp 11

ip 10.10.11.254

priority 150

preempt

N3K-2:

inter vlan 11

ip add 10.10.11.253 255.255.255.0

hsrp 11

ip 10.10.11.254

priority 120

preempt

N3K-3:

inter vlan 11

ip add 10.10.11.251 255.255.255.0

hsrp 11

ip 10.10.11.254

devinliu1991 · ‎10-31-2014

Also the switchport mode and trunk etherchannel state are checked, one uplink on N3K-3 is blocked because of the STP, all normal.

pille1234 · ‎11-01-2014

Hallo,

I suppose this behavior is by design.

In a vPC-setup the two nexus are running in an active-standby setup from a control plane perspective only. In fact even the HSRP standby router is aware of the VMAC and whenever it receives a frame addressed for this VMAC it routes the packet to the destination no matter the HSRP state. Thus from a dataplane perspective Nexus switches always run active-active.

While there is no vPC config present here, I assume the same rules apply in your case.

Regards

devinliu1991 · ‎11-01-2014

Thanks for your reply,

I had assumed the same before I had another scenario yesterday.

In this case, N3K-1 keeps on to be the active device of VLAN 11 and I have also created a new VLAN 101 on N3K-1 with IP address 10.10.101.1/24.

Since N3K-2 which is in standby state doesn't has the route to 10.10.101.0/24, the packet with VLAN 11 VMAC to be the destination would be dropped if N3K-2 handled it.

But in my test, PC had got ICMP replies from N3K-1 which means that the packet had got through the HSRP standby device in layer-2.

pille1234 · ‎11-02-2014

Hm that's queer. Would you mind showing us the output from N3k-2 for

show ip route 10.10.101.1

show ip route 0.0.0.0

show hsrp interface vlan 11

show mac address-table vlan 11 | inc VMAC

show mac address-table vlan 101

and on N3k-1

show hsrp interface vlan 11

show mac address-table vlan 11 | inc VMAC

show hsrp interface vlan 101

show mac address-table vlan 101

devinliu1991 · ‎11-04-2014

on N3K-2

N3K-2# show ip route 10.10.101.1
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

Route not found

N3K-2# show ip route 0.0.0.0
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

0.0.0.0/0, ubest/mbest: 1/0
*via 10.129.4.1, [1/0], 3d07h, static

N3K-2#show hsrp interface vlan 11
Vlan11 - Group 11 (HSRP-V1) (IPv4)
Local state is Standby, priority 120 (Cfged 120), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 120
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 2.123000 sec(s)
Virtual IP address is 10.10.11.254 (Cfged)
Active router is 10.10.11.252, priority 150 expires in 0.173000 sec(s)
Standby router is local
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac0b (Default MAC)
3 state changes, last state change 3d07h
IP redundancy name is hsrp-Vlan11-11 (default)

N3K-2# show mac address-table vlan 11 | include 0000.0c07.ac0b
* 11 0000.0c07.ac0b dynamic 610 F F Po1

no mac information for VLAN 101 on N3K-2

on N3K-1

N3K-1# show hsrp interface vlan 11
Vlan11 - Group 11(HSRP-V1) (IPv4)
Local state is Active, priority 150 (Cfged 150), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 150
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 1.705000 sec(s)
Virtual IP address is 10.10.11.254 (Cfged)
Active router is local
Standby router is 10.10.11.253 , priority 120 expires in 2.865000 sec(s)
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac0b (Default MAC)
1 state changes, last state change 3d07h
IP redundancy name is hsrp-Vlan11-11 (default)

no mac information for vlan 11 about VMAC

N3K-1# show hsrp brief
P indicates configured to preempt.
|
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan11 11 150 P Active local 10.10.11.253 10.10.11.254
(conf)

pille1234 · ‎11-04-2014

Just to be clear we are talking here about your second scenario, right?

From the output we see N3K-2 has a default route. Would it be possible that your ping from pc to VLAN 101 reached its destination via default route 10.129.4.1?

N3K-2# show mac address-table vlan 11 | include 0000.0c07.ac0b
* 11 0000.0c07.ac0b dynamic 610 F F Po1

What a pitty, i was hoping to see a "G" in front of the *.

on N3K-1

no mac information for vlan 11 about VMAC

Er no, I don't think this is possible. There needs to be a static entry belonging to SUP or something like that. Could you confirm this output?

Regards

devinliu1991 · ‎11-04-2014

Hi,

Yes we are talking about the second scenario, for security concern I had changed the IP address, hope you could understand.

VLAN 101 was created on N3K-1 for testing, and the device 10.129.4.1 doesn't have the route to reach 10.10.101.0/24. Only N3K-1 knows about it. So I think in the second scenario, when two N3Ks forms a group, the standby device doesn't handle the VMAC.

N3K-2# show mac address-table vlan 11 | include 0000.0c07.ac0b
* 11 0000.0c07.ac0b dynamic 610 F F Po1

I have checked again, and the flag is *.

on N3K-1

no mac information for vlan 11 about VMAC

I used the "show mac address-table" command to check out again, and I could only see dynamic entries.

devinliu1991 · ‎11-04-2014

And I have also found a relevant bug, but is for N5K, and the condition is hsrp not working.

more details https://tools.cisco.com/bugsearch/bug/CSCts46891

I just think that the condition I had met is also a bug, because when I have the same topology, similar ip addresses, but change the devices to 3560-X, everything works well.

Question about NX-OS HSRP