Why does an ACL applied in an outbound direction on a VLAN effect traffic that goes across the Nexus...

Dennis McLaughlin · ‎07-13-2012

Lets consider a situation where you would need an ACL on a VLAN in an outbound direction. This may be needed for security reasons or you want to log a certain stream of traffic going out of the VLAN on the Nexus 7000 platform. The reason is beyond the scope of this test.

Now lets look at the topology diagram provided. We have a host that is trying to get to the HSRP gateway 192.168.50.1. N7k1 is the HSRP active so N7k1 should get the traffic destined for the HSRP gateway mac address.

Now traffic could go directly from the access switch to N7k1 and the ACL would not be applied as this is strictly a layer 2 path from the host, through the access switch, to N7k1. In our instance traffic is hashed from our access switch to N7k2 the standby HSRP gateway. This will then require the traffic to traverse the peer-link to get to N7k1.

The normal expectancy for traffic destined to the gateway mac address is to route out of the VLAN. This is the most common reason we would have the destination mac in the packet to be that of the HRSP or SVI mac address.

When putting the packet on the peer-link this will cause the packet to be essentially routed from VLAN 50 to VLAN 50. You can see this behavior not only causes the TTL field to decrement it will also cause the outbound ACL to be applied as it leaves the VLAN and is routed back into the VLAN. The attached sniffer output will show we decrement the TTL to 254 as it comes out of the N7k2. If this were strictly a layer 2 topology we would not decrement the TTL value and it would egress N7k2 as TTL 255.

Ethanalyzer capture on N7k1 of ping packet with decremented TTL to VLAN 50 SVI on N7k1

7K1# ethanalyzer local interface inband capture-filter "dst host 192.168.50.2" limit-captured-frames 1 detail

Capturing on inband

Frame 1 (146 bytes on wire, 114 bytes captured)

Arrival Time: Jul 10, 2012 18:47:01.841932000

Internet Protocol, Src: 192.168.50.50 (192.168.50.50), Dst: 192.168.50.2 (192.168.50.2)

Version: 4

Header length: 20 bytes

Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)

0000 00.. = Differentiated Services Codepoint: Default (0x00)

.... ..0. = ECN-Capable Transport (ECT): 0

.... ...0 = ECN-CE: 0

Total Length: 100

Identification: 0x4aa5 (19109)

Flags: 0x00

0.. = Reserved bit: Not Set

.0. = Don't fragment: Not Set

..0 = More fragments: Not Set

Fragment offset: 0

Time to live: 254

Protocol: ICMP (0x01)

Header checksum: 0x8c6e [correct]

[Good: True]

[Bad : False]

Source: 192.168.50.50 (192.168.50.50)

Destination: 192.168.50.2 (192.168.50.2)

Internet Control Message Protocol

Type: 8 (Echo (ping) request)

Lets take a step back and understand why we are routing this packet. When the traffic with a destination mac of the gateway comes into N7k2. The N7k2 will see the mac address in its local mac table. M1 and F1 modules will have different output as the F1 doesn’t do its own Layer 3 lookups. For an M1 linecard this local programing will say the following.

7K2# show mac address-table vlan 50

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

G 50 0000.0c07.ac01 static - F F vPC Peer-Link(R)

1) Mac address “0000.0c07.ac01”

2) Route this mac address locally.

3) My vPC peer owns the address

4) Send it on the peer-link

Because I have the mac address in my local mac table with the “G” and “R” flags set N7k2 will route the packet. The routing will not re-write anything but will decrement the TTL because it is going through the routing function. This is the point where the packet will leave the VLAN and is subjected to the outbound ACL on the VLAN. If it is written to deny, the traffic will drop. If it is permitted then the traffic is sent on the peer-link for the N7k1 to process the packet.

N7k2 will not respond to the packet because he doesn’t own the IP. N7k2 is the HSRP standby. Gateway Mac and Routed Mac allow the N7k2 to route for its peer N7k1, but what it doesn’t do is allow it to answer for IP addresses that it doesn’t own. So if N7k2 is the HSRP standby it must send IP HSRP destined traffic to the active for processing. In this case N7k1 is HSRP active so it must receive the packet to respond to the ICMP request. If it were just a packet with the HSRP destination MAC and an IP address out of the subnet then N7k2 would route the packet without N7k1 intervention.

But you say this behavior doesn’t make sense. Why would we want to route a packet into the same VLAN it came out of? It seems to be very inefficient. Why don’t we just switch the packet and send it over the peer-link?

The reason we route the packet is due to the internal mechanisms on how vPC works. It is part of how traffic destined to gateway mac of the peer is handled by vPC. This behavior is only seen when the destination mac address has the gateway flag set in the mac table. And the destination IP address is either the SVI of N7k1 or the HSRP VIP. Lets take a look at what that will look like on an M1 linecard.

7K2# show mac address-table vlan 50

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

G 50 0000.0c07.ac01 static - F F vPC Peer-Link(R)

G 50 001b.54c2.77c1 static - F F vPC Peer-Link(R)

G 50 0024.986f.d5c1 static - F F sup-eth1(R)

* 50 0025.84e6.8dc7 dynamic 570 F F Po50

As you can see we have the gateway MAC flag set on the HSRP MAC, N7k1 SVI MAC, and the local N7k2 SVI MAC. The host doesn’t have a gateway MAC flag set so this will be layer 2 switched to Po50.

So lets now look at what N7k1 shows in its MAC table to see what it has programed on an M1 linecard.

7K# show mac address-table vlan 50

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

G 50 0000.0c07.ac01 static - F F sup-eth1(R)

G 50 001b.54c2.77c1 static - F F sup-eth1(R)

G 50 0024.986f.d5c1 static - F F vPC Peer-Link(R)

50 0025.84e6.8dc7 dynamic 330 F F vPC Peer-Link

As you can see we have the same output only on N7k1, we forgo the Gateway MAC flag for the host MAC address even though we send it on the vPC peer-link.

We have established that when a packet is received on a vPC peer and the destination mac address has the gateway mac flag set. The Nexus will in fact route the packet back into the same VLAN to send it on the peer-link to the neighbor who owns the mac address. Under normal circumstances we will only use the HSRP mac or the SVI mac to get out of the VLAN anyway. So this is only a factor when your destination IP address is that of the N7k1 SVI or the HSRP VIP when N7k1 is active and the traffic ingresses the N7k2. In our case when you ping the gateway you will have the destination mac of the HSRP or SVI. When the packet is received on N7k2 it will route the packet for N7k1 only to be found that the L3 IP address is destined across the peer-link to N7k1.

Finally lets consider an ACL that is applied to an SVI and the behavior we will see with certain traffic. Since IP ACLs applied to VLANs only apply when traffic is ingressing or egressing the VLAN, regular layer 2 traffic is unaffected by this ACL. It is when traffic leaves or enters the VLAN that we permit or deny based on the ACL configuration.

ICMP ping from host attached to the access switch to the N7k1 SVI lab testing

(Link from access switch link to N7k1 in vPC is shut down for testing)

Configuration

N7k2

ip access-list 1

10 permit ip host 192.168.50.50 host 192.168.50.2

interface vlan 50

ip access-group 1 out

Host#

Ping 192.168.50.2

!!!!!!!!!!!!!!!!!!!!

This ACL will allow only host 192.168.50.50 to communicate with host 192.168.50.2. In our test this is all that is needed to establish a working ping between the host and N7k1 SVI VLAN 50 IP. This will validate the test as a working baseline.

Next we will make the permit statement into a deny statement. This will now stop our ICMP request from getting past N7k2.

N7k2

ip access-list 1

10 deny ip host 192.168.50.50 host 192.168.50.2

20 permit ip any any

interface vlan 50

ip access-group 1 out

Host#

Ping 192.168.50.2

.............

Why does it drop the ICMP request?

It drops the packet due to the destination N7k1 SVI mac for the request is set with the gateway mac flag in the mac table on N7k2. The ACL now is in play for any mac address with the gateway MAC flag set, when the packet needs to traverse the peer-link. This packet must go “out” the VLAN to be routed then back “in” the VLAN before it goes across the peer-link.

Summary

The gateway mac and routed mac fields in the mac table allow the Nexus 7000 to locally route traffic without intervention by the vPC peer device. When a packet is received on the N7k and the destination mac has the “G and R” flags set it will locally route the packet. If the destination IP happens to be in the same VLAN then it will send it across the peer-link where the SVI or HSRP VIP resides to be processed.

Why does an ACL applied in an outbound direction on a VLAN effect traffic that goes across the Nexus 7000 vPC peer-link in the same VLAN?