Re: Delayed loss of connectivity after adding mac access-list

remitprosupport · ‎03-25-2011

Hello all,

I have a catalyst 2960 (12.2(53)SE2) that I've added several mac ACL's to. This was almost 2 days ago, but this morning one host connected to a port where an access-group was assigned lost connectivity with the rest of the network. Connectivity was only re-established by removing the access-group from the interface in question.

I re-added the access list several times, and every time about 10 minutes later my monitoring software starts alerting that it can't connect. On the host in question I'm unable to ping out to anything, even it's default gateway. Connectivity is again only established when I remove the access-group.

This also started happening again an hour ago on a different port assigned to the same access-list/access-group. The same symptoms also persist (connection only re-established when the access-group is removed, loses connectivity after a delayed period once it's re-enabled).

I checked the mac address table and the mac for either host in the table hasn't changed. If anyone can shed some light on this it would be greatly appreciated.

Thanks,

Dan

boss.silva · ‎03-25-2011

Hey Daniel,

Can you attach the switch's config? Taking a look at the config perhaps can lead us to something. Please also tell us the MAC address of the gateway of the subnet, as well as the host's MAC address.

I was not able to find a bug reported for this version of IOS.

Just as a side note: MAC ACLs apply to non-ip traffic only. Perhaps the ARP entries of the host expired, and when it tried to talk to the gateway for instance, it would not be able to if its MAC was denied as a source, or the MAC of the router was denied as a destination.

-Bruno Silva.

remitprosupport · ‎03-25-2011

Thanks Bruno for the reply. Here's the config. The first node in question is on port 19, with a mac address of 0007.e9c8.4517. The second is on port 21, and has a mac adress of f4ce.4637.4337. The default gateway is actually on another switch. Port 25 on this switch is trunked through a GBe link via an SPD port to another switch, and that switch has a connection to the firewall. The trunk port on the firewall has a mac address of d0d0.fd45.64fa.

I should also mention that when I was unable to ping the default gateway from either of these hosts, hosts on the same subnet were also not able to ping them, which tells me the problem is something to do with the switch and not the access at the firewall (which for the record is an ASA5505).

Please let me know if there's anything else you need from me.

Current configuration : 6945 bytes
!
! Last configuration change at 19:50:17 UTC Fri Mar 25 2011
! NVRAM config last updated at 19:27:25 UTC Fri Mar 25 2011
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname officesw2
!
boot-start-marker
boot-end-marker
!
enable secret 5 XXXXXXX
enable password 7 XXXXXXX
!
!
!
no aaa new-model
switch 1 provision ws-c2960s-24ps-l
authentication mac-move permit
ip subnet-zero
!
!
!
!

!

mac access-list extended infra

permit host 0007.e9c8.4517 any

permit host f4ce.4637.4337 any

permit host 0014.38a1.b688 any

permit host 0008.a12a.500d any

mac access-list extended voip

permit host 0010.4917.c93c any

permit host 0010.4917.c91c any

permit host 0010.4917.c918 any

permit host 0010.4917.c93d any

permit host 0010.4917.c915 any

permit host 0010.4917.c919 any

permit host 0010.4917.c935 any

permit host 0010.4917.c93b any

permit host 0010.4917.c931 any

permit host 0010.4917.c917 any

permit host 0010.4917.c91b any

permit host 0010.4917.c920 any

permit host 0010.4917.c939 any

permit host 0010.4917.c91a any

mac access-list extended voip-system

permit host b8ac.6f91.0545 any

permit host b8ac.6f91.0547 any

permit host 0010.4919.2d37 any

spanning-tree mode pvst

spanning-tree etherchannel guard misconfig

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

interface FastEthernet0

description Management

ip address 192.168.250.20 255.255.255.0

!

interface GigabitEthernet1/0/1

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/2

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/3

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/4

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/5

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/6

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/7

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/8

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/9

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/10

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/11

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/12

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/13

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/14

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/15

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/16

description Phone

switchport access vlan 125

mac access-group voip in

spanning-tree portfast

!

interface GigabitEthernet1/0/17

description Phone Switch

switchport access vlan 125

spanning-tree portfast

!

interface GigabitEthernet1/0/18

description Phone Server

switchport access vlan 125

spanning-tree portfast

!

interface GigabitEthernet1/0/19

description Old Jira

switchport access vlan 100

spanning-tree portfast

!

interface GigabitEthernet1/0/20

description New File and JIRA

switchport access vlan 100

mac access-group infra in

spanning-tree portfast

!

interface GigabitEthernet1/0/21

description Recept Printer

switchport access vlan 100

spanning-tree portfast

!

interface GigabitEthernet1/0/22

description Main Printer

switchport access vlan 100

mac access-group infra in

spanning-tree portfast

!

interface GigabitEthernet1/0/23

description WWW2

switchport access vlan 100

mac access-group infra in

!

interface GigabitEthernet1/0/24

description New File and Jira

switchport access vlan 100

mac access-group infra in

!

interface GigabitEthernet1/0/25

description Uplink to SW1

switchport trunk allowed vlan 100,125,150,200

switchport mode trunk

!

interface GigabitEthernet1/0/26

shutdown

!

interface GigabitEthernet1/0/27

shutdown

!

interface GigabitEthernet1/0/28

shutdown

!

interface Vlan1

no ip address

shutdown

!

no ip http server

ip http secure-server

ip sla enable reaction-alerts

!

line con 0

line vty 0 4

password 7 XXXXXX

login

line vty 5 15

password 7 XXXXXXX

login

!

ntp logging

ntp clock-period 22518667

ntp source FastEthernet0

ntp peer 192.168.250.200

end

boss.silva · ‎03-25-2011

Ok. Can you add the following lines as the first in the infra access-list:

permit host 0007.e9c8.4517 any 0x800 0x0

permit host 0007.e9c8.4517 any 0x806 0x0

and assign it again to port gig1/0/19 and test? This will explicitly allow ARP.

Also, if the problem occurs very shortly after the access-list is applied, you can run debug icmp (if not too much traffic on the switch ofc) and see if the switch will send a Host unreachable message to the host on port gig1/0/19, indicating it's not allowed to send out traffic because there is no permit statement for its MAC.

If the problem occurs, take a look and see if the hitcnt increment as well.

Try it out with a vlan access-map, something like this:

access-list 10 permit ip host ip-of-host any

mac access-list extended test

permit host 0007.e9c8.4517 any 0x800 0x0

permit host 0007.e9c8.4517 any 0x806 0x0

vlan access-map ip_mac 10

action forward

match ip address 10

vlan access-map ip_mac 20

action forward

match mac address test

vlan filter ip_mac vlan-list 100

Let me know how the testing goes.

PS: Jira is the best bug tracking system! Did you know the name of it came from the Godzilla (Gojira)?

-Bruno Silva.

remitprosupport · ‎03-28-2011

Thanks for the suggestions. I've implemented your first suggestion (setting the ethertypes). I've also re-instated the access-list on the other node without setting the ether-types. Both are working for now, so I'm going to give them a couple days to fail.

I'll update this thread then with the outcome.

Thanks again!

remitprosupport · ‎03-30-2011

Bruno,

A few days ago when I added your suggestions to my config, I added the ethertypes allowing ARP and IP to the mac for the printer, which was f4ce.4637.4337. This worked fine for the past couple days. This morning though, our JIRA server lost connectvity (with a mac of 0007.e9c8.4517). I added the ethertype rules and it began working again.

About 15 minutes ago though, one of our users brought to my attention that the printer I had applied your suggested rules to wasn't working. Thankfully I had a continuous ping to that printer running to it, and I was able to figure out by the sequence numbers that it hadn't been accessible for almost 2 hours. I left the access-list statements in place and removed the access-group from the port and it started working again. I re-added the access-group, and it's still working.

Whatever's causing this doesn't have an immediate effect, but removing the access group or making any change in the access-list seems to temporarily allow access.

Do you have any other suggestions?

Thanks,

Dan

boss.silva · ‎03-31-2011

If you are sure that the access-lists that are blocking the traffic suddenly, i suggest you open a TAC case as this is just weird. This gotta be a bug.