ARP Broadcast not visible in VLAN until port-security is disabled

[Martin Insel]

Hello Community,

we are testing VoIP in our institute and investigating an issue with our VoIP phones.

A couple of phones spradic lose there connection to the SIP server of our provider.

We figured out that the problem probably is caused by the port-security of the access port where the VoiP phone is connected to.

We captured the packets of the port an saw that the phone periodically sends unicast arp requests for it's gateway as long as it has the information about the destination mac of the gateway in its arp cache.

When the information is removed from the arp cache of the phone, it starts sending arp broadcast every few seconds. The problem is that we can't see the arp broadcast on other ports with this vlan. 

The port-security adds the mac of the phone to the port with every arp request. 

The behaviour immediately ends when we disable the port-security on the port. The arp request arrives on all ports in the vlan and in this case the gateway (our firewall) replys to it and the phone starts a new session to the sip server.

We can't see a violation in port-security debug log and port-securtiy statistic

Dec 6 18:28:25 c3650- 372809: Dec 6 17:28:25: PSECURE: psecure_packet_enqueue: psecure receives a packet: addr = 805e.0c01.0af9, swidb = Gi3/0/2, vlan = 632, linktype = NullPak
Dec 6 18:28:25 c3650- 372811: Dec 6 17:28:25: PSECURE: swidb = GigabitEthernet3/0/2 mac_addr = 805e.0c01.0af9 vlanid = 632
Dec 6 18:28:25 c3650- 372812: Dec 6 17:28:25: PSECURE: Adding address vlan 632 805e.0c01.0af9 to port-security
Dec 6 18:28:25 c3650- 372814: Dec 6 17:28:25: PSECURE: psecure_update_address_counts: Incrementing the count for dynamic addresses on vlan 632 and also updating totals
Dec 6 18:28:25 c3650- 372815: Dec 6 17:28:25: PSECURE: Adding 805e.0c01.0af9 as dynamic on port Gi3/0/2 for vlan 632

Dec 6 18:28:25 c3650- 372816: Dec 6 17:28:25: PSECURE: psecure_packet_enqueue: psecure receives a packet: addr = 805e.0c01.0af9, swidb = Gi3/0/2, vlan = 632, linktype = NullPak
Dec 6 18:28:25 c3650- 372818: Dec 6 17:28:25: PSECURE: swidb = GigabitEthernet3/0/2 mac_addr = 805e.0c01.0af9 vlanid = 632
Dec 6 18:28:25 c3650- 372819: Dec 6 17:28:25: PSECURE: Address vlan 632 805e.0c01.0af9 exists in HA table with feature psec and type dynamic
Dec 6 18:28:25 c3650- 372820: Dec 6 17:28:25: PSECURE: Adding 805e.0c01.0af9 as dynamic on port Gi3/0/2 for vlan 632

Dec 6 18:28:26 c3650- 372826: Dec 6 17:28:25: PSECURE: psecure_packet_enqueue: psecure receives a packet: addr = 805e.0c01.0af9, swidb = Gi3/0/2, vlan = 632, linktype = NullPak
Dec 6 18:28:26 c3650- 372828: Dec 6 17:28:25: PSECURE: swidb = GigabitEthernet3/0/2 mac_addr = 805e.0c01.0af9 vlanid = 632
Dec 6 18:28:26 c3650- 372829: Dec 6 17:28:25: PSECURE: Address vlan 632 805e.0c01.0af9 exists in HA table with feature psec and type dynamic
Dec 6 18:28:26 c3650- 372830: Dec 6 17:28:25: PSECURE: Adding 805e.0c01.0af9 as dynamic on port Gi3/0/2 for vlan 632

Dec 6 18:28:26 c3650- 372831: Dec 6 17:28:26: PSECURE: psecure_packet_enqueue: psecure receives a packet: addr = 805e.0c01.0af9, swidb = Gi3/0/2, vlan = 632, linktype = NullPak
Dec 6 18:28:26 c3650- 372833: Dec 6 17:28:26: PSECURE: swidb = GigabitEthernet3/0/2 mac_addr = 805e.0c01.0af9 vlanid = 632
Dec 6 18:28:26 c3650- 372834: Dec 6 17:28:26: PSECURE: Address vlan 632 805e.0c01.0af9 exists in HA table with feature psec and type dynamic
Dec 6 18:28:26 c3650- 372835: Dec 6 17:28:26: PSECURE: Adding 805e.0c01.0af9 as dynamic on port Gi3/0/2 for vlan 632

Dec 6 18:28:26 c3650- 372847: Dec 6 17:28:26: PSECURE: psecure_packet_enqueue: psecure receives a packet: addr = 805e.0c01.0af9, swidb = Gi3/0/2, vlan = 632, linktype = NullPak
Dec 6 18:28:26 c3650- 372849: Dec 6 17:28:26: PSECURE: swidb = GigabitEthernet3/0/2 mac_addr = 805e.0c01.0af9 vlanid = 632
Dec 6 18:28:26 c3650- 372850: Dec 6 17:28:26: PSECURE: Address vlan 632 805e.0c01.0af9 exists in HA table with feature psec and type dynamic
Dec 6 18:28:26 c3650- 372851: Dec 6 17:28:26: PSECURE: Adding 805e.0c01.0af9 as dynamic on port Gi3/0/2 for vlan 632

interface GigabitEthernet1/0/33
switchport access vlan 176
switchport mode access
switchport voice vlan 632
switchport port-security maximum 48
switchport port-security violation restrict
switchport port-security aging time 1
switchport port-security aging static
switchport port-security
no cdp enable
authentication control-direction in
authentication event fail retry 0 action authorize vlan 176
authentication event no-response action authorize vlan 176
authentication host-mode multi-auth
authentication timer reauthenticate server
authentication timer inactivity server dynamic
authentication violation protect
dot1x pae authenticator
dot1x timeout quiet-period 61
dot1x timeout server-timeout 5
dot1x timeout tx-period 3
dot1x timeout supp-timeout 5
storm-control broadcast level pps 500 100
storm-control action trap
no lldp transmit
spanning-tree portfast
spanning-tree bpduguard enable
end

sh version
Cisco IOS XE Software, Version 16.06.04
Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.4, RELEASE SOFTWARE (fc3)

 

For us it seems that port-security prevents the distribution of the arp broadcast.

If there are more information needed we will provide them but maybe someone knows about this behaviour.

Thanks in advance and best regards

Martin Insel 

[Deepak Kumar]

did you evaluated with removing 

storm-control broadcast level pps 500 100
storm-control action trap

[Martin Insel]

Thank you for this advice,

but after checking the the storm-control broadcast statistic while facing the issue we forgoe the opportunity to remove the two lines.

The current value for the affected port stays at 0.

Device#show storm-control broadcast | i Gi3/0/2
Gi3/0/2 Forwarding 500 pps 100 pps 0 pps Trap B

Best regards

Martin

ss

[Georg Pauwen]

Hello,

what brand/models are these phones ?

Not sure why the:

--> switchport port-security maximum 48

is used. With Cisco VoIP phones, you would only need 2 (with e.g. Avaya or other non-Cisco brands 3 as far as I recall).

What if you append the 'vlan access' to the port security commands ?

[Martin Insel]

We use three different phone models from Yealink but the issue don't refers to a special one. 

We will generally rework our port-scurity configuration because we also saw that the configuration is nonsense.

Meanwhile we saw drops in the Control Plane Policing (CPP) which increased constantly.

We compared the configuration with other switches in our network and they have configured the default rate of 600.

Device#sh platform hardware fed switch 1 qos queue stats internal cpu policer

                         CPU Queue Statistics
============================================================================================
                                              (default)  (set)     Queue
QId PlcIdx  Queue Name                Enabled   Rate     Rate      Drop(Bytes)
-----------------------------------------------------------------------------
12   0      BROADCAST                   Yes     600       200      147426782269

We reseted the cpp values of the affected switch to default and since then the broadcast drops stay at 0.

Now we observe the behaviour.

If the issue appears again we apply the reworked port-security configuration.

thank you!

regards

Martin