Unknown Protocol Drop observed on router interfaces

[shuaib.khalid]

Dear Friends,

Couple of months ago we have upgraded our network infrastructure, before the activity we had only one core router both primary and secondary WAN links were terminated on the same router. Now we have two core routers primary WAN link is terminated on primary router (Cisco 2911) and secondary link is on secondary router (Cisco 2811). Both core routers are connected in core switches (Cisco 3560). The juniper firewalls are also connected in the core switches which is gateway for our LAN. I have attached the logical core network design for reference.

We created BVI interfaces on both routers and bridge two physical interfaces in it.

Primary router = int BVI1 (fa0/0/0 & Gi0/1) IP addresses 66.77.88.13 & 1.1.1.3 sec

Secondary router = int BVI1 (fa0/0 & fa0/1) IP addresses 66.77.88.11 & 1.1.1.4 sec

To control the redundancy we have configured HSRP on both BVI interfaces,

HSRP VIP: 66.77.88.9 & 1.1.1.1 sec

Firewall is forwarding internet and data traffic to HSRP VIPs.

We have also configured EIGRP to get the secondary WAN link routes on primary router, in case of primary link down the primary router will forward the traffic to secondary router. The EIGRP neighbouship is also build on BVI interfaces. I have also BGP neighbourship with my Service Provider for Internet and Data MPLS connectivity.

The above scenario was working fine but a week ago the whole WAN connectivity went down and there was no failover to secondary router. I was unable to access the routers from the LAN. When I checked the routers through console the IP routing was disabled and all routing protocols and static routes were vanished from both routers. No log found for the incident. I reconfigured the both routers, connectivity went up for some hours and went down again, the situation of both routers was same. I faced this issue for whole week.

When I dig out the issue I observed that there were unkown protocol drops on both router interfaces connected to core switches. I am running STP on switched network including core switches and no DTP or LLDP is enabled, CDP is also enabled on all cisco devices. I am unable to understand which type of traffic from the switches is causing for "Unknown Protocol Drops", please also tell me that is this issue can be cause for the incident stated above?

[acampbell]

Hi,

On your 3560 interfaces tat face the routers can you check that DTP is definately off

!
interface gig 0/5
switchport nonegotiate
!

Routers will not understand DTP packets and show them as unknown protocol

Hope this helps

Regards
Alex

[Richard Burts]

Alex has suggested checking on one of the common causes of unknown protocol drops which is DTP. But there are other kinds of traffic from a switched environment which might cause unknown protocol drops. It might be helpful if the original poster would share with us the output from the core switch of the commands show interface gi0/5 and show interface switchport gi0/5.

While the unknown protocol drops are an interesting problem, the real problem here is what is causing the router to change its configuration and to disable ip routing. I find it difficult to see how an unknown protocol drop could have that effect.

Am I correct in understanding that the router continues to run when this problem happens (there is not a reboot or reload)? If so then perhaps one step that I would suggest would be to enable logging buffer debug with a fairly large logging buffer. Then when the problem happens again log into the console and do a show log to get what is in the logging buffer. Hopefully there will be sign of some event at the time that the problem begins.

HTH

Rick

[shuaib.khalid]

Hi Bros,

Thanks for your responses,

Below are the required configs. DTP status on both switches is also attached (data & time was not set on switches which i have set now)

Primary router

interface GigabitEthernet0/1
description *** CONNECTED WITH PRIMARY SWITCH ****
no ip address
ip flow ingress
ip flow egress
ip virtual-reassembly in
duplex auto
speed auto
bridge-group 1

Secondary Router:

interface FastEthernet0/1
description *** CONNECTED WITH BACKUP SWITCH ***
no ip address
ip flow ingress
ip flow egress
ip virtual-reassembly
duplex auto
speed auto
bridge-group 1

Core Switch1

interface GigabitEthernet0/5
description *****PRI-ROUTER*****
switchport access vlan 30
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 30
switchport mode access

Core Switch2

interface GigabitEthernet0/5
description ****SEC-ROUTER****
switchport access vlan 30
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 30
switchport mode access

[acampbell]

Hi,

Why have you configured the switch ports to be both access & trunk ?

If the routers are purely just end devices in vlan 30 then reconfigure
your switch ports :-

int gig 0/5
shut
description *****PRI-ROUTER*****
no switchport trunk encapsulation dot1q
no switchport trunk allowed vlan 30
switchport mode access
switchport access vlan 30
switchport nonegotiate
no shut
!

Regards
Alex

[shuaib.khalid]

Hi Alex,

Thank you for guidance, I have to take some downtime for this configuration change because this port contains traffic of all switched network towards primary router, i will let you know after getting it done.

Regards,

Shuaib
 

[Richard Burts]

Shuaib

In the original post you tell us that " the IP routing was disabled and all routing protocols and static routes were vanished from both routers." I would like clarification on a couple of points.

- was this the condition on both routers, that IP routing was disabled and all routing protocols and routes were removed?

- has this happened more than once?

HTH

Rick

[shuaib.khalid]

Hi Rick,

Yes I face same issue on both routers for multiple time, at that time i was unable to ping my routers from LAN until i reload them. I have checked the CPU usage which was normal and also checked the hardware health of both routers with Cisco authorized vendor and they also reported that they are fine.

[Richard Burts]

It is interesting that it happens on both routers and that it has happened multiple times. How often does this happen (how frequently)? I will repeat my suggestion about setting up logging buffered with a good sized buffer and then monitoring what is in the logs as a way to identify what is going on.

HTH

Rick

[Amit Goyal]

Hi Shuaib,

Do you have snmp-server commands enabled on your WAN routers?

snmp-server community <community name> RW

With the above command in place without any ACL, any one from outside can disable the "ip routing" on your device.

in ideal scenario you must see a log generated whenever "ip routing" is disabled. But you may not be able to see the logs because of below bug.

https://tools.cisco.com/bugsearch/bug/CSCup37781

You may need to change the snmp command as per below

snmp-server community <community name> RO

HTH

-Amit