Interface Flapping

[jigar kothari]

Hi All,

I am having an issue with one of router interface flapping, Here are the logs.

May 22 07:54:28.917 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down

May 22 07:54:28.929 EDT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor 192.168.88.1 (FastEthernet0/0) is down: interface down

May 22 07:54:34.917 EDT: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.0.104.52 port 514 started - reconnection

May 22 07:54:39.917 EDT: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.0.105.17 port 514 started - reconnection

May 22 08:02:44.881 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up

May 22 08:02:44.997 EDT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor 192.168.88.1 (FastEthernet0/0) is up: new adjacency

May 22 08:02:50.881 EDT: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.0.104.52 port 514 started - reconnection

May 22 08:02:55.881 EDT: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.0.105.17 port 514 started - reconnection

2811#sh ver
Cisco IOS Software, 2800 Software (C2800NM-ENTBASEK9-M), Version 12.4(24)T3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Tue 23-Mar-10 06:43 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

Clev2811MPLS uptime is 15 weeks, 6 days, 3 hours, 30 minutes
System returned to ROM by power-on
System restarted at 05:48:05 EST Thu Jan 31 2013
System image file is "flash:c2800nm-entbasek9-mz.124-24.T3.bin"

Please suggest the solution if you have come across the issue.

[Richard Burts]

You have shown us one instance where Fast0/0 went down and 8 minutes later it came back up. We do not have nearly enough information yet to offer any accurate assessment of the issue. For starters can you supply information about these things:

- what is connected to Fast0/0? Is it a back to back connection to another router? Is it to a switch?

- Since there is an EIGRP neighbor on Fast0/0 there is a router somewhere in that connection. Can you provide details about how it is connected?

- did the router at 192.168.88.1 record any events in its log at this time?

- the logs indicate that there was a reconnection to logging servers in  10.0.104 and 10.0.105 at this same time can you tell us where these servers are an how they are connected? And what happened at this same time that caused them to reconnect?

- is it possible that the cable connecting to Fast0/0 was disconnected at this time?

- is it possible that whatever is on the other end of the connection to Fast0/0 had an issue at this time?

HTH

Rick

[jigar kothari]

Hi Richard,

Thanks for your reply.

- what is connected to Fast0/0? Is it a back to back connection to another router? Is it to a switch? Cisco 3750 Switch

- Since there is an EIGRP neighbor on Fast0/0 there is a router somewhere in that connection. Can you provide details about how it is connected?

- did the router at 192.168.88.1 record any events in its log at this time?

Log : 192.168.88.1 (3750 Switch)

May 22 12:02:48.681: %LINK-3-UPDOWN: Interface FastEthernet1/0/21, changed state to up

May 22 12:02:49.688: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/21, changed state to up

May 22 12:02:49.755: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.0.104.52 Port 514 started - reconnection

May 22 12:02:57.548: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel103, changed state to up

May 22 12:02:59.612: %DUAL-5-NBRCHANGE: EIGRP-IPv4 99: Neighbor 172.17.17.65 (Tunnel103) is up: new adjacency

May 22 12:03:07.111: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/15, changed state to up

May 22 12:03:08.361: %LINK-3-UPDOWN: Interface FastEthernet1/0/15, changed state to up

May 22 12:03:17.262: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to upMay 22 12:02:48.681: %LINK-3-UPDOWN: Interface FastEthernet1/0/21, changed state to up
May 22 12:02:49.688: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/21, changed state to up
May 22 12:02:49.755: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.0.104.52 Port 514 started - reconnection
May 22 12:02:57.548: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel103, changed state to up
May 22 12:02:59.612: %DUAL-5-NBRCHANGE: EIGRP-IPv4 99: Neighbor 172.17.17.65 (Tunnel103) is up: new adjacency
May 22 12:03:07.111: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/15, changed state to up
May 22 12:03:08.361: %LINK-3-UPDOWN: Interface FastEthernet1/0/15, changed state to up
May 22 12:03:17.262: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up

- the logs indicate that there was a reconnection to logging servers in 10.0.104 and 10.0.105 at this same time can you tell us where these servers are an how they are connected? And what happened at this same time that caused them to reconnect?

This are the solarwinds and Redcell Server. I checked on Physical server logs of application and server, no instance of droping tracffi or disconnection.

- is it possible that the cable connecting to Fast0/0 was disconnected at this time?

No

- is it possible that whatever is on the other end of the connection to Fast0/0 had an issue at this time?

I have attached the logs of the switch connected to 3750 above.

[Richard Burts]

There are several things in the logs that make it difficult to interpret what is going on.

- the time stamps of these logs are 4 hours off from the first set of logs. This makes it difficult to know whether this is really logs from a different time period or that the time on the routers and switches is not in sync with each other.

- there is no message in the second set of logs about anything that could be neighbor to 192.168.88.1. This implies either that it really is from a different time period or that it is not from the device that was running EIGRP.

- there are two separate entries indicating that  FastEthernet1/0/15 changed from down to up but no messages indicating that it went down.

Given that we have very little in the way of reliable information it is difficult to find what is the problem. But at this point I believe there is evidence that there are problems on the switch (why would an interface come up twice but not have any message about going down). Perhaps this problem on the switch is what caused the problem on the 2811 router.

HTH

Rick

[jigar kothari]

Exactly Rick, its quite wierd. Is it anything wrong with IOS:

3750Primary#sh ver                            
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 25-Sep-09 08:32 by sasyamal
Image text-base: 0x01000000, data-base: 0x02B00000

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE6, RELEASE SOFTWARE (fc1)

Clev3750Primary uptime is 1 day, 6 hours, 48 minutes
System returned to ROM by power-on
System restarted at 08:01:47 EDT Wed May 22 2013
System image file is "flash:c3750-ipbasek9-mz.122-52.SE.bin"

[Leo Laohoo]

Jigar,

Can you post the output to the following commands:

1.  Router:  sh interface F0/0;

2.  Switch:  sh interface ;

3.  Switch:  sh controller e

[jigar kothari]

Hi Leo,

ROuter:

sh int fa0/0

FastEthernet0/0 is up, line protocol is up

  Hardware is MV96340 Ethernet, address is 001b.54dd.a538 (bia 001b.54dd.a538)

  Internet address is 192.168.88.2/29

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:01, output 00:00:00, output hang never

  Last clearing of "show interface" counters 2d04h

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 294000 bits/sec, 176 packets/sec

  5 minute output rate 762000 bits/sec, 207 packets/sec

     29253332 packets input, 39052138 bytes

     Received 58140 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog

     0 input packets with dribble condition detected

     33402677 packets output, 3784889169 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

sh int fa1/0/15

FastEthernet1/0/15 is up, line protocol is up (connected)

  Hardware is Fast Ethernet, address is 001f.27e0.c591 (bia 001f.27e0.c591)

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:42, output 00:00:01, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     8300 packets input, 1035225 bytes, 0 no buffer

     Received 1849 broadcasts (1847 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 1847 multicast, 0 pause input

     0 input packets with dribble condition detected

     101121 packets output, 7638046 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

     Transmit FastEthernet1/0/15              Receive

      7643356 Bytes                          1035917 Bytes                   

        17786 Unicast frames                    6455 Unicast frames          

        81060 Multicast frames                  1848 Multicast frames        

         2347 Broadcast frames                     2 Broadcast frames        

            0 Too old frames                  452583 Unicast bytes           

            0 Deferred frames                 258744 Multicast bytes         

            0 MTU exceeded frames                774 Broadcast bytes         

            0 1 collision frames                   0 Alignment errors        

            0 2 collision frames                   0 FCS errors              

            0 3 collision frames                   0 Oversize frames         

            0 4 collision frames                4656 Undersize frames        

            0 5 collision frames                   0 Collision fragments     

            0 6 collision frames      

            0 7 collision frames                4654 Minimum size frames     

            0 8 collision frames                6453 65 to 127 byte frames   

            0 9 collision frames                1850 128 to 255 byte frames  

            0 10 collision frames                  4 256 to 511 byte frames  

            0 11 collision frames                  0 512 to 1023 byte frames 

            0 12 collision frames                  0 1024 to 1518 byte frames

            0 13 collision frames                  0 Overrun frames          

            0 14 collision frames                  0 Pause frames            

            0 15 collision frames     

            0 Excessive collisions                 0 Symbol error frames     

            0 Late collisions                      0 Invalid frames, too large

            0 VLAN discard frames                  0 Valid frames, too large 

            0 Excess defer frames                  0 Invalid frames, too small

        70851 64 byte frames                       0 Valid frames, too small 

        28431 127 byte frames         

            5 255 byte frames                      0 Too old frames          

         1875 511 byte frames                      0 Valid oversize frames   

           31 1023 byte frames                     0 System FCS error frames 

            0 1518 byte frames                     0 RxPortFifoFull drop frame

            0 Too large frames        

            0 Good (1 coll) frames    

            0 Good (>1 coll) frames  

sh int fa0/0
FastEthernet0/0 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 001b.54dd.a538 (bia 001b.54dd.a538)
  Internet address is 192.168.88.2/29
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters 2d04h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 294000 bits/sec, 176 packets/sec
  5 minute output rate 762000 bits/sec, 207 packets/sec
     29253332 packets input, 39052138 bytes
     Received 58140 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     33402677 packets output, 3784889169 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

Switch:

sh int fa1/0/15
FastEthernet1/0/15 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 001f.27e0.c591 (bia 001f.27e0.c591)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:42, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     8300 packets input, 1035225 bytes, 0 no buffer
     Received 1849 broadcasts (1847 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1847 multicast, 0 pause input
     0 input packets with dribble condition detected
     101121 packets output, 7638046 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Show Controller ethernet-controller:

     Transmit FastEthernet1/0/15              Receive
      7643356 Bytes                          1035917 Bytes                   
        17786 Unicast frames                    6455 Unicast frames          
        81060 Multicast frames                  1848 Multicast frames        
         2347 Broadcast frames                     2 Broadcast frames        
            0 Too old frames                  452583 Unicast bytes           
            0 Deferred frames                 258744 Multicast bytes         
            0 MTU exceeded frames                774 Broadcast bytes         
            0 1 collision frames                   0 Alignment errors        
            0 2 collision frames                   0 FCS errors              
            0 3 collision frames                   0 Oversize frames         
            0 4 collision frames                4656 Undersize frames        
            0 5 collision frames                   0 Collision fragments     
            0 6 collision frames      
            0 7 collision frames                4654 Minimum size frames     
            0 8 collision frames                6453 65 to 127 byte frames   
            0 9 collision frames                1850 128 to 255 byte frames  
            0 10 collision frames                  4 256 to 511 byte frames  
            0 11 collision frames                  0 512 to 1023 byte frames 
            0 12 collision frames                  0 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames          
            0 14 collision frames                  0 Pause frames            
            0 15 collision frames     
            0 Excessive collisions                 0 Symbol error frames     
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                  0 Valid frames, too large 
            0 Excess defer frames                  0 Invalid frames, too small
        70851 64 byte frames                       0 Valid frames, too small 
        28431 127 byte frames         
            5 255 byte frames                      0 Too old frames          
         1875 511 byte frames                      0 Valid oversize frames   
           31 1023 byte frames                     0 System FCS error frames 
            0 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames        
            0 Good (1 coll) frames    
            0 Good (>1 coll) frames   

[Leo Laohoo]

Thanks for the output.  Here are some of my "concerns":

I do not see any line errors from the router interface, HOWEVER, I did notice that the interface counters were recently cleared (I'll ask a follow-up question shortly).

The "sh interface" of the switch does not show any line errors at all but the counters were NOT cleared.  I am, however, curious with the output to the "sh controller ether" particularly to the presense of "Undersized Frames".  I've never encountered that before so I won't be able to comment any further to the indication of it.

Someone, and I don't remember who, recently posted an old Cisco link detailing exactly the explanation and potential causes of each of the lines in the "sh controller ethernet" counter commands.

My question is:  IF the counters of the router was recently cleared, was the switch recently rebooted as well?

Version 12.2(52)SE

I would consider upgrading to 12.2(55)SE7.  The IOS you are using ain't really that good.

[jigar kothari]

Leo, thanks for rthe reply. I will try upgrading the switch, check if that changes anything.