BGP ATT MPLS

[Andy Guley]

I am having an issue with an ATT MPLS Ethernet circuit with BGP.  I have a 140 MPLS circuit network with ATT and this is the only site where I am having this type of issue.  

From CER to the CPE BGP will flap and the circuit will go up and down.  The circuit will finally go down hard.  I see down down on my router.  ATT tests the circuit and say they are testing clean to their equipment.  If I go into my equipment and shut no shut the physical interface that connects to the ATT equipment the port goes UP UP and BGP is re-established with no issues.  

*Sep 26 23:37:36.319: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

*Sep 26 23:37:37.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

*Sep 26 23:37:51.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up

*Sep 26 23:37:52.319: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

*Sep 26 23:37:53.195: %BGP-5-ADJCHANGE: neighbor 172.X.X.X Up

*Sep 26 23:40:07.319: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

*Sep 26 23:40:08.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

*Sep 26 23:40:08.319: %BGP-5-NBR_RESET: Neighbor 172.X.X.X reset (Interface flap)

*Sep 26 23:40:08.339: %BGP-5-ADJCHANGE: neighbor 172.X.X.X Down Interface flap

*Sep 26 23:40:08.339: %BGP_SESSION-5-ADJCHANGE: neighbor 172.X.X.X  IPv4 Unicast topology base removed from session  Interface flap

*Sep 26 23:45:20.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up

*Sep 26 23:45:21.319: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

*Sep 26 23:45:22.527: %BGP-5-ADJCHANGE: neighbor 172.X.X.X  Up

*Sep 26 23:46:53.319: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

*Sep 26 23:46:54.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

*Sep 26 23:46:54.319: %BGP-5-NBR_RESET: Neighbor 172.X.X.X reset (Interface flap)

*Sep 26 23:46:54.339: %BGP-5-ADJCHANGE: neighbor 172.X.X.X  Down Interface flap

*Sep 26 23:46:54.339: %BGP_SESSION-5-ADJCHANGE: neighbor 172.X.X.X  IPv4 Unicast topology base removed from session  Interface flap

*Sep 26 23:46:56.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up

*Sep 26 23:46:59.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

*Sep 27 00:01:16.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up

*Sep 27 00:01:17.319: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

*Sep 27 00:01:24.171: %BGP-5-ADJCHANGE: neighbor 172.X.X.X  Up

*Sep 27 00:02:18.319: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

*Sep 27 00:02:19.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

*Sep 27 00:02:19.319: %BGP-5-NBR_RESET: Neighbor 172.X.X.X  reset (Interface flap)

*Sep 27 00:02:19.339: %BGP-5-ADJCHANGE: neighbor 172.X.X.X  Down Interface flap

*Sep 27 00:02:19.339: %BGP_SESSION-5-ADJCHANGE: neighbor 172.X.X.X  IPv4 Unicast topology base removed from session  Interface flap

*Sep 27 00:10:11.971: %LINK-5-CHANGED: Interface GigabitEthernet0/1, changed state to administratively down

*Sep 27 00:10:16.963: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

*Sep 27 00:10:20.319: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up

*Sep 27 00:10:21.319: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

*Sep 27 00:10:23.911: %BGP-5-ADJCHANGE: neighbor 172.X.X.X Up

[Mark Malone]

Hi

do you see anything on your local physical interface that may be causing it ? inputs , crcs etc or is it clean nothing incrementing

Is the mtu the same as the other circuits ? nothing different setup, same config setup as the other 140 circuits in general bgp config

Have ATT even swapped out the local mux in case its a lower layer issue , if its constantly happening I would request that to rule out their kit

If its happening frequently you could run a debug to the specific neighbour send it to your buffer and try and capture the more relevant info as that only show you its flapping

[Andy Guley]

Thanks for the response.

MTU 1500 bytes, BW 10000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full Duplex, 100Mbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 63
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 15000 bits/sec, 11 packets/sec
  5 minute output rate 15000 bits/sec, 11 packets/sec
     827422 packets input, 330216082 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     887479 packets output, 591924085 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     13 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

MTU size is the same as it is on the rest of the circuits. 

ATT is requesting inside wiring be verified at the site.  I am having LCON swap out the cat6 cable.  I highly doubt it's going to be the cabling.  But I cant contest until this is done. 

[Mark Malone · Accepted solution]

Hi

Looking at that output there's definitely a layer 1 issue there whether its cabling or connecting mux t fault , 9/10 its ISP side but you will most likely have to replace local cabling to rule them out

  13 lost carrier, 0 no carrier, 0 pause output

lost carrier and no carrier—The carrier is an electrical signal that Ethernet devices use to detect whether the wire is currently being used by another transmitting station.

The lost carrier counter increases each time a carrier sense loss occurs. This happens when the hardware is transmitting a frame onto the wire and does not see its own carrier wave on the Ethernet. The absence of the carrier signal increments the no carrier counter.

lost carrier

Description: Cisco IOS sh interfaces counter. The number of times the carrier was lost in transmission. Common Causes: Check for a bad cable. Check the physical connection on both sides.