Re: MPLS site never came back online

spurs2k18 · ‎05-09-2018

Ok Ill try and keep this as short as possible. Last week we had our MPLS service interrupted. Well once services were restored, all sites (we have 15 sites on MPLS) came back up except one. First thing I did was contact ATT. They were able to loop smart jack, and verified all was good on their end. They even sent tech out 3 different times to verify DMARC. Well this store never came back up, and is still down. I dont know if something got hosed on config during the outage but WIC showing yellow / amber light, from which I was reading indicates there is not a connection to a remote site (which I assume is ATT). Here is the config for that site that is down. Not familiar with BGP so not sure that when I saw remote router ID was set to 0.0.0.0 where other sites had a 12.x.x.x ip assigned to it. The neighbor I'm assuming would be ATT? Not too sure. Thanks!!!

ip dhcp database flash:dhcp.log
no ip dhcp use vrf connected
ip dhcp excluded-address 10.1.77.250 10.1.77.254
ip dhcp excluded-address 172.18.77.250 172.18.77.254
!
ip dhcp pool voice
network 172.18.77.0 255.255.255.0
option 150 ip 192.168.3.32 192.168.3.33
default-router 172.18.77.254
dns-server 192.168.2.42 192.168.2.43
domain-name abc.local
!
ip dhcp pool data
network 10.1.77.0 255.255.255.0
domain-name abc.local
dns-server 192.168.2.42 192.168.2.43
default-router 10.1.77.254
!
!
no ip domain lookup
ip domain name abc.local
!
voice-card 0
!
!
!
voice service voip
fax protocol t38 ls-redundancy 0 hs-redundancy 0 fallback pass-through g711ulaw
!
!
!
!
!
!
!
!
!
!
!
application
global
service alternate default

!

class-map match-any voice-media
match ip dscp ef
class-map match-any call-control
match ip dscp cs3
match ip dscp af31
class-map match-any AutoQoS-VoIP-RTP-Trust
match ip dscp ef
class-map match-any AutoQoS-VoIP-Control-Trust
match ip dscp cs3
match ip dscp af31
!
!
policy-map AutoQoS-Policy-Trust
class AutoQoS-VoIP-RTP-Trust
priority percent 70
class AutoQoS-VoIP-Control-Trust
bandwidth percent 5
class class-default
fair-queue
policy-map qos-wan
class voice-media
priority 64
class call-control
bandwidth 8
class class-default
fair-queue
random-detect
!
!
!
!
interface FastEthernet0/0
no ip address
shutdown
duplex auto
speed auto
!
interface FastEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
interface FastEthernet0/1/0
description IP Phone
switchport mode trunk
switchport voice vlan 2
load-interval 30
spanning-tree portfast
service-policy output AutoQoS-Policy-Trust
!
interface FastEthernet0/1/1
description WIRELESS_AP
switchport mode trunk
load-interval 30
spanning-tree portfast
service-policy output AutoQoS-Policy-Trust
!
interface FastEthernet0/1/2
description ATA 186
switchport mode trunk
switchport voice vlan 2
load-interval 30
spanning-tree portfast
service-policy output AutoQoS-Policy-Trust
!
interface FastEthernet0/1/3
switchport stacking-partner interface FastEthernet0/3/0
!
interface FastEthernet0/3/0
switchport stacking-partner interface FastEthernet0/1/3
!
interface FastEthernet0/3/1
description NSB-POS
switchport access vlan 3
spanning-tree portfast
service-policy output AutoQoS-Policy-Trust
!
interface FastEthernet0/3/2
description NSB-POS
switchport access vlan 3
spanning-tree portfast
service-policy output AutoQoS-Policy-Trust
!
interface FastEthernet0/3/3
description Internet MoH Device
load-interval 30
spanning-tree portfast
service-policy output AutoQoS-Policy-Trust
!
interface Serial0/2/0
ip address 12.84.112.54 255.255.255.252
ip flow ingress
encapsulation ppp
load-interval 30
service-module t1 timeslots 1-24
service-policy output qos-wan
!
interface Vlan1
description data vlan
ip address 10.1.77.254 255.255.255.0
!
interface Vlan2
description voice vlan
ip address 172.18.77.254 255.255.255.0
!
interface Vlan3
ip address 10.134.77.254 255.255.255.0
!
router bgp 65015
no synchronization
bgp router-id 12.84.112.54
bgp log-neighbor-changes
redistribute connected
neighbor 12.84.112.53 remote-as 7018
no auto-summary
!
!
ip flow-export version 5
ip flow-export destination 192.168.2.20 2055
ip flow-top-talkers
top 20
sort-by bytes
!
ip http server
ip http secure-server
ip http timeout-policy idle 600 life 86400 requests 10000
!
ip access-list extended srst-test
permit eigrp any any
permit tcp any eq telnet any
permit tcp any eq 22 any
deny ip any host 192.168.3.32
deny ip any host 192.168.3.33
deny ip any 192.168.3.0 0.0.0.255
ip access-list extended srst-test2
permit eigrp any any
permit tcp any any eq telnet
permit tcp any any eq 22
deny ip any any
!
logging 192.168.2.20
access-list 7 permit 10.10.0.1
access-list 7 permit 192.168.2.112
access-list 7 permit 192.168.2.118
access-list 7 permit 192.168.2.34
access-list 7 permit 192.168.3.32
access-list 7 permit 192.168.2.19
access-list 7 permit 192.168.2.20
access-list 7 permit 192.168.2.200
access-list 7 permit 64.132.239.145
access-list 7 permit 192.168.32.150
snmp-server community nw@comsol RW 7
!
!
!

sgt2111usmc · ‎05-09-2018

"remote router ID was set to 0.0.0.0 where other sites had a 12.x.x.x ip assigned to it."

What do your logs show? Did you verify that ATT still has the same BGP ASN (65015) on their end? Can you ping your neighbor (12.84.112.53)?

Do the following:

clear ip bgp (on that router that hasn't come up. Wait a minute see if bgp came back up)

ping 12.84.112.53

sho ip bgp neighbors | s 12.84.112.53

show ip bgp summary

show ip bgp summary | s 12.84.112.52

Thanks,

Matt

spurs2k18 · ‎05-09-2018

Yes, the ASN is correct but unable to hit 53. I will attempt clear ip bgp this afternoon when tech gets on site (have to remote in his laptop to access router via putty), and will post results

cofee · ‎05-09-2018

Hi,

Can you verify if interface Serial0/2/0 is showing as up/up? Are you able to ping your bgp neighbor 12.84.112.53 ?

Please run these commands and provide the output "sh ip bgp summary" and "sh ip bgp neighbors".

spurs2k18 · ‎05-09-2018

Serial int is set to down/down, and unable to bring up. I replaced the WIC just to verify that wasnt the issue. Now I'm unable to hit .53, but for some strange reason, Meraki AP is now showing up in the inventory but with not gateway to the internet. I dont think it would be the actual router, but I'm almost out of options.

cofee · ‎05-09-2018

I would assume most likely layer 1 connectivity is the issue. Won't make any sense to check anything else before layer 1/2 connectivity is restored between CE and PE routers.

spurs2k18 · ‎05-09-2018

I have ATT on site right now as we speak. Have asked them to test connection from WIC directly to their end (there is an extension from ATT original DMARC to suite). Will update shortly

sgt2111usmc · ‎05-09-2018

That's correct.

spurs2k18 · ‎05-09-2018

ATT didnt give me much info, and its hard to get in contact with dispatched techs on site for whatever reason, but what I am wondering about is that if other MPLS sites (which all touch base back home at corp office), each have a different "remote-router ID" the down site needs one as well. Not sure what the remote-router would be as in if its something I would need to get from ATT or if I could use anything. Still working on it

cofee · ‎05-09-2018

Selection process for BGP router ID:

•Use the address configured by the bgp router-id command.
•Use the Loopback interface address with the highest IP address.
•Use the highest IP address of the interface.

You can manually configure your global IPV4 address to be used as BGP router-ID. Quite possible BGP is already using it as the BGP router-ID if it's the highest IP address assigned to a physical interfaces and you don't have any loopbacks configured.

spurs2k18 · ‎05-09-2018

So this is from a MPLS site thats currently up.

hou080-rt1#sh ip bgp nei
BGP neighbor is 12.84.19.229, remote AS 7018, external link
BGP version 4, remote router ID 12.122.124.240 <----Down site has this set to 0.0.0.0 this is what im confused on
BGP state = Established, up for 1w0d
Last read 00:00:03, last write 00:00:52, hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
Route refresh: advertised and received(old & new)
Address family IPv4 Unicast: advertised and received
Message statistics:
InQ depth is 0
OutQ depth is 0
Sent Rcvd
Opens: 2 2
Notifications: 0 0
Updates: 2 1580
Keepalives: 43619 43616
Route Refresh: 0 0
Total: 43623 45198
Default minimum time between advertisement runs is 30 seconds

cofee · ‎05-09-2018

That's normal. Because at one point you had eBGP peering with this neighbor and now you don't, therefore it's showing the router-id of a down eBGP neighbor as 0.0.0.0. Once the connection comes back up remote router's router-id will be replaced by a valid router-id.

spurs2k18 · ‎05-09-2018

Ok great. Thats what i was assuming but not sure. I will be on site in 2 days to further troubleshoot issue.