Duplicate Ip address + Huge outage in the network

[satishmothukri]

Hi All,           

We were seening below messaage on Core switches during outage last week.During the issue time all switches were not rechable from outside network where as routers are reachable.

Note : Switches started pinging from outside once i clear PE learned routes using clear ip bgp <PE IP> soft in command.

:12:32 gmt: %STANDBY-3-DUPADDR: Duplicate address 145.18.44.251 on Vlan19, sourced by 0000.0c07.ac01
Dec 12 18:12:36 gmt: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 110: Neighbor 10.172.31.13 (Vlan903) is up: new adjacency
Dec 12 18:12:40 gmt: %STANDBY-6-STATECHANGE: Vlan302 Group 1 state Active -> Speak
Dec 12 18:12:40 gmt: %STANDBY-6-STATECHANGE: Vlan32 Group 1 state Active -> Speak
Dec 12 18:12:46 gmt: %STANDBY-6-STATECHANGE: Vlan302 Group 1 state Standby -> Active
Dec 12 18:12:46 gmt: %STANDBY-6-STATECHANGE: Vlan32 Group 1 state Standby -> Active
Dec 12 18:12:47 gmt: %STANDBY-6-STATECHANGE: Vlan302 Group 1 state Active -> Speak
Dec 12 18:12:49 gmt: %LINK-3-UPDOWN: Interface Vlan12, changed state to down
Dec 12 18:12:49 gmt: %STANDBY-6-STATECHANGE: Vlan12 Group 1 state Active -> Init
Dec 12 18:12:52 gmt: %LINK-3-UPDOWN: Interface Vlan12, changed state to up
Dec 12 18:12:52 gmt: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 110: Neighbor 10.172.13.3 (Vlan990) is down: holding time expired
Dec 12 18:12:53 gmt: %STANDBY-6-STATECHANGE: Vlan32 Group 1 state Active -> Speak
Dec 12 18:12:53 gmt: %STANDBY-6-STATECHANGE: Vlan302 Group 1 state Standby -> Active
Dec 12 18:12:53 gmt: %PIM-5-DRCHG: DR change from neighbor 0.0.0.0 to 145.18.3.250 on interface Vlan12 (vrf default)
Dec 12 18:12:55 gmt: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 110: Neighbor 10.172.31.5 (Vlan901) is up: new adjacency
Dec 12 18:12:57 gmt: %LINK-3-UPDOWN: Interface Vlan26, changed state to down

Below is the setup:

Core SW1 ( HSRP+ EIGRP) ----> (EIGRP+EBGP) Core Router1(CE Router) ----PE Router ---MPLS cloud.
                 |                                                |
      Trunk   |                                    IBGP    |

Core SW2 ( HSRP+ EIGRP) ----> (EIGRP+EBGP) Core Router2(CE Router) ----PE Router ---MPLS cloud.

Cross connectivity is there between Core SW1 to Core Router 2 and Core Sw2 to Core Router 1 for redundancy.

Note : On Core switches (WS-C6509+ MSFC3 (R7000)) , HSRP group is 1 for all Vlan's configured.
HSRP and EIGRP was fluctuated during issue time and all the switches are reachable after executing command clear ip bgp <PE IP> soft in on core routers.
Dampening is configured on PE router.

I have checked below things:

Root switch for vlan 19 on all access switches is Core Switch1.
Eigrp hello and hold down timers on Cire switches and core routers are defualt ones.

Observation :

On access sw3 , 3 com switch is connected on port 16/17 which is being used for LAB devices connectivity purpose.

Any inputs from any one to know root cause for this issue.It happened twice.

Thanks ,
M S K       

[mahmoodmkl]

Hi

these messages r usually caused due to stp loop.
please check u r topology to find any issues

Sent from Cisco Technical Support iPhone App

[Srin_G]

this is due to loop, we had the same issue.One of the Unix guys plugged in their vio equipment configured with layer 3 etherchannel and STP got screwed. Do you have rootguard or bpduguard enabled in your switched network?

[Giuseppe Larosa]

Hello Satish,

the log messages are clearly related to a bridging loop that happened in your network.

The fact that you have gained access to the network after issuing clear ip bgp PE-IP on core routers may be a side effect.

During the fault the PE routers may have advertised to core routers reachability of the IP subnets that have been affected by the bridging loop, and the clear ip bgp may have fixed redistribution of EIGRP into BGP on core routers making the EIGRP routes installed again in the IP routing table of the core routers instead of eBGP routes coming from PE routers.

You should investigate on spanning-tree activity.using show spanning-tree detail

That third-party device 3com switch connected to two different switch ports on switch3 may be part of the issue. It may be wise to disable one of the two ports.

On PE routers the use of site of origin extended community may avoid in the future advertisement of site internal subnets from the SP backbone to the CE/core routers.So you should also check with SP tech people if this tuning can be performed.

Hope to help

Giuseppe

[milan.kulik]

Hi,

how are the ports to which the 3Com switch was connected configured?

Access ports? Trunks?

I remember a problem with 3Com switch some years ago - it did not forward muticasts by defalt.

And Cisco PVSTP BPDUs are just muticast frames fro the 3Com point of view!

So it could create an STP loop, I guess?

I also agree with Guiseppe:

Are you running a mutual EIGRP/BGP redistribution in your LAN?

If yes, what happens if some of your LAN subnets becomes unreachable on one of your BGP routers?

Will it accept the same prefix advertised from MPLS?

If yes, what happens when the subnet is reachable via EIGRP again?

BR,

Milan