CSS11501 and OSPF problems

allen.external · ‎09-29-2004

I pluged my CSS into the network and this Caused OSPF to start flapping. Does anybody know why this would happen. Nothing else was effected just OSPF. I had to unplug the CSS inorder to stablize the network. This was not a spanning tree problem as I had that turned off. Any help from anybody would be great. The models were CSS11501 v7.20 and a 7500 v.12.1. Again, I am desperate as I cannot plug my CSS back into the network until I find an answer. Any suggestions would be great.

Justin

Gilles Dufour · ‎09-30-2004

how many interface did you plug to the CSS ?

What vlan was configured ?

Do you have ospf enable on the CSS ?

Is ospf traffic supposed to go through the CSS ?

Did you enable some debug ospf on the routers to get a reason for the flap ?

What about a sniffer trace to see what is happening.

What's the exact software version of the CSS ? 7.20(??) ?

If you can answer to these questions I'm sure you'll find the answer to your problem.

Gilles.

allen.external · ‎09-30-2004

Sorry this one is going to be a book. Only one interface E1 on Vlan 1. I dont have ospf enabled. It is a one armed config. I also had all the services disabled but did have VRRP and APP running. I did not have the ISC port pluged in. Only E1. I could not leave it running as the entire segment was down. I did do a write up on what ospf did. I took the css back into the lab and tried to recrete the problem and it worked fine. Active version: sg0720003

What happened to OSPF when we plugged the CSS into the network?

In the scenario of the CSS, OSPF went down due to the fact OSPF was only passing one way traffic. For some reason this only effected OSPF. As you can see below A1d is receiving hellos from E2d. You can see this through the transition from EXSTART to DOWN on A1d.

Sep 28 15:07:10.007: %OSPF-5-ADJCHG: Process 1, Nbr 11.1.1.2 on FastEthernet10/0

/0 from FULL to DOWN, Neighbor Down: Dead timer expired

Sep 28 15:09:30.014: %OSPF-5-ADJCHG: Process 1, Nbr 11.1.1.2 on FastEthernet10/0

/0 from EXSTART to DOWN, Neighbor Down: Dead timer expired

Sep 28 15:11:10.019: %OSPF-5-ADJCHG: Process 1, Nbr 11.1.1.2 on FastEthernet10/0

/0 from EXSTART to DOWN, Neighbor Down: Too many DBD retransmitions

Sep 28 15:12:10.018: %OSPF-5-ADJCHG: Process 1, Nbr 11.1.1.2 on FastEthernet10/0

/0 from DOWN to DOWN, Neighbor Down: Ignore timer expired

.Sep 28 15:14:00.040: %OSPF-5-ADJCHG: Process 1, Nbr 11.1.1.2 on FastEthernet10/

0/0 from EXSTART to DOWN, Neighbor Down: Dead timer expired

If you look on the other neighbor E2d we can see he never receives a response back from A1d and goes directly to a DOWN state.

.Sep 28 15:11:16.848: %OSPF-5-ADJCHG: Process 1, Nbr 11.1.1.1 on FastEthernet10/

0/0 from EXSTART to DOWN, Neighbor Down: Too many DBD retransmitions

.Sep 28 15:12:16.847: %OSPF-5-ADJCHG: Process 1, Nbr 11.1.1.1 on FastEthernet10/

0/0 from DOWN to DOWN, Neighbor Down: Ignore timer expired

Exstart: Routers are trying to establish the initial sequence number that is going to be used in the information exchange packets. The sequence number insures that routers always get the most recent information. One router will become the primary and the other will become secondary. The primary router will poll the secondary for information. Normally this would not be a problem and E2d became the DR as it should but since the HSRP owner was A1d, which is the router that was having the problem, and it did not know who to send the traffic to or who was DR/BDR it was dropping any traffic trying to leave the subnet. It could not decide whether it was the DR or the BDR due to the packets being sent by E2d.

Conclusion

Most likely traffic was getting the .24 network but it could not leave due to A1d, (the gateway at the time), not knowing where to send the packets too. We still do not know why the CSS did this to OSPF. I am still working on that. I just wanted to show you how OSPF was reacting when we plugged the CSS into the network.

Gilles Dufour · ‎10-01-2004

OSPF uses multicast ip address to exchange hello packets.

IP addresses are 224.0.0.5 and 224.0.0.6.

So, ALL devices in the vlan should receive OSPF sent by the routers.

So unless the CSS was sending corrupted ospf frames, I don't see how it could prevent yout E2d device to not receive the multicast traffic.

What kind of switch is connecting your routers and the CSS ?

I still believe a sniffer trace on this vlan is needed to identify is the CSS sends ospf traffic and to see if Hello packets are correctly being sent and received by the routers.

One remark about the CSS version.

You should better use a later version.

The one you have is like running IOS 12.2(1).

It's the very first release in this train.

Go for 7.20(x) where is x is the highest value available on CCO or better go for 7.30(x).

Regards,

Gilles.

allen.external · ‎10-01-2004

Yeah, I tried the sniffer trace thing in the lab I found it was only spewing VRRP 224.0.0.18, ICMP Keepalives, and traffic going from port 1024 to 5001. My guess is the traffic on the ports 1024 and 5001 is APP. But that is a guess. I did not see any sighns of OSPF Traffic.

allen.external · ‎10-08-2004

Well it did it again. It takes it about 20min for OSPF to start going nuts. I plug them into the lab everything works fine. I plug them into the production network OSPF dies after 20min or so. This is the log and it keeps doing this over and over. Mean while we cannot route traffic. Does anybody have any ideas. Could it be hardware related? We have the same version of code in the lab. Differnt hardware though. Any help would be much much much much much much appriciated. This is just bizzare. It only effects OSPF and nothing else. STP is fine. HSRP is fine.

Oct 7 19:19:22.276: OSPF: Neighbor change Event on interface FastEthernet10/0/0

Oct 7 19:19:22.276: OSPF: DR/BDR election on FastEthernet10/0/0

Oct 7 19:19:22.276: OSPF: Elect BDR 10.1.1.1

Oct 7 19:19:22.276: OSPF: Elect DR 10.1.1.2

Oct 7 19:19:22.276: DR: 10.1.1.2 (Id) BDR: 10.1.1.1 (Id)

Gilles Dufour · ‎10-12-2004

did you capture a sniffer trace ?

Did you try to ping 224.0.0.5 and 224.0.0.6 to see if traffic was flowing correctly to all devices ?

Gilles.

allen.external · ‎10-12-2004

Yeah, I got a trace and a debug ospf adj. This is the thing. The local area network is fine. Here is another funny thing. It only happens after 30min. I checked for layer 2 loops and could not find any. Why 30 min? Is something filling up? Is some kind of timer expiring? I can ping anything, except the router haveing the problem. The trace show nothing abnormal. It is sending and recieving APP, VRRP, and ICMP keepalives. This is bizzar. It only happens on one router and HSRP never flinches. Ospf is the only thing bouncing and only on one router. On the trace I can see where I was trying to ping the router and not getting a response back. I compared the two routers, the one having the problem and the working router. The only difference is hardware version on the VIP and Bay card. The configs are exactly the same. I mean this should not be happening. Is there some incorrect setting on my CSS that would cause this, ospf to bounce? IE. If I had a ref. index wrong would this cause something. Why 30 min? Again, thanks a ton for your help.

ehirsel · ‎11-30-2004

I was wondering if you still had the issue? If not, what was the resolution to your problem? If so, can you post a diagram of your topology (i.e., where the css is in relation to the routers, and the layer 2 devices/switches involved?

Thanks.

allen.external · ‎11-30-2004

Well not really. I talked to the Devs. at cisco and they say that code was very buggy. So after going round and round with cisco on what the problem could possibly be I decided to upgrade the code. I have not had a chance to plug them back into the network yet. Well the same network. I pluged them into our 16network and they work fine but no OSPF on that network. So, out next outtage window, I will try again to plug them into the serverfarm network and see what happens. But the Developers from cisco could not figure it out either. But thanks for asking.

allen.external · ‎12-01-2004

Oh almost forgot here is the network layout.

ehirsel · ‎12-01-2004

What type of devices are A1D10-02 and E2D08-02? Layer 2 switches? Layer 3 switches? Or something else?

When you put the css devices in your production network, do you bring both css's online? Do you have your problem if only one css is introduced?

How are the css connections defined to the upstream devices? Trunk ports or access ports?

allen.external · ‎12-01-2004

Layer 2 Enterasys switches. Just layer 2. As far as bringing them online. I have tried Both at the same time and one at a time. Got the same result. They are access-ports. We are not using trunkports to the CSSs. I looked for layer2 loops. did not find any. Also it takes exactly 28 min. for OSPF to die. Took sniffer traces. did not find anything. I disconnect the CSSs and ospf starts acting normal. CSS configs

configure

!*************************** GLOBAL ***************************

bridge spanning-tree disabled

app

app session x.x.x.x

logging buffer 64000

ip route 0.0.0.0 0.0.0.0 x.x.x.x 1

!************************* INTERFACE *************************

interface e1

description "Interface to 24Net"

interface e2

description "DNS Server x.x.x.x"

interface e3

description "DNS Server x.x.x.x"

interface e8

isc-port-one

!************************** CIRCUIT **************************

circuit VLAN3

ip address x.x.x.x x.x.x.x

ip virtual-router 200 priority 200 preempt

ip redundant-vip 200 x.x.x.x

ip critical-service 200 DNS2

ip critical-service 200 DNS1

!************************** SERVICE **************************

service DNS1

ip address x.x.x.x

protocol udp

port 53

keepalive port 53

redundant-index 81

keepalive retryperiod 2

keepalive frequency 2

keepalive maxfailure 2

service DNS2

ip address x.x.x.x

port 53

keepalive retryperiod 2

keepalive frequency 2

keepalive maxfailure 2

keepalive port 53

protocol udp

redundant-index 81

!*************************** OWNER ***************************

owner DNS_LoadBalance

content DNS_Redundancy

vip address x.x.x.x

add service DNS1

add service DNS2

protocol udp

port 53

!*************************** GROUP ***************************

group DNS

vip address x.x.x.x

add service DNS1

add service DNS2