Upgrading between a 3750 stack and a 3850 stack

[Andrew Reid]

Hi, I wonder could someone shed some light on this issue we are having here. We are trying to do a migration between a 3750 stack and a 3850 stack over a week. Each stack in is a new cabinet so we will move a few patch panels over each night and re patch the clients into the new 3850 stack.

So what we have to start with is a 3750 stack with 2 uplinks to the core (6500's). OSPF running and working perfectly. So we now connect a trunk between the 3850 and the 3750 stack. The only ip address on the new stack is one for management for now, however the same vlan interfaces have been configured as are on the 3750 ready for the final move over , but all of these interfaces are shut down.

So we can connect clients into the 3850, they then trunk over to the svi on the 3750 and then on to the bigger network. So to bring the migrations a little closer to a full change over we move one of the L3 links over to the 3850.

I would have expected that link to pretty much not be getting used at this point and all of the traffic to still go over the trunk to the 3750 SVI and out of that uplink.

This is not what is happening, once the second uplink is connected into the 3850 all traffic from clients cnnected on the 3850 are no longer able to connect to the network. From the 3850 switch itself I can ping out of the uplink to the full network and I assume for some reason because OSPF is now active on the 3850 something strange is happening to stop the vlan traffic going to the svi's over on the 3750.

Once the L3 uplink is moved over to the 3850 the port it was on when connected to the 3750 is shut, so no issues with the ip in 2 places. Also the interfaces set to no passive on both of the uplinks, all other interfaces passive by default ( for OSPF)

I've tried having a search about but cannot see any info on this. Does anyone have a reason why this is happening ?

Thanks

[chrihussey]

Very strange.

Are the L3 uplinks VLAN interfaces, if so what are they, or are the IPs configured directly on the physical switch ports?

Are the two stacks also OSPF peering with each other? If so how, across which VLAN?

[Andrew Reid]

The L3 ospf links on the stacks to the core are using ip addresses with 30 bit masks. No vlans or anything on the uplink ports. They are purely L3.

There is no L3 peering between the stacks, they are purely L2 trunks.

It just seems somehow that the traffic seems to favor trying to use the L3 uplink on the 3850 instead of going via the svi's on the 3750. Not sure whats up.

Andy

[Jon Marshall]

Andy

So vlan 100 is passive on both stacks ?

What does a traceroute from a client attached to the 3850 to a remote IP show ie. trying to work out which direction traffic is failing.

Jon

[Andrew Reid]

Yes the same vlans are configured on each switch, only one has an SVI address.

When the L3 link was moved I was able to get out of the 3850 via the moved link. But the clients were for some reason no longer trying to get to the SVi on the other switch once OSPF was active on the 3850.

[Andrew Reid]

I had a Laptop connected to the 3850 when I did the moving of the L3 link with continuous pings to several locations on the network and all pings failed. The both L3 links are back in the 3750 for now.

[Jon Marshall]

When you say the clients were no longer trying to get to the SVIs on the 3750 how do you know ie. you were running pings but were you also doing traceroutes ?

Reason I am asking is I am trying to work out where the traffic is getting dropped and it may be that the ping is going via the 3750 but the problem is somewhere else.

Jon

[Andrew Reid]

Hi, thanks so far for the input.

So for the OSPF everything is setup ok, in fact the L3 uplink port is configured the same way that ii was on the 3750, and the OSPF itself is configured the same.

If there is really only one way out from the vlans via the 3750 svi's could there be confusion how to get back to the switch.

As I see if the traffic on the vlan on the 3850 has to go via the svi on the 3750 to get to the outside world and the L3 link on the 3850 really is not doing anything at this point. So what is stopping the L2 traffic going across the trunk to the 3750 svi.

[chrihussey]

So to clarify, when you dropped the one L3 link to the 3750 nothing changed and you still had connectivity from the 3850 stack. It wasn't until you brought up the L3 link to the 3850 and established OSPF peers that the problem occurred. Correct?

What if you bring up the L3 link to the 3850 stack without OSPF?

Also, can you elaborate what you mean by:

"But before any L3 uplinks were moved to it I had to put an ip route to the network via the SVI to be able to access it remotely. There is def something strange when you mix L3 and L2 and the SVI is on a different SW"

Where was this static route configured?

Thanks.

[Jon Marshall]

Until you bring up an client vlan SVIs on the 3850 it doesn't matter what routes the 3850 knows about, client traffic still has to go via the 3750.

So the main difference I can see is that with your current setup both core switches get routes for the client vlans direct from the 3750. When you migrate one of the links to the 3850 the core switch that this link previously connected to is now reliant on receiving those routes from the other core switch across the L3 interconnect.

I am assuming it does and you are not doing any route filtering but we would need more details.

Jon

[Andrew Reid]

Actually another example I feel of this issue is. In another location I have a trunk connected to a switch that has L3 to the core. I simply put an ip address on it that's in the same subnet of one of the vlans. Everything is great and I can get to it from anywhere on the network.

But on this example where I am having the problem. I did the same thing only OSPF was configured.

But before any L3 uplinks were moved to it I had to put an ip route to the network via the SVI to be able to access it remotely. There is def something strange when you mix L3 and L2 and the SVI is on a different SW.

[chrihussey]

Yeah, I would not have expected the results you are getting. Unfortunately you are in a production environment so identifying exactly what is happening will be difficult.  The configs may help.

Just tossing out these thoughts as they may help if you try it again:

1- Set the load-interval on the L3 uplinks to 30 to get a better idea of the traffic utilization and flow.

2- Pay close attention to the TX and RX traffic. Is it possible the traffic is leaving the stacks fine and it is the return traffic that is failing?

3- Does the OSPF router ID in both 6500s update correctly?

4- Verify the routing in the 6500 cores for the destination networks.

[Joseph W. Doherty]

For assistance, you may need to post additional information.