Management VLAN Migration

[Derick-Angel]

Hello all,

I'm having an issue with some of my access switches and am hoping that you may be able to help. My network consists of 7 separate buildings across the compound, let's call them Buildings 1-7. Building 1 contains my core (C4510R), DMZ (C3850), some server switches (2x C3850/ 1x 4500X), and 10 additional access switches (3x C3850 / 7x C2960C/CX). Building 2 has 1 stack of 4x C3850 and 9 access C2960C/CX.  Building 3 has 1 stack of 4x C3850, 1x server C3850, and 17 access switches (6x C2960C, 4x 2960CX, 7x C3560CX).  

When I came on board last year, the previous network engineer has laid out plans for an IP migration and started it in some of the buildings. Each building had their own /24 network and a /28 management subnet allocated for each building. This was to provide uniformity and say that the management addresses for each building will be in the same ranges with the same VLAN ID (VLAN 145) (192.168.0.177, 192.168.1.177, 192.168.3.177, etc).  For 5 of the buildings on my compound, that is sufficient but buildings 1 and 3 have more than 14 switches to manage. I cannot expand the management subnet in those buildings due to the IP addresses before and after being in use for other subnets (printers, servers, etc.). Our solution is to flatten the network for management and create a shared /25 subnet for all of the devices. Let's call it VLAN 148 (192.168.20.0-127).

This is all to set the stage for two problems that I am having. 

Problem #1: On VLAN 145, from my management workstation, I can SSH to the stack in building 2 and 5 of the access switches, but cannot SSH or ping the other 5. However, I am able to SSH to the stack and can ping and SSH the other 5 switches from there. I have checked the configs on the access switches and confirmed that they have the same VTY commands, IP SSH V2 enabled, and IP Routing. I am not sure what else the issue could be. I have the same issue with several other buildings across the compound. The problem does not seem to be isolated to a specific model of switch or IOS version as there is a combination of C2960CX and C3856CX running 15.2(4)E2 and newer that I cannot reach but I can reach another switch in the same building with the same model and IOS version. 

Problem #2: I am trying to roll-out the larger VLAN 148. I added the IP addresses to the majority of the switches across the compound. There has been no change in which switches I am able to SSH to and those which I cannot. This lead me to believe that the VLAN was working fine. I started to pull back the VLAN 145 addresses and problem #2 started. The trunk ports between the access switches and stacks are trunking VLANs 145,148, and any other VLANS necessary for that room. As soon as I remove 145 from the trunk port but keep everything else the same, I can no longer ping or SSH the IP address for VLAN 148. I do not know why one would cause any effect on the other. 

Unfortunately, I cannot paste a copy of my configs due to the network classification levels.

I'm hoping that someone may have come across this before and might have an idea of what I am missing. 

[balaji.bandi]

Fist i would check VLAN created - and where is STP bridge for the VLANs

Second, if the VLAN stretched all over, where is the Layer 3 interface are you able to ping Locally or next hop ? ( that is reachability)

Third how is your routing in Place ? Static or any IGP (like OSPF or eigrp?)

Forth - check any ACL created and added for security reasons.

You need to provide some example config for us to understand wat is configured. like a dummy config with IP address in it to understand better.

[Derick-Angel]

I am not sure where the STP bridge is for the VLANs. I believe that it should be the core for VLAN 148 and the local stack for 145. spanning-tree mode pvst and spanning-tree extend system-id is enabled on both switches. 

All routing for VLAN 148 is done by the Core in Building 1. VLAN 145 is still processed by the stack in each building. When VLAN 145 and 148 are trunking to the access switch, I can ping from my workstation in building 1 to the core, the stack for Building 2 (next hop), and to the access switch on the VLAN 148 address (192.168.10.35). When I remove VLAN 145 from the trunk, I can ping to the core and to the stack/next hop but cannot ping the access switch. I can ping the access switch from inside the core and stack but if I try to SSH, I get an error stating "CBC Ciphers got moved out of default config. Please configure ciphers as required to (to match peer ciphers)." I'm confused as to why I get that error message now but not when the other VLAN is included. 

Routing is static and handled by the core. We have a device for data encryption between each building with a static route between building 1 and the respective building. VLAN 145 is unique to each building so all routing for that VLAN is handled by the stack for the respective building. The encrypting device does have a separate VLAN 148 built into it to flatten the network and spread the entire VLAN across the compound. 

There are no ACLs on the switches in question that would separate one VLAN from the other.

I am working on getting a sanitized version of the config uploaded. I have to wait for the powers that be to approve the cross-network transfer but will hopefully have it ready to upload tomorrow morning. 

[Reza Sharifi]

Hi,

In general, you do not need to breakdown the management vlan per building. All you need is a /24 or a /25, simply assign an IP address with an SVI to each layer-2 switch and make sure each switch has a default-gateway pointing to the core or whatever device is doing the routing for the management vlan. Also, the management vlan needs to be added to each trunk port on all access switches. So, I would check to make sure all access switches do actually have a default-gateway configured.

HTH

[Derick-Angel]

@Reza Sharifi That is what I am attempting to correct. The previous network engineer set up the network so that each building has it's own /24 and the management VLAN (145) is part of that. I am trying to implement a single /24 (VLAN 148) across the entire network for all of the devices to share. The new VLAN (192.168.10.0 /25) is on all of the access switches, as well as the default-gateway of 192.162.10.1. The trunk port is currently trunking the traffic VLAN, 145, and 148 and I can access the switch on either IP address. When I remove VLAN 145, I can no longer access the device on VLAN 148. I cannot figure out why removing 145 has any effect on 148. 

I also cannot figure out why I can acces one switch directly from my workstation but cannot access another on the same VLAN but I am able to access it via SSH from inside another switch on the network. 

[Derick-Angel]

@balaji.bandi I was finally able to get a copy of my sanitized configs uploaded. Please let me know if there is something that you can find that I've overlooked. Both of these switched are in building 3. One I can SSH/ping to directly and the other I can only SSH/ping from inside another switch but I cannot identify what is causing the issue. They are both using VLANs 145 and 148.

[balaji.bandi]

as i see the config 

success one do not have any default gateway ?

the failed one has

ip default-gateway XXX.XXX.15.130   ( your IP config on vlan 145 and 148 does not match 3rd octet of IP address that is 15. so check and correct it.)

next I would like to see 

show IP route (working and not working)

[Derick-Angel]

Correct, there is no default gateway set on the successful switch. I added a default gateway to the fail switch while troubleshooting to see if that would correct the issue but it was unsuccessful. 

The third octet is different between the VLANs due to the different subnets. VLAN 145 is part of a larger /24 assigned to each building (192.168.1.0/24). VLAN 145s entire scope is 192.168.1.176/28. The third octet changes with each building (ex. 1.0 for building 1, 2.0 for building 2, 3.0 for building 3, etc.) VLAN 148 is a /25 that will be distributed across the entire network (192.168.10.0/25).  The default gateway listed is for the user VLAN. I tried changing it to 192.168.10.1, which is the core address for VLAN 148 but it did not make a difference. 

show IP route for success switch shows:

Default gateway is not set

Host    Gateway          Last Use          Total Uses       Interface

ICMP redirect cache is empty

show IP route for fail switch shows:

Gateway of last resort is not set
192.168.0.0/16 is variably subnetted, 2 subnets, 2 masks
C    192.168.10.0/25 is directly connected, Vlan148
L    192.168.10.38/32 is directly connected, Vlan 148
192.168.24.0/24 is variably subnetted, 2 subnets, 2 masks
C    192.168.24.0/24 is directly connected, Vlan 92
L    192.168.24.130/32 is directly connected, Vlan 92