Unable to ping/telnet to management interface of switch

[Charlie Jones]

Good morning,

A couple of weeks ago we upgraded about 50 access layer switches at a branch office.  All of them are WS-3560-24PS.  The switches were upgraded to IOS 12.2(55)SE6. Once the update was completed, all of the switches were reloaded to complete the upgrade.   One of the 50 switches showed up in our monitoring application as being down after the reload.  We had someone at the office plug in a laptop so we could console into it and the switch configuration looked correct.  The switch is working normally (PC's and phones working normally), but we cannot ping or telnet into this one switch.  Below is a breakdown of this site in terms of topology:

Core - 2 6509

Distribution - 2 3750g

Access - 3560 switches

Layer two looks to be running normally in that vtp is being updated and cdp is working as well.   The trunk interfaces from this switch to the distribution layer switch are up (each gig interface on this 3560 goes to one of the 3705g switches). 

On this switch, I have erased the config and deleted the vlan.dat.  I reapplied the config and re-enabled VTP and this switch is still not accessible.  Any suggestions?

I should mention that the management interface is  vlan 1. I have tried giving this management interface a different IP address in case there was a duplicate IP and that does not work.  Other switches that were upgraded and connect up into this 3750g stack work fine.

[Richard Burts]

I would like to go back to my question of yesterday about arp between the problematic switch and the upstream. The output of debug arp shows that the problematic router is generating requests for arp for the gateway address but not receiving any response. The output of debug arp on the upstream router shows that it receives the request from the problematic switch. It does not seem to show that the upstream sends any response. But I am not sure how long that debug was running. So I would ask Charlie to run debug arp on the upstream switch (preferable the switch that is the active router in HSRP). While the debug is running then do the ping from the problematic router. This should generate the arp request to upstream. Let the debug run long enough to be sure whether the switch is sending a response or not.

HTH

Rick

[paul driver]

Charlie

CDP from previous posts:

Switch 1

Device ID            Local Intrfce         Holdtme   Capability    Platform   Port ID

CHI-3560-2602       Gig 1/0/2             162            S I      WS-C3560-2Gig 0/1

Switch 2

Device ID            Local Intrfce         Holdtme   Capability    Platform   Port ID

CHI-3560-2602       Gig 1/0/2             147            S I      WS-C3560-2Gig 0/2

Now from you current posting:

UPLINK SWITCH 1 -

Only one trunk interface is up

GIG1/0/1 - this is not your link to the problem switch as stated in your previous cdp post

Port        Mode         Encapsulation  Status        Native vlan

Gi1/0/1     on           802.1q         trunking      1

Port        Vlans in spanning tree forwarding state and not pruned

Gi1/0/1     1,851,853 Port        Vlans in spanning tree forwarding state and not pruned
Gi1/0/1     1,851,853

UPLINK SWITCH 2 -

Only one trunk  interface shows up

GIG1/0/1 - this is not your link to the problem switch as stated in your previous cdp post

Port        Mode         Encapsulation  Status        Native vlan

Gi1/0/1     on           802.1q         trunking      1

Port        Vlans in spanning tree forwarding state and not pruned
Gi1/0/1     1,26,126,850

Problematic Switch

#sh int trunk

Port        Mode             Encapsulation  Status        Native vlan
Gi0/1       on               802.1q         trunking      1
Gi0/2       on               802.1q         trunking      1

Port        Vlans in spanning tree forwarding state and not pruned

Gi0/1       1,25,27,29,31,125,127,129,131,503,851,853,935,977,985,999,1001

Gi0/2       26,28,30,32,126,128,130,132,830,850,984,990,998

Port        Vlans in spanning tree forwarding state and not pruned -

Gi0/1       1,25,27,29,31,125,127,129,131,503,851,853,935,977,985,999,1001
Gi0/2       26,28,30,32,126,128,130,132,830,850,984,990,998

I can see some anomalies regards trunking and stp:

How are these two Uplink switches connected to the problem switch? I only see 1 trunk on each uplink switch and this does not seem to be going to the problem switch?

Can you post your running config in file attachments for all 3 switches:

Please don't forget to rate this post if it has been helpful.

[Charlie Jones]

Good morning,

It's been awhile since I've had a chance to work on this, and today I have an update of sorts.  We decided to swap out this problematic switch in the hope it was a problem with the switch.  Unfortuntely, this did not correct the problem.  The problem must lie somewhere else within this office.

Here are the configs for the trunks and the vlan interface on the problematic switch:

interface GigabitEthernet0/1

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

speed 1000

duplex full

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape 10 0 0 0

queue-set 2

priority-queue out

mls qos trust cos

auto qos trust

rmon collection history 10101 owner campusmanager buckets 10 interval 300

!

interface GigabitEthernet0/2

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

speed 1000

duplex full

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape 10 0 0 0

queue-set 2

priority-queue out

mls qos trust cos

auto qos trust

rmon collection history 10102 owner campusmanager buckets 10 interval 300

!

interface Vlan1

ip address 10.19.0.112 255.255.255.0

!

ip default-gateway 10.19.0.1

Here is the config of switch 1 uplink that this switch trunks to

interface GigabitEthernet1/0/2

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

speed 1000

duplex full

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape  10  0  0  0

queue-set 2

mls qos trust dscp

auto qos voip trust

rmon collection history 10102 owner campusmanager buckets 10 interval 300

Here is the config for the switch 2 trunk:

interface GigabitEthernet1/0/2

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

speed 1000

duplex full

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape  10  0  0  0

queue-set 2

mls qos trust dscp

auto qos voip trust

To give you an idea of the network topology, it follows Cisco's practice of a core/distribution/access model.

Core - Pair of 6509

distribution - Pair of 3750G

access- 3550 switches

At this access layer, there are other switches that are trunked up to the 3750G's, and those switches are working normally.  Also, there are PC's/phones plugged into this switch, and those are working normally.  In fact, I tested pinging one of the hosts on this switch, and I can ping the host from somewhere else on the network.  However from the problematic switch, I cannot ping this host.

[Charlie Jones]

Problem solved.  I ended up looking at a configuration from an existing switch at this branch, and the switch having the problem had the command "vlan dot1q tag native".  As soon as I did a no vlan dot1q tag native, the problem was gone.

[stephen.stack · Accepted solution]

Charlie

thats great news, but unfortunate in a way that it was not found eariler

Not my text but the; “vlan dot1q tag native” which will prevent the double-encapsulation  attacks.  This command globally works on all switchport trunks on that  entire Ethernet switch.  This command will make sure that the native  VLAN is always tagged on every trunk on the switch.  This is a great  best practice and takes care of the issue with a single command.  This  command should be entered in every switch in the campus." (my bold)

This caused every untagged ingress frame is dropped, so the traffic replies were not reaching the control plane of the switch.

==========================
http://www.rConfig.com 

A free, open source network device configuration management tool, customizable to your needs!

- Always vote on an answer if you found it helpful

[Charlie Jones]

If this is configured globally, then what would have been the fix to keep this command active?  Would we need to add something to the trunk ports on this switch?  Now, I'm curious.

[stephen.stack]

No, i've checked it out, and i've not seen a way to disable on a per-port basis. It's all-for-one, or none-at-all

Really gald you located the issue.

==========================
http://www.rConfig.com 

A free, open source network device configuration management tool, customizable to your needs!

- Always vote on an answer if you found it helpful