Unable to ping/telnet to management interface of switch

[Charlie Jones]

Good morning,

A couple of weeks ago we upgraded about 50 access layer switches at a branch office.  All of them are WS-3560-24PS.  The switches were upgraded to IOS 12.2(55)SE6. Once the update was completed, all of the switches were reloaded to complete the upgrade.   One of the 50 switches showed up in our monitoring application as being down after the reload.  We had someone at the office plug in a laptop so we could console into it and the switch configuration looked correct.  The switch is working normally (PC's and phones working normally), but we cannot ping or telnet into this one switch.  Below is a breakdown of this site in terms of topology:

Core - 2 6509

Distribution - 2 3750g

Access - 3560 switches

Layer two looks to be running normally in that vtp is being updated and cdp is working as well.   The trunk interfaces from this switch to the distribution layer switch are up (each gig interface on this 3560 goes to one of the 3705g switches). 

On this switch, I have erased the config and deleted the vlan.dat.  I reapplied the config and re-enabled VTP and this switch is still not accessible.  Any suggestions?

I should mention that the management interface is  vlan 1. I have tried giving this management interface a different IP address in case there was a duplicate IP and that does not work.  Other switches that were upgraded and connect up into this 3750g stack work fine.

[stephen.stack]

Hi

Can the switch ping out to the network? is the ip default-gateway set?

Regards

Stephen

==========================
http://www.rConfig.com 

A free, open source network device configuration management tool, customizable to your needs!

- Always vote on an answer if you found it helpful

[Charlie Jones]

HI,

The switch cannot ping out.   The default gateway is set as well.

[stephen.stack]

can you provide partial relevant configuration? Also, try a debug ip icmp when pinging it to see if the icmp packets are reaching the switch

Regards

==========================
http://www.rConfig.com 

A free, open source network device configuration management tool, customizable to your needs!

- Always vote on an answer if you found it helpful

[Charlie Jones]

I enabled ICMP debugging.  I pinged the switch from another switch in this IDF and nothing shows in the log of the failed switch.

Interface Vlan1

ip address 10.19.0.112 255.255.255.0

ip default-gateway 10.19.0.1

ip classless

[glen.grant]

  Make sure ip routing is not turned on the 3560,  otherwise the default gateway statement  is no good.  If routing is turned on then you would need a default static route  pointing to the 3750 gateway address for that vlan and get rid of the default gateway statement.   Is this switch trunked or just an access port ?  Check trunk setup , native vlan for trunk etc...    You should be able to get to the switch from 3750 even if the gateway is no good because it is directly attached to the 3750 which is where i assume the vlan 1 subnet originates.  Source ping the switch from the vlan 1 SVI .

[Richard Burts]

Perhaps the output of show ip interface brief or of show interface status might shed some light on the issue?

I also wonder what would be in the output of show arp from the problem switch.

HTH

Rick

[Charlie Jones]

Here is the information you requested.

Interface              IP-Address      OK? Method Status                Protocol

Vlan1                  10.19.0.112     YES manual up                    up

FastEthernet0/1        unassigned      YES unset  down                  down

FastEthernet0/2        unassigned      YES unset  up                    up

FastEthernet0/3        unassigned      YES unset  up                    up

FastEthernet0/4        unassigned      YES unset  down                  down

FastEthernet0/5        unassigned      YES unset  down                  down

FastEthernet0/6        unassigned      YES unset  up                    up

FastEthernet0/7        unassigned      YES unset  up                    up

FastEthernet0/8        unassigned      YES unset  up                    up

FastEthernet0/9        unassigned      YES unset  up                    up

FastEthernet0/10       unassigned      YES unset  up                    up

FastEthernet0/11       unassigned      YES unset  up                    up

FastEthernet0/12       unassigned      YES unset  up                    up

FastEthernet0/13       unassigned      YES unset  up                    up

FastEthernet0/14       unassigned      YES unset  down                  down

FastEthernet0/15       unassigned      YES unset  up                    up

FastEthernet0/16       unassigned      YES unset  down                  down

FastEthernet0/17       unassigned      YES unset  up                    up

FastEthernet0/18       unassigned      YES unset  up                    up

FastEthernet0/19       unassigned      YES unset  down                  down

FastEthernet0/20       unassigned      YES unset  up                    up

FastEthernet0/21       unassigned      YES unset  up                    up

FastEthernet0/22       unassigned      YES unset  up                    up

FastEthernet0/23       unassigned      YES unset  down                  down

FastEthernet0/24       unassigned      YES unset  up                    up

GigabitEthernet0/1     unassigned      YES unset  up                    up

GigabitEthernet0/2     unassigned      YES unset  up                    up

sh arp

Protocol  Address          Age (min)  Hardware Addr   Type   Interface

Internet  10.19.0.1               0   Incomplete      ARPA

Internet  10.19.0.112             -   001b.0c7c.1340  ARPA   Vlan1

[Richard Burts]

Thanks for the information. It is somewhat helpful, but not enough to diagnose the full problem. It does confirm the IP address of the VLAN interface seems to be correct.

The fact that the arp table shows incomplete for the mac address of the gateway 10.19.0.1 indicates that arp is failing to resolve the IP address to a mac address. This typically indicates some layer 2 problem. Can you post the output of show interface status, and indentify for us the ports on the switch that connect to the upstream switch?

HTH

Rick

[stephen.stack]

You may also want to check what vlans are allowed an the upstream trunk links with a show int trunk. Do this on both sides of the trunks from the affected switch and ensure vlan1 is allowed

Regards

Sent from Cisco Technical Support iPhone App

[Charlie Jones]

I think this will hit the last two questions. I compared this output to another switch that we upgraded, and the information matches the functional switch.

Trunks on problematic switch

Gi0/1                        connected    trunk        full   1000 10/100/1000BaseTX SFP
Gi0/2                        connected    trunk        full   1000 10/100/1000BaseTX SFP

Trunk Uplink Switch 1
Gi1/0/1                      connected    trunk        full   1000 10/100/1000BaseTX

Trunk Uplink Switch 2
Gi1/0/2                      connected    trunk        full   1000 10/100/1000BaseTX

Show int trunk from problematic switch

Port        Mode             Encapsulation  Status        Native vlan
Gi0/1       on               802.1q         trunking      1
Gi0/2       on               802.1q         trunking      1

Port        Vlans allowed on trunk
Gi0/1       1-4094
Gi0/2       1-4094

Port        Vlans allowed and active in management domain
Gi0/1       1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi0/2       1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001

Port        Vlans in spanning tree forwarding state and not pruned
Gi0/1       1,25,27,29,31,125,127,129,131,503,851,853,935,977,985,999,1001
Gi0/2       26,28,30,32,126,128,130,132,830,850,984,990,998

Show int trunk from Switch 1

Port        Mode         Encapsulation  Status        Native vlan
Gi1/0/1     on           802.1q         trunking      1
Gi1/0/2     on           802.1q         trunking      1
Gi1/0/3     on           802.1q         trunking      1
Gi1/0/4     on           802.1q         trunking      1
Gi1/0/5     on           802.1q         trunking      1
Gi1/0/6     on           802.1q         trunking      1
Gi1/0/7     on           802.1q         trunking      1
Gi1/0/25    on           802.1q         trunking      1

Port        Vlans allowed on trunk
Gi1/0/1     1-4094
Gi1/0/2     1-4094
Gi1/0/3     1-4094
Gi1/0/4     1-4094
Gi1/0/5     1-4094
Gi1/0/6     1-4094
Gi1/0/7     1-4094
Gi1/0/25    1-4094

Port        Vlans allowed and active in management domain
Gi1/0/1     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/2     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/3     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/4     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/5     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/6     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/7     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/25    1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001

Port        Vlans in spanning tree forwarding state and not pruned
Gi1/0/1     1,851,853
Gi1/0/2     1
Gi1/0/3     1,851,853
Gi1/0/4     1,851,853
Gi1/0/5     1,851,853
Gi1/0/6     1,851,853
Gi1/0/7     1,851,853
Gi1/0/25    1,25-32,125-132,503,830,850-851,853,935,977,984-985,990,998-999,1001

Show int trunk from Switch 2

Port        Mode         Encapsulation  Status        Native vlan
Gi1/0/1     on           802.1q         trunking      1
Gi1/0/2     on           802.1q         trunking      1
Gi1/0/3     on           802.1q         trunking      1
Gi1/0/4     on           802.1q         trunking      1
Gi1/0/5     on           802.1q         trunking      1
Gi1/0/6     on           802.1q         trunking      1
Gi1/0/7     on           802.1q         trunking      1
Gi1/0/25    on           802.1q         trunking      1

Port        Vlans allowed on trunk
Gi1/0/1     1-4094
Gi1/0/2     1-4094
Gi1/0/3     1-4094
Gi1/0/4     1-4094
Gi1/0/5     1-4094
Gi1/0/6     1-4094
Gi1/0/7     1-4094
Gi1/0/25    1-4094

Port        Vlans allowed and active in management domain
Gi1/0/1     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/2     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/3     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/4     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/5     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/6     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/7     1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi1/0/25    1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001

Port        Vlans in spanning tree forwarding state and not pruned
Gi1/0/1     1,26,126,850
Gi1/0/2     1,26,126
Gi1/0/3     1,26,126,850
Gi1/0/4     1,26,126,850
Gi1/0/5     1,26,126,850
Gi1/0/6     1,26,126,850
Gi1/0/7     1,26,126,850
Gi1/0/25    1,25-32,125-132,503,830,850-851,853,935,977,984-985,990,998-999,1001

[Richard Burts]

Thanks for the additional information. So far it all looks reasonable (and looks like it should be working).

Perhaps you could post the output of show cdp neighbor from the problem switch and also from both of the upstream switches.

Where is the address 10.19.0.1 located? (on which device)

HTH

Rick

[stephen.stack]

All looks ok

Not sure if this was done or can be done. But try to remove vlan 1 completely. Else default it's config shut it down and start configuring it from scratch - bring it up only when configured. Have had similar issues with svi's in the past

Regards

Sent from Cisco Technical Support iPhone App

[Charlie Jones]

Below is the cdp information you requested.   10.19.0.1 resides on the core switches via hrsp.

Problematic Switch

CHI-3560-2602#sh cdp neighbors gi0/1
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
CHI-3750G-2601   Gig 0/1           126              S I   WS-C3750G Gig 1/0/2
CHI-3560-2602#sh cdp neighbors gi0/2
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
CHI-3750G-2602   Gig 0/2           131              S I   WS-C3750G Gig 1/0/2

Trunk Switch 1

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID            Local Intrfce         Holdtme   Capability    Platform   Port ID
CHI-3560-2601       Gig 1/0/1             173            S I      WS-C3560-2Gig 0/1

Trunk Switch 2

CHI-3750G-2602#sh cdp neighbors gi1/0/2
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID            Local Intrfce         Holdtme   Capability    Platform   Port ID
CHI-3560-2602       Gig 1/0/2             165            S I      WS-C3560-2Gig 0/2

[Richard Burts]

Perhaps we have a clue about the problem. I see that from the problem switch we believe that upstream switch 1 is using interface gi1/0/2

CHI-3750G-2601   Gig 0/1           126              S I   WS-C3750G Gig 1/0/2

but on the upstream switch1 it think is it using interface Gig1/0/1

CHI-3560-2601       Gig 1/0/1             173            S I      WS-C3560-2Gig 0/1

I am not sure what causes this mismatch. But I suspect that this is related to the problem.

HTH

Rick