01-22-2013 07:09 AM - edited 03-07-2019 11:14 AM
Good morning,
A couple of weeks ago we upgraded about 50 access layer switches at a branch office. All of them are WS-3560-24PS. The switches were upgraded to IOS 12.2(55)SE6. Once the update was completed, all of the switches were reloaded to complete the upgrade. One of the 50 switches showed up in our monitoring application as being down after the reload. We had someone at the office plug in a laptop so we could console into it and the switch configuration looked correct. The switch is working normally (PC's and phones working normally), but we cannot ping or telnet into this one switch. Below is a breakdown of this site in terms of topology:
Core - 2 6509
Distribution - 2 3750g
Access - 3560 switches
Layer two looks to be running normally in that vtp is being updated and cdp is working as well. The trunk interfaces from this switch to the distribution layer switch are up (each gig interface on this 3560 goes to one of the 3705g switches).
On this switch, I have erased the config and deleted the vlan.dat. I reapplied the config and re-enabled VTP and this switch is still not accessible. Any suggestions?
I should mention that the management interface is vlan 1. I have tried giving this management interface a different IP address in case there was a duplicate IP and that does not work. Other switches that were upgraded and connect up into this 3750g stack work fine.
Solved! Go to Solution.
01-22-2013 10:02 AM
Sorry, that was my fault. Here is the correct informaiton on the uplink switches
Switch 1
Device ID Local Intrfce Holdtme Capability Platform Port ID
CHI-3560-2602 Gig 1/0/2 162 S I WS-C3560-2Gig 0/1
Switch 2
Device ID Local Intrfce Holdtme Capability Platform Port ID
CHI-3560-2602 Gig 1/0/2 147 S I WS-C3560-2Gig 0/2
01-22-2013 10:05 AM
Or did you show the incorrect cdp neighbor on the upstream switch 1? Since the neighbor it is reporting seems to not be the problem switch?
Trunk Switch 1
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone
Device ID Local Intrfce Holdtme Capability Platform Port ID
CHI-3560-2601 Gig 1/0/1 173 S I WS-C3560-2Gig 0/1
l am now noticing that it is 3560-2601 and not 3560-2602
HTH
Rick
01-22-2013 10:11 AM
OK. Let us take a slightly different approach. We know that part of the difficulty is that the problematic switch is not able to arp for the 10.19.0.1 address (it shows incomplete in the arp table). You have told us that this address is HSRP on the upstream switches. So can the problematic switch arp (and ping) to the physical interface address (not the shared address) on either or both of the upstream switches\?
Also I would be curious to know if the upstream switches can arp for the management address of the problematic switch?
HTH
Rick
01-22-2013 11:39 AM
I went to the upstream switches and attempted to ping the management IP of the problematic switch, and that times out. When I checked the arp on the upstream switches, I see the following:
Internet 10.19.0.112 0 001b.0c7c.1340 ARPA Vlan1
Is that what you were looking to see?
01-22-2013 12:43 PM
Yes this is what I was looking for. It demonstrates that there is successful communication between the upstream switch and the problematic switch. So the questions in this thread about where interfaces configured correctly, were the trunks set up correctly, were the right VLANs allowed on the trunk, etc are all answered now. We have successful communication.
The next test I would like to do is to see if the problematic switch can arp (and ping) to the interface address of the upstream switch.
HTH
Rick
01-22-2013 01:34 PM
Im unable to ping the upstream switches. Below is the arp table from the problematic switch
Internet 10.19.0.109 0 Incomplete ARPA
Internet 10.19.0.110 0 Incomplete ARPA
01-22-2013 02:44 PM
I am a bit surprised at this. But I believe that it is quite helpful. Something is preventing the switch from arp to what should be locally connected addresses. Would you turn on debug arp, try the ping, and post the output of the debug?
It might also be helpful to turn on debug arp on the upstream switch, try ping from the problematic switch, and post output to see if the arp gets to the upstream switch.
It would seem that either the problematic switch is not sending the arp request or that the upstream is not sending the reply. These tests should show which it is.
HTH
Rick
01-23-2013 06:37 AM
Here is the arp debug from the problematic switch to the DG.
CHI-3560-2602#ping 10.19.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.19.0.1, timeout is 2 seconds:
.Jan 23 08:31:01.667 CST: IP ARP: creating incomplete entry for IP address: 10.19.0.1 interface Vlan1
.Jan 23 08:31:01.667 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.19.0.1 0000.0000.0000 Vlan1
.Jan 23 08:31:01.667 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.50.1.17 0000.0000.0000 Vlan1
.Jan 23 08:31:02.673 CST: IP ARP throttled out the ARP Request for 10.19.0.1.
.Jan 23 08:31:03.672 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.50.1.17 0000.0000.0000 Vlan1
.Jan 23 08:31:03.680 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.19.0.1 0000.0000.0000 Vlan1
.Jan 23 08:31:04.687 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:04.687 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:05.693 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.19.0.1 0000.0000.0000 Vlan1.
.Jan 23 08:31:06.700 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:07.673 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.50.1.17 0000.0000.0000 Vlan1
.Jan 23 08:31:07.707 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.19.0.1 0000.0000.0000 Vlan1
.Jan 23 08:31:07.707 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:08.713 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:09.720 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.19.0.1 0000.0000.0000 Vlan1.
.Jan 23 08:31:10.727 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:10.727 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:11.733 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.19.0.1 0000.0000.0000 Vlan1.
.Jan 23 08:31:12.740 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:13.746 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.19.0.1 0000.0000.0000 Vlan1
.Jan 23 08:31:13.746 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:14.753 CST: IP ARP throttled out the ARP Request for 10.19.0.1
.Jan 23 08:31:15.676 CST: IP ARP: creating incomplete entry for IP address: 10.50.1.17 interface Vlan1
.Jan 23 08:31:15.676 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.50.1.17 0000.0000.0000 Vlan1
.Jan 23 08:31:15.760 CST: IP ARP: sent req src 10.19.0.112 001b.0c7c.1340,
dst 10.19.0.1 0000.0000.0000 Vlan1.
Success rate is 0 percent (0/5)
When I tried to ping the problematic switch from an upstream switch, I did not get any debug messages.
When I pinged the DG from the problematic switch and had debugging enabled on an upstream switch, I was seeing some log entries in the upstream switch
Jan 23 08:36:26.109 CST: IP ARP: rcvd req src 10.19.0.112 001b.0c7c.1340, dst 10.50.1.17 Vlan1
Jan 23 08:36:26.193 CST: IP ARP: rcvd req src 10.19.0.112 001b.0c7c.1340, dst 10.19.0.1 Vlan1
01-23-2013 06:54 AM
Charlie,
I know you've rebuilt the config on the switch. Take my earlier suggestion and remove the SVI (interface vlan 1) and re-add it if you can. as it's not working, i wont do any harm right now.
Regards
==========================
http://www.rConfig.com
A free, open source network device configuration management tool, customizable to your needs!
- Always vote on an answer if you found it helpful
01-23-2013 07:02 AM
I can't remove it completely since it is vlan 1. I removed the IP address, shutdown the port, added the IP address, and enabled the port again and it is not working. I also tried giving the vlan interface a different IP, and i get the same result.
01-23-2013 07:09 AM
Odd, i can remove it on some lab kit i have
LABSW#conf t
Enter configuration commands, one per line. End with CNTL/Z.
LABSW(config)#int vlan 1
LABSW(config-if)#ip add 1.1.1.1 255.255.255.0
LABSW(config-if)#no shut
LABSW#sh ip int bri
Interface IP-Address OK? Method Status Protocol
Vlan1 1.1.1.1 YES manual up up
LABSW#conf t
Enter configuration commands, one per line. End with CNTL/Z.
LABSW(config)#no int vlan 1
LABSW#sh run int vlan 1
^
% Invalid input detected at '^' marker.
LABSW#sh ip int bri
Interface IP-Address OK? Method Status Protocol
==========================
http://www.rConfig.com
A free, open source network device configuration management tool, customizable to your needs!
- Always vote on an answer if you found it helpful
01-23-2013 07:06 AM
Charile,
From this problem switch upto the cores - where in that path are pings succesfull - what switch is able to ping the cores?
res
Paul
01-23-2013 09:15 AM
From the problematic switch, I can only ping its VLAN interface, I can't ping anything upstream from it. From the upstream switches, I can ping the default gateway but not the IP of the problematic switch.
01-23-2013 09:26 AM
Okay from the good switch directly attached to the problem switch, and the problem switch can you post:
Running config
sh int trunk
sh vtp status
sh ip int brief
sh vlan bri
sh cdp neighbour
res
Paul
Please don't forget to rate this post if it has been helpful.
01-23-2013 09:36 AM
Here you go. I cleaned it up so it wasn't as long.
UPLINK SWITCH 1
Port Mode Encapsulation Status Native vlan
Gi1/0/1 on 802.1q trunking 1
Port Vlans allowed on trunk
Gi1/0/1 1-4094
Port Vlans allowed and active in management domain
Gi1/0/1 1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Port Vlans in spanning tree forwarding state and not pruned
Gi1/0/1 1,851,853
CHI-3750G-2601#sh vtp status
VTP Version : 2
Configuration Revision : 235
Maximum VLANs supported locally : 1005
Number of existing VLANs : 36
VTP Operating Mode : Client
VTP Domain Name : CHI
VTP Pruning Mode : Enabled
VTP V2 Mode : Enabled
VTP Traps Generation : Disabled
MD5 digest : 0xE5 0x68 0x65 0x50 0x48 0x2F 0x23 0x4A
Configuration last modified by 10.19.0.254 at 4-12-12 13:49:06
CHI-3750G-2601#sh ip int brief
Interface IP-Address OK? Method Status Protocol
Vlan1 10.19.0.109 YES NVRAM up up
GigabitEthernet1/0/1 unassigned YES unset up up
#sh vlan brief
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Gi1/0/8, Gi1/0/9, Gi1/0/10, Gi1/0/11, Gi1/0/12, Gi1/0/17, Gi1/0/18, Gi1/0/19
Gi1/0/20, Gi1/0/21, Gi1/0/22, Gi1/0/23, Gi1/0/24, Gi1/0/26, Gi1/0/27, Gi1/0/28
1001 NO-ACCESS active
1002 fddi-default act/unsup
1003 trcrf-default act/unsup
1004 fddinet-default act/unsup
1005 trbrf-default act/unsup
UPLINK SWITCH 2
#sh int trunk
Port Mode Encapsulation Status Native vlan
Gi1/0/1 on 802.1q trunking 1
Port Vlans allowed on trunk
Gi1/0/1 1-4094
Port Vlans allowed and active in management domain
Gi1/0/1 1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Port Vlans in spanning tree forwarding state and not pruned
Gi1/0/1 1,26,126,850
#sh vtp status
VTP Version : 2
Configuration Revision : 235
Maximum VLANs supported locally : 1005
Number of existing VLANs : 36
VTP Operating Mode : Client
VTP Domain Name : CHI
VTP Pruning Mode : Enabled
VTP V2 Mode : Enabled
VTP Traps Generation : Disabled
MD5 digest : 0xE5 0x68 0x65 0x50 0x48 0x2F 0x23 0x4A
Configuration last modified by 10.19.0.254 at 4-12-12 13:49:06
#sh ip int brief
Interface IP-Address OK? Method Status Protocol
Vlan1 10.19.0.110 YES NVRAM up up
GigabitEthernet1/0/1 unassigned YES unset up up
#sh vlan bri
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Gi1/0/8, Gi1/0/9, Gi1/0/10, Gi1/0/11, Gi1/0/12, Gi1/0/13, Gi1/0/14,
1001 NO-ACCESS active
1002 fddi-default act/unsup
1003 trcrf-default act/unsup
1004 fddinet-default act/unsup
1005 trbrf-default act/unsup
Problematic Switch
#sh int trunk
Port Mode Encapsulation Status Native vlan
Gi0/1 on 802.1q trunking 1
Gi0/2 on 802.1q trunking 1
Port Vlans allowed on trunk
Gi0/1 1-4094
Gi0/2 1-4094
Port Vlans allowed and active in management domain
Gi0/1 1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Gi0/2 1,25-32,125-132,503,803-804,830,850-851,853,935,977,984-985,990,998-999,1001
Port Vlans in spanning tree forwarding state and not pruned
Gi0/1 1,25,27,29,31,125,127,129,131,503,851,853,935,977,985,999,1001
Gi0/2 26,28,30,32,126,128,130,132,830,850,984,990,998
#sh vtp status
VTP Version capable : 1 to 3
VTP version running : 2
VTP Domain Name : CHI
VTP Pruning Mode : Enabled
VTP Traps Generation : Disabled
Device ID : 001b.0c7c.1300
Configuration last modified by 10.19.0.254 at 4-12-12 13:49:06
Feature VLAN:
--------------
VTP Operating Mode : Client
Maximum VLANs supported locally : 1005
Number of existing VLANs : 36
Configuration Revision : 235
MD5 digest : 0xE5 0x68 0x65 0x50 0x48 0x2F 0x23 0x4A
0xC1 0x88 0x19 0x24 0x22 0x30 0x8F 0x5B
#sh ip int brief
Interface IP-Address OK? Method Status Protocol
Vlan1 10.19.0.112 YES manual up up
GigabitEthernet0/1 unassigned YES unset up up
GigabitEthernet0/2 unassigned YES unset up up
#sh vlan bri
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active
1001 NO-ACCESS active
1002 fddi-default act/unsup
1003 trcrf-default act/unsup
1004 fddinet-default act/unsup
1005 trbrf-default act/unsup
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: