Unable to ping mgt IP address of the switch but it pass the traffic

[rishisemwal]

We have more than 40 cisco switches in the network that has lost IP connectivity via the management IP.  All ports, access, and network trunks, are up and passing data and voice traffic but we cannot ping some time all the access switches via the management IP.   It has been working fine from many years but last few weeks it has gone up/down a few times.  All the switches are reachable from core switch and it will be reachable from the local network once we ping the any management IP of the switch.  I do not understand why it is happening.

[marce1000]

 - Could be a bug in the software version  they are running , upgrade to latest advisory , especially if currently on an older version , 

 M.

[Richard Burts]

The OP mentions 40 switches in the network. I am not clear whether the problem is impacting all of the switches or impacting some of the switches. Can you clarify?

The OP says it has been working many years, so the suggestion to upgrade code might be a good idea. 

Is it possible that there have been some changes in the network which might be causing this problem?

Are there some switches that you can access while they are experiencing the problem? If so when the problem is occurring please access the switch(es) and post the output of show interface status and show arp.

[rishisemwal]

ASW01#sh int status

Port Name Status Vlan Duplex Speed Type
Gi1/0/1 "VTC" notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/2 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/3 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/4 "RESERVED FOR AP c notconnect 33 auto auto 10/100/1000BaseTX
Gi1/0/5 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/6 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/7 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/8 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/9 connected 11 a-full a-100 10/100/1000BaseTX
Gi1/0/10 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/11 connected 21 a-full a-1000 10/100/1000BaseTX

Port Name Status Vlan Duplex Speed Type
Gi1/0/12 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/13 connected 11 a-full a-100 10/100/1000BaseTX
Gi1/0/14 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/15 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/16 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/17 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/18 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/19 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/20 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/21 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/22 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/23 notconnect 11 auto auto 10/100/1000BaseTX

Port Name Status Vlan Duplex Speed Type
Gi1/0/24 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/25 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/26 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/27 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/28 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/29 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/30 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/31 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/32 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/33 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/34 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/35 "Test-Switch" notconnect 1 auto auto 10/100/1000BaseTX

Port Name Status Vlan Duplex Speed Type
Gi1/0/36 "Test-AP" notconnect 313 auto auto 10/100/1000BaseTX
Gi1/0/37 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/38 connected 11 a-full a-1000 10/100/1000BaseTX
Gi1/0/39 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/40 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/41 disabled 33 auto auto 10/100/1000BaseTX
Gi1/0/42 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/43 notconnect 11 auto auto 10/100/1000BaseTX
Gi1/0/44 connected trunk a-full a-100 10/100/1000BaseTX
Gi1/0/45 "AP-FTS" connected 33 a-full a-1000 10/100/1000BaseTX
Gi1/0/46 "Switch-Sanjeev" connected trunk a-full a-100 10/100/1000BaseTX
Gi1/0/47 "Switch-Diwan" connected trunk a-full a-100 10/100/1000BaseTX

Port Name Status Vlan Duplex Speed Type
Gi1/0/48 "UPLink To UNPOL" connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/49 "UPLink To Core" connected trunk a-full a-1000 1000BaseLX SFP
Gi1/0/50 "UPLink To Core" notconnect 1 auto auto Not Present
Gi1/0/51 "UPLink To Core" notconnect 1 auto auto Not Present
Gi1/0/52 "UPLink To Core" notconnect 1 auto auto Not Present
Fa0 disabled routed auto auto 10/100BaseTX
ASW01#

ASW01#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.155.11.121 48 bc26.c71e.91bf ARPA Vlan35
Internet 10.155.35.1 45 bc26.c71e.91bf ARPA Vlan35
Internet 10.155.35.20 - 94d4.69c7.9741 ARPA Vlan35
Internet 10.155.35.27 237 00d6.febd.5c43 ARPA Vlan35
Internet 10.155.35.30 105 a478.06c9.cbd9 ARPA Vlan35
Internet 10.155.35.31 74 00d6.fe8d.a7c1 ARPA Vlan35
Internet 10.155.35.38 16 84eb.ef1d.b059 ARPA Vlan35

[Richard Burts]

Thank you for the output that I requested. In that output it looks like the ports on this switch are connected to vlan 11. Is that correct?

The show arp shows addresses in vlan 35 (including this which I believe would be the address of this switch Internet 10.155.35.20 - 94d4.69c7.9741 ARPA Vlan35, is this correct?) Can you confirm that the gateway address for this switch is one of the addresses in the output, and if so which one? And can you confirm that this switch was experiencing the problem when you gathered the output?

And would you post the output of these commands:

show ip interface brief

show ip route

[rishisemwal]

Hello,

1. Yes its correct data port are on vlan 11.
2. Yes this switch IP is 10.155.35.20
3. Gateway on this switch is 10.155.35.1 ( valn 35 is  a management Ips of the switches )
4. This switch is also among the others one which are experiencing the problem. It looks like management vlan is entering in sleeping mode and after some time it starts pinging automatically. This management vlan is up/down after few minutes.

ASW01 #sh ip int brie

Interface              IP-Address      OK? Method Status                Protocol

Vlan1                  unassigned      YES NVRAM  administratively down down

Vlan11                 unassigned      YES unset  up                    up

Vlan35                 10.155.35.20    YES NVRAM  up                    up

FastEthernet0          unassigned      YES NVRAM  administratively down down

GigabitEthernet1/0/1   unassigned      YES unset  down                  down

GigabitEthernet1/0/2   unassigned      YES unset  down                  down

GigabitEthernet1/0/3   unassigned      YES unset  up                    up

GigabitEthernet1/0/4   unassigned      YES unset  down                  down

GigabitEthernet1/0/5   unassigned      YES unset  up                    up

GigabitEthernet1/0/6   unassigned      YES unset  up                    up

GigabitEthernet1/0/7   unassigned      YES unset  up                    up

GigabitEthernet1/0/8   unassigned      YES unset  down                  down

GigabitEthernet1/0/9   unassigned      YES unset  up                    up

GigabitEthernet1/0/10  unassigned      YES unset  up                    up

GigabitEthernet1/0/11  unassigned      YES unset  up                    up

GigabitEthernet1/0/12  unassigned      YES unset  down                  down

GigabitEthernet1/0/13  unassigned      YES unset  up                    up

GigabitEthernet1/0/14  unassigned      YES unset  up                    up

GigabitEthernet1/0/15  unassigned      YES unset  down                  down

GigabitEthernet1/0/16  unassigned      YES unset  up                    up

GigabitEthernet1/0/17  unassigned      YES unset  down                  down

GigabitEthernet1/0/18  unassigned      YES unset  up                    up

GigabitEthernet1/0/19  unassigned      YES unset  up                    up

GigabitEthernet1/0/20  unassigned      YES unset  down                  down

GigabitEthernet1/0/21  unassigned      YES unset  down                  down

GigabitEthernet1/0/22  unassigned      YES unset  up                    up

GigabitEthernet1/0/23  unassigned      YES unset  down                  down

GigabitEthernet1/0/24  unassigned      YES unset  up                    up

GigabitEthernet1/0/25  unassigned      YES unset  up                    up

GigabitEthernet1/0/26  unassigned      YES unset  up                    up

GigabitEthernet1/0/27  unassigned      YES unset  up                    up

GigabitEthernet1/0/28  unassigned      YES unset  down                  down

GigabitEthernet1/0/29  unassigned      YES unset  up                    up

GigabitEthernet1/0/30  unassigned      YES unset  down                  down

GigabitEthernet1/0/31  unassigned      YES unset  up                    up

GigabitEthernet1/0/32  unassigned      YES unset  down                  down

GigabitEthernet1/0/33  unassigned      YES unset  up                    up

GigabitEthernet1/0/34  unassigned      YES unset  up                    up

GigabitEthernet1/0/35  unassigned      YES unset  down                  down

GigabitEthernet1/0/36  unassigned      YES unset  down                  down

GigabitEthernet1/0/37  unassigned      YES unset  down                  down

GigabitEthernet1/0/38  unassigned      YES unset  up                    up

GigabitEthernet1/0/39  unassigned      YES unset  down                  down

GigabitEthernet1/0/40  unassigned      YES unset  down                  down

GigabitEthernet1/0/41  unassigned      YES unset  administratively down down

GigabitEthernet1/0/42  unassigned      YES unset  down                  down

GigabitEthernet1/0/43  unassigned      YES unset  down                  down

GigabitEthernet1/0/44  unassigned      YES unset  up                    up

GigabitEthernet1/0/45  unassigned      YES unset  up                    up

GigabitEthernet1/0/46  unassigned      YES unset  up                    up

GigabitEthernet1/0/47  unassigned      YES unset  up                    up

GigabitEthernet1/0/48  unassigned      YES unset  up                    up

GigabitEthernet1/0/49  unassigned      YES unset  up                    up

GigabitEthernet1/0/50  unassigned      YES unset  down                  down

GigabitEthernet1/0/51  unassigned      YES unset  down                  down

GigabitEthernet1/0/52  unassigned      YES unset  down                  down

ASW01#sh ip route

Default gateway is 10.155.35.1

Host               Gateway           Last Use    Total Uses  Interface

ICMP redirect cache is empty

ASW01#

[Richard Burts]

Thank you for the additional information. Now that we know the IP of the gateway I see something interesting. Looking into the arp table I see these entries:

Internet 10.155.11.121 48 bc26.c71e.91bf ARPA Vlan35
Internet 10.155.35.1 45 bc26.c71e.91bf ARPA Vlan35

Why are there 2 devices in vlan 35 with the same mac address? Is it possible that 10.155.11.121 was a recent addition/change?

[Joseph W. Doherty]

I'm wondering if some newly occurring broadcast flood?  It's the type of thing that might hit multiple devices control plane, at the same time.

[rishisemwal]

1. This is a laptop ip address 10.155.11.121 not switch and its working fine,  no idea how its learning through vlan 35.
2. I did  show arp command in the core switch I found that all the vlan’s gateway mac address is the same. Is it correct ?
3.  Why management vlan 35 is going in sleeping mode. I ping switch IP from another switch then it starts ping.  

Internet  10.155.11.1             -   bc26.c71e.91bf  ARPA   Vlan11

Internet  10.155.21.1             -   bc26.c71e.91bf  ARPA   Vlan21

Internet  10.155.33.1             -   bc26.c71e.91bf  ARPA   Vlan33

Internet  10.155.34.1             -   bc26.c71e.91bf  ARPA   Vlan34

Internet  10.155.35.1             -   bc26.c71e.91bf  ARPA   Vlan35

Internet  10.155.62.1             -   bc26.c71e.91bf  ARPA   Vlan62

Internet  10.155.114.1            -   bc26.c71e.91bf  ARPA   Vlan114

# I have checked one of the distribution switch I found same entry here Internet 10.155.11.121 74 bc26.c71e.91bf ARPA Vlan35

DSW#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.155.11.121 74 bc26.c71e.91bf ARPA Vlan35
Internet 10.155.35.1 34 bc26.c71e.91bf ARPA Vlan35
Internet 10.155.35.10 - a478.06c9.b8d9 ARPA Vlan35
Internet 10.155.35.19 66 94d4.69c7.e141 ARPA Vlan35
Internet 10.155.35.20 67 94d4.69c7.9741 ARPA Vlan35
Internet 10.155.35.50 89 cc5a.538f.4941 ARPA Vlan35
DSW#

[Richard Burts]

I am not clear what is going on here and why/how multiple devices are having the same mac address. I do know that having multiple devices in the same vlan using the same mac address can cause problems. And I believe this is your problem.