1 VLAN can not communicate with 1 other VLAN on my layer 3 switch interVLAN routing

[kylemiller452225788]

got an odd one I haven't encountered before.  I have one VLAN with devices on it that can't communicate to 1 of my other vlans.  I have a layer 3 switch with several interface VLANs following the scheme 10.x.a.x, 10.x.b.x, 10.x c.x etc.  I had full connectivity between everything and all of the sudden 1 of my VLANs cant contact any device on another VLAN ( Important VLAN containing DC,DHCP, DNS ).  No ACLs have been placed that would have effected it or any other major network changes. I have tried deleting the interface VLAN and recreating it, restarting the switch stack, and even upgrading the IOS to a more recent release, but it did not solve anything.  Again the confusing part is that it had been working for several weeks now and it just decided to stop working...  Additionally confusing is I can ping through my switch and firewall out to the internet from the troubled VLAN.  Just not that one important VLAN. Any help would be appreciated, Thanks.

[balaji.bandi]

Can you post the configuraiton of the device

show version

show run

show vla

show spann brief

show ip interface brief

show ip route.

[kylemiller452225788]

Due to confidentially reasons I will have to censor out some information but here is the crucial information.

show ver:

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
lanbasek9 Smart License lanbasek9
None Subscription Smart License None

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 56 WS-C3850-48T 16.9.5 CAT3K_CAA-UNIVERSALK9 INSTALL
2 56 WS-C3850-48T 16.9.5 CAT3K_CAA-UNIVERSALK9 INSTALL

Configuration register is 0x102

Show VLAN:

VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Gi2/0/15, Gi2/0/22
5 x Management active Gi1/0/17, Gi1/0/21, Gi1/0/22, Gi1/0/26, Gi1/0/30, Gi1/0/39, Gi1/0/45, Gi2/0/10
Gi2/0/13, Gi2/0/17, Gi2/0/26
10 Data active Gi1/0/1, Gi1/0/2, Gi1/0/3, Gi1/0/4, Gi1/0/9, Gi1/0/10, Gi1/0/11, Gi1/0/12
Gi1/0/13, Gi1/0/14, Gi1/0/15, Gi1/0/16, Gi1/0/18, Gi1/0/19, Gi1/0/20, Gi1/0/23
Gi1/0/25, Gi1/0/27, Gi1/0/28, Gi1/0/29, Gi1/0/31, Gi1/0/32, Gi1/0/34, Gi1/0/35
Gi1/0/36, Gi1/0/38, Gi1/0/42, Gi1/0/43, Gi1/0/46, Gi1/0/47, Gi2/0/1, Gi2/0/2
Gi2/0/4, Gi2/0/5, Gi2/0/6, Gi2/0/7, Gi2/0/8, Gi2/0/9, Gi2/0/11, Gi2/0/12
Gi2/0/14, Gi2/0/16, Gi2/0/18, Gi2/0/20, Gi2/0/21, Gi2/0/30, Gi2/0/32, Gi2/0/39
11 x Production active
13 x Voice active Gi1/0/15, Gi1/0/27, Gi2/0/39
14 x IT Department active
16 x Security active Gi1/0/24, Gi1/0/44
17 xServers active
18 x Printers active
20 Voice active Gi1/0/1, Gi1/0/2, Gi1/0/3, Gi1/0/4, Gi1/0/5, Gi1/0/6, Gi1/0/7, Gi1/0/9, Gi1/0/11
Gi1/0/12, Gi1/0/13, Gi1/0/14, Gi1/0/16, Gi1/0/18, Gi1/0/19, Gi1/0/20, Gi1/0/23
Gi1/0/24, Gi1/0/25, Gi1/0/28, Gi1/0/29, Gi1/0/31, Gi1/0/32, Gi1/0/34, Gi1/0/35
Gi1/0/36, Gi1/0/38, Gi1/0/42, Gi1/0/43, Gi1/0/44, Gi1/0/46, Gi1/0/47, Gi2/0/1
Gi2/0/2, Gi2/0/4, Gi2/0/5, Gi2/0/6, Gi2/0/7, Gi2/0/8, Gi2/0/9, Gi2/0/11
Gi2/0/12, Gi2/0/14, Gi2/0/16, Gi2/0/18, Gi2/0/19, Gi2/0/20, Gi2/0/21, Gi2/0/23
Gi2/0/24, Gi2/0/25, Gi2/0/27, Gi2/0/28, Gi2/0/29, Gi2/0/30, Gi2/0/31, Gi2/0/32
Gi2/0/33, Gi2/0/34, Gi2/0/35, Gi2/0/36, Gi2/0/37, Gi2/0/38, Gi2/0/40, Gi2/0/41
Gi2/0/42, Gi2/0/43, Gi2/0/44, Gi2/0/45, Gi2/0/46, Gi2/0/47, Gi2/0/48
30 VLAN0030 active Gi1/0/5, Gi1/0/6, Gi1/0/7
31 Ring_Central active
33 8x8_Voice_VLAN active Gi1/0/10
123 Voice 2 active Gi2/0/23, Gi2/0/24, Gi2/0/25
666 Blackhole active
1002 fddi-default act/unsup
1003 token-ring-default act/unsup
1004 fddinet-default act/unsup
1005 trnet-default act/unsup

VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1 enet 100001 1500 - - - - - 0 0
5 enet 100005 1500 - - - - - 0 0
10 enet 100010 1500 - - - - - 0 0
11 enet 100011 1500 - - - - - 0 0
13 enet 100013 1500 - - - - - 0 0
14 enet 100014 1500 - - - - - 0 0
16 enet 100016 1500 - - - - - 0 0
17 enet 100017 1500 - - - - - 0 0
18 enet 100018 1500 - - - - - 0 0
20 enet 100020 1500 - - - - - 0 0
30 enet 100030 1500 - - - - - 0 0
31 enet 100031 1500 - - - - - 0 0
33 enet 100033 1500 - - - - - 0 0
123 enet 100123 1500 - - - - - 0 0
666 enet 100666 1500 - - - - - 0 0
1002 fddi 101002 1500 - - - - - 0 0
1003 tr 101003 1500 - - - - - 0 0
1004 fdnet 101004 1500 - - - ieee - 0 0
1005 trnet 101005 1500 - - - ibm - 0 0

Remote SPAN VLANs
------------------------------------------------------------------------------

Primary Secondary Type Ports
------- --------- ----------------- ------------------------------------------

show spann:

VLAN0001
Spanning tree enabled protocol ieee
Root ID Priority 32769
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p

VLAN0005
Spanning tree enabled protocol ieee
Root ID Priority 32773
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32773 (priority 32768 sys-id-ext 5)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/17 Desg FWD 4 128.17 P2p Edge
Gi1/0/21 Desg FWD 4 128.21 P2p Edge
Gi1/0/22 Desg FWD 4 128.22 P2p Edge
Gi1/0/26 Desg FWD 4 128.26 P2p Edge
Gi1/0/30 Desg FWD 4 128.30 P2p Edge
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/39 Desg FWD 4 128.39 P2p Edge
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p
Gi2/0/10 Desg FWD 4 128.106 P2p Edge
Gi2/0/13 Desg FWD 4 128.109 P2p Edge
Gi2/0/26 Desg FWD 4 128.122 P2p Edge

VLAN0010
Spanning tree enabled protocol ieee
Root ID Priority 32778
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1 Desg FWD 4 128.1 P2p Edge
Gi1/0/3 Desg FWD 4 128.3 P2p Edge
Gi1/0/4 Desg FWD 4 128.4 P2p Edge
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/11 Desg FWD 4 128.11 P2p Edge
Gi1/0/13 Desg FWD 4 128.13 P2p Edge
Gi1/0/15 Desg FWD 4 128.15 P2p Edge
Gi1/0/16 Desg FWD 4 128.16 P2p Edge
Gi1/0/19 Desg FWD 4 128.19 P2p Edge
Gi1/0/20 Desg FWD 4 128.20 P2p Edge
Gi1/0/23 Desg FWD 19 128.23 P2p Edge
Gi1/0/27 Desg FWD 4 128.27 P2p Edge
Gi1/0/28 Desg FWD 4 128.28 P2p Edge
Gi1/0/31 Desg FWD 4 128.31 P2p Edge
Gi1/0/32 Desg FWD 4 128.32 P2p Edge
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/34 Desg FWD 4 128.34 P2p Edge
Gi1/0/35 Desg FWD 19 128.35 P2p Edge
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/38 Desg FWD 4 128.38 P2p Edge
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi1/0/42 Desg FWD 4 128.42 P2p Edge
Gi1/0/43 Desg FWD 4 128.43 P2p Edge
Gi1/0/46 Desg FWD 4 128.46 P2p Edge
Gi2/0/2 Desg FWD 4 128.98 P2p Edge
Gi2/0/3 Desg FWD 4 128.99 P2p
Gi2/0/4 Desg FWD 4 128.100 P2p Edge
Gi2/0/5 Desg FWD 19 128.101 P2p Edge
Gi2/0/6 Desg FWD 4 128.102 P2p Edge
Gi2/0/7 Desg FWD 4 128.103 P2p Edge
Gi2/0/8 Desg FWD 4 128.104 P2p Edge
Gi2/0/9 Desg FWD 4 128.105 P2p Edge
Gi2/0/11 Desg FWD 4 128.107 P2p Edge
Gi2/0/14 Desg FWD 4 128.110 P2p Edge
Gi2/0/16 Desg FWD 4 128.112 P2p Edge
Gi2/0/18 Desg FWD 4 128.114 P2p Edge
Gi2/0/30 Desg FWD 4 128.126 P2p Edge
Gi2/0/32 Desg FWD 4 128.128 P2p Edge
Gi2/0/39 Desg FWD 4 128.135 P2p Edge

VLAN0011
Spanning tree enabled protocol ieee
Root ID Priority 32779
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32779 (priority 32768 sys-id-ext 11)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p

VLAN0013
Spanning tree enabled protocol ieee
Root ID Priority 32781
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32781 (priority 32768 sys-id-ext 13)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/15 Desg FWD 4 128.15 P2p Edge
Gi1/0/27 Desg FWD 4 128.27 P2p Edge
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p
Gi2/0/39 Desg FWD 4 128.135 P2p Edge

VLAN0014
Spanning tree enabled protocol ieee
Root ID Priority 32782
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32782 (priority 32768 sys-id-ext 14)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p

VLAN0016
Spanning tree enabled protocol ieee
Root ID Priority 32784
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32784 (priority 32768 sys-id-ext 16)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/24 Desg FWD 19 128.24 P2p Edge
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi1/0/44 Desg FWD 19 128.44 P2p Edge
Gi2/0/3 Desg FWD 4 128.99 P2p

VLAN0017
Spanning tree enabled protocol ieee
Root ID Priority 32785
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32785 (priority 32768 sys-id-ext 17)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p

VLAN0018
Spanning tree enabled protocol ieee
Root ID Priority 32786
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32786 (priority 32768 sys-id-ext 18)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p

VLAN0020
Spanning tree enabled protocol ieee
Root ID Priority 32788
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32788 (priority 32768 sys-id-ext 20)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1 Desg FWD 4 128.1 P2p Edge
Gi1/0/3 Desg FWD 4 128.3 P2p Edge
Gi1/0/4 Desg FWD 4 128.4 P2p Edge
Gi1/0/5 Desg FWD 4 128.5 P2p Edge
Gi1/0/6 Desg FWD 4 128.6 P2p Edge
Gi1/0/7 Desg FWD 4 128.7 P2p Edge
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/11 Desg FWD 4 128.11 P2p Edge
Gi1/0/13 Desg FWD 4 128.13 P2p Edge
Gi1/0/16 Desg FWD 4 128.16 P2p Edge
Gi1/0/19 Desg FWD 4 128.19 P2p Edge
Gi1/0/20 Desg FWD 4 128.20 P2p Edge
Gi1/0/23 Desg FWD 19 128.23 P2p Edge
Gi1/0/24 Desg FWD 19 128.24 P2p Edge
Gi1/0/28 Desg FWD 4 128.28 P2p Edge
Gi1/0/31 Desg FWD 4 128.31 P2p Edge
Gi1/0/32 Desg FWD 4 128.32 P2p Edge
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/34 Desg FWD 4 128.34 P2p Edge
Gi1/0/35 Desg FWD 19 128.35 P2p Edge
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/38 Desg FWD 4 128.38 P2p Edge
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi1/0/42 Desg FWD 4 128.42 P2p Edge
Gi1/0/43 Desg FWD 4 128.43 P2p Edge
Gi1/0/44 Desg FWD 19 128.44 P2p Edge
Gi1/0/46 Desg FWD 4 128.46 P2p Edge
Gi2/0/2 Desg FWD 4 128.98 P2p Edge
Gi2/0/3 Desg FWD 4 128.99 P2p
Gi2/0/4 Desg FWD 4 128.100 P2p Edge
Gi2/0/5 Desg FWD 19 128.101 P2p Edge
Gi2/0/6 Desg FWD 4 128.102 P2p Edge
Gi2/0/7 Desg FWD 4 128.103 P2p Edge
Gi2/0/8 Desg FWD 4 128.104 P2p Edge
Gi2/0/9 Desg FWD 4 128.105 P2p Edge
Gi2/0/11 Desg FWD 4 128.107 P2p Edge
Gi2/0/14 Desg FWD 4 128.110 P2p Edge
Gi2/0/16 Desg FWD 4 128.112 P2p Edge
Gi2/0/18 Desg FWD 4 128.114 P2p Edge
Gi2/0/19 Desg FWD 4 128.115 P2p Edge
Gi2/0/24 Desg FWD 4 128.120 P2p Edge
Gi2/0/25 Desg FWD 4 128.121 P2p Edge
Gi2/0/28 Desg FWD 4 128.124 P2p Edge
Gi2/0/29 Desg FWD 4 128.125 P2p Edge
Gi2/0/30 Desg FWD 4 128.126 P2p Edge
Gi2/0/31 Desg FWD 4 128.127 P2p Edge
Gi2/0/32 Desg FWD 4 128.128 P2p Edge
Gi2/0/34 Desg FWD 4 128.130 P2p Edge

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------

Gi2/0/35 Desg FWD 4 128.131 P2p Edge
Gi2/0/36 Desg FWD 4 128.132 P2p Edge
Gi2/0/37 Desg FWD 4 128.133 P2p Edge
Gi2/0/38 Desg FWD 4 128.134 P2p Edge
Gi2/0/40 Desg FWD 4 128.136 P2p Edge
Gi2/0/41 Desg FWD 4 128.137 P2p Edge
Gi2/0/42 Desg FWD 4 128.138 P2p Edge
Gi2/0/43 Desg FWD 4 128.139 P2p Edge
Gi2/0/44 Desg FWD 4 128.140 P2p Edge
Gi2/0/45 Desg FWD 4 128.141 P2p Edge
Gi2/0/47 Desg FWD 4 128.143 P2p Edge

VLAN0030
Spanning tree enabled protocol ieee
Root ID Priority 32798
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32798 (priority 32768 sys-id-ext 30)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/5 Desg FWD 4 128.5 P2p Edge
Gi1/0/6 Desg FWD 4 128.6 P2p Edge
Gi1/0/7 Desg FWD 4 128.7 P2p Edge
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p

VLAN0031
Spanning tree enabled protocol ieee
Root ID Priority 32799
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32799 (priority 32768 sys-id-ext 31)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p

VLAN0033
Spanning tree enabled protocol ieee
Root ID Priority 32801
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32801 (priority 32768 sys-id-ext 33)
Address x.x.x
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p

VLAN0123
Spanning tree enabled protocol ieee
Root ID Priority 32891
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32891 (priority 32768 sys-id-ext 123)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p
Gi2/0/24 Desg FWD 4 128.120 P2p Edge
Gi2/0/25 Desg FWD 4 128.121 P2p Edge

VLAN0666
Spanning tree enabled protocol ieee
Root ID Priority 33434
Address x.x.x
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 33434 (priority 32768 sys-id-ext 666)
Address 2c31.24b9.da00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/8 Desg FWD 4 128.8 P2p
Gi1/0/33 Desg FWD 4 128.33 P2p
Gi1/0/37 Desg FWD 4 128.37 P2p
Gi1/0/40 Desg FWD 4 128.40 P2p
Gi1/0/41 Desg FWD 4 128.41 P2p
Gi2/0/3 Desg FWD 4 128.99 P2p

Show IP route:

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is x.x.x.x to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via x.x.x.x
10.0.0.0/8 is variably subnetted, 22 subnets, 2 masks
C 10.x.31.0/24 is directly connected, Vlan30
L 10.x.31.1/32 is directly connected, Vlan30
C 10.x.38.0/24 is directly connected, Vlan10
L 10.x.38.1/32 is directly connected, Vlan10
C 10.y.38.0/24 is directly connected, Vlan31
L 10.y.38.250/32 is directly connected, Vlan31
C 10.z.5.0/24 is directly connected, Vlan5
L 10.z.5.1/32 is directly connected, Vlan5
C 10.z.11.0/24 is directly connected, Vlan11
L 10.z.11.1/32 is directly connected, Vlan11
C 10.z.13.0/24 is directly connected, Vlan13
L 10.z.13.1/32 is directly connected, Vlan13
C 10.z.14.0/24 is directly connected, Vlan14
L 10.z.14.1/32 is directly connected, Vlan14
C 10.z.16.0/24 is directly connected, Vlan16
L 10.z.16.1/32 is directly connected, Vlan16
C 10.z.17.0/24 is directly connected, Vlan17
L 10.z.17.1/32 is directly connected, Vlan17
C 10.z.18.0/24 is directly connected, Vlan18
L 10.z.18.1/32 is directly connected, Vlan18
C 10.z.33.0/24 is directly connected, Vlan33
L 10.z.33.1/32 is directly connected, Vlan33

Show run:

interface GigabitEthernet1/0/1
switchport access vlan 10
switchport mode access
switchport voice vlan 20
spanning-tree portfast

interface Vlan1
no ip address
!
interface Vlan5
description x
ip address 10.z.5.1 255.255.255.0
!
interface Vlan10
ip address 10.x.38.1 255.255.255.0
!
interface Vlan11
description x
ip address 10.z.11.1 255.255.255.0
!
interface Vlan13
description x

ip address 10.z.13.1 255.255.255.0
ip helper-address x.x.x.x
!
interface Vlan14
description x
ip address 10.z.14.1 255.255.255.0
!
interface Vlan16
description x
ip address 10.z.16.1 255.255.255.0
!
interface Vlan17
description x
ip address 10.z.17.1 255.255.255.0
!
interface Vlan18
description x
ip address 10.z.18.1 255.255.255.0
!
interface Vlan20
no ip address
!
interface Vlan30
ip address 10.x.31.1 255.255.255.0
!
interface Vlan31
description x
ip address 10.y.31.1  255.255.255.0
ip helper-address x.x.x.x
!
interface Vlan33
ip address x.x.x.x 255.255.255.0
ip helper-address x.x.x.x
!
ip forward-protocol nd
no ip http server
ip http authentication local
no ip http secure-server
ip http secure-active-session-modules none
ip http active-session-modules none
ip route 0.0.0.0 0.0.0.0 x.x.x.x
ip ssh maxstartups 3
ip ssh version 2

Show IP interface brief:

Interface IP-Address OK? Method Status Protocol
Vlan1 unassigned YES NVRAM up up
Vlan5 10.z.5.1 YES NVRAM up up
Vlan10 10.x.38.1 YES NVRAM up up
Vlan11 10.z.11.1 YES NVRAM up up
Vlan13 10.z.13.1 YES NVRAM up up
Vlan14 10.z.14.1 YES NVRAM up up
Vlan16 10.z.16.1 YES NVRAM up up
Vlan17 10.z.17.1 YES NVRAM up up
Vlan18 10.z.18.1 YES NVRAM up up
Vlan20 unassigned YES unset up up
Vlan30 10.x.31.1 YES NVRAM up up
Vlan31 10.y.38.250 YES NVRAM up up
Vlan33 10.z.33.1 YES NVRAM up up
GigabitEthernet0/0 unassigned YES NVRAM administratively down down
GigabitEthernet1/0/1 unassigned YES unset up up
GigabitEthernet1/0/2 unassigned YES unset down down
GigabitEthernet1/0/3 unassigned YES unset up up
GigabitEthernet1/0/4 unassigned YES unset up up
GigabitEthernet1/0/5 unassigned YES unset up up
GigabitEthernet1/0/6 unassigned YES unset up up
GigabitEthernet1/0/7 unassigned YES unset up up
GigabitEthernet1/0/8 unassigned YES unset up up
GigabitEthernet1/0/9 unassigned YES unset down down
GigabitEthernet1/0/10 unassigned YES unset down down
GigabitEthernet1/0/11 unassigned YES unset up up
GigabitEthernet1/0/12 unassigned YES unset down down
GigabitEthernet1/0/13 unassigned YES unset up up
GigabitEthernet1/0/14 unassigned YES unset down down
GigabitEthernet1/0/15 unassigned YES unset up up
GigabitEthernet1/0/16 unassigned YES unset up up
GigabitEthernet1/0/17 unassigned YES unset up up
GigabitEthernet1/0/18 unassigned YES unset down down
GigabitEthernet1/0/19 unassigned YES unset up up
GigabitEthernet1/0/20 unassigned YES unset up up
GigabitEthernet1/0/21 unassigned YES unset up up
GigabitEthernet1/0/22 unassigned YES unset up up
GigabitEthernet1/0/23 unassigned YES unset up up
GigabitEthernet1/0/24 unassigned YES unset up up
GigabitEthernet1/0/25 unassigned YES unset down down
GigabitEthernet1/0/26 unassigned YES unset up up
GigabitEthernet1/0/27 unassigned YES unset up up
GigabitEthernet1/0/28 unassigned YES unset up up
GigabitEthernet1/0/29 unassigned YES unset down down
GigabitEthernet1/0/30 unassigned YES unset up up
GigabitEthernet1/0/31 unassigned YES unset up up
GigabitEthernet1/0/32 unassigned YES unset up up
GigabitEthernet1/0/33 unassigned YES unset up up
GigabitEthernet1/0/34 unassigned YES unset up up
GigabitEthernet1/0/35 unassigned YES unset up up
GigabitEthernet1/0/36 unassigned YES unset down down
GigabitEthernet1/0/37 unassigned YES unset up up
GigabitEthernet1/0/38 unassigned YES unset up up
GigabitEthernet1/0/39 unassigned YES unset up up
GigabitEthernet1/0/40 unassigned YES unset up up
GigabitEthernet1/0/41 unassigned YES unset up up
GigabitEthernet1/0/42 unassigned YES unset up up
GigabitEthernet1/0/43 unassigned YES unset up up
GigabitEthernet1/0/44 unassigned YES unset up up
GigabitEthernet1/0/45 unassigned YES unset down down
GigabitEthernet1/0/46 unassigned YES unset up up
GigabitEthernet1/0/47 unassigned YES unset down down
GigabitEthernet1/0/48 10.22.37.252 YES NVRAM down down
GigabitEthernet1/1/1 unassigned YES unset down down
GigabitEthernet1/1/2 unassigned YES unset down down
GigabitEthernet1/1/3 unassigned YES unset down down
GigabitEthernet1/1/4 unassigned YES unset down down
Te1/1/1 unassigned YES unset down down
Te1/1/2 unassigned YES unset down down
Te1/1/3 unassigned YES unset down down
Te1/1/4 unassigned YES unset down down
GigabitEthernet2/0/1 unassigned YES unset down down
GigabitEthernet2/0/2 unassigned YES unset up up
GigabitEthernet2/0/3 unassigned YES unset up up
GigabitEthernet2/0/4 unassigned YES unset up up
GigabitEthernet2/0/5 unassigned YES unset up up
GigabitEthernet2/0/6 unassigned YES unset up up
GigabitEthernet2/0/7 unassigned YES unset up up
GigabitEthernet2/0/8 unassigned YES unset up up
GigabitEthernet2/0/9 unassigned YES unset up up
GigabitEthernet2/0/10 unassigned YES unset up up
GigabitEthernet2/0/11 unassigned YES unset up up
GigabitEthernet2/0/12 unassigned YES unset down down
GigabitEthernet2/0/13 unassigned YES unset up up
GigabitEthernet2/0/14 unassigned YES unset up up
GigabitEthernet2/0/15 unassigned YES unset down down
GigabitEthernet2/0/16 unassigned YES unset up up
GigabitEthernet2/0/17 unassigned YES unset down down
GigabitEthernet2/0/18 unassigned YES unset up up
GigabitEthernet2/0/19 unassigned YES unset up up
GigabitEthernet2/0/20 unassigned YES unset down down
GigabitEthernet2/0/21 unassigned YES unset down down
GigabitEthernet2/0/22 unassigned YES unset down down
GigabitEthernet2/0/23 unassigned YES unset down down
GigabitEthernet2/0/24 unassigned YES unset up up
GigabitEthernet2/0/25 unassigned YES unset up up
GigabitEthernet2/0/26 unassigned YES unset up up
GigabitEthernet2/0/27 unassigned YES unset down down
GigabitEthernet2/0/28 unassigned YES unset up up
GigabitEthernet2/0/29 unassigned YES unset up up
GigabitEthernet2/0/30 unassigned YES unset up up
GigabitEthernet2/0/31 unassigned YES unset up up
GigabitEthernet2/0/32 unassigned YES unset up up
GigabitEthernet2/0/33 unassigned YES unset down down
GigabitEthernet2/0/34 unassigned YES unset up up
GigabitEthernet2/0/35 unassigned YES unset up up
GigabitEthernet2/0/36 unassigned YES unset up up
GigabitEthernet2/0/37 unassigned YES unset up up
GigabitEthernet2/0/38 unassigned YES unset up up
GigabitEthernet2/0/39 unassigned YES unset up up
GigabitEthernet2/0/40 unassigned YES unset up up
GigabitEthernet2/0/41 unassigned YES unset up up
GigabitEthernet2/0/42 unassigned YES unset up up
GigabitEthernet2/0/43 unassigned YES unset up up
GigabitEthernet2/0/44 unassigned YES unset up up
GigabitEthernet2/0/45 unassigned YES unset up up
GigabitEthernet2/0/46 unassigned YES unset down down
GigabitEthernet2/0/47 unassigned YES unset up up
GigabitEthernet2/0/48 unassigned YES unset down down
GigabitEthernet2/1/1 unassigned YES unset down down
GigabitEthernet2/1/2 unassigned YES unset down down
GigabitEthernet2/1/3 unassigned YES unset down down
GigabitEthernet2/1/4 unassigned YES unset down down
Te2/1/1 unassigned YES unset down down
Te2/1/2 unassigned YES unset down down
Te2/1/3 unassigned YES unset down down
Te2/1/4 unassigned YES unset down down

----------------------------------------------------------------------------------------------------

The main issue is between VLANs 13 and VLAN 10.  From VLAN 10 I can ping everything on 13 but from 13 I can not ping anything on 10.

[balaji.bandi]

high level looks ok

make sure you have command in the config

ip routing  (which you may have but not seen in the config)

can you also post below testing outcomes :

show ip arp vlan 10

show ip arp vlan 13

show sdm prefer

show interface vlan 13

show interface vlan 10

from the switch are you able to ping VLAN 10 and VLAN 13 ? can you post an example ping?

Maybe code has a buggy, but we can only confirm based on the output.

[kylemiller452225788]

IP routing is enabled and all other VLANs are functioning normally besides 13.

pings from VLAN 10 to a device located on VLAN 10:

XXX-3850#ping 10.X.38.11 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.x.38.11, timeout is 2 seconds:
Packet sent with a source address of 10.x.38.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms

Pings from VLAN 13 to a device located on VLAN 10:

XXX-3850#ping 10.x.38.11 source vlan 13
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.x.38.11, timeout is 2 seconds:
Packet sent with a source address of 10.z.13.1
.....
Success rate is 0 percent (0/5)

Pings from VLAN 10 to a device located on VLAN 13:

XXX-3850#ping 10.z.13.4 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.z.13.4, timeout is 2 seconds:
Packet sent with a source address of 10.x.38.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms

Pings from VLAN 13 to a device located on VLAN 13:

XXX-3850#ping 10.z.13.4 source vlan 13
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.z.13.4, timeout is 2 seconds:
Packet sent with a source address of 10.z.13.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 MS

They only pings that are not successful are the ones coming from VLAN 13 to VLAN 10.

Show IP arp vlan 10:

Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.x.38.1 - aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.x.x 0 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.88 0 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.5 0 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.89 84 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.12 74 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.99 0 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.10 0 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.11 0 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.91 0 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.95 0 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.86 34 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.16 0 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.81 185 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.223 128 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.239 67 aaaa.bbbb.cccc ARPA Vlan10
Internet 10.x.38.108 0 aaaa.bbbb.cccc ARPA Vlan10

Show IP arp vlan 13:

Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.z.13.1 - aaaa.bbbb.cccc ARPA Vlan13
Internet 10.z.13.4 1 aaaa.bbbb.cccc ARPA Vlan13

Show sdm prefer:

Showing SDM Template Info

This is the Advanced template.
Number of VLANs: 4094
Unicast MAC addresses: 32768
Overflow Unicast MAC addresses: 512
L2 Multicast entries: 4096
Overflow L2 Multicast entries: 512
L3 Multicast entries: 4096
Overflow L3 Multicast entries: 512
Directly connected routes: 16384
Indirect routes: 7168
STP Instances: 4096
Security Access Control Entries: 3072
QoS Access Control Entries: 2560
Policy Based Routing ACEs: 1024
Netflow ACEs: 768
Flow SPAN ACEs: 512
Tunnels: 256
LISP Instance Mapping Entries: 256
Control Plane Entries: 512
Input Netflow flows: 8192
Output Netflow flows: 16384
SGT/DGT (or) MPLS VPN entries: 4096
SGT/DGT (or) MPLS VPN Overflow entries: 512
Wired clients: 2048
MACSec SPD Entries: 256
MPLS L3 VPN VRF: 127
MPLS Labels: 2048
MPLS L3 VPN Routes VRF Mode: 7168
MPLS L3 VPN Routes Prefix Mode: 3072
MVPN MDT Tunnels: 256
L2 VPN EOMPLS Attachment Circuit: 256
MAX VPLS Bridge Domains : 64
MAX VPLS Peers Per Bridge Domain: 8
MAX VPLS/VPWS Pseudowires : 256
These numbers are typical for L2 and IPv4 features.
Some features such as IPv6, use up double the entry size;
so only half as many entries can be created.
* values can be modified by sdm cli.

Show interface vlan 13:

Vlan13 is up, line protocol is up , Autostate Enabled
Hardware is Ethernet SVI, address is aaaa.bbbb.cccc (bia aaaa.bbbb.cccc)
Description: x
Internet address is 10.z.13.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:02:00, output 00:02:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1579128 packets input, 123799958 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
1134528 packets output, 77112427 bytes, 0 underruns
0 output errors, 2 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

show interface vlan 10:

Vlan10 is up, line protocol is up , Autostate Enabled
Hardware is Ethernet SVI, address is aaaa.bbbb.cccc (bia aaaa.bbbb.cccc)
Internet address is 10.x.38.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/45420/6 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 15432000 bits/sec, 5087 packets/sec
5 minute output rate 9831000 bits/sec, 3958 packets/sec
53916688009 packets input, 60324034591565 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
43280181551 packets output, 53138123896250 bytes, 0 underruns
0 output errors, 2 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

------------------------------------------------------------------------------------------------

Thanks for taking the time to look through this.  I had someone else look at this issue and they also thought it might be a bug which prompted me to do a restart and try a different IOS.  I have also tried layer 1 troubleshooting steps such as: Changing the physical switchport, Trying different cables, trying the device on a different known working switch.

[balaji.bandi]

If the device has long uptime try reboot - what happenss if you move that IP address to a different VLAN (not mentioned as voice) 

create a new vlan XX name XXXX(not voice)

[kylemiller452225788]

Device only has an uptime of 2 weeks cause I did a restart during the IOS upgrade.  I tried creating a new VLAN 10.z.33.0/24.  This VLAN could not communicate with  VLAN 10 either.  I'm starting to think their just might be something wrong with the switch stack.

@paul driver I shutdown the SVI and it still could not ping so im thinking its not a duplicate IP issue.

[balaji.bandi]

uptime of 2 weeks cause I did a restart during the IOS upgrade  - is this working before the upgrade?

[kylemiller452225788]

No, I did the upgrade to see if that would fix the issue. It was on version 16.3.4 and I upgraded to 16.9.5.

[paul driver]

Hello

If you can, Shut down SVI 13 on the L3 Sw and test the ping again from vlan 10, if it works then you have a duplicate addressing which is negating proper inter-vlan communication between those vlans.

[amikat]

Hi,

Can you please post the Gi2/0/23, Gi2/0/24, Gi2/0/25 interfaces configuration as these are assigned to Vlan20 and Vlan123 named "Voice" and "Voice 2" respectively which sounds rather strange.

Thanks & Regards,

Antonin

[kylemiller452225788]

Figured out the issue.... Improper default gateway set on the devices.