Inter-Vlan Routing Issue

[amatorjnr]

I have this weird problem I'm facing on my network.

I have inherited an inter-vlan configuration with a layer 3 switch and everything works out fine.

Recently I had to add a new server to the network and I've noticed something weird.

All computers are on a different vlan and can communicate with each other and other servers.

example. PC = Vlan 2

Servers= vlan 3

New Server = vlan 4

everything works, pings go through all vlans.

BUT

After a while, both PC's and Servers on Vlan 2 and 3 can't ping New Server on Vlan 4.

Until

1. after i disconnect new server network cable and reconnect it,

2. or shutdown and renable port New Server is connected to

3. or i log in to the layer 3 switch and ping New Server

4. or I use a computer on vlan 4 to ping New Server.

AFter performing any of the options 1-4. All other host can ping New Server again for a while, then it stops.

I've noticed so far that until I'm continously pinging NEw Server, pings from all other vlans stop going through.

what could be the cause.

PS. New Server port = Access

Inter-vlan routing works

I've changed network cable twice

switch doesn't show any errors on interface

[rasmus.elmholt]

Can you share the configuration on all 3 ports and the 3 SVI's.

And show ip arp of the L3 switch when the problem occurs?

The PC and the 2 servers are all directly connected to the L3 Switch?

[amatorjnr]

One thing I've realized just now is, all my Vlans have the same MAC address, so I suspect that's whats causing the issue.

Show log doesn't give any meaningful information.

And no, PC is not directly connected to the same switch as SERVER.

Vlan3 is up, line protocol is up , Autostate Enabled
Hardware is Ethernet SVI, address is 0008.e3ee.ec45 (bia 0008.e3ee.ec45)
Description: SERV_VLAN
Internet address is 172.16.13.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:12, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
L3 out Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
IPv6 L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
IPv6 L3 out Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
625 packets input, 80755 bytes, 0 no buffer
Received 826 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 packets output, 0 bytes, 0 underruns
0 output errors, 1 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

Vlan2 is up, line protocol is up , Autostate Enabled
Hardware is Ethernet SVI, address is 0008.e3ee.ec45 (bia 0008.e3ee.ec45)
Description: FLEX
Internet address is 172.16.14.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
L3 out Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
IPv6 L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
IPv6 L3 out Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
42486 packets input, 3963320 bytes, 0 no buffer
Received 97123 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 packets output, 0 bytes, 0 underruns
0 output errors, 1 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

Vlan4 is up, line protocol is up , Autostate Enabled
Hardware is Ethernet SVI, address is 0008.e3ee.ec45 (bia 0008.e3ee.ec45)
Description: NEW_SERV
Internet address is 172.16.15.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
L3 out Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
IPv6 L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
IPv6 L3 out Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
17002 packets input, 1763766 bytes, 0 no buffer
Received 31318 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 packets output, 0 bytes, 0 underruns
0 output errors, 1 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

[Giuseppe Larosa]

Hello ,

>> all my Vlans have the same MAC address, so I suspect that's whats causing the issue.

No , this is normal in Cisco implementation on many multillayer switch platforms and it is not a problem as they are in different L2 broadcast domains.

Rather I notice that the countes for L3 switches packet in and out are stuck at 0

L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
L3 out Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes

Can you post a show version of your L3 switch ?

The above can be just a cosmetic issue the counters at 0 or a sign of an issue like CEF disabled on SVIs

To verify STP stability you can use

show spanning-tree detail | include topology

check also show ip arp for the new server IP address over time.

Hope to help

Giuseppe

[amatorjnr]

This is the results of "show version" 

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.09.00.E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 19-Jul-16 15:19 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: 15.0(1r)SG12
JDS-CORE-SWI uptime is 1 day, 13 hours, 29 minutes
Uptime for this control processor is 1 day, 13 hours, 31 minutes
System returned to ROM by power-on
System image file is "flash1:unknown"
Jawa Revision 3, Winter Revision 0x0.0x41

Last reload reason: power-on

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Information for 'WS-C4500X-16'
License Level: ipbase Type: Permanent
Next reboot license Level: ipbase

cisco WS-C4500X-16 (MPC8572) processor (revision 9) with 4194304K bytes of physical memory.
Processor board ID
MPC8572 CPU at 1.5GHz, Cisco Catalyst 4500X
Last reset from PowerUp
13 Virtual Ethernet interfaces
32 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2101

Show spanning-tree | include topology outputs nothing 

[Giuseppe Larosa]

Hello,

>> Show spanning-tree | include topology outputs nothing 

you need to add the detail keyword

>> show spanning-tree detail | include topology

It is a C4500X with IOS XE

Hope to help

Giuseppe

[amatorjnr]

Number of topology changes 14 last change occurred 01:10:56 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300
Number of topology changes 859 last change occurred 00:00:20 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 14, notification 0, aging 300
Number of topology changes 700 last change occurred 00:00:20 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 14, notification 0, aging 300
Number of topology changes 147 last change occurred 00:00:13 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 21, notification 0, aging 300
Number of topology changes 171 last change occurred 00:00:50 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300
Number of topology changes 314 last change occurred 00:01:08 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300
Number of topology changes 84 last change occurred 00:51:22 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300
Number of topology changes 125 last change occurred 00:20:18 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300
Number of topology changes 8 last change occurred 1d00h ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300
Number of topology changes 118 last change occurred 00:04:49 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300
Number of topology changes 8 last change occurred 1d00h ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300
Number of topology changes 7 last change occurred 1d00h ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300
Number of topology changes 14 last change occurred 19:34:19 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 0, notification 0, aging 300

[Giuseppe Larosa]

Hello ,

focus on the second and third VLAN in the list

use

show spanning-tree summary to know the list of VLANs

then for the second and the third that appear to be very active in STP

>>Number of topology changes 859 last change occurred 00:00:20 ago
Times: hold 1, topology change 35, notification 2
Timers: hello 0, topology change 14, notification 0, aging 300
Number of topology changes 700 last change occurred 00:00:20 ago

use

show spanning-tree vlan X detail

Look for the interface that caused the last topology change

This will help you find the problematic switch(es) and links

Hope to help

Giuseppe

[MHM Cisco World]

there is broadcast storm I think .

[rasmus.elmholt]

Either that or just a loop in the network. Does show logging give any usefull information? MAC-Flapping?

[amatorjnr]

Nope. show logging doesn't show any mac-flapping issue

[amatorjnr]

I think so too but I can't find where its coming from cos all my troubleshooting leads to a dead end 

[paul driver]

Hello

On the switchport the server is connected - check it speed/duplex also make sure the port is defined as an access port and stp portfast is enabled.

On the server is it network card configuration correct - if it’s a windows server you can reset its network binding and api stack with the following command after which you need to reload the server and reapply it ip addressing.

start:run cmd

netsh interface ipv4 reset reset.log

netsh winsock reset reset.log

shutdown -r -f -t 5

[amatorjnr]

Yes, the port is set as an access port and stp portfast is enabled.

The server is also configured correctly, its not a windows server

[Kevin Kilgore]

Disconnect the new server and connect a laptop. See if it still occurs.