Loop in /24 Subnet, No ping beyond .1 and .2

[BashedRoot]

Running 3650 switch. I have this odd issue where I cannot get VLAN 26 pinging on it's /24, only gateway .1 and first public IP .2

This is what I see, which doesn't look normal. How do I fix this?

The server itself is configured fine (Centos 7)

[root@localhost ~]# cat /etc/sysconfig/network-s*/ifcfg-em1
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="em1"
UUID="******"
DEVICE="em1"
ONBOOT="yes"
IPADDR="xxx.xxx.30.2"
PREFIX="24"
GATEWAY="xxx.xxx.30.1"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_PRIVACY="no"

Cisco3650#show ip route xxx.xxx.30.0
Routing entry for xxx.xxx.30.0/24, 2 known subnets
  Attached (2 connections)
  Variably subnetted with 2 masks
C        xxx.xxx.30.0/24 is directly connected, Vlan26
L        xxx.xxx.30.1/32 is directly connected, Vlan26

Show run looks normal?

Cisco3650#show run int vlan 26
Building configuration...

Current configuration : 85 bytes
!
interface Vlan26
 description Server 26
 ip address xxx.xxx.30.1 255.255.255.0
end

[paul driver]

Hello

Have you tried another device into the same port of the server and tested again - If doesn't work then:

Check to see you have vlan26 int the vtp database of this switch and you have a complete arp entry for the server

sh vlan brief
Sh mac-address interface xxxxx ( port of server)
sh ip arp | in xxxx.xxxx.xxx ( mac adr of server)


Try changing the ip address of SVI26 to another address and see if that works and amend the D/G of your client.


res
Paul

[BashedRoot]

Please see my last post above (2nd below original) about 4 VLANs I found out doing the same.

26   VLAN26                           active    Gi1/0/2

Cisco3650#sh mac address interface Gi1/0/21
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  26    1803.73f2.ac0c    DYNAMIC     Gi1/0/21
Total Mac Addresses for this criterion: 1

[BashedRoot]

Cisco3650#show interface vlan 25
Vlan25 is up, line protocol is up 
  Hardware is Ethernet SVI, address is 002a.107d.8771 (bia 002a.107d.8771)
  Description: Server 25
  Internet address is xxx.xxx.98.97/28
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:01:43, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
     29290785 packets input, 3369330201 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     25858889 packets output, 5107315841 bytes, 0 underruns
     0 output errors, 1 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

	 
	 
Cisco3650#show interface vlan 26
Vlan26 is up, line protocol is up 
  Hardware is Ethernet SVI, address is 002a.107d.8765 (bia 002a.107d.8765)
  Description: Server 26
  Internet address is xxx.xxx.30.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 3000 bits/sec, 6 packets/sec
     1728395 packets input, 328212204 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     3396151 packets output, 1084431946 bytes, 0 underruns
     0 output errors, 2 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

	 
	 
Switch2#show interface vlan 27
Vlan27 is down, line protocol is down 
  Hardware is Ethernet SVI, address is 002a.10b7.79d5 (bia 002a.10b7.79d5)
  Description: Server 27
  Internet address is xxx.xxx.105.57/29
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 3d12h, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1018522 packets input, 188065862 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     1155057 packets output, 140817767 bytes, 0 underruns
     0 output errors, 1 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
	 
	 
	 
Switch2#show interface vlan 28
Vlan28 is up, line protocol is up 
  Hardware is Ethernet SVI, address is 002a.10b7.79d9 (bia 002a.10b7.79d9)
  Description: Server 28
  Internet address is xxx.xxx.107.57/29
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 7000 bits/sec, 5 packets/sec
  5 minute output rate 4000 bits/sec, 5 packets/sec
     1497000 packets input, 281725987 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     1636510 packets output, 200109257 bytes, 0 underruns
     0 output errors, 1 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

[BashedRoot]

Ok I just noticed something. This is happening to all four new VLANs I created last week. That's odd. None of their assigned IPs ping past the first public IP. The rest show this same error.

[root@localhost ~]# ping xxx.xxx.107.59
PING xxx.xxx.107.59 (xxx.xxx.107.59) 56(84) bytes of data.
From xxx.xxx.107.58 icmp_seq=1 Destination Host Unreachable
From xxx.xxx.107.58 icmp_seq=2 Destination Host Unreachable
From xxx.xxx.107.58 icmp_seq=3 Destination Host Unreachable
From xxx.xxx.107.58 icmp_seq=4 Destination Host Unreachable

This is how I create the VLANs:

Switch> en
Switch# conf t
Switch(config)#vlan 26
#name vlan26
#show vlan

Add IPs to VLAN:

Switch> en
switch# conf t
int vlan 26
ip address #ipumber-gateway# #netmask#

Assign VLAN to Port:

switch> en
switch# conf t
switch# int Gi1/0/x
Switch(config)# switchport access vlan26
Switch(config)# switchport mode access

[paul driver]

Hello

Have you tried testing with another device connect to that port?

Can you post the switch config please.

sh run
sh mac address-table count | in Total
sh mac address-table aging-time

res
paul

[BashedRoot]

Please *read* my post. This is happening with 4 new VLANs I created. I stated that twice. It is not a server issue nor a port issue.

Cisco3650#sh mac address-table count | in Total
Total Mac Addresses    : 0
Total Mac Addresses    : 2
Total Mac Addresses    : 35
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 1
Total Mac Addresses    : 2
Total Mac Addresses    : 1
Total Mac Addresses    : 34
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 6
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 2
Total Mac Addresses    : 1
Total Mac Addresses    : 1
Total Mac Address Space Available: 32655
Cisco3650#sh mac address-table aging-time
Global Aging Time:  300
Vlan    Aging Time
----    ----------

[paul driver]

Hello

The reason why I am asking this is that you show your test sourcing from just one host but if you say this not the case and all other routing is fine apart from those new L3 interfaces your recently created then it could be down to a possible bug.

what iOS are you running and licence base 

also what SDM template is applied to the switch?

res

paul

[BashedRoot]

> what iOS are you running and licence base 

> also what SDM template is applied to the switch?

How do I pull these details? I had to get rid of a previous network guy a while back and not sure how it was set up.

[Jon Marshall]

"sh ver" and "sh sdm prefer"

Jon

[Jon Marshall]

The devices you are trying to ping, are they connected to the L3 switch or are they on other switches connecting back to the L3 switch ?

Jon

[BashedRoot]

Switch #1 = Primary L3

Switch #2 = Trunk L2

The method I use to create VLANs and assign IPs, ports have always worked until these 4, oddly enough.

Technology Package License Information: 

-----------------------------------------------------------------
Technology-package                   Technology-package
Current             Type             Next reboot  
------------------------------------------------------------------
ipbasek9            Permanent        ipbasek9

cisco WS-C3650-24TS (MIPS) processor (revision N0) with 866081K/6147K bytes of memory.
Processor board ID FDO2027E0M9
25 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
252000K bytes of Crash Files at crashinfo:.
1611414K bytes of Flash at flash:.
0K bytes of  at webui:.
0K bytes of Dummy USB Flash at usbflash0:.

Base Ethernet MAC Address          : 
Motherboard Assembly Number        : 
Motherboard Serial Number          : 
Model Revision Number              : N0
Motherboard Revision Number        : A0
Model Number                       : WS-C3650-24TS
System Serial Number               : 


Switch Ports Model              SW Version        SW Image              Mode   
------ ----- -----              ----------        ----------            ----   
*    1 28    WS-C3650-24TS      16.3.1            CAT3K_CAA-UNIVERSALK9 INSTALL

Cisco3650#sh sdm prefer
Showing SDM Template Info

This is the Advanced (high scale) template.
  Number of VLANs:                                 4094
  Unicast MAC addresses:                           32768
  Overflow Unicast MAC addresses:                  512
  IGMP and Multicast groups:                       8192
  Overflow IGMP and Multicast groups:              512
  Directly connected routes:                       16384
  Indirect routes:                                 7168
  Security Access Control Entries:                 3072
  QoS Access Control Entries:                      2560
  Policy Based Routing ACEs:                       1024
  Netflow ACEs:                                    768
  Wireless Input Microflow policer ACEs:           256
  Wireless Output Microflow policer ACEs:          256
  Flow SPAN ACEs:                                  256
  Tunnels:                                         256
  Control Plane Entries:                           512
  Input Netflow flows:                             8192
  Output Netflow flows:                            16384
  SGT/DGT and MPLS VPN entries:                    3840
  SGT/DGT and MPLS VPN Overflow entries:           512
These numbers are typical for L2 and IPv4 features.
Some features such as IPv6, use up double the entry size;
so only half as many entries can be created.

[Jon Marshall]

So are you running VTP or are do you create the vlan on both switches separately ?

Jon

[BashedRoot]

I create the VLANs only on the switches they're physically ported to. Since the #2 switch is a trunk, I use the ip routing method from switch #1 to #2 for the VLANs, in order to utilize maximum TCAM allowance (IP Routing). It's always worked this way.

Now, VLAN 25 and 26 are on the main switch #1. 27/28 are on #2. These 4 were created the same day last week to setup new Dell boxes. It wasn't until yesterday I found out the IPs are not pinging past the gateway and first public IP of that assigned subnet, whether /29 or /24, etc.

[Jon Marshall]

Okay, so working with x.x.107.59 which you cannot ping.

I am assuming as this is in vlan 28 it is connected to switch 2. So from the 3650 can you try pinging it and then check the arp table on the 3650 and see if there is an entry for it.

Can you also confirm that no firewalls on the end devices could be blocking ping.

Jon

[BashedRoot]

Let's start with VLAN 26 on switch #1.

Cisco3650#show arp xxx.xxx.30.0
Cisco3650#show arp xxx.xxx.30.1
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  xxx.xxx.30.1             -   002a.107d.8765  ARPA   Vlan26
Cisco3650#show arp xxx.xxx.30.2
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  xxx.xxx.30.2            62   1803.73f2.ac0c  ARPA   Vlan26
Cisco3650#show arp xxx.xxx.30.3
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  xxx.xxx.30.3             0   Incomplete      ARPA   
Cisco3650#show arp xxx.xxx.30.4
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  xxx.xxx.30.4             0   Incomplete      ARPA   
Cisco3650#ping xxx.xxx.30.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to xxx.xxx.30.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
Cisco3650#ping xxx.xxx.30.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to xxx.xxx.30.3, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)