Re: Vlan issue after ios upgrade

LIBJCFPL · ‎12-24-2019

I recently upgraded the IOS of a Cisco Catalyst 3650 to Fuji-16.9.4. Ever since the upgrade, the vlan keeps going up and down. This is a display of the log:

*Dec 24 02:33:44.463: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/2, changed state to down
*Dec 24 02:33:45.463: %LINK-3-UPDOWN: Interface Vlan15, changed state to down
*Dec 24 02:33:45.470: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.1 (Vlan15) is down: interface down
*Dec 24 02:33:46.463: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan15, changed state to down
*Dec 24 02:33:47.519: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/2, changed state to up
*Dec 24 02:34:18.799: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/2, changed state to down
*Dec 24 02:34:21.856: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/2, changed state to up
*Dec 24 02:34:52.859: %LINK-3-UPDOWN: Interface Vlan15, changed state to up
*Dec 24 02:34:53.860: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan15, changed state to up
*Dec 24 02:34:54.096: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.1 (Vlan15) is up: new adjacency
*Dec 24 02:35:10.688: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/2, changed state to down
*Dec 24 02:35:11.688: %LINK-3-UPDOWN: Interface Vlan15, changed state to down
*Dec 24 02:35:11.694: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.1 (Vlan15) is down: interface down
*Dec 24 02:35:12.688: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan15, changed state to down
*Dec 24 02:35:13.740: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/2, changed state to up
*Dec 24 02:35:44.745: %LINK-3-UPDOWN: Interface Vlan15, changed state to up
*Dec 24 02:35:45.746: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan15, changed state to up
*Dec 24 02:35:46.858: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.1 (Vlan15) is up: new adjacency

These are the settings related on the interface and the vlan:
interface GigabitEthernet1/1/2
switchport trunk allowed vlan 15
switchport mode trunk
speed nonegotiate

interface Vlan1
ip address 10.10.36.1 255.255.254.0
ip helper-address 10.10.4.30
ip helper-address 10.10.4.32

interface Vlan15
ip address 172.18.5.2 255.255.255.0

router eigrp JCL-EIGRP

router eigrp JCPL-EIGRP

address-family ipv4 unicast autonomous-system 100

topology base
exit-af-topology
network 10.10.36.1 0.0.0.0
network 172.18.5.2 0.0.0.0

exit-address-family

Is there anywhere I should be looking at?

ngkin2010 · ‎12-24-2019

Hi,

The SVI (interface VLAN15) will UP only if there is any physical port (either access or trunk) that belong to VLAN15 is UP.

E.g. if GigabitEthernet1/1/2 is the only interface that belong to VLAN15, when Gi1/1/2 go down, the SVI will down afterward.

Could you issue "show vlan" and "show interface trunk" to double verify?

LIBJCFPL · ‎12-24-2019

This is what I get from the commands

MARION-3650-24#sh vlan

VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Gi1/0/1, Gi1/0/2, Gi1/0/3, Gi1/0/4, Gi1/0/5, Gi1/0/6, Gi1/0/7, Gi1/0/8, Gi1/0/9, Gi1/0/10, Gi1/0/11, Gi1/0/12, Gi1/0/13, Gi1/0/14, Gi1/0/15, Gi1/0/16, Gi1/0/17
Gi1/0/18, Gi1/0/19, Gi1/0/20, Gi1/0/21, Gi1/0/22, Gi1/0/23, Gi1/0/24, Gi1/1/1, Gi1/1/3, Gi1/1/4
15 EVPL-VLAN active
1002 fddi-default act/unsup
1003 token-ring-default act/unsup
1004 fddinet-default act/unsup
1005 trnet-default act/unsup
3179 MARION active

VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1 enet 100001 1500 - - - - - 0 0
15 enet 100015 1500 - - - - - 0 0
1002 fddi 101002 1500 - - - - - 0 0
1003 tr 101003 1500 - - - - - 0 0
1004 fdnet 101004 1500 - - - ieee - 0 0
1005 trnet 101005 1500 - - - ibm - 0 0
3179 enet 103179 1500 - - - - - 0 0

Remote SPAN VLANs
------------------------------------------------------------------------------

Primary Secondary Type Ports
------- --------- ----------------- ------------------------------------------

MARION-3650-24#sh int
MARION-3650-24#sh interfaces trunk

Port Mode Encapsulation Status Native vlan
Gi1/1/2 on 802.1q trunking 1

Port Vlans allowed on trunk
Gi1/1/2 15

Port Vlans allowed and active in management domain
Gi1/1/2 15

Port Vlans in spanning tree forwarding state and not pruned
Gi1/1/2 15
MARION-3650-24#

Leo Laohoo · ‎12-24-2019

Reboot the switch again.

LIBJCFPL · ‎12-24-2019

I have rebooted the switch and the issue is still there.

Leo Laohoo · ‎12-24-2019

Good.
Now go find out what that downstream client. Maybe that requires a reboot?

LIBJCFPL · ‎12-24-2019

Rebooted the 4500 and issue still exists. This is what I see in the logs:

*Dec 24 19:16:09.635: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.6.2 (Vlan74) is down: holding time expired
*Dec 24 19:16:09.943: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.2 (Vlan15) is down: holding time expired
*Dec 24 19:16:32.802: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.2 (Vlan15) is up: new adjacency
*Dec 24 19:16:33.458: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.6.2 (Vlan74) is up: new adjacency
*Dec 24 19:17:07.383: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.2 (Vlan15) is down: holding time expired
*Dec 24 19:17:30.683: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.2 (Vlan15) is up: new adjacency
*Dec 24 19:17:50.803: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.2 (Vlan15) is down: holding time expired

Leo Laohoo · ‎12-24-2019

Hold on ... a 4500 on the other end?
Post the complete output to the command "sh interface Gi1/1/2)".
Post the complete output to the same command on the 4500.
I want to see what is going on why the EIGRP keeps flapping.

LIBJCFPL · ‎12-24-2019

This is an EVPL with Verizon. The gigabit interface is connected to a canoga box, which in turn is connected to the 4500. Below is the result:

MARION-3650-24#sh int gi1/1/2
GigabitEthernet1/1/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 00f6.63f6.f61a (bia 00f6.63f6.f61a)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 254/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2000 bits/sec, 3 packets/sec
5 minute output rate 3000 bits/sec, 2 packets/sec
202958 packets input, 48915417 bytes, 0 no buffer
Received 10888 broadcasts (10888 multicasts)
0 runts, 0 giants, 0 throttles
2393 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 10888 multicast, 0 pause input
0 input packets with dribble condition detected
191143 packets output, 42505404 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

For the 4500, this is what is on my config:

interface TenGigabitEthernet1/1
description ** Connection to NJRoom **
switchport mode trunk
!
interface TenGigabitEthernet1/2
description ** Connection to GovDocs **
switchport mode trunk
!
interface TenGigabitEthernet1/3
description ** Connection to Lending **
switchport mode trunk
!
interface TenGigabitEthernet1/4
description ** Connection to Reference **
switchport mode trunk
!
interface TenGigabitEthernet1/5
description ** Connection to ChildRoom **
switchport mode trunk
!
interface TenGigabitEthernet1/6
description ** Connection to Literacy **
switchport mode trunk
!
interface TenGigabitEthernet1/7
description ** Connection to TechServ **
switchport mode trunk
!
interface TenGigabitEthernet1/8
switchport mode trunk
!
interface TenGigabitEthernet1/9
!
interface TenGigabitEthernet1/10
!
interface TenGigabitEthernet1/11
switchport trunk allowed vlan 1,11,13-15,42,71,74,76,168,170
switchport mode trunk
!
interface TenGigabitEthernet1/12
switchport trunk allowed vlan 1,11,13-15,42,71,74,76,168,170
switchport mode trunk
!
interface TenGigabitEthernet1/13
switchport mode access
!
interface TenGigabitEthernet1/14
switchport mode access
!
interface TenGigabitEthernet1/15
switchport trunk allowed vlan 11,13-15,42,71,74,76,168,170
switchport mode trunk
!
interface TenGigabitEthernet1/16
switchport mode trunk
!
interface TenGigabitEthernet2/1
switchport mode access
shutdown
spanning-tree portfast
!
interface TenGigabitEthernet2/2
shutdown
!
interface TenGigabitEthernet2/3
shutdown
!
interface TenGigabitEthernet2/4
shutdown
!
interface TenGigabitEthernet2/5
shutdown
!
interface TenGigabitEthernet2/6
shutdown
!
interface TenGigabitEthernet2/7
shutdown
!
interface TenGigabitEthernet2/8
switchport trunk allowed vlan 11,13-15,71,74,76,168,170
switchport mode trunk
!
interface Vlan1
ip address 10.10.4.5 255.255.254.0
!
interface Vlan11
ip address 172.18.1.1 255.255.255.0
!
interface Vlan13
ip address 172.18.2.1 255.255.255.0
!
interface Vlan14
ip address 172.18.4.1 255.255.255.0
!
interface Vlan15
ip address 172.18.5.1 255.255.255.0
!
interface Vlan42
ip address 172.18.9.1 255.255.255.0
!
interface Vlan71
ip address 172.18.3.1 255.255.255.0
!
interface Vlan74
ip address 172.18.6.1 255.255.255.0
!
interface Vlan76
ip address 172.18.8.1 255.255.255.0
!
interface Vlan168
no ip address
shutdown
!
interface Vlan170
ip address 172.18.7.1 255.255.255.0
!
!
router eigrp JCPL-EIGRP
!
address-family ipv4 unicast autonomous-system 100
!
topology base
exit-af-topology
network 10.10.4.1 0.0.0.0
network 10.10.4.5 0.0.0.0
network 172.18.1.1 0.0.0.0
network 172.18.2.1 0.0.0.0
network 172.18.3.1 0.0.0.0
network 172.18.4.1 0.0.0.0
network 172.18.5.1 0.0.0.0
network 172.18.6.1 0.0.0.0
network 172.18.7.1 0.0.0.0
network 172.18.8.1 0.0.0.0
network 172.18.9.1 0.0.0.0
exit-address-family
!
ip forward-protocol nd
no ip http server
!
ip route 0.0.0.0 0.0.0.0 10.10.4.2
!

Leo Laohoo · ‎12-24-2019

@LIBJCFPL wrote:

reliability 254/255

2393 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

Layer 1 issue.

balaji.bandi · ‎12-24-2019

As per the Logs the Port going up and down. to confirm waht is wrong ?

1.have you checked physical cable, make sure they are intact, (worth checking)

2. After upgrade, is the switch rebooted ? is this single switch or stack ?

3. what is the device connected otehr side to GigabitEthernet1/1/2 ? do you see any logs other side ?

can you post below output

show version

show switch

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

LIBJCFPL · ‎12-24-2019

1. Physical cable has been checked and replaced.

2. Switch has been rebooted after upgrade. Single switch.

3. It is connected to a 4500. This is what I see on it:

Dec 24 07:34:30.346: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.2 (Vlan15) is up: new adjacency
*Dec 24 07:34:30.346: EIGRP: Enqueueing UPDATE on Vl15 - paklen 0 nbr 172.18.5.2 tid 0 iidbQ un/rely 0/1 peerQ un/rely 0/0
*Dec 24 07:34:30.348: EIGRP: Requeued unicast on Vlan15
*Dec 24 07:34:30.351: EIGRP: Sending UPDATE on Vl15 - paklen 0 nbr 172.18.5.2 tid 0
*Dec 24 07:34:30.351: AS 100, Flags 0x1:(INIT), Seq 23780/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:32.352: EIGRP: Sending UPDATE on Vl15 - paklen 0 nbr 172.18.5.2, retry 1, RTO 3000 tid 0
*Dec 24 07:34:32.353: AS 100, Flags 0x1:(INIT), Seq 23780/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.337: EIGRP: Received UPDATE on Vl15 - paklen 0 nbr 172.18.5.2
*Dec 24 07:34:35.337: AS 100, Flags 0x1:(INIT), Seq 2211/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.352: EIGRP: Sending UPDATE on Vl15 - paklen 0 nbr 172.18.5.2, retry 2, RTO 4500 tid 0
*Dec 24 07:34:35.353: AS 100, Flags 0x1:(INIT), Seq 23780/2211 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.356: EIGRP: Received UPDATE on Vl15 - paklen 44 nbr 172.18.5.2
*Dec 24 07:34:35.356: AS 100, Flags 0x8:(EOT), Seq 2212/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.360: EIGRP: Received ACK on Vl15 - paklen 0 nbr 172.18.5.2
*Dec 24 07:34:35.360: AS 100, Flags 0x0:(NULL), Seq 0/23780 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.361: EIGRP: Enqueueing UPDATE on Vl15 - paklen 0 tid 0 iidbQ un/rely 0/1 serno 1-2349
*Dec 24 07:34:35.363: EIGRP: Sending UPDATE on Vl15 - paklen 793 tid 0
*Dec 24 07:34:35.364: AS 100, Flags 0x0:(NULL), Seq 23781/0 interfaceQ 0/0 iidbQ un/rely 0/0 serno 1-2349Dec 24 07:34:30.346: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.18.5.2 (Vlan15) is up: new adjacency
*Dec 24 07:34:30.346: EIGRP: Enqueueing UPDATE on Vl15 - paklen 0 nbr 172.18.5.2 tid 0 iidbQ un/rely 0/1 peerQ un/rely 0/0
*Dec 24 07:34:30.348: EIGRP: Requeued unicast on Vlan15
*Dec 24 07:34:30.351: EIGRP: Sending UPDATE on Vl15 - paklen 0 nbr 172.18.5.2 tid 0
*Dec 24 07:34:30.351: AS 100, Flags 0x1:(INIT), Seq 23780/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:32.352: EIGRP: Sending UPDATE on Vl15 - paklen 0 nbr 172.18.5.2, retry 1, RTO 3000 tid 0
*Dec 24 07:34:32.353: AS 100, Flags 0x1:(INIT), Seq 23780/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.337: EIGRP: Received UPDATE on Vl15 - paklen 0 nbr 172.18.5.2
*Dec 24 07:34:35.337: AS 100, Flags 0x1:(INIT), Seq 2211/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.352: EIGRP: Sending UPDATE on Vl15 - paklen 0 nbr 172.18.5.2, retry 2, RTO 4500 tid 0
*Dec 24 07:34:35.353: AS 100, Flags 0x1:(INIT), Seq 23780/2211 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.356: EIGRP: Received UPDATE on Vl15 - paklen 44 nbr 172.18.5.2
*Dec 24 07:34:35.356: AS 100, Flags 0x8:(EOT), Seq 2212/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.360: EIGRP: Received ACK on Vl15 - paklen 0 nbr 172.18.5.2
*Dec 24 07:34:35.360: AS 100, Flags 0x0:(NULL), Seq 0/23780 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1
*Dec 24 07:34:35.361: EIGRP: Enqueueing UPDATE on Vl15 - paklen 0 tid 0 iidbQ un/rely 0/1 serno 1-2349
*Dec 24 07:34:35.363: EIGRP: Sending UPDATE on Vl15 - paklen 793 tid 0
*Dec 24 07:34:35.364: AS 100, Flags 0x0:(NULL), Seq 23781/0 interfaceQ 0/0 iidbQ un/rely 0/0 serno 1-2349

LIBJCFPL · ‎12-24-2019

MARION-3650-24#sh ver
Cisco IOS XE Software, Version 16.09.04
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.9.4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 22-Aug-19 17:33 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.56, RELEASE SOFTWARE (P)

MARION-3650-24 uptime is 20 hours, 57 minutes
Uptime for this control processor is 21 hours, 0 minutes
System returned to ROM by Reload command
System image file is "flash:cat3k_caa-universalk9.16.09.04.SPA.bin"
Last reload reason: Reload command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
ipservicesk9 Smart License ipservicesk9
None Subscription Smart License None

Smart Licensing Status: UNREGISTERED/EVAL MODE

cisco WS-C3650-24PS (MIPS) processor (revision K0) with 832441K/6147K bytes of memory.
Processor board ID FDO2024Q027
2 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
1609272K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : 00:f6:63:f6:f6:00
Motherboard Assembly Number : 73-15899-05
Motherboard Serial Number : FDO20232KX1
Model Revision Number : K0
Motherboard Revision Number : B0
Model Number : WS-C3650-24PS
System Serial Number : FDO2024Q027

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 28 WS-C3650-24PS 16.9.4 CAT3K_CAA-UNIVERSALK9 BUNDLE

Configuration register is 0x102

MARION-3650-24#sh switch
Switch/Stack Mac Address : 00f6.63f6.f600 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
*1 Active 00f6.63f6.f600 1 V03 Ready

Kavin Patel · ‎12-24-2019

Hi,

It looks like interface VLAN 15 might keep on going down. I would suggest to take a look at that interface and enter the commands

int VLAN 15

autostate exclude

no shutdown

Which will keep it always up regardless of any ports in VLAN 15 being up or not

Also, I noticed that only VLAN 15 is allowed on the trunk. I would suggest to allow VLAN 1 also, as some protocols still use VLAN 1 such as CDP etc.

So check and see after allowing VLAN 1 on that interface and keeping it up.

LIBJCFPL · ‎12-24-2019

Do you mean: switchport auto-state exclude

Am unable to input autostate exclude