Networking issues on Cisco Catalyst 9500-48Y4C switch

[jshayne]

Hi,

I have two questions about the Cisco Catalyst 9500-48Y4C switch:

1. I am able to connect two uplink ports with two PCs using QSFP cables (status: connected). However, two PCs could NOT ping each other. Could anyone please let me know how to solve this issue? 

2. For app-hosting feature, this link [1] mentions:
"Catalyst 9500 High Performance, and 9600 series switches do not support AppGigabitEthernet interface. The connectivity for applications hosted on these models is achieved through management interface via loopback from any front panel ports."

Could anyone please help to explain what it means by "... through management interface via loopback from any front panel ports"?
If I understand correctly, that means the containers can be accessed via front panel ports. I would be grateful if you could give me an example of that. Additionally, is it possible to access the containers via uplink ports?

[1] https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9300-series-switches/white-paper-c11-742415.html

[balaji.bandi]

1. I am able to connect two uplink ports with two PCs using QSFP cables (status: connected). However, two PCs could NOT ping each other. Could anyone please let me know how to solve this issue? 

what is the IOS XE Code ?

what SFP speed ?

what ethernet card PC has ?

what is the PC 1 and PC2 IP address ?.

can you share the config of the PC Port connected ? (show run interface x/x show interface x/x)

2. For app-hosting feature, this link [1] mentions:

Container networking on the document mentioned what ports can be used. (since there is no appgigethernport available on cat 9500)

depends on IOS Code running, some new changes on 17.X  code

[jshayne]

Hi Balaji,

Thank you for your reply.

1. The connection between PCs and the switch (layer 2) is fine. It took me some time to figure it out. Therefore, I am sure the L2 connection is okay.
I am sure the IP addresses on PC1 and PC2 are fine. For example, when I performed a loop test between two NIC ports on PC1 (i.e., using a cable connecting two NIC ports), I was able to ping between them. That also means the NIC side is fine. Is it possible to use 100 Gbps uplink ports to connect to two PCs? I did not find any document about that.

2. Yes, I got it. How can I have the connectivity for applications achieved through the management interface via loopback from any front panel ports? It is really confusing to me. 

[balaji.bandi]

The port comes up, so is your PC is 100Gig port? what speed?

we have asked for some input before - let m ask again :

can you share the config of the PC Port connected? (show run interface x/x show interface x/x)

refer app hosting config guide :

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/prog/configuration/173/b_173_programmability_cg/application_hosting.html

[jshayne]

As I said, the L2 connection is fine. The PCs have 100 Gbps NICs and the speed is set to 100 Gbps as well. Running ethtool on PCs told me that the links are connected. Please let me know what you are trying to figure out. That helps me a lot.
If you want to see the configuration on the switch, here it is:

show int HundredGigE1/0/49
HundredGigE1/0/49 is up, line protocol is up (connected)
Hardware is Hundred Gigabit Ethernet, address is 8024.8f2f.44b1 (bia 8024.8f2f.44b1)
MTU 1500 bytes, BW 100000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Gb/s, link type is force-up, media type is QSFP 100G CU2M
Fec is auto
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:01, output hang never
Last clearing of "show interface" counters 2w4d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
72259 packets input, 11205482 bytes, 0 no buffer
Received 72233 broadcasts (56753 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 56753 multicast, 0 pause input
0 input packets with dribble condition detected
422745 packets output, 49262624 bytes, 0 underruns
Output 1 broadcasts (0 multicasts)
0 output errors, 0 collisions, 8 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

show int HundredGigE1/0/51
HundredGigE1/0/51 is up, line protocol is up (connected)
Hardware is Hundred Gigabit Ethernet, address is 8024.8f2f.44b3 (bia 8024.8f2f.44b3)
MTU 1500 bytes, BW 100000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Gb/s, link type is auto, media type is QSFP 100G CU2M
Fec is auto
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 2w4d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
225017 packets input, 24615384 bytes, 0 no buffer
Received 225006 broadcasts (209657 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 209657 multicast, 0 pause input
0 input packets with dribble condition detected
844420 packets output, 76776386 bytes, 0 underruns
Output 0 broadcasts (0 multicasts)
0 output errors, 0 collisions, 7 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

For the app hosting, I saw the document already. If you know how to have the connectivity for applications achieved through the management interface via loopback from any front panel ports, please let me know. Thanks.

[MHM Cisco World]

can you share STP status for both ports ?

[balaji.bandi]

the output looks interesting, and i see the ports are up.

So end device able to auto-negotiate with a switch with a 100Gb connection.

the ports are up, do you see MAC address table on that switch where end device connected.

For quick test before we go to any next level of diagnosis

what is the PC1 IP and what is PC2 IP

for testing create on switch

config t

!

interface vlan1

IP address x.x.x.x x.x.x.x   <- same IP range you used for PC1 and PC2

no shutdown

!

check from switch are you able to ping PC1 and PC2

post show IP arp ?

[jshayne]

For sure, it is not IP configuration issue on PCs. They are both in the same range 123.123.123.0/24. Again, with the loop test (direct cable between two PCs), I could ping from one PC to another. 

Assigning IP to vlan 1 did not work. It was unreachable from PCs to vlan 1's IP address (same IP range). I even assigned IP address to the uplink port before but there is no luck. When pinging between two PCs, I saw the switch arp table was empty.

[balaji.bandi]

 do you see MAC address table on that switch where the end device is connected?

yes heard that your PC to PC is back to working - that is expected.

if this is test Switch, can you post below :

show version

show run

show mac address table.

Note: I was not testd100GB to the end device directly using a PC. with UCS yes.

[jshayne]

Hi,

I did not see MAC table updated when the end device was connected, especially when I tried to ping the PCs. 

Sorry I could not show switch's configuration in detail. Here is some information that I hope will be useful for you:

show version
Cisco IOS XE Software, Version 17.03.04
Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.4, RELEASE SOFTWARE (fc3)

show mac address-table
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
All 0100.0ccc.cccc STATIC CPU
All 0100.0ccc.cccd STATIC CPU
All 0180.c200.0000 STATIC CPU
All 0180.c200.0001 STATIC CPU
All 0180.c200.0002 STATIC CPU
All 0180.c200.0003 STATIC CPU
All 0180.c200.0004 STATIC CPU
All 0180.c200.0005 STATIC CPU
All 0180.c200.0006 STATIC CPU
All 0180.c200.0007 STATIC CPU
All 0180.c200.0008 STATIC CPU
All 0180.c200.0009 STATIC CPU
All 0180.c200.000a STATIC CPU
All 0180.c200.000b STATIC CPU
All 0180.c200.000c STATIC CPU
All 0180.c200.000d STATIC CPU
All 0180.c200.000e STATIC CPU
All 0180.c200.000f STATIC CPU
All 0180.c200.0010 STATIC CPU
All 0180.c200.0021 STATIC CPU
All ffff.ffff.ffff STATIC CPU
1 8024.8f2f.451f STATIC Vl1

[balaji.bandi]

For me, if you do not see the MAC table on the interface where the end device is connected, so switch does not know layer2 information.

i suspect its shows as up a layer 1 level and failing layer2 level somehow.

at this stage, I can only say upgrade to 17.6.4 and test it or if this is critical for your business Raise a TAC to investigate for you.

Twinax cable we used mostly connect to switch to switch, never tested connecting to PC directly (this may be not support I am in guess)

[jshayne]

For the networking issue with uplink ports, I am not sure what you meant by STP status. Here are some information. Please let me know what you suspect and let me know if you need more input. Thanks.

------ On switch side ---------------

show int HundredGigE1/0/49 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
--------- ----------------- ---------- --------- --------- ---------
Hu1/0/49 0.0 17.0 0.0 0.0 35.5

High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- ----------------- ---------- --------- --------- ---------
Hu1/0/49 0.00 0.00 1.80 0.82 0.06

High Alarm High Warn Low Warn Low Alarm
Current Threshold Threshold Threshold Threshold
Port Lane (milliamperes) (mA) (mA) (mA) (mA)
--------- ---- --------------- ---------- --------- --------- ---------
Hu1/0/49 1 0.0 33.3 0.0 35.9 3.0
Hu1/0/49 2 0.0 33.3 0.0 35.9 3.0
Hu1/0/49 3 0.0 33.3 0.0 35.9 3.0
Hu1/0/49 4 0.0 33.3 0.0 35.9 3.0

Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Hu1/0/49 1 -40.0 -5.4 2.5 0.9 -40.0
Hu1/0/49 2 -40.0 -5.4 2.5 0.9 -40.0
Hu1/0/49 3 -40.0 -5.4 2.5 0.9 -40.0
Hu1/0/49 4 -40.0 -5.4 2.5 0.9 -40.0

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Hu1/0/49 1 N/A -0.8 -0.8 -0.8 -0.8
Hu1/0/49 2 N/A -0.8 -0.8 -0.8 -0.8
Hu1/0/49 3 N/A -0.8 -0.8 -0.8 -0.8
Hu1/0/49 4 N/A -0.8 -0.8 -0.8 -0.8

show int HundredGigE1/0/51 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
--------- ----------------- ---------- --------- --------- ---------
Hu1/0/51 0.0 17.0 0.0 0.0 35.5

High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- ----------------- ---------- --------- --------- ---------
Hu1/0/51 0.00 0.00 1.80 0.82 0.06

High Alarm High Warn Low Warn Low Alarm
Current Threshold Threshold Threshold Threshold
Port Lane (milliamperes) (mA) (mA) (mA) (mA)
--------- ---- --------------- ---------- --------- --------- ---------
Hu1/0/51 1 0.0 33.3 0.0 35.9 3.0
Hu1/0/51 2 0.0 33.3 0.0 35.9 3.0
Hu1/0/51 3 0.0 33.3 0.0 35.9 3.0
Hu1/0/51 4 0.0 33.3 0.0 35.9 3.0

Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Hu1/0/51 1 -40.0 -5.4 2.5 0.9 -40.0
Hu1/0/51 2 -40.0 -5.4 2.5 0.9 -40.0
Hu1/0/51 3 -40.0 -5.4 2.5 0.9 -40.0
Hu1/0/51 4 -40.0 -5.4 2.5 0.9 -40.0

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Hu1/0/51 1 N/A -0.8 -0.8 -0.8 -0.8
Hu1/0/51 2 N/A -0.8 -0.8 -0.8 -0.8
Hu1/0/51 3 N/A -0.8 -0.8 -0.8 -0.8
Hu1/0/51 4 N/A -0.8 -0.8 -0.8 -0.8

------ On PC side---------------

PC1:

sudo ethtool enp4s0f0
[sudo] password for client:
Settings for enp4s0f0:
Supported ports: [ FIBRE ]
Supported link modes: 25000baseCR/Full
25000baseSR/Full
50000baseCR2/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
50000baseSR2/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None BaseR RS
Advertised link modes: 25000baseCR/Full
50000baseCR2/Full
100000baseCR4/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: None BaseR RS
Speed: 100000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

PC2:

sudo ethtool enp1s0f0
[sudo] password for test:
Settings for enp1s0f0:
Supported ports: [ Backplane ]
Supported link modes: 25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
50000baseSR2/Full
100000baseSR2/Full
100000baseCR2/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None BaseR RS
Advertised link modes: 25000baseCR/Full
25000baseKR/Full
50000baseCR2/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseSR2/Full
100000baseCR2/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: None BaseR RS
Speed: 100000Mb/s
Duplex: Full
Port: None
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

[MHM Cisco World]

show spanning-tree vlan X <<-

I want this 

[jshayne]

This is my default vlan setting:

show vlan

VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Twe1/0/1, Twe1/0/2, Twe1/0/3
Twe1/0/4, Twe1/0/5, Twe1/0/6
Twe1/0/7, Twe1/0/8, Twe1/0/9
Twe1/0/10, Twe1/0/11, Twe1/0/12
Twe1/0/13, Twe1/0/14, Twe1/0/15
Twe1/0/16, Twe1/0/17, Twe1/0/18
Twe1/0/19, Twe1/0/20, Twe1/0/21
Twe1/0/22, Twe1/0/23, Twe1/0/24
Twe1/0/25, Twe1/0/26, Twe1/0/27
Twe1/0/28, Twe1/0/29, Twe1/0/30
Twe1/0/31, Twe1/0/32, Twe1/0/33
Twe1/0/34, Twe1/0/35, Twe1/0/36
Twe1/0/37, Twe1/0/38, Twe1/0/39
Twe1/0/40, Twe1/0/41, Twe1/0/42
Twe1/0/43, Twe1/0/44, Twe1/0/45
Twe1/0/46, Twe1/0/47, Twe1/0/48
Hu1/0/49, Hu1/0/50, Hu1/0/51
Hu1/0/52
2 VLAN0002 active
1002 fddi-default act/unsup

show spanning-tree vlan 1

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 32769
Address 8024.8f2f.4480
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)
Address 8024.8f2f.4480
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Hu1/0/49 Desg FWD 200 128.49 P2p
Hu1/0/51 Desg FWD 200 128.51 P2p

[MHM Cisco World]

show interfaces transceiver properties

Name : ---
Administrative Speed: ---
Administrative Duplex: ---
Administrative Auto-MDIX: off/on <<- Make sure this must be ON
Administrative Power Inline: ---
Operational Speed: ---
Operational Duplex: ---
Operational Auto-MDIX: off/on <<-  this must be ON