Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- Cisco Community
- Technology and Support
- Networking
- Switching
- Re: Nexus 9396px very strange routing issue
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
823
Views
0
Helpful
5
Replies
Nexus 9396px very strange routing issue
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-02-2018 07:45 PM - edited 03-08-2019 02:06 PM
We got new connection from ISP 40GE link (4x10GE in LACP) I have created SVI vlan3 and configured IP and set default route but very strange thing happened
Nexus 9k switche (L3)
interface Vlan3
description HAND_OF
no shutdown
ip address 74.xx.xx.2/30
Default route
ip route 0.0.0.0/0 74.xx.xx.1
Now lets ping GW, all looks good
sw1# ping 74.xx.xx.1
PING 74.xx.xx.1 (74.xx.xx.1): 56 data bytes
64 bytes from 74.xx.xx.1: icmp_seq=0 ttl=63 time=1.294 ms
64 bytes from 74.xx.xx.1: icmp_seq=1 ttl=63 time=1.11 ms
64 bytes from 74.xx.xx.1: icmp_seq=2 ttl=63 time=0.88 ms
64 bytes from 74.xx.xx.1: icmp_seq=3 ttl=63 time=0.765 ms
64 bytes from 74.xx.xx.1: icmp_seq=4 ttl=63 time=1.889 ms
Lets ping google DNS 8.8.4.4
sw1# ping 8.8.4.4 PING 8.8.4.4 (8.8.4.4): 56 data bytes Request 0 timed out Request 1 timed out Request 2 timed out Request 3 timed out Request 4 timed out
lets ping other google DNS 8.8.8.8 (WTH! what is this DUP! ?)
sw# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=56 time=9.197 ms
64 bytes from 8.8.8.8: icmp_seq=0 ttl=56 time=9.454 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=12.586 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=12.767 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=8.426 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=8.578 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=9.919 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=10.049 ms (DUP!)
64 bytes from 8.8.8.8: icmp_seq=4 ttl=56 time=8.414 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=56 time=8.477 ms (DUP!)
Lets ping any other public IP
sw1# ping 172.217.12.174
PING 172.217.12.174 (172.217.12.174): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out
I have called ISP and they also spend 8 hours to debug and won't able to see anything they clear arp, route whatever possible but no luck.
Finally i reload my Nexus 9k switch but still same result its not working, what do you think guys what is going on?
NOTE:
from outside world i can ping my router outside interface 74.xx.xx.2 so that is also very strange :(
Show version:
Software BIOS: version 07.41 NXOS: version 7.0(3)I4(7) BIOS compile time: 10/12/2015 NXOS image file is: bootflash:///nxos.7.0.3.I4.7.bin NXOS compile time: 6/28/2017 14:00:00 [06/28/2017 21:53:29] Hardware cisco Nexus9000 C9396PX Chassis Intel(R) Core(TM) i3- CPU @ 2.50GHz with 16401396 kB of memory. Processor Board ID SAL2006Y9CQ Device name: sw-core1-bns003-2-7 bootflash: 51496280 kB Kernel uptime is 0 day(s), 6 hour(s), 45 minute(s), 24 second(s) Last reset at 843568 usecs after Fri Mar 2 21:09:15 2018 Reason: Reset Requested by CLI command reload System version: 7.0(3)I4(7) Service:
Labels:
- Labels:
-
Other Switching
5 Replies 5
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-03-2018 05:29 AM
Hi Satish,
This is very strange indeed.
We need to understand quite a lot about your situation, so I would like to ask you to perform a rather extensive set of tests for me:
First, please be so kind to post the following complete outputs from your device:
terminal length 0
show logging log
show ip arp
show ip route 74.xx.xx.1 show ip route 8.8.8.8 show ip route 8.8.4.4 show ip route 172.217.12.174 show forwarding route 74.xx.xx.1 detail show forwarding route 8.8.8.8 detail show forwarding route 8.8.4.4 detail show forwarding route 172.217.12.174 detail show forwarding adjacency 74.xx.xx.1 detail show system internal forwarding route 0.0.0.0/0 detail
Second, I would like to have a close look on the traffic going in and out of your CPU. Please start this command:
ethanalyzer local interface inband limit-captured-frames 0 write bootflash:support.pcap
This command will record all the traffic received and sent by the CPU of your switch into the "support.pcap" file in the bootflash: filesystem.
Keep this command running, open another session to the switch, and ping the offending locations:
ping 8.8.8.8 ping 8.8.4.4 ping 172.217.12.174
After you're done with the pings, stop the ethanalyzer by pressing <Ctrl><C> and please post all the outputs, plus the support.pcap file from the bootflash: filesystem to this case.
Thanks!
Best regards,
Peter
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-03-2018 07:06 AM - edited 03-03-2018 07:11 AM
Thanks for showing Interest. Here is the full output and i am sending directly to you on IM to not expose original IPs, if anyone need pcap just let me know i will send it to them (hope you guys understand)
here is the output.
swt-core1# show logging log
2018 Mar 3 04:29:44 swt-core1 %USER-0-SYSTEM_MSG: before access to bkout_cfg - clis
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-PS_FOUND: Power supply 1 found (Serial number DCB1951X1IL)
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number DCB1951X1IL)
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-PS_STATUS: PowerSupply 1 current-status is PS_OK
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-PS_FOUND: Power supply 2 found (Serial number DCB1951X1G3)
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number DCB1951X1G3)
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-PS_STATUS: PowerSupply 2 current-status is PS_OK
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-FAN_DETECT: Fan module 1 (Serial number ) Fan1(sys_fan1) detected
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-FAN_STATUS: Fan module 1 (Serial number ) Fan1(sys_fan1) current-status is FAN_OK
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-2-FANMOD_FAN_OK: Fan module 1 (Fan1(sys_fan1) fan) ok
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-FAN_DETECT: Fan module 2 (Serial number ) Fan2(sys_fan2) detected
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-FAN_STATUS: Fan module 2 (Serial number ) Fan2(sys_fan2) current-status is FAN_OK
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-2-FANMOD_FAN_OK: Fan module 2 (Fan2(sys_fan2) fan) ok
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-FAN_DETECT: Fan module 3 (Serial number ) Fan3(sys_fan3) detected
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-5-FAN_STATUS: Fan module 3 (Serial number ) Fan3(sys_fan3) current-status is FAN_OK
2018 Mar 3 04:29:46 swt-core1 %PLATFORM-2-FANMOD_FAN_OK: Fan module 3 (Fan3(sys_fan3) fan) ok
2018 Mar 3 04:29:46 swt-core1 Mar 3 04:29:46 %KERN-3-SYSTEM_MSG: [ 0.229971] Allocating netns hash table - kernel
2018 Mar 3 04:29:46 swt-core1 %USER-2-SYSTEM_MSG: <<%USBHSD-2-MOUNT>> logflash: online - usbhsd
2018 Mar 3 04:29:46 swt-core1 Mar 3 04:29:46 %KERN-3-SYSTEM_MSG: [ 6.492066] Total windows = 3 - kernel
2018 Mar 3 04:29:46 swt-core1 Mar 3 04:29:46 %KERN-2-SYSTEM_MSG: [ 9.211535] hwport mode=6 - kernel
2018 Mar 3 04:29:46 swt-core1 Mar 3 04:29:46 %KERN-3-SYSTEM_MSG: [ 22.676612] fcfwd: major=0 res=233. Registered with unlocked/compat ioctl - kernel
2018 Mar 3 04:29:48 swt-core1 %VDC_MGR-5-VDC_STATE_CHANGE: vdc 1 state changed to create pending
2018 Mar 3 04:29:49 swt-core1 %DAEMON-2-SYSTEM_MSG: <<%ASCII-CFG-2-CONF_CONTROL>> Binary restore - ascii-cfg[13642]
2018 Mar 3 04:29:50 swt-core1 netstack: Registration with cli server complete
2018 Mar 3 04:30:03 swt-core1 %USER-2-SYSTEM_MSG: ssnmgr_app_init called on ssnmgr up - aclmgr
2018 Mar 3 04:30:09 swt-core1 %USER-0-SYSTEM_MSG: end of default policer - copp
2018 Mar 3 04:30:11 swt-core1 %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary
2018 Mar 3 04:30:11 swt-core1 %CARDCLIENT-2-FPGA_BOOT_PRIMARY: MIFPGA booted from Primary
2018 Mar 3 04:30:11 swt-core1 %CARDCLIENT-2-FPGA_BOOT_PRIMARY: GEM_MIFPGA booted from Primary
2018 Mar 3 04:30:11 swt-core1 %MODULE-5-ACTIVE_SUP_OK: Supervisor 1 is active (Serial number: SAL2006Y9CQ)
2018 Mar 3 04:30:11 swt-core1 %VDC_MGR-5-VDC_STATE_CHANGE: vdc 1 state changed to create in progress
2018 Mar 3 04:30:11 swt-core1 %PLATFORM-5-MOD_STATUS: SUP sub Module 1 current-status is MOD_STATUS_ONLINE/OK
2018 Mar 3 04:30:11 swt-core1 %IM-5-IM_INTF_STATE: mgmt0 is DOWN in vdc 1
2018 Mar 3 04:30:11 swt-core1 %ETH_PORT_CHANNEL-5-CREATED: port-channel1 created
2018 Mar 3 04:30:11 swt-core1 %USER-2-SYSTEM_MSG: IPV6 Netlink thread init successful - icmpv6
2018 Mar 3 04:30:11 swt-core1 Mar 3 04:30:11 %KERN-3-SYSTEM_MSG: [ 57.091059] ICMPV6 daemon PID = 14913 - kernel
2018 Mar 3 04:30:11 swt-core1 %VDC_MGR-5-VDC_STATE_CHANGE: vdc 1 state changed to active
2018 Mar 3 04:30:11 swt-core1 %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online
2018 Mar 3 04:30:11 swt-core1 %VDC_MGR-5-VDC_HOSTNAME_CHANGE: vdc 1 hostname changed to swt-core1
2018 Mar 3 04:30:12 swt-core1 last message repeated 1 time
2018 Mar 3 04:30:12 swt-core1 %ETHPORT-5-IF_UP: Interface loopback0 is up
2018 Mar 3 04:30:13 swt-core1 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.14891
2018 Mar 3 04:30:13 swt-core1 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.14725
2018 Mar 3 04:30:14 swt-core1 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.14891
2018 Mar 3 04:30:18 swt-core1 %IM-5-IM_INTF_STATE: mgmt0 is UP in vdc 1
2018 Mar 3 04:30:53 swt-core1 %VDC_MGR-5-VDC_STATE_CHANGE: vdc 1 state changed to updating
2018 Mar 3 04:30:53 swt-core1 %VDC_MGR-5-VDC_STATE_CHANGE: vdc 1 state changed to active
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/1 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/2 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/3 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/4 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/5 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/6 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/7 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/8 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/9 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/10 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/11 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/12 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/13 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/14 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/15 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/16 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/17 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/18 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/19 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/20 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/21 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/22 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/23 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/24 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/25 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/26 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/27 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/28 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/29 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/30 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/31 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/32 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/33 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/34 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/35 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/36 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/37 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/38 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/39 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/40 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/41 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/42 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/43 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/44 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/45 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/46 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/47 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/48 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet2/1 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet2/2 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet2/3 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet2/4 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet2/5 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet2/6 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet2/7 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet2/8 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet2/9 is down (None)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet2/10 is down (Administratively down)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet2/11 is down (Administratively down)
2018 Mar 3 04:30:55 swt-core1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet2/12 is down (Administratively down)
2018 Mar 3 04:30:55 swt-core1 %PLATFORM-5-MOD_STATUS: LC sub Module 1 current-status is MOD_STATUS_ONLINE/OK
2018 Mar 3 04:30:56 swt-core1 %MODULE-5-MOD_OK: Module 1 is online (Serial number: SAL2006Y9CQ)
2018 Mar 3 04:30:56 swt-core1 %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/45, hardware type changed to 10G
2018 Mar 3 04:30:56 swt-core1 %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/46, hardware type changed to 10G
2018 Mar 3 04:30:56 swt-core1 %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/47, hardware type changed to 10G
2018 Mar 3 04:30:56 swt-core1 %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/48, hardware type changed to 10G
2018 Mar 3 04:30:57 swt-core1 %ETHPORT-5-SPEED: Interface Ethernet1/46, operational speed changed to 10 Gbps
2018 Mar 3 04:30:57 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/46, operational duplex mode changed to Full
2018 Mar 3 04:30:57 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/46, operational Receive Flow Control state changed to off
2018 Mar 3 04:30:57 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/46, operational Transmit Flow Control state changed to off
2018 Mar 3 04:30:58 swt-core1 Mar 3 04:30:57 %KERN-3-SYSTEM_MSG: [ 103.002038] registered device ps-inb - kernel
2018 Mar 3 04:30:57 swt-core1 %ETHPORT-5-SPEED: Interface port-channel1, operational speed changed to 10 Gbps
2018 Mar 3 04:30:57 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface port-channel1, operational duplex mode changed to Full
2018 Mar 3 04:30:57 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel1, operational Receive Flow Control state changed to off
2018 Mar 3 04:30:57 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel1, operational Transmit Flow Control state changed to off
2018 Mar 3 04:30:58 swt-core1 %ASCII-CFG-2-CONF_CONTROL: System ready
2018 Mar 3 04:30:58 swt-core1 %ETHPORT-5-SPEED: Interface Ethernet1/45, operational speed changed to 10 Gbps
2018 Mar 3 04:30:58 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/45, operational duplex mode changed to Full
2018 Mar 3 04:30:58 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/45, operational Receive Flow Control state changed to off
2018 Mar 3 04:30:58 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/45, operational Transmit Flow Control state changed to off
2018 Mar 3 04:30:58 swt-core1 %ETHPORT-5-SPEED: Interface Ethernet1/47, operational speed changed to 10 Gbps
2018 Mar 3 04:30:58 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/47, operational duplex mode changed to Full
2018 Mar 3 04:30:58 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/47, operational Receive Flow Control state changed to off
2018 Mar 3 04:30:58 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/47, operational Transmit Flow Control state changed to off
2018 Mar 3 04:30:59 swt-core1 %ETHPORT-5-SPEED: Interface Ethernet1/48, operational speed changed to 10 Gbps
2018 Mar 3 04:30:59 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/48, operational duplex mode changed to Full
2018 Mar 3 04:30:59 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/48, operational Receive Flow Control state changed to off
2018 Mar 3 04:30:59 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/48, operational Transmit Flow Control state changed to off
2018 Mar 3 04:31:02 swt-core1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/48 is up
2018 Mar 3 04:31:02 swt-core1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1: first operational port changed from none to Ethernet1/48
2018 Mar 3 04:31:02 swt-core1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/45 is up
2018 Mar 3 04:31:02 swt-core1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/46 is up
2018 Mar 3 04:31:02 swt-core1 %ETHPORT-5-IF_UP: Interface Ethernet1/48 is up in mode access
2018 Mar 3 04:31:02 swt-core1 %ETHPORT-5-IF_UP: Interface port-channel1 is up in mode access
2018 Mar 3 04:31:02 swt-core1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/47 is up
2018 Mar 3 04:31:02 swt-core1 %ETHPORT-5-IF_UP: Interface Ethernet1/45 is up in mode access
2018 Mar 3 04:31:02 swt-core1 %ETHPORT-5-IF_UP: Interface Ethernet1/46 is up in mode access
2018 Mar 3 04:31:02 swt-core1 %ETHPORT-5-IF_UP: Interface Ethernet1/47 is up in mode access
2018 Mar 3 04:34:19 swt-core1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface port-channel1 is down(Config change)
2018 Mar 3 04:34:19 swt-core1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: Ethernet1/48 is down
2018 Mar 3 04:34:19 swt-core1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: Ethernet1/45 is down
2018 Mar 3 04:34:19 swt-core1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: Ethernet1/46 is down
2018 Mar 3 04:34:19 swt-core1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: Ethernet1/47 is down
2018 Mar 3 04:34:20 swt-core1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1: first operational port changed from Ethernet1/48 to none
2018 Mar 3 04:34:20 swt-core1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface port-channel1 is down (Administratively down)
2018 Mar 3 04:34:20 swt-core1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/46 is down(Config change)
2018 Mar 3 04:34:20 swt-core1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/45 is down(Config change)
2018 Mar 3 04:34:20 swt-core1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/47 is down(Config change)
2018 Mar 3 04:34:20 swt-core1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/48 is down(Config change)
2018 Mar 3 04:34:20 swt-core1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/46 is down (Channel admin down)
2018 Mar 3 04:34:20 swt-core1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/45 is down (Channel admin down)
2018 Mar 3 04:34:20 swt-core1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/47 is down (Channel admin down)
2018 Mar 3 04:34:20 swt-core1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/48 is down (Channel admin down)
2018 Mar 3 04:34:21 swt-core1 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on 10.5.0.100@pts/0
2018 Mar 3 04:34:53 swt-core1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/45 is down (Administratively down)
2018 Mar 3 04:34:53 swt-core1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/46 is down (Administratively down)
2018 Mar 3 04:34:53 swt-core1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/47 is down (Administratively down)
2018 Mar 3 04:34:53 swt-core1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/48 is down (Administratively down)
2018 Mar 3 04:35:03 swt-core1 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/45 is admin up .
2018 Mar 3 04:35:03 swt-core1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/45 is down (Channel admin down)
2018 Mar 3 04:35:03 swt-core1 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/46 is admin up .
2018 Mar 3 04:35:03 swt-core1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/46 is down (Channel admin down)
2018 Mar 3 04:35:03 swt-core1 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/47 is admin up .
2018 Mar 3 04:35:03 swt-core1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/47 is down (Channel admin down)
2018 Mar 3 04:35:03 swt-core1 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/48 is admin up .
2018 Mar 3 04:35:03 swt-core1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/48 is down (Channel admin down)
2018 Mar 3 04:35:04 swt-core1 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on 10.5.0.100@pts/0
2018 Mar 3 04:35:10 swt-core1 %ETHPORT-5-IF_ADMIN_UP: Interface port-channel1 is admin up .
2018 Mar 3 04:35:10 swt-core1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operational members)
2018 Mar 3 04:35:11 swt-core1 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on 10.5.0.100@pts/0
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-SPEED: Interface Ethernet1/46, operational speed changed to 10 Gbps
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/46, operational duplex mode changed to Full
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/46, operational Receive Flow Control state changed to off
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/46, operational Transmit Flow Control state changed to off
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-SPEED: Interface port-channel1, operational speed changed to 10 Gbps
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface port-channel1, operational duplex mode changed to Full
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel1, operational Receive Flow Control state changed to off
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel1, operational Transmit Flow Control state changed to off
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-SPEED: Interface Ethernet1/45, operational speed changed to 10 Gbps
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/45, operational duplex mode changed to Full
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/45, operational Receive Flow Control state changed to off
2018 Mar 3 04:35:11 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/45, operational Transmit Flow Control state changed to off
2018 Mar 3 04:35:12 swt-core1 %ETHPORT-5-SPEED: Interface Ethernet1/47, operational speed changed to 10 Gbps
2018 Mar 3 04:35:12 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/47, operational duplex mode changed to Full
2018 Mar 3 04:35:12 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/47, operational Receive Flow Control state changed to off
2018 Mar 3 04:35:12 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/47, operational Transmit Flow Control state changed to off
2018 Mar 3 04:35:12 swt-core1 %ETHPORT-5-SPEED: Interface Ethernet1/48, operational speed changed to 10 Gbps
2018 Mar 3 04:35:12 swt-core1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/48, operational duplex mode changed to Full
2018 Mar 3 04:35:12 swt-core1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/48, operational Receive Flow Control state changed to off
2018 Mar 3 04:35:12 swt-core1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/48, operational Transmit Flow Control state changed to off
2018 Mar 3 04:35:15 swt-core1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/45 is up
2018 Mar 3 04:35:15 swt-core1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/46 is up
2018 Mar 3 04:35:15 swt-core1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/48 is up
2018 Mar 3 04:35:15 swt-core1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/47 is up
2018 Mar 3 04:35:15 swt-core1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1: first operational port changed from none to Ethernet1/45
2018 Mar 3 04:35:15 swt-core1 %ETHPORT-5-IF_UP: Interface Ethernet1/45 is up in mode access
2018 Mar 3 04:35:15 swt-core1 %ETHPORT-5-IF_UP: Interface port-channel1 is up in mode access
2018 Mar 3 04:35:15 swt-core1 %ETHPORT-5-IF_UP: Interface Ethernet1/46 is up in mode access
2018 Mar 3 04:35:15 swt-core1 %ETHPORT-5-IF_UP: Interface Ethernet1/47 is up in mode access
2018 Mar 3 04:35:15 swt-core1 %ETHPORT-5-IF_UP: Interface Ethernet1/48 is up in mode access
2018 Mar 3 04:37:43 swt-core1 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on 10.5.0.100@pts/0
2018 Mar 3 05:31:57 swt-core1 last message repeated 1 time
2018 Mar 3 05:31:57 swt-core1 %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP private mode packet. Drop count: 1 - ntpd[14661]
2018 Mar 3 07:22:06 swt-core1 %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from 5.101.40.10 - dcos_sshd[21513]
2018 Mar 3 07:46:35 swt-core1 %DAEMON-3-SYSTEM_MSG: NTP Receive dropping message: Received NTP private mode packet. Drop count: 2 - ntpd[14661]
swt-core1# show ip arp
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies D - Static Adjacencies attached to down interface
IP ARP Table for context default
Total number of entries: 1
Address Age MAC Address Interface
74.xx.xx.1 00:00:25 cce1.7fc6.00c5 Vlan3
swt-core1# show ip route 74.xx.xx.2
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
74.xx.xx.2/32, ubest/mbest: 1/0, attached
*via 74.xx.xx.2, Vlan3, [0/0], 10:19:27, local
swt-core1# show ip route 8.8.8.8
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
0.0.0.0/0, ubest/mbest: 1/0
*via 74.xx.xx.1, [1/0], 10:19:35, static
swt-core1# show ip route 8.8.4.4
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
0.0.0.0/0, ubest/mbest: 1/0
*via 74.xx.xx.1, [1/0], 10:19:43, static
swt-core1# show ip route 172.217.12.174
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
0.0.0.0/0, ubest/mbest: 1/0
*via 74.xx.xx.1, [1/0], 10:19:56, static
swt-core1# show forwarding route 74.xx.xx.1 detail
slot 1
=======
Prefix 74.xx.xx.1/32, No of paths: 1, Update time: Sat Mar 3 04:35:45 2018
74.xx.xx.1 Vlan3
swt-core1# show forwarding route 8.8.8.8 detail
slot 1
=======
Prefix 0.0.0.0/0, No of paths: 1, Update time: Sat Mar 3 04:35:45 2018
Vobj id: 2 Partial Install: No
74.xx.xx.1 Vlan3
swt-core1# show forwarding route 8.8.4.4 detail
slot 1
=======
Prefix 0.0.0.0/0, No of paths: 1, Update time: Sat Mar 3 04:35:45 2018
Vobj id: 2 Partial Install: No
74.xx.xx.1 Vlan3
swt-core1# show forwarding route 172.217.12.174 detail
slot 1
=======
Prefix 0.0.0.0/0, No of paths: 1, Update time: Sat Mar 3 04:35:45 2018
Vobj id: 2 Partial Install: No
74.xx.xx.1 Vlan3
swt-core1# show forwarding adjacency 74.xx.xx.1 detail
slot 1
=======
IPv4 adjacency information
next-hop rewrite info interface Origin AS Peer AS Neighbor
-------------- --------------- ------------- ---------- --------- --------------
74.xx.xx.1 cce1.7fc6.00c5 Vlan3
swt-core1# show system internal forwarding route 0.0.0.0/0 detail
slot 1
=======
0.0.0.0/0 , Vlan3
Dev: 1 , RPF Enabled: N , VPN: 1
AdjIdx: 0x186a6, LIF: Vlan3 (0x3 ),
DMAC: cce1.7fc6.00c5 SMAC: 00c8.8b85.2c05
swt-core1#
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-03-2018 12:08 PM - edited 03-03-2018 12:10 PM
Hi Satish,
I have reviewed the PCAP file you have sent me via private message, and the outputs attached to this post.
I do not see any outstanding issue in the outputs here. All the routing including the hardware programming I could verify so far correctly points to the proper next hop and its MAC address.
Just to confirm, would you be so kind to pull one additional output for me?
show system internal forwarding adjacency entry 0x186a6 detail
From the Wireshark capture, I can conclude this:
- Your switch is properly sending all outbound IP traffic to the next hop identified with the MAC address cc:e1:7f:c6:00:c5. This is correct and entirely as expect. Also, there are no duplicate packets being sent out.
- Pings returning from 8.8.8.8 are returning in two copies each - that is why you see the DUP! alert in your ping. Checking their IP header, they must be duplicated by their sender - each packet from 8.8.8.8 carries unique IP identification number, so they cannot be duplicated on the path; if they were, their IP identification number would have been the same. I do not believe that your switch has anything to do with this duplication.
- Packets sent from your switch to other destinations (8.8.4.4, 172.217.12.174) are properly sent to the next hop but there is no response.
From this, I have to conclude that there is nothing that would point to a problem with your switch. I believe that this is up to your ISP to find a problem in his network.
What I would suggest:
- Have the ISP perform a packet capture on 74.xx.xx.1 and share the results with you. If that packet capture shows the pings for 8.8.4.4 and 172.217.12.174 properly coming in to 74.xx.xx.1 in just a single copy, while no return packets are passed back to your switch then this is very clearly a problem inside the ISP's cloud. Once 74.xx.xx.1 receives packets from you properly, it makes its own autonomous routing decisions which your switch cannot influence anymore.
- If the ISP cannot perform a packet capture for whatever reasons, let him at least install accounting ACLs on the interface that goes back to your switch that would match 8.8.8.8, 8.8.4.4, and 172.217.12.174 in particular, and permit everything, and perform the ping tests again. If the ACLs show that packets from you have been properly received in exactly the count they have been sent in, and not duplicated, while there have been no responses sent back to you, it is again a clear indication of the problem in the ISP's network.
Sorry for not being able to solve your problem here but based on what I could see so far, I do not have a reason to point my finger to your switch as the source of problems.
Best regards,
Peter
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-03-2018 12:49 PM
Peter,
Thanks you so much analysis, I will send you all required info,
This is interesting that I can ping my switch outside interfec from outside world so it's indication anyone can reach to my switch public interface.
My ISP side they have juniper switch and I will see if they can perform task what you requested.
I have one more option which I can put other same switch and try if that fix or give some clue.
This is very strange issue.
Thanks you so much analysis, I will send you all required info,
This is interesting that I can ping my switch outside interfec from outside world so it's indication anyone can reach to my switch public interface.
My ISP side they have juniper switch and I will see if they can perform task what you requested.
I have one more option which I can put other same switch and try if that fix or give some clue.
This is very strange issue.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-03-2018 07:08 PM
Here is the requested output:
swt-core1# show system internal forwarding adjacency entry 0x186a6 detail slot 1 ======= Device: 1 Index: 0x186a6 dmac: cce1.7fc6.00c5 smac: 00c8.8b85.2c05 e-lif: 0x3
Learn, share, save
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide
Customers Also Viewed These Support Documents
