Denali 16.3.5b sh aaa servers - Does anybody know what is this ?

Rosa Ladera · ‎06-22-2018

from a ssh session :

ssh -l netadm simson

cat3850-wc2#sh wireless summary

Access Point Summary

               Total    Up    Down
-----------------------------------
802.11a/n          1     1       0
802.11b/g/n        0     0       0
All APs            1     1       0

Client Summary

Current Clients : 1
Excluded Clients: 0
Disabled Clients: 0

cat3850-wc2#sh aaa servers
packet_write_wait: Connection to XXX.XX.XX.XXX port 22: Broken pipe

from another ssh session

ssh -l netadm simson

cat3850-wc2#sh logging
Syslog logging: enabled (0 messages dropped, 170 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 358 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
File logging: disabled
Persistent logging: disabled

No active filter modules.

Trap logging: level debugging, 362 message lines logged
Logging to 147.65.1.96 (udp port 514, audit disabled,
link up),
319 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging Source-Interface: VRF Name:
Vlan14

Log Buffer (4096 bytes):
nged state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/27, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/33, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/35, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/36, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/41, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/43, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/44, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/45, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/7, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/8, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/9, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/10, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/11, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/12, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/13, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/14, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/15, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/16, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/17, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/18, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/19, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/20, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/22, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/24, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/25, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/26, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/34, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/35, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/39, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/40, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/41, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/43, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/0/47, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel20, changed state to up
%HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
%HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
%HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
%HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
%HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
%HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
%SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 147.65.1.96 port 514 started - reconnection
cat3850-wc2#
cat3850-wc2#
cat3850-wc2#
cat3850-wc2#term mon
cat3850-wc2#
%STACKMGR-6-STANDBY_ELECTED:Switch 1 R0/0: stack_mgr: Switch 6 has been elected STANDBY.
%IOSXE_REDUNDANCY-6-PEER: Active detected switch 6 as standby.
cat3850-wc2#sh sw
Switch/Stack Mac Address : cc5a.53d5.de00 - Foreign Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active cc5a.53d5.6080 14 V06 Ready
2 Member c444.a0e9.d580 2 V06 Ready
3 Member cc5a.53e8.7000 4 V06 Ready
4 Member 706b.b920.0b80 6 V06 Ready
5 Member cc5a.53e8.9080 8 V06 Ready
6 Standby cc5a.53e8.9980 10 V06 HA sync in progress
7 Member 0000.0000.0000 0 V06 Removed

%LINK-3-UPDOWN: Interface GigabitEthernet7/1/1, changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet7/1/2, changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet7/1/3, changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet7/1/4, changed state to down
%LINK-3-UPDOWN: Interface TenGigabitEthernet7/1/1, changed state to down
%LINK-3-UPDOWN: Interface TenGigabitEthernet7/1/2, changed state to down
%LINK-3-UPDOWN: Interface TenGigabitEthernet7/1/3, changed state to down
%LINK-3-UPDOWN: Interface TenGigabitEthernet7/1/4, changed state to down
cat3850-wc2#
%HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 7: EMP_RELAY: Channel UP!
%HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
%HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
cat3850-wc2#
%STACKMGR-1-STACK_LINK_CHANGE:Switch 7 R0/0: stack_mgr: Stack port 1 on switch 7 is down
%STACKMGR-1-STACK_LINK_CHANGE:Switch 7 R0/0: stack_mgr: Stack port 2 on switch 7 is down
%STACKMGR-1-STACK_LINK_CHANGE:Switch 7 R0/0: stack_mgr: Stack port 1 on switch 7 is up
%STACKMGR-1-STACK_LINK_CHANGE:Switch 7 R0/0: stack_mgr: Stack port 2 on switch 7 is up
cat3850-wc2#
%STACKMGR-6-SWITCH_ADDED:Switch 7 R0/0: stack_mgr: Switch 7 has been added to the stack.
%STACKMGR-6-SWITCH_ADDED:Switch 7 R0/0: stack_mgr: Switch 7 has been added to the stack.
cat3850-wc2#
%PLATFORM_PM-6-FRULINK_INSERTED: 2x1G 2x10G uplink module inserted in the switch 7 slot 1
cat3850-wc2#
%PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te7/1/4
cat3850-wc2#
%LINK-3-UPDOWN: Interface TenGigabitEthernet7/1/4, changed state to up
cat3850-wc2#
%LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet7/1/4, changed state to up
cat3850-wc2#
%SIF_MGR-1-FAULTY_CABLE:Switch 7 R0/0: sif_mgr: High hardware interrupt seen on switch 7
cat3850-wc2#
cat3850-wc2#sh sw
Switch/Stack Mac Address : cc5a.53d5.de00 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active cc5a.53d5.6080 14 V06 Ready
2 Member c444.a0e9.d580 2 V06 Ready
3 Member cc5a.53e8.7000 4 V06 Ready
4 Member 706b.b920.0b80 6 V06 Ready
5 Member cc5a.53e8.9080 8 V06 Ready
6 Standby cc5a.53e8.9980 10 V06 Ready
7 Member cc5a.53d5.de00 12 V06 Ready

Georg Pauwen · ‎06-23-2018

Hello,

what part of this output is your question about ?

Rosa Ladera · ‎06-25-2018

A reboot

TOM FRANCHINA · ‎11-08-2018

Rosa,

We were getting this:

12966: Oct 16 12:38:36: %SIF_MGR-1-FAULTY_CABLE:Switch 3 R0/0: sif_mgr: High hardware interrupt seen on switch 3 12967: Oct 16 12:38:37: %SIF_MGR-1-FAULTY_CABLE:Switch 4 R0/0: sif_mgr: High hardware interrupt seen on switch 4

Then the switch would reboot. Check the date on your stacking cables. If they were manufactured in 2015.... we have found that Denali 16.3.5b we reboot the stack when these cables are in the mix.

We are working with TAC on this.... no solution but to replace the cable with a new cable of 2017 or newer

Thanks,

Tom

Rosa Ladeira · ‎11-08-2018

All stack cables date 2017.

georgehewittuk1 · ‎11-08-2018

I mean looks like you switch stack is having issues so I would priortise looking into that.

-Not sure on your model but run a show switch stack-ports summary (do a show switch and see your options.. if need be.)

- There should be crash logs generated which you could analyse showing root cause of it.

Rosa Ladeira · ‎11-09-2018

cat3850-wc2#sh sw detail
Switch/Stack Mac Address : cc5a.53d5.de00 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
1 Standby cc5a.53d5.6080 14 V06 Ready
2 Member c444.a0e9.d580 2 V06 Ready
3 Member cc5a.53e8.7000 4 V06 Ready
4 Member 706b.b920.0b80 6 V06 Ready
5 Member cc5a.53e8.9080 8 V06 Ready
6 Member cc5a.53e8.9980 10 V06 Ready
*7 Active cc5a.53d5.de00 12 V06 Ready

Stack Port Status Neighbors
Switch# Port 1 Port 2 Port 1 Port 2
--------------------------------------------------------
1 OK OK 7 2
2 OK OK 1 3
3 OK OK 2 4
4 OK OK 3 5
5 OK OK 4 6
6 OK OK 5 7
7 OK OK 6 1

cat3850-wc2#

Rosa Ladeira · ‎11-09-2018

cat3850-wc2#show switch stack-ports summary
Sw#/Port# Port Status Neighbor Cable Length Link OK Link Active Sync OK #Changes to LinkOK In Loopback
-------------------------------------------------------------------------------------------------------------------
1/1 OK 7 50cm Yes Yes Yes 1 No
1/2 OK 2 50cm Yes Yes Yes 1 No
2/1 OK 1 50cm Yes Yes Yes 1 No
2/2 OK 3 50cm Yes Yes Yes 1 No
3/1 OK 2 50cm Yes Yes Yes 1 No
3/2 OK 4 50cm Yes Yes Yes 1 No
4/1 OK 3 50cm Yes Yes Yes 1 No
4/2 OK 5 50cm Yes Yes Yes 1 No
5/1 OK 4 50cm Yes Yes Yes 1 No
5/2 OK 6 50cm Yes Yes Yes 1 No
6/1 OK 5 50cm Yes Yes Yes 1 No
6/2 OK 7 50cm Yes Yes Yes 1 No
7/1 OK 6 50cm Yes Yes Yes 1 No
7/2 OK 1 50cm Yes Yes Yes 1 No

Rosa Ladeira · ‎11-09-2018

#dir
Directory of flash:/

62095 -rw- 2097152 Nov 1 2018 14:48:06 -03:00 nvram_config
62083 -rw- 15954556 Feb 26 2018 09:24:36 -03:00 cat3k_caa-guestshell.16.03.05b.SPA.pkg
62084 -rw- 22301472 Feb 26 2018 09:24:33 -03:00 cat3k_caa-rpbase.16.03.05b.SPA.pkg
62085 -rw- 266035828 Feb 26 2018 09:24:36 -03:00 cat3k_caa-rpcore.16.03.05b.SPA.pkg
15521 -rw- 9089660 Feb 26 2018 09:24:35 -03:00 cat3k_caa-srdriver.16.03.05b.SPA.pkg
15522 -rw- 212812400 Feb 26 2018 09:24:35 -03:00 cat3k_caa-wcm.16.03.05b.SPA.pkg
31041 -rw- 13423224 Feb 26 2018 09:24:35 -03:00 cat3k_caa-webui.16.03.05b.SPA.pkg
31042 -rw- 4748 Feb 26 2018 09:24:59 -03:00 packages.conf
31043 -rw- 1110 Nov 1 2018 11:37:13 -03:00 bootloader_evt_handle.log
77601 drwx 4096 Aug 27 2018 13:00:11 -03:00 core
85361 drwx 4096 Nov 1 2018 11:41:27 -03:00 .prst_sync
7762 drwx 4096 Feb 26 2018 09:27:34 -03:00 .rollback_timer
31044 drwx 4096 Nov 1 2018 11:37:27 -03:00 dc_profile_dir
31046 drwx 4096 Feb 26 2018 09:27:42 -03:00 gs_script
31047 -rw- 65301 Nov 1 2018 11:37:44 -03:00 memleak.tcl
31048 drwx 4096 Feb 26 2018 09:28:12 -03:00 .installer
31049 -rw- 2097152 Nov 1 2018 14:48:07 -03:00 nvram_config_bkup
62086 -rw- 2728 Nov 1 2018 14:10:18 -03:00 vlan.dat
77603 drwx 4096 Feb 26 2018 13:57:48 -03:00 onep
62087 -rw- 4682 Jul 26 2018 16:50:43 -03:00 webauth_login.html-25-07
62089 -rw- 16378 Jul 26 2018 16:21:29 -03:00 web_auth_image.jpg
62090 -rw- 231232 Jul 26 2018 16:54:16 -03:00 webauth_login.html
62091 -rw- 2574 Jul 27 2018 13:00:53 -03:00 web_auth_aup.html
62088 -rw- 2574 Jul 26 2018 16:51:45 -03:00 web_auth_aup.html-25-07
62092 -rw- 4682 Jul 27 2018 15:31:34 -03:00 webauth_login-25-07.html
62093 -rw- 4718 Jul 27 2018 15:36:52 -03:00 login-27-07.html
62094 -rw- 4718 Jul 27 2018 15:37:47 -03:00 webauth_login-27-07.html

1624104960 bytes total (993935360 bytes free)
cat3850-wc2#cd core
cat3850-wc2#dir
Directory of flash:/core/

11 drwx 4096 Feb 26 2018 09:27:34 -03:00 modules
77602 -rw- 1 Nov 9 2018 08:03:14 -03:00 .callhome

1624104960 bytes total (993935360 bytes free)
cat3850-wc2#cd mod
cat3850-wc2#cd modules
cat3850-wc2#dir
Directory of flash:/core/modules/

No files in directory

1624104960 bytes total (993935360 bytes free)
cat3850-wc2#

Rosa Ladeira · ‎11-09-2018

cat3850-wc2#more bootloader_evt_handle.log
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...
boothelper event handler starting...