Mysterious Monitoring Gaps on Switch 4 Times a Day

[martinezaw]

I have an aggregation 3750X stack switch at one of our schools... Multiple IDFs connect to it. It then connects out to our core switch There are I believe 7 switches in the stack.

On our monitoring software, 4 times a day, close to the same times  roughly 1AM  7AM 1PM and 7PM. This is just a school so it's not like there are some backups going on. No servers are at this school... Just regular user traffic.

I do notice that the Memory I/O stays around 78%. Could that be part of the problem. What could be happening on an interval like that???

XXXX-MDF-Stack#sh mem
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
 Address      Bytes     Prev     Next Ref     PrevF    NextF Alloc PC  what
Processor    46E5A34   179092736    62550620   116542116   113674304    94536284
      I/O    E000000    16777216    13004616     3772600     2111392     2369804
Driver te    2C00000     4194304          44     4194260     4194260     4194260

sh  ver..

XXXX-MDF-Stack uptime is 1 week, 3 days, 8 minutes
System returned to ROM by power-on
System restarted at 17:07:57 EDT Mon Sep 17 2018
System image file is "flash:/c3750e-universalk9-mz.150-1.SE3/c3750e-universalk9-mz.150-1.SE3.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: lanbase
License Type: Default. No valid license found.
Next reload license Level: lanbase

cisco WS-C3750X-48P (PowerPC405) processor (revision A0) with 262144K bytes of memory.
Processor board ID FDO1526P128
Last reset from power-on
6 Virtual Ethernet interfaces
1 FastEthernet interface
364 Gigabit Ethernet interfaces
14 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 30:E4:DB:FC:A8:80
Motherboard assembly number     : 73-12553-05
Motherboard serial number       : FDO152705VE
Model revision number           : A0
Motherboard revision number     : C0
Model number                    : WS-C3750X-48PF-S
Daughterboard assembly number   : 800-32727-01
Daughterboard serial number     : FDO15270278
System serial number            : FDO1526P128
Top Assembly Part Number        : 800-31324-02
Top Assembly Revision Number    : C0
Version ID                      : V02
CLEI Code Number                : COMJZ00ARB
Hardware Board Revision Number  : 0x03


Switch Ports Model              SW Version            SW Image                 
------ ----- -----              ----------            ----------               
     1 54    WS-C3750X-48       15.0(1)SE3            C3750E-UNIVERSALK9-M     
     2 54    WS-C3750X-48       15.0(1)SE3            C3750E-UNIVERSALK9-M     
     3 54    WS-C3750X-48       15.0(1)SE3            C3750E-UNIVERSALK9-M     
*    4 54    WS-C3750X-48P      15.0(1)SE3            C3750E-UNIVERSALK9-M     
     5 54    WS-C3750X-48       15.0(1)SE3            C3750E-UNIVERSALK9-M     
     6 54    WS-C3750X-48P      15.0(1)SE3            C3750E-UNIVERSALK9-M     
     7 54    WS-C3750X-48P      15.0(1)SE3            C3750E-UNIVERSALK9-M     


Switch 01
---------
Switch Uptime                   : 1 week, 3 days, 8 minutes
Base ethernet MAC Address       : 4C:00:82:20:D1:80
Motherboard assembly number     : 73-12552-10
Motherboard serial number       : FDO17220B62
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48T-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO17212L8T
System serial number            : FDO1721H3V9
Top assembly part number        : 800-31326-08
Top assembly revision number    : B0
Version ID                      : V05
CLEI Code Number                : COMJT00ARE
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase


Switch 02
---------
Switch Uptime                   : 1 week, 3 days, 8 minutes
Base ethernet MAC Address       : 4C:00:82:20:CC:00
Motherboard assembly number     : 73-12552-10
Motherboard serial number       : FDO17220B0X
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48T-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO17212Y6V
System serial number            : FDO1721H3V0
Top assembly part number        : 800-31326-08
Top assembly revision number    : B0
Version ID                      : V05
CLEI Code Number                : COMJT00ARE
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase


Switch 03
---------
Switch Uptime                   : 1 week, 3 days, 8 minutes
Base ethernet MAC Address       : 4C:00:82:20:CD:00
Motherboard assembly number     : 73-12552-10
Motherboard serial number       : FDO17220B17
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48T-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO172200HZ
System serial number            : FDO1721H3V7
Top assembly part number        : 800-31326-08
Top assembly revision number    : B0
Version ID                      : V05
CLEI Code Number                : COMJT00ARE
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase


Switch 05
---------
Switch Uptime                   : 1 week, 3 days, 9 minutes
Base ethernet MAC Address       : 4C:00:82:1C:8C:00
Motherboard assembly number     : 73-12552-10
Motherboard serial number       : FDO172130MP
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48T-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO17212YGA
System serial number            : FDO1721Z319
Top assembly part number        : 800-31326-08
Top assembly revision number    : B0
Version ID                      : V05
CLEI Code Number                : COMJT00ARE
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase
          

Switch 06
---------
Switch Uptime                   : 1 week, 3 days, 9 minutes
Base ethernet MAC Address       : B0:00:B4:B1:91:80
Motherboard assembly number     : 73-12553-11
Motherboard serial number       : FDO18071QPQ
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48P-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO180716Z0
System serial number            : FDO1807Z1P8
Top assembly part number        : 800-31324-09
Top assembly revision number    : B0
Version ID                      : V06
CLEI Code Number                : CMMPS00DRA
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase


Switch 07
---------
Switch Uptime                   : 1 week, 3 days, 1 minute
Base ethernet MAC Address       : E0:5F:B9:C2:35:80
Motherboard assembly number     : 73-12553-04
Motherboard serial number       : FDO144919FV
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48PF-L
Daughterboard assembly number   : 800-32727-01
Daughterboard serial number     : FDO14490HAP
System serial number            : FDO1449R05E
Top assembly part number        : 800-31324-01
Top assembly revision number    : A0
Version ID                      : V01
CLEI Code Number                : COMJZ00ARA
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase

[Leo Laohoo]

1. The stack is running 15.0(1)SE train. This is a very bad (and buggy) train. I wouldn't recommend anyone use this.
2. The stack is 7 switches. That's too much.
Regarding the IOS, stick with the latest 12.2(55)SE or 15.0(2)SE with 12.2(55)SE considered as "the" most stable IOS that ever came out of Cisco.
Regarding the "size" of the stack, break it up. Don't go over six (6) switches in a stack because you'll see some funny stuff happening: It can't handle some of the management and the load.

[martinezaw]

I was considering that. We are limited to

---

@Leo Laohoo wrote:
1. The stack is running 15.0(1)SE train. This is a very bad (and buggy) train. I wouldn't recommend anyone use this.
2. The stack is 7 switches. That's too much.
Regarding the IOS, stick with the latest 12.2(55)SE or 15.0(2)SE with 12.2(55)SE considered as "the" most stable IOS that ever came out of Cisco.
Regarding the "size" of the stack, break it up. Don't go over six (6) switches in a stack because you'll see some funny stuff happening: It can't handle some of the management and the load.

---

We upgraded to 15.0(2)SE and still have the issues. I thought about the stack being too large. When you say the load could too much, in what way? I haven't really seen CPU or memory being overloaded. What do you think may be being overutilized?

[Leo Laohoo]

Post the complete output to the following commands:
1. sh version; and
2. dir flash1:

[martinezaw]

SHOW VERSION

MCMS-MDF-Stack#sh ver
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(2)SE9, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Tue 01-Dec-15 06:19 by prod_rel_team

ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(58r)SE1, RELEASE SOFTWARE (fc1)

MCMS-MDF-Stack uptime is 2 weeks, 2 days, 19 hours, 10 minutes
System returned to ROM by power-on
System restarted at 17:56:48 EDT Mon Nov 19 2018
System image file is "flash:/c3750e-universalk9-mz.150-2.SE9/c3750e-universalk9-mz.150-2.SE9.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: lanbase
License Type: Permanent
Next reload license Level: lanbase

cisco WS-C3750X-48 (PowerPC405) processor (revision A0) with 262144K bytes of memory.
Processor board ID FDO1721Z319
Last reset from power-on
6 Virtual Ethernet interfaces
1 FastEthernet interface
364 Gigabit Ethernet interfaces
14 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 4C:00:82:1C:8C:00
Motherboard assembly number     : 73-12552-10
Motherboard serial number       : FDO172130MP
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48T-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO17212YGA
System serial number            : FDO1721Z319
Top Assembly Part Number        : 800-31326-08
Top Assembly Revision Number    : B0
Version ID                      : V05
CLEI Code Number                : COMJT00ARE
Hardware Board Revision Number  : 0x05


Switch Ports Model              SW Version            SW Image                 
------ ----- -----              ----------            ----------               
     1 54    WS-C3750X-48       15.0(2)SE9            C3750E-UNIVERSALK9-M     
     2 54    WS-C3750X-48       15.0(2)SE9            C3750E-UNIVERSALK9-M     
     3 54    WS-C3750X-48       15.0(2)SE9            C3750E-UNIVERSALK9-M     
     4 54    WS-C3750X-48P      15.0(2)SE9            C3750E-UNIVERSALK9-M     
*    5 54    WS-C3750X-48       15.0(2)SE9            C3750E-UNIVERSALK9-M     
     6 54    WS-C3750X-48P      15.0(2)SE9            C3750E-UNIVERSALK9-M     
     7 54    WS-C3750X-48P      15.0(2)SE9            C3750E-UNIVERSALK9-M     


Switch 01
---------
Switch Uptime                   : 1 day, 23 hours, 58 minutes
Base ethernet MAC Address       : 4C:00:82:20:D1:80
Motherboard assembly number     : 73-12552-10
Motherboard serial number       : FDO17220B62
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48T-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO17212L8T
System serial number            : FDO1721H3V9
Top assembly part number        : 800-31326-08
Top assembly revision number    : B0
Version ID                      : V05
CLEI Code Number                : COMJT00ARE
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase


Switch 02
---------
Switch Uptime                   : 1 day, 23 hours, 58 minutes
Base ethernet MAC Address       : 4C:00:82:20:CC:00
Motherboard assembly number     : 73-12552-10
Motherboard serial number       : FDO17220B0X
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48T-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO17212Y6V
System serial number            : FDO1721H3V0
Top assembly part number        : 800-31326-08
Top assembly revision number    : B0
Version ID                      : V05
CLEI Code Number                : COMJT00ARE
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase


Switch 03
---------
Switch Uptime                   : 1 day, 23 hours, 58 minutes
Base ethernet MAC Address       : 4C:00:82:20:CD:00
Motherboard assembly number     : 73-12552-10
Motherboard serial number       : FDO17220B17
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48T-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO172200HZ
System serial number            : FDO1721H3V7
Top assembly part number        : 800-31326-08
Top assembly revision number    : B0
Version ID                      : V05
CLEI Code Number                : COMJT00ARE
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase

          
Switch 04
---------
Switch Uptime                   : 1 day, 23 hours, 57 minutes
Base ethernet MAC Address       : 30:E4:DB:FC:A8:80
Motherboard assembly number     : 73-12553-05
Motherboard serial number       : FDO152705VE
Model revision number           : A0
Motherboard revision number     : C0
Model number                    : WS-C3750X-48PF-S
Daughterboard assembly number   : 800-32727-01
Daughterboard serial number     : FDO15270278
System serial number            : FDO1526P128
Top assembly part number        : 800-31324-02
Top assembly revision number    : C0
Version ID                      : V02
CLEI Code Number                : COMJZ00ARB
License Level                   : lanbase
License Type                    : Default. No valid license found.lanbase
Next reboot licensing Level     : lanbase


Switch 06
---------
Switch Uptime                   : 1 day, 23 hours, 54 minutes
Base ethernet MAC Address       : B0:00:B4:B1:91:80
Motherboard assembly number     : 73-12553-11
Motherboard serial number       : FDO18071QPQ
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48P-L
Daughterboard assembly number   : 800-32727-03
Daughterboard serial number     : FDO180716Z0
System serial number            : FDO1807Z1P8
Top assembly part number        : 800-31324-09
Top assembly revision number    : B0
Version ID                      : V06
CLEI Code Number                : CMMPS00DRA
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase


Switch 07
---------
Switch Uptime                   : 1 day, 23 hours, 57 minutes
Base ethernet MAC Address       : E0:5F:B9:C2:35:80
Motherboard assembly number     : 73-12553-04
Motherboard serial number       : FDO144919FV
Model revision number           : A0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48PF-L
Daughterboard assembly number   : 800-32727-01
Daughterboard serial number     : FDO14490HAP
System serial number            : FDO1449R05E
Top assembly part number        : 800-31324-01
Top assembly revision number    : A0
Version ID                      : V01
CLEI Code Number                : COMJZ00ARA
License Level                   : lanbase
License Type                    : Permanent
Next reboot licensing Level     : lanbase


Configuration register is 0xF

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

DIR FLASH1:

MCMS-MDF-Stack# dir flash1:
Directory of flash1:/

    2  -rwx       45870   Dec 6 2018 05:00:39 -05:00  config.text
    3  -rwx        5731   Dec 6 2018 05:00:39 -05:00  private-config.text
    4  -rwx       45856   Dec 5 2018 05:00:39 -05:00  config.text.backup
    5  -rwx        5731   Dec 5 2018 05:00:39 -05:00  private-config.text.backup
    6  -rwx        3436   Jan 1 2006 19:01:56 -05:00  vlan.dat
    7  drwx         512  Feb 28 1993 19:10:21 -05:00  c3750e-universalk9-mz.150-1.SE3
  471  -rwx        2072  Feb 28 1993 20:18:37 -05:00  multiple-fs
  472  drwx         512  Feb 28 1993 19:05:36 -05:00  crashinfo_ext
  478  drwx         512  Feb 28 1993 19:00:08 -05:00  crashinfo
  484  drwx         512  Nov 19 2018 17:29:14 -05:00  c3750e-universalk9-mz.150-2.SE9
 1010  drwx         512  Nov 19 2018 17:29:17 -05:00  dc_profile_dir

[Leo Laohoo]

Ok, now post the complete output to the command "dir flash1:crashinfo_ext".
I see the uptime of the stack at 2 days and running 15.0(2)SE9.

[martinezaw]

Yes, that switch did really go down that day. Usually though the device is really up when the alerts happen.

Here's the output attached. (not sure how current it is since it's timestamp is 1993 but the clock is correct on the switch)

[Leo Laohoo]

Is there only one crashinfo file? This one is old.
Post the complete output to the command "dir flash1:crashinfo_ext".

[martinezaw]

The file I attached above is the only one in flash1: It should have all the input within it. I could not paste inside the forum because of the character limit.

There is no more recent crashfile.