06-29-2018 08:09 AM - edited 03-08-2019 03:31 PM
Hi,
We recently set up a 9300 stack (3 switches) and all has been fine. Recently however whilst on the switch it keeps reporting the following kernal messages every 30 seconds or so, can anyone shed some light?
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: INFO: rcu_sched self-detected stall on CPU
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: INFO: rcu_sched self-detected stall on CPU
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: ^I2-...: (1 GPs behind) idle=ddd/2/0 softirq=12597740/12597741 fqs=170140
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: ^I (t=525024 jiffies g=2182353 c=2182352 q=228949)
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: ffffffff81a2df40 ffff88023fc83c70 ffffffff81097244 0000000000000002
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: ffffffff81a2df40 ffff88023fc83c88 ffffffff81099452 0000000000000003
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: ffff88023fc83cb8 ffffffff810bd619 ffff88023fc94f80 ffffffff81a2df40
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: Call Trace:
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: <IRQ> [<ffffffff81097244>] sched_show_task+0xa4/0x100
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff81099452>] dump_cpu_task+0x32/0x40
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff810bd619>] rcu_dump_cpu_stacks+0x89/0xe0
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff810c0d59>] rcu_check_callbacks+0x439/0x750
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff81099944>] ? sched_clock_cpu+0x94/0xa0
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff810c385c>] update_process_times+0xac/0xe0
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: ^I6-...: (1 GPs behind) idle=309/2/0 softirq=20071489/20071490 fqs=170140
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: ^I (t=525024 jiffies g=2182353 c=2182352 q=228949)Open
Solved! Go to Solution.
08-03-2018 01:10 AM
hi joe.doherty,
We've finally had our switch replaced due to what appeared to be a hardware fault. The switch would start throwing up the error logs and would eventually completely crash, after being up for over 24 hours (even out of the stack). Cisco support eventually came to the conclusion that it needed to be replaced. We did try upgrading to the latest firmware and changing the stack leads etc, but that didn't help with us.
Matt
06-29-2018 08:11 AM
06-29-2018 08:13 AM
@Leo Laohoo wrote:
What is the exact firmware version the stack is running on?
The three switches in the stack are all on Cisco IOS XE Software, Version 16.06.03
06-29-2018 08:49 AM
07-03-2018 01:27 AM
@Leo Laohoo wrote:
Please raise a TAC Case. 16.6.3 is not "mature" yet. I believe this issue could be similar to CSCvj58046.
Thanks for your reply. I think i'll contact our supplier as they are brand new switches. The log seems to only mention switch 2 of the stack interestingly, so wondering if there is a problem with that one only. The three switch are identical and all on the same firmware etc.
08-02-2018 08:19 AM
08-03-2018 01:10 AM
hi joe.doherty,
We've finally had our switch replaced due to what appeared to be a hardware fault. The switch would start throwing up the error logs and would eventually completely crash, after being up for over 24 hours (even out of the stack). Cisco support eventually came to the conclusion that it needed to be replaced. We did try upgrading to the latest firmware and changing the stack leads etc, but that didn't help with us.
Matt
01-16-2019 07:05 AM
Joe,
Were you able to resolve your issue that generated the logs? I am starting to get the same. Switch is stable and functioning.
Jan 16 01:32:31.199: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088428: Jan 16 01:32:31.199: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088429: Jan 16 01:32:31.324: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088430: Jan 16 01:32:31.324: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088431: Jan 16 01:32:31.449: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088432: Jan 16 01:32:31.449: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00
10-07-2020 06:26 AM
Hi All,
I am having same issue. I am running on 16.9.4.
084406: Oct 5 04:41:30.279 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084407: Oct 5 04:41:30.279 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084408: Oct 5 04:42:20.295 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084409: Oct 5 04:42:20.295 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084410: Oct 5 04:43:10.312 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084411: Oct 5 04:43:10.312 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084412: Oct 5 04:43:53.505 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: *** Detected Flash issues. Please issue the reload slot <#> command at your earliest convenience ***
084413: Oct 5 04:44:00.329 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084414: Oct 5 04:44:00.329 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084415: Oct 5 04:44:50.346 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084416: Oct 5 04:44:50.346 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084417: Oct 5 04:45:40.364 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084418: Oct 5 04:45:40.364 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084419: Oct 5 04:46:30.382 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084420: Oct 5 04:46:30.382 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084421: Oct 5 04:47:20.399 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084422: Oct 5 04:47:20.399 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449s
10-07-2020 02:16 PM
Avoid using 16.9.X train. Go straight to 16.12.4 and see if this improves things.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide