Cisco Catalyst 9300 Kernal logs ??

[m.s.rees1]

Hi,

We recently set up a 9300 stack (3 switches) and all has been fine. Recently however whilst on the switch it keeps reporting the following kernal messages every 30 seconds or so, can anyone shed some light?

*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: INFO: rcu_sched self-detected stall on CPU
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: INFO: rcu_sched self-detected stall on CPU
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: ^I2-...: (1 GPs behind) idle=ddd/2/0 softirq=12597740/12597741 fqs=170140
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: ^I (t=525024 jiffies g=2182353 c=2182352 q=228949)
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: ffffffff81a2df40 ffff88023fc83c70 ffffffff81097244 0000000000000002
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: ffffffff81a2df40 ffff88023fc83c88 ffffffff81099452 0000000000000003
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: ffff88023fc83cb8 ffffffff810bd619 ffff88023fc94f80 ffffffff81a2df40
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: Call Trace:
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: <IRQ> [<ffffffff81097244>] sched_show_task+0xa4/0x100
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff81099452>] dump_cpu_task+0x32/0x40
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff810bd619>] rcu_dump_cpu_stacks+0x89/0xe0
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff810c0d59>] rcu_check_callbacks+0x439/0x750
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff81099944>] ? sched_clock_cpu+0x94/0xa0
*Jun 29 11:55:13.608: %IOSXE-4-PLATFORM: Switch 2 R0/0: kernel: [<ffffffff810c385c>] update_process_times+0xac/0xe0
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: ^I6-...: (1 GPs behind) idle=309/2/0 softirq=20071489/20071490 fqs=170140
*Jun 29 11:55:13.608: %IOSXE-3-PLATFORM: Switch 2 R0/0: kernel: ^I (t=525024 jiffies g=2182353 c=2182352 q=228949)Open

[Leo Laohoo]

What is the exact firmware version the stack is running on?

[m.s.rees1]

---

@Leo Laohoo wrote:
What is the exact firmware version the stack is running on?

---

The three switches in the stack are all on Cisco IOS XE Software, Version 16.06.03

[Leo Laohoo]

Please raise a TAC Case. 16.6.3 is not "mature" yet. I believe this issue could be similar to CSCvj58046.

[m.morris]

@Leo Laohoo wrote:
Please raise a TAC Case. 16.6.3 is not "mature" yet. I believe this issue could be similar to CSCvj58046.

Thanks for your reply. I think i'll contact our supplier as they are brand new switches. The log seems to only mention switch 2 of the stack interestingly, so wondering if there is a problem with that one only. The three switch are identical and all on the same firmware etc.

---

[joe.doherty]

Having the same issue here did TAC give you an answer to your question?
Hardware 9300-48 P
V16.6.2
3 Stacked switches. We open a TAC case last night because one of the 3 switches was crashing. But TAC recommended nothing at the moment since they rebooted the stack switch 3 has been stable for about 16 hours but is showing these logs.
Aug 2 09:33:13.563: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 55 ce b2 2f 25 70 00 d2 91 d0 da e2 fe a5 9b 01
Aug 2 09:33:13.563: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.563: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.563: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.563: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.563: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.563: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.563: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.563: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.564: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.564: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.564: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.564: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 64 94
Aug 2 09:33:13.564: %IOSXE-4-PLATFORM: Switch 3 R0/0: kernel: oobhal: 87 b8 3a 18 d4 ca 0e 5b cb 83 4a af f2 5c 00 00

[m.s.rees1 · Accepted solution]

hi joe.doherty,

We've finally had our switch replaced due to what appeared to be a hardware fault. The switch would start throwing up the error logs and would eventually completely crash, after being up for over 24 hours (even out of the stack). Cisco support eventually came to the conclusion that it needed to be replaced. We did try upgrading to the latest firmware and changing the stack leads etc, but that didn't help with us.

Matt

[michael brock]

Joe,

Were you able to resolve your issue that generated the logs? I am starting to get the same. Switch is stable and functioning. 

Jan 16 01:32:31.199: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088428: Jan 16 01:32:31.199: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088429: Jan 16 01:32:31.324: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088430: Jan 16 01:32:31.324: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088431: Jan 16 01:32:31.449: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00
088432: Jan 16 01:32:31.449: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: oobhal: 00 00 18 00 00 00 00 00 00 00 00 00 00

[Bhavik88498]

Hi All,

I am having same issue. I am running on 16.9.4.

084406: Oct 5 04:41:30.279 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084407: Oct 5 04:41:30.279 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084408: Oct 5 04:42:20.295 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084409: Oct 5 04:42:20.295 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084410: Oct 5 04:43:10.312 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084411: Oct 5 04:43:10.312 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084412: Oct 5 04:43:53.505 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: *** Detected Flash issues. Please issue the reload slot <#> command at your earliest convenience ***
084413: Oct 5 04:44:00.329 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084414: Oct 5 04:44:00.329 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084415: Oct 5 04:44:50.346 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084416: Oct 5 04:44:50.346 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084417: Oct 5 04:45:40.364 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084418: Oct 5 04:45:40.364 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084419: Oct 5 04:46:30.382 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084420: Oct 5 04:46:30.382 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449
084421: Oct 5 04:47:20.399 EDT: %IOSXE-3-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): previous I/O error to superblock detected
084422: Oct 5 04:47:20.399 EDT: %IOSXE-2-PLATFORM: Switch 4 R0/0: kernel: EXT2-fs (sda3): error: ext2_readdir: bad page in #360449s

[Leo Laohoo]

Avoid using 16.9.X train.  Go straight to 16.12.4 and see if this improves things.