IPS module issue

[m-stawiski]

Hi guys,

We have an issue with one of the node in our ASA cluster - basically module is in Unresponsive state.

From what I can see there is probably issue with memory - I am suspecting one of the memory module but would like to know if someone have experienced something like that before?

Call Trace:
 [<ffffffff8026d44a>] oom_kill_process+0x88/0x238
 [<ffffffff805103e8>] ? _spin_lock+0x9/0xc
 [<ffffffff8026d7c4>] ? badness+0x1a0/0x1e3
 [<ffffffff8026d93b>] __out_of_memory+0x134/0x14b
 [<ffffffff8026da8a>] out_of_memory+0x138/0x16a
 [<ffffffff8027101c>] __alloc_pages_internal+0x367/0x427
 [<ffffffff80294012>] alloc_pages_current+0xbe/0xc7
 [<ffffffff802a262c>] pipe_write+0x28c/0x4da
 [<ffffffff8029bb3a>] do_sync_write+0xe7/0x12d
 [<ffffffff8024ca44>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff8050ec5c>] ? thread_return+0x3d/0xd4
 [<ffffffff80220733>] ? default_spin_lock_flags+0x9/0xe
 [<ffffffff805103e8>] ? _spin_lock+0x9/0xc
 [<ffffffff8029c3aa>] vfs_write+0xae/0x137
 [<ffffffff8029c4f7>] sys_write+0x47/0x70
 [<ffffffff8020b6aa>] system_call_fastpath+0x16/0x1b
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 175
Active_anon:704 active_file:0 inactive_anon:720
 inactive_file:1 unevictable:259 dirty:0 writeback:0 unstable:0
 free:1848 slab:678 mapped:99 pagetables:13 bounce:0
Node 0 DMA free:3564kB min:12kB low:12kB high:16kB active_anon:768kB inactive_anon:720kB active_file:0kB inactive_file:0kB unevictable:0kB present:3760kB pages_scanned:1460 all_unreclaimable? no
lowmem_reserve[]: 0 888 888 888
Node 0 DMA32 free:3828kB min:3804kB low:4752kB high:5704kB active_anon:2048kB inactive_anon:2160kB active_file:0kB inactive_file:4kB unevictable:1036kB present:909996kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 3*4kB 14*8kB 19*16kB 20*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3564kB
Node 0 DMA32: 8*4kB 6*8kB 5*16kB 3*32kB 0*64kB 2*128kB 1*256kB 2*512kB 0*1024kB 1*2048kB 0*4096kB = 3840kB
773 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
234749 pages RAM
6899 pages reserved
210 pages shared
3947 pages non-shared
Out of memory: kill process 561 (bunzip2) score 72 or a child
Killed process 561 (bunzip2)
cpio: short read
Killed
switch_root: bad init /loader.run: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

Regards,

Marcin

[semenoa@mosenergosbyt.ru]

I have a similar problem on IPS module on the ASA5585X chassis.

IPS  module does not load and displays "Kernel panic - not syncing: Out of memory and no killable processes..." message.

Boot trace: 

NET: Registered protocol family 2
IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Simple Boot Flag at 0x34 set to 0x1
Adding htlb page ffff880037000000 phys 0000000037000000 page ffffe20000c08000
Adding htlb page ffff880036e00000 phys 0000000036e00000 page ffffe20000c01000
Adding htlb page ffff880036c00000 phys 0000000036c00000 page ffffe20000bfa000
Adding htlb page ffff880036a00000 phys 0000000036a00000 page ffffe20000bf3000
Adding htlb page ffff880036800000 phys 0000000036800000 page ffffe20000bec000
Adding htlb page ffff880036600000 phys 0000000036600000 page ffffe20000be5000
Adding htlb page ffff880036400000 phys 0000000036400000 page ffffe20000bde000
Adding htlb page ffff880036200000 phys 0000000036200000 page ffffe20000bd7000
Adding htlb page ffff880036000000 phys 0000000036000000 page ffffe20000bd0000
Adding htlb page ffff880035e00000 phys 0000000035e00000 page ffffe20000bc9000
Adding htlb page ffff880035c00000 phys 0000000035c00000 page ffffe20000bc2000
Adding htlb page ffff880035a00000 phys 0000000035a00000 page ffffe20000bbb000
Adding htlb page ffff880035800000 phys 0000000035800000 page ffffe20000bb4000
Adding htlb page ffff880035600000 phys 0000000035600000 page ffffe20000bad000
Adding htlb page ffff880035400000 phys 0000000035400000 page ffffe20000ba6000
Adding htlb page ffff880035200000 phys 0000000035200000 page ffffe20000b9f000
Adding htlb page ffff880035000000 phys 0000000035000000 page ffffe20000b98000
Adding htlb page ffff880034e00000 phys 0000000034e00000 page ffffe20000b91000
Adding htlb page ffff880034c00000 phys 0000000034c00000 page ffffe20000b8a000
Adding htlb page ffff880034a00000 phys 0000000034a00000 page ffffe20000b83000
Adding htlb page ffff880034800000 phys 0000000034800000 page ffffe20000b7c000
Adding htlb page ffff880034600000 phys 0000000034600000 page ffffe20000b75000
Adding htlb page ffff880034400000 phys 0000000034400000 page ffffe20000b6e000
Adding htlb page ffff880034200000 phys 0000000034200000 page ffffe20000b67000
Adding htlb page ffff880034000000 phys 0000000034000000 page ffffe20000b60000
Adding htlb page ffff880033e00000 phys 0000000033e00000 page ffffe20000b59000
Adding htlb page ffff880033c00000 phys 0000000033c00000 page ffffe20000b52000
Adding htlb page ffff880033a00000 phys 0000000033a00000 page ffffe20000b4b000
Adding htlb page ffff880033800000 phys 0000000033800000 page ffffe20000b44000
Adding htlb page ffff880033600000 phys 0000000033600000 page ffffe20000b3d000
Adding htlb page ffff880033400000 phys 0000000033400000 page ffffe20000b36000
Adding htlb page ffff880033200000 phys 0000000033200000 page ffffe20000b2f000
Adding htlb page ffff880033000000 phys 0000000033000000 page ffffe20000b28000
Adding htlb page ffff880032e00000 phys 0000000032e00000 page ffffe20000b21000
Adding htlb page ffff880032c00000 phys 0000000032c00000 page ffffe20000b1a000
Adding htlb page ffff880032a00000 phys 0000000032a00000 page ffffe20000b13000
Adding htlb page ffff880032800000 phys 0000000032800000 page ffffe20000b0c000
Adding htlb page ffff880032600000 phys 0000000032600000 page ffffe20000b05000
Adding htlb page ffff880032400000 phys 0000000032400000 page ffffe20000afe000
Adding htlb page ffff880032200000 phys 0000000032200000 page ffffe20000af7000
Adding htlb page ffff880032000000 phys 0000000032000000 page ffffe20000af0000
Adding htlb page ffff880031e00000 phys 0000000031e00000 page ffffe20000ae9000
Adding htlb page ffff880031c00000 phys 0000000031c00000 page ffffe20000ae2000
Adding htlb page ffff880031a00000 phys 0000000031a00000 page ffffe20000adb000
Adding htlb page ffff880031800000 phys 0000000031800000 page ffffe20000ad4000
Adding htlb page ffff880031600000 phys 0000000031600000 page ffffe20000acd000
Adding htlb page ffff880031400000 phys 0000000031400000 page ffffe20000ac6000
Adding htlb page ffff880031200000 phys 0000000031200000 page ffffe20000abf000
Adding htlb page ffff880031000000 phys 0000000031000000 page ffffe20000ab8000
Adding htlb page ffff880030e00000 phys 0000000030e00000 page ffffe20000ab1000
Adding htlb page ffff880030c00000 phys 0000000030c00000 page ffffe20000aaa000
Adding htlb page ffff880030a00000 phys 0000000030a00000 page ffffe20000aa3000
Adding htlb page ffff880030800000 phys 0000000030800000 page ffffe20000a9c000
Adding htlb page ffff880030600000 phys 0000000030600000 page ffffe20000a95000
Adding htlb page ffff880030400000 phys 0000000030400000 page ffffe20000a8e000
Adding htlb page ffff880030200000 phys 0000000030200000 page ffffe20000a87000
Adding htlb page ffff880030000000 phys 0000000030000000 page ffffe20000a80000
Adding htlb page ffff88002fe00000 phys 000000002fe00000 page ffffe20000a79000
Adding htlb page ffff88002fc00000 phys 000000002fc00000 page ffffe20000a72000
Adding htlb page ffff88002fa00000 phys 000000002fa00000 page ffffe20000a6b000
Adding htlb page ffff88002f800000 phys 000000002f800000 page ffffe20000a64000
Adding htlb page ffff88002f600000 phys 000000002f600000 page ffffe20000a5d000
Adding htlb page ffff88002f400000 phys 000000002f400000 page ffffe20000a56000
Adding htlb page ffff88002f200000 phys 000000002f200000 page ffffe20000a4f000
HugeTLB registered 2 MB page size, pre-allocated 3876 pages
report_hugepages: Using 64 pages from low memory at ffff88002f200000 HugeTLB FS
msgmni has been set to 15716
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
fakephp: Fake PCI Hot Plug Controller Driver
ACPI: CPU0 (power states: C1[C1] C2[C3])
processor ACPI_CPU:00: registered as cooling_device0
ACPI: Processor [CP00] (supports 8 throttling states)
ACPI: CPU1 (power states: C1[C1] C2[C3])
processor ACPI_CPU:01: registered as cooling_device1
ACPI: CPU2 (power states: C1[C1] C2[C3])
processor ACPI_CPU:02: registered as cooling_device2
ACPI: CPU3 (power states: C1[C1] C2[C3])
processor ACPI_CPU:03: registered as cooling_device3
ACPI: CPU4 (power states: C1[C1] C2[C3])
processor ACPI_CPU:04: registered as cooling_device4
ACPI: CPU5 (power states: C1[C1] C2[C3])
processor ACPI_CPU:05: registered as cooling_device5
ACPI: CPU6 (power states: C1[C1] C2[C3])
processor ACPI_CPU:06: registered as cooling_device6
ACPI: CPU7 (power states: C1[C1] C2[C3])
processor ACPI_CPU:07: registered as cooling_device7
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
ipmi message handler version 39.2
ipmi device interface
IPMI System Interface driver.
ipmi_si: Unable to find any System Interface(s)
IPMI SMB Interface driver
IPMI Watchdog: driver initialized
Copyright (C) 2004 MontaVista Software - IPMI Powerdown via sys_reboot.
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Ъserial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
serial 0000:00:12.3: PCI INT B -> GSI 17 (level, low) -> IRQ 17
0000:00:12.3: ttyS2 at I/O 0xbc08 (irq = 17) is a 16550A
brd: module loaded
loop: module loaded
lpc: version 0.1 (Oct 22 2010)
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Uniform Multi-Platform E-IDE driver
ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
ide-gd driver 1.18
Driver 'sd' needs updating - please use bus_type methods
Driver 'sr' needs updating - please use bus_type methods
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0000:00:1a.7: PCI INT C -> GSI 18 (level, low) -> IRQ 18
ehci_hcd 0000:00:1a.7: EHCI Host Controller
ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1a.7: debug port 1
ehci_hcd 0000:00:1a.7: irq 18, io mem 0xdb9de000
ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 6 ports detected
ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2
ehci_hcd 0000:00:1d.7: debug port 1
ehci_hcd 0000:00:1d.7: irq 16, io mem 0xdb9df000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 6 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1a.0: UHCI Host Controller
uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1a.0: irq 16, io base 0x0000bc20
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
uhci_hcd 0000:00:1a.1: UHCI Host Controller
uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1a.1: irq 17, io base 0x0000bc40
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
uhci_hcd 0000:00:1a.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1a.2: UHCI Host Controller
uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:1a.2: irq 18, io base 0x0000bc60
usb usb5: configuration #1 chosen from 1 choice
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6
uhci_hcd 0000:00:1d.0: irq 16, io base 0x0000bc80
usb usb6: configuration #1 chosen from 1 choice
hub 6-0:1.0: USB hub found
hub 6-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7
uhci_hcd 0000:00:1d.1: irq 17, io base 0x0000bca0
usb usb7: configuration #1 chosen from 1 choice
hub 7-0:1.0: USB hub found
hub 7-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
uhci_hcd 0000:00:1d.2: irq 18, io base 0x0000bcc0
usb usb8: configuration #1 chosen from 1 choice
hub 8-0:1.0: USB hub found
hub 8-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
PNP: No PS/2 controller found. Probing ports directly.
i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
rtc_cmos 00:09: RTC can wake from S4
rtc_cmos 00:09: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day, 114 bytes nvram
i2c /dev entries driver
i801_smbus 0000:00:1f.3: PCI INT C -> GSI 18 (level, low) -> IRQ 18
device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
cpuidle: using governor ladder
ioatdma 0000:00:16.0: can't derive routing for PCI INT A
ioatdma 0000:00:16.0: PCI INT A: no GSI
ioatdma 0000:00:16.0: Intel(R) I/OAT DMA Engine found, 1 channels, device version 0x30, driver version 3.64
ioatdma 0000:00:16.0: DCA is disabled in BIOS
ioatdma 0000:00:16.1: can't derive routing for PCI INT B
ioatdma 0000:00:16.1: PCI INT B: no GSI
ioatdma 0000:00:16.1: Intel(R) I/OAT DMA Engine found, 1 channels, device version 0x30, driver version 3.64
ioatdma 0000:00:16.1: DCA is disabled in BIOS
ioatdma 0000:00:16.2: can't derive routing for PCI INT C
ioatdma 0000:00:16.2: PCI INT C: no GSI
ioatdma 0000:00:16.2: Intel(R) I/OAT DMA Engine found, 1 channels, device version 0x30, driver version 3.64
usb 1-4: new high speed USB device using ehci_hcd and address 2
Clocksource tsc unstable (delta = 4686777628 ns)
ioatdma 0000:00:16.2: DCA is disabled in BIOS
ioatdma 0000:00:16.3: can't derive routing for PCI INT D
ioatdma 0000:00:16.3: PCI INT D: no GSI
ioatdma 0000:00:16.3: Intel(R) I/OAT DMA Engine found, 1 channels, device version 0x30, driver version 3.64
ioatdma 0000:00:16.3: DCA is disabled in BIOS
ioatdma 0000:00:16.4: can't derive routing for PCI INT A
ioatdma 0000:00:16.4: PCI INT A: no GSI
ioatdma 0000:00:16.4: Intel(R) I/OAT DMA Engine found, 1 channels, device version 0x30, driver version 3.64
ioatdma 0000:00:16.4: DCA is disabled in BIOS
ioatdma 0000:00:16.5: can't derive routing for PCI INT B
ioatdma 0000:00:16.5: PCI INT B: no GSI
ioatdma 0000:00:16.5: Intel(R) I/OAT DMA Engine found, 1 channels, device version 0x30, driver version 3.64
ioatdma 0000:00:16.5: DCA is disabled in BIOS
ioatdma 0000:00:16.6: can't derive routing for PCI INT C
ioatdma 0000:00:16.6: PCI INT C: no GSI
ioatdma 0000:00:16.6: Intel(R) I/OAT DMA Engine found, 1 channels, device version 0x30, driver version 3.64
ioatdma 0000:00:16.6: DCA is disabled in BIOS
ioatdma 0000:00:16.7: can't derive routing for PCI INT D
ioatdma 0000:00:16.7: PCI INT D: no GSI
ioatdma 0000:00:16.7: Intel(R) I/OAT DMA Engine found, 1 channels, device version 0x30, driver version 3.64
ioatdma 0000:00:16.7: DCA is disabled in BIOS
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
IPv6: Loaded, but is disabled by default. IPv6 may be enabled on individual interfaces.
NET: Registered protocol family 10
Kernel panic - not syncing: Out of memory and no killable processes...