Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
4220
Views
12
Helpful
31
Replies

Unknown output from router

Go to solution
Mike Buyarski
Level 3
Level 3

‎12-08-2015 02:24 PM - edited ‎03-05-2019 02:53 AM

i was in our router and out of the blue i got this.

Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel: ucode_pkt_PPE: page allocation failure. order:5, mode:0x40d0
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel: Pid: 4618, comm: ucode_pkt_PPE Not tainted 2.6.32.59 #5
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel: Call Trace:
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81089c2d>] ? __alloc_pages_nodemask+0x592/0x5c9
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b0431>] ? kmalloc_large_node+0x5b/0x8a
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b16ab>] ? __kmalloc_node+0x28/0xe8
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa01298f9>] ? userMemAlloc+0x135/0x230 [icp_qa_al]
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128731>] ? dev_mem_alloc+0x58/0xd1 [icp_qa_al]
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128825>] ? mem_ioctl+0x7b/0x271 [icp_qa_al]
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810c98e8>] ? dput+0x3b/0x13a
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810bbaa2>] ? __fput+0x1ab/0x1e2
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810f0db1>] ? compat_sys_ioctl+0x1be/0x465
Dec  8 16:18:14: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b8ef0>] ? filp_close+0x60/0x6c
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff8102c085>] ? sysenter_dispatch+0x7/0x2b
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32 per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:  186, btch:  31 usd:  74
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:  186, btch:  31 usd:  82
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: active_anon:332252 inactive_anon:111254 isolated_anon:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  active_file:7648 inactive_file:130154 isolated_file:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  unevictable:5541 dirty:1 writeback:0 unstable:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  free:57754 slab_reclaimable:6679 slab_unreclaimable:9596
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  mapped:106373 shmem:86094 pagetables:5213 bounce:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA free:10148kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15236kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 1545 3561 3561
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32 free:205016kB min:3304kB low:4128kB high:4956kB active_anon:897852kB inactive_anon:299840kB active_file:6952kB inactive_file:99700kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1582324kB mlocked:0kB dirty:0kB writeback:0kB mapped:273156kB shmem:303672kB slab_reclaimable:7364kB slab_unreclaimable:6524kB kernel_stack:640kB pagetables:11596kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:91 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 0 2016 2016
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal free:15852kB min:4312kB low:5388kB high:6468kB active_anon:431156kB inactive_anon:145176kB active_file:23640kB inactive_file:420716kB unevictable:22164kB isolated(anon):0kB isolated(file):0kB present:2064384kB mlocked:0kB dirty:4kB writeback:0kB mapped:152336kB shmem:40704kB slab_reclaimable:19352kB slab_unreclaimable:31856kB kernel_stack:2808kB pagetables:9256kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:132 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 0 0 0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA: 3*4kB 3*8kB 4*16kB 2*32kB 2*64kB 3*128kB 3*256kB 1*512kB 2*1024kB 1*2048kB 1*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 10148kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32: 3538*4kB 2330*8kB 1586*16kB 1771*32kB 1409*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 205016kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal: 2363*4kB 423*8kB 133*16kB 29*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 16020kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: 229325 total pagecache pages
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: 0 pages in swap cache
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Free swap  = 0kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Total swap = 0kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] Unable to allocate memory mem_info->kmalloc_ptr
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] dev_mem_alloc(): userMemAlloc failed
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: ucode_pkt_PPE: page allocation failure. order:5, mode:0x40d0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Pid: 4618, comm: ucode_pkt_PPE Not tainted 2.6.32.59 #5
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Call Trace:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81089c2d>] ? __alloc_pages_nodemask+0x592/0x5c9
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b0431>] ? kmalloc_large_node+0x5b/0x8a
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b16ab>] ? __kmalloc_node+0x28/0xe8
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa01298f9>] ? userMemAlloc+0x135/0x230 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128731>] ? dev_mem_alloc+0x58/0xd1 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128825>] ? mem_ioctl+0x7b/0x271 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81038f7c>] ? finish_task_switch+0x4d/0xcf
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81338c2a>] ? thread_return+0x3e/0xcc
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810c98e8>] ? dput+0x3b/0x13a
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810f0db1>] ? compat_sys_ioctl+0x1be/0x465
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81338e04>] ? __schedule+0x3d/0x58
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff8102c085>] ? sysenter_dispatch+0x7/0x2b
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32 per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:  186, btch:  31 usd: 134
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: active_anon:332307 inactive_anon:111273 isolated_anon:7
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  active_file:4049 inactive_file:53790 isolated_file:62
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  unevictable:5541 dirty:0 writeback:0 unstable:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  free:138403 slab_reclaimable:5724 slab_unreclaimable:9597
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  mapped:99734 shmem:86094 pagetables:5212 bounce:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA free:10148kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15236kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 1545 3561 3561
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32 free:263800kB min:3304kB low:4128kB high:4956kB active_anon:897960kB inactive_anon:299836kB active_file:3724kB inactive_file:44712kB unevictable:0kB isolated(anon):28kB isolated(file):120kB present:1582324kB mlocked:0kB dirty:0kB writeback:0kB mapped:267296kB shmem:303556kB slab_reclaimable:7120kB slab_unreclaimable:6524kB kernel_stack:640kB pagetables:11576kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 0 2016 2016
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal free:279664kB min:4312kB low:5388kB high:6468kB active_anon:431268kB inactive_anon:145256kB active_file:12472kB inactive_file:170448kB unevictable:22164kB isolated(anon):0kB isolated(file):128kB present:2064384kB mlocked:0kB dirty:0kB writeback:0kB mapped:131396kB shmem:40820kB slab_reclaimable:15776kB slab_unreclaimable:31860kB kernel_stack:2808kB pagetables:9272kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:146 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 0 0 0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA: 3*4kB 3*8kB 4*16kB 2*32kB 2*64kB 3*128kB 3*256kB 1*512kB 2*1024kB 1*2048kB 1*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 10148kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32: 7592*4kB 3502*8kB 2734*16kB 2238*32kB 1409*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 263920kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal: 27525*4kB 12603*8kB 3639*16kB 338*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 280092kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: 149528 total pagecache pages
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: 0 pages in swap cache
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Free swap  = 0kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Total swap = 0kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] Unable to allocate memory mem_info->kmalloc_ptr
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] dev_mem_alloc(): userMemAlloc failed
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: ucode_pkt_PPE: page allocation failure. order:5, mode:0x40d0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Pid: 4618, comm: ucode_pkt_PPE Not tainted 2.6.32.59 #5
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Call Trace:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81089c2d>] ? __alloc_pages_nodemask+0x592/0x5c9
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b0431>] ? kmalloc_large_node+0x5b/0x8a
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b16ab>] ? __kmalloc_node+0x28/0xe8
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa01298f9>] ? userMemAlloc+0x135/0x230 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128731>] ? dev_mem_alloc+0x58/0xd1 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128825>] ? mem_ioctl+0x7b/0x271 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff8109823e>] ? handle_mm_fault+0x3fe/0x7be
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810c98e8>] ? dput+0x3b/0x13a
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810f0db1>] ? compat_sys_ioctl+0x1be/0x465
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff8133cdb2>] ? do_page_fault+0x308/0x325
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff8102c085>] ? sysenter_dispatch+0x7/0x2b
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32 per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:  186, btch:  31 usd:  29
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: active_anon:332260 inactive_anon:111273 isolated_anon:7
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  active_file:2558 inactive_file:25922 isolated_file:62
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  unevictable:5541 dirty:0 writeback:0 unstable:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  free:168798 slab_reclaimable:4856 slab_unreclaimable:9597
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  mapped:96806 shmem:86094 pagetables:5212 bounce:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA free:10148kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15236kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 1545 3561 3561
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32 free:287116kB min:3304kB low:4128kB high:4956kB active_anon:897772kB inactive_anon:299836kB active_file:2168kB inactive_file:23296kB unevictable:0kB isolated(anon):28kB isolated(file):120kB present:1582324kB mlocked:0kB dirty:0kB writeback:0kB mapped:264856kB shmem:303556kB slab_reclaimable:6628kB slab_unreclaimable:6524kB kernel_stack:640kB pagetables:11576kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 0 2016 2016
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal free:377928kB min:4312kB low:5388kB high:6468kB active_anon:431268kB inactive_anon:145256kB active_file:8064kB inactive_file:80392kB unevictable:22164kB isolated(anon):0kB isolated(file):128kB present:2064384kB mlocked:0kB dirty:0kB writeback:0kB mapped:122368kB shmem:40820kB slab_reclaimable:12796kB slab_unreclaimable:31860kB kernel_stack:2808kB pagetables:9272kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 0 0 0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA: 3*4kB 3*8kB 4*16kB 2*32kB 2*64kB 3*128kB 3*256kB 1*512kB 2*1024kB 1*2048kB 1*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 10148kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32: 9199*4kB 4007*8kB 3167*16kB 2423*32kB 1409*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 287236kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal: 35864*4kB 17456*8kB 5018*16kB 449*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 377888kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: 120187 total pagecache pages
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: 0 pages in swap cache
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Free swap  = 0kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Total swap = 0kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] Unable to allocate memory mem_info->kmalloc_ptr
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] dev_mem_alloc(): userMemAlloc failed
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: ucode_pkt_PPE: page allocation failure. order:5, mode:0x40d0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Pid: 4618, comm: ucode_pkt_PPE Not tainted 2.6.32.59 #5
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Call Trace:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81089c2d>] ? __alloc_pages_nodemask+0x592/0x5c9
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b0431>] ? kmalloc_large_node+0x5b/0x8a
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b16ab>] ? __kmalloc_node+0x28/0xe8
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa01298f9>] ? userMemAlloc+0x135/0x230 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128731>] ? dev_mem_alloc+0x58/0xd1 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128825>] ? mem_ioctl+0x7b/0x271 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81038f7c>] ? finish_task_switch+0x4d/0xcf
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81338c2a>] ? thread_return+0x3e/0xcc
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810c98e8>] ? dput+0x3b/0x13a
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810f0db1>] ? compat_sys_ioctl+0x1be/0x465
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff8133cdb2>] ? do_page_fault+0x308/0x325
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81338e04>] ? __schedule+0x3d/0x58
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff8102c085>] ? sysenter_dispatch+0x7/0x2b
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:    0, btch:   1 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32 per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal per-cpu:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    0: hi:  186, btch:  31 usd:  27
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    2: hi:  186, btch:  31 usd:  29
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    4: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: active_anon:332260 inactive_anon:111273 isolated_anon:7
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  active_file:2558 inactive_file:27108 isolated_file:62
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  unevictable:5541 dirty:0 writeback:0 unstable:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  free:167508 slab_reclaimable:4856 slab_unreclaimable:9597
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  mapped:96806 shmem:86094 pagetables:5212 bounce:0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA free:10148kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15236kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 1545 3561 3561
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32 free:287384kB min:3304kB low:4128kB high:4956kB active_anon:897772kB inactive_anon:299836kB active_file:2168kB inactive_file:23108kB unevictable:0kB isolated(anon):28kB isolated(file):120kB present:1582324kB mlocked:0kB dirty:0kB writeback:0kB mapped:264856kB shmem:303556kB slab_reclaimable:6628kB slab_unreclaimable:6524kB kernel_stack:640kB pagetables:11576kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 0 2016 2016
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal free:372500kB min:4312kB low:5388kB high:6468kB active_anon:431268kB inactive_anon:145256kB active_file:8064kB inactive_file:85568kB unevictable:22164kB isolated(anon):0kB isolated(file):128kB present:2064384kB mlocked:0kB dirty:0kB writeback:0kB mapped:122368kB shmem:40820kB slab_reclaimable:12796kB slab_unreclaimable:31860kB kernel_stack:2808kB pagetables:9272kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: lowmem_reserve[]: 0 0 0 0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA: 3*4kB 3*8kB 4*16kB 2*32kB 2*64kB 3*128kB 3*256kB 1*512kB 2*1024kB 1*2048kB 1*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 10148kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 DMA32: 9223*4kB 4017*8kB 3174*16kB 2423*32kB 1409*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 287524kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Node 0 Normal: 34450*4kB 17465*8kB 5018*16kB 449*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 372304kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: 121407 total pagecache pages
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: 0 pages in swap cache
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Free swap  = 0kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Total swap = 0kB
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] Unable to allocate memory mem_info->kmalloc_ptr
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] dev_mem_alloc(): userMemAlloc failed
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: ucode_pkt_PPE: page allocation failure. order:5, mode:0x40d0
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Pid: 4618, comm: ucode_pkt_PPE Not tainted 2.6.32.59 #5
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: Call Trace:
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81089c2d>] ? __alloc_pages_nodemask+0x592/0x5c9
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b0431>] ? kmalloc_large_node+0x5b/0x8a
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810b16ab>] ? __kmalloc_node+0x28/0xe8
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa01298f9>] ? userMemAlloc+0x135/0x230 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128731>] ? dev_mem_alloc+0x58/0xd1 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffffa0128825>] ? mem_ioctl+0x7b/0x271 [icp_qa_al]
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81038f7c>] ? finish_task_switch+0x4d/0xcf
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff81338c2a>] ? thread_return+0x3e/0xcc
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel:  [<ffffffff810c98e8>] ? dput+0x3b/0x13a
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] Unable to allocate memory mem_info->kmalloc_ptr
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] dev_mem_alloc(): userMemAlloc failed
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] Unable to allocate memory mem_info->kmalloc_ptr
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] dev_mem_alloc(): userMemAlloc failed
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] Unable to allocate memory mem_info->kmalloc_ptr
Dec  8 16:18:16: %IOSXE-4-PLATFORM: R0/0: kernel: [error] dev_mem_alloc(): userMemAlloc failed

Not entirely sure what this is. Buy it does not look good.

just for grins: a sh ver

Cisco IOS XE Software, Version 03.13.03.S - Extended Support Release
Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(3)S3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 28-May-15 15:58 by mcpre


Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


ROM: IOS-XE ROMMON

R-BAY-TW uptime is 22 weeks, 6 days, 19 hours, 19 minutes
Uptime for this control processor is 22 weeks, 6 days, 19 hours, 21 minutes
System returned to ROM by reload at 21:57:44 CDT Tue Jun 30 2015
System restarted at 22:01:22 CDT Tue Jun 30 2015
System image file is "bootflash:isr4300-universalk9.03.13.03.S.154-3.S3-ext.SPA.bin"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.


Technology Package License Information:

-----------------------------------------------------------------
Technology    Technology-package           Technology-package
              Current       Type           Next reboot
------------------------------------------------------------------
appx             None             None             None
uc               None             None             None
security         securityk9       Permanent        securityk9
ipbase           ipbasek9         Permanent        ipbasek9

cisco ISR4331/K9 (1RU) processor with 1687301K/6147K bytes of memory.
Processor board ID FLM1914W038
3 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
3223551K bytes of flash memory at bootflash:.

Configuration register is 0x2102

Labels:
0 Helpful
1 Accepted Solution

Accepted Solutions

Go to solution
Masoud Pourshabanian
VIP Alumni
VIP Alumni

‎12-08-2015 03:11 PM

Hello,

This is an IOS bug. You need to upgrade your IOS.

Please check the below link. It is not exactly the same,  but very similar.

https://tools.cisco.com/quickview/bug/CSCuu85404

Masoud

View solution in original post

0 Helpful
31 Replies 31

Go to solution
Masoud Pourshabanian
VIP Alumni
VIP Alumni

‎12-08-2015 03:11 PM

Hello,

This is an IOS bug. You need to upgrade your IOS.

Please check the below link. It is not exactly the same,  but very similar.

https://tools.cisco.com/quickview/bug/CSCuu85404

Masoud

0 Helpful

Go to solution
Mike Buyarski
Level 3
Level 3
In response to Masoud Pourshabanian

‎12-08-2015 07:24 PM

that bug is for a AP while what we have is a ISR4331

0 Helpful

Go to solution
Masoud Pourshabanian
VIP Alumni
VIP Alumni
In response to Mike Buyarski

‎12-08-2015 07:36 PM

It was just a sample. The bug related to this line in your log

kernel: ucode_pkt_PPE: page allocation failure. order.

It is an OS bug.

Masoud

0 Helpful

Go to solution
Junaid Shah
Level 1
Level 1

‎12-09-2015 05:07 AM

I also have Cisco 4331 and just noticed exactly the same logs on my router. Opening case with Cisco now, will provide update as i learn more. Thanks

0 Helpful

Go to solution
Mike Buyarski
Level 3
Level 3
In response to Junaid Shah

‎12-09-2015 06:19 AM

I c well i have the new IOS ready but i have not reloaded the router yet. does your router have the same version of software?

0 Helpful

Go to solution
Junaid Shah
Level 1
Level 1
In response to Mike Buyarski

‎12-09-2015 12:31 PM

I use this IOS

isr4300-universalk9.03.14.02.S.155-1.S2-std.SPA.bin" i.e. 3.14

and my output of commands that were earlier mentioned is pretty much same i.e. this process "cpp_sp_svr" is consuming lot of memory. I have not noticed these logs again and its been few hours. Are these logs appearing frequently on your router.

show platform software process list rp active sort memory
Name                     Pid    PPid  Group Id  Status    Priority  Size
------------------------------------------------------------------------------
linux_iosd-imag        22775   22095     22775  R               20  2442084352
fman_fp_image          30526   30019     30526  S               20  2033225728
cpp_cp_svr             29240   28413     29240  S               20  1560596480
qfp-ucode-utah         31403   30689     31403  S               20  1477009408
fman_rp                21367   20306     21367  S               20  1059160064
cpp_sp_svr             30090   29616     30090  S               20  1025232896
cpp_driver             29137   28551     29137  S               20  961929216

show platform software process list rp active sort memory
Name                     Pid    PPid  Group Id  Status    Priority  Size
------------------------------------------------------------------------------
linux_iosd-imag        22775   22095     22775  S               20  2442084352
fman_fp_image          30526   30019     30526  S               20  2033225728
cpp_cp_svr             29240   28413     29240  S               20  1560596480
qfp-ucode-utah         31403   30689     31403  S               20  1477009408
fman_rp                21367   20306     21367  S               20  1059160064

Also just shared show tech and these logs with Cisco TAC so waiting to hear from him.

0 Helpful

Go to solution
Mike Buyarski
Level 3
Level 3
In response to Junaid Shah

‎12-09-2015 12:45 PM

in the past 7 hours i get that output i posted 5 times. so not frequent but often enough.

0 Helpful

Go to solution
Vinit Jain
Cisco Employee
Cisco Employee
In response to Junaid Shah

‎12-09-2015 06:24 PM

If the memory of the cpp_sp_svr process is increasing frequently, You may eventually notice the router crash when the router runs totally out of memory. Could you also get the other output in a file:

show platform software memory qfp-service-process qfp active

Thanks

Vinit

Thanks
--Vinit
0 Helpful

Go to solution
Richard Burts
Hall of Fame
Hall of Fame
In response to Junaid Shah

‎12-09-2015 06:25 AM

A number of the log messages relate to problems with memory allocation. This could very easily be a bug in the OS, or it might be a hardware issue. Please do post back to the forum and let us know what happens in your case with Cisco.

HTH

Rick 

HTH

Rick
0 Helpful

Go to solution
Masoud Pourshabanian
VIP Alumni
VIP Alumni
In response to Richard Burts

‎12-10-2015 12:58 PM

Edited

0 Helpful

Go to solution
Mike Buyarski
Level 3
Level 3
In response to Junaid Shah

‎12-10-2015 09:28 AM

Did TAC get back to you on a solution?

0 Helpful

Go to solution
Junaid Shah
Level 1
Level 1
In response to Mike Buyarski

‎12-10-2015 12:03 PM

Yes, they did. I somehow don't agree and also it seems that whats true for ISR 1000 is not true for 4331 or 4451x. I am new to this and I hope I am not violating any policy here by sharing their response.

The error message reported indicate we failed to allocate memory due to a low memory condition.

From the following output we see qfp-ucode-utah   process is holding a large amount of memory which indicate a leak here:

---- show platform software process list R0 ----

Name                     Pid    PPid  Group Id  Status    Priority  Size

qfp-ucode-utah         31403   30689     31403  S               20  1477009408

So I suspect we are hitting software bug CSCut67872 which fixed the memory leak under qfp-ucode-uta.

Here is the bug link: https://tools.cisco.com/bugsearch/bug/CSCut67872/?reffering_site=dumpcr

This bug is fixed in 15.5(2)S2 and later version.

Go to solution
Richard Burts
Hall of Fame
Hall of Fame
In response to Junaid Shah

‎12-10-2015 12:22 PM

Thank you for sharing what they told you about this problem. I am confident that you are not violating any policy in sharing this information. And it does reinforce that this problem is a memory leak due to a software defect.

HTH

Rick

HTH

Rick
0 Helpful

Go to solution
Vinit Jain
Cisco Employee
Cisco Employee

‎12-09-2015 07:15 AM

Hi,

Could you please confirm if you have DMVPN IPSec tunnel on the router.

Also, please share the output of below command:

show platform software memory qfp-service-process qfp active
show platform software process list rp active sort memory

Thanks

Vinit

Thanks
--Vinit
0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card