PKT_INFRA-PQMON-6-QUEUE_DROP

evan.r.moore · ‎06-26-2015

(ASR 9001 running 5.1.2.)

Yesterday I had this five times:

%PKT_INFRA-PQMON-6-QUEUE_DROP : Taildrop on XIPC queue 5 owned by ipv4_io (jid=240)

It seemed roughly to match some odd events with reaching NTP servers, keeping up BGP sessions, and a customer complaint.

Does anyone know how to make sense of this? Job 240 isn't running any more, although it seems to exist:

RP/0/RSP0/CPU0:tr0#show proc 240
Job Id: 240
Executable path: /disk0/iosxr-infra-5.1.2/0x204/bin/fim_cli
Instance #: 10
Version ID: 00.00.0000
Respawn: ON
Respawn count: 0
Max. spawns per minute: 12
Last started: Not yet spawned
Process state: Hold
Package state: Normal
core: MAINMEM
Max. core: 0
Placement: None
startup_path: /pkg/startup/fim_cli.startup
-------------------------------------------------------------------------------

From what little information I can find, something involved in inter-process communication filled. How do I determine just what it was? How to know what caused it to fill? How to prevent it from recurring?

Does anyone have ideas?

ERM

Bitwrangler for FirstLight Fiber Corp

xthuijs · ‎06-27-2015

the xipc queue is a queue that receives packets that are punted. there are several different queues (and associated priorities). This Q in question is for ipv4_io, as the message says indeed, and is used to handle punt traffic generally for netio and beyond.

To identify what the punt reasons have been you could look at the following statistics:

show lpts pifib hardware police location 0/5/cpu0

show ipv4 traffic

to get some impressions as to what had been going on.

another one that you could try is:

run xipc_show -j <jobid>

cheers!

xander

evan.r.moore · ‎06-27-2015

RP/0/RSP0/CPU0:tr0.zr.nycm.ny#show ipv4 traffic

IP statistics:
Rcvd: 57437729 total, 56014349 local destination
218 format errors, 0 bad hop count
9690 unknown protocol, 0 not a gateway
0 security failures, 457393 bad source, 2044294959 bad header
21537 with options, 0 bad, 15 unknown
Opts: 2560 end, 0 nop, 3 basic security, 0 extended security
0 strict source rt, 0 loose source rt, 2559 record rt
1 stream ID, 42 timestamp, 18916 alert, 0 cipso
Frags: 315 reassembled, 27 timeouts, 0 couldn't reassemble, 909 fragments received
33428273 fragmented, 0 fragment count, 0 fragment max drop
Bcast: 0 sent, 0 received
Mcast: 3518022 sent, 1406157 received
Lisp: 0 encapped in v4, 0 decapped from v4
0 encapped in v6, 0 decapped from v6
0 encap errors, 0 decap errors
Drop: 0 encapsulation failed, 4458 no route, 0 too big
Sent: 244683340 total

ICMP statistics:
Sent: 0 admin unreachable, 24039375 network unreachable
1635672 host unreachable, 187 protocol unreachable
68044 port unreachable, 840 fragment unreachable
5356218 time to live exceeded, 11 reassembly ttl exceeded
10746 echo request, 441130 echo reply
0 mask request, 0 mask reply
0 parameter error, 0 redirects
30643474 total
Rcvd: 0 admin unreachable, 1064 network unreachable
235044 host unreachable, 0 protocol unreachable
727386 port unreachable, 0 fragment unreachable
81 time to live exceeded, 0 reassembly ttl exceeded
441130 echo request, 10181 echo reply
0 mask request, 0 mask reply
79 redirect, 0 parameter error
0 source quench, 223 timestamp, 0 timestamp reply
0 router advertisement, 0 router solicitation
1420945 total, 983 checksum errors, 0 unknown

UDP statistics:
5875194 packets input, 8679468 packets output
180 checksum errors, 161021 no port
0 forwarded broadcasts

TCP statistics:
50118902 packets input, 51377164 packets output
1034 checksum errors, 480642 no port
RP/0/RSP0/CPU0:tr0.zr.nycm.ny#show l
l2snoop l2tp l2tpv2 l2vpn
lacp lamptest lcc led
license line lisp lldp
lmp lmrib local logging
loopback lpts lrd lwm
RP/0/RSP0/CPU0:tr0.zr.nycm.ny#show lpts pifib hard police ?
location Location of Pre-IFIB
RP/0/RSP0/CPU0:tr0.zr.nycm.ny#show lpts pifib hard police loca ?
0/0/0 Fully qualified line card specification
0/0/CPU0 Fully qualified line card specification
WORD Fully qualified line card specification
RP/0/RSP0/CPU0:tr0.zr.nycm.ny#show lpts pifib hard police loca 0/0/cPU0 ?
| Output Modifiers
<cr>
RP/0/RSP0/CPU0:tr0.zr.nycm.ny#show lpts pifib hard police loca 0/0/cPU0
-------------------------------------------------------------
Node 0/0/CPU0:
-------------------------------------------------------------
Burst = 100ms for all flow types
-------------------------------------------------------------
FlowType Policer Type Cur. Rate Def. Rate Accepted Dropped TOS Value
---------------------- ------- ------- ---------- ---------- -------------------- -------------------- ----------
unconfigured-default 100 Static 2500 2500 0 0 01234567
L2TPv2-fragment 185 Static 10000 10000 0 0 01234567
Fragment 101 Static 2500 2500 590 0 01234567
OSPF-mc-known 102 Static 2000 2000 0 0 01234567
OSPF-mc-default 103 Static 1500 1500 0 0 01234567
OSPF-uc-known 104 Static 2000 2000 0 0 01234567
OSPF-uc-default 105 Static 1000 1000 0 0 01234567
ISIS-known 143 Static 2000 2000 928456 0 01234567
ISIS-default 144 Static 1500 1500 0 0 01234567
BFD-known 150 Static 9600 9600 0 0 01234567
BFD-default 160 Static 45340 9600 0 0 01234567
BFD-MP-known 178 Static 11520 11520 0 0 01234567
BFD-MP-0 179 Static 128 128 0 0 01234567
BFD-BLB-known 183 Static 11520 11520 0 0 01234567
BFD-BLB-0 184 Static 128 128 0 0 01234567
BFD-SP-0 182 Static 512 512 0 0 01234567
BGP-known 106 Static 2500 2500 45658377 2660 01234567
BGP-cfg-peer 107 Static 2000 2000 140 0 01234567
BGP-default 108 Static 1500 1500 1959520 0 01234567
PIM-mcast-default 109 Static 2000 2000 0 0 01234567
PIM-mcast-known 176 Static 2000 2000 0 0 01234567
PIM-ucast 110 Static 1500 1500 0 0 01234567
IGMP 111 Static 3000 3000 0 0 01234567
ICMP-local 112 Static 1500 1500 437662 0 01234567
ICMP-app 152 Static 1500 1500 0 0 01234567
ICMP-control 140 Static 1000 1000 76445844 0 01234567
ICMP-default 153 Static 1500 1500 80429197 0 01234567
ICMP-app-default 152 Static 1500 1500 202 0 01234567
LDP-TCP-known 113 Static 2500 2500 136561 0 01234567
LDP-TCP-cfg-peer 114 Static 2000 2000 0 0 01234567
LDP-TCP-default 115 Static 1500 1500 0 0 01234567
LDP-UDP 116 Static 2000 2000 1406059 0 01234567
All-routers 117 Static 1000 1000 0 0 01234567
LMP-TCP-known 168 Static 2500 2500 0 0 01234567
LMP-TCP-cfg-peer 169 Static 2000 2000 0 0 01234567
LMP-TCP-default 170 Static 1500 1500 0 0 01234567
LMP-UDP 171 Static 2000 2000 0 0 01234567
RSVP-UDP 118 Static 2000 2000 0 0 01234567
RSVP-default 154 Static 500 500 202 415 01234567
RSVP-known 177 Static 7000 7000 0 0 01234567
IKE 119 Static 100 100 0 0 01234567
IPSEC-known 120 Static 400 400 0 0 01234567
IPSEC-default 121 Static 100 100 0 0 01234567
MSDP-known 122 Static 300 300 0 0 01234567
MSDP-cfg-peer 123 Static 200 200 0 0 01234567
MSDP-default 124 Static 100 100 0 0 01234567
SNMP 125 Static 300 300 3344280 0 01234567
SSH-known 127 Static 300 300 0 0 01234567
SSH-default 128 Static 200 200 432079 0 01234567
HTTP-known 129 Static 400 400 0 0 01234567
HTTP-default 130 Static 200 200 0 0 01234567
SHTTP-known 161 Static 400 400 0 0 01234567
IFIB_FT_SHTTP_DEFAULT 162 Static 200 200 0 0 01234567
TELNET-known 131 Static 200 200 0 0 01234567
TELNET-default 132 Static 200 200 0 0 01234567
CSS-known 133 Static 200 200 0 0 01234567
CSS-default 134 Static 200 200 0 0 01234567
RSH-known 135 Static 200 200 0 0 01234567
RSH-default 136 Static 200 200 0 0 01234567
UDP-known 137 Static 2500 2500 0 0 01234567
UDP-listen 138 Static 2500 2500 0 0 01234567
UDP-cfg-peer 155 Static 2500 2500 0 0 01234567
UDP-default 163 Static 3500 3500 95235 0 01234567
TCP-known 156 Static 2500 2500 0 0 01234567
TCP-listen 157 Static 2500 2500 0 0 01234567
TCP-cfg-peer 158 Static 2000 2000 0 0 01234567
TCP-default 164 Static 2000 2000 270603 0 01234567
Mcast-known 159 Static 2500 2500 0 0 01234567
Mcast-default 165 Static 2000 2000 0 0 01234567
Raw-listen 166 Static 2500 2500 0 0 01234567
Raw-default 167 Static 2500 2500 726287 1375911123 01234567
Ip-Sla 139 Static 1000 1000 0 0 01234567
EIGRP 145 Static 1500 1500 0 0 01234567
RIP 146 Static 1500 1500 0 0 01234567
L2TPv3 141 Static 400 400 0 0 01234567
PCEP 142 Static 200 200 0 0 01234567
GRE 147 Static 10000 10000 0 0 01234567
VRRP 148 Static 1000 1000 0 0 01234567
HSRP 149 Static 400 400 0 0 01234567
MPLS-oam 151 Static 250 250 0 0 01234567
L2TPv2-default 172 Static 2000 2000 0 0 01234567
L2TPv2-known 181 Static 2500 2500 0 0 01234567
DNS 173 Static 2000 2000 0 0 01234567
RADIUS 174 Static 2000 2000 0 0 01234567
TACACS 175 Static 2000 2000 0 0 01234567
NTP-default 126 Static 200 200 5730 0 01234567
NTP-known 180 Static 200 200 12178 0 01234567
AMT 186 Static 4000 4000 0 0 01234567
MIPv6 188 Static 0 0 0 0 01234567
SDAC-TCP 187 Static 5000 5000 0 0 01234567
ONEPK 189 Static 0 0 0 0 01234567

------------------------
statistics:
Packets accepted by deleted entries: 3191269
Packets dropped by deleted entries: 2893
Run out of statistics counter errors: 0

RP/0/RSP0/CPU0:tr0.zr.nycm.ny#run xipc_show -j 240
RP/0/RSP0/CPU0:tr0.zr.nycm.ny#exit

Looking at it, BGP has suffered, but mostly it's raw_default. Is there any way to tell more?

ERM

Bitwrangler for FirstLight Fiber Corp