ASR9K Monitor Session Garbage Output

[wj343]

I have the following monitor-session config deployed on my ASR9010 router running IOS-XR 6.4.2:

monitor-session mirror ethernet
  destination interface TenGigE0/0/0/23
!
interface TenGigE0/0/0/1
  monitor-session mirror ethernet
!

Theoretically, this should mirror all traffic from TenGigE0/0/0/1 to TenGigE0/0/0/23. However, I have a server hooked up to the other end of TenGigE0/0/0/23 and am only seeing garbage traffic:

21:23:49.011305 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.011687 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.011784 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.011958 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.012611 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.012645 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 22, Flags [Command], length 46
21:23:49.013179 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.013478 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.013612 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Poll], length 46
21:23:49.014410 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.014708 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 19, Flags [Poll], length 46
21:23:49.014927 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.014945 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.015274 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.015277 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 19, Flags [Poll], length 46
21:23:49.015464 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46
21:23:49.015951 xx:xx:xx:xx:xx:xx (oui Unknown) Null > xx:xx:xx:xx:xx:xx (oui Unknown) Unknown DSAP 0x40 Information, send seq 0, rcv seq 20, Flags [Command], length 46

All of the packets received on the mirrored port conform to the following characteristics:

• The source and destination MAC address are valid and correct
• The "type" field in the MAC header is always 0x0000
• The packet is always 60 bytes

I'm only pushing a few hundred megabits of information, so that's much less than the recommended 15% or 1.5Gbps. I've also tried capturing with a different server, which yields the same results.

Is there any way for me to tell if IOS-XR is somehow corrupting the packet output? Or am I setting up my capture incorrectly somewhere?

[balaji.bandi]

interface TenGigE0/0/0/1

is this Layer 2 or Layer 3 port ?

try mirror first 128bytes to start with, if you not looking full header.

interface TenGigE0/0/0/1
  monitor-session mirror ethernet
  mirror first 128

or try other way :

monitor-session TESTING 
  destination interface TenGigE0/0/0/23
!
interface TenGigE0/0/0/1
  monitor-session TESTING
 mirror first 128
!

post - show monitor-session TESTING status

[wj343]

I ended up moving my SPAN port from TenGigE0/0/0/23 to TenGigE0/0/0/20 and that worked ... for a few weeks, now the issue randomly presents itself again. I wonder whether this is a bug in IOS XR?

I did a NP capture for counter PARSE_SPAN_LPBK on the source port (port being mirrored), and the resulting packet seems to be valid with the correct MAC addresses, ethertype 0x0800, and IPv4 version 0x4 with IHL 0x5.

monitor np counter PARSE_SPAN_LPBK np5 location 0/0/CPU0

I then tried to capture the packet on the output NP (NP6 for TenGigE0/0/0/20). The first 32 bytes of this packet looked like some internal headers, but starting from byte 33 I was able to see the valid ethernet header with correct MAC address, ethertype, IPv4 version, and IHL.

monitor np counter PARSE_FAB_RECEIVE_CNT np6 location 0/0/CPU0

Finally, I tried doing a NP capture for counter MDF_TX_WIRE on the same NP. I found that here, the MAC addresses are still valid, but the ethertype is now 0x0000, if presuming this was an IPv4 packet, the IPv4 version is still 0x4 but now the IHL is 0x0.

monitor np counter MDF_TX_WIRE.1 np6 location 0/0/CPU0

So it looks like the packet is somehow being corrupted when moving from PARSE_FAB_RECEIVE_CNT to MDF_TX_WIRE?

[wj343 · Accepted solution]

This is definitely a very interesting problem ... I was messing around and added "l2transport" to my destination interface, that somehow magically resolved the issue. Removing it results in the issue happening again. I find that very strange since I've never had l2transport enabled on the destination interface, yet it seemed to work perfectly fine before.

interface TenGigE0/0/0/20
  l2transport
  !
!