on 03-20-2011 10:04 AM - edited on 12-22-2017 09:20 AM by Gagandeep Kaur
This document provides some extra documentation and use cases on the use of port spanning or port mirroring.
You can monitor traffic passing in & out of a set of L2 or L3 Ethernet interfaces (including bundle-Ether).
ASR 9000 is the only platform implementing SPAN on XR (Only support on ethernet linecards, not on SIP-700.)
You can use SPAN/Mirror in the follow scenarios
- L2 & L3 interfaces.
- Local, R-SPAN, and PW-SPAN only (no ER SPAN.)
- Scale limits:
8 monitor sessions
800 total source ports
1.5 Gig bidirectional replication limit toward fabric for bundle interfaces and 10 Gig ports.
Guideline: ~ 10% - 15% total bandwidth can be mirrored system-wide
- Source ports: Physical, EFPs, and bundles interfaces (L2 & L3)
- Destination ports: Ethernet interfaces, EFPs, and PW-SPAN. (No bundle) [ only L2 transport interfaces are supported as destination ports]
- Ability to use ACL's to define which traffic is to be captured
- Capture multicast traffic is possible
Note: some of the functionality mentioned are enhancements to the XR 4.0.1 release, this document assumes you are using this release or later.
A good reference on the terminology of SPAN/Mirror can be found here:
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/span.pdf
SPAN mirrors what is on the wire
For ingress, this means packets are mirrored before QOS, ACL, and encapsulation rewrite operations.
For egress, this means packets are mirrored after QOS, ACL, and encapsulation rewrite operations.
User can configure to mirror first 64 upto 256 bytes of the packet.
Note: The actual mirrored packet will be the configured size plus 4-byte trailling CRC.
Sample config:
interface GigabitEthernet0/6/0/20 l2transport
monitor-session PW
mirror first 100 <== valid range: [64, 256], inclusively
!
!
Note: The mirrored packet received at sniffer will have the size of 104
(4-byte of trailing CRC added by transmit MAC layer.)
“permit/deny” determines the behavior of the regular traffic (forwarded or dropped)
“capture” determines whether the packet is mirrored to the SPAN destination.
On SPAN: mirror traffic on the wire (regardless with or without ACL.)
ACL on ingress direction:
SPAN will mirror traffic even regular traffic dropped by ACL: Always mirror!
ACL on egress direction
Will mirror if regular traffic is forwarded (Permit)
Will not mirror if regular traffic is dropped (Deny.)
Inconsistent configurations:
“acl” is configured on SPAN source port but
ACL has no “capture” keyword:
No traffic gets mirrored.
“acl” is NOT configured on SPAN source port but
ACL has “capture” keyword:
Mirroring traffic as normal, no ACL performed.
The ACL can also be an L2 ACL :
ethernet-services access-list esacl_t2
10 deny 1234.5678.90ab 0000.0000.0000 any capture
monitor-session TEST
destination interface GigabitEthernet0/1/0/2 (<<<< this is NP3)
!
interface GigabitEthernet0/1/0/14 (<<<< this is NP2)
ipv4 address 5.5.1.1 255.255.255.0
monitor-session TEST
acl
!
load-interval 30
ipv4 access-group span ingress
!
ipv4 access-list span
10 permit ipv4 any host 1.1.1.10 capture
15 permit ipv4 any host 239.1.1.1 capture
20 permit ipv4 any host 2.2.2.100
30 permit ipv4 any any
Sample TRAFFIC GEN: (sending multicast in this example)
tgn rate 1000
L2-dest-addr 0100.5E01.0101
L2-src-addr 0003.A0FD.28A8
L3-src-addr 5.5.1.2
L3-dest-addr 239.1.1.1
Checking NP2: (the port that we are spanning)
Show global stats counters for NP2, revision v3
Read 12 non-zero NP counters:
Offset Counter FrameValue Rate (pps)
-------------------------------------------------------------------------------
22 PARSE_ENET_RECEIVE_CNT 5478 1001
31 PARSE_INGRESS_DROP_CNT 3 1
33 RESOLVE_INGRESS_DROP_CNT 5474 1000
(there is no mcast recipient for this mcast addr, but we’re still replicating, see red line)
40 PARSE_INGRESS_PUNT_CNT 1 0
50 MODIFY_RX_SPAN_CNT 5475 1000
54 MODIFY_FRAMES_PADDED_CNT 5475 1000
68 RESOLVE_INGRESS_L3_PUNT_CNT 1 0
104 LOOP 1 0
224 PUNT_STATISTICS 9 2
480 RESOLVE_IPM4_ING_RTE_DROP_CNT 5475 1000
565 UIDB_TCAM_MISS_AGG_DROP 3 1
570 UIDB_TCAM_MISS_PORT4_DROP_FOR_HOST 3 0
NP3 is the span monitor interface:
Show global stats counters for NP3, revision v3
Read 16 non-zero NP counters:
Offset Counter FrameValue Rate (pps)
-------------------------------------------------------------------------------
22 PARSE_ENET_RECEIVE_CNT 36 0
23 PARSE_FABRIC_RECEIVE_CNT 79656 1000
30 MODIFY_ENET_TRANSMIT_CNT 79655 1000
Packets received from fabric and sent off to the Ethernet on the span port!
For PW span to work, you need to define a local monitor session with a destination pseudo wire. You apply that span session to the interface of interest and define an xconnect group that also leverages that span session as one of the pw ends.
On the remote side where the PW terminates, you just configure regular VPWS.
Here an example:
On the Local Side, besides my Span configuration, there is also a local cross connect between the interested session we want to span over the PW
l2vpn
xconnect group TEST
p2p TEST
interface GigabitEthernet0/1/0/39
! port 39 is the port where we apply the span on.
interface GigabitEthernet0/1/0/20.100
! this is just a random AC to have traffic flowing between the spanned port.
!
AC configuration:
interface GigabitEthernet0/1/0/20.100 l2transport
encapsulation dot1q 100
rewrite ingress tag pop 1 symmetric
! the tag is popped because the other XCON end is a plain ethernet without vlan. The explanation and use cases of tag popping can be found a related
! Tech note article.
Regular VPWS configuration:
RP/0/RSP0/CPU0:A9K-TOP#sh run l2vpn
l2vpn
xconnect group PW-SPAN
p2p PW-SPAN_1
interface GigabitEthernet0/0/0/39
neighbor 2.2.2.2 pw-id 1
!
!
!
interface GigabitEthernet0/0/0/39
load-interval 30
transceiver permit pid all
l2transport
!
!
the neighbor in the l2vpn configuration is the LDP neighbor ID
between which the PW is built.
Show on remote side:
RP/0/RSP0/CPU0:A9K-TOP#show l2vpn xcon group PW-SPAN det
Group PW-SPAN, XC PW-SPAN_1, state is up; Interworking none
AC: GigabitEthernet0/0/0/39, state is up
Type Ethernet
MTU 1500; XC ID 0x4000a; interworking none
Statistics:
packets: received 0, sent 16570475
bytes: received 0, sent 994228500
! packets received from the PW are sent out hte Attachment circuit's interface. The analyzer is connected to G0/0/0/39
PW: neighbor 2.2.2.2, PW ID 1000, state is up ( established )
PW class not set, XC ID 0x4000a
Encapsulation MPLS, protocol LDP
PW type Ethernet, control word disabled, interworking none
PW backup disable delay 0 sec
Sequencing not set
MPLS Local Remote
------------ ------------------------------ -----------------------------
Label 16002 16027
Group ID 0xa40 0x2
Interface GigabitEthernet0/0/0/39 PW/TM/MS
MTU 1500 1500
Control word disabled disabled
PW type Ethernet Ethernet
VCCV CV type 0x2 0x2
(LSP ping verification) (LSP ping verification)
VCCV CC type 0x6 0x6
(router alert label) (router alert label)
(TTL expiry) (TTL expiry)
------------ ------------------------------ -----------------------------
MIB cpwVcIndex: 4294705162
Create time: 04/04/2011 14:36:42 (00:20:07 ago)
Last time status changed: 04/04/2011 14:36:42 (00:20:07 ago)
Statistics:
packets: received 16570475, sent 0
bytes: received 994228500, sent 0
! Packets received on the Pseudo Wire from the SPAN port
NOTE: Pseudo Wire counters on the span side are not incrementing.That is the XCON group "cisco" in this picture config example.
This is intentional. You can review the SPANNING also with this command:
RP/0/RSP1/CPU0:A9K-BOTTOM#sh monitor-session counters
Monitor-session PW_TM_MS
GigabitEthernet0/1/0/39
Rx replicated: 58488205 packets, 3743245120 octets
Tx replicated: 58488206 packets, 3743245184 octets
Non-replicated: 0 packets, 0 octets
R-SPAN is natively support with the capability of ASR9000 to do vlan imposition:
monitor-session MS2
destination interface gig0/2/0/19.10
!
interface gig0/2/0/12.10 l2transport
encapsulation dot1q 10 <<< Monitoring vlan 10 traffic
monitor-session MS2
!
interface gig0/2/0/19.10 l2transport (*)
encapsulation dot1q 100 <<< VLAN 100 will get imposed.
!
(*) Monitor destination could be any supported destination interface regardless of monitor source
n/a
Xander Thuijs, CCIE #6775
Sr. Tech Lead ASR9000
hey michael! thank you!! that's very nice to hear!! so hey for this situation, some options are possible.
natively, this won't work since as you noted there is only one destination interface per monitor session.
what you could do is, if you have a few ports to spare is to make a loopback cable between 2 interfaces, say X and Y. Then your destination interfaces are A B and C.
Now you create Y as an L2 transport interface, and A,B and C also as l2transport interfaces either with or without vlan as needed. Then create a bridge group and domain call it SPAN or so.
add the interfaces Y, A, B and C in there. disable mac learning in the bridge domain. This will result in flooding to all ports except originator, so we replicate the incoming packets effectively over to all ports in the BD.
then in the span session you make X the destination interface.
Now all packets spanned to X get looped via cable to Y into the BD and flooded to A B and C.
acceptable?
cheers!
xander
Wow, that's creative as heck Xander haha. I like it. Let me run it by the customer and see if this method will work for him. Either way, this is interesting to note. Thanks for the prompt response!
-Michael
Hello Xander,
Is RSPAN supported on GRE interfaces (as source port) on ASR9k, 5.3.3?
Thanks,
Dan
Hi dan,
spanning from a GRE (as a source) or towards (ERSPAN) is currently not working for span...
xander
Brilliant !!! Who would have thought :-)
Thanks Michael for asking the question and Xander to provide such brilliant solution that benefits us all .... got to take my hat off :-)
Thanks.
Hey Xander,
Just quick question about SPAN setup on ASR9006, version 4.3.1. does it support mutli 10Gi to mirror mutli 10Gi for typhoon line card? how's CPU or memory consumption for the device if we need to run this span on 24/7 all the time ?
thanks,
Eric.
for typhoon linecards your limit is merely the pps performance of the NPU that needs to replicate.
depending on the LC type and packet size the "standard" rate of pps of the NPU has some cycles "to spare".
these cycles can be used for recirculation that is required for span.
note that from fab (egress) or from line (ingress) are always high priority for the NPU to process. pipeline replication (eg span) is lower priority. so spanned traffic would never push away in or egress traffic.
If you have small apcket size and using a 36x10 (6 intf per NPU) there will be little pps to spare for span, but for a high performance card like 24x10 (3 intf per npu) or a MOD160 with a 2x10G mpa (1intf per npu) and as packet sizes increase there is room for line rate replication.
xander
Hi Xander,
Greetings.
My customer is implementing Monitor-Session on ASR9K.
They have a Bundle as the source interface and the same bundle has multiple sub-interfaces. They have configured Monitor-Session under the main interface and not under the sub-interfaces. Will this work?
Their destination is a Gig interface with the below config.
interface GigabitEthernet0/0/0/25
description 0401.04_TRNPHI22_X35
monitor-session Troubleshooting ethernet
Thanks in Advance,
Sachin A
hi sachin,
you can apply the monitor session on the main or subinterface, but only traffic matching that interface will get spanned. if there is no ip add on the main interface, then there is no forwarding on it either and therefore will not span anything.
the destination interface has to be of the l2transport kind.
oh, the config you have on the destination interface has a monitor session applied, this monitor session is configured on the bundle (sub)interface, and the monitor configuration points to g 0/0/0/25 as destination interface.
xander
Hello xthuijs,
I have this sceneario on a (NV - ASR9010 - IOS 5.1.1)
interface Bundle-Ether18
description xxxxxx
ipv4 address 10.3.84.58 255.255.255.252
bundle load-balancing localize threshold links 1
ipv4 access-group ABF-SALIDA ingress
The interfaces that belongs to the BE are:
TenGigE0/0/1/1 Full-duplex 10000Mb/s Active
TenGigE0/1/1/0 Full-duplex 10000Mb/s Active
When I try to apply the monitor session to the BE, this is what router shows:
interface bundle-ether 18
monitor-session VOIP_CMTS ethernet
!! SEMANTIC ERRORS: This configuration was rejected by
!! the system due to semantic errors. The individual
!! errors with each failed configuration command can be
!! found below.
interface Bundle-Ether18
monitor-session VOIP_CMTS ethernet
!!% 'CfgMgr' detected the 'fatal' condition 'This configuration has not been verified and can not be accepted by the system.'
But if I apply to another interface that is not BE it works perfect.
monitor-session VOIP_CMTS ethernet
destination interface GigabitEthernet0/0/0/4
interface GigabitEthernet0/0/0/4
l2transport
What I´m doing wrong?
Thanks for your help.
hi evmartinez, this is supported and should work. it could be that the bundle intf is in an inconsistent state (check show bundle be18).
it is generally caused by the bundle not reporting its state correctly to sysdb which is read by config manager.
some bugs for bundles in that regard exist, 511 is not a good release for that.
you can mitigate this possibly with an RSP switchover and let the system stabilize and reconfigure it again just as you're doing it.
cheers xander
hi dmitry, you probably want to look at:
PARSE_ENET_RCV (receive from wire)
PARSE_FAB_RCV (receive from fabric for transmission to wire)
this because if you look at TX WIRE you may miss the packets that you got from fabric but say got dropped by ACL or QOS
PARSE_LOOP_ (these are packets that are recevied for a loop for instance when BVI is in play or spanned/netflow traffic), they need to be processed also.
an ACL takes about 25% hit on the total PPS by itself. with ACL span you can control a bit better what you want to span.
so without ACL you'd span everything basically making it a 1 for 1 hit (50% loss).
with ACL you take a base hit of 25% and then an additional hit for those packets that match. so depending on what your acl matches, it can be quite beneificial to handle span with an ACL, in case you're only interested in a subset of the traffic!
cheers!
xander
In regards to limits:
1.5 Gig bidirectional replication limit toward fabric for bundle interfaces and 10 Gig ports.
and also
http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r4-1/interfaces/configuration/guide/hc41asr9kbook/hc41span.pdf
it states:
Performance Impact with Traffic Mirroring
Cisco recommends that you do not mirror more than 15% of your total transit traffic. On Ten Gigabit Ethernet interfaces or bundle interfaces there is a limit of 1.5G on each ingress amount to be mirrored and 1.5G on each egress amount to be mirrored.
We have a 10Gig port being SPANned on an ASR9010 and it far exceeds 1.5G (below is the destination port):
monitor-session No1 ethernet
destination interface TenGigE0/1/1/4
TenGigE0/1/1/4 is up, line protocol is up
Interface state transitions: 49
Hardware is TenGigE, address is 5087.896c.1f3c (bia 5087.896c.1f3c)
Layer 1 Transport Mode is LAN
Description: SPAN port destination
Internet address is Unknown
MTU 1514 bytes, BW 10000000 Kbit (Max: 10000000 Kbit)
reliability 255/255, txload 88/255, rxload 0/255
Encapsulation ARPA,
Full-duplex, 10000Mb/s, link type is force-up
output flow control is off, input flow control is off
loopback not set,
Last input never, output 00:00:00
Last clearing of "show interface" counters never
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 3454031000 bits/sec, 462697 packets/sec
We do not see any performance hit. Is there something we are missing?
Thanks!
hi hank, the 1.5G limitation is a fabric bw issue on trident. on trident LCs you're bound by the pps and bw and generally at larger packet sizes the bw is the first one you'll hit.
typhoon and tomahawk LCs dont have that bw limitation, since they have higher bw FIA asics.
the limitation for those cards is really pps and that only.
cheers
xander
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: