nexus1000v local SPAN can't capture inbound traffic

[davidcheung]

hi all,

I just configured local SPAN on nexus1000v (version 1.3d).

local SPAN source and destination is on same VEM.

my config is like below:

monitor session 3

  source interface Vethernet13 both

  destination interface Vethernet170

  destination interface Vethernet36

  no shut

SPAN session is up.

But we can't see any inbound traffic to the source VM.

(10.16.185.4,5,6 is the IPs of SPAN source)

[root@davidzhangRHEL ~]# tcpdump -i eth1

tcpdump: WARNING: eth1: no IPv4 address assigned

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes

11:46:07.644551 IP 10.16.185.3.hsrp > 224.0.0.2.hsrp: HSRPv0-hello 20: state=sta                                                ndby group=1 addr=10.16.185.1

11:46:07.654771 IP 10.16.185.2.hsrp > 224.0.0.2.hsrp: HSRPv0-hello 20: state=act                                                ive group=1 addr=10.16.185.1

11:46:07.961735 IP 10.16.185.6.https > 10.16.184.196.50254: S 3897896960:3897896                                                960(0) ack 1838046824 win 8192 <mss 1460,nop,wscale 8,sackOK,timestamp 74329579                                                 2654766205>

11:46:07.962955 IP 10.16.185.6.https > 10.16.184.196.50254: R 1:1(0) ack 2 win 0

11:46:10.644950 IP 10.16.185.3.hsrp > 224.0.0.2.hsrp: HSRPv0-hello 20: state=sta                                                ndby group=1 addr=10.16.185.1

11:46:10.657615 IP 10.16.185.2.hsrp > 224.0.0.2.hsrp: HSRPv0-hello 20: state=act                                                ive group=1 addr=10.16.185.1

11:46:11.081231 IP 10.16.185.5.https > 10.16.184.197.58538: S 1850399261:1850399                                                261(0) ack 3055844595 win 8192 <mss 1460,nop,wscale 8,sackOK,timestamp 74329891                                                 2654662655>

11:46:11.081970 IP 10.16.185.5.https > 10.16.184.197.58538: R 1:1(0) ack 2 win 0

11:46:11.957381 IP 10.16.185.5.https > 10.16.184.196.42161: S 1862096740:1862096                                                740(0) ack 970410175 win 8192 <mss 1460,nop,wscale 8,sackOK,timestamp 74329978 2                                                654770202>

11:46:11.958705 IP 10.16.185.5.https > 10.16.184.196.42161: R 1:1(0) ack 2 win 0

11:46:12.089401 IP 10.16.185.6.https > 10.16.184.197.45604: S 2733719434:2733719                                                434(0) ack 3290215780 win 8192 <mss 1460,nop,wscale 8,sackOK,timestamp 74329992                                                 2654663683>

11:46:12.090735 IP 10.16.185.6.https > 10.16.184.197.45604: R 1:1(0) ack 2 win 0

11:46:12.956018 IP 10.16.185.6.https > 10.16.184.196.50302: S 2275642708:2275642                                                708(0) ack 3286673454 win 8192 <mss 1460,nop,wscale 8,sackOK,timestamp 74330078                                                 2654771200>

11:46:12.956838 IP 10.16.185.6.https > 10.16.184.196.50302: R 1:1(0) ack 2 win 0

11:46:13.552716 IP 10.16.185.4.61913 > 10.2.222.111.5723: P 3867141198:386714222                                                3(1025) ack 4146771556 win 508

11:46:13.645770 IP 10.16.185.3.hsrp > 224.0.0.2.hsrp: HSRPv0-hello 20: state=sta                                                ndby group=1 addr=10.16.185.1

11:46:13.654427 IP 10.16.185.2.hsrp > 224.0.0.2.hsrp: HSRPv0-hello 20: state=act                                                ive group=1 addr=10.16.185.1

11:46:13.817143 IP 10.16.185.4.61913 > 10.2.222.111.5723: . ack 180 win 508

1000v# module vem 12 execute vemcmd show span

VEM SOURCE IP NOT CONFIGURED.

HW SSN ID            DST LTL/IP  ERSPAN ID  HDR VER
        0                    68      local
1000v# show monitor internal errors
1) Event:E_DEBUG, length:96, at 163774 usecs after Thu Sep 22 15:12:17 2011
    [102] eth_span_phy_if_init_runtime_info(1051): im_get_ifindex_span_mode_list returned 0x40e30005

2) Event:E_DEBUG, length:96, at 684704 usecs after Thu Sep 22 15:12:04 2011
    [102] eth_span_phy_if_init_runtime_info(1051): im_get_ifindex_span_mode_list returned 0x40e30005

anybody any suggestion. Your help is highly appreciated.

[mipetrin]

Hi David,

Are you saying that you see outbound traffic but do not see inbound, or you don't see anything traffic at all?

Please provide following output:

show monitor session 3

show monitor internal info session 3

show monitor internal info interface vethernet 13

show monitor internal info interface vethernet 36

show monitor internal info interface vethernet 170

show interface virtual | egrep "(13|36|170)"

module vem 12 execute vemcmd show span

module vem 12 execute vemcmd show port

Thanks,

Michael

[davidcheung]

Hi Michael,

Thanks. You are correct. We are able to see outbound traffic from SPAN source but not inbound traffic to SPAN source.

Note: I have done vmotion for the SPAN source and SPAN destination virtual machines.

Please see the below output which you requested.

1000v# show monitor session 1
   session 1
---------------
type              : local
state             : up
source intf       :
    rx            : Veth13
    tx            : Veth13
    both          : Veth13
source VLANs      :
    rx            :
    tx            :
    both          :
filter VLANs      : filter not specified
destination ports : Veth170   Veth36

1000v#

show monitor internal info session 1

Session 1 info:
FSM state: SESSION_STATE_OPER_ACTIVE
State reason: 0

*** ADMIN DATA ***
Session state: NO SHUT

Ingress sources
---------------
phy if: Veth13
port ch:
vlans:

Egress sources
--------------
phy if: Veth13
port ch:
vlans:

Destinations:
-------------
Veth170, Veth36

PSS source list:
----------------
Veth13

PSS destination list:
---------------------
Veth170, Veth36

*** RUNTIME DATA ***
hw_ssn_id: 0
destination index: 0x4fa3 (multicast di)

oper rx: Veth13
oper tx: Veth13
oper dest: Veth170, Veth36
oper dest for di: Veth170, Veth36
programmed rx: Veth13
programmed tx: Veth13
programmed dest: Veth170, Veth36
programmed dest for di: Veth170, Veth36

programmed filter rx:1500
programmed filter tx:

Lock Info: resource [Session ID(0x1)]
  type[0] p_gwrap[(nil)]
      FREE @ 97236 usecs after Sun Sep 25 12:14:25 2011
  type[1] p_gwrap[(nil)]
      FREE @ 580143 usecs after Tue Sep 27 01:53:06 2011
  type[2] p_gwrap[(nil)]
      FREE @ 520203 usecs after Sun Sep 25 12:48:23 2011
0x1
Use lock event history for more details

1000v# terminal length 0
1000v# show monitor internal info interface vethernet 13
Interface info:
if_index: 1c0000c0
source for ssn 1, src_dir 3
state: up
layer: 2
mode: access
Access vlan: 1500
Interface is not in switchport monitor mode
No Entries in SDB for if_index 0x1c0000c0
1000v# show monitor internal info interface vethernet 36
Interface info:
if_index: 1c000230
destination for ssn 1
state: up
layer: 2
mode: access
Access vlan: 1500
Interface is not in switchport monitor mode
The port is MISCONFIGURED, being not span destination but used as such
No Entries in SDB for if_index 0x1c000230
1000v# show monitor internal info interface vethernet 170
Interface info:
if_index: 1c000a90
destination for ssn 1
state: up
layer: 2
mode: access
Access vlan: 1500
Interface is not in switchport monitor mode
The port is MISCONFIGURED, being not span destination but used as such
No Entries in SDB for if_index 0x1c000a90
1000v# show interface virtual | egrep "(13|36|170)"
Veth13      Net Adapter 1  VM451             11  esx905
Veth36      Net Adapter 4  VM510             11  esx905
Veth113     Net Adapter 2  VM808             12  esx902
Veth130     Net Adapter 1  VMSDE449          12  esx902
Veth131     Net Adapter 3  VM809             10  esx904
Veth132     Net Adapter 2  VMSDE449          12  esx902
Veth134     Net Adapter 2  VM510             11  esx905
Veth135     Net Adapter 2  VM511             8   esx901
Veth136     Net Adapter 2  VM465             9   esx903
Veth137     Net Adapter 1  VM472             11  esx905
Veth138     Net Adapter 3  VM470             10  esx904
Veth139     Net Adapter 3  VM472             11  esx905
Veth151     Net Adapter 1  VMSDE436          11  esx905
Veth152     Net Adapter 2  VMSDE436          11  esx905
Veth170     Net Adapter 2  RHEL5             11  esx905

1000v# module vem 11 execute vemcmd show port
  LTL    IfIndex   Vlan    Bndl  SG_ID Pinned_SGID  Type  Admin State  CBL Mode   Name
    8          0   3969       0     32          32  VIRT     UP    UP    4 Access
    9          0   3969       0     32          32  VIRT     UP    UP    4 Access
   10          0   1513       0     32           7  VIRT     UP    UP    4 Access
   11          0   3968       0     32          32  VIRT     UP    UP    4 Access
   12          0   1514       0     32           8  VIRT     UP    UP    4 Access
   13          0      1       0     32          32  VIRT     UP    UP    0 Access
   14          0   3971       0     32          32  VIRT     UP    UP    4 Access
   15          0   3971       0     32          32  VIRT     UP    UP    4 Access
   16   1a0a0000   1600 T   307      0          32  PHYS     UP    UP    4  Trunk vmnic0
   18   1a0a0200    616 T   306      2          32  PHYS     UP    UP    4  Trunk vmnic2
   19   1a0a0300      1 T   305      3          32  PHYS     UP    UP    1  Trunk vmnic3
   20   1a0a0400      1 T   305      4          32  PHYS     UP    UP    1  Trunk vmnic4
   21   1a0a0500   1600 T   307      5          32  PHYS     UP    UP    4  Trunk vmnic5
   23   1a0a0700      1 T   304      7          32  PHYS     UP    UP    1  Trunk vmnic7
   24   1a0a0800      1 T   304      8          32  PHYS     UP    UP    1  Trunk vmnic8
   25   1a0a0900    616 T   306      9          32  PHYS     UP    UP    4  Trunk vmnic9
   48   1b0a0000   1500       0     32           3  VIRT     UP    UP    4 Access VM510 ethernet3
   49   1b0a0010    620       0     32           9  VIRT     UP    UP    4 Access VM510 ethernet2
        pvlan isolated 616 620
   50   1b0a0020   1500       0     32           4  VIRT     UP    UP    4 Access VM510 ethernet1
   51   1b0a0030   1620       0     32           0  VIRT     UP    UP    4 Access VM480 ethernet2
        pvlan isolated 1600 1620
   52   1b0a0040    620       0     32           2  VIRT     UP    UP    4 Access VM480 ethernet1
        pvlan isolated 616 620
   53   1b0a0050   1502       0     32           3  VIRT     UP    UP    4 Access VM480 ethernet0
   54   1b0a0060   1509       0     32           4  VIRT     UP    UP    4 Access fiserv-f5 ethernet2
   55   1b0a0070    620       0     32           9  VIRT     UP    UP    4 Access fiserv-f5 ethernet1
        pvlan isolated 616 620
   56   1b0a0080   1512       0     32           7  VIRT     UP    UP    4 Access fiserv-f5.eth0
   57   1b0a0090   1620       0     32           5  VIRT     UP    UP    4 Access VM459 ethernet2
        pvlan isolated 1600 1620
   58   1b0a00a0    620       0     32           2  VIRT     UP    UP    4 Access VM459 ethernet1
        pvlan isolated 616 620
   59   1b0a00b0   1501       0     32           3  VIRT     UP    UP    4 Access VM459 ethernet0
   60   1b0a00c0    620       0     32           2  VIRT     UP    UP    4 Access VM476 ethernet2
        pvlan isolated 616 620
   61   1b0a00d0   1501       0     32           4  VIRT     UP    UP    4 Access VM476 ethernet1
   62   1b0a00e0   1620       0     32           0  VIRT     UP    UP    4 Access VM476 ethernet0
        pvlan isolated 1600 1620
   63   1b0a00f0    620       0     32           2  VIRT     UP    UP    4 Access VM451 ethernet3
        pvlan isolated 616 620
   64   1b0a0100   1620       0     32           5  VIRT     UP    UP    4 Access VM451 ethernet2
        pvlan isolated 1600 1620
   65   1b0a0110   1500       0     32           3  VIRT     UP    UP    4 Access VM451 ethernet0
   66   1b0a0120   1620       0     32           0  VIRT     UP    UP    4 Access VMSDE440 ethernet1
        pvlan isolated 1600 1620
   67   1b0a0130   1508       0     32           4  VIRT     UP    UP    4 Access VMSDE440 ethernet0
   68   1b0a0140   1509       0     32           3  VIRT     UP    UP    4 Access VM501 ethernet0
   72   1b0a0180   1620       0     32           0  VIRT     UP    UP    4 Access VMSDE436 ethernet1
        pvlan isolated 1600 1620
   73   1b0a0190   1508       0     32           3  VIRT     UP    UP    4 Access VMSDE436 ethernet0
   74   1b0a01a0    620       0     32           2  VIRT     UP    UP    4 Access VM477 ethernet3
        pvlan isolated 616 620
   75   1b0a01b0   1620       0     32           5  VIRT     UP    UP    4 Access VM477 ethernet1
        pvlan isolated 1600 1620
   76   1b0a01c0   1501       0     32           3  VIRT     UP    UP    4 Access VM477 ethernet0
   77   1b0a01d0   1620       0     32           0  VIRT     UP    UP    4 Access VMSDE434 ethernet1
        pvlan isolated 1600 1620
   78   1b0a01e0   1508       0     32           4  VIRT     UP    UP    4 Access VMSDE434 ethernet0
   79   1b0a01f0   1620       0     32           5  VIRT     UP    UP    4 Access VM454 ethernet3
        pvlan isolated 1600 1620
   80   1b0a0200    620       0     32           9  VIRT     UP    UP    4 Access VM454 ethernet2
        pvlan isolated 616 620
   81   1b0a0210   1501       0     32           4  VIRT     UP    UP    4 Access VM454 ethernet0
   82   1b0a0220   1620       0     32           0  VIRT     UP    UP    4 Access VM815 ethernet1
        pvlan isolated 1600 1620
   83   1b0a0230   1507       0     32           3  VIRT     UP    UP    4 Access VM815 ethernet0
   87   1b0a0270   1620       0     32           0  VIRT     UP    UP    4 Access VMSDE405 ethernet1
        pvlan isolated 1600 1620
   88   1b0a0280   1509       0     32           3  VIRT     UP    UP    4 Access VMSDE405 ethernet0
   89   1b0a0290   1620       0     32           5  VIRT     UP    UP    4 Access VMSDE424 ethernet1
        pvlan isolated 1600 1620
   90   1b0a02a0   1509       0     32           3  VIRT     UP    UP    4 Access VMSDE424 ethernet0
   91   1b0a02b0    620       0     32           9  VIRT     UP    UP    4 Access VM472 ethernet2
        pvlan isolated 616 620
   92   1b0a02c0   1620       0     32           0  VIRT     UP    UP    4 Access VM472 ethernet1
        pvlan isolated 1600 1620
   93   1b0a02d0   1500       0     32           4  VIRT     UP    UP    4 Access VM472 ethernet0
   94   1b0a02e0   1508       0     32           4  VIRT     UP    UP    4 Access VMSDE431 ethernet1
   95   1b0a02f0   1620       0     32           5  VIRT     UP    UP    4 Access VMSDE431 ethernet0
        pvlan isolated 1600 1620
   96   1b0a0300   1620       0     32           0  VIRT     UP    UP    4 Access VM496 ethernet2
        pvlan isolated 1600 1620
   97   1b0a0310   1501       0     32           3  VIRT     UP    UP    4 Access VM496 ethernet1
   98   1b0a0320   1500       0     32           4  VIRT     UP    UP    4 Access VM496 ethernet0
   99   1b0a0330   1620       0     32           5  VIRT     UP    UP    4 Access VM510 ethernet0
        pvlan isolated 1600 1620
  100   1b0a0340   1500       0     32           4  VIRT     UP    UP    4 Access RHEL5 ethernet1
  101   1b0a0350   1512       0     32           8  VIRT     UP    UP    4 Access RHEL5.eth0
  102   1b0a0360   1620       0     32           0  VIRT     UP    UP    4 Access VM452 ethernet3
        pvlan isolated 1600 1620
  103   1b0a0370    620       0     32           2  VIRT     UP    UP    4 Access VM452 ethernet2
        pvlan isolated 616 620
  104   1b0a0380   1500       0     32           3  VIRT     UP    UP    4 Access VM452 ethernet0
  304   16000028      1 T     0     32          32  VIRT     UP    UP    1  Trunk
  305   16000029      1 T     0     32          32  VIRT     UP    UP    1  Trunk
  306   1600002a    616 T     0     32          32  VIRT     UP    UP    4  Trunk
  307   1600002b   1600 T     0     32          32  VIRT     UP    UP    4  Trunk
1000v# module vem 11 execute vemcmd show span

VEM SOURCE IP NOT CONFIGURED.

HW SSN ID            DST LTL/IP  ERSPAN ID  HDR VER
        0                  4408      local
1000v#

[mipetrin]

Hi David,

Thanks for getting all the output. I see that you have a TAC case open on this matter and are having a live troubleshooting session with the TAC.  That is the best way forward for this issue, as nothing immediately stands out. Seeing the issue first hand always helps.

Work with the TAC moving forward and once final resolution is available, please update this thread. This is so others who may be hitting your issue also know what to do.

Thanks,

Michael

[mipetrin · Accepted solution]

All,

Looking at the TAC service request, this issue has been tracked and resolved via the following Bug:

CSCtt03189 - VM's must be pinned to same SGID for SPAN to work

Externally found moderate (Sev3) bug: R-Resolved

Symptom:

- Packets inbound to the span source interface are missing from span traffic.

Conditions:

- Host has Port-channel(s) configured in vPC host mode either using sub-group cdp or mac-pinning option.

- Local span session is configured with source interface(s) and destination vethernet interface(s).

Workaround:

- Configure the local span destination interface(s) on an access VLAN that is not used in the system.

- Use "show vlan" command to identify which vlans are not used in the system so that they can be configured as access vlans on span destination vethernet ports

This bug fix will be available in an upcoming release of Nexus 1000v.

Thanks,

Michael

[CARL LINDAHL]

I looked this bug up in the bug tool and see that it is fixed in "release-pending". Am I correct to assume that it is NOT fixed in 4.2.1.SV1(4a)?

On a side note, anyone out there running 4.2.1.SV1(4a)?

[mipetrin]

Hi Carl,

That is correct. This issue is fixed in the next version, yet to be released.

Thanks,

Michael

[davidcheung]

Hi Michael,

Thank you very much for your update.

( I was on leave and went to a training course after Cisco helped to find workaround for the issue. This is the reason why I just come online)

Best Regards,