VM lost network connectivity after period of time

[techmail4sam]

Hi,

There is a port-profile "vmdata192" for VM data traffic on VSM and "test_acl" ACL is been associated with this port-profile as follows.

n1000v(config)# port-profile vmdata192

n1000v(config-port-prof)# ip port access-group test_acl out

n1000v(config-port-prof)# copy run start

[########################################] 100%

n1000v(config-port-prof)#

VM1's NIC is assigned this port-profile.

Everything works fine according to defined ACLs, but after a period of time VM have no network connectivity.

Once I remove ESX host from N1KV and again add to N1KV it works fine again.

What could be the cause of this problem?

Thanks

I

[Robert Burns]

Can you please post the ACL config.

Thanks,

Robert

[techmail4sam]

Hi,

ACL configuration is as follows.

IP access list test_acl
        statistics per-entry
        10 permit icmp any 192.168.4.73/32 echo-reply [match=1261]
        20 permit icmp any 192.168.4.73/32 echo [match=4]
        30 permit tcp any eq www 192.168.4.73/32 [match=91]
        40 permit tcp any 192.168.4.73/32 eq www [match=5]
        50 permit udp any range netbios-ns netbios-dgm any [match=15060]
        60 permit tcp any eq 139 any [match=413]
        70 permit tcp any eq 445 any [match=231]
        80 permit udp any eq domain 192.168.4.73/32 [match=3]

192.168.4.73 is VM's IP address.

[Robert Burns]

1. Does the veth for this VM just disapear when this issue happens?

2.  Is there a set time cycle this occurs or is it random?

Can you post "show int vethxxxx" on the VMs virtual interface before and after this occurs?

Thanks,

Robert

[Robert Burns]

Also can you post a "show run".

Feel free to remove any confidential data/addressing if needed.

Robert

[techmail4sam]

Hi,

Currently, In working condition it looks like as follows.

show interface vethernet 2
Vethernet2 is up
    Port description is esx - 41 - w2k3 ent sp2, Network Adapter 1
    Hardware is Virtual, address is 0050.5683.12ee
    Owner is VM "esx - 41 - w2k3 ent sp2", adapter is Network Adapter 1
    Active on module 3
    VMware DVS port 33
    Port-Profile is vmdata192
    Port mode is access
    Rx
    6863 Input Packets 4256 Unicast Packets
    8 Multicast Packets 2599 Broadcast Packets
    916131 Bytes
    Tx
    186581 Output Packets 53481 Unicast Packets
    1270 Multicast Packets 131830 Broadcast Packets 7 Flood Packets
    43113011 Bytes
    5 Input Packet Drops 0 Output Packet Drops

once problem reoccurs, I will post output of same command.

Attached file contains output of "show running-config" command.

Thanks

[Robert Burns]

Can you also confirm that when you VM loses connectivity that your VEM host is still connected.

Next time you notice the problem can you issue:

"show mod" and verify all your modules are still connected.

Thanks,

Robert

[techmail4sam]

Hi,

Problem reoccured.

I checked all the things you suggested.

show module- displays VSM & VEM both.

show interface brief

--------------------------------------------------------------------------------
Port     VRF          Status IP Address                     Speed    MTU
--------------------------------------------------------------------------------
mgmt0     --           up     192.168.4.104                  1000     1500

--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth3/2        192    eth  trunk  up      none                        100(D) --

--------------------------------------------------------------------------------
Interface     VLAN   Type Mode   Status  Reason                   MTU
--------------------------------------------------------------------------------
Veth1         192    virt access up      none                     1500
Veth2         192    virt access up      none                     1500

Both VM's interface Veth1 & Veth2 are up.

Veth1 have connectivity to Veth2 but not to outside network.

Veth2 have connectivity neither to Veth1 nor outside network.

Thanks

[admin11111]

Can you please collect the output of the following command when problem is recreated.

module vem 3 execute vemcmd show port

module vem 3 execute vemcmd show trunk

module vem 3 execute vemcmd show bd

module vem 3 execute vemcmd show l2 all

[techmail4sam]

Hi,

Output of suggested commands is as follows.

Attached file is a snapshot of vCenter switch configuration.

n1000v(config)# module vem 3 execute vemcmd show port
  LTL    IfIndex   Vlan    Bndl  SG_ID Pinned_SGID  Type  Admin State  CBL Mode   Name
    8          0   3969       0      2           2  VIRT     UP    UP    4 Access l20
    9          0   3969       0      2           2  VIRT     UP    UP    4 Access l21
   10          0    190       0      2           2  VIRT     UP    UP    4 Access l22
   11          0   3968       0      2           2  VIRT     UP    UP    4 Access l23
   12          0    191       0      2           2  VIRT     UP    UP    4 Access l24
   13          0      1       0      2           2  VIRT     UP    UP    0 Access l25
   14          0   3967       0      2           2  VIRT     UP    UP    4 Access l26
   16   1a020100      1 T     0      2           2  PHYS     UP    UP    4  Trunk vmnic1
   47   1b020000    192       0      2           2  VIRT     UP    UP    4 Access esx - 41 - w2k3 ent sp2.eth0

n1000v(config)# module vem 3 execute vemcmd show trunk
Trunk port 16 native_vlan 1 CBL 4
vlan(1) cbl 4, vlan(190) cbl 4, vlan(191) cbl 4, vlan(192) cbl 4,

n1000v(config)# module vem 3 execute vemcmd show bd
BD 1, vdc 1, vlan 1, 0 ports
Portlist:
Multicast Group Table:
Group 239.255.255.250 RID 1 Multicast LTL 4407
    16

BD 190, vdc 1, vlan 190, 2 ports
Portlist:
     10  l22
     16  vmnic1

BD 191, vdc 1, vlan 191, 2 ports
Portlist:
     12  l24
     16  vmnic1

BD 192, vdc 1, vlan 192, 2 ports
Portlist:
     16  vmnic1
     47  esx - 41 - w2k3 ent sp2.eth0

Multicast Group Table:
Group 239.255.255.250 RID 1 Multicast LTL 4407
    16

BD 3967, vdc 1, vlan 3967, 1 ports
Portlist:
     14  l26

BD 3968, vdc 1, vlan 3968, 3 ports
Portlist:
      1  inband
      5  inband port security
     11  l23

BD 3969, vdc 1, vlan 3969, 2 ports
Portlist:
      8  l20
      9  l21

n1000v(config)# module vem 3 execute vemcmd show l2 all

Bridge domain 1 brtmax 1024, brtcnt 36, timeout 300
    Dynamic MAC 00:0f:ea:93:78:26  LTL    16             timeout        34
    Dynamic MAC 00:0f:ea:92:be:3b  LTL    16             timeout        24
    Dynamic MAC 00:0d:60:c9:48:7a  LTL    16             timeout        97
    Dynamic MAC 00:14:85:93:f4:0b  LTL    16             timeout       264
     Static MAC 00:02:3d:60:64:02  LTL     5             timeout         0
    Dynamic MAC 00:05:5d:4a:5e:82  LTL    16             timeout        34
    Dynamic MAC 00:1d:92:0a:af:0e  LTL    16             timeout         6
    Dynamic MAC 00:05:5d:4a:cf:a0  LTL    16             timeout       119
     Static MAC 00:02:3d:30:64:02  LTL     3             timeout         0
    Dynamic MAC 00:19:d1:3e:9a:98  LTL    16             timeout       101
    Dynamic MAC 00:14:85:94:b2:6b  LTL    16             timeout        10
    Dynamic MAC 00:14:85:98:99:6f  LTL    16             timeout       129
    Dynamic MAC 00:19:d1:61:47:28  LTL    16             timeout         6
    Dynamic MAC 00:0f:ea:92:97:2f  LTL    16             timeout       139
Bridge domain 191 brtmax 1024, brtcnt 2, timeout 300
    Dynamic MAC 00:0c:29:8f:7a:64  LTL    16             timeout        29
    Dynamic MAC 00:02:3d:20:64:02  LTL    12             timeout        55
Bridge domain 192 brtmax 1024, brtcnt 1, timeout 300
     Static MAC 00:50:56:83:12:ee  LTL    47             timeout     66934
Bridge domain 3967 brtmax 1024, brtcnt 0, timeout 300
Bridge domain 3968 brtmax 1024, brtcnt 3, timeout 300
    Dynamic MAC 00:0c:29:8f:7a:64  LTL    11             timeout        29
     Static MAC 00:02:3d:60:64:02  LTL     5             timeout         0
     Static MAC 00:02:3d:20:64:02  LTL     1             timeout     66883
Bridge domain 3969 brtmax 1024, brtcnt 2, timeout 300
    Dynamic MAC 00:02:3d:40:64:02  LTL     8             timeout         0
    Dynamic MAC 00:0c:29:8f:7a:50  LTL     9             timeout         0

Thanks

[admin11111]

When this problem occurs,  is the ip address of the interfaces valid, ( just wondering if the ip address is not renewed through DHCP after some time, if it is DHCP assigned ip addres.  Does  Pinging  the loopback ip addresses, and VM's own ip address work once we hit this situation. (To clarify that TCP/IP on the host does not have problems)

Once the problem is hit, to verify if ACL is causing problem, follow the steps below, it will give more insight into the problem

1.Any traffic that is not matched to any of the above configured rules will be dropped automatically. So if you introduce the new rule

  "permit ip any any" at the end for short span of time, and see if the network connectivity is restored or not. If the network connectivity

   is restored, it points to that original ACL rule configuration is dropping packets which is needed for network connectivity. Then we have to

   add more "permit" rules to allow packets to restore the network connectivity.

   Also verifying the interface packet counters on VM side and N1K interface will be useful, to clarify that packets are reaching on N1K interface from this VM.

2. Verify ACL statistics by running  the "show ip access list"  and monitor the statistics counters. In the system configuration attached above, statistics is

enabled for this acl.

3. Does the problem happen even without ACL configuration. If the problem is reproducible in farily short amout of time, you can quicky verify it

by removing the ACL on this interface and see how system behaves.

4. After running Steps 1 (and Step 3 (if tested)), and after verifying ACL is causing the problem, logs on VEM is needed to see the debug this issue.

    Step a: Verify that acls are installed properly on DP (please provide this info)

          a.module vem < module no> execute vemcmd show acl

          b. module vem < module no> excute vemcmd show pinst 

   We need packet level debugging.

    Packet level debugging, so use with caution:  Make sure that packet is reached VEM from that VM while DP logs are enabled.

     c.module vem <module no> execute vemdebug sfacl all  (enable acl logging)

     d. module vem <module no> execute vemlog show all     (provide this output)

     e. module vem <module no> execute vemlog sfacl none (disable acl logging)

     need to make sure that Packet from VM is reached the VEM between Step C and Step E.

Srini

[admin11111]

Hi,

It doesn't shows any ACL in "show pinst" commands output.

Required output is as follows.

n1000v(config-acl)# module vem 3 execute vemcmd show acl
Acl-id Ref-cnt  Type  Numrules  Stats
    1      2    IPv4      7    disabled
    2      2    IPv4      7     enabled
n1000v(config-acl)# module vem 3 execute vemcmd show pinst
VEM Version: 4.0.4.1.1.27-0.4.2
VSM Version: 4.0(4)SV1(1)
System Version: VMware ESX 4.0.0 Releasebuild-171294
n1000v(config-acl)#

Thanks

[admin11111]

Did you get a chance to run the following steps and see what findings are

Step 1.  (try acl "permit ip any any") once you hit the problem.

Step 2.  If possible, Run without acl configuration on interface, and see if problem can be reproduced.

what are the stats counters reporting.