Nexus 7K packet loss issue

[peter.cheng]

Hello all,

Our scenario is as below:

                   DMZ switch ----- PC

                    |             |

                    |             |

                    |             |

                  FW         FW   (Checkpoint with VRRP connecting to N7k using VLAN 16)

                    |             |

                    L2 Switch -------- Laptop

                    | |           | |

                N7k-1 ----  N7k-2   (Peer Link Between N7k)

                    | |           | |

                    | |           | |

                   Inside switch ---- Server (VLAN16)

When user ping from DMZ switch PC to Server in the Inside switch, the packet loss and long response time happen intermittently.

But when we ping from Inside switch with another VLAN (VLAN12) to the server, it's okay. VLAN12 and VLAN16's  gateway are on N7k with HSRP.

So N7k's inter-vlan routing seems to be okay, but through FW has problem.

L2 switch and Inside switch connect to N7k with vPC. ALL the PC/Server are in VLAN 16 and their default gateway is to N7k.

If I connect a laptop which default gw is FW's VRRP ip, and PC at DMZ ping the laptop is okay.

When user ping from inside to DMZ we can see a icmp redirect message, and I don't know whether it could be the problem to cause the intermittent packet loss?

Thanks.

Peter

[Roman Rodichev]

Peter, if you try to ping Nexus2 SVI from checkpoint2, does it work? Are you running nxos5.1.2?

[peter.cheng]

Hello Roman,

Thanks for your response.

Whwn we ping from Nexus to the checkpoint FW, it's okay.

The NX-OS version is 5.0(5).

[peter.cheng]

Hello all,

I've escalate to Cisco TAC and got the answer.

It's caused by the lots of icmp redirect packet and make Nexus performance impact.

When Nexus replies the icmp redirect packet, it uses its real ip instead of HSRP VIP so that the PC/Server still send traffic to N7K.

So every time PC/Server send out every packet and N7K need to reply icmp redirect .

Cisco IOS doesn't have such issue since it will reply icmp redirect with source ip : VIP.

Peter

[nickeoannidis]

Hi Peter,

I had this issue with icmp redirects on nx 7009. It was when uploading files to a vmware datastore via nexus 7k / 5k / UCS fabric. The storage protocol is FCOE - so i kept looking at it for the issue when infact it was the switching path to the mgmt vmkernel - the default gatetway on the 7k had ip redirects by default - and packets kept looping impacting file upload to the vsphere datastore.

I hope this helps someone else in the future, and thanks for your explaination above Peter!

Regards,

Nick