6500 - High CPU utilization - Jumbo Frames

[Adnan Fakruddin]

Hello All,

I'm observing very high CPU utilization on 6509 when even a ping with packet size 9000 (Jumbo Frame) is being sent. The issue started when I added a Nexus 2232 10G switch to our Nexus (Server Farm) environment. Basically have 2 6500 with trunks to the Nexus 5500 series switches with multiple N2K FEXs. The uplinks are all 20G (port-channels) & Jumbo frames is cofigured on the trunks to NEXUS and as a class-map on N5K.

Other changes in environment were creating a dedicated VLAN for backup environement on each of the 10G servers. As of now when the backups commence during off-hours the core CPU averages between 60-70%.

The process on cpu dont give much as the highest utilized process are slcp, snmp engine, check heap which dont exceed 2-3%

There doesn't seem to be any performance degrdation, but concern is that (correct me if im wrong here) the core 6500 CPU should normally be lower than 10-15%. (I have SUP 2T as well).

Any information would be much appreciated.

Thank you.

Adnan

[dave cooper]

Out of curiosity did you ever find out what was causing the issues. We are running into the same problem

[Adnan Fakruddin]

Hello Dave,

The issue was with one of our servers which had 2x10GE & 2x1GE ports on it. The newly connected 10GE ports were in a new vlan created for backup purposes. However the new IP address for the 10GE NICs had the same FQDN as the 1GE ports, an administrative error, happens all the time:)

So when the backup would run for the server the cpu on 6500 would go wild. 

We performed a debug using "debug netdr capture" and then ran "show netdr capture packets" couple of times to identify the server causing interrupts on the CPU and resulting in high CPU utilization. These cmds need to be run at the precise time the CPU is hitting high. I would strongly recommend though to check with TAC, as your situation may be different.

In general though for 10GE ports connected to NEXUS environment need some care, I strongly suggest the below based on experience:

1. Make sure the 10GE NICs on the servers have the latest firmware. We've had issues with the firmware on the NIC and caused lot of time troubleshooting.

2. Identify your traffic patterns (example backup traffic during off-hours, which servers have most network bandwidth utilization) and make sure that you allot ports for servers accordingly on the 10G Nexus 2K FEXs due to over-subscription constraints. Backup environments are demanding and you don't want 4-5 servers with backups running for long hours at the same time on the same FEX. But if you must, sit with the backup/storage admin and try to schedule the backups in a way that the uplink to N5Ks is not overwhelmed. Also make sure that your uplinks from N2K to N5K have are at least 40 Gig. 

3. If you've enabled Jumbo Frames, make sure they're enabled end-end from not just the network point of view but the entire environment like VMware servers, etc.

4. Server administrators may try to play with the 10GE NIC settings, thinking they will get super high speed. Make sure they DONT. Pass them the document on this link. But if they still wanna play around, do it in a test environment and go through carefully what they're setting and trying to achieve.

I hope your issue gets resolved.

Regards,

Adnan