Introduction
The Cisco MDS switch family switches support interface configurations such as Trunks and PortChannels, Fibre Channel Port Rate Limiting, Extended Credits, and N Port Virtualization. Full line-rate Fibre Channel ports provide at least 255 buffer credits standard. Adding credits lengthens distances for Fibre Channel SAN extension. Using extended credits, up to 4095 buffer credits from a pool of more than 6000 buffer credits for a module can be allocated to ports as needed to greatly extend the distance for Fibre Channel SANs.
Problem
User is getting following error messages after upgrading the system to version 5.0.4.
Event Description:RMON_ALERT WARNING(4) Falling:iso.3.6.1.4.1.9.9.289.1.2.1.1.38.17305600=0 <= 0:65500, 4 Severity Level:2
Event Description:RMON_ALERT WARNING(4) Rising:iso.3.6.1.4.1.9.9.289.1.2.1.1.38.17305600=10 >= 10:65500, 4 Severity Level:2
Requirements
Basic understanding of Cisco MDS switches and NX OS commands.
NX OS Slow Drain Device Detection and Congestion Avoidance
All data traffic between end devices in a SAN fabric is carried by Fibre Channel Class 3, and in some cases, Class 2 services that use link-level, per-hop-based, and buffer-to-buffer flow control. These classes of service do not support end-to-end flow control. When there are slow devices attached to the fabric, the end devices do not accept the frames at the configured or negotiated rate. The slow devices lead to ISL credit shortage in the traffic destined for these devices and they congest the links.
Slow Drain Device Detection and Congestion Avoidance is focused mainly on the edge ports that are connected to slow drain devices. The goal is to avoid or minimize the frames stuck condition in the edge ports due to slow drain devices that are causing ISL blockage. No-credit timeout drops all packets once the slow drain is detected using the configured thresholds. The lesser frame timeout value helps to alleviate the slow drain condition that affects the fabric by dropping the packets on the edge ports sooner than the time they actually get timed out.
Resolution
The alert message is because by default the feature Slow Drain Device Detection and Congestion Avoidance is automatically enabled for version 5.0(4).
These RMON email alerts correspond to the output of
show system internal snmp credit-not-available
Module: 1 Number of events logged: 36
-------------------------------------------------------------------------------------------------------------
Port Threshold Rising/Falling Interval(s) Event Time Type Duration of time not available
-------------------------------------------------------------------------------------------------------------
Fcx/y 10/0(%) 1 Sat Feb 26 17:21:24 2011 Rising 10%
This reads that the fc interface waited 100ms (10% of 1 Second) before receiving an R_RDY (credit) to send.
Consider contacting the vendor(s) of the devices attached to the interfaces generating these alarms. Tell the Vendor(s) that withholding R_RDY over 100ms is a symptom of a problem.
This credit-not-available within 100ms is not a big issue, unless a many credits are not available for longer durations, such as 500ms.
To see if it is causing frame drops, use the following command,
"show logging onboard flow-control timeout-drops"
This command will display the total timeout discards per interface and the timestamp in the latest specified duration.
A timeout discard is a frame that was dropped because it was in the fabric for over 500ms, usually because an end device did not grant permission (R_RDY) for the switch to send to it.
You can see the defaults for port monitor with this command:
switch#show port-monitor
Related Information
Cisco MDS 9000 Family NX-OS Interfaces Configuration Guide
Buffer to buffer credits alerts
