udld fail ?

[James Lasky]

Dear experts

I've a Cisco 6500 (12.2(33)) connected to a juniper EX with a 2 x 10Gb port channel.

Apparently the Cisco gear has disabled one out of the two ten giga interface after some flapping of the other one and due to UDLD that is currently non configured as aggressive nor bidirectional (not supported by Juniper gear).

Among the two gears LACP fast is running.

I kindly ask any feedback if it's something already experienced by somebody.

Cheers

James

[InayathUlla Sharieff]

Need to check the reason of the port going down.

Configure the udld aggressive and check UDLD reason is it due to what parameter?

Other option to check will be to move the cable to different port and see if the port comes up else it could be problem with the sfp / cable.

[James Lasky]

Not sure to catch your point.

I do not want to enable aggressive, it was a temporary behaviour and the links are WAN links not inside the DC.

Here you can see the logs:

232641: May 31 11:16:41.330 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/7, changed state to up

232642: May 31 11:16:41.330 CET: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/7, changed state to up

232643: May 31 11:16:49.431 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/7, changed state to down

232644: May 31 11:16:49.459 CET: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/7, changed state to down

232645: May 31 11:16:49.451 CET: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/7, changed state to down

232646: May 31 11:16:49.455 CET: %LINK-SP-3-UPDOWN: Interface TenGigabitEthernet2/7, changed state to down

232647: May 31 11:16:54.831 CET: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/7, changed state to up

232655: May 31 11:16:54.831 CET: %LINK-SP-3-UPDOWN: Interface TenGigabitEthernet2/7, changed state to up

232648: May 31 11:16:55.775 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/5, changed state to down

232649: May 31 11:16:55.795 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel20, changed state to down

232650: May 31 11:16:55.803 CET: %LINK-3-UPDOWN: Interface Port-channel20, changed state to down

232651: May 31 11:16:55.803 CET: %PIM-5-NBRCHG: neighbor 10.x.x.x DOWN on interface Port-channel20 non DR

232652: May 31 11:16:55.803 CET: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/5, changed state to down

232653: May 31 11:16:55.819 CET: %BGP-5-ADJCHANGE: neighbor 10.x.x.x Down Interface flap

232654: May 31 11:16:55.819 CET: %BGP-5-ADJCHANGE: neighbor 10.x.x.x Down Interface flap

232656: May 31 11:16:55.771 CET: %UDLD-SP-4-UDLD_PORT_DISABLED: UDLD disabled interface Te2/5, unidirectional link detected   <----------------------

232657: May 31 11:16:55.771 CET: %PM-SP-4-ERR_DISABLE: udld error detected on Te2/5, putting Te2/5 in err-disable state       <----------------------

232658: May 31 11:16:55.803 CET: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/5, changed state to down

232659: May 31 11:16:55.803 CET: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface Port-channel20, changed state to down

232660: May 31 11:16:55.803 CET: %LINK-SP-3-UPDOWN: Interface Port-channel20, changed state to down

232661: May 31 11:16:55.803 CET: %LINK-SP-3-UPDOWN: Interface TenGigabitEthernet2/5, changed state to down

[Tagir Temirgaliyev]

sh int te2/7 trans

sh int te2/5 trans

try to change FO patchcords

try to change SFP

try to measure FO links by OLTS and OTDR

[James Lasky]

xxx#sh int te2/5 transceiver
Transceiver monitoring is disabled for all interfaces.

ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
If device is externally calibrated, only calibrated values are printed.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
NA or N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).

Optical Optical
Temperature Voltage Current Tx Power Rx Power
Port (Celsius) (Volts) (mA) (dBm) (dBm)
---------- ----------- ------- -------- -------- --------
Te2/5 25.5 0.00 59.2 -- 1.1 -3.1

xxx#sh int te2/7 transceiver
Transceiver monitoring is disabled for all interfaces.

ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
If device is externally calibrated, only calibrated values are printed.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
NA or N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).

Optical Optical
Temperature Voltage Current Tx Power Rx Power
Port (Celsius) (Volts) (mA) (dBm) (dBm)
---------- ----------- ------- -------- -------- --------
Te2/7 28.4 0.00 48.9 -- -1.0 -7.0

[Carlos Villagran]

Hi! 

LACP provides unidirectional failure capabilities, I honetly do not understand why it is being reported as UDLD failure. 

LACP will exchange LACPBPDUs and lets say host A will expect to receive certain flags from peer to start the bundling and when not received it sends an Expired flag which peer B reacts shutting down the port. According to Juniper EX documentation you can use LACP as a workaround. 

http://kb.juniper.net/InfoCenter/index?page=content&id=KB13314&actp=search

As I understand, you do not have any UDLD configuration at all in the switch, right?

So in summary, it seems you are actually having an unidirectional link failure and LACP is shutting down the port, for some reason it is being reported as UDLD log.  

Allow me some time to go through some tests. 

Have a nice day, best regards!

JC

[James Lasky]

Hi carlos

Indeed i have udld configured but normal not aggressive hence i should not get interface disabled!

I am wondering if there is a resonable reason for the issue since EX do not support udld and why c6500 disable the working interface.

Cheers

James

[Carlos Villagran]

Hi!

Firstly I would like to point you to this information provided by Peter Paluch about explicit and implicit  unidirectional failure detection system in UDLD:

https://supportforums.cisco.com/discussion/12091831/udld-normal-question

Accordin to this, there are three conditions that when met put the interface in err-disble no matter if it is configured as Aggressive or Normal mode, which are:

• UDLD messages received from a neighbor do not contain the tuple that matches the receiving switch and port in their echo portion. This would indicate a split fiber or a truly unidirectional link.
• UDLD messages received from a neighbor contain the same sender as the receiving switch and its port. This would indicate a self-looped port.
• UDLD message received from a neighbor contain a different list of entries in the echo portion (i.e. a different set of neighbors) than the set of neighbors heard by this switch on this port. This would indicate a shared segment with a partial impaired connectivity between connected switches.

Regarding your second question:

Well, I am just talking from my own experience and as stated in the above documentation, UDLD is not really broadly documented. 

I have had around three TAC cases where certain devices and I am not exactly talking about 3rd party devices, do not treat UDLD packets as expected, and this does not seem to be related to a bug or something like that but because there is some nature in the protocols not really cooperating. 

As per my observations, if the UDLD enable port does detect that the link is up/up and does not receive UDLD information at all from the neighbor it goes automatically to err-disable (Of course this happen really fast). So this kind of contradict the three statements mentioned above.

If UDLD detects an up/up interface and absolutely no info is detected from the neighbor then it goes to err-disable (Be aware that this information is carried along with CDP packets). I have not had the chance to actually isolate this behavior but I will try to do it by tomorrow.

Hope it helps, best regards!

JC

[James Lasky]

Hi JC,

indeed my udld sessions are not bidirectional and it worked well until the mentioned issue.

Cheers

James

XXX#sh udld tenGigabitEthernet 2/5

Interface Te2/5
---
Port enable administrative configuration setting: Follows device default
Port enable operational state: Enabled
Current bidirectional state: Unknown
Current operational state: Advertisement
Message interval: 7
Time out interval: 5
No neighbor cache information stored

XXX#sh udld tenGigabitEthernet 2/7

Interface Te2/7
---
Port enable administrative configuration setting: Follows device default
Port enable operational state: Enabled
Current bidirectional state: Unknown
Current operational state: Advertisement
Message interval: 7
Time out interval: 5
No neighbor cache information stored

[James Lasky]

Hi carlos

Indeed i have udld configured but normal not aggressive hence i should not get interface disabled!

I am wondering if there is a resonable reason for the issue since EX do not support udld and why c6500 disable the working interface.

Cheers

James