07-10-2017 05:23 AM
Hi All,
I have recently completed the installation of ISL links, after the initial configuration they were all ok, until I checked them again this morning and to my surprise I saw the errDisabled error on two links.
Please help me with the resolution of the below errors:
fc1/45 20 E on trunking lwcr TE 8 --
fc1/46 20 E on errDisabled lwcr -- -- --
fc1/47 20 E on trunking lwcr TE 8 --
fc1/48 20 E on errDisabled lwcr -- -- --
-------------------------------------------------------------------------------
Interface Status Speed
(Gbps)
-------------------------------------------------------------------------------
sup-fc0 up 1
-------------------------------------------------------------------------------
Interface Status IP Address Speed MTU
-------------------------------------------------------------------------------
mgmt0 up 10.114.60.31/24 1 Gbps 1500
MDP-9148-FABA# show int fc1/46
fc1/46 is down (Error disabled - bit error rate too high)
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 20:2e:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port vsan is 20
Receive data field Size is 2112
Beacon is turned off
5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec,0 bytes/sec, 0 frames/sec
1939 frames input,144644 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
2929 frames output,180016 bytes
0 discards,0 errors
8 input OLS,5 LRR,2 NOS,0 loop inits
19 output OLS,18 LRR, 6 NOS, 0 loop inits
Interface last changed at Fri Jul 7 00:07:55 2017
Last clearing of "show interface" counters :never
MDP-9148-FABA# show int fc1/48
fc1/48 is down (Error disabled - bit error rate too high)
Port description is ISL
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 20:30:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port vsan is 20
Receive data field Size is 2112
Beacon is turned off
5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec,0 bytes/sec, 0 frames/sec
293 frames input,30648 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
289 frames output,23812 bytes
0 discards,0 errors
8 input OLS,4 LRR,4 NOS,0 loop inits
12 output OLS,12 LRR, 4 NOS, 0 loop inits
Interface last changed at Fri Jul 7 00:03:45 2017
Last clearing of "show interface" counters :never
Best Regards,
Thabiso Mathabathe
07-10-2017 01:14 PM
Hi
Can you please provide additional information:
- are you running over dark fibre
- what is the distance ?
- which FC speed ?
- please add exact spec of SFP ? is it
DS-SFP-FC4G-MR |
1/2/4-Gbps Fibre Channel-Longwave 4-km, SFP, LC |
Does the SFP appear on this list
http://www.cisco.com/c/en/us/products/collateral/storage-networking/mds-9000-series-multilayer-switches/product_data_sheet09186a00801bc698.html#ConfigurationLimitsOM
07-11-2017 01:04 AM
Hi Walter,
Thank you for your prompt reply:
Herewith more information:
- What is the distance ?
a. Less than 10KMs
- are you running over dark fibre
a. No
- which FC speed ?
a. 8gb
- please add exact spec of SFP:
DS-SFP-FC8G-LW |
Desc: 8Gbps Fibre Channel LW SFP+, LC |
Fabric A
VCP (MDS 9513) Fabric Ports |
MDP (9148S) Fabric Port |
Operation Mode |
Comment |
FC1/46 |
FC1/45 |
TE |
VSAN20 |
FC6/42 |
FC1/46 |
NotConnected |
Check Physical Connection |
FC5/39 |
FC1/47 |
NotConnected |
Check Physical Connection |
FC11/27 |
FC1/48 |
NotConnected |
Check Physical Connection |
Fabric B
VCP (MDS 9513) Fabric Ports |
MDP (9148S) Fabric Port |
Operation Mode |
Comment |
FC1/46 |
FC1/45 |
TE |
VSAN 20 |
FC6/47 |
FC1/46 |
TE |
VSAN 20 |
FC5/39 |
FC1/47 |
TE |
VSAN 20 |
FC6/29 |
FC1/48 |
E |
VSAN 20 |
I hope the above information will suffice.
Best Regards,
Thabiso
07-11-2017 02:30 AM
Hi Thabiso
DS-SFP-FC8G-LW |
1310 |
SMF |
9.0 9.0 9.0 |
2.125 4.250 8.500 |
6.2 mi (10 km) 6.2 mi (10 km) 6.2 mi (10 km) |
Is good for 10km distance !
fc 1/48 shows: Hardware is Fibre Channel, SFP is long wave laser cost reduced
I don't believe that this is a DS-SFP-FC8G-LW ! can you please check !
"switchport ignore bit-errors" command the error "errDisable" is now gone.
I would not advise to do this; your problem is not solved; the performance will be severely impacted !
Please provide the output of "show interface fc 6/29 and fc 1/48
If the error message says vsan x is isolated, the command
"show interface fc 6/29 trunk vsan x" will give you the reason of the isolation !
07-11-2017 03:00 AM
Hi Walter,
As requested:
Herewith Port details for fc6/29 and fc1/48
FABBCOMPARK# show int fc6/29 transceiver details
fc6/29 sfp is present
Name is CISCO-FINISAR
Manufacturer's part number is FTLF1428P3BNV-C1
Revision is A
Serial number is FNS19491NTU
Cisco part number is 10-2459-01
Cisco pid is DS-SFP-FC8G-LW
FC Transmitter type is long wave laser cost reduced
FC Transmitter supports long distance link length
Transmission medium is single mode (SM) laser
Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
Nominal bit rate is 8500 Mb/s
Link length supported for 9/125um fiber is 10.100 km
Cisco extended id is unknown (0x0)
No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
SFP Diagnostics Information:
----------------------------------------------------------------------------
Alarms Warnings
High Low High Low
----------------------------------------------------------------------------
Temperature 40.54 C 75.00 C -5.00 C 70.00 C 0.00 C
Voltage 3.32 V 3.63 V 2.97 V 3.46 V 3.13 V
Current 43.00 mA 85.00 mA 15.00 mA 80.00 mA 20.00 mA
Tx Power -2.43 dBm 3.50 dBm -15.70 dBm 0.50 dBm -11.70 dBm
Rx Power -4.52 dBm 3.50 dBm -23.47 dBm 0.50 dBm -19.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
FABBCOMPARK# show int fc6/29
fc6/29 is up
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 21:5d:00:0d:ec:6c:c4:c0
Peer port WWN is 20:30:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is off
snmp link state traps are enabled
Port mode is E
Port vsan is 20
Speed is 8 Gbps
Rate mode is dedicated
Transmit B2B Credit is 64
Receive B2B Credit is 250
B2B State Change Number is 14
Receive data field Size is 2112
Beacon is turned off
admin fec state is down
oper fec state is down
5 minutes input rate 40 bits/sec,5 bytes/sec, 0 frames/sec
5 minutes output rate 40 bits/sec,5 bytes/sec, 0 frames/sec
8079 frames input,404816 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
8058 frames output,408588 bytes
0 discards,0 errors
15 input OLS,20 LRR,1 NOS,0 loop inits
16 output OLS,4 LRR, 7 NOS, 0 loop inits
250 receive B2B credit remaining
64 transmit B2B credit remaining
64 low priority transmit B2B credit remaining
Interface last changed at Mon Jul 10 14:53:18 2017
Last clearing of "show interface" counters :never
-----------------------------------------------------------------------------------------------------------------------
MDP-9148-FABA# show int fc1/48 transceiver details
fc1/48 sfp is present
Name is CISCO-FINISAR
Manufacturer's part number is FTLF1428P3BNV-C1
Revision is A
Serial number is FNS19500RBE
Cisco part number is 10-2459-01
Cisco pid is DS-SFP-FC8G-LW
FC Transmitter type is long wave laser cost reduced
FC Transmitter supports long distance link length
Transmission medium is single mode (SM) laser
Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
Nominal bit rate is 8500 Mb/s
Link length supported for 9/125um fiber is 10.100 km
Cisco extended id is unknown (0x0)
No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
SFP Diagnostics Information:
----------------------------------------------------------------------------
Alarms Warnings
High Low High Low
----------------------------------------------------------------------------
Temperature 33.28 C 75.00 C -5.00 C 70.00 C 0.00 C
Voltage 3.32 V 3.63 V 2.97 V 3.46 V 3.13 V
Current 35.13 mA 85.00 mA 15.00 mA 80.00 mA 20.00 mA
Tx Power -1.87 dBm 3.50 dBm -15.70 dBm 0.50 dBm -11.70 dBm
Rx Power -4.16 dBm 3.50 dBm -23.47 dBm 0.50 dBm -19.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
MDP-9148-FABA# show int fc1/48
fc1/48 is up
Port description is ISL
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 20:30:8c:60:4f:79:99:d0
Peer port WWN is 21:5d:00:0d:ec:6c:c4:c0
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is E
Port vsan is 20
Speed is 8 Gbps
Rate mode is dedicated
Transmit B2B Credit is 250
Receive B2B Credit is 64
B2B State Change Number is 14
Receive data field Size is 2112
Beacon is turned off
admin fec state is down
oper fec state is down
5 minutes input rate 32 bits/sec,4 bytes/sec, 0 frames/sec
5 minutes output rate 32 bits/sec,4 bytes/sec, 0 frames/sec
8064 frames input,408884 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
8085 frames output,405112 bytes
0 discards,0 errors
8 input OLS,4 LRR,4 NOS,0 loop inits
19 output OLS,20 LRR, 4 NOS, 0 loop inits
64 receive B2B credit remaining
250 transmit B2B credit remaining
235 low priority transmit B2B credit remaining
Interface last changed at Mon Jul 10 14:45:20 2017
Last clearing of "show interface" counters :never
07-11-2017 03:14 AM
Hi Walter,
With regards to the fc6/29 and fc1/48 config info:
The problem was that fc1/48 only allowed VSAN 2 to trunk.
I ran the "switchport trunk allowed VSAN all" command and sorted.
07-11-2017 03:48 AM
FABBCOMPARK# show int fc6/29
fc6/29 is up
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 21:5d:00:0d:ec:6c:c4:c0
Peer port WWN is 20:30:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is off <================ should be on ?
snmp link state traps are enabled
07-11-2017 05:05 AM
Hi Walter,
I believe that is the earlier response before I ran the "switchport trunk allowed VSAN all" command:
Herewith latest output:
FABBCOMPARK# show int fc6/29
fc6/29 is trunking
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 21:5d:00:0d:ec:6c:c4:c0
Peer port WWN is 20:30:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 20
Speed is 8 Gbps
Rate mode is dedicated
Transmit B2B Credit is 64
Receive B2B Credit is 250
B2B State Change Number is 14
Receive data field Size is 2112
Beacon is turned off
admin fec state is down
oper fec state is down
Trunk vsans (admin allowed and active) (1-6,8-10,20)
Trunk vsans (up) (1,20)
Trunk vsans (isolated) (2-6,8-10)
Trunk vsans (initializing) ()
5 minutes input rate 96 bits/sec,12 bytes/sec, 0 frames/sec
5 minutes output rate 288 bits/sec,36 bytes/sec, 0 frames/sec
9559 frames input,489020 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
10982 frames output,618140 bytes
0 discards,0 errors
17 input OLS,22 LRR,1 NOS,0 loop inits
17 output OLS,4 LRR, 8 NOS, 0 loop inits
250 receive B2B credit remaining
64 transmit B2B credit remaining
64 low priority transmit B2B credit remaining
Interface last changed at Tue Jul 11 12:09:08 2017
Last clearing of "show interface" counters :never
Fabric B Looks Good (All ports are trunking)
fc1/45 20 E on trunking lwcr TE 8 --
fc1/46 20 E on trunking lwcr TE 8 --
fc1/47 20 E on trunking lwcr TE 8 --
fc1/48 20 E on trunking lwcr TE 8 --
07-13-2017 10:26 AM
Hi
I see a few vsan's which are isolated: Trunk vsans (isolated) (2-6,8-10)
Please run
"show interface fc 6/29 trunk vsan x" which will give you the reason of the isolation !
Regarding biterrors please read
http://www.cisco.com/c/en/us/support/docs/storage-networking/mds-9513-multilayer-director/210451-FC-port-down-due-to-Error-disabled-bi.html
Bit errors occur because of the following reasons:
I assume you are not impacted by https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuo56792
no switchport ignore bit-errors
Note: It is not advisable to leave above setting on indefinitely, rather to be used during troubleshooting sessions.
Note: Regardless of the setting of the switchport ignore bit-errors command, a switch generates a syslog message when the BER threshold is exceeded.
07-17-2017 07:15 AM
07-17-2017 01:01 PM
Hi Thabiso
It's hard to read ! I think it reads "3/125" which would be SMF !
Can you please provide us a diagram end to end (which fibre type SMF, MMF, which SFP (LW, SF)
Walter.
PS. of course there could be a bad cable and/or dirty connector !
07-13-2017 04:40 AM
Hi Walter,
I trust this comment finds you well.
Please assist as to why we get the Bit-Error on the ports? On your previous comment you mention that "ignoring the Bit-Error" on the switchport does not help and might cause issues in future. How do I resolve the matter?
How do I allocate sufficient B2B credits on the ports to alleviate the bit-error.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide