Hi

thabiso.mathabathe1 · ‎07-10-2017

Hi All,

I have recently completed the installation of ISL links, after the initial configuration they were all ok, until I checked them again this morning and to my surprise I saw the errDisabled error on two links.

Please help me with the resolution of the below errors:

fc1/45 20 E on trunking lwcr TE 8 --
fc1/46 20 E on errDisabled lwcr -- -- --
fc1/47 20 E on trunking lwcr TE 8 --
fc1/48 20 E on errDisabled lwcr -- -- --

-------------------------------------------------------------------------------
Interface Status Speed
(Gbps)
-------------------------------------------------------------------------------
sup-fc0 up 1

-------------------------------------------------------------------------------
Interface Status IP Address Speed MTU
-------------------------------------------------------------------------------
mgmt0 up 10.114.60.31/24 1 Gbps 1500

MDP-9148-FABA# show int fc1/46

fc1/46 is down (Error disabled - bit error rate too high)
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 20:2e:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port vsan is 20
Receive data field Size is 2112
Beacon is turned off
5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec,0 bytes/sec, 0 frames/sec
1939 frames input,144644 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
2929 frames output,180016 bytes
0 discards,0 errors
8 input OLS,5 LRR,2 NOS,0 loop inits
19 output OLS,18 LRR, 6 NOS, 0 loop inits
Interface last changed at Fri Jul 7 00:07:55 2017

Last clearing of "show interface" counters :never

MDP-9148-FABA# show int fc1/48

fc1/48 is down (Error disabled - bit error rate too high)
Port description is ISL
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 20:30:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port vsan is 20
Receive data field Size is 2112
Beacon is turned off
5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec,0 bytes/sec, 0 frames/sec
293 frames input,30648 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
289 frames output,23812 bytes
0 discards,0 errors
8 input OLS,4 LRR,4 NOS,0 loop inits
12 output OLS,12 LRR, 4 NOS, 0 loop inits
Interface last changed at Fri Jul 7 00:03:45 2017

Last clearing of "show interface" counters :never

Best Regards,

Thabiso Mathabathe

Walter Dey · ‎07-10-2017

Hi

Can you please provide additional information:

- are you running over dark fibre

- what is the distance ?

- which FC speed ?

- please add exact spec of SFP ? is it

DS-SFP-FC4G-MR

1/2/4-Gbps Fibre Channel-Longwave 4-km, SFP, LC

Does the SFP appear on this list

http://www.cisco.com/c/en/us/products/collateral/storage-networking/mds-9000-series-multilayer-switches/product_data_sheet09186a00801bc698.html#ConfigurationLimitsOM

thabiso.mathabathe1 · ‎07-11-2017

Hi Walter,

Thank you for your prompt reply:

Herewith more information:

- What is the distance ?

a. Less than 10KMs

- are you running over dark fibre

a. No

- which FC speed ?

a. 8gb

- please add exact spec of SFP:

DS-SFP-FC8G-LW

Desc: 8Gbps Fibre Channel LW SFP+, LC

Furthermore, kindly note that after running the "switchport ignore bit-errors" command the error "errDisable" is now gone.
However I have one link on Fabric B which goes into "Isloated" mode whenever I enable Trunk mode on the MDS 9513 Port FC6/29. This is strange because both port on both fabrics are on the same VSAN ID (below highlighted):

Fabric A

VCP (MDS 9513) Fabric Ports	MDP (9148S) Fabric Port	Operation Mode	Comment
FC1/46	FC1/45	TE	VSAN20
FC6/42	FC1/46	NotConnected	Check Physical Connection
FC5/39	FC1/47	NotConnected	Check Physical Connection
FC11/27	FC1/48	NotConnected	Check Physical Connection

Fabric B

VCP (MDS 9513) Fabric Ports	MDP (9148S) Fabric Port	Operation Mode	Comment
FC1/46	FC1/45	TE	VSAN 20
FC6/47	FC1/46	TE	VSAN 20
FC5/39	FC1/47	TE	VSAN 20
FC6/29	FC1/48	E	VSAN 20

I hope the above information will suffice.

Best Regards,

Thabiso

Walter Dey · ‎07-11-2017

Hi Thabiso

DS-SFP-FC8G-LW

1310

SMF

9.0

2.125

4.250

8.500

6.2 mi (10 km)

Is good for 10km distance !

fc 1/48 shows: Hardware is Fibre Channel, SFP is long wave laser cost reduced

I don't believe that this is a DS-SFP-FC8G-LW ! can you please check !

"switchport ignore bit-errors" command the error "errDisable" is now gone.

I would not advise to do this; your problem is not solved; the performance will be severely impacted !

Please provide the output of "show interface fc 6/29 and fc 1/48

If the error message says vsan x is isolated, the command

"show interface fc 6/29 trunk vsan x" will give you the reason of the isolation !

thabiso.mathabathe1 · ‎07-11-2017

Hi Walter,

As requested:

Herewith Port details for fc6/29 and fc1/48

FABBCOMPARK# show int fc6/29 transceiver details
fc6/29 sfp is present
Name is CISCO-FINISAR
Manufacturer's part number is FTLF1428P3BNV-C1
Revision is A
Serial number is FNS19491NTU
Cisco part number is 10-2459-01
Cisco pid is DS-SFP-FC8G-LW
FC Transmitter type is long wave laser cost reduced
FC Transmitter supports long distance link length
Transmission medium is single mode (SM) laser
Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
Nominal bit rate is 8500 Mb/s
Link length supported for 9/125um fiber is 10.100 km
Cisco extended id is unknown (0x0)

No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
SFP Diagnostics Information:
----------------------------------------------------------------------------
Alarms Warnings
High Low High Low
----------------------------------------------------------------------------
Temperature 40.54 C 75.00 C -5.00 C 70.00 C 0.00 C
Voltage 3.32 V 3.63 V 2.97 V 3.46 V 3.13 V
Current 43.00 mA 85.00 mA 15.00 mA 80.00 mA 20.00 mA
Tx Power -2.43 dBm 3.50 dBm -15.70 dBm 0.50 dBm -11.70 dBm
Rx Power -4.52 dBm 3.50 dBm -23.47 dBm 0.50 dBm -19.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

FABBCOMPARK# show int fc6/29
fc6/29 is up
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 21:5d:00:0d:ec:6c:c4:c0
Peer port WWN is 20:30:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is off
snmp link state traps are enabled
Port mode is E
Port vsan is 20
Speed is 8 Gbps
Rate mode is dedicated
Transmit B2B Credit is 64
Receive B2B Credit is 250
B2B State Change Number is 14
Receive data field Size is 2112
Beacon is turned off
admin fec state is down
oper fec state is down
5 minutes input rate 40 bits/sec,5 bytes/sec, 0 frames/sec
5 minutes output rate 40 bits/sec,5 bytes/sec, 0 frames/sec
8079 frames input,404816 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
8058 frames output,408588 bytes
0 discards,0 errors
15 input OLS,20 LRR,1 NOS,0 loop inits
16 output OLS,4 LRR, 7 NOS, 0 loop inits
250 receive B2B credit remaining
64 transmit B2B credit remaining
64 low priority transmit B2B credit remaining
Interface last changed at Mon Jul 10 14:53:18 2017

Last clearing of "show interface" counters :never

-----------------------------------------------------------------------------------------------------------------------

MDP-9148-FABA# show int fc1/48 transceiver details
fc1/48 sfp is present
Name is CISCO-FINISAR
Manufacturer's part number is FTLF1428P3BNV-C1
Revision is A
Serial number is FNS19500RBE
Cisco part number is 10-2459-01
Cisco pid is DS-SFP-FC8G-LW
FC Transmitter type is long wave laser cost reduced
FC Transmitter supports long distance link length
Transmission medium is single mode (SM) laser
Supported speeds are - Min speed: 2000 Mb/s, Max speed: 8000 Mb/s
Nominal bit rate is 8500 Mb/s
Link length supported for 9/125um fiber is 10.100 km
Cisco extended id is unknown (0x0)

No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
SFP Diagnostics Information:
----------------------------------------------------------------------------
Alarms Warnings
High Low High Low
----------------------------------------------------------------------------
Temperature 33.28 C 75.00 C -5.00 C 70.00 C 0.00 C
Voltage 3.32 V 3.63 V 2.97 V 3.46 V 3.13 V
Current 35.13 mA 85.00 mA 15.00 mA 80.00 mA 20.00 mA
Tx Power -1.87 dBm 3.50 dBm -15.70 dBm 0.50 dBm -11.70 dBm
Rx Power -4.16 dBm 3.50 dBm -23.47 dBm 0.50 dBm -19.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

MDP-9148-FABA# show int fc1/48
fc1/48 is up
Port description is ISL
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 20:30:8c:60:4f:79:99:d0
Peer port WWN is 21:5d:00:0d:ec:6c:c4:c0
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is E
Port vsan is 20
Speed is 8 Gbps
Rate mode is dedicated
Transmit B2B Credit is 250
Receive B2B Credit is 64
B2B State Change Number is 14
Receive data field Size is 2112
Beacon is turned off
admin fec state is down
oper fec state is down
5 minutes input rate 32 bits/sec,4 bytes/sec, 0 frames/sec
5 minutes output rate 32 bits/sec,4 bytes/sec, 0 frames/sec
8064 frames input,408884 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
8085 frames output,405112 bytes
0 discards,0 errors
8 input OLS,4 LRR,4 NOS,0 loop inits
19 output OLS,20 LRR, 4 NOS, 0 loop inits
64 receive B2B credit remaining
250 transmit B2B credit remaining
235 low priority transmit B2B credit remaining
Interface last changed at Mon Jul 10 14:45:20 2017

Last clearing of "show interface" counters :never

thabiso.mathabathe1 · ‎07-11-2017

Hi Walter,

With regards to the fc6/29 and fc1/48 config info:

The problem was that fc1/48 only allowed VSAN 2 to trunk.

I ran the "switchport trunk allowed VSAN all" command and sorted.

Walter Dey · ‎07-11-2017

FABBCOMPARK# show int fc6/29
fc6/29 is up
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 21:5d:00:0d:ec:6c:c4:c0
Peer port WWN is 20:30:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is off <================ should be on ?
snmp link state traps are enabled

thabiso.mathabathe1 · ‎07-11-2017

Hi Walter,

I believe that is the earlier response before I ran the "switchport trunk allowed VSAN all" command:

Herewith latest output:

FABBCOMPARK# show int fc6/29
fc6/29 is trunking
Hardware is Fibre Channel, SFP is long wave laser cost reduced
Port WWN is 21:5d:00:0d:ec:6c:c4:c0
Peer port WWN is 20:30:8c:60:4f:79:99:d0
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 20
Speed is 8 Gbps
Rate mode is dedicated
Transmit B2B Credit is 64
Receive B2B Credit is 250
B2B State Change Number is 14
Receive data field Size is 2112
Beacon is turned off
admin fec state is down
oper fec state is down
Trunk vsans (admin allowed and active) (1-6,8-10,20)
Trunk vsans (up) (1,20)
Trunk vsans (isolated) (2-6,8-10)
Trunk vsans (initializing) ()
5 minutes input rate 96 bits/sec,12 bytes/sec, 0 frames/sec
5 minutes output rate 288 bits/sec,36 bytes/sec, 0 frames/sec
9559 frames input,489020 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
10982 frames output,618140 bytes
0 discards,0 errors
17 input OLS,22 LRR,1 NOS,0 loop inits
17 output OLS,4 LRR, 8 NOS, 0 loop inits
250 receive B2B credit remaining
64 transmit B2B credit remaining
64 low priority transmit B2B credit remaining
Interface last changed at Tue Jul 11 12:09:08 2017

Last clearing of "show interface" counters :never

Fabric B Looks Good (All ports are trunking)

fc1/45 20 E on trunking lwcr TE 8 --
fc1/46 20 E on trunking lwcr TE 8 --
fc1/47 20 E on trunking lwcr TE 8 --
fc1/48 20 E on trunking lwcr TE 8 --

Walter Dey · ‎07-13-2017

Hi

I see a few vsan's which are isolated: Trunk vsans (isolated) (2-6,8-10)

Please run

"show interface fc 6/29 trunk vsan x" which will give you the reason of the isolation !

Regarding biterrors please read

http://www.cisco.com/c/en/us/support/docs/storage-networking/mds-9513-multilayer-director/210451-FC-port-down-due-to-Error-disabled-bi.html

Bit errors occur because of the following reasons:

Faulty or bad cable.
Faulty or bad GBIC or SFP.
GBIC or SFP is specified to operate at 1 Gbps but is used at 2 Gbps.
GBIC or SFP is specified to operate at 2 Gbps but is used at 4 Gbps.
Short haul cable is used for long haul or long haul cable is used for short haul.
Momentary sync loss.
Loose cable connection at one or both ends.
Improper GBIC or SFP connection at one or both ends.

I assume you are not impacted by https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuo56792

FC interface disabled due to 'bit error rate too high' when rate is low

CSCuo56792

no switchport ignore bit-errors

Note: It is not advisable to leave above setting on indefinitely, rather to be used during troubleshooting sessions.

Note: Regardless of the setting of the switchport ignore bit-errors command, a switch generates a syslog message when the BER threshold is exceeded.

thabiso.mathabathe1 · ‎07-17-2017

Hi Walter,

I trust you are well today.

Please see attached pic with cable information and advise if the cable in use is appropriate for the Long-Wave Inter-Switch-Link connection.

Best Regards,

Thabiso

Walter Dey · ‎07-17-2017

Hi Thabiso

It's hard to read ! I think it reads "3/125" which would be SMF !

Can you please provide us a diagram end to end (which fibre type SMF, MMF, which SFP (LW, SF)

Walter.

PS. of course there could be a bad cable and/or dirty connector !

thabiso.mathabathe1 · ‎07-13-2017

Hi Walter,

I trust this comment finds you well.

Please assist as to why we get the Bit-Error on the ports? On your previous comment you mention that "ignoring the Bit-Error" on the switchport does not help and might cause issues in future. How do I resolve the matter?

How do I allocate sufficient B2B credits on the ports to alleviate the bit-error.

ISL Trunks errDisabled (bit error rate too high)