Interconnecting MDS 9148 with 2 different Fiber Providers

[trombidz1]

Hi

I would like discuss the following issue :

We have 2 HP SAN EVA 8400 ( one at the production site and one at our Disaster Recovery site ) replicating using a fiber channel service from ( 2* FC dark fiber from PROVIDER A )

They are linked together  with 4 Cisco MDS 9148 Multilayer Fabric Switch (2 fabric ). 

We have recently purchased a new fiber service ( 2* 1Gb FC managed service from PROVIDER B) in order to create a port channel  with the 2 running ones and we have some problem.

We disconnected the PROVIDER A FC ( running for 10 years now )  and left the SAN replication during a couple of days running with the new PROVIDER B 1 Gb FC to make sure the link was stable and we had no issue.

Then we tried to connect the 2 FC together from the 2 providers and tried to create a Port channel.

However we discovered that to create a PortChannel, interfaces must have the same configurations ( speed, mode, rate )

In our case , the interface FC 1/1 using PROVIDER A link is running at 4 Gb and the interface FC 1/2 using PROVIDER B is running at 1 Gb

PROVIDER A dark fiber

PROVIDER B fiber managed service

fc1/1 is trunking     Hardware is Fibre Channel, SFP is CWDM-1550     Port WWN is 20:01:00:05:9b:72:b1:60     Peer port WWN is 20:01:00:05:9b:72:b1:a0     Admin port mode is E, trunk mode is on     snmp link state traps are enabled     Port mode is TE     Port vsan is 1     Speed is 4 Gbps     Rate mode is dedicated     Transmit B2B Credit is 32     Receive B2B Credit is 32     B2B State Change Number is 14     Receive data field Size is 2112     Beacon is turned off     Trunk vsans (admin allowed and active) (1)     Trunk vsans (up)                       (1)     Trunk vsans (isolated)                 ()     Trunk vsans (initializing)             ()     5 minutes input rate 4416 bits/sec, 552 bytes/sec, 6 frames/sec     5 minutes output rate 91616 bits/sec, 11452 bytes/sec, 11 frames/sec       51537358280 frames input, 50955393731852 bytes         0 discards, 0 errors         0 CRC,  0 unknown class         0 too long, 0 too short       223256095067 frames output, 408667816663528 bytes         32 discards, 0 errors       34 input OLS, 36 LRR, 51 NOS, 0 loop inits       53 output OLS, 35 LRR, 45 NOS, 0 loop inits       32 receive B2B credit remaining       32 transmit B2B credit remaining       28 low priority transmit B2B credit remaining     Interface last changed at Fri Dec 14 17:02:48 2012

fc1/2 is trunking     Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)     Port WWN is 20:02:00:05:9b:72:b1:60     Peer port WWN is 20:02:00:05:9b:72:b1:a0     Admin port mode is E, trunk mode is on     snmp link state traps are enabled     Port mode is TE     Port vsan is 1     Speed is 1 Gbps     Rate mode is dedicated     Transmit B2B Credit is 32     Receive B2B Credit is 32     B2B State Change Number is 14     Receive data field Size is 2112     Beacon is turned off     Trunk vsans (admin allowed and active) (1)     Trunk vsans (up)                       (1)     Trunk vsans (isolated)                 ()     Trunk vsans (initializing)             ()     5 minutes input rate 48 bits/sec, 6 bytes/sec, 0 frames/sec     5 minutes output rate 48 bits/sec, 6 bytes/sec, 0 frames/sec       39496053 frames input, 2696451760 bytes         0 discards, 0 errors         0 CRC,  0 unknown class         0 too long, 0 too short       167653989 frames output, 230540944084 bytes         70 discards, 0 errors       40377 input OLS, 345 LRR, 4248 NOS, 0 loop inits       5186 output OLS, 1154 LRR, 39568 NOS, 0 loop inits       32 receive B2B credit remaining       32 transmit B2B credit remaining       28 low priority transmit B2B credit remaining     Interface last changed at Fri Dec 14 16:50:42 2012

So we tried to force the FC 1/1 to run at 1 Gb ( same speed as FC 1/2 ) to create the PortChannel, by doing the basic command on the interface : Switchport speed 1000

Since that , our SAN is logging every 10/15 minutes the following errors :

---------- Problem Found: CUST: CA informational: EVA8400_LIVE: Write resources on the inter site link have been reduced. at Fri 14 Dec 2012 17:21:04 GMT+01:00 ----------

---------- Problem Found: CUST: CA warning: EVA8400_DR: Excessive out of order message rate on the inter site link. at Fri 14 Dec 2012 17:27:04 GMT+01:00 ----------

---------- Problem Found: CUST: CA warning: EVA8400_DR: Excessive data exchange retry rate on the inter site link. at Fri 14 Dec 2012 17:41:34 GMT+01:00 ----------

---------- Problem Found: CUST: CA warning: EVA8400_LIVE: Excessive data exchange retry rate on the inter site link. at Fri 14 Dec 2012 17:41:49 GMT+01:00 ----------

We have the following questions :

Have you already seen that kind of situation ?

Is there a best practice  to reduce the interface speed in order to create the PortChannel with identical interface setup ?

Your help and advice would be much appreciated.

Regards

[blueyonderguy]

the RTT delay by each link is different and this is causing packets to arrive out of order.

not sure if port channeling between two different links or different characteristics is good practive.

[franjime]

Would also point out that you are seeing a lot of OLS/LRR/NOS errors - that link from the provider may be experiencing connectivity issues.