cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

1070
Views
0
Helpful
2
Replies
trombidz1
Beginner

Interconnecting MDS 9148 with 2 different Fiber Providers

Hi

I would like discuss the following issue :

We have 2 HP SAN EVA 8400 ( one at the production site and one at our Disaster Recovery site ) replicating using a fiber channel service from ( 2* FC dark fiber from PROVIDER A )

They are linked together  with 4 Cisco MDS 9148 Multilayer Fabric Switch (2 fabric ). 

We have recently purchased a new fiber service ( 2* 1Gb FC managed service from PROVIDER B) in order to create a port channel  with the 2 running ones and we have some problem.

We disconnected the PROVIDER A FC ( running for 10 years now )  and left the SAN replication during a couple of days running with the new PROVIDER B 1 Gb FC to make sure the link was stable and we had no issue.

Then we tried to connect the 2 FC together from the 2 providers and tried to create a Port channel.

However we discovered that to create a PortChannel, interfaces must have the same configurations ( speed, mode, rate )

In our case , the interface FC 1/1 using PROVIDER A link is running at 4 Gb and the interface FC 1/2 using PROVIDER B is running at 1 Gb

PROVIDER A dark fiber

PROVIDER B fiber managed service

fc1/1 is trunking

    Hardware is Fibre Channel, SFP is CWDM-1550

    Port WWN is 20:01:00:05:9b:72:b1:60

    Peer port WWN is 20:01:00:05:9b:72:b1:a0

    Admin port mode is E, trunk mode is on

    snmp link state traps are enabled

    Port mode is TE

    Port vsan is 1

    Speed is 4 Gbps

    Rate mode is dedicated

    Transmit B2B Credit is 32

    Receive B2B Credit is 32

    B2B State Change Number is 14

    Receive data field Size is 2112

    Beacon is turned off

    Trunk vsans (admin allowed and active) (1)

    Trunk vsans (up)                       (1)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             ()

    5 minutes input rate 4416 bits/sec, 552 bytes/sec, 6 frames/sec

    5 minutes output rate 91616 bits/sec, 11452 bytes/sec, 11 frames/sec

      51537358280 frames input, 50955393731852 bytes

        0 discards, 0 errors

        0 CRC,  0 unknown class

        0 too long, 0 too short

      223256095067 frames output, 408667816663528 bytes

        32 discards, 0 errors

      34 input OLS, 36 LRR, 51 NOS, 0 loop inits

      53 output OLS, 35 LRR, 45 NOS, 0 loop inits

      32 receive B2B credit remaining

      32 transmit B2B credit remaining

      28 low priority transmit B2B credit remaining

    Interface last changed at Fri Dec 14 17:02:48 2012

fc1/2 is trunking

    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)

    Port WWN is 20:02:00:05:9b:72:b1:60

    Peer port WWN is 20:02:00:05:9b:72:b1:a0

    Admin port mode is E, trunk mode is on

    snmp link state traps are enabled

    Port mode is TE

    Port vsan is 1

    Speed is 1 Gbps

    Rate mode is dedicated

    Transmit B2B Credit is 32

    Receive B2B Credit is 32

    B2B State Change Number is 14

    Receive data field Size is 2112

    Beacon is turned off

    Trunk vsans (admin allowed and active) (1)

    Trunk vsans (up)                       (1)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             ()

    5 minutes input rate 48 bits/sec, 6 bytes/sec, 0 frames/sec

    5 minutes output rate 48 bits/sec, 6 bytes/sec, 0 frames/sec

      39496053 frames input, 2696451760 bytes

        0 discards, 0 errors

        0 CRC,  0 unknown class

        0 too long, 0 too short

      167653989 frames output, 230540944084 bytes

        70 discards, 0 errors

      40377 input OLS, 345 LRR, 4248 NOS, 0 loop inits

      5186 output OLS, 1154 LRR, 39568 NOS, 0 loop inits

      32 receive B2B credit remaining

      32 transmit B2B credit remaining

      28 low priority transmit B2B credit remaining

    Interface last changed at Fri Dec 14 16:50:42 2012

So we tried to force the FC 1/1 to run at 1 Gb ( same speed as FC 1/2 ) to create the PortChannel, by doing the basic command on the interface : Switchport speed 1000

Since that , our SAN is logging every 10/15 minutes the following errors :

---------- Problem Found: CUST: CA informational: EVA8400_LIVE: Write resources on the inter site link have been reduced. at Fri 14 Dec 2012 17:21:04 GMT+01:00 ----------

---------- Problem Found: CUST: CA warning: EVA8400_DR: Excessive out of order message rate on the inter site link. at Fri 14 Dec 2012 17:27:04 GMT+01:00 ----------

---------- Problem Found: CUST: CA warning: EVA8400_DR: Excessive data exchange retry rate on the inter site link. at Fri 14 Dec 2012 17:41:34 GMT+01:00 ----------

---------- Problem Found: CUST: CA warning: EVA8400_LIVE: Excessive data exchange retry rate on the inter site link. at Fri 14 Dec 2012 17:41:49 GMT+01:00 ----------

We have the following questions :

Have you already seen that kind of situation ?

Is there a best practice  to reduce the interface speed in order to create the PortChannel with identical interface setup ?

Your help and advice would be much appreciated.

Regards

2 REPLIES 2
blueyonderguy
Beginner

the RTT delay by each link is different and this is causing packets to arrive out of order.

not sure if port channeling between two different links or different characteristics is good practive.

franjime
Cisco Employee

Would also point out that you are seeing a lot of OLS/LRR/NOS errors - that link from the provider may be experiencing connectivity issues.