12-19-2012 12:57 AM
Hi
I would like discuss the following issue :
We have 2 HP SAN EVA 8400 ( one at the production site and one at our Disaster Recovery site ) replicating using a fiber channel service from ( 2* FC dark fiber from PROVIDER A )
They are linked together with 4 Cisco MDS 9148 Multilayer Fabric Switch (2 fabric ).
We have recently purchased a new fiber service ( 2* 1Gb FC managed service from PROVIDER B) in order to create a port channel with the 2 running ones and we have some problem.
We disconnected the PROVIDER A FC ( running for 10 years now ) and left the SAN replication during a couple of days running with the new PROVIDER B 1 Gb FC to make sure the link was stable and we had no issue.
Then we tried to connect the 2 FC together from the 2 providers and tried to create a Port channel.
However we discovered that to create a PortChannel, interfaces must have the same configurations ( speed, mode, rate )
In our case , the interface FC 1/1 using PROVIDER A link is running at 4 Gb and the interface FC 1/2 using PROVIDER B is running at 1 Gb
PROVIDER A dark fiber | PROVIDER B fiber managed service |
fc1/1 is trunking Hardware is Fibre Channel, SFP is CWDM-1550 Port WWN is 20:01:00:05:9b:72:b1:60 Peer port WWN is 20:01:00:05:9b:72:b1:a0 Admin port mode is E, trunk mode is on snmp link state traps are enabled Port mode is TE Port vsan is 1 Speed is 4 Gbps Rate mode is dedicated Transmit B2B Credit is 32 Receive B2B Credit is 32 B2B State Change Number is 14 Receive data field Size is 2112 Beacon is turned off Trunk vsans (admin allowed and active) (1) Trunk vsans (up) (1) Trunk vsans (isolated) () Trunk vsans (initializing) () 5 minutes input rate 4416 bits/sec, 552 bytes/sec, 6 frames/sec 5 minutes output rate 91616 bits/sec, 11452 bytes/sec, 11 frames/sec 51537358280 frames input, 50955393731852 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 223256095067 frames output, 408667816663528 bytes 32 discards, 0 errors 34 input OLS, 36 LRR, 51 NOS, 0 loop inits 53 output OLS, 35 LRR, 45 NOS, 0 loop inits 32 receive B2B credit remaining 32 transmit B2B credit remaining 28 low priority transmit B2B credit remaining Interface last changed at Fri Dec 14 17:02:48 2012 | fc1/2 is trunking Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:02:00:05:9b:72:b1:60 Peer port WWN is 20:02:00:05:9b:72:b1:a0 Admin port mode is E, trunk mode is on snmp link state traps are enabled Port mode is TE Port vsan is 1 Speed is 1 Gbps Rate mode is dedicated Transmit B2B Credit is 32 Receive B2B Credit is 32 B2B State Change Number is 14 Receive data field Size is 2112 Beacon is turned off Trunk vsans (admin allowed and active) (1) Trunk vsans (up) (1) Trunk vsans (isolated) () Trunk vsans (initializing) () 5 minutes input rate 48 bits/sec, 6 bytes/sec, 0 frames/sec 5 minutes output rate 48 bits/sec, 6 bytes/sec, 0 frames/sec 39496053 frames input, 2696451760 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 167653989 frames output, 230540944084 bytes 70 discards, 0 errors 40377 input OLS, 345 LRR, 4248 NOS, 0 loop inits 5186 output OLS, 1154 LRR, 39568 NOS, 0 loop inits 32 receive B2B credit remaining 32 transmit B2B credit remaining 28 low priority transmit B2B credit remaining Interface last changed at Fri Dec 14 16:50:42 2012 |
So we tried to force the FC 1/1 to run at 1 Gb ( same speed as FC 1/2 ) to create the PortChannel, by doing the basic command on the interface : Switchport speed 1000
Since that , our SAN is logging every 10/15 minutes the following errors :
---------- Problem Found: CUST: CA informational: EVA8400_LIVE: Write resources on the inter site link have been reduced. at Fri 14 Dec 2012 17:21:04 GMT+01:00 ----------
---------- Problem Found: CUST: CA warning: EVA8400_DR: Excessive out of order message rate on the inter site link. at Fri 14 Dec 2012 17:27:04 GMT+01:00 ----------
---------- Problem Found: CUST: CA warning: EVA8400_DR: Excessive data exchange retry rate on the inter site link. at Fri 14 Dec 2012 17:41:34 GMT+01:00 ----------
---------- Problem Found: CUST: CA warning: EVA8400_LIVE: Excessive data exchange retry rate on the inter site link. at Fri 14 Dec 2012 17:41:49 GMT+01:00 ----------
We have the following questions :
Have you already seen that kind of situation ?
Is there a best practice to reduce the interface speed in order to create the PortChannel with identical interface setup ?
Your help and advice would be much appreciated.
Regards
12-19-2012 03:29 AM
the RTT delay by each link is different and this is causing packets to arrive out of order.
not sure if port channeling between two different links or different characteristics is good practive.
12-19-2012 05:59 AM
Would also point out that you are seeing a lot of OLS/LRR/NOS errors - that link from the provider may be experiencing connectivity issues.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide