4451-x WAN Latency Problem

[scbctaps307]

Hello,

Hoping someone can assist with a problem we're experiencing across our WAN as follows:

• We have a fiber WAN with 2 pairs that is approximately 21 Miles (35 KM's) in length.
• At each end we have a 4451-x router.
• The routers have been enabled with VPN and encryption.
• When data traverses across the WAN the routers seem to experience very high latency and eventually reboot by themselves.
• The 21 mile length of fiber cable has been tested from one router's fiber ports to the other router's fiber port and it is stable.  Therefore, our conclusion is that the problem is either the SFP transceivers or the routers.  The transmit power and the receive power of the 4 sfp transceivers is well within the range as stated by the GLC-ZX-SMD documentation.  Transmit is approximately 2.5 dBm and Receive is approximately 8.0 dBm.

Has anyone experience something similar or does anyone have any suggestions?

Thanks!

[Joseph W. Doherty]

Gig link on a 4451, eh?  You licensed for 1 or 2 gig throughput?

Encrypted VPN, eh? Is fragmentation possible?

What's you bandwidth utilization look like, and concurrent CPU usage look like?

Please further clarify/define meaning of very high latency?

What IOS version are you running?

[scbctaps307]

Thanks for the reply Joseph,

We're licensed for 1 gig throughput.

IOS version is IOS XE 03.13.00.S and boot flash is "bootflash:/isr4400-universalk9.03.13.00.S.154-3.S-ext.SPA.bin".

As for the latency, we used a program named "ping plotter" that would send icmp echo packets between the routers.  Approximately 45 minutes into the test you could see packets accumulating into what I presume to be the buffer.  Application performance would degrade and then the router reboots.

Someone mentioned that the length of the fiber WAN is shorter than the minimal required by the SFP transceivers.  The WAN is 21 Miles (35 KM's) and the sfp transceivers minimum distance is 43.4 miles (69.84 KM's).  It was suggested we use attenuating cables at one end.  Either 6dB or 3dB attenuating cables.

The CPU and bandwidth utilization is minimal as the link is not live in production at the moment. When we enable the line for testing and have data tx/rx through the VPN link the bandwidth consumption is minimal.  On that note can you please send me the commands to check the bandwidth and the concurrent cpu utilization.  I usually do the "show processes cpu" and show "memory commands" but if there's alternatives please let me know. 

Hope this helps and greatly appreciate the assistance.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I'm familiar with Ping Plotter.  What are you actually pinging?

Yea, I too was wondering about the ZX at 35 KM.  There's general rules for when you need to attenuate the signal.  The critical issue being the received dBm, and what the receiver supports for minimum and maximum.  (From a quick read of ZX specs, you often need an attenuator for distances under 40 Km.)

ZX optics often have additional module stats, such as received dBm, and the min/max.

If you're over driving the receiver's dBm, assuming you don't damage the optical module, I would not have expected ping time increases or router reloads; I would have expected corrupted frames.  So, I wonder if your IOS version has bugs (if so, maybe something triggered by the optical module being out of spec).  You might try, if Cisco has any for a 4451, an IOS version tagged for stability.