File Transfers Dropping over site to site VPN tunnels (GRE ove IPSec)

[jwhite100]

We have an issue on our network that when we transfer files from site to site using windows the transfer drops and we get the error message "network not available". It doesn't happen all the time but 9 times out of ten it does! Some tunnels work better than others. We currently have 4 sites.

I have posted on another forum. We have tried many things but no one has been able to resolve the issue so far. Please check my other post for an update.

http://www.networking-forum.com/viewtopic.php?f=35&t=19354&p=120181#p120181

[paulbrazier]

Good morning all,

I am working with James on this issue and though I would share some more information with you.

Attached is a diagram our prevoius network manager created of the site tunnels, this should help you understand the layout.

We have 4 sites:

1. Primary hub - Wokingham, UK 10Mb fibre leased line

2. Reading, UK (VEE) 7Mb fibre lease line

3. Tampa, US 1Mb Serviced office

4. Maidenhead, UK 1Gb ehternet transit provided - Site has 2 * 7201's on different external networks and participate in BGP with our own RIPE address space.

We have been left to manage a fairly complex setup with failry minimal knowledge cisco knowledge :-)

------------------------------------------

New findings
On my latest finding I have disabled a link to Blue Square House (seconhdary tunnel to Maidenhead) so we are linking from Wokingham to that site using Blue Square 3.

Now becuase BSH is the active BGP router i.e. announcing our address space we will be routing through this to get to BS3 internal IP to kick off the tunnel, this seems wrong to me?

However all other site are not this complex i.e. 2 routers and BGP and we still have issues there.

I believe James is looking into keepalives as this is something we do not use.

Any other info required please shout.

Many thanks for all your help so far.

Regards

Paul

[Marcin Latosiewicz]

Paul, James,

I have not been following on both of the threads fully, so forgive me if I'm asking for something already discussed/checked.

I beleive:

- Keepalives are ment as isakmp keepalives - and they were not configured as far as I understand? "show run | i keep" will show you.

Them not being enabled will basically say that tunnel itself does not drop (even tho connectivity MIGHT be impacted between IKE sites)

- if it's GRE keepalives you're looking for - they are not supported with tunnel protection.

My eariler suggestion was to check "show crypto engine accel stati" during the issue to see if there is something wrong there.

I would advise a simple and effective test during scheduled window - disable encryption on both sides and check if the issue perists.

If the issue does not persist without crypto (tunnel protection) it's either crypto accelarator problem or a hidden MTU problem.

What is the maximum ICMP packet size you can ping through the GRE tunnel with Df-bit set?

Remember that ip policy will not impact locally generated packets. (unless it's "ip local policy ...")

Marcin

[Lei Tian]

Hi,

I have looked the trace file. was it captured between Volume to Blue Square? The traffic direction is from 172.16.8.1 to 172.16.33.100, is 172.16.8.1 in Volume?

Based on the trace, I can see some packets been dropped from 172.16.8.1 to 172.16.33.100, which cause lot retransmissions. Probably you can identify where does the packet been dropped first. It could be dropped by Router interface, or vpn card, or provider. After we know where the packet been dropped, it will be easier to find the solution.

HTH,

Lei Tian

[paulbrazier]

Hi Marcin,

VOL-GATEWAY#show run | i keep
crypto isakmp keepalive 3600
no keepalive
no keepalive

I believe there are a few ways to setup GRE ipsec protected tunnels, the guy who did ours used tunnel protection, however on a test enviroment using Cisco Professional wizzard (sorry) it seemed to use crypto maps?

VOL-GATEWAY#show crypto engine accel stati

Device:   Onboard VPN
Location: Onboard: 0
        :Statistics for encryption device since the last clear
         of counters 1475003 seconds ago
              131135714 packets in                   131135704 packets out
            83412267287 bytes in                   83024152649 bytes out
                     88 paks/sec in                         88 paks/sec out
                    452 Kbits/sec in                       450 Kbits/sec out
               69454566 packets decrypted             61681138 packets encrypted
            47919849488 bytes before decrypt       35486928075 bytes encrypted
            45333650429 bytes decrypted            37690502220 bytes after encrypt
                      0 packets decompressed                 0 packets compressed
                      0 bytes before decomp                  0 bytes before comp
                      0 bytes after decomp                   0 bytes after comp
                      0 packets bypass decompr               0 packets bypass compres
                      0 bytes bypass decompres               0 bytes bypass compressi
                      0 packets not decompress               0 packets not compressed
                      0 bytes not decompressed               0 bytes not compressed
                  1.0:1 compression ratio                1.0:1 overall
                Last 5 minutes:
                  17819 packets in                       17819 packets out
                     59 paks/sec in                         59 paks/sec out
                 236369 bits/sec in                     237824 bits/sec out
                2335953 bytes decrypted                6137127 bytes encrypted
                  63133 Kbits/sec decrypted             165868 Kbits/sec encrypted
                  1.0:1 compression ratio                1.0:1 overall

----------------------------

On one of the links I have removed tunnel protection and copied files over, did seem a little better however still failed!

C:\Users\paulb>ping -f -l 1372 172.16.33.100

Pinging 172.16.33.100 with 1372 bytes of data:
Reply from 172.16.33.100: bytes=1372 time=8ms TTL=125
Reply from 172.16.33.100: bytes=1372 time=8ms TTL=125
Reply from 172.16.33.100: bytes=1372 time=8ms TTL=125
Reply from 172.16.33.100: bytes=1372 time=8ms TTL=125

Ping statistics for 172.16.33.100:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 8ms, Maximum = 8ms, Average = 8ms

C:\Users\paulb>ping -f -l 1373 172.16.33.100

Pinging 172.16.33.100 with 1373 bytes of data:
Reply from 10.1.255.1: Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for 172.16.33.100:
    Packets: Sent = 4, Received = 1, Lost = 3 (75% loss),

Thanks for your post!

[paulbrazier]

Hi Lei

Correct the trace was from Volume 172.16.0.0/20 to Bluesqaure 172.16.33.0/24

Do I just try an position my laptop on different areas of the network to check where the packets are dropped?

We have an ISP managed router in Volume and we buy transit at bluesquare with our own routers so will be easier to monitor that end.

Have not done too much with packet tracing on a switched network any pointers welcome

Thanks

Paul

[Lei Tian]

Hi,

Can the provider help to test the link?

Here is a SPAN configuration guide for 3560, other cisco switches use similar syntax. You can use SPAN to trace packets on your switch. The initial trace file you provided missed lot packets, not sure if it is sniffer's filter problem, a lot packets are not seen on either end.

Regards,

Lei Tian

[paulbrazier]

Thanks Lei,

FYI the filter was applied after the capture on the TCP stream so I believe that packets were being dropped.

I shall look a little more into SPAN have heard it being talked about before.

Thanks

Paul

[jwhite100]

I've just done a bit of testing. It seems that at present transfers are only going in one direction. Please have a look at my findings:

	Date	Time	Size	Result	Notes
To BSQ	20/09/2010	5pm	121MB	Failed
To BSQ	20/09/2010	5.10pm	121MB	Failed
From BSQ	20/09/2010	5pm	58MB	OK
From BSQ	20/09/2010	5.10pm	388MB	OK
To Vee	20/09/2010	5.30pm	121MB	Failed
From Vee	20/09/2010	5.36pm	258MB	OK
To Tampa	20/09/2010	05/01/1900	121MB	Failed
From Tampa	17/09/2010	4pm	2.5GB	OK
From Tampa	20/09/2010	6pm	16MB	OK
From Tampa	20/09/2010	6.10pm	130MB
To Tampa	20/09/2010	6.1	17MB	OK