Slow network performance while using TFTP

[enriquebs]

HI all,

The thing is that TFTP traffic goes really slow while crossing our MPLS/MP-BGP network, at present I?m trying to upload an IOS image (25MB) to a remote PE and it seems to don?t finish untill more than 1 hour...

Here is the output of the "copy tftp: disk0:" command:

Loading c7300-k91p-mz.122-27.SBC5.bin from 172.26.26.99 (via GigabitEthernet0/0): !!!!.!.!!!..!!!.!!!.!!!!!!!!...!!!.!.!!!!!.!!.!.!!!!!!!.!!!!.!!.!!.!!!!!!.!!!!!!!!!!!.!!!!!!!.!!!!!!!!!!!!.!!!.!.!!.!!.!!!!.!!!!!!!!!!!!.!!..!!!!!!.!!!!!!...!!!.!!!!!..!!..!!!..!!.!!.!!!.!!.!!!!!!.!!..!!.!!..!!...!!!.!!!!!!..!!!!!!.!!.!!.!!..!!.!!!!!!!!!!!!!!!.!!!!.!!..!!!!!!!!!.!!!.!!!!.!!.!!!.!!..!.!!!!!!!!.!!.!!!..!!!!!.!!!!.!!.!!..!!.!!.!!..!!..!!!!!!!!.!!.!!..!!..!!!.!!!.!!..!!.!!!!!!!!!!!!!!!!!.!!.!!.!!..!!.!!..!!.!!.!!.!!!!!.!!!!!!!..!!!..!.!!!.!.!.!!!!.!!.!!.!!.!!!!!.!!.!!.!!.!!!!!!!!!..!!!.!!!.!.!!!.!!..!!!!!!!!!.!!!..!!.!!.!!!!!!!.!!!!!!!.!!!!..!!..!!!!!!!!.!!!!!!.!!!!!!.!!!!!!!!!..!!..!!!!!!!!.!!!!!!!.!!!!!!!.!!..!!!!!!!!..!!!!!!!!!!!!!!..!!!!!!!!!!!!..!!!.!!.!!!!..!!!..!!!!!!.!!.!!!!!.!!!!..!!.!!.!!!!!.!!!!!!!!.!!!!.!!..!!.!!!!..!!.!.!!.!!!!!!.!!!.!!!!.!!.!!.!!.!!!!!!!.!!!!!!!!!!!.!!!!.!!.!!!..!.!!!!!.!!.!!.!!!!.!!.!!!!!!..!!.!.!!!.!!!.!!.!!!!..!.!!!!!!..!.!!.!!.!!!!!.!!!!!!!!!!!.!!.!!!!!...!!!!!!.!!!.!!!!!!!!.!!.!!..!!!!.!!.!!!!!!!!..!.!!..!.!..!!!.!!..!..!!.!!!!!!.!!!!!.!!!.!!.!..!!!!.!!!!!!!!..!!.!!!!!!!..!!!.!!..!!..!!!.!.!!!!.!!!!!.!!!!.!!.!!.!!!!!.!!.!!!!.!!.!!.!.!!!!!!!!!!!!.!!!.!!!!!!!!.!!!...!.!!!.!!.!!!.!!!!!!.!!!!.!!.!!.!!.!!.!.!!!!!!!.!!!.!.!!!.!!!!!!!!!.!.!!!!.!!!!!..!!!!!!.!!.!!!!!!!!!!!!!.!!.!!!!!.!!..!!.!!!.!!.!!!.!!.!!.!.!!!!!!!!!!!!.!!!.!!!!.!!!!!!!!!!.!!.!!.!!!.!!.!.!!!!!!.!!!..!!!!!!.!!!!!!!.!.!!!.!!!!!!.!!!.!.!!!!.!!!!!!!..!!!!!.!!!..!!..!!.!.!!!!.!!!..!!!!!!!!!!.!!!!..!!!.!!!!!..!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!.!.!!!.!.!!!.!!!.!!.!!.!!..

and so on......

Has someone experimented something like this?

Thanks in advance,

Kike

[ariela]

Hi Kike,

no problems like yours here, but ...

have you already tryed an extended ping with different mtu?

HTH

Andrea

[enriquebs]

HI Andrea,

Yes, and we don?t loss any packet, extended ping works really good...

[ariela]

Have you already tryed to use a local tftp server (directly connected)?

Regards

Andrea

[enriquebs]

umm, not really... It is a remote site and I must use remote TFTP.

Anyway it is not a big issue because I?m updating the IOS and it is temporary job... but I would like to know why it is happening just with this class of traffic...

[ariela]

mmm ... maybe a disk0 issue?

It sounds like this bug: CSCef26370

I've had something like your problem with a 1751, but without MPLS. I think MPLS is not the problem!

maybe a cpu (too slow) and/or memory (not much) problem ...

HTH

Andrea

[enriquebs]

I don?t think so... because I have the same problem in the eight PE devices (7304) and each one has two different disks, disk0 and stby-disk0... so... I think hardware is working fine. The same problem is happening in the P devices (7604)... it seems like something missconfigured in the network but I can?t find it...

[desai.jaideep]

Hi

I have too faced problems during WAN tftp transfers.The only logic I can infer is that the TFTP server checks for the integrity of each packet.In WAN transition, they may be altered or lost or somehow they coudn't make it to the place where the packet intigrity is being checked.

Thats why sometimes you may get CRC error on TFTP server.

Regards

JD

[enriquebs]

OK that?s a good response but I spent 1.30 hour to complete the upload of a 25MG IOS... and the network has not traffic load at present, just the management traffic... I think it is too long, isn?t it?

[Richard Burts]

The symptoms do look like there might be a traffic loading issue. I wonder what an extended ping with lots of packets would show.

There is an alternative to consider in this situation. You might try transfering the image via FTP instead of TFTP. The transfer with FTP is more efficient than with TFTP. Especially over WAN links where congention might be an issue I have had much better success with transferring images with FTP than TFTP.

HTH

Rick

[enriquebs]

I haven't tried via FTP. Anyway it isn't a production environment, so... WAN links are point to point fiber links don?t shared with other people... network is not congested.

I don?t wanna find any other method to upload the images, just "why" it is happening :)

[ariela]

Hi enriquebs,

"TFTP is not well suited to large files for the following reasons.

TFTP has no inherent integrity check. There is no way to determine

what one side sent is what the other received. There is no way to

restart TFTP transfers from anywhere other than the beginning. TFTP

is a lock step protocol. Only one packet may be in flight at any one

time. There is no slow start or smart backoff mechanism in TFTP, but

very simple timeouts."

(RFC 3617)

your problem is timeouts ... but why, I don't know.

A very basic troubleshooting will check, after a trace, step-by-step all hops (cpu, mem, int errors). If you please, try a tftp from tftp server to all hops, and check where you have a packet loss.

Maybe it does the trick

HTH

Andrea

generic advice: check duplexing, enable UDLD

[enriquebs]

Hi Andrea,

Thanks a lot for the information :)

I?ll do that and I?ll post the results.

[enriquebs]

Here is the output when I try the TFTP to the remote PE router:

Loading c7300-k91p-mz.122-27.SBC5.bin from 172.26.26.99 (via GigabitEthernet0/0): !!!!!!!!!!.!!.!!..!!.!!!!!!!!!.!.!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!.!..!!!!!!!!!.!!!!!!!..!!!!!.!!..!!!.!!!!!!!.!.!.!!.!!.!!!.!.!!!!!.!!!.!.!!!..!!!!!.!!!!.!!.!!!.!!.!!!!!!!.!!!.!!.!!!!!!!!.!!..!!!!!!.!!!..!!!!.!!..!!.!!!.!!...!!.!..!!.!!!!!.!!!!!!!.!!.!!.!!!!.!!.!.!!!!!.!!.!!.!!!.!!!.!!!!!!!.!!!.!!!!!!.!!!!!..!!.!!...!!.!!...!!!!!!.!!!!!.!!.!!!.!!.!!.!!.!!.!!!.!!!...!!.!!!!!!!!.!!!!!!!.!!!!!!!!!!!!!!.!!.!!.!..!!!!!!..!!!!!!!..!!!!!!!!!!!!!!!.!!!!!!.!!!!.!!...!!!!.!!!.!!!!..!!!!!!!!!!!!!!!!!!!!...!!!!.!.!!!!!.!!!!!.!!..!!!.!!...!!...!!!!.!.!!!!!!!.!!!!!!!!!.!!.!!!!!!..!!.!!.!!!!!!!!!!..!!.!.!!.!.!!!!!!!!!.!!!..!!.!!!.!!!.!!..!..!!!!.!!!!!!!!!!.!!!!.!!!.!!.!!!!.!!!!!!!!!.!!!.!!!!.!!.!!!..!!!!!!!!!!!!!!..!!!.!!!!!!!!!!!!..!!!!.!!.!!!.!!!!!!.!!!!.!!!!!..!!.!!.!!.!!!!.!!..!!.!!!.!!..!!.!!!...!!!!!..!!.!!!.!...!!!!.!!!!.!.!!!!.!!!!!.!!!!!!!!!.!!!!!..!!.!!!!!!!!.!!!!!.!!..!!!.!!!.!!.!!..!!!!.!!!!!.!!.!!.!!!!.!!!.!!!!.!!!!..!!!.!!!!!.!!!!!.!!!.!!.!!.!!!!.!!.!!!!!!

And here is to the closer PE router:

Loading c7300-k91p-mz.122-27.SBC5.bin from 172.26.26.99 (via Vlan3021): !!!!!!!!!!!!!!!!!!!!!.!.!!!!!!!..!!!.!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!.!...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!...!!!!!!!!!!!!!!!!..!!!!!!!..!.!!!!!!!!!.!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!

It continues going too slow but seems better than the remote PE.

[ariela]

well,

The closer PE cpu/mem are ok when you try the download? No errors on 'sh int'?

Regards

Andrea