50% packet loss to/from 2621 router

[cmcfarling]

I have a Cisco 2621 router in front of a Watchguard Firebox III 700. The interface (FastEthernet0/1) IP on the Cisco facing my LAN is 100.200.300.1, for example. The IP on the FBIII external interface is 100.200.300.2.

Using any computer behind the FBIII, if I ping the Cisco at 100.200.300.1, 50% of the packets are dropped. Likewise, from the Cisco, if I ping the FBIII at 100.200.300.2 50% of packets are dropped.

Any packets passing through the Cisco (the router is not the source or destination) seem to be fine, i.e. no packet loss.

As a result when I try to copy the system image from the Cisco to a TFTP server behind the FBIII, some data gets through but the copy eventually fails. The copy status on the Cisco console looks something like this

.!!.!...!.!...!...!!.....

A period represents a timeout and a bang represents 10 packets sent.

I'm leaning toward the issue being with the Cisco router but I'm not positive. I'm wondering if anyone has seen this behavior and has any helpful hints.

[JORGE RODRIGUEZ]

Could you provide show interface fe0/1 statistics.. as a base line record the stats on Fe0/1 , then clear counters and note any crc errors on the interface, as well as your T1's.

Im just curious on the interface stats.

[cmcfarling]

cisco2621#show int FastEthernet0/1

FastEthernet0/1 is up, line protocol is up

Hardware is AmdFE, address is 0008.a3b3.b6a1 (bia 0008.a3b3.b6a1)

Description: SBC primary T1

Internet address is 64.171.123.1/24

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:02:30, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Queueing strategy: fifo

Output queue 0/40, 0 drops; input queue 0/75, 0 drops

5 minute input rate 71000 bits/sec, 41 packets/sec

5 minute output rate 647000 bits/sec, 64 packets/sec

1516167236 packets input, 432529126 bytes

Received 743516 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast

0 input packets with dribble condition detected

1320432057 packets output, 4259644730 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 2452001 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

[JORGE RODRIGUEZ]

you have high deferred transmissions on the interface.. whats the routers cpu utilization.

Is this packet loss anomaly suddently developed ? were you able to ping this interface witout any packet losses in the past?

[cmcfarling]

It's been this way for a while. I can't say if it has always been this way though. It's just that I decided to finally try to get to the bottom of it. Attached is the cpu utilization. This output shows it being minimal. Even in this state I get the same packet loss.

[JORGE RODRIGUEZ]

Im definately missing something , I don't see anything in the configuration you provided to all causing packet losses, we all know this is only happening on interfatce fe0/1, so it is narrowed to that interface, as a practice I always hardcode fe interfaces at both ends to 100 full duplex even if no crc's or other errors are seen..

I looked of any bugs on your ios 12.0 code but could not find anything on this anomaly.

I will take a second look on bugs.

I would do the following

1- Hardcode trans speed (both ends )

2- Upgrade code from 12.0

[2pparish]

You need to use debug ip packet and debug ip icmp

to find out how those packets are being forwarded or dropped.

The simple solution, like someone said, is use these commands

debug ip packet

debug ip icmp

good luck to you :)

[scottholwerda]

You should hard code the speed and duplex on both sides to prevent auto negotiation conflicts.

Scott

[cmcfarling]

If this were a speed/duplex issue, wouldn't it stand to reason that all traffic through this interface would be affected and not just traffic to/from the 2621?

[cmcfarling]

Problem solved.

By enabling debug ip packet I was able to gleen a little more info from the log. I saw that CEF switching was coming into play. Disabling CEF seems to have solved the problem. I don't know enough about CEF to determine why it would cause this behavior though. Should I expect any detrimental effects by disabling CEF?

[Richard Burts]

Chris

I am glad that the debug that we suggested was able to help you find the issue. I am surprised that it was caused by CEF and it is likely that an upgrade to more recent code would resolve it.

CEF is an enhanced packet switching path in IOS. If CEF is disabled then you are left with switching paths of fast switching, process switching, etc. CEF has some performance advantage over fast switching (there is no need to process switch the first packet every time the router is forwarding to a destination not present in the fast switching cache) and fast switching certainly has a performance advantage over process switching. So if you have disabled CEF there will be some performance difference. Whether it will be enough to notice or enough to be detrimental is hard for us to tell without knowing a good deal more about the volume of traffic and kind of traffic being processed by the router.

HTH

Rick