50% packet loss to/from 2621 router

[cmcfarling]

I have a Cisco 2621 router in front of a Watchguard Firebox III 700. The interface (FastEthernet0/1) IP on the Cisco facing my LAN is 100.200.300.1, for example. The IP on the FBIII external interface is 100.200.300.2.

Using any computer behind the FBIII, if I ping the Cisco at 100.200.300.1, 50% of the packets are dropped. Likewise, from the Cisco, if I ping the FBIII at 100.200.300.2 50% of packets are dropped.

Any packets passing through the Cisco (the router is not the source or destination) seem to be fine, i.e. no packet loss.

As a result when I try to copy the system image from the Cisco to a TFTP server behind the FBIII, some data gets through but the copy eventually fails. The copy status on the Cisco console looks something like this

.!!.!...!.!...!...!!.....

A period represents a timeout and a bang represents 10 packets sent.

I'm leaning toward the issue being with the Cisco router but I'm not positive. I'm wondering if anyone has seen this behavior and has any helpful hints.

[Richard Burts]

Chris

The symptom of 50 % packet loss (especially if it really is exactly 50 %) is frequently the result of having 2 routes in the routing table and one of them works and one does not. When the router is generating packets (traffic from the router not traffic through the router) it will send packet by packet over both routes and 50 % of the packets get lost.

Can you post the output of show ip route from the router?

HTH

Rick

[cmcfarling]

Sure...

cisco2621#show ip route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP

i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

* - candidate default, U - per-user static route, o - ODR

P - periodic downloaded static route

Gateway of last resort is 71.128.141.41 to network 0.0.0.0

71.0.0.0/30 is subnetted, 1 subnets

C 71.128.141.40 is directly connected, Serial0/0.1

64.0.0.0/24 is subnetted, 1 subnets

C 64.171.123.0 is directly connected, FastEthernet0/1

67.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

S 67.153.169.20/30 [1/0] via 67.155.215.249

C 67.155.215.248/29 is directly connected, FastEthernet0/0

B* 0.0.0.0/0 [20/0] via 71.128.141.41, 5w3d

FastEthernet0/1 is the interface I'm referring to. Using real IP's, issuing ping from the 2621 to 64.171.123.2 I see 50% packet loss.

[Richard Burts]

Chris

Thanks for posting the additional information. It does not show what I had thought it might and there is not an indication here that it might be the issue with 2 routes which I had thought it might be.

Would you post the output of show arp? It might also be helpful to turn on debug ip icmp, try the ping again, and post the debug output.

HTH

Rick

[jj-zhou]

Hi cmcfarling :

the Firebox maybe not respond the icmp ack packet ?

[JORGE RODRIGUEZ]

Did you ruled out the physical aspect? duplex transmission missmatch etc.. on all devices including your computer .

[cmcfarling]

Here's the show arp output:

cisco2621#show arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 64.171.123.200 39 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.201 20 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.202 17 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.1 - 0008.a3b3.b6a1 ARPA FastEthernet0/1

Internet 64.171.123.2 39 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.61 0 Incomplete ARPA

Internet 64.171.123.35 128 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.36 123 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.37 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.38 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.39 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.40 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.41 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.42 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 67.155.215.250 - 0008.a3b3.b6a0 ARPA FastEthernet0/0

Internet 67.155.215.249 2 00a0.c811.2ed0 ARPA FastEthernet0/0

Here's a ping attempt with the resulting log output with debugging on:

cisco2621#ping 64.171.123.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 64.171.123.2, timeout is 2 seconds:

!.!.!

Success rate is 60 percent (3/5), round-trip min/avg/max = 1/1/1 ms

cisco2621#show log

Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)

Console logging: level debugging, 375 messages logged

Monitor logging: level debugging, 0 messages logged

Buffer logging: level debugging, 346 messages logged

Trap logging: level informational, 36 message lines logged

Log Buffer (4096 bytes):

45w3d: ICMP: echo reply rcvd, src 64.171.123.2, dst 64.171.123.1

45w3d: ICMP: echo reply rcvd, src 64.171.123.2, dst 64.171.123.1

45w3d: ICMP: echo reply rcvd, src 64.171.123.2, dst 64.171.123.1

cisco2621#

I believe the physical connections are ruled out. Have verified the duplex settings are correct on all devices. Besides, only traffic to/from the 2621 seems to be affected. Traffic through the 2621 is not experiencing packet loss/performance issues.

[Richard Burts]

Chris

Thanks for the additional information. Unfortunately it does not seem to point to the answer. I am surprised that it shows receiving a response but does not show sending the request. But while I think about that I will suggest something else that we can try. Would you turn on debugging for ip packet (with an access list), attempt the ping, and post the debug output.

- first create an access list to use with debug:

access-list 199 permit ip host 64.171.123.2 any

access-list 199 permit ip any host 64.171.123.2

- then run debug using the access list:

debug ip packet 199

- then try the ping

- then capture and post the debug output

- then remember to turn off the debug

HTH

Rick

[anandramapathy]

is the firewall & router directly connected ?

Try this if they are connected through a switch

connect another PC with the IP 64.171.123.3 & ping both the firewall interface & the router interface.

see what results you get

[cmcfarling]

I'll work on the access lists. In the meantime I setup another computer on that network segment at 64.171.123.3 for example. From that host, when I ping the 2621 I get 50% packet loss. When I ping the Watchguard I get 0% loss. It definitely seems to be an issue with the router.

[Richard Burts]

Chris

Perhaps it would help us if we knew a bit more about the topology of the network. In looking at the ARP table that you posted it looks like most of the addresses are behind the firewall from the router (they all have MAC of 0090.7f1f.ad22) and there are 3 addresses at MAC 00a0.cc52.5ab3. So what is the other box on the network?

When you connected another computer was it behind the firewall also? Do you have the same experience of packet loss if you ping to the .200 or .201 or .202 addresses?

cisco2621#show arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 64.171.123.200 39 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.201 20 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.202 17 00a0.cc52.5ab3 ARPA FastEthernet0/1

Internet 64.171.123.1 - 0008.a3b3.b6a1 ARPA FastEthernet0/1

Internet 64.171.123.2 39 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.61 0 Incomplete ARPA

Internet 64.171.123.35 128 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.36 123 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.37 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.38 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.39 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.40 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.41 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 64.171.123.42 78 0090.7f1f.ad22 ARPA FastEthernet0/1

Internet 67.155.215.250 - 0008.a3b3.b6a0 ARPA FastEthernet0/0

Internet 67.155.215.249 2 00a0.c811.2ed0 ARPA FastEthernet0/0

HTH

Rick

[cmcfarling]

There's nothing on that segment other than the external Firebox interface, the FastEthernet0/1 Cisco interface and another host with the 3 IP's noted (.200,.201,.202). That segment is defined as a VLAN on an HP switch.

If I ping 64.171.123.1 from the host at 64.171.123.200 I get 50% packet loss

No packet loss when pinging .200 (or .201 or .202) from behind the Firebox. From the internet, if I ping the Firebox at 64.171.123.2 there is no packet loss. Pinging the 2621 at 64.171.123.1 from the internet results in packet loss though. If you were to ping that address you should see packet loss.

[cmcfarling]

BTW, BGP routing is employed on this router. Could that have anything to do with it?

[Richard Burts]

Chris

I had noticed the routing table default route was learned from BGP, so was aware that BGP was running. I believe that it is highly unlikely that BGP has anything to do with it.

I did take your suggestion and pinged the router interface. Actually I pinged both interfaces. And I am getting the same behavior on both of them. As a detail I am getting about 40% loss, not 50%. And that convinces me that it is not an extra route in the routing table as I had originally thought it might be.

I am wondering if the router is throttling its own packets for some reason. Is there any shaping or policing of traffic or any QOS configured on the router? Perhaps you can post the config of the router?

HTH

Rick

[cmcfarling]

See attachment for config.