Pinging a WAN interface from a ISP monitoring platform

[kgreene]

Hi:

We are going round and round with Cogent on this ping health check issue. Our ACL's have been in place for some time. Recently we had a link failure and when the T-1 returned to service, Cogent claimed they could not ping the WAN serial interface anymore. Infact, they stated they could not ping the other T-1 interface also.

I far as I can see within the incoming ACL for the fist T-1. we have this set of rules that should allow it:

ip access-list extended September-2010-199-2.29
permit tcp any 199.4.110.0 0.0.0.255 established
deny   ip 199.4.110.0 0.0.0.255 any log-input
deny   ip 10.0.0.0 0.255.255.255 any log-input
deny   ip 192.168.0.0 0.0.255.255 any log-input
deny   ip 172.16.0.0 0.15.255.255 any log-input
deny   ip 224.0.0.0 31.255.255.255 any log-input
deny   ip host 255.255.255.255 any log-input
permit icmp any any echo
permit icmp any any echo-reply
deny   icmp any any log-input

(the rest of the ACL was deleted for security reasons)

The outgoing ACL has this rule:

ip access-list extended NewInternet_v1.51
permit ip 199.4.110.0 0.0.0.255 any
permit ip host 140.174.40.90 any
permit ip 206.184.209.0 0.0.0.255 any
permit icmp any any echo
permit icmp any any echo-reply
deny icmp any any

From an external IP, you can not ping the WAN IP address. You can ping through the router to the LAN addresses.

I have tried turning off the outbound ACL and even I turned off the incoming ACL with no effect.

Running out of ideas here.

So...

It is either they can not send pings correctly (unlikely) or something is wrong in the actual ACL (I have no clue where).

Any suggestions would be great.

The Router is a Cisco 2621 with dual T-1's and two FE ports.

Thanks.

Kevin

[gatlin007]

Your ACL does look good.  If you can get them on the phone while they try to ping your interface it would be helpful.  Run a 'debug ip icmp' on your router.  Then from config mode execute a 'logging monitored debugging'; after this from privileged exec mode execute a 'term mon'.  Immediately execute a 'show proc cpu' and make sure your CPU isn't spiked.  I doubt it would be as this debug command is generally always safe unless someone is flooding you with ICMP.

At this point the ICMP echo and echo-reply messages your router receives and generates should stream across your telnet/ssh session.  I wouldn't think that it would be so verbose that you couldn't spot them trying to ping your WAN interface.  This will give you some idea as to if the service providers echo packet ever reaches you.  It will also make you aware if your router is sending the echo-reply.  At that point check your routing table for were your router is sending it.  Sometimes if you have redundant links you may be sending it out the other router.

After troubleshooting run a 'undebug all' from privileged mode to make sure you are not vulerable to an ICMP attack.

Chris

[Richard Burts]

Kevin

first a couple of observations and then a couple of questions and possible issues:

- if they are trying to ping the router interface then the problem can not be the outbound access list. An outbound access list will not filter traffic that is generated by the router itself.

- you state that you removed the inbound access list and the problem continued. If correct then this was good troubleshooting and clearly demonstrates that the problem is not related to access list filtering (and the part of the access list that you posted certainly looks like it would not cause problems with ping).

= it is not clear from your post whether other traffic is working and the problem is only with ping to the interface or whether the problem is that no traffic works through the interface. Can you clarify whether the interface is working for other things?

= can you ping from your router to any remote resource, specifying the problematic interface as the source of the ping (using extended ping for this)?

= can you ping from your router to the connected interface of the ISP router, specifying the problematic interface as the source of the ping (using extended ping for this)?

= are you running CDP on the T1 interfaces? If so posting the output of show cdp neighbor detail might be helpful.

= you mention that there are 2 T1s and that there was an outage. Is it possible that the cables for the T1s got swapped in the process of troubleshooting and resolving the outage?

If you can provide this information we may be in a better position to help resolve your issue.

HTH

Rick

[kgreene]

Hi:

Sorry I have not responded sooner. Been a bit busy this week.

The router (2621) is in operation and does pass traffic. The ACL's are on the right interface as otherwise, many things would not work.

From our remote office (a ADSL provided by ATT), I have an established VPN connection to the VPN server located behind the edge router (the 2621). In addition, the people who access the various web services have not complained.

Again from the remote office, I can ping any interface on the inside lan without a problem. But I can not ping the WAN IP of the serial interface. I can ping Cogent's end of the circuit however.

Feel free to ping the interface yourself to see if there is some weird.

Cogent end of the WAN interface: 140.174.40.89

My end of the WAN link: 140.174.40.90

I have tried as I mentioned even turning off BOTH ACL's (hated to do this) and tried pinging the WAN with no luck. By both, I mean the incoming and outgoing ACL's. The other T-1 does not have an outgoing ACL for some reason but it has been like this for several years.

You asked if I could ping the interface from the CLI. I did and it works.

cisco2621# ping 140.174.40.90

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.90, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/13/20 ms
cisco2621# ping 140.174.40.89

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.89, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/8/8 ms
cisco2621#

However if I try to ping Google as an example:

cisco2621#ping www.google.com
Translating "www.google.com"...domain server (199.4.110.11) [OK]

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 74.125.19.147, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
cisco2621#

It does not work on this router. However - This does work on my remote location router.

I am not running CDP on the WAN interfaces. Should I???

I am wanting to resolve this. Any ideas would be great.

Kevin

[Richard Burts]

Kevin

First let me address the CDP question. Many people that I know do not run CDP on interfaces that face outside of their network and consider CDP on outward facing interfaces to be a security issue. I do not advocate running CDP on outward facing interfaces in normal circumstances. There are some circumstances where running CDP can provide very helpful troubleshooting information, and perhaps this might be a situation where CDP information could be useful. Of course if Cogent is not running CDP then you will not get anything useful. But it might be worth it to try turning on CDP for the T1 interfaces (I would do both), wait for a couple of minutes and try show cdp neighbor detail. And then turn off CDP.

When I attempt ping I get 100% packet loss to both addresses. And when I attempt traceroute I get into the Cogent network but not to anything that appears to be close to your subnet. So it looks to me like Cogent is doing some filtering, which may or may not relate to your particular issue. And that might explain why the ping to Google from the router fails. Do you know if ping from the router to Google worked before these problems started?

It is good news that traffic through the router works ok and that the VPN sessions establish without problem. It makes me wonder if the issue is something about the interfaces themselves. Would you post the output of traceroute on the router to the Cogent interface address and traceroute on the router to its own serial interface address?

HTH

Rick

[Richard Burts]

Kevin

This is a very strange situation and I do not yet see a clear answer about what is going on.

First let me address an observation in one of my previous posts. I said that my attempts to ping either your interface or the Cogent interface results in 100% packet loss. That appears to be an issue with the firewall running on my laptop (with which I have been having some problems). I have changed setting in my laptop firewall and now I do get the results that you describe: ping to the Cogent address is successful but ping to your address fails.

My sense is that Cogent is blocking traffic. I base this on the combination of factors including:

- ping to their interface works while ping to your interface fails (and you now have proof that the ping to your interface is received on your router and that your router is attempting to respond).

- ping from your router to an address near your router is successful while ping from your router to a remote address fails.

- traceroute to the Cogent address and to your address gets part way into their network and then fails.

But I have not yet come up with a way to convince Cogent of this.

I will continue to think about this. Perhaps if you could provide some additional details about your environment, about the problem that impacted the T1, and about what was done to recover from that episode we might get some better understanding.;

HTH

Rick

[edit] let me acknowledge that my response shows up much earlier in the thread than I expected, which may make it difficult to follow the chronology of the responses.

And then let me agree with Chris that it is important to respond appropriately when our wives call. And I would suggest that in addition to the show ip route that he suggests, that it might be helpful to do a traceroute to that address and see if the traceroute shows what is the next hop, and how many hops it goes.

[kgreene]

Hi:

I enabled CDP on the router and then enabled the CDP on both serial interfaces.

I then did:

cisco2621#show cdp neighbor detail

cisco2621#show cdp neighbor detail

cisco2621#show cdp neighbor detail

cisco2621#

And saw nothing (I waited about 5 minutes).

If I do a traceroute to either the local WAN port on either serial card, I see nothing. It comes with no results.

cisco2621# traceroute  140.174.40.90

Type escape sequence to abort.
Tracing the route to d1-2-1-3-17.a01.snfcca02.us.ce.verio.net (140.174.40.90)

  1  *  *  *
  2  *  *  *
  3  *  *  *
  4  *  *  *
  5  *  *  *
  6  *  *  *
  7  *  *  *
  8  *  *  *
  9  *  *  *
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *

Same with the other serial port.

As a side note: the IOS is: c2600-i-mz.123-6.bin

It is has been working for a long time (years) without a problem.

I have done a hardware rest with no differences. I have done a software reset with no differences.

It is almost like there is some device in the middle of the Cogent WAN that is filtering traffic. Am I paranoid???

Cogent claims there is no filtering done on their routers.

Not sure I believe this anymore.

Kevin

[Richard Burts]

Kevin

Thanks for trying CDP. I am not entirely surprised that it did not work but thought that it was worth a try. It probably means that Cogent has it turned off on their router (or that their router is not Cisco). So make sure it is turned off on your router.

I am surprised that the ping to the remote interface did work but that the traceroute did not. I have another experiment that I would ask you to try.

- do show interface for both T1s and note the input and output packet counters.

- do an extended ping to the 40.90 address and in the extended ping send 5000 packets

- do show interface for both T1s and note the input and output packet counters. are they both going up? are they going up on the same interface?

HTH

Rick

[kgreene]

OK... Thanks for the email.

Here is the ping test. I hope it helps.

Kevin

cisco2621#show interfaces serial 0/1
Serial0/1 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 199 Internet connection (HDLC)
  Internet address is 140.174.40.90/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 12/255, rxload 9/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/19460/0 (size/max/drops/flushes); Total output drops: 2052
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/2038 (size/max total/threshold/drops)
     Conversations  0/40/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 57000 bits/sec, 77 packets/sec
  30 second output rate 76000 bits/sec, 91 packets/sec
     10517425 packets input, 2760598167 bytes, 0 no buffer
     Received 76575 broadcasts, 0 runts, 82 giants, 0 throttles
     162012 input errors, 44859 CRC, 88332 frame, 0 overrun, 0 ignored, 28820 abort
     7873081 packets output, 2062179523 bytes, 0 underruns
     0 output errors, 0 collisions, 34 interface resets
     0 output buffer failures, 0 output buffers swapped out
     15 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#show interfaces serial
% Incomplete command.

cisco2621#ping 140.174.40.90 repeat 5000

Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 140.174.40.90, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

(lines edited out to save space)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (5000/5000), round-trip min/avg/max = 12/13/36 ms
cisco2621#show interfaces serial 0/1
Serial0/1 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 199 Internet connection (HDLC)
  Internet address is 140.174.40.90/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 30/255, rxload 18/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:02, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/19460/0 (size/max/drops/flushes); Total output drops: 2052
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/2038 (size/max total/threshold/drops)
     Conversations  0/40/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 110000 bits/sec, 141 packets/sec
  30 second output rate 183000 bits/sec, 167 packets/sec
     10530583 packets input, 2761850861 bytes, 0 no buffer
     Received 76584 broadcasts, 0 runts, 82 giants, 0 throttles
     162012 input errors, 44859 CRC, 88332 frame, 0 overrun, 0 ignored, 28820 abort
     7888603 packets output, 2063978495 bytes, 0 underruns
     0 output errors, 0 collisions, 34 interface resets
     0 output buffer failures, 0 output buffers swapped out
     15 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#ping 140.174.40.89 repeat 5000

Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 140.174.40.89, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

(lines edited out to save space)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (5000/5000), round-trip min/avg/max = 4/7/220 ms
cisco2621#show interfaces serial 0/1
Serial0/1 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 199 Internet connection (HDLC)
  Internet address is 140.174.40.90/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 23/255, rxload 14/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/19460/0 (size/max/drops/flushes); Total output drops: 2052
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/2038 (size/max total/threshold/drops)
     Conversations  0/40/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 90000 bits/sec, 127 packets/sec
  30 second output rate 143000 bits/sec, 164 packets/sec
     10538523 packets input, 2762551800 bytes, 0 no buffer
     Received 76591 broadcasts, 0 runts, 82 giants, 0 throttles
     162012 input errors, 44859 CRC, 88332 frame, 0 overrun, 0 ignored, 28820 abort
     7898918 packets output, 2065075360 bytes, 0 underruns
     0 output errors, 0 collisions, 34 interface resets
     0 output buffer failures, 0 output buffers swapped out
     15 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

[Richard Burts]

Kevin

Thanks for doing this testing for me. The results are interesting and a bit puzzling. When you ping your own address the packet counters should go up twice as much as when you ping the remote address. But the counters here do not do that. The packet counters go up more when you ping the remote than when you ping your own address. And when you ping your own address 5000 times there should be 10000 input packets and the input counters only went up 7940.

Could I get you to repeat the test and this time get the output of show interface for both of the T1 interfaces?

HTH

Rick

[edit] If there were NetFlow data available it might help. I am not sure whether a router running that version of code has support for NetFlow. Can you tell me whether that router supports NetFlow?

[kgreene]

Hi:

I do not think it does Netflow. I think the 2621 XM does but the older 2621.

I will rerun the tests... This is the other serial interface.

I will leave for home, My wife just called...

Kevin

Serial 0/0

cisco2621#show interfaces serial 0/0
Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 206 Internet connection (PPP)
  Internet address is 131.103.116.66/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 12/255, rxload 96/255
  Encapsulation PPP, LCP Open
  Open: IPCP, loopback not set
  Last input 00:00:04, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:02:10
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 584000 bits/sec, 128 packets/sec
  30 second output rate 76000 bits/sec, 108 packets/sec
     15144 packets input, 8555346 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     12706 packets output, 1161554 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#ping 131.103.116.65 repeat 5000

Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 131.103.116.65, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

(edited...)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (5000/5000), round-trip min/avg/max = 4/9/220 ms
cisco2621#show interfaces serial 0/0
Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 206 Internet connection (PPP)
  Internet address is 131.103.116.66/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 5/255, rxload 77/255
  Encapsulation PPP, LCP Open
  Open: IPCP, loopback not set
  Last input 00:00:15, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:03:36
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 469000 bits/sec, 72 packets/sec
  30 second output rate 34000 bits/sec, 54 packets/sec
     23287 packets input, 13587805 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     19368 packets output, 1756526 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#

cisco2621#clear counters serial 0/0
Clear "show interface" counters on this interface [confirm]
cisco2621#show interfaces serial 0/0
Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 206 Internet connection (PPP)
  Internet address is 131.103.116.66/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 3/255, rxload 67/255
  Encapsulation PPP, LCP Open
  Open: IPCP, loopback not set
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:00:03
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 407000 bits/sec, 53 packets/sec
  30 second output rate 22000 bits/sec, 37 packets/sec
     164 packets input, 240902 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     85 packets output, 3801 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#ping 131.103.116.66 repeat 5000

Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 131.103.116.66, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(edited for ease of reading)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (5000/5000), round-trip min/avg/max = 12/15/40 ms
cisco2621#show interfaces serial 0/0
Serial0/0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: 206 Internet connection (PPP)
  Internet address is 131.103.116.66/30
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 15/255, rxload 77/255
  Encapsulation PPP, LCP Open
  Open: IPCP, loopback not set
  Last input 00:00:06, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:01:37
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  30 second input rate 469000 bits/sec, 142 packets/sec
  30 second output rate 96000 bits/sec, 125 packets/sec
     13073 packets input, 5435841 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     11634 packets output, 1113674 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

cisco2621#

[Richard Burts]

Kevin

Thank you for running these tests that I asked for. They are not showing anything that points toward a problem. As I go through these results and as I re-read the posts in this thread I think that I perhaps did not understand the problem correctly. I assumed that it was some problem on the local subnet and that the ping attempt was coming primarily from the connected router. I now believe that those assumptions were faulty - my apologies.

If the problem has to do with remote sources of the ping then I go back to the suggestion made by Chris. debug ip icmp would be an excellent test. First make sure that the monitor logging level is set to debug (which is level 7) and then do terminal monitor (or if you are connected on the router console then make sure that logging console is set to debug). Then turn on debug ip icmp. If a ping request gets to your router it should then generate debug output.

HTH

Rick

[kgreene]

Hi:

I will run the tests suggested by Chris.

The problem - to restate is:

Cogent monitors their customers by pinging the serial WAN interface on a periodic basis (every 5 minutes as I recall). If the ping fails after several ping tests in a row, they will call the customer within a few minutes to see if there is some sort of power failure or perhaps an equipment failure. Normally they are very good about this.

The problem now is that from an external location somewhere in the Gogent network, their monitoring platform can not ping the serial interface from the outside. It is possible to ping one of the hosts on the inside LAN. The public /24 and the smaller /27 block can be pinged from the outside. The issue is that the two WAN IP addresses - those that face the external world, can not be pinged. If you want to ping an internal host, try 199.4.110.11 (the name server). I should note that a traceroute to an internal host starts but never gets anywhere near the WAN. (See below). It can however be pinged:

hayward>ping 199.4.110.11

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 199.4.110.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/21/24 ms
hayward>

The router is in production use and seems to work fine otherwise. The two incoming ACL's are active and working (and on the right interface).

I will run the test next.

Thanks for all the help so far.

Kevin

hayward>traceroute 199.4.110.11

Type escape sequence to abort.
Tracing the route to ns1.cellmail.com (199.4.110.11)

  1 adsl-68-127-30-198.dsl.pltn13.pacbell.net (68.127.30.198) 4 msec 0 msec 0 msec
  2 192.0.2.100 12 msec 16 msec 12 msec
  3 64.164.107.130 12 msec 12 msec 12 msec
  4 151.164.93.229 12 msec 12 msec 16 msec
  5 151.164.101.122 44 msec 32 msec 52 msec
  6 po5-3.core01.sjc04.atlas.cogentco.com (154.54.13.97) 12 msec 16 msec 16 msec
  7 te3-2.mpd01.sjc04.atlas.cogentco.com (66.28.4.49) 16 msec *  *
  8 te0-0-0-6.mpd22.sfo01.atlas.cogentco.com (154.54.28.81) 16 msec
    te0-0-0-6.mpd21.sfo01.atlas.cogentco.com (154.54.2.165) 16 msec
    te0-0-0-6.mpd22.sfo01.atlas.cogentco.com (154.54.28.81) 12 msec
  9 te7-3.mpd01.sfo01.atlas.cogentco.com (154.54.3.222) 16 msec 12 msec 12 msec
10 gi0-1.ca01.sfo01.atlas.cogentco.com (154.54.5.90) 12 msec 16 msec 12 msec
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *

[kgreene]

Hello:

From the debug information, I got the following information.

I was pinging the 2nd T-1 from my remote office (the .193 address).

ICMP: echo reply sent, src 131.103.116.66, dst 68.127.30.193
Sep 14 01:59:26: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 30 packets
ICMP: echo reply sent, src 131.103.116.66, dst 68.127.30.193
ICMP: echo reply sent, src 131.103.116.66, dst 68.127.30.193
ICMP: echo reply sent, src 131.103.116.66, dst 68.127.30.193

hayward>ping 140.174.40.90

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 140.174.40.90, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
hayward>

Then in the problem router...

84.72(123) (Serial0/1 ) -> 199.4.110.40(123), 1 packet
ICMP: echo reply sent, src 140.174.40.90, dst 68.127.30.193
ICMP: echo reply sent, src 140.174.40.90, dst 68.127.30.193
ICMP: echo reply sent, src 140.174.40.90, dst 68.127.30.193
ICMP: echo reply sent, src 140.174.40.90, dst 68.127.30.193

So it looks like the reply is being sent OK...

Ideas??? Somewhere the packets are dropped.

Thanks.

Kevin

[gatlin007]

Kevin,

First off you are a good man.  We should all pay attention when our wives call.

If you execute a 'show ip route 68.127.30.193' from the troublesome router is the next hop over the interface you expect it to be?  Is the gateway for this route the optimum path?

From the ACL's you posted it doesn't look like this traffic would be dropped.  Could it be that this traffic is crossing an interface that may have security policy that conflicts with it?  Or perhaps it's matching a null route?


Chris