ICMP "TTL Expirate in Transmit" does not work

tsvetomir_gospodinov · ‎02-29-2012

The router is Cisco 7606 (RSP-720-3CXL-10GE is the supervisor).

The router does not return TTL Message. (traceroute command behind this router does not work).

Tracing route to abv.bg [194.153.145.104]

over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 95.87.0.49

2 * * * Request timed out.

3 5 ms 2 ms 3 ms 145.128.158.95-rev.novatelbg.net [95.158.128.145]

4 5 ms 5 ms 11 ms 145.236.236.64

5 14 ms 5 ms 7 ms novatel.bix.bg [193.169.198.109]

6 5 ms 4 ms 5 ms netinfo.bix.bg [193.169.198.168]

7 5 ms 4 ms 4 ms abv.bg [194.153.145.104]

Trace complete.

nkarpysh · ‎02-29-2012

Hello,

Not clear what is the problem. From trace I see that it is reaching final destination. Why do you expect TTL Expire message to arrive?

Nik

HTH,
Niko

tsvetomir_gospodinov · ‎02-29-2012

Ok. Think as traceroute throw the router does not work. ("2 * * * Request timed out.", here should be the router ip address). Ping to router is OK.

nkarpysh · ‎02-29-2012

Ok Got it -- thnaks.

Possibly it sends reply but with some delay - that is why answer considered as timed out. Do you see any delay pinging that router? Can you please try pinging that with time 1 in ping to lower response time expected.

Also do you see that host dropping traceback ICMP continiously or on random tries?

Have you check the CPU load on it? (show proc cpu sort and show proc cpu history)

Nik

HTH,
Niko

tsvetomir_gospodinov · ‎02-29-2012

C:\Documents and Settings\TrakiaCable>ping 95.87.0.1 -t

Pinging 95.87.0.1 with 32 bytes of data:

Reply from 95.87.0.1: bytes=32 time=2ms TTL=254

Reply from 95.87.0.1: bytes=32 time=3ms TTL=254

Reply from 95.87.0.1: bytes=32 time=5ms TTL=254

Reply from 95.87.0.1: bytes=32 time=6ms TTL=254

Reply from 95.87.0.1: bytes=32 time=9ms TTL=254

Reply from 95.87.0.1: bytes=32 time<1ms TTL=254

Reply from 95.87.0.1: bytes=32 time=1ms TTL=254

Reply from 95.87.0.1: bytes=32 time=2ms TTL=254

Reply from 95.87.0.1: bytes=32 time=43ms TTL=254

Reply from 95.87.0.1: bytes=32 time=5ms TTL=254

Reply from 95.87.0.1: bytes=32 time=9ms TTL=254

Reply from 95.87.0.1: bytes=32 time=2ms TTL=254

Reply from 95.87.0.1: bytes=32 time=8ms TTL=254

Reply from 95.87.0.1: bytes=32 time=2ms TTL=254

Reply from 95.87.0.1: bytes=32 time=1ms TTL=254

Ping statistics for 95.87.0.1:

Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 43ms, Average = 6ms

---

sz-edge1#sho processes cpu | exclude 0.00%__0.00%__0.00%

CPU utilization for five seconds: 7%/2%; one minute: 9%; five minutes: 9%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

2 100 48057 2 0.07% 0.01% 0.00% 0 Load Meter

3 268 861 311 0.07% 0.09% 0.06% 1 SSH Process

7 1555360 91579 16983 0.00% 1.38% 1.10% 0 Check heaps

12 97516 1003195 97 0.00% 0.03% 0.04% 0 ARP Input

13 208 251720 0 0.00% 0.01% 0.00% 0 ARP Background

26 24 234349 0 0.07% 0.00% 0.00% 0 IPC Deferred Por

28 34200 744806 45 0.00% 0.01% 0.00% 0 IPC Seat Manager

61 2476 240211 10 0.00% 0.01% 0.00% 0 TTY Background

137 84 240099 0 0.07% 0.00% 0.00% 0 Heartbeat Proces

161 65504 181319 361 0.00% 0.02% 0.00% 0 BGP Router

164 50724 8275 6129 0.00% 0.01% 0.00% 0 Per-minute Jobs

165 528 240352 2 0.00% 0.03% 0.02% 0 Per-Second Jobs

196 364 7436175 0 0.00% 0.02% 0.00% 0 ACE Tunnel Task

232 320 7435337 0 0.00% 0.01% 0.00% 0 IP ARP Retry Age

233 7315316 129181230 56 3.03% 2.96% 2.96% 0 IP Input

240 1117480 6981 160074 0.00% 0.76% 0.85% 0 BGP Scanner

270 712 30294279 0 0.15% 0.11% 0.10% 0 Ethernet Msec Ti

274 396 7435969 0 0.00% 0.01% 0.00% 0 IPAM Manager

337 996 64081 15 0.00% 0.01% 0.00% 0 CDP Protocol

341 194864 3642995 53 0.23% 0.19% 0.17% 0 ADJ resolve proc

347 552 474885 1 0.07% 0.01% 0.00% 0 TCP Timer

348 5584 23695 235 0.00% 0.01% 0.00% 0 TCP Protocols

397 180 342205 0 0.07% 0.00% 0.00% 0 FM core

412 97160 120142 808 0.07% 0.06% 0.05% 0 HIDDEN VLAN Proc

528 512 45394 11 0.00% 0.02% 0.00% 0 RIP Timers

531 543420 6810671 79 0.00% 0.19% 0.22% 0 Port manager per

591 324292 6812868 47 0.23% 0.15% 0.15% 0 IGMP Input

592 880 4786899 0 0.00% 0.04% 0.05% 0 PIM Process

593 756 2396940 0 0.07% 0.02% 0.00% 0 Mwheel Process

594 8668 1220566 7 0.07% 0.01% 0.00% 0 MRIB Trans

597 752 11934184 0 0.07% 0.05% 0.05% 0 MFIB_mrib_write

598 13056 340348 38 0.07% 0.00% 0.00% 0 MFIB_mrib_read

607 34548 492738 70 0.00% 0.01% 0.00% 0 SNMP ENGINE

----

DF bit unreachables All other unreachables

Interval (millisecond) 1 1

Interface # DF bit unreachables # All other unreachables

--------- --------------------- ------------------------

GigabitEthernet1/43 0 0

GigabitEthernet5/3 0 0

Vlan304 0 5

Vlan1001 0 2120

Vlan1402 0 132352

Vlan1500 0 0

Vlan1501 0 0

Vlan2402 0 927

Vlan2414 0 14

Vlan3021 0 0

nkarpysh · ‎02-29-2012

Ok,

Just one more test. Can you try to do the traceroutes and on the router sz-edge1 where they fail to be sent back pls do

"show ip traffic | i time exceeded" - those are counters for expired TTL icmp sent. Want to check if those grow or stay stale.

One more thing - can you also check config on this router interface which should respond ro these ICMP (it is incoming L3 int). See if there are any "no ip unreachable" or "no ip redirect" configured.

Nik

HTH,
Niko

tsvetomir_gospodinov · ‎02-29-2012

As I see it is sporadic:

Tracing route to 95.87.0.202 over a maximum of 30 hops

1 1 ms <1 ms <1 ms 95.87.0.49

2 * * <1 ms edge-sz.trakiacable.net [95.87.0.1]

3 35 ms 33 ms 33 ms 95.87.0.202

Trace complete.

---

sz-edge1#show ip traffic | i time exceeded

0 info reply, 6888 time exceeded, 0 parameter problem

---

interface GigabitEthernet1/1

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/2

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/3

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/4

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/5

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/6

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/7

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/8

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

shutdown

!

interface GigabitEthernet1/9

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/10

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/11

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/12

switchport

switchport access vlan 1001

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet1/13

no ip address

shutdown

!

interface GigabitEthernet1/14

no ip address

shutdown

!

interface GigabitEthernet1/15

no ip address

shutdown

!

interface GigabitEthernet1/16

no ip address

shutdown

!

interface GigabitEthernet1/17

no ip address

shutdown

!

interface GigabitEthernet1/18

no ip address

shutdown

!

interface GigabitEthernet1/19

no ip address

shutdown

!

interface GigabitEthernet1/20

no ip address

shutdown

!

interface GigabitEthernet1/21

no ip address

shutdown

!

interface GigabitEthernet1/22

no ip address

shutdown

!

interface GigabitEthernet1/23

no ip address

shutdown

!

interface GigabitEthernet1/24

no ip address

shutdown

!

interface GigabitEthernet1/25

no ip address

shutdown

!

interface GigabitEthernet1/26

no ip address

shutdown

!

interface GigabitEthernet1/27

no ip address

shutdown

!

interface GigabitEthernet1/28

no ip address

shutdown

!

interface GigabitEthernet1/29

no ip address

shutdown

!

interface GigabitEthernet1/30

no ip address

shutdown

!

interface GigabitEthernet1/31

no ip address

shutdown

!

interface GigabitEthernet1/32

no ip address

shutdown

!

interface GigabitEthernet1/33

no ip address

shutdown

!

interface GigabitEthernet1/34

no ip address

shutdown

!

interface GigabitEthernet1/35

no ip address

shutdown

!

interface GigabitEthernet1/36

no ip address

shutdown

!

interface GigabitEthernet1/37

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 2414

switchport mode trunk

speed 1000

!

interface GigabitEthernet1/38

no ip address

shutdown

!

interface GigabitEthernet1/39

no ip address

shutdown

!

interface GigabitEthernet1/40

no ip address

shutdown

!

interface GigabitEthernet1/41

no ip address

shutdown

!

interface GigabitEthernet1/42

no ip address

shutdown

!

interface GigabitEthernet1/43

ip address x.x.x.x y.y.y.y

ip pim dense-mode

!

interface GigabitEthernet1/44

no ip address

ip pim dense-mode

!

interface GigabitEthernet1/45

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1501

switchport mode trunk

switchport nonegotiate

!

interface GigabitEthernet1/46

switchport

switchport access vlan 1500

switchport trunk encapsulation dot1q

switchport trunk native vlan 1500

switchport trunk allowed vlan 1500,2896

switchport mode trunk

switchport nonegotiate

!

interface GigabitEthernet1/47

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 3021

switchport mode trunk

switchport nonegotiate

!

interface GigabitEthernet1/48

switchport

switchport access vlan 1500

switchport mode access

switchport nonegotiate

!

interface GigabitEthernet5/1

ip address x.x.x.x y.y.y.y

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 304

switchport mode trunk

!

interface GigabitEthernet5/2

no ip address

shutdown

!

interface GigabitEthernet5/3

ip address x.x.x.x y.y.y.y

ip pim dense-mode

!

interface TenGigabitEthernet5/4

description Novatel

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1402,2402,2896

switchport mode trunk

!

interface TenGigabitEthernet5/5

no ip address

shutdown

!

interface Vlan1

no ip address

!

interface Vlan101

no ip address

shutdown

!

interface Vlan304

ip address x.x.x.x y.y.y.y

!

interface Vlan1001

ip address x.x.x.x y.y.y.y

!

interface Vlan1402

ip address x.x.x.x y.y.y.y

!

interface Vlan1500

ip address x.x.x.x y.y.y.y

ip pim dense-mode

!

interface Vlan1501

ip unnumbered Vlan1500

ip pim sparse-mode

ip igmp static-group z.z.z.z

ip igmp access-group BBCMulticast

!

interface Vlan2402

ip address x.x.x.x y.y.y.y

!

interface Vlan2414

ip address x.x.x.x y.y.y.y

!

interface Vlan2896

no ip address

no ip igmp snooping

!

interface Vlan3021

ip address x.x.x.x y.y.y.y

ip pim dense-mode

ip igmp join-group z.z.z.z

ip igmp query-max-response-time 5

ip igmp query-interval 10

no ip igmp snooping

!

nkarpysh · ‎02-29-2012

SO basically this router is sending TTL expired packets:

sz-edge1#show ip traffic | i time exceeded

0 info reply, 6888 time exceeded, 0 parameter problem

Did you try to do multiple traces through this router to see if that umber is growing?

It seems that due to some internal CPU realization the packet is just got delaid and initiator not receiving it in time count that as drop. To check it in more detail we may need to setup SPAN on ingress port to see the time when trace ICMP is coming to this port and when ICMP reply for TTL expired is sent to consider it further.

Nik

HTH,
Niko

tsvetomir_gospodinov · ‎03-05-2012

I found this while examining cef:

sz-edge1#sh cef interface vlan 2402

Vlan2402 is up (if_number 84)

Corresponding hwidb fast_if_number 84

Corresponding hwidb firstsw->if_number 84

Internet address is x.x.x.x/30

ICMP redirects are never sent

IP unicast RPF check is disabled

Output features: HW Shortcut Installation

IP policy routing is disabled

BGP based policy accounting on input is disabled

BGP based policy accounting on output is disabled

Hardware idb is Vlan2402

Fast switching type 28, interface type 147

IP CEF switching enabled

IP CEF switching turbo vector

IP Null turbo vector

IP prefix lookup IPv4 mtrie generic

Input fast flags 0x0, Output fast flags 0x0

ifindex 83(83)

Slot unknown (255) Slot unit 2402 VC -1

IP MTU 1500

Is it relevant to this problem ?