facing ping drop and traceroute issues on specific IP,s of Avaya Servers

Muhammad Nasir Jalil · ‎02-09-2017

Hi Expert Community,

i am facing very strange issue from last 3 to 4 days and this issue is related to Avaya servers and due to this facing a lot of voice quality issues sometimes empty voice calls and sometimes during voice call the agents are not able to hear customers for a while and than voice stream comes back

when i try to ping and traceroute 1 or 2 avaya servers IP from Access switches (for rest all the servers ping and traceroute is fine) i am facing issues and ping and traceroute output is listed below

AVAYA-SW#ping 192.168.20.56 repeat 20

Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to 192.168.20.56, timeout is 2 seconds:
!!!!!.!!.!!.!!.!!.!!
Success rate is 75 percent (15/20), round-trip min/avg/max = 1/2/17 ms

AVAYA-SW#traceroute 192.168.20.56

Type escape sequence to abort.
Tracing the route to 192.168.20.56

1 192.168.1.2 0 msec 0 msec 0 msec (( Core-sw ip)
2 * * *
3 * * *
4 * * *

and with the same switch all other servers are working fine and the output of ping and traceroute are listed below

AVAYA-SW#ping 192.168.20.24 repeat 20

Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to 192.168.2.24, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (20/20), round-trip min/avg/max = 1/2/9 ms

AVAYA-SW#traceroute 192.168.20.24

Type escape sequence to abort.
Tracing the route to 192.168.20.24

1 192.168.1.2 0 msec 0 msec 0 msec (((Core SW IP)
2 192.168.20.24 0 msec 0 msec 0 msec

these Access SW,s are connected to Core directly and now issue is this that why i am facing the ping drop and traceroute issues on one or two specific servers and rest all servers (almost more than 10 to 15 ) are working fine i checked STP is working fine no loop etc and when we ping this IP 20.56 from PC its pinging fine

i need you expert suggestions about this that how can we overcome this issue? this issue is related to avaya servers( which in my opinion is yes) or there is something else we can troubleshoot in our network

thanks in advance

Mark Malone · ‎02-09-2017

If your on the switch the Avaya servers are directly connected to an you ping them do you see drops ? what do the interfaces where the servers connect show any errrors crcs/inputs etc ?

Are any of the interfaces your tracing over under congestion at all or are you going over the same path to the working servers ?

Muhammad Nasir Jalil · ‎02-09-2017

Hi Mark,

thanks for your reply below is the output of the interface on which Avaya server is connected and not facing any kind of congestion and also going to the same path and same switch with the working servers

AVAYA-SW#sho interfaces gigabitEthernet 0/1
GigabitEthernet0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 1005.ca7c.0101 (bia 1005.ca7c.0101)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4000 bits/sec, 2 packets/sec
5 minute output rate 9000 bits/sec, 6 packets/sec
113943 packets input, 23256786 bytes, 0 no buffer
Received 1772 broadcasts (6 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 6 multicast, 0 pause input
0 input packets with dribble condition detected
368128 packets output, 44830275 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Mark Malone · ‎02-09-2017

Ok that's good its clean and if your on that switch directly AVAYA-SW and pinging the server on G0/1 are you dropping packets ? just wondering if its is the nic causing it on the server

Aswell on the CUCM for Cisco there is a design featire that prevents you constantly pinging it , its a security feature to stop you getting DDOS on your voice system , I wonder if AVAYA have that too ? , basically the CUCM stops every 5th or 6th ping to break the flow but it doesn't do anything to actual prod traffic but it makes it look like pings are dropping

https://supportforums.cisco.com/discussion/11177711/dropping-6th-icmp-packets

Muhammad Nasir Jalil · ‎02-09-2017

but there are almost more than 20 servers of Avaya and all are pinging and tracerouting fine only issues with 2 or 3 servers

Mark Malone · ‎02-09-2017

Ok well that rules out the security settings then like Cisco call managers , if you ping directly on the switch the effected servers are connected to are you dropping ?

Muhammad Nasir Jalil · ‎02-09-2017

yes i am pinging from the switch on which servers are directly connected and ping out put is here

AVAYA-SW#ping 192.168.20.56 repeat 20

Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to 192.168.20.56, timeout is 2 seconds:
!!!!!.!!.!!.!!.!!.!!
Success rate is 75 percent (15/20), round-trip min/avg/max = 1/2/9 ms
AVAYA-SW#ping 192.168.20.24 repeat 20

Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to 192.168.20.24, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (20/20), round-trip min/avg/max = 1/2/17 ms
AVAYA-SW#

Mark Malone · ‎02-09-2017

Ok cool then that's the server nic I agree its AVAYA and not the switches , your not traversing any other network equipment , the port is clean to indicate cable and connections are good but when you ping directly connected you still drop packets and its only the case on these certain few servers , everything else is working fine which again confirms its not the network or switch

My next step if it was my servers I would ask AVAYA to check the server NIC make sure its running same firmware etc as the working servers and if possible either upgrade the firmware or replace a nic as a test on one of the effected servers see if it resolves it

I don't think there is anything you can do to fix it from your end