Problems with EX90 and MX300 devices dropping registration

Steve Coady · ‎07-24-2013

Hello

My company purhcased this TeleP solution. For the most part, it has worked fine. However, I have had several devices lose registration.

These devices were setup and workign well.

There have been no network configuration changes.

TAC case support wants me to console into the device, however the console cable is a special kind that I have been advised I need to make myself.

I did receive one of these cables from TAC and was advise d to download software. I did, but had no success. I "googled" this

software and found ALOT of people had problems with this console cable and software.

Troubleshooting steps I have taken are:

I Have Restarted Device, no success

I Have Power Cycled Device, no success

I Have Replaced Ethernet Cable, no success

I moved device To New Lan Jacks And Switch Ports, no success

I Reset And Tried To Re-Provision, no success

So far, my only option has been to RMA the devices that stop registering

Questions I have are:

What can I do to resolve this special console cable issue. I belive I can issue command to correct CODEC, once I gain console access

Are there any other troubleshooting steps I should try?

sMc

Bryan Deaver · ‎07-24-2013

Hey Steve,

Normally for registration issues I would not recommend nor see the need for accessing the console.

You can find some information on the console cable that was likely sent to you and the drivers at this link:

https://supportforums.cisco.com/docs/DOC-27936

The cable will allow you access the codec even when the codec is not accessible on the network. For example, in some instances where the codec will not boot up properly, the console cable can help to understand why and in many cases recover the codec.

This Cisco Live presentation covered some of the use and access for the console cable:

https://supportforums.cisco.com/docs/DOC-29766

Now, if the device is losing registration because it is crashing, then possible the console cable would be good but normally the system should come back up and in that case you would want to work with the tac to review the historical log files to determine why the codec crashes. In later release of the software, the codec may go into "maintenance mode" when it crashes which provides choices of what you want to do to collect information and recover the device. Yet another reference link for you for maintenance mode:

https://supportforums.cisco.com/docs/DOC-34116

Normally for registration issues, you would want to look more on the protocol being used. For example SIP and determine when and why the device is losing registration. This may entail collecting more information such as SIP packet traces from the codec. Check this troubleshooting guide and specifically look at the section on enabling SIP/H323 traces:

http://www.cisco.com/en/US/docs/telepresence/endpoint/codec-c-series/tc6/troubleshooting_guide/tc_troubleshooting_guide_tc60.pdf

It is unclear to me how RMAing the codec would relate to issues with it not being able to register unless there was something going on with the lower layers such as the NIC failing on the device. Maybe you can unicast me the tac case number where the troubleshooting of a registration problem led to an RMA to get better context of this issue.

Hope this helps guide you to resolution of your issues.

Bryan

Vernon Depee · ‎07-24-2013

Steve,

Have you tried to access these devices via SSH? If they lose registration, but still have network connection, SSH should still be available if enabled.

When the devices unregister, do they still respond to pings?

Steve Coady · ‎07-24-2013

Bryan

Thanks for the response and the links.

The RMA ghas been my only option because TAC could not help unless they had a log. The only way to get the log was via consoie cable.

Vernon

Thanks for the response. The devices cannot be pinged and are not available to ssh.

Again, there was no network change attributable to this. No configs were changed. Other devices on the same LAN still work fine!

sMc

Steve Coady · ‎07-24-2013

aded info:

The devices are attached to an Cisco 3750 switch with routing enabled.

sMc

Bryan Deaver · ‎07-24-2013

Does show cdp neighbor detail from the 3750 show information about what the IP address on the codec thinks it is? What does the touch panel on show you?

Yes, console access in this case could be helpful if there is no other access to the device. I would guess that something changed possibly on the codec with regards to the vlan it is on but again if you are running later versions of TC software we should have codec ip address infomation which is sent back to the 3750. Again, what does the touch show you when in this state as well as what do you observe on the EX90 screen?

If are you at the point in an issue where you are going to RMA the device, as a last step I would recommend that you try a factory reset; on the EX90 it is tricky but doable from the power button method. Again check the troubleshooting guide reference. I normally don't recommend a facory reset as a troubleshoot step (since for one it wipes out good diagnostic information from the device) but again if you are going to be doing an RMA anyway, then try it.

In what briefly I am hearing from you though, there should not be a need for an RMA here if this is an issue that occurs more than once.

Steve Coady · ‎07-24-2013

Bryan

Both the switch and the Touch screen show the same ip info.

Show cdp ne det

Device ID: SEP00506008580A

Entry address(es):

IP address: 10.x.x.30

Platform: CTS-CODEC-MX200/MX300, Capabilities: Host Phone

Interface: GigabitEthernet6/0/24, Port ID (outgoing port): eth0

Holdtime : 170 sec

Version :

TC6.0.1 65adebe

advertisement version: 2

Management address(es):

PP3750S1#ping 10.x.x.30

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.x.x.30, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

The Vlan is /24

sMc

Bryan Deaver · ‎07-24-2013

Good that something is working; we see CDP packets coming in. The codec is using DHCP and does get this address from the server? This is not using static address, correct?

The address and vlan information you see from the touch panel settings menu reflects the reality of what is configured on the switch side?

As you said, restarting the EX90 from the touch panel does not cause it to come back up, yes?

There are a few things that can be checked from the codec via console connection. Would be good to get the output of xstatus, xconfig, the /var/log/console file as well as ifconfig and ethtool -S eth0.

Nothing off on the switch side when the codec is in this state from the show interface stats? Are you using port security on the switch?

The same issue has happened on more than one of the codecs then? Do you have other codecs besides EX/MX systems?

Steve Coady · ‎07-24-2013

Bryan

The device does have a static ip.

The switch interface is configured as follows:

interface GigabitEthernet6/0/24

description VIDEO MX300 LrgConf Alternate

switchport access vlan xxx

switchport mode access

srr-queue bandwidth share 10 10 60 20

queue-set 2

priority-queue out

mls qos trust dscp

auto qos voip trust

spanning-tree portfast

end

Retarting from Touch panel does not fix the issue

No port security

sMc

Bryan Deaver · ‎07-25-2013

Again, since we are showing CDP information on the switch, this means that the CDP multicast packets are coming in to the switch.

No guarantee that the codec is receiving packets though. If you do a 'show arp | include ' from the switch does it show that it was able to resolve the address?

Again, nothing in the show interface for that switch port that looks abnormal?

Thinking at this point what a packet capture while spanning that port would show. But before going doing that path, again getting access to the codec while in this state (again through the console port) will be a good step if there is nothing else that we can determine through the touch panel nor the switch side. At that point though, I am going to recommend that you take the Q/A from this thread and open up a TAC case to have a CSE interactively troubleshoot this with you.

Steve Coady · ‎07-25-2013

If you do a 'show arp | include ' from the switch does it show that it was able to resolve the address?

router#sh ip arp | inc 10.x.x.30

router#sh mac add add 0050.6008.580a

Mac Address Table

-------------------------------------------

Vlan Mac Address Type Ports

---- ----------- -------- -----

router#

switch1# sh mac add int gi6/0/24

Mac Address Table

-------------------------------------------

Vlan Mac Address Type Ports

---- ----------- -------- -----

x 0050.6008.580a DYNAMIC Gi6/0/24

Total Mac Addresses for this criterion: 1

switch1#

I am including this to show that ALL other devices on same vlan, several on same switch, others on different switches ARE being resolved and are working fine.

router#sh ip arp | inc 10.x.x

Internet 10.x.x.15 233 d867.d971.9113 ARPA Vlanx

Internet 10.x.x.12 2 d867.d971.9138 ARPA Vlanxxx

Internet 10.x.x.13 233 d867.d971.913c ARPA Vlanxxx

Internet 10.x.x.10 228 d867.d971.9446 ARPA Vlanxxx

Internet 10.x.x.11 101 d867.d971.9428 ARPA Vlanxxx

Internet 10.x.x.6 - 3cdf.1e8a.9f4c ARPA Vlanxxx

Internet 10.x.x.7 231 3cdf.1efc.9e42 ARPA Vlanxxx

Internet 10.x.x.1 161 3037.a6ab.bdd1 ARPA Vlanxxx

Internet 10.x.x.28 233 0050.6008.5803 ARPA Vlanxxx

Internet 10.x.x.29 62 0050.6008.556a ARPA Vlanxxx

Internet 10.x.x.24 57 0050.6006.3c47 ARPA Vlanxxx

Internet 10.x.x.23 230 d867.d971.988a ARPA Vlanxxx

Internet 10.x.x.20 26 0050.6008.5837 ARPA Vlanxxx

Internet 10.x.x.21 233 d867.d971.98ac ARPA Vlanxxx

router#

Again, nothing in the show interface for that switch port that looks abnormal?

GigabitEthernet6/0/24 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is dc7b.940c.0498 (bia dc7b.940c.0498)

Description: VIDEO MX300 LrgConf Alternate

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:17, output hang never

Last clearing of "show interface" counters 14w1d

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 1000 bits/sec, 1 packets/sec

125852 packets input, 13244462 bytes, 0 no buffer

Received 125852 broadcasts (61410 multicasts)

3 runts, 0 giants, 0 throttles

3 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 61410 multicast, 0 pause input

0 input packets with dribble condition detected

2855367 packets output, 230517120 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

There are (3) runts and input errors in last 14 weeks.

I do appreciate you assistance.

sMc

Bryan Deaver · ‎07-25-2013

OK, so a guess here is that the codec is not receiving the frames from the switchport. Don't see it in the arp table; and I would assume that if you ping the address and then check the arp table you would see an incomplete entry for the ip address.

One off case, I could see issues possibly with the nic on the codec. Happening more than once for you seems odd to me. We don't currently have a way on the touch panel to see the ethernet stats for the codec so to dig further, it would be some of the information I had indicated before which once in this state would be the console connection.

How often have you see this issue with the same symptoms?

Again, since you seem to have one in this state currently, I would recommend getting a case open on this so we can track it. There is the factory reset option from the touch panel (again that would be my last resort option) but from what you seem to be seeing, I don't hold high hopes that it will make a difference. You can ping my privately with the case number so I can track what happens on it.

Steve Coady · ‎07-25-2013

router #ping 10.x.x.30

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.x.x.30, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

router#sh ip arp | inc 10.x.x.30

Internet 10.x.x.30 0 Incomplete ARPA

How often have you see this issue with the same symptoms?

We implemented this solution last summer/fall. Since that time (4) devices have had this same problem.

Several at my local site and 1 in Chicago site.

A couple devices experienced this within a couple weeks of eachother.

I have already done a factory rest on the device in questuion, no success

sMc