E20 hang with Sip Config - TE4.1.1 - Strange

[RAMEEZ RAHIM]

We see that few of our E20's - most of them on internet hangs as soon as the endpoint is booted.

If the endpoint is booted without network cable, then the E20 is stable.

If the SIP configuration is removed, the E20 is stable.

As soon as we put in the SIP config and the E20 is registering over SIP, the device hangs.

We can see from root that the Tandberg application to fail - tsh from root cli says "unable to connect application"

Is this a known issue with TE4.1.1.

If we disable SIP and use only H323 - device is stable.

VCS - X7.2.1

[Danny De Ridder]

Hard to tell what's going on without any logfiles. Please attach historical logfiles. Your E20 seems to have crashed.

[Danny De Ridder]

The logfiles can be retrieved using a WEB browser pointed to your E20.

[RAMEEZ RAHIM]

Attaching Historical logs

[Danny De Ridder · Accepted solution]

Hello,

the logfiles you sent from the E20 show an issue with the Marvell chip which is an ethernet controller inside the E20.

Feb 26 07:59:02 (none) main: FPGA programmed OK

Feb 26 07:59:02 (none) main: marvell.c:  ioctl(9, -2146669310, ...)==-1, errno==14, Bad address

Feb 26 07:59:02 (none) main: platform/marvell/marvell.c:132: marvell_ioctl: Assertion `0 && "Marvell ioctl Failure"' failed.

Feb 26 07:59:02 (none) main: Received signal SIGABRT (6) in thread 0x4082a4c0, TID 1704

Feb 26 07:59:02 (none) main: Registers:

Feb 26 07:59:02 (none) main: R0:     00000000 R1:   000006a8 R2:    00000006 R3:   000006a8

Feb 26 07:59:02 (none) main: R4:     00000006 R5:   40826bdc R6:    40826000 R7:   0000010c

Feb 26 07:59:02 (none) main: R8:     00000bdc R9:   00000000 R10:   4082a4c0 FP:   be976974

Feb 26 07:59:02 (none) main: PC:     407206f8 IP:   be976978 SP:    be97695c LR:   407206c4

Feb 26 07:59:02 (none) main: ERR:    00000000 CPSR: 20000010 FAULT: 00000000 TRAP: 00000000

Feb 26 07:59:02 (none) main: OLDMSK: 00000000

Our defect tracking system is down, so could not find a matching ddts for this, but to me, this looks like hardware.

Will verify with engineering.

The recent change you talk about can cause E20s to crash. There's a defect open on this :

CSCue59199 - Boot: SIPAUTH error: unable to clear signature (gState=1)

When we get challenged on a SIP presence subscribe message, the E20 crashes. When that happens you will see a message like "Authorization to SUBSCRIBE got proxy-challenged in 407 SUBSCRIBE".

Now the historical logfiles you sent for the specific E20 does not match that ddts. Do you have other E20s crashing when you enable SIP? If so, can you send the historical logfiles of such a device which now no longer crashes with SIP disabled?

Below is the sequence prior to the crash.

Could you collect some SIP debugs and try enable SIP on 1 E20 please?

From the tsh do :

log ctx sippacket debug 9

When the unit crashes, disable SIP again and forward the logfiles to verify.

Mar  3 11:43:43 (none) main: User admin(1001) successfully executed  command '/Presence/Subscribe/Start URI: 91000001' from  sweet-brew-7.cisco.com.

Mar  3 11:43:43 (none) main: 8241.76 SipStack I: SipEv: Active Subscribe to '

sip:91000001@ciscolab.com

' of type 'presence', unsolicited=0

Mar  3 11:43:43 (none) main: 8241.76 SipStack I: SipUa GRUU added OK

Mar  3 11:43:43 (none) main: 8241.77 SipPacket   PacketDump: Proto: SIP, Name: SUBSCRIBE

sip:91000001@ciscolab.com

SIP/2.0, Direction: Outgoing, RemoteAddress: 10.106.93.69:5061, GroupEntity:

6f22769f93245019@171.69.87.88

, Time: 8241765, Content: !!!<

Mar  3 11:43:43 (none) main: 8241.77 SipPacket   SUBSCRIBE

sip:91000001@ciscolab.com

SIP/2.0

Mar  3 11:43:43 (none) main: 8241.77 SipPacket   Via: SIP/2.0/TLS  171.69.87.88:5061;branch=z9hG4bKb13dc154a422586a9cb016c9d7ac0a3f.1;rport

Mar  3 11:43:43 (none) main: 8241.77 SipPacket   Call-ID:

6f22769f93245019@171.69.87.88

Mar  3 11:43:43 (none) main: 8241.77 SipPacket   CSeq: 101 SUBSCRIBE

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   Contact:

<9100123>

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   From:

<9100123>

;tag=92c18ad11538b1bd

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   To:

<91000001>

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   Max-Forwards: 70

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   Route:

<10.106.93.69>

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   User-Agent: TANDBERG/257 (TE4.1.1.271887Beta1)

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   Expires: 3600

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   Event: presence

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   Accept: application/pidf+xml

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   Content-Length: 0

Mar  3 11:43:43 (none) main: 8241.78 SipPacket  

Mar  3 11:43:43 (none) main: 8241.78 SipPacket   >!!!

Mar  3 11:43:43 (none) main: 8242.02 SipPacket   PacketDump: Proto: SIP,  Name: SIP/2.0 407 Proxy Authentication Required, Direction: Incoming,  RemoteAddress: 10.106.93.69:5061, GroupEntity:

6f22769f93245019@171.69.87.88

, Time: 8242018, Content: !!!<

Mar  3 11:43:43 (none) main: 8242.02 SipPacket   SIP/2.0 407 Proxy Authentication Required

Mar  3 11:43:43 (none) main: 8242.02 SipPacket   Via: SIP/2.0/TLS  171.69.87.88:5061;branch=z9hG4bKb13dc154a422586a9cb016c9d7ac0a3f.1;received=171.69.87.88;rport=36924

Mar  3 11:43:43 (none) main: 8242.02 SipPacket   Call-ID:

6f22769f93245019@171.69.87.88

Mar  3 11:43:43 (none) main: 8242.03 SipPacket   CSeq: 101 SUBSCRIBE

Mar  3 11:43:43 (none) main: 8242.03 SipPacket   From:

<9100123>

;tag=92c18ad11538b1bd

Mar  3 11:43:43 (none) main: 8242.03 SipPacket   To:

<91000001>

;tag=aaa367e278a7ece0

Mar  3 11:43:43 (none) main: 8242.03 SipPacket   Server: TANDBERG/4120 (X7.2.1)

Mar  3 11:43:43 (none) main: 8242.03 SipPacket   Proxy-Authenticate:  Digest realm="akgvcsc1.ciscolab.com",  nonce="bb98866b8387ba58270573abceefb75e19dcde22e51b036493413ef0b5cd",  opaque="AQAAAK5Mba8WRqO56xvFJpIWzjCx51zZ", stale=FALSE, algorithm=MD5,  qop="auth"

Mar  3 11:43:43 (none) main: 8242.04 SipPacket   Content-Length: 0

Mar  3 11:43:43 (none) main: 8242.04 SipPacket  

Mar  3 11:43:43 (none) main: 8242.04 SipPacket   >!!!

Mar  3 11:43:43 (none) main: Authorization to SUBSCRIBE got proxy-challenged in 407 SUBSCRIBE

Mar  3 11:43:43 (none) main: Received signal SIGSEGV (11) in thread 0x4084d450, TID 1809

Mar  3 11:43:43 (none) main: Illegal memory access at: 0x8

Mar  3 11:43:43 (none) main: Registers:

[RAMEEZ RAHIM]

Hi Danny,

Yes you are right. We are getting a crash following a SIP Presence Subscribe.

Jan  1 06:15:48 (none) main: Authorization to SUBSCRIBE got proxy-challenged in 407 SUBSCRIBE

Jan  1 06:15:48 (none) main: Received signal SIGSEGV (11) in thread 0x4084d450, TID 1826

Jan  1 06:15:48 (none) main: Illegal memory access at: 0x8

Jan  1 06:15:48 (none) main: Registers:

Jan  1 06:15:48 (none) main: R0:     4192f1f9 R1:   000003c0 R2:    8a2ebf18 R3:   00000000

Jan  1 06:15:48 (none) main: R4:     00000000 R5:   4192ebdc R6:    00000001 R7:   00000078

Jan  1 06:15:48 (none) main: R8:     4192ebd4 R9:   00000000 R10:   00000008 FP:   4084cacc

Jan  1 06:15:48 (none) main: PC:     007cfe30 IP:   000005c8 SP:    4084ca90 LR:   007cfd94

Jan  1 06:15:48 (none) main: ERR:    00000017 CPSR: 20000010 FAULT: 00000008 TRAP: 0000000e

Jan  1 06:15:48 (none) main: OLDMSK: 00000000

1778.69 SipPacket   PacketDump: Proto: SIP, Name: SIP/2.0 407 Proxy Authentication Required, Direction: Incoming, RemoteAddress: 115.112.251.34:5060, GroupEntity: 5212593f6032700f@203.99.192.27, Time: 1778684, Content: !!!<

1778.69 SipPacket   SIP/2.0 407 Proxy Authentication Required

1778.69 SipPacket   Via: SIP/2.0/TCP 203.99.192.27:5060;branch=z9hG4bK8b0a6dba8f2ccee420664579e4ec3240.1;received=203.99.192.27;rport=39352;ingress-zone=DefaultSubZone

1778.69 SipPacket   Call-ID: 5212593f6032700f@203.99.192.27

1778.69 SipPacket   CSeq: 101 SUBSCRIBE

1778.69 SipPacket   From: <>ABC@video.gg.com>;tag=54020edd62302085

1778.69 SipPacket   To: <>609160037@video.gg.com>;tag=260b1bce948ae702

1778.69 SipPacket   Server: TANDBERG/4120 (X7.2.1)

1778.69 SipPacket   Proxy-Authenticate: Digest realm="vcsc.video.gg.com", nonce="d34ecc64f5e7b0abc9067dce57099b1be740b6cd099122e377633655d4b1", opaque="AQAAAET4omQDHsmcwJls30RE53TlDEMt", stale=FALSE, algorithm=MD5, qop="auth"

1778.69 SipPacket   Proxy-Authenticate: NTLM realm="vcsc.video.gg.com", qop="auth", targetname="vcsc.video.gg.com"

1778.69 SipPacket   Content-Length: 0

1778.69 SipPacket  

1778.70 SipPacket   >!!!

[RAMEEZ RAHIM]

The only change we had is enable NTLM auth on VCS-Control. Prevously it was local.

So, my question is would only an NTLM challenge crash the E20..?

[Danny De Ridder · Accepted solution]

Since you confirmed that the issue your are encountering is indeed CSCue59199, can you mark this thread as being "answered" ?

The reason why E20 is crashing is due to that ddts I mentioned :

CSCue59199 - Boot: SIPAUTH error: unable to clear signature (gState=1)

That ddts is not fixed in any release yet, but there are ways to prevent the E20 being challenged using NTLM so it no longer crashes. VCS changes must be done to avoid the NTLM going out to the E20.

The bug has more info found at bugtoolkit.

Conditions:

VCS registers E20 as an unauthenticated device because the SubZone where E20 registers to is set to "Do not check credentials". After the E20 registers, it will update its Presence information. VCS needs to Authenticate a device before providing Presence, hence the VCS sends an NTLM Challenge (SIP 407 Proxy Auth Required). Once E20 receives this SIP message, it crashes

Workaround:

Make sure the Default Zone and the SubZone where the E20 registers are set to "Treat as Authenticated" or "Check Credentials". The E20 device must be authenticated by the VCS in the SUBSCRIBE process.

If the Presence is provided by a Neighbor VCS, make sure the following configuration is set correctly in the Neighbor Zones in both sides:

SIP Authentication Trust Mode: On

In this way, the VCS will identify as Authenticated Devices all the messages coming from Neighbor Devices that have already been authenticated by the Neighbor VCS

Finally, make sure NTLM Protocol Challenges is set to "Auto" in all VCS's. In X7.2 and later, this setting can be found in AD Configurations. In X7.1, you can find it under Device Authentication Configuration.

[Martin Koch]

What kind of network is that? Did you ever sniff on the network whats going on there?

I have had put a sip enabled E20 with TE4.1.1 on a public non firewalled IP address

(just to see how it behaves) and it never crashed.

You have the typical sip scans which will cause the system to ring, but that does not

sould like the issue here.

I would see if there is some other strange packets going on, like CDP, ...

Put a network tracer, a switch with a mirror port or a hub in between the system and its

network and sniff whats happening.

[RAMEEZ RAHIM]

The E20's are connecting from Home Internet - mostly behind a DSL.

The recent change we had is, enabled AD auth for Jabber on VCS with NTLM.

So, we enabled check credentials on the VCS-C's traversal zone.

I will try to get a capture from the wire and post.