3750 Switch Marking ShoreTel PoE Phones - Faulty.

[Paul Armstrong]

All,

We are running into an issue with a new 3750 - 24 port switch. The switch is setting ports that phones plug into as 'faulty'. 

Doug-SW1#sh power inline

Module   Available     Used     Remaining
          (Watts)     (Watts)    (Watts)
------   ---------   --------   ---------
1           495.0       30.8       464.2
Interface Admin  Oper       Power   Device              Class Max
                            (Watts)                           
--------- ------ ---------- ------- ------------------- ----- ----
Gi1/0/1   auto   off        0.0     n/a                 n/a   30.0
Gi1/0/2   auto   off        0.0     n/a                 n/a   30.0
Gi1/0/3   auto   off        0.0     n/a                 n/a   30.0
Gi1/0/4   auto   off        0.0     n/a                 n/a   30.0
Gi1/0/5   auto   off        0.0     n/a                 n/a   30.0
Gi1/0/6   auto   on         15.4    Ieee PD             3     30.0
Gi1/0/7   auto   faulty     0.0     n/a                 n/a   30.0
Gi1/0/8   auto   off        0.0     n/a                 n/a   30.0
Gi1/0/9   auto   faulty     0.0     n/a                 n/a   30.0
Gi1/0/10  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/11  auto   faulty     0.0     n/a                 n/a   30.0
Gi1/0/12  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/13  auto   on         15.4    Ieee PD             3     30.0
Gi1/0/14  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/15  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/16  auto   faulty     0.0     n/a                 n/a   30.0
Gi1/0/17  auto   faulty     0.0     n/a                 n/a   30.0
Gi1/0/18  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/19  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/20  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/21  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/22  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/23  auto   off        0.0     n/a                 n/a   30.0
Gi1/0/24  auto   off        0.0     n/a                 n/a   30.0

We've tried:

1. Swapping out cabling.

2. Rebooting the switch.

3. Admin-ing down the ports.

4. Un-applying power to the ports / re-applying power.

But, there are some phones that simply will not power-up. Also, the 'faulty' status follows the phone if we move it to a different port.

The debugs show:

Mar 14 10:37:48.141 CDT: ILP State_Machine ( Gi1/0/11 ): State= ILP_PD_DETECTED_S, Event= IEEE_TSTART_FAULT_EV
Mar 14 10:37:48.141 CDT: **** Instance 0 : Sub Address 0 *****
Mar 14 10:37:48.141 CDT: ***************************************

Mar 14 10:37:48.141 CDT: ILP uses DC Disconnect(Gi1/0/11): state= ILP_PD_DETECTED_S, event= IEEE_TSTART_FAULT_EV
Mar 14 10:37:48.141 CDT: ILP Power apply to ( Gi1/0/11 ) Okay
Mar 14 10:37:48.141 CDT: ILP Power Accounting RELEASE_PWR ( Gi1/0/11 ) Okay, pwr_used= 30800
Mar 14 10:37:48.141 CDT: ILP Power problem detected ( Gi1/0/11 )

I found a thread on our phone vendor's website that discusses this issue, but none of the work-arounds worked for us.

http://www.shoretelforums.com/forums/administrators/2240-cisco-3750-not-supplying-poe-initially.html

I also found a couple of DDTS that describes these: debugs.
DDTS:

CSCtr74459

CSCed09484

Both of which have no work-arounds for them.

We then took-away the 3750 and plugged in a 3560 to power-up the phones and all the phones powered-up without a problem.

So, this is a platform specfic issue.

[Matthew Blanshard]

What version of code are you running on your switch?  Check out CSCsw18530.

-Matt

[Paul Armstrong]

We're running:

c3750e-universalk9-mz.122-55.SE3

(C3750E-UNIVERSALK9-M), Version 12.2(55)SE3

[Matthew Blanshard]

Try and configure "power inline port 2x-mode" on the port and shut/no shut it and see if it powers up the phone,

-Matt

[rsimoni]

Hi Paul,

I suggest you to follow the PoE troubleshooting guide and try to manually set the power inline consumption. Likely these phones are not IEEE compliant.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/g_power_over_ethernet.html

If you still have issue go for a TAC case as I could not find any known issue.

Riccardo

[Paul Armstrong]

I did try setting the power inline on one of the effected ports - but the problem remained.

We did not try the 2x-mode on the port.

Thank you for the input. I've already opened up a TAC case.

[Matthew Blanshard]

What's the case number?  PM me if you don't want to post it here.

-Matt

[Matthew Blanshard]

Just to document here, but this looks to be a good cantidate for the power inline 2x-port mode.  Basically this portion of the debug:

Mar 14 10:37:48.141 CDT: ILP State_Machine ( Gi1/0/11 ): State= ILP_PD_DETECTED_S, Event= IEEE_TSTART_FAULT_EV

That means that we are receiving a voltage spike back from the device which is over threshold and we are shutting down the port to prevent damage to our components. 

CSCsw18530 implemented this power inline 2x-port mode method to handle devices which spike the current like this but don't actually fall out of the IEEE 802.3af spec.  The momentary spike is out of spec, but the power is measured as an average and the average doesn't fall out of spec (though it's close). 

You will need to manually configure that on any port that has this problem.  Only certain platforms are vulnurable to this spike from the PD device and others can handle it without issue.  That's why it is working correctly on the 3560 switches and not on the 3750x switches. 

If after configuring that and it still doesn't work then that means the shoretel phone is out of IEEE spec and the electrical resistance on the phone is too low and allowing too much current through when the initial detection is going on causing the voltage to fall out of spec. 

-Matt

[Paul Armstrong]

Matt,

Thank you for the clarification. I have passed this information onto my team.

Thanks again.

[Leo Laohoo]

Hi Paul,

Can you try something?  Can you do a TDR test on the "faulty" interfaces?

1.  Command:   test cable tdr inter ;

2.  Wait for approximately between 5 to 7 seconds.

3.  Command:  sh cable tdr inter ;

4.  Please post your output.

[Paul Armstrong]

All,

We have confirmation.

By adding the command: power inline port 2x-mode - does resolve this issue. The phone will correctly power-up.

I appreciate the input and assistance on this issue.

[Matthew Blanshard · Accepted solution]

Awesome.  If it's resolved please mark the thread as answered. 

-Matt

[nextira_grob]

Same problems on C3560-48PS, running 12.2(55)SE6.

No possibillity to configure mentioned configuration on the interface.

We've than exchanged the switch, but problem still occured on the same port.

So might be the end-device be the problem? (Alcatel ipTouch)

This issue machting with known bug CSCsw18530

CSCsw18530 - Power Imax / Tstart errors may be reported with some third party PDs

The fix is present in the following platforms only :
3750-E, 3560-E, 3750-X, 3560-X, 3560-C. The other platforms do not support 2X power mode and the workaround would be to use a longer cable.

Workaround:
Longer cable (>50ft) seems to fix the issue in most cases.

It seems weird to install a longer cable - but we'll try it out now...

[Paul Armstrong]

We recently upgraded our IDF switches and noticed that we didn't have any issues with the ShoreTel phones being marked as 'faulty'.

The code we're running on those switches is: 15.0(2)SE2 - which is the latest code.

You may want to try upgrading to the latest code and see if the problem persists.

[Matthew Blanshard]

You are running a platform that there is no software workaround for the ShoreTel phones sending too much voltage.  On the platforms where you can configure the command we are able to absorb some of the spike in voltage, but on the other platforms the hardware is not capable of it and we have to disable the port to avoid damage to the hardware.

So you really only have 3 options. 

1. Get ShoreTel to stop spiking the voltage towards the switch to out of spec values
2. Try a longer cable (may/may not help)
3. Attach the phones to one of the platforms listed in CSCsw18530 and configure that workaround

-Matt