Mitel5324 voip phone takes 30 mins to boot on Cisco3850 switches

[d.lachapelle]

Hi guys, I'm more of a network guy than a telecom guy (we do have 2 of those guys that take care of the voip side), but I'm hoping someone has seen this issue before.

We are primarily a Cisco 3750, 3750G, 3750X shop in terms of switches but recently got a bunch of 3850Xs.

We noticed that the Mitel phones are now taking 30 minutes to fully load up and get a dialtone on the new switches compared to the old 3750s.  We have tried phones that were previously registered, brand new, etc.  We also confirmed that when we take the non-working phones to the 3750s they boot up immediately as usual.  If we take them back to the 3850s they take about 30 minutes.  This is strictly POE only, we do not use injectors or power blocks on our phones.  We have confirmed this on multiple stacks, with all horizontal cabling removed from equation.

We also noticed that this only applies with the data vlan + voice vlan type port configuration, NOT the trunk-type config.

data + voice vlan config (takes 30 minutes):
interface GigabitEthernet3/0/27
 switchport access vlan 323
 switchport mode access
 switchport voice vlan 445
 spanning-tree portfast

trunk-type config boots right away:
interface GigabitEthernet2/0/9
 switchport access vlan 323
 switchport trunk native vlan 323
 switchport trunk allowed vlan 1,323,423,1002-1005
 switchport mode trunk
 spanning-tree portfast trunk

We have confirmed this on Mitel phone model 5324 so far.  Anyone have any ideas here?  DHCP scopes have plenty of room, no issue there.  Options and helper addresses must be set right as it works eventually and with the 3750s.  We could really use some help on this as it's setting us back on our voip deployment.  I am going to open a tac case on this.

[Charles Hill]

To see exactly what is happening on the interface, I would create a monitor session on the switch and do a packet capture.  Hopefully you will see what the holdup is.

I'm seeing different vlans assigned to the working interface and the slow interface.

With both ports configured with the same two vlans only, does it still take 30 minutes to boot up?

[d.lachapelle]

We have a capture and it isn't giving us much.

The vlans don't seem to matter, it doesn't work with whatever vlans we try.  In terms of config we even went as simple as:

switchport mode access

switchport voice vlan 445

spanning-tree portfast

Still doesn't work.  The same exact config works everywhere else on the 3750s, we can't get one to boot in a normal amount of time on the 3850s.

[johnd2310]

Hi,

Do Mitel phones use LLDP-MED? If they do,then you might want to look at LLDP-MED interoperability issues with the 3850. Have you tried a different IOS on the cisco switches?

Thanks

John

[d.lachapelle]

Hi John, thanks for the reply.  It looks like they can use both CDP & LLDP.  I have confirmed lldp is turned off on both 3750 & 3850 switches.

[Charles Hill]

Do you mind uploading the packet capture? 

[d.lachapelle]

It doesn't like me trying to upload the .pcapng file so i renamed it to .txt.  Just rename the extension to .pcapng and hopefully it will work.  We only grabbed this from an RSPAN so we're not getting CDP/LLDP stuff.  We're going to try to get the capture from the switch itself so we can grab that info hopefully.

thanks for helping me out

[Charles Hill]

Hey d.lachapelle,

I see a ip address conflict on your network.  Looks like it may be the pc and phone grabbing the same ip address.

The two duplicate ip addresses are on the folowing mac addresses.
0023.2456.1a98(Gpro computer)  0800.0f83.4513(Mitel)

There is dhcp traffic from 11:27 to around 11:43 or later. 

At 11:43 I see the Mitel phone sending out another dhcp discover packet.

To verify you could assign an available static ip address to the Mitel phone and see if that resolves the issue.

Just make sure the ip address that you give the Mitel phone is not being used. 

[d.lachapelle]

I'm not sure that's the issue as the phone works immediately on the 3750.  We can also reproduce this with multiple phones, and all ports and 3850s react the same as we tried different stacks and different ports on switches in those stacks.  We also tried different VLANs, so a duplicate IP would only hit one vlan and one phone, right?

We have also confirmed that the Mitel 5220 phone works perfectly on the 3850X but the Mitel 5324 and 5330 models do NOT.

We are trying to get a better capture which includes the CDP info as we didn't have that in the first cap.

[Charles Hill]

Here is a line from the packet capture that states there is a address conflict.

[Duplice IP address detected for 10.100.23.43 (00:23:24:56:1a:98) - also in use by 08:00:0f:83:45:13 (frame 51)]

[d.lachapelle]

More information on this.  We confirmed this is NOT affecting Mitel 5220 phones but IS affecting Mitel 5324s and 5330s.

Here is exactly what we saw on the screen during this time from the beginning of plugging it in:

10:15 waiting for lldp, release version, waiting for dhcp, dhcp discovery
10:15 fails and goes to "option 128 missing"
10:18 "Use superkey to send pin"
10:18 phone got ip in voice vlan and I could ping it for 50 seconds, then phone reboots but does NOT lose power
10:19 goes back to beginning waiting for lldp, release version, waiting for dhcp, dhcp discovery
10:19 option 128 missing again
10:21 got ip in voice vlan again, back to "use superkey to send pin".  I can ping that voice vlan ip again for 43 seconds until it reboots. screen was still showing "use superkey to send pin"
10:22 back to beginning waiting for lldp, release version, waiting for dhcp, back to dhcp discovery, fails
10:24 back to having an ip and asking for pin.  
10:25 reboots again after 30-58 seconds
10:26 back to beginning

rinse and repeat for approx 30 minutes total.  Eventually it locks up and stays at the superkey forever, which is what we want.  This also happens a little differently on phones that are already superkey entered, but the result is the same.

[Charles Hill]

I found a site that discusses possible reasons for option 128 missing. 

When plugging in a Mitel IP phone (5312, 5320, 5330, 5340, 5360, etc) the first thing it does is try to get an IP address so it can communicate on the network. Option 128 is a Mitel legacy option that tells the phones where the Mitel 3300 PBX lives. Typically it would read something like 192.168.1.2

Option 128 has been superseded by option 125 and now only exist to provide backward compatibility with legacy Mitel telephones.

It suggest:

What to do if your Boot ROM is out of date:

This is one of the simplest things to correct. The answer is to upgrade the boot ROM. To do this, simply hold the '7' key while booting the phone. It will ask you if you want to configure Teleworker (TW) mode. Say yes and enter the IP address of your Mitel 3300 IPC PBX. Save the change and allow the phone to boot. Doing this bypasses option 125/128 and will use what ever IP address is handed out first. The phone will upgrade it's Boot ROM. Once the phone has upgraded, reset the phone again while holding the '7' key and delete the option. The phone should not boot up using options 125/128.

If this doesn't help, there are a couple more options on the following link.

http://www.mitelforums.com/articles/option-128-missing.php

[d.lachapelle]

We solved the problem although the solution is not necessarily to our liking.  

We have determined that CDP is causing this issue.  When we enabled global LLDP and the phone boots immediately.  

[Charles Hill]

Thanks for the update.

Glad that you guys got it resolved!

[d.lachapelle]

Thanks cehill.  We're just not sure what effect LLDP will have on our switches, we've never had to turn this on before with these phones.