cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
24068
Views
54
Helpful
33
Comments
Comments
George Stefanick
VIP Alumni
VIP Alumni

I have to be honest here. I have issues with these comments.

1. PSK is starting to be the more preferred means of security because large enterprises don't want 1 single AD account on all phones and they dont want individual AD account on phones because of the amount of time to takes to manage these accounts through RMAs and deployment.

This talk about going to EAP is poop. Sounds more like "we have a issue, but since we really cant figure it out lets just say you should be using EAP". Lets not kid ourselves PSK is very secure if you manage the KEY properly.

Regarding CSCtt38270 - you are right that this bug is not open, so it won't get fixed unless something changes.  For this bug to get fixed, the following will need to occur:

  • there must exist a customer who is determined to use PSK
  • this customer must have a TAC case open
  • the TAC engineer must reopen the bug
  • the customer must be willing and able to reproduce the bug, and to collect over-the-air traces, debugs, logs, etc., and to run test code from engineering as needed

2. I have thousands of phones and we are starting our upgrade next week. I have to say Im worried. Im very worried. My organization has spent millions as a customer of Cisco I shouldnt feel this way.

I am one of many orgs that have more cisco phones then the population of some small towns.

I think its very poor for this to be communicated in this manner.

Sorry - I have nothing but love for ya .. but this is poop just my 2 cents

migilles
Cisco Employee
Cisco Employee

WPA2+CCKM has been a recommended security type for enterprise deployments ever since the 7921 started to ship in January 2007.

Even with the 1st gen 7920, the recommendation was WPA+CCKM; once CCKM was added.

PSK is primarily for home or small/business deployments, which is why  PSK is also   referred to as WPA/WPA2 Personal and why 802.1x methods  are called WPA/WPA2   Enterprise.

I have spoken to many customers around the world over many years with   enterprise deployoments where users are roaming frequently and almost   all of them have implemented CCKM.

And the reason for recommending WPA2+CCKM is to take advantage of the seamless roaming via CCKM and not per se from a security perspective.

This should not come as a surprise to anyone.

However, we are not saying that PSK is not secure, but it definitely doesn't have the additional security level of potentially having unique 802.1x accounts per device, where an account can be revoked as necessary preventing that device from accessing the network.

Sure you could enable MAC based authentication, but that is pretty much considered as legacy security; plus would need to either manage the lists ont the WLC, AP or RADIUS were the list is stored anyways as with 802.1x.

So why not go with WPA2+CCKM which offers better seamless roaming and less chance of a packet exchange during a roam (less packets exchanged).

There are certainly deployment types that should go with PSK vs WPA2+CCKM, hence my reply in regards to the "Never PSK" comment in the thread above.

1)  "Never PSK" is a bit harsh.  There are definitely scenarios where  PSK can be used.  But if frequent roaming occurs, then obviously  WPA2+CCKM utilizing PEAP or EAP-FAST is the recommendation.

Customers may go with PSK if they do not want to manage or purchase a RADIUS server, or the size of the deployment doesn't warrant it. 

However local radius is an option in the Cisco WLAN Controller and Cisco Autonomous AP, so that is an option for the smaller deployments.

Or maybe even a slim chance that the deployed AP doesn't support WPA/WPA2 Enterprise methods.

As for CSCtt38270, there were some customers encountering some intermittent roaming gaps and it seemed that it may have been due to an interop issue with the 792x and PSK.  After design review and the need for a quick solution, some customers decided to move to WPA2+CCKM, which is better for highly mobile users; less voice gap during roaming as there is no need to redo the key handshake.  With CCKM, just a reassociation request and reassociation response; less than 100 ms voice gap, typically 50-80 ms.

So in some situations, it was a win-win, where the issue faced was resolved and using a feature eliminates the need for re-keying, therefore diminishes the chance of any frame during the key exchange to fail to get to the other end and avoid any EAP timeouts, etc.

In other situations, the issue with PSK was not reproducible.

So what Aaron stated, is spot on.

Since we can't reproduce this in Cisco labs, we would more than likely need to rely on the customer deployment to troubleshoot any issues related to this.

And to get to the BU, TAC must escalate to the BU; therefore the need for a TAC case.

And of course, we would need the traces/logs capturing the issue to get to any root cause.

If you have a customer impacted by an issue, then please raise a TAC case.

Sometimes, there are  things in the wireless world that are just seen at a single location and not reproducible in labs or other deployments.

WLAN is much more complex and has many more influences than a wired solution.

Hope this helps!

If should you have any further concerns, please provide some detailed info about your concern and we will be happy to help.

George Stefanick
VIP Alumni
VIP Alumni

I appreciate the quick response as always.

Understand I'm not trying to bust your chops, you and Aaron have my full respect and attention. But something needs to be said if we think making accounts for each device or one account for all phones is the answer to this problem. Let me be frank and tell you as a wifi engineer managing thousands of phones it is not the answer.

Large customers are now looking at PSK and moving to PSK because of the amount of management and excess overhead it requires to manage AD accounts. In fact in the last 12 months I have personally spoken to 3 other very large Cisco customers with thousands of phones (most Im sure you know) who they themselves are moving to PSK. (One of which actually contacted me about my thoughts about this thread because they are in the middle of their EAP to PSK conversion). One customer in fact has 4,000 AD accounts for phones, but they can only account for 3,000 phones. Over the years as phones were replaced due to RMA and other means they were never deleted because of the management burden.

As for the recommendation -- I read the deployment guides and I dont recall any of the guides stating "Cisco Recommendation." Rather it states, if you are deploying 802.1X, it is recomended to use CCKM.

My point is this. WPA(2) PSK is by far a better choice of roaming on phones. Is it not? It negates the new of a MSK key, PMK Key, radius servers, AD accounts and the over head associated with the management of the radius accounts. Clearly PSK shouldn't be deployed on all devices. But there are means and ways to deploy Cisco phones with a PSK in a secure manner for example with Wavelink whereby the PSK key is protected from end users.

My organization is moving our thousands of phones away from EAP for the very reasons noted above.

I have an upgrade plan for next week to SR1 and I sure hope we dont get hit with this bug as most of our phones are WPA2/PSK. It is comforting to know <I think, that you can not reproduce the problem>. Im opening a TAC case tomorrow, perhaps you can grab it and we can discuss further.

migilles
Cisco Employee
Cisco Employee

We are simply promoting use of CCKM for enterprise deployments where frequent roaming occurs.

As you are aware, CCKM requires an 802.1x account, but this doesn't mean you have to have an account for each device. 

We have 792x customers using the traditional method where each device has its own account and we have customers where they have a single account for all devices.

And in the unique account scenarios (e.g. healthcare & retail), the accounts would be IT managed with complex passwords.

Per your comments, it appears that you appear to fall into the category of "not wanting to manage a RADIUS server" as outlined in my previous statement.

Customers may go with PSK if they do not want to manage or purchase a  RADIUS server, or the size of the deployment doesn't warrant it. 

We have the Cisco Bulk Deployment Utility (BDU) available, which can help deploy common or unique credentials easily.  Can reference a CSV file containing MAC address, username & password to create unique templates.

It doesn't support certs, but doesn't appear that EAP-TLS or PEAP with server validation is a concern for you.

FYI, PSK is NOT a better choice than CCKM in regards to roaming performance.

As mentioned, CCKM is 2 frames, where PSK has those 2 frames plus the 4-way and 2-way for the key exchange, therefore more time off-network and creating a larger audio gap during the roaming event.

PSK is simply a secure method that is easy to deploy.

If the pre-shared key is compromised, then any device can access the network.

Where as if you use unique accounts, you can disable an account on a 1 by 1 basis as needed vs having to change the key for the entire network.

Anyways, not trying to push WPA2+CCKM on you.

Simply stating that it is the preferred security method for enterprise deployments where frequent roaming occurs.

FYI, I am the author of the Cisco WLAN Endpoint Deployment Guides (792x, 9971, Cius).

Yes, definitely should enable CCKM if using 802.1x to avoid doing  the full EAP handshake for each roam; not only to take that load off of  the RADIUS but to also decrease roam times (reduce voice gaps while  roaming).

I will add some more verbiage in the Roaming and CCKM  sections in the guide to make it clear that CCKM is the recommended  deployment model for enterprise deployments where frequent roaming  occurs.

If you are currently using 1.4(2) or later and not encountering any issues with PSK, then you should not encounter any issues with 1.4(3)SR1.

If you do encounter any issues, simply open a TAC case for assistance.

FYI I am not a member of Cisco TAC, but the folks that are including Aaron know how to contact me if necessary (i.e. there is an issue requiring BU assistance).

Aaron
Cisco Employee
Cisco Employee

Hi George,

Regarding this:

"Large customers are now looking at PSK and moving to PSK because of the  amount of management and excess overhead it requires to manage AD  accounts. In fact in the last 12 months I have personally spoken to 3  other very large Cisco customers with thousands of phones (most Im sure  you know) who they themselves are moving to PSK."

There is certainly no requirement that, just because you're using WPA2-Enterprise, your user credentials have to reside in AD.  Certainly if your security model is that each phone has to have unique credentials, then it makes sense to have AD be your identity store.  (But then, if that's your security model, you couldn't possibly use PSK!)

The simplest method for deploying WPA2-Enterprise as recommended with the 792x phones, is to configure the WLC as the local EAP authentication server for the voice WLAN.  The WLC does a good job of handling EAP-FAST or PEAP authentications for the phones, in large deployments.  (In very large deployments, it is a good idea to tune the "advanced eap" and client exclusion settings, so that 802.1X exclusion kicks in if needed, so that you can effectively perform admission control after a widespread outage ... [note to self: document this in a tech tip].)

All that said ... I agree that WPA2/PSK should work well.  If we have a large deployment that is moving to PSK, then let's make sure that it does work well ... if you should encounter the CSCtt38270 syndrome, then be sure to open a TAC case, and we'll work with the BU to get this resolved.  (It may well involve extensive debugging on the WLCs, APs, and in the phones, with coordinated over the air sniffs, and will likely involve running test firwmare on the phones.)

Aaron

migilles
Cisco Employee
Cisco Employee

Yeah I think local radius feature in the Cisco WLC would be fine for smaller deployments, but for larger 802.1x deployments, one would want to go with Cisco ACS as that is a central storage point, which can get replicated to other servers (i.e. don't have to worry about entering the accounts into every WLC).

As far as CSCtt38270 goes, I have just reopened this as this is under further investigation currently.

It would no't be resolved before 1.4(4) goes out though.

As Aaron mentioned, if using PSK, you would want to decrease the EAPOL key timeout from 1000 ms to 400 ms and increase the # of retries from 2 to 4 as outlined in the 7925 Deployment Guide.

George Stefanick
VIP Alumni
VIP Alumni

Thank you ..

All good discussion around a very important topic that impacts a lot of us. I think someone coming across this tread can learn from the various input.

Migilles thank you for considering the reopening of the bug.

scott.stapleton
Level 1
Level 1

Hi Aaron,

Thanks for the response. I appreciate the effort of the post. Fair enough if it is just addressing voice-only deployments. You can't address all varieties of designs/deployments in a post after all but obviously we're often thinking of the bigger picture in designs.

With regards to authentication, this is a topic George and I had discussed previously. His issue with 802.1X in this context is with the administrative burden of managing individual user accounts and I completely agree. It comes down to the usual security --> usability trade-off. The 802.1X alternative then is a single account; essentially a service account. My big issue with this is the significant risk in taking down your whole VoWLAN if something happens to that account. I have seen service accounts disabled and even deleted by ICT staff and scripts when performing clean-ups of AD and just fat-fingering of accounts. The network team generally doesn't even have access to AD, but systems, desktop and even service desk teams often do. Your VoWLAN continuing to function could potentially rest on often large numbers of often inexperienced ICT staff.

Lately I have been thinking that a good middle-ground between these two issues and using PSK is to use 802.1X with a single service account but have that service account stored on the WLC(s) themselves. In this way, you limit the risk to only network teams who have access to the WLC. This risk is far, far smaller. This is essentially what you've proposed above!

My only issue then comes back to the security. The irony of using the more secure PEAPv0 is that because BDU doesn't support pushing out root certificates, the phones will not be able to perform mutual authentication and risk a MiTM attack. Obviously you can enable mutual auth but per below, how do you get the root CA cert on the phones in a large scale, enterprise-friendly method (i.e. - not the web interface) on each individual phone.

EAP-FAST using Phase 0 also means having to rely on anonymous PAC provisioning. The risk is very small I would expect, but still present… like most complex security risks, until of course someone goes and makes a GUI front-end script-kiddie friendly tool to automate the process (a la Firesheep).

With the above in mind:

  * Is there any easy way of getting private PKI root certificates on to the phones (does Wavelink support deployment of these)?

  * Do the phones have public PKI root certificates installed on them, out of the box, if someone wants to go down this path?

  * Is there a way to retrieve the PSK or account password on a 792x phone?

If I could get the root CA certificate onto phones in an enterprise-friendly manner I would be happy running WPA2-Enterprise using PEAPv0+CCKM using a single service account hosted on the WLC! If not, then it is a toss-up between keeping the PSK secure vs. potential MiTM from not performing server authentication when using PEAPv0.

Cheers again for the useful post.

migilles
Cisco Employee
Cisco Employee

It was thought that a customer was encountering CSCtt38270, therefore it was reopened, but after further investigation it appears that there are other issues (design and config) that are attributing to voice gaps.

This is currently in I state but it could very well be put back in U state.

migilles
Cisco Employee
Cisco Employee

Server Validation (requiring root CA cert on client) with PEAPe is optional.

This is simply to prevent communicating with a rogue RADIUS sever.

Currently certificates must be installed through the 792x phone's web interface.

The CSR for a client cert must be run from the 792x phone's web interface.

The manufacturing certificate (MIC) is pre-installed which can be used as the client cert for EAP-TLS.

The CA chain that signed the MIC is also pre-installed and can be exported to enable in the trust list go the RADIUS server.

The PSK or password can not be exported from flash.

A template containing the network profile config can be exported but an encryption key must be specified where that template will be encrypted and can only be imported to another 792x phone.

Also if production 802.1x accounts are being deleted intentionally or accidentally then there are bigger issues in play.

muranskycotech
Level 1
Level 1

I'm a little more interested in the 2.4Ghz information, since I currently don't have the luxury of dual-band APs. I'm pretty extensively Aironet 1242G APs at this point with plans to implement (at least) 1262AGN APs next year.

What guidance should be considered in that scenario?

Darren Ramsey
Level 4
Level 4

Going to agree with George on this one, PSK should work correctly.

We are a Cisco shop utilizing Catalyst 3/4/6K, Nexus, UCS, MDS, ONS, ASA, CUCM and over the years had more cases with Wireless than all the others platforms combined. I would classify us as a medium sized healthcare system (6000+ employees and around 1000 beds) and have always used PSK for 792x and Vocera. Sure we could use PEAP/Radius like we do with our laptops/tablets/carts etc, but PSK is just easier for phones and our key is protected. Been running Aironet since 2002 and WLC since just after the Airespace acquisition and have found plenty of bugs for the WBU over the years, (GLBP compatibility and several 7.4 SSO bugs to name a few) and have wasted more than our fair share of time playing the "convince TAC we really have a problem so that we might escalate this to WBU" game.

So if Cisco knows there is a problem with 792x/PSK and has not fixed it, then I'll call that out as blatant negligence around a device that is used to provide patient care and improve life safety in our environment. It's not like we are using Aruba and 7921, or WLC and Spectralink, we are using Cisco on Cisco...So dust off the gear used for the useless AssureWave program, run some debugs and find and fix the darn bug. Show us some value for being a loyal Cisco customer and using Cisco for both WLC and VoWLAN.

migilles
Cisco Employee
Cisco Employee

Hello Darren.

We appreciate your comments on this matter.

Are you responding to this post because you have users that are currently experiencing issues with their 792x phones when using PSK?

For CSCtt38270, the due diligence has been performed by our BU and unfortunately this rare issue falls into the small bucket which can not be resolved on the 7925 platform. 

This issue is specific to the 7925G & 7925G-EX platforms when using PSK; does not exist with the 7921G or 7926G platforms.  This issue is also not Cisco AP specific.

Originally this thread was worded in a way where it sounded like this issue occurs very frequently / for every roam if using PSK, but that is definitely not the case!  I can personally attest to that as I have performed 1000s of roams in our lab environments with our 7925 phones using PSK and have been unable to reproduce the issue.

This issue is very rare indeed!

But if this issue does occur, then worse case is there could be a 1 second gap if on call assuming that the default EAP settings are used. 

With PSK, EAPOL data retries could potentially occur in regular scenarios even when the CSCtt38270 signature is not matched.

As per my 7925G Deployment Guide, if using PSK it is recommended to reduce the EAPOL key timeout from 1000 ms (default) to 400 ms and increase the EAPOL key max retries from 2 (default) to 4 on the Cisco WLAN Controller.

Please refer to the details outlined in the bug for more info.

Also FYI, the WLAN endpoints such as 792x, 9971, DX650 are from the Telepresence and IP Phone BU.

The Wireless Networking BU is responsible for the wireless infrastructure (APs, WLAN Controllers, etc.) who we work very closely with.

Thank you.

Darren Ramsey
Level 4
Level 4

Thanks for the quick reply. Not sure if we have seen the issue as our customer base (nurses) would likely just complain amongst one another and drive on rather than calling in a ticket to the help desk. We are around 80% 7921 and as they are retired/replaced with 7925 over time, I could see the probability of the condition increasing.

Didn't mean to give you a hard time,  but still waiting on a stable 7.4 to run on 4 WLC HA-SKUs that have been sitting idle since purchased in January. 7.0.x has been very stable for us and just don't need any trouble with PSK right now nor 7.4/7.5 bugs....

babai
Level 1
Level 1

Hi 

Could you please let me know why EAP-FAST with CCKM is more recommended than EAP-TLS with CCKM. 

Is it only because of Bulk Deployment Issues in managing client certificates or it is slower (roaming) compared to EAP-FAST.

The reason of asking this question is most of the guides recommend EAP-FAST than EAP-TLS with CCKM. 

Thanks in advance.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: