I bought 2 Cisco WAP581 wireless APs. I am having a problem with the wireless going offline after a day or two.
My Windows 10 laptop comes up and says there is no internet access after 24 hours. From the wireless laptop I cannot ping the WAP581 from the wireless side. I also have an iPad which states no internet. The WAP581 has a DHCP IP address. I can ping the WAP581 from a wired PC in the same network. There are 2 SSIDs on WAP581. If I try to connect to the other SSID my laptop will not connect stating there is no internet. I do have internet on the wired PC. If I reboot the WAP581 AP it all starts working again. What is going on?
I am in the USA with version WAP581-A-K9 V01. I am running firmware 18.104.22.168.
I replaced Cisco WAP371 APs with these WAP581 units and never had this problem. The WAP581 are connected to a SG300-28 switch in L3 mode on a trunk port with 2 VLANs defined to the WAP581 AP. The 2 WAP581 AP are setup in single point mode using DHCP off the SG300-28 switch. I am using a SG300-10MPP switch to power the WAP581 APs connected to the SG300-28 using a trunk port.
I did not state I am running both bands with band steering on. If you need me to start a support case send me contact info. If you want me try something I will be happy to as this is my home network with complete access all the time. Let's get this problem fixed.
It's been almost 2 months since my last update. I have still been patiently waiting and sending Cisco all the debug files. But, this has been so repetitive and it's basically sending them the same stuff they already have over and over. I've copied and pasted my last email I sent to them below. There were a few emails I sent prior to this with debug files, but because it was so repetitive I decided to let them add up and try to make sense out of it as much as I could. Then I sent this email March 4th. Carlos, (my service advisor) also asked me about my system setup. I'm not sure why because I explained all that before, but I gave him an update on that as well. This literally has been a pain in my rear, and I feel like I'm working part time for Cisco because of how much time I've put into this. I've given them everything they asked for and need. They have to figure this out now. It's been too long. I have a few months left of this in me before I transition to Ubiquiti. Unfortunately I made the decision 2 years ago to go with Cisco because of their reliability and support, but the reliability has proved me wrong! So far this has been a horrible investment. Hopefully Cisco fixes this soon. My email is below... FYI, Carlos sent an email back saying thanks for the info and he would share the info with the team and follow up with me once they all review it. Also, I'm not attaching everything here for personal reasons, but just a pic of my setup... and one more thing, I've mentioned this thread multiple times to them prior to this email, but here you see me mentioning it again.
This is going to be a lengthy update. If you have any questions, please respond inline to my email so that there is no misunderstanding. I’ve attached a zipped folder that relates to everything that I’m going to talk about and is organized accordingly. Each bold underlined title below relates to the folders within the folder attached so you know what I’m talking about.
Residence Server copy
I’ve sent this document to you before, but I’m sending it again to you to answer your question. Looking at the excel sheet, click on the static DHCP tab. There you will find every single thing on the network except for 3 iPads, 3 iPhones, and 2 MacBook Pros. All the devices on the excel sheet, you can see whether they are wired or on wifi. Remember this is a very large residence with a small business operating out of it that is NOT data driven. At times, these WAP’s went down when there wasn’t even anyone at the residence. Typically, especially during covid there were only 3-4 people at this residence.
Looking at the Static DHCP tab on the excel sheet again, you will see the Hikvision section. There are 30 active ip cameras hardwired that do not allow more than 4mbps each. Those cameras are split between two POE switches that connect to my RV345P. My NVR is basically like a DVR for those cameras to be recorded. The NVR is also connected to the RV345P directly. So basically, there is no more than 120mbps that my Cisco RV345p is handling for my cameras at any given time. I’ve never had any camera problems or my network being slow in regards to anything else.
Not sure if this adds any value, but my internet is Xfinity, with 1gig download and 25 Mbps upload. So those are my internet limits.
Cisco Support (01/20/21)
So you know my internet limits. We typically only stream on our Apple TV 4k’s. We have a few of those on the excel sheet but we usually only use 3 max at any given time. However, we can rule these out because they are all hardwired. The only other thing that I thought of is viewing the cameras at home on our iPad or iPhones. We can view one channel stream at 4 mbps or 16 channel stream at 2mbs on each stream which is a total of 32mbps.
So I ended up doing a load test for you. I got all our iPhones and iPads and made sure to put them in the room with the "10.0.1.17” AP to have them connect to it. I double checked that they were connected, then opened up the camera streams on the iPhones and iPads. I even had a bunch of tabs open on my laptop with YouTube videos running. My laptop was always on the 5g network. Please see the screens shots in the "Cisco Support (01/20/21)" "Load Test screenshots for 10.0.1.17" folder. Screenshots “1A” and “1B” you can see I load tested on the 2.4G network. No issues or crashes presented. Screenshots “2A” and “2B” and screenshots “3A” and “3B” you can see I load tested on the 5G network. No issues or crashes presented.
My conclusion is that it’s not anything that I’m doing on the network or wifi. Also, I’m not the only one having this issue with the WAP’s crashing and I believe the majority of people that have this WAP are experiencing this issue same issue. https://community.cisco.com/t5/small-business-wireless/wap581-with-wireless-offline-problem/m-p/4293497#M10794
Cisco Support (01/31/21)
Config file creds (same as before):
WAP (10.0.1.14) went down down again. I’ve attached the requested files. However, after connecting it back to the power source (RV345p), I logged into the AP and it said “5G disabled” on the dashboard. Please see the screenshots. I looked at the POE section of the RV345p and everything looked normal.
Cisco Support (02/18/21)
Config file creds (same as before):
This WAP (10.0.1.13) hasn’t crashed before. Also it’s not a part of the cluster. I’ve attached the requested files. However, I know this is a completely separate issue unrelated to what we have been dealing with. We will have to make a separate ticket for this issue which is directly related to the RV345p. It may be due to it’s long "system up time.” Who know’s but this is another issue we will have to fix. Please look at the "Router 10.0.1.1” folder to see the screenshots/pictures I took. In one picture you can see the ethernet cord disconnected from the router. Yes, this cord is the one coming for WAP (10.0.1.13). You can see the POE light is still “On” for the port on the router, even though the wire is pulled out. This makes no sense, but it’s probably a firmware bug. I’ve never seen this happen before, but I usually restarted my router every month or 2 at least when trying to figure out the WAP issue that you and I are trying to figure out.
Cisco Support (02/22/21)
Config file creds (same as before):
This WAP (10.0.1.13) went down again. Same results as before. Right now you might be thinking my router is causing a power supply issue but "Cisco Support (02/25/21)” will rule that out. Please read below.
Cisco Support (02/25/21)
Config file creds (same as before):
WAP (10.0.1.17) went down down again. (Since the last time I’ve sent you something on it.) I’ve attached the requested files.
WAP 10.0.1.17 and WAP 10.0.1.16 are not connected to the POE supply from the RV345p. They have their own POE injectors. The product ID on the POE injector is SB-PWR-INJ2-NA. Therefore, my RV345p is not causing any power issues to the AP’s. Plus all the people in this thread that are experiencing the same issue with these AP’s don’t all have the same setup as me. https://community.cisco.com/t5/small-business-wireless/wap581-with-wireless-offline-problem/m-p/4293497#M10794
This is all the information I have. What’s the status on your end?
We all truly appreciate your perseverance on this, just so frustrating that you've had to put so much effort yourself into this. This problem is so easily reproducible with only TWO WAP581s, Cisco could easily reproduce this in their lab, if they even have an actual lab?
It does make you wonder if the person(s) driving this support case even have any hardware, or if they are just overseas looking at crash dumps and have no access to a real lab? sadly I wouldn't be surprised if this is the case, companies are outsourcing so much anymore these things just don't surprise me. Hardware companies like this used to have actual physical labs where they would work on problems like this.
Now they put it back onto the end users...and more and more users continue to add new cases to this discussion, same scenario, and still nothing from Cisco. And another user just posted he opened up yet ANOTHER case with Cisco... they don't even realize there is already your case working the issue..
so now at least two separate support engineers chasing down the bug, with probably no real hardware or lab to work on it from..
*SIGH*...... maybe by 2030 they will identify the issue
I needed to add that I have this problem as well, on two WAP581 devices.
when I see this issue here, it first stops reacting in the 2.4ghz network, and I see my smart home devices go off-line (while still showing as 'connected' in the monitor screen). After a while from that point most of the time the 5ghz network also start to manifest issues, and even the admin webpage stops responding, and in that points, the only remedy is reboot.
I've tried playing with every possible setting in the interface, with no-avail. I must admit I'm very disappointed from Cisco 'small bustiness' WAP products. Even before I owned a WAP150 which also gave me issues when trying to use 3 networks at once.
I'm pretty certain this is a firmware issue (as it 'degrades' over time, and solves itself after reboot), and the total lack of support for this product line is mind-baffling. (I see that mainly the switches get timely firmware updates)
Personally I lost hope, and moved to ruckus APs which solved the issue. (There is nothing like a stable wifi which just 'works').
Hopefully I would be able to sell the existing Cisco WAPs I own in eBay and recover some of my sunk cost
Having used Cisco devices in the enteprise, my experience is that these Small Business devices just don't have the same DNA. There is no comparison.
However, I wouldn't rush out to buy Ubiquiti just yet. Have a look at Aironet 1832 with embedded Mobility Express. I am very happy with it.
But even before that, I would advise this: Get a Wifi Analyzer and make sure that your APs aren't suffering radio interference, from your own or neighbors' APs. I've had clients unable to associate with APs 10 feet away because of interference. And even if they did associate, they suffered continuous disconnections, authentication failures etc. Once I moved to clear channels, the problems went away. Make sure your wifi analyzer can show bonded channels (e.g. 40 MHz channels) as otheriwise you're not seeing the whole picture.
I retired from Cisco after 12 years before starting a a company that builds networks in large custom homes. Among the roles I had at Cisco were Systems Engineering Manager for Enterprise customers, and Channel Partners in the Western US. I remember at least 3 instances where we received a bad part from a supplier, such as a $0.05 resistor or memory module, which was then built into a product's hardware, requiring us to replace the product.
I think that is what has happened with the WAP581s; there is a batch of WAP581s that hit the market with a bad component built into a circuit board. Here's the evidence for this hypothesis: We've been using the same product families over and over for many, many customers; RV series routers, SG series smart switches, and WAP581s, with the same firmware (the latest version in each device) and very similar configs. It's as close to a cookie-cutter solution as possible where everything works and our customers have been delighted.
Until now. We installed the same setup in our most recent customer's network and immediately began experiencing the WAP581 failure described in this thread. After spending thousands of dollars in man-hours troubleshooting the problem, we finally replaced the WAP581s with Linksys' LAPAC2600c and everything works great. Thankfully, our older networks with the same hardware, firmware and similar configs continue to run flawlessly.
As with others who have reported this issue, I am extremely disappointed in Cisco's lack of response. Cisco has such a positive culture, with 'Customer Satisfaction' as a leading corporate goal, but somehow the ball is getting dropped with this issue.
To Cisco, this WAP581 product issue is real. Please address it and restore your loyal customers’ and Channel Partners’ faith in you
I managed to create case for this issue with Cisco, Cisco engineer did some changes to my unit that I copied to the other 2 units. That all happened 5 March 2021. Everything worked fine till Monday morning. Over the weekend maybe 1 user connected. Monday morning 7am as I walked into the office first AP was frozen rebooted number 1, worked fine after reboot. At the same time remote accessed all the rest of the AP's and everyone was online and working. 10:30am number 2 AP froze up and I had to reboot.......... Will see what Cisco say and keep you updated.
I have the same problem as all of you we have 4 AP clustered with POE switch SG 350.
we have to regularly restart 2 APs so one of the two is the master of the 4 APs.
the problem arises even with the classic power supplies we have abandoned this option for the moment because the AP are difficult to access.
Thanks for all of your replies. Hopefully this issue gets resolved because I don't want to spend anymore time or money swapping out hardware. I'm copying and pasting the last few emails between my service advisor and I. I'm happy that it seems like they are really going to fix the issue, but then upset because it almost seems like they didn't even read my long email in full when I asked them about my Cisco RV345p router. Like seriously, I don't want to call in again from scratch and start a new ticket for my router when my service advisor is completely capable, especially spending so much time on this wap issue. I want him to do it because he knows my complete system.
"To" is from me and "From" is from my service advisor.
At this point in time the update that I have is that we have a firmware available that gives the option to set up a schedule reboot for the WAP581 so that they reboot during so that before they start seeing the disconnects, they reboot, and you won’t see the interruptions that you are seeing. I want to be clear with you that this is not the fix. We have identified some other issues within the software that are being fixed and hopefully that beta firmware becomes available in the coming week or 2. But for now that schedule reboot is what we have.
Let me know if you want the reboot firmware.
We have only been working on this case on the WAP581 issues, what RV345P issue?
From my email i sent you recently. Please go back to that email for the attached related document.
Updated from my service advisor:
WAP581_22.214.171.124-telnet.tar” last week still had performance side-effect w/ all the cluster/sw watchdog/mem fixes, our DEV team was fixing it over the weekend. They will address the performance side-effect and provided me final build later tonight/early tomorrow
Hopefully this seals the deal. Testing time...
We have the fixed beta image.
We fixed :
Cluster daemon crash
SW watchdog optimization
Memory Management – memory exhaustion was observed when excessive traffic injects.
Please upgrade all the WAP581 to this image and monitor. Send me all feedback you have.
Awesome, thanks so much for the heads up! I'm keeping my fingers crossed this fixes it once and for all..
again congrats and kudos for all your hard work and time in pushing them to get this fixed....
Keep us posted on your status..
Update with my email response to Carlos. Some good news some bad... still the same problem.
2 of the WAP's went down about 12 hours later after installing the FW updates to them. I upgraded all of the WAP’s on Saturday afternoon, and 2 of them went down sometime Sunday morning.
Attached are the requested files for the downed WAP’s
Config file creds (same as before):
WAP’s that went down (Please see associated folder in zip file attached):
10.0.1.13 (Not part of the Cluster, (Remember, because I wanted the 2.4 GHz radio turned off on this one))
I’ve also attached a screenshots folder in the zip file attached. There you can see a screenshot of each WAP running active firmware version 126.96.36.199-mem. Also you will see "Channel Management" screenshot and "5GHz radio grayed out 10.0.1.14” screenshot. On the 10.0.1.14 WAP it shows the message that the 5G radio is disabled. Yes this is probably because of the RV345p, but on the screenshot "5GHz radio grayed out 10.0.1.14” you obviously see the 5GHz section grayed out because of the message and the radio not being enabled. So now looking in "Channel Management” screenshot (In single point setup) you see 5GHz radio on 10.0.1.14 shown as “up.” This is incorrect because it’s down. Please advise your developers and have them fix this bug.
- Performance (load times) of the dashboard on the WAP’s appear to be much quicker.
- 2.4 Ghz performance in the 20 MHz band increased from the usual 50 Mbps to now 95 Mbps
- 5 Ghz performance in the 20/40 MHz band increased from the usual 215 Mbps to now 280 Mbps
I've noticed you mention 2 WAPs go down in most of your posts.
Is it always the same 2? Have you connected them directly to the PoE switch with short, known-good cables to rule out the wall wiring having something wrong (one of the power pairs being broken or having a questionable connection, for example)?
FYI, you also mention your router shows "PoE" without anything connected. If you don't know how PoE is detected and supplied, here's a quick primer: the PoE sending device checks for low resistance across the power pairs. If low resistance is detected, it supplies power. It doesn't care if there's anything plugged into the RJ45 jack or not. No resistance, it'll apply power to the pins. If your router is the only thing that serves PoE, try a power injector on another port. The PoE logic on that port might be blown up/fried in some way and causing PoE fluctuations.
Just some thoughts and stuff I've seen in 15 years of supporting networking/VoIP (which includes thousands of PoE devices).
Thanks, but unfortunately it's random and only the ones within the cluster. I have all WAP 581's. I have one not associated in the cluster is because I needed to turn off the 2.4 Ghz radio only on that device based on the site survey. Also the Cat6 cables have been tested and retested, all checking out.
So the latest debug FW Cisco sent me, which I talked about in my previous post seems to improve the performance and stability of the WAP's dramatically, however the main issue was still happening. So, Cisco sent me a SG350-52MP 52-Port Gigabit PoE Managed Switch to test out with.
Heres my response email after receiving the switch:
Updated the rv345p and cisco switch you sent me to the latest firmware available. Also assigned 10.0.1.5 to the switch on the rv345p using static dhcp. Shut the system down. Plugged all the ap’s into the new cisco switch. Used port G49 on the switch and plugged it into port 16 of the rv345p. Turned the system on and everything is up and running.
Here is another follow up response email I sent because everything was working great:
Just wanted to send you an update. For nearly almost 8 days I haven’t seen any issues arise and none of the AP’s have went down. Please see the attached screenshot. Look at the “Total PoE Power Consumption.” You’ll notice it shows 183. In this link: https://www.cisco.com/c/en/us/products/collateral/wireless/small-business-500-series-wireless-access-points/datasheet-c78-738872.html it states that the peak power of this AP is 18 watts.
So basically, I'm thinking it's a power issue now. At least for my previous setup not being able to supply it. But the system was based on the specs that Cisco told me and they said it would all work perfectly together.
Carlos told me to run these commands:
Show power inline
Show LLDP Neighbor
Show CDP Neighbor
Show interface status
It shows that each WAP is using 25.5 watts. Way over the peak 18 watts Cisco claims. So now for 2 weeks I haven't run into any issues with using the new switch supplying power to the WAP's using the same ethernet cables. Literally everything else stayed the same, just the new power supply.