Re: AIR-AP2802I-B-K9 AP offline following upgrade to 8.3.140.0 on <1G interface

Ethan Grinnell · ‎03-01-2018

Howdy,

I wanted to share an interesting issue with 2802 APs. I completed an upgrade to 8.3.140.0 and found that many of our 2802s did not return to operation. After a bit of tinkering and bouncing ports I found that the ones that were down were all connected to 1G interfaces and had their port speed set to auto negotiate 100M with the command "speed auto 100"or had physical issues preventing the link from coming up as 1G. Even stranger is that the link is actually up on both sides, it's just wasn't functional. The AP never shows up in CDP on the switch. I can't confirm if other Wave 2 APs are affected, we don't have 3802s or 1850s, but none of the IOS APs have this problem. 8.3.133.0, 8.5.110.0, and 8.5.120.0 all work without issue and allow the 2802 AP to connect and function off of 100M ports.

Just FYI

Leo Laohoo · ‎03-01-2018

I believe this is a bug. Kindly raise a Cisco TAC Case.

Ethan Grinnell · ‎03-02-2018

I did, the TAC agent is working on recreating it in the lab. But since I didn't find anything in the bug reports or here I thought I'd let it be known, so others don't have to waste time trouble shooting

Leo Laohoo · ‎03-02-2018

@Ethan Grinnell wrote:
The AP never shows up in CDP on the switch.

Hold on ... I've seen this before.

What is the exact model of your Cisco switch? What is the IOS version?

Ethan Grinnell · ‎03-02-2018

I had it on a few different switches, 6800IA-48-FPD, WS-C3650-48PS , and WS-C3750-48PS. I'd have to check on software versions, the 3750 probably had 12.2(55)SE11

Ethan Grinnell · ‎03-02-2018

Switch Ports Model                     SW Version            SW Image                 
------ ----- -----                     ----------            ----------               
*    1 52    WS-C2960S-48FPS-L         15.2(2)E7             C2960S-UNIVERSALK9-M               
*    1 52    C6800IA-48FPD             15.2(3m)E8            C6800IA-UNIVERSALK9-M    
*    4 52    WS-C3650-48PD             03.06.06E             cat3k_caa-universalk9 INSTALL       
*    1 52    WS-C3750-48P              12.2(55)SE10          C3750-IPBASEK9-M

The 3750 has 100M access ports, so we put the AP off of a 1G SFP to bring it back to operation

Leo Laohoo · ‎03-02-2018

Put the AP back to the 2960S and do the following:

1. Command: test cable tdr interface <PORT>;

2. Wait for approximately 5 to 7 seconds;

3. Command: sh cable tdr interface <PORT>; and

4. Post the complete output to #3.

Ethan Grinnell · ‎03-02-2018

Why? The hardware works fine as long as the interface is 1G. For 1G to function it needs all 4 pairs operational, 100M doesn't necessarily. This isn't the first time Cisco has broken connectivity for things that are running less than 1G, it's gotten pretty common lately. It seems that they test a single case and call the software "validated". This is the first time I've seen this on an AP though.

You misunderstand, this post isn't a question. I'm stating how changing software versions broke stuff as a service to other people so they don't have to waste time wondering what happened.

Leo Laohoo · ‎03-02-2018

Ethan Grinnell wrote:
The AP never shows up in CDP on the switch.

I've seen this several times. If this is what I'm suspecting, then it's a faulty cable. The TDR function of the 2960S will be able to determine which pair is faulty and how far away is the fault. If CDP isn't showing up, then I'm suspecting it's Pair B. If the AP will negotiate to 100 Mbps then it's Pair D.

That's just a suggestion. If not, then it's OK.

Ethan Grinnell · ‎03-02-2018

Exactly, if the cable was faulty it wouldn't come up at 1G, but it does and works just fine. It's the opposite of what you are suggesting.

When it's configured for 100M it doesn't pass data, but the link is up. We did have one AP with a wiring issue, but again, this isn't a single AP. There were lots of them on multiple access switches. All of them behaved the same, 100M configuration doesn't work, configuring the port to allow 1G fixes it right away.

t.goldpergel · ‎03-03-2018

We have same problem after 8.3.140 upgrade. No MAC address is shown on interface, but links are OK, however AP says the link speed is 1000. We have 3560v2 switches with 10/100 Mbps access ports. I am sure, the problem is connected with the the OS of 2802I APs.

Ethan Grinnell · ‎03-09-2018

They confirmed it's a bug, https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi14641

The workaround of using a power injector didn't work for me, we had one affected AP that was already on an injector.

t.goldpergel · ‎03-10-2018

+1, here was the same.

8.5.120.0 solved the issue. But, of course "zombie" APs could back to life only after successful upgrade when they were connected to gigabit capable ports.