3850 Switch Stack won't auto upgrade newly added switch

[Jim Matuska]

We had 2 3850-48P-S Switches in a stack running version 03.06.06E IOS.  I added a 3rd switch running 03.06.05E and after enabling auto upgrade with the "software auto-upgrade enable" command I rebooted the 3rd switch expecting it to upgrade to 03.06.06E but it reboots back into 03.06.05E.  It is not showing any errors in the stack but I know it is best to keep the stack software at the same version.  I would rather not reboot the whole stack either but need to know why the 3rd switch doesn't auto upgrade or what the easiest way to upgrade it to match software would be without disrupting the first 2 switches in the stack.  What do you think?  

From Show Version

Switch Ports Model              SW Version        SW Image              Mode

------ ----- -----              ----------        ----------            ----
*    1 56    WS-C3850-48P       03.06.06E         cat3k_caa-universalk9 INSTALL
     2 56    WS-C3850-48P       03.06.06E         cat3k_caa-universalk9 INSTALL
     3 56    WS-C3850-48P       03.06.05E         cat3k_caa-universalk9 INSTALL

From show switch

Switch#   Role    Mac Address     Priority Version  State
------------------------------------------------------------
*1       Active   Removed_______     12     V07     Ready
 2       Standby Removed_______     10     V07     Ready
 3       Member Removed_______     8      V03     Ready

[Leo Laohoo]

Hmmm ... 3.6.6, eh? How's the CPU like?
Have a look at this very "popular" bug: CSCuo14511.

[Jim Matuska]

CPU Usage doesn't look bad to me at all. 

Since CPU isn't high and that bug is related to CPU usage I don't see why that bug would apply and prevent auto upgrade of IOS from working.  

3850-Stack#show process cpu
Core 0: CPU utilization for five seconds: 10%; one minute: 11%; five minutes: 11%
Core 1: CPU utilization for five seconds: 1%; one minute: 2%; five minutes: 2%
Core 2: CPU utilization for five seconds: 2%; one minute: 3%; five minutes: 2%
Core 3: CPU utilization for five seconds: 0%; one minute: 3%; five minutes: 2%

[Leo Laohoo]

---

@Jim Matuska wrote:

Since CPU isn't high and that bug is related to CPU usage I don't see why that bug would apply and prevent auto upgrade of IOS from working. 

---

Read between the lines:  Avoid using early release of 3.6.X because of the bug.  Kindly consider upgrading to the latest 3.6.X (if there is a "must" to use 3.6.X).

Check the graph: sh proc cpu h

[Jim Matuska]

Some CPU cores show high historical utilization but others look ok to me (see below).  I'm not stuck with 3.6.x as this stack is just an access switch for a building and changing code shouldn't be an issue as the only features I am using are vtp, and trunking.  

What version would you recomend upgrading to?  

show proc cpu h

History information for system:

    554444444444333334444433333555554444444444444444444433333555
100
 90
 80
 70
 60
 50
 40
 30
 20
 10 **                         *****                         ***
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

         1      1       2       2      1   2   1    3  1       2
    555658545554945556552555455504545559845945544555055855555551
100
 90
 80
 70
 60
 50
 40
 30                                        *        *
 20      *      *       *       *      *   *        *  *       *
 10 *****#* *** # ******#*** ***# * ***#* *# **# ***#**********#
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

    322232223222222232222222222222222222222222222222222222222222222222222222
    044504680361834904169244926583639275848486428326853693489439322864494349
100
 90
 80
 70
 60
 50
 40
 30 *  ** *** * *  **  **   * *** * * *** * **  *  *** **  **  *   **  *   *
 20 ************************************************************************
 10 ************************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

History information for core 0:

    111111111111111111111111111111111111     1111111111111111111
    533333222223333333333222221111111111999992222244444333331111
100
 90
 80
 70
 60
 50
 40
 30
 20 *
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

    111111211111112111111121111113111311121111311211111112111111
    454233243536344344353484725240432125324444534735633244555347
100
 90
 80
 70
 60
 50
 40                                           *
 30                       *      *   *        *  *
 20  *    *  * *  *    *  * * *  *   * * *    *  * **    ****  *
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

    333223243333432222322223233233333333432443226332332423234222223332332223
    531782756044339988579862622814287764728096791419949150823449330617169560
100
 90
 80
 70
 60                                             *
 50        *                            *   *   *
 40 *      **   *     *            **** *  ***  *   *  *    *      *   *
 30 *********************************************************  *  **********
 20 ************************************************************************
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

History information for core 1:

    54444411111     11111555553333355555222222222211111     3333
100
 90
 80
 70
 60
 50
 40
 30
 20
 10 *                    *****     *****
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

          2       2       3      1   8   1   13  2 1     3
    565557966555452447454535656550344565574462865040555570466675
100
 90                                  *
 80                                  *
 70                                  *
 60                                  *
 50                                  *
 40                                  *        *
 30       *               *          *        *          *
 20       *       *       *          *   *    *  *       *
 10 ******#***** **  * * *#*******   #****  **#*** ******# *****
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

    833323239422333553239242333294232243234393324424733383227322332222394323
    587058517666316196498967110541939565548093594490602584662587428903093579
100         *           *                   *                          *
 90 *       *           *       *           *           *              *
 80 *       *           *       *           *       *   *              *
 70 *       *           *       *           *       *   *   *          *
 60 *       *       *   *       *           *       *   *   *          *
 50 *       **     **   * *     *     *   * *       *   *   *          *
 40 ***  *  **    **** ** *     **    **  * * * ** **  **   **         *** *
 30 ****************** *********************************************  ******
 20 ************************************************************************
 10 ************************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

History information for core 2:

    533333     222222222211111333332222222222     33333222221111
100
 90
 80
 70
 60
 50
 40
 30
 20
 10 *
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

          2       2       1      1   1   1    3  1       2
    334355333445453454445326243339444833574443545656434444443665
100
 90
 80
 70
 60
 50
 40                                           *
 30                                           *
 20       *       *              *   *   *    *  *       *
 10     ***    * ** *   * **     *   *  **    # ****     *   ***
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

    432344342342944424325543443423347244823344243442343342443329433223364336
    038635088633638587256343671756363308855941417745278705565945665885952944
100             *                                              *
 90             *                       *                      *
 80             *                       *                      *
 70             *                   *   *                      *       *
 60             *       *           *   *                      *       *   *
 50      * *    * ** *  **  ** *   **  **        *   *    **   **      *   *
 40 *  *** * ** **** *  *** ** * * ** *** **** ****  **** **** ****  ***** *
 30 *********** ********************* ******** *************** *************
 20 ************************************************************************
 10 ************************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

History information for core 3:

    222222222222222222222222222222223333322222222223333322222111
100
 90
 80
 70
 60
 50
 40
 30
 20
 10
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

         2              1              13  1   3    1
    345433333433445335549533343386325467124044397443444952333438
100
 90
 80
 70
 60
 50
 40                                            *
 30                                     *      *
 20      *              *              **      *
 10   *  *        *  ** **      **  * ***  *   #*   *  **      *
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

    424493513243444392231324833334421312442334436422432242232442223933432432
    642661694332076374057197242968977397340063513823677279710539151746635403
100     *           *                                              *
 90     *           *                                              *
 80     *           *       *                                      *
 70     *           *       *                                      *
 60     * *         *       *                   *                  *
 50 *  ** *      ** *      **    **           * **  *   *    *     *  *
 40 * *** *   * *** *  *   **  ****     **  *** **  **  *    **    * **  *
 30 * ***** * *******  * *********** * *** *******  *** **** *** **********
 20 ************************************************************************
 10 ************************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

[paul driver]

Hello

The easiest way would I guess to copy the new iOS off flash1/2 onto sw3

copy flash1:xxx06.03.06e.bin flash3:

verify /md5 flash3:xxx06.03.06e.bin

[Jim Matuska]

That's what I just tried, however I think I may have made things worse as we were running in install mode and when I ran the software install command using the .bin file I received an out of space warning.  Before trying again I did a reboot on the 3rd switch and when it came back up it now shows the switches in bundle mode which makes me think the 3rd switch booted directly to the .bin file.  I'm wondering if I should take and format the flash-3: file sysem for the 3rd switch then re-run the software install on the .bin file for switch 3.  Not sure if this will put the other switches back into install mode though.  Any thoughts?  

Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 56    WS-C3850-48P       BUNDLE UNKNOWN    IMAGE NAME UNKNOWN    BUNDLE
     2 56    WS-C3850-48P       BUNDLE UNKNOWN    IMAGE NAME UNKNOWN    BUNDLE
     3 56    WS-C3850-48P       BUNDLE UNKNOWN    IMAGE NAME UNKNOWN    BUNDLE

[paul driver]

Hello

If you haven’t the space on the flash3; then

I am sure you can make the space

Can you:

dir  flash1:

dir flash3;

If the above didn’t work tou could also just remove thr switch switch from the stack - erase the config and  install the new iOS and then re-add it to the stack 

[Jim Matuska]

dir flash-1:
Directory of flash:/

 7755  -rw-     2097152   Oct 3 2019 13:31:11 -07:00  nvram_config
 7747  -rw-    83293932  Jun 21 2017 20:28:38 -07:00  cat3k_caa-base.SPA.03.06.06E.pkg
 7748  -rw-     3982012  Jun 21 2017 20:28:39 -07:00  cat3k_caa-drivers.SPA.03.06.06E.pkg
 7749  -rw-    33788908  Jun 21 2017 20:28:38 -07:00  cat3k_caa-infra.SPA.03.06.06E.pkg
 7750  -rw-    43074880  Jun 21 2017 20:28:39 -07:00  cat3k_caa-iosd-universalk9.SPA.152-2.E6.pkg
 7751  -rw-    28394380  Jun 21 2017 20:28:38 -07:00  cat3k_caa-platform.SPA.03.06.06E.pkg
 7752  -rw-   111233856  Jun 21 2017 20:28:39 -07:00  cat3k_caa-wcm.SPA.10.2.160.0.pkg
 7753  -rw-        1236  Jun 21 2017 20:28:53 -07:00  packages.conf
23233  drwx        4096  Jun 21 2017 20:48:29 -07:00  dc_profile_dir
 7754  -rw-         676   Oct 7 2019 12:25:51 -07:00  vlan.dat
 7746  -rw-   303772864   Oct 7 2019 11:14:04 -07:00  cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin

1621966848 bytes total (1009029120 bytes free)

dir flash-3:
Directory of flash-3:/

23234  -rw-     2097152   Oct 3 2019 13:31:11 -07:00  nvram_config
23236  -rw-         676   Oct 7 2019 12:00:22 -07:00  vlan.dat
23235  -rw-   303772864   Oct 7 2019 11:14:04 -07:00  cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin
77443  -rw-    83293932   Oct 7 2019 11:24:21 -07:00  cat3k_caa-base.SPA.03.06.06E.pkg
77444  -rw-     3982012   Oct 7 2019 11:24:21 -07:00  cat3k_caa-drivers.SPA.03.06.06E.pkg
77445  -rw-    33788908   Oct 7 2019 11:24:21 -07:00  cat3k_caa-infra.SPA.03.06.06E.pkg
23239  -rw-        1237  Nov 18 2016 20:01:35 -08:00  packages.conf
77446  -rw-    43074880   Oct 7 2019 11:24:21 -07:00  cat3k_caa-iosd-universalk9.SPA.152-2.E6.pkg
77447  -rw-    28394380   Oct 7 2019 11:24:22 -07:00  cat3k_caa-platform.SPA.03.06.06E.pkg
77448  -rw-   111233856   Oct 7 2019 11:24:22 -07:00  cat3k_caa-wcm.SPA.10.2.160.0.pkg
23240  -rw-    83297376  Nov 18 2016 20:01:30 -08:00  cat3k_caa-base.SPA.03.06.05E.pkg
23241  -rw-     4012732  Nov 18 2016 20:01:30 -08:00  cat3k_caa-drivers.SPA.03.06.05E.pkg
23242  -rw-    33786860  Nov 18 2016 20:01:30 -08:00  cat3k_caa-infra.SPA.03.06.05E.pkg
23243  -rw-    43048256  Nov 18 2016 20:01:30 -08:00  cat3k_caa-iosd-universalk9.SPA.152-2.E5.pkg
23244  -rw-    28343180  Nov 18 2016 20:01:30 -08:00  cat3k_caa-platform.SPA.03.06.05E.pkg
   11  drwx        4096   Oct 3 2019 09:52:44 -07:00  dc_profile_dir
23245  -rw-   111260480  Nov 18 2016 20:01:31 -08:00  cat3k_caa-wcm.SPA.10.2.150.0.pkg

1622147072 bytes total (705167360 bytes free)

[paul driver]

Hello

Try to perform a software clean 

# software clean 

[Jim Matuska]

I formatted the flash-3: on the 3rd switch to free up space and then tried to install the 3.6.6E IOS and received an error about unable to determine free memory on switch 3.  I'm trying to figure why that is occuring and where to go from here.  The 3rd switch is not in production yet, but the first 2 switches are, but I still don't want to reload the 3rd switch without any ios as it currently stands.  What can you recomend, is there a way to get rid of the memory error or bypass it and get the 3rd switch to the same version? 

3850-Stack#software install file flash:cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin switch 3
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 3
[3]: % Failed to determine free memory on switch 3, operation aborted

[paul driver]

Hello

Not had that before -however as you ah e now formatted the flash on sw3 and your receiving that error - I would now personally remove it from the stack and re-cable the remaining production switch so you ha e a full stack- ring then with sw3 I would try to  format it agiajn and re-install the iOS 

[Leo Laohoo]

---

@Jim Matuska wrote:
[3]: % Failed to determine free memory on switch 3, operation aborted

---

I've seen this error message before.   This error message appears when the stack has a very high up time. Here's the fix:
1. Make sure the BIN files are in each of the switch members.
2. Reboot the entire stack.
3. Perform a "software clean force switch all".
4. Convert Bundle Mode to Install Mode:  software install file flash-1:IOS_filename.bin new force verb
5. [OPTIONAL] Reboot the stack.

---

@Jim Matuska wrote:
I'm not stuck with 3.6.x as this stack is just an access switch for a building and changing code shouldn't be an issue as the only features I am using are vtp, and trunking.

---

The latest 16.6.X will do fine for a stack doing nothing but Layer 2.
To keep things simpler, STAY AWAY from 16.9.X and later.  This is because Smart Licensing is enabled is enabled starting from 16.9.X and later.