Hmmm this just gets weirder.

sokarlsson · ‎05-05-2014

Have received our first couple of 6800ia switches and tried to get them to work on a 6880-X in VSS. All works fine until the 6800IA concludes it has an older image then the one included in 15.1SY(2) that is on the 6880-X.

The feature set on the 6880-X is Advanced Enterprice

This is what I see on the 6880-X

Router#show fex
FEX       FEX              FEX                        FEX
Number    Description      State            Model              Serial
---------------------------------------------------------------------------
101       FEX0101          img_dnld         C6800IA-48TD       FOC1742S2L4
121       FEX0121          img_dnld         C6800IA-48TD       FOC1742S2KK
Router#

Router#
*May 6 06:30:33.528: %SATMGR-SW1-5-IMAGE_VERSION_MISMATCH: Image version mismatch detected for FEX 101: image download initiated. FEX will reboot on completion.

This is what I see on both 6880IA

FEX-101#

Could not buffer tarfile...using multiple downloads
examining image...
%Error opening tftp://192.1.1.1/c6800ia-universalk9-mz.150-2.EX5.bin (Timed out)
%Error opening flash:update/info (No such file or directory)
ERROR: Image is not a valid IOS image archive.

I can not find any documentation on how the upgrade works on CCO. Is the ip address 192.1.1.1 a default loopback on the parent switch or do I need to create it? Been dealing with FEX on Nexus switches before (which is pretty similar) but never had to config anything else then fex association on the parent switch ports and it have worked.

sokarlsson · ‎05-06-2014

Hmmm this just gets weirder. I rebooted the active switch in the VSS pair and after the switchover to the other sup one of the FEX:s came up upgraded. The other one is stuck in rommon and can not be booted using the image on the flash. Opened a TAC case

mathieu.gouret · ‎05-15-2014

hello,

I'm in the same situation as you. I have an IA that does not download the OS since 6800. I rebooted the VSS and now my IA is rommon. Have you had a response from cisco tac?

Mathieu

edv evosoft · ‎09-04-2014

Hi,

look here

http://www.cisco.com/c/en/us/support/docs/field-notices/638/fn63838.html

BR

Eduard Gheorghiu · ‎09-24-2014

Hello

The name of the .bin should be short, like c6800ia-image.bin and copy the file directly under flash: on the IA. If you use the whole path like flash:/update/etc... it won't work (see error below).

Thanks,

Eduard

switch: BOOT=flash:/update/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin

switch: boot

Loading "flash:/update/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin"...flash:/update/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin: magic number mismatch: bad mzip file

Error loading "flash:/update/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin"

Interrupt within 5 seconds to abort boot process.

Boot process failed...

Valter Popeskic · ‎02-02-2017

Hi,

This is the list of steps which worked for me together with solved issue relating the missing boot variable.

It's for later version of IOS but I hope it will help either way.

https://howdoesinternetwork.com/2016/6880-issu

Regards,

Valter

c39213679 · ‎05-15-2014

Me too. I have 5 switches from different stacks stuck in ROMON after the stack doing a 6880 initiated upgrade......

c39213679 · ‎05-18-2014

Followed this method from TAC and got my 6800ia switches working.

1. Check the FEX file and extract it into boot disk:
dir microcode:
Sh fex image bundle location

archive tar /xtract microcode:FEX bootdisk: <<<< will extract .bin file from FEX
You should get a directory that looks like c6800ia-universalk9-mz.150-2.EX5, which if
you look inside of will have a c6800ia-universalk9-mz.150-2.EX5.bin file.

dir bootdisk:

2. Copy c6800ia-universalk9-mz.150-2.EX5.bin to tftp server

3. Delete "c6800ia-universalk9-mz.150-2.EX5.bin" from 6800IA

4. Copy image from TFTP into 6800IA

you should be able to do this via TFTP and the use of the management port on the 6800IA (left-hand top-most port)
Now I connected the management port of the fex and set up an IP address. The update folder I deleted before already.

switch: set IP_ADDR 1.1.1.1/255.255.255.0
switch: set DEFAULT_ROUTER 1.1.1.2

switch: ping 1.1.1.2 <TFTP server ip>
ping 1.1.1.2 with 32 bytes of data ...
Host 192.168.101.55 is alive.

switch: copy tftp://1.1.1.2/c6800ia-universalk9-mz.150-2.EX5.bin flash:update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin

5. Booting by boot command
switch: boot flash:update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

FEX-111#

switch:

switch: set IP_ADDR 1.1.1.1/255.255.255.0

switch: set DEFAULT_ROUTER 1.1.1.2

switch: delete flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin

Are you sure you want to delete "flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin" (y/n)?y

File "flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin" deleted

switch: copy tftp://1.1.1.2/c6800ia-universalk9-mz.150-2.EX5.bin flash:update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin

...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

File "tftp://1.1.1.2/c6800ia-universalk9-mz.150-2.EX5.bin" successfully copied to "flash:update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin"

switch: boot

Loading "flash:/c6800ia-universalk9-mz.150-2.EX2/c6800ia-universalk9-mz.150-2.EX2.bin"...flash:/c6800ia-universalk9-mz.150-2.EX2/c6800ia-universalk9-mz.150-2.EX2.bin: no such file or directory

Error loading "flash:/c6800ia-universalk9-mz.150-2.EX2/c6800ia-universalk9-mz.150-2.EX2.bin"

Interrupt within 5 seconds to abort boot process.

Loading "flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin"...Verifying image flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin.................................Image passed digital signature verification

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ (etc)
File "flash:/update/c6800ia-universalk9-mz.150-2.EX5/c6800ia-universalk9-mz.150-2.EX5.bin" uncompressed and installed, entry point: 0x3000

executing...

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

m-mueller3 · ‎05-06-2014

You have to set the ip tftp source Interface that it works. The internal usage for this is Vlan1006.

sokarlsson · ‎05-16-2014

I have a case open and instructions on how to unpack the FEX image from the 6880-X image and tftp it over but the fex refuse to boot on the manually extracted file as well. Just updated the case.

You can try this as well and see if any one else is lucky. I suspect everyone is running the same 6880-X version since there is only one available for download.

I used the 688-X as tftp-server and connected a cable between mgmt0 on the 6880 and the EOB ethernet port (top left corner) of the 6800ia. I guess one can forget using USB...

####### Instruction start
Copy image via TFTP and boot 6800IA in case the IA-image is corrupted

SETUP TFTP SERVER :

1. a. If you have a Lab TFTP Server :

Connect 6800IA TFTP Port (left hand topmost port) to the TFTP Server .

b. If you don’t have a TFTP Server :

i. Using USB Stick/compact-flash copy the image to bootdisk.

Internally image available at :

/release/150/bin/150-2.EX5/.3DES (Please refer to the compatibility chart above for right image)

bash-3.2$ ls /release/150/bin/150-2.EX5/.3DES/c6800ia*

/release/150/bin/150-2.EX5/.3DES/c6800ia-universalk9-mz.150-2.EX5 <<<< Can be used directly

/release/150/bin/150-2.EX5/.3DES/c6800ia-universalk9-tar.150-2.EX5 <<<< Will need to extract as below.

OR

1. Get the image present on the controller by executing following on Controller

archive tar /xtract microcode:FEX bootdisk:

IA-PARENT#archive tar /xtract microcode:FEX bootdisk:

c6800ia-universalk9-mz.150-2.5.10.MK1D/ (directory)

extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/info (821 bytes)

c6800ia-universalk9-mz.150-2.5.10.MK1D/html/ (directory)

extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/html/foo.html (0 bytes)

extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/c6800ia-universalk9-mz.150-2.5.10.MK1D.bin (15414400 bytes)

extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/dc_default_profiles.txt (71241 bytes)

extracting info (118 bytes)

2. Identify the FEX image to use : It will have a .bin suffix in the directory created above.

IA-PARENT#dir bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1D

Directory of bootdisk:/c6800ia-universalk9-mz.150-2.5.10.MK1D/

23 -rw- 821 Apr 10 2014 13:54:22 +05:00 info

24 drw- 0 Apr 10 2014 13:54:24 +05:00 html

32 -rw- 15414400 Apr 10 2014 13:54:52 +05:00 c6800ia-universalk9-mz.150-2.5.10.MK1D.bin <<<<<====== FEX IMAGE

33 -rw- 71241 Apr 10 2014 13:54:52 +05:00 dc_default_profiles.txt

1981050880 bytes total (348753920 bytes free)

3. Copy the FEX image to the bootdisk: and "RENAME" the file . It should NOT MATCH the directory name or the FEX image name.

IA-PARENT#copy bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1D/c6800ia-universalk9-mz.150-2.5.10.MK1D.bin bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1

Destination filename [c6800ia-universalk9-mz.150-2.5.10.MK1]?

Copy in progress...CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

15414400 bytes copied in 7.488 secs (2058547 bytes/sec)

IA-PARENT#

ii. Make the Controller Box as the TFTP Server like this :-

tftp-server bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1 <<<====RENAMED FILE

iii.Connect the IA TFTP Port with an Interface on Controller Box.

2.

a.Setup the IP ADDRESS and GATEWAY on the 6800IA

switch: IP_ADDR=8.10.8.19/255.255.0.0 (Based on the connection to tftp network)

switch: DEFAULT_ROUTER=8.10.0.1 (IP address of the gateway to reach tftp-server or

IP address of the controller interface)

b. If using Controller as TFTP-SERVER

Setup the IP ADDRESS on the Controller interface

This is same as the DEFAULT_ROUTER configured on 6800IA

4. Check the reachability of the tftp server.

switch: ping 202.153.144.25

ping 202.153.144.25 with 32 bytes of data ...

Host 202.153.144.25 is alive.

5. Copy the image to the 6800IA :

switch: copy tftp://x.x.x.x/<imagename> flash:<imagename>

6. Set the boot variable :

switch: BOOT=flash:/<image-name>

Example image-name = c6800ia-universalk9-mz.150-2.EX4.bin

7. Check if the auto-boot is enabled, if not :

switch: MANUAL_BOOT=no

8. Boot the 6800IA

switch: boot

tgrundbacher · ‎07-02-2014

Same problem here with 15.1(2)SY3 on a 6880-X-LE VSS and a stack of 2x C6800IA-48TDs.

I'm wondering whether I have to do this for every future IA client addition...

tgrundbacher · ‎07-18-2014

Just to let you know, the problem went away by rebooting the whole VSS...

Aninda Chatterjee · ‎09-06-2014

Gentlemen,

I understand that a lot of you are having trouble getting your IA clients upgraded. I am going to try and address your concerns and talk about how this process works.

I am going to take the example of a 6880-X running 15.1(2)SY2 with the IA client already booted up and running with the EX5 code. Now, let's say you're upgrading the 6880-X to 15.1(2)SY3 - this has the EX6 microcode, which implies that on reboot, the IA client needs to be automatically upgraded.

The first thing to understand is that communication between the IA client and the parent happens over a dedicated VLAN interface that is created on boot up for both. On sup2T setups, this is most likely VLAN 1006 and on 6880-X setups this is most likely VLAN 1013.

On the parent, this VLAN is given the IP address 192.168.1.1/24 while on the IA client this is given the IP address 192.168.1.<fex ID>/24.

Let's say the parent has come up in 15.1(2)SY3 now. The IA client first boots up with its older, existing image (EX5) in our case and comes up as FEX0. If you're consoled into the FEX, you'd see the host name to be FEX0 at this stage.

Now it initializes its SVI for communication with the parent and moves over to the configured FEX ID. At this stage, it attempts a download of the microcode from the parent. Once the image is downloaded, the FEX reboots and now it boots with the new image.

Unfortunately, there are several defects around this entire process which can break it leading to situations that y'all run into.

For this reason, here are some troubleshooting tips to narrow down where exactly the issue is.

Troubleshooting:

- Verify that your SVI on both the parent and the IA client is up/up. There is a defect (which is still internal) that causes the SVI on either the parent or the client to be up/down. Due to this, there is no reachability between the parent and the client over this SVI and thus the image never gets downloaded.

Typically, if the SVI is down on the IA client, you'd get socket errors on the FEX console. If the SVI is down on the parent, you'd get timed out errors on the FEX console. This is also why a reboot resolves the issue. When you reboot the VSS, the SVI is reinitialized on bootup and it probably did not run into that race condition issue again and the SVI came up fine.

- Verify that you can ping between the SVIs. On the parent, the SVI is put into a FEX management VRF. So when pinging from the parent, you need to initiate the ping from the VRF. Use ping vrf <vrf name> <IP address of the FEX SVI> for this. On the FEX, the SVI is not under any VRF, so you can directly ping the parent using ping 192.168.1.1.

- Verify that your interfaces in the fabric port-channel are bundled. You might see one in a waiting state some times (w). This is another defect that is fixed in 15.1(2)SY3. However, this is not something you should be extremely concerned about with regards to the download. As long as at least one interface is bundled, it should be okay.

If you still have questions or are continuing to have trouble, please feel free to inbox me here or directly send me an email. Please make sure you specify your SR number (if you have a TAC case open) and I'll take a look at whats happening in the SR.

Regards,

Aninda

relsethagen · ‎10-30-2014

Aninda,

I have 6880 VSS pair running 15.1(2)SY3 and ran into the issue where FEX cannot receive upgraded image due to the SVI for VLAN 1013 being in UP/Down state. I was able to clear the interface VLAN 1013 and bring the SVI into an UP/UP state. The FEX was then able to upgrade without issue. You mentioned that this bug is internal, do you or anyone else know if this or these other FEX upgrade bugs are to be fixed in the November release? These issues leave us with concern about the viability of this platform in any high profile deployments if we need to worry this much about the upgrade process.

Router#sho ip int brief

Interface IP-Address OK? Method Status Protocol

Te1/1/1 unassigned YES NVRAM up up

.

Vlan1 unassigned YES NVRAM administratively down down

Vlan1013 192.1.1.1 YES unset up down

Note the VLAN1013 interface used by the FEX is down

Note you cannot shut no shut this interface

Router#conf t

Enter configuration commands, one per line. End with CNTL/Z.

Router(config)#int vlan 1013

% This interface cannot be modified

Clearing the interface resolved the down state

Router#clear int vlan 1013

Router#sho ip int brief

Interface IP-Address OK? Method Status Protocol

Te1/1/1 unassigned YES NVRAM up up

.

Aninda Chatterjee · ‎10-30-2014

Hi,

This FEX internal SVI is put into a restricted mode which is why you cannot shut/no-shut it. The issue arises from a timing problem due to which the SVI is not initialized in software.

Unfortunately, this specific concern is not fixed in the upcoming November release (15.1(2)SY4).

I can work with you on any other FEX related concerns that you're seeing if you'd like to talk about it.

Regards,

Aninda

6800 ia upgrade loop on 6880-X w 15.1SY