6800 ia upgrade loop on 6880-X w 15.1SY

[sokarlsson]

Have received our first couple of 6800ia switches and tried to get them to work on a 6880-X in VSS. All works fine until the 6800IA concludes it has an older image then the one included in 15.1SY(2) that is on the 6880-X.

The feature set on the 6880-X is Advanced Enterprice

This is what I see on the 6880-X

Router#show fex
FEX       FEX              FEX                        FEX                  
Number    Description      State            Model              Serial      
---------------------------------------------------------------------------
101       FEX0101          img_dnld         C6800IA-48TD       FOC1742S2L4
121       FEX0121          img_dnld         C6800IA-48TD       FOC1742S2KK
Router#

Router#
*May  6 06:30:33.528: %SATMGR-SW1-5-IMAGE_VERSION_MISMATCH: Image version mismatch detected for FEX 101: image download initiated.  FEX will reboot on completion.

This is what I see on both 6880IA

FEX-101#

 Could not buffer tarfile...using multiple downloads
examining image...
%Error opening tftp://192.1.1.1/c6800ia-universalk9-mz.150-2.EX5.bin (Timed out)
%Error opening flash:update/info (No such file or directory)
ERROR: Image is not a valid IOS image archive.

I can not find any documentation on how the upgrade works on CCO. Is the ip address 192.1.1.1 a default loopback on the parent switch or do I need to create it? Been dealing with FEX on Nexus switches before (which is pretty similar) but never had to config anything else then fex association on the parent switch ports and it have worked.

[dean holroyd]

I have had this problem. I opened a TAC case via our support provider for this, who could only point me to this thread!

A few weeks ago when testing the 6807-XL with FEX, I managed to reboot the device before the FEX image downloaded, which wiped off the image from the IA clients, and I had to go through the switch: 'rommon' recovery procedure as others have pointed out, so I won't repeat that

However, here is a another insight. Yesterday, I installed two 6880-X devices in a remote location. I only had console access to the devices, and none of the management interfaces were connected. I found I was in the booting loop

6880-X#sh fex
FEX       FEX              FEX                        FEX                  
Number    Description      State            Model              Serial      
---------------------------------------------------------------------------
101       FEX0101          img_dnld         C6800IA-48TD       FCWXXXXXXX
102       FEX0102          img_dnld         C6800IA-48TD       FCWXXXXXXX 

Dec  1 04:01:05.703: fex_chassis: Reg Response - version mismatch
Dec  1 04:01:05.703:     fex_chassis (101)189c.5d66.8e80: during state registered, got event 11(version_mismatch)
Dec  1 04:01:05.703: @@@ fex_chassis (101)189c.5d66.8e80: registered -> version_mismatch
Dec  1 04:01:05.703: image incompatible: image to download:tftp://192.1.1.1/c6800ia-universalk9-mz.150-2.EX6.bin
Dec  1 04:01:05.717: fexx_cm:: async svi op 2 (1006, 192.1.1.101, 255.255.255.0) completed, status:success(1)
Dec  1 04:01:05.717: %SATMGR-5-IMAGE_DOWNLOAD_START: Download of image to FEX started.  FEX will reboot on completion.
Dec  1 04:02:21.823:     fex_chassis (101)189c.5d66.8e80: during state version_mismatch, got event 12(download_fail)
Dec  1 04:02:21.823: @@@ fex_chassis (101)189c.5d66.8e80: version_mismatch -> discovery
Dec  1 04:02:21.823: fex_chassis: Image download fail. 101

From previous experience, I know that if you have CoPP enabled, then you must allow TFTP connectivity for 192.1.1.0/24 as the FEX uses to that communicate with the chassis for the download. I already had this, but removed CoPP just in case. 

I thought that I am going to have to upgrade manually, but had no mgmt or console access to the FEX, so I did extracted the FEX image from the microcode as in the rommon recovery procedure, and turned the chassis into a TFTP server for the FEX image. I then telnet'd to the FEX via its management VRF (which is only possible if the FEX is still alive and not in rommon

6880-X(config)#archive tar /xtract microcode:FEX bootdisk:
6880-X(config)#copy bootdisk:/c6800ia-universalk9-mz.150-2.EX6/c6800ia-universalk9-mz.150-2.EX6.bin bootdisk:6800IA-EX6.bin
6880-X(config)#tftp-server bootdisk:6800IA-EX6.bin
6880-X(config)#ip tftp source-interface Vlan1006
6880-X(config)#^Z
6880-X#telnet 192.1.1.101 /vrf __SVI_fex_mgmt_vrf__
Trying 192.1.1.101 ... Open

User Access Verification

Password: 
FEX-101>en
Password: 
FEX-101#copy tftp://192.1.1.1/6800IA-EX6.bin flash:c6800ia-universalk9-mz.150-2.EX6.bin
Destination filename [c6800ia-universalk9-mz.150-2.EX6.bin]? 
Accessing tftp://192.1.1.1/6800IA-EX6.bin...
Loading 6800IA-EX6.bin from 192.1.1.1 (via Vlan1006): !!!!!!!!!!!

The image downloaded, and I was just about to verify it, when the FEX rebooted itself. I thought I had done something wrong, but after a few minutes, the FEX came back on line. I had another FEX which I hadn't even touched yet, and I saw this was also online.

#sh fex
FEX       FEX              FEX                        FEX                  
Number    Description      State            Model              Serial      
---------------------------------------------------------------------------
101       FEX0101          online           C6800IA-48TD       FCW1XXXXXXX
102       FEX0102          online           C6800IA-48TD       FCW1XXXXXXX

From this, I gathered that when I configured 'ip tftp source-interface Vlan1006' which is the FEX vlan interface, the auto download actually worked and beat me to the FEX software upgrade.

I tried this on another chassis which had the same problem, and changed the TFTP source interface to the FEX vlan, and after 20 mins the FEXs were on line

[sokarlsson]

Very good catch Dean!

[andrewswanson]

Many thanks for the post. I had a FEX stuck in the "%SATMGR-SW2-5-IMAGE_VERSION_MISMATCH" loop after upgrading a C6807-XL from 151-2.SY1 to 151-2.SY4a. I had changed the "ip tftp source-interface" setting a few weeks before. I changed it to "ip tftp source-interface vlan 1006" and the FEX upgraded successfully.

Thanks again
Andy

[cyyuen2000]

Thanks for the help. You save my day.

[Cristian Ruta]

hi,

I also had this problem on upgrading from 151-2.SY3 to152-1.SY0a

The loop was every 3 minutes: SATMGR-SW2-5-IMAGE_VERSION_MISMATCH: Image version mismatch detected for FEX 105: image download initiated.  FEX will reboot on completion.

On sh fex: 105       FEX0105          img_dnld         C6800IA-48FPD  

but there was no new file on the flash, also no traffic on the uplinks.

I copied the extracted image through tftp manually on every fex, restarted them and it finally worked.

Next time i will copy the image on the fex before starting the upgrade.

Thanks for sharing this solution!