ISE patch downloads broken?

[duanetoler]

Every ISE patch I download is corrupt in some manner.  No matter how I download it, the MD5sum never matches, and the file contents aren't recognized.  I opened a TAC case, but the customer has 8x5xNBD support only

I even downloaded via curl directly at command line, bypassing weird browser behaviors.  This still didn't change things.  Here's a transcript, to show what I mean:

* Download via CURL: 
[det@signet ~]$ curl -k 'https://download-ssc.cisco.com/files/swc/sec/4_SDSP_14768670_1636006591795/1/ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz?...' -o ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 2678M 100 2678M 0 0 9262k 0 0:04:56 0:04:56 --:--:-- 10.0M 

* md5sum:
[det@signet ~]$ md5sum ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
7763e9463d308e7e607fbae0aa05b663

* "file" command:
[det@signet ~]$ file ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz  
ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz: data

* Attempt to decompress, just to check:
[det@signet ~]$ gzip -d ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
gzip: ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz: not in gzip format

* View with tar:
[det@signet ~]$ tar -tzvf ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now  

I don't get it... what am I missing?  I've done this a dozen times in the past, all the same way, and it's been fine.. until now.

[balaji.bandi]

instead of download in to box, download local computer and try ?

[thomas]

I just use a web browser on https://cs.co/ise-software and I don't recall the last time I had a problem.

I could not get the file to download with curl and all of "?..." options at the end of the URL so there must be some other sorcery you are doing that allows you to get the download going. I used

 curl --include --location '{url}' -o {filename} 

and it redirected me the maximum 50 times then stopped.

Meanwhile, my Patch6 download finished and so I did md5 and shasum but neither matched the published hashes but I did get the same hash as you for MD5! You may want to submit the issue to TAC.

md5 ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
MD5 (ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz) = 7763e9463d308e7e607fbae0aa05b663

rhash --md5 ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
7763e9463d308e7e607fbae0aa05b663 ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz

shasum ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
3c14d576e709a36df2b4b64b6b09491ee91fdcee ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz

I also downloaded 2.7 Patch 9 using my browser since it is the latest and the MD5 also did not match:

md5 ise-patchbundle-2.7.0.356-Patch9-23012204.SPA.x86_64.tar.gz
MD5 (ise-patchbundle-2.7.0.356-Patch9-23012204.SPA.x86_64.tar.gz) = 9a9db9ff11705ae8e26f26b1f0fb155c

rhash --md5 ise-patchbundle-2.7.0.356-Patch9-23012204.SPA.x86_64.tar.gz
9a9db9ff11705ae8e26f26b1f0fb155c ise-patchbundle-2.7.0.356-Patch9-23012204.SPA.x86_64.tar.gz

[Leo Laohoo]

---

@thomas wrote:
Meanwhile, my Patch6 download finished and so I did md5 and shasum but neither matched the published hashes but I did get the same hash as you for MD5! You may want to submit the issue to TAC.

---

In my humble opinion, paying customers should not be made responsible for reporting "documentation bug".  This is a Cisco-internal process gone wrong and it should be handled internally because the "priorities" of getting a document bug fixed is very dependent on "who" reported the bug.  

In my experience, I have seen document bugs fixed within hours after reporting it to a Cisco staff (not TAC) and I have seen document bugs, until now, remaining unfixed and copied across multiple documents even after TAC admits to the bugs.

[duanetoler]

The magic I did was:  As the file was downloading in my browser, I clicked the downloads button to show the download progress, right clicked on that, did "Copy Address" (or Copy Link, browser-dependent), then pasted that to curl (wrapped in quotes!).  The URL is timed with the various parameters, so it only lasts about 10-ish minutes or whatever.  [strange i could not type the "sorc..." word; the Reply button is forbidding it. WTF?]

As for your results, that's very interesting! I'm glad to know it's not just me!  I did send all of this in a TAC case last night.

Thanks to everyone who has replied and followed-up on this!  This is pretty cool, seeing the entire might of The Internet come out to address a problem!  I saw everyone's updates over the weekend, so thank you @Rich R and @Russell Smoak for contacting TAC and PSIRT.  I got an update from my TAC case support who said the problem has been corrected at Akamai.  I'll download a patch today and test the MD5SUM for it (unless one of you beats me to it first).

Thank you all!

[Leo Laohoo]

Do not waste any time contacting Cisco TAC.  Whatever someone did over the weekend is beyond any control of TAC.  This is the first (of four) thread that talks about corrupted file getting downloaded.  Escalate and reach out to your Cisco Account Manager because this could be something bigger.  

The other threads are:

• https://community.cisco.com/t5/network-access-control/ise-3-2-patch-3/m-p/4908357
• https://community.cisco.com/t5/network-security/fmc-and-ftd-software-corrupt-images-invalid-md5sums/m-p/4908255
• https://community.cisco.com/t5/network-security/error-when-trying-to-untar-cisco-fmc-virtual-vmware-image-7-2-4/m-p/4908387

Most importantly:  

• DO NOT DOWNLOAD ANYTHING until this is sorted. 
• If the file(s) have already been downloaded, do not make any attempts to unpack them.  
• I do not even want to spell out what needs to happen if the file(s) have been unpacked.  

[Kasun Bandara]

@duanetoler  like @Leo Laohoo said, i have seen many reports of corrupted files today. contact local cisco team and same time cisco TAC.

[Leo Laohoo]

---

@Kasun Bandara wrote:
i have seen many reports of corrupted files today. contact local cisco team and same time cisco TAC.

---

@Kasun Bandara, 

If I missed any thread, add those threads I have missed to the three I have enumerated.  

Personally, I hope this is nothing but an internal fault and nothing sinister.  After watching what happened to DolarWinds and threeCX, I am not taking any chances.  

[Rich R]

I've also just tested - the files on (at least some, maybe all of) the Akamai servers are corrupt! 
Download on browser & wget (--no-cache) works fine, but same incorrect hashes both ways.
Given that the file is supposed to have been there since 03-Nov-21 either:
- Akamai has corrupted the files on disk or network or
- Somebody has tampered with the files (no change to file size).
Unfortunately Akamai is not serving the Last Modified header so we can't see if the file has been recently modified.

> gzip -t ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
gzip: ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz: not in gzip format

> file ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz: data

So no recognisable file header - looks like the file has just been corrupted on disk or network rather than intentionally altered.

More test results:
AP Mobility Express .tar and .zip file images - both corrupt in exactly the same way.
c900-universalk9-mz.SPA.159-3.M8.bin IOS image - not corrupt
ise-3.2.0.542a.SPA.x86_64.iso - not corrupt
ISE-3.2.0.542a-virtual-SNS3695-2400.ova - not corrupt

I'm going to take a wild guess here and say somebody at Akamai has accidentally or inadvertently turned on some kind of in-line processing of archive (tar/zip/gz) files which is causing them to be corrupted.

So I'd say until further notice assume any files in those formats are corrupt.  Not all the posts Leo has linked actually specified what the corrupted files were but all those mentioned seem to fit this profile.

[Leo Laohoo]

Thanks for the update, @Rich R. 

Rated.

[Russell Smoak]

Hello Richard.  Confirming that PSIRT received your direct message.  We have engaged the right IT folks and the issue of a being investigated.   

-Russell Smoak

VP Cisco Security & Trust

[Naresh Ginjupalli]

Hi Team,

Cisco ISE Team is aware of the Issue with the download of ISE files and we have root caused the problem and working towards getting it fixed. Will keep this thread update when we are complete on this. 

[Leo Laohoo]

@Naresh Ginjupalli, 

Thanks for looking into this. 

1. What about the other files (and not just ISE)?  
2. Can we please get some update if-and-when this issue gets fixed?

[Naresh Ginjupalli · Accepted solution]

We are pleased to inform you that the previously reported issue with ISE files being corrupted while downloading from Cisco site has been successfully resolved. Our dedicated technical team has been hard at work to pinpoint and address the root cause, which was related to Akamai Cloud hosting services.

We understand the inconvenience this issue may have caused and apologize for any disruptions it may have brought to your work. Your satisfaction is our top priority, and we are committed to providing you with a seamless and reliable experience.

Should you encounter any further issues or have any questions, please do not hesitate to reach out to our support team. We are here to assist you and ensure that your experience with our services is as smooth as possible.

Once again, thank you for your patience and understanding during this time. We truly value your business and remain dedicated to delivering the high-quality service you expect from us.

[Leo Laohoo]

@Naresh Ginjupalli, 

What about other files (other than ISE)?  

There have been other threads affecting FMC/FTD and CP-8821 firmware.  

CP-8821:

• https://community.cisco.com/t5/wireless/can-t-verify-checksum-of-cp8821-firmware/m-p/4908468

FMC/FTD:  

• https://community.cisco.com/t5/network-security/fmc-and-ftd-software-corrupt-images-invalid-md5sums/m-p/4908255
• https://community.cisco.com/t5/network-security/error-when-trying-to-untar-cisco-fmc-virtual-vmware-image-7-2-4/m-p/4908387