topic Re: ISE patch downloads broken? in Network Access Control

ISE patch downloads broken?

duanetoler — Sat, 19 Aug 2023 01:41:25 GMT

Every ISE patch I download is corrupt in some manner. No matter how I download it, the MD5sum never matches, and the file contents aren't recognized. I opened a TAC case, but the customer has 8x5xNBD support only

I even downloaded via curl directly at command line, bypassing weird browser behaviors. This still didn't change things. Here's a transcript, to show what I mean:

* Download via CURL: 
[det@signet ~]$ curl -k 'https://download-ssc.cisco.com/files/swc/sec/4_SDSP_14768670_1636006591795/1/ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz?...' -o ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 2678M 100 2678M 0 0 9262k 0 0:04:56 0:04:56 --:--:-- 10.0M 

* md5sum:
[det@signet ~]$ md5sum ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
7763e9463d308e7e607fbae0aa05b663

* "file" command:
[det@signet ~]$ file ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz  
ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz: data

* Attempt to decompress, just to check:
[det@signet ~]$ gzip -d ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
gzip: ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz: not in gzip format

* View with tar:
[det@signet ~]$ tar -tzvf ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

I don't get it... what am I missing? I've done this a dozen times in the past, all the same way, and it's been fine.. until now.

Re: ISE patch downloads broken?

balaji.bandi — Sat, 19 Aug 2023 09:24:42 GMT

instead of download in to box, download local computer and try ?

Re: ISE patch downloads broken?

thomas — Sat, 19 Aug 2023 13:44:53 GMT

I just use a web browser on https://cs.co/ise-software and I don't recall the last time I had a problem.

I could not get the file to download with curl and all of "?..." options at the end of the URL so there must be some other sorcery you are doing that allows you to get the download going. I used

 curl --include --location '{url}' -o {filename}

and it redirected me the maximum 50 times then stopped.

Meanwhile, my Patch6 download finished and so I did md5 and shasum but neither matched the published hashes but I did get the same hash as you for MD5! You may want to submit the issue to TAC.

md5 ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
MD5 (ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz) = 7763e9463d308e7e607fbae0aa05b663

rhash --md5 ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
7763e9463d308e7e607fbae0aa05b663 ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz

shasum ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
3c14d576e709a36df2b4b64b6b09491ee91fdcee ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz

I also downloaded 2.7 Patch 9 using my browser since it is the latest and the MD5 also did not match:

md5 ise-patchbundle-2.7.0.356-Patch9-23012204.SPA.x86_64.tar.gz
MD5 (ise-patchbundle-2.7.0.356-Patch9-23012204.SPA.x86_64.tar.gz) = 9a9db9ff11705ae8e26f26b1f0fb155c

rhash --md5 ise-patchbundle-2.7.0.356-Patch9-23012204.SPA.x86_64.tar.gz
9a9db9ff11705ae8e26f26b1f0fb155c ise-patchbundle-2.7.0.356-Patch9-23012204.SPA.x86_64.tar.gz

Re: ISE patch downloads broken?

Leo Laohoo — Sun, 20 Aug 2023 10:56:53 GMT

@thomas wrote:
Meanwhile, my Patch6 download finished and so I did md5 and shasum but neither matched the published hashes but I did get the same hash as you for MD5! You may want to submit the issue to TAC.

In my humble opinion, paying customers should not be made responsible for reporting "documentation bug". This is a Cisco-internal process gone wrong and it should be handled internally because the "priorities" of getting a document bug fixed is very dependent on "who" reported the bug.

In my experience, I have seen document bugs fixed within hours after reporting it to a Cisco staff (not TAC) and I have seen document bugs, until now, remaining unfixed and copied across multiple documents even after TAC admits to the bugs.

Re: ISE patch downloads broken?

Leo Laohoo — Sun, 20 Aug 2023 11:49:35 GMT

Do not waste any time contacting Cisco TAC. Whatever someone did over the weekend is beyond any control of TAC. This is the first (of four) thread that talks about corrupted file getting downloaded. Escalate and reach out to your Cisco Account Manager because this could be something bigger.

The other threads are:

Most importantly:

DO NOT DOWNLOAD ANYTHING until this is sorted.
If the file(s) have already been downloaded, do not make any attempts to unpack them.
I do not even want to spell out what needs to happen if the file(s) have been unpacked.

Re: ISE patch downloads broken?

Kasun Bandara — Sun, 20 Aug 2023 07:07:52 GMT

@duanetoler like @Leo Laohoo said, i have seen many reports of corrupted files today. contact local cisco team and same time cisco TAC.

Re: ISE patch downloads broken?

Leo Laohoo — Sun, 20 Aug 2023 07:23:35 GMT

@Kasun Bandara wrote:
i have seen many reports of corrupted files today. contact local cisco team and same time cisco TAC.

@Kasun Bandara,

If I missed any thread, add those threads I have missed to the three I have enumerated.

Personally, I hope this is nothing but an internal fault and nothing sinister. After watching what happened to DolarWinds and threeCX, I am not taking any chances.

Re: ISE patch downloads broken?

Rich R — Sun, 20 Aug 2023 14:24:00 GMT

I've also just tested - the files on (at least some, maybe all of) the Akamai servers are corrupt!
Download on browser & wget (--no-cache) works fine, but same incorrect hashes both ways.
Given that the file is supposed to have been there since 03-Nov-21 either:
- Akamai has corrupted the files on disk or network or
- Somebody has tampered with the files (no change to file size).
Unfortunately Akamai is not serving the Last Modified header so we can't see if the file has been recently modified.

> gzip -t ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
gzip: ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz: not in gzip format

> file ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz
ise-patchbundle-2.7.0.356-Patch6-21110108.SPA.x86_64.tar.gz: data

So no recognisable file header - looks like the file has just been corrupted on disk or network rather than intentionally altered.

More test results:
AP Mobility Express .tar and .zip file images - both corrupt in exactly the same way.
c900-universalk9-mz.SPA.159-3.M8.bin IOS image - not corrupt
ise-3.2.0.542a.SPA.x86_64.iso - not corrupt
ISE-3.2.0.542a-virtual-SNS3695-2400.ova - not corrupt

I'm going to take a wild guess here and say somebody at Akamai has accidentally or inadvertently turned on some kind of in-line processing of archive (tar/zip/gz) files which is causing them to be corrupted.

So I'd say until further notice assume any files in those formats are corrupt. Not all the posts Leo has linked actually specified what the corrupted files were but all those mentioned seem to fit this profile.

Re: ISE patch downloads broken?

Leo Laohoo — Sun, 20 Aug 2023 12:49:20 GMT

Thanks for the update, @Rich R.

Rated.

Re: ISE patch downloads broken?

Russell Smoak — Mon, 21 Aug 2023 00:43:10 GMT

Hello Richard. Confirming that PSIRT received your direct message. We have engaged the right IT folks and the issue of a being investigated.

-Russell Smoak

VP Cisco Security & Trust

Re: ISE patch downloads broken?

Naresh Ginjupalli — Mon, 21 Aug 2023 00:49:41 GMT

Hi Team,

Cisco ISE Team is aware of the Issue with the download of ISE files and we have root caused the problem and working towards getting it fixed. Will keep this thread update when we are complete on this.

Re: ISE patch downloads broken?

Leo Laohoo — Mon, 21 Aug 2023 04:06:23 GMT

@Naresh Ginjupalli,

Thanks for looking into this.

What about the other files (and not just ISE)?
Can we please get some update if-and-when this issue gets fixed?

Re: ISE patch downloads broken?

Naresh Ginjupalli — Mon, 21 Aug 2023 04:55:44 GMT

We are pleased to inform you that the previously reported issue with ISE files being corrupted while downloading from Cisco site has been successfully resolved. Our dedicated technical team has been hard at work to pinpoint and address the root cause, which was related to Akamai Cloud hosting services.

We understand the inconvenience this issue may have caused and apologize for any disruptions it may have brought to your work. Your satisfaction is our top priority, and we are committed to providing you with a seamless and reliable experience.

Should you encounter any further issues or have any questions, please do not hesitate to reach out to our support team. We are here to assist you and ensure that your experience with our services is as smooth as possible.

Once again, thank you for your patience and understanding during this time. We truly value your business and remain dedicated to delivering the high-quality service you expect from us.

Re: ISE patch downloads broken?

Leo Laohoo — Mon, 21 Aug 2023 05:22:17 GMT

@Naresh Ginjupalli,

What about other files (other than ISE)?

There have been other threads affecting FMC/FTD and CP-8821 firmware.

CP-8821:

https://community.cisco.com/t5/wireless/can-t-verify-checksum-of-cp8821-firmware/m-p/4908468

FMC/FTD:

Re: ISE patch downloads broken?

Naresh Ginjupalli — Mon, 21 Aug 2023 05:33:50 GMT

Other files are also validated and they are working fine as well.

Re: ISE patch downloads broken?

Leo Laohoo — Mon, 21 Aug 2023 06:09:02 GMT

Thank you very much @Naresh Ginjupalli & @Russell Smoak for sorting this out.

Re: ISE patch downloads broken?

Rich R — Mon, 21 Aug 2023 08:06:06 GMT

Ditto - thanks @Russell Smoak & @Naresh Ginjupalli.

Re: ISE patch downloads broken?

duanetoler — Mon, 21 Aug 2023 13:48:46 GMT

The magic I did was: As the file was downloading in my browser, I clicked the downloads button to show the download progress, right clicked on that, did "Copy Address" (or Copy Link, browser-dependent), then pasted that to curl (wrapped in quotes!). The URL is timed with the various parameters, so it only lasts about 10-ish minutes or whatever. [strange i could not type the "sorc..." word; the Reply button is forbidding it. WTF?]

As for your results, that's very interesting! I'm glad to know it's not just me! I did send all of this in a TAC case last night.

Thanks to everyone who has replied and followed-up on this! This is pretty cool, seeing the entire might of The Internet come out to address a problem! 🙂 I saw everyone's updates over the weekend, so thank you @Rich R and @Russell Smoak for contacting TAC and PSIRT. I got an update from my TAC case support who said the problem has been corrected at Akamai. I'll download a patch today and test the MD5SUM for it (unless one of you beats me to it first).

Thank you all!

Re: ISE patch downloads broken?

duanetoler — Mon, 21 Aug 2023 16:03:06 GMT

As others have reported, I have also verified the patch downloads are now working correctly again with matching md5sum and sha512sum.

Re: ISE patch downloads broken?

AnilKumar95946 — Thu, 24 Aug 2023 08:50:39 GMT

Hi duanetoler,

How do we check md5 value on an ISE node. I was able to check on our Linux SFTP server but not able to find the command on Cisco ISE. Could you pls let me know how it works? I have manually uploaded the upgrade file to the ISE node and now want to confirm MD5 value. Ise Version 2.6.