TAC have managed to replicate this from my ACS backups - and have raised bug CSCtw59271 for me for this issue:
|Random Network Device corruption after upgrade from ACS 5.2 to 5.3.|
After application upgrade from ACS 5.2 to 5.3 some Network Devices experience corruption. (Not all NDs are corrupt, only a few).* Symptom 1: Some Network Devices give the following error on clicking them: ?This System Failure occurred: Has empty AVPAir. Your changes have not been saved. Click ok to return to the list page"
* Symptom 2: Some Network Devices which were working before the upgrade start failing authentication with reason "NDG is not known or has the wrong key". Once the TACACS key is modified/or just edited to be the same key, they start passing authentication.Conditions:
Upgrade of ACS 5.2 to 5.3.Workaround:
Modifies the TACACS+ shared secret of the Network Device, re-enter the same key and save the Network device
No fix - but the workaround is just what I was doing - for a device not authenticating, make any change to the TACACS key and then put it back - and auth works again. For a corrupt device - just delete and re-add. Annoying - but once you know, it's not a big issue.
We upgraded a few weeks ago using the upgrade bundle from 5.2 to 5.3.
The upgrade itself went fairly smoothly - but I had to manually reboot each ACS (primary and secondary) during the upgrade - instead of them rebooting themselves automatically. Had to sit on my hands for an hour to stop me rebooting it in case it really was still doing something - but gave up and rebooted in the end and came back up fine.
Also had some very odd issues with network devices seemingly being 'corrupted' aswell.
I did a fresh install at 5.2 - and used the bulk import to import all our ND's from the CVS file - and I've found (on 5.2 aswell) that some of them look ok - but they don't authenticate (and no messages in the ACS View at all - not even saying eg. wrong tacacs key or IP etc) - until you make some sort of change to the tacacs key - eg. add a '1' onto the end of the string - and then remove it again (back to the same key) - and it suddenly starts working. TAC seem to think this may be 'non unicode characters' issue in the key - but lots of our keys are the same - and I created the CSV file with all devices (eg. copy & paste) - so don' t see how some work and some don't - and I would have thought that the import tool should pick that up anyway?
Since the 5.3 upgrade - I then had some issues with some ND's showing a very odd error when you clicked on them in the network devices list - "This System Failure occurred: Has empty AVPair.. Your changes have not been saved. Click ok to return to the list page" - so you couldn't even view what was in the ND. Each ND needed to be manually deleted - and then re-added - and then worked fine - so I think this is an upgrade ND-corruption issue - but TAC can't replicate or see anything in any backups etc. Not a major issue as we just deleted ND's and re-created - but a bit of a pain.
Anyone else seen any similar issues?
Apart from that - all is good with 5.3. Quite a few little things seem to have been fixed along the way aswell.
I had that same issue with importing from a CVS file. However, it was with 5.2. Very strange indeed.
On a side note, It seems I can no longer authenticate to my child domain. Everything looks fine, including the directory groups and the policies. Pretty annoying.
I had the same issue with the TACACS keys in 5.2. Nothing shows up in the logs for some devices. Copy and pasting the key or even resubmitting and it works.
Can anyone shed some light on whether I can restore the backup made on ACS5.1 to the freshly installed ACS5.3 ?
Secondly, can I have ACS administrators/users athenticate using an external Identity Store, i.e. Microsoft AD ?
I've seen the TAC guys say they've restored a 5.2 backup onto a 5.3 - so I guess it must be possible - but haven't done it myself.
I beleive ACS administrators have to be local ACS users - don't think they can be linked to AD. If it is possible - let me know!
There's also the ADE user (admin) - from the ADE CLI - it looks like you can define a TACACs server for that aswell - but I wasn't sure about the sanity of having the login to the ADE relying on ACS - if you're trying to login to ADE to fix ACS - so I didn't try that myself!
Ok, let's call them ACS users, not administrators. Our client has a strict requirement to have all user ID integrated with just one Identity source which is Microsoft AD. What's ADE user, Rob ?
You can upgrade from ACS 5.1 directly to ACS 5.3. See
Note there have been some issues with log collection starting after upgrade to ACS 5.3, as reported earlier in this thread
There is a patch scheduled to be released in about a week that will resolve one of these issues:
CSCtu15651 ACS view upgrade failure
and it may be worth waiting to upgrade until that patch becomes available
What would be the less painfull and more preferred way to have ACS5.3 running with data and configuration from ACS5.1?
Would it be easier to restore the backup done on ACS5.1 to ACS5.3 or I have to have ACS5.1 freshly installed, restored the backup and then upgrade to ACS5.3 ?
Another thing I ran into while researching on potential methods of upgrade to ACS5.3
But first of all I wanted to see how the restore on ACS5.3 works. To do it I first made a backup to the remote software repository via TFTP and then deleted all configuration for all devices, profiles, policies and users from the server. The next logical step is to try a restore. I followed the above mentioned Cisco's guide and was suprised that it didn't work.
Copying the output from ACS CLI:
acs53/admin# restore acs53-ACS53-111212-1630.tar.gpg repository Backup
Restore requires a reboot to successfully complete. Continue? (yes/no) [yes] ?
find: backup/cars: No such file or directory
% No operating system data found in this backup. Use the 'application option to restore an app-specific backup
Question 1: Why the heck does ACS expects to find any operating system data if it is just the backup of the configuration
Question 2: What is the application option to restore app-specific backup?
These are all application CLI options available:
acs53/admin# application ?
install Install An Application Bundle
remove Uninstall An Application
reset-config Reset application configuration to factory defaults
start Start an Application
stop Stop an Application
upgrade Upgrade An Application Bundle
Question 3: What am I doing wrong ?