03-12-2025 05:44 AM
Hi Everyone, I recently reimaged my DNA Servers (3-node cluster) to 2.3.3.5 after a failed upgrade, plan is to reimage, update packages as per previous then restore backup. The reimage went well, all ok there, I then went to Software management selected all ensuring all versions matched that to what I previously used for 2.3.3.5 then clicked Install. All was good with packages installing until it got to 'Application Visibility Service' where it froze at 90% and it eventually failed after a couple hours. After investigating the below error was present
DEPLOYMENT_ERROR - Exception in task - Maximum wait time 5400 seconds exceeded for the following services to be ready: network-design-service, spf-service-manager-service, apic-em-network-programmer-service, application-recognition-service
When i checked these other dependent services I found the network-visibility, NCP–BASE & automation-core (NCP – SYSTEM) had all been updated to 2.1.720.61514 which is for 2.3.7.7 and the network-visibility service was not starting because of this. Prior to starting the package installs these services were all correctly on 2.1.515.60238 for 2.3.3.5. Unsure why its even touched these.
I've undeployed network-visibility, but unable to undeploy NCP-BASE & automation core, anyone know how to get these 3x services back to 2.1.515.60238 as I think this will fix the problem. Current Package status output below. Just to also mention that these 3x services on 2.1.515.60238 are now also showing in Software management in GUI as available to install but if I attempt to install I get a warning stating "Warning packages not specified’ then nothing happens. many thanks.
btw I do have a TAC Case logged for this too but waiting next steps.
NAME DISPLAY_NAME DEPLOYED AVAILABLE STATUS PROGRESS
---------------------------------------------------------------------------------------------------------------------------------------
access-control-application Access Control Application 2.1.515.60238 2.1.720.61514 DEPLOYED
ai-network-analytics AI Network Analytics 2.9.24.406 3.3.58.21 DEPLOYED
app-hosting Application Hosting 1.9.02209020733 2.3.12406270628 DEPLOYED
application-policy Application Policy 2.1.515.117391 2.1.720.117227 DEPLOYED
application-registry Application Registry 2.1.515.117391 2.1.720.117227 DEPLOYED
application-visibility-service Application Visibility Service 2.1.515.117391 2.1.720.117227 DEPLOYMENT_ERROR - Exception in task - Maximum wait time 5400 seconds exceeded for the following services to be ready: network-design-service, spf-service-manager-service, apic-em-network-programmer-service, application-recognition-service
appsvc-remediation Application and Service Remediation - 1.1.32 NOT_DEPLOYED
assurance Assurance - Base 2.3.3.463 2.3.7.7006 DEPLOYED
automation-core NCP - Services 2.1.720.60128 - DEPLOYED
base-provision-core Automation - Base 2.1.515.60238 2.1.720.61514 DEPLOYED
catalyst-center-api-catalog Catalyst Center API Catalog - 1.15.69 NOT_DEPLOYED
cisco-dna-docs Documentation and Help - 2.1.720.60128 NOT_DEPLOYED
cloud-connectivity-contextual-content Cloud Connectivity - Contextual Content 2.4.1.338 2.11.2.542 DEPLOYED
cloud-connectivity-data-hub Cloud Connectivity - Data Hub 1.8.43 1.15.57 DEPLOYED
cloud-connectivity-tethering Cloud Connectivity - Tethering 2.30.1.72 2.36.1.47 DEPLOYED
cloud-provision-core Cloud Device Provisioning Application 2.1.515.60238 2.1.720.60128 DEPLOYED
command-runner Command Runner 2.1.515.60238 2.1.720.60128 DEPLOYED
device-onboarding Device Onboarding 2.1.515.60238 2.1.720.60128 DEPLOYED
disaster-recovery Disaster Recovery 2.1.515.360031 2.1.720.360029 DEPLOYED
dna-core-apps Network Experience Platform - Core 2.1.515.60238 2.1.720.60128 DEPLOYED
dnac-platform Cisco DNA Center Platform 1.8.1.137 1.15.105.0 DEPLOYED
dnac-search Cisco DNA Center Global Search 1.8.1.10 1.15.1.12 DEPLOYED
endpoint-analytics AI Endpoint Analytics 1.7.702 1.11.1185 DEPLOYED
group-based-policy-analytics Group-Based Policy Analytics 2.3.3.35 2.3.7.140 DEPLOYED
icap-automation Automation - Intelligent Capture 2.1.515.60238 2.1.720.60128 DEPLOYED
image-management Image Management 2.1.515.60238 2.1.720.61514 DEPLOYED
ise-bridge Cisco Identity Services Engine Bridge 2.1.515.450 2.1.716.90701 DEPLOYED
machine-reasoning Machine Reasoning 2.1.515.210125 2.1.720.210013 DEPLOYED
ncp-system NCP - Base 2.1.720.60128 - DEPLOYED
ndp-base-analytics Network Data Platform - Base Analytics 1.8.239 2.3.7.750156 DEPLOYED
ndp-platform Network Data Platform - Core 1.8.396 1.9.5055 DEPLOYED
ndp-ui Network Data Platform - Manager 1.8.244 1.9.5001 DEPLOYED
network-visibility Network Controller Platform - 2.1.720.61514 NOT_DEPLOYED
path-trace Path Trace 2.1.515.60238 2.1.720.61514 DEPLOYED
platform-ui Cisco Catalyst Center UI 1.7.1.339 1.7.7.71 DEPLOYED
rbac-extensions RBAC Extensions 2.1.515.1900002 2.1.720.1900009 DEPLOYED
rogue-management Rogue and aWIPS 2.5.0.20 2.9.0.654 DEPLOYED
sd-access SD Access 2.1.515.60238 2.1.720.61514 DEPLOYED
sensor-assurance Assurance - Sensor 2.3.3.375 2.3.7.7003 DEPLOYED
sensor-automation Automation - Sensor 2.1.515.60238 2.1.720.60128 DEPLOYED
ssa Stealthwatch Security Analytics 2.1.515.1090110 2.1.720.1090100 DEPLOYED
support-services Support Services 2.1.510.880029 2.1.720.880010 DEPLOYED
system System 1.7.769 1.8.232 DEPLOYED
system-commons System Commons 2.1.515.60238 2.1.720.61514 DEPLOYED
system-remediation System Remediation - 1.4.0 NOT_DEPLOYED
umbrella Cisco Umbrella 2.1.515.590102 2.1.720.590098 DEPLOYED
wide-area-bonjour Wide Area Bonjour 2.4.511.75063 2.4.718.75196 DEPLOYED
03-12-2025 09:27 PM
Hi Mark. Yes, please open a TAC case. This issue has been seen before with restoring 2.3.3.5 where it does exactly as you state and updates the latest available package for that service, even when not intended.
As for removing those packages, that's probably not going to happen. Once those packages updated, it updated the schema inside those services and we can't revert that. It's most likely going to be a reimage again, but we need to identify why this happened first.
03-13-2025 02:29 AM
Thanks @maflesch , it is as you state, after a 2 hour TAC call of them attempting to remove packages etc they advised best to reimage again, this time they asked me just to reimage the 1x node and get that back fully operational first with my backup restored then the other 2x can be reimaged and added to cluster in due course. They couldn't find the cause of the problem.
03-14-2025 07:07 AM
Reimaged 1x node to 2.3.3.5 and after doing a 'maglev package status' all base/default Services are on correct versions as expected but concerning that 'available' is showing non-compatible versions available which look to be for 2.3.7.7, wonder if this the cause of my previous attempt and when i installed the additional packages required it automatically update some of these some being core NCP Services. awaiting to hear back form TAC and holding off installing anything, really dont wont to rebuild for a 3rd time ha
03-14-2025 07:25 AM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide