DNA Center Appliance No Longer Accepting Connections via HTTP/HTTPS

[zachartl]

Hello,

We've a DNA Center Appliance no longer accepting connections to the GUI Interface via HTTP/HTTPS. The CIMC Interface is reachable and so is the Maglev shell. I can access and use the maglev configuration wizard and I've tried using that to reinstantiate the configuration parameters but the configuration hangs on the kong certificate installation and aborts. I've queried the maglev package status and get this;

$ maglev package status
ERROR: HTTPSConnectionPool(host='kong-frontend.maglev-system.svc.cluster.local', port=443): Max retries exceeded with url: /api/system/v1/catalog/package?allVer sions=false&repository=main&keepForever=false (Caused by NewConnectionError('<ur llib3.connection.VerifiedHTTPSConnection object at 0x7fabaa5446d0>: Failed to es tablish a new connection: [Errno -3] Temporary failure in name resolution',))

I've ran this openssl command and get this repsonse;

$ echo | openssl s_client -showcerts -servername localhost - connect:443 2>/dev/ null | openssl x509 -inform pem -noout -text
unable to load certificate
139899937572504:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c :701:Expecting: TRUSTED CERTIFICATE

I'm thinking this a bad thing given the other support threads I've reviewed in relation to this symptom. Is there a way to get the data store off of this appliance if need be to either import into another DNA cluster or as a backup and then reimport the data store once the appliance is re-imaged, if it comes to that?

Thank you!

Terry

[Flavio Miranda]

Hi

 Not long ago I worked with cisco TAC in a similar problem where DNAC was no longer respondiing HTTP/HTTPS and they restart service. The problem is, I dont remember which service.

 I imagine that you already reload it.

[balaji.bandi]

what DNAC version ? have you rebooted and tested ?

[edust]

Most likely you have internal certificates that have expired or have lost a "symbolic link". Check the following three commands. If you see anything expired or the commands do not work, the only resolution is to open a case with Cisco TAC.

sudo maglev-config certs info

for f in scheduler.conf controller-manager.conf admin.conf kubelet.conf; do echo "$f"; sudo awk -F ":" '/certificate-authority-data/ {print $2}' /etc/kubernetes/$f | tr -d ' ' |base64 -d | openssl x509 -noout -dates -issuer -issuer_hash;done

for f in scheduler.conf controller-manager.conf admin.conf kubelet.conf; do echo "$f"; sudo awk -F ":" '/client-certificate-data/ {print $2}' /etc/kubernetes/$f | tr -d ' ' |base64 -d | openssl x509 -noout -dates -issuer -issuer_hash -subject;done

[zachartl]

Hello,

I was finally able to circle back around and get to this situation. All 3 commands, yielded output. The first command I ran from the maglev shell, the other two, from the bash shell.

$ sudo maglev-config certs info
--------------------------------------------------------------------------------
certificate start date end date
--------------------------------------------------------------------------------
credentialmanager.pem May 12 21:03:59 2022 GMT May 12 21:03:59 2023 GM T
kong.pem May 12 21:03:59 2022 GMT May 12 21:03:59 2023 GM T
kube-worker-1.pem May 12 21:03:59 2022 GMT May 12 21:03:59 2023 GM T
maglev-registry.pem May 12 21:03:59 2022 GMT May 12 21:03:59 2023 GM T
apiserver.crt Dec 12 14:04:35 2019 GMT Feb 16 02:44:31 2023 GM T
apiserver-kubelet-client.crt Dec 12 14:04:35 2019 GMT Feb 16 02:44:32 2023 GM T
front-proxy-ca.crt Sep 3 02:27:53 2020 GMT Sep 1 02:27:53 2030 GM T
front-proxy-client.crt Sep 3 02:27:53 2020 GMT Feb 16 02:44:32 2023 GM T
admin.conf Dec 12 14:04:35 2019 GMT Feb 16 02:44:33 2023 GM T
scheduler.conf Dec 12 14:04:35 2019 GMT Feb 16 02:44:33 2023 GM T
controller-manager.conf Dec 12 14:04:35 2019 GMT Feb 16 02:44:34 2023 GM T
--------------------------------------------------------------------------------

maglev@maglev-master-1:~$ for f in scheduler.conf controller-manager.conf admin.conf kubelet.conf; do echo "$f"; sudo awk -F ":" '/certificate-authority-data/ {print $2}' /etc/kubernetes/$f | tr -d ' ' |base64 -d | openssl x509 -noout -dates -issuer -issuer_hash;done
scheduler.conf
notBefore=Dec 12 14:04:35 2019 GMT
notAfter=Sep 7 14:04:35 2022 GMT
issuer= /CN=kube-ca
d43a9042
controller-manager.conf
notBefore=Dec 12 14:04:35 2019 GMT
notAfter=Sep 7 14:04:35 2022 GMT
issuer= /CN=kube-ca
d43a9042
admin.conf
notBefore=Dec 12 14:04:35 2019 GMT
notAfter=Sep 7 14:04:35 2022 GMT
issuer= /CN=kube-ca
d43a9042
kubelet.conf
notBefore=Dec 12 14:04:35 2019 GMT
notAfter=Sep 7 14:04:35 2022 GMT
issuer= /CN=kube-ca
d43a9042
maglev@maglev-master-1:~$ for f in scheduler.conf controller-manager.conf admin.conf kubelet.conf; do echo "$f"; sudo awk -F ":" '/client-certificate-data/ {print $2}' /etc/kubernetes/$f | tr -d ' ' |base64 -d | openssl x509 -noout -dates -issuer -issuer_hash -subject;done
scheduler.conf
notBefore=Dec 12 14:04:35 2019 GMT
notAfter=Feb 16 02:44:33 2023 GMT
issuer= /CN=kube-ca
d43a9042
subject= /CN=system:kube-scheduler
controller-manager.conf
notBefore=Dec 12 14:04:35 2019 GMT
notAfter=Feb 16 02:44:34 2023 GMT
issuer= /CN=kube-ca
d43a9042
subject= /CN=system:kube-controller-manager
admin.conf
notBefore=Dec 12 14:04:35 2019 GMT
notAfter=Feb 16 02:44:33 2023 GMT
issuer= /CN=kube-ca
d43a9042
subject= /O=system:masters/CN=kubernetes-admin
kubelet.conf

notBefore=Dec 12 14:04:35 2019 GMT
notAfter=Sep 3 02:27:55 2021 GMT
issuer= /CN=kube-ca
d43a9042
subject= /O=system:nodes/CN=system:node:10.1.253.50
maglev@maglev-master-1:~$

It appears the second kubelet.conf cert may've expired?

I will open a TAC Case. Thank you for having provided these helpful commands.

Terry