PKI Trustpoint - NDES Autoenrollment issues (-> FlexVPN)

[VirtualAccess36632]

Hi,
I am experiencing very unexpected result while trying to auto-enroll certificates with CISCO and NDES CA (Enterprise) (configured with OTP - standard)

1. Noticed that my IOS-XE (universalk9.16.12.01) is acting when configuring auto-enroll 90. To my understanding it should attempt to auto-enroll at 90% of certificates life time (which is configured to 1 day with template)

What it actually does it attempt to auto-enroll almost immediately after 

```

Next enrollment attempt:
17 seconds
* Configuration will not be saved after enrollment *
```

2. When I remove auto-enroll option, it seem to work fine but ... It never saves client certificate, hence VPN is down.
I can only see CA certificate within that trustpoint ( using wri mem / copy run start makes no difference) 

3. for a very strange reason I see that Cisco always duplicates requests and in result two certificates are generated on NDES side 

4. CRL revocation seems to be there only for decoration purposes, no requests - counters are not increasing  
   
This configuration works absolutely fine with fixed length NDES passwords, Cisco collects it's client cert, its available via sh cry cert - It still generates 2 certificates but VPN works

Can someone assist me with tested config or explain where the problem is. Other Linux clients seems to work fine with this setup. 

This looks very similar to this bug https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvb96706

Here are relevant configs and logs 

IR1101#sh run | sec trustpoint NDES-233
crypto pki trustpoint NDES-233
enrollment retry count 3
enrollment retry period 5
enrollment mode ra
enrollment url http://ndes.vatest.com:80/certsrv/mscep/mscep.dll
usage ike
serial-number none
fqdn ndes.vatest.com
ip-address 10.100.233.248
subject-name C=IE, CN=NDES-MSCEP-RA
subject-alt-name ndes.vatest.com
revocation-check crl
rsakeypair NDES-233 2048 2048
auto-enroll 90


IR1101(ca-trustpoint)#auto-enroll ?
<0-100> renewal percentage
regenerate Regenerate keys on re-enrollment
<cr> <cr>

IR1101(ca-trustpoint)#auto-enroll 90
IR1101(ca-trustpoint)#
IR1101(ca-trustpoint)#
IR1101(ca-trustpoint)#
IR1101(ca-trustpoint)#ex
IR1101(config)#

IR1101(config)#cry pki enroll NDES-233
% Start certificate enrollment ..
% Create a challenge password. You will need to verbally provide this
password to the CA Administrator in order to revoke your certificate.
For security reasons your password will not be saved in the configuration.
Please make a note of it.

Password:
Re-enter password:

% The subject name in the certificate will include: C=IE, CN=NDES-MSCEP-RA
% The subject name in the certificate will include: ndes.vatest.com
% The IP address in the certificate is 10.100.233.248

Request certificate from CA? [yes/no]: yes
% Certificate request sent to Certificate Authority
% The 'show crypto pki certificate verbose NDES-233' commandwill show the fingerprint.

IR1101(config)#
IR1101(config)#do sh cry pki trustpoi st | sec Trustpoint NDES-233
Trustpoint NDES-233:
Issuing CA certificate configured:
Subject Name:
cn=NDES-NDES-CA,dc=NDES,dc=vatest,dc=com
Fingerprint MD5: 63B155A9 68556345 2EF5C4F7 83B70377
Fingerprint SHA1: 694FA451 3EDD38B3 A1B6364C C2D2E5D5 E760D235
Last enrollment status: Granted
Next enrollment attempt:
17 seconds   <- this happens when I add auto-enroll option to trustpoint ********************* and it fails since it issues another SCEP chalengepassowd with already used OTP password *********
* Configuration will not be saved after enrollment *
State:
Keys generated ............. Yes (Usage Keys, non-exportable)
Issuing CA authenticated ....... Yes
Certificate request(s) ..... Yes

LOGs (after issuing crypto pki enroll) 
Dec 7 18:13:37.637: %PKI-6-CERT_ENROLL_MANUAL: Manual enrollment for trustpoint NDES-233
Dec 7 18:13:52.328: CRYPTO_PKI_SCEP: Client sending GetCACert request
Dec 7 18:13:52.334: CRYPTO_PKI_SCEP: Client received CA and RA certificate
Dec 7 18:13:52.381: %PKI-6-CSR_FINGERPRINT:
CSR Fingerprint MD5 : EC9D32D7F758B66F89F848FB4EB4EEFE
CSR Fingerprint SHA1: 05509F3BF032DD718B0084FF04CB791A9B4C0F83
Dec 7 18:13:52.381: CRYPTO_PKI: Signature Certificate Request Fingerprint MD5: EC9D32D7 F758B66F 89F848FB 4EB4EEFE
Dec 7 18:13:52.382: CRYPTO_PKI: Signature Certificate Request Fingerprint SHA1: 05509F3B F032DD71 8B0084FF 04CB791A 9B4C0F83
Dec 7 18:13:52.465: CRYPTO_PKI_SCEP: Client sending PKCSReq
Dec 7 18:13:52.532: %PKI-6-CSR_FINGERPRINT:
CSR Fingerprint MD5 : 08DD9CC70BEFA3543F076CB1A6CACFC7
CSR Fingerprint SHA1: DEA5BBC86ADBD61FF1C8CD3528968997EF231202
Dec 7 18:13:52.532: CRYPTO_PKI: Encryption Certificate Request Fingerprint MD5: 08DD9CC7 0BEFA354 3F076CB1 A6CACFC7
Dec 7 18:13:52.533: CRYPTO_PKI: Encryption Certificate Request Fingerprint SHA1: DEA5BBC8 6ADBD61F F1C8CD35 28968997 EF231202
Dec 7 18:13:52.619: CRYPTO_PKI_SCEP: Client sending PKCSReq
Dec 7 18:13:52.650: CRYPTO_PKI_SCEP: Client received CertRep - GRANTED (057ABECEE71D90B3F6F3549A193EC7F6)
Dec 7 18:13:52.708: CRYPTO_PKI_SCEP: Client received CertRep - GRANTED (7A8A298927008B4D3CF2A49EC0F2F798)
Dec 7 18:14:18.051: %PKI-6-CERT_ENROLL_AUTO: Auto initial enrollment for trustpoint NDES-233
Dec 7 18:14:18.052: CRYPTO_PKI_SCEP: Client sending GetCACert request
Dec 7 18:14:18.058: CRYPTO_PKI_SCEP: Client received CA and RA certificate
Dec 7 18:14:18.105: %PKI-6-CSR_FINGERPRINT:
CSR Fingerprint MD5 : 1226B5E3B1155B7A66774FEC573A968E
CSR Fingerprint SHA1: 676E09BAE410CF15B73AF01A6929B90AB5579955
Dec 7 18:14:18.105: CRYPTO_PKI: Signature Certificate Request Fingerprint MD5: 1226B5E3 B1155B7A 66774FEC 573A968E
Dec 7 18:14:18.106: CRYPTO_PKI: Signature Certificate Request Fingerprint SHA1: 676E09BA E410CF15 B73AF01A 6929B90A B5579955
Dec 7 18:14:18.189: CRYPTO_PKI_SCEP: Client sending PKCSReq
Dec 7 18:14:18.251: %PKI-6-CSR_FINGERPRINT:
CSR Fingerprint MD5 : 887D951E2F3D6A33C5215C3B2C518907
CSR Fingerprint SHA1: 5D696D674394FA4BBAD816733592B4BE314F68A5
Dec 7 18:14:18.251: CRYPTO_PKI: Encryption Certificate Request Fingerprint MD5: 887D951E 2F3D6A33 C5215C3B 2C518907
Dec 7 18:14:18.252: CRYPTO_PKI: Encryption Certificate Request Fingerprint SHA1: 5D696D67 4394FA4B BAD81673 3592B4BE 314F68A5
Dec 7 18:14:18.335: CRYPTO_PKI_SCEP: Client sending PKCSReq
Dec 7 18:14:18.368: CRYPTO_PKI_SCEP: Client received CertRep - REJECTED (057ABECEE71D90B3F6F3549A193EC7F6)
Dec 7 18:14:18.368: %PKI-6-CERTREJECT: Certificate enrollment request was rejected by Certificate Authority for Trustpoint NDES-233
Dec 7 18:14:18.387: CRYPTO_PKI_SCEP: Client received CertRep - REJECTED (7A8A298927008B4D3CF2A49EC0F2F798)
Dec 7 18:14:18.387: %PKI-6-CERTREJECT: Certificate enrollment request was rejected by Certificate Authority for Trustpoint NDES-233

Edit:

More logs, which confirms the above

CISCO is unable to save router cert and re:enrolment is kicking in

HTTP/1.1 200 OK
Content-Type: application/x-x509-ca-ra-cert
Server: Microsoft-IIS/10.0
Date: Wed, 08 Dec 2021 11:34:31 GMT
Connection: close
Content-Length: 4123

Content-Type indicates we have received CA and RA certificates.

Dec 8 11:34:31.582: CRYPTO_PKI_SCEP: Client received CA and RA certificate
Dec 8 11:34:31.582: CRYPTO_PKI:crypto_process_ca_ra_cert(trustpoint=NDES-233)

Dec 8 11:34:31.585: The PKCS #7 message contains 4 certificates.
Dec 8 11:34:31.586: CRYPTO_PKI:crypto_pkcs7_insert_ra_certs found RA certs

Dec 8 11:34:31.590: CRYPTO_PKI:crypto_pkcs7_insert_ra_certs found RA certs

Dec 8 11:34:31.592: CRYPTO_PKI: Capabilites already obtained CA_CAP_POST_ACCEPTED CA_CAP_RENEWAL CA_CAP_SHA_1 CA_CAP_SHA_256 CA_CAP_SHA_512
Dec 8 11:34:31.593: CRYPTO_PKI: transaction CRYPTO_REQ_CERT completed
Dec 8 11:34:31.593: CRYPTO_PKI: status:
Dec 8 11:34:31.631: %PKI-6-CSR_FINGERPRINT:
CSR Fingerprint MD5 : 90A1F5011D32D0EA30AA925C8EC7AE74
CSR Fingerprint SHA1: 8202CC323A29FDDF5C8B614315F46AE58D8BF107
Dec 8 11:34:31.631: CRYPTO_PKI: Signature Certificate Request Fingerprint MD5: 90A1F501 1D32D0EA 30AA925C 8EC7AE74
Dec 8 11:34:31.632: CRYPTO_PKI: Signature Certificate Request Fingerprint SHA1: 8202CC32 3A29FDDF 5C8B6143 15F46AE5 8D8BF107
Dec 8 11:34:31.633: PKI:PKCS7 to issuer cn=NDES-NDES-CA,dc=NDES,dc=vatest,dc=com serial
48 00 00 00 05 FF 27 3B AE F5 60 F0 BA 00 00 00
00 00 05

Dec 8 11:34:31.637: CRYPTO_PKI: Deleting cached key having key id 93
Dec 8 11:34:31.637: CRYPTO_PKI: Attempting to insert the peer's public key into cache
Dec 8 11:34:31.637: CRYPTO_PKI:Peer's public inserted successfully with key id 94
Dec 8 11:34:31.639: CRYPTO_PKI: Expiring peer's cached key with key id 94
Dec 8 11:34:31.639: PKI: Trustpoint NDES-233 has no router cert and loaded
Dec 8 11:34:31.639: PKI: Signing pkcs7 with NDES-233 trustpoint temp self-signed cert
Dec 8 11:34:31.717: CRYPTO_PKI_SCEP: Client sending PKCSReq
Dec 8 11:34:31.717: CRYPTO_PKI: locked trustpoint NDES-233, refcount is 1
Dec 8 11:34:31.719: CRYPTO_PKI: http connection opened
Dec 8 11:34:31.719: CRYPTO_PKI: Sending HTTP message

Dec 8 11:34:31.719: CRYPTO_PKI: Reply HTTP header:
HTTP/1.0
Host: ndes.vatest.com

Dec 8 11:34:31.737: CRYPTO_PKI: unlocked trustpoint NDES-233, refcount is 0
Dec 8 11:34:31.775: %PKI-6-CSR_FINGERPRINT:
CSR Fingerprint MD5 : 71488938DF6D9BE840BC703783489916
CSR Fingerprint SHA1: 9340573AEDA48E2BE8C70DE7919F388E40EBC120
Dec 8 11:34:31.776: CRYPTO_PKI: Encryption Certificate Request Fingerprint MD5: 71488938 DF6D9BE8 40BC7037 83489916
Dec 8 11:34:31.776: CRYPTO_PKI: Encryption Certificate Request Fingerprint SHA1: 9340573A EDA48E2B E8C70DE7 919F388E 40EBC120
Dec 8 11:34:31.778: PKI:PKCS7 to issuer cn=NDES-NDES-CA,dc=NDES,dc=vatest,dc=com serial
48 00 00 00 05 FF 27 3B AE F5 60 F0 BA 00 00 00
00 00 05

Dec 8 11:34:31.782: CRYPTO_PKI: Deleting cached key having key id 94
Dec 8 11:34:31.782: CRYPTO_PKI: Attempting to insert the peer's public key into cache
Dec 8 11:34:31.782: CRYPTO_PKI:Peer's public inserted successfully with key id 95
Dec 8 11:34:31.784: CRYPTO_PKI: Expiring peer's cached key with key id 95
Dec 8 11:34:31.784: PKI: Trustpoint NDES-233 has no router cert and loaded
Dec 8 11:34:31.784: PKI: Signing pkcs7 with NDES-233 trustpoint temp self-signed cert
Dec 8 11:34:31.865: CRYPTO_PKI_SCEP: Client sending PKCSReq
Dec 8 11:34:31.865: CRYPTO_PKI: locked trustpoint NDES-233, refcount is 1
Dec 8 11:34:31.868: CRYPTO_PKI: http connection opened
Dec 8 11:34:31.868: CRYPTO_PKI: Sending HTTP message

Dec 8 11:34:31.868: CRYPTO_PKI: Reply HTTP header:
HTTP/1.0
Host: ndes.vatest.com

Dec 8 11:34:31.893: CRYPTO_PKI: unlocked trustpoint NDES-233, refcount is 0
Dec 8 11:34:31.893: CRYPTO_PKI: locked trustpoint NDES-233, refcount is 1
Dec 8 11:34:31.893: CRYPTO_PKI: locked trustpoint NDES-233, refcount is 2
Dec 8 11:34:31.894: CRYPTO_PKI: unlocked trustpoint NDES-233, refcount is 1
Dec 8 11:34:31.894: CRYPTO_PKI: received msg of 2508 bytes
Dec 8 11:34:31.894: CRYPTO_PKI: Reply HTTP header:
HTTP/1.1 200 OK
Content-Type: application/x-pki-message
Server: Microsoft-IIS/10.0
Date: Wed, 08 Dec 2021 11:34:31 GMT
Connection: close
Content-Length: 2342

Dec 8 11:34:31.900: CRYPTO_PKI: Prepare global revocation service providers
Dec 8 11:34:31.903: CRYPTO_PKI: Deleting cached key having key id 95
Dec 8 11:34:31.903: CRYPTO_PKI: Attempting to insert the peer's public key into cache
Dec 8 11:34:31.903: CRYPTO_PKI:Peer's public inserted successfully with key id 96
Dec 8 11:34:31.904: CRYPTO_PKI: Expiring peer's cached key with key id 96
Dec 8 11:34:31.910: CRYPTO_PKI: Remove global revocation service providers
Dec 8 11:34:31.910: The PKCS #7 message has 1 verified signers.
Dec 8 11:34:31.910: signing cert: issuer cn=NDES-NDES-CA,dc=NDES,dc=vatest,dc=com serial 4800042D43EDB53E1A175000004
Dec 8 11:34:31.911: Signed Attributes:

Dec 8 11:34:31.911: CRYPTO_PKI_SCEP: Client received CertRep - GRANTED (057ABECEE71D90B3F6F3549A193EC7F6)
Dec 8 11:34:31.911: CRYPTO_PKI: status = 100: certificate is granted
Dec 8 11:34:31.951: The PKCS #7 message contains 1 certs and 0 crls.
Dec 8 11:34:31.953: CRYPTO_PKI: can not find router cert in certrep
Dec 8 11:34:31.953: CRYPTO_PKI: can not find crl in certrep
Dec 8 11:34:31.953: CRYPTO_PKI: unlocked trustpoint NDES-233, refcount is 0

Dec 8 11:34:41.953: CRYPTO_PKI: locked trustpoint NDES-233, refcount is 1
Dec 8 11:34:41.953: CRYPTO_PKI: locked trustpoint NDES-233, refcount is 2
Dec 8 11:34:41.953: CRYPTO_PKI: unlocked trustpoint NDES-233, refcount is 1
Dec 8 11:34:41.953: CRYPTO_PKI: received msg of 2508 bytes
Dec 8 11:34:41.954: CRYPTO_PKI: Reply HTTP header:
HTTP/1.1 200 OK
Content-Type: application/x-pki-message
Server: Microsoft-IIS/10.0
Date: Wed, 08 Dec 2021 11:34:31 GMT
Connection: close
Content-Length: 2342

Dec 8 11:34:41.960: CRYPTO_PKI: Prepare global revocation service providers
Dec 8 11:34:41.963: CRYPTO_PKI: Deleting cached key having key id 96
Dec 8 11:34:41.963: CRYPTO_PKI: Attempting to insert the peer's public key into cache
Dec 8 11:34:41.963: CRYPTO_PKI:Peer's public inserted successfully with key id 97
Dec 8 11:34:41.964: CRYPTO_PKI: Expiring peer's cached key with key id 97
Dec 8 11:34:41.969: CRYPTO_PKI: Remove global revocation service providers
Dec 8 11:34:41.970: The PKCS #7 message has 1 verified signers.
Dec 8 11:34:41.970: signing cert: issuer cn=NDES-NDES-CA,dc=NDES,dc=vatest,dc=com serial 4800042D43EDB53E1A175000004
Dec 8 11:34:41.971: Signed Attributes:

Dec 8 11:34:41.971: CRYPTO_PKI_SCEP: Client received CertRep - GRANTED (7A8A298927008B4D3CF2A49EC0F2F798)
Dec 8 11:34:41.971: CRYPTO_PKI: status = 100: certificate is granted
Dec 8 11:34:42.010: The PKCS #7 message contains 1 certs and 0 crls.
Dec 8 11:34:42.012: CRYPTO_PKI: can not find router cert in certrep
Dec 8 11:34:42.012: CRYPTO_PKI: can not find crl in certrep
Dec 8 11:34:42.012: CRYPTO_PKI: All enrollment requests completed for trustpoint NDES-233.
Dec 8 11:34:42.012: CRYPTO_PKI: All enrollment requests completed for trustpoint NDES-233.
Dec 8 11:34:42.012: CRYPTO_PKI: All enrollment requests completed for trustpoint NDES-233.
Dec 8 11:34:42.012: %PKI-6-AUTOSAVE: Running configuration saved to NVRAM[OK]
Dec 8 11:34:42.192: CRYPTO_PKI: locked trustpoint SLA-TrustPoint, refcount is 1
Dec 8 11:34:42.192: CRYPTO_PKI: unlocked trustpoint SLA-TrustPoint, refcount is 0
Dec 8 11:34:42.192: CRYPTO_PKI: locked trustpoint TP-self-signed-3199919222, refcount is 1
Dec 8 11:34:42.193: CRYPTO_PKI: unlocked trustpoint TP-self-signed-3199919222, refcount is 0
Dec 8 11:34:42.193: CRYPTO_PKI: locked trustpoint NDES-233, refcount is 2
Dec 8 11:34:42.194: CRYPTO_PKI: unlocked trustpoint NDES-233, refcount is 1
Dec 8 11:34:44.630: %SYS-2-PRIVCFG_ENCRYPT: Successfully encrypted private config file
Dec 8 11:34:44.630: CRYPTO_PKI: Setting renewal timers
Dec 8 11:34:44.630: CRYPTO_PKI: set re-enroll timer to 30-second
Dec 8 11:34:44.630: CRYPTO_PKI: unlocked trustpoint NDES-233, refcount is 0
Dec 8 11:34:44.630: CRYPTO_PKI: All enrollment requests completed for trustpoint NDES-233.

[VirtualAccess36632]

Hint:

Dec 8 11:34:42.012: CRYPTO_PKI: can not find router cert in certrep
Dec 8 11:34:42.012: CRYPTO_PKI: can not find crl in certrep

Apparently, SCEP works slightly different on Linux, it only needed a key pair with correct label. Here is a command:

crypto key generate rsa exportable general-keys modulus <modulus size> label <label name>