Hi,
We have a Cisco CSR1000V deployed in AWS with the following AMI :
https://aws.amazon.com/marketplace/pp/B00EV8VWR2?qid=1488972774924&sr=0-4&ref_=srh_res_product_title
This CSR has a kron scheduled to let it copy by FTP its configuration and copy it directly to its running-config.
Issue is that randomly, (1 out of 3) the CSR fails to write the downloaded configuration to its running-config.
From the debug FTP, we see the CSR is able to download the configuration, but cannot write it, and generate this message : "%SYS-3-CONFIG_SYS_ERR: System running-config write error; configuration failed".
Cannot figure out why, as if we do the same command manually it works or it may also work with a kron. It seems to only appear if the command is executed from a kron.
The kron configuration is given to the CSR using User data :
"ios-config-0001=\"file prompt quiet\"\n",
"ios-config-0002=\"event manager environment RTBPriv ",{"Ref":"PrivRouteTable"},"\"\n",
"ios-config-0005=\"event manager environment ENIPriv ",{"Ref":"ENIPrivate2"},"\"\n",
"ios-config-0006=\"event manager environment RTBPub ",{"Ref":"PubRouteTable"},"\"\n",
"ios-config-0007=\"event manager environment ENIPub ",{"Ref":"ENIPublic2"},"\"\n",
"ios-config-0008=\"kron policy-list SETUP\"\n",
"ios-config-0009=\"cli copy ftp://X:X@X.X.X.X/",{"Ref":"AWS::Region"},"/config-csr2.txt running-config\"\n",
"ios-config-0010=\"cli wr\"\n",
"ios-config-0011=\"kron occurrence SETUP in 5 oneshot\"\n",
"ios-config-0012=\"policy-list SETUP\"\n",
"ios-config-0015=\"do wr\"\n",
"ios-config-0016=\"do reload\"\n"
And the full debug when it happens :
Mar 3 11:45:29.682: Major 1, Minor 0
*Mar 3 11:45:29.682: Timer Event SETUP
*Mar 3 11:45:29.683: Call parse_cmd 'copy ftp://X:X@X.X.X.X/config-test.txt running-config'
*Mar 3 11:45:29.743: FTP: 220 10.231.192.130 FTP server ready
*Mar 3 11:45:29.743: FTP: ---> USER X
*Mar 3 11:45:29.770: FTP: 331 Password required for X.
*Mar 3 11:45:29.770: FTP: ---> ****
*Mar 3 11:45:29.799: FTP: 230 User X logged in.
*Mar 3 11:45:29.799: FTP: ---> TYPE I
*Mar 3 11:45:29.826: FTP: 200 Type set to I
*Mar 3 11:45:29.826: FTP: ---> PASV
*Mar 3 11:45:29.853: FTP: 227 Entering Passive Mode
*Mar 3 11:45:29.879: FTP: ---> RETR eu-west-1/config-test.txt
*Mar 3 11:45:29.905: FTP: 150 Opening BINARY mode data connection for eu-west-1/config-test.txt (4216 bytes)
*Mar 3 11:45:29.957: FTP: ---> QUIT
*Mar 3 11:45:29.983: FTP: 226 Transfer complete.
*Mar 3 11:45:30.042: FTP: 220 10.231.192.130 FTP server ready
*Mar 3 11:45:30.042: FTP: ---> USER X
*Mar 3 11:45:30.070: FTP: 331 Password required for X.
*Mar 3 11:45:30.070: FTP: ---> ****
*Mar 3 11:45:30.099: FTP: 230 User X logged in.
*Mar 3 11:45:30.099: FTP: ---> TYPE I
*Mar 3 11:45:30.126: FTP: 200 Type set to I
*Mar 3 11:45:30.126: FTP: ---> PASV
*Mar 3 11:45:30.153: FTP: 227 Entering Passive Mode
*Mar 3 11:45:30.179: FTP: ---> RETR eu-west-1/config-test.txt
*Mar 3 11:45:30.206: FTP: 150 Opening BINARY mode data connection for eu-west-1/config-test.txt (4216 bytes)
*Mar 3 11:45:30.206: FTP: ---> QUIT
*Mar 3 11:45:30.232: FTP: 226 Transfer complete.
*Mar 3 11:45:30.295: FTP: 220 10.231.192.130 FTP server ready
*Mar 3 11:45:30.295: FTP: ---> USER X
*Mar 3 11:45:30.321: FTP: 331 Password required for X.
*Mar 3 11:45:30.321: FTP: ---> ****
*Mar 3 11:45:30.351: FTP: 230 User X logged in.
*Mar 3 11:45:30.351: FTP: ---> TYPE I
*Mar 3 11:45:30.378: FTP: 200 Type set to I
*Mar 3 11:45:30.378: FTP: ---> PASV
*Mar 3 11:45:30.405: FTP: 227 Entering Passive Mode (
*Mar 3 11:45:30.432: FTP: ---> RETR eu-west-1/config-test.txt
*Mar 3 11:45:30.458: FTP: 150 Opening BINARY mode data connection for eu-west-1/config-test.txt (4216 bytes)
*Mar 3 11:45:30.510: FTP: ---> QUIT
*Mar 3 11:45:30.536: FTP: 226 Transfer complete.
*Mar 3 11:45:30.596: FTP: 220 10.231.192.130 FTP server ready
*Mar 3 11:45:30.596: FTP: ---> USER X
*Mar 3 11:45:30.623: FTP: 331 Password required for X.
*Mar 3 11:45:30.623: FTP: ---> ****
*Mar 3 11:45:30.651: FTP: 230 User X logged in.
*Mar 3 11:45:30.651: FTP: ---> TYPE I
*Mar 3 11:45:30.678: FTP: 200 Type set to I
*Mar 3 11:45:30.678: FTP: ---> PASV
*Mar 3 11:45:30.705: FTP: 227 Entering Passive Mode
*Mar 3 11:45:30.732: FTP: ---> RETR eu-west-1/config-test.txt
*Mar 3 11:45:30.758: FTP: 150 Opening BINARY mode data connection for eu-west-1/config-test.txt (4216 bytes)
*Mar 3 11:45:30.810: FTP: ---> QUIT
*Mar 3 11:45:30.836: FTP: 226 Transfer complete. //We can see FTP download is fine
*Mar 3 11:45:32.154: %SYS-3-CONFIG_SYS_ERR: System running-config write error; configuration failed
*Mar 3 11:45:32.154: Kron CLI return 0
The file does not look corrupted as if I run the command manually it works. Only if the command is executed from a kron generates this issue. Same, if I use a temp file: it works if I do it manually, but failed if performed with a kron. Also, it does sometime work with a kron. It may work with the first two Cisco CSR instances, and issue will appear with the third one. All of them use the same file, which make me think file is not corrupted. However there is probably an issue with the Cisco CSR instance, but I cannot figure out what. All of our CSR deployments use the same AMI, but for some reason, sometimes issue appears on one of them.
We want to automate this task, and so we don't to do it manually.
Could you help us understanding why we have this behaviour?
