ACS 5.2 fails to send files to sftp server after installing patc

ROBERTO GIANA · ‎07-22-2011

Hi

After we have installed patch 5 on several ACS 5.2 server they aren't able anymore to write their backups to the sftp servers. I tried to search on the bug tool kit, but it seems to be broken when searching for the keyword "sftp". It's the same when I try to do a "copy logs" with sftp as destination.

running a debug I can see:

acs/admin# copy logs sftp://10.1.115.11/

Collecting logs...

Username: backupuser

Password:

6 [16376]: transfer: cars_xfer.c[301] [admin]: sftp copy out of /var/tmp/ADElogs.tar.gz requested

6 [16376]: transfer: cars_xfer_util.c[412] [admin]: resolved server to 10.1.115.11

7 [16383]: transfer: sftp_copy.c[75] [daemon]: Executing SFTP command: /usr/bin/scp -o StrictHostKeyChecking=no /var/tmp/ADElogs.tar.gz backupuser@10.1.115.11://ADElogs.tar.gz

% Error: Transfer failed

3 [16376]: transfer: sftp_copy.c[230] [admin]: sftp_copy ERROR: command execution failed

3 [16376]: copy: cm_copy.c[1226] [admin]: Logs archive transfer to url sftp://10.1.115.11/ failed retcode=-306

acs/admin#

Is anybody else seeing this problem?

Jatin Katyal · ‎07-22-2011

Please get the following debugs from the ACS CLI

debug copy 7

debug transfer 7

Regards,

Jatin

~Jatin

ROBERTO GIANA · ‎07-25-2011

It's almost the same as with the copy command:

acs/admin# acs backup acs repository WUG-Test

6 [13229]: transfer: cars_xfer.c[120] [admin]: sftp copy out of /opt/backup/backup-acs-110725-0920-1311578444/acs-110725-0920.tar.gpg requested

6 [13229]: transfer: cars_xfer_util.c[343] [admin]: resolved server to 10.1.115.11

7 [13229]: transfer: cars_xfer_util.c[359] [admin]: copying file to remote server: 10.1.115.11 with full path /acs-110725-0920.tar.gpg

7 [13350]: transfer: sftp_copy.c[75] [daemon]: Executing SFTP command: /usr/bin/scp -o StrictHostKeyChecking=no /opt/backup/backup-acs-110725-0920-1311578444/acs-110725-0920.tar.gpg backupuser@10.1.115.11:/acs-110725-0920.tar.gpg

3 [13229]: transfer: sftp_copy.c[230] [admin]: sftp_copy ERROR: command execution failed

% Error: Failed to perform ACS backup: SSH connect error

acs/admin#

acs/admin# ssh 10.1.115.11 ********

********@10.1.115.11's password:

C:\Windows\system32>

C:\Windows\system32> exit

acs/admin#

BTW: Doing an SFTP with other tools to the same server works. Even with a Cisco Call Manager.

Jatin Katyal · ‎07-25-2011

Please confirm if we have the same DH diffie-hellman-group14 on the server side.
Make sure we have SSHv2 configured on the server.

Regards,
Jatin

~Jatin

ROBERTO GIANA · ‎07-25-2011

Yes, we have. Please see the logs from a Putty connection:

2011-07-25 13:38:30 Server version: SSH-2.0-WeOnlyDo 2.1.3

2011-07-25 13:38:30 We claim version: SSH-2.0-PuTTY_Release_0.60

2011-07-25 13:38:30 Using SSH protocol version 2

2011-07-25 13:38:30 Using Diffie-Hellman with standard group "group14"

2011-07-25 13:38:30 Doing Diffie-Hellman key exchange with hash SHA-1

2011-07-25 13:38:31 Host key fingerprint is:

BTW: It worked before upgrading from patch 4 to patch 5!

sanjayb · ‎08-05-2011

Hi

We have run into the same issue whith SFTP. The issue is if the backup transfer takes more than 60 seconds the connection will drop. I have timed my backup transfer many times and it disconnects exactly in 60 seconds. My servers ( ACS and SFTP) are on the same network so there is no latency issues. I have tried will all flavors of the patch levels and the situations is the same . We have been told it will be fixed only in 5.3. which may be some time in october

Michael Famin · ‎09-21-2011

Hi,

Just installet a couple of new ACS 5.2 and have exactly the same problem.

Anyone got it to work yet?

Waiting for 5.3 is not an option since we go live with a lot of 802.1x config, and I do not want to live without backups...

ROBERTO GIANA · ‎09-21-2011

After installing patch 6 I was able to see in the debugs, that the systems own ssh process isn't able to access its "known_hosts" file. Obviously a bug of the patches. But as we do not have access to the Linux only TAC is able to fix it.

See the threat "ACS can't access ssh known_hosts file".

Michael Famin · ‎09-22-2011

Hmmm,

After examining the debug I realized that it is'nt SFTP, it is actually SCP that is used for the file transfer. So I asked our UNIX guys (which were providing the backup server) for an SCP server instead of SFTP. And I was the able to get a manual copy to work!!! The file was an backup to the localfilesystem done earlier.

CLI Initiated Copy
====================

ACS/admin# copy disk:Manual-110921-1832.tar.gpg sftp://10.X.X.X/home/XXXXXX/drop
Username: XXXXXX

Password:
6 [31774]: transfer: cars_xfer.c[301] [admin]: sftp copy out of /localdisk/Manual-110921-1832.tar.gpg requested
6 [31774]: transfer: cars_xfer_util.c[412] [admin]: resolved server to 10.X.X.X

7 [31781]: transfer: sftp_copy.c[75] [daemon]: Executing SFTP command: /usr/bin/scp -o StrictHostKeyChecking=no /localdisk/Manual-110921-1832.tar.gpg XXXXXX@10.107.199.36:/home/XXXXXX/drop//Manual-110921-1832.tar.gpg
7 [31774]: transfer: cars_xfer_util.c[433] [admin]: sftp xfer succeeded

I will however not work with the backup script

CLI Initiated Backup
====================

ACS/admin# acs backup test repository Westlife
6 [32125]: transfer: cars_xfer.c[120] [admin]: sftp copy out of /opt/backup/backup-test-110922-1733-1316705616/test-110922-1733.tar.gpg requested
6 [32125]: transfer: cars_xfer_util.c[343] [admin]: resolved server to 10.X.X.X.X

7 [32125]: transfer: cars_xfer_util.c[359] [admin]: copying file to remote server: 10.X.X.X with full path /home/XXXXXX/drop/test-110922-1733.tar.gpg
7 [32247]: transfer: sftp_copy.c[75] [daemon]: Executing SFTP command: /usr/bin/scp -o StrictHostKeyChecking=no /opt/backup/backup-test-110922-1733-1316705616/test-110922-1733.tar.gpg komscp@10.107.199.36:/home/XXXXXX/drop/test-110922-1733.tar.gpg
3 [32125]: transfer: sftp_copy.c[137] [admin]: sftp_copy ERROR: timeout!
3 [32125]: transfer: sftp_copy.c[230] [admin]: sftp_copy ERROR: command execution failed
% Error: Failed to perform ACS backup: SSH connect error

Can not really see the difference between the scp commands, except the path of the sourcefile.

I realize that I do not have the latest patches installed in my ACS, so I will upgrade a testserver to patchlevel 6 tomorrow and see whats happening. Anyway, the retcode=-306 seems to be because of that my backup server did not accept SCP, only SFTP connections...

ROBERTO GIANA · ‎09-22-2011

Please do not confuse the command "scp" with the protocol "scp". ;-) Using the command scp you can decide which transfer method you would like to use. Usually you can use "scp" or "sftp" as transfer method.

For example for PuTTy you can control it by using "pscp -scp ..." or "pscp -sftp ..."

Michael Famin · ‎09-23-2011

OK, Sorry for that.

What I ment to point out was that the ACS is using the SCP protocol.

From the Releasenote:(http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_system/5.2/release/notes/acs_52_rn.html#wp190802)

SFTP Copy

In ACS 5.2, SSH File Transfer Protocol (SFTP) is implemented by Secure Copy Protocol (SCP).

And our backupserver has different accounts if you are using the sftp or scp protocol. If I use the sftp account the transfer fails with retcote=-306, but if I use the scp account it works.

sanjayb · ‎09-23-2011

This is a known bug. The SFTP File transfer will fail if the time it takes to backup is more than a minute. I have tested it many times and has been verified by cisco, the fix will only be done in 5.3

Michael Famin · ‎09-23-2011

Hmm,

Thats the BugID: CSCtn78315 you mention.

It seems strange to me though, because if I backup to localdisk and then manually copy the file with the command:

copy disk:Manual-110923-1723.tar.gpg sftp://10.X.X.X/home/XXXXXX/drop

It takes less than 5 seconds to copy it successfully...

Might open a TAC Case on this.

Michael Famin · ‎10-10-2011

Hi again,

Opened a TAC case on this, and the TAC identified it as the 60 seconds bug (CSCtn78315), and told me it was fixed in

release 5.3. So, I downloaded and upgraded on my test ACS server.

Unfortionally it did'nt do any good. The behavior is still the same.

We'll see what the TAC says...

Michael Famin · ‎11-04-2011

Hi,

My problem was that I had an too long password. The password can not be longer than 16 characters.

To new BugID created from the TAC Case: CSCtt47076 and CSCtt46924

ACS 5.2 fails to send files to sftp server after installing patch 5

SFTP Copy