CUCM 8.6 backup SFTP timeout issue

[mark.rollinson]

Hi,

We are running 2 x CUCM appliances at 2 seperate sites with a provider/subscriber setup and having some issues with backing up via SFTP backing up to a windows server.

We have a scheduled backup running on each of the devices, backing up to a local server at each site. The problem we have is that the link between these two sites has very little bandwidth and as a result, when backing up the remote device .tar file the backup times out after 15 mins. This means we have a complete backup of all of the files, but they are spread over the 2 sites and would like a complete backup at each site.

As I understand it, this is as a result of CUCM not detecting that the .tar file has increased in size in 15 mins. Having run wireshark during the backup I can see that the CUCM appliance is sending a reset packet to stop the backup transmissin to the server. Windows does not update the metadata of the file until the file has been completely transferred, but this take more than 15 mins. I can get the backup to complete if I manually update the properties of the .tar file during the backup which proves that the config is correct.

A Linux server would solve the issue as the OS can be set to update the file metadata but unfortunately this is not an option in this case

So.....what I am after is a way to increase this 15 min timer on the CUCM appliance to enable the backup to complete. Any ideas? Would an upgrade to 9.1 get round this issue or would the same thing happen?

Any help would be greatly appreciated.

Cheers

mark

[jimander]

We are seeing a similiar issue - what windows SFTP Server are you using? We are using BitVise WinSSHd and on Server 2003 the filesize increases during the transfer but on 2008 it does not- something diferent in the way Windows is handling the files.

In my testing Titan FTP does update the filesize during the transfer on Windows 7, so I assume 2008 would behave the same. We are unable to move to Titan though, so we are left with Cisco allowing us to update the backup timeout counter or increasing the BW between sites.

You can decrease the size of the TFTP directory by removing the CIUS load files - that's almost 500mb right there, it might be enough to get you under the 15 minute mark.

[btanjutco]

Hi Mark,

Did this get resolved? I am also running into a similar situation, CUCM 8.6.2 running Freeftpd. It is also timing out when backup the Subscriber from a remote location.

Thanks

[jimander]

I 'fixed' my issue - I found a relatively new switch to the DIR command (/R) in Server 2008 that 'Displays alternate data streams of the file'. The side effect is that it updates the metadata of the open files, and DRF sees the size increasing.

So I wrote myself a batch file and a scheduled job that does a dir /R on the CUCM/UC backup directories every 10 minutes and all my backups work now. Tested on 2 different servers/clusters.

[JoeRevelle]

I'm having the same issue. Do you have the batch file that you created to resolve this issue? Do you mind sharing? I know it will work but I'm not sure how to write the batch file.

[jimander]

My script is one line, runs every ten minutes - this command will perform it recursively on all directories below E:\IPT-Backups.

dir /R /S E:\IPT-Backups

Jim

[JoeRevelle]

Jim,

Thanks, I'll give it a try.

Joe

[JoeRevelle]

Hi Jim,

I created the following batch file: dir /R /S E:\UCM Backups

I then went into Task Scheduler and setup a Scheduled Task for it to repeat every 10 min from 1:00 - 2:00 am every morning for the duration of 1 hour.

I just ran the task (with the time edited to run a test) and got the follow message in the backup status "ERROR: SFTP transfer failed as backup size was not increasing for the past 15 minutes. Either there is not enough disk space or network transfer rate is too slow with the configured SFTP Server. Please either free some space on SFTP device or check network connectivity and then run a fresh backup, Backup Completed..."

I reviewed the "task History" and found that the batch file run every 10 min and it was successful.

I know it should work because if I run the "dir /r" command manually the backup will complete successfully.

Any thoughts on this? How do you have the batch setup to run on your system?

Joe

[jimander]

I think it is because you have a space in your directory name - it's probably trying to dir E:\UCM - you'll need to put it in quotes or eliminate the space.

[JoeRevelle]

I got the batch file to work fine if I run it manually every 10 min and the backup will complete successfully. what I did was take the batch fie and put it into the same folder as the backup with the following script:

dir /R

CALL cmd.exe

should this work or should I just put in "dir /r" as it's already in the correct directory? or should I put in :

CALL cmd.exe

dir /R

?

Joe

[JoeRevelle]

Jim,

I think I just got it to work using the following: dir /R /S G:\"UCM Backup"

(Just like you said)

I moved it from a 2008 server to a 2012 server and just let it run while I went to lunch. When I returned, it completed successfully (using the task scheduler).

it must have been an issue with the 2008 server. I'll see how it runs tonight.

Thanks for your help with this issue.

Joe

[richardmoorehead]

I have the scheduler configured correctly within Windows Server 2012, and the history shows it running successfully.  DRS Backups worked the first day but have errored since.

CUCM version is 9.1.2.11900-12 and the SFTP program is Solarwinds.  Backup size is around 23GB.  Running a manual backup works 100% of the time.

There are known DRS bugs with my particular version of CUCM and I know Solarwinds is not officially supported, but I wanted to put this out there in case anyone else has similar problems.

[David Cave]

I'm having this issue as well. Thank for the DIR workaround - I'll give that a shot

[TJ]

Hi Mark,

A bug has been filed and seems to be fixed in ES 9.1.2.12024-1. The bug ID - CSCul38509

Regards

TJ

[Daniel Williams]

This is the answer!  I was hitting this bug.  The test outlined in the bug was spot on.

Now to update!

Thanks!