Log Collection API - PushtoSFTPServer

[jonathan Monestel]

Hello Team,

I have a small problem with the API and the requests.

I am able to push the the request to download logs from the CUCM servers, however I am encountering this error on CUCM 9.1.2 and 10.5.x

When I use the SOAP/AXL request the below error is encountered. I also learned that using "utils os secure permissive" fixes the problem however for obvious reasons I don't want to disable this on my clusters. (I am using labs for now)

Any help will highly appreciated on how to download the traces with secure enforce.

This is a snippet of the error on the SOAP traces: (I have removed the IP addresses, but they are correct, using CLI directly works)

2016-07-22 14:44:45,515 INFO  [http-bio-443-exec-32] LogCollection.LogCollectionUtils - SendFiletoSFTPClient(): sending 1 file(s) via SFTP.

2016-07-22 14:44:45,528 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - Entered bldSendFileCmds fileNames

2016-07-22 14:44:45,530 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - Changed dir /tmp

2016-07-22 14:44:45,530 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - File transfer /var/log/active/cm/trace/cti/sdl/SDL001_200_000001.txt.gz

2016-07-22 14:44:45,530 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - The dir name is /var/log/active/cm/trace/cti/sdl

2016-07-22 14:44:45,530 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - The file name is SDL001_200_000001.txt.gz

2016-07-22 14:44:45,531 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - sendFiles(): subfolder to create=cm/trace/cti/sdl

2016-07-22 14:44:45,531 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - file exists

2016-07-22 14:44:45,532 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - --> XMLBuilder:: connect

2016-07-22 14:44:45,532 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - UserName test

2016-07-22 14:44:45,532 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - hostName <IP ADDRESS>

2016-07-22 14:44:45,532 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - Port 22

2016-07-22 14:44:45,533 INFO  [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - SFTPLinuxScriptClient.connect(): Executing command with sudo: /usr/local/cm/bin/setsoapvar  /var/log/active/cm/trace/tct/tmp/batchCmdscom.cisco.ccm.serviceability.tct.tctutil.SFTPLinuxScriptClient@5a31fa -o Port=22 test@<IP ADDRESS> --taos-logfile /var/log/active/cm/trace/tct/tmp/sessionLogcom.cisco.ccm.serviceability.tct.tctutil.SFTPLinuxScriptClient@5a31fa <PASSWORD>...

2016-07-22 14:44:45,590 ERROR [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - Process /usr/local/cm/bin/setsoapvar  /var/log/active/cm/trace/tct/tmp/batchCmdscom.cisco.ccm.serviceability.tct.tctutil.SFTPLinuxScriptClient@5a31fa -o Port=22 test@<IP ADDRESS> --taos-logfile /var/log/active/cm/trace/tct/tmp/sessionLogcom.cisco.ccm.serviceability.tct.tctutil.SFTPLinuxScriptClient@5a31fa <PASSWORD> returned non-zero value:1

2016-07-22 14:44:45,590 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - Process output:

2016-07-22 14:44:45,591 DEBUG [http-bio-443-exec-32] tctutil.SFTPLinuxScriptClient - Process error:

2016-07-22 14:44:45,594 ERROR [http-bio-443-exec-32] LogCollection.LogCollectionUtils - SendFiletoSFTPClient(): Download failed for matching files. Reason:

2016-07-22 14:44:45,594 DEBUG [http-bio-443-exec-32] LogCollection.LogCollectionUtils - LogCollectionUtils.SendFiletoSFTPClient::Finished

2016-07-22 14:44:45,595 ERROR [http-bio-443-exec-32] LogCollection.LogCollectionUtils - Error encountered during downloading files to SFTP server.

2016-07-22 14:44:45,595 ERROR [http-bio-443-exec-32] LogCollection.LogCollectionUtils - Thrwoing exceptionjava.rmi.ServerException: Error encountered during downloading files to SFTP server.

2016-07-22 14:44:45,595 ERROR [http-bio-443-exec-32] LogCollection.LogCollectionBindingImpl - selectLogFiles(): Remote ExceptionError encountredjava.rmi.ServerException: Error encountered during downloading files to SFTP server.

This is the request:

I am testing with just 5min of traces to avoid long delays.

<!--LogCollection API - SelectLogFiles - Request-->

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soap="http://schemas.cisco.com/ast/soap">

   <soapenv:Header/>

   <soapenv:Body>

      <soap:selectLogFiles>

         <soap:FileSelectionCriteria>

            <soap:ServiceLogs>

               <soap:item>Cisco CTIManager</soap:item>

            </soap:ServiceLogs>

            <soap:SystemLogs>

               <soap:item></soap:item>

            </soap:SystemLogs>

            <soap:SearchStr></soap:SearchStr>

            <soap:Frequency>OnDemand</soap:Frequency>

            <soap:JobType>PushtoSFTPServer</soap:JobType>

            <soap:ToDate></soap:ToDate>

            <soap:FromDate></soap:FromDate>

            <soap:TimeZone></soap:TimeZone>

            <soap:RelText>minutes</soap:RelText>

            <soap:RelTime>5</soap:RelTime>

            <soap:Port>22</soap:Port>

            <soap:IPAddress>servername</soap:IPAddress>

            <soap:UserName>username</soap:UserName>

            <soap:Password>password</soap:Password>

            <soap:ZipInfo>false</soap:ZipInfo>

            <soap:RemoteFolder>/tmp</soap:RemoteFolder>

         </soap:FileSelectionCriteria>

      </soap:selectLogFiles>

   </soapenv:Body>

</soapenv:Envelope>

[dstaudt]

This looks very similar to an issue we helped another customer resolve recently.  In that case the issue was a misconfiguration of a particular SE Linux security subsystem setting on CUCM.  These SELinux settings should be handled automatically by the CUCM install and upgrade scripts - it was not determined how the setting got into the incorrect state.  Note that disabling the SELinux protections via the CLI (as you describe) also allowed the request to succeed as a (not permanent) workaround.

For now we can suggest the end-customer (if not you) go ahead and a case with Cisco TAC to request assistance with updating the SELinux poiicy to its correct state.  The TAC SR 638356601 and DevNet Ticket 736 can be referenced to help TAC understand the issue.  If you need help from the DevNet side, please go ahead and open a DevNet ticket here: https://developer.cisco.com/site/devnet/support/

[jonathan Monestel]

Thank you for the response,

I am seeing the same behavior on a mixture of 9 and 10 versions across more than 50 clients we manage. I am a little hesitant that this is a SELinux policy misbehavior or one time installation issue.

I don't see TAC fixing more than 200 Servers to be honest. Was a bug opened on this case you mentioned?

[dstaudt · Accepted solution]

It looks like a defect was opened/resolved for this against the 11.5 train:

CSCuy85786

CUCM SELinux policies for customer API that looks into CUCM traces

I do not see that an Engineering Special (patch release) has been requested for this as of yet.  Please open a DevNet support ticket referring to the SR and defect #s and we can work with you to request an ES for you to verify/test.  Note, customers should contact TAC to discuss obtaining a fix for CSCuy85786, i.e. via Engineering Special.

[jonathan Monestel]

Thank you.

Patching all clients into 11.5 is not going to be an option and getting TAC root accounts for all servers may not be viable. Probably the best (not very acceptable) option is to create the script to set the OS permissive, download logs and set the OS enforced after that. At least I know now is not something missing on my end.

I'll see how that works. I appreciate your answers.

[dstaudt]

It's quite possible that the fix could be backported to other, earlier release trains.