Cloud Lookup Timeout

[True Warrior]

Hello All,

One of our SourceFire device has a Malware license and recently we enabled Malware blocking. We created a File policy with the action Block Malware with reset. From the Malware events, I can see the action as Cloud Lookup Timeout which makes me to think that the malware was not blocked as the device wasn't able to do a lookup to the SourceFire cloud.

I'm able to do a nslookup to api.amp.sourcefire.com. We have other devices with the action as Malware Cloud Lookup and the cloud lookup is working. I'm wondering why the cloud lookup works for most of the detection and not for some. I haven't received any health alerts saying that there are some issues with the cloud. I checked the file sizes and its not that large (195.3125KB). Any suggestions?

[David Janulik]

Hello,

First of all, is that device an FMC (firepower management device)? Following document needs to be updated. The sourcefire.com was deprecated.

https://www.cisco.com/c/en/us/support/docs/security/sourcefire-fireamp-private-cloud-virtual-appliance/118290-technote-fireamp-00.html

If that's AMP public cloud than any integration will point to our server in cisco.com domain. This link gives you the required server list overview: https://www.cisco.com/c/en/us/support/docs/security/sourcefire-amp-appliances/118121-technote-sourcefire-00.html

Second for sample analysis we support following file types. Maybe the file submitted was in a file format out of the list. Can you be more specific, on what failed?

https://www.cisco.com/c/en/us/support/docs/security/advanced-malware-protection-endpoints/118711-technote-fireamp-00.html

Finally did you update the FMC from 6.1 to 6.2.1? (or did you update to other versions)

This is an important, so I can help you sort this out.

Thanks

David

[True Warrior]

Hello David,

Thanks for the response. Please see the below response for your questions.

Yes, Its a Cisco Firepower Management Center 2000 running on v6.1.0 and all the sensors running on the same version.

The below are the file types that are affected and I can confirm that are supported file types.

MSEXE
MSOLE2
RTF
SWF

I'm adding a screenshot that shows Cloud Lookup Timeout in Malware Events. 

[David Janulik]

Did you notice following error in the FMC logs?

error with the SFDataCorrelator,

SFDataCorrelator:imcloudpool [WARN] Can't register: Can't parse data

[True Warrior]

I extracted all the gzipped files and below is the message that I see.

root@xxxxxx:/var/tmp# grep -i 'SFDataCorrelator:imcloudpool' messages*
messages.1:Aug 4 14:52:56 xxxxxx SF-IMS[6999]: [5801] SFDataCorrelator:imcloudpool [WARN] Failed to connect to the cloud: Can't connect to host cloud-sa.amp.sourcefire.com
messages.1:Aug 5 00:45:35 xxxxxx SF-IMS[6999]: [5801] SFDataCorrelator:imcloudpool [WARN] bogus slot state (already replied, 5) txid (54212 54212)
messages.2:Jul 27 11:24:06 xxxxxx SF-IMS[6999]: [5801] SFDataCorrelator:imcloudpool [WARN] bogus slot state (already replied, 5) txid (30123 30123)
messages.2:Jul 27 14:02:16 xxxxxx SF-IMS[6999]: [5801] SFDataCorrelator:imcloudpool [WARN] bogus slot state (already replied, 5) txid (13509 13509)
messages.3:Jul 18 09:42:33 xxxxxx SF-IMS[6999]: [5801] SFDataCorrelator:imcloudpool [WARN] bogus slot state (already replied, 5) txid (21024 21024)
messages.3:Jul 19 13:20:10 xxxxxx SF-IMS[6999]: [5801] SFDataCorrelator:imcloudpool [WARN] bogus slot state (query got removed, 4) txid (39843 56227)
messages.4:Jul 11 22:06:08 xxxxxx SF-IMS[6999]: [9425] SFDataCorrelator:imcloudpool [WARN] bad txid: 0
messages.4:Jul 13 09:20:15 xxxxxx SF-IMS[6999]: [9406] SFDataCorrelator:imcloudpool [INFO] Joining imcloud query thread
messages.4:Jul 13 09:20:15 xxxxxx SF-IMS[6999]: [5801] SFDataCorrelator:imcloudpool [INFO] starting imcloud query thread
messages.4:Jul 13 09:20:16 xxxxxx SF-IMS[6999]: [5801] SFDataCorrelator:imcloudpool [WARN] failed to encode query

[David Janulik]

I checked the output above, as I primary focus on AMP product, the only think FMC point me to check required server addresses for cloud lookup. I believe the FMC sensor does the cloud lookup action. This link gives you the required server list overview and please check it using curl command with "curl -v -k ServerAddress". So does the firewall allows all traffic transmitt to following server addresses?

https://www.cisco.com/c/en/us/support/docs/security/sourcefire-amp-appliances/118121-technote-sourcefire-00.html

Also can you reproduce that on demand and grab FMC logs? I understand you use AMP public cloud, is that correct?

If you're behind proxy server the curl command will look like this:

where --proxy switch has the correct Proxy IP Port

and at the end I used dynamic analysis web page Threatgrid, which you can edit with the hostname from the server list above

curl -vk --proxy http://10.0.0.10:8080 https://panacea.threatgrid.com/

[True Warrior]

The FMC is located in Europe.
I have provided the curl output for the URL used for Dynamic Analysis.
I have screenshots for the AMP connection & DA.

root@pl-ips-mgmt01:~# curl -v -k https://mgmt.eu.amp.cisco.com
* Rebuilt URL to: https://mgmt.eu.amp.cisco.com/
* Trying 52.30.217.226...
* Connected to mgmt.eu.amp.cisco.com (52.30.217.226) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=CA; L=San Jose; O=Cisco Systems, Inc.; CN=eu.amp.cisco.com
* start date: Mar 23 17:04:21 2017 GMT
* expire date: Mar 23 17:04:18 2019 GMT
* issuer: C=US; O=HydrantID (Avalanche Cloud Corporation); CN=HydrantID SSL ICA G2
* SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET / HTTP/1.1
> Host: mgmt.eu.amp.cisco.com
> User-Agent: curl/7.48.0
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Wed, 09 Aug 2017 09:22:21 GMT
< Server: Apache
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< X-XSS-Protection: 1; mode=block
< X-Request-Id: 83d68dff-3c8d-4c73-ab1c-0244303a09b2
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Status: 403 Forbidden
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
<
* Connection #0 to host mgmt.eu.amp.cisco.com left intact

root@pl-ips-mgmt01:~# curl -v -k https://panacea.threatgrid.com
* Rebuilt URL to: https://panacea.threatgrid.com/
* Trying 199.36.143.68...
* Connected to panacea.threatgrid.com (199.36.143.68) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: jurisdictionC=US; jurisdictionST=California; businessCategory=Private Organization; serialNumber=C1183477; C=US; ST=California; L=San Jose; businessCategory=Private Organization; serialNumber=C1183477; O=Cisco Systems, Inc.; CN=panacea.threatgrid.com
* start date: Mar 28 00:00:00 2016 GMT
* expire date: Jun 15 23:59:59 2018 GMT
* issuer: C=US; O=GeoTrust Inc.; CN=GeoTrust EV SSL CA - G4
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: panacea.threatgrid.com
> User-Agent: curl/7.48.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: nginx/1.10.0 (Ubuntu)
< Date: Wed, 09 Aug 2017 09:26:21 GMT
< Content-Length: 0
< Connection: keep-alive
< Location: /login?next=%2F
< X-TB-HOST: 121
< Strict-Transport-Security: max-age=31536000; includeSubdomains
<
* Connection #0 to host panacea.threatgrid.com left intact

[David Janulik]

Hi,

there is only one think comes to my mind, as the connectivity is flowing well. Do you have any SSL inspection or content inspection? Does the problem comes often in day - to - day basis?

[True Warrior]

Hello David,

No we don't do SSL inspection. We have the below policies enabled on the sensors along with the other generic polices like ACL, health and system.

Malware Policy
Intrusion Policy.

[NetEngineer]

Did you ever find a resolution for this issue?  I'm having a similar issue and progress is slow.  

[juancarlosmartinez]

Hi Guys,

I got the same probleam, here is some steps to tropubleshoot, Cisco TAC mentioned coulbe a false positive. Is a bug CSCut77594  who affect some versions.

## Note ##
- The following steps should be done *while the issue is happening*.

1. SSH to the FMC and gain root access.

$ sudo su

2. Check for DNS lookup issues

nslookup service.brightcloud.com
nslookup database.brightcloud.com

Expected Result:

Name: service.brightcloud.com
Address: X.X.X.X
Address: X.X.X.X

Name: database.brightcloud.com
Address: X.X.X.X
Address: X.X.X.X
Address: X.X.X.X

3. If you can successfully resolve the names on FMC, check for connectivity

[ For the service.brightcloud.com IPs ]

telnet <IP> 80 -e !
wget http://<IP>

[ For the database.brightcloud.com IPs ]

telnet <IP> 80 -e !
wget http://<IP>
telnet <IP> 443 -e !

telnet X.X.X.X -e !

4. If you are able to telnet and wget to all IPs, but still the issue persisted....please do the following and provide the SSH session log.

# Note #
- Root access as before is required and should be done *while the issue is happening*

- Enable Session logging to file on the SSH session and ...

+ Disable CloudAgent

root@FMCPrime:/var/log# pmtool disablebyid CloudAgent

+ Run CloudAgent in Debug mode

root@FMCPrime:/var/log# CloudAgent --debug --config /etc/sf/cloudagent.conf --peers-config /etc/sf/device_cap.conf --ccfg /etc/sf/bca.cfg > /var/tmp/CloudAgent.debug &

5. Wait for 1 or 2 min in debug mode while the issue is happening.

6. Open another SSH session to the FMC and grain root access

$ sudo su

7. Kill the CloudAgent service and start the agent again.

- Get the current Process ID (PID)

root@FMCPrime:/Volume/home/admin# pgrep CloudAgent

Example Output
898

- Kill CloudAgent

root@FMCPrime:/Volume/home/admin# pkill CloudAgent

- Enalbe Cloud Agent

root@FMCPrime:/Volume/home/admin# pmtool enablebyid CloudAgent

- Check the PID

root@FMCPrime:/Volume/home/admin# pgrep CloudAgent

Example Output

1997

Make sure the Process ID before you kill CloudAgent and After you kill / start the agent are not the same. In the above example its 898 before and 1997 after.