Re: Cisco WSA "Anti-Malware and Reputation" GUI not working

Martin Kyrc · ‎09-30-2023

I can't show AMP configuartion on GUI. Our AsyncOS version is 14.0.5 (S695)

I found older bug connected to this behaviour (https://bst.cisco.com/bugsearch/bug/CSCuv48754), but in our case DNS is working and "telnet panacea.threatgrid.com 443" has connectivity.

amp_logs shows "Warning: The File Reputation service is not reachable", but no more details. The same story after HW restart.

- What else can I check?
- How can I restart or troubleshoot "file reputation service" (see amp_log message)?
- How can I change AMP configuration on WSA with CLI? (ampconfig exists in ESA, but I need to change it in WSA)

balaji.bandi · ‎09-30-2023

Do you have License (i am sure you have just checking) -

make sure you test the connectivity using right interface. (depends on the interface you looking to connect to cloud, by default its DATA - if you looking to use Manangment then you need to use M1

there is not much configuration you can do on WSA

check some troubleshooting tips :

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKSEC-3771.pdf

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

amojarra · ‎10-01-2023

Hi @Martin Kyrc

[1] As Balaji mentioned, it is best to first check the connectivity, to do this , kindly just type telnet in the CLI and press enter,

then choose the correct interface ( it is better to test with all interfaces one by one )

please try this URL port 443 : cloud-sa.amp.cisco.com

[2] you can restart AMP service from CLI > diagnostic > services > AMP > RESTART

[3] if that din't fixed the issue , there might be some certificate issue, it is better to change the AMP logs to trace or Debug

to do this : CLI > logconfig > EDIT > choose the number associated with amp_logs > press enter > change the log level to trace > press enter to finish the wizard and commit

[4] check displayalerts to see if there is any app fault.

[5] if none of them works, please open a TAC case, we need to check things from Backend

Regards,

Amirhossein Mojarrad

+++++++++++++++++++++++++++++++++++++++++++++++++++

++++ If you find this answer helpful, please rate it as such ++++

+++++++++++++++++++++++++++++++++++++++++++++++++++

Martin Kyrc · ‎10-01-2023

thanks for reply...

@balaji.bandi license is there, DNS routing is through M1 and "internet" routing is through DATA interface. there is no problem with DNS resolving or "inrternet" connectivity. I thing, the issue is connected to AMP service.

@amojarra
[1] we have connectivity to cloud-sa.amp.cisco.com (on port 443). there is no problem with DNS resolving or/and data connectivity to internet.

[2] AMP service was running but I see in the system log this: "Warning: The File Reputation service is not reachable." After restarting AMP service (diagnostic -> services -> amp -> restart) I can see in the log this:

Mon Oct 2 08:08:15 2023 Info: File reputation service initialized successfully
Mon Oct 2 08:08:15 2023 Info: The following file type(s) can be sent for File Analysis:Executables, Document, Microsoft Documents, Database, Miscellaneous, Encoded and Encrypted, Configuration, Email, Archived and compressed. To allow analysis of new file type(s), go to Security Services > File Reputation and Analysis.
Mon Oct 2 08:08:15 2023 Info: connecting to /tmp/reporting_listener.sock.root [try #0 of 20]
Mon Oct 2 08:08:15 2023 Info: File Analysis service initialized successfully
Mon Oct 2 08:08:15 2023 Info: connected to /tmp/reporting_listener.sock.root [try #0 of 20]
Mon Oct 2 08:08:15 2023 Info: AMP started with Max faster connection limit 200 and Max slower connection limit 200 with file location for slow connections at "/data/tmp/amp/slow_conn"
Mon Oct 2 08:08:15 2023 Info: connecting to /tmp/amp_cb_fastrpc.sock [try #0 of 20]
Mon Oct 2 08:08:15 2023 Info: connected to /tmp/amp_cb_fastrpc.sock [try #0 of 20]
Mon Oct 2 08:09:17 2023 Info: File reputation service initialized successfully
Mon Oct 2 08:09:17 2023 Info: The following file type(s) can be sent for File Analysis:Executables, Document, Microsoft Documents, Database, Miscellaneous, Encoded and Encrypted, Configuration, Email, Archived and compressed. To allow analysis of new file type(s), go to Security Services > File Reputation and Analysis.
Mon Oct 2 08:09:17 2023 Info: connecting to /tmp/reporting_listener.sock.root [try #0 of 20]
Mon Oct 2 08:09:17 2023 Info: File Analysis service initialized successfully
Mon Oct 2 08:09:17 2023 Info: connected to /tmp/reporting_listener.sock.root [try #0 of 20]
Mon Oct 2 08:09:17 2023 Info: AMP started with Max faster connection limit 200 and Max slower connection limit 200 with file location for slow connections at "/data/tmp/amp/slow_conn"
Mon Oct 2 08:09:17 2023 Info: connecting to /tmp/amp1_cb_fastrpc.sock [try #0 of 20]
Mon Oct 2 08:09:17 2023 Info: connected to /tmp/amp1_cb_fastrpc.sock [try #0 of 20]
Mon Oct 2 08:09:18 2023 Info: File reputation service initialized successfully
Mon Oct 2 08:09:18 2023 Info: The following file type(s) can be sent for File Analysis:Executables, Document, Microsoft Documents, Database, Miscellaneous, Encoded and Encrypted, Configuration, Email, Archived and compressed. To allow analysis of new file type(s), go to Security Services > File Reputation and Analysis.
Mon Oct 2 08:09:18 2023 Info: connecting to /tmp/reporting_listener.sock.root [try #0 of 20]
Mon Oct 2 08:09:18 2023 Info: File Analysis service initialized successfully
Mon Oct 2 08:09:18 2023 Info: connected to /tmp/reporting_listener.sock.root [try #0 of 20]
Mon Oct 2 08:09:18 2023 Info: AMP started with Max faster connection limit 200 and Max slower connection limit 200 with file location for slow connections at "/data/tmp/amp/slow_conn"
Mon Oct 2 08:09:18 2023 Info: connecting to /tmp/amp2_cb_fastrpc.sock [try #0 of 20]
Mon Oct 2 08:09:18 2023 Info: connected to /tmp/amp2_cb_fastrpc.sock [try #0 of 20]
Mon Oct 2 08:09:45 2023 Warning: The File Reputation service is not reachable.
Mon Oct 2 08:09:45 2023 Info: stunnel process health check pid [66711]
Mon Oct 2 08:09:45 2023 Warning: The File Reputation service is not reachable.
Mon Oct 2 08:09:45 2023 Info: stunnel process health check pid [66711]
Mon Oct 2 08:09:45 2023 Warning: The File Reputation service is not reachable.
Mon Oct 2 08:09:45 2023 Info: stunnel process health check pid [66711]

Till 08:09:18 is AMP service running correct, but starting 08:09:45 is not reachable?

[3] after changing log level to trace, I can't see any problem with certificate:

Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG7[34396097280]: Starting certificate verification: depth=0, subject=/CN=amp.cisco.com/O=Cisco Systems Inc./L=San Jose/ST=California/C=US
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG6[34396097280]: SICAP CERT: checking validity of certificate.
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG5[34396097280]: Certificate accepted: depth=0, subject=/CN=amp.cisco.com/O=Cisco Systems Inc./L=San Jose/ST=California/C=US
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG7[34396097280]: SSL state (connect): SSLv3 read server certificate A
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG3[34396097280]: Unknown protocol encountered with version 0
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG7[34396097280]: SSL state (connect): SSLv3 read server key exchange A
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG3[34396097280]: Unknown protocol encountered with version 0
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG7[34396097280]: SSL state (connect): SSLv3 read server done A
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG3[34396097280]: Unknown protocol encountered with version 0
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG7[34396097280]: SSL state (connect): SSLv3 write client key exchange A
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG3[34396097280]: Unknown protocol encountered with version 0
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG7[34396097280]: SSL state (connect): SSLv3 write change cipher spec A
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG3[34396097280]: Unknown protocol encountered with version 0
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG3[34396097280]: TLS 1.2 Handshake : Finished.
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG7[34396097280]: SSL state (connect): SSLv3 write finished A
Mon Oct 2 08:26:26 2023 Debug: stunnel: 2023.10.02 08:26:26 LOG7[34396097280]: SSL state (connect): SSLv3 flush data

[4] displayalerts shows only above issue with File Analysis service (is not reachable).

Many thanks for hints, but the issue is not solved. I'll try to open TAC case.

Martin

Martin Kyrc · ‎10-02-2023

@amojarra maybe new findings. I do packet capture to dst IP 52.21.117.50:443 and I see that TCP 3-way handshake is correct, SSL-handshake started correctly but I see packet retransmission for "Client Key Exchange". After 9-times retransmission is connection closed. There is no reason for that. Maybe that's the reason why AMP is not running correctly (and I can see amp configuration page on GUI). I need disable AMP only, but GUI page is not possible to load and change/disable amp. Is it possible disable amp (file reputation) using CLI?

Martin Kyrc · ‎10-03-2023

It seems, that an issue is connected with update service - the device is unable to get updates and I think this is a core issue.

In the updater_logs - there is a message "Warning: Failed to download release notifications: Received invalid notification data". What does it mean? Why is it happened? How can I solve this issue?

Wed Oct 4 01:55:28 2023 Info: Started wbrsd timer handler for updates, interval 300 sec.
Wed Oct 4 01:55:28 2023 Info: Scheduled next wbrsd update to occur at Wed Oct 4 02:00:28 2023
Wed Oct 4 01:55:28 2023 Info: Started wbrs timer handler for updates, interval 300 sec.
Wed Oct 4 01:55:28 2023 Info: Scheduled next wbrs update to occur at Wed Oct 4 02:00:28 2023
Wed Oct 4 01:55:28 2023 Info: ytc Starting YTC Category Update Thread
Wed Oct 4 01:55:28 2023 Info: Started talos_intelligence timer handler for updates, interval 300 sec.
Wed Oct 4 01:55:28 2023 Info: Scheduled next talos_intelligence update to occur at Wed Oct 4 02:00:28 2023
Wed Oct 4 01:55:28 2023 Info: trusted_root starting
Wed Oct 4 01:55:28 2023 Info: internal_cert waiting for new updates
Wed Oct 4 01:55:28 2023 Info: firestone waiting for new updates
Wed Oct 4 01:55:28 2023 Info: merlin waiting for new updates
Wed Oct 4 01:55:28 2023 Info: talos_catchange_prenotify waiting for new updates
Wed Oct 4 01:55:28 2023 Info: timezones waiting for new updates
Wed Oct 4 01:55:28 2023 Info: sophos waiting for new updates
Wed Oct 4 01:55:28 2023 Info: howto waiting for new updates
Wed Oct 4 01:55:28 2023 Info: wbrs waiting for new updates
Wed Oct 4 01:55:28 2023 Info: cato waiting for new updates
Wed Oct 4 01:55:28 2023 Info: talos_intelligence waiting for new updates
Wed Oct 4 01:55:28 2023 Info: mcafee waiting for new updates
Wed Oct 4 01:55:28 2023 Info: wbrsd waiting for new updates
Wed Oct 4 01:55:28 2023 Info: support_request waiting for new updates
Wed Oct 4 01:55:28 2023 Info: amp waiting for new updates
Wed Oct 4 01:55:28 2023 Info: trusted_root waiting for new updates
Wed Oct 4 01:55:28 2023 Info: openssh_key waiting for new updates
Wed Oct 4 01:55:28 2023 Info: cisco_cloud_services waiting for new updates
Wed Oct 4 01:55:28 2023 Info: trafmon waiting for new updates
Wed Oct 4 01:55:28 2023 Info: ytc waiting for new updates
Wed Oct 4 01:55:28 2023 Info: smart_agent waiting for new updates
Wed Oct 4 01:55:28 2023 Info: avc waiting for new updates
Wed Oct 4 01:55:28 2023 Info: Started handler for updates; updates are now started
Wed Oct 4 01:55:28 2023 Info: Started release notification thread
Wed Oct 4 01:55:59 2023 Warning: Failed to download release notifications: Received invalid notification data
Wed Oct 4 01:55:59 2023 Info: Scheduled next release notification fetch to occur at Wed Oct 4 04:55:59 2023
Wed Oct 4 01:55:59 2023 Info: Dynamic manifest fetch failure: Received invalid update manifest response
Wed Oct 4 01:57:00 2023 Info: Dynamic manifest fetch failure: Received invalid update manifest response
Wed Oct 4 01:58:00 2023 Info: smart_agent capable of receiving updates again
Wed Oct 4 01:58:01 2023 Info: Dynamic manifest fetch failure: Received invalid update manifest response
Wed Oct 4 01:58:02 2023 Info: Scheduled next update to occur at Wed Oct 4 02:03:02 2023
Wed Oct 4 02:00:28 2023 Info: Scheduled next wbrsd update to occur at Wed Oct 4 02:05:28 2023
Wed Oct 4 02:00:28 2023 Info: Scheduled next wbrs update to occur at Wed Oct 4 02:05:28 2023
Wed Oct 4 02:00:28 2023 Info: Scheduled next talos_intelligence update to occur at Wed Oct 4 02:05:28 2023
Wed Oct 4 02:03:02 2023 Info: Starting scheduled update
Wed Oct 4 02:03:03 2023 Info: Dynamic manifest fetch failure: Received invalid update manifest response
Wed Oct 4 02:04:04 2023 Info: Dynamic manifest fetch failure: Received invalid update manifest response
Wed Oct 4 02:05:05 2023 Info: Dynamic manifest fetch failure: Received invalid update manifest response

Any idea how to fix it?

amojarra · ‎10-04-2023

Hi @Martin Kyrc

From the appliance's CLI, please configure the dynamic update host to the new update server.

vWSA-Testing.com> updateconfig

Choose the operation you want to perform:
- SETUP - Edit update configuration.
[]> dynamichost

Enter new manifest hostname:port
[update-manifests.ironport.com:443]> update-manifests.sco.cisco.com:443

Choose the operation you want to perform:
- SETUP - Edit update configuration.
[]> <Enter>

vWSA-Testing.com> commit

Please enter some comments describing your changes:
[]> <Enter, or comment>

vWSA-Testing.com> updatenow

Success - All component updates requested

NOW, VERIFY UPDATES AND SOFTWARE UPGRADE WORKING OVER THE TAIL:

vWSA-Testing.com> tail

30. "updater_logs" Type: "Updater Logs" Retrieval: FTP Poll

Enter the number of the log you wish to tail.
[]> 30

and kindly check this FN:Field Notice: FN - 72502 - Secure Web, Secure Management, and Secure Email Virtual Appliances Might Not Receive Updates After January 13, 2023 - Configuration Change Recommended - Cisco

to see if the conditions matches you.

Regards,

Amirhossein Mojarrad

+++++++++++++++++++++++++++++++++++++++++++++++++++

++++ If you find this answer helpful, please rate it as such ++++

+++++++++++++++++++++++++++++++++++++++++++++++++++

Martin Kyrc · ‎10-13-2023

dynamic host changed from "update-manifests.ironport.com:443" to "update-manifests.sco.cisco.com:443". the same results.

note: I checked another WSA devices, where AMP GUI page is running, there is mix od ironport.com and sco.cisco.com domain and everything is running.

updater_log:

Fri Oct 13 11:29:33 2023 Debug: app_update feature key disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "app_update"
Fri Oct 13 11:29:33 2023 Debug: csa updates disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "csa"
Fri Oct 13 11:29:33 2023 Debug: graymail updates disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "graymail"
Fri Oct 13 11:29:33 2023 Debug: dlp feature key disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "dlp"
Fri Oct 13 11:29:33 2023 Debug: mailbox_remediator updates disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "mailbox_remediator"
Fri Oct 13 11:29:33 2023 Debug: mcafee feature key disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "mcafee"
Fri Oct 13 11:29:33 2023 Debug: kronos updates disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "kronos"
Fri Oct 13 11:29:33 2023 Debug: eaas updates disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "eaas"
Fri Oct 13 11:29:33 2023 Debug: retro_scanner feature key disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "retro_scanner"
Fri Oct 13 11:29:33 2023 Debug: threatfeeds updates disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "threatfeeds"
Fri Oct 13 11:29:33 2023 Debug: postx updates disabled
Fri Oct 13 11:29:33 2023 Debug: Skipping update request for "postx"
Fri Oct 13 11:29:33 2023 Debug: Acquiring dynamic manifest from update-manifests.sco.cisco.com:443
Fri Oct 13 11:29:33 2023 Debug: Sending client manifest: <XML CUT>
Fri Oct 13 11:29:33 2023 Debug: Network Participation: Connecting to proxy: my-company-proxy:3128
Fri Oct 13 11:29:34 2023 Debug: Network Participation: Attempting to connect to host: update-manifests.sco.cisco.com port: 443
Fri Oct 13 11:29:35 2023 Debug: Network Participation: Successfully connected to host: update-manifests.sco.cisco.com port: 443
Fri Oct 13 11:29:35 2023 Debug: Network Participation: Closed connection to host: update-manifests.sco.cisco.com port: 443
Fri Oct 13 11:29:35 2023 Info: Dynamic manifest fetch failure: Received invalid update manifest response
Fri Oct 13 11:29:39 2023 Debug: Received remote command to get application status
Fri Oct 13 11:29:39 2023 Debug: Received remote command to get application status
Fri Oct 13 11:29:52 2023 Debug: Received remote command to get application status

I checked FN, but we use HW appliance in our case, with valid smart licencing. I can't check "showlicense" on the WSA box. Based on FN description, I think, this is not our problem. Am I sure?