IPS V7 Global Correlation

[learnsec]

Dear all,

IPS Correlation update will be done through the Management interface right? So I should confirm the ability of the IPS Management IP address to be able to access internet right?

I did so, but still not able to have global correlation update, what I am having each time I enable global correlation is a boost of traffic generated from the IPS and directed to the outside that is consuming the total internet link bandwidth.

What could be the reason behind this boost, and how may I troubleshoot the reason why the correlation is not being updated.

Regards,

[Dustin Ralich]

IPS Correlation update will be done through the Management interface right? So I should confirm the ability of the IPS Management IP address to be able to access internet right?

Correct, and yes.

I did so, but still not able to have global correlation update. how may I troubleshoot the reason why the correlation is not being updated.

Take a look at the Global Correlation section of this document (it covers enabling, connectivity requirements, testing/troubleshooting).

[learnsec]

hello dustin,

the link you provided do not help troublshoot why i am not able to have successfull GC update.

i tried the following:

1- disbale IPS mgt port,

2-assign IPS mgt port ip address to a laptop and test internet connectivity. (Successfull)

3- try to ping(from the laptop)  "update-manifests.ironport.com" it resolves to ip address "204.15.82.17" but ping was not successfull.

4- try to open on Internet explorer http://update-manifests.ironport.com nothing opens.

can you help?

[Dustin Ralich]

i tried the following:
1- disbale IPS mgt port,
2-assign IPS mgt port ip address to a laptop and test internet connectivity. (Successfull)
3- try to ping(from the laptop)  "update-manifests.ironport.com" it resolves to ip address "204.15.82.17" but ping was not successfull.
4- try to open on Internet explorer http://update-manifests.ironport.com nothing opens.

While I believe I see where you are going with your testing, did you follow the all the troubleshooting steps found in the document I referenced?

Specifically, did you:

1. Ensure that the sensor's Management IP address has full, unfiltered outbound Internet access (at least DNS, HTTP, and HTTPS)? If necessary, did you update outbound access-lists on your firewalls/routers to permit this?
2. Ensure the sensor's clock is accurate/correct (since SSL/TLS is involved in the initial GC update connection)?
3. Confirm if your network environment makes use of a proxy server/system? If so, the sensor needs to be configured with the relevant information so that it can properly proxy its GC update connection.

If after confirming/correcting the above items, you are still unable to determine why this appears to fail, please copy/paste a new 'show stat global' command output into this discussion for review.

[learnsec]

Hello Dustin,

concerning the points you mentioned above, please find hereafter my answers:

1) IPS Mgt IP has full unfiltered outbound Internet access.

2) the sensor clock is synchronized with our NTP Server

3) no proxy is used.

!

IPS# show statistics global-correlation

Network Participation:

   Counters:

      Total Connection Attempts = 0

      Total Connection Failures = 0

      Connection Failures Since Last Success = 0

   Connection History:

Updates:

   Status Of Last Update Attempt = Failed

   Time Since Last Successful Update = never

   Counters:

      Update Failures Since Last Success = 3

      Total Update Attempts = 3

      Total Update Failures = 3

   Update Interval In Seconds = 300

   Update Server = update-manifests.ironport.com

   Update Server Address = Unknown

   Current Versions:

      config = 0

      drop = 0

      ip = 0

      rule = 0

Warnings:

IPS#

!

Regards,

[Dustin Ralich]

IPS# show statistics global-correlation
Updates:
   Status Of Last Update Attempt = Failed
   Time Since Last Successful Update = never
   Counters:
      Update Failures Since Last Success = 3
      Total Update Attempts = 3
      Total Update Failures = 3
   Update Interval In Seconds = 300
   Update Server = update-manifests.ironport.com
   Update Server Address = Unknown
   Current Versions:
      config = 0
      drop = 0
      ip = 0
      rule = 0
Warnings:
IPS#

Looks like your sensor was/is unable to resolve the GC Update Server name to an IP address (see the bolded line in your output above). I.e. Either no DNS server is configured on the sensor or the configured server(s) is/are not valid/working. An example of configuring a DNS server can be found here.

[learnsec]

hello dustin,

the problem was in routing, once routing is fixed global corrolation worked fine.

but a new problem occured within the same exercice related to sensor health,

event retreival status is critical!!!

i restarted the sensor but same issue, how may fix this critical problem?

[Dustin Ralich · Accepted solution]

once routing is fixed global corrolation worked fine.

Excellent.

a new problem occured within the same exercice related to sensor health,
event retreival status is critical!!! i restarted the sensor but same issue, how may fix this critical problem?

This is actually not related... the sensor's Event Retrieval health metric tracks how much time has passed since a remote monitoring application/device has logged into the sensor and pulled copies of its event store. By default, this will be Red (as by-default there isn't a remote monitoring system configured against any given sensor installation, as-is). This doesn't indicate anything is wrong on the sensor, just that nothing is copying its event store contents remotely (really just an FYI, especially for users that have compliancy reasons, etc. that require they ensure logs are being pulled).

If you have a remote monitoring system (SDEE client such as: CS-MARS, IME, or a 3rd-party system) that is up and running 24x7, configured properly with the sensor's login credentials, then this metric should change to Green. If you do not have such a monitoring system, you can disable this particular health metric.

[learnsec]

Hello Dustin,

you are right, the problem is that for CS-Mars the IPS is not anymore reacheable.

also i have a routing problem between the IPS and the NTP Server.

can i add a static route on IPS in order to do not change the IPS gateway?

although i added a static route on the IPS gateway(ASA) for the ntp server and CS-Mars but still unreacheable!

in similar cases how can we proceed ?

thanks in advance,

[Dustin Ralich]

also i have a routing problem between the IPS and the NTP Server. can i add a static route on IPS in order to do not change the IPS gateway?

There is not a supported method of adding static routes on the sensor for it's Management traffic. It is expected that the configured default-gateway will have a proper, working routing table.

although i added a static route on the IPS gateway(ASA) for the ntp server and CS-Mars but still unreacheable! in similar cases how can we proceed ?

Fix the routing problem (and/or ACL, NAT, etc. problem) present on your Layer 3 devices (i.e. ensure that everything is reachable by everything else). Start with the sensor and work hop-by-hop to your CS-MARS, confirming the routing table entries present on each hop. Then, do the reverse (from the CS-MARS to the sensor).

[learnsec]

hello again,

the routing issue was solved for CS-Mars but not yet for NTP Server.

i  keep asking the above question:

"can i add a static route on IPS in order to do not change the IPS gateway?"

in addition, i had an error on CS-Mars concerning this IPS which is

"CS-MARS module: csips detected a conflicting certificate for device with IP: 172.34.42.10"

i tested connectivity it asked me to "accept new certificate " then i redicover device and it seem problem solved till now, but is that normal? i didnt change IPS Certificate just gateway address.

thank you,

regards,

[Dustin Ralich · Accepted solution]

in addition, i had an error on CS-Mars concerning this IPS which is 
"CS-MARS module: csips detected a conflicting certificate for device with IP: 172.34.42.10"
i tested connectivity it asked me to "accept new certificate " then i redicover device and it seem problem solved till now, but is that normal? i didnt change IPS Certificate just gateway address.

When/if you modify the sensor's Management IP address (host-ip config entry), the sensor regenerates it's self-signed TLS certificate. So, yes, that would be expected in your case. The default-gateway is configured by that same entry and you changed it.

[mik.gustafsson]

Hi,

I had the exact same problem that I solved to day.

Full connectivity but still the error:

# sh statistics global-correlation

Network Participation:

   Counters:

      Total Connection Attempts = 0

      Total Connection Failures = 0

      Connection Failures Since Last Success = 0

   Connection History:

Updates:

   Status Of Last Update Attempt = Failed

   Time Since Last Successful Update = 3826 minutes

   Counters:

      Update Failures Since Last Success = 764

      Total Update Attempts = 22747

      Total Update Failures = 806

   Update Interval In Seconds = 300

   Update Server = update-manifests.ironport.com

   Update Server Address = 204.15.82.17

   Current Versions:

      config = 1236210407

      drop = 1312830724

      ip = 1312830846

      rule = 1312744926

# sh events error error warning past 12:00

evError: eventId=1304592381890230981 severity=error vendor=Cisco

  originator:

    hostId: xxxxxxxx

    appName: collaborationApp

    appInstanceId: 458

  time: 2011/08/11 00:38:28 2011/08/11 02:38:28 GMT+01:00

  errorMessage: name=errUnclassified A global correlation update failed: Failed download of ibrs/1.1/drop/default/1313021562 :

  URI does not contain a valid ip address

Messages, like this one, in the category - Reputation update failure - were logged 49 times in the last 14699 seconds.

I found a tip when searching that worked for me :

Issue the: dns-secondary-server disable to flush DNS wait for GC to update again.

Thanks to: http://doublef.org/archives/cisco-ips-global-correlation-update-failures 

HTH

Edit: I see a difference in our output, you don't have the ip address in update server field:

Update Server Address = Unknown

Might not bee the same problem.

[learnsec]

thank you mikael,

it was a problem related to routing, Global Correlation issue was solved but another issue appeared

thanks