Sourcefire DC 3500 policy pushes hang or fail

[khendrick512]

Hi,

Whenever any policy changes occur that require a push, I am unable to consistently apply the policy to several appliances.  I have five sets of access control policies that I will push out, and out of all these, maybe one will successfully run and apply to the appliance within a matter of a couple of minutes or so.  On the remaining 10 or so appliances, the will remain in a "polling" state on the Task Status page, and will eventually result in a failure, sometimes after 24 hours or more. 

Error messages are sometimes simply "Access Control policy apply failed", and other times I see "Failed to update state: DB connection was lost Previous state: Remote status: Applying policy".

On the appliances, I have tried accessing the CLI and restarting the ActionQueueScraper, but that doesn't seem to resolve the issue.

I am running DC 5.4.1.2, and my sensors are on 5.4.0.2.

Any thoughts about troubleshooting steps I can take?

Thank you!

[ankojha]

Hi,

Did you check the /var/log/messages for failed ones just to narrow down if the policy push is failing at the sensor or at the DC itself.

If the reason in the logs shows that is due to RPC timeout then we need to increase the interval for RPC connection so that it can push the policy to the remote devices.

Thanks,

Ankita

[khendrick512]

The timeouts I see in the log (using cat /var/log/messages | grep) are entries like this:

Mar 28 16:12:44 <DC_NAME> SF-IMS[31339]: [1466] SFDataCorrelator:RRDClient [INFO] read timed out

Mar 28 12:51:54 <DC_NAME> SF-IMS[7603]: [3703] sftunneld:sf_ssl [ERROR] Unable to connect to port 8305 (IPv4): Connection timed out

When I grep on rpc, I see normal looking entries for that.