No Ping Response from C-SPC after Upgrade to 2.8.1

[CommKeeper]

I ran the GUI update to the lastest version last night and all went well. However this morning I was told server alerts were going off since they could no longer get a ping response from the server. I logged into the admin user of the CLI and I can see where ICMP type 8 is allowed on the firewall of the C-SPC.

Does anyone else have this issue? We have stopped ping checks for now, but would like to resume so we can ensure our inventory is happening regularly.

[brawall]

Is it still not replying to ping requests? I checked and it looks like your collector has uploaded multiple times today so it seems to have at least upgraded properly. It would make sense to lose pings for a while while the CSPC rebooted, which can sometimes take a while.

On a side note, it looks like you are getting uploads silenced due to them coming in so frequently. I would check the configuration to prevent this from happening.

Thank you,

Brandon

[CommKeeper]

They have not worked all day. We had to silence it in our monitoring system.

How often should it upload?

[jofrumki]

Uploads to SNTC are typically scheduled for at most once a week since we do not collect any syslog data and contract and EOX information does not fluctuate that frequently.

[mdsmith200]

I have the same issue.  Version 2.8.1 installed on Saturday morning, and my monitoring system has been unable to ping since the upgrade.  I am able to log into the web interface, and my Sunday morning collection and upload job ran without issue.  Did a firewall get enabled during the upgrade?

[CommKeeper]

@mdsmith200 wrote:

I have the same issue.  Version 2.8.1 installed on Saturday morning, and my monitoring system has been unable to ping since the upgrade.  I am able to log into the web interface, and my Sunday morning collection and upload job ran without issue.  Did a firewall get enabled during the upgrade?

---

From what I could see, the firewall was allowing ICMP, but nothing was responding. Not sure if there is another layer of firewall I am not seeing. Either way, we still cannot ping our C-SPC, but its up and working.

[brawall]

We've upgraded some collectors internally and don't have the ping issue. What monitoring system are you pinging from? Are you able to ping it from your workstation? Are you able to ping out from the collector itself? 

[brawall]

Can you login to CLI as root user and temporarily disable iptables to see if you can ping? 

service iptables stop

[trapasso]

My CSPC server just upgraded to version 2.8.1 this morning and the server stopped replying to pings after the reboot.  I am able to get access to the server via SSH and https but cannot ping.

The suggestion brawall was dead on.  Shutting down iptables did workaround the issue, I am able to ping with iptables stopped.  I hope there is a version 2.8.2 out soon to fix this issue or a quick fix I can apply.

[brawall]

Can you please PM me the output of "iptables -L" as root user?

Thank you,

Brandon

[brawall · Accepted solution]

It looks like iptables is being overwritten and the icmp permit rule is not there anymore after upgrade. This looks like a bug but for now instead of disabling iptables you can run "iptables -L --line-numbers" to get the numbered rules and in the INPUT chain add the icmp permit with "iptables -I INPUT 13 --match icmp --protocol icmp --icmp-type any --jump ACCEPT" where 13 is the line number at the end of INPUT chain. If there is a LOG_DROP entry at the end of the input chain, change the number to whatever line is 1 above it.

Thanks,

Brandon

[CommKeeper]

Thanks for the reply. Working on changing this now.

[CommKeeper]

This works. Pinging is alive again.

[jvujcich]

Adding the ICMP to the IPTABLEs worked for me, But, after a restart of the collector, the ping issues came back. The IPTABLES ICMP entry had to be re-added. How do I make it stick?