out of file descriptors for accept()

[ryan-hitch]

CentOS 7
NSO 4.5.5.1

I am seeing the following when trying to perform a check-sync or sync-from on several hundred devices, which causes NSO to crash:

out of file descriptors for accept() – process limit reached

System file limits are as follows:

cat /proc/sys/fs/file-max

100000

ulimit -a

open files (-n) 16384

The number of open files never seems to go above ~5000 before the crash occurs. Has anyone else seen this? Perhaps there are other OS or Java parameters that need adjusting?

cat /proc/sys/fs/file-nr

5005 0    100000

[joepak · Accepted solution]

Here is a couple of resources:

https://techzone.cisco.com/t5/Network-Services-Orchestrator/File-Descriptor-Limits-for-NCS/ta-p/1238611

and

- Need to break the hard limit for the number of FD for a process. This can be done in two ways


	1.	OS Level:             
		a.	In order to know how many file descriptors are available at the OS Level (normally it is 1024), please execute:
		ulimit -a | grep 'open files'
		b.	For increasing FD in system level you will need to add the following lines to the /etc/security/limits.conf file:

			*    soft nofile <value>
			*    hard nofile <value>
				root soft nofile <value>
				root hard nofile <value>

			e.g 

			*    soft nofile 32000
			*    hard nofile 32000
			root soft nofile 32000
			root hard nofile 32000

		c.	To verify this you will have to logout from your current session, connect again and execute the command given above. (ulimit -a | grep 'open files')
	2.	NSO Level:
		a.	In order to know how many file descriptors are available to NSO, please execute:
												 ps -ef | grep "fd" OR  ps -aef | grep ncs.smp
		b.	By default even these will be 1024 and in order to change this you will need to perform the following steps:

								  *       Stop NSO if it is running.
								  *       Open the /etc/init.d/ncs file and add the new ulimit value below the “heart” argument

																<Other Configurations>
																heart="--heart"
																ulimit -n <valu

Let me know if this information was helpful. It would seem that the issue is down to the OS level.

[ryan-hitch]

Thanks! I do not have access to https://techzone.cisco.com - perhaps its a Cisco internal site? The info is great though, I had increased the OS level limits, but did not know that a change to /etc/init.d/ncs was also required without this document.

[joepak]

My apologies as that link is in fact, internal to us. But it basically shares similar info :) I'm glad it was helpful!

thank you for rating this post!

[joepak]

Also, what NED are you using? And versions?

[rodriry06]

Spoiler

Hi Joey,

I did not find that file in the path your posted but found in /etc/security/limits.conf . 

Is this the right file, and what is the max number that is safe to reconfigure with?  
Should we incrementally test to see and are there specific increments we need to adhere to?

# - nofile - max number of open file descriptors