LDAP connection exceed maximum simultaneous connections that was setup

mizwan.saib-c · ‎07-04-2017

Hi Cisco Support Coummnity,

My IronPort was configured to do verification to ldap server through port 3268. It was configured to establish 10 maximum concurrent connection. However, when I run netstat. It shown output shown below. Which I can conclude, connection establish exceed 10 connection concurrently. Is this a normal behavior? Because recently connection between IronPort and ldap server spike out and remain high. Kindly advise. Thank you.

P/s:

- IronPort: 192.168.200.3

- LDAP 1 : 10.21.1.175

-LDAP 2 : 10.21.1.176

Choose the information you want to display:
1. List of active sockets.
2. State of network interfaces.
3. Contents of routing tables.
4. Size of the listen queues.
5. Packet traffic information.
[1]> 1

1. IPv4 only.
2. IPv6 only.
[1]> 1

Show network addresses as numbers? [N]> y

Avoid truncating addresses? [N]> y

Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 192.168.200.3.443 10.27.3.242.56655 TIME_WAIT
tcp4 0 0 192.168.200.3.23238 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.443 10.27.3.242.56654 TIME_WAIT
tcp4 0 0 192.168.200.3.443 10.27.3.242.56653 TIME_WAIT
tcp4 0 0 192.168.200.3.443 10.27.3.242.56652 TIME_WAIT
tcp4 0 0 127.0.0.1.8089 127.0.0.1.38362 TIME_WAIT
tcp4 0 0 192.168.200.3.443 10.27.3.242.56651 TIME_WAIT
tcp4 0 0 192.168.200.3.18367 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.43651 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.55441 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.443 10.27.3.242.56650 TIME_WAIT
tcp4 0 0 192.168.200.3.47022 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.12393 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.36576 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.22 10.27.3.242.56649 ESTABLISHED
tcp4 0 0 192.168.200.3.36891 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.41198 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.1325 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.443 10.27.3.242.56648 TIME_WAIT
tcp4 0 0 192.168.200.3.58338 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.14715 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.16392 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.3589 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.17781 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.51739 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.443 10.27.3.242.56647 TIME_WAIT
tcp4 0 0 192.168.200.3.443 10.27.3.242.56646 TIME_WAIT
tcp4 0 0 192.168.200.3.443 10.27.3.242.56645 TIME_WAIT
tcp4 0 0 192.168.200.3.19815 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.17858 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.443 10.27.3.242.56640 TIME_WAIT
tcp4 0 0 192.168.200.3.443 10.27.3.242.56639 TIME_WAIT
tcp4 0 0 192.168.200.3.443 10.27.3.242.56638 TIME_WAIT
tcp4 0 0 192.168.200.3.443 10.27.3.242.56637 TIME_WAIT
tcp4 0 0 192.168.200.3.443 10.27.3.242.56636 TIME_WAIT
tcp4 0 0 192.168.200.3.40093 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.55146 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.34839 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.21914 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.42288 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.35754 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.10385 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.45065 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.39546 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.17164 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.59595 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.46151 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.12355 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.58900 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.49603 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.14215 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.25371 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.17944 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.47024 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.51802 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.45725 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.37540 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.41744 10.21.1.176.3268 ESTABLISHED
tcp4 0 0 192.168.200.3.8205 10.21.1.175.3268 ESTABLISHED
tcp4 0 0 127.0.0.1.9997 127.0.0.1.33809 ESTABLISHED
tcp4 0 0 127.0.0.1.33809 127.0.0.1.9997 ESTABLISHED
tcp4 0 0 192.168.200.3.25 *.* LISTEN
tcp4 0 0 192.168.42.42.443 *.* LISTEN
tcp4 0 0 192.168.200.3.443 *.* LISTEN
tcp4 0 0 192.168.200.3.80 *.* LISTEN
tcp4 0 0 127.0.0.1.9997 *.* LISTEN
tcp4 0 0 127.0.0.1.8089 *.* LISTEN
tcp4 0 0 192.168.200.3.83 *.* LISTEN
tcp4 0 0 192.168.200.3.82 *.* LISTEN
tcp4 0 0 192.168.200.3.21 *.* LISTEN
tcp4 0 0 192.168.42.42.22 *.* LISTEN
tcp4 0 0 192.168.200.3.22 *.* LISTEN
tcp4 0 0 127.0.0.1.53 *.* LISTEN
tcp4 0 0 127.0.0.1.5432 *.* LISTEN
udp4 0 0 192.168.200.3.35461 10.31.1.77.514
udp4 0 0 192.168.200.3.58547 10.31.1.77.514
udp4 0 0 192.168.200.3.36411 10.31.1.77.514
udp4 0 0 192.168.200.3.161 *.*
udp4 0 0 127.0.0.1.53 *.*
udp4 0 0 192.168.200.3.31856 192.168.1.110.514
udp4 0 0 192.168.200.3.4118 10.31.1.77.514
udp4 0 0 127.0.0.1.61879 127.0.0.1.61879
udp4 0 0 127.0.0.1.514 *.*

Libin Varghese · ‎07-04-2017

Hi,

This appears to be linked to the below defect.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCur70268/?reffering_site=dumpcr

The defect was terminated due to lack of customer sightings and the only mentioned workaround would be to lower the concurrent connections count further.

You can certainly open a TAC case if you feel the defect is business impacting and request the defect be re-opened.

Currently, the number of connections per host might be two more than twice or thrice the maximum configured connections setting.

- Libin V

mizwan.saib-c · ‎07-04-2017

Hi Libin,

Thank you for your response.

Do this defect will suddenly cause our total number of connection from IronPort to ldap server increase?

This defect shows its symptoms since beginning of time or only being triggered after certain conditon?

Do any firmware upgrade will solved this issue? If yes, can you suggest which AsynOS should we upgrade? Our current firmware is 9.7.1-066, model c170.

Kindly advise

Kindly advise.

Libin Varghese · ‎07-04-2017

The defect currently shows high number of connections are seen if External Authentication are used.

It does not mention if there would be a sudden increase in the number of connections and it currently not marked as fixed on newer Async OS releases as well.

The defect status is currently terminated so there are no plans that I see of this getting fixed in the near future.

- Libin V

mizwan.saib-c · ‎07-04-2017

Okay. Noted. Thank you so much Libin. Have a good day!

mizwan.saib-c · ‎07-05-2017

Hi Libin,

I have another doubt. Let say we configured IronPort to have 10 of maximum session to ldap.

Do this will be one to one connection from IronPort to ldap?

Example 1:

Src :- 192.168.200.3.3421(IronPort) Dest:- 10.21.1.167:3268(Ldap)

Consider: One session and one connection.

Example 2:

Src: 192.168.200.3.2321(IronPort) Dest:- 10.21.1.167:3268(Ldap)

Src: 192.168.200.3.2331(IronPort) Dest:- 10.21.1.167:3268(Ldap)

Src: 192.168.200.3.2322(IronPort) Dest:- 10.21.1.167:3268(Ldap)

Consider: One session, many connection.

Which one is true?

Kindly advise.

Thanks

Libin Varghese · ‎07-05-2017

The maximum number of concurrent connections is the count for number of connections over port 3268 at a given time.

There can be multiple LDAP requests over once active session, however would still be considered a single session.

- Libin V

mizwan.saib-c · ‎07-05-2017

Hi Libin.

Noted. Thank you for the explanation.

We found out that every about 20 mili second, IronPort will do connection request to LDAP server. However, this request do not contain any AD authentication process. After several seconds, this connection will be dropped automatically. Do this is normal IronPort behaviour with LDAP? Why? Kindly advise.

Many thanks.

Libin Varghese · ‎07-05-2017

The ESA does not normally open connections that often.

The ESA keeps the concurrent LDAP connections open for 6 hours or 10.000
queries(whatever comes first).

Seeing connections every 20ms would be strange. I would suspect another network appliance closing the connections which is causing the ESA to open connections again. This connection interruptions are often caused by an firewall/gateway in between the Email Security Appliance and the LDAP servers. This can also be caused by the LDAP server itself closing the connection early.You could try setting up a capture from the ESA to the LDAP server IP to monitor the traffic.

Do note that LDAP is integrated with the ESA for different types of queries and not just AD authentication.

- Libin V

mizwan.saib-c · ‎07-08-2017

Hi Libin,

Do you have anything that I can refer on integration between ESA and LDAP such as official documentation from Cisco? As I do really understand deeply how this integration occur in order to provide normal behavior of this activity to my customer.

Libin Varghese · ‎07-12-2017

I do not see a documentation apart from the defect I shared earlier on in the post.

If there is a specific issue the customer would like to troubleshoot on then a TAC case would certainly be helpful to get additional insight directly from TAC after they review what is happening.

- Libin V