cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
761
Views
5
Helpful
8
Replies

CISCO ACE Syslog

michellp
Level 1
Level 1

I have in the syslogs of our ACE an entry that matches below code.

302022
Error Message    %ACE-6-302022: Built TCP connection id for
interface:real-address/real-port (mapped-address/mapped-port) to interface:real-address/real-port (mapped-address/mapped-port)
 Explanation    This informational message is logged when a TCP connection slot between two hosts is created.

The question I have is about the first part with the real-address/port and mapped-address/port. Where is this mapping done or where is it taken from? Where can I find it?

Anyone have an idea?

1 Accepted Solution

Accepted Solutions

Hi Michell,

It was a good discussion and i am glad i was of help:)

Having said that i agree with your assumption here. We could have for sure verified if we could actually see that connection in "show conn address 2.3.4.5" and in detail it also shows whether the connection is in "reuse" pool or not. Like below:

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
552511     1  in  TCP   5    10.82.235.243:64618   10.86.212.33:23       ESTAB
          [ idle time   : 00:00:00,   byte count  : 6896       ]
          [ elapsed time: 00:00:50,   packet count: 167        ]
550931     1  out TCP   5    10.86.212.33:23       10.82.235.243:64618   ESTAB
          [ conn in reuse pool : FALSE]
          [ idle time   : 00:00:00,   byte count  : 7692       ]
          [ elapsed time: 00:00:50,   packet count: 131        ]

 

That shall give you an idea. Also, you can see how front end connection is mapped to the backend. Like below:

switch/Admin# show np 1 me-stats "-c 552511 -vvv"


Connection ID:seq: 552511[0x86e3f].0
  Other ConnID    : 550931[0x86813].0
  Proxy ConnID    : 0[0x0].0
  Next Q    : 0[0x0]

10.82.235.243:64618 -> 10.86.212.33:23 [RX-NextHop: CP] [TX-NextHop: TX]
  Flags:  PAT: No  DynNAT: No  Implicit PAT: No On_Reuse: No
  L3 Protocol     : IPv4                L4 Protocol    : 6
  Inbound Flag    : 1
  Interface Match : Yes
    Interface MatchID: 5

Please look at the bold part. If you see i took the connection ID from show conn output, NP would be 1 here and can be 2,3 and 4 depending on which NP handles the connection (ace 20 has two NP's and ace 30 has 4) and that's all you need to see which front end connection is mapped to which one at the back end. That should help in your further TS. Also, the above output gives lot of other details which i omitted here since we don't need to go over it.

Regards,

Kanwal

 

 

 

 

View solution in original post

8 Replies 8

Kanwaljeet Singh
Cisco Employee
Cisco Employee

Hi Michell,

The real address here would be your client IP address and mapped address/port would be NAT IP address/port(if NAT is configured) and second part would be server side interface VLAN and real-address/port and mapped-address/port would be "actual server's ip address and port".

It would easy to co-relate if you have an output of "show conn" and also the corresponding above message.

Regards,

Kanwal

Hi Kanwal,

I'll be a bit more specific so you can get a better point of view in regards to the issue that I have. I will use fictitious addresses, It will not matter I believe. Anyway

We have a webserver and on that webserver an external client is trying to access an particular url. Now the setup is as follows [client pc]<=>(internet) <=>[firewall]<=>[ACE]<=>[webserver]

Now this client pc has global ip address 1.1.1.1 and our webservers global address is 5.5.5.5. Internally 5.5.5.5 is natted to 10.10.10.1(ACE) and then the ace forwards to webserver 192.168.1.1

Now the syslog should look like this (focus on the bold):

%ACE-6-302022: Built TCP connection 0x1a02d1 for vlan10:1.1.1.1/63422 (1.1.1.1/63422) to vlan20:10.10.10.1/443 (192.168.1.1/80)

But instead it looks like this:

%ACE-6-302022: Built TCP connection 0x1a02d1 for vlan10:1.1.1.1/63422 (2.3.4.5/63422) to vlan20:10.10.10.1/443 (192.168.1.1/80)

The webserver has an ip filter so only white listed ip's may access this url. ip 1.1.1.1 is on that list as it is the global ip address of the external client. The webserver however denies access because it somehow sees the request comming from the mapped address 2.3.4.5. I cannot add this ip to the list because it seems this mapped address is random, so different mapped ip in every tcp stream. Now the destination address and mapping makes sense as this is configured on the ace. The source part is not, as far as I can see. So where does this mapping come from? Why does this happen? And this does not happen for all external clients, but for many and so also this particular one.

 

Hi Michell,

Where is this IP "2.3.4.5" in your ACE configuration (if you co-relate with original IP schema)? I assume that you are doing source NAT which makes the ACE to change the client IP address. By default, ACE will take the packet from FW, do destination NAT and forward it to the server. Simple L3-L4 loadbalancing. If you do source NAT, ACE will do source as well as destination NAT before forwarding the packet.

You have 443 and 80 as ports mentioned so i assume you are doing SSL termination which will make the ACE to behave as proxy i.e one front end stream and one backend stream. But without source NAT, ACE still uses the original client IP to open the backend connection with server. With source NAT it will use IP from NAT pool.

We need to find out why ACE is changing the client IP? What is making ACE to do that. We will have that answer in configuration. If you can share your configuration, i can take a look.

Regards,

Kanwal

HI Kanwal,

 

Thanks for your reply. This "2.3.4.5" is nowhere in the config. And no, we do not do source NAT only destination.

Unfortunately due to security I cannot just share the config with you unless I go and change/mask all sensitive information which is a bit of a pain.

Let me ask you this, there is a parameter:

parameter-map type http HTTP_PARAMETERS
  server-conn reuse
  case-insensitive
  persistence-rebalance

This are referenced in:

policy-map multi-match VIPs
  class BLABLANL_VIP
    loadbalance vip inservice
    loadbalance policy Blabla_https-policy
    loadbalance vip icmp-reply
    appl-parameter http advanced-options HTTP_PARAMETERS
    ssl-proxy server proxy-BLA

Could this server-conn reuse option explain this? If I understand the reuse option correcty and correct me if I'm wrong; The ace has the reuse option to reduce the amount of connections to the webserver. Now, if a clients sets up a tcp connection and it uses the same tcp parameters, it can reuse an existing connection to the webserver. Now let's reflect this on our situation. The client (1.1.1.1) sets up a connection. The ACE realizes it can use an existing connection to the webserver (previously setup by 2.3.4.5) and resues this connection. Now the ip filter on the webserver comes in place as it sees a connection that was initially setup by  2.3.4.5 and says 403 since this ip is not white listed. Does this sound like a plausible explenation for this behaviour?

 

Hi Michell,

ACE uses existing server connections at the backend which are IDLE and have same TCP options. If all connections are active, then ACE would open a new connection since it cannot use any IDLE connection.

So for ACE to actually REUSE an IDLE connection, that connection has to be established but in your case that won't be possible since that IP is not known to server.

What do you see in "show conn address 2.3.4.5"?

Also, if you are using a TCP connection reuse it is best practice to use PAT on interface. For more details please check out user guide.

Regards,

Kanwal

Hi Kanwal,

You are right about the idle connections. However, the ACE could indeed REUSE that specific IDLE connection (initially setup by 2.3.4.5) Here's why I believe it can;

The webserver serves multiple url's. The default page is accessible to everyone, no ip filter here. The ip filter only applies to a specific url. Now, if a connection was setup by a client with ip 2.3.4.5 (or any ip for that matter) to the default page and that client disconnected, that specific connection would remain IDLE on the ACE's backend and placed in the REUSE pool.

Now client 1.1.1.1 comes along and connects to the specific url using same tcp options. The request arrives at the ACE's backend and the ACE looks for an idle connection in the REUSE pool, finds one and uses it.

The request get forwarded to the server. The server sees some tcp packets trying to access the url that has the ip filter. So the webserver tries to match the source ip against the list.

Now my guess is, since the ACE has mapped the real-ip against some random ip (2.3.4.5 or any other ip)  that came from the REUSE pool, the server will see this random ip (2.3.4.5)  as the source compare to the ip filter and deny access. The http 403 is then returned to the ACE. The ACE does a reverse mapping, translates 2.3.4.5 back to 1.1.1.1 and forwrds the 403 deny to the client. Sounds plausible no?

 

Anyway to test this theory I removed the REUSE parameter from the ACE. From the ACE's syslogs I see no more REUSE or random ip's. The client (1.1.1.1) does no longer get the 403. So it seems my theory is correct. I don't know if it's exaclty as I say, but at least to a degree it seems to be correct.

I believe our issue is solved. I will do some more testing just to be sure. In any case, Kanwal you support has been helpful, you've pointed me in the right direction I guess. Thanks for that!

Hi Michell,

It was a good discussion and i am glad i was of help:)

Having said that i agree with your assumption here. We could have for sure verified if we could actually see that connection in "show conn address 2.3.4.5" and in detail it also shows whether the connection is in "reuse" pool or not. Like below:

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
552511     1  in  TCP   5    10.82.235.243:64618   10.86.212.33:23       ESTAB
          [ idle time   : 00:00:00,   byte count  : 6896       ]
          [ elapsed time: 00:00:50,   packet count: 167        ]
550931     1  out TCP   5    10.86.212.33:23       10.82.235.243:64618   ESTAB
          [ conn in reuse pool : FALSE]
          [ idle time   : 00:00:00,   byte count  : 7692       ]
          [ elapsed time: 00:00:50,   packet count: 131        ]

 

That shall give you an idea. Also, you can see how front end connection is mapped to the backend. Like below:

switch/Admin# show np 1 me-stats "-c 552511 -vvv"


Connection ID:seq: 552511[0x86e3f].0
  Other ConnID    : 550931[0x86813].0
  Proxy ConnID    : 0[0x0].0
  Next Q    : 0[0x0]

10.82.235.243:64618 -> 10.86.212.33:23 [RX-NextHop: CP] [TX-NextHop: TX]
  Flags:  PAT: No  DynNAT: No  Implicit PAT: No On_Reuse: No
  L3 Protocol     : IPv4                L4 Protocol    : 6
  Inbound Flag    : 1
  Interface Match : Yes
    Interface MatchID: 5

Please look at the bold part. If you see i took the connection ID from show conn output, NP would be 1 here and can be 2,3 and 4 depending on which NP handles the connection (ace 20 has two NP's and ace 30 has 4) and that's all you need to see which front end connection is mapped to which one at the back end. That should help in your further TS. Also, the above output gives lot of other details which i omitted here since we don't need to go over it.

Regards,

Kanwal

 

 

 

 

Hi Michell,

Replied again because my last message had some typos.

Regards,

Kanwal

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: