WMI is not working through NAT'd address

jtowry · ‎12-04-2013

Hello,

I have 2 MS servers that I am trying to pull WMI statistics from. When I access the server with its real address of 10.0.1.224 it is successful. When I try to access it with the NAT'd address of 10.5.20.224 it fails. I have inspect dcerpc enabled with a class-map. Pings, traceroutes, RDP's work with no issues to the NAT'd address of 10.5.20.224

!

object network PerryCo-224-ext

host 10.0.1.224

object network PerryCo-224

host 10.5.20.224

!

nat (ITS_inside,ITS-outside) source static Orion-srv Orion-srv-ext destination static PerryCo-224 PerryCo-224-ext no-proxy-arp

!

class-map MSRPC

match port tcp eq 135

!

policy-map type inspect dcerpc MSRPC-MAP

parameters

endpoint-mapper epm-service-only lookup-operation

timeout pinhole 0:03:00

policy-map global_policy

class inspection_default

.

inspect icmp error

class MSRPC

inspect dcerpc MSRPC-MAP

This log entry shows up:

%ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows; Connection for tcp src ITS_inside:10.1.1.51/21342 dst ITS-outside:10.0.1.224/135 denied due to NAT reverse path failure

And here is a debug dcerpc from the connection attempt:

DCERPC-PKT: bind id:2 - ITS_inside:10.1.1.51/6085 to ITS-outside:10.0.1.224/135.

DCERPC-EV: bind has non-epm uuid 99fcfec4-5260-101b-bbcb-00aa0021347a.

DCERPC-EV: bind with ctx_num:2

DCERPC-EV: retrieve ctx_id: 0, if-uuid: 99fcfec4

DCERPC-EV: retrieve ctx_id: 1, if-uuid: 99fcfec4

DCERPC-PKT: bind_ack id:2 - ITS-outside:10.0.1.224/135 to ITS_inside:10.1.1.51/6085.

DCERPC-EV: bind_ack with ctxnum_result:2

DCERPC-EV: ctxid/result:0/0 accepted, ack_reason:0 tsyn: 8a885d04

DCERPC-EV: ctxid/result:1/3 rejected, ack_reason:3 tsyn: 00000000

DCERPC-PKT: request id:2 - ITS_inside:10.1.1.51/6085 to ITS-outside:10.0.1.224/135.

DCERPC-EV: valid_ctxid: alt:0 ctxnum:2 j:0 val:0 valid:1

DCERPC-PKT: response id:2 - ITS-outside:10.0.1.224/135 to ITS_inside:10.1.1.51/6085.

DCERPC-PKT: bind id:3 - ITS_inside:10.1.1.51/6101 to ITS-outside:10.0.1.224/135.

DCERPC-EV: ISystemActivator UUID found

DCERPC-EV: bind with ctx_num:1

DCERPC-EV: retrieve ctx_id: 1, if-uuid: 000001a0

DCERPC-PKT: bind_ack id:3 - ITS-outside:10.0.1.224/135 to ITS_inside:10.1.1.51/6101.

DCERPC-EV: bind_ack with ctxnum_result:1

DCERPC-EV: ctxid/result:1/0 accepted, ack_reason:0 tsyn: 8a885d04

DCERPC-PKT: auth id:3 - ITS_inside:10.1.1.51/6101 to ITS-outside:10.0.1.224/135.

DCERPC-PKT: request id:3 - ITS_inside:10.1.1.51/6101 to ITS-outside:10.0.1.224/135.

DCERPC-EV: valid_ctxid: alt:0 ctxnum:1 j:0 val:1 valid:1

DCERPC-PKT: request with opnum:4 call_id:3.

DCERPC-PKT: response id:3 - ITS-outside:10.0.1.224/135 to ITS_inside:10.1.1.51/6101.

DCERPC-EV: RCI :: Found ORPC_THAT

DCERPC-EV: RCI :: Found OBJ_REF_CUSTOM

DCERPC-EV: RCI:: Prop Sizes = 240

DCERPC-EV: RCI:: Prop Sizes = 600

DCERPC-EV: RCI :: Found Prop_out property

DCERPC-EV: RCI :: found OBJREF_STANDARD

DCERPC-EV: RCI :: Found Embedded IP address = 10.0.1.224

DCERPC-EV: pre NAT/PAT: intercepted 10.0.1.224:0.

DCERPC-EV: post NAT/PAT: 10.0.1.224:0 translated to 10.5.20.224:0.

DCERPC-EV: RCI :: Found Embedded IP address = 10.0.1.224 Port = 704

DCERPC-EV: pre NAT/PAT: intercepted 10.0.1.224:49154.

DCERPC-EV: post NAT/PAT: 10.0.1.224:49154 translated to 10.5.20.224:49154.

DCERPC-EV: emb_timeout 0:03:00

DCERPC-EV: pinhole already exists TCP 10.1.1.51/0 to 10.0.1.224:49154.

DCERPC-EV: RCI :: Length changed after nat message Original Length: 1032, New Length: 1048

DCERPC-PKT: updated checksum and forward packet.

Thanks in advance!

Jeff

Jouni Forss · ‎12-05-2013

Hi,

Can you issue a "packet-tracer" command for the traffic that the logs mentions having Asymmetric NAT rules?

packet-tracer input ITS_inside tcp 10.1.1.51 12345 10.0.1.224 135

Though now that I look at it closely it seems to me that the target address should rather be 10.5.20.224 since that is defined as the NAT address for the destination in the "nat" command.

nat (ITS_inside,ITS-outside) source static Orion-srv Orion-srv-ext destination static PerryCo-224 PerryCo-224-ext no-proxy-arp

nat (ITS_inside,ITS-outside) source static destination static

- Jouni

jtowry · ‎12-05-2013

Hi Jouni,

I did a packet trace and it works. In the DCERPC debug the packet that it says the original length is 1032 and the new length is 1048 shows up as a malformed packet in the wireshark trace.When I attempt WMI to the real address this doesnt happen. Thanks.

packet-tracer input ITS_inside tcp 10.1.1.51 12345 10.5.20.224 135

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

MAC Access list

Phase: 2

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

nat (ITS_inside,ITS-outside) source static Orion-srv Orion-srv-ext destination static PerryCo-224 PerryCo-224-ext no-proxy-arp

Additional Information:

NAT divert to egress interface ITS-outside

Untranslate 10.5.20.224/135 to 10.0.1.224/135

Phase: 3

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group ITS-inside in interface ITS_inside

access-list ITS-inside extended permit ip any4 any4

Additional Information:

Phase: 4

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (ITS_inside,ITS-outside) source static Orion-srv Orion-srv-ext destination static PerryCo-224 PerryCo-224-ext no-proxy-arp

Additional Information:

Static translate 10.1.1.51/12345 to 75.141.84.6/12345

Phase: 5

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 6

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 7

Type: INSPECT

Subtype: inspect-dcerpc

Result: ALLOW

Config:

class-map MSRPC

match port tcp eq 135

policy-map global_policy

class MSRPC

inspect dcerpc MSRPC-MAP

service-policy global_policy global

Additional Information:

Phase: 8

Type: FOVER

Subtype: standby-update

Result: ALLOW

Config:

Additional Information:

Phase: 9

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:

nat (ITS_inside,ITS-outside) source static Orion-srv Orion-srv-ext destination static PerryCo-224 PerryCo-224-ext no-proxy-arp

Additional Information:

Phase: 10

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 11

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 12

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 11478337, packet dispatched to next module

Result:

input-interface: ITS_inside

input-status: up

input-line-status: up

output-interface: ITS-outside

output-status: up

output-line-status: up

Action: allow

Jouni Forss · ‎12-05-2013

Hi,

To be honest I can't tell what the problem is since I am not familiar on the actual connection you are trying to form and debug.

I would imagine it might have something to do with the "inspect" itself?

As you said, the "packet-tracer" goes through but I assume that the "inspect" does something to this traffic in between that makes it fail. Is the "inspect" supposed to be essential for this connection to work or can it even be removed?

Dont know if I can be much help with this as I dont know anything about the actual connection and how its supposed to behave.

- Jouni

knakke · ‎06-04-2014

Good Morning,

I'm having the same problem. To resolve, we had to implement a double nat as a work around until we can find a long term solution.

I wish there was a better way to implement it. The issue resides in the DCOM procedure that WMI uses to connect to the remote machine.

Here is a link to the MS KB: http://support.microsoft.com/kb/248809 explaining how DCOM affects the nat.

Here is a summary of what I saw happening on wire shark here is what I saw happening.

1) Server 1 initiates WMI call to Remote Client NAT Address

2) Remote Client responds to Server 1.

3) Server 1 responds with to Remote Client using the machines real NIC IP Address.

Just a note:

Also added the Remote Client NAT Address to Server 1 host file to see if WMI would use that address, and it doesn't. It appears the DCOM only uses the real NIC IP Address of the Remote Client machine.

I hope this helps.