ASA5510 SMTP problems

Adam Hudson · ‎07-09-2012

Up until recently one of my sites was able to get to a postini subnet. Then we started recieving "host unreachable" e-mails. Postini told us SMTP traffic was not getting let in. I've compared the current config to a config that was saved before the issue popped up and found really no noticeable difference.

I tried a packet tracer trace with no luck:

==============================

SiteB-Firewall# packet-tracer input outside tcp 11.2.2.36 12345 65.19.0.0 25

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 0.0.0.0 0.0.0.0 outside

Phase: 3

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: outside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

============================================

Attached is a sanitized config. I'm not entirely convinced it's a firewall issue, but I need to some successful testing to prove otherwise.

Julio Carvajal · ‎07-09-2012

Hello,

So only the subnet 65.19.0.0 255.255.240.0 should be able to access the SMTP server?

The packet tracer is not properly build

Try this one

packet-tracer input outside tcp 65.19.0.30 1025 25.107.253.3 eq 25

Right now your ASA is setup to allow connections only from 65.19.0.0 255.255.240.0 to the SMTP server.

Rate all the helpful posts

Julio

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

Adam Hudson · ‎07-10-2012

Julio, yes, only that subnet should be able to reach my SMTP server.

Here's the packet trace:

SiteB-Firewall# packet-tracer input outside tcp 65.19.0.30 1025 25.107.$

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 2

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

static (inside,outside) tcp interface smtp 11.2.2.36 smtp netmask 255.255.255.255

nat-control

match tcp inside host 11.2.2.36 eq 25 outside any

static translation to 25.107.253.3/25

translate_hits = 0, untranslate_hits = 9453

Additional Information:

NAT divert to egress interface inside

Untranslate 25.107.253.3/25 to 11.2.2.36/25 using netmask 255.255.255.255

Phase: 3

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group incoming in interface outside

access-list incoming extended permit tcp object-group Postini interface outside eq smtp

object-group network Postini

network-object 65.19.0.0 255.255.240.0

Additional Information:

Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 5

Type: VPN

Subtype: ipsec-tunnel-flow

Result: ALLOW

Config:

Additional Information:

Phase: 6

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:

static (inside,outside) tcp interface smtp 11.2.2.36 smtp netmask 255.255.255.255

nat-control

match tcp inside host 11.2.2.36 eq 25 outside any

static translation to 25.107.253.3/25

translate_hits = 0, untranslate_hits = 9453

Additional Information:

Phase: 7

Type: NAT

Subtype: host-limits

Result: ALLOW

Config:

static (inside,outside) tcp interface smtp 11.2.2.36 smtp netmask 255.255.255.255

nat-control

match tcp inside host 11.2.2.36 eq 25 outside any

static translation to 25.107.253.3/25

translate_hits = 0, untranslate_hits = 9453

Additional Information:

Phase: 8

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 9

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 41247, packet dispatched to next module

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: allow

If I'm understanding this right, we're testing to make sure the outside interface and my postini subnet can talk. That looks like that was successful, that's good, but only half of the communication.

Testing the other half of the coummuncation, if all of my assumptions have been correct, is making sure the outside interface passes the SMTP traffic back correctly to the inside network. Below is the packet trace I tried, it failed:

SiteB-Firewall# packet-tracer input inside tcp 24.106.253.3 1025 11.2.2$

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

MAC Access list

Phase: 2

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 3

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 11.2.2.0 255.255.255.0 inside

Phase: 4

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Phase: 5

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 6

Type: NAT

Subtype: host-limits

Result: ALLOW

Config:

nat (inside) 0 0.0.0.0 0.0.0.0

nat-control

match ip inside any BAD_INT_1 any

no translation group, implicit deny

policy_hits = 0

Additional Information:

Phase: 7

Type: NAT

Subtype: rpf-check

Result: DROP

Config:

nat (inside) 1 11.2.2.0 255.255.255.0

nat-control

match ip inside 11.2.2.0 255.255.255.0 inside any

dynamic translation to pool 1 (No matching global)

translate_hits = 0, untranslate_hits = 0

Additional Information:

Result:

input-interface: inside

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

What's our next step?

Adam Hudson · ‎07-10-2012

Just got a Mail Host Unreachable email from Postini so were the packet tracer results incorrect then?

Adam Hudson · ‎07-11-2012

Can anyone make heads or tails of that last packet tracer output?

Julio Carvajal · ‎07-11-2012

Hello Adam,

Sorry I could not respond this before, I was working on some other things.

You are using Port-forwarding to make this happen.

Por-forwarding is only used for incoming connections, for outbound connections the server will use a PAT or NAT rule.

So as we can see on packet tracer 1 everything looks good from the ASA perspective.

1-Traffic arrives on the outside interface on por 25

2-ASA checks the ACL and allows the packet

3-ASA does the right nat tranlastion

4-ASA creates an entry on the XLATE, CONN and Local-Host table

5-ASA send's the packet out the right interface

6-ASA receives the response from the SMTP server, based on the entry on the XLATE and CONN table he will perform the nat for the reply

7-Packet will reach the outside client

Why is the second PT not working?

A/Because as I said you have a port-forwarding rule for the SMTP server and that only works for incoming traffic

not outbound traffic ( to make it work you will need to add a Global statement but this will not solve the problem)

Please add the following

capture capout interface outside trace match tcp any host interface_ip eq 25

capture capin interface inside trace match tcp any host SMTP_SERVER_IP eq 25

Then generate the traffic and provide us the :

-Show cap capout

-Show cap capin

Regards,

Julio

Rate all the helpful posts

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

Adam Hudson · ‎07-11-2012

@jcarvaja I'll try that next.

Here's the test from the outside interface, it fails as well.

SiteB-Firewall# packet-tracer input outside tcp 24.106.253.3 1025 11.2.2.36 25 detail

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 11.2.2.0 255.255.255.0 inside

Phase: 3

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd7c09ef8, priority=11, domain=permit, deny=true

hits=39, user_data=0x5, cs_id=0x0, flags=0x0, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

What do I need to try next in this config?

Julio Carvajal · ‎07-11-2012

Hello Adam,

add the following

access-list incoming line 1 permit ip any host 11.2.2.36 25

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

Adam Hudson · ‎07-11-2012

Here's the results:

SiteB-Firewall# capture capout interface outside trace match tcp any host 25.107.253.3 eq 25

SiteB-Firewall# capture capin interface inside trace match tcp any host 11.2.2.36 eq 25

NOTE: I did not manually generate traffic here.

SiteB-Firewall# sh cap capin

2 packets captured

1: 14:26:39.200154 65.19.1.143.35442 > 11.2.2.36.25: S 1481583325:1481583325(0) win 5744

2: 14:26:39.682948 65.19.1.134.40522 > 11.2.2.36.25: S 688312206:688312206(0) win 5744

2 packets shown

SiteB-Firewall# sh cap capout

5 packets captured

1: 14:26:33.201710 65.19.1.143.35442 > 25.107.253.3.25: S 3750468370:3750468370(0) win 5744

2: 14:26:33.683085 65.19.1.134.40522 > 25.107.253.3.25: S 15701922:15701922(0) win 5744

3: 14:26:34.168051 65.19.1.159.33872 > 25.107.253.3.25: S 422177870:422177870(0) win 5744

4: 14:26:39.200062 65.19.1.143.35442 > 25.107.253.3.25: S 3750468370:3750468370(0) win 5744

5: 14:26:39.682871 65.19.1.134.40522 > 25.107.253.3.25: S 15701922:15701922(0) win 5744

5 packets shown

SiteB-Firewall#

====

SiteB-Firewall# SiteB-Firewall# packet-tracer input outside tcp 25.107.253.3 1025 11.2.2.36 25 detail

Phase: 1

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd84e46f0, priority=12, domain=capture, deny=false

hits=4963, user_data=0xd7c01278, cs_id=0x0, l3_type=0x0

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0000.0000.0000

Phase: 2

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd7b31b58, priority=1, domain=permit, deny=false

hits=129043, user_data=0x0, cs_id=0x0, l3_type=0x8

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0000.0000.0000

Phase: 3

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 4

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 11.2.2.0 255.255.255.0 inside

Phase: 5

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd7b375a0, priority=500, domain=permit, deny=true

hits=1, user_data=0x6, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip=25.107.253.3, mask=255.255.255.255, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

====

SiteB-Firewall# sh cap capin

10 packets captured

1: 14:26:39.200154 65.19.1.143.35442 > 11.2.2.36.25: S 1481583325:1481583325(0) win 5744

2: 14:26:39.682948 65.19.1.134.40522 > 11.2.2.36.25: S 688312206:688312206(0) win 5744

3: 14:26:51.200276 65.19.1.143.35442 > 11.2.2.36.25: S 1481583325:1481583325(0) win 5744

4: 14:26:51.680201 65.19.1.134.40522 > 11.2.2.36.25: S 688312206:688312206(0) win 5744

5: 14:26:58.167883 65.19.1.159.33872 > 11.2.2.36.25: S 691920151:691920151(0) win 5744

6: 14:27:15.197774 65.19.1.143.35442 > 11.2.2.36.25: S 1326979564:1326979564(0) win 5744

7: 14:27:15.677867 65.19.1.134.40522 > 11.2.2.36.25: S 330226058:330226058(0) win 5744

8: 14:27:46.166983 65.19.1.159.33872 > 11.2.2.36.25: S 1755337294:1755337294(0) win 5744

9: 14:28:03.191899 65.19.1.143.35442 > 11.2.2.36.25: S 465997866:465997866(0) win 5744

10: 14:28:03.670299 65.19.1.134.40522 > 11.2.2.36.25: S 1234871544:1234871544(0) win 5744

10 packets shown

SiteB-Firewall# sh cap capout

13 packets captured

1: 14:26:33.201710 65.19.1.143.35442 > 25.107.253.3.25: S 3750468370:3750468370(0) win 5744

2: 14:26:33.683085 65.19.1.134.40522 > 25.107.253.3.25: S 15701922:15701922(0) win 5744

3: 14:26:34.168051 65.19.1.159.33872 > 25.107.253.3.25: S 422177870:422177870(0) win 5744

4: 14:26:39.200062 65.19.1.143.35442 > 25.107.253.3.25: S 3750468370:3750468370(0) win 5744

5: 14:26:39.682871 65.19.1.134.40522 > 25.107.253.3.25: S 15701922:15701922(0) win 5744

6: 14:26:51.200200 65.19.1.143.35442 > 25.107.253.3.25: S 3750468370:3750468370(0) win 5744

7: 14:26:51.680125 65.19.1.134.40522 > 25.107.253.3.25: S 15701922:15701922(0) win 5744

8: 14:26:58.167639 65.19.1.159.33872 > 25.107.253.3.25: S 422177870:422177870(0) win 5744

9: 14:27:15.197530 65.19.1.143.35442 > 25.107.253.3.25: S 3750468370:3750468370(0) win 5744

10: 14:27:15.677653 65.19.1.134.40522 > 25.107.253.3.25: S 15701922:15701922(0) win 5744

11: 14:27:46.166769 65.19.1.159.33872 > 25.107.253.3.25: S 422177870:422177870(0) win 5744

12: 14:28:03.191670 65.19.1.143.35442 > 25.107.253.3.25: S 3750468370:3750468370(0) win 5744

13: 14:28:03.670070 65.19.1.134.40522 > 25.107.253.3.25: S 15701922:15701922(0) win 5744

13 packets shown

Julio Carvajal · ‎07-11-2012

Hello Adam,

Can you install wireshark on the server and run a capture?

Based on the captures packets are reaching the server but there is no reply

Regards,

Julio

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

Adam Hudson · ‎07-11-2012

Results:

SiteB-Firewall# packet-tracer input outside tcp 25.107.253.3 1025 11.2.2.36 25 detail

Phase: 1

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd84e46f0, priority=12, domain=capture, deny=false

hits=245849, user_data=0xd7c01278, cs_id=0x0, l3_type=0x0

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0000.0000.0000

Phase: 2

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd7b31b58, priority=1, domain=permit, deny=false

hits=131772, user_data=0x0, cs_id=0x0, l3_type=0x8

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0000.0000.0000

Phase: 3

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 4

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 11.2.2.0 255.255.255.0 inside

Phase: 5

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group incoming in interface outside

access-list incoming extended permit tcp any host 11.2.2.36 eq smtp

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd7b3c5f8, priority=12, domain=permit, deny=false

hits=0, user_data=0xd6874100, cs_id=0x0, flags=0x0, protocol=6

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=11.2.2.36, mask=255.255.255.255, port=25, dscp=0x0

Phase: 6

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd7b347d0, priority=0, domain=permit-ip-option, deny=true

hits=14315, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 7

Type: VPN

Subtype: ipsec-tunnel-flow

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd83106c8, priority=12, domain=ipsec-tunnel-flow, deny=true

hits=11490, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 8

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

out id=0xd84e3c28, priority=12, domain=capture, deny=false

hits=372, user_data=0xd7c01bf8, cs_id=0xd84e3538, reverse, flags=0x0, protocol=6

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=11.2.2.36, mask=255.255.255.255, port=25, dscp=0x0

Phase: 9

Type: NAT

Subtype: rpf-check

Result: DROP

Config:

static (inside,outside) tcp interface smtp 11.2.2.36 smtp netmask 255.255.255.255

nat-control

match tcp inside host 11.2.2.36 eq 25 outside any

static translation to 25.107.253.3/25

translate_hits = 0, untranslate_hits = 11143

Additional Information:

Forward Flow based lookup yields rule:

out id=0xd7bfb070, priority=5, domain=nat-reverse, deny=false

hits=11104, user_data=0xd7bfacf0, cs_id=0x0, flags=0x0, protocol=6

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=11.2.2.36, mask=255.255.255.255, port=25, dscp=0x0

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

Adam Hudson · ‎07-12-2012

@jcarvaja: That last capture was after adding the line: access-list incoming line 1 permit ip any host 11.2.2.36 25

To my ACL.

Do you see anything on the captures or packet-trace that would help me out or at least point me in the right direction?

Julio Carvajal · ‎07-12-2012

Hello Adam,

The packet tracer's and captures are not the right ones:

no cap capout

no cap capin

cap capout interface outside trace match tcp any host 11.255.2.1 eq 25

cap capin interface inside trace match tcp any host 11.2.2.36 eq 25

packet-tracer input tcp 25.107.253.3 1025 11.255.2.1 25

Regards,

Julio

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

Adam Hudson · ‎07-12-2012

cap capout interface outside trace match tcp any host 11.255.2.1 eq 25

cap capin interface inside trace match tcp any host 11.2.2.36 eq 25

SiteB-Firewall# packet-tracer input outside tcp 25.107.253.3 1025 11.255.2.1 25

Phase: 1

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

MAC Access list

Phase: 2

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

MAC Access list

Phase: 3

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 4

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 11.255.2.1 255.255.255.255 identity

Phase: 5

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: NP Identity Ifc

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

SiteB-Firewall# sh capture capin

0 packet captured

0 packet shown

SiteB-Firewall# sh capture capout

1 packet captured

1: 16:39:58.911543 24.106.253.3.1025 > 10.255.2.1.25: S 1576173544:1576173544(0) win 8192

1 packet shown

Julio Carvajal · ‎07-13-2012

Hello Adam,

I just reviewed the entire configuration one more time and I saw what is going on here.

Please remove the entire captures one more time:

no cap capin

no cap capout

You are trying to connect from the outside world to the following IP:11.255.2.1

-That ip belongs to the inside interface.

*****ASA speaking, you will not be able to access a distant interface***********

example: from a inside host you cannot ping or ssh or telnet the outside interface

example 2: from the outside world you will not be able to ping or ssh or telnet the outside interface

You will need to connect to this IP address 25.107.253.3

That is why you have : static (inside,outside) tcp interface smtp 11.2.2.36 smtp netmask 255.255.255.255

So:

1-access-list incoming permit tcp any host 11.2.2.36

2- packet-tracer input outside tcp 4.2.2.2 1025 25.107.253.3 and provide me the result

3- cap capout interface outside match tcp any host 25.107.253.3 eq 25

4-cap capin interface inside match tcp any host 11.2.2.36 eq 25

5-Generate real traffic

6-Send me the show cap capin, show cap capout

Julio

CSC it's a free support community take your time to rate all the engineer's responses that help you resolving your problems.

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC