FTP and WWW services aren't accessible externally, but are on the LAN

[rgould001]

I have Lubuntu 14.04 LTS with vsftpd and the lamp stack (apache2, etc.) installed. Everything works fine on the LAN. Externally, the services don't acknowledge connection attempts. I can see packets reaching the machine with `tcpdump-i eth0 -A`, but the machine doesn't send a corresponding ACK. `sudo ufw status` reports "Status:inactive". I don't have any rules configured in "iptables"; `iptables -L` has blank output.

To make matters more confusing, I can use the external WAN IP assigned to that machine and the hostname from inside the network and connections are successful. To use the WAN IP and hostname externally, I get the same behavior of Ubuntu not sending ACK packets.

    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      1494/mysqld
    tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      1277/dnsmasq
    tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      984/vsftpd
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      977/sshd
    tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      669/cupsd
    tcp6       0      0 :::80                   :::*                    LISTEN      2175/apache2
    tcp6       0      0 :::22                   :::*                    LISTEN      977/sshd
    tcp6       0      0 ::1:631                 :::*                    LISTEN      669/cupsd

Without divulging too much information, the LAN is a class A private 10.x.x.x/22 network. The server has one NIC assigned a static address in the business VLAN. The server's other NIC has a private class C 192.168.x.x/24 static address in the DMZ.

An nmap scan of both internal addresses shows identical output, shown below:

    Nmap scan report for 192.168.x.x/10.x.x.x
    Host is up (0.0018s latency).
    Not shown: 996 closed ports
    PORT     STATE SERVICE
    21/tcp   open  ftp
    22/tcp   open  ssh
    80/tcp   open  http
    3306/tcp open  mysql
    Nmap done: 1 IP address (1 host up) scanned in 0.36 seconds

A scan on the external IP shows all ports as "filtered", meaning nmap didn't get a SYN/ACK or ACK response.

    Starting Nmap 5.21 ( http://nmap.org ) at 2015-02-10 06:28     UTC
    Host is up.
    PORT     STATE    SERVICE
    21/tcp   filtered ftp
    22/tcp   filtered ssh
    80/tcp   filtered http
    3306/tcp filtered mysql

[David paull]

Does the device know where to send the acknowledgement to?

Sounds like a server type issue to me really -- hopefully someone can help.
 

[rgould001]

The server acknowledges devices on the LAN, but not externally/from the internet. I enabled MySQL for remote usage, made the necessary firewall access-list changes, and watched it not ACK the MySQL packets.

[David paull]

I'm aware of the successful ACK when sending to devices on the LAN.

So what I'm hearing is the following:

A server responds properly to 10.x.x.x (some local address)

A server doesn't respond properly to 59.x.x.x (some internet address)

I'll formulate my question differently -- does the device know where to send a packet destined for the internet -- or just where to send a packet destined for the local network?

--

When you say that you "watch it not ACK" --

How are you watching it not ACK? 

Are you capturing what's leaving that server's NIC? 

Are you capturing an input interface on a device connected to that server?

Are you capturing on some firewall a few hops away?

Are there ACL's on any other device?

--

You've got to realize that you've done one of two things here:

Presented a firewalling problem in the proper forum and supplied application / server data, but no routing/firewalling data.

Presented an application / server problem in a firewalling forum.

--

I promise I'm trying to help but realize most of the network engineers on my team literally can't configure Microsoft Outlook, let alone explain or understand how a mysql server works.  Edited to add: When we suspect a server issue, our answer is to have someone reboot that server, which actually works a good bit of the time.

--

Run a capture on the inside and outside interfaces of your firewall.  I would like to see the packet going into your device, coming out of your device, and never getting anything in response.

[rgould001]

Thank you for the thorough response. I'll address each item as much as possible. This ended up being a server issue, not a firewall issue. Since you took the time to respond, so will I.

I temporarily put a Windows 7 laptop in place of the server, installed MySQL, VNC, and FTP services, and was able to access said services without problems. This lead me to believe it was an issue with Ubuntu. I ran nmap scans after adding each access-list statement and each one was listed as "open", leading me to believe the statements are correct. After reading your questions below, one comment lead me to experiment with changing the default route.

My responses to your statements are in red.

So what I'm hearing is the following:
A server responds properly to 10.x.x.x (some local address) Correct
A server doesn't respond properly to 59.x.x.x (some internet address) Correct

I'll formulate my question differently -- does the device know where to send a packet destined for the internet -- or just where to send a packet destined for the local network?
If by "does the device know where to send a packet destined for the internet" you meant something like, "what's the default route? via NIC1 on 10.x.x.x or NIC2 on 102.168.x.x?", then the default route was over the LAN, not the DMZ. I disconnected from the LAN and the default route changed to the DMZ NIC. Everything worked.

When you say that you "watch it not ACK", How are you watching it not ACK?, Are you capturing what's leaving that server's NIC?, Are you capturing an input interface on a device connected to that server?, Are you capturing on some firewall a few hops away?, Are there ACL's on any other device? - I was running tcpdump -i eth1 -A, a command-line equivalent of of Wireshark. I think in my earlier post I had "-i eth0", but that was a typo; I was listening on the correct interface. There are no ACLs anywhere else.

You've got to realize that you've done one of two things here:
Presented a firewalling problem in the proper forum and supplied application / server data, but no routing/firewalling data. The firewall access-list statements are configured correctly. I verified this by putting a windows 7 laptop in place of the server and the services I installed communicated properly on the internet. I was able to enable and disable ACLs, which I then verified with the appropriate nmap scans.

Presented an application / server problem in a firewalling forum. This ended up being a server problem. The default route was on the LAN, and not on the DMZ as it should have been.

Run a capture on the inside and outside interfaces of your firewall.  I would like to see the packet going into your device, coming out of your device, and never getting anything in response. - I work in the news industry, so I am unable to interrupt communication on the firewall, as we have feeds coming into the network constantly. I ran the capture on the server itself, listening on the DMZ NIC. I used tcpdump -i eth1 -A. When attempting service connections externally, from the internet, I saw SYN packets reach the server, but never saw the server respond with SYN/ACK or ACK packets. When I attempted the same connections to the server from the LAN or another host on the DMZ, with either the DNS name or IP address, the server received SYN packets and successfully sent SYN/ACK and ACK packets. After changing the default route, the tcpdump output shows the proper SYN, SYN/ACK, ACK communication. .