Solved: I was able to test this idea

hreftools2014cisco · ‎12-17-2014

Hi...

My problem is: outgoing tcp is blocked on almost all ports, and I have not configured any rules in the router to cause that. That probably sounds impossible so let me give you the background.

0. Upstream ISP (gi0/1) has confirmed they are not blocking any ports. Cisco 1941 K9. Version dump is below. Moreover, this is how simple the situation is:

Embedded-Service-Engine0/0 is administratively down, line protocol is down
GigabitEthernet0/0 is up, line protocol is up
  Outgoing access list is not set
  Inbound  access list is not set
GigabitEthernet0/1 is up, line protocol is up
  Outgoing access list is not set
  Inbound  access list is 110

Also please note that there are zero known problems with inbound traffic from public internet to the servers in the cabinet ("LAN"). The only problem is with outbound traffic, from the servers to remote public systems. There are no additional hardware or software firewall layers involved.

1. I know Windows really well. The Windows firewall is open. Tests within the LAN all work. Tests from 3 machines within the LAN all give the same result. I have tried opening some sample ports explicitly in the Windows firewall and that makes no difference. I have tested with the Windows anti-malware software temporarily suspended. Always the same result; outgoing connections from LAN to public internet fail on most ports.

2. This is how I'm checking whether the ports are open:

telnet portquiz.net 465

telnet portquiz.net 8088

telnet portquiz.net 3389

Those all fail to connect.

telnet portquiz.net 80 ---- works

Surprisingly (to me), outbound traffic to a few ports does work. Port 25 works. Port 80 works. Port 53 works. I do not know a quick way to test all the outbound ports so I'll just summarize: most outbound ports are blocked and a few are open.

3. I am seriously wondering whether this is a bug in the version of IOS that I have (details below). I have read the bug report CSCtt19027 and that does not list my IOS version, and, as far as I can tell, my inbound ACL rules are working. Perhaps it is an "related" bug. This is a production router etc etc and I am loathe to change the IOS version because there seem to be different types of problems at every release and I don't have the confidence to fix anything fast enough. However if someone wants to convince me otherwise, I am listening.

The workaround for CSCtt19027 is to install the "security license." How? Should I go down that path?

I have read http://www.cisco.com/c/en/us/td/docs/ios/15_1/release/notes/15_1m_and_t/151-4MCAVS.html ; and I do not see an exact match for my situation. I did notice there was trouble reported when using a range, and I do have a range as part of my inbound rules.

I can provide my inbound ACL details if anything thinks that can matter. It seems to matter - but from a theoretical point of view - it should not. Right???!???

4. I don't have netflow software / analysis tools going (yet). What is the easiest way to confirm that it is the Cisco router cutting off this outbound traffic, keeping in mind this is a production system and I do not want to overload/crash the router.

Thank you for reading this far. Looking forward to your comments. The 'show version' info follows.

#show version
Cisco IOS Software, C1900 Software (C1900-UNIVERSALK9-M), Version 15.1(4)M4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 20-Mar-12 17:58 by prod_rel_team

ROM: System Bootstrap, Version 15.0(1r)M15, RELEASE SOFTWARE (fc1)

 uptime is 1 week, 4 days, 21 hours, 8 minutes
System returned to ROM by reload at 03:40:30 UTC Sat Dec 6 2014
System restarted at 03:41:55 UTC Sat Dec 6 2014
System image file is "flash0:c1900-universalk9-mz.SPA.151-4.M4.bin"
Last reload type: Normal Reload
Last reload reason: Reload Command


This product contains cryptographic features (... snip ....)

Cisco CISCO1941/K9 (revision 1.0) with 491520K/32768K bytes of memory.
Processor board ID FTX164481T7
2 Gigabit Ethernet interfaces
1 terminal line
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity disabled.
255K bytes of non-volatile configuration memory.
250880K bytes of ATA System CompactFlash 0 (Read/Write)

(snip)


Technology Package License Information for Module:'c1900'

-----------------------------------------------------------------
Technology    Technology-package           Technology-package
              Current       Type           Next reboot  
------------------------------------------------------------------
ipbase        ipbasek9      Permanent      ipbasek9
security      securityk9    Permanent      securityk9
data          None          None           None

Configuration register is 0x2102

Karsten Iwen · ‎12-17-2014

Do you have configured one of the firewall-features (CBAC or ZBF) on your router? If not, then it could be that the answer-packets to your communication can't flow back because your ACL 110 is blocking it. Test the following:

ip inspect name FW tcp router-traffic
ip inspect name FW udp router-traffic
ip inspect name FW icmp router-traffic
ip inspect name FW ftp
int gig 0/1
 ip inspect FW out

With that a very basic statefull firewall is enabled on the router and return-traffic is allowed automatically. With that your ACL 110 should be cleaned up and shouldn't include any entries for return-traffic. Please share that ACL.

For this statefull inspection you need the security license, but you have that installed on your router.

View solution in original post

Jon Marshall · ‎12-20-2014

From your original post -

Surprisingly (to me), outbound traffic to a few ports does work. Port 25 works. Port 80 works. Port 53 works

that doesn't surprise me because if you look at your acl you have specific entries for the above ports for traffic in both directions eg.

access-list 110 permit tcp any any eq www
access-list 110 permit tcp any eq www any

so when an internal client requests a web page the return traffic matches the second entry.

Where it is not working is where in your acl you have not specified two entries per port but have relied on the object-group feature to do that for you.

Now I have never used object-groups in IOS but from a quick read my understanding is that it should open the port both ways so you don't have to explicitly have entries for traffic both ways.

So, either my understanding of how object-groups works is wrong or they are simply not working as they should be but I think that is your issue.

As a quick test pick one of the ports that isn't working and have explicit entries for the traffic both ways, as above, and see if it works.

If it does then it is definitely an issue with the object-group in your acl.

Jon

View solution in original post

Karsten Iwen · ‎12-17-2014

Do you have configured one of the firewall-features (CBAC or ZBF) on your router? If not, then it could be that the answer-packets to your communication can't flow back because your ACL 110 is blocking it. Test the following:

ip inspect name FW tcp router-traffic
ip inspect name FW udp router-traffic
ip inspect name FW icmp router-traffic
ip inspect name FW ftp
int gig 0/1
 ip inspect FW out

With that a very basic statefull firewall is enabled on the router and return-traffic is allowed automatically. With that your ACL 110 should be cleaned up and shouldn't include any entries for return-traffic. Please share that ACL.

For this statefull inspection you need the security license, but you have that installed on your router.

hreftools2014cisco · ‎12-18-2014

My config does not mention CBAC nor ZBF.

I do not understand why this involves return-traffic. The traffic originates on the LAN side of the router (gi 0/0) and tries to out the public/ISP side (gi0/1).

Even if I test a port such as 990, which is allowed inbound, the outbound telnet test fails.

telnet portquiz.net 990

I will read about the "ip inspect" feature - thank you for the tip.

hreftools2014cisco · ‎12-18-2014

I found https://learningnetwork.cisco.com/docs/DOC-7832 talking about "ip inspect".... ah ha. I will have more news in ~12-24 hours. I will mark your answer as the correct one if it pans out. THANK YOU.

hreftools2014cisco · ‎12-19-2014

After reading more about CBAC (Context Based Access Control), I do not think it is the solution for me. Let me try a picture:

Most of the traffic in this story originates in the public internet, comes in to one of the servers, and gets a reply. That all works.

A tiny amount of traffic originates on a server and tries to get out to the public internet. That is failing. An example would be our monitoring software that checks whether Remote Desktop is running on a remote public machine in the Amazon cloud by trying a connection on port 3389. That is failing. Connections to portquiz.net where they promise that all ports answer, all the time, also fail.

I have deliberately tested tcp ports where I know the inbound traffic arrives fine, for example, 3389 or 8088. I don't see how "inspecting" the outbound traffic is going to make it any more likely for the inbound traffic to arrive, since packets are already arriving on the inbound side.

I think an IOS bug is preventing the outbound traffic but I do not know how to prove that.

Comments welcome -

hreftools2014cisco · ‎12-20-2014

This is what my ACL 110 has:

access-list 110 permit ip any object-group ABC
access-list 110 permit object-group RequiredPorts any any
access-list 110 permit udp any any eq domain
access-list 110 permit udp any eq domain any
access-list 110 permit tcp any any eq domain
access-list 110 permit tcp any eq domain any
access-list 110 permit icmp any any echo-reply
access-list 110 permit icmp any any echo
access-list 110 permit icmp any any source-quench
access-list 110 permit icmp any any packet-too-big
access-list 110 permit icmp any any time-exceeded
access-list 110 deny   icmp any any
access-list 110 permit tcp any any eq www
access-list 110 permit tcp any eq www any
access-list 110 permit tcp any eq 443 any
access-list 110 permit ip 208.x.x.0 0.0.0.255 any
access-list 110 permit tcp any eq ftp any
access-list 110 permit tcp any eq ftp-data any
access-list 110 permit tcp any eq smtp any
access-list 110 permit tcp any any eq smtp

This is what my requiredports list looks like:

object-group service RequiredPorts
 description Inbound TCP and UDP ports required at cabinet F13
 tcp eq ftp-data
 tcp eq ftp
 tcp eq 22
 tcp eq domain
 tcp eq www
 tcp eq 443
 tcp eq nntp
(snip)
 tcp eq 3389
 tcp eq 6389
 tcp range 8000 8099
 tcp eq 8169
(snip)
 udp eq domain

I do not think many people use this syntax,

access-list 110 permit object-group RequiredPorts any any

but I read it in a book and liked it because it seemed self-documenting. Maybe it's wrong or maybe it doesn't really work and that's why people don't use it. Comments welcome. I did realize afterwards that it was tricky to edit the list of required ports.

Still, I don't have any evidence to believe I have a problem with inbound traffic.

Thank you, forum readers.

Karsten Iwen · ‎12-20-2014

With the proper inspection you could remove all the lines for returning traffic which would clean up your ACL and make it much more secure.

Up to now I never used the object groups on a router in that way, can't tell if it really works. But I would expect that you allow only destination-ports what would be wrong.

For your bug-theory, just upgrade to a more recent version.

Jon Marshall · ‎12-20-2014