01-22-2012 02:13 PM - edited 03-07-2019 04:29 AM
Hello, I think I have run into a massive bug with IOS 15.
I've been trying to get a 1941 (SEC-K9) router that has a 4-port EHWIC-4ESG-P module configured for a few days now.
I have everything working as planned - SMTP, IMAP, DNS are all fine from the outside to the internal DMZ hosts - with the exception of serving traffic from the outside to an internal DMZ HTTP web server. NAT is setup statically -- I can establish a connection from outside the network to the internal server on TCP 80, the HTTP GET is sent, and the web server DOES try and send the return traffic (I have verified this with tcpdump on the web server itself) .. BUT, the router does not forward this return traffic back to the remote client.
Equally strange, with some of the web pages that have only a few lines of HTML, the router will pass the HTML back to the client but NOT any of the images. In fact, when the remote clients attempt to hit http://site.tld/some_image.jpg .. the connection opens, but just hangs.
One other thing to mention, I CAN complete HTTP requests from the LAN > DMZ web server, and the pages load correctly as they should.
I'm totally at a loss, I had a CCNP briefly looks at the problem, and all he said was he'd never seen this behavior before and thought it might be a bug.
Really strange.. here are the relevant bits in my config:
version 15.2
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime localtime
service password-encryption
service sequence-numbers
!
hostname rtr
!
boot-start-marker
boot system flash0 c1900-universalk9-mz.SPA.152-2.T.bin
boot-end-marker
!
!
security authentication failure rate 10 log
security passwords min-length 6
logging userinfo
logging buffered 51200 warnings
no logging console
no logging monitor
!
!
clock timezone Pacific -8 0
!
no ipv6 cef
no ip source-route
no ip gratuitous-arps
ip auth-proxy max-login-attempts 5
ip admission max-login-attempts 5
no ip domain lookup
ip name-server 4.2.2.2
ip port-map user-submission port tcp 587
ip port-map user-rsync port tcp 873
ip port-map user-apc port tcp 3551
ip port-map user-smtps port tcp 465
ip port-map user-imaps port tcp 44993
ip inspect log drop-pkt
ip inspect audit-trail
ip inspect udp idle-time 1800
ip inspect dns-timeout 7
ip inspect tcp idle-time 14400
no ip cef
!
multilink bundle-name authenticated
!
parameter-map type inspect global
log dropped-packets enable
no ip ftp passive
ip scp server enable
!
class-map type inspect match-any all-traffic-cm
match protocol http
match protocol https
match protocol user-rsync
match protocol user-smtps
match protocol user-submission
match protocol ftp
match protocol dns
match protocol ntp
match protocol ssh
match protocol tcp
match protocol udp
match protocol icmp
class-map type inspect match-any dmz-protocols-cm
match protocol http
match protocol smtp
match protocol user-smtps
match protocol user-submission
match protocol user-imaps
match protocol dns
match protocol tcp
class-map type inspect match-all out-dmz-cm
match class-map dmz-protocols-cm
match access-group name out-dmz-acl
class-map type inspect match-all permit-all-cm
match access-group name permit-all-acl
match class-map all-traffic-cm
!
policy-map type inspect out-dmz-pm
class type inspect out-dmz-cm
inspect
class class-default
drop log
policy-map type inspect permit-all-pm
class type inspect permit-all-cm
inspect
class class-default
drop log
!
zone security DMZ
zone security OUT
zone-pair security OUT_to_DMZ source OUT destination DMZ
service-policy type inspect out-dmz-pm
zone-pair security DMZ_to_OUT source DMZ destination OUT
service-policy type inspect permit-all-pm
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description $ETH-WAN$
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
zone-member security OUT
duplex auto
speed auto
pppoe enable group global
pppoe-client dial-pool-number 1
no mop enabled
!
interface GigabitEthernet0/1
description $ETH-SW-LAUNCH$$INTF-INFO-GE 0/0$$ETH-LAN$
ip address 10.10.10.1 255.255.255.248
ip access-group 100 in
no ip redirects
no ip unreachables
no ip proxy-arp
duplex auto
speed auto
no mop enabled
!
interface GigabitEthernet0/0/0
description LAN
switchport access vlan 2
switchport trunk native vlan 2
no ip address
no mop enabled
!
interface GigabitEthernet0/0/1
description DMZ
switchport access vlan 3
switchport trunk native vlan 3
no ip address
no mop enabled
!
interface GigabitEthernet0/0/2
no ip address
shutdown
!
interface GigabitEthernet0/0/3
no ip address
shutdown
!
interface Vlan1
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
no mop enabled
!
interface Vlan2
ip address 172.16.1.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly in
zone-member security LAN
ip tcp adjust-mss 1412
no mop enabled
!
interface Vlan3
ip address 172.16.98.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip flow ingress
ip flow egress
ip nat inside
ip virtual-reassembly in
zone-member security DMZ
no mop enabled
!
interface Vlan4
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly in
shutdown
no mop enabled
!
interface Dialer1
ip address X.X.X.28 255.255.255.248
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1492
ip flow ingress
ip flow egress
ip nat outside
ip virtual-reassembly in
zone-member security OUT
encapsulation ppp
dialer pool 1
ppp ..snip..
!
ip forward-protocol nd
!
no ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip flow-top-talkers
top 100
sort-by bytes
!
ip nat inside source list DMZ interface Dialer1 overload
ip nat inside source static tcp 172.16.98.4 25 X.X.X.23 25 extendable
ip nat inside source static tcp 172.16.98.4 53 X.X.X.23 53 extendable
ip nat inside source static udp 172.16.98.4 53 X.X.X.23 53 extendable
ip nat inside source static tcp 172.16.98.5 80 X.X.X.23 80 extendable
ip nat inside source static tcp 172.16.98.4 465 X.X.X.23 465 extendable
ip nat inside source static tcp 172.16.98.4 587 X.X.X.23 587 extendable
ip nat inside source static tcp 172.16.98.4 993 X.X.X.23 993 extendable
ip nat inside source static tcp 172.16.98.4 993 X.X.X.23 44993 extendable
ip nat inside source static tcp 172.16.98.11 25 X.X.X.24 25 extendable
ip nat inside source static 172.16.98.11 X.X.X.24
ip nat inside source static tcp 172.16.98.4 53 X.X.X.25 53 extendable
ip nat inside source static udp 172.16.98.4 53 X.X.X.25 53 extendable
ip nat inside source static tcp 172.16.98.5 80 X.X.X.25 80 extendable
ip nat inside source static tcp 172.16.98.4 53 X.X.X.27 53 extendable
ip nat inside source static udp 172.16.98.4 53 X.X.X.27 53 extendable
ip nat inside source static tcp 172.16.98.5 80 X.X.X.27 80 extendable
!
ip access-list standard DMZ
permit 172.16.98.0 0.0.0.255
!
ip access-list extended out-dmz-acl
permit tcp any host 172.16.98.4 eq smtp
permit tcp any host 172.16.98.4 eq domain
permit udp any host 172.16.98.4 eq domain
permit tcp any host 172.16.98.5 eq www
permit tcp any host 172.16.98.11 eq smtp
permit tcp any host 172.16.98.4 eq 465
permit tcp any host 172.16.98.4 eq 587
permit tcp any host 172.16.98.4 eq 36124
permit tcp any host 172.16.98.11 eq 5223
ip access-list extended permit-all-acl
permit ip any any
!
no cdp run
!
!
!
!
control-plane
!
!
!
line con 0
exec-timeout 0 0
logging synchronous
transport output none
line aux 0
exec-timeout 0 0
transport output none
line 2
exec-timeout 0 0
no activation-character
no exec
transport preferred none
transport input all
transport output none
stopbits 1
line vty 0 4
access-class 23 in
privilege level 15
logging synchronous
transport input ssh
transport output ssh
line vty 5 15
access-class 23 in
privilege level 15
logging synchronous
login authentication local_auth
transport input ssh
transport output ssh
!
scheduler allocate 20000 1000
!
end
01-22-2012 11:43 PM
I'm wondering now if there is some issue triyng to NAT one of the PPPoE static (inside global) address I have been allocated. EDIT: Actually, that doesn't make sense as I can translate them to my mail and dns servers just fine.
Another thought is, even though I do not have the router's http server running, could the router's http server be having some undesirable effect?
I just do not comprehend why this works:
ip nat inside source static tcp 172.16.98.4 25 X.X.X.23 25 extendable
and this does not work:
ip nat inside source static tcp 172.16.98.5 80 X.X.X.23 80 extendable
It's extremely frustrating .. I have even tested changing the web server's listening port to 8888 and tried:
ip nat inside source static tcp 172.16.98.5 8888 X.X.X.23 80 extendable
..but the router still kills the return traffic somehow/somewhere
01-23-2012 01:24 AM
Hi,
I have two suggestions:
- In the config, I can see that CEF is disabled, any reason for that? If not then lets enable it and then test again
- As the NAT config seems to be Ok, still the webservices are not working in the way it should, I would suggest you to disable CBAC/Zone Based Firewall from the configuration as a testing step and then check again. This might help you in narrowing down the issue to ZBF or NAT. I would have asked you to run the debugs but somehow I have seen many a times ZBF can have this effect so if possible, lets disable it from the inside and outside interfaces and then check if your HTTP server responds back to internet users
Hope it helps.
Neeraj
01-23-2012 08:05 PM
Hi Neeraj:
I'm fortunate that Cisco TAC was able to resolve the issue. The problem turned out to be a hardcoded MTU of 1492 on the Dialer interface, which, when removed fixed the issue right away.
This MTU value was recommended by the ISP however for the PPPoE connection.
Thank you
01-23-2012 11:38 PM
Thanks for posting the resolution...it turned out to be a minor thing causing a major issue
I'll keep this in mind for future. thanks again.
Neeraj
01-24-2012 12:25 AM
Hello seattleit,
Hmmm, I am glad to see your issue resolved but I am confused by the solution the TAC provided. The IP MTU of 1492 is indeed the correct MTU for PPPoE interfaces. If this is increased, the resulting datagrams may exceed the maximum size of an Ethernet frame (the 'E' in PPPoE). So the TAC suggested to remove the ip mtu command from your Dialer without any further caring?
Best regards,
Peter
01-24-2012 12:46 AM
Hi Peter,
Yes, once the no ip mtu on the Dialer was removed, the web server was able to negotiate MSS and the http traffic began to serve. I also find this a bit odd, as I thought I configured the Dialer interface correctly with an MTU value of 1492.
This was definitely a painful troubleshooting experience, as I had spent many hours and packet captures trying to debug the configuration. Thankfully I gave in and opend the SR with TAC, but I will do some testing in the next day or two to make sure traffic is running optimally.
Cheers
01-24-2012 12:52 AM
Hello,
Personally, I would be willing to suggest a few more tests with MTU, MSS clamping and ICMP permissions as I find this issue utterly odd - but I do not want to break your working config and do not want to cause you any more outages.
Best regards,
Peter
01-24-2012 09:35 AM
Hi Peter,
I just ran a few tests in clamping the MSS down to 1452 and allowing ICMP and it seems to work. I have to head into the office and will tweak this a little more when I get home this evening.
Best
01-26-2012 06:22 AM
A side note on the MTU configuration - you might want to try setting mtu 1492 instead of ip mtu 1492 on the dialer. The former should cause PPP to correctly negotiate the link MTU as part of the LCP negotiations. I have had similar experiences, and after switching from the latter to the former they cleared up. It's likely that your ISP just NACK's any requests for larger MRRU from your router and forces the negotiation to use 1492. debug ppp nego will show all the gory details.
01-26-2012 06:33 AM
When I read the original post my first thought was that it sounded like a problem with MTU. And I was interested to read what seems to have solved the problem. I have a different suggestion about how the problem might have been dealt with. I notice that this command is on VLAN 2 (LAN) but not on VLAN 3 (DMZ)
ip tcp adjust-mss 1412
I suggest that putting this command on the DMZ interface would also have resolved the problem.
HTH
Rick
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide