cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3041
Views
4
Helpful
10
Replies

IOS 15 Bug?

seattleit
Level 1
Level 1

Hello, I think I have run into a massive bug with IOS 15.

I've been trying to get a 1941 (SEC-K9) router that has a 4-port EHWIC-4ESG-P module configured for a few days now.

I have everything working as planned - SMTP, IMAP, DNS are all fine from the outside to the internal DMZ hosts - with the exception of serving traffic from the outside to an internal DMZ HTTP web server. NAT is setup statically -- I can establish a connection from outside the network to the internal server on TCP 80, the HTTP GET is sent, and the web server DOES try and send the return traffic (I have verified this with tcpdump on the web server itself) .. BUT, the router does not forward this return traffic back to the remote client.

Equally strange, with some of the web pages that have only a few lines of HTML, the router will pass the HTML back to the client but NOT any of the images. In fact, when the remote clients attempt to hit http://site.tld/some_image.jpg .. the connection opens, but just hangs.

One other thing to mention, I CAN complete HTTP requests from the LAN > DMZ web server, and the pages load correctly as they should.

I'm totally at a loss, I had a CCNP briefly looks at the problem, and all he said was he'd never seen this behavior before and thought it might be a bug.

Really strange.. here are the relevant bits in my config:

version 15.2

no service pad

service tcp-keepalives-in

service tcp-keepalives-out

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime localtime

service password-encryption

service sequence-numbers

!

hostname rtr

!

boot-start-marker

boot system flash0 c1900-universalk9-mz.SPA.152-2.T.bin

boot-end-marker

!

!

security authentication failure rate 10 log

security passwords min-length 6

logging userinfo

logging buffered 51200 warnings

no logging console

no logging monitor

!

!

clock timezone Pacific -8 0

!

no ipv6 cef

no ip source-route

no ip gratuitous-arps

ip auth-proxy max-login-attempts 5

ip admission max-login-attempts 5

no ip domain lookup

ip name-server 4.2.2.2

ip port-map user-submission port tcp 587

ip port-map user-rsync port tcp 873

ip port-map user-apc port tcp 3551

ip port-map user-smtps port tcp 465

ip port-map user-imaps port tcp 44993

ip inspect log drop-pkt

ip inspect audit-trail

ip inspect udp idle-time 1800

ip inspect dns-timeout 7

ip inspect tcp idle-time 14400

no ip cef

!

multilink bundle-name authenticated

!

parameter-map type inspect global

log dropped-packets enable

no ip ftp passive

ip scp server enable

!

class-map type inspect match-any all-traffic-cm

match protocol http

match protocol https

match protocol user-rsync

match protocol user-smtps

match protocol user-submission

match protocol ftp

match protocol dns

match protocol ntp

match protocol ssh

match protocol tcp

match protocol udp

match protocol icmp

class-map type inspect match-any dmz-protocols-cm

match protocol http

match protocol smtp

match protocol user-smtps

match protocol user-submission

match protocol user-imaps

match protocol dns

match protocol tcp

class-map type inspect match-all out-dmz-cm

match class-map dmz-protocols-cm

match access-group name out-dmz-acl

class-map type inspect match-all permit-all-cm

match access-group name permit-all-acl

match class-map all-traffic-cm

!

policy-map type inspect out-dmz-pm

class type inspect out-dmz-cm

  inspect

class class-default

  drop log

policy-map type inspect permit-all-pm

class type inspect permit-all-cm

  inspect

class class-default

  drop log

!

zone security DMZ

zone security OUT

zone-pair security OUT_to_DMZ source OUT destination DMZ

service-policy type inspect out-dmz-pm

zone-pair security DMZ_to_OUT source DMZ destination OUT

service-policy type inspect permit-all-pm

!

interface Embedded-Service-Engine0/0

no ip address

shutdown

!

interface GigabitEthernet0/0

description $ETH-WAN$

no ip address

no ip redirects

no ip unreachables

no ip proxy-arp

zone-member security OUT

duplex auto

speed auto

pppoe enable group global

pppoe-client dial-pool-number 1

no mop enabled

!

interface GigabitEthernet0/1

description $ETH-SW-LAUNCH$$INTF-INFO-GE 0/0$$ETH-LAN$

ip address 10.10.10.1 255.255.255.248

ip access-group 100 in

no ip redirects

no ip unreachables

no ip proxy-arp

duplex auto

speed auto

no mop enabled

!

interface GigabitEthernet0/0/0

description LAN

switchport access vlan 2

switchport trunk native vlan 2

no ip address

no mop enabled

!

interface GigabitEthernet0/0/1

description DMZ

switchport access vlan 3

switchport trunk native vlan 3

no ip address

no mop enabled

!

interface GigabitEthernet0/0/2

no ip address

shutdown

!

interface GigabitEthernet0/0/3

no ip address

shutdown

!

interface Vlan1

no ip address

no ip redirects

no ip unreachables

no ip proxy-arp

no mop enabled

!

interface Vlan2

ip address 172.16.1.1 255.255.255.0

no ip redirects

no ip unreachables

no ip proxy-arp

ip nat inside

ip virtual-reassembly in

zone-member security LAN

ip tcp adjust-mss 1412

no mop enabled

!

interface Vlan3

ip address 172.16.98.1 255.255.255.0

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

ip flow egress

ip nat inside

ip virtual-reassembly in

zone-member security DMZ

no mop enabled

!

interface Vlan4

no ip address

no ip redirects

no ip unreachables

no ip proxy-arp

ip nat inside

ip virtual-reassembly in

shutdown

no mop enabled

!

interface Dialer1

ip address X.X.X.28 255.255.255.248

no ip redirects

no ip unreachables

no ip proxy-arp

ip mtu 1492

ip flow ingress

ip flow egress

ip nat outside

ip virtual-reassembly in

zone-member security OUT

encapsulation ppp

dialer pool 1

ppp ..snip..

!

ip forward-protocol nd

!

no ip http server

ip http access-class 23

ip http authentication local

ip http secure-server

ip http timeout-policy idle 60 life 86400 requests 10000

ip flow-top-talkers

top 100

sort-by bytes

!

ip nat inside source list DMZ interface Dialer1 overload

ip nat inside source static tcp 172.16.98.4 25 X.X.X.23 25 extendable

ip nat inside source static tcp 172.16.98.4 53 X.X.X.23 53 extendable

ip nat inside source static udp 172.16.98.4 53 X.X.X.23 53 extendable

ip nat inside source static tcp 172.16.98.5 80 X.X.X.23 80 extendable

ip nat inside source static tcp 172.16.98.4 465 X.X.X.23 465 extendable

ip nat inside source static tcp 172.16.98.4 587 X.X.X.23 587 extendable

ip nat inside source static tcp 172.16.98.4 993 X.X.X.23 993 extendable

ip nat inside source static tcp 172.16.98.4 993 X.X.X.23 44993 extendable

ip nat inside source static tcp 172.16.98.11 25 X.X.X.24 25 extendable

ip nat inside source static 172.16.98.11 X.X.X.24

ip nat inside source static tcp 172.16.98.4 53 X.X.X.25 53 extendable

ip nat inside source static udp 172.16.98.4 53 X.X.X.25 53 extendable

ip nat inside source static tcp 172.16.98.5 80 X.X.X.25 80 extendable

ip nat inside source static tcp 172.16.98.4 53 X.X.X.27 53 extendable

ip nat inside source static udp 172.16.98.4 53 X.X.X.27 53 extendable

ip nat inside source static tcp 172.16.98.5 80 X.X.X.27 80 extendable

!

ip access-list standard DMZ

permit 172.16.98.0 0.0.0.255

!

ip access-list extended out-dmz-acl

permit tcp any host 172.16.98.4 eq smtp

permit tcp any host 172.16.98.4 eq domain

permit udp any host 172.16.98.4 eq domain

permit tcp any host 172.16.98.5 eq www

permit tcp any host 172.16.98.11 eq smtp

permit tcp any host 172.16.98.4 eq 465

permit tcp any host 172.16.98.4 eq 587

permit tcp any host 172.16.98.4 eq 36124

permit tcp any host 172.16.98.11 eq 5223

ip access-list extended permit-all-acl

permit ip any any

!

no cdp run

!

!

!

!

control-plane

!

!

!

line con 0

exec-timeout 0 0

logging synchronous

transport output none

line aux 0

exec-timeout 0 0

transport output none

line 2

exec-timeout 0 0

no activation-character

no exec

transport preferred none

transport input all

transport output none

stopbits 1

line vty 0 4

access-class 23 in

privilege level 15

logging synchronous

transport input ssh

transport output ssh

line vty 5 15

access-class 23 in

privilege level 15

logging synchronous

login authentication local_auth

transport input ssh

transport output ssh

!

scheduler allocate 20000 1000

!

end

10 Replies 10

seattleit
Level 1
Level 1

I'm wondering now if there is some issue triyng to NAT one of the PPPoE static (inside global) address I have been allocated. EDIT: Actually, that doesn't make sense as I can translate them to my mail and dns servers just fine.

Another thought is, even though I do not have the router's http server running, could the router's http server be having some undesirable effect?

I just do not comprehend why this works:

ip nat inside source static tcp 172.16.98.4 25 X.X.X.23 25 extendable

and this does not work:

ip nat inside source static tcp 172.16.98.5 80 X.X.X.23 80 extendable

It's extremely frustrating .. I have even tested changing the web server's listening port to 8888 and tried:

ip nat inside source static tcp 172.16.98.5 8888 X.X.X.23 80 extendable

..but the router still kills the return traffic somehow/somewhere

Hi,

I have two suggestions:

- In the config, I can see that CEF is disabled, any reason for that? If not then lets enable it and then test again

- As the NAT config seems to be Ok, still the webservices are not working in the way it should, I would suggest you to disable CBAC/Zone Based Firewall from the configuration as a testing step and then check again. This might help you in narrowing down the issue to ZBF or NAT. I would have asked you to run the debugs but somehow I have seen many a times ZBF can have this effect so if possible, lets disable it from the inside and outside interfaces and then check if your HTTP server responds back to internet users

Hope it helps.

Neeraj

Hi Neeraj:

I'm fortunate that Cisco TAC was able to resolve the issue. The problem turned out to be a hardcoded MTU of 1492 on the Dialer interface, which, when removed fixed the issue right away.

This MTU value was recommended by the ISP however for the PPPoE connection.

Thank you

Thanks for posting the resolution...it turned out to be a minor thing causing a major issue

I'll keep this in mind for future. thanks again.

Neeraj

Hello seattleit,

Hmmm, I am glad to see your issue resolved but I am confused by the solution the TAC provided. The IP MTU of 1492 is indeed the correct MTU for PPPoE interfaces. If this is increased, the resulting datagrams may exceed the maximum size of an Ethernet frame (the 'E' in PPPoE). So the TAC suggested to remove the ip mtu command from your Dialer without any further caring?

Best regards,

Peter

Hi Peter,

Yes, once the no ip mtu on the Dialer was removed, the web server was able to negotiate MSS and the http traffic began to serve. I also find this a bit odd, as I thought I configured the Dialer interface correctly with an MTU value of 1492.

This was definitely a painful troubleshooting experience, as I had spent many hours and packet captures trying to debug the configuration. Thankfully I gave in and opend the SR with TAC, but I will do some testing in the next day or two to make sure traffic is running optimally.

Cheers

Hello,

Personally, I would be willing to suggest a few more tests with MTU, MSS clamping and ICMP permissions as I find this issue utterly odd - but I do not want to break your working config and do not want to cause you any more outages.

Best regards,

Peter

Hi Peter,

I just ran a few tests in clamping the MSS down to 1452 and allowing ICMP and it seems to work. I have to head into the office and will tweak this a little more when I get home this evening.

Best

A side note on the MTU configuration - you might want to try setting mtu 1492 instead of ip mtu 1492 on the dialer.  The former should cause PPP to correctly negotiate the link MTU as part of the LCP negotiations.  I have had similar experiences, and after switching from the latter to the former they cleared up.  It's likely that your ISP just NACK's any requests for larger MRRU from your router and forces the negotiation to use 1492.  debug ppp nego will show all the gory details.

When I read the original post my first thought was that it sounded like a problem with MTU. And I was interested to read what seems to have solved the problem. I have a different suggestion about how the problem might have been dealt with. I notice that this command is on VLAN 2 (LAN) but not on VLAN 3 (DMZ)

ip tcp adjust-mss 1412

I suggest that putting this command on the DMZ interface would also have resolved the problem.

HTH

Rick

HTH

Rick
Review Cisco Networking for a $25 gift card