HELP Ios ZBF Drops TCP traffic with "Invalid Segment"

[stathis_iku]

Hi All,

After more than a week researching this i am just about ready to throw the box off the window.

In the office we have four zones configured on an ISR-G2 router. Namely the outside, inside, dmz and byod.

I am trying to inspect a set of protocols on traffic going from the byod to the inside but the router keeps droping all tcp traffic with the following message:

156277: Apr  7 16:34:36.679 EET: %FW-6-DROP_PKT: Dropping tcp session 10.10.100.44:58773 10.10.1.3:80 on zone-pair BYOD_IN class BYOD_TO_NAS due to  Invalid Segment with ip ident 17171

What I find is interesting is that the box seems to be dropping only tcp traffic, udp and icmp goes through properly

This is definately a firewall issue since when I change the related class map to pass traffic then traffic goes through as expected.

The box is running the following ios.

Version 15.4(3)M3, RELEASE SOFTWARE (fc2)

I have posted below the relevant part from my config for your consideration.

Please let me know if you see anything out of place.

I would also be happy if you could point me to some document that I may have missed that may explain this behavior.

I thank you all in advance,

Stathis

Config follows:

========================

INTERFACE CONFIGIRATION

========================

interface Vlan40
description BYOD_NETWORK
ip address 10.10.100.2 255.255.255.0
ip helper-address 10.10.1.4
no ip redirects
no ip proxy-arp
ip nat inside
ip virtual-reassembly in
zone-member security BYOD
ip tcp adjust-mss 1412
end

interface Vlan75
description Management Vlan
ip address 10.10.1.101 255.255.255.0
ip helper-address 10.10.1.4
no ip redirects
no ip proxy-arp
ip authentication mode eigrp 100 md5
ip authentication key-chain eigrp 100 EIGRP_KEYS
ip flow ingress
ip nat inside
ip virtual-reassembly in
zone-member security in-zone
ip tcp adjust-mss 1412
ip policy route-map Distribute
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 7 xxxxx
end

============================

BYOD TO SERVER POLICIES

============================

Policy Map type inspect BYOD_IN_POLICY
Class BYOD_TO_NAS
Inspect
Class BYOD_TO_IN_DNS
Inspect
Class class-default
Drop log

Class Map type inspect match-all BYOD_TO_NAS
Match class-map BYOD_TO_NAS_PROTOCOLS
Match access-group name BYOD_TO_NAS

Class Map type inspect match-any BYOD_TO_NAS_PROTOCOLS
Match protocol http
Match protocol https
Match protocol ftp
Match protocol microsoft-ds
Match protocol nfs
Match protocol cifs
Match protocol netbios-dgm
Match protocol netbios-ns
Match protocol netbios-ssn
Match protocol ftps

Extended IP access list BYOD_TO_NAS
10 permit ip 10.10.100.0 0.0.0.255 host 10.10.1.3

===============================

SERVER TO BYOD POLICIES

===============================

Policy Map type inspect IN_TO_BYOD_ALLOWED
Class NAS_TO_BYOD
Inspect
Class class-default
Drop log

Class Map type inspect match-all NAS_TO_BYOD
Match class-map IN_TO_BYOD_GENERAL_PROTOCOLS
Match access-group name NAS_TO_BYOD

Class Map type inspect match-any IN_TO_BYOD_GENERAL_PROTOCOLS (id 264)
Match protocol tcp
Match protocol udp
Match protocol icmp

Extended IP access list NAS_TO_BYOD
10 permit ip host 10.10.1.3 10.10.100.0 0.0.0.255

[Philip D'Ath]

Are you able to upgrade to 15.4(3)M5?

Try changing this:

Class Map type inspect match-all BYOD_TO_NAS
  Match class-map BYOD_TO_NAS_PROTOCOLS
  Match access-group name BYOD_TO_NAS

to:

Class Map type inspect match-all BYOD_TO_NAS
  Match access-group name BYOD_TO_NAS

And see if that has any impact.  I don't think there is much advantage to specifying the protocols as well.  And if you are missing a protocol it will break it - and well, it is broken at the moment.

[stathis_iku]

Hi Philip,

Thanks very much for taking a look at this.

Actually I don't think i can upgrade the ios and in fact I think I have already tried to remove the protocols and allow all as you suggested but i think this has to impact.

I will give it another shot tomorrow since i am not in the office right now.

Do you think this is related to the version of the ios are there any reported bugs?

Thanks again,

[Philip D'Ath]

I think there is a 10% change it is the IOS version.  But 15.4(3)M5 is the current maintenance release in that train - so the router has like got two service packs of bug fixes available for it that have not been applied.

Might as well apply bug fix releases when they are available if you are having an issue.

I am more suspicious about the protocol selection.

[stathis_iku]

Sure ok.

I 'll try to get that release and see if anything changes.

It just does not make any sense to me that the same box is allowing inter zone traffic for the outside to inside and vice versa but is dropping byod to inside and vice versa. I also tried an intrazone config and its droping everything there as well. 

Essentially i changed the config on the interface from BYOD to inside and it still dropped everything.

Anyway i ll try to work on this tomorrow and will let you know how it goes.

Kind regards,

Stathis

[stathis_iku]

Hi Philip,

I just tried your suggestion and we can see that with just the inspect in place there is no change.

So if we remove the protocols everything gets dropped. Again we see the same message on the firewall

159393: Apr 8 12:34:11.947 EET: %FW-6-DROP_PKT: Dropping tcp session 10.10.100.43:49266 10.10.1.3:5222 on zone-pair BYOD_IN class BYOD_TO_NAS due to Invalid Segment with ip ident 1262

I am trying to get a hold of the new ios version but i fear this will not resolve this.

Any ideas will be greatly appreciated.

[stathis_iku]

Hi again,

We were able to get the new firmware after all and unfortunately there is no change.

We still get the same messages all tcp traffic is still blocked between the zones with the same generic "Invalid Segment with ip ident" message.

If anyone could shed some light into this i would be grateful.

[Philip D'Ath]

Are you able to attach your config to the thread as a file attachment?

[stathis_iku]

Hi Philip,

I have attached a sanitised version of the config as requested.

Please let me know if you see anything that would explain this behavior.

I thank you in advance

[Philip D'Ath]

There servers and devices definitely only have a single NIC in them, and there is only a single default gateway in each network?  If traffic went by one path and came back by another you might be able to get an error like this.

[Philip D'Ath]

Looking at just the case you posted, "10.10.100.44:58773 to 10.10.1.3:80", it using the BYOD_IN zone-pair, which is using the BYOD_IN_POLICY service policy.  Traffic is going from VLAN40 to VLAN75.

This class of traffic is current set to "pass" rather than "inspect", so it wont create a return path to allow the return traffic.

policy-map type inspect BYOD_IN_POLICY
 class type inspect BYOD_TO_NAS
   pass log
 class type inspect BYOD_TO_IN_DNS
   inspect
 class type inspect BYOD_TO_DC
   pass log
 class class-default
 drop log

[stathis_iku]

Hi Philip,

All servers and clients have the right DHCP confifuration with a single default gateway on each network.

The configuration has been modified with pass since the inspect drops all tcp traffic as i have explained.

When we change to inspect, as it should be, we see the logs that i have already posted and all tcp (and only tcp) traffic blocked.

Wireshark reveals nothing of interest.

Do you see anything on the config that would suggest this madness?

Many thanks,

[Philip D'Ath]

Negative, I do not see anything else that would cause this issue.

Hold everything, you have a route-map on vlan75, Distribute_traffic, which is also matching this traffic and forcing it out somewhere different.

Can we do a quick experiment?  Change back to using "inspect" and remote "Distribute_traffic" from vlan75.  Then you can put it back.

If this makes it work then we need to modify Distribute_traffic so that it does not apply to this traffic.

[Philip D'Ath]

You probably need to change the access-lists used by Distribute_traffic from being like:

ip access-list extended Mgt_Network
 permit ip host 10.10.1.4 any
 permit ip host 10.10.1.3 any

to being like:

ip access-list extended Mgt_Network
 deny ip 10.10.1.4 10.10.100.0 0.0.0.255
 deby ip 10.10.1.3 10.10.100 0.0.0.255
 permit ip host 10.10.1.4 any
 permit ip host 10.10.1.3 any

[stathis_iku]

Hi Philip,

Yeah that actually does make some sense. Let me try this and i will get back to you with the results shortly.