IP Fragments and "port 16191"

[fields.james]

This issue has been raised previously; it's gained some visibility recently out at SANS although so far no one has an answer. I and a number of other folks occaisionaly see two or three alerts in a row regarding IP fragments, with the source and destination field set to 16191 (according to the IDS anyway).

Packet captures from the IDSMv2 show that what is really happening is a bunch of data is either being overwritten or inserted composed of the repeated hex byte 0x3f. It always seems to start at the offset where the source port would appear, and any double-byte field after that will translate as 16191 (0x3f3f = 16191).

These always seem to be packets to and from Windows machines (in my environment). This seems to match what others have reported. They also seem to only be "seen" by Cisco IDS sensors (don't know if it's all version 4 sensors or not, and don't know if this is ever seen by other products - only Cisco users have spoken up about it so far). It also seems to frequently involve active directory servers or SQL clusters. I've so far neither seen in my area nor heard from others that this happens with any *nix servers.

I am leaning towards thinking there is a bug in fragment reassembly (or something like that) causing this, that is only triggered with certain kinds of packets. I have quite a few examples I've captured on the IDS sensors, but I am trying to get a "real" sniffer trace that would include what was happening before and after with a given machine.

--James

[pbobby]

I concur. Have been seeing this traffic on my DMZ sensors for a couple of weeks now. I do not see it internally.

Here's the captured packet from the No initial frag (the incomplete dgram is just more 0x3f stuff)

Frame 1 (70 on wire, 70 captured)

Arrival Time: Jan 1, 1970 01:00:00.000000000

Time delta from previous packet: 0.000000000 seconds

Time relative to first packet: 0.000000000 seconds

Frame Number: 1

Packet Length: 70 bytes

Capture Length: 70 bytes

Ethernet II

Destination: 00:50:53:8d:68:8c (Cisco_8d:68:8c)

Source: 00:0e:d7:b0:96:a0 (00:0e:d7:b0:96:a0)

Type: IP (0x0800)

Internet Protocol, Src Addr: 192.168.0.221 (192.168.0.221), Dst Addr: xx.xxx.189.33 (66.194.189.33)

Version: 4

Header length: 20 bytes

Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)

0000 00.. = Differentiated Services Codepoint: Default (0x00)

.... ..0. = ECN-Capable Transport (ECT): 0

.... ...0 = ECN-CE: 0

Total Length: 56

Identification: 0x0000

Flags: 0x00

.0.. = Don't fragment: Not set

..0. = More fragments: Not set

Fragment offset: 0

Time to live: 238

Protocol: TCP (0x06)

Header checksum: 0x0b65 (incorrect, should be 0x0b57)

Source: 192.168.0.221 (192.168.0.221)

Destination: xx.xxx.189.33 (xx.xxx.189.33)

Transmission Control Protocol, Src Port: 16191 (16191), Dst Port: 16191 (16191), Seq: 1061109567

Source port: 16191 (16191)

Destination port: 16191 (16191)

Sequence number: 1061109567

Header length: 12 bytes (bogus, must be at least 20)

0000 00 50 53 8d 68 8c 00 0e d7 b0 96 a0 08 00 45 00 .PS.h.........E.

0010 00 38 00 00 00 00 ee 06 0b 65 c0 a8 00 dd xx xx .8.......e......

0020 bd 21 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f .!??????????????

0030 3f 3f 0b b3 01 bd 40 63 f1 01 40 63 f1 01 06 04 ??....@c..@c....

0040 00 00 c8 3d 00 00 ...=..

[fields.james]

Just to help with data collection, let me add a couple of things and ask if you would provide similar:

1. I mentioned before that at least in my network, these always seem to come to or from Active Directory servers or SQL cluster machines. I typically see them on the internal network (we don't have any of those other two types of machines in our DMZ) although I have seen one or two "hits" of this from the Internet trying to come in. Does this fit the description of your own machines that seem to be involved?

2. We use almost exclusively Compaq servers. I'm not on the team that manages them, but I'll give a 95% confident statement that these are running Windows 2000.

I'm trying to find all the patterns I can - I'm not smart enough to dig a whole lot deeper, but hopefully anything we can establish as a pattern will help with ultimate identification.

I am attaching an edited capture in pcap format - edited because I only want to include the single packet that resulted in the alert.

--James

[mlhall]

If you have an incomplete datagram the missing information is filled with 0x3f which is '?'. This is a feature not a bug :)

If you are still seeing 0x3f bytes in a datagram that is complete, then we are talking about a bug. I would be interested in seeing any trace with packets that cause that.

[mlhall]

My comments apply to version 3.x. Are you running 4.x and still getting -x3f?

[fields.james]

Yes, we're running version 4.x on IDSMv2 blades. I think we're at S94.

[mlhall]

Okay. Thanks we are looking into this now.

[mjuckett]

I am seeing this traffic as well. I am seeing about 150 attempts a day from the internet to our firewall. The signatures that I see fire are 1204-No Initial Frag and 1208-Incomplete Datagram, source and dest port 16191. This is seen by a 4235 running 4.1(4)S94. I saw a post on incidents.org where another Cisco user is seeing this occur on 3 signatures.

http://www.incidents.org/diary.php?date=2004-05-18&isc=46270c26d5494471304e344a661994e7

I thought perhaps this extra info might be of some use.

Thanks.

[pbobby]

Same, v4.1-4-S94

Also my target is an Oracle server running under Solaris 2.8, not Compaq unfortunately.

We use AD within our corp. but these packets have only been seen at the dmz point.

Nice to see Cisco looking in to it.

[fields.james]

I was the guy that reported that. They had a note a few days previously, something like "we're getting reports of fragmented traffic towards port 16191" but no other details, and a call for someone to submit more information.

--James

[pberis]

Preliminary analysis based on the extremely limitted data available, I would say that this is most likely caused by a high level of fragmented UDP traffic. The packets in these logs ( as well as from earlier messages ) are only trigger packets. In the case of the 12xx series of alarms the trigger packet is a (partially) reassembled datagram built from the data kept by the Fragment Reassembly Unit (FRU) ... any portions of the d-gram that are missing are replaced with '0x3f'. It appears that the FRU was only able to queue the final fragment, probably due to buffer limits imposed on it.

To avoid this problem try changing the FragmentReassembly settings ( try increasing 'IPReassembleMaxFrags' ). You will probably also need to change the 'FragmentThreshold' settings for these signatures.

[a.arndt]

I agree that UDP fragments could explain some of these alarms (particularly for the folks noting the activity in relation to very specific network conditions and involved systems), but I think it’s more important that we now know what the Fragment Reassembly Unit (FRU) is up to.

The info about the FRU using padding is very important. This changes how someone will react to these alarms because we now know 0x3f is a pad put in by the IDS to cover any missing fragment and not some variation of a NOOP sled that is part of someone’s buffer overflow attack!

One question/concern. If you increase the max number of fragments, do you not also increase the performance cost for this particular group of signatures? It would appear the default for "IPReasembleMaxFrags" is currently set at 10000 and while that sounds like a lot, it seems to me that it stands to reason that this setting is actually woefully inadequate if a sensor is monitoring a link where a lot of IPSec traffic (or other activity where fragmentation is quite normal) is moving through. The problem now is figuring how much is too much when in comes to configuring the "IPReasembleMaxFrags" variable... On a related point, can we expect Cisco to produce tuned versions of the 12xx series of signatures in order to adjust 'FragmentThreshold' variables in an appropriate manner?

By the way, the 0x3f padding has obviously caused many of us to pull our collective hair out trying to identify the cause of these alarms, particularly because it was unique to Cisco IDS sensors. Information like this needs to be in the user documentation, IMHO. I don’t think the thread at the ISC or here would have started if we knew about the padding. Instead, though I can only speak for myself now, I think the focus would have been on figuring out if the apparent volume of fragmentation was normal for the segment(s) monitoring by the Cisco IDS and correcting any discovered configuration errors (either in the network or the attached systems).

Just my $LOCAL_CURRENCY = 0.02 + $LOCAL_TAXES

Alex Arndt

[fields.james]

I do not consider this matter closed; one of the other Cisco engineers indicated this padding is a feature of 3.x sensors but in 4.x indicates a bug.

One of the other engineers said "preliminary analysis based on extremely limited data available" - so what can I provide that will help to make things more certain? I am willing to provide whatever settings are requested as well as further sniffer traces as required.

--James

[darin.marais]

Sorry to dig this one up again. I am not sure that I have understood the resolution.

After reading this thread I have understood that if you are receiving this event with port 16191, this means increase the IPRreasembleMaxFrags parameter on the sensor until you don’t see it anymore.

Is this the answer???