Cisco FireSIGHT Forwarder for HP Arcsight CEF

[dohurd]

The forwarder is an eStreamer client that converts eStreamer data collected from FireSIGHT into a ArcSight Common Event Format (CEF) format for input into Arcsight's ESM platform. ArcSight CEF is a syslog and text-based alternative to Arcsight's Smart Connector however it does not have support for packet payload yet. This functionality will be provided via the ESM action connector and will be made available soon as part of the HP ArcSight CEF certified connector. 

[MervAhYoung1]

Have you had any luck getting ArcSight to successfully (and fully) parse events from FireSIGHT Defense Center (via eStreamer), without any Parser overrides or customised FlexConnectors?

It seems ArcSight only officially supports FireSIGHT/Sourcefire Defense Center v5.3.1, but doesn't support 5.4.x.

Has CISCO/Sourcefire made any changes to the eStreamer message format between 5.3.1 and 5.4.x?

Any ideas?

[dohurd]

5.3.1 was the last version they tested.  Their Smart Connector should continue to work identically and receive exactly the same data from FMC 5.4.x as it does with 5.3.1.  Same holds for future versions of FMC.  Obviously, Arcsight needs to add to/enhance the connector to specifically request new stuff in new versions.  The Arcsight connector is not comprehensive however and only collects IDS/IPS/Impact and Packet.  Maybe a few other event types.  Not 100 percent sure.

Arcsight is pushing all of their partners into CEF however so they wont be enhancing the connector until a) enough customers complain or b) they realize that the number of Cisco devices that eStreamer will be able to collect events from will increase 20 x and that it makes sense to double down.  I'm not optimistic however.

There are changes in 5.4 but they are not huge.  The first chapter of every new eStreamer Integration Guide always covers all the changes in that new version.

[MervAhYoung1]

I've integrated FireSIGHT 5.4.1.1 with ArcSight SIEM using eStreamer event flows into the Connector tier.  Works best with ArcSight SmartConnector 7.1.4 but still requires some customizations with map files for those events which aren't properly categorized.  Works fine for intrusion, Malware and file block events. Now trying to figure out why eStreamer isn't including the managed device IP in the event to easily identify which IPS sensor detected the event.

[harshadsstsi]

Hi MervAhYoung1 ,

I am facing same issue , I am not able to see the individual IPS IP address in Arcsight.

Did you get any resolution ?

Thanks.

[MervAhYoung1]

Hi harshadsstsi,

Yes, I eventually got this working!

In your eStreamer Server configuration (on FireSIGHT) you need to make sure you enable Intrusion Event Extra Data, so ArcSight can pull METADATA off FireSIGHT.

In your ArcSight Connector configuration, set 'Request RNA events' to 'true' and also set 'Intrusion Event Version' to 5.1.1.

Ensure the 'agent.properties' file in your ArcSight Connector Container has the following entries:

agents[0].request.meta.data=true
agents[0].request.meta.data.version=4
 

Edit the 'Runtime Parameters' in your SmartConnector's Destination and make sure 'Preserve Raw' events is enabled.

Now, Stop the Connector process and Start it again to re-initiate the eStreamer connection to the FireSIGHT DC.

As IPS events are being fed into FireSIGHT and then sent onto ArcSight, you will notice you'll start getting METATADATA:123 events into your ArcSight ESM Console.  These are the events which will contain detailed information on which FirePOWER IPS sensor has been allocated which External ID (e.g. event.deviceExternalId) by the FireSIGHT DC.

Take note of the External ID assigned to the originating IPS sensor which triggered the event.

You will then need to create a custom map.X.properties file (e.g. map.0.properties) in your Connector's '/opt/arcsight/connectors/connector_X/current/user/agent/map' directory.

The contents should look something like this (as an example obviously):

event.deviceVendor,event.deviceProduct,event.deviceExternalId,set.event.deviceAddress,set.event.deviceHostName
Sourcefire,Sourcefire Management Console eStreamer,1,192.168.1.1,FirePOWER_IPS_1
 

I usually create this file in a directory call 'map' and then I ZIP up the file from the 'map' directory.  Upload the custom map file to your Connector and then apply/upload it to the Container which you have configured your eStreamer Client on.

Give it a few minutes, and then you'll start seeing IPS events in the ESM Console with the Device Address and Device Hostname set to the correct IPS sensor which triggered the event.  The key to this mapping is the 'External ID', so make sure you search for all the METADATA:123 raw events and look at what the External ID is for the originating sensor.

METADATA:123 can aslo give you a lot of other useful information, such as the Access Control Policy which triggered the event and the Intrusion Policy etc...

Give it a try and see how you go... good luck!

[harshadsstsi]

Hi MervAhYoung1, 

Thank you very much for your very quick response :-) 

Any information on how to send "security intelligence events" to arcsight.

Estreamer client optionnon foresight doesn't have specific option for Sec. Intel.events.

[v_terekhin]

Hi. FireSIGHT/Sourcefire Defense Center v6.0 supports a syslog ArcSight?

[MervAhYoung1]

Hi Harshadsstsi,

Unfortunately, ArcSight is very limited in the type of events you can actually select to "pull" from the eStreamer Server.

Also, I do not believe you can actually select SI type events to send from FireSIGHT/eStreamer.

If you want to extract SI events, I think the best way is to remotely query the FireSIGHT DB directly using some customised scripting.

Sorry, I haven't really looked into this to be honest.  For now I'm just pulling Intrusion, RNA, File/Malware and trigger packet events from FireSIGHT into ArcSight.

Cheers,

Merv

[MervAhYoung1]

Hi v_terekhin,

You can send FireSIGHT events into ArcSight via normal Syslogs, however, the events will not be as comprehensive as compared to using eStreamer to pull events into ArcSight.

Also, ArcSight doesn't really parse FireSIGHT Syslog messages very well, so you will need to do some customisations via map files in your ArcSight Container.

As far as I'm aware, ArcSight "officially" supports up to eStreamer v5.3.1. Anything newer is not supported.  I'm running FireSIGHT v5.4.1.3 and I've had to do some fairly extensive customisations for my use cases in ArcSight.

I haven't had the pleasure of testing/running FireSIGHT 6.0 as yet because it doesn't support HA peering between DC's... yet! :-(

Cheers!

Merv

[mymorristribe]

Hi, have you or anyone else here been successful with using a parser override on v5 Firesite?  In our estreamer connectors pulling from v4 Sourcefires, we use something like this:

/fcp/sourcefire/sourcefire_api.sdkmapper.properties configured as follows:

event.deviceExternalId=__regexToken(DetectionEngineName,".*?/(.*)")
event.deviceInboundInterface=__regexToken(DetectionEngineName,"(.*)/.*?")

This automatically pulls the SF sensor name and populates it into the device InboundInterface and ExternalID fields in Arcsight.

But, now that we are using v5 Firesight DCs, the estreamer connector parser over-ride doesn't work.  So, we're back to map.x.properties files which are a pain since the sensors change often and we manage many DCs.

[MervAhYoung1]

From FS DC v5.x onwards, I believe the Metadata event format (as seen by ArcSight is):

EventMap: [RecordType=>123] , [RecordLength=>20] ,[DetectionEngineName=>192.168.1.1] , [DetectionEngineId=>1]

You'll  need to modify your regex expression accordingly.

e.g. 

event.deviceExternalId=__regexToken(DetectionEngineName=> ([^,\:].*?))

event.deviceInboundInterface=__regexToken(DetectionEngineName=> ([^,\:].*?))

[mymorristribe]

Merv,

Thanks for the reply.  You seem to know what I'm talking about, so that's nice.  I've found no other support forum with this particular topic.  Unfortunately, the example you provided didn't work.  The agent.log output shows this:

[2016-05-29 04:26:25,707][FATAL][default.com.arcsight.agent.parsers.j][constructAlertFromValues]
com.arcsight.agent.parsers.operation.WrongArgumentsException: Wrong number of arguments, expected :2, received :1, key[0]=DetectionEngineName=> ([^,:].*?), value[0]=null
        at com.arcsight.agent.parsers.operation.BaseOperation.assertSize(BaseOperation.java:41)
        at com.arcsight.agent.parsers.operation.regexTokenOperation.getResult(regexTokenOperation.java:102)
        at com.arcsight.agent.parsers.i$c_.a(i$c_.java:1395)
        at com.arcsight.agent.parsers.i.a(i.java:763)
        at com.arcsight.agent.sdk.a.q.a(q.java:128)
        at com.arcsight.agent.sdk.a.q.a(q.java:118)
        at com.arcsight.agent.sourcefire.api.c.a(c.java:1281)
        at com.arcsight.agent.sourcefire.api.c.a(c.java:1000)
        at com.arcsight.agent.sourcefire.api.ab.a(ab.java:854)
        at com.arcsight.agent.sourcefire.api.ab.a(ab.java:584)
        at com.arcsight.agent.sourcefire.api.c.a(c.java:1627)
        at com.arcsight.agent.util.d.k.a(k.java:218)
        at com.arcsight.agent.util.d.h.a(h.java:358)
        at com.arcsight.agent.util.d.i.s(i.java:216)
        at com.arcsight.agent.util.d.h.run(h.java:207)
        at java.lang.Thread.run(Thread.java:745)

I tried a few variations like adding in quotes since it looks like you were expecting an IP address, but we use 8-10 character strings for the DetectionEngineName.  So far, I haven't had any luck with this.  Thoughts?

Update: 29 May. 

I was able to get some mappings using  event.deviceInboundInterface=__regexToken(DetectionEngineName,(.*))

It's strange.  It will sometimes map, but not usually.  Still experimenting. 

[MervAhYoung1]

What does the raw event look like?  Can you post it here?

You'll need to enable 'Preserve Raw Events' in your SmartConnector's runtime parameters, and then copy the 'raw event' field from ESM Console.

[mymorristribe]

Ok, so I didn't really know about the "Raw Event" field before, and now that it's enabled, this whole thing makes a lot more sense.  HUGE THANKS for that.  What I see in the Raw Event field is that in only a few events (about 10% of the events named METADATA) the [DetectionEngineName=> sensor_name] field shows up.  In the majority of the METADATA events, that particular entry is missing (goes from [DetectionEngineId] right to [BlockLength] or some other field.  In ALL of the events that are actual Sourcefire Intrusion events, the [DetectionEngineName] field is missing--always.  Any ideas why this might be? 

So, looking closer, my parser over-ride using simply DetectionEngineName,(.*)) works perfectly, but of course, only on events where the DetectionEngineName field actually shows up in the Raw Event.

[MervAhYoung1]

Yep, I know exactly why.

Sourcefire/FireSIGHT DC simply does NOT sent the 'DetectionEngineName' in any of the intrusion events. Which brainiac thought up that solution??

You will notice in the METADATA:123 events, the DetectionEngineName correlates to a DetectionEngineId.

Every IPS sensor peered with the DC is allocated a unique DetectionEngineId#.

This is the only way you can correlate the 'DetectionEngineName' with the 'DetectionEngineId'.

Once you have this, you can then map the two together in the SmartConnector Container under the '/opt/arcsight/connectors/connector_1/current/user/agent/map/map.0.properties'.

For e.g.

event.deviceProduct,event.deviceExternalId,set.event.deviceAddress,set.event.deviceHostName

Sourcefire Management Console eStreamer,1,192.168.1.1,ips-sensor-01-name

Sourcefire Management Console eStreamer,2,192.168.1.2,ips-sensor-02-name

Note: DetectionEngineId = deviceExternalId in v5.4.x of eStreamer