cisco net flow ,export to collector.

[sarahr202]

Hi everybody.

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps6601/prod_white_paper0900aecd80406232.html

How Does the Router or Switch Determine Which Flows to Export to the NetFlow Collector Server?

A  flow is ready for export when it is inactive for a certain time (ie: no  new packets received for the flow); or if the flow is long lived  (active) and lasts greater than the active timer (ie: long FTP  download). Also, the flow is ready for export when a TCP flag indicates  the flow is terminated (i.e. FIN, RST flag). Their are timers to  determine if a flow is inactive or if a flow is long lived and the  default for the inactive flow timer is 15 seconds and the active flow  timer is 30 minutes.

The above paragraph answers one question when a router/switch deems a particular is ready  to be export to collector. For example. An inactive flow will be ready to be exported after 15 seconds since the flow was first cached.  On the other hand an active flow will be ready to be exported to collector after 30 min since the time the flow was cached.

  The following is from the link above:

In general, the NetFlow cache is constantly  filling with flows and software in the router or switch is searching the  cache for flows that have terminated or expired and these flows are  exported to the NetFlow collector server.

But how often this flows are sent in a report to a collector? every 5 min?  every 20 min? 

. Does a router keep  checking cache and the second a flow becomes ready for export , router will send it to collector? . Or router/switch will keep checking the cache for the flows that are ready to be exported until certain time. For example let say router will check the cache for 5 sec. two second elapsed, router finds two flow f1,f2 that are ready to be exported to collector, another 2 two seconds elapsed, router find two more flow f3 f4 ready to be exported. After one last second, router find one more flow f5 ready to exported. Since 5 seconds are up , router needs to send these  five flows to collector.

So my question can be sum up as:

But how often this flows are sent in a report to a collector? every 5 min?  every 20 min? 

thanks and have a great day.

[Richard Burts · Accepted solution]

Your question suggests an approach where the router will attempt to batch up several flows and export them al at one time. It is my understanding that this is not what the router does. The router is searching the cache for flows that qualify for export (inactive for 15 seconds or active for 30 minutes. When the router finds one of these flows it immediately exports that flow.

HTH

Rick

[Giuseppe Larosa · Accepted solution]

Hello Sarah,

a router configured for netflow accounting and netflow exporting to a netflow collector continously exports flow statistics to the collector. The more the traffic is varied ( the more flows exist) the more export packets are sent  to the collector.

On the collector side one or more aggregation schemes can be configured and the duty of collector is to aggregate the export data every N minutes and to produce a file ( the file can be  a text file or a binary file).

The file produced by the collector for each router can then be further aggregated and processed by second line servers like the netflow analyzer.

The amount of traffic generated by a router for netflow exporting can be high and the netflow cache is a precious resource with a finite number of entries. The router cannot wait 5 or 20 minutes before sending the export packet, Clearly the export packet describes multiple flows that have expired or that needs to be exported because the netflow cache is near to be full.

An internet backbone router can produce on the collector a file every ten minutes with hundreds of thousands of different flows, each flow accounted with its number of bytes seen in the conversation!

If the flow is exported, a new entry for the same flow can be created on the router netflow cache and then exported again at a later time,

For the way the collector works this fact of exporting the same flow several times is not a problem during aggregation the single entries for the same flow sum up  and the total bytes in the aggregation time interval is calculated and reported in the collector file.

So building netflow export packets can be an heavy duty for a router and this is the reason why local aggregation caches have been introduced. With local aggregation caches the router performs locally one or more aggregations with different schemes and then exports these pre-aggregated data to the collector.

On high end platforms scalability is achieved by the use of sampling : actually only one packet every N is passed to Netflow (with N that can be 1000 typically). Some short life flows like a simple DNS query can be lost but this method allows in any case to make good measures for the long live flows.

So a big internet service provider needs to deploy many collector servers each of them receiving flows from a small subset of routers. Data from all the collector need to be aggregated to get the whole picture.

These setups are used to study the BGP peering relationships between providers and with customers.

Netflow is also used for security purposes and in that case the level of detail of flow aggregation is the highest as possible.

Hope to help

Giuseppe

[sarahr202]

Thanks Giusepppe and Rick.

So building netflow export packets can be an heavy duty for a router  and this is the reason why local aggregation caches have been  introduced. With local aggregation caches the router performs locally  one or more aggregations with different schemes and then exports these  pre-aggregated data to the collector.

Normally  a router sends flows to a collector the second they qualify for export. It puts a lot of stress on a router. That is why we came up with local aggregation cache. What does it mean to us?  Does it mean now router will export the flows to aggregation cache and then perform some aggregation and send the bunch of flows together to collector.

Did i understand it correctly ?

Thanks and have a great day.   ( gotta run  for work)

[Giuseppe Larosa · Accepted solution]

Hello Sarah,

your understanding is correct.

The router uses more memory resources to host a local aggregation cache or a few local aggregation caches, but it has to transmit less export packets as it is exporting pre-aggregated information instead or raw flow info.

As a result of this it uses less cpu resources for the netflow export process

The collector has to be configured to handle correctly the data received from a router that uses local aggregation caches. on the collector there are aggregation schemes corresponding to the pre-aggregated local flow cache.

Hope to help

Giuseppe