We got around this problem through our netflow collector software. Typical netflow collectors dump it all to some SQL backend and you have to trawl through it and its extremely inefficient especially if you are interested in a small part of it.
Our netfow collector is an application that works a little like a firewall does. You only collect on what you are interested in...
ie, You create a datapoint and associate an access-list style of filter with it. (if you are familiar with IOS access lists then you will get the idea). So when the netflow hits the collector, only traffic matching the filter associated with each datapoint gets recorded to the datapoint. By default, we only record the size of each flow because we use this as a billing technique. However when a customer requests more detail we record full details of all flows matching the filter where necessary.
We are an ISP and use this technique for billing our broadband customers in real-time. It enables us to create itemised invoices breaking down bandwidth into different protocols, address ranges, etc.
Its commercialware and I haven't read up fully on acceptable use for these discussion forums. So I will avoid turning this into a spam advertisement and instead I suggest contacting me directly (pculmsee@wiredcity.com.au) and I can explain things in more detail.
(as an added bonus, the software also can be used to monitor SNMP, NT Perfmon API, SQL transactions, etc)
regards
Paul
