Security Network Analytics doesn't add new exporter

[dijix1990]

Recently I found that my Security Network Analytics (with last update-smc-ROLLUP20240709-7.5.0) doesn't add new exporter.

For example I have 132 exporters for 3 Netflow collectors, I configured new device (c8200) for export netflow and realized that it didn't apeear as exporter. I tried to add it manually but nothing happened. I did research and found out that after deleting some exporters Security Network Analytics creates them again with new devices. I think it because of version 7.5, for version 7.4 it worked perfectly, exporters appear immediately.

maybe someone bumped into this problem?

[DanielLarsson63527]

Hi,

I haven't experienced this perticular issue with 7.5.x release (yet). But i have experienced plenty of exporter-misconfigurations (mainly netflow v9/fnf template configuration errors on the exporters) and I think I read somewhere that there was a change in 7.5.x for the required fields to accept/add exporters.

Normally i just parse through logs/collectors/datastores to see what the issue is. This was a ton easier when the product was not locked down for troubleshooting behind TAC (but if your linux skills are good, you can get a challenge from TAC and just fix the linux-shell for future sessions so you don't require TAC do find out whats wrong).

But this specific issue you can at least do some basic troubleshooting through the web GUI of FC.
Just check in /lancope/var/sw/data/ ... and search for a log-file with your exporter ip-address and see what the template look like when it arrives to FC. (will be support->browse files ... /sw/data in GUI).

There are required fields that must be exported in the netflow-template before it is a "valid" Exporter. Maybe you have a file there and can see if it's a template mismatch? If you don't have a file there, i would start looking into messing with the C8200 netflow configurations just to see if you can make it work. In general though, if the export reaches FC I have not seen a template mismatch not being logged somewhere.

HTH
-Daniel

[dijix1990]

Thanks for sharing your experience, I will check your advice. BTW it's not only c8200 it happens for every platform, Asa, firepower, isr800/isr2900/isr4400 etc

[DanielLarsson63527]

Hmm,

Now that sure sounds like an issue of another level. That said, the platforms you mentioned above does handle netflow a bit differently then let's say a C9000 switch...is it literally all the platforms ... nothing that works?

Just tried and I don't have that issue, although I'm not on the july update (20240709) just yet, i have a *very* strict rule with cisco and *NEVER* applies any patches or updates unless it's been out a month for extensive beta-testing on other networks than mine. Security and everything, yes but it does make things work longterm. Have lost count on the amount of times I break this rule only to learn the hard lesson that I should have waited .... 7.5.0 was one of those that didn't go through proper testing before release. (services not starting, patches not working properly, collectors not exporting flows properly etc).

That said, patches are in general fine in my exprience with stealthwatch as long as you wait about a month before applying the first one. So maybe it's the 20240709 rollup that has issue.... ROLLUP20240515-7.5.0 works at least.

I would still check into the log files to see what's going on, but probably bad patch if it affects all platforms you have!
(you can see all log files from GUI in 7.5 at least, inventory -> view appliance statistics -> support -> browse files)

HTH
-Daniel

[dijix1990]

I talked with my colleagues and they said it started after upgrade from 7.4 to 7.5 version, and we installed all patches but it didn't help. BTW I installed 7.4 version again with ip addresses from 7.5 and everything started to work properly. So 7.5.0 version is extreme untested with a lot of bugs

[DanielLarsson63527]

I agree that the 7.5.0 version is not really working as good as it should. I was actually surprised about how quickly it became the recommended version (no way it was tested properly with that quick transition). I've had tons of issues as well, just not the one you are experiencing.

I haven't had this issue with the rollup i mentioned above, so far.

I suspect you need to TAC for Root and just clean-up interface/exporter in CLI in your install to get around this.

HTH
-Daniel

[dijix1990]

I saw the logs 

https://172.18.144.50/swa/files/sw/data/templates.xml - there is nothing especial

https://172.18.144.50/swa/files/sw/today/logs/sw.log -  the same is nothing especial

Don't know why, I have 140 exporters, when I add new one it doesn't appears, I delete 5 of them and after some time I have 140 exporters with new one. In total 141 after deleting

and with Release 7.4.2 everythink is ok

[DanielLarsson63527]

I think under normal conditions you should not see anything special in "templates.xml".
What usually happens if there is a mismatch in the netflow export there is a logfile like this: <ip of your exporter>_<id>_<source_int> but it only exists if there is something wrong with your netflow config template.

I think you are just hitting a bug in the rollup-patch. Did you apply the patch directly on 29th of july? If so, try and do a MD5-check on your file compared to the one available right now.

Maybe check the syslog-logs of FC and SMC and see what happens when you add/delete exporters?

Edit: Digging around a little bit more and there are a couple of known bugs at least that is related to similar issues you are experiencing in 7.5.0. I haven't hit any of them myself, but there are a few bugs related to "exporter cleaning" services not working properly. Since they messed up big with the 7.5.0 train and removed root-access even for Non-FIPS installs...you cannot fix that without TAC, if only root would still be available and you could run the tool that would fix it...pretty sure the interface/exporter clean-up script/tool available from root would fix it.

HTH
-Daniel

[dijix1990]

Yeah I saw the logs after deleting on the FC and SMC and there was nothing suspicious. Now I want to try to build new environment with version 7.5.0 and try it.

I installed patch two days ago. I checked MD5 now and they are the same 

[DanielLarsson63527]

Hi again,

So i labbed this up today and can confirm that patch is messing with exporters.
Also learned that on top of that, it also has some issues with "hybrid" configurations.

So today I was messing around with adding Datastores/Nodes to a Non-DS environment and it was just impossible to add it to manager running anything with a mistmatch in Rollup-patches...problem is, ISO is without Rollups so had to "land" on a temporary SMC to update it and fix that.

Once that was working, tried the 20240709-01 udate, and had same issues that you are describing here. Also couldn't add Datastores/Nodes to SMC with Custom certificates/PKI signed certificates. Rolled back to 20240515-01 ... things started to work again. Tried with 20240610-01 and exporters were working, but Datanode was unable to join SMC.

The way I see it 20240515-01 is still the Stable patch.
Don't know if you can/is allowed to try it for security reasons, but 20240709-01 seems to have issues, hidden ones.

Unfortunately i have no fix for it other than running something that I know is working.
I did see tons of errors in logs in 20240709-01 for "authentications" and stuff like that, shouldnt mess with exporter configurations though. Unfortunately don't gave root for this temporary SMC so couldn't confim that fixing exporter-tables fixed it in 20240709-01, but if I do i will let you know.

-HTH
Daniel

[dijix1990]

I just instaled SMC 7.4.2 and there is no problem, so I decided to use it and delete 7.5 version at all