Netflow Suddengly Not Working on Cisco 3650

[fdharmawan]

Hi Guys,

Suddenly the netflow on my 3650 switch is not working. I have compared the config from the one back in 2021, it's the same, nothing was changed. But when I compare to the other switch which netflow is working, I realized that the current entries is exceeding the cache. Is this the reason why? Below are the output of show flow monitor statistics.

Switch#show flow monitor Netflow-OUT statistics
Cache type: Normal (Platform cache)
Cache size: 10000
Current entries: 33280

Flows added: 152302
Flows aged: 119022
- Active timeout ( 30 secs) 117049
- Inactive timeout ( 15 secs) 1973

Switch#show flow monitor Netflow-IN statistics
Cache type: Normal (Platform cache)
Cache size: 10000
Current entries: 16896

Flows added: 82508
Flows aged: 65612
- Active timeout ( 30 secs) 64116
- Inactive timeout ( 15 secs) 1496

Below are the config for the netflow itself

flow record ORION-FLOW
match ipv4 tos
match ipv4 protocol
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
match interface input
match flow direction
collect interface output
collect counter bytes long
collect counter packets long
collect timestamp absolute first
collect timestamp absolute last
flow record ORION-FLOW-OUT
match ipv4 tos
match ipv4 protocol
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
match interface output
match flow direction
collect interface input
collect counter bytes long
collect counter packets long
collect timestamp absolute first
collect timestamp absolute last
flow exporter ORION-EXPORTER
description NETFLOW Export to ORION
destination 192.168.0.1
source Loopback0
transport udp 2055
flow monitor ORION-MONITOR
description ORION Netflow Monitor
exporter ORION-EXPORTER
cache timeout active 30
record ORION-FLOW
flow monitor ORION-MONITOR-OUT
description ORION Netflow Monitor Out
exporter ORION-EXPORTER
cache timeout active 30
record ORION-FLOW-OUT

Any idea why? Thank you.

[M02@rt37]

Hello @fdharmawan 

It seems that the number of entries in the NetFlow cache has exceeded the configured limit on both NetFlow monitors (`Netflow-OUT` and `Netflow-IN`). The cache size is set to 10,000 entries, but the current entries are well above that limit.

This might be causing issues with NetFlow functionality. You can consider a few steps to address this:

Increase Cache Size:
If your switch resources allow, you might consider increasing the cache size. This can be done using the following commands:

flow monitor Netflow-OUT
cache size <new_size>

Review and Optimize Flow Records:
Review your flow records (`ORION-FLOW` and `ORION-FLOW-OUT`) and make sure you are not collecting unnecessary information. Fine-tune your flow records to collect only the data you need.

Adjust Cache Timeout:
The cache timeout for active flows is set to 30 seconds. Depending on your network traffic patterns, you might adjust this timeout to age out flows more quickly.

flow monitor Netflow-OUT
cache timeout active <new_timeout>

Check the overall resource utilization on your switch to ensure it has sufficient memory and processing power to handle the configured NetFlow parameters.

[fdharmawan]

Hi Mozart,

Thank you for the reply. I have several follow up queries, hope you don't mind.

1. Is there any risk on increasing the netflow cache size? On this article, it said that it is recommended not to increase the cache flow size. While I cannot find anything from Cisco. The current switch memory utilisation is 45-50%. So with the current 10k cache size and one of them hits 33k, I think increasing it to 40k is minimum, right?
2. Is there some fine tuning article that you can recommend to me? Since I was following up somebody else's configuration example on the internet, and not knowing the full extent of what the command might do to the netflow.

Thank you.

[M02@rt37]

@fdharmawan 

Increasing the NetFlow cache size itself doesn't inherently pose a risk. However, it's important to consider the available resources on your switch. Since you mentioned that the current memory utilization is around 45-50%, you may have some headroom to increase the cache size. Keep in mind that a larger cache size will consume more memory, so it's crucial to monitor memory usage after making changes. As for recommendations against increasing the cache size, it's possible that such advice is given to avoid excessive memory usage on devices with limited resources. In your case, since the memory utilization is within a reasonable range, you can cautiously increase the cache size and monitor the impact.

Fine-tuning NetFlow requires a good understanding of your network traffic patterns and the specific information you want to capture. 

• Define Relevant Flow Records: Carefully choose the fields in your flow records based on the information you need. Your current configuration seems to capture various fields (IPv4 TOS, protocol, source/destination addresses, transport ports, interface, flow direction), which is a good start.

• Active and Inactive Timeout Values: Adjust the cache timeout active value based on the typical duration of active flows in your network. If flows tend to be longer-lived, increasing this value might prevent premature aging.

• Review Exporter Configuration: Ensure that the exporter configuration aligns with your network's requirements. Verify the destination IP address, source interface, and UDP port. Also, confirm that the configured loopback interface is reachable.

• Consider Flexible NetFlow: If your platform supports it, Flexible NetFlow provides more flexibility in defining flow records and templates. It allows you to customize the exported information more precisely.

[balaji.bandi]

If i were you  :

1. i take maintenance window - reload the Switch and check.

2. i will upgrade to latest code any stable version (see if that fix the issue)

[fdharmawan]

Hi Balaji,

That's what I thought, because I did not change anything from 2021. Plus, I saw this line on the log of the switch.
An invalid command line [show platform software netflow FP active flow ] passed into mcprp_dc_handler

Did some Googling, Cisco said it was a bug, but not the very same line, but similar (CSCvs25280 and CSCvi58066). My code is 16.09.06 and the current recommended is 16.12.10a.

[johnlloyd_13]

hi,

can you still ping the orion NPM 192.168.0.1 from the switch/SVI?

can you remove then re-apply the netflow config under the monitored interface?

interface x/x
no ip flow monitor <MONITOR NAME> input
no ip flow monitor <MONITOR NAME> output
ip flow monitor <MONITOR NAME> input
ip flow monitor <MONITOR NAME> output

[fdharmawan]

Hi John,

I did that already, but unfortunately did not help.