cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
492
Views
0
Helpful
5
Replies

3825 issues after IOS upgrade (cpu, netflow)

Nikolaos Milas
Beginner
Beginner

Hello,

I have just upgraded two routers 3825 from Version 12.4(9)T2 to 12.4(24)T8 (Advanced Enterprise).

Things seem to be running smoothly after upgrade (no reported serious issues) but CPU load has increased significantly. On one of the routers it went up from 15-25% average, to 60-70% average (in the few hours we are watching it).

Can someone suggest if this is expected and normal behavior or not?

After the upgrade, I noticed that both routers have issues with netflow exporting.

My configuration was:

ip flow-cache timeout active 1
ip flow-export source Loopback0
ip flow-export version 9
ip flow-export interface-names
ip flow-export destination 195.251.xxx.xxx 9995
ip flow-top-talkers
 top 10
 sort-by bytes
 cache-timeout 3600000

After the upgrade, the line:

ip flow-export interface-names

has been removed automatically and if I add it manually, it still is not displayed.

Router interface netflow configuration is using only "ip flow ingress" statements on all interfaces (as is usually suggested).

After upgrade, the automatically-modified (during upgrade) running-configuration file has removed from all interfaces the statement:

ipv6 flow ingress

No ipv6 reporting is being made since then. (I tried to enter the same statement again manually, but it did not accept it.)

Can you please suggest a reason for this behavior and indicate possible solutions?

Here follow some diagrams (upgrade point is clearly visible) from one of the two routers:

Main interface (connection to ISP):

Some more interfaces:

Please advise.

Nick

5 Replies 5

Nikolaos Milas
Beginner
Beginner

Let me add the following info to the high CPU problem:

1. High interrupt percentage (without any particularly significant network load):

noa-pen#show processes cpu sorted
CPU utilization for five seconds: 78%/31%; one minute: 73%; five minutes: 73%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 114     3031176     4953163        611 44.00% 40.95% 40.79%   0 IP Input         
  42      101512        5694      17827  1.83%  1.79%  1.78%   0 Per-Second Jobs  
 178       10772     1160649          9  0.47%  0.45%  0.46%   0 HQF Shaper Backg
  51        1780          96      18541  0.31%  0.03%  0.00%   0 Per-minute Jobs  
   2         544        1134        479  0.07%  0.03%  0.02%   0 Load Meter       
 153         836       26245         31  0.07%  0.03%  0.02%   0 TCP Timer        
 306        4864       27319        178  0.07%  0.07%  0.07%   0 IPv6 Input       
 179         720       55993         12  0.07%  0.02%  0.02%   0 RBSCP Background
 307        2008       34419         58  0.07%  0.07%  0.07%   0 IPv6 ND          
 141        2388        8692        274  0.07%  0.08%  0.07%   0 CEF: IPv4 proces
  19        5960       29324        203  0.07%  0.07%  0.07%   0 ARP Input        
 228         796       17104         46  0.07%  0.02%  0.00%   0 OSPF-120 Hello   
  12          28        5544          5  0.00%  0.00%  0.00%   0 IPC Deferred Por
  11          72        5544         12  0.00%  0.00%  0.00%   0 IPC Periodic Tim
  10           0           1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager
...

Never in the past did we have such loads:

The arrow shows the upgrade time.

Here is the primary (connection to ISP) interface config, for your reference. (This config is similar to other interfaces config.):

interface GigabitEthernet0/0.102
 encapsulation dot1Q 102
 ip address 62.xxx.xxx.xxx 255.255.255.254
 ip access-group 100 in
 ip access-group 101 out
 ip verify unicast reverse-path 103
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nbar protocol-discovery
 ip flow ingress
 ip nat outside
 ip virtual-reassembly
 no ip mroute-cache
 ipv6 address 2001:648:2FFD:xxx:xxx::2/126
 ipv6 enable
 no keepalive
 no cdp enable
 crypto map vpn

Please advise.

Thanks,
Nick

With regard to the IPv6 issues, I found this nice article:

http://commandline.ninja/2013/10/11/ipv6-flow-export-with-flexible-netflow/

which revived our IPv6 netflow export.

I also found two interfaces (with many subinterfaces each) having CEF disabled ("no ip route-cache cef"). Interestingly, when I enabled CEF on these interfaces, it fixed both the CPU load problem AND the IPv4 netflow export problem!!

I can understand why this fixes CPU load, but not why it fixes IPv4 netflow export as well!

If someone can provide an explanation/insight on this, I would appreciate it!

All the best,
Nick

In the end I found that IPv4 netflow did not produce correct output (despite my initial impressions) with the configuration described earlier.

I'll have to switch to Flexible IPv4 netflow and see if thinks work properly this way.

I'll update this thread with any new information.

(I have started questioning my decision to upgrade these routers, even though their IOS was 9 years old. Argh.)

Nick

I am still struggling with this.

I temporarily switched to IPv4 FNF (flexible netflow) but I obviously (as I found later) ran into this problem:

https://supportforums.cisco.com/discussion/11686271/switching-flexible-netflow-increases-cpu-load

After our routers suffered 100% load, I had to switch back to normal IPv4 netflow (with CEF fully enabled) and IPv4 netflow seems to be working correctly, but currently I am testing with:

ip flow ingress
ip flow egress

...configured ONLY on the ISP interface.

However, IPv6 FNF exporting (the only available IPv6 netflow export method on this IOS version) does not seem to be working properly. It magnifies traffic and packets by an estimated factor of about 100K. That is, IPv6 traffic that is really around 100packets/sec and 100 Kbps appears as: 1 G packets / sec and 10Gbps respectively.

I have been obliged to stop (at least temporarily) IPv6 recording, because it distorts real traffic graphs (data is collected and graphs are generated by nfdump/nfsen).

Until now, I have not been able to find a solution. If someone can advise, I will appreciate it.

       flow exporter IPv6
         destination 195.251.xxx.xxx
         source Loopback0
         transport udp 9995
       !
       flow monitor IPv6
         record netflow ipv6 original-output
         exporter IPv6
         transport udp 9995

And on the interfaces:

       ipv6 flow monitor IPv6 input

Perhaps the "record" statement should be configured differently?

What may be wrong?

Thanks in advance,
Nick

It finally worked (IPv6 FNF). I had to configure a user-defined record, because the pre-defined records would not produce correct results, as I described in my earlier posts, for reasons I have not been able to find.

Here is the working configuration:

   flow exporter IPv6
     destination 10.10.10.10
     source Loopback0
     transport udp 9995
   !
   !
   flow record ipv6_record
     match ipv6 protocol
     match ipv6 source address
     match ipv6 destination address
     match transport source-port
     match transport destination-port
     match flow direction
     collect routing source as
     collect routing destination as
     collect ipv6 dscp
     collect ipv6 source mask
     collect ipv6 destination mask
     collect transport tcp source-port
     collect transport tcp destination-port
     collect transport tcp flags
     collect interface input
     collect interface output
     collect counter bytes
     collect counter packets
     collect timestamp sys-uptime first
     collect timestamp sys-uptime last
   !
   !
   flow monitor IPv6
     record ipv6_record
     exporter IPv6
   !

   and on all IPv6-enabled router interfaces:

    ipv6 flow monitor IPv6 input

All the best,
Nick

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: