Well it lasted 20 days with

alexnswr34 · ‎01-01-2015

Hi all,

I have an 1841 router acting as ADSL gateway. It has maxed out RAM and HWIC-1ADSL module installed.

Some time after upgrade from IOS 15.1(4)M6 to IOS 15.1(4)M9 the router started to behave somewhat strange. Every 3-5 days routing process stops, that is if I do "sh run" I get "no ip routing" in configuration which wasn't there before. It appears that config was not edited by anyone either. On top of that router stops resolving DNS. A simple reboot fixes the DNS issue.

I get following in the logs (sh log):

003801: Jan 2 05:39:13.569 AEST: %SYS-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (1/0),process = DNS Server.
-Traceback= 0x63BB1550z 0x63BB1BF8z 0x610C92F8z 0x610C935Cz 0x610CA4FCz 0x61306CC8z 0x61306DD0z 0x61309AD8z 0x6130B11Cz 0x62C873F4z 0x62C873D8z
003802: Jan 2 07:59:53.603 AEST: %SYS-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (1/1),process = DNS Server.
-Traceback= 0x63BC658Cz 0x63BB1530z 0x63BB1BF8z 0x610C92F8z 0x610C93B4z 0x610CA4FCz 0x61306E10z 0x61309AD8z 0x6130B11Cz 0x62C873F4z 0x62C873D8z

Has anybody experienced something like this?

Is this a 15.1(4)M9 issue?

Reza Sharifi · ‎01-01-2015

Hi,

-Traceback= 0x63BB1550z 0x63BB1BF8z 0x610C92F8z 0x610C935Cz 0x610CA4FCz 0x61306CC8z 0x61306DD0z 0x61309AD8z 0x6130B11Cz 0x62C873F4z 0x62C873D8z

Traceback usually indicates a memory issue/software crash. Try loading a different version and see if the issue disappears.

HTH

alexnswr34 · ‎01-01-2015

Thanks Reza.

I have loaded back the 15.1(4)M6.

Will see how it'll behave now.

Just out of curiosity could memory problems cause "no ip routing" to appear in the config?

Reza Sharifi · ‎01-01-2015

Hi

Just out of curiosity could memory problems cause "no ip routing" to appear in the config?

I have never seen that before but of course that does not mean there isn't a bug in IOS that is causing that.

Now that you have loaded 15.1(4)M6, lets see if the issue goes away.

HTH

alexnswr34 · ‎01-03-2015

Same thing happened again today with 15.1(4)M6 loaded. First the Internet stopped working so I've logged into the router and sure enough got "no ip routing" int the config along with "! No configuration change since last restart". Routing capability was restored after "ip routing" command was issued, as expected (ping 8.8.8.8 came back with responses). Next there was DNS resolution issue as per above which got fixed after reboot.

I've looked in the logs after reboot an found following:

000059: Jan 3 12:27:36.189 AEST: %ALIGN-3-SPURIOUS: Spurious memory access made at 0x631586D4z reading 0x1C
000060: Jan 3 12:27:36.189 AEST: %ALIGN-3-TRACE: -Traceback= 0x631586D4z 0x631768E8z 0x63176998z 0x63177590z 0x631C2438z 0x631C2658z 0x62FE3ED0z 0x62FE4258z

The Output interpreter suggests that there might be an IOS bug or a memory leak.

  WARNING: The following processes are currently holding more than 1 MB of memory
  with 'Freed' memory less that 'Holding' memory:
    'Chunk Manager' (Holding 2106432 bytes)
    'DSL State Machin' (Holding 1008300 bytes)
  This is considered to be high. High memory usage can be normal, or can indicate
  a memory leak.

Everything seems to point to the problems with the memory.

Here is the output of "sh align"

Alignment data for:
1841 Software (C1841-ADVENTERPRISEK9-M), Version 15.1(4)M6, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Compiled Thu 14-Feb-13 03:21 by prod_rel_team

No alignment data has been recorded.

Total Spurious Accesses 1, Recorded 1

Address Count Traceback
1C 1 0x631586D4z 0x631768E8z 0x63176998z 0x63177590z
0x631C2438z 0x631C2658z 0x62FE3ED0z 0x62FE4258z

Output interpreter suggests that there might be a bug in the IOS.
Therefore M6 and M9 contain the same bug? What about M7 or M8?

I've had M6 for quite a while before I've upgraded to M9 and never had such a problem before. Could memory issues be related to the above mentioned behavior?

Reza Sharifi · ‎01-03-2015

Hi,

Can you post the output of "sh ver"?

Thanks,

alexnswr34 · ‎01-03-2015

sh ver
Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 15.1(4)M6, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Thu 14-Feb-13 03:21 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

C1841 uptime is 20 hours, 32 minutes
System returned to ROM by reload at 12:05:32 AEST Sat Jan 3 2015
System restarted at 12:07:34 AEST Sat Jan 3 2015
System image file is "flash:c1841-adventerprisek9-mz.151-4.M6.bin"
Last reload type: Normal Reload

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 1841 (revision 7.0) with 352256K/40960K bytes of memory.
Processor board ID FHK12252492
2 FastEthernet interfaces
1 ATM interface
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity disabled.
191K bytes of NVRAM.
1958896K bytes of USB Flash usbflash0 (Read/Write)
125440K bytes of ATA CompactFlash (Read/Write)

License Info:

License UDI:

-------------------------------------------------
Device# PID SN
-------------------------------------------------
*0 CISCO1841 FHK12252492

Configuration register is 0x2102

Reza Sharifi · ‎01-03-2015

Thanks,

I thought there maybe something wrong with your config register but it is correct.

I think, this is memory issue with these IOS versions. It could also be that these IOS versions are a lot newer than your device (1841) which is pretty old and they may not work together well.

Neil Haswell · ‎01-03-2015

So far my 887 router is holding with the 15.4(3)m

ios. I will let you know the first time it bails out but it was doing it twice a day and started just about a week ago. Before I had the same iOS for a year or more . Is this built in obsolescence ?

alexnswr34 · ‎01-04-2015

15.4.3M1(ED) is the latest version for 887

https://software.cisco.com/download/release.html?mdfid=281993180&flowid=48111&softwareid=280805680&release=15.4.3M1&relind=AVAILABLE&rellifecycle=ED&reltype=latest

Neil Haswell · ‎01-24-2015

Well it lasted 20 days with the new IOS before it died and exhibited the same problem.

Any ideas?

alexnswr34 · ‎01-26-2015

I've cleared out NVRAM on my router then reloaded configuration back via TFTP, it has been running for 4 days, 3 hours, 10 minutes so far. I'll report back when/if it'll stuff up again.

I've also removed IPS part from config. as IPS puts quite a lot of strain on the CPU without the AIM module.

Neil Haswell · ‎01-24-2015

Well it lasted 20 days with the new IOS before it died and exhibited the same problem.

Any ideas?

alexnswr34 · ‎01-04-2015

OK

It looks like 1841 has a specific version built for it unlike the rest of the 1800 series. Not sure if hardware revision matters (my is v.7) but I'd assume that all hardware revisions should work with a given IOS version written for a given platform.

I have forgotten about the USB stick in my 1841 which I have taken out now. I recall that this issue started to happen sometime after I have inserted the USB stick into the router. It's a long shot but who knows...

Neil Haswell · ‎01-02-2015

I have a very similar problem with C800-UNIVERSALK9-M Version 15.1(4r)M, RELEASE SOFTWARE (fc1) running on a cisco 887.

I am upgrading to 15.3-3.M3 to see if it is any more stable.

Routing and DNS processes stop - IOS 15.1(4)M9