Cisco 2600 running config changed by itself - turned off routing

[Drew Clark]

Hello,

I'm looking for help from the experts here to confirm some facts:

We have a 2600 (yes, still running) at our Edge interfacing the ISP. The last couple days, overnight, the router would just stop routing. All links were up, router was accessible. Restarting the router brought things back.

After not seeing anything wrong in the config, we switched to a backup 2600 last night. Again, overnight, the same thing happened. This time, I found the running config had been changed! The default route had been deleted and "no ip routing" had been set.

Can anyone confirm that IOS cannot change this config on its own? Is there any technical reason this config would have changed without user intervention? If not, we may have been hacked. Telnet is not accessible from the Internet on either FE interface, but may have been accessed via a guest network. Thanks for any insight.

[rvarelac]

Hi dclark005, 

Well that´s a weird behavihor , next time this happen I would run a "Show version" and check the last reload reason , also enable logs and point them to a log server (it could be a simple PC running a software for example kiwi syslog server). 

 With that information we can have and idea of what´s going on . 

Regards ,

[Richard Burts]

Show version may provide helpful information. Also if the router has learned authoritative time (especially if it is running NTP) then show run will have the last time the config was changed (assuming that the config change was after the last reboot.

It might also be useful to enable accounting for level 15 commands which will clearly indicate in someone has issues the config t command.

HTH

Rick

[rskov]

dclark005,

We started having this same problem yesterday also.  We were running a 2621 router and switched to a 2811 and have the same problem.  Did you figure out what is happening?

Ross

[Drew Clark]

Thank you Rick and rvarelac for the replies.

It happened again two hours ago. I have to be on site on console to gain access. I will run your suggestions and post what I find. It sounds like you are confirming what I thought - the config cannot be changed without a user changing it - IOS won't change it on me in response to an event or a trigger, correct?

rskov - That's crazy you're having the same problem. No luck yet. My first thought was a security breach. I changed the enable password yesterday with no luck. Like you, we tried on two separate 2600 routers. Perhaps there's a security hole in the firmware? What version of IOS are you running? I'll post mine when on-site. Also, looking for similarities - who is your ISP?

[rskov]

We use Enventis for our ISP.   

The IOS on the 2811 we are using now is 12.4(2)T4.

We noticed that 2 commands were added to the running config.

no ip routing

no ip cef

The startup config is not changed.  We don't understand how the running config is being changed.  Since the startup config isn't changed a reload fixes the problem.

[Richard Burts]

One time I observed IOS dynamically put an interface into the admin down state in response to a network event. Up to that moment I was certain that the only way an interface got to admin down was when a person configured it. Since then I have been extremely cautious about saying that IOS would never do something (like change a default route).

While I can not state with certainty that IOS did not make the changes, I am certainly thinking that it is much more likely to be a person who is doing this. I continue to think that looking for timestamps of the last config change, time of last reboot, and logging of level 15 commands would be effective steps.

It also occurs to me that if you have not already done it, that changing passwords on the router, especially the enable password, would be a good idea.

I would think that you might also want to evaluate the possibilities of remote access to the router and perhaps to put restrictions on remote access.

- you might require SSH instead of telnet since SSH will require a user name and password (and SSH is more secure than telnet anyway).

- you might configure an access list and apply it to the vty using access-class to restrict remote access to the router.

- do you permit SNMP read/write access to the router? That is another avenue through which the change could be made. Perhaps you want to restrict where SNMP will be accepted from.

HTH

Rick

[Drew Clark]

Ross,

That exact behavior is what we're experiencing. Startup config isn't touched, and run config gets no ip def and no ip routing added to it. A reload or restart fixes it.

Richard,

These are good suggestions; thank you. One of the first things I tried was changing the enable password. Just now, I logged back into the tty and tried to enable, using the new password (which I had used successfully last night), and the password has reverted to the previous password! This makes me even more suspicious. Is it possible an entire ghost config is being loaded somehow, either by IOS or a person? I will most definitely check SNMP...that's a good thought.

Ross, have you been able to make any more progress on tracking this down?

My "sh ver" is below:

OSBA# sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IPBASE-M), Version 12.3(6c), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Tue 20-Jul-04 05:24 by kellythw
Image text-base: 0x80008098, data-base: 0x80ED06CC

ROM: System Bootstrap, Version 12.2(8r) [cmong 8r], RELEASE SOFTWARE (fc1)

OSBA uptime is 1 day, 4 hours, 21 minutes
System returned to ROM by reload
System image file is "flash:c2600-ipbase-mz.123-6c.bin"

cisco 2621XM (MPC860P) processor (revision 0x400) with 126976K/4096K bytes of memory.
Processor board ID JMX0849L0TE (4186345228)
M860 processor: part number 5, mask 2
Bridging software.
X.25 software, Version 3.0.0.
2 FastEthernet/IEEE 802.3 interface(s)
1 Serial network interface(s)
32K bytes of non-volatile configuration memory.
32768K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

Thanks!

[Drew Clark]

Ross,

Curious - do you have "ip http server" in your config file? I just realized this is on, and upon testing, I can access the router web server from the outside world. Using admin/<<enable pw>> I can run any config command I want via the web GUI, and any changes do NOT show up in the syslog.

I'm going to disable and see if this fixes the problem.

[Drew Clark]

Ross,

I've done a couple things today to try to troubleshoot. If I can't figure it out on this next outage, I'm simply going to replace the unit, seeing how this one is at least 15 years old and EOL.

1. I power-cycled the router so sh ver shows a power-on event. If, after the next outage, it shows it restarted due to "reload", this is a clear indication someone on a vty reloaded the config (according to Cisco docs).

2. I enabled debug logging and a syslog server to log all events.

Hopefully this will shed some light on the situation. I also disabled snmp community private RW per Richard's suggestion. 

Rick, I ran out of time to setup Accounting - hopefully the logs show some info. I'm still perplexed about the enable password reverting....

Ross, if you find out anything else, will you post?

Thanks, guys.

[Richard Burts]

The comment about http server/http secure server is a helpful reminder about another vehicle for remote access to the router. Along with SNMP I would either disable these, or (if you actually use them) make sure that they are doing authentication and that the passwords have been changed.

While there is often some value in replacing equipment that is as old as this router, I would caution you that what is happening probably has little to do with the age of the router. And if you replace this old router with a newer router that is configured the same, then whatever is allowing someone to access your old router will probably allow them to access the new router.

HTH

Rick

[rskov]

Switching from a 2621 to a 2811 did not solve the problem. We ended up running the command auto secure to harden the 2811 router we are using now.  I answered no to the firewall part as that seemed to slow the router down considerably.  Here is an article on the auto secure command.  The auto secure command changes all your passwords and sets up accounting for you, it also runs a bunch of other commands to disable unneeded features.

http://www.firewall.cx/cisco-technical-knowledgebase/cisco-routers/865-how-to-secure-your-cisco-router-using-cisco-autosecure-feature.html

Our router has been up for almost 24 hours now with out any outages and/or config changes.

[Drew Clark]

Thanks, Ross. I will attempt to run this as well. After disabling the http server interface, I haven't seen any more issues either.

[rskov]

http and https were both disabled on our router.

[Fahad Wasi]

 Hi,

 pls check the console and telnet password that you have given to this 2600 series Router, are you sure that passwords are complex ?

 Also pls check which version of IOS are you using, make sure that the router has new version of IOS.

 IOS is also a software for network device and if it is not updated, then their are chances that any hacker can hack the device.

 Thanks