Where can we collect debugs and logs
Console --- All the debugs/information messages are given as an output on line console. This means that all the output is pumped onto the serial cable that is connected to the router.
To enable type:
Router# logging console <level>
To disable type:
Router# no logging console
Advantage: Message can be seen on the console session and no command required to send the message to the Serial connection client such as "terminal monitor" in case of ssh/telnet session.
Disadvantage: High load on CPU. Might bring the CPU if the debug output is verbose as the console line has limited data rate and real time output which is more than what the Serial connection can handle slows the performance of the router and causes strain on CPU.
Monitor --- All debug/information messages are sent as output on line tty (the current active line). This is irrespective of the fact if this is a ssh or a telnet session. The log output is sent in the same TCP session active for ssh/telnet.
To enable type:
Router# logging monitor <level>
To disable type:
Router# no logging monitor
Advantage: Message can be seen on the ssh/telnet session in real time though "terminal monitor" command is required to get the output on the current active tty line. Also, compared to console logging this mechanism is not limited to the data rate of Serial line.
Disadvantage: Not a very high load on CPU as compared to logging console but still since the output is real time can bring down the router/slow down router's performance based on verbosity of logging.
Buffer --- All debug/information messages are stored in router's internal logging buffer. Does not generate real time output on the screen and can be seen on a later time based on buffer size. Buffer defined should not be too big and should leave enough space so that router has enough free memory for other memory intensive operations.
To enable type:
Router# logging buffered <size in bytes> <level>
To disable type:
Router# no logging buffered
To check logs stored in buffer:
Router# show log
Advantage: Since the messages are not generated real time and instead stored in internal logging buffer, load on CPU is less. Logs can be viewed at a later time.
Disadvantage: Limited by the size of buffer size defined. Logs will be tail dropped as the buffer is filled.
Syslog --- All debug/information messages are sent to an external Syslog server. Syslog uses UDP port number 514 so logging messages are sent on a UDP stream to the syslog server. Make sure that IP connectivity between Syslog server and the router is there and UDP port 514 is open. Since the messages are sent on an unreliable UDP stream, the Syslog Server should be local the router.
To enable type:
Router# logging <ip of syslog server>
Router# logging trap <level>
To disable type:
Router# no logging <ip of syslog server>
Best Practices:
1. Persistent Issue
Always use logging buffered for such issues. Do not send logs to monitor or console as real time output hampers performance.
Commands:
Router(config)# no logging console
Router(config)# no logging monitor
Router(config)# exit
Router# show memory stat
/* Check available memory under I/O Memory */
Router# configure terminal
Router(config)# logging buffered <size based on above output> debugging
Router(config)# exit
/* Capture the problem or call in question */
Router# clear log
[confirm]
/* Once the call has been made or appropriate problematic action has be completed issue the following commands */
Router# terminal len 0
/* Wait for 5 to 10 seconds and make sure you are logging the output on your telnet/ssh/serial connection client */
Router# show log
Router# terminal default len
2. Intermittent Issue:
Use Syslog server to capture the outputs since we can not keep the logs in syslog server due to the limitation of buffer size and impact on router's internal memory and CPU due to prolonged debugging.
Router(config)# no logging console
Router(config)# no logging monitor
Router(config)# no logging buffered
Router(config)# logging <ip of syslog server>
Router(config)# logging trap debugging
Now the messages would be sent to the ip mentioned above. Make sure IP connectivity is there and UDP port 514 is open. Also, it is a good idea to have this server local to the router.
Regardless of the type of logging being done, always issue the following commands:
Router(config)#service timestamps log datetime msec
This ensures that the debug output will have millisecond level accuracy and will be in local time instead of GMT.
Common Problems -
1. Debug output not visible.
Check id debugs are enabled:
Router# show debug
Check the logging level:
Router# show run | section logging
Router# show log
2. Debug output is being missed.
Always capture debugs in internal buffer. If real time output is captured, output might be missed because of the verbosity of the debugs. Also, capture debugs in logging buffer after entering "term len 0" so that the output does not have multiple annoying "More" statements.
Hope this helps in gaining clarity on debugging process. Any suggestions, discussions, modifications and corrections are welcomed.
References:
Using Debug Commands:
http://www.cisco.com/en/US/docs/ios/debug/command/reference/db_01.html
Cisco MultiService Debug lookup Tool:
http://www.cisco.com/en/US/tech/tk1077/technologies_tech_note09186a0080207ec6.shtml
Cheers
--
Udit Mehrotra
