Introduction
This document explains how to recover D9036 encoder which is having issue with identifying daughter cards due to log system full in daughter cards.
Problem Description
Customer might see below message from its GUI(most likely firmware version under 1.5.27) even though it has cards on the slot with proper LED status.
In below example, the encoder was having proper cards on slot 1,2,5 properly
Components Used
D9036,putty(Hypertermimal)
Troubleshooting Steps
1. Connect D9036 serial cable between D9036 and laptop
make sure the serial parameter is(or use ssh if it is available)
- baud rate : 115200
- data bit : 8
- parity : none
- stop bit : 1
2. Once you connect the encoder via serial cable(or ssh), make sure some of boards are not recognized properly with below commands on hardware layer
#more /sys/devices/virtual/misc/pcie-slot-ctrl0/info
Make sure all boards information having actual cards physically all should be Present : Yes, Power:Good, State:On,Link-up.
Otherwise we can assume that typical slot has an issue with hardware level.
In this scenario, all hardware level communications are supposed to be ok.
3. If all H/W level commnucation are ok, then check from software level by typing
#get_mvp_hosts
You probably don't see board 1,5,6 information from here. With this information, we are able to say that there is an issue with commnucation between mainboard to board 1,5,6
4. For the first try, you may try warm and cold-reboot.
5. If it can't resolve the issue.Next step could be firmware updating to higher version than 1.5.27 via Upgrade-it.
you can download the latest firmware version from CISCO.COM
6. When the upgrade is done, check the commnucation status again with "get_mvp_hosts" again.
Hopfully, you are able to see rest of boards as well which were not before.
In this exampe: slot 1,5,6
7. Go to individual board which was having issue with recognizing from motherboard
In this example, we will use mvi01 card
#ssh "board_aliasname"
(or ip address from get_mvp_hosts command output)
password is root
8. Check filesystem usage of dauther card by typing "df-h"
9. You probably see /var/log file system usage is 100% with some reason
* In below screenshot, it is showing 1% but you will see 100% from your faulty encoder
10. Then go to the log partition then remove old log files manually
#cd /var/log
# rm -rf *.gz
11. To be sure, check the file system usage again.
12. As a final step,check H/W level commnucation status as well as S/W level again.
#get_mvp_hosts
#more /sys/devices/virtual/misc/pcie-slot-ctrl0/info
If commands outputs are similar to above.
Now, the encoder should be able to work without issue.