CSP-OS Troubleshooting Tips

kramesh · ‎12-16-2021

This is a troubleshooting self help document for CSP5000 platform using CSPOS.

Problem 1 : Increase the timeout value for CSP GUI Session to facilitate image upload

I was trying to create a service using Nagios log service ova image which is about 2.5GB. I logged into to UI and went to configuration-> Repository -> '+' which is add file to repository -> gave the file path in my local computer and clicked upload.

But the GUI session would time out resulting in aborted upload.

Solution 1 :

Increase the Session time out value to high value (8192) Currently that field is blank

Go to Administration-> Host-> Session Idle Timeout (Seconds):
Set Session Idle Timeout (Seconds): to value you prefer (Max is 8192 Seconds)
Log out of the GUI and log back in for the change to take effect

I was trying to create a service using Nagios log service ova image which is about 2.5GB. I logged into to UI and went to configuration-> Repository -> '+' which is add file to repository -> gave the file path in my local computer and clicked upload. But the GUI session would time out resulting in aborted upload. Solution 1 : Increase the Session time out value to high value (8192) Currently that field is blank Go to Administration-> Host-> Session Idle Timeout (Seconds): Set Session Idle Timeout (Seconds): to value you prefer (Max is 8192 Seconds) Log out of the GUI and log back in for the change to take effect

Problem 2 : Unable to ping VM when using SRIOV

Commands to check port, traffic stats, vnic stats, ovs/sriov Mac learning table, MAC address assignment to confirm the correct ordering of Vnic etc.

Problem 3 : Low traffic throughput across interfaces in a VNF

Problem 4 : No response on VNF console

Problem/Task 5: Create/poweron/poweroff of vnfs with SRIOV interfaces stuck if CSP is not rebooted since last year

Spoiler

Create/poweron/poweroff of vnfs with SRIOV interfaces stuck if CSP is not rebooted since last year. This is due to a bug: CSCvq14748

To confirm that you are hitting this bug, please verify the traceback in the CSP logs (messages file) with the traceback mentioned in the bug above.

This bug has been fixed in 2.4.3 onwards. If the customer is ok with reboot of CSP, the reboot will fix it.

But to fix it without reboot, the TAC will have to go into the linux shell and excute some linux commands. These wont impact traffic or any current running vnfs on any pnics.

1. Identify the sriov interfaces used. if there are multiple sriov interfaces used the following steps need to be done on all pnics. Say the customer is using sriov "enp94s0f0"

2. login to the linux shell using the CSP: https://techzone.cisco.com/t5/CSP2100-5000/Loading-and-using-CSP-2100-Debug-Plugin/ta-p/1003804

3. Execute

Identify the pnics using SRIOV (see image below)
From the output above pnics enabled for sriov will have values other than 0 under the "enabled vFs" column

cd /sys/class/net/<pnic-name>/devices
ls -ltrh virtf* (you will see that the files have previous year dates)
touch -h virtfn* (make all files from the previous "ls' command current)
repeat this for all sriov pnics being used.

4. Now reboot the vnf and it should work.

Create/poweron/poweroff of vnfs with SRIOV interfaces stuck if CSP is not rebooted since last year. This is due to a bug: CSCvq14748 To confirm that you are hitting this bug, please verify the traceback in the CSP logs (messages file) with the traceback mentioned in the bug above. This bug has been fixed in 2.4.3 onwards. If the customer is ok with reboot of CSP, the reboot will fix it. But to fix it without reboot, the TAC will have to go into the linux shell and excute some linux commands. These wont impact traffic or any current running vnfs on any pnics. 1. Identify the sriov interfaces used. if there are multiple sriov interfaces used the following steps need to be done on all pnics. Say the customer is using sriov "enp94s0f0" 2. login to the linux shell using the CSP: https://techzone.cisco.com/t5/CSP2100-5000/Loading-and-using-CSP-2100-Debug-Plugin/ta-p/1003804 3. Execute Identify the pnics using SRIOV (see image below) From the output above pnics enabled for sriov will have values other than 0 under the "enabled vFs" column cd /sys/class/net/<pnic-name>/devices ls -ltrh virtf* (you will see that the files have previous year dates) touch -h virtfn* (make all files from the previous "ls' command current) repeat this for all sriov pnics being used. 4. Now reboot the vnf and it should work.