Slow response to APIC servers after upgrade

[kz-support]

Hello

On the ACI factory in one of the sites after upgrading to version from 4.2(7w) 5.3(2d) we record a slow response from the APIC servers when executing configuration and show commands via CLI.

The problem is especially noticeable when executing various “show run” commands and is noticed on all APIC servers in the 2nd site (RCOD)

Example of command execution time on sites:

Site1 version ACI4.2(7w) execution time “showepg <epg> ” - 00:05 sec

Site2 version ACI5.3(2d) execution time “showepg <epg>” - 00:39 sec

Site1 version ACI 4.2(7w) execution time “show run leaf <leaf> interface ethernet <port>” - 00:50 sec

Site2 version ACI 5.3(2d) execution time “show run leaf <leaf> interface ethernet <port>” - 02:15 sec

Site1 version ACI 4.2(7w) execution time “show run vpc context leaf <leaf1 leaf2> int vpc <vpc>” - 1:10 sec

Site2 version ACI 5.3(2d) execution time “show run vpc context leaf <leaf1 leaf2> int vpc <vpc>” – 2:45 sec

Site1 ACI version 4.2(7w) execution time “show run template policy-group <pg>” – 0:04 sec

Site2 ACI version 5.3(2d) execution time “show run template policy-group <pg>” – from 0:04 to 0:08 sec

Previously, the 2nd site had version 15.2(3e), these problems with slowing down the execution of commands via CLI were not observed.

[AshSe]

Hello @kz-support 

The issue you are describing is a common concern when upgrading Cisco ACI (Application Centric Infrastructure) environments to newer versions. Performance degradation in CLI command execution, especially for configuration and "show" commands, can occur due to several factors introduced in the newer version. Below are some potential causes and troubleshooting steps to address the issue:

---

Potential Causes

1. Increased Resource Utilization in Newer Versions:

   • Newer ACI versions (like 5.3(2d)) often introduce additional features, enhancements, and bug fixes, which can increase the resource utilization (CPU, memory, etc.) on the APIC controllers. This can lead to slower response times for CLI commands.

2. Database Changes:

   • Upgrades often involve changes to the underlying database schema or indexing. If the database is not optimized or if there are inefficiencies in the new schema, it can result in slower query execution times for commands like show run or show epg.

3. Increased Data Volume:

   • Over time, the ACI fabric may accumulate more configuration data, endpoints, policies, and other objects. The newer version may process this data differently, leading to longer execution times for commands that query or display this information.

4. Code Optimization Differences:

   • The newer version may have changes in how certain commands are processed or executed, which could inadvertently introduce inefficiencies.

5. APIC Hardware Limitations:

   • If the APIC hardware is older or underpowered, it may struggle to handle the increased demands of the newer ACI version.

6. Bugs in the New Version:

   • It's possible that the version 5.3(2d) has a bug or inefficiency causing the slow response times. Cisco often releases patches or updates to address such issues.

---

Troubleshooting Steps

1. Check APIC Resource Utilization:

   • Use the acidiag commands to check the CPU, memory, and disk utilization on the APIC controllers.

     acidiag fnvread
     acidiag avread
     acidiag sysinfo

   • Look for any resource bottlenecks that could be causing the slow response times.

2. Verify Database Health:

   • Check the health of the APIC database using the following command:

     acidiag dbcheck

   • If there are any database inconsistencies or issues, they may need to be resolved.

3. Check for Known Bugs:

   • Review the Cisco ACI release notes and bug tracker for version 5.3(2d) to see if there are any known issues related to slow CLI performance. If a bug is identified, upgrading to a newer patch or version may resolve the issue.

4. Compare Configuration Complexity:

   • Compare the configuration complexity (number of EPGs, policies, endpoints, etc.) between Site1 and Site2. If Site2 has significantly more objects, it could explain the slower response times.

5. Enable Debugging for CLI Commands:

   • Enable debugging for CLI commands to identify where the delay is occurring. For example:

     debug cli

   • This can help pinpoint whether the delay is due to database queries, processing, or other factors.

6. Test with a Different APIC Node:

   • If the issue is isolated to a specific APIC node, try executing the commands on a different APIC node in the same site to rule out hardware-specific issues.

7. Check Network Latency:

   • Ensure there is no network latency or connectivity issue between the APIC controllers and the devices being queried (e.g., leaf switches).

8. Upgrade to a Newer Version:

   1. If the issue persists and is identified as a bug in version 5.3(2d), consider upgrading to a newer version of ACI (if available) that addresses the issue.

9. Engage Cisco TAC:

   • If the issue cannot be resolved through the above steps, open a case with Cisco TAC. Provide them with the following information:
     • APIC logs (techsupport files)
     • Output of acidiag commands
     • Details of the commands experiencing delays
     • Comparison of performance between Site1 and Site2

---

Workarounds

1. Use the GUI or API:

   • If the CLI is slow, consider using the ACI GUI or API for configuration and monitoring tasks. These interfaces may not be affected by the same performance issues.

2. Optimize Command Usage:

   • For commands like show run, try to narrow the scope of the query (e.g., specify specific objects or filters) to reduce the amount of data being processed and displayed.

---

Next Steps

1. Perform the troubleshooting steps outlined above to identify the root cause of the issue.
2. If no resolution is found, escalate the issue to Cisco TAC for further investigation.
3. Monitor Cisco's release notes and advisories for any updates or patches that address CLI performance issues in version 5.3(2d).

By systematically addressing the potential causes, you should be able to identify and resolve the performance degradation in Site2.

Hope This Helps!!!

AshSe

Forum Tips: 

1. Insert photos/images inline - don't attach.
2. Always mark helpful and correct answers, it helps others find what they need.
3. For a prompt reply, kindly tag @name. An email will be automatically sent to the member.

[kz-support]

Thank you for you answer, and could you tell me will there be any impact on system performance when using the following suggested commands:

debug cli

acidiag fnvread
acidiag avread
acidiag sysinfo
acidiag dbcheck

I mean could I apply them during the working process, or I must plan th MW for it?

[AshSe]

Great question! It's always important to understand the potential impact of diagnostic commands on system performance, especially in a production environment. Here's a breakdown of the commands you mentioned and their potential impact:

1. debug cli
   • Enabling debugging can generate additional logs and consume some CPU and memory resources on the APIC.
   •  
2. acidiag fnvread
   • It has minimal impact on system performance and can be safely run during normal operations.
   •  
3. acidiag avread
   • It has minimal impact on system performance and can be safely run during normal operations.
   •  
4. acidiag sysinfo
   • It has minimal impact on system performance and can be safely run during normal operations.
   •  
5. acidiag dbcheck
   • In most cases, the impact is minimal, but if the APIC is already under heavy load, it could cause a temporary performance degradation.

Summary

1. Safe to Run During Normal Operations:
   1. acidiag fnvread
   2. acidiag avread
   3. acidiag sysinfo
2. Minimal Impact but Monitor Resource Usage:
   1. debug cli
   2. acidiag dbcheck (safe in most cases, but monitor resource usage or plan for MW if the system is under heavy load)

In most cases, these commands are safe to run during normal operations without requiring a maintenance window. However, always monitor the system's resource utilization and ensure the APIC is not under heavy load before running diagnostic commands.

Hope This Helps!!!

AshSe

Community Etiquette: 

1. Insert photos/images inline - don't attach.
2. Always mark helpful and correct answers, it helps others find what they need.
3. For a prompt reply, kindly tag @Name. An email will be automatically sent to the member.

[kz-support]

We've upgraded  to the version 6.0(8f) but problem still the same

and another quiestion - I didn't find the command 'debug cli' on the APIC CLI, and no any description on this command, is this a right syntax?