Ok good datapoint. So let's prove your assumption, then you can foucs on haivng your DevOps folks to spread out their API requests across the controllers (either statically or using a LB).
A simple check you can do against APIC1 vs. APIC2/3/4/5, is check the usage of NGINX (service that processes API requests) by using 'top'.
DCVLab-APIC1# top | grep nginx
22142 root 20 0 10.4g 9.7g 276708 S 0.9 15.6 5196:50 nginx.b+
22142 root 20 0 10.4g 9.7g 276708 S 2.4 15.6 5196:50 nginx.b+
22142 root 20 0 10.4g 9.7g 276708 S 1.0 15.6 5196:50 nginx.b+
22142 root 20 0 10.4g 9.7g 276708 S 1.3 15.6 5196:50 nginx.b+
22142 root 20 0 10.4g 9.7g 276708 S 5.6 15.6 5196:50 nginx.b+
CPU % Usage
Memory % Usage
Another option is to check the hits against the access.log which monitors all API requests coming into nginx service.
DCVLab-APIC1# pwd
/var/log/dme/log
DCVLab-APIC1# tail -f access.log
127.0.0.1 (::ffff:10.0.0.1) - - [14/May/2024:10:28:44 -0400]"GET /api/class/fabricPod.json?page-size=75000&page=0 HTTP/1.1" 200 236 "-" "python-requests/2.31.0"
127.0.0.1 (::ffff:10.0.0.1) - - [14/May/2024:10:28:44 -0400]"GET /api/mo/topology/pod-1.json?query-target=children&target-subtree-class=fabricNode&page-size=75000&page=0 HTTP/1.1" 200 2921 "-" "python-requests/2.31.0"
127.0.0.1 (::1) - - [14/May/2024:10:28:46 -0400]"GET /api/node/class/topSystem.json?query-target-filter=eq(topSystem.role, \x22controller\x22) HTTP/1.1" 200 1337 "-" "DC/Go-http-client/1.1"
//snip
log_format proxy_ip '$remote_addr ($http_x_real_ip) - $remote_user [$time_local]'
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
Again, you're trying to compare APIC1 with other controllers activity levels. Let us know what you find.
Robert
