"Is there a reason we see Microbursts on switches when the average load is say 20-30% of the 10G link?"
Yes, there's a reason. BTW, 20 to 30% utilization only means 20 to 30% of the available quantity of bits were transmitted during some measured time interval. Every frame is transmitted at 100%.
"if the port can sustain line rate with no queuing then why do we see Microbursts?"
Oversubscription.
For example, assume there's 11 gig ports feeding to your 10g port, and all send a frame at the same time. 11 gig into 10 gig don't fit, so the excess 1 gig will either be queued or dropped. (Actually if all 11 frame were sent at the exactly the same time, 10 frames would need to be queued or dropped. However, the 10g interface could transmit 10 of the 11 received frames before those 11 gig ports could provide the next frame.)
"I see we can alleviate some of this with the command below
qos queue-softmax-multiplier"
Yes, often, but not always, true.
"How does it work then, say on a 9K switch, does each port have its own buffer, and if it needs more it borrows an amount from the shared pool? and the command above allows them to borrow more from the pool?"
I would need to reread how the 9K switches work. Cisco keep making minor changes in their various switches.
Yes, I believe the 9K has some buffers reserved per port (like some of the earlier switches did).
Yes, a port might borrow from a shared pool, but the softmax mainly adjusts logical queue limits, which may, or may not, require actually obtaining buffers from the shared pool.
"Also, if the port has no QOS enabled, i.e FIFO, this is one queue with more buffer reserved?"
Unsure you can actually disable QoS in the 9K (and/or since the 3650/3850 switches).
On earlier switches, port egress queues had their own hard buffer reservations too. Again, without researching it, don't know exactly how the 9K switches work. Do remember the 3650/3750 series had "knobs" to make all kinds of port/queue/buffer adjustments (which I never understood until a Cisco employee published a paper on these forums that really explained how they worked - which might still be available).
"if we enable QOS does this make the queue space smaller for say normal traffic as it divides the buffer space up between the queues?"
Again, unsure that's an option on the 9Ks. On the 3560/3750 you could enable/disable QoS, and when it was disabled, the default buffers allocations did seem to provide more buffer space for the one egress port queue compared to the default QoS using 4 egress queues per port. However, when QoS was enabled, you could reallocate buffers as desired, and remap as desired, i.e. you could direct all traffic to just one of the four egress queues and tweak its buffers and queue limits pretty much as desired.
