10-15-2012 10:07 AM
All,
I have six Dell Equallogic SANs connected to two Cisco 3750X (Stacked) in a redundant fashion. I have enabled Jumbo Frames (9000) and I configured a Port-Channel connecting back to the Distribution Layer. Keep in mind, our SAN and Server traffic reside in the same VLAN. Our Virtualization Platform is Citrix / Xenserver. I'm currently seeing many Output Drops on the 3750X connecting the SANs. I believe the buffers are almost full. Any thoughts? Possible design recommendations?
Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50, peak 119 @ 1w2d):
49 in free list (20 min, 150 max allowed)
29870278 hits, 23 misses, 69 trims, 69 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 85 @ 1w2d):
23 in free list (10 min, 150 max allowed)
292659 hits, 39 misses, 117 trims, 117 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50, peak 260 @ 1w2d):
50 in free list (5 min, 150 max allowed)
7118336 hits, 87 misses, 261 trims, 261 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 16, permanent 10, peak 16 @ 1w2d):
0 in free list (0 min, 100 max allowed)
59 hits, 3 misses, 6 trims, 12 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 4, permanent 0, peak 7 @ 1w2d):
4 in free list (0 min, 4 max allowed)
325240 hits, 13959 misses, 27855 trims, 27859 created
0 failures (0 no memory)
Interface buffer pools:
RxQFB buffers, 2040 bytes (total 904, permanent 904):
896 in free list (0 min, 904 max allowed)
1676806 hits, 0 misses
RxQ0 buffers, 2040 bytes (total 1200, permanent 1200):
700 in free list (0 min, 1200 max allowed)
12989722 hits, 0 misses
RxQ1 buffers, 2040 bytes (total 128, permanent 128):
5 in free list (0 min, 128 max allowed)
18683526 hits, 766906 fallbacks
RxQ2 buffers, 2040 bytes (total 128, permanent 128):
1 in free list (0 min, 128 max allowed)
809774 hits, 6326 fallbacks, 0 trims, 0 created
6326 failures (0 no memory)
RxQ3 buffers, 2040 bytes (total 128, permanent 128):
1 in free list (0 min, 128 max allowed)
6272522 hits, 51299 fallbacks
RxQ4 buffers, 2040 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
761106 hits, 906039 misses
RxQ5 buffers, 2040 bytes (total 128, permanent 128):
64 in free list (0 min, 128 max allowed)
64 hits, 0 misses
RxQ6 buffers, 2040 bytes (total 128, permanent 128):
0 in free list (0 min, 128 max allowed)
128 hits, 0 misses
RxQ7 buffers, 2040 bytes (total 192, permanent 192):
61 in free list (0 min, 192 max allowed)
513149 hits, 0 misses
RxQ8 buffers, 2040 bytes (total 64, permanent 64):
0 in free list (0 min, 64 max allowed)
3405180 hits, 3390852 misses
RxQ9 buffers, 2040 bytes (total 1, permanent 1):
0 in free list (0 min, 1 max allowed)
1 hits, 0 misses
RxQ10 buffers, 2040 bytes (total 64, permanent 64):
1 in free list (0 min, 64 max allowed)
5460093 hits, 822752 fallbacks
RxQ11 buffers, 2040 bytes (total 16, permanent 16):
0 in free list (0 min, 16 max allowed)
16 hits, 0 misses
RxQ12 buffers, 2040 bytes (total 96, permanent 96):
0 in free list (0 min, 96 max allowed)
96 hits, 0 misses
RxQ13 buffers, 2040 bytes (total 16, permanent 16):
0 in free list (0 min, 16 max allowed)
16 hits, 0 misses
RxQ15 buffers, 2040 bytes (total 4, permanent 4):
0 in free list (0 min, 4 max allowed)
16195688 hits, 16195685 misses
IPC buffers, 2048 bytes (total 300, permanent 300):
290 in free list (150 min, 500 max allowed)
822814 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
Jumbo buffers, 9240 bytes (total 200, permanent 200):
200 in free list (0 min, 200 max allowed)
0 hits, 0 misses
Header pools:
10-18-2012 07:50 AM
Can you post the configuration? Enable flow-control. I also like the global command "buffer tune automatic"
http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_02.html#wp1181763
10-22-2012 08:01 AM
How big of an impact will "buffer tune automatic"make? I will have to upgrade my 3750X so I can use that command. I do have flow-control enabled on the switchports. Is it necessary to enable flow-control on the uplink connections going to the Distribution Layer? Thanks
10-26-2012 11:56 AM
I think configuring flow-control on the uplinks would also help. Although I am not positive this is needed. I think the buffer tune automatic is worth it. Before upgrading you could try increasing "huge buffers max allowed". Start with 8.
10-26-2012 12:12 PM
I assume you're talking about adjusting the Huge Buffers pool? If not, I don't see the command "huge buffers max allowed" globally or under the interfaces. I'm on IOS 122-55.SE3. Thanks for your help!
10-29-2012 07:51 AM
The command is "buffers huge max-free #".
After readying this some more, it may be better to adjust the hold-queue on the interface. Can you post a show interface for a port experiencing output drops?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide