Netapp snapmirror affectiing CIFS and ISCSI traffic

[carl_townshend]

Hi All

We are having an issue with some slow performance on some netapp devices

We have a VIF of 6 ports in an etherchannel to a 3750X switch, these ports never max out, however we do see output drops.

The server team are saying that when a snapmirror is running to the other filer all of the cifs and iscsi traffic is affected.

Any ideas what this could be?

cheers

[Anthony LaRosa]

Do the drops continuously increment if the counters are cleared? Are you doing any type of QOS?

Output is generally congestion related.

Regards,

Anthony 

[carl_townshend]

Hi
Yes the drops are always incrementing.
There is no qos enabled on the switch at all.
The ports never go above 40 percent utilisation, this can't be congestion?

[Anthony LaRosa]

NetApp snapmirror is a heavy process and the 3750 does not have a very large shared buffer pool per ASIC, even with mls qos disabled.

Could you post the output of the following commands:

sh platform pm if-numbers

show platform port-asic stats drop (interface ID for review)

If you include your output from show version I can set it up in my lap and verify the commands/load.

Regards,

Anthony 

[carl_townshend]

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Gi1/0/1   1    1    1    1/1  1    1    1    local     Yes     Yes
Gi1/0/2   2    2    2    1/0  1    2    2    local     Yes     Yes
Gi1/0/3   3    3    3    1/3  1    3    3    local     Yes     Yes
Gi1/0/4   4    4    4    1/2  1    4    4    local     Yes     Yes
Gi1/0/5   5    5    5    1/5  1    5    5    local     Yes     Yes
Gi1/0/6   6    6    6    1/4  1    6    6    local     Yes     Yes
Gi1/0/7   7    7    7    1/9  1    7    7    local     Yes     Yes
Gi1/0/8   8    8    8    1/8  1    8    8    local     Yes     Yes
Gi1/0/9   9    9    9    1/11 1    9    9    local     Yes     Yes
Gi1/0/10  10   10   10   1/10 1    10   10   local     Yes     Yes
Gi1/0/11  11   11   11   1/13 1    11   11   local     Yes     Yes
Gi1/0/12  12   12   12   1/12 1    12   12   local     Yes     Yes
Gi1/0/13  13   13   13   1/15 1    13   13   local     Yes     Yes
Gi1/0/14  14   14   14   1/14 1    14   14   local     Yes     Yes
Gi1/0/15  15   15   15   1/17 1    15   15   local     Yes     Yes
Gi1/0/16  16   16   16   1/16 1    16   16   local     Yes     Yes
Gi1/0/17  17   17   17   1/19 1    17   17   local     Yes     Yes
Gi1/0/18  18   18   18   1/18 1    18   18   local     Yes     Yes
Gi1/0/19  19   19   19   1/23 1    19   19   local     Yes     Yes
Gi1/0/20  20   20   20   1/22 1    20   20   local     Yes     Yes

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Gi1/0/21  21   21   21   1/25 1    21   21   local     Yes     Yes
Gi1/0/22  22   22   22   1/24 1    22   22   local     Yes     Yes
Gi1/0/23  23   23   23   1/27 1    23   23   local     Yes     Yes
Gi1/0/24  24   24   24   1/26 1    24   24   local     Yes     Yes
Gi1/1/1   25   25   25   0/6  1    25   25   local     Yes     Yes
Gi1/1/2   26   26   26   0/7  1    26   26   local     Yes     Yes
Gi1/1/3   27   27   27   0/20 1    27   27   local     Yes     Yes
Gi1/1/4   28   28   28   0/21 1    28   28   local     Yes     Yes
Te1/1/1   29   29   29   0/0  1    1    29   local     Yes     Yes
Te1/1/2   30   30   30   0/14 1    2    30   local     Yes     Yes
Gi2/0/1   55   55   1    1/1  2    1    1    remote    No      Yes
Gi2/0/2   56   56   2    1/0  2    2    2    remote    No      Yes
Gi2/0/3   57   57   3    1/3  2    3    3    remote    No      Yes
Gi2/0/4   58   58   4    1/2  2    4    4    remote    No      Yes
Gi2/0/5   59   59   5    1/5  2    5    5    remote    No      Yes
Gi2/0/6   60   60   6    1/4  2    6    6    remote    No      Yes
Gi2/0/7   61   61   7    1/9  2    7    7    remote    No      Yes
Gi2/0/8   62   62   8    1/8  2    8    8    remote    No      Yes
Gi2/0/9   63   63   9    1/11 2    9    9    remote    No      Yes
Gi2/0/10  64   64   10   1/10 2    10   10   remote    No      Yes
Gi2/0/11  65   65   11   1/13 2    11   11   remote    No      Yes
Gi2/0/12  66   66   12   1/12 2    12   12   remote    No      Yes
Gi2/0/13  67   67   13   1/15 2    13   13   remote    No      Yes

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Gi2/0/14  68   68   14   1/14 2    14   14   remote    No      Yes
Gi2/0/15  69   69   15   1/17 2    15   15   remote    No      Yes
Gi2/0/16  70   70   16   1/16 2    16   16   remote    No      Yes
Gi2/0/17  71   71   17   1/19 2    17   17   remote    No      Yes
Gi2/0/18  72   72   18   1/18 2    18   18   remote    No      Yes
Gi2/0/19  73   73   19   1/23 2    19   19   remote    No      Yes
Gi2/0/20  74   74   20   1/22 2    20   20   remote    No      Yes
Gi2/0/21  75   75   21   1/25 2    21   21   remote    No      Yes
Gi2/0/22  76   76   22   1/24 2    22   22   remote    No      Yes
Gi2/0/23  77   77   23   1/27 2    23   23   remote    No      Yes
Gi2/0/24  78   78   24   1/26 2    24   24   remote    No      Yes
Gi2/1/1   79   79   25   0/6  2    25   25   remote    No      Yes
Gi2/1/2   80   80   26   0/7  2    26   26   remote    No      Yes
Gi2/1/3   81   81   27   0/20 2    27   27   remote    No      Yes
Gi2/1/4   82   82   28   0/21 2    28   28   remote    No      Yes
Te2/1/1   83   83   29   0/0  2    1    29   remote    No      Yes
Te2/1/2   84   84   30   0/14 2    2    30   remote    No      Yes
Po1       488  488  0    1/0  10   1    1    local     No      No
Po2       496  496  0    1/0  10   2    2    local     No      No
Po3       504  504  0    16/0  10   3    3    local     No      No
Po4       512  512  0    16/0  10   4    4    local     No      No
Po10      560  560  0    1/0  10   10   10   local     No      No
Po15      600  600  0    1/0  10   15   15   local     No      No

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Po18      624  624  0    0/0  10   18   18   local     No      No
Po24      672  672  0    1/0  10   24   24   local     No      No
Po25      680  680  0    1/0  10   25   25   local     No      No
St1       872  872  0    16/0  1    0    0    internal  Yes     Yes
St2       873  873  0    16/0  2    0    0    internal  No      Yes

show platform port-asic stats drop gigabitEthernet 1/0/10

  Interface Gi1/0/10 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 1997915
    Queue 4
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 5
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 6
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 7
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

show platform port-asic stats drop gigabitEthernet 1/1/1

  Interface Gi1/1/1 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 299
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 2367368611
    Queue 4
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

#sh ver
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:11 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x02400000

ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(53r)SE1, RELEASE SOFTWARE (fc1)

NETAPP-SW uptime is 5 years, 14 weeks, 2 days, 13 hours, 15 minutes
System returned to ROM by power-on
System restarted at 22:26:21 UTC Sun Nov 14 2010
System image file is "flash:/c3750e-universalk9-mz.122-53.SE2/c3750e-universalk9-mz.122-53.SE2.bin"

[Anthony LaRosa]

Thanks.

Can you provide the name of the port-channel

Also the output of the following for the interfaces/ether-channel in discussion:

show etherchannel summary

show etherchannel load-balance

show interfaces (Port channel in question)

The information provided in your previous post will help determine if the ports in the etherchannel are using the same ASIC, aswell as potential options for remediation.

Regards,

Anthony 

[carl_townshend]

Number of channel-groups in use: 9
Number of aggregators:           9

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)          -        Gi1/0/23(P) Gi1/0/24(P)
2      Po2(SU)          -        Gi1/0/21(P) Gi1/0/22(P)
3      Po3(SU)          -        Gi2/0/23(P) Gi2/0/24(P)
4      Po4(SU)          -        Gi2/0/21(P) Gi2/0/22(P)
10     Po10(SU)        LACP      Gi1/0/13(P) Gi1/0/14(P) Gi2/0/13(P)
                                 Gi2/0/14(P)
15     Po15(SU)        LACP      Gi1/0/15(P) Gi1/0/16(P) Gi2/0/15(P)
                                 Gi2/0/16(P)
18     Po18(SU)         -        Gi1/1/1(P)  Gi1/1/2(P)  Gi1/1/3(P)
                                 Gi2/1/1(P)  Gi2/1/2(P)  Gi2/1/3(P)
24     Po24(SU)        LACP      Gi1/0/5(P)  Gi1/0/7(P)  Gi1/0/9(D)
                                 Gi2/0/5(P)  Gi2/0/7(D)  Gi2/0/9(P)
25     Po25(SD)        LACP      Gi1/0/6(D)  Gi1/0/8(D)  Gi1/0/10(D)
                                 Gi2/0/6(D)  Gi2/0/8(D)  Gi2/0/10(D)

EtherChannel Load-Balancing Configuration:
        src-dst-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
  IPv4: Source XOR Destination IP address
  IPv6: Source XOR Destination IP address

sh interfaces port-channel 25
Port-channel25 is down, line protocol is down (notconnect)
  Hardware is EtherChannel, address is d0d0.fd61.468a (bia d0d0.fd61.468a)
  Description: NETAPP-ROCFILER22-STORAGE
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Auto-duplex, Auto-speed, link type is auto, media type is unknown
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 02:01:42, output hang never
  Last clearing of "show interface" counters 19w6d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 7287099
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     36109689113 packets input, 26877698694815 bytes, 0 no buffer
     Received 2396371 broadcasts (2357764 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 2357764 multicast, 597 pause input
     0 input packets with dribble condition detected
     46476081999 packets output, 54401611474758 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out