cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2775
Views
15
Helpful
12
Replies

Slow server to server copy

galaga
Level 1
Level 1

Slow server 2008 to server 2012 copy,copies time out 

copies from 2012 to 2008 are fine

both server are VM

cisco nexus 5k on each end

mpls in between 

nexus interface report no errors or bandwidth seems fine

looking for troubleshooting tips would like to make sure this is not a network issue 

thanks 

12 Replies 12

nazimkha
Level 4
Level 4

I would do a packet capture where the server is connected. Look for TCP re-transmissions packet drops.

 

I recently debugged a similar issue and found out that the server is setting the MTU as 1500 with no fragment bit set. Once the MTU was adjusted (as the server team said that they have to set the no fragment bit) it was all good.

 

In another instance, the server gateway (SVI) was on the Nexus 9K and the ip redirects was enabled...disabled and it was all good

 

 

 

What MTU adjustments were made ?

Did a ping test to determine the WAN MTU and then reduced the MTU from the server application as they could not disable 'DO-NOT-FRAGMENT' flag

Ok thanks

mojafri
Cisco Employee
Cisco Employee

Hi,

 

So Win 2012 to 2008 works fine? You mean to say in reverse direction? Or its all together a different VMs? If same, what trace route is saying from 2008-->2012 & 2012-->2008? Is it taking multiple hops? 

 

Regards,

MJ

Yes reverse direction works good trace route is 8 hops same each way

Cool..! then do you see tcp re-transmission? Is there any difference in tcp window size? 

What could be the difference in windows size value ? 

 

copy to windows size value 11595 source length 60 windows scaling factor -1 (unknown)
copy from windows size value 510 source length 1514 windows scaling factor -1 (unknown)

Hi,

 

So file transfer is kinda "uploading"- on receiving end if window size is small you will "Window full" messages on wireshark. Do you see any? Any re-transmission? 

If you don't see any QoS/queuing drop or fwm drop as @paul driver mentioned, you have to focus on vmware & OS layer. 

 

1. Do both VMs reside on same datastore? What type of datastore both VMs are using? Local/SAN(FC/FCoE)/remote(NFS) 

2. Do you see disk/memory utilization going high on target VM during copy to target machine? 

3. Adapter type you are using for both VMs? vmxnet or vmxnet3

4. Hosts where both VMs reside, does that have same hardware specification? Specifically adapter type/bandwidth?

5. Do you have RSS/tcp offload enabled on Windows adapter? 

6. Any increment in error or rx_no_bufs counters? SSH to esxi and get the below outputs: 

vsish
/> cd /net/pNics/vmnicX/
/> cat stats
 X:=> vmnic ID used by host where VM traffic is traversing. 

 

Please rate if you find it helpful.  

 

Regards,

MJ

here is some of the wireshark output, not a big wireshark guy

Windows file size on is small (on destination ?) you can see below

adapter type is vmxnet3

dont see any high memory or disk utilization 

for this question 5. Do you have RSS/tcp offload enabled on Windows adapter?  enabled on both

 

4. Hosts where both VMs reside, does that have same hardware specification? Specifically adapter type/bandwidth?

similar hardware, same bandwidth,,speed 1000

 

17259 41.287265 10.6.10.88 10.99.12.159 SMB2 1514 Write Request Len:65536 Off:15859712 File: WindowsRefreshPkg.msu [TCP segment of a reassembled PDU]
17260 41.287267 10.6.10.88 10.99.12.159 TCP 1514 53823 → 445 [ACK] Seq=15959292 Ack=31348 Win=513 Len=1460 [TCP segment of a reassembled PDU]
17304 41.287373 10.6.10.88 10.99.12.159 SMB2 1514 Write Request Len:65536 Off:15925248 File: WindowsRefreshPkg.msu [TCP segment of a reassembled PDU]

10.46.10.91 10.100.12.167
Frame 17440: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) on interface 0
Interface id: 0 (\Device\NPF_{46344EB4-273B-4FB7-8DD4-874F74508860})
Encapsulation type: Ethernet (1)
Arrival Time: Oct 24, 2018 14:09:23.794619000 Eastern Daylight Time
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1540404563.794619000 seconds
[Time delta from previous captured frame: 0.000003000 seconds]
[Time delta from previous displayed frame: 0.000003000 seconds]
[Time since reference or first frame: 41.298536000 seconds]
Frame Number: 17440
Frame Length: 1514 bytes (12112 bits)
Capture Length: 1514 bytes (12112 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:tcp:nbss:smb2:data]
[Coloring Rule Name: SMB]
[Coloring Rule String: smb || nbss || nbns || nbipx || ipxsap || netbios]
Ethernet II, Src: Vmware_9c:11:3d (00:50:56:9c:11:3d), Dst: All-HSRP-routers_00 (00:00:0c:07:ac:00)
Destination: All-HSRP-routers_00 (00:00:0c:07:ac:00)
Address: All-HSRP-routers_00 (00:00:0c:07:ac:00)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: Vmware_9c:11:3d (00:50:56:9c:11:3d)
Address: Vmware_9c:11:3d (00:50:56:9c:11:3d)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.46.136.91, Dst: 10.100.128.167
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 1500
Identification: 0x619e (24990)
Flags: 0x4000, Don't fragment
Time to live: 128
Protocol: TCP (6)
Header checksum: 0x0000 [validation disabled]
[Header checksum status: Unverified]
Source: 10.6.10.88
Destination: 10.99.12.159
Transmission Control Protocol, Src Port: 53823, Dst Port: 445, Seq: 16220504, Ack: 31348, Len: 1460
Source Port: 53823
Destination Port: 445
[Stream index: 4]
[TCP Segment Len: 1460]
Sequence number: 16220504 (relative sequence number)
[Next sequence number: 16221964 (relative sequence number)]
Acknowledgment number: 31348 (relative ack number)
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 1... = Push: Set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: ·······AP···]
Window size value: 513
[Calculated window size: 513]
[Window size scaling factor: -1 (unknown)]
Checksum: 0x2363 [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
[SEQ/ACK analysis]
[Bytes in flight: 131400]
[Bytes sent since last PSH flag: 65700]
[Timestamps]
[Time since first frame in this TCP stream: 30.286256000 seconds]
[Time since previous frame in this TCP stream: 0.000003000 seconds]
TCP payload (1460 bytes)
TCP segment data (34 bytes)
[Reassembled PDU in frame: 17489]
TCP segment data (1426 bytes)
[46 Reassembled TCP Segments (65652 bytes): #17395(1378), #17396(1460), #17397(1460), #17398(1460), #17399(1460), #17400(1460), #17401(1460), #17402(1460), #17403(1460), #17404(1460), #17405(1460), #17406(1460), #17407(1460), #17408(1460), ]
NetBIOS Session Service
Message Type: Session message (0x00)
Length: 65648
SMB2 (Server Message Block Protocol version 2)
SMB2 Header
Server Component: SMB2
Header Length: 64
Credit Charge: 1
Channel Sequence: 0
Reserved: 0000
Command: Write (9)
Credits requested: 0
Flags: 0x00000000
Chain Offset: 0x00000000
Message ID: Unknown (388)
Process Id: 0x0000feff
Tree Id: 0x00000001
Session Id: 0x0000bc0000000065
Signature: 00000000000000000000000000000000
[Response in: 19684]
Write Request (0x09)
StructureSize: 0x0031
Data Offset: 0x0070
Write Length: 65536
File Offset: 16121856
GUID handle File: Windows6.1-KB958830-x64-RefreshPkg.msu
Channel: None (0x00000000)
Remaining Bytes: 0
Write Flags: 0x00000000
Blob Offset: 0x00000000
Blob Length: 0
Channel Info Blob: NO DATA
Data (65536 bytes)
Data: ccf428b0e463b53594d69041a0ab28760445a44470d01fcb...
[Length: 65536]

Hello

Do you have any span sessions open?

Show monitor session

show queuing interface x/x

show platform fwm info pif ethernet x/x | in drops

 


Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Hello,

 

On destination, on spanning session open on source none

for show queuing interface Eth1/49 same before and after copy

on destination, intface is part of VPC on source it is not

Server guys seem to think it network and vice versa ? Any guess ?

 

for this command, my os does not have pif show platform fwm info pif ethernet Eth1/50 | in drops

on source I get this:

show platform fwm event err
on source switch during copy
4) Event:E_DEBUG, length:128, at 500486 usecs after Wed Nov 7 19:11:44 2018
[102] Failed to convert cdce (ig 0 ul 1 switch_id abe subswitch_id 0 endnode
_id 0 local_id 0) to an ifindex (mac 0050.569c.5ef5)

show queuing interface Eth1/49 (Before copy)
Ethernet1/49 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 100

RX Queuing
qos-group 0
q-size: 100160, HW MTU: 1500 (1500 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Pkts received over the port : 1601006029
Ucast pkts sent to the cross-bar : 3115452303
Mcast pkts sent to the cross-bar : 2777159843
Ucast pkts received from the cross-bar : 3230866123
Pkts sent to the port : 3656676645
Pkts discarded on ingress : 3361217
Per-priority-pause status : Rx (Inactive), Tx (Inactive)

show queuing interface Eth1/50 (before copy)
Ethernet1/50 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 100

RX Queuing
qos-group 0
q-size: 100160, HW MTU: 1500 (1500 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Pkts received over the port : 1212312339
Ucast pkts sent to the cross-bar : 1485181613
Mcast pkts sent to the cross-bar : 4017626235
Ucast pkts received from the cross-bar : 662075870
Pkts sent to the port : 3679966500
Pkts discarded on ingress : 4471807
Per-priority-pause status : Rx (Inactive), Tx (Inactive

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: