10-30-2018 05:07 PM
Slow server 2008 to server 2012 copy,copies time out
copies from 2012 to 2008 are fine
both server are VM
cisco nexus 5k on each end
mpls in between
nexus interface report no errors or bandwidth seems fine
looking for troubleshooting tips would like to make sure this is not a network issue
thanks
10-31-2018 06:27 AM
I would do a packet capture where the server is connected. Look for TCP re-transmissions packet drops.
I recently debugged a similar issue and found out that the server is setting the MTU as 1500 with no fragment bit set. Once the MTU was adjusted (as the server team said that they have to set the no fragment bit) it was all good.
In another instance, the server gateway (SVI) was on the Nexus 9K and the ip redirects was enabled...disabled and it was all good
11-08-2018 04:39 AM
What MTU adjustments were made ?
11-08-2018 09:11 AM
11-08-2018 12:01 PM
10-31-2018 06:54 AM
Hi,
So Win 2012 to 2008 works fine? You mean to say in reverse direction? Or its all together a different VMs? If same, what trace route is saying from 2008-->2012 & 2012-->2008? Is it taking multiple hops?
Regards,
MJ
10-31-2018 07:41 AM
10-31-2018 08:45 AM
Cool..! then do you see tcp re-transmission? Is there any difference in tcp window size?
11-01-2018 05:16 AM
What could be the difference in windows size value ?
copy to windows size value 11595 source length 60 windows scaling factor -1 (unknown)
copy from windows size value 510 source length 1514 windows scaling factor -1 (unknown)
11-02-2018 09:19 PM
Hi,
So file transfer is kinda "uploading"- on receiving end if window size is small you will "Window full" messages on wireshark. Do you see any? Any re-transmission?
If you don't see any QoS/queuing drop or fwm drop as @paul driver mentioned, you have to focus on vmware & OS layer.
1. Do both VMs reside on same datastore? What type of datastore both VMs are using? Local/SAN(FC/FCoE)/remote(NFS)
2. Do you see disk/memory utilization going high on target VM during copy to target machine?
3. Adapter type you are using for both VMs? vmxnet or vmxnet3
4. Hosts where both VMs reside, does that have same hardware specification? Specifically adapter type/bandwidth?
5. Do you have RSS/tcp offload enabled on Windows adapter?
6. Any increment in error or rx_no_bufs counters? SSH to esxi and get the below outputs:
vsish
/> cd /net/pNics/vmnicX/
/> cat stats
X:=> vmnic ID used by host where VM traffic is traversing.
Please rate if you find it helpful.
Regards,
MJ
11-07-2018 10:22 AM
here is some of the wireshark output, not a big wireshark guy
Windows file size on is small (on destination ?) you can see below
adapter type is vmxnet3
dont see any high memory or disk utilization
for this question 5. Do you have RSS/tcp offload enabled on Windows adapter? enabled on both
4. Hosts where both VMs reside, does that have same hardware specification? Specifically adapter type/bandwidth?
similar hardware, same bandwidth,,speed 1000
17259 41.287265 10.6.10.88 10.99.12.159 SMB2 1514 Write Request Len:65536 Off:15859712 File: WindowsRefreshPkg.msu [TCP segment of a reassembled PDU]
17260 41.287267 10.6.10.88 10.99.12.159 TCP 1514 53823 → 445 [ACK] Seq=15959292 Ack=31348 Win=513 Len=1460 [TCP segment of a reassembled PDU]
17304 41.287373 10.6.10.88 10.99.12.159 SMB2 1514 Write Request Len:65536 Off:15925248 File: WindowsRefreshPkg.msu [TCP segment of a reassembled PDU]
10.46.10.91 10.100.12.167
Frame 17440: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) on interface 0
Interface id: 0 (\Device\NPF_{46344EB4-273B-4FB7-8DD4-874F74508860})
Encapsulation type: Ethernet (1)
Arrival Time: Oct 24, 2018 14:09:23.794619000 Eastern Daylight Time
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1540404563.794619000 seconds
[Time delta from previous captured frame: 0.000003000 seconds]
[Time delta from previous displayed frame: 0.000003000 seconds]
[Time since reference or first frame: 41.298536000 seconds]
Frame Number: 17440
Frame Length: 1514 bytes (12112 bits)
Capture Length: 1514 bytes (12112 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:tcp:nbss:smb2:data]
[Coloring Rule Name: SMB]
[Coloring Rule String: smb || nbss || nbns || nbipx || ipxsap || netbios]
Ethernet II, Src: Vmware_9c:11:3d (00:50:56:9c:11:3d), Dst: All-HSRP-routers_00 (00:00:0c:07:ac:00)
Destination: All-HSRP-routers_00 (00:00:0c:07:ac:00)
Address: All-HSRP-routers_00 (00:00:0c:07:ac:00)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Source: Vmware_9c:11:3d (00:50:56:9c:11:3d)
Address: Vmware_9c:11:3d (00:50:56:9c:11:3d)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.46.136.91, Dst: 10.100.128.167
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 1500
Identification: 0x619e (24990)
Flags: 0x4000, Don't fragment
Time to live: 128
Protocol: TCP (6)
Header checksum: 0x0000 [validation disabled]
[Header checksum status: Unverified]
Source: 10.6.10.88
Destination: 10.99.12.159
Transmission Control Protocol, Src Port: 53823, Dst Port: 445, Seq: 16220504, Ack: 31348, Len: 1460
Source Port: 53823
Destination Port: 445
[Stream index: 4]
[TCP Segment Len: 1460]
Sequence number: 16220504 (relative sequence number)
[Next sequence number: 16221964 (relative sequence number)]
Acknowledgment number: 31348 (relative ack number)
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 1... = Push: Set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: ·······AP···]
Window size value: 513
[Calculated window size: 513]
[Window size scaling factor: -1 (unknown)]
Checksum: 0x2363 [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
[SEQ/ACK analysis]
[Bytes in flight: 131400]
[Bytes sent since last PSH flag: 65700]
[Timestamps]
[Time since first frame in this TCP stream: 30.286256000 seconds]
[Time since previous frame in this TCP stream: 0.000003000 seconds]
TCP payload (1460 bytes)
TCP segment data (34 bytes)
[Reassembled PDU in frame: 17489]
TCP segment data (1426 bytes)
[46 Reassembled TCP Segments (65652 bytes): #17395(1378), #17396(1460), #17397(1460), #17398(1460), #17399(1460), #17400(1460), #17401(1460), #17402(1460), #17403(1460), #17404(1460), #17405(1460), #17406(1460), #17407(1460), #17408(1460), ]
NetBIOS Session Service
Message Type: Session message (0x00)
Length: 65648
SMB2 (Server Message Block Protocol version 2)
SMB2 Header
Server Component: SMB2
Header Length: 64
Credit Charge: 1
Channel Sequence: 0
Reserved: 0000
Command: Write (9)
Credits requested: 0
Flags: 0x00000000
Chain Offset: 0x00000000
Message ID: Unknown (388)
Process Id: 0x0000feff
Tree Id: 0x00000001
Session Id: 0x0000bc0000000065
Signature: 00000000000000000000000000000000
[Response in: 19684]
Write Request (0x09)
StructureSize: 0x0031
Data Offset: 0x0070
Write Length: 65536
File Offset: 16121856
GUID handle File: Windows6.1-KB958830-x64-RefreshPkg.msu
Channel: None (0x00000000)
Remaining Bytes: 0
Write Flags: 0x00000000
Blob Offset: 0x00000000
Blob Length: 0
Channel Info Blob: NO DATA
Data (65536 bytes)
Data: ccf428b0e463b53594d69041a0ab28760445a44470d01fcb...
[Length: 65536]
11-02-2018 04:07 PM
Hello
Do you have any span sessions open?
Show monitor session
show queuing interface x/x
show platform fwm info pif ethernet x/x | in drops
11-07-2018 11:25 AM
Hello,
On destination, on spanning session open on source none
for show queuing interface Eth1/49 same before and after copy
on destination, intface is part of VPC on source it is not
Server guys seem to think it network and vice versa ? Any guess ?
for this command, my os does not have pif show platform fwm info pif ethernet Eth1/50 | in drops
on source I get this:
show platform fwm event err
on source switch during copy
4) Event:E_DEBUG, length:128, at 500486 usecs after Wed Nov 7 19:11:44 2018
[102] Failed to convert cdce (ig 0 ul 1 switch_id abe subswitch_id 0 endnode
_id 0 local_id 0) to an ifindex (mac 0050.569c.5ef5)
show queuing interface Eth1/49 (Before copy)
Ethernet1/49 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 100
RX Queuing
qos-group 0
q-size: 100160, HW MTU: 1500 (1500 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Pkts received over the port : 1601006029
Ucast pkts sent to the cross-bar : 3115452303
Mcast pkts sent to the cross-bar : 2777159843
Ucast pkts received from the cross-bar : 3230866123
Pkts sent to the port : 3656676645
Pkts discarded on ingress : 3361217
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
show queuing interface Eth1/50 (before copy)
Ethernet1/50 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 100
RX Queuing
qos-group 0
q-size: 100160, HW MTU: 1500 (1500 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Pkts received over the port : 1212312339
Ucast pkts sent to the cross-bar : 1485181613
Mcast pkts sent to the cross-bar : 4017626235
Ucast pkts received from the cross-bar : 662075870
Pkts sent to the port : 3679966500
Pkts discarded on ingress : 4471807
Per-priority-pause status : Rx (Inactive), Tx (Inactive
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide