WAE 4.1.3b: preposition issues in one sense

[ruben.montes]

Hi,

I have recently upgraded to 4.1.3b. I'm using new CIFS functionality and I'm having preposition problems. In one direction is working fine, but in the other direction i'm having "Network initialization error". I know this error appears when the WAEs are not inline, but customer assures me they are.

If there are connectivity probelms (like a firewall dropping packets), would I have the same error message? When the WAE does the Preposition, what kind of traffic is expected traversing the firewall? Both WAE are working inline.

What else could I check?

Regards,

Ruben

[darkfact]

Are the WAE's configured to use directed mode?  If there is a firewall between devices, this can be configured to allow traffic between devices to be encapsulated using udp.  The inline adapters just need to be configured with an IP address (If they aren't already) and directed mode turned on in network settings.  You may need to add an inspection rule in the firewall for the specific udp port to allow through (Default 4050.)  Hope that helps.

[ruben.montes]

Hi,

customer has told me the Firewall has the same rules in both sites (remember preposition is working in one direction). I have upgradedto 4.1.5f and result is the same:

%WAAS-CIFSAO-4-131237: (190068) Preposition ID 6274 failed, reason: network initialization error, retrying in 30 seconds.

How can I troubleshoot this?

Regards,

Ruben

[Michael Korenbaum]

Ruben,

This message can mean a few things:

1)  As previously suggested redirection to the server side WAE did not occur properly (e.g. missing wccp redirect statement, FW in path striping option 0x21 on the SYN packet from the client WAE, or traffic going "around" an inline WAE via some alternating routing path, etc.).

2) The server side WAE can't ping or resolve the hostname of the server you're trying to preposition from (do a ping/dnslookup from the CLI to verify).

3) The credentials supplied in the preposition directive don't have sufficient rights to access the server.  On the CLI of both WAEs us the following command to check:

        WAE#windows-domain diagnostics smbclient "-L -U "

   The output should list the shares available on this server.

As side note I always suggest using an Administrator account when setting up preposition jobs to avoid any potential credential issues.

Another way to troubleshoot this would be to take packet captures on both WAEs right as you enable the preposition job.

On the client side WAE you will disable this preposition job:

     WAE(config)#no accelerator cifs preposition 6274

Then open another set of CLI sessions to both WAEs and use the tcpdump or tethereal command appropriate to your situation (depends on what type of redirection you are using).

See the following document on taking packet captures:

https://www.myciscocommunity.com/thread/8027?tstart=0

Once the captures are started re-enable your prepostion job

    WAE(config)#accelerator cifs preposition 6274

Once you receive the failure stop the captures with Ctrl C and FTP them off to your PC to view with wireshark (www.wireshark.org), or open a TAC case with the above information confirmed/gathered and new sysreports.

Cheers,

Mike Korenbaum

Cisco WAAS PDI Help Desk

http://www.cisco.com/go/pdihelpdesk

[smnambia]

Ruben,

I have also seen this message when

1)  There is an ACL for WCCP at the core location and the edge WAE is not permitted.

3)  The wrong location was selected in the setup of the PP(preposition) job.  The location should be of the the WAE that is close to the server.

When you click on Configure--Preposition---the preposition directive name and then click on Browse next to Root Share Directories, check if you are able to browse the share?

Check on the core/datacenter router or switch that has wccp configured if you are using redirect-list ,if yes then in the acl do you have a permit statement for the remote/edge wae ip?

Regards,

Smita Nambiar

Cisco PDI Helpdesk

http://www.cisco.com/go/pdihelpdesk

[ruben.montes]

Hi,

there's no WCCP here, it's inline.

I'll check the config following your suggestions: I'll post the results.

Thanks in advance,

Ruben

[ruben.montes]

Hi,

I have verified and the name is resolved correctly on both WAEs. The location in the preposition task is also correct. I cannot check for the moment the SMB access to the folder as I don't have the password (yet).

I have captured some traffic, and I can see traffic coming and going from the Edge WAE (10.241.17.253) and the server (10.161.101.113) in the port 445.

I attach the trace to see if anybody can give me a hint.

Regards,

Ruben

[Michael Korenbaum]

Ruben,

Where/ how was this packet capture taken?  All I see is a series of TCP 3-way hand shakes, but nothing else between the WAE and server?

It is also clear that TFO auto discovery is failing because the sequence number on the ACK packet is not jumping as it would normally do on an optimized flow.

So, without having more details it would seem that something is preventing TFO auto -discovery from occuring for this flow, resulting in a failed preposition job.  I would check the FW/ security device logs between these two WAEs to see if there are any entries that correlate to the connections listed in your packet capture.  This may point to the source of the problem (e.g. the FW stripping the TCP options one of the WAEs is placing in the SYN or SYN-ACK packets).

Cheers,

Mike Korenbaum

Cisco WAAS PDI Help Desk

http://www.cisco.com/go/pdihelpdesk

[ruben.montes]

Hi,

this has been taken in the "edge" WAE with the command:

tcpdump -i bond0 -s 1600 -w packets1.cap

So you think it's a firewall issue? Customerr firewall is in front of the servers, not in the optimized path.

Regards,

[smnambia]

Ruben,

Also on the edge wae sh run can you see the preposition configuration in the cli?

Like what is the output of sh run | i prepos from the edge wae?

Are there any alarms on the CM or the edge wae device with respect to secure-store or any other alarms (get the sh alarms detail output)?

From the CM when you telnet to edge wae device gui ,under CifsAO --Preposition are you able to view the preposition policy created?

For Preposition to work following communication should not be blocked

Client to branch WAE

Branch WAE to data center WAE

Branch WAE to file server

Data center WAE to file server 

Regards

-Smita

Cisco PDI Helpdesk

http://www.cisco.com/go/pdihelpdesk

http://www.cisco.com/go/pdihelp

[Michael Korenbaum]

Ruben,

You would need simultaneous packet captures on both WAEs to confirm this is the case, but I believe there is a device stripping the TCP options on the way back to this Edge WAE.  The reason I suspect this is because if you look at the SYN-ACK packets in the options field where you would expect TCP option 0x21 from the Core WAE are a series of NOPs, which in my experience has indicated a device (FW, Satellite provider, etc.) has stripped the TCP options.

If this is a production system, I would highly suggest you open a TAC case and provide them Edge and Core WAE sysreports, as well as packet captures from both WAEs while you are attempting to restart the preposition job.

If this is a Demo or PoC system you could open a case with the PDI Help Desk and provide the same since your company Dimension Data is a qualified partner.

Cheers,

Mike Korenbum

Cisco WAAS PDI Help Desk 

http://www.cisco.com/go/pdihelpdesk

[Patrick Moubarak]

Hi Ruben,

If you already checked that connectivity is ok, I would investiguate asymmetric routing...

I've seen this "Network initialization error" multiple times and everytime it was caused by asymmetric routing. The easiest way I found to check is to start a session and check:

show stats connection in both WAAS devices. If you see PT Asym (Server or client) then here's your problem...

Hope that helps,

Patrick