WCCP problems with Websense and Nexus

[Jon-Finngard Moe]

Hi everyone,

I have a Websense V5000G2 web proxy connected to a pair of Nexus 7010s (NX-OS v6.0 (4)).

I am using WCCP to redirect web traffic to the proxy, but it only works periodically.

It seems the service (10) is lost every 1-10 minutes.

The wccp debug log look like this:

---------------------------------

2013 Oct 21 11:28:13.296309 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120f3

2013 Oct 21 11:28:14.297118 wccp: WCCP-EVNT: vrf outer service 10: Here_I_Am packet from 10.0.100.4 w/bad recive_id 0x120f2. Expected 0x120f3.

2013 Oct 21 11:28:14.297212 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120f3

2013 Oct 21 11:28:15.299743 wccp: WCCP-EVNT: vrf outer service 10: Here_I_Am packet from 10.0.100.4 w/bad recive_id 0x120f2. Expected 0x120f3.

2013 Oct 21 11:28:15.299841 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120f3

2013 Oct 21 11:28:18.301365 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120f3

2013 Oct 21 11:28:18.301558 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120f4

2013 Oct 21 11:28:24.307567 wccp: WCCP-EVNT: vrf outer service 10: Ignoring duplicate Redirect_Assignment packet from 10.0.100.4 w/ Receive ID:0x120f4 Key Chg Num:0x1

2013 Oct 21 11:28:28.311902 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120f4

2013 Oct 21 11:28:28.312094 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120f5

2013 Oct 21 11:28:38.322341 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120f5

2013 Oct 21 11:28:38.322534 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120f6

2013 Oct 21 11:28:48.332817 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120f6

2013 Oct 21 11:28:48.333011 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120f7

2013 Oct 21 11:28:58.345556 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120f7

2013 Oct 21 11:28:58.345754 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120f8

2013 Oct 21 11:29:23.355861 wccp: WCCP-PKT: vrf outer service 10: Sending Removal_Query packet to 10.0.100.4 w/ Receive ID 0x120f9

2013 Oct 21 11:29:23.355896 wccp: WCCP-EVNT: vrf outer service 10: Sending Removal_Query packet to 10.0.100.4 w/ Receive ID 0x120f9

2013 Oct 21 11:29:28.355719 wccp: Service 10 VRF outer lost on WCCP Router 172.20.1.2

2013 Oct 21 11:29:28.355753 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 1 routers, 1 usable WCCP clients, change #0x3506

2013 Oct 21 11:29:28.356149 wccp: Service 10 VRF outer lost on WCCP Router 172.20.1.3

2013 Oct 21 11:29:28.356177 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 0 routers, 1 usable WCCP clients, change #0x3507

2013 Oct 21 11:29:28 KA-15-N7010-01 %WCCP-1-SERVICELOST: Service 10 VRF outer lost on WCCP Client 10.0.100.4

2013 Oct 21 11:29:28.356400 wccp: Service 10 VRF outer lost on WCCP Client 10.0.100.4

2013 Oct 21 11:29:28.356533 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 0 routers, 0 usable WCCP clients, change #0x3508

2013 Oct 21 11:29:28.356887 wccp: WCCP-EVNT: vrf outer service 10: Flushing Mask/Value info from TCAM

2013 Oct 21 11:29:28.356923 wccp: Group 10 vrf outer batch building: Added policy 1 with opmode = DEL

2013 Oct 21 11:29:28.356942 wccp: Group 10 vrf outer batch building: Added policy 4 with opmode = DEL

2013 Oct 21 11:29:28.356960 wccp: Group 10 vrf outer batch building: Added policy 2 with opmode = DEL

2013 Oct 21 11:29:28.356977 wccp: Group 10 vrf outer batch building: Added policy 3 with opmode = DEL

2013 Oct 21 11:29:28.363234 wccp: WCCP-EVNT: Send Batch to SPM: Req id:0x97ac4ea, Group id 10, Change flags 0x0 

2013 Oct 21 11:29:28.363316 wccp: wccp_spm_update_policy_state: Setting policy 1 state for interface Vlan194, group 10 to SG_POLICY_NOT_PRESENT

2013 Oct 21 11:29:28.363379 wccp: wccp_spm_update_policy_state: Setting policy 4 state for interface Vlan198, group 10 to SG_POLICY_NOT_PRESENT

2013 Oct 21 11:29:28.363426 wccp: wccp_spm_update_policy_state: Setting policy 2 state for interface Vlan1872, group 10 to SG_POLICY_NOT_PRESENT

2013 Oct 21 11:29:28.363472 wccp: wccp_spm_update_policy_state: Setting policy 3 state for interface Vlan1874, group 10 to SG_POLICY_NOT_PRESENT

2013 Oct 21 11:29:28.363621 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 0 routers, 0 usable WCCP clients, change #0x3508

2013 Oct 21 11:29:28.745396 wccp: WCCP-EVNT: Rx from SPM: Req id:0x97ac4ea, Policy ID:0, OpMode:DEL, Interface:All, Type:Group Batch Request, Request status:OK, Error code:0x0, Error string:None

2013 Oct 21 11:30:19.428266 wccp: WCCP-EVNT: vrf outer service 10: Here_I_Am packet from 10.0.100.4: WCI length: 32, WCI flags: 0x2

2013 Oct 21 11:30:19.428453 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x0

2013 Oct 21 11:30:19.428647 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120fa

2013 Oct 21 11:30:29.439228 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120fa

2013 Oct 21 11:30:29.439281 wccp: Service 10 VRF outer acquired on WCCP Client 10.0.100.4

2013 Oct 21 11:30:29.439469 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 0 routers, 1 usable WCCP clients, change #0x3509

2013 Oct 21 11:30:29.439772 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120fb

2013 Oct 21 11:30:36.446039 wccp: WCCP-EVNT: vrf outer service 10: Dropping Redirect_Assignment packet from 10.0.100.4 w/o our router id

2013 Oct 21 11:30:39.449309 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120fb

2013 Oct 21 11:30:39.449505 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120fc

2013 Oct 21 11:30:39.449995 wccp: Service 10 VRF outer acquired on WCCP Router 172.20.1.3

2013 Oct 21 11:30:39.450020 wccp: Service 10 VRF outer acquired on WCCP Router 172.20.1.2

2013 Oct 21 11:30:39.450049 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 2 routers, 1 usable WCCP clients, change #0x350a

2013 Oct 21 11:30:49.473603 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120fc

2013 Oct 21 11:30:49.473786 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120fd

2013 Oct 21 11:30:58.469231 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 2 routers, 1 usable WCCP clients, change #0x350a

2013 Oct 21 11:30:58.469270 wccp: WCCP-EVNT: vrf outer service 10: Received valid Redirect_Assignment packet from 10.0.100.4 w/ Receive ID:0x120fd Key Chg Num:1, Num sets 1, Num values 2

2013 Oct 21 11:30:58.469600 wccp: wccp_spm_update_policy_state: Setting policy 1 state for interface Vlan194, group 10 to ADD_PENDING

2013 Oct 21 11:30:58.469666 wccp: Group 10 vrf outer batch building: Added policy 1 with opmode = ADD

2013 Oct 21 11:30:58.469683 wccp: wccp_spm_update_policy_state: Setting policy 4 state for interface Vlan198, group 10 to ADD_PENDING

2013 Oct 21 11:30:58.469733 wccp: Group 10 vrf outer batch building: Added policy 4 with opmode = ADD

2013 Oct 21 11:30:58.469750 wccp: wccp_spm_update_policy_state: Setting policy 2 state for interface Vlan1872, group 10 to ADD_PENDING

2013 Oct 21 11:30:58.469804 wccp: Group 10 vrf outer batch building: Added policy 2 with opmode = ADD

2013 Oct 21 11:30:58.469820 wccp: wccp_spm_update_policy_state: Setting policy 3 state for interface Vlan1874, group 10 to ADD_PENDING

2013 Oct 21 11:30:58.469870 wccp: Group 10 vrf outer batch building: Added policy 3 with opmode = ADD

2013 Oct 21 11:30:58.493906 wccp: WCCP-EVNT: Send Batch to SPM: Req id:0x97ad805, Group id 10, Change flags 0x80 

2013 Oct 21 11:30:58.494013 wccp: WCCP-EVNT: vrf outer service 10: Setting WCCP Mask/Value for Key:<10.0.100.4, 1>

2013 Oct 21 11:30:58.494028 wccp: Service 10 VRF outer Received 2 new mask/value elements

2013 Oct 21 11:30:58.966947 wccp: WCCP-EVNT: Rx from SPM: Req id:0x97ad805, Policy ID:0, OpMode:ADD, Interface:All, Type:Group Batch Request, Request status:OK, Error code:0x0, Error string:None

2013 Oct 21 11:30:58.967015 wccp: wccp_spm_update_policy_state: Setting policy 1 state for interface Vlan194, group 10 to ADD_SUCCESS

2013 Oct 21 11:30:58.967111 wccp: wccp_spm_update_policy_state: Setting policy 4 state for interface Vlan198, group 10 to ADD_SUCCESS

2013 Oct 21 11:30:58.967164 wccp: wccp_spm_update_policy_state: Setting policy 2 state for interface Vlan1872, group 10 to ADD_SUCCESS

2013 Oct 21 11:30:58.967212 wccp: wccp_spm_update_policy_state: Setting policy 3 state for interface Vlan1874, group 10 to ADD_SUCCESS

2013 Oct 21 11:30:58.967265 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 2 routers, 1 usable WCCP clients, change #0x350a

2013 Oct 21 11:30:59.470932 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120fd

2013 Oct 21 11:30:59.471107 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120fe

2013 Oct 21 11:31:05.476448 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 2 routers, 1 usable WCCP clients, change #0x350a

2013 Oct 21 11:31:05.476487 wccp: WCCP-EVNT: vrf outer service 10: Received valid Redirect_Assignment packet from 10.0.100.4 w/ Receive ID:0x120fe Key Chg Num:2, Num sets 1, Num values 2

2013 Oct 21 11:31:05.476502 wccp: WCCP-EVNT: vrf outer service 10: Redirect_Assignment packet had no change in assignment info

2013 Oct 21 11:31:19.491341 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120fe

2013 Oct 21 11:31:19.491532 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x120ff

2013 Oct 21 11:31:29.501710 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x120ff

2013 Oct 21 11:31:29.501889 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x12100

2013 Oct 21 11:31:39.512223 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x12100

2013 Oct 21 11:31:39.512415 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x12101

2013 Oct 21 11:31:49.522771 wccp: WCCP-EVNT: vrf outer service 10: Here_I_Am packet from 10.0.100.4 w/bad recive_id 0x0. Expected 0x12101.

2013 Oct 21 11:31:49.522863 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x12101

2013 Oct 21 11:31:59.533154 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x12101

2013 Oct 21 11:31:59.533342 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x12102

2013 Oct 21 11:32:19.563394 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x12102

2013 Oct 21 11:32:19.563585 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x12103

2013 Oct 21 11:32:27.564558 wccp: WCCP-EVNT: vrf outer service 10: Built new router view: 2 routers, 1 usable WCCP clients, change #0x350a

2013 Oct 21 11:32:27.564597 wccp: WCCP-EVNT: vrf outer service 10: Received valid Redirect_Assignment packet from 10.0.100.4 w/ Receive ID:0x12103 Key Chg Num:1, Num sets 1, Num values 2

2013 Oct 21 11:32:27.564611 wccp: WCCP-EVNT: vrf outer service 10: Redirect_Assignment packet had no change in assignment info

2013 Oct 21 11:32:29.573491 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x12103

2013 Oct 21 11:32:29.573675 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x12104

2013 Oct 21 11:32:39.583731 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x12104

2013 Oct 21 11:32:39.583925 wccp: WCCP-PKT: vrf outer service 10: Sending I_See_You packet to 10.0.100.4 w/ Receive ID 0x12105

2013 Oct 21 11:32:49.594335 wccp: WCCP-PKT: vrf outer service 10: Received valid Here_I_Am packet from 10.0.100.4 w/ Receive ID: 0x12105

..

..

WCCP config:

-----------

ip access-list WS_REDIRECT

  10 deny ip 10.0.100.4/32 any  (websense)

  20 permit ip 10.137.10.1/32 any (This is my test client)

ip wccp 10 redirect-list WS_REDIRECT

interface Vlan194  

  ip wccp 10 redirect in

and so on for the other vlans....

Any ideas on what might be causing this?

Best regards,

J

[Andrew Smale]

Hi J,

Are you still having this issue? We have almost exactly the same setup as you except with a V10000 appliance and see the same symptons you see. Our N7K's are running 6.1(4a). We have been having a horrid time getting help from Websense who is trying to push the blame on the Nexus side.

If we solve this problem I will post an update in case anyone else can benefit.

Thanks,

Andrew