topic Re: CSM bridge mode urgent issue. in Application Networking

CSM bridge mode urgent issue.

csco10387876 — Thu, 06 Sep 2007 05:22:40 GMT

Hi,

I have a pair of CSM running 4.2.6 (tried 4.2.7 too) on cat 6500 sup 720 chassis.

config is following :

vlan 902 server

ip address 192.168.1.36 255.255.255.224 alt 192.168.1.37 255.255.255.224

vlan 100 client

ip address 192.168.1.36 255.255.255.224 alt 192.168.1.37 255.255.255.224

vserver VS_MWINA_WWW

virtual 192.168.1.59 tcp www

serverfarm SF_MWINA_W

replicate csrp sticky

replicate csrp connection

persistent rebalance

inservice

real R_PARKINSON

address 192.168.1.42

inservice

real R_GUEDEL

address 192.168.1.39

inservice

serverfarm SF_MWINA_W

nat server

no nat client

real name R_SRV1 8098

inservice

real name R_SRV2 8098

inservice

I am sniffing on the PO to the CSM module and what I see is the SYN goin from the chassis to the blade, nothing else. then sometimes it goes well and I have SYN/ACK and ACKs following.

Any help would be greatly appreciated.

Re: CSM bridge mode urgent issue.

freost — Thu, 06 Sep 2007 17:54:04 GMT

I dont see a gateway on the client vlan. What are the reals default gateway?

Re: CSM bridge mode urgent issue.

csco10387876 — Fri, 07 Sep 2007 05:38:53 GMT

Hi,

This is bridge mode, the DG of the server is the firewall 😉

Re: CSM bridge mode urgent issue.

Gilles Dufour — Fri, 07 Sep 2007 06:27:12 GMT

if you sniff the portchannel, what do you see on the server side ? Is the SYN forwarded ?

This a basic L4 setup, so the CSM should forward the SYN immediately to a server as long as there is one available.

Is this in production or a lab for testing ?

There are some debug commands we can run, but only if you can reproduce with 1 specific client and control the traffic sent by this client.

Gilles.

Re: CSM bridge mode urgent issue.

csco10387876 — Fri, 07 Sep 2007 06:35:08 GMT

Sadly this is production.

I see the syn on the client side, not on the server side(based on mac address).

Those blade are not heavily used so we can try the debug if they are not too intrusive.

yes the setup is really basic and I don't see what could be the problem and I fear that it happens again when the site will be full production).

What is strange is that for the csm, all reals servers are ok, ping ok, tcp probe ok.

Luc

Re: CSM bridge mode urgent issue.

Gilles Dufour — Fri, 07 Sep 2007 08:16:54 GMT

what do you mean by based on mac address ??

You filter on dst mac address ? src mac ?

The CSM will rewrite the src mac...

Re: CSM bridge mode urgent issue.

csco10387876 — Fri, 07 Sep 2007 08:23:45 GMT

The syn packet is

source mac: firewall

dest mac : csm vip mac

the capture was source based ip

Luc

Re: CSM bridge mode urgent issue.

Gilles Dufour — Fri, 07 Sep 2007 10:33:36 GMT

you can capture a 'sho mod csm X tech proc 4' to see if there is any LB reject.

Gilles.

Re: CSM bridge mode urgent issue.

csco10387876 — Sat, 08 Sep 2007 07:22:48 GMT

Gilles,

I will try that on monday morning.

Thanks,

Re: CSM bridge mode urgent issue.

csco10387876 — Mon, 10 Sep 2007 04:52:59 GMT

Gilles,

Here is the output :

show module csm 1 tech-support processor 4

Software version: 4.2(7)

--------------------------------------------------------------

------------------------ LB Cfg Stats ------------------------

--------------------------------------------------------------

LB ACL pool 524256 / 524288

LB Server NAT pool 261888 / 262144

LB Redir URL pool 1572832 / 1572864

LB common pool 4193712 / 4194304

--------------------------------------------------------------

------------------------ LB Statistics -----------------------

--------------------------------------------------------------

Total Rslt Msgs Discarded 0

Total Sess Seq Mismatched 8 (4)

Total Sess Reused w/o End 2

Total Ref Invalid Session 3 (1)

Total Unknown Type Msgs 0

Total Per-pkt balanced 0

Total Close Session Msgs Tx 0

Total Reassign Conn Msgs Tx 0

Total Cookie SC Msgs Tx 0

Total Dest Msgs to TCP Tx 329880 (138101)

LB Rjct: rule not match 0

LB Rjct: no real 5

LB Rjct: acl denied 0

LB Rjct: no cfg policy 9 (3)

LB Rjct: svr init denied 0

LB Rjct: invalid rlid 0

LB Rjct: VIP max-conn 0

LB Rjct: L7 ver mismatched 0

LB Rjct: L7 max parse len 0

LB Rjct: L7 SSL format 0

LB Rjct: L7 parser 0

LB Rjct: no rslt buffer 0

LB Rjct: no lkup buffer 0

LB Rjct: no cookie buffer 0

LB Rjct: no cl NAT port 0

Total New Session Msgs Rx 329906 (138101)

Total End Session Msgs Rx 329892 (138122)

Total Cookie Rslt Msgs Rx 0

Total Header Hash Rslt Msgs Rx 0

Total Rtcode Rslt Msgs Rx 0

Total Other Rslt Msgs Rx 0

Total End Parsing Msgs Rx 0

Total Slowpath Msgs Tx 0

Total Slowpath Msgs Rx 0

Console Enabled Flag 0

--------------------------------------------------------------

Free Result Buffers 655359

Free Ck Hash Buffers 98303

Free Ck Analysis Buffers 4095

Free Ck Sm-Analysis Buffers 49151

Free Retcode Result Buffers 3583

Free LeastConn Buffers 16383

I don't see anything particularly wrong 😉

Re: CSM bridge mode urgent issue.

Gilles Dufour — Mon, 10 Sep 2007 07:26:59 GMT

LB Rjct: no real 5

LB Rjct: no cfg policy 9 (3)

So the CSM was not able to loadbalance the SYN only 14 times. That's really not a lot.

This makes me wonder if the SYN was really dropped. I feel like it must have been transmitted but you didn't see it.

Gilles.

Re: CSM bridge mode urgent issue.

csco10387876 — Mon, 10 Sep 2007 07:40:17 GMT

If it was transmitted, ok I didn't see it but I don't see where it would have gone.

The csm is a fine blade but sometimes not easy to trouleshoot I find.

With our config I don't see what could cause it to stop working.

Tech Proc 1 give me this

scsm1 tech proc 1

Software version: 4.2(7)

--------------------------------------------------------------

--------------------- SESSION Statistics ---------------------

--------------------------------------------------------------

Current time 438570 324085 1

Aborted rx 152564848 2673378996 10183

Total Packets rx 163666741 101777820 387

Packets Dropped 80262 59218 0

Packets Drop Stale Connection 22473 16390 0

Packets Drop No More Sessions 0 0 0

Packets Drop No VLAN 233026 172035 0

Packets Drop Bad Checksum 0 0 0

Packets Drop IP Fragments 0 0 0

Packets Drop SI with no SMAC 0 0 0

Packets Drop: SI, Route Mode, no DMAC 116827 115609 0

Packets Drop: Not IP, SNAP 0 0 0

Packets Drop: Zero L3 offset 0 0 0

Packets Drop: vlan/vs Force Drop 204 0 0

Packets Drop: Slowpath limit exceeded 0 0 0

Packets Drop: LP non-ip, non-arp 0 0 0

Packets Drop: TCP/UDP with zero port 1 0 0

Packets Drop: CDP 0 0 0

Packets Spanning Tree DMAC 0 0 0

Packets Repeat: Slowpath limit exceeded 0 0 0

Packets Rx on secondary vlan 0 0 0

Packets Slowpath 5056349 3584950 13

Packets Shakira 0 0 0

Packets High Priority 467142 346215 1

Packets Session Hit 43583067 12829485 48

Packets New Sessions 333858 142719 0

New Session- source route checks 79701 22473 0

New Session- source ecmp route 0 0 0

Packets Repeat 114240674 84857415 323

Packets Repeat Reverse Frag 0 0 0

Packets Repeat and Slowpath 0 0 0

Packets Force Repeat 0 0 0

Packets One Shot 0 0 0

Packets bad parse 0 0 0

Packets Session Hit TCP+NAT 0 0 0

Packets Session Hit TCP 1364769 591465 2

Packets Session Hit NAT 42218298 12238019 46

Packets Session Hit Slw 0 0 0

Packets Session FIN 664593 283296 1

Packets Dropped- SYN+ACKs 0 0 0

Packet, Transmit retries 0 0 0

SYN Packets routed (w/o conn) 115956 115143 0

Packets routed (w/o conn) 0 0 0

Packets routed (w/o conn), bad enc 0 0 0

Packets routed (w/o conn), FT 0 0 0

Packets with no SMAC, sent to slowpath 539 0 0

there are quite a lot of drops here.