topic Re: CSM: create coredump in Application Networking

CSM: create coredump

mruuth — Tue, 10 Oct 2006 13:14:15 GMT

Hi all,

I have a problem with CSM and I want to create a coredump. Is it possible?

Regards

Mats

Re: CSM: create coredump

Gilles Dufour — Wed, 11 Oct 2006 08:08:07 GMT

Mats,

I would not recommend getting a core dump without first getting the advice from a Cisco TAC engineer.

The CSM core dump is a bunch of bitd and bytes very difficult to analyse.

So, it should be captured really as a last resort.

There are most probable other engineering command to get before forcing a core.

Gilles.

Re: CSM: create coredump

mruuth — Wed, 11 Oct 2006 11:37:20 GMT

Gilles,

I was asking because I had a strange problem yesterday but the problem was fixed when we moved to the standby. Now I want to reload the CSM, but before that, I want to save the information in core to disk. Otherwise I dont't have any information to give TAC except

text describing what happened. Right now I don't have an good standbyCSM.

Regards

Mats

Re: CSM: create coredump

Gilles Dufour — Wed, 11 Oct 2006 12:49:39 GMT

Mats,

as I said, csm core files are extremely difficult to analsyse. You can't load them into a CSM and start playing with it like GDB.

If you give a core file to the TAC they will just send it to us and we won't be able to find root cause with just a core.

Therefore, before going for the core dump, we need several 'show tech' captured at regular interval [liek 5 min] and sniffer traces of the CSM port-channel.

This is in case the problem is traffic related.

For other type of issues, there might be different command to use.

We will need these commands to be captured at the time the CSM is active and the problem occuring.

You should maybe go to the TAC now and have them tell you what info to capture.

Gilles.

Re: CSM: create coredump

mruuth — Wed, 11 Oct 2006 13:26:45 GMT

Gilles,

Ok then I reload the module. The problem I did have, was that in one of eight bridges, the servers was behaving like they were on the client Vlan. After activating of the standby CSM everything was OK.

Regards

Mats

Re: CSM: create coredump

Gilles Dufour — Wed, 11 Oct 2006 13:42:18 GMT

was it for connection originated by the server ?

or connection from client to server ?

Direct or through vip ?

What did you see happening with the traffic ?

I'm not sure what you mean by acting as if on client vlan.

Did the server go down and then up recently ?

Did you capture the CSM arp table at that time ?

Gilles.

Re: CSM: create coredump

mruuth — Thu, 12 Oct 2006 06:04:07 GMT

Gilles,

Connection was from client to server and every application server was ok. The servers was on servervlan regarding arp in CSM. A beginners problem with bridged configuration i that servers are still on the old(client) vlan and not moved to the new servervlan. In that case you get entrys with "sho mod csm x conn vserver", but with "sho real detail" you get entrys in "total conn failures". This behavior was what I saw and that is what you see when servers are on client vlan. We did a snoop at one server and we could see SYN from client and SYN ACK from server but CSM didn't react to that. Where did that packet go?

You could make a connection directly to a server but not via a VIP.

with my initial question I was hoping to get the possibilities to create a crashdump to complement other information for TAC.

Regards

Mats

Re: CSM: create coredump

Gilles Dufour — Thu, 12 Oct 2006 06:47:43 GMT

Mats,

the dump would definitely be useless for this kind of issue.

Most important is a sniffer trace, and then some engineering command to capture internal arp entries and forwarding table [this info is not included in the core dump].

There was a recent issue with SYN/ACK misrouted by the CSM.

It was fixed in 4.1.7 and 4.2.3

If you have a version older than these, it might be a good hit.

CSCsb46265: CSM: syn/ack misrouted - using wrong encap id - after arp

Gilles.

Re: CSM: create coredump

mruuth — Thu, 12 Oct 2006 08:22:08 GMT

Gilles,

I'm running 4.2.4. Could you advice me about which commands I shall do, so I'm prepared if the problem happens again.

Regard

Mats

Re: CSM: create coredump

mruuth — Thu, 12 Oct 2006 10:10:27 GMT

Gilles,

Since a couple of weeks we get a lot of these messages "Invalid encaps ID for get info." After upgrade end of July it has been no such messages until now. You can notice they often comes in 5-minutes interval (ARP-refresh real).

We have IOS 12.1(20)E2

Re: CSM: create coredump

Gilles Dufour — Thu, 12 Oct 2006 12:37:51 GMT

This is a known issue and a ddts has been opened. However, this is a cosmetic issue.

It should not affect CSM operations.

CSCek31065: IOS/CSM generates CSM9: Invalid encaps ID for get info messages

Gilles.

Re: CSM: create coredump

mruuth — Fri, 13 Oct 2006 08:03:52 GMT

Gilles,

What kind of information should I collect when (hopefully not) I stumble on the problem with a failed bridge in CSM?

Your truly

Mats

Re: CSM: create coredump

Gilles Dufour — Fri, 13 Oct 2006 09:37:53 GMT

you will need to access the csm console.

Then enter venus mode.

Then capture

arp_cache_print

vlan_tree_dump_raw

!<--- repeat the command for all the vlans involved - client vlan and server vlan --->

Then from ios, enable the following debug

'debug module ContentSwitchingModule X packet x.x.x.x/32' where x.x.x.x is a client that you control to limit the amount of traffic that will hit the debug.

Then open a connection that fails.

Make sure to capture a csm portchannel trace [portchannel not vlan as the csm might send the packet in the wrong vlan, so the portchannel will tell us where the packet is coming out].

Gilles.

Re: CSM: create coredump

mruuth — Fri, 13 Oct 2006 10:29:40 GMT

Gilles,

Thanks for a nice conversation and advices. I will follow your instructions if (hopefully not) the trouble occurs again.

Regards

Mats