CSM: create coredump

[mruuth]

Hi all,

I have a problem with CSM and I want to create a coredump. Is it possible?

Regards

Mats

[Gilles Dufour]

Mats,

I would not recommend getting a core dump without first getting the advice from a Cisco TAC engineer.

The CSM core dump is a bunch of bitd and bytes very difficult to analyse.

So, it should be captured really as a last resort.

There are most probable other engineering command to get before forcing a core.

Gilles.

[mruuth]

Gilles,

I was asking because I had a strange problem yesterday but the problem was fixed when we moved to the standby. Now I want to reload the CSM, but before that, I want to save the information in core to disk. Otherwise I dont't have any information to give TAC except

text describing what happened. Right now I don't have an good standbyCSM.

Regards

Mats

[Gilles Dufour]

Mats,

as I said, csm core files are extremely difficult to analsyse. You can't load them into a CSM and start playing with it like GDB.

If you give a core file to the TAC they will just send it to us and we won't be able to find root cause with just a core.

Therefore, before going for the core dump, we need several 'show tech' captured at regular interval [liek 5 min] and sniffer traces of the CSM port-channel.

This is in case the problem is traffic related.

For other type of issues, there might be different command to use.

We will need these commands to be captured at the time the CSM is active and the problem occuring.

You should maybe go to the TAC now and have them tell you what info to capture.

Gilles.

[mruuth]

Gilles,

Ok then I reload the module. The problem I did have, was that in one of eight bridges, the servers was behaving like they were on the client Vlan. After activating of the standby CSM everything was OK.

Regards

Mats

[Gilles Dufour]

was it for connection originated by the server ?

or connection from client to server ?

Direct or through vip ?

What did you see happening with the traffic ?

I'm not sure what you mean by acting as if on client vlan.

Did the server go down and then up recently ?

Did you capture the CSM arp table at that time ?

Gilles.

[mruuth]

Gilles,

Connection was from client to server and every application server was ok. The servers was on servervlan regarding arp in CSM. A beginners problem with bridged configuration i that servers are still on the old(client) vlan and not moved to the new servervlan. In that case you get entrys with "sho mod csm x conn vserver", but with "sho real detail" you get entrys in "total conn failures". This behavior was what I saw and that is what you see when servers are on client vlan. We did a snoop at one server and we could see SYN from client and SYN ACK from server but CSM didn't react to that. Where did that packet go?

You could make a connection directly to a server but not via a VIP.

with my initial question I was hoping to get the possibilities to create a crashdump to complement other information for TAC.

Regards

Mats

[Gilles Dufour]

Mats,

the dump would definitely be useless for this kind of issue.

Most important is a sniffer trace, and then some engineering command to capture internal arp entries and forwarding table [this info is not included in the core dump].

There was a recent issue with SYN/ACK misrouted by the CSM.

It was fixed in 4.1.7 and 4.2.3

If you have a version older than these, it might be a good hit.

CSCsb46265: CSM: syn/ack misrouted - using wrong encap id - after arp

Gilles.

[mruuth]

Gilles,

I'm running 4.2.4. Could you advice me about which commands I shall do, so I'm prepared if the problem happens again.

Regard

Mats

[mruuth]

Gilles,

Since a couple of weeks we get a lot of these messages "Invalid encaps ID for get info." After upgrade end of July it has been no such messages until now. You can notice they often comes in 5-minutes interval (ARP-refresh real).

We have IOS 12.1(20)E2

[Gilles Dufour]

This is a known issue and a ddts has been opened. However, this is a cosmetic issue.

It should not affect CSM operations.

CSCek31065: IOS/CSM generates CSM9: Invalid encaps ID for get info messages

Gilles.

[mruuth]

Gilles,

What kind of information should I collect when (hopefully not) I stumble on the problem with a failed bridge in CSM?

Your truly

Mats

[Gilles Dufour · Accepted solution]

you will need to access the csm console.

Then enter venus mode.

Then capture

arp_cache_print

vlan_tree_dump_raw

!<--- repeat the command for all the vlans involved - client vlan and server vlan --->

Then from ios, enable the following debug

'debug module ContentSwitchingModule X packet x.x.x.x/32' where x.x.x.x is a client that you control to limit the amount of traffic that will hit the debug.

Then open a connection that fails.

Make sure to capture a csm portchannel trace [portchannel not vlan as the csm might send the packet in the wrong vlan, so the portchannel will tell us where the packet is coming out].

Gilles.

[mruuth]

Gilles,

Thanks for a nice conversation and advices. I will follow your instructions if (hopefully not) the trouble occurs again.

Regards

Mats