Shared VLAN FWSM

[John Apricena]

Hello All,

We run a 6500 with an FWSM with about 10 virtual contexts. Within these virtual contexts we have a "shared" vlan. Basically I have added an interface to each context that is in the same network. This way using route statements I can communicate from one vlan (call in management) to the rest of the VLANs. So, I can perform monitoring, syslog, WSUS, etc.

The basic setup was I assigned the 6500 an IP of the VLAN for example I will use vlan 105. I gave the 6500 an IP and added it to the firewall vlan-group. I then added this interface into every FWSM context, and placed a route statement in each context forrwarding the data to the 6500. The 6500 then knew which context to route the data to. This worked with no issue, and I was able to route between FWSMs through the backend, so I would get caught up with asymetrical routing.

The problem occured yesterday when we lost power to our DC, since our generator was restarted. Currently due to Sandy our DC is running on a generator and this was restarted manually by the building. I don't know why this occured, but I'm sure I will find out in the coming days. When the systems came back up all was fine, except for the ability to route between the contexts. No longer am I able to ping the 6500 inter vlan 105 from the FWSMs that all have an IP on the same scope as the 6500 vlan 105. All the FWSMs can ping eachother's shared interface, but can't ping the 6500.

Ex.

FWSM A Int Vlan 105 - 10.10.10.1   Can ping FWSM B but not 6500

FWSM B Int Vlan 105 - 10.10.10.2 - Can ping FWSM A but not 6500

6500 Int Vlan 105 - 10.10.10.254 - Cannot ping FWSM A or B

I confirmed all the system configs are the same from the backups, however it just doens't work now. I'm assuming the problem is somewhere between the 6500 and FWSM. Could something have become faulty on the backend of the 6500? My question is what could I be missing since the configs are all the same that they used to be, yet the traffic doesn't route the way it should be? Thanks for all the help and I can provide any attached diagrams/configs if you'd like.

[Jennifer Halim]

It does indeed sound like a backend issue on the 6500 itself.

You can further troubleshoot it with the TAC or try to reload it and see if it solves the issue.

[John Apricena]

Hey Jennifer,

Thanks for the advice, this is what I'll do.

[John Apricena]

I have further troubleshooted this issue and here is what I found. As i've said we had a complete power reset in our datacenter, and when the devices came up the VLAN we are "sharing" accross multiple contexts can no longer communicaate between the 6500 and FWSM.

We run multiple contexts and from the system and admin conexts I can ping the 6500 SVI on the shared VLAN, however on all the other FWSM's I cannot ping the 6500's SVI. Can anyone advise as to why this may be? I am allowing all traffic through the FWSMS and have the intervlan vlan on all the FWSMs with a different IP on the same network. Thanks in advance!

[Jennifer Halim · Accepted solution]

sounds like it got stuck within the hardware. As 6500 switch is all hardware based, best would be to open a TAC case so they can perform hardware level troubleshooting.

At this point, it doesn't sound like it is a configuration or software issue.

[John Apricena]

Hey Jennifer,

Thank you so much for your help with this. I have one more question though. Currently, we have two FWSM blades in our 6500, however we only use one of them. We are just waiting on the license for failover. Our primary FWSM cannot ping the 6500 "shared" interface that I am using across all the FWSMs, however the secondary one can. I ran a show module and all passed successfully, My question is with the recent power surge, could there be a chance that the 6500 doesn't know which FWSM to route data to and possibly does the 6500 think that our bypass FWSM is the current. Again there is no failover configured, but there are two FWSM blades in the 6500. I only ask this because our primary FWSM CANNOT ping the 6500 shared interface but our secondary CAN. Thanks again for your help!

[Jennifer Halim]

There is a possibility if they both have exactly the same configuration, however, are not in failover pair. Both might assume they are the only FWSM in the network if both have the exact same configuration but not in failover pair.

If that is the case, I would strongly recommend shutting down one of the FWSM, and since the secondary one is the one working, and i am assuming passing traffic as well, I would shut the primary until they are put in failover pair.

[Anthony Chillino]

Thanks again Jennifer! We shut down one of the FWSMs, and opened up a TAC with Cisco. Currently they said te FWSM is functioning fine and the issue is somewhere on the 6500.