VCS 7.2.1 - Head scratcher...

[Darren Goulden]

Hi,

A strange one, I've just deployed a new VCS on 7.2.1 (virtual), I logged onto it via its web interface, navigated to 'Maintenenace>Logon Accounts>Admin Accounts' and changed the admin password, when I clicked 'Save' the request just timed out.  I lost connectivity to the box from the server I was working on, couldn't ssh or web onto it so I rebooted the VCS, tried again and this time the connection timed out before I reached the admin accounts page, I tried to connect to the VCS from another server in the same subnet as the first server, no issues, both ssh and the web interface are working fine, checked my firewall rules, they are fine, routing, that was fine (which is why I could reach it before it timed out).

Performed a wireshark on the firewall and the VCS and I can see the packets getting to the VCS but nothing returns, it's like the VCS is dropping them.  I can ping the server from the VCS and I get a reply though which says to me the VCS is just dropping the packets from my server.

Any suggestions on what I can try?  My problem is the server in question is running my TMS so I kinda need it to talk to the VCS!

Thanks
Darren

[aostense]

Hi Darren,

My first thoughts are:

- Could it be a local browser problem (tried more than one browser from the same server)?

- Are the servers in different VLANs (any restictions/Inter-VLAN routing)?

- Any inspection/security between the servers?

What "Session time out" timers do you have set for https timeouts on the VCS (default=30mins)? I guess you are using all default configurations on the configuration page System > System on the VCS?

Any other users that use the admin account at the same time?

If the TCP connection terminates you will see this behavior in your browser (logged out).

Are you able to add the VCS to TMS? Which problems do you see? Is SNMP enabled?

Hope this helps, but I'm not sure it will, as this is a real head scratcher.. =)

Arne

[Darren Goulden]

Hi Arne,

- Could it be a local browser problem (tried more than one browser from the same server)?

     - Yep, tried both IE and FF (and ssh with putty)

- Are the servers in different VLANs (any restictions/Inter-VLAN routing)?

     - Different segments of the network so different VLANs with a firewall between them (relevent rules added)

- Any inspection/security between the servers?

     - Just an ASA

What "Session time out" timers do you have set for https timeouts on the VCS (default=30mins)? I guess you are using all default configurations on the configuration page System > System on the VCS?

     - All at defaults

Any other users that use the admin account at the same time?

     - No, it doesn't seem to be related to the account as I can't access any interface once it decides it doesn't like the traffic from my TMS server

Are you able to add the VCS to TMS? Which problems do you see? Is SNMP enabled?

     - SNMP is enabled (with TMS support) but adding the system to TMS is a negative 'System not found!'

From the wireshark traces I've taken it doesn't look like the traffic is being blocked, the traffic reaches the VCS, the VCS just doesn't respond, it's wierd...

[awinter2]

Darren,

seeing this is a VM VCS, do you see anything at all in the VCS console in ESXi/vCenter?

The fact that the VCS receives packets but fails to reply to them (Unless the VCS replies with ICMP unreachable), sounds almost like the VCS is missing a default gateway, which is odd since from what you describe, you could previously communicate with the VCS across subnets.

Would you be able to access the VCS web interface and/or SSH from the same subnet as the VCS itself, to see if there is a difference in behavior?

The next step would be to collect a snapshot from the VCS and raise a TAC case, albeit that might be hard to obtain if you can't properly browse or SSH to the VCS to begin with.

[aostense]

How did you assign the network ports in ESX? How are they configured (ip/gw/sub/duplex/etc)?

Could you also do a Wireshark trace on the VCS itself (not the ASA FW), and see if the VCS really isn't sending anything back on the requests?

Cheers,

Arne

[Darren Goulden]

Hi Guys, the VCS is configured with full IP details, and I've run a trace on the firewall and the VCS (as described in my original post) and I'm not seeing any returning packets on either, I will escalate this to a TAC case.

Cheers