cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
418
Views
0
Helpful
1
Replies

snapshot recent data wiped & restoret to old data

jalmishal
Level 1
Level 1

Hi,

I would appreciate quick response as this is in production environment.

currently we’ve 3X UCS 5108 chases configured in HA with full blades mix (B200 M2, B230 M1) using two 6296 UCS Fabric interconnect connected to two Cisco MDS 9148 SAN Switch with one EMC VNX 5300 storage . and all are boot from SAN

Yesterday our technical guy do some laying cables and seems he has touch the FC cable (in between UCS ~ Fabric Interconnect) without notice that the cable been disconnected, and after nearly an hour found the issue and retrieve the cable again, during the cable interruption, the ESXi do migrate the blade server 7 to blade server 8, through vMotion and after return the cable did not return them back automatically, but we returned them manually,

but the shock was that we noticed that all the recent data is wiped out in snapshot and data has been restored automatically to old data dated 27th Feb!

This is seriously causing a lot of issues.

as show in attached picture taken from EMC VNX got warning "The following hosts have initiators that no longer have an active connection to the storage system:( ,,,,,,,,,)."


Kindly help

1 Reply 1

Kirk J
Cisco Employee
Cisco Employee

Greetings.

You first step will probably need to be to engage VMware/OS vendors to confirm file system level issues are cleared up.

When you reboot the blades (with the initiators in question) does your storage see the initiators again?

You can check your local initiator Flogi status by going to nxos on each fi:

#connect nxos

nxos#show npv flogi

Do you see the WWPNs for the initiators in question at the FI level?

Do you have any error messages about the vsan in question ?

If you haven't rebooted the blades yet, you may want to try that.

I have seen a number of cases where ESXi stops trying to re-stablish a Plogi login to storage after a while (following a layer 1 issue that disrupted FC connectivity).

Do we know that your MDS FC ports are all up?

Does the MDS see the initiator WWPNs (#show flogi )

 

There are a lot of layers of stuff to check that is more than can easily be discussed on a forum post.

I would normally refer you to open a TAC case, but your Blades referenced are  End Of Support.

Running production on equipment you can't open a case for is not an enviable position to be in :(

 

https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-b-series-blade-servers/115764-ucs-san-tshoot-00.html

 

Thanks,

Kirk...

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: