May also apply to future appliances that make use of a FusionIO card
After an ungraceful shutdown of the SAP Appliance (Loss of Power/OS system Crash) the system file system check (fsck) fails on /dev/saplogvg/saplog
SUSE will boot into single-user-mode or repair mode
Subsequent reboots of the system will also end up in this state
/dev/saplogvg/saplog will fail to mount
Once in single user mode you will also the following messages on the console screen or issuing var/log/dmesg | tail -100:
[ 418.142765] fioinf Fusion-io ioDrive 320GB 0000:84:00.0: Powercut detected
[ 418.822921] fioinf Fusion-io ioDrive 320GB 0000:84:00.0: recovered data log append point 866:126959616.
[ 418.897230] fioinf Fusion-io ioDrive 320GB 0000:84:00.0: Creating block device fioa: major: 252 minor: 0 sector size: 512...
[ 418.973952] fioa: unknown partition table
[ 419.016524] fioinf Waiting for /dev/fioa to be created
[ 420.705203] fioinf Fusion-io ioDrive 320GB 0000:87:00.0: found end of data log.
[ 420.743656] fioinf Fusion-io ioDrive 320GB 0000:87:00.0: No powercut detected on data log
[ 420.960294] fioinf Fusion-io ioDrive 320GB 0000:87:00.0: recovered data log append point 250:178633728.
[ 421.036381] fioinf Fusion-io ioDrive 320GB 0000:87:00.0: Creating block device fiob: major: 252 minor: 16 sector size: 512...
[ 421.112808] fiob: unknown partition table
[ 421.158446] fioinf Waiting for /dev/fiob to be created
Cause / Problem Description
This is expected behaviour of the FusionIO cards if they are not cleanly shut down the drives may take a long time to come up and will fail to be mounted during boot.
Failure to mount the drives will result in fsck failure which will stop the OS from booting
Udev will wait 180 seconds for the driver to load, then it will exit. In most cases, this is plenty of time, even with multiple ioDrives installed. But if the drives were shut down improperly, loading the driver and attaching the drives takes longer than the 180 seconds. In this case, udev will exit. The driver will not exit, but will continue working on attaching the drives.
There is not always a problem when udev exits early. The drivers will eventually load, and then you will be able to use the attached block devices. But, if the drivers do take too long to load, and udev does exit, and file systems are set to be mounted in the fstab, then the system file system check (fsck) will fail, and the system will stop booting. In most distributions the user will drop into a single-user mode, or repair mode.
Conditions / Environment
SAP HANA M-Sized Appliance (C460)
SUSE Linux (OS)
Can also apply to any UCS server that has a FusionIO card and FusionIO cards are set to mount during boot (i.e /etc/fstab)
This condition occurs when the system is brought down ungracefully and will occur on next boot. This was discovered during a power outage but there may be other scenarios that could result in this behaviour
1. After the System boots into repair mode hash out the /dev/saplogvg/saplog line in /etc/fstab
2. Reboot the system 3. Run Fsck on /dev/saplogvg/saplog to make sure the file system is okay 4. The system at this point should boot normally but /sap/log will not be mounted. Uncomment out the /dev/saplogvg/saplog line in /etc/fstab
5. Reboot the system. Since this will be a graceful reboot from the regulare SUSE linux mode, the driver will go through proper shut down procedure and should come up without any issues
1. Please try and avoid rebuilding the volume group which will blow away sap/log data which will then require the users to preform a full restore of the HANA DB.
2. In the above method, the data should be present on the disk and should be recoverable (The issue is that the FusionIO disks are not coming up quick enough) Should the data/filesystem be corrupt then you could rebuild the volume group once this is complete then use HANA Studio to preform a restore
Hello Guys,My Cisco ACI fabric have 1 APIC server-M2 (Fimware 2.2(2e)), 1 Spine N9K-9336PQ, 2 Leaf N9K-9396PX using fimware 12.2(2e). can i upgrade my apic fimware to 4.2(3i), spine and leaf fimware to 14.2(3i) ?
Got a fabric module that refused to boot up on a new switch. Upon inspecting the log I've come across this. 2020 Mar 26 17:14:17 N9K-C9504 %$ VDC-1 %$ %MODULE-2-MOD_DIAG_FAIL: Module 22 (Serial number: ) reported failure due to fatal error in device ...
Cisco launched their solution for hybrid cloud solution for the Microsoft Azure public cloud back in September of 2017. Since that time, Cisco has enjoyed great success with installations around the world from Poland to Australia and many points in ...
Hi All, I want asking about problem with APIC and how to resolve it.Let's say, I have three APIC Controller in fabric and all of the fabric go down.1. what happens when all apic controllers in fabric go down?2. How to troubleshoot this problem?3. If ...