cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
8395
Views
0
Helpful
23
Replies

Error running VSOM in MSP RHEL Server

Ritchie Nasayao
Level 1
Level 1

Hi,

I have encountered problem in loading VSOM from MSP RHEL Server. Login page does not show value. Error loading /media1 during bootup process of RHEL. It says "mounting local filesystem error" and "mount: structure needs cleaning". After checking service cisco status, VSOM is down and at the end "/media1 is not mounted".

Anyone has idea or has encountered this kind of problem, please advise. Thank you.

2 Accepted Solutions

Accepted Solutions

I'm not sure why you are deleting any files anywhere, Ritchie.  The mrtg.cfg file is completely unrelated to the issue.

If you've run the xfs_repair binary with the -n flag, it's only going to report on the issues it locates, and not actually modify or repair them.  I was hoping you'd share the output with the forum before proceeding, but ultimately that's up to you.

Shanky's post also mentions the xfs_repair command, but it *will* modify the filesystem (and likely correct your problem).

Scott Olsen Solutions Specialist Bulletproof Solutions Inc. Web: www.bulletproofsi.com

View solution in original post

Hi Ritchie,

I think the option we are trying to do repair is not  helping us here,

Please re-run the below command to repair, also follow the above the above the above steps get the partition up.

++ Run xfs_repair for /media1

  #  xfs_repair -L /dev/sdc1 -l /dev/sdc2

-L represent Force Log Zeroing, let me know if any query.

Thanks

Shanky

View solution in original post

23 Replies 23

Scott Olsen
Level 6
Level 6

Yeah, it sounds as though you have some serious file system / storage array problems.  What is the health of the RAID array =(Can check via WebBIOS on boot)?

If the array is healthy or 'optimal', chances are things have become corrupted at the filesystem level.  You'll need to find a way to check/repair your filesystems.  

Best of luck.

Scott Olsen Solutions Specialist Bulletproof Solutions Inc. Web: www.bulletproofsi.com

Hi Scott,

Thank you for the response.

RAID array is all good. RHEL is booting up and running up to its GUI. It is the application that is not running well. I can still open the VSM page through the browser but only up to the login page. It says error in html object "The page is not available. Please check URL". Its Domain pull-down option only says "Loading Value". I checked the /media1 content and I cant find the other two files I got before. It only contains the getstoragestatus file. The other two files that I got before is missing. It also says that /media1 is not mounted. When I try to mount it, it says structure needs cleaning.

What is the exact version of VSM you are running?   7.2.0?  Also, are you using a supported browser to access the page?

Final question; Have you or someone else been manually modifying the media repositories? Adding a new repo, resizing them, etc....

Do you have video data on the repo you need to retain, or are you just attempting to get it functional.  The kernel is informing you that something is wrong with the XFS filesystem on the repo... which is likely pretty bad news.

Scott Olsen Solutions Specialist Bulletproof Solutions Inc. Web: www.bulletproofsi.com

That is correct, 7.2.0. I am using the supported browser as per documentation.

The system is already running. The power just went down and the server was not shutdown properly. No modification was done in the server.

I may just set aside the video recording and consider having it deleted maybe if I cannot restore it. I need to determine what is that "structure needs cleaning". There is only one error I can see during boot up and it is that "/media1 is not mounted".

It may be best to raise it to TAC if I cannot find any other answer to my issue.

Hi Ritchie, 

 There is something wrong with  media partitions, so we mainly need to run xfs(file system) repair so as to get this up. 

Please provide me the below output through vsom server console, login as "localadmin".

Output of  proc/partitions

#cat /proc/partitions

++ Output of  /etc/fstab :

#cat /etc/fstab

++ output of diskfree

#df -hk

++ Output for mount

# mount -a

Let me know the output to find my next plan.

Thanks 

Shanky

Hi Shanky,

please see below output:

>>>>

[root@VSOM10 ~]# cat /proc/partitions
major minor #blocks name

8 0 153600 sda
8 1 152586 sda1
8 16 127926272 sdb
8 17 7999983 sdb1
8 18 8000000 sdb2
8 19 32000000 sdb3
8 20 8000000 sdb4
8 21 8000000 sdb5
8 22 8000000 sdb6
8 23 55926255 sdb7
8 32 7680509952 sdc
8 33 7680363264 sdc1
8 34 146654 sdc2
[root@VSOM10 ~]#
[root@VSOM10 ~]#
[root@VSOM10 ~]#
[root@VSOM10 ~]# cat /etc/fstab
LABEL=/ / ext3 defaults 1 1
LABEL=/usr/BWhttpd /usr/BWhttpd ext3 defaults 1 2
LABEL=/var /var ext3 defaults 1 2
LABEL=/mysql/data /mysql/data ext3 defaults 1 2
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sdc4 swap swap defaults 0 0
LABEL=media1 /media1 xfs rw,nodiratime,noatime,logbufs=2,logdev=/dev/sdc2 0 0
[root@VSOM10 ~]#
[root@VSOM10 ~]#
[root@VSOM10 ~]#
[root@VSOM10 ~]# df -hk
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 7749180 2766332 4582852 38% /
/dev/sdb5 7749200 2964904 4384296 41% /usr/BWhttpd
/dev/sdb3 30997316 1826588 27570728 7% /var
/dev/sdb7 54173584 1405488 49971784 3% /mysql/data
/dev/sda1 147764 11458 128677 9% /boot
tmpfs 2069352 4 2069348 1% /dev/shm
[root@VSOM10 ~]#
[root@VSOM10 ~]#
[root@VSOM10 ~]# mount -a
mount: Structure needs cleaning
[root@VSOM10 ~]#

>>>>

Please let me know. Thank you.

Interesting... I've never seen those mount options on an XFS media repo for VSM before:

,logbufs=2,logdev=/dev/sdc2 0 0

Looks like this is attempting to put the journal log for the FS on a different physical device than the XFS block device.

Did you add these, or were they already there? 

You can try running a consistency check on the volume as indicated by the error, but I'm not sure if the binaries for this are in the RHEL image to do so?  I don't see 'xfs_repair' on the lab box I have access to... but it's not a RHEL server.

If you have data on this repo that you need to retain, I strongly recommend a TAC case.

Scott Olsen Solutions Specialist Bulletproof Solutions Inc. Web: www.bulletproofsi.com

Nevermind.  Looks like I was still localadmin.  If you su to root with 'sudo su -', they'll be in your path and you'll have the ability to run them.

*CAUTION* - If you have data on this repo you can't lose, stop listening to me now and refer to TAC.  If you still want to continue and try to resolve this yourself, check out the following command below;

(as root)

# xfs_repair -n -l /dev/sdc2 /dev/sdc1

... and share the output (or act on it...up to you).

The -n instructs 'no modify', which means it will simply check the status of the XFS filesystem and report what it finds.

Cheers!

Scott Olsen Solutions Specialist Bulletproof Solutions Inc. Web: www.bulletproofsi.com

Hi Scott,

Those items in the repo are already there from the first time I checked after the error. I am still trying to look on options of restoring the existing setup and configuration so as not to re-run installation again. The first time I tried to run the xfs_repair option, it already indicated that mount: Structure needs cleaning. I have also tried deleting the other mrtg.cfg file but nothing has changed nor improved.

Hi Ritchie,

Thanks for your response,

Please follow the below steps to recover the media partition '/media1'.

++ Stop the cisco services

  services cisco stop

++ su to root with 'sudo su -'

++ Run xfs_repair for /media1

  #  xfs_repair -L /dev/sdc1 -l /dev/sdc2

  #  cd /media1

  #  ls -lrt (check if you are able to run)

  # Mount the FS 'mount -a'

  # check 'df -hk'

++ Restart the cisco services

  service cisco start

I think the above steps would be helpful for you, please let me know if you have any concern.(Do rate if you find this helpful)

Thanks

Shanky

Hi Shanky,

Please see below. I have just cut the sequence though:

[root@VSOM10 etc]# xfs_repair -n -l /dev/sdc2 /dev/sdc1
Phase 1 - find and verify superblock...
Phase 2 - using external log on /dev/sdc2
- scan filesystem freespace and inode maps...
block (22,669089) already used, state 2
....

....

block (22,669103) already used, state 2

block (22,689073) already used, state 2

....

....

block (22,689087) already used, state 2

block (23,962241) already used, state 2

...

...

block (23,962255) already used, state 2
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
...

...

...

- agno = 20
imap claims in-use inode 671088923 is free, would correct imap
- agno = 21
- agno = 22
imap claims a free inode 738197545 is in use, would correct imap and clear inode
imap claims in-use inode 738197546 is free, would correct imap
- agno = 23
7dd4cb90: Badness in key lookup (length)
bp=(bno 2760508544, len 131072 bytes) key=(bno 2760508544, len 8192 bytes)
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28

...

...

...

- agno = 125
- agno = 126
- agno = 127
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 10
- agno = 7
- agno = 11
- agno = 8
- agno = 9
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
entry "5618b547.tmp" at block 0 offset 1720 in directory inode 738887674 references free inode 738197545
would clear inode number in entry at offset 1720...
- agno = 32
- agno = 40
- agno = 41
- agno = 42
- agno = 43
- agno = 44
- agno = 45
- agno = 46
- agno = 33
- agno = 31
- agno = 47
- agno = 48
- agno = 49
- agno = 50
- agno = 51
- agno = 52
- agno = 53
- agno = 54
- agno = 55
- agno = 56
- agno = 57
- agno = 58
- agno = 59
- agno = 60
- agno = 62
- agno = 63
- agno = 61
- agno = 64
- agno = 65
- agno = 66
- agno = 67
- agno = 68
- agno = 69
- agno = 70
- agno = 71
- agno = 72
- agno = 73
- agno = 74
- agno = 75
- agno = 76
- agno = 77
- agno = 78
- agno = 79
- agno = 80
- agno = 81
- agno = 82
- agno = 83
- agno = 84
- agno = 85
- agno = 86
- agno = 87
- agno = 88
- agno = 89
- agno = 90
- agno = 91
- agno = 92
- agno = 93
- agno = 94
- agno = 95
- agno = 96
- agno = 97
- agno = 98
- agno = 99
- agno = 100
- agno = 101
- agno = 102
- agno = 103
- agno = 104
- agno = 105
- agno = 106
- agno = 107
- agno = 108
- agno = 110
- agno = 111
- agno = 109
- agno = 112
- agno = 113
- agno = 114
- agno = 115
- agno = 116
- agno = 117
- agno = 118
- agno = 119
- agno = 120
- agno = 122
- agno = 123
- agno = 121
- agno = 124
- agno = 125
- agno = 126
- agno = 127
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem ...
entry "5618b547.tmp" in directory inode 738887674 points to free inode 738197545, would junk entry
bad hash table for directory inode 738887674 (no data entry): would rebuild
- traversal finished ...
- moving disconnected inodes to lost+found ...
disconnected inode 671088923, would move to lost+found
disconnected inode 738197546, would move to lost+found
disconnected inode 771845995, would move to lost+found
Phase 7 - verify link counts...
would have reset inode 771845995 nlinks from 0 to 1
No modify flag set, skipping filesystem flush and exiting.
[root@VSOM10 etc]#

Well, it definitely found issues in the XFS filesystem. Fortunately, it looks like it can repair them if you rerun the command without the -n flag.  After that you could likely mount the filesystem as the media repository as intended.

sudo su -

/etc/init.d/cisco stop

xfs_repair -l /dev/sdc2 /dev/sdc1

mount -a 

if this returns without error, check to see it's actually mounted with

df -h

You should see the /mediaX repository listed there.  Then restart the Cisco services:

/etc/init.d/cisco start

Cheers!

Scott Olsen Solutions Specialist Bulletproof Solutions Inc. Web: www.bulletproofsi.com

Hi Scott/Shanky,

I run the xfs_repair and I was able to mount /media1 xfs.

>>>>

[root@VSOM10 etc]# xfs_repair -L /dev/sdc1 -l /dev/sdc2
Phase 1 - find and verify superblock...
Phase 2 - using external log on /dev/sdc2
- zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
- scan filesystem freespace and inode maps...
block (22,669089) already used, state 2

...

...

...

block (22,669103) already used, state 2
block (22,689073) already used, state 2

...

...

block (22,689087) already used, state 2
block (23,962241) already used, state 2

...

...

block (23,962255) already used, state 2
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1

...

...

- agno = 20
correcting imap
- agno = 21
- agno = 22
imap claims a free inode 738197545 is in use, correcting imap and clearing inode
correcting imap
- agno = 23
7e6e8b90: Badness in key lookup (length)
bp=(bno 2760508544, len 131072 bytes) key=(bno 2760508544, len 8192 bytes)
- agno = 24
- agno = 25

...

...

- agno = 127
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1

...

...

- agno = 39
entry "5618b547.tmp" at block 0 offset 1720 in directory inode 738887674 references free inode 738197545
clearing inode number in entry at offset 1720...
- agno = 40

...

...

- agno = 127
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
bad hash table for directory inode 738887674 (no data entry): rebuilding
rebuilding directory inode 738887674
- traversal finished ...
- moving disconnected inodes to lost+found ...
disconnected inode 671088923, moving to lost+found
disconnected inode 738197546, moving to lost+found
disconnected inode 771845995, moving to lost+found
Phase 7 - verify and correct link counts...
done
[root@VSOM10 etc]#

>>>>

A new problem came up with mySQL database server.. I have attached the screenshot.

Hi Ritchie,

Good to know know you were able to repair the FS, Let me know the output for below command.

# df -hk

# service cisco status

# service cisco restart

Thanks

Shanky

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: