media server is oversubscribed

[gtoscano1]

I have a media server reported that its being oversubscribed, utilizing 26tb of the 28tb it has. When I check the vsmc, i see its holding shelved video from cameras that I moved to other servers previously, which I understand as it should hold that video for the normal retention period. 

However, I also see a bunch of camera that are holding more video than they should. For example, a handful of 6030 cameras are set on a template for 14 days, but they are storing 30 days worth of video. Im not entirely sure why this is happening, because its not happening for all camera using this template. 

In any case, I need to free up space because the server storage should be utilized this much, especially after I moved cameras to other media servers. I would expect each day to get more space back on this server but its not happening. 

Any ideas how to resolved this, or manually delete shelved files from the server directly? I can make my way around the cli but Im not well versed really. 

thank you

[Scott Olsen]

I hear you...

I have to start off by saying that the only real supported way to accomplish this is to engage Cisco TAC.  If you want to resolve this yourself, I'll still attempt to assist you, but in no way do I take any responsibility for anything that could potentially go wrong.

Cisco has this really strange and totally counterintuitive thing regarding Camera Templates and Retention Days... so the first question is;

1)  These 6030 cameras that are *currently* on a template for 14 days, but are storing 30 days of video, were they previously assigned to a *different* template, or was the template retention days *changed*?

The reason I ask is because the 'Expiration Dates' for each 5 minute video file are only updated *moving forward*.  You cannot modify expiration dates for any archive video that's already been recorded.

There are a couple ways I've found to work around this... but it really depends on your particular situation.

[gtoscano1]

Scott, the template was originally for 30 days for all cameras models (4 cameras models). When I changed the each template it looked like they updated properly and groomed, but a bunch of cameras did not. It addition I moved cameras to record to other servers going forward, but their shelved video past the 14 days still exist on the server. 

I would appreciate any help, the smart net elapsed last month so I can't engage Cisco tac unfortunately. Thanks

[Scott Olsen]

Well, it looks like Raghav is trying to help you out.

If you want to personally know more, and you are comfortable crafting SELECT statements in mysql, I could provide you some information on what I've discovered as far as how actual archives are handled based on data in the SMD_FILE table on each "UMS" platform.

[gtoscano1]

scott and raghav,

Thank you both for the assistance. Given the current status of the project I'm working on, I have decided to just allow the original 30 days to elapse to get the storage back to normal.

When I have some time, I plan to investigate this a little further using the commands raghav provided so I can understand the system a little better.

Scott, I would very much appreciate you sharing the information you have on digging through MySQL. I have seen what you mention done before, I just haven't done it myself. I would love to know what you are doing, but I'm going to test in a lab environment to get a better understanding. Please post here or message me, thank you so much both!

[Raghav-Rao]

Hi,

thanks for the info, let me help you further on this issue.

Cisco doesn't recommend to more the smd files or delete them manually which could create unexpected behavior with VSM server.

VSM run the groomer process every 10 mins to groom the expired smd files which are part of DB. i'm guessing smd files wouldn't be part of DB and left on the media partition without been groomed off after expiration. we may need to push this data back to DB so Groomer can remove them safely. something went wrong we don't know what, with very little info, we might have to take look at the support report to understand the cause. but for now,

we need more info on the smd files on the media partitions. run the below commands

1. vscheck

2. df -f

3. find /media? -mtime +14 (this command will list all the files older than 14 days). find the oldest files from this list and share it.

4. how many camera are recording more than 14 days?

5. a. export LD_LIBRARY_PATH=/usr/BWhttpd/lib

    b. /usr/BWhttpd/bin/reposdbtestclient dumpstoragemonshmem

[gtoscano1]

raghav, see attached file with output for 1, 2, and 3. The answer to 4 is many. I checked my other media server and I see the issue exists there too where the video should be groomed for a handful of cameras after 14 days but its not. 

I did not performed step 5, do I just type the commands exactly as you say there?

I just tried to create a new template with 14 days recording and applied it to a camera. I was hoping it would automatically groom down to 14 days from 30, but it did not, and this is what Scott mentioned would happen (which confuses the hell out of me lol). See that attached image with the information from the server vsmc. I cut and past the 3 lines of information so you could read across it. You can clearly see it is set to a 14 day loop, but the first frame is 7/5 and the last frame is 8/4 (30 days). 

thank you for the help guys, I really need to fix this. 

[Raghav-Rao]

thank you for the output and snap shot. looks like few camera have data from july 5.

please run the below command and share us the output.

1. sudo /usr/BWhttpd/mysql/bin/mysql --defaults-file=/usr/BWhttpd/mysql/ums/ums.cnf UMS -e "select * from recording"

2. sudo /usr/BWhttpd/mysql/bin/mysql --defaults-file=/usr/BWhttpd/mysql/ums/ums.cnf UMS -e "select recording_id, date_dir, from_unixtime(expire_time) from smd_file where recording_id=10008;"

[gtoscano1]

raghav, 

See attached file. I only got output running the first command, none from the second. What are these commands trying to accomplish?

thank you for your help

[Raghav-Rao]

sorry, i can't find the attachment. first command indicates the recording data available for media server, second command indicates the recording data for specific camera recording. with second command output none, DB entries could be missing for this SMD files, below are command to import the entries back to DB so, groomer could remove them off safely.

run the below commands,

1. export LD_LIBRARY_PATH=/usr/BWhttpd/lib (NO output will seen for this command)

2. /usr/BWhttpd/bin/recoversmd -start 1467730908 -stop 1470452822 (wait till the command is returned, don't interrupt or exit the session)

after the import, please wait for the groomer to remove files, this process may take some time. let us know, how it goes.