08-21-2008 03:00 PM
Hello all,
Hopefully someone can help me out as I dont know linux at all to verify some information.
I have MARS configured to backup to a NFS server. I recently received a notification from MARS stating that one of the backups failed.
Because I do not have access to the NFS server (Managed Services is why), what other ways can I verify that the NFS backup is succeeding? Whenever I attempt to use the GUI to pull raw messages from the NFS server it essentially times out.
Anything on the cli available? Any and all help/ideas are appreciated.
MARS is running 4.2.6 ( 2458 ). Yea, old, but customer absolutely refuses to give us the time of day to upgrade it.
08-22-2008 09:38 PM
You can setup a second NFS server temporarity to see if that works. You can even run NFS on windows with the help of some software/add-on (as per MARS docs).
Alternatively you could go to MARS >> Admin >> System Maintenance >> Set Runtime Logging Levels
pnarchiver = DEBUG
And see if you get any more meaningful logs.
Regards
Farrukh
08-25-2008 06:24 AM
FYI,
It isn't until a later release that Cisco actually includes
A) more meaningful logs for NFS
B) Event corrolation for NFS failures for the MARS appliance so it will notify more then just the account associated with "pnadmin"
08-25-2008 06:29 AM
"MARS is running 4.2.6 ( 2458 ). Yea, old, but customer absolutely refuses to give us the time of day to upgrade it."
well, they still get to check the box when the auditor visits;-) sad isn't it. The NFS logging enhancements is one of MANY reasons to make sure MARS is kept up to date. FWIW, we implemented our own NFS monitoring years ago because it would frequently break with no alerts being generated.
08-25-2008 07:03 AM
Mind if I ask how you monitor the NFS?
08-25-2008 07:07 AM
Each day has it's own directory in the archive.
We just check the modify time on the directory. The NFS archiving has gotten MUCH better, but back in the day this worked pretty good to let us know when it was broken.
08-25-2008 07:24 AM
Ah. Makes sense, and was my first instinct. Unfortunately, we're a managed service provider. For liability issues, we do not have access to view the server.
08-25-2008 04:39 AM
Also there is a 'pnexp' command available on the CLI. Even tough it is meant to export 4.x database before importing it into a new 5.x box (Generation 1). It can be used for 'immediate backups' as per the Cisco engineer on this thread:
Once you enter the pnexp CLI
Do the following:
export config 10.1.1.1:/archive.
Then enter the following command:
pnexp> status
Data exporting process is currently running, please use command 'log {all|recent}' to view running logs and/or progress.
pnexp> log
Aug 25 09:44:47.192 2008@LM_INFO@Thread 1024:Number of events exported: 1227522, ~0.18% completed, overall speed = 30688.05 eps
Aug 25 09:44:47.192 2008@LM_INFO@Thread 1024:Estimated-Time-To-Complete: 6 hours 18 minutes
CTRL-C
As you can see I'm doing a full backup here (config+event/report data etc), don't run that just to 'test' NFS. Its better to use the 'config' option :)
Regards
Farrukh
08-25-2008 06:12 AM
You can use tcpdump from the command line. Try this:
tcpdump -X host
I'm not sure how intelligent it is, but some level of NFS decoding is performed by tcpdump and it may give you enough of a hint to get you going in the right direction.
08-25-2008 07:17 AM
Thats actually a really good idea. Of course, I keep thinking about how NFS is UDP so that a packet capture wouldn't be all that great. But If i do the dump at the same time requesting a file, i can confirm that data is indeed on the server and they're talking.
Thanks for the input!
08-25-2008 07:22 AM
Actually, it's TCP...at least the version I'm on now. I do think it used to be UDP though. Another reason to patch;-) Even with UDP, there will be application level acknowledgments, so tcpdump should work fine.
08-26-2008 01:14 PM
So, interesting enough, here is that tcpdump for communication to my NFS server. As you can see, it'd not really seeing a response from the NFS server. But the amusing part is, this is actually from when the NFS - Get Archived Files worked.
[pnadmin]$ tcpdump -i eth0 ip host 10.17.1.252
tcpdump: listening on eth0
16:08:36.407598 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)
16:08:37.400025 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)
16:08:39.400024 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)
16:08:43.400466 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)
16:08:51.400025 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)
16:09:07.400358 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)
16:09:39.400034 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)
16:10:39.400033 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)
16:11:39.400041 pnmars.1059879952 > 10.17.1.252.nfs: 108 getattr [|nfs] (DF)
16:13:07.541864 10.17.1.252.netbios-dgm > 10.17.1.255.netbios-dgm: NBT UDP PACKET(138)
08-26-2008 01:56 PM
interesting. pulling files should definitely result in a whole lot more activity than that. You don't really know what is happening "behind the scenes". MARS may not actually fetch data it has local for example (i.e. previously requested data, or data in the DB). The getattr should result in a response too.
08-26-2008 02:03 PM
I've actually expanded the tcpdump capture to get everything and then looked through it. For example, when I am doing that it isn't capturing my browsing the webgui. So how broken is tcpdump?
08-26-2008 02:05 PM
I'm not aware of it ever being broken in MARS, but I wouldn't have paid that much attention if it was. The MARS box isn't using both interfaces is it?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide