Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Options
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
71
Views
0
Helpful
0
Comments
Options
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
on 01-24-2014 03:49 PM
This document was generated from CDN thread
Created by: James Maudlin on 20-05-2011 10:25:02 AM
<h1>
Overview: WebEx Social and NFS</h1>
WebEx Social uses NFS for a few main purposes, including, but not limited to:
<ul>
<li>
Storage of uploaded documents (both Doc Libraries and attachments to Posts)</li>
<li>
Sync of logs from each WebEx Social role to a diagnostics directory on the NFS server every 30 minutes for purposes of troubleshooting</li>
<li>
Storage of ActiveMQ data</li>
</ul>
In order to address these needs, each WebEx Social node (this is *all* nodes,
not just those with the WebEx Social role) will create the following NFS mounts:
<ul>
<li>
/WebEx Socialdata/cms: Used for Document Library and Post Attachment document storage</li>
<li>
/WebEx Socialdata/jms: Used for storing the ActiveMQ (JMS) database and other files required for proper JMS node failover</li>
<li>
/WebEx Socialdata/diagnostics: Used for copying log files to the NFS server for easier retrieval in the event that troubleshooting is needed</li>
<li>
/WebEx Socialdata/backup: Used for RDF/Social Graph node backup</li>
</ul>
On the WebEx Social nodes (again, this is all nodes, not just
those with the WebEx Social role), these will be mounted as the /WebEx Socialdata
volume. If NFS is not setup properly - or even if no NFS server is
provided - the nodes will build /WebEx Socialdata as a local volume on that VM.
This can present a few different issues, including, but not limited to:
<ul>
<li>
For WebEx Social nodes: If enough documents are uploaded, it can fill up the /WebEx Socialdata node and cause the WebEx Social node performance to slow to a crawl</li>
<li>
For ActiveMQ (JMS) nodes: If you have multiple JMS
nodes, they will both attempt to be active at the same time and will not
be able to share data, as that data is shared via a common NFS
directory</li>
<li>
For all nodes: Logs can potentially consume all of the
space of the volume on which they live, plus the logs will not be
centralized on the NFS server for easier gathering</li>
</ul>
<h1>
Verifying NFS is currently mounted</h1>
The first thing to do if you suspect there is an NFS-related issue is to verify if the various NFS mounts are setup properly.
The easiest way to do this is to ssh into the node as root and run ¿cat
/etc/fstab¿. Here¿s an example of what that would look like (the NFS
server mounts are highlighted in red):
[root@WebEx Social-1 ~]# cat /etc/fstab
/dev/vg_os/lv_root / ext3 defaults 1 1
/dev/vg_os/lv_var /var ext3 defaults 1 2
/dev/vg_os/lv_opt /opt ext3 defaults 1 2
/dev/vg_os/lv_recovery /recovery ext3 defaults 1 2
/dev/vg_os/lv_WebEx Socialdata /WebEx Socialdata ext3 defaults 1 2
/dev/vg_os/lv_localdisk /localdisk ext3 defaults 1 2
/dev/vg_os/lv_tmp /tmp ext3 defaults 1 2
/dev/vg_os/lv_altroot /altroot ext3 defaults 1 2
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/vg_os/lv_swap swap swap defaults 0 0
nfs.cisco.com:/export/WebEx Socialexport/diagnostics /WebEx Socialdata/diagnostics nfs defaults 0 0
nfs.cisco.com:/export/WebEx Socialexport/jms /WebEx Socialdata/jms nfs defaults 0 0
nfs.cisco.com:/export/WebEx Socialexport/cms /WebEx Socialdata/cms nfs defaults 0 0
nfs.cisco.com:/export/WebEx Socialexport/backup /WebEx Socialdata/backup nfs defaults 0 0
none /opt/ramfs/logs dbcfs defaults,uid=0,gid=0,mode=0770,size=128M,wproc=java,dest=/opt/cisco/WebEx Social/tomcat/logs 0 0
If those are missing, then the node is definitely not using the NFS
server and it will be storing the docs locally in the /WebEx Socialdata
directory. Refer to the linked pages below for details on how to repair
the issue depending on the type of node that has the issue (e.g.: WebEx Social
role vs JMS role vs Social Graph role).
<h1>
Verifying write access on the NFS mounts</h1>
Assuming you see the mounts listed properly in fstab via the steps
above, then the next thing to verify is that you have write permissions
to the NFS shares. To do this:
<ol>
<li>
ssh to the node in question and log in as root</li>
<li>
Type "cd /WebEx Socialdata/diagnostics/logs"</li>
<li>
Type "ls -la" and you should see multiple directories - one for each
node type (Message-Queue, Monitor, etc) and one named "common"</li>
<li>
Type "touch zz", which will create a 0 byte file named "zz" in the /WebEx Socialdata/diagnostics/logs directory</li>
<li>
Type "ls -la" again to verify that you see the "zz" file. If you do
not, or if you received a permission denied message when you tried to
create the file, then you need to look at the permissions of the
exported directories on the NFS server. Either the exports config is
incorrect or the directories do not have the proper permissions to allow
you to write files. Refer to the NFS Permissions link below for steps
on how to fix this.</li>
<li>
Once you have confirmed that the "zz" file exists, then you can delete
it by typing "rm zz" and answering yes to the "rm: remove regular empty
file `zz'?" question that appears</li>
<li>
At this point, you have confirmed that you can read and write to the NFS mount</li>
</ol>
Subject: RE: Troubleshoot NFS
Replied by: Alan Sears on 06-09-2011 07:37:43 AM
James,
In this post you refer to links to another document to repair NFS connectivity for different node types. Can you provide those? I believe I'm running into this issue.
Thanks!
Subject: RE: Troubleshoot NFS
Replied by: Emanuel M Bola on 06-09-2011 10:43:27 AM
Hi James,
Which version of WebEx Social are you using at the moment? There are some changes to the NFS configuration between the Lakota & Chinook releases.
Thanks,
Manny
Subject: RE: Troubleshoot NFS
Replied by: Alan Sears on 06-09-2011 10:46:04 AM
Emanual,
I'm currently running 2.1.2.20000.346.
Thanks!
Alan
Subject: RE: Troubleshoot NFS
Replied by: Emanuel M Bola on 06-09-2011 11:15:34 AM
Hi Alan,
First off, check/correct the status of the rpcidmapd service.
service rpcidmapd status ?
&
chkconfig --list rpcidmapd ?
If the service is stopped, and the startup sequence is off for all run levels, then do the following.
1) service rpcidmapd start
2) chkconfig --add rpcidmapd
Then, if you need to remount the nfs mounts run the following script.
/opt/software/scripts/WebEx Socialos.sh mountnfs4 <nfsip or dns name>:/<nfs mount name> /WebEx Socialdata
mount -l or df -kh to verify the mounts.
Thanks,
Manny
Subject: RE: Troubleshoot NFS
Replied by: Alan Sears on 06-09-2011 02:26:45 PM
That did it...thanks!
Subject: RE: Troubleshoot NFS
Replied by: Alan Sears on 06-09-2011 02:26:45 PM
That did it...thanks!
Created by: James Maudlin on 20-05-2011 10:25:02 AM
<h1>
Overview: WebEx Social and NFS</h1>
WebEx Social uses NFS for a few main purposes, including, but not limited to:
<ul>
<li>
Storage of uploaded documents (both Doc Libraries and attachments to Posts)</li>
<li>
Sync of logs from each WebEx Social role to a diagnostics directory on the NFS server every 30 minutes for purposes of troubleshooting</li>
<li>
Storage of ActiveMQ data</li>
</ul>
In order to address these needs, each WebEx Social node (this is *all* nodes,
not just those with the WebEx Social role) will create the following NFS mounts:
<ul>
<li>
/WebEx Socialdata/cms: Used for Document Library and Post Attachment document storage</li>
<li>
/WebEx Socialdata/jms: Used for storing the ActiveMQ (JMS) database and other files required for proper JMS node failover</li>
<li>
/WebEx Socialdata/diagnostics: Used for copying log files to the NFS server for easier retrieval in the event that troubleshooting is needed</li>
<li>
/WebEx Socialdata/backup: Used for RDF/Social Graph node backup</li>
</ul>
On the WebEx Social nodes (again, this is all nodes, not just
those with the WebEx Social role), these will be mounted as the /WebEx Socialdata
volume. If NFS is not setup properly - or even if no NFS server is
provided - the nodes will build /WebEx Socialdata as a local volume on that VM.
This can present a few different issues, including, but not limited to:
<ul>
<li>
For WebEx Social nodes: If enough documents are uploaded, it can fill up the /WebEx Socialdata node and cause the WebEx Social node performance to slow to a crawl</li>
<li>
For ActiveMQ (JMS) nodes: If you have multiple JMS
nodes, they will both attempt to be active at the same time and will not
be able to share data, as that data is shared via a common NFS
directory</li>
<li>
For all nodes: Logs can potentially consume all of the
space of the volume on which they live, plus the logs will not be
centralized on the NFS server for easier gathering</li>
</ul>
<h1>
Verifying NFS is currently mounted</h1>
The first thing to do if you suspect there is an NFS-related issue is to verify if the various NFS mounts are setup properly.
The easiest way to do this is to ssh into the node as root and run ¿cat
/etc/fstab¿. Here¿s an example of what that would look like (the NFS
server mounts are highlighted in red):
[root@WebEx Social-1 ~]# cat /etc/fstab
/dev/vg_os/lv_root / ext3 defaults 1 1
/dev/vg_os/lv_var /var ext3 defaults 1 2
/dev/vg_os/lv_opt /opt ext3 defaults 1 2
/dev/vg_os/lv_recovery /recovery ext3 defaults 1 2
/dev/vg_os/lv_WebEx Socialdata /WebEx Socialdata ext3 defaults 1 2
/dev/vg_os/lv_localdisk /localdisk ext3 defaults 1 2
/dev/vg_os/lv_tmp /tmp ext3 defaults 1 2
/dev/vg_os/lv_altroot /altroot ext3 defaults 1 2
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/vg_os/lv_swap swap swap defaults 0 0
nfs.cisco.com:/export/WebEx Socialexport/diagnostics /WebEx Socialdata/diagnostics nfs defaults 0 0
nfs.cisco.com:/export/WebEx Socialexport/jms /WebEx Socialdata/jms nfs defaults 0 0
nfs.cisco.com:/export/WebEx Socialexport/cms /WebEx Socialdata/cms nfs defaults 0 0
nfs.cisco.com:/export/WebEx Socialexport/backup /WebEx Socialdata/backup nfs defaults 0 0
none /opt/ramfs/logs dbcfs defaults,uid=0,gid=0,mode=0770,size=128M,wproc=java,dest=/opt/cisco/WebEx Social/tomcat/logs 0 0
If those are missing, then the node is definitely not using the NFS
server and it will be storing the docs locally in the /WebEx Socialdata
directory. Refer to the linked pages below for details on how to repair
the issue depending on the type of node that has the issue (e.g.: WebEx Social
role vs JMS role vs Social Graph role).
<h1>
Verifying write access on the NFS mounts</h1>
Assuming you see the mounts listed properly in fstab via the steps
above, then the next thing to verify is that you have write permissions
to the NFS shares. To do this:
<ol>
<li>
ssh to the node in question and log in as root</li>
<li>
Type "cd /WebEx Socialdata/diagnostics/logs"</li>
<li>
Type "ls -la" and you should see multiple directories - one for each
node type (Message-Queue, Monitor, etc) and one named "common"</li>
<li>
Type "touch zz", which will create a 0 byte file named "zz" in the /WebEx Socialdata/diagnostics/logs directory</li>
<li>
Type "ls -la" again to verify that you see the "zz" file. If you do
not, or if you received a permission denied message when you tried to
create the file, then you need to look at the permissions of the
exported directories on the NFS server. Either the exports config is
incorrect or the directories do not have the proper permissions to allow
you to write files. Refer to the NFS Permissions link below for steps
on how to fix this.</li>
<li>
Once you have confirmed that the "zz" file exists, then you can delete
it by typing "rm zz" and answering yes to the "rm: remove regular empty
file `zz'?" question that appears</li>
<li>
At this point, you have confirmed that you can read and write to the NFS mount</li>
</ol>
Subject: RE: Troubleshoot NFS
Replied by: Alan Sears on 06-09-2011 07:37:43 AM
James,
In this post you refer to links to another document to repair NFS connectivity for different node types. Can you provide those? I believe I'm running into this issue.
Thanks!
Subject: RE: Troubleshoot NFS
Replied by: Emanuel M Bola on 06-09-2011 10:43:27 AM
Hi James,
Which version of WebEx Social are you using at the moment? There are some changes to the NFS configuration between the Lakota & Chinook releases.
Thanks,
Manny
Subject: RE: Troubleshoot NFS
Replied by: Alan Sears on 06-09-2011 10:46:04 AM
Emanual,
I'm currently running 2.1.2.20000.346.
Thanks!
Alan
Subject: RE: Troubleshoot NFS
Replied by: Emanuel M Bola on 06-09-2011 11:15:34 AM
Hi Alan,
First off, check/correct the status of the rpcidmapd service.
service rpcidmapd status ?
&
chkconfig --list rpcidmapd ?
If the service is stopped, and the startup sequence is off for all run levels, then do the following.
1) service rpcidmapd start
2) chkconfig --add rpcidmapd
Then, if you need to remount the nfs mounts run the following script.
/opt/software/scripts/WebEx Socialos.sh mountnfs4 <nfsip or dns name>:/<nfs mount name> /WebEx Socialdata
mount -l or df -kh to verify the mounts.
Thanks,
Manny
Subject: RE: Troubleshoot NFS
Replied by: Alan Sears on 06-09-2011 02:26:45 PM
That did it...thanks!
Subject: RE: Troubleshoot NFS
Replied by: Alan Sears on 06-09-2011 02:26:45 PM
That did it...thanks!
Labels:
Getting Started
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:
Customers Also Viewed These Support Documents