Yes, always a good idea and is recommended for the reasons you mention. It can also in some cases, case one-way audio or video as well. Video conferencing uses real time video, so you want to make sure anything that could delay or hinder that communication such as stateful/deep packet inspection of agl on the firewall is turned off.
A really good explanation can be found in the "What About Routers/Firewalls with SIP/H.323 ALG?" section of the VCS Deployment Guide on pg 58.
