First, you're using an unsupported version of the Java Plug-in. You must be using either 1.4.2_10 (if you do not have the DST patch) or 1.4.2_12 (if you do have the DST patch). You are using 1.5.0. You need to remove all installed Java Plug-ins, then install the one correct version.
Next, the most common causes of this error are either a firewall or access-list between the client and server which could be blocking TCP ports 1683, 1684, or 8088. Make sure the client can telnet to the server on each of those ports. For example:
telnet server 1683
The other common cause is the server's IP address was recently changed, but you did not delete the gatekeeper.ior files after the change. If this is the case, shutdown dmgtd on the server, delete all gatekeeper.ior files under NMSROOT, then restart dmgtd.
Finally, if this is a multi-home server, make sure you configure the proper inbound client interface in NMSROOT/lib/vbroker/gatekeeper.cfg. See that file for instructions.
