Try to telnet to the smtp server with the port number and attempt a "helo" and see if the connection is being refused. In the logs it indicates a connection refused message which implies that you need to allow the lms server to connect your lotus server.
[ Thu May 26 09:47:12 ICT 2011 ] WARN [MailUtility : isMailServerWorking] : While trying to send HELO, the mail server [172.28.38.11] had thrown this exception :
java.net.ConnectException: Connection refused: connect
[ Thu Apr 14 16:19:55 ICT 2011 ] WARN [MailUtility : isMailServerWorking] : While trying to send HELO, the mail server [172.30.83.2] had thrown this exception :
java.net.ConnectException: Connection refused: connect