What do you mean? You want to stop people from PHYSICALLY accessing the console port but leave the data cabinets open?
If this is the case, then the only thing I can think of is to put a "lock" to stop people from plugging anything into the console port.
In the meantime you can enable TACACS and/or RADIUS on your console port. Disable Telnet and only allow SSH.