It is best and easy to use a ACL to restrict the access to other users.
I will look around , how to restrict SSH without using of ACL.
but till then u can try this, if you want:
access-list 5 permit (Hostname) 0.0.0.0
access-list 5 deny any any
then apply to Line vty
line vty 0 4
! Only allow ssh
tranport input ssh
! Apply our access list for incoming connections
access-class 5 in
! Finally, apply an exec-timeout, which will disconnect an idle connection
! The timeout is 10 minutes and 0 seconds
exec-timeout 10 0