Here's a link for boot loop recovery.
Since it's a new install you probably don't have anything to recover though:::
https://supportforums.cisco.com/docs/DOC-26689#comment-14559
Reading the system requirements for the Express version, one would think any version of ESXi is supported.
Looking at the system requirements for the Standard and Pro versions though it appears 5.1 is the latest version.
VMWare Version
Express - ESXi 4.1 or later
Standard - ESXi 5 or ESXi 5.1
Pro - ESXi 5 or ESXi 5.1
Try with ESXi 5.1 and I'll bet it works!