I'd highly recommend getting on a call with the TAC for doing this. You may very well encounter issues and the TAC can work through them with you in real time if you open a ticket first.
That said, most use cases would involve upgrading instead of rolling back a patch. 6.4.0.4 is currently the recommended release. It's hard to imagine a condition where you would want to move backwards on an already older release.