That depends on how your switch is configured to process DOT1X and MAB. The standard and usual case is to process DOT1X first, and then timeout (if no supplicant response is received) to MAB. With IBNS 2.0 you can even do DOT1X and MAB concurrently - although, that has its pros and cons. There is nothing wrong with switching the order around (MAB first , then DOT1X) but that could have issues of its own - usually one or more device types that give you the most grief.
Have a look at your switch config and see what your DOT1X timer values are like - 5 minutes does sound a bit long.
If you have foreign endpoints connected to your network (e.g. from other organizations) then the supplicant should reply, and then ISE should reject them (since these are foreign endpoints). If you fail DOT1X then the PC should revert to sending non-DOT1X packets, and your switch should process that as unknown MAB endpoints, which I guess lands them in the guest portal redirection Authorization Result.
Windows has its own timers that will restart a failed DOT1X process - can't remember - I think it's 10 minutes or 20 minutes - the Wired Auto Service will keep trying to see if it can connect via 802.1X.
