Brief overview:
If Auth Fail send radius-reject because client has not met your authc requirements
If user not found in your identity source used send radius-reject. This could be internal endpoints/AD/etc. or all of the above depending on configuration.
If process fail send radius-reject. Process fail could be mab/dot1x process failing/terminating depending on configuration.
Description of options:
Reject: Send ‘Access-Reject’ back to the NAD
Continue: Continue to authorization regardless of authc outcome
Drop: Drop the request and do not respond to the NAD. During this the NAD deems ISE as dead.
HTH!