First it helps if I use the correct port, ssh was on 2202. Second, my ad_access_filter wasn't correct in sssd.conf for the user. I added the user to users to be checked by duo but made an exception for the user in /etc/pam.d/sshd to skip duo auth for that user.
auth [success=1 default=ignore] pam_succeed_if.so user = ansible
auth [success=1 default=ignore] /lib64/security/pam_duo.so