we can disable the "Smartcard Removal Feature" using the smartcard-removal-disconnect command:
group-policy name attributes
[no] smartcard-removal-disconnect {enable | disable}
Disabling this option configures the Client to leave existing tunnels connected when a SmartCard is removed. Enabling this option configures the Client to disconnect existing tunnels when a SmartCard is removed.
This option is enabled by default. The no form of the command removes the command from the configuration and reverts to the default value