I have configured two ACS 4.2.124 Patch 17 ACS Servers on Windows 2008. The Domain is a Windows 2008 AD. I configured Group mappings for some AD Groups where some test laptops are member of the groups. We also have installed a internal Microsoft CA and have configured a GPO to rollout Client Certificates on the workstations. On the ACS Server we have configured different groups with different VLAN assignment. The mapping only works if under external user databases and default one group is configurate. All clients authenticated come in that default group, so no mapping from the domain is performed.
Attached is the output from the Authentication Log from one ACS Server.
UTH 01/16/2012 13:10:16 I 1915 3192 0x11 pvAuthenticateUser: authenticate 'host/080199C.WBS.ADS' against CSDB
AUTH 01/16/2012 13:10:16 I 3092 3192 0x11 pvCopySession: setting session group ID to 0.
AUTH 01/16/2012 13:10:16 I 2838 3192 0x11 pvCheckUnknownUserPolicy: session group ID is 0, calling pvAuthenticateUser.
AUTH 01/16/2012 13:10:16 I 1915 3192 0x11 pvAuthenticateUser: authenticate 'host/080199C.WBS.ADS' against Windows Database
AUTH 01/16/2012 13:10:16 I 0750 3192 0x11 External DB [NTAuthenDLL.dll]: Starting MSCHAP authentication for user [host/080199C.WBS.ADS]
AUTH 01/16/2012 13:10:16 I 1479 3192 0x11 External DB [NTAuthenDLL.dll]: Checking Domain WBS.ADS is present in Domain Filter List permit,WBS.ADS
AUTH 01/16/2012 13:10:16 I 2017 3192 0x11 External DB [NTAuthenDLL.dll]: Got WorkStation S-ACS1
AUTH 01/16/2012 13:10:16 I 2018 3192 0x11 External DB [NTAuthenDLL.dll]: Attempting Windows authentication for user 080199C$
AUTH 01/16/2012 13:10:16 I 2076 3192 0x11 External DB [NTAuthenDLL.dll]: Windows authentication SUCCESSFUL (by LogonServer V-DC1) and (by LogonDomain WBS)
AUTH 01/16/2012 13:10:16 I 1716 3192 0x11 External DB [NTAuthenDLL.dll]: User mapped to ACS group id 
AUTH 01/16/2012 13:10:16 I 2853 3192 0x11 pvCheckUnknownUserPolicy: setting session group ID to 3.
AUTH 01/16/2012 13:10:16 I 4320 3192 0x11 Final group map: 3.
This group ID 3 is the configured group for the default external user databases. But we need to setup the group mapping from AD to ACS group because of the VLAN Assignment.
When you go to External User Databases > Database Group Mappings > Windows Database > Click on the Appropriate domain > does the ACS display the list of the AD Groups or does it give you a "Failed to Enumerate Windows Groups error"?
If you are able to see the groups, can you see and select the appropriate ones for the Group Mapping?
Would you kindly share your Group Mapping configuration on a screenshot?
Have you tried to configure Manual Group Mapping? NOTE: You will need the exact Windows Group name in order for it to work.
Also, is the ACS application installed on a Windows Domain Controller or Member Server?
NOTE: Remember that the ACS 4.x does not support Windows Server 2008 R2 as the backend DC.
You are correct. On ACS 4.x, if you create a Group Mapping entry like "Domain Admins, Domain Users, Acccount Operators" the ACS will only match a username with that combination when it belongs to all three groups.
As you realized the ACS Group Mapping performs a logical AND to the combination instead of a Logical OR. I am glad you figured it out.