In my experience and what others have reported, custom app deployments can be very particular with deployment requirements. If the app can be deployed while creating it's own log file, that would help. You might be able to see more information inside of the Agent Logs for the device inside of Systems Manager. Alternatively, you can look at the logs from the local machine in /var/log/m_agent.log, m_agent.log.0.bz2, m_agent.log.1.bz2, etc. Meraki has documentation on how to enable additional logging that will help you investigate here, there's a specific MacOS section: https://documentation.meraki.com/SM/Other_Topics/Systems_Manager_Logging_and_Troubleshooting#macOS
I've had apps install only if I use certain switches or options - even if the software did not require it. Trial and error, and understanding the logs when/if they're helpful will be needed here. Depending on where it's failing in the deployment, you might even be able to schedule a script to run with the app that can provide more info on it as well.
I'll add 2 previous threads I was involved with related to custom deployments. Both were for Windows devices but the concept is the same and the same steps can be take on MacOS.
- https://community.meraki.com/t5/Mobile-Device-Management/Custom-App-Install-Failed-Log/m-p/156316#M9448
- https://community.meraki.com/t5/Mobile-Device-Management/Windows-custom-APPs-failing-to-install-need-guidance/m-p/170331#M9648
Hopefully this helps get you in the right direction!
