We are re-architecting a typical server - agent product for AWS SaaS. The UI and configuration DB will be on the cloud while the agents will be deployed on-prem. The problem is that 100K agents need to periodically poll the server if there are any configuration updates for them. This seems like a typical hybrid cloud scenario but I couldnt find any standard approach for this. Approaches I have come across:
- Deploy a cloud connector on-prem that is responsible for polling and updating the agents (1 request vs 100K requests every 5 mins). The downside is that the saas customer needs to manage this connector.
- Use Confluent Kafka cloud bridge and register for config change events
- Put the config on S3 that the agents can poll. Better than our own API but still the requests are expensive.
- AWS direct connect, yet to explore this fully but wondering if this is needed for basic config pull.
Any other approaches that might work? Thanks!
