The license server supports global instances of Control Room with high availability and disaster recovery capabilities. Regular health checks of the applications trigger failover processes to maintain continuity of the service.

License Server (LS) is hosted centrally in the US. All the instances of the Control Room across the globe which are using GUID based license or cloud license connect to this server. Being a critical service the license server has high availability and disaster recovery capabilities. Below diagram illustrates the detailed infrastructure level architecture. For more information about cloud license, see Cloud licensing FAQ.

Consider two regions, Oregon (primary) and Virginia (standby) regions. The standby Virginia region has a worker job which regularly checks for the application and the database health of the primary Oregon region. The below diagram is about the optimal state.

License server optimal state

If the application is unresponsive, the server checks the primary master database. If both the application and database repeatedly fail health checks, the worker job starts a failover process.

The following three scenarios can trigger disaster recovery state and starts the failover process:
  • If the primary application is down (all 3 dynos are inactive) but the master database is active, the worker job takes no action. However, the alert and monitoring system notifies the team.
  • If the primary application is active but the master database is down, Heroku automatically assigns a standby database from another availability zone in the same region.
  • If both the primary application and the master database are down, the worker job triggers the failover script.
If either of the three scenarios mentioned above occurs, the failover script is activated and the following workflow is implemented:
  • The follower database in the Virginia region syncs all commits with the master database in the Oregon region and then stops following it.
  • The database in the Virginia region becomes the new master database.
  • The Virginia application connects to the new master database in the Virginia region.
  • The Web Application Firewall (WAF) connects to the Virginia application's router.
  • The connection between the WAF and Oregon is removed.
License server disaster recovery state

When the Oregon region becomes active again, manual intervention (of reconnecting the Web Application Firewall (WAF)) is needed to restore Oregon as the primary application.