Expanding and Maintaining the RosettaHealth Platform with Zero Downtime
Our SaaS-based solution, which makes electronic health record exchange affordable and ubiquitous, has grown substantially from its beginnings three years ago.
For the first two years, we found that scheduled maintenance during low traffic times of the week were not too difficult to coordinate. However in the last year this has rapidly changed, we found ourselves with a system that needed to be kept running -- while also scaled out, upgraded and reconfigured to handle the changing needs of our customers.
As such, we had to devise a plan for incrementally overhauling the running system without ever taking it offline. My boss and coworker in this effort jokingly equated it to taking a Cessna aircraft and transforming it into a fighter jet mid-flight, which is no small task.
The seamless growth of the system was accomplished through standing up software load balancers that would increasingly handle the load as we diverted traffic to them. Because in many cases the public facing TCP ports are not the ports our services are listening on, all we had to do was disable the port redirection. It was by the removal of redirection on a port-by-port basis that we gradually diverted all traffic through the load balancers.
Once we had the public facing endpoints load balanced, we were then able to reconfigure individual instances of a load-balanced group to perform their own internal calls through the load balancers.
These few changes to our service have resulted in monumental differences in our ability to manage the environment. Upgrades and reconfigurations are routinely done during business hours as opposed to waiting until late at night for a quiet period. We’re also able to expand out the environment to meet our customers’ needs without ever having to declare an outage for system maintenance.
These are the notable items encountered when scaling out the RosettaHealth platform:
- Adjusted TCP keepalives
- Increased maximum database connections
- Addressed slow leaks of database connections by third party connectors
- Increased maximum allowed file handles
- Reduced use of swap file
- Optimized SQL
Lastly, we would like to acknowledge the following articles, which we used in our research efforts:
For more information about RosettaHealth, an innovative, secure, standards-based SaaS interoperability platform that eases the movement of electronic medical records across organizational boundaries, please contact Phillip here.