Amazon RDS Blue/Green Deployments: A Silver Lining to Database Updates at Scale with Reduced Downtime
Change management in databases, in a nutshell, is difficult, risky, and highly time-consuming and thus requires a great deal of groundwork and testing in a synchronized staging environment before the change can be propagated to the corresponding production environment. Every minor and major change on a database from performing version upgrades, software patches, and underlying maintenance updates; to the more complicated schema-level changes such as modifying column data types, altering primary keys, dropping a column (no less on a production database), or, if you’re anticipating adventure, even dropping an entire table. In many of these cases, particularly the ones involving schema migrations, you have to play dice with the predicted downtime and performance metrics and hope that it does not decide to roll a zero. The initial testing sometimes do not coincide with the implementation of the changes at the production layer and thus, this results in unpredictable but mandatory save-the-day sprees.
Although, one of the many perks of working with data and databases—due to the nature of data being abundant and critical—is getting to work with complexity and scale quickly; this scalability comes with a cost. And that is even minor changes in the system require downtime. Sometimes, when the change is unforeseen and unplanned, and you work with databases where the size exceeds terabytes of storage and billions of rows, the downtime can last for hours and even days. So, how exactly do we avoid or reduce this downtime?
Brief intro to Blue/Green deployments
Disclaimer: Blue/Green Deployments have nothing to do with blue or even green and everything to do with deployments. While, some claim that you may as well call them Red/Black Deployments but for the scope of this article, we shall be using the more classic term i.e. the Blue/Green Deployment.
As described by Jez Humble and David Farley in their Continuous Delivery book, “The idea is to have two identical versions of your production environment, which we’ll call blue and green… Users of the system are routed to the green environment, which is the currently designated production. We want to release a new version of the application. So we deploy it to the blue environment… This does not in any way affect the operation of the green environment. We can run smoke tests against the blue environment to check it is working properly. When we’re ready, moving to the new version is as simple as changing the router configuration to point to the blue environment instead of the green environment. The blue environment thus becomes production. This switchover can typically be performed in much less than a second. If something goes wrong, we simply switch the router back to the green environment.”
In a straightforward way, it is a strategy for implementing change in any software product by orchestrating dual environments, both synchronised, for the said product and switching them as per the latest changes in each environment by employing the use of traffic routing.
Blue/Green deployments for RDS
The trickier part of the problem set arises when we want to implement the concept of Blue/Green Deployments to database change management. However, with Amazon RDS Blue/Green Deployment, you can achieve a seamless mechanism for deploying your databases changes to production with minimal downtime.
The basic workflow
The basic workflow of a database change deployed by utilizing Amazon RDS Blue/Green Deployment includes creating the deployment, making the necessary configuration, parametric, or schema changes to the green environment (the staging environment), performing testing on the staging environment, and finally the switching over of the blue (the live production) environment to the green environment. Sounds exceptionally easy, right? It definitely is.
Let’s walk you through each step of using RDS Blue/Green Deployment towards the deployment of a database change.
1. Provisioning the deployment
The Blue/Green Deployment for RDS can be created using the AWS Console or the Command Line client. However, if you prefer to work with automation scripts, using the AWS SDK to manage AWS services simply takes the cake. We’ll be using AWS SDK for Python (boto3) here to tinker with the Blue/Green Deployment.
When you create the deployment for a database instance, RDS copies the blue environment’s topology to a staging area, along with its configured features while establishing replication between the two environments.
However, after the creation of the staging environment, the data is loaded to the new database in the “lazy loading” mode which prevents quick availability of the database instance. Wait till the data is completely loaded before switching over to avoid performance degradation at the database level.
2. Let’s introduce the changes
Once the deployment is up and running, you can make your required changes to the green environment with predictable downtime. These changes may include upgrading the major or minor DB engine version, changing database parameters by attaching a new parameter group to the database in the green environment, or making schema changes in the staging environment.
Make sure the schema changes might only be replication-compatible such as adding new columns at the end of table, adding or dropping indexes. However, renaming columns or tables should be avoided, as such modifications will disrupt replication.
3. Performing the switchover
After you have made the changes in the green deployment, you can perform the switchover whereby the green environment is promoted to replace the blue environment by rerouting the traffic.
But, before the RDS performs the switchover for the deployment, it performs certain guardrail checks to make sure both the—blue and green—environments are in healthy state, the replication is working as expected, no active writes or DDL operations are running on the primary database etc. Further, you can specify a timeout value for the switchover between 30 seconds and 3,600 seconds—in case the switchover exceeds this period, any changes to the deployment are rolled back.
Post-switchover
After the switchover is complete, the green environment starts serving the production traffic with the updated changes in place. Meanwhile, the database in the previous blue environment is kept running. You can either delete the database instance or use it for further testing (after rebooting the instance).
This is it, you have successfully made changes to a possibly humongous database without batting an eye with the help of Blue/Green Deployments for RDS.
