Agile, devops, continuous integration, continuous delivery, scrum, kanban, automation... How many times a day do you hear one of these words? I come across these words every single day whether it be in a book or during a conversation with a co-worker. There is a reason for it though. Everybody is trying to improve their team's performance so that they deliver their products to the market faster than their competition. And these words/techniques/practices, if adopted well, help you achieve this. The best book I can recommend to understand their benefit is The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations
Make sure to read this book if you are at all interested in improving your own and your team's performance.
If you have read it, you know that one of the biggest and most important pillars of a DevOps team is continuous integration. It means that every commit is integrated to the mainline (master/trunk) branch all the time.
Now imagine that you have a big improvement to do in your project. Your team decided to switch from Peewee ORM to SQLAlchemy. How would one normally go about doing this task? He would open a new branch and do everything there. As this is a big task, it would take him days, if not weeks, to finish it. At the end, he would merge this branch with the master. He would probably have lots of conflicts. He would eventually merge everything and deploy it to production. Users would probably complain that some things are not working at all. Something went wrong while merging the conflicts. It would take him a couple more days to fix everything.
You can't blame this guy. Merging is difficult and scary. You should blame the process. The team was not continually integrating their code. If they were, they would not have had to merge all the things at the end. They would integrate every commit and as commits are small, it would not be difficult to merge.
"But this is a big task. How can you integrate all your commits to the master all the time and deploy to production? Users would see the incomplete work in progress" I hear you say. Let me show you how you can accomplish this feat painlessly both for you and your users.
Feature Toggles
The first thing we need to understand is Feature Toggles. This is a simple but powerful concept. Imagine you developed a big feature. But you are too afraid to open it up for thousands of users. You just want to test it with a couple hundred users and gradually increase this number. You can do something like this:
if feature_is_turned_on_for_this_user(feature_name, user):
return the_new_interface
else:
return the_old_interface
That is it. Feature Toggles are simply an if statement with a fancy name :)
Branch by Abstraction
This one is more interesting and is implemented in several steps.
Step I is a situation described above. There are some parts of you code that use Peewee ORM code that you want to replace:
In Step II, we create an abstraction layer for the Peewee code and make one of the clients to work with this abstraction layer using Feature Toggles:
In Step III, we move all the clients to work with this abstraction layer only:
In Step IV, we add the new SQLAlchemy code and make one of the clients to work with the new code using Feature Toggles:
In Step V, we move all the clients to use the new code:
At this point you have successfully and undetectably moved your old Peewee code to the new SQLAlchemy code. If something goes wrong, you can easily bring back the old code until you fix the bug in the new code. How? Feature Toggles, baby!
If everything is working okay, you may even delete the abstraction layer:
That is it. No branches, no merges needed. Everything has been done in the master branch.
"This is all good and dandy but some code would be great" I hear you say. Cool. Let's go. But remember that this is an imaginary case and I will be using Python-like pseudo-code.
Step I
Some parts of your software uses Peewee code that you want to replace:
def client_one():
...
peewee = Peewee('my_database')
users = peewee.get_users()
...
def client_two():
...
peewee = Peewee('my_database')
new_user = peewee.create_user(username='rahmonov', password='rockstar')
...
def client_three():
...
peewee = Peewee('my_database')
user = peewee.get_user(username='rahmonov')
Step II
We add an abstraction layer and use it in one of the clients:
class DatabaseManager:
def __init__(self, database_url, use_sqlalchemy=False)
self.use_sqlalchemy = use_sqlalchemy
self.peewee = Peewee('database_url')
def get_users(self):
users = self.peewee.get_users()
return users
def get_user(self, username):
user = self.peewee.get_user(username=username)
return user
def create_user(self, username, password):
new_user = self.peewee.create_user(username=username, password=password)
return new_user
def client_one():
...
db = DatabaseManager('my_database')
users = db.get_users()
...
Nothing really changed. We have DatabaseManager that acts just like Pewee. At this point, you should make sure that all your unit tests are passing.
Step III
We use the new DatabaseManager in all of our clients:
def client_one():
...
db = DatabaseManager('my_database')
users = db.get_users()
...
def client_two():
...
db = DatabaseManager('my_database')
users = db.create_user(username='rahmonov', password='rockstar')
...
def client_three():
...
db = DatabaseManager('my_database')
user = db.get_user(username='rahmonov')
Step IV
We add the new SQLAlchemy code to the DatabaseManager class and use it in one of the clients:
class DatabaseManager:
def __init__(self, database_url, use_sqlalchemy=False)
self.use_sqlalchemy = use_sqlalchemy
self.peewee = Peewee('database_url')
self.sqlalchemy = SQLAlchemy('database_url')
def get_users(self):
if self.use_sqlalchemy:
users = self.sqlalchemy.retrieve_all_users()
else:
users = self.peewee.get_users()
return users
def get_user(self, username):
if self.use_sqlalchemy:
user = self.sqlalchemy.fetch_user(username=username)
else:
user = self.peewee.get_user(username=username)
return user
def create_user(self, username, password):
if self.use_sqlalchemy:
new_user = self.sqlalchemy.create_user(username=username, password=password)
else:
new_user = self.peewee.create_user(username=username, password=password)
return new_user
features = {
'sqlalchemy': {'client_one': True, 'client_two': False, 'client_three': False}
}
def client_one():
...
db = DatabaseManager('my_database', use_sqlalchemy=features['sqlalchemy']['client_one'])
users = db.get_users()
...
Note the features dictionary which we are using to show how feature-toggles work. Ideally, those configurations should be stored in a database or something else that you can change without deploying your code. Imagine something goes wrong. You will just have to set the value to False and it will start using the old code automatically without any deployment.
Step V
We use the new code in all of our client codes:
features = {
'sqlalchemy': {'client_one': True, 'client_two': True, 'client_three': True}
}
def client_one():
...
db = DatabaseManager('my_database', use_sqlalchemy=features['sqlalchemy']['client_one'])
users = db.get_users()
...
def client_two():
...
db = DatabaseManager('my_database', use_sqlalchemy=features['sqlalchemy']['client_two'])
users = db.create_user(username='rahmonov', password='rockstar')
...
def client_three():
...
db = DatabaseManager('my_database', use_sqlalchemy=features['sqlalchemy']['client_three'])
user = db.get_user(username='rahmonov')
Make sure that all unit tests pass. If something goes wrong, turn off the feature for everybody, fix the bug and turn it back on.
When everything is good, go on to the next step:
Step V (optional)
You can optionally delete the abstraction layer and the dead Peewee code as well.
def client_one():
...
sqlalchemy = SQLAlchemy('database_url')
users = sqlalchemy.retrieve_all_users()
...
def client_two():
...
sqlalchemy = SQLAlchemy('my_database')
new_user = sqlalchemy.create_user(username='rahmonov', password='rockstar')
...
def client_three():
...
sqlalchemy = SQLAlchemy('my_database')
user = sqlalchemy.fetch_user(username='rahmonov')
That's it! You have successfully worked on a big improvement without using branches, i.e. directly on the master branch. Add small commits and unit tests to this mix and you will have the famous Trunk Based Development implemented, which is a theme for another post.
Let me know in the comments if you have any questions.
Fight on!