Autodeploy Jekyll using bitbucket post-commit service hooks and Flask

Screenshot 1

A very nice and clean way to deploy a Jekyll (or any other static site) is to setup a very small application on your site’s webserver to listen for post commit hooks from Bitbucket. This allows you to have your site automatically updated and regenerated every time you push to your repository.

Unfortunately it requires quite a bit of setup, so this blog posts shows you how to go about getting it working using a very simple Flask application.

The code for this blog post can be seen on my Github account →


What Is Going To Happen

When everything is setup, the following will happen:

  1. We write a new blog post locally (or via the Bitbucket online file editor). We commit our changes and push them to our Bitbucket repo.
  2. Bitbucket sends a POST HTTP request (using basic authentication) to our webserver (http://example.com/githook/ for example)
  3. A very simple Flask application on our webserver is listening on that predefined URL for incoming requests and notices Bitbuckets request.
  4. Our Flask application calls a bash script
  5. That bash script will pull the new changes from the Bitbucket repo to our webserver over SSH
  6. The bash script will then regenerate Jekyll’s static files from those changes
  7. Once everything is finished, we send an email to confirm that the changes have been succesfully pulled in.

What We Are Going To Use

  • An existing Jekyll site. I’m going to assume you have an existing Jekyll site setup on your webserver (although this should work with any static-generated site)
  • Python + pip. I’m going to assume that you have both of these already installed on your machine.
  • Virtualenv/Virtualenvwrapper. This will manage our python environment. I will assume you are familiar with why we would want to use this and that they are both already installed . Otherwise, see this section of the docs for more information on the benefits of virtualenv
  • Flask microframework. This is a fantastic little microframework written in Python that makes it very easy to setup a bloat-free web application. We will install this.
  • Nginx. This is our webproxy. I won’t go through the setup of this and assume that it is already installed (or you are comfortable using something else)
  • Gunicorn. This will act as our application server. We will install this.
  • Supervisor. For starting and stopping our app. Again, I’m not going to go into why you might want to use this, it’s simply what I am using. I assume that you already have this installed

All of the code for this blogpost is on Github


1. Setup Our Webserver Environment

For this blog post, our virtualenv path is assumed to be /srv, meaning our applications virtualenv folder will be /srv/bitbucket.githooks/...

>>> mkvirtualenv --no-site-packages bitbucket.githooks
>>> cdvirtualenv
>>> git clone https://github.com/timmyomahony/bitbucket-jekyll-hook app
>>> add2virtualenv app/src/  # Make sure the app is on our python path
>>> cd app
>>> pip install -r requirements.txt

Before we go on, let’s look at our app’s layout:

.
├── .ssh
│   └── id_rsa
├── bin
│   └── update_jekyll.sh
├── etc
│   ├── gunicorn.conf.py
│   ├── nginx.conf
│   └── supervisor.conf
├── README.md
├── requirements.txt
└── src
    ├── app.py
    ├── auth.py
    └── __init__.py
  • .ssh/id_rsa is our Bitbucket private key
  • src/app.py is our Flask application.
  • src/auth.py is a Flask plugin to allow basic authentication so people can’t troll us.
  • bin/update_jekyll.sh is the bash script that will do the actual pull and regeneration of our jekyll site when an authentic request comes in from Bitbucket
  • etc/* are all configuration files to run our application

Now let’s create some log files:

>>> cd ../      # i.e. /srv/bitbucket.githooks/
>>> mkdir logs
>>> mkdir logs/supervisor
>>> mkdir logs/nginx
>>> mkdir logs/gunicorn
>>> touch logs/supervisor/access.log
>>> touch logs/supervisor/errors.log
>>> touch logs/nginx/access.log
>>> touch logs/nginx/errors.log
>>> touch logs/gunicorn/access.log
>>> touch logs/gunicorn/errors.log

Let’s configure nginx - etc/nginx.conf:

server {
        listen                  80;
        root                    /srv/bitbucket.githooks/;
        server_name             example.com/githook/;
        access_log              /srv/bitbucket.githooks/logs/nginx/access.log;
        error_log               /srv/bitbucket.githooks/logs/nginx/error.log;
        client_max_body_size    20m;

        location / {
                proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_redirect off;
                if (!-f $request_filename) {
                        proxy_pass http://127.0.0.1:50100;
                        break;
                }
        }
}

and symlink the configuration and restart the nginx process:

$ sudo ln -s /srv/bitbucket.githooks/app/etc/nginx.conf /etc/nginx/conf.d/bitbucket.githooks.conf
$ sudo /etc/init.d/nginx restart

Now make sure our paths and ports are correct in our gunicorn configuration - etc/gunicorn.conf:

bind = "127.0.0.1:50100"
accesslog = "/srv/bitbucket.githooks/logs/gunicorn/access.log"
errorlog = "/srv/bitbucket.githooks/logs/gunicorn/errors.log"
workers = 1

Now setup supervisor - etc/supervisor.conf - so that we can easily start and stop our application:

[program:githooks]
command=/srv/bitbucket.githooks/bin/gunicorn app:app -c ./etc/gunicorn.conf.py
directory=/srv/bitbucket.githooks/app/
user=gunicorn
autostart=true
autorestart=true
stderr_logfile = /srv/bitbucket.githooks/logs/supervisor/errors.log
stdout_logfile = /srv/bitbucket.githooks/logs/supervisor/access.log
stopsignal=INT

and also symlink and load the config (we won’t actually run it yet so it’s ok if supervisor gives out about there being an error - we need to configure the actual flask application first):

$ sudo ln -s /srv/bitbucket.githooks/app/etc/supervisor.conf /etc/supervisor/conf.d/bitbucket.githooks.conf`
$ sudo supervisorctl
supervisor> reread
supervisor> update
supervisor> stop githooks

2. Setup Our Bitbucket Post Commit Hook

We need to tell bitbucket that we want it to actually make a post-commit request to our webserver from bitbucket.org when our codebase changes. We do this through our repository settings.

Read detailed documentation on regarding this step on bitbucket.org

First go to your repo settings:

Screenshot 2

Click “Hooks” on the lefthand side. From the dropdown in the center of the screen select “POST” and click “Add Hook”. This will bring up a modal dialog.

We are going to use basic authentication, so add something like the following:

http://bob:mypassword1234@example.com/githook/

Screenshot 3

You’ll notice I have my hook setup on a subdomain of my site instead of a path.


3. Setup A Bitbucket SSH Keypair

When we receive a post-commit hook request from Bitbucket we will want to pull down the repo changes to our webserver. We can’t enter our username and password here so we will instead setup a private/public keypair that can authenticate us in absentia.

Warning: The following approach is using a keypair without a passphrase which means that anyone who gets a hold of the private key has access to your entire bitbucket account. This is less then ideal. A better approach is to use a keypair with a passphrase, but use ssh-agent to manage the passphrase so that it only has to be entered once

First create a keypair on your local development machine (this is not my actual keypair!):

$ ssh-keygen
Generating public/private rsa key pair.
/Enter file in which to save the key (/Users/timmy/.ssh/id_rsa): /Users/timmy/Desktop/id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/timmy/Desktop/id_rsa.
Your public key has been saved in /Users/timmy/Desktop/id_rsa.pub.
The key fingerprint is:
63:f5:c7:98:a7:2d:e0:c3:06:e0:e1:f7:c0:1d:dd:1a timmyomahony@macbook-pro
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|           . .   |
|      o   o E .  |
|     o + o o *   |
|      o S o = +  |
|       o B . =   |
|          * o .  |
|         . . .   |
|                 |
+-----------------+

When prompted, make sure not to enter any passphrase. This will create two keys: id_rsa (private key) and id_rsa.pub (public key).

Go to your account settings (not repo settings like in the last step) and add the public key (again, this is not my actual public key):

Screenshot 4

Now go to your webserver and paste the contents of your private key into .ssh/id_rsa. You should now be able to pull your repo over SSH.

Again, read detailed documentation regarding this step on bitbucket.org if you are having trouble with this →


4. Setup The Flask Application

First let’s have a look at our Flask application in src/app.py

from flask import request
from flask import Flask
from werkzeug.contrib.fixers import ProxyFix
from subprocess import Popen, PIPE
from email.mime.text import MIMEText
import smtplib
from auth import requires_auth

app = Flask(__name__)

EMAIL_TO = 'foo@bar.com'
EMAIL_FROM = 'baz@bar.com'
EMAIL_LOGIN_USERNAME = 'joe@bar.com'
EMAIL_LOGIN_PASSWORD = '1234'

SCRIPT_PATH = '/path/to/virtualenv/app/bin/update_jekyll.sh'

def send_email(subject, body):
  server = smtplib.SMTP('smtp.gmail.com:587')
  server.starttls()
  server.login(EMAIL_LOGIN_USERNAME, EMAIL_LOGIN_PASSWORD)

  msg = MIMEText(body)
  msg['To'] = EMAIL_TO
  msg['From'] = EMAIL_FROM
  msg['Subject'] = subject

  server.sendmail(EMAIL_FROM, EMAIL_TO, msg.as_string())
  server.quit()

@app.route('/', methods=['POST', ])
@requires_auth
def update_jekyll():
  try:
    (stdout, stderr) = Popen([SCRIPT_PATH,], stdout=PIPE).communicate()
    send_email("Bitbucket post commit hook succesfully executed", "Response: %s" % body)
    return 'Success'
  except:
    send_email("Error with Bitbucket post commit hook", "There was an error performing the script associated with the post commit hook")

app.wsgi_app = ProxyFix(app.wsgi_app)

if __name__ == '__main__':
  app.run()

This is a simple Flask app that has one URL/view which listens for incoming authenticated POST requests only (doesn’t reply to GETs) and runs our accomanying bash script. It then sends an email via Gmail to report whether or not the operation was succesful.

We need to make sure we have the following variables configured:

EMAIL_TO = 'foo@bar.com'
EMAIL_FROM = 'baz@bar.com'
EMAIL_LOGIN_USERNAME = 'joe@bar.com'
EMAIL_LOGIN_PASSWORD = '1234'

SCRIPT_PATH = '/srv/bitbucket.githooks/app/bin/update_jekyll.sh

Note that we aren’t taking into consideration what sort of POST data Bitbucket is actually sending us. We could improve this script by reading the POST data and seeing if the commit that caused Bitbucket to contact our server was actually made to the master branch but for the moment we are just happy getting the request.

Also note that there is a @requires_auth decorator on our view to allow for basic authentication. This needs to also be configured.

Open src/auth.py

from functools import wraps
from flask import request, Response

def check_auth(username, password):

    '''
    This function is called to check if a username /
    password combination is valid.
    '''

    return username == 'foo' and password == 'baz'

def authenticate():

    '''Sends a 401 response that enables basic auth'''

    return Response(
    'Could not verify your access level for that URL.\n'
    'You have to login with proper credentials', 401,
    {'WWW-Authenticate': 'Basic realm="Login Required"'})

def requires_auth(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        auth = request.authorization
        if not auth or not check_auth(auth.username, auth.password):
            return authenticate()
        return f(*args, **kwargs)
    return decorated

and make sure to change username foo and password baz to the values you used when setting up the POST commit hook on Bitbucket.org

Finally, lets have a look at the bash script - bin/update_jekyll.sh

#!/bin/bash
cd /srv/jekyll.timmyomahony.com/timmyomahony.com/
sudo -u admin git pull origin master
chown -R admin:admin ./*
jekyll build

This is a very primative script. It is being run as root and simply pulls our repo and builds our jekyll project. You can add more to this if needs be but as-is it covers the basics.


5. Start Up The App

Now we have everything setup, we should be able to launch the app using supervisor

$ sudo supervisorctl
supervisor> start githooks

Every time you make a change to your codebase now, Bitbucket should contact your server and your server should automatically update!