--- author: '' category: '' date: 2013/06/03 00:31:40 description: '' link: '' priority: '' slug: deploying-django-into-my-cheap-vps tags: python title: Deploying Django Into My Cheap VPS type: text updated: 2013/06/03 00:31:40 url_type: '' --- I am preparing to open my cheap site-and-blog-hosting service to the public at some point, so I needed to do some groundwork into deployment. Consider that the host that will run it will have very limited resources, so I needed to find lean and cheap solutions when possible, but at the same time, I want to achieve reasonable reliability and ease of deployment. Since this is a testing server, I want it to have git master deployed. I don't want *automatic* deployment, but I want to deploy often, meaning several times daily. I preferred simple tools instead of complex tools, lightweight tools with just enough features instead of heavier, more fully-featured tools. Your choices on each step could and probably should be different than mine, depending on your situation, requirements and personal preferences. So, here's my notes from how it's done currently. This is not meant as a HOWTO, just a description of what seems to be working well enough so far. .. TEASER_END Preparing the System -------------------- Along this deployment we'll use a lot of stuff. So, lots of stuff needs to be installed and configured before we can *start* deployment. .. sidebar:: Not Covered Here * Setting up your email server. * Setting up DBMS 1) Get rid of all services that will not be needed. Ideally nothing should be running on the system for which you don't have a compelling reason. 2) Install and setup DBMS. I got MySQL via the Ubuntu package. 3) Install other things. In this case, that meant: * Virtualenv * Redis (Used to create job queues) * Nginx (For reverse proxying) * Gatling (For static file serving) * monit (For service monitoring/restarting) * Exim4 (For SMTP service) Choices made here: ~~~~~~~~~~~~~~~~~~ Virtualenv instead of Ubuntu/Debian packages The chance to ensure I am getting the exact same version of everything as in my development machine makes this a no-brainer. Also, it means I can deploy this as a non-root user. MySQL instead of PostgreSQL It's not a big deal for the testing server, really. It could be SQLite and you may not even notice. Monit instead of Supervisor or Circus I couldn't quite get Circus+Chaussette to work correctly, and Supervisor uses more resources than monit. Monit has a nice web "control panel". Exim4 instead of Sendmail, Postfix, or something else No strong preference, except ease of configuration. If I can make Django use a sendmail binary instead of a SMTP socket to send email, then this is out and nullmailer is in, to avoid a long-lived process. Nginx instead of Apache I heard good things, and Apache bores me, so why not try something different? Gatling instead of Nginx I already had a Gatling setup here, and it's neat way to deal with virtual domains, plus low resource usage made me want to keep it. Also, it being a separate service from Nginx means if Nginx dies, the sites will stay alive. Getting the Code Up There ------------------------- Since the project is hosted at GitHub, it's very reasonable to just use git. Since it's pure python, virtualenv and pip are good for dependency handling. .. sidebar:: System Packages Since this installs lxml, it requires the system to have gcc and a bunch of development libraries, so a script like this will still need some manual tweaks to the system before it will "just work" .. sidebar:: Not Covered Here * Differences between production/development Django settings * Git training * How to build/install packages efficiently using pip So, I created a user at the VPS, and started writing a script that should get everything in place, presented here with much more comment that it has in real life:: #/bin/sh if [ ! -d nikola-server ] then # This means this is an initial deployment or else I have nuked # everything. So start by cloning the repo and creating the # virtualenv it will use git clone git://github.com/ralsina/nikola-server.git virtualenv nikola-server/venv fi # Go into the repo and rollback any changes to settings.py cd nikola-server git checkout alva/alva/settings.py # Update from master git pull # Override settings.py with the deployment version, which # has things like the proper DB settings and such. # Yes I know I could override it using an env. variable # bu this works too. cp ../settings.py alva/alva/settings.py # Activate the venv . venv/bin/activate # Enter the django project's folder and install all it requires cd alva pip install -r requirements.txt # These are requirements for deployment which the development # setup doesn't need (MySQL driver, gunicorn) pip install mysql-python pip install gunicorn # Bring the deployment server's DB up to speed ./manage.py syncdb # Perform any necessary DB migrations ./manage.py migrate allauth.socialaccount ./manage.py migrate allauth.socialaccount.providers.twitter ./manage.py migrate blogs # Put all the static files in the right place so the right # webserver will pick it up ./manage.py collectstatic --noinput Choices made here: ~~~~~~~~~~~~~~~~~~ A simple shell script instead of Puppet or Chef. This is not complicated stuff, it doesn't need much, a shell script is good enough. I can always grow it into a recipe later on. I will surely do that when/if this moves from testing into production, specially since I will want to have all the services configured from a versioned repo. Gunicorn instead of something else I heard good things. No particularly strong preference. Web Servers ----------- This deployment involves a number of webservers: Gunicorn, running in ``localhost:9000`` No configuration, it uses the command line. Gatling, running in ``184.82.108.14:80`` serving static files: Configured in ``/etc/default/gatling``:: DAEMON_OPTS="-e -v -D -S -F -U -u nobody -c /var/www" DAEMON_OPTS="-c /srv/www -P 2M -d -v -p 80 -F -S -i 184.82.108.14" DAEMON="gatling" Nginx, running as reverse proxy for Gunicorn in ``184.82.108.15:80``: Configured in ``/etc/nginx/sites-enabled/default``:: server { #listen 80; ## listen for ipv4; this line is default and implied #listen [::]:80 default ipv6only=on; ## listen for ipv6 listen 184.82.108.15:80; index index.html index.htm; server_name donewithniko.la; # no security problem here, since / is alway passed to upstream root /path/to/test/hello; location / { proxy_pass_header Server; proxy_set_header Host $http_host; proxy_redirect off; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Scheme $scheme; proxy_connect_timeout 10; proxy_read_timeout 10; proxy_pass http://localhost:9000/; } } Decisions Made Here ~~~~~~~~~~~~~~~~~~~ Treating the Django app and the generated static sites as different services. By chance, this VPS comes with two IP addresses, which means I can use different software in port 80. I took advantage of that to treat this as two completely separate products. The reason is that it's possible that it makes more sense to have two actual servers, one for the Django app and one for serving the generated static sites, and I wanted to make uncompromised decisions for each, and to have the chance to try them and tweak them. Running Things -------------- .. sidebar:: About startstop.sh Monit wants everything to have a .pid file where it's process number is stored. Gunicorn can provide that, but Django-rq's rqworker doesn't. Also, I wanted to run those services completely under the "alva" user. So I wrote a wrapper script that works like this:: startstop.sh start|stop NAME USERNAME COMMAND ARG1 ..... ARGN And it will start that command, with the arguments, as that user, and create a PID file with the correct thing in it. I could have written start/stop scripts for each, but why not write only one. The code is a bit embarrasing though, so not showing it. Things should start and stay started. Process should not run as root if not needed. Other than that, I have no strong requirements. Here's the monit configuration (``/etc/monit/conf.d/alva.conf``) , which should be fairly self-explanatory except for the ``startstop.sh`` thing explained in the sidebar:: check process gunicorn with pidfile /home/alva/gunicorn.pid start program = "/bin/startstop.sh start gunicorn alva /usr/bin/writelog /home/alva/gunicorn.log /home/deployer/nikola-server/venv/bin/python /home/deployer/nikola-server/alva/manage.py run_gunicorn --bind=127.0.0.1:9000" stop program = "/bin/startstop.sh stop gunicorn" if failed host 127.0.0.1 port 9000 protocol http then restart check process rqworker with pidfile /home/alva/rqworker.pid start program = "/bin/startstop.sh start rqworker alva /usr/bin/writelog /home/alva/rqworker.log /home/deployer/nikola-server/venv/bin/python /home/deployer/nikola-server/alva/manage.py rqworker" stop program = "/bin/startstop.sh stop rqworker" check process nginx with pidfile /var/run/nginx.pid start program = "/etc/init.d/nginx start" stop program = "/etc/init.d/nginx stop" if failed host 184.82.108.15 port 80 protocol http then restart check process gatling with pidfile /var/run/gatling.pid start program = "/etc/init.d/gatling start" stop program = "/etc/init.d/gatling stop" if failed host 184.82.108.14 port 80 protocol http then restart check process redis with pidfile /var/run/redis.pid start program = "/etc/init.d/redis-server start" stop program = "/etc/init.d/redis-server stop" check exim4 with pidfile /var/run/exim4/exim.pid start program = "/etc/init.d/exim4 start" stop program = "/etc/init.d/exim4 stop" if failed host 127.0.0.1 port 25 then restart check mysql with pidfile /var/run/mysqld/mysqld.pid start program = "/etc/init.d/mysql start" stop program = "/etc/init.d/mysql stop" if failed host 127.0.0.1 port 3306 then restart So, that starts everything I want running, and hopefully will *keep it* running. How To Deploy ------------- .. sidebar:: SSH Security Logins via SSH are all done using separate, secure private/public key pairs. First a bit of CLI action:: ssh -l deployer burst.ralsina.me sh -x deploy.sh Then, via monit, restart gunicorn and rqworker. And that's it. There's still some things that could be done to make it more streamlined but it's good enough at this point.