Commits

Mikhail Korobov  committed 958f557

Initial import

  • Participants

Comments (0)

Files changed (20)

+syntax: glob
+
+#IDE files
+.settings/*
+.project
+.pydevproject
+.cache/*
+
+#temp files
+*.pyc
+*.pyo
+*.orig
+*~
+
+#misc files
+pip-log.txt
+
+#os files
+.DS_Store
+Thumbs.db
+
+#setup files
+build/
+dist/
+.build/
+MANIFEST
+Copyright (c) 2010, Matt Croydon, Mikhail Korobov
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+    * Neither the name of the tastypie nor the
+      names of its contributors may be used to endorse or promote products
+      derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL MATT CROYDON BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+include *.txt
+include *.rst
+===================================================
+django-fab-deploy: Opionated django deployment tool
+===================================================
+
+django-fab-deploy is intended to be an easy deployment and management
+solution for django projects using mercurial, fabric, virtualenv, pip,
+nginx and apache with mod_wsgi. The supported OS is Debian Lenny.
+
+This software is very opionated. It is a collection of fabric scripts
+that work well together for my projects.
+
+Several projects can be deployed on the same VPS using django-fab-deploy.
+One project can be deployed on several servers. Projects are isolated
+with virtualenv.
+
+Please don't use OpenVZ or Virtuozzo VPS's for deployment! Use XEN or KVM or
+real servers instead. OpenVZ has very serious issues with memory management
+(VIRT is counted and limited instead of RSS or something) so apache (and a
+lot of other software like mysql's InnoDB engine) is totally unusable on
+OpenVZ while being memory-wise and performant on XEN.
+
+Requirements
+============
+
+* python 2.5+
+* your project is stored in mercurial repository
+* Debian Lenny server or VPS with ssh access. I don't have other servers
+  so e.g. Ubuntu is untested but it will possibly work with some small changes.
+* South is used for migrations
+* Fabric trunk (http://github.com/bitprophet/fabric)
+* jinja2
+* Optional: django-compress is used for css and js bundling
+
+I tested the setup only with mysql.
+
+License
+=======
+
+Licensed under a MIT license.
+
+
+What should it be good for
+==========================
+
+1. Deploying several small django sites on one Debian Lenny VPS.
+
+2. Deploying and managing django project with several hg branches that should
+   go to different servers (e.g. staging and production).
+
+3. Repeatable deployments
+
+Getting started
+===============
+
+1. Install django-fab-deploy
+2. Make shure that your project match this structure:
+
+        my_project
+            ...
+            hosting          <- this folder should be copied from django-fab-deploy
+                ...
+            reqs             <- a folder with project's pip requirement files
+                all.txt      <- include all other files in here using pip '-r' syntax
+                active.txt   <- put recently modified apps here
+                ...
+            static           <- static files folder
+                ...
+
+            fabfile.py       <- your project's Fabric deployment script, see below
+            config.py        <- this file should be included in settings.py and ignored in .hgignore
+            config.server.py <- this is a production config template, see below
+            settings.py
+            manage.py
+
+   ``config.py`` trick is also known as ``local_settings.py``.
+
+3. Copy 'hosting' folder from django-fab-deploy to your project and adjust
+   web server configs if it is needed. Basic configs are good starting point and
+   should work as-is. TODO: easier way.
+
+4. Make shure your .hgignore has these lines::
+
+        syntax: regexp
+        ^hosting/generated/(?!noremove)
+        ^hosting/backups/(?!noremove)
+        ^config.py
+
+5. Create fabfile.py. It should provide functions with your server-specific
+   environment variables and you own deployment utils. Example::
+
+        from fabric.api import *
+        from fab_deploy import *
+
+        def stage():
+
+            # how to connect via ssh:
+            env.hosts = ['my-stage-server.com']   # host
+            env.user = 'user'                     # user (must not be root)
+
+            # instance parameters
+            env.conf = dict(
+
+                # distinct instance name
+                INSTANCE_NAME = "my_site",
+
+                # server name. It will be used for web server configs.
+                SERVER_NAME = "my-site.example.com",
+
+                # DB credentials
+                DB_NAME = 'my_site_testing',
+                DB_PASSWORD = '123',
+
+                # apache and mod_wsgi config
+                PROCESSES = 1,
+                THREADS = 5,
+
+                # port should be distinct from other instances' ports
+                APACHE_PORT = 8083,
+
+                # named hg branch that will be active by default
+                HG_BRANCH = 'default',
+
+                # any other parameters. They will be available in config
+                # templates as template variables
+                VERSION = 'STAGING',
+            )
+            update_env()
+
+        def prod():
+            env.hosts = ['my-site.com']
+            env.user = 'user'
+            env.conf = dict(
+
+                # this should be different if stage and production
+                # instances share the same server
+                INSTANCE_NAME = "my_site",
+
+                SERVER_NAME = "my-site.com",
+
+                # DB credentials
+                DB_NAME = 'my_site_production',
+                DB_PASSWORD = '345',
+
+                # apache and mod_wsgi config
+                PROCESSES = 5,
+                THREADS = 15,
+
+                # port should be distinct from other instances'
+                # ports on the same server
+                APACHE_PORT = 8083,
+
+                # named hg branch that will be active by default
+                HG_BRANCH = 'production',
+
+                # any other parameters. They will be available in config
+                # templates as template variables
+                VERSION = 'PROD',
+            )
+            update_env()
+
+        stage() # use stage versions as default
+
+6. Create config.server.py. Example::
+
+        #config file for environment-specific settings
+        DEBUG = False
+        DATABASES = {
+            'default': {
+                'ENGINE': 'django.db.backends.mysql',
+                'NAME': '{{ DB_NAME }}',
+                'USER': 'root',
+                'PASSWORD': '{{ DB_PASSWORD }}',
+                'HOST': '',
+                'PORT': '',
+                'OPTIONS': {
+                    "init_command": "SET storage_engine=INNODB"
+                },
+            }
+        }
+        MEDIA_URL = 'http://{{ SERVER_NAME }}/static/'
+
+
+7. You should be able to run ``fab full_deploy`` from project root now. Run it.
+   'stage' server will be configured: neccessary system and python packages
+   will be installed, apache and ngnix will be configured, virtualenv is
+   created and project is on the server. If you want to deploy on prod server,
+   run ``fab prod full_deploy``.
+
+   Project sources will be available under ``~/src/<project_name>``, virtualenv
+   will be placed in ``~/envs/<project_name>``.
+
+8. TODO: this step should be eliminated.
+   Finish some tasks that were not handled by django-fab-tools:
+
+   a) For now mysql should be installed manually:
+
+        $ aptitude install mysql-server
+
+   b) Then you should create a DB using mysql shell:
+
+        CREATE DATABASE db_name DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci
+
+   c) Then perform the 'syncdb' step on your server:
+
+        $ ./manage syncdb
+
+   d) And then 'migrate' step (from local machine)
+
+        $ fab migrate
+
+   e) Django session tables MUST be MyISAM. If the default engine is InnoDB
+      then the following command should be performed in mysql shell:
+
+        alter table django_session engine=myisam;
+
+   f) Configuring the email server:
+
+        $ dpkg-reconfigure exim4-config
+
+9. You project should be now up and running.
+
+
+Some common tasks (dig into source code for more)
+=================================================
+
+1. Deploy changes on default server:
+
+        $ fab push
+
+2. Deploy changes on another server, update pip requirements and
+   perform migrations:
+
+        $ fab prod push:pip_update,migrate
+
+3. Update requirements specified in reqs/active.txt:
+
+        $ fab pip_update
+
+4. Update requirements specified in reqs/my_apps.txt:
+
+        $ fab pip_update:my_apps
+
+5. Remotely change hg branch
+
+        $ fab up:my_branch
+
+TODO: provide complete list of commands
+

File fab_deploy/__init__.py

+from fabric.api import *
+
+from fab_deploy.deploy import *
+from fab_deploy.commands import touch, mysqldump, pip_install, pip_update, restart_apache
+from fab_deploy.django_commands import migrate, manage, syncdb, compress, test, coverage

File fab_deploy/commands.py

+#coding: utf-8
+from datetime import datetime
+from fabric.api import run, env, local, cd
+from fab_deploy.utils import run_as
+
+@run_as('root')
+def restart_apache():
+    run('/etc/init.d/apache2 restart')
+
+def touch():
+    run('touch %s/hosting/generated/django.wsgi' % env.conf['SRC_DIR'])
+
+def pip_install(what='active', options=''):
+    run('pip install %s -E %s -r %s/reqs/%s.txt' % (options, env.conf['ENV_DIR'], env.conf['SRC_DIR'], what))
+    touch()
+
+def pip_update(what='active', options='', restart=True):
+    run('pip install %s -U -E %s -r %s/reqs/%s.txt' % (options, env.conf['ENV_DIR'], env.conf['SRC_DIR'], what))
+    if restart:
+        touch()
+
+def mysqldump(dir='hosting/backups'):
+    now = datetime.now().strftime("%Y.%m.%d-%H.%M")
+    db = env.conf['DB_NAME']
+    password = env.conf['DB_PASSWORD']
+    with cd(env.conf['SRC_DIR']):
+        run('mysqldump -uroot -p%s %s > %s/%s%s.sql' % (password, db, dir, db, now))
+

File fab_deploy/deploy.py

+#coding: utf-8
+from __future__ import with_statement
+from fabric.api import env, run, local, settings, cd
+from fabric.contrib.files import upload_template
+
+from fab_deploy.utils import run_as, upload_hosting_template
+from fab_deploy.commands import touch, pip_install, pip_update
+from fab_deploy.django_commands import compress, migrate, syncdb, coverage
+from fab_deploy.system import install_system_packages, setup_backports, setup_locale, install_vcs
+
+def full_deploy():
+    install_system_packages()
+    setup_backports()
+    install_vcs()
+
+    make_virtualenv()
+    make_clone()
+    make_hgrc()
+
+    pip_install('all')
+
+    make_wsgi()
+    setup_web_server()
+    update_config()
+
+
+def make_virtualenv():
+    with settings(warn_only=True):
+        run('mkdir %s' % env.conf['ENV_BASE'])
+        run('mkdir %s' % env.conf['SRC_BASE'])
+    with cd(env.conf['ENV_BASE']):
+        run('virtualenv --no-site-packages %s' % env.conf['INSTANCE_NAME'])
+
+
+def push(*args):
+    ''' Run it instead of hg push. '''
+    allowed_args = set(['force', 'notest', 'syncdb', 'migrate', 'pip_update'])
+    for arg in args:
+        if arg not in allowed_args:
+            print 'Invalid argument: %s' % arg
+            print 'Valid arguments are: %s' % allowed_args
+            return
+
+    repo = 'ssh://%s@%s/%s/%s/' % (env.user, env.hosts[0], env.conf['SRC_BASE'], env.conf['INSTANCE_NAME'])
+    local('hg push %s' % repo)
+    with cd(env.conf['SRC_BASE']):
+        with cd(env.conf['INSTANCE_NAME']):
+            output = run('hg up')
+            updated = '0 files updated, 0 files merged, 0 files removed, 0 files unresolved' not in output
+    if updated or 'force' in args:
+        if 'pip_update' in args:
+            pip_update('active_apps', restart=False)
+        if 'syncdb' in args:
+            syncdb()
+        if 'migrate' in args:
+            migrate()
+        compress()
+        touch()
+#        if 'notest' not in args:
+#            coverage()
+
+
+def update_config():
+    upload_template('config.server.py', '%s/config.py' % env.conf['SRC_DIR'], env.conf, True)
+    touch()
+
+
+def up(branch=None):
+    branch = branch or env.conf['HG_BRANCH']
+    with cd(env.conf['SRC_BASE']):
+        with cd(env.conf['INSTANCE_NAME']):
+            run('hg up -C %s' % branch)
+    compress()
+    touch()
+
+
+def make_clone():
+    with cd(env.conf['SRC_BASE']):
+        with settings(warn_only=True):
+            run('mkdir %s' % env.conf['INSTANCE_NAME'])
+            with cd(env.conf['INSTANCE_NAME']):
+                run('hg init')
+    local('hg push ssh://%s@%s/%s/%s/' % (env.user, env.hosts[0], env.conf['SRC_BASE'], env.conf['INSTANCE_NAME']))
+    with cd(env.conf['SRC_BASE']):
+        with cd(env.conf['INSTANCE_NAME']):
+            run('hg up -C %s' % env.conf['HG_BRANCH'])
+    update_config()
+
+
+def make_hgrc():
+    upload_hosting_template('hgrc', '%s/.hg/hgrc' % env.conf['SRC_DIR'])
+
+def make_wsgi():
+    upload_hosting_template('django.wsgi', '%s/hosting/generated/django.wsgi' % env.conf['SRC_DIR'])
+
+
+@run_as('root')
+def setup_web_server():
+    name = env.conf['INSTANCE_NAME']
+
+    upload_hosting_template('apache.config', '/etc/apache2/sites-available/%s' % name)
+    upload_hosting_template('nginx.config', '/etc/nginx/sites-available/%s' % name)
+
+    with settings(warn_only=True):
+        run('ln -s /etc/nginx/sites-available/%s /etc/nginx/sites-enabled/%s' % (name, name))
+        run('rm /etc/nginx/sites-enabled/default')
+        run('rm /etc/apache2/sites-enabled/default')
+    run('a2ensite %s' % name)
+    setup_locale()
+    run('/etc/init.d/nginx restart')

File fab_deploy/django_commands.py

+#coding: utf-8
+from __future__ import with_statement
+from fabric.api import env, cd, local, run
+from fab_deploy.commands import mysqldump
+
+def manage(command, on_server=True):
+    if on_server:
+        with cd(env.conf['SRC_DIR']):
+            run ('%s/bin/python ./manage.py %s' % (env.conf['ENV_DIR'], command))
+    else:
+        local('./manage.py %s' % command)
+
+def migrate(params='', do_backup=True):
+    if do_backup:
+        mysqldump('before-migrate/')
+    manage('migrate --noinput %s' % params)
+
+def syncdb():
+    manage('syncdb --noinput')
+
+def compress():
+    from django.conf import settings
+    if 'compress' in settings.INSTALLED_APPS:
+        manage('synccompress')
+
+def test():
+    pass
+#    with cd(env.conf['SRC_DIR']):
+#        run('source %s/bin/activate; ./bin/runtests.sh' % env.conf['ENV_DIR'])
+
+def coverage():
+    pass
+#    with cd(env.conf['SRC_DIR']):
+#        run('source %s/bin/activate; ./bin/runcoverage.sh' % env.conf['ENV_DIR'])
+

File fab_deploy/system.py

+#coding: utf-8
+from __future__ import with_statement
+from fabric.api import run, settings
+from fabric.contrib.files import append
+from fab_deploy.utils import run_as
+
+@run_as('root')
+def install_system_packages():
+    with settings(warn_only=True):
+        run('aptitude update')
+
+    to_install = [
+        'apache2', 'python2.5', 'memcached', 'mutt', 'nginx',
+        'libjpeg-dev', 'libmysqlclient15-dev', 'zlib1g-dev',
+        'build-essential', 'python-dev', 'python-setuptools',
+        'libapache2-mod-wsgi', 'python-profiler', 'libapache2-mod-rpaf',
+        'screen', 'locales-all'
+    ] # + mysql-server
+
+    run('aptitude install -y %s' % " ".join(to_install))
+    run('easy_install -U pip')
+    run('pip install -U virtualenv')
+
+
+@run_as('root')
+def setup_backports():
+    run("echo 'deb http://www.backports.org/debian lenny-backports main contrib non-free' > /etc/apt/sources.list.d/backports.sources.list")
+    run('wget -O - http://backports.org/debian/archive.key | apt-key add -')
+    with settings(warn_only=True):
+        run('aptitude update')
+
+@run_as('root')
+def install_vcs():
+    run("aptitude -t lenny-backports install -y mercurial")
+    run("aptitude install -y git-core subversion bzr")
+
+
+@run_as('root')
+def setup_locale():
+    append('/etc/apache2/envvars', ['export LANG="en_US.UTF-8"', 'export LC_ALL="en_US.UTF-8"'])
+    run('/etc/init.d/apache2 stop')
+    run('/etc/init.d/apache2 start')
+
+
+#@run_as('root')
+#def install_backup_system():
+#    run('aptitude install -y s3cmd ruby rubygems libxml2-dev libxslt-dev libopenssl-ruby')
+#    run('gem install rubygems-update')
+#    run('/var/lib/gems/1.8/bin/update_rubygems')
+#    run('gem install astrails-safe --source http://gemcutter.org')
+
+#def prepare_backups():
+#    run('mkdir -p backups/before-migrate')
+#
+#    gen_dir = '%s/hosting/backup/generated/' % env.conf['SRC_DIR']
+#    tpl_dir = 'hosting/backup/tpl/'
+#
+#    def gen_template(name):
+#        upload_template(tpl_dir+name, gen_dir+name, env.conf, True)
+#
+#    gen_template('crontab')
+#    gen_template('db.rb')
+#    gen_template('files.rb')
+#    run('crontab -u %s %s' % (env.user, gen_dir+'crontab'))
+
+

File fab_deploy/utils.py

+from functools import wraps
+from fabric.contrib.files import upload_template
+from fabric.api import env
+
+def run_as(user):
+    def decorator(func):
+        @wraps(func)
+        def inner(*args, **kwargs):
+            old_user = env.user
+            env.user = user
+            result = func(*args, **kwargs)
+            env.user = old_user
+            return result
+        return inner
+    return decorator
+
+def upload_hosting_template(name, to):
+    upload_template('./hosting/templates/'+name, to, env.conf, True)
+
+def update_env():
+    HOME_DIR = '/home/%s' % env.user
+    ENV_BASE, SRC_BASE = 'envs', 'src'
+
+    env.conf.update({
+       'HOME_DIR': HOME_DIR,
+       'ENV_BASE': ENV_BASE,
+       'SRC_BASE': SRC_BASE,
+       'ENV_DIR': HOME_DIR + '/' + ENV_BASE + '/' + env.conf['INSTANCE_NAME'],
+       'SRC_DIR': HOME_DIR + '/' + SRC_BASE + '/' + env.conf['INSTANCE_NAME'],
+    })

File hosting/__init__.py

+#coding: utf-8
+

File hosting/backups/before-migrate/noremove

Empty file added.

File hosting/backups/noremove

Empty file added.

File hosting/generated/__init__.py

+#coding: utf-8
+

File hosting/generated/noremove

Empty file added.

File hosting/templates/apache.config

+Listen localhost:{{ APACHE_PORT }}
+<VirtualHost *:{{ APACHE_PORT }}>
+    ServerName {{ SERVER_NAME }}
+    ServerAlias www.{{ SERVER_NAME }}
+    ServerAdmin example@example.com
+
+    WSGIDaemonProcess {{ INSTANCE_NAME }} user=nadovmeste group=nadovmeste processes={{ PROCESSES }} threads={{ THREADS }}
+    WSGIProcessGroup {{ INSTANCE_NAME }}
+
+    WSGIScriptAlias / {{ SRC_DIR }}/hosting/generated/django.wsgi
+    <Directory {{ SRC_DIR }}/hosting/generated>
+        Order deny,allow
+        allow from all
+    </Directory>
+
+    ErrorLog /var/log/apache2/{{ INSTANCE_NAME }}-error.log
+    ErrorDocument 500 {{ SRC_DIR }}/templates/500.html
+
+    # Possible values include: debug, info, notice, warn, error, crit, alert, emerg
+    LogLevel error
+</VirtualHost>

File hosting/templates/django.wsgi

+import os
+import sys
+import site
+
+# prevent errors with 'print' commands
+sys.stdout = sys.stderr
+
+def add_to_path(dirs):
+    # Remember original sys.path.
+    prev_sys_path = list(sys.path)
+
+    # Add each new site-packages directory.
+    for directory in dirs:
+        site.addsitedir(directory)
+
+    # Reorder sys.path so new directories at the front.
+    new_sys_path = []
+    for item in list(sys.path):
+        if item not in prev_sys_path:
+            new_sys_path.append(item)
+            sys.path.remove(item)
+    sys.path[:0] = new_sys_path
+
+add_to_path([
+     os.path.normpath('{{ ENV_DIR }}/lib/python2.5/site-packages'),
+     os.path.normpath('{{ SRC_DIR }}' + '/..'),
+     '{{ SRC_DIR }}'
+])
+
+os.environ['DJANGO_SETTINGS_MODULE'] = '{{ INSTANCE_NAME }}.settings'
+
+#print sys.path
+
+import django.core.handlers.wsgi
+application = django.core.handlers.wsgi.WSGIHandler()

File hosting/templates/hgrc

+# [paths]
+# bitbucket = ssh://hg@bitbucket.org/me/myrepo/
+
+# [hooks]
+# changegroup.push = screen -d -m hg push bitbucket

File hosting/templates/nginx.config

+server {
+    listen 80;
+    server_name {{ SERVER_NAME }} www.{{ SERVER_NAME }};
+    access_log /var/log/nginx/{{ SERVER_NAME }}.access.log;
+    charset utf-8;
+    client_max_body_size 8m;
+
+    gzip_types text/html text/plain text/xml text/css application/javascript application/x-javascript;
+
+    location / {
+        proxy_pass http://localhost:{{ APACHE_PORT }};
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    }
+
+    location /static {
+        root {{ SRC_DIR }};
+        autoindex off;
+        expires 1M;
+    }
+
+    location /media {
+        root {{ ENV_DIR }}/src/django/django/contrib/admin;
+        expires 10m;
+        autoindex off;
+    }
+
+#    location /static/admin_media {
+#        root {{ SRC_DIR }};
+#        autoindex off;
+#        expires 10m;
+#    }
+
+    #error_page  404  /404.html;
+
+    # redirect server error pages to the static page /50x.html
+    error_page   500 502 503 504  /50x.html;
+    location = /50x.html {
+        root   /var/www/nginx-default;
+    }
+}
+#!/usr/bin/env python
+from distutils.core import setup
+
+version='0.0.1'
+
+setup(
+    name='django-fab-deploy',
+    version=version,
+    author='Mikhail Korobov',
+    author_email='kmike84@gmail.com',
+
+    packages=['fab_deploy'],
+
+    url='http://bitbucket.org/kmike/django-fab-deploy/',
+    download_url = 'http://bitbucket.org/kmike/django-fab-deploy/get/tip.zip',
+    license = 'MIT license',
+    description = """ Opionated django deployment tool """,
+
+    long_description = open('README.rst').read(),
+    requires = ['Fabric', 'jinja2', 'South'],
+
+    classifiers=(
+        'Development Status :: 3 - Alpha',
+        'Environment :: Web Environment',
+        'Framework :: Django',
+        'Intended Audience :: Developers',
+        'License :: OSI Approved :: MIT License',
+        'Programming Language :: Python',
+        'Topic :: Software Development :: Libraries :: Python Modules',
+    ),
+)