1. Mikhail Korobov
  2. django-fab-deploy
  3. Issues
Issue #32 resolved

Fabric 1.1 and new-style tasks

Vladimir Mihailenco
created an issue

As I remember we discussed that it would be great to have class-based tasks in django-fab-deploy: http://readthedocs.org/docs/fabric/en/1.1.1/usage/tasks.html#new-style-tasks

What do you think about it? It looks like a great candidate for 1.0/2.0 rewrite :) Currently i see following issues that can be solved with such rewrite:

  • true inheritance

  • library usage (not from CLI). Currently we have following code to accomplish it:

{{{ def do(param1=None): param1 = param1 or env.conf.PARAM1 # such line for every param }}}

which can be simplified:

{{{ @with_not_none_settings def do(**kwargs): env.conf.PARAM1 # is already populated by decorator }}}

We can also extends fabric Task class to get rid of global env variable.

Personally I don't like uppercased settings, but I guess it is insufficient reason :)

  • I believe we have to use signal-based approach. Most useful/interesting signals:

{{{ update_env(env) - can be used to inject variable to env.conf setup(env) - install libraries before_deploy(env) - can be used to stop services etc deploy(env) - actual deploy after_deploy(env, failed=False) - start/restart services }}}

So actual deploy function is simply sending needed signals.

PS As usual I am interested in such modifications and can do some work myself.

Comments (11)

  1. Vladimir Mihailenco reporter

    To give you idea how task will look:

    class PgDump(Task):
        name = 'pg_dump'  # autogenerate?
    
        def update_conf(self, conf):
            if not conf.get('dirname'):
                conf.dirname = conf.env_dir + '/var/backups'
    
        @property
        def filename(self):
            return '%s%s.pgc' % (self.db_name, now)
    
        @property
        def filepath(self):
            return posixpath.join(self.project_dir, self.dirname,  # project_dir = self.conf.project_dir
                                  self.filename)
    
        def cmd(self):
            return 'pg_dump --format=c --file=%(filepath)s %(db_name)s'
    
        def do(self):
            return run(self.cmd % self.conf)
    
  2. Mikhail Korobov repo owner

    What I like about 1.1 tasks:

    • Namespaces. They are great. No more stupid fab_deploy.crontab.crontab_update.
    • Sane way to implement class-based tasks. Somewhere between django-fab-deploy 0.0.1 and 0.1 I actually tried to write class based tasks with some success (there was a hack to export class methods as module methods) but drops the implementation because class based tasks didn't provide much benefit at that time but they increased the codebase significantly.

    The main issue with class-based tasks is that they are not going to magically solve all problems :)

    For example:

    def syncdb(params=''):
        """ Runs syncdb management command. """
        manage('syncdb --noinput %s' % params)
    

    versus

    class Syncdb(Task):
        """ Runs syncdb management command. """
    
        name = 'syncdb'
    
        def run(params=''):
            Manage().run('syncdb --noinput %s' % params)
    

    What problem is solved by such conversion? It seems that replacing all function-based tasks with class-based tasks will only lead to code bloat without much benefits. Standard holywar "inheritance vs callbacks", "OOP vs FP", etc. :)

    The class-based approach can be very useful for high-level commands (e.g. 'deploy').

    The main issue in django-fab-deploy now is how to replace a deployment step without rewriting the rest of tasks. E.g. 'migrate' calls 'mysqldump', 'push' calls 'migrate' and 'deploy' calls 'push', what should we do to make 'deploy' call 'pgdump' instead of 'mysqldump'? Just rewriting all these tasks to classes won't help.

    This issue is now solved by global environment configuration values but if it can be solved better with class-based tasks (or class-based deployment modules, e.g. VCSEngine instead of vcs subpackages) then it'll be great.

    Signals could help, but the issue with signals is that order of execution is often important, that's why the steps are now called explicitly. But they may also help.

    As for env.conf.OPTION_NAME, I thought they are constants and should be uppercased according to pep-0008.

    with_settings_or_none decorator looks useful. Can you please elaborate on why does it require class-based tasks?

    Thanks for bringing this issue and starting the discussion!

  3. Vladimir Mihailenco reporter

    what should we do to make 'deploy' call 'pgdump' instead of 'mysqldump'?

    Ideally we should not have such hardcoded deploy function... I have 4 projects and I don't use original deploy function, because I have to stop some services, restart celery etc. Signals can handle rest:

    • import fab_deploy.postgresql
    • module adds handlers to particular signal
    • ...
    • profit! :)

    And most time I just want to deploy sources, migrate DB and restart apache/celery, but I don't want every time to install/configure Apache/nginx. I think deploy() should just send few signals, but not to intent to do anything. So user can define it's own deploy like this:

    def deploy():
        # every import registers signal handler
        # with every import we select feature we want to use
        from fab_deploy import codebase
        from fab_deploy import apache
        from fab_deploy import django
        fab_deploy.deploy() # send signals
    

    Signals could help, but the issue with signals is that order of execution is often important

    I think it should not be hard to add priorities to signal handlers.

    It seems that replacing all function-based tasks with class-based tasks will only lead to code bloat without much benefits.

    Agree. But I think there will be very few such simple functions.

    with_settings_or_none decorator looks useful. Can you please elaborate on why does it require class-based tasks?

    It does not require class-based tasks, but it requires major rewrites of codebase, because most function now relies on env.conf values. And global env variable looks ugly, that is why I wrap it to self.conf: https://bitbucket.org/vladimir_webdev/django-fab-deploy-ext/src/59d893ee9c96/fab_deploy_ext/postgresql.py#cl-31 . There is no much sense in it, but at least I feel myself little bit happier :) And if I want to use command from python, I can write something like this:

    PgDump(conf=dict(filename='/var/dump.sql')).do()
    

    As for env.conf.OPTION_NAME, I thought they are constants and should be uppercased according to pep-0008.

    Yep, there is nothing wrong with it (most apps including Django have uppercased settings). But if I take my previous example it will look like this:

    PgDump(conf=dict(FILENAME='/var/dump.sql')).do()
    

    Fab-deploy has a lot of configuration options and I think lower-cased settings will make code to look cleaner. Just a thought, nothing more.

  4. Vladimir Mihailenco reporter

    My current attempt lives here: https://bitbucket.org/vladimir_webdev/django-fab-deploy-plus/ . It is backward incompatible in many ways. Some features that you may want to backport:

    • task can be configured in several ways, for example, postgres.Dump task can use var_dir. I can set this var in following way:
    dict(var_dir='/var',
          postgres_var_dir='/var/postgres') # postgres is module name
          postgres_dump_var_dir='/var/postgres/dump') # post is module, dump is task name
    

    This var is accessible via var_dir both in tasks and templates only for Dump task.

    • vars can contain another vars:
    dict(log_dir='%(env_dir)s/log')
    

    I use this in my local python config.

    • I pass host in config as address attribute:
        env.conf = setup_conf(**conf)
        env.hosts = [env.conf.address]
    

    In such way I can configure host from my local config.

    • Other changes are mostly clean ups and using class-based tasks features.
  5. Mikhail Korobov repo owner

    Hi Vladimir,

    I'm very busy right now (startup launch, you know). Hope I'll find some time to look at your repository after a week or two. Thanks for willing to contribute and not just forking!

  6. Mikhail Korobov repo owner

    Hi Vladimir, I'm closing this ticket because it is technically fixed :) All tasks are new-style now and task collections are now class based in django-fab-deploy. Spin-off tickets are #36 and #37.

    I finally came up with a conclusion that it makes more sense to have task collections as classes, not individual tasks. Take a look at https://github.com/fabric/fabric/issues/573 for some reasoning. It seems Jeff also thinks so and fabric may eventually get class-based task collections. In order to get things done at present time django-fab-deploy uses https://github.com/kmike/fabric-taskset as 'task collection engine'.

  7. Log in to comment