HTTPS SSH

Tabletop planning

What is this?

This is a django project for site that is used to organize tabletop roleplaying games in club with limited capacity. Usually this project is deployed on https://playhard.kiev.ua/.

Features are simple:

  1. Only russian texts are available! But the project is almost i18n-friendly.
  2. Registration/auth.
  3. Games: create, join, leave, kick.
  4. Calendar (no integrations, very simple email notifications).
  5. Very specific level/experience/cost system based on number of played or mastered games.
  6. Simple statistics.
  7. Finished games report system with moderation: for statistics and for level/cost system.
  8. Simple "news".

Screenshots

calendar Future games list Future games list Game edit

site admin guide

  1. there are many "live settings" than can be configured after admin login: default session duration, maximum number of parallel games, default xp coset, etc. This all can be configured by visiting url "/admin/settings/"
  2. player and maset XP tables can be configured by visiting "/admin/accounting/playerlevelinfo/" and "/admin/accounting/masterlevelinfo/"
  3. game tickets can be created here: "/admin/accounting/ticket/add/"
  4. "Game master" accounts should be confirmed here: "/admin/users/user/"
  5. You can manually create tags here: "/admin/tags/"
  6. tag normalization can be performed by creating "synonyms" objects here: "/admin/tags/tagsynonym/". Example: "Source tag regex"="^d(n|-n-|&|&)d.?4\.?0?(ed.*)?$", "Target tag name"="dnd-4.0". But target tags should be created for this normalization to work

local setup

  1. install needed software. Example for ubuntu:

    sudo apt-get install git python-virtualenv python-pip libxml2-dev libxslt-dev python-dev libsqlite3-dev libjpeg-dev zlib1g-dev node-less postgresql-server-dev-all libmagickwand-dev graphicsmagick-imagemagick-compat python-coverage libssl-dev libffi-dev libmemcached-dev libopenblas-dev npm
    sudo npm install --global jshint
    sudo npm install --global csslint
    
  2. setup virtual environment with virtualenv or virtualenvwrapper

  3. activate virtualenv

  4. install requirements:

    pip install -r requitements_dev.pip
    
  5. Copy default dev settings:

    cp tabletop_planning/settings_dev.example.py tabletop_planning/settings_dev.py
    
  6. open, examine and modify tabletop_planning/settings_dev.py.

  7. init project and run tests:

    python manage.py migrate
    python manage.py test
    python manage.py createsuperuser
    
  8. to enable social login you'll need to set several api keys and secrets in settings. Please do not save them in repository. If you are one of main developers then you can ask imposeren for keys that work on 127.0.0.1:8000

Note

Django-compressor is used and enabled even for development server. It's better to develop with compression enabled as results may differ for compressed and non-compressed static files.

This may cause longer load times but on production compression will lead to faster loading of pages

Deployment

TODO

  • docs are written for those who already know how to deploy with nginx+uwsg: give more details.
  • Deploy procedure have changed but instructions have not: update instructions.
  • example configs are hard tied to specific paths: make paths easier to change.
  • Django settings specific for this project are not documented anywhere: document them.

Install packages the same way as for local setup.

Install ngins-extras, uwsgi and memcached:

sudo apt-get install nginx-extras uwsgi memcached gettext postgresql-client postgresql postgresql-contrib

Required jenkins plugins:

  • Job Cacher plugin
  • AnsiColor
  • Pipeline Utility Steps
  • Bitbucket Plugin
  • Static Analysis Collector
  • Warnings Plug-in
  • Cobertura Plugin

List of required variables:

DJANGO_SA_GOOGLE_KEY='SECRET'
DJANGO_SA_GOOGLE_SECRET='SECRET'

DJANGO_SA_VK_KEY='SECRET'
DJANGO_SA_VK_SECRET='SECRET'

DJANGO_SA_FACEBOOK_KEY='SECRET'
DJANGO_SA_FACEBOOK_SECRET='SECRET'

DJANGO_SETTINGS_MODULE='tabletop_planning.settings_openshift'

EMAIL_USER='EMAIL_USERNAME
EMAIL_PASSWORD='EMAIL_PASSWORD'

LC_ALL='en_US.UTF-8'
LANG='en_US.UTF-8'

If you are deploying with jenkins then DEPLOY_TARGET variable should also be set to directory where you deploy

Databases creation:

createuser -d -P sentry
# enter password
createuser -d -P playhard
# enter password
createdb -T template0 -E utf-8 -l ru_RU.UTF-8 -O sentry sentry
createdb -T template0 -E utf-8 -l ru_RU.UTF-8 -O playhard playhard

Periodic (cron) tasks

On production you should setup some management commands to run periodically. Example usage:

17 *    *  *    *       /bin/run-parts /home/jenkins/revisions/current/deployment/cron/hourly
*/7 *    *  *    *       /bin/run-parts /home/jenkins/revisions/current/deployment/cron/eight_per_hour
35 1    *  *    *       /bin/run-parts /home/jenkins/revisions/current/deployment/cron/daily
* *     *  *    *       /bin/run-parts /home/jenkins/revisions/current/deployment/cron/minutely
45 1    *  *    1       /bin/run-parts /home/jenkins/revisions/current/deployment/cron/weekly
55 1    1  *    *       /bin/run-parts /home/jenkins/revisions/current/deployment/cron/monthly

Note

scripts in this example depend on /home/jenkins/deploy/.extra_env.conf file that in my setup is generated by jenkins and have all required environment variables in it. See some scipt contents for details (e.g. deployment/cron/hourly/regular_acivity.sh)

Project Structure

  1. common static files and template live in: ./tabletop_planning/: static, templates
  2. app static files and templates live in ./{{ app_name }}/: static,``templates``
  3. setting and root url conf are located under tabletop_planning. Some common utilities are located there too.
  4. django 1.7 is used so signal handlers are assigned under some_app.apps.AppConfig_instance.ready method
  5. base template: tabletop_planning/templates/base.html

Dummy mail server

python -m smtpd -n -c DebuggingServer localhost:1025

Backend Guide

  1. Write documentation. Example: this documentation. Also write doctrings for function.

  2. write tests! See games.tests.test_views.AddGameTestCase.

  3. Use model_utils.Choices (https://django-model-utils.readthedocs.org/en/latest/utilities.html#choices) for choices. It's recommended to use PositiveSmallIntegerField for such fields

  4. Use django.shortcuts.render. This will run context processors.

  5. Do NOT use processing for post_save, pre_save signals. Just write this code in model.save() method (Explicit is better than implicit). e.g.:

    def save(self, *args, **kwargs):
         created = False
         if self.pk is None:
             created = True
         if not self.some_field:
             self.some_field = ...
         super(ModelClass, self).save(*args, **kwargs)
         if created:
             self.do_some_processing()
    

    You should use post/pre_save signals only for processing of models in external apps where you can't change code

  6. Use fat models (or forms) and thin views (http://redbeacon.github.io/2014/01/28/Fat-Models-a-Django-Code-Organization-Strategy/)

  7. use QS.select_related for "lists" of objects which need to access FKs

  8. use QS.iterator() if you need to process huge amount of objects once. E.g.

    from django.db.models import Count
    from some_app.models import SomeModel
    from django.db import reset_queries
    
    i = 0
    
    all_qs = SomeModel.objects.all().values_list('some_field', flat=True).annotate(some_count=Count('some_field')).filter(some_count__gt=1)
    
    while True:
        sub_qs = all_qs[i:i+500]
        if not sub_qs.exists():
            break
        for instance in sub_qs.iterator():
            j = 0
            instance.do_some_processing()
        i += 500
        reset_queries()
    
  9. use with open(...) as f: context manager if you work with files.

  10. check pid's life if you are working with external scripts

  11. always use unicode literals instead of plain strings. Use django.utils.encoding.force_unicode if you are not sure what you get as input

  12. Use celery for long running tasks

  13. if you are going to return queryset in some function try not to evaluate it because it may later be filtered again and your evaluation will make unnecesary request to DB. Examples:

    if not queryset:  # this is bad! qs._fetch_all() will be called
        return None
    
    if not queryset.exists(): # this is better because simpler query will hit DB
        return None
    

    exception! If you are going to use query in current function to filter something based on results of this query then it's better to evaluate it earlier. But this will consume more memory so you still should avoid it when thousands of objects are returned. And it's also recommended to get results by values_list(field_name, flat=True) because it will reduce amount of data returned by DB

    some_ids = list(some_qs.values_list('pk', flat=True))
    # ^^^ list() evaluates query and disables paged iteration over it (all results are fetched at once)
    
    if not some_ids:
        return SomeModel.objects.none()
    
    return SomeModel.objects.filter(parent_id__in=some_ids)
    # ^^^ this will use fetched ids directly without nested queries
    
  14. If you need to build cache key that accepts any object (but object still have to support unicode(obj)) with ANY length then use tabletop_planning.utils.get_cache_key. Example:

    cache_key = get_cache_key(u'some_app.views.some_view', target, query)
    cache.set(cache_key, value, 24*60*60)
    

    Warning

    there is method with same name in django.utils.cache module.

Frontend and frontend defined in backend

Custom and frequently used Django template tags

  1. automatically padded image thumbnails:

    {% load common_tags %}
    {% image_block game.image 100 %}
    {% image_block game.image "300x500" %}
    

HTML layout, CSS, JS

  1. Use different CSS classes for styling and for JS bindings. Classes for JS bindings shoud start with "js-"

  2. {% spaceless %} is used in base template. Consider this when you work with <li> elements

  3. if contents of a <div> is longer than 10 lines than add comment with id or class of this div when you close it. Example:

    <div id="demo-1">
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
    </div> {# div#demo-1 #}
    
    <div class="phones-list dim">
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
    </div> {# div.phones-list.dim #}
    
  4. It's better to use some JS linter. JSHint is fine and can be used with SublimeText

  5. Use translations in templates. Example:

    <a href=...>{{ _("Отличное предложение") }}</a>
    
    {% load i18n %}
    {% trans "И это надо перевести тоже!" %}
    
    {% url 'some_view_name' 'arg1_value' arg2 as some_url %}
    {% blocktrans %}
      Привет <a href="{{ some_url }}">всем</a>
    {% endblocktrans %}
    
    {% blocktrans with item_title=item.title %}
        Штука {{ item_title }}!
    {% endblocktrans %}
    
  6. do not use hardcoded urls in HTML or JS. Use {% url %} tag instead.

  7. Do not embedd JS in templates. Use <script type="text/javascript" src="{% static 'js/some_scipt.js' %}"></script> instead. There may be some exceptions (see below).

  8. Do not use hardcoded messages in JS. You can set them in template or you may use javascript catalog (https://docs.djangoproject.com/en/dev/topics/i18n/translation/#module-django.views.i18n).

    Example of js messages in template:

    <script type="text/javascript">
      window.phoneError = "{{ _('Необходимо ввести номер полностью')|escapejs }}";
    </script>
    

    See tabletop_planning/templates/blocks/js_settings.js for examples

  9. Do you have anything to add?

Forms

crispy-forms and django-parsley

It's recommended to use crispy-forms app for all form definitions and django-parsley for form validation on clientside. Please avoid writing html for forms.

Examples:

  • games.forms.GameEditForm.
  • games.forms.GameReportFormset.

More info:

template context variables

While request_context_processors process request you will have several variables in all views' templates contexts:

{{ PROJECT_NAME }}