Mike Fletcher committed 3daa679

Review and tweaking of the entire deployment hierarchy

Comments (0)

Files changed (12)


 Alternate Deployment Patterns
+This document attempts to discuss alternate patterns for deploying
+TurboGears |version|.  It is written with the assumption that you
+have at least read the :ref:`deploy_standard`, as most documents
+will simply discuss the differences from a standard deployment.
 .. note::
    New developers should likely use the :ref:`deploy_standard` (if possible).
    The choices involved in alternate installations can be daunting if you
    aren't yet familiar with the various components.
-Questions to answer for yourself:
-* :ref:`deploy_web_server` -- what application will actually accept
-  the http requests from the client?  This is normally a service that must
-  be installed as "root" in order to claim the standard ports (80 or 443)
-* :ref:`deploy_which_database` -- what server will you use to store your
-  model data?
-* :ref:`deploy_source` -- will you deploy with source-code-checkouts, built
-  eggs, whole-application binary checkouts, zc.buildout packages,
-  PIP packages or puppet scripts?
 .. _deploy_web_server:
 Web-Server Choice
-The most common way to install TurboGears |version| for production use
-is to use the `Apache`_ web-server running the mod_wsgi extension
-to host TurboGears in a reasonably performant easily deployed
-and flexible configuration.
+The web-server, which actually receives and processes HTTP requests
+from clients and converts them to WSGI requests for your TurboGears
+project, can significantly impact the performance and scalability of
+your site.
-There are many other approaches to deploying a production environment,
-and choosing which approach is best for your needs is a non-trivial
-* Apache with mod_wsgi tends to be the "default" choice.  It is widely
-  available, well documented, stable and easily supported by Linux
-  sysadmins in production environments.
-* Apache with mod_proxy or mod_rewrite can be used to have Apache handle
-  "the rest" of your site, while passing only your TurboGears application's
-  requests through to a Paster server running on a non-privileged port.
-* Apache with FastCGI can be used if necessary, such as when you do not have
-  control of your Apache server or a policy requires suexec or the like for
-  all "user" scripts.
-* `Nginx`_ is often preferred by those who need speed above all else
-* The built-in Paster server will occasionablly be used by those
-  who are deploying small internal sites with no more than a handful
-  of users.
-* IIS users may want to experiment with the WSGI support from the
+* :ref:`apache_mod_wsgi` -- the standard way to deploy on Apache
+* :ref:`apache_mod_proxy` -- runs Apache as a front-end with
+  a Paste web-server running on a local port.  Allows you to run
+  the Paste server as *any* user
+* :ref:`FastCGI` -- runs Apache as a front-end with a `FastCGI`
+  process using Mod-Rewrite to make the CGI appear at the correct
+  point in the server's URL-space.
+* `paster serve production.ini` -- while not recommended for large
+  or high-traffic sites, Paste's web-server can often serve for small
+  internal sites with few users.  See :ref:`deploy_daemon` for a
+  discussion of how to keep your server running.
+* :ref:`Nginx` -- an alternative asynchronous high-performance web-server
+  which can reverse-proxy TurboGears
+* :ref:`Light HTTPD <lighttpd_fcgi>` -- has built-in FastCGI support, so can
+  be used to reverse-proxy TurboGears
+* `Twisted Web2`_ -- likely only of interest if you are already using
+  Twisted for your application and simply want to host a TurboGears
+  application within it.  Twisted's WSGI implementation is *not*
+  heavily optimized, so should not be used for high-performance sites.
+* MS-IIS users may want to experiment with the WSGI support from the
   `ISAPI-WSGI`_ project.
 .. todo:: document use of `isapi-wsgi`_ with TurboGears
+.. todo:: Difficulty: Hard. Document use of IIS with TurboGears thru a proxy.
 .. _`Apache`:
-.. _`Nginx`:
 .. _`ISAPI-WSGI`:
 .. _deploy_which_database:
 Database Choice
-Normally users choose either MySQL or PostgreSQL as their production
-database back-end, but Oracle or MSSQL can also be used.  The built-in
-SQLite database should not be used for production sites as a general
-rule.  Obviously if you have used MongoDB/Ming you will need to deploy
-against a MongoDB database instead.
+If you are using SQLAlchemy (the default ORM for TurboGears |version|),
+then by-and-large your choice of database back-end is a matter of preference.
-TurboGears 2 provides a solid HTTP server built in, and for many
-internal corporate deployments or low traffic sites you can just fire
-up the TurboGears |version| app and point people at it.
+* :ref:`deploy_postgresql` -- is a robust, mature, well documented
+  free database server which meets or exceeds most new user's needs.
+* MySQL -- allows you to trade robustness (ACID compliance, for instance)
+  for raw speed and some exotic features that are add-ons for PostgreSQL
+* Oracle -- if your site is an Oracle shop with specialized Oracle admins
+  it may be appropriate to use an Oracle DB for your TurboGears application
+* SQLite -- can be used for *extremely* small sites (e.g. a local web-server
+  intended solely to be used by a single user).  It is *extremely* easy to
+  set up and comes with later versions of Python.
+* MSSQL -- if you are already using MSSQL for your site, and have admins who
+  maintain the servers, it may be appropriate to use MSSQL for your TurboGears
+  application.
-This can be as simple as running::
+.. _`Twisted Web2`:
-  paster serve production.ini
-But it's also likely that you may want to automatically restart your
-TurboGears |version| app if the server reboots, or you may want to set
-it up as a windows service. Unfortunately these things can be very
-operating system specific, but fortunately they aren't
-TurboGears specific.
+.. todo:: Add section on "repeatable deployment options"


    $ sudo -u www-data svn checkout file:///var/svn/myapp/production myapp
    $ cd myapp
    $ sudo -u www-data bash
+   $ mkdir python-eggs
    $ source /usr/local/pythonenv/myapp/bin/activate
    $ python develop
    $ exit
 by default :ref:`modwsgi_deploy <deploy_modwsgi_deploy>` will have specified
-that `production.ini` is in the root directory of this checkout.  See
+that `production.ini` is in the root directory of this checkout. See
 :ref:`deploy_ini_scc` for details on why you might **not** want that file to be
 checked into your main repository.
+Similarly, you will need to make sure that your Beaker session and cache
+directories are not sub-directories of the source code checkout if you
+are planning on deleting and re-checking-out the source for each release.
+(See :ref:`deploy_ini_beaker` for details).


 .. _deploy_code:
-Deploying Your Project
+Deploying Your Project Code
 There are a number of ways you can deploy your application's code in
 production.  While there is an "officially" standard way to deploy


 .. _deploy_db:
-Deploy a Production Database
+Production Database
 Most production sites will use a dedicated database server rather than
 relying on the in-process SQLite engine.  Dedicated servers are generally
    use strong passwords for all accounts, *even* if you only expose the
    DB on a "trusted" port.
+Either PostgreSQL or MySQL is a good default choice for a database server,
+using either one is considered part of a :ref:`deploy_standard` and should
+"just work".
 .. _deploy_postgresql:
-PostgreSQL is a mature, robust, efficient ACID database server.  It
+PostgreSQL is a mature, robust, efficient `ACID`_ database server.  It
 is available for all major platforms, and has GUI administrative tools
-(though almost all "serious" users use the robust command-line tools).
+(though almost all "serious" users use the command-line tools).
+.. _`ACID`:
 PostgreSQL is very well packaged on most Linux distributions, generally
 the packages will automatically create a default `database cluster`
 so that all you need to do is to create a user and a database, then
 configure your application to use that database:
+Create (DB) User and Database
 .. code-block:: bash
     $ sudo aptitude install postgresql
     $ sudo -u postgres createdb --owner=username databasename
 at this point you have a database server and a user account that can
-access (just) the one database you've created.  If you want, you can
+access (just) the one database you've created.
+Test Database Connection
+If you want, you can
 test the database using the command-line psql client from PostgreSQL:
 .. code-block:: bash
     command in psql!  This is a raw connection to the database and
     you are logged in as the owner of the database.
+Alter Production Config
 Once you are satisfied that your database is defined and accessible,
-you alter your production.ini file.  The SQLAlchemy URL should point
-at the database you've created:
+you can alter your :ref:`deploy_ini` file to reference it.  The
+SQLAlchemy URL should point at the database you've created:
 .. code-block:: ini
    check the production.ini file into your development repository, instead
    check it into your configuration-management database (e.g. etckeeper),
    and restrict the file's read permissions as appropriate to allow only the
-   server process to read it.
+   server process (www-data) to read it.
+   See :ref:`deploy_ini_scc`
+Install Driver
 You need to add a PostgreSQL database driver to  your VirtualEnv to
-be able to access the server:
+be able to access the server.
 .. code-block:: bash
     (tg2env)$ easy_install psycopg2
-Now you can initialize your application's database:
+Initialize Database
+Now you can initialize your application's database (see :ref:`deploy_ini` for
+how to create the `production.ini` file):
 .. code-block:: bash
     (tg2env)$ paster setup-app production.ini
     (tg2env)$ paster serve production.ini
 Obviously this is only scratching the surface of PostgreSQL installation
 and maintenance.  For further information:
 .. _`The PostgreSQL Docs`:
-.. todo:: Document setup of MySQL
-.. todo:: Document setup of MongoDB
-.. todo:: Document setup of Oracle (low priority)
-.. todo:: Document setup of MSSQL (low priority)
 What's Next?
 * :ref:`deploy_standard` -- if you are deploying your application, you likely want
   to continue working through the standard deployment pattern
+* :ref:`deploy_which_database` -- discusses how to go about choosing an alternate
+  database engine.
+* :ref:`dbdriverinstall` -- discusses initial setup of database drivers
+.. todo:: Priority high: Document setup of MySQL
+.. todo:: Priority low: Document setup of Oracle
+.. todo:: Priority low: Document setup of MSSQL
+.. todo:: Priority low: Document deployment issues with SQLite
+.. todo:: Priority medium: Document setup of MongoDB/Ming (not here)
+.. todo:: Priority low: Document setup of CouchDB (not here)


 Which one you choose is likely up to your familiarity level with the
 particular tool.
+.. note::
+   For *extremely* small non-critical sites, it can sometimes be expedient
+   to use the `screen` tool to start a `paster serve production.ini`
+   process and then disconnect from the screen.  This isn't recommended,
+   as a power-cycle of the machine will require you to rush back from
+   your vacation to ssh in and re-start the server, but sometimes you
+   do this kind of thing just to get the job done *now*.
 .. todo:: Provide sample init script
 .. todo:: Provide sample upstart
 .. todo:: Provide sample supervisord config


 FastCGI/WSGI -- Running TurboGears |version| behind Apache
-:status: Draft
+..  warning::
+    We recommend that, where possible, you use :ref:`apache_mod_wsgi`
+    as that is part of the :ref:`deploy_standard`
-Under many conditions, native support for Python in the form of mod_wsgi
-or mod_python will not be available.  Furthermore, you will not be able to
-run CherryPy as a webserver, since your webhost insists that everything pass
-through port 80... which is being served by Apache.  Because TurboGears
-implements the WSGIServer interface, we can use flup to interface between
-FastCGI and CherryPy.  We'll also show you how to use virtualenv in this
+FastCGI is an appropriate choice when:
+* mod_wsgi and mod_python are not available
+* you cannot run the Pylons/Paste web-server directly on port 80 (likely
+  because Apache is already running on port 80)
+* mod_fcgi and mod_rewrite *are* available (common)
+Because TurboGears implements the WSGIServer interface, we can use
+`flup`_ to interface between FastCGI and Pylons.  We'll also show you
+how to use :ref:`virtualenv` in this setup.
 This document is closely analogous to _Pylon's instructions for CGI:,+FastCGI+and+mod_rewrite
 but have a number of key differences.
-Apache configuration
+Apache Configuration
 Discussing what Apache directives should be set is beyond the scope of this
 have the latter enabled by default; some require you to explicitly enable
 FastCGI via their control panel (Dreamhost is one such host).
 If you setup your own virtualenv according to the instructions on the
-installation page, you will need to install flup, with::
+installation page, you will need to install `flup`_, with:
-    $ setuptools -i flup
+.. code-block:: bash
+    $ easy_install flup
 If you are not using virtualenv, check to see if flup is installed or not
 with `import flup`.
+.. _`flup`:
 Dispatch scripts
-In the `dispatch.fcgi` file, you will need the following boilerplate code::
+In the `dispatch.fcgi` file, you will need the following boilerplate code:
+.. code-block:: python
     #!/usr/bin/env python
-    turbogears = '/path/to/turbogears'
-    inifile = 'development.ini'
+    turbogears = '/usr/local/turbogears/myapp'
+    inifile = 'production.ini'
     import sys, os
     sys.path.insert(0, turbogears)
     from paste.deploy import loadapp
     RewriteRule   ^(static/.*)$ - [L]
-The first two lines (Options and AddHandler) are not strictly necessary,
-depending on your webserver's configuration.
+The first two lines (Options and AddHandler) may not be strictly necessary,
+depending on your web server's configuration.
-Pylons tweaking
+Proxy Mount Point Fix
 Using this method, Turbogears/Pylons wrongly thinks that dispatch.fcgi
-is a part of the URL. One fix for this known to work is to add this
-Middleware (see config/ to your install::
-    class FastCGIFixMiddleware(object):
-        """Remove dispatch.fcgi from the SCRIPT_NAME
-        mod_rewrite doesn't do a perfect job of hiding it's actions to the
-        underlying script, which causes TurboGears to get confused and tack
-        on dispatch.fcgi when it really shouldn't. This fixes that problem as a
-        Middleware that fiddles with the appropriate environment variable
-        before any processing takes place.
-        """
-        def __init__(self, app, global_conf=None):
-   = app
-        def __call__(self, environ, start_response):
-            environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'].replace('/dispatch.fcgi', '')
-            return, start_response)
-And then, below, in `make_app()`::
-    app = FastCGIFixMiddleware(app, global_conf)
+is a part of the URL. See :ref:`deploy_ini_mountpoint` for how to fix
+this in your production.ini.


 Running TurboGears |version| behind Apache with Mod Proxy
-:status: Draft
+By running your TurboGears |version| application behind
+:ref:`Apache <deploy_apache>` you can take advantage of Apache's
+HTTPS abilities or have it serve your static files, but keep your
+Paste server independent of the Apache server.
-.. contents::
-    :depth: 2
+This can allow, for instance, wsgi applications to be run as
+regular Unix users instead of under the www-data user account.
-By running your TurboGears |version| application behind Apache you
-can take advantage of Apache's HTTPS abilities or have it serve
-your static files.
-Using Apache As A Reverse Proxy
+.. note:: We recommend the use of :ref:`apache_mod_wsgi` where
+   possible, as it is part of the :ref:`deploy_standard` and
+   should provide better performance in general.
 TurboGears Configuration
-If you're mounting your TurboGears |version| app at the root of the
-website, there's nothing you need to do.  But if you're mounting it
-somewhere else, you need to edit production.ini to include these
+..  warning::
+    You will need a :ref:`deploy_ini` for your application.  There
+    are significant security implications to a Production Config file,
+    do **not** just copy your development.ini file!
-  [app:main]
-  use = egg:your_project_name
-  filter-with = proxy-prefix
-  # Usual options here
-  [filter:proxy-prefix]
-  use = egg:PasteDeploy#prefix
-  prefix = /wherever_your app_is mounted
-basically this just tells paster where your app is going to be mounted
-so that it can manage the URL's for you properly.
-.. warning:: You will also want to make sure that you disable the debugger middleware.
-Make sure you have this line in production.ini ::
-   full_stack = False
+If you are not mounting your application at the "root" of your site,
+you will need to configure a proxy filter in your `production.ini` file.
+See :ref:`deploy_ini_mountpoint` for details.
 Apache Configuration
 Here is how to configure Apache 2 as a reverse proxy for your
 TurboGears2 application.
 Now you should be able to see your webpage in full TurboGears glory at
 the address configured as ``ServerName`` above.
 Setting The Correct Charset
 The default templates used by TurboGears specify ``utf-8`` as a
 charset.  The Apache default charset, returned in the ``Content-Type``


 When you are finished, you can continue on to :ref:`deploy_apache_enable`.
 Possible Issues
+Print Statements
+If you have used print statements anywhere in your codebase, you can
+expect your Mod-WSGI applications to crash.  Mod-WSGI will error out
+if there is *any* attempt to write to stdout (which is what print does
+by default).  Use the logging module instead of print throughout
+your codebase.
+Widget Resource Race Condition
 In multiple process load balanced deployments (such as this one) it is
 very possible that a given request will pull resources from multiple


 Your production config file looks much like your `development.ini` file,
 but you will generally need to make a number of changes to make your config
 "production ready".  There are a lot of warnings in this document because
-there are a lot of ways to seriously compromize your security by
+there are a lot of ways to seriously compromise your security by
 mis-handling or mis-configuring a `production.ini` file.
-.. note:
+.. note::
    Throughout this document we'll refer to this file as `production.ini`.
    The file can be named anything you like, and there can be multiple versions,
    such as having `myapp-staging.ini` and `myapp-production.ini` to
 .. warning::
-   You **MUST** set debug=false in your `production.ini`, the Paste
-   web-server in debug mode provides interactive debugging which allows
+   You **MUST** set debug=false in your `production.ini`, Paste
+   in debug mode provides interactive debugging which allows
    any user of the site to run arbitrary Python code!  This must **never**
    happen on a production site (or even a development site which is
-   exposed to the other machines), as it will give any visitor to the
+   exposed to other machines), as it will give any visitor to the
    site complete control of your server.
 In your configuration, you will find a line that looks like this in the
-   # WARGING == If debug is not set to false, you'll get the interactive
+   # WARNING == If debug is not set to false, you'll get the interactive
    # debugger on production, which is a huge security hole.
    debug = false
 You will need to alter the SQLAlchemy database URL to reflect your production
 database.  See :ref:`deploy_db`.
-.. warning:
+.. warning::
    Keep in mind that anyone who has access to
-   this file will now be able to connect to your database. See :ref:`deploy_ini_scc`
+   this file will now be able to connect to your database.  The
+   SQLAlchemy URL includes the username and password to log in as
+   your DB user.
+   See :ref:`deploy_ini_scc`
 Change Your Keys
 These values should not be shared.  See :ref:`deploy_ini_scc`
+.. _deploy_ini_beaker:
 Check File-Storage Locations
-You will likely replace your entire application checkout directory every time
+..  warning::
+    This section does not apply if you are :ref:`deploying your code <deploy_code>`
+    using an :ref:`egg <tgeggdeployment>` (which is the :ref:`deploy_standard`).
+    It likely applies if you :ref:`deploy_checkout`.
+You may be planning to replace your entire application checkout directory every time
 you re-deploy your application, so things such as persistent session-storage,
-and cache directories should be located outside your checkout.
+and cache directories should be located outside your checkout.  By default the
+quick-started application will use %(here)s variables to control where the
+cache and session data is stored.  If your production.ini is in your source-code
+checkout (see :ref:`deploy_ini_scc` for issues with this), this will be a
+directory that will potentially be deleted frequently, and you will need to
+specify an alternate location.
-If you are deploying with an egg, you can continue to use %(here)s in the config
-to reference the "deployment directory" (where you production.ini file is),
-and this will be outside the egg's path.  (This is the :ref:`deploy_standard`)
-If you are using a source-code-checkout into the deployment location, the
-appropriate location is somewhat open to sysadmin preference, but a good
-default choice would be `/var/local/myappname`, which would create config
-lines like this:
+The appropriate location for application data-storage is somewhat open to
+sysadmin preference, but a good default choice would be
+`/var/local/myappname`, which would require config lines like this:
 .. code-block:: ini
 See :ref:`caching` and :ref:`session` for discussions of the Beaker system
 along with alternative deployment options, such as the use of :ref:`memcache`.
-See :ref:`deploy_code` for dicussions of how to deploy
+See :ref:`deploy_code`.
 Check Log-file Options
 See :ref:`config_logging` for more details.
+.. _deploy_ini_mountpoint:
+Configure Proxy Mount Point
+..  warning:: This section **only** applies to "proxied" sites, which are
+    *not* part of the :ref:`deploy_standard`.
+If you are **not** mounting your application at the root of your site
+(i.e. you are mounting your application as a sub-site of some larger site)
+**and** are using a non-embedded WSGI environment (such as a reverse proxy)
+then you will need to configure TurboGears so that it knows how to
+resolve application URLs from that base URL.
+.. code-block:: ini
+  [app:main]
+  ...
+  filter-with = proxy-prefix
+  [filter:proxy-prefix]
+  use = egg:PasteDeploy#prefix
+  prefix = /wherever_your_app_is mounted
+See the `PasteDeploy Documentation`_ for details on the prefix middleware
+being configured here.
+.. _`PasteDeploy documentation`:
 Test your Config
 .. code-block:: bash
-   $ sudo -u paster server production.ini
+   $ sudo -u www-data server production.ini
 .. _deploy_ini_scc:
 Check In Your Config
-.. warning:
+.. warning::
    Your `production.ini` contains secrets, keys, passwords, and everything
    else an attacker would need to crack your application and potentially


 it can be run and this documentation only begins to cover the
 available approaches.
-Standard Deployments
 .. toctree::
    :maxdepth: 1
-Alternative Deployments
 These sections describe non-standard approaches to deployment.  You
 should not likely use these unless you are comfortable with web
 development and deployment or you have some particular need which
    :maxdepth: 1
+   Checkout
-   Checkout
-.. todo:: Difficulty: Hard. Document use of IIS with TurboGears thru a proxy.
+.. todo:: Document processes for repeatable local-only releases: Local PyPI,
+   PIP, recordeggs, whole-virtualenv checkin/checkout.


 Lighttpd and FastCGI
-.. highlight:: bash
+.. todo:: clean up the LightHTTPD + FastCGI deployment documentation
+.. todo:: consolidate the 3 FastCGI documents (Mod-FastCGI, NGINX FastCGI
+   and LightHTTPD FastCGI) as well as the Mod-Proxy stuff.
 Lighttpd has strong build-in FastCGI support. This makes FCGI the method of choice to deploy a TurboGears2 application in a production environment.
 This is exactly the workload expected for a webserver.
 If you want to use more than one physical core, start more python processes.
-.. highlight:: python
 You need a script to start the FastCGI server with your application.
 Additionally you have to load the paths to your virtual environment.
     filter-with = proxy-prefix
 And as an additional section::
     use = egg:PasteDeploy#prefix
     prefix = /
 This will force the URL transmitted from Lighttpd to TurboGears to "/".
 Reload lighttpd to enable the changes. You should now have a process named "" with several threads.


 NGINX Web Server
-Nginx is a very fast asynchronous web server that can be used in front
-of TurboGears |version| in very high load environments.
+Nginx is a very fast asynchronous web server.  This means that it
+handles all IO using non-blocking sockets rather than threads or
+processes, which allows it to scale to extremely large numbers of
+connected clients (on the order of 10,000 simultaneous clients).
-   .. toctree::
-      :maxdepth: 1
+Nginx support for WSGI applications (and TurboGears in particular)
+is still very much experimental, but the following patterns may
-      load_balance.rst
+* can provide :ref:`reverse-proxy/load-balancing<nginx_load_balance>`
+  for multiple Paste web-servers
+* TurboGears should be compatible with `uWSGI`_, which should be
+  compatible with Nginx, (this is a reverse-proxy setup as well)
+* has `FastCGI support`_ which, with some effort likely can be used
+  to host TurboGears |version|
-Another alternative that has yet to be explored by the doc crew is
-`uWSGI <>`_.
+.. toctree::
+   :maxdepth: 1
+   load_balance.rst
+.. _`FastCGI support`:
+.. _`uWSGI`:
+.. todo:: Need to test and document these options better
+   if we're going to keep them in the official documentation.