1. Nate Aune
  2. silverlining


Ian Bicking  committed 9b2007a Merge


Comments (0)

Files changed (32)

File docs/conf.py Modified

View file
  • Ignore whitespace
  • Hide word diff
 html_theme = 'nature'
 html_theme_path = ['_theme']
-html_favicon = 'favicon.ico'
+#html_favicon = 'favicon.ico'
 # The style sheet to use for HTML and HTML Help pages. A file of that name
 # must exist either in Sphinx' static/ path, or in one of the custom paths

File docs/disabling-sites.txt Added

View file
  • Ignore whitespace
  • Hide word diff
+.. -*- mode: rst -*-
+ Disabling Sites
+When hosting a site it's likely that from time to time it will be
+necessary to make it unavailable to the general public while some
+maintenance is carried out.  Silver Lining provides a mechanism for
+allowing this.  It is important to recognise that it is *applications*
+that are disabled via this mechanism, rather than locations, so if the
+same application is accessible via two different locations (even if it
+is two different *deployments* of the application), and it is
+disabled, then it will appear to be disabled at both locations.  This
+is because if your application uses a database, then all different
+deployments of the application will share the database, and cleaning
+up databases is one of the key use-cases for this feature.
+Disabling an application
+The following command will disable an application called blog::
+    $ silver disable --by-name blog
+You can also disable the application by giving a path that it appears
+    silver disable --by-location www.example.com/blog
+Enabling an application
+To re-enable an application you use the ``silver enable`` command,
+which takes the same options as silver disable command.
+The disabled special location
+When an application is disabled, visitors to it will be shown a page
+explaining that the site is unavailable (similarly there is a
+not-found location which is shown when a URL is not resolved).  It is
+possible to replace this page by uploading a new WSGI application to
+the special location "disabled".  To restore the default application,
+use the ``silver activate`` command to point the disabled location to
+Headers for disabled apps
+It is sensible to return a "503 Service Unavailable" response from a
+disabled app and use the following headers:
+:Cache-Control: no-store, no-cache, max-age=0
+:Pragma: no-cache
+Access To disabled apps
+While an application is disabled, it is still possible to access it
+via the command line tools.  ``silver backup``, ``restore``, ``run``
+etc. will all work as usual.  It's also possible to access the site
+through a web browser.  At present the application is only made
+available to clients connecting from localhost, so to view the site
+and access management facilities you will probably need to set up an
+SSH tunnel and manually update ``/etc/hosts`` on your own machine.

File docs/index.txt Modified

View file
  • Ignore whitespace
  • Hide word diff
+    disabling-sites

File docs/todo.txt Modified

View file
  • Ignore whitespace
  • Hide word diff
 <http://bitbucket.org/ianb/silverlog>`_).  Though I feel like some
 kind of management programming interface is better, and then a
 "deployed" app would use that interface.  Actually, "interface" is
-probably saying too much; documented and conscious server layout that
-an application can be written to inspect.
+probably saying too much; it should be sufficient to simply have a
+documented and conscious server layout that an application can be
+written to inspect.
 There should be a way to get a list of all log files for a host/app.
 Maybe there should be a standard way to effect the Python standard
 some indication of how often the pinging should happen (i.e., if the
 ping is cheap do it often, if it's expensive be more reserved).
+CloudKick has started doing a bunch of monitoring things, and has an
+API.  A quick way of doing monitoring might simply be via CloudKick
+(or maybe just an option).
 update, but before the update was really made live (the internal
 request mechanism can be run on any instance, even one that is not
 live yet).  If this returned a non-2xx result, then the update would
-be aborted.
+be aborted (or if it's a test script, exit with an error code).
 Backup/Restore/Revert on update
 I think this should be run before test-before-activate, but if
 test-before-activate fails then the databases should also be reverted.
+There are fancy ways of doing really fast checkpoints, like backups
+without actually backing things up.  These should be accessible even
+in a VPS, but they require some fancy sysadmin knowledge that I don't
 Cron is a higher priority.
-I'm not even sure what this *means*, but I know it needs to be
-implemented in some way.  I'm kind of holding off because I don't want
-apps/servers to get configured a lot.  But for reusable application
-packages *something* has to be implemented.  And I suspect there are
-places where this is really sensible.
-I will probably resist this until it is clear what it should mean,
-which means an actual circumstance where it is needed arises.
+For persistence of the jobs, Redis seems quite possible (small enough
+that I don't feel bad having it always installed, seems relatively
+robust and easy to use).
 API Keys
 This is something that might be good to manage organization-wide.
 Which just makes the workflow slightly different.
+In some ways this is just global configuration (as opposed to
+application-specific configuration).  This same configuration could
+potentially be used for things like admin email addresses, logging
+information, etc.  Applications would have to pull it in, but it would
+be updated on a machine basis, not a per-application basis (though
+possibly you could have per-application overrides?)
 Inter-application communication
 It feels a bit crude.  Maybe another command besides ``silver`` for
 pre-packaged apps?  Or another set of subcommands.
+Middleware Packs
+Sometimes you want to apply middleware in a deployment-specific
+fashion (not application-specific).  Some examples:
+* Site disabling
+* Error catching
+* Password protecting a site before it is public
+* Ad hoc authorization
+I'd like the ability to apply a piece of middleware to an
+application.  This wouldn't *just* be WSGI middleware, it would
+probably include some extra information (description, etc).  Also a
+PHP equivalent version ("fat" middleware) would be nice to include.
+Using PHP's output buffer operations you can simulate middleware.
+I'm not sure what the command would look like.  Maybe just::
+  silver apply-middleware path/to/middleware LOCATION
+There's also be ``show-middleware`` and ``remove-middleware``
+operations, I suppose.  Like applications, there's a missing notion of
+versions here.  Also middleware would have to be compatible with the
+libraries of the hosted application, as they would run in the same
+process.  Probably middleware libraries would be added after the path,
+and the middleware should be written to be as library-version-agnostic
+as possible.  There should be a fairly finite number of middleware
+Configuration should apply to middleware, though maybe a second set of
+configuration to avoid overlap.
+Configuration Management
+Now that there's a ``--config`` option you can have
+deployment-specific configuration of your application, but it's only
+exposed through ``silver update``.  It should be exposed as a separate
+operation, to view, edit, backup, etc. the configuration, you
+shouldn't have to deploy just to change the configuration.  A complete
+set of commands might be:
+Show the configuration (all past revisions)::
+  silver config-query LOCATION
+  # To show info about files:
+  silver config-query LOCATION --files
+  # To download config:
+  silver config-query LOCATION --dump=path/
+Then operations to actually modify the configuration::
+  # Upload new configuration:
+  silver config LOCATION path/to/config
+  # Revert configuration to some previous version ("PREV" literal, or
+  # some revision number as shown by config-query):
+  silver config LOCATION --revert VERSION
+  # Remove configuration:
+  silver config LOCATION --delete
+  # Copy configuration from another site:
+  silver config LOCATION --copy=SOURCE_LOCATION
+Doing version control on the configuration would be nice.  Also
+supporting an app that did generic through-the-web application
+configuration (needs a DevAuth-style thing again).  Such configuration
+would just do through-the-web-editing of the source files, and
+probably call the validation function if available.  It might also
+allow for operations like reverting, seeing logs, etc.
 Application Versions
 You can use `mod_negotiation
 <http://httpd.apache.org/docs/2.2/mod/mod_negotiation.html>`_ to serve
-static files without extensions.
+static files without extensions.  These are files that include headers
+(that indicate things like Content-Type) as well as the body.
+Probably they'd have a different extension.
 Security Headers
 *might* want to enable it globally (not as part of the application).
 Specifically for static file headers.
+Maybe a middleware pack would handle this case.
 Short-term list

File silverlining/commands/create_node.py Modified

View file
  • Ignore whitespace
  • Hide word diff
-            files={'/root/.ssh/authorized_keys': config.get('root_authorized_keys')},
+            ex_files={'/root/.ssh/authorized_keys': config.get('root_authorized_keys')},
         public_ip = resp.public_ip[0]
         config.logger.notify('Status %s at IP %s' % (

File silverlining/commands/deactivate.py Modified

View file
  • Ignore whitespace
  • Hide word diff
 def command_deactivate(config):
     if not config.args.node:
         config.args.node = appdata.normalize_location(config.args.locations[0])[0]
-    if config.args.disable:
-        for location in config.args.locations:
-            ssh('www-mgr', config.args.host,
-                '/usr/local/share/silverlining/mgr-scripts/activate-instance.py %s disabled'
-                % location)
+    if config.args.keep_prev:
+        option = ['--keep-prev']
-        if config.args.keep_prev:
-            option = ['--keep-prev']
-        else:
-            option = []
-        ssh('www-mgr', config.node_hostname,
-            ['/usr/local/share/silverlining/mgr-scripts/remove-host.py']
-            + option + config.args.locations)
+        option = []
+    ssh('www-mgr', config.node_hostname,
+        ['/usr/local/share/silverlining/mgr-scripts/remove-host.py']
+        + option + config.args.locations)

File silverlining/commands/disable.py Added

View file
  • Ignore whitespace
  • Hide word diff
+from cmdutils import CommandError
+from silversupport.appdata import normalize_location
+from silversupport.shell import ssh
+def command_disable(config, enable=False):
+    node, location, appname = None, None, None
+    if config.args.by_name is not None:
+        appname = config.args.by_name
+    if config.args.by_location is not None:
+        location = config.args.by_location
+        node, _path = normalize_location(location)
+    if config.args.node is not None:
+        node = config.node_hostname
+    if location and not appname:
+        stdout, _stderr, _returncode = ssh(
+            'www-mgr', node,
+            '/usr/local/share/silverlining/mgr-scripts/get-application.py',
+            fail_on_error=True)
+        appname = stdout.strip()
+    if node is None or appname is None:
+        raise CommandError("Unable to determine target node or appname.")
+    ssh('www-mgr', node,
+        '/usr/local/share/silverlining/mgr-scripts/disable.py %s %s'
+        % ('--enable' if enable else '', appname))

File silverlining/commands/enable.py Added

View file
  • Ignore whitespace
  • Hide word diff
+from silverlining.commands import disable
+def command_enable(config):
+    return disable.command_disable(config, enable=True)

File silverlining/commands/query.py Modified

View file
  • Ignore whitespace
  • Hide word diff
 def command_query(config):
     stdout, stderr, returncode = ssh(
         'www-mgr', config.node_hostname,
-        'cat /var/www/appdata.map; echo "END" ; ls /var/www',
+        'cat /var/www/appdata.map; echo "END"; '
+        'cat /var/www/disabledapps.txt; echo END; '
+        'ls /var/www',
     hosts = {}
     lines = [line.strip()
     site_instances = {}
     instance_site = {}
     sites = set()
+    disabled = set()
+    # parse appdata.map
     while 1:
         if not lines:
         hostname, path, data = line.split(None, 2)
         instance_name = data.split('|')[0]
-        hosts[hostname+path] = instance_name
+        hosts[hostname + path] = instance_name
+    # parse disabledsites.txt
+    while 1:
+        if not lines:
+            break
+        line = lines.pop(0)
+        if line == 'END':
+            break
+        disabled.add(line)
+    # parse directory listing
     for line in lines:
-        if line == 'appdata.map':
+        if line == 'appdata.map' or line == 'disabledapps.txt':
         match = re.match(r'^(?:([a-z0-9_.-]+)\.(.*)|default-[a-z]+)$',
     for hostname, site in hosts.items():
         if site in ('disabled', 'notfound'):
             special_hosts.setdefault(site, []).append(hostname)
+        elif site in disabled:
+            special_hosts.setdefault('disabled', []).append(hostname)
     for site in sorted(sites):
         if len(sites) > 1:
             notify('Site: %s' % site)

File silverlining/commands/run.py Modified

View file
  • Ignore whitespace
  • Hide word diff
 def command_run(config):
+    if not hasattr(config.args, 'unknown_args'):
+        raise CommandError("You may not place any arguments before 'run'")
     args = config.args
     out = StringIO()
     zip = zipfile.ZipFile(out, 'w')

File silverlining/commands/serve.py Modified

View file
  • Ignore whitespace
  • Hide word diff
 def serve_python(config, appconfig):
-    dir = config.args.dir
+    dir = os.path.abspath(config.args.dir)
     if os.path.exists(os.path.join(dir, 'bin', 'python')):
         # We are in a virtualenv situation...
         cmd = [os.path.join(dir, 'bin', 'python'),
             while 1:
-                proc = subprocess.Popen(cmd, cwd=dir, env=environ)
+                try:
+                    proc = subprocess.Popen(cmd, cwd=dir, env=environ)
+                except:
+                    config.logger.warn('Error running command: %s' % ' '.join(cmd))
+                    raise
                 if proc.returncode == 3:
                     # Signal to do a restart
     ## FIXME: -X would also be an alternative to -DFOREGROUND; not sure which is better
     run([exe_name, '-f', conf_file,
          '-d', config.args.dir, '-DFOREGROUND'])
 def _turn_sigterm_into_systemexit():

File silverlining/commands/setup_node.py Modified

View file
  • Ignore whitespace
  • Hide word diff
 def setup_rsync(config, source, dest, delete=False):
     cwd = os.path.abspath(os.path.join(__file__, '../..'))
-    options = ['--quiet', '-prvC']
+    options = ['--quiet', '-ErvC']
     if delete:
     stdout, stderr, returncode = run(

File silverlining/config.py Modified

View file
  • Ignore whitespace
  • Hide word diff
                 DriverClass = libcloud_get_driver(getattr(Provider, provider.upper()))
             self._driver = DriverClass(self['username'], self['secret'])
+            if getattr(self.args, 'debug_libcloud', False):
+                print 'XXX', self.args.debug_libcloud, self._driver.connection.conn_classes
+                from libcloud.base import LoggingHTTPConnection, LoggingHTTPSConnection
+                fp = open(self.args.debug_libcloud, 'a')
+                LoggingHTTPConnection.log = LoggingHTTPSConnection.log = fp
+                self._driver.connection.conn_classes = (
+                    LoggingHTTPConnection, LoggingHTTPSConnection)
         return self._driver

File silverlining/mgr-scripts/disable.py Added

View file
  • Ignore whitespace
  • Hide word diff
+#!/usr/bin/env python
+import sys
+sys.path.insert(0, '/usr/local/share/silverlining/lib')
+import optparse
+from silversupport import disabledapps
+from silversupport import appdata
+parser = optparse.OptionParser(
+    usage='%prog application')
+    '--enable',
+    action='store_true',
+    help='renable a previously disabled application')
+def main():
+    options, args = parser.parse_args()
+    application, = args
+    in_use = any(True for instance, uses in appdata.all_app_instances().items()
+                 if instance.startswith(application + '.') and uses)
+    if not in_use:
+        sys.stderr.write("%s is not in use in appdata.map\n" % application)
+        return 1
+    if options.enable:
+        if not disabledapps.is_disabled(application):
+            sys.stderr.write("%s is not disabled\n" % application)
+            return 1
+        disabledapps.enable_application(application)
+    else:
+        if disabledapps.is_disabled(application):
+            sys.stderr.write("%s is already disabled\n" % application)
+            return 1
+        disabledapps.disable_application(application)
+    return 0
+if __name__ == '__main__':
+    sys.exit(main())

File silverlining/mgr-scripts/get-application.py Added

View file
  • Ignore whitespace
  • Hide word diff
+#!/usr/bin/env python
+"""Find the application for a location"""
+import sys
+sys.path.insert(0, '/usr/local/share/silverlining/lib')
+from silversupport import appdata
+def main():
+    hostname, path = appdata.normalize_location(sys.argv[1])
+    instance = appdata.instance_for_location(hostname, path)
+    appname = instance.split('.')[0]
+    print appname
+    return 0
+if __name__ == '__main__':
+    sys.exit(main())

File silverlining/runner.py Modified

View file
  • Ignore whitespace
  • Hide word diff
 parser = argparse.ArgumentParser(
-## FIXME: these options should also be available in the subparsers:
     '-p', '--provider',
     help="Answer yes to any questions")
+    '--debug-libcloud', metavar='FILENAME',
+    help="Write any libcloud interactions (HTTP request log)")
 add_verbose(parser, add_log=True)
 subcommands = parser.add_subparsers(dest="command")
-    default='name *karmic*',
+    default='name *lucid*',
     help='Image to use, can be "id 10", and can contain wildcards like "name *karmic*".  '
     'Default is "name *karmic*" which will select an Ubuntu Karmic image.')
     "they must all be on the same node.  Can be a wildcard.")
-    '--disable', action='store_true',
-    help="Set the host to the status disabled, pointing it at the disabled application (good for a temporary removal)")
     '--keep-prev', action='store_true',
     help="Keep the prev.* host activate (by default it is deleted)")
     '--info', action='store_true',
     help="Show information about how the configuration is created")
+parser_disable = subcommands.add_parser(
+    'disable', help="Temporarily disable an application")
+parser_enable = subcommands.add_parser(
+    'enable', help="Re-enable a disabled application")
+for subparser in (parser_disable, parser_enable):
+    group = subparser.add_mutually_exclusive_group(required=True)
+    group.add_argument('--by-name', metavar="APPNAME",
+                       help="Identify the application by its name")
+    group.add_argument('--by-location', metavar="LOCATION",
+                       help="Identify the application by its location")
+    subparser.add_argument('--node', metavar="NODE_HOSTNAME",
+                           help="Node to act on")
 for subparser in subcommands._name_parser_map.values():
-    ## FIXME: these options should also be available in the subparsers:
         '-p', '--provider',

File silverlining/server-root/etc/ufw/ufw.conf Added

View file
  • Ignore whitespace
  • Hide word diff
+# /etc/ufw/ufw.conf
+# set to yes to start on boot
+# set to one of 'off', 'low', 'medium', 'high'

File silverlining/server-root/etc/varnish/default.vcl Modified

View file
  • Ignore whitespace
  • Hide word diff
     remove req.http.X-Forwarded-For;
     set req.http.X-Forwarded-For = client.ip;
     if (req.request == "POST") {
-        pass;
+        return(pass);
     if (req.request != "GET" && req.request != "HEAD" &&
         req.request != "PUT" && req.request != "POST" &&
         req.request != "DELETE") {
         # Non-RFC2616 or CONNECT which is weird. #
-        pass;
+        return(pass);
     if (req.http.Authorization) {
         # Not cacheable by default #
-        pass;
+        return(pass);
 sub vcl_fetch {
-    if(obj.http.Pragma ~ "no-cache" ||
-       obj.http.Cache-Control ~ "no-cache" ||
-       obj.http.Cache-Control ~ "private") {
-            pass;
+    if(beresp.http.Pragma ~ "no-cache" ||
+       beresp.http.Cache-Control ~ "no-cache" ||
+       beresp.http.Cache-Control ~ "private") {
+            return(pass);
-    if (obj.status >= 300) {
-        pass;
+    if (beresp.status >= 300) {
+        return(pass);
     # Django regularly sends pages with Set-Cookie and cache control, 
     # we'll ignore Cache-Control in that case, as there's no point to
     # caching something that sets a cookie.
-    if (obj.http.Set-Cookie) {
-        unset obj.http.Cache-Control;
-        pass;
+    if (beresp.http.Set-Cookie) {
+        unset beresp.http.Cache-Control;
+        return(pass);
-    if (obj.http.Cache-Control ~ "max-age" || obj.http.Expires) {
-        unset obj.http.Set-Cookie;
-        deliver;
+    if (beresp.http.Cache-Control ~ "max-age" || beresp.http.Expires) {
+        unset beresp.http.Set-Cookie;
+        return(deliver);
-    pass;
+    return(pass);
 sub vcl_hit {
     if (!obj.cacheable) {
-        pass;
+        return(pass);
     if (req.http.Cache-Control ~ "no-cache") {
             return (restart);
-    deliver;
+    return(deliver);

File silverlining/server-root/lib/ufw/user.rules Added

View file
  • Ignore whitespace
  • Hide word diff
+:ufw-user-output - [0:0]
+:ufw-user-forward - [0:0]
+:ufw-user-limit - [0:0]
+:ufw-user-limit-accept - [0:0]
+### RULES ###
+### tuple ### allow any 22 any in
+-A ufw-user-input -p tcp --dport 22 -j ACCEPT
+-A ufw-user-input -p udp --dport 22 -j ACCEPT
+### tuple ### allow tcp 80 any in
+-A ufw-user-input -p tcp --dport 80 -j ACCEPT
+### END RULES ###
+-A ufw-user-limit -m limit --limit 3/minute -j LOG --log-prefix "[UFW LIMIT BLOCK] "
+-A ufw-user-limit -j REJECT
+-A ufw-user-limit-accept -j ACCEPT

File silverlining/server-sync-scripts/dpkg-query.txt Modified

View file
  • Ignore whitespace
  • Hide word diff
 libblas3gf	1.2-2
 libc6-dev	2.9-4ubuntu6
 libexpat1	2.0.1-4
-libgdal1-1.5.0	1.5.2-3ubuntu1
 libgfortran3	4.3.3-5ubuntu4
 libgif4	4.1.6-6
 libgmp3c2	2:4.2.4+dfsg-2ubuntu1
 libgomp1	4.3.3-5ubuntu4
-libhdf4g	4.1r4-22
-libhdf5-serial-1.6.6-0	1.6.6-4ubuntu1
 libjasper1	1.900.1-5.1
 libjpeg62	6b-14
 liblapack3gf	3.1.1-6
 libltdl7	2.2.6a-1ubuntu1
 libmpfr1ldbl	2.4.0-1ubuntu3
-libmysqlclient15off	5.1.30really5.0.75-0ubuntu10
 libneon27-gnutls	0.28.2-6.1
 libnetcdf4	1:3.6.2-3.1
 libogdi3.2	3.2.0~beta1-3.1
 xml-core	0.12
 zip	2.32-1

File silverlining/server-sync-scripts/update-from-server.sh Modified

View file
  • Ignore whitespace
  • Hide word diff
 rsync root@$SERVER:/var/www/README.txt www-README.txt
 echo rsync root@$SERVER:/etc/init.d/silverlining-setup silverlining-setup
 rsync root@$SERVER:/etc/init.d/silverlining-setup silverlining-setup
-echo rsync root@$SERVER:/etc/postgresql/8.3/main/pg_hba.conf pg_hba.conf
-rsync root@$SERVER:/etc/postgresql/8.3/main/pg_hba.conf pg_hba.conf
 echo ssh root@$SERVER '"dpkg-query -W" >' dpkg-query.txt
 ssh root@$SERVER "dpkg-query -W" > dpkg-query.txt

File silverlining/server-sync-scripts/update-server-script.sh Modified

View file
  • Ignore whitespace
  • Hide word diff
 disabled / default-disabled|general_debug|/dev/null|python|
 " > /var/www/appdata.map
-chown www-mgr:www-mgr /var/www/appdata.map
+touch /var/www/disabledapps.txt
+chown www-mgr:www-mgr /var/www/appdata.map /var/www/disabledapps.txt
 ## Now setup Apache.  Ubuntu installs 000-default, which we don't
 ## want, so we delete it, and make sure the necessary modules are

File silversupport/appconfig.py Modified

View file
  • Ignore whitespace
  • Hide word diff
 from silversupport.env import is_production
 from silversupport.shell import run
 from silversupport.util import asbool, read_config
+from silversupport.disabledapps import DisabledSite, is_disabled
 __all__ = ['AppConfig']
 class AppConfig(object):
     """This represents an application's configuration file, and by
     def from_instance_name(cls, instance_name):
         """Loads an instance given its name; only valid in production"""
-        return cls(os.path.join('/var/www', instance_name, 'app.ini'))
+        return cls(os.path.join(DEPLOYMENT_LOCATION, instance_name, 'app.ini'))
     def from_location(cls, location):
             execfile(sitecustomize, ns)
     def get_app_from_runner(self):
-        """Returns the WSGI app that the runner indicates"""
+        """Returns the WSGI app that the runner indicates
+        """
         assert self.platform == 'python', (
             "get_app_from_runner() shouldn't be run on an app with the platform %s"
             % self.platform)
             runner = 'config:%s' % runner
             global_conf = os.environ.copy()
             global_conf['SECRET'] = get_secret()
-            return loadapp(runner, name=spec,
-                           global_conf=global_conf)
+            app = loadapp(runner, name=spec,
+                          global_conf=global_conf)
         elif runner.endswith('.py'):
             ## FIXME: not sure what name to give it
             ns = {'__file__': runner, '__name__': 'main_py'}
             execfile(runner, ns)
             spec = spec or 'application'
             if spec in ns:
-                return ns[spec]
+                app = ns[spec]
                 raise Exception("No application %s defined in %s"
                                 % (spec, runner))
             raise Exception("Unknown kind of runner (%s)" % runner)
+        if is_production() and is_disabled(self.app_name):
+            disabled_appconfig = AppConfig.from_location('disabled')
+            return DisabledSite(app, disabled_appconfig.get_app_from_runner())
+        else:
+            return app
     def canonical_hostname(self):
         """Returns the 'canonical' hostname for this application.
         """Synchronize this application (locally) with a remote server
         at the given host.
-        dest_dir = os.path.join('/var/www', instance_name)
+        dest_dir = os.path.join(DEPLOYMENT_LOCATION, instance_name)
         self._run_rsync(host, self.app_dir, dest_dir)
     def sync_config(self, host, config_dir):

File silversupport/disabledapps.py Added

View file
  • Ignore whitespace
  • Hide word diff
+"""Routines for managing the list of disabled applications"""
+# The file contains application names rather than instance names or locations,
+# as what we really care about is databases, and they are shared between
+# deployments of the same application.
+# The format of the disabled apps file is at present simply a list of
+# application names.  In the future it might well be desirable to extend this
+# with options for the DisabledSite WSGI middleware, such as a list of IPs
+# which can be allowed to access the site, or a set of http auth credentials
+# which allow access.  Happily the file should be empty except in exceptional
+# circumstances, and we can therefore just change the data format without
+# concern for backwards compatibility.
+import os.path
+DISABLED_APPS_FILE = '/var/www/disabledapps.txt'
+class DisabledSite(object):
+    """This WSGI app is returned instead of our application when it is disabled
+    """
+    def __init__(self, real_app, disabled_app):
+        self._app = real_app
+        self._disabled = disabled_app
+    def __call__(self, environ, start_response):
+        """Delegates to the disabled app unless some conditions are met"""
+        if environ.get('silverlining.update'):
+            return self._app(environ, start_response)
+        # Allow connections from localhost.
+        client = environ['REMOTE_ADDR']
+        if client.strip() in ('localhost', ''):
+            return self._app(environ, start_response)
+        return self._disabled(environ, start_response)
+def is_disabled(application_name):
+    """Return True if the application has been disabled"""
+    if not os.path.exists(DISABLED_APPS_FILE):
+        return False
+    with open(DISABLED_APPS_FILE) as file_:
+        lines = [line.strip() for line in file_]
+        return application_name in lines
+def disable_application(application_name):
+    """Adds application_name to the list of disabled applications"""
+    if not is_disabled(application_name):
+        with open(DISABLED_APPS_FILE, 'a') as file_:
+            file_.write(application_name)
+            file_.write('\n')
+def enable_application(application_name):
+    """Removes application_name from the list of disabled applications"""
+    lines = []
+    with open(DISABLED_APPS_FILE, 'r') as file_:
+        for line in file_:
+            line = line.strip()
+            if line != application_name:
+                lines.append(line)
+    with open(DISABLED_APPS_FILE, 'w') as file_:
+        file_.write('\n'.join(lines))

File silversupport/service/postgis-pg_hba.conf Modified

View file
  • Ignore whitespace
  • Hide word diff
 # access to all databases is required during automatic maintenance
 # (custom daily cronjobs, replication, and similar tasks).
-# This is for general access databases:
+# This is for general databases access:
 local   all         all                               trust
 # Database administrative login by UNIX sockets
-local   all         postgres                          ident sameuser
+local   all         postgres                          ident
 # "local" is for Unix domain socket connections only
-local   all         all                               ident sameuser
+local   all         all                               ident
 # IPv4 local connections:
 host    all         all          md5
 # IPv6 local connections:
 host    all         all         ::1/128               md5

File silversupport/service/postgis.py Modified

View file
  • Ignore whitespace
  • Hide word diff
 class Service(AbstractService):
-    ## Note that PostGIS only works with 8.3, even though 8.4 is the more
-    ## modern version available on Karmic
     packages = [
-        'postgresql-8.3',
-        'postgresql-8.3-postgis',
+        'postgresql-8.4',
+        'postgresql-8.4-postgis',
-        'postgresql-client-8.3',
+        'postgresql-client-8.4',
-                            '/etc/postgresql/8.3/main/pg_hba.conf')
+                            '/etc/postgresql/8.4/main/pg_hba.conf')
             run(['chown', 'postgres:postgres',
-                 '/etc/postgresql/8.3/main/pg_hba.conf'])
-            run(['/etc/init.d/postgresql-8.3', 'restart'])
+                 '/etc/postgresql/8.4/main/pg_hba.conf'])
+            run(['/etc/init.d/postgresql-8.4', 'restart'])
         stdout, stderr, returncode = run(
             ['psql', '-U', 'postgres', '--tuples-only'],
             run(['createdb', '-U', 'postgres', 'template_postgis'])
             parts = ['CREATE LANGUAGE plpgsql;\n']
             parts.append("UPDATE pg_database SET datistemplate='true' WHERE datname='template_postgis';")
-            for filename in ['lwpostgis.sql', 'lwpostgis_upgrade.sql',
+            for filename in ['postgis.sql',
                 filename = os.path.join(
-                    '/usr/share/postgresql-8.3-postgis', filename)
+                    '/usr/share/postgresql/8.4/contrib', filename)
                 fp = open(filename)

File silversupport/service/postgresql-pg_hba.conf Modified

View file
  • Ignore whitespace
  • Hide word diff
 # (custom daily cronjobs, replication, and similar tasks).
-# This is for general access databases:
+# This is for general databases access:
 local   all         all                               trust
 # Database administrative login by UNIX sockets

File silversupport/shell.py Modified

View file
  • Ignore whitespace
  • Hide word diff
     This will use sudo for some users, and ssh in directory in other cases.
-    if isinstance(command, (list, tuple)):
-        command = ' '.join(conditional_shell_escape(i) for i in command)
     if user == 'www-data':
         # This is a bit tricky:
         user = 'www-mgr'
-        command = 'sudo -H -u www-data %s' % shell_escape(command)
+        if isinstance(command, (list, tuple)):
+            command = ' '.join(conditional_shell_escape(i) for i in command)
+        else:
+            command = conditional_shell_escape(command)
+        command = 'sudo -H -u www-data %s' % command
+    elif isinstance(command, (list, tuple)):
+        command = ' '.join(conditional_shell_escape(i) for i in command)
     ssh_args = kw.pop('ssh_args', [])
     return run(['ssh'] + ssh_args + ['-l', user, host, command], **kw)
     return run(['apt-get', 'install', '-q=2', '-y', '--force-yes'] + packages, **kw)
-_end_quote_re = re.compile(r"^('*)(.*?)('*)$")
+_end_quote_re = re.compile(r"^('*)(.*?)('*)$", re.S)
 _quote_re = re.compile("'+")

File silversupport/util.py Modified

View file
  • Ignore whitespace
  • Hide word diff
         n += 1
         result = '%s_%03i' % (name, n)
-def asbool(obj):
+def asbool(obj, default=ValueError):
     if isinstance(obj, (str, unicode)):
         obj = obj.strip().lower()
         if obj in ['true', 'yes', 'on', 'y', 't', '1']:
         elif obj in ['false', 'no', 'off', 'n', 'f', '0']:
             return False
+            if default is not ValueError:
+                return default
             raise ValueError(
                 "String is not true/false: %r" % obj)
     return bool(obj)
 def read_config(filename):
     if not os.path.exists(filename):
         raise ValueError('No file %s' % filename)
             raise KeyError(key)
             return self.default

File tests/functional/runtest.py Modified

View file
  • Ignore whitespace
  • Hide word diff
         if run_stage(stage, 'logs'):
             print 'Doing log check'
-            ssh('www-data', name, 'rm /var/log/silverlining/apps/functest/*')
+            ssh('www-data', name, ['bash', '-c', 'rm /var/log/silverlining/apps/functest/*'])
             url = 'http://%s/test/update' % name
             resp = urllib.urlopen(url).read()
             text, _, _ = ssh('www-data', name,
-                             'cat /var/log/silverlining/apps/functest/errors.log',
+                             ['cat', '/var/log/silverlining/apps/functest/errors.log'],
             text_lines = ''.join(text.strip().splitlines(True)[1:-1]).strip()
             assert text_lines == """\

File tests/unit/test_disabledapps.py Added

View file
  • Ignore whitespace
  • Hide word diff
+"""Tests for the functionality around disabled applications"""
+import os
+import os.path
+import tempfile
+from contextlib import contextmanager, nested
+import sys
+import shutil
+from webtest import TestApp
+from silversupport import disabledapps
+from silversupport.appconfig import AppConfig
+from silversupport import appdata
+ROOT = os.path.join(os.path.dirname(__file__),  '../..')
+def monkeypatch(module, global_, replacement):
+    """Replace module.global_ with replacement"""
+    if isinstance(module, str):
+        __import__(module)
+        module = sys.modules[module]
+    old = getattr(module, global_)
+    setattr(module, global_, replacement)
+    try:
+        yield
+    finally:
+        setattr(module, global_, old)
+def temporary_directory():
+    """Make and then remove a temporary directory"""
+    dirname = tempfile.mkdtemp()
+    try:
+        yield dirname
+    finally:
+        shutil.rmtree(dirname)
+def patch_disabled_apps_path(dirname):
+    """Changes the location of the disabled apps list to be within dirname
+    Returns the new path of the file
+    """
+    with monkeypatch(disabledapps, 'DISABLED_APPS_FILE',
+                     os.path.join(dirname, 'disabledapps.txt')):
+        try:
+            yield disabledapps.DISABLED_APPS_FILE
+        finally:
+            if os.path.exists(disabledapps.DISABLED_APPS_FILE):
+                os.remove(disabledapps.DISABLED_APPS_FILE)
+def test_addition():
+    """Adding an application should correctly write the file"""
+    with temporary_directory() as tempdir:
+        with patch_disabled_apps_path(tempdir) as path:
+            disabledapps.disable_application('testapp')
+            with open(path) as file_:
+                lines = [line.strip() for line in file_]
+                assert lines == ['testapp'], lines
+def test_is_disabled():
+    """We should be able to identify disabled apps"""
+    with temporary_directory() as tempdir:
+        with patch_disabled_apps_path(tempdir):
+            assert not disabledapps.is_disabled('testapp')
+            disabledapps.disable_application('testapp')
+            assert disabledapps.is_disabled('testapp')
+def test_removal():
+    """Removing applications should work correctly"""
+    with temporary_directory() as tempdir:
+        with patch_disabled_apps_path(tempdir) as path:
+            disabledapps.disable_application('testapp-a')
+            disabledapps.disable_application('testapp-b')
+            disabledapps.disable_application('testapp-c')
+            disabledapps.enable_application('testapp-b')
+            with open(path) as file_:
+                lines = [line.strip() for line in file_]
+                assert lines == ['testapp-a', 'testapp-c'], lines
+## Utilities to help set up a disabled site
+def patch_deployment_location(dirname):
+    """Replace /var/www with dirname, and set is_production to True"""
+    return nested(
+        patch_disabled_apps_path(dirname),
+        monkeypatch(appdata, 'APPDATA_MAP',
+                    os.path.join(dirname, 'appdata.map')),
+        monkeypatch('silversupport.appconfig', 'DEPLOYMENT_LOCATION', dirname),
+        monkeypatch('silversupport.appconfig', 'is_production', lambda: True))
+def install_default_disabled(dirname):
+    """Copy default-disabled into dirname"""
+    default_disabled = os.path.join(
+        ROOT, 'silverlining/server-root/var/www/default-disabled')
+    shutil.copytree(default_disabled,
+                    os.path.join(dirname, 'default-disabled'))
+    with open(os.path.join(dirname, 'appdata.map'), 'w') as file_:
+        file_.write(
+            "disabled / default-disabled|general_debug|/dev/null|python|\n")
+def install_sample_app(dirname):
+    """copy sample app into dirname"""
+    path = os.path.join(dirname, 'sampleapp.0')
+    os.mkdir(path)
+    with open(os.path.join(path, 'app.ini'), 'w') as file_:
+        file_.write('[production]\n')
+        file_.write('app_name = sampleapp\n')
+        file_.write('runner = app.py#application\n')
+    with open(os.path.join(path, 'app.py'), 'w') as file_:
+        file_.write(
+            'from wsgiref.simple_server import demo_app as application\n')
+    appdata.add_appdata('sampleapp.0', ['www.example.com'])
+def disabled_site():
+    with temporary_directory() as tempdir:
+        with patch_deployment_location(tempdir):
+            install_default_disabled(tempdir)
+            install_sample_app(tempdir)
+            disabledapps.disable_application('sampleapp')
+            app_config = AppConfig.from_location('www.example.com')
+            app = app_config.get_app_from_runner()
+            yield app
+def test_loading_disabled_site():
+    """Loading a site should return a DisabledSite when the site is disabled"""
+    with disabled_site() as app:
+        assert isinstance(app, disabledapps.DisabledSite), app
+def test_disabled_site_is_disabled():
+    with disabled_site() as app:
+        test_app = TestApp(app)
+        response = test_app.get(
+            '/', headers={'X-Forwarded-For':', localhost'},
+            status=503)
+def test_disabled_site_commandline_internal_access():
+    with disabled_site() as app:
+        test_app = TestApp(app)
+        response = test_app.get(
+            '/', headers={'X-Forwarded-For':', localhost'},
+            extra_environ={'silverlining.update': True},
+            status=200)
+        assert response.body.startswith('Hello world!'), str(response)
+def test_disabled_site_local_access():
+    with disabled_site() as app:
+        test_app = TestApp(app)
+        response = test_app.get(
+            '/', headers={'X-Forwarded-For':'localhost, localhost'})
+        assert response.status.startswith('200')
+        assert response.body.startswith('Hello world!')

File tests/unit/test_rewrite_map.py Modified

View file
  • Ignore whitespace
  • Hide word diff
     for line in tests:
         input, output = line.split(None, 1)
         hostname, path = input.split('^')
-        path_match, data = rewritemap.lookup(hostname, path)
+        path_match, data, _rest = rewritemap.lookup(hostname, path)
         result = '%s %s' % (path_match, data)
         assert result == output, (
             "Bad result: %r -> %r" % (line, result))