1. nowells
  2. django-lifestream


django-lifestream / lifestream / tests / feeds / atom / blogger_blog.xml

<?xml version='1.0' encoding='UTF-8'?><?xml-stylesheet href="http://www.blogger.com/styles/atom.css" type="text/css"?><feed xmlns='http://www.w3.org/2005/Atom' xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/' xmlns:georss='http://www.georss.org/georss'><id>tag:blogger.com,1999:blog-9238405</id><updated>2009-11-11T10:55:21.127-08:00</updated><title type='text'>Agile Testing</title><subtitle type='html'>Thoughts on testing and systems infrastructure with an agile, mostly Pythonic, twist.</subtitle><link rel='http://schemas.google.com/g/2005#feed' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/posts/default'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/'/><link rel='hub' href='http://pubsubhubbub.appspot.com/'/><link rel='next' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default?start-index=26&amp;max-results=25'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email></author><generator version='7.00' uri='http://www.blogger.com'>Blogger</generator><openSearch:totalResults>379</openSearch:totalResults><openSearch:startIndex>1</openSearch:startIndex><openSearch:itemsPerPage>25</openSearch:itemsPerPage><entry><id>tag:blogger.com,1999:blog-9238405.post-3285683763226093783</id><published>2009-11-10T16:48:00.000-08:00</published><updated>2009-11-10T16:48:21.422-08:00</updated><title type='text'>Google using buildbot for Chromium continuous integration</title><content type='html'>Via &lt;a href="http://twitter.com/benbangert"&gt;Ben Bangert&lt;/a&gt;, this &lt;a href="http://build.chromium.org/buildbot/waterfall/console"&gt;gem of a page&lt;/a&gt; showing the continuous integration status for the Chromium project at Google. It's cool to see that they're using buildbot. But just like Ben says -- I wish they open sourced the look and feel of that buildbot status page ;-)&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-3285683763226093783?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/3285683763226093783/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=3285683763226093783' title='3 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/3285683763226093783'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/3285683763226093783'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/11/google-using-buildbot-for-chromium.html' title='Google using buildbot for Chromium continuous integration'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>3</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-8629435591153782959</id><published>2009-11-10T12:04:00.000-08:00</published><updated>2009-11-10T12:04:03.006-08:00</updated><title type='text'>NFS troubleshooting with iostat and lsof</title><content type='html'>&lt;b&gt;Scenario&lt;/b&gt;: you mount a volume exported from a NetApp on several Linux clients via NFS&lt;br /&gt;&lt;br /&gt;&lt;b&gt;Problem&lt;/b&gt;: you see constant high CPU usage on the NetApp, and some of the Linux clients become sluggish, primarily in terms of I/O&lt;br /&gt;&lt;br /&gt;&lt;b&gt;Troubleshooting steps&lt;/b&gt;:&lt;br /&gt;&lt;br /&gt;1) If iostat is not already on the clients, install the &lt;a href="http://pagesperso-orange.fr/sebastien.godard/documentation.html"&gt;sysstat&lt;/a&gt; utilities.&lt;br /&gt;&lt;br /&gt;2) On each client mounting from the filer, or on a representative sample of the clients, run iostat with -n so that it shows NFS-related statistics. The following&lt;br /&gt;command will run iostat every 5 seconds and show NFS stats in a nicely tabulated output:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;# iostat -nh 5&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;3) Notice which client exhibits the most NFS operations per second, and correlate it with the NFS volume on that client which shows the most NFS reads and/or writes per second.&lt;br /&gt;&lt;br /&gt;At this point you found the most likely culprit in terms of sending NFS traffic to the filer (there could be several client machines in this position, for example if they are part of a cluster).&lt;br /&gt;&lt;br /&gt;5) If not already installed, download and install &lt;a href="http://freshmeat.net/projects/lsof/"&gt;lsof&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;6) Run lsof on the client(s) discovered in step 4, and grep for the directory representing the mount point of the NFS volume with the most reads and/or writes. For example:&lt;br /&gt;&lt;pre class="code"&gt;&amp;nbsp;&lt;/pre&gt;&lt;pre class="code"&gt;# lsof | grep /var/log&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;This will show you, among other things, which processes are accessing which files under that directory. Usually something will jump out at you in terms of things that are going on outside of the ordinary. In my case, it was logrotate kicking off from a daily cron and compressing a huge log file -- since the log file was on a volume NFS-mounted from the filer, this caused the filer to do extra work, hence its increased CPU usage.&lt;br /&gt;&lt;br /&gt;That's about it. Of courser these steps can be refined/modified/added to -- but even in this simple form, they can help you pinpoint NFS issues fairly quickly.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-8629435591153782959?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/8629435591153782959/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=8629435591153782959' title='0 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/8629435591153782959'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/8629435591153782959'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/11/nfs-troubleshooting-with-iostat-and.html' title='NFS troubleshooting with iostat and lsof'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>0</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-8491072546269562393</id><published>2009-11-05T13:55:00.000-08:00</published><updated>2009-11-05T13:55:52.200-08:00</updated><title type='text'>Automated deployments with Puppet and Fabric</title><content type='html'>I've been looking into various configuration management/automated deployment tools lately. At OpenX we used &lt;a href="http://code.google.com/p/slack/"&gt;slack&lt;/a&gt;, but I wanted something with a bit more functionality than that (although I'm not badmouthing slack by any means -- it can definitely be bent to your will to do pretty much whatever you need in terms of automating your deployments).&lt;br /&gt;&lt;br /&gt;From what I see, there are 2 types of configuration management tools:&lt;br /&gt;&lt;ol&gt;&lt;li&gt;The first type I call '&lt;b&gt;pull&lt;/b&gt;', which means that the servers pull their configurations and their marching orders in terms of applying those configurations from a centralized location -- both slack and Puppet are in this category. I think this is great for initial configuration of a server. As I described in another post, you can have a server bootstrap itself by installing Puppet (or slack) and then 'call home' to the central Puppet master (or slack repository) and get all the information it needs to configure itself&lt;/li&gt;&lt;li&gt;The second type I call '&lt;b&gt;push&lt;/b&gt;', which means that you send configurations and commands to a list of servers from a centralized location -- Fabric is in this category. I think this is a more appropriate mode for application-specific deployments, where you might want to deploy first to a subset of servers, then push it to all servers.&lt;/li&gt;&lt;/ol&gt;So, as a rule of thumb, I think it makes sense to use a tool like Puppet for the initial configuration of the OS and of the packages required by your application (things like MySQL, Apache, Tomcat, Tornado, Nginx, or whatever your application relies on). When it comes time to deploy your application, I think a tool like Fabric is more appropriate, since it gives you more immediate and finer-grained control over what you want to do.&lt;br /&gt;&lt;br /&gt;I also like the categorization of these tools done by the people at ControlTier. Check out their &lt;a href="http://blog.controltier.com/2009/04/new-whitepaper-achieving-fully.html"&gt;blog post on Achieving Fully Automated Provisioning &lt;/a&gt; (which also links to a white paper &lt;a href="http://www.dtosolutions.com/storage/downloads/FullyAutomatedProvisioning_Whitepaper.pdf"&gt;PDF&lt;/a&gt;) for a nice diagram of hierarchy of deployment tools:&lt;br /&gt;&lt;ul&gt;&lt;li&gt;at the bottom you have tools that install or launch the initial OS on physical servers (via Kickstart/Jumpstart/Cobbler) or on virtual machines/cloud instances (via various vendor tools, or by rolling your own)&lt;/li&gt;&lt;li&gt;in the middle you have what they call 'system configuration' tools, such as Puppet/Chef/SmartFrog/cfengine/bcfg2&lt;/li&gt;&lt;li&gt;at the top you have what they call 'application service deployment' tools, such as Fabric/Capistrano/Func -- and of course their own ControlTier tool&lt;/li&gt;&lt;/ul&gt;In a &lt;a href="http://agiletesting.blogspot.com/2009/09/bootstrapping-ec2-images-as-puppet.html#c6668282733983700258"&gt;comment&lt;/a&gt; on one of my posts,&amp;nbsp; Damon Edwards from ControlTier calls Fabric a "command dispatching tool", as opposed to Puppet, which he calls a "configuration management tool". I think this relates to the 2 types of tools I described above, where you 'push' or 'dispatch' commands with Fabric, and you 'pull' configurations and actions with Puppet.&lt;br /&gt;&lt;br /&gt;Before I go on, let me just say that in my evaluation of different deployment tools, I quickly eliminated the ones that use XML as their configuration language. In my experience, many tools that aim to be language-neutral end up using XML as their configuration language, and then they try to bend XML into a 'real' programming language, thus ending up reinventing the wheel badly. I'd rather use a language I like (Python in my case) as the glue around the various tools in my toolchain. Your mileage may vary of course.&lt;br /&gt;&lt;br /&gt;OK, enough theory, let's see some practical examples of Puppet and Fabric in action. While Fabric is very easy to install and has a minimal learning curve, I can't say the same about Puppet. It takes a while to get your brain wrapped around it, and there isn't a lot of great documentation online, so for this reason I warmly recommend that you go buy &lt;a href="http://www.amazon.com/Pulling-Strings-Puppet-Configuration-Management/dp/1590599780"&gt;the book&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;Puppet examples&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;The way I organize things in &lt;a href="http://reductivelabs.com/trac/puppet/"&gt;Puppet&lt;/a&gt; is by creating a module for each major package I need to configure. On my puppetmaster server, under &lt;i&gt;/etc/puppet/modules&lt;/i&gt;, I have directories such as &lt;i&gt;apache2&lt;/i&gt;, &lt;i&gt;mysqlserver&lt;/i&gt;, &lt;i&gt;nginx&lt;/i&gt;, &lt;i&gt;scribe&lt;/i&gt;, &lt;i&gt;tomcat&lt;/i&gt;, &lt;i&gt;tornado&lt;/i&gt;. Under each such directory I have 2 directories, one called &lt;i&gt;files&lt;/i&gt; and one called &lt;i&gt;manifests&lt;/i&gt;. I keep files and directories that I need downloaded to the puppet clients under &lt;i&gt;files&lt;/i&gt;, and I create manifests (series of actions to be taken on the puppet clients) under &lt;i&gt;manifests&lt;/i&gt;. I usually have a single manifest file called &lt;i&gt;init.pp&lt;/i&gt;.&lt;br /&gt;&lt;br /&gt;Here's an example of the init.pp manifest file for my tornado module:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;class tornado {&lt;br /&gt; $tornado = "tornado-0.2"&lt;br /&gt; $url = "http://mydomain.com/download"&lt;br /&gt;&lt;br /&gt; $tornado_root_dir = "/opt/tornado"&lt;br /&gt; $tornado_log_dir = "/opt/tornado/logs"&lt;br /&gt; $tornado_src_dir = "/opt/tornado/$tornado"&lt;br /&gt;&lt;br /&gt; Exec {&lt;br /&gt;  logoutput =&amp;gt; on_failure,&lt;br /&gt;  path =&amp;gt; ["/bin", "/sbin", "/usr/bin", "/usr/sbin", "/usr/local/bin",  "/usr/local/sbin"]&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; file { &lt;br /&gt;  "$tornado_root_dir":&lt;br /&gt;  ensure =&amp;gt; directory,&lt;br /&gt;  recurse =&amp;gt; true,&lt;br /&gt;  source =&amp;gt;  "puppet:///tornado/bin";&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; file { &lt;br /&gt;  "$tornado_log_dir":&lt;br /&gt;  ensure =&amp;gt; directory,&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; package {&lt;br /&gt;  ["curl", "libcurl3", "libcurl3-gnutls", "python-setuptools", "python-pycurl", "python-simplejson", "python-memcache", "python-mysqldb", "python-imaging"]:&lt;br /&gt;  ensure =&amp;gt; installed;&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; define install_pkg ($pkgname, $extra_easy_install_args = "", $module_to_test_import) {&lt;br /&gt;  exec {&lt;br /&gt;   "InstallPkg_$pkgname":&lt;br /&gt;   command =&amp;gt; "easy_install-2.6 $extra_easy_install_args $pkgname",&lt;br /&gt;   unless =&amp;gt; "python2.6 -c 'import $module_to_test_import'",&lt;br /&gt;   require =&amp;gt; Package["python-setuptools"];&lt;br /&gt;  }&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; install_pkg {&lt;br /&gt;  "virtualenv":&lt;br /&gt;  pkgname =&amp;gt; "virtualenv",&lt;br /&gt;  module_to_test_import =&amp;gt; "virtualenv";&lt;br /&gt;&lt;br /&gt;  "boto":&lt;br /&gt;  pkgname =&amp;gt; "boto",&lt;br /&gt;  module_to_test_import =&amp;gt; "boto";&lt;br /&gt;&lt;br /&gt;  "grizzled":&lt;br /&gt;  pkgname =&amp;gt; "grizzled",&lt;br /&gt;  module_to_test_import =&amp;gt; "grizzled.os";&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; $oracle_root_dir = "/opt/oracle"&lt;br /&gt; &lt;br /&gt; case $architecture {&lt;br /&gt;  i386, i686: { &lt;br /&gt;   $oracle_instant_client_pkg = "instantclient_11_2-linux-i386"&lt;br /&gt;   $oracle_instant_client_dir = "instantclient_11_2"&lt;br /&gt;  }&lt;br /&gt;  x86_64: { &lt;br /&gt;   $oracle_instant_client_pkg = "instantclient_11_1-linux-x86_64"&lt;br /&gt;   $oracle_instant_client_dir = "instantclient_11_1"&lt;br /&gt;  }&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; package {&lt;br /&gt;  ["libaio-dev", "gcc"]:&lt;br /&gt;  ensure =&amp;gt; installed;&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; file { &lt;br /&gt;  "$oracle_root_dir":&lt;br /&gt;  ensure =&amp;gt; directory;&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; exec {&lt;br /&gt;  "InstallOracleInstantclient":&lt;br /&gt;  command =&amp;gt; "(cd $oracle_root_dir; wget $url/$oracle_instant_client_pkg.tar.gz; tar xvfz $oracle_instant_client_pkg.tar.gz; rm $oracle_instant_client_pkg.tar.gz; &lt;br /&gt;cd $oracle_instant_client_dir; ln -s libclntsh.so.11.1 libclntsh.so); echo $oracle_root_dir/$oracle_instant_client_dir &amp;gt; /etc/ld.so.conf.d/oracleinstantclient.conf; ldconfig",&lt;br /&gt;  creates =&amp;gt; "$oracle_root_dir/$oracle_instant_client_dir",&lt;br /&gt;  require =&amp;gt; File[$oracle_root_dir];&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; $cx_oracle = "cx_Oracle-5.0.2"&lt;br /&gt; exec {&lt;br /&gt;  "InstallCxOracle":&lt;br /&gt;  command =&amp;gt; "(cd $oracle_root_dir; wget $url/$cx_oracle.tar.gz; tar xvfz $cx_oracle.tar.gz; rm $cx_oracle.tar.gz; cd $oracle_root_dir/$cx_oracle; export ORACLE_HO&lt;br /&gt;ME=$oracle_root_dir/$oracle_instant_client_dir; python2.6 setup.py install)",&lt;br /&gt;  unless =&amp;gt; "python2.6 -c 'import cx_Oracle'",&lt;br /&gt;  require =&amp;gt; [Package["libaio-dev"], Package["gcc"], Exec["InstallOracleInstantclient"]];&lt;br /&gt; }&lt;br /&gt;&lt;br /&gt; exec {&lt;br /&gt;  "InstallTornado":&lt;br /&gt;  command =&amp;gt; "(cd $tornado_root_dir; wget $url/$tornado.tar.gz; tar xvfz $tornado.tar.gz; rm $tornado.tar.gz; cd $tornado; python2.6 setup.py install)",&lt;br /&gt;  creates =&amp;gt; $tornado_src_dir,&lt;br /&gt;  unless =&amp;gt; "python2.6 -c 'import tornado.web'",&lt;br /&gt;  require =&amp;gt; [File[$tornado_root_dir], Package["python-pycurl"], Package["python-simplejson"], Package["python-memcache"], Package["python-mysqldb"]];&lt;br /&gt; }&lt;br /&gt;}&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;I'll go through this file from the top down. At the very top I declare some variables that are referenced throughout the file. In particular, $url points to the location where I keep large files that I need every puppet client to download. I could have kept the files inside the tornado module's &lt;i&gt;files&lt;/i&gt; directory, and they would have been served by the puppetmaster process, but I prefered to use Apache for better performance and scalability. Note that I do this only for relatively large files such as tar.gz archives.&lt;br /&gt;&lt;br /&gt;The Exec stanza (note upper case E) defines certain parameters that will be common to all 'exec' actions that follow. In my case, I specify that I only want to log failures, and I also specify the path for the binaries called in the various 'exec' actions -- this is so I don't have to specify that path each and every time I call 'exec' (alternatively, you can specify the full path to each binary that you call).&lt;br /&gt;&lt;br /&gt;The next 2 stanzas define files and directories that I want created on the puppet client nodes. Both 'exec' and 'file' are what is called '&lt;a href="http://reductivelabs.com/trac/puppet/wiki/TypeReference"&gt;types&lt;/a&gt;' in Puppet lingo. I first specify that I wanted the directory &lt;i&gt;/opt/tornado&lt;/i&gt; created on each node, and by setting '&lt;i&gt;recurse=&amp;gt;true&lt;/i&gt;' I'm saying that the contents of that directory should be taken from a source which in my case is "&lt;i&gt;puppet:///tornado/bin&lt;/i&gt;". This translates to a directory called &lt;i&gt;bin&lt;/i&gt; which I created under &lt;i&gt;/etc/puppet/modules/tornado/files&lt;/i&gt;. The contents of that directory will be copied over via the puppet internal communication protocol to the destination &lt;i&gt;/opt/tornado&lt;/i&gt; by each Puppet client node.&lt;br /&gt;&lt;br /&gt;The 'package' type that follows specifies the list of packages I want installed on the client nodes. Note that I don't need to specify &lt;b&gt;how&lt;/b&gt; I want those packages installed, only &lt;b&gt;what&lt;/b&gt; I want installed. Puppet's language is mostly &lt;b&gt;declarative&lt;/b&gt; -- you tell Puppet what you want done, and it does it for you, using OS-specific commands that can vary from one client node to another. It so happens in my case that I know my client nodes all run Ubuntu, so I did specify Ubuntu/Debian-specific package names.&lt;br /&gt;&lt;br /&gt;Next in my manifest file is a function definition. You can have these definitions inline, or in a separate manifest file. In my case, I declare a function called &lt;i&gt;'install_pkg&lt;/i&gt;' which takes 3 arguments: the package name, any extra arguments to be passed to the installer, and a module name to test the installation with. The function runs the easy_install command via the '&lt;i&gt;exec&lt;/i&gt;' type, but only if the specified module wasn't already installed on the system.&lt;br /&gt;&lt;br /&gt;A paranthesis: the Puppet docs don't recommend the overuse of the '&lt;i&gt;exec&lt;/i&gt;' type, because it strays away from the declarative nature of the Puppet language. With &lt;i&gt;exec&lt;/i&gt;, you specifically tell the remote node how to run a specific command, not merely what to do. I find myself using &lt;i&gt;exec&lt;/i&gt; very heavily though. I means that I don't grokk Puppet fully yet, but it also means that Puppet doesn't have enough native types yet that can hide OS-specific commands.&lt;br /&gt;&lt;br /&gt;One important thing to keep in mind is that for every &lt;i&gt;exec&lt;/i&gt; action that you write, you need to specify a condition which becomes true after the successful completion of the action. Otherwise &lt;i&gt;exec&lt;/i&gt; will be called each and every time the manifest will be inspected by the puppet nodes. Examples of such conditions:&lt;br /&gt;&lt;ul&gt;&lt;li&gt;'&lt;i&gt;creates&lt;/i&gt;' -- specifies a file or directory that gets created by the exec action; if the file or directory is already there, exec won't be called&lt;br /&gt;&lt;/li&gt;&lt;li&gt;'&lt;i&gt;unless&lt;/i&gt;' -- specifies a condition that, if true, results in exec not being called. In my case, this condition is the import of a given Python module, but it can be any shell command that returns 0&lt;/li&gt;&lt;/ul&gt;Another thing to note in the &lt;i&gt;exec&lt;/i&gt; action is the '&lt;i&gt;require&lt;/i&gt;' parameter. You'll find yourself using '&lt;i&gt;require&lt;/i&gt;' over and over again. It is a critical component of Puppet manifests, and it is so important because it allows you to &lt;b&gt;order&lt;/b&gt; the actions in the manifest. Without it, actions would be executed in random order, which is most likely something you don't want. In my function definition, I require the existence of the package python-setuptools, and I do it because I need the easy_install command to be present on the remote node.&lt;br /&gt;&lt;br /&gt;After defining the function '&lt;i&gt;install_pkg&lt;/i&gt;', I call it 3 times, with various parameters, thus installing 3 Python packages -- virtualenv, boto and grizzled. Note that the syntax for calling a function is funky; it's one of the many things I don't necessarily like about Puppet, but it's an evil you learn to deal with.&lt;br /&gt;&lt;br /&gt;Next up in my manifest file is a &lt;i&gt;case&lt;/i&gt; statement based on the &lt;i&gt;$architecture&lt;/i&gt; variable. Puppet makes several such variables available to your manifests, based on facts gathered from the remote nodes via &lt;a href="http://reductivelabs.com/trac/puppet/tags/facter%2Crecipe"&gt;Facter&lt;/a&gt; (which comes with Puppet).&lt;br /&gt;&lt;br /&gt;Moving along, we have a package definition, a file definition -- both should be familiar by now -- followed by 3 exec actions:&lt;br /&gt;&lt;ul&gt;&lt;li&gt;&lt;i&gt;InstallOracleInstantclient&lt;/i&gt; performs the download and unpacking of this package, followed by some ldconfig incantations to actually make it work&lt;/li&gt;&lt;li&gt;&lt;i&gt;InstallCxOracle&lt;/i&gt; downloads and installs the cx_Oracle Python package (not a trivial feat at all in and of itself); note that for this action, the &lt;i&gt;require&lt;/i&gt; parameter contains &lt;i&gt;Package["libaio-dev"], Package["gcc"], Exec["InstallOracleInstantclient"]&lt;/i&gt; -- so we're saying that these 2 packages, and the Instantclient Oracle libraries need to be installed before attempting to even install cx_Oracle&lt;br /&gt;&lt;/li&gt;&lt;li&gt;&lt;i&gt;InstallTornado&lt;/i&gt; -- pretty self-explanatory, with the observation that the &lt;i&gt;require&lt;/i&gt; parameter again points to a directory and several packages that need to be on the remote node before the installation of Tornado is attempted&lt;/li&gt;&lt;/ul&gt;Whew. Nobody said Puppet is easy. But let me tell you, when you get everything working smoothly (after much pulling of hair), it's a great feeling to let a node 'phone home' to the puppetmaster server and configure itself unattended in a matter of minutes. It's worth the effort and the pain.&lt;br /&gt;&lt;br /&gt;One more thing here: once you have a module with manifests and files defined properly, you need to define the set of nodes that this module will apply to. The way I do it is to have the following files on the puppet master, in /etc/puppet/manifests:&lt;br /&gt;&lt;br /&gt;1) A file called &lt;i&gt;modules.pp&lt;/i&gt; which imports the modules I have defined, for example:&lt;br /&gt;&lt;pre class="code"&gt;import "common" &lt;br /&gt;import "tornado"&lt;br /&gt;&lt;/pre&gt;('common' can be a module where you specify actions that are common across all types of nodes)&lt;br /&gt;&lt;br /&gt;2) A file called &lt;i&gt;nodetemplates.pp&lt;/i&gt; which contains definitions for 'node templates', i.e. classes of nodes that have the same composition in terms of modules they import and actions they perform. For example:&lt;br /&gt;&lt;pre class="code"&gt;node basenode {&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; include common&lt;br /&gt;}&lt;br /&gt;&lt;br /&gt;node default inherits basenode {&lt;br /&gt;}&lt;br /&gt;&lt;br /&gt;node webserver inherits basenode {&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; include scribe&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; include apache2&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; $required_apache2_modules = ["rewrite", "proxy", "proxy_http", "proxy_balancer", "deflate", "headers", "expires"]&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; apache2::module {&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; &amp;nbsp;&amp;nbsp;&amp;nbsp; $required_apache2_modules:&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; &amp;nbsp;&amp;nbsp;&amp;nbsp; ensure =&amp;gt; 'present',&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; }&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; include tomcat&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; include tornado&lt;br /&gt;}&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;Here I defined 3 types of nodes: basenode (which includes the 'common' module), default (which applies to any machine not associated with a specific node definition) and webserver (which includes modules such as apache2, tomcat, tornado, and also requires that certain apache modules be enabled).&lt;br /&gt;&lt;br /&gt;3) A file called nodes.pp which maps actual machine names of the Puppet clients to node template definitions. For example:&lt;br /&gt;&lt;pre class="code"&gt;node "web1.mydomain.com" inherits webserver {}&lt;br /&gt;&lt;/pre&gt;4) A file called site.pp which ties together all these other files. It contains:&lt;br /&gt;&lt;pre class="code"&gt;import "modules"&lt;br /&gt;import "nodetemplates"&lt;br /&gt;import "nodes"&amp;nbsp;&lt;/pre&gt;&lt;pre class="code"&gt;&lt;/pre&gt;&lt;br /&gt;Much more documentation on node definition and node inheritance can be found on the Puppet wiki, especially in the &lt;a href="http://reductivelabs.com/trac/puppet/wiki/LanguageTutorial"&gt;Language Tutorial&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;Fabric examples&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;In comparison with Puppet, &lt;a href="http://www.nongnu.org/fab/"&gt;Fabric&lt;/a&gt; is a breeze. I wanted to live on the cutting edge, so I installed the latest version (alpha, pre-1.0) from &lt;a href="http://github.com/bitprophet/fabric"&gt;github&lt;/a&gt; via:&lt;br /&gt;&lt;br /&gt;git clone &lt;a class="git_url_facebox" href="git://github.com/bitprophet/fabric.git" rel="#git-clone"&gt;git://github.com/bitprophet/fabric.git&lt;/a&gt;&lt;br /&gt;&lt;br /&gt;I also easy_install'ed paramiko, which at this time brings down paramiko-1.7.6 (the Fabric documentation warns against using 1.7.5, but I assume 1.7.6 is OK).&lt;br /&gt;&lt;br /&gt;Then I proceeded to create a so-called 'fabfile', which is a Python module containing fabric-specific functions. Here is a fragment of a file I called fab_nginx.py:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;from __future__ import with_statement&lt;br /&gt;import os&lt;br /&gt;from fabric.api import *&lt;br /&gt;from fabric.contrib.files import comment, sed&lt;br /&gt;&lt;br /&gt;# Globals&lt;br /&gt;&lt;br /&gt;env.user = 'myuser'&lt;br /&gt;env.password = 'mypass'&lt;br /&gt;env.nginx_conf_dir = '/usr/local/nginx/conf'&lt;br /&gt;env.nginx_conf_file = '%(nginx_conf_dir)s/nginx.conf' % env&lt;br /&gt;&lt;br /&gt;# Environments&lt;br /&gt;&lt;br /&gt;&lt;br /&gt;def prod():&lt;br /&gt;    """Nginx production environment."""&lt;br /&gt;    env.hosts = ['nginx1', 'nginx2']&lt;br /&gt;&lt;br /&gt;def test():&lt;br /&gt;    """Nginx test environment."""&lt;br /&gt;    env.hosts = ['nginx3']&lt;br /&gt;&lt;br /&gt;# Tasks&lt;br /&gt;&lt;br /&gt;def disable_server_in_lb(hostname):&lt;br /&gt;    require('hosts', provided_by=[nginx,nginxtest])&lt;br /&gt;    comment(env.nginx_conf_file, "server %s" % hostname, use_sudo=True)&lt;br /&gt;    restart_nginx()&lt;br /&gt;&lt;br /&gt;def enable_server_in_lb(hostname):&lt;br /&gt;    require('hosts', provided_by=[nginx,nginxtest])&lt;br /&gt;    sed(env.nginx_conf_file, "#server %s" % hostname, "server %s" % hostname, use_sudo=True)&lt;br /&gt;    restart_nginx()&lt;br /&gt;&lt;br /&gt;def restart_nginx():&lt;br /&gt;    require('hosts', provided_by=[nginx,nginxtest])&lt;br /&gt;    sudo('/etc/init.d/nginx restart')&lt;br /&gt;    is_nginx_running()&lt;br /&gt;&lt;br /&gt;def is_nginx_running(warn_only=False):&lt;br /&gt;    with settings(warn_only=warn_only):&lt;br /&gt;        output = run('ps -def|grep nginx|grep -v grep')&lt;br /&gt;        if warn_only:&lt;br /&gt;            print 'output:', output&lt;br /&gt;            print 'failed:', output.failed&lt;br /&gt;            print 'return_code:', output.return_code&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;Note that in its 0.9 and later versions, Fabric uses the '&lt;i&gt;env&lt;/i&gt;' &lt;a href="http://docs.fabfile.org/0.9/usage/env.html"&gt;environment dictionary&lt;/a&gt; for configuration purposes (it used to be called 'config' pre-0.9).&lt;br /&gt;&lt;br /&gt;My file starts by defining or assigning global &lt;i&gt;env&lt;/i&gt; configuration variables, for example &lt;i&gt;env.user&lt;/i&gt; and &lt;i&gt;env.password&lt;/i&gt; (which are special pre-defined variables that I assign to, and which are used by Fabric when connecting to remote hosts via the ssh functionality provided by paramiko). I also define my own variables, for example &lt;i&gt;env.nginx_conf_dir&lt;/i&gt; and &lt;i&gt;env.nginx_conf_file&lt;/i&gt;. This makes it easy to pass the env dictionary as a whole when I need to format a string. Here's an example from another fab file:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;cmd = 'mv -f %(crt_egg)s %(backup_dir)s' % env&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;I then have 2 function definitions in my fab file: one called &lt;i&gt;prod&lt;/i&gt;, which sets env.hosts to a list of production nginx servers, and one called &lt;i&gt;test&lt;/i&gt;, which does the same but sets env.hosts to test nginx servers.&lt;br /&gt;&lt;br /&gt;Next I have the actions or tasks that I want performed on the remote hosts. Note the &lt;i&gt;require&lt;/i&gt; function (similar in a way to the parameter used in Puppet manifests), which says that the function will only be executed if the given variable in the env dictionary has been assigned to (in my case, the variable is hosts, and I require that the value need to have been provided by either the prod or the test function). This is a useful mechanism to ensure that certain things have been defined before attempting to run commands on the remote servers.&lt;br /&gt;&lt;br /&gt;The first task is called &lt;i&gt;disable_server_in_lb&lt;/i&gt;. It takes a host name as a parameter, which is the server that I want disabled in the nginx configuration file. I use the handy '&lt;i&gt;comment&lt;/i&gt;' function available in fabric.contrib.files to comment out the lines that contain '&lt;i&gt;server HOSTNAME&lt;/i&gt;' in the nginx configuration. The comment function can be invoked with sudo rights on the remote host by passing &lt;i&gt;use_sudo=True&lt;/i&gt;.&lt;br /&gt;&lt;br /&gt;The task also calls another function defined in my fab file, &lt;i&gt;restart_nginx&lt;/i&gt;. This taks simply calls '&lt;i&gt;/etc/init.d/nginx restart&lt;/i&gt;' on the remote host, then verifies that nginx is running by calling &lt;i&gt;is_nginx_running&lt;/i&gt;.&lt;br /&gt;&lt;br /&gt;By default, when running a command on the remote host, if the command returns a non-zero code, it is considered to have failed by Fabric, and execution stops. In most cases, this is exactly what you want. In case you just want to run a command to get the output, and you don't care if it fails, you can set &lt;i&gt;warn_only=True&lt;/i&gt; before running the command. I show an example if this in the &lt;i&gt;is_nginx_running &lt;/i&gt;function.&lt;br /&gt;&lt;br /&gt;The other main task in my fabfile is &lt;i&gt;enable_server_in_lb&lt;/i&gt;. Here I use another handy function offered by Fabric -- the &lt;i&gt;sed&lt;/i&gt; function. I substitute '&lt;i&gt;#server&amp;nbsp; HOSTNAME&lt;/i&gt;' with '&lt;i&gt;server HOSTNAME&lt;/i&gt;' in the nginx configuration file, then I restart nginx.&lt;br /&gt;So now that we have the fabfile, how do we actually perform the tasks we defined? Let's assume we have a server called 'web1.mydomain.com' that we want disabled in nginx. We want to test our task first in a test environment, so we would call:&lt;br /&gt;&lt;pre class="code"&gt;fab -f fab_nginx.py test disable_server_in_lb:web1.mydomain.com&lt;br /&gt;&lt;/pre&gt;(note the syntax for passing parameters to a function/task)&lt;br /&gt;&lt;br /&gt;By specifying test on the command line before specifying the task, I ensure that Fabric first calls the function named 'test' in the fabfile, which sets the hosts to the test nginx servers.&lt;br /&gt;&lt;br /&gt;Once I'm satisfied that this works well in the test environment, I call:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;fab -f fab_nginx.py prod disable_server_in_lb:web1.mydomain.com&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;For a real deployment procedure, let's say for deploying tornado-based servers that are behind one or more nginx load balancer, I would do something like this:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;fab -f fab_nginx.py prod disable_server_in_lb:web1.mydomain.com&lt;br /&gt;fab -f fab_tornado.py prod deploy&lt;br /&gt;fab -f fab_nginx.py prod enable_server_in_lb:web1.mydomain.com&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;This will deploy my new application code to web1.mydomain.com. Of course I can script this and call the above sequence for all my production servers. I assume here that I have another fabfile called fab_tornado.py and a task defined in in which does the actual deployment of the application code (most likely by downloading and easy_install'ing an egg).&lt;br /&gt;&lt;br /&gt;That's it for today. It's been more like a whirlwind through two types of automated deployment tools -- Puppet/pull and Fabric/push. I didn't do justice to either of these tools in terms of their full capabilities, but I hope this will still be useful for some people as a starting point into their own explorations.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-8491072546269562393?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/8491072546269562393/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=8491072546269562393' title='3 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/8491072546269562393'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/8491072546269562393'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/11/automated-deployments-with-puppet-and.html' title='Automated deployments with Puppet and Fabric'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>3</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-1754210905399288790</id><published>2009-10-13T14:21:00.000-07:00</published><updated>2009-10-13T14:21:35.348-07:00</updated><title type='text'>Thierry Carrez on running your own Ubuntu Enterprise Cloud</title><content type='html'>&lt;a href="http://fnords.wordpress.com/"&gt;Thierry Carrez&lt;/a&gt;, who works in the Ubuntu Server team, has a great series of blog posts on how to run your own Ubuntu Enterprise Cloud. I haven't had a chance to&amp;nbsp; try this yet, but it's high on my TODO list. Thierry uses the &lt;a href="http://www.ubuntu.com/products/whatisubuntu/serveredition/cloud/uec"&gt;Ubuntu Enterprise Cloud&lt;/a&gt; product (which has been part of Ubuntu server starting with 9.04) together with &lt;a href="http://www.eucalyptus.com/"&gt;Eucalyptus&lt;/a&gt;. Here are the links to Thierry's posts:&lt;br /&gt;&lt;ul&gt;&lt;li&gt;&lt;a href="http://fnords.wordpress.com/2009/10/04/run-your-own-uec-part-1/"&gt;Part 1&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="http://fnords.wordpress.com/2009/10/07/run-your-own-uec-part-2/"&gt;Part 2&lt;/a&gt;&lt;/li&gt;&lt;li&gt;&lt;a href="http://fnords.wordpress.com/2009/10/13/run-your-own-uec-part-3/"&gt;Part 3&lt;/a&gt;&lt;br /&gt;&lt;/li&gt;&lt;/ul&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-1754210905399288790?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/1754210905399288790/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=1754210905399288790' title='1 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/1754210905399288790'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/1754210905399288790'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/10/thierry-carrez-on-running-your-own.html' title='Thierry Carrez on running your own Ubuntu Enterprise Cloud'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>1</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-1877175784560418350</id><published>2009-10-09T12:51:00.000-07:00</published><updated>2009-10-09T12:53:10.244-07:00</updated><title type='text'>Compiling, installing and test-running Scribe</title><content type='html'>I went to the &lt;a href="http://www.cloudera.com/hadoop-world-nyc"&gt;Hadoop World&lt;/a&gt; conference last week and one thing I took away was how Facebook and other companies handle the problem of scalable logging within their infrastructure. The solution found by Facebook was to write their own logging server software called Scribe (more details on the &lt;a href="http://www.facebook.com/note.php?note_id=32008268919"&gt;FB blog&lt;/a&gt;).&lt;br /&gt;&lt;br /&gt;Scribe is mentioned in one of the best presentations I attended at the conference -- '&lt;a href="http://www.borthakur.com/ftp/hadoopworld.pdf"&gt;Hadoop and Hive Development at Facebook&lt;/a&gt;' by Dhruba Borthakur and Zheng Shao. If you look at page 4, you'll see the enormity of the situation they're facing: 4 TB of compressed data (mostly logs) handled every day, and 135 TB of compressed data scanned every day. All this goes through Scribe, so that gives me a warm fuzzy feeling that it's indeed scalable and robust. For more details on Scribe, see the &lt;a href="http://sourceforge.net/apps/mediawiki/scribeserver/index.php?title=Main_Page"&gt;wiki page&lt;/a&gt; of the project. It's my intention here to detail the steps needed for compiling and installing it, since I found that to be a non-trivial process to say the least. I'm glad Facebook open-sourced Scribe, but its packaging could have been a bit more straightforward. Anyway, here's what I did to get it to run. I followed roughly the same steps on Ubuntu and on Gentoo.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;1) Install pre-requisite packages&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;On Ubuntu, I had to install the following packages via apt-get:  g++, make, build-essential, flex, bison, libtool, mono-gmcs, libevent-dev.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;2) Install the boost libraries&lt;br /&gt;&lt;/b&gt;&lt;br /&gt;&lt;b&gt;Very important&lt;/b&gt;: scribe needs boost 1.36 or newer, so make sure you don't have older boost libraries already installed. If you install libboost-* in Ubuntu, it tries to bring down 1.34 or 1.35, which will &lt;b&gt;NOT&lt;/b&gt; work with scribe. If you have libboost-* already installed, you need to uninstall them. Now. Trust me, I spent several hours pulling my hair on this one.&lt;br /&gt;&lt;br /&gt;- download the latest boost source code from SourceForge (I got boost 1.40 from &lt;a href="http://downloads.sourceforge.net/project/boost/boost/1.40.0/boost_1_40_0.tar.gz?use_mirror=softlayer"&gt;here&lt;/a&gt;)&lt;br /&gt;&lt;br /&gt;- untar it, then cd into the boost directory and run:&lt;br /&gt;&lt;br /&gt;$ ./boostrap.sh&lt;br /&gt;$ ./bjam&lt;br /&gt;$ sudo ./bjam install&lt;br /&gt;&lt;br /&gt;&lt;b&gt;3) Install thrift and fb303&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;- get thrift source code with git, compile and install:&lt;br /&gt;&lt;br /&gt;$ git clone git://git.thrift-rpc.org/thrift.git&lt;br /&gt;$ cd thrift&lt;br /&gt;$ ./bootstrap.sh&lt;br /&gt;$ ./configure&lt;br /&gt;$ make&lt;br /&gt;$ sudo make install&lt;br /&gt;&lt;br /&gt;- compile and install the Facebook fb303 library:&lt;br /&gt;&lt;br /&gt;$ cd contrib/fb303&lt;br /&gt;$ ./bootstrap.sh&lt;br /&gt;$ make&lt;br /&gt;$ sudo make install&lt;br /&gt;&lt;br /&gt;- install the Python modules for thrift and fb303:&lt;br /&gt;&lt;br /&gt;$ cd TOP THRIFT DIRECTORY&lt;br /&gt;$ cd lib/py&lt;br /&gt;$ sudo python setup.py install&lt;br /&gt;$ cd TOP THRIFT DIRECTORY&lt;br /&gt;$ cd contrib/fb303/py&lt;br /&gt;$ sudo python setup.py install&lt;br /&gt;&lt;br /&gt;To check that the python modules have been installed properly, run:&lt;br /&gt;&lt;br /&gt;$ python -c 'import thrift' ; python -c 'import fb303'&lt;br /&gt;&lt;br /&gt;&lt;b&gt;4) Install Scribe&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;- download latest source code from SourceForge (I got it from &lt;a href="http://downloads.sourceforge.net/project/scribeserver/Scribe/Scribe%20version%202.01/scribe-version-2.01.tar.gz?use_mirror=softlayer"&gt;here&lt;/a&gt;)&lt;br /&gt;&lt;br /&gt;- untar, then run:&lt;br /&gt;&lt;br /&gt;$ cd scribe&lt;br /&gt;$ ./bootstrap.sh&lt;br /&gt;$ make&lt;br /&gt;$ sudo make install&lt;br /&gt;$ sudo ldconfig  (this is necessary so that the boost shared libraries are loaded)&lt;br /&gt;&lt;br /&gt;- install Python modules for scribe:&lt;br /&gt;&lt;br /&gt;$ cd lib/py&lt;br /&gt;$ sudo python setup.py install&lt;br /&gt;&lt;br /&gt;- to test that scribed (the scribe server process) was installed correctly, just run 'scribed' at a command line; you shouldn't get any errors&lt;br /&gt;- to test that the scribe Python module was installed correctly, run&lt;br /&gt;$ python -c 'import scribe'&lt;br /&gt;&lt;br /&gt;&lt;b&gt;5) Initial Scribe configuration&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;- create configuration directory -- in my case I created /etc/scribe&lt;br /&gt;- copy one of the example config files from TOP_SCRIBE_DIRECTORY/examples/example*conf to /etc/scribe/scribe.conf -- a good one to start with is example1.conf&lt;br /&gt;- edit /etc/scribe/scribe.conf and replace file_path (which points to /tmp) to a location more suitable for your system&lt;br /&gt;- you may also want to replace max_size, which dictates how big the local files can be before they're rotated (by default it's 1 MB, which is too small -- I set it to 100 MB)&lt;br /&gt;- run scribed either with nohup or in a screen session (it doesn't seem to have a daemon mode):&lt;br /&gt;&lt;br /&gt;$ scribed -c /etc/scribe/scribe.conf&lt;br /&gt;&lt;br /&gt;&lt;b&gt;6) Test run&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;To test Scribe, you can install it on a remote machine, configure scribed on that machine to use a configuration file similar to examples/example2client.conf, then change remote_host in the config file to point to the central scribe server configured in step 5.&lt;br /&gt;&lt;br /&gt;Once scribed is configured and running on the remote machine, you can test it with a nice utility written by Silas Sewell, called &lt;a href="http://silassewell.googlecode.com/svn/trunk/2009/05/05/scribe_pipe"&gt;scribe_pipe&lt;/a&gt;. For example, you can pipe an Apache log file from the remote machine to the central scribe server by running:&lt;br /&gt;&lt;br /&gt;cat apache_access_log | ./scribe_pipe apache.access&lt;br /&gt;&lt;br /&gt;On the scribe server, you should see at this point a directory called apache.access under the main file_path directory, and files called apache.access_00000, apache.access_00001 etc (in chunks of max_size bytes).&lt;br /&gt;&lt;br /&gt;I'll post separately about actually using Scribe in production. I hope this post will at least get you started on using Scribe and save you some headaches during its installation process.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-1877175784560418350?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/1877175784560418350/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=1877175784560418350' title='1 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/1877175784560418350'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/1877175784560418350'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/10/compiling-installing-and-test-running.html' title='Compiling, installing and test-running Scribe'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>1</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-5992400591813266232</id><published>2009-10-06T10:38:00.000-07:00</published><updated>2009-10-06T10:38:17.076-07:00</updated><title type='text'>Brandon Burton on 'Automation is the cloud'</title><content type='html'>Great &lt;a href="http://www.inatree.org/2009/10/06/automation-is-the-cloud/"&gt;post&lt;/a&gt; from Brandon Burton, my ex-colleague at &lt;a href="http://reliam.com/"&gt;RIS/Reliam&lt;/a&gt;, on why automation is the foundation of cloud computing. Brandon discusses automation at various levels, starting with virtualization and networking, then moving up the layers and covering OS, configuration management and application deployment. Highly recommended.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-5992400591813266232?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/5992400591813266232/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=5992400591813266232' title='0 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/5992400591813266232'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/5992400591813266232'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/10/brandon-burton-on-automation-is-cloud.html' title='Brandon Burton on &apos;Automation is the cloud&apos;'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>0</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-6128979176228764327</id><published>2009-09-24T21:54:00.000-07:00</published><updated>2009-09-25T09:02:56.346-07:00</updated><title type='text'>Pybots success stories and a call for help</title><content type='html'>Any report of the death of the &lt;a href="http://pybots.org/"&gt;Pybots&lt;/a&gt; project is an exaggeration. But not by much. First, some history.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;Some history&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;The idea behind the &lt;b&gt;Pybots&lt;/b&gt; project is to allow people to run automated tests for their Python projects, while using Python binaries built from the very latest source code from the Python subversion repository.&lt;br /&gt;&lt;br /&gt;The idea originated from &lt;a class="reference" href="http://glyf.livejournal.com/"&gt;Glyph&lt;/a&gt;, of &lt;a class="reference" href="http://twistedmatrix.com/trac/"&gt;Twisted&lt;/a&gt; fame. He sent out a &lt;a class="reference" href="http://mail.python.org/pipermail/python-dev/2006-July/067366.html"&gt;message&lt;/a&gt; to the python-dev mailing list in which he said:&lt;br /&gt;&lt;blockquote&gt;&lt;i&gt;"I would like to propose, although I certainly don't have time to implement, a program by which Python-using projects could contribute buildslaves which would run their projects' tests with the latest Python trunk. This would provide two useful incentives: Python code would gain a reputation as generally well-tested (since there is a direct incentive to write tests for your project: get notified when core python changes might break it), and the core developers would have instant feedback when a "small" change breaks more code than it was expected to."&lt;/i&gt;&lt;br /&gt;&lt;div class="attribution"&gt;&lt;br /&gt;&lt;/div&gt;&lt;/blockquote&gt;This was back in July 2006. I volunteered to maintain a buildbot master (running on a server belonging to the PSF) and also to rally a community of people interested in running this type of tests. The hard part was (and still is) to find people willing to donate client machines to act as build slaves for a particular project, and even more so people willing to keep up with the status of their build slaves. The danger here, as in any continuous integratin system, is that once the status turns to red and doesn't go back to green, people start to ignore the failed steps. Even if those steps exhibit new and interesting failures, it's too late at this point (this is related to the &lt;a href="http://en.wikipedia.org/wiki/Broken_windows_theory"&gt;broken windows theory&lt;/a&gt;).&lt;br /&gt;&lt;br /&gt;The project starting fairly strong, gained some momentum, but then slowly ran out of steam. It was a combination of me not having the time to do the rallying, and of people not being interested in participating in the project anymore. At the height of its momentum, in early 2007, the Pybots farm consisted of 11 buildslaves running automated tests for more than 20 Python projects, including Twisted, Django, SQLAlchemy, MySQLdb, Bazaar, nose, twill, Storm, Trac, CherryPy, Genshi, Roundup. Pretty much a who's who of the Python project world.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;Early success stories&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;Here are some examples of bugs discovered by the buildslaves in the Python farm:&lt;br /&gt;&lt;ul&gt;&lt;li&gt;new keywords 'as' and 'with' in Python 2.6 causing &lt;a href="http://agiletesting.blogspot.com/2006/09/pay-attention-to-new-with-and-as.html"&gt;problems&lt;/a&gt; for projects that had variables with those names &lt;/li&gt;&lt;li&gt;Python install step failing even though all unit tests were passing (this underscores the &lt;a href="http://agiletesting.blogspot.com/2006/08/on-importance-of-functional-testing.html"&gt;importance of functional test&lt;/a&gt;&lt;a href="http://agiletesting.blogspot.com/2006/08/on-importance-of-functional-testing.html"&gt;ing&lt;/a&gt;)&lt;/li&gt;&lt;li&gt;platform-specific issues -- for example Bazaar issues on Windows due to TCP client behavior, Twisted issues on Red Hat 9 due to multicast behavior, Python core issues on OS X due to string formatting errors&lt;/li&gt;&lt;/ul&gt;(for a more thorough overview of the Pybots project, including lessons learned, see also my &lt;a href="http://agilistas.org/presentations/pycon07/pybots.pdf"&gt;PyCon07 presentation&lt;/a&gt;)&lt;br /&gt;&lt;br /&gt;&lt;b&gt;Recent signs of life and more success stories&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;In the last month or so there has been a flurry of activity related to the Pybots farm. It all started with an upgrade of the buildbot version on the machine hosting the Pybots buildmaster. This broke the master's configuration file, so the Pybots status page went completely dark.&lt;br /&gt;&lt;br /&gt;As a result, &lt;a href="http://holdenweb.blogspot.com/"&gt;Steve Holden&lt;/a&gt; posted a plea for help answered by a few people who showed interest in adding build slaves to the project. In parallel, &lt;a href="http://jcalderone.livejournal.com/"&gt;Jean-Paul Calderone&lt;/a&gt; jumped in to help on the buildmaster side and he managed to fix the buildbot upgrade issue (thanks, JP!) &lt;a href="http://www.traceback.org/"&gt;David Stanek&lt;/a&gt; also expressed interest in taking a more active role on the buildmaster side.&lt;span style="color: black;"&gt;&lt;/span&gt;&lt;br /&gt;&lt;br /&gt;Jean-Paul also sent more success stories to the &lt;a href="http://lists2.idyll.org/listinfo/pybots"&gt;Pybots mailing list&lt;/a&gt;. Here they are, verbatim, with his permission:&lt;br /&gt;&lt;br /&gt;&lt;i&gt;"&lt;b&gt;The skip story:&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;The Twisted pybots slave started skipping every Twisted test one day.  I noticed and filed &lt;a href="http://twistedmatrix.com/trac/ticket/3703" target="_blank"&gt;http://twistedmatrix.com/trac/&lt;wbr&gt;&lt;/wbr&gt;ticket/3703&lt;/a&gt; (which goes into a bit of detail about why this happened).  This happened to come up during the PyCon language summit, so there was some real-time discussion about it, resulting in a Python bug being filed, &lt;a href="http://bugs.python.org/issue5571" target="_blank"&gt;http://bugs.python.org/&lt;wbr&gt;&lt;/wbr&gt;issue5571&lt;/a&gt;.  Then, as that ticket shows, Benjamin Peterson was nice enough to fix the incompatibility.&lt;br /&gt;&lt;br /&gt;&lt;b&gt; The array/buffer story:&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;The Twisted pybots slave started to fail some Twisted tests one day. ;) The tests in question were actually calling into some PyCrypto code, so this failure wasn't in Twisted directly.  PyCrypto loads some bytes into an array.array and then tries to hash them (for some part of its random pool API).  I filed &lt;a href="http://bugs.python.org/issue6071" target="_blank"&gt;http://bugs.python.org/&lt;wbr&gt;&lt;/wbr&gt;issue6071&lt;/a&gt; on which someone explained that hashlib switched over to the new buffer API, lost support for hashing anything that only provides the old buffer API, and that array.array still only supports the old buffer API.  This one hasn't been fixed yet, but it sounds like Gregory Smith plans to fix it before 2.7 is released.&lt;br /&gt;&lt;br /&gt;There are other success stories too, incompatible changes that are more like bugs on the Twisted side than on the Python side (assuming one is generous and believes that incompatible changes in Python can actually be Twisted bugs ;).  Things like typos that didn't result in syntax errors in an older version of Python but became syntax errors in newer versions (in particular, a variable was defined as 0x+80000000 instead of 0x80000000 - the former actually being valid syntax in 2.5 but became illegal in 2.6)."&lt;/i&gt;   &lt;br /&gt;&lt;br /&gt;My hope is that stories like these will convince more people about the usefulness of running tests for their projects against 'live' changes in the Python trunk (or other Python branches). I am not aware of any other testing project that accomplishes this for other programming languages.&lt;br /&gt;&lt;br /&gt;In particular, if there is enough interest, we can also configure the Pybots master to trigger test runs for your project of choice &lt;b&gt;using Py3k binaries&lt;/b&gt;! Think how cool you'll appear to your grandchildren!&lt;br /&gt;&lt;br /&gt;&lt;b&gt;How you can help&lt;/b&gt;&lt;br /&gt;&lt;br /&gt;If you want to be involved in the Pybots project, please subscribe to the Pybots mailing list and show your interest by sending a message to the list. Here are some resources to get you started:&lt;br /&gt;&lt;br /&gt;&lt;ul&gt;&lt;li&gt;&lt;a href="http://agiletesting.blogspot.com/2006/08/setting-up-pybots-buildslave.html"&gt;Setting up a Pybots buildslave&lt;/a&gt; (scroll to the end for an update)&lt;/li&gt;&lt;li&gt;&lt;a href="http://wiki.python.org/moin/PyBotsFaq"&gt;Pybots FAQ&lt;/a&gt; &lt;/li&gt;&lt;li&gt;Pybots Google Code &lt;a href="http://code.google.com/p/pybots/"&gt;page&lt;/a&gt;&lt;br /&gt;&lt;/li&gt;&lt;li&gt;Examples of &lt;a href="http://pybots.googlecode.com/svn/trunk/slave/"&gt;buildslave scripts&lt;/a&gt; that trigger the automated builds and tests&lt;/li&gt;&lt;li&gt;Sidnei Da Silva's experiences on &lt;a href="http://lists2.idyll.org/pipermail/pybots/2006-September/000098.html"&gt;setting up a Win32 buildslave&lt;/a&gt;&lt;br /&gt;&lt;/li&gt;&lt;/ul&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-6128979176228764327?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/6128979176228764327/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=6128979176228764327' title='2 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/6128979176228764327'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/6128979176228764327'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/09/pybots-success-stories-and-call-for.html' title='Pybots success stories and a call for help'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>2</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-2672314302965645700</id><published>2009-09-24T11:25:00.000-07:00</published><updated>2009-09-24T11:25:06.380-07:00</updated><title type='text'>Jeff Roberts on a scalable DNS scheme for EC2</title><content type='html'>My ex-colleague from OpenX, Jeff Roberts, has another great blog post on '&lt;a href="http://www.vim-fu.com/better-dns-scheme-for-amazons-ec2-cloud/"&gt;A Scalable DNS Scheme for Amazon's EC2 Cloud&lt;/a&gt;'. If you need to deploy an internal DNS infrastructure in EC2, you have to read this post. It's based on battle-tested experience.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-2672314302965645700?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/2672314302965645700/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=2672314302965645700' title='0 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/2672314302965645700'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/2672314302965645700'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/09/jeff-roberts-on-scalable-dns-scheme-for.html' title='Jeff Roberts on a scalable DNS scheme for EC2'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>0</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-4485457284072909240</id><published>2009-09-14T16:07:00.000-07:00</published><updated>2009-09-14T16:29:39.711-07:00</updated><title type='text'>A/B testing and online experimentation at Microsoft</title><content type='html'>Via &lt;a href="http://glinden.blogspot.com/2009/09/experiments-and-performance-at-google.html"&gt;Greg Linden&lt;/a&gt;, I found a great presentation from Ronny Kohavi on "&lt;b&gt;&lt;a href="http://exp-platform.com/expMicrosoft.aspx"&gt;Online experimentation at Microsoft&lt;/a&gt;&lt;/b&gt;". All kinds of juicy nuggets of information on how to conduct meaningful A/B testing and other types of controlled online experiments.&lt;br /&gt;&lt;br /&gt;One of my favorite slides is 'Key Lessons', from which I quote:&lt;br /&gt;&lt;br /&gt;&lt;ul&gt;&lt;li&gt;&lt;i&gt;"Avoid the temptation to try and build optimal features through extensive planning without early testing of ideas"&lt;/i&gt;&lt;/li&gt;&lt;li&gt;&lt;i&gt;"Experiment often"&lt;/i&gt;&lt;/li&gt;&lt;li&gt;&lt;i&gt;"Try radical ideas. You may be surprised"&lt;/i&gt;&lt;/li&gt;&lt;/ul&gt;&lt;br /&gt;The entire presentation is highly recommended. You can tell that this wisdom was earned in the school of hard knocks, which is the best school there is in my experience, at least for software engineering.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-4485457284072909240?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/4485457284072909240/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=4485457284072909240' title='1 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/4485457284072909240'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/4485457284072909240'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/09/ab-testing-and-online-experimentation.html' title='A/B testing and online experimentation at Microsoft'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>1</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-4540382936631468861</id><published>2009-09-02T17:44:00.000-07:00</published><updated>2009-09-02T17:50:53.961-07:00</updated><title type='text'>Bootstrapping EC2 images as Puppet clients</title><content type='html'>I've been looking at &lt;a href="http://reductivelabs.com/trac/puppet/wiki"&gt;Puppet&lt;/a&gt; lately as an alternative to &lt;a href="http://code.google.com/p/slack/"&gt;slack&lt;/a&gt; for automated deployment and configuration management. I can't say I love it, but I think it's good enough that it warrants banging your head against the wall repeatedly until you learn how to use it. I do wish it was written in Python, but hey, you do what you need to do. I did look at &lt;a href="http://www.nongnu.org/fab/"&gt;Fabric&lt;/a&gt;, and I might still use it for 'push'-type deployments, but it has nowhere near the features that Puppet has (and its development and maintenance just &lt;a href="http://lists.gnu.org/archive/html/fab-user/2009-04/msg00041.html"&gt;changed hands&lt;/a&gt;, which makes it too cutting edge for me at this point.)&lt;br /&gt;&lt;br /&gt;But this is not a post about Puppet -- although I promise I'll blog about that too. This is a post on how to get to the point of using Puppet in an EC2 environment, by automatically configuring EC2 instances as Puppet clients once they're launched.&lt;br /&gt;&lt;br /&gt;While the mechanism I'll describe can be achieved by other means, I chose to use the Ubuntu EC2 AMIs provided by &lt;a href="http://alestic.com/"&gt;alestic&lt;/a&gt;. As a parenthesis, if you're thinking about using Ubuntu in EC2, do yourself a favor and read &lt;a href="http://twitter.com/esh"&gt;Eric Hammond&lt;/a&gt;'s blog (which can be found at alestic.com) He has a huge number of amazingly detailed posts related to this topic, and they're all worth your while to read.&lt;br /&gt;&lt;br /&gt;Unsurprisingly, I chose a mechanism provided by the alestic AMIs to bootstrap my EC2 instances -- specifically, passing &lt;a href="http://alestic.com/2009/06/ec2-user-data-scripts"&gt;user-data scripts&lt;/a&gt; that will be automatically run on the first boot of the instance. You can obviously also bake this into your own custom AMI, but the alestic AMIs already have this hook baked in, which I LIKE (picture Borat's voice). What's more, Eric kindly provides another way to easily run custom scripts within the main user-data script -- I'm referring to his runurl script, detailed in this &lt;a href="http://alestic.com/2009/08/runurl"&gt;blog post&lt;/a&gt;. Basically you point runurl at a URL that contains the location of another script that you wrote, and runurl will download and run that script. You can also pass parameters to runurl, which will in turn be passed to your script.&lt;br /&gt;&lt;br /&gt;Enough verbiage, let's see some examples.&lt;br /&gt;&lt;br /&gt;Here is my user-data file, whose file name I am passing along as a parameter when launching my EC2 instances:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;&lt;br /&gt;#!/bin/bash -ex&lt;br /&gt;&lt;br /&gt;cat &amp;lt;&amp;lt;EOL &amp;gt; /etc/hosts&lt;br /&gt; localhost.localdomain localhost&lt;br /&gt;  puppetmaster&lt;br /&gt;&lt;br /&gt;# The following lines are desirable for IPv6 capable hosts&lt;br /&gt;::1 ip6-localhost ip6-loopback&lt;br /&gt;fe00::0 ip6-localnet&lt;br /&gt;ff00::0 ip6-mcastprefix&lt;br /&gt;ff02::1 ip6-allnodes&lt;br /&gt;ff02::2 ip6-allrouters&lt;br /&gt;ff02::3 ip6-allhosts&lt;br /&gt;EOL&lt;br /&gt;&lt;br /&gt;wget -qO/usr/bin/runurl run.alestic.com/runurl&lt;br /&gt;chmod 755 /usr/bin/runurl&lt;br /&gt;runurl ec2web.mycompany.com/upgrade/apt&lt;br /&gt;runurl ec2web.mycompany.com/customize/ssh&lt;br /&gt;runurl ec2web.mycompany.com/customize/vim&lt;br /&gt;runurl ec2web.mycompany.com/install/puppet&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;&lt;br /&gt;The first thing I do in this script is to add an entry to the /etc/hosts file pointing at the IP address of my puppetmaster server. You can obviously do this with an internal DNS server too, but I've chosen not to maintain my own internal DNS servers in EC2 for now.&lt;br /&gt;&lt;br /&gt;My script then retrieves the runurl utility from alestic.com, puts it in /usr/bin and chmod's it to 755. Then the script uses runurl and points it at various other scripts I wrote, all hosted on an internal web server.&lt;br /&gt;&lt;br /&gt;For example, the contents of upgrade/apt are:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;&lt;br /&gt;#!/bin/bash&lt;br /&gt;apt-get update&lt;br /&gt;apt-get -y upgrade&lt;br /&gt;apt-get -y autoremove&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;&lt;br /&gt;For ssh customizations, my scripts downloads a specific .ssh/authorized_keys file, so I can ssh to the new instance using certain ssh keys.&lt;br /&gt;&lt;br /&gt;To install and customize vim, I have customize/vim:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;&lt;br /&gt;#!/bin/bash&lt;br /&gt;apt-get -y install vim&lt;br /&gt;wget -qO/root/.vimrc http://ec2web.mycompany.com/configs/os/.vimrc&lt;br /&gt;echo 'alias vi=vim' &amp;gt;&amp;gt; /root/.bashrc&lt;/pre&gt;&lt;br /&gt;&lt;br /&gt;...where .vimrc is a customized file that I keep under the document root of the same web server where I keep my scripts.&lt;br /&gt;&lt;br /&gt;Finally, install/puppet looks like this:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;&lt;br /&gt;#!/bin/bash&lt;br /&gt;apt-get -y install puppet&lt;br /&gt;wget -qO/etc/puppet/puppetd.conf http://ec2web.mycompany.com/configs/puppet/puppetd.conf&lt;br /&gt;/etc/init.d/puppet restart&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;&lt;br /&gt;Here I am installing puppet via apt-get, then I'm downloading a custom puppetd.conf configuration, which points at puppetmaster as its server name (instead of the default, which is puppet). Finally, I restart puppet so that the new configuration takes effect.&lt;br /&gt;&lt;br /&gt;Note that I want to keep these scripts to the bare minimum that allows me to:&lt;br /&gt;&lt;br /&gt;1) ssh into the instance in case anything goes wrong&lt;br /&gt;2) install and configure puppet so the instance can talk to the puppetmaster&lt;br /&gt;&lt;br /&gt;The actual package and application installations and customizations on my newly launched image will be done through puppet, by associating the instance hostname with a node that is defined on the puppetmaster; I am also adding more entries to /etc/hosts as needed using puppet-specific mechanisms such as the 'host' type (as promised, blog post on this forthcoming...)&lt;br /&gt;&lt;br /&gt;Note that you need to make sure you have good security for the web server instance which is serving your scripts to runurl; Eric Hammond talks about using S3 for that, but it's too complicated IMO (you need to sign URL and expire them, etc.) In my case, I preferred to use an internal Apache instance with basic HTTP authentication, and to only allow traffic on port 80 from certain security groups within EC2 (my Apache server doubles as the puppetmaster BTW).&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-4540382936631468861?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/4540382936631468861/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=4540382936631468861' title='11 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/4540382936631468861'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/4540382936631468861'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/09/bootstrapping-ec2-images-as-puppet.html' title='Bootstrapping EC2 images as Puppet clients'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>11</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-4852944756110092249</id><published>2009-09-02T15:05:00.000-07:00</published><updated>2009-09-02T15:05:44.049-07:00</updated><title type='text'>Is your hosting provider Reliam?</title><content type='html'>Some exciting news from RIS Technology, the hosting company I used to work for. They changed their name to &lt;a href="http://www.reliam.com/"&gt;Reliam&lt;/a&gt;, which stands for Reliable Internet Application Management. And I think it's an appropriate name, because RIS has always been much more involved into the application stack than your typical hosting provider. When I was there, we rolled out Django applications, Tomcat instances, MySQL, PostgreSQL and Oracle installations, and we maintained them 24x7, which required a deep understanding of the applications. We also provided the glue that tied all the various layers together, from deployment to monitoring.&lt;br /&gt;&lt;br /&gt;Since I left, RIS/Reliam has invested heavily in a virtual infrastructure that can be combined where it makes sense with physical dedicated servers. The DB layer is usually dedicated, since the closest you are to bare metal, the better off you are in terms of database access. But the application layer can easily be virtualized and scaled on demand. So you can the scaling benefit of cloud computing, and the performance benefit of dedicated servers.&lt;br /&gt;&lt;br /&gt;Here are some stats for the infrastructure that RIS/Reliam used for supporting traffic during the recent Miss Universe event (they host missuniverse.com and missusa.com):&lt;br /&gt;&lt;br /&gt;* 135 virtual servers running the Web application&lt;br /&gt;* 9 virtual servers running mysql-proxy&lt;br /&gt;* 1 master DB server and 5 read-only slave DB servers running MySQL&lt;br /&gt;* 301.52 Mbps bandwidth&lt;br /&gt;* 33,750 concurrent users&lt;br /&gt;* over 150K concurrent sessions per second&lt;br /&gt;&lt;br /&gt;An interesting note is that they used round robin DNS to load balance between the mysql proxies and had all proxies configured to use the master and all five slaves. They managed to get mysql-proxy 0.7.2 running with this &lt;a href="http://bazaar.launchpad.net/%7Ediego-fmpwizard/mysql-proxy/bug-43424/annotate/head%3A/lib/rw-splitting.lua"&gt;patch&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;So...what's the point of this note? It's a shout-out to my friends at RIS/Reliam, and a warm recommendation for them in case you need a hosting provider with strong technical capabilities that cover cloud/hybrid computing, system architecture design, application deployment and deep application monitoring and graphing.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-4852944756110092249?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/4852944756110092249/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=4852944756110092249' title='2 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/4852944756110092249'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/4852944756110092249'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/09/is-your-hosting-provider-reliam.html' title='Is your hosting provider Reliam?'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>2</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-7782930542264633664</id><published>2009-08-25T08:33:00.000-07:00</published><updated>2009-08-25T08:33:38.356-07:00</updated><title type='text'>New presentation and new job</title><content type='html'>I gave a presentation last night on 'Agile and Automated Testing Techniques and Tools' to the Pasadena Java User Group. It was a version of the talk I gave earlier this year to the XP/Agile SoCal User Group. This time I posted the &lt;a href="http://www.slideshare.net/ggheorghiu/agile-testing-pasadena-jug-aug2009-1902652"&gt;slides&lt;/a&gt; to Slideshare.&lt;br /&gt;&lt;br /&gt;If you look attentively at the first slide, you'll notice I have a new job at Evite, as a Sr. Systems Architect. I started a couple of weeks ago, and it's been great. Expect more blog posts on automated deployments to the cloud (using Ubuntu images from &lt;a href="http://alestic.com/"&gt;Alestic&lt;/a&gt;), on continuous integration/build/release management processes, on Hadoop, and on any interesting stuff that comes my way.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-7782930542264633664?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/7782930542264633664/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=7782930542264633664' title='4 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/7782930542264633664'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/7782930542264633664'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/08/new-presentation-and-new-job.html' title='New presentation and new job'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>4</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-5460072493207766003</id><published>2009-07-28T08:39:00.000-07:00</published><updated>2009-07-28T08:39:45.298-07:00</updated><title type='text'>noSQL databases? map-reduce? Erlang? it's all in this cartoon</title><content type='html'>Hilarious &lt;a href="http://highscalability.com/nsfw-hilarious-fault-tolerance-cartoon"&gt;cartoon&lt;/a&gt; (not sure why it's titled 'Fault Tolerance' though) seen on the &lt;a href="http://highscalability.com/"&gt;High Scalability&lt;/a&gt; blog. Captures very well the spirit and hype of our times in the IT world.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-5460072493207766003?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/5460072493207766003/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=5460072493207766003' title='3 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/5460072493207766003'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/5460072493207766003'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/nosql-databases-map-reduce-erlang-its.html' title='noSQL databases? map-reduce? Erlang? it&apos;s all in this cartoon'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>3</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-5717448781330085150</id><published>2009-07-27T14:30:00.000-07:00</published><updated>2009-07-27T14:31:34.659-07:00</updated><title type='text'>Python well represented in NASA's Nebula cloud</title><content type='html'>I found out today from the &lt;a href="http://groups.google.com/group/cloud-computing"&gt;cloud-computing mailing list&lt;/a&gt; about NASA's &lt;a href="http://nebula.nasa.gov/"&gt;Nebula&lt;/a&gt; project. Here's what the '&lt;a href="http://nebula.nasa.gov/about"&gt;About&lt;/a&gt;' page of the project's web site says:&lt;br /&gt;&lt;br /&gt;&lt;i&gt;"NEBULA is a Cloud Computing environment developed at NASA Ames Research Center, integrating a set of open-source components into a seamless, self-service platform. It provides high-capacity computing, storage and network connectivity, and uses a virtualized, scalable approach to achieve cost and energy efficiencies."&lt;/i&gt;&lt;br /&gt;&lt;br /&gt;The &lt;a href="http://nebula.nasa.gov/services"&gt;Services&lt;/a&gt; page has some nice architectural diagrams. I wasn't surprised to see that their VM enviroment is managed via &lt;a href="http://www.eucalyptus.com/"&gt;Eucalyptus&lt;/a&gt;. I also shouldn't have been surprised by the large number of Python modules and applications they're using, especially on the client side. Pretty much all the frontend applications are Python bindings for the various backend technologies they're using (such as LUSTRE, RabbitMQ, Subversion). Of course Trac is there too.&lt;br /&gt;&lt;br /&gt;But the most interesting thing for Python fans will be undoubtedly their selection for the Web application framework. Maybe again unsurprisingly, they chose...Django:&lt;br /&gt;&lt;br /&gt;&lt;i&gt;"After an extensive trade study, the NEBULA team selected Django, a python-based web application framework, as the first and primary application environment for the Cloud. NEBULA users have access to an extensive collection of open-source django "apps", providing features ranging from simple blogs, wikis, and discussion forums, to more advanced collaboration suites, image processing, and more."&lt;/i&gt;&lt;br /&gt;&lt;br /&gt;Other interesting tidbits from those diagrams:&lt;br /&gt;&lt;br /&gt;&lt;ul&gt;&lt;li&gt;deployments are automated with &lt;a href="http://www.nongnu.org/fab/"&gt;Fabric&lt;/a&gt;&lt;/li&gt;&lt;li&gt;distributed automated testing is done with &lt;a href="http://selenium-grid.seleniumhq.org/"&gt;Selenium Grid&lt;/a&gt;&lt;/li&gt;&lt;li&gt;continuous integration is done with &lt;a href="http://cruisecontrol.sourceforge.net/"&gt;CruiseControl &lt;/a&gt;&lt;br /&gt;&lt;/li&gt;&lt;li&gt;for the database backend, they use a MySQL cluster with &lt;a href="http://www.mysql.com/products/enterprise/drbd.html"&gt;DRBD&lt;/a&gt; &lt;/li&gt;&lt;li&gt;for the file system they use &lt;a href="http://wiki.lustre.org/index.php/Main_Page"&gt;LUSTRE&lt;/a&gt; &lt;/li&gt;&lt;li&gt;queuing is done with &lt;a href="http://www.rabbitmq.com/"&gt;RabbitMQ&lt;/a&gt; (which is written in Erlang)&lt;/li&gt;&lt;li&gt;search and indexing is done with &lt;a href="http://lucene.apache.org/solr/"&gt;SOLR&lt;/a&gt;&lt;/li&gt;&lt;/ul&gt;All in all, an interesting mix of technologies. Besides Python, Java and Erlang are well represented, as expected. Not a bad model to follow if you want to build your own private cloud environment.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-5717448781330085150?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/5717448781330085150/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=5717448781330085150' title='2 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/5717448781330085150'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/5717448781330085150'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/python-well-represented-in-nasas-nebula.html' title='Python well represented in NASA&apos;s Nebula cloud'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>2</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-229152473288311190</id><published>2009-07-26T10:18:00.000-07:00</published><updated>2009-07-26T10:18:56.636-07:00</updated><title type='text'>How to roll your own Amazon EC2 image</title><content type='html'>Jeff Roberts, the &lt;a href="http://www.vim-fu.com/"&gt;vim-fu&lt;/a&gt; guru, does it again with a great post on "&lt;a href="http://www.vim-fu.com/?p=416"&gt;Bundling versioned AMIs rapidly in Amazon's EC2&lt;/a&gt;". It's a step-by-step guide on how to roll your own AMI, bundle it and upload it to S3, while keeping it versioned at the same time. Highly recommended.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-229152473288311190?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/229152473288311190/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=229152473288311190' title='0 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/229152473288311190'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/229152473288311190'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/how-to-roll-your-own-amazon-ec2-image.html' title='How to roll your own Amazon EC2 image'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>0</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-927477432184067438</id><published>2009-07-21T12:17:00.000-07:00</published><updated>2009-07-21T12:17:44.230-07:00</updated><title type='text'>Automated testing of production deployments</title><content type='html'>When you work as a systems engineer at a company that has a large scale system infrastructure, sooner or later you realize that you need to automate pretty much everything you do. You can't afford not to, if you want to keep up with the ever-present demands of scaling up and down the infrastructure.&lt;br /&gt;&lt;br /&gt;The main promise of cloud computing -- infinite elastic scaling based on demand -- is real, but you can only achieve it if you &lt;a href="http://agiletesting.blogspot.com/2009/04/experiences-deploying-large-scale.html"&gt;automate your deployments&lt;/a&gt;. It's fairly safe to say that most teams that are involved in such infrastructures have achieved high levels of automation. Some fearless teams practice &lt;a href="http://startuplessonslearned.blogspot.com/search/label/continuous%20deployment"&gt;continuous deployment&lt;/a&gt;, others do frequent &lt;a href="http://agiletesting.blogspot.com/2009/07/dark-launching-and-other-lessons-from.html"&gt;dark launches&lt;/a&gt;. All these practices are great, but my thesis is that in order to achieve fearlessness you need &lt;b&gt;automated tests of your production deployments&lt;/b&gt;.&lt;br /&gt;&lt;br /&gt;Note the word '&lt;b&gt;production&lt;/b&gt;' -- I believe it is necessary to go one step beyond running automated tests in an isolated staging environment (although that is a very good thing to do, especially if staging mirrors production at a smaller scale). That next step is to run your test harness in production, every time you deploy. And deployment, at a fast moving Web company these days, can happen multiple times a day. Trust me, with no automated tests in place, you'll never get rid of that nagging feeling in the pit of your stomach that you might have broken things horribly, in &lt;b&gt;production&lt;/b&gt;.&lt;br /&gt;&lt;br /&gt;So how do you go about writing automated tests for your deployments? I wrote a while ago about &lt;a href="http://agiletesting.blogspot.com/2008/01/checklist-automation-and-testing.html"&gt;automating and testing your system setup checklists&lt;/a&gt;. Even testing small things such as 'is httpd/mysqld/postfix setup to run at boot time' will go a long way in achieving peace of mind.&lt;br /&gt;&lt;br /&gt;Assuming you have a list of things to test (it can be just a couple of critical things for starters), how and when do you run the tests? Again, you can do the simplest thing that works -- a bash shell that iterates through your production servers and runs the test scripts remotely on the servers via ssh. Some things I test this way these days are:&lt;br /&gt;&lt;br /&gt;* do local MySQL databases on servers in a particular cluster contain the same data in certain tables? (this shows me that things are in sync across servers)&lt;br /&gt;* is MySQL replication working as expected across the cluster of read-only slaves?&lt;br /&gt;* are periodic operations happening as expected (here I can do a simple tail of a log file to figure it out)&lt;br /&gt;* are certain PHP modules correctly installed?&lt;br /&gt;* is Apache serving a number of requests per second that is not too high, but not too low either (where high and low are highly dependent on your traffic and application obviously)&lt;br /&gt;&lt;br /&gt;I run these tests (and many others) each time I push a change to production. No matter how small the change can seem, it can have unanticipated side effects. I found that having tests that probe the system from as many angles as possible are the most efficient -- the angles in my case being Apache, MySQL, PHP, memcached for example. I also found that this type of testing (push-based if you want) is very good at showing discrepancies between servers. If you see a server being out of wack this way, then you know you need to attempt to fix it, or even terminate it and deploy a new one.&lt;br /&gt;&lt;br /&gt;Another approach in your automated testing strategy is to run your test harness periodically (via cron for example) and also to write the harness in a proper language (Python comes to mind), integrated into a test framework. You can have the results of the tests emailed to you in case of failure. The advantage of this approach is that you can have things run automatically without your intervention (in the first approach, you still have to remember to run the test suite!).&lt;br /&gt;&lt;br /&gt;The ultimate in terms of automated testing is to integrate it with your monitoring infrastructure. If you use Nagios for example, you can easily write plugins that essentialy probe for the same things that your tests probe for. The advantage of this approach is that the tests will run every time Nagios runs, and you can set up alerts easily. One disadvantage is that it can slow down your monitoring, depending on the number of tests you need to run on each server. Monitoring typically happens very often (every 5 minutes is a common practice), so it may be overkill to run all the tests every 5 minutes. Of course, this should be configurable in your monitoring tool, so you can have a separate class of checks that only happen every N hours for example.&lt;br /&gt;&lt;br /&gt;In any case, let me assure you that even if you take the first approach I mentioned (ssh into all servers and run commands remotely that way), you'll reap the rewards very fast. In fact, you'll like it so much that you'll want to keep adding more tests, so you can achieve more inner peace. It's a sure way to becoming test infected, but also to achieve deployment nirvana.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-927477432184067438?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/927477432184067438/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=927477432184067438' title='6 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/927477432184067438'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/927477432184067438'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/automated-testing-of-production.html' title='Automated testing of production deployments'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>6</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-883435935533908730</id><published>2009-07-17T11:35:00.000-07:00</published><updated>2009-07-17T11:35:45.051-07:00</updated><title type='text'>Managing multiple MySQL instances with MySQL Sandbox</title><content type='html'>MySQL doesn't support multi-master replication, i.e. you can't have one MySQL instance acting as a replication slave to more than one master. There are times when you need this functionality, for example for disaster recovery purposes, where you have a machine with tons of CPU, RAM and disk running several MySQL instances, each being a replication slave to a different MySQL master.&lt;br /&gt;&lt;br /&gt;One tool I've used for easy management of multiple MySQL instances on the same box is &lt;a href="http://mysqlsandbox.net/"&gt;MySQL Sandbox&lt;/a&gt;. It's nothing fancy -- a Perl module which offers a collection of scripts -- but it does make your life much easier.&lt;br /&gt;&lt;br /&gt;To install MySQL Sandbox, download it from its &lt;a href="https://launchpad.net/mysql-sandbox/+download"&gt;Launchpad page&lt;/a&gt;, then run 'perl Makefile.PL; make; make install'. You also need to download a MySQL binary tarball which will serve as a common base used by your MySQL instances.&lt;br /&gt;&lt;br /&gt;Here's an &lt;a href="http://agilistas.org/code/mysql-sandbox/setup_sandbox_instance.sh.txt"&gt;example&lt;/a&gt; of a script I wrote which creates a new MySQL Sandbox instance under a common directory (/var/mysql_slaves in my case). The script takes 2 arguments: a database name, and the name of the MySQL master from which that database is replicated. The script automatically increments the port number that the new sandbox instance will listen on, then creates the instance via a call like this:&lt;br /&gt;&lt;br /&gt;&lt;br /&gt;&lt;pre&gt;/usr/bin/make_sandbox /usr/local/src/mysql-5.1.32-linux-x86_64-glibc23.tar.gz \&lt;br /&gt;        --upper_directory=/var/mysql_slaves \&lt;br /&gt; --sandbox_directory=$SLAVEDB_NAME --sandbox_port=$LAST_PORT_NUMBER \&lt;br /&gt; --db_user=$SLAVEDB_NAME --db_password=PASSWORD \&lt;br /&gt; --no_confirm&lt;/pre&gt;&lt;br /&gt;As a result, there will be a new directory called $SLAVEDB_NAME under /var/mysql_slaves, which serves as the sandbox for the newly created MySQL instance. The script also adds some lines related to replication to the new MySQL instance configuration file (which is /var/mysql_slaves/my.sandbox.cnf).&lt;br /&gt;&lt;br /&gt;To start the instance, run &lt;br /&gt;&lt;pre&gt;/var/mysql_slaves/$SLAVEDB_NAME/start&lt;/pre&gt;&lt;br /&gt;To stop the instance, run&lt;br /&gt;&lt;pre&gt;/var/mysql_slaves/$SLAVEDB_NAME/start&lt;/pre&gt;&lt;br /&gt;To go to a MySQL prompt for this instance, run&lt;br /&gt;&lt;pre&gt;/var/mysql_slaves/$SLAVEDB_NAME/use&lt;/pre&gt;&lt;br /&gt;At this point, you still don't have a functioning slave. You need to load the data from the master. One way to do this is to run mysqldump on the master with options such as ' --single-transaction --master-data=1'. This will include the master information (binlog name and position) in the DB dump.&lt;br /&gt;&lt;br /&gt;The next step is to transfer the DB dump over to the box running MySQL Sandbox, and load it into the MySQL instance. I use a script similar to &lt;a href="http://agilistas.org/code/mysql-sandbox/create_and_load_instance_db.sh.txt"&gt;this&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;You should now have a MySQL instance that acts as a replication slave to a specific master server. Repeat this process to set up other sandboxed MySQL instances that are slaves to other masters.&lt;br /&gt;&lt;br /&gt;Note that MySQL Sandbox already includes some replication-related utilities (which I haven't used) and also an admin-type tool called sbtool. The &lt;a href="http://search.cpan.org/%7Egmax/MySQL-Sandbox-3.0.04/lib/MySQL/Sandbox.pm"&gt;documentation&lt;/a&gt; is pretty good.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-883435935533908730?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/883435935533908730/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=883435935533908730' title='0 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/883435935533908730'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/883435935533908730'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/managing-multiple-mysql-instances-with.html' title='Managing multiple MySQL instances with MySQL Sandbox'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>0</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-8851114287272432456</id><published>2009-07-14T07:25:00.000-07:00</published><updated>2009-07-14T07:25:46.572-07:00</updated><title type='text'>Kent Langley's '10 rules for launching a web site'</title><content type='html'>The advice in this &lt;a href="http://www.productionscale.com/home/2009/7/13/10-simple-practices-for-launching-web-sites-successfully.html"&gt;blog post&lt;/a&gt; by &lt;a href="http://www.productionscale.com/"&gt;Kent Langley&lt;/a&gt; resonates with my experiences launching Web infrastructures of all types, large and small. Deploy early and often, automate your deployments, use version control, create checklists, have a rollback plan -- these are all very sensible things to do.&lt;br /&gt;&lt;br /&gt;I would add one more very important thing that seems to be missing from the list: &lt;b&gt;have an extensive suite of automated tests&lt;/b&gt; to check that your deployment steps did the right thing. Many people just stop at the automation step, and don't go beyond that to the testing step. It will come back to haunt them in the long run. But this is fodder for another blog post, which will be coming real soon now ;-)&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-8851114287272432456?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/8851114287272432456/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=8851114287272432456' title='2 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/8851114287272432456'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/8851114287272432456'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/kent-langleys-10-rules-for-launching.html' title='Kent Langley&apos;s &apos;10 rules for launching a web site&apos;'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>2</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-1190118649994567205</id><published>2009-07-13T17:19:00.000-07:00</published><updated>2009-07-13T17:21:47.678-07:00</updated><title type='text'>Greatest invention since sliced bread: vimdiff</title><content type='html'>If you work at the ssh command prompt all day long (like I do), and if you need to compare text files and merge differences (like I do), then make sure you check out &lt;a href="http://www.vim.org/htmldoc/diff.html"&gt;vimdiff&lt;/a&gt; (thanks to &lt;a href="http://www.vim-fu.com/"&gt;Jeff Roberts&lt;/a&gt; for bringing it to my attention).&lt;br /&gt;&lt;br /&gt;If you run 'vimdiff file1 file2', the tool will split your screen vertically, with file1 displayed in a vim session on the left and file2 on the right. The differences between the 2 files will be highlighted. To jump from difference to difference, use &lt;b&gt;]c&lt;/b&gt; (forward) and &lt;b&gt;[c&lt;/b&gt; (backward). When the cursor is on a difference block, use &lt;b&gt;:diffget&lt;/b&gt; or &lt;b&gt;do&lt;/b&gt; to merge the difference from the other file into the file where the cursor is; use &lt;b&gt;:diffput&lt;/b&gt; or &lt;b&gt;dp&lt;/b&gt; to merge the other way. To jump from one file's window to the other, use &lt;b&gt;Ctrl-w-w&lt;/b&gt;. Google vimdiff for other tips and tricks. Definitely a good tool to have in your arsenal.&lt;br /&gt;&lt;br /&gt;If you have the luxury of a graphical enviroment, I also recommend &lt;a style="font-weight: bold;" href="http://meld.sourceforge.net/"&gt;meld&lt;/a&gt; (thanks to Chris Nutting for the tip).&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-1190118649994567205?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/1190118649994567205/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=1190118649994567205' title='2 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/1190118649994567205'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/1190118649994567205'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/greatest-invention-since-sliced-bread.html' title='Greatest invention since sliced bread: vimdiff'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>2</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-2100628087247323321</id><published>2009-07-13T14:57:00.000-07:00</published><updated>2009-07-13T14:57:51.085-07:00</updated><title type='text'>Recommended blog: Elastician</title><content type='html'>&lt;a href="http://www.elastician.com/"&gt;Elastician&lt;/a&gt; is the blog of Mitch Garnaat, the author of the amazingly useful &lt;a href="http://code.google.com/p/boto/"&gt;&lt;b&gt;boto&lt;/b&gt;&lt;/a&gt; Python library -- a collection of modules for managing AWS resources (EC2, S3, SQS,&amp;nbsp; SimpleDB and more recently&amp;nbsp;&lt;a href="http://www.elastician.com/2009/05/using-ec2-cloudwatch-in-boto.html"&gt;CloudWatch&lt;/a&gt;).&lt;br /&gt;&lt;br /&gt;Mitch has a great picture on what he calls the '&lt;a href="http://www.elastician.com/2009/05/cloud-computing-hierarchy-of-needs.html"&gt;Cloud Computing Hierarchy of Needs&lt;/a&gt;' (in a reference to Maslow's self-actualization &lt;a href="http://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs"&gt;hierarchy&lt;/a&gt;). Very insightful.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-2100628087247323321?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/2100628087247323321/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=2100628087247323321' title='0 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/2100628087247323321'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/2100628087247323321'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/recommended-blog-elastician.html' title='Recommended blog: Elastician'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>0</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-6539334617377595191</id><published>2009-07-10T08:54:00.000-07:00</published><updated>2009-07-10T08:54:30.099-07:00</updated><title type='text'>Python mock testing techniques and tools</title><content type='html'>This is an article I wrote for &lt;a href="http://pymag.phparch.com/"&gt;Python Magazine&lt;/a&gt; as part of the 'Pragmatic Testers' column. &lt;a href="http://ivory.idyll.org/blog"&gt;Titus&lt;/a&gt; and I have taken turns writing the column, although we haven't produced as many articles as we would have liked.&lt;br /&gt;&lt;br /&gt;Here is the content of my article, which appeared in the February 2009 issue of PyMag:&lt;br /&gt;&lt;br /&gt;Mock testing is a controversial topic in the area of unit testing. Some people swear by it, others swear at it. As always, the truth is somewhere in the middle. &lt;br /&gt;&lt;br /&gt;Let's get some terminology clarified: when people say they use mock objects in their testing, in most cases they actually mean stubs, not mocks. The difference is expanded upon with his usual brilliance by Martin Fowler in his article "&lt;a href="http://www.martinfowler.com/articles/mocksArentStubs.html"&gt;Mocks aren't stubs&lt;/a&gt;". &lt;br /&gt;&lt;br /&gt;In his revised version of the article, Fowler uses the terminology from Gerard Meszaros's 'xUnit Test Patterns' book. In this nomenclature, both stubs and mocks are special cases of 'test doubles', which are 'pretend' objects used in place of real objects during testing.&amp;nbsp; Here is Meszaros's definition of a &lt;a href="http://xunitpatterns.com/Test%20Double.html"&gt;test double&lt;/a&gt;:&lt;br /&gt;&lt;i&gt;&lt;br /&gt;&lt;/i&gt;&lt;br /&gt;&lt;i&gt;Sometimes it is just plain hard to test the system under test (SUT) because it depends on other components that cannot be used in the test environment. This could be because they aren't available, they will not return the results needed for the test or because executing them would have undesirable side effects. In other cases, our test strategy requires us to have more control or visibility of the internal behavior of the SUT. &lt;/i&gt;&lt;br /&gt;&lt;br /&gt;&lt;i&gt;When we are writing a test in which we cannot (or chose not to) use a real depended-on component (DOC), we can replace it with a Test Double. The Test Double doesn't have to behave exactly like the real DOC; it merely has to provide the same API as the real one so that the SUT thinks it is the real one!&amp;nbsp;&lt;/i&gt;&lt;br /&gt;&lt;br /&gt;These 'other components' that cannot be used in a test environment, or can only be used with a high setup cost, are usually &lt;b&gt;external resources&lt;/b&gt; such as database servers, Web servers, XML-RPC servers. Many of these resources may not be under your control, or may return data that often contains some &lt;b&gt;randomness&lt;/b&gt; which makes it hard or impossible for your unit tests to assert things about it.&lt;br /&gt;&lt;br /&gt;So what is the difference between stubs and mocks? Stubs are used to return &lt;b&gt;canned data&lt;/b&gt; to your SUT, so that you can make some assertions on how your code reacts to that data. This eliminates randomness from the equation, at least in the test environment. Mocks, on the other hand, are used to specify &lt;b&gt;expectations&lt;/b&gt; on the behavior of the object called by your SUT. You indicate your expectations by specifying that certain methods of the mock object need to be called by the SUT in a certain order and with certain arguments. &lt;br /&gt;&lt;br /&gt;Fowler draws a further distinction between stubs and mocks by saying that stubs are used for &lt;b&gt;“state verification”&lt;/b&gt;, while mocks are used for &lt;b&gt;“behavior verification”&lt;/b&gt;. When we use state verification, we assert things about the state of the SUT after the stub returned the canned data back to the SUT. We don't care &lt;b&gt;how&lt;/b&gt; the stub obtained that data, we just care about the final result (the data itself) and about how our SUT processed that data. When we use behavior verification, not only do we care about the data, but we also make sure that the SUT made the correct calls, in the correct order, and with the correct parameters, to the object representing the external resource.&lt;br /&gt;&lt;br /&gt;If readers are still following along after all this theory, I'm fairly sure they have at least two questions:&lt;br /&gt;&lt;br /&gt;1) when exactly do I use mock testing in my overall testing strategy?; &lt;i&gt;and&lt;/i&gt;&lt;br /&gt;2) if I do use mock testing, should I use mocks or stubs?&lt;br /&gt;&lt;br /&gt;I already mentioned one scenario when you might want to use mock testing: when your SUT needs to interact with external resources which are either not under your control, or which return data with enough randomness to make it hard for your SUT to assert anything meaningful about it (for example external weather servers, or data that is timestamped). Another area where mock testing helps is in simulating error conditions which are not always under your control, and which are usually hard to reproduce. In this case, you can mock the external resource, simulate any errors or exceptions you want, and see how your program reacts to them in your unit tests (for example, you can simulate various HTTP error codes, or database connection errors).&lt;br /&gt;&lt;br /&gt;Now for the second question, should you use mocks or stubs? In my experience, stubs that return canned data are sufficient for simulating the external resources and error conditions I mentioned. However, if you want to make sure that your application interacts correctly with these resources, for example that all the correct connection/disconnection calls are made to a database, then I recommend using mocks. One caveat of using mocks: by specifying expectations on the behavior of the object you're mocking and on the interaction of your SUT with that object, you couple your unit tests fairly tightly to the implementation of that object. With stubs, you only care about the external interface of the object you're mocking, not about the internal implementation of that object.&lt;br /&gt;&lt;br /&gt;Enough theory, let's see some practical examples. I will discuss some unit tests I wrote for an application that interacts with an external resource, in my case a SnapLogic server. I don't have the space to go into detail about &lt;a href="https://www.snaplogic.org/trac"&gt;SnapLogic&lt;/a&gt;, but it is a Python-based Open Source data integration framework. It allows you to unify the access to the data needed by your application through a single API. Behind the scenes, SnapLogic talks to database servers, CSV files, and other data sources, then presents the data to your application via a simple unified API. The main advantage is that your application doesn't need to know the particular APIs for accessing the various external data sources.&lt;br /&gt;&lt;br /&gt;In my case, SnapLogic talks to a MySQL database and presents the data returned by a SELECT SQL query to my application as a list of rows, where each row is itself a list. My application doesn't know that the data comes from MySQL, it just retrieves the data from the SnapLogic server via the SnapLogic API. I encapsulated the code that interacts with the SnapLogic server in its own class, which I called SnapLogicManager. My main SUT is passed a SnapLogicManager object in its __init__ method, then calls its methods to retrieve the data from the SnapLogic server.&lt;br /&gt;&lt;br /&gt;I think you know where this is going – SnapLogic is an external resource as far as my SUT is concerned. It is expensive to set up and tear down, and it could return data with enough randomness so I wouldn't be able to make meaningful assertions about it. It would also be hard to simulate errors using the real SnapLogic server. All this indicates that the SnapLogicManager object is ripe for mocking.&lt;br /&gt;&lt;br /&gt;My application code makes just one call to the SnapLogicManager object, to retrieve the dataset it needs to process:&lt;br /&gt;&lt;br /&gt;&lt;code&gt;&lt;br /&gt;rows = self.snaplogic_manager.get_attrib_values()&lt;br /&gt;&lt;/code&gt;&lt;br /&gt;&lt;br /&gt;Then the application processes the rows (list of lists) and instantiates various data structures based on the values in the rows. For the purpose of this article, I'll keep it simple and say that each row has an attribute name (element #0), and attribute value (element #1) and an attribute target (element #2). For example, an attribute could have the name “DocumentRoot”, the value “/var/www/mydocroot” and the target “apache”. The application expects that certain attributes are there with the correct target. If they're not, it raises an exception.&lt;br /&gt;&lt;br /&gt;How do we test that the application correctly instantiates the data structure, and correctly reacts to the presence or absence of certain attributes? You guessed it, we use a mock SnapLogicManager object, and we return canned data to our application.&lt;br /&gt;&lt;br /&gt;I will show here how to achieve this using two different Python mock testing frameworks: &lt;a href="http://code.google.com/p/pymox/wiki/MoxDocumentation"&gt;Mox&lt;/a&gt;, written by Google engineers, and &lt;a href="http://code.google.com/p/mock/"&gt;Mock&lt;/a&gt;, written by &lt;a href="http://www.voidspace.org.uk/python/weblog/index.shtml"&gt;Michael Foord&lt;/a&gt;.&lt;br /&gt;&lt;br /&gt;Mox is based on the Java EasyMock framework, and it does have a Java-esque feel to it, down to the CamelCase naming convention. Mock feels more 'pythonic' – more intuitive and with cleaner APIs. The two frameworks also differ in the way they set up and verify the mock objects: Mox uses a record/replay/verify pattern, whereas Mock uses an action/assert pattern. I will go into these differences by showing actual code below.&lt;br /&gt;&lt;br /&gt;Here is a unit test that uses Mox:&lt;br /&gt;&lt;br /&gt;&lt;code&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; def test_get_attrib_value_with_expected_target(self):&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # We return a SnapLogic dataset which contains attributes with correct targets&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; canned_snaplogic_rows = [&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; [u'DocumentRoot', u'/var/www/mydocroot', u'apache'],&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; [u'dbname', u'some_dbname', u'database'],&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; [u'dbuser', u'SOME_DBUSER', u'database'],&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; ]&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Create a mock SnapLogicManager&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mock_snaplogic_manager = mox.MockObject(SnapLogicManager)&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Return the canned list of rows when get_attrib_values is called&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mock_snaplogic_manager.get_attrib_values(self.appname, self.hostname).AndReturn(canned_snaplogic_rows)&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Put all mocks created by mox into replay mode&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mox.Replay(mock_snaplogic_manager)&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Run the test&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; myapp = MyApp(self.appname, self.hostname, mock_snaplogic_manager)&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; myapp.get_attr_values_from_snaplogic()&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Verify all mocks were used as expected&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mox.Verify(mock_snaplogic_manager)&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # We test that attributes with correct targets are retrieved correctly&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; assert '/var/www/mydocroot' == myapp.get_attrib_value_with_expected_target("DocumentRoot", "apache")&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; assert 'some_dbname' == myapp.get_attrib_value_with_expected_target("db_name", "database")&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; assert 'SOME_DBUSER' == myapp.get_attrib_value_with_expected_target("db_user", "database")&lt;br /&gt;&lt;/code&gt;&lt;br /&gt;&lt;br /&gt;Some explanations are in order. With the Mox framework, when you instantiate a MockObject, it is in 'record' mode, which means it's waiting for you to specify expectations on its behavior. You specify these expectations by telling the mock object what to return when called with a certain method. In my example, I tell the mock object that I want the list of canned rows to be returned when I call its 'get_attrib_values' method: mock_snaplogic_manager.get_attrib_values(self.appname, self.hostname).AndReturn(canned_snaplogic_rows)&lt;br /&gt;&lt;br /&gt;I only have one method that I am recording the expectations for in my example, but you could have several. When you are done recording, you need to put the mock object in 'replay' mode by calling &lt;code&gt;mox.Replay(mock_snaplogic_manager)&lt;/code&gt;. This means the mock object is now ready to be called by your application, and to verify that the expectations are being met.&lt;br /&gt;&lt;br /&gt;Then you call your application code, in my example by passing the mock object in the constructor of MyApp: &lt;code&gt;myapp = MyApp(self.appname, self.hostname, mock_snaplogic_manager)&lt;/code&gt;. My test then calls &lt;code&gt;myapp.get_attr_values_from_snaplogic()&lt;/code&gt;, which in turn interacts with the mock object by calling its get_attrib_values() method.&lt;br /&gt;&lt;br /&gt;At this point, you need to verify that the expectations you set happened correctly. You do this by calling the Verify method of the mock object: &lt;code&gt;mox.Verify(mock_snaplogic_manager)&lt;/code&gt;.&lt;br /&gt;&lt;br /&gt;If any of the methods you recorded were not called, or where called in the wrong order, or with the wrong parameters, you would get an exception at this point and your unit tests would fail.&lt;br /&gt;&lt;br /&gt;Finally, you also assert various things about your application, just as you would in any regular unit test. In my case, I assert that the &lt;code&gt;get_attrib_value_with_expected_target&lt;/code&gt; method of MyApp correctly retrieves the value of an attribute.&lt;br /&gt;&lt;br /&gt;This seems like a lot of work if all you need to do is to return canned data to your application. Enter the other framework I mentioned, Mock, which lets you specify canned return values very easily, and also allows you to assert certain things about the way the mock objects were called without the rigorous record/replay/verify pattern.&lt;br /&gt;&lt;br /&gt;Here's how I rewrote my unit test using Mock:&lt;br /&gt;&lt;br /&gt;&lt;code&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; def test_get_attrib_value_with_expected_target(self):&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # We return a SnapLogic dataset which contains attributes with correct targets&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; canned_snaplogic_rows = [&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; [u'DocumentRoot', u'/var/www/mydocroot', u'apache'],&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; [u'dbname', u'some_dbname', u'database'],&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; [u'dbuser', u'SOME_DBUSER', u'database'],&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; ]&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Create a mock SnapLogicManager&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mock_snaplogic_manager = Mock()&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Return the canned list of rows when get_attrib_values is called&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mock_snaplogic_manager.get_attrib_values.return_value = canned_snaplogic_rows&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Run the test&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; myapp = MyApp(self.appname, self.hostname, mock_snaplogic_manager)&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; myapp.get_attr_values_from_snaplogic()&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Verify that mocks were used as expected&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mock_snaplogic_manager.get_attrib_values.assert_called_with(self.appname, self.hostname)&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # We test that attributes with correct targets are retrieved correctly&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; assert '/var/www/mydocroot' == myapp.get_attrib_value_with_expected_target("DocumentRoot", "apache")&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; assert 'some_dbname' == myapp.get_attrib_value_with_expected_target("db_name", "database")&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; assert 'SOME_DBUSER' == myapp.get_attrib_value_with_expected_target("db_user", "database")&lt;br /&gt;&lt;/code&gt;&lt;br /&gt;&lt;br /&gt;As you can see, Mock allows you to specify the return value for a given method of the mock object, in my case for the 'get_attrib_values' method. Mock also allows you to verify that the method has been called with the correct arguments. I do that by calling &lt;code&gt;assert_called_with&lt;/code&gt; on the mock object. If you just want to verify that the method has been called at all, with no regard to the arguments, you can use &lt;code&gt;assert_called&lt;/code&gt;.&lt;br /&gt;&lt;br /&gt;There are many other things you can do with both Mox and Mock. Space doesn't permit me to go into many more details here, but I strongly encourage you to read the documentation and try things out on your own. &lt;br /&gt;&lt;br /&gt;Another technique I want to show is how to simulate exceptions using the Mox framework. In my unit tests, I wanted to verify that my application reacts correctly to exceptions thrown by the SnapLogicManager class. Those exception are thrown for example when the SnapLogic server is not running. Here is the unit test I wrote:&lt;br /&gt;&lt;br /&gt;&lt;code&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; def test_get_attr_values_from_snaplogic_when_errors(self):&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # We simulate a SnapLogicManagerError and verify that it is caught properly&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Create a mock SnapLogicManager&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mock_snaplogic_manager = mox.MockObject(SnapLogicManager)&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Simulate a SnapLogicManagerError when get_attrib_values is called&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mock_snaplogic_manager.get_attrib_values(self.appname, self.hostname).AndRaise(SnapLogicManagerError('Boom!'))&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Put all mocks created by mox into replay mode&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mox.Replay(mock_snaplogic_manager)&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Run the test&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; myapp = MyApp(self.appname, self.hostname, mock_snaplogic_manager)&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; myapp.get_attr_values_from_snaplogic()&lt;br /&gt;&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Verify all mocks were used as expected&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; mox.Verify(mock_snaplogic_manager)&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; # Verify that MyApp caught and logged the exception&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; line = get_last_line_from_log(self.logfile)&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; assert re.search('myapp - CRITICAL - get_attr_values_from_snaplogic --&amp;gt; SnapLogicManagerError: \'Boom!\'', line)&lt;br /&gt;&lt;/code&gt;&lt;br /&gt;&lt;br /&gt;I used the following Mox API for simulating an exception: &lt;code&gt;mock_snaplogic_manager.get_attrib_values(self.appname, self.hostname).AndRaise(SnapLogicManagerError('Boom!')).&lt;br /&gt;&lt;/code&gt;&lt;br /&gt;&lt;br /&gt;To verify that my application reacted correctly to the exception, I checked the application log file, and I made sure that the last line logged contained the correct exception type and value.&lt;br /&gt;&lt;a href="http://draft.blogger.com/goog_1247240264437"&gt;&lt;br /&gt;&lt;/a&gt;&lt;br /&gt;Space does not permit me to show a Python-specific mock testing technique which for lack of a better name I call 'namespace overriding' (actually this is a bit like monkey patching, but for testing purposes; so maybe we can call it monkey testing?). I refer the reader to my blog post on '&lt;a href="http://agiletesting.blogspot.com/2006/12/mock-testing-examples-and-resources.html"&gt;Mock testing examples and resources&lt;/a&gt;' and I just quickly describe here the technique. Imagine that one of the methods of your application calls &lt;code&gt;urllib.urlretrieve&lt;/code&gt; in order to download a file from an external Web server. Did I say external Web server, as in 'external resource not under your control'? I did, so you know that mock testing will help. My blog post shows how you can write a mocked_urlretrieve function, and override the name urllib.urlretrieve in your unit tests with your mocked version mocked_urlretrieve. Simple and elegant. The blog post also shows how you can return various canned valued from the mocked version of urlretrieve, based on different input values.&lt;br /&gt;&lt;br /&gt;I started this article by saying that mock testing is a controversial topic in the area of unit testing. Many people feel that you should not use mock testing because you are not testing your application in the presence of the real objects on which it depends, so if the code for these objects changes, you run the risk of having your unit tests pass even though the application will break when it interacts with the real objects. This is a valid objection, and I don't recommend you go overboard with mocking every single interaction in your application. Instead, limit your mock testing, as I said in this article, to resources whose behavior and returned data are hard to control.&lt;br /&gt;&lt;br /&gt;Another important note: whatever your unit testing strategy is, whether you use mock testing techniques or not, do not forget that you also need to have functional test and integration tests for your application. Integration tests especially do need to exercise all the resources that your application needs to interact with. For more information on different types of testing that you need to consider, please see my blog posts '&lt;a href="http://agiletesting.blogspot.com/2006/04/should-acceptance-tests-be-included-in.html"&gt;Should acceptance tests be included in the continuous build process?&lt;/a&gt;' and '&lt;a href="http://agiletesting.blogspot.com/2006/08/on-importance-of-functional-testing.html"&gt;On the importance of functional testing&lt;/a&gt;'.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-6539334617377595191?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/6539334617377595191/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=6539334617377595191' title='2 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/6539334617377595191'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/6539334617377595191'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/python-mock-testing-techniques-and.html' title='Python mock testing techniques and tools'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>2</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-4472023454739813778</id><published>2009-07-09T13:51:00.000-07:00</published><updated>2009-07-09T13:51:10.984-07:00</updated><title type='text'>Setting system-wide environment variables on RedHat-based machines</title><content type='html'>I keep forgetting this, so I'm committing it to long-term memory. If you have a RedHat-based operating system (RH, CentOS etc) and you need to set certain environment variables so they're available to every user, one good place to do it is to drop a script ending in .sh in /etc/profile.d. Then export your desired environment variables there.&lt;br /&gt;&lt;br /&gt;Here's an example from a CentOS machine I have:&lt;br /&gt;&lt;br /&gt;&lt;pre class="code"&gt;# cd /etc/profile.d&lt;br /&gt;# cat java.sh &lt;br /&gt;export JAVA_HOME=/usr/java/default&lt;br /&gt;&lt;/pre&gt;&lt;br /&gt;Note that you can do whatever else you need in these scripts -- for example you can set up aliases etc. Every script in /etc/profile.d which ends in .sh gets sourced in /etc/profile.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-4472023454739813778?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/4472023454739813778/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=4472023454739813778' title='1 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/4472023454739813778'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/4472023454739813778'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/setting-system-wide-environment.html' title='Setting system-wide environment variables on RedHat-based machines'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>1</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-5023362329211972525</id><published>2009-07-06T14:50:00.000-07:00</published><updated>2009-07-06T14:50:19.299-07:00</updated><title type='text'>Resource monitoring and graphing with Cacti in EC2</title><content type='html'>My colleague &lt;a href="http://www.vim-fu.com/"&gt;Jeff Roberts&lt;/a&gt; just posted a blog entry on '&lt;a href="http://www.vim-fu.com/?p=300"&gt;Using Cacti to monitor a large scale infrastructure in Amazon’s EC2&lt;/a&gt;'. Highly recommended for people interested in monitoring and graphing their system resources across hundreds of nodes in Amazon EC2.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-5023362329211972525?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/5023362329211972525/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=5023362329211972525' title='0 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/5023362329211972525'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/5023362329211972525'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/resource-monitoring-and-graphing-with.html' title='Resource monitoring and graphing with Cacti in EC2'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>0</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-9159460486427664011</id><published>2009-07-02T14:29:00.000-07:00</published><updated>2009-07-02T14:29:58.326-07:00</updated><title type='text'>Dark launching and other lessons from Facebook on massive deployments</title><content type='html'>I came across this &lt;a href="http://www.facebook.com/note.php?note_id=96390263919"&gt;note&lt;/a&gt; from the Engineering team at Facebook which talks about how they managed to smoothly launch their recent 'pick a username' feature. The title of the note is, appropriately enough, 'Hammering usernames' -- this is of course because they were expecting their infrastructure to be hammered.&lt;br /&gt;&lt;br /&gt;In the note I saw for the first time a name for a strategy that teams I've been involved with have applied before: '&lt;span style="font-weight: bold;"&gt;dark launching&lt;/span&gt;'. Essentially, dark launching is releasing a new feature to a subset of your users, mostly with no UI changes, but otherwise exercising all the parts of your infrastructure involved in serving that feature. A good strategy to apply when you're dealing with massive, large-scale deployments, and when you want to see how your infrastructure behaves in conditions that are as close to production as possible. Because remember, there's nothing like production! Your careful load/stress testing exercises in a lab environment ain't gonna cut it.&lt;br /&gt;&lt;br /&gt;The note from Facebook has all sorts of other nuggets of wisdom related to massive infrastructure deployments. I recommend subscribing to the RSS feed for 'Engineering @ Facebook's Notes'.&lt;br /&gt;&lt;br /&gt;While googling for 'dark launching', I also came across this very good &lt;a href="http://www.25hoursaday.com/weblog/2008/06/19/DarkLaunchesGradualRampsAndIsolationTestingTheScalabilityOfNewFeaturesOnYourWebSite.aspx"&gt;post&lt;/a&gt; by Dare Obasanjo. Recommended reading.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-9159460486427664011?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/9159460486427664011/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=9159460486427664011' title='4 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/9159460486427664011'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/9159460486427664011'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/07/dark-launching-and-other-lessons-from.html' title='Dark launching and other lessons from Facebook on massive deployments'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>4</thr:total></entry><entry><id>tag:blogger.com,1999:blog-9238405.post-6327449147692714713</id><published>2009-06-30T11:38:00.000-07:00</published><updated>2009-06-30T11:38:12.310-07:00</updated><title type='text'>A redbot from mnot</title><content type='html'>mnot == &lt;a href="http://www.mnot.net/personal/resume.html"&gt;Mark Nottingham&lt;/a&gt; (RESTful Web Services guru and &lt;a href="http://www.mnot.net/blog/"&gt;blogger&lt;/a&gt;)&lt;br /&gt;&lt;br /&gt;redbot == &lt;a href="http://mnot.github.com/redbot/"&gt;Resource Expert Droid&lt;/a&gt; == a testing tool written in Python that &lt;i&gt;"checks HTTP resources to see how they use HTTP, makes suggestions, and finds common protocol mistakes"&lt;/i&gt;&lt;br /&gt;&lt;br /&gt;Use this tool if you want to see how well your Web application does in terms of HTTP connections, content negotiation, caching and validation. Very useful both for traditional Web sites and for Web services.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/9238405-6327449147692714713?l=agiletesting.blogspot.com'/&gt;&lt;/div&gt;</content><link rel='replies' type='application/atom+xml' href='http://agiletesting.blogspot.com/feeds/6327449147692714713/comments/default' title='Post Comments'/><link rel='replies' type='text/html' href='https://www.blogger.com/comment.g?blogID=9238405&amp;postID=6327449147692714713' title='0 Comments'/><link rel='edit' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/6327449147692714713'/><link rel='self' type='application/atom+xml' href='http://www.blogger.com/feeds/9238405/posts/default/6327449147692714713'/><link rel='alternate' type='text/html' href='http://agiletesting.blogspot.com/2009/06/redbot-from-mnot.html' title='A redbot from mnot'/><author><name>Grig Gheorghiu</name><uri>http://www.blogger.com/profile/17863511617654196370</uri><email>noreply@blogger.com</email><gd:extendedProperty xmlns:gd='http://schemas.google.com/g/2005' name='OpenSocialUserId' value='02287375960894298151'/></author><thr:total xmlns:thr='http://purl.org/syndication/thread/1.0'>0</thr:total></entry></feed>