Forget Chef or Puppet – Automate with Sprinkle

Miso has a relatively standard server architecture for a medium-level traffic Rails application. We use Linode to host our application and we provision a number of VPS instances that make up our infrastructure. We have a load balancer equipped with Nginx and Varnish, we have an application server that runs our Rails application to serve dynamic requests, we have a master-slave database setup, and a cache server that has memcached and redis.

Early on, we manually setup these servers by hand by installing packages, compiling libraries, tweaking configurations and installing gems. We then manually documented these steps to a company wiki which would allow us to mechanically follow the steps and setup a new instance of any of our servers.

Automating our Setup

We decided recently that we wanted a more robust way of provisioning and configuring our different types of servers. Mechanically following steps from a wiki is inefficient, error-prone and likely to become inaccurate. What we needed was a system that would allow us to provision a new application or database server with just a single command. Here were additional requirements for our desired setup:

  • Simple to setup and configure
  • Lightweight system using familiar syntax
  • Preferably utilizing ssh and commands in the vein of Capistrano
  • Modular components that can be assembled to setup each server
  • Locally executable such that I can provision a server from my local machine
  • Doesn’t require the target machine to have anything installed prior

The set-up and provisioning tools which are most popular for open-source seem to be Puppet and Chef. Both are great tools that have a lot of features and flexibility, and are well-suited when you have a large infrastructure with tens or hundreds of servers for a project. However, Chef and Puppet recipes are not trivial to create or manage and by default they are intended to be managed from a remote server. We felt that there was unnecessary complexity and overhead for our simple provisioning purposes. Thanks for the corrections in the comments regarding the remote server requirement for Puppet/Chef.

For smaller infrastructures like ours, we felt a better tool would be easier to manage and setup. Ideally, something that is akin to deploying our code with Capistrano. A tool that can be managed and run locally and that can be maintained easily. After some exploration, we stumbled upon a ruby tool called Sprinkle, probably best known by one example automated script called Passenger Stack.

There are several aspects of Sprinkle that made this our tool of choice. For one, it is locally managed and the setup is done leveraging the rock-solid Capistrano deployment system. Also, even though Sprinkle is written in Ruby, the tool does not require Ruby or anything else to be installed on the target servers since the automated setup is executed on your development machine and communicates with the remote server using only SSH. The best part is that there are only a few concepts required to understand and use this system.

Understanding Sprinkle

The rest of this article is intended to be a long and comprehensive overview of Sprinkle. While I have read many blog posts about Sprinkle written across several different years, I hadn’t seen a post that covered each aspect of Sprinkle in full detail. The goal is that by the end of this post, you should be able to understand and write Sprinkle scripts as well as execute them. Let’s start off by exploring the four major concepts that make up Sprinkle and that will be used to build out your server recipes: Packages, Policies, Installers, and Verifiers.


A package defines one or more things to provision onto the server. There is a lot of flexibility in a way a package is defined but fundamentally this represents a “component” can be installed. Packages are sets of installations, options, and verifications, grouped under a meaningful name. The basic structure of a package is like this:

# packages/some_name.rb
package :some_name do
  description 'Some text'
  version '1.2.3'
  requires :another_package

  # ...installers...

  verify { ...verifiers... }

Note that defining a package does not install the package by default. A package is only installed when explicitly mentioned in a policy. You can also specify recommendations and/or optional package dependencies in a package as well:

# packages/foo_name.rb
package :foo_name do
  requires :another_package
  recommends :some_package
  optional :other_package

  # ...installers...
  verify { ...verifiers... }

You can also create virtual aliased packages that are various alternatives for the same component type. For instance, if I wanted to give people a choice over the database to use when provisioning:

# packages/database.rb
package :sqlite3, :provides => :database do
  # ...installers and verifiers...

package :postgresql, :provides => :database do
  # ...installers and verifiers...

package :mysql, :provides => :database do
  # ...installers and verifiers...

You can now reference that you want to install a :database and the script will ask you which provision you want to install. For more information on packages, the best place is looking at the source file itself.


A policy defines a set of particular “packages” that are required for a certain server (app, database, etc). All policies defined will be run and all packages required by the policy will be installed. So whereas defining a “package” merely defines it, defining a “policy” actually causes those packages to be installed. A policy is very simple to define:

# Define target machine used for the "app" role
role :app, ""
# Installs the packages specified with a 'requires' when the script executes
policy :myapp, :roles => :app do
  requires :some_package
  requires :nginx
  requires :postgresql
  requires :rails

A role merely defines what server the commands are run on. This way, a single Sprinkle script can provision an entire group of servers specifying different roles similar to using Capistrano directly. You may specify as many policies as you’d like. If the packages you’re requiring are properly defined with verification blocks, then no software will be installed twice, so you may require a webserver on multiple packages within the same role without having to wait for that package to install repeatedly.

For more information on policies, the best place is looking at the source file itself.


Installers are mechanisms for getting software onto the target machine. There are different scripts that allow you to download/compile software from a given source, use packaging systems (such as apt-get), copy files, install a gem, or even just inject content into existing files. Common examples of installers are detailed below.

To run an arbitrary command on the server:

package :foo do
  # Can be any line executed in the shell
  runner "run_some_command --now"

To install using the aptitude package manager in Ubuntu:

package :foo do
  apt 'foo-package'
  # Supports only installing dependencies
  apt('bar-package') { dependencies_only true }

That would install the ‘foo-package’ when you run the ‘foo’ package. To install a ruby gem:

package :foo do
  gem 'foo-gem'
  # Supports specifying version, source, repository, build_docs and build_flags
  gem 'bar-gem' do
    version "1.2.3"
    source ''
    build_docs false

To upload arbitrary text into a file on the target:

package :foo do
  push_text 'some random text', '/etc/foo/bar.conf'
  # Supports sudo access for a file
  push_text 'some random text', '/etc/foo/bar.conf', :sudo => true

You can also replace text in a target file:

# packages/foo.rb
package :foo do
  replace_text 'original foo', 'replacement bar', '/etc/foo/bar.conf'
  # Supports sudo access for a file
  replace_text 'original foo', 'replacement bar', '/etc/foo/bar.conf', :sudo => true

To transfer a file to a remote target file:

package :foo do
  # Transfers are recursive by default so whole directories can be moved
  transfer 'file/some_folder', '/etc/some_folder'
  # Supports sudo access for a file
  transfer 'file/foo.file', '/etc/foo.file', :sudo => true
  # Also a file can have "render" passed which runs the template through erb 
  # You can access variables to output the file dynamically, or pass explicit locals
  foo_port = 8080
  transfer 'file/foo.file', '/etc/foo.file', :render => true, 
            :locals => { :bar_port => 80 }

To run a rake task as part of a package on target:

package :foo, :rakefile => "/path/to/Rakefile" do
  rake 'foo-task'

To install a library from a given source path:

package :foo do
  source ''
  # Supports prefix, builds, archives, enable, with, and more
  source '' do
    prefix    '/usr/local'
    archives  '/tmp'
    builds    '/tmp/builds'
    with      'pgsql'

Installers also support installation hooks at various points during the install process which vary depending on the installer used. An example of how to use hooks is as follows:

package :foo do
  apt 'foo-package' do
    pre :install, 'echo "Beginning install..."'
    post :install, 'echo "Completing install!"'

Multiple hooks can be specified for any given step and each installer has multiple steps for which hooks can be configured. For more information on installers, the best place is looking at the source folder itself.


Verifiers are convenient helpers which you can use to check if something was installed correctly. You can use this helper within a verify block for a package. Sprinkle runs the verify block to find out whether or not something is already installed on the target machine. This way things never get done twice and if a package is already installed, then the task will be skipped.

Adding a verify block for every package is extremely important, be diligent to have an appropriate verifier for every installer used in a package. This will make the automated scripts much more robust and reusable on any number of servers. This also ensures that an installer works as expected and tests the server after installation as well.

There are many different types of verifications, for each one there are installers for which they are particularly useful. For instance, if I wanted to see if an aptitude package was installed correctly:

package :foo do
  apt 'foo-package'
  apt 'bar-package'
  verify do
    has_apt 'foo-package'
    has_apt 'bar-package'

This will only install the package on a target if not already installed and verifies the installation after the package runs. If we wanted to check that a gem exists:

package :foo do
  gem 'foo-gem', :version => "1.2.3"
  gem 'bar-gem'
  verify do
    has_gem 'foo-gem', '1.2.3'
    # or verify that ruby can require a gem
    ruby_can_load 'bar-gem'

If you want to check if a directory, file or executable exists:

package :foo do
  mkdir '/var/some/dir'
  touch 'var/some/file'
  runner 'touch /usr/bin/abinary' do
    post :install, "chmod +x /usr/bin/abinary"

  verify do
    has_directory '/var/some/dir'
    has_file      '/etc/apache2/apache2.conf'
    has_executable 'abinary'

You can also check if a process is running:

package :foo do
  apt 'memcached'

  verify do
    has_process 'memcached'

For more information on verifiers, the best place is looking at the source folder itself.

Putting Everything Together

Once you understand the aforementioned concepts, building automated recipes for provisioning becomes quite straightforward. Simply define packages (with installers and verifiers) and then group them into ‘policies’ that run on target machines. Generally, you can have a deploy.rb file and an install.rb file that are defined as follows:

# deploy.rb
# SSH in as 'root'. Probably not the best idea.
set :user, 'root'
set :password, 'secret'

# Just run the commands since we are 'root'.
set :run_method, :run

# Be sure to fill in your server host name or IP.
role :app, '83.434.34.234'

default_run_options[:pty] = true

The install file tends to define the various policies for this sprinkle script:

# install.rb
require 'packages/essential'
require 'packages/git'
require 'packages/nginx'
require 'packages/rails'
require 'packages/mongodb'

policy :myapp, :roles => :app do
  requires :essential
  requires :git
  requires :nginx
  requires :rails
  requires :mongodb

deployment do
  delivery :capistrano

  source do
    prefix   '/usr/local'
    archives '/usr/local/sources'
    builds   '/usr/local/build'

Then you should store your packages in a subfolder aptly named ‘packages’:

# packages/git.rb
package :git, :provides => :scm do
  description 'Git version control client'
  apt 'git-core'

  verify do
    has_executable 'git'

You can store your assets (configuration files, etc) in “assets” folder and access them from your packages to upload. Once all the packages and policies have been defined appropriately you can execute the sprinkle script on the command line with:

sprinkle -c -s install.rb

And sprinkle is off to the races, setting up all the policies on the target machines.

Further Reading

There are several other good posts about Sprinkle:

In addition, there are a lot of good examples of sprinkle recipes:

Let me know what you think of all this in the comments and if you have any related comments or questions. What do you use to automate and manage your servers?

Vendor – Bringing Bundler to iOS

Using and sharing iOS libraries is tragically difficult given the maturity of the framework, the number of active developers, and the size of the open source community.  Compare with Rails which has RubyGems and Bundler, it’s no surprise that Rails/Ruby has a resource like The Ruby Toolbox, while iOS development has…nothing?

Are iOS developers fundamentally less collaborative than Rails developers?  When developing on Rails, if I ever find myself developing anything remotely reusable, I can almost always be certain that there is a gem for it (probably with an obnoxiously clever name).

I don’t think the spirit of collaboration is lacking in the iOS developer community; rather, there are a few fundamental challenges with iOS library development:

  • Libraries are hard to integrate into a project.  Granted, it’s not *that* hard to follow a brief set of instructions, but why can’t this process be more streamlined?
  • No standardized versioning standard.  Once a library is integrated into a project, there is no standard way of capturing which version of the library was used.
  • No dependency specification standard (this is a big problem).  Why does facebook-ios-sdk embed it’s own JSON library when there are better ones available?  So many libraries come embedded with common libraries that we all use – and worse than that, who knows what version they’re using!  Not only can this lead to duplicate symbols, but library developers essentially have to start from scratch instead of leveraging other existing libraries.

Of course, coming up with a naming, versioning, and dependency standard for iOS libraries and convincing everyone to adopt it is a daunting task.  One possible approach is follow the example of Homebrew, a popular package manager for OS X.  Homebrew turned installing and updating packages on OS X into a simple process.  Instead of convincing everyone to comply to some standard, Homebrew maintains a set of formulas that helps describe commonly used packages.  These formulas allow Homebrew to automate the installation process as well as enforce dependencies.  This works well for Homebrew, although it puts the burden of maintaining package specifications in one place, rather then distributed as it is with Ruby gems.

There seems to be a need for some type of solution here.  When we write Rails libraries, we write the readme first to help us understand what problem we’re trying to solve (Readme Driven Development).  Below is the readme for an iOS packaging system called Vendor.

Vendor – an iOS library management system

Vendor makes the process of using and managing libraries in iOS easy.  Vendor leverages the XCode Workspaces feature introduced with XCode 4 and is modeled after Bundler. Vendor streamlines the installation and update process for dependent libraries.  It also tracks versions and manages dependencies between libraries.

Step 1) Specify dependencies

Specify your dependencies in a Vendors file in your project’s root.

source ""
lib "facebook-ios-sdk"  # Formula specified at source above
lib "three20"
lib "asi-http-request", :git => ""
lib "JSONKit", :git => ""

Step 2) Install dependencies

vendor install
git add Vendors.lock

Installing a vendor library gets the latest version of the code, and adds the XCode project to the workspace.  As part of the installation process, the library is set up as a dependency of the main project, header search paths are modified, and required frameworks are added.  The installed version of the library is captured in the Vendors.lock file.

After a fresh check out of a project from source control, the XCode workspace may contain links to projects that don’t exist in the file system because vendor projects are not checked into source control. Run `vendor install` to restore the vendor projects.

Other commands

# Updating all dependencies will update all libraries to their latest versions.
vendor update
# Specifying the dependency will cause only the single library to be updated.
vendor update facebook-ios-sdk

Adding a library formula

If a library has no framework dependencies, has no required additional compiler/linker flags, and has an XCode project, it doesn’t require a Vendor formula. An example is JSONKit, which may be specified as below. However, if another Vendor library requires JSONKit, JSONKit must have a Vendor formula.

lib "JSONKit", :git => ""

However, if the library requires frameworks or has dependencies on other Vendor libraries, it must have a Vendor formula.  As with Brew, a Vendor formula is some declarative Ruby code that is open source and centrally managed.

An example Vendor formula might look like:

require 'formula'

class Three20 < Formula
  url ""
  libraries libThree20.a
  frameworks "CoreAnimation"
  header_path "three20/Build/Products/three20"
  linker_flags "ObjC", "all_load"
  vendors "JSONKit"


Using iOS libraries is way harder than it should be, which has negatively impacted the growth of the open source community for iOS.  Even if I was only developing libraries for myself, I would still want some kind of packaging system to help me manage the code.  Vendor essentially streamlines the flow described by Jonas Williams here.  Unfortunately, programmatically managing XCode projects isn’t supported natively, but people have implemented various solutions, such as Three20Victor Costan, and XCS.

Open Questions

  • Is there an existing solution for this?
  • Would this be a useful gem for your iOS development?
  • Why hasn’t anyone built something like this already? Impossible to build?