CodePath

Introduction

In my last post, I discussed the failings of to_json in Rails and how we approached writing our public APIs using RABL and JSON Templates in the View. Since that post, we have continued expanding our Developer API and adding new applications to our Application Gallery.

The approach of using views to intelligently craft our JSON APIs has continued to prove useful again and again as we strive to create clean and simple APIs for developers to consume. We have used the as_json approach on other projects for years in the past and I can say that I have not missed that strategy even once since we moved to RABL. To be clear, if your APIs and data models are dead simple and only for internal or private use and a one-to-one mapping suits your needs, then RABL may be overkill. RABL really excels once you find yourself needing to craft flexible and more complex APIs rich with data and associated information.

The last post was all about explaining the reasons for abandoning to_json and the driving forces behind our creation of RABL. This post is all about how to actually use Rails 3 and RABL in a psuedo real-world way to create a public API for your application. In other words, how to empower an “application” to become a developer’s platform.

Authentication

The first thing to determine when creating a platform is the authentication mechanism that developers will use when interacting with your API. If your API is read-only and/or easily cached on a global basis, then perhaps no authentication is necessary. In most cases though, your API will require authentication of some kind. These days OAuth has become a standard and using anything else is probably a bad idea.

The good news is that we are well past the point where you must implement an OAuth Provider strategy from scratch in your application. Nevertheless, I would strongly recommend you become familiar with the OAuth Authentication Protocol before trying to create a platform. Starting with this in mind, your next task is to pick an OAuth gem for Ruby. Searching Github, you can see there are many libraries to choose from.

The best one to start with is Rails OAuth Plugin which will provide a drop-in solution both to consuming and providing APIs in your Rails 3 applications. The documentation is fairly decent and coupled with the introduction blog post, I think reiterating the steps to set this up would be unnecessary. Follow the steps and generate an OAuth Provider system into your Rails 3 application. This is what will enable the “OAuth Dance” where developers can generate a request token and sign with that token for a user to retrieve and access token. Once the developer gets an access token, they can use that to interact with your APIs. The easiest way for developers to consume your API is to use a “consumer” OAuth library such as Ruby OAuth or equivalent in the client language.

Building your API

Now that we have authentication taken care of in our application, the next step is to create APIs for developers to interact with on our platform. For this, we can re-use our existing controllers or create new ones. You already likely have html representations of your application for users to interact with. These views are declared in a “index.html.erb” file or similar and describe the display for a user in the browser. When building APIs, the simplest way to think about them is to think of the JSON API as “just another view representation” that lives alongside your HTML templates.

To demonstrate how easy building an API is, let’s consider a sample application. This application is a simple blogging engine that has users and posts. Users can create posts and then their friends can read their streams. For now, we just want this to be a blogging platform and we will leave the clients up to developers to build using our APIs. First, let’s use RABL by declaring the gem in our Gemfile:

# Gemfile
gem 'rabl'

Next, we need to install the new gem into our gemset:

$ bundle install

Next, let’s generate the User and Post tables that are going to be used in this example application:

$ rails g model User first_name:string last_name:string age:integer
$ rails g model Post title:string body:text user_id:integer
$ rake db:migrate db:test:prepare

Nothing too crazy, just defining some models for user and posts with the minimum attributes needed. Let’s also setup model associations:

# app/models/user.rb
class User < ActiveRecord::Base
has_many :posts
end

# app/models/post.rb
class Post < ActiveRecord::Base
belongs_to :user
end

Great! So we now have a user model and a post model. The user can create many posts and they can be retrieved easily using the excellent rails has_many association. Now, let’s expose the user information for a particular user in our API. First, let’s generate the controller for Users:

$ rails g controller Users show

and of course setup the standard routes for the controller in routes.rb:

# config/routes.rb
SampleApiDemo::Application.routes.draw do
resources :users
end

Next, let’s protect our controller so that the data can only be accessed when a user has authenticated, and setup our simple “show” method for accessing a user’s data:

# app/controllers/users_controller.rb
class UsersController < ApplicationController
before_filter :oauth_required

respond_to :json, :xml
def show
@user = User.find_by_id(params[:id])
end
end

As you can see the controller is very lean here. We setup a before_filter from the oauth-plugin to require oauth authentication and then we define a “show” action which retrieves the user record given the associated user id. We also declare that the action can respond to both json and xml formats. At this point, you might wonder a few things.

First, “Why include XML up there if we are building a JSON API?” The short answer is because RABL gives this to us for free. The same RABL declarations actually power both our XML and JSON apis by default with no extra effort! The next question might be, “How does the action know what to render?”. This is where the power of the view template really shines. There is no need to declare any response handling logic in the action because Rails will automatically detect and render the associated view once the template is defined. But what does the template that powers our XML and JSON APIs look like? Let’s see below:

# app/views/users/show.rabl
object @user

# Declare the properties to include
attributes :first_name, :last_name

# Alias 'age' to 'years_old'
attributes :age => :years_old

# Include a custom node with full_name for user
node :full_name do |user|
[user.first_name, user.last_name].join(" ")
end

# Include a custom node related to if the user can drink
node :can_drink do |user|
user.age >= 21
end

The RABL template is all annotated above hopefully explaining fairly clearly each type of declaration. RABL is an all-ruby DSL for defining your APIs. Once the template is setup, testing out our new API endpoint is fairly simple. Probably the easiest way is to temporarily comment out the “oauth_required” line or disable in development. Then you can simply visit the endpoint in a browser or using curl:

// rails c
// User.create(...)
// rails s -p 3001
// http://localhost:3001/users/1.json
{
user: {
first_name: "Bob",
last_name: "Hope",
years_old: "92",
full_name: "Bob Hope",
can_drink: true
}
}

With that we have a fully functional “show” action for a user. RABL also let’s us easily reuse templates as well. Let’s say we want to create an “index” action that lists all our users:

# app/views/users/index.rabl
object @users

# Reuse the show template definition
extends "users/show"

# Let's add an "id" resource for the index action
attributes :id

That would allow us to have an index action by re-using the show template and applying it to the entire collection:

// rails s -p 3001
// http://localhost:3001/users.json
[
{
user: {
id : 1,
first_name: "Bob",
last_name: "Hope",
years_old: "92",
full_name: "Bob Hope",
can_drink: true
}
},
{
user: {
id: 2,
first_name: "Alex",
last_name: "Trebec",
years_old: "102",
full_name: "Alex Trebec",
can_drink: true
}
}
]

Now that we have an “index” and “show” for users, let’s move onto the Post endpoints. First, let’s generate a PostsController:

$ rails g controller Posts index

and append the posts resource routes:

# config/routes.rb
SampleApiDemo::Application.routes.draw do
resources :users, :only => [:show, :index]
resources :posts, :only => [:index]
end

Now, we can fill in the index action by retrieving all posts into a posts collection:

# app/controllers/posts_controller.rb
class PostsController < ApplicationController
before_filter :login_or_oauth_required
respond_to :html, :json, :xml

def index
@posts = Post.all
end
end

and again we define the RABL template that serves as the definition for the XML and JSON output:

# app/views/posts/index.rabl

# Declare the data source
collection @posts

# Declare attributes to display
attributes :title, :body

# Add custom node to declare if the post is recent
node :is_recent do |post|
post.created_at > 1.week.ago
end

# Include user as child node, reusing the User 'show' template
child :user do
extends "users/show"
end

Here we introduce a few new concepts. In RABL, there is support for “custom” nodes as well as for “child” association nodes which can also be defined in RABL or reuse existing templates as shown above. Let’s check out the result:

[
{
post: {
title: "Being really old",
body: "Let me tell you a story",
is_recent: false,
user: {
id : 52,
first_name: "Bob",
last_name: "Hope",
years_old: 92,
full_name: "Bob Hope",
can_drink: true
}
}
},
{ ...more posts... }
]

Here you can see we have a full posts index with nested user association information and a custom node defined as well. This is where the power of JSON Templates lies. The incredible flexibility and simplicity that this approach affords you. Renaming an object or attribute is as simple as:

# app/views/posts/index.rabl

# Every post is now contained in an "article" root
collection @posts => :articles

# The attribute "first_name" is now "firstName" instead
attribute :first_name => :firstName

Another benefit is the ability to to use template inheritance which affords seamless reusability. In addition, the partial system allows you to embed a RABL template within another to reduce duplication.

Wrapping Up

For a full rundown of the functionality and DSL of RABL, I encourage you to check out the README and the Wiki. In addition, I would like to point out excellent resources by other users of RABL and encourage you to read teohm’s post as well as Nick Rowe’s overview of RABL as well. Please post any other resources in the comments!

My intention with this post was to provide a solid introductory guide for creating an API and a data platform easily in Rails. If there are topics that you would be interested in reading about more in-depth related to API design or creation or building out a platform, please let us know and we will consider it for a future post.

At Miso, we have been very busy in the last few months building out a large number of public APIs for our Developer Platform. In a short time, we have already seen early versions of applications built on our platform for Chrome, Windows Mobile 7, Blackberry, Playbook, XBMC among others. This has been very exciting to see the community embrace our platform and leverage our data to power additional services or bring our service to a new group of users. In this post, we will discuss how we started out building our APIs using Rails and ‘to_json’, why we became frustrated with that approach and how we ended up building our own library for API generation.

Our public APIs are designed to be unsurprising and intuitive for a developer. We chose OAuth 1.0a (soon to support OAuth 2) because this is already familiar to developers and there is rich library support across languages for this authentication strategy. The endpoints are for the most part RESTful with GET retrieving and POST / DELETE used to modify the data associated with a user. We also tried to design our API responses as simply as possible, giving every attribute a readable name, keeping node hierarchies relatively flat and not including unnecessary information from our database schema. While this may seem like good design goals when building a Public API, you might find yourself surprised at how difficult this can be using Rails and the baked in ‘to_json’ serialization that the framework provides.

Rails ‘to_json’ API generation

Let’s start by discussing the canonical approach Rails provides for generating APIs. The idea is to have the JSON and XML responses be deeply tied to the model schemas with one-to-one mappings in most cases between the database columns and the api output. This is great if you are working on internal APIs and you wish to dump the data directly into a response but less great for well-designed public APIs. Let’s look at how rendering with this to_json approach works in practice:

# app/controllers/posts_controller.rb
#...
  respond_to :json, :xml
  def index
    @posts = Post.all
    respond_with(@posts)
  end
#...

Now, when this controller action is invoked the @post object automatically has the ‘to_json’ method called which takes the ActiveRecord model and converts seamlessly to JSON output. The model can also optionally be given parameters in the model by overriding the as_json method with options:

# app/models/post.rb
class Post
  def as_json(options={})
    super(options.merge(:methods => [...], :only => [...], :include => [...])
  end
end

This would then render the associated JSON response based on the options specified. In the simplest cases this would be all you need and the Rails way works without a problem. As long as your database schema is deeply coupled with the API output then all is well. As mentioned there are also a few choices to customize the to_json output and alter the output:

• only – Only show column names in the output as specified in this list
• except – Show all column names except the ones specified in this list
• methods – Include these methods nodes (without any arguments) as nodes in the output
• include – Add child nodes (potentially nested) based on associations within the object

Alright, to recap: There is a model which contains a specific database schema and some methods. There are limited options to transforming that into JSON as described above by passing a hash of options. These options can be passed in as defaults with the ‘as_json’ method in the model itself or through the controller action.

The unravelling of ‘to_json’ begins

Well wait just a minute, what if in different API responses, you want to include different api output options? What if you want to override the ‘as_json’ defaults? Not too bad you can just do:

# app/controllers/posts_controller.rb
#...
  respond_to :json, :xml
  def index
    @posts = Post.all
    respond_with(@posts) do |format|
      format.json { render :json => @posts.to_json(:include => [...], :methods => [...]) }
    end
  end
#...

Using those settings you can change the JSON output on a per action basis. This system as described is a good high level overview of the Rails JSON generation approach. This can work in very simple and naive applications, but it should not be hard to imagine how this approach can be restrictive as well as verbose. Suppose I want to render a JSON output with only a few columns, also adding several methods and including nested options for multiple associations? That might look something like this:

# app/controllers/posts_controller.rb
#...
  respond_to :json, :xml
  def index
    @posts = Post.all
    respond_with(@posts) do |format|
      format.json { render :json => @posts.to_json(
         :only => [:title, :body, :created_at, :tags, :category],
         :include => [
            :likes => { :only => [:created_at], :include => [:author] },
            :comments => { only => [:created_at, :body], :include => [:author]  },
            :user => { :only => [:first_name, :last_name}, :methods => [:full_name] },
         :methods => [:likes_count, :comments_count])
      }
    end
  end
#...

This action code is already starting to smell a bit funny even here. This is a lot of bulk in the controller and quite redundant. This doesn’t even really seem like this belongs in the controller. This is more of a view or template concern discussing the details of a particular JSON representation. You could move that all to the model inside a method, but that actually makes things harder to follow. Already this method of generating JSON doesn’t feel quite right and begins to break down.

You may think this example is a contrived case or poor API design but consider that there’s actually not that much going on here. This type of response is commonplace in almost any public API you will see on the web. In fact, it is actually much simpler then many in the wild. Compare the above to the Instagram API.

More Frustrations with API Generation

The issues above were just the beginning of the issues we ran up against using the ‘to_json’ method because that approach is interested in ‘serializing’ a database object while we are interested in creating a relevant representation for our public API platform. The ‘serialization’ of the object so directly just didn’t quite fit what we were trying to do.

The easiest way to demonstrate the limitations that frustrated us is to show relevant examples. Let’s start with a simple idea. In our system we have ‘posts’ and we have the idea of ‘liking a post’. In our API we want to return if the authenticated user ‘liked’ a particular post in the feed. Something like:

[ { post : { title : "...", liked_by_user : true }, ...]

Notice the node ‘liked_by_user’ which contains whether or not a user has liked the given post. Assuming we have this method in the model:

class Post
  # Returns true if given user has liked the post.
  # @user.liked_by_user?(@user) => true
  def liked_by_user?(user)
    self.likes.exists?(:user_id => user.id)
  end
end

We simply want to get this boolean value into the API response with the node name ‘liked_by_user’. How would we do this in ‘to_json’? How do we pass an argument to a method? After doing some research, it was apparent that this was not particularly easy or intuitive. It would be nice to have a simple way to pass multiple arguments to a method in the model without jumping through hoops.

Let’s move onto another example. Suppose we want to change the ‘user’ association to be aliased as an ‘author’ node in the output. Let’s say we have:

class Post
  belongs_to :user
end

and we want to have the output be:

[ { post : { title : "...", author : { first_name : "...", last_name : "..." }  }, ...]

What if I just need a minor change to the value of an attribute before inserting it into the JSON? What if I need a custom node in the JSON that is not needed in the model directly? What if I want to include the value of a method only if a condition on the record is met? What if I want to reduce duplication and render a JSON hash as a child of the parent response? What if I want to glue a couple of attributes from the user to the post? Change the model and fill it with this display logic every time? Fill our controllers with complicated JSON display options? Workaround the problems by fighting with ‘to_json’ and/or monkeypatching it?

Perhaps a better approach

As we came against these issues and many more while we designed and implemented our public APIs, we butted our heads against ‘to_json’ again and again. Often we wanted the attributes defined in the schema to be renamed or modified for the representation, or we wanted to omit attributes, or we wanted to include attributes if a condition was met, we wanted to handle polymorphic associations in a clean and easy way, we wanted to keep a flat hierarchy by ‘gluing’ attributes from the child to the parent.

Furthermore, the model and/or controller was getting filled up with tons of json specific details that had nothing to do with model or business logic. In fact, these JSON responses and verbose declarations didn’t seem to belong in the model or the controller at all and were cluttering up our code. In fact, true to MVC these details of the response seemed much more appropriate in a view of some kind. This idea of storing the JSON in a view sparked an experiment. Why not just generate the JSON in a template and move all of the display details out of the model and the controller. What if the API could be crafted easily in the view where a JSON representation belongs?

We agreed that implementing APIs in a view made the most sense both conceptually and practically. The next question becomes what templating language to use to generate these APIs? Forming the XML or JSON manually in ‘erb’ seemed verbose and error-prone. Using builder seemed silly since we wanted to build APIs that work primarily in JSON. Indeed, none of the default templating languages we grab for seemed to fit. We didn’t want to painstakingly handcraft nodes manually, we just wanted a simple way to declare how our APIs should look that afforded us the flexibility we needed.

We investigated a wealth of different libraries that seemed to fit the bill from tequila, to json_builder, to argonaut and many more attempts to solve this problem. Clearly we weren’t the only ones that had experienced the pain of ‘to_json’. Perusing the READMEs of any of these libraries quickly revealed people fed up with the limitations same as we had become. Problem was every option we could find didn’t work for one reason or another. Either the syntax was awkward, the libraries weren’t maintained, there were too many bugs, or the templates became verbose and difficult to manage. After reviewing the available options, we decided to try and design our own library for creating APIs. One that would solve all the problems we had encountered thus far.

The Ruby API Builder Language

We embarked on a thought experiment before building the library. What were our frustrations with existing libraries and tools? Where do we want the JSON options to live? How did we want to specify them? What options did we want to have? What language or syntax should we use to define the output? How do we keep the options DRY and intuitive?

Early on we decided we wanted the JSON output to be defined in the views. Logic that belonged in the models would stay there where it belonged, but this was rarely the case. Most of the options were simply crafting the JSON response and clearly belongs in a template. So that meant a file living in the views folder within Rails. We also decided we didn’t want to learn a new language and that Ruby was as good an API builder as any. Why not just leverage a simple Ruby DSL to build our APIs? Why not support inheritance and partials for our APIs? Why not allow the same template to describe both the JSON and XML responses for our API?

From these design questions and several days of work, the RABL gem was born. We started using this approach and fell in love with it immediately. All of a sudden, generating APIs was easy and intuitive. Even the most complex or custom API output was very simple and maintainable through the use of inheritance, partials and custom nodes. All of this was kept neatly tucked away in a view template where it belonged without requiring any extra code in the models or worse the controller actions.

Stay Tuned

Since we built RABL, we have gotten excellent feedback from the community. We have deployed in production all of our Public APIs using RABL and we couldn’t be happier. Please checkout the README and let us know what you think! We would love to hear your experiences with building APIs on Rails or Sinatra. This post is a setup for a thorough step-by-step tutorial we plan to publish soon on generating clean JSON and XML APIs in Rails 3 using RABL.