Monkeypod - API Design and Virtualization

At work, we have just launched a new tool called Monkeypod that allows easy design of REST APIs in a browser. It automatically creates Swagger documentation and a virtualized version of the API serving dummy data instantly. We're finding it very helpful internally, and wanted to open it up to the community for a private beta. Please sign up if you do work with APIs and give us your feedback.

Sign up at http://monkeypod.io

Follow Monkeypod on Twitter at http://twitter.com/monkeypodio

Monkeypod is written using MongoDB, Swagger, Scala, Akka, Casbah, and a lot of the techniques that I have written about in this blog. I'll do a deeper technical architecture dive at some point, but if you'd like to read more about the project, please check out  http://bit.ly/1uHBuoq

Cheers!

Mobile Enterprise Integration with Scala, MongoDB and Swagger

Check out my post on Enterprise Integration and some of the tools we have built for it using MongoDB, Scala and Swagger:

Enterprise Integration for Mobile Applications

Scala Either Example - Responding to Failures

A lot of Scala developers I know regularly use Option to write concise code in situations where something may or may not be returned by a given function. By getting familiar with the functions available on the Option class, you can chain many operations together without a lot of boilerplate or defensive protection against nonexistent values. Something like:

getAnOption
  .flatMap(transformIntoAnotherOption)
  .map(changeThing)
  .filter(thingIsTheRightType)

Assuming in this contrived bit of pseudo-code that getAnOption and transformIntoAnotherOption are functions that return Option types, this chain of function calls will short-circuit if any call returns a None. For example, if transformIntoAnotherOption returns None, then the expression evaluation is complete without ever running changeThing.

But what if I want to know which part of a complex expression like the one above returned None? It's tricky and less elegant when using an Option expression which simply evaluates to Some or None.

Let's say I'm writing a database application that requires a two-phase lookup, either phase of which may fail to find the requested record. I want to be able to:

• Look up an id based on a user-provided String
• If we found an id, look up the Record
• If we found a Record, validate it, process it, display it in an app, etc

The driver I am using returns Option[Record] from find calls. For this simple example, I mocked up a call with a frequent random failure that looks like this:

def databaseCall[A](value: A) = 
  if (nextInt % 5 == 0) None else Some(value)

def nextInt = math.abs(scala.util.Random.nextInt)

And two wrapper calls to represent our two phases of lookup:

def lookupId(key: String): Option[Int] = 
  databaseCall(nextInt)

def lookupRecord(id: Int): Option[Record] =
  databaseCall(Record(id))

Chaining Option calls is a natural fit if we just need to find a record and return it (or not):

def findRecordByKey(key: String): Option[Record] =
  databaseCall(key)
  .flatMap(databaseCall)
  .filter(validateIt)
  .map(processIt)
  .foreach(displayInApp)

The code above will sometimes call displayInApp with a value, and sometimes it will not. It is concise and avoids cluttering the logic with explicit conditionals and error handling. But it does not allow us to provide detailed feedback about what part of the process failed. Say we want to log whenever a databaseCall fails. We have to break up our nice single expression with calls to isDefined and error logging code. Not exactly elegant functional monadic wizardry that will get you invited to the cool kids' parties.

Either FTW

The scala.Either class can be used in ways very similar to Option. Either represents an instance of either Left or Right. By convention, Left is the "bad" or failure result and Right is the "good" one. In our example, that means Right is a value we found in the database and Left is a richer version of None where we can put a sensible message, an Exception or whatever we like. Profit!

First thing, let's change our database functions so they return Either instead of Option. Conveniently, Option has the functions toLeft and toRight for just this purpose. We'll use toRight in our example, which returns a Right containing the value of the Option it is called on if it is a Some. toRight() takes an argument that will be used to construct a Left if the Option is None. In our case, we'll provide a straightforward error message to use in case of failures:

def lookupId(key: String): Either[String, Int] = 
  databaseCall(nextInt).toRight("failed to find id based on key")

def lookupRecord(id: Int): Either[String, Record] =
  databaseCall(Record(id)).toRight("failed to find record based on id")

Easy enough. Here's how we can chain the calls to these functions:

lookupId("key").right
  .flatMap(lookupRecord(_))

Pretty similar to how we use Options. Calling right() on the result of lookupId("key") projects the Either as a Right. The contained value is the passed to lookupRecord and so on. If the Either is a Left, lookupRecord is never called. So just as with Option, we can chain our calls, evaluating to an Either in the end.

So we end up with either a Right(someRecordInstance) or a Left(someErrorMessage). We can use pattern matching to process this result, or you can use the fold() function, which takes two function arguments. The first is run in the case of a Left, the second in the case of a Right. So that's just super for us. Here's the whole thing:

lookupId("key").right
  .flatMap(lookupRecord(_)).fold(
    s => println("ERROR - " + s),
    s => println("looked up record " + s))

Running this code a thousand or so times prints a bunch of lines like the following:

looked up record Record(2073762079)
ERROR - failed to find id based on key
ERROR - failed to find record based on id
ERROR - failed to find record based on id
looked up record Record(1377010886)
looked up record Record(1665383097)
looked up record Record(646489647)

We now know when we failed to resolve an id to a key. We know if the record itself was not found. And assuming all went well, we get our Record result returned, validated, processed and displayed in our app in one neat expression. And we get invited to the cool kids' parties.

Download a buildable version of this code from github, and enjoy using Either in your projects.

Introduction to Casbah - The Scala MongoDB Driver

I've been meaning to point out the sheer coolness of the Scala MongoDB driver - Casbah. Written and maintained by 10Gen's own Brendan McAdamas, it is a truly elegant way to interact with MongoDB. If you've been using the Java driver, this might be a fun way to try out some Scala.

Here are some examples, none of which will be too surprising, since they look so much like interacting with Mongo via the shell client. Really that's the nice thing about Casbah - it's much closer to using the Python driver or JavaScript shell than the Java map-style driver.

I'll demonstrate some more advanced usage in the future, but this will get you started.

    //connect to a MongoDB server
    val mongo = MongoConnection("anduin")

    //create/connect to a collection
    val coll = mongo("casbah_examples")("movies")

    //drop collection
    coll.dropCollection

    //create some documents
    val pf = MongoDBObject("title" -> "Pulp Fiction",
                           "director" -> "Quentin Tarantino",
                           "foo" -> "bar",
                           "actors" -> List("John Travolta", "Samuel L Jackson"),
                           "year" -> 1994)

    val sw = MongoDBObject("title" -> "Star Wars",
                           "director" -> "George Lucas",
                           "cast" -> List("Harrison Ford", "Carrie Fisher"),
                           "year" -> 1977)

     val fc = MongoDBObject("title" -> "Fight Club",
                           "director" -> "David Fincher",
                           "cast" -> List("Brad Pitt", "Edward Norton"),
                           "year" -> 1999)

    //add some documents
    coll += pf
    coll += sw
    coll += fc
    val pfid = MongoDBObject("_id" -> pf.get("_id"))

    //update a document
    //increment a value (or set it if the value does not exist)
    coll.update(pfid, $inc("likes" -> 1))

    //set a value
    coll.update(pfid, $set("year" -> 1994))

    //remove a field
    coll.update(pfid, $unset("foo"))

    //add a value to an array - create it if need be
    coll.update(pfid, $push("comebackKids" -> "John Travolta"))

    //add a bunch of values to an array
    coll.update(pfid, $pushAll("comebackKids" -> ("Bruce Willis", "Uma Thurman")))

    //add to a set (only adds values that were not there already)
    coll.update(pfid, $addToSet("actors") $each("John Travolta","Groucho Marx","Harpo Marx","Alyssa Milano","Bruce Willis","Uma Thurman", "Jim Carrey"))

    //remove last element of an array - use -1 to remove the first element
    coll.update(pfid, $pop("actors" -> 1))

    //remove matching elements from a field
    coll.update(pfid, $pull("actors" -> "Alyssa Milano"))

    //remove more than one matching element from a field
    coll.update(pfid, $pullAll("actors" -> ("Groucho Marx", "Harpo Marx")))

    //rename a field
    coll.update(pfid, $rename("actors" -> "cast"))

    //query documents
    //find one by an exact field match
    coll.findOne(MongoDBObject("title" -> "Star Wars"))

    //find by regex
    coll.find(MongoDBObject("title" -> ".*F".r))

    //leave out the ids
    coll.find(MongoDBObject(),MongoDBObject("_id" -> 0))

    //include only certain fields without ids
    coll.find(MongoDBObject(), MongoDBObject("title" -> 1, "cast" -> 1, "_id" -> 0))

    //find most recent
    coll.find(MongoDBObject()).sort(MongoDBObject("year" -> -1)).limit(1)

    //remove
    coll.remove(MongoDBObject("title" -> "Fight Club"))

Scalatra MongoDB Executable Jar Template

Just a quick note that I have updated my giter8 template for a simple Scalatra project using MongoDB. It now uses library versions that are available for Scala 2.9.1 and updates some dependency versions. These dependencies/plugins target SBT 0.10.1 and probably won't work well with other versions. I also added the assembly plugin to make it easy to build an executable jar. I didn't bother to branch since the old one was broken due to some dependency issues.

To start a new project, do the following:

1. Install giter8


2. g8 JanxSpirit/scalatra-mongodb


3. cd *project-name*


4. sbt


5. jetty-run


6. browse to http://localhost:8080/test


7. If you want an executable jar, run 'assembly' at the SBT prompt


8. If you want to create a standalone shell command wrapping that jar, run something like 'cat src/main/resources/execute_jar.sh target/scalatra-mongodb-project-assembly-1.0.jar > target/scalatra_mongo'

Hopefully this template is helpful. Cheers!

Scalatra MongoDB template project using SBT 0.10.x

I spent some time this weekend getting a basic Scalatra/Casbah/MongoDB project (much like the one I covered in this post) set up using SBT 0.10.0.

If you have looked at the newest version of SBT, you know that a lot has changed. SBT 0.7.x projects will not build under the newer versions without a bit of migration. I ran into some hiccups along the way, so I set up a template project to avoid having to figure it out again. It uses Nathen Hamblen's fantastic giter8(g8) tool to allow you to set up a new project in seconds. Hopefully other g8 users find this template useful.

Changes in SBT 0.10.x

I just want to note some of the changes I ran into in setting up this very simple project:

• build.sbt file - SBT 0.10.x offers an alternative to the /project/build/Project.scala configuration file - you can put a *.sbt file in the root of your project and configure your project there using the new SBT DSL. You can still configure your project using Scala, but the API and file locations have changed. See the SBT documentation for details.


• "jetty" context for the Jetty dependency - this one confused me as I didn't see it documented anywhere. The Jetty dependency in build.sbt should not use the "test" scope as it did with SBT 0.7.x but rather the "jetty" scope. Using "test" will cause the jetty-run command to silently fail in a most unhelpful way.


• Jetty support no longer built-in - Unlike SBT 0.7.x which provided commands like jetty-run, jetty-stop and prepare-webapp for any Project extending DefaultWebProject, SBT 0.10.x requires a plugin to support web applications. See the project's homepage for setup and my template project for an example.

Template Project

You'll need to install SBT 0.10.x - README

You'll also need to install g8

Both tools are very easy one-time installs and worth having. Once they are installed, you should be able to run:

g8 JanxSpirit/scalatra-mongodb
cd *name-of-app*
sbt
> update
> jetty-run

You'll then be able to browse to the test resource to make sure it's working.

What's inside

The servlet provides two resources:

1. 'test' - a 'Hello World' type resource just to make sure everything is working.


2. 'msgs' - GET msgs to see all records currently in the database and a form to add another record - the form issues a POST to the same resource to add records

The project includes a few basic tests to get you started. Run 'sbt test' to see the test output.

The template allows you to configure where your MongoDB server is running, the name of your servlet and other parameters.

All source code is here: https://github.com/JanxSpirit/scalatra-mongodb.g8 I'd welcome any comments or suggestions or pull requests. Happy coding!

Java Without Semicolons: An Introduction to Scala - Part 1: Iteration

This series of posts was inspired by a suggestion from Martin Oedersky at Scala Days 2011 as well as something I have heard Dick Wall say on several occasions. To paraphrase Dick, you can get started with Scala by converting Java code to Scala with few changes beyond getting rid of the semicolons. It may not take advantage of many of Scala's more powerful features, but it's a great way to get started and an excellent jumping off point to learning Scala's richer language features.

Scala is a deep and flexible language that allows an enormous degree of freedom to library developers and advanced users, but those features can be daunting to new developers trying to take them all in. So, what areas should a Java programmer getting started with Scala start with? This blog series will cover Scala features that Java programmers should be able to put to immediate use. In other words, I'll suggest some of the first things a newcomer to Scala can try after getting rid of the semicolons.

Setup

You're encouraged to follow along with these examples in the Scala REPL and use them as a jumping off point for your own exploration. Installing Scala is easy - if you already have Java set up, just unzip the Scala distribution, set SCALA_HOME and add $SCALA_HOME/bin to your path. You can run the REPL by simply typing 'scala' in a shell. Further instructions at scala-lang.org or typesafe.com. I'll be using Scala 2.9.0.1 which is the latest version at the time of this writing.

Iteration

As a longtime Java developer, I was immediately drawn to Scala's alternatives to for-loop iteration. Java loops are easy enough to write, but can be buggy and confusing to read and comprehend.

This post will take a look at some functions Scala Collections provide to iterate that are as easy to read and understand as they are to write.

Function Literals

Assuming that Java is the only language you are familiar with, there's one important Scala language feature you'll need to understand in these examples, and that is function literals.

A function literal is analogous to String or int literals in Java in that it is a value that can be provided directly where it is used.

In Scala, you can define a function inline - that is pass a bit of functionality right where it is needed. This is what a simple function literal looks like:

x => x + 1

You can read this as "a function that take one argument x and returns the result of adding 1 to the value of x".

Scala's collections have special functions that take functions as arguments to be applied to the members of the collection. So unlike a Java for loop, where you get a handle on each member in the collection one at a time and do something to it, in Scala you can tell the collection to apply some functionality to all of its members and the actual iteration over elements is handled internally. Building on our example:

val someInts = List(4, 8, 15, 16, 23, 42)
val incrementedInts = someInts.map(x => x + 1)

The map function takes a single-argument function as its argument. It applies that function to every element in the List and returns a new List whose contents are the result of applying the function argument. In the example above, each element of the list is incremented by 1 and the result looks like this:

List(5, 9, 16, 17, 24, 43)

Note that the original list still exists unmodified.

In Scala code you will often see a shorter form of function literals. Scala allows us to substitute a _ character for the function parameters in many cases, so the above map call could be written as:

val incrementedInts = someInts.map(_ + 1)

The two are functionally identical, and it's up to you which you want to use.

In some cases the function literal can be even shorter. When the function passed as an argument takes a single argument itself, Scala will automatically pass each element of the collection to it without even needing the _. The following examples are functionally equivalent and use the foreach function, which is executed only for side effects and does not return a value (it returns Unit, equivalent to Java's void return type).

someInts.foreach(x => println(x))
someInts.foreach(println(_))
someInts.foreach(println)

Now let's say we only want the subset of our list that is odd numbers. The filter function is useful for such operations. It takes a function argument that returns a boolean to determine whether an element should be included in the output:

someInts.filter(x => x % 2 == 1)
//someInts.filter(_ % 2 == 1)

Maybe we want the first number in the list that is greater than 20. We can use find:

someInts.find(x => x > 20)
//someInts.find(_ > 20)

If you want to discover whether all elements of a collection satisfy some requirement, use forall, which takes a function that returns boolean and itself returns a boolean:

someInts.forall(x => x > 0)
//someInts.forall(_ > 0)

You can get the first n elements that meet a certain requirement by using takeWhile. The following example returns a new list containing all of the words up to the first word that begins with a vowel:

val words = List("the","quick","brown","fox","jumped","over","the","lazy","dog")
words.takeWhile(w => !"aeiou".contains(w(0)))
// returns List(the, quick, brown, fox, jumped)

If you wanted to retain the remainder of the list separately, you can pass the same function to span, which returns a tuple of two Lists:

words.span(w => !"aeiou".contains(w(0)))
// returns (List(the, quick, brown, fox, jumped),List(over, the, lazy, dog))

The last example of these sort of collection iteration functions I'll demonstrate is foldLeft. It's a bit different in that it does not return a collection, but rather a single value as a result of applying a two-argument function to each element of a list and the result of the last function execution. You need to provide it a seed value, and the first iteration will execute the function with the seed value and first element of the list. In this example we'll get the sum of all numbers with more than one digit:

someInts.foldLeft(0)((a,b) => if (b.toString.length > 1) a+b else a)

In the example, foldLeft is seeded with a 0. The function is then called with 0 and 4, returning 0. Then 0 and 8 returning 0. Then 0 and 15 returning 15, and so on. Folding operations take a bit of getting used to, but are a powerful functional programming paradigm.

For Comprehension

Scala does have a for loop structure that may look more familiar to java developers. In Scala it's called a for comprehension, and it can be used in very powerful ways. It can also be used to iterate over collections in a more imperative way:

for (word <- words) {
  println(word)
}

But, often the functions on the various Scala collection classes allow you to perform common iteration tasks without using loops. This change can result in fewer more readable lines of code. In Java, even the contrived examples above would necessetate loops, temporary variables, temporary lists, and, in general, more boilerplate code not relating to business logic. The Scala collection libraries allow us to focus on the code that performs the actions we're really interested in. And it makes for some nifty one-liners:

//find all numbers from 1 to a million that are evenly divisible by 3 and 5, 
//remove all instances of the digit '9' and sum the resulting numbers
(1 to 1000000).filter(i => i % 3 == 0 && i % 5 == 0).map(_.toString.replace("9","").toInt).sum 

That's it for this installment. Hopefully someone getting started with Scala finds these examples helpful as a jumping off point.