Janx Spirit

Introduction to Casbah - The Scala MongoDB Driver

2011-11-20T20:45:00.000-08:00

I've been meaning to point out the sheer coolness of the Scala MongoDB driver - Casbah. Written and maintained by 10Gen's own Brendan McAdamas, it is a truly elegant way to interact with MongoDB. If you've been using the Java driver, this might be a fun way to try out some Scala.

Here are some examples, none of which will be too surprising, since they look so much like interacting with Mongo via the shell client. Really that's the nice thing about Casbah - it's much closer to using the Python driver or JavaScript shell than the Java map-style driver.

I'll demonstrate some more advanced usage in the future, but this will get you started.


    //connect to a MongoDB server
    val mongo = MongoConnection("anduin")

    //create/connect to a collection
    val coll = mongo("casbah_examples")("movies")

    //drop collection
    coll.dropCollection

    //create some documents
    val pf = MongoDBObject("title" -> "Pulp Fiction",
                           "director" -> "Quentin Tarantino",
                           "foo" -> "bar",
                           "actors" -> List("John Travolta", "Samuel L Jackson"),
                           "year" -> 1994)

    val sw = MongoDBObject("title" -> "Star Wars",
                           "director" -> "George Lucas",
                           "cast" -> List("Harrison Ford", "Carrie Fisher"),
                           "year" -> 1977)

     val fc = MongoDBObject("title" -> "Fight Club",
                           "director" -> "David Fincher",
                           "cast" -> List("Brad Pitt", "Edward Norton"),
                           "year" -> 1999)

    //add some documents
    coll += pf
    coll += sw
    coll += fc
    val pfid = MongoDBObject("_id" -> pf.get("_id"))

    //update a document
    //increment a value (or set it if the value does not exist)
    coll.update(pfid, $inc("likes" -> 1))

    //set a value
    coll.update(pfid, $set("year" -> 1994))

    //remove a field
    coll.update(pfid, $unset("foo"))

    //add a value to an array - create it if need be
    coll.update(pfid, $push("comebackKids" -> "John Travolta"))

    //add a bunch of values to an array
    coll.update(pfid, $pushAll("comebackKids" -> ("Bruce Willis", "Uma Thurman")))

    //add to a set (only adds values that were not there already)
    coll.update(pfid, $addToSet("actors") $each("John Travolta","Groucho Marx","Harpo Marx","Alyssa Milano","Bruce Willis","Uma Thurman", "Jim Carrey"))

    //remove last element of an array - use -1 to remove the first element
    coll.update(pfid, $pop("actors" -> 1))

    //remove matching elements from a field
    coll.update(pfid, $pull("actors" -> "Alyssa Milano"))

    //remove more than one matching element from a field
    coll.update(pfid, $pullAll("actors" -> ("Groucho Marx", "Harpo Marx")))

    //rename a field
    coll.update(pfid, $rename("actors" -> "cast"))

    //query documents
    //find one by an exact field match
    coll.findOne(MongoDBObject("title" -> "Star Wars"))

    //find by regex
    coll.find(MongoDBObject("title" -> ".*F".r))

    //leave out the ids
    coll.find(MongoDBObject(),MongoDBObject("_id" -> 0))

    //include only certain fields without ids
    coll.find(MongoDBObject(), MongoDBObject("title" -> 1, "cast" -> 1, "_id" -> 0))

    //find most recent
    coll.find(MongoDBObject()).sort(MongoDBObject("year" -> -1)).limit(1)

    //remove
    coll.remove(MongoDBObject("title" -> "Fight Club"))

Scalatra MongoDB Executable Jar Template

2011-09-24T23:43:00.000-07:00

Just a quick note that I have updated my giter8 template for a simple Scalatra project using MongoDB. It now uses library versions that are available for Scala 2.9.1 and updates some dependency versions. These dependencies/plugins target SBT 0.10.1 and probably won't work well with other versions. I also added the assembly plugin to make it easy to build an executable jar. I didn't bother to branch since the old one was broken due to some dependency issues.

To start a new project, do the following:

Install giter8

g8 JanxSpirit/scalatra-mongodb

cd *project-name*

jetty-run

browse to http://localhost:8080/test

If you want an executable jar, run 'assembly' at the SBT prompt

If you want to create a standalone shell command wrapping that jar, run something like 'cat src/main/resources/execute_jar.sh target/scalatra-mongodb-project-assembly-1.0.jar > target/scalatra_mongo'

Hopefully this template is helpful. Cheers!

Scalatra MongoDB template project using SBT 0.10.x

2011-06-12T13:44:00.000-07:00

I spent some time this weekend getting a basic Scalatra/Casbah/MongoDB project (much like the one I covered in this post) set up using SBT 0.10.0.

If you have looked at the newest version of SBT, you know that a lot has changed. SBT 0.7.x projects will not build under the newer versions without a bit of migration. I ran into some hiccups along the way, so I set up a template project to avoid having to figure it out again. It uses Nathen Hamblen's fantastic giter8(g8) tool to allow you to set up a new project in seconds. Hopefully other g8 users find this template useful.

Changes in SBT 0.10.x

I just want to note some of the changes I ran into in setting up this very simple project:

build.sbt file - SBT 0.10.x offers an alternative to the /project/build/Project.scala configuration file - you can put a *.sbt file in the root of your project and configure your project there using the new SBT DSL. You can still configure your project using Scala, but the API and file locations have changed. See the SBT documentation for details.

"jetty" context for the Jetty dependency - this one confused me as I didn't see it documented anywhere. The Jetty dependency in build.sbt should not use the "test" scope as it did with SBT 0.7.x but rather the "jetty" scope. Using "test" will cause the jetty-run command to silently fail in a most unhelpful way.

Jetty support no longer built-in - Unlike SBT 0.7.x which provided commands like jetty-run, jetty-stop and prepare-webapp for any Project extending DefaultWebProject, SBT 0.10.x requires a plugin to support web applications. See the project's homepage for setup and my template project for an example.

Template Project

You'll need to install SBT 0.10.x - README

You'll also need to install g8

Both tools are very easy one-time installs and worth having. Once they are installed, you should be able to run:


g8 JanxSpirit/scalatra-mongodb
cd *name-of-app*
sbt
> update
> jetty-run

You'll then be able to browse to the test resource to make sure it's working.

What's inside

The servlet provides two resources:

'test' - a 'Hello World' type resource just to make sure everything is working.

'msgs' - GET msgs to see all records currently in the database and a form to add another record - the form issues a POST to the same resource to add records

The project includes a few basic tests to get you started. Run 'sbt test' to see the test output.

The template allows you to configure where your MongoDB server is running, the name of your servlet and other parameters.

All source code is here: https://github.com/JanxSpirit/scalatra-mongodb.g8 I'd welcome any comments or suggestions or pull requests. Happy coding!

Java Without Semicolons: An Introduction to Scala - Part 1: Iteration

2011-06-05T17:18:00.001-07:00

This series of posts was inspired by a suggestion from Martin Oedersky at Scala Days 2011 as well as something I have heard Dick Wall say on several occasions. To paraphrase Dick, you can get started with Scala by converting Java code to Scala with few changes beyond getting rid of the semicolons. It may not take advantage of many of Scala's more powerful features, but it's a great way to get started and an excellent jumping off point to learning Scala's richer language features.

Scala is a deep and flexible language that allows an enormous degree of freedom to library developers and advanced users, but those features can be daunting to new developers trying to take them all in. So, what areas should a Java programmer getting started with Scala start with? This blog series will cover Scala features that Java programmers should be able to put to immediate use. In other words, I'll suggest some of the first things a newcomer to Scala can try after getting rid of the semicolons.

Setup

You're encouraged to follow along with these examples in the Scala REPL and use them as a jumping off point for your own exploration. Installing Scala is easy - if you already have Java set up, just unzip the Scala distribution, set SCALA_HOME and add $SCALA_HOME/bin to your path. You can run the REPL by simply typing 'scala' in a shell. Further instructions at scala-lang.org or typesafe.com. I'll be using Scala 2.9.0.1 which is the latest version at the time of this writing.

Iteration

As a longtime Java developer, I was immediately drawn to Scala's alternatives to for-loop iteration. Java loops are easy enough to write, but can be buggy and confusing to read and comprehend.

This post will take a look at some functions Scala Collections provide to iterate that are as easy to read and understand as they are to write.

Function Literals

Assuming that Java is the only language you are familiar with, there's one important Scala language feature you'll need to understand in these examples, and that is function literals.

A function literal is analogous to String or int literals in Java in that it is a value that can be provided directly where it is used.

In Scala, you can define a function inline - that is pass a bit of functionality right where it is needed. This is what a simple function literal looks like:

x => x + 1

You can read this as "a function that take one argument x and returns the result of adding 1 to the value of x".

Scala's collections have special functions that take functions as arguments to be applied to the members of the collection. So unlike a Java for loop, where you get a handle on each member in the collection one at a time and do something to it, in Scala you can tell the collection to apply some functionality to all of its members and the actual iteration over elements is handled internally. Building on our example:


val someInts = List(4, 8, 15, 16, 23, 42)
val incrementedInts = someInts.map(x => x + 1)

The map function takes a single-argument function as its argument. It applies that function to every element in the List and returns a new List whose contents are the result of applying the function argument. In the example above, each element of the list is incremented by 1 and the result looks like this:


List(5, 9, 16, 17, 24, 43)

Note that the original list still exists unmodified.

In Scala code you will often see a shorter form of function literals. Scala allows us to substitute a _ character for the function parameters in many cases, so the above map call could be written as:


val incrementedInts = someInts.map(_ + 1)

The two are functionally identical, and it's up to you which you want to use.

In some cases the function literal can be even shorter. When the function passed as an argument takes a single argument itself, Scala will automatically pass each element of the collection to it without even needing the _. The following examples are functionally equivalent and use the foreach function, which is executed only for side effects and does not return a value (it returns Unit, equivalent to Java's void return type).


someInts.foreach(x => println(x))
someInts.foreach(println(_))
someInts.foreach(println)

Now let's say we only want the subset of our list that is odd numbers. The filter function is useful for such operations. It takes a function argument that returns a boolean to determine whether an element should be included in the output:


someInts.filter(x => x % 2 == 1)
//someInts.filter(_ % 2 == 1)

Maybe we want the first number in the list that is greater than 20. We can use find:


someInts.find(x => x > 20)
//someInts.find(_ > 20)

If you want to discover whether all elements of a collection satisfy some requirement, use forall, which takes a function that returns boolean and itself returns a boolean:


someInts.forall(x => x > 0)
//someInts.forall(_ > 0)

You can get the first n elements that meet a certain requirement by using takeWhile. The following example returns a new list containing all of the words up to the first word that begins with a vowel:


val words = List("the","quick","brown","fox","jumped","over","the","lazy","dog")
words.takeWhile(w => !"aeiou".contains(w(0)))
// returns List(the, quick, brown, fox, jumped)

If you wanted to retain the remainder of the list separately, you can pass the same function to span, which returns a tuple of two Lists:


words.span(w => !"aeiou".contains(w(0)))
// returns (List(the, quick, brown, fox, jumped),List(over, the, lazy, dog))

The last example of these sort of collection iteration functions I'll demonstrate is foldLeft. It's a bit different in that it does not return a collection, but rather a single value as a result of applying a two-argument function to each element of a list and the result of the last function execution. You need to provide it a seed value, and the first iteration will execute the function with the seed value and first element of the list. In this example we'll get the sum of all numbers with more than one digit:


someInts.foldLeft(0)((a,b) => if (b.toString.length > 1) a+b else a)

In the example, foldLeft is seeded with a 0. The function is then called with 0 and 4, returning 0. Then 0 and 8 returning 0. Then 0 and 15 returning 15, and so on. Folding operations take a bit of getting used to, but are a powerful functional programming paradigm.

For Comprehension

Scala does have a for loop structure that may look more familiar to java developers. In Scala it's called a for comprehension, and it can be used in very powerful ways. It can also be used to iterate over collections in a more imperative way:


for (word <- words) {
  println(word)
}

But, often the functions on the various Scala collection classes allow you to perform common iteration tasks without using loops. This change can result in fewer more readable lines of code. In Java, even the contrived examples above would necessetate loops, temporary variables, temporary lists, and, in general, more boilerplate code not relating to business logic. The Scala collection libraries allow us to focus on the code that performs the actions we're really interested in. And it makes for some nifty one-liners:


//find all numbers from 1 to a million that are evenly divisible by 3 and 5, 
//remove all instances of the digit '9' and sum the resulting numbers
(1 to 1000000).filter(i => i % 3 == 0 && i % 5 == 0).map(_.toString.replace("9","").toInt).sum

That's it for this installment. Hopefully someone getting started with Scala finds these examples helpful as a jumping off point.

Sneaking Scala Through the Alley - Test Java Maven Web Services with Scala Specs

2011-02-13T12:00:00.000-08:00

It's been said before, but a great way to introduce Scala at work is in writing tests. In my case, that means writing functional tests for web services built with Maven. I've found the Specs library excellent for this purpose, and after sorting out a bit of setup, it's integrated into my build well. Some things I like about this approach:

Tests are concise and quick to write
Specs can be written in natural language, user stories, or however you normally describe software functionality
Output is natural language, so identifying the source of any bugs, regressions, etc is easy
Test code is generally such that developers who are unfamiliar with Scala can understand it quickly
Scala and Specs make it trivial to write focused, single-assertion tests with little boilerplate
Write empty specs as requirements emerge and bugs are found then implement them when you have time
You get to use Scala at work

This example will use Maven and a very simple Java REST service built with Jersey. In the real world you'd probably be using Spring and a lot of other libraries, but I've tried to distill this example down to the barest essentials in order to focus on Specs and how it plays with Maven.

Envision a beer service. It accepts requests for beer by type and size in ounces. It should return a 404 Not Found response if the beer is not a type it knows (Corona, Guiness or Fat Tire). If someone tries to order a beer smaller than a pint, the service should respond with a 400 Bad Request response. And if the service is able to fulfill the request, it should return a Json Beer representation.

As you work with your design team to specify these behaviors, you can write a first pass at your specs. The great thing about specs is how well they translate from the plain English description above:


"Calling GET /beer" should {
  "when called with" in {
    "a sizeOunces argument of 4" in {
      "return status 400" >> { }
    }

    "a beer name 'Budweiser'" in {
      "return status 404" >> { }
    }

    "beer name 'Guiness' and sizeOunces 16" in {
      "return status 200" >> { }
      "return a Beer representation" in {
        "with" in {
          "name 'Guiness'" >> {}
          "sizeOunces 16" >> {}
        }
      }
    }
  }
}

You can read each spec from its outermost part to inner, for example: "Calling GET /beer should when called with beer name 'Guiness' and sizeOunces 16 return status 200". You can leave these specs empty until you implement the corresponding functionality. Some build environments like Hudson display the unimplemented tests separately from the passing or failing tests. These unimplemented tests are a great aspect of using Specs. They convey to others what the expected functionality is, though it may not yet be implemented, and it serves as a checklist for you the developer. As new requirements emerge, you can immediately write empty specs that reflect them.

Setting up pom.xml
In order to use Scala in a Maven project, you'll need to set up a few plugins, dependencies. etc. I'll highlight the ones that may be unfamiliar to Java developers, and the entire pom.xml is shown below.

Repositories
I'm not going to get into repository configuration much, since everyone's setup is different. One repository to be aware of when doing Scala development is Scala Tools. It's built into SBT, but you'll need to add it to your Maven repository or project pom to use it in a Maven project.


        <repository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>

Dependencies
Scala
This example uses version 2.8.0 of the Scala library.


        <!-- Scala -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.8.0</version>
        </dependency>

Specs
Specs is a library for Behavior Driven Development in Scala. It integrates easily with mock libraries, ScalaTest for generating test data, and various IDEs and build tools. Scala Specs can be run as JUnit tests, allowing integration into Java development environments like Eclipse, IntelliJ and NetBeans as well as CI systems like Hudson and Bamboo. It has an excellent lightweight syntax that is easy to read and write. I'll only be scratching the surface of Specs' functionality here, but you should check out the excellent documentation for more examples.


        <!-- Specs -->
        <dependency>
            <groupId>org.scala-tools.testing</groupId>
            <artifactId>specs_2.8.0</artifactId>
            <version>1.6.5</version>
        </dependency>

Dispatch
Dispatch is a wrapper around HttpClient that provides a very terse DSL syntax and the ability to make Http calls in a Scalactic way. The overloaded operators throw some people off so if you prefer to use HttpClient or any other Http library, that works too. I have found that once I got familiar with Dispatch syntax, I quite like it, and it meshes well with Specs to allow me to quickly bang out a lot of test calls.


        <!-- Dispatch -->
        <dependency>
            <groupId>net.databinder</groupId>
            <artifactId>dispatch-http_2.8.0</artifactId>
            <version>0.7.7</version>
            <scope>test</scope>
        </dependency>

Lift-Json
Dispatch provides support for Json through the lift-json library, which is a part of the Lift framework. It provides straightforward serialization and deserialization of Scala case classes as well as clear XPath-like parsing of Json.


        <dependency>
            <groupId>net.databinder</groupId>
            <artifactId>dispatch-lift-json_2.8.0</artifactId>
            <version>0.7.7</version>
            <scope>test</scope>
        </dependency>

And the rest...
You'll also need to add JUnit and Jersey for this example project. See the complete pom.xml below for specifics.

Build
You'll need to tell Maven to look for Scala tests.


        <testSourceDirectory>${project.basedir}/src/test/scala</testSourceDirectory>

Compile the Scala Tests
Configure the Scala compiler plugin so that your Scala tests get built like so:


            <!-- Compile Scala -->
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Surefire
Clarify that you would indeed like to run the Scala tests by telling the Maven Surefire plugin when and what to run. In this example we flout JUnit convention and run things that end in Spec. Observant types will notice that the configuration says **/*Spec.java even though we're writing tests in Scala. Just go with it.


            <!-- Tests -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.4.3</version>
                <configuration>
                    <useSystemClassLoader>false</useSystemClassLoader>
                    <argLine>-Xmx512m</argLine>
                    <includes>
                        <include>**/*Spec.java</include>
                    </includes>
                </configuration>
            </plugin>

Start and Stop Jetty
This configuration will run Jetty for the duration of the build. The sophisticated among you may wish to start and stop the server at different phases of the build. Determining an appropriate strategy is left as an exercise to the reader.


            <!-- Run Jetty -->
            <plugin>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>maven-jetty-plugin</artifactId>
                <version>6.1.24</version>
                <configuration>
                    <connectors>
                        <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector">
                            <port>2342</port>
                        </connector>
                    </connectors>
                    <stopKey>stop</stopKey>
                    <stopPort>9999</stopPort>
                </configuration>
                <executions>
                    <execution>
                        <id>start-jetty</id>
                        <phase>process-classes</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <daemon>true</daemon>
                        </configuration>
                    </execution>
                    <execution>
                        <id>stop-jetty</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>stop</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Beer Service
With the Maven setup behind us, we'll throw together a simple Jersey service so there's something to test. Here's a Beer JavaBean and a service class to serve some Beers.

/src/main/java/com/janxspirit/Beer.java


package com.janxspirit;

import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement
public class Beer {
    
    private String name;
    private int sizeOunces;

    public Beer() {}

    public Beer(String name, int sizeOunces) {
        this.name = name;
        this.sizeOunces = sizeOunces;
    }

    @XmlElement(name = "name")
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    @XmlElement(name = "sizeOunces")
    public int getSizeOunces() {
        return sizeOunces;
    }

    public void setSizeOunces(int sizeOunces) {
        this.sizeOunces = sizeOunces;
    }
}

/src/main/java/com/janxspirit/BeerResource.java


package com.janxspirit;

import java.util.ArrayList;
import java.util.List;
import javax.ws.rs.Path;
import javax.ws.rs.GET;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;

@Path("/")
public class RestResource {

    private List beerList;

    public RestResource() {
        beerList = new ArrayList();
        beerList.add("Corona");
        beerList.add("Guiness");
        beerList.add("Fat Tire");
    }

    @GET
    @Produces("application/json")
    @Path("beers/{name}")
    public Response getBeer(@PathParam("name") String name, @QueryParam("sizeOunces") int sizeOunces) {
        if (name == null || !beerList.contains(name)) {
            return Response.status(Status.NOT_FOUND).entity(String.format("Sorry we don't have any %s", name)).build();
        }
        if (sizeOunces < 8) {
            return Response.status(Status.BAD_REQUEST).entity("Sorry no kid-size beers.").build();
        }
        Beer beer = new Beer(name, sizeOunces);
        System.out.println(String.format("Pouring %s ounces of %s", sizeOunces, name));
        return Response.ok(beer).build();
    }
}

/src/main/java/JsonContextResolver.java
In order to get the default Json library to render the responses correctly, you need to add a class like the following. There are other ways, but in general its unfortunate that Jersey doesn't do the right thing out of the box at least presenting integer attributes unquoted in Json. We explicitly set the sizeOunces property to be non-String below.


package com.janxspirit;

import com.sun.jersey.api.json.JSONConfiguration;
import com.sun.jersey.api.json.JSONJAXBContext;
import javax.ws.rs.ext.ContextResolver;
import javax.ws.rs.ext.Provider;
import javax.xml.bind.JAXBContext;

@Provider
public class JsonContextResolver implements ContextResolver<JAXBContext> {

     private JAXBContext context;
     private Class[] types = {Beer.class};

     public JsonContextResolver() throws Exception {
         this.context =
    new JSONJAXBContext(
      JSONConfiguration.mapped().nonStrings("sizeOunces").build(), types);
    }

    public JAXBContext getContext(Class<?> objectType) {
        for (Class type : types) {
             if (type == objectType) {
                 return context;
             }
         }
         return null;
     }
}

web.xml
A last bit of setup is to configure your web.xml to fire up the Jersey servlet. This goes in the usual place - src/main/webapp/WEB-INF/web.xml


<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
    <display-name>scala_test_jersey</display-name>
    <servlet>
        <servlet-name>RestResourcer</servlet-name>
        <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
        <init-param>
            <param-name>com.sun.jersey.config.property.packages</param-name>
            <param-value>com.janxspirit</param-value>
        </init-param>
    </servlet>
    <servlet-mapping>
        <servlet-name>RestResourcer</servlet-name>
        <url-pattern>/*</url-pattern>
    </servlet-mapping>
</web-app>

Writing the Specs
Now the fun part! This is actually much less involved than all the boilerplate above, but hopefully the setup is helpful in getting you to the good stuff more quickly.

The first thing I did was create a simple case class and a function to convert the Dispatch responses into a simple object with a status code and body. I'd love to hear any better approaches to this, but in any event, this does work.


  def toResponse(req: Request): StatusCodeBody = {
    val http = new Http
    http x (req as_str) {
      case (status, _, ent, _) => StatusCodeBody(status, EntityUtils.toString(ent.get))
    }
  }

case class StatusCodeBody (status: Int, body: String)

Now we fill in the empty spec bodies. In most cases that's a one-liner to make the Http call and then a simple equality test against the response object. Here's what a simple call to the service looks like:


        val app = "scala_test_jersey"
        val getBeerRes = toResponse(:/("localhost", 2342) / app / "beers" / "Budweiser")
        "return status 404" >> { getBeerRes.status must_== 404 }

As you can see, assembling an Http call with dispatch is very straightforward. If you don't like the overloaded operators, I suggest you use trusty old HttpClient.

Here's all you need to do to add a query param to a request:


        val getBeerRes = toResponse(:/("localhost", 2342) / app / "beers" / "Corona" <<? Map("sizeOunces" -> 4))
        "return status 400" >> { getBeerRes.status must_== 400 }

These examples also show one of the Specs matchers 'must_==' The Specs library provides a lot of matchers and if you need some new ones, it's very easy to create them.

For the last two specs we'll do another simple check of the Http response code as well as deserialize the Json response body and inspect the result. To do that, I've created a simple case class. Lift-Json takes care of the rest. Here's what that looks like:


case class JsonBeer(name: String, sizeOunces: Int)


        val getBeerRes = toResponse(:/("localhost", 2342) / app / "beers" / "Guiness" <<? Map("sizeOunces" -> 16))
        "return status 200" >> { getBeerRes.status must_== 200 }
        "return a Beer representation" in {
          val beer = parse(getBeerRes.body).extract[JsonBeer]
          "with" in {
            "name 'Guiness'" >> {beer.name mustEqual "Guiness"}
            "sizeOunces 16" >> {beer.sizeOunces must_== 16}
          }
        }

Pretty slick.

And now the moment of truth. You can run


mvn clean test

or get fancier and run it in your IDE. This screenshot shows output in Netbeans.

There's a bit more setup for the spec class. I've set these tests to share variables and run sequentially. The complete code for BeerSpec.scala and pom.xml is below.

I hope this setup guide has been helpful and encourages some folks to dip their toe in with Scala testing. Happy coding!

/src/test/scala/BeerSpec.scala


import org.specs._
import dispatch._
import net.liftweb.json.JsonParser._
import net.liftweb.json.JsonAST._
import net.liftweb.json.JsonDSL._
import org.apache.http.util.EntityUtils

class BeerSpec extends SpecificationWithJUnit {

  implicit val formats = net.liftweb.json.DefaultFormats // Brings in default date formats etc.
  val app = "scala_test_jersey"
  setSequential()
  shareVariables()

  "Calling GET /beers" should {
    "when called with" in {

      "name 'Budweiser'" in {
        val getBeerRes = toResponse(:/("localhost", 2342) / app / "beers" / "Budweiser")
        "return status 404" >> { getBeerRes.status must_== 404 }
      }

      "name 'Corona' and sizeOunces 4" in {
        val getBeerRes = toResponse(:/("localhost", 2342) / app / "beers" / "Corona" <<? Map("sizeOunces" -> 4))
        "return status 400" >> { getBeerRes.status must_== 400 }
      }

      "beer name 'Guiness' and sizeOunces 16" in {
        val getBeerRes = toResponse(:/("localhost", 2342) / app / "beers" / "Guiness" <<? Map("sizeOunces" -> 16))
        "return status 200" >> { getBeerRes.status must_== 200 }
        "return a Beer representation" in {
          val beer = parse(getBeerRes.body).extract[JsonBeer]
          "with" in {
            "name 'Guiness'" >> {beer.name mustEqual "Guiness"}
            "sizeOunces 16" >> {beer.sizeOunces must_== 16}
          }
        }
      }
    }
  }

  def toResponse(req: Request): StatusCodeBody = {
    val http = new Http
    http x (req as_str) {
      case (status, _, ent, _) => StatusCodeBody(status, EntityUtils.toString(ent.get))
    }
  }
}

case class JsonBeer(name: String, sizeOunces: Int)
case class StatusCodeBody (status: Int, body: String)

pom.xml


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.janxspirit</groupId>
    <artifactId>scala_test_jersey</artifactId>
    <packaging>war</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>scala_test_jersey</name>
    <url>http://maven.apache.org</url>
  
    <repositories>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
    </repositories>
    
    <dependencies>

        <!-- Scala -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.8.0</version>
        </dependency>
        
        <!-- Dispatch -->
        <dependency>
            <groupId>net.databinder</groupId>
            <artifactId>dispatch-http_2.8.0</artifactId>
            <version>0.7.7</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>net.databinder</groupId>
            <artifactId>dispatch-lift-json_2.8.0</artifactId>
            <version>0.7.7</version>
            <scope>test</scope>
        </dependency>

        <!-- Specs -->
        <dependency>
            <groupId>org.scala-tools.testing</groupId>
            <artifactId>specs_2.8.0</artifactId>
            <version>1.6.5</version>
        </dependency>

        <!-- JUnit -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.8.1</version>
        </dependency>

        <!-- Jersey -->
        <dependency>
            <groupId>com.sun.jersey</groupId>
            <artifactId>jersey-json</artifactId>
            <version>1.3</version>
        </dependency>
        <dependency>
            <groupId>com.sun.jersey</groupId>
            <artifactId>jersey-server</artifactId>
            <version>1.3</version>
        </dependency>

    </dependencies>

    <build>
        <testSourceDirectory>${project.basedir}/src/test/scala</testSourceDirectory>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.9.1</version>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>2.0.2</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>

            <!-- Compile Scala -->
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- Compile Java -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>

            <!-- Tests -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.4.3</version>
                <configuration>
                    <useSystemClassLoader>false</useSystemClassLoader>
                    <argLine>-Xmx512m</argLine>
                    <includes>
                        <include>**/*Spec.java</include>
                    </includes>
                </configuration>
            </plugin>

            <!-- Run Jetty -->
            <plugin>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>maven-jetty-plugin</artifactId>
                <version>6.1.24</version>
                <configuration>
                    <connectors>
                        <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector">
                            <port>2342</port>
                        </connector>
                    </connectors>
                    <stopKey>stop</stopKey>
                    <stopPort>9999</stopPort>
                </configuration>
                <executions>
                    <execution>
                        <id>start-jetty</id>
                        <phase>process-classes</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <daemon>true</daemon>
                        </configuration>
                    </execution>
                    <execution>
                        <id>stop-jetty</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>stop</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Create an Executable Scala Jar with SBT

2011-01-27T22:30:00.000-08:00

A friend was asking about building an executable Jar with Scala and SBT. This ended up working for me.

In a fresh SBT project, create the following files:

project/build/AssemblyProject.scala
(this file is lifted in its entirety from here - thanks to Nathan Hamblen)


import sbt._

trait AssemblyProject extends BasicScalaProject {
def assemblyExclude(base: PathFinder) = base / "META-INF" ** "*"
def assemblyOutputPath = outputPath / assemblyJarName
def assemblyJarName = artifactID + "-assembly-" + version + ".jar"
def assemblyTemporaryPath = outputPath / "assembly-libs"
def assemblyClasspath = runClasspath
def assemblyExtraJars = mainDependencies.scalaJars
def assemblyPaths(tempDir: Path, classpath: PathFinder, extraJars: PathFinder, exclude: PathFinder => PathFinder) = {
val (libs, directories) = classpath.get.toList.partition(ClasspathUtilities.isArchive)
for(jar <- extraJars.get ++ libs) FileUtilities.unzip(jar, tempDir, log).left.foreach(error)  
val base = (Path.lazyPathFinder(tempDir :: directories) ##)    (descendents(base, "*") --- exclude(base)).get }   
lazy val assembly = assemblyTask(assemblyTemporaryPath, assemblyClasspath, assemblyExtraJars, assemblyExclude) dependsOn(compile)  
def assemblyTask(tempDir: Path, classpath: PathFinder, extraJars: PathFinder, exclude: PathFinder => PathFinder) =
 packageTask(Path.lazyPathFinder(assemblyPaths(tempDir, classpath, extraJars, exclude)), assemblyOutputPath, packageOptions)
}

project/build/Project.scala


import sbt._

class ExecutableProject(info: ProjectInfo) extends DefaultProject(info) with AssemblyProject {
override def mainClass = Some("RunMe")
}

src/main/scala/RunMe.scala


object RunMe {
def main(args: Array[String]) = {
println("I run.")
}
}

Now in your sbt shell, run:

reload
compile
assembly

That will create an executable jar in the target/scala_*version* directory. You can run it with something like:

java -jar target/scala_2.8.1/runme_2.8.1-assembly-1.0.jar
I run.

Thanks to Nathan Hamblen for the AssemblyProject trait.

A Quick WebApp with Scala, MongoDB, Scalatra and Casbah

2011-01-25T22:13:00.000-08:00

Here's a nice way to get a web service up and running quickly using Scala, Scalatra, SBT and MongoDB. I'm going to assume you have Scala installed but nothing beyond that. Feel free to skip the episodes you've already seen.

Install SBT
Simple Build Tool (SBT) is a build tool for Scala, and it's just super. To install it, download the jar from here: http://code.google.com/p/simple-build-tool/downloads/list

Put that jar someplace and the make a little script along the lines of this:

java -Xmx1512M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512m -jar `dirname $0`/sbt-launch.jar "$@"

Save that alongside the jar you downloaded. Make it executable and add it to your PATH.

MongoDB
MongoDB ( http://www.mongodb.org ) is a document database, meaning it persists JSON-like documents of arbitrary structure. It's very fast and offers very flexible ad hoc querying. A MongoDB document looks like this:

{
_id : ObjectId("4d2167a24d2fa1a764a6f0fa"),
date : "Sun Jan 02 2011 22:07:30 GMT-0800 (PST)",
body : "My blog post!",
upvotes : 3,
downvotes : 0,
comments : [
{
author : "The Dude",
body : "The Dude abides"
},
{
author : "Walter",
body : "Over the line!"
}
]
}

You don't create joins in the database - each document is schema-less and unrelated to other documents, even those in its collection (sort of like a SQL table in MongoDB parlance), But you can do cool stuff like index on those nested comments. Here's how to install it and get started:

Download the tarball for your architecture: http://www.mongodb.org/downloads

Expand it somewhere and add the bin directory to your PATH if you want to.

Run something like this to start the MongoDB server:

$MONGO_HOME/bin/mongod --dbpath=/path/to/place/to/save/mongodb/data

Connect to it and try it out like so:

$MONGO_HOME/bin/mongo

'mongo' is the MongoDB shell, which is a complete Javascript environment and connects to the local server on the default port. Create the example document above like this:

> use temp
switched to db temp
> db.demo.insert({date:new Date(), body:'My blog post!', upvotes: 3, dowvotes: 0, comments: [{author: 'The Dude', body: 'The Dude abides'},{author: 'Walter', body: 'Over the line!'}]})
> db.demo.find()

It's an excellent tool and a lot of fun. There are great docs at http://mongodb.org

Creating the SBT Project
Let's create an application. Make yourself a directory, navigate to it and run sbt. It will guide you through a wizard experience thusly:

$ sbt
Project does not exist, create new project? (y/N/s) y
Name: Demo
Organization: com.demo
Version [1.0]:
Scala version [2.7.7]: 2.8.1
sbt version [0.7.4]:
Getting Scala 2.7.7 ...
...more stuff...
[success] Successfully initialized directory structure.
...more stuff...
[info] Building project Demo 1.0 against Scala 2.8.1
[info]    using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
>

Leave that shell open to the sbt prompt. In your editor of choice, create a file in your project directory named

project/build/Project.scala

Add the following to it:

import sbt._
class build(info: ProjectInfo) extends DefaultWebProject(info) {
// scalatra
val sonatypeNexusSnapshots = "Sonatype Nexus Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
val sonatypeNexusReleases = "Sonatype Nexus Releases" at "https://oss.sonatype.org/content/repositories/releases"
val scalatra = "org.scalatra" %% "scalatra" % "2.0.0.M2"

// jetty
val jetty6 = "org.mortbay.jetty" % "jetty" % "6.1.22" % "test"
val servletApi = "org.mortbay.jetty" % "servlet-api" % "2.5-20081211" % "provided"

//casbah
val casbah = "com.mongodb.casbah" %% "casbah" % "2.0.1"
}

In your sbt shell, run

reload

and then

update

SBT will download dependencies ending with a message of success.

Add a Scalatra GET handler
Scalatra is refreshingly simple. To create a Scalatra servlet with a simple GET handler, using Scala's support for XML literals, create a file like src/main/scala/WebApp.scala with the following content:

import org.scalatra._

class WebApp extends ScalatraServlet {
get("/hello") {
<html><head><title>Hello!</title></head><body><h1>Hello</h1></body></html>
}
}

web.xml

One last bit of setup is to add the Scaltra servlet to the web.xml. Create a file named /src/main/webapp/WEB-INF/web.xml with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE web-app
PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.2//EN"
"http://java.sun.com/j2ee/dtds/web-app_2_2.dtd">
<web-app>

<servlet>
<servlet-name>WebApp</servlet-name>
<servlet-class>WebApp</servlet-class>
</servlet>

<servlet-mapping>
<servlet-name>WebApp</servlet-name>
<url-pattern>/*</url-pattern>
</servlet-mapping>
</web-app>

Back at the SBT prompt you just need to

reload

and then do the following to start jetty and listen for source file changes:

jetty-run
~prepare-webapp

Open a browser to http://localhost:8080/hello and you should get a working Scalatra app. Change the message to something more interesting, like "Hello World" or "Goodnight Moon", save the source file, and SBT will recompile and redeploy. Just reload the web browser to see your changes.

Parameters

Add a path parameter to your GET call thusly:


get("/hello/:name") {
<html><head><title>Hello!</title></head><body><h1>Hello {params("name")}!</h1></body></html>
}

Save the file, browse to http://localhost:8080/hello/handsome

Oh yeah!

A Form

Add a form like this:

 get("/msgs") {
<body>
<form method="POST" action="/msgs">
Author: <input type="text" name="author"/><br/>
Message: <input type="text" name="msg"/><br/>
<input type="submit"/>
</form>
</body>
}

You can see the fruits of your labor at http://localhost:8080/msgs

Casbah

Casbah is the officially supported Scala driver for Mongo. It wraps the Java driver and makes accessing MongoDB straightforward using idiomatic Scala. http://api.mongodb.org/scala/casbah/2.0.2/index.html

You already have the driver dependency in your project, but you'll need a few new imports:

import com.mongodb._
import com.mongodb.casbah.Imports._
import scala.xml._

And a MongoDB connection that can be shared across the application:

val mongo = MongoConnection()
val coll = mongo("blog")("msgs")

Add a method to handle the form POST:


post("/msgs") {
val builder = MongoDBObject.newBuilder
builder += "author" -> params("author")
builder += "msg" -> params("msg")

coll += builder.result.asDBObject
redirect("/msgs")
}

Slick, right? That's all it takes to add a new document to the "msgs" collection in the "blog" db. You don't even need to create the db and collection first - just make sure MongoDB is running on the local host at the default port 27017.

Now you should adjust the form page to display the existing messages. This code treats iterates over the whole collection - in a real-world application you'd want paging, querying, filtering etc.

Add this code to your get() method just after the body tag:

 
<ul>
{for (l <- coll) yield <li>          
 From: {l.getOrElse("author", "???")} -                           
 {l.getOrElse("msg", "???")}</li>}                       
 </ul>

Once your app builds and you reload the page, your form should work to add new messages with authors and display them.

In Conclusion

I've found this stack a great way to whip up a quick example, prototype or test. You can use SBT to build a war (the 'package' command) and deploy it to any container you want.

I've barely scratched the surface of any of these bits, but hopefully piqued some interest in some cool tools. There are great docs and examples for all of these projects, so dig in and have fun!

Source code:

project/build/Project.scala:


import sbt._
class build(info: ProjectInfo) extends DefaultWebProject(info) {
// scalatra
val sonatypeNexusSnapshots = "Sonatype Nexus Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
val sonatypeNexusReleases = "Sonatype Nexus Releases" at "https://oss.sonatype.org/content/repositories/releases"
val scalatra = "org.scalatra" %% "scalatra" % "2.0.0.M2"

// jetty
val jetty6 = "org.mortbay.jetty" % "jetty" % "6.1.22" % "test" servletApi = "org.mortbay.jetty" % "servlet-api" % "2.5-20081211" % "provided"

//casbah
val casbah = "com.mongodb.casbah" %% "casbah" % "2.0.1"
}

src/main/scala/WebApp.scala:


import org.scalatra._
import com.mongodb.casbah.Imports._
import scala.xml._
import com.mongodb._

class WebApp extends ScalatraServlet {

val mongo = MongoConnection()
val coll = mongo("blog")("msgs")

get("/hello/:name") {
<html><head><title>Hello!</title></head><body><h1>Hello {params("name")}!</h1></body></html>
}

get("/msgs") {
<body>
 <ul>
   {for (l <- coll) yield <li>        
   From: {l.getOrElse("author", "???")} -       
   {l.getOrElse("msg", "???")}</li>}           
 </ul>
 <form method="POST" action="/msgs">
   Author: <input type="text" name="author"/><br/>
   Message: <input type="text" name="msg"/><br/>
   <input type="submit"/>
 </form>
</body>
}

post("/msgs") {
val builder = MongoDBObject.newBuilder
builder += "author" -> params("author")
builder += "msg" -> params("msg")

coll += builder.result.asDBObject
redirect("/msgs")
}
}