Archive for the ‘Scala’ Category

Play! Framework 2 (2.1) Scala with Slick made easy (With example)

by @mgonto, original article can be found here

Hi everyone!

First of all, I want to tell you that the original post is from here (Blogeek).

Today I’ve seen some videos about Slick and Play! Framework 2.1 and I got excited, so I thought, let’s code an app with Slick. What did I realize? It wasn’t that simple. Play! isn’t yet thought to work with Slick very well.

First of all, if you want to see this example complete you can go to

https://github.com/mgonto/slick-play2-example

In Slick, you create table definitions for the class you want to use. However, all of the imports for the Table, the columns, etc. are driver dependant. So, when you change the driver, you don’t want to change the import in the code. You just want to change a configuration in the application.conf from Play! Framework. That’s exactly what I did.

So, let’s see first how it’s implemented. Every Driver (H2, Postgres, etc) extends from a trait called ExtendedProfile. So, I needed my table classes to use some profile that was injected by me. Once I realized that I needed to make some DI I thought about Cake Pattern. For more information about the Cake Pattern please viisit Cake pattern

So, the first thing I implemented was a Trait Profile with an abstract Extended Profile value:

package models

import slick.driver.ExtendedProfile

trait Profile {
  val profile: ExtendedProfile
}

After this, I needed to create the Users table. That Users table needed the profile, so I’d create a UserModule which would need the Profile trait. Then, as I’d mix with a Profile trait, I’d import the things from the abstract ExtendedProfile field and everything would compile. Then I create the users table with an ID and a Name (pretty simple)

package models

case class User(id: Option[Int], name : String)

trait UserComponent {
  this: Profile =>

  import profile.simple._

  object Users extends Table[User]("users") {
    def id = column[Int]("id", O.PrimaryKey)
    def name =  column[String]("name", O.NotNull)
    def * = id.? ~ name <> (User, User.unapply _)

    def add(user: User)(implicit session: Session) = {
      this.insert(user)
    }

    def countByName(name: String)(implicit session: Session) = {
      (for {
        user <- Users
        if (user.name === name)
      } yield(user)).list.size
    }

  }
}

After this, I create a DataAccessLayer class (DAL) which would receive the ExtendedProfile as a constructor parameter and then would include the other traits (Profile and UserModule)

package models

import slick.driver.ExtendedProfile

class DAL(override val profile: ExtendedProfile) extends UserComponent with Profile {

  import profile.simple._

  def create(implicit session: Session): Unit = {
    Users.ddl.create //helper method to create all tables
  }
}

After this, I have everything to create a DAL from a given profile. However, now I needed a way to configure the profile, so I added a new property in the application.conf.

slick.db.driver=scala.slick.driver.H2Driver

Then, I created a trait that would import this driver from the conf and create the database to do the queries and the dal with the corresponding profile. It has two methods, getDb and getDal which will be then used by two other classes.

package models

import slick.session.Database
import play.api.db.DB
import play.api.Application
import slick.driver.ExtendedProfile

trait DBeable {

  val SLICK_DRIVER = "slick.db.driver"
  val DEFAULT_SLICK_DRIVER = "scala.slick.driver.H2Driver"

  def getDal(implicit app : Application) : DAL = {
    val driverClass = app.configuration.getString(SLICK_DRIVER).getOrElse(DEFAULT_SLICK_DRIVER)
    val driver = singleton[ExtendedProfile](driverClass)
    new DAL(driver)
  }

  def getDb(implicit app : Application) = {
    Database.forDataSource(DB.getDataSource())
  }

  private def singleton[T](name : String)(implicit man: Manifest[T]) : T =
    Class.forName(name + "$").getField("MODULE$").get(man.runtimeClass).asInstanceOf[T]

}

The only things left are creating the DB once the app starts and having a singleton object to be able to ask for the dal and the database to do the queries from a controller. This is done as following:

import models.{User, DBeable, AppDB}
import play.api.db.DB
import play.api.GlobalSettings
import play.api.Application
import slick.session.Session

object Global extends GlobalSettings with DBeable {

  override def onStart(app: Application) {
    implicit val application = app
    lazy val database = getDb
    lazy val dal = getDal
    database.withSession {
      implicit session: Session =>
        dal.create
    }
  }
}
package models
import play.api.Play.current
object AppDB extends DBeable {

  lazy val database = getDb
  lazy val dal = getDal

}

And that’s it :). Now you have everything configured to start coding. I’ve tried if everything was working with a test as following.

class UserSpec extends Specification {

  "User" should {

    "be saved" in {
      running(FakeApplication(additionalConfiguration = inMemoryDatabase())) {
        AppDB.database.withSession {
          implicit session: Session =>
            AppDB.dal.Users.add(User(Some(2), "hola"))
            AppDB.dal.Users.countByName("pepe") must beEqualTo(0)
            AppDB.dal.Users.countByName("hola") must beEqualTo(1)

        }
      }
    }
  }
}

And that’s it 🙂 Now you have Play! configured for running with Slick.

Cya people!

Advertisements

First steps with Scala revisited once more: Scala’s implicit revenge

Last post might have left a bitter taste in scala’s fans mouth. In there, after developing a very simple script with scala, bash and python, I found the latter one better suited for the task.

But my first attempt with scala, was just an excuse to show some pretty powerful scala features, like Type Inference, collections, higher order functions, functional programming and some other goodies.

So this time I’ll take a more pragmatic -and less pedagogic- approach, and will just try to mimic the python script.

Here’s my first attempt:

#!/bin/sh
exec scala -savecompiled "$0" "$@"
!#

import java.io._

val total_files = new File(".").listFiles.filter(_.getName.endsWith(".textile"))
val translated_files = total_files.filter { file => 
  new BufferedReader(new FileReader(file)).readLine.
    indexOf("Esta página todavía no ha sido traducida al castellano") == -1
}

val total_size = total_files.map(_.length).sum / 1000
val translated_size = translated_files.map(_.length).sum / 1000
val size_percent = translated_size * 100 / total_size

println( "translated size: %dkb/%dkb %d%% (pending %dkb %d%%)" 
  format(translated_size, total_size, size_percent, total_size-translated_size, 100-size_percent)
)

val total_count = total_files.length
val translated_count = translated_files.length
val count_percent = translated_count * 100 / total_count

println( "translated files: %d/%d %d%% (pending %d %d%%)" 
  format(translated_count, total_count, count_percent, total_count-translated_count, 100-count_percent)
)

You can compare it to the python version below:

#! /usr/bin/env python
# -*- coding: utf-8 -*-

import fnmatch
import os

total_files = [file for file in os.listdir('.') if fnmatch.fnmatch(file, '*.textile')]
translated_files = [file for file in total_files if "Esta página todavía no ha sido traducida al castellano" not in open(file).read()]

total_size = sum([os.path.getsize(file) for file in total_files]) / 1000
translated_size = sum([os.path.getsize(file) for file in translated_files]) / 1000
translated_percent= translated_size * 100 / total_size

print "translated size: %dkb/%dkb %d%% (pending %dkb %d%%)" % \
      (translated_size, total_size, translated_percent, total_size-translated_size, 100-translated_percent)

total_count=len(total_files)
translated_count=len(translated_files)
translated_percent= translated_count * 100 / total_count

print "translated files: %d/%d %d%% (pending %d %d%%)" % \
      (translated_count, total_count, translated_percent, total_count-translated_count, 100-translated_percent)

Well, no such a big deal, both scripts are functionally equivalent, number of lines is approximately the same, I guess it’s now just a matter of personal taste.

Of course there are quite a lot of differences, scala runs on the JVM, is type safe and compiled. On the other hand, Python is more lightweight and a little less verbose.

That’s specially true for the “new BufferedReader(new FileReader(file)).readLine” part of the script, so I think we can improve it.

Scala’s io package

We can take advantage of Scala’s io package. Add the following import: “import scala.io.Source._” and just issue:

val translated_files = total_files.filter { file => 
  fromFile(file).mkString.
    indexOf("Esta página todavía no ha sido traducida al castellano") == -1
}

This reads the whole file, in order to read just the first line (like we did using the BufferedReader) do like this:

val translated_files = total_files.filter { file => 
  fromFile(file).getLines.take(1).mkString.
    indexOf("Esta página todavía no ha sido traducida al castellano") == -1
}

The current scala io package isn’t quite mature as java’s libraries. In fact, if you look at the documentation, apparently it was meant to be used mainly for source files. Work is being done on a new standard io library named scala-io. Luckily, you can always turn to the good old java libraries.

Well, it’s a little better, but we can make it more elegant and, at the same time, learn a very powerful scala feature.

Implicit conversions: Scala’s answer to open classes

Ruby is pretty famous, among other things, for letting you easily add new methods to existing classes, like this:

class Integer
  def squared
    self * self
  end
end

That will allow you to do this:

>> 4.squared
=> 16

Now in Scala, being a type-safe statically compiled language, that sort of magic is not so easy to accomplish, nevertheless they found a pretty elegant and safe way to do it.

You don’t touch the original class, you just define a new one, with the desired methods to add, and define a function that will implicitly convert from one type to the other.

To give it a try we will use the interactive scala interpreter, better known as REPL, which stands for Read-Evaluate-Print Loop.

So just type “scala” and you’ll be taken to the REPL. There we’ll create the class with the new method to enchance the Int class.

scala> class EnhancedInt(x: Int) { def squared = x * x }

And with the following code we’ll tell scala to automatically convert a regular Int to a EnhancedInt where the context demands it:

scala> implicit def IntToEnchancedInt(x: Int) = new EnhancedInt(x)

and after that you can try it

scala> 4.squared
res0: Int = 16

You can check this article, from where I took the example, for a much detailed explanation.

Enhancing the File class

What we want to do, is to let the class File tell us if the file is translated or not.

To do that we’ll just create a DocumentationFile class, like this:

class DocumentationFile(val file: File) {
  val isTranslated = (
    fromFile(file).getLines.take(1).mkString.
      indexOf("Esta página todavía no ha sido traducida al castellano") == -1
  )
}

And then add the corresponding implicit def to tell scala how to convert from a regular File to a DocumentationFile:

implicit def fileToDocumentationFile(file: File): DocumentationFile = new DocumentationFile(file)

And now, we can get the translated files with this neat line of code

val translated_files = total_files.filter(_.isTranslated)

Note that just like we did in the example with the Int class, with a couple of lines of code, we are enhancing an existing -and closed- Java class using scala. As we said earlier, it’s a pretty powerful feature, that should be used with caution. After a couple of these tricks, with many implicit conversions going around, following the code can get a bit messy.

This is the complete source code:

#!/bin/sh
exec scala -savecompiled "$0" "$@"
!#

import java.io._
import scala.io.Source._

val total_files = new File(".").listFiles.filter(_.getName.endsWith(".textile"))
val translated_files = total_files.filter(_.isTranslated)

val total_size = total_files.map(_.length).sum / 1000
val translated_size = translated_files.map(_.length).sum / 1000
val size_percent = translated_size * 100 / total_size

println( "translated size: %dkb/%dkb %d%% (pending %dkb %d%%)" 
  format(translated_size, total_size, size_percent, total_size-translated_size, 100-size_percent)
)

val total_count = total_files.length
val translated_count = translated_files.length
val count_percent = translated_count * 100 / total_count

println( "translated files: %d/%d %d%% (pending %d %d%%)" 
  format(translated_count, total_count, count_percent, total_count-translated_count, 100-count_percent)
)

implicit def fileToDocumentationFile(file: File): DocumentationFile = new DocumentationFile(file)

class DocumentationFile(val file: File) {
  val isTranslated = (
    fromFile(file).getLines.take(1).mkString.
      indexOf("Esta página todavía no ha sido traducida al castellano") == -1
  )
}

Conclusion

Now the differences between the python and the scala version aren’t so huge anymore. It is true that right now scala is lacking a more powerful IO standard library for dealing with files, but you can always use the vastly tested java libraries.

After this I think that scala has somehow recovered it’s reputation for scripting purposes, and at the same time we could have a look at implicit conversions, a powerful scala feature.

First steps with Scala, a functional alternative to bash scripts…

update: after getting some feedback I decided to make a bash and python version of this script. Check it out here.

another update: I revisited the scala version of the script, and at the same time talked a little bit about Implicit conversions, Scala’s answer to ruby’s open classes. Have a look at it here.

Those who know me are aware that I’ve been following play framework, and actively taking part of it’s community, for a couple of years.

Playframework 2.0 is right around the corner, and it’s core is programmed in Scala, so it’s a wonderful opportunity to give this object-oriented / functional hybrid beast a try…

Like many others, I will pick a very simple script to give my first steps…

Finding an excuse to give Scala a try

With a couple of friends we are on the way to translate play framework documentation to spanish (go have a look at it at http://playdoces.appspot.com/, by the way, you are more than welcome to collaborate with us)

The documentation is composed of a bunch of .textile files, and I had a very simple and silly bash script to track our advance. Every file that has not yet been translated has the phrase “todavía no ha sido traducida” in it’s first line

echo pending: `grep "todavía no ha sido traducida" * | wc -l` / `ls | wc -l`

Which produced something like

pending: 40 / 63

Pretty simple, right?

I just wanted to develop a simple scala script to count the translated files, and also it’s size, to know how much work we had ahead.

Scala as a scripting language

Using scala as a scripting language is pretty simple. Just enter some scala code in a text file, and execute it with “scala file.scala“. You can also try it with the interactive interpreter, better knonw as REPL (well, it’s not really an interpreter, but a Read-Evaluate-Print Loop, that’s where the REPL name comes from).

In linux, you can also excute them directly from the shell marking the scala file as executable and adding these lines to the beginning of the file.

#!/bin/sh
exec scala "$0" "$@"
!#

Tip: you can speed up A LOT script execution by adding a -savecompiled like it says on the scala command man page, like this:

#!/bin/sh
exec scala -savecompiled "$0" "$@"
!#

Classes and type inference in scala

So I created a DocumentationFile, with a name, length and an isTranslated property.

class DocumentationFile(val file: File) {

  val name = file.getName
  val length = file.length
  val isTranslated = (firstLine.indexOf("Esta página todavía no ha sido traducida al castellano") == -1)

  def firstLine = new BufferedReader(new FileReader(file)).readLine

}

Scala takes away a lot of boilerplate code. The constructor is right there, along with the class declaration. In our case, the DocumentationFile constructor takes a java.io.File as argument.

Scala also makes heavy use of type inference to alleviate us from having to declare every variable’s type. That’s why you don’t have to specify that name is a String, length a Long and isTranslated a Boolean. You still have to declare types on method’s arguments, but usually you can omit them everywhere else.

Working with collections

Next I needed to get all textile files from the current directory, instantiate a DocumentationFile for each of them, and save them in an Array for later processing.

import java.io._

val docs = new File(".").listFiles
  .filter(_.getName.endsWith(".textile"))   // process only textile files
  .map(new DocumentationFile(_))

Technically speaking is just one line of code. The “_” is just syntactic sugar, we could have written it in a more verbose way like this:

val docs = new File(".").listFiles
  .filter( file => file.getName.endsWith(".textile") )   // process only textile files
  .map( file => new DocumentationFile(file) )

Or if you are a curly braces fun:

val docs = new File(".").listFiles
  .filter { file => 
    file.getName.endsWith(".textile")         // process only textile files
  }   
  .map { file => 
    new DocumentationFile(file)
  }

Higher order functions

Once we have all textile files, we’ll need the translated ones.

val translated = docs.filter(_.isTranslated)

Here we are passing the filter method a function as parameter (that’s what is called a higher order function). That function is evaluated for every item in the Array, and if it returns true, that item is added to the resulting Array. The “_.isTranslated” stuff is once again just syntactic sugar. We could have also written the function as follows:

val translated = docs.filter( (doc: DocumentationFile) => doc.isTranslated )

Functional versus imperative: To var or not to var

Now I need to calculate the quantity and size of the translated and not yet translated files. Counting the files is pretty easy, just have to use “translated.length” to know how many files have been translated so far. But for counting their size I have to sum the size of each one of them.

This was my first attempt:

var translatedLength = 0L
translated.foreach( translatedLength += _.length ) 

In scala we can declare variables with the “var” and “val” keywords, the first ones are mutable, while the later one ar immutables. Mutable variables are read-write, while immutable variables can’t be reassigned once their value has been established (think of them like final variables in Java).

While scala allows you to work in an imperative or functional style, it really encourages the later one. Programming in scala, kind of the scala bible, even teaches how to refactor your code to avoid the use of mutable variables, and get your head used to a more functional programming style.

These are several ways I’ve found to calculate it in a more functional style (thanks to stack overflow!)

val translatedLength: Long = translated.fold(0L)( (acum: Long, element: DocumentFile) => acum + element.length )

//type inference to the rescue
val translatedLength = translated.foldLeft(0L)( (acum, element) => acum + element.length )

//syntactic sugar
val translatedLength = translated.foldLeft(0L)( _ + _.length )

// yes, if statement is also an expression, just like the a ? b : c java operator.
val translatedLength = if (translated.length == 0) 0 else translated.map(_.length).sum

I’ve finally settled with this simple and short form:

val translatedLength = translated.map(_.length).sum
val docsLength = docs.map(_.length).sum

Default parameters and passing functions as arguments

Now I have all the information I needed, so I just have to show it on screen. I also wanted to show the file size in kbs.

Once again this was my first attempt:

println( 
  "translated size: " + asKB(translatedLength) + "/" + asKB(docsLength) + " " + 
  translatedLength * 100 / docsLength + "% "
)

println( 
  "translated files: " + translated.length + "/" + docs.length + " " + 
  translated.length * 100 / docs.length + "% "
)

def asKB(length: Long) = (length / 1000) + "kb"

And this was the output:

translated size: 256kb/612kb 41% 
translated files: 24/64 37% 

Well, it worked, but it could definitely be improved, there was too much code duplication.

So I created a function that took care of it all:

def status(
  title: String = "status", 
  current: Long, total: Long, 
  format: (Long) => String = (x) => x.toString): String = {

  val percent = current * 100 / total

  title + ": " + format(current) + "/" + format(total) + " " +
  percent + "%" +
  " (pending " + format(total - current) + " " +
  (100-percent) + "%)"
}

The only tricky part is the format parameter. It’s just a higher order function, that by default just converts the passed number to a String.

We use that function like this:

println( 
  status("translated size", translatedLength, docsLength, (length) => asKB(length) ) 
)

println( 
  status("translated files", translated.length, docs.length) 
)

And that’s it.

It’s really easy to achieve this kind of stuff using scala as a scripting language, and on the way you may learn a couple of interesting concepts, and give your first steps into functional programming.

In the next article, I have a look at a bash and python version of this script, to compare the scripting capabilities of each of them.

This is the complete script, here you have a github gist and you can also find it in the play spanish documentation project.

#!/bin/sh
exec scala -savecompiled "$0" "$@"
!#

import java.io._

val docs = new File(".").listFiles
  .filter(_.getName.endsWith(".textile"))   // process only textile files
  .map(new DocumentationFile(_))

val translated = docs.filter(_.isTranslated)    // only already translated files

val translatedLength = translated.map(_.length).sum
val docsLength = docs.map(_.length).sum

println( 
  status("translated size", translatedLength, docsLength, (length) => asKB(length) ) 
)

println( 
  status("translated files", translated.length, docs.length) 
)

def status(
  title: String = "status", 
  current: Long, total: Long, 
  format: (Long) => String = (x) => x.toString): String = {

  val percent = current * 100 / total

  title + ": " + format(current) + "/" + format(total) + " " +
  percent + "%" +
  " (pending " + format(total - current) + " " +
  (100-percent) + "%)"
}

def asKB(length: Long) = (length / 1000) + "kb"

class DocumentationFile(val file: File) {

  val name = file.getName
  val length = file.length
  val isTranslated = (firstLine.indexOf("Esta página todavía no ha sido traducida al castellano") == -1)

  override def toString = "name: " + name + ", length: " + length + ", isTranslated: " + isTranslated

  def firstLine = new BufferedReader(new FileReader(file)).readLine

}

updates:

I added some tips and advices I got from all the positive feedback to this article:

    2012-01-13 changed title, from “First steps with Scala, say goodbye to bash scripts…” to the current one, it’s less polemic, but more accurate.
%d bloggers like this: