What is old is new ?

Having dabbled deeply in the Javascript world for a few weeks, and looking at some trends, it seems like either a whole lot of Java devs made their way into JS world and are trying some of the same patterns again, or certain patterns have a way of self emerging.  For example, React as a component framework is reminiscent of JSF only client side, and we hope sucks less.  But the basic idea and promise, sounds pretty familiar.  Or for example, things like immutable-js looks like some ideas where based on the Java Collection API.  Then we have FOAM and TypeScript bringing strong typing back.  FOAM is interesting getting all meta on stuff.  We’ve seen that pattern before except we used to code gen from the tables, or use some other home grown declarative approach.

This post isn’t a dig on the JavaScript landscape not at all, the quality of and velocity of innovation is pretty remarkable, especially for a core language that really doesn’t do that much by itself, and was invented in 10 days. I’ll write another post on my reflections on this subject, now back to the old and the new…

I remember when I was a young lad pursing my second undergraduate degree (don’t presume I finished the first, because I didn’t), I had to write a paper to get accepted into the computer science program.  I remember reflecting on how math doesn’t change, and even though you are teaching me antiquated methods, I recognize that the patterns are transferable.  Apparently, me telling them that their tech was not up to date didn’t piss them off enough to exclude me, or maybe I was spot on with that analysis.

Now after about 18 years in the industry, I think that reflection was spot on.  I’ve been watching client/server become request/response, and now single page apps look a lot more client/server.  It was do all rendering on the server and now its do all rendering on the client.  On the backend, we had don’t do any processing in the data store (kill those sprocs!), but then map/reduce comes around and says, wait do the processing close the the data.  Sorry man, we were wrong, that was a good idea after all, we just need to tweak it a bit.  Doesn’t docker remind you of the JVM conceptually, it’s something like a virtual machine that sits on another machine, but doesn’t require the whole OS, in other words I can run several dockers on one box, just like I can run several JVM processes, albeit JVM will run out of memory much sooner!.  Tech trends come and go, and a lot of them sound the same as before, sometimes they improve on the past, sometimes they make the same mistakes as the past.  In general, it always back to square one:

• Pick the right tool for the job.
• Don’t add unnecessary layers of abstraction where they aren’t necessary, or aren’t bringing real value.
• Don’t trust that whatever X framework is promising without some tangible demonstration.  The academic promise, if its nothing more than that, can not be trusted, and even when it works, has limits.
• Whatever fail safe pure layer you create, some asshole will find a way to leak it, trust me, I’m often that asshole.
• Beware of easy to start hard to finish.  The industry is all about time to market, time to market, bring the value.  But remember 80% of the time and effort is spent maintaining and supporting the software.  Maintenance and sustainability are crucial.  Regression test are great, but the compiler gives you stuff for free.

So next time you want to poke fun at the COBOL, ok now its the legacy Java, because its 10 or 20 years old.  Think to yourself, will that transpiler that was just invented yesterday be around in 2 years even?  Software that keeps adding value over time is hard.  If you work somewhere that has some 5, 10, 20 year old code, instead of cursing at that shit, maybe stop and realize that its amazing that stinky crap still works, and its still providing value (and maybe your paycheck!), do you think your shitty ass code will be around that long?

I think the microservice trend is fine and I can see that value.  I like that it forces the division between layers, and decomposes problems, its largely about risk management, and quick deployments.  But on the other hand it’s also a cop out to believe that software can’t have any longevity.  Maybe its a dying art, maybe I’m just an old fogey, maybe the crap created today really won’t be around in 10 years, none of it and I should just accept that.  But seeing as how the patterns keep coming back and fading away just like fashion, I’m thinking a sturdy pairs of jeans will do just fine.  They might take awhile to put on, but once they are on, I can do anything. And if I can have quick time to market and maintainability, I’m picking that one… Scalalala

Logstash and Playframework

I’m not sure why google let me down in regards to hooking up logstash and Play, but it sent me on some pretty weird paths. So I’m going to share what we did to get it working that is pretty simple in the end.

Play uses logback, first rule is don’t try to include a new version of logback in your build, that will cause you conflicts, the ootb Play dependencies are all you need, at the time I did this, we were using Play 2.3.8.

In your logback config, logger.xml, just wire up the appender you want, the tcp or udp one like this:

	<!– for udp –>
	<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashSocketAppender">
	<host>logstash_server/host>
	<port>logstash_port</port>
	<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
	</appender>
	<!– for tcp –>
	<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
	<host>logstash_server/host>
	<port>logstash_port</port>
	<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
	</appender>
	<!– include the appender –>
	<root level="INFO">
	<appender-ref ref="LOGSTASH" />
	</root>

view raw

gistfile1.xml

hosted with ❤ by GitHub

Then on the logstash config side create a new input:

	input {
	udp {
	codec => json
	port => XXXX
	type => logback
	}
	}

view raw

gistfile1.txt

hosted with ❤ by GitHub

This puts your data into a new type (called “logstash”) in the ES indices so that the json mappings don’t conflict with anything else. That’s it.

Retry

One of the things I love about functional programming, and Scala in particular is that it feels like you can make new language constructs.  One common pattern I run into is having to do something a few times until hopefully it works.  This is especially relevant when you are calling some remote endpoint that might be busy, slow, or down or whatever.

I found this little gem on stackoverflow:

	@annotation.tailrec
	final def retry[T](n: Int)(fn: => T): T = {
	util.Try { fn } match {
	case util.Success(x) => x
	case _ if n > 1 => retry(n – 1)(fn)
	case util.Failure(e) => throw e
	}
	}

view raw

gistfile1.scala

hosted with ❤ by GitHub

What’s going on here?  We take an Int and a function.  Then try to call the function.  If it successful then return its value, else recursively call ourselves again until we run out of tries.  If we haven’t succeeded in a certain number of tries then throw the exception the function gave us.

If you don’t fully understand what the @annotation.tailrec does, do yourself a favor and read this article.  It does a great job of explaining it.  But in a nutshell, the Scala compiler can optimize a tail recursive function into a loop, which means no more stack overflows.  Surprisingly, this feature is rare in a lot of modern day languages.  I think in Ruby you can compile it in, Python is philosophically opposed to it.  The more purely functional languages like Clojure, Erlang, and Haskell of course would do better.  Sorry Javascript, you have to trampoline too, how sad for you…  But I’m sure there are a bunch of transpilers that can fix that up, I wonder what Scala.js does there?

Ok so enough of that, the real story is in how easy it in to use this guy.

	retry(10) {
	// to some stuff that might need a few kicks in the butt before it works
	}

view raw

gistfile1.scala

hosted with ❤ by GitHub

It’s just that easy you can retry how ever many times you like else throw your exception…

Hot Threads

If you have ever used elasticsearch and you haven’t discovered the hot_threads endpoint, take a look at it.  It basically figures out which threads on each node are consuming the most cpu and then gives you a stack trace.  Its like a rest endpoint for thread dump around the cluster. It looks something like this:

In a previous project we were running elasticsearch embedded in our monolith, so this was pretty handy, because it gave you a view into not only what ES was doing but the whole server.

I had a issue the other day where my monitoring was not helping me out, and I needed a stack dump, but I was having some trouble getting one.  So I decided I don’t want to be in this position again standing there holding my dick, so I thought I wonder if I could just lift this thing out of elasticsearch, or at least borrow some code.

Well it turns out it was easier than I thought.   I poked around in github found the HotThreads class.  Since I already had the elasticsearch api included in my project, all I had to do in Play was just wire up a quick controller like this, and then map in the routes file.

	import org.elasticsearch.monitor.jvm.HotThreads
	import play.api.mvc._
	object HotThreadsController extends BaseController {
	def getHotThreads = Action {
	Ok(new HotThreads().detect())
	}
	}

view raw

gistfile1.scala

hosted with ❤ by GitHub

That’s it, then you have a nice admin endpoint to see running threads.  It’s not across the cluster its per node, that’s a job for another time, but still pretty cool!

Either you’re with us or against us

So I was happily going on with my Scala journey avoid exceptions everywhere.  Then I ran into this use case where I sorta needed them.  But I really liked not having try/catch all over the place mucking up stuff.  So here’s this use case:

I’m in the middle tier, people call me via Rest, then I call sql, sprocs and more sprocs, Rest, your mom’s old couch, whatever and make that rationale and pass it back up.  The thing is that Rest service I’m calling might poop out on me, and I’d like to just pass up the json it sends me to the guy above me.  So I have this situation where I need to return two things.

I know exceptions can work here, I can return my object in the good case, or an exception in the bad case, but I really hate checked exceptions, not to get into a whole theological discussion about it, but I’m definitely in the camp of exceptions are best for exceptional stuff.  That whole FinderException crap EJB started, pisses me off.  Sun really went ape shit with that stuff.  I digress..

So how can I return two different things.  Well I could create some wrapper object thingy.  Yeah that would work, but its so icky, and then, I’m going to have to down cast stuff, or have a bunch of icky wrappers.  I don’t think that will work.  Then I discovered Either.

A common use of Either is as an alternative to Option for dealing with possible missing values. In this usage, scala.None is replaced with a Left which can contain useful information.Right takes the place of Some. Convention dictates that Left is used for failure and Right is used for success.

This is the medicine I was looking for.  I can return this, or I can return that.  OMG, genius!!!  What is even better, is that the convention is that the good stuff is on the right, and the bad stuff on the left.  Now Scala is even jelling with my political sensibilities.  Wow, mind blow.

So I can write stuff like this which is pretty fun:

 

	// if there is something there Cache it
	// clearly Martin Odersky is not a progressive, well or left handed 🙂
	if (entityEither.isRight && entityEither.right.get.isDefined){
	Cache.set(getCacheKey(entityName, lookupKey), entityEither.right.get.get, 60)
	}

view raw

gistfile1.txt

hosted with ❤ by GitHub

So back to how I use this.  When I call into my store like this:

	def getEntitiesByKeyFromDatastore[T](
	entityName: String,
	lookupKey: (String, String),
	reads: Reads[T]): Either[DataStoreResponse, List[T]]

view raw

gistfile1.txt

hosted with ❤ by GitHub

I send back up an EIther.  Which has a list in this case and then the raw DataStoreResponse on the left.  So if something bad happens, you get the raw DataStoreResponse, else the object(s) you were looking for. So in my case where I’m a service sending json back up in the good or bad case, things end up like this:

	def getExceptionTemplateByEnv(templateId: String) = Action {
	Core.getExceptionTemplatesByEnv(templateId).fold (
	outputDataStoreResponse,
	(templates) => Ok(Json.toJson(templates))
	)
	}
	def outputDataStoreResponse: (DataStoreResponse) => Result = {
	(dataStoreResponse) => NotFound(dataStoreResponse.jsValue)
	}

view raw

gistfile1.txt

hosted with ❤ by GitHub

That’s so much nicer that try/catch all over the place.  But the thing is, this is stable, resilient code, it handles failure well.  But it doesn’t hurt your eyes to look at, is very concise, and reads well.

Chain Gang

So as my Scala journey has progressed.  It has really stepped up since I’ve switched jobs and been able to focus on it for a few weeks without distraction.  I’ve found that I’ve crossed into the functional style.  If statements annoy me now, and anything mutable causes me concern.  I was doing really well, getting the hang of things and then I ran into an issue.

Here was my use case.  I’ve got a deep domain, its heavily customized per customer in many ways.  The way I’m dealing with this right now, is that the actual field names of wide domain objects have customized labels per client.  We are talking 100 columns or more that can be customized.  I knew I needed to get all meta data with it, and use reflection, no problem, been there before.  But what I wanted to do, was read data from the store, then apply the label just in time, I don’t want to do a bunch of “select * as name from”  …, but rather get data, and then transform it.  Partially, because that would perform badly, partially because I’m going to have to transform the same data again and again, I don’t want that logic all wrapped up in the data access layer, and frankly its not available there, well yet anyway.

So that’s my use case, here’s my technical problem.  I had this list I needed to iterate over using a normal map type flow.  As I was going over that list, I was modifying some json.  What I ended up having was a nice functional map activity that had to pass in some json, and then return some json, for the next guy to transform.  It meant I had to create a var, and I found myself being annoyed greatly by this, I really wanted a val not a var.

Here is the before:

	def transformField(value: JsValue, fieldName: String, displayName: String): JsValue = {
	value.transform(
	(__ \ fieldName).json.update(
	__.read[JsString].map { s => Json.obj("DisplayName" -> displayName, "Value" -> s)}
	)
	)
	match {
	case s: JsSuccess[JsObject] => s.get
	case e: JsError => {
	Logger.error("Errors: " + JsError.toFlatJson(e).toString())
	JsNull
	}
	}
	}
	def transform(envId: String, value: JsValue, exception: SomeObject, overridableFieldsMap : HashMap[String, String]): JsValue = {
	var newValue = value
	typeOf[SomeObject].members.filter(!_.isPrivate)
	.filter(_.isMethod)
	.filter(_.asMethod.isAccessor)
	.filter(_.asMethod.fullName.startsWith("services."))
	.foreach(
	methodScope => {
	var term = methodScope.asTerm
	var mirror = runtimeMirror(exception.getClass.getClassLoader)
	val instanceMirror = mirror.reflect(exception)
	val fieldMirror = instanceMirror.reflectField(term)
	val fieldName = term.name.toString
	val displayName = overridableFieldsMap.getOrElse(fieldName, fieldName)
	newValue = transformField(newValue, fieldName, displayName)
	}
	)
	newValue
	}

view raw

gistfile1.txt

hosted with ❤ by GitHub

So you can see what is wrong here, looking at transform().  I’m taking an object, modifying it, and then sending it back, mutating stuff as I go.  I really wanted to pass the output of one function as input to the next function.  I knew that is what I wanted, but I wasn’t sure how to do it, and I didn’t know the right words to google.  In my mind I thought it should wind up being some like f(f(f(value))), which felt really recursive, only I’m going thru this list as I do stuff, so not really.

Then I found this blog post: http://danielwestheide.com/blog/2013/01/23/the-neophytes-guide-to-scala-part-10-staying-dry-with-higher-order-functions.html

In the middle he talks about “Composing a transformation pipeline”.  That is exactly what I needed, I wanted to pass the output from one function as the input to the next function.

So here’s how it ended up after applying that idea:

	def transform[T: TypeTag: ClassTag](json: JsValue, bean: T, overridableFieldsMap : HashMap[String, String]): JsValue = {
	val functionCalls: Seq[(JsValue) => JsValue] = typeOf[T].members.filter(!_.isPrivate)
	.filter(_.isMethod)
	.filter(_.asMethod.isAccessor)
	.filter(_.asMethod.fullName.startsWith("services."))
	.map(
	methodScope => {
	var term = methodScope.asTerm
	var mirror = runtimeMirror(bean.getClass.getClassLoader)
	val instanceMirror = mirror.reflect(bean)
	val fieldMirror = instanceMirror.reflectField(term)
	val fieldName = term.name.toString
	val displayName = overridableFieldsMap.getOrElse(fieldName, fieldName)
	(x:JsValue) => {
	transformField(x, fieldName, displayName)
	}
	}
	).toSeq
	scala.Function.chain(functionCalls)(json)
	}

view raw

gistfile1.txt

hosted with ❤ by GitHub

Ok so don’t worry about all the reflection the key bits are how this:

newValue = transformField(newValue, fieldName, displayName)

view raw

gistfile1.txt

hosted with ❤ by GitHub

Becomes this:

(x:JsValue) => {transformField(x, fieldName, displayName)}

view raw

gistfile1.txt

hosted with ❤ by GitHub

Instead of taking in the JsValue object and then returning it back for the next guy to mutate, I return a function which accepts a JsValue and returns one.  So when this map activity finishes I end up with a Seq of functions.  The built in Scala Function.chain() function takes a Seqs of functions and applies the output of one to the input to the next.  From the Scala docs that business looks like this.

def chain[a](fs: Seq[(a) ⇒ a]): (a) ⇒ a

Given a sequence of functions f1, …, fn, return the function f1 andThen ... andThen fn.

So read about partial functions and composition.  I’m finding that learning the functional vernacular really helps you google stuff to find what you are looking for.  As a professional day to day programmer, what I need is to make these tools part of my everyday toolbox.  They aren’t just esoteric academic things, they are tools I can use to solve real problems, which is why I’m blogging about this, so others might see how I used this capability to solve a problem.

 

Thoughts on working in higher education for a decade and a half

This is from a email I really wanted to send to a client but felt might be taking it a little too far:

 

As a final note that isn’t really here nor there, but it completely underscores a problem that I’ve recognized in this industry for years.  IT departments, for whatever reason, are really driven by faculty concerns, which IMHO, are often misinformed.  The issue is there seems to be this force that is often overly conservative and apprehensive about change.  There is this constant push back against moving any new innovations out the door.  Over time, years, eventually, students/faculty and others start complaining about the pace of innovation, and why hasn’t their system moved forward at a faster pace.  I find it very ironic, because IMHO is appears to be the same people who created the snail pace in the first place.  What’s even more ironic, is that the originally risk adverse culture ultimately, leads to a far more disruptive and risker process of switching systems.  I don’t know if its a problem of leadership at various places within higher ed that created this problem, or that faculty just have too much power/influence, or maybe they are just old fogeys and talk about how they remember when bread was 25 cents a loaf all the time, and time will eventually fix this issue.

 

It seems in the case that this same force is once again at play stalling a X.X.X upgrade that could happen sooner rather than later, and I think this unfortunate.  What I really don’t understand is how real cloud providers like Canvas are breaking through this dynamic?  Because there you don’t have any control over when and where the upgrades happen irrespective of whether there are cosmetic changes or not.  How can they being doing so well, when the experience in our world is that most organizations behave pretty much like yours?  It’s very confusing to me.

 

Do faculty not use google, facebook, mobile apps, or any other SaaS cloud based services that can change under their feat at any moment, how do these faculty handle that?  Do the see a new button or user experience in gmail, faint, have a heart attack, go on sabbatical for a week to recover?  How does that work?

 

At the end of the day, I’ll assume this is all about managing client expectations, how you establish the engagement will work from the start.  If you follow through with what you say and consistently re-message the engagement message, the path of trust becomes wider and things move along smoother.  I still think there are unique political structures and people problems in higher education that make change difficult.  There is an awful lot of consensus decision making and group think that ultimately slows things down and leads to poor overall outcomes.

implicit voodoo

For awhile in Scala, I was really fighting the idea of giving up IoC/dependency injection.  When Spring came around I was in J2EE hell.  Dealing with EJB deployment descriptors and porting things from one bad app server to the next, realizing that write once/run anywhere was a total lie when it came to the J2EE spec.  Spring was just what the doctor ordered, say good by to complicated deployment descriptors, JNDI, all that nonsense.  Of course over time Spring turn into XML hell, which was the replaced by annotations.  I’ve used some other IoC frameworks as well, and sorta got used to doing things that way.

The fact Scala has built in singletons via Object, pretty quickly convinced me I didn’t need IoC anymore.  It was sorta like duh, yeah, that makes total sense we should just have stuff like that built in.  The next really common Java’ism I had to rethink was context.  Java loves context.  You have the servlet context, thread local context, context for this, context for that, there is a context is just about every api you encounter.  You need these little things to hold onto state while you do something.  Basically, you want to push some little bit of data somewhere and pull it out way down in your stack, and if you had to pass that around to every method call you make you’d have to touch like 40,000 api’s 1/2 of which you can’t control, and so it goes.  But we’ve all encountered how fragile these little guys are.  They have some magic name that gets stuck in a constant someone, which is one point of failure when the name changes.  Then if on the consumer side, if that guy didn’t get put there, you are just screwed, and you just sorta hope whoever or whatever was supposed to put in there did so.

Implicit Parameters

One way to avoid all this context nonsense in Scala is with implicit parameters.  An implicit parameter is an optional parameter you can define on a method.  You can either implicitly call a function with a value for it, or if you don’t provide the value, the compiler will look for a value in the scope somewhere.

You mark a parameter as being implicit in your function definition like so:

	def isModerator()(implicit user: User, request: RequestHeader): Boolean = {
	..
	}

view raw

gistfile1.txt

hosted with ❤ by GitHub

So now you can call this in the normal way, like this:

	var user : User = new User
	var request : RequestHeader = new RequestHeader
	isModerator(user, request)

view raw

gistfile1.txt

hosted with ❤ by GitHub

or if these parameters can be set as implicit somewhere in the scope, and they magically get applied, like this:

	implicit val user = currentUser
	implicit val request = currentRequest
	isModerator()

view raw

gistfile1.txt

hosted with ❤ by GitHub

In a web app, this sort of stuff is great for the user, request, role, customer, or whatever normal state things you have that you need often, but don’t necessarily want to have to pass all around.

Implicit Methods

Ok so now the real voodoo.  You can in fact magically insert functions into existing classes.  Ruby has something like this called monkey patching, C# has something called extensions, the other dynamic languages have something similar as well.  The problem with the dynamic languages is that one framework could monkey patch something in with one name and another framework uses the same name, and then you have a whole problem.  In our happy statically typed world of Scala, the compiler will catch these problems for you and bitch.

Ok so how does this work.  I’ll give a little example where I tried it out.  In a typical web app, you have some domain classes modeling your database layer. Then you start through those into your view and you realize there are some things you need that might be based on the domain but need some dynamic generation or whatever.  In my example, I have a meeting for a web conferencing tool.  Most of the meeting info is in the database, but then I have to dynamically create a join url in the view, because there are user parameters, and checksums, and what have you. In Java, you might have to create another class that decorates your domain object.  Or you might build up some maps and pass them down.  But what if I could just add a joinUrl() method to my meeting object, so when I’m iterating over a list in the view I can just spit that out, without any more bother.

Here’s how that might work:

	class JoinUrlMethod(meeting:Meeting) {
	def joinUrl()(implicit user:User) : String = {
	MeetingManager.joinMeeting(meeting, user)
	}
	}
	object Conversions {
	implicit def meetingWithJoinUrl(meeting:Meeting) = new JoinUrlMethod(meeting)
	}

view raw

gistfile1.txt

hosted with ❤ by GitHub

The first thing you do is define your class that models the method you want to add.  The create an Object which defines the implicit method, giving it a name and tying it to your class that has the implementation. Then where you want to use this, you simply bring in the Conversions object in the import and it will magically be available in a Play template, or any other Scala code where you might want to use it:

	@import model.Conversions._
	…
	@for(meeting <- meetings) {
	…
	@if(meeting.isJoinable) {
	<a class="btn btn-primary" href='@meeting.joinUrl'> @Messages("tool.view.join")</a>
	}
	}
	…

view raw

gistfile1.txt

hosted with ❤ by GitHub

You can read about how monkey patching causes problems in dynamic languages.  Readability, collisions, etc, etc.  You can see in Scala, because things are strongly typed, and you have to actually import the implicit method conversion to have it work, that the complier helps you avoid some of those pitfalls.  Its still probably a valid argument that these types of tools should be used sparingly, but you can see how powerful they are and how they can make other code much more readable and clean.

Slicker than most

Let’s start with a song from my college days

So as my Scala journey progressed it was time to pick a persistent engine.  I looked at Anorm and decided on Slick.  Mostly because its supported by TypeSafe which I would assume with their bankroll and influence is a safe option.  Technically, I like that I can still fall back on raw sql if I need to, because my experience with every Java ORM is that you need that option at one point or another.

So the first thing to do was get my model going.  I quickly discovered something called case classes.

case class Meeting (id: Option[Long], meetingEid: String, name: String,   
                    var host: Option[String], var hostSalt:Option[String],  
                    welcomeMessage:Option[String],dialNumber:Option[String],  
                    voiceBridge:Option[String],startTime:DateTime,duration:Int)  

So that is sorta like an immutable POJO in the Java world, but in one line!  It comes with hashcode(), equals(), toString(), all built in. Already I was digging things, that would have probably need like 50 lines of code in Java, albeit my IDE would have vomited most of it out for me, but still, its very concise, me likes.

Based on patterns I found googling around, the next business was to create the Meetings class.  This guy is sorta like what you might do with an hbm file or annotations to describe how columns map to your model.

class Meetings extends Table[Meeting]("MEETING") {  
   def id = column[Long]("ID", O.PrimaryKey, O.AutoInc)  
   def meetingEid = column[String]("MEETING_EID", O.NotNull)  
   def name = column[String]("NAME", O.NotNull)  
   def host = column[String]("HOST", O.Nullable)  
   def hostSalt = column[String]("HOST_SALT", O.Nullable)  
   def welcomeMessage = column[String]("WELCOME_MSG", O.Nullable)  
   def dialNumber = column[String]("DIAL_NUMBER", O.Nullable)  
   def voiceBridge = column[String]("VOICE_BRIDGE", O.Nullable)  
   def startTime = column[DateTime]("START_TIME", O.NotNull)  
   def duration = column[Int]("DURATION", O.NotNull)  
   def uniqueName = index("IDX_MEETING_EID", meetingEid, unique = true)  
   def newMeeting = meetingEid ~ name  
   def * = id.? ~ meetingEid ~ name ~ host.? ~ hostSalt.? ~ welcomeMessage.? ~ 
      dialNumber.? ~ voiceBridge.? ~ startTime ~ duration <> 
      (Meeting.apply _, Meeting.unapply _)  
}  

So most of that is probably pretty obvious until you get to

def * = id.? ~ meetingEid ~ name ~ host.? ~ hostSalt.? ~ welcomeMessage.? ~ 
           dialNumber.? ~ voiceBridge.? ~ startTime ~ duration <> 
           (Meeting.apply _, Meeting.unapply _)  

What the hell is that crap ?  Yeah, I didn’t know either.  What does <> mean ?  What about def * ? Eventually (like hours later)  I came across this, http://stackoverflow.com/questions/13906684/scala-slick-method-i-can-not-understand-so-far which explains about Mapped Projections and Comprehensions.  If you want to understand it read that post, does a much better job than I can, that guy should write a book.  Basically it comes down to, that is how you map a row into your backing object and vice versa.  Its amazing concise and powerful.  Think about all the really verbose ways you’ve seen that done before, and it will really stop you in your tracks.  On the flip side try googling for “<>” or “_” or “def *”.  Once you understand it, its awesome and super easy, but trying to find the information to explain it, that was sorta hard for me.  I thought I was a pretty good goolge user, sheesh

Ok so now we have our DAO which might look something like this:

object MeetingDAO {  
   val table = new Meetings  
   def createMeeting(meeting: Meeting): Long = DB.withSession { 
       implicit session: Session =>  
    table.insert(meeting)  
   }  
   def listMeetings: List[Meeting] = DB.withSession { 
       implicit session: Session =>  
       Query(table).list  
   }  
   def deleteMeeting(meetingId: Long): Option[Meeting] = DB.withSession {  
        implicit session: Session =>  
     val query = for {  
       meeting <- table if meeting.id === meetingId  
     } yield meeting  
     val retVal = query.firstOption  
     query.delete  
     retVal  
   }  
}  

That’s all pretty straight forward, other than the implicit parameters and Option junk right ?  Ok, so Option I first found annoying, but now I think its awesome. Option basically eliminates NPE’s and null checks from your code.  Its a way to type that something can have a value or not.  To access the real value you do someOptionVar.get, and you can test if it has a value with someOptionVar.isEmpty.  All the extra .get calls annoyed me at first, but then when I saw how all the null checks disappeared, but my code was still safe, I had a different opinion.

What else is going on there?  Oh, query.firstOption.  So in Scala a List is called a Seq.  query here is a Seq.   To get the first element as an Option, you can call firstOption. Then back in my calling code I can make the “null” check using the Option.

1:   val entity = MeetingDAO.deleteMeeting(meetingId.toLong)  
2:     if (entity.isEmpty) {  
3:      Home.flashing("warning" -> s"Meeting '${meetingId}' was not found.")  
4:     } else {  
5:      Home.flashing("success" -> s"Meeting '${entity.get.name}' has been deleted.")  
6:     }</span>  

I’ve just begun to touch the service with Slick, I haven’t tried doing complicated queries or even relationships yet, so I suspect will have some more posts about that, when I get to it.  I’ll save implicit parameters and implicit methods for another time as well.

First Project With Scala

So I started my first project with Scala about six months ago.  I had hired a new guy to build some automated testing out for our main application, and I figured that was a fairly low risk type of project to kick the tires out on.  I knew I was going to be using Web Driver to do most of the work, so it seemed largely irrelevant to me what language was actually pushing the commands out, they are probably all roughly the same.  Also, the resource I hired was still in school, so this would be a chance to see how quickly someone with limited coding experience, could pick up Scala and use it for something.  This project wasn’t going to have to have any fancy design or be overly complicated, so we could ease into it without getting too sophisticated in the functional world yet. 

The project is here: https://github.com/johntbush/sakai-test-suite.  I wanted to use cucumber because our application is pretty large and complex and I had another tester guy who already knew the functional aspects of the application, so I figured he could write cucumber tests and feed them to my automation Scala newbie to implement.  We stuck with maven for building, junit, and other tools we were familiar with.  This was a classic ease into Scala type of project.

Within a few days I had the basic skeleton working. I actually struggled more with getting cucumber to work with scala and junit, more than anything else.  Really the stupid phantomjs driver (nice idea, not a fan) and getting everything pulled into jenkins was much harder than anything to do with the Scala language.  Just basic Selenium funkiness, especially with the iframes in our app too some time to figure out, the dsl in Scala was really nice to work with.  This lead me to my first conclusion, the frameworks matter. You can have all the nails in the world, but without a hammer to drive them in, you are screwed.  The nails being power of Scala, hammer the frameworks… ok you get the idea.  You weren’t expecting a fucking poet here were you ?

Singletons Are Built In

So the first cool thing I discovered was that Scala has built in singletons.  If you come from Java world, that was like a big deal to me.  Because its either dependency injection (I really didn’t need to be that fancy in this app), or having to create a whole class with single instance of itself with static method and all that.  Yeah, its not hard, and yeah I’ve done it a million times, but it sure was nice to just go:

object Config extends Config  
class Config {  
   val systemProperties = System.getProperties()  
   val targetServer: String = loadProperty ("target.server","https://nightly.cle.rsmart.com/portal")  
   val sakaiVersion : String = loadProperty("sakai.version", "2.9.1")  
   val sakaiDistro : String = loadProperty("sakai.distro", "ani")  
   val defaultAdminEid : String = loadProperty("sakai.admin.eid", "admin")  
   val defaultAdminPassword : String = loadProperty("sakai.admin.pwd", "admin")  
   def defaultCourseSiteId = "course-test-1"  
   def defaultCourseSiteTitle = "Course Site Test 1"  
   def defaultInstructorEid = "instructor1"  
   def defaultStudentEid = "student01"  
   def defaultInstructorPassword = "password"  
   def defaultStudentPassword = "password"  
   def loadProperty (name : String, defaultValue : String) = {  
    if (!StringUtils.isEmpty(systemProperties.getProperty(name))) {  
     systemProperties.getProperty(name)  
    } else {  
     defaultValue  
    }  
   }  
 }  

Now I had a nice Config object, defaulted by a properties file, overridable by system properties.  I could call from anywhere else and not worry about injection or any of that business.  Nothing fancy, quick and dirty, does the job:

    Portal.navigateToPage(Config.targetServer)

You can pass functions around?!

So this is something every language but Java seems to have, but when you’ve been living without it for a long time… its like a conjugal visit.  I had one piece of code that was basically the same as this other piece of code except for a block in the middle.  So I move the two blocks into functions, and then invoked the surrounding method by passing that in and applying it at the right spot.  Its too much code to share here, if you can see:  https://github.com/johntbush/sakai-test-suite/blob/master/src/test/scala/com/anisakai/test/pageobjects/SiteManageTool.scala, look at the createProjectSite() and createCourseSite().  I think there is a better way to do that now that I know more, but whatever this is part of my journey.

More bad language, unfiltered opinions, and voodoo coming

Ok, all for now, next post I’ll talk about my new project with Play! and emberjs, where we learn about implicit parameters, and implicit methods and all kinds of awesome voodoo that makes for some really concise and powerful code.  Stay tuned more coming….