Maven Release Plugin Horrors with git

If you have spent any time trying to use the Maven release plugin, you probably hate it.  It wants to run your build twice, typically you have to wait 20 minutes to hours depending on the size of your project to figure out that it barfed and then try something else, barf again.  At some point you either dig in really hard, or start looking for other solutions.

As is just about typical with every major block I have, just when I’m about to kill myself, my dog, or anyone that comes near me, I figure things out.

So my story is this, we have the Maven release plugin working great locally, but doing releases from laptops is not ideal, and of course on the weekend I needed to do a release and the VPN was painful to run the test suite over.  I decided it was about time to get this thing working in jenkins, how hard could it be, throw it up there, wire in the maven goals, bada bing right ?

Not so much.

Ran the release goals, everything is awesome, then it goes to push the thing back into git:

I do some googling, find out that jenkins checks out git code into a detached head, so you need to deal with that.  After a few failed attempts doing things that seemed much more complicated then they needed to be I landed on this:

Basically you need to just pick the “check out to specific local branch” option and set your branch.  Grab a beer, wait another 25 minutes and then bamn!:

Well this one got got me stuck forever.  Convinced it was some credential problem in jenkins trying all sorts of things messing around with the git plugin, credentials, you name it nothing helping.  Started pondering on “this is why people hate Java”…

Then finally I was sitting there staring at that url,

could not read Username for ''

Thinking about where have I seen that before somewhere else.  Oh yeah, its in the maven pom.xml:

I changed the git url to this:

And hell yeah it finally worked.  It makes sense if you think about how git is handling security that the http protocol isn’t going to work here since since we are detached from the original git pull of the code.  Anyway, it was much pain and suffering and I couldn’t believe stackoverflow and google let me down, it took way too much time to figure this out.  That is why I’m blogging about it, so some other poor sorry soul won’t have to kill their dog over it.

As a side note, I ran into this while looking for a solution:

This guys approach seems pretty interesting.  It eliminates the two builds and seems really simple and straightforward to implement.  I was just about to try it out, when I got the past this.  But for sure when I find some time, or next time this poops out, I might be giving that a go.

For the record, I have other builds using the SBT release plugin, and I went really deep on that too, I have battle scars from that one too, a tail for another day…


What is old is new ?

the_wheelHaving dabbled deeply in the Javascript world for a few weeks, and looking at some trends, it seems like either a whole lot of Java devs made their way into JS world and are trying some of the same patterns again, or certain patterns have a way of self emerging.  For example, React as a component framework is reminiscent of JSF only client side, and we hope sucks less.  But the basic idea and promise, sounds pretty familiar.  Or for example, things like immutable-js looks like some ideas where based on the Java Collection API.  Then we have FOAM and TypeScript bringing strong typing back.  FOAM is interesting getting all meta on stuff.  We’ve seen that pattern before except we used to code gen from the tables, or use some other home grown declarative approach.

This post isn’t a dig on the JavaScript landscape not at all, the quality of and velocity of innovation is pretty remarkable, especially for a core language that really doesn’t do that much by itself, and was invented in 10 days. I’ll write another post on my reflections on this subject, now back to the old and the new…

I remember when I was a young lad pursing my second undergraduate degree (don’t presume I finished the first, because I didn’t), I had to write a paper to get accepted into the computer science program.  I remember reflecting on how math doesn’t change, and even though you are teaching me antiquated methods, I recognize that the patterns are transferable.  Apparently, me telling them that their tech was not up to date didn’t piss them off enough to exclude me, or maybe I was spot on with that analysis.

Now after about 18 years in the industry, I think that reflection was spot on.  I’ve been watching client/server become request/response, and now single page apps look a lot more client/server.  It was do all rendering on the server and now its do all rendering on the client.  On the backend, we had don’t do any processing in the data store (kill those sprocs!), but then map/reduce comes around and says, wait do the processing close the the data.  Sorry man, we were wrong, that was a good idea after all, we just need to tweak it a bit.  Doesn’t docker remind you of the JVM conceptually, it’s something like a virtual machine that sits on another machine, but doesn’t require the whole OS, in other words I can run several dockers on one box, just like I can run several JVM processes, albeit JVM will run out of memory much sooner!.  Tech trends come and go, and a lot of them sound the same as before, sometimes they improve on the past, sometimes they make the same mistakes as the past.  In general, it always back to square one:

  • Pick the right tool for the job.
  • Don’t add unnecessary layers of abstraction where they aren’t necessary, or aren’t bringing real value.
  • Don’t trust that whatever X framework is promising without some tangible demonstration.  The academic promise, if its nothing more than that, can not be trusted, and even when it works, has limits.
  • Whatever fail safe pure layer you create, some asshole will find a way to leak it, trust me, I’m often that asshole.
  • Beware of easy to start hard to finish.  The industry is all about time to market, time to market, bring the value.  But remember 80% of the time and effort is spent maintaining and supporting the software.  Maintenance and sustainability are crucial.  Regression test are great, but the compiler gives you stuff for free.

So next time you want to poke fun at the COBOL, ok now its the legacy Java, because its 10 or 20 years old.  Think to yourself, will that transpiler that was just invented yesterday be around in 2 years even?  Software that keeps adding value over time is hard.  If you work somewhere that has some 5, 10, 20 year old code, instead of cursing at that shit, maybe stop and realize that its amazing that stinky crap still works, and its still providing value (and maybe your paycheck!), do you think your shitty ass code will be around that long?

I think the microservice trend is fine and I can see that value.  I like that it forces the division between layers, and decomposes problems, its largely about risk management, and quick deployments.  But on the other hand it’s also a cop out to believe that software can’t have any longevity.  Maybe its a dying art, maybe I’m just an old fogey, maybe the crap created today really won’t be around in 10 years, none of it and I should just accept that.  But seeing as how the patterns keep coming back and fading away just like fashion, I’m thinking a sturdy pairs of jeans will do just fine.  They might take awhile to put on, but once they are on, I can do anything. And if I can have quick time to market and maintainability, I’m picking that one… Scalalala

Logstash and Playframework

I’m not sure why google let me down in regards to hooking up logstash and Play, but it sent me on some pretty weird paths. So I’m going to share what we did to get it working that is pretty simple in the end.

Play uses logback, first rule is don’t try to include a new version of logback in your build, that will cause you conflicts, the ootb Play dependencies are all you need, at the time I did this, we were using Play 2.3.8.

In your logback config, logger.xml, just wire up the appender you want, the tcp or udp one like this:

Then on the logstash config side create a new input:

This puts your data into a new type (called “logstash”) in the ES indices so that the json mappings don’t conflict with anything else. That’s it.


One of the things I love about functional programming, and Scala in particular is that it feels like you can make new language constructs.  One common pattern I run into is having to do something a few times until hopefully it works.  This is especially relevant when you are calling some remote endpoint that might be busy, slow, or down or whatever.

I found this little gem on stackoverflow:

What’s going on here?  We take an Int and a function.  Then try to call the function.  If it successful then return its value, else recursively call ourselves again until we run out of tries.  If we haven’t succeeded in a certain number of tries then throw the exception the function gave us.

If you don’t fully understand what the @annotation.tailrec does, do yourself a favor and read this article.  It does a great job of explaining it.  But in a nutshell, the Scala compiler can optimize a tail recursive function into a loop, which means no more stack overflows.  Surprisingly, this feature is rare in a lot of modern day languages.  I think in Ruby you can compile it in, Python is philosophically opposed to it.  The more purely functional languages like Clojure, Erlang, and Haskell of course would do better.  Sorry Javascript, you have to trampoline too, how sad for you…  But I’m sure there are a bunch of transpilers that can fix that up, I wonder what Scala.js does there?

Ok so enough of that, the real story is in how easy it in to use this guy.

It’s just that easy you can retry how ever many times you like else throw your exception…

Hot Threads

If you have ever used elasticsearch and you haven’t discovered the hot_threads endpoint, take a look at it.  It basically figures out which threads on each node are consuming the most cpu and then gives you a stack trace.  Its like a rest endpoint for thread dump around the cluster. It looks something like this:

Screen Shot 2015-03-03 at 11.32.00 PM

In a previous project we were running elasticsearch embedded in our monolith, so this was pretty handy, because it gave you a view into not only what ES was doing but the whole server.

I had a issue the other day where my monitoring was not helping me out, and I needed a stack dump, but I was having some trouble getting one.  So I decided I don’t want to be in this position again standing there holding my dick, so I thought I wonder if I could just lift this thing out of elasticsearch, or at least borrow some code.

Well it turns out it was easier than I thought.   I poked around in github found the HotThreads class.  Since I already had the elasticsearch api included in my project, all I had to do in Play was just wire up a quick controller like this, and then map in the routes file.

That’s it, then you have a nice admin endpoint to see running threads.  It’s not across the cluster its per node, that’s a job for another time, but still pretty cool!

Screen Shot 2015-03-03 at 11.07.12 PM

Either you’re with us or against us

So I was happily going on with my Scala journey avoid exceptions everywhere.  Then I ran into this use case where I sorta needed them.  But I really liked not having try/catch all over the place mucking up stuff.  So here’s this use case:

I’m in the middle tier, people call me via Rest, then I call sql, sprocs and more sprocs, Rest, your mom’s old couch, whatever and make that rationale and pass it back up.  The thing is that Rest service I’m calling might poop out on me, and I’d like to just pass up the json it sends me to the guy above me.  So I have this situation where I need to return two things.

I know exceptions can work here, I can return my object in the good case, or an exception in the bad case, but I really hate checked exceptions, not to get into a whole theological discussion about it, but I’m definitely in the camp of exceptions are best for exceptional stuff.  That whole FinderException crap EJB started, pisses me off.  Sun really went ape shit with that stuff.  I digress..

So how can I return two different things.  Well I could create some wrapper object thingy.  Yeah that would work, but its so icky, and then, I’m going to have to down cast stuff, or have a bunch of icky wrappers.  I don’t think that will work.  Then I discovered Either.

A common use of Either is as an alternative to Option for dealing with possible missing values. In this usage, scala.None is replaced with a Left which can contain useful information.Right takes the place of Some. Convention dictates that Left is used for failure and Right is used for success.

This is the medicine I was looking for.  I can return this, or I can return that.  OMG, genius!!!  What is even better, is that the convention is that the good stuff is on the right, and the bad stuff on the left.  Now Scala is even jelling with my political sensibilities.  Wow, mind blow.

So I can write stuff like this which is pretty fun:


So back to how I use this.  When I call into my store like this:

I send back up an EIther.  Which has a list in this case and then the raw DataStoreResponse on the left.  So if something bad happens, you get the raw DataStoreResponse, else the object(s) you were looking for. So in my case where I’m a service sending json back up in the good or bad case, things end up like this:

That’s so much nicer that try/catch all over the place.  But the thing is, this is stable, resilient code, it handles failure well.  But it doesn’t hurt your eyes to look at, is very concise, and reads well.

Chain Gang

So as my Scala journey has progressed.  It has really stepped up since I’ve switched jobs and been able to focus on it for a few weeks without distraction.  I’ve found that I’ve crossed into the functional style.  If statements annoy me now, and anything mutable causes me concern.  I was doing really well, getting the hang of things and then I ran into an issue.

Here was my use case.  I’ve got a deep domain, its heavily customized per customer in many ways.  The way I’m dealing with this right now, is that the actual field names of wide domain objects have customized labels per client.  We are talking 100 columns or more that can be customized.  I knew I needed to get all meta data with it, and use reflection, no problem, been there before.  But what I wanted to do, was read data from the store, then apply the label just in time, I don’t want to do a bunch of “select * as name from”  …, but rather get data, and then transform it.  Partially, because that would perform badly, partially because I’m going to have to transform the same data again and again, I don’t want that logic all wrapped up in the data access layer, and frankly its not available there, well yet anyway.

So that’s my use case, here’s my technical problem.  I had this list I needed to iterate over using a normal map type flow.  As I was going over that list, I was modifying some json.  What I ended up having was a nice functional map activity that had to pass in some json, and then return some json, for the next guy to transform.  It meant I had to create a var, and I found myself being annoyed greatly by this, I really wanted a val not a var.

Here is the before:

So you can see what is wrong here, looking at transform().  I’m taking an object, modifying it, and then sending it back, mutating stuff as I go.  I really wanted to pass the output of one function as input to the next function.  I knew that is what I wanted, but I wasn’t sure how to do it, and I didn’t know the right words to google.  In my mind I thought it should wind up being some like f(f(f(value))), which felt really recursive, only I’m going thru this list as I do stuff, so not really.

Then I found this blog post:

In the middle he talks about “Composing a transformation pipeline”.  That is exactly what I needed, I wanted to pass the output from one function as the input to the next function.

So here’s how it ended up after applying that idea:

Ok so don’t worry about all the reflection the key bits are how this:

Becomes this:

Instead of taking in the JsValue object and then returning it back for the next guy to mutate, I return a function which accepts a JsValue and returns one.  So when this map activity finishes I end up with a Seq of functions.  The built in Scala Function.chain() function takes a Seqs of functions and applies the output of one to the input to the next.  From the Scala docs that business looks like this.

def chain[a](fs: Seq[(a) ⇒ a]): (a) ⇒ a

Given a sequence of functions f1, …, fn, return the function f1 andThen ... andThen fn.

So read about partial functions and composition.  I’m finding that learning the functional vernacular really helps you google stuff to find what you are looking for.  As a professional day to day programmer, what I need is to make these tools part of my everyday toolbox.  They aren’t just esoteric academic things, they are tools I can use to solve real problems, which is why I’m blogging about this, so others might see how I used this capability to solve a problem.