I’m not sure why google let me down in regards to hooking up logstash and Play, but it sent me on some pretty weird paths. So I’m going to share what we did to get it working that is pretty simple in the end.
Play uses logback, first rule is don’t try to include a new version of logback in your build, that will cause you conflicts, the ootb Play dependencies are all you need, at the time I did this, we were using Play 2.3.8.
In your logback config, logger.xml, just wire up the appender you want, the tcp or udp one like this:
Then on the logstash config side create a new input:
This puts your data into a new type (called “logstash”) in the ES indices so that the json mappings don’t conflict with anything else. That’s it.
One of the things I love about functional programming, and Scala in particular is that it feels like you can make new language constructs. One common pattern I run into is having to do something a few times until hopefully it works. This is especially relevant when you are calling some remote endpoint that might be busy, slow, or down or whatever.
I found this little gem on stackoverflow:
What’s going on here? We take an Int and a function. Then try to call the function. If it successful then return its value, else recursively call ourselves again until we run out of tries. If we haven’t succeeded in a certain number of tries then throw the exception the function gave us.
Ok so enough of that, the real story is in how easy it in to use this guy.
It’s just that easy you can retry how ever many times you like else throw your exception…
If you have ever used elasticsearch and you haven’t discovered the hot_threads endpoint, take a look at it. It basically figures out which threads on each node are consuming the most cpu and then gives you a stack trace. Its like a rest endpoint for thread dump around the cluster. It looks something like this:
In a previous project we were running elasticsearch embedded in our monolith, so this was pretty handy, because it gave you a view into not only what ES was doing but the whole server.
I had a issue the other day where my monitoring was not helping me out, and I needed a stack dump, but I was having some trouble getting one. So I decided I don’t want to be in this position again standing there holding my dick, so I thought I wonder if I could just lift this thing out of elasticsearch, or at least borrow some code.
Well it turns out it was easier than I thought. I poked around in github found the HotThreads class. Since I already had the elasticsearch api included in my project, all I had to do in Play was just wire up a quick controller like this, and then map in the routes file.
That’s it, then you have a nice admin endpoint to see running threads. It’s not across the cluster its per node, that’s a job for another time, but still pretty cool!