Here’s how I made my own slow query log with a Kibana dashboard. See ya Appdynamics and Solarwinds.
First of all if you are using SQL in Scala take a look at Scalikejdbc. I’ve been around the block with various ORM solutions and I always come back to wanting something closer to SQL, its easier to tune, and easier to communicate with DBA’s and other parts of your organization. SQL is the interface to relational database’s, IMHO, arbitrary DSL’s invented simply for syntax sugar, or usually easy to start hard to finish. Scalikejdbc is a great blend of SQL with efficiency tools that make binding and result set manipulations easy and idiomatic.
So the first part of the puzzle is leveraging the built in mechanism that lets you turn on sql logging. See the docs for more help.
Now you can create a function to be called when your sql query completes using GlobalSettings.taggedQueryCompletionListener. You’ll notice I’m using the mapped diagnostic context, you can read more about that here. I’ll explain how I get things into that in a bit, but it’s basically other contextual info that I want to log, like which database, driver, etc.
Ok next up. This is how I shoved the contextual info into the tags so that I could pull it out in the log. Most of my queries look something like this:
I have a number of different databases I connect to, I use NamedDB to look them up and get a session. I noticed that the DBSession object in Scalikejdbc has tags, you can also put them on the queries, but I didn’t want to have to do that to all my existing code, and I’m pretty sure I’d forget to do it on the exact query that was causing all my pain, needed a better way. So I created a new NamedDB case class that would do it for me:
I still had to update my existing code, but simple search and replace of NamedDB to TaggedDB, done!
I had to do some hacking since Scala doesn’t like you extending case classes, and it’s not fully implemented, but you get the idea: you can put tags into the session. My dbName is like “server/databaseName” so I split that out and then grab some other stuff from the connection that I’m interested in logging. You could do more here if you need it.
Ok so now thats done. Let’s talk about how I got my logs to go into a special index in Elasticsearch. I decided to bypass logstash once I found this logback Elasticsearch appender. Basically you can just wire that into your logback config, and you are done. It seems performant, it makes calls in a future, etc, etc.
The trick to the configuration is that logger at the top of this blog, that’s the one you want to connect up. In this way all your existing logs go where they go, but send this slow query log stuff to Elasticsearch.
Every property that you put in as a “tag” you need to wire up in this logback config to get it to output into the ES json. This seems a bit verbose and maybe now I’m seeing why the JVM has 40k loggers, people just simple get these APIs wrong, but in any event there is a way.
That’s it. Now you’ll end up the a new index in ES with your slow query data that includes the sql, the parameters, how long the query took, and all the contextual info you need to root cause your issues. Wire it up in a Kibana dashboard and make pretty pictures, hook it up to alerts, etc, etc. And it didn’t cost you a million dollars for some root detection tool.
I’d show you some data but then I’d have to kill you, so you are going to just use your imagination. If you’ve read the blog this far, I know you are a smart person and have a bit of an imagination.
Latest Update 1-14-2016
Update. I found the reliance on Logback problematic using this in another places that rely on a different logging implementations, like Apache Storm needs Log4j2. So rather than get in the business of supporting every single Java logging impl (why are there so many!), I decided to just take the writing to ES into my own hands.
The ES Java API is good for somethings, but in this case I didn’t want to bring in a bunch of dependencies. So I borrowed this guys thin http client to Elasticsearch for Scala, https://github.com/gphat/wabisabi/blob/master/src/main/scala/wabisabi/Client.scala, and then queued up stuff and flush in batches (in a Future of course). This gets hammered in some pretty intense workflows and works great (in the millions of logs per day). That assumes you’ve done the work to make your ES cluster perform well, that can be a real pain depending on what you are doing, story for another time…