If you have ever used elasticsearch and you haven’t discovered the hot_threads endpoint, take a look at it. It basically figures out which threads on each node are consuming the most cpu and then gives you a stack trace. Its like a rest endpoint for thread dump around the cluster. It looks something like this:
In a previous project we were running elasticsearch embedded in our monolith, so this was pretty handy, because it gave you a view into not only what ES was doing but the whole server.
I had a issue the other day where my monitoring was not helping me out, and I needed a stack dump, but I was having some trouble getting one. So I decided I don’t want to be in this position again standing there holding my dick, so I thought I wonder if I could just lift this thing out of elasticsearch, or at least borrow some code.
Well it turns out it was easier than I thought. I poked around in github found the HotThreads class. Since I already had the elasticsearch api included in my project, all I had to do in Play was just wire up a quick controller like this, and then map in the routes file.
That’s it, then you have a nice admin endpoint to see running threads. It’s not across the cluster its per node, that’s a job for another time, but still pretty cool!