Web Interface for OpenLava

Openlava is a GPL fork of LSF 4, and as such is a pretty fully featured job scheduler.  I use frequently LSF in my day job, and use Openlava as a substitute when I just need a familiar environment to test things out.  I felt Openlava lacked a good scripting interface, as (like LSF) it provides only a C API for job management, so I wrote some Python bindings for Openlava.

To test it, I wrote a web interface to Openlava that enables users to view detailed information on jobs, queues, users, and hosts, and for administrators to perform management activities such as opening and closing queues and hosts.  Users can submit new jobs too using a web form.

In addition to the web interface, there is a RESTful API, that can be used to request information programmatically.  To test the API, I wrote a client tools package that removes any complexity from dealing with the remote cluster, and a bunch of command line tools that mimic the behavior of the Openlava CLI enabling remote submission and management of  the remote cluster.

You can view a demo of the web interface, view the documentation, or download the server or client tools from Github.

Loguino Data Aquisition for Arduino

I’ve been interested in capturing data from the seven since the moment I could drive it. I’ve looked at commercial data loggers, and there are plenty of good options, I use the ETB Digidash on the Fury, and it’s very competent, not least in the analysis software.

For the seven though, I didn’t feel there was a good fit for a number of reasons:

  • not many data logging tools support reading data from the megasquirt ECU, which is critical to get any information about the engine at runtime.
  • Many data loggers are geared towards providing an in car display system, but I already have a working dashboard that I like.
  • Usually they are limited to a number of channels, each channel having a predefined function or purpose.
So I decided one day to make my own, my first step was the bifferboard, this is a 486pc on a chip, it was easy to program since it runs linux, but it wasn’t the right tool for the job.  I’d hear lots of good things about Arduino, an embedded platform, and once I got my head around the API, it was clear this was the perfect solution, and shortly thereafter loguino was born.
Loguino itself is a generic logging platform, it functions a little bit like log4perl/java/etc in that you have a logging class that accepts log messages, and logger outputs that take those messages and do something meaningful with them.  Most of the time this is simply write the message to a disk, or output it over the serial line, but it could be used to control shift lights or any other task too.
Data is generated by pollers, pollers are polled periodically and generate messages which are picked up by the loggers.  The first task was to write one for the megasquirt, this allows me to pull engine data directly into the logging tool. The next step was gps speed and positioning, and then support for accelerometers and gyroscopes.
I chose a very simple logging format, rather than use columns, I decided to use key value pairs, this means the tool can be completely agnostic about formatting, time, and everything else.  My view is that once data is downloaded of the loguino, it will go into a database and be queried.  Having seen google chart, and flot, I decided there was no real need to use excel or write complex software, instead it should be a case of adding and removing series, and scaling them.  This spawned metriflot, a data visualization package, and was a great way for me to learn some Ajax and django.
Having tested loguino over the summer, I’ll be doing a permanent install over the winter with wheel speed sensors, brake temperature, pressure, and all sorts of other sensors, and generally tidying up the packaging.  Next year I’ll be able to concentrate on the analysis where I want to have more intelligent data trending, for example:
  • Is the engine temperature hot because of the outside temperature, speed, or a fault?
  • Is the oil pump failing or was i just not driving hard enough?
There is a lot of potential here, as well as a good chance for me to learn more about electronics and data analysis, and hopefully it will benefit others too.
Loguino was released under the GPL license, and is hosted on google code.

Metriflot

After lamenting about the difficulty of getting specific usage and utilization reports out of the scheduling system, I decided to write a tool to help.  What I basically want is a tool that can collect metrics on one or more systems, graph any combination of metrics and not much else.  There are plenty of tools to chose from, and we use quite a few of them already.

There’s ganglia, which is great for capturing cluster data, and it is used extensively, however I need to deal in exact values, so the average queue time a year ago isn’t acceptable, i need real, hard data.  I could tweak RRD, but at the end of the day its also not that pretty, and prettiness counts, especially when dealing with stake holders.

There is RTM, which is based on cacti, its pretty good and it has soon good graphs out the box, but it doesn’t have the graphs I need, its not very pretty, and its also not very solid.  I’m never 100% confident in the data I get from RTM, and in all honesty, its just overkill for my simple graphing needs.

Lastly, there is platform analytics, but its really a windows tool, yes it does pretty things, and is very extensible, but I can probably write the tool I want faster than even installing analytics.

The end result is metriflot, a catonation of metric, and flot, the latter being the ajax graphing library i’m using.  It meets my immediate requirement which is to create some simple graphs of esoteric cluster usage statistics, and in the process learned some more Django, Ajax, and Flot.

I’ve uploaded the source to google code here.