Graph all the things

RRDtool is a great way of storing metric data, however the charts it produces look dated now, especially considering how good modern JS based visualization libraries such as D3 have become. I wanted to understand RRDtool better, so this gave me an opportunity to write a front end for RRDTool that uses NVD3 to display RRD data as charts.

The Live Example uses data from the Mosquitto MQTT Server as a source.
Example

Installation

RRDViewer is hosted on Git Hub, you can download the zip file, or you can clone/fork your own copy of the repository.

First setup a Django project, Once you have the code, install using the supplied setup.py. If you are using git, be sure to use the –recursive flag, or otherwise ensure all sub modules are downloaded before running setup.

$ sudo python setup.py install

Then add ‘rrdviewer’ to your INSTALLED_APPS list in settings.py:

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    ....
    'rrdviewer',
    ....
)

Then setup your GRAPH_DIR in settings.py to point to a directory containing a bunch of RRD files. RRDViewer will recursively search that directory for RRD files to include.

GRAPH_PATH='/var/lib/rrd_data'

Once the application is installed, run collectstatic to copy the static files to their correct location:

$ sudo python manage.py collectstatic

Finally, setup your urls.py to include the urls for rrdviewer at your preferred location:

urlpatterns = patterns('',
    url(r'^rrdviewer/', include('rrdviewer.urls')),
)

You can now restart the web browser and visit the RRDViewer page.

Graph all the things

MQTT2RRD is a daemon written in python that connects to an MQTT server, subscribes to selected topics, and stores values in RRD round-robin databases for later graphing.

The code is hosted on git hub, and is licensed under the GPL.

Download

Usage

usage: mqtt2rrd.py [-h] {stop,restart,start} ...
optional arguments:
  -h, --help            show this help message and exit
  --config_file CONFIG_FILE
                        The location of the config file
  --no_daemon           Do not spawn a daemon, stay in the foreground

Configuration

Configuration is via a configuration file, by default mqtt2rrd looks in the following locations:

/etc/mqtt2rrd.conf
~/.mqtt2rrd.conf

You can specify the location of the configuration on the command line using the –config_file argument. For example:

mqtt2rrd.py start --config_file=/home/bob/mqtt2rrdtest.conf

Configuration is stored using an ini file using the python ConfigParser format. The following configuration options are available:

daemon

pid_file
The PID file location, used to store the PID of the daemon process.
data_dir
Configures the root directory to store the RRD files in. RRDs are in sub directories corresponding to their topic name. For example, a message with topic “foo/bar/baz” would be stored as $DATA_DIR/foo/bar/baz.rrd. The default data directory is /var/lib/mqtt2rrd.
user
The user to run as, when a username and group is specified, and the script is executed as root, then the script will demote itself to the specified user and group
group
The group to run as, when a username and group is specified, and the script is executed as root, then the script will demote itself to the specified user and group

logging

Logging is done using the python logging class, at this time you can configure the log file, and the log level.

log_level
The level to log at, 10:debug and above, 20:info and above, 30: warn and above, 40: error and above, 50: Critical log messages only. The default is to log at level info and above.
log_file
The file to log to, by default this is /var/log/mqtt2rrd.log

Example Log Output

2014-06-20 18:14:02,531: INFO: Connected: test.mosquitto.org:1883

2014-06-20 18:14:02,584: INFO: Connected to server.
2014-06-20 18:14:02,584: INFO: Subscribing to topic: #
2014-06-20 18:14:02,643: INFO: Message received on topic revspace/sensors/co2 with QoS 0 and payload 560 
2014-06-20 18:14:02,643: INFO: Updating: /tmp/revspace/sensors/co2.rrd with value: 560.0
2014-06-20 18:14:02,644: INFO: Message received on topic revspace/sensors/2-t1 with QoS 0 and payload 22 
2014-06-20 18:14:02,644: INFO: Updating: /tmp/revspace/sensors/2-t1.rrd with value: 22.5
2014-06-20 18:14:02,644: INFO: Message received on topic revspace/sensors/2-t3 with QoS 0 and payload 20 
2014-06-20 18:14:02,644: INFO: Updating: /tmp/revspace/sensors/2-t3.rrd with value: 20.5
2014-06-20 18:14:02,644: INFO: Message received on topic revspace/sensors/6-t0 with QoS 0 and payload 20 
2014-06-20 18:14:02,644: INFO: Updating: /tmp/revspace/sensors/6-t0.rrd with value: 20.0
2014-06-20 18:14:02,645: INFO: Message received on topic revspace/sensors/20-t0 with QoS 0 and payload 23 
2014-06-20 18:14:02,645: INFO: Updating: /tmp/revspace/sensors/20-t0.rrd with value: 23.0
2014-06-20 18:14:02,645: INFO: Message received on topic revspace/sensors/2-t2 with QoS 0 and payload 22 
2014-06-20 18:14:02,645: INFO: Updating: /tmp/revspace/sensors/2-t2.rrd with value: 22.75

mqtt

client_id
The client identifier is a 23 byte string that identifies an MQ Telemetry Transport client. Each identifier must be unique to only one connected client at a time. The default is: MQTT2RRD Client.
username
The username to use when connecting to the server. If not configured, will attempt to connect anonymously. The default is anonymous.
password
The password to use when connecting to the server. If not configured, will attempt to connect anonymously. The default is anonymous.
hostname
The hostname or IP address of the MQTT server. The default is localhost.
port
The port to connect to on the MQTT server, the default is 1883.
keepalive
The number of seconds in between sending a keepalive packet to the server. The default is 60 seconds.
subscriptions
A comma separated list of topics to subscribe to, can use # has a wildcard, the default is to subscribe to all topics.

Topic Specific

Topic specific configuration can be made by creating a section with the name of a topic. The following options are supported:

step
The step time in seconds to use when creating the RRD graph. The default is 60.
friendly_name
A friendly name to give the topic, this is stored in the info file for the RRD, and may be used for reference later.
archives
A comma seperated list of RRA configuration strings to be used when creating the graph. The default is: RRA:AVERAGE:0.5:2:30,RRA:AVERAGE:0.5:5:288,RRA:AVERAGE:0.5:30:336,RRA:AVERAGE:0.5:60:1488,RRA:AVERAGE:0.5:720:744,RRA:AVERAGE:0.5:1440:265

Configration Example

[daemon]
data_dir = /tmp
pid_file = /tmp/mqtt2rrd.pid
user = mqtt
group = mqtt

[logging]
log_file = /tmp/mqtt2rrd.log
log_level = 10


[mqtt]
client_id = "MQTT_to_RRD_TEST"
hostname = test.mosquitto.org
port= 1883
username="bob"
password="t0ps3cr3t!"
keepalive=60
subscriptions=/mytopics/#,/other/important/topic

[/other/important/topic]
friendly_name = "My Important Topic"
step=30
archives=RRA:AVERAGE:0.5:1:120,RRA:AVERAGE:0.5:5:288

You can measure engine RPM using the negative terminal of the coil, however you need to isolate the 12v circuit of the coil, from the 5v levels used in the Arduino. This can be done using an optical isolator circuit as detailed below.

The following schematic shows how to connect the circuit:

RPM Module Schematic

Assembly List

The following components, or their equivalent are required to build the RPM module.

Label Part Type Properties
4n25 DIP – 6 pins package DIP (Dual Inline) [THT]; hole size 1.0mm,0.508mm; true; chip label IC; pins 6; pin spacing 300mil
Arduino Arduino Uno (Rev3) type Arduino UNO (Rev3)
C1 Ceramic Capacitor package 100 mil [THT, multilayer]; capacitance 100nF; voltage 6.3V
C2 Ceramic Capacitor package 100 mil [THT, multilayer]; capacitance 100nF; voltage 6.3V
D1 Rectifier Diode package 300 mil [THT]; type Rectifier; part # 1N4001
D2 Zener Diode package Melf DO-213 AB [SMD]; breakdown voltage 5.1V; type Zener; power dissipation 0.5W; part # 1N4732A
R1 390 Ω Resistor package THT; tolerance ±5%; bands 4; resistance 390Ω; pin spacing 400 mil
R2 4.7k Ω Resistor package THT; tolerance ±5%; bands 4; resistance 4.7kΩ; pin spacing 400 mil

Example Code

volatile unsigned long timePointsOpen, timePointsClosed, lastChange;
volatile unsigned int numBangs;

void pointsOpening(){
    unsigned long t;
    t = millis();
    if (lastChange > 0){
        timePointsClosed += (t - lastChange);
    }
    lastChange = t;
    ++numBangs;
}

void setup(){
  pinMode(3, INPUT);
  digitalWrite(3, HIGH);
}

void loop(){
  // Number of coil pulses
  numBangs = 0;
  attachInterrupt(0, pointsOpening, RISING);
  delay(50);
  detachInterrupt(0);
  numBangs = ((60000/50)*numBangs)/2;
  Serial.print("RPM: ");
  Serial.println(numBangs);
}
Made with cherries, almonds, brazil nuts and a few walnuts. In the baking tray ready for the oven.

This is my recipe for flap jack, I use this when I’m hiking, its filling, calorie dense, and tastes good

Ingedients

  • 600g Rolled Oats
  • 300g Butter
  • 300g Sugar
  • 300g Honey
  • 200g Glace Cherries (chopped)
  • 200g Chopped Nuts (Walnuts/Brazil Nuts/Almonds/etc)

Method

  1. Melt sugar, honey and butter in a sauce pan, and stir until the sugar is disolved.
  2. Add all other dry ingredients, mix well until the mixture is a nice sticky goo
  3. Empty into baking tray and bake for about 10-15minutes at 180C until golden on top
  4. Place in fridge until set

This contains about 7400 Calories, and makes enough for 10 GIANT flap jacks weighing about 200g each.

The West Highland Way is a Long Distance Footpath running from Milngavie to Fort William. This trail has been on my list for a while, and as its close to home, and I have a spare week, now is the time. Of course, at 96 miles, its not really that long, and would normally take me around 4 to five days to complete. As I need to get back in shape for some peak bagging in the Himalayas, I decided to add some side trips.

The Plan

Day 1

Leave Milngavie following the west highland way for about 20 miles with seven days food. Aim to camp at the campground at Millarochy or in some woodland nearby.

Day 2

Leave Millarochy and take a side trip to climb Ben Lomond, camp nearby Inversnaid.

Day 3

Leave the West Highland way at Bein Glas farm, climb Beinn Chabhair, An Caisteal, Beinn a’ Chroin , Glas Tulaichean and Cruach Ardrain before sleeping in woods near Inverlochlarig

Day 4

Leaving Inverlochlarig and climbing Stob Binnein and Ben More, before descending into Crainlarich for lunch. Post lunch regain the West Highland way towards Tyndrum and turn left towards Cononish, then ascend regain WHW, Ascend Ben Dubhchraig, and camp somewhere nearby.

Day 5

Climb Ben Oss, return to Tyndrum for lunch and a resupply before following the West Highland Way as far as Gleann Achadh-innis Chailean, follow the Glean before climbing Beinn Mhanach before finding a suitable camp spot.

Day 6

Continue on to the summits of Beinn Achaladair, Beinn a Chreachain, Beinn an Dothaidh, Bein Dorain, and return to Bridge of Orchy, Camp on shore of Loch Tulla.

Day 7

Leave Loch Tulla, climbing Meall a’ Bhuiridh, Creise and camp in Kinlochmore, resupplying as required.

Day 8

Leave Kinlochmore, heading east and attain the ridge to the peaks of Sgurr Eilde Mor, Binnein Beag, Binnein Mor, Na Gruagaichean, Stob Coire a’ Chairn, An Gearanach, Am Bodach, Sgurr a Mhaim, Stob Ban, Mullach Nan Coirean camping a few miles out of Fort William

Day 9

Arrive in Fort William, find accomodation, food, shower and resupply.

This gives me the flexibility to skip sections due to bad weather, (which no doubt there will be) or time, but still push quite hard.

The Gear

Backpack

ULA Catalyst 70L pack

Sleeping System

Hydration and Cooking System

Hygene/First Aid

Small first aid kit containing:

  • Medical tape
  • Bbandage
  • Sam splint
  • Pain killers
  • Suture strips
  • Iodine solution.

Hygene Kit containing:

  • Baby Wipes
  • Toothbrush/Tooth paste

Clothing

  • GT-2000 Shoes
  • Darn Tough Socks (2 Pairs)
  • Icebreaker Boxers (2 Pairs)
  • Running Shorts
  • Merino T-Shirt
  • Sweater
  • Goretex Pants
  • Goretex Jacket
  • Gloves
  • Hat

Food

Nutrition is an important part of any trip, if not the single most important part, especially when you are pushing for distance.  I wish to lose a few lbs on this trip, so I’ve set a daily calorie target of 3000calories.  Yes, that’s three thousand calories.  The recommended daily intake for an adult male is 2500 calories, that assumes a normal day sitting in the office pushing keys on a keyboard.  On this trip Ill likely burn between 4000 and 5000 calories a day, without at least 3000 calories a day, I’ll simply run out of steam after a few days as my body fails to get the energy it requires.

A typical day will consist of the following:

Breakfast

  • 1 Packet of Pop Tarts, these were my staple on the Pacific Crest Trail, and provide 400 calories per packet (4 packets per box)
  • Banana/Fresh fruit, when available, I’ll add fresh fruit to breakfast, this will likely be the first couple of days.

Lunch

  • Pork Pie (350 Calories)
  • Sandwich/Burrito 500(Calories)

Snacks

  • Home made Flap Jack (800 Calories)
  • Chocolate Bar (100 Calories)
  • Trail Mix (200 Calories)

Dinner

  • Pasta/Rice meal 250 Calories
  • Noodle Soup 250 Calories
  • Pudding or other desert (200 Calories)

This driver provides access to the MegaSquirt ECU with a level of abstraction above the MegaSquirt serial interface, enabling users with no knowledge of the MegaSquirt commands to access the register data easily without caring about the underlying communication required.

The MegaSquirt is a low cost engine control unit with support for fueling, ignition and idle control.

Wiring

The MegaSquirt is connected to the Arduino using the RS232 output on the MegaSquirt, a MAX232 chip, and the Serial interface on the arduino board.

Synopsis

 
#include >MegaSquirt.h<
void setup(){
 MegaSquirt::begin();
 
 byte status;
 String signature;
 status=MegaSquirt::signature(&signature);
 switch (status){
  case MS_COMM_SUCCESS:
   Serial.print("MegaSquirt Signature: ");
   Serial.println(signature);
   break;
  case MS_ERR_COMM_TIMEOUT:
   Serial.println(" FAILURE: Communication Timeout");
   break;
  case MS_ERR_COMM_OVERFLOW:
   Serial.println(" FAILURE: Communication Overflow");
   break;
  default:
   Serial.println(" FAILURE: Unknown Error"); 
 }

String revision;
 status=MegaSquirt::revision(&revision);

 uint16_t uptime;
 status=MegaSquirt::seconds(&uptime);

 MegaSquirtData registers;
 byte regTable[MS_TABLE_LENGTH];
 status=MegaSquirt::getData(regTable);

 registers.loadDAta(regTable);

 Serial.println(registers.lambda1()/10,DEC )
}

Download

You can download the ZIP file to import into the IDE here, or you can fork your own.

The LIS331HH is an ultra low-power full-scale three axis MEMS linear accelerometer.  The device also features ultra low-power operational modes that allow advanced power saving and smart sleep to wake-up functions.  I wrote a library for Arduino that provides an object oriented interface to these devices using the I2C bus.

The project is hosted on GitHub here and is licensed under the GPL.

Download

You can download the ZIP file to import into the IDE here, or you can fork your own.

Usage

Be sure to include the LIS331 and Wire libraries in your project first.

setPowerStatus(int status);

Set the power saving status of the device, options are: LR_POWER_OFF, LR_POWER_NORM, LR_POWER_LOW1, LR_POWER_LOW2,LR_POWER_LOW3

setXEnable(bool enable)

Enable or disable X axis

setYEnable(bool enable)

Enable or disable Y axis

setZEnable(bool enable)

Enable or disable Z axis

string lis.getXValue(int16_t &value);

Reads the X value and assigns it to value.

string lis.getYValue(int16_t &value);

Reads the Y value and assigns it to value.

string lis.getZValue(int16_t &value);

Reads the Z value and assigns it to value.

Example

You can download an example project here.

The ELM327 IC is a multi-function OBD2 Interpreter that can be used to query information about most relatively new vehicles.  The ELM327 connects to the vehicle using the OBD2 port, and provides an RS232 Interface.   I recently purchased an OBD2 shield form SparkFun which uses an ELM clone, the STN1110.  I wanted to add OBD2 support to Loguino, but first I needed to be able to communicate using the ELM protocol.

Subsequently I created a the ELM327 class, which contains both high level and low level methods to get information from the controller.  The low level class is used to send arbitrary commands to the ELM chip, there are two main methods,

  • getBytes which requests a specific PID and parses the returned bytes into an array of values.
  • runCommand which sends an arbitrary command to the ELM device, and parses the output into a buffer which contains the response from the ELM.

The high level class provides methods to gather metrics from the controller, each method parses the response from the ELM device into the actual value.  The following code illustrates how to use the class:

#define ELM_TIMEOUT 9000
#define ELM_BAUD_RATE 9600
#define ELM_PORT Serial3

byte status;
Elm327 Elm;
status=Elm.begin();
if (status != ELM_SUCCESS){
    Serial.print("Error: ");
    Serial.println(status);
}
int temp;
Serial.print("OBD2 Coolant Temperature: ");
status=Elm.coolantTemperature(temp);
if (status  == ELM_SUCCESS)
{
    Serial.println ("Pass");
    Serial.print(" Value: ");
    Serial.println(i);
}else{   
    Serial.print("Error: ");
    Serial.println(status);
}

byte values[2];
status=getBytes("01","41","0C",values,2);
if (status == ELM_SUCCESS){
    Serial.print("Elm returned: ");
    Serial.print(values[0], HEX);
    Serial.print(" ");
    Serial.println(values[1], HEX);
    Serial.print("Which is: ");
    Serial.print(((values[0]*256)+values[1])/4, DEC);
    Serial.println(" RPM");
}else{   
    Serial.print("Error: ");
    Serial.println(status);
}

char data[20];
status=runCommand("ATI",data,20);
Serial.print("Current version is: ");
Serial.println(data);

Feature requests, bugs, suggestions and other feedback always appreciated.