MQTT to RRD Daemon: Graph all the things!!!

Graph all the things

MQTT2RRD is a daemon written in python that connects to an MQTT server, subscribes to selected topics, and stores values in RRD round-robin databases for later graphing.

The code is hosted on git hub, and is licensed under the GPL.

Download

Usage

usage: mqtt2rrd.py [-h] {stop,restart,start} ...
optional arguments:
  -h, --help            show this help message and exit
  --config_file CONFIG_FILE
                        The location of the config file
  --no_daemon           Do not spawn a daemon, stay in the foreground

Configuration

Configuration is via a configuration file, by default mqtt2rrd looks in the following locations:

/etc/mqtt2rrd.conf
~/.mqtt2rrd.conf

You can specify the location of the configuration on the command line using the –config_file argument. For example:

mqtt2rrd.py start --config_file=/home/bob/mqtt2rrdtest.conf

Configuration is stored using an ini file using the python ConfigParser format. The following configuration options are available:

daemon

pid_file
The PID file location, used to store the PID of the daemon process.
data_dir
Configures the root directory to store the RRD files in. RRDs are in sub directories corresponding to their topic name. For example, a message with topic “foo/bar/baz” would be stored as $DATA_DIR/foo/bar/baz.rrd. The default data directory is /var/lib/mqtt2rrd.
user
The user to run as, when a username and group is specified, and the script is executed as root, then the script will demote itself to the specified user and group
group
The group to run as, when a username and group is specified, and the script is executed as root, then the script will demote itself to the specified user and group

logging

Logging is done using the python logging class, at this time you can configure the log file, and the log level.

log_level
The level to log at, 10:debug and above, 20:info and above, 30: warn and above, 40: error and above, 50: Critical log messages only. The default is to log at level info and above.
log_file
The file to log to, by default this is /var/log/mqtt2rrd.log

Example Log Output

2014-06-20 18:14:02,531: INFO: Connected: test.mosquitto.org:1883

2014-06-20 18:14:02,584: INFO: Connected to server.
2014-06-20 18:14:02,584: INFO: Subscribing to topic: #
2014-06-20 18:14:02,643: INFO: Message received on topic revspace/sensors/co2 with QoS 0 and payload 560 
2014-06-20 18:14:02,643: INFO: Updating: /tmp/revspace/sensors/co2.rrd with value: 560.0
2014-06-20 18:14:02,644: INFO: Message received on topic revspace/sensors/2-t1 with QoS 0 and payload 22 
2014-06-20 18:14:02,644: INFO: Updating: /tmp/revspace/sensors/2-t1.rrd with value: 22.5
2014-06-20 18:14:02,644: INFO: Message received on topic revspace/sensors/2-t3 with QoS 0 and payload 20 
2014-06-20 18:14:02,644: INFO: Updating: /tmp/revspace/sensors/2-t3.rrd with value: 20.5
2014-06-20 18:14:02,644: INFO: Message received on topic revspace/sensors/6-t0 with QoS 0 and payload 20 
2014-06-20 18:14:02,644: INFO: Updating: /tmp/revspace/sensors/6-t0.rrd with value: 20.0
2014-06-20 18:14:02,645: INFO: Message received on topic revspace/sensors/20-t0 with QoS 0 and payload 23 
2014-06-20 18:14:02,645: INFO: Updating: /tmp/revspace/sensors/20-t0.rrd with value: 23.0
2014-06-20 18:14:02,645: INFO: Message received on topic revspace/sensors/2-t2 with QoS 0 and payload 22 
2014-06-20 18:14:02,645: INFO: Updating: /tmp/revspace/sensors/2-t2.rrd with value: 22.75

mqtt

client_id
The client identifier is a 23 byte string that identifies an MQ Telemetry Transport client. Each identifier must be unique to only one connected client at a time. The default is: MQTT2RRD Client.
username
The username to use when connecting to the server. If not configured, will attempt to connect anonymously. The default is anonymous.
password
The password to use when connecting to the server. If not configured, will attempt to connect anonymously. The default is anonymous.
hostname
The hostname or IP address of the MQTT server. The default is localhost.
port
The port to connect to on the MQTT server, the default is 1883.
keepalive
The number of seconds in between sending a keepalive packet to the server. The default is 60 seconds.
subscriptions
A comma separated list of topics to subscribe to, can use # has a wildcard, the default is to subscribe to all topics.

Topic Specific

Topic specific configuration can be made by creating a section with the name of a topic. The following options are supported:

step
The step time in seconds to use when creating the RRD graph. The default is 60.
friendly_name
A friendly name to give the topic, this is stored in the info file for the RRD, and may be used for reference later.
archives
A comma seperated list of RRA configuration strings to be used when creating the graph. The default is: RRA:AVERAGE:0.5:2:30,RRA:AVERAGE:0.5:5:288,RRA:AVERAGE:0.5:30:336,RRA:AVERAGE:0.5:60:1488,RRA:AVERAGE:0.5:720:744,RRA:AVERAGE:0.5:1440:265

Configration Example

[daemon]
data_dir = /tmp
pid_file = /tmp/mqtt2rrd.pid
user = mqtt
group = mqtt

[logging]
log_file = /tmp/mqtt2rrd.log
log_level = 10


[mqtt]
client_id = "MQTT_to_RRD_TEST"
hostname = test.mosquitto.org
port= 1883
username="bob"
password="t0ps3cr3t!"
keepalive=60
subscriptions=/mytopics/#,/other/important/topic

[/other/important/topic]
friendly_name = "My Important Topic"
step=30
archives=RRA:AVERAGE:0.5:1:120,RRA:AVERAGE:0.5:5:288

6 thoughts on “MQTT to RRD Daemon: Graph all the things!!!

  1. Hi, I’m having a wee bit of trouble with this. I’ve got a sensor, thats recording current.. mqtt2rrd, is subscribed to it, but its just writing zeros to the rrdfile. ( i have checked the red-database using rrdtool dump. )

    I’ve tyed a couple of things, but nothing seems to stop it from writing zeros.. Any help, greatly appreciated.

    root@edison:~/MQTT2RRD-master# ./mqtt2rrd.py start –config_file=mqtt2rrd.conf –no_daemon
    mqtt mqtt
    2014-11-02 08:10:54,256: INFO: Running as: root
    2014-11-02 08:10:54,261: INFO: Connected: localhost:1883
    2014-11-02 08:10:54,265: INFO: Connected to server.
    2014-11-02 08:10:54,265: INFO: Subscribing to topic: ticc3201/current
    2014-11-02 08:10:59,949: INFO: Message received on topic ticc3201/current with QoS 0 and payload 0
    2014-11-02 08:10:59,949: INFO: Updating: /var/mqtt2rrd/ticc3201/current.rrd with value: 0.258
    2014-11-02 08:11:09,070: INFO: Message received on topic ticc3201/current with QoS 0 and payload 0
    2014-11-02 08:11:09,070: INFO: Updating: /var/mqtt2rrd/ticc3201/current.rrd with value: 0.258

    heres my .conf file

    root@edison:~/MQTT2RRD-master# more mqtt2rrd.conf
    [daemon]
    data_dir = /var/mqtt2rrd
    pid_file = /tmp/mqtt2rrd.pid
    user = mqtt
    group = mqtt

    [logging]
    log_file = /tmp/mqtt2rrd.log
    log_level = 20

    [mqtt]
    client_id = “MQTT_to_RRD_TEST”
    hostname = localhost
    port= 1883
    #username=”bob”
    #password=”t0ps3cr3t!”
    keepalive=60
    subscriptions=ticc3201/current

    [ticc3201/current]
    friendly_name = “current”
    step=5
    archives=DS:current:GAUGE:15:0:15,RRA:AVERAGE:0.5:1:120,RRA:AVERAGE:0.5:5:288

  2. Found the issue, on line 193 of the python code, its formatted as % d, when in fact for my data it needs to be % f

    Now logging data, very helpful!

  3. Hi,

    thank you for your program.

    For information, I had to replace
    import mosquitto, rrdtool
    by
    import paho.mqtt.client as paho, rrdtool

    and
    client = mosquitto.Mosquitto(get_config_item(“mqtt”, “client_id”, “MQTT2RRD Client”))
    by
    client = paho.Client(get_config_item(“mqtt”, “client_id”, “MQTT2RRD Client”))

    to use your program with the last version of Mosquitto.

    It may help others 😉

    Alex

  4. Hi,
    Just sent a pull request on GitHub, to use the paho mqtt library instead of the mosquitto one. Just noticed Alex’s comment stating the same although I suggested importing the library as mosquitto instead of paho.

  5. being trying to get this to work I habe mosquito and rrdtool installed but when ever i run mqtt2rrd.py it gets this error ( i placed mqtt2rrd in the same directory as rrdtools and mosquitto are located..
    Traceback (most recent call last):
    File “/usr/bin/mqtt2rrd.py”, line 22, in
    import rrdtool, mosquitto
    ImportError: No module named rrdtool

    any suggestions to what might be wrong ( ubuntu 16.04)

Leave a Reply

Your email address will not be published. Required fields are marked *