MQTT2RRD is a daemon written in python that connects to an MQTT server, subscribes to selected topics, and stores values in RRD round-robin databases for later graphing.
The code is hosted on git hub, and is licensed under the GPL.
Usage
usage: mqtt2rrd.py [-h] {stop,restart,start} ... optional arguments: -h, --help show this help message and exit --config_file CONFIG_FILE The location of the config file --no_daemon Do not spawn a daemon, stay in the foreground
Configuration
Configuration is via a configuration file, by default mqtt2rrd looks in the following locations:
/etc/mqtt2rrd.conf ~/.mqtt2rrd.conf
You can specify the location of the configuration on the command line using the –config_file argument. For example:
mqtt2rrd.py start --config_file=/home/bob/mqtt2rrdtest.conf
Configuration is stored using an ini file using the python ConfigParser format. The following configuration options are available:
daemon
- pid_file
- The PID file location, used to store the PID of the daemon process.
- data_dir
- Configures the root directory to store the RRD files in. RRDs are in sub directories corresponding to their topic name. For example, a message with topic “foo/bar/baz” would be stored as $DATA_DIR/foo/bar/baz.rrd. The default data directory is /var/lib/mqtt2rrd.
- user
- The user to run as, when a username and group is specified, and the script is executed as root, then the script will demote itself to the specified user and group
- group
- The group to run as, when a username and group is specified, and the script is executed as root, then the script will demote itself to the specified user and group
logging
Logging is done using the python logging class, at this time you can configure the log file, and the log level.
- log_level
- The level to log at, 10:debug and above, 20:info and above, 30: warn and above, 40: error and above, 50: Critical log messages only. The default is to log at level info and above.
- log_file
- The file to log to, by default this is /var/log/mqtt2rrd.log
Example Log Output
2014-06-20 18:14:02,531: INFO: Connected: test.mosquitto.org:1883 2014-06-20 18:14:02,584: INFO: Connected to server. 2014-06-20 18:14:02,584: INFO: Subscribing to topic: # 2014-06-20 18:14:02,643: INFO: Message received on topic revspace/sensors/co2 with QoS 0 and payload 560 2014-06-20 18:14:02,643: INFO: Updating: /tmp/revspace/sensors/co2.rrd with value: 560.0 2014-06-20 18:14:02,644: INFO: Message received on topic revspace/sensors/2-t1 with QoS 0 and payload 22 2014-06-20 18:14:02,644: INFO: Updating: /tmp/revspace/sensors/2-t1.rrd with value: 22.5 2014-06-20 18:14:02,644: INFO: Message received on topic revspace/sensors/2-t3 with QoS 0 and payload 20 2014-06-20 18:14:02,644: INFO: Updating: /tmp/revspace/sensors/2-t3.rrd with value: 20.5 2014-06-20 18:14:02,644: INFO: Message received on topic revspace/sensors/6-t0 with QoS 0 and payload 20 2014-06-20 18:14:02,644: INFO: Updating: /tmp/revspace/sensors/6-t0.rrd with value: 20.0 2014-06-20 18:14:02,645: INFO: Message received on topic revspace/sensors/20-t0 with QoS 0 and payload 23 2014-06-20 18:14:02,645: INFO: Updating: /tmp/revspace/sensors/20-t0.rrd with value: 23.0 2014-06-20 18:14:02,645: INFO: Message received on topic revspace/sensors/2-t2 with QoS 0 and payload 22 2014-06-20 18:14:02,645: INFO: Updating: /tmp/revspace/sensors/2-t2.rrd with value: 22.75
mqtt
- client_id
- The client identifier is a 23 byte string that identifies an MQ Telemetry Transport client. Each identifier must be unique to only one connected client at a time. The default is: MQTT2RRD Client.
- username
- The username to use when connecting to the server. If not configured, will attempt to connect anonymously. The default is anonymous.
- password
- The password to use when connecting to the server. If not configured, will attempt to connect anonymously. The default is anonymous.
- hostname
- The hostname or IP address of the MQTT server. The default is localhost.
- port
- The port to connect to on the MQTT server, the default is 1883.
- keepalive
- The number of seconds in between sending a keepalive packet to the server. The default is 60 seconds.
- subscriptions
- A comma separated list of topics to subscribe to, can use # has a wildcard, the default is to subscribe to all topics.
Topic Specific
Topic specific configuration can be made by creating a section with the name of a topic. The following options are supported:
- step
- The step time in seconds to use when creating the RRD graph. The default is 60.
- friendly_name
- A friendly name to give the topic, this is stored in the info file for the RRD, and may be used for reference later.
- archives
- A comma seperated list of RRA configuration strings to be used when creating the graph. The default is: RRA:AVERAGE:0.5:2:30,RRA:AVERAGE:0.5:5:288,RRA:AVERAGE:0.5:30:336,RRA:AVERAGE:0.5:60:1488,RRA:AVERAGE:0.5:720:744,RRA:AVERAGE:0.5:1440:265
Configration Example
[daemon] data_dir = /tmp pid_file = /tmp/mqtt2rrd.pid user = mqtt group = mqtt [logging] log_file = /tmp/mqtt2rrd.log log_level = 10 [mqtt] client_id = "MQTT_to_RRD_TEST" hostname = test.mosquitto.org port= 1883 username="bob" password="t0ps3cr3t!" keepalive=60 subscriptions=/mytopics/#,/other/important/topic [/other/important/topic] friendly_name = "My Important Topic" step=30 archives=RRA:AVERAGE:0.5:1:120,RRA:AVERAGE:0.5:5:288
Hi, I’m having a wee bit of trouble with this. I’ve got a sensor, thats recording current.. mqtt2rrd, is subscribed to it, but its just writing zeros to the rrdfile. ( i have checked the red-database using rrdtool dump. )
I’ve tyed a couple of things, but nothing seems to stop it from writing zeros.. Any help, greatly appreciated.
root@edison:~/MQTT2RRD-master# ./mqtt2rrd.py start –config_file=mqtt2rrd.conf –no_daemon
mqtt mqtt
2014-11-02 08:10:54,256: INFO: Running as: root
2014-11-02 08:10:54,261: INFO: Connected: localhost:1883
2014-11-02 08:10:54,265: INFO: Connected to server.
2014-11-02 08:10:54,265: INFO: Subscribing to topic: ticc3201/current
2014-11-02 08:10:59,949: INFO: Message received on topic ticc3201/current with QoS 0 and payload 0
2014-11-02 08:10:59,949: INFO: Updating: /var/mqtt2rrd/ticc3201/current.rrd with value: 0.258
2014-11-02 08:11:09,070: INFO: Message received on topic ticc3201/current with QoS 0 and payload 0
2014-11-02 08:11:09,070: INFO: Updating: /var/mqtt2rrd/ticc3201/current.rrd with value: 0.258
heres my .conf file
root@edison:~/MQTT2RRD-master# more mqtt2rrd.conf
[daemon]
data_dir = /var/mqtt2rrd
pid_file = /tmp/mqtt2rrd.pid
user = mqtt
group = mqtt
[logging]
log_file = /tmp/mqtt2rrd.log
log_level = 20
[mqtt]
client_id = “MQTT_to_RRD_TEST”
hostname = localhost
port= 1883
#username=”bob”
#password=”t0ps3cr3t!”
keepalive=60
subscriptions=ticc3201/current
[ticc3201/current]
friendly_name = “current”
step=5
archives=DS:current:GAUGE:15:0:15,RRA:AVERAGE:0.5:1:120,RRA:AVERAGE:0.5:5:288
Found the issue, on line 193 of the python code, its formatted as % d, when in fact for my data it needs to be % f
Now logging data, very helpful!
Thanks Andrew,
I’ve updated the code on GitHub.
Cheers
David
Hi,
thank you for your program.
For information, I had to replace
import mosquitto, rrdtool
by
import paho.mqtt.client as paho, rrdtool
and
client = mosquitto.Mosquitto(get_config_item(“mqtt”, “client_id”, “MQTT2RRD Client”))
by
client = paho.Client(get_config_item(“mqtt”, “client_id”, “MQTT2RRD Client”))
to use your program with the last version of Mosquitto.
It may help others 😉
Alex
Hi,
Just sent a pull request on GitHub, to use the paho mqtt library instead of the mosquitto one. Just noticed Alex’s comment stating the same although I suggested importing the library as mosquitto instead of paho.
being trying to get this to work I habe mosquito and rrdtool installed but when ever i run mqtt2rrd.py it gets this error ( i placed mqtt2rrd in the same directory as rrdtools and mosquitto are located..
Traceback (most recent call last):
File “/usr/bin/mqtt2rrd.py”, line 22, in
import rrdtool, mosquitto
ImportError: No module named rrdtool
any suggestions to what might be wrong ( ubuntu 16.04)