For those who wish to convert RRD data bases to CSV, Json and or send to influxdb

Here is a set of software for people to use. I created it for a network monitoring project that i had using BPi R1- it would store data locally on the machine (in a very small RRD format, but you lose precision over time) plus push to remote database InfluxDB or can be pulled to be stored at a higher precision large format. for MicrosoftSQL or other databases,

but it works with all software that uses an RRD Database, i.e. smokeping, ntop(ntopng) cactus, etc.
Basically for ease of use, since I had 50 remote networking monitoring devices to install. Pretty much nationwide.
I made it as simple as possible to set up.

You simply plug the device in. It acts as a transparent bridge or router and it will automatically configure to DHCP and start sending data. Since Influx is so easy to use, it’s based on that for pushing data. All one has to do is flash your SD card insert it in the BPi-r1 or r2 and it will automatically create a database in InfluxDB. The unique identifier is based on the MAC address of the device. Here a screenshot of the device’s output to InfluxDB via Grafana:

RRD2CSV_JSON_INFLUX.tar.gz (2.2 MB)




But anyway, I thought it wou;d be useful for those who wish to push data to their emonCMS database as it is simply a converter for other software. All you would need to do is edit the json string to something compatible with what ever IoT database you with to use and push (curl) the string to your account.

Okay, good luck, have fun.

5 Likes

here an updated variant of the above software but specifically for the openwrt platform .

it works very well. you can configure it many ways

it will use the CollectD software for network and hardware info, if using it for your IOT archival data you just need to configure to output a CSV data base every once in a while to update to influxdb via the software
its default send is every 30 seconds but that is adjustable up to 4 hours or more .

just edit influxsend to adjust timing setting
to tell the software where your csv are located edit cp-csv
and to edit influxdb setting edit settings in Sendinflux.pl

In the openwrt platform all the high write data is stored in memory so at reboot all data is lost so if you are worried about that leave the setting for timer at 30sec all you need to do once you copy it to your Openwrt / dir is set up a small bash script to run at boot that first copies the runfiles folder to the /tmp then execute influxsend afterward . if you want long data storage just insert USB key or harddrive and point rrd storage to that key via collectd rrdtools

runfiles.zip (2.2 MB)

for basic setup the SD is smaller then 100megs with freeboard IOT node-red… and you should be able to port to many off the shelf routers. if you leave out node red it only 45- 60 megs the entire OS and IOT system

sorry for some reason every time I compress these file for some reason one import file is missing Sendinflux.pl I re compressed this time with tar and uploaded it to Github… please let me know if you have issue with the file

I check it several times and the file is there so I believe the package is not broken this time …