I have been investigating the transfer of data from the CGX to Emoncms. I know the battery monitor will perform this task, but it looks a bit klunky to me.
I am investigating the way OpenHab reads data from the ccgx. They use MQTT, & Json to read & sort the data which seems to me to be more elegant. JSON needs this :: GitHub - dvdln/jsonpath-object-transform: Transform an object literal using JSONPath., & a little code from OpenHab.
The reason I feel this is a good option is because I need quite a bit of data from the CCGX. I have a split system, where the Solar PV, inverters, Victron Multiplus 2, & batteries are about 30M from all the house loads. I have a IoTaWatt for the house, & Emoncms on an intel NUC. I am on a 180 degree rural 2 phase system, so have 2 Solaredge PV inverters & 2 Multiplus 2, with a CCGX having control. I have 12Kw of ´Zenaji´ Lithium Titanate batteries (Australian made). Because I need to read the Battery information from the CCGX, I might as well use it for all the data it can supply.
The disadvantage of the current approach to reading ccgx data is that it is a bit complicated, & requires a restart of the config file if it requires modification.
My problem is I am not a programmer, so will need some assistance. Is this idea worth pursuing?.
I have done some more research. It seems I need a Mosquitto Client installed, then from the client the reads from the CCGX can be enabled. There is info on the Victron site, & implementations have been made for Influxdb & Home Assistant. There are some issues in that some fields come back with a ¨null¨ value which can cause problems.
The good thing is if this is implemented, no changes need to be made to the CCGX (or Venus implementation: means updates will not change user files on the CCGX). I feel Jsonpath as modified by OpenHab (where they filtered the Null values for instance) would be required.
Does the CCGX publish data to an external MQTT server, or does it act as an MQTT server itself?
If the former then you’d probably need to set up the Mosquitto server, though that’s really not very difficult, and I think the command-line clients would get installed at the same time.
If the latter then you probably don’t specifically need a Mosquitto client, though they might be handy for debugging.
Assuming the former, once you have the data you want being published to an MQTT topic, getting it into emoncms shouldn’t really be too hard. If the data doesn’t need modifying then everything might just work with the tools that already exist. If it does then that might be possible by plugging together some stuff in NodeRED.
I’d suggest the first stage should be to reach the point where you can use the mosquitto command line client to connect to an MQTT server and view the data coming from the CCGX. If you can do that, which really shouldn’t be too hard, then it should be possible to get a clearer idea of how much more work is required.
In that case, I’d run that second “mosquitto_sub” command in the linked page and see what gets displayed. If you want most/all of them to appear as inputs in emoncms then quite possibly it’s just a case of hooking up the MQTT import script in emoncms. If you only want a few then perhaps filtering them using NodeRED and using it to submit them to emoncms via the API might be a sensible way forward.
It may need a little guidance, and I’m afraid I’m probably not the one to do that where NodeRED is concerned as I have very limited experience with it myself, but I don’t think it should be a hideously complex job.
I would like to install the following (when edited) scripts then be able to use them when necessary. I will need both read & write capability because I want to install a Victron PV Mppt with a string of panels to charge my batteries (which is basically an upgrade to my PV system, but not grid connected directly) This is needed because if the main batteries are depleted, & the grid is down, the Victron might not start when the sun comes up. By installing a PV charger, the system will start (using 3s x3p string 250w
ie ~2Kw, recycled panels, & a BlueSolar 150/35 MPPT connected to my CCGX.) I would like all this data visible in EmonCMS.
So, I need the ´Keep alive´ script running, then the required data read, then decoded. Keep in mind this data will contain ´null´ data (non numerical) when the PV inverters are asleep, which is why I think the Json datatransform script? might be necessary too. Where are these scripts installed? Is there a page/ tutorial I can access? (looking at pages such as Steve´s tutorial is OK, but how does it fit in?
I feel in the future it might be good to have an expandable system where the parameters can be defined in a table possibly externally accessible to make it more understandable. There seem to be other HW that might be able to use such a setup (BMS systems, other battery systems, PV inverters, etc), so it might make maintenance & expandibility easier in the future. Btw, I repeat I am not a s/w person, so I can fiddle, but don´t have any experience in development.
To assist the documentation (Tutorial?), I would be happy to summarize this topic so others don´t have to chase this info as I have done.
One thing I did not mention in the last communication is that the ´Keep Alive´ function will need to be enabled/disabled, because not all applications will need it. ( If there was a setup panel, this could be a switch) btw, I do like containerising items so we do not have to re-invent the wheel all the time.
If there is a tutorial for the bits I’ve used, I’m afraid I didn’t find it. Mostly I muddled my way through things until I had a reasonable idea of what was going on and what needed to happen.
In my case I have some power monitors and code that reads them and publishes the data to an MQTT server (Mosquitto, in this case). That bit is probably not relevant for you. Then I’m using scripts/services/emoncms_mqtt/emoncms_mqtt.php from the emoncms sources to read data from the MQTT server and push it into emoncms. There’s very little configuration required as long as the MQTT server details and the correct topic name are set in the main configuration. In that same directory there’s a file emoncms_mqtt.service which configures the system to start the script and keep it running (assuming it isn’t already), but I’d not worry about that until you know it would be the most sensible approach in the first place.
There’s some information in the README.md here that might be useful. They even suggest something that should work for a keepalive on Linux, at least whilst you’re trying to get everything set up.
I suspect that if you can’t avoid having to mangle the data because emoncms won’t like it otherwise then you’re going to have to start playing with NodeRED though. The first step has to be to run
mosquitto_sub -h ccgx -t 'N/#' -v
and see what the data you’re interested in looks like though. It would probably be helpful to see the nulls being used in context to determine a sensible way forward.
Command in process atm. My reading of EmonCMS is that MQTT is basically used as a server, not a receiver. But Mosquitto_client is loaded, so available.
What I am suggesting (but do not have the skills to implement) is to use code similar to the OH code, where they have modified Jsonpath-object-transform to perform the required functions. I will also be using the MQTT client.
I really want to keep everything in EmonCMS, mainly because I am using an IoTaWatt to monitor all the circuits in my house, where the generation is 30M away, & instead of doubling the monitoring, The system is a rural 2 phase, 240-0-240v supply. I will use the data from the CCGX for the following:
CCGX
2 of SolarEdge SE5000 inverters
2 of Victron Multiplus 2 ch/inverters
DC Battery Current smart shunt
Inbuilt Victron circuit monitoring
12Kw Zenaji Lithium Titanate batteries
BlueSolar 150/35 MPPT interface,
so as you can see it will be much easier to read the data from the CCGX (with all its encompassed issues).
If I interpret the response to one of my other posts correctly, emoncms may well just skip input messages with non-numeric values, so the whole issue of nulls could turn out to be something that can be ignored in which case there may well be no need for NodeRED.
Thanks James. As a non-programmer, I think it will be easier to use Node-Red for me. It will send data to EmonCMS using it, & I will not have to worry about the Venus GX going to sleep, etc.
The Node-Red implementation is only pre-release, but is an addition to the standard Venus-GX firmware, so will not be damaged during future firmware updates. I need to read many registers from the Venus GX, including the Battery SOC, & hopefully I will be able to also access the data from the SolarEdge SE5000 inverters (x2).
Anyway, have nothing to lose. I was given a faulty Venus GX which I have diagnosed (faulty DC-DC module) so I will be able to try it soon when the parts arrive.
Thanks for the help. I will probably start a new thread on the Installation & setup of the Node-Red implementation. (If & when I repair the Venus GX!)
I’ve just schwitched from running dbus-emoncms on the Cerbo to use Node-red for posting data to my local emoncms server. It works very well and virtually all the data from my Victron setup can be uploaded.
You need to install and use the special victron and emoncms nodes, join the data you want to monitor and post it to your emoncms account. Very simple.
James & Lykke,
thanks. I have set up a Vivtron Venus GX that I repaired (Faulty Beagleboard, & dc-dc chip on cape) Yet to test it to see if the VE Bus working) but everything else seems to. Loaded Venus-OS-Large, & it is working on the bench.
I am waiting on my installer to commission some more panels. I am installing abt 2Kw panels into a Victron MPPT 150/35 that will ´kick start´ the system if the batteries are depleted by extended grid failure. (After the batteries are charged, the excess is to be fed thru the Multiplis 2, to the grid to either be locally used, or fed to the grid.
I now have after the extras, 20Kw on the roof.