Uploading MATE data from an OutbackCC to emoncms

Hi guys,

Seeing you guys are doing wonderful work here. I have no php knowlege but will be happy if you guide me. I have a csv file that gets generated every 20 seconds. I have installed emoncms but cant upload the data.

charger_mode,error_mode,pv_voltage,timestamp,daily_kwh,daily_ah,aux_mode,pv_current,address,charge_current,bat_voltage
2,0,87,5/20/2017 9:50,1.8,35,3,13,A,22.4,49.8
2,0,85,5/20/2017 9:50,1.8,36,3,13,A,22.7,49.8
2,0,83,5/20/2017 9:50,1.8,36,3,14,A,23,49.9
2,0,81,5/20/2017 9:51,1.8,36,3,14,A,22.8,51
2,0,80,5/20/2017 9:51,1.8,36,3,14,A,22.7,51.3
2,0,83,5/20/2017 9:52,1.8,36,3,14,A,23.4,51.8
2,0,84,5/20/2017 9:52,1.8,36,3,14,A,23.4,51.9
2,0,84,5/20/2017 9:53,1.8,37,3,14,A,23.7,52
2,0,82,5/20/2017 9:53,1.8,37,3,15,A,23.6,52.1
2,0,83,5/20/2017 9:53,1.8,37,3,15,A,23.6,52.2
2,0,85,5/20/2017 9:54,1.9,37,3,15,A,23.7,52.2
2,0,87,5/20/2017 9:54,1.9,37,3,14,A,23.7,52.3
2,0,85,5/20/2017 9:54,1.9,37,3,15,A,24.1,52.3
2,0,83,5/20/2017 9:55,1.9,37,3,15,A,24.2,52.3
2,0,81,5/20/2017 9:55,1.9,38,3,15,A,24.2,52.4
2,0,87,5/20/2017 9:56,1.9,38,3,14,A,24.3,51
2,0,81,5/20/2017 9:56,1.9,38,3,15,A,24.4,50.8
2,0,82,5/20/2017 9:56,1.9,38,3,15,A,24.5,50.7
2,0,83,5/20/2017 9:57,1.9,38,3,15,A,24.6,50.5
2,0,83,5/20/2017 9:57,1.9,39,3,15,A,24.7,50.5
2,0,84,5/20/2017 9:58,1.9,39,3,15,A,24.9,50.5
2,0,82,5/20/2017 9:58,1.9,39,3,15,A,25.1,50.4
2,0,84,5/20/2017 9:59,2,39,3,15,A,25.3,50.4

I need to build those beautiful graphs guys.

Cheers.

Since none of us (as near as I know) can read minds,
itā€™d be a BIG help if you told us more about your system.

Is the emoncms you installed on a Windows machine? A Linux box? Which version of Windows / Linux is it running on? Is it the pre-built version? If so, what is the version number? Or did you build it yourself? What have you tried thatā€™s not working? What else have you tried? Did you change anything? If so, what was it?

Do you see where Iā€™m going with this? The more info you can give us, the more likely someone will be able to help you. Please be as descriptive as possible.

1 Like

Thanks Bill for your quick response. It is installed on a Windows 10 PC. I have followed this guide https://github.com/emoncms/emoncms/blob/master/docs/WindowsInstall.md to do the install.

Also, I have a batch file generating the above .csv on a Windows PC every 20 seconds. Hope this helps

It would be helpful to know what that .bat (or .cmd, if thatā€™s the case) file is doing.
Iā€™ve bumped your account up a notch, to enable you to post it

1 Like

Thanks.

below is the content of the batch file.

@echo off
cls
:start
C:\python27\python.exe C:\mate\matecomV2.py %*

TIMEOUT /T 20
goto start

Thatā€™s a start. But what does your Python script do? We need to look at that too.

1 Like
import csv
import os
import os.path
from sys import argv
#from pathlib2 import Path

def write_to_csv(data):
    statinfo = None
    file_existed = os.path.isfile('matecomdata.csv')
    if (file_existed):
        statinfo = os.stat('matecomdata.csv')
    
    file = open("matecomdata.csv", 'a')
    #print (statinfo.st_size)
        
    if not file_existed or (file_existed and statinfo.st_size == 0):
        line = ""
        for key,value in data.iteritems():
            if len(line) == 0:
                line+=key
            else:
                line+=","+key
        file.write(line)
        file.write("\n")
    line = ""
    for key,value in data.iteritems():
        if len(line) == 0:
            line+=repr(value)
        else:
            #line+=","+value
            #line+=repr(value)
            line+=","+repr (value)
    file.write(line)
    file.write("\n")   

if __name__ == "__main__":
    data = {'daily_ah': '144.0', 'pv_current': '0', 'charger_mode': '00', 'error_mode': '000', 'pv_voltage': '18', 'aux_mode': '03', 'address': 'A', 'charge_current': '0.0', 'daily_kwh': '7.8', 'bat_voltage': '51.1'}
    write_to_csv(data)

csvmake.txt (1.2 KB)

# pyMate controller
# Author: Jared Sanson <[email protected]>
#
# Allows communication with an Outback Systems MATE controller panel,
# which provides diagnostic information of current charge state and power use
#
# Currently only supports the MX status page (from an Outback MX charge controller)
#
# NOTE: This is intended for communication with the MATE's RS232 port, not Outback's proprietary protocol.

import serial
import csvmake
import datetime

from value import Value
from time import gmtime, strftime

class MXStatusPacket(object):
    """
    Represents an MX status packet, containing useful information
    such as charge current and PV voltage.
    """
    def __init__(self, packet):
        fields = packet.split(',')
        self.address = fields[0]
        self.timestamp = strftime('%Y-%m-%d %H:%M:%S', gmtime())
        # fields[1] unused
        self.charge_current = Value(float(fields[2]) + (float(fields[6]) / 10.0), resolution=1)
        self.pv_current = Value(fields[3], resolution=0)
        self.pv_voltage = Value(fields[4], resolution=0)
        self.daily_kwh = Value(float(fields[5]) / 10.0, resolution=1)
        self.aux_mode = fields[7]
        self.error_mode = fields[8]
        self.charger_mode = fields[9]
        self.bat_voltage = Value(float(fields[10]) / 10, resolution=1)
        self.daily_ah = Value(float(fields[11]), resolution=1)
	# fields[12] unused

        chk_expected = int(fields[13])
        chk_actual = sum(ord(x)-48 for x in packet[:-4] if ord(x)>=48)
        if chk_expected != chk_actual:
            raise Exception("Checksum error in received packet")


class MateCom(object):
    """
    Interfaces with the MATE controller on a specific COM port.
    Must be a proper RS232 port with RTS/DTR pins.
    """
    def __init__(self, port, baudrate=19200):
        self.ser = serial.Serial(port, baudrate, timeout=2)

        # Provide power to the Mate controller
        self.ser.setDTR(True)
        self.ser.setRTS(False)

        self.ser.readline()

    def read_status(self):
        ln = self.ser.readline().strip()
        return MXStatusPacket(ln) if ln else None


if __name__ == "__main__":
    # Test
    mate = MateCom('COM3')
    status = mate.read_status()
    #print status.__dict__
    print 'Writing to CSV'
    csvmake.write_to_csv(status.__dict__)
    print 'CSV Output written'
    #{'charger_mode': '00', 'error_mode': '000', 'pv_voltage': 18V, 'charge_current': 0.0A, 'daily_kwh': 7.8kWh, 'daily_ah': 144.0Ah, 'pv_current': 0A, 'address': 'A', 'aux_mode': '03', 'bat_voltage': 51.1V}

matecom.txt (2.6 KB)

Here attached

You can modify it to write directly to emoncms. It is a modified script from a wonderful guy, Jared. It works on Outback CC.

Thansk.

OK. Now we know the inverter charge controller youā€™re reading is made by Outback Systems.
I donā€™t know if any of the other users have an Outback charge controller, but if they do, theyā€™ll be in a better position to help you. Unfortunately, I donā€™t have any experience with any of Outbackā€™s products.

My interaction with you was to try and gather as much info as possible, so you could (hopefully) get help a bit quicker

1 Like

Oh Oh, it is not an Inverter, it is a Charge Controller. So thanks Bill. Eagerly waiting for the help. till thenā€¦ cheers.

Can you also provide some sample output from makecom.py with this line

#print status.__dict__

uncommented ("#" removed) so we can see the raw data?

There are numerous ways to do what you ask. What you explicitly ask is how to upload data from a CSV text file that get written to (or created) every 20secs but that may not be the best route. The less moving parts, so to speak, the better, so we could edit the mate.com to post directly to emoncms or to emonhub, via http (or mqtt?) via CSV or JSON. Or we could change the csvmake module to send the data to emonhub or emoncms rather than a text file.

Is the serial interfaced device on the same PC as emoncms and the .bat file?

What else do you have setup or in the pipeline to interface or monitor?

Do you have emonHub installed?

I am not familiar with ā€œOutback Systemsā€ nor do I know how widely they are used, ideally the ultimate solution for the ā€œOEM + emonā€ eco-sytem would be to write a custom interfacer for emonhub so that error logging, auto start up and posting to emoncms were all taken care of by existing emonhub code and only the addition of code to read the device was needed.

1 Like

Hello Paul

Thanks for your response.

  1. I will provide the #print status.dict to you as soon as i get back home in 4 hrs time or less.
  2. Kindly advise and assist with the best method to implement this.
  3. Yes, the serial interface is on the same PC with the .bat file.
  4. Nothing for now.
  5. Am yet to install emonHub? should I go ahead and do that ? can you give a head up by guiding me on how to do that.
  6. Suggest the best practice for me.

Thanks a lot .

Hello Paul,

find below the output after running #print status.dict_

{ā€˜charger_modeā€™: ā€˜02ā€™, ā€˜error_modeā€™: ā€˜000ā€™, ā€˜pv_voltageā€™: 77, ā€˜timestampā€™: ā€˜2017-05-25 16:52:47ā€™, ā€˜daily_kwhā€™: 4.9, ā€˜daily_ahā€™: 89.0, ā€˜aux_modeā€™: ā€˜03ā€™, ā€˜pv_currentā€™: 1, ā€˜addressā€™: ā€˜Aā€™, ā€˜charge_currentā€™: 2.7, ā€˜bat_voltageā€™: 52.0}

Thanks.

Hello Everyone. Need help to fix this. Need somebody to come to my aid. Thanks as I wait your assistance.

@babaegun Have patience. None of the people who have helped you so far get paid for doing so. Everybody here except for 3 (Glyn, Trystan and Gwil) give their time, skills, knowledge and experience for free. And believe it or not, they also have a life away from OEM.

So please donā€™t always expect an immediate reply. While we try to answer everybody as soon as possible, it may be that the expert who can best answer you has other more important matters to attend to.

Thank you.

1 Like

Alright. Thanks.

@babaegun I had started looking at this yesterday morning but I ran out of time.

The example at the end of the matecom.txt file had engineering units and I could not figure out where they come from which is why I asked for the print status.__dict__ output, that was different and trying to work out why raised questions over what the `Value() functions were for.

I have since sought out the original repo

and found the value module is part of the the software you are using and since you have removed all the unit strings from those functions (hence the different status.__dict__ output) they are now effectively still adding ā€œā€ to some fields which is confusing and unnecessarily complex, so I have removed them and therefore any need for the ā€œvalueā€ module.

The 'address': 'A' would be an issue for emoncms as it only accepts numeric data so I have simply excluded it, if it is data you need then you will have to fashion a way of converting it to a number and send that instead.

So I have altered the matecom.txt to send a json request to emoncms so that the field names are used, otherwise all that code would be useless too, The data is actually received from the controller in a CSV format and the original script converted that CSV to a key:value json-like dict, applied some scaling and appended unit strings, You then removed the text from the strings (that continued to be appended) and converted the dict contents to CSV, to be written to file.

You will need to edit the script below for your own emoncms domain/IP and apikey, you can also change the node id, I only set it to 30 so as not to clash with any frequently used node ids. The node id is used in emoncms to group the inputs as a single source/device. thinking about it now you could probably even use `nodeid=ā€œOutbackCCā€ rather than a number.

This isnā€™t intended as a finished item, there is no error checking or handling, but nor is there in the original script, if a non-float slipped through to one of those lines containing calculations, it would crash, likewise if the serial connection wasnā€™t successful.

Currently if the request send fails it will crash, if it is a successful send but the response isnā€™t ā€œokā€ it will not try and resend, so there is plenty of scope for improvement. You could also break out the domain/IP, apikey and nodeid to variables etc.

# pyMate controller
# Author: Jared Sanson <[email protected]>
#
# Allows communication with an Outback Systems MATE controller panel,
# which provides diagnostic information of current charge state and power use
#
# Currently only supports the MX status page (from an Outback MX charge controller)
#
# NOTE: This is intended for communication with the MATE's RS232 port, not Outback's proprietary protocol.

import serial
#import csvmake
#import datetime
import time
import json
import urllib2

#from value import Value
#from time import gmtime, strftime

class MXStatusPacket(object):
    """
    Represents an MX status packet, containing useful information
    such as charge current and PV voltage.
    """
    def __init__(self, packet):
        fields = packet.split(',')
        # self.address = fields[0]
        # self.timestamp = strftime('%Y-%m-%d %H:%M:%S', gmtime())
        # fields[1] unused
        self.charge_current = (float(fields[2]) + (float(fields[6]) / 10.0))
        self.pv_current = fields[3]
        self.pv_voltage = fields[4]
        self.daily_kwh = (float(fields[5]) / 10.0)
        self.aux_mode = fields[7]
        self.error_mode = fields[8]
        self.charger_mode = fields[9]
        self.bat_voltage = (float(fields[10]) / 10)
        self.daily_ah = (float(fields[11]))
        # fields[12] unused

        chk_expected = int(fields[13])
        chk_actual = sum(ord(x)-48 for x in packet[:-4] if ord(x)>=48)
        if chk_expected != chk_actual:
            raise Exception("Checksum error in received packet")


class MateCom(object):
    """
    Interfaces with the MATE controller on a specific COM port.
    Must be a proper RS232 port with RTS/DTR pins.
    """
    def __init__(self, port, baudrate=19200):
        self.ser = serial.Serial(port, baudrate, timeout=2)

        # Provide power to the Mate controller
        self.ser.setDTR(True)
        self.ser.setRTS(False)

        self.ser.readline()

    def read_status(self):
        ln = self.ser.readline().strip()
        return MXStatusPacket(ln) if ln else None


if __name__ == "__main__":
    # Test
    mate = MateCom('COM3')
    status = mate.read_status()
    #print status.__dict__
    #print 'Writing to CSV'
    #csvmake.write_to_csv(status.__dict__)
    #print 'CSV Output written'
    #{'charger_mode': '00', 'error_mode': '000', 'pv_voltage': 18V, 'charge_current': 0.0A, 'daily_kwh': 7.8kWh, 'daily_ah': 144.0Ah, 'pv_current': 0A, 'address': 'A', 'aux_mode': '03', 'bat_voltage': 51.1V}
    url = 'https://emoncms.org/input/post.json?node=30&apikey=abc123&time=' + str(int(time.time())) + '&json=' + json.dumps(status.__dict__, separators=(',', ':')) + ''
    print(url)
    reply = urllib2.urlopen(url).read()
    print reply

I hope this works for you, it will at least get you going in the right direction.

1 Like

@pb66

I am extremely grateful for your assistance. I will try it out and give you a feedback.

Cheers.

@pb66 It worked flawlessly. I am grateful.

@pb66 @Bill.Thomson @RobertK @Robert.Wall

So guys, I feel it will be incomplete if i dont share the beautiful graphs from my solar setup. It is simply amazing. I am grateful to all of you. I now have real time graphs and a good insight into my 3kw Solar Setup.

2 Likes