Directly connecting to Optical Pulse Counter with RPi?

nellyt · 2016-11-19T07:30:33.274Z

Nearly …!
Fixing the indent on the first log message and all is OK.

Data arrives in local and remote EMONCMS

My function is


    def receiver(self, cargo):
    
        # Create a frame of data in "emonCMS format"
        f=[]
        try:
            f.append(float(cargo.timestamp))
            f.append(cargo.nodeid)
            for i in cargo.realdata:
                f.append(i)
            f.append(cargo.rssi)
            self._log.debug(str(cargo.uri) + " adding frame to buffer => "+ str(f))
        except:
            self._log.warning("Failed to create emonCMS frame " + str(f))

        # Append to bulk post buffer
        self.buffer.append(f)

Thanks for your help @pb66
I’ll try to tidy up this thread and summarise once complete.
Looks like I’ve still got some issues in correctly generating my feeds and calculating instant power from the accumulating pulses but I should do some more reading/browsing of forums before mentioning.

pb66 · 2016-11-19T12:36:31.535Z

nellyt:

Thanks for your help @pb66

Thanks for your help on reaching a solution. I’ve submitted a PR

github.com/openenergymonitor/emonhub

Bugfix to prevent emoncmshttp interfacer crashing other threads

openenergymonitor:emon-pi ← pb66:patch-2

opened 11:47AM - 19 Nov 16 UTC

pb66

+11 -9

See https://community.openenergymonitor.org/t/directly-connecting-to-optical-pul…se-counter-with-rpi/1294?u=pb66

If/when this gets accepted it will be automatically included in future emonPi updates, but the edits made to your local copy will prevent those changes being pulled in via git without first undoing those changes precisely or reseting to a “clean” copy of the file using git checkout -- src/interfacers/EmonHubEmoncmsHTTPInterfacer.py from the emonhub directory, which will permanently remove the edits.

You may be better off using git stash from within the emonhub directory before updating, to temporarily undo the edits, stashing them away and then if the changes are not pulled in during an update you can use git stash apply to reapply the edits.

glyn.hudson · 2016-11-20T18:57:07.363Z

Merged, thanks.

nellyt · 2017-06-21T09:14:39.781Z

An update from me, unrelated to the last dozen or so emails fixing a bug with Paul/pb66
I have just returned to this after a down time to try to address some unreliable pulse detections.
I have a standard optical sensor connected to my GPIO pins and using Paul’s script above.
I am running on a pi 2 Model B.
My sensor was on an extended cable about 4-5 m long.
I was receiving strange output from the script with multiple OFFs with no intervening ONs
My meter produces 500 impulses/kWh.
The ON pulse seems very short…

I suspected issues with pullups/pulldowns or cable length or noise on the edge transitions, but rejected those as the cause after some experiments.
I then came across this https://raspberrypi.stackexchange.com/questions/14105/how-does-python-gpio-bouncetime-parameter-work see the last few answers.

I believe I was detecting the RISING edge (to LED on) but by the time the handler polled the data it was back low again. This seems to explain why I was getting two OFF’s for every pulse and no ON.

I believe I have solved my problem by -

Changing my event detect from GPIO.BOTH to GPIO.FALLING
Adding time.sleep(0.1) at the start of the eventHandler
Changing if status to if not status in processpulse

I now have a nice clean power curve with no sporadic spikes or drop outs to zero. ie it is now working fine.

This is just FYI.
No assistance required, thanks

IrishLads · 2018-03-03T19:55:48.800Z

Neil & @pb66 ,

Could you please summarize the following?

How you connected the optical sensors to the GPIO pins?
How you start pulses.py (have you automated this for a restart, etc?)
How pulses.py interfaces with your configuration?

My intention is to have one pi with 1 sensor in my basement where my solar meter is, and 1 pi with 1 sensor in my garage where I can monitor my net meter which is outside.

I’m assuming if I start with the 1st pi in the basement, that the second pi can send data over mqtt or some other interface to that one?

Thanks

CelsoHenriques · 2018-03-04T12:13:38.181Z

IrishLads, see this thread:

Open collector Hardware

You can use the pulse input on the RJ45 connector. Using a device with an SO output has been mentioned a few times - I’m not sure about this forum but certainly in the ‘Archived’ section, and the emonPi input is exactly the same as on the emonTx (so if you find something about the emonTx, it also applies to the emonPi).

It doesn’t have all you need but it may have something usefull that might help you regarding using an rpi with pulse sensor. But as i dont understand programing neither eletronic, i can’t help you more.
Regards,
Celso

Robert.Wall · 2018-03-04T13:55:42.691Z

There’s a very recent response from Paul (pb66) citing the exact details here. But I can’t help with moving the data from one base to another. I’m guessing you need something like Nodered for that, but I know nothing about it, so I didn’t respond.

pb66 · 2018-03-04T14:11:45.835Z

IrishLads:

How you connected the optical sensors to the GPIO pins?

pb66:

their data pins are connected to gpio pins 13 and 15, and are powered by pins 9 and 17 so that a simple 1x5 header socket could be plugged directly on the gpio pins (9-17 odds).

Rather than chop the rj45’s off the pulse counters I use a surface mount double rj45 socket with just 4 wires between the sockets and the Pi gpio.

I used surface mount Ethernet sockets and a few “jumper wires” with one end in the “punch down” connectors of the Ethernet socket box and the other loose ends I used a spot of super glue to make a 5pin header from the 5 single pole connectors to make the connection(s) more manageable when plugging onto the gpio.

IrishLads:

How you start pulses.py (have you automated this for a restart, etc?)

Minimum setup for Solar + Grid Monitoring

when it was originally written it was started as one of several scripts and before systemd in Jessie, but a simple service unit could be used to start and manage ir as a daemon yes.

Another “quick and dirty” method is to use rc.local, but this is less reliable and being phased out in Raspbian Stretch.

IrishLads:

How pulses.py interfaces with your configuration?

Minimum setup for Solar + Grid Monitoring

That script posts to an emonhub socket,

I think the basic settings would be

[[Pulse]]
        Type = EmonHubSocketInterfacer
        [[[init_settings]]]
                port_nb = 50012
        [[[runtimesettings]]]
                pubchannels = ToEmonCMS,
                timestamped = True

added to the interfacers section of emonhub and possibly an entry in the nodes section too, But I don’t use the same version of emonhub so I can’t easily define that off-the top of my head. Once you have the optical counter mounter and physically connected, with just the above settings in emonhub.conf and pulses.py running, you should get data appearing in emonhub.log and from there it should be easy to add a nodes definition and see the data arrive in emoncms.

An example would be something like

[[18]]
    nodename = pulsecounter
    [[[rx]]]
        names = pulsecnt1, pulsecnt2
        scales = 1, 1
        units = Wh, Wh

but I’m guessing a bit here and it will partially depend on any mods you make to pulses.py, the spec of your meter pulses and your choice of labels.

With regard to using multiple Pi’s you have options, although unless you have reason to do otherwise, I would recommend just changing the node id in one of the pulses.py scripts and use the same target host for both scripts to deposit their data into the same emonhub socket on the one device, just edit the host = "localhost" line to point the slave Pi’s pulses.py at the master Pi’s emonhub socket, using a fixed ip if needed.

As long as they are on the same LAN you need no other setting than possibly opening the “50012” port in the UFW firewall if it’s enabled on the master Pi.

Robert.Wall:

There’s a very recent response from Paul (pb66) citing the exact details here

??? I thought I was loosing my marbles, I couldn’t remember writing anything along these lines recently. That post was written a year ago! But thanks for linking, I had forgotten about that post and there are some useful links about gpio etc in there.

Robert.Wall · 2018-03-04T14:20:43.517Z

pb66:

??? I thought I was loosing my marbles,

No - a victim of editing. I’d linked to a recent link, then edited out the link to the link, leaving the original target.

I think this might usefully become a ‘Learn’ topic?

pb66 · 2018-03-04T14:39:23.391Z

Robert.Wall:

I think this might usefully become a ‘Learn’ topic?

I think there are several topics in here that could be useful

using the emonhub socket interfacer
- setting one up, the options etc
- using one across the LAN (ie handling UFW rules and using hostnames/IP addresses and ports
- creating a simple script that posts data to an emonhub interfacer
Using the Pi’s GPIO
- Physical conns
- Addressing the gpio pins
- catching pulses
- hooking up surface mount Ethernet sockets to expand the Pi’s gpio to connect pulse counters and temp sensors with rj45 plugs
Writing a simple systemd unit to manage a bespoke script

but I wouldn’t know where to begin, that’s why I just tend to post stuff on the forum.

IrishLads · 2018-03-05T14:20:58.242Z

When i try to run pulses.py on a raspberry pi 3 running the latest emoncms image, I’m getting the following:

pi@emonpi(ro):~$ python pulses.py 
 Traceback (most recent call last):
 File "pulses.py", line 118, in <module>
  main()
 File "pulses.py", line 103, in main
  GPIO.add_event_detect(pulse_pin1, GPIO.BOTH, callback=eventHandler1, bouncetime=bounce)
 RuntimeError: Failed to add edge detection

Seems like the script is not able to communicate with the GPIO?

I get a similar result when using this gpio test script gpiotest.py

Thinking that this might be a case of the gpio not being accessible, I edited my conf to comment out the [[RFM2Pi]] interfacer since it looked like that was potentially tying up the gpio board. No change.

Anything I’m missing?

pb66 · 2018-03-05T21:51:08.086Z

Sorry, I’ve not seen that before. I have just done a quick internet search and most (but far from all) discussions around that error seem to point to permissions or incorrect pin refs.

I’m afraid it’s going to be a while before I can get to checking this and it was quite a long time ago I wrote that script, I put it into service and it’s worked ever since, I’ve not had any cause to look at it again since, so off-hand I can’t really think of anything to try beyond checking the pin references are ok, perhaps even try running the script as sudo???

IrishLads · 2018-03-05T22:26:54.787Z

Ah,yeah, permissions issue apparently - running it as sudo worked; will have to make sure I figure out how to run with elevated permissions in systemd / startup.

Now however I’m getting an socket error:

socket.error [Errno 111] Connection refused

I have opened up 50012 using the command in the script

sudo ufw allow 50012

and
sudo ufw -f enable

I’ll look around the forums to see if this is seen by others.

IrishLads · 2018-03-05T22:37:33.196Z

Nevermind, Ok, I had the:

[[Pulse]]
    Type = EmonHubSocketInterfacer
    [[[init_settings]]]
            port_nb = 50012
    [[[runtimesettings]]]
            pubchannels = ToEmonCMS,
            timestamped = True

In the wrong section. ( I had it at the end of the conf so it was under the nodes section)

Now when i touch pin 13, i’m getting a pulse detection and it is writing to the node 18 pulsecnt1 as expected

Now that I have that working, I need to set up a wH Accumulator, right?

pb66 · 2018-03-05T22:48:28.424Z

IrishLads:

Now that I have that working, I need to set up a wH Accumulator, right?

Personally I’ve never had much joy with the whaccumulator process, I use the “total pulse count to pulse increment” process.

If you search “total pulse count to pulse increment” here on the forum you can find more info, in particular the “Wh accumulator - strange behavior” thread discusses the issues you could have.

IrishLads · 2018-03-06T17:28:37.692Z

Thanks Paul,

If I want to sync to my current meter reading, say its 40644 kwH, would I set pulse_id =

{1:40644000,2:0}

My node is configured as:

[[18]]
nodename = pulsecounter
[[[rx]]]
    names = pulsecnt1, pulsecnt2
    scales = 1, 1
    units = Wh, Wh

I’m testing the script right now and it doesn’t always increment the pulse_id value:

1520357220.42 18 40664020 0
1520357277.69 18 40664020 0

pb66 · 2018-03-06T18:19:07.938Z

I can understand why you might want to do this, but if you must, then consider trying a virtual feed, the data storage in emoncms is in 32bit floats, this equates to around 7 digits of accuracy so the next increment to 40664020 0 will be 40664030 0 and depending on how your processing is set up, you may need to see that whole jump in one update, it maybe that every update that doesn’t reach that increment is ignored/lost, so effectively it might only be registering use that is over 10 Wh in any one interval, that could be a large portion of your usage.

IrishLads · 2018-03-06T18:46:40.827Z

I want to track the actual meter reading as closely as possible (i need to report it monthly manually to earn my solar renewable energy credits for generation).

Is there a better way to do this?

pb66 · 2018-03-06T22:56:57.173Z

IrishLads:

Is there a better way to do this?

pb66:

but if you must, then consider trying a virtual feed,

Basically the datatype you are using (phpfina or phptimeseries) cannot offer you the scope and resolution to do what you want. You need to store the change in pulsecount and then add the starting meter reading back in afterwards, either by manually calculating it each month or by setting up a virtual feeds so that you are recording the “change in pulse count” but displaying the “meter tracking” because the virtual feed adds the starting reading to every value when graphed or displayed.

Alternatively you could look at other storage engines, but there may be other consequences, eg the apps module only works with phpfina (without hacking).

TBH, if you need to be submitting “meter readings” then that’s what you should be doing “reading the meter”, I’m pretty familiar with all this stuff to the point it’s pretty predictable, but if I submit a meter reading, I read the meter.

If the meter isn’t electronically “readable” eg registers accessible via modbus or infrared etc, and it’s not easily accessed, maybe you should be looking at a remote camera or cctv so you can remotely take a peek at the meter once an month.

Mikeo · 2018-05-07T21:31:53.972Z

Hi Paul, I’m struggling to find more info on the connections between the optical sensors and the GPIO pins. I’ve made my own surface mount ethernet sockets with jumper wires connected to each punch down. Looking at the RJ45 on the optical sensor it looks like there are connections to Pins 1,2, 5 and 6.

Sorry to be a pain, could you please advise what GPIO you have connected 1,2,5 and 6 to?
I’ve searched through other posts and the archive and can’t seem to find any schematics or guides for this.

Thanks

Mike