Maximum number of pulses

Robert.Wall · 29 June 2020 20:09

What confused me, which is why I spelled out what the two methods available in the emonTx do, was

when it was obvious that you were not using the pulse count of the emonTx running emonLibCM, instead comparing the analogue energy measured by the emonTx against the pulse input of the RPi.

It looks as if the rate limit will be a little under 5 pulses per second. That’s a good finding and we need to remember it in case another user comes across the problem later. It should really go in the documentation.

dBC · 29 June 2020 22:44

A very good finding indeed.

Or change the sleep value. I’m no python guru but Brian’s findings suggest you can only have one pending event while sleeping. I rate limit my pulse counting to ~12Hz primarily as part of the debouncing/deglitching. 5Hz seems a bit restrictive. I think that works out at 18kW doesn’t it? A 3-phase Tesla charger can pump 22kW. Does anyone know the history of the 200 msec sleep?

Robert.Wall · 29 June 2020 23:18

That’s pre-supposing one pulse = 1 Wh. It’s a common but not standard value, the Optical pulse sensor page in Resources gives 4000 pulses per kWh as the “fastest”, and I think that isn’t the fastest I’ve seen reported/mentioned in the forum. IIRC, it was someone in Continental Europe.

It seemed a good idea at the time?
As I haven’t the faintest clue about how the Pi and emonHub works, I couldn’t hazard a better guess. But it looks to me as if rather more of the pulse handling needs to be done as a low level service inside the Pi itself and rather less in emonHub and the interfacer.

Reading the slope off the graphs, I reckon it was rate limiting at 3.9 kW (first graph) or 3.5 kW (second graph) so that’s not far off 1 per second, not 5.
(That seemed so ridiculous that as a check, I looked at the slope of the emonTx with emonLibCM - it came to about 12 kW, which is close enough to Brian’s 11.4 kW.)

At that rate, I’d suggest that the direct pulse input is unusable except for a 16 A PV infeed, if the meter pulse equates to 1 Wh, even then it might rate-limit at or near maximum production.

dBC · 30 June 2020 00:24

Now that you mention it, the 3-phase meter at that house with the Tesla only pulses at 250 per kWh. Maybe they tune the rate to ensure maximum power doesn’t have it flashing insanely fast.

Bill.Thomson · 30 June 2020 02:15

I’m no Python expert myself, but I do know that Pyhton is a multi-thread capable language.
The module Advanced Python Scheduler I use to control various events can have more than one
pending event. Maybe multi-threading is the way to go?

dBC · 30 June 2020 03:13

In this old thread, @danbates says “time.sleep() could be a millisecond, one second, 5000 seconds, makes no difference, as long as the program is running the interrupts will happen and callbacks are initiated immediately.”

Dan, does that include the case where multiple interrupts arrive during the one sleep() call?

Bill.Thomson · 30 June 2020 03:19

That sounds about right. Here’s a snippet from APS that does what you’re speaking of:
(that is, I think it does, but I may be off on a tangent too. As I mentioned earlier, I’m no
Python expert, I just dabble with it)

def main():
    sched.add_cron_job(read_meter, second='1,6,11,16,21,26,31,36,41,46,51,56')
    sched.add_cron_job(read_water_htr, second='3,8,13,18,23,28,33,38,43,48,53,58')
    sched.add_cron_job(get_garage_temp, minute='*/1', second='0')
    sched.add_cron_job(send_to_pvo, minute='*/5', second='2')
    sched.add_cron_job(wrt_daily_data, hour='23', minute='59', second='52')
    sched.add_cron_job(wrt_monthly_separator, day='1', hour='0', minute='0', second='3')

    while True:
       time.sleep(1)

if __name__ == "__main__":
    main()

Without the sleep construct, the script runs once then exits.

dBC · 30 June 2020 03:29

add_cron_job makes me think each of those is a completely separate unix process launched by the cron daemon.

Bill.Thomson · 30 June 2020 03:32

It appears that way, but it’s just a name. Advanced Python Scheduler has three different ways
to schedule a job. the add_cron_job syntax is very similar to cron syntax, hence the name.
Job scheduling is handled entirely by APS. i.e. cron is not used.

borpin · 30 June 2020 06:23

I’m pretty sure the callback on the interrupt will fire even in the sleep mode.

What it needs is to limit how often the read() sends data to the main process. I’m sure I’ve seen one of the other interfacers do that. It is not handling the interrupt that takes the time, it is the rest of the processing. This would simply mean that instead of reporting every pulse, it would report the pulses in the last n seconds.

This was part of the reason for introducing (not merged yet) an MQTT JSON option whereby a single MQTT publish is done for every frame - the publish can be a significant block of the processing time. I have also seen emoncms drop MQTT messages if a value is published too quickly onto the same topic - this only really matters if the data frame is timestamped.

yes but if it is too short, emonhub takes up too much processing time (there is a thread somewhere on that) hence the sleep. It should possibly be refactored into a full callback rather than a polling architecture perhaps, but that is way beyond my Python!

danbates · 30 June 2020 08:22

As already hinted at by @Bill.Thomson, any number of callbacks of the same callback can happen any time during ‘sleep’.
Absolutely right @borpin

borpin · 30 June 2020 08:23

I think we are agreeing…

danbates · 30 June 2020 08:25

Coffee must’ve just kicked in

danbates · 30 June 2020 08:37

I did that scripto for turning the time between pulses into a Wattage… There was a problem with fast pulses at the time. I cranked up the priority of the process and that helped a lot. It was surprisingly accurate given rPi runs a non-RTOS.
I would have added a decay to the Wattage ‘reading’ too, so if there was suddenly a drop in pulse freq. the value in emonCMS wouldn’t be stuck at some kWs.
But yeah, if fast pulses are a problem I think you’re idea Brian of holding back the pulse counts in an accumulating variable somewhere before all the MQTT mess is a good one.

danbates · 30 June 2020 09:32

If you want to get an idea of how fast a pulse rate the input can handle, jump the gpio to another pin set as an output and send it high and low at different rates.
I can’t remember how fast it was that it started dropping triggers but it was fast, fast enough, I just sent a screenshot to T&G one day with the results, about 2 years ago that was.

borpin · 30 June 2020 10:12

That is what I have done. Not going full tilt but seems to work OK.

2020-06-30 10:43:08,444 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 94
2020-06-30 10:43:09,446 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 95
2020-06-30 10:43:10,020 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 96
2020-06-30 10:43:10,070 DEBUG    pulse2     75 NEW FRAME :
2020-06-30 10:43:10,074 DEBUG    pulse2     75 Timestamp : 1593510190.0
2020-06-30 10:43:10,077 DEBUG    pulse2     75 From Node : 3
2020-06-30 10:43:10,081 DEBUG    pulse2     75    Values : [96]
2020-06-30 10:43:10,084 DEBUG    pulse2     75 Sent to channel(start)' : ToEmonCMS
2020-06-30 10:43:10,087 DEBUG    pulse2     75 Sent to channel(end)' : ToEmonCMS
2020-06-30 10:43:10,277 DEBUG    MQTT       Publishing: emon/3/Pulse 96
2020-06-30 10:43:10,336 DEBUG    MQTT3      Publishing: emon/3/Pulse 96
2020-06-30 10:43:10,340 DEBUG    MQTT2      Publishing: emon/3/Pulse 96
2020-06-30 10:43:10,577 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 97
2020-06-30 10:43:11,131 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 98
2020-06-30 10:43:11,689 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 99
2020-06-30 10:43:12,017 DEBUG    pulse2     76 NEW FRAME :
2020-06-30 10:43:12,020 DEBUG    pulse2     76 Timestamp : 1593510192.0
2020-06-30 10:43:12,023 DEBUG    pulse2     76 From Node : 3
2020-06-30 10:43:12,026 DEBUG    pulse2     76    Values : [99]
2020-06-30 10:43:12,029 DEBUG    pulse2     76 Sent to channel(start)' : ToEmonCMS
2020-06-30 10:43:12,033 DEBUG    pulse2     76 Sent to channel(end)' : ToEmonCMS
2020-06-30 10:43:12,118 DEBUG    MQTT       Publishing: emon/3/Pulse 99
2020-06-30 10:43:12,191 DEBUG    MQTT3      Publishing: emon/3/Pulse 99
2020-06-30 10:43:12,194 DEBUG    MQTT2      Publishing: emon/3/Pulse 99
2020-06-30 10:43:12,247 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 100
2020-06-30 10:43:12,801 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 101
2020-06-30 10:43:13,359 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 102
2020-06-30 10:43:13,917 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 103
2020-06-30 10:43:14,056 DEBUG    pulse2     77 NEW FRAME :
2020-06-30 10:43:14,069 DEBUG    pulse2     77 Timestamp : 1593510194.0
2020-06-30 10:43:14,081 DEBUG    pulse2     77 From Node : 3
2020-06-30 10:43:14,085 DEBUG    pulse2     77    Values : [103]
2020-06-30 10:43:14,107 DEBUG    pulse2     77 Sent to channel(start)' : ToEmonCMS
2020-06-30 10:43:14,111 DEBUG    pulse2     77 Sent to channel(end)' : ToEmonCMS
2020-06-30 10:43:14,228 DEBUG    MQTT3      Publishing: emon/3/Pulse 103
2020-06-30 10:43:14,232 DEBUG    MQTT2      Publishing: emon/3/Pulse 103
2020-06-30 10:43:14,258 DEBUG    MQTT       Publishing: emon/3/Pulse 103
2020-06-30 10:43:14,471 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 104
2020-06-30 10:43:15,031 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 105
2020-06-30 10:43:15,589 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 106
2020-06-30 10:43:16,045 DEBUG    pulse2     78 NEW FRAME :
2020-06-30 10:43:16,048 DEBUG    pulse2     78 Timestamp : 1593510196.0
2020-06-30 10:43:16,052 DEBUG    pulse2     78 From Node : 3
2020-06-30 10:43:16,057 DEBUG    pulse2     78    Values : [106]
2020-06-30 10:43:16,062 DEBUG    pulse2     78 Sent to channel(start)' : ToEmonCMS
2020-06-30 10:43:16,067 DEBUG    pulse2     78 Sent to channel(end)' : ToEmonCMS
2020-06-30 10:43:16,148 DEBUG    Dummy-5    pulse2 : Pulse Channel 15 pulse: 107
2020-06-30 10:43:16,165 DEBUG    MQTT2      Publishing: emon/3/Pulse 106
2020-06-30 10:43:16,170 DEBUG    MQTT3      Publishing: emon/3/Pulse 106
2020-06-30 10:43:16,185 DEBUG    MQTT       Publishing: emon/3/Pulse 106

In the init, define the start point

self.last_time = time.time()

In the read check for when last called - fixed time of 2 seconds.

        if not self.pulse_received:
            return False
        elif self.last_time + 2 > time.time():
            return False

        self.pulse_received = False

        self.last_time += 2

Yes, I know it needs tidying up.

There is also a race condition between setting the flag to false and reading the number of pulses. I think at worst, the ‘read’ would return the same pulse number twice.

Instead of setting a received flag would it be better to keep a ‘last count’ and compare the new count against the last count?

There is also the point that the accumulator does not need to be a dict. We started off with reading several pins in the one interfacer but decided to move to multiple interfacers one for each pin if required.

danbates · 30 June 2020 10:26

nah, should fit into the rest fine as it is

borpin · 30 June 2020 20:55

I’ve updated the PR if anyone is interested in testing it.

danbates · 1 July 2020 08:26

I’ve had a bash and had the following error:

2020-07-01 09:24:08,086 INFO     MainThread Creating EmonHubPulseCounterInterfacer 'pulse2' 
2020-07-01 09:24:08,087 INFO     MainThread pulse2 : Pulse pin set to: 15
2020-07-01 09:24:08,088 ERROR    MainThread Unable to create 'pulse2' interfacer: No access to /dev/mem.  Try running as root!

borpin · 1 July 2020 08:28

Interesting, what setup? Mine is a vanilla Raspbian with the emonhub repo cloned and installed using the emonhub install script.