Missing threading.Lock() functionality in emonhub?

emonHub
1

Hi, am have been using EmonHub for some smaller projects before but now I am planning a bigger project involving a number of interfaces, including multiple modbus RTU devices.

Before I want to commit to using EmonHub I just wanted to check something. I realise that Emonhub employs threading for the independent running of interfacers but does not make use of the threading.Lock() functionality, which has been designed to temporarily block threads from performing certain tasks while one thread is performing a critical activity such as reading data from a shared physical device. Why is that? Is EmonHub just catching the errors when two interfacers are trying to access the same physical device or am I missing something here?

From looking at the emonhub.conf file structure it seems I should be able to create several interfacers of the same type but different names, and expect those to work independently. My main worry is how to avoid them clashing with each other by repeatedly accessing the same physical Modbus infrastructure.

Any advice will be appreciated. Thanks!
Frank

2

Hi @pb66, I realise that my previous post may not have been specific enough. Here is my proposal for adding the threading.Lock() to the emonhub in order to avoid device access collisions:

  1. create the lock globally just before hub itself is created, e.g. TLock = threading.Lock()
  2. then pass the lock to each of the interfacers upon their creation as an argument.
  3. invoke the lock every time an interfacer accesses external resources and devices by using self.lock.acquire() and self.lock.release().

What do you think? Would that work? Is there a better way of doing it? Or is it not really required because another mechanism is taking care of avoiding those collisions?

3

Hi Frank, I did see your post and gave it some thought, but I have very little, if any, input on the “emonpi” variant of emonhub that is included on the emonSD image and since most users tend to opt for the pre-built image there seems little point in maintaining a competing version.

You raise a valid point about device access but I have not yet formed an opinion about the suggested changes.

There is already a threading concern regarding devices in the emonpi variant of emonhub, it’s a loose end that I identified in my “experimental” version and highlighted when the emonpi variant was in development, but development was out of my hands at that time and it took on a different direction.

Currently the settings for all interfacers are all managed by the main thread, so, for example if the jee interfacer is receiving from the serial port and a setting is changed simultaneously there could theoretically be 2 threads trying to access the same serial port at the same time from within emonhub. This I was going to resolve by changing the settings code to queue the writes to the serial port via the interfacer “routing” queues rather than writing directly to the port. However my routing queues were removed in favour of a pubsub library implementation that wasn’t thread safe, that has since been removed but the routing and queing is still not as it was intended. Plus there have been 2 independent PR’s for writing to the jee interfacers serial port (emonpi or rfm2pi) one of which has been merged, neither of which accommodate the settings.

However, I believe your target issue is possibly when the user configures the same interface more than once (correct?) if so I can see the potential issue, but I would have expected this to only be in error (correct?) as I cannot offhand think of why a user would duplicate interfacers unless using multiple devices, eg it is not unheard of to use an rfm2pi and jeelink on the same RPi, they are essentially a GPIO and USB versions of the same device, and this is perfectly acceptable and works well.

How do you propose the lock would work? Would it lock all instances of a certain type of interfacer or lock just the interfacers targeting the exact same device address/id?

Assuming the known threading issue for settings is resolved, is there a real need for the lock? (just questioning out loud, I’m not inferring there isn’t a need) if a user accidentally duplicates a interfacer config exactly, the config will not load due to the need for unique section headers (interfacer names) and if the name was different the setup would probably fail on the second device if the first already has control (eg serial ports), but yes it is better to be safe than sorry, if it doesn’t impact speed or usability too much, ie the point of threading the interfacers is so that no devices were sat waiting for another device to respond and finish before it could then do it’s thing. locking for all/any external device each time any is accessed might cause blocking, where as per device id/address might be complex (maybe not?).

Ideally, I believe that should never happen or be needed, or do you have other thoughts?

IMO, there should be a single interfacer for any one Modbus network bus, to which there can be multiple devices defined and accessed, I’ve no idea if the current Modbus implementation works that way, I have been working on my own Modbus “interfacer” script that does work exactly that way, but it is unlikely to manifest as an emonhub interfacer as I do not use the “emonpi” variant and I no longer develop for the original version, so it is likely to be an independent emonhubesque script that posts to an emonhub socket interfacer for now.

This is at the very least a good point to raise and I will link to it from the emonhub development discussion so that if/when I get back into emonhub dev, it will be easy to find and review at that time.

4

Hi Paul,

thanks for sharing your comprehensive thoughts.

I believe the conflict issue is the same for all interfacers sharing any serial communication ports RS232 or RS485 (Modbus RTU). In my case I am using around 4-5 Modbus RTU devices connected to the same port and hardware adaptor (USB to RS485), with each of these being operated through a separate interfacer instance. Some of these devices are identical (e.g. several Modbus kWh meters) and are using the same interfacer class while others are of a different type and have their own class. I created my own (rather simple) software based on threading and using a simple lock as explained in my previous post. The lock is “acquired” by the interfacer instance just before serial port access and “released” just after. From the moment the interfacer/thread that has acquired the lock all execution of other threads/interfacers is paused until the lock is released. It works very well, but my code lacks many features that EmonHub has and therefore I am contemplating switching to EmonHub.

It is my understanding that all interfacers are acting on the same logical level with the same level of priority. Creating a queue-based system to buffer the serial port interaction as you seem to be proposing seems a bit complicated to me and introduces a two-tiered structure. In contrast to that the threading.Lock() functionality seems much simpler to use. However, I am really no expert in these matters, I just wanted to see how the current code takes care of those conflicts.

Another, related point that I need to check with you is where the frequency of interfacers performing their data collection work can be set. My interfacers need to be run at different intervals.

IMO, there should be a single interfacer for any one Modbus network bus, to which there can be multiple devices defined and accessed, I’ve no idea if the current Modbus implementation works that way, I have been working on my own Modbus “interfacer” script that does work exactly that way, but it is unlikely to manifest as an emonhub interfacer as I do not use the “emonpi” variant and I no longer develop for the original version, so it is likely to be an independent emonhubesque script that posts to an emonhub socket interfacer for now.

Sounds interesting and I am intrigued if this could work for me. The thing is I not only read data from the interfacers, I also control devices through interfacers by reading control data from the database and using the data as setpoints for the controls. Are you aware of any EmonHub interfacers doing that also?

Thanks again for your consideration.