emonTxV3 stops and requires reboot (emonhub)

TrystanLea · 2017-10-26T18:58:46.218Z

Sorry setting the logging to info wont help of course as info messages are still logged when debug level is set

I’ve got a preliminary fix for the issue above so far that stops the creation of the buffered data array when the apikey is not set but items are still added to the queue which Im still working on https://github.com/openenergymonitor/emonhub/compare/emon-pi...openenergymonitor:fix-emoncmsorg-buffering?expand=1

pb66 · 2017-10-26T20:49:48.130Z

TrystanLea:

not logging the adding of the frame to the buffer, im not sure that its that useful anyway

It is useful when the data isn’t being sent as it’s the only record that something happened with the data. The one log message could also include a total buffer size to reduce the number of separate entries.

The problem lies with the interfacer working directly on the data in the queue so each failed send results in every queued frame being added to the buffer again. All the queue’s should, on a healthy system be at or close to zero most of the time. The data must be consumed by the interfacer, either sent, buffered (elsewhere) or discarded.

TrystanLea:

will buffer data indefinitely

Yep, that is a big potential issue, I think the code should be written in a way to ensure the queues cannot grow and as a safety thing we add an overall max size based predominantly on how big we could go without breaking the system or loosing performance and then introduce a very low warning level so that even the lightest backup in the queues will result in a warning level log, but there is still plenty of headroom. These are just safety nets though, the intention must be to aim for near zero queue lengths.

bgrattan · 2017-10-26T20:55:34.889Z

If I’m following correctly, the easiest fix for me is to comment out the complete section of [[emoncmsorg]]. By doing this, the DEBUG logging quieted down quite a bit and, every 10 seconds, the INFO level gives me this:

2017-10-26 16:51:51,185 INFO MQTT Publishing: emon/emontx3/rssi -27
2017-10-26 16:51:51,187 INFO MQTT Publishing: emonhub/rx/8/values 525.8496,562.1888,154,2,121.7311488,300,300,300,300,300,300,0
2017-10-26 16:51:51,189 INFO MQTT Publishing: emonhub/rx/8/rssi -27

I’ll see if things are more stable from this point. Much appreciated to everyone for the help.
Bob

pb66 · 2017-10-26T21:10:04.524Z

bgrattan:

If I’m following correctly, the easiest fix for me is to comment out the complete section of [[emoncmsorg]].

yes, but there is a quick fix being tested as we speak that should be pushed to the repo tonight or tomorrow.

bgrattan:

the INFO level gives me this

I wouldn’t bother with INFO level logging, it was never fully implemented, INFO doesn’t give you enough info to be useful but it is enough traffic to hide the warnings etc. Stick with DEBUG or WARNING, I always recommend DEBUG until a system is entirely stable as once you recognize a fault you need historic debugging info,

TrystanLea · 2017-10-26T21:42:03.506Z

Quick fix has been pushed to the emonhub emon-pi branch repo, thanks to @pb66
http://github.com/openenergymonitor/emonhub

and is available now via standard emonpi update.

For those not sending data to emoncms.org or another remote server, you can disable this interfacer fully either by commenting out he complete section of [[emoncmsorg]] as @bgrattan has done, or set senddata = 0, both in emonhub.conf.

pb66 · 2017-10-26T21:46:43.545Z

TrystanLea:

For those not sending data to emoncms.org or another remote server, you can disable this interfacer fully either by commenting out he complete section of [[emoncmsorg]] as @bgrattan has done, or set senddata = 0, both in emonhub.conf.

Or just comment out the one Type = EmonHubEmoncmsHTTPInterfacer line (this like commenting/removing the whole interfacer entry, will also stop the internal queue forming)

pb66 · 2017-10-27T21:59:25.226Z

Just adding a link in here to the EmonHub still dying? thread as it is the same runaway logging issue.

bgrattan · 2019-07-18T19:55:41.332Z

Hello,
I’m still having the same problem. It never really stopped after the changes in config but only happened every six weeks or so and I let it go. Now it has happened several times in the last week. The emonTxV3 stops transmitting to the pi which appears to be running fine. My equipment is:

House power monitoring system (USA) with the following:

RPI 3 loaded with flashed OEM software on 32 GB SD (latest updates)
emonTXV3 + AC/AC power, 
RFM69Pi_433 (purchased July 2017)
2 x SCT023R (200A:50MA) on 220 mains (North America 2 x 110)
emonTH V2 (purchased Oct 2017)
--------
weewx weather station (VP2) on Node 0

Only the emonTxV3 is affected.

config.txt (7.5 KB) emoncmsLog.txt (1.9 KB) updateLog.txt (15.0 KB)

When it’s working everything is fine and very accurate. Will be happy to send more info.
Thanks in advance.

Bob

Robert.Wall · 2019-07-18T21:29:47.221Z

bgrattan:

The emonTxV3 stops transmitting to the pi which appears to be running fine.

Having looked down this thread, everything appears to be concentrating on emonHub and emonCMS. I appreciate that you’ve said the emonTx stops transmitting. Is your evidence for “Only the emonTxV3 is affected” that you have an emonTH and because you continue to receive that even when the emonTx is no longer received, therefore you are reasonably certain that the emonPi would be receiving the emonTx if it was transmitting?

EmonTx problems are often traced to power supply issues. You write that your mains is 110 V - 0 - 110 V. Is that correct because the normal North American domestic supply is 120 V - 0 - 120 V. Have you measured the mains voltage, and if so, what is it normally?

You write that the problem occurs every 6 weeks or so. Does it happen on the same day in the week (or often on the same day), or around the same time of day? Does it relate to anything in particular that happens in your house, or anything your neighbours do (those who are fed by the same transformer)?

Are you running the standard default sketch with no changes, other than calibration?

Do you have anything else (apart from a.c. adapter and c.t’s) connected to the emonTx?

bgrattan · 2019-07-18T21:49:04.153Z

Thanks for the quick reply, Robert.
Yes, the emonpi continues to run getting data from my weather station and the emonTH. So far it has never stopped.
The mains are +/- 120 volts and the emonTX is adjusted for the US transformer I have been using. I can’t detect any sort of pattern from the emonTX shutting down. The LED is off. When I cycle the power I get the normal LED and the flashing and it restarts sending data. It is about 3 feet from the pi. Everything is default settings except for the calibration adjustments in config. The whole thing is very accurate.

Robert.Wall · 2019-07-19T12:03:20.902Z

As you haven’t said anything about temperature sensors, I assume you don’t have any.

I’ve plugged the values for your US a.c. adapter into my LTSpice model, and that indicates that the power supply inside the emonTx should work, with worst-case components, down to 7.89 V rms out of the a.c. adapter, i.e. about 83 V line - neutral, again given a worst-case (-5%) adapter.

This doesn’t rule out the possibility that it’s a brown-out, but it’s about a 30% drop in the system voltage, so quite severe. If it’s easy, is it possible to feed it from another circuit as a test?

There remains to possibility that it’s a dry joint somewhere in the emonTx, possibly the RFM69 even (because if that doesn’t respond, the sketch hangs).
Have you inspected the emonTx - especially the soldering to the RFM69?

bgrattan · 2019-07-19T14:33:44.074Z

The only inputs to the emonTX3 are 4 current transformers. I have attached a vrms graph which shows when the shutdown happened, July 14-15. The house voltage fluctuates but I wouldn’t think it would be enough to shut it down. The circuit is connected directly to a panel breaker with about one foot of cable.

I haven’t examined the emonTX3 as I’m somewhat nervous about unplugging the high amp feeds. Once unplugged, should these feeds be shorted until they are re-plugged into the unit? Best way to short?

At one point, I had a second RPI with RFM69 running and it remained operational (emonTH and weather station recording) when the emonTX dropped out. That was a couple of years ago.

Is there some sort of handshaking going on between the emonTX and RFM69 or is it just a straight transmission without acknowledgement?

Sorry, these are probably silly questions but I’m quite a novice here. Thanks. Bob

emonTx3vrms.pdf (335.2 KB)

Robert.Wall · 2019-07-19T15:48:41.234Z

bgrattan:

Is there some sort of handshaking going on between the emonTX and RFM69 or is it just a straight transmission without acknowledgement?

No handshake, the emonTx simply broadcasts the data and forgets all about it.

But, I’ve just checked the dates and it would appear you have an early emonTx V3.4., not the later V3.4.4 (I think yours will have “emonTx V3.4 Oct 2014” printed on the p.c.b.) In that case, I was wrong with my 7.89 V / 83 V, as that reflects a component change to improve the low voltage performance. The voltage yours needs (still given a worst-case set of components and transmitting on full power) is 11.32 V out of the a.c. adapter, or 118.9 V line-neutral. But unless you’ve changed the defaults in JeeLib, you don’t transmit on full power, and with the default power those voltages change to 7.97 V or 83.6 V. That’s almost the same, and well below the lowest voltage you recorded - but that doesn’t mean there wasn’t a big dip just as it transmitted the data, and that would be enough to crash the emonTx or the RFM69CW.

bgrattan:

I haven’t examined the emonTX3 as I’m somewhat nervous about unplugging the high amp feeds. Once unplugged, should these feeds be shorted until they are re-plugged into the unit? Best way to short?

It’s very wise to be careful. I don’t know which particular c.t. you have, so I’ve got to say, don’t unplug them if they are on load. Most proper c.t’s have a protective bidirectional diode across the winding that should protect you and it when you unplug it. But I wouldn’t trust that. The most reliable way would be to isolate the circuits they are on while you look. Failing that, isolate, then you need to connect the tip and sleeve of the jack plug together, so a short length of bare copper wire twisted around the tip and again round the sleeve should suffice (but look at your c.t. first - some have a screw or a shorting link for this very purpose). When you’ve done that, you can safely liven up the circuit again. Reverse the procedure when you’ve finished.

bgrattan · 2019-07-20T20:09:32.456Z

Robert,
I’m using two SCT023R CTs on the mains. I will try shorting the plug with a lead clip or something like that as I don’t like to put my hands in the breaker box if it’s possible to avoid it. I don’t think these CTs have protective diodes.

I’m having my hot water boiler replaced next week so probably won’t try anything until that is finished, hopefully in several days.

Is there anything that needs to be updated on the emonTX3 that I should do while I have it opened up?

Thanks again for your help and I’ll probably get back to you in a week or so. Hope your weather is better in the UK as we will most likely hit 100 F. today and tomorrow.

Bob

Robert.Wall · 2019-07-20T20:39:35.992Z

bgrattan:

Is there anything that needs to be updated on the emonTX3 that I should do while I have it opened up?

Let’s not even go there. I understand why you’re tempted, but we don’t want to introduce any more variables into the problem.

bgrattan:

I don’t think these CTs have protective diodes.

There’s no suggestion that they have, so we must assume that they have not, therefore you must isolate, unplug and short the c.t. before energising the circuit again. And the same again when plugging back in.

Simsala · 2019-07-20T20:45:30.503Z

In order to find out whether the problem is due to the voltage level, the emontx could also be powered via a USB power supply instead of the voltage reference transformer.

bgrattan · 2019-07-20T21:27:34.111Z

I am using the ac/ac transformer because it was recommended as the most accurate. With USB power what sort of calibration changes would I have to make? Thanks.

Robert.Wall · 2019-07-20T23:12:48.392Z

bgrattan:

With USB power what sort of calibration changes would I have to make?

None.

But you should remove the link on the p.c.b. next to the battery holder.

The USB power supply is presumed to have a larger reservoir capacitor that will carry the emonTx over short brown-outs, whereas in order to avoid inaccuracies due to distorting the waveform from the a.c. adapter, the emonTx’s internal supply uses the smallest practical reservoir capacitor, which barely carries it over one mains cycle (of ours - for you at 60 Hz, the position is slightly less critical, but it still won’t carry it over 2 cycles).

bgrattan:

it was recommended as the most accurate

Not strictly true, in fact the USB supply should be more accurate, because it won’t distort the measured wave shape. But the internal supply may well be quieter, meaning that the reading you get with no actual current flowing should be smaller.

pb66 · 2019-07-21T08:43:21.439Z

With the emonTx located only 3 foot from the emonBase, I would be more than tempted to use a usb lead and usb to serial adapter to connect the emonTx directly to the emonBase. This would give you a better power (potentially more accurate data due to not powering via the AC:AC) and no risk of RF issues, regardless of whether the RFM module.is defective or not. The down side is if the cable and adaptor are on show it’s not the best looking option. But it the rfm module is terminally ill, a usb adapter is much cheaper than a new emonTx if you are not able to replace the rfm module yourself.

bgrattan · 2019-07-21T16:12:52.282Z

Paul, how difficult and expensive would it be to replace the RFM module on the emonTXv3? If it’s SMT components that have to be removed and replaced then I’ll pass.

I can try the direct USB from the Pi using the FTDI I purchased with the emonTX. I did notice in the notes that this method of power is not recommended except for programming.

I also have a Pi power supply I could use to directly connect it to 5V through the USB on board. However, I guess this would not solve anything if it’s the RFM.

Once the emonTX is connected to the Pi via USB port (/dev/ttyAMA0) does the communication begin automatically or do I need to change something in the config?
Also, once I begin using DC power (either USB or FTDI) and I cut the PCB link, do I leave the AC/AC power connected (I realize that it no longer powers the unit) or just remove it?

Sorry for all of these naïve questions but I’m out of my league here and want to be sure I am doing things correctly. Many thanks.
Bob