emonTxV3 stops and requires reboot (emonhub)

yes, but there is a quick fix being tested as we speak that should be pushed to the repo tonight or tomorrow.

I wouldn’t bother with INFO level logging, it was never fully implemented, INFO doesn’t give you enough info to be useful but it is enough traffic to hide the warnings etc. Stick with DEBUG or WARNING, I always recommend DEBUG until a system is entirely stable as once you recognize a fault you need historic debugging info,

1 Like

Quick fix has been pushed to the emonhub emon-pi branch repo, thanks to @pb66
http://github.com/openenergymonitor/emonhub

and is available now via standard emonpi update.

For those not sending data to emoncms.org or another remote server, you can disable this interfacer fully either by commenting out he complete section of [[emoncmsorg]] as @bgrattan has done, or set senddata = 0, both in emonhub.conf.

Or just comment out the one Type = EmonHubEmoncmsHTTPInterfacer line (this like commenting/removing the whole interfacer entry, will also stop the internal queue forming)

Just adding a link in here to the EmonHub still dying? thread as it is the same runaway logging issue.

Hello,
I’m still having the same problem. It never really stopped after the changes in config but only happened every six weeks or so and I let it go. Now it has happened several times in the last week. The emonTxV3 stops transmitting to the pi which appears to be running fine. My equipment is:

House power monitoring system (USA) with the following:

RPI 3 loaded with flashed OEM software on 32 GB SD (latest updates)
emonTXV3 + AC/AC power, 
RFM69Pi_433 (purchased July 2017)
2 x SCT023R (200A:50MA) on 220 mains (North America 2 x 110)
emonTH V2 (purchased Oct 2017)
--------
weewx weather station (VP2) on Node 0

Only the emonTxV3 is affected.

config.txt (7.5 KB) emoncmsLog.txt (1.9 KB) updateLog.txt (15.0 KB)

When it’s working everything is fine and very accurate. Will be happy to send more info.
Thanks in advance.

Bob

Having looked down this thread, everything appears to be concentrating on emonHub and emonCMS. I appreciate that you’ve said the emonTx stops transmitting. Is your evidence for “Only the emonTxV3 is affected” that you have an emonTH and because you continue to receive that even when the emonTx is no longer received, therefore you are reasonably certain that the emonPi would be receiving the emonTx if it was transmitting?

EmonTx problems are often traced to power supply issues. You write that your mains is 110 V - 0 - 110 V. Is that correct because the normal North American domestic supply is 120 V - 0 - 120 V. Have you measured the mains voltage, and if so, what is it normally?

You write that the problem occurs every 6 weeks or so. Does it happen on the same day in the week (or often on the same day), or around the same time of day? Does it relate to anything in particular that happens in your house, or anything your neighbours do (those who are fed by the same transformer)?

Are you running the standard default sketch with no changes, other than calibration?

Do you have anything else (apart from a.c. adapter and c.t’s) connected to the emonTx?

Thanks for the quick reply, Robert.
Yes, the emonpi continues to run getting data from my weather station and the emonTH. So far it has never stopped.
The mains are +/- 120 volts and the emonTX is adjusted for the US transformer I have been using. I can’t detect any sort of pattern from the emonTX shutting down. The LED is off. When I cycle the power I get the normal LED and the flashing and it restarts sending data. It is about 3 feet from the pi. Everything is default settings except for the calibration adjustments in config. The whole thing is very accurate.

As you haven’t said anything about temperature sensors, I assume you don’t have any.

I’ve plugged the values for your US a.c. adapter into my LTSpice model, and that indicates that the power supply inside the emonTx should work, with worst-case components, down to 7.89 V rms out of the a.c. adapter, i.e. about 83 V line - neutral, again given a worst-case (-5%) adapter.

This doesn’t rule out the possibility that it’s a brown-out, but it’s about a 30% drop in the system voltage, so quite severe. If it’s easy, is it possible to feed it from another circuit as a test?

There remains to possibility that it’s a dry joint somewhere in the emonTx, possibly the RFM69 even (because if that doesn’t respond, the sketch hangs).
Have you inspected the emonTx - especially the soldering to the RFM69?

The only inputs to the emonTX3 are 4 current transformers. I have attached a vrms graph which shows when the shutdown happened, July 14-15. The house voltage fluctuates but I wouldn’t think it would be enough to shut it down. The circuit is connected directly to a panel breaker with about one foot of cable.

I haven’t examined the emonTX3 as I’m somewhat nervous about unplugging the high amp feeds. Once unplugged, should these feeds be shorted until they are re-plugged into the unit? Best way to short?

At one point, I had a second RPI with RFM69 running and it remained operational (emonTH and weather station recording) when the emonTX dropped out. That was a couple of years ago.

Is there some sort of handshaking going on between the emonTX and RFM69 or is it just a straight transmission without acknowledgement?

Sorry, these are probably silly questions but I’m quite a novice here. Thanks. Bob

emonTx3vrms.pdf (335.2 KB)

No handshake, the emonTx simply broadcasts the data and forgets all about it.

But, I’ve just checked the dates and it would appear you have an early emonTx V3.4., not the later V3.4.4 (I think yours will have “emonTx V3.4 Oct 2014” printed on the p.c.b.) In that case, I was wrong with my 7.89 V / 83 V, as that reflects a component change to improve the low voltage performance. The voltage yours needs (still given a worst-case set of components and transmitting on full power) is 11.32 V out of the a.c. adapter, or 118.9 V line-neutral. But unless you’ve changed the defaults in JeeLib, you don’t transmit on full power, and with the default power those voltages change to 7.97 V or 83.6 V. That’s almost the same, and well below the lowest voltage you recorded - but that doesn’t mean there wasn’t a big dip just as it transmitted the data, and that would be enough to crash the emonTx or the RFM69CW.

It’s very wise to be careful. I don’t know which particular c.t. you have, so I’ve got to say, don’t unplug them if they are on load. Most proper c.t’s have a protective bidirectional diode across the winding that should protect you and it when you unplug it. But I wouldn’t trust that. The most reliable way would be to isolate the circuits they are on while you look. Failing that, isolate, then you need to connect the tip and sleeve of the jack plug together, so a short length of bare copper wire twisted around the tip and again round the sleeve should suffice (but look at your c.t. first - some have a screw or a shorting link for this very purpose). When you’ve done that, you can safely liven up the circuit again. Reverse the procedure when you’ve finished.

Robert,
I’m using two SCT023R CTs on the mains. I will try shorting the plug with a lead clip or something like that as I don’t like to put my hands in the breaker box if it’s possible to avoid it. I don’t think these CTs have protective diodes.

I’m having my hot water boiler replaced next week so probably won’t try anything until that is finished, hopefully in several days.

Is there anything that needs to be updated on the emonTX3 that I should do while I have it opened up?

Thanks again for your help and I’ll probably get back to you in a week or so. Hope your weather is better in the UK as we will most likely hit 100 F. today and tomorrow.

Bob

Let’s not even go there. I understand why you’re tempted, but we don’t want to introduce any more variables into the problem.

There’s no suggestion that they have, so we must assume that they have not, therefore you must isolate, unplug and short the c.t. before energising the circuit again. And the same again when plugging back in.

In order to find out whether the problem is due to the voltage level, the emontx could also be powered via a USB power supply instead of the voltage reference transformer.

I am using the ac/ac transformer because it was recommended as the most accurate. With USB power what sort of calibration changes would I have to make? Thanks.

None.

But you should remove the link on the p.c.b. next to the battery holder.

The USB power supply is presumed to have a larger reservoir capacitor that will carry the emonTx over short brown-outs, whereas in order to avoid inaccuracies due to distorting the waveform from the a.c. adapter, the emonTx’s internal supply uses the smallest practical reservoir capacitor, which barely carries it over one mains cycle (of ours - for you at 60 Hz, the position is slightly less critical, but it still won’t carry it over 2 cycles).

Not strictly true, in fact the USB supply should be more accurate, because it won’t distort the measured wave shape. But the internal supply may well be quieter, meaning that the reading you get with no actual current flowing should be smaller.

With the emonTx located only 3 foot from the emonBase, I would be more than tempted to use a usb lead and usb to serial adapter to connect the emonTx directly to the emonBase. This would give you a better power (potentially more accurate data due to not powering via the AC:AC) and no risk of RF issues, regardless of whether the RFM module.is defective or not. The down side is if the cable and adaptor are on show it’s not the best looking option. But it the rfm module is terminally ill, a usb adapter is much cheaper than a new emonTx if you are not able to replace the rfm module yourself.

Paul, how difficult and expensive would it be to replace the RFM module on the emonTXv3? If it’s SMT components that have to be removed and replaced then I’ll pass.

I can try the direct USB from the Pi using the FTDI I purchased with the emonTX. I did notice in the notes that this method of power is not recommended except for programming.

I also have a Pi power supply I could use to directly connect it to 5V through the USB on board. However, I guess this would not solve anything if it’s the RFM.

Once the emonTX is connected to the Pi via USB port (/dev/ttyAMA0) does the communication begin automatically or do I need to change something in the config?
Also, once I begin using DC power (either USB or FTDI) and I cut the PCB link, do I leave the AC/AC power connected (I realize that it no longer powers the unit) or just remove it?

Sorry for all of these naïve questions but I’m out of my league here and want to be sure I am doing things correctly. Many thanks.
Bob

Answering the ones I can:

Very difficult - and you could easily damage your emonTx, and that becomes expensive. You need to simultaneously melt the solder on two rows of 7 connections - see the picture
https://wiki.openenergymonitor.org/index.php/File:Emontx_V3_4_top.jpg
(It’s the green piggy-back pcb top left.)

But it might establish that it is not the RFM.

Removing the link (it’s on a two-pin header, so nothing to physically cut - just pull the jumper of both pins and put it back on one only for safe keeping) removes the d.c. power. You still want the a.c. to measure the mains voltage.

You might need to edit the sketch so that the output is in an acceptable format for emonHub to parse. The present sketch does that by default, but without checking back and knowing the version you have, I can’t say for certain. If you download the present one, then if you’ve calibrated yours and have a copy of your running sketch from which you can copy the calibration and compile it, I’d do that.

I think it will then work - but Paul will no doubt confirm this.

As Robert says, it is extremely difficult to change the RFM module, the modules themselves are pretty cheap, around a fiver maybe? But at best, it’s a nerve wracking job that is a total gamble and could cost you a emonTx.

You are not the first person to say this. I’m pretty sure wherever you read that, it’s just the way it’s worded. What it really means is to run the emonTx with the USB programmer AND the AC:AC link in place should be avoided except for programming perhaps, with that link un-linked, you can run the emonTx via the programmer forever, it’s just a different 5vdc supply, albeit a fairly low-current one, but there’s plenty of a run of the mill emonTx.

I has assumed you were on a later FW sketch, if that is the case, you simply need to define a new interfacer in emonhub.conf, pretty simple to do from the emoncms config page.

Since you already have a programmer I think you should get that hooked up to see what FW version you have and check the serial output.

Do you have SSH access to the emonBase? It doesn’t matter if you don’t, you could alternatively download and use the Arduino IDE on a PC, your choice. The emonBase does have an “emonUpload” program on it that can check and update the emonTx to the latest FW. If you have made calibration changes (to the emonTx not to emonhub.conf) you probably already have a preferred way to talk to the emonTx via serial.

I would also be tempted to see if the rfm module continues transmitting for an extended time whilst you have got the USB programmer attached, for that you should probably remove the link, but just to get the FW version and possibly update, you would be fine with the link in place still. However it might not be relevant if you are going to use USB instead of RFM, but it might be useful to know for sure if it’s the PSU or RFM. As a matter of interest, do you log line voltage? Is there any pattern like the line voltage is always low when the emonTx stops transmitting for example? It should be easy to see the periods that the emonTX wasn’t sending data as the graph will show a straight line between the last good data point and the first datapoint after a restart.

Paul,
It’s possible there is a low voltage cause. I have attached a graph which shows July 14 and 20 drop outs, although it dips even lower in other places without shutting down.
vrmsgraph.pdf (383.9 KB)

I have loaded emonUpload on my Pi and will try reflashing the emonTXv3. My CT calibrations were made in the conf file so I’m not sure how to do it otherwise. Will these “scales” be affected (adjusted for SCT023R) by the new firmware in the emonTX?

Thanks and also thanks to Robert for his comments earlier.

Bob