IotaWatt Update - nearly complete

Hardware IoTaWatt
34

Thank @overeasy, I think I have spotted the source of this bug. On the standard install of emoncms $time_now is set by the input in the power_to_kwh processor:

"https://github.com/emoncms/emoncms/blob/master/Modules/process/process_processlist.php#L290

but on emoncms.org its being set using php time(), its therefore setting $kwh_inc = 0; which explains the graph your getting.

"https://github.com/emoncms/emoncmsorg/blob/master/emoncms/Modules/input/process_model.php#L265

I need to double check this and ensure that there will not be any unintended consequences of making the change but I think that may well be the cause, thanks for highlighting it.

How large a bulk upload do you do per request when your catching up after several days say?

35

I cut it off when the entire URL string is >= 1000. The number of update periods and span of time that includes is a function of the number of input channels being reported and the posting interval.

In this case, I have 15 channels at 10 second intervals. So say about 50 characters per interval, say 60 or 70 for the basics (path, apikey, time) and it comes to roughly 18 or 19 entries covering a period of about 180 seconds.

This is an important issue because while during recovery the URL can by up to 1000 characters, the program has a bulk-send attribute that causes it to aggregate the specified number of postings into a single bulk post during regular operation. Setting that to one = real time updates while I think I’m running at three right now so I only send you a transaction every 30 seconds. Conceivably someone could set the bulk to say 18 and end up with the same scenario as recovery every 3 minutes.

The bulk feature makes real-time widgets lag a bit, but I get my local real-time right from the horses mouth. It has the potential to unload network traffic significantly.

36

@overeasy great thanks

37

As I see it adafruit have the Adafruit Feather HUZZAH with ESP8266 WiFi but are the only supplier of the Adalogger FeatherWing - RTC + SD Add-on For All Feather Boards. It looks like they are now $25.90 for the pair and $16.40 for international shipping. I have been waiting for ages for the Adalogger to come back into stock which it finally has today.

In the meantime I have been doing some reading and I don’t want to be critical of Bob’s tremendous development effort in any way but a few dumb questions have lodged in my head:

  1. I understand the SD card is a physical connection so any replacement should work. eg This Micro SD card module for 39c.

  2. The real time clock (PCF8523) is no so common but it doesn’t seem so impressive either. A DS3231 RTC Module for $1.28 would probably be more accurate but the programming library would be different. How much effort would it be for the software to support this or a similar module?

  3. I understand that ongoing power issues with the ESP8622 DevBoard which led to the AdaFruit Feather ESP8266 being adopted. What doesn’t the $2.70 ESP-12F Module have that the IotaWatt needs?

  4. One application is a pump shed. Can the NodeMCU ESP8266 devkit still be used with the shields getting it to reset periodically to work with the power issues?

The main thing that I like about the first two options is that they are not tied to a single supplier.

38

I went with the ADAfruit feather system for the 2.0 version to get the SD and RTC as part of a modular system. Although the ESP is a bit more than the street price for nodeMCU, in the big picture its really still cheap for what it is compared to something like an Arduino. Even the SainSmart duinos (which are fine) are still relatively expensive. [quote=“flywire, post:37, topic:2989”]
$16.40 for international shipping
[/quote]

My biggest disappointment with AdaFruit. They charge me around $13 and I live less than 200 miles from them. I have complained with no response.

Yeah, the SD is just an SPI device. The Adalogger is really just a connector. Just make sure the Clk, Miso, Mosi, and CS (gpio15) are connected along with 3.3v and GND. Be careful because the pinout of the microSD is different from the full size.

The library supports both the DS3231 and the DS1307, so it would be trivial to adapt to them. Accuracy isn’t an issue. The AC synchronization is handled by the ESP millisecond clock. The RTC, while an important part of any datalogger, needs only to be within a few seconds, and IotaWatt checks the NTP server every hour and adjusts.

There are no power issues that I am aware of with the nodeMCU 0.9 or 1.0. My house runs on a 1.0 and has been running uninterrupted for the past two weeks. The ADC boards could easily be modified to accept the nodeMCU, or an adapter board could be made pretty inexpensively (maybe with an RTC and SDsocket).[quote=“flywire, post:37, topic:2989”]
One application is a pump shed. Can the NodeMCU ESP8266 devkit still be used with the shields getting it to reset periodically to work with the power issues?
[/quote]

Once again, there are no powr issues, but the IotaWatt reboots in about 15 seconds, fills in the log gaps, and keeps right on going pretty much uneventfully.[quote=“flywire, post:37, topic:2989”]
The main thing that I like about the first two options is that they are not tied to a single supplier.
[/quote]

I like that too. That’s why it’s open. Hopefully this will catch on when I finish the software and there will be a few hardware options.

39

Hi bob - was curious why you never used the the built in ADC. ( or maybe you did and did not like the results)

i built something similar to yours. only based on the emontx form factor- inter changeable software and hardware from arduino to esp ( wemos d1) the only difference I guess Is i used the onboard ADC to do the voltage calculation and it left a full 8 ADC to stack up to 4 units or what ever number of free CS your board supports… 32 cts on a esp and 36 ct if used on adruino and enmontx sheild. ( do not have 36 ct so i can not attest to whether it works well with 36 ct s but it will support it _ I only have 12 cts - but it works fine with those)

but yours looks nicer then my hand built ones

40

Several reasons:

  • One isn’t enough.
  • It has a 1 volt limitation.
  • Not sure how accurate the 1v reference is
  • Probably nowhere near as fast as the MCPs (though I don’t know that).
  • Wanted to support multiple voltage inputs for three phase and more accurate split phase monitoring.
  • Had to do a lot of experimentation to learn how to use the MCP fast and accurately, another piece of hardware would have doubled that.
  • Doesn’t really do anything significant to the cost but increases the software complexity.

All that said, I’m starting to use the ESP32 and it has 16 ADC channels (if you want to give up some digital pins). I may take advantage of them.

41

Latest version coming along well. Now supports dynamic configuration so you never have to restart.

Also now has output channels that are functions of input channels, much like eMonCMS inputs. The output channels are available in the local graphing application.

input.PNG617×738 36.5 KB

Output channels are configured with a “calculator” like interface:
output.PNG618×851 32.4 KB

There’s a demo version at iotawatt.com that you can play around with. It has limited functionality but demonstrates the look and feel.

I’ve received quite a few inquiries, from here and GitHub, for hardware. Other efforts have not panned out, so I am moving ahead to make hardware available as soon as possible.

4 Likes
42

Very nice, Bob!

This is what OEM should aspire to be.

1 Like
43

Look very very good!
One thing i could not find: does it support 3 phase measurement with only 1 VT? (i read that 3phase and 3VT work)

44

The only way I know to accurately measure three phase power is to treat
each leg as an independent load and add the three together. That said,
you can probably get pretty close by making assumptions about voltage of
two legs relative to a third measured leg. I think one of the eMon
boxes does that, and its probably reasonably close for applications
where the legs are loaded symmetrically. I wouldn’t count on it to
measure individual circuits in a three phase panel where the legs are
loaded differently.

But to answer your question more directly, no, there is no software to
perform the necessary phase shifts to estimate three phase power based
off one VT. IoTaWatt makes it easy to connect three VTs.

45

Wow, very inspiring, great job Bob!

46

Looks really impressive. Got a few questions:

  • How do you configure it for 3-phase? I see you can setup multiple VTs but how do you configure which phase is which in the software? (I’ve only looked at the demo online but cannot see how you would do it).
  • What size SD is required? Github mentions 16GB but can we get away with much smaller, eg 4GB?
  • How do you determine what voltage each board should run at? For example I plan to use some SCT-013-000 and some HWCT-004 CT coils - 3.3v or 1.2v on the switch?
  • I have a location that I’d like to use two “units” to get 28 channels in total, can you see any issues if I share the 3 VT inputs between the two units (ie. 2x (huzzah+3xadc boards))?

Here in the UK there are a couple of suppliers that offer the adafruit Huzzah and datalogger so that’s good.

Any progress on production options? I could do with 16 ADC boards…

47

When you specify more than one VT, a voltage reference specification appears in the CT input edit dialogue. Was in the old app, dropped out in the remake, is in my current version. Was always in the code.

Its all about the log file. Currently whites 256bytes every 5 seconds. That’s about 1.6GB per year. My current design will probably use half that. So depends…

You should ask Mr. Wall for a lesson in burden/turns/current, or read his excellent tutorials in the learning section. The reason I added 1.2V in the adafruit feather design was that I have some revenue grade .33mv CTs that I came across on eBay for short money. Truth be told, with the 12bit ADCs, even those work fine at 3.3V. I have abandoned 1.2V in my latest design. I have an HCWT-004 monitoring a solar inverter at one installation. 3.3v 24ohm burden. The thing has been within 0.1Kwh of the (Solar-Edge) inverter web monitor application, which is revenue grade because it is used as a basis for SREC (Solar Renewable Energy Credit) payments. Not bad for a $1.98 CT.

I see a potential problem in that one side of the VT goes to ground through a 10uf capacitor. If the polarity is reversed between the two IoTaWatt devices, and any elements of the DC circuitry are connected (like the ring of any 3.5 mm stereo jack), you will have a short of the 9VAC, potentially through the DC circuits of one or both units. So the answer is it can be done but has to be done with strict observance of polarity.

I had gone with the Adafruit components in the hope that I could convince Adafruit to build an ADC boads to round out the hardware and make it available to people. They don’t even answer emails, and their stuff is relatively expensive, especially with their shipping. So I have given up on them and gone back to the $5 (free shipping) nodeMCUs which work fine. I have put the RTC and SD on a board with two ADCs for a 14 channel + VT 3.3v design that is working great - all SMD except for the nodeMCU.

The final board went to PCBway today, should be back in a week. I have sourced an enclosure, have samples, and am waiting for a quote on machining the openings. I have an excellent local assembly company that is quoting board production. So I will need to decide soon if I go to market with it.

Not really sure what you are doing. I get the sense that you may be associated with the OEM people. Last I heard from Trystan Lea, he played with the device I sent him for half a day, left it connected as voltmeter for a few weeks, then it went off the air. Not a word from these folks, so I am considering that overture as dead as the Adafruit effort.

IMG_0698.JPG1936×2592 1.25 MB

IMG_0699.JPG1936×2592 1.06 MB

IMG_0700.JPG2592×1936 1.07 MB

2 Likes
48

Not associated, I’m a long-time user of the OEM stuff and have contributed some code in the past. I write software and integrate systems, particularly for smart home/energy applications. I’ve got a big project at the moment that requires me to monitor just over 50 circuits in different parts of a building, some of which are three-phase. Hence my interest in your board, as I don’t see anything else available that supports a sensible number of circuits and can use an existing radio infrastructure (ie. site-wide WiFi).

Sounds like I need to wait a bit longer… do let me know when you are think you may be in a position to go to production.

Matthew

49

I’ll let you know. Now I seem to recall that you said you from down-under. I should have some boxes in a month or two at the most. If you want some boards, I buy these prototype boards in batches of ten and typically use only a few. You can have a few for shipping if you are inclined to populate them.

50

If you want to do something complementary to this effort, take a look at Amazon Web Services with an eye toward using it as a web data repository for IoTaWatt. All it really needs is a flat file, and the size of it is trivial by AWS standards. The IoTaWatt does the heavy lifting and a JS routine can implement the output channel scripts in a front end to any app. Would be fast and cheap.

I’ll be going there when I finish this hardware sprint.

51

I updated the online version with the development version that allows you to select from the available VT when there are more than one.

There is also support in the program to allow correction to virtually any phaseshift, so with a single VT, you might get satisfactory results by adding 120 or 240 to the phaselead on the other legs. This is similar in principle to the way the eMon handles three-phase with a single VT, but quite a bit different in implementation. With nearly ten times the sample rate, the accuracy should be improved. I haven’t had a chance to get to a three-phase site and test it, but when the time comes, maybe you could do a comparison. Both methods can be run simultaneously on the same IoTaWatt and the results compared.

52

Still very interested @overeasy, just been a bit flat out with emoncms developments, which I also need to post here about, your new board design and the enclosure in particular looks really nice!

53

@overeasy
Hi Bob
I’m following your project with interest, I can test your hardware at a UK three phase site if needed.

Regards
Dave