Does the emonTx Shield need an actual Arduino?

Dave · 18 December 2016 01:32

Thanks Paul
Still no luck, the print is now.
11 0 2 0 0 23692 0 71

And the logs .

2016-12-18 01:17:24,217 INFO     MainThread EmonHub emonHub 'emon-pi' variant v1.2
2016-12-18 01:17:24,220 INFO     MainThread Opening hub...
2016-12-18 01:17:24,253 INFO     MainThread Logging level set to DEBUG
2016-12-18 01:17:24,256 INFO     MainThread Creating EmonHubSerialInterfacer 'SerialArduino' 
2016-12-18 01:17:24,292 DEBUG    MainThread Opening serial port: /dev/ttyUSB0 @ 115200 bits/s
2016-12-18 01:17:24,295 DEBUG    MainThread Setting SerialArduino subchannels: ['ToRFM12']
2016-12-18 01:17:24,299 DEBUG    MainThread Interfacer: Subscribed to channel' : ToRFM12
2016-12-18 01:17:24,323 DEBUG    MainThread Setting SerialArduino pubchannels: ['ToEmonCMS']
2016-12-18 01:17:24,326 DEBUG    MainThread Interfacer: Subscribed to channel' : ToRFM12
2016-12-18 01:17:24,353 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi' 
2016-12-18 01:17:24,382 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
2016-12-18 01:17:26,432 WARNING  MainThread Device communication error - check settings
2016-12-18 01:17:26,436 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2016-12-18 01:17:27,440 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2016-12-18 01:17:28,445 INFO     MainThread Setting RFM2Pi quiet: 0 (0q)
2016-12-18 01:17:29,449 INFO     MainThread Setting RFM2Pi baseid: 5 (5i)
2016-12-18 01:17:30,454 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2016-12-18 01:17:31,459 DEBUG    MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2016-12-18 01:17:31,464 DEBUG    MainThread Interfacer: Subscribed to channel' : ToRFM12
2016-12-18 01:17:31,467 DEBUG    MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2016-12-18 01:17:31,471 DEBUG    MainThread Interfacer: Subscribed to channel' : ToRFM12
2016-12-18 01:17:31,566 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT' 
2016-12-18 01:17:31,600 INFO     MainThread MQTT Init mqtt_host=127.0.0.1 mqtt_port=1883 mqtt_user=emonpi
2016-12-18 01:17:31,688 DEBUG    MainThread MQTT Subscribed to channel' : ToEmonCMS
2016-12-18 01:17:31,773 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg' 
2016-12-18 01:17:31,781 DEBUG    MainThread emoncmsorg Subscribed to channel' : ToEmonCMS
2016-12-18 01:17:31,883 INFO     MQTT       Connecting to MQTT Server
2016-12-18 01:17:32,070 INFO     MQTT       connection status: Connection successful
2016-12-18 01:17:32,202 DEBUG    MQTT       CONACK => Return code: 0
2016-12-18 01:17:32,404 INFO     MQTT       on_subscribe
2016-12-18 01:17:33,276 WARNING  MainThread SerialArduino thread is dead
2016-12-18 01:17:34,633 WARNING  MainThread SerialArduino thread is dead
2016-12-18 01:17:35,826 WARNING  MainThread SerialArduino thread is dead
2016-12-18 01:17:36,908 WARNING  MainThread SerialArduino thread is dead
2016-12-18 01:25:21,045 INFO     emoncmsorg sending: http://192.168.10.34/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[]&sentat=1482024321
2016-12-18 01:25:21,159 WARNING  MainThread SerialArduino thread is dead
2016-12-18 01:25:21,377 WARNING  emoncmsorg send failure: wanted 'ok' but got 'Error: Format error, json string supplied is not valid
'
2016-12-18 01:25:22,230 WARNING  MainThread SerialArduino thread is dead
2016-12-18 01:25:23,267 WARNING  MainThread SerialArduino thread is dead
2016-12-18 01:25:24,426 WARNING  MainThread SerialArduino thread is dead

Regards
Dave

pb66 · 18 December 2016 14:04

Looking back through the various bits of log I see no evidence of the SerialArduino thread ever being alive, i wonder if there is something being transmitted by the emonTx at start up that emonhub doesn’t like. Have you disabled the various start up messages in the emonTx sketch?

All my USB connected emonTx’s have custom firmware that have a debug mode to enable the startup messages etc, the normal running is nice clean formatted data only. I have connected my USB nodes with the debugging messages to emonhub without problems, but the emonHub I use is not so prone to crashing as the emonpi variant.

It’s worth me mentioning but unlikely to be the issue here that this USB connection cannot be hot plugged, I have no idea what the outcome maybe on the emonpi variant but currently I need to restart the interfacer (or emonhub) to see any disconnected/reconnected devices, emonhub doesn’t fail but it doesn’t manage to “pick up where it left off” (yet?). I haven’t looked that deeply into why yet but for now just assume it’s not hot-plug safe.

I have not tested (the emonpi variant of) emonhub at 115200 baud either, but I do not suspect that to be an issue. Having said that it is an easy thing to exclude by editing the baud in 2 places, the sketches setup() and emonhub.conf.

Could you confirm the fault still occurs when there are no debug/setup messages passed and maybe try a lower baud until we see it work?

P.S. your Vrms no longer needs to be sent in integer form, if you wanted to you could alter the print line to

    Serial.print(" "); Serial.print(emontx.Vrms*0.01);

but that definitely isn’t causing an issue and can be scaled in emonhub.conf or emoncms instead, I just thought I would mention it is not compulsory likewise for the temperatures I guess.

        Serial.print(" "); Serial.print(emontx.temp[j]*0.1);

Dave · 18 December 2016 19:40

Thanks for your help Paul
You were indeed right, the startup messages were crashing the thread, I’ve removed all of the startup messages and its passing the data but its complaining about data length.
Also is it picking up the node id in the data string?
Your right about not being able to hot plug, every time I unplug, I have to reboot the Pi (I’m running on a standard pi but using a pre setup SD Card image)

2016-12-18 19:34:35,285 INFO     MainThread EmonHub emonHub 'emon-pi' variant v1.2
2016-12-18 19:34:35,288 INFO     MainThread Opening hub...
2016-12-18 19:34:35,291 INFO     MainThread Logging level set to DEBUG
2016-12-18 19:34:35,314 INFO     MainThread Creating EmonHubSerialInterfacer 'SerialArduino' 
2016-12-18 19:34:35,349 DEBUG    MainThread Opening serial port: /dev/ttyUSB0 @ 115200 bits/s
2016-12-18 19:34:35,356 DEBUG    MainThread Setting SerialArduino subchannels: ['ToRFM12']
2016-12-18 19:34:35,360 DEBUG    MainThread Interfacer: Subscribed to channel' : ToRFM12
2016-12-18 19:34:35,375 DEBUG    MainThread Setting SerialArduino pubchannels: ['ToEmonCMS']
2016-12-18 19:34:35,379 DEBUG    MainThread Interfacer: Subscribed to channel' : ToRFM12
2016-12-18 19:34:35,413 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi' 
2016-12-18 19:34:35,420 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
2016-12-18 19:34:37,449 WARNING  MainThread Device communication error - check settings
2016-12-18 19:34:37,457 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2016-12-18 19:34:38,461 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2016-12-18 19:34:39,466 INFO     MainThread Setting RFM2Pi quiet: 0 (0q)
2016-12-18 19:34:40,472 INFO     MainThread Setting RFM2Pi baseid: 5 (5i)
2016-12-18 19:34:41,476 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2016-12-18 19:34:42,483 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT' 
2016-12-18 19:34:42,488 INFO     MainThread MQTT Init mqtt_host=127.0.0.1 mqtt_port=1883 mqtt_user=emonpi
2016-12-18 19:34:42,617 DEBUG    MainThread MQTT Subscribed to channel' : ToEmonCMS
2016-12-18 19:34:42,813 INFO     MQTT       Connecting to MQTT Server
2016-12-18 19:34:43,054 INFO     MQTT       connection status: Connection successful
2016-12-18 19:34:43,059 DEBUG    MQTT       CONACK => Return code: 0
2016-12-18 19:34:43,364 INFO     MQTT       on_subscribe
2016-12-18 19:35:03,880 DEBUG    SerialArduino 1 NEW FRAME : 11 3 4 1 0 238.49 8.50 0
2016-12-18 19:35:03,936 WARNING  SerialArduino 1 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:08,881 DEBUG    SerialArduino 2 NEW FRAME : 11 2 5 1 0 238.37 8.50 0
2016-12-18 19:35:09,018 WARNING  SerialArduino 2 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:09,093 INFO     MQTT       Unexpected disconnection
2016-12-18 19:35:09,219 INFO     MQTT       Connecting to MQTT Server
2016-12-18 19:35:09,284 INFO     MQTT       Could not connect...
2016-12-18 19:35:10,389 INFO     MQTT       Connecting to MQTT Server
2016-12-18 19:35:10,398 INFO     MQTT       Could not connect...
2016-12-18 19:35:11,554 INFO     MQTT       Connecting to MQTT Server
2016-12-18 19:35:11,666 INFO     MQTT       connection status: Connection successful
2016-12-18 19:35:11,738 DEBUG    MQTT       CONACK => Return code: 0
2016-12-18 19:35:11,958 INFO     MQTT       on_subscribe
2016-12-18 19:35:13,871 DEBUG    SerialArduino 3 NEW FRAME : 11 4 4 1 0 237.99 8.50 0
2016-12-18 19:35:13,928 WARNING  SerialArduino 3 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:18,690 DEBUG    SerialArduino 4 NEW FRAME : 11 1 4 1 0 238.35 8.50 0
2016-12-18 19:35:18,837 WARNING  SerialArduino 4 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:23,699 DEBUG    SerialArduino 5 NEW FRAME : 11 1 4 1 0 238.43 8.50 0
2016-12-18 19:35:23,816 WARNING  SerialArduino 5 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:28,562 DEBUG    SerialArduino 6 NEW FRAME : 11 3 5 2 0 238.18 8.50 0
2016-12-18 19:35:28,660 WARNING  SerialArduino 6 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:33,719 DEBUG    SerialArduino 7 NEW FRAME : 11 3 5 1 0 237.46 8.50 0
2016-12-18 19:35:33,878 WARNING  SerialArduino 7 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:38,474 DEBUG    SerialArduino 8 NEW FRAME : 11 2 4 1 0 238.37 8.50 0
2016-12-18 19:35:38,526 WARNING  SerialArduino 8 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:43,456 DEBUG    SerialArduino 9 NEW FRAME : 11 3 3 2 0 238.40 8.50 0
2016-12-18 19:35:43,487 WARNING  SerialArduino 9 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:48,389 DEBUG    SerialArduino 10 NEW FRAME : 11 2 4 3 0 238.38 8.50 0
2016-12-18 19:35:48,496 WARNING  SerialArduino 10 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:53,199 DEBUG    SerialArduino 11 NEW FRAME : 11 2 4 2 0 238.26 8.50 0
2016-12-18 19:35:53,306 WARNING  SerialArduino 11 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:35:58,142 DEBUG    SerialArduino 12 NEW FRAME : 11 1 4 2 0 238.19 8.60 0
2016-12-18 19:35:58,168 WARNING  SerialArduino 12 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']
2016-12-18 19:36:03,022 DEBUG    SerialArduino 13 NEW FRAME : 11 1 4 2 0 237.91 8.60 0
2016-12-18 19:36:03,068 WARNING  SerialArduino 13 RX data length: 7 is not valid for datacodes ['h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'h', 'L']

EmonHub

[[11]]
    nodename = emonTx_Node11
    [[[rx]]]
       names = Shed_Submain, Shed_SolarPV, Shed_Sockets, Shed_Lights, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacodes = h,h,h,h,h,h,h,h,h,h,h,L
       scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

pb66 · 18 December 2016 21:19

Hi Dave, the data being passed from the emonTx via serial to emonHub doesn’t need reconstructing.

is instructing emonhub to unpack the 26 byte values as 11 ints and 1 unsigned long, but since there are only 7 values provided, emonhub objects.

You can use a “0” datacode to specify a value doesn’t need further decoding so you have 3 options

datacode = 0 to specify an undefined number of un-encoded values
datacodes = 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 to specify 12 un-encoded values
omit a datacode(s) entry as the default datacode for a serialinterfacer is already “0”

The leading node id is correct. There is another way of doing it where you can add a node id to the serialinterfacer settings in emonhub.conf instead, but that is only suitable if you only have 1 serial device or they have a way of using a unique id in their device path (hence the programable eeproms and udev rules) as when you have more than one device they compete for USB0, USB1 etc on a first come first served basis so in effect the node ids become useless.

When say you “have to reboot” are you saying that restarting emonhub or the interfacer doesn’t work?

Dave · 18 December 2016 22:12

Thanks Paul, changing the datacode to 0 worked and the data is now coming up in the inputs.
I think its sensible to write the node id into the sketch as you would with using wireless so you know what’s what.
I have noticed that its taking the pulse as a temperature, would you recommend moving the pulse before the temperature, ie VOLTAGE - PULSE - TEMPERATURE / S
TBH I was resetting the pi to try and rule out not restarting something properly while trying to get things working.

Also would you recommend using MQTT rather than HTTP posting? I presume you add the MQTT server in the receiving server?

Regards
Dave

pb66 · 18 December 2016 23:19

The definition in emonhub.conf must match what ever is being sent from the sketch, you can have whatever you like as long as it is the same both ends. What is the data format you want or expect? I see there are 2 temps shown on you inputs page, so going by the comment, I assume you are sending power1, power2, power3, power4, Vrms, Temp1, pulse, is that right?

I agree the most logical place for the variable part of the payload (ie 0 - 6 temp sensors) is at the end, but ultimately it is your choice, that is the whole point of user defined nodes in emonhub (I’m not a fan of all the pre-defined nodes).

Using the right tool for the job is the best way of getting the job done correctly, it depends on your specific application, again there is no “one size fits all” way of doing things. It depends on where you are posting and what else you want to do with your data etc.

IMO the way MQTT is used for OEM is not mature enough to replace http, I’m itching to go all MQTT, but there are a whole load of reasons I can’t or won’t. I currently use http, but having said that the very basic http implementation in the emonpi variant of emonhub is secondary to the MQTT and therefore doesn’t have all the benefits.

fabiopbx · 19 December 2016 15:20

thats what I ment, testing one by one.

Im having some weird errors, was wondering if anyone has come across this and knows why… on my windows machine i can no longer program the Tx’s or read from the serial… all 3 adaptors, many reboots and driver reinstall along with the different machine…

yet on linux (the raspberrypi) reads them fine :o

idea, maybe im missing something super obvious?

errors i get are:

_avrdude: ser_open(): can't open device "\\.\COM3": Access is denied._

_or_

_Error opening serial port 'COM3'. (Port busy)_

fabiopbx · 19 December 2016 15:35

mhmm somehow it works, if plug the devices backwards… open the serial monitor and then plug them in properly… odd

pb66 · 19 December 2016 15:49

Sometime this can happen on windows when using the Aduino IDE if you have multiple “IDE” windows open and use the serial console, ie when you try to upload a sketch from onw window when you have the serial console connected and attached to a second “IDE” window.

That’s not quite what I meant, I do all at once so I can compare, calibrate and soak test all as a group from a single test load and voltage supply, then mark the AC:AC’s (and CT’s ?) if you apply specific calibrations.

fabiopbx · 19 December 2016 15:53

ahh i see sounds good, i was trying to get the firmware there and that was giving me head aches

well i got it now, tho i have been unable to change the eeprom of the adaptors using Linux or the FT_PROG utility…

pb66 · 19 December 2016 16:01

When using the FT_PROG utility can you see the serial numbers? are they the same for all your adapters?
(possibly A50285BI or A9M9DY3R are the more common fake serial numbers)

They will probably function OK as serial adapters but you will not be able to use udev rules effectively unless they at least have different serial numbers, I also like to give mine friendly names too.

fabiopbx · 19 December 2016 16:19

I believe its these, i cant see the SERIAL or anything the app says, it detected an adaptor but cant access them or something… then goes on about waiting for windows to enumerate them. as you say likely fakes, they can read serial ok but yh no udev rules, theres no amount of unique data i could use to udev them

Damn emonCMS/emonHUB wont stop making feeds

pb66 · 19 December 2016 16:33

Not if they are A50285BI’s, they do not have a writable eeprom, they do not even have different serial numbers exposed they are total clones.

If you bought them on ebay you could order some more from somewhere else and ask for a refund for those, if it’s an oversea’s seller Ebay will force a refund (inc P&P) without you having to return them at your cost, it will be up to the seller to collect or pre-pay for the return trip.

fabiopbx · 19 December 2016 19:22

now i cant read data off any of them any ideas or are they just crappy ones?

Dave · 19 December 2016 22:00

@fabiopbx I had success with my ones on fleebay, I didn’t check the serial numbers but confirmed that the EEPROM was writeable.

Regards
Dave

pb66 · 19 December 2016 22:56

Do you mean you can’t read the configuration of the device or that it is not passing serial data?

I don’t use the obvious clones, I have a sack of them in the garage that have been refunded, but not returned. If it doesn’t have an eeprom, it’s not a FTDI ft232rl so there’s no telling what it will or won’t do. personally I wouldn’t waste my time with them.

fabiopbx · 20 December 2016 09:28

yh, no serial data… sometimes one of them works… then it stops they all worked before tho very odd

pb66 · 21 December 2016 14:29

Have you got the voltage selector set to 5v not 3.3v?

This only alters the output VCC and the emonTx expects 5v at the 6pin header, supplying 3v’ish into a 5v to 3.3v reg might cause a problem? I initially had a couple that wouldn’t upload new sketches when I had selected 3.3v, this turned out to be power related as selecting 5v allowed me to upload again, but I have not seen an issue where the normal serial output wasn’t available when the emonTx was running. Is the LED on the emonTx flashing whilst you cannot get any serial output?

fabiopbx · 21 December 2016 14:32

nah its 5V.

I’m seeing these on dmesg

ftdi_sio ttyUSB0: usb_serial_generic_read_bulk_callback - urb stopped: -32
ftdi_sio ttyUSB0: usb_serial_generic_read_bulk_callback - urb stopped: -32
ftdi_sio ttyUSB1: usb_serial_generic_read_bulk_callback - urb stopped: -32
ftdi_sio ttyUSB1: usb_serial_generic_read_bulk_callback - urb stopped: -32

pb66 · 21 December 2016 14:39

Sorry that’s not something I’ve seen.

If they are giving you trouble from the outset, just get new ones whilst you are able to do it at the sellers expense. Right now you can get Ebay to refund you on the grounds they are fake or faulty on delivery, a couple of weeks down the line and you will be at the mercy or the sellers warranty (or lack of!) unless you shell out for new ones out of your own pocket.