Posting data & backing up

haffle · 7 November 2019 14:54

I currently post data from my three emonTx units directly to emoncms.org & I have set my emonPi to post data to emoncms.org

Is this the best thing to have done? Would I have been better to post the emonTx data to the emonPi, or even post to both the emonPi AND emoncms.org?

One reason that I ask is WRT to backing up data; AIUI from the online Guide Import / Backup - Guide | OpenEnergyMonitor it is not possible to download or backup data held on emoncms.org - which seems strange

In case it is relevant, I do have access to my own webspace, and I have a static IP. (The reason I didn’t use it is because I am now to this and wasn’t sure how to set emoncms up on my webspace).

TIA

Bill.Thomson · 7 November 2019 17:20

Indeed you would’ve. (posting to both)

A matter of economics, if nothing else. Serving up bandwidth costs money.

haffle · 7 November 2019 17:46

Thanks @Bill.Thomson ; if I change the 3 EmonTx’s to post to my emonPi locally (via a fixed WLAN IP 192.168.1.xxx), would the emonPi then automatically post data from all four nodes to emoncms.org, or would I have to change a setting within the emonPi too?

Whilst I’d rather not have to change my Feeds, Dashboards & Apps etc if I can help it, being able to backup my data from all four nodes (henceforward held on the Pi) in future does have a strong appeal. Or maybe I should just be brave and migrate my Dashboard data over to the emonPi based data now in order to prevent future confusion.

This is what the emonhub on my Pi currently says (I have redacted my API key (& mqtt_passwd)).
.
.
.

#######################################################################
#######################       Interfacers       #######################
#######################################################################

[interfacers]
### This interfacer manages the RFM12Pi/RFM69Pi/emonPi module
[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
        com_baud = 38400                        # 9600 for old RFM12Pi
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,

        group = 210
        frequency = 433
        baseid = 5                              # emonPi / emonBase nodeID
        calibration = 230V                      # (UK/EU: 230V, US: 110V)
        quiet = true                            # Disable quite mode (default enabled) to enable RF packet debugging, show packets which fail crc
        # interval =  0                         # Interval to transmit time to emonGLCD (seconds)
        

[[MQTT]]

    Type = EmonHubMqttInterfacer
    [[[init_settings]]]
        mqtt_host = 127.0.0.1
        mqtt_port = 1883
        mqtt_user = emonpi
        mqtt_passwd = X!X!X!X!X!X!X!X!X!

    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,

        # emonhub/rx/10/values format
        # Use with emoncms Nodes module
        node_format_enable = 1
        node_format_basetopic = emonhub/

        # emon/emontx/power1 format - use with Emoncms MQTT input
        # http://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
        nodevar_format_enable = 1
        nodevar_format_basetopic = emon/

[[emoncmsorg]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = https://emoncms.org
        apikey = X!X!X!X!X!X!XX!X!X!X!X!X!X!X!X!XX!X!X!X!XXX!X!X!XX!X!X!X!X
        senddata = 1                    # Enable sending data to Emoncms.org
        sendstatus = 1                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list
        sendinterval= 10                # Bulk send interval to Emoncms.org in seconds

#######################################################################
#######################          Nodes          #######################
#######################################################################

borpin · 7 November 2019 17:59

No BUT don’t change the URL just duplicate this section and change the IP and API Key so Emonhub sends the data to the local Pi as well as emoncms.org. I know the emonhub on the EmonPi uses MQTT to do this, but I prefer using the HTTP API.

No that is not right!

Question, do you use the CT input on the EmonPi at all or just the EmonTXs?

You will need to create the feeds etc on the EmonPi for each EmonTX.

The configuration guide is emonhub/configuration.md at emon-pi · openenergymonitor/emonhub · GitHub

borpin · 7 November 2019 18:07

Slightly confused here. Is there anything on the ‘Inputs’ page of your local EmonCMS?

haffle · 7 November 2019 18:15

On the Pi Inputs page there is an errant Node 3 (which I don’t know what it is and hasn’t updated any Input for 4 days & has no Feeds), and a Node 5 (the Pi) which correctly shows two CTs, one Temp & one VRMS Input.

borpin · 7 November 2019 18:45

EmonHub has always been a bit of a mystery to me but here goes

I am presuming your EmonTXs are communicating with the EmonHub on the EmonPi and sending the data to emoncms.org.

I would therefore expect the data to also be sent via MQTT to the EmonPi as well.

AIUI (and I’m sure someone will put be right so I will learn if I’m wrong), what happens is, that data comes in on the RFM2Pi interface and is published on a pubchannel. The MQTT and Emoncmsorg interfaces listen (subscribe) to subchannels.

[[RFM2Pi]]
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,

and

[[MQTT]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,

[[emoncmsorg]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,

What you posted above is just the top part of the emonhub.conf I presume and below that are a load of node definitions?

What node numbers are your EmonTXs?

Have a look in the log. Do you see

NEW FRAME : OK 10

where the fiirst number matches the number of your EmonTXs? If so, what is in the log after that.

You will need to set the log level to DEBUG

haffle · 7 November 2019 19:12

Thanks Brian @borpin . I’ll study your reply more fully, but in the interim herewith the entire Emonhub config. Also …

The emonPi is Node 5; I’d given the 3 EmonTx’s names for Nodes within espwifi1912 (the control panel for the new ESP8266 that Trystan sent to me), and embarrassingly I can’t now work out what their Node numbers are … but presume they are three out of Nodes 7, 8, 9 & 10.

I rather thought that the 3 EmonTxShield were communicating directly to emoncms.org , but TBH I never really understood this aspect.

I don’t think that I have ever done anything WRT MQTT.
.

#######################################################################
#######################      emonhub.conf     #########################
#######################################################################
### emonHub configuration file, for info see documentation:
### https://github.com/openenergymonitor/emonhub/blob/emon-pi/conf/emonhub.conf
#######################################################################
#######################    emonHub  settings    #######################
#######################################################################

[hub]
### loglevel must be one of DEBUG, INFO, WARNING, ERROR, and CRITICAL
loglevel = DEBUG
### Uncomment this to also send to syslog
# use_syslog = yes
#######################################################################
#######################       Interfacers       #######################
#######################################################################

[interfacers]
### This interfacer manages the RFM12Pi/RFM69Pi/emonPi module
[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
        com_baud = 38400                        # 9600 for old RFM12Pi
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,

        group = 210
        frequency = 433
        baseid = 5                              # emonPi / emonBase nodeID
        calibration = 230V                      # (UK/EU: 230V, US: 110V)
        quiet = true                            # Disable quite mode (default enabled) to enable RF packet debugging, show packets which fail crc
        # interval =  0                         # Interval to transmit time to emonGLCD (seconds)
        

[[MQTT]]

    Type = EmonHubMqttInterfacer
    [[[init_settings]]]
        mqtt_host = 127.0.0.1
        mqtt_port = 1883
        mqtt_user = emonpi
        mqtt_passwd = Xredacted!X!X!X!!X!X!X!X!XX!X!X!X!X!X

    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,

        # emonhub/rx/10/values format
        # Use with emoncms Nodes module
        node_format_enable = 1
        node_format_basetopic = emonhub/

        # emon/emontx/power1 format - use with Emoncms MQTT input
        # http://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
        nodevar_format_enable = 1
        nodevar_format_basetopic = emon/

[[emoncmsorg]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = https://emoncms.org
        apikey = Xredacted!X!X!X!!X!X!X!X!XX!X!X!X!X!X
        senddata = 1                    # Enable sending data to Emoncms.org
        sendstatus = 1                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list
        sendinterval= 10                # Bulk send interval to Emoncms.org in seconds

#######################################################################
#######################          Nodes          #######################
#######################################################################

[nodes]

## See config user guide: https://github.com/openenergymonitor/emonhub/blob/emon-pi/conf/emonhub.conf

[[5]]
    nodename = emonpi
    [[[rx]]]
        names = power1,power2,power1pluspower2,vrms,t1,t2,t3,t4,t5,t6,pulsecount
        datacodes = h, h, h, h, h, h, h, h, h, h, L
        scales = 1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
        units = W,W,W,V,C,C,C,C,C,C,p

[[6]]
    nodename = emontxshield
    [[[rx]]]
       names = power1, power2, power3, power4, vrms
       datacode = h
       scales = 1,1,1,1,0.01
       units =W,W,W,W,V

[[7]]
   nodename = emontx4
   [[[rx]]]
      names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
      datacodes = h,h,h,h,h,h,h,h,h,h,h,L
      scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
      units =W,W,W,W,V,C,C,C,C,C,C,p

[[8]]
    nodename = emontx3
    [[[rx]]]
       names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacodes = h,h,h,h,h,h,h,h,h,h,h,L
       scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[9]]
   nodename = emontx2
   [[[rx]]]
      names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
      datacode = h
      scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
      units =W,W,W,W,V,C,C,C,C,C,C,p

[[10]]
    nodename = emontx1
    [[[rx]]]
       names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacode = h
       scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[11]]
    nodename = 3phase
    [[[rx]]]
       names = powerL1, powerL2, powerL3, power4, Vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacodes = h,h,h,h,h,h,h,h,h,h,h,L
       scales = 1,1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p
       
[[12]]
    nodename = 3phase2
    [[[rx]]]
       names = powerL1, powerL2, powerL3, power4, Vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacodes = h,h,h,h,h,h,h,h,h,h,h,L
       scales = 1,1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[13]]
    nodename = 3phase3
    [[[rx]]]
       names = powerL1, powerL2, powerL3, power4, Vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacodes = h,h,h,h,h,h,h,h,h,h,h,L
       scales = 1,1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[14]]
    nodename = 3phase4
    [[[rx]]]
       names = powerL1, powerL2, powerL3, power4, Vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacodes = h,h,h,h,h,h,h,h,h,h,h,L
       scales = 1,1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[19]]
   nodename = emonth1
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[20]]
   nodename = emonth2
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[21]]
   nodename = emonth3
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[22]]
   nodename = emonth4
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[23]]
    nodename = emonth5
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

[[24]]
    nodename = emonth6
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

[[25]]
    nodename = emonth7
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

[[26]]
    nodename = emonth8
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

borpin · 7 November 2019 19:17

Ah, you’re using the WiFi modules (I should have remembered). In that case just configure the MQTT settings in the EmonESP config page to match the EmonPi and the data should just appear on the inputs if they are all on the same LAN.

[edit]
For future ref when posting code or bash output, put in 3 ‘backticks’ (found at the top left of the keyboard normally) on a line on their own, then the code, then 3 more backticks on a line following the code.

    ```
    code
    ```

pb66 · 7 November 2019 19:26

Hi Julian, You are correct, your emontx’s are all reporting direct to emoncms.org via http using emonESP’s. To log the same data locally in parralel to sending to emoncms.org, you would need to use the emonESP’s MQTT settings. emonhub would play no part in this arrangement.

However before you do!

Take a look at the sync module as that was designed to do exactly what you need regards “backing up or exporting” your existing emoncms.org data to a local emoncms (first used for migrating from emoncms.org to local emonSD when emoncms.org became a chargeable service). But it needs to be done before you start setting up any local feeds or posting data via mqtt as you cannot insert older data, you must start with the oldest data first, ie the emoncms.org stuff b4 the new local reporting.

I see Brian has already posted about the MQTT stuff

haffle · 8 November 2019 09:40

borpin:

What node numbers are your EmonTXs?

Have a look in the log. Do you see
NEW FRAME : OK 10
where the fiirst number matches the number of your EmonTXs? If so, what is in the log after that.

You will need to set the log level to DEBUG

Herewith my emonpi.local log file taken just now; do I take it that the EmonTx’s are not reporting through the emonPi as only Node 5 shows up? The Values I do recognise - they are CT1, CT2, CT1+CT2, Vrms and Temp oC (my office is very cold!)

Is the “Discarding RX frame ‘unreliable content’” line a problem?


2019-11-08 09:24:42,731 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 4 32 24 54 64 0 36 33 0 32 5 40 0 36 8 32 0 253 128 0 34 (-102)
2019-11-08 09:24:44,651 DEBUG    RFM2Pi     740959 NEW FRAME : OK 5 7 0 36 0 43 0 170 89 105 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2019-11-08 09:24:44,655 DEBUG    RFM2Pi     740959 Timestamp : 1573205084.65
2019-11-08 09:24:44,656 DEBUG    RFM2Pi     740959 From Node : 5
2019-11-08 09:24:44,657 DEBUG    RFM2Pi     740959    Values : [7, 36, 43, 229.54, 10.5, 0, 0, 0, 0, 0, 0]
2019-11-08 09:24:44,658 DEBUG    RFM2Pi     740959 Sent to channel(start)' : ToEmonCMS
2019-11-08 09:24:44,658 DEBUG    RFM2Pi     740959 Sent to channel(end)' : ToEmonCMS
2019-11-08 09:24:44,817 DEBUG    MQTT       Publishing: emon/emonpi/power1 7
2019-11-08 09:24:44,819 DEBUG    MQTT       Publishing: emon/emonpi/power2 36
2019-11-08 09:24:44,820 DEBUG    MQTT       Publishing: emon/emonpi/power1pluspower2 43
2019-11-08 09:24:44,822 DEBUG    MQTT       Publishing: emon/emonpi/vrms 229.54
2019-11-08 09:24:44,824 DEBUG    MQTT       Publishing: emon/emonpi/t1 10.5
2019-11-08 09:24:44,825 DEBUG    MQTT       Publishing: emon/emonpi/t2 0
2019-11-08 09:24:44,827 DEBUG    MQTT       Publishing: emon/emonpi/t3 0
2019-11-08 09:24:44,829 DEBUG    MQTT       Publishing: emon/emonpi/t4 0
2019-11-08 09:24:44,830 DEBUG    MQTT       Publishing: emon/emonpi/t5 0
2019-11-08 09:24:44,832 DEBUG    MQTT       Publishing: emon/emonpi/t6 0
2019-11-08 09:24:44,833 DEBUG    MQTT       Publishing: emon/emonpi/pulsecount 0
2019-11-08 09:24:44,835 INFO     MQTT       Publishing: emonhub/rx/5/values 7,36,43,229.54,10.5,0,0,0,0,0,0
2019-11-08 09:24:46,277 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 0 129 192 162 130 40 160 32 0 8 160 58 67 32 38 1 0 144 64 0 240 (-100)
2019-11-08 09:24:47,320 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 0 32 27 224 44 96 168 20 2 176 0 32 32 0 0 0 56 172 166 18 240 (-98)
2019-11-08 09:24:47,859 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 6 132 129 34 130 32 2 34 180 41 46 32 15 24 52 94 16 36 0 51 1 (-100)
2019-11-08 09:24:49,642 DEBUG    RFM2Pi     740960 NEW FRAME : OK 5 4 0 37 0 41 0 155 89 105 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2019-11-08 09:24:49,646 DEBUG    RFM2Pi     740960 Timestamp : 1573205089.64
2019-11-08 09:24:49,647 DEBUG    RFM2Pi     740960 From Node : 5
2019-11-08 09:24:49,648 DEBUG    RFM2Pi     740960    Values : [4, 37, 41, 229.39000000000001, 10.5, 0, 0, 0, 0, 0, 0]
2019-11-08 09:24:49,649 DEBUG    RFM2Pi     740960 Sent to channel(start)' : ToEmonCMS
2019-11-08 09:24:49,649 DEBUG    RFM2Pi     740960 Sent to channel(end)' : ToEmonCMS
2019-11-08 09:24:49,736 DEBUG    MQTT       Publishing: emon/emonpi/power1 4
2019-11-08 09:24:49,738 DEBUG    MQTT       Publishing: emon/emonpi/power2 37
2019-11-08 09:24:49,740 DEBUG    MQTT       Publishing: emon/emonpi/power1pluspower2 41
2019-11-08 09:24:49,741 DEBUG    MQTT       Publishing: emon/emonpi/vrms 229.39
2019-11-08 09:24:49,743 DEBUG    MQTT       Publishing: emon/emonpi/t1 10.5
2019-11-08 09:24:49,744 DEBUG    MQTT       Publishing: emon/emonpi/t2 0
2019-11-08 09:24:49,745 DEBUG    MQTT       Publishing: emon/emonpi/t3 0
2019-11-08 09:24:49,748 DEBUG    MQTT       Publishing: emon/emonpi/t4 0
2019-11-08 09:24:49,750 DEBUG    MQTT       Publishing: emon/emonpi/t5 0
2019-11-08 09:24:49,752 DEBUG    MQTT       Publishing: emon/emonpi/t6 0
2019-11-08 09:24:49,754 DEBUG    MQTT       Publishing: emon/emonpi/pulsecount 0
2019-11-08 09:24:49,756 INFO     MQTT       Publishing: emonhub/rx/5/values 4,37,41,229.39,10.5,0,0,0,0,0,0
2019-11-08 09:24:54,635 DEBUG    RFM2Pi     740961 NEW FRAME : OK 5 7 0 38 0 45 0 96 90 105 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2019-11-08 09:24:54,639 DEBUG    RFM2Pi     740961 Timestamp : 1573205094.63
2019-11-08 09:24:54,640 DEBUG    RFM2Pi     740961 From Node : 5
2019-11-08 09:24:54,641 DEBUG    RFM2Pi     740961    Values : [7, 38, 45, 231.36, 10.5, 0, 0, 0, 0, 0, 0]
2019-11-08 09:24:54,641 DEBUG    RFM2Pi     740961 Sent to channel(start)' : ToEmonCMS
2019-11-08 09:24:54,642 DEBUG    RFM2Pi     740961 Sent to channel(end)' : ToEmonCMS
2019-11-08 09:24:54,750 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 0 35 187 56 36 43 27 87 128 64 1 68 48 44 4 12 63 144 134 41 64 (-102)
2019-11-08 09:24:54,867 DEBUG    MQTT       Publishing: emon/emonpi/power1 7
2019-11-08 09:24:54,869 DEBUG    MQTT       Publishing: emon/emonpi/power2 38
2019-11-08 09:24:54,871 DEBUG    MQTT       Publishing: emon/emonpi/power1pluspower2 45
2019-11-08 09:24:54,873 DEBUG    MQTT       Publishing: emon/emonpi/vrms 231.36
2019-11-08 09:24:54,874 DEBUG    MQTT       Publishing: emon/emonpi/t1 10.5
2019-11-08 09:24:54,876 DEBUG    MQTT       Publishing: emon/emonpi/t2 0
2019-11-08 09:24:54,878 DEBUG    MQTT       Publishing: emon/emonpi/t3 0
2019-11-08 09:24:54,880 DEBUG    MQTT       Publishing: emon/emonpi/t4 0
2019-11-08 09:24:54,881 DEBUG    MQTT       Publishing: emon/emonpi/t5 0
2019-11-08 09:24:54,883 DEBUG    MQTT       Publishing: emon/emonpi/t6 0
2019-11-08 09:24:54,885 DEBUG    MQTT       Publishing: emon/emonpi/pulsecount 0
2019-11-08 09:24:54,887 INFO     MQTT       Publishing: emonhub/rx/5/values 7,38,45,231.36,10.5,0,0,0,0,0,0
2019-11-08 09:24:56,081 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 16 36 192 19 136 16 161 112 86 162 41 32 0 168 26 131 56 1 128 178 16 (-100)
2019-11-08 09:24:59,704 DEBUG    RFM2Pi     740962 NEW FRAME : OK 5 7 0 36 0 43 0 197 89 105 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2019-11-08 09:24:59,708 DEBUG    RFM2Pi     740962 Timestamp : 1573205099.7
2019-11-08 09:24:59,708 DEBUG    RFM2Pi     740962 From Node : 5
2019-11-08 09:24:59,709 DEBUG    RFM2Pi     740962    Values : [7, 36, 43, 229.81, 10.5, 0, 0, 0, 0, 0, 0]
2019-11-08 09:24:59,710 DEBUG    RFM2Pi     740962 Sent to channel(start)' : ToEmonCMS
2019-11-08 09:24:59,710 DEBUG    RFM2Pi     740962 Sent to channel(end)' : ToEmonCMS
2019-11-08 09:24:59,901 DEBUG    MQTT       Publishing: emon/emonpi/power1 7
2019-11-08 09:24:59,903 DEBUG    MQTT       Publishing: emon/emonpi/power2 36
2019-11-08 09:24:59,905 DEBUG    MQTT       Publishing: emon/emonpi/power1pluspower2 43
2019-11-08 09:24:59,907 DEBUG    MQTT       Publishing: emon/emonpi/vrms 229.81
2019-11-08 09:24:59,909 DEBUG    MQTT       Publishing: emon/emonpi/t1 10.5
2019-11-08 09:24:59,911 DEBUG    MQTT       Publishing: emon/emonpi/t2 0
2019-11-08 09:24:59,913 DEBUG    MQTT       Publishing: emon/emonpi/t3 0
2019-11-08 09:24:59,915 DEBUG    MQTT       Publishing: emon/emonpi/t4 0
2019-11-08 09:24:59,917 DEBUG    MQTT       Publishing: emon/emonpi/t5 0
2019-11-08 09:24:59,919 DEBUG    MQTT       Publishing: emon/emonpi/t6 0
2019-11-08 09:24:59,920 DEBUG    MQTT       Publishing: emon/emonpi/pulsecount 0
2019-11-08 09:24:59,922 INFO     MQTT       Publishing: emonhub/rx/5/values 7,36,43,229.81,10.5,0,0,0,0,0,0

borpin · 8 November 2019 10:05

As said above, this is redundant wrt your question as the data is going direct from the TX via WiFi.

Not usually.

If you set up the MQTT on each WiFi module, this will be logged in the main emoncms log on receipt.

haffle · 8 November 2019 11:22

Thanks Paul. Within the Sync module in emonpi.local are several Local feeds (which are those created on the emonPi), and a load of Remote feeds (those created on the EmonTx’s). I Downloaded the first of the Tx feeds and that turned the Remote to Both for that feed. After a few moments it showed “Local behind Remote by 21 points”, with the value increasing by 1 slowly. (The only way to get that number to go back down is to re-download, and I am not quite sure why this would be / what it is “saying”).

But when I go back to the Log no mention is made of anything but Node 5 (the Pi); so I am not clear about quite what this Sync is doing & I’d rather clear it up in my mind before clicking Download on the rest of the feeds.

Maybe I Download all of the feeds within the Sync module and then log in to the EmonTx’s in turn, and fill in the emonPi’s LAN IP in the MQTT Server field? And in the MQTT Base-topic field just put something like “emontx3_house” and “emontx3_garage” etc etc? When done, all Inputs & Feeds will be stored on the emonPi rather than (or as well as) emoncms.org, and from there I can update my Dashboards & Apps to get their data from the Pi instead on emoncms.org. Or have I got this all wrong?

borpin · 8 November 2019 12:24

I think, you can just leave the Pi to sync from emoncms.org (not used sync myself as I do not have a remote account). Probably stops any confusion.

Which log? if emonhub, then no you won’t.

So these are the feeds on emoncms.org created by the EmonTX sending data there.

I expect this is telling you that the EmonTX has sent the data to the remote, and the local (Pi) is picking that data up from the remote, but there is a short delay. sounds reasonable.

haffle · 8 November 2019 13:31

Not sure; this screen grab taken an hour or so later shows the two Feeds that I have Sync’d (emonpi.local / Sync), and the “points” values continue to rise. I’m not sure how to interpret this. NB: If I click on Download against a Feed again, it initially says “Local & Remote are the same”, and then refreshes to different by 1 point … 2…3 etc after each automatic Next Update.

Am I right in thinking that the Tx Feed that I have Sync’d is posting initially to emoncms.org and then subsequently being sync’d back on to the emonPi? And that when I have Sync’d all of the Tx feeds in this way I can then tell the Tx’s to post to the Pi via MQTT with no loss of past data (other than perhaps a small amount lost during this handover process) as per @pb66 's comment above?

borpin · 8 November 2019 13:40

TBH I am lost as to your setup here.

What does the EmonPi feeds page look like and what does the emoncms.org Feeds page look like?

haffle · 8 November 2019 14:16

Apologies for any confusion. I have one emonPi & three EmonTx’s - which all connect to emoncms.org via wifi; all four units have a fixed LAN IP. The whole system has been communicating very reliably for the past couple of weeks, ever since swapping out some ESP8266 modules on the Tx’s with some newer ones from @TrystanLea. Communicating via RF is not possible due to distances.

I’ve attached two PDFs showing the Feeds page from the emonPi and from emoncms.

Towards the bottom of the emonPi page you can see the two Feeds that I Downloaded/Sync’d via emoncms.org (values 233.2 and 249.3); the former was probably a mistake by me as I have sync’d back some data that was already on the Pi . Not sure what has gone wrong with the Office Temp Max & Mins, but that isn’t very important right now.

On the emoncms page you can see that several recently added Min/Max Feeds aren’t working properly, but again, ignore them for now.

emonpi.local.pdf (46.0 KB)
Emoncms - feed list 2.pdf (87.8 KB)

And herewith an emonPi.local Sync Feeds page.

EmonPi- sync view.pdf (113.9 KB)

haffle · 12 November 2019 10:31

Hi All,

Sorry to be a nuisance, but I’d really like to put this to bed. The essence of the issue is sync’ing past data back from emoncms.org to my emonPi.local , so that I can then route future data from my emonPi & x3 Tx’s to the emonPi, as this will allow me to make backups etc etc.

Last week I downloaded two feeds within emonPi.local > Emoncms Sync: Feeds > Download. These feeds then showed “Local behind Remote by xyz points"; looking at it again today, those values have risen to 34,409 & 33,481. But I am not sure what this is telling me. Is this correct, and therefore should I click on Download on the remaining Feeds? Or is there an error that needs sorting prior to Downloading more Feeds?

Am I right to think that when I have successfully sync’d all of the Feed’s past data I should then tell the three Tx’s to (via the espwifiXXXX log in page) to send future data to the Pi via either the Emoncms section or the MQTT section? I’m not quite sure how they differ.

TIA

TrystanLea · 12 November 2019 11:19

Hello Julian

When you first downloaded the feeds and saw “Local behind Remote by 905 points” this means that there are 905 points on emoncms.org that you can download to bring your local copy up to date.

905 points x 10s interval = 9050 seconds or 2.5 hours of data-logging.

Your more recent values 34,409 x 10s interval = 344090 seconds ~ 4 days (as expected from your posts)

To download the new data just click on Download again. The sync module does not automatically keep the local feeds up to date.

haffle · 12 November 2019 11:34

Thanks @TrystanLea; now I understand what it is saying. Do I take it that what I now need to do is …

Download all of the Feeds within Emoncms Sync in turn, and then
Wait a couple of minutes for that to populate across, and then
Tell the x3 Tx’s to send future data to the Pi. ***

Presumably I will then loose a short period’s worth of data, but this won’t be an issue in terms of data completeness / accuracy.

*** If so, should I do this through the Emoncms section or the MQTT section of the espwifiXXXX page?

If via Emoncms, would I be best to use something like /emoncms/house , /emoncms/garage etc etc in the Path field, or just use /emoncms for all Tx’s?
If via MQTT, it looks that I should use emon/house , emon/garage etc in the Base-topic field. Am I best to leave the Username & Password fields blank? Does authentication cause issues elsewhere?

TIA