I noticed today some issues with data coming from my EmonTX and in the process of investigating this found something else.
I’m currently really unlucky, in that the timing of the data coming into emonhub, is right on the EmonCMS feed boundary. The result is, as the data coming in oscillates around the boundary (natural variation of timing), some values are missed and others overwritten. I am using an older version of emonhub so this may have been noticed and fixed.
This is the emonhub log:
2019-12-02 16:28:50,094 DEBUG 14166 NEW FRAME : 1575304130.09 10 148 6 0 0 0 0 0 0 141 94 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-02 16:28:50,100 DEBUG 14166 Timestamp : 1575304130.09
2019-12-02 16:28:50,102 DEBUG 14166 Node : 10
2019-12-02 16:28:50,104 DEBUG 14166 Values : [1684, 0, 0, 0, 24205, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0]
2019-12-02 16:28:50,136 DEBUG 14166 Append to 'emonCMS47' buffer => time: 1575304130.09, data: [10, 1684, 0, 0, 0, 24205, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14166
2019-12-02 16:28:50,183 DEBUG 14166 Append to 'emonCMS77' buffer => time: 1575304130.09, data: [10, 1684, 0, 0, 0, 24205, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14166
2019-12-02 16:28:50,239 INFO emonCMS47 sending: http://192.168.7.47/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304130.09,10,1684,0,0,0,24205,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304130
2019-12-02 16:28:50,284 INFO emonCMS77 sending: http://192.168.7.77/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304130.09,10,1684,0,0,0,24205,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304130
2019-12-02 16:28:50,352 DEBUG emonCMS47 acknowledged receipt with 'ok' from http://192.168.7.47/emoncms
2019-12-02 16:28:50,436 DEBUG emonCMS77 acknowledged receipt with 'ok' from http://192.168.7.77/emoncms
2019-12-02 16:28:59,986 DEBUG 14167 NEW FRAME : 1575304139.99 10 145 6 0 0 0 0 0 0 136 94 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-02 16:28:59,992 DEBUG 14167 Timestamp : 1575304139.99
2019-12-02 16:28:59,994 DEBUG 14167 Node : 10
2019-12-02 16:28:59,996 DEBUG 14167 Values : [1681, 0, 0, 0, 24200, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0]
2019-12-02 16:29:00,028 DEBUG 14167 Append to 'emonCMS77' buffer => time: 1575304139.99, data: [10, 1681, 0, 0, 0, 24200, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14167
2019-12-02 16:29:00,030 DEBUG 14167 Append to 'emonCMS47' buffer => time: 1575304139.99, data: [10, 1681, 0, 0, 0, 24200, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14167
2019-12-02 16:29:00,133 INFO emonCMS77 sending: http://192.168.7.77/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304139.99,10,1681,0,0,0,24200,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304140
2019-12-02 16:29:00,135 INFO emonCMS47 sending: http://192.168.7.47/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304139.99,10,1681,0,0,0,24200,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304140
2019-12-02 16:29:00,263 DEBUG emonCMS47 acknowledged receipt with 'ok' from http://192.168.7.47/emoncms
2019-12-02 16:29:00,272 DEBUG emonCMS77 acknowledged receipt with 'ok' from http://192.168.7.77/emoncms
2019-12-02 16:29:09,995 DEBUG 14168 NEW FRAME : 1575304149.99 10 140 6 0 0 0 0 0 0 111 94 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-02 16:29:10,001 DEBUG 14168 Timestamp : 1575304149.99
2019-12-02 16:29:10,003 DEBUG 14168 Node : 10
2019-12-02 16:29:10,005 DEBUG 14168 Values : [1676, 0, 0, 0, 24175, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0]
2019-12-02 16:29:10,057 DEBUG 14168 Append to 'emonCMS77' buffer => time: 1575304149.99, data: [10, 1676, 0, 0, 0, 24175, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14168
2019-12-02 16:29:10,063 DEBUG 14168 Append to 'emonCMS47' buffer => time: 1575304149.99, data: [10, 1676, 0, 0, 0, 24175, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14168
2019-12-02 16:29:10,160 INFO emonCMS77 sending: http://192.168.7.77/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304149.99,10,1676,0,0,0,24175,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304150
2019-12-02 16:29:10,164 INFO emonCMS47 sending: http://192.168.7.47/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304149.99,10,1676,0,0,0,24175,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304150
2019-12-02 16:29:10,284 DEBUG emonCMS47 acknowledged receipt with 'ok' from http://192.168.7.47/emoncms
2019-12-02 16:29:10,288 DEBUG emonCMS77 acknowledged receipt with 'ok' from http://192.168.7.77/emoncms
2019-12-02 16:29:20,038 DEBUG 14169 NEW FRAME : 1575304160.04 10 147 6 0 0 0 0 0 0 138 94 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-02 16:29:20,040 DEBUG 14169 Timestamp : 1575304160.04
2019-12-02 16:29:20,041 DEBUG 14169 Node : 10
2019-12-02 16:29:20,041 DEBUG 14169 Values : [1683, 0, 0, 0, 24202, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0]
2019-12-02 16:29:20,111 DEBUG 14169 Append to 'emonCMS47' buffer => time: 1575304160.04, data: [10, 1683, 0, 0, 0, 24202, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14169
2019-12-02 16:29:20,113 DEBUG 14169 Append to 'emonCMS77' buffer => time: 1575304160.04, data: [10, 1683, 0, 0, 0, 24202, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14169
2019-12-02 16:29:20,215 INFO emonCMS47 sending: http://192.168.7.47/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304160.04,10,1683,0,0,0,24202,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304160
2019-12-02 16:29:20,217 INFO emonCMS77 sending: http://192.168.7.77/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304160.04,10,1683,0,0,0,24202,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304160
2019-12-02 16:29:20,348 DEBUG emonCMS77 acknowledged receipt with 'ok' from http://192.168.7.77/emoncms
2019-12-02 16:29:20,354 DEBUG emonCMS47 acknowledged receipt with 'ok' from http://192.168.7.47/emoncms
2019-12-02 16:29:23,745 DEBUG 14170 NEW FRAME : 1575304163.74 19 205 0 0 0 108 1 23 0
2019-12-02 16:29:23,749 DEBUG 14170 Timestamp : 1575304163.74
2019-12-02 16:29:23,750 DEBUG 14170 Node : 19
2019-12-02 16:29:23,752 DEBUG 14170 Values : [205, 0, 364, 23]
2019-12-02 16:29:23,790 DEBUG 14170 Append to 'emonCMS77' buffer => time: 1575304163.74, data: [19, 205, 0, 364, 23], ref: 14170
2019-12-02 16:29:23,793 DEBUG 14170 Append to 'emonCMS47' buffer => time: 1575304163.74, data: [19, 205, 0, 364, 23], ref: 14170
2019-12-02 16:29:23,893 INFO emonCMS77 sending: http://192.168.7.77/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304163.74,19,205,0,364,23]]&sentat=1575304163
2019-12-02 16:29:23,896 INFO emonCMS47 sending: http://192.168.7.47/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304163.74,19,205,0,364,23]]&sentat=1575304163
2019-12-02 16:29:24,020 DEBUG emonCMS77 acknowledged receipt with 'ok' from http://192.168.7.77/emoncms
2019-12-02 16:29:24,025 DEBUG emonCMS47 acknowledged receipt with 'ok' from http://192.168.7.47/emoncms
2019-12-02 16:29:29,940 DEBUG 14171 NEW FRAME : 1575304169.94 10 146 6 0 0 0 0 0 0 151 94 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-02 16:29:29,942 DEBUG 14171 Timestamp : 1575304169.94
2019-12-02 16:29:29,942 DEBUG 14171 Node : 10
2019-12-02 16:29:29,943 DEBUG 14171 Values : [1682, 0, 0, 0, 24215, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0]
2019-12-02 16:29:29,985 DEBUG 14171 Append to 'emonCMS47' buffer => time: 1575304169.94, data: [10, 1682, 0, 0, 0, 24215, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14171
2019-12-02 16:29:30,005 DEBUG 14171 Append to 'emonCMS77' buffer => time: 1575304169.94, data: [10, 1682, 0, 0, 0, 24215, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14171
2019-12-02 16:29:30,087 INFO emonCMS47 sending: http://192.168.7.47/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304169.94,10,1682,0,0,0,24215,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304170
2019-12-02 16:29:30,107 INFO emonCMS77 sending: http://192.168.7.77/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304169.94,10,1682,0,0,0,24215,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304170
2019-12-02 16:29:30,203 DEBUG emonCMS47 acknowledged receipt with 'ok' from http://192.168.7.47/emoncms
2019-12-02 16:29:30,229 DEBUG emonCMS77 acknowledged receipt with 'ok' from http://192.168.7.77/emoncms
2019-12-02 16:29:39,877 DEBUG 14172 NEW FRAME : 1575304179.88 10 149 6 0 0 0 0 0 0 153 94 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-02 16:29:39,879 DEBUG 14172 Timestamp : 1575304179.88
2019-12-02 16:29:39,879 DEBUG 14172 Node : 10
2019-12-02 16:29:39,880 DEBUG 14172 Values : [1685, 0, 0, 0, 24217, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0]
2019-12-02 16:29:39,890 DEBUG 14172 Append to 'emonCMS77' buffer => time: 1575304179.88, data: [10, 1685, 0, 0, 0, 24217, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14172
2019-12-02 16:29:39,952 DEBUG 14172 Append to 'emonCMS47' buffer => time: 1575304179.88, data: [10, 1685, 0, 0, 0, 24217, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14172
2019-12-02 16:29:39,991 INFO emonCMS77 sending: http://192.168.7.77/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304179.88,10,1685,0,0,0,24217,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304179
2019-12-02 16:29:40,053 INFO emonCMS47 sending: http://192.168.7.47/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304179.88,10,1685,0,0,0,24217,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304180
2019-12-02 16:29:40,109 DEBUG emonCMS77 acknowledged receipt with 'ok' from http://192.168.7.77/emoncms
2019-12-02 16:29:40,169 DEBUG emonCMS47 acknowledged receipt with 'ok' from http://192.168.7.47/emoncms
2019-12-02 16:29:50,014 DEBUG 14173 NEW FRAME : 1575304190.01 10 148 6 0 0 0 0 0 0 151 94 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-02 16:29:50,021 DEBUG 14173 Timestamp : 1575304190.01
2019-12-02 16:29:50,023 DEBUG 14173 Node : 10
2019-12-02 16:29:50,025 DEBUG 14173 Values : [1684, 0, 0, 0, 24215, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0]
2019-12-02 16:29:50,076 DEBUG 14173 Append to 'emonCMS47' buffer => time: 1575304190.01, data: [10, 1684, 0, 0, 0, 24215, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14173
2019-12-02 16:29:50,116 DEBUG 14173 Append to 'emonCMS77' buffer => time: 1575304190.01, data: [10, 1684, 0, 0, 0, 24215, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14173
2019-12-02 16:29:50,180 INFO emonCMS47 sending: http://192.168.7.47/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304190.01,10,1684,0,0,0,24215,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304190
2019-12-02 16:29:50,220 INFO emonCMS77 sending: http://192.168.7.77/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304190.01,10,1684,0,0,0,24215,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304190
2019-12-02 16:29:50,337 DEBUG emonCMS77 acknowledged receipt with 'ok' from http://192.168.7.77/emoncms
2019-12-02 16:29:50,404 DEBUG emonCMS47 acknowledged receipt with 'ok' from http://192.168.7.47/emoncms
2019-12-02 16:29:59,924 DEBUG 14174 NEW FRAME : 1575304199.92 10 146 6 0 0 0 0 0 0 146 94 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-02 16:29:59,931 DEBUG 14174 Timestamp : 1575304199.92
2019-12-02 16:29:59,932 DEBUG 14174 Node : 10
2019-12-02 16:29:59,934 DEBUG 14174 Values : [1682, 0, 0, 0, 24210, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0]
2019-12-02 16:29:59,967 DEBUG 14174 Append to 'emonCMS47' buffer => time: 1575304199.92, data: [10, 1682, 0, 0, 0, 24210, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14174
2019-12-02 16:29:59,987 DEBUG 14174 Append to 'emonCMS77' buffer => time: 1575304199.92, data: [10, 1682, 0, 0, 0, 24210, 30000, 30000, 30000, 30000, 30000, 30000, 0, 0], ref: 14174
2019-12-02 16:30:00,070 INFO emonCMS47 sending: http://192.168.7.47/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304199.92,10,1682,0,0,0,24210,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304200
2019-12-02 16:30:00,091 INFO emonCMS77 sending: http://192.168.7.77/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1575304199.92,10,1682,0,0,0,24210,30000,30000,30000,30000,30000,30000,0,0]]&sentat=1575304200
2019-12-02 16:30:00,193 DEBUG emonCMS47 acknowledged receipt with 'ok' from http://192.168.7.47/emoncms
2019-12-02 16:30:00,213 DEBUG emonCMS77 acknowledged receipt with 'ok' from http://192.168.7.77/emoncms
and what I see recorded is
2019-12-02 16:28:50, 1684.0
2019-12-02 16:29:00, 1676.0
2019-12-02 16:29:10, null
2019-12-02 16:29:20, 1683.0
2019-12-02 16:29:30, 1685.0
2019-12-02 16:29:40, null
2019-12-02 16:29:50, 1684.0
2019-12-02 16:30:00, 1678.0
Because the sent data is oscillating across the recording boundary, I’m missing values quite regularly.
It also seems to me that emoncms ignores the sentat value. From the logs, using this value might smooth out some of the variation.
Two ways I can think of stopping this;
- throttle the sending to the feed interval
- and/or avoid the usual boundaries when sending data.
A throttle time would be calculated from the time of the first received value + the specified interval on that interface rather than just a minimum time gap otherwise there would be drift introduced every time the data was over the interval time. A running ‘next send at’ value.
Alternatively (or in addition), as emoncms seems to always use 5 or 10 second Feed intervals, emonhub steers well away from those and uses timestamps ending in 2 or 7.
For now I’m going to reboot the TX and see if I’m lucky and move the sending time away from a boundary!