Problem with data from MQTT feed seeming to be "lost"

Tom_sky · 2022-06-06T14:41:54.486Z

I hope I’m putting this query in the rignt place.

I’ve got some data coming in to Emoncms from some temperature sensors. I’m just at the stage of checking/calibrating them, so I’ve got them all hooked up to the same remote sensor, and then dumping data into Emoncms via MQTT/Node Red/MQTT again (all the same Emonpi MQTT broker).

The data appears to be arriving ok as inputs, and then setup as feeds. However, after a while, some of the data appears to be forgotten, leading to graphs like this one:

image1127×570 61.8 KB

The data looks like this:


"Unix timestamp", "test:Temperature1_D577C60264FF", "test:Temperature1_3C41F649D7BE", "test:Temperature1_3CF9F649B585", "test:Temperature1_3C79F6482613"
1654504290, 18.6, 19.0, 19.2, 18.4
1654504320, 18.6, 19.0, 19.2, 18.4
1654504350, 18.6, 19.0, 19.2, 18.4
1654504380, 18.6, 19.0, 19.2, 18.4
1654504410, 18.5, 19.0, 19.2, 18.3
1654504440, 18.6, 19.0, 19.2, 18.4
1654504470, 18.6, 19.0, 19.2, 18.4
1654504500, null, null, null, null
1654504530, null, null, null, null
1654504560, null, null, null, null
1654504590, null, null, null, null
1654504620, 19.2, null, null, null
1654504650, 19.1, null, null, null
1654504680, 19.1, null, null, null
1654504710, 19.0, null, null, null
1654504740, 18.9, null, null, null
1654504770, 18.9, null, null, null
1654504800, 18.9, null, null, null
1654504830, 18.8, null, null, null
1654504860, 18.8, null, null, null
1654504890, 18.8, null, null, null
1654504920, 18.7, null, null, null
1654504950, 18.7, null, null, null
1654504980, 18.6, null, null, null
1654505010, 18.6, null, null, null
1654505040, 18.6, null, null, null
1654505070, 18.6, null, null, null
1654505100, 18.6, null, null, null
1654505130, 18.6, null, null, null
1654505160, 18.6, null, null, null
1654505190, 18.6, null, null, null
1654505220, 18.6, null, null, null
1654505250, 18.6, null, null, null
1654505280, 18.6, null, null, null
1654505310, 18.6, null, null, null
1654505340, 18.6, null, null, null
1654505370, 18.6, null, null, null
1654505400, 18.6, null, null, null
1654505430, 18.6, null, null, null
1654505460, 18.6, null, null, null
1654505490, 18.6, null, null, null
1654505520, 18.6, null, null, null
1654505550, 18.6, null, null, null
1654505580, 18.6, null, null, null
1654505610, 18.6, null, null, null
1654505640, 18.6, null, null, null
1654505670, 18.6, null, null, null
1654505700, 18.6, null, null, null
1654505730, 18.6, null, null, null
1654505760, 18.6, null, null, null
1654505790, 18.6, null, null, null
1654505820, 18.6, null, null, null
1654505850, 18.6, null, null, null
1654505880, 18.6, null, null, null
1654505910, 18.6, null, null, null
1654505940, 18.6, null, null, null
1654505970, 18.6, null, null, null
1654506000, 18.6, null, null, null
1654506030, 18.6, null, null, null
1654506060, 18.6, null, null, null
1654506090, 18.6, null, null, null
1654506120, 18.7, null, null, null
1654506150, 18.7, null, null, null
1654506180, 18.7, null, null, null
1654506210, 18.7, null, null, null
1654506240, 18.7, null, null, null
1654506270, 18.7, null, null, null
1654506300, 18.7, null, null, null
1654506330, 18.7, null, null, null
1654506360, 18.7, null, null, null
1654506390, 18.7, null, null, null
1654506420, 18.7, null, null, null
1654506450, 18.7, null, null, null
1654506480, 18.7, null, null, null
1654506510, 18.7, null, null, null
1654506540, 18.7, null, null, null
1654506570, 18.7, null, null, null
1654506600, 18.7, null, null, null
1654506630, 18.7, null, null, null
1654506660, 18.7, null, null, null
1654506690, 18.7, null, null, null
1654506720, 18.7, null, null, null
1654506750, 18.7, null, null, null
1654506780, 18.8, null, null, null
1654506810, 18.7, null, null, null
1654506840, 18.7, null, null, null
1654506870, 18.8, null, null, null
1654506900, 18.8, null, null, null
1654506930, 18.8, null, null, null
1654506960, 18.8, null, null, null
1654506990, 18.8, null, null, null
1654507020, 18.8, null, null, null
1654507050, 18.8, null, null, null
1654507080, 18.8, null, null, null
1654507110, 18.8, null, null, null
1654507140, 18.8, null, null, null
1654507170, 18.7, null, null, null
1654507200, 18.6, null, null, null
1654507230, 18.6, null, null, null
1654507260, 18.5, null, null, null
1654507290, 18.4, null, null, null
1654507320, 18.4, null, null, null
1654507350, 18.3, null, null, null
1654507380, 18.3, null, null, null
1654507410, 18.3, null, null, null
1654507440, 18.3, null, null, null
1654507470, 18.2, null, null, null
1654507500, 18.2, null, null, null
1654507530, 18.2, null, null, null
1654507560, 18.1, null, null, null
1654507590, 18.1, null, null, null
1654507620, 18.1, null, null, null
1654507650, 18.1, null, null, null
1654507680, 18.1, null, null, null
1654507710, 18.1, null, null, null
1654507740, 18.1, null, null, null
1654507770, 18.1, null, null, null
1654507800, 18.1, null, null, null
1654507830, 18.2, null, null, null
1654507860, 18.2, null, null, null
1654507890, 18.2, null, null, null
1654507920, 18.2, null, null, null
1654507950, 18.2, null, null, null
1654507980, 18.2, null, null, null
1654508010, 18.2, null, null, null
1654508040, 18.3, null, null, null
1654508070, 18.3, null, null, null
1654508100, 18.3, null, null, null
1654508130, 18.3, null, null, null
1654508160, 18.3, null, null, null
1654508190, 18.3, null, null, null
1654508220, 18.3, null, null, null
1654508250, 18.3, null, null, null
1654508280, 18.3, null, null, null
1654508310, 18.4, null, null, null
1654508340, 18.4, null, null, null
1654508370, 18.4, null, null, null
1654508400, 18.4, null, null, null
1654508430, 18.4, null, null, null
1654508460, 18.4, null, null, null
1654508490, 18.4, null, null, null
1654508520, 18.5, null, null, null
1654508550, 18.5, null, null, null
1654508580, 18.5, null, null, null
1654508610, 18.6, null, null, null
1654508640, 18.6, null, null, null
1654508670, 18.6, null, null, null
1654508700, 18.6, null, null, null
1654508730, 18.6, null, null, null
1654508760, 18.6, null, null, null
1654508790, 18.7, null, null, null
1654508820, 18.7, null, null, null
1654508850, 18.7, null, null, null
1654508880, 18.7, null, null, null
1654508910, 18.7, null, null, null
1654508940, 18.7, null, null, null
1654508970, 18.7, null, null, null
1654509000, 18.7, null, null, null
1654509030, 18.7, null, null, null
1654509060, 18.7, null, null, null
1654509090, 18.7, null, null, null
1654509120, 18.8, null, null, null
1654509150, 18.7, null, null, null
1654509180, 18.8, null, null, null
1654509210, 18.8, null, null, null
1654509240, 18.8, null, null, null
1654509270, 18.8, null, null, null
1654509300, 18.8, null, null, null
1654509330, 18.8, null, null, null
1654509360, 18.8, null, null, null
1654509390, 18.8, null, null, null
1654509420, 18.8, null, null, null
1654509450, 18.8, null, null, null
1654509480, 18.8, null, null, null
1654509510, 18.8, null, null, null
1654509540, 18.9, null, null, null
1654509570, 18.9, null, null, null
1654509600, 18.9, null, null, null
1654509630, 18.9, null, null, null
1654509660, 18.9, null, null, null
1654509690, 18.9, null, null, null
1654509720, 18.9, null, null, null
1654509750, 18.9, null, null, null
1654509780, 18.9, null, null, null
1654509810, 18.9, null, null, null
1654509840, 18.9, null, null, null
1654509870, 18.9, null, null, null
1654509900, 18.9, null, null, null
1654509930, 18.9, null, null, null
1654509960, 18.9, null, null, null
1654509990, 18.9, null, null, null
1654510020, 18.9, null, null, null
1654510050, 18.9, null, null, null
1654510080, 18.9, null, null, null
1654510110, 19.0, null, null, null
1654510140, 19.0, null, null, null
1654510170, 19.0, null, null, null
1654510200, 19.0, null, null, null
1654510230, 19.0, null, null, null
1654510260, 19.9, null, null, null
1654510290, 19.6, null, null, null
1654510320, 19.6, null, null, null
1654510350, 19.6, null, null, null
1654510380, 19.5, null, null, null
1654510410, 20.8, null, null, null
1654510440, 20.0, null, null, null
1654510470, 19.8, null, null, null
1654510500, 19.8, null, null, null
1654510530, 19.6, null, null, null
1654510560, 19.6, null, null, null
1654510590, 19.5, null, null, null
1654510620, 19.4, null, null, null
1654510650, 19.4, null, null, null
1654510680, 19.4, null, null, null
1654510710, 19.3, null, null, null
1654510740, 19.3, null, null, null
1654510770, 19.3, null, null, null
1654510800, 19.3, null, null, null
1654510830, 19.2, null, null, null
1654510860, 19.2, null, null, null
1654510890, 19.2, null, null, null
1654510920, 19.2, null, null, null
1654510950, 19.2, null, null, null
1654510980, 19.2, null, null, null
1654511010, 19.1, null, null, null
1654511040, 19.1, null, null, null
1654511070, 19.1, null, null, null
1654511100, 19.1, null, null, null
1654511130, 19.1, null, null, null
1654511160, 19.1, null, null, null
1654511190, 19.1, null, null, null
1654511220, null, null, null, null
1654511250, 19.1, null, null, null
1654511280, 19.1, null, null, null
1654511310, 19.1, null, null, null
1654511340, 19.1, null, null, null
1654511370, 19.1, null, null, null
1654511400, 19.0, null, null, null
1654511430, 19.0, null, null, null
1654511460, 19.0, null, null, null
1654511490, 19.0, null, null, null
1654511520, 19.0, null, null, null
1654511550, 19.0, null, null, null
1654511580, 19.0, null, null, null
1654511610, 19.0, null, null, null
1654511640, 19.0, null, null, null
1654511670, 19.0, null, null, null
1654511700, 19.0, null, null, null
1654511730, 19.0, null, null, null
1654511760, 19.0, null, null, null
1654511790, 19.0, null, null, null
1654511820, 19.0, null, null, null
1654511850, 19.0, null, null, null
1654511880, 19.0, null, null, null
1654511910, 19.0, null, null, null
1654511940, 19.0, null, null, null
1654511970, 19.0, null, null, null
1654512000, 19.0, null, null, null
1654512030, 19.0, null, null, null
1654512060, 19.0, null, null, null
1654512090, 19.0, null, null, null
1654512120, 19.0, null, null, null
1654512150, 19.0, null, null, null
1654512180, 19.0, null, null, null
1654512210, 19.0, null, null, null
1654512240, 19.0, null, null, null
1654512270, 19.0, null, null, null
1654512300, 19.0, null, null, null
1654512330, 19.0, null, null, null
1654512360, 19.0, null, null, null
1654512390, 19.0, null, null, null
1654512420, 19.0, null, null, null
1654512450, 19.0, null, null, null
1654512480, 19.0, null, null, null
1654512510, 18.9, null, null, null
1654512540, 19.0, null, null, null
1654512570, 19.0, null, null, null
1654512600, 19.0, null, null, null
1654512630, 19.0, null, null, null
1654512660, 19.0, null, null, null
1654512690, 18.9, null, null, null
1654512720, 19.0, null, null, null
1654512750, 19.0, null, null, null
1654512780, 19.0, null, null, null
1654512810, 19.0, null, null, null
1654512840, 18.9, null, null, null
1654512870, 19.0, null, null, null
1654512900, 18.9, null, null, null
1654512930, 19.0, null, null, null
1654512960, 18.9, null, null, null
1654512990, 18.9, null, null, null
1654513020, 18.9, null, null, null
1654513050, 19.0, null, null, null
1654513080, 18.9, null, null, null
1654513110, 19.0, null, null, null
1654513140, 19.0, null, null, null
1654513170, 19.0, null, null, null
1654513200, 19.0, null, null, null
1654513230, 19.0, null, null, null
1654513260, 19.0, null, null, null
1654513290, 19.0, null, null, null
1654513320, 19.0, null, null, null
1654513350, 19.0, null, null, null
1654513380, 19.0, null, null, null
1654513410, 19.0, null, null, null
1654513440, 19.0, null, null, null
1654513470, 19.0, null, null, null
1654513500, 19.0, null, null, null
1654513530, 19.0, null, null, null
1654513560, 18.9, null, null, null
1654513590, 18.9, null, null, null
1654513620, 18.9, null, null, null
1654513650, 18.9, null, null, null
1654513680, 18.9, null, null, null
1654513710, 18.9, null, null, null
1654513740, 18.9, null, null, null
1654513770, 18.9, null, null, null
1654513800, 18.9, null, null, null
1654513830, 18.9, null, null, null
1654513860, 18.9, null, null, null
1654513890, 18.9, null, null, null
1654513920, 18.9, null, null, null
1654513950, 18.9, null, null, null
1654513980, 18.9, null, null, null
1654514010, 18.9, null, null, null
1654514040, 18.9, null, null, null
1654514070, 18.9, null, null, null
1654514100, 18.9, null, null, null
1654514130, 18.9, null, null, null
1654514160, 18.9, null, null, null
1654514190, 18.9, null, null, null
1654514220, 18.9, null, null, null
1654514250, 18.9, null, null, null
1654514280, 18.9, null, null, null
1654514310, 18.9, null, null, null
1654514340, 18.9, null, null, null
1654514370, 18.9, null, null, null
1654514400, 18.9, null, null, null
1654514430, 18.9, null, null, null
1654514460, 18.9, null, null, null
1654514490, 18.9, null, null, null
1654514520, 18.9, null, null, null
1654514550, 18.9, null, null, null
1654514580, 18.9, null, null, null
1654514610, 18.9, null, null, null
1654514640, 18.9, null, null, null
1654514670, 18.9, null, null, null
1654514700, 19.0, null, null, null
1654514730, 19.0, null, null, null
1654514760, 19.0, null, null, null
1654514790, 19.0, null, null, null
1654514820, 19.0, null, null, null
1654514850, 19.0, null, null, null
1654514880, 19.0, null, null, null
1654514910, 19.0, null, null, null
1654514940, 19.0, null, null, null
1654514970, 19.0, null, null, null
1654515000, 19.0, null, null, null
1654515030, 19.0, null, null, null
1654515060, 19.0, null, null, null
1654515090, 19.1, null, null, null
1654515120, 19.1, null, null, null
1654515150, 19.1, null, null, null
1654515180, 19.1, null, null, null
1654515210, 19.1, null, null, null
1654515240, 19.1, null, null, null
1654515270, 19.1, null, null, null
1654515300, 19.1, null, null, null
1654515330, 19.1, null, null, null
1654515360, 19.1, null, null, null
1654515390, 19.1, null, null, null
1654515420, 19.1, null, null, null
1654515450, 19.1, null, null, null
1654515480, 19.1, null, null, null
1654515510, 19.1, null, null, null
1654515540, 19.2, null, null, null
1654515570, 19.2, null, null, null
1654515600, 19.2, null, null, null
1654515630, 19.2, null, null, null
1654515660, 19.2, null, null, null
1654515690, 19.2, null, null, null
1654515720, 19.2, null, null, null
1654515750, 19.2, null, null, null
1654515780, 19.2, null, null, null
1654515810, 19.2, null, null, null
1654515840, 19.2, null, null, null
1654515870, 19.2, null, null, null
1654515900, 19.2, null, null, null
1654515930, 19.2, null, null, null
1654515960, 19.2, null, null, null
1654515990, 19.3, null, null, null
1654516020, 19.3, null, null, null
1654516050, 19.3, null, null, null
1654516080, 19.3, null, null, null
1654516110, 19.3, null, null, null
1654516140, 19.3, null, null, null
1654516170, 19.3, null, null, null
1654516200, 19.3, null, null, null
1654516230, 19.3, null, null, null
1654516260, 19.3, null, null, null
1654516290, 19.3, null, null, null
1654516320, 19.3, null, null, null
1654516350, 19.3, null, null, null
1654516380, 19.3, null, null, null
1654516410, 19.3, null, null, null
1654516440, 19.3, null, null, null
1654516470, 19.3, null, null, null
1654516500, 19.3, null, null, null
1654516530, 19.3, null, null, null
1654516560, 19.3, null, null, null
1654516590, 19.3, null, null, null
1654516620, 19.3, null, null, null
1654516650, 19.3, null, null, null
1654516680, 19.3, null, null, null
1654516710, 19.3, null, null, null
1654516740, 19.3, null, null, null
1654516770, 19.3, null, null, null
1654516800, 19.3, null, null, null
1654516830, 19.3, null, null, null
1654516860, 19.3, null, null, null
1654516890, 19.3, null, null, null
1654516920, 19.3, null, null, null
1654516950, 19.4, null, null, null
1654516980, 19.4, null, null, null
1654517010, 19.4, null, null, null
1654517040, 19.4, null, null, null
1654517070, 19.4, null, null, null
1654517100, 19.4, null, null, null
1654517130, 19.4, null, null, null
1654517160, 19.4, null, null, null
1654517190, 19.4, null, null, null
1654517220, 19.4, null, null, null
1654517250, 19.4, null, null, null
1654517280, 19.4, null, null, null
1654517310, 19.4, null, null, null
1654517340, 19.4, null, null, null
1654517370, 19.4, null, null, null
1654517400, 19.4, null, null, null
1654517430, 19.4, null, null, null
1654517460, 19.4, null, null, null
1654517490, 19.4, null, null, null
1654517520, 19.4, null, null, null
1654517550, 19.4, null, null, null
1654517580, 19.4, null, null, null
1654517610, 19.4, null, null, null
1654517640, 19.4, null, null, null
1654517670, 19.4, null, null, null
1654517700, 19.4, null, null, null
1654517730, 19.4, null, null, null
1654517760, 19.4, null, null, null
1654517790, 19.4, null, null, null
1654517820, 19.4, null, null, null
1654517850, 19.4, null, null, null
1654517880, 19.4, null, null, null
1654517910, 19.4, null, null, null
1654517940, 19.5, null, null, null
1654517970, 19.4, null, null, null
1654518000, 19.4, null, null, null
1654518030, 19.4, null, null, null
1654518060, 19.5, null, null, null
1654518090, 19.5, null, null, null
1654518120, 19.5, null, null, null
1654518150, 19.5, null, null, null
1654518180, 19.5, null, null, null
1654518210, 19.5, null, null, null
1654518240, 19.5, null, null, null
1654518270, 19.5, null, null, null
1654518300, 19.5, null, null, null
1654518330, 19.5, null, null, null
1654518360, 19.5, null, null, null
1654518390, 19.5, null, null, null
1654518420, 19.5, null, null, null
1654518450, 19.5, null, null, null
1654518480, 19.5, null, null, null
1654518510, 19.6, null, null, null
1654518540, 19.6, null, null, null
1654518570, 19.6, null, null, null
1654518600, 19.6, null, null, null
1654518630, 19.6, null, null, null
1654518660, 19.6, null, null, null
1654518690, 19.6, null, null, null
1654518720, 19.6, null, null, null
1654518750, 19.6, null, null, null
1654518780, 19.6, null, null, null
1654518810, 19.6, null, null, null
1654518840, 19.6, null, null, null
1654518870, 19.6, null, null, null
1654518900, 19.6, null, null, null
1654518930, 19.6, null, null, null
1654518960, 19.6, null, null, null
1654518990, 19.6, null, null, null
1654519020, 19.6, null, null, null
1654519050, 19.6, null, null, null
1654519080, 19.6, null, null, null
1654519110, 19.6, null, null, null
1654519140, 19.7, null, null, null
1654519170, 19.7, null, null, null
1654519200, 19.7, null, null, null
1654519230, 19.7, null, null, null
1654519260, 19.7, null, null, null
1654519290, 19.7, null, null, null
1654519320, 19.7, null, null, null
1654519350, 19.7, null, null, null
1654519380, 19.7, null, null, null
1654519410, 19.7, null, null, null
1654519440, 19.7, null, null, null
1654519470, 19.7, null, null, null
1654519500, 19.7, null, null, null
1654519530, 19.7, null, null, null
1654519560, 19.7, null, null, null
1654519590, 19.8, null, null, null
1654519620, 19.8, null, null, null
1654519650, 19.8, null, null, null
1654519680, 19.8, null, null, null
1654519710, 19.8, null, null, null
1654519740, 19.8, null, null, null
1654519770, 19.8, null, null, null
1654519800, 19.8, null, null, null
1654519830, 19.8, null, null, null
1654519860, 19.8, null, null, null
1654519890, 19.8, null, null, null
1654519920, 19.8, null, null, null
1654519950, 19.8, null, null, null
1654519980, 19.8, null, null, null
1654520010, null, null, null, null
1654520040, null, null, null, null
1654520070, null, null, null, null
1654520100, null, null, null, null
1654520130, null, null, null, null
1654520160, null, null, null, null
1654520190, null, null, null, null
1654520220, null, null, null, null
1654520250, null, null, null, null
1654520280, 19.8, null, null, null
1654520310, 19.8, null, null, null
1654520340, 19.8, null, null, null
1654520370, 19.8, null, null, null
1654520400, 19.8, null, null, null
1654520430, 19.8, null, null, null
1654520460, 19.9, null, null, null
1654520490, 19.8, null, null, null
1654520520, 19.9, null, null, null
1654520550, 19.8, null, null, null
1654520580, 19.8, 19.9, 20.1, 19.6
1654520610, 19.8, null, null, null
1654520640, 19.8, null, null, null
1654520670, 19.8, null, null, null
1654520700, 19.8, null, null, null
1654520730, 19.8, null, null, null
1654520760, 19.8, null, null, null
1654520790, 19.8, null, null, null
1654520820, 19.8, null, null, null
1654520850, 19.8, null, null, null
1654520880, 19.8, null, null, null
1654520910, 19.8, null, null, null
1654520940, 19.8, null, null, null
1654520970, 19.8, null, null, null
1654521000, 19.8, null, null, null
1654521030, 19.8, null, null, null
1654521060, 19.7, null, null, null
1654521090, 19.7, null, null, null
1654521120, 19.7, null, null, null
1654521150, 19.8, null, null, null
1654521180, 19.7, null, null, null
1654521210, 19.7, null, null, null
1654521240, 19.7, null, null, null
1654521270, 19.7, null, null, null
1654521300, 19.7, null, null, null
1654521330, 19.7, null, null, null
1654521360, 19.7, null, null, null
1654521390, 19.7, null, null, null
1654521420, 19.7, null, null, null
1654521450, 19.7, null, null, null
1654521480, 19.7, null, null, null
1654521510, 19.6, null, null, null
1654521540, 19.6, null, null, null
1654521570, 19.7, null, null, null
1654521600, 19.6, null, null, null
1654521630, 19.6, null, null, null
1654521660, 19.6, null, null, null
1654521690, 19.6, null, null, null
1654521720, 19.6, null, null, null
1654521750, 19.6, null, null, null
1654521780, 19.6, null, null, null
1654521810, 19.6, null, null, null
1654521840, 19.6, null, null, null
1654521870, 19.6, null, null, null
1654521900, 19.6, null, null, null
1654521930, 19.6, null, null, null
1654521960, 19.6, null, null, null
1654521990, 19.6, null, null, null
1654522020, 19.6, null, null, null
1654522050, 19.6, null, null, null
1654522080, 19.6, null, null, null
1654522110, 19.6, null, null, null
1654522140, 19.6, null, null, null
1654522170, 19.6, null, null, null
1654522200, 19.6, null, null, null
1654522230, 19.6, null, null, null
1654522260, 19.6, null, null, null
1654522290, 19.6, null, null, null
1654522320, 19.6, null, null, null
1654522350, 19.6, null, null, null
1654522380, 19.6, null, null, null
1654522410, 19.5, null, null, null
1654522440, 19.6, null, null, null
1654522470, 19.5, null, null, null
1654522500, 19.5, null, null, null
1654522530, 19.5, null, null, null
1654522560, 19.5, null, null, null
1654522590, 19.5, null, null, null
1654522620, 19.5, null, null, null
1654522650, 19.5, null, null, null
1654522680, 19.5, null, null, null
1654522710, 19.5, null, null, null
1654522740, 19.5, null, null, null
1654522770, 19.5, null, null, null
1654522800, 19.5, null, null, null
1654522830, 19.5, null, null, null
1654522860, 19.5, null, null, null
1654522890, 19.5, null, null, null
1654522920, 19.5, null, null, null
1654522950, 19.5, null, null, null
1654522980, 19.5, null, null, null
1654523010, 19.5, null, null, null
1654523040, 19.5, null, null, null
1654523070, 19.5, null, null, null
1654523100, 19.5, null, null, null
1654523130, 19.4, null, null, null
1654523160, 19.5, null, null, null
1654523190, 19.4, null, null, null
1654523220, 19.5, null, null, null
1654523250, 19.5, null, null, null
1654523280, 19.5, null, null, null
1654523310, 19.4, null, null, null
1654523340, 19.4, null, null, null
1654523370, 19.5, null, null, null
1654523400, 19.4, null, null, null
1654523430, 19.4, null, null, null
1654523460, 19.4, null, null, null
1654523490, 19.4, null, null, null
1654523520, 19.4, null, null, null
1654523550, 19.4, null, null, null
1654523580, 19.4, null, null, null
1654523610, 19.4, null, null, null
1654523640, 19.4, null, null, null
1654523670, 19.4, null, null, null
1654523700, 19.4, null, null, null
1654523730, 19.4, null, null, null
1654523760, 19.4, null, null, null
1654523790, 19.4, null, null, null
1654523820, 19.4, null, null, null
1654523850, 19.4, null, null, null
1654523880, 19.4, null, null, null
1654523910, 19.4, null, null, null
1654523940, 19.4, null, null, null
1654523970, 19.4, null, null, null
1654524000, 19.4, null, null, null
1654524030, 19.4, null, null, null
1654524060, 19.4, null, null, null
1654524090, 19.4, null, null, null
1654524120, 19.4, null, null, null
1654524150, 19.4, null, null, null
1654524180, 19.4, null, null, null
1654524210, 19.4, null, null, null
1654524240, 19.4, null, null, null
1654524270, 19.4, null, null, null
1654524300, 19.4, null, null, null
1654524330, 19.4, null, null, null
1654524360, 19.4, null, null, null
1654524390, 19.4, null, null, null
1654524420, 19.4, null, null, null
1654524450, 19.4, null, null, null
1654524480, 19.4, null, null, null
1654524510, 19.4, null, null, null
1654524540, 19.4, null, null, null
1654524570, 19.4, null, null, null
1654524600, 19.4, null, null, null
1654524630, 19.4, null, null, null
1654524660, 19.4, null, null, null
1654524690, 19.4, null, null, null
1654524720, 19.4, null, null, null
1654524750, 19.4, null, null, null
1654524780, 19.4, null, null, null
1654524810, 19.4, null, null, null
1654524840, 19.4, null, null, null
1654524870, 19.4, null, null, null
1654524900, 19.4, null, null, null
1654524930, 19.4, null, null, null
1654524960, 19.4, null, null, null
1654524990, 19.4, null, null, null
1654525020, 19.4, null, null, null
1654525050, 19.4, null, null, null
1654525080, 19.4, null, null, null
1654525110, 19.4, null, null, null
1654525140, 19.4, null, null, null
1654525170, 19.4, null, null, null
1654525200, 19.4, null, null, null
1654525230, 19.4, null, null, null
1654525260, 19.4, null, null, null
1654525290, 19.4, null, null, null
1654525320, 19.4, null, null, null
1654525350, 19.4, null, null, null
1654525380, 19.4, null, null, null
1654525410, 19.4, null, null, null
1654525440, 19.4, null, null, null
1654525470, 19.4, null, null, null
1654525500, 19.4, null, null, null
1654525530, 19.4, null, null, null
1654525560, 19.4, null, null, null
1654525590, 19.4, null, null, null
1654525620, 19.4, null, null, null
1654525650, 19.4, 19.6, 19.6, 19.1
1654525680, 19.4, 19.6, 19.6, 19.1
1654525710, 19.4, 19.6, 19.6, 19.1
1654525740, 19.4, 19.6, 19.6, 19.1
1654525770, 19.4, 19.6, 19.6, 19.2
1654525800, 19.4, 19.6, 19.6, 19.2
1654525830, 19.4, 19.6, 19.6, 19.2
1654525860, 19.4, 19.6, 19.7, 19.2
1654525890, 19.4, 19.6, 19.7, 19.3
1654525920, null, null, null, null

After a while, the data at e.g. 1654525650 will be forgotten, but the last few data points, whatever they are, are always available. Also, one of the feeds seems to be ok?

I’ve had a google and a search trhough the forum and I can’t seem to find anyone having similar issues.

Has anyone seen anything like this before?

Tank · 2022-06-06T18:57:18.136Z

MQTT Node Red and missing topics - Integrations / NodeRED - OpenEnergyMonitor Community

Tom_sky · 2022-06-06T19:32:26.715Z

On the face of it, that seems to be an unrelated problem.

The data is getting into Emoncms, it’s just that Emoncms is “forgetting” the data. For example, the data above re-read now:

"Unix timestamp", "test1:Temperature1_D577C60264FF", "test1:Temperature1_3C41F649D7BE", "test1:Temperature1_3CF9F649B585", "test1:Temperature1_3C79F6482613"
1654525290, 19.4, null, null, null
1654525300, 19.4, null, null, null
1654525310, 19.4, null, null, null
1654525320, 19.4, null, null, null
1654525330, 19.4, null, null, null
1654525340, 19.4, null, null, null
1654525350, 19.4, null, null, null
1654525360, 19.4, null, null, null
1654525370, 19.4, null, null, null
1654525380, 19.4, null, null, null
1654525390, 19.4, null, null, null
1654525400, 19.4, null, null, null
1654525410, 19.4, null, null, null
1654525420, 19.4, null, null, null
1654525430, 19.4, null, null, null
1654525440, 19.4, null, null, null
1654525450, 19.4, null, null, null
1654525460, 19.4, null, null, null
1654525470, 19.4, null, null, null
1654525480, 19.4, null, null, null
1654525490, 19.4, null, null, null
1654525500, 19.4, null, null, null
1654525510, 19.4, null, null, null
1654525520, 19.4, null, null, null
1654525530, 19.4, null, null, null
1654525540, 19.4, null, null, null
1654525550, 19.4, null, null, null
1654525560, 19.4, null, null, null
1654525570, 19.4, null, null, null
1654525580, 19.4, null, null, null
1654525590, 19.4, null, null, null
1654525600, 19.4, null, null, null
1654525610, 19.4, null, null, null
1654525620, 19.4, null, null, null
1654525630, 19.4, null, null, null
1654525640, 19.4, null, null, null
1654525650, 19.4, null, null, null
1654525660, 19.4, null, null, null
1654525670, 19.4, null, null, null
1654525680, 19.4, null, null, null
1654525690, 19.4, null, null, null
1654525700, 19.4, null, null, null
1654525710, 19.4, null, null, null
1654525720, 19.4, null, null, null
1654525730, 19.4, null, null, null
1654525740, 19.4, null, null, null
1654525750, 19.4, null, null, null
1654525760, 19.4, null, null, null
1654525770, 19.4, null, null, null
1654525780, 19.4, null, null, null
1654525790, 19.4, null, null, null
1654525800, 19.4, null, null, null
1654525810, 19.4, null, null, null
1654525820, 19.4, null, null, null
1654525830, 19.4, null, null, null
1654525840, 19.4, null, null, null
1654525850, 19.4, null, null, null
1654525860, 19.4, null, null, null
1654525870, 19.4, null, null, null
1654525880, 19.4, null, null, null
1654525890, 19.4, null, null, null
1654525900, 19.4, null, null, null
1654525910, 19.4, null, null, null
1654525920, 19.4, null, null, null
1654525930, 19.4, null, null, null
1654525940, 19.4, null, null, null
1654525950, 19.4, null, null, null
1654525960, 19.4, null, null, null
1654525970, 19.4, null, null, null
1654525980, 19.4, null, null, null
1654525990, 19.4, null, null, null
1654526000, 19.4, null, null, null
1654526010, 19.4, null, null, null
1654526020, 19.4, null, null, null
1654526030, 19.4, null, null, null
1654526040, 19.4, null, null, null
1654526050, 19.4, null, null, null
1654526060, 19.4, null, null, null
1654526070, 19.4, null, null, null
1654526080, 19.4, null, null, null
1654526090, 19.5, null, null, null
1654526100, 19.4, null, null, null
1654526110, 19.5, null, null, null
1654526120, 19.5, null, null, null
1654526130, 19.5, null, null, null
1654526140, 19.5, null, null, null

So the data was there, but is not now.

Anyway, I’ve set up parallel feeds from the same inputs, using PHPTIMESERIES and not PHPFINA. I’ll run it overnight and see whether there are more gaps this time.

Tom_sky · 2022-06-07T08:04:09.063Z

Right, I’m declaring this mystery “fixed” rather than “solved”.

Here’s a graph with the same data input being logged two different ways:

image1127×573 64.4 KB

^^^missing data hidden

image1138×569 84.5 KB

^^^missing data shown

The top 4 series are in PHPFINA. The series appended with ___ are in PHPTIMESERIES.

So, looks like variable time interval logging is the best.

Vster · 2022-06-09T15:26:32.279Z

It’s not ‘best’, they’re two separate ways to store data.

PHPFINA (fixed interval - no averaging) is for data that turns up at fixed intervals, like you read the solar PV input every 10 secs. You store it in a PHPFINA with 10 sec intervals, it’s fast and compact, as there is no timestamp data stored, each reading has it’s own timeslot.

PHPTIMESERIES is for data that changes irregularly, like the wind speed or car battery voltage. When that data is stored, it’s stored with a timestamp. This takes space in the database, but as it irregular it saves space compared to FINA which would have blank slots with no data.

Your problem above, as shown by straight lines, is that there is data missing, that was never stored in the database, so the straight lines are just joining available data points together.

If EmonCMS was ‘forgetting’ data, there would be hundreds of people here screaming/shouting while comparing log files.

Tom_sky · 2022-06-09T15:59:03.138Z

Vster:

If EmonCMS was ‘forgetting’ data, there would be hundreds of people here screaming/shouting while comparing log files.

OK, well it’s forgetting my data! See above - the CSV showed data at a time point, and then later on, that time point was showing null.

The problem is only adjacent to OEM - it’s temp data coming from a Tasmota ESP with DS18B20 “compatible” sensors, using Emoncms as a data logger only, so I’m not expecting anyone to leap to solve the issue, and it seems to have been resolved by switching storage intervals.

Robert.Wall · 2022-06-09T16:13:00.134Z

As Lee says, two different database algorithms for two different situations.

What I suspect he left out for clarity is, when you use PHPFINA (it’s got a sightly different name in the latest emonCMS: Fixed Interval TimeSeries), if the data fails to arrive, it stores a NULL instead. When you draw a graph, if the time interval for the graph is longer than one pixel per value (I’m open to correction on the detail, but the principle is the same), then the graph routine picks values from the data (rows in the database) at intervals to make the graph. If a value is a null, it skips it even though an adjacent row has a good value, i.e. it doesn’t interpolate if it finds a null.

Vster:

PHPTIMESERIES is for data that changes irregularly,

It would be more accurate to put that as “PHPTIMESERIES is for data that arrives irregularly”.

Tom_sky · 2022-06-09T19:57:35.871Z

Robert.Wall:

if the data fails to arrive, it stores a NULL instead

Aha!

I’ve exported the data directly as a CSV:

using 10s interval

date	136	137	140	141	145
1654588800	null	null	18.8	18.1	null
1654588810	null	null	18.8	18.1	null
1654588820	null	null	18.8	18.2	null
1654588830	null	null	18.8	18.2	null
1654588840	null	null	18.8	18.2	null
1654588850	null	null	18.8	18.2	null
1654588860	null	null	18.8	18.2	null

using 5s interval

date	136	137	140	141	145
1654588800	null	null	null	null	null
1654588805	18.8	18.1	18.8	18.1	18.8
1654588810	null	null	null	null	null
1654588815	18.8	18.1	18.8	18.1	18.8
1654588820	null	null	null	null	null
1654588825	18.8	18.2	18.8	18.2	18.8
1654588830	null	null	null	null	null
1654588835	18.8	18.2	18.8	18.2	18.8
1654588840	null	null	null	null	null
1654588845	18.8	18.2	18.8	18.2	18.8
1654588850	null	null	null	null	null
1654588855	18.8	18.2	18.8	18.2	18.8
1654588860	null	null	null	null	null

So, the data wasn’t lost, it was hiding between the null data points!

Robert.Wall:

If a value is a null, it skips it even though an adjacent row has a good value, i.e. it doesn’t interpolate if it finds a null.

It looks like this is applied to more than just the graph plotting. How can I avoid this phenomena in future?

Bill.Thomson · 2022-06-09T20:07:46.564Z

Tom_sky:

using 5s interval

It sounds like you’ve got emonCMS set up to expect data at 5 second intervals,
but it’s in fact arriving at 10 second intervals.

Have you tried creating a new feed that expects data at 10 second intervals?

Tom_sky · 2022-06-09T20:20:06.801Z

Bill.Thomson:

It sounds like you’ve got emonCMS set up to expect data at 5 second intervals

I didn’t realise that 5s was a selectable interval! I’ve just experimented, and leaving it on “select” does indeed set it to 5s…

Bill.Thomson:

Have you tried creating a new feed that expects data at 10 second intervals?

I’ve just turned the breadboard back on, with a fresh sensor (with a fresh ID#) and set it up with some test feeds. I’ll report back once it’s collected a bit of data.

Bill.Thomson · 2022-06-09T20:21:10.587Z

Tom_sky · 2022-06-09T21:16:16.278Z

Ok, I set up the Tasmota to deliver data every 15s (teleperiod 15).

Then I set up 4 feeds from the 1 input: fixed 5s (“select”), fixed 10s, fixed 15s, and variable.

I then gather some data:

Date	Fixed_5s	Fixed_10s	Fixed_15s	Variable
1654806900	null	21.4	21.4	null
1654806905	21.4	21.4	21.4	21.4
1654806910	null	null	21.4	null
1654806915	null	null	21.4	null
1654806920	21.4	21.4	21.4	21.4
1654806925	null	21.4	21.4	null
1654806930	null	21.4	21.4	null
1654806935	21.4	21.4	21.4	21.4
1654806940	null	null	21.4	null
1654806945	null	null	21.4	null
1654806950	21.4	21.4	21.4	21.4
1654806955	null	21.4	21.4	null
1654806960	null	21.4	21.4	null
1654806965	21.4	21.4	21.4	21.4
1654806970	null	null	21.4	null
1654806975	null	null	21.4	null
1654806980	21.4	21.4	21.4	21.4
1654806985	null	21.4	21.4	null
1654806990	null	21.4	21.4	null
1654806995	21.4	21.4	21.4	21.4
1654807000	null	null	21.4	null
1654807005	null	null	21.4	null
1654807010	21.4	21.4	21.4	21.4
1654807015	null	21.4	21.4	null
1654807020	null	21.4	21.4	null
1654807025	21.4	21.4	21.4	21.4
1654807030	null	null	21.4	null
1654807035	null	null	21.4	null
1654807040	21.4	21.4	21.4	21.4
1654807045	null	21.4	21.4	null
1654807050	null	21.4	21.4	null
1654807055	21.4	21.4	21.4	21.4
1654807060	null	null	21.4	null
1654807065	null	null	21.4	null
1654807070	21.4	21.4	21.4	21.4
1654807075	null	21.4	21.4	null
1654807080	null	21.4	21.4	null
1654807085	21.4	21.4	21.4	21.4

I’m not fully sure what’s going on here, so I won’t speculate what this means.

Robert.Wall · 2022-06-09T22:44:53.018Z

It looks as if it’s behaving exactly as designed (PHPFINA, or Fixed Interval TimeSeries in the latest emonCMS, that is).

Think of the emonCMS PHPFINA Feed as a line of boxes moving along a conveyor. If there’s a box going past every 5 s but data falls onto the conveyor every 15 s, two boxes out of three will move on empty, with no data in them (a NULL value). You need to tweak the timing of any device sending to emonCMS to be just slightly less than one of the intervals you can choose. If there’s already data in the box and it hasn’t moved on, it will overlay the data already in there, which is lost. But you won’t have null values in the Feed.

The same applies to any Feed and any timebase - and the boxes move on the Internet minute, or multiples or sub-multiples thereof. i.e., a 10 s feed happens on the minute and at 10, 20 etc seconds past.

If you can always guarantee that the source will send the data in the middle of a time slot by Internet time, you can safely match the rates exactly.

Where the sender, the emonTx for example, depends on a crystal tolerance for its timebase, or there might be transmission delays, we set it to send every 9.96 seconds and use a 10 s feed interval.

Tom_sky · 2022-06-10T11:53:29.214Z

Thanks for the clear explanation - it’s a feature of the system which I didn’t appreciate. It’s a shame that it doesn’t offer to interpolate null values from the good data adjacent, but also understandable.

Robert.Wall:

Where the sender, the emonTx for example, depends on a crystal tolerance for its timebase, or there might be transmission delays, we set it to send every 9.96 seconds and use a 10 s feed interval.

It’s also a shame that Tasmota doesn’t offer the option of a teleperiod <10s or with resolution <1s. Advantages of getting the purpose built monitor, I guess!

I’ll select a suitable shorter period (14s? 29s? 59s?) and go from there. I don’t need the data to be reported that frequently, anyway - I’d only set it up to report that frequently so I could play around with it.

Thanks to everyone for your input - I’ve definitely learned something new.