Stuck with feeds not showing up

MikeC · 22 November 2017 21:44

Hi All,

Im stuck with an issue where some of my feeds are not graphing in EmonCMS. I have EmonCMS installed on a local ubuntu VM has been running through many updates of EmonCMS. I have recently cleaned out a lot of my feeds by deleting some unused and re-setting up all my inputs.

Issue seems isolated to some but not all PHPFNIA feeds. The latest value shows up in the feeds page but when graphed they are not visible.

Below is a quick dump of my administration\server information.

Server Information
Emoncms	Version	9.8.10 | 2017.08.17
Modules	Administration | EventProcesses | Feed | Input | CoreProcess | Schedule | Time | User | Visualisation
Server	OS	Linux 4.4.0-38-generic
Host	UbuntuServer UbuntuServer (127.0.1.1)
Date	2017-11-23 10:43:05 NZDT
Uptime	10:43:05 up 3 days, 17:19, 0 users, load average: 2.18, 1.78, 1.70
HTTP	Server	Apache/2.4.7 (Ubuntu) HTTP/1.1 CGI/1.1 80
Database	Version	MySQL 5.5.58-0ubuntu0.14.04.1
Host	localhost (127.0.0.1)
Date	2017-11-23 10:43:05 (UTC 13:00‌)
Stats	Uptime: 321571 Threads: 18 Questions: 11551303 Slow queries: 0 Opens: 131 Flush tables: 1 Open tables: 124 Queries per second avg: 35.921
Memory	RAM	
Used 14.56%
Total: 5.93 GB Used: 883.68 MB Free: 5.06 GB
Swap	
Used 0.78%
Total: 510 MB Used: 3.95 MB Free: 506.04 MB
Disk	Mount	Stats
/	
Used 64.59%
Total: 15.25 GB Used: 9.85 GB Free: 4.61 GB
PHP	Version	5.5.9-1ubuntu4.22 (Zend Version 2.5.0)
Modules	apache2handler | bcmath | bz2 | calendar | Core v5.5.9-1ubuntu4.22 | ctype | curl | date v5.5.9-1ubuntu4.22 | dba | dom v20031129 | ereg | exif v1.4 | fileinfo v1.0.5-dev | filter v0.11.0 | ftp | gd | gettext | hash v1.0 | iconv | json v1.3.2 | libxml | mbstring | mcrypt | mhash | mysql v1.0 | mysqli v0.1 | openssl | pcre | PDO v1.0.4dev | pdo_mysql v1.0.2 | Phar v2.0.2 | posix | readline v5.5.9-1ubuntu4.22 | redis v3.1.4 | Reflection | session | shmop | SimpleXML v0.1 | soap | sockets | SPL v0.2 | standard v5.5.9-1ubuntu4.22 | sysvmsg | sysvsem | sysvshm | tokenizer v0.1 | wddx | xml | xmlreader v0.1 | xmlwriter v0.1 | Zend OPcache v7.0.3FE | zip v1.11.0 | zlib v2.0 |

Any help would be most appreciated.

Kind Regards,

Mike Christiansen.

MikeC · 22 November 2017 22:08

Looks like its all the feeds referenced in the error “WARN|PHPFina.php|post() padding max block size exeeded” Anyone know how to address these errors? Cheers, MikeC

pb66 · 22 November 2017 22:49

Hi Mike

I’m struggling to get my head around what the issue could be

Tells us the feeds have not been written too for a significant period of time and the error is being thrown rather than having a huge empty file with data at the beginning and the end. phpfina feeds use space as time tick by whether there is data saved or not.So not using a feed for a year (for example) will only remain smaller than one that is used every day, until, you try and write to that unused feed, at which time it will try and place the latest data one year later in the files structure and backfill with null data points, it the position within the file that gives the data it’s timestamp.

I imagine the feeds were not used for a long time and now you are trying to use them again, the feed looks like it is updating on the feeds page, but the data isn’t actually being saved because you’ve reached the “padding max block size” threshold above, that why the graphs show nothing.

Do you really want to revive these long time dormant feeds that will jump in size as soon as you do write any data to them?

I think your choices are to start a fresh new feed, or you may be able to extend the threshold by editing this line

github.com

emoncms/emoncms/blob/acdbed74c23ee62b8ee43f15621d6547fd7358eb/Modules/feed/engine/PHPFina.php#L196


      
          $fh = fopen($this->dir.$id.".dat", 'c+');
          if (!$fh) {
              $this->log->warn("post() could not open data file id=$id");
              return false;
          }
          
          // Write padding
          $padding = ($pos - $last_pos)-1;
          
          // Max padding = 1 million datapoints ~4mb gap of 115 days at 10s
          $maxpadding = 1000000;
          
          if ($padding>$maxpadding) {
              $this->log->warn("post() padding max block size exeeded id=$id, $padding dp");
              return false;
          }
          
          if ($padding>0) {
              $padding_value = NAN;
              
              if ($last_pos>=0 && $padding_mode!=null) {

MikeC · 23 November 2017 02:56

Thanks Paul,

As you suggested I just deleted all the old feeds and started with fresh ones. That has solved all the issues thanks.

Kind Regards,

MikeC

MyForest · 11 November 2020 12:55

Just to confirm, bumping up maxpadding does allow a feed to be “resurrected”.

I have some devices I add and remove every few years and the nature of the device changes, but I want all the history.

I am doing this after syncing to github:

sed --in-place s/3153600/31536000/ Modules/feed/engine/PHPFina.php

I’m not bothered about feed sizes, I have a 12TB drive they can go on

Thanks for the hint @pb66

(and apologies for bringing an old thread back from the dead, but it was the best hit in the results and I wanted others to know that altering the setting works)

borpin · 11 November 2020 13:25

I have seen this when the Feed Interval is different to the Feed itself.

If you set that to match the actual feed interval, it might work. (example feed is the Octopus Consumption data which is half hourly).