Performance Modification: One Second Logging Resolution

SolarMill · 2016-07-06T19:00:18.276Z

We are hosting our own EmonCMS server and generating tons of awesome data, however it would be very useful for our specific application if we could have the highest resolution possible. We understand that this may not be ideal for everyone, but it would be extremely useful for our purposes.

Does anyone have guidance on what we would need to do to implement this functionality?

Paul · 2016-07-06T22:14:22.393Z

Why would this be useful for your specific application?
I understand the ethos of SolarMill, but fail to see why you think that 1 second logging would be useful?
How would you make use of this resolution?

Paul

glyn.hudson · 2016-07-07T09:12:08.366Z

I believe it’s only emoncms.org that has a limit of once per 10s. I think emoncms installed on your own server will log as fast as the hardware will allow. Please correct me if I’m wrong @TrystanLea @nchaveiro

nchaveiro · 2016-07-07T10:41:42.071Z

Hi, the minimal limit for the API is 1s, but also depends on the used feed interval. Mysql supports any interval starting from 1s.
If there is any other ‘hard’ limit on emoncms.org, @trystanlea knows the answer.

SolarMill · 2016-07-07T14:20:43.725Z

Hey Paul,

That’s a great question and thanks for asking. There are actually several reasons why this is important to us, some more than others. As I expand on some of them, I’d like to preface our request for help with a few caveats and acknowledgements:

This is not a feature we are asking to be included in the standard distribution and it will not likely be of benefit to 98% of the OpenEmon community. We appreciate that a lot of thought and consideration has gone into the current design decisions and that in most cases 5 seconds is the ideal compromise between resolution and storage. It certainly should never be used on the publicly hosted EmonCMS. This is for local, experimental usage only. However, there may be other power users who might find this documentation and mod useful, so we want to share details of this mod for the benefit of others.
Even in our own customized system, this isn’t a level of resolution we’d be using on most of our inputs and feeds. It would be absurd to log environmental data at anything more than the standard 90sec interval. We need this for about 5-10 specific data inputs, which are exclusively for recording the movements of a CNC machine, the changing load on our router spindle, the varying intensity of sunlight as clouds pass overhead (PhotoTX light sensor logged into EmonCMS), the responsiveness of our charge controller, the surge of motor startups, testing appliances, etc.

For those that haven’t read our previous post on 48V DC monitoring, we are running a custom built solar powered CNC router with our own in-house DC monitoring solution (hall effect sensors, custom firmware/sketch/ loaded on a basic arduiono with RF transmitter). We can measure separately how much power our spindle, motion control system, computer, and dust collection system are using, as well as realtime performance of our batteries, charge controller, and Outback Inverter. We doing active research in the field of motion dynamics, sustainable manufacturing, and energy usage in robotic systems. This is what a one hour CNC routing job looks like, complete with 3 tool changes.

The scale at which many of our CNC machining operations occur is frequently in the 5-15 second range. For a 5sec sample rate, that gives us 3 unique data points to describe a 15 second machining pass, as opposed to 15 data points.

The scale at which most of our manual finishing and woodworking operations occur is at most 5 seconds per operation. Tasks like cutting a 2x4 on a miter saw typically take only 3 seconds. That’s doesn’t even fully occupy a single data point, making capturing and displaying such an event impossible. It’s simply a no go. At 1sec resolution, we’d have at least 3 full data points; a beginning, a middle, and an end.

Not only do we manufacture our own products, we also test existing equipment to compare things like startup load, running power, and smoothness. We are building a database of profiles for certain types of machinery. Increasing the resolution from 5sec to 1sec increases the likelihood of us being able to capture power spikes and transients by 500%, as well as being able to determine the duty cycle for PID controlled thermal systems. For example, we are using a lulzbot mini for 3D printing and recording the graphs. This is the same part 3D Printed in ABS (left grouping) compared to a one hour print in PLA (right grouping):

If we look closer, we see that the frequency that the bed and nozzle are cycling on/off is about the same as the sample rate, which means a single point for high and a single point for low. At 1sec resolution, we would have clear plateaus at top and bottom.

Screen Shot 2016-07-06 at 9.38.02 PM.png2186×1374 515 KB

Since we are doing primary level research, our performance needs are higher than a residential grade user who is more concerned with hourly, daily, and monthly figures. We have already made several steps towards enabling this functionality. The relevant sensors have modified firmware installed and are already transmitting at 1sec intervals.

After much deliberation, I decided that the OpenEmon system and the related community are more in line with our values than anything else out there. I also like where it’s headed. There are a lot of modules in the right place and the right time for the type of projects I am working on. There are times when having a truly lab grade solution would be nice but at the moment simply going from 5sec to 1sec would still be a major gain.

If that’s not enough reason, we have several more, but I think we’ve covered enough column space for one post.

Thanks,
Bert

Paul · 2016-07-07T22:27:58.010Z

Hi Bert

You’re explanation helps, and it’s good that you recognise that a lab grade solution would probably be the more robust ‘commercial’ solution, especially if it’s mission critical.
I just have concerns that the sparse data points, and averaging functions could mask issues instead of highlighting them.

…However, it would be interesting to determine the absolute limits of emoncms… perhaps @TrystanLea may be able to advise.

Paul

SolarMill · 2016-07-07T23:24:17.207Z

That’s a good point about missing data.

If we can make an attempt at getting this running on our system, we can give instant feedback to the community on what data quality (%) we are able to record with 3 nodes transmitting at the highest theoretical level the SQL database can log . It could be lower quality, like 80%, or it could turn out to be the same as our existing 5 second feeds which is typically 96%. There’s even a chance it could be higher! We have no problem receiving the data on the radio, so I don’t think that will be a problem.

The only way we’ll find out is to test.

We already have the following nodes transmitting at 1 second intervals:

EmonDC
PhotoTX
EmonTXv3 High Speed

All total, that’s 12 inputs that we would like to log at the highest possible level of detail. Right now the system is receiving the data not the radio, but choosing to store only 1 per 5.

SolarMill · 2016-07-12T19:43:11.757Z

@TrystanLea, do you have any thoughts on this? Where does it need to be hacked?

TrystanLea · 2016-07-13T11:53:40.490Z

I think the two places in the standard emoncms install are:

github.com

emoncms/emoncms/blob/master/Modules/process/Views/process_ui.php#L130

    
      
                      <input id="new-feed-tag" type="text" pattern="[a-zA-Z0-9-_: ]+" required style="width:4em; border-right: none; border-bottom-right-radius: 0; border-top-right-radius: 0;" title="<?php echo dgettext('process_messages','Please enter a feed tag consisting of alphabetical letters, A-Z a-z 0-9 - _ : and spaces'); ?>" placeholder="<?php echo dgettext('process_messages','Tag'); ?>" />
                  </div>
                  <input id="new-feed-name" type="text" pattern="[a-zA-Z0-9-_: ]+" required style="width:6em" title="<?php echo dgettext('process_messages','Please enter a feed name consisting of alphabetical letters, A-Z a-z 0-9 - _ : and spaces'); ?>" placeholder="<?php echo dgettext('process_messages','Name'); ?>" />
              </div>
          </div>
          
          
<div class="input-prepend">
              <span class="add-on feed-engine-label"><?php echo dgettext('process_messages','Engine'); ?></span>
              <div class="btn-group">
                  <select id="feed-engine" class="input-medium">
                      <?php // All supported engines must be here, add to engines_hidden array in settings.php to hide them from user ?>
                      <option value="5">PHPFINA Fixed Interval No Averaging</option>
                      <option value="2">PHPTIMESERIES Variable Interval No Averaging</option>
                      <option value="0">MYSQL TimeSeries</option>
                      <option value="8">MYSQL Memory (RAM data lost on power off)</option>
                      <option value="10">CASSANDRA TimeSeries</option>
                  </select>
                  <select id="feed-interval" class="input-mini">
                      <option value=""><?php echo dgettext('process_messages','Select interval'); ?></option>
                      <option value="5">5<?php echo dgettext('process_messages','s'); ?></option>
                      <option value="10">10<?php echo dgettext('process_messages','s'); ?></option>

and if you wish to use PHPFina:

github.com

emoncms/emoncms/blob/master/Modules/feed/engine/PHPFina.php#L35

    
      
              }
              
          // #### \/ Below are required methods
          
          
    /**
               * Create feed
               *
               * @param integer $id The id of the feed to be created
               * @param array $options for the engine
              */
              public function create($id,$options)
              {
                  $id = (int)$id;
                  $interval = (int) $options['interval'];
                  if ($interval<5) $interval = 5;
                  
                  // Check to ensure we dont overwrite an existing feed
                  $feedname = "$id.meta";
                  if (!file_exists($this->dir.$feedname)) {
                      // Set initial feed meta data
                      $meta = new stdClass();

I think PHPTimeSeries is 1s minimum already.

There is limiter on emoncms.org to stop all posting faster than 5s but its not present on the self-install version.

SolarMill · 2016-07-13T17:33:26.740Z

Awesome! Thanks. We’ll give this a shot this afternoon. Would we have to create a new feed to log at 1s or can we modify the properties of existing feeds, provided we make these adjustments?

TrystanLea · 2016-07-13T17:49:06.086Z

Great, they will need to be new feeds

SolarMill · 2016-07-13T18:43:50.286Z

That looks like it works! Missing a few data points here and there, but overall the data is better fitted than before. We’ll update when we have more data.

kelster101757 · 2018-02-20T14:31:55.254Z

SolarMill:

(hall effect sensors, custom firmware/sketch/ loaded on a basic arduiono with RF transmitter).

Hello, I’m looking at doing something similar monitoring cnc spindles. How have you got on and what was your final solution? Any help appreciated.

Thanks

SolarMill · 2018-02-21T17:32:40.943Z

It works perfectly. We have over 2 years of non-stop data. We use the hall effect sensor board to get an analog voltage signal which feeds into a stand alone arduino that does the ADC. The arduino has a radio transmitter that sends the data to our EmonPi. This could be done many different ways (you could use an EmonPi directly without the radio bounce but you’d need to mod the case to get the analog signals in).