Tasks module

Emoncms
,
1

Hi,

I have recently being working developing the Groups module (pretty much ready to go). As part of the project the clients want to be able to have notifications. I have read the forum and I know from what the experts say that notifications are best achieved with NodeRed, but we need a solution in emonCMS.

The two basic notifications required are:

  1. based on feed values and operations (like when feed goes above a value)

  2. inactivity of an input

Also to make life easy when managing many installations, we want to be able to set the notifications from the Groups view for single feeds or in bulk (many users in one go)

First thoughts:

  • Number 1 can be achivied with input processors, so we would only need to make it possible to configure them from the Groups view
  • Number 2 cannot be achieved with input processors as they are trigered by an input update, so it’s imposible to detect an input not updating.

I found a blog from Trystan about a script/module “Notify on inactive feeds”. So the actual check of inactivity has already been done. The question is if that is the best way to do it.

In our original conversations we thought about a Notifications module that runs as a cron task (which I understand is almost the same than the “Notify on inactive feeds” but letting the user define the feeds to check). But thinking of it I think it is very limited and I am getting to another idea: a module to define tasks to be run periodically. Those “tasks” are run at specific times (defined by a frequency, maybe only one time or even have an expiry date). The way to set up a task would be similar to configuring input processes, in fact after having studied the Process module I can see that most of the work is done already. We would use all the existing processes and make new ones (like one for detecting feed inactivity). Also, it would be the Process module who runs the tasks (great achivement this Process module :slight_smile:

Other application of this Tasks module could be the recent feature added to emoncms.org: the monday email notification. An another: combined with the Groups module this would allow periodical aggregation of feeds into a another feed from different users. I am sure we can think of more.

As a final pro: input processors trigered by input updates may be run when not needed therefor using resources, here there is an example on how to get an email notification if temperature has been above 20 for more then an hour. Those checks happen every time the input is updated, but we only need to check once every half hour (as we would do with the Tasks module).

Also having to do with the use of server resources there might be a con: how would the extra load of dealing with all the tasks affect the server performance?. I am thinking more in big systems like emoncms.org

Any idea about all of this will be very much apprecciated

2

Thanks😏
Ok my view on this is that the concept of a task implemented on emoncms can be considered more like a function that agregates processes and is usable from inputs or feeds. We still have the problem of triggering tasks, so back to the start…
If the only issue needed to solve is the no data received and hence no trigger of a processlist then the cron like thing is required. There is something called redis scheduler that might be useful.

3

Thanks nchaveiro for your reply,

Yes, clearly I have to use the cron for the inactivity notification.

What I need to decide is if

I make another version of the “Notify of inactive feeds” making it possible to set them up from the Groups view. Any other notifications would have to be set up one by one in the user’s Inputs view (I will not be able to add processors to user inputs from the Groups view, in the Groups module we only deal with feeds)

OR

go for the big mission and do the Tasks module that triggers processes on a specific time. I would make a new processor to get the last value of a feed and that would work as the input to the processes. Doing it this way I will be able to set up all the notifications from the Groups view

I think I have to go for the second option. Let see what comes up, I’ll update
here

4

On the brainsstorm phase:
How about a new system feed type that gets triggered at a regular interval (admin configurable) and not usable on graphs.
Since this feed does not depend on any input, you can use it as a global task where a processlist deals with checking idle inputs/feed.

5

Right, it has taken me a while to understand what you mean, but after looking into the Feed module and Virtual Feeds I think I get it. Now it makes more sense what you said before:

my view on this is that the concept of a task implemented on emoncms can be considered more like a function that agregates processes and is usable from inputs or feeds

Also I see the “Source feed” processor which is the one I’d also need in the beginning of any task

Well, this looks like a very neat solution. I need more thinking

6

That’s it,
We can even create new processors: ‘Input last updated’ and ‘Feed last updated’ that returns seconds since last update of an input or a feed respectively.

7

Ok, clearly it makes lots of sense to make a feed engine and use the infrastructure there is already in the feed module. But i have a conceptual problem with it. A task is, as you said, an aggregation of processes while I understand a feed as a set of datapoints. How would this tasks be called “Time triggered feed”? Does it make sense to display them in the feeds view?

9

Interesting approaches.

I like the concept of a task module which could be a trigger source for processes. Im not so sure about the idea of creating a feed engine, that sounds cumbersome to add feeds on top of feeds your trying to monitor somehow… ?

I have a basic cron alert script on emoncms.org that monitors activity and sends me an email alert if something goes out of bounds, here’s an example alert definition (this is just hardcoded in the script - but it could be extended so that alerts are registered using an emoncms alert module):

$alerts = array(
     // Disk use alerts
    array("alertid"=>1, "feedid"=>FEEDID, "morethan"=>160000, "reset"=>150000, "emailto"=>EMAILS, "message"=>"Emoncms.org disk space more than X%", "repeat_period"=>3600*24),

It fetches the feed last value using the entry in redis, e.g:

$lastvalue = $redis->hmget("feed:lastvalue:$feedid",array('time','value'));

But its essentially the cron approach, similar to the original notify module…

10

@TrystanLea like virtual feeds, that new ‘system scheduled feeds’ wont get accounted for last update time.

Alternative 2: could be a system input, trigered at a regular fixed interval. Same concept of the above.

Alternative 3: abstract the concept of feed/input/action/task and like the inactive module, just send an email to the feed owner when a pre configured time has passed, make this time configured per feed. Personaly, for me this is enought.

Other options are welcome.

11

Hi,

Finally I went for the stand alone module as I didn’t see it clear to put in the Feeds, they have things in common but they are not the same. I have more thoughts about inputs, virtual feeds and tasks being ProcessLists with different triggers that i hope I’ll write down in another post soon.

For now you can find the module here. If anybody tries it I’d appreciate comments, if you find something not working please open an issue.

12

11:28:08 Apply changes to tasks Error 1101: BLOB, TEXT, GEOMETRY or JSON column ‘frequency’ can’t have a default value SQL Statement: ALTER TABLE emoncms.tasks CHANGE COLUMN frequency frequency TEXT NULL DEFAULT 0

The frequency column is a text type column. This doesn’t work

13

Thanks for pointing that out.

The frequency column is a text type column. This doesn’t work

The frequency is a “text” because is storing a json string

[…] column ‘frequency’ can’t have a default value […]

Interesting enough I was not having that problem, the frequency column was created in the database but no default value was added. I have removed the “default” from the schema.

Cheers