Emoncms.org performance improvement

Emoncms
1

Happy to say that I’ve managed to improve the performance of emoncms.org significantly which should sort the issue with feeds being slow to update that has reoccurred quite a bit recently.

The problem was that large user accounts with several hundreds of inputs per account where slowing down the input processing due partly to the time required to load the inputs. If an account had 100 inputs the input processing script had to make 100+ redis calls to load the inputs object. Each call has a degree of latency associated with it, which adds up, slowing the processing down.

The solution I found was to use Redis pipelining, where you can batch together a block of redis calls into one request. Following the example here How to pipeline with phpredis by Josh Sherman I modified the get_inputs function in the input_model to use this approach. The result is about a 10x speed improvment for an accounts with 100+ inputs.

Here’s the new get_inputs function with pipelining:

// USES: redis input & user
private function redis_get_inputs($userid)
{
    $userid = (int) $userid;
    if (!$this->redis->exists("user:inputs:$userid")) $this->load_to_redis($userid);

    $dbinputs = array();
    $inputids = $this->redis->sMembers("user:inputs:$userid");

    $pipe = $this->redis->multi(Redis::PIPELINE);
    foreach ($inputids as $id) $row = $this->redis->hGetAll("input:$id");
    $result = $pipe->exec();
    
    foreach ($result as $row) {
        if ($row['nodeid']==null) $row['nodeid'] = 0;
        if (!isset($dbinputs[$row['nodeid']])) $dbinputs[$row['nodeid']] = array();
        $dbinputs[$row['nodeid']][$row['name']] = array('id'=>$row['id'], 'processList'=>$row['processList']);
    }
    
    return $dbinputs;
}

I will commit these changes to the emoncms master branch soon.

1 Like
2

Thanks @TrystanLea, much appreciated.