Emoncms.org running behind

We’re seeing a bit of lag again today due to higher than normal load, inputs and feeds for 1/6th of users are running 10 mins behind at the moment.

Its catching up slowly now. A large part of the reason for these slow downs are the result of the number of active feeds growing from ~28,000 a year ago to ~38,000 today. We are working again on increasing the scaling capacity, see our post and accompanying details here We are hiring: Emoncms.org development role available more to come on this soon.

I’ve been able to implement a number of performance improvements over the last week that should give a bit more time before the more substantial architecture changes are needed, including:

  • using unix sockets rather than tcp for redis which give a processing time reduction of about 40%
  • reduction in calls to redis exist and then get, where the result of get being null can be used to check the key’s existence instead.
  • automated clean up of references to feeds that have been deleted in input processing lists, eliminating a substantial amount of redis and mysql queries that where resulting from attempts to find deleted feeds.
  • balancing of queue load

The queues are all processing at a good rate as a result and the server load is about 25% lower