We’re seeing a bit of lag again today due to higher than normal load, inputs and feeds for 1/6th of users are running 10 mins behind at the moment.
Its catching up slowly now. A large part of the reason for these slow downs are the result of the number of active feeds growing from ~28,000 a year ago to ~38,000 today. We are working again on increasing the scaling capacity, see our post and accompanying details here We are hiring: Emoncms.org development role available more to come on this soon.
I’ve been able to implement a number of performance improvements over the last week that should give a bit more time before the more substantial architecture changes are needed, including:
- using unix sockets rather than tcp for redis which give a processing time reduction of about 40%
- reduction in calls to redis exist and then get, where the result of get being null can be used to check the key’s existence instead.
- automated clean up of references to feeds that have been deleted in input processing lists, eliminating a substantial amount of redis and mysql queries that where resulting from attempts to find deleted feeds.
- balancing of queue load
The queues are all processing at a good rate as a result and the server load is about 25% lower