Archives

Pinterest’s follower graph store, built on Redis

This is a good, high-availability Redis configuration; sharded by userid across 8192 shards, with a Redis master/slave pair of instances for each set of N shards. I like their use of two redundancy systems — hot slave and backup snapshots:

We run our cluster in a Redis master-slave configuration, and the slaves act as hot backups. Upon a master failure, we failover the slave as the new master and either bring up a new slave or reuse the old master as the new slave. We rely on ZooKeeper to make this as quick as possible. Each master Redis instance (and slave instance) is configured to write to AOF on Amazon EBS. This ensures that if the Redis instances terminate unexpectedly then the loss of data is limited to 1 second of updates. The slave Redis instances also perform BGsave hourly which is then loaded to a more permanent store (Amazon S3). This copy is also used by Map Reduce jobs for analytics. As a production system, we need many failure modes to guard ourselves. As mentioned, if the master host is down, we will manually failover to slave. If a single master Redis instance reboots, monit restart restores from AOF, implying a 1 second window of data loss on the shards on that instance. If the slave host goes down, we bring up a replacement. If a single slave Redis instance goes down, we rely on monit to restart using the AOF data. Because we may encounter AOF or BGsave file corruption, we BGSave and copy hourly backups to S3. Note that large file sizes can cause BGsave induced delays but in our cluster this is mitigated by smaller Redis data due to the sharding scheme.

(tags: graph redis architecture ha high-availability design redundancy sharding)

Flower Filter

‘A simple time-decaying approximate membership filter’ — like a Bloom filter with time decay. See also http://eng.42go.com/flower-filter-an-update/ for some notes on the non-independence of survival probabilities, and how that imposes negligible differences in practice.

(tags: bloom-filter algorithms coding probabilistic approximate time decay)

Spybike

This is brilliant. ‘covert bicycle GPS tracker; Notifies you by SMS if your bicycle moves; Online tracking’. ‘Spybike is a covert tracking device that is hidden inside your bicycle steerer tube. The device is disguised to look like a normal head set cap to avoid suspicion. If someone steals your bike, you can use SpyBike to track their movements online and on your mobile.’ More details: http://www.integratedtrackers.com/GPSTrack/pdf/Spybike_Instructions_2.pdf

(tags: spybike cycling theft gps tracking)

No Time To Spare [infographic]

‘On August 2, 2005, a fully-loaded Air France Airbus A340 arriving from Paris crash-landed at Toronto’s Pearson International Airport and caught fire. Only 4 of the 8 exits were usable, yet all 309 people on board made it off the aircraft in two minutes, before it was consumed by flames. Here, five of the passengers recount their escape.’

(tags: infographics travel air accidents fire airbus safety escape a340)