Slicer: Auto-sharding for datacenter applications
Paper from Google describing one of their internal building block services:
A general purpose sharding service. I normally think of sharding as something that happens within a (typically data) service, not as a general purpose infrastructure service. What exactly is Slicer then? It has two key components: a data plane that acts as an affinity-aware load balancer, with affinity managed based on application-specified keys; and a control plane that monitors load and instructs applications processes as to which keys they should be serving at any one point in time. In this way, the decisions regarding how to balance keys across application instances can be outsourced to the Slicer service rather than building this logic over and over again for each individual back-end service. Slicer is focused exclusively on the problem of balancing load across a given set of backend tasks, other systems are responsible for adding and removing tasks.
interesting.(tags: google sharding slicer architecture papers)
Cherami: Uber Engineering’s Durable and Scalable Task Queue in Go – Uber Engineering Blog
a competing-consumer messaging queue that is durable, fault-tolerant, highly available and scalable. We achieve durability and fault-tolerance by replicating messages across storage hosts, and high availability by leveraging the append-only property of messaging queues and choosing eventual consistency as our basic model. Cherami is also scalable, as the design does not have single bottleneck. […] Cherami is completely written in Go, a language that makes building highly performant and concurrent system software a lot of fun. Additionally, Cherami uses several libraries that Uber has already open sourced: TChannel for RPC and Ringpop for health checking and group membership. Cherami depends on several third-party open source technologies: Cassandra for metadata storage, RocksDB for message storage, and many other third-party Go packages that are available on GitHub. We plan to open source Cherami in the near future.
(tags: cherami uber queueing tasks queues architecture scalability go cassandra rocksdb)
The Perfect Weapon: How Russian Cyberpower Invaded the U.S. – The New York Times
This is scary shit. It’s amazing how Russia has weaponised transparency, but I guess it’s not new to observers of “kompromat”: https://en.wikipedia.org/wiki/Kompromat
(tags: kompromat russia cyberpower cyberwar security trump us-politics dnc)