To solve hard problems, you need to use bricolage
In a talk about a neat software component he designed, Bruce Haddon observed that there is no way that the final structure and algorithmic behavior of this component could have been predicted, designed, or otherwise anticipated. Haddon observed that computer science serves as a source of core ideas: it provides the data structures and algorithms that are the building blocks. Meanwhile, he views software engineering as a useful set of methods to help design reliable software without losing your mind. Yet he points out that neither captures the whole experience. That’s because much of the work is what Haddon calls hacking, but what others would call bricolage. Simply put, there is much trial and error: we put ideas to together and see where it goes.
This is a great post, and I agree (broadly). IMO, most software engineering requires little CS, but there are occasional moments where a single significant aspect of a project requires a particular algorithm, and would be kludgy, hacky, or over-complex to solve without it.(tags: bricolage hacking cs computer-science work algorithms)
Getting Real About Distributed System Reliability
I have come around to the view that the real core difficulty of [distributed] systems is operations, not architecture or design. Both are important but good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations. This is quite different from the view of unbreakable, self-healing, self-operating systems that I see being pitched by the more enthusiastic NoSQL hypesters. Worse yet, you can’t easily buy good operations in the same way you can buy good software—you might be able to hire good people (if you can find them) but this is more than just people; it is practices, monitoring systems, configuration management, etc.
(tags: reliability nosql distributed-systems jay-kreps ops)
Don’t use Hadoop – your data isn’t that big
see also HN comments: https://news.ycombinator.com/item?id=6398650 , particularly davidmr’s great one:
I suppose all of this is to say that the amount of required parallelization of a problem isn’t necessarily related to the size of the problem set as is mentioned most in the article, but also the inherent CPU and IO characteristics of the problem. Some small problems are great for large-scale map-reduce clusters, some huge problems are horrible for even bigger-scale map-reduce clusters (think fluid dynamics or something that requires each subdivision of the problem space to communicate with its neighbors). I’ve had a quote printed on my door for years: Supercomputers are an expensive tool for turning CPU-bound problems into IO-bound problems.
I love that quote!(tags: hadoop big-data scaling map-reduce)
-
Gilt ran a stress-test of Riak to replace Voldemort (I think) in a shadow stack, with good results:
Riak’s strong performance suggests that, should we pursue implementation, it will withstand our unique traffic needs and prove reliable. As for the Gilt-Basho team’s strong performance: It was amazing that we were able to accomplish so much in just a week’s time! Thanks again to Seth and Steve for making this possible.
THE LONG DARK, a first-person post-disaster survival sim by Hinterland — Kickstarter
wow this looks great.
The Long Dark is a thoughtful, first-person survival simulation that emphasizes quiet exploration in a stark, yet hauntingly beautiful, post-disaster setting. The breathtakingly picturesque Pacific Northwest frames the backdrop for the drama of The Long Dark.
(tags: games survival via:fp eclaire the-long-dark kickstarter)
The Rational Choices of Crack Addicts – NYTimes.com
“The key factor is the environment, whether you’re talking about humans or rats,” Dr. Hart said. “The rats that keep pressing the lever for cocaine are the ones who are stressed out because they’ve been raised in solitary conditions and have no other options. But when you enrich their environment, and give them access to sweets and let them play with other rats, they stop pressing the lever.”