Justin's Linklog Posts

Dong detection in LEGO Universe

great example of how Minecraft solved the problem the easy way — by simply not making an MMO, the whole problem effectively goes away

(tags: penis funny games lego lego-universe minecraft gaming mmo ugc)

HTTP/2 is here, let’s optimize! – Velocity SC 2015 – Google Slides

Changes which server-side developers will need to start considering as HTTP/2 rolls out. Remove domain sharding; stop concatenating resources; stop inlining resources; use server push.

(tags: http2 http protocols streaming internet web dns performance)

Five different ways to handle leap seconds with NTP

Without switching to chronyd, ntpd -x sounds not too suboptimal:

With ntpd, the kernel backward step is used by default. With ntpd versions before 4.2.6, or 4.2.6 and later patched for this bug, the -x option (added to /etc/sysconfig/ntpd) can be used to disable the kernel leap second correction and ignore the leap second as far as the local clock is concerned. The one-second error gained after the leap second will be measured and corrected later by slewing in normal operation using NTP servers which already corrected their local clocks.

(tags: ntpd ntp chronyd clocks time synchronization via:fanf linux leap-seconds)

The Agency – NYTimes.com

Russia’s troll farms. Ladies and gentlemen — the future

(tags: future abuse trolls russia trolling politics social-media twitter facebook)

Ireland’s media silenced over MP’s speech about Denis O’Brien

this is appalling. And of course we can only find out about it from overseas media because our own media is quaking in their boots :(

(tags: media ireland he-who-cannot-be-named censorship omgwtfbbq law libel injunctions high-court)

How Ireland’s same-sex marriage referendum played out on Twitter

nice clear data there

(tags: ireland ssm marref history twitter hashtags yesequality)

murbul comments on The security issue of Blockchain.info’s Android Wallet is not about system’s entropy. It’s their own BUGs on PRNG again!

I was in the middle of writing a breakdown of what went wrong, but you’ve beat me to it. Basically, they have a LinuxSecureRandom class that’s supposed to override the standard SecureRandom. This class reads from /dev/urandom and should provide cryptographically secure random values. They also seed the generator using SecureRandom#setSeed with data pulled from random.org. With their custom SecureRandom, this is safe because it mixes the entropy using XOR, so even if the random.org data is dodgy it won’t reduce security. It’s just an added bonus. BUT! On some devices under some circumstances, the LinuxSecureRandom class doesn’t get registered. This is likely because /dev/urandom doesn’t exist or can’t be accessed for some reason. Instead of screaming bloody murder like any sensible implementation would, they just ignore that and fall back to using the standard SecureRandom. If the above happens, there’s a problem because the default implementation of SecureRandom#setSeed doesn’t mix. If you set the seed, it replaces the entropy entirely. So now the entropy is coming solely from random.org. And the final mistake: They were using HTTP instead of HTTPS to make the webservice call to random.org. On Jan 4, random.org started enforcing HTTPS and returning a 301 Permanently Moved error for HTTP – see https://www.random.org/news/. So since that date, the entropy has actually been the error message (turned into bytes) instead of the expected 256-bit number. Using that seed, SecureRandom will generate the private key for address 1Bn9ReEocMG1WEW1qYjuDrdFzEFFDCq43F 100% of the time. Ouch. This is around the time that address first appears, so the timeline matches. I haven’t had a thorough look at what they’ve replaced it with in the latest version, but initial impressions are that it’s not ideal. Not disastrous, but not good.

(tags: bugs android fail securerandom random prng blockchain.info bitcoin http randomness entropy error-checking)

CommonMark

A strongly specified, highly compatible implementation of Markdown

(tags: reference markdown commonmark specs formatting text compatibility)

GitTorrent

‘A Decentralized GitHub’. nifty

(tags: distributed git github bittorrent bitcoin gittorrent dvcs)

I Fooled Millions Into Thinking Chocolate Helps Weight Loss

“Slim by Chocolate!” the headlines blared. A team of German researchers had found that people on a low-carb diet lost weight 10 percent faster if they ate a chocolate bar every day. It made the front page of Bild, Europe’s largest daily newspaper, just beneath their update about the Germanwings crash. From there, it ricocheted around the internet and beyond, making news in more than 20 countries and half a dozen languages. It was discussed on television news shows. It appeared in glossy print, most recently in the June issue of Shape magazine (“Why You Must Eat Chocolate Daily”, page 128). Not only does chocolate accelerate weight loss, the study found, but it leads to healthier cholesterol levels and overall increased well-being. The Bild story quotes the study’s lead author, Johannes Bohannon, Ph.D., research director of the Institute of Diet and Health: “The best part is you can buy chocolate everywhere.” I am Johannes Bohannon, Ph.D. Well, actually my name is John, and I’m a journalist. I do have a Ph.D., but it’s in the molecular biology of bacteria, not humans. The Institute of Diet and Health? That’s nothing more than a website. Other than those fibs, the study was 100 percent authentic. My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.

(tags: chocolate journalism science diet food churnalism pr bild health clinical-trials papers peer-review research)

Snake-Oil Superfoods

mainly interesting for the dataviz and the Google-Doc-driven backend. wish they published the script though

(tags: google snake-oil superfoods food dataviz bubble-race-chart graphics infographics google-docs spreadsheets)

Three Questions to Answer When Reporting an Error

Very long, but tl;dr:

the trick to creating an effective error message is to answer the 3 Questions within your message: What is the error? What was the probable cause of the error? What is the probable remedy?

(tags: errors ui ux reporting logging coding)

Volvo says horrible ‘self-parking car accident’ happened because driver didn’t have ‘pedestrian detection’

Grim meathook future, courtesy of Volvo:

“The Volvo XC60 comes with City Safety as a standard feature however this does not include the Pedestrian detection functionality […] The pedestrian detection feature […] costs approximately $3,000.

But even if it did have the feature, Larsson says the driver would have interfered with it by the way they were driving and “accelerating heavily towards the people in the video.” “The pedestrian detection would likely have been inactivated due to the driver inactivating it by intentionally and actively accelerating,” said Larsson. “Hence, the auto braking function is overrided by the driver and deactivated.” Meanwhile, the people in the video seem to ignore their instincts and trust that the car assumed to be endowed with artificial intelligence knows not to hurt them. It is a sign of our incredible faith in the power of technology, but also, it’s a reminder that companies making AI-assisted vehicles need to make safety features standard and communicate clearly when they aren’t.

(tags: self-driving-cars cars ai pedestrian computer-vision volvo fail accidents grim-meathook-future)

iPhone UTF-8 text vulnerability

‘Due to how the banner notifications process the Unicode text. The banner briefly attempts to present the incoming text and then “gives up” thus the crash’. Apparently the entire Springboard launcher crashes.

(tags: apple vulnerability iphone utf-8 unicode fail bugs springboard ios via:abetson)

Schedule Recurring AWS Lambda Invocations With The Unreliable Town Clock (UTC)

The Unreliable Town Clock (UTC) is a new, free, public SNS Topic (Amazon Simple Notification Service) that broadcasts a “chime” message every quarter hour to all subscribers. It can send the chimes to AWS Lambda functions, SQS queues, and email addresses. You can use the chime attributes to run your code every fifteen minutes, or only run your code once an hour (e.g., when minute == “00”) or once a day (e.g., when hour == “00” and minute == “00”) or any other series of intervals. You can even subscribe a function you only want to run only once at a specific time in the future: Have the function ignore all invocations until it’s after the time it wants. When it is time, it can perform its job, then unsubscribe itself from the SNS Topic.

(tags: alestic aws lambda cron time clock periodic-tasks recurrence hacks)

Soylent, Neoliberalism and the Politics of Life Hacking – CounterPunch: Tells the Facts, Names the Names

Soylent’s not purchased by the Mark Zuckerbergs or the Larry Pages or the other tech aristocrats […] Rather, it’s been taken up by white-collar workers and students destined for perpetual toil in the digital mills. Their embrace of life hacking represents the internalisation of management practices by the managed themselves.

(tags: life-hacks soylent food politics taylorism efficiency capitalism work life)

Working with Apache Spark: Or, How I Learned to Stop Worrying and Love the Shuffle | Cloudera Engineering Blog

some good Spark optimization tips

(tags: spark performance optimization rdd emr big-data cloudera tips akka)

Elements of Scale: Composing and Scaling Data Platforms

Great, encyclopedic blog post rounding up common architectural and algorithmic patterns using in scalable data platforms. Cut out and keep!

(tags: architecture storage databases data big-data scaling scalability ben-stopford cqrs druid parquet columnar-stores lambda-architecture)

ISIS vs. 3D Printing | Motherboard

Morehshin Allahyari, an Iranian born artist, educator, and activist [….] is working on digitally fabricating [the] sculptures [ISIS destroyed] for a series called “Material Speculation” as part of a residency in Autodesk’s Pier 9 program. The first in the series is “Material Speculation: ISIS,” which, through intense research, is modeling and reproducing statues destroyed by ISIS in 2015. Allahyari isn’t just interested in replicating lost objects but making it possible for anyone to do the same: Embedded within each semi-translucent copy is a flash drive with Allahyari’s research about the artifacts, and an online version is coming. In this way, “Material Speculation: ISIS,” is not purely a metaphorical affront to ISIS, but a practical one as well. Allahyari’s work is similar to conservation efforts, including web-based Project Mosul, a small team and group of volunteers that are three-dimensionally modeling ISIS-destroyed artifacts based on crowd-sourced photographs. “Thinking about 3D printers as poetic and practical tools for digital and physical archiving and documenting has been a concept that I’ve been interested in for the last three years,” Allahyari says. Once she began exploring the works, she discovered a thorough lack of documentation. Her research snowballed. “It became extremely important for me to think about ways to gather this information and save them for both current and future civilizations.”

(tags: 3d-printing fabrication scanning isis niniveh iraq morehshin-allahyari history preservation archives archival)

Kubernetes for developers

great intro

(tags: kubernetes ops docker containers rocket deployment packaging)

A Piece of Apple II History Cracks Open — May 24, 2015

Lovely description of cracking (ie. copy-protection removal) in the Apple-II era. Very reminiscent of the equivalent in the C=64 scene, from my experience. ;)

(tags: history c=64 apple-ii personal-computers archive cracks copy-protection hacking)

Deploying Elastic Beanstalk Applications from Docker Containers – Elastic Beanstalk

oh wow, this actually sounds pretty cool

(tags: docker aws ec2 beanstalk deployment ops containers)

TIL we have more gravity than Canada

‘Early gravity mapping efforts in the 1960s revealed that the Hudson Bay area in particular exerts a weaker gravitational force. Since less mass equals less gravity, there must be less mass underneath these areas.’ informed!

(tags: gravity canada geode earth science hudson-bay mass)

SolarCapture Packet Capture Software

Interesting product line — I didn’t know this existed, but it makes good sense as a “network flight recorder”. Big in finance.

SolarCapture is powerful packet capture product family that can transform every server into a precision network monitoring device, increasing network visibility, network instrumentation, and performance analysis. SolarCapture products optimize network monitoring and security, while eliminating the need for specialized appliances, expensive adapters relying on exotic protocols, proprietary hardware, and dedicated networking equipment.

(tags: corvil timing metrics measurement latency network solarcapture packet-capture financial performance security network-monitoring)

Top 10 data mining algorithms in plain English

This is a phenomenally useful ML/data-mining resource post — ‘the top 10 most influential data mining algorithms as voted on by 3 separate panels in [ICDM ’06’s] survey paper’, but with a nice clear intro and description for each one. Here’s the algorithms covered:

1. C4.5 2. k-means 3. Support vector machines 4. Apriori 5. EM 6. PageRank 7. AdaBoost 8. kNN 9. Naive Bayes 10. CART

(tags: svm k-means c4.5 apriori em pagerank adaboost knn naive-bayes cart ml data-mining machine-learning papers algorithms unsupervised supervised)

Developer believes he can turn digital game into global hit

g’wan the Colm!

(tags: colm-larkin guild-of-dungeoneering games press)

Trend Micro Locality Sensitive Hash

a fuzzy matching library. Given a byte stream with a minimum length of 512 bytes, TLSH generates a hash value which can be used for similarity comparisons. Similar objects will have similar hash values which allows for the detection of similar objects by comparing their hash values. Note that the byte stream should have a sufficient amount of complexity. For example, a byte stream of identical bytes will not generate a hash value.

(tags: nilsimsa sdhash ssdeep locality-sensitive hashing algorithm hashes trend-micro tlsh hash fuzzy-matching via:adulau)

Eric Brewer interview on Kubernetes

What is the relationship between Kubernetes, Borg and Omega (the two internal resource-orchestration systems Google has built)? I would say, kind of by definition, there’s no shared code but there are shared people. You can think of Kubernetes?—?especially some of the elements around pods and labels?—?as being lessons learned from Borg and Omega that are, frankly, significantly better in Kubernetes. There are things that are going to end up being the same as Borg?—?like the way we use IP addresses is very similar?—?but other things, like labels, are actually much better than what we did internally. I would say that’s a lesson we learned the hard way.

(tags: google architecture kubernetes docker containers borg omega deployment ops)

‘Can People Distinguish Pâté from Dog Food?’

Ugh.

Considering the similarity of its ingredients, canned dog food could be a suitable and inexpensive substitute for pâté or processed blended meat products such as Spam or liverwurst. However, the social stigma associated with the human consumption of pet food makes an unbiased comparison challenging. To prevent bias, Newman’s Own dog food was prepared with a food processor to have the texture and appearance of a liver mousse. In a double-blind test, subjects were presented with five unlabeled blended meat products, one of which was the prepared dog food. After ranking the samples on the basis of taste, subjects were challenged to identify which of the five was dog food. Although 72% of subjects ranked the dog food as the worst of the five samples in terms of taste (Newell and MacFarlane multiple comparison, P<0.05), subjects were not better than random at correctly identifying the dog food.

(tags: pate food omgwtf science research dog-food meat economics taste flavour)

Redditor runs the secret Python code in Ex Machina

and finds:

when you run with python2.7 you get the following: ISBN = 9780199226559 Which is Embodiment and the inner life: Cognition and Consciousness in the Space of Possible Minds. and so now I have a lot more respect for the Director.

(tags: python movies ex-machina cool books easter-eggs)

Metalwoman beer recipe

via the Dublin Ladies Beer Society ;)

(tags: metalman metalwoman recipes beer brewing hops dlbs)

Linux futex_wait() bug

major bug in kernel versions 3.14 – 3.18 on Haswell hardware

(tags: haswell linux futex_wait futexes kernel bugs hang)

repo

‘The multiple repository tool’. How Google kludged around the split-repo problem when you don’t have a monorepo.

(tags: kludges git monorepo monorepi google android aosp repo coding version-control dvcs)

Declaratively Provision Docker Images Using Nix

I really wish Docker/CoreOS would look at copying some of the deterministic-build ideas from Nix; see also http://gregoryszorc.com/blog/2014/10/13/deterministic-and-minimal-docker-images/

(tags: build packaging docker nix nix-docker deterministic-builds nixos apollo brazil)

Please stop calling databases CP or AP

In his excellent blog post […] Jeff Hodges recommends that you use the CAP theorem to critique systems. A lot of people have taken that advice to heart, describing their systems as “CP” (consistent but not available under network partitions), “AP” (available but not consistent under network partitions), or sometimes “CA” (meaning “I still haven’t read Coda’s post from almost 5 years ago”). I agree with all of Jeff’s other points, but with regard to the CAP theorem, I must disagree. The CAP theorem is too simplistic and too widely misunderstood to be of much use for characterizing systems. Therefore I ask that we retire all references to the CAP theorem, stop talking about the CAP theorem, and put the poor thing to rest. Instead, we should use more precise terminology to reason about our trade-offs.

(tags: cap databases storage distcomp ca ap cp zookeeper consistency reliability networking)

Input: Fonts for Code

Non-monospaced coding fonts! I’m all in favour…

As writing and managing code becomes more complex, today’s sophisticated coding environments are evolving to include everything from breakpoint markers to code folding and syntax highlighting. The typography of code should evolve as well, to explore possibilities beyond one font style, one size, and one character width.

(tags: input fonts via:its typography code coding font text ide monospace)

Apache HTrace

a Zipkin-compatible distributed-system tracing framework in Java, in the Apache Incubator

(tags: zipkin tracing trace apache incubator java debugging)

Intel speeds up etcd throughput using ADR Xeon-only hardware feature

To reduce the latency impact of storing to disk, Weaver’s team looked to buffering as a means to absorb the writes and sync them to disk periodically, rather than for each entry. Tradeoffs? They knew memory buffers would help, but there would be potential difficulties with smaller clusters if they violated the stable storage requirement. Instead, they turned to Intel’s silicon architects about features available in the Xeon line. After describing the core problem, they found out this had been solved in other areas with ADR. After some work to prove out a Linux OS supported use for this, they were confident they had a best-of-both-worlds angle. And it worked. As Weaver detailed in his CoreOS Fest discussion, the response time proved stable. ADR can grab a section of memory, persist it to disk and power it back. It can return entries back to disk and restore back to the buffer. ADR provides the ability to make small (<100MB) segments of memory “stable” enough for Raft log entries. It means it does not need battery-backed memory. It can be orchestrated using Linux or Windows OS libraries. ADR allows the capability to define target memory and determine where to recover. It can also be exposed directly into libs for runtimes like Golang. And it uses silicon features that are accessible on current Intel servers.

(tags: kubernetes coreos adr performance intel raft etcd hardware linux persistence disk storage xeon)

streamtools: a graphical tool for working with streams of data | nytlabs

Visual programming, Yahoo! Pipes style, back again:

we have created streamtools – a new, open source project by The New York Times R&D Lab which provides a general purpose, graphical tool for dealing with streams of data. It provides a vocabulary of operations that can be connected together to create live data processing systems without the need for programming or complicated infrastructure. These systems are assembled using a visual interface that affords both immediate understanding and live manipulation of the system.

(tags: via:akohli streaming data nytimes visual-programming coding)

MappedBus

a Java based low latency, high throughput message bus, built on top of a memory mapped file; inspired by Java Chronicle with the main difference that it’s designed to efficiently support multiple writers – enabling use cases where the order of messages produced by multiple processes are important. MappedBus can be also described as an efficient IPC mechanism which enable several Java programs to communicate by exchanging messages.

(tags: ipc java jvm mappedbus low-latency mmap message-bus data-structures queue message-passing)

Amazon’s Drone Delivery Patent Just Feels Like Trolling At This Point

Oh dear, Amazon.

These aren’t actual technologies yet. […] All of which underscores that Amazon might never ever ever ever actually implement delivery drones. The patent paperwork was filed nearly a year after Amazon’s splashy drone program reveal on 60 Minutes. At the time we called it revolutionary marketing because, you know, delivery drones are technical and logistical madness, not to mention that commercial drone use is illegal right now. Although, in fairness the FAA did just relax some rules so that Amazon could test drones. At this point it feels like Amazon is just trolling. It’s trolling us with public relations BS about its future drones, and it’s trolling future competitors — Google is also apparently working on this — so that if somebody ever somehow does anything relating to drone delivery, Amazon can sue them. If I’m wrong, I’ll deliver my apology via Airmail.

(tags: amazon trolling patents uspto delivery drones uavs competition faa)

Red Hat on rkt vs Docker

This is like watching a train-wreck in slow motion on Groundhog Day. We, in the broader Linux and open source community, have been down this path multiple times over the past fifteen years, specifically with package formats. While there needs to be room for experimentation, having two incompatible specs driven by two startups trying to differentiate and in direct competition is *not* a good thing. It would be better for the community and for everyone who depends on our collective efforts if CoreOS and Docker collaborated on a standardized common spec, image format, and distribution protocol. To this end, we at Red Hat will continue to contribute to both initiatives with the goal of driving convergence.

(tags: rkt docker appc coreos red-hat dpkg rpm linux packaging collaboration open-source)

Migration to, Expectations, and Advanced Tuning of G1GC

Bookmarking for future reference. recommended by one of the GC experts, I can’t recall exactly who ;)

(tags: gc g1gc jvm java tuning performance ops migration)

Deploy a registry – Docker Documentation

Looks like it’s pretty feasible to run a private Docker registry on every host, backed by S3 (according to the ECS team’s AMA). SPOF-free — handy

(tags: docker registry ops deployment s3)

How to change Gradle cache location

$GRADLE_USER_HOME, basically — it may also be possible to set from the Gradle script itself too

(tags: gradle build caching environment unix cache)

Internet of 404’s

“An archive of the former Internet of Things”

(tags: archive iot things internet nabaztag startups acquisitions tumblr gadgets history)

Memory Layouts for Binary Search

Key takeaway:

Nearly uni­ver­sally, B-trees win when the data gets big enough.

(tags: caches cpu performance optimization memory binary-search b-trees algorithms search memory-layout)

Understanding the Docker Cache for Faster Builds

good advice. see also the Best Practices official doc at https://docs.docker.com/articles/dockerfile_best-practices/

(tags: docker build packaging cache best-practices tips)

Your Google Algorithm Cheat Sheet: Panda, Penguin, and Hummingbird

Interesting that GOOG are still doing these big-bang releases — I guess crunching the data to come up with new weights/rules is a heavyweight, time-consuming process

(tags: google search ranking releases panda penguin hummingbird weighting)

Dublin Bike Theft Survey Results

Dublin Cycling Campaign’s survey results: estimated 20,000 bikes stolen per year in Dublin; only 1% of thefts results in a conviction

(tags: dublin bikes cycling theft crime statistics infographics dcc)

DRUG PUMP’S SECURITY FLAW LETS HACKERS RAISE DOSE LIMITS

The Hospira drug pump vulnerabilities described here sound pretty horrific

(tags: drugs drug-pumps hospira exploits vulnerabilities security root dosage limits)

Making End-to-End Tests Work

+1 to ALL of this. We are doing exactly the same in Swrve and it has radically improved our release quality

(tags: end-to-end testing acceptance-tests tests system-tests lmax)

How to do named entity recognition: machine learning oversimplified

Good explanation of this NLP tokenization/feature-extraction technique. Example result: “Jimi/B-PER Hendrix/I-PER played/O at/O Woodstock/B-LOC ./O”

(tags: named-entities feature-extraction tokenization nlp ml algorithms machine-learning)

The Discovery of Apache ZooKeeper’s Poison Packet – PagerDuty

Excellent deep dive into a production issue. Root causes: crappy error handling code in Zookeeper; lack of bounds checking in ZK; and a nasty kernel bug.

(tags: zookeeper bugs error-handling bounds-checking oom poison-packets pagerduty packets tcpdump xen aes linux kernel)

The Injector: A new Executor for Java

This honestly fits a narrow niche, but one that is gaining in popularity. If your messages take > 100?s to process, or your worker threads are consistently saturated, the standard ThreadPoolExecutor is likely perfectly adequate for your needs. If, on the other hand, you’re able to engineer your system to operate with one application thread per physical core you are probably better off looking at an approach like the LMAX Disruptor. However, if you fall in the crack in between these two scenarios, or are seeing a significant portion of time spent in futex calls and need a drop in ExecutorService to take the edge off, the injector may well be worth a look.

(tags: performance java executor concurrency disruptor algorithms coding threads threadpool injector)

KillBiller

Excellent mobile-phone plan comparison site for the Irish market, using apps which you install and which analyse your call history, data usage, etc. over the past month to compute the optimal plan based on your usage. Pretty amazing results in my case! The only downside is the privacy policy, which allows the company to resell your usage data (anonymised, and in aggregate) — I’d really prefer if this wasn’t the case :(

(tags: mobile-phones shopping tesco emobile 3g 4g ireland plans comparison-shopping killbiller via:its)

Family in No poster Says YES to Marriage Equality | Amnesty International

Beyond the politics, the risks of stock photo usage are pretty evident too:

“In 2014, as a young family, we did a photo shoot with a photographer friend to get some nice shots for the family album. No money was exchanged – we got nice photos for free, they got nice images for their portfolio. As part of this agreement, we agreed to let them upload them to a stock photo album. We knew that these were available for purchase and we gave permission. Perhaps, naïvely, we imagined that on the off chance that any was ever selected, it might be for a small magazine or website. To confirm, we have not received any money for the photo – then or now, and nor do we expect any. We were surprised and upset to see that the photo was being used as part of a campaign with which we do not agree. We completely support same-sex marriage, and we believe that same-sex couples’ should of course be able to adopt, as we believe that they are equally able to provide children with much-needed love and care. To suggest otherwise is offensive to us, and to many others.”

(tags: ssm ireland politics amnesty stock-photos ip rights photos campaigns ads)

Lambda: Bees with Frickin’ Laser Beams

a HTTP testing tool in AWS Lambda. nice enough, but still a toy…

(tags: lambda aws node javascript hacks http load-testing)

Why Loggly loves Apache Kafka

Some good factoids about Loggly’s Kafka usage and scales

(tags: scalability logging loggly kafka queueing ops reliabilty)

Patterns for building a resilient and scalable microservices platform on AWS

Some good details from Boyan Dimitrov at Hailo, on their orchestration, deployment, provisioning infra they’ve built

(tags: deployment ops devops hailo microservices platform patterns slides)

hyperlogsandwich

A probabilistic data structure for frequency/k-occurrence cardinality estimation of multisets. Sample implementation

(tags: via:patrickmcfadin hyperloglog cardinality data-structures algorithms hyperlogsandwich counting estimation lossy multisets)

“Trash Day: Coordinating Garbage Collection in Distributed Systems”

Another GC-coordination strategy, similar to Blade (qv), with some real-world examples using Cassandra

(tags: blade via:adriancolyer papers gc distsys algorithms distributed java jvm latency spark cassandra)

Five Takeaways on the State of Natural Language Processing

Good overview of the state of the art in NLP nowadays. I particularly like word2vec interesting:

Embedding words as real-numbered vectors using a skip-gram, negative-sampling model (word2vec code) was mentioned in nearly every talk I attended. Either companies are using various word2vec implementations directly or they are building diffs off of the basic framework. Trained on large corpora, the vector representations encode concepts in a large dimensional space (usually 200-300 dim).

(tags: word2vec nlp tokenization machine-learning language parsing doc2vec skip-grams data-structures feature-extraction via:lemonodor)

Smarter testing Java code with Spock Framework

hmm, looks quite nice as a potential next-gen JUnit replacement for unit tests

(tags: java testing bdd tests junit unit-tests spock via:trishagee)

Tots To Travel

‘Baby Friendly Holidays | Child, Toddler & Family Villas | France | Spain | Portugal | Italy’. Joe swears by it, will give it a go next year

(tags: holidays vacation travel europe kids children via:joe)

How the NSA Converts Spoken Words Into Searchable Text – The Intercept

This hits the nail on the head, IMO:

To Phillip Rogaway, a professor of computer science at the University of California, Davis, keyword-search is probably the “least of our problems.” In an email to The Intercept, Rogaway warned that “When the NSA identifies someone as ‘interesting’ based on contemporary NLP methods, it might be that there is no human-understandable explanation as to why beyond: ‘his corpus of discourse resembles those of others whom we thought interesting’; or the conceptual opposite: ‘his discourse looks or sounds different from most people’s.’ If the algorithms NSA computers use to identify threats are too complex for humans to understand, it will be impossible to understand the contours of the surveillance apparatus by which one is judged.  All that people will be able to do is to try your best to behave just like everyone else.”

(tags: privacy security gchq nsa surveillance machine-learning liberty future speech nlp pattern-analysis cs)

awslabs/aws-lambda-redshift-loader

Load data into Redshift from S3 buckets using a pre-canned Lambda function. Looks like it may be a good example of production-quality Lambda

(tags: lambda aws ec2 redshift s3 loaders etl pipeline)

Call me maybe: Aerospike

‘Aerospike offers phenomenal latencies and throughput — but in terms of data safety, its strongest guarantees are similar to Cassandra or Riak in Last-Write-Wins mode. It may be a safe store for immutable data, but updates to a record can be silently discarded in the event of network disruption. Because Aerospike’s timeouts are so aggressive–on the order of milliseconds — even small network hiccups are sufficient to trigger data loss. If you are an Aerospike user, you should not expect “immediate”, “read-committed”, or “ACID consistency”; their marketing material quietly assumes you have a magical network, and I assure you this is not the case. It’s certainly not true in cloud environments, and even well-managed physical datacenters can experience horrible network failures.’

(tags: aerospike outages cap testing jepsen aphyr databases storage reliability)

Emojineering Part 1: Machine Learning for Emoji Trends – Instagram Engineering

Instagram figuring out meanings from Emoji usage contexts using ML. ????

(tags: instagram emoji cool language text internet web speech communication trends machine-learning analysis)

Call me maybe: Elasticsearch 1.5.0

tl;dr: Elasticsearch still hoses data integrity on partition, badly

(tags: elasticsearch reliability data storage safety jepsen testing aphyr partition network-partitions cap)

In the privacy of your own home

I didn’t know about this:

Last spring, as 41,000 runners made their way through the streets of Dublin in the city’s Women’s Mini Marathon, an unassuming redheaded man by the name of Candid Wueest stood on the sidelines with a scanner. He had built it in a couple of hours with $75 worth of parts, and he was using it to surreptitiously pick up data from activity trackers worn on the runners’ wrists. During the race, Wueest managed to collect personal info from 563 racers, including their names, addresses, and passwords, as well as the unique IDs of the devices they were carrying.

(tags: dublin candid-wueest privacy data marathon running iot activity-trackers)

Cyclists! Why do they ride in the middle of the road?

sense!

(tags: cycling roads cars driving safety uk)

David P. Reed on the history of UDP

‘UDP was actually “designed” in 30 minutes on a blackboard when we decided pull the original TCP protocol apart into TCP and IP, and created UDP on top of IP as an alternative for multiplexing and demultiplexing IP datagrams inside a host among the various host processes or tasks. But it was a placeholder that enabled all the non-virtual-circuit protocols since then to be invented, including encapsulation, RTP, DNS, …, without having to negotiate for permission either to define a new protocol or to extend TCP by adding “features”.’

(tags: udp ip tcp networking internet dpr history protocols)

Oops: Instagram forgot to renew its SSL certificate

hooray for cert renewal pain

(tags: certs ssl renewal expiry instagram outages lifecycle web https)

s3.amazonaws.com “certificate verification failed” errors due to crappy Verisign certs and overzealous curl policies

Seth Vargo is correct. Its not the bit length of the key which is at issue, its the signature algorithm. The entire keychain for the s3.awsamazon.com key is signed with SHA1withRSA: https://www.ssllabs.com/ssltest/analyze.html?d=s3.amazonaws.com&s=54.231.244.0&hideResults=on At issue is that the root verisign key has been marked as weak because of SHA1 and taken out of the curl bundle which is widely popular, and this issue will continue to cause more and more issues going forwards as that bundle makes it way into shipping o/s distributions and aws certification verification breaks.

(tags: curl s3 amazon aws ssl tls certs sha1 rsa key-length security cacerts)

Cassandra moving to using G1 as the default recommended GC implementation

This is a big indicator that G1 is ready for primetime. CMS has long been the go-to GC for production usage, but requires careful, complex hand-tuning — if G1 is getting to a stage where it’s just a case of giving it enough RAM, that’d be great. Also, looks like it’ll be the JDK9 default: https://twitter.com/shipilev/status/593175793255219200

(tags: cassandra tuning ops g1gc cms gc java jvm production performance memory)

The Colossal Shop

ThisIsColossal now have a shop! bookmarking for some lovely gifts

(tags: art design shop colossal shopping christmas gifts)

Eight lessons learned hacking on GitHub Pages for six months

Pages is actually pretty solid — nice one GitHub

(tags: github api pages html web jekyll hosting)

ShellCheck

Static code analysis for shell scripts (via Tony Finch)

(tags: bash cli sh linux shell coding static-analysis lint)

‘Microservice AntiPatterns’

presentation from last week’s Craft Conference in Budapest; Tammer Saleh of Pivotal with a few antipatterns observed in dealing with microservices.

(tags: microservices soa architecture design coding software presentations slides tammer-saleh pivotal craft)

Kappa

‘a command line tool that (hopefully) makes it easier to deploy, update, and test functions for AWS Lambda.’ much needed IMO — Lambda is too closed

(tags: aws lambda mitch-garnaat coding testing cli kappa)

Vault

HashiCorp’s take on the secrets-storage system. looks good

(tags: hashicorp deployment security secrets authentication vault storage keys key-rotation)

Everything Science Knows Right Now About Standing Desks | Co.Design

“Overall, current evidence suggests that both standing and treadmill desks may be effective in improving overall health considering both physiological and mental health components.”

(tags: standing-desks treadmill-desks desks exercise health work workplace back sitting standing)

Race conditions on Facebook, DigitalOcean and others

good trick — exploit eventual consistency and a lack of distributed transactions by launching race-condition-based attacks

(tags: attacks exploits race-conditions bugs eventual-consistency distributed-transactions http facebook digitalocean via:aphyr)

StackShare

‘Discover and discuss the best dev tools and cloud infrastructure services’ — fun!

(tags: stackshare architecture stack ops software ranking open-source)

OWASP KeyBox

a web-based SSH console that centrally manages administrative access to systems. Web-based administration is combined with management and distribution of user’s public SSH keys. Key management and administration is based on profiles assigned to defined users. Administrators can login using two-factor authentication with FreeOTP or Google Authenticator . From there they can create and manage public SSH keys or connect to their assigned systems through a web-shell. Commands can be shared across shells to make patching easier and eliminate redundant command execution.

(tags: keybox owasp security ssh tls ssl ops)

32-bit overflow in BitGo js code caused an accidental 85 BTC transaction fee

Yes, this is a fucking 32-bit integer overflow. Whatever software was used, it calculated the sum of all inputs using 32-bit variables, which overflow at about 20 BTC if signed or 40 BTC if not. The fee was supposed to be 0xC350 = 50,000 satoshis, but it turned out to be 0x2,0000,C350 = 8,589,984,592 satoshis. Captains of the industry. If they were captains of any other industry, like say for example automotive, we’d have people dying in car crashes between two stationary vehicles.

(tags: bitcoin fail bitgo javascript bugs 32-bit overflow btc)

Eight Docker Development Patterns

good Docker tips

(tags: tips docker ops deployment)

Google Online Security Blog: A Javascript-based DDoS Attack [the Greatfire DDoS] as seen by Safe Browsing

We hope this report helps to round out the overall facts known about this attack. It also demonstrates that collectively there is a lot of visibility into what happens on the web. At the HTTP level seen by Safe Browsing, we cannot confidently attribute this attack to anyone. However, it makes it clear that hiding such attacks from detailed analysis after the fact is difficult. Had the entire web already moved to encrypted traffic via TLS, such an injection attack would not have been possible. This provides further motivation for transitioning the web to encrypted and integrity-protected communication. Unfortunately, defending against such an attack is not easy for website operators. In this case, the attack Javascript requests web resources sequentially and slowing down responses might have helped with reducing the overall attack traffic. Another hope is that the external visibility of this attack will serve as a deterrent in the future.

(tags: google security via:nelson ddos javascript tls ssl safe-browsing networking china greatfire)

Amazon EC2 Container Service team AmA

a few answers here. Mostly people pointing out shortcomings and the team asking them to start a thread on their forum though :(

(tags: ec2 ecs docker aws ops ama reddit)

Cluster-Based Architectures Using Docker and Amazon EC2 Container Service

In this post, we’re going to take a deeper dive into the architectural concepts underlying cluster computing using container management frameworks such as ECS. We will show how these frameworks effectively abstract the low-level resources such as CPU, memory, and storage, allowing for highly efficient usage of the nodes in a compute cluster. Building on some of the concepts detailed in the earlier posts, we will discover why containers are such a good fit for this type of abstraction, and how the Amazon EC2 Container Service fits into the larger ecosystem of cluster management frameworks.

(tags: docker aws ecs ec2 ops hosting containers mesos clusters)

Kubernetes compared to Borg

‘Here are four Kubernetes features that came from our experiences with Borg.’

(tags: google ops kubernetes borg containers docker networking)

attacks using U+202E – RIGHT-TO-LEFT OVERRIDE

Security implications of in-band signalling strikes again, 43 years after the “Blue Box” hit the mainstream. Jamie McCarthy on Twitter: “.@cmdrtaco – Remember when we had to block the U+202E code point in Slashdot comments to stop siht ekil stnemmoc? https://t.co/TcHxKkx9Oo” See also http://krebsonsecurity.com/2011/09/right-to-left-override-aids-email-attacks/ — GMail was vulnerable too; and http://en.wikipedia.org/wiki/Unicode_control_characters for more inline control chars. http://unicode.org/reports/tr36/#Bidirectional_Text_Spoofing has some official recommendations from the Unicode consortium on dealing with bidi override chars.

(tags: security attacks rlo unicode control-characters codepoints bidi text gmail slashdot sanitization input)

Meet the e-voting machine so easy to hack, it will take your breath away | Ars Technica

The AVS WinVote system — mind-bogglingly shitty security.

If an election was held using the AVS WinVote, and it wasn’t hacked, it was only because no one tried. The vulnerabilities were so severe, and so trivial to exploit, that anyone with even a modicum of training could have succeeded. They didn’t need to be in the polling place—within a few hundred feet (e.g., in the parking lot) is easy, and within a half mile with a rudimentary antenna built using a Pringles can. Further, there are no logs or other records that would indicate if such a thing ever happened, so if an election was hacked any time in the past, we will never know. I’ve been in the security field for 30 years, and it takes a lot to surprise me. But the VITA report really shocked me—as bad as I thought the problems were likely to be, VITA’s five-page report showed that they were far worse. And the WinVote system was so fragile that it hardly took any effort. While the report does not state how much effort went into the investigation, my estimation based on the description is that it was less than a person week.

(tags: security voting via:johnke winvote avs shoup wep wifi windows)

‘Continuous Deployment: The Dirty Details’

Good slide deck from Etsy’s Mike Brittain regarding their CD setup. Some interesting little-known details: Slide 41: database schema changes are not CD’d — they go out on “Schema change Thursdays”. Slide 44: only the webapp is CD’d — PHP, Apache, memcache components (Etsy.com, support and back-office tools, developer API, gearman async worker queues). The external “services” are not — databases, Solr/JVM search (rolling restarts), photo storage (filters, proxy cache, S3), payments (PCI-DSS, controlled access). They avoid schema changes and breaking changes using an approach they call “non-breaking expansions” — expose new version in a service interface; support multiple versions in the consumer. Example from slides 50-63, based around a database schema migration. Slide 66: “dev flags” (rollout oriented) are promoted to “feature flags” (long lived degradation control). Slide 71: some architectural philosophies: deploying is cheap; releasing is cheap; gathering data should be cheap too; treat first iterations as experiments. Slide 102: “Canary pools”. They have multiple pools of users for testing in production — the staff pool, users who have opted in to see prototypes/beta stuff, 0-100% gradual phased rollout.

(tags: cd deploy etsy slides migrations database schema ops ci version-control feature-flags)

Etsy’s Release Management process

Good info on how Etsy use their Deployinator tool, end-to-end. Slide 11: git SHA is visible for each env, allowing easy verification of what code is deployed. Slide 14: Code is deployed to “princess” staging env while CI tests are running; no need to wait for unit/CI tests to complete. Slide 23: smoke tests of pre-prod “princess” (complete after 8 mins elapsed). Slide 31: dashboard link for deployed code is posted during deploy; post-release prod smoke tests are run by Jenkins. (short ones! they complete in 42 seconds)

(tags: deployment etsy deploy deployinator princess staging ops testing devops smoke-tests production jenkins)

Makerbot’s Saddest Hour | TechCrunch

I’ve been speaking to a few people [at Makerbot] who prefer to remain anonymous and most of my contacts there are gone (the head of PR was apparently fired) and don’t want to talk. But the new from inside is troubling. The mass-layoffs are blamed on low revenue and one former employee wrote “Company was failing. Couldn’t pay vendors, had to downsize.” Do I think Makerbot will sink? At this point I don’t know.

(tags: makerbot 3d-printing startups downsizing layoffs ouch)

credstash

‘CredStash is a very simple, easy to use credential management and distribution system that uses AWS Key Management System (KMS) for key wrapping and master-key storage, and DynamoDB for credential storage and sharing.’

(tags: aws credstash python security keys key-management secrets kms)

distributed scalability systems coding server-side erlang devops networking reliability)

Internet Scale Services Checklist

good aspirational checklist, inspired heavily by James Hamilton’s seminal 2007 paper, “On Designing And Deploying Internet-Scale Services”

(tags: james-hamilton checklists ops internet-scale architecture operability monitoring reliability availability uptime aspirations)

FBI admits flaws in hair analysis over decades

Wow, this is staggering.

The Justice Department and FBI have formally acknowledged that nearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all trials in which they offered evidence against criminal defendants over more than a two-decade period before 2000. [….] The review confirmed that FBI experts systematically testified to the near-certainty of “matches” of crime-scene hairs to defendants, backing their claims by citing incomplete or misleading statistics drawn from their case work. In reality, there is no accepted research on how often hair from different people may appear the same. Since 2000, the lab has used visual hair comparison to rule out someone as a possible source of hair or in combination with more accurate DNA testing. Warnings about the problem have been mounting. In 2002, the FBI reported that its own DNA testing found that examiners reported false hair matches more than 11 percent of the time.

(tags: fbi false-positives hair dna biometrics trials justice experts crime forensics inaccuracy csi)

The missing MtGox bitcoins

Most or all of the missing bitcoins were stolen straight out of the MtGox hot wallet over time, beginning in late 2011. As a result, MtGox operated at fractional reserve for years (knowingly or not), and was practically depleted of bitcoins by 2013. A significant number of stolen bitcoins were deposited onto various exchanges, including MtGox itself, and probably sold for cash (which at the bitcoin prices of the day would have been substantially less than the hundreds of millions of dollars they were worth at the time of MtGox’s collapse). MtGox’ bitcoins continuously went missing over time, but at a decreasing pace. Again by the middle of 2013, the curve goes more or less flat, matching the hypothesis that by that time there may not have been any more bitcoins left to lose. The rate of loss otherwise seems unusually smooth and at the same time not strictly relative to any readily available factors such as remaining BTC holdings, transaction volumes or the BTC price. Worth pointing out is that, thanks to having matched up most of the deposit/withdrawal log earlier, we can at this point at least rule out the possibility of any large-scale fake deposits — the bitcoins going into MtGox were real, meaning the discrepancy was likely rather caused by bitcoins leaving MtGox without going through valid withdrawals.

(tags: mtgox bitcoin security fail currency theft crime btc)

Bank of the Underworld – The Atlantic

Prosecutors analyzed approximately 500 of Liberty Reserve’s biggest accounts, which constituted 44 percent of its business. The government contends that 32 of these accounts were connected to the sale of stolen credit cards and 117 were used by Ponzi-scheme operators. All of this activity flourished, prosecutors said, because Liberty Reserve made no real effort to monitor its users for criminal behavior. What’s more, records showed that one of the company’s top tech experts, Mark Marmilev, who was also arrested, appeared to have promoted Liberty Reserve in chat rooms devoted to Ponzi schemes.

(tags: scams fraud crime currency the-atlantic liberty-reserve ponzi-schemes costa-rica arthur-budovsky banking anonymity cryptocurrency money-laundering carding)

I was a Lampedusa refugee. Here’s my story of fleeing Libya – and surviving

‘The boy next to me fell to the floor and for a moment I didn’t know if he had fainted or was dead – then I saw that he was covering his eyes so he didn’t have to see the waves any more. A pregnant woman vomited and started screaming. Below deck, people were shouting that they couldn’t breathe, so the men in charge of the boat went down and started beating them. By the time we saw a rescue helicopter, two days after our boat had left Libya with 250 passengers on board, some people were already dead – flung into the sea by the waves, or suffocated downstairs in the dark.’

(tags: lampedusa migration asylum europe fortress-europe italy politics immigration libya refugees)

Run your own high-end cloud gaming service on EC2

Using Steam streaming and EC2 g2.2xlarge spot instances — ‘comes out to around $0.52/hr’. That’s pretty compelling IMO

(tags: aws ec2 gaming games graphics spot-instances hacks windows steam)

Running Arbitrary Executables in AWS Lambda

actually an officially-supported mode. huh

(tags: lambda aws architecture ops node.js javascript unix linux)

Exclusive: Chopra says ECB’s threats to Ireland were ‘outrageous’ – Independent.ie

The letters urged the then-government to commit to structural reforms and restructuring of the financial sector. “That is not their job,” Mr Chopra said. “Their mandate is to meet inflation. And if you lecture the ECB as to how they might go about that, they talk about their independence. “But when it comes to lecturing others about fiscal policy or structural policy, they’re not at all hesitant. I’m not surprised that the people in Ireland were very upset about these letters from [Jean-Claude] Trichet.”

(tags: trichet banking ireland politics ajai-chopra ecb history)

Writing Minecraft Plugins – The Book

wow, Walter Higgins’ book (from Peachpit Press) is looking great

(tags: books reading minecraft walter-higgins javascript)

Pinball

Pinterest’s Hadoop workflow manager; ‘scalable, reliable, simple, extensible’ apparently. Hopefully it allows upgrades of a workflow component without breaking an existing run in progress, like LinkedIn’s Azkaban does :(

(tags: python pinterest hadoop workflows ops pinball big-data scheduling)

HACKERS COULD COMMANDEER NEW PLANES THROUGH PASSENGER WI-FI

Boeing 787 Dreamliner jets, as well as Airbus A350 and A380 aircraft, have Wi-Fi passenger networks that use the same network as the avionics systems of the planes

(tags: air-gap security planes boeing a380 a350 dreamliner networking firewalls avionics)

Tips for debugging EC2 Container Service

some basic ECS tips from Gilt

(tags: gilt ecs tips ops advice ec2 aws)

_Blade: a Data Center Garbage Collector_

Essentially, add a central GC scheduler to improve tail latencies in a cluster, by taking instances out of the pool to perform slow GC activity instead of letting them impact live operations. I’ve been toying with this idea for a while, nice to see a solid paper about it

(tags: gc latency tail-latencies papers blade go java scheduling clustering load-balancing low-latency performance)

SCADA systems online, and a horror story about a non-airgapped Boeing 747 engine management system

747’s are big flying Unix hosts. At the time, the engine management system on this particular airline was Solaris based. The patching was well behind and they used telnet as SSH broke the menus and the budget did not extend to fixing this. The engineers could actually access the engine management system of a 747 in route. If issues are noted, they can re-tune the engine in air. The issue here is that all that separated the engine control systems and the open network was NAT based filters. There were (and as far as I know this is true today), no extrusion controls. They filter incoming traffic, but all outgoing traffic is allowed.

(tags: air-gap planes boeing security 747 solaris unix)

Squarespace

Nice, simple “build a website” platform. Keeping this one bookmarked for the next time someone non-techie asks me for the simplest way to do just that (thanks for the tip, Oisin)

(tags: via:oisin blog cms design hosting web-design web websites)

Extracting Structured Data From Recipes Using Conditional Random Fields

nice probabilistic/ML approach to recipe parsing

(tags: nytimes recipes parsing text nlp machine-learning probabilistic crf++ algorithms feature-extraction)

Large-scale cluster management at Google with Borg

Google’s Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior. We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it.

(tags: via:conall clustering google papers scale to-read borg cluster-management deployment packing reliability redundancy)

Keeping Your Car Safe From Electronic Thieves – NYTimes.com

In a normal scenario, when you walk up to a car with a keyless entry and try the door handle, the car wirelessly calls out for your key so you don’t have to press any buttons to get inside. If the key calls back, the door unlocks. But the keyless system is capable of searching for a key only within a couple of feet. Mr. Danev said that when the teenage girl turned on her device, it amplified the distance that the car can search, which then allowed my car to talk to my key, which happened to be sitting about 50 feet away, on the kitchen counter. And just like that, open sesame.

(tags: security fail cars keys signal proximity keyless-entry prius toyota crime amplification power-amplifiers 3db keyless)

Closed access means people die

‘We’ve paid 100 BILLION USD over the last 10 years to “publish” science and medicine. Ebola is a massive systems failure.’ See also https://www.techdirt.com/articles/20150409/17514230608/dont-think-open-access-is-important-it-might-have-prevented-much-ebola-outbreak.shtml : ‘The conventional wisdom among public health authorities is that the Ebola virus, which killed at least 10,000 people in Liberia, Sierra Leone and Guinea, was a new phenomenon, not seen in West Africa before 2013. […] But, as the team discovered, that “conventional wisdom” was wrong. In fact, they found a bunch of studies, buried behind research paywalls, that revealed that there was significant evidence of antibodies to the Ebola virus in Liberia and in other nearby nations. There was one from 1982 that noted: “medical personnel in Liberian health centers should be aware of the possibility that they may come across active cases and thus be prepared to avoid nosocomial epidemics.”

(tags: deaths liberia ebola open-access papers elsevier science medicine reprints)

Making Pinterest — Learn to stop using shiny new things and love MySQL

‘The third reason people go for shiny is because older tech isn’t advertised as aggressively as newer tech. The younger companies needs to differentiate from the old guard and be bolder, more passionate and promise to fulfill your wildest dreams. But most new tech sales pitches aren’t generally forthright about their many failure modes. In our early days, we fell into this third trap. We had a lot of growing pains as we scaled the architecture. The most vocal and excited database companies kept coming to us saying they’d solve all of our scalability problems. But nobody told us of the virtues of MySQL, probably because MySQL just works, and people know about it.’ It’s true! — I’m still a happy MySQL user for some use cases, particularly read-mostly relational configuration data…

(tags: mysql storage databases reliability pinterest architecture)

Microservices and elastic resource pools with Amazon EC2 Container Service

interesting approach to working around ECS’ shortcomings — bit specific to Hailo’s microservices arch and IPC mechanism though. aside: I like their version numbering scheme: ISO-8601, YYYYMMDDHHMMSS. keep it simple!

(tags: versioning microservices hailo aws ec2 ecs docker containers scheduling allocation deployment provisioning qos)

Please Kill Me (Eventually) | Motherboard

There is much that the wise application of technology can do to help us ease off this mortal coil, instead of tormenting ourselves at the natural end of life in a futile, undignified and excruciating attempt to keep it somehow duct-taped on. Train more people in geriatrics, for example. Learn new ways to make life safe, healthy, fun and interesting for the old. Think like a community, a brotherhood, not like atomized competing individuals a few of whom can somehow “beat the system” of the universe. Maybe it is better to examine clearly what we are with a view to understanding and acceptance than it is to try to escape what perhaps should be our inevitable ending.

(tags: death mortality cryogenics alcor geriatrics life singularity mind-uploading ray-kurzweil)

CGA in 1024 Colors – a New Mode: the Illustrated Guide

awesome hackery. brings me back to my C=64 demo days

(tags: pc cga graphics hacks art 1024-colours)

Keywhiz

‘a secret management and distribution service [from Square] that is now available for everyone. Keywhiz helps us with infrastructure secrets, including TLS certificates and keys, GPG keyrings, symmetric keys, database credentials, API tokens, and SSH keys for external services — and even some non-secrets like TLS trust stores. Automation with Keywhiz allows us to seamlessly distribute and generate the necessary secrets for our services, which provides a consistent and secure environment, and ultimately helps us ship faster. […] Keywhiz has been extremely useful to Square. It’s supported both widespread internal use of cryptography and a dynamic microservice architecture. Initially, Keywhiz use decoupled many amalgamations of configuration from secret content, which made secrets more secure and configuration more accessible. Over time, improvements have led to engineers not even realizing Keywhiz is there. It just works. Please check it out.’

(tags: square security ops keys pki key-distribution key-rotation fuse linux deployment secrets keywhiz)

Bigcommerce Status Page blasts IBM Softlayer Object Storage service

This is pretty heavy stuff:

Bigcommerce engineers have been very pro-active in working with our storage provider, IBM Softlayer, in finding solutions. Unfortunately, it takes two parties to come to a solution. In this case, IBM Softlayer intentionally let their Object Storage cluster fall into disrepair and chose not to scale it. This has impacted Bigcommerce, IBM and many other Softlayer customers. Our engineers placed too much trust in IBM Softlayer and that’s on us. However, the catastrophic failures to see metrics and rapidly scale capacity, the decisions to let hard drives sit at 90% utilization for weeks and months, the cascading failures of an undersized cluster of 52 nodes for the busiest data center in their business speaks to IBM Softlayer’s lack of concern for their customers. We found this out 3 days ago.

(tags: softlayer bigcommerce outages shambles ibm fail object-storage storage iaas cloud)

Subscribing AWS Lambda Function To SNS Topic With aws-cli

how to use the AWS command line tools to do this

(tags: aws aws-cli cli lambda sns hacks)

Yelp Product & Engineering Blog | True Zero Downtime HAProxy Reloads

Using tc and qdisc to delay SYNs while haproxy restarts. Definitely feels like on-host NAT between 2 haproxy processes would be cleaner and easier though!

(tags: linux networking hacks yelp haproxy uptime reliability tcp tc qdisc ops)

Amazon Machine Learning

Upsides of this new AWS service: * great UI and visualisations. * solid choice of metric to evaluate the results. Maybe things moved on since I was working on it, but the use of AUC, false positives and false negatives was pretty new when I was working on it. (er, 10 years ago!) Downsides: * it could do with more support for unsupervised learning algorithms. Supervised learning means you need to provide training data, which in itself can be hard work. My experience with logistic regression in the past is that it requires very accurate training data, too — its tolerance for misclassified training examples is poor. * Also, in my experience, 80% of the hard work of using ML algorithms is writing good tokenisation and feature extraction algorithms. I don’t see any help for that here unfortunately. (probably not that surprising as it requires really detailed knowledge of the input data to know what classes can be abbreviated into a single class, etc.)

(tags: amazon aws ml machine-learning auc data-science)

Rob Pike’s 5 rules of optimization

these are great. I’ve run into rule #3 (“fancy algorithms are slow when n is small, and n is usually small”) several times…

(tags: twitter rob-pike via:igrigorik coding rules laws optimization performance algorithms data-structures aphorisms)

AWS Lambda Event-Driven Architecture With Amazon SNS

Any message posted to an SNS topic can trigger the execution of custom code you have written, but you don’t have to maintain any infrastructure to keep that code available to listen for those events and you don’t have to pay for any infrastructure when the code is not being run. This is, in my opinion, the first time that Amazon can truly say that AWS Lambda is event-driven, as we now have a central, independent, event management system (SNS) where any authorized entity can trigger the event (post a message to a topic) and any authorized AWS Lambda function can listen for the event, and neither has to know about the other.

(tags: aws ec2 lambda sns events cep event-processing coding cloud hacks eric-hammond)

Texting at the wheel kills more US teenagers every year than drink-driving

Texting while behind the wheel has overtaken drink driving as the biggest cause of death among teenagers in America. More than 3,000 teenagers are killed every year in car crashes caused by texting while driving compared to 2,700 from drink driving. The study by Cohen Children’s Medical Center also discovered that 50 per cent of students admit to texting while driving.

(tags: texting sms us driving car-safety safety drink-driving)

China’s Great Cannon

Conducting such a widespread attack clearly demonstrates the weaponization of the Chinese Internet to co-opt arbitrary computers across the web and outside of China to achieve China’s policy ends.  The repurposing of the devices of unwitting users in foreign jurisdictions for covert attacks in the interests of one country’s national priorities is a dangerous precedent — contrary to international norms and in violation of widespread domestic laws prohibiting the unauthorized use of computing and networked systems.

(tags: censorship ddos internet security china great-cannon citizen-lab reports web)

Sirius: An open end-to-end voice and vision personal assistant and its implications for future warehouse scale computers

How to build an Intelligent Personal Assistant: ‘Sirius is an open end-to-end standalone speech and vision based intelligent personal assistant (IPA) similar to Apple’s Siri, Google’s Google Now, Microsoft’s Cortana, and Amazon’s Echo. Sirius implements the core functionalities of an IPA including speech recognition, image matching, natural language processing and a question-and-answer system. Sirius is developed by Clarity Lab at the University of Michigan. Sirius is published at the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2015.’

(tags: sirius siri cortana google-now echo ok-google ipa assistants search video audio speech papers clarity nlp wikipedia)

Why We Will Not Be Registering easyDNS.SUCKS – blog.easydns.org

If you’re not immersed in the naming business you may find the jargon in it hard to understand. The basic upshot is this: the IPC believes that the mechanisms that were enacted to protect trademark holders during the deluge of new TLD rollouts are being gamed by the .SUCKS TLD operator to extort inflated fees from trademark holders.

(tags: shakedown business internet domains dns easydns dot-sucks scams tlds trademarks ip)

Data privacy is as important as tax, Google exec warns Noonan – Independent.ie

Yep, that would be Google requesting more regulation ;)

(tags: google regulation ireland privacy data-protection)

Russia just made a ton of Internet memes illegal – The Washington Post

In post-Soviet Russia, you don’t make memes. Memes make (or unmake?) you. That is, at least, the only conclusion we can draw from an announcement made this week by Russia’s three-year-old media agency/Internet censor Roskomnadzor, which made it illegal to publish any Internet meme that depicts a public figure in a way that has nothing to do with his “personality.”

(tags: memes photoshop russia freedom web internet funny humour roskomnadzor censorship sad-keanu)

OG-Commons/Guavate.java

‘Utilities that help bridge the gap between Java 8 and Google Guava. Guava has the {@link FluentIterable} concept which is similar to streams. In many ways, fluent iterable is nicer, because it directly binds to the immutable collection classes. However, on balance it seems wise to use the stream API rather than {@code FluentIterable} in Java 8.’

(tags: guava java-8 java fluentiterable streams fluent coding)