Skip to content

Justin's Linklog Posts

Links for 2018-02-15

Links for 2018-02-14

  • Single Trapped Atom Captures Science Photography Competition’s top prize – EPSRC website

    An image of a single positively-charged strontium atom, held near motionless by electric fields, has won the overall prize in a national science photography competition, organised by the Engineering and Physical Sciences Research Council (EPSRC). ‘Single Atom in an Ion Trap’, by David Nadlinger, from the University of Oxford, shows the atom held by the fields emanating from the metal electrodes surrounding it. The distance between the small needle tips is about two millimetres. When illuminated by a laser of the right blue-violet colour the atom absorbs and re-emits light particles sufficiently quickly for an ordinary camera to capture it in a long exposure photograph. The winning picture was taken through a window of the ultra-high vacuum chamber that houses the ion trap.

    (tags: atom photography science strontium ion-traps light)

Links for 2018-02-12

  • Thousands of websites hijacked by hidden crypto-mining code after Browsealoud hacked

    The affected sites all use a fairly popular plugin called Browsealoud, made by Brit biz Texthelp, which reads out webpages for blind or partially sighted people. This technology was compromised in some way – either by hackers or rogue insiders altering Browsealoud’s source code – to silently inject Coinhive’s Monero miner into every webpage offering Browsealoud. For several hours today, anyone who visited a site that embedded Browsealoud inadvertently ran this hidden mining code on their computer, generating money for the miscreants behind the caper. A list of 4,200-plus affected websites can be found here: they include The City University of New York (cuny.edu), Uncle Sam’s court information portal (uscourts.gov), Lund University (lu.se), the UK’s Student Loans Company (slc.co.uk), privacy watchdog The Information Commissioner’s Office (ico.org.uk) and the Financial Ombudsman Service (financial-ombudsman.org.uk), plus a shedload of other .gov.uk and .gov.au sites, UK NHS services, and other organizations across the globe. Manchester.gov.uk, NHSinform.scot, agriculture.gov.ie, Croydon.gov.uk, ouh.nhs.uk, legislation.qld.gov.au, the list goes on.

    (tags: browsealoud accessibility http sri coinhive monero hacks ico nhs)

Links for 2018-02-09

  • How the Game Genie worked

    “Sometimes it was really easy to find cheats, because the code was very straightforward, and sometimes it was a massive pain in the arse,” recalls Jon. “In simple terms, if a game started you with three lives I’d set up the logic analyser to stop when it found the value three being written to RAM. Then I’d use the Game Genie to change that 3 to say a 5, reboot the game and see if I started with 5 lives. If not, then I’d let it find the next time it wrote 3 into RAM and try that. “Infinite lives codes were always the best. Once I’d found where in RAM the lives value was stored I’d then monitor when it got decremented. What I was looking for was where the game’s original coder used -most likely – the DEC A (&H3D) instruction after reading the lives value from RAM, and then storing it back into RAM. If I found this then all I had to do was swap out the DEC A (&H3D) decrement operation with a NOP (&H00), which performed no operation. So the lives value would be left as-is and voila the player had infinite lives.”

    (tags: games gameboy game-genie via:its logic-analysers reverse-engineering history hacking)

  • Last orders: Ireland’s vanishing ‘quirky’ shopfronts – in pictures | Cities | The Guardian

    Graphic designer Trevor Finnegan spent seven years documenting traditional shopfronts throughout Ireland.
    Lovely examples of a vanishing vernacular style.

    (tags: architecture ireland rural shopfronts signs history)

  • Russia Did It, Y’all. And Nobody Fucking Cares.

    That’s right, that’s CRAZY LIBERAL CONSPIRACY THEORIST George W. Bush […] saying it’s still an open question whether Russia actually successfully rigged the 2016 election. What a Code Pink Occupy Democracy Now liberal George W. Bush is being, to even ask that question!

    (tags: wonkette elections donald-trump 2016 us-politics george-w-bush hacking)

  • Car Hacker’s Handbook

    Modern cars are more computerized than ever. Infotainment and navigation systems, Wi-Fi, automatic software updates, and other innovations aim to make driving more ­convenient. But vehicle technologies haven’t kept pace with today’s more hostile security environment, leaving ­millions vulnerable to attack. The Car Hacker’s Handbook will give you a deeper understanding of the computer systems and embedded software in modern ­vehicles. It begins by examining vulnerabilities and providing detailed explanations of communications over the CAN bus and ­between devices and systems. Then, once you have an understanding of a vehicle’s communication network, you’ll learn how to intercept data and perform specific hacks to track vehicles, unlock doors, glitch engines, flood communication, and more.
    Creative Commons Attribution-Noncommercial-ShareAlike license.

    (tags: cars books hacking exploits can-bus)

Links for 2018-02-07

  • Pubs reveal drinks firms’ exclusive deals

    ‘You could be offered €100,000 – it’s big money’: Pubs reveal drinks firms’ exclusive deals; Heineken has already been accused of using its clout to squeeze out rivals.

    (tags: heineken beer ireland craft-beer payola scams pubs)

  • Horslips respond angrily to xenophobic #irexit use of their hit “Dearg Doom”

    Some of you may have spotted that the saddos in the Eirexit conference had the feckin’ temerity to use Dearg Doom as a soundtrack and to show the image of the album cover on the big screen. Needless to say, they didn’t ask us. If they had, we’d have pointed out that we wouldn’t piss on them if they were on fire -which they’re unlikely to be, anytime soon. Five hundred damp, self regarding eejits being patronised by the Crazy Frog lookalike Nigel Farage … isn’t going to set the heather blazing in the near future. Horslips stood for a hopeful, outward looking, inclusive vision of Ireland with plenty of drink and a Blue Range Rover. This lot stand for a diminished, fearful, xenophobic state. Little Irelanders. Checking out whether we can do them for copyright infringement. We’ll keep you posted.Feel free to share.
    legends.

    (tags: horslips music 1970s irexit facebook copyright)

  • why Cheddar Man was dark skinned

    ‘But why should that be surprising? He’s over 10,000 years old, while mutations that led to white skin [the depigmentation gene SLC24A5] only began to spread widely [across Europe] 5,800 years old!’

    (tags: europe history prehistory skin-colour cheddar-man race skin slc24a5 genetics david-grimes)

Links for 2018-02-02

  • Amazon Aurora Parallel Query is Available for Preview

    Looks very nifty (at least once it’s GA)

    Parallel Query improves the performance of large analytic queries by pushing processing down to the Aurora storage layer, spreading processing across hundreds of nodes. With Parallel Query, you can run sophisticated analytic queries on Aurora tables with an order of magnitude performance improvement over serial query processing, in many cases. Parallel Query currently pushes down predicates used to filter tables and hash joins. 

    (tags: parallel aurora amazon mysql sql performance joins architecture data-model)

Links for 2018-01-30

Links for 2018-01-29

  • Key metrics for RabbitMQ monitoring

    Good suggestions from Datadog

    (tags: rabbitmq mq monitoring metrics graphite ops)

  • Amazing thread from @gavinsblog on the Strava leak

    ‘This often led to the same results you see with Strava. In low population countries, or countries with low smartphone penetration, it was often easy to detect Westerners (usually soldiers) in remote areas. this usually led to being able to identify bases and other types of things based solely on social data. Iraq, Afghanistan = always easy to find US troops (Instagram being a common sharing tool). Same true of IDF troops in staging areas before invasion of Gaza in 2014. and the same true in 2014 with Russian troops in Ukraine. All too easy. Of course the other thing you might be nosey about [is] known military facilities. Social geotagging can give you staff/visitor lists if you persist long enough. the difference between this technique and Strava was you could usually quickly deduce first name/last name if you wanted, and infer other social profiles eg LinkedIn -> FB -> FB friends -> work colleagues. Not only that but it was possible to automate.’

    (tags: strava privacy military security geotagging geodata gavin-sheridan)

  • My £300 32Amp Charging Station Install

    good writeup of a DIY EV car charger install

    (tags: ev cars diy car-chargers home)

Links for 2018-01-28

  • Strava app gives away location of secret US army bases

    This is a privacy nightmare. Even with anonymized userids the data was far too user-specific.

    The details were released by Strava in a data visualisation map that shows all the activity tracked by users of its app, which allows people to record their exercise and share it with others. The map, released in November 2017, shows every single activity ever uploaded to Strava – more than 3 trillion individual GPS data points, according to the company. The app can be used on various devices including smartphones and fitness trackers like Fitbit to see popular running routes in major cities, or spot individuals in more remote areas who have unusual exercise patterns.

    (tags: strava privacy fail army us-army data)

Links for 2018-01-27

  • ‘A Look into 30 Years of Malware Development from a Software Metrics Perspective’

    ‘During the last decades, the problem of malicious and unwanted software (malware) has surged in numbers and sophistication. Malware plays a key role in most of today’s cyber attacks and has consolidated as a commodity in the underground economy. In this work, we analyze the evolution of malware since the early 1980s to date from a software engineering perspective. We analyze the source code of 151 malware samples and obtain measures of their size, code quality, and estimates of the development costs (effort, time, and number of people). Our results suggest an exponential increment of nearly one order of magnitude per decade in aspects such as size and estimated effort, with code quality metrics similar to those of regular software. Overall, this supports otherwise confirmed claims about the increasing complexity of malware and its production progressively becoming an industry.’

    (tags: malware coding metrics software history complexity arms-race)

Links for 2018-01-25

  • Rocket Lab secretly launched a disco ball satellite on its latest test flight – The Verge

    I’m quite conflicted about this — I think I like it:

    Shaped a bit like a disco ball, the Humanity Star is a 3-foot-wide carbon fiber sphere, made up of 65 panels that reflect the Sun’s light. The satellite is supposed to spin in space, too, so it’s constantly bouncing sunlight. In fact, the probe is so bright that people can see it with the naked eye. The Humanity Star’s orbit also takes it all over Earth, so the satellite will be visible from every location on the planet at different times. Rocket Lab has set up a website that gives real-time updates about the Humanity Star’s location. People can find out when the satellite will be closest to them, and then go outside to look for it. The goal of the project is to create “a shared experience for all of humanity,” according to Rocket Lab.

    (tags: rocket-lab disco-balls satellites humanity-star orbit space)

  • 3D Scans of 7,500 Famous Sculptures, Statues & Artworks: Download & 3D Print Rodin’s Thinker, Michelangelo’s David & More | Open Culture

    oh my.

    (tags: 3d-printing art history british-museum models cool)

  • ‘DolphinAttack: Inaudible Voice Commands’ [pdf]

    ‘Speech recognition (SR) systems such as Siri or Google Now have become an increasingly popular human-computer interaction method, and have turned various systems into voice controllable systems(VCS). Prior work on attacking VCS shows that the hidden voice commands that are incomprehensible to people can control the systems. Hidden voice commands, though hidden, are nonetheless audible. In this work, we design a completely inaudible attack, DolphinAttack, that modulates voice commands on ultrasonic carriers (e.g., f > 20 kHz) to achieve inaudibility. By leveraging the nonlinearity of the microphone circuits, the modulated low frequency audio commands can be successfully demodulated, recovered, and more importantly interpreted by the speech recognition systems. We validate DolphinAttack on popular speech recognition systems, including Siri, Google Now, Samsung S Voice, Huawei HiVoice, Cortana and Alexa. By injecting a sequence of inaudible voice commands, we show a few proof-of-concept attacks, which include activating Siri to initiate a FaceTime call on iPhone, activating Google Now to switch the phone to the airplane mode, and even manipulating the navigation system in an Audi automobile. We propose hardware and software defense solutions. We validate that it is feasible to detect DolphinAttack by classifying the audios using supported vector machine (SVM), and suggest to re-design voice controllable systems to be resilient to inaudible voice command attacks.’ via Zeynep (https://twitter.com/zeynep/status/956520320504123392)

    (tags: alexa siri attacks security exploits google-now speech-recognition speech audio acm papers cortana)

Links for 2018-01-24

  • Targeted Audio Adversarial Examples

    This is phenomenal:

    We have constructed targeted audio adversarial examples on speech-to-text transcription neural networks: given an arbitrary waveform, we can make a small perturbation that when added to the original waveform causes it to transcribe as any phrase we choose. In prior work, we constructed hidden voice commands, audio that sounded like noise but transcribed to any phrases chosen by an adversary. With our new attack, we are able to improve this and make an arbitrary waveform transcribe as any target phrase.
    The audio examples on this page are impressive — a little bit of background noise, such as you might hear on a telephone call with high compression, hard to perceive if you aren’t listening out for it. Paper here: https://arxiv.org/abs/1801.01944 (Via Parker Higgins, https://twitter.com/xor )

    (tags: papers audio adversarial-classification neural-networks speech-to-text speech recognition voice attacks exploits via:xor)

Links for 2018-01-22

Links for 2018-01-18

Links for 2018-01-17

  • Boost your immunity: Cold and flu treatments suppress innate immune system

    The next time you feel a cold coming on, maybe what you really want is just a little teensy bit of innate immune suppression, not an immunity boost. Over-the-counter medications like ibuprofen and antihistamines should help you feel better. Meanwhile, sit back while your acquired B and T cells do the rest. And if you aren’t yet sick, stay up-to-date on your vaccines, including the yearly influenza vaccine. Most importantly, practice vigorous hand washing — after all, the skin is also a component of your natural defenses and one that actually can be enhanced by good hygiene. Take care of yourself by keeping a balanced diet, maintaining good sleep habits, and minimizing stress. These are interventions that have been shown to help keep your immune system at its best. These alone can “boost” your odds of staving off an infection this cold season.

    (tags: immunity health immune-system colds b-cells t-cells flu)

  • Sarah Jeong’s hilarious Twitter thread on Bitcoin

    “People are sick of the Federal Reserve, sick of bailouts, sick of inflation. You know what we need? Internet money with the usability of PGP and the reliability of BART” and much, much more

    (tags: bitcoin funny sarah-jeong comedy lols pgp twitter threads)

  • How To Measure the Working Set Size on Linux

    A nifty metric:

    The Working Set Size (WSS) is how much memory an application needs to keep working. Your app may have populated 100 Gbytes of main memory, but only uses 50 Mbytes each second to do its job. That’s the working set size. It is used for capacity planning and scalability analysis. You may never have seen WSS measured by any tool (I haven’t either). OSes usually show you virtual memory and resident memory, shown as the “VIRT” and “RES” columns in top. Resident memory is real memory: main memory that has been allocated and page mapped. But we don’t know how much of that is in heavy use, which is what WSS tells us. In this post I’ll introduce some new things I’ve developed for WSS estimation: two Linux tools, and WSS profile charts. The tools use either the referenced or the idle page flags to measure a page-based WSS, and were developed out of necessity for another performance problem.
    (via Amy Tobey)

    (tags: via:amytobey memory linux rss wss proc ps processes metrics working-set-size ram)

Links for 2018-01-15

  • The likely user interface which led to Hawaii’s false-alarm incoming-ballistic-missile alert on Saturday 2018-01-13

    @supersat on Twitter: “In case you’re curious what Hawaii’s EAS/WEA interface looks like, I believe it’s similar to this. Hypothesis: they test their EAS authorization codes at the beginning of each shift and selected the wrong option.” This is absolutely classic enterprisey, government-standard web UX — a dropdown template selection and an easily-misclicked pair of tickboxes to choose test or live mode.

    (tags: testing ux user-interfaces fail eas hawaii false-alarms alerts nuclear early-warning human-error)

  • The Death of Microservice Madness in 2018

    Quite a good set of potential gotchas, which I’ve run into myself, including: ‘Real world systems often have poorly defined boundaries’ ‘The complexities of state are often ignored’ ‘The complexitities of communication are often ignored’ ‘Versioning can be hard’ ‘Microservices can be monoliths in disguise’

    (tags: architecture devops microservices services soa coding monoliths state systems)

  • Do algorithms reveal sexual orientation or just expose our stereotypes?

    ‘A study claiming that artificial intelligence can infer sexual orientation from facial images caused a media uproar in the Fall of 2017. […] Michal Kosinski, who co-authored the study with fellow researcher Yilun Wang, initially expressed surprise, calling the critiques “knee-jerk” reactions. However, he then proceeded to make even bolder claims: that such AI algorithms will soon be able to measure the intelligence, political orientation, and criminal inclinations of people from their facial images alone.’ ‘In [this paper], we have shown how the obvious differences between lesbian or gay and straight faces in selfies relate to grooming, presentation, and lifestyle? — ?that is, differences in culture, not in facial structure. […] We’ve demonstrated that just a handful of yes/no questions about these variables can do nearly as good a job at guessing orientation as supposedly sophisticated facial recognition AI. Therefore?—?at least at this point?—?it’s hard to credit the notion that this AI is in some way superhuman at “outing” us based on subtle but unalterable details of our facial structure.’

    (tags: culture facial-recognition ai papers facial-structure sexual-orientation lgbt computer-vision)

  • Shanzhai ?? China & its Contents

    As he drinks Sino-coffee for around RMB 10, Comrade X might well be wearing the latest ‘ZARE’ couture while watching the TV news streaming on his HiPhone.[2] Back in Guangdong, his girlfriend — a sales consultant at a small stall in one of Shenzhen’s many wholesale electronics markets — sports a ‘high-end replica’ ?? Louis Vuitton bag and makes a living selling ‘domestically produced’ ?? and ‘smuggled’ ?? smartphones. The imitation products that festoon the couple’s lives are part of ‘shanzhai ?? China’. Shanzhai, the word means roughly ‘mass-produced imitation goods’, has created a Chinese landscape that is littered with products derided by the media, Chinese and international, as ‘copycat’, ‘guerrilla counterfeits’ and ‘knockoffs’, all the work of thieves.[3] Those who feel that their intellectual property and copyright has been infringed by shanzhai producers describe the products as ‘rubbish’, ‘piracy in disguise’ and ‘hooligan’.[4] Regardless of such righteous outrage, shanzhai — the producers, the products and the mentality — continues to flourish as an essential, quasi-legitimate shadow dimension of the Chinese economy. And, in practical terms, shanzhai products give disenfranchised ‘non-consumers’ of the orthodox economy — that is, people who would like to own but can’t afford the ‘original’ products — cut-price access to high-end technologies, as well as offering aspirational shoppers consumer satisfaction.

    (tags: shanzai china fakes consumerism hiphone smartphones copycat knockoffs imitation consumption)

  • Don Norman on “Human Error”, RISKS Digest Volume 23 Issue 07 2003

    It is far too easy to blame people when systems fail. The result is that over 75% of all accidents are blamed on human error. Wake up people! When the percentage is that high, it is a signal that something else is at fault — namely, the systems are poorly designed from a human point of view. As I have said many times before (even within these RISKS mailings), if a valve failed 75% of the time, would you get angry with the valve and simply continual to replace it? No, you might reconsider the design specs. You would try to figure out why the valve failed and solve the root cause of the problem. Maybe it is underspecified, maybe there shouldn’t be a valve there, maybe some change needs to be made in the systems that feed into the valve. Whatever the cause, you would find it and fix it. The same philosophy must apply to people.

    (tags: don-norman ux ui human-interface human-error errors risks comp.risks failures)

Links for 2018-01-14

Links for 2018-01-12

  • google/highwayhash: Fast strong hash functions: SipHash/HighwayHash

    HighwayHash: ‘We have devised a new way of mixing inputs with AVX2 multiply and permute instructions. The multiplications are 32×32 -> 64 bits and therefore infeasible to reverse. Permuting equalizes the distribution of the resulting bytes. The internal state occupies four 256-bit AVX2 registers. Due to limitations of the instruction set, the registers are partitioned into two 512-bit halves that remain independent until the reduce phase. The algorithm outputs 64 bit digests or up to 256 bits at no extra cost. In addition to high throughput, the algorithm is designed for low finalization cost. The result is more than twice as fast as SipTreeHash. We also provide an SSE4.1 version (80% as fast for large inputs and 95% as fast for short inputs), an implementation for VSX on POWER and a portable version (10% as fast). A third-party ARM implementation is referenced below. Statistical analyses and preliminary cryptanalysis are given in https://arxiv.org/abs/1612.06257.’ (via Tony Finch)

    (tags: siphash highwayhash via:fanf hashing hashes algorithms mac google hash)

  • Brain Cells Share Information With Virus-Like Capsules – The Atlantic

    …a gene called Arc which is active in neurons, and plays a vital role in the brain. A mouse that’s born without Arc can’t learn or form new long-term memories. If it finds some cheese in a maze, it will have completely forgotten the right route the next day. “They can’t seem to respond or adapt to changes in their environment,” says Shepherd, who works at the University of Utah, and has been studying Arc for years. “Arc is really key to transducing the information from those experiences into changes in the brain.” Despite its importance, Arc has been a very difficult gene to study. Scientists often work out what unusual genes do by comparing them to familiar ones with similar features—but Arc is one-of-a-kind. Other mammals have their own versions of Arc, as do birds, reptiles, and amphibians. But in each animal, Arc seems utterly unique—there’s no other gene quite like it. And Shepherd learned why when his team isolated the proteins that are made by Arc, and looked at them under a powerful microscope. He saw that these Arc proteins assemble into hollow, spherical shells that look uncannily like viruses. “When we looked at them, we thought: What are these things?” says Shepherd. They reminded him of textbook pictures of HIV, and when he showed the images to HIV experts, they confirmed his suspicions. That, to put it bluntly, was a huge surprise. “Here was a brain gene that makes something that looks like a virus,” Shepherd says. That’s not a coincidence. The team showed that Arc descends from an ancient group of genes called gypsy retrotransposons, which exist in the genomes of various animals, but can behave like their own independent entities.* They can make new copies of themselves, and paste those duplicates elsewhere in their host genomes. At some point, some of these genes gained the ability to enclose themselves in a shell of proteins and leave their host cells entirely. That was the origin of retroviruses—the virus family that includes HIV.

    (tags: brain evolution retroviruses viruses genes arc gag proteins memory biology)

Links for 2018-01-11

  • [1801.02780] Rogue Signs: Deceiving Traffic Sign Recognition with Malicious Ads and Logos

    Well, so much for that idea.

    We propose a new real-world attack against the computer vision based systems of autonomous vehicles (AVs). Our novel Sign Embedding attack exploits the concept of adversarial examples to modify innocuous signs and advertisements in the environment such that they are classified as the adversary’s desired traffic sign with high confidence. Our attack greatly expands the scope of the threat posed to AVs since adversaries are no longer restricted to just modifying existing traffic signs as in previous work. Our attack pipeline generates adversarial samples which are robust to the environmental conditions and noisy image transformations present in the physical world. We ensure this by including a variety of possible image transformations in the optimization problem used to generate adversarial samples. We verify the robustness of the adversarial samples by printing them out and carrying out drive-by tests simulating the conditions under which image capture would occur in a real-world scenario. We experimented with physical attack samples for different distances, lighting conditions, and camera angles. In addition, extensive evaluations were carried out in the virtual setting for a variety of image transformations. The adversarial samples generated using our method have adversarial success rates in excess of 95% in the physical as well as virtual settings.

    (tags: signs road-safety roads traffic self-driving-cars cars avs security machine-learning computer-vision ai)

  • The Stress of Remote Working – Martin De Wulf – Medium

    There is a lot of good to say about remote working, and I see a lot of rabid defence of the practice. That said, I have been working remotely for a little more than 5 years now, and I now must acknowledge that it does not come without stress. This might come as a surprise for some, but in the end, I think that remote working has taken some toll on me over the last two years, especially when I went almost fully remote for a year.
    I have to say, I agree with this 100% — I spent a few years remote working full time, and by the end of it I was absolutely delighted to return to a mainly office-based job.

    (tags: business work life coding teleworking remote-work stress anxiety mental-health)

  • Best way designing a GDPR compliant datalake using AWS services : aws

    interesting thread at Reddit

    (tags: gdpr reddit aws tips design services ops)

Links for 2018-01-10

Links for 2018-01-09

Links for 2018-01-04

Links for 2018-01-03

Links for 2018-01-01

  • Steven Bellovin on Bitcoin

    When you engineer a system for deployment you build it to meet certain real-world goals. You may find that there are tradeoffs, and that you can’t achieve all of your goals, but that’s normal; as I’ve remarked, “engineering is the art of picking the right trade-off in an overconstrained environment”. For any computer-based financial system, one crucial parameter is the transaction rate. For a system like Bitcoin, another goal had to be avoiding concentrations of power. And of course, there’s transaction privacy. There are less obvious factors, too. These days, “mining” for Bitcoins requires a lot of computations, which translates directly into electrical power consumption. One estimate is that the Bitcoin network uses up more electricity than many countries. There’s also the question of governance: who makes decisions about how the network should operate? It’s not a question that naturally occurs to most scientists and engineers, but production systems need some path for change. In all of these, Bitcoin has failed. The failures weren’t inevitable; there are solutions to these problems in the acdemic literature. But Bitcoin was deployed by enthusiasts who in essence let experimental code escape from a lab to the world, without thinking about the engineering issues—and now they’re stuck with it. Perhaps another, better cryptocurrency can displace it, but it’s always much harder to displace something that exists than to fill a vacuum.

    (tags: steven-bellovin bitcoin tech software systems engineering deployment cryptocurrency cypherpunks)

Links for 2017-12-19

Links for 2017-12-18

Links for 2017-12-15

Links for 2017-12-14

Links for 2017-12-13

Links for 2017-12-12

  • The Case for Learned Index Structures

    ‘Indexes are models: a B-Tree-Index can be seen as a model to map a key to the position of a record within a sorted array, a Hash-Index as a model to map a key to a position of a record within an unsorted array, and a BitMap-Index as a model to indicate if a data record exists or not. In this exploratory research paper, we start from this premise and posit that all existing index structures can be replaced with other types of models, including deep-learning models, which we term learned indexes. The key idea is that a model can learn the sort order or structure of lookup keys and use this signal to effectively predict the position or existence of records. We theoretically analyze under which conditions learned indexes outperform traditional index structures and describe the main challenges in designing learned index structures. Our initial results show, that by using neural nets we are able to outperform cache-optimized B-Trees by up to 70% in speed while saving an order-of-magnitude in memory over several real-world data sets. More importantly though, we believe that the idea of replacing core components of a data management system through learned models has far reaching implications for future systems designs and that this work just provides a glimpse of what might be possible.’ Excellent follow-up thread from Henry Robinson: https://threadreaderapp.com/thread/940344992723120128 ‘The fact that the learned representation is more compact is very neat. But also it’s not really a surprise that, given the entire dataset, we can construct a more compact function than a B-tree which is *designed* to support efficient updates.’ […] ‘given that the model performs best when trained on the whole data set – I strongly doubt B-trees are the best we can do with the current state-of-the art.’

    (tags: data-structures ml google b-trees storage indexes deep-learning henry-robinson)

  • Internet protocols are changing

    per @mnot. HTTP/2; TLS 1.3; QUIC and UDP; and DOH (DNS over HTTP!)

    (tags: crypto encryption http https protocols http2 tls quic udp tcp dns tunnelling)