Skip to content

Justin's Linklog Posts

Links for 2014-01-30

Published January 30, 2014

Links for 2014-01-29

Published January 29, 2014

Links for 2014-01-28

Published January 28, 2014

Links for 2014-01-27

Published January 27, 2014

  • Extending graphite’s mileage

    Ad company InMobi are using graphite heavily (albeit not as heavily as $work are), ran into the usual scaling issues, and chose to fix it in code by switching from a filesystem full of whisper files to a LevelDB per carbon-cache:

    The carbon server is now able to run without breaking a sweat even when 500K metrics per minute is being pumped into it. This has been in production since late August 2013 in every datacenter that we operate from.
    Very nice. I hope this gets merged/supported.

    (tags: graphite scalability metrics leveldb storage inmobi whisper carbon open-source)

  • BBC News – Pair jailed over abusive tweets to feminist campaigner

    When a producer from BBC Two’s Newsnight programme tracked Nimmo down after he had sent the abuse, the former call centre worker told him: “The police will do nothing, it’s only Twitter.”

    (tags: bbc bullying social-media twitter society uk trolls trolling abuse feminism cyberbullying)

  • If You Used This Secure Webmail Site, the FBI Has Your Inbox

    TorMail was a Tor-based webmail system, and apparently its drives have been imaged and seized by the FBI. More info on the Freedom Hosting seizure:

    The connection, if any, between the FBI obtaining Freedom Hosting’s data and apparently launching the malware campaign through TorMail and the other sites isn’t spelled out in the new document. The bureau could have had the cooperation of the French hosting company that Marques leased his servers from. Or it might have set up its own Tor hidden services using the private keys obtained from the seizure, which would allow it to adopt the same .onion addresses used by the original sites. The French company also hasn’t been identified. But France’s largest hosting company, OVH, announced on July 29, in the middle of the FBI’s then-secret Freedom Hosting seizure, that it would no longer allow Tor software on its servers. A spokesman for the company says he can’t comment on specific cases, and declined to say whether Freedom Hosting was a customer. “Wherever the data center is located, we conduct our activities in conformity with applicable laws, and as a hosting company, we obey search warrants or disclosure orders,” OVH spokesman Benjamin Bongoat told WIRED. “This is all we can say as we usually don’t make any comments on hot topics.”

    (tags: fbi freedom-hosting hosting tor tormail seizures ovh colo servers)

  • Sky parental controls break many JQuery-using websites

    An 11 hour outage caused by a false positive in Sky’s anti-phishing filter; all sites using the code.jquery.com CDN for JQuery would have seen errors.

    Sky still appears to be blocking code.jquery.com and all files served via the site, and more worryingly is that if you try to report the incorrect category, once signing in on the Sky website you an error page. We suspect the site was blocked due to being linked to by a properly malicious website, i.e. code.jquery.com and some javascript files were being used on a dodgy website and every domain mentioned was subsequently added to a block list.
    (via Tony Finch)

    (tags: via:fanf sky filtering internet uk anti-phishing phish jquery javascript http web fps false-positives)

  • Coders performing code reviews of scientific projects: pilot study

    ‘PLOS and Mozilla conducted a month-long pilot study in which professional developers performed code reviews on software associated with papers published in PLOS Computational Biology. While the developers felt the reviews were limited by (a) lack of familiarity with the domain and (b) lack of two-way contact with authors, the scientists appreciated the reviews, and both sides were enthusiastic about repeating the experiment. ‘ Actually sounds like it was more successful than this summary implies.

    (tags: plos mozilla code-reviews coding science computational-biology biology studies)

  • Caught with our Pantis down

    The views expressed by [the Iona Institute] – especially in relation to gay people – are very much at odds with the liberal secular society that Ireland has become. Indeed, Rory O’Neill suggested that the only time he experiences homophobia is online or at the hands of Iona and Waters. When they’re done with that, they can ask why Iona is given so much room in the media. In any other country in the world, an organisation as litigious as Iona would never be asked to participate in anything.

    (tags: homophobia ireland john-waters iona-institute politics catholicism religion libel defamation rte the-irish-times)

  • Scotch Whiskey flavour wheels

    mine’s a Smoky/Spicy/Medicinal, thanks

    (tags: scotch whiskey whisky alcohol dataviz flavour)

Links for 2014-01-24

Published January 24, 2014

Links for 2014-01-22

Published January 22, 2014

  • Ukrainian government targeting protesters using threatening SMS messages

    The government’s opponents said three recent actions had been intended to incite the more radical protesters and sow doubt in the minds of moderates: the passing of laws last week circumscribing the right of public assembly, the blocking of a protest march past the Parliament building on Sunday, and the sending of cellphone messages on Tuesday to people standing in the vicinity of the fighting that said, “Dear subscriber, you are registered as a participant in a mass disturbance.” [….] The phrasing of the message, about participating in a “mass disturbance,” echoed language in a new law making it a crime to participate in a protest deemed violent. The law took effect on Tuesday. And protesters were concerned that the government seemed to be using cutting-edge technology from the advertising industry to pinpoint people for political profiling. Three cellphone companies in Ukraine — Kyivstar, MTS and Life — denied that they had provided the location data to the government or had sent the text messages, the newspaper Ukrainskaya Pravda reported. Kyivstar suggested that it was instead the work of a “pirate” cellphone tower set up in the area.

    (tags: targeting mobile-phones sms text-messaging via:tjmcintyre geotargeting protest ukraine privacy surveillance tech 1984)

  • UK porn filter blocks game update that contained ‘sex’ in URL

    Staggeringly inept. The UK national porn filter blocks based on a regexp match of the URL against /.*sex.*/i — the good old “Scunthorpe problem”. Better, it returns a 404 response. This is also a good demonstration of how web filtering has unintended side effects, breaking third-party software updates with its false positives.

    The update to online strategy game League of Legends was disrupted by the internet filter because the software attempted to access files that accidentally include the word “sex” in the middle of their file names. The block resulted in the update failing with “file not found” errors, which are usually created by missing files or broken updates on the part of the developers.

    (tags: uk porn filtering guardian regular-expressions false-positives scunthorpe http web league-of-legends sex)

  • Register article on Amazon’s attitude to open source

    This article is frequently on target; this secrecy (both around open source and publishing papers) was one of the reasons I left Amazon.

    Of the sources with whom we spoke, many indicated that Amazon’s lack of participation was a key reason for why people left the company – or never joined at all. This is why Amazon’s strategy of maintaining secrecy may derail the e-retailer’s future if it struggles to hire the best talent. […] “In many cases in the big companies and all the small startups, your Github profile is your resume,” explained another former Amazonian. “When I look at developers that’s what I’m looking for, [but] they go to Amazon and that resume stops … It absolutely affects the quality of their hires.” “You had no portfolio you could share with the world,” said another insider on life after working at Amazon. “The argument this was necessary to attract talent and to retain talent completely fell on deaf ears.”

    (tags: amazon recruitment secrecy open-source hiring work research conferences)

  • Chinese Internet Traffic Redirected to Small Wyoming House

    ‘That address — which is home to some 2,000 companies on paper — was the subject of a lengthy 2011 Reuters investigation that found that among the entities registered to the address were a shell company controlled by a jailed former Ukraine prime minister; the owner of a company charged with helping online poker operators evade an Internet gambling ban; and one entity that was banned from government contracts after selling counterfeit truck parts to the Pentagon.’

    (tags: china internet great-firewall dns wyoming attacks security not-the-onion)

  • James Friend | PCE.js – Classic Mac OS in the Browser

    This is a demo of PCE’s classic Macintosh emulation, running System 7.0.1 with MacPaint, MacDraw, and Kid Pix. If you want to try out more apps and games see this demo.
    Incredible. I remember using this version of MacPaint!

    (tags: javascript browser emulation mac macos macpaint macdraw claris kid-pix history desktop pce)

Links for 2014-01-21

Published January 21, 2014

  • likwid

    ‘Lightweight performance tools’.

    Likwid stands for ‘Like I knew what I am doing’. This project contributes easy to use command line tools for Linux to support programmers in developing high performance multi-threaded programs. It contains the following tools: likwid-topology: Show the thread and cache topology likwid-perfctr: Measure hardware performance counters on Intel and AMD processors likwid-features: Show and Toggle hardware prefetch control bits on Intel Core 2 processors likwid-pin: Pin your threaded application without touching your code (supports pthreads, Intel OpenMP and gcc OpenMP) likwid-bench: Benchmarking framework allowing rapid prototyping of threaded assembly kernels likwid-mpirun: Script enabling simple and flexible pinning of MPI and MPI/threaded hybrid applications likwid-perfscope: Frontend for likwid-perfctr timeline mode. Allows live plotting of performance metrics. likwid-powermeter: Tool for accessing RAPL counters and query Turbo mode steps on Intel processor. likwid-memsweeper: Tool to cleanup ccNUMA memory domains.
    No kernel patching required. (via kellabyte)

    (tags: via:kellabyte linux performance testing perf likwid threading multithreading multicore mpi numa)

  • Backblaze Blog » What Hard Drive Should I Buy?

    Because Backblaze has a history of openness, many readers expected more details in my previous posts. They asked what drive models work best and which last the longest. Given our experience with over 25,000 drives, they asked which ones are good enough that we would buy them again. In this post, I’ll answer those questions.

    (tags: backblaze backup hardware hdds storage disks ops via:fanf)

Links for 2014-01-20

Published January 20, 2014

Links for 2014-01-17

Published January 17, 2014

  • Transport Minister planning to make hi-vis jackets mandatory for cyclists

    The minister also spoke of a number of new transport initiatives, such  as mandatory use of high visibility jackets by cyclists.

    (tags: cycling safety law ireland leo-varadkar)

  • The Malware That Duped Target Has Been Found

    a Windows ‘RAM scraper’ trojan known as Trojan.POSRAM, which was used to attack the Windows-based point-of-sales systems which the POS terminals are connected to. part of an operation called Kaptoxa. ‘The code is based on a previous malicious tool known as BlackPOS that is believed to have been developed in 2013 in Russia, though the new variant was highly customized to prevent antivirus programs from detecting it’ … ‘The tool monitors memory address spaces used by specific programs, such as payment application programs like pos.exe and PosW32.exe that process the data embossed in the magnetic strip of credit and debit cards data. The tool grabs the data from memory.’ … ‘The siphoned data is stored on the system, and then every seven hours the malware checks the local time on the compromised system to see if it’s between the hours of 10 a.m. and 5 p.m. If so, it attempts to send the data over a temporary NetBIOS share to an internal host inside the compromised network so the attackers can then extract the data over an FTP … connection.’ http://www.pcworld.com/article/2088920/target-credit-card-data-was-sent-to-server-in-russia.html says the data was then transmitted to another US-based server, and from there relayed to Russia, and notes: ‘At the time of its discovery, Trojan.POSRAM “had a zero percent antivirus detection rate, which means that fully updated antivirus engines on fully patched computers could not identify the software as malicious,” iSight said.’ Massive AV fail.

    (tags: kaptoxa trojans ram-scrapers trojan.posram posram point-of-sale security hacks target credit-cards pin ftp netbios smb)

  • Full iSight report on the Kaptoxa attack on Target

    ‘POS malware is becoming increasingly available to cyber criminals’ … ‘there is growing demand for [this kind of malware]’. Watch your credit cards…

    (tags: debit-cards credit-cards security card-present attacks kaptoxa ram-scrapers trojans point-of-sale pos malware target)

  • The Target hack and PCI-DSS

    Both Heartland Payment Systems and Hannaford Bros. were in fact certified PCI-compliant while the hackers were in their system. In August 2006, Wal-Mart was also certified PCI-compliant while unknown attackers were lurking on its network. […] “This PCI standard just ain’t working,” says Litan, the Gartner analyst. “I wouldn’t say it’s completely pointless. Because you can’t say security is a bad thing. But they’re trying to patch a really weak [and] insecure payment system [with it].”
    Basically, RAM scrapers have been in use in live attacks, sniffing credentials in the clear, since 2007. Ouch.

    (tags: ram-scrapers trojans pins pci-dss compliance security gartner walmart target)

  • ISPAI responds to TD Patrick O’Donovan’s bizarre comments regarding “open source browsers”

    ISPAI is rather dismayed and somewhat confused by the recent press release issued by Deputy Patrick O’Donovan (FG). He appears to be asking the Oireachtas Communications Committee (of which he is a member) to investigate: “the matter of tougher controls on the use of open source internet browsers and payment systems”  which he claims “allow users to remain anonymous for illegal trade of drugs weapons and pornography.” Deputy O’Donovan would do well to ask the advice of industry experts on these matters given that legislating to curtail the use of such legitimate software or services, which may be misused by some, is neither practical nor logical. Whether or not a browser is open source bears no relevance to its ability to be the subject of anonymous use. Indeed, Deputy O’Donovan must surely be confusing and conflating different technical concepts? In tracing illegal activities, Law Enforcement Agencies and co-operating parties will use IP addresses – users’ choice of browser has little relevance to an investigation of criminal activity. Equally, it may be that the Deputy is uncomfortable with the concept of electronic payment systems but these underpin the digital economy which is bringing enormous benefit to Ireland. Yes, these may be misused by criminals but so are cash and traditional banking services. Restricting the growth of innovative financial services is not the solution to tackling cyber criminals who might be operating what he describes as “online supermarkets for illegal goods.” Tackling international cybercrime requires more specialist Law Enforcement resources at national level and improved international police cooperation supported by revision of EU legislation relating to obtaining server log evidence existing in other jurisdictions.

    (tags: ispai open-source patrick-o-donovan fine-gael press-releases tor darknet crime)

Links for 2014-01-16

Published January 16, 2014

Don’t use Timers with exponentially-decaying reservoirs in Graphite

Published January 16, 2014

A common error when using the Metrics library is to record Timer metrics on things like API calls, using the default settings, then to publish those to a time-series store like Graphite. Here’s why this is a problem.

By default, a Timer uses an Exponentially Decaying Reservoir. The docs say:

‘A histogram with an exponentially decaying reservoir produces quantiles which are representative of (roughly) the last five minutes of data. It does so by using a forward-decaying priority reservoir with an exponential weighting towards newer data. Unlike the uniform reservoir, an exponentially decaying reservoir represents recent data, allowing you to know very quickly if the distribution of the data has changed.’

This is more-or-less correct — but the key phrase is ‘roughly’. In reality, if the frequency of updates to such a timer drops off, it could take a lot longer, and if you stop updating a timer which uses this reservoir type, it’ll never decay at all. The GraphiteReporter will dutifully capture the percentiles, min, max, etc. from that timer’s reservoir every minute thereafter, and record those to Graphite using the current timestamp — even though the data it was derived from is becoming more and more ancient.

Here’s a demo. Note the long stretch of 800ms 99th-percentile latencies on the green line in the middle of this chart:

However, the blue line displays the number of events. As you can see, there were no calls to this API for that 8-hour period — this one was a test system, and the user population was safely at home, in bed. So while Graphite is claiming that there’s an 800ms latency at 7am, in reality the 800ms-latency event occurred 8 hours previously.

I observed the same thing in our production systems for various APIs which suffered variable invocation rates; if rates dropped off during normal operation, the high-percentile latencies hung around for far longer than they should have. This is quite misleading when you’re looking at a graph for 10pm and seeing a high 99th-percentile latency, when the actual high-latency event occurred hours earlier. On several occasions, this caused lots of user confusion and FUD with our production monitoring, so we needed to fix it.

Here are some potential fixes.

  • Modify ExponentiallyDecayingReservoir to also call rescaleIfNeeded() inside getSnapshot() — but based on this discussion, it appears the current behaviour is intended (at least for the mean measurement), so that may not be acceptable. Another risk of this is that it leaves us in a position where the percentiles displayed for time T may actually have occurred several minutes prior to that, which is still misleading (albeit less so).

  • Switch to sliding time window reservoirs, but those are unbounded in size — so a timer on an unexpectedly-popular API could create GC pressure and out-of-memory scenarios. It’s also the slowest reservoir type, according to the docs. That made it too risky for us to adopt in our production code as a general-purpose Timer implementation.

  • Update, Dec 2017: as of version 3.2.3 of Dropwizard Metrics, there is a new SlidingTimeWindowArrayReservoir reservoir implementation, which is a drop-in replacement for SlidingTimeWindowReservoir, with much more acceptable memory footprint and GC impact. It costs roughly 128 bits per stored measurement, and is therefore judged to be ‘comparable with ExponentiallyDecayingReservoir in terms of GC overhead and performance’. (thanks to Bogdan Storozhuk for the tip)

  • What we eventually did in our code was to use this Reporter class instead of GraphiteReporter; it clears all Timer metrics’ reservoirs after each write to Graphite. This is dumb and dirty, reaching across logical class boundaries, but at the same time it’s simple and comprehensible behaviour: with this, we can guarantee that the percentile/min/max data recorded at timestamp T is measuring events in that timestamp’s 1-minute window — not any time before that. This is exactly what you want to see in a time-series graph like those in Graphite, so is a very valuable feature for our metrics, and one that others have noted to be important in comparable scenarios elsewhere.

Here’s an example of what a graph like the above should look like (captured from our current staging stack):

Note that when there are no invocations, the reported 99th-percentile latency is 0, and each measurement doesn’t stick around after its 1-minute slot.

Another potential bug fix for a related issue, would be to add support to Metrics so that it can use Gil Tene’s LatencyUtils package, and its HdrHistogram class, as a reservoir. (Update: however, I don’t think this would address the “old data leaking into newer datapoints” problem as fully.) This would address some other bugs in the Exponentially Decaying Reservoir, as Gil describes:

‘In your example of a system logging 10K operations/sec with the histogram being sampled every second, you’ll be missing 9 out of each 10 actual outliers. You can have an outlier every second and think you have one roughly every 10. You can have a huge business affecting outlier happening every hour, and think that they are only occurring once a day.’

Eek.

Links for 2014-01-15

Published January 15, 2014

Links for 2014-01-14

Published January 14, 2014

Links for 2014-01-13

Published January 13, 2014

Links for 2014-01-11

Published January 11, 2014

  • Growing up unvaccinated: A healthy lifestyle couldn’t prevent many childhood illnesses.

    I understand, to a point, where the anti-vaccine parents are coming from. Back in the ’90s, when I was a concerned, 19-year-old mother, frightened by the world I was bringing my child into, I was studying homeopathy, herbalism, and aromatherapy; I believed in angels, witchcraft, clairvoyants, crop circles, aliens at Nazca, giant ginger mariners spreading their knowledge to the Aztecs, the Incas, and the Egyptians, and that I was somehow personally blessed by the Holy Spirit with healing abilities. I was having my aura read at a hefty price and filtering the fluoride out of my water. I was choosing to have past life regressions instead of taking antidepressants. I was taking my daily advice from tarot cards. I grew all my own veg and made my own herbal remedies. I was so freaking crunchy that I literally crumbled. It was only when I took control of those paranoid thoughts and fears about the world around me and became an objective critical thinker that I got well. It was when I stopped taking sugar pills for everything and started seeing medical professionals that I began to thrive physically and mentally.

    (tags: health medicine science vaccination disease slate)

Links for 2014-01-09

Published January 9, 2014

Links for 2014-01-07

Published January 7, 2014

Links for 2014-01-06

Published January 6, 2014

Links for 2014-01-02

Published January 2, 2014

  • Dogs like to excrete in alignment with the Earth’s magnetic field

    Dogs preferred to excrete with the body being aligned along the North-south axis under calm magnetic field conditions.

    (tags: dogs poo excrement shit magnetic-field earth zoology papers)

  • Paul Graham and the Manic Pixie Dream Hacker

    Under Graham’s influence, Mark [Zuckerberg], like many in Silicon Valley, subscribes to the Manic Pixie Dream Hacker ideal, making self-started teenage hackers Facebook’s most desired recruiting targets, not even so much for their coding ability as their ability to serve as the faces of hacking culture. “Culture fit”, in this sense, is one’s ability to conform to the Valley’s boyish hacker fantasy, which is easier, obviously, the closer you are to a teenage boy. Like the Manic Pixie Dream Girl’s role of existing to serve the male film protagonist’s personal growth, the Manic Pixie Dream Hacker’s job is to embody the dream hacker role while growing the VC’s portfolio. This is why the dream hacker never ages, never visibly develops interests beyond hardware and code, and doesn’t question why nearly all the other people receiving funding look like him. Like the actress playing the pixie dream girl, the pixie dream boy isn’t being paid to question the role for which he has been cast. In this way, for all his supposed “disruptiveness”, the hacker pixie actually does exactly what he is told: to embody, while he can, the ideal hacker, until he is no longer young, mono-focused, and boyish-seeming enough to qualify for the role (at that point, vested equity may allow him to retire). And like in Hollywood, VCs will have already recruited newer, younger ones to play him.

    (tags: hackers manic-pixie-dream-girl culture-fit silicon-valley mark-zuckerberg paul-graham y-combinator vc work investment technology recruitment facebook ageism equality sexism)

  • The How and Why of Flapjack

    Flapjack aims to be a flexible notification system that handles: Alert routing (determining who should receive alerts based on interest, time of day, scheduled maintenance, etc); Alert summarisation (with per-user, per media summary thresholds); Your standard operational tasks (setting scheduled maintenance, acknowledgements, etc). Flapjack sits downstream of your check execution engine (like Nagios, Sensu, Icinga, or cron), processing events to determine if a problem has been detected, who should know about the problem, and how they should be told.

    (tags: flapjack notification alerts ops nagios paging sensu)

Links for 2013-12-27

Published December 27, 2013

  • Dublin Cycle Planner needs a health warning – Irish Cycle

    An extensive catalogue of shitty routing. Poor…

    It’s expected that any new mapping and routing systems will have errors which will need to be ironed out but the level of issues with the NTA Cycle Planner is far beyond what you’d expect in a light and quiet beta launch. It’s beyond acceptable for a public PR launch directing people to a route planner with no clear warnings. It looks like a rush job which allows junior minister Alan Kelly to get his name in another press release before the end of the year.

    (tags: cycling dublin commute mapping nta ireland maps)

  • Reflected hidden faces in photographs revealed in pupil

    The pupil of the eye in a photograph of a face can be mined for hidden information, such as reflected faces of the photographer and bystanders, according to research led by Dr. Rob Jenkins, of the Department of Psychology at the University of York and published in PLOS ONE (open access).
    (via Waxy)

    (tags: via:waxy future zoom-and-enhance privacy photography eyes photos)

Links for 2013-12-23

Published December 23, 2013

  • Jesse Willms, the Dark Lord of the Internet – Taylor Clark – The Atlantic

    “It was an out-and-out hijacking,” LeFevre told me. “They counterfeited our product, they pirated our Web site, and they basically directed all of their customer service to us.” At the peak of Willms’s sales, LeFevre says, dazzlesmile was receiving 1,000 calls a day from customers trying to cancel orders for a product it didn’t even sell. When irate consumers made the name dazzlesmile synonymous with online scamming, LeFevre’s sales effectively dropped to zero. Dazzlesmile sued Willms in November 2009; he later paid a settlement.

    (tags: scams hijacking ads affiliate one-wierd-trick health dieting crime)

Links for 2013-12-21

Published December 21, 2013

Links for 2013-12-19

Published December 19, 2013

Links for 2013-12-16

Published December 16, 2013

Links for 2013-12-13

Published December 13, 2013

  • Karlin Lillington on DRI’s looming victory in the European Court of Justice

    If the full European Court of Justice (ECJ) accepts the opinion of its advocate general in a final ruling due early next year – and it almost always does – it will prove a huge vindication of Ireland’s small privacy advocacy group, Digital Rights Ireland (DRI). Its case against Irish retention laws, which began in 2006, forms the basis of this broader David v Goliath challenge and initial opinion. The advocate general’s advice largely upholds the key concerns put forward by DRI against Ireland’s laws. Withholding so much data about every citizen, including children, in case someone commits a future crime, is too intrusive into private life, and could allow authorities to create a “faithful and exhaustive map of a large portion of a person’s [private] conduct”. Retained data is so comprehensive that they could easily reveal private identities, which are supposed to remain anonymous. And the data, entrusted to third parties, is at too much risk of fraudulent or malicious use. Cruz Villalón argues that there must be far greater oversight to the retention process, and controls on access to data, and that citizens should have the right to be notified after the fact if their data has been scrutinised. The Irish Government had repeatedly waved off such concerns from Digital Rights Ireland in the past.

    (tags: dri rights ireland internet surveillance data-retention privacy eu ecj law)

Links for 2013-12-11

Published December 11, 2013

Links for 2013-12-10

Published December 10, 2013

Links for 2013-12-09

Published December 9, 2013

  • Cyanite

    a metric storage daemon, exposing both a carbon listener and a simple web service. Its aim is to become a simple, scalable and drop-in replacement for graphite’s backend.
    Pretty alpha for now, but definitely worth keeping an eye on to potentially replace our burgeoning Carbon fleet…

    (tags: graphite carbon cassandra storage metrics ops graphs service-metrics)

  • Twitter tech talk video: “Profiling Java In Production”

    In this talk Kaushik Srenevasan describes a new, low overhead, full-stack tool (based on the Linux perf profiler and infrastructure built into the Hotspot JVM) we’ve built at Twitter to solve the problem of dynamically profiling and tracing the behavior of applications (including managed runtimes) in production.
    Looks very interesting. Haven’t watched it yet though

    (tags: twitter tech-talks video presentations java jvm profiling testing monitoring service-metrics performance production hotspot perf)

  • Spy agencies in covert push to infiltrate virtual world of online gaming

    [MMOGs], the [NSA] analyst wrote, “are an opportunity!”. According to the briefing notes, so many different US intelligence agents were conducting operations inside games that a “deconfliction” group was required to ensure they weren’t spying on, or interfering with, each other.

    (tags: spies spying games mmog online surveillance absurd east-germany funny warcraft)

  • Ryan Lizza: Why Won’t Obama Rein in the N.S.A.? : The New Yorker

    Fantastic wrap-up of the story so far on the pervasive global surveillance story.

    The history of the intelligence community, though, reveals a willingness to violate the spirit and the letter of the law, even with oversight. What’s more, the benefits of the domestic-surveillance programs remain unclear. Wyden contends that the N.S.A. could find other ways to get the information it says it needs. Even Olsen, when pressed, suggested that the N.S.A. could make do without the bulk-collection program. “In some cases, it’s a bit of an insurance policy,” he told me. “It’s a way to do what we otherwise could do, but do it a little bit more quickly.” In recent years, Americans have become accustomed to the idea of advertisers gathering wide swaths of information about their private transactions. The N.S.A.’s collecting of data looks a lot like what Facebook does, but it is fundamentally different. It inverts the crucial legal principle of probable cause: the government may not seize or inspect private property or information without evidence of a crime. The N.S.A. contends that it needs haystacks in order to find the terrorist needle. Its definition of a haystack is expanding; there are indications that, under the auspices of the “business records” provision of the Patriot Act, the intelligence community is now trying to assemble databases of financial transactions and cell-phone location information. Feinstein maintains that data collection is not surveillance. But it is no longer clear if there is a distinction.

    (tags: nsa gchq surveillance spying privacy dianne-feinstein new-yorker journalism long-reads us-politics probable-cause)

Links for 2013-12-07

Published December 7, 2013

  • Same Old Stories From Sean Sherlock

    Sherlock’s record is spotty at best when it comes to engagement. Setting aside the 80,680 people who were ignored by the minister, he was hostile and counter productive to debate from the beginning, going so far as to threaten to pull out of a public debate because a campaigner against the [‘Irish SOPA’] SI would be in attendance. His habit of blocking people online who publicly ask him tough yet legitimate questions has earned him the nickname “Sherblock”.

    (tags: sean-sherlock sherblock labour ireland politics blocking filtering internet freedom copyright emi music law piracy debate twitter)

  • Smart Metering in the UK is FCUKED

    Most utilities don’t want smart metering.  In fact they seem to have used the wrong dictionary.  It is difficult to find anything smart about the UK deployment, until you realise that the utilities use smart in the sense of “it hurts”.  They consider they have a perfectly adequate business model which has no need for new technology.  In many Government meetings, their reluctant support seems to be a veneer for the hope that it will all end in disaster, letting them go back to the world they know, of inflated bills and demands for money with menaces. […] Even when smart meters are deployed, there is no evidence that any utility will use the resulting data to transform their business, rather than persecute the consumer.  At a recent US conference a senior executive for a US utility which had deployed smart meters, stated that their main benefit was “to give them more evidence to blame the customer”.  That’s a good description of the attitude displayed by our utilities.

    (tags: smart-metering energy utilities uk services metering consumer)

  • Kelly “kellabyte” Sommers on Redis’ “relaxed CP” approach to the CAP theorem

    Similar to ACID properties, if you partially provide properties it means the user has to _still_ consider in their application that the property doesn’t exist, because sometimes it doesn’t. In you’re fsync example, if fsync is relaxed and there are no replicas, you cannot consider the database durable, just like you can’t consider Redis a CP system. It can’t be counted on for guarantees to be delivered. This is why I say these systems are hard for users to reason about. Systems that partially offer guarantees require in-depth knowledge of the nuances to properly use the tool. Systems that explicitly make the trade-offs in the designs are easier to reason about because it is more obvious and _predictable_.

    (tags: kellabyte redis cp ap cap-theorem consistency outages reliability ops database storage distcomp)

  • Building a Balanced Universe – EVE Community

    Good blog post about EVE’s algorithm to load-balance a 3D map of star systems

    (tags: eve eve-online algorithms 3d space load-balancing sharding games)

  • Virtual Clock – Testing Patterns Encyclopedia

    a nice pattern for unit tests which need deterministic time behaviour. Trying to think up a really nice API for this….

    (tags: testing unit-tests time virtual-clock real-time coding)

  • We’re sending out the wrong signals in bid to lure the big data bucks – Independent.ie

    Simon McGarr on Ireland’s looming data-protection train-crash.

    Last week, during the debate of his proposals to increase fees for making a Freedom of Information request, Brendan Howlin was asked how one of his amendments would affect citizens looking for data from the State’s electronic databases. His reply was to cheerfully admit he didn’t even understand the question. “I have no idea what an SQL code is. Does anyone know what an SQL code is?” Unlike the minister, it probably isn’t your job to know that SQL is the computer language that underpins the data industry. The amendment he had originally proposed would have effectively allowed civil servants to pretend that their computer files were made of paper when deciding whether a request was reasonable. His answer showed how the Government could have proposed such an absurd idea in the first place. Like it or not – fair or not – these are not the signals a country that wanted to build a long-term data industry would choose to send out. They are the sort of signals that Ireland used to send out about Financial Regulation. I think it’s agreed, that approach didn’t work out so well.

    (tags: foi ireland brendan-howlin technology illiteracy sql civil-service government data-protection privacy regulation dpa)