Skip to content

Justin's Linklog Posts

Links for 2012-05-23

Published May 23, 2012

  • Copyright Review Committee Submission : ‘This site is intended to give the public a chance to comment on, and hopefully [collaboratively] improve, the text of a proposed submission to the [Irish] Copyright Review Commission.’ (ie. CRC2012, deadline 31 May.)
    (tags: crc2012 copyright ireland law collaboration)

  • Dropwizard : ‘a Java framework for developing ops-friendly, high-performance, RESTful web services. Developed by Yammer to power their JVM-based backend services, Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, lightweight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality HTTP+JSON web service in the shortest time possible.’ From Coda Hale/Yammer; includes Guava, Jetty, Jersey, Jackson, Metrics, slf4j. Pretty good baseline to start any new Java service with….
    (tags: framework http java rest web jersey guava jackson jetty json web-services yammer)

Links for 2012-05-22

Published May 22, 2012

  • satellite rescue abandoned due to patents : ‘SES and Lockheed Martin explored ways to attempt to bring the functioning [AMC-14] satellite into its correct orbital position, and subsequently began attempting to move the satellite into geosynchronous orbit by means of a lunar flyby (as done a decade earlier with HGS-1). In April 2008, it was announced that this had been abandoned after it was discovered that Boeing held a patent on the trajectory that would be required. At the time, a lawsuit was ongoing between SES and Boeing, and Boeing refused to allow the trajectory to be used unless SES dropped its case.’ In. credible. http://www.spacedaily.com/reports/Boeing_Patent_Shuts_Down_AMC_14_Lunar_Flyby_Salvage_Attempt_999.html notes ‘Industry sources have told SpaceDaily that the patent is regarded as legal “trite”, as basic physics has been rebranded as a “process”, and that the patent wouldn’t stand up to any significant level of court scrutiny and was only registered at the time as “the patent office was incompetent when it came to space matters”‘, but still — who’d want to go up in court against Boeing?
    (tags: boeing space patenting via:hn funny sad lockheed-martin ses amc-14 business-process patents)

Links for 2012-05-20

Published May 20, 2012

  • Nikola Tesla Wasn’t God And Thomas Edison Wasn’t The Devil : Correcting some egregious misconceptions about an Oatmeal comic regarding Tesla and Edison — explaining some realities about invention, scientific progress, and the history of electricity. “I’d contend that nearly every invention in the engineering or sciences is an improvement on what has come before – such as Tesla’s improvements to alternating current. That’s what innovation is. It’s a social process that occurs in a social context. As Robert Heinlein once said, “When railroading time comes you can railroad — but not before.” In other words, inventions are made in the context of scientific and engineering understanding. Individuals move things forward – some faster than others – but in the end, the most intelligent person in the world can’t invent the light bulb if the foundation isn’t there.”
    (tags: nikola-tesla history electricity innovation invention progress science thomas-edison the-oatmeal)

  • Jamming Tripoli: Inside Moammar Gadhafi’s Secret Surveillance Network : The very scary future of state control, censorship, and totalitarianism in the age of the internet. A presentation from Amesys, a subsidiary of Bull S.A. “explained the significance of Eagle to a government seeking to control activities inside its borders. Warning of an “increasing need of high-level intelligence in the constant struggle against criminals and terrorism,” the document touted Eagle’s ability to capture bulk Internet traffic passing through conventional, satellite, and mobile phone networks, and then to store that data in a filterable and searchable database. This database, in turn, could be integrated with other sources of intelligence, such as phone recordings, allowing security personnel to pick through audio and data from a given person all at once, in real time or by historical time stamp. In other words, instead of choosing targets and monitoring them, officials could simply sweep up everything, sort it by time and target, and then browse through it later at their leisure. The title of the presentation — ”From Lawful to Massive Interception” — gestured at the vast difference between so-called lawful intercept (traditional law enforcement surveillance based on warrants for specific phone numbers or IP addresses) and what Amesys was offering.”
    (tags: massive-interception future state-control censorship privacy internet email totalitarianism libya amesys bull-sa gadhafi surveillance)

Links for 2012-05-16

Published May 16, 2012

Links for 2012-05-15

Published May 15, 2012

  • Digital Rights Forum – Online Privacy : ‘The Digital Rights Forum is a public debate on the important issues surrounding digital rights, with each event designed around the general over-arching topic of digital rights, puls a more narrowly focused subject. On Friday, the 18th of May, the forum will tackle the issue of Online Privacy. With our lives ever more integrated with the web and social media, staying safe online is becoming an increasing concern to everyone. From mobile apps to websites and email, protecting our personal information and online privacy has never been more complicated and more important. Faced with software vulnerabilities such as contacts being leaked onto the Internet by mobile application providers, the increasing push toward revealing more private and personal information on social networks, and attempts by some to protect their businesses through litigation or processes which require the disclosure of personal information, the modern digital landscape has made protecting one’s privacy more difficult than ever before. With this in mind, this Digital Rights Forum will discuss the current state of data protection and online privacy in the current context of social networks and mobile applications.’ Featuring Billy Hawkes (the DPC, no less!), and Devore from Boards.
    (tags: dpc digital-rights ireland politics online security privacy data-protection)

Links for 2012-05-13

Published May 13, 2012

  • An IDE is not enough : Very thought-provoking response to that ‘Light Table’ demo which went round the aggregators a couple of weeks back. ‘The fundamental reason IDEs have dead-ended is that they are constrained by the syntax and semantics of our programming languages. Our programming languages were all designed to be used with a text editor. It is therefore not surprising that our IDEs amount to tarted-up text editors. Likewise our programming languages were all designed with an imperative semantics that efficiently matches the hardware but defies static visualization. Indeed it would be a miracle if we could slap a new IDE on top of an old language and magically alter its syntactic and semantic assumptions. I don’t believe in miracles. Languages and IDEs have co-evolved and neither can change without the other also changing. That is why three years ago I put aside my IDE work to focus on language design. Getting rid of imperative semantics is one of the goals. Another is getting rid of source text files (as well as ASTs, which carry all the baggage of a textual encoding minus the readability). This has turned out to be really really hard. And lonely – no one wants to even talk about these crazy ideas. Nevertheless I firmly believe that so long as we are programming in decendants of assembly language we will continue to program in descendants of text editors.’ (via Chris Horn)
    (tags: via:cjhorn ide programming coding programming-languages semantics syntax source-code text)

  • Open Data Structures : A free-as-in-speech as well as -beer textbook of data structures, covering a great range, including some I hadn’t heard of before. Here’s the full list: ArrayStack, FastArrayStack, ArrayQueue, ArrayDeque, DualArrayDeque, RootishArrayStack, SLList, DLList, SEList, SkiplistSSet, SkiplistList, ChainedHashTable, LinearHashTable, BinaryTree, BinarySearchTree, Treap, ScapegoatTree, RedBlackTree, BinaryHeap, MeldableHeap, AdjacencyMatrix, AdjacencyLists, BinaryTrie, XFastTrie, and YFastTrie
    (tags: algorithms books data-structures computer-science coding tries skiplists arrays queues heap trees graphs hashtables)

Links for 2012-05-12

Published May 12, 2012

  • Chronon DVR for Java : “record entire execution of your Java app; play it back on any machine”. Other features: time-travelling debugger — step backwards, jump to any point in execution, designed for long running programs; post-execution logging — add log statements after the program has run, and see what it would have logged. Looks extremely nifty, but I wonder how big those recording files get…
    (tags: debugging via:peakscale eclipse chronon dvr java coding logging jvm)

In Dublin? Hear me talk about AWS network monitoring!

Published May 11, 2012

Reminder to Dublin-based readers — next week, Amazon (my employers) will be putting on Under the Hood at Amazon, billed as ‘A night of Beer, Pizza and Cloud Computing for Software Developers’. I’ll be speaking at it.

It’s partially a recruiting event, but even if you’re not looking for a new job, please come along. It’s also useful for us to talk about some details of what we’ve been doing in Dublin, since we’ve been operating to date with a pretty low profile, and in reality there’s some very interesting stuff going on here… particularly the product I’ll be talking about, naturally.

Also, there’ll be free beer and some Kindles to be won ;)

It’s next Thursday night, in our offices in Kilmainham. More info on this Facebook page.

Links for 2012-05-10

Published May 10, 2012

  • FF Chartwell : OpenType font to display charts/graphs using ligatures. ‘Designed by Travis Kochel, FF Chartwell is a typeface for creating simple graphs. Driven by the frustration of creating graphs within design applications and inspired by typefaces such as FF Beowolf and FF PicLig, Travis saw an opportunity to take advantage of OpenType technology to simplify the process. Using OpenType ligatures, strings of numbers are automatically transformed into charts. The data remains in a text box, allowing for easy updates and styling. It’s really easy to use; you just type a simple series of numbers like: ‘10+13+37+40’, turn on Stylistic Alternates or Stylistic Set 1 and a graph is automatically created.’ (via Simon)
    (tags: ligatures via:sboyle fonts hacks charts dataviz ui)

  • McGarr Solicitors’ sternly-worded letter to Newspaper Licencing Ireland Ltd : In response to a letter received by a charity, warning of dire penalties for ‘reproducing copyright content without permission’, since doing so ‘is theft’. It gets better, since in correspondence they were then informed that “a licence is required to link directly to an online article even without uploading any of the content directly onto your own website”. Looking forward to seeing how this one plays out…
    (tags: law ireland scams shakedown copyright nli licensing linking hyperlinks)

  • Goodbye, CouchDB : ‘From most model-using code, using [Percona] MySQL looks exactly the same as using CouchDB did. Except it’s faster, and the DB basically never fails.’
    (tags: couchdb mysql nosql databases storage percona via:peakscale)

Links for 2012-05-09

Published May 9, 2012

  • Diageo Screw BrewDog : Giant booze multinational screws tiny Scottish microbrewery. “Diageo (the main sponsor) approached us at the start of the meal and said under no circumstances could the award be given to BrewDog. They said if this happened they would pull their sponsorship from all future BII events and their representatives would not present any of the awards on the evening. We were as gobsmacked as you by Diageo’s behaviour. We made the wrong decision under extreme pressure. We were blackmailed and bullied by Diageo. We should have stuck to our guns and gave the award to BrewDog.”
    (tags: brewdog diageo bii awards beer brewing dirty-tricks)

Links for 2012-05-04

Published May 4, 2012

Links for 2012-05-03

Published May 3, 2012

Links for 2012-04-30

Published April 30, 2012

Links for 2012-04-24

Published April 24, 2012

Links for 2012-04-23

Published April 23, 2012

  • First Music Contact – Music3.0 : ‘We talk a lot about what the world of music and artists will look like five or ten years from now. But for changes to happen then, the conversations need to happen now. We believe that the next big thing in music is not going to ever appear on a stage. After the record industry (music 1.0) and the live music industry (music 2.0), it’s time to pay more attention to innovation (music 3.0) and what can come from constructively disrupting how the music industry operates. It’s time to open up the shop. It’s time for unvested interests to see if they can use existing data and ecosystems to make a better music business. For far too long, music has been a conservative sector which views the influence of outside forces with abject suspicion and rank horror. Chalk this down to some bad experiences over the last 15 years due to misunderstandings with and ignorance of the tech and telecoms worlds. Chalk this down to rampant music industry egos which lead folks to believe no-one else can sell music bar music players. Chalk it down to fear of disruption. So, it’s time for change. You can’t keep doing the same things in the same way and hope you won’t make the same mistakes again. It’s time to listen to and learn from smart people in other areas. It’s time for people who have innovative ideas or even just the stirrings of innovative ideas to take stock from people who operate in other areas and who deal with ideas, technology and the valuable currency of innovation every single working day. It’s time for some different talking which is going to lead to some very different make-and-do experiences.’ Looks excellent. (via Jim Carroll)
    (tags: music future technology internet disruption music-industry ireland via:jimcarroll)

Links for 2012-04-19

Published April 19, 2012

  • A Kiva success story : Pretty cool testimonial to Kiva’s effects on the ground. ‘Thanks to Mariano’s entrepreneurship and skills, and partially to the [microfinance] loans offered to him, as he said: now, his children are attending to school, something his generation couldn’t afford to, and he is able to save some money for his retirement as he won’t have any pension when that moment comes.’ plus, I liked this detail: ‘Meeting Mariano was funny, because at the beginning he was not convinced we were not there from the lending organization to check on him.’ (via Eoin)
    (tags: kiva microfinance loans developing-world peru small-world)

  • Scale Something: How Draw Something rode its rocket ship of growth : Membase, surprise answer. In general it sounds like they had a pretty crazy time — rebuilding the plane in flight even more than usual. “This had us on our toes and working 24 hours a day. I think at one point we were up for around 60-plus hours straight, never leaving the computer. We had to scale out web servers using DNS load balancing, we had to get multiple HAProxies, break tables off MySQL to their own databases, transparently shard tables, and more. This was all being done on demand, live, and usually in the middle of the night. We were very lucky that most of our layers were scalable with little or no major modifications needed. Helping us along the way was our very detailed custom server monitoring tools which allowed us to keep a very close eye on load, memory, and even provided real time usage stats on the game which helped with capacity planning. We eventually ended up with easy to launch “clusters” of our app that included NGINX, HAProxy, and Goliath servers all of which independent of everything else and when launched, increased our capacity by a constant. At this point our drawings per second were in the thousands, and traffic that looked huge a week ago was just a small bump on the current graphs.”
    (tags: scale scalability draw-something games haproxy mysql membase couchbase)

  • Scaling: It’s Not What It Used To Be : skamille’s top 5 scaling apps. “1. Redis. I was at a NoSQL meetup last night when someone asked “if you could put a million dollars behind one of the solutions presented here tonight, which one would you choose?” And the answer that one of the participants gave was “None of the above. I would choose Redis. Everyone uses one of these products and Redis.” 2. Nginx. Your ops team probably already loves it. It’s simple, it scales fabulously, and you don’t have to be a programmer to understand how to run it. 3. HAProxy. Because if you’re going to have hundreds or thousands of servers, you’d better have good load balancing. 4. Memcached. Redis can act as a cache but using a real caching product for such a purpose is probably a better call. And finally: 5. Cloud hardware. Imagine trying to grow out to millions of users if you had to buy, install, and admin every piece of hardware you would need to do such a thing.”
    (tags: scaling nginx memcached haproxy redis)

  • Clay Shirky Q&A: online creativity and intellectual property | Comment is free | guardian.co.uk : Good discussion and some great points, particularly this one for pro-copyright comments from “creative class” types: “there are few absolutes in copyright. To the question of motivation, if no copyright equaled no work, the fashion business would collapse, as their products are not covered by copyright. Money is one form of reward, but there are others (many non-fiction authors make more money doing things ancillary to their writing than they do from the writing, and then there is the explosion in labors of love), and copyright is one way to arrange the flow of money, but it’s a less good one than it used to be, because we are in an environment that makes that model of control less salient, and the other forms of reward moreso. So the logic of “It’s copyright or chaos” isn’t holding up well.”
    (tags: copyright clay-shirky the-guardian creative-commons fashion)

Links for 2012-04-16

Published April 16, 2012

  • The Cybercrime Wave That Wasn’t – NYTimes.com : MSFT researchers discover fundamental scientific failures in almost all data on cybercrime/spam/malware damages. ‘In numeric surveys, errors are almost always upward: since the amounts of estimated losses must be positive, there’s no limit on the upside, but zero is a hard limit on the downside. As a consequence, respondent errors — or outright lies — cannot be canceled out. Even worse, errors get amplified when researchers scale between the survey group and the overall population. […] The cybercrime surveys we have examined exhibit exactly this pattern of enormous, unverified outliers dominating the data. In some, 90 percent of the estimate appears to come from the answers of one or two individuals. In a 2006 survey of identity theft by the FTC, two respondents gave answers that would have added $37 billion to the estimate, dwarfing that of all other respondents combined.’ my opinion: this is what happens when PR drives the surveys — numbers tend to inflate to make headlines
    (tags: fail science pr press cybercrime ms via:mark-russinovitch data surveys spam malware viruses phishing)

  • Censoring The Pirate Bay is Useless, Research Shows : ‘The assumption of BREIN and the court was that a blockade of The Pirate Bay would lower the number of infringers at [Dutch ISPs Ziggo and XS4ALL], but new research from the University of Amsterdam shows that this is not the case. […] The claim that The Pirate Bay blockade by Ziggo and XS4ALL leads to a decrease of copyright infringement by their subscribers via BitTorrent transfers must be rejected. There is no significant effect of this measure. […] ‘Ziggo and XS4ALL subscribers who use BitTorrent apparently found different routes other than ‘The Pirate Bay’ to share files, and remain active as seeders to upload files to others.’ Unfortunately the paper is in Dutch, however
    (tags: holland brein ziggo xs4all bittorrent piratebay piracy research data)

  • French ‘Three Strikes’ Law Slashes Piracy, But Fails to Boost Sales : Hadopi report says piracy dropped in France by between 17% and 66% during 2011, while Hadopi was in force; however the SEVN report on 2011 notes that legitimate sales of video dropped by 2.7%, ironically blaming ‘the continually high level of piracy despite counter measures adopted under the HADOPI law’ (http://www.dvd-intelligence.com/display-article.php?article=1676), and the SNEP report on 2011 sales of audio indicates that the market dropped by 3.9% (http://www.telecompaper.com/news/french-online-music-worth-eur-110-mln-in-2011-study). Hard not to come to a conclusion that actions against piracy do not improve sales
    (tags: france hadopi legal music piracy sales revenues sevn snep video)

  • Why the New Aesthetic isn’t about 8bit retro, the Robot Readable World, computer vision and pirates | : ‘The New Aesthetics, or at least the aspect I’m looking at, is inspired by computer vision. And computer vision is at the point now that computer graphics was at 30 years ago. The New Aesthetics isn’t concerned with retro 8bit graphics of the past, but the 8bit graphics designed for machines of the now.’ — ie, The Robot Readable World, etc. Great essay, and exciting stuff
    (tags: art design new-aesthetic retro robotics graphics computer-vision)

  • HotelClub : a decent hotel search/booking site, recommended by On The Record’s Jim Carroll ?(@jimcarrollOTR on Twitter): ‘@sineadgleeson use HotelClub – good range of hotels & prices. Or use Hotel Tonight app for real last minute stuff’
    (tags: hotels travel recommended booking)

  • Metricfire – Powerful Application Metrics Made Easy : Irish “metrics as a service” company, Python-native; they’ve just gone GA and announced their pricing plans
    (tags: python metrics service-metrics)

Links for 2012-04-13

Published April 13, 2012

  • Ask For Forgiveness Programming – Or How We’ll Program 1000 Cores : Nifty concept from IBM Research’s David Ungar — “race-and-repair”. Simply put, allow lock-free lossy/inconsistent calculation, and backfill later, using concepts like “freshener” threads, to reconcile inconsistencies. This is a familiar concept in distributed computing nowadays thanks to CAP, but I hadn’t heard it being applied to single-host multicore parallel programming before — I can already think of an application in our codebase…
    (tags: race-and-repair concurrency coding ibm parallelism parallel david-ungar cap multicore)

  • Operations, machine learning and premature babies – O’Reilly Radar : good post about applying ML techniques to ops data. ‘At a recent meetup about finance, Abhi Mehta encouraged people to capture and save “everything.” He was talking about financial data, but the same applies here. We’d need to build Hadoop clusters to monitor our server farms; we’d need Hadoop clusters to monitor our Hadoop clusters. It’s a big investment of time and resources. If we could make that investment, what would we find out? I bet that we’d be surprised.’ Let’s just say that if you like the sound of that, our SDE team in Amazon’s Dublin office is hiring ;)
    (tags: ops big-data machine-learning hadoop ibm)

Links for 2012-04-12

Published April 12, 2012

  • Graft punk: Breaking the law to help urban trees bear fruit : This is brilliant. I find it pretty offensive that “ornamental” fruit trees are chosen by urban councils, so that fruit doesn’t fall on the path and become slippery or whatever — come on, that’s just what trees do! ‘They’re covertly grafting — a practice of connecting two branches in a way that will allow their vascular tissues to join together — fruit tree limbs onto the trunks of ornamental cherry, plum, and pear trees.’
    (tags: public roads trees nature city urban fruit guerrilla grafting)

  • The Cake Cafe map of Ireland : ‘Now that Dublin is in our bag, on our Tea Towel and across our Aprons, The Cake Café is going to create a new map of Ireland. We want to fill this map with all of your favorite places in land. Please send us locations that turn you on, fire your imaginations, or just fulfill your dreams; what ever you think should be included. Please pass the request on to friends in far flung parts of the land so they too can send their suggestions; natural or unnatural, animal or man made, a view, a corner of a field, an island or even a journey or hidden places to enjoy a picnic. — thecakecafe /at/ gmail.com’. Their map of Dublin is a work of genius — I love that they include a decent chunk of the Northside, which was a notable failure of the Alljoy Design version. I can’t wait to see what they come up with for Ireland.
    (tags: cake-cafe ireland maps mapping crowdsourcing dublin design tea-towels)

Links for 2012-04-06

Published April 6, 2012

Links for 2012-04-04

Published April 4, 2012

  • A one-line software patent – and a fix : Just another sad story of how software patenting made a standard useless. “I had once hoped that JBIG-KIT would help with the exchange of scanned documents on the Internet, facilitate online inter-library loans, and make paper archives more accessible to users all over the world. However, the impact was minimal: no web browser dared to directly support a standardized file format covered by 23 patents, the last of which expired today. About 25 years ago, large IT research organizations discovered standards as a gold mine, a vehicle to force users to buy patent licenses, not because the technology is any good, but because it is required for compatibility. This is achieved by writing the standards very carefully such that there is no way to come up with a compatible implementation that does not require a patent license, an art that has been greatly perfected since.”
    (tags: via:fanf patents jbig1 swpats scanning standards rand frand licensing)

Links for 2012-03-31

Published March 31, 2012

  • Girls and coding: female peer pressure scares them off | Education | The Observer : ‘Coding and digital prowess is still niche at a young age, self-taught by the studious. It is often considered a bit nerdy in senior school, where it is not currently taught as a part of the curriculum, although this is changing in senior schools from September 2012. Therefore, generally speaking, those who code have taught themselves. Teaching yourself something that should really be covered as a part of lessons is a bit like doing extra homework – why, ask many teens, would anyone do that? There is no way the majority of hormonally challenged, desperate-to-find-their-place-in-the-world teenage girls would risk ridicule or isolation by doing such a thing – let alone be open and proud about it. (Boys of the same age have different social challenges and do not measure their societal worth so much by peer review.)’
    (tags: girls coding education peer-pressure software teaching kids)

Links for 2012-03-30

Published March 30, 2012

Links for 2012-03-27

Published March 27, 2012

Links for 2012-03-22

Published March 22, 2012

Links for 2012-03-20

Published March 20, 2012

Links for 2012-03-18

Published March 18, 2012

  • Colm McCarthy: This burden of bank debt is simply not sustainable : Powerful burn-the-bondholders editorial from Colm McCarthy in the Indo. ‘No other eurozone member has incurred bank-related debt under ECB duress. There are no provisions in the Maastricht Treaty, in the Stability and Growth Pact or in any other pact or international treaty which grant this power to the ECB, nor was any eurozone member state ever asked to accede to such an arrangement. Commissioner Rehn’s Latin phrase (“pacta sunt servanda”) has no pact to refer to, insofar as these imposed debts are concerned. Ireland never signed a pact or treaty which empowered the ECB to behave in this fashion. One can only speculate as to the ECB’s motives, since it does not deign to explain. European banks have come to rely heavily on unsecured bond financing and the ECB may have felt that no bank bondholder should suffer losses, in order to encourage the survival of this market in bank debt. If this was the motive, the policy is being paid for, not by the ECB, but by Irish taxpayers and sovereign bondholders and financed by European taxpayers and the IMF. There is no pact which confers powers of taxation on the ECB.’
    (tags: bondholders ireland finance colm-mccarthy bailout)

Links for 2012-03-14

Published March 14, 2012

Links for 2012-03-13

Published March 13, 2012

Links for 2012-03-11

Published March 11, 2012

  • I left my shutter open for 30 seconds in the wilderness at 10.30pm, under a full moon : Amazing shot. With a sufficiently long exposure, it looks like midday — no colour correction applied. (via fp)
    (tags: via:fp pictures photos night colour landscapes long-exposure photography)

  • Occursions : ‘Our goal is to create the world’s fastest extendable, non-transactional time series database for big data (you know, for kids)! Log file indexing is our initial focus. For example append only ASCII files produced by libraries like Log4J, or containing FIX messages or JSON objects. Occursions was built by a small team sick of creating hacks to remotely copy and/or grep through tons of large log files. We use it to index around a terabyte of new log data per day. Occursions asynchronously tails log files and indexes the individual lines in each log file as each line is written to disk so you don’t even have to wait for a second after an event happens to search for it. Occursions uses custom disk backed data structures to create and search its indexes so it is very efficient at using CPU, memory and disk.’
    (tags: logs search tsd big-data log4j via:proggit)