Articles on ScoutLabs

Twitter FirehoseDrinking from the Twitter Firehose

The Twitter Firehose is a phrase that refers to the roughly 50 million daily tweets on everything from Wall Street reform to Justin Bieber’s favorite cereal. There are hundreds of tools on the market designed to monitor and measure Twitter data by allowing users to easily search and track Twitter trends. One of the big problems with most of these products is that they don’t have access to the full Twitter Firehose. Typically they might only have access to 10%-20% of all Twitter posts on any given day. There’s a couple reasons for this.

  • The cost and technical challenge of handling so much data
  • The cost – or privilege – to access the Firehose

Who’s Tapping it?

Giants like Microsoft, Yahoo, and Google have reportedly paid hundreds of millions of dollars for access to the full Twitter data stream. Surprisingly Twitter has decided to grant access to the full Firehose to a number of startups including; Twazzup, Collecta, CrowdEye, Scoopler, Kosmix, Chainn Search and Ellerdale.  No one is quite sure, at least I’m not, how much it’s costing these startups (if anything), and why they were chosen when so many others have been denied. I should also mention that there are also a small number of mid-size companies that fall somewhere between the giants and startups that now have access to the full Firehose – including Jive and Converseon.

A large number of social media monitoring companies – whose goal is to collect and consolidate all relevant social media data for its clients – are left out of the party. Which means even great products like Radian6 and Scout Labs have a significant hole in their data inventory – full Twitter Firehose access. Both companies expect to have full access at some point, but when isn’t yet clear.

What Does the Future Hold?

Some experts expect Twitter to eventually be more generous in granted access to the Firehose. However, this would be opposed to the direction they’ve taken recently in switching to their own URL shortening service (rather than using, unveiling their own iPad application, and requiring third-party Twitter applications to access user accounts through the OAuth authentication standard. The latter move caused problems for a number of high profile tools like TweetDeck and a number of widely used WordPress Twitter plugins.