Reading the Twitter Stream API with PHP

Monitoring a term like ‘Facebook’ with the Twitter Search API can result in hundreds of tweets every minute. If you continuously issue requests for the next set of data, you’ll eventually get a message that reads, “You have been rate limited. Enhance your calm.” Essentially, it’s not possible to refresh the results from the search API fast enough and you will lose a significant portion of the data. The Twitter Stream API lets you get around this by serving up a firehose of tweets on nearly any topic as they happen. Twitter also recently announced a new Site Stream API which is designed for 3rd party sites. That product may suit your needs better.

The process is fairly simple. Open a connection and then suck it all in. While this example is specific to Twitter, it could easily be implemented for any other service that provides an open socket and continually pumps data through it. There are three stages to the code and you’ll expand on section three to process the data for your own use.

  1. Connecting to the API
  2. Creating the request
  3. Handling/Processing the data

In the first part, we establish a connection. We’re going to want to keep this connection open – so we use PHP’s built in socket connection function fsockopen(). For REST based services it often makes sense to use curl because of the simplicity and wide range of browser like support. Alas, curl also likes requests to end, so it’s a non-starter. This code will be running continuously, so you may also need to remove any command line execution time limits in your PHP configuration or simply include a call to set_time_limit(0).

Next we’ll create the initial request. I used an array to hold my query data. This is handy because other methods support additional parameters which I won’t go into here. The track parameter takes a comma delimited list of values. The following examples come from the Twitter documentation on methods and parameters.

Track examples:
Twitter will return statuses which contain: TWITTER, twitter, “Twitter”, twitter., #twitter and @twitter.
Twitter will not return statuses which contain: TwitterTracker and
helm’s-alee will return statuses which contain: helm’s-alee
helm’s-alee will not return statuses which contain: #helm’s-alee
twitter api,twitter streaming will return statuses such as: “The Twitter API is awesome” and “The twitter streaming deal is fast”.
twitter api,twitter streaming will not return statuses such as: “I’m new to Twitter”.
chirp search,chirp streaming will return statuses such as: “Listening to the @chirp talk on search”, “I’m at Chirp talking about search!”, and “loving this search talk #chirp”.

The parameters and login information are then formatted into our GET request. You can use Basic Authentication on this API which removes the hastle of setting up OAuth. After we fwrite() these values to the server, Twitter will immediately begin sending our stream.

The final part is to process the data. The loop code will run for each and every tweet received. Beware, not everything that comes over the wire will be usable data! NOOP and other noise will come as well, so be sure to check that you can parse what you receive. Twitter also doesn’t guarantee the order of the Tweets, although they’re roughly chronological.

Lastly, and this is a PHP gotcha, on 32-bit platforms, json_decode() and other functions will munge big integers leaving a useless Tweet Id in the object you get back. Twitter’s new snowflake ids are 64-bit. Use the string representation of the Tweet Id, stored in id_str, instead.


$query_data = array('track' => 'facebook');
$user = 'username';	// replace with your account
$pass = 'password';	// replace with your account

$fp = fsockopen("", 80, $errno, $errstr, 30);
	print "$errstr ($errno)\n";
} else {
	$request = "GET /1/statuses/filter.json?" . http_build_query($query_data) . " HTTP/1.1\r\n";
	$request .= "Host:\r\n";
	$request .= "Authorization: Basic " . base64_encode($user . ':' . $pass) . "\r\n\r\n";
	fwrite($fp, $request);
		$json = fgets($fp);
		$data = json_decode($json, true);
			// Do something with the data!
This entry was posted in PHP and tagged , , . Bookmark the permalink.

18 Responses to Reading the Twitter Stream API with PHP

  1. Anders Borg says:

    Thanks for a very concrete and to the point example.

    The only problem I have now is that I get a server error after a while, and I don’t think I can keep a PHP script “alive” forever.

  2. Erik says:

    Actually from the command line PHP can keep it alive, but in the browser the Apache (or whatever HTTPD server you use) will time the connection out after a while.

  3. argenisleon says:


    I was trying to connect with
    $fp = fsockopen(“”, 80, $errno, $errstr, 30);

    but was getting a “Conection refused” Error. I change the line to use a SSL connection a works perfect

    $fp = fsockopen(“ssl://”, 443, $errno, $errstr, 30);

  4. I had difficulty getting your example to work at first, but then I read that the Streaming API requires you to use HTTPS, so where you have…

    $fp = fsockopen("", 80, $errno, $errstr, 30);

    …you need to change it to…

    $fp = fsockopen("ssl://", 443, $errno, $errstr, 30);

    …in order for it to work.

  5. Ahmad Amin says:

    It is stated on twitter website that you need to use ssl so you guys are correct.

  6. Kamal Salem says:

    I’m trying to test the code above as-is, just put in my username and pssword, I have no idea what’s going wrong.

    I put the code in a file on my local server (WAMP) and opened it in a browser, is this the correct way to test it? if so, it keeps saying “waiting for localhost…” to no end, and nothing gets displayed, I used print_r($data); to try and display the data returned from the request.

  7. Jgordon says:

    Greetings !

    It is possible to have more than 1 word as a parameter for the search? for example :

    i would like to extract tweets with the word “twitter” or “facebook” inside the tweet.

    something like /1/statuses/filter.json?track=facebook&track=twitter

    is this possible? or there is other way to achieve this?

    Thanks a lot

  8. Jose Cardenas says:

    Thanks Erik for the great code. Very easy to implement and test, but I’m having an issue when I try to submit it with multiple keywords. I find when I have two keywords that it only pulls the stream from the 1st keyword. I’ve specifically tested with keywords that I know have lots of messages and ones that I personally have that have none and neither work with the second keyword. I’ve created a web page to display the stream with filters and it works great. Any ideas why this could be?


  9. Jose Cardenas says:

    Update: I found a new example that added this line:$request .= urlencode(implode($keywords, ‘,’)) . ” HTTP/1.1\r\n”; and just added track to the end of the json line.

    Everything works great except when I try submitting keywords from another page through a form submit. I figured out that I needed to encode the string to create the array and then pass it. I’m finding that once it adds the string to the URL it has an extra “+” inbetween the keywords so I get a %2C+ instead of just %2C. Through the form I enter the keywords as follows “keyword1, keyword2”, no quotes. If I hard code the keywords within the same page it works great and doesn’t have the ‘+’. That’s the only thing that I see different. Any suggestions on this would be greatly appreciated!

  10. Regardt says:


    The code above still assumed “basic auth” Twitter has now essentially enforced oAuth so this example won’t work as it stands.

  11. Kibe Wachira says:

    Thank you very much for this awesome code, really solved everything i wanted to do.

  12. Enrique says:

    Is there a way to monitor or retrieve historical tweets/stats using stream api from multiple accounts? 1 cron per monitored account maybe?

    Thanks for the post, very useful

  13. Aalaap Ghag says:

    Regardt, this code still works just fine. Just use ssl:// and port 443.

  14. BillD says:

    This doesn’t work. The only “data” I’m getting back is “Array”.

  15. Pingback: Test » Acceder a Streaming API de PHP

  16. don says:

    thanks for the post – really helpful. I have just two quick questions:

    1. Once I open the socket and everything is running correctly – how would I close it again. i.e. at a later date this code has all executed and is still running but I need to close the connection?

    2.If I search for two keywords – is there any way to determine which tweets were related to which keywords? i.e. it comes back with a variable stating the keyword which was referenced?

  17. Brad Bonkoski says:

    This code is officially dead… basic auth has been retired.

  18. dhanu says:

    I got HTTP/1.1 401 Unauthorized from json data. can u please help how to correct this?