FollowStream not working

Create issue
Issue #2 resolved
Rune Halvorsen repo owner created an issue

Apparently FollowStream doesn't deliver data as it should. Needs test and a good list of user IDs to test with

Comments (15)

  1. Former user Account Deleted

    I've tried debugging on my own twitter id and just posting test twits when ready to debug (hint it's possible to remove/delete twits again). The Spritzer stream works fine. Using Python 2.5.2 on Ubuntu. Been debugging on it today but couldn't find any reason why it behaves as it does. When sniffing the raw network traffic the messages come in on the tcp stream but readline keeps blocking and doesn't return (and there is newlines in the stream). Maybe something deep in urllib2 or httplib? could it be timeouts as follow.json takes quite a while to reply before the stream starts? or could it be the carriage returns in beginning of the stream? or something third? :-) I've changed to just use pycurl as curl seems to handle it ok for some reason...

  2. Rune Halvorsen reporter
    • changed status to open

    Working on a test suite today, and I also had problems with follow/track. The only difference between those streams is if they are GET or POST requests. Since there seems to be errors when connecting to my test server, as well as the twitter server, it seems you might be right about there being something deep in httplib or urllib. I'll try comparing output from curl and urllib2 to see if there's a difference.

  3. Rune Halvorsen reporter

    Found one bug, where on slow connections, twitter will send a newline every once in a while to make sure the connection is kept alive. Fix for this will be in shortly. Second issue, the one with not getting a tweet yielded as soon as it has arrived over the wire is related to buffering in the _fileobject class in the socket module, which is what urllib uses to enable readline on streams. The default buffer size is 8192, which fits quite a few tweets. Lowering the buffer size seemingly fixed the issue. It's not ideal though, digging into the socket internals. httplib might be a better alternative.

    Also, twitter doesn't seem to follow their own spec wrgt newlines/carriage return. Docs say CR delimits tweets, but it looks like we're getting CRLF

  4. Former user Account Deleted

    There seems to be a problem using TrackStream -- it was working fine, but now it seems to just be closing the connection remotely. Anyone else having a problem lately with this?

    stream = tweetstream.TrackStream(username, password, trends)

    for tweet in stream: print tweet


    Traceback (most recent call last): File "./twitter-stream", line 16, in <module> for tweet in stream: File "/usr/lib/python2.5/site-packages/tweetstream-0.3.1-py2.5.egg/tweetstream/", line 167, in next self._init_conn() File "/usr/lib/python2.5/site-packages/tweetstream-0.3.1-py2.5.egg/tweetstream/", line 148, in _init_conn raise ConnectionError(exception.reason) tweetstream.ConnectionError: <ConnectionError (104, 'Connection reset by peer')>

  5. Former user Account Deleted

    I believe the URLs have changed, you can look it up on the api wiki and try modifying it if you like. I'll have a look next week when I'm back from holiday

  6. Rune Halvorsen reporter

    I've updated the twitter urls in the latest version. Hope this fixes it, but will try to get some tests written

  7. Former user Account Deleted

    Hi there, just tried TweetStream and TrackStream, both returning a 404 - Wireshark says a 401 is being presented first then a 404. ANy fixes for this? I'm using the packaged version from pypi.

  8. Rune Halvorsen reporter

    (Reply via

    That's weird. Can't reproduce it here at the moment. A couple of questions:

    1. Do you still get the same problem? Might have been a fail whale error on their end or some other intermitent condition 2. Did it work for you before or is this the first time you're testing it? 3. Did you have any other twitter apps running that might have been using up all your requests? I believe there is still an api limit in palce. 4. You grabbed the latest from pypi within a week or so? There were url fixes made rencently 5. Do you have any exotic characters in your user name that I might not be encoding properly? 6. Ditto with password. you might try changing it temporarily to something with only boring ascii in it and try to log in to check.

    If the problem persists for you, can you include the exception you get and the snippet of code that caused it?

  9. Former user Account Deleted

    my problem with follow is that beyond 2000 users ids it blocks. It's not twitter, because I have shadow access, allowing many more to follow. Could it be a buffer limit in the POST? Laurent.

  10. Former user Account Deleted

    Hi, first of all, thanks for writing this library. Is just what I was looking for.

    I'm having one problem. I'm using the Track subclass and it works OK when I'm tracking popular keywords (for example, one of the trends at any given time). But when I try to track less common keywords, i don't get any tweet, even though I'm tweeting that keyword myself.

    Just in case, if I tweet something using one of the popular keywords I do get my own tweets, so I don't think it has to do with my account.

    Any ideas on what could be happening?

    Thanks, Jorge

  11. Former user Account Deleted

    I get the EXACT same error that Anonymous #7 is getting. I got tweetstream for pypi.

    Trying to use TrackStream -- repeatedly returning a 404. Trace shows, first 401 was received, then after trying basic authentication with supplied credentials, a 404 was received. (The funny thing is everthing works perfectly on a different machine. Neither machine uses a proxy).

    File "/usr/lib/python2.6/site-packages/tweetstream-0.3.1-py2.6.egg/tweetstream /", line 167, in next self._init_conn() File "/usr/lib/python2.6/site-packages/tweetstream-0.3.1-py2.6.egg/tweetstream /", line 141, in _init_conn self._conn = File "/usr/lib/python2.6/", line 397, in open response = meth(req, response) File "/usr/lib/python2.6/", line 510, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.6/", line 429, in error result = self._call_chain(*args) File "/usr/lib/python2.6/", line 369, in _call_chain result = func(*args) File "/usr/lib/python2.6/", line 855, in http_error_401 url, req, headers) File "/usr/lib/python2.6/", line 833, in http_error_auth_reqed return self.retry_http_basic_auth(host, req, realm) File "/usr/lib/python2.6/", line 843, in retry_http_basic_auth return, timeout=req.timeout) File "/usr/lib/python2.6/", line 397, in open response = meth(req, response) File "/usr/lib/python2.6/", line 510, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.6/", line 435, in error return self._call_chain(*args) File "/usr/lib/python2.6/", line 369, in _call_chain result = func(*args) File "/usr/lib/python2.6/", line 518, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 404: Not Found

  12. Rune Halvorsen reporter

    Hey, #12

    Can you try with the latest version? You're apparently using 0.3.1, which is quite old. 0.3.5 should be the latest on pypi. If it still breaks, could you either get a bitbucket account, or mail me, so I can try to debug it with you? Cheers

  13. Former user Account Deleted

    readline() buffers, you'll need to make your own readline that does a read(1) and return if "\n" is found

  14. Log in to comment