rss release gets marked read even though it wasn't downloaded yet

[Former user]

Just as instructed on the site, I set up my uTorrent to automatically download new releases from the RSS feed. Weeks later I noticed that some of the releases were marked "read" even though they haven't been downloaded by uTorrent. It's really a problem since I have to go check my list and see which ones are downloaded properly and which ones are not. For those who are missed by uTorrent, I just have to click "Send to RSS" again.

[Shana Project]

Can you please let me know if those releases are hosted on nyaa torrents? I'm fairly sure they would be since that seems to be the main cause of what you describe.

This issue seems to have been getting more prevalent and I'm still a bit unsure on how to solve it. I'll try to figure something out sooner rather than later =(

[karappo-kun]

Sorry for late reply. I was on the lookout for a chance to encounter the bug again. Just now Log Horizon ep16 (HorribleSubs) and Chu2byou Ren Lite ep2 (UTW) got marked read again and as usual "Send to RSS" fixed it. While I was examining my "All Followed Releases" section I found something that might give a clue as to the nature of the bug.

The image below shows the rss release for Buddy Complex ep2 (Anime-Koi) and it was marked "read". I checked my Downloads folder but I only have the ep1, which means my uTorrent haven't downloaded it yet.

I tried the "Send to RSS" again but to my surprised it won't show up in uTorrent. I tried to download the torrent file instead but to my surprised I can't download it.

It seems that the bug manifests when the direct link to the torrent file no longer exists (or is broken temporarily) and Shana Project rss marks the item "read" regardless of the fact that the link is broken.

(Sorry for my bad English)

[Shana Project]

Hmmm...

This just raises more questions. We actually do a check to make sure the "file" exists, unfortunately it looks like nyaa isn't throwing a 404, but instead returning an html page (instead of a torrent) in some cases...

I'll have to have a think on this >_>

[karappo-kun]

How about storing a cached copy of the torrent file instead of linking them on third-party site? I think that would remove the broken link problem..

[Shana Project]

That's always been a possibility... It looks like I might have to make the change to actually do it... shouldn't be too difficult I guess. I'm just a bit unsure of how it should work if that release is getting deleted (because usually that means its a bad release). Usually thats fine thanks to "v2's" of files, but of Horrible are posting broken links after the working ones... hmmmmm.

[Shana Project]

Issue #36 was marked as a duplicate of this issue.

[Shana Project]

Now caching torrents after at least one person clicks on it.

[Tim M]

This appears to have reared its head again. I was having the same issue when this was originally made and it seemed to be fixed back when it was originally resolved. But I've noticed now (actually have for a while - months or more, just hadn't gotten around to looking into it) that this appears to be happening again. Very frequently I have seen torrents get marked as read even thought uTorrent didn't grab them, and so I missed out on them. Due to this, I have to go back through all the time and verify everything that was grabbed or not, making it not as "set & forget" as I'd like. I'm running uTorrent 2.2.1 (the last good, ad-free version). If you'd like I can try to start trying to track which ones are missed.

[Shana Project]

I'm honestly at a loss for what else we can do to fix this... the last thing we could do is to not mark the torrent as downloaded until we've literally pushed every last byte to the client... but it's a) somewhat difficult (I think... I've actually no idea how to do it), and b) would likely be fairly resource heavy (ie: potential performance hit).

There is one other possibility that I can think of... but I'm not sure what the best way to test it is. How often does your client check for updates, and how often are you seeing this issue happen?

[Tim M]

I'm using the default in uTorrent for RSS updating/checking (never changed it), which looks to be every 15 minutes. Verified by checking the rss.update_interval in Preferences > Advanced. I see the issue happen probably every few days give or take. As I said I haven't really been tracking it so it is hard to say exactly, but that feels about right.

[karappo-kun]

I just confirmed that the problem with my RSS torrents has something to do on my ISP's side. It took me long to realize that they were implementing some sort of traffic shaping on BitTorrent traffic. Surprisingly, they also extended the policy on RSS feeds that have torrent links in it. I don't know why. The torrent shaping start out at around 8:00 PM and ends at around 12:00 AM midnight (my timezone is +8:00 UTC by the way).

I found out about this while I was waiting out for [HorribleSubs] Anne Happy - 06 [720p].mkv to be downloaded at around 10:00 PM. The torrent appeared on Shana Project's "New Followed Releases" and while I was being impatient, I decided to use uTorrent's "Update Feed" option to immediately fetch the RSS feed. The feed didn't download, I tried closing uTorrent and tried again. Same thing happened. 12:00 AM passed and the traffic shaping was lifted. It was then that the uTorrent began downloading RSS feed and successfully fetched the torrent with no problems.

As a temporary measure, I'll probably just write a script to terminate uTorrent at 8:00 PM and wake it up at around 1:00 AM to fetch RSS feeds.

Edit: Caching torrents actually worked and RSS problem was fixed but sometime around January 2015 it came back again. I think it was that time my ISP started implementing that damn traffic shaping policy.

[Shana Project]

I'd be surprised if the ISP was blocking the RSS feed itself (also, that wouldn't explain why things disappear). I guess it could be blocking the .torrent file though...

When you download a file via RSS, we mark it as read as we send the file to your computer, so if they're blocking it at that point (detecting that the file is a torrent file - which is easy over http) then that could do it... We could try offering an https version of the RSS feed... that could work? But we've had all sorts of issues with https in general, so I'm not sure I want to =\

[karappo-kun]

I think your last suggestion might help:

"I'm honestly at a loss for what else we can do to fix this... the last thing we could do is to not mark the torrent as downloaded until we've literally pushed every last byte to the client... but it's a) somewhat difficult (I think... I've actually no idea how to do it), and b) would likely be fairly resource heavy (ie: potential performance hit)."

You can do this by operating your own BitTorrent tracker (the one that uses http protocol instead of udp). After you've fetched the torrent file from Nyaa (or other sources like TokyoTosho), you may want to tamper the torrent file to add your own tracker before you cached it on your server.

This is usually the default list of trackers for the torrent files fetched from Nyaa.

The new tracker entry added to the torrent file may look something like this: http://tracker.shanaproject.com/{user_id}/{release_id}/announce

Where user_id corresponds to user's account and release_id corresponds to the unique ID of the file being downloaded (a particular episode from a particular anime being subbed by a particular fansubbing group).

Now that you have a tracker, you can keep tabs of how much of user's BitTorrent client has downloaded so far for each file (or many pieces it has already downloaded). What you can do now is that you may want to mark the release "Read" ONLY when the user's BT client had actually downloaded at least 20 pieces of the file. This is your undeniable proof that the user's BitTorrent client has actually fetched and parsed the RSS properly and is starting to download the file.

Having an extra tracker actually has an added benefit. It makes the releases more resistant to downtime (just in case Nyaa's torrent site or its tracker server goes down, which really happens once in a while). You may also want to keep the torrent caching feature intact for added reliability.

This is actually just a suggestion and I know how hard it is to implement such a BitTorrent tracker so it is up to you to accept my suggestion. But if you're really up for it then you may want to check GitHub for tracker implementations and do a fork when you find one.

[Shana Project]

@karappo-kun sorry for the slow response - this is actually a really cool idea (why didn't I think of it?).

I have written and operated trackers before (so forking aside, implementing this would be trivial), but I'm not sure I want to add this sort of thing from a perspective of minimizing SP as a target for legal action.

I will think about it; but I think its fairly unlikely due to (1) legal fears; and (2) my lack of time to work on SP (other than bug fixes).