1. Jason Carr Avatar Jason Carr
  2. Untitled project
  3. LaunchBox
  4. Issues

Additional Scraper Options

Issue #285 new
CriticalCid created an issue

Including EmuMovies as an additional media scraper was awesome but I would suggest adding some more websites for game information’s. TheGamesDB is a great database but when it comes to less known games or japan exclusives it comes to its limits.

I know that the most known game databases like GameFAQs or MobyGames don’t want that scrapers download their game info’s and artwork. So I suggest some great sites which support scrapers.

Giantbomb: Huge database which features many unknown and special interest games. API: http://www.giantbomb.com/api/documentation

Co-Optimus: I think the name says it all. This site is specialized for info’s about the local and online multiplayer/co-op capabilities but features other games metadata as well. It would also come in handy for my request of scraping and saving the number of local players ;) API: http://api.co-optimus.com/games.php?params=true

Metacritic: No need to say something about this. AFAIK it hasn’t an official API, but with Google I found some unofficial ones. I don’t know how good they work but an inclusion of the metacritic score similar as the IMDB score in Kodi would be awesome.

I hope with all those additional scrapers it would be possible to find nearly every game and fill most of the metadata gaps.

Comments (41)

  4. CriticalCid reporter

    Hey Jason, I'm really glad that I could help. I hope this request (and the number of local players ;)) gets a really high priority on your roadmap because in my opinion the metadata is the core of LaunchBox. Anyways, I just found 2 other databases with APIs:

    IDGB: A pretty new website which wants to become the IMDB for Video Games. It's still in Beta but it looks really good so far. The API was just released in August this year. API: https://www.igdb.com/forums/general/blog/topics/igdb-dot-coms-api-now-launched-officially

    Gamesradar: One of your typical video game websites with News, Reviews, and of course a games database. API: http://dl.gamesradar.com/api/GamesRadar_Developer_API_Documentation.pdf

  5. Jason Carr repo owner

    Holy crap, IGDB...that sure sounds promising. We'll keep our eyes open on that one. Perhaps we won't need to build our own database after all...

    Very, very cool. Thanks again.

  9. CriticalCid reporter

    Oh I have forgotten one of the most important scraper options when it comes to Arcade games! Please include support for the official History.dat file from Arcade-History:

    http://www.arcade-history.com/?page=download

    You won't find a better and more complete database when it comes to arcade games. It has more then 50.000 arcade games in it and is updated every month when a new version of MAME is released. Even mobygames and thegmaesdb have both not even 2000 games in their databases. And also it would solve the problem for the other non-MAME arcade systems which are using mandatory ROM names like the Demul systems (Atomiswave, Naomi, etc) because the mandatory filename scheme is already included in it. And with Emumovies wihich is using the mandatory filenames as well we have the perfect combo for metadata and artwork. I know you are currently working with Mobygames on an API but the History.dat should has the same priority.

  10. Psico Lock

    Mobygames is a real pain in the ass, they are absurdely scrict with the data they accept, they prefer to not include a game if you don´t have premium quality covers and a serious review of the game.And thinking about games of the 70-80´s or panchiko mahjong games i think that is a "littlle" problematic. Yeah, I know i´m really angry with this people. And besides they prefer to name japanese games with special diacritics that made mad a lot of scrappers and western systems os-programs.

  11. CriticalCid reporter

    Thats right. But at least you will have high quality covers for the games in your LaunchBox setup.

    I vote for History.dat, Giantbomb and Mobygames to be top priority to integrate in LaunchBox. Together with TGDB, EmuMovies and Wikipedia it will cover most likely nearly every game you can imagine.

    And if Co-Optimus is included as well you will even know with how many players you can play these games ;)

  13. CriticalCid reporter

    Agreed. In fact is GameFAQs my primary source besides Wikipedia for retro game information’s. BUT they don’t have an API for scrapers so you can’t easily get metadata from them and they don’t seem to plan to integrate one in the near future. They even block your IP address if you try to scrap metadata and artwork without an API. The Advanced Launcher Add-On for Kodi uses GameFAQs as scraper but after 10-15 games your IP is blocked.

  14. Jason Carr repo owner

    Wow, good to know Cid. Oscar, Moby being strict is probably a good thing overall. It's good to have a combination of strict providers and more lenient ones, I'd say. Moby also seems to have more obscure games than anyone, from what I've found.

  15. Psico Lock

    My guess is put our asses to work. I´m upgrading the games database with a bunch of data but i can´t update all alone. I think is better to improve the games database than asking jason for infinite databases to add. We are a good group of users. Let´s make a team and do it!

  16. Jason Carr repo owner

    Yeah, I would run with that if I was convinced that TGDB was a great solution overall. I started out believing that, which is why I integrated it into LaunchBox to begin with. Unfortunately it's uptime and stability over the past couple years has been so horrid that I cannot trust it going forward. I don't like to keep putting in time adding more and more scrapers, to be honest (it's not much fun to develop), but it's a very important aspect of LaunchBox. And sadly I just can't trust TGDB with that, especially when it goes down so often.

  17. Max Payne

    Even if we put our asses to work and we add every game in the existence, as Jason said, it is not uncommon for TGBD to going down and we can't rely on just one database, we don't know if their data will be available forever. Having one or two alternatives besides the current ones, is always good.

  18. Brad Cheyne

    No, the GamesDB has issues MobyGames, GiantBomb, EmuMovies and Wikipedia don't have. That would be the ideal 4.

  19. Max Payne

    @CriticalCid It seems I have found this: https://github.com/Soleone/gamefaqs

    From the description: This is a simple gem which gives you access to information about all games (any platform) from GameFAQs.com

    You can search for games by title and platform, and then view Reviews, FAQs, Cheats and Answers for it.

  20. Brad Cheyne

    Unless it is official we will not use it. Unofficial Metadata scraping can result in ban's for the users. This has been proven on other sites in the past.

    Edit: Looked at the link, it is also very old (probably broken) and in the wrong programming language.

  22. Brad Cheyne

    We've already looked at a lot of API's and cases where unauthorized access was gained to metadata. We classify this as any form of API not released by the site owners / developers. Usually this is ok if one of these sites has an API page. Wikipedia, Steam, GamesDB etc. These all have easily accessible API's EmuMovies requires contacting the owners but they state they love front ends and are ok with this and in other cases we are helping the developers develop an API (or updated API), like MobyGames. This has been something we've done a lot of research on for over a year, and Jason longer before that.

    Edit: Can we also move this to the forums not the bitbucket pages? The forums are for discussing these various issue pages.

  24. Psico Lock

    With Mobygames I recomend some workaround that detects special fonetic chars like Bishōjo (ō is ou) Gokūden (ū is uu) and convert to the typical fonetic translation. Nobody names the roms with this special characters and this straight name policy of mobygames results on failing to scrap info for a ton of japanese games.

  28. Chris

    This would be awesome you would be able to get rating systems from igdb.com for different regions and it's extra data for missing pictures etc. I think you need a API key to get the information but its free and could be added into the options search tap like EmuMovies and Retro Achievements.

  30. Gerardo Alcira

    I'd like to add, that if this feature is low priority or not coming at all, this proposal:

    Add an experimental method of a simple image search built into LB

  35. Conar McVicker

    MODDB comes to mind as another good scraper for some things. Would be nice to be able to find mod info in launchbox automatically

  37. Chris Kant

    maybe additional scrapper infos are not needed if adding data into the gamesDB would be easier and level of quality better.

  40. Andrew Russell

    Could one of the scraping sources pull fan translation status from romhacking.net? Maybe it could add an option to visit the corresponding translation page and download.

  41. Glauco Fox

    Please consider adding a functionality to play videos from an online source. We don’t go over every single game everytime we start LaunchBox, so I think it would be a huge gain in hard drive space if we could just load a video inside a regular game detail launchbox/bigbox window directly from youtube/emumovies

  42. Log in to comment
Assignee
Type
enhancement
Priority
major
Status
new
Votes
89
Watchers
85
Jira Software: the preferred issue tracker for Bitbucket. Join the team!