Slow/Random Crashes -- Massive Library

[Former user]

I am a long time user of Launchbox and have a bug/enhancement request:

Library XML file has 35k entries and is 56Mb is size!!

5.8, latest release

My instance of Launchbox is consistently very very slow and hit the point of non usability. I am a avid collector and completionist. I have an extensive collection of roms and isos that many have metadata where available. This I know taxes the application.

I suggest the following change to correct this bug and also improve performance:

• Split the database XML on a per platform basis
• Maintain synced master XML for cross platform user quires that only load on specific user requests and unloads to platform specific once title is queried
• Add ability to purge titles from XML database that lack gameDB data (helps users sanitize unwanted or otherwise undocumented games while boosting performance with a slimed down DB.

[Jason Carr]

Hi Anonymous, I've put some extensive work into optimization with the data access and I've previously come to some conclusions. Splitting up the XML file will do nothing to help performance, because the XML file is only read once at startup and reading even a 56 MB XML file is not causing major performance issues. Pretty much the only thing we could do to speed up data access is to use an actual database (such as SQL Server, MySQL, SQLite, or similar). I am considering this for the future.

If you're not running the latest beta, please upgrade to the beta, as I expect that will fix some of your stability issues. I'm not sure what you mean by very, very slow, but typically libraries of even 15,000 games should run very well on modern hardware. What kind of performance issues are you seeing? Where are you having to wait and how long are you having to wait?

Historically the metadata has not been cause for performance issues; the images have been the biggest problem and the hardest thing to optimize.

[Robert Taylor]

I am on a i7 machine with 16gb of RAM on Win 10 64x

My issues with performance are as follows:

Scroll lag - usable but stuttery Clicking on new platform - noticeable 3-5 second wait Change category - again 3-5 seconds Search: 2-4 seconds each character I type (regardless of platform) Creating custom rules: 10-30 second lag and sometimes crashes when using drop downs to customize custom rules Right click on game/title - intermittent lag of 1-2 seconds Startup: takes about 30 seconds - Launchbox app and data on SSD

[Robert Taylor]

As far as images go, try this:

ImageMagick

It is a FOSS program that you could build into a auto hot key script. You could add a new menu called "optimize images" in the tools menu. This program can smash down the file sizes of each image which ought to bring about a nice boost in performance.

[Robert Taylor]

I had 19GB of image data and now shrunk to 6GB w/ 21k files. A lot of work was done:

run via CMD: mogrify -strip -interlace Plane -sampling-factor 4:2:0 -quality 90% *.jpg I ran this to each folder to help massively reduce image file sizes. Might want to run it on your end to cut the hosting bills down a big notch. Might help maintain profit margins which could go in part to R&D time.

Delete all image folders per platform except "back", "front" and "screenshot". These three were the only important ones to me. I did not mind clearing out fanart and erasing the banners/logos.

After all this work, I still have virtually the same lag as stated above. I did update to the latest beta. I do not know what to do next. I have done everything in my power to optimize.

[Jason Carr]

How big is your collection, Robert?

[Brad Cheyne]

Yea this is strange... I have 15k entries, tons of images, metadata, video and music and I don't really have a scrolling lag, I wonder if it is another application hindering the performance? Do you have RAM monitor software or anything to monitor and track your PC stats in real time?

Edit: 35.6GB is what my LB folder just got totaled at.

[Brad Cheyne]

Actually, here is something we may have glossed over, have you left LaunchBox open all night (or in your case several days and nights) to let it all get cached? When you are scrolling through your list of all games do they all 1 by 1 in a row start popping in images? If so, LaunchBox hasn't cached all of the image you have yet and since your library is large this can take a while. At 15k items this takes easily a n hour or so.

It depends on your CPU mostly and we say over night as a precaution so users leave enough time to let it do it's thing. You may need to scroll the library down once in a while too, but letting it go on it's own should work fine.

[Robert Taylor]

That might be it. I ran imagemagick to compress my image data and deleted unwanted images.

I checked my cache, its about 90MB, just from a fresh job. I deleted everything in the cache because I wanted Launchbox to load my newly optimized images. I will leave it running for a day and do a couple scrolls under "all".

[Brad Cheyne]

The optimization may not matter as much in this instance though except for maybe make the process take a bit less time.

[Brad Cheyne]

Any news on the performance?

[Robert Taylor]

I let the program sit over night. My cache file is almost 2GB big and wont change. I am still experience lag during UI operations and slow start up times. Again, I am on an SSD with image data and install data on the SSD. My roms for small systems like NES, SNES, Gensis are on the same SSD. I can run PS2 games in software with no GPU acceleration and play Fall out 4 on max. I got a GTX 980, I should have no issues. I have not seen a new beta yet. I might be an issue with the beta. It used to preform slightly faster in 5.0.

[Brad Cheyne]

Ok, well it was worth a shot. For clarity's sake too, does the lag only happen when you have All games showing for every system or even when you have just a single system of games open? Even with 15k I can notice some lag when scrolling with them all showing but I generally only look at a system at a time or all of my favorites for my library so I never see the whole library almost ever.

[Jason Carr]

Hi Robert, I'll admit that 35,000 games might be the biggest we've seen. I haven't done a lot of testing with that big of a collection, so it may just be so big that it's getting exponentially slower past 20,000 games or so. If you would, send me your LaunchBox.xml file to [e-mail] so I can test on my system. Thanks!

[Robert Taylor]

Optimizing/shrinking my images massively help scroll lag. That is non-existence. However, the lag I get is clicking on platforms and custom filters. I get a 10-30 second load. I get the application to crash when I open the options or other settings menu. I can click on roms and they launch virtually instantly. I think what is going on is that each time I click on a category/filter or platform, the lag comes full force--possibly tabulating the image data for the requested query. Search menu is still very very laggy and delayed, even when searching in one system with about 1k titles.

I would suggest using progressive image loading--may save of CPU process when launching a new platform. Jpg allows files to load progressively showing a blurry fast image that takes less cycles to load and a sharp image once the scroll wheel stops. This may make the interface more snappy.

Also, try adding some predictive logic that unloads and preloads image data depending on user query. Ex: load only the first 50 "front covers ONLY" one any given filter or category. Progressively load in low quality images 50 at a time during scroll and once user stops--sharpen up images. Once a user clicks on a game and the details pane shows up, now and ONLY now, load fan art, backs and extra image data. Never let this data hit the ram or CPU unless specifically requested by user. I think this along with packing image magic with an optimization routine that is optional for users to run may seal the deal.

Finally, I like the database idea you have. I would go with mysql lite, should also do some more boosts to performance.

I got iTunes with album art and over 100k pieces of media; nothing is stopping this app from achieving the same.

[Robert Taylor]

Have you checked out my XML file and attempted to load it. I just want to rule out my system. My PC should be able to handle this kind of data assuming certain optimizations are present.

[Jason Carr]

Alright, I tested your XML file in the latest beta. I'm seeing wait times of between 2 and 3 seconds when switching between sidebar filters. That seems acceptable to me considering a collection of 35,000 games. The search results are taking about the same amount of time to populate once you stop typing.

Startup for me took about 15 seconds, which I admit could be improved. I recently added a splash screen with progress notifications to the latest betas, which helps, but obviously doesn't do anything to make it faster.

Custom filters are a known problem area right now. For large collections I'm aware that it's mostly unusable and I plan to review those very soon.

I'm not seeing any delays on right-click, but I suspect that has to do with the images you have in your collection.

I do appreciate the details here; we will continue to focus on improving performance. At this point though I think it would be wise to review the problematic performance areas and look for faster ways to do things rather than to revamp the entire data storage model.

[Jason Carr]

Tested on two systems which are both Haswell i7s with 16 GB of RAM. Knowing what LaunchBox is doing, though, I expect the CPU to be the bottleneck.

[Jason Carr]

Issue #608 was marked as a duplicate of this issue.

[Martin Alejandro Liguori]

Any fix for this? bigbox and launchbox are imposible to use with normal Arcade PC hardware and normal Arcade game library.

Any normal arcade hardware is in the range of core 2 duo/quad and i3, most people will use old computers to make their arcades since almost all emulators including ps2 emulation will reach top framerate and speed with a core2quad 3ghz.

I tested all your builds in my arcade core2quad 3.2ghz, 4gb ram with windows 7 x64 (screen resolution 800x600 and 1024x768) and when you reach 10k games the scrolling/menu changing/general usage is just imposible and very slow.

My collection is for 20k games.

Is there any hope for people with arcade hardware?

This issue and ledblinky are the only issues preventing me to use bigbox on my arcade machine, I switched back to my old front end but I wish I could use this one.

[Weltall of Id]

Only because I have seen this, I want to point out that I am at 10k+ games and I am not experiencing a major lag, although I am guessing my view of laggy might be different. It is slow for me only when it needs to cache in pictures, but after doing so it is quite fluid.

I am on a Core2Quad @3.2GHz, 8GBram, Windows 8 64bit, two screens one 1920x1080 and one 1280x1024 (not sure if that matters).

I actually have assigned to LaunchBox to use 2GB ram for caching, but I have never seen it even reach 1GB. Then again after a I scroll for a couple of minutes I find my game and play.

I do not use movies, just pictures.

[Brad Cheyne]

There can be lag in some spots, but what also might be considered lag might just be the program processing. A lot of factors are at play though: Other software installed that is running (like active monitoring software), the parts in the system as it pertains to just simply not enough, and LaunchBox over a network.

I don't specifically get lag, but LaunchBox can easily become unresponsive at almost 20k games.

[Robert Taylor]

Still an issue on 6.0 beta. I have no scroll lag and crashes from context menus are gone from my tests. I still however have MASSIVE load times when clicking on a new platform or category. Search still very very slow. Loads each character I type. Again, I am thinking a new database structure would help. My cache is set to 4gb, the max value. I have tried leaving the app open in the background and testing-I still have lag.

[Robert Taylor]

I may have said this already, but I think you should add progressive image loading. Ex: Load only the images for the first page and a little more hidden from view below the top scroll position. As the user scrolls down their games list, dynamically unload the images out of view about minus a preset buffer and start loading new images that are out of view below the current window plus a buffer. This implementation ought to vastly speed up launchbox. The app would now only need to load less data versus all image data on the selection custom filter or category.

[Brad Cheyne]

Issue #859 was marked as a duplicate of this issue.

[Jason Carr]

We've made tons of changes and performance improvements since this ticket was opened.