Benchmark rendering
Issue #17
resolved
I think this is a good time to see what city sizes do we support in a decent amount of rendering time / FPS. For a start it would be nice to know if we can parse/display the 10sq km city. Another option would be to use the simple city builder to generate really big cities (ping me if you need help with this).
Comments (14)
-
-
reporter
Ok, looks like we have a lot of work to do :/. A 10^2 km should be an easy one to handle and one of the things we should aim for is (relatively) fast loading.
IMHO we should tackle the three things in parallel: - If the parsing alone is taking 2:30 minutes then we should create an issue in Carlos repo to see if he can speed things up. Also it would be an opportunity to work closer with him on his codebase. - 27 secs for generating the roads is not bad, but we should definitely improve. For this city we should aim to no more than 5 secs, although 10 secs may be an acceptable upper bound. - Regarding rendering, we should get in touch with Louise (@JChoclin has her contact and can make the proper introductions) and Ian and ask for advice. We know that she has been working on improving Gazebo performance, so she may be able to help (esp now that we have a concrete case to share). - It would be great if we could have a more standard way of doing time benchmarks. Again, maybe Louise can give us pointers on how to do this.
Finally, maybe it would be worth splitting this issue in three separate one to track each item on its own issue.
-
reporter
- marked as critical
@agalbachicar @JChoclin I'm raising the priority of this one, as we should get some concrete improvements for next week. Please, organize between yourselves to take the most of Javier working the next two days.
-
- changed status to open
-
@JChoclin, here is the link from @ernestmc. However there are tools provided in Gazebo respository for performace. I will be using those to compare with the same tools.
-
Preliminary results for darpa example:
[Msg] ------------------------------------------------- [Msg] [MANIFOLD]: Start file parsing... [Msg] [MANIFOLD]: End file parsing. [Msg] [MANIFOLD]: Time consumed: 3.05991 s [Msg] [MANIFOLD]: Memory consumed: 10 MB [Msg] ------------------------------------------------- [Msg] ------------------------------------------------- [Msg] [PLUGIN]: Start loading visuals into Gazebo... [Msg] [PLUGIN]: End loading visuals into Gazebo. [Msg] [PLUGIN]: Time consumed: 0.0726194 s [Msg] [PLUGIN]: Memory consumed: 0 MB [Msg] ------------------------------------------------- [Msg] ------------------------------------------------- [Msg] [PLUGIN]: Start moving camera... [Msg] [PLUGIN]: End loading visuals into Gazebo. [Msg] [PLUGIN]: Time consumed: 0.000118892 s [Msg] [PLUGIN]: Memory consumed: 0 MB [Msg] ------------------------------------------------- -
reporter
Nice. Can't wait to see what happens with the 10km^2 :)
-
Preliminary results for 10km sq city example:
[Msg] ------------------------------------------------- [Msg] [MANIFOLD]: Start file parsing... [Msg] [MANIFOLD]: End file parsing. [Msg] [MANIFOLD]: Time consumed: 189.985 s [Msg] [MANIFOLD]: Memory consumed: 18446744073709551557 MB [Msg] ------------------------------------------------- [Msg] ------------------------------------------------- [Msg] [PLUGIN]: Start loading visuals into Gazebo... [Msg] [PLUGIN]: End loading visuals into Gazebo. [Msg] [PLUGIN]: Time consumed: 3.65402 s [Msg] [PLUGIN]: Memory consumed: 15 MB [Msg] ------------------------------------------------- [Msg] ------------------------------------------------- [Msg] [PLUGIN]: Start moving camera... [Msg] [PLUGIN]: End loading visuals into Gazebo. [Msg] [PLUGIN]: Time consumed: 0.000149274 s [Msg] [PLUGIN]: Memory consumed: 0 MB [Msg] -------------------------------------------------As you can see memory consumption is not really relevant. It gives an overflow, but time is giving us important information. Something important to note is that smoothing in case of distances larger than 10feet is enabled.
I'm going to track memory consumption and CPU usage through top cmd to get an overview of the whole process with and without the plugin.
-
Something else, loading the visuals of roads and waypoints lasts seconds, 3.7 in the last case, but Gazebo rendering costs much more and it is not something we can simple measure. At least not programatically.
-
Final Results
Normal usage
gzserver
gzclient
10KM SQ
gzserver
gzclient
Inside plugin measured times:
[Msg] ------------------------------------------------- [Msg] [MANIFOLD]: Start file parsing... [Msg] [MANIFOLD]: End file parsing. [Msg] [MANIFOLD]: Time consumed: 204.918 s [Msg] [MANIFOLD]: Memory consumed: 18446744073709551526 MB [Msg] ------------------------------------------------- [Msg] Name: [city_1] [Msg] Version: [1.0] [Msg] Number of segments: 1282 [Msg] Number of zones: 0 [Msg] Waiting for master. [Msg] Connected to gazebo master @ http://127.0.0.1:11345 [Msg] Publicized address: 10.34.0.65 libGL error: failed to open drm device: Permission denied libGL error: failed to load driver: i965 [Msg] ------------------------------------------------- [Msg] [PLUGIN]: Start loading visuals into Gazebo... [Msg] [PLUGIN]: End loading visuals into Gazebo. [Msg] [PLUGIN]: Time consumed: 4.86805 s [Msg] [PLUGIN]: Memory consumed: 18446744073709551502 MB [Msg] ------------------------------------------------- [Msg] ------------------------------------------------- [Msg] [PLUGIN]: Start moving camera... [Msg] [PLUGIN]: End loading visuals into Gazebo. [Msg] [PLUGIN]: Time consumed: 0.000127453 s [Msg] [PLUGIN]: Memory consumed: 0 MB [Msg] -------------------------------------------------Conclusions
From comparison between normal behavior and with 10Km squared city and time consumption for each part we can say that there is no important memory consumption when creating messages to Gazebo and sending --> This can be seen in the time and memory report. Also, we need to improve how Manifold library creates its graph and parses the RNDF file.
@JChoclin @andres_fortier do you agree?
Graphs were done with a bash script that parses top information of gzserver and gzclient and Libre Office Calc
-
reporter
@agalbachicar thanks a lot for the reports! I just created https://bitbucket.org/osrf/manifold/issues/5/parser-performance-review so we can keep these issues in sync. I would like to get some in-depth details of where are we spending time on the visuals, is that possible? Ideally I would like to have our times cut in a half (between 1.5s and 2s), but hard to say if this is possible without seeing some per-function costs.
-
@andres_fortier I've created issue #30 to tackle the in-depth performance testing.
-
- changed status to closed
-
- changed status to resolved
- Log in to comment
@andres_fortier Some preliminar and not methodical results of CPU usage and RAM memory.
While parsing 10KMSQ City:
While generating roads and arrows (whithout SPline interpolation):
While evertyhing was rendered:
When closing the app:
And the result:
There was a point at which the zoom out breaks so I think we should scale the city or at least give it as a paramenter the scale factor.
Just a thought. As far as I can see, we can improve a lot, at least the road and waypoint creation, by sending lanes to a pool of threads to be rendered. This should be done easily.