Improving the Accuracy of Point Cloud (Lidar) data for Vehicles
Welcome to the Wiki! Here we discuss how to make the lidar sensor more accurate by improving the accuracy of the collision boundaries (relative to the original shape of the object) which are the surfaces used to calculate the point-clouds generated by the lidar. Below is an example of what we can achieve with the method I will discuss (note you can increase accuracy as you like). We hope you can contribute to our pool of improved assets so that everyone can benefit! (see last section below for details on how to contribute)
The most natural way to create vehicle animations in Unreal is using 'skeletal meshes' and currently all vehicles in Carla are of this type. Unfortunately, Unreal Engine does not allow using the original mesh as collision boundary for these types of objects due to it's high computational cost. From https://docs.unrealengine.com/en-US/PythonAPI/class/SkeletalMesh.html: "enable_per_poly_collision (bool): Uses skinned data for collision data. Per poly collision cannot be used for simulation, in most cases you are better off using the physics asset". Instead, the Unreal engine automatically approximates the vehicle as a simple box for the body and 4 spheres for the wheels, which looks nothing close to a real vehicle, and this is pretty much how it looks with the lidar sensor. Here, judge for yourself:
What Can We Do?
There are two options here:
- Use static meshes and move them around as if you were moving props - this requires significant redesign of Carla and it's unclear how you would do the animations of the suspension/wheels
- Create an approximate custom collision boundary that can be used with skeletal meshes
This Wiki discusses option no.2 and includes tutorials I created for the general community. While it requires some knowledge of 3D modeling, hopefully the tutorial will allow more contributions so we can have better looking vehicles in lidar in no time! I am not an artist and I learned all of this stuff by myself so you can too!
Before we begin, a lot of credit goes to Francisco E who published this clip https://www.youtube.com/watch?v=SEH4f0HrCDM through which I learned how to import custom collision in Unreal.
- Install Unreal Engine 4.24 and follow all the installation instructions in the Carla project
- Install Blender 2.80 or newer from https://www.blender.org/download/
- Install the VHACD Plugin for Blender using the instructions in https://github.com/andyp123/blender_vhacd. This plugin automatically creates an approximation of a selected object using a collection of convex hulls. See more details here: https://github.com/kmammou/v-hacd
- If you are new to Blender checkout this series https://www.youtube.com/watch?v=ppASl6yaguU or this Udemy course https://www.udemy.com/course/blender-3d-from-zero-to-hero/?pmtag=MRY1010
Creating Custom Collision for Skeletal Mesh
First, we summarize all the steps and a detailed tutorial will follow for each one:
- In Unreal Editor: Add collision boundaries for the wheels
- In Unreal Engine Editor: export the skeletal mesh asset of a vehicle to an FBX file
- Import the FBX file into Blender (free open source 3D modeling software)
- In Blender: add meshes that are convex hulls (this is an Unreal requirement for computational efficiency) to form the new collision boundary (the hardest and most time-consuming part!)
- In Blender: export the custom collision boundaries into an FBX file
- In Unreal Engine Editor: import the new asset from FBX into the Carla Project as an Unreal asset file (static mesh)
- In Unreal Engine Editor: import the custom collision into the physics asset for this specific vehicle so that it will be used for calculating collisions with this vehicle
- Create constraints that connect the different joints and define the physics of all parts
It is worth pointing out that the goal of this repo is to collect all the unreal physics asset files with custom collisions in one place, so other people can use it!
1. Define Custom Collision for Wheels in Unreal Editor
2. Exporting Vehicle to FBX
3-4. Import to Blender and Create Custom Boundary
Below you will find 2 clips that gives an example of this process but before you watch it let me explain a bit about the logic that goes into this process:
If you select the entire car and create the collision boundary with the VHACD Plugin you will get a very ugly approximation and the wheels will contains sharp edges which will mess-up the drive on the road (it will make them collide with the road every time it's not straight). It's important that the wheels have smooth boundaries around them. If you used convex decomposition on the car's body the mirror would still not look right.
For computer vision applications, we need some important features of the vehicles, such as side mirrors, wheels that look right, etc. It is with this in mind that I break down the vehicle into several parts.
Generally the steps are:
- Cut out the bottom parts of the wheels, the side mirrors and the top part of the car's body to create the first boundary using the VHACD tool
- Cut out the bottom half of the car to create the second boundary (top part of the car) using the VHACD tool
- Create separate boundaries for side mirrors using the VHACD tool
Be very careful about naming the objects!! Each boundary should have "UCX_" at the beginning and the rest of the name has to be EXACTLY the same as the original mesh
The full example is shown in these two clips:
5. Export from Blender to FBX
This clip: https://youtu.be/aJPyskYjzWo explains this simple step. Follow it exactly. Select only the original car object + all the newly added objects for collision. Make sure that in the export menu you check "selected objects" and select only "Mesh".
6.-8. Import Custom Collision to Carla Project + Define Physics
The final results:
at the end of the clip you can see the new collision boundaries in gray (only those with really good vision can see it!)
How to Contribute?
Please consider submitting your creations so that everyone can benefit. The process is very simple:
- Create a branch with the name "dev/<user_name>" and upload the new physics asset you created.
- Create a pull request to let us know you submitted it
- We will occasionally test the newly submitted physics assets and copy them to the master branch