1. Andre Netzeband Avatar Andre Netzeband
  2. Untitled project
  3. DeepDriving

Wiki

Clone wiki

DeepDriving / DeepDrivingEvaluate

View History

Deep-Driving

Evaluate the Network

Run the Evaluation

A checkpoint can be evaluated against the validation data. This is done by starting the script eval.py:

cd <repository-path/python/scripts

python eval.py

At the end of evaluation, a table is shown, which contains error values for every output signal. Furthermore a comparison with the error-values from the original implementation is provided. The comparison is given as percentage, where 100% means, that the error level was the same like the error on the original implementation. As reference values the values from the PhD Thesis of Chenyi Chen are used (see here).

The pre-trained checkpoint on this repository should lead to the following error values:

Full Summary:  ( Error: 16.41,  SD: 16.67 )

|  Type  | Angle  |   LL   |   ML   |   MR   |   RR   | DistLL | DistMM | DistRR |   L    |   M    |   R    | DistL  | DistR  |  Fast  |
+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+
|  MAE   |   0.03 |   0.16 |   0.14 |   0.13 |   0.16 |   3.27 |   3.04 |   3.62 |   0.16 |   0.28 |   0.19 |   2.47 |   2.47 |   0.29 |
|   SD   |   0.08 |   0.36 |   0.39 |   0.40 |   0.36 |   5.91 |   4.82 |   7.10 |   0.48 |   0.82 |   0.52 |   5.81 |   5.91 |   0.41 |
+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+
|MAE/Ref |  95.8% |  82.7% |  87.8% |  84.6% |  88.9% |  64.3% |  64.1% |  45.4% |  52.2% |  90.3% |  64.3% |  27.8% |  22.7% |  98.8% |
| SD/Ref |  92.9% |  66.7% |  93.3% |  89.3% |  67.5% |  65.0% |  61.6% |  56.5% |  67.7% | 114.5% |  80.7% |  45.0% |  40.4% | 100.0% |

Error Comparison

The following image visualizes the comparison of the measured error values with the error values from the original implementation. As you can see the pre-trained model achieves on almost all output signals a better error than the original implementation. Only the standard deviation of the value M seems to be some percent higher than in the original project:

Error Values

Next Step

Updated