Wiki
Clone wikiDeepDriving / DeepDrivingEvaluate
Deep-Driving
Evaluate the Network
Run the Evaluation
A checkpoint can be evaluated against the validation data. This is done by starting the script eval.py
:
cd <repository-path/python/scripts python eval.py
At the end of evaluation, a table is shown, which contains error values for every output signal. Furthermore a comparison with the error-values from the original implementation is provided. The comparison is given as percentage, where 100% means, that the error level was the same like the error on the original implementation. As reference values the values from the PhD Thesis of Chenyi Chen are used (see here).
The pre-trained checkpoint on this repository should lead to the following error values:
Full Summary: ( Error: 16.41, SD: 16.67 ) | Type | Angle | LL | ML | MR | RR | DistLL | DistMM | DistRR | L | M | R | DistL | DistR | Fast | +--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+ | MAE | 0.03 | 0.16 | 0.14 | 0.13 | 0.16 | 3.27 | 3.04 | 3.62 | 0.16 | 0.28 | 0.19 | 2.47 | 2.47 | 0.29 | | SD | 0.08 | 0.36 | 0.39 | 0.40 | 0.36 | 5.91 | 4.82 | 7.10 | 0.48 | 0.82 | 0.52 | 5.81 | 5.91 | 0.41 | +--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+ |MAE/Ref | 95.8% | 82.7% | 87.8% | 84.6% | 88.9% | 64.3% | 64.1% | 45.4% | 52.2% | 90.3% | 64.3% | 27.8% | 22.7% | 98.8% | | SD/Ref | 92.9% | 66.7% | 93.3% | 89.3% | 67.5% | 65.0% | 61.6% | 56.5% | 67.7% | 114.5% | 80.7% | 45.0% | 40.4% | 100.0% |
Error Comparison
The following image visualizes the comparison of the measured error values with the error values from the original implementation. As you can see the pre-trained model achieves on almost all output signals a better error than the original implementation. Only the standard deviation of the value M seems to be some percent higher than in the original project:
Next Step
Updated