Changing point selection while tracking selected points

This might be intentional, but it has been a bit annoying for me so I thought I'd bring it up.

When I'm tracking selected points, I occasionally accidentally press up or down on the arrow keys and it switches to tracking the newly selected point. I would expect changing the selection while tracking is in progress to either a) stop tracking, or b) not stop tracking, but not change the points being tracked either.