First, an example ghost marker that quickly pops in and out of existence:

The marker appears to have been inferred from the Back Marker that the upper-right ray passes through, and the Shin Marker I've selected here:

This problem doesn't just create garbage data, it has effectively stolen 1 camera's worth of data from both the Back Marker, and Shin Marker:

The Back Marker was actually picked up on 4 cameras, and the Shin Marker on 3 Cameras.
A possible solution would be to take any ray that passes through another marker to reach its destination, and instead contribute it to the marker it passed through. This would remove any "confirmation" of the ghost marker's existence, and allow the remaining ghost-contributing ray to be used for good, instead of evil.

Edit: I do understand I could require 3+ cameras for marker validation, but in a 6 camera setup, occlusion would become a larger problem.