I like the idea, and if these pruned false positives are rare enough I think it would be a great safety measure - you could even have the driver verbally label things, which might have both training (in the AI sense) and awareness (for the humans) benefits (https://en.wikipedia.org/wiki/Pointing_and_calling).
Note that "rare enough" needn't be all that rare, just that if it is multiple every second or something it clearly wouldn't be practical.
I'm not sure how well I would respond to a sudden alarm when everything has been deemed safe until a moment ago. I'd potentially need several seconds to figure out what is going on.
I would hazard a guess that an approach that might be more fruitful be to have a meter that dynamically adjusts the "riskiness" of the situation between a yellow, orange, and red zone, so that the driver (a) has to pay attention constantly, (b) gets information constantly, and hence (c) has a better chance to react earlier. I know if I see a "danger meter" getting into the orange zone, I'm not going to wait until it goes into the red before I start paying attention.
"I like the idea" doesn't mean I will like any given implementation of the idea - I certainly agree that there's plenty of room to get the interface wrong.
I want the driver paying attention constantly, but I want them paying attention to the environment, not a meter. A periodic "things are unusually interesting, what's going on?" query seems like a way to motivate that. But any actual attempt at a solution should be validated in testing...
I like the idea, and if these pruned false positives are rare enough I think it would be a great safety measure - you could even have the driver verbally label things, which might have both training (in the AI sense) and awareness (for the humans) benefits (https://en.wikipedia.org/wiki/Pointing_and_calling).
Note that "rare enough" needn't be all that rare, just that if it is multiple every second or something it clearly wouldn't be practical.