This is one major advantage to Apple sharing a foundation across all their devices. Vision Pro introduced eye tracking to their systems as a new input modality, and now it trickles down to their other platforms.
I am surprised CarPlay didn’t have voice control before this though.
Is this really likely to be downstream of the Vision Pro implementation? I would think that eye-tracking with specialized hardware at a fixed position very close to the eye is very different to doing it at a distance with a general purpose front facing camera.
Typically eye-trackers work by illuminating the eyes with near infrared light and using infrared cameras. This creates a higher contrast image of the pupils, etc. I assume Apple is doing this in the Vision Pro. Eye-tracking can also be done with just visible light, though. Apple has the benefit of knowing where all the user interface elements are on screen, so eye-tracking in this on the iPhone or iPad doesn't need to be high precision. Knowledge of the position of the items can help to reduce the uncertainty of what is being fixated on.
So there isn’t much more to it than getting a good resolution image of the eye from any distance, every millisecond.
Precision is the issue because we are mostly moving our eyes in about 8 directions, there’s no precision because we don’t know how to measure focusing of our eye lens with a camera yet (unless that too is just a matter of getting a picture).
Squinting would be the closest thing to physically expressing focusing. So the camera needs to know I’m looking left with my eye, followed by a squint to achieve precision. Seems stressful though.
Gonna need AI just to do noise cancelling of involuntary things your eyes do like pupil dilation, blinking.
Not the hardware side, but the software side. Implementing all the various behaviours and security models for the eye tracking and baking it into SwiftUI, means that it translates over easier once they figure out the hardware aspect. But the iPads and iPhones with FaceID have had eye tracking capabilities for a while, just not useful in UI.
CarPlay devices (car components) are essentially playing a streaming video of a hidden display generated by the phone. CarPlay also lets those devices send back touch events to trigger buttons and other interactions. Very little process is done on the vehicle.
BTW if you are plugged in to CarPlay and take a screen shot, it will include the hidden CarPlay screen.
CarPlay is rendered by the phone itself, so it's not strictly a function of how powerful the car infotainment is. You've been able to talk to Siri since the beginning of CarPlay so additional voice control is really just an accessibility thing
Some cars already have a voice control button on the wheel for their existing system which, if done correctly, is overriden by Siri+CarPlay. Which is really nice when it works.
I’m able to control carplay by just saying “Hey Siri.” Siri’s abilities tend to fluctuate based on what Apple is doing on the server, the phase of the moon and whether I remembered to sacrifice a chicken that morning, but otherwise, it seems to work fine.
I am surprised CarPlay didn’t have voice control before this though.