A team of researchers has developed a new software tool called "Side Eye" that can extract sound and reconstruct its waves from silent videos and even photos. According to the study, the tool is based on a machine learning system that can identify the gender of the speaker in any silent picture or video and determine the words being spoken in the pictured context.
According to a professor of electrical and computer engineering and computer science at Northeastern University, obtaining sound from images and even silent videos has become possible using the new tool. He explained that the speaker’s voice and its waves cause light waves and rays to bend and break. Therefore, if, during the photography process, the camera lens is able to capture a large enough amount of data from this light change, it becomes possible to reconstruct the sound waves and identify the speaker’s gender. Sometimes, it even becomes possible to accurately determine the identity.
The tool uses image stabilization technology to monitor changes in the sound waves affecting imaging sensors during the photo or video capture process.