The Microsoft HoloLens viewer is a device based on mixed reality that has the ability to trace the real space around itself, translating it into virtual space.
This allows in fact to make virtual objects interact with the real world, improving the immersion sensation of the wearer. By developing applications for HoloLens, spatial mapping can therefore be exploited and, through the use of specific objects, or through the Windows Device Portal, one can take advantage of the reconstruction of the environment. This reconstruction, however, is coarse, being constituted by a triangular mesh with vertices too distant to be useful in many applications. There is therefore the need, in order to develop applications that take full advantage of the spatial mapping capability, to be able to derive a more accurate reconstruction. However, this is not possible without access to sensor data. With the update of Windows 10 of April 2018 it was possible to use the HoloLens device in Research Mode, which is a mode that allows access to sensor data. The main problem of Research Mode, however, is to be a modality reserved for researchers. It can not therefore be used to develop applications for the public, but it is a useful tool to be exploited in professional contexts.
The project aims to exploit sensor data to reconstruct a more accurate 3D image of the real world than can be obtained without them. This is made possible through the Recording tool of the HoloLensForCV project, which, loaded on the device and made to run, records a video from the four environmental understanding cameras. Then connecting the viewer to a computer we can download the frames of these videos through the recorder console program, available in the HoloLens for CV project.
Once the frames are recovered, the images of the two frontal cameras (right and left) are used to derive a 3D image. To do this, first of all we need to obtain some specific parameters of the camera through a process called calibration. With the information obtained we can then proceed to rectification, acting on the images to correct the distortions and make them suitable for 3D reconstruction. After doing this we build the disparity map and then the 3D image.
At the end we can have a 3D image taken from two images of stereo cameras, which is an alternative reconstruction to the one already available on the device.