|
Several projects described in this article have synchronized multiple video streams as input. Therefore, in order to correctly simulate the imaging process of the cameras in a computer, a mathematical camera model is required. Such formulation is detailed later, where the correspondence between a real camera and its computational equivalent is presented. After that, we briefly describe the process of camera calibration and the imaging geometry of stereo cameras. Although the projects in this article are usually tailored to human actors, the fundamental principles described here can also be applied to a larger class of real-world subjects, like animals.
This description concludes with a description of important computer vision algorithms employed by several projects proposed in this article. In particular, we briefly describe how to perform background subtraction, and how to calculate optical flow, scene flow and SIFT features. The information contained in a 3D scene can be captured to a 2D image plane by a camera as follows: first, a lens collects the incident illumination. Afterwards, the light rays are deflected towards a focal point, and at the end, the deflected rays create an image of the observed scene. In the following, we will describe the mathematical framework formapping the 3D world space to the 2D image plane, the process of camera calibration and the geometry of stereo cameras. The matrix K can be referred to as the calibration matrix and its entries are called the intrinsic parameters of the camera. The principal point in the image plane is at position at the intersection of the optical axis with the image plane. The coefficients represent the focal length of the camera in terms of pixel dimensions in x and y directions, respectively. The focal length f of the camera, and mx and my represent the number of pixels per unit distance in image coordinates in x and y respectively. Therefore, a real-world camera can be represented by ten parameters.
However, unfortunately the physical properties of lenses make the previous image formation process geometrically deviate from the ideal pinhole model. Geometric deviations are typically caused by radial or tangential distortion artifacts. Radial distortion happens since a real lens bents light rays towards the optical center by more or less than the ideal amount. Tangential distortion are caused by the bad alignment of the individual lenses in an optical system with respect to the overall optical axis. The imaging process of a real-world camera is simulated by a mathematical camera model. Camera calibration is the process of determining the parameters of the mathematical model that optimally reflect the geometric and photometric imaging properties of the real camera. The most important calibration step is the geometric calibration, where the parameters of the imaging model are estimated. The majority of the calibration algorithms take into account radial and tangential lens distortions and derive these parameters from images of a calibration object with known physical dimensions, such as a checkerboard pattern. The parameters are estimated by means of an optimization procedure that modifies the model parameters until the predicted appearance of the calibration object optimally aligns with the captured images.
Additionally, color calibration can be applied to ensure correct color reproduction under a given illumination setup. White balancing is the simplest color calibration procedure that computes multiplicative scaling factors from an image of a purely white or gray object. We also developed a more sophisticated relative calibration procedure that assure color-consistency across the cameras.