|
Another important component contributing to a realistic look of a virtual human is the surface texture. A possible way to reproduce the appearance of a real-world actor is to reconstruct a consistent surface texture from images showing the subject. However, a static texture cannot reproduce dynamic details, such as wrinkles in the apparel. In our projects, we use dynamic surface textures that incorporate such timevarying details. The multiple video streams are recorded in our studio by cameras providing high frame-rates, high resolution and precise color reproduction. Therefore, realistic virtual actors are generated by combining the multiple synchronized footage with the model's pose at each particular frame. The computational equivalent of the human skeleton is a kinematic skeleton, that mathematically models a hierarchical arrangement of joints and interconnecting bones. The human skeleton is usually approximated by a collection of kinematic sub-chains, where the relative orientation between one segment and the subsequent one in the hierarchy is controlled via a rigid body transformation. It jointly describes a rotational and a translational transformation between the local coordinate frames of adjacent rigid bodies.
The translational components of the rigid body transformations are implicitly represented by the bone lengths and the joints model the rotational components. Since the bone lengths are constant, the pose of the skeleton is fully-specified by the rotation parameters for each joint and an additional translational parameter for the root. A method to robustly segment a person in the foreground of a scene from the background is necessary. Due to its robustness, we decided to use the color-based method originally proposed in, which incorporates an additional criterion to prevent shadows from being erroneously classified as part of the scene foreground. The technique employs per-pixel color statistics for each background pixel that is represented by a mean image and a standard-deviation image, with each pixel value being a three-vector comprising all three color channels. The statistics is generated from consecutive input image frames of the background scene without an object in the foreground, in order to incorporate the pixel intensity variations due to noise and natural illumination changes. The background subtraction method classifies an image pixel p(px, py) as foreground if the color of p(px, py) differs in at least one RGB channel by more than an upper threshold Tu from the background distribution. If this difference is smaller than the lower threshold Tl in all channels, it is classified as a background pixel. All pixels which fall in between these thresholds are possibly in shadow areas and can be classified depending on the amount of variation in the hue value. The difference in hue can be calculated.
Optical flow is the projection of the 3D motion field of a real-world dynamic scene into the 2D image plane of a recording camera. Algorithms used to calculate optical flow attempt to find correlations between adjacent frames, generating a vector field showing where each pixel or region in one frame moved to in the next frame. In computer vision, a number of simplifying assumptions are usually made to compute the optical flow from the pixel intensities of two consecutive images. The basic assumption is that the pixel intensity does not significantly change between two subsequent frames, the so-called intensity constancy constraint. In order to make the problem well-posed, additional assumptions need to be made about the smoothness of the optical flow field in a local spatial neighborhood. In the differential optical flow approach by Lucas and Kanade, the flow is assumed to be constant in a small neighborhood. Alternatively, a hierarchical variant can be employed that incorporates flow estimates from multiple levels of an image pyramid into its final result. Using the same basic formulation, a large number of algorithms have been proposed in the literature. In general, they are based on the Horn-Schunck model, that additionally uses a global smoothness constraint to regularize the optical flow computation. An example is the dense optical flow method by Black. The approach is based on a statistical framework that enables the robust estimation of flow fields addressing violations of the intensity constancy and spatial smoothness assumptions. As a result, the method is able to deal with discontinuities in the flow field. Recently, Brox proposed a multiresolution warping-based method for dense optical flow that uses a continuous, rotationally invariant energy functional. The energy functional E is composed by a weighted sum between a data term ED and a smoothing term ES. By this means, the overall energy functional becomes more robust against intensity value changes.
The smoothness term ES takes into account neighboring information to improve the calculation of the flow field by penalizing its total variation. The global minimum solution is found via a multiscale approach. One starts by solving a coarse, smoothed version of the problem. Thereafter, the coarse solution is used as initialization for solving a refined version of the problem until step by step the original problem is solved. Additionally, the energy functional E is designed using the non-linearized data terms and linearizations are computed during the numerical scheme used to solve it. By this means, the overall method improves the convergence of the solution to the global minimum, generating more accurate results.