M.

Tech Project Report Phase-I

3D VIDEO RECONSTRUCTION FROM MULTIPLE VIEWS:AN FEM PARADIGM

By M.Sudhakar (EE10M04) Adviser: Dr. Soumya Jana Department of Electrical Engineering Co-Adviser: Dr. Raja Gopal Department of Civil Engineering

Indian Institute of Technology Hyderabad

Ordinance Factory Estate , Yeddumailaram, Medak-502205.

Abstract
The aim of the project is to study various 3D surface reconstruction methods and try to develop more generalized and appropriate 3D video reconstruction and motion estimation technique which mainly aims at 3D telepresence system. The source of motivation for choosing this project is from the The possibility of being able to acquire 3D information from 2D images has been a major eld of interest in recent years.We observed that there are many low-level problems to be addressed. We considered representation of 2D topological surface embedded in 3D space is one of the most interesting and challenging problems. So, we have studied various approaches and found that FEM, Space Curving, Shadow Curving and Variational approach are some of the interesting and suitable methods for 3D surface reconstruction. In variational method Euler-Legrange equations are deduced from the variational principle to provide a set of PDEs which potentially provides an ecient and robust way of achieving the surface evolution and to deal automatically with changes in the surface topology during deformation.The approach directly works on 3D Euclidean space based on a level set formulation.In this project we have also considered the problem of 3D video compression. To address this problem we have studied 2d video compression using motion vectors, Finite Element Method and we want to extend this knowledge for 3D video compression.

Contents
1 Introduction 1.1 3D Telepresence System . . . . . . . . . . . . . . . . . . . . . 1.2 Literature Survey . . . . . . . . . . . . . . . . . . . . . . . . 2 Approach to Problem 2.1 Space Curving . . . . . . . . . . . . . . . . 2.1.1 Advantages of Space Curving:- . . . 2.1.2 Limitations of Space Curving:- . . . 2.2 Shadow Curving . . . . . . . . . . . . . . . 2.2.1 Procedure . . . . . . . . . . . . . . 2.3 Variational Approach . . . . . . . . . . . . 2.3.1 Procedure . . . . . . . . . . . . . . 2.3.2 Rened Model . . . . . . . . . . . . 2.3.3 Energy Functional Embedded With 2.4 Finite Element Method Implication . . . . 2.4.1 Hand Motion Analysis . . . . . . . 2.4.2 Relate FEM to Image Processing . 2.5 Compression using Motion Vectors . . . . 2.5.1 Compression in 3D Video . . . . . 3 Future Approach . 2 2 3 5 5 8 8 8 8 9 9 12 13 13 14 15 15 16 17

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Level Set Term: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

Chapter 1 Introduction
In the present world we want the things to be much more innovative and more appealing. This is even more in the eld of communication. Right from the days of sending letters, using birds to convey messages, there is an urge for better communication. Although we are now enjoying the avors of great inventions like Telephone and Television which gave us the power to talk or even see the person who is very far from us, we still feel that we can add much more powerful features to these existing technologies which makes your lives even more easier and comfortable. The main motivation for choosing this project is the necessity for life like representation of the scene or person in the eld of communication and medical image processing. Creating life like representation is nothing but reconstructing the whole 3D model of a scene or person with the help 2D images. This problem is addressed by many people in many ways but it is still an open problem.

1.1

3D Telepresence System

3D Telepresence is nothing but having 3D life like representation of the person while communicating. We are trying to develop a framework for this system which takes multiple views of the object as input and create 3D model of the object. There are many challenges and problems to be addressed. The overall block diagram of the framework is given by

Figure 1.1: Block Diagram of the System Overview As shown in the gure the input to transmitter(Encoder) is the an array of 2D images captured in dierent views covering whole 3600 space around the object(person). In the transmitter(Encoder) 3D reconstruction of the object(person or scene) is done from 2D images captured in multiple views. The reconstructed 3D image is compressed and transmitted over the communication channel. The receiver decodes and displays rendered 3D model in the 3D-Display. One of the major challenges we observed is signal representation. Intuitively the amount of data which represents the 3D object will be very high compared to that of single 2D image. So we have to have spare representation of the signal to achieve real time transmission and reception. One interesting method of representation we come across is Finite element method.To avoid temporal redundancy in 3D video we apply motion estimation concept which are using in 2D video.

1.2

Literature Survey

During the past decades many methods have been proposed to reconstruct the 3D shape of objects from images.Among these some are based on the apparent contours to compute the visual hull[4].Space carving was also proposed in[4][5] where photo-consistency is used to carve voxels.In Space curving we have the disadvantage that we cannot capture the concavity of the object. The shadow curving is the renement to the space curving[6].The shadow curving is the more time consuming problem because of the number of iterations needed for capture concavity.Even though we come across this 3

process, there is no guarantee for capture the concavity in a better fashion. In this process the 3D reconstruction from arbitrarily distributed camera cluster which will produces partial 3D meshes, instead of dense point clouds, which are combined on the renderer to create a unied model of the user[9].In this method there is no mentioning of the camera positions and there is no mathematical justication for this method.The problem of curve evolution driven by a PDE has been recently studied both from the theoretical standpoint and from the viewpoint of implementation with the development of level set methods that can eciently and robustly solve those PDEs. A nice recent exposition of the level set methods and of many of their applications can be found in [3]. The path we will follow to attack the stereo problem from that angle is a variational one. we will compute the Euler-Lagrange equations of this functional, thereby obtaining a set of necessary conditions, in eect a set of partial differential equations,which we will solve as a time evolution problem by a level set method.And use the mesh mechanism [9] for represent the surface with minimum data.For compression we approach the motion estimation criteria.

Chapter 2 Approach to Problem

In this chapter we describe how we approached various methods and how we arrived towards more generalized framework of the problem.In this process we come across method i.e.,Space Curving,Shadow Curving,Variational approach.Space Curving have the grid in which we will curve the space for getting the shape of object.Each element in grid called voxel.The voxel size judge the eciency of reconstruction.The inability to capture concavity of Space curving we will go for shadow curving.Here we will curve the shadow in iterative process, it is time consuming one.Now we will go for variational approach where we nd the fuction for surface by appling Level Set Method to Euler-Legranges PDEs which are evolved from variational principles of data.After this we try to resolve problem of compression in Finite Element Method point of view, where the surface can be represent by interpolative basis or mesh nodes.After we acheiving this our view is to apply the motion vectors to the FEM modelled images to achieve better compression for tele presence application.

2.1

Space Curving

Space curving is a method that generates a texture-mapped threedimensional model of a scene computed from multiview calibrated two-dimensional photographs. In this framework the reconstruction is represented using an array of voxels, and each voxel is assigned a probability of it existing in the model[4][5].

The procedure for shadow curving involves: In this reconstruction we initially form a grid .For better reconstruction we have select the grid in such a way that the voxel size is small. We will form a silhouette for one image.The silhouette is projected onto the voxel array. Any voxels that lie outside the silhouette are carved away, leaving only points inside the model[4]. Using just one camera, we end up with a dinosaur-prism a single camera provides no information on depth. Adding more views renes the shape.As number of views increase the shape will be more rened.

SPACE CURVING FOR DINOSAR ARTIFACT

image of dragon from 36 views

silhoutte of one image

camera arrangement

Result of one curving

Result after 3 curvings

Result after 36 curvings after renement after colour The above demo gives the felling of space curving with number of 2D views.It is taken from http://mathworks.com. 7

2.1.1

Advantages of Space Curving:-

Robust and simple Wide changes with viewpoints cope with occlusion

2.1.2

Limitations of Space Curving:-

Need of multiple camera for good modeling. Some Concavities are cannot be modeling

2.2

Shadow Curving

As we discussed in previous section the space curving has the disadvantage that it cannot capture the concavities.Self shadows are further visual cues.Self shadow indicates concavities.To overcome this we will adopt the procedure that itervatively curve the shadow. While we know the object and where the concavity is present , we will focus on the area such a way we will curve it.In this we will focus the light on the shadow area, then we will take the image such a way we will curve the area that is intersection of both light and camera projection[6].In second iteration we will rotate the light and do the same procedure such a will we will curve the shadow of the object for better reconstruction.

2.2.1

Procedure

Upper bound known from space curving as prior estimate. Here we curve shadow for better approximation of object. As the light source moves new carvable areas are removed.

In above diagrams we shown the underlying procedure of shadow curving. It has the disadvantage that even though we curve object for number of iterations, we will not get accurate reconstruction. 8

2.3

Variational Approach

The idea that is put forward is that the methods of curve and surface evolutions which have been developed in computer vision under the name of snakes, in the context of PDE driven evolving curves can be used eectively for solving 3D vision problems such as stereo and motion analysis. It is based up on the denition of a variational principle that must be satised by the surfaces of the objects in the scene and their images. The EulerLagrange equations which are deduced from the variational principle provide a set of PDEs which are used to deform an initial set of surfaces which then move towards the object to be detected. The Level Set implementation of these PDEs potentially provides an ecient and robust way of achieving the surface evolution and to deal automatically with changes in the surface topology during the deformation.

2.3.1

Procedure

Function for nding matching The establishing correspondence between two views can be done by matching function.Write matching function for pixel i ,nd maxima in another image at pixel j ,Then 3D can form by intersecting corresponding optical rays[3] < mi , ci >,i=1,2,3,..n

To know more about it we should learn about the image formation. Image Formation We will assume here that the cameras perform a perspective projection of the 3Dworld on the retinal plane as shown in Figure. The optical center, noted C in the gure, is the center of projection and the image of the 3D point M is the pixel m at the intersection of the optical ray < C, m > and the retinal plane R.As described in many recent papers in computer vision, this 9

operation can be conveniently described in projective geometry by a matrix operation. The projective co ordinates of the pixel m (a 3 x 1 vector) are obtained by applying a3 x 4 matrix P1 to the projective coordinates of the 3D point M (a 4 x 1 vector)[8][1].

Fig:(a)The focal plane(x, y) is parellel to the retinal plane (x1 , y1 ) and at a distance of 1 from it This matrix is called the perspective projection matrix. If we express the matrix P1 in the coordinate system (C , x, y , z ) shown in the Fig. , it then takes a very simple form: P1 = [I3 0] ,where I3 is the 3 x 3 identity matrix.If we now move the camera by applying to it a rigid transformation described by the rotation R and the translation vector t, the expression of the matrix P changes accordingly and becomes:P2 = [RT RT t] . With these preliminaries in mind we are ready to proceed .Assume objects in image are modeled as the graph of an unknown smooth function z = f (x, y ). A point M of co-ordinates [x, y, f (x, y)]T is seen as two pixels m1 and m2 whose co-ordinates(gi (x, y), hi (x, y)), i = 1, 2, can be easily computed as functions ofx, y, f (x, y)and the coecients of the perspective projection matrices P1 and P2 [1]. The objects are perfectly Lambertian, we must have I1 (m1 ) = I2 (m2 ) for all pixels in correspondence, i.e. which are the images of the same 3D point.This reasoning immediately leads to the variational problem of nding a suitable function f dened, to be rigorous, over an op en subset of the focal plane of the rst camera which minimizes the following integral: C1 (f ) = (((I1 (m1 (x; y)) I2 (m2 (x; y)))2 dxdy (2.1)

Our rst variational problem is thus to nd a function f in some suitable functional space that minimizes the error measure C1 (f ). The corresponding Euler-Lagrange equation is readily obtained[1][3]: m2 m1 I2 )=0 (2.2) (I1 I2 )( I1 f f The values of m1 and m2 are functions of f which are easily computed. The f f terms involving I1 and I2 are computed from the images. 10

Stratagies for solving One standard strategy is to consider that the function f is also a function f (x, y, t) of time and to solve the following PDE ft = (f ) = (I1 I2 )( I1 m1 f I2 m2 ) f (2.3)

with some initial condition f (x, y, 0) = f0 (x, y). We thus see appear for the rst time the idea that the shape of the objects in the scene, described by the function f , is obtained by allowing a surface of equation z = f (x, y, t) to evolve over time, starting from some initial conguration z = f (x, y, 0), according to some PDE, to hopefully converge toward the real shape of the objects in the scene when time goes to innity. If you add regularization term to C1 (f ) we cannot get converge. so, we will go to other approach. Level Set implementation of PDE The Level Set implementation of this PDE s provide ecient , robust way to deal with surfaces,surfaces with deformation. Consider the family of surfaces S dened by S(x, y, t) = [x, y, f (x, y, t)]T . The parameters x and y are used to parameterize the surface, t is the time. The unit normal to this surface is the vectorN = 1 2 [ f T , 1]T , the velocity vector is St = [0, 0, ft ]T and
1+ f

hence the evolution of the surface can be written [1] St = (f ) (1 + f 2) N (2.4)

This expression of the evolution of the surface directly leads to a straight forward application of the level set methods. Consider a function u(x, y, z, t) whose zero level set is the surface S , i.e. at each time instant t, the set of points (x, y , z )such that u(x, y, z, t) = 0 is identical to the surface S . Note that the function u can b e considered a temporal sequence of volumetric images. Two subtle points here (i)The LSM designed for closed surfaces.here the graph is not closed.(ii) | u|is dened for on s not for whole (x,y,z).But the term needed for all points. Better function for matching: we can extend C1 (f ) (i) dierence can be replace by correlation,because is images formed by fronto plane to surface of object(move along the matching criteria). (ii) relax the hypothesis & take into account the orientation of tangent plane 11

to the surface of object(move along the matching & shape criteria complexity axis).

2.3.2

Rened Model

In previous case we taken the object as function of x&y,but here it will take u(x, y, z) = 0 for smooth reconstruction.It leads naturally to implement surface by level set method. Let us consider a family of smooth surfaces S :(v, w, t) S(v, w, t) where(v, w) parameterize the surface and t is the time. It is in general not possible to nd a single mapping S from R2 to R3 that describes the entire surface of the objects but we do not have to worry about this since our result will in fact be independent of the parametrization we choose. The objects in the scene correspond to a surface S(v, w) and our goal is, starting from an initial surface S0 (v, w), to derive apartial dierential equation[1][3] St = N (2.5)

where N is the inner unit normal to the surface, which, when solved with initial conditions S(v, w, 0) = S0 (v, w), will yield a solution that closely approximatesS(v, w).An intresting point is the above evolution equation can be solved using the level set method which has the advantage of coping automatically with several objects in the scene.In detail, the surface S are at each time instant the zero level sets of function u :R4 R: u(S, t) = 0 u u ,N = N = | u| | u| Error measure becomes
n

(2.6) (2.7)

C(S, N ) =

(S, N, v, w)d =
i,j=1,i=j

1 < Ii , Ij > |Ii ||Ij |

(2.8) (2.9)

d = |Su Sw |dvdw = h(v, w)dvdw

Su Sw d plays the role of dxdy in our previous analysis, S that of f , and N = |Su Sw | ,the unit normal vector to the surface S , that of f . Note (X, N ) the value of at the point X of R3 and the point N of S2 .Let us now consider the folowing error measure:

C(S, Sv , Sw ) =

(S(u, w), N (v, w))h(v, w)dvdw

(2.10)

Where the intergral is taken over the surface S. 12

2.3.3

Energy Functional Embedded With Level Set Term:

we seek zero level set of that minimizes the energy functional: E() =

(X, N )()| |dXdY dZ +

()| |dXdY dZ

(2.11)

Having represented the object to be reconstructed as a level set from the beginning, the integral can however be done on the whole 3D space by introducing the term ()| | in the integrand[2]. The Euler-Lagrange equation for of the functional can be shown equal to E = = ()( N + k + k ) t (2.12)

1 In our approach we use a reguralized form of () is () = 2 +2 Using this approximation, the algorithm has the tendency to compute a global minimizer. One of the reasons is that, the Euler-Lagrange equation acts only locally, on a few level surfaces around = 0 using the original Dirac function, while by the regularized form, the equation acts on all level sets[1][2].The second term is smoothness term.

=
1

E = = ()( N + k + k ) t Ij Ii ) + (Ij Ij )( ) ij = (Ii Ii )( X X m,n

(2.13) (2.14)

2.4

Finite Element Method Implication

The FEM is a numerical procedure for obtaining approximate solutions to many of the problems encountered in engineering analysis.FEM use interpolative basis to reconstruct the total image. Here we see a problem which is solved using the FEM. Given an initial model representing general knowledge of the object, and incomplete or missing information about geometry or material properties. The method is based on iterative analysis of the dierence between the actual and predicted behaviour.Large dierences indicate that an objects properties are not captured properly by the model describing it. These error are due to aws in the model parameter estimation such as geometry and material properties. P is sparse points, Q is set of correspondence[7] of guide search.

13

Figure 2.1: General approach

2.4.1

Hand Motion Analysis

Since large deformations occur during any hand motion, physical laws should be used to model not only the skeletal motion, but also the nature of the deformations in soft tissues. Research has shown that nite element theory can be used to model near-correct muscular motion.Frame-to-frame correspondence recovery is based on the iterative analysis of the directed Hausdor distance between the model and the next frame in the sequence.Graphically interpreting nonlinear behavior through animation allows us to verify and visualize displacement results. An obvious high pressure around the base of the thumb is readily visible.

(a) Motion model of hand(left and right) and its analysis

14

2.4.2

Relate FEM to Image Processing

FEM has many applications in the eld of image processing. Many parts of the body will be reconstucted as 3D models with the help FEM from the slices of 2D imges like CT-Scan, MRI-Scan etc.FEM model representation of suface is nothing but interpolative basis selection, where we select the node points to represent total object.For transmission of 3D data we need more bandwidth.In real world it is not possible to dedicate more bandwidth for single user.So, we do compression of this 3D data by modeling using FEM.We use the mesh nodes for represent surface,from which we will reconstruct the total object.Any point on 3D surface can be represent by using the function of nodes points.My teammate Mr.Srikanth is also subjecting the same problem with the help of compressive sensing framework.Our objective is to select the node points in such way that we will reconstruct the object with better representation.In the above hand model we disscussed about the changing the properties by using the motion of hand[7].But out problem is to change the node points such way that we will get the good representation of object.

2.5

Compression using Motion Vectors

In video editing motion vectors are used to compress video by storing the changes to an image from one frame to the next. In a video sequences exhibit very close similarity, except for the fact that the objects or the parts of a frame in general may get somewhat displaced in position. This assumption is mostly valid except for the frames having signicant change of contents. The motion estimation block in a video codec computes the displacement between the current frame and a stored past frame that is used as the reference. We consider a pixel belonging to the current frame, in association with its neighborhood as the candidates block. The dierence in position between the candidates and its match in the reference frame is dened as the motion vector.The motion vector is stored in the place of candidate block with better prediction.The dierence betweeen predicted block and the candidate block called prediction error.While we are coding we will code the only prediction error instead of total candidate block.At reconstruction we will get back the candidate block from the reference frame(Intraframe) and the prediction error[10]. The video sequence consists of Intraframe coded pictures(I-pictures) Interframe predicted pictures(P-pictures) 15

 Bi-directionally predicted pictures(B-pictures)

P frame formed from I . B formed from I , P

2.5.1

Compression in 3D Video

The compression in 3D video can be done by incorporating motion estimation for 3D frames.The motion estimation of 3D video is gaining momentum because of the advancement in 3D technologies.Here we proposed a idea to nd the motion vectors for 3D video. Here we have the 3D mesh model.Here we want to nd the node displacement instead of block displacement.By incorporating this node displacement we make less computation, more compression.And we have to take care of the occluded points in the ob-

3D encoder 3D decoder ject. The Encoder uses the 3D frames.The 3D frames are FEM modeled,which can be reconstruct from the nodes.We will nd the motion estimation from the reference frame to candidate frame by estimating the motion of nodes. We have the Decoder running internally in Encoder.Because to calculate the motion vectors from the encoded 3D surface.The 3D surface which is formed by using the motion vector based on FEM models gives the better compression.For nding matching we will go for one of existed matching methods [10].The reconstruction of 3D video in decoder is done by adding the motion vectors to the nodes in reference frame.

16

Chapter 3 Future Approach

In report we presented the approaches to reconstruct the 3D surface of an object from a sequence of calibrated images.We presented the different methods for 3D reconstruction .i.e., Space curving, Shadow curving,variational approach.Among these space curving provide you a appropriate reconstruction if the voxel should be low and the number of image taken for curving is more.The shadow curving which is developed to overcome the incapability of Space curving is give a good solution.The design of the variational principle allows us to clearly incorporate the hypothesis we make about the objects in the scene and how we obtain correspondence between image points.The Euler-Legrange equation which are deduced from the variational principle provide a set of PDEs which are used to deform an initial set of surfaces which then move towards the object to be detected.The level set implementation of these PDEs provides an ecient and robust way of achieving the surface evolution and to deal automatically with changes in the surface topology during the deformation.If we represent the object in the level set form we can incorporate any priori information in the energy functional in order to guide the surface evolution. Our future approach is to nd the better function which will produce the surface with less computation by using the level set implementation on sparse set of data.For obtain the sparse set of data we have to nd the constraints on camera position, which we can reduce the more redundancy.And then by using the interpolative basis (FEM) we are going to represent object.FEM is the method which is successfully implemented in many medical application.FEM gives the better representation of object with less number of points or interpolative basis or mesh.Then by using the motion estimation of these models we are going to give the better way of compression.The motion vectors for FEM models is nothing but the movement of the node points.By 17

tracking the node points movement we can get the better representation with less amount of data.By combing all this various techniques we are going to implement a robust algorithm for real time application.Such a way we can made 3D virtual environment in real life.

18

Bibliography
[1] Olivier Faugeras,Renaud Keriven,Complete Dense Stereovision using Level Set Methods,ECCV- Volume I ,1998 [2] El-Melegy, M.T.; Al-Ashwal, N.H.; , A variational technique for 3D reconstruction from multiple views, Computer Engineering and Systems, 2007. ICCES 07. International Conference on , vol., no., pp.38-43, 27-29 Nov. 2007 [3] Faugeras, O.; Keriven, R.; , Variational principles, surface evolution, PDEs, level set methods and the stereo problem, Biomedical Imaging, 2002. 5th IEEE EMBS International Summer School on , vol., no., pp. 83 pp., 15-23 June 2002 [4] A. W. Fitzgibbon, G. Cross, and A. Zisserman,3D Structure from Multiple Images of Large-Scale Environments, Springer LNCS 1506, pages 155170, 1998 [5] Kutulakos, K.N.; Seitz, S.M.; , A theory of shape by space carving, Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on , vol.1, no., pp.307-314 vol.1, 1999 [6] Silvio Savarese , Marco Andreetto , Holly Rushmeier , Fausto Bernardini , Pietro Perona,3d reconstruction by shadow carving: Theory and practical evaluation ,International Journal of Computer Vision 71(3), 305336, 2007International Journal of Computer Vision 71(3), 305336, 2007 [7] Tsap, L.V.; Goldgof, D.B.; Sarkar, S.; , Nonrigid motion analysis based on dynamic renement of nite element models, Computer Vision and Pattern Recognition, 1998. Proceedings. 1998 IEEE Computer Society Conference on , vol., no., pp.728-734, 23-25 Jun 1998 [8] Zhengyou Zhang; , Flexible camera calibration by viewing a plane from unknown orientations, Computer Vision, 1999. The Proceedings of the

19

Seventh IEEE International Conference on , vol.1, no., pp.666-673 vol.1, 1999 [9] Kurillo, G. Vasudevan, R. Lobaton, E. Bajcsy, R. , A Framework for Collaborative Real-Time 3D Teleimmersion in a Geographically Distributed Environment, Multimedia, 2008. ISM 2008. Tenth IEEE International Symposium on , vol., no., pp.111-118, 15-17 Dec. 2008 [10] Yu-Cheng Fan; Shu-Fen Wu; Bing-Lian Lin; , Three-Dimensional Depth Map Motion Estimation and Compensation for 3D Video Compression, Magnetics, IEEE Transactions on , vol.47, no.3, pp.691-695, March 2011

20