Virtual Set Extensions Tutorial

This tutorial is about extending an existing set with a virtual set created with 3d computer graphics. It is the written procedure I wrote after working on such a project. It may also have relevance to any project integrating live action and cgi. See the references link at the end for another tutorial on combining cgi with a photo. Camera mapping is a related technique for creating textured 3d objects quickly from a live action photograph which allows some minor virtual camera movement. A link to a tutorial on camera mapping can also be found at the end of this tutorial.

Planning
What resolution will the project be rendered at? This has implications for both modeling detail and the resolution required for photographic textures. How close will the camera get to the virtual set elements? Is it a locked off shot? If it isn’t you may be able to use camera matching software to match the camera position and focal length. What will the virtual set consist of – what physical objects need virtual duplication and what virtual objects need to be created anew?

At the shoot / On the set
Take a measure devices, lighting reference sphere, reflection sphere, camera - digital and film, pen and paper. Make sure you are working with the final set as it will be shot before following these procedures.

Measurements
Get as many dimensions of the set as practicable. You will need measuements of reference objects that will be modeled and used to match the virtual camera's position and focal length to the plate. Also take measurements so reference objects can be position relative to each other in the scene. These reference objects don't necesarily need to be ones that will be visible in the final render. Try to get measurements of any straight edged objects that cover a long distance in the frame aligning these objects will give good camera position acuracy. Try to get some measurements of straight objects that go from close to far from the camera. This will help match the focal length especialy with wide angled lens distortion. Take measurements of any of the practical elements that will have virtual duplicates or replacements. Measure the camera's position relative to the reference objects so the virtual camera can be positioned - you can match by eye using your reference objects but you might not get a chance to record the camera focal length so could be useful.

Camera settings
Take note of focal length and details so you can match the depth of field the camera is set at for the final plate.

Reference photos
Take some set reference photos from a lot of different angles with a f/l set to minimize wide angle lens distortion. Take photos for references of geometry that you don’t have measurements for - try to include a scale next to the objects so you can later work out the dimensions and position relative to other objects. Try to get some at right angles so that you can see the ratio between the vertical and horizontal aspects without foreshortening.

Texture photos
For textures take photos with the lens at right angles to the surface and make sure the resolution is at least double that of the final size as it will appear at its closest on screen. If the texture varies, take multiple shots to allow for some variation.

Environment photos
Take some shots that can be used as environment maps. Cubic from six directions. Take photos of a mirrored ball from the direction of the camera for the scene as a whole and or for individual objects.

Light reference photos
When the lights have been set up as per the final shot, get shots of lighting reference sphere. Remember to get references from it at all appropriate locations where the light differs. This should help determine the direction, intensity & colour of the light on the set. Take some photos of position and size of lights or take notes so that the hardness or softness of virtual lights can be determined.

Model some physical objects for camera matching
Model some of the objects that appear on the physical set using the dimensions taken at the shoot so that you can duplicate the camera setting – position and focal length in the next step. Model the objects on the default x,y,z axis rather than rotating them to match any camera angle. Its a lot more convenient to model on these axis. The camera is the object that isn’t necessarily aligned to any axis not the geometry. Position the objects acurately relative to each other.

Match the camera
The position and focal length of the virtual camera need to match that of the physical one. Make sure your resolution and aspect ratio and other camera details are set up correctly before matching the camera. A reminder to confirm that the shot you are matching is the very final shot with the correct composition and any post processing. If it’s not a locked off shot you may get good results by using camera matching software. The image of the shot to be matched should be set up in your software by attaching it to the camera in such a way that altering your camera position or focal length doesn’t affect the plate as seen through the camera in any way. In Maya this is set via a cameras Image Plane. Once the shot is set up in this way, adjust the camera position and focal length until you have matched the reference objects modeled in the previous step so that they correct in relation to the shots physical counterpart.

Match the lighting
Use the lighting reference sphere images to position, colour and attenuate the virtual lights in the scene. If you don’t have lighting reference sphere images, place virtual objects to replace physical ones and compare and contrast the lighting. You need to think about wether you are going to use the same lighting for both the virtual replacement of physical objects and the set extension or have light exclusivly linked for each of these two. This decision will have an impact on light type and placement amongst other paramaters.

Build the virtual set extension
No need to build parts that won’t be seen through the matched camera. Model and texture the virtual set extension.

Light the virtual set extension
You may want to set up the lighting created in "Match the lighting" step so that it exclusively lights that geometry and doesn’t affect the rest of the virtual set extension. And set up the virtual set extension lighting to exclude lighting the physical set object replacements in the "Match the lighting" step so that the two sets of lights do not impact one another. Alternatively you may now want to use the same lights to illuminate both the replacement objects and the virtual set extension. If you plane on the second alternative then adjust the light in the "Match the lighting" step.