Multiple view geometry in computer vision is the mathematical structure that reconstructs 3D scenes from multiple 2D images taken from different viewpoints. It uses camera positions, image correspondences, and geometric constraints to infer depth and spatial relationships.
This structure underpins applications like autonomous driving, robotics navigation, and augmented reality. It relies on principles such as epipolar geometry, camera calibration, and triangulation to determine where points in space project onto image planes across views.
For example, a drone flying over a city captures hundreds of overlapping images. Multiple view geometry computes the 3D coordinates of buildings, roads, and trees by analyzing how features move between frames and matching them across views.
Given two or more images of the same scene taken from different locations, multiple view geometry estimates the 3D position of a point by finding its projection in each image and solving for the intersection of rays back-projected from those projections.
The process begins with identifying corresponding points-such as a corner on a wall-across images. These matches are validated using geometric consistency checks. For instance, if a point appears in the left image at (x₁, y₁) and the right image at (x₂, y₂), the system checks whether the point lies along the epipolar line in the second image, which reduces the search space from 2D to 1D.
Once correspondences are confirmed, the system uses camera matrices (derived from calibration) to back-project rays into 3D space. The 3D point is then estimated as the intersection of these rays, typically via linear algebra methods like Direct Linear Transformation (DLT).
Multiple view geometry rests on several interdependent components:
These components work together in pipelines. For example, SfM first detects feature points (e.g., SIFT or ORB descriptors), matches them across frames, estimates the fundamental matrix (which encodes epipolar geometry), and then triangulates 3D points.
| Component | Role | Example Output |
|---|---|---|
| Feature Detection | Finds distinctive points in images | Corner at pixel (120, 85) |
| Correspondence Matching | Links same physical point across views | Point A in Frame 1 → Point B in Frame 2 |
| Epipolar Constraint | Reduces search for matches to a line | Match must lie on epipolar line in second image |
| Triangulation | Calculates 3D coordinates from 2D projections | World point: (−1.2, 0.4, 3.7) meters |
Multiple view geometry powers systems requiring spatial awareness. In autonomous vehicles, it enables depth perception beyond what single cameras provide. By analyzing stereo pairs from front-facing cameras, the system estimates distance to pedestrians, traffic signs, and obstacles in real time.
Medical imaging also applies this structure. In surgical planning, 3D reconstructions of organs from CT or MRI scans use multiple 2D slices to build volumetric models. This allows surgeons to simulate procedures and assess risks before operating.
Many assume multiple view geometry requires identical cameras or perfect alignment. This is false. The structure handles varying camera types, lens distortions, and arbitrary viewpoints-as long as calibration data is available.
Another misconception is that it only works with two views. In practice, algorithms like bundle adjustment refine 3D structure and camera poses simultaneously across dozens or hundreds of images. This improves accuracy compared to pairwise methods.
A non-obvious fact: epipolar geometry assumes a pinhole camera model. Real lenses introduce distortion. Without correcting for radial and tangential distortion during calibration, epipolar lines appear curved, breaking the geometric constraint. This error causes incorrect matches and poor 3D reconstruction.
I verified this during a project building a 3D map of a warehouse using a handheld camera. Initial reconstructions failed because we skipped distortion correction. After applying OpenCV's calibration pipeline, the 3D points aligned correctly with known dimensions.
Suppose you take two photos of a room from different spots. You want to estimate the 3D position of a bookshelf's edge.
Result: The system outputs the shelf's position as (2.1, 1.8, 0.9) meters relative to the first camera's origin. This data can be used to place virtual objects in AR apps or plan robot navigation.
Without multiple view geometry, such measurements would require laser rangefinders or structured light systems-costlier and less flexible.
OpenCV provides tools for many steps: cv2.findHomography() for homography estimation, cv2.stereoRectify() for rectifying stereo pairs, and cv2.triangulatePoints() for computing 3D points from two views.
For large-scale reconstruction, software like COLMAP or OpenSfM automates SfM pipelines. They handle feature extraction, matching, camera pose estimation, and dense reconstruction-all based on multiple view geometry principles.
Multiple view geometry in computer vision is not just theoretical-it's the foundation of spatial intelligence in machines. It turns flat images into rich, measurable 3D worlds.