Dynamic Programming

CONTENTS

Historically, this approach was first proposed in the nineteen seventies and then extended in many works. Dynamic programming algorithms have been used for both feature- and intensity-based stereo. They implement the maximum likelihood or Bayesian decision rules derived from particular probability models of stereo images and desired 3-D surfaces (in terms of 2-D epipolar profiles). Generally, only a single continuous optical surface is assumed resulting in the ordering constraint along each pair of the conjugate epipolar scan-lines. In accord with the ordering constraint, the corresponding pixels are in the same order along the conjugate scan-lines.

Markov chain model of a profile

The profile models can account for symmetry of stereo channels, visibility of 3-D points, and image discontinuities due to occlusions in each channel. Basic nonuniform image distortions can also be taken into account. Partial occlusions of surface points impose the visibility constraint on x-disparities along an epipolar profile in the symmetric coordinates (x,d) (see Figure 2):

dx,y − 1 ≤ dx−0.5,y ≤ dx,y + 1;    dx,y − 2 ≤ dx−1,y ≤ dx,y + 2.
The visibility constraint allows us to model a set of admissible epipolar profiles with a Markov chain of transitions between the nodes of a planar graph of profile variants (GPV) shown also in Figure 2:
Figure 3: Graph of profile variants (GPV) and admissible transitions for the visibility states.
Transitions between the nodes form all the profile variants. Depending on the incoming and outcoming transition, the node represents either a binocularly visible point (BVP), or a monocularly visible one (MVP) occluded in the left or the right image of a stereo pair. Each GPV-node v = [x,d,s] has three visibility states s indicating the binocular (s = B) or only monocular (s = M1 or M2) observation by the stereo camera 1 or 2, respectively. Only eight transitions shown in Figure 3 are allowed in a GPV.

The Markov chain model with a stationary probability distribution describes the expected shape and smoothness of a pro file by transition and marginal probabilities of the visibility states. This allows for probabilistic ordering of the pro file variants in Figure 2 that have the same similarity with respect to image signals. Let Pr(v|v′) be a probability of transition from a preceding node v′ to a current node v. The differences x−x′ and d−d′ between the x-coordinates and x-disparities in these GPV-nodes are uniquely speci fied by the visibility states s and s′. Therefore, the transition probabilities can be denoted as Pr(s|s′).


Figure 4: GPV for the symmetric dynamic programming stereo.

The GPV in Figure 4 with a narrow "tube" of x-disparities demonstrates both the uniform internal transitions and specifi c uppermost/lowermost transitions to ensure the equilibrium conditions are preserved in all the nodes. Let M substitute for M1 or M2 so that the transition probabilities are Pr(B|B) = p(B|B); Pr(M1|B) = Pr(M2|B) = p(M|B); Pr(B|M1) = Pr(B|M2) = p(B|M); Pr(M1|M1) = Pr(M2|M2) = p(M|M), and Pr(M1|M2) = p0(M|M). The equilibrium conditions suggest that each node in a profile has the same marginal probability of the visibility state Pr(B). Thus the transition probabilities and the resulting marginal probabilities of the visibility states in the profi les are expressed in terms of the two transition probabilities p(B|M) and p(M|B) as follows:

Thus to have a stationary probability distribution of the Markov chain in the equilibrium, only seven of the allowable eight transitions have nonzero probability and the transition M2→M1 should be absent: p0(M1|M2) = Pr(x,d,M1|x−1,d,M2) = 0. To retain the equilibrium conditions at the boundaries of the GPV in Figure 4, the transition probabilities for the extreme GPV-nodes at the uppermost and lowermost boundaries with the incomplete incoming and outcoming transitions are as follows: Probability of each particular profile d = ((xi,di,si): i = 1,2,…,N) containing N GPV-nodes is proportional to the product of the transitional probabilities for the successive nodes:
where N(s"|s′) denote the number of transitions from the visibility state s′ to s" along the profile d.

Return to the local table of contents

Return to the global table of contents

Markov chain model of image signals

Let a symmetric photometric model specify local signal distortions in the images g1 and g2 with respect to the unobserved "ideal" cyclopean, or ortho-image g of a 3-D scene. These local distortions are described by positive transfer factors a varying over the field of view of each camera and by random noise N of signal sensors:

g1:x1,y1 = a1:x,ygx,y + N1:x,y;    g2:x2,y2 = a2:x,ygx,y + N2:x,y
where gx,y is the signal in a 3-D point of the distortion-less cyclopean image of an observed surface and g1:x1,y1, g2:x2,y2 are the signals in the corresponding pixels of the distorted stereo pair. The positive transfer factors a and the sign-alternate random noise r specify the multiplicative and additive parts of the ortho-image distortions, respectively. We assume that the noise is independent in the pixels, has zero expectation and the same variance for all pixels, and its probability decreases steadily with the absolute value (like in the Gaussian case).

The transfer factors represent most regular part of image distortions that cannot be independent for adjacent binocularly visible points (BVP) in order to retain visual resemblance between the pair of images. Their interdependencies are roughly described with a symmetric difference model assuming a limited direct proportion between each grey level difference in the adjacent cyclopean BVPs and the two corresponding noiseless differences in the stereo images, i.e. the differences if the additive noise is absent:

.
Here (x′,y′), (x,y) are the neighbouring BVPs along the same epipolar profile and e1, e2 denote the positive "difference" transfer factors describing the interdependence between the neighbouring "amplitude" factors a over the scene. The difference factors vary within a given range E = [emin, emax] where 0 < emin ≤ emax. The difference model admits large deviations between the corresponding grey levels, but the deviations retain visual resemblance of the images because of the approximate direct proportions between the corresponding grey level differences.

Assuming that both the ortho-image signals g and the image random noise N1, N2 are statistically independent, the above difference model results in a Markov chain of image signals corresponding to the successive BVPs and in the independent image signals corresponding to the MVPs along a given profile, respectively. Under a given surface geometry, different statistical estimates of the ortho-image g and transfer factors a1, a2 can be derived on the basis of particular assumptions about the random noise probability distributions and the transfer factors variations. The estimates involve the corresponding image signals for the BVPs. The match between the estimated cyclopean, or ortho-image and the initial stereo images to within a given range of the transfer factors provides a theoretically justified part of a quantitative intensity-based similarity measure for stereo images. A heuristic regularising part of the measure corresponds to the MVPs because the difference model does not suggest how to estimate image distortions for the MVPs. These estimates need additional prior assumptions about the links between the MVPs and the neighboring BVPs.

Return to the local table of contents

Return to the global table of contents

Dynamic programming reconstruction of a profile

For brevity, the coordinate y is omitted below. Let d = ((xi,di,si): i = 1,2,…,n) denote a digital profile with n successive nodes in a GPV (here, i is a serial number of a GPV-node along the profile). Let g = (gi : i = 1,2,…,n), g1 = (g1:i : i = 1,2,…, n1), and g2 = (g2:i : i = 1,2,…,n2), be sequences of the ortho-image, left image, and right image signals for the profile d, respectively. Here, for brevity, gi denotes gxi, g1:i denotes g1:xi+di/2, and g2:i denotes g2:xi−di/2.

For the assumed image model, the dissimilarities of the corresponding signals from the cyclopean signal in the GPV-node with the visibility state B are |g1:i − a1:igi| and |g1:i − a1:igi| where (a1:i, a2:i: i = 1,2,…,n) is a sequence of the transfer factors. Given these factors, the simplest estimate of the unknown cyclopean signal is obtained by minimising the maximal of the two node-wise dissimilarities:

Here, the cyclopean signal gi is estimated in such a way as to minimise the maximum of the two squared or absolute pixel-wise noise values in the images (in both the cases the estimate is the same). The cyclopean signal transferred to the image under the relative transfer factor α is indicated by "hat" (^). The point-wise signal dissimilarity Di ≡ Dy(xi,di,si| g1:xi+di/2, g2:xi−di/2) for a BVP (xi,di,si=B) in a profile d is therefore as follows:

The total signal dissimilarity for a profile d is

where the set Ωxi,di,si constrains the nodes that can be just before the node (xi,di,si) in a continuous profile:
and each MVP (si = M1 or M2) has a certain constant "dissimilarity" Docc to account for occlusions without signal correspondences.

Dynamic programming (DP) exhausts all the profiles under the constraints Ω and finds the profile giving the global minimum of the total dissimilarity. The search involves a successive pass along the x-axis of the GPV. At any current location xi, all the possible GPV-nodes (xi,di,si) are examined in order to calculate and store, for each node, the local, potentially optimal (provided that this node belongs to the optimum solution) backward transition to one of the preceding nodes specified by the set Ω. Let S(xi,di,si) denote the sum of signal dissimilarity values along the whole potentially optimum backward path for every node (xi,di,si) where i = 1, 2, …, n; di = dmin, dmin+1, …, dmax, and si = M1, B, M2. Then the basic recurrent DP computation is as follows:

where Tback(xi,di,si) is the indicator of the potentially optimum backward transition.

After passing a given range of the x-coordinates, the optimal profile minimising the total signal dissimilarity is recovered by the backward pass using the stored potentially optimal backward transitions:

Spatial interdependence of the transfer factors along the profile is more naturally taken into account if one chooses the floating rather than fixed range of the factor for each point i depending on the neighbouring factor along the profile. The floating range constrains deviations of the differences between the cyclopean signals in neighbouring BVPs along the profile transferred to the images (below, i′B is the index of the preceding BVP being closest to the BVP i):

.

Because the floating range varies along every potentially optimal solution, the global minimisation cannot be attained. But the DP algorithm can still approximately minimise the resulting total dissimilarity, too. Figures below exemplify results of the symmetric DP stereo algorithm applied to the stereo pair "Newspaper" of size 256×384:

Left and right images of the pair "Newspaper" (the canonical stereo geometry).
Reconstructed digital disparity map (after median smoothing in the moving 7×3 window)
and the corresponding cyclopean image.

Notice that the DP reconstruction is reasonably good for the textured surface of newspaper whereas highly uniform background areas result in arbitrary random deviations of the adjacent profiles (due to their independent reconstruction).

Return to the local table of contents

Return to the global table of contents