Analysis and synthesis of multiview images using wavelets based on voxel patterns

doi:10.21203/rs.3.rs-1973485/v1

Download PDF

Research Article

Analysis and synthesis of multiview images using wavelets based on voxel patterns

https://doi.org/10.21203/rs.3.rs-1973485/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Multiview wavelets based on voxel patterns of autostereoscopic multiview three-dimensional (3D) displays are proposed. Direct and inverse continuous wavelet transforms of binary and grayscale multiview images were performed. With modified wavelet coefficients, 3D images with the changed parallax and depth were obtained. Multiview images were displayed using lenticular plates. The visual picture confirms the correctness of all synthesized images. The results can be applied to various types of autostereoscopic 3D displays.

Graphical Systems

Electrical Engineering

autostereoscopic display

3D display

multiview wavelet

image analysis

image synthesis

In this paper, multiview (MV) image in the image plane of autostereoscopic three-dimensional (3D) displays (ASD) with a lenticular/barrier plate [1–3] are considered. This plane can be split into logical cells, one cell per lenticular/barrier element. Noteworthy, some specific arrangements consisting of several fragments can be recognized in MV image. Such arrangements can be treated as combinations of certain basic voxel patterns (VP) [4, 5]. Significantly, the multiview, integral, plenoptic, and light-field displays have a similar structure of the image in the image plane.

In 3D imaging, the wavelets are used for various purposes, for example, image fusion [6], gesture recognition [7], video coding [8, 9], etc. Mostly, the known wavelets are in use. For MV images, specially designed wavelets based on VP were proposed, similarly to Fresnel patterns [10, 11]. VPs can be thought of as a kind of “elementary particles” in the image plane of ASD. Previous MV wavelets were based on Haar wavelets [12], B-splines [13], and Marr wavelets [14]. Among them, the Haar-based MV wavelets are directional, while the MV image shows no signs of directionality. Besides, as compared to the MV wavelets [13, 14], the proposed wavelets have an essentially simpler structure allowing real-time processing.

To build pattern-based (PB) wavelets, VPs were used. Technically, in each discrete plane, PB MV wavelet is represented by an individual function as in Refs. [4, 5]. The output of the continuous wavelet transform (CWT) of a MV image is the two-dimensional (2D) array of wavelet coefficients for each depth plane, comprising the 3D array of CWT coefficients. This array can be used in the image analysis to estimate the depth and location of the 3D image (a particular branch of 3D image processing), as well as in the inverse wavelet transform to generate a visual 3D object.

Forward and inverse wavelet transforms in visualization are the two main research methodologies - analysis and synthesis - used in engineering. Each of them is of great importance in itself, but only together they provide the greatest progress. For example, image synthesis is important in imaging; the depth analysis is a particular branch of 3D image processing. Combining the two, we can make up a complete cycle. In the cycle, coefficients of CWT array can be modified, and the result of the inverse transform would be another/modified 3D image. A couple of examples of such modifications were demonstrated in the paper: the inverse depth and the dimension of the parallax. The inverse depth was a special problem of the integral imaging for years and this non-trivial transformation which could hardly be made in another way than a simple modification of coefficients proposed in this paper. Similarly, there is no obvious known way to change the dimensions. The results are confirmed by visual 3D images.

The testing images were not made especially for this paper. Instead, we used the existing images of various kinds (multiview and light-field) from different sources. These images were made years ago by different authors. Their characteristics (resolution, contrast, etc.) might be not be satisfactory today, but we only use them to demonstrate the principle. Moreover, such a “random/independent” choice of the testing images (together with the positive results of each case) confirms the reliability of the proposed technique.

The proposed wavelets could be applied to binary/gray-scale multiview/integral/light-field/plenoptic images of autostereoscopic 3D displays with the parallax barrier or the microlens array.

We arranged the paper as follows. In Sec. 2, we recall the properties of the voxel patterns and calculate their spectra. In Sec. 3, we describe the MV wavelets in one and two dimensions. The experimental results are presented in Sec. 4, including the intentionally modified 3D objects. Section 5 briefly outlines the connections between the 3D autostereoscopic displays, multiview wavelets, and 3D interaction in real time. Section 6 contains a brief discussion, and Sec. 7 concludes the paper.

In this Section, we describe the voxel patterns themselves and calculate their spectra. The basic element of VPs is the symmetric rectangular unit pulse of the unit width commonly defined as,

$$\text{rect}\left(x\right)=\left\{\begin{array}{c}0, \left|x\right|>\frac{1}{2}\\ \frac{1}{2}, \left|x\right|=\frac{1}{2}\\ 1, \left|x\right|<\frac{1}{2}\end{array}\right.$$

The single pulse of the width w, phase θ, and the amplitude a is expressed as

$${u}_{0}=a\text{rect}\left(\frac{\text{x}}{w}\right)$$

Let the integer d ≠ 0 be a distance from the screen, as in [14]; the distance in front of the screen of ASD is positive, behind it – negative. In space, the distance corresponds to a plane. The zeroth plane is intentionally omitted (zero partition means an absence of the pattern). The first depth plane (the sign does not matter) is the screen itself. Each VP corresponds to certain discrete depth plane, and the d-th pattern consists of n = |d| non-negative pulses of the identical amplitude and width; n = 1, 2, … In other words, according to [13], VP is a finite series of rectangular pulses with the unit height; the width of each pulse is c/n, where c is the size of the logical cell. Similar to [13], [14], VP can be written as a linear combination of the identical rectangular pulses as follows,

$${P}_{n}\left(x\right)=\sum _{i=1}^{n}\text{rect}\left(\frac{nx-{ip}_{n}-{\text{s}}_{n}}{c}\right), n>1$$

where n is the index of the discrete depth plane, c is the size of the cell, x is the coordinate, and (ip_n + s_n)/c is the phase shift of the i-th pulse; in the latter expression, the initial phase of the first pulse (i.e., the position of its center) is

$${s}_{n}=\left(1\pm \frac{1}{2n}\right)c=c\bullet \left\{\begin{array}{c}1+\frac{1}{2n}\equiv \frac{1}{2n}, d>0\\ 1-\frac{1}{2n}, d<0\end{array}\right.$$

and the period of pulses (the distance between the centers of repeated pulses) is

$${p}_{n}=\left(1\pm \frac{1}{n}\right)c$$

The graph of the period as a function of n is shown in Fig. 1.

The interval (gap) between the pulses of VP is

$${\delta }_{n}=\left(1-\frac{1}{n}\pm \frac{1}{n}\right)c=c\bullet \left\{\begin{array}{c}1, d>0\\ 1-\frac{2}{n}, d<0\end{array}\right.$$

Note that in the positive depth planes, the gap does not depend on the distance.

Substituting the period and phase into Eq. (3), we get the formula for the voxel patterns,

$${P}_{n}\left(x\right)=\sum _{i=0}^{n-1}\text{rect}\left(n\frac{x}{c}-i\left(1\pm \frac{1}{n}\right)-\left(1\pm \frac{1}{2n}\right)\right), n>1$$

The examples of series of the rectangular pulses comprising VPs are shown in Fig. 2. The implied intensity range of MV images is [0, 1], which can be easily transformed into the typical range of digital images [0, 255].

The main properties of VPs [4], [5] in MV images are as follows. In the 1D case, the summed length of all pulses is equal to the size of the cell. In the 2D case, the voxel patterns are the products of two 1D functions in the orthogonal directions, and the summed area of all pulses is equal to the area of the cell. With the same n (i.e., across the same number of the cells) the pulses of VPs for positive and negative planes are distributed differently, see Fig. 1. Depending on the sign of d, the left/right edge of the first/last pulse coincides with the left/right edge of the cell. Specifically, the first and last partitions are aligned either to the outer or to the inner boundaries of the outmost (leftmost and rightmost) cells of the pattern. By Eq. (6), there are two cases: a wide gap c and the maximum separated pulses at the positive distances in front of the screen, and a narrow gap and the maximum close pulses at the negative distances behind the screen. We may call these cases as separated and close pulses, resp.

All patterns including the patterns with n = 1, 2 obey the general rules; although, for n = 1, neither the period nor the gap has a physical meaning, and effectively, the negative sign is only used in Eqs. (5), (6) in this case. Besides, for n = 2, the gap between the pulses may be zero. These features might look uncertain, and to avoid confusion, we explain these cases separately.

In the case of n = 1, the same pattern matches both positive and negative distances ± 1; neither period nor gap have a meaning. The initial phase is,

$${s}_{1}=\frac{c}{2}$$

Therefore,

$${P}_{1}\left(x\right)=\text{rect}\left(\frac{\text{x}}{c}-\frac{1}{2}\right)$$

The initial phase and the period of the pulses by Eqs. (4), (5) are,

$${s}_{2}=\text{c}\bullet \left\{\begin{array}{c}\frac{1}{4}, d>0\\ \frac{3}{4}, d<0\end{array}\right.$$

$${p}_{2}=\text{c}\bullet \left\{\begin{array}{c}\frac{3}{2}, d>0\\ \frac{1}{2}, d<0\end{array}\right.$$

Besides, the gap δ_2- at the second negative distance is 0,

$${\delta }_{2}=\left\{\begin{array}{c}c, d>0\\ 0, d<0\end{array}\right.$$

i.e., two adjacent pulses of the width c/2 compose a single pulse of the total width c, as shown in Fig. 3(b),

Therefore,

$${P}_{2}\left(x\right)=\left\{\begin{array}{c}rect\left(2\frac{x}{c}-\frac{1}{4}\right)+rect\left(2\frac{x}{c}-\frac{7}{4}\right), d>0\\ rect\left(2\frac{x}{c}-\frac{3}{4}\right)+rect\left(2\frac{x}{c}-\frac{5}{4}\right)\equiv rect\left(\frac{x}{c}-1\right), d<0\end{array}\right.$$

Two particular cases of VPs with n = 1 and n = 2 are shown in Fig. 3.

Let’s calculate the Fourier transform (FT) of 1D VPs. Note that we use the definition of the Fourier transform without 2π,

$$F\left(k\right)=\underset{-\infty }{\overset{\infty }{\int }}f\left(x\right){e}^{-ikx}dx$$

Also, recall that FT of a shifted function f(x-s) is the FT of the original function multiplied by the exponential,

$${F}_{s}\left(k\right)={e}^{-iks}F\left(k\right)$$

Then, FT of the single pulse Eq. (2) is,

$$F\left(\omega \right)=aw\bullet \text{sinc}\frac{kw}{2}$$

where k is the wavenumber, and the unnormalized sinc function is

$$\text{sinc}z=\frac{\text{sin}z}{z}$$

In three specific cases, FT is as follows.

(a) an elementary pulse with w = c/n, a = A,

$${F}_{1}=2\frac{A}{k}\text{sin}\frac{kc}{2n}=\frac{Ac}{n}\text{sinc}\frac{kc}{2n}$$

(b) a typical combination of the elementary pulses (a pair of narrow pulses) with w = c/n, a = A, symmetrically displaced around the origin by ± s. Thus,

$${F}_{2}=4\frac{A}{k}\text{sin}\frac{kc}{2n}\text{cos}\left(ks\right)=2\frac{Ac}{n}\text{sinc}\frac{kc}{2n}\text{cos}\left(ks\right)$$

$${F}_{-}=2\frac{A}{qk}\bullet \text{sinc}\frac{qkc}{2}=Ac\bullet \text{sinc}\frac{qkc}{2}$$

Consequently, the FT of VP at the first plane is,

$${F}_{p1}=Ac\bullet \text{sinc}\frac{kc}{2}$$

Then, there are two pulses with w = c/2 in VP of the second order, which are displaced either by c/4 or 3c/4, as shown in Figs. 3(a), (c). In the case of the adjacent pulses (s = c/4), by Eq. (19), FT is,

$${F}_{p2-}=2\frac{A}{k}\text{sin}\frac{kc}{2}=Ac\bullet \text{sinc}\frac{kc}{2}$$

Note that Eq. (22) is identical to Eq. (21), i.e., F_p2- = F_p1. In the case of separated pulses (s = 3c/4), FT of the pattern is,

$${F}_{p2+}=2\frac{A}{k}\text{sin}\frac{kc}{2}\left(4{\text{cos}}^{2}\frac{kc}{4}-3\right)$$

The graphical illustrations of these three particular cases of FT are shown in Fig. 4.

FP of VPs of higher orders can be calculated similarly, as the sum of pairs Eq. (116), conditionally plus the central pulse Eq. (18) for odd n,

$${F}_{pn}=\left\{\begin{array}{c}\sum _{i=0}^{\frac{n}{2}}{F}_{2}-{F}_{-}, n even\\ \sum _{i=0}^{\frac{n}{2}}{F}_{2}-{F}_{-}+{F}_{1}, n odd\end{array}\right.$$

To illustrate Eq. (24), FT of VPs up to 16th order is shown in Fig. 5.

Note that the minimum (the second extremum after DC term) of the FT of the close pulses is on the right side from the dotted vertical line k = 2π in Fig. 5, while the second extremum of the separated pulses on the left side. This corresponds to the periods of VPs by Eq. (5) shown in Fig. 1.

In this Section, we describe the wavelets and calculate their spectra.

Generally, the wavelets should satisfy some important properties, namely, the zero mean (no bias) and the square norm one (fixed power/energy); the orthogonality is not a requirement for the continuous wavelet transform (CWT).

The french hat wavelet [15, 16] obviously satisfies the requirements. One can consider the french hat wavelet as a positive rectangular pulse biased ½ down, or as two adjacent Haar wavelets of the opposite signs, see Fig. 6. In the first case, it is a biased VP; therefore, the french hat wavelet will be the MV wavelet of the first order.

Other MV wavelets with compact support (which are identically zero outside some finite interval) can be built in a similar manner. To make MV wavelets satisfy the zero mean property, a constant bias can be subtracted from the VP (across the limited interval of n cells for the n-th depth plane). However, the property of the square norm one becomes violated. Nevertheless, by changing the amplitude of the biased VP, the condition can be satisfied again. Thus, the wavelet with a proper bias and normalization coefficient satisfies the necessary conditions.

Consider two cases: the one-dimensional (1D) wavelets for ASD with the horizontal parallax only (HPO) and the two-dimensional (2D) wavelets for ASD with so-called full parallax (FP). The spectra will be also analyzed.

3.1 1D wavelets

HPO wavelets are the functions of one coordinate. As intended, we take VP, bias and normalize it; thus we obtain the MV wavelet satisfying the conditions of zero mean and square norm one. According to this suggestion, the general formula for the 1D MV wavelets of the order n is as follows,

$${\psi }_{1Dn}\left(x\right)={h}_{1Dn}\left({P}_{n}\left(x\right)-{b}_{1Dn}\right)$$

where h_1Dn is the normalization coefficient, b_1Dn is the bias; the VPs P_n with the period p_n and phase s_n, are given by Eqs. (4), (5), (7). The coefficient and bias will be calculated in this Section.

The overall picture of the one-dimensional (1D) PB wavelets for ASD with the horizontal parallax only (HPO) built this way is shown in Fig. 7.

The support interval of the wavelets is,

$$I=\left\{\begin{array}{c}2c, n=1\\ nc, n>1\end{array}\right.$$

Two particular cases (n = 1 and n = 2, p = q = 2) are shown in Fig. 8.

Note the different alignment of the pulses to the cells. The wavelet ψ₁ shown in Fig. 8(b) is the single rectangular pulse biased by ½; compare it with the French hat Fig. 6 to find a full identity of the shape (the phase is different, though). The absolute values of the wavelets ψ₂ are identical, but the sighs are opposite, ψ₂₊ = - ψ_2-, see Fig. 8(a), (c). Other wavelets may also have identical shapes but the different phases. Specifically, ψ₁ is ψ_2- shifted by one half of the cell (in the former, the positive pulse occupies the whole cell; in the latter, the pulse is shared between two cells, exactly as in the french hat wavelet). Briefly speaking, Fig. 8(a) shows the negative french hat, Fig. 8(b) the shifted french hat, while Fig. 8(c) shows the french hat wavelet itself.

PB wavelet of n-th depth plane occupies n cells (except for n = 1). The bias is,

$${b}_{1Dn}=\left\{\begin{array}{c}\frac{1}{2}, n=1\\ \frac{1}{n}, n > 1\end{array}\right.$$

Particularly, b_1D1 = b_1D2+ = b_1D2- =1/2. With the bias by Eq. (27), PB wavelets have zero mean.

Let’s calculate their normalized power (energy). In the case of n = 1, there is a single positive pulse with the width c and the amplitude of ½ and two negative pulses with the width c/2 and the amplitude of ½, so their total squared area is c. For higher n, there are n narrow pulses (their total summed width = c) of the height of 1–1/n plus the bias (which height = 1/n) spread across n-1 cells. Then,

$${‖f‖}^{2}=4{\left(\frac{n-1}{n}\right)}^{2}$$

Therefore, the normalization coefficient of 1D case is,

$${h}_{1Dn}=\frac{1}{‖f‖}=\left\{\begin{array}{c}1, n=1\\ \frac{1}{2}\frac{n}{n-1}, n>1\end{array}\right.$$

It is interesting that with n = 1, we may use the coefficient for n = 2, due to the similar shape of these wavelets. The normalization coefficient of these three cases equals 1: h_1D1 = h_1D2+ = h_1D2- =1.

Using Eqs. (27), (29), the general formula for 1D PB wavelet Eq. (25) can be rewritten as follows,

$${\psi }_{1Dn}\left(x\right)=\left\{\begin{array}{c}{P}_{1}\left(x\right)-\frac{1}{2}, n=1\\ \frac{1}{2}\frac{n}{n-1}\left({P}_{n}\left(x\right)-\frac{1}{n}\right), n>1\end{array}\right.$$

where P_n is defined by Eq. (7).

Similarly to the scaling function of the Haar wavelet, the single rectangular pulse could work as a scaling function at the first depth plane,

$${{\phi }}_{1\text{D}} = \frac{1}{2}\text{rect}\left(\frac{\text{x}}{\text{c}}-1\right)$$

Now, let’s find FT of the wavelets. For the first order (the single pulse with n = 1plus the negative bias with n = 1, q = 2), FT by Eqs. (18), (20) is

$${F}_{\psi 1}=4\frac{\text{A}}{k}\text{sin}\frac{kc}{2}{\text{sin}}^{2}\frac{kc}{4}$$

It is shown in Fig. 9 (a).

Note the absence of DC component in FT of the wavelet; except for this, the graph is quite similar to the FT of VP; cf. Figure 4(a).

Similar to the VPs of the first order, two pulses of the corresponding PB wavelets are biased by -½; the FT of both cases have the opposite signs,

$${F}_{\psi 2}=\pm 4\frac{A}{k}\text{sin}\frac{kc}{2}{\text{sin}}^{2}\frac{kc}{4}$$

Again, both FTs have 0 at the origin; see Fig. 9(b). Note that the moduli of all three mentioned FT for n = 1, 2 are identical.

FTs of the PB wavelets of the higher orders can be found similarly to VPs as sums of FTs of paired pulses for the even n (plus FT of the centered pulse for the odd n) using Eqs. (18), (116), (20). The calculated spectra of the wavelets are shown in Fig. 10.

Note some particular regions in the spectra, where FT is zero or close to zero. These regions in Fig. 10 are as follows: near zero, exactly on the line 2π, and a high-frequency range > 3π. These spectral regions are not covered by the proposed wavelets, and in general, may cause negative effects. Thus, we might need some special means to cover these “holes” in the spectrum. Namely, the first region can be covered by the wide single positive pulse of the double width, the second region by a series of the short pulses with period = c, and the third one by a chirp function which period changes from c/2 to the physically minimal period (in a digital screen, 2 pixels), see Fig. 11. The centers of the pulses of the infinite and chirp signals are aligned to the edge of the cell (at least the first pulse of each chirp frequency); the number of pulses of these signals should be enough to cover the whole MV image.

3.2 2D wavelets

Consider the wavelets with the 2D parallax (FP) in a similar manner as 1D. The 2D wavelets for ASD with the full parallax (FP) are the functions of two coordinates, i.e., the products of two 1D functions in the orthogonal directions. 2D PB wavelets at two planes between − 5 and + 5 are shown in Fig. 12; the special cases n ≤ 2 in Fig. 13.

The width and period of pulses in each row/column of a 2D PB wavelet are the same as in the corresponding 1D case. Note the smaller bias and the changed normalization coefficient (as those for 1D Ricker and 2D Marr wavelets of different dimensions, but both derived from the Gaussian). Namely, the bias of the 2D case is different because the wavelet is spread across n x n cells,

$${b}_{2Dn}=\left\{\begin{array}{c}-\frac{1}{4}, n=1\\ -\frac{1}{{n}^{2}}, n>1\end{array}\right.$$

There are n² peaks of the height of 1–1/n² (their total area = 1) plus the bias (height = 1/n²) across the area n²-1 remaining for the bias. Therefore,

$${‖f‖}^{2}=4{\left(\frac{{n}^{2}-1}{{n}^{2}}\right)}^{2}$$

and the normalization coefficient of 2D PB wavelets is

$${h}_{2Dn}=\left\{\begin{array}{c}1, n=1\\ \frac{1}{2}\frac{{n}^{2}}{{n}^{2}-1}, n>1\end{array}\right.$$

The general formula for the 2D PB MV wavelets built as the normalized and biased product of the orthogonal 1D VPs is as follows,

$${\psi }_{2Dn}\left(x,y\right)={h}_{2Dn}\left({P}_{n}\left(x\right){P}_{n}\left(y\right)-{b}_{2Dn}\right)$$

where h_2Dn is the normalization coefficient, b_2Dn is the bias, VPs P_n with their period and phase are given by Eqs. (4), (5), (7).

Based on the above formulas Eq. (34), (36), Eq. (37) can be rewritten,

$${\psi }_{2Dn}\left(x\right)=\left\{\begin{array}{c}{P}_{1}\left(x\right){P}_{1}\left(y\right)-\frac{1}{4},n=1\\ \frac{1}{2}\frac{{n}^{2}}{{n}^{2}-1}\left({P}_{n}\left(x\right){P}_{n}\left(y\right)-\frac{1}{{n}^{2}}\right), n>1\end{array}\right.$$

The 2D scaling function is the product of two scaling functions Eq. (31),

$${\phi }_{2D} =\frac{1}{4}\left(\text{rect}\left(\frac{x}{c}\right)-1\right)\bullet \left(\text{rect}\left(\frac{y}{c}\right)-1\right)$$

Its normalization coefficient is ¼.

In experiments, the PB wavelets were applied to two testing images made by different authors: the binary MV image of the tetrahedron [4] with two levels of brightness (the black and the white colors only) and to a gray-scale light-field image of books [17]. The latter image is available for free [17]; its magnified part is shown in Fig. 18 below. The image of books is processed in original colors. All calculations are verified and confirmed visually.

The base of the tetrahedron lies in the xy-plane and the camera (more exactly, an array of virtual cameras) is “in front of” the object at a certain z > 0, see Fig. 14(a). FP image of the tetrahedron is shown in Fig. 14(b) in pseudo colors in this and similar illustrations, where the white color means intensity zero and the black color means one.

For visual verification, the MV image was printed on a desk jet printer and displayed using two orthogonal crossed lenticular plates with 15 lines per inch; correspondingly, the size of the cell of the printed image was 1.69 mm. The visual appearance of the tetrahedron in 3D (technically, two photographs making up a stereo pair) is shown in Fig. 15.

With certain practical skills, one can watch a stereo pair with naked eyes; a 3D picture is merged in our mind [18, 19]. The visual picture confirms, particularly, that the vertex of the tetrahedron is closer to our eyes than the base, as in the model Fig. 14(a).

4.1 Direct transform

The result of the direct wavelet transform of an MV image is a 3D array which contains several 2D arrays for the discrete planes. The wavelet coefficients for the image of the tetrahedron are shown in Fig. 16 at two depth planes. The cross-sections of these 2D arrays along the thin lines are shown in Fig. 17.

Another testing image is the light-field image of books on a desk, see Fig. 18.

The result of the direct wavelet transform of the light-field image is shown in Fig. 19 at two depth planes.

The titles of the books can be hardly recognized in Fig. 18, but, one can read the words “Optical Systems” and “Fluid Dynamics” in Fig. 19. This shows the selectivity of the proposed wavelet transform by depth.

4.2 Direct and inverse transforms

The output of the direct transform can be naturally used as the input of the inverse wavelet transform. The inverse transforms of all planes can be merged (in a simple case like this, just added) into a whole MV image. Thus, we may restore a whole 3D object from the arrays of coefficients by planes.

In this numerical experiment, two options were applied: MV wavelets in both direct and inverse transforms, and MV wavelets in the direct transform but VPs in the inverse transform. In the latter case, VPs Eq. (7) were considered as a sort of scaling function for each depth plane.

The object restored with the first option, is shown in Fig. 20(a). The restored MV image looks similar to the original, as expected; compare it with Fig. 14(b). With the second option, the restored 3D MV image has smaller details, as shown in Fig. 20(b). Also note that using the scaling function, the positive intensity range [0, 1] seems to be restored more correctly.

(a) (b)

Figure 20. (a) Restored MV image of the tetrahedron calculated using wavelets. (b) Restored MV image of the tetrahedron using voxel patterns in inverse transform.

The image was displayed in 3D similarly to the original tetrahedron; see Fig. 21 and compare it with Fig. 15. The visual observation confirms that the restored 3D object is matching the original.

With both options, the resulting intensity might be distorted in some points because of occasional overlap of the wavelets. To overcome that, we can additionally apply an amplitude-limiter to ensure the output range of [0, 1] known a priori; however, this method is unacceptable for the gray-scale objects. Anyway, the second option automatically provides non-negative result.

For the image of books, the restored image reproduces the structure of the original light-field image correctly; see Fig. 22, although an essential vignetting of cells in the original image was not taken into account.

To confirm that the scaling functions seem to be the better functions for the inverse transform, the numerical difference between the original and two restored MV images was calculated, see Fig. 23. The difference is definitely smaller in the case of VPs. This suggests the following combined cycle: the direct transform using PB wavelets and the inverse transform using VPs as the scaling functions.

4.3 Modification of coefficients prior to inverse transform

Furthermore, with modified CWT coefficients, 3D objects different from the original can be constructed. Two modifications were made to illustrate that: the reversed depth and the dimensions of the parallax (the 2D parallax transformed into 1D parallax).

In the first case, the sign of the depth of each 2D array was reversed. The MV image of the tetrahedron with the reversed depth is shown in Fig. 24; compare it with Fig. 14(b).

The stereoscopic photograph of the tetrahedron with the reversed depth is shown in Fig. 25; compare it with its original counterpart in Fig. 15. As expected, the visual position of the vertex of the modified object is behind the base. Note that such a modification makes sense for a wireframe 3D object, but could be improper for a textured object because of so-called “pseudoscopic effect” [20, 21].

In the second case, CWT arrays obtained with 2D wavelets were used in the inverse transform using 1D wavelets; some planes of that CWT array were shown in Fig. 16. The MV image with the changed dimension of the parallax (2D → 1D) is shown in Fig. 26.

To display the HPO image with 1D parallax, one lenticular plate was enough. The stereoscopic photograph of the displayed HPO tetrahedron is shown in Fig. 27.

Figure 27 confirms that with modified dimensions, the structure of the object remains the same; the vertex is in front of the base. Compare the photograph of the modified object with its original counterpart Fig. 15.

When we are not just watching a movie, we would naturally need to point locations in the 3D content. In the 2D case, it is our regular behavior in front of computer. We used to click mouse in order to select, copy, move the text or graphical objects, etc. We do that always everywhere in any computer. But even in the case of the movie, we might need to control the sound level, to make a pause, etc. as some kind of “limited” interaction. What about the 3D case?

For interaction, we need at least a pointer and a pointing device. Our approach is to use a 3D cursor and a regular computer mouse with wheel. The practically implemented 3D cursor was presented in [22].

Instead of using some special 3D joysticks etc., we propose using a regular computer mouse with a wheel. The movement in a plane parallel to the screen (xy-plane) is made in a regular manner, just by moving the mouse vertically or horizontally, as we used to do. The movement along the z-axis (different distances from the screen) is made by the rotation of the wheel: the scroll down means “forward” or “farther” from an observer; the scroll up means “backward” or “closer” to the observer. Controlling the depth this way seems to be convenient for users.

This way, we can draw and control the cursor anywhere within the limits of the virtual 3D space. However, another problem appears. The cursor pattern is physically drawn in the image plane of ASD over a background image. Therefore, the 3D cursor partially overwrites the previously displayed content, even if it should be behind a 3D object. In such a case, the 3D cursor seems to “penetrate” into an object) and may look like a hole in the 3D object. To avoid such a “collision”, we need to know locations of the currently shown 3D objects. Indeed, if we have a computer model of all objects and their current locations in space, we might overcome the problem with ease. However, the full database might not be accessible every time. Nevertheless, using the multiview wavelets, we can perform the local wavelet analysis of the multiview image in neighboring lateral and longitudinal points, and find out if anything is in proximity. Then, we can apply proper countermeasures, e.g., make the cursor blinking, reduce its contrast, make a sound, and so on.

A couple of examples of the entire cycle of transforms in 3D interaction could be as follows. To find if a 3D object approaches another 3D object, we may analyze the depth of the underlying surface in real-time by the wavelet analysis, and when the depth of the surface (the result of analysis) would be close to the depth of the surface of another object, we may, e.g., re-synthesize one surface (recolor it, make it blink, etc.), and show the synthesized object in 3D. This way, we may draw additional attention of the operator to an approaching object.

In another example, we may prevent one 3D object from “penetrating” into another 3D object. To avoid that, we may do the following. When the first object is behind the second object in the 3D space (by the depth analysis of the wavelet coefficients), we may either hide it (i.e., refrain from drawing) or transform somehow (e.g., re-synthesize it with a reduced contrast) reducing the visual effect of the “penetration”. Besides, the array needed to determine the close neighbors can be as small as 3x3 cells in 3 depth planes, i.e., the required time could be approx. 30 times less than in the estimation given in Sec. 4.4. In that case, the time would be less than 25–30 msec, which is essentially less than the average reaction time of humans, about 200–250 msec [23]. Moreover, the optimized C + + program might run several times faster. In that case, the delay becomes practically negligible and would not produce any noticeable delay for an operator.

The authors hope that this could be a practical step toward the visual real-time interaction with 3D content in 3D space, and real-time 3D applications using the proposed MV wavelets become closer. For example, a fast 3D cursor with a proximity sensing can be efficiently implemented using the proposed wavelets.

We performed the continuous wavelet transform of MV images with 1D and 2D parallax (displayed using the vertical and orthogonal lenticular plates). This way, we model a small area of a whole screen of ASD. Due to the optical similarity of the lens camera and the pinhole camera, the results are also valid for ASD with the barrier plate.

The direct and inverse wavelet transforms using the proposed wavelets (as well as [13], [14]) are not a universal instrument to analyze/synthesize arbitrary images. We should emphasize that they only work well with the MV images. Applying these wavelets to other sorts of images might be meaningless as, e.g., the fresnelets [6], [7] are only useful with Fresnel holograms. Besides, the PB wavelets only provide a fast but approximate operation with 3D images because each voxel in the model [14] represents a rhomboidal area in space, but not a point. However, in many cases, it is acceptable because of a fast real-time result. And if we would need a more exact coordinates, we may apply the exact methods such as the ray tracing within the limited region, which could also be fast.

The calculation of the proposed wavelet functions is simpler than the calculation of other wavelets. The structure of the wavelets directly corresponds to the structure of the MV image. The time characteristics of a Python code running a desktop computer 3.4 GHz, 16 GB RAM was measured at the discrete depth planes from − 6 to + 6. The time to generate the 1D wavelet was less than 15 msec, 2D wavelet less than 31 msec; the total time for CWT of the image with 30x30 cells was 15.6–31.25 msec (1D transform) and 48.9–781.3 msec (2D transform), depending on the depth plane.

The testing images were not made especially for this paper. Instead, the existing multiview and light-field images made by independent authors were used. This confirms the reliability of the method. With that, the characteristics of the original images (such as resolution, contrast, etc.) were not modified; only demonstrated were the transformations using the proposed wavelets.

Due to their simpler structure, the calculation of the proposed wavelet functions themselves is simpler than the calculation of many known wavelets. Thus, the convolution (the most time-consuming operation of the wavelet transform) with the constant functions (within a plateau of a pulse) could be, in principle, calculated more efficiently than with other arbitrary functions. Besides, the structure of the wavelets directly corresponds to the structure of the multiview image. For example, the wavelets can be applied to the 3D controls in autostereoscopic displays.

Some practical requirements of this technique are as follows. The elementary view images of the MV image in the image plane should be rectified; at least because VPs in non-rectified MV images were not observed. The square cell should comprise an integer number of pixels (n2 pixels). MV image should comprise whole cells; at least, the whole cell must lie in the left upper corner. However, the latter is not a fundamental restriction, just a limitation of the current version of the computer program.

The author hopes that the proposed MV wavelets could be a practical step towards the real-time interaction with 3D content in the visual 3D space.

The multiview wavelets based on the voxel patterns of the autostereoscopic 3D displays were built. The description of the wavelets is made in the spatial and spectral domains. The direct and inverse wavelet transforms were demonstrated on binary/gray-scale images. Also, examples of modifications of 3D images using the proposed multiview wavelets were provided: the reversed depth and the changed dimension of the parallax. The correctness of the restored image was verified in each case. A regular computer mouse with a scroll wheel can effectively used in the 3D interaction in real-time. The proposed method could be a practical step towards a simple and natural 3D interaction in the autostereoscopic 3D displays using the proposed wavelets. The role of the multiview wavelets in 3D imaging (autostereoscopic displays) and 3D interaction is highlighted.

The results can be used in the multiview/light-field/integral image processing in general, as well as in the depth analysis of 3D multiview images and the 3D interaction in particular. The potential applications include designing, editing, arranging, and transforming the 3D images interactively in the multiview, integral, and plenoptic images.

Statement

There are no potential conflicts of interest.

Lee B., Park S.G., Hong K., Hong J.: Design and Implementation of Autostereoscopic Displays. SPIE Press, Bellingham,WA (2016).
Lueder E.: 3D Displays. Wiley, Chichester, UK (2012).
Dodgson N.A.: Autostereoscopic 3D displays. IEEE Computer 38, 31–36 (2005).
Son J.Y., Javidi B., Saveljev V.: Synthesizing 3D images based on voxels. Proc. SPIE 5202, 1–11 (2003).
Son J.Y., Saveljev V., Kim S.K., Javidi B.: Pixel patterns for voxels in contact-type 3D imaging systems. Jpn. J. Appl. Phys. 45, 798–803 (2006).
Rubio-Guivernau J.L., Gurchenkov V., Luengo-Oroz M.A., Duloquin L., Bourgine P., Santos A., Peyrieras N., Ledesma-Carbayo M.J.: Wavelet-based image fusion in multi-view three-dimensional microscopy. Bioinformatics 28, 238–245 (2012).
Aryanfar A., Yaakob R., Halin A.A., Sulaiman M.N., Kasmiran K.A., Mohammadpour L.: Multi-view human action recognition using wavelet data reduction and multi-class classification. Procedia Comp. Sci. 62, 585–592 (2015).
Yang W., Wu F., Lu Y., Cai J., Ngan K.N., S Li.: Scalable multiview video coding using wavelet. Proc. IEEE Int. Symp. Circ. S. 6078–6081 (2005).
Gelman A., Dragotti P.L., Velisavljevic V., Multiview image coding using depth layers and an optimized bit allocation. IEEE T. Image Process 21, 4092–4105 (2012).
Liebling M., Blu T., Unser M.: Fresnelets - a new wavelet basis for digital holography. Proc. SPIE 4478, 347–352 (2001).
Liebling M., Blu T., Unser M.: Fresnelets: a new wavelet basis for digital holography. IEEE T. Image Process. 12, 29–43 (2003).
Saveljev V.: Wavelets and continuous wavelet transform for autostereoscopic multiview images. J. Electr. Eng. 4, 19–23 (2016).
Saveljev V., Palchikova I.: Analysis of autostereoscopic three-dimensional images using multiview wavelets. Appl. Optics 55, 6275–6284 (2016).
Saveljev V., Palchikova I.: Analytical model of multiview autostereoscopic 3D display with a barrier or a lenticular plate. J. Inf. Disp. 19, 99–110 (2018).
Bendjoya P., Slezak É., Wavelet analysis and applications to some dynamic systems. Celest. Mech. Dyn. Astr. 56, 231–262 (1993).
Kurokawa Y., Mizutani Y., Mayuzumi M., Frequency filtering algorithms of plate wave AE for source location. J. Acoust. Emiss. 24, 145–152 (2006).
Georgiev T.: Gallery and Lightfield Data, http://www.tgeorgiev.net/. Accessed 15 June 2017 (2017).
Parkin A.: Digital Imaging Primer, p. 218. Springer, London, UK (2016).
Wikipedia, Stereoscopy: Freeviewing. https://en.wikipedia.org/wiki/Stereoscopy. Accessed 09 November 2021.
Burckhardt C.B., Collier R.J., Doherty E.T., Formation and inversion of pseudoscopic images. Appl. Optics 7, 627–631 (1968).
Yeom J., Hong K., Jeong Y., Jang C., Lee B.: Solution for pseudoscopic problem in integral imaging using phase-conjugated reconstruction of lens-array holographic optical elements. Opt. Express 22, 13659–13670 (2014).
Saveljev V., Son J.-Y., Yim C., Heo G.: Three-dimensional interactive cursor based on voxel patterns for autostereoscopic displays, Journal of Information Display 23, 137–150 (2022).
Jain A., Bansal R., Kumar A., and Singh K.D.: A Comparative Study of Visual and Auditory Reaction Times on the Basis of Gender and Physical Activity Levels of Medical First Year Students. Int. J. Appl. Basic Med. Res. 5, 124–127 (2015).

Download PDF

Version 1

posted

You are reading this latest preprint version

Analysis and synthesis of multiview images using wavelets based on voxel patterns

Status:

Version 1

Abstract

Figures

Introduction

Voxel Patterns

Pattern-based Wavelets

3.1 1D wavelets

3.2 2D wavelets

Numerical And Visual Experiments

4.1 Direct transform

4.2 Direct and inverse transforms

4.3 Modification of coefficients prior to inverse transform

Interaction

Discussion

Conclusion

Declarations

References

Status:

Version 1