Poznan University of Technology
3D Codec

This page presents a technology for efficient coding of 3D video in Multi-view plus Depth (MVD) representation.

The new 3D video systems will include glassless displays and will provide realistic impression of depth as well as controllable stereoscopic base-line distance. In such 3D video systems the description of a 3D scene should be richer than just a stereo pair. Some applications will need many views to be available at the receiver. For example, future autostereoscopic displays are expected to present simultaneously even 50 different views corresponding to cameras with parallel optical axes equally spaced within an interval of order of the human inter-ocular distance (about 64 millimeters). Such dense spacing of the views yields strong similarity between the neighboring views that can exploited for compression. Moreover, in the receiver many virtual views may be efficiently synthesized using the Depth-Image-Based Rendering (DIBR) [3,6,19] and for transmission the MVD format often may be limited to only 2-3 views accompanied with the corresponding depth maps [1]. In a realistic example of a system with an autostereoscopic display only 3 views with 3 depth maps are transmitted (Fig. 1).

Fig. 1. An example of a 3D video system where 3 views with 3 depth maps are transmitted and used for synthesis of many virtual views.

The proposed technology is backward compatible with HEVC standard, so one of the views called base view can be decoded by a legacy HEVC decoder (video only). The remaining data (video and depth) can only be decoded by the 3D decoder, because additional syntax structures are used in the bitstream (fig. 2). For both videos and depth maps hierarchical view coding structure similar to MVC is used: the already coded views are used as references for prediction of the subsequent views. There are three main inter-view prediction mechanizms used:

• view synthesis prediction with Disoccluding Region Coding,
• disparity-compensation prediction (MVC-like),
• depth-base motion prediction (DBMP).

Fig 2. Proposed codec structure for 3-view MVD.

The detailed technical description can be found in [1].

In october 2011, the codec has been submitted as a proposal to Call for Proposal [2] issued by Motion Picture Experts Group (MPEG) on behalf of International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC). After the submission, all proposals have been extensively tested both subjectively and objectively and the proposal from Poznan University Technolgy has been qualified as one of the best performing.

Even after the resolution of CfP, we have continued to develop and assess our codec in finer details. The objective and subjective experiments that we have performed [3,4,5] proved the results attained by MPEG and also provided in-depth look into the performace of our technology.

The evaluation methodology, can be found in [2,6].

Table 1. Input views position and synthesized output view position for 2-view and 3-view configurations.

Below, it can find an results of coding with our technology for 2-view case and 3-view case for all 8 sequences defined in CfP. The results include

• stereoscopic (side by side) AVI files of 2 synthesized middle views
• and bitstreams with executable decoder in RAR archive

Fig 3. Synthesized stereopair of Poznan Street sequence in side by side format created based on 3 videos and 3 depth maps encoded at 1950 kbps.

Fig 4. Synthesized stereopair of Poznan Street sequence in side by side format created based on 3 videos and 3 depth maps encoded at 1180 kbps.

Fig 5. Synthesized stereopair of Poznan Street sequence in side by side format created based on 3 videos and 3 depth maps encoded at 710 kbps.

Fig 6. Synthesized stereopair of Poznan Street sequence in side by side format created based on 3 videos and 3 depth maps encoded at 410 kbps.

Table 2. AVI file with synthesized stereo pair from decoded video and depth maps, bitstreams and executable decoder for 3 view case.

Table 3. AVI file with synthesized stereo pair from decoded video and depth maps, bitstreams and executable decoder for 2 view case.

References

• [1] Marek Domanski, Tomasz Grajek, Damian Karwowski, Krzysztof Klimaszewski, Jacek Konieczny, Maciej Kurc, Adam Luczak, Robert Ratajczak, Jakub Siast, Olgierd Stankiewicz, Jakub Stankowski, Krzysztof Wegner
  „Technical Desciption of Poznan University of Technology proposal for Call on 3D Video Coding Technology”
  ISO/IEC JTC1/SC29/WG11, MPEG 2011 / M22697, Geneva, Switzerland, November 2011
• [2] "Call for Proposals on 3D Video Coding Technology"
  ISO/IEC JTC1/SC29/WG11 MPEG2011/N12036, Geneva, Switzerland, March 2011
• [3] Marek Domański, Jacek Konieczny, Maciej Kurc, Robert Ratajczak, Jakub Siast, Olgierd Stankiewicz, Jakub Stankowski, Krzysztof Wegner
  „3D video compression by coding of disoccluded regions”
  IEEE The International Conference on Image Processing (ICIP), Orlando, USA, 30 wrzesień – 3 październik 2012
• [4] Marek Domański, Tomasz Grajek, Damian Karwowski, Krzysztof Klimaszewski, Jacek Konieczny, Maciej Kurc, Adam Łuczak, Robert Ratajczak, Jakub Siast, Olgierd Stankiewicz, Jakub Stankowski, Krzysztof Wegner
  „New Coding Technology For 3D Video With Depth Maps As Proposed For Standardization Within Mpeg”
  19th International Conference on Systems, Signals and Image Processing (IWSSIP), Vienna, Austria, 11-13 Kwiecień 2012
• [5] Marek Domański, Tomasz Grajek, Damian Karwowski, Krzysztof Klimaszewski, Jacek Konieczny, Maciej Kurc, Adam Łuczak, Robert Ratajczak, Jakub Siast, Olgierd Stankiewicz, Jakub Stankowski, Krzysztof Wegner
  „Coding of Multiple Video+Depth Using HEVC Technology and Reduced Representations of Side Views and Depth Maps”
  Picture Coding Symposium, Kraków, Polska, 2012
• [6] Filip Lewandowski, Mateusz Paluszkiewicz, Tomasz Grajek, Krzysztof Wegner
  „Subjective Quality Assessment Methodology for 3D Video Compression Technology”
  IEEE International Conference on Signals and Electronic Systems – ICSES 2012, Wrocław, Poland, Wrzesień 2012