Detailed Overview of HEVC/H.

265
Prepared by Shevach Riabtsev
The author is thankful to every individual who has reviewed the presentation and provided comments/suggestions

High Level Syntax

HEVC Encoder
Ref Ref Ref
Input Video

+
Motion Est.
MVs

Residual

Inter

T&Q

Bit-Stream
CABAC

Reference samples
Mode

Motion Comp.

Intra Pred.

Intra/Inter Decision
Intra

MVs/Intra modes

Intra Est.

Quantized residuals Reconstructed

Ref. Ref Ref Ref

DPB

SAO

Deblk.

SAO Params Est. Filter Control

+ +

Q-1& T-1
SAO params

Notes: In addition to AVC/H.264, SAO and SAO Params Estimation added. SAO Params Est. can be executed right after deblocking or right after the reconstruction as shown in the figure. HEVC similarity with AVC/H.264 allows quick upgrading of existing AVC/H.264 solutions to HEVC ones.

Bitstream Structure

VPS

SPS

PPS

Slice Header

Slice Data Picture #1

* * * *

Slice Header

Slice Data

Slice Header

Slice Data

* * * *

Picture #k

High-Level Syntax ( Sequence, Picture level ) Sequencelevel: VPS, SPS and PPS
VPS specify multi-layer structure

SPS specify a single layer: Profile, tier, level Picture Size Max/Min CTU size, CU depth Max/Min TU size, TU depth ..
PPS specify a single layer, on/off tools: Sign Data Hiding Transform Skip Weighted Prediction Tiles, WPP

Picture Syntax Slice Header

Reference Frames the list of reference pictures in DPB is explicitly signaled in slice header (unlike to AVC/H.264 where MMCO or sliding window mode is used). Not mentioned pictures are marked as unused for reference and should be removed from DPB respectively. Notice that explicit signaling of the reference pictures enhances error resilience. Indeed, if a decoder detects that one of the reference pictures (mentioned in the list) is not exist in DPB then this picture got lost. Deblocking Parameters

Picture Syntax (2)

Coding Tree Block (CTB): Picture is partitioned into square coding tree blocks (CTBs). The size N of the CTBs is chosen by the encoder (16x16, 32x32, 64x64). Luma CTB covers a square picture area of N N samples and the corresponding chroma CTBs cover each (N/2) (N/2) samples (in 4:2:0 format). Coding Tree Units (CTU): The luma CTB and the two chroma CTBs, together with the associated syntax, form a coding tree unit (CTU). The CTU is the basic processing unit similar to MB in prior standards.

Coding Block (CB): Each CTB can be further partitioned into multiple coding blocks (CBs). The size of the CB can range from the same size as the CTB to a minimum size (88).
Coding Unit (CU) The luma CB and the chroma CBs, together with the associated syntax, form a coding unit (CU). Each CU can be either Intra or Inter predicted.

CTU Syntax

64x64 CTU

CTU Syntax (2) All CUs in a CTU are encoded (traversed) in ZScan order:

64x64 CTU

CTU Syntax (3)

Formally CTU specifies quad-tree traversed in inorder.

Note: unlike to prior standards where MB header is followed by data, in HEVC headers are dispersed: CTU Header

CU Hdr

CU Data CTU

* * * * CU Hdr

CU Data

CU Syntax (1)
Prediction Block (PB): Each CB is partitioned in 1, 2 or 4 prediction blocks (PBs).
Prediction Unit (PU): The luma PB and the chroma PBs, together with the associated syntax, form a prediction unit (PU). Intra:

2Nx2N Inter:

NxN (only if CB size is smallest CB size, i.e. CU = SCU)

2Nx2N

NxN

2NxN

Nx2N

Sub-partitions (e.g. NxN) are allowed if CU = SCU

CU Syntax (2)
Inter Assymetric Partitions (conditioned by amp_enabled_flag in SPS):

nLx2N

nRx2N

2NxnU

2NxnD

Why assymetric partitions are beneficial:

2NxnU

2NxnD

nLx2N

nRx2N

CU Syntax (3)
Notes: The smallest luma PB size is 4 8 or 8 4 samples (where 4x8 and 8x4 are permitted only for uni-directional predictions, no bi-prediction < 8x8 allowed).

Chroma PBs mimic corresponding luma partition with the scaling factor 1/2 for 4:2:0.
Assymetric splitting is also applied to chroma CBs.

CU Syntax (4)
Transform Block (TB) : Each luma CB can be quadtree partitioned into one, four or larger number of TBs. The number of transform levels is controlled by max_transform_hierarchy_depth_inter and max_transform_hierarchy_depth_intra.
Example.

CB divided into two TB levels (the block #1 is split into four blocks):

1,0

1,1 1,3

0
1,2

0 2 3

1,0

1,1

1,2

1,3

CU Syntax (5)
Notes Unlike to H.264/AVC where TB PB, prediction and transform partitioning are independent, i.e. TB can contain several PBs and vice versa. Reported by some experts that prediction discontinuities on PB boundaries within TB are smoothed by transform and quantization. If PB and TB boundaries coincide then the discontinuities are observed increased.

Restrictions/Constraints
a) HEVC disallows 16x16 CTBs for level 5 and above (4K TV). Motivation: 16x16 CTBs add overheads for decoders to target 4K TV:

Up to 10% increase in worst-case decode time

Add storage for SAO params. b) Maximal CTU size shall be less than or equal to 4 * RawCtuBits / 3.

The variable RawCtuBits is derived as

RawCtuBits=CtbSizeY * CtbSizeY * BitDepthY +2 * ( CtbWidthC * CtbHeightC ) * BitDepthC

Illustration: Lets take CtbSizeY=16 (as in AVC/H.264). Then RawCtuBits = 16*16*8+2*8*8*8 = 3072 and maximal CTB bit-size is 4*3072/3 = 4096 bits respectively. Notice that in AVC/H.264 the maximal MB bit-size is 3200 bits.

Inter Prediction/ Motion Compensation

Luma Motion Compensation Details (1)

Fractional interpolation for luma samples uses 8-tap filter for both half-pels and quarter-pels (although some positions actually reduced to 7-tap filter). Notice that in AVC/H.264 the motion compensation is executed in two serial stages: 6-tap filter for half-pels bilinear filter for quarter-pels So, AVC/H.264 gives the same complexity as HEVC 8-tap filter but two-stage latency (HEVC motion compensation filter can be executed in one stage). In the following slides we illustrate as positions a0,0 through r0,0 are specified.
A - 1, - 1 A 0, - 1 a 0, - 1 b 0, - 1 c 0, - 1 A 1, - 1 A 2, - 1

A - 1,0 d - 1,0 h - 1,0 n - 1,0 A - 1,1

A 0,0 d 0,0 h 0,0 n 0,0 A 0,1

a 0,0 e 0,0 i 0,0 p 0,0 a 0,1

b 0,0 f 0,0 j 0,0 q 0,0 b 0,1

c 0,0 g 0,0 k 0,0 r 0,0 c 0,1

A 1,0 d 1,0 h 1,0 n 1,0 A 1,1

A 2,0 d 2,0 h 2,0 n 2,0 A 2,1

A - 1,2

A 0,2

a 0,2

b 0,2

c 0,2

A 1,2

A 2,2

Luma Motion Compensation Details (2) Quarter-pels a0,0, c0,0, d0,0, n0,0 and half-pels b0,0, h0,0 are derived from nearest integer
positions. The quarter-pels a0,0, c0,0, d0,0, n0,0 are derived by the 7-tap filter and the half-pels b0,0, h0,0 by 8-tap filter. a0,0 , b0,0 and c0,0 are computed by horizontal filtering, while d0,0 , h0,0 and n0,0 by vertical filtering.
A - 1, - 1 A 0, - 1 a 0, - 1 b 0, - 1 c 0, - 1 A 1, - 1 A 2, - 1

A - 1,0 d - 1,0 h - 1,0 n - 1,0 A - 1,1

A 0,0 d 0,0 h 0,0 n 0,0 A 0,1

a 0,0 e 0,0 i 0,0 p 0,0 a 0,1

b 0,0 f 0,0 j 0,0 q 0,0 b 0,1

c 0,0 g 0,0 k 0,0 r 0,0 c 0,1

A 1,0 d 1,0 h 1,0 n 1,0 A 1,1

A 2,0 d 2,0 h 2,0 n 2,0 A 2,1

A - 1,2

A 0,2

a 0,2

b 0,2

c 0,2

A 1,2

A 2,2

Luma Motion Compensation Details (3) Half-pel j0,0 is derived by applying the 8-tap filter vertically to nearest half-pels: b0,3 , b0,2 ,
b0,1 , b0,0 , b0,1 , b0,2 , b0,3 , b0,4 . Notice that j0,0 can be determined only after b0,0 has been computed (see the previous slide). Quarter-pels e0,0 and p0,0 are derived by applying the 7-tap filter vertically to nearest quarter-pels. Notice that e0,0 and p0,0 can be determined only after a0,0 has been computed (see the previous slide).
A - 1, - 1 A 0, - 1 a 0, - 1 b 0, - 1 c 0, - 1 A 1, - 1 A 2, - 1

A - 1,0 d - 1,0 h - 1,0 n - 1,0 A - 1,1

A 0,0 d 0,0 h 0,0 n 0,0 A 0,1

a 0,0 e 0,0 i 0,0 p 0,0 a 0,1

b 0,0 f 0,0

c 0,0 g 0,0 k 0,0 r 0,0 c 0,1

A 1,0 d 1,0 h 1,0 n 1,0 A 1,1

A 2,0 d 2,0 h 2,0 n 2,0 A 2,1

j00
q 0,0 b 0,1

A - 1,2

A 0,2

a 0,2

b 0,2

c 0,2

A 1,2

A 2,2

Luma Motion Compensation Details (4)

Quarter-pel i0,0 is derived by applying the 8-tap filter vertically to nearest quarter-pels: a0,3, a0,2, a0,1, a0,0, a0,1, a0,2, a0,3, a0,4

Quarter-pel k0,0 is derived by applying the 8-tap filter vertically to nearest quarter-pels: c0,3,
c0,2, c0,1, c0,0, c0,1, c0,2, c0,3, c0,4
A - 1, - 1 A 0, - 1 a 0, - 1 b 0, - 1 c 0, - 1 A 1, - 1 A 2, - 1

A - 1,0 d - 1,0 h - 1,0 n - 1,0 A - 1,1

A 0,0 d 0,0 h 0,0 n 0,0 A 0,1

a 0,0 e 0,0 i 0,0 p 0,0 a 0,1

b 0,0 f 0,0 j 0,0 q 0,0 b 0,1

c 0,0 g 0,0 k 0,0 r 0,0 c 0,1

A 1,0 d 1,0 h 1,0 n 1,0 A 1,1

A 2,0 d 2,0 h 2,0 n 2,0 A 2,1

A - 1,2

A 0,2

a 0,2

b 0,2

c 0,2

A 1,2

A 2,2

Luma Motion Compensation Details (5) Quarter-pels f0,0 , g0,0 , q0,0 , r0,0 are derived by applying the 7-tap filter vertically to the nearest quarter-pels.

A - 1, - 1

A 0, - 1

a 0, - 1

b 0, - 1

c 0, - 1

A 1, - 1

A 2, - 1

A - 1,0
d - 1,0 h - 1,0 n - 1,0 A - 1,1

A 0,0
d 0,0 h 0,0 n 0,0 A 0,1

a 0,0
e 0,0 i 0,0 p 0,0 a 0,1

b 0,0
f 0,0 j 0,0 q 0,0 b 0,1

c 0,0
g 0,0 k 0,0 r 0,0 c 0,1

A 1,0
d 1,0 h 1,0 n 1,0 A 1,1

A 2,0
d 2,0 h 2,0 n 2,0 A 2,1

A - 1,2

A 0,2

a 0,2

b 0,2

c 0,2

A 1,2

A 2,2

Chroma Motion Compensation

The fractional interpolation for the chroma is similar to the luma except that the 4-tap filter is applied instead of 8-taps in luma and accuracy is 1/8-pel.

Notes/Conclusions

1. Luma interpolation can be performed in two serial stages: half-pel and quarterpel.

2. For motion compensation of NxM block its required to load (N+7)x(M+7) reference block (3 left/above, 4 right and below).
3. 8 tap filter coefficients: { -1, 4, -11, 40, 40, -11, 4, -1 } 4. 7-tap filter coefficients: { -1, 4, -10, 58, 17, -5, 1 }, non-symmetric. 5. 8-tap MC interpolation filter expands DR of intermediate results to 22 bits: -pel interpolation expands DR for 8-bits input to 16 bits.

The next -pel interpolation expands DR to 22 bits.

Dynamic Range Estimation

Worst case for 1/2-pel filter : b0,0 = A3,0 + 4*A2,0 11*A1,0 + 40*A0,0 + 40*A1,0 11*A2,0 + 4*A3,0 A4,0 Maximal value of b0,0 is 88*255 = 22440, the minimal value is -24*255 = -6120. The same limits are also correct for h0,0 . Worst case for 1/4-pel filter:

a0,0 = A3,0 + 4*A2,0 10*A1,0 + 58*A0,0 + 17*A1,0 5*A2,0 + A3,0

Maximal value of a0,0 is 80*255 = 20400, the minimal value is -16*255 = -4080. The same limits are also correct for c0,0, d0,0, n0,0 . -4080 a0,0, c0,0, d0,0, n0,0 20400 -6120 b0,0, h0,0 22440 So, the values of a0,0, c0,0, d0,0, n0,0 , b0,0, h0,0 are within 16 bits.

Dynamic Range Estimation (2)

Second step of interpolation (when neighboring half-pel and quarter-pel samples are uses): e0,0 = ( a0,3 + 4*a0,2 10*a0,1 + 58*a0,0 + 17*a0,1 5*a0,2 + a0,3 ) >> 6 Taking into account that a0,k is in the range [-4080 .. 20400] the expression in the parenthesis gives the following limits:
-80*4080= - 326400 a0,3 + 4*a0,2 10*a0,1 + 58*a0,0 + 17*a0,1 5*a0,2 + a0,3 80*20400=1632000

So, the dynamic range increased to 22 bits (incl. sign bit).

After shifting by 6 the dynamic range reduced to 16 bits: -5100 e0,0 25500
j0,0 = ( b0,3 + 4*b0,2 11*b0,1 + 40*b0,0 + 40*b0,1 11*b0,2 + 4*b0,3 b0,4 ) >> 6 Taking into account that b0,k is in the range [-6120 .. 22440] the expression in the parenthesis gives the following limits:
-88*6120 = - 538560 b0,3 + 4*b0,2 11*b0,1 + 40*b0,0 + 40*b0,1 11*b0,2 + 4*b0,3 b0,4 88*22440=1974720

As in the case with e0,0 the dynamic range in calculation of j0,0 is increased to 22 bits. After the shift by 6, the dynamic range is reduced to 16 bits.

Intra Prediction

Overview

33 angular predictions for both luma and chroma. Two non-directional predictions (DC, Planar). PB sizes from 44 up to 6464. Like in AVC/H.264 Luma intra prediction mode is coded predictivly and chroma intra mode is derived from the luma one.

Coding & Derivation Luma Intra Prediction Mode (1)

Unlike to AVC/H.264 three most probable modes: MPM0, MPM1 and MPM2 are considered. The following figure reveals the logic for derivation of MPMs:
Left available ?
N Y Y

Top available ?
N

Left Intra ?
Y

Top Intra ?
Y

CandA=Left

CandA=DC

CandB=Top

CandB=DC

CandA = CandB?
Y

MPM0=candA MPM1=candB

MPM0 && MPM1 != Planar?

Y

CandA < 2
Y N

MPM2 = Plane

MPM0+MPM1<2? MPM0 = Plane MPM1 = DC MPM2 = Vertical MPM0=candA MPM1=(MPM0+29)%32+2 MPM2=(MPM0-1)%32+2

Y N

MPM2 = Vert

MPM2 = DC

Coding & Derivation Luma Intra Prediction Mode (2) Encoder side: If the current luma intra prediction mode is one of three MPMs, prev_intra_luma_pred_flag set to 1 and the MPM index (mpm_idx) is signaled. Otherwise, the index of the current luma intra prediction mode excluding the three MPMs is transmitted to the decoder by using a 5-bit fixed length code (rem_intra_luma_pred_mode).

Coding & Derivation Luma Intra Prediction Mode (3) Decoder side: Upon derivation of MPMs, collect them in candModeList[3]= [MPM0, MPM1, MPM2]. The following code outlines how luma intra mode is derived:
If prev_intra_pred_flag = true then IntraPredMode = CandList [mpm_idx] Else { sort CandList in ascending order } if candModeList[0] is greater than candModeList[1], swap two values. if candModeList[0] is greater than candModeList[2], swap two values. if candModeList[1] is greater than candModeList[2], swap two values. Read 5 bits to rem_intra_luma_pred_mode IntraPredMode = rem_intra_luma_pred_mode if IntraPredMode >= candList[ 0 ] IntraPredMode ++ if IntraPredMode >= candList[ 1 ] IntraPredMode ++ if IntraPredMode >= candList[ 2 ] IntraPredMode ++

EndElse

Coding & Derivation Chroma Intra Prediction Mode Unlike AVC/H.264, chroma intra prediction mode is derived from luma and from the syntax element intra_chroma_pred_mode ( signaled each PB ) according to the following table:

intra_chroma_pred _mode 0 1 2 3 4

Luma IntraPredMode 0 34 26 10 1 0 26 0 34 10 1 26 10 0 26 34 1 10 1 0 26 10 34 1 X ( 0 <= X <= 34 ) 0 26 10 1 X

Implementation Angular Intra Prediction (1)

At most 4N+1 neighbor pixels are required (in contrast to H.264/AVC, belowleft samples are exploited in HEVC).
Top CTU Top-left CTU 8x8
Top-left predictor

8x8
Top predict.

16x16
Top-right predictors

Left CTU

Left predictors

Current PB

8x8

Current CTU

16x16
Below Left predictors

Implementation Angular Intra Prediction (2) To improve the intra prediction accuracy, the projected reference sample location is computed with 1/32 sample accuracy, bi-linear interpolation is used. In angular mode predicted sample PredSample(x,y) is derived as follow:
predSample[ x ][ y ] = ( ( 32 iFact )*ref[ x+iIdx+1 ] + iFact*ref[ x+iIdx+2] + 16 ) >> 5 predSample[ x ][ y ] = ( ( 32 iFact )*ref[ y+iIdx+1 ] + iFact*ref[ y+iIdx+2] + 16 ) >> 5

The parameter iIdx and iFact denote the index and the multiplication factor determined by the intra prediction mode (can be extracted via LUTs).
The weighting factor iFact remains constant across predicted row or column that facilitates SIMD implementations of angular intra predictions.

Planar Mode In AVC/H.264, the plane intra mode requires two multiplications per sample
predL[ x, y ] = Clip1Y( ( a + b * ( x 7 ) + c * ( y 7 ) + 16 ) >> 5 )

plus overhead calculation per 16x16 block in determination of the parameters a,b and c. Totally the plane mode takes at most three multiplications per sample.
In HEVC, intra planar mode requires four multiplications per sample:
predSamples[ x ][ y ] = ( ( nT 1 x ) * p[ 1 ][ y ] + ( x + 1 ) * p[ nT ][ 1 ] + ( nT 1 y ) * p[ x ][ 1 ] + ( y + 1 ) * p[ 1 ][ nT ] + nT ) >> ( Log2( nT ) + 1 )

So, HEVC planar mode is more complex than in AVC/H.264.

Motion Vector Prediction

Overview
Effective motion data prediction techniques have been adopted in HEVC in order to reduce motion data portion in the stream. Unlike other standards (e.g. AVC/H.264), the HEVC adopted competitive motion vector prediction for both regular and merge (HEVC replacement of AVC/H.264 skip/direct) modes, i.e. several predictor candidates are competing for the prediction and the right candidate is signaled in the stream. Unlike AVC/H.264, in regular (advanced motion vector prediction or in AMVP in jargon of HEVC) MV prediction the co-located temporal MV is considered if one of spatial candidates is non-available or redundant. In merge mode (HEVC replacement of AVC/H.264 skip/direct), the set of merge candidates can include a temporal candidate (if one or more spatial candidates are non-available or redundant). Including of temporal MV prediction in both regular and merge modes improves error resilience. On the other hand an additional storage of co-located MVs of reference frames is required. Spatial and temporal MVP candidates can be derived independently and in parallel

Merge Mode: Removal of Same Motion Candidates (Prunning)

In earlier versions of HEVC, a candidate removed if any of previous candidate has the same motion. When NumMergeCands=5 the detection of all redundant candidates requires 10 comparisons per PU. In latest HEVC to reduce complexity of merge list generation, only 5 comparisons are executed (instead of 5) for detection of duplication and the temporal candidate is excluded from comparisons (for the sake of parallelism).
B2
comparison

B1

B0

A1 A0

Note: Due to limiting of comparisons and exempting the temporal candidate from prunning process, redundant candidates can appear in the merge list.

Merge Mode: Additional Candidates

If merge list is not full (i.e. #candidates<NumMergeCands) then additional virtual candidates appended.

Merge Mode: List Construction in Encoding

Derive candidate list (spatial & temporal )

Prunning Process remove duplicates (restricted)

If merge list full? No Yes Add virtual candidates

Select candidate for encoding

Merge Mode: List Construction in Decoding

Derive candidate list (spatial & temporal )

CABAC
Prunning Process remove duplicates (restricted) Decode merge_idx

If merge_idx< current list size? No Yes Add virtual candidates

index

Use candidate pointed by merge_idx for decoding

Advanced Motion Vector Prediction (AMVP) Motion vector is predicted from five spatial neighbors: B0, B1, B2, A0, A1 (see the figure below) and one co-located temporal MV. Only two motion candidates are chosen among six neighbors and the selected predictor is explicitly signaled (mvp_lx_flag).

B2 Prediction Unit Col. Block

B1

B0

A0 A1

Advanced Motion Vector Prediction (AMVP) cont.

The first motion candidate (left candidate) is chosen from {A0, A1} The second motion candidate (top candidate) is chosen from {B0, B1, B2} If both candidates are available and have the same motion data, one is excluded If one of the above candidates is not available then the temporal MV is used unless the temporal prediction mode disabled If the number of available candidates is less than 2, zero MV is added

Transform

Overview The standard supports 32x32, 16x16, 8x8 and 4x4 DCT-like transforms and 4x4 DST-like
transform. Notice that DST-like 4x4 transform is allowed only for intra mode. Each transform is specified by 8-bits signed digital transform matrix T. To perform all transform operations its required 32-bits precision.

The inverse transform can be executed as follows:

a) Z = X T, where X is the input matrix of quantized coefficients (16-bits per coefficient, for more detailed analysis refer to JCTVC-L0332)

b) Scaling and clipping of Z to guarantee that the output values are within 16-bits. c) Y = Z T Notice that in encoder architecture the step (c) can be coupled with quantization: once first row of Y completed the quantization of the first row is started.

Transform Implementation
Notice AVC/H.264 where transform coefficients are dyadic in 4x4 case and near dyadic (i.e. from the form 2^n, 2^n-1, 2^n+1) in 8x8 case and hence AVC/H.264 transform can be multiplication-free. In HEVC transform operations are not multiplication-free. Indeed, let the multiplication takes 3 cycles, shift or addition 1 cycle . Therefore if all coeffs are near dyadic we can use only shifts and additions, otherwise we need a multiplier (because the alternative of shifts and adds hurts performance). As well as in AVC/H.264 the transforms in HEVC are separable and can be performed as sequence of two 1D transforms (vertical and horizontal): for (i=0, i<N, i++ ) { 1D Transform on column i // vertical } Scaling (right shift by 7) & Saturation for (j=0, j<N, j++ ) { 1D Transform on row j }

// horizontal

Comments
As well as in previous standards HEVC DCT works well on flat areas, but fails on areas with noise, contours and other peculiarities of the signal. HEVC DCT is efficient for big size of blocks but it looses efficiency on smaller blocks.

Beginning from 16x16 transforms visual artifacts are noticeable. The more the transform size the more artifacts are observable. Deblocking can reduce artifacts on TB boundaries, while artifacts inside a TB can be reduced only by SAO. Therefore its recommended to apply SAO when large transform sizes (32x32) are used.

Transform Implementation (2)

HW Aspects of Implementation 1D 8x8 case

P0 butterfly block (free of multiplications), Z = P0 X

Pc permutation matrix

HW Aspects of Transform 1D 8x8 case (2) A1 butterfly block (no multiplications), Q = A1 * Z{1..4}

B1 , B2 4x4 matrices (multiplication by these matrices can be implemented in fast form)

4x4 DST (Discrete Sine Transform) 4x4 DST is applied only for Intra prediction. The 4x4 DST matrix from HEVC spec:

29 74 84 55

55 74 29 84

74 0 74 74

84 74 55 29

Motivation: Intra prediction is based on the top and left neighbors. Prediction accuracy is more for the pixels located near to top/left neighbors than those away from it. In other words, residual of pixels which are away from the top/left neighbor usually be larger then pixels near to neighbors. Therefore DST transform is more suitable to code such kind of residuals, since DST basis function start with low and increase further which is different from conventional DCT basis function. Reported 4x4 DST provides some performance gain, about 1%, against DCT. For bigger sizes the gain is negligible.

As well as DCT, DST can be implemented using "fast" form

Performance Result According to JCTVC-G757:

For 8x8 transform required 2.47 cycles per sample

For 16x16 transform required 3.35 cycles per sample

For 32x32 transform required 4.59 cycles per sample The above results are obtained on x86 and ARM with SIMD operations (MMX/SSE on x86, and NEON on ARM). 4.59 cycles per sample for 32x32 can be a bottleneck on some platforms.

Entropy Coding

Overview

HEVC specifies only one entropy coding method CABAC, comparing to two CABAC and CAVLC as in H.264/AVC. HEVC CABAC encoding involves three main functions: a) b) c) Binarization - maps the syntax elements to binary symbols (bins). Context modeling - estimates the probability of the bins Arithmetic coding - compresses the bins to bits based on the estimated probability

Memory Requirements In AVC/H.264 context of some syntax elements (e.g. mvd) depends on top and left values. This dependence on top values requires line buffers, may be issue for 4K resolutions.

In HEVC the dependence from top values in selection of context is almost removed.
For example, unlike to AVC/H.264, in HEVC mvd is coded without the need of knowing neighboring mvd values.

Residual Coding: Overview Each Transform Block (TB) is divided into 4x4 sub-blocks (coefficient groups). Processing starts with the last significant coefficient and proceeds to the DC coefficient in the reverse scanning order. Coefficient groups are processed sequentially in the reverse order (from bottomright to top-left) as illustrated in the following figure:

Coefficient groups for 8x8 TB, for different scans

Residual Coding: Scanning Order

Note: Experiments show that including horizontal and vertical scans for large TBs offers little compression efficiency, so vertical and horizontal scans are limited to 4x4 and 8x8 sizes.

Residual Coding: Multi-Level Significance Unlike to AVC/H.264, three level significance is used, intermediate level is added to exploit transform block sparsity: Level0: coded_block_flag is signaled for each TB (transform block) to specify the significance of the entire TB. Level1: (intermediate level): if coded_block_flag =1 then each TB divided into 4x4 coefficient groups (CG) where the significance of the entire CG is signaled (by coded_sub_block_flag).
a) The coded_sub_block_flag syntax elements are signaled in reverse order (from bottom-right towards top-left) according to selected scan.
b) The coded_sub_block_flag is not signaled for the last CG (i.e. the CG which contains the last level). Motivation: a decoder can infer significance since the last level is present. c) The coded_sub_block_flag is not signaled for the group including the DC position

Residual Coding: Multi-Level Significance (cont.)

Level2: If coded_sub_block_flag=1 then significant_coeff_flag are signaled to specify the significance of individual coefficients.

a)The significant_coeff_flag are signaled in the reverse order (from bottom-right towards top-left) according to selected scan.

Notes:
The coded_sub_block_flag of CG containing of DC coefficient (i.e. (0,0)

position) is not coded and inferred to 1. Motivation to improve coding efficiency, since the probability of this CG to be totally zero is low.
If current CG contains last coefficient in a TB then the coded_sub_block_flag is

not coded and inferred to 1.

Residual Coding: Multi-Level Significance (Example)

16x16 TU with 4x4 coefficient groups:

--------------------------------------------------------| 5 2 0 1 | 1 0 0 0 | 1 1 1 0 | 0 0 0 0 | | 5 0 2 1 | 2 0 1 1 | 0 2 1 0 | 0 0 0 0 | | 0 1 0 0 | 1 0 1 1 | 0 1 1 1 | 0 0 0 0 | | 0 0 0 1 | 1 1 1 1 | 0 0 1 0 | 0 0 0 0 | --------------------------------------------------------| 1 1 0 0 | 1 0 1 0 | 0 1 0 1 | 0 0 0 0 | | 2 1 0 1 | 0 0 1 0 | 1 0 0 1 | 0 0 0 0 | | 0 0 1 0 | 0 0 1 0 | 1 0 0 0 | 0 0 0 0 | | 1 0 0 1 | 1 0 1 0 | 0 0 0 0 | 0 0 0 0 | --------------------------------------------------------| 0 1 1 1 | 0 1 0 1 | 1 1 1 0 | 0 1 0 0 | | 0 0 1 0 | 0 1 1 1 | 0 0 0 1 | 0 0 0 0 | | 1 0 0 1 | 0 0 0 0 | 1 1 0 0 | 1 0 0 0 | | 0 0 1 0 | 0 0 0 0 | 1 1 0 0 | 0 0 0 0 | --------------------------------------------------------| 0 0 0 0 | 0 0 0 0 | 0 0 0 0 | 0 0 0 0 | | 0 0 0 0 | 0 0 0 0 | 0 0 0 0 | 0 0 0 0 | | 0 0 0 0 | 0 0 0 0 | 0 0 0 0 | 0 0 0 0 | | 0 0 0 0 | 0 0 0 0 | 0 0 0 0 | 0 0 0 0 | --------------------------------------------------------For this CG coded_sub_block_flag is not coded

Last coeff

Residual Coding: Levels At the start of residual block, coordinates of last significant coefficient are signaled (last_significant_coeff_x, last_significant_coeff_y) Coding starts backward from last significant coefficient toward (0,0). The coding process of each CG generally consists of five separate loops (passes), which provides some benefits for parallelization :

1. 2. 3. 4. 5.

significant_coeff_flag loop coeff_abs_level_greater1_flag loop coeff_abs_level_greater2_flag (at most one flag is coded) coeff_sign_flag loop coeff_abs_level_remaining loop

Hint for parallelization: Grouping of syntax elements of same type enables parallel processing. For example, while coeff_abs_level_greater1_flag are proceeding, the significance map contexts for the next CG can be pre-calculated.

Residual Coding: significant_coeff_flag significant_coeff_flag - indicates whether the transform coefficient is non-zero.
1 bin regular coding, 3 context models. Context model derivation for 8x8 and higher TBs - context depends on the significant_coeff_group_flag of the neighboring right CG and lower (sl) CGs and on the coefficient position in the current CG. Motivation: to avoid data dependencies within a CG and to benefit parallezation with negligible coding loss if contexts depend on significance of immediately preceeding coefficints (around 0.1% as reported in JCTVC-I0296). Coding direction
Current CG Bottom CG Right CG

Here sr denotes significant_coeff_group_flag of the right CG and sl denotes significant_coeff_group_flag of the lower CG.

Context model derivation for 4x4 TB is completely position based:

Not coded inferred

Residual Coding: significant_coeff_flag (cont.) Notes:

significant_coeff_flag is not coded and inferred to 1 when the last significant

coefficient points at it.

significant_coeff_flag is not coded and inferred to 1 when

coded_sub_block_flag is true and all the other coefficients are zero (i.e. all coefficients are zero except the (0,0), specified by inferSbDcSigCoeffFlag).
In case of 4x4 TB the significant_coeff_flag is not coded. The

significant_coeff_flag is inferred to 1 if the last significant coefficient points at it, otherwise inferred to 0.

Residual Coding: coeff_abs_level_greater1_flag coeff_abs_level_greater1_flag - indicates (if signaled) whether the transform coefficient has absolute value > 1.
1 bin regular coding, 24 context models.
Only first eight coeff_abs_level_greater1_flags in a CG are coded, the rest is inferred to 0. Motivation to reduce regularly (context) coded bins, especially at the high bit-rate, and improve CABAC throughput. Indeed, at most 8 coeff_abs_level_greater1_flags are coded in one CG instead of 16. There are 4 context model sets for luma ( denoted as 0, 1, 2 and 3) and 2 for chroma (denoted as 4 and 5), the number of context models in each set is 4. The derivation of context mode consists of two steps: the inference of context set and the derivation of the model inside the selected set, the following table reveals the context set derivation:

Luma
# coeff_abs_level_greater1_flags
in previous CG

Chroma
>0 1 3 0 4 4 >0 5 5

0 0 2

CG with DC CG without DC

Residual Coding: coeff_abs_level_greater1_flag (cont.)

Context model derivation within the selected context set depending on the number of trailing ones and the number of coefficient levels larger than 1 in the current CG: If coeff_abs_level_greater1_flag is the first in the current CG then the context model equals to 1. Else If the revious coefficient in current CG is more than 1 (i.e. the previous coeff_abs_level_greater1_flag=1) then the context model equal to 0. Else If the number of trailing ones is 1 (i.e. the previous coeff_abs_level_greater1_flag=0 and the pre-previous coeff_abs_level_greater1_flag=1 or not exist) then the model is 2

Else (if trailing ones is 2 or more) then the model is 3.

Residual Coding: coeff_abs_level_greater2_flag coeff_abs_level_greater2_flag - indicates (if signaled) whether the transform coefficient has absolute value > 2. Unlike coeff_abs_level_greater1_flag, this flag is signaled once.
1 bin regular coding, 6 context models. If all coeff_abs_level_greater1_flag are 0, the coeff_abs_level_greater2_flag is not signaled and inferred to 0. Context model derivation:

Luma
# coeff_abs_level_greater1_flags in previous CG

Chroma
>0 1 3 0 4 4 >0 5 5

0 0 2

CG with DC CG without DC

Notice that the derivation of context model for coeff_abs_level_greater2_flag is identical to the derivation of context set of coeff_abs_level_greater1_flag.

Residual Coding: coeff_sign_flag

coeff_sign_flag : substantial proportion in compressed stream 15-20%.

coeff_sign_flags are bypass coded

Sign Data Hiding (SDH): optional mode, for each CG the sign of the last nonzero coefficient (in the reverse scan) is omitted. Instead, the sign is embedded in the parity of the sum of the levels, if the sum is even then the hidden sign is +, otherwise .

Encoder can be required to modify coefficients to embed the sign (potentially quantization noise can be increased).
If the distance in scan order between the first and the last nonzero coefficient is less than 4 then SDH is not used. Notice that the fixed value 4 was experimentally chosen (see JCTVC-I0156). Probably that value can be a bad choice on some streams. If only one nonzero coefficient is present in CG, then SDH is not activated. When SDH is beneficial? When the percentage of sign bits is substantial (it is expected to happen when the bit-rate is low). Disadvantages of SDH: More complexity and Increase of quantization noise (potentially)

Residual Coding: example implementation of SDH in encoder

If the parity does not match the omitted sign, the encoder has to change the value of one of the nonzero coefficients in the current CG. Low complexity implementation (details can be found in JCTVC-H0481): If #nonzero coeffs > 1 and distance (in scan) between the first and the last nozero coefficient >= 4 { For each nonzero and non-last (in reverse scan) quantized coefficient qCoef Do { Calculate delta = abs(qCoef) * q_scale tCoef where tCoef is transformed coefficient (prior to quantization). }

If there is nonzero delta values, find the minima minNzDelta among abs( delta ) { If minNzDelta >0 adjust qCoef = qCoef +1 Else [minNzDelta <0] adjust qCoef = qCoef -1 } Else [ all delta values are zero ] Take most high frequence coeff and adjust it.
}

Residual Coding: coeff_abs_level_remaining coeff_abs_level_remaining remaining absolute value of the coefficient level. All bins are bypass coded (to increase throughput). The total level is derived as:
Level = 1 + coeff_abs_level_greater1_flag+ coeff_abs_level_greater2_flag+ coeff_abs_level_remaining

Binarization - HEVC employs adaptive GolombRice coding for small values and switches to Exp-Golomb code for larger values. The Golomb-Rice parameter cRiceParam depends on previous levels. The transition point to Exp-Golomb is when the unary code length equals 4. The maximal codeword for coeff_abs_level_remaining is kept within 32 bits.

Residual Coding - Example

Scan_pos Coefficients significantFlag gr1Flag gr2Flag signFlag levelRem

15 0 0

14 1 1 0

13 -1 1 0

12 0 0

11 2 1 1 0

10 4 1 1

9 -1 1 0

8 -4 1 1

7 4 1 1

6 2 1 1

5 -6 1

4 4 1

3 7 1

2 6 1 not coded

1 -12 1

0 18 1

0 3

1 3

0 3

0 1

11

17

Coeff-1

Residual Coding - Notes On SW decoder: the processing of residuals take ~8% of the computation for 4K video. It is a challenge to speedup or parallelize the residual coding loop by SW There are multiple branches within the loops. There are data dependency among adjacent data. The loop counts in the four loops can be different (challenge for loop unrolling) For coeff_abs_level_remaining the binarization type depends on the previous coefficient level.

Deblocking

Overview
The deblocking filter is applied to all samples adjacent to PB or TB boundaries, with the following exceptions: picture boundaries

slice boundaries (if deblocking is explicitly disabled in sequence level)

tile boundaries (if deblocking is explicitly disabled in sequence level) Granularity is 8x8 grid or higher (unlike AVC/H.264 where granularity is 4x4)

32
Deblocked 8X8 8X8 Non-deblocked

16x16
8X8

32

16X16

16X16

Deblocking Algorithm

1. For each edge of 8x8 grid determine the filter strength (Bs) 2. According to the filter strength and the average quantization parameter (QP) determine two thresholds: tC and 3. According to the values of edge pixels and tC , modify (if needed) the pixels

Top Line Buffer For deblocking top reference storage is necessary: 4 luma top lines

2 chroma top lines

p3 p2 p1 p0

Horizontal Edge Vertical Edge

p3 p2 p1 p0 q0 q1 q2 q3

q0 q1 q2 q3

p0 .. p2 luma pixels are modifiable while p0 .. p3 are taken into consideration p0 chroma is modifiable, while p0 , p1 are taken into consideration
Page 75

Determination of Filter Strength

The strength of the deblocking filter is represented by three values: 0 - no deblocking 1 - weak 2 strong Notice that in AVC/H.264 there are five strengths and more complicated derivation of boundary strengths. P and Q are two adjacent TB or PB blocks then the filter strength Bs is specified as: If one of the blocks (P or Q) is intra Bs = 2 Else if P and Q belong to different TBs and P or Q has at least one non-zero transform coefficient then Bs = 1 Else if the reference pictures of P and Q are not equal then Bs = 1 Else if P and Q has difference number of MVs then Bs = 1 Else if the difference between x or y motion vector component of P and Q is equal or greater than one integer sample then Bs = 1 Else Bs = 0

Determination thresholds tC and Thresholds tC and are derived by the following table:

Where Q = [ (QPQ + QPP +1)>>1 + (beta_offset_div2<<1) ]

Vertical Edge Filtering (1) derivation d on/off decision for all 4 lines/columns based on two lines/columns: p3,0 p2,0 p1,0 p0,0 p3,1 p2,1 p1,1 p0,1 p3,2 p2,2 p1,2 p0,2 p3,3 p2,3 p1,3 p0,3 Derivation of d: dp0 = Abs( p2,0 2*p1,0 + p0,0 ) dp3 = Abs( p2,3 2*p1,3 + p0,3 ) dq0 = Abs( q2,0 2*q1,0 + q0,0 ) dq3 = Abs( q2,3 2*q1,3 + q0,3 ) dpq0 = dp0 + dq0 dpq3 = dp3 + dq3 dp = dp0 + dp3 dq = dq0 + dq3 d = dpq0 + dpq3 Notice that lines #1 and #2 dont participate in calculation of d q0,0 q1,0 q2,0 q3,0 q0,1 q1,1 q2,1 q3,1 q0,2 q1,2 q2,2 q3,2 q0,3 q1,3 q2,3 q3,3

Vertical Edge Filtering (2) derivation dSam0 and dSam1 dSam0 = 0 dSam3 = 0 If ( (dpq0 ( >> 2 )) And ((Abs( p3,0 p0,0 ) + Abs( q0,0 q3,0 ) < ( >> 3 )) And (Abs( p0,0 q0,0 ) < ( 5*tC + 1 ) >> 1) ) Then { dSam0 =1 } If ( (dpq3 ( >> 2 )) And ((Abs( p3,3 p0,3 ) + Abs( q0,3 q3,3 ) < ( >> 3 )) And (Abs( p0,3 q0,3 ) < ( 5*tC + 1 ) >> 1) ) Then { dSam3 =1 }

Vertical Edge Filtering (3) derivation dE, dEp and dEq dE= 0, takes values 0,1,2 dEp = 0, takes values 0,1 dEq = 0, takes values 0,1 If d < Then dE = 1 if dSam0 = 1 and dSam3 = 1 then dE = 2 // strong filter, modify p0 .. p2, q0 .. q2 If dp <( + ( >> 1 ) ) >> 3 then dEp = 1

If dq <( + ( >> 1 ) ) >> 3 then dEq = 1

EndIf

Vertical Edge Filtering (4) core

If dE = 2 // strong filtering, p0 .. p2, q0 .. q2 are modied { for each k=0..3 { p0,k = Clip3( p0,k2*tC, p0,k+2*tC, ( p2,k + 2*p1,k + 2*p0,k + 2*q0,k + q1,k + 4 ) >> 3 ) p1,k = Clip3( p1,k2*tC, p1,k+2*tC, ( p2,k + p1,k + p0,k + q0,k + 2 ) >> 2 ) p2,k = Clip3( p2,k2*tC, p2,k+2*tC, ( 2*p3,k + 3*p2,k + p1,k + p0,k + q0,k + 4 ) >> 3 ) q0,k = Clip3( q0,k2*tC, q0,k+2*tC, ( p1,k + 2*p0,k + 2*q0,k + 2*q1,k + q2,k + 4 ) >> 3 ) q1,k = Clip3( q1,k2*tC, q1,k+2*tC, ( p0,k + q0,k + q1,k + q2,k + 2 ) >> 2 ) q2,k= Clip3( q2,k2*tC, q2,k+2*tC, ( p0,k + q0,k + q1,k + 3*q2,k + 2*q3,k + 4 ) >> 3 ) } }

Vertical Edge Filtering (5) core

Else if dE = 1 // weak filtering, p0,p1 and q0,q1 are modified { for each k=0..3 {
= ( 9 * ( q0,k p0,k ) 3 * ( q1,k p1,k ) + 8 ) >> 4 if (Abs() < tC*10) { = Clip3( tC, tC, ) p0,k = Clip1Y( p0,k + ) q0,k = Clip1Y( q0,k ) } if dEp = 1 // modify p1 { p = Clip3( (tC >> 1), tC >> 1, ( ( ( p2,k + p0,k + 1 ) >> 1 ) p1,k + ) >>1 ) p1,k = Clip1Y( p1,k + p ) } if dEq = 1 // modify q1 { q = Clip3( (tC >> 1), tC >> 1, ( ( ( q2,k + q0,k + 1 ) >> 1 ) q1,k + ) >>1 ) q1,k = Clip1Y( q1,k + q ) } }

Sample Adaptive Offset (SAO)

Page 83

Background Idea: Quantization makes reconstructed and original blocks differ. The quantization error is not uniformly distributed among pixels. There is a bias in distortion around edges which can be eliminated/reduced.

Background (cont.) In addition to bias in quantization distortion around edges, systematic errors related to specific ranges of pixel values can also occur. Both types of the above systematic errors (or biases) are corrected in SAO.

Overview
SAO is the second post-processing tool (after deblocking) accepted in HEVC/H.265. SAO is applied after deblocking. For efficient HW implementation SAO can be coupled with deblocking in MB-loop.

SAO can be optionally turned off or applied only on luma samples or only on chroma samples (regulated by slice_sao_luma_flag and slice_sao_chroma_flag ).
SAO parameters can be either explicitly signalled in CTU header or inherited from left or above CTUs.

As well as deblocking SAO is adaptively applied on pixels. Unlike to deblocking SAO is adaptively applied to all pixels.
There are two types of SAO: Edge Type - offset depends on edge mode (signaled by SaoTypeIdx = 2) Band Type - offset depends on the sample amplitude (SaoTypeIdx = 1) Note: chroma CTBs share the same SaoTypeIdx.

Edge Type SAO In case of Edge type, the edge is searched across one of following directions ( the direction is signaled by sao_eo_class parameter, once per CTU):

Notes: Sample labeled p indicates a current sample. Two samples labeled n0 and n1 specify two neighboring samples along the chosen direction. The edge detection is applied to each sample. According to the results the sample is classified into five categories (EdgeIdx) :

Edge Type SAO (cont.) According to EdgeIdx the corresponding sample offset (signaled by sao_offset_abs and sao_offset_sign) is added to the current sample. Up to 12 edge offsets (4 luma, 4 Cb chroma and 4 Cr chroma) are signaled per CTU. To reduce the bit overhead there is a particular merge mode (signaled by sao_merge_up_flag and sao_merge_left_flag flag) which enables a direct inheritance of SAO parameters from top or left CTU.

Reported that SAO reduces ring and mosquitos artifacts and improve subjective quality for low compression ratio video.

Band Type SAO The pixel range from 0..255 is uniformly split into 32 bands and the sample values belonging to four consecutive bands are modified by adding the values denoted as band offsets.

Band offsets are signaled in CTU header.

SAO Design Points

For SAO left and top lines of pixels need to keep in a memory. Pipeline chain: a) QTR Deblock+ SAO decisions SAO b) QTR + SAO decisions Deblock SAO According to the schema (a), during the deblocking process, statistical information is processed and the decision on SAO parameters is made. Due to the fact that SAO parameters are determined in Deblock stage we cant apply CABAC in parallel to Deblocking. The schema (b) enables to parallel Deblock and CABAC with negligible coding efficiency loss.

Quality

SAO is enabled (QP = 32 )

SAO is disabled (QP = 32 )

Paralleling Tools

Overview

HEVC adopted three in-built parallel processing tools: Slices Tiles Wavefronts (WPP)

Slices

As in H.264/AVC, slices are groups of CTUs in scan order, separated by start code.

There are two types of slices: Dependent Non dependent (normal).

Slice #0 Slice #1 Independent Slice Dependent Slice

Slice #2
Slice #3 Slice #4

Slice #3 Independent Slice

Slices - Dependent Slices

Dependencies of dependent slice include the following: Slice Header Dependency: Short slice header is used where the missing elements are copied from the preceding normal slice. Context Models Dependency: CABAC context models are not initialized to defaults at the beginning of a dependent slice. Spatial Prediction Dependency: No breaking intra and motion vector prediction.

Slices - Dependent Slices

Restriction: Each dependent slice must be preceded by a non-dependent slice. The picture always starts with a normal slice, followed by zero or more dependent slices. Rationale for using of dependent slices: Allow data associated with a particular wavefront thread or tile to be carried in a separate NAL unit, and thus make that data available to a system for fragmented packetization with lower latency than if it were all coded together in one slice.

Slices: Pros and Cons

Pros: Slices are effective for:

Network packetization (MTU size matching)

Low-delay applications. Indeed, to start transmission of the encoded data earlier, a current slice may be already transmitted, while encoding the next slice in the picture. Parallel processing (slice are self-contained, excepting deblocking) but decoder has to perform some preprocessing to identify entry points. Fast resynchronization in case of bitstream errors or packet loss.

Slices: Pros and Cons

Cons: Penalty on rate distortion performance is incurred due to the breaking of dependencies at slice boundaries Overhead is added since each slice is preceded by the slice header.

Tiles Tiles are rectangular groups of CTUs. Tiles are transmitted in raster scan order, and the CTUs inside each tile are also processed in raster scan order. All dependencies are broken at tile boundaries. The entropy coding engine is reset at the start of each tile and flushed at the end of the tile. Only the deblocking filter can be optionally applied across tiles, in order to reduce visual artifacts.

Tiles (cont.)

At the end of each tile CABAC is flushed and consequently the tile ends at byte boundary. The tile entry points (actually offsets) are signaled at the start of picture in order to enable to a decoder to process tiles in parallel. Due to high area/perimeter ratio square tiles are more beneficial than rectangular ones (since the perimeter represents the boundaries where the dependencies are broken).

Tiles: Pros & Cons

Pros: Friendly to multi-core implementations, can be built by simply replicating the single core designs. Composition of a picture (4K TV) from multiple rectangular sources which are encoded independently. With slices we can compose only horizontal stripes.

Cons:
Pre-defined tile structure makes MTU size matching challenging. Breaking intra and motion vector prediction across tile boundaries deteriorates coding efficiency.

Wavefronts (WPP) Picture is divided into rows of CTUs. The first row is processed in an ordinary way.

The second row can be delayed until two first CTUs of the first row completed.
The third row can be processed after two first CTUs of the second row have been made, etc.

Wavefronts (WPP) The context models of the entropy coder in each row are inferred from those in the preceding row with a small fixed processing lag. Actually the context models are inherited from the second CTU of the previous row.

CABAC is flushed after the last CTU of each row, making each row to end at byte boundary.
No breaking of dependencies across rows of CTUs.

Wavefronts (Cont.)

CABAC is flushed at the end of each CTU row in order to make each row to end at byte boundary and to facilitate parallel processing. CABAC is reset at the end of each CTU row in order to enable parallel processing.

Wavefronts (Cont.)

At the start of each CTU row CABAC contexts are inherited from the above row (after the second CTU of the above row finished, in order to minimize training penalty). In other words cabac context derivation crosses row boundaries. It requires some synchronization among cores.

Entry points of each CTU row are explicitly signaled in picture/slice header.

Wavefronts (WPP) Pros & Cons

Pros: Good for architectures with shared cache, e.g. overlapping of search areas: Unlike tiles intra and motion vector prediction across CTU rows enabled.

Cons: MTU size matching challenging with wavefronts. Frequent cross-core data communication, inter-processor synchronization for WPP is complex.

Wavefronts (WPP) Pros & Cons

Pros: Good for architectures with shared cache, e.g. overlapping of search areas: Unlike tiles intra and motion vector prediction across Search area of core #(n-1) CTU rows enabled.
Currently processed already processed Core #(n-1)

Core #n

Search area of core #n

Notes

Wavefront parallel encoding is reported to give BD-rate degradation around 1.0% compared to a non-parallel mode. Bitrate savings from 1% to 2.5% are observed at same QP for Wavefront against Tiles (each row is encompassed by single tile). Wavefronts and Tiles cant co-exist in single frame.

Appendix: HM Test Configuration

1. All Intra Main configuration - encoder_intra_main.cfg High efficiency (10 bits per pixel) - encoder_intra_he10.cfg

2. Random Access Main configuration - encoder_randomaccess_main.cfg High efficiency (10 bits per pixel) - encoder_randomaccess_he10.cfg

HM Test Configuration (cont.)

3. Low Delay (DPB buffer contains two or more reference frames, each inter frame can utilize bi-prediction from previous references): Main configuration - encoder_lowdelay_main.cfg High efficiency (10 bits per pixel) - encoder_lowdelay_he10.cfg

4. Low Delay (DPB buffer contains single reference frame) : I P P P P Main configuration - encoder_lowdelay_P_main.cfg High efficiency (10 bits per pixel) - encoder_lowdelay_P_he10.cfg