Joint Explicit FMO map and Error Concealment for Wireless Video Transmission

Jantana Panyavaraporn* and Supavadee Aramvith*

Abstract H.264/AVC has a new error-resilient tool called Flexible Macroblock Ordering (FMO). In this paper, we present a method to generate explicit FMO map based on distortion by simulating spatial and temporal error concealment at the encoder and a new error concealment method at decoder for wireless network. The error concealment method at the decoder is applied according to the residual information derived from the distortion simulation at the encoder that generated the FMO map. Our simulation result indicate the our proposed method of explicit FMO map generation and error concealment scheme could reduce the number of undecodable MB and improve quality of video under wireless network.

I.INTRODUCTION Wireless transmission has been used to transmit or receive multimedia such as image, sound, video streaming and video conference that can offer an acceptable quality of service to the end user. Nevertheless, the bandwidth of channel is limited, so the use of efficient coding scheme to compress the video signal is necessary. H.264/AVC [1] is the best candidate codec for wireless multimedia application. One of the features of H.264/AVC is the adoption of a robust error resilience tool at the encoder known as Flexible Macroblock Ordering (FMO) [2] that allows flexible means of macroblock grouping. FMO can significantly enhance the robustness of data losses by managing the spatial relationship between regions that are coded in each slice and also serves as a macroblock level interleaving tool that can spread out consecutive burst error. Some recent work in FMO studied different methods on how to quantify the importance of a MB. In [3], the error sensitivity of a MB is determined by computing a MB PSNR parameter. In [4], a MB impact factor is computed which depends on some information derived from the used pixels. In [5], a distortion measure based on the mean square error of the original and reconstructed pixels are used. In [6], a MB importance factor is computed based on two distortion measures, a distortion of the coded MB and a distortion if the MB is lost and concealed and the number of bits of a particular MB. In [7], number of macroblock coded bit-count has been investigated as an indicator for a choice of FMO map of each frame. Using FMO alone may still not be good enough to provide acceptable quality of video data in term of human visual perception and peak signal to noise ratio (PSNR) when

transmitted by wireless channel. As a way to improve the visual quality, error concealment is very useful; since the decoded frame that have lost some macroblocks may still have spatial and temporal redundancy. Several approaches to error concealment have been proposed to enhance the performance. The reference software of H.264/AVC also uses a temporal error concealment based in boundary matching algorithm (BMA) [8]. In [9], a motion vector recovery method is proposed using optical flow in MPEG2. A set of error concealment scheme to improve error resilience ability for video consumer application is presented in [10]. In [11], frame copy and motion vector copy is used to recover frame loss in the JM standard reference software to improve quality of video. In [12], initial approach to combine error concealment and pre-defined FMO has been proposed. Nevertheless the relation between FMO and error concealment has not been exploited. In this paper, we analyze the relation between explicit FMO and error concealment and propose the joint approach of using explicit FMO and error concealment by using encoded residual information. We investigate the use of distortion measure that takes into consideration spatial and temporal error concealment schemes simulated at the encoder in designing explicit FMO map and combined with error concealment method at decoder using residual information from the simulated error concealment method at encoder. This paper is organized as follows. Some background on FMO and spatial and temporal error concealment schemes are presented in Section II. Section III describes our proposed method of using FMO together with error concealment simulation at encoder and combined with error concealment at decoder. In Section IV, the simulation results are discussed. Conclusions and future works are given in Section V II. FMO IN H.264/AVC The Flexible Macroblock Ordering (FMO) in H.264 allows flexible means of macroblock grouping. The order of coding macroblock for each picture can be done in a non-rasterscan order. The groups of macroblocks, called Slice, can be altered differently for each picture and are independently intra-predicted. With the current implementation of H.264, each macroblock can be mapped to a particular slice through the MBAmap data structure. However, the macroblock order

978-1-4244-4522-6/09/$25.00 2009 IEEE

1269

ISCIT 2009

within a slice must be in ascending order. The FMO scheme supports six different slice group map types; fives of them are predefined macroblock mapping that can be specified through the picture parameter sets (PPS). FMO type 6, i.e., explicit slice group map type, is the most general type where the entire MBAmap is actually coded into a picture parameter set. The explicit slice group map type (Type 6) allows the full flexibility of assigning macroblock to any slice; it also requires slightly higher number of extra header bits. There is no rule on how to assign slice group mapping for the explicit type and the FMO pattern can assigned differently for each picture through the first picture header and can be altered in subsequent pictures. This allows the option to turn on and off the FMO feature while coding the sequence. Using FMO can thus effectively limit any error-propagated macroblock within one slice. Overheads required for FMO are acceptable, it has been observed that the total overhead incurred by omitting inpicture prediction is normally less than 12-15% with eight slice groups, bit rate of 32 kbps at 10 frames per second and rate-control enabled [13]. III. ERROR CONCEALMENT IN H.264/AVC Errors can occur during the transmission of a compressed video stream. The effect of these errors is that parts of the video sequence cannot be decoded properly. In addition, errors contribute to loss of quality and appearance of visible artifacts in the decoded output. In real-time applications, often retransmission is not possible; so the missing parts of the video have to be concealed. To be able to conceal the errors, the spatial and temporal correlation within the video sequence can be exploited. A. Spatial Error Concealment The simplest and often used method is weight averaging [13]. Each pixel p ( i, j ) of a missing macroblock is interpolated as a linear combination of the nearest pixels in the boundaries using (1) as illustrated in Fig. 1.

Fig. 1. weighted averaging

Temporal Error Concealment This kind of error concealment takes advantage of the temporal correlation of the sequence to conceal the error. There are two types of temporal error concealment: nonmotion compensated and motion compensated. The simplest and often used method is the non-motion compensated temporal error concealment, it assumes that the motion vector of the MB in the previous frame is zero; hence the macroblock at the corresponding position in the previous frame is used for concealment. Motion compensated error concealment however tries to estimate the missing motion vectors in the current frame using the previous frames then motion compensation is performed to reconstruct the missing data. The simplest motion compensated error concealment is motion vector copy [11], where the motion vectors in the previous frame is used for motion compensation in the current frame. IV.JOINT EXPLICIT FMO MAP AND ERROR CONCEALMENT FOR WIRELESS
VIDEO TRANSMISION

B.

p ( i, j ) =

d R pL + d L pR + d B pT + d B pT d L + d R + dT + d B

(1)

nearest pixel pL = p ( i, 0 ) in the left boundary.

d L is the distance between the interpolated pixel and the d R is the distance between the interpolated pixel and the

nearest pixel pR = p ( i, N + 1) in the right boundary. dT is the distance between the interpolated pixel and the nearest pixel pT = p ( 0, j ) in the top boundary. d B is the distance between the interpolated pixel and the

Specially, with the fast growing demand for accessing multimedia content through the internet and wireless networks, robust video delivery has become an important issue. To increase the error resilience against the harsh conditions imposed during transmission. H.264/AVC has a new error resilient tool called FMO. FMO can be used to improve the error concealment performance at the decoder if the spatial and temporal characteristics of the error can be exploited which leads to improved delivery of video quality against the wireless channel impairments. In this work a framework is proposed that uses explicit FMO in combination with error concealment method. We propose new method to generate explicit FMO map based on distortion by simulating spatial and temporal error concealment at the encoder. For the decoding process, we use residual information from encoder process in the error concealment to recover the information of the lost macroblock. A block diagram of the proposed wireless video transmission system is shown in Fig. 2.

nearest pixel pB = p ( N + 1, j ) in the bottom boundary. N is size of the block.

1270

Fig. 2. Block diagram of proposed method

Explicit FMO Map using Distortion Measure from Spatial and Temporal Error Concealment We proposed to use the explicit map type of FMO and compute distortion measure from spatial and temporal error concealment simulation at the encoder to determine the importance of a particular macroblock. The important macroblocks will then be interleaved to different slice groups to improve error resiliency. To generate MB-to-slice group map, a two pass encoding scheme is used. During the first pass the necessary information of the macroblocks are collected such as the motion vector and the residual mean absolute difference (MAD). This information is used to generate the appropriate MB-to-slice group map. In the second pass, the video is encoded with FMO enabled using the generated explicit FMO map. From our previous work in [7], the number of MB coded bit count is used as a spatial indicator of MB importance. In [16], a distortion measure due to the error concealment is used as a temporal indicator of MB importance. In this work, we propose another indicator, a residual information, derived from the mean absolute difference (MAD) of the macroblock due to motion estimation and the potential distortion measure of a given error concealment technique. The residual information is computed after the first pass of the encoding sequence. First, we assume that each MB in the frame will be lost and concealed by either spatial or temporal error concealment. If the MB is lost and concealed then a certain amount of distortion is incurred, measure as Dis using (2). We compute two types of distortion, one is if spatial error concealment (weight interpolation) is used, and the other if a temporal error concealment (motion vector copy) is used. After computing the distortion due to error concealment, we then compare the distortions with the MAD of the MB, this is the residual information, res , computed using (3). If the resulting residual information is less when weight interpolation is used, then it means the MB can be better concealed by using spatial error concealment. For each frame, the distortion of using spatial or temporal error concealment of each MB is compared with the MAD and the resulting residual information is computed as in (2) and (3). (2) Dis = f k ( x, y ) f k ( x, y)
( x, y)

A.

Where frame k is the current frame, fk ( x, y ) is the concealed pixel value at coordinate ( x, y ) , f k ( x, y ) is original video sequence, Dis(m, n) is the distortion if a MB is lost and concealed at coordinate (m, n) , res is a residual information and L is the damaged area due to packet loss. For each frame, the number of MB that can be better concealed (lower residual information) using spatial or temporal error concealment is determined. If the majority of the MB for that frame can be better concealed using spatial error concealment, then the residual information for that frame is computed assuming spatial error concealment will be used for the entire frame. The same is also done if the majority of MBs in the frame can be better concealed using temporal error concealment. The residual information, res , is then used to sort the macroblocks in descending order. The macroblocks are then assigned to different slice groups beginning with the macroblock with the highest residual information and following the order or the sorted list. An example of the MBAmap of the 9th frame of the news sequence using the slice group mapping technique based on error concealment simulation is shown in Fig. 3.
3 6 5 3 7 1 6 0 2 4 7 6 0 5 2 7 1 3 5 0 7 4 6 0 3 2 4 6 4 3 2 1 7 0 3 5 7 1 1 4 7 6 1 4 6 0 2 5 2 6 3 2 5 7 1 3 0 1 2 1 3 6 0 2 7 5 0 6 2 4 7 1 3 2 0 4 0 4 5 0 1 4 3 1 5 3 4 6 5 7 5 4 2 6 0 5 7 1 2

Fig. 3. MBAmap of 9 th frame of News sequence

The steps of generate FMO map. 1. Simulate spatial error concealment (SEC), compute fk ( x, y ) and calculate distortion of spatial error concealment, DisSEC , by (2) Simulate temporal error concealment (TEC), compute fk ( x, y ) and calculate distortion of temporal error concealment, DisTEC , by (2) 3. Compare DisSEC and DisTEC for each MBs. Count the number of MBs with DisSEC < DisTEC , denoted as MBcntSEC . Also count the number of MBs with DisTEC < DisSEC and denote as MBcntTEC

2.

res =

( m , n L )

Disk (m, n) MADk (m, n)

(3)

1271

4.

Generate a new explicit FMO map based on MBcntSEC and MBcntTEC of each frame: IF MBcntSEC > MBcntTEC , use Spatial-FMO and use DisSEC in (3) to compute res . ELSE MBcntSEC < MBcntTEC , use Temporal-FMO and use DisTEC in (3) to compute res .

channels, the maximum Doppler frequency parameter is set to 1 Hz and 40 Hz respectively. The average packet error rate (PER) is 0.1 and bit-error rate (BER) of 0.06 for a fixed packet size of 80-bit. We used the average PSNR and the number of undecodable MBs to evaluate the performance of our proposed technique. Having a lower number of undecodable MB indicates improved visual quality but it does not directly translate to a better average PSNR because not all MBs have the same effect on the PSNR. Table 1 summarizes the average PSNR of the test videos in the scenario of slow and fast fading channels. For the slow fading case the results show that the PSNR improvement of the proposed method is up to 5 dB compared to No FMO, [7] and [16]. For fast fading case, the proposed method can improve PSNR of up to 4.51 dB. As mentioned, the quality measurement of video in term of the PSNR depends on the location of the bit errors and also the error concealment method applied. Nevertheless, there is no direct relation between the number of undecodable macroblock and the PSNR as several other factors have to be taken into account. Table 2 shows the number of undecodable macroblocks for slow and fast fading channel conditions. For slow fading case, the proposed method can reduce, on the average, the number of undecodable macroblocks of up to 77% compared to No FMO, [7] and [16].While in the case of fast fading, the average number of undecodable macroblocks could be reduce of up to 81%. Fig. 4-5 show the average PSNR curve of the news test sequence under slow and fast fading conditions. From the curves, it can be clearly observed the PSNR improvement of the proposed method. In some cases, the number of undecodable MBs with explicit FMO using bit count information is very close with the proposed technique for sequences with low motion content. If there is little motion, such as news, bit count information based on spatial information is better can reduce the number of undecodable MB more than the proposed technique, but the difference is slight. Overall, using explicit FMO based on temporal information on the average can improve the PSNR with the number of undecodable MB being comparable. This is because of the unequal importance of each MB. VI.CONCLUSIONS

5.

Sort the MB address for each frame based on the residual information, res , and assign slice group id 0 to 7 to the sorted list to assign important MB to different slice groups.

Error Concealment method However, error-resilience in the encoder still cannot guarantee to make the data stream fully decodable in high error rate condition. If channel coding is used at the encoder, at the decoder the received video packets from the wireless channel undergo channel decoding to correct some of the errors introduced during transmission. Practical channel encoding and decoding schemes for video transmission do not provide perfect error recovery from transmission errors because this would require large bandwidth overhead. In practice a certain amount of error can be tolerated in the decoder since the human visual perception can tolerate some degree of distortion and visual artifacts In order to further reduce quality degradation caused by error, error concealment techniques at decoder are required. The choice of error concealment techniques used would contribute to the improvement of the received video quality. In this paper, we proposed an error concealment scheme to recover from MB loss and modify algorithm by using residual information, res, in (3) and calculate pixel loss from (4).
i f n ( x, y ) = p resn + f n ( x, y ) + ( 1 p ) f n 1 ( x, y )

B.

(4)

pixel in error-free condition at the coordinate ( x, y ) , f n 1 ( x, y ) is the reconstructed pixel the previous frame at the
i coordinate ( x, y ) , and resn is residual information.

when p is packet error rate, f n ( x, y ) is the reconstruct

V.SIMULATIONS AND RESULTS Several video sequences are encoded using the baseline profile at level 3.0 for the experiments. Standard test video sequences are used namely, mother, mobile, news and salesman. Each sequence is encoded for a total of 100 frames with frame rate of 10 fps. The rate-control is enabled at fixed bit rate of 32 kbps. Detailed description of the other H.264 JM codec parameter usage can be found in [14]. The wireless channel is simulated using a Rayleigh fading wireless channel simulator. The details of the simulator can be found in [15]. To simulate the effects of slow and fast fading

In this paper, the framework of using joint explicit FMO map based on distortion simulated from spatial and temporal error concealment at the encoder and a new error concealment at decoder. Our simulation results show the benefit of using explicit FMO especially when using higher number of slice group per picture and apply new error concealment scheme. The amount of undecodable MBs is decreased of up to 81% compared to no FMO and the PSNR improvement is up to 5 dB. The use of error concealment at the decoder and its effectiveness combined with explicit FMO is currently under study.

1272

ACKNOWLEDGMENT This work has been supported in part by CHE-AUN/SeedNet Scholarship.

TABLE I
COMPARISION OF AVERAGE PSNR

mother slow 28.48 31.58 31.96 32.35 fast 25.11 28.09 27.38 28.85

moblie slow 15.30 16.53 16.50 16.78 fast 12.65 14.99 14.91 15.47

news Slow 24.00 26.73 25.77 27.99 fast 20.56 23.31 23.39 24.89

salesman slow 25.89 29.15 29.73 30.89 fast 24.59 27.87 27.12 27.66

REFERENCES
[1] ITU-T Recommendation H.264: "Advance video coding for generic audiovisual, March 2005. [2] P. Lambert, W. D. Neve, Y. Dhondt, and R.V. de Walle, Flexible Macroblock Ordering in H.264/AVC,Journal of Visual Comms. and Image Repres, vol. 17, pp. 358-375, April 2006. [3] J. Lie, and K. Ngi Ngan, An Error Sensitivity-based Redundant Macroblock Strategy for Robust Wireless Video Transmission, Intl Conf. on Wireless Networks, Comms. and Mobile Computing, 2005. [4] Y. Dhondt, P. Lambert, R. Van der Walle, A Flexible Macroblock scheme for Unequal Error Protection, IEEE Interc. on Image Proc.(ICIP), October 2006. [5] N. Thomos, S.Argyropoulos, N.V.Boulgouris, and M.G.Strintzis: "Error resilient transmission of H.264/AVC streams using FMO", 2nd European Workshop on the Integration of Knowledge, Semantic and Digital Media Technologies(EWIMT05), London, UK, November 2005. [6] S. Im, A.J. Pearmain, Unequal error protection with the H.264 flexible macroblock ordering, Proc. of SPIE, Visual Comms. and Image Proc., 2005. [7] W. Hantanong, and S. Aramvith, Analysis of Macroblock-toslice Group Mapping for H.264 Video Transmission over Packet-Based Wireless Fading Channel, 48th Midwest Symp. on Circuit and Systems, Vol. 2, pp.1541-1544, August 2005. [8] T.Chen, X.Zhang, and Y.Q.Shi, Error Concealment using Refined Boundary Matching Algorithm, IEEE Trans. on Circ. and Syst. for Video Tech., July 2003. [9] J.W.Suh and Y.S.Ho, Error Concealment Technique for Digital TV, IEEE Trans. Broadcasting, December 2002. [10] Y.Xu, and Y.Zhou, H.264 Video Communication Based Refined Error Concealment Schemes, IEEE Trans. Consumer Electronics, November 2004. [11] ITU-T Recommendation H.264,"Frame Loss Error Concealment for H.264/AVC, July 2005. [12] Agrafiotis, D., Bull, D.R., Chiew, T.K., Ferre, P., and Nix, A, Enhanced error concealment for video transmission over WLANS, Proc. Intl Workshop on Image Analysis for Multimedia Interactive Services, 2005. [13] S. Wenger and M. Horowitz, FMO: Flexible Macroblock Ordering, JVT-C089, Fairfax (USA), May 2002. [14] A. Michael, K. Shring, and G. Sullivan, Proposed H.264/MPEG-4 AVC Reference Software Manual, Joint Video Team, Doc. JVTM012, Plama (Spain), October 2004. [15] T.C. Chen, et al., A Real-Time Software Based End-To-End Wireless Visual Comms. Simulation Platform, Proc. SPIE Visual Comms. and Image Proc., vol. 3, pp. 1068-1074, May 1995. [16] J. Panyavaraporn, R. Cajote, S. Aramvith, Performance Analysis of FMO using Bit-count and Distortion measure for H.264 Wireless Video Transmission, Intl Workshop on SISB 2007, Bangkok, Thailand, Nov 2007.

TABLE III
COMPARISION OF NUMBER OF UNDECODABLE MBS

mother slow No FMO FMO_bitcnt [7] FMO_Dist [16] Proposed 1038 764 777 759 fast 4978 1039 1130 1076

moblie slow 1382 394 470 316 fast 4168 860 933 810 slow 1424 651 692 658

news fast 5754 1220 1298 1318

salesman slow 1383 563 547 558 fast 5011 1115 1164 1124

Fig. 4. Comparison of PSNR among 5 methods with respect to clean channel for News test sequence under slow fading channel

Fig. 5. Comparison of PSNR among 5 methods with respect to clean channel for News test sequence under fast fading channel

1273