pe Image Processing ‘Syllabus Pixel transforms, color transforms, histogram processing, histogram equalization, filtering, convolution, Fourier transformation and its applications in sharpening, blurring and noise removal. Contents 2.1 Pixel Transforms 2.2 Color Transforms 2.3 Histogram Processing 2.4 Histogram Equalization 2.5 Filtering 2.6 Convolution 27 Fourier Transformation 2.8 Applications in Sharpening, Blurring and Noise Removal (2-1)Computer Vision i tors 2 Pein opera sing transformations are point operators, in which the y by the value of the associated input pix formation or parameters). Adjustments 4, The most basic image proces value of each output pixel is determined sole! odin (plus, potentially, some globally collectes : brightness and contrast are examples of such operators. Point processes are another name for such procedures. Pixel Transforms ‘* A function that takes one or more input images and produces an output image is called as a image processing operator. This can be written as in the continuous domain, g(x) = (F(X) oF g(x) = UG), ---, fr(%)), 1) ‘+ where x is in the function D-dimensional domain (typically D = 2 for images) and the functions f and g are the operate across a range, which can be the scalar or vector-valued, for example, for color images or 2D motion. For discrete (sampled) images, the domain consists of a finite number of pixel locations, x = (ij), and this can be written aij) = hEG)). (212) * Two commonly used in point processes operation are multiplication and addition with a constant, g(x) = af(x)+b. @13) * The gain and the bias parameters, a > 0 and b, are frequently referred to as the contrast and brightness controls, respectively. * bias and gain parameters can also be spatially varying is given as, B(x) = alx)f (x) +b(x), : 14) For example, when modeling vignetting in an optical system or simulating the graded density filter used by photographers to selectively darken the sky. Multiplicative gain (both global and spatially variable) is a linear operation that follows the notion of superposition, hi +f) = biG) +(e) 15) | Image squaring (which is frequently employed to derive a local estimate of the energy in a band-pass filtered signal) is not a linear operator. The linear mix extensively used dyadic (two-input) operator, 8G) = (1-af(x) + af, (0, a8 operator is another TEOHNICAL PUBLICATIONS® an pas orinonndpoComputer Vision Image Processing By varying @ from 0 — 1, this operator can be used to perform a temporal cross-dissolve between two images or videos, as see’ ‘ras a component of the image morphing algorithms slide shows and film production, ‘The gamma correction, which is used to remove the non-linear mapping between input radiance and quantization pixel values, is a widely used non-linear transform that is typically performed to images before further processing, To invert the sensor's gamma mapping, with a gamma value of y BX) = [FQ] (2.1.7) .2 being an acceptable fit for most digital cameras. Color Transforms While color images can be treated as arbitrary vector-valued functions or collections of the independent bands, it usually makes sense to think about them as highly correlated signals with strong connections to the image formation process, sensor design, and human perception. Consider the effect of adding a constant value to all three channels to brighten a picture. Adding the same value to each éolor channel not only enhances the perceived intensity of each pixel, but it can also change the hue and saturation of the a (a) Source image (b) Extracted foreground (¢)Alpha matte a (4) New composite C object F ‘shown in grayscale pixel. 2.4 Image matting and compositing ‘After modifying (e.g,, brightening) the luminance Y, chromaticity coordinates or even simpler color ratios can be utilized to re-compute a valid RGB image with the same hue and saturation. For improved visibility, Fig, 2.2.1 illustrates some color ratio images multiplied by the middle gray value. Color balancing (for example, to compensate for incandescent lighting) can be achieved by multiplying each channel with a different scale factor, or by a more involved process of mapping to XYZ color space, changing the nominal white point, and mapping back to RGB, which can be written down using a linear 3 3 color twist transform matrix. Compositing and matting It’s common to wish to clip a foreground object from one image and set it on top of another in various picture editing and visual effects tools. Matting is the technique of TECHNICAL PUBLICATIONS® - an up-tust for knowiodgeImage 2-4 Computer Vision is the proce: of inco compositing Poratn ge, whereas yund object's % rious atts). The foreground Objects inter 8 it into another image (without ol wiation tothe thee color RGB cha Nnels, ret i en these two stag- that describes wrency is the polar o, teach pixel . Transpai reg amount of opacity or fractional coverage at ES eran cite P fea a inside the object are fully opaque (a = 1), ie . a ae the o Pixels i : aoe Objet ae completely transparent (a ~ 0) The perceived ee a at he only binary opacities are employed are hidden by pixels or 1B varying smoothly between these two extremes. eliminating an object from an ima} a! ee yee : B a oF c ) C) @ Fig, 22.2 Compositing equation C = (1 ~ a)B + oF . The images are taken from a close-up ofthe region of the hair in the upper right part of the lion. ‘To composite a new (or foreground) image on top of an old (background) image, the over operator, first proposed by Porter and Duff (1984) and then studied extensively by Blinn, is used, C= (1-a)B +aF. 21) This operator attenuates the influence of the background image B by a factor (I) and then adds in the color (and opacity) values corresponding to the foreground layer, as shown in Fig, 2.2.2. In many situations, it is convenient to represent the foreground colors in Pre-multiplied form, i. to store (and manipulate) the aF values directly. For various reasons, including the ability to blur or resample (eg, rotate) alpha-matted images without any additional complexities, the pre-multiplied RGBA representation is preferable (just treating each RGBA band independently). Because the pure un-multiplied foreground colors F remain constant (or move slowly) in the neighborhood of the objet they are employed for matting utilizing local color consistency. TECHNICAL PUBLICATIONS? - en uptinasttrinocaooComputer Vision 28 Histogram Processing A histogram is a graph that shows frequency of anything. Usually histogram have bars that represent frequency of occurring of data in the whole data set. The histogram of a picture, like other histograms, shows frequency. In contrast, a picture histogram shows the frequency of pixel intensity levels. In an image histogram, the gray level intensi are shown on the x axis, and the frequency of these intensities is shown on the y axis. The x axis and the y axis are the two axes of a histogram. The x axis contains event whose frequency you have to count. The y axis contains frequency. ees Fig, 2.3.1 Histogram Processing ‘The histogram of a digital image with gray levels in the range (0, L-1] is a discrete function of the form H(t) = nk where rk is the k gray level and nk is the number of pixels in the image having the level rk A normalized histogram is given by the equation p(k) = nk/n for k=0,1,2,....L-1 P(rk) gives the estimate of the probability of occurrence of gray level rk. The sum of all. components of a normalized histogram is equal to 1.The histogram plots are simple plots of p(rk)=nk versus rk. The components of the histogram are concentrated on the low (dark) side of the gray scale in the dark image. In the case of a bright image, the histogram components are skewed to the gray scale's high end. A low contrast image's histogram will be narrow and centered near the middle of the gray scale. The histogram components in the high contrast image cover a wide range of gray scale. The end result will be an image with a lot of gray level details and a wide dynamic range. TECHNICAL PUBLICATIONS - an upthust or hrowndge‘ Computer Vision = an Histogram ea" Vawcotaaiage ere en = : oat " Number of pixels aaa es 13s. ostz | y Ul ~ L— Load ll yk oa bo 025 050 0.75 1.00 0.00 025 050 075 1.09 77 075 100 0} 9 200 emer Seater Pe tenet Pa tenaiy Fig. 2.3.2 Histogram equalization Histogram Equalization Equalization of histograms is a typical approach for improving the appearance of photographs. Assume that we have a largely dark image. The visual detail is compressed towards the dark end of the histogram, and the histogram is skewed towards the lower end of the grey scale. If we could ‘stretch out’ the grey levels at the dark end to produce a more uniformly distributed histogram then the image would become much clearer. Let there be a continuous function with r being gray levels of the image to be enhanced. The range of r is (0, 1] with r = 0 representing black and r = 1 representing white. The transformation function is of the form. One approach might be to look at the darkest and brightest pixel values in an image and map them to pure black and pure white. Another approach might be to find the average value in the image, push it towards middle gray, and expand the range so that it more closely fills the displayable values S=T(r) where 0 Pq = Weighted Pei median Pax Fig. 2.5.5 Weighted median filter with weight matrix for sort EX convolution Overlay a revolving mask on the image. This is how it is done. Place the mask's center at the middle of each picture element. Multiply the corresponding parts, then put them together, and paste the result into the image element where you want to set the mask's center. To begin, breakdown the input signal into a series of impulses, each of which can be seen as a scaled and shifted delta function. Second, each impulse produces an output that is a scaled and shifted replica of the impulse response. Finally, by summing these scaled and shifted impulse responses, the overall output signal may be obtained. In other applications, the impulse response is referred to by a different term. The impulse response is known as the filter kernel, the convolution kernel, or simply the kernel if the system under consideration is a filter. The point spread function is the name given to the impulse response in image processing. While these phrases are employed in a variety of ways, they all refer to the same signal produced by a system when the input is a delta function. Convolution, like multiplication, addition, and integration, is a formal mathematical operation. Convolution takes two signals and produces a third signal, whereas addition takes two numbers and produces a third number. TECHNICAL PUBLICATIONS® -an ups for krowedboFig, 26. Delta function and impulse response Convolution is utilized in many branches of mathematics, including statistics, Convolution is a mathematical term that describes the relationshi three signals of interest: The input signal, the impulse response, and the 0 linear systems. When convolution is employed with linear systems, the in Fig. 262. A linear system having an impulse response, h[n], receives an x(n}, and produces an output signal, y{n}-x{n] * hin] = y[n] in equation form, Expressed in words, the input signal convolved with the i : ds the the output gaa Just as edlton represented by te pda aComputer Vision 2-15 Image Processing The mask is represented by the red box, and the values in orange are the mask's values. The image owns the black color box and its values. The value for the the image will now be calculated as, (52) + (4"4) + 2°8) + ("10 10+16+16+10 = 32 rst pixel of First pixel Place 52 in the original image at the first index and repeat this procedure for each pixel of the image. Fourier Transformation A complex-valued function of frequency, whose magnitude (absolute value) is the amount of that frequency present in the original function, and whose argument is the phase offset of the fundamental sinusoid at that frequency, is the Fourier transform of a function of time. Although the Fourier transform is not confined to time functions, the original function's domain is typically referred to as the time domain. The Fourier Transform is an important image processing tool which is used to divide a picture into its sine and cosine components. The image in the Fourier or frequency domain is represented by the output of the transformation, whereas the spatial domain equivalent is represented by the input image. Each point in the Fourier domain image indicates a frequency contained in the spatial domain image. E Fiter Trverse afouer | exp] unction |x| Fourier Ate) 7 | wat woven § fox) ey) Fig. 2.7.1 Fourier transformation 4-D Fourier Transformation and its Inverse Fourier transform is one of the most commonly used techniques in (linear) signal processing and control theory. It provides one-to-one transform of signals from / to a time-domain representation {(t) to/from a frequency domain representation F(E). It allows a frequency content (spectral) analysis of a signal, FT is suitable for periodic TECHNICAL PUBLICATIONS® -an up Prst or knowodgeImage Computer Vision 2.16 Freee en the windowed FT or the linear integra ae aie signals. If the signal is not periodic oe transformation with time (spatially in 2D) localized basis function, Fm filters can be used. F{f(t)} = F(E), where £ [Hz = -1 ] is a frequency and 27 [5-1 ]is the angular frequency Fourier Tx Inverse Fourier Tx F@= J fe*at Fe = J Fer ag Convolution (in functional analysis) is an operation on two functions f and h, which Produces a third function (f * h), often used to create a modification of one of the input functions. Convolution is an integral ‘mixing’ values of two functions, i., of the function 1(t), which is shifted and overlayed with the function f(t) or vice-versa. The limits can be constraint to the interval [0, t], because we assume zero values of functions for the negative argument. t t (HO = HH = Stent ar = fie ar 0 0 Fourier Tx, properties Property f(t) Fe Linearity af, (0) +b E(t) a, (6) + bF, (8) Duality Lig) FCS) Convolution (orxo FQ) Product fg) FOO Time Shift F(t-t,) etl f(t) Frequency shift oF FE-E) Differentiation ate) 2nigFE) at Multiplication by t ro) Time scaling fat TECHNICAL PUBLICATIONS® - an uptnst lr nowedge Eeecomputer Vision 2-17 mage Processing a ae pres ah 1w--f-ao & ‘Area function f() ‘Area in freq, FO)- J Hae Area under F() Parseval’s th, Jameaes J i@erae fenergy =F energy Discrete Fourier transform Let f(n) be an input signal (a sequence), n= 0,...,N~-1, Let F(k) be a Frequency spectrum (the result of the discrete Fourier transformation) of a signal f(n). If there is a single variable, continuous function f(n) , then Fourier transformation F (k) may be given as, Discrete Fourier transformation N-L RK)= LD Fine n=0 Inverse discrete Fourier transformation Because the DFT is a sampled Fourier Transform, it does not contain all of the frequencies that make up an image, but only a sufficient number of samples to fully characterize the spatial domain image. The number of frequencies is the same as the number of pixels in the spatial domain image, indicating that the spatial and Fourier domain images are of equal size. Fora square image of size NxN, the two-dimensional DFT is given by : N-1N-1 on (kits Ay Fk p- X D fijpe inl N 0 j-0 TECHNICAL PUBLICATIONS® -an up.tust orknowedgerE —— Image Proc : sponding correspon Th term is the basis FUNCT TT ppain. The Value of each point Fy he exponential im is th bi oti Fourier space, and f(a,b) is the pic bya 1g the spatial picture hb the associated base fun ial picture with the liplying sation. summing the result, as shown in the eq _ a OFF Transformed Origir image imege Fig. 27.2 Applying DFT and IDFT Fast Fourier Transform (FFT) er a x0}. Aw eu Xi] x(2]o—-—) fi ex [ a x4) hy 2 x13) 61 py Xx(4] ate (t) wa xo a x8) Point wi OFF oo fess we xTlo—-— oy ‘xm Fig. 2.73 Fast Fourier Transform (FFT) A Fast Fourier Transform (FFT) is an algorithm that computes the discrete Fouret transform (DFT) of a sequence, or its inverse (IDFT). Fourier analysis converts a sign! from its original domain (often time or space) to a representation in the frequency domain and vice versa. The DFT is obtained by decomposing a sequence of values int? components of different frequencies. This operation useful in many fields, but 80 up-hatfr knowledge aComputer Vision 2-19 Image Processing computing it directly from the definition is often too slow to be practical. A Fast For Transform (FFT) is an efficient algorithm to compute the discrete Fourier transform and its inverse. Statement : FFT has the complexity O(Nlog 2N). Example (according to numerical recepies in C) : A sequence of N='106 , 1 tssecond computer. FFT 30 seconds of CPU time. DFT 2 weeks of CPU time, ie, 1,209,600 seconds, which is about 40,000 x more. 2.0 Fourier Transform ‘The Fourier transform can be applied to a larger number of dimensions. Many signals, for example, are 2D space functions defined on an x-y plane. Depending on whether the 2D signal is periodic or discrete, the two-dimensional Fourier transform has four possible forms. ‘periodic, continuous signal, continuous, aperiodic spectrum toy) = Jf Fu yer du dy Fy = Jf (yer day Where u and v are spatial frequencies in x and y directions, respectively, and F(x,y) is the 2D spectrum of f(x,y). Aperiodic, discrete signal, continuous, periodic spectrum Fuy= LL fmm metry) ideas ge UV te .) ,i2n(umy, + vny,) du dv iv J Jrume 00 ffm, n] = Where X and Y are periods ofthe signal in x and y directions, respectively, and u,=5 and vy =1/Y are the intervals between consecutive samples in the spectrum F{k,I] e Removal E21 Applications in Sharpening, Blurring and N ‘The enhancement of images through the use of sharpening and noise reduction processes, which need some form of neighborhood processing, is another popular application of image processing. TECHNICAL PUBLICATIONS® - an vp tnt for knowiedgeImage Proc Campa Vata a Blurring An image-Low Pass Filtering Convolution in the spatial dor 's one of the most esse! tical t0 simple multiplication ing main is ident | features of Fourier Transforms, spatial e hha i's called. The most basic filter jg “ a most important aspects of Four, 4 ial dot s Transforms is that convolution in the spat aie fundamental convolution filters yo. a frequency domain, which i -convolve option blurs an image in the $ convolution filters (kernels). A low pass AIS, square array with equal weights. One 0! the frequency domain, Using modest, $44 tial domain : eae oe ts Bae the most basic filter. This is the same as pass filter. A square array with equal wei ly shaped filter offere neighborhood or local average. The Gaussanweighit Gireularly shapes ‘d by either -gaussian-blur or eee wipes ode aoxot IEW pass bluting ee ean te ope component, and there is no (or a zero) phase (or imaginary) compo ae Spatial domain, this filter would be analogous to a circularly shaped averaging convolution filter Because convolution in the spatial domain is identical to multiplication in the frequency domain, all we have to do is forward Fourier transform the image, multiply the filter wi the magnitude image, and then inverse Fourier transform the product. We can see that in the frequency domain, a tiny convolution filter corresponds to a huge circle Multiplication is done with a -compose multiply setting and -composite. (kernels). I's referred to as a toy, n filter corresponds to a huge circle So let's try it with two different sizes of circular filters, (radius 24) and the other with a diameter of 32 ( spectrum and the masked spectrum, masked magnitude. ‘one with a diameter of 48 (radius 32). (radius 16). We show both the however the blurred effect is created using the » Asa result, we can see that the image with the smaller diameter filtet TECHNICAL PUBLICATIONS®- a upstatforinoniodpo‘computer Vision 2-21 Image Processing has more blurring. In the generated photos, we also notice a ‘ringing’ or ‘ripple’ effect at the edges. This happens because, as we learned earlier, the Fourier Transform of a circle is ajinc function, which has decreasing oscillations as it moves away from the center. The jinc function and oscillations, on the other hand, are in the spatial domain rather than the frequency domain, as we saw before. Sharpening An Image - High Boost Filtering The simplest technique to sharpen an image is to apply a high pass filter to it (without stretching it) and then mix it with the original. In this scenario, the high pass filtering is performed in the frequency domain, with the output being translated back to the spatial domain and blended with the original image. Fig. 2.8.2 Sharpen an image is to apply a high pass filter, Noise Removal - Notch Filtering Patterned noise can be found in a lot of noisy photos. In the frequency domain, this type of noise is easier to remove because the pattems appear as a pattern of a few dots or lines. Remember that a basic sine wave is a repeating pattern that appears in the spectrum as simply three dots. To get rid of the noise, all one has to do is manually mask (or notch) away the dots or lines in the magnitude image. This is accomplished by translating to the frequency domain, creating a grayscale version of the spectrum, masking the dots or lines, thresholding it, multiplying the binary mask image with the magnitude image, and then transforming back to the spatial domain, Let's try it on the clown image, which has a dither-like diagonally striped pattem. We start by transforming the clown image into magnitude and phase images. TECHNICAL PUBLICATIONS® - an up-to knowedgeComputer Vision ratyde and phase images “age from the magnitude, but we add Following that, we construct the eee to gray level 1, 50 that gray Pi Spectrum, a bash scrip, is also available jy as input, or the original image may be ide before generating the spectrum, jy the log scaling constant, so in fact, yoy Fig. 2.8.3 Clown image into maa extra step to push any image values at gray zero can be preserved for the masked areas. help with this, The magnitude image can be used provided and it will be converted to the magnitu fact, the script does an adequate job of estimating a4 may leave off the -s argument. We can detect four bright star-like spots in the spectrum, one in each quadrant. The bright dot and lines in the middle are ignored since thay represent the image's DC or zero frequency components, i.e. the constant intensy regions. So we'll have to mask out those star-like spots immediately. We could open this image in ImageMagick and measure the locations and sizes of the star-like spots using te display functions, then use -draw to fill them with black circles or rectangles. However, i is easier to perform this interactively with a tool like GIMP or Photoshop and then threshold the output to a binary picture with ImageMagick. The outcomes are displayed below. a Fig. 2.8.4 Binary picture with ImageMagick ‘You can even color in the mask before removing it. Now we just multiply the mask by the magnitude and translate back to the spatial domain using the result and the ph® image. A Gaussian shaped taper, as we did earlier, or even a simple linear taper 0f5 pixels can be used. It's worth noting that -blur radiusx65000 will result in a linear tape Using the script filter, both of the examples above become much easier. TECHNICAL PUBUICATIONS® on upna rk up-thnst fr knowledge oT