The Nile Programming Language: Declarative

Stream Processing for Media Applications

Dan Amelang
Viewpoints Research Institute
UCSD PhD Student

Southern California Programming Languages and

Systems Workshop – Spring 2011
2D Vector Graphics
 State of the Art – 10,000+ lines of...
Cairo (Firefox, Linux) Skia (Chrome, Android)
... ...
if (lxi > fill_start) if (currE->fLastY == curr_y) {
{ if (currE->fCurveCount < 0) {
ADD_SATURATE_8 (ap + fill_start, if (((SkCubicEdge*)currE)->updateCubic())
fill_size * N_X_FRAC (8), {
lxi - fill_start); SkASSERT(currE->fFirstY == curr_y + 1);
fill_start = lxi; newX = currE->fX;
} goto NEXT_X;
else if (lxi < fill_start) }
{ } else if (currE->fCurveCount > 0) {
ADD_SATURATE_8 (ap + lxi, if (((SkQuadraticEdge*)currE)->updateQuadratic())
N_X_FRAC (8), {
fill_start - lxi); newX = currE->fX;
} goto NEXT_X;
if (rxi < fill_end) }
{ }
ADD_SATURATE_8 (ap + rxi, ...
fill_size * N_X_FRAC (8),
fill_end – rxi);
...
type Color = (a, r, g, b : Real) CompositeTextures (t1 : Texture, t2 : Texture, c : Compositor) : Texture
type Point = (x, y : Real) ⇒ DupZip (t1, t2) → c
type Matrix = (a, b, c, d, e, f : Real)
type Bezier = (A, B, C : Point) ExpandSpans : EdgeSpan >> PixelCoverage
type EdgeSpan = (x, y, c, l : Real) ∀ (x, y, c, l)
type EdgeSample = (x, y, a, h : Real) if c
type PixelCoverage = (x, y, c, ic : Real) >> (x, y, c, 1 - c)
if l > 0
type Texture = Point >> Color << (x + 1, y, 1, l - 1)
type Compositor = (Color, Color) >> Color
ExtractSamplePoints : PixelCoverage >> Point
| (a : Real) | : Real ∀ (x, y, _, _)
{ -a if a < 0, a } >> (x, y)

(a : Real) ◁ (b : Real) : Real ApplyTexture (t : Texture) : EdgeSpan >> (Color, PixelCoverage)

{a if a < b, b} ⇒ ExpandSpans → DupZip (ExtractSamplePoints → t, (→))

(a : Real) ▷ (b : Real) : Real DecomposeBeziers : Bezier >> EdgeSample

{a if a > b, b} ∀ (A, B, C)
inside = (⌊ A ⌋ = ⌊ C ⌋ ∨ ⌈ A ⌉ = ⌈ C ⌉)
(a : Real) ~ (b : Real) : Real if inside.x ∧ inside.y
(a + b) / 2 P = ⌊ A ⌋ ◁ ⌊ C ⌋
w = P.x + 1 - (C.x ~ A.x)
(M : Matrix) ⊗ (A : Point) : Point h = C.y - A.y
(M.a × A.x + M.c × A.y + M.e, M.b × A.x + M.d × A.y + M.f) >> (P.x + 1/2, P.y + 1/2, w × h, h)
else
TransformBeziers (M : Matrix) : Bezier >> Bezier ABBC = (A ~ B) ~ (B ~ C)
∀ (A, B, C) min = ⌊ ABBC ⌋
>> (M ⊗ A, M ⊗ B, M ⊗ C) max = ⌈ ABBC ⌉
nearmin = | ABBC - min | < 0.1
UniformColor (C : Color) : Texture nearmax = | ABBC - max | < 0.1
∀ _ M = {min if nearmin, max if nearmax, ABBC}
>> (C.a, C.a × C.r, C.a × C.g, C.a × C.b) << (M, B ~ C, C) << (A, A ~ B, M)

CompositeOver : Compositor CombineEdgeSamples : EdgeSample >> EdgeSpan

∀ (a, b) (x, y, A, H) = 0
>> a + b × (1 - a.a) ∀ (x', y', a, h)
if y' = y
ClipBeziers (min, max : Point) : Bezier >> Bezier if x' = x
∀ (A, B, C) A' = A + a
bmin = A ◁ B ◁ C H' = H + h
bmax = A ▷ B ▷ C else
inside = min ≤ bmin ∧ bmax ≤ max l = {x' - x - 1 if |H| > 0.5, 0}
outside = bmax ≤ min ∨ max ≤ bmin >> (x, y, |A| ◁ 1, l)
if inside.x ∧ inside.y A' = H + a
>> (A, B, C) H' = H + h
else if outside.x ∨ outside.y else
cA = min ▷ A ◁ max >> (x, y, |A| ◁ 1, 0)
cC = min ▷ C ◁ max A' = a
>> (cA, cA ~ cC, cC) H' = h
else >> (x, y, |A| ◁ 1, 0)
ABBC = (A ~ B) ~ (B ~ C)
nearmin = | ABBC - min | < 0.1 Rasterize : Bezier >> EdgeSpan
nearmax = | ABBC - max | < 0.1 ⇒ DecomposeBeziers → SortBy (@x) → SortBy (@y) → CombineEdgeSamples
M = {min if nearmin, max if nearmax, ABBC}
<< (M, B ~ C, C) << (A, A ~ B, M)
 Talk Overview

• The Nile language

• Dataflow
• Data manipulation
• Graphics rendering in Nile
• Parallelism in Nile
• The “Frankenstein” Application
• Future Work
• Related Work
 Nile Dataflow

• Processes communicate asynchronously via unidirectional,

homogenous, typed streams of structured data
• Single input / single output (with some exceptions)
 Rendering Pipeline

TransformBeziers (matrix) →
ClipBeziers (min, max) →
Rasterize →
ApplyTexture (texture) →
WriteToImage (image)
 Rasterization Pipeline

Rasterize : Bezier >> EdgeSpan

⇒ DecomposeBeziers →
SortBy (@x) → SortBy (@y) →
CombineEdgeSamples
 Texturing Pipeline

ApplyTexture (t : Texture) : EdgeSpan >> (Color, PixelCoverage)

⇒ ExpandSpans →
DupZip (ExtractSamplePoints → t, (→))
 Data Manipulation

• Statically typed with type inference

• Single assignment
• User-defined record types and operators
• Pattern matching
• Syntax for stream I/O
 Data Manipulation

Sum : Real >> Real

s = 0
∀ x
s' = s + x
>> s
>> s
 User-defined Types and Operators

type Point = (x, y : Real)

type Matrix = (a, b, c, d, e, f : Real)
type Bezier = (A, B, C : Point)

(M : Matrix) × (A : Point) : Point

(M.a × A.x + M.c × A.y + M.e,
M.b × A.x + M.d × A.y + M.f)

TransformBeziers (M : Matrix) : Bezier >> Bezier

∀ (A, B, C)
>> (M × A, M × B, M × C)
 Gezira: A 2D Vector Graphics Renderer
Written in Nile
• In under 400 lines of code:
• Anti-aliased rasterization of Bezier shapes
• Affine transformation
• Geometry clipping
• 26 compositing operators
• Texture transformation and extend styles
• Bilinear and bicubic filters
• Gaussian blur
• Multi-stop linear and radial color gradients
• Pen stroke paths with 3 join styles and 3 cap styles
• Geometry bounds calculation
 Workflow

Nile-to-C C compiler
core.nl libgezira.a
gezira.h
bezier.nl
gezira.c
...

libnile.a

libgezira.a app.exe

app.c
Intra-pipeline Parallelism
Inter-pipeline Parallelism
 Instruction-level SIMD Parallelism

(M : Matrix) × (A : Point) : Point

(M.a × A.x + M.c × A.y + M.e,
M.b × A.x + M.d × A.y + M.f)

TransformBeziers (M : Matrix) : Bezier >> Bezier

∀ (A, B, C)
>> (M × A, M × B, M × C)
Process-level Data Parallelism
 Nile Runtime

• Multi-threaded
• Load balancing
• Heap balancing
• User space synchronization
 Speedup on 40 core machine

35

30

25
Speedup

20

15

10

0
1 10 20 30 40

Cores
Meet “Frank”
 Future Work
• More applications
• Data compression (zlib, png)
• Audio/video decoding
• 3D graphics
• Image processing
• More compiler backends
• OpenCL (for GPUs)
• Javascript
• More language features
• Sliding windows
• Feedback networks
• Binary streams
 Related Work

• Programming models
• Kahn Process Networks [Kahn74, Kahn76]
• Dataflow Process Networks [Lee95]
• Programming languages
• Lucid [Wadge85]
• VAL [Acherman79], later SISAL [Mcgraw85]
• Id [Arvind90], later Parallel Haskell (pH) [Nikhil93]
• Streamit [Thies02]
• Single Assignment C [Scholz03] and S-Net [Grelck07]
Extra slides...
 Model of Graphics Rendering?
From Computer Graphics: Principles and Practice [Foley et al.]:
Once the ET has been formed, the following processing steps for the scan-line
algorithm are completed:

1. Set y to the smallest y coordinate that has an entry in the ET, that is, y for the
first nonempty bucket.
2. Initialize the AET to be empty.
3. Repeat until the AET and ET are empty;
(a) Move from ET bucket y to the AET edges whose ymin = y (add entering edges).
(b) Remove from the AET those entries for which y = ymax (edges not involved in
the next scan line), then sort the AET on x.
(c) Fill in desired pixel values on scan line y by using pairs of x coordinates from
the AET (suitably rounded).
(d) Increment y by 1 (to the coordinate of the next scan line).
(e) For each nonvertical edge remaining in the AET, update x for the new y.
TODO: size graph of cairo vs. gezira