Data Dependences: CS 524 - High-Performance Computing

Data Dependences
CS 524 – High-Performance
Computing
Motivating Example: Superscalar Execution
CS 524 (Au 2004/05)- Asim Karim @ LUMS 2

Data Dependences
 Fundamental execution ordering constraints:
S1: a = b + c
S2: d = a * 2
S3: a = c + 2
S4: e = d + c + 2
 S1 must execute before S2 (flow-dependence)
 S2 must execute before S3 (anti-dependence)
 S1 must execute before S3 (output-dependence)
 But, S3 and S4 can execute concurrently
S1
S2
S3 S4
Types of Dependences (1)
 Three types are usually defined:
 Flow-dependence occurs when a variable which is assigned a
value in one statement say S1 is used in another statement,
say S2 later. Written as S1 δf S2
 Anti-dependence occurs when a variable which is used in one
statement say S1 is assigned a value in another statement, say
S2, later. Written as S1 δa S2
 Output dependence occurs when a variable which is assigned
a value in one statement say S1 is later reassigned in another
statement say S2. Written as S1 δo S2

Types of Dependences (2)
 Type of dependence found by using IN and OUT sets
for each statement
 IN(S):
the set of memory locations read by statement S
 OUT(S): the set of memory locations written by statement S
 A memory location may be included in both IN(S) and
OUT(S)
 If S1 is executed before S2 in sequential execution, then
 OUT(S1) ∩ IN(S2) ≠ {} ==> S1 δf S2
 IN(S1) ∩ OUT(S2) ≠ {} ==> S1 δa S2
 OUT(S1) ∩ OUT(S2) ≠ {} ==> S1 δo S2

Data Dependence in Loops (1)
 Associate an instance to each statement and determine
dependences between the instances
 For example, we say S1(10) to mean the instance of S1 when i
= 10
do i = 1, 50
S1: A(i) = B(i-1) + C(i)
S2: B(i) = A(i+2) + C(i)
end do

Data Dependence in Loops (2)
do i = 1, 50
S1: A(i) = B(i-1) + C(i)
S2: B(i) = A(i+2) + C(i)
end do
S1(1): A(1) = B(0) + C(1)

S2(1): B(1) = A(3) + C(1)
S1(2): A(2) = B(1) + C(2)
S2(2): B(2) = A(4) + C(2)
S1(3): A(3) = B(2) + C(3)
S2(3): B(3) = A(5) + C(3)
... ...
S1(50): A(50) = B(49) + C(50)
S2(50): B(50) = A(52) + C(50)
Data Flow Dependence
 Data written by some statement instance is later read by
some statement instance, in the serial execution of the
program. This is written as S1 δf S2.
do i = 3, 50
S1: A(i+1) = ...
S2: ... = A(i-2) ...
end do
S1
S2

Data Anti-Dependence
 Data read by some statement instance is later written by
some statement instance, in the serial execution of the
program. This is written as S1 δa S2.
do i = 1, 50
S1: A(i-1) = ...
S2: ... = A(i+1) ...
end do
S1
S2

Iteration Space Graph (1)
 Nested loops define an iteration space:
do i = 1, 4
do j = 1, 4
A(i,j) = B(i,j) + C(j)
end do
end do
 Sequential execution (traversal order):
j

Iteration Space Graph (2)
 Dimensionality of iteration space = loop nest level
Not restricted to be rectangular

 Triangular iteration spaces are common in scientific codes
do i = 1, 5
do j = i, 5
A(i,j) = B(i,j) + C(j)
end do
end do
 Sequential execution (traversal order):
j

Sequential Execution
 Ordering of execution
 Given two iterations (i1, j1) and (i2, j2) (with positive loop
steps): we say (i1, j1) PCD (i2, j2) if and only if either (i1 < i2)
or [(i1 = i2) AND (ji < j2)].
 This rule can be extended to multi-dimensional iteration
spaces.
 A vector (d1, d2) is positive, if (0, 0) PCD (d1, d2) i.e.,
its first (leading) non-zero component is positive.
 PCD = precedes = ordering symbol for vectors

Dependences in Loop Nests
do i1 = L1, U1
do i2 = L2, U2
...
do in = Ln, Un
BODY(i1, i2,..., in)
end do end do ... end do
 There is a dependence in a loop nest if there are
iterations I = (i1, i2,...,in) and J = (j1, j2,...,jn) and some
memory location M such that
1. I PCD J
2. BODY(I) and BODY(J) reference M
3. There is no intervening iteration K that accesses M, I PCD K
PCD J
Distance and Direction Vectors
 Assume a dependence from BODY(I = (i1, i2,...,in)) and
BODY(J = (j1, j2,...,jn)).
 The distance vector d = (j1 – i1, j2 – i2,…, jn – in)
Define the sign function sgn(x1) of scalar x1:
- if x1 < 0
sgn(x1) = 0 if x1 = 0
+ if x1 > 0
 The direction vector = (sgn(d1), sgn(d2),…,sgn(dn)
where dk = jk- ik for k = 1,…,n.

Example of Dependence Vectors
do i = 1, N
do j = 1, N
A(i, j) = A(i, j-3) + A(i-2, j)
+ A(i-1, j+2) + A(i+1, j-1)
end do
end do
RHS Type Distance Direction

reference vector vector
A(i, j-3) Flow (0, 3) (0, +)
A(i-2, j) Flow (2, 0) (+, 0)
A(i-1, j+2) Flow (1, -2) (+, -)
A(i+1, j-1) Anti (1, -1) (+, -)
Validity of Loop Permutation (1)
Before interchange After interchange
do i = 1, N do j = 1, N
do j = 1, N do i = 1, N
... ...
end do end do
end do end do
j (+, -) prevents interchange j
(-+)
i i

Validity of Loop Permutation (2)
 Loop permutation is valid if
 alldependences are satisfied after interchange
 Geometric view: source (of depended statements) still
executed before sink
 Algebraic view: permuted dependence direction vector is
lexicographically non-negative

Iteration Space Blocking (Tiling)
 A tile in an n-dimensional iteration space is an n-
dimensional subset of the iteration space
A tile is defined by a set of boundaries regularly spaced apart
 Each tile boundary is an (n – 1)-dimensional plane

Validity of Loop Blocking
 Loop blocking is valid if
 All dependences still satisfied in tiled execution order of
iteration space graph (i.e. source before sink)
 Geometric view: No two dependences cross any tile
boundary in opposite directions
 Algebraic view: Dot product of all dependence distance
vectors with normal to tile boundary has same sign

Summary
 Data dependency analysis is essential to avoid
changing the meaning of the program when
performance optimization transformations are done
 Data dependency analysis is essential in design of
parallel algorithms from sequential code
 Data dependences must be maintained in all loop
transformations; otherwise the transformation is illegal
 Data dependency exist in a loop nest when dependence
vector is lexicographically non-negative

Data Dependences: CS 524 - High-Performance Computing

Uploaded by

Copyright:

Available Formats

Data Dependences: CS 524 - High-Performance Computing

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Dependences: CS 524 - High-Performance Computing

Uploaded by

Copyright:

Available Formats

Data Dependences

CS 524 (Au 2004/05)- Asim Karim @ LUMS 2

CS 524 (Au 2004/05)- Asim Karim @ LUMS 4

CS 524 (Au 2004/05)- Asim Karim @ LUMS 5

CS 524 (Au 2004/05)- Asim Karim @ LUMS 6

S1(1): A(1) = B(0) + C(1)

CS 524 (Au 2004/05)- Asim Karim @ LUMS 8

CS 524 (Au 2004/05)- Asim Karim @ LUMS 9

CS 524 (Au 2004/05)- Asim Karim @ LUMS 10

CS 524 (Au 2004/05)- Asim Karim @ LUMS 11

 PCD = precedes = ordering symbol for vectors

CS 524 (Au 2004/05)- Asim Karim @ LUMS 12

CS 524 (Au 2004/05)- Asim Karim @ LUMS 14

RHS Type Distance Direction

j (+, -) prevents interchange j

CS 524 (Au 2004/05)- Asim Karim @ LUMS 16

CS 524 (Au 2004/05)- Asim Karim @ LUMS 17

CS 524 (Au 2004/05)- Asim Karim @ LUMS 18

CS 524 (Au 2004/05)- Asim Karim @ LUMS 19

CS 524 (Au 2004/05)- Asim Karim @ LUMS 20

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.