BDDC For Mixed-Hybrid Formulation of Flow in Porous Media With Combined Mesh Dimensions
BDDC For Mixed-Hybrid Formulation of Flow in Porous Media With Combined Mesh Dimensions
1 Institute of Mathematics, Academy of Sciences of the Czech Republic, Žitná 25, 115 67 Prague 1, Czech Republic
2 Institute for Nanomaterials, Advanced Technology and Innovation, Technical University of Liberec,
Bendlova 1407/7, 461 17 Liberec 1, Czech Republic
3 Department of Mathematics and Statistics, University of Maryland, Baltimore County,
1000 Hilltop Circle, Baltimore, MD 21250, USA
SUMMARY
We extend the Balancing Domain Decomposition by Constraints (BDDC) method to flows in porous media
discretised by mixed-hybrid finite elements with combined mesh dimensions. Such discretisations appear
when major geological fractures are modelled by 1D or 2D elements inside three-dimensional domains. In
this set-up, the global problem as well as the substructure problems have a symmetric saddle-point structure,
containing a ‘penalty’ block due to the combination of meshes. We show that the problem can be reduced
by means of iterative substructuring to an interface problem, which is symmetric and positive definite. The
interface problem can thus be solved by conjugate gradients with the BDDC method as a preconditioner.
A parallel implementation of this algorithm is incorporated into an existing software package for subsurface
flow simulations. We study the performance of the iterative solver on several academic and real-world
problems. Numerical experiments illustrate its efficiency and scalability. Preprint version.
Submitted 2013
KEY WORDS: Iterative substructuring, BDDC, saddle-point problems, mixed-hybrid methods, fractured
porous media, subsurface flow
1. INTRODUCTION
A detailed description of flow in porous media is essential for building mathematical models
with applications in, for example, water management, oil and gas recovery, carbon dioxide (CO2 )
sequestration or nuclear waste disposal. In order to set up a reliable numerical model, one needs to
have a good knowledge of the problem geometry and input parameters. For example, the flow of
water in granite rock, which is a suitable site for nuclear waste disposal, is driven by the complex
system of vugs, cavities and fractures with various topology and sizes. These alter the effective
permeability, and therefore should be accurately accounted for in the numerical model. There are
two main approaches: either the fractures are considered as free-flow regions, or the fractures
contain debris and are also modelled as porous media with specific permeabilities. In the first case,
a unified approach to modelling free-flow and porous media regions can be provided by the so called
∗ Correspondence to: J. Šı́stek, Institute of Mathematics, Academy of Sciences of the Czech Republic, Žitná 25,
115 67 Prague 1, Czech Republic. E-mail: sistek@math.cas.cz
Contract/grant sponsor: The research was supported by Czech Science Foundation under project GA ČR 14-02067S, by
Academy of Sciences of the Czech Republic through RVO:67985840, and by Ministry of Education, Youth and Sports
under project CZ.1.05/2.1.00/01.0005. J. Šı́stek acknowledges the computing time on HECToR provided through the
PRACE-2IP project (FP7 RI-283493).
Preprint version.
2 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
Stokes-Brinkman equation, which reduces to either the Stokes or Darcy model in certain parameter
limits, e.g., within the Multiscale Mixed Finite-Element (MsMFE) framework [1]. In this paper, we
consider the latter case, and apply Darcy’s law to the flow in the reservoir and in the fractures as well;
see [2] for a related approach. In either case, the preferential flow in large geological dislocations
and their intersections should be considered as two- and one-dimensional flows, respectively. Due
to the quite complex structure of the domains, the discretisation is performed using finite element
methods (FEM). The resulting meshes are therefore unstructured, and they combine different spatial
dimensions (line elements in 1D, triangles in 2D, and tetrahedrons in 3D). The systems of linear
equations obtained from the FEM discretisation are often very large, so that using direct methods
is prohibitive and iterative solvers are warranted. The systems are typically also bad-conditioned
due to the mixing of spatial dimensions, large jumps in permeability coefficients and presence of
elements of considerably different sizes, and so they are challenging for iterative solvers as well.
The matrices have a saddle-point structure
T
A B , (1)
B −C
where A is symmetric positive definite on the kernel of B , C is symmetric positive semi-definite,
T
and it is positive definite on the kernel of B . The ‘penalty’ block C 6= 0 arises from connecting
meshes of different spatial dimensions. The iterative solution of systems with this structure is a
frequently studied topic; see, for example, [3, 4, 5, 6], the monographs [7, 8] or [9, Chapter 9]
and the references therein. However, efficient methodologies for solving saddle-point problems are
typically problem dependent.
In this paper, we develop a robust and scalable solver for linear systems with the saddle-point
structure as in (1) with the block C either zero or nonzero. The solver is tailored to the mixed-
hybrid formulation of flow in porous media using the lowest order Raviart-Thomas (RT0 ) finite
elements with combined mesh dimensions (1D, 2D and 3D). In particular, we adapt the Balancing
Domain Decomposition by Constraints (BDDC) method to this type of problems.
The BDDC method is currently one of the most popular methods of iterative substructuring. It has
been proposed independently in [10, 11, 12]; see [13, 14] for the proof of equivalence. Even though
BDDC has been originally formulated for elliptic problems, it has been successfully extended,
for example, beyond elliptic cases [15, 16] and to multiple levels [17, 18]. An optimal set-up has
been studied in [19, 20, 21, 22]. A closely related BDDC preconditioner for vector field problems
discretised with Raviart-Thomas finite elements has been studied in [23].
We are interested in applications of the BDDC method to saddle-point problems. If C = 0 in (1),
one possible approach is to use an algebraic trick and constrain the iterative solution of the indefinite
problem into a balanced subspace, which is sometimes also called benign, where the operator
is positive definite; see [15] for the Stokes problem, and [5, 24, 25] for flow in porous media.
However, due to the mixed-hybrid formulation and possible coupling of meshes with different
spatial dimensions, C 6= 0 in general, and we will favour an alternative, dual approach here.
Our methodology is as follows. The mixed-hybrid formulation [26, 27] is used in order to modify
the saddle-point problem to one which is symmetric and positive definite by means of iterative
substructuring. In particular, we introduce a symmetric positive definite Schur complement with
respect to interface Lagrange multipliers, corresponding to a part of block C . The reduced system is
solved by the preconditioned conjugate gradient (PCG) method, and the BDDC method is used as
a preconditioner. From this perspective, our work can be viewed as a further extension of [6]. Our
main effort here is in accommodating the BDDC solver to flows in porous media with combined
mesh dimensions. In addition, the presentation of the BDDC algorithm is driven more by an efficient
implementation, while it is more oriented towards underlying theory in [6]. We take advantage
of the special structure of the blocks in matrix (1) studied in detail in [26, 28, 29]. In particular,
the nonzero structure of block C resulting from a combination of meshes with different spatial
dimensions is considered in [30]. We describe our parallel implementation of the method and
study its performance on several benchmark and real-world problems. Another original contribution
of this paper is proposing a new scaling operator in the BDDC method suitable for the studied
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 3
problems. We note that if there is no coupling of meshes with different spatial dimensions present
in the discretisation, the block C = 0 in (1) and our method is almost identical to the one introduced
in [6].
The paper is organised as follows. In Section 2, we introduce the model problem. In Section 3, we
describe the modelling of fractured porous media and combining meshes of different dimensions.
In Section 4, we introduce the substructuring components and derive the interface problem. In
Section 5, we formulate the BDDC preconditioner. In addition, the selection of interface weights for
BDDC is studied in detail in Section 6. In Section 7, we describe our parallel implementation, and in
Section 8 we report the numerical results and parallel performance for benchmark and engineering
problems. Finally, Section 9 provides a summary of our work.
Our notation does not, for simplicity, distinguish between finite element functions and
corresponding algebraic vectors of degrees of freedom, and between linear operators and matrices
within a specific basis—the meaning should be clear from the context. The transpose of√a matrix
is denoted by superscript T and the energy norm of a vector x is denoted by kxkM = xT M x,
where M is a symmetric positive definite matrix.
2. MODEL PROBLEM
Let Ω be an open bounded polyhedral domain in R3 . We are interested in the solution of the
following problem, combining Darcy’s law and the equation of continuity written as
k−1 u + ∇p = −∇z in Ω, (2)
∇ · u = f in Ω, (3)
p = pN on ∂ΩN , (4)
u · n = 0 on ∂ΩE , (5)
subject to boundary conditions on ∂Ω = ∂ΩN ∪ ∂ΩE , where ∂ΩN stands for the part of the
boundary with natural (Dirichlet) boundary condition, and ∂ΩE for the part with essential
(Neumann) boundary condition. In applications, the variable u describes the velocity of the fluid
and p the pressure (head) in an aquifer Ω, k is a symmetric positive definite tensor of the hydraulic
conductivity, −∇z = (0, 0, −1)T is the gravity term, and n is the outer unit normal vector of ∂Ω. The
term ∇z is present due to the fact, that u satisfies u = −k∇ph , where ph = p + z is the piezometric
head. For a thorough discussion of application background we refer, e.g., to monographs [31, 32].
Let T be the triangulation of domain Ω consisting of NE simplicial elements with characteristic
size h. We introduce a space
H(Ω; div) = v : v ∈ L2 (Ω); ∇ · v ∈ L2 (Ω) and v · n = 0 on ∂ΩE ,
(6)
equipped with the standard norm. Let V ⊂ H(Ω, div) be the space consisting of the lowest order
Raviart-Thomas (RT0 ) functions and let Q ⊂ L2 (Ω) be the space consisting of piecewise constant
functions on the elements of the triangulation T . We refer, e.g., to monograph [33] for a detailed
description of the mixed finite elements and the corresponding spaces.
In the mixed finite element approximation of problem (2)–(5) we look for a pair {u, p} ∈ V × Q
that satisfies
Z Z Z Z
k−1 u · v dx − p∇ · v dx = − pN v · n ds − vz dx, ∀v ∈ V, (7)
Ω Ω ∂ΩN Ω
Z Z
− q∇ · u dx = − f q dx, ∀q ∈ Q. (8)
Ω Ω
In the discrete formulation, we need pN and f only sufficiently regular so that the integrals in the
weak formulation (7)–(8) make sense, namely pN ∈ L2 (∂ΩN ), f ∈ L2 (Ω).
Next, we recall the mixed-hybrid formulation. It was originally motivated by an effort to modify
the saddle-point problem (7)–(8) to one which leads to symmetric positive definite matrices.
Preprint version.
4 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
Nevertheless, this formulation is also suitable for a combination of meshes with different spatial
dimensions, which will be described in detail in the next section.
Let F denote the set of inter-element faces of the triangulation T . We now introduce several
additional spaces. First, let us define the space V−1 by relaxing the condition of continuity of the
normal components in the space V on inter-element boundaries F . More precisely, we define local
spaces Vi for each element T i ∈ T , i = 1, . . . , NE , by
and put V−1 = V1 × · · · × VNE . Next, we define the space of Lagrange multipliers Λ consisting
of functions that take constant values on individual inter-element faces in F ,
Λ = λ ∈ L2 (F) : λ = v · n|F , v ∈ V .
(10)
Equation (13) imposes a continuity condition on the normal component of the velocity (also
called normal flux) u · n across F which guarantees that u ∈ V. This condition also implies the
equivalence of the two formulations (7)–(8) and (11)–(13). We note that the Lagrange multipliers λ
can be interpreted as the approximation of the trace of p on F , see [34] for details.
Let us now write the matrix formulation corresponding to (11)–(13) as
A B T BF T
u g
B 0 0 p = f . (14)
BF 0 0 λ 0
3. MODELLING OF FRACTURES
In this section, we recall the main ideas of the discrete model of the flow in fractured porous
media that is based on connection of meshes of different dimensions as described in [30]. Let us
denote the full domain by Ω3 = Ω. Next, consider lower-dimensional domains Ωd−1 ⊂ Ωd , d = 2, 3,
such that Ω2 consists of polygons and Ω1 consists of line segments. We will also assume that
∂Ω1 ⊂ ∂Ω2 ⊂ ∂Ω3 . The first condition requires that a domain of a lower dimension cannot poke
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 5
out of the domain of higher dimension, while the second condition prevents domains of lower
dimension from having boundaries in the interior of domains of higher dimension. We impose these
conditions to avoid technical difficulties in the analysis. However, numerical evidence suggests that
these conditions are not necessary, and in fact, they are not satisfied for the real-world problems
presented in Section 8.2.
For every dimension d = 1, 2, 3, we introduce a triangulation Td of the domain Ωd that consists
of finite elements Tdi , i = 1, . . . , NEd and satisfies the compatibility conditions
[
i
Td−1 ⊂ Fd , where Fd = ∂Tdk \ ∂Ωd , (15)
k
i
Td−1 ∩ ∂Tdj is either Td−1
i
or ∅, (16)
for every i ∈ {1, . . . , NEd−1 }, j ∈ {1, . . . , NEd }, and d = 2, 3. This means that elements of a lower
dimension match faces of elements of the higher dimension.
We consider equations (3)–(5) on the domains Ωd , d = 1, 2, 3, completed by a slight modification
of the Darcy’s law (2):
ud
kd−1 + ∇pd = −∇z, (17)
δd
where ud stands for the velocity integrated over the cross-section for d = 1, 2, i.e. the units of u3 ,
u2 , and u1 are ms−1 , m2 s−1 , and m3 s−1 , respectively. In addition, δ3 = 1, δ2 is the thickness of a
fracture, and δ1 is the cross-section of a 1D preferential channel. The effective fluid source f2 on Ω2
is given as
f2 = δ2 f˜2 + u+ + −
3 · n + u3 · n ,
−
(18)
where f˜2 is the density of external fluid sources, and the normal fluxes from the two faces of the
3D continuum surrounding the fracture are given through the Robin (also called Newton) boundary
conditions
u+ + + +
3 · n = σ3 (p3 − p2 ), (19)
u−
3 ·n = −
σ3− (p−
3 − p2 ). (20)
+/− −
In the last formula, σ3 > 0 are the transition coefficients (cf. [2] for possible choices) and p+
3 , p3
are the traces of pressure p3 on the two sides of the fracture. The effective fluid source f1 on Ω1 is
similar, X
f1 = δ1 f˜1 + uk2 · nk , (21)
k
where f˜1 is the density of external fluid sources. In the 3D ambient space, the 1D channel can be
connected to k faces of 2D fractures. Thus
is the normal flux from the connected fracture k , σ2k > 0 is the transition coefficient, and pk2 is the
trace of pressure p2 on the face of fracture k .
In the following, we describe the discrete mixed-hybrid formulation of the problem. The
formulation and discussion of the continuous problem can be found in [30]. Let us consider spaces
d
NE
Y
V −1
= V1−1 × V2−1 × V3−1 , Vd−1 = Vi (Tdi ), Q = Q1 × Q2 × Q3 , Qd = L2 (Ωd ) . (23)
i=1
For the definition of the space Λ, we cannot follow (10) directly, since e.g. on Ω2 , we need to
distinguish values of λ3 on two sides of a fracture. Thus, we introduce a separate value for every
non-boundary side of every element:
n o
Λ(Tdi ) = λ ∈ L2 (∂Tdi \ ∂Ωd ) : λ = v · n|∂Tdi , v ∈ Vd , (24)
Preprint version.
6 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
where Vd is defined in the same way as the space V but on the domain Ωd . Further, we identify
values on faces/points that are not aligned to the fractures/channels:
d
n NE o
Y
Λd = λ ∈ Λ(Tdi ); λ|∂Tdi = λ|∂T j on face F = ∂Tdi ∩ ∂Tdj if F ∩ Ωd−1 = ∅ . (25)
d
i=1
with
d "
NE
3 X Z #
X 1 −1
a (u, v) = kd ud · vd dx , (29)
d=1 i=1 Tdi δd
d
NE
3 X
"Z #
X
b (u, q) = − qd (∇ · ud ) dx , (30)
d=1 i=1 Tdi
d "
NE
3 X Z #
X
bF (u, λ) = λ|∂Tdi (ud · n) ds , (31)
d=1 i=1 ∂Tdi \∂Ωd
d "
NE
3 X Z #
X
c (p, q) = σd pd−1 qd−1 ds , (32)
d=2 i=1 ∂Tdi ∩Ωd−1
d "
NE
3 X Z #
X
cF (p, µ) = − σd pd−1 µd ds , (33)
d=2 i=1 ∂Tdi ∩Ωd−1
d "
NE
3 X Z #
X
c (λ, µ) =
e σd λd µd ds , (34)
d=2 i=1 ∂Tdi ∩Ωd−1
d d
NE Z
3 X NE Z
3 X
X X
hg, vi = − pN (v · n) ds− vz dx, (35)
d=1 i=1 ∂Tdi ∩∂ΩN d=1 i=1 Tdi
3 Z
X
hf, qi = − δd f˜d qd dx. (36)
d=1 Ω
BT T
A BF u g
T
B −C −CF p = f . (37)
BF −CF −C e λ 0
We note that (37) is related to (26)–(28) in the same way as (14) is related to (11)–(13). The
main difference in the structure of the matrices between (37) and (14) is the additional block
T
C CF
C= related to the normal fluxes between dimensions and arising from (19)–(20)
CF C e
and (22) via (32)–(34). In particular, the modified right-hand side of the continuity equation for
two-dimensional and one-dimensional elements, f2 and f1 , incorporates pressure unknowns on 2D
and 1D elements, and traces of pressure on 3D and 2D elements at the fracture, which are nothing
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 7
but the Lagrange multipliers on 3D and 2D elements in the mixed-hybrid method. Consequently,
+/− +/−
p3 = λ3 in (19)–(20), and pk2 = λk2 in (22).
Assuming δd bounded and greater than zero, and using the fact that kd corresponds to a symmetric
positive definite matrix, we see from (29) that block A in (37) is symmetric positive definite. Block
C is symmetric positive semi-definite since
d "
NE
3 X Z #
X
c(p, p) + 2cF (p, λ) + e
c(λ, λ) = σd (pd−1 − λd )2 ds . (38)
d=2 i=1 ∂Tdi ∩Ωd−1
The following theorem is a standard result, e.g. [33, Theorem 1.2]. Here, we rewrite it in
a form suitable for our setting and we verify the assumptions for the specific blocks of the
matrix in (37). We will further denote Q = Q × Λ, p = (p, λ) ∈ Q, q = (q, µ) ∈ Q, and b(u, q) =
b(u, q) + bF (u, µ).
Theorem 1
Let natural boundary conditions (4) be prescribed at a certain part of the boundary, i.e. ∂ΩN,d 6= ∅
for at least one d ∈ {1, 2, 3}. Then the discrete mixed-hybrid problem (37) has a unique solution.
Proof
Let us first investigate the structure of the matrix in (37) more closely. Let us number the unknowns
within each block of (37) with respect to spatial dimension d ∈ {1, 2, 3}. The matrix then takes the
form of 9×9 blocks,
T T
A11 B11 BF ,11
T T
A22 B22 BF ,22
T T
A 33 B 33 B F ,33
T
B11 −C11 −CF ,12
T
B22 −C22 −CF ,23 . (39)
B 33
BF ,11
BF ,22 −CF ,12 −Ce22
BF ,33 −CF ,23 −C33
e
Suppose for a moment that we solve a problem only on domain Ωd , d ∈ 1, 2, 3 (i.e. Ωi = ∅ for
i 6= d). If no natural boundary conditions are imposed, there is a single −1 entry on each row
T T
of Bdd and a single +1 entry on each row of BF ,dd . Since Ωd is a simply connected set, the
T T T
matrix B dd = [ Bdd BF ,dd ] has a nontrivial nullspace of constant vectors. Enforcing natural boundary
T
condition on a part of Ωd changes the +1 value on the corresponding row of matrix BF ,dd to 0, in
T
which case B dd has only a trivial nullspace, i.e. full column rank (see e.g. [33, Section IV.1] or [26,
Lemma 3.2]).
The nullspace becomes more complicated for domains with fractures, in which case Ωd typically
has more simply connected components separated by fractures (cf. Fig. 1). Let us denote them Ωck ,
k = 1, . . . , nc , regardless of the dimension. In particular, Ωck , k = 1, . . . , nci will be components
without natural condition boundary, i.e. ∂Ωck ∩ ∂ΩN = ∅, while for k = nci + 1, . . . , nc we get
components with prescribed natural boundary condition. We also denote χk ∈ Q the characteristic
vector of the component Ωck that takes value +1 for degrees of freedom associated with elements
and faces of the component Ωck . With such notation, the basis of the nullspace of the whole matrix
T T
B11 BF ,11
T T T
B = B22 BF ,22
(40)
T T
B33 BF ,33
Preprint version.
8 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
Next, we show that matrix C is not only symmetric positive semi-definite, as seen from (38), but
T T
also positive definite on null B . To this end, take p ∈ null B , a vector that is piecewise constant on
c
components, having value pk,d on the component Ωk of dimension d for k = 1, . . . , nci , and value
pk,d = 0 for other components. Then pT Cp = 0 implies p = pk,d = 0. Indeed, every component Ωck
of dimension d = 2, 3 has some component Ωcj of dimension d − 1 on its boundary, and therefore
all pk,d have the same value, cf. (38). This value is zero, since there is at least one component with
natural boundary condition.
Applying the congruence transformation,
T "A #
T
A B =
I T
I A−1 B . (41)
B −C BA−1 I − BA−1 B + C I
T
Matrix A is symmetric positive definite (SPD) from (29) and therefore BA−1 B is SPD on range
T T
B , which is the orthogonal complement of the nullspace of B . Thus BA−1 B + C is SPD on
whole Q. From the Sylvester law of inertia, the number of positive, negative and zero eigenvalues is
preserved by the congruence transformation. Since the block diagonal matrix on the right-hand side
of (41) has only (strictly) positive and (strictly) negative eigenvalues, the matrix on the left-hand
side also must be nonsingular, and problem (37) has a unique solution.
Figure 1. Example of a two-dimensional problem: even single fracture gives rise to two components of the
two-dimensional mesh (left); example of three (shaded) triangles sharing a single Lagrange multiplier in 3D
(right).
4. ITERATIVE SUBSTRUCTURING
For our purposes of combining meshes with different spatial dimensions, we define substructures
as subsets of finite elements in the mesh rather than parts of a physical domain, cf. [9].
To begin, let us define the combined triangulation T123 as the union of triangulations for each
spatial dimension, i.e., T123 = T1 ∪ T2 ∪ T3 . The triangulation T123 is subsequently divided into
substructures Ωi , i = 1, . . . , NS . Note that in general a substructure can contain finite elements of
different dimensions. We define the interface Γ as the set of degrees of freedom shared by more than
one substructure. Note that Γ ⊂ Λ for the current setting. Thus, let us split the Lagrange multipliers λ
into two subsets. First, we denote by λΓ the set of multipliers corresponding to the interface Γ.
The remaining multipliers, corresponding to substructure interiors, will be denoted by λI . The
interface Γi of substructure Ωi is defined as a subset of Γ corresponding to ∂Ωi . Next, let ΛiΓ be
defined as the space of Lagrange multipliers corresponding to Γi , i = 1, . . . , NS , and define a space
ΛΓ = Λ1Γ × · · · × ΛN
Γ .
S
(42)
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 9
The substructure problems are obtained by assembling contributions of finite elements in each Ωi ,
Ai B iT BFiT
BFiT
i i
,I ,Γ u g
Bi −C i −CFiT
,I
iT
−CF ,Γ pi f i
i i ei eiT λi = 0 ,
i = 1, . . . , NS , (43)
BF ,I −CF ,I −C II −C ΓI I
i
BF ,Γ
i
−CF ,Γ −Cei
ΓI −Cei
ΓΓ
λiΓ 0
where the blocks Ai are block diagonal with blocks corresponding to finite element matrices, and the
i
blocks C 6= 0 only if the substructure Ωi contains coupling of elements of different dimensions. We
note that the global problem (37) could be obtained from (43) by further assembly at the interface.
In the iterative substructuring (see e.g. [9]), we first reduce the problem to substructure interfaces.
In our context we can eliminate normal fluxes, pressure unknowns, and Lagrange multipliers at
interiors of substructures, and we can define the substructure Schur complements S i : ΛiΓ 7→ ΛiΓ ,
i = 1, . . . , NS , formally as
−1
Ai B iT iT iT
h i BF ,I BF ,Γ
Si = C i
eΓΓ + i
BF i
−CF ei
−C Bi −C i −CFiT
,I
−CFiT,Γ . (44)
,Γ ,Γ ΓI
i i i eiT
BF ,I −CF ,I −Ce
II −C ΓI
However, in an implementation of a Krylov subspace iterative method, we only need to compute the
matrix-vector product S i λiΓ for a given vector λiΓ . Therefore, the matrix is not constructed explicitly,
and the multiplication is obtained as follows.
Algorithm 2
Given λiΓ ∈ ΛiΓ , determine S i λiΓ ∈ ΛiΓ in the following two steps:
Ai B iT iT iT
i
BF ,I w BF ,Γ
i
B −C i −CFiT
,I q i = − −CFiT
,Γ
λiΓ , (45)
i i i i iT
BF ,I −CF ,I −Ce
II
µI −Ce
ΓI
Next, let Λ
b Γ be the space of global degrees of freedom, such that the values of degrees of freedom
from ΛΓ coincide on Γ. The vectors of the local substructure degrees of freedom λiΓ ∈ ΛiΓ and the
vector of the global degrees of freedom λΓ ∈ Λ b Γ are related by
λiΓ = Ri λΓ , i = 1, . . . , NS , (47)
where Ri are the restriction operators. More specifically, each Ri is a 0-1 matrix such that every row
contains one entry equal to one, and Ri RiT = I . The global Schur complement Sb : Λ bΓ → Λ b Γ can
be obtained as
NS
X
Sb = RiT S i Ri . (48)
i=1
Equation (48) represents the formal assembly of substructure Schur complements into the global
Schur complement. The global Schur complement system, which we would like to solve iteratively,
reads
Sλ
b Γ = bb, (49)
Preprint version.
10 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
PNS iT i
where vector bb = i=1 R b is obtained from substructure reduced right-hand sides
−1 i
Ai B iT iT
h i BF ,I g
bi = i
BF i
−CF ei
−C Bi −C i −CFiT
,I
fi . (50)
,Γ ,Γ ΓI
i i ei 0
BF ,I −CF ,I −C II
In our implementation, we factor and store the matrices from (45). The factors are then used to
compute the vectors bi in (50), and especially in Algorithm 2, which is used in connection to
formula (48) to evaluate λΓ → Sλ b Γ within each iteration of a Krylov subspace iterative method.
This algorithm allows a straightforward parallel implementation. After an approximate solution
of (49) is found, solution in interiors of substructures, including all primal variables, is recovered
from (43) using the factors from (45).
Remark 3
There are other ways to derive the interface problem (49). The authors of [6, 34] use a mixed-
hybrid formulation with Lagrange multipliers introduced only at the interface Γ as their starting
point. While problem (49) is equivalent to the interface problems considered in [6, 34], the
substructure problems therein have a different structure from (43). In particular, there are no
blocks corresponding to λiI , and the matrices corresponding to Ai are no longer block diagonal
element-wise. Next, the authors of [28, 30] construct explicit Schur complement with respect to the
whole block of Lagrange multipliers λ and they show that, due to the special structure of A, the
complement is both sparse and reasonably cheap to construct. If this was performed substructure
by substructure, this could be again seen as an intermediate step in obtaining problem (49) by
additional elimination of the interior Lagrange multipliers λI . However, this would again lead to
different substructure problems based on explicit local Schur complements.
The following result allows an application of the BDDC method to problems with fractures.
Theorem 4
Let natural boundary conditions (4) be prescribed at a certain part of the boundary, i.e. ∂ΩN,d 6= ∅
for at least one d ∈ {1, 2, 3}. Then the matrix Sb in (49) is symmetric positive definite.
Proof
T
Using the notation of (40)–(41), let us introduce a matrix S = BA−1 B + C . The matrix S
is symmetric positive definite by Theorem 1 (and its proof). Applying another congruence
transformation to S and denoting the rows corresponding to the interface Lagrange multipliers by
subscript Γ and the interior by I , we obtain
−1 T
T
S SΓI I SII I SII SΓI
S = II = −1 −1 T . (51)
SΓI SΓΓ SΓI SII I SΓΓ − SΓI SII SΓI I
−1 T
Since the matrix on the left-hand side is SPD, both diagonal blocks SII and SΓΓ − SΓI SII SΓI are
also symmetric positive definite from the Sylvester law of inertia. It remains to note that the Schur
−1 T
complement Sb in (49) is symmetric positive definite because Sb = SΓΓ − SΓI SII SΓI .
Theorem 4 allows us to use the conjugate gradient method for the iterative solution of (49). In the
next section, we describe the BDDC method used as a preconditioner for Sb.
In this section, we formulate the BDDC method for the solution of (49). The algorithm can be
viewed as a generalisation of [6]. However, we follow the original description from [11], which
better reflects our implementation.
One step of BDDC provides a two-level preconditioner for the conjugate gradient method applied
to solving problem (49). It is characterised by the selection of certain coarse degrees of freedom
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 11
based on primary degrees of freedom at interface Γ. The main coarse degrees of freedom in this
paper are arithmetic averages over faces, defined as subsets of degrees of freedom shared by the
same two substructures. In addition, corner coarse degrees of freedom, defined as any selected
Lagrange multiplier at the interface, are used. Substructure edges, defined as subsets of degrees of
freedom shared by several substructures, may also appear (see Remark 5).
The BDDC method introduces constraints which enforce continuity of functions from ΛΓ at
coarse degrees of freedom among substructures. This gives raise to the space Λ e Γ , which is given as
the subspace of ΛΓ of functions which satisfy these continuity constraints. In particular,
bΓ ⊂ Λ
Λ e Γ ⊂ ΛΓ . (52)
Remark 5
In three spatial dimensions, several triangular elements can be connected at a single Lagrange
multiplier in a star-like configuration (cf. Fig. 1). A similar statement holds for line elements
considered in two- or three-dimensional space. This fact may lead to the presence of substructure
edges and even vertices (defined as degenerate edges consisting of a single degree of freedom), and
we may prescribe also edge averages as constraints. As mentioned above, we also select corners
as coarse degrees of freedom. While essentially any degree of freedom at the interface Γ can be a
corner, we select them by the face-based algorithm [21]. This algorithm considers all vertices as
corners, and, in addition, it selects three geometrically well distributed degrees of freedom from the
interface between two substructures sharing a face into the set of corners. Although considering
corners is not the standard practice with RT0 finite elements, in our experience, corners improve
convergence for numerically difficult problems, as can be observed for the engineering applications
presented in Section 8.
We now proceed to the formulation of operators used in the BDDC method. The choice of
constraints determines the construction of matrices Di . Each row of Di defines one coarse degree of
freedom at substructure Ωi , e.g., a corner corresponds to a single 1 entry at a row and an arithmetic
average to several 1’s at a row. The coarse basis functions ΦiΓ , one per each substructure coarse
degree of freedom, are computed by augmenting the matrices from (43) with Di , and solving the
augmented systems with multiple right-hand sides
Ai B iT BFiT
BFiT
0
Xi 0
,I ,Γ
i i iT iT
B −C −CF ,I −CF ,Γ 0 Zi 0
i i
i
ei eiT
F ,I −CF ,I −CII −C ΦI = 0 , i = 1, . . . , NS ,
B 0 (53)
ΓI
Bi i ei ei iT Φi 0
F ,Γ −C F ,Γ − C ΓI −C ΓΓ D Γ
0 0 0 Di 0 Li I
where I is the identity matrix, and X i , Z i , and ΦiI are auxiliary matrices not used any further. As
i
shown in [35], the local coarse matrix SCC is obtained as a side product of solving (53) as
Ai B iT BF iT
BFiT
i
,I ,Γ X
i
iT iT iT iT B i −C i −CF iT
,I −CF iT
,Γ Zi i
SCC = X Z ΦI ΦΓ i eiT Φi = −L . (54)
i
BF ,I −CF −C ei −C
,I II ΓI I
BFi
,Γ −CF i
,Γ −C ei
ΓI −C ei
ΓΓ
ΦiΓ
i
Let us define, similarly to (47), operators RC that relate vectors of local coarse degrees of
i
freedom µC to the vector of global coarse degrees of freedom µC as
µiC = RC
i
µC . (55)
The global coarse matrix SCC is then obtained by assembling the local contributions as
NS
X
iT i i
SCC = RC SCC RC . (56)
i=1
Preprint version.
12 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
rΓi = W i Ri rΓ , i = 1, . . . , NS . (59)
i
2. Compute the substructure corrections ηΓ∆ by solving the local Neumann problems
Ai B iT BFiT iT
BF 0
xi 0
,I ,Γ
i i iT iT
B −C −CF ,I −CF ,Γ 0 zi 0
i i
i
B −C − Cei − eiT
C 0 ηI∆ = 0 , i = 1, . . . , NS .
F ,I F ,I II ΓI i i
Bi −CF i
−C ei −Cei DiT ηΓ∆ rΓ
F ,Γ ,Γ ΓI ΓΓ
i
0 0 0 Di 0 l 0
(60)
i
3. Compute the coarse correction ηC by collecting the coarse residual
NS
X
iT iT i
rC = RC ΦΓ rΓ , (61)
i=1
SCC ηC = rC , (62)
We note that the factorisations of the matrices from (53) are also used for each solution of (60).
In order to apply the existing BDDC theory for elliptic problems (e.g. [13, 36]) to the proposed
preconditioner, we introduce some additional notation and make a few observations. Due to (44),
the substructure corrections in (60) can be written equivalently as
i i
−S i DiT ηΓ∆ rΓ
= , (65)
Di 0 li 0
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 13
Next, let us formally write the operators and vectors in the block form
1 1
λΓ R1 W1 S
.. .. . .. ..
λΓ = . , R = . , W = , S = . (67)
.
NS NS NS
λΓ R W S NS
By grouping the steps of Algorithm 6 and using (65), the operator of the BDDC preconditioner
can be formally written as
MBDDC = RT W Se−1 W R, (68)
where
!T !
i iT
−1 ! NS NS
−S D IΛiΓ
X X
−1
Se−1 = − iT iT iT iT
diag IΛiΓ 0 + RC ΦΓ SCC RC ΦΓ .
i=1,...,NS Di 0 0
i=1 i=1
(69)
The first term in Se−1 corresponds to substructure corrections and the second term to the coarse
correction (steps 2 and 3 of Algorithm 6), and IΛiΓ is the identity matrix in ΛiΓ . From (68) and (69),
one can readily see that MBDDC is symmetric.
Assumption 7
Let us assume that
null S i ⊥ null Di . (70)
In order to satisfy Assumption 7, we must prescribe enough coarse degrees of freedom as
constraints along with Robin boundary conditions (19)–(20) and (22) at each fracture within
substructure Ωi . Since constrains in Di are linearly independent, DiT has full column rank. In
particular, Assumption 7 is satisfied when arithmetic averages are used on each substructure face
(and eventually edge) as constraints.
Lemma 8
The operator Se−1 in preconditioner MBDDC is symmetric and positive definite on the space Λ
eΓ.
Proof
The space Λ
e Γ is decomposed into the substructure spaces and the coarse space,
Λ e Γ∆ ⊕ Λ
eΓ = Λ e ΓC . (71)
To achieve this splitting, each local space ΛiΓ is decomposed into subspaces
corresponding to the substructure space Λe i , and the coarse space on substructure Ωi , Λ e ΓC |Ωi ,
Γ∆
i
respectively. To analyse this decomposition, let us recall that S is a positive semi-definite matrix
and write (66) in detail as
From (73),
range (S i ΦiΓ ) ⊂ range DiT ⊥ null Di , (75)
which in turn, similarly to [36, Lemma 8], gives for any χi∆ ∈ null Di and ξC
i
∈ range ΦiΓ
χiT i i
∆ S ξC = 0. (76)
Preprint version.
14 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
From (74), the matrix ΦiΓ has full column rank, and
null Di ∩ range ΦiΓ = 0. (77)
Finally, from Assumption 7 and (72),
null S i ⊂ range ΦiΓ . (78)
Decomposition of the subdomain space (72) implies decomposition of a function ζ i ∈ ΛiΓ to
ζ i = ζ∆i
+ ζCi , where ζ∆i
∈ null Di , ζCi ∈ range ΦiΓ , and ζ∆
iT i i
S ζC = 0 by (76).
Let us first analyse the substructure corrections. Following [3, Section 3.3], the matrix of (65) is
invertible due to Assumption 7. If we define, in addition, a matrix Qi with orthonormal columns
forming a basis of null Di , i.e.
range Qi = null Di , QiT Qi = I, (79)
we have −1
−S i DiT IΛiΓ −1
= −Qi QiT S i Qi QiT .
IΛiΓ 0 (80)
Di 0 0
−1
The matrix QiT S i Qi is symmetric positive definite, and consequently, for any ζ i ∈ ΛiΓ ,
−1 iT i
− ζ iT Qi QiT S i Qi Q ζ ≤0 (81)
with equality if ζ i = ζCi ∈ range ΦiΓ .
i
Next, let us turn towards the coarse correction. Formula (54) for SCC can be written equivalently
as
i
SCC = −ΦiT i i
Γ S ΦΓ . (82)
Since the term on the right-hand side is just a (negative) Galerkin projection of the positive semi-
definite matrix S i , matrix SCC
i
is symmetric negative semi-definite. If at least one substructure is
equipped with natural boundary conditions, the matrix SCC assembled by (56) becomes symmetric
−1
negative definite, and so is SCC .
We have just verified the negative definiteness of the principal parts of Se−1 , and the desired
positive definiteness is obtained through the change of sign in front of the braces in (69).
In view of Lemma 8, the standard condition number bound follows from [13, Lemma 2].
Theorem 9
The condition number κ of the preconditioned operator MBDDC Sb satisfies
RR W λΓ
2
T
S
κ ≤ ω = max 2 . (83)
λΓ ∈Λ
eΓ kλΓ kS
The norms in (83) are induced by the matrix S defined in (67) for all functions λΓ ∈ ΛeΓ.
In addition, in the case of a single mesh dimension in either 2D or 3D, and under the assumption
of substructure-wise constant hydraulic conductivities, it has been also derived in [6, Lemma 5.5
and Theorem 6.1] that the condition number bound ω satisfies
2
H
ω ≤ C 1 + log , (84)
h
where H is the characteristic size of geometric substructures. We note that the bound (84) implies
that, for a fixed relative subdomain size H/h, the condition number is independent of the problem
size.
It is worth emphasising that Theorem 9 is also valid for combined mesh dimensions. However,
several simplifications are employed in [6] to obtain (84), which are not satisfied in the set-up
considered in this paper. In particular (i) hydraulic conductivity coefficient here is, in general, not
substructure-wise constant nor isotropic, (ii) it is not clear whether, in presence of fractures, the
interpolation operator onto a conforming mesh introduced in [6] can be constructed and bounded in
the H 1 norm.
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 15
Let us now discuss the choice of entries in the diagonal weight matrices W i . These matrices play
an important role in the BDDC method, both in the theory, cf. Theorem 9 or [6, 13, 37], and in
the computations, cf. [19, 38]. Three possible choices are also studied numerically in Section 8.2.
The basic choice is presented by the arithmetic average taken from values at the neighbouring
substructures. In this simplest construction, the entry corresponding to Lagrange multiplier λiΓ,j is
given by the inverse counting function as
i 1
Wjj = , (85)
card(Ij )
where card(Ij ) is the number of substructures in the set Ij of indices of substructures to which λiΓ,j
i
belongs. For 2D or 3D meshes without fractures, Wjj = 1/2 for Raviart-Thomas elements. However,
since several two-dimensional fractures can meet in our setting, smaller weights can occasionally
appear at such regions.
While arithmetic average is sufficient for problems with homogeneous coefficients, it is well
known that for problems with large variations in material properties along the interface, it is
necessary to incorporate their values into the (weighted) average to obtain a robust method. This
gives rise to the ρ-scaling, for which
i ρi
Wjj = Pcard(I ) , (86)
j
k=1 ρk
where ρk is a material characteristic for substructure Ωk . This choice is robust with respect to jumps
in coefficients across the interface, cf. [6, 9]; however, coefficients are assumed constant for each
substructure. This requirement is very restrictive for practical computations with quickly varying
coefficients, and we employ a generalisation which takes into account the material coefficient of the
element to which the Lagrange multiplier λiΓ,j corresponds. In our case, we use ρi = d/tr (k−1 ),
where d ∈ {1, 2, 3} is the dimension of the element T i . This value can be seen as a representative
hydraulic conductivity on the element.
Finally, we propose a modification of the popular scaling by diagonal stiffness [19]. In the
usual diagonal stiffness approach, the optimal weight, which is the diagonal entry of the Schur
complement, is approximated by the diagonal entry of the original substructure matrix. However,
this is not directly applicable to the indefinite system (43), as, in general, matrix C i contains
only seldom nonzeros on the diagonal. For this reason, we approximate the diagonal of the Schur
complement as
i i 1
Wjj =C eΓΓ,jj + i , (87)
Akk
where the index k corresponds to the row in block Ai of the element face to which the Lagrange
multiplier λiΓ,j belongs.
Using the diagonal stiffness scaling in connection with the standard Lagrange finite elements may
lead to poor convergence for problems with rough interface [19, 38], for which the diagonal stiffness
can vary quickly even for smooth problems with constant coefficients on uniform meshes. This is a
severe issue for practical computations, in which graph partitioners are typically used for creating
substructures. However, this issue is not as pronounced for Raviart-Thomas elements, for which
only one element contributes to the stiffness on the diagonal at an interface degree of freedom, and
thus irregularities caused by changing number of elements contributing to an interface weight cannot
occur. On the other hand, an advantage of the diagonal stiffness scaling is the fact, that—unlike the
ρ-scaling—it takes into account the shape and relative sizes of elements, which vary considerably
in engineering applications, as well as the effect of δd introduced in (29) and (36). Unless stated
otherwise, scaling (87) is used in the computations presented in Section 8.
Preprint version.
16 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
The basis for an efficient parallel implementation of the method described in previous sections was
obtained by combining two existing open-source software packages: the finite element package
Flow123d † (version 1.6.5) for underground fluid flow simulations and the BDDC-based solver
BDDCML ‡ (version 2.0) used for the solution of the resulting system of equations. However, minor
changes have been made to both codes to support the specific features, such as the weights (86)
and (87).
The Flow123d package has been developed for modelling complex behaviour of underground
water flow and pollution transport. However, only the simple flow in a fully saturated porous
media described by Darcy’s law is considered in this paper. To accurately account for fractures
in the medium, such as granite rock, the solver allows us to combine finite elements of different
dimensions: the three-dimensional elements of porous media are combined with two-dimensional
elements modelling planar fractures, which may be in turn connected in one-dimensional elements
for channels. Raviart-Thomas elements are consistently used throughout such discretisation.
Although the fractures are also modelled as porous media, their hydraulic conductivity is by
orders of magnitude higher than that of the main porous material of the domain. In addition, the
finite element discretisations are typically not uniform within the domain, and the relative sizes
of elements may also vary by orders of magnitude. Both these aspects give rise to very poorly
conditioned linear systems, which are very challenging for iterative solvers. The Flow123d solver
has been developed for over 10 years and it is written in C/C++ programming language with object-
oriented design and parallelism through MPI.
The BDDCML is a library for solving algebraic systems of linear equations by means of the
BDDC method. The package supports the Adaptive-Multilevel BDDC method [22] suitable for very
high number of substructures and computer cores, although we only use the standard (non-adaptive
two-level) BDDC method from [6, 11] for the purpose of this paper. The BDDCML library is
typically interfaced by finite element packages, which may provide the division into substructures.
This feature is used in our current implementation, in which the division into non-overlapping
substructures is constructed within the Flow123d using the METIS (version 5.0) package [39].
One substructure is assigned to a processor core in the current set-up of the parallel solver, although
BDDCML is more flexible in this respect. The library performs the selection of additional corners
by the face-based algorithm from [21]. The BDDCML package is written in Fortran 95 and
parallelised through MPI.
The BDDCML solver relies on a serial instance of the MUMPS direct solver [40] for the solution
of each local discrete Dirichlet problem (45) as well as for the solution of each local discrete
Neumann problem (60). The coarse problem (62) is solved by a parallel instance of MUMPS. The
main difference from using BDDCML for symmetric positive definite problems is the need to use
the LDLT factorisation of general symmetric matrices for problems (45), which are saddle-point
(i.e. indefinite) systems in the present setting.
Although the original system (37) is indefinite, system (49) is symmetric positive definite, which
allows the use of the preconditioned conjugate gradient (PCG) method. One step of BDDC is used
as the preconditioner within the PCG method applied to problem (49). The matrix of problem (49)
is not explicitly constructed in the solver, and only its actions on vectors are computed following
(45)–(48).
Remark 10
In our implementation, we change the sign neither in the action of S i (46) nor in the action of the
preconditioner MBDDC (64). Since both are then strictly negative definite, the product MBDDC Sb is
the same as if both signs were changed, and the PCG method runs correctly. In this way, no changes
are necessary in an implementation developed for symmetric positive definite problems.
† http://flow123d.github.io
‡ http://users.math.cas.cz/
˜sistek/software/bddcml.html
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 17
8. NUMERICAL RESULTS
In this section, we investigate the performance of the algorithm and its parallel implementation
on several benchmark problems in 2D and 3D, and on two geoengineering problems of existing
localities in 3D. For the two benchmark problems without fractures we perform weak scaling tests.
For the benchmark problem with fractures and for the geoengineering problems, we perform strong
scaling tests with the problem size fixed and increasing number of processor cores. In all cases, the
PCG iterations are run until the relative norm of residual kr(k) k/kbbk < 10−7 . If not stated otherwise,
the proposed scaling by diagonal stiffness (87) is used within the averaging operator of BDDC.
time (sec)
N n n/N nΓ nf nc its. cond.
set-up PCG solve
2 207k 103k 155 1 2 7 1.37 8.3 1.6 9.9
4 440k 110k 491 5 10 8 1.60 12.2 2.2 14.4
8 822k 103k 1.2k 13 26 9 1.78 11.0 2.5 13.5
16 1.8M 111k 2.8k 33 66 8 1.79 14.3 2.7 17.0
32 3.3M 104k 5.9k 74 148 9 1.79 12.1 3.3 15.4
64 7.2M 113k 13.0k 166 332 9 1.85 14.8 4.4 19.2
Table I. Weak scaling test for the 2D square problem, each substructure problem contains approx. 100k
unknowns.
Preprint version.
18 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
Figure 2. Example of solution to the model square problem containing only 2D elements, plot of pressure
head with mesh (left) and velocity vectors (right).
Figure 3. Example of solution to the model cube problem containing only 3D elements, plot of pressure
head with mesh (left) and velocity vectors (right).
time (sec)
N n n/N nΓ nf nc its. cond.
set-up PCG solve
2 217k 108k 884 1 3 11 2.88 11.7 2.3 14.0
4 437k 109k 2.3k 6 18 12 3.04 11.7 2.5 14.2
8 945k 118k 5.7k 21 63 15 12.00 15.4 4.0 19.3
16 1.6M 103k 12.8k 56 168 16 6.58 12.9 4.0 17.0
32 3.4M 106k 29.8k 132 401 18 10.10 15.4 5.2 20.6
64 6.1M 95k 59.6k 307 931 19 16.58 13.7 6.3 20.0
Table II. Weak scaling test for the 3D cube problem, each substructure problem contains approx. 100k
unknowns.
The next benchmark problem is considerably more complicated. It consists again of a unit cube,
which now contains four planar fractures aligned with diagonals of a 2D cross-section. These four
planar fractures meet at a 1D channel in the centre of the cross-section. Therefore, the problem
contains the full possible combination of 3D, 2D and 1D finite elements. The tensor k is isotropic,
thus it is just a scalar multiple of identity. The corresponding scalar value is set to 10, 1, and 0.1 for
1D, 2D, and 3D elements, respectively.
We perform a strong scaling test with this problem, keeping the mesh size fixed with
approximately 2.1 million elements and 14.6 million degrees of freedom. In Fig. 5, the
computational mesh and the resulting pressure head and velocity fields are presented. This scaling
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 19
25 25
set-up set-up
PCG PCG
20 solve 20 solve
15 15
time [s]
time [s]
10 10
5 5
0 0
100 101 102 100 101 102
number of processors number of processors
Figure 4. Weak scaling test for the 2D square problem (left), and the 3D cube problem (right), approx. 100k
unknowns per core. Computational times separately for set-up and PCG phases, and their sum (solve).
test was computed on the Cray XE6 supercomputer Hector at the Edinburgh Parallel Computing
Centre. This supercomputer is composed of 2816 nodes, each containing two AMD Opteron
Interlagos processors with 16 cores at 2.3 GHz. GNU compilers version 4.6 were employed.
Results of the strong scaling test are summarised in Table III, and the computing times are
visualised also in Fig. 6 together with the parallel speed-up. The reference value for computing
speed-up is the time on 16 cores, and the speed-up on np processors is computed as
16 t16
snp = , (88)
tnp
Figure 5. Example of solution to the model cube problem containing 1D, 2D, and 3D elements, plot of
pressure head with mesh and fractures (left) and velocity vectors (right).
Preprint version.
20 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
time (sec)
N n/N nΓ nf nc its. cond.
set-up PCG solve
16 912k 47k 53 159 26 59.3 171.6 84.5 256.2
32 456k 65k 126 380 48 2091.0 90.1 109.8 200.0
64 228k 86k 301 914 81 1436.1 36.8 77.1 114.0
128 114k 116k 689 2076 109 2635.8 14.3 43.1 57.4
256 57k 151k 1436 4365 164 1700.5 6.7 31.2 38.0
512 28k 196k 3021 9244 254 42614.5 4.0 26.9 30.9
Table III. Strong scaling test for the cube problem with 1D, 2D, and 3D elements, size of the global problem
is n = 14.6M unknowns.
103
set-up
PCG
102 103 solve
optimal
speed-up
time [s]
101
102
set-up
0
10 PCG
solve
optimal
10-1 1 101 1
10 102 103 10 102 103
number of processors number of processors
Figure 6. Strong scaling test for the cube problem with 1D, 2D, and 3D elements and 14.6M unknowns,
computational time (left) and speed-up (right) separately for set-up and PCG phases, and their sum (solve).
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 21
∆y =12100 m
∆z =1410 m
∆x =9940 m
z
y
x
Figure 7. The problem of the Melechov locality containing 2D and 3D elements, mesh contains 2.1M
elements and 15M unknowns. Plot of the piezometric head. Data by courtesy of Jiřina Královcová.
Figure 8. The problem of the Melechov locality; the system of fractures (left) and an example division into
64 substructures (right).
time (sec)
N n/N nΓ nf nc its. cond.
set-up PCG solve
16 934k 36k 32 96 40 53.0 131.4 144.1 275.6
32 467k 54k 76 228 70 878.3 47.5 112.9 160.4
64 233k 82k 186 561 67 202.4 17.4 50.2 67.7
128 117k 116k 528 1592 69 237.6 7.9 23.1 31.1
256 58k 155k 1235 3747 96 5577.0 4.0 14.7 18.8
512 29k 207k 2699 8256 106 1658.1 2.2 8.3 10.5
1024 15k 271k 5711 17581 119 11554.5 2.1 7.0 9.2
Table IV. Strong scaling test for the problem of the Melechov locality containing 2D and 3D elements, size
of the global problem is n = 15M unknowns.
this locality is also a valuable site for experimental geological measurements performed inside the
tunnel.
The model aims at describing the flow in the granite rock surrounding the tunnel. The
computational mesh consists of 1.1 million elements leading to 7.8 million unknowns. The mesh
with the plot of resulting piezometric head is presented in Fig. 10. The system of fractures and an
example division into 256 substructures are visualised in Fig. 11. The hydraulic conductivity of the
fractures is 10−7 ms−1 , while the conductivity of the outer material is 10−10 ms−1 , the transition
coefficient σ3 = 1 s−1 , and the effective thickness of fractures δ2 = 1.1 m.
Preprint version.
22 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
103
set-up
PCG
2 3
10 10 solve
optimal
speed-up
time [s]
1
10
102
set-up
100 PCG
solve
optimal
10-1 1 101 1
10 102 103 10 102 103
number of processors number of processors
Figure 9. Strong scaling test for the problem of the Melechov locality containing 2D and 3D elements and
15M unknowns, computational time (left) and speed-up (right) separately for set-up and PCG phases, and
their sum (solve).
Although the mesh contains fewer finite elements than the one of the Melechov locality model,
this problem is considerably more complicated. This is caused mainly by the presence of relatively
very small and irregularly shaped finite elements in the vicinity of the tunnel and near the cross-
sections of fractures (see Fig. 12) generated by the mesh generator.
The results of a strong scaling test are summarised in Table V. As before, the times are also plotted
in Fig. 13. Although the number of iterations is not independent of the number of substructures, the
growth is still small. Consequently, the computing times, and especially the time for set-up, scale
very well over a large range of numbers of substructures. The observed super-optimal scaling may
be related to faster factorisation of the smaller local problems by the direct solver for indefinite
matrices.
Table VI summarises an experiment performed to analyse the effect of using corners in the
construction of the coarse space in BDDC. As has been mentioned in Section 5, using Raviart-
Thomas finite elements does not lead to ‘natural’ corners as cross-points shared by several
substructures. On the other hand, the notion of corners was generalised to any selected interface
degree of freedom, at which continuity of functions from the coarse space is required. Such
generalisation is important for the well-posedness of the local problems for unstructured meshes e.g.
in elasticity analysis [21]. This is also the default option for BDDCML, in which selection of corners
is performed at each face between two substructures. Adding corners improves the approximation
properties of the coarse space at the cost of increasing the size of the coarse problem. Table VI
compares the convergence for variable number of substructures without and with constraints at
corners. Column ‘with corners’ in Table VI corresponds to results in Table V, and it is repeated for
comparison. We can see that while the effect of corners on convergence is small for smaller number
of substructures, the improvement of the coarse problem and the approximation of BDDC becomes
more significant for higher numbers of cores. Looking at times in Table VI, the additional time spent
in the set-up phase due to higher number of constraints when using corners is compensated by the
lower number of PCG iterations, resulting in lower overall times. Thus, using additionally selected
corners appears beneficial for complicated engineering problems like this one.
In the final experiment, we compare the effect of different averaging techniques on the
convergence of BDDC. In Table VII, results of the strong scaling test for arithmetic averaging (85),
the modified ρ-scaling (86), and the proposed scaling by diagonal stiffness (87) are summarised.
The final column corresponds to the results from Table V, which are repeated here for comparison.
Table VII suggests, that while the simple arithmetic averaging does not lead to satisfactory
convergence for this problem, the modified ρ-scaling and the diagonal scaling mostly lead to similar
convergence. However, while the former provides slightly better convergence for several cases, it
also leads to irregularities for certain divisions, for which the BDDC method with this averaging
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 23
2655 m (length)
458 m (height)
1754 m (width)
Figure 10. The Bedřichov tunnel problem containing 2D and 3D elements, mesh contains 1.1M elements
and 7.8M unknowns. Plot of the piezometric head. Data by courtesy of Dalibor Frydrych.
Figure 11. The Bedřichov tunnel problem; the system of fractures (left) and an example division into 256
substructures (right).
Figure 12. Example of difficulties in the mesh of the Bedřichov tunnel problem; detail of the tunnel geometry
with fine elements (left) and enforced refinement at an intersection of fractures (right).
converges rather poorly. Therefore, the proposed scaling (87) can be recommended as the most
robust choice among the three tested options.
Preprint version.
24 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
time (sec)
N n/N nΓ nf nc its. cond.
set-up PCG solve
32 245k 20k 106 322 112 1514.1 110.3 144.0 254.3
64 123k 28k 192 597 63 117.7 42.2 36.0 78.3
128 61k 45k 413 1293 75 194.4 13.4 16.8 30.3
256 31k 72k 902 2791 119 526.7 4.2 10.9 15.1
512 15k 110k 2009 6347 137 1143.4 1.8 7.1 9.0
1024 8k 155k 4575 14725 173 897.0 1.6 8.0 9.7
Table V. Strong scaling test for the problem of the Bedřichov tunnel containing 2D and 3D elements, size
of the global problem is n = 7.8M unknowns.
103
set-up
PCG
3
102 10 solve
optimal
speed-up
time [s]
101
102
set-up
100 PCG
solve
optimal
10-1 1 101 1
10 102 103 10 102 103
number of processors number of processors
Figure 13. Strong scaling test for the problem of the Bedřichov tunnel containing 2D and 3D elements and
7.8M unknowns, computational time (left) and speed-up (right) separately for set-up and PCG phases, and
their sum (solve).
Table VI. Effect of using corners. Problem of the Bedřichov tunnel containing 2D and 3D elements, size of
the global problem is n = 7.8M unknowns. In column ‘without corners’, no additional corners are selected
in BDDC. In column ‘with corners’, additional corners are selected.
9. CONCLUSION
A parallel solver for the mixed-hybrid finite element formulation based on Darcy’s law has been
presented. The software combines an existing package Flow123d developed for problems in
geophysics with BDDCML, a parallel library for solution of systems of linear algebraic equations
by the BDDC method.
In geoengineering applications, the mathematical model is applied to geometries with the
presence of fractures. In the present approach, the flow in these fractures is also modelled by Darcy’s
law, although the hydraulic conductivity of the porous media is considered by orders of magnitude
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 25
Table VII. Comparison of different averaging techniques for the Bedřichov tunnel containing 2D and 3D
elements, size of the global problem is n = 7.8M unknowns.
higher. These fractures are modelled by finite elements of a lower dimension. In the discretised
model, 1D, 2D and 3D finite elements are coupled together through Robin boundary conditions.
These coupling terms lead to a modification of the usual saddle-point matrix of the system, in which
a new non-zero block appears on the diagonal.
The BDDC method is employed for the solution of the resulting system of linear algebraic
equations. BDDC is based on iterative substructuring, in which the problem is first reduced to
the interface among substructures. The Schur complement is not built explicitly. Instead, only
multiplications by the matrix are performed through solving a discrete Dirichlet problem on each
substructure. In the setting of the mixed-hybrid problem, the interface is built only as a subset of
the block of Lagrange multipliers, while remaining unknowns belong to interiors of substructures.
Although the original problem is symmetric indefinite, the system reduced to the interface is
symmetric positive definite. This is also shown to hold for the case with fractures in the present
paper. Consequently, the PCG method is used for the solution of the reduced problem. However,
unlike the symmetric positive definite problems, a direct solver for symmetric indefinite matrices
needs to be used for the factorisation and repeated solution of local problems on substructures.
One step of the BDDC method is used as the preconditioner for the PCG method run on the
interface problem. A modification of the diagonal stiffness scaling has been introduced. It is
motivated by difficult engineering problems, for which it performs significantly better than other
two applicable choices—the arithmetic averaging and the modified ρ-scaling. Arithmetic averages
over faces between substructures are used as the basic constraints defining the coarse space. In
addition, corners are selected from unknowns at the interface using the face-based algorithm. While
corners are not required by the theory, they are shown to improve both the convergence and the
computational times for complicated problems.
The performance of the resulting solver has been investigated on three benchmark problems in
2D and 3D. Both weak and strong scaling tests have been performed. On benchmark problems with
single mesh dimension, the expected optimal convergence independent of number of substructures
has been achieved. Correspondingly, the resulting parallel scalability has been nearly optimal for
the weak scaling tests up to 64 computer cores.
The strong scaling tests were presented for a benchmark problem of a unit cube and for two
engineering problems containing large variations in element sizes and hydraulic conductivities,
using up to 1024 computer cores and containing up to 15 million degrees of freedom. The
convergence for the unit cube problem with all three possible dimensions of finite elements slightly
deteriorated by using more substructures, and this translated to sub-optimal parallel performance.
However, for the two engineering applications, in which only 3D and 2D elements are combined,
the BDDC method has also maintained good convergence properties with the growing number of
substructures, resulting in optimal, or even super-optimal parallel scalability of the solver. It has been
also shown that the proposed modification of the diagonal stiffness scaling plays an important role
in achieving such independence for the challenging engineering problems presented in the paper.
ACKNOWLEDGEMENT
Preprint version.
26 J. ŠÍSTEK, J. BŘEZINA, B. SOUSEDÍK
The authors are grateful to Jiřina Královcová and Dalibor Frydrych for providing the geoengineering models.
REFERENCES
1. Gulbransen AF, Hauge VL, Lie KA. A multiscale mixed finite-element method for vuggy and naturally-fractured
reservoirs. SPE Journal 2010; 15(2):395–403.
2. Martin V, Jaffré J, Roberts JE. Modeling fractures and barriers as interfaces for flow in porous media. SIAM Journal
on Scientific Computing 2005; 26(5):1667–1691.
3. Benzi M, Golub GH, Liesen J. Numerical solution of saddle point problems. Acta Numerica 2005; 14:1–137.
4. Dohrmann CR, Lehoucq RB. A primal-based penalty preconditioner for elliptic saddle point systems. SIAM Journal
on Numerical Analysis 2006; 44(1):270–282.
5. Tu X. A BDDC algorithm for mixed formulation of flow in porous media. Electronic Transactions on Numerical
Analysis 2005; 20:164–179.
6. Tu X. A BDDC algorithm for flow in porous media with a hybrid finite element discretization. Electronic
Transactions on Numerical Analysis 2007; 26:146–160.
7. Vassilevski PS. Multilevel block factorization preconditioners: matrix-based analysis and algorithms for solving
finite element equations. Springer, 2008.
8. Elman HC, Silvester DJ, Wathen AJ. Finite elements and fast iterative solvers: with applications in incompressible
fluid dynamics. Numerical Mathematics and Scientific Computation, Oxford University Press, 2005.
9. Toselli A, Widlund OB. Domain Decomposition Methods—Algorithms and Theory, Springer Series in
Computational Mathematics, vol. 34. Springer-Verlag, 2005.
10. Cros JM. A preconditioner for the Schur complement domain decomposition method. Domain Decomposition
Methods in Science and Engineering, Herrera I, Keyes DE, Widlund OB (eds.). National Autonomous University
of Mexico (UNAM), México, 2003; 373–380. 14th International Conference on Domain Decomposition Methods,
Cocoyoc, Mexico, January 6–12, 2002.
11. Dohrmann CR. A preconditioner for substructuring based on constrained energy minimization. SIAM J. Sci.
Comput. 2003; 25(1):246–258.
12. Fragakis Y, Papadrakakis M. The mosaic of high performance domain decomposition methods for structural
mechanics: Formulation, interrelation and numerical efficiency of primal and dual methods. Computer Methods
in Applied Mechanics and Engineering 2003; 192:3799–3830.
13. Mandel J, Sousedı́k B. BDDC and FETI-DP under minimalist assumptions. Computing 2007; 81:269–280.
14. Sousedı́k B, Mandel J. On the equivalence of primal and dual substructuring preconditioners. Electronic
Transactions on Numerical Analysis 2008; 31:384–402.
15. Li J, Widlund OB. BDDC algorithms for incompressible Stokes equations. SIAM Journal on Numerical Analysis
2006; 44(6):2432–2455.
16. Tu X, Li J. A balancing domain decomposition method by constraints for advection-diffusion problems.
Communications in Applied Mathematics and Computational Science 2008; 3(1):25–60.
17. Tu X. Three-level BDDC in three dimensions. SIAM Journal on Scientific Computing 2007; 29(4):1759–1780.
18. Mandel J, Sousedı́k B, Dohrmann CR. Multispace and multilevel BDDC. Computing 2008; 83(2-3):55–85.
19. Klawonn A, Rheinbach O, Widlund OB. An analysis of a FETI-DP algorithm on irregular subdomains in the plane.
SIAM Journal on Numerical Analysis 2008; 46(5):2484–2504.
20. Mandel J, Sousedı́k B. Adaptive selection of face coarse degrees of freedom in the BDDC and the FETI-DP iterative
substructuring methods. Computer Methods in Applied Mechanics and Engineering 2007; 196(8):1389–1399.
21. Šı́stek J, Čertı́ková M, Burda P, Novotný J. Face-based selection of corners in 3D substructuring. Mathematics and
Computers in Simulation 2012; 82(10):1799–1811.
22. Sousedı́k B, Šı́stek J, Mandel J. Adaptive-Multilevel BDDC and its parallel implementation. Computing 2013;
95(12):1087–1119.
23. Oh DS, Widlund OB, Dohrmann CR. A BDDC algorithm for Raviart-Thomas vector fields. Technical Report TR-
951, Courant Institue of Mathematical Sciences Feb 2013. Department of Computer Science.
24. Sousedı́k B. Nested BDDC for a saddle-point problem. Numerische Mathematik 2013; 125(4):761–783.
25. Tu X. A three-level BDDC algorithm for a saddle point problem. Numerische Mathematik 2011; 119(1):189–217.
26. Maryška J, Rozložnı́k M, Tůma M. Mixed-hybrid finite element approximation of the potential fluid flow problem.
Journal of Computational and Applied Mathematics 1995; 63:383–392.
27. Oden J, Lee J. Dual-mixed hybrid finite element method for second-order elliptic problems. Mathematical Aspects
of Finite Element Methods, Lecture Notes in Mathematics, vol. 606, Galligani I, Magenes E (eds.). Springer Berlin
Heidelberg, 1977; 275–291.
28. Maryška J, Rozložnı́k M, Tůma M. Schur complement systems in the mixed-hybrid finite element approximation
of the potential fluid flow problem. SIAM Journal on Scientific Computing 2000; 22(2):704–723.
29. Maryška J, Severýn O, Vohralı́k M. Numerical simulation of fracture flow with a mixed-hybrid FEM stochastic
discrete fracture network model. Computational Geosciences 2005; 8:217–234.
30. Březina J, Hokr M. Mixed-hybrid formulation of multidimensional fracture flow. Proceedings of the 7th
international conference on Numerical methods and applications, NMA’10, Springer-Verlag: Berlin, Heidelberg,
2011; 125–132.
31. Bear J. Dynamics of Fluids in Porous Media. Dover, 1988.
32. Chen Z, Huan G, Ma Y. Computational Methods for Multiphase Flows in Porous Media. SIAM, 2006.
33. Brezzi F, Fortin M. Mixed and Hybrid Finite Element Methods. Springer-Verlag, 1991.
34. Cowsar LC, Mandel J, Wheeler MF. Balancing domain decomposition for mixed finite elements. Mathematics of
Computation 1995; 64(211):989–1015.
Preprint version.
BDDC FOR POROUS MEDIA WITH COMBINED MESH DIMENSIONS 27
35. Pultarová I. Preconditioning of the coarse problem in the method of balanced domain decomposition by constraints.
Math. Comput. Simulation 2012; 82(10):1788–1798.
36. Mandel J, Dohrmann CR, Tezaur R. An algebraic theory for primal and dual substructuring methods by constraints.
Appl. Numer. Math. 2005; 54(2):167–193.
37. Mandel J, Dohrmann CR. Convergence of a balancing domain decomposition by constraints and energy
minimization. Numerical Linear Algebra with Applications 2003; 10(7):639–659.
38. Čertı́ková M, Šı́stek J, Burda P. On selection of interface weights in domain decomposition methods. Proceedings
of Programs and Algorithms of Numerical Mathematics 16, Dolnı́ Maxov, Czech Republic, June 3–8, 2012. Institute
of Mathematics AS CR, 2013; 35–44.
39. Karypis G, Kumar V. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on
Scientific Computing 1998; 20(1):359–392.
40. Amestoy PR, Duff IS, L’Excellent JY. Multifrontal parallel distributed symmetric and unsymmetric solvers.
Computer Methods in Applied Mechanics and Engineering 2000; 184:501–520.
41. Mandel J, Sousedı́k B, Šı́stek J. Adaptive BDDC in three dimensions. Mathematics and Computers in Simulation
2012; 82(10):1812–1831.
Preprint version.