Chapter 7
Chapter 7
Replication
Why replicate
Assume a simple model in which we make a copy of a specific part of a system
(meaning code and data).
• Increase reliability: if one copy does not live up to specifications, switch
over to the other copy while repairing the failing one.
• Performance: simply spread requests between different replicated parts
to keep load balanced, or to ensure quick responses by taking proximity
into account.
The problem
Having multiple copies, means that when any copy changes, that change
should be made at all copies: replicas need to be kept the same, that is, be
kept consistent.
Issue
Guaranteeing global ordering on conflicting operations may be a costly
operation, downgrading scalability. Solution: weaken consistency requirements
so that hopefully global synchronization can be avoided
Essential
A data store is a distributed collection of storages:
Consistency and replication Data-centric consistency models
Some notations
Read and write operations
• Wi (x)a: Process Pi writes value a to x
• Ri (x)b: Process Pi reads value b from x
• All data items initially have value NIL
Possible behavior
We omit the index when possible and draw according to time (x-axis):
Sequential consistency
Definition
The result of any execution is the same as if the operations of all processes
were executed in some sequential order, and the operations of each individual
process appear in this sequence in the order specified by its program.
Example
Three concurrent processes (initial values: 0)
Process P1 Process P2 Process P3
x ← 1; y ← 1; z ← 1;
print(y,z); print(x,z); print(x,y);
Example
Three concurrent processes (initial values: 0)
Process P1 Process P2 Process P3
x ← 1; y ← 1; z ← 1;
print(y,z); print(x,z); print(x,y);
Causal consistency
Definition
Writes that are potentially causally related must be seen by all processes in the
same order. Concurrent writes may be seen in a different order by different
processes.
A number of schedules
Time −→
S1 x=0 x=x+1 x=0 x=x+2 x=0 x=x+3 Legal
S2 x=0 x=0 x=x+1 x=x+2 x=0 x=x+3 Legal
S3 x=0 x=0 x=x+1 x=0 x=x+2 x=x+3 Illegal
S4 x=0 x=0 x=x+3 x=0 x=x+1 x=x+2 Illegal
Grouping operations
Entry consistency: Definition
• Accesses to locks are sequentially consistent.
• No access to a lock is allowed to be performed until all previous writes
have completed everywhere.
• No data access is allowed to be performed until all previous accesses to
locks have been performed.
Grouping operations
Entry consistency: Definition
• Accesses to locks are sequentially consistent.
• No access to a lock is allowed to be performed until all previous writes
have completed everywhere.
• No data access is allowed to be performed until all previous accesses to
locks have been performed.
Basic idea
You don’t care that reads and writes of a series of operations are immediately
known to other processes. You just want the effect of the series itself to be
known.
Grouping operations
A valid event sequence for entry consistency
Observation
Entry consistency implies that we need to lock and unlock data (implicitly or
not).
Question
What would be a convenient way of making this consistency more or less
transparent to programmers?
Eventual consistency
Definition
Consider a collection of data stores and (concurrent) write operations. The
strores are eventually consistent when in lack of updates from a certain
moment, all updates to that point are propagated in such a way that replicas
will have the same data stored (until updates are accepted again).
Program consistency
P is a monotonic problem if for any input sets S and T , P(S) ⊆ P(T ).
Observation: A program solving a monotonic problem can start with
incomplete information, but is guaranteed not to have to roll back when
missing information becomes available. Example: filling a shopping cart.
Eventual consistency
Consistency and replication Data-centric consistency models
Eventual consistency
Definition
Consider a collection of data stores and (concurrent) write operations. The
strores are eventually consistent when in lack of updates from a certain
moment, all updates to that point are propagated in such a way that replicas
will have the same data stored (until updates are accepted again).
Program consistency
P is a monotonic problem if for any input sets S and T , P(S) ⊆ P(T ).
Observation: A program solving a monotonic problem can start with
incomplete information, but is guaranteed not to have to roll back when
missing information becomes available. Example: filling a shopping cart.
Important observation
In all cases, we are avoiding global synchronization.
Eventual consistency
Consistency and replication Data-centric consistency models
Continuous Consistency
We can actually talk about a degree of consistency
• replicas may differ in their numerical value
• replicas may differ in their relative staleness
• there may be differences regarding (number and order) of performed
update operations
Conit
Consistency unit ⇒ specifies the data unit over which consistency is to be
measured.
Continuous consistency
Consistency and replication Data-centric consistency models
Example: Conit
Continuous consistency
Consistency and replication Data-centric consistency models
Example: Conit
Continuous consistency
Consistency and replication Client-centric consistency models
Note
The only thing you really want is that the entries you updated and/or read at A,
are in B the way you left them in A. In that case, the database will appear to be
consistent to you.
Consistency and replication Client-centric consistency models
Basic architecture
The principle of a mobile user accessing different replicas of a
distributed database
Consistency and replication Client-centric consistency models
Monotonic reads
Consistency and replication Client-centric consistency models
Monotonic reads
Example
Automatically reading your personal calendar updates from different servers.
Monotonic reads guarantees that the user sees all updates, no matter from
which server the automatic reading takes place.
Example
Reading (not modifying) incoming mail while you are on the move. Each time
you connect to a different e-mail server, that server fetches (at least) all the
updates from the server you previously visited.
Monotonic reads
Consistency and replication Client-centric consistency models
Monotonic reads
Definition
If a process reads the value of a data item x, any successive read operation on
x by that process will always return that same or a more recent value.
Monotonic reads
Consistency and replication Client-centric consistency models
Monotonic writes
Example
Updating a program at server S2 , and ensuring that all components on which
compilation and linking depends, are also placed at S2 .
Example
Maintaining versions of replicated files in the correct order everywhere
(propagate the previous version to the server where the newest version is
installed).
Monotonic writes
Consistency and replication Client-centric consistency models
Monotonic writes
Definition
A write operation by a process on a data item x is completed before any
successive write operation on x by the same process.
OK Not OK
Not OK OK
Monotonic writes
Consistency and replication Client-centric consistency models
OK
Not OK
OK
Not OK
Example
Updating your Web page and guaranteeing that your Web browser shows the
newest version instead of its cached copy.
OK
Not OK
OK
Not OK
Example
See reactions to posted articles only if you have the original posting (a read
“pulls in” the corresponding write operation).
Replica placement
Essence
Figure out what the best K places are out of N possible locations.
Replica placement
Essence
Figure out what the best K places are out of N possible locations.
• Select best location out of N − K for which the average distance to clients
is minimal. Then choose the next best server. (Note: The first chosen
location minimizes the average distance to all clients.) Computationally
expensive.
Replica placement
Essence
Figure out what the best K places are out of N possible locations.
• Select best location out of N − K for which the average distance to clients
is minimal. Then choose the next best server. (Note: The first chosen
location minimizes the average distance to all clients.) Computationally
expensive.
• Select the K -th largest autonomous system and place a server at the
best-connected host. Computationally expensive.
Replica placement
Essence
Figure out what the best K places are out of N possible locations.
• Select best location out of N − K for which the average distance to clients
is minimal. Then choose the next best server. (Note: The first chosen
location minimizes the average distance to all clients.) Computationally
expensive.
• Select the K -th largest autonomous system and place a server at the
best-connected host. Computationally expensive.
• Position nodes in a d-dimensional geometric space, where distance
reflects latency. Identify the K regions with highest density and place a
server in every one. Computationally cheap.
Content replication
Distinguish different processes
A process is capable of hosting a replica of an object or data:
• Permanent replicas: Process/machine always having a replica
• Server-initiated replica: Process that can dynamically host a replica on
request of another server in the data store
• Client-initiated replica: Process that can dynamically host a replica on
request of a client (client cache)
Content replication
The logical organization of different kinds of copies of a data store into
three concentric rings
Server-initiated replicas
Counting access requests from different clients
Content distribution
Consider only a client-server combination
• Propagate only notification/invalidation of update (often used for caches)
• Transfer data from one copy to another (distributed databases): passive
replication
• Propagate the update operation to other copies: active replication
Note
No single approach is the best, but depends highly on available bandwidth and
read-to-write ratio at replicas.
Content distribution
Consistency and replication Replica management
Content distribution
Consistency and replication Replica management
Content distribution
Observation
We can dynamically switch between pulling and pushing using leases: A
contract in which the server promises to push updates to the client until the
lease expires.
Content distribution
Consistency and replication Replica management
Content distribution
Observation
We can dynamically switch between pulling and pushing using leases: A
contract in which the server promises to push updates to the client until the
lease expires.
Content distribution
Consistency and replication Replica management
Content distribution
Observation
We can dynamically switch between pulling and pushing using leases: A
contract in which the server promises to push updates to the client until the
lease expires.
Content distribution
Consistency and replication Replica management
Content distribution
Observation
We can dynamically switch between pulling and pushing using leases: A
contract in which the server promises to push updates to the client until the
lease expires.
• State-based leases: The more loaded a server is, the shorter the
expiration times become
Content distribution
Consistency and replication Replica management
Content distribution
Observation
We can dynamically switch between pulling and pushing using leases: A
contract in which the server promises to push updates to the client until the
lease expires.
Question
Why are we doing all this?
Content distribution
Consistency and replication Replica management
Replicated-object invocations
Problem when invocating a replicated object
Replicated-object invocations
Primary-based protocols
Primary-backup protocol
Primary-based protocols
Primary-backup protocol
Primary-based protocols
Primary-backup protocol with local writes
Primary-based protocols
Primary-backup protocol with local writes
Replicated-write protocols
Quorum-based protocols
Assume N replicas. Ensure that each operation is carried out in such a way
that a majority vote is established: distinguish read quorum NR and write
quorum NW . Ensure:
1. NR + NW > N (prevent read-write conflicts)
2. NW > N/2 (prevent write-write conflicts)
∀W : val(W ) > 0
value vi of x at server Si :
N
vi = vinit + ∑ TW [i, k ]
k=1
Approach
Let every server Sk maintain a view TWk [i, j] of what it believes is the value of
TW [i, j]. This information can be gossiped when an update is propagated.
Approach
Let every server Sk maintain a view TWk [i, j] of what it believes is the value of
TW [i, j]. This information can be gossiped when an update is propagated.
Note
Question
To what extent are we being pessimistic here: where does δi /(N − 1) come
from?
Question
To what extent are we being pessimistic here: where does δi /(N − 1) come
from?
Note
Staleness can be done analogously, by essentially keeping track of what has
been seen last from Si (see book).
Monotonic-read consistency
When client C wants to read at server S, C passes its read set. S can pull in
any updates before executing the read operation, after which the read set is
updated.
Monotonic-read consistency
When client C wants to read at server S, C passes its read set. S can pull in
any updates before executing the read operation, after which the read set is
updated.
Monotonic-write consistency
When client C wants to write at server S, C passes its write set. S can pull in
any updates, executes them in the correct order, and then executes the write
operation, after which the write set is updated.
Writes-follows-reads consistency
When client C wants to write at server S, C passes its read set. S can pull in
any updates, executes them in the correct order, and then executes the write
operation, after which the write set is updated.
Caches at ISPs
Internet Service Providers also place caches to (1) reduce cross-ISP traffic
and (2) improve client-side performance. May get nasty when a request needs
to pass many ISPs.
Consistency and replication Example: Caching and replication in the Web
Cooperative caching
Consistency and replication Example: Caching and replication in the Web
Web-cache consistency
How to guarantee freshness?
To prevent that stale information is returned to a client:
• Option 1: let the cache contact the original server to see if content is still
up to date.
• Option 2: Assign an expiration time Texpire that depends on how long ago
the document was last modified when it is cached. If Tlast modified is the
last modification time of a document (as recorded by its owner), and
Tcached is the time it was cached, then
• Database copy: the edge has the same as the origin server
• Content-aware cache: check if a (normal query) can be answered with
cached data. Requires that the server knows about which data is cached
at the edge.
• Content-blind cache: store a query, and its result. When the exact same
query is issued again, return the result from the cache.