Chapter 5 Synchronization

1
Chapter 5 Synchronization

• Clocks and Synchronization Algorithms
• Lamport Timestamps and Vector Clocks
• Distributed Snapshots
• Termination Detection
• Election Algorithms
• Distributed Mutual Exclusion
• Transactions concurrency control

2
What Do We Mean By Time?

• Monotonic increasing
• Useful when everyone agrees on it
• UTC is Universal Coordinated Time.
• NIST operates on a short wave radio frequency WWV
  and transmits UTC from Colorado.

3
Clock Synchronization

• When each machine has its own clock, an event
  that occurred after another event may
  nevertheless be assigned an earlier time.

4
Time

• Time is complicated in a distributed system.
• Physical clocks run at slightly different rates
  so they can drift apart.
• Clock makers specify a maximum drift rate ?
  (rho).
• By definition
• 1-? lt dC/dt lt 1? where C(t) is the
  clocks time as a function of the real time.

5
Clock Synchronization

• The relation between clock time and UTC when
  clocks tick at different rates.

6
Clock Synchronization

• 1-? lt dC/dt lt 1?
• A perfect clock has dC/dt 1
• Assuming 2 clocks have the same max drift rate ?.
  To keep them synchronized to within a time
  interval delta, ?, they must re-sync every ?/2?
  seconds.

7
Cristians Algorithm

• One of the nodes (or processors) in the
  distributed system is a time server TS
  (presumably with access to UTC). How can the
  other nodes be synced?
• Periodically, at least every ?/2? seconds, each
  machine sends a message to the TS asking for the
  current time and the TS responds.

8
Cristian's Algorithm

• Getting the current time from a time server.

9
Cristians Algorithm

• Should the client node simply force his clock to
  the value in the message??
• Potential problem if clients clock was fast,
  new time may be less than his current time, and
  just setting the clock to the new time might make
  time appear to run backwards on that node.
• TIME MUST NEVER RUN BACKWARDS. There are many
  applications that depend on the fact that time is
  always increasing. So new time must be worked in
  gradually.

10
Cristians Algorithm

• Can we compensate for the delay from when TS
  sends the response to T1 (when it is received)?
• Add (T1 T0)/2. If no outside info is
  available.
• Estimate or ask server how long it takes to
  process time request, say R. Then add (T1 T0
  R)/2.
• Take several measurements and taking the smallest
  or an average after throwing out the large values.

11
The Berkeley Algorithm

• The server actively tries to sync the clocks of a
  DS. This algorithm is appropriate if no one has
  UTC and all must agree on the time.
• Server polls each machine by sending his
  current time and asking for the difference
  between his and theirs. Each site responds with
  the difference.
• Server computes average with some compensation
  for transmission time.
• Server computes how each machine would need to
  adjust his clock and sends each machine
  instructions.

12
The Berkeley Algorithm

1. The time daemon asks all the other machines for
   their clock values
2. The machines answer
3. The time daemon tells everyone how to adjust
   their clock

13
Analysis of Sync Algorithms

• Cristians algorithm N clients send and receive
  a message every ?/2? seconds.
• Berkeley algorithm 3N messages every ?/2?
  seconds.
• Both assume a central time server or coordinator.
  More distributed algorithms exist in which each
  processor broadcasts its time at an agreed upon
  time interval and processors go through an
  agreement protocol to average the value and agree
  on it.

14
Analysis of Sync Algorithms

• In general, algorithms with no coordinator have
  greater message complexity (more messages for the
  same number of nodes). Thats the price you pay
  for equality and no-single-point-of-failure.
• With modern hardware, we can achieve loosely
  synchronized clocks. This forms the basis for
  many distributed algorithms in which logical
  clocks are used with physical clock timestamps to
  disambiguate when logical clocks roll over or
  servers crash and sequence numbers start over
  (which is inevitable in real implementations).

15
Logical Clocks

• What do we really need in a clock? For many
  applications, it is not necessary for nodes of a
  DS to agree on the real time, only that they
  agree on some value that has the attributes of
  time.
• Attributes of time X(t) has the sense or
  attributes of time if it is strictly increasing.
• A real or integer counter can be used. A real
  number would be closer to reality, however, an
  integer counter is easier for algorithms and
  programmers. Thus, for convenience, we use an
  integer which is incremented anytime an event of
  possible interest occurs.

16
Logical Clocks in a DS

• What is important is usually not when things
  happened but in what order they happened so the
  integer counter works well in a centralized
  system.
• However, in a DS, each system has its own logical
  clock, and you can run into problems if one
  clock gets ahead of others. (like with
  physical clocks)
• We need a rule to synchronize the logical clocks.

17
Lamport Clocks

• Lamport defined the happens-before relation for
  DS.
• A ? B means A happens before B.
• If A and B are events in the same process and A
  occurs before B then A ? B is true.
• If A is the event of a message being sent by one
  process-node and B is the event of that message
  being received by another process, then then A ?
  B is true. (A message must be sent before it is
  received).
• Happens-before is the transitive closure of 1 and
  2. That is, if A?B and B?C, then A?C.
• Any other events are said to be concurrent.

18
Lamport Clocks
P
Q
R
q1
p1
q2
p2
r1
q3
r2
p3

• p1?q2 and q1?p2 and q1?q2 but p1 and q1 are
  incomparable. p1?q3 and p1?r2 Does p1 ? r1?

19
Lamport Clocks

• Desired properties
• (1) anytime A? B , C(A) lt C(B), that is the
  logical clock value of the earlier event is less
• (2) the clock value C is increasing (never runs
  backwards)

20
Lamport Clocks Rules

• An event is an internal event or a message send
  or receive.
• The local clock is increased by one for each
  message sent and the message carries that
  timestamp with it.
• The local clock is increased for an internal
  event.
• When a message is received, the current local
  clock value, C, is compared to the message
  timestamp, T. If the message timestamp, T C,
  then set the local clock value to C1. If T gt C,
  set the clock to T1. If TltC, set the clock to
  C1.

21
Lamport Clocks
P
Q
R
2
5
3
6
4
6
7
8
7

• Anytime A? B , C(A) lt C(B)
• However, C(E) lt C(F) doesnt mean E? F
• (ex event 6 on P may not proceed event 7 on Q)

22
Lamport Timestamps

• If you need a total ordering, (distinguish
  between event 6 on P and event 6 on Q) use
  Lamport timestamps.
• Lamport timestamp of event A at node i is (C(A),
  i)
• For any 2 timestamps T1(C(A),I) and T2(C(B),J)
• If C(A) gt C(B) then T1 gt T2.
• If C(A) lt C(B) then T1 lt T2.
• If C(A) C(B) then consider node numbers. If
  IgtJ then T1 gt T2. If IltJ then T1 lt T2. If IJ
  then the two events occurred at the same node, so
  since their clock C is the same, they must be the
  same event.

23
Lamport Timestamps
P1
P2
P3
(2,2)
(5,1)
(3,3)
(6,2)
(4,3)
(6,1)
(7,2)
(8,3)
(7,1)

• (6,1) ? (6,2) and (4,3) ? (6,2)

24
Lamport Timestamps

1. Three processes, each with its own clock. The
   clocks run at different rates.
2. Lamport's algorithm corrects the clocks.

25
Exercise Lamport Clocks and Timestamps
1 2 3 4
5 6
7
A B C

• Assuming the only events are message send and
  receive, what are the clocks at 1-7

26
Why Vector Timestamps

• Lamport timestamps gives us the property if A ? B
  then C(A) lt C(B). But it doesnt give us the
  property if C(A) lt C(B) then A?B. (if C(A) lt
  C(B), A and B may be concurrent or incomparable,
  but never B?A).

1,1 2,1
1,2
2,2
A1 B2 C3
Lamport timestamp of 1,1 lt 2,2 but the events are
unrelated
1,3 2,3
27
Why Vector Timestamps

• Also, Lamport timestamps do not detect causality
  violations. Causality violations are caused by
  long communications delays in one channel that
  are not present in other channels or a non-FIFO
  channel.

A B C
28
Causality Violation

• Causality violation example A gets a message
  from B that was broadcast to all nodes. A
  responds by broadcasting an answer to all nodes.
  C gets As answer to B before it receives Bs
  original message.
• How can B tell that this message is out of order?

A B C
29
Causality Solution

• The solution is vector timestamps Each node
  maintains an array of counters.
• If there are N nodes, the array has N integers
  V(N). V(I) C, the local clock, if I is the
  designation of the local node.
• In general, V(X) is the latest info the node has
  on what Xs local clock is.
• Gives us the property e ? f iff ts(e) lt ts(f)

30
Vector Timestamps

• Each site has a local clock incremented at
  each event (not according to Lamport clocks) The
  vector clock timestamp is piggybacked on each
  message sent. RULES
• Local clock is incremented for a local event and
  for a send event. The message carries the vector
  time stamp.
• When a message is received, the local clock is
  incremented by one. Each other component of the
  vector is increased to the received vector
  timestamp component if the current value is less.
  That is, the maximum of the two vector components
  is the new value.

31
Vector Timestamps and Causal Violations

• C receives message (2,1,0) then (0,1,0)
• The later message causally precedes the first
  message if we define how to compare timestamps
  right

A B C
32
Vector Clock Comparison

• VC1 gt VC2 if for each component j, VC1j gt
  VC2j, and for some component k, VC1k gt VC2k
• VC1 VC2 if for each j, VC1j VC2j
• Otherwise, VC1 and VC2 are incomparable and the
  events they represent are concurrent

1 2 3 4
Clock at point 1 (2,1,0) 2 (2,2,0) 3
(2,1,1) 4 (2,1,2)
A B C
33
Vector Clock Exercise

• Assuming the only events are send and receive
• What is the vector clock at events 1-6?
• Which events are concurrent?

6
1 2 5 3 4
A B C
34
Matrix Timestamps

• Matrix timestamps can be used to give each node
  more information about the state of the other
  nodes.
• Each site keeps a 2 dimensional time table
• If Tij,k v then site i knows that site j
  is aware of all events at site k up to v
• Row x is the view of the vector clock at site x

As TT A B C A 3 2
3 B 1 2 0 C 2 2 3
A B C
35
Global State

• Matrix timestamps is one way of getting
  information about the distributed system.
  Another way is to sample the global state.
• The Global state is the combination of the states
  of all the processors and channels at some time
  which could have occurred.
• Because there is no way of recording states at
  the exact same time at every node, we will have
  to be careful how we define this.

36
Global State

• There are many reasons for wanting to sample the
  global state take a snapshot.
• deadlock detection
• finding lost token
• termination of a distributed computation
• garbage collection
• We must define what is meant by the state of a
  node or a channel.

37
Defining Global State

• There are N processes P1Pn. The state of the
  process Pi is defined by the system and
  application being used.
• Between each pair of processors, Pi and Pj, there
  is a one-way communications channel Ci,j.
  Channels are reliable and FIFO, ie, the messages
  arrive in the order sent. The contents of Ci,j
  is an ordered list of messages Li,j (m1, m2,
  m3). The state of the channel is the messages in
  the channel and their order.
• Li,j (m1, m2, ) is the channel from Pi to Pj
  and m1 (head or front) is the next message to be
  delivered.

38
Defining Global State

• It is not necessary for all processors to be
  interconnected, but each processor must have at
  least one incoming channel and one outgoing
  channel and it must be possible to reach each
  processor from any other processor (graph is
  strongly connected).

2
1
4
3
39
Defining Global State

• The Global state is the combination of the states
  of all the processors and channels.
• The state of all the channels, L, is the set of
  messages sent but not yet received.
• Defining the state was easy, getting the state is
  more difficult.
• Intuitively, we say that a consistent global
  state is a snapshot of the DS that looks to the
  processes as if it were taken at the same instant
  everywhere.

40
Defining Global State

• For a global state to be meaningful, it must be
  one that could have occurred.
• Suppose we observe processor Pi (getting state
  Si) and it has just received a message m from
  processor Pk. When we observe processor Pk to
  get Sk, it should have sent m to Pi in order for
  us to have a consistent global state. In other
  words, if we get Pks state before it sent
  message m and then get Pis state after it
  received m, we have an inconsistent global state.

Pi Pk
Pi Pk
41
Consistent Cut

• So we say that the global state must represent a
  consistent cut.
• One way of defining a consistent cut is that the
  observations resulting in the states Si should
  all occur concurrently (as defined using vector
  clocks).
• Also, a consistent cut is one where all the
  events before the cut happen-before the ones
  after the cut or are unrelated (uses
  happens-before relation).

42
Global State (1)

1. A consistent cut
2. An inconsistent cut

43
Algorithm for Distributed Snapshot

• Well known algorithm by Chandy and Lamport
• When instructed, each processor will stop other
  processing and record its state Pi, send out
  marker messages and record the sequence of
  messages arriving on each incoming channel until
  a marker comes in (this will enable us to get the
  channel state Ci,j).
• At end of algorithm, initiator or other
  coordinator collects local states and compiles
  global state.

44
Chandy Lamport Snapshot

1. Organization of a process and channels for a
   distributed snapshot

45
Chandy Lamport Snapshot

• One processor starts the snapshot by recording
  his own local state and immediately sends a
  marker message M on each of its outgoing
  channels. (This indicates the causal boundary
  between before the local state was recorded and
  after). It begins to record all the messages
  arriving on all incoming channels. When it has
  received markers from all incoming channels, it
  is done.
• When a processor who was not the initiator
  receives the marker for the first time, it
  immediately records its local state, sends out
  markers on all outgoing channels. It begins
  recording the received message sequence on all
  incoming channels other than the one it just
  received the marker on. When a marker has been
  received on each incoming channel, the processor
  is done with its part of the snapshot.

46
Chandy Lamport Snapshot

• Process Q receives a marker for the first time
  and records its local state
• Q records all incoming message
• Q receives a marker for its incoming channel and
  finishes recording the state of the incoming
  channel

47
Snapshot
M
Recorded
a
M
State S2
2
2
1
4
b
3
c

• Node 2 initiates snapshot

48
Snapshot
Recorded
M
State S2
2
2
M
1
4
b
3
c

• Node 2 initiates snapshot

49
Snapshot
Recorded
State S2
2
2
M
State S1
1
1
b
4
4
4
State S4
M
M
d
3

• Node 2 initiates snapshot

50
Snapshot
Recorded
State S2
2
2
M
State S1
1
1
M
d
4
4
4
State S4
L3,2 b
3
M
51
Snapshot
Recorded
State S2
2
2
State S1
1
1
d
4
4
4
State S4
L3,2 b
3
L1,2 empty
M
L4,2 empty
52
Snapshot
Recorded
State S2
2
2
State S1
1
1
4
4
4
State S4
M
L3,2 b
M
3
3
3
L1,2 empty
L4,2 empty
State S3
L3,1 d

53
Snapshot
Recorded
State S2
2
2
State S1
1
1
M
M
4
4
4
State S4
L3,2 b
3
3
3
L1,2 empty
L4,2 empty
State S3
L3,1 d

54
Snapshot
Recorded
State S2
2
2
2
State S1
1
1
1
4
4
4
State S4
L3,2 b
3
3
3
L1,2 empty
L4,2 empty
State S3
L3,1 d

55
Chandy Lamport Snapshot
2
2
2
1
1
1
4
4
4
3
3
3

• Uses O(E) messages where E is the number of
  edges. Time bound is dependent on the topology
  of the graph.

56
Next Termination detection