Data Structures in Graph Algorithms Robert E. Tarjan Princeton University HP Labs http://www.cs.princeton.edu/courses/ archive/spring11/cos423/

Outline • • • •

Shortest paths Minimum Spanning Trees Amortization Heaps

Shortest Paths

Directed graph with arc weights b

–10

25 c

–15 6 40 a 6 30 d 16 f e 18 2 0 11 3 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 –10 k 48 l 14 12 5 16 j –20 –27 m n –14

path weight = sum of arc weights along path Goal: find a minimum-weight path from s to t, for given pairs of vertices s, t weights: costs, travel times, lengths,… Henceforth think of weights as lengths; a minimum-weight path is shortest (but we allow negative lengths)

Path s, g, d, e, f, t Length 22 + 2 + 16 + 30 + 18 = 88 Shortest? b 25 –10 c

–15 6 40 a 6 16 30 d f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 –10 k 48 l 14 12 5 16 j –20 –27 m n –14

Versions of shortest path problem Single pair: from one s to one t Single source: from one s to each possible t Single sink: from each possible s to one t All pairs: from each possible s to each possible t Single-source problem is central: equivalent to single-sink problem (reverse arc directions) all pairs = n single-source problems single-pair algorithms at least partially solve a single-source (or single-sink) problem

Special cases Graph is undirected No negative arcs No cycles Graph is planar Graph is a “road network” …

Negative cycles A negative cycle is a cycle whose total length is negative. If there are no negative cycles and there is some path from s to t (t is reachable from s), then there is a shortest path from s to t that is simple (it contains no repeated vertices): deletion of a cycle from the path does not increase the length of the path.

If a negative cycle is reachable from s, then there are arbitrarily short paths from s to every vertex on the cycle: just repeat the cycle. If there are negative cycles, the problem of finding a shortest simple path is NP-hard. Revised goal: Find a shortest path from s to t for each of the given pairs s, t, or find a negative cycle.

Notation G = (V, A): graph with vertex set V and arc set A n = |V|, m = |A|, assume n > 1 s: source vertex for single-source or single-pair problem t: target vertex for single-sink or single-pair problem (v, w): arc from v to w c(v, w) = length of arc (v, w) c(P) = length of path P

Single–source problem: find shortest paths from s to each vertex reachable from s, or find a negative cycle reachable from s. Method: iterative improvement. For each vertex v, maintain the length d(v) of the shortest path from s to v found so far. Look for shorter paths by repeatedly examining arcs (v, w). If d(v) + c(v, w) < d(w), there is a shorter path to w: decrease d(w) to d(v) + c(v, w). Stop when no such improvement is possible.

Labeling algorithm for w ∈ V do d(w) ← ∞; d(s) ← 0; while ∃(v, w) ∈ A ∋ d(v) + c(v, w) < d(w) do label(w): d(w) ← d(v) + c(v, w) Each iteration of the while loop is a labeling step. (The Operations Research literature calls such a step a relaxation of (v, w).)

Theorem 1: If the algorithm stops, d(w) if finite is the length of a shortest path from s to w; d(w) = ∞ ↔ w is unreachable from s. The theorem implies that if there is a negative cycle reachable from s, the labeling algorithm never stops.

Theorem 2: If there are no negative cycles, the algorithm stops, and the number of labeling steps is at most the number of simple paths (no repeated vertex). The bound on the number of steps given by the proof of Theorem 2 is exponential, and indeed the labeling algorithm takes exponential time in the worst case. To make the algorithm efficient, we must choose the order of steps carefully.

Before addressing how to choose labeling steps, we extend the algorithm to find shortest paths, not just their lengths. To do this, we maintain a parent p(w) for each vertex w: p(w) is the next-to-last vertex on the shortest path to w found so far.

Labeling algorithm with parents for w ∈ V do {d(w) ← ∞; p(w) ← null}; d(s) ← 0; while ∃(v, w) ∈ E ∋ d(v) + c(v, w) < d(w) do {d(w) ← d(v) + c(v, w); p(w) ← v}

Lemma 1: Any cycle of arcs (p(x), x) is negative. Theorem 3: If there are no negative cycles, the arcs (p(v), v) form a tree T rooted s (no arc enters s, one arc enters each vertex other than s, and there are no cycles) containing exactly the vertices reached from s. When the algorithm stops, T is a shortest path tree (SPT): every path in T is shortest. Corollary 1: G contains either a shortest path tree rooted at s or a negative cycle reachable from s.

To make the shortest path algorithm efficient, we begin by doing the labeling steps vertexby-vertex instead of arc-by-arc. We partition the vertices into three sets: U (unlabeled), L (labeled), and S (scanned). Each vertex in U has not been reached from s; each vertex in L may have outgoing arcs that give shorter paths, and each vertex in S has been reached and its arcs have been checked since its distance last changed.

The scanning algorithm for w ∈ V do {d(w) ← ∞; p(w) ← null}; d(s) ← 0; U ← V – {s}; L ← {s}; S ← { }; while some v ∈ L do scan(v): {for each arc (v, w) out of v do if d(v) + c(v, w) < d(w) then {d(w) ← d(v) + c(v, w); p(w) ← v; move w to L}; move v to S}

Graph Representation For each vertex, store the set of outgoing arcs Store arc sets in lists, or in arrays, which can be subarrays of one big array Array representation saves space (no pointers), improves locality of access

Efficient scanning orders General graph: Breadth-first scanning (Bellman-Ford) L = queue, add new labeled vertices to back Non-negative arc lengths: Shortest-first scanning (Dijkstra) L = heap, distances are keys Acyclic graph: Topological scanning

Breadth-first scanning L = s:0 scan s b

–10

25 c

–15 6 40 a 6 16 30 d f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 –10 k 48 l 14 12 5 16 j –20 –27 m n –14

S = s:0 L = a:3, g:22, j:16

scan a b

–10

25 c

–15 6 40 a 6 16 30 d f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 –10 k 48 l 14 12 5 16 j –20 –27 m n –14

S = s:0, a:3 scan g, scan j L = g:21, j:16, d:9 b

–10

25 c

–15 6 40 a 6 16 30 d f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 –10 k 48 l 14 12 5 16 j –20 –27 m n –14

S = s:0, a:3, j:16 L = d:9, k:39, g:20

scan d b

–10

25 c

–15 6 40 a 6 16 30 d f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 –10 k 48 l 14 12 5 16 j –20 –27 m n –14

S = s:0, a:3, j:16, d:9 L = k:21, g:20, h:9, e:25 b

–10

scan k, scan g 25 c

–15 6 40 a 6 16 30 d f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 –10 k 48 l 14 12 5 16 j –20 –27 m n –14

S = s:0, a:3, j:16, d:9, k:21, g:20 L = h:9, e:25, i:11, m:26 b

–10

…

25 c

–15 6 40 a 6 16 30 d f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 –10 k 48 l 14 12 5 16 j –20 –27 m n –14

Running time of breadth-first scanning Define passes through the queue: pass 0 = scanning of s pass k + 1 = scanning of all vertices added to the queue during pass k After pass k, each vertex having a shortest path from s of k arcs has correct distance → all distances correct after pass n – 1 → algorithm stops after ≤n passes, or never Each pass scans each vertex at most once → O(nm) time

Negative cycle detection Lazy: count passes. If count exceeds n, stop: there must be a negative cycle. Such a cycle can be found by following parent pointers. (Exercise: prove this.) Eager: test for a cycle of parent pointers during each labeling step. How?

If d(v) + c(v, w) < d(w), follow parent pointers from v until reaching w (negative cycle found) or s This method takes Θ(n) time per labeling step, increasing the running time to Θ(n2m) Better: The predecessor pointers define a tree rooted at s. Instead of starting at v and visiting its ancestors looking for w, start at w and visit its descendants looking for v Why better?

d(v) + c(v, w) < d(w) We disassemble the subtree rooted at w as we traverse it looking for v. Each deletion of a vertex from T is preceded by a labeling of the same vertex; this labeling pays for the deletion. Furthermore each vertex other than w deleted from T will be labeled again later: its distance is not minimum. This method is subtree disassembly.

We store the vertices of T in a doubly-linked circular list in preorder (all descendants of a vertex are consecutive, each child follows its parent, not necessarily consecutively), using pointers a (after) and b (before). If d(v) + c(v, w) < d(w), we visit the descendants of w, deleting each from T. If v is visited, a cycle exists. If not, we decrease d(w) and reinsert w into T as a child of v. Optionally, for each vertex x ≠ w deleted from T, we can decrease d(x) by almost as much as the decrease in d(w).

Scanning with subtree disassembly and distance updates for w ∈ V do {d(w) ← ∞; p(w) ← null; b(w) ← null}; d(s) ← 0; U ← V – {s}; L ← {s}; S ← { }; a(s) ← s; b(s) ← s; while some v ∈ L do {for each arc (v, w) out of v do {δ ← d(w) – d(v) – c(v, w); if δ > 0 then {d(w) ← d(w) – δ; p(w) ← v; move w to L; x ← b(w); b(w) ← null; y ← a(w); while b(p(y)) = null do if y = v then stop: negative cycle else {d(y) ← d(y) – δ + ε; b(y) ← null; y ← a(y)}; a(x) ← y; b(y) ← x; a(w) ← a(v); b(a(w)) ← w; b(w) ← v; a(v) ← w}}; move v to S}

The before pointers indicate whether a vertex is in T: x in T ↔ b(x) ≠ null If the algorithm detects a negative cycle, the cycle can be found by following parent pointers from v (or from w). If w = s, there is a negative cycle. After the while loop, y is the vertex after the last (now deleted) descendant of w in preorder.

The while loop removes each proper descendant y of w from T and decreases its distance by δ – ε, where δ = d(w) – d(v) + c(v, w) and ε is the smallest representable positive number (if arc lengths are integers, ε = 1); we cannot reduce d(y) by an additional ε, because we must make sure that y is labeled again before the algorithm stops. (Why?)

Not only does subtree disassembly detect negative cycles eagerly, it speeds up the scanning algorithm in practice: it results in fewer scans of vertices whose distance from s is not yet minimum. Thus it is an important heuristic even if eager negative cycle detection is not needed. Theorem 4: Breadth-first scanning with subtree disassembly, with or without distance updates, runs in O(nm) time and stops as soon as a cycle of parent pointers exists.

Non-negative arc lengths Use greedy method: Shortest-first scanning (Dijkstra’s algorithm): Scan a labeled vertex v with d(v) minimum, breaking a tie arbitrarily.

The scanning algorithm for w ∈ V do {d(w) ← ∞; p(w) ← null}; d(s) ← 0; U ← V – {s}; L ← {s}; S ← { }; while some v ∈ L do scan(v): {for each arc (v, w) out of v do if d(v) + c(v, w) < d(w) then {d(w) ← d(v) + c(v, w); p(w) ← v; move w to L}; move v to S}

L = s:0 scan s b

1

25 c

5 6 40 a 6 30 d 16 f e 18 2 0 11 3 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 0 k 48 l 14 12 5 16 j 2 7 m n 0

S = s:0 L = a:3, j:16, g:22

scan a b

1

25 c

5 6 40 a 6 30 d 16 f e 18 2 0 11 3 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 0 k 48 l 14 12 5 16 j 2 7 m n 0

S = s:0, a:3 scan d L = d:9, j:16, g:21 b

1

25 c

5 6 40 a 6 30 d 16 f e 18 2 0 11 3 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 0 k 48 l 14 12 5 16 j 2 7 m n 0

S = s:0, a:3, d:9 scan h L = h:9, j:16, g:21, k:21, e:25 b

1

25 c

5 6 40 a 6 30 d 16 f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 0 k 48 l 14 12 5 16 j 2 7 m n 0

S = s:0, a:3, d:9, h:9 scan j L = j:16, g:21, k:21, e:25, i:29 b

1

25 c

5 6 40 a 6 30 d 16 f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 0 k 48 l 14 12 5 16 j 2 7 m n 0

S = s:0, a:3, d:9, h:9, j:16 L = g:20, k:21, e:25, i:29 b

1

scan g… 25 c

5 6 40 a 6 30 d 16 f e 18 2 3 0 11 25 14 18 22 g 12 40 h 20 i s 15 18 12 8 10 t 16 4 6 0 k 48 l 14 12 5 16 j 2 7 m n 0

Lemma 1: If arc lengths are non-negative, shortest-first scanning maintains the invariant that d(x) ≤ d(y) if x is scanned and y is labeled. Theorem 1: If arc lengths are non-negative, shortest-first scanning scans each vertex at most once.

If arc lengths can be negative, Dijkstra’s algorithm can take exponential time, even if NO cycles Exercise: Give a class of examples that show this

Implementation: L = heap (priority queue) with key of v = d(v) find v to scan: delete-min(L) label(w): after decreasing d(w), if w ∈ U then insert(w, L) else (w ∈ L) decrease-key(w, d(w), L)

Dijkstra’s algorithm for w ∈ V do {d(w) ← ∞; p(w) ← null}; d(s) ← 0; U ← V – {s}; L ← {s}; S ← { }; while ∃ v ∈ L do {v ← delete-min(L); for each arc (v, w) out of v do if d(v) + c(v, w) < d(w) then {d(w) ← d(v) + c(v, w); p(w) ← v; if w ∉ L then insert(w, L) else decrease-key(w, L)}; move v to S}

≤n inserts, ≤n delete-mins, ≤m decrease-keys L = implicit heap or pairing heap: O(mlgn) L = Fibonacci heap or rank-pairing heap: O(m + nlgn)

Minimum Spanning Trees

Problem: Given a connected graph with edge weights, find a spanning tree of minimum total edge cost: a minimum spanning tree (MST)

A connected undirected graph with edge weights 20

27 12 16

21

4 3

18

25 19

15

10

30

17 25

32

14

8

Minimum spanning tree? 20

27 12 16

21

4 3

18

22 19

15

10

30

17 25

32

14

8

Applications Clustering (single-linkage): 1909, skull classification in anthropology Network design: 1920’s, Moravian electrical network Traveling salesperson problem lower bounds: 1971, Held & Karp

When can we conclude that an edge is in the MST? When can we conclude that an edge is not in the MST? (tie-breaking by edge numbering)

Build an MST by edge coloring blue = accepted (added to MST) red = rejected (no longer a candidate) cut: a partition of the vertex set into two nonempty parts X, Y. An edge with one end in X and one in Y crosses the cut. Blue rule: Given a cut, color blue the minimumweight edge crossing it (good edge) Red rule: Given a cycle, color red its maximumweight edge (bad edge)

Generalized greedy method Begin with all edges uncolored. Repeatedly apply the blue and red rules, in any order, until all edges are colored. At all times, the blue edges form a set of trees, called the blue trees. Initially, no blue edges: each vertex forms a one-vertex blue tree. Once there is only one blue tree, containing all the vertices, it is an MST.

Classical algorithms Global greedy (Kruskal, 1956): Process the edges in increasing order by weight. To process an edge, if its ends are in different blue trees color it blue; otherwise, color it red. Single-source greedy (Jarník 1929, Kruskal 1956, Prim 1957, Dijkstra 1958): Choose a start vertex s. While the blue tree B containing s is not spanning, color blue the minimum-weight edge with exactly one end in B.

Concurrent greedy (Borůvka, 1926): For each blue tree, choose the minimum-weight edge with one end in the tree. Color all the chosen edges blue. Repeat until there is only one blue tree. All three of these algorithms are special cases of the generalized greedy algorithm.

Concurrent greedy 20

27 12 16

21

4 3

18

25 19

15

10

30

17 25

32

14

8

Concurrent greedy 20

27 12 16

21

4 3

18

25 19

15

10

30

17 25

32

14

8

Concurrent greedy 20

27 12 16

21

4 3

18

25 19

15

10

30

17 25

32

14

8

Implementations Single-source scanning: Use a heap H of vertices not in the blue tree B but connected to it by at least one edge k(v) = minimum weight of an arc connecting v to blue tree (p(v), v) = connecting arc of minimum weight E = edge set of graph

for v ∈ V do k(v) ← ∞; H ← make-heap; initialize B to contain s and no edges; for (s, v) ∈ E do {k(v) ← c(s, v); p(v) ← s; insert(v, H)}; while H ≠ { } do {v ← delete-min(H); add (p(v), v) to B; for (v, w) ∈ E do if c(v, w) < k(w) and w not in B then {k(w) ← c(v, w); p(w) ← v; if w not in H then insert(w, H) else decrease-key(w, k(v), H)}

Graph representation: for each vertex, set of incident edges. Each edge is in two such sets. n – 1 insertions, m – n + 1 decrease-keys →O(mlgn) time (implicit heap or pairing heap) O(m + nlgn) time (Fibonacci heap or rank-pairing heap) (vs. O(n2) original implementation: heap = unordered set)

Global greedy: Sort edges by weight or store in a heap with key = weight Need a data structure to keep track of the vertex sets of the blue trees. Initially each vertex is in its own tree. Operations (first cut): find(x): return the set containing x unite(A, B): unite the sets A and B

Process the edges in sorted order. process(v, w): if find(v) ≠ find(w) then {unite(find(v), find(w)), color (v, w) blue} Stop after n – 1 edges are colored blue. The running time is dominated by the sorting time, or by the heap operations if a heap is used: O(mlgn)

Concurrent greedy: after k passes, each blue tree contains at least 2k vertices → ≤lgn passes To do a pass: For each blue tree, keep track of the minimum-weight connecting edge, initially null. For each arc, find the blue trees containing its ends; update the minimumweight connecting arcs of these blue trees appropriately. (If both ends in the same tree, can delete the edge.) To find blue trees: either use disjoint set data structure or find connected components of the blue edges by graph search and number each component.

O(m) time per pass → O(mlgn) time total Cleanup: Before a pass, number the blue trees. Assign to each edge the tree numbers of its ends. Sort the edges lexicographically by edge number. Delete all edges with both ends numbered the same, and of each group of edges with the same pair of numbers, delete all but the minimum-weight arc. After cleanup, at most b2/2 edges, if b blue trees

With cleanup, after pass k, b ≤ n/2k → total time for concurrent greedy with cleanup is O(min{n2, mlgn}) Is O(nlgn + m) fastest? (Is sorting inherent in MST-finding?)

Concurrent greedy with cleanup For certain families of graphs, concurrent greedy with cleanup runs in O(n) time Requirements: (i) All graphs in the family are sparse: m = O(n) (ii) The family is closed under edge contraction: combine both ends of the edge into a single vertex, with an edge to any vertex that was adjacent to either end

If m ≤ cn, concurrent greedy with cleanup takes O(c(n + n/2 + n/ 4 +…)) = O(n) time Trees: m < n, closed under contraction, no cleanups needed Planar graphs: m < 3n, closed under contraction For concurrent greedy (with or without cleanups) to run in O(m) time on a arbitrary graph, we need a way to “thin” a graph: using red rule, color red all but O(n) edges, in O(m) time.

Faster algorithms for general graphs O(mlglgn) Yao 1975, packets Run concurrent greedy algorithm, but with only m/lgn edges. To do this, group edges incident to each vertex into packets, each of size lgn (with at most one small packet per vertex). Give the main algorithm only the minimumweight edge in each packet. Time to find packet minima is O(mlglgn).

O(mlg*n) Fredman & Tarjan 1984, F-heaps Store the set of edges incident to each blue tree in an F-heap (or rank-pairing heap). Run single-source greedy until blue tree is big enough; then choose an unconnected source and run single-source from it. Repeat until all blue trees are big enough. Clean up. Repeat. Algorithm is a hybrid of single-source and concurrent greedy with cleanup. Each round takes O(m) time. If blue trees have size at least k before a round, they have size at least 2k after → lg*n rounds.

O(mlglg*n) Gabow et al. 1986, F-heaps + packets O(m) Karger et al. 1995, thinning + random sampling O(mα(n)) Chazelle 1998, soft heaps + complicated hybrid algorithm O(minimum) Pettie & Ramachandran 2002, Chazelle’s algorithm with fixed-depth recursion + brute force for small subproblems

The power of random sampling Concurrent greedy + thinning How to thin?

A related question: MST verification Given a spanning tree T, is it an MST? Yes, if and only if every non-tree edge (v, w) has maximum weight on the cycle formed with the path in T joining v and w Proof: Red rule Use the same idea to thin: given any forest (set of vertex-disjoint trees), can color red any non-tree edge whose ends are in the same tree and whose weight is maximum on the cycle formed with tree edges

Thinning using a forest 20

27 12 16

21

4 3

18

22 19

15

10

30

17 25

32

14

8

Thinning using a forest 20

27 12 16

21

4 3

18

22 19

15

10

30

17 25

32

14

8

How to find maxima on tree paths? For now, assume O(m) How to find a good forest? Best is an MST, but too expensive to compute Good enough: an MSF (minimum spanning forest) of a random sample of the edges. (The sample subgraph may not be connected)

Randomized minimum spanning tree algorithm Concurrent greedy with occasional thinning Let b = #blue trees, initially n, e = #uncolored edges, initially m c = a constant to be chosen later while b > 1 do if e < cb then one pass of concurrent greedy else thinning step

Thinning step Sample the uncolored edges by adding each edge with probability ½, independently Find an MSF of the sample by applying the MST algorithm recursively to each connected component of the sample Color red all sampled edges not in the MSF and all non-sampled edges maximum on a cycle with MSF edges After thinning, expected #uncolored edges ≤ 2b

Expected running time R(e) ≤ O(e) + R(e – b/2) if sparse ≤ O(e) + R(e/2) + R(2b) if dense Sparse: e < cb → b/2 > e/(2c) → e – b/2 < e(1 – 1/(2c)) Dense: e ≥ cb → 2b ≤ 2e/c → e/2 + 2b ≤ e(1/2 + 4e/c) c = 5 →R(e) ≤ O(e) + R(9e/10) = O(e)

Amortized Efficiency

Example: Counting in binary

Number Binary 0 0 1 1 2 10 3 11 4 100 5 101 6 110 7 111 8 1000 9 1001

Total cost 0 1 3 4 7 8 10 11 15 16

Cost to add 1 1 2 1 3 1 2 1 4 1

Total cost to count to n Cost to add 1 = number of trailing 1’s + 1 = decrease in number of 1’s + 2 Worst-case cost to add 1: lg(n + 1) + 1 Total cost to count to n: nlgn ?

Amortize: to liquidate a debt by installment payments. From Medieval Latin: to reduce to the point of death. In analysis of algorithms: to pay for the total cost of a sequence of operations by charging each operation an equal (or appropriate) amount. (A little stretch, but there you have it.)

Amortized Efficiency Coin-operated computer bit flip costs $1 $2 per addition suffices: unspent $ = #1’s ≥ 0 total cost = 2n - #1’s ≤ 2n

Amortization (banker) Assign $ to each operation (amortized cost) Keep track of savings (or borrowings) in state of data structure If no debt after all operations, total cost ≤ sum of amortized costs

Amortization (physicist) Switch perspective: start with savings, derive cost per operation Assign “potential” Φ to each state of data structure Define “amortized cost” of an operation to be actual cost plus net change in potential: ai = ti + Φi – Φi – 1 Thus ti = ai + Φi – 1 – Φ i

total actual cost = total amortized cost + initial Φ – final Φ ≤ total amortized cost if initial Φ = 0, final Φ ≥ 0 (no net borrowing) Binary counting: Φ = number of 1’s Φ0 = 0, Φn ≥ 0 Amortized cost to add one = 2 → total actual cost ≤ 2n

Frequency of multiple carries Observation: a cost of k + 1 or more occurs at most n/2k times (out of n) Proof of observation via a potential function: Fix k. Let Φ = n mod 2k. Each add increases Φ by one, unless cost is k + 1 or more. (We call the add expensive.) In this case n mod 2k = 2k – 1, so Φ decreases by 2k – 1. This can happen at most n/2k times out of n: Φ = n - e2k ≥ 0, where e = #expensive adds.

Heaps

Heap(priority queue): contains a set of items x, each with a key k(x) from a totally ordered universe, and associated information. We assume no ties in keys. Basic Operations: make-heap: Return a new, empty heap. insert(x, H): Insert x and its info into heap H. delete-min(H): Delete the item of min key from H.

Additional Operations: find-min(H): Return the item of minimum key in H. meld(H1, H2): Combine item-disjoint heaps H1 and H2 into one heap, and return it. decrease-key(x, k, H): Replace the key of item x in heap H by k, which is smaller than the current key of x. delete(x, H): Delete item x from heap H. Assumption: Heaps are item-disjoint.

A heap is like a dictionary but no access by key; can only retrieve the item of min key: decrease-key(x, k, H) and delete(x, H) are given a pointer to the location of x in heap H Applications: Priority-based scheduling and allocation Discrete event simulation Network optimization: Shortest paths, Minimum spanning trees

Lower bound from sorting Can sort n numbers by doing n inserts followed by n delete-min’s. Since sorting by binary comparisons takes Ω(nlgn) comparisons, the amortized time for either insert or delete-min must be Ω(lgn). One can modify any heap implementation to reduce the amortized time for insert to O(1) → delete-min takes Ω(lgn) amortized time.

Our goal O(lgn) amortized time for delete-min and delete O(1) amortized time for all other operations

Representation: Heap-ordered tree Heap order: k(p(x)) ≤ k(x) for all nodes x. Defined for rooted trees, not just binary trees Heap order → item in root has min key → find-min takes O(1) time What tree structure? How to implement heap operations?

Three heap implementations Implicit heap: Very simple, fast, small space. O(lgn) worst-case time per operation except for meld. Pairing heap: O(lgn) amortized time per operation including meld, simple, selfadjusting. Rank-pairing heap: Achieves our goal.

Heap-ordered tree: internal representation Store items in nodes of a rooted tree, in heap order. Find-min: return item in root. Insert: replace any null child by a new leaf containing the new item x. To restore heap order, sift up: while x is not in the root and x has key less than that in parent, swap x with item in parent.

Delete-min or delete: Delete item. To restore heap order, sift down: while empty node is not a leaf, fill with item of smallest key in children. Either delete empty leaf, or fill with item from another leaf, sift moved item up, and delete empty leaf. (Allows deletion of an arbitrary leaf, so tree shape can be controlled) Decrease-key: sift up. Choice of leaf to add or delete is arbitrary: add level-by-level, delete last-in, first-out.

A binary heap Numbers in nodes are keys. Numbers next to nodes are order of addition. 5 1

25 2 21

3 40 5

4 24

30

8

9

10 16 6

12 7

insert 7 5 1

25

10 3

2

40

21

5

4 24

30

7

8

9

10

16 6

12 7

delete-min: remove item in root, sift empty node down 5 1

7

10 3

2

25

21

5

4 24

30

40

8

9

10

16 6

12 7

End of sift-down Swap item in last leaf into empty leaf; sift up.

7 1

21

3

2

25

24

5

4

8

10

30

40

9

10

16 6

12 7

7 1

21 2 24

3 25 5

4 40

30

8

9

10 16 6

12 7

Implicit binary heap Binary tree, nodes numbered in addition order root = 1 children of v = 2v, 2v + 1 p(v) = v/2 → no pointers needed! Can store in array insert: add node n + 1 delete: delete node n depth = lgn

5 1

7

10 3

2

25

21

16

5

4 24

30

40

8

9

10

12

6

7

1

2

3

4

5

6

7

8

9

10

5

7

10

21

25

16

21

24

30

40

Each operation except meld takes O(lgn) time: insert takes ≤lgn comparisons (likely O(1)) delete takes ≤2lgn comparisons (likely lgn + O(1)) Can reduce comparisons (but not data movement) to lglgn worst-case for insert, lgn + lglgn for delete Instead of binary, can make tree d-ary. Some evidence suggests 4-ary is best in practice.

Heap-ordered tree: external representation Store items in external nodes of a binary tree, in any order. To fill internal nodes, run a tournament: bottomup, fill each internal node with item of smaller key in children. Find-min: return root. Primitive operation link: combine two trees by creating a new root with old roots as children, filling with item of smaller key in old roots.

A link takes one comparison and O(1) time. We will build all operations out of links and cuts. 6 8

6

First: alternative ways to represent tournaments

Full representation 5 5

7 7

21 21

24 28

24

7

12

16

5

18

7

18

5

30

5

27

10

Half-full representation Store each item once, at highest node

5

7 16

21 18

24 28

12

27

30

10

Left-full representation Swap siblings to make left children full

5

7 16

21 18

24 28

30

27

12

10

Heap-ordered representation Heap-ordered tree, children contain items that lost links, most recent first

5

7 21

18

24

30

28

16 12

27

10

Half-ordered representation Binary tree: first child, next sibling representation of heap-ordered tree 7 21

18

24

30

28

16 12

5

27

10

Half-ordered representation Half order: all items in left subtree larger than item in node Half tree: root has 21 null right child 24 28

5 7 16 27

18 30

12

10

Linking half trees One comparison, O(1) time 10

+

8

8

10 A

B A

B

Half-tree representation: Left and right child pointers Parent pointers if decrease-key, delete allowed Heap operations: find-min: return item in root make-heap: return a new, empty half tree insert: create a new, one-node half tree, link with existing half tree meld: link two half trees

delete-min: Delete root. Cut edges along right path down from new root. Roots of the resulting half trees are the losers to the old root. Must link these half trees. How?

Delete-min

5 7 21 24

28

16 27

18 30

12

28

10

Link half trees in pairs, top-down. Then take bottom half tree and link with each new half tree, bottom-up Pairing heap

Delete-min

5 7 21 24

28

16 27

18 30

12

28

10

After top-down pairing links

7

16 10

21 24 28

27

18 30

12

28

After bottom-up links 7 10 16

27

21 28 24 28

18 30

12

Remaining heap operations: decrease key of x in heap H: Remove x and its left subtree (becomes a new half tree). Replace x by its right child. Decrease k(x). Link the old half tree with the new half tree rooted at x. delete x in heap H: Decrease key of x to – ∞; delete-min.

decrease key 18 to 11 Remove half tree rooted at 18 Replace by right child of 18

5 7

21 24 28

16 27

18 30

12

28

10

Link old and new half trees

11

5

30

7 21

24 28

16 27

12 28

10

5 11 7

30 21 24 28

16 27

12 28

10

Analysis of pairing heaps Almost identical to amortized analysis of splay tree (a type of self-adjusting search tree) Amortized time per operation is O(lgn) Fast in practice but NOT O(1) for decrease-key O(lgn) is not tight for decrease-key; tight bound is unknown (Ω(lglgn))

Inefficiency in pairing heaps: Links during inserts, melds, decrease-keys can combine two trees of very different sizes, increasing the potential by Θ(lgn). To avoid such bad links, we use ranks: Each node has a non-negative integer rank Missing nodes have rank –1 Rank of root = 1 + rank of left child rank of tree = rank of root

Store ranks with nodes. Link only roots of equal rank. Rank of winning root increases by one: r 10

+

r

r+1

8

8 r 10

A

B A

B

A heap is a set of half trees, not just one half tree (can’t link half trees of different ranks) Representation of a heap: a circular singly-linked list of roots of half trees, with the root of minimum key (the min-root) first Circular linking → catenation takes O(1) time Node ranks depend on how operations are done

find-min: return item in min-root make-heap: return a new, empty list insert: create new one-node half tree of rank 0, insert in list, update min-root 5

+

4

9

6

4

8

5

9

6

8

meld: catenate lists, update min-root 4

9

6

2

9

6

+

2

4

8

8

5

5

7

7

delete-min: Delete min-root. Cut edges along right path down from new root to give new half-trees. Link roots of equal rank until no two roots have equal rank. Form list of remaining half trees. To do links, use array of buckets, one per rank. Add each tree to its bucket, link with tree in bucket if there is one, add tree formed by link to next bucket.

delete-min: numbers are keys 0

6

8

9 7 2 4 1

5

3

delete-min: numbers are ranks 5

3

4 3 2 1 0

3

2

0

1

2

3

4

5

0 1 2 3 4 5 6

3

3

2

0

1

4

6

Each link takes O(1) time, forming final tree takes O(max rank) time → delete-min takes O(max rank + #links) time Amortize links: let Φ = # half trees One unit of time per link , ΔΦ = –1 → link takes 0 amortized time

make heap, find-min, meld do not change Φ, insert increases Φ by 1 → each takes O(1) amortized time delete-min takes O(#new half trees + max rank) amortized time How many new half trees? What is max rank?

A perfect half tree of rank 4 4 3 2

2

1 0

1 0

0

1 0

0

1 0

0

0

What about decrease-key? Add parent pointers to data structure decrease key(x, k, H): Remove x and its left subtree (becomes a new half tree). Replace x by its right child. Change key of x to k. Add x to the list of half tree roots. Update the min-root. But: rank differences are no longer all 1, half trees are no longer perfect, max rank can grow beyond O(lgn)

decrease-key(x, k, H): numbers are ranks

y

z 3 2

x

1 0

1 0

0

2

1 0

0

1 0

0

0

y

x

z

1 0

3 0

2

1 0

1 0

0

1 0

0

0

Solution: maintain limit rank differences. Allowed rank differences are 1,1 (perfect); 1,2; and 0,j for any j > 1 (2,1 = 1,2; j,0 = 0,j) At end of decrease-key of x, set r(x) = r(left(x)) + 1; start from old parent y of x and walk up path toward root, decreasing ranks to restore the invariant:

while p(y) ≠ null and r(y) too big do {let y be i,j with i ≤ j; if j – i > 1 then r(y) ← r(y) – i else r(y) ← r(y) – i + 1; y ← p(y)}; if p(y) = null then r(y) = r(left(y)) + 1 In successive steps, r(y) decreases by a nonincreasing amount

decrease-key(x, k, H)

9 8 7

7

6

5 6

y

3 6

2 6

x 0

7

9

x 8

6

7

7

6

5 6

y

3

2 0

7

9

x 8

6

6

7

4

5 2

y

3

2 0

delete(x, H): decrease-key(x, –∞, H); delete-min(H) delete-min: Delete min-root. Cut edges along right path down from new root to give new half-trees. Set rank of each new root = 1 + rank of left child. Link roots of equal rank until no two roots have equal rank. Form list of remaining half trees.

Delete-min: numbers are keys 0

6

8

9 7 2 4 1

5

3

Delete-min: numbers are ranks 5

3

4 4 2 1 1

3

2

4

4

2

1

1

5

0 1 2 3 4 5 6

3

3

2

2

3

4

6

Each link takes O(1) time, forming final tree takes O(max rank) time → delete-min takes O(max rank + #links) time

Data structure: (type-2) rank-pairing heap Invariant on rank differences → max rank is O(lgn) Amortize rank decrease steps as well as links → amortized time of decrease-key is O(1), other amortized time bounds the same

For any non-root x, s(x) + 1 ≥ Fr(x) + 3 For any root x, s(x) ≥ Fr(x) + 2 Proof for non-roots by induction on h(x): s(null) + 1 = 0 + 1 = F2 s(leaf) + 1 = 1 + 1 ≥ F3 h(x) > 0: s(x) + 1 = s(left(x)) + 1 + s(right(x)) + 1 ≥ Fr(x) + 2 + Fr(x) + 1 = Fr(x) + 3 if x is 1,1 or 1,2; immediate if x is 0,j n ≥ Fr(x) + 2 ≥ φr(x) → r(x) ≤ lgφn < 1.44043lgn

How to amortize rank decrease steps and links? Such a step must decrease Φ Φ(x) = i + j + c if x is i,j (c any constant) → when r(x) decreases by δ, Φ(x) decreases by 2δ, Φ(p(x)) increases by δ, net change is –δ A link converts a root to a 1,1 → Φ(root) > Φ(x) if x is 1,1

Choosing c = –2, Φ(root) = 1 gives Φ(x) = 0 if x 1,1 Almost works! Problem: 0,2 nodes also have Φ = 0, but such nodes need Φ = 1 when they become roots. Need extra potential. Only one 1,1 node can decrease in rank during a decrease-key: choose c = –1, give 1,1 nodes potential 0 instead of 1.

Φ(x) = r(x) – r(left(x)) if root = 0 if 1,1 = 2r(x) – r(left(x)) – r(right(x)) – 1 otherwise make-heap (ΔΦ = 0), find-min (ΔΦ = 0), insert (ΔΦ = 1), meld (ΔΦ = 0) take O(1) amortized time

delete-min: Each new root needs one unit of Φ, which it has unless it is a 1,1 node. Only lgφn 1,1 nodes on a path (at most one per rank) → creation of new half trees increases Φ by at most lgφn Each link takes 0 amortized time → delete-min takes O(lgn) amortized time

decrease-key: Creating new subtree takes O(1) amortized time (ΔΦ ≤ 2). Each rank decrease step reduces Φ unless it is of a 1,1 node. Only one 1,1 node can decrease in rank: can only decrease by 1, previous decrease must be by at least 2 → decrease-key takes O(1) amortized time

Ongoing Work on Heaps • Rank-pairing heaps with worst-case bounds (and one tree per heap): with Gerth Brodal & George Lagogiannis • Simplified soft heaps (heaps that can make errors): with Haim Kaplan & Uri Zwick

“Exercises” • Linear-time deterministic MST algorithm? • Linear-time optimum branching algorithm? (directed MST) • Tighter analysis of pairing heaps?