Problems with Network Coding in Overlay Networks Tomislav Nad Advisor: Arvind Krishnamurthy Network coding is recently proposed mechanism [6] in which intermediate network nodes are not only limited to routing the data, but can also encode or decode it. Network coding is interesting because if network coding is allowed, multicast can achieve the theoretical throughput upper bound, which is generally not possible without network coding. In this paper, we discuss some problems that arise when network coding is used in overlay networks, and identify some possible solutions. We primarily focus on performance of coding the data, and on throughput benefits in realistic model of overlay networks. I. Introduction Until recently, content distribution on the Internet relied on placing dedicated servers at the edge of the Internet. However, during recent years, much research has gone into implementing large scale data distribution systems using end-system cooperation (peer-to-peer systems). Main advantages of such fully distributed systems over centralized systems are cost-efficiency, self-scalability, and ability to efficiently multicast data. All users that use these systems share resources of their commodity PCs (storage, CPU time, bandwidth), so no expensive dedicated servers are necessary. Peer-to-peer systems are also inherently self-scalable, in that the bandwidth capacity of the system increases as more nodes arrive: as each node arrives to the system and requests service, it also provides service to the system and thus increases total system bandwidth. This causes virtually limitless scalability of system for no additional cost. Thus, such systems are especially useful for efficiently distributing large files to a large number of users. Perhaps the most interesting benefit of peer-to-peer systems is possibility of efficient multicast. Although there is a theoretical possibility to do multicast at network layer, IP multicast service at the network layer in backbone networks is not widely available. Due to this, recent research has examined the feasibility of implementing multicast in the application-layer, by using overlay networks (e.g., [1], [2], [3], [4], [5]). The general approach is to have end-nodes self organize into an overlay network, on top of densely connected IP-layer network. The data can then be multicasted along the edges of such overlay network by using unicast transport services. Unlike IP multicast, application-layer multicast does not require multicast support in the network layer. However, multicast on overlay networks does not perform as well as IP multicast because it is impossible to completely prevent multiple overlay edges from traversing the same physical link, causing redundant traffic on the physical link. Network coding is a recently proposed mechanism for improving throughput when multicasting. Basing principle of network coding is to encode at the interior nodes

of the network, and it was shown [6] that a sender can multicast data to a set of receivers at the multicast capacity of the network (theoretical upper bound) provided one allows network coding. Most previous work on network coding is largely theoretical and assumes detailed knowledge of network topology. In this paper we focus on practical issues of network coding. The remainder of this paper is organized as follows. Section II presents network coding in more detail. Section III gives overview of previous research on network coding, especially research focusing on practical issues. Section IV discusses some problems which arise when using network coding in overlay networks, and Section V concludes the paper. II. Network Coding Network coding is a recently proposed mechanism based on a simple idea first stated by Ahlswede et al. [6]. The basic principle behind network coding is to allow and encourage encoding of data at intermediate nodes. It was shown [6] that when network coding is allowed, the sender can multicast data at the multicast capacity of the network. This is generally not the case when network coding is not allowed. However, unlike any traditional approaches in which intermediate nodes are simple routers, network coding requires that intermediate nodes have the capacity to encode the data.

a)

b) Figure 1. Network coding example

To illustrate how network coding improves multicast throughput we consider the classical network coding example. In Figure 1 consider the multicast of two data bits, b1 and b2, from the source S to receivers t1 and t2. Figure 1(a) shows the capacity of links in the network. Without network coding, it is impossible to send both bits to t1 and t2. However, network coding is allowed, both bits can be sent to both t1 and t2. Figure 1(b) shows such a scheme, where "+" denotes XORing of two bits. In this scheme, node 3 XORs bits b1 and b2 and sends it towards receivers. From b1 and b1+b2, t1 can recover both b1 and b2. Similarly, t2 can also recover both data bits. In example network from Figure 1, we can achieve multicast throughput of 2 when using network coding. In this network, such throughput is unachievable when using traditional approaches. For example, multicast throughput of 1 can be achieved using the usual all-widest-paths multicast tree, and multicast throughput of 1.5 can be achieved by packing Steiner trees. In fact, it was shown [6] that if we regard multicast throughput as information flow, then max-flow of point-to-multipoint communication is: Fmcast = min F (t ) , t∈M

where F(t) is a max-flow from source s to t, and M is the set of nodes which are multicast receivers. Furthermore, multicast max-flow can not be achieved when using IP multicast, but it is achievable when using network coding [6]. Going back to example in Figure 1, we can see that F(t1)=F(t2)=2, and thus maximum multicast throughput is 2. We have already demonstrated how this throughput can be achieved when using network coding, it can be verified that without network coding, the achievable throughput is only 1.5. III. Related Work Network coding was introduced by Ahlswede et al. [6], who showed that if network coding is used, a source can multicast information at a rate of the smallest minimum cut between the source and any receiver, which is not always possible with routing alone. Li et al. [7] showed that it is sufficient for the encoding functions at the interior nodes to be linear and that finite symbol size is sufficient. All linear coding operations are defined as linear combinations over finite fields with fixed element lengths. Koetter and Medard [8] presented an algebraic framework for linear coding and showed how to find the coefficients of the linear coding and decoding functions. They also showed that the field of size h|T| is sufficient, where h is multicast capacity and |T| is the number of receivers. Jaggi, Sanders, et al. [9, 10, 11] considered acyclic networks and showed how to find encoding and decoding coefficients in polynomial time. They also showed that the linear encoding functions can be designed randomly.

Most previous work on network coding is largely based on theoretical calculations and assumes detailed centralized knowledge of network topology. However in real networks, the problem becomes much more complicated. In real networks, information travels asynchronously in packets, and packets are subject to random delays and losses on every edge. Edges often have unknown capacity, and there are frequent link failures. Even if network topology was stable, it would be difficult to obtain knowledge on topology, or to arrange broadcast of that knowledge to every node. Furthermore, most

edges in real networks are bi-directional which results in numerous cycles, which generally present a problem for most proposed algorithms. Another limitation of past research is that in most cases it assumes that network coding is done at network-layer, where each node corresponds to a router. However, most routers currently do not even support IP multicast, and it is unlikely that they will support any kind of network coding in the near future. Therefore, we believe that if network coding is implemented, it would have to be on overlay networks. Gkantsidis and Rodriguez [12] explored application of network coding to unstructured overlay networks. They proposed an unstructured peer-to-peer network which uses network coding to improve content propagation through the system, and claim this system might offer significant improvements over traditional peer-to-peer systems such as BitTorrent. However, it remains unclear whether benefit of network coding in this scenario is the increase of achievable throughput, or whether network coding just allows us to easier reach the throughput which is achievable with just routing. IV. Problems with Network Coding in Overlay Networks In this section, we address some of the problems which can arise when using network coding in overlay network setting. Some of the problems such as choosing encoding and decoding coefficients, dealing with node and link failures and additions, and buffering were already discussed in [12] and [13]. We primarily focus on speed of encoding and decoding and feasibility of linear coding over finite fields, and on throughput benefits of network coding in overlay networks. A. Coding Speed Speed at which the data can be encoded and decoded is extremely important, but it was largely ignored in previous network coding research. Intrinsic property of network coding is that the data will be encoded at intermediate nodes, which means that total encoding overhead will be constantly increasing as the data travels through the network. Because of this, a system which uses network coding must be able to very efficiently encode the data at every node. Unfortunately, this problem is even harder to solve than is immediately obvious. In most general case in network coding, the data block is encoded as a linear combination of data blocks over a finite field, often GF(2^8) or GF(2^16) [12]. "+" operation in Galois field is usually defined as a simple XOR, but "X" operation is more complicated and is usually defined as multiplication modulo irreducible polynomial. File Size 100 kB 1 MB 5 MB

100 B 10.7 s 1700 s 8590 s

Block Size 1 kB 10 kB 1.7 s 0.2 s 170 s 16 s 870 s 170 s

Figure 2. Experimental encoding times Figure 2 shows encoding times for some file and block sizes. Each file is divided into blocks (symbols). Each encoded block is a linear combination (in GF(2^8)) of all data blocks. Our encoding function performs operations on finite fields by using Binary

finite field library 0.001. Similar encoding times were reported Byers et al. [14]. As can be seen, encoding times reach hundreds of seconds even for relatively small file sizes. Since peer-to-peer systems are usually used to distribute large amounts of data, usually in order of hundred of MBs, a simple encoding scheme such as linearly combining all data blocks in finite fields is clearly infeasible. Although increasing block size can greatly decrease encoding time, there are several problems with this approach. Firstly, since TCP packet size is ~1.4 kB, we believe that block size of above 1.4 kB could cause many problems due to splitting block into multiple packets. Main problem we identify is that a node would have to receive all packets of a block before it can use that block for encoding or decoding, which causes further overhead. Even more serious problem with large block size is that it would decrease total number of blocks. Since each block is basically a symbol in information theoretic sense, it is possible that there would not be sufficient number of symbols to achieve optimal multicast throughput (see [7], [8]), and there would be no gain in using network coding. We identify two mechanisms which could decrease the encoding time: a) Encoding by computing linear combination of constant degree Suppose that each block (symbol) is a linear combination of d random blocks (symbols). We call d the degree of linear combination, and keep it constant. It is easy to see that in this case, the encoding time stays constant as the number of data blocks increases, so this method could be suitable for multicasting a large amount of data. However, there are also drawbacks. At this time, it is not clear what degree is sufficient to achieve network multicast capacity. It is also not clear whether there would be some structure in the resulting coefficient matrix which could be used to improve performance of decoding. If there is no such structure, decoding time would still increase as the number of data blocks increases. b) Encoding by just using XORs This method is based on Digital Fountain [14] approach which uses XOR-based erasure codes to transfer bulk data. In the digital fountain approach, servers construct erasure codes by XOR-ing data blocks, and nodes can efficiently reconstruct the original content of size n from roughly a subset of any n symbols from a large universe of encoded symbols. However, the sets of symbols acquired by nodes are likely to overlap substantially, so care must be taken to enable them to collaborate effectively. We propose a similar approach, but in which intermediate nodes would also be allowed to code the data. The advantage if this approach over doing linear combinations in finite fields is much faster speed of both encoding and decoding. Since encoding is just XOR-ing of at most d blocks (where d is the degree of encoded block), it is extremely fast. Even more interesting property of this approach is that decoding can be done incrementally, as the receiver receives new encoded data blocks. This is a sharp contrast to traditional approaches to network coding where all the encoded data blocks must be received before decoding can be done. We identify two main problems with this approach. Similarly to problem we identified with encoding approach described in previous section, it is not clear what

degree of encoded blocks is sufficient to achieve network multicast capacity; moreover, it is not clear whether multicast capacity can be achieved with just XORs (which can be thought of as an addition in GF(2^8)). The second problem is of practical nature and involves selecting the data blocks which will be combined to generate the encoded block. If suboptimal choice is made, it is possible that the encoded block will contain only the information which was already sent, resulting in unnecessary overhead. This problem is also present in the original Digital Fountain, and it can be expected that it will be much harder to solve when there are multiple nodes which encode the data. In this discussion we only focused on encoding performance and ignored the decoding. In case when encoding is done by computing linear combination in finite field, decoding largely involves inverting the matrix of all the encoding coefficients, and applying the inverted matrix to the encoded blocks. Although this can be a costly operation, we primarily focused on encoding performance because encoding occurs at all nodes and the overhead accumulates as data travels through the network, while decoding is done only once. Even so, any encoding scheme which allows for fast or incremental decoding would clearly be advantageous. B. Throughput Benefits in Overlay Networks Throughput benefits of using network coding in physical networks have been proven [6]; however, situation is not clear when network coding is used in overlay networks. Overlay links usually span multiple physical links, and multiple overlay links often span the same physical link, thus sharing the bandwidth. Furthermore, the characteristics and topology of underlying physical links are usually not available to overlay nodes, resulting in difficulty in choosing coding coefficients.

Although it is possible to construct a theoretical example of overlay network in which use of network coding would result in improved throughput, we believe it would be impossible to do this in a realistic model of overlay networks. Realistically, the nodes in overlay networks are end-systems which are placed at the edge of Internet, and are connected to each other through various Autonomous Systems. End-systems have no control over physical links, and usually have little or no knowledge of their characteristics. This is best modeled by a 'cloud' model, in which there is a central cloud which represents Internet and its various ASs, and each end-system is connected to this cloud with a physical link. This physical link is a link from end-system to ISP, and its characteristics are generally known, but topology or characteristics of physical links in the central cloud are not known to end-systems. Therefore, the central cloud is just a black box which performs routing, and all end-system (overlay nodes) are connected to this black box, forming a star topology. We postulate that in this model, there is no throughput gain by using network coding. Our argument is based on the fact that network coding seems to be beneficial (throughputwise) only in the case when correlated data must be sent through bottleneck link and then multicasted to multiple receivers. If different receivers already have disjunctive portions of the original data, then network coding can be used the compute the linear combination of the data which is then sent through the bottleneck link and multicasted to all the receivers. Since each receiver already has a portion of data, it is able recover the entire

data from the newly received data. In this case, throughput benefit is achieved by computing the linear combination of the correlated data, thus compressing it. However, since our model of overlay networks has star topology, scenario with bottleneck link will never occur, and thus there would be no benefits of using network coding. If we go back to Figure 1, we can see that the bottleneck link is between node 3 and node 4. Thus, node 3 sends encoded data to the node 4, which then multicasts it to t1 and t2. Both t1 and t2 can then recover the original data. However, if we imagine this example in cloud model, nodes would not have direct link to each other, instead, all links would go through the central cloud. We can imagine that source S would have output capacity of 2, nodes 1, 2 and 4 would have input capacity of 1 and output capacity of 2, node 3 would have input capacity of 2 and output capacity of 1, and receivers t1 and t2 would have input capacity of 2. In this case, multicast throughput of 2 can easily be achieved without network coding: source S sends b1 to node 1 and b2 to node 2, nodes 1 and 2 then send the data to t1 and t2.

V. Conclusion In this paper, we described some of the problems which arise when using network coding in overlay network settings. Primary problems we discuss are encoding speed and throughput benefits in overlay setting. Fast encoding is extremely important in network coding because encoding overhead accumulates as data travels through the network. Unfortunately, most encoding mechanisms that the past theoretical research assumes have unsatisfactory performance. We identify two coding mechanisms which significantly improve encoding performance; however, both mechanisms have drawbacks which should be researched further.

We also discuss throughput benefits in overlay networks and claim that network coding does not offer any throughput benefits in realistic model of overlay networks. We are currently working on a formal proof of this. Although the primary benefit of network coding was thought to be the increase in achievable multicast throughput, other benefits were recently identified: better heuristics for content propagation [12], easier load balancing [15], faster computation of the strategies to achieve the optimal throughput, even if this throughput is achievable without network coding [16]. Therefore, even if network coding does not cause the increase of multicast throughput in overlay networks, it is likely that other benefits are present. Further research should focus on whether these benefits can be used by practical systems in realistic settings.

References:

[1] Y. Chu, S. G. Rao, S. Seshan, and H. Zhang, "A Case for End System Multicast," IEEE Journal on Selected Areas in Communications, pp. 1456-1471, October 2002. [2] M. Castro, P. Druschel, A.-M. Kermarrec, and A. Rowstron, "Scribe: A Large-Scale and Decentralized Application-Level Multicast Infrastructure," IEEE Journal on Selected Areas in Communications, pp. 1489-1499, October 2002. [3] S. Banerjee, B. Bhattacharjee, and C. Kommareddy, "Scalable Application Layer Multicast," in Proc. Of ACM SIGCOMM, August 2002. [4] J. Liebeherr, M. Nahas, and W. Si, "Application-Layer Multicasting With Delaunay Triangulation Overlays," IEEE Journal on Selected Areas in Communications, pp. 1472-1488, October 2002. [5] Y. Zhu, B. Li, and J. Guo, "Multicast with Network Coding in Application-Layer Overlay Networks," IEEE Journal on Selected Areas in Communications, Vol. 22(1), January 2004. [6] R. Ahlswede, N. Cai, S. R. Li, and R. W. Yeung, "Network Information Flow," IEEE Transactions of Information Theory, July 2000. [7] S.-Y. R. Li, R. W. Yeung, and N. Cai, "Linear Network Coding," IEEE Transactions on Information Theory, IT-49(2):371-381, February 2003. [8] R. Koetter and M. Medard, "An algebraic approach to network coding," IEEE/ACM Transactions on Networking, to appear. [9] S. Jaggi, P. A. Chou, and K. Jain, "Low complexity optimal algebraic multicast codes," Proc. Int'l Symp. Information Theory, June 2003. [10] P. Sander, S. Egner, and L. Tolhuizen, "Polynomial time algorithms for network information flow," Symposium on Parallel Algorithms and Architectures (SPAA), pp. 286-294, June 2003. [11] S. Jaggi, P. Sanders, P. A. Chou, M. Effros, S. Egner, K. Jain, and L. Tolhuizen, "Polynomial time algorithms for network code construction," IEEE Transactions on Information Theory, submitted. [12] C. Gkantsidis and P. R. Rodriguez, "Network Coding for Large Scale Content Distribution," Microsoft Research, 2004. [13] P. A. Chou, Y. Wu, and K. Jain, "Practical network coding," Allerton Conference on Communication, Control, and Computing, October 2003. [14] J. W. Byers, M. Luby, M. Mitzenmacher, and A. Rege, "A Digital Fountain Approach to Reliable Distribution of Bulk Data," SIGCOMM, 1998. [15] T. Noguchi, T. Matsuda, and M. Yamamoto, "Performance Evaluation of New Multicast Architecture with Network Coding," IEICE Trans. Commun., Vol. E86B(6), June 2003 [16] Z. Li, B. Li, D. Jiang, and L. C. Lau, "On Achieving Optimal End-to-End Throughput in Data Networks: Theoretical and Empirical Studies," Technical Report, Department of Electrical and Computer Engineering, University of Toronto, February 2004.