Georg-August-Universität Göttingen Institut für Informatik
NOSQL Databases Dr. Lena Wiese Institut für Informatik Research Group Knowledge Engineering Fakultät für Mathematik und Informatik Georg-August Universität Göttingen
August/September 2016 Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
1 / 49
Georg-August-Universität Göttingen Institut für Informatik
Short CV Dr. Lena Wiese University of Göttingen (Research Group Leader Knowledge Engineering) University of Hildesheim (Visiting Professor for Databases) National Institute of Informatics, Tokyo, Japan Robert Bosch India Ltd., Bangalore, India Master/PhD: TU Dortmund Teaching and Research NoSQL databases (lecture, seminars, projects) Database security (encryption for Cassandra and HBase)
Web: http://wiese.free.fr/ Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
2 / 49
Georg-August-Universität Göttingen Institut für Informatik
Conference Announcement BTW’17 17th Conference on Database Systems for Business, Technology, and Web Conference of German Database community (sponsored by the German Informatics Society GI) March 6th through March 10th 2017 at the University of Stuttgart in Germany http://btw2017.informatik.uni-stuttgart.de/ Research and Industry Track, Demo Track, Workshops, Tutorials, Student Program, Dissertation Awards, Data Science Challenge Paper deadline: 23.9.2016 Data Science Challenge deadline: 17.10.2016
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
3 / 49
Georg-August-Universität Göttingen Institut für Informatik
Copyright Notice
Several pictures in this talk taken from my Master’s level text book (in English): Lena Wiese: Advanced Data Management for SQL, NoSQL, Cloud and Distributed Databases c 2015 DeGruyter/Oldenbourg
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
4 / 49
Georg-August-Universität Göttingen Institut für Informatik
Introduction :: Content
Overview 1
Introduction Content New Requirements
2
Graph Databases
3
XML Databases
4
Key-Value Stores
5
Document Stores
6
Column Stores
7
BigTable Databases
8
Polyglot Data Base Architectures
9
Conclusion
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
5 / 49
Georg-August-Universität Göttingen Institut für Informatik
Introduction :: Content
Content SQL Tabular row-wise storage: Relational Databases (RDBs) Query Language: SQL versus
NOSQL (Not Only SQL) Graph Databases XML Databases Key-value Stores Column Stores Bigtable Databases Object Databases and Object-Relational Databases ... Dr. Lena Wiese
Knowledge
{K∃} Engineering
NOSQL Databases
6 / 49
Georg-August-Universität Göttingen Institut für Informatik
Introduction :: Content
What is a Database System? A database system is required to manage huge amounts of data in an efficient, persistent, reliable, consistent, non-redundant way for multiple users Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
7 / 49
Georg-August-Universität Göttingen Institut für Informatik
Introduction :: New Requirements
New requirements Data are organized in complex structures (example: social networks)
foe
foo
me you
Data are constantly changing (frequent updates) Data are distributed on a huge number of interconnected servers (example: cloud storage) Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
8 / 49
Georg-August-Universität Göttingen Institut für Informatik
Introduction :: New Requirements
New requirements
Data are organized in complex structures (example: social networks) Data are constantly changing (frequent updates) write1 read1 write2
write3
write4 read2 write5
Data are distributed on a huge number of interconnected servers (example: cloud storage)
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
8 / 49
Georg-August-Universität Göttingen Institut für Informatik
Introduction :: New Requirements
New requirements Data are organized in complex structures (example: social networks) Data are constantly changing (frequent updates) Data are distributed on a huge number of interconnected servers (example: cloud storage) S3 data
S2
S4
user S1
S5 Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
8 / 49
Georg-August-Universität Göttingen Institut für Informatik
Introduction :: New Requirements
New requirements
Data are organized in complex structures (example: social networks) Data are constantly changing (frequent updates) Data are distributed on a huge number of interconnected servers (example: cloud storage) Revival of non-relational data models for novel applications
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
8 / 49
Georg-August-Universität Göttingen Institut für Informatik
Graph Databases :: Background
Overview 1
Introduction
2
Graph Databases Background Graph Management Systems
3
XML Databases
4
Key-Value Stores
5
Document Stores
6
Column Stores
7
BigTable Databases
8
Polyglot Data Base Architectures Knowledge
9
{K∃}
Conclusion
Engineering
Dr. Lena Wiese
NOSQL Databases
9 / 49
Georg-August-Universität Göttingen Institut für Informatik
Graph Databases :: Background
Why Graph Databases? Links between data items are important Example: Social Networks Recommender Systems Semantic Web Geographic Information Systems Bioinformatics ...
Name: Bob Age: 27 knows Name: Alice Age: 34
knows Name: Charlene Age: 29 K
Knowledge
dislikes
{ ∃} Engineering
Dr. Lena Wiese
NOSQL Databases
10 / 49
Georg-August-Universität Göttingen Institut für Informatik
Graph Databases :: Background
Why Graph Databases? Links between data items are important Social Networks Recommender Systems Semantic Web Example: Geographic Information Systems Bioinformatics ...
City: Hildesheim Population: 102T 35km City: Hannover Population: 522T
45km City: Braunschweig Population: 248T K
Knowledge
65km
{ ∃} Engineering
Dr. Lena Wiese
NOSQL Databases
10 / 49
Georg-August-Universität Göttingen Institut für Informatik
Graph Databases :: Graph Management
Property Graph Model A Property Graph is a directed multigraph Stores information (properties) in vertices and on edges A Property is a key-value pair like “Name: Alice” Sometimes multi-value properties: one key, list of values
For vertices and edges: predefined property key called Id with unique identifier value
Id: 4 Id: 1 Name: Alice Age: 34
Id: 2 Name: Bob Age: 27
Id: 6
Id: 5 Id: 3 Name: Charlene K∃ Age: 29
Knowledge
{ } Engineering
Dr. Lena Wiese
NOSQL Databases
11 / 49
Georg-August-Universität Göttingen Institut für Informatik
Graph Databases :: Graph Management
Property Graph Model: Paths Paths are serial concatenations of edges End vertex of one edge is start vertex of next edge on the path
Id: 4 Label: knows
Id: 2 Type: Person Name: Bob Age: 27
Id: 1 Type: Person Name: Alice Age: 34
Id: 5 Label: knows Id: 3 Type: Person Name: Charlene Age: 29 Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
12 / 49
Georg-August-Universität Göttingen Institut für Informatik
Graph Databases :: Graph Management
Property Graph Model: Paths Path “friends-of-friends” concatenates two edges with “Label: knows” Paths can be used as normal edges
Id: 4 Label: knows Id: 1 Type: Person Name: Alice Age: 34
Id: 2 Type: Person Name: Bob Age: 27
Path friends-of-friends
Id: 5 Label: knows Id: 3 Type: Person Name: Charlene Age: 29 Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
12 / 49
Georg-August-Universität Göttingen Institut für Informatik
Graph Databases :: Systems
Open Source Systems The TinkerPop http://tinkerpop.apache.org/ graph processing stack: a set of open source graph management modules
Neo4J graph database http://neo4j.com/ Cypher query language START alice = (people_idx, name, "Alice") MATCH (alice)-[:knows]->(aperson) RETURN (aperson)
HyperGraphDB: http://www.hypergraphdb.org/ Graph may contain hyperedges that combine more than two nodes Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
13 / 49
Georg-August-Universität Göttingen Institut für Informatik
XML Databases :: Background
Overview 1
Introduction
2
Graph Databases
3
XML Databases Background Numbering Schemes Systems
4
Key-Value Stores
5
Document Stores
6
Column Stores
7
BigTable Databases
8
Polyglot Data Base Architectures Knowledge
9
{K∃}
Conclusion
Engineering
Dr. Lena Wiese
NOSQL Databases
14 / 49
Georg-August-Universität Göttingen Institut für Informatik
XML Databases :: Background
XML XML: Extensible Markup Language Defined by the WWW Consortium (W3C) Intended as a document markup language (not a database language) Tags divide documents into sections Tag: label for a section of data Element: section of data beginning with and ending with matching Inside an element: arbitrary text other elements (“nesting”) Nothing (“empty element”): abbreviate to
Standardized query languages: XPath and XQuery
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
15 / 49
Georg-August-Universität Göttingen Institut für Informatik
XML Databases :: Background
Tree Model of XML Data 0 reservationsystem 1 Buergermeisterkapelle hotel 7 3 5 Hildesheim 2 pricesgl name location hotelID h1 6 8 65 Euro Hildesheim 65 Euro 4 Buergermeisterkapelle
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
16 / 49
Georg-August-Universität Göttingen Institut für Informatik
XML Databases :: Numbering Schemes
Numbering Scheme assigns each node of an XML tree a unique identifier (a label or node ID which is usually a number) Important for database application with frequent updates: How many nodes have to be renumbered in an update?
simplest scheme: preorder traversal of tree increasing a counter for each node: root node is numbered as the first node before numbering any other node this is done recursively for all child nodes
Renumbering: all nodes in the worst case Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
17 / 49
Georg-August-Universität Göttingen Institut für Informatik
XML Databases :: Systems
Open Source Systems
eXistDB: http://exist-db.org/ numbering scheme that virtually expands the tree into a complete tree such that not all node IDs correspond to existing nodes eXistDB offers several user APIs: RESTful API, XML:DB API, XML-RPC API, SOAP AP
BaseX: http://basex.org/ Numbering scheme: Pre/Dist/Size Several language bindings as well as a REST API, an XQJ API and a XML:DB API
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
18 / 49
Georg-August-Universität Göttingen Institut für Informatik
Key-Value Stores :: Background
Overview
Dr. Lena Wiese
1
Introduction
2
Graph Databases
3
XML Databases
4
Key-Value Stores Background Systems MapReduce Systems
5
Document Stores
6
Column Stores
7
BigTable Databases
8
Polyglot Data Base Architectures
9
Conclusion
Knowledge
{K∃} Engineering
NOSQL Databases
19 / 49
Georg-August-Universität Göttingen Institut für Informatik
Key-Value Stores :: Background
Key-Value Stores A key value pair is a tuple of two strings hkey, valuei You can get (or delete) a value from the store by key Schema-less: you can put arbitrary key-value pairs into the store value = store.get(key) store.put(key, value) store.delete(key)
Values can have other data types than just strings Values can even be a list or array of atomic values Simple but quick Simple data structure No advanced query language Good for “data-intensive” applications Application is responsible or combining key-value pairs into more K∃ complex objects
Knowledge
{ } Engineering
Dr. Lena Wiese
NOSQL Databases
20 / 49
Georg-August-Universität Göttingen Institut für Informatik
Key-Value Stores :: Systems
Open Source Systems
Redis: http://redis.io/ in-memory key-value store data types: string, linked lists, unsorted set, sorted set, hash, bit array, hyperloglog
Riak-KV: http://basho.com/products/riak-kv/ key-value pairs called Riak objects grouped into buckets convergent replicated data types (CRDTs) Riak’s search functionality based on Apache Solr (Yokozuna)
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
21 / 49
Georg-August-Universität Göttingen Institut für Informatik
Key-Value Stores :: MapReduce
MapReduce Applied at Google Jeffrey Dean / Sanjay Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters”, OSDI’04: Sixth Symposium on Operating System Design and Implementation, 2004.
“The computation takes a set of input key/value pairs, and produces a set of output key/value pairs. The user of the MapReduce library expresses the computation as two functions: Map and Reduce.” Four basic steps 1 2 3 4
split input key-value pairs into disjunct subsets compute map function on each input subset group all intermediate values by key (shuffle) reduce values of each group Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
22 / 49
Georg-August-Universität Göttingen Institut für Informatik
Key-Value Stores :: MapReduce
MapReduce: Example split
map
shuffle
sentence1
(word3 ,1)
(word1 ,(1,1,1))
(word4 ,1)
(word2 ,(1))
sentence2
server1
sentence3
reduce server4
(word1 ,3) (word2 ,1)
(word3 ,1) (word1 ,1)
sentence4
server2
sentence5
(word1 ,1)
(word3 ,(1,1,1))
(word2 ,1)
(word4 ,(1,1))
server5
(word3 ,3) (word4 ,2)
(word4 ,1)
sentence6 sentence7 Dr. Lena Wiese
server3
Knowledge
{K∃}
(word3 ,1) (word1 ,1)
Engineering
NOSQL Databases
23 / 49
Georg-August-Universität Göttingen Institut für Informatik
Key-Value Stores :: Systems
Open Source Systems Apache Hadoop: http://hadoop.apache.org/ Hadoop Distributed File System (HDFS)
Apache Spark: http://spark.apache.org/ data flow programming model on top of Hadoop
Apache Pig: http://pig.apache.org/ express parallel execution of data analytics tasks input={(’alice’,{’charlene’,’emily’}), (’bob’,{’david’,’emily’})}; output = FOREACH input GENERATE $0, FLATTEN($1);
Apache Hive: http://hive.apache.org/ querying and data management layer can serialize tables as files in HDFS HiveQL queries are compiled into Hadoop MapReduce tasks
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
24 / 49
Georg-August-Universität Göttingen Institut für Informatik
Document Stores :: Background
Overview 1
Introduction
2
Graph Databases
3
XML Databases
4
Key-Value Stores
5
Document Stores Background Systems
6
Column Stores
7
BigTable Databases
8
Polyglot Data Base Architectures
9
Conclusion
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
25 / 49
Georg-August-Universität Göttingen Institut für Informatik
Document Stores :: Background
JSON: JavaScript Object Notation human-readable text format more compact than XML nesting of key-value pairs { "firstName":"Alice", "lastName" :"Smith", "age":31, "address" :{ "street":"Main Street", "number":12, "city":"Newtown", "zip":31141 } , "telephone":[935279,908077,278784]
{K∃} Engineering
} Dr. Lena Wiese
Knowledge
NOSQL Databases
26 / 49
Georg-August-Universität Göttingen Institut für Informatik
Document Stores :: Systems
Open Source Systems MongoDB: https://www.mongodb.org/ BSON storage format (binary JSON representation) db.persons.find(age$lt: 34)
CouchDB: http://couchdb.apache.org/ retrieval process with views defined as map function function(doc) { if(doc.lastname && doc.age) { emit(doc.lastname, doc.age); } }
Couchbase: http://www.couchbase.com SQL-like query language
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
27 / 49
Georg-August-Universität Göttingen Institut für Informatik
Column Stores :: Background
Overview 1
Introduction
2
Graph Databases
3
XML Databases
4
Key-Value Stores
5
Document Stores
6
Column Stores Background Column Compression Systems
7
BigTable Databases
8
Polyglot Data Base Architectures Knowledge
9
{K∃}
Conclusion
Engineering
Dr. Lena Wiese
NOSQL Databases
28 / 49
Georg-August-Universität Göttingen Institut für Informatik
Column Stores :: Background
Why Column Stores? A row store is a row-oriented relational database Data are stored in tables On disk, data in a row are stored consecutively Currently used in most commercially successful RDBMSs
A column store is a column-oriented relational database Data are stored in tables On disk, data in a column are stored consecutively In use since the 1970s but less successful than row stores
Example BookLending
BookID ReaderID ReturnDate 123 225 25-10-2011 234 347 31-10-2011 Storage order in row store: 123,225,25-10-2011,234,347,31-10-2011 Storage order in column store: 123,234,225,347,25-10-2011,31-10-2011 Dr. Lena Wiese
NOSQL Databases
Knowledge
{K∃} Engineering
29 / 49
Georg-August-Universität Göttingen Institut für Informatik
Column Stores :: Background
Advantages of Column Stores Only columns (attributes) that are needed are read from disk into main memory, because a memory page contains only values of a column Values in a column (that is, values of the same attribute domain) can be compressed better when stored consecutively (“locality”) Iterating or aggregating over values in a column can be done quickly, because they are stored consecutively For example, summing up all values in a column, finding the average, maximum...
Adding new columns to a table is easy Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
30 / 49
Georg-August-Universität Göttingen Institut für Informatik
Column Stores :: Column Compression
Column Compression Columns may contain lots of repetitions of values Compression can be more effective on columns Option 1: run-length encoding run-length: how many repetitions of a value are stored consecutively?
Option 2: bit-vector encoding create a bit vector for each value in the column
Option 3: dictionary encoding create a dictionary for single values or sequences of values
Option 4: frame of reference encoding store off-set from a reference point
Option 5: differential encoding store off-set from previous value
Stavros Harizopoulos / Daniel Abadi / Peter Boncz, “Column-Oriented Database Systems”, VLDB Tutorial, 2009 Dr. Lena Wiese
NOSQL Databases
Knowledge
{K∃} Engineering
31 / 49
Georg-August-Universität Göttingen Institut für Informatik
Column Stores :: Column Compression
Example: Run-Length Encoding BookLending
BID 123 386 938 123 234
RID 225 225 225 347 347
RD 25-10-2012 20-10-2012 27-10-2012 25-11-2012 31-10-2012
Store ReaderID (RID) in run-length encoding count number of consecutive repetitions format: (value, start row, run-length) RID: ( (225, 1, 3), (347, 4, 2) )
Answer queries on compressed format How many books does each reader have? SELECT RID, COUNT(*) FROM BookLending GROUP BY RID Just return (the sum of) the run-lengths for each ReaderID value K∃ Result: (225, 3), (347, 2)
Knowledge
{ } Engineering
Dr. Lena Wiese
NOSQL Databases
32 / 49
Georg-August-Universität Göttingen Institut für Informatik
Column Stores :: Systems
Systems
MonetDB: https://www.monetdb.org/ open source “column store pioneers”
Apache Parquet: http://parquet.apache.org/ implements column striping: transform nested data to columns
Commercial systems SAP HANA HP Vertica IBM DashDB
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
33 / 49
Georg-August-Universität Göttingen Institut für Informatik
BigTable Databases :: Background
Overview 1
Introduction
2
Graph Databases
3
XML Databases
4
Key-Value Stores
5
Document Stores
6
Column Stores
7
BigTable Databases Background Storage Organization Systems
8
Polyglot Data Base Architectures Knowledge
9
{K∃}
Conclusion
Engineering
Dr. Lena Wiese
NOSQL Databases
34 / 49
Georg-August-Universität Göttingen Institut für Informatik
BigTable Databases :: Background
Google BigTable Fay Chang / Jeffrey Dean / Sanjay Ghemawat / Wilson C. Hsieh / Deborah A. Wallach / Mike Burrows / Tushar Chandra / Andrew Fikes / Robert E. Gruber, “Bigtable: A Distributed Storage System for Structured Data”, OSDI, 2006 “A Bigtable is a sparse, distributed, persistent, multi-dimensional sorted map” Google BigTable is indexed by a row key, column key, and a timestamp Map: ( row:string, column:string, time:int64) → string A Big Table may have an unbounded number of columns. Columns are grouped into sets called column families.
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
35 / 49
Georg-August-Universität Göttingen Institut für Informatik
BigTable Databases :: Background
BigTable & HBase Data Structure Store data that is accessed together in a column family Columns in a single column family can vary arbitrarily for each row. Only fetch column families of columns that are required by query Data locality: Store data in a column family together on disk
table Library
row key BID
column family LendingInfo
Title
Author
25-10-2012
25-11-2012
123
Databases
Miller
Mayer
Green
Title
Author
20-10-2012
386
Algorithms
Jacobs
Mayer
Title
Author
27-10-2012
938
Programming
Brown
Mayer
234 Dr. Lena Wiese
column family BookInfo
Title
Author
31-10-2012
SQL
Smith
Green
Knowledge
{K∃} Engineering
NOSQL Databases
36 / 49
Georg-August-Universität Göttingen Institut für Informatik
BigTable Databases :: Storage Organization
Writing to memory tables and data files The most recent writes are collected in a main memory table (memtable) of fixed size. All data records written to the on-disk store will only be appended to the existing records. Once written, these records are read-only and cannot be modified: they are immutable data files. Any modification of a record must hence also be simulated by appending a new record in the store. Deletions are treated by writing a new record (tombstone) for a key. Main memory
flush
Disk Knowledge
write
memtable
Sorted file n
...
Sorted file 2
Sorted file 1
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
37 / 49
Georg-August-Universität Göttingen Institut für Informatik
BigTable Databases :: Storage Organization
Reading from memory tables and data files The downside of immutable data files is that they complicate the read process: retrieving all the relevant data that match a user query requires combining records from several on-disk data files and the memtable.
This combination may affect records for different search keys that are spread out across several data files; but it may also apply to records for the same key of which different versions exist in different data files. In other words, all sorted data files have to be searched for records matching the read request. Disk
Main memory memtable
Sorted file n
...
Sorted file 2
combine read
block buffer
Dr. Lena Wiese
combine
Sorted file 1
Knowledge
{K∃} Engineering
NOSQL Databases
38 / 49
Georg-August-Universität Göttingen Institut für Informatik
BigTable Databases :: Systems
Open Source Systems
Apache Cassandra: http://cassandra.apache.org/ column families in a keyspace CQL: SQL-like query language INSERT INTO bookinfo (bookid, title, author) VALUES (1002,’Databases’,’Miller’);
Apache HBase: http://hbase.apache.org/ stores tables in namespaces tables contain column families
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
39 / 49
Georg-August-Universität Göttingen Institut für Informatik
Polyglot Data Base Architectures :: Polyglot Persistence
Overview 1
Introduction
2
Graph Databases
3
XML Databases
4
Key-Value Stores
5
Document Stores
6
Column Stores
7
BigTable Databases
8
Polyglot Data Base Architectures Polyglot Persistence Lambda Architecture Multi-Model Databases Knowledge
9
{K∃}
Conclusion
Engineering
Dr. Lena Wiese
NOSQL Databases
40 / 49
Georg-August-Universität Göttingen Institut für Informatik
Polyglot Data Base Architectures :: Polyglot Persistence
Polyglot Data Management
Data management layer has to handle contradictory requirements access patterns: write-heavy workloads vs read-heavy workloads data model: data of different structures access method: web application access via REST vs programmatic access vs query language
Consider a database and storage architecture that includes all these requirements (well, at least some...) Polyglot Persistence Lambda Architecture Multi-Model Databases Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
41 / 49
Georg-August-Universität Göttingen Institut für Informatik
Polyglot Data Base Architectures :: Polyglot Persistence
Polyglot Persistence
Choose as many databases as needed Fowler, M.J., Sadalage, P.J.: NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence. Prentice Hall (2012)
Example: Apache Drill http://drill.apache.org/ Apache Drill is inspired by the ideas developed in Google’s Dremel system Melnik, S., Gubarev, A., Long, J.J., Romer, G., Shivakumar, S., Tolton, M., Vassilakis, T.: Dremel: interactive analysis of web-scale datasets. Proceedings of the VLDB Endowment 3(1-2), 330–339 (2010)
Introduces an integration layer decomposing queries in to several subqueries redirecting queries to the appropriate databases recombining the results obtained from the accessed databases Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
42 / 49
Georg-August-Universität Göttingen Institut für Informatik
Polyglot Data Base Architectures :: Polyglot Persistence
Polyglot Persistence analytical query
graph traversal
Integration layer
write-heavy SQL query transaction
RESTbased access
Query decomposition Query redirection Result recombination Synchronization
graph database
key-value store
SQL database
in-memory store
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
43 / 49
Polyglot Data Base Architectures :: Lambda Architecture
Georg-August-Universität Göttingen Institut für Informatik
Lambda Architecture For real-time / streaming data Combination of a slower batch processing layer and a speedier stream processing layer Speed layer: only the most recent data delivered in several real-time views Batch layer: data stored in an append-only and immutable fashion in a “master dataset” delivered in so-called batch views Serving layer: makes batch views accessible to user queries by maintaining indexes
User queries answered by merging data from batch views and real-time views Open source implementation following the ideas of a lambda architecture is Apache Druid http://druid.io/ (streaming data K in real-time nodes and batch data in historical nodes)
Knowledge
{ ∃} Engineering
Dr. Lena Wiese
NOSQL Databases
44 / 49
Georg-August-Universität Göttingen Institut für Informatik
Polyglot Data Base Architectures :: Lambda Architecture
Lambda Architecture Batch layer
ap
pen
d
Master data set
Batch view 1
Serving layer Index 1
Batch view 2
Index 2
Batch view 3
Index 3
Batch view 4
Index 4
Data stream Speed view 1
d
n pe
ap
Speed layer Recent data set
Speed view 2
merge
Speed view 3 Speed view 4
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
45 / 49
Polyglot Data Base Architectures :: Multi-Model Databases
Georg-August-Universität Göttingen Institut für Informatik
Multi-Model Databases Data in a single store but providing access to the data with different APIs (according to different data models) Either support different data models directly inside the database engine or offer layers for additional data models on top of a single-model engine OrientDB http://orientdb.com/ a document API, an object API, and a graph API (Java Graph API is compliant with Tinkerpop) extensions of the SQL standard to interact will all three APIs
ArangoDB https://www.arangodb.com/ a graph API, a key-value API and a document API Query language AQL (ArangoDB query language) resembles SQL but adds several database-specific extensions to it
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
46 / 49
Polyglot Data Base Architectures :: Multi-Model Databases
Georg-August-Universität Göttingen Institut für Informatik
Multi-Model Databases
graph traversal
Graph layer
write-heavy transaction RESTbased access
key-value store
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
47 / 49
Georg-August-Universität Göttingen Institut für Informatik
Conclusion
Overview 1
Introduction
2
Graph Databases
3
XML Databases
4
Key-Value Stores
5
Document Stores
6
Column Stores
7
BigTable Databases
8
Polyglot Data Base Architectures
9
Conclusion
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
48 / 49
Georg-August-Universität Göttingen Institut für Informatik
Conclusion
Conclusion
Many, many other data models than just relational tables Lots of different query languages (no standards) Problems with reliability (no long-term experience, open source development teams) Which database you choose depends on your needs
Knowledge
{K∃} Engineering
Dr. Lena Wiese
NOSQL Databases
49 / 49
