FoundationDB Summit 2018

FoundationDB Summit registration will be available directly outside of 3AB at the Washington State Convention Center.

This talk will provide a birds-eye view of the current state of the FoundationDB project and its community. This will cover some details of interesting deployments and collaborators as well as community projects of note. To make sure that attendees, both new to the project and more seasoned, have good context on the workings of the community, this will cover current and emerging community process.

This presentation will walk through what is happening in the internals of the FoundationDB in response to user operations. The talk will focus on explaining how FoundationDB provides external consistency, and how FoundationDB scales as more processes join a cluster.

FoundationDB is an integral part of the architecture of SnowflakeDB, a cloud SQL analytics database. FDB is effectively their OLTP engine and has allowed them to build some truly amazing and differentiating features.

Snowflake have been operating it for 4+ years, and as they have grown, FDB has continued to scale from tens of servers to thousands of servers at present.

In this talk Ashish will present why FDB was their preferred choice, what purposes it serves, and the reliability and scale of their FDB clusters.

Ryan will demonstrate how FoundationDB can be applied to solve real business problems today and how to map common infrastructure components like logs, tables, and indexes into a cohesive system within FoundationDB. His example applies these techniques to a problem ClickFunnels was facing in mid-2018, which required scanning millions of end-user data points for each of their tens of thousands of customers multiple times per hour. Through custom bitmap indexes built on top of FoundationDB, queries which simply wouldn't finish now take milliseconds, which enables new use cases never thought possible.

Wavefront has been running FoundationDB in production for years. Our fleet consists of 500 instances of FoundationDB spanning 35 Wavefront clusters. About 400 of those are running FoundationDB as both memory & ssd systems.

Wavefront Ops will show how we manage FoundationDB at scale, including automatically provisioning new clusters, expanding/contracting existing clusters and how we replace an entire FoundationDB fleet.

We’ll include a live demo showing how we use Terraform to quickly add capacity and how we use Ansible to manage exclusions (or replace an entire running fleet).

The discussion will also cover the architecture design of a single Fdb instance, including disk layouts and instance store caching that enable us to achieve memory write rates of 800KHz and ssd write rates of 70KHz.

Document Layer is a stateless micro server on top of FoundationDB that allows management of JSON documents at large scale. It exposes the traits of FoundationDB through document data model, such as fully ordered documents, consistent indexes and serializable transactions. It does all this while maintaining wire compatibility with MongoDB® API. As the compatibility is done at wire level, existing MongoDB® tools and drivers work seem less. In this talk, we explore how FoundationDB core strengths play well together with Document data model to make it easier to use and reliable database.

JanusGraph is a popular open source property graph database that serves a variety of transactional and analytical use cases. It was originally designed to run on top of a number of different distributed storage engines including Apache Cassandra and Apache HBase. This talk will discuss the new JanusGraph FoundationDB storage adapter which adds distributed ACID support to JanusGraph. Topics will include an intro to JanusGraph and FoundationDB layer development followed by a deep dive into how the property graph model and read/write access patterns have been mapped on to FoundationDB. We will also discuss how FoundationDB's unique features  address a number of JanusGraph pain points and discuss further improvements that it will enable going forward.

This is intended to be a somewhat quick how-to on how to build useful primitives using FoundationDB features to help avoid conflicts in highly-concurrent systems. In particular, thinking about minimizing contention and not running into correctness issues related to multiple workers going at once. This will be demonstrated by explaining the design of a few simple patterns that are often useful within larger applications. This could include implementing (or avoiding) an auto-incrementing primary key replacement, including the naïve approach, an approach using a max_index, and an approach using versionstamps. Another application might be a high-contention queue (like our high-contention queue layer), where applications must be able to both append to the end of a growing thing as well as allow for multiple readers to pop from the beginning. Another might be something like a semaphore where you might have multiple requests at once attempting to take or release locks and doing that in an efficient way might be important. (In principle, this could be shortened to one example to meet the 20 minute time limit.)

The current FDB disaster recovery solution uses asynchronous replication, therefore failover might result in data loss.  

For Snowflake, this is an unacceptable business risk.

Snowcannon is a synchronous, distributed queueing system built within FDB.  An FDB cluster can synchronously replicate transactions to Snowcannon, and Snowcannon asychronously pushes these transactions to a secondary cluster.  The queueing system has three benefits: (1) It can replay rollbacks and redos (2) It allows us to bring down the secondary cluster for maintenance while the primary remains online. (3) In the event that both primary and secondary cluster fails, it allows recovery from backups with zero data loss.

This talk will go over the design of Snowcannon, including the replication protocol, failover switch, queueing architecture, and the quorum protocol.

This presentation will explain the role of FDB's storage engine component, how the storage engine's requirements are influenced by FDB's distributed architecture, and it will give a brief overview of the upcoming first storage engine to be designed with FDB's specific read and commit paths in mind.  In addition to higher write throughput, this new design will have a smaller space footprint via key prefix compression and a configurable ability to enable long running read transactions by trading off a reasonable amount of disk space to persist old data versions on disk.

FoundationDB has a number of different configurations for replicating data between regions. This presentation will go through these different options and discuss the tradeoffs between them.

The basis for many open source projects is to solve a problem and deliver value.  But the growth and adoption comes from the way the project is setup for positive governance, community growth and sustainability.  Nithya will talk about the elements needed to set up a community for growth and opening doors to new contributors.

Snowflake is an analytics database provided as a cloud service. The Snowflake data warehouse uses a new SQL database engine that built from scratch and was designed for the cloud. In this talk Sameet will describe the design of Snowflake's novel multi-cluster, shared-data cloud architecture. The talk will highlight some of the key features of Snowflakelike extreme and instant elasticity, support for semi-structured and schema-less data, and data sharing. The talk will also present upcoming features.

Current backups and restore mechanism available for FDB can be quite resource and time intensive operation. Senthil, will present a mechanism to do instant backup and restores across FDB clusters that are equivalent. The mechanism relies on a snapshottable file-system made available to TLogs and Storage processes.

Hotspot, are a source of problems for all databases. On FDB they can result in a system-wide performance degradation. This talk will demonstrate how SnowflakeDB, which uses FDB as its OLTP database, keeps track of FDB transactions and maps data to partitions on FDB storage nodes. The enables detection of hotspots, and discusses ways to handle them.

This talk discusses the performance testing of FoundationDB done at Snowflake Computing. This includes discussion of a TPC-C workload that has been used to discover performance regressions and to test the performance effects of new features, rolling upgrades, and various cluster configurations.

FoundationDB's serialization library provides an abstraction that allows it to be replaced with a new serialization format without changing (much) client code. In this talk, Andrew will describe a flatbuffers implementation that is nearly a drop-in replace for the current serialization, and how it allows rolling upgrades and backward compatibility. Andrew will also describe how this might be tested in the simulator.

Entity Store is a FoundationDB layer that implements a data model for versioned entities with fine-grained authorization and lineage. It is implemented as a Python library exposing its own API above the FoundationDB Python bindings. Versioning of entities is automatic, with modification to immutable fields resulting in a new version rather than mutation. Versions form a parentage tree and can be explicitly selected for use. Read and write authorizations are separately recorded at the level of individual fields (a.k.a “cell-based security.”) Lineage is recorded via labeled, directed multigraphs and used to represent version parentage and other forms of provenance.

Each entity is modeled as a collection of objects that represent its distinct versions, along with a “core” object for non-versioned fields. Objects are schemaless and consist of any number of fields with values. Fields can be single-valued or set-valued, and either case can optionally be indexed. Fields can also have large blob values.

FoundationDB was chosen because it allows rich data models to be flexibly mapped to its Key-Value Store. Required features such as versioning and lineage were straightforward to incorporate into the layer. We rely on FoundationDB’s distributed transactions to allow multiple clients concurrent access to entities without danger of erroneous results or data corruption.

Our use case for Entity Store is as a backend for a metadata manager in a service for data science and machine learning pipelines. These pipelines interact with multiple types of entities, including data sets, notebooks, machine learning models, and experiments. Each of these entities has associated metadata that must be versioned, protected by authorization, and tracked for lineage. The metadata manager uses GraphQL to define schemas for entity types, queries, and mutations.