Skip to main content

Episode 11: Bitmaps and the Future of Real-Time Data Processing

Lars Kamp

Some studies estimate that nine out of ten copies of data are precomputed. Precomputation requires a lot of engineering and batch processing. Compare this to what you can achieve when instead computing raw data, which reduces the amount of data you need to manage, store, and secure by up to 90%. Yet, some precomputation has often still been required because of bottlenecks in I/O, storage, or compute.

FeatureBase is the first analytical database built entirely on bitmaps.

Bitmaps lay out data differently from both the row-oriented layout of transactional databases and the columnar layout of analytical databases; bitmaps store data at the value. Due to the nature of bitmaps, the data pertaining to each unique value within a row or column can be accessed independently without having to scan the row or column. The I/O for typical analytical workloads is only a fraction of that of traditional analytical queries.

Bitmaps are more efficient when it comes to storing, transporting, and managing data—they are orders of magnitude faster than today's popular cloud warehouses, and also an order of magnitude more efficient at storing data. Their efficiency makes them ideal for real-time processing and artificial intelligence workloads.

In fact, that's what positions FeatureBase as the database between real-time streaming engines like Kafka on one end, and cloud warehouses as long-term storage engines on the other end. FeatureBase is the working memory in-between the two.

Higinio "H.O." Maycotte is Founder and CEO at FeatureBase. In this session, we explore the mathematical pillars of databases and bitmaps. We cover:

The data footprint and scale of some of FeatureBase's customers is nothing short of breathtaking. One of their advertising customers processes 120 billion updates a day—that's 1.38 million updates per second. FeatureBase allowed them to reduce their server count from 1,000 servers to just 11, saving them millions of dollars per year.

The team at FeatureBase has invested over $30 million in R&D and nine years of their lives to advance the use of bitmaps in databases. Watch this fascinating session with H.O. to learn more about math, bitmaps, and modern real-time processing data architecture.