1 of 29

TSDB: 1 year in

Ganesh Vernekar

November, 2019

2 of 29

About me

TSDB: 1 year in

| 2

Ganesh Vernekar

Software Engineer, Grafana Labs

Prometheus GSoC’18 student

Prometheus Member, TSDB Maintainer

Graduated this year from IIT Hyderabad

@_codesome ganesh@grafana.com

3 of 29

What is TSDB (Time Series DataBase)?

TSDB: 1 year in

  • Storage engine of Prometheus 2.x
  • Independent repo in the past prometheus/tsdb
  • Now a part of Prometheus repo, inside tsdb directory

| 3

4 of 29

Statistics

TSDB: 1 year in

  • 500+ commits since Prometheus 2.0 release
  • 60+ contributors
  • 771 stars before archiving

| 4

5 of 29

Some selected features/enhancements

TSDB: 1 year in

| 5

6 of 29

Backfilling

TSDB: 1 year in

  • Issue before: cannot have overlapping blocks
  • Vertical query merging and compaction in Feb. 2019

| 6

7 of 29

Backfilling

TSDB: 1 year in

  • No recommended way to backfill yet
  • Community is taking it forward

| 7

8 of 29

WAL compression

TSDB: 1 year in

  • Optional snappy compression for WAL

​

​

​

​

  • Can save up to 50% of WAL size without compromising on performance

| 8

9 of 29

WAL read optimizations

TSDB: 1 year in

| 9

10 of 29

WAL read optimizations

TSDB: 1 year in

| 10

11 of 29

WAL read optimizations

TSDB: 1 year in

| 11

12 of 29

Allocation/memory optimization for compaction

TSDB: 1 year in

| 12

13 of 29

Allocation/memory optimization for compaction

TSDB: 1 year in

35% allocations

​

​

​

​

19% allocations

| 13

14 of 29

Allocation/memory optimization for compaction

TSDB: 1 year in

6.5% allocations

​

​

​

​

(Don’t have the numbers)

| 14

15 of 29

Various optimizations for the queries

TSDB: 1 year in

| 15

16 of 29

Various optimizations for the queries

TSDB: 1 year in

​

​

​

​

{foo=~”bar|baz”} => {foo=”bar”} or {foo=”baz”}

Make Grafana dashboard queries faster

| 16

17 of 29

Various optimizations for the queries

TSDB: 1 year in

| 17

18 of 29

Reuse Chunk Iterators

TSDB: 1 year in

| 18

19 of 29

Reuse Chunk Iterators

TSDB: 1 year in

| 19

20 of 29

Reuse Chunk Iterators

TSDB: 1 year in

| 20

21 of 29

Analyse churn

TSDB: 1 year in

| 21

22 of 29

Read Only TSDB

TSDB: 1 year in

  • Interface/implementation for Read Only Mode TSDB
  • Safe to use this interface with a live tsdb
  • TSDB cli already uses it

| 22

23 of 29

Upcoming features/enhancements

TSDB: 1 year in

| 23

24 of 29

Lifting index size limit

TSDB: 1 year in

| 24

25 of 29

Lifting index size limit

TSDB: 1 year in

  • 32 bit postings currently (reference of series in index)
    • Index size is limited to 64 GiB
  • Not difficult to hit the limit
    • 8M series with block spanning 9 days - 20 GiB

| 25

https://grafana.com/blog/2019/10/31/lifting-the-index-size-limit-of-prometheus-with-postings-compression/

26 of 29

64 bit postings

TSDB: 1 year in

GSoC 2019 work by Alec Wang (GH: naivewong)

  • 64 bit postings - practically unlimited index size
  • Use prefix compression to store postings (48 bit prefix)
  • Same performance and no increase in index size

​

| 26

011001

011010

101100

101101

101110

101111

0110

01

10

1011

00

01

10

11

https://grafana.com/blog/2019/10/31/lifting-the-index-size-limit-of-prometheus-with-postings-compression/

27 of 29

Isolation

TSDB: 1 year in

  • TSDB has A C and D from ACID, but not I
    • Some progress in the past by Brian and Goutham
  • Temporarily abandoned after that
  • Plan is to add it this year
    • Rebase the work of Brian and Goutham :P

| 27

28 of 29

Improved checkpointing of WAL

TSDB: 1 year in

Potential speedup of WAL replay by a big factor

| 28

29 of 29

Thanks! Questions?

@_codesome