State of

Data + AI

STATE OF DATA + AI
 We’re in the
golden age of
data and AI

STATE OF DATA + AI 2
INTRO

In the 6 months since ChatGPT launched, the world has woken up to the vast potential
of AI. The unparalleled pace of AI discoveries, model improvements and new products
on the market puts data and AI strategy at the top of conversations across every
organization around the world. We believe that AI will usher in the next generation of
product and software innovation, and we’re already seeing this play out in the market.
The next generation of winning companies and executives will be those who understand
and leverage AI.

In this report, we examine patterns and trends in data and AI adoption across more
than 9,000 global Databricks customers. By unifying business intelligence (BI) and AI
applications across companies’ entire data estates, the Databricks Lakehouse provides
a unique vantage point into the state of data and AI, including which products and
technologies are the fastest growing, the types of data science and machine learning
(DS/ML) applications being developed and more.

STATE OF DATA + AI 3
Here are the major stories we uncovered:

Companies are adopting Open source wins in today’s Organizations are increasingly
machine learning and large data and AI markets. Eight out using the Lakehouse for data
language models (LLMs) of 10 of our most widely warehousing, as evidenced
at a rapid pace. Natural adopted AI and machine by the high growth of data
language processing (NLP) learning products are based integration tools dbt and
is dominating use cases, on open source. Fivetran, and the accelerated
with an accelerated focus adoption of Databricks SQL.
on LLMs.

We hope that by sharing these trends, data leaders will be able to benchmark
their organizations and gain insights that help inform their strategies for an
era defined by data and AI.

STATE OF DATA + AI 4
Summary of
Key Findings

1 DATA SCIENCE AND MACHINE LEARNING:

NLP AND LLMS ARE IN HIGH DEMAND

• The number of companies using SaaS LLM APIs (used to access

services like ChatGPT) grew 1310% between the end of November
2022 and the beginning of May 2023

• NLP accounts for 49% of daily Python data science library usage,
making it the most popular application

• Organizations are putting substantially more models into

production (411% YoY growth) while also increasing their ML
experimentation (54% YoY growth)

• Organizations are getting more efficient with ML; for every three
experimental models, roughly one is put into production, compared
to five experimental models a year prior

STATE OF DATA + AI 5
2 FASTEST-GROWING DATA
AND AI PRODUCTS

• BI is the top data and AI market, but

growth trends in other markets show that
companies are increasingly looking at
3 ADOPTION AND
MIGRATION TRENDS

more advanced data use cases

• The fastest-growing data and AI product • 61% of customers migrating to the

is dbt, which grew 206% YoY by number Lakehouse are coming from on-
of customers prem and cloud data warehouses

• Data integration is the fastest-growing • The volume of data in Delta Lake

data and AI market on the Databricks has grown 304% YoY
Lakehouse with 117% YoY growth
• The Lakehouse is increasingly
being used for data warehousing,
including serverless data
warehousing with Databricks
SQL, which grew 144% YoY

STATE OF DATA + AI 6
 Methodology: How did Databricks
create this report?

The State of Data + AI is built from fully aggregated, anonymized data collected
from our customers based on how they are using the Databricks Lakehouse
and its broad ecosystem of integrated tools. This report focuses on machine
learning adoption, data architecture (integrations and migrations) and use cases.
The customers in this report represent every major industry and range in size
from startups to many of the world’s largest enterprises.

Unless otherwise noted, this report presents and analyzes data from February 1,
2022, to January 31, 2023, and usage is measured by number of customers.
When possible, we provide YoY comparisons to showcase growth trends over time.

STATE OF DATA + AI 7
Data Science and
Machine Learning
NATURAL LANGUAGE PROCESSING AND LARGE
LANGUAGE MODELS ARE IN HIGH DEMAND

Across all industries, companies leverage data science and

machine learning (DS/ML) to accelerate growth, improve
predictability and enhance customer experiences. Recent
advancements in large language models (LLMs) are propelling
companies to rethink AI within their own data strategies.
Given the rapidly evolving DS/ML landscape, we wanted to
understand several aspects of the market:

• Which types of DS/ML applications are companies investing

in? In particular, given the recent buzz, what does the data
around LLMs look like?

• Are companies making headway on operationalizing

their machine learning models (MLOps)?

STATE OF DATA + AI 8
 Time Series
Time Series

SPECIALIZED PYTHON DS/ML

Speech Recognition
LIBRARIES FROM FEBRUARY 2022
TO JANUARY 2023
Simulations &
Optimizations

Recommender Systems

DS/ML Applications
Natural
Language 
Processing

Industry Data Modeling

Graph

Note: This chart reflects the unique Geospatial

number of notebooks using ML
libraries per day in each of the
categories. It includes libraries used
Computer Vision
for the particular problem-solving use
cases mentioned. It does not include
libraries used in tooling for data
preparations and modeling. Anomaly Detection
& Segmentation

STATE OF DATA + AI 9
Natural language processing dominates
machine learning use cases

To understand how organizations are applying AI and Our second most popular DS/ML application is
ML within the Lakehouse, we aggregated the usage simulations and optimization, which accounts for 30% of
of specialized Python libraries, which include NLTK, all use cases. This signals organizations are using data to
Transformers and FuzzyWuzzy, into popular data science model prototypes and solve problems cost-effectively.
use cases.1 We look at data from these libraries because
Python is on the cutting edge of new developments in ML,
advanced analytics and AI, and has consistently ranked In our data set, 49% of
as one of the most popular programming languages in specialized Python libraries
recent years. used are associated with NLP

Our most popular use case is natural language processing

(NLP), a rapidly growing field that enables businesses to Many of the DS/ML use cases are predominantly
gain value from unstructured textual data. This opens the leveraged by specific industries. While they take up a
door for users to accomplish tasks that were previously smaller share of the total, they are mission-critical for
too abstract for code, such as summarizing content many organizations. For example, time series includes
or extracting sentiment from customer reviews. In our forecasting, a use case that is especially popular in
data set, 49% of libraries used are associated with NLP. industries such as Retail and CPG, which rely heavily
LLMs also fall within this bucket. Given the innovations on the ability to forecast the need for every item in
launched in recent months, we expect to see NLP take every store.
off even more in coming years as it is applied to use
cases like chatbots, research assistance, fraud detection,
content generation and more.

1. This data does not include general-purpose ML libraries, including

scikit-learn or TensorFlow.

STATE OF DATA + AI 10
 USE OF LARGE LANGUAGE MODELS (LLMS)

March 24, 2023

Dolly Launch

Nov 30, 2022

Transformer-Related ChatGPT Launch
Libraries
Number of Customers

SaaS LLM APIs

LLM Tools

Feb Mar Apr May June July Aug Sept Oct Nov Dec Jan Feb Mar Apr May
2022 2023

Note: There are several popular types of Python libraries that are commonly used for LLMs.
These libraries provide pretrained models and tools for building, training and deploying LLMs.
We have rolled these libraries up into groupings based on the type of functionality they provide.

Data consistently dips in the last week of December due to seasonality.

STATE OF DATA + AI 11
Large language models are
the “it” tool Organizations can leverage LLMs either by
using SaaS LLM APIs to call services like
ChatGPT from OpenAI or by operating their
LLMs are currently one of the hottest and most-watched areas own LLMs in-house.
in the field of NLP. LLMs have been instrumental in enabling
Thinking of building your own modern LLM
machines to understand, interpret and generate human
application? This approach could entail
language in a way that was previously impossible, powering
the use of specialized transformer-related
everything from machine translation to content creation to Python libraries to train the model, as well as
virtual assistants and chatbots. LLM tools like LangChain to develop prompt
interfaces or integrations to other systems.

Transformer-related libraries had been growing in popularity

LLM DEFINITIONS
even before ChatGPT thrust LLMs into the public consciousness.
◊ Transformer-related libraries:
Within the last 6 months, our data shows two accelerating
Python libraries used to train LLMs
trends: organizations are building their own LLMs, which models (example: Hugging Face)
like Dolly show can be quite accessible and inexpensive. And,
◊ SaaS LLM APIs: Libraries used to access
they are using proprietary models like ChatGPT. Transformer-
LLMs as a service (example: OpenAI)
related libraries, such as Hugging Face, which are used to train
LLMs, have the highest adoption within the Lakehouse. ◊ LLM tools: Toolchains for working
with and building proprietary LLMs
(example: LangChain)
The second most popular type is SaaS LLMs, which are used
to access models like OpenAI. This category has grown
exponentially in parallel with the launch of ChatGPT: the
number of Lakehouse customers using SaaS LLMs grew
an impressive 1310% between the end of November 2022 and
the beginning of May 2023. (In contrast, transformer-related
libraries grew 82% in this same period.)

STATE OF DATA + AI 12
Machine learning experimentation and production
take off across industries

The increasing demand for ML solutions and the growing MLflow Model Registry launched in May 2021. Overall, the
availability of technologies have led to a significant number of logged models has grown 54% since February
increase in experimentation and production, two distinct 2022, while the number of registered models has grown
parts of the ML model lifecycle. We look at the logging 411% over the same period. This growth in volume
and registering of models in MLflow, an open source suggests organizations are understanding the value of
platform developed by Databricks, to understand how ML investing in and allocating more people power to ML.
is trending and being adopted within organizations.

LOGGED MODELS AND REGISTERED MODELS AND ML PRODUCTION

ML EXPERIMENTATION Production models have undergone the experimentation
During the experimentation phase of ML, data scientists phase and are then deployed in real-world applications. They
develop models designed to solve given tasks. After training are typically used to make predictions or decisions based on
the models, they test them to evaluate their accuracy, new data. Registering a model is the process of recording and
precision, recall (the percentage of correctly predicted storing metadata about a trained model in a centralized location
positive instances out of all actual positive instances), that allows users to easily access and reuse existing models.
and more. These metrics are logged (recorded) in order to Registering models prior to production enables organizations to
analyze the various models’ performance and identify which ensure consistency and reliability in model deployment and scale.
approach works best for the given task.
We have chosen registered models to represent ML production
We have chosen logged models as a proxy to measure ML because the MLflow Model Registry is designed to manage
experimentation because the MLflow Tracking Server is models that have left the experimentation phase through the
designed to facilitate experiment tracking and reproducibility. rest of their lifecycle.

STATE OF DATA + AI 13
Organizations test numerous approaches and variables saw that for roughly every five experimental models, one
before committing an ML model to production. We was registered. Recent advances in ML, such as improved
wanted to understand, “How many models do data open source libraries like MLflow and Hugging Face, have
scientists experiment with before moving to production?” radically simplified building and putting models into
production. The result is that 34% of logged models are
Our data shows the ratio of logged to registered models now candidates for production today, an improvement
is 2.9 : 1 as of January 2023. This means that for roughly from over 20% just a year ago.
every three experimental models, one model will get
registered as a candidate for production. This ratio has
improved significantly from just a year prior, when we

RATIO OF LOGGED VS.

REGISTERED MODELS
Number of Models

2.9 : 1
Ratio of Logged to Registered
Feb Mar Apr May June July Aug Sept Oct Nov Dec Jan
Models in Jan 2023 2022 2023

STATE OF DATA + AI 14
The Modern Data
and AI Stack
Over the last several years, the trend toward building
open, unified data architectures has played out in our
own data. We see that data leaders are opting to preserve
choice, leverage the best products and deliver innovation
across their organizations by democratizing access to
data for more people.

STATE OF DATA + AI 15
TOP 5 AI AND MACHINE LEARNING PRODUCTS

Plotly and Dash

Hugging Face

John Snow Labs

Customers

Labelbox
Customers

NVIDIA
of of
Number
Number

Feb Mar Apr May June July Aug Sept Oct Nov Dec Jan
2022
Feb Mar Apr May June July Aug Sept Oct Nov 2023
Dec Jan
2022 2023

STATE OF DATA + AI 16
Top AI and ML Products
As companies integrate data science and ML into
their business strategies, many leaders are looking for
guidance on the right tools to add to their arsenals. The
Rising
Databricks Lakehouse integrates with a growing number Star
of AI and ML solutions to support these use cases.

One of our most interesting findings is that open source Launched in October 2022, LangChain is an
is dominating the top ranks; 3 out of our 5 most widely open source framework for developing LLM
applications. As a new integration, LangChain
adopted AI and ML products on the Lakehouse are based
does not qualify for this year’s Top 5 AI and
on open source. This indicates a growing sentiment across ML Products list. But its accelerated growth
industries: open platforms and products are critical to with the Databricks Lakehouse is worth
today’s AI and ML strategies. We anticipate this trend to highlighting, as it speaks volumes about the
continue with the rise of generative AI. Many organizations current state of the industry.

want to leverage LLMs but share concerns over issues

The LangChain and Databrick integration
such as data privacy for their sensitive data. Open source launched at the end of April 2023. In just
and open models empower organizations to build LLMs three months, LangChain became the third
without relying on third-party proprietary tools. most popular AI and ML product with our
customers, following Plotly and Dash and
Hugging Face.

This data point supports a very clear

message: enterprises want to use generative
AI within their businesses now.

STATE OF DATA + AI 17
FASTEST-GROWING DATA AND AI PRODUCTS

dbt 206%

Fivetran 181%

Informatica 174%

Qlik Data Integration 152%

Esri 145%

Looker 141%

Hugging Face 110%

Lytics 101%

Great Expectations 100%

Kepler.gl 95%

0% 50% 100% 150% 200%

Year-Over-Year Growth by Number of Customers

STATE OF DATA + AI 18
 DBT IS THE FASTEST-GROWING DATA
AND AI PRODUCT OF 2023
The data ecosystem is undergoing a major transition,
and selecting the right products is critical for companies
looking to take advantage of the newest innovations.
Because the Databricks Lakehouse is used broadly across
this ecosystem, it provides unique insights into how
customers adopt hundreds of data products and services.

We discovered that as companies move quickly to

develop more advanced use cases, they are investing
in newer products that produce trusted data sets for
reporting, ML modeling and operational workflows. Hence,
we see the rapid rise of data integration products. dbt, a
data transformation tool, and Fivetran, which automates
data pipelines, are our two fastest-growing data and AI
products. This suggests a new era of the data integration
market, with challenger tools making headway as
companies shift to prioritize DS/ML initiatives. With Great
Expectations from Superconductive in the ninth spot,
a full 50% of our fastest-growing products represent
the data integration category.

STATE OF DATA + AI 19
 GROWTH OF DATA AND AI MARKETS

Business Intelligence

Data Governance
& Security

Data Science &

Machine Learning

Data Integration
Number of Customers

Feb Mar Apr May June July Aug Sept Oct Nov Dec Jan
2022 2023

Note: In this chart, we count the number of customers deploying one or more data and AI products in each category. These four
categories do not encompass all products. Databricks products, such as Unity Catalog, are not included in this data.

20
Data and AI markets: business intelligence is
standard, organizations invest in their machine
learning foundation

To understand how organizations are prioritizing their While BI is often where organizations start their data
data initiatives, we aggregated all data and AI products on journey, companies are increasingly looking at more
the Databricks Lakehouse and categorized them into four advanced data and AI use cases.
core markets: BI, data governance and security, DS/ML,
and data integration. Our data set confirms that BI tools DEMAND FOR DATA INTEGRATION PRODUCTS
IS GROWING FAST
are more widely adopted across organizations relative to
more nascent categories — and they continue to grow, We see the fastest growth in the data integration market.
with a 66% YoY increase in adoption. This aligns with the These tools enable a company to integrate vast amounts
broader trend of more organizations performing data of upstream and downstream data in one consolidated
warehousing on a Lakehouse, covered in the next section, view. Data integration products ensure that all BI and DS/
Views from the Lakehouse. ML initiatives are built on a solid foundation.

While it’s easier for smaller markets to experience

Data integration is the faster growth, at 117% YoY increased adoption, the data
fastest-growing market, integration market is growing substantially faster than BI.
This trend dovetails with the rapid growth of ML adoption
with 117% YoY growth
we see across the Lakehouse, covered in the DS/ML
section of the report.

STATE OF DATA + AI 21
Views from
the Lakehouse
MIGRATION AND DATA
FORMAT TRENDS

Data migration is a major undertaking: it can be risky,

expensive and delay companies’ timelines. It’s not a
task to jump into lightly. As organizations run into the
limitations, scalability challenges and the cost burden
of legacy data platforms, they are increasingly likely
to migrate to a new type of architecture.

STATE OF DATA + AI 22
Migration trends: SOURCE OF NEW CUSTOMER 
the best data warehouse MIGRATIONS TO DATABRICKS
is a Lakehouse

The Lakehouse Platform is an attractive ple

u lti ions
alternative to traditional data warehouses M at
gr
Mi
because it supports advanced use cases and
DS/ML, allowing organizations to boost their
overall data strategy. As evidenced by the most 12%

Ha
se
popular data and AI products, with BI and data

doo
ou
ata Wareh
integration tools at the top, organizations are
39%

p
increasingly using the data lakehouse for data
warehousing. To better understand which legacy 27%
platforms organizations are moving away from,

em D
we look at the migrations of new customers

Pr
to Databricks.

n- O
22%
An interesting takeaway is that roughly half of the
companies moving to the Lakehouse are coming
from data warehouses. This includes the 22% Cl
oud use
Data Wareho
that are moving from cloud data warehouses.
It also demonstrates a growing focus on running
data warehousing workloads on a Lakehouse
and unifying data platforms to reduce cost.

STATE OF DATA + AI 23 23
Rising tides: the volume VOLUME OF DATA MANAGED,
of data in Delta Lake BY STORAGE FORMAT
has grown 304% YoY

As the volume of data explodes, an increasingly

large proportion is in the form of semi-structured
and unstructured data. Previously, organizations
had to manage multiple different platforms for
their structured, unstructured and semi-structured
data, which caused unnecessary complexity and

Volume of Data
high costs. The Lakehouse solves this problem by
providing a unified platform for all data types
and formats.

Delta Lake is the foundation of the Databricks

Lakehouse. The Delta Lake format encompasses
structured, unstructured and semi-structured
data. Use has surged over the past 2 years.
When compared to the steady, flat or declining Jan Jan Jan Jan Jan
2019 2020 2021 2022 2023
growth in other storage formats (e.g., text, JSON
and CSV), our data shows that a growing number
of organizations are turning to Delta Lake to
Delta Text CSV Avro
manage their data. In June 2022, Delta Lake
surpassed Parquet as the most popular data lake Parquet ORC JSON
source, reaching 304% YoY growth.

STATE OF DATA + AI 24
Data warehousing grows,
with emphasis on serverless

Over the past 2 years, companies have vastly increased their usage
of data warehousing on the Lakehouse Platform. This is especially
Data Lakehouse
demonstrated by use of Databricks SQL ­— the serverless data
Warehouse Platform
warehouse on the Lakehouse — which shows 144% YoY growth.
This suggests that organizations are increasingly ditching traditional
data warehouses and are able to perform all their BI and analytics
on a Lakehouse.

DATA WAREHOUSING
ON LAKEHOUSE WITH
DATABRICKS SQL
Number of Customers

Note: There is a spike in October 2021

as a result of the ungated preview
launch of Databricks SQL, followed by
General Availability in December 2021.

Data consistently dips in the last Jan July Jan July Jan
week of December due to seasonality. 2021 2021 2022 2022 2023

STATE OF DATA + AI 25
CONCLUSION

Generation AI
We’re excited that companies are progressing into more
advanced ML and AI use cases, and the modern data
and AI stack is evolving to keep up. Along with the rapid
growth of data integration tools (including our fastest
growing, dbt), we’re seeing the rapid rise of NLP and LLM
usage in our own data set, and there’s no doubt that the
next few years will see an explosion in these technologies.
It’s never been more clear: the companies that harness
the power of DS/ML will lead the next generation of data.

STATE OF DATA + AI 26
About Databricks

Databricks is the data and AI company. More than 9,000

organizations worldwide — including Comcast, Condé Nast and
over 50% of the Fortune 500 — rely on the Databricks Lakehouse
Platform to unify their data, analytics and AI. Databricks is
headquartered in San Francisco, with offices around the globe.
Founded by the original creators of Apache Spark™, Delta Lake
and MLflow, Databricks is on a mission to help data teams solve
the world’s toughest problems. To learn more, follow Databricks
on Twitter, LinkedIn and Instagram.

DISCOVER LAKEHOUSE

© Databricks 2024. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation | Terms of Use

STATE OF DATA + AI 27