Clustering is one of the most useful techniques in exploratory data analysis.

Even when you do not know in advance what kinds of groups exist in your data, clustering can help you discover meaningful patterns such as:

• Customer segments
• Product groups
• Behavioral patterns
• Regional differences
• Types of survey respondents

However, clustering always comes with one important question:

How many clusters should we create?

For example, in K-Means Clustering, one of the most widely used clustering algorithms, you need to decide the number of clusters, usually called K, before running the algorithm.

If K is too small, groups that should be separated may be combined into one cluster. If K is too large, groups that should belong together may be split into unnecessarily small pieces, making the result harder to interpret.

This is why methods such as the Silhouette Method and the Elbow Method are often used to help choose a reasonable number of clusters.

In this article, we will focus on the Silhouette Method and discuss:

• What the Silhouette Method is
• How to interpret the Silhouette Score
• What to watch out for when using it
• How it differs from the Elbow Method

Exploratory recently added support for the Silhouette Method in K-Means clustering, so we will also look at how you can use it inside Exploratory at the end.

What Is the Silhouette Method?

The Silhouette Method is a way to evaluate how well each data point fits within the cluster it has been assigned to.

It answers two questions at the same time:

1. Is this data point close to other data points in the same cluster?
2. Is this data point far away from data points in other clusters?

A good clustering result should satisfy both conditions.

In other words, a data point should be:

• Similar to other points in the same cluster
• Clearly different from points in other clusters

A metric that measures how well it satisfy these two is called Silhouette Score.

What Is the Silhouette Score?

The Silhouette Score ranges from -1 to 1.

Let’s say we have several data points in a two-dimensional space like the one below.

Now, suppose we divide these data points into three clusters.

Let’s calculate the Silhouette Score for the point indicated by the arrow.

The Silhouette Score can be calculated as follows.

Cohesion measures how similar a point is to the other points in the same cluster. More specifically, it is calculated by summing the distances from that point to all the other points in the same cluster.

Separation, on the other hand, is the distance from that point to all the points in the nearest neighboring cluster. The larger this value is, the farther the point is from the other cluster.

The Silhouette Score is calculated by subtracting cohesion from separation, then dividing the result by the larger of the two values.

For example, suppose the cohesion is 40 and the separation is 80.

In this case, the Silhouette Score is 0.5.

As you can see from the formula, when separation is large and cohesion is small, the numerator becomes larger, which makes the Silhouette Score higher. This means the point is closer to other points in its own cluster and farther away from points in other clusters.

On the other hand, when separation is small and cohesion is large, the numerator gets closer to zero or even becomes negative, which makes the Silhouette Score lower. This means the point is farther from other points in its own cluster and closer to points in another cluster.

Hence the Silhouette Score ranges between -1 and 1, and -1 being the worst and 1 being the best.

Close to 1: Good

A score close to 1 means the point fits well within its assigned cluster and is clearly separated from other clusters.

This is the ideal situation.

Around 0: Ambiguous

A score around 0 means the point is near the boundary between two clusters.

This does not always mean something is wrong, but it suggests that the separation between clusters is weak.

Negative: Possible Misclassification

A negative score means the point may be closer to another cluster than to the cluster it was assigned to.

If many data points have negative scores, you should be careful when interpreting the clustering result.

Three Silhouette Metrics That Help Evaluate Clustering Quality

So far, we have talked about the Silhouette Score for a single data point.

In practice, we calculate the score for all data points and then summarize the result. The most common summary is the average Silhouette Score.

However, looking only at the average can be misleading.

To evaluate clustering results more carefully, it is useful to look at three metrics together:

• Average Silhouette Score
• Minimum Silhouette Score
• Percentage of Negative Silhouette Scores

Together, these metrics help you understand both the overall quality and whether there are problematic assignments.

1. Average Silhouette Score: Overall Quality

The average Silhouette Score is the average of the Silhouette Scores across all observations.

It answers the question:

Overall, how well are the data points assigned to their clusters?

For example, suppose we try different values of K and get the following results.

In this case, K = 3 has the highest average Silhouette Score, so it becomes a strong candidate.

However, the average can hide problems.

For example, some data points may be poorly assigned even if the overall average looks good. This is why we should also check the other two metrics.

2. Minimum Silhouette Score: The Worst Assignment

The Minimum Silhouette Score is the lowest score among all observations.

It answers the question:

How bad is the worst cluster assignment?

For example:

The average scores are almost the same, but K = 4 has a much worse minimum score.

This may indicate:

• An outlier
• Noise in the data
• Overlapping clusters
• An unsuitable value of K

That said, the minimum score is affected by a single observation. A very low minimum score does not automatically mean that the number of clusters is wrong.

It is best to treat it as a warning signal rather than a final decision rule.

3. Percentage of Negative Silhouette Scores

The percentage of negative Silhouette Scores tells you what percentage of observations have a score below 0.

It answers the question:

How many data points may be closer to another cluster than to their assigned cluster?

For example:

In this case, K = 4 has a slightly higher average score, but it also has a much higher percentage of negative scores.

That is a warning.

It may mean that K = 4 creates clusters that look good on average but leave many observations poorly assigned.

In this situation, K = 3 may be a more stable and reliable choice, even though its average score is slightly lower.

How to Use the Three Metrics Together

Here is a simple way to think about the three metrics.

A practical workflow is:

1. Start with the K that has the highest average Silhouette Score.
2. Check whether the percentage of negative scores is low.
3. Check whether the minimum score is not extremely low.
4. Compare the actual cluster profiles.
5. Choose the K that is meaningful and useful for your analysis.

The goal is not simply to choose the highest score, it is to find clusters that balance among:

• Separation
• Stability
• Interpretability
• Practical usefulness

Silhouette Method vs. Elbow Method

Another popular method for choosing the number of clusters is the Elbow Method.

Silhouette Method and Elbow Method are both useful, but they evaluate different aspects of clustering.

Elbow Method

The Elbow Method looks at how compact the clusters are.

In K-Means clustering, the algorithm tries to minimize the distance between each data point and its cluster center. This distance is often summarized as within-cluster sum of squares, or WSS.

As K increases, WSS always decreases.

As we increase the number of clusters from 1 to 2, then from 2 to 3, the distance (WSS) tends to decrease substantially at first. However, after a certain point, the amount of decrease starts to level off.

The point where this decrease changes from steep to gradual is called the “elbow,” and the K value, or number of clusters, at this elbow is considered a good candidate for the optimal number of clusters.

The challenge is that, in real-world data, the elbow is not always clear. Sometimes the curve gradually decreases without an obvious bend.

Silhouette Method

The Silhouette Method, on the other hand, evaluates how well clusters are separated.

It considers both:

• How close points are within the same cluster
• How far they are from neighboring clusters

In simple terms:

• The Elbow Method asks, “How compact are the clusters?”
• The Silhouette Method asks, “How well separated are the clusters?”

Both are useful, and in practice it is often best to look at both.

However, when comparing different values of K, the Silhouette Method has a practical advantage because its score is easier to interpret and compare.

This is especially true when you look at not only the average score, but also the minimum score and the percentage of negative scores.

A Practical Workflow for Choosing the Number of Clusters

Here is a practical way to use the Silhouette Method.

1. Choose a Range of K Values

Start with a reasonable range, such as K = 2 to K = 10.

Technically, you can create many clusters, but if the result becomes too hard to explain, it may not be useful in practice.

2. Compare the Average Silhouette Scores

As a rough guideline:

However, these are only rough guidelines.

For survey data, behavioral data, and many real-world business datasets, scores can often be lower. A “weak” score does not automatically mean the clusters are useless.

The real question is whether the clusters are meaningful for your purpose.

3. Check the Percentage of Negative Scores

Look at how many observations may be poorly assigned.

If one K has a slightly higher average score but a much higher percentage of negative scores, you may want to be cautious.

4. Check the Minimum Score

The minimum score can help you identify outliers or problematic assignments.

Again, do not overreact to a single low value, but use it as a warning signal.

5. Compare Similar Candidate K Values

Do not automatically choose the K with the highest average Silhouette Score.

If several values of K have similar scores, compare them as candidates.

6. Profile the Clusters

Once you have a few candidate K values, create the clusters and examine their characteristics.

For example, in customer segmentation, you may want clusters that can be explained in simple language, such as:

“High-frequency customers with high spending and low discount usage.”

If the clusters are hard to explain, they may not be useful even if the score looks good.

7. Use Domain Knowledge

Clustering is an unsupervised learning method, which means there is no correct answer label.

Unlike regression or classification, we cannot simply compare predictions against known outcomes.

This means the final decision should be made by combining quantitative metrics, visual analysis, and domain knowledge.

The Silhouette Method supports your decision, but it does not replace your judgment.

Using the Silhouette Method in Exploratory

Starting with Exploratory v15.5, you can use the Silhouette Method in the K-Means Clustering under Analytics view.

Choosing the most optimized number of clusters

When you run K-Means clustering, Exploratory automatically evaluates multiple values of K using the Silhouette Method and show you the results.

You can compare different cluster counts and choose reasonable candidates.

For example, suppose you analyze this employee survey data, which you can download from here.

The average Silhouette Score may suggest that K = 2 is the best option, while K = 3 is the second-best option. However, when you look at the percentage of negative scores, you may find that K = 2 has more observations that are not clearly separated.

In this case, you may want to compare K = 2 and K = 3 by looking at the actual cluster profiles.

In Exploratory, you can use visualizations such as radar charts, box plots, and scatter plots to compare the characteristics of each cluster.

For example, with K = 2, the result may simply separate respondents into a “high satisfaction” group and a “low satisfaction” group.

On the other hand, with K = 3, the result may further separate the high-satisfaction respondents into two different patterns.

In this case, K = 3 may be more useful, even if K = 2 has a slightly better average score.

This is where cluster interpretation becomes important.

You can also use Exploratory’s AI Summary to get help interpreting the clustering result, including advice on the number of clusters and explanations of the cluster characteristics.

Checking Silhouette Scores by Cluster

Exploratory also shows Silhouette Score metrics for each cluster.

This is useful because the overall score may hide differences across clusters.

For example, some clusters may be well separated, while another cluster may contain many ambiguous observations.

By checking the average Silhouette Score and the percentage of negative scores for each cluster, you can identify which clusters are stable and which ones should be interpreted with caution.

This helps you avoid treating all clusters as equally reliable.

Summary

Clustering is useful for many types of analysis, but choosing the number of clusters can be confusing.

The Silhouette Method helps you by answering questions such as:

• Are the clusters clearly separated?
• Are we creating too many groups?
• Are there many poorly assigned observations?
• Which K gives us a more stable and interpretable result?

The key is not to choose the highest score alone.

Instead, use the Silhouette Method as part of a broader workflow that combines:

• Quantitative evaluation
• Visual exploration
• Cluster profiling
• AI-assisted interpretation
• Human judgment

What I personally find exciting about the new Silhouette Method support in Exploratory is that it connects these steps into one workflow.

You can evaluate the clustering result statistically, explore the clusters visually, get interpretation support from AI, and then make the final decision based on your own knowledge and purpose.

That is the kind of workflow that helps people use data to make better decisions.

Try with Exploratory

If you have wanted to try cluster analysis, or if you have used clustering before but struggled with choosing the right number of clusters, this is a good time to try it in Exploratory.

👉 Download the latest version of Exploratory

https://exploratory.io/download

If you do not have an Exploratory account yet, you can sign up and start a 30-day free trial here:

https://exploratory.io/

If your trial has already expired, you can launch the latest version and click “Extend Trial” to try it again.

How to Find the Right Number of Clusters with Silhouette Method was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.

In recent years, gradient boosting algorithms such as XGBoost, LightGBM, and CatBoost have become popular choices for building accurate predictive models on tabular business data.

At Exploratory, we added support for CatBoost in version 15.5, based on requests from our users.

So, what is CatBoost?

To understand why CatBoost is useful, it helps to look at the kind of data we often work with in the real world.

Business data does not always come neatly packaged as numbers.

Customer data often includes fields such as customer segment, region, industry, and subscription plan. Marketing data may include campaigns, ad channels, and traffic sources. Survey data often contains response categories and demographic attributes.

In other words, business data is full of categorical variables.

CatBoost is a machine learning algorithm designed to build high-performing predictive models, especially when your data contains many categorical variables.

In this post, I’ll explain what CatBoost is, how it differs from XGBoost and LightGBM, and why it is especially useful for business data with many categories.

CatBoost can be particularly effective when your data includes variables such as:

• Customer segment
• Region
• Product category
• Industry
• Subscription plan
• Campaign
• Ad channel
• Survey responses

We’ll cover:

• What problem CatBoost was designed to solve
• How it differs from Random Forest, XGBoost, and LightGBM
• Why CatBoost handles categorical variables well
• What kinds of data CatBoost is good for
• When you should consider using CatBoost

If you’d like to learn more about LightGBM, you may also find this article helpful:

• Introduction to LightGBM: How It Differs from Random Forest and XGBoost

The Problem CatBoost Tries to Solve

Suppose you want to predict customer churn based on data like this:

In this data, Monthly Revenue is numeric, but everything else is categorical.

In many real-world business datasets, categorical variables are not a minor. They are often a major part of the data.

But most machine learning algorithms cannot directly understand text values such as:

These values need to be converted into numbers before they can be used by a model.

This is where CatBoost starts to become interesting.

Before going deep down into CatBoost, let’s refresh our memory on how Boosting model has evolved over the last decades, starting from Random Forest to CatBoost.

Random Forest: Many Independent Trees

Random Forest builds many decision trees independently and combines their results.

Each tree follows a process like this:

1. Randomly sample rows from the data.
2. Randomly select a subset of variables when splitting.
3. Let each tree make its own prediction.

The final prediction is based on the average prediction for regression, or the majority vote for classification.

The core idea is simple: by combining many independent trees, Random Forest reduces the instability of a single decision tree and produces more stable predictions.

However, the trees do not learn from each other.

Each tree is built independently. One tree does not try to fix the mistakes made by another tree.

As a result, Random Forest is often easy to use and fairly robust, but it may not achieve the same level of predictive accuracy as modern boosting algorithms.

XGBoost: Trees That Correct Previous Mistakes

XGBoost uses a method called gradient boosting.

Instead of building many trees independently, it builds trees sequentially. Each new tree tries to correct the mistakes made by the trees that came before it.

For example, in a regression problem, the first tree makes an initial prediction. Then the model calculates the errors, or residuals, between the actual values and the predicted values.

RowActual ValuePredicted ValueResidual11073215141389–1

The next tree is trained not to predict the original target directly, but to predict these residuals.

Why?

Because the residuals tell the model how it should adjust its predictions to reduce the loss.

Conceptually, the model evolves like this:

• Tree 1 → Initial prediction
• Tree 2 → Corrects the errors from Tree 1
• Tree 3 → Corrects the remaining errors
• Tree 4 → Continues improving the prediction

The updated prediction can be thought of as:

New prediction = Prediction from previous trees + Learning rate × Prediction from the new tree

This process often produces higher accuracy than Random Forest.

However, as data becomes larger and more complex, building many trees sequentially can become computationally expensive.

LightGBM: Built for Speed and Scale

LightGBM is also a gradient boosting algorithm, but it was designed to make boosting faster and more scalable.

It uses several techniques to improve performance, including:

• Histogram-based splitting
• Leaf-wise tree growth
• Gradient-based One-Side Sampling, or GOSS
• Exclusive Feature Bundling, or EFB

LightGBM is especially useful when:

• The dataset has many rows.
• There are many explanatory variables.
• Training speed matters.
• You need to scale to larger data.

Because of this, LightGBM has become one of the most widely used machine learning algorithms for practical business applications.

For more details, see:

• Introduction to LightGBM: How It Differs from Random Forest and XGBoost

So What Makes CatBoost Different?

CatBoost is also a gradient boosting algorithm, just like XGBoost and LightGBM.

The overall learning process is similar: it builds many trees sequentially, and each new tree helps improve the predictions made so far.

But CatBoost has one major design focus:

CatBoost was designed to handle categorical variables effectively.

This makes it especially useful when your data includes categorical variables with many possible values, such as:

• Product categories
• Campaign IDs
• Regions
• Stores
• User segments
• Features used by customers
• Survey responses

These are sometimes called high-cardinality categorical variables.

To understand why CatBoost matters, let’s first look at the common ways categorical variables are handled.

The Problem with One-Hot Encoding

A common way to handle categorical variables is one-hot encoding.

For example, suppose you have a subscription plan column like this:

One-hot encoding turns each category into a separate column:

This works fine when the number of categories is small.

But what if you have:

• 500 product categories
• 1,000 regions
• 10,000 campaign IDs

Then one-hot encoding creates a very large number of columns.

This can cause several problems:

• Higher memory usage
• Slower training
• Sparse data
• Higher risk of overfitting

So one-hot encoding is not always ideal, especially for business data with many categorical variables.

Target Encoding: A More Informative Alternative

Another approach is called target encoding.

Instead of turning each category into many binary columns, target encoding replaces each category with a statistic calculated from the target variable.

For example, if you are predicting churn, you might calculate the churn rate for each subscription plan:

Then the model can use this information directly.

For example:

Starter → 0.67
Business → 0.00

This gives the model useful information:

“Customers on the Starter plan tend to churn more often.”

Target encoding can be very powerful, especially for categorical variables with many levels.

But it also has a serious risk.

The Risk of Data Leakage

Suppose you have this small dataset:

If you calculate the churn rate for the Starter plan, you get:

2 / 3 = 0.67

So you might encode the Starter plan as:

Starter → 0.67

But there is a problem.

When you calculate the value 0.67, you are using the target value, Churn, from all rows, including the row you are trying to predict.

For Customer A, the encoded value includes Customer A’s own churn result.

That means you are using the answer to create an input variable.

This is a form of data leakage.

When this happens, the model may look very accurate on the training data, but it may perform poorly on new data because it has learned from information that would not be available in a real prediction setting.

This is one of the problems CatBoost was designed to address.

Ordered Target Statistics

CatBoost uses a technique called Ordered Target Statistics to reduce leakage when encoding categorical variables.

The idea is simple:

When calculating the encoded value for a row, use only the rows that come before it.

Let’s use the same example.

Suppose the data is arranged in a random order:

Assume the overall churn rate, used as the prior, is 50%.

CatBoost converts the Plan variable like this:

What is happening here?

For Customer A, there are no previous Starter customers, so CatBoost uses the prior:

A → No previous Starter customers → Use 0.50

For Customer B, the only previous Starter customer is A:

B → Previous Starter customers: A → Churn rate = 1 / 1 = 1.00

For Customer C, the previous Starter customers are A and B:

C → Previous Starter customers: A and B → Churn rate = (1 + 0) / 2 = 0.50

For Customer D, the previous Starter customers are A, B, and C:

D → Previous Starter customers: A, B, and C → Churn rate = (1 + 0 + 1) / 3 = 0.67

The key point is this:

The encoded value for each row is calculated without using that row’s own target value.

So CatBoost can still use the useful information that Starter customers tend to churn more often, while reducing the leakage that can happen with naive target encoding.

This is one of the main reasons CatBoost works well with categorical variables.

Ordered Boosting

CatBoost also uses another important idea called Ordered Boosting.

In standard gradient boosting, the model is trained on the full training data. Then it makes predictions on that same training data and calculates the errors. The next tree is trained to correct those errors.

At first glance, this seems reasonable.

But there is a subtle issue.

When the model predicts a row in the training data, that row has already been used to train the model. In other words, the model is predicting a row it has already seen.

In production, the situation is different. The model is asked to predict new data that it has not seen before.

This difference between the training-time prediction situation and the production-time prediction situation is sometimes called prediction shift.

CatBoost tries to reduce this shift by using a random order of the data.

When calculating the prediction for a row, CatBoost uses only the data that comes before that row in the random order.

For example, if the data is ordered as A, B, C, D, then when calculating the error for C, the model uses only A and B. It does not use C’s own target value to help predict C.

This makes the training process more similar to the real-world prediction process, where the model must predict data it has not seen before.

Ordered Target Statistics helps reduce leakage when encoding categorical variables.

Ordered Boosting helps reduce prediction shift during boosting.

Together, these ideas are a big part of what makes CatBoost different from XGBoost and LightGBM.

Symmetric Trees

Another important feature of CatBoost is that it uses symmetric trees, also called oblivious trees.

In a regular decision tree, different branches can use different split conditions.

Conceptually, it may look like this:

In a symmetric tree, all nodes at the same depth use the same split condition.

Conceptually, it looks like this:

This means, one of the trees would look something like this.

At first, this may sound restrictive.

A regular decision tree can be more flexible because each branch can choose its own split condition.

But CatBoost is not trying to build one perfect tree.

It is a boosting model. It builds many relatively small trees and combines them.

One tree may capture a broad pattern. The next tree may correct part of the remaining error. Later trees continue refining the prediction.

So even if each individual symmetric tree is less flexible, the full boosted model can still be very expressive.

The symmetric tree structure also has several advantages:

• Faster prediction
• Better memory efficiency
• Simpler and more stable tree structure
• Lower risk of overfitting
• Good performance when many trees are combined

This is another example of CatBoost’s design philosophy: use a more controlled tree structure, then gain flexibility by combining many trees through boosting.

What CatBoost Cannot Protect You From

At this point, it may sound like CatBoost solves data leakage.

But that is not quite true.

CatBoost helps reduce leakage caused by target encoding of categorical variables.

It does not protect you from leakage already present in the data itself.

For example, suppose you are predicting customer churn and you include variables such as:

• Churn date
• Refund amount after cancellation
• Number of support tickets after cancellation
• Final billing status
• Cancellation reason

These variables would not be available at the time you actually need to make the prediction.

If you include them, any model can produce artificially high accuracy.

CatBoost cannot fix that.

The basic rule remains the same:

Use only the information that would be available at the time of prediction.

This is true for CatBoost, LightGBM, XGBoost, Random Forest, and any other predictive model.

How XGBoost and LightGBM Handle Categorical Variables

By the way, it is important to clarify one point here.

Using categorical variables with XGBoost or LightGBM does not automatically mean you have a leakage problem.

XGBoost, LightGBM, and CatBoost use different strategies for categorical variables.

For example, LightGBM can learn category-based splits such as:

Plan in {Starter, Pro}

versus:

Plan in {Business, Enterprise}

The resulting rule may look like:

if Plan is Starter or Pro:
    go left
else:
    go right

CatBoost takes a different approach.

Instead of only grouping categories, it can convert a category into a target-based statistic in a leakage-aware way.

For example:

Starter
↓
Average churn rate among previous Starter customers

The important point is that these models have different design goals.

LightGBM emphasizes speed and scalability.

CatBoost emphasizes effective use of categorical variables while reducing leakage and prediction shift.

XGBoost is a strong general-purpose boosting algorithm with broad adoption and flexibility.

What Kind of Data Is CatBoost Good For?

CatBoost is especially useful for business datasets that contain many categorical variables.

Examples include:

Customer Data

• Customer segment
• Region
• Industry
• Subscription plan
• Account type

Marketing Data

• Ad channel
• Campaign ID
• Traffic source
• Landing page
• Creative type

Product Data

• Product category
• Brand
• SKU
• Store
• Supplier

Survey Data

• Demographic attributes
• Response categories
• Segments
• Preference groups

SaaS Data

• Plan type
• Customer segment
• Feature usage category
• Acquisition channel
• Support category

If your dataset has many categorical variables, especially variables with many unique values, CatBoost is often worth trying.

Which Model Should You Use?

So which model should you choose: Random Forest, XGBoost, LightGBM, or CatBoost?

In practice, the best answer is usually:

Try multiple models, evaluate them properly, tune the parameters, and choose the one that performs best on validation or test data.

That said, the following guideline can be helpful as a starting point:

This does not mean one model is always better than the others. It means each model has a different strength.

Random Forest is stable and easy to use.

XGBoost is powerful and flexible.

LightGBM is fast and scalable.

CatBoost is strong when categorical variables play an important role.

For many business datasets, especially customer, marketing, survey, and SaaS data, CatBoost is a very practical option to consider.

Summary

Random Forest, XGBoost, LightGBM, and CatBoost are all tree-based ensemble models, but they have different design philosophies.

• Random Forest focuses on stability.
• XGBoost focuses on predictive accuracy.
• LightGBM focuses on speed and scalability.
• CatBoost focuses on categorical variables.

CatBoost’s biggest advantage is that it can use information from categorical variables while reducing the leakage risk that often comes with naive target encoding.

This makes it especially useful for business data, where categorical variables are often central to the problem.

If you have mainly used XGBoost or LightGBM so far, CatBoost is worth trying, especially when your data contains many categorical variables.

You may find that it performs surprisingly well with less preprocessing than you expected.

Try CatBoost in Exploratory

Exploratory supports CatBoost starting with version 15.5.

You can build CatBoost models directly from the Analytics view using the UI.

The basic steps are:

1. Open the Analytics view.
2. Select CatBoost.
3. Choose the target variable.
4. Choose the explanatory variables.
5. Click Run.

You can also tune model parameters from the Settings dialog to improve predictive accuracy while controlling overfitting.

If you work with data that contains many categorical variables, CatBoost is definitely worth trying.

You can download the latest version of Exploratory here:

👉 Download Exploratory

https://exploratory.io/download

If you do not have an account yet, you can sign up and start a 30-day free trial here:

https://exploratory.io/

If your trial has already expired, you can launch the latest version and click “Extend Trial” to try it again.

Introduction to CatBoost: Why It Works So Well with Categorical Data was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’re excited to announce the release of Exploratory v15.5! 🎉

Although v15.5 is a minor release in version number, it includes several major enhancements to Exploratory’s Analytics features. This release significantly expands what you can do with machine learning, clustering, and statistical testing.

In particular, this release focuses on helping users:

• Build more accurate machine learning models.
• Tune machine learning parameters more easily.
• Evaluate clustering results more objectively.
• Run statistical tests that are commonly used in A/B testing, survey analysis, and business analytics.

As AI increasingly supports data analysis and report generation, it becomes even more important for analysts to understand the models, assumptions, and statistical results behind the analysis.

Exploratory v15.5 is an important step toward making analytics more powerful, transparent, and trustworthy.

Here are the highlights.

CatBoost Model Added to Machine Learning

Exploratory v15.5 adds CatBoost as a new machine learning model.

CatBoost is a boosting-based machine learning model, similar to XGBoost and LightGBM. In recent years, it has become widely used in the data science community, especially because of its strong handling of categorical variables.

Business data often contains many categorical variables, such as region, product category, customer type, marketing channel, job role, or industry. CatBoost can be a strong option when your data includes these types of variables.

Exploratory already supports XGBoost and LightGBM. With the addition of CatBoost, you can now compare multiple high-performance machine learning models and choose the one that works best for your data and use case.

If you want to improve prediction accuracy, especially with data that includes many categorical variables, we encourage you to try CatBoost.

Redesigned Parameter Tuning UI for Machine Learning Models

Tuning parameters is an important part of improving machine learning models.

However, models such as XGBoost, LightGBM, CatBoost, and Random Forest have many parameters. It is not always easy to understand what each parameter does, where to find it, or what value to use.

In v15.5, we redesigned the parameter setting UI for machine learning models.

Parameters are now grouped by their roles and purposes, and the layout is made more consistent across machine learning models.

This makes it easier to find parameters related to areas such as:

• Model complexity
• Learning strategy
• Row and column sampling
• Overfitting control
• Validation data and early stopping

We also added recommended values and practical hints for each parameter.

For example, you can now see what is likely to happen when a value is set higher or lower. This helps you adjust parameters even if you are not familiar with every technical term.

Machine learning is not just about building a model once. It is about reviewing the result, adjusting the settings, and improving the model. The redesigned UI makes this tuning process more intuitive and accessible.

Silhouette Method Added to K-Means Clustering

When using K-Means clustering, you need to decide the number of clusters in advance.

But choosing the right number of clusters is often difficult.

Until now, Exploratory supported the Elbow Method to help choose the number of clusters. The Elbow Method is widely used, but sometimes the “elbow” in the chart is not clear, which can make the decision subjective.

In v15.5, we added the Silhouette Method.

The Silhouette Method evaluates how well each data point fits within its assigned cluster, while also considering how well it is separated from other clusters.

The resulting silhouette score helps you evaluate the number of clusters more quantitatively.

In Exploratory, you can also check not only the average silhouette score, but also the percentage of observations with negative silhouette scores.

This is important because a high average score alone can be misleading. If many observations have negative silhouette scores, it may indicate that those observations are not assigned to appropriate clusters.

By looking at both the average silhouette score and the percentage of negative scores, you can make a more realistic decision about the number of clusters.

We also added silhouette scores by cluster to the summary table.

This allows you to see which clusters are well formed and which clusters may be weaker or less clearly separated.

Starting with this release, the Silhouette Method is the default method for K-Means Clustering. The Elbow Method is still supported and can be selected from the settings if you want to use it.

Two-Sample Proportion Test Added

Exploratory v15.5 adds the Two-Sample Proportion Test.

This test is used to check whether the proportion of True values is statistically different between two groups.

For example, you can use it to answer questions such as:

• Is the conversion rate different between A and B in an A/B test?
• Is the purchase rate different between two customer groups?
• Is the click-through rate different between two campaigns?
• Is the response rate different between two demographic groups?

Until now, you could use the Chi-Square Test to examine differences in proportions across multiple groups.

However, when your main goal is to directly compare the proportions between two groups, such as in an A/B test, the Two-Sample Proportion Test is often a more natural fit.

With this new test, it is now easier to analyze proportion differences in marketing analytics, product analytics, survey analysis, and other common business use cases.

One-Sample Proportion Test and One-Sample t-Test Added

This release also adds two one-sample tests:

• One-Sample Proportion Test
• One-Sample t-Test

These tests are used to check whether the proportion or average in your sample is statistically different from a predefined benchmark, target, or hypothesized value.

For example, with the One-Sample Proportion Test, you can answer questions such as:

• Is the conversion rate higher than the target value of 10%?
• Is the satisfaction rate different from the expected value of 80%?
• Is the defect rate higher than the acceptable threshold of 5%?

With the One-Sample t-Test, you can answer questions such as:

• Is the average revenue higher than the target value?
• Is the average satisfaction score different from the benchmark?
• Is the average processing time shorter than the expected value?

These tests make it possible to evaluate your data against a target or benchmark, not only against another group.

Note: Table Column Width Can Now Be Adjusted

We also made an improvement to Notes.

In the Note Editor, you can use tables to organize your analysis results and explanations. With this release, you can now adjust table column widths with your mouse, and the adjusted widths are preserved when the note is published.

Note Editor:

Published Note:

This makes it easier to improve the readability of tables and create cleaner, more polished analytical reports.

Summary of New Features in Exploratory v15.5

Here are the main additions and improvements in Exploratory v15.5.

• Added CatBoost as a new machine learning model.
• Redesigned the parameter setting UI for machine learning models.
• Added the Silhouette Method to K-Means clustering.
• Added silhouette scores by cluster to the clustering summary.
• Added the Two-Sample Proportion Test.
• Added the One-Sample Proportion Test.
• Added the One-Sample t-Test.
• Improved Notes so table column widths can be adjusted and preserved after publishing.

This release also includes many bug fixes and other improvements. For details, please see the release notes.

Try Exploratory v15 !

Exploratory v15 includes many new features designed to support trustworthy data analysis in the age of AI.

In v15.0, we introduced major improvements to make data wrangling workflows more visible, explainable, documented, and easier to fix when errors occur.

Exploratory v15 - The 3 Rs for Trustworthy Data Analysis in the AI Era

With v15.5, we are making major improvements to Analytics.

You can now not only prepare clean and reproducible data, but also use machine learning, clustering, and statistical testing to gain deeper insights from that data.

As AI increasingly helps with analysis and report writing, it becomes more important for analysts to understand how models are built, how parameters affect results, and how statistical results should be interpreted.

We believe Exploratory v15.5 is an important milestone toward building a trustworthy analytics foundation for the AI era.

Please download the latest version and try it out.

👉 Download Exploratory

https://exploratory.io/download

If you don’t have an account yet, you can start a 30-day free trial here.

👉 Start Free Trial

https://exploratory.io/

That’s all for today.

If you have any feedback about Exploratory v15, please feel free to comment below!

We will continue working hard to improve Exploratory and make data analysis more trustworthy and accessible in the AI era.

Thank you!

Kan,
CEO/Exploratory

Exploratory v15.5 was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.

I hope you are doing well.

This is Kan from Exploratory.

We are very excited to announce the release of Exploratory v15! 🎉

The theme of this release is:

“The 3 Rs for Trustworthy Data Analysis in the AI Era”

• Readability — Data transformation processes should be easy to read and understand.
• Reproducibility — Analysis results should be reproducible.
• Reliability — Reproducibility can be maintained even if the data & structure change.

Today, AI is making it easier than ever to generate charts, dashboards, reports, and even analytical insights automatically.

However, no matter how beautifully a dashboard or analysis result is generated, if we don’t understand how the underlying data was processed and transformed, it becomes difficult to trust the result with confidence.

Questions like:

• “Where did this number come from?”
• “How transformations, aggregations, and filters were applied?”
• “Can this analysis still be reproduced with the new data?”

are becoming more important than ever in the AI era.

To address this challenge, Exploratory v15 introduces new capabilities designed to improve the readability, reproducibility, and reliability of data preparation and analysis workflows.

Here are some of the major new features in v15:

• Step Diagram to visualize data transformation pipelines
• AI Step Summary to explain data transformation steps in natural language
• Automatic Document Generation for the entire data preparation process
• AI-Powered Auto Fix for data transformation errors
• AI Summaries for Charts and Analytics
• Enhancements to Summary View, Charts, Tables, and Performance

Below is a quick overview of the new features in Exploratory v15.

Step Diagram: Visualizing Data Transformation Pipelines

Exploratory v15 introduces the Step Diagram, which allows you to visually trace the flow of data transformations from charts, dashboards, and reports all the way down to source data.

You can now easily understand:

• Where the data is sourced from
• How the data is transformed and lead to specific charts or KPIs
• Where and how data are combined (join & merge)
• Where data frame was branched off

The Step Diagram can also be accessed directly from charts and numbers (metrics) inside dashboards and notes.

Dashboard

Note

This makes it much easier to answer questions like:

“How exactly was this number created?”

AI Step Summary

We also added AI-generated Step Summaries, which explain data transformation steps in natural language.

For example:

• “Filling NA values with previous values after sorting data by purchase date”
• “Joining customer data with order data using Customer ID”
• “Aggregating sales by region”
• “Keeping only customers with more than two purchases”

This makes data transformation workflows easier to understand, even for users who are not familiar with underlying code or detailed transformation logic.

Automatic Documentation Generation for Data Preparation

Exploratory v15 can now automatically generate documentation for the entire data preparation process.

Using AI-generated summaries, Exploratory creates a structured document that includes:

• An overview of the entire workflow
• Descriptions of each transformation step
• Organize by grouped sections based on transformation purpose

The generated document can then be saved directly as an Exploratory Note, allowing you to edit, expand, and share it with your team.

This is especially useful for:

• project handoffs
• team reviews
• supplemental documentation for reports
• explaining analysis logic
• auditing and validation of data preparation workflows

Until now, dashboards and notes could be shared easily, but the context behind the data preparation process was often difficult to communicate because documenting it manually takes significant time.

With automatic documentation generation, you can now share not only the final result, but also the context behind how the result was created.

AI-Powered Auto Fix for Data Transformation Errors

From the very beginning of Exploratory, reproducibility has been one of the core design principles.

Exploratory records data transformations as step-by-step workflows (at the right-hand side) and automatically manages relationships between:

• transformation steps
• data frames
• charts
• analytics
• dashboards and notes

This enables reproducible data analysis workflows.

However, maintaining reproducibility is not always easy because real-world data changes constantly.

For example:

• column names changed in the source file
• values are presented in a different format, which causes data types change
• renamed columns or data type changes break downstream transformations

To help address these problems, Exploratory v15 introduces AI-powered Auto Fix for data transformation errors.

Rather than simply analyzing the error message itself, the AI reviews the surrounding data transformation context and suggests practical fixes while preserving the integrity of the transformation pipeline.

For example, it can:

• find similar column names and update column name references
• add appropriate data type conversion steps
• disable the step that causes the downstream error in the pipeline
• fix syntax errors in calculations

Users can review the suggested fixes and apply the appropriate fix.

This makes it easier to maintain reproducible and reliable data transformation pipeline even when there are accidental changes in source data or in the data transformation steps.

AI Summaries for Charts and Analytics

Analytics

Analytics view in Exploratory automatically generates analysis reports, but analytical outputs often contain a large amount of information (or, too much info!).

AI Summary helps users quickly identify the most important findings before diving into the detailed report.

This makes interpreting analysis results much more efficient.

Charts

AI Summary for Charts explains patterns, trends, and statistical signals in an easy-to-understand way.

For example, with XmR charts, AI Summary can automatically identify whether special-cause variation signals are present beyond expected variation.

This helps users notice important insights that might otherwise be overlooked.

Enhancements to Summary View

Interactive Filtering

You can now click bars directly in Summary View charts to keep or exclude related records interactively.

Outlier Analysis

We also added a new Outlier Analysis feature to Summary View.

This allows users to quickly visualize how outliers in one variable relate to other variables in the dataset.

Custom Charts

Exploratory v15 introduces Custom Charts, allowing users to write R code directly inside Chart View to create highly customizable visualizations.

By leveraging R’s rich visualization packages, users can now create charts beyond standard built-in chart types directly within Exploratory.

Drag-and-Drop Column Width Adjustment

You can now resize columns in pivot tables, summary tables, and regular tables using drag-and-drop.

This makes it easier to improve readability and optimize table layouts for dashboards and reports.

Performance Improvements

Exploratory v15 also includes major performance improvements.

Key areas include:

• Faster Exploratory startup
• Faster loading for Notes and Dashboards

Startup performance improvements are especially noticeable for users with many projects.

Additionally, Exploratory now upgrades its R foundation to R 4.5 and R packages.

One notable package upgrade is the dplyr v1.2, which significantly improves the performance of functions such as:

• case_when()
• if_else()

These functions are heavily used in Exploratory’s conditional calculation steps.

Try Exploratory v15

In the AI era, trustworthy analysis requires understanding how data was created, transformed, and analyzed.

Exploratory v15 is designed to support exactly that.

We hope you will try the latest version and experience the new capabilities yourself.

👉 Download Exploratory
​Download Exploratory

If you do not yet have an account, you can start a 30-day free trial.

👉 Start Your Free Trial
​Start Free Trial

That’s all for today.

If you have any feedback about Exploratory v15, please feel free to comment below!

We will continue working hard to improve Exploratory and make data analysis more trustworthy and accessible in the AI era.

Thank you!

Kan,
CEO/Exploratory

Exploratory v15 — The 3 Rs for Trustworthy Data Analysis in the AI Era was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Originally published on Exploratory. — Link.

We are entering a new phase of data science.

You don’t write most of the code anymore, but AI does.

At first glance, this feels like a liberation. For years, learning data science meant learning how to code in Python, R, SQL, etc. Now, you can simply describe what you want, and the code gets generated for you.

This makes you wonder…

If AI can generate the code, do we still need to learn programming like R or Python at all for data science?

Well, it depends…

The Real Problem: Answers without Trust

In many areas of machine learning we can evaluate our results against something concrete. There is a ground truth. A label.

And we can evaluate how well we are doing with a set of well-defined metrics.

For this type of data science, you may never have to look at the underlying code that builds prediction models, so what’s the point of learning how to code? You may ask.

However, when it comes to data analysis side of data science things are very different.

Because, there is no clear pre-defined answer that we can evaluate against at the end.

You are not verifying result against a particular answer, instead you are building an answer. You are exploring data in order to understand something that was previously unknown.

And this makes a difference between generating code for data analysis side of data science and for building prediction models side of data science.

In fact, the real difficulty in data analysis has always been how you build the answers you can trust, not how to write code.

In order to trust the result, you have to have trust in the process that produced the result. You have to be confident that each data transformation such as filter, calculations, aggregation, combine with other data, cleaning text data, etc. is doing exactly what you intended.

Not approximately.

“Close enough” is not enough in data analysis or building metrics or KPIs, it has to be ‘Exact and Accurate’ because people (or even agents!) will make decisions based on the result.

Readability, the Most Important Part of Programming Language

This leads to an uncomfortable but unavoidable truth:

If you cannot read the code, you cannot trust the result.

This is where Readability becomes the most critical feature when you choose programming language for data analysis.

In the context of AI generated code, people often talk about the accuracy of the code, how fast the performance, what is the cost, which AI model to use, etc.

But in the context of data analysis, none of these are as important as Readability.

From Readability to Trust

Before making further discussion, I want to make it clear.

When I say Readability, it is not just about aesthetics or personal preference. It is not about whether code looks “clean” or “elegant.”

It is about whether you can understand what the code is actually doing.

When AI generates code, it is easy to assume that the result is correct. But small details matter. A slightly different grouping, an unintended filter, a silent drop of missing values, any of these can change the outcome in meaningful ways.

If the code is readable, you can pause and inspect it. You can follow the transformation step by step and confirm that it aligns with your intention. The code becomes something you can reason about.

Here’s an example of R code that provides high readability.

sales %>%  
   filter(year >= 2020) %>%  
   group_by(region) %>%  
   summarize(total_sales = sum(sales))

It can be read as:

“Filters data for 2020 or later, and calculate total sales for each group of region.”

It’s readable, and you can understand what it’s trying to do.

If it is not readable, you are left with a different experience. You are not verifying the result, you are just trusting it faithfully. And in data analysis, blind trust is not a viable strategy.

Trust, then, is not something AI gives you.

It is something you build through readability.

From Readability to Reproducibility

But this is only the beginning.

In a business context, data analysis rarely happens only once. Data changes. Questions evolve. The same analysis needs to be run again next week, next month, or with slightly different conditions.

At that point, the question becomes:

“Can you reuse what you already did before?”

If you understand the code, the answer is simple. You run it again. And if needed, you adjust a condition, extend a calculation, etc. The original logic in data transformation and analysis remains intact, and the analysis evolves naturally over time.

If you don’t, it breaks.

You go back to AI. You describe the task again. The AI generates something similar, but may not be identical. Small differences creep in. Over time, you accumulate multiple versions of “the same” analysis that are, in fact, not quite the same.

This is the problem when you don’t have reproducible data analysis pipeline. And if you can’t reproduce you’ll lose trust quickly.

Again, the question is not whether you or AI can write, but it is whether the code, either generated by AI or written by you, is readable enough to be reused with confidence.

From Readability to Interactivity

There is one more dimension, and it is perhaps the most subtle.

Data analysis is not a linear process. It is not a sequence of predefined steps. It is a iterative process, a back-and-forth between you and the data, as if you are having a conversation with data.

You might begin with something simple:

“Show me sales by year.”

But almost immediately, your thinking moves forward.

-> What about monthly trends?

-> What about regional differences?

-> What about repeat customers?

-> Is there a correlation with marketing spend?

Each question generates an answer, which generates a new question, which generates an answer and so on. It’s an iterative process and the center of the process is you as a curious and creative being.

And you want this process to be fast, as fast as the speed of thought.

But when every step requires writing a prompt, waiting for AI to respond, reading the generated code, and then inspecting whether the result is correct, that flow is interrupted. The conversation slows down.

You are no longer exploring, you are coordinating with a tool.

However, something different happens when the code is readable.

You stop relying on prompts for every step. Instead, you begin to adjust the code directly. A grouping changes. A filter is added. A calculation is modified. The feedback loop tightens.

Now, the pace of analysis starts to match the pace of your thinking.

That is what true exploratory data analysis feels like.

It’s interactive and iterative.

The Shift We Often Miss

It is tempting to think that AI reduces the importance of programming.

In reality, it does the opposite.

It shifts the role of programming from writing code to understanding code, at least in a context of data analysis.

And once that shift happens, the most important property of programming language is no longer what it can do.

It is how easily you can read what it does.

Because that is what determines whether you can:

• trust the result
• reproduce the process
• and interact with the data in a meaningful way

And all of that begins in the same place.

Readability.

3 Technical Foundations of R That Enables Readability

If readability is the key to trust, reproducibility, and interactivity, then the next question is:

Which data science language provides high readability?

And the short answer answer is R.

R provides by far the most readable code. When I say R, I don’t necessary mean the base R, but R with ‘tidyverse’, which is a set of packages that most of the modern R practitioners use.

R’s tidyverse code resembles the description of the data operation in English as we have seen above.

sales %>%  
   filter(year >= 2020) %>%  
   group_by(region) %>%  
   summarise(total_sales = sum(sales))

But, the tidyverse didn’t become highly readable by itself. It was R’s unique foundation that enables it.

Clause Wilke has articulated this so well in his blog post when he talks about why R is better than Python.

• Python is not a great language for data science. Part 2: Language features

And among many reasons listed in the post, I think the following 3 characteristics of R are especially important for enabling R to be highly readable.

1. Separation of Logic vs Logistics
2. Built-in Vectorization
3. Support for Non-Standard Evaluation (NSE)

These three work together to determine whether code feels like a clear expression of thought or a set of instructions for a machine.

In order to highlight these unique features, as Clause does in his blog post, I’m going to compare R against Python. Python is often used by most of the AI models when you ask data science / analysis related questions without specifying your preference of language.

1. Separation of Logic vs Logistics

Every piece of data analysis code contains two layers:

• Logic → What you want to do
• Logistics → How the computer executes it

For example:

“Calculate total sales by region for 2020 and after”

This is logic.

But implementing it may involve:

• loops
• indexing
• temporary variables
• restructuring outputs

This is logistics.

Example with R (Tidyverse)

Now, here is how you write in order to achieve the above with R (tidyverse).

sales %>%  
   filter(year >= 2020) %>%  
   group_by(region) %>%  
   summarise(total_sales = sum(sales))

This is almost a direct translation of the question:

• filter year >= 2020
• group by region
• summarize total sales

You see only the logic, almost no visible logistics

Example with Python (Panda)

Now, here is an example code with Python with Panda library to do the same.

sales[sales['year'] >= 2020] \
    .groupby('region') \
    .agg(total_sales=('sales', 'sum')) \
    .reset_index()

Still readable, but notice the logistics creeping in:

• sales['year'] → how to access column
• agg(...) → syntax detail
• ('sales', 'sum') → implementation detail
• .reset_index() → structural fix

These are not part of the question. They are part of the machinery.

If AI decided to write Python code without the panda library, this is how it would look like. You can see the true nature of Python:

result = {}

for row in sales:    
    if row['year'] >= 2020:        
       region = row['region']   
       if region not in result:     
           result[region] = 0  
       result[region] += row['sales']

Now we are fully in logistics mode:

• Manual loop
• Dictionary initialization
• Key checks
• Accumulation logic

This code does the same as the above R example, but it forces you to think like a machine.

Why This Difference Exists

This difference is not accidental. It comes from the design philosophy of the languages.

R (tidyverse) is designed for expressing intent. It uses data-centric grammar (verbs like filter, mutate, summarize, etc.) And this is only possible because the following features of R, which I’m going to talk about in the next sections.

• Built around vectorized operations
• Supports non-standard evaluation (NSE)

On the other hand, Python is designed for general-purpose programming. It’s built around objects and structures, and it requires explicit references like (df['column']).

It lacks native data-manipulation grammar. And even with a library like pandas, Python still carries its general-purpose DNA.

When logistics dominates,

• Code becomes harder to read
• Verification becomes harder
• Modification becomes slower

In R code, logic dominates. And this makes R code closer to human thinking.

2. Built-in Vectorization

Now let’s go one level deeper.

Even if you separate logic from logistics, you still need a way to apply that logic to data.

This is where vectorization comes in.

With vectorization, you can apply an operation to an entire column at once, instead of looping row by row.

Let’s say you want to make a simple calculation like “multiple sales by 1.1”.

With R, you can write something like this.

sales * 1.1

This applies to all values in the sales column (or variable).

No loop. No special function. Just the logic.

But in Python, you must iterate explicitly.

[s * 1.1 for s in sales]

With the pandas or the NumPy library it could be close to R.

df['sales'] * 1.1

In Python, vectorization is not part of the language itself unlike R. Instead, it is outsourced to libraries. Therefore, you need to deal with the difference among the libraries:

• Python lists → not vectorized
• NumPy arrays → vectorized
• pandas Series → vectorized (differently)
• Polars Series → vectorized (again differently)

Each has:

• different APIs
• different behaviors
• different edge cases

So you end up writing with the numpy package:

import numpy as np

sales = np.array(sales)
sales * 1.1

Or, with the panda package:

pd.Series(sales) * 1.1

And this gets worse when you want to calculate conditionally.

Let’s say if you want to calculate something like this.

“If sales > 100, apply 10% discount”

In the case of R, it’s pretty simple.

ifelse(sales > 100, sales * 0.9, sales)

You have the following three.

• condition
• calculation
• fallback

But in the case of Python, it becomes convoluted.

[s * 0.9 if s > 100 else s for s in sales]

Now:

• explicit iteration
• inline conditional logic
• more cognitive load

Even with pandas package,

df['sales'] = np.where(df['sales'] > 100, df['sales'] * 0.9, df['sales'])

It’s better, but still:

• referencing columns repeatedly
• using external function (np.where)
• mixing libraries (pandas + NumPy)

Why this matters

Vectorization is not just about performance. It’s about thinking at the right level of abstraction.

When vectorization is built into the language:

• You think in columns and transformations

When it is not:

• You fall back to rows and loops

3. Non-Standard Evaluation (NSE)

Now we reach the most important layer.

Even if you can think in logic and can operate on vectors, you still need a way to express your idea naturally.

This is where NSE (Non-Standard Evaluation) comes in.

What is Non-Standard Evaluation?

NSE (Non-Standard Evaluation) means that you can write code that looks like you are directly operating on the data, even though the data lives inside a data frame.

In simpler terms, you can refer to columns as if they were normal variables.

R has NSE support, so you can specify the column (variable) names in functions like this.

df %>%
  mutate(profit = sales - cost)

You can read this code literary as:

• profit equals sales minus cost

No quotes, no indexing, and no function around the column name. You simply type column names as if you write in a report.

On the other hand, Python doesn’t have NSE, hence the column (variable) needs to be specified as below.

df['profit'] = df['sales'] - df['cost']

Notice that

• Columns must be accessed as strings ('sales')
• Repeated df[...]

This may seem small, but it becomes worse as we expand.

Example 1: Grouping and Summarizing

Let’s say you want to:

“Group by region, calculate average sales”

In R, you will write like the below.

df %>%
  group_by(region) %>%
  summarise(avg_sales = mean(sales))

This can be easily read as ‘group by region, calculate average sales’.

But with Python, you will need to write like this.

df.groupby('region')['sales'].mean().reset_index(name='avg_sales')

• 'region', 'sales' → string references
• chaining syntax tied to structure
• .reset_index() → logistics

This is when non-programmers start losing it.

And it gets even worse.

Example 2: Sorting based on Calculation

Let’s push further. If you want to do something like:

“Sort by cosine of sales”

With R, it’s pretty simple.

df %>%
  arrange(cos(sales))

With Python, not so much.

import numpy as np

df.assign(
    cos_sales=lambda d: np.cos(d['sales'])
).sort_values('cos_sales') \ 
 .drop(columns=['cos_sales'])

Now we see:

• temporary column
• lambda function
• repeated references
• cleanup step

We’re not comparing the capability between R and Python. Both codes can do the same job. But which one would you be more comfortable when you are accountable for the produced result?

NSE turns code into a language for expressing ideas, rather than constructing the computational instruction.

Bringing It All Together

These three principles are not independent. They reinforce each other.

When all three are present,

• You express logic, not logistics
• You operate on data as a whole
• You write expressions, not instructions

The result is the code that reads like the question you are asking.

The Shift in the AI Era

In the past, programming was about writing code. And because humans wrote that code, readability mattered, primarily as a matter of maintainability and collaboration.

But today, we live in a different time.

AI writes a significant portion of the code we use, and that changes the role of the human entirely.

You are no longer the author or even the maintainer of the code. You are the reviewer of it.

You didn’t write it. You didn’t think through each line as it was constructed. And yet, you are responsible for deciding whether the result produced by the code is correct.

That responsibility cannot be delegated to AI or others.

And this is where the hierarchy of importance flips.

Readability is no longer a “nice-to-have.” It becomes the foundation upon which everything else depends.

Because only when code is readable can you:

• verify its correctness
• trust the result
• reuse the logic
• and modify it to satisfy ever evolving questions you have.

Reliability, reproducibility, and interactivity are not separate concerns.

They are all downstream of one thing.

That’s Readability.

Pick the Language That Gives You Readability

If your goal is to work effectively with AI-generated code, you need a language where:

• the code reflects the logic directly
• transformations are easy to follow
• small changes can be made without friction

This is where R, especially with the tidyverse, stands out.

Not because it can do something others cannot.

But because it expresses ideas in a way that is intuitively understandable.

When AI generates R code, what you see is not machinery, you see intent.

And that makes all the difference.

Why This Matters for Exploratory

By the way, this perspective is not something we arrived at recently.

It is, in fact, the original reason we built Exploratory in the first place.

About 10 years ago, when we started Exploratory, we made a deliberate decision: build UI on top of R and the tidyverse.

• Introducing Exploratory Desktop — UI for R

At the time, this was not the most obvious choice to many. But we believed something that, in hindsight, has only become more true over time.

Data analysis is not just about capability.

It is about how easily humans can understand and work with that capability.

We saw that the R & dplyr (a part of tidyverse) offered something unique.

Not just a set of powerful functions, but a way of expressing data transformations that was readable, consistent, and aligned with how we think. That readability, combined with flexibility, made it ideal for exploratory data analysis.

So we built an UI experience on top of it.

The goal was simple: to make the power of R accessible to non-technical users without a need of writing code.

That was the beginning of Exploratory.

What is the bottleneck for data analysis today?

10 years later, things have changed dramatically.

It is now 2026, and we are deep in the era of AI and LLM (large language models). Writing code is no longer the hard part. AI can write R, Python, SQL — almost instantly and entirely.

And this begs a question.

If writing code is no longer the bottleneck, then what is?

We started asking our customers,

What is the bottleneck for data analysis today in the era of AI?

And the answers were surprisingly consistent.

The bottlenecks were:

Reliability and Reproducibility.

This brings us back to the core idea of this post.

If reliability and reproducibility are the problems, then how can we solve?

And the answer, as we’ve seen, is readability that is supported by the right interface — AI prompt and UI.

This is where Exploratory fits in, especially in the AI era.

Data Analysis in Exploratory

In Exploratory, you can type in how you want to transform and prepare data in the prompt UI, then AI generates R (tidyverse) code. Not as something hidden behind the scenes, but as something you can see, read, and understand.

You can read the code with a help of the code description to see what it does and what you can expect from it.

But it doesn’t disappear into somewhere it’s hard to find like Excel where the calculations and transformations are spread across a bunch of cells.

It becomes part of a visible, inspectable data wrangling step.

Each step is preserved as part of a pipeline — a sequence of data transformations that you can revisit, modify (such as changing the data source files or underlying SQL queries, updating calculations, changing the data filter conditions, etc.), and rerun at any time.

You can see the updated data visualized by Summary view where each column shows its summary information with automatically generated chart and metrics.

Or, simply use Chart view where you can visualize the data with various chart types and chart features with point-and-click.

This makes a powerful data exploratory experience, which is highly visual, iterative, and interactive.

Over time, this changes how you work.

You are no longer repeatedly asking AI to regenerate the same logic. You are refining and evolving an analysis that you understand.

When new data arrives, you don’t start over. You update the source step and rerun the process.

When a new question emerges, you don’t need to start from scratch. You add a new step and continue the flow.

At this point, your analysis becomes:

• Reliable, because you can verify every step
• Reproducible, because the process is preserved
• Interactive, because you can move at the speed of your thinking

And this is not simply because AI is generating code.

It is because that code is readable, and because the environment is designed to support that readability.

Final Thought

If there is one idea to take away from all of this, it is this:

The biggest shift in the AI era is not who writes the code. It is who is responsible for understanding the code AI generates in the world of Data Analysis.

In the past, you wrote the code, so you naturally understood it.

Today, AI writes the code.

Which means understanding no longer comes for free.

It becomes something you must actively regain.

And the only way to do that is through readability.

When code is generated automatically, the value no longer lies in producing it.

The value lies in being able to read it instantly, verify it confidently, and adapt it flexibly.

That is why programming itself hasn’t become less important.

It has simply changed its role.

From:

writing code

To:

understanding and validating code

Therefore, in this AI era, the choice of language is not about capability alone, but it’s about readability.

It is about whether you can understand what has been generated quickly.

Because that is what helps you trust the result.

That is what makes it reusable.

That is what makes it interactive.

And that is why R, especially tidyverse, feels increasingly aligned with the future of data science.

Try Exploratory!

If this idea resonates with you, that in the AI era, the ability to read and understand code matters more than ever, then the best way to experience it is to try it yourself.

In Exploratory, you can describe how you want to transform your data, and AI will generate readable R (tidyverse) codefor you. Not as a black box, but as something you can inspect, understand, and build upon.

You’ll quickly notice the difference.

Instead of repeatedly asking AI to regenerate answers, you begin to work with the analysis itself, verifying it, refining it, and evolving it over time.

👉 Download Exploratory

https://exploratory.io/download

If you don’t have an account yet, you can sign up here and start a 30-day free trial:

https://exploratory.io/

If your trial has expired but you’d like to try the latest AI features, simply launch the newest version and use the Extend Trial option.

If you have any questions or feedback, feel free to reach out directly: kan@exploratory.io

I’d love to hear how you’re using Exploratory to better understand your data.

Kan Nishida

CEO, Exploratory

In the Age of AI, R’s Readability Becomes a Superpower was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the past decade, the world of machine learning and AI has been dominated by one idea: deep learning.

Neural Networks — the algorithms behind deep learning — has transformed fields like:

• computer vision
• speech recognition
• natural language processing

Neural networks have transformed fields like image recognition, speech processing, and natural language understanding. Models such as transformers and convolutional neural networks now power everything from ChatGPT to self-driving cars.

Given that success, it seemed inevitable that deep learning would replace traditional machine learning everywhere.

But something interesting happened.

In the world of tabular data, the structured datasets used by most businesses, deep learning never completely took over.

Instead, algorithms like XGBoost and LightGBM continue to dominate many real-world machine learning applications.

This sounds surprising to some. But once you understand the nature of tabular data and what deep learning is good at (and not good at), it starts to make sense.

Most Businesses Data Are Tabular Data

Most organizations are not training AI models on billions of images or internet-scale text corpora.

Instead, they work with data that looks something like this:

Columns represent different concepts:

• age
• income
• purchase behavior
• geographic region

Unlike images or language, there is no inherent structure connecting these variables.

Images have spatial structure

Language has sequential structure

But, tabular data is different. It’s simply a collection of variables describing some phenomenon. And it doesn’t have the kind of structure that images and text have.

Tabular data

Deep learning thrives when data has hierarchical structure (images, language).

Think of the modern Transformer, which is one of the most important algorithms in deep learning and AI today. When it builds a model on a given text data, it takes the sequence of text and the relation between words, sentences, etc. into account, and predicts the next word.

But, tabular data typically does not have such sequence and structure. Each value in a given variable is typically independent from other values.

And it turned out that this difference matters a lot.

Why Tree Based ML Models Work So Well

Machine Learning models like XGBoost, LightGBM, Random Forest, etc. are still the most common algorithms among practitioners who are building prediction models with business data, and they all share a common fundamental architecture.

That is Decision Trees.

Decision trees approach the problem in a way that feels very natural for this type of data.

Instead of learning abstract representations, they learn rules.

For example:

Or:

These kinds of conditional rules often show up in real-world datasets, and Decision trees are extremely good at discovering them.

The Rise of XGBoost

Around the mid-2010s, one algorithm in the tree family became very popular.

That algorithm was XGBoost.

XGBoost implemented gradient boosting in a highly optimized way and quickly became the default choice for many machine learning practitioners working with tabular data.

It builds trees sequentially, each one correcting the mistakes of the previous model.

For several years, it dominated the eyes of data science practitioners who were building prediction models for business data.

But as datasets grew larger, people began to encounter a new problem.

Training these models could take a long time.

Enter LightGBM

In 2017, researchers at Microsoft introduced a new boosting framework called LightGBM.

The goal wasn’t to reinvent gradient boosting.

Instead, the idea was to make boosting lighter.

The word Light in LightGBM refers to being lightweight in computation and memory usage.

Several clever design decisions helped achieve this:

• trees grow leaf-wise, focusing computation on the most useful splits
• features are converted into histogram bins to reduce split evaluations
• rows with large gradients are prioritized using GOSS
• sparse features are compressed using exclusive feature bundling (EFB)

Level-wise vs Leaf-wise growth

Level-wise (XGBoost)

Leaf-wise (LightGBM)

LightGBM grows trees where the loss decreases most, focusing computation on the most informative parts of the model.

Together, these ideas dramatically reduce the amount of computation required to train models.

For people experimenting with machine learning models by trying many features and hyper-parameters, this speed improvement made a huge difference.

Why Deep Learning Often Struggles?

It’s not that people stopped trying, in fact many researchers have been trying applying neural networks to tabular datasets.

But very often, Decision tree based boosting models such as XGBoost, LightGBM, still performed better.

The reason is surprisingly simple.

Deep learning excels when the data contains rich internal structure.

Images contain spatial patterns. Language contains grammatical patterns.

Tabular data usually does not.

Instead, tabular datasets often contain:

• diverse mix of variables
• engineered features (artificially created extra variables based on original variables)
• sparse categorical encodings (one-hot encoding)
• nonlinear feature interactions

Decision trees are well suited to discovering these kinds of patterns.

What This Means in Practice

In many real-world machine learning projects, the workflow often looks like this:

1. Build a baseline model.
2. Try a tree boosting algorithm.
3. Improve the model through feature engineering and parameter tuning.

And very often, the models that end up performing best are based on boosting algorithms such as XGBoost, LightGBM, etc.

These algorithms have become reliable workhorses for tabular data problems.

Trying These Models Yourself

One of the motivations behind building Exploratory was to make various data science tools easier to use, and building prediction models with machine learning models is one of them.

In Exploratory, you can train tree based models such as:

• Random Forest
• XGBoost
• LightGBM

directly from an interactive interface.

You can take a look at this how-to note for more details on how to use LightGBM.

Instead of writing large amounts of code, you can focus on exploring your data, building features, and comparing models.

If you work with tabular data such as customer behavior, financial data, operational metrics, etc., it’s definitely worth trying them.

A Final Thought

As it turned out, the deep learning revolution didn’t eliminate traditional machine learning algorithms.

Instead, it clarified something important.

Different types of data require different tools.

For images and language, as you know, deep learning dominates.

But, for tabular data, tree based algorithms like XGBoost and LightGBM remain some of the most powerful methods available.

One of the things I appreciate about boosting algorithms like XGBoost and LightGBM is how practical they are.

You don’t need massive infrastructure or complicated neural network architectures. You start with your data, build some features (if required), and let the model discover useful patterns.

In many cases, the results are surprisingly good!

In Exploratory, you can train models such as Random Forest, XGBoost, and LightGBM directly from an interactive interface and compare their performance on your dataset.

Sometimes the easiest way to understand the strengths of these models is simply to see how they perform on your own data.

Download Exploratory

You can start using XGBoost, LightGBM, Random Forest, and other models today in the latest version of Exploratory.

👉 Download Exploratory

https://exploratory.io/download

If you don’t have an account yet, sign up here to start your 30-day free trial.

https://exploratory.io/

If your trial has expired, simply launch the latest version and use the Extend Trial option.

If you have questions or feedback, feel free to contact me at kan@exploratory.io .

We’d love to hear how you’re using Exploratory to uncover insights in your data.

Kan Nishida

CEO, Exploratory

Why Deep Learning Didn’t Replace Tree Models for Tabular Data was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you work with tabular data (table data), the kind of structured data found in business analytics, finance, marketing, or operations, you’ve probably encountered three popular machine learning algorithms:

• Random Forest
• XGBoost
• LightGBM

All three rely on decision trees, and they can all produce very strong predictive models. But they were designed with different priorities in mind.

Random Forest emphasizes simplicity and robustness. XGBoost focuses on highly optimized gradient boosting. LightGBM was created to make boosting faster and more scalable.

So, which one to choose?

It depends…

And this is why I wrote this blog post.

Understanding why LightGBM was created and how it works makes it much easier to decide whether it’s the right tool for your problem.

The Problem LightGBM Was Designed to Solve

By the mid-2010s, gradient boosting had already proven to be one of the most powerful techniques for predictive modeling on tabular data, or structured data if you will.

In particular, XGBoost had become extremely popular after dominating many machine learning competitions.

However, as datasets continued to grow, practitioners began to encounter new challenges.

The performance.

Datasets with millions of rows become normal, and variables (or features) grow thousands, which caused sparse features produced by one-hot encoding.

Gradient boosting was powerful, but it could also be computationally heavy. This means that it takes time to build models with boosting algorithms when the data size is big.

Researchers at Microsoft set out to redesign parts of the algorithm so it could handle large datasets more efficiently.

The result was LightGBM, and they released it as open source in 2017.

Why Is It Called “LightGBM”?

LightGBM stands for ‘Light Gradient Boosting Machine’.

The word “Light” does not refer to the speed of light.

Instead, it refers to the algorithm being lightweight in computation and memory usage.

The goal was to create a gradient boosting system that could:

• train faster
• use less memory
• scale to larger datasets

while still maintaining strong predictive performance.

Before diving into LightGBM’s innovations, it helps to understand how the three algorithms differ conceptually.

Random Forest: Many Independent Trees

Random Forest builds many trees independently.

Each tree:

1. samples the dataset randomly
2. selects variables (or features) randomly when splitting
3. produces its own prediction

The final prediction is simply the average (regression) or majority vote (classification).

Key idea is that many independent trees reduce variance and improve stability compared to a single tree (Decision Tree).

But the trees do not learn from each other.

XGBoost: Trees That Correct Mistakes

XGBoost uses a technique called gradient boosting. Instead of building independent trees, it builds trees sequentially to correct the errors made by the previous trees.

Boosting algorithms are really performing a form of gradient descent in function space.

It builds the first tree and predict and calculate the errors (or loss). If it was a regression problem then the errors can be the residual between the actual and predicted values. And this is called ‘Gradient’.

The next tree will be built to predict the gradient values, not the actual values because the gradient tells the model how to move predictions to reduce loss.

And combining the predicted value from the first tree and the predicted values from the second tree will become the new predicted values as a model.

prediction_new = prediction_from_first_tree + learning_rate × prediction_from_second_tree

Conceptually, the model evolves like this:

• Tree1 → initial prediction
• Tree2 → fix errors from Tree1
• Tree3 → fix remaining errors
• Tree4 → continue improving

This process usually leads to more accurate models than Random Forest.

However, as datasets grow larger, training boosting models can become computationally expensive.

That is where LightGBM comes in.

LightGBM: Designed to scale

LightGBM does not change the basic idea of gradient boosting.

Instead of optimizing only the boosting algorithm, LightGBM also optimizes:

• how trees grow
• how rows are sampled
• how splits are evaluated
• how features are represented

to scale to very large datasets while maintaining strong accuracy.

These improvements come from four key innovations:

• Leaf-wise tree growth
• Histogram-based splitting
• GOSS (Gradient-based One-Side Sampling)
• EFB (Exclusive Feature Bundling)

Let’s walk through them one by one.

1. Leaf-Wise Tree Growth

One of the most distinctive features of LightGBM is leaf-wise tree growth.

Traditional tree algorithms such as Random Forest and XGBoost grow trees level-wise.

At each depth of the tree, all nodes are expanded.

Example:

        Root
       /    \
      A      B
     / \    / \
    C   D  E   F

Every level of the tree expands evenly, and it produces balanced trees, which are stable and predictable.

But, they may waste computation expanding branches that do not significantly improve predictions.

LightGBM takes a different approach. Instead of expanding all nodes at the same depth, it expands the leaf that produces the greatest reduction in loss.

Example:

        Root
       /    \
      A      B
     / \
    C   D
   /
  E

The tree grows where the model improves most.

This approach allows LightGBM to reach strong predictive performance with fewer splits.

The trade-off is that trees can become deeper in certain branches, so LightGBM provides parameters such as max_depth and num_leaves to control model complexity.

2. Histogram-Based Splitting

Another important optimization in LightGBM is histogram-based splitting.

Standard tree algorithms may evaluate many possible split thresholds for continuous features.

Example:

Age ≤ 21
Age ≤ 22
Age ≤ 23
Age ≤ 24

LightGBM speeds this up using histogram binning.

Instead of evaluating every unique value, continuous features are grouped into bins.

Example:

Original values:

23, 25, 27, 29, 35

Converted to bins:

20–25
25–30
30–40

Now the algorithm evaluates splits only on bin boundaries.

This dramatically reduces the number of candidate splits and speeds up training.

3. GOSS (Gradient-Based One-Side Sampling)

Training boosting models on large datasets normally requires processing all rows.

LightGBM introduces GOSS (Gradient-based One-Side Sampling) to reduce the number of rows used during training.

The key idea is based on how boosting works.

In boosting algorithms, each data point has a gradient value indicating how much the model needs to adjust its prediction.

• Rows with large gradients represent predictions where the model is making large errors.
• Rows with small gradients are already well predicted.

While typical boosting algorithms including XGBoost randomly sample the data, LightGBM uses this gradient information to sample the data.

• Keeps all rows with large gradients
• Keeps only a subset of rows with small gradients

Example:

Dataset: 100,000 rows

• Top 20% largest gradients → keep all (20,000 rows)
• Remaining 80% → sample 10% (8,000 rows)

This wya, it need to use only 28,000 rows instead of 100,000 rows.

This significantly reduces computation while preserving important learning signals.

4. EFB (Exclusive Feature Bundling)

Many modern datasets contain high-dimensional sparse features, especially when categorical variables are one-hot encoded.

Example:

These features are mutually exclusive, only one can be active in each row.

Instead of treating them separately, LightGBM bundles them into one feature.

Original:

Bundled:

Now the algorithm evaluates splits on one feature instead of three.

This reduces feature dimensionality and speeds up training.

How to Choose Each Model?

Each algorithm has strengths.

Random Forest

Good when:

• you want a simple baseline
• datasets are relatively small
• minimal tuning is preferred

XGBoost

Good when:

• datasets are moderate in size
• you want strong predictive performance
• stability and extensive tuning options are important

LightGBM

LightGBM works particularly well when:

• datasets are large
• feature dimension is high
• features are sparse
• training time matters

Practical Recommendation

A common workflow in machine learning projects is:

1. Start with Random Forest as a baseline.
2. Try XGBoost or LightGBM to improve performance.
3. Prefer LightGBM when datasets become large or training time becomes a bottleneck.

Because of its efficiency and scalability, LightGBM has become a popular choice for many machine learning tasks with tabular data.

Final Thought

Random Forest, XGBoost, and LightGBM all rely on decision trees, but they represent different philosophies:

• Random Forest focuses on robust ensembles
• XGBoost focuses on optimized gradient boosting
• LightGBM focuses on efficient and scalable boosting

Understanding these differences helps you choose the right tool, and explains why LightGBM has become an important algorithm in modern machine learning.

Try LightGBM with Exploratory!

You can try LightGBM with Exploratory v14.5 or later versions.

• Go to Analytics view.
• Select LightGBM.
• Select a Target Variable.
• Select Explanatory Variables (Features)
• Click Run button.

You can take a look at this how-to note for more details on how to use LightGBM.

Download Exploratory

You can start using LightGBM today in the latest version of Exploratory.

👉 Download Exploratory v14

https://exploratory.io/download

If you don’t have an account yet, sign up here to start your 30-day free trial.

https://exploratory.io/

If your trial has expired but you’d like to try the new features, simply launch the latest version and use the Extend Trial option.

If you have questions or feedback, feel free to contact me at kan@exploratory.io .

We’d love to hear how you’re using Exploratory to uncover insights in your data.

Kan Nishida
CEO, Exploratory

LightGBM Explained: How It Differs from Random Forest and XGBoost was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Showing values directly on a chart is incredibly powerful.

Your audience first sees the pattern in the data. Then they see the exact labels and numbers behind the pattern.

That combination is often what makes a chart insightful.

Most charting systems place labels exactly at the coordinates of the data point.

For small datasets, this works perfectly.

But when you have many data points that are close to each other, your chart would become something like this.

Instead of helping the reader, the labels start fighting each other.

Typical problems appear immediately:

• Labels overlap each other
• Labels cover the data points
• Values become unreadable
• Analysts start manually adjusting labels

What was supposed to clarify the chart ends up making it harder to read.

This problem is known as label overlap or label collision.

And it turns out to be surprisingly difficult to solve.

The Idea: Let Labels Move (Just a Little)

In Exploratory v14.3 we introduced Auto-Positioning for Labels.

The idea sounds simple:

Instead of fixing labels exactly on the data points (or coordinates), allow them to move slightly until they no longer overlap.

But there are a few important constraints.

The labels must:

• Avoid overlapping each other
• Avoid being overlapped by marker (e.g. line, bar, etc.)
• Stay visually close to the original point
• Indicate clearly which data point it represents
• Preserve readability

When the system finds a better layout, the labels reposition automatically.

If a label moves away from its original point, a leader line (arrow) connects the label back to the data point.

The result is a chart that remains readable even with many labels.

Our First Attempt: Physics Simulation

This is a solved problem in R with a package called ‘ggrepel’, but while Exploratory is built on top of R system we use Plotly JS (Java Script) for chart rendering. So we needed to build an auto-positioning system in JS layer.

So, our first approach was to use d3-force, a physics simulation library in JS.

The idea was appealing.

Labels repel each other like particles with repulsive force.

Eventually they should settle into positions where nothing overlaps.

In theory.

In practice, it didn’t work.

When many values are densely packed the labels tend to push each other away and scatter outside the chart area or across the entire screen.

So we gave up on that approach.

The Algorithm That Worked: Simulated Annealing

Instead, we adopted an algorithm called Simulated Annealing, which was used in D3-Labeler.

This is a classic optimization technique inspired by metallurgy.

When metal cools slowly, its atoms settle into a stable structure.

Simulated Annealing follows a similar idea:

1. Start with the current label layout
2. Move labels slightly in random directions
3. Evaluate whether the layout improves
4. Gradually reduce the randomness over time

After many small adjustments, the system converges toward a layout with minimal overlaps and good readability.

The key advantage is that labels stay close to their original positions, rather than flying across the chart.

Engineering the System

To integrate this into Exploratory, we created a new module that wraps the chart rendering process.

The workflow looks like this:

1. Exploratory renders the chart normally
2. The wrapper module analyzes the label positions
3. The auto-positioning algorithm runs
4. Labels are repositioned if collisions are detected

This adjustment happens automatically after the chart rendering.

Keeping It Fast (Even with Hundreds of Labels)

Optimization algorithms can become slow when the number of labels increases.

To keep performance smooth, we introduced spatial grid partitioning.

Instead of checking collisions between every pair of labels, the system divides the chart into small grid cells.

Labels only need to check nearby neighbors inside the same grid.

This dramatically reduces the number of collision checks.

As a result, the system remains responsive even with 500+ labels.

Additional Improvements

We added a few additional features to make the system more practical.

Leader Lines

If a label moves far from its original point, a leader line automatically appears.

This maintains the visual connection between label and data point.

Error Bar Awareness

Bar charts with error bars introduce another challenge.

Labels must avoid overlapping the confidence interval lines.

The algorithm takes error bar ranges into account when calculating positions.

How to Enable Auto-Positioning

Using the Auto-Positioning feature is simple.

1. Open the chart property dialog.
2. Click the gear icon at the top of the chart.
3. Then go to the Values tab.
4. Enable Show Values.

When the Position is set to Automatic, Exploratory automatically adjusts label positions to avoid overlaps.

If labels move far from their original point, arrows appear to indicate the connection.

You can control the color and the arrow visibility using the Arrow Display Threshold setting.

Increasing the threshold hides shorter arrows and reduces clutter.

For a newly created charts, when you enable to show the values (labels) on chart the auto-positioning is automatically set by default. For existing charts that you created before v14.3, you want to manually switch the Position to Automatic.

Improving Placement Accuracy

You can also control the optimization effort.

The setting ‘# Tries to Improve Accuracy’ determines how many optimization trials are performed.

• Increase the value → more accurate placement (slower)
• Decrease the value → faster calculation (less precise)

A Small Feature That Makes Powerful Data Exploration

In Exploratory v14.3 we introduced this auto-positioning system using Simulated Annealing and spatial optimization.

At first glance, this might look like a small feature.

But in practice, it changes something fundamental about how you explore data.

Exploratory Data Analysis is not just about creating charts.

It is about discovering things you did not expect to see.

That kind of discovery often happens when you start looking closely at individual data points.

For example, imagine a scatterplot showing 190 countries.

At first, you might look for a particular country you are interested in.

But once the labels become readable, something else begins to happen.

You start to notice patterns you did not plan to look for.

• Which countries are close to each other?
• Which countries behave similarly?
• Which countries stand apart from the rest?

These moments of discovery are the essence of exploratory data analysis.

However, if the labels overlap and become unreadable, those discoveries become much harder.

Analysts end up spending time manually adjusting labels instead of exploring the data itself.

Our goal with Exploratory has always been to build a tool that helps people think better with data.

Not by automating the thinking process, but by removing the friction that gets in the way of exploration.

Auto-positioning labels is one of those small features that quietly makes exploration easier.

And when exploration becomes easier, new insights often follow.

That is why we believe this feature helps make Exploratory a better environment for true Exploratory Data Analysis.

Try Auto-Positioning Today

You can start using the Auto-Positioning feature today in the latest version of Exploratory.

👉 Download Exploratory v14

https://exploratory.io/download

If you don’t have an account yet, sign up here to start your 30-day free trial.

https://exploratory.io/

If your trial has expired but you’d like to try the new features, simply launch the latest version and use the Extend Trial option.

If you have questions or feedback, feel free to contact me at kan@exploratory.io .

We’d love to hear how you’re using Exploratory to uncover insights in your data.

Kan Nishida

CEO, Exploratory

Auto-Positioning Labels: Keep Your Charts Readable Automatically was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.

This new feature works like having a professional editor — and a data analyst — right beside you as you write. Except it never gets tired, always respond fast, and can analyze your charts with deep statistical knowledge.

It’s designed to help you create high-quality analysis reports quickly, clearly, and effortlessly.

Why Reporting Is the Most Underrated (and Most Painful) Step in Data Analysis

In any real-world data analysis workflow, running the analysis is only half the job. The other half — often the harder half — is explaining what you discovered.

But let’s be honest: most of us don’t enjoy writing reports. Sometimes, we aren’t sure how to describe what charts are showing. So, we end up sending a Slack message with “here’s the chart” or copying & pasting charts in PowerPoint/Slides and hoping the audience magically understands it.

As a result:

• Screenshots pile up in Slack threads.
• PowerPoint slides become chart image dumps.
• Insights get lost because they’re never clearly communicated.

This is the communication problem in Data Science workflow that we wanted to solve with the new AI Note Editor.

Meet AI Note Editor: Turn Comments & Charts Into a Clear, Polished Report

With AI Note Editor, your analysis report writing workflow will become something like this.

1. Add your charts.
2. Write a few comments, if you like.
3. Get a complete data analysis reports generated.
4. Edit it as it fits your needs.
5. Share your report with others!

Based on your comments and charts, AI Note Editor generates a polished, structured, ready-to-share report for you, right inside Exploratory.

No switching tools.

No copy & paste to Power-Point.

No asking somebody else to create reports.

No staring at a blinking cursor trying to think of the right words.

Your data analysis stays where it belongs — inside the Exploratory workflow — and AI handles the writing.

Automatic Chart Interpretation

AI Note Editor doesn’t just improve your writing, it can also interpret your charts and explain what’s happening in the data. This is something that no general-purpose writing tool can do.

Give it a chart and the AI will:

• Identify key patterns
• Explain strengths and weaknesses
• Describe trends, anomalies, or outliers
• Summarize relationships between variables
• Highlight important signals
• Put everything into intuitive, natural language

For example, given a radar chart like the one below, AI will describe the key patterns, strengths, weaknesses, and overall story behind the data.

Detect Trends & Signals

Chart interpretation not only interprets numerical values but also provides context-aware interpretations, such as trends and signals within the data.

For example, with XmR charts (control charts), it can detect whether a signal is present and explain what it indicates accordingly.

AI Note Editor that comes with a deep statistical knowledge

Moreover, for charts containing statistical information, the AI Note Editor provides interpretations based on that statistical data.

For example, for a scatter plot showing the relationship between two variables, it can explain how strong (or weak) the correlation is, what the relationship implies, and whether it is statistically significant or not.

AI Note Editor Can Analyze Data and Explain What’s There

You’ve probably experienced something like these before:

• “I know what I see with the chart… but how do I put it into words?”
• “I think this trend is important, but I’m not confident how to explain it.”
• “I saw the chart, but I didn’t even notice it until someone pointed it out.”

AI Note Editor can address such concerns by:

• Helping you verbalize what the data shows
• Pointing out things you might have missed
• Ensuring your analysis is complete
• Reducing the risk of overlooking important signals
• Improving the clarity and quality of your reports — instantly

AI Note Editor is not just about writing faster. It’s about analyzing data better and communicating more clearly.

It can help you Write Better

AI Note Editor also includes a set of tools to improve your writing:

• Summarize long text
• Fix grammar and spelling
• Improve clarity and tone
• Refine wording and expression
• Translate your writing

Whether you’re writing an internal update, a weekly KPI brief, or a full analysis report, AI Note Editor makes the report writing process faster and better.

Create Your Own Report Format with Custom Prompts

Exploratory provides an “Analysis Report” style out of the box — but let’s be honest, not everyone writes reports the same way.

You have your own reporting needs and might need to write in different formats or style to fit your clients or audience’s needs by project to project.

That’s why AI Note Editor includes Custom Prompts support.

Just use ‘Run Custom Prompt’ and describe the format you want with Markdown instructing a specific structure with:

• Headings and subheadings
• Bullet points
• Summary sections
• Business-style executive summaries
• Step-by-step analysis
• Narrative storytelling
• Even templates that match your corporate writing guidelines

Just tell the AI the structure you want — and it will generate the full report in that format.

Save & Reuse Prompt Templates

Once you create a prompt you like, you can save it as a template.

Your templates appear in the Template list, ready to use anytime.

This is incredibly useful for reports you produce regularly — weekly reports, monthly summaries, recurring analyses, etc. Just pick a template and generate a polished report in seconds — consistent, clean, and always in the right style.

A Gallery of Prompt Examples to Get You Started

To help you get started, we have prepared a collection of prompt examples in the AI Note Editor Gallery. You can browse through the prompts, copy & paste them, and tweak however you like.

You can create sophisticated report formats immediately — even if you’ve never written a prompt before.

Share Templates with Your Team

AI Note Editor templates can also be shared across teams.

Your team members can directly import them by clicking the “Import” button from the “AI Prompt Template” dialog mentioned above.

This means:

• Standardized reporting
• Consistent communication style
• Faster onboarding for new members
• Reduced back-and-forth editing
• Higher-quality reports across the organization

A New Standard for Data Communication

AI Note Editor isn’t just a writing tool — it’s a communication tool with a deep statistical knowledge. It helps you:

• Turn analysis into narrative
• Turn charts into discoveries and insights
• Turn discoveries into stories
• Turn insights into action

No more copy & paste dumps in Power-Point slides.

No more staring at charts wondering how to describe what you see.

No more wasting time wondering what to write and how to explain.

AI Note Editor gives you clarity.

Your audience gets deeper understanding of your discoveries.

And your analysis becomes dramatically more impactful.

This is what the future of data communication looks like — and it lives directly inside Exploratory.

Try AI Note Editor Today!

You can start using AI Note Editor right now in the latest version of Exploratory.

👉 Download Exploratory v14
https://exploratory.io/download

If you don’t have an Exploratory account yet, please sign up here to try it out. The first 30 days are a free trial period!

If your trial has already expired but you want to try the new AI features, simply launch the latest version and use the “Extend Trial” option in the dialog — or contact us directly.

For questions or feedback, feel free to reach out:
📧 support@exploratory.io

We can’t wait to see the reports you create — and how AI Note Editor helps you communicate insights faster, clearer, and more confidently than ever before.

AI Note Editor: Create Reports 10x Faster, 10x Better was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Preparing a release note used to take a few hours every release. With AI Function, it now takes less than a few minute.

Every time we ship a new version, we go through dozens of GitHub issues — bug fixes, enhancements, and new features — and turn them into a release note that our users can easily understand.

This used to be a very manual process.

For each issue we had to:

1. Read the issue title and description
2. Decide whether it is a bug fix, enhancement, or documentation change
3. Assign it to a functional category (Data Wrangling, Chart, Analytics, etc.)
4. Rewrite the title so users understand what changed and why it matters

When there are 50–100 issues per release, this quickly becomes tedious.

Today, we automate most of this workflow using AI Function and AI Note Editor inside Exploratory.

In this post, I’ll show you exactly how we do it.

My hope is that this gives you ideas for how you can use AI Function to automate your own text-based workflows.

What is AI Function?

AI Function in Exploratory lets you create a function using a prompt so that AI (LLM) processes each row of your data.

You can ask AI to:

• analyze text
• classify information
• summarize content
• generate new text
• transform messy data

all directly inside your data workflow.

For example, if you have a column containing customer feedback comments, you can simply write a prompt like:

Classify the text into several groups.

AI will analyze each row and assign a category.

You can learn more about AI Function here.

• https://blog.exploratory.io/data-science-2-0-a-new-era-of-text-data-analysis-b6430baadba7

I personally use AI Function almost every day.

When we built this feature, I realized something interesting:

There was far more text data in my daily work than I had ever noticed before.

Issue logs.

Customer feedback.

Meeting notes.

Support conversations.

Task descriptions.

These are all valuable data sources, but without the right tools they’re difficult to analyze or automate.

AI Function changes that.

Example: Automating Our Release Notes

Let me show you a real example from our workflow.

We manage all development work in GitHub Issues.

These issues include:

• bug reports
• feature requests
• internal development tasks
• documentation updates

When we release a new version, we publish the closed issues for that milestone as the release note.

But raw GitHub issue titles are not written for users — they’re written for developers.

So we need to transform them.

Specifically, we need to:

1. Categorize issues by product area
2. Identify whether they are bug fixes or enhancements
3. Rewrite the title so users clearly understand the change

Before AI Function, I did this manually.

Now it’s automated.

Step 1: Import GitHub Issues

First, we import GitHub issues directly into Exploratory.

For example, we can import issues for the milestone v14.5.

Once imported, the dataset contains columns like:

• Issue title
• Issue body
• Labels
• Status
• Milestone

Take a look at this how-to note for details.

• How to Import Github Issue Data

Now that the data is imported, it’s time to work with AI Function.

Step 2: Categorize Issues by Product Area

First, we categorize each issue into a functional area.

For this, I create an AI Function with a prompt like this:

Based on the title and the body text, categorize text to one of the following groups.

AI Function
AI Prompt
Summary View
Table View
Data Source
Data Wrangling
Chart
Analytics
Note
Dashboard
Parameter
Project
Publish
Install
Document
Others

Instead of letting AI invent categories, I give it a predefined list.

This improves consistency.

I also provide two columns as input:

• Title
• Body

The title text and the body text would look like the below on the actual Github issue page.

This gives the model enough context

When executed, the AI analyzes the issue text and assigns an appropriate category for each row.

Step 3: Identify Issue Type

Next, we classify whether the issue is:

• a bug fix
• an enhancement
• documentation
• other

The prompt is simple:

Identify if a given sentence indicates whether it is an issue fix, a product enhancement, documents, or others.

Identify if a given sentence indicates whether it is an issue fix, a product enhancement, documents, or others.

This correctly categorize them into the right group.

Step 4: Rewrite the Issue Title

The final step is improving the issue titles.

GitHub issue titles are often short and technical.

For example:

AI Function: Don’t run the cached step when duplicating the data frame

That’s clear to developers, but not to most users.

We already built an internal system to improve and clean up the issue title and generate the Release Note, but we’ve realized that we can simply use AI Function to rewrite a better title for each issue based on the title and the body text

So I use AI Function again to generate a better description.

Prompt:

Based on the ‘title’ and ‘body’ text, write a one or two sentence summary that clearly explains what the issue is and why it matters to users.

In the AI Function dialog, I’m setting ‘title’ and ‘body’ columns as the Target Columns.

The result:

Original: 

AI Function: Don't run the cached step when duplicating the data frame

New:

Duplicating a data frame currently triggers re-execution of AI Function steps, which can cause unnecessary processing time and API costs.

Much clearer.

And this happens for every issue automatically.

That’s all!

Now that I have categorized the issues and improved the text I can show them in a table format.

From Raw Issues to Clean Release Notes

After these steps, we now have structured information:

• Issue category
• Issue type
• Improved description

This makes it easy to organize them into a release note.

Next Step: Generate Release Note with AI Note Editor

Once the data is prepared, we take it one step further.

We use AI Note Editor inside Exploratory to generate the release note itself.

Check out this introductory blog post for AI Note Editor.

• AI Note Editor: Create Reports 10x Faster, 10x Better

I’ll write another blog post explaining exactly how we do this, so stay tuned!

The Real Power: Reproducibility

The biggest benefit of this workflow is reproducibility.

Once the AI Function steps are created, we can reuse them.

What if the new issues have been added in the last minutes?

You can:

1. Re-import issues for the new milestone
2. Run the workflow

What if we will need to generate a release note for another version in future?

1. Click on the Data Source step and open the Import dialog
2. Update the milestone
3. Run the same workflow

We release new versions often, so being able to repeat the same workflow automatically is important for us.

In any future releases I can simply click a button to re-import data, and the AI Functions automatically process the new issues and produce clean, user-friendly descriptions.

No manual rewriting required.

This is the real power of combining:

• data workflows
• AI functions
• reproducibility

That is the real power of building an automated data wrangling system with AI Function that is reproducible and can be used with any future incoming data.

Why Not Just Copy and Paste the Issues into ChatGPT?

At this point, you might be thinking:

“Couldn’t I just copy the list of issues into ChatGPT and ask it to do the same thing?”

Yes, you certainly could.

But once you try to use that approach in a real workflow, several practical limitations quickly appear.

What makes AI Function inside Exploratory powerful is not just that it uses AI. It’s that AI becomes part of a data workflow.

Here are some key differences.

1. Iterative Workflow Development

When working with AI Function, you can build your workflow step by step.

For example:

1. Start with issue categorization
2. Review the results
3. Improve the prompt
4. Run it again
5. Add another AI Function step for issue type classification
6. Add another step to rewrite titles

Each step produces a visible column of results, which makes it easy to:

• review
• refine prompts
• improve output quality

This iterative workflow is much harder to do when you are simply pasting text into an AI chat interface.

2. Control Over the Output

When AI is part of your data workflow, you have much more control over the result.

For example, you can:

• specify allowed categories
• combine multiple columns as context
• inspect the output row by row
• filter and fix problematic cases

This is important if you care about quality and consistency, not just quick answers.

3. Reproducibility

This is perhaps the most important difference.

Once you build the workflow, it becomes reproducible.

Every time you import a new set of issues, you can run the exact same steps and get consistent results.

With copy-paste AI workflows, you would have to:

• prepare the text
• paste it into the AI
• rewrite prompts
• manually format the results

every single time.

AI Function turns that process into something you can run repeatedly with one click.

4. AI Works Directly on Your Data

Instead of copying and pasting text back and forth, AI Function works directly on the data frame in Exploratory.

This means you can combine AI with other data operations, such as:

• filtering rows
• joining datasets
• grouping and summarizing results
• visualizing patterns

AI becomes just another data transformation step inside the workflow.

5. Scales to Large Datasets

Chat interfaces are great for small tasks, but they quickly become cumbersome when working with hundreds or thousands of rows of data.

AI Function processes your data row by row, allowing you to apply the same logic consistently across the entire dataset.

6. Part of a Larger Data System

Finally, AI Function integrates with the rest of Exploratory’s capabilities:

• data wrangling
• visualization
• machine learning
• dashboards
• notes

In our case, the output of AI Function becomes the input for AI Note Editor, which then generates the release notes themselves.

This creates a complete pipeline:

GitHub Issues → AI Function → Enriched Data → AI Note Editor → Release Note

The Real Difference

Using AI in a chat interface is great for one-off tasks.

But AI Function allows you to turn those tasks into reusable data workflows.

And once that happens, something interesting occurs:

AI stops being a tool you occasionally use, and becomes part of your everyday data process.

That’s the real power of AI Function inside Exploratory.

Key Takeaways: Why AI Function Is a Game Changer

This example is not really about release notes.

It’s about something much bigger.

Many everyday workflows involve unstructured text data:

• GitHub issues
• customer feedback
• support tickets
• meeting notes
• task descriptions
• survey comments

Traditionally, these workflows required manual reading and interpretation, which made them difficult to automate.

That’s why many teams simply accept them as manual work.

But AI Function changes this.

Instead of writing complicated scripts or building custom AI pipelines, you can simply describe the task in plain language and apply it directly to your data.

In the release note example above, AI Function helped automate tasks that previously required manual effort:

• Categorizing issues by product area
• Identifying whether they are bug fixes or enhancements
• Rewriting issue titles into user-friendly descriptions

Once the workflow is built, it becomes reusable and reproducible.

Every new release can go through the exact same process automatically.

What used to be a repetitive manual task becomes a data workflow you run with one click.

And this idea applies far beyond release notes.

Anywhere you have rows of text data, AI Function can help you:

• classify
• summarize
• transform
• generate structured information

directly inside your data workflow.

This is why we built AI Function inside Exploratory.

Not to replace human thinking, but to remove the repetitive parts of working with text, so you can focus on the insights and decisions that matter.

AI Function turns messy text data into something you can actually work with, and once that happens, many workflows that used to be manual suddenly become automated.

Try AI Function with Your Own Data

If you work with text data such as:

• issue logs
• support tickets
• customer feedback
• meeting notes
• task lists

You can build similar workflows with AI Function.

If you’re new to AI Function, I recommend starting with the examples available in the Create AI Function menu.

Try AI Functions Today!

You can start using AI Functions today in the latest version of Exploratory.

👉 Download Exploratory v14

https://exploratory.io/download

If you don’t have an Exploratory account yet, please sign up here to try it out. The first 30 days are a free trial period!

If your trial has already expired but you want to try the new AI features, simply launch the latest version and use the “Extend Trial” option in the dialog, or contact us (support@exploratory.io) directly.

If you have any questions or feedback, please contact me at kan@exploratory.io

We’d love to hear what data wrangling system you build with AI Functions!

Kan

CEO/Exploratory

Turn GitHub Issues into Release Notes with AI was originally published in learn data science on Medium, where people are continuing the conversation by highlighting and responding to this story.