This page intentionally left blank. Inside front cover.

POWER QUERY BEYOND

THE USER INTERFACE
Solving Advanced Data
Cleaning Problems using M

by
Chandeep Chhabra

Holy Macro! Books

PO Box 541731
Merritt Island, FL 32953
Power Query Beyond the User Interface
Solving Advanced Data Cleaning Problems using M
© 2025 Tickling Keys, Inc.
All rights reserved. No part of this book may be reproduced or transmitted in any form or by any
means, electronic or mechanical, including photocopying, recording, or by any information or stor-
age retrieval system without permission from the publisher.
All terms known in this book known to be trademarks have been appropriately capitalized. Trade-
marks are the property of their respective owners and are not affiliated with Holy Macro! Books
Every effort has been made to make this book as complete and accurate as possible, but no war-
ranty or fitness is implied. The information is provided on an “as is” basis. The authors and the
publisher shall have neither liability nor responsibility to any person or entity with respect to any
loss or damages arising from the information contained in this book.
First Publication: April 2025
Author: Chandeep Chhabra
Copy Editor: Kitty Wilson
Tech Editors: Geert Delmulle
Indexer: Cheryl Lenser
Compositor: Bronkella Publishing
Cover Design: Chris Dorning
Published by: Holy Macro! Books, PO Box 541731, Merritt Island FL 32954
Distributed by Independent Publishers Group, Chicago, IL
ISBN 978-1-61547-081-5 (print) 978-1-61547-167-6 (digital)
Library of Congress Control Number: 2024951813
Version: 20241228

ii
 iii

Contents at a Glance
Introduction............................................................................................................................x
Chapter 0: Warming Up.......................................................................................................... 1
Chapter 1: Working with Lists................................................................................................15
Chapter 2: Working with Records..........................................................................................37
Chapter 3: Working with Tables.............................................................................................50
Chapter 4: Navigation............................................................................................................63
Chapter 5: Manipulating Between Lists, Records, and Tables.................................................70
Chapter 6: If...then...else.......................................................................................................80
Chapter 7: Iteration...............................................................................................................88
Chapter 8: Custom Functions...............................................................................................130
Chapter 9: Errors.................................................................................................................147
Chapter 10: Nested Tables...................................................................................................159
Chapter 11: Patterns and Recipes........................................................................................181
Chapter 12: Miscellaneous Tips and Tricks...........................................................................209
Solutions.............................................................................................................................220
Index...................................................................................................................................229
iv POWER QUERY BEYOND THE USER INTERFACE
Introduction............................................................................................................................x
Who This Book Is For.........................................................................................................................x
How to Use This Book...................................................................................................................... xi
Chapter 0: Warming Up.......................................................................................................... 1
Power Query’s User Interface.......................................................................................................... 1
What Is M, and Why Do We Need It?.............................................................................................. 2
Where Is M in Power Query?........................................................................................................... 4
M Syntax: A Few Basics.................................................................................................................... 6
Example 1: Writing a Simple M Query............................................................................................. 7
Example 2: Writing Another Simple M Query.................................................................................. 9
Feed Power Query What It Eats..................................................................................................... 11
Order of Evaluation........................................................................................................................ 13
Chapter 1: Working with Lists................................................................................................15
What Is a List?................................................................................................................................ 15
How to Create a List....................................................................................................................... 15
How to Extract a List from a Table.................................................................................................. 16
How Lists Behave........................................................................................................................... 18
Example 1: Expanding Dynamic Columns....................................................................................... 19
Example 2: Expanding Dynamic Columns, Part 2........................................................................... 25
Example 3: Dynamically Unpivoting Other Columns...................................................................... 26
Lists of Lists (a.k.a. Nested Lists).................................................................................................... 30
Example 4: Removing Errors from Multiple Columns (Using a List of Lists)................................... 31
Exercises......................................................................................................................................... 35
Chapter 2: Working with Records..........................................................................................37
Visualizing a Record........................................................................................................................ 37
Creating a Record........................................................................................................................... 37
Extracting a Record........................................................................................................................ 38
How Do Records Behave?.............................................................................................................. 39
Example 1: Summing Columns of a Table....................................................................................... 42
Example 2: Extracting an Intermediate Step from a Query............................................................ 44
Exercises......................................................................................................................................... 48
Chapter 3: Working with Tables.............................................................................................50
What Are Tables?........................................................................................................................... 50
How to Create a Table.................................................................................................................... 50
Cross-Join Example Using #table.................................................................................................... 53
What Can You Do with a Table (Beyond the UI)?........................................................................... 55
Chapter 4: Navigation............................................................................................................63
Navigation...Meaning?................................................................................................................... 63
Lookup Operators........................................................................................................................... 64
Exercises......................................................................................................................................... 68
 v
Chapter 5: Manipulating Between Lists, Records, and Tables.................................................70
Nesting Versus Transforming.......................................................................................................... 70
Nesting Example: Creating a Table from Records........................................................................... 72
Transformation Example: Removing Random Blank Values........................................................... 74
Exercises......................................................................................................................................... 78
Chapter 6: If...then...else.......................................................................................................80
The IF Syntax.................................................................................................................................. 80
Two if Examples.............................................................................................................................. 80
Using if in Applied Steps................................................................................................................. 82
Using List.Contains Instead of or.................................................................................................... 83
Using List.ContainsAll Instead of and............................................................................................. 85
Exercise.......................................................................................................................................... 87
Chapter 7: Iteration...............................................................................................................88
Iteration in English......................................................................................................................... 88
Understanding each _ (underscore)............................................................................................... 88
Nested Iterations Using List.Transform........................................................................................... 96
Understanding List.Accumulate................................................................................................... 101
Understanding List.Generate........................................................................................................ 120
Exercises....................................................................................................................................... 129
Chapter 8: Custom Functions...............................................................................................130
The Syntax.................................................................................................................................... 130
A Slightly Useful Function: Calculating PMT................................................................................. 131
Practical Example 1: Calculating Fiscal Quarters.......................................................................... 134
Practical Example 2: A Custom Function Like Excel’s TRIM.......................................................... 138
Exercises....................................................................................................................................... 146
Chapter 9: Errors.................................................................................................................147
Row-Level and Step-Level Errors.................................................................................................. 147
IFERROR in Power Query: try and otherwise............................................................................... 147
Using Only try............................................................................................................................... 151
try and catch: An Easier Way to Capture Errors........................................................................... 156
Exercises....................................................................................................................................... 158
Chapter 10: Nested Tables...................................................................................................159
What Are Nested Tables?............................................................................................................. 159
Nested Tables Example: Concatenating Customers...................................................................... 159
Nested Tables: Comparing Columns Between Expanded and Nested Tables............................... 162
Nested Tables: Determining Employee Stint Duration................................................................. 169
Customizing Local Grouping......................................................................................................... 173
Chapter 11: Patterns and Recipes........................................................................................181
Example 1: The Double Headers Problem.................................................................................... 181
Example 2: Removing Junk Rows and Combining Data................................................................ 190
vi POWER QUERY BEYOND THE USER INTERFACE
Example 3: Combining Data with Inconsistent Column Headers................................................. 198
Example 4: Capturing and Reporting Errors................................................................................. 205
Exercise........................................................................................................................................ 208
Chapter 12: Miscellaneous Tips and Tricks...........................................................................209
Power Query Keywords................................................................................................................ 209
Miscellaneous Power Query Tricks............................................................................................... 213
Solutions.............................................................................................................................220
Chapter 1: Working with Lists...................................................................................................... 220
Chapter 2: Working with Records................................................................................................. 222
Chapter 4: Navigation................................................................................................................... 223
Chapter 5: Manipulating Between Lists, Records, and Tables...................................................... 224
Chapter 6: if then else.................................................................................................................. 224
Chapter 7: Iteration...................................................................................................................... 225
Chapter 8: Custom Functions....................................................................................................... 225
Chapter 9: Errors.......................................................................................................................... 227
Chapter 11: Patterns and Recipes................................................................................................ 227
Index...................................................................................................................................229
Foreword vii

Foreword
One of the most flattering things for any teacher is for one of their students to become a teacher
in their own right and to start teaching you things. The book you are reading is an example of that.
When we wrote M Is for (Data) Monkey, Miguel and I set a goal to teach how to use Power Query to
solve common business problems. As you might have guessed from the original title, our first thought
was to explore the M language more thoroughly, but it became apparent to us as we began writing
that there was just so much that could be done via the user interface. In what was one of the first
books in the Power Query space, we decided that our priority was to focus on teaching people to
use Power Query to get the job done. While we did cover M, we filled our book with examples where
reaching past the user interface really wasn’t necessary. When we released Master Your Data, we
rewrote and expanded our M code section, but we stopped short of including extensive examples
of solving practical business problems that could not be solved via the user interface. Our intention
at that point was really to say “Hey, if you do want to dive into this language, here are the basics to
get you into the journey.”
Little did I know that in 2016, Chandeep would encounter Power Query for the first time, pick up
our book to start his journey, and eventually write a book to pick up where we left off. And honestly,
I could not be happier about this!
One of the greatest things about Power Query is that it gives you the ability to visualize the output
of your actions. Every action you take gets recorded, and you can preview the results in the output
window—at least until you can’t. And at that point, you need to be able to figure out your goal,
determine the syntax needed to achieve it, and visualize the results without being able to actually
see the intermediate values. I honestly think that is one of the most difficult jumps to make when
programming, whether with VBA, DAX, SQL, or M. Can you visualize the virtual tables or lists that
are being used to generate your output? How do you validate that your vision is correct?
One of the most important things that Chandeep does here is illustrate these intermediate steps
during his problem-solving journey. Getting to the end results is important, but understanding how
you got there is critical to being able to develop your own solutions in the future, and Chandeep’s
explanations will help you do exactly that. From the foundational concepts of tables, lists, and records
and how to move between them, to advanced concepts like List.Generate, Chandeep does a
great job of explaining the concepts in a layered way, using business-focused examples. He’s even
included practice sets and examples for you to hone your skills.
That’s enough from me, though. It’s now time for Chandeep to show truly that M is for (us) data
monkey(s). 😊
Ken Puls, FCPA, FCMA, MVP
Co-author of M Is for (Data) Monkey and Master Your Data with Power Query in Excel and Power BI
https://excelguru.ca, https://skillwave.training
viii POWER QUERY BEYOND THE USER INTERFACE

Dedication
Dedicated to the three most important women in my life: Daya, Manjeet, and Sonika.
Acknowledgments ix

Acknowledgments
A big THANK YOU to:
Ashutosh Zawar for everything that I’ve learned from him. Ken Puls for helping me write this book
from the very start. Darin Spence for consistently throwing nasty data cleaning problems at me.
Geert Delmulle for proving his valuable feedback. (This book would have been a mess without him.
😁
If you still find it a mess, I am to be blamed for that. ) Bill Jelen for being so generous and allowing
me to extend the deadline. I bet he initially thought that I wouldn’t complete this at all. My friend
and business partner, Sourabh Kushwaha, for proofreading this book multiple times. The incredible
copy editor, Kitty Wilson, for putting up with my shenanigans and patiently fixing my bad English.
And all the genius Power Query minds that I have learned a lot from, including Imke Feldmann, Ben
Gribaudo, Rick De Groot, Mellissa De Korte, Victor Wang, Brian Julius, Bhavya Gupta, Owen Price,
Mike Girvin, Chris Webb, Aditya Darak, Marcel Beug, Daniil Maslyuk, and even people behind
aliases like mma173.
x POWER QUERY BEYOND THE USER INTERFACE

Introduction
I participated in World Excel Champ (a worldwide competition to find the nerdiest Excel jockey)
back in 2016. I could create fancy charts and cube-driven pivot tables, and I could even break the
spreadsheet with array formulas. I thought I’d do well in the contest. The first two rounds were
indeed charts and formulas, and I did okay. Round 3 was Power Query questions, and I kid you not,
that is when I opened the Power Query window for the first time in my life.
I was zapped, staring at whole new window of tools and functionality opened on top of Excel. I
couldn’t do a thing, and I lost! But that got me thinking: If Power Query is being tested at a global
competition, it might be a thing in the future.
For the next few months, I laid my hands on all the Power Query materials I could get on the web,
including the book M Is for (Data) Monkey by Ken Puls and Miguel Escobar. There’s a good chance
that you’ve also read that book, or maybe its successor, Master Your Data.
Each time I learned something new in Power Query, I kicked myself for not having learned Power
Query earlier. Turning pages, I’d say to myself, “This is so awesome. Why didn’t I learn this sooner?”
But until that time, I had been the user interface guy. I just knew the clicks to solve most problems.
Fast forward to early 2018, when I met up with my close friend Ashutosh, who runs a data analytics
company in Mumbai. The awesomeness of Power Query was bound to dominate our conversation.
In the midst of the chatter with lemonade and wada-pav (Indian burger), he mentioned that his team
was grappling with Power Query’s limitations. I couldn’t believe his words.
I stopped drinking the lemonade and said, “Grappling with Power Query! Really? It’s the most
awesome automation tool ever built in Excel. Maybe your team doesn’t know how to use it.” I tried
to quash his point without even listening to his problem.
Ashutosh is a smartass. He didn’t mind that I wasn’t drinking the lemonade anymore and turned his
laptop to show me a peculiar case. Let me summarize the case to the best of my memory.
One of his clients had shared with him CSV files for sales data, but each file had slightly different
column headers, and there was no real pattern to the changes. He asked, “How would you dynamically
combine data from all these CSV files, given that the column headers aren’t always the same, and
there’s no pattern to them?”
The problem was indeed hard to solve (at least with the Power Query user interface), and I felt the
same way I’d felt when I’d faced Power Query for the first time in 2016. There was work to be done
beyond the UI.
I scraped the Internet again and found a whole new world behind Power Query’s user interface and
that was the M language. I guess that was a reason Ken titled his book M Is for (Data) Monkey. 😁
In the next few months, I took my Power Query skills to a whole new level. Although M isn’t the
most friendly-looking code, once you learn how it works and what it can do, every bit of its ugliness
is worth it.
In this book, I’ve put together solutions to many of the problems that business users like Ashutosh
struggle with every single day—problems that are clearly beyond the scope of Power Query’s UI
capabilities. This book will turn you into an advanced data monkey. (I really wanted to be able to
say that.)😁
Who This Book Is For
I have made a few assumptions about you as the reader of this book:
• You already know the “clicks” of Power Query, either in Excel or Power BI.
• You already know how to create basic queries that can combine data, remove columns, filter
and unpivot rows, and the like.
Introduction xi
• You have come across a few data cleaning problems that have made you scratch your head
and wonder, “How am I going to do that?”
• Most importantly, you could relate to Ashutosh’s plight in the Introduction. (Don’t tell me
😜
you skipped that! )
If you were mostly nodding through this list: Congratulations! You are my ideal reader! Enjoy the book!
How to Use This Book
I’ve written this book in a structured way, leveling up the complexity one page at a time. It’s ideal
for you to read the book in its written order. Fundamental building blocks like lists, records, tables,
and navigation are discussed in the first few chapters. More complex topics like iterations, custom
functions, and patterns and recipes are then built on top of these fundamentals in later chapters.
I did start with a story, but this isn’t a story book. To truly master M, you need to practice. I’ve got
two suggestions for you:
• After you read a concept or an example, close the book and practice it on your own. This will
help you recall the concepts and apply the learning to your own data.
• Spend some time with the exercises that appear at the end of most chapters. Do your best
to solve those questions before looking at the solutions.
I’ve provided companion files for you to practice, tweak, and tinker with. You can download them
from https://tinyurl.com/pq-book or by scanning the QR code below.
Chapter 1: Working with Lists 15

Chapter 1: Working with Lists

After warming up, there is no better place to start your M workout than with a list. Lists can make
your queries a lot more robust and adaptive to changes in your data. In this chapter, we’ll talk about
what lists are, explore their nuances and behavior, and use them to solve a few practical problems.
Note: For this chapter, we’ll be using the file Lists.xlsx (which you’ll find in the Lists folder).
As you work through this chapter, you’ll be able to walk through the examples by using
the queries provided in this file.

What Is a List?
Refer to the query ListExample.
In the query ListExample, I’ve hard-coded a list that contains a few numbers and letters.

Figure 1-1 This is how a list looks in Power Query.

Here are a few things to keep in mind about lists:
• In the Power Query UI, a list appears as a single column, and the list icon appears next to the
query’s name. Note that a table with one column is not a list. (You’ll learn more about tables
later in this book.)
• A list can store any value (a number, text, a date, or even a structured value like a table or
another list).
• A list is typed in curly brackets ({ }), and the list items are separated by commas—for
example, ={1, 2, "Apple"}. In this example, because Apple is a text value, it is wrapped
in double quotes.
• A list doesn’t have a header. To help you understand that something is a list, Power Query
adds the default header label List (which you can’t change).
• A list uses zero-based sequencing, which means the position of the first value in a list is 0.
For example, in Figure 1-1, the letter E is in position 4 and not 5 (even though it appears in
position 5 in the UI).
How to Create a List
Refer to the query My First List.
Let’s start off by creating a hard-coded list:
1. In Power Query, create a blank query (Home à New Source à Other Sources à Blank Query).
2. Name this query My First List.
3. Write the following code in the formula bar:
= {"Hello", "How", "are", "you?"}
16 POWER QUERY BEYOND THE USER INTERFACE
This is what you get:

Figure 1-2 A list is created inside curly brackets.

Refer to the query Interesting List.
Now let’s look at a slightly more interesting example of creating a list:
1. Create another blank query (Home à New Source à Other Sources à Blank Query).
2. Name the new query Interesting List.
3. Write the following code in the formula bar:
={"Chandeep",100,{"Sleeping","Running","Creating lists"}}

Here we have created a nested list (a list inside a list) by using curly brackets within the outer curly
brackets.

Figure 1-3 An example of a nested list.

At this point, creating a nested list might seem useless, but there are several advanced problems
you can solve by using the nested list structure (as discussed later in this chapter).
How to Extract a List from a Table
Refer to the query ExtractList.
A lot of Power Query functions require you to work with lists. You’ve already seen that you can
create a static list by hard-coding list values in curly brackets. Sometimes, though, you need to use
Chapter 1: Working with Lists 17
dynamic lists. Let’s start with a simple example of creating a dynamic list by extracting a column of
a table as a list.
Consider this table:

Figure 1-4 A table with three columns.

From this table, say that we’d like to extract the Sales column as a list. We can do so in two simple
steps:
1. Create a new step by clicking the fx button next to the formula bar. Power Query creates a new
step that automatically refers to the previous step, Source.
2. Add the column name Sales in square brackets (so you have = Source[Sales] in the
formula bar) and press Enter. The Sales column is extracted as a list.

Figure 1-5 Extracting a column as a list.

Let’s pause to reflect on what just happened:
• Notice that Source is the name of the previous step, which refers to the three-column table
in Figure 1-4.
• By referencing the column name in square brackets ([ ]) after the table reference, we extract
that column as a list.
Note: You don’t need to wrap the column name in quotation marks, even if it contains
spaces:
= table[column name] // returns the column as a list.
However, you do need the pound sign and quotes if the column name contains special
characters, such as an exclamation point. In that case, the M code looks like this:
= table[#"column name!"]
18 POWER QUERY BEYOND THE USER INTERFACE
Not surprisingly, you can create the same list by right-clicking on the Sales column and choosing
Drill Down. So, we’ve done nothing remarkable here, but allow me a few more paragraphs before I
start to blow your mind. 😉
How Lists Behave
Refer to the query ListBehavior.
A common behavior of a list is to expand to create rows. To see this in action, let’s take a slightly
more practical use case. Take a look at this three-column table:

Figure 1-6 A table with three columns.

Say that you’d like to add a column to this table and duplicate each row for the years 2019 and 2020.
The result should look like this:

Figure 1-7 The expected output with 10 rows: 5 for 2019 and 5 for 2020.
To create a list with two values for each row of the table, we follow these steps:
1. Create a custom column (Add Column à Custom Column).
2. Name the new column Year.
3. In the Custom Column Formula box, create the list ={2019, 2020} and click OK.

Figure 1-8 Using curly brackets to create a list.

Chapter 1: Working with Lists 19
We’ve now created a list with two values for each row of the table.

Figure 1-9 An expandable list with two values for each row in the Year column.
Notice the little expand button on the Year column. If you click the expand button and select Expand
to New Rows, you end up with values for both years in the table, as shown here. Sweet!

Figure 1-10 When you expand the Year column, the table includes a row for each year.
To summarize, when you create a list in each row of a table, the list can be expanded to create more
rows (or columns). This is incredibly useful when you want to dynamically expand a table’s rows (or
columns).
Example 1: Expanding Dynamic Columns
Refer to the query Dynamic Expansion.
Let’s level up to a trickier problem. Say that we are working with three tables in the Lists.xlsx file
(on the sheet named Sales Data), as shown here, and we want to append them into a single table.
20 POWER QUERY BEYOND THE USER INTERFACE

Figure 1-11 Three tables with the same columns.

Let’s fire up Power Query and gather the three tables by following these steps:
1. In the Power Query window, create a blank query (Home à New Source à Other Sources à
Blank Query).
2. Rename the new query Dynamic Expansion.
3. In the formula bar, write the function = Excel.CurrentWorkbook() to retrieve all the
tables in the current Excel file.

Figure 1-12 Excel.CurrentWorkbook() returns all tables and named ranges in the current Excel
file.
To append only the Year tables, we can conveniently apply a filter on the Name column to keep the
values that start with the string "Year".
Chapter 1: Working with Lists 21

Figure 1-13 Filtered tables that begin with Year.

At this stage, I am pretty sure you’re familiar with this view, which has nested tables waiting to be
expanded:

Figure 1-14 The Content column has three tables that can be expanded.
Here’s the challenge: When we expand these tables, the column names get hard-coded, which means
any new columns added to the source data will not dynamically expand. Bad!

Figure 1-15 Upon expansion, the column names are hard-coded into the formula.
But what if we want to automatically show any new columns that are added? Before we look at how to
do that, we need to take a step back and understand the function used in the Expanded Content step:
22 POWER QUERY BEYOND THE USER INTERFACE
= Table.ExpandTableColumn(
#"Filtered Rows", // 1
"Content", // 2
{"Sales Rep", "Customer", "Amount"}, // 3
{"Sales Rep", "Customer", "Amount"} // 4
)

Note that:
1. The first input is a reference to the table in the previous step (i.e., #"Filtered Rows").
2. The second input is the hard-coded name of the column to be expanded (i.e., "Content";
refer to Figure 1-14).
3. The next input is a list of columns to be expanded. Notice the curly braces:
{"Sales Rep", "Customer", "Amount"}.
4. The last input is a list to rename the columns that are expanded, and it is optional.
A Logical Way to Think About the Solution
Here is a two-step process that will get us the result we want:
1. Start by creating a dynamic list of all the possible column names in the three tables (from the
Content column; refer to Figure 1-14).
2. Once we have the list of columns, replace the hard-coded list (in Figure 1-15) with our
dynamic list, and it should work.
Let’s give this a shot.
Step 1: Creating a List of Columns
Here is how we can create a dynamic list of column names:
1. Click the fx button next to the formula bar to create a new step in between the steps Filtered
Rows and Expanded Content.
2. Write the following M code in the formula bar to get a list of column names from the three
tables:
=Table.ColumnNames(
Table.Combine(#"Filtered Rows"[Content])
)

3. Right-click the new step and rename it ColList.

Figure 1-16 Creating a new step that returns a dynamic list with column names from the three tables.
Chapter 1: Working with Lists 23
Let’s dissect the formula from the inside out:
• #"Filtered Rows" is the name of the table in the previous step (refer to Figure 1-14).
• In the event that we reference the column name ([Content]) after the table name, we’ll
get a list that contains three tables like the one shown here:

Figure 1-17 This is how the list looks if we extract the Content column from the previous step (Filtered
Rows).
• The Table.Combine function accepts a list that contains the tables that are to
be appended). Sure enough, we have that, so it is legit to write the full formula as
Table.Combine(#"Filtered Rows"[Content]). However, this function is literally
going to append the data from all three tables (regardless of the column names), and we
don’t want that to happen yet.
• Therefore, we have the final wrapper Table.ColumnNames, which accepts a table as input
and extracts the column names as a list that is not hard-coded. (In comparison, see the formula
with the hard-coded list in Figure 1-15.)
Step 2: Plugging the List Back into the Expanded Content Step
Now if we go to the Expanded Content step, we get an error because we added a step in between,
and Expanded Content no longer refers to the correct table.

Figure 1-18 The Expanded Content step returns an error.

Let’s fix this. Here is the revised M code for the Expanded Content step:
=Table.ExpandTableColumn(
#"Filtered Rows",
"Content",
ColList
)
24 POWER QUERY BEYOND THE USER INTERFACE
And here’s how it differs from the original code:
• The first input is now #"Filtered Rows" (the table on which expansion will happen).
• The "Content" column name remains the same because it was hard-coded.
• The list to be expanded now refers to the ColList step that we’ve created.
• We don’t want any renaming to be done, so we’ve removed the optional renaming list.

Figure 1-19 The Table.ExpandTableColumn function now references the dynamic list of columns.
Now, if any new columns are added in the base tables, the query is robust enough to expand them
automatically.
Gotcha! 🤕 Because we omitted the last input (renaming the list) from the
Table.ExpandTableColumn function, the function might return an error if any of the
expanded columns have the same name as the existing columns. An easy solution for this
is to make sure the existing table columns are renamed before the Expanded Content step.
Recap: Feed Power Query What It Eats
It’ll be worthwhile to see what it looks like to feed Power Query what it eats (my cool lingo from the
previous chapter). Take a close look at the input and output values of the functions used in creating
the ColList and Expanded Content steps:
ColList =
Table.ColumnNames(
Table.Combine(#"Filtered Rows"[Content])
) // 1 Creating Col List

#"Expanded Content" =
Table.ExpandTableColumn(
#"Filtered Rows",
"Content",
ColList
) // 2 Expanded Content

Note that:
1. The Table.Combine function can only work if we input a list containing tables. The output it
returns is a single table. Table.ColumnNames can only work if the input is a table. The output
it returns is a list of column names.
2. Table.ExpandTableColumn can only work if the third argument is a list. The output it
returns is appended tables with expanded columns.
Note that the output value of each function matches the input value of the wrapper function. Here
is a table to help you see this vividly:
Chapter 1: Working with Lists 25

Figure 1-20 The output values of the functions match the input values of the wrapper functions.
I said this earlier, but I’d like to say it again: Pay attention to the input and output values as you learn
different M functions. A mismatch between what a formula is asking for and what you are feeding
in the formula will at the very least return an error and will definitely lead to frustration. 😣
Example 2: Expanding Dynamic Columns, Part 2
Refer to the query Dynamic Expansion Part 2.
In the previous example, our intermediate query looked like this right before we expanded the
Content column:

Figure 1-21 Expanding the nested tables hard-codes the names of the columns.
To make the expanded columns dynamic, we created a dynamic list of column names and then
plugged that list into the Table.ExpandTableColumn function. Very important: This allowed us
to retain the existing Name column along with the expanded columns (refer to Figure 1-19).
But what if we’re only interested in expanding the columns from the tables in the Content column
and don’t want to retain any of the existing columns?
We can do that—and it is way easier. Here’s how:
1. Create a new step by clicking the fx button.
2. Write the following code in the formula bar:
= Table.Combine (#"Filtered Rows"[Content])

This combines all the tables in the Content column. Neat!Right-click the new step and rename
it CombinedAllTables.
26 POWER QUERY BEYOND THE USER INTERFACE

Figure 1-22 Creating a new step to append all tables from the Content column.
Example 3: Dynamically Unpivoting Other Columns
Refer to the query PivotedData.
Now let’s consider an interesting unpivoting example where lists can come very handy. Take a look
at this data:

Figure 1-23 Cross-tabulated (pivoted) data shows sales for product and color, by month.
The data is obviously pivoted (or cross-tabulated) with month-ending dates in columns and product
and color in rows.
A typical unpivoting technique in Power Query would be to select both the Product and Color
columns, right-click, and select Unpivot Other Columns. And boom! You’d be done!
However, if you do that, you get output like this, where the Product and Color columns are hard-
coded as a list:

Figure 1-24 The Unpivoted Other Columns step hard-codes the Product and Color columns in the
formula.
Chapter 1: Working with Lists 27
What if we add a Category column as the third column in the source data?

Figure 1-25 A new Category column is added.

Now if we refresh the query with the new column added, the Category column is also unpivoted
because only the Product and Color columns were hard-coded in the formula.

Figure 1-26 The Category column was also unpivoted since it was not included in
Table.UnpivotOtherColumns.
How do we fix this in such a way that the Product and Color columns aren’t hard-coded but are
formed using a dynamic list? Let’s think this through. We could do it like this:
1. Extract all column names as a list.
2. Filter the list for only non-date values (because only dates are supposed to be unpivoted).
3. Plug the filtered list into the Table.UnpivotOtherColumns function instead of hard-cod-
ing the column names.
This should work. Let’s give it a shot.
Step 1: Extract All Columns as a List
Let’s start by creating a list of all column names:
1. Create a new step in between Source and Unpivoted Other Columns.
2. Write the following code in the formula bar to extract all the column names as a list:
= Table.ColumnNames(Source)

3. Rename the new step ColNames. (Renaming custom steps and using CamelCase for clarity
is a good hygiene practice.)
28 POWER QUERY BEYOND THE USER INTERFACE

Figure 1-27 Extracting all columns from the Source step as a list.
Step 2: Filter the List for Non-Date Values
To filter the list, we can use another list function, List.Select, which checks each item of the list
against a condition. In this case, our condition can be loosely written like this:
= List.Select(
Column Name List,
each value <> a date
)

Let’s wrap the existing M code in the following List.Select function:

Figure 1-28 List.Select returns a list of values that are not dates.
Momentarily refer to the query ListMagic.
Okay, I might have stretched your gray cells a bit too much here. There is a lot going on in these two
lines. Let me explain:
• Table.ColumnNames(Source) gets us the list of all column names (refer to Figure 1-27).
• For each value in the list, we use the try keyword (which is synonymous with Excel’s IFERROR
function) to convert each value into a date:
each (try Date.From(_))

If we write the same code slightly differently, the results are more visual:
Chapter 1: Working with Lists 29

Figure 1-29 Using List.Transform and the try keyword returns a record.
• The output of List.Transform will not be a filtered list but a list where each value is tried
using the Date.From function to transform it into a date.
• The try keyword returns a record for each row with a field HasError that has TRUE or FALSE
as output. We get TRUE as the output if an error occurred (i.e., if the value was not converted
to a date) and otherwise FALSE.
• Now the Product and Color columns and the newly added Category column contain text
values, and our formula will return an error because it cannot convert them into dates.
• The error value is (by default) stored in a HasError field as TRUE or FALSE. So, we simply
extract TRUE or FALSE from the HasError field.
Note: When we replace List.Transform with List.Select, it evaluates each value
of list against a TRUE/FALSE condition. All values where the condition returns TRUE are
retained, and the rest are removed.
Another important thing that could easily be overlooked is that
try Date.From(_) returns a record, and in order to refer to the HasError field from
the record, we have to surround try Date.From(_) with parentheses, like this:
(try Date.From(_)) [Has Error].
It’s okay if your eyes glazed over just now. You’ll get a more detailed understanding of working with
each _ (underscore), records, and try...otherwise in the next few chapters.
Step 3: Plug the Filtered List into Table.UnpivotOtherColumns
Refer to the query PivotedData.
In our PivotedData query, if we go to the Unpivoted Other Columns step, Power Query shows an
error. This is expected because we added a step before it, causing it to point to the ColNames step
rather than to the correct table (via the Source step). Let’s fix that.
30 POWER QUERY BEYOND THE USER INTERFACE

Figure 1-30 The hard-coded list has been replaced with a dynamic list of columns (i.e., ColNames).
In the function Table.UnpivotOtherColumns:
• Change the first input to Source (because the Source table is the table on which unpivoting
will happen).
• Ensure that the list of columns refers to the dynamic list (step) we’ve created (i.e., ColNames).
• To omit an additional step, rename the Attribute column Month Date.
Now, if any new (non-date) columns are added to our base table, they’ll automatically be left out of
unpivoting. How awesome is that?! 😎
Lists of Lists (a.k.a. Nested Lists)
Refer to the query Creating List of Lists.
We can use nested lists—that is, lists of lists, where one or more lists are wrapped in another list—to
solve more advanced problems. Before we jump to a practical example of nested lists, let’s warm
up a bit by creating a nested list. Take a look at this M code:
= { // Outer List
{1,2}, // Nested List 1
{"A","B"}, // Nested List 2
{"C",4} // Nested List 3
}

This code creates three nested lists within a list.

Figure 1-31 Three nested lists within a list. Each list is surrounded by curly brackets.
Chapter 1: Working with Lists 31
Notice a few things:
• The outer list is wrapped in curly brackets (as you’d expect).
• The inner lists are separated with commas and also wrapped in curly brackets.
• All items of the inner lists are also separated by commas.
And that’s about it. We’ve created a nested list. Now let’s use this list of lists in a practical setting.
Example 4: Removing Errors from Multiple Columns
(Using a List of Lists)
Refer to the query Replacing Errors.
Consider this table in the Source step, which contains errors. In this case, the source data from Excel
had a few #N/A errors, and Power Query also marked each of those cells with “Error.”

Figure 1-32 A table with errors in the Date and Sales columns.
Say that we’d like to replace each error with a null value. Sure enough, we could do that in the user
interface, but what if we want the replacement to be dynamic, so that errors are replaced in all
columns even if columns are added (or removed)?
To solve this problem, we can take hints from the UI and see what M code gets generated if we
replace errors in the Date and Sales columns with null values:
1. Select the Date and Sales columns.
2. Select Transform à Replace Values drop-down à Replace Errors.
3. In the box that appears, enter null (all lowercase because Power Query is case-sensitive).
The errors are replaced with null values, giving us code like this:

Figure 1-33 The Table.ReplaceErrorValues function uses a nested list to replace errors in each
column.
32 POWER QUERY BEYOND THE USER INTERFACE
Notice a few interesting things:
• Table.ReplaceErrorValues needs a table to work with (i.e., the Source step) in the first
argument.
• In the second argument, the function accepts a nested list. Each inner list has two parts: the
column name (as a text value) and the replacement value (null).
• To make this formula dynamic, we need to produce a nested list, and each inner list must
have two values: the column name (as a text value) and null.
• Also, the dynamic nested list should contain all columns from the Source step.
Let’s get to work.
Step 1: Creating a List
Let’s start by generating column names as a list (not as a nested list—yet):
1. Create a new step in between the Source and Replaced Errors steps.
2. Write this M code in the formula bar to get the list of column names:
= Table.ColumnNames (Source)

3. Rename the new step AllColsList.

Figure 1-34 Extracting all the columns in the Source step as a list.
Step 2: Converting a List into a List of Lists
So far, we’ve been able to produce a list of columns (refer to Figure 1-34), but we need a nested list
structure to make this work. Let’s revise our M code further, as shown here:
Chapter 1: Working with Lists 33

Figure 1-35 Using List.Transform to create a list of lists.

What just happened?
• We started by wrapping the existing formula in the List.Transform function.
• The first argument is a list to be transformed (that’s our column names list—
Table.ColumnNames(Source)).
• The second argument is the transformation. In this case, we’d like to nest each value into a list.
• The underscore (_) represents each value, and the curly brackets are used to nest each value
in a list.
The only problem: We need to have two values in each list: a column name and null. We already
have the column name. Now let’s get the null.
Step 3: Adding Null Values to the Nested Lists
Here is a slight modification to the code to add null values to the nested lists:

Figure 1-36 Appending null to each list.

34 POWER QUERY BEYOND THE USER INTERFACE
Note the following:
• Along with the underscore, we append (using a comma) a second value in the list (null).
• A preview of each list now shows two values—the column name and null—just like we
wanted. 😍
Step 4: Plugging the List of Lists into Table.ReplaceErrorValues
Once again, the Replaced Errors step must have errored out because we added a step in between,
so let’s fix that.

Figure 1-37 Replacing the hard-coded list with the dynamic list of lists.
In the third step, Replaced Errors:
• The first argument refers to Source (i.e., the table).
• The second input is the dynamic nested list we created as a new step (i.e., AllColsList).
Now if any error shows up in any column of the source table, it’ll automatically be replaced by a
null value.
Chapter 1: Working with Lists 35

Exercises
Here are a few exercises to test your understanding. Ready?
Note: Open the Excel file List Exercises - Unsolved in the Lists folder and then use the
Power Query Editor to work through these exercises.
Exercise 1: Create a List
Create a list of lowercase letters from a to j. The output should look like this:

Figure 1-38 Expected output.

Exercise 2: Extract and Sum a Column as a List
From the three-column table shown here, extract the Amount column as a list and sum it. The output
should be a single value: the sum of the Amount column.

Figure 1-39 A table with three columns with the expected output below (i.e., the sum of the Amount
column).
36 POWER QUERY BEYOND THE USER INTERFACE
Exercise 3: Dynamically Remove Junk Columns
Here is a table with a few columns. Notice that the last two columns have the ~ suffix at the end of
the column name (and are junk).
Your task is to remove all the columns that have the ~ symbol at the end of the column name. Make
sure that your solution is dynamic and the column names are not hard-coded.

Figure 1-40 A table with junk columns (with a ~ suffix on the column names).
Exercise 4: Dynamically Rename Columns
The table below has the suffix 1 with each column name. Your task is to dynamically rename all the
columns by removing the 1. The solution should not have any hard-coded column names.

Figure 1-41 Column names suffixed with the junk value 1.

Note: When you’re done with these exercises, you can compare your answers against the
ones I provide in the Solutions chapter at the end of this book or refer to the Excel file List
Exercises - Solved or do both. 😁
 Loved it ?
A person holding a sign

Description automatically generated

👉🏻Buy the book!

Buy the book!