Lab #2

Grouping and Aggregation

Practice DQL Basics - Labs
CONTENTS
1. Introduction ........................................................................................................................................3
2. Command: Summarize ........................................................................................................................3
⚙ 2.1 Lab – Summarize with the Count Function ................................................................................................................ 3
⚙ 2.2 Lab – Log Source............................................................................................................................................................... 4
2.3 Additional Practice ................................................................................................................................................................ 5
3. Command: Sort ...................................................................................................................................5
⚙ 3.1 Lab – Sort by Timestamp ................................................................................................................................................ 5
⚙ 3.2 Lab – Sort by Log Source Count.................................................................................................................................... 6
3.3 Additional practice ................................................................................................................................................................ 6
4. Naming ................................................................................................................................................7
⚙ 4.1 Lab – Provide a Name ...................................................................................................................................................... 7
⚙ 4.2 Lab – Sort by totallogs.................................................................................................................................................... 7
5. Working with Fields ........................................................................................................................... 8
⚙ 5.1 Lab – fields ......................................................................................................................................................................... 8
⚙ 5.2 Lab – fieldsAdd ................................................................................................................................................................. 8
5.3 Additional Practice ................................................................................................................................................................ 8
6. Calculations ........................................................................................................................................ 9
⚙ 6.1 Lab – Calculate the Percent of Error Log Events....................................................................................................... 9
6.2 Additional Practice.............................................................................................................................................................. 10

2
1. Introduction
This lab builds on your previous knowledge of DQL with the addition of methods for grouping and aggregating
data, as well as adding fields to your log entries.

2. Command: Summarize
The summarize command groups together records with the same values for a specified field and aggregates
them. It's important to remember that you can only use summarize once in a query, but you can run multiple
steps within the command.

Resources

� summarize
count()
countIf()
filterOut

⚙ 2.1 Lab – Summarize with the Count Function

A large number of logs may be returned in a particular timeframe. You can use the count function to provide
the total number of logs found that match your criteria within the timeframe. Build the query of the logs you
want returned, and then add: | summarize count()
1. Navigate to Logs and Events.
2. Enter the following query that counts the number of logs in the last 1 minute:
fetch logs, from:now() -1m
| summarize count()
3. Run query.
The results will look something like this:

This is returning the number of logs found within the last minute.

3
 ⚙ 2.2 Lab – Log Source
In many cases, you will want to know the origin of log files. To do this, you can use the log.source parameter
within your query.
1. Building on the query from 2.1, add the log source as shown here:
fetch logs, from:now() -1m
| summarize count(), by:{log.source}
2. Take note of the surrounding braces.
3. Run query.
The results will look something like this:

4
2.3 Additional Practice
Try these additional options for practice:
1. Fetch the count of all logs summarized by status:
fetch logs
| summarize count(), by:{status}
2. Try adding in the count by status for the last hour.
3. Try filtering out (filterOut) logs that do not have a status. i.e., status == “NONE”
fetch logs, from:now() -1h
| filterOut status == "NONE"
| summarize count(), by:{status}

3. Command: Sort
Sorting records is standard practice with any type of query. In DQL, use the sort command to classify results
in either ascending (default) or descending order.
Resources

� sort

⚙ 3.1 Lab – Sort by Timestamp

1. Enter the following query: fetch logs, from: -10m | limit 10
• How are the results sorted? Extend or shorten the timeframe as needed.
• Did you notice we didn’t need to use the “now” parameter?
2. Add the line: |sort timestamp
• Do you see a difference in how they are sorted? When it isn’t specified, DQL assumes you
mean to sort in ascending order.
3. Try |sort timestamp desc
• What is the difference in the results now?
The results will look something like this:

5
⚙ 3.2 Lab – Sort by Log Source Count
In the previous labs, we added a summary of count by log source. To sort by the count of log sources, we can
add this line: | sort `count()`
1. Enter the full query as:
fetch logs, from:now() -10m
| summarize count(), by:{log.source}
| sort `count()`
2. Add the option so the list is sorted with the log source with the most logs at the top.
The results will be similar to:

3.3 Additional practice

Practice fetching logs sorted in the following ways:
1. By timestamp descending, and then by log source:
fetch logs, from: now() -10m
| sort timestamp desc, log.source
2. By loglevel, then by status
3. By process technology: | sort process.technology desc

6
4. Naming
You may have noticed above in 3.2, that when we both summarized and sorted by the count() of logs that it
was a little awkward. A best practice is to use naming for these values, instead of referring to the command.
Then these values can be more easily reused.

⚙ 4.1 Lab – Provide a Name

1. First, use summarize to get a count of log events in the last hour summarized by the log
source.
fetch logs, from:now() -1h
| summarize count(), by:{log.source}
2. Next, provide a name for the count
fetch logs, from:now() -1h
| summarize totallogs = count(), by:{log.source}
The results will now look something like this:

⚙ 4.2 Lab – Sort by totallogs

1. Use the name to sort the logs by source from the highest to the lowest number.
fetch logs, from: -1h
| summarize totallogs = count(), by:{log.source}
|sort totallogs desc
2. Refer to lab 3.2 to compare the differences.

7
5. Working with Fields
Resources

� fields
fieldsAdd
� formatTimestamp

fieldsRemove
fieldsRename

⚙ 5.1 Lab – fields

When reviewing results, you may find that there is extraneous data you don’t find is helpful or necessary for
the query you are building. Use the fields command to list only the fields you want to see in the results.
1. Enter the simple query to fetch logs from the last hour.
• Which fields do you see listed in the results?
2. Add the following line to the query: |fields timestamp, status, content, log.source
• What do you notice in the results?

⚙ 5.2 Lab – fieldsAdd

Use fieldsAdd to include additional information in your results, based on the evaluation of a function. You
can also use it to overwrite an existing field.
1. Use fieldsAdd and formatTimestamp to add the following line to the query from Lab 5.1:
| fieldsAdd dateOnly = formatTimestamp(timestamp, format:"MM-dd-YYYY")
2. Run the query.
The results will look something like this, with the new field added at the end:

5.3 Additional Practice

Explore using fieldsRemove and fieldsRename to alter the returned results of your query.

8
6. Calculations
Now that you have seen how to summarize data, use functions, add names, and create fields, let’s pull that all
together into a query that performs a calculation and then stores it in a field.
Resources

� round
toString
concat

⚙ 6.1 Lab – Calculate the Percent of Error Log Events

Let’s calculate the percentage of log events that belong to the ERROR category.
1. Enter the following query that sets two values:
fetch logs, from:now() -1h
| summarize totallogs = count(), errorlogs = countIf(loglevel=="ERROR")
2. Run query.

The results will look something like this:

3. Next, use those values to calculate the percentage of ERROR logs and add it to a field. This is where
naming is important because we can use totallogs and errorlogs in the formulas:
fetch logs, from:now() -1h
| summarize totallogs = count(), errorlogs = countIf(loglevel=="ERROR")
| fieldsAdd percentErrorLogs = ((toDouble(errorlogs)/toDouble(totallogs))*100

The results should look something like this:

9
6.2 Additional Practice
Try using the following functions to clean up the display of the percentage:
a. Use round to remove decimal places.
b. Use toString to convert it to a string and concat to add a “%” sign.
c. Remove the fields so only the percentErrorLogs remains.
d. Go to Actions and select ‘Pin to dashboard’ to display the value.
fetch logs, from:now() -1h
| summarize totallogs = count(), errorlogs = countIf(loglevel=="ERROR")
| fieldsAdd percentErrorLogs =
concat(toString(round((toDouble(errorlogs)/toDouble(totallogs))*100)),"%")
| fieldsRemove errorlogs, totallogs

10