FortiAnalyzer

SQL and Datasets (supplementary material)

FortiAnalyzer 6.4
© Copyright Fortinet Inc. All rights reserved.
Last Modified:
Wednesday, February 16, 2022
Lesson Overview

Datasets and SQL

SQL Functions and Operators

FortiAnalyzer Functions and

Macros
Datasets and SQL
Datasets
• Datasets are SQL SELECT queries to the database
• Data populates a chart
ADOM specific!
Reports > Report Definitions > Datasets

Dataset (example App-Sessions-By

Category)
4
 Designing SQL Queries
• FortiAnalyzer uses SQL as the local database
Test that queries are well-
• Proper query syntax required formed and all keywords are
spelled correctly

SQL queries are

not case-sensitive

5
SQL – The Declarative Language

SELECT dstip as destination_ip, count(*) as Session

FROM $log WHERE $filter and dstip is not null GROUP BY
dstip ORDER BY session desc LIMIT 7

• Declarative language: Describes what needs to be done rather than how to do it

• All information in the database is represented as tables
• Each table consists of a set of rows and columns
• Two types of tables: User tables and System tables

6
Basic Data Manipulation Constructs (DML)
• SELECT
• Retrieve and display data from one This is the only query
or more database tables (read-only query)
• SELECT … FROM … WHERE
statement used by
FortiAnalyzer for reports
• INSERT
• Add new rows of data into a table
• INSERT INTO … VALUES …
• UPDATE
• Modify existing data in a table
• UPDATE … SET … WHERE
• DELETE
• Remove rows of data from a table
• DELETE FROM … WHERE

7
 SELECT Statement
• The SELECT statement retrieves the log data you want from the database
• Must specify criteria using a recognized/supported clause

Clauses must be coded in a

specific sequence!

Clause Definition
FROM Selects the table or views.
WHERE Sets the conditions (all rows that do not satisfy the condition are eliminated)
GROUP BY Collects data across multiple records and groups the results by one or more columns.

ORDER BY Orders the results by rows.

LIMIT Limits the number of records returned based on a limit value. OFFSET clause can be used with the
LIMIT clause to offset the results by a set value.

8
SELECT and FROM
• Use the SELECT query to ask specific questions of the database

SELECT column FROM log_type

Column from database that contains The log type under which the data is
the value(s) you want to retrieve contained (ie. Traffic, Web filter, etc.)
• When designing queries for SQL reports on the FortiAnalyzer device, the Log Type
is assigned to a variable called $log

SELECT dstip as destination_ip FROM $log

SELECT *
returns all
data
9
Multiple Log Types
• Search multiple log types
• Combine the data so that you can compare and contrast information

SELECT dstip, hostname FROM $log-traffic, $log-webfilter

Log type syntax Log type
$log-attack Attack log
$log-dlp DLP log
$log-event Event log
$log-netscan NetScan log
$log-app-ctrl Application control log
$log-emailfilter Email filter log
$log-traffic Traffic log
$log-virus Anti-virus log
$log-webfilter Web filter log

10
WHERE
• The WHERE clause requests data with certain characteristics
• The expression specifies a stored value in the database

SELECT column FROM log_type WHERE expression1 and expression2 not in

expression3

Criteria you want to Can use multiple expressions

specify separated by AND/OR/NOT statements
SELECT dstip as destination_ip FROM $log WHERE $filter and dstip is
not null

11
GROUP BY
• GROUP BY statement is usually used in conjunction with aggregate functions to
group data by one or more columns.
• Returns one output row for each group
• Can form groups within groups
• Each item in the SELECT list produces a single value per set

SELECT column, aggregate_function FROM log_type WHERE

expression1 and expression2 not in expression3 GROUP BY column

If GROUP BY is used without aggregates,

it is similar to the DISTINCT clause.

SELECT dstip as destination_ip, count(*) as session FROM $log

WHERE $filter and dstip is not null GROUP BY dstip

12
ORDER BY
• By default, rows of an SQL query result table are not arranged in a particular order

SELECT column, aggregate_function FROM log_type WHERE expression1

and expression2 not in expression3 GROUP BY column ORDER BY
column_name|column_number asc|desc

Can sort data by Can sort data in ascending (asc)

column name or or descending (desc) order. By
column number default, sorts in ascending order.

SELECT dstip as destination_ip, count(*) as session FROM $log WHERE

$filter and dstip is not null GROUP BY dstip ORDER BY session desc

13
LIMIT and OFFSET
• The LIMIT clause limits the number of records retrieved from the query result
• Useful in large deployments to help limit the CPU/memory usage for reports
• Can be combined with ORDER BY asc to get the “top <x> results”

SELECT column, aggregate_function FROM log_type WHERE expression1

and expression2 not in expression3 GROUP BY column ORDER BY
column_name|column_number asc|desc LIMIT number OFFSET number

Specify how many records to return Specify how many records to skip

SELECT dstip as destination_ip, count(*) as session FROM $log WHERE

$filter and dstip is not null GROUP BY dstip ORDER BY session desc
LIMIT 7 OFFSET 1

14
Creating a Dataset in FortiAnalyzer
Reports > Report Definitions > Datasets

15
Analyzing the Dataset Test Results
Reports > Report Definitions > Datasets

16
Lesson Progress

Datasets and SQL

SQL Functions and Operators

FortiAnalyzer Functions and

Macros
SQL Functions and Operators
“Normal” Functions vs. Aggregate Functions

Aggregate functions “Normal” functions

Use the entire column of data as their input and Operate on each element in the column of data
produce a single output

19
NULLIF
• NULLIF function takes two arguments: if the first two arguments are equal, then
NULL is returned. Otherwise, the first argument is returned.

SELECT NULLIF(expression1, expression2)

Must be values that are of the

same datatype

• NULL represents unknown data—it is not equal to zero

20
COALESCE
• Returns the first of its arguments that is not NULL. NULL is returned only if all
arguments are NULL

SELECT coalesce(catdesc,‘unknown’) as category,

coalesce(root_domain(hostname), ‘unknown’) as domain FROM $log
GROUP BY category, domain

21
Aggregate Functions
• Aggregate functions perform a calculation on a set of values in a column and return
a single value

Aggregate functions
AVG(expression) Returns the average value
COUNT(expression) Returns the number of rows
COUNT(*) Returns all rows, even if some columns contain a NULL value

FIRST(expression) Returns the first value

LAST(expression) Returns the last value
MAX(expression) Returns the largest value
MIN(expression) Returns the smallest value
SUM(expression) Returns the sum

22
Operators
• Reserved word or character used primarily in the WHERE clause to perform various
operations
• Arithmetic operators
• Comparison operators
• Logical operators

23
Arithmetic Operators
• Perform mathematical operations on two expressions of one or more of the data
types of the numeric data type category

Operator Description
+ Addition – adds values on either side of the operator
- Subtraction – Subtracts right hand operand from left hand operand
* Multiplication – Multiplies values on either side of the operator
/ Division – Divides left hand operand by right hand operand
% Modulus – Divides left hand operand by right hand operand and returns
remainder

24
Comparison Operators
• Test whether two expressions are the same
• Can be used on all expressions except text, ntext, or image data types

Operator Description
= Equal to
> Greater than
< Less than
>= Greater than or equal to
<= Less than or equal to
<> Not equal to
!= Not equal to (not ISO standard)
!< Not less than (not ISO standard)
!> Not greater than (not ISO standard)

25
Logical Operators
• Test for the truth of some condition
• Return a boolean data type with a value of TRUE, FALSE, or UNKNOWN

Operator Description
ALL TRUE if all of a set of comparisons are TRUE.
AND TRUE if both Boolean expressions are TRUE.
ANY TRUE if any one of a set of comparisons are TRUE.
BETWEEN TRUE if the operand is within a range.
EXISTS TRUE if a subquery contains any rows.
IN TRUE if the operand is equal to one of a list of expressions.
LIKE TRUE if the operand matches a pattern.
NOT Reverses the value of any other Boolean operator.
OR TRUE if either Boolean expression is TRUE.
SOME TRUE if some of a set of comparisons are TRUE.

26
Lesson Progress

Datasets and SQL

SQL Functions and Operators

FortiAnalyzer Functions and

Macros
FortiAnalyzer Functions and Macros
root_domain
• root_domain(hostname)
• Retrieves the root domain of the fully qualified domain name (FQDN)

SELECT devid, root_domain(hostname) as website FROM

$log WHERE‘user’=‘USER01’ GROUP BY devid, hostname
ORDER BY hostname LIMIT 7

29
nullifa
• nullifna(expression)
• Inverse operation of COALESCE
• Can be used to filter out values with N/A and n/a from logs
• SQL syntax  SELECT NULLIF(NULLIF(<value>, 'N/A'), 'n/a')

SELECT coalesce(nullifna(‘user’), ‘srcip’) as user_src,

coalesce(nullifna(root_domain(hostname)),’unknown’) as domain FROM
$log WHERE dstport=‘80’ GROUP BY user_src, domain ORDER BY
user_src LIMIT 7

If user is n/a, the source IP is

displayed, otherwise it returns the
user name

30
FortiAnalyzer Functions: email_domain, email_user
• email_domain: Retrieves anything after the @ symbol in an email address
• email_user: Retrieves anything before the @ symbol in an email address

SELECT ‘from’ as source, email_user(‘from’) as e_user,

email_domain(‘from’) as e_domain FROM $log LIMIT 5 OFFSET 10

Source e_user e_domain

[email protected] user11 example.com
[email protected] user12 hostname.com
[email protected] user13 exampleXYZ.com
[email protected] user14 hostnameXYZ.com
[email protected] user15 example.com

31
FortiAnalyzer Functions: from_dtime, from_itime
• from_dtime(bigint): Returns device timestamp without time zone
• from_itime(bigint): Returns FortiAnalyzer’s timestamp without time zone

SELECT itime, from_itime(itime) as faz_local_time, dtime,

from_dtime(dtime) as dev_local_time FROM $log LIMIT 3

32
Macros
• FortiAnalyzer Date/Time macros

Macros PostgreSQL syntax Result

$hour_of_day to_char(from_itime("itime"), 'HH24:00') 18:00
$HOUR_OF_DAY to_char(from_itime("itime"), 'YYYY-MM-DD 2015-01-01 18:00
HH24:00')
$day_of_week to_char(from_itime("itime"), '"WDAY" D-Dy') WDAY 2-Mon

$DAY_OF_WEEK XXX XXX

$day_of_month to_char(from_itime("itime"), 'DD') 01
$DAY_OF_MONTH to_char(from_itime("itime"), 'YYYY-MM-DD') 2015-01-01

$month_of_year to_char(from_itime("itime"), 'YYYY-MM') 2015-01

$MONTH_OF_YEAR XXX XXX

33
Lesson Progress

Datasets and SQL

SQL Functions and Operators

FortiAnalyzer Functions and

Macros