0% found this document useful (0 votes)
15K views

Infobright ICE User Guide

Uploaded by

peterpan107
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
15K views

Infobright ICE User Guide

Uploaded by

peterpan107
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 62

Infobright Community Edition 3.4.

2
GA
USER GUIDE

WWW.INFOBRIGHT.COM
Copyright Notice
The materials provided herein are Copyright © 2005-2010 Infobright Inc.
All rights reserved.
CONFIDENTIAL: The information contained in this document is the property of Infobright Inc. Except a s
specifically authorized in writing by Infobright, the holder of this document sh all keep the information contained
herein confidential and sh all protect s ame in whole or in part from disclosure or dis semination to third parties.
If these materials were purcha sed a s a digital download, Infobright hereby grants the purcha ser permission to
reproduce a single copy (print or download) of the materials without prior written permission.
If these materials were purcha sed in printed form, no part of these materials sh all be reproduced or retransmitted
by any means, electronic, mechanical, photocopying, recording, or otherwise without written permis sion from
Infobright.
Document Revision 3.4.2 GA-10.07.28
C ONT ENT S
1. About Infobright................................................................................................. 1
Infobright Overview............................................................................................................................................. 1
Infobright and MySQL......................................................................................................................................... 1

2. Setting up Infobright.......................................................................................... 3
Technical Requirements....................................................................................................................................... 3
Linux for Infobright.............................................................................................................................................. 4
Installing Infobright.............................................................................................................................................. 4
Windows Installation Instructions.............................................................................................................. 4
Linux Installation Instructions..................................................................................................................... 5
RPM and DPKG Install.......................................................................................................................... 5
TAR Install............................................................................................................................................... 6
Windows Upgrade Instructions...................................................................................................................7
Linux Upgrade Instructions......................................................................................................................... 8
RPM or DPKG Upgrade........................................................................................................................ 8
TAR Upgrade.......................................................................................................................................... 9
Confguring Infobright....................................................................................................................................... 10
Confguration Tips and Examples............................................................................................................. 13

3. Using Infobright............................................................................................... 14
Starting and Stopping the Infobright Server................................................................................................... 14
Windows................................................................................................................................................ 14
Linux and Solaris.................................................................................................................................. 14
Linux....................................................................................................................................................... 14
Working with the Infobright Server................................................................................................................. 15
Windows................................................................................................................................................ 15
Linux and Solaris.................................................................................................................................. 15
Linux....................................................................................................................................................... 15
Checking the Infobright Version....................................................................................................................... 16
Infobright is the default Storage Engine.......................................................................................................... 17
About Log Files................................................................................................................................................... 17
About Errors........................................................................................................................................................ 18
About SQL Command Syntax........................................................................................................................... 18
About SQL ISO Standards................................................................................................................................. 18

4. Managing Infobright Tables.............................................................................. 20


About the Infobright Database Files................................................................................................................ 20
About Supported Data Types............................................................................................................................ 21
Creating and Dropping Tables.......................................................................................................................... 22
About Column Options......................................................................................................................................23
NULL and NOT NULL............................................................................................................................... 23
Lookup Columns......................................................................................................................................... 23
Unsupported Column Options.................................................................................................................. 23
Unsupported Indices Options.................................................................................................................... 24
Converting Oracle DDL to Infobright ............................................................................................................. 24
Converting SQL Server to Infobright .............................................................................................................. 24
Converting MySQL (MyISAM) to Infobright................................................................................................. 25
Viewing Table Information................................................................................................................................ 25
Viewing Compression Ratio Statistics............................................................................................................. 26
Viewing Table Level Compression Ratio Statistics................................................................................. 27
Viewing Column Compression Ratio Statistics....................................................................................... 28
Comparison of Calculated Compression Ratio to Physical Size........................................................... 28

5. Data Manipulation Statements.......................................................................... 29


Unsupported Data Manipulation Commands (INSERT, UPDATE, DELETE) ..........................................29

6. Character Set Support...................................................................................... 30


Supported Character Sets.................................................................................................................................. 30
Collations and Comparisons ............................................................................................................................ 30
Padding................................................................................................................................................................ 31

7. Importing and Exporting Data in Infobright........................................................ 32


About Transactions............................................................................................................................................. 32
Using AUTOCOMMIT, COMMIT and ROLLBACK Commands........................................................ 32
About Transaction Behavior....................................................................................................................... 33
Failure Handling.......................................................................................................................................... 33
About Export Differences in Infobright........................................................................................................... 33
CHAR(n) Data Type Values........................................................................................................................ 34
Escape Characters........................................................................................................................................ 34
Exporting NULL Values.............................................................................................................................. 34
Infobright Import/Export Syntax..................................................................................................................... 34
Importing Data............................................................................................................................................. 34
Exporting Data............................................................................................................................................. 35
Optional FIELDS Clause............................................................................................................................. 35
Importing Files with Invalid Values..........................................................................................................35
Importing Data Using Linux Pipes...................................................................................................................36
About Import Errors........................................................................................................................................... 36
About Export Errors........................................................................................................................................... 37
Sample Script (Create Table, Import Data, Export Data)................................................................ 38
Exporting and Importing Query Results......................................................................................................... 39

8. Running Queries in Infobright............................................................................ 40


About the Knowledge Grid............................................................................................................................... 40
About Knowledge Nodes........................................................................................................................... 40
Running Queries................................................................................................................................................. 40
Enabling Queries to be Redirected to the MySQL Engine..................................................................... 41
Viewing Queries Redirected to the MySQL Engine................................................................................ 41
Terminating a Query................................................................................................................................... 42
Creating VIEWs in Infobright........................................................................................................................... 42
Create VIEW Syntax.................................................................................................................................... 42
Select Syntax Supported in Infobright............................................................................................................. 42
Select Syntax................................................................................................................................................. 42
Join Syntax.................................................................................................................................................... 43
Union Syntax................................................................................................................................................ 43
Subqueries..................................................................................................................................................... 43
Query Performance............................................................................................................................................. 44

9. Infobright Backup and Recovery.......................................................................45


Backup Procedure............................................................................................................................................... 45
Restore Procedure............................................................................................................................................... 45

A. Functions and Operators.................................................................................. 46


Infobright Optimizer – Supported Functions and Operators.......................................................................46
Comparison Functions and Operators..................................................................................................... 46
Logical Operators.........................................................................................................................................46
Control Flow Functions.............................................................................................................................. 47
String Functions........................................................................................................................................... 47
String Comparison Functions.................................................................................................................... 48
Numeric Functions...................................................................................................................................... 48
Date and Time Functions............................................................................................................................ 49
Text Search and Other Functions............................................................................................................... 51
Group By Aggregate Functions................................................................................................................. 51
Group By Modifers..................................................................................................................................... 51

B. Infobright Data Tools....................................................................................... 52


Infobright Confguration Manager................................................................................................................... 52
Running the Infobright Confguration Manager ................................................................................... 52
Charset Migration Tool.......................................................................................................................................52
Running the Charset Migration Tool ....................................................................................................... 52
Log Structure................................................................................................................................................ 53
Collations-conversion-fle Structure......................................................................................................... 53

C. Linux Tuning Settings...................................................................................... 55


System Settings for Red Hat Enterprise Linux and CentOS......................................................................... 55
Disable SElinux ........................................................................................................................................... 55
Swappiness .................................................................................................................................................. 55
Disable Unused Processes.......................................................................................................................... 55
File System Settings............................................................................................................................................ 55
Ensure CacheFolder is on a Fast Local Disk............................................................................................ 55
Larger Readahead ....................................................................................................................................... 55
Use XFS File System for Data Directories ................................................................................................56
noatime ......................................................................................................................................................... 56
Deadline Elevator ....................................................................................................................................... 56
Increase ulimit to Support Large Data Volume or Users....................................................................... 56
Note on how to detect ulimit problem..................................................................................................... 57
1 ABOUT INFOBRIGHT 1

1. About Infobright
Infobright Overview
Thank you for choosing to install Infobright Community Edition (ICE) 3.4.2 GA. Infobright is
a column-oriented, high performance analytic engine designed for analytic applications and
data marts that need fast query response across large data volumes. Infobright was designed
specifcally for large volume data analytics applications with up to 50TB of data.
Infobright uses a unique and patent-pending approach to compressing, storing, and
processing data that allows it to be installed and run on commodity hardware with little or
no DBA intervention. Infobright requires little tuning to support ad hoc or complex business
analytic queries.
Infobright is a database engine utilizing the MySQL database environment. As such,
Infobright is fully compatible with all MySQL-compliant Business Intelligence tools and
utilizes the MySQL administrative interface to reduce the learning curve for system
administrators.
Infobright Community Edition provides a versatile, highly-compressed database system
optimized for analytic-type queries. The ratio of possible compression and the speed of data
import and retrieval are optimized at the expense of some transactional features of the engine
performance, like the frequent data updating.
Infobright executes complex or ad hoc queries across vast amounts of data with a low cost of
ownership.

Infobright and MySQL


ICE 3.4.2 GA combines the Infobright storage engine with MySQL server implementation.
Infobright consists of several layers. The upper layers are provided by the MySQL server
implementation, and the lower layers are provided by Infobright.
Infobright includes both its own optimizer and executor along with the storage engine. The
MySQL query engine can be used with Infobright ; however, since the MySQL storage engine
interface is row oriented, it can not take full advantage of the column orientation or the
Knowledge Grid and hence query execution via this path is reduced. Queries will be directed
to the Infobright optimizer whenever possible.
Infobright ships with the full MySQL binaries required, including the MyISAM storage
engine. MyISAM is used to store catalog information (as with other storage engines) and you
can use the MyISAM instance for other purposes but joining MyISAM and Infobright tables
may result in reduced performance as the MySQL query engine will be used.

MySQL provides: Infobright provides:


• Mature connectors, tools and • Load function that compresses data
resources • Column-oriented storage engine
INFOBRIGHT 3.4.2 USER G UIDE
1 ABOUT INFOBRIGHT 2

• Interconnectivity and • Knowledge Grid metadata layer that contains


certifcation with BI Tools information about the compressed data
• Management services and • Optimizer/executor that uses the Knowledge
utilities Grid
Infobright and MySQL are integrated as shown below:

Since other storage engines, like InnoDB and Falcon, are not included in the Infobright
distribution, they must be run as separate instances (executables). If you wish to combine
other storage engines with Infobright, you will need to look at a database federation
application (some BI tools provide this).

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 3

2. Setting up Infobright
Technical Requirements
Before installing Infobright, review the following technical requirements.

INFOBRIGHT TECHNICAL REQUIREMENTS


Requirement Description

Platforms Windows XP (32-bit only)


Windows Server 2003/2008 (64-bit only)
Red Hat Enterprise Linux 5
Debian “Lenny”
CentOS 5.2
Ubuntu 8.04 (32-bit only)
Fedora 9 (32-bit only)

Processor Architecture Intel 64-bit


Intel 32-bit
AMD 64-bit
AMD 32-bit

For Personal Evaluation and/or Application Development

CPU Speed 32-bit: 1.6GHz minimum


2.0GHz or faster dual or quad core recommended
64-bit: 1.8GHz minimum
2.0GHz or faster dual or quad core recommended

Memory 32-bit: 1GB minimum


2GB or more recommended
64-bit: 2GB minimum
4GB or more recommended

For Multi-User Evaluation or Production Deployment

CPU Speed 64-bit: 2.0GHz minimum


2.0GHz or faster dual or quad core recommended

Memory 64-bit: 4GB minimum


16GB or more recommended (and at least 2GB per core)

Important: 32-bit platforms are for solution testing purposes only, and not recommended for

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 4

performance or multi-user testing, or production deployments.

Linux for Infobright


Infobright has been optimized for various ‘favours’ of Linux. While Infobright can be run
‘out of the box’ on any supported Linux platform, there are a number of tuning opportunities
to improve performance.
See Appendix C, “Linux Tuning Settings“ for a list of tuning suggestions.

Installing Infobright
The Infobright installation packages are provided as an RPM, DEB, PKG, .exe, or tarball. For
non-Windows platforms, the user installing Infobright must be the root user or a user with
the necessary permissions to install fles, create the user mysql and create the group mysql.

Window s Installation Instruction s


1. Download the install package (for example, infobright-3.4.2-win32.exe) to the Windows
machine on which you are installing Infobright, and double click on the infobright-3.4.2-
win32.exe fle to launch the Install Wizard. Click Next to continue.
2. Click I Agree to accept the GPL license agreement.
3. By default ICE is installed in C:\Program Files\Infobright. To change the default location,
either enter the folder name in the feld or click Browse… to select desired install location
on your computer. Click Install to accept the install location.
4. Please wait while the Install Wizard completes the installation.
5. Choose if you want Infobright to start on completion of the installation. Click Finish to
complete the installation.
6. The Install Wizard automatically creates ICE as a Windows Service, which allows the
Infobright server to be started and stopped automatically when you boot or shutdown
Windows. If you do not want ICE to start on boot, open the Services window from the
Control Panel and change the Startup Type for Infobright from “Automatic” to “Manual”.
7. The Install Wizard automatically determines the optimum memory settings based on the
physical memory of the system. You may change these settings by editing the fle
brighthouse.ini within the data directory.
Important: The memory settings assume that there are no other services on the machine
consuming signifcant memory. If this is not the case, please lower the memory
settings for Infobright.
See “Recommended Memory Confgurations” later in this chapter.
8. To uninstall ICE, select “Infobright Uninstall” under the Infobright program group in the
Windows Start Menu:
Start/All Programs/Infobright/Infobright Uninstall

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 5

Linux Installation Instruction s


RPM and DPKG Install
To install Infobright on Linux using the rpm or dpkg package:
1. Download the installation package from
www.infobright.org/Download/ICE/
2. Obtain root user access, and run:

rpm -i infobright_version_name.rpm [--prefix=path]


or
dpkg -i infobright_version_name.deb

Important: Do not install in the root or home directories due to possible MySQL permission
checking issues during install, start up, and/or load.
If you use the rpm --prefx option, you should manually create a softlink to the
Infobright install directory from /usr/local/infobright .
3. To change the default install options, after installation run:
/usr/local/infobright/postconfig.sh

You can run this script at any time after installation to change the datadir, CacheFolder,
socket, or port. The script must be run as root, and Infobright must not be running.

INFOBRIGHT INSTALL OPTIONS


Parameter Description

Datadir Path to the directory where tables will be created and stored. Use
a high-performance storage such as a RAID.

Cachedir Path to the directory where temporary fles will be created and
stored. Should be located on a fast drive, possibly not the same as
the data. Allow at least 100 GB of free space (depending on
database size).

Port Listening port for the Infobright server instance.

Socket Socket connection point for client connections. (The socket


connection point will be created during the Infobright
installation.)

4. The installation determines the optimum memory settings based on the physical memory
of the system. You may change these settings by editing the fle brighthouse.ini within the
data directory.
Important: The memory settings assume that there are no other services on the machine
consuming signifcant memory. If this is not the case, please lower the memory
settings for Infobright.
INFOBRIGHT 3.4.2 USER G UIDE
2 SETTING UP INFOBRIGHT 6

See “Recommended Memory Confgurations” later in this chapter.


5. To uninstall Infobright, run:

rpm -e infobright
or
dpkg -r infobright

TAR Install
To install Infobright on Linux using the tarball package:
1. Obtain root user access
2. Change to the parent location in which you want to install (e.g. /usr/local)

cd /usr/local

Important: Do not install in the root or home directories due to possible MySQL permission
checking issues during install, start up, and/or load.
3. Unpack the tarball, which will create the product directory (e.g. infobright-3.4.2-p1-
x86_64_ice and create a symbolic link ‘infobright’ to the product folder

gunzip < /path/to/infobright-3.4.2-p1-x86_64_ice.tar.gz | tar xvf -


ln -s /usr/local/infobright-3.4.2-p1-x86_64_ice infobright
cd /usr/local/infobright

4. Run the install script with the “--help” fag to check for system confguration and provide
examples of directory parameters

./install-infobright.sh –help

Parameters required:

--datadir=infobright data folder


[--datadir=/usr/local/infobright/data]
--cachedir=infobright cache folder
[--cachedir=/usr/local/infobright/cache]
--config=mysql conf file to be created
[--config=/etc/my-ib.cnf]
--port=infobright server port [--port=5029]
--socket=socket file to be used by this server
[--socket=/tmp/mysql-ib.sock]
--user=user to be created if not exist [--user=mysql]

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 7

--group=user group to be created if not exist [--group=mysql]

INFOBRIGHT COMMAND-LINE PARAMETERS


Parameter Description

Datadir Path to the directory where tables will be created and stored. Use
a high-performance storage such as a RAID.

Cachedir Path to the directory where temporary fles will be created and
stored. Should be located on a fast drive, possibly not the same as
the data. Allow at least 100 GB of free space (depending on
database size).

Port Listening port for the Infobright server instance.

Confg MySQL confguration fle. (The confguration fle will be created


with defaults during the Infobright installation.)

Socket Socket connection point for client connections. (The socket


connection point will be created during the Infobright
installation.)

User System user who can run the Infobright server instance. User will
be created if it does not exist. The default user is mysql.

Group System group for the above user. Group will be created if it does
not exist. The default group is mysql.

5. Run the install script again, this time with directory parameters. If parameters are used
that already exist, an error will occur (for example running the same script with
parameters twice)
Example command

./install-infobright.sh --datadir=/usr/local/infobright/data
--cachedir=/usr/local/infobright/cache --port=5029 --config=/etc/my-ib.cnf
--socket=/tmp/mysql-ib.sock --user=mysql --group=mysql

6. Change the default memory confguration by editing the fle brighthouse.ini within the
data directory. See “Recommended Memory Confgurations” later in this chapter.
Important: It is critical that you increase the memory settings for systems running more than
2GB of physical memory or performance will be severely impacted.

Window s Upgrade Instructions


1. Please follow the Windows Installation Instructions. The Install Wizard automatically
detects a previous version of ICE and upgrades your ICE installation while preserving
your data and confguration settings. The install procedure automatically runs the
Confguration Manager.

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 8

2. Stop the Infobright server from the Start Menu items.


3. Run the Charset Migration Tool from the Windows command line:
cd "C:\Program Files\Infobright\bin"
chmt.exe –datadir=\absolute\path\to\data\directory

4. Start the Infobright server from the Start Menu items.


5. Create or ensure that the directory c:\tmp exists (necessary for step 6)
6. Run the MySQL Upgrade utility from the Windows command line:
cd "C:\Program Files\Infobright\bin"
mysql_upgrade.exe --defaults-file="c:\Program Files\Infobright\my-ib.ini"
-uroot --tmpdir=c:\tmp

Important: The MySQL Upgrade utility may display several errors regarding the use of locks
with log tables and errors requiring table upgrades. The errors are all handled
automatically by Infobright and/or the upgrade utility and can be ignored.

7. Stop and start the Infobright server from the Start Menu items.

Linux Upgrade Instruction s


RPM or DPKG Upgrade
To upgrade using the rpm or deb package, simply run the installation command and the
package will automatically identify that Infobright is already installed and switch to upgrade
mode. Your confguration settings and data will not be changed during the upgrade.
Important: If the previous installation was done using the tarball package, you must
upgrade using the tarball package (see instructions below) or contact Infobright
Support to move from a tar install to a package install.
To upgrade Infobright on Linux using the rpm or deb package:
1. Obtain root user access
2. Run either:
rpm -U infobright-version-platform.rpm
dpkg -i infobright-version-platform.deb

3. Change to user mysql


4. Run the Charset Migration Tool (as user mysql):
cd /usr/local/inforbright
./bin/chmt –datadir=/absolute/path/to/data/directory

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 9

5. Start the Infobright server:


/etc/init.d/mysqld-ib start

6. Run mysql upgrade tool to upgrade data folder:


cd /usr/local/inforbright
./bin/mysql_upgrade --defaults-file=/etc/my-ib.cnf --user=root --tmpdir=/tmp

Important: The MySQL Upgrade utility may display several errors regarding the use of locks
with log tables and errors requiring table upgrades. The errors are all handled
automatically by Infobright and/or the upgrade utility and can be ignored.

7. Re-start the Infobright server:


/etc/init.d/mysqld-ib restart

/usr/local/infobright/bin/mysqld --version

TAR Upgrade
To upgrade Infobright on Linux using the tarball package:
1. Unpack the tarball into a temporary folder. Use the gunzip utility for unpacking:
cd /path/to/temp/
gunzip < /path/to/infobright-3.4.2-x86_64.tar.gz | tar xvf -

2. Stop the Infobright server:


/etc/init.d/mysqld-ib stop

3. Run the install script with the "--upgrade” and “--confg” fags and pass in the
confguration fles of the previously installed version:
./install-infobright.sh --upgrade --config=/etc/my-ib.cnf

4. Run the Charset Migration Tool (as user mysql):


cd /usr/local/inforbright
./bin/chmt –datadir=/absolute/path/to/data/directory

5. Start the Infobright server and run the mysql_upgrade utility:


/etc/init.d/mysqld-ib start
cd /usr/local/infobright
./bin/mysql_upgrade --defaults-file=/etc/my-ib.cnf --user=root --tmpdir=/tmp

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 10

Important: The MySQL Upgrade utility may display several errors regarding the use of locks
with log tables and errors requiring table upgrades. The errors are all handled
automatically by Infobright and/or the upgrade utility and can be ignored.

6. Re-start the Infobright server:


/etc/init.d/mysqld-ib restart

7. Confrm the build version as IB_3.4.2_r8940_9191:


/usr/local/infobright/bin/mysqld –-version

Configuring Infobright
The Infobright confguration fle is called brighthouse.ini and is located in the data
subdirectory within your Infobright installation directory. The confguration fle is a text fle
containing the Infobright confguration parameters. See the Infobright installation package
for a sample brighthouse.ini fle.
Important: It is critical that you specify increased memory settings for systems running more
than 2GB of physical memory to ensure optimal performance.
Each parameter is shown on a separate line and uses the following form:
ParameterName=ParameterValue
If a parameter is not present in the confguration fle or if the confguration fle does not exist,
the default values are used. Blank lines and comments (lines starting with #) are ignored.
Be sure to customize the following parameters to optimize performance. These parameters
are case-sensitive and must be typed as shown.

INFOBRIGHT TUNING PARAMETERS


Parameter Syntax Value Description

ServerMainHeapSize=size Not less than 320 Size of the main memory heap in
Default: 600 the server process, in MB. The
larger the heap size, the more
effectively the server works.
However, the sum of the heap
sizes in the server and the loader
should not exceed physical
memory installed in the machine,
otherwise performance decreases
radically.

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 11

INFOBRIGHT TUNING PARAMETERS


Parameter Syntax Value Description
LoaderMainHeapSize=size Not less than 320 Size of the memory heap in the
Default: 320 loader process, in MB. The sum of
the heap sizes in the server and
the loader should not exceed
physical memory installed in the
machine, otherwise performance
decreases radically.

CacheFolder=directory Directory name This is a mandatory parameter.


Default: none Path to the directory where
temporary fles will be created
and stored. This is set as one of
the install script parameters.

AllowMySQLQueryPath= 0, 1 Set to 1 to allow queries that are


number Default: 0 not supported in the Infobright
Optimizer to be handled by the
MySQL query engine. Queries
that take the MySQL path will
have reduced performance.

Note: The values are commented out (preceded by #) in the brighthouse.ini fle which
causes them to default to the application minimum allowed values of 600 and 320 for
ServerMainHeapSize and LoaderMainHeapSize respectively.

INFOBRIGHT ADDITIONAL PARAMETERS


Parameter Syntax Value Description

KNFolder=directory Directory name Directory where the Knowledge


Default: Grid is stored. If not specifed,
BH_RSI_Repository these fles are located in a
subdirectory of the data
directory. Allow free space of at
least 1% of database size
(compressed).

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 12

INFOBRIGHT ADDITIONAL PARAMETERS


Parameter Syntax Value Description
ControlMessages=number 0, 1, 2, 3 Set to 2 to turn the control
Default: 0 messages on with timestamps.
This is usually needed by
Infobright to support
performance investigation.
1 removes timestamp and
session number information, and
generally not used in favour of 2.
3 is new in Infobright 3.4 and
adds information for resource
management (total and free
memory and CPU cores).

INFOBRIGHT 3.4.2 USER G UIDE


2 SETTING UP INFOBRIGHT 13

C onfiguration Tips and Example s


Important: You should confgure memory settings to ensure optimal performance.
The following table shows sample memory confgurations for different systems.

RECOMMENDED MEMORY CONFIGURATIONS


System Server Main Heap Size Loader Main Heap Size
Memory

64GB 48000 800

48GB 32000 800

32GB 24000 800

16GB 10000 800

8GB 4000 800

4GB 1300 400

2GB 600 320

In most cases, the loader does not beneft from larger memory settings. However, increasing
the LoaderMainHeapSize can help when:
• a table to be loaded has very long text values, or
• the table has many columns (e.g., 1000 columns).
You can use more memory at import if you are planning to execute several concurrent load
tasks to different data tables. However, disk access may become a bottleneck.
ServerMainHeapSize should be as large as possible but safely smaller than the amount of
physical memory in the machine. If performance decreases because of memory swapping by
the operating system, try to set lower heap sizes. We also recommend decreasing the heap
size if many users are running queries in parallel.
Important: Infobright may use additional memory for heavy loads or queries. Also, other
applications on your server will use memory for their processes. It is important
that the total of ServerMainHeapSize and LoaderMainHeapSize is less than the
total available physical memory. If the system needs to swap memory,
performance will be severely impacted.

INFOBRIGHT 3.4.2 USER G UIDE


3 USING INFOBRIGHT 14

3. Using Infobright
Starting and Stopping the Infobright Server
Windows
The Windows Install Wizard automatically creates Infobright as a Windows Service, which
allows the Infobright server to be started and stopped automatically when you boot or
shutdown Windows.
 To manually start the Infobright server, from the Windows Start Menu run:

Start/All Programs/Infobright/Infobright Start

 To manually stop the Infobright server, from the Windows Start Menu run:

Start/All Programs/Infobright/Infobright Stop

Linux
You can start and stop the Infobright server the same way you would start and stop the
original MySQL server (mysqld). Before using the Infobright server, see “Starting and
Stopping MySQL Automatically” in the MySQL 5.1 Reference Manual.
Important: It is recommended that you run Infobright using MySQL user credentials rather
than root for security reasons.
 To start the Infobright server on Linux, run:
/etc/init.d/mysqld-ib start
/etc/init.d/mysqld-ib start

 To start/stop the Infobright server during system boot/shutdown use the mysqld-ib
script in /etc/init.d/ for start and stop services. Use run level 2 3 4 5 to start the service,
and run level 0 1 6 to stop.
 The following are sample commands to create services:

(Ubuntu) update-rc.d mysql-ib.server start 99 2 3 4 5 . stop 01 0 1 6 .


(CentOS) chkconfig --add mysqld-ib
chkconfig --level 2345 mysqld-ib on
chkconfig --level 016 mysqld-ib off

INFOBRIGHT 3.4.2 USER G UIDE


3 USING INFOBRIGHT 15

Working with the Infobright Server


You can use the tools provided with MySQL, such as the mysql client program, with the
Infobright server. For more information, see “Tutorial” in the MySQL 5.1 Reference Manual..
You can also use GUI tools, such as the MySQL Workbench provided by MySQL AB, to query
Infobright databases in a more graphical manner.
You can use the mysql client program to perform the following actions. For more
information, see “Tutorial“ in the MySQL 5.1 Reference Manual.

Windows
 To connect to the Infobright command line interface, run :

Start/All Programs/Infobright/Infobright Command Line Client

To enable remote connections to Infobright you need to grant connection permissions in


Infobright. From within the mysql shell run the following grant privileges commands:

mysql> grant all privileges on *.* to ‘root’@‘localhost’ with grant option;


Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on *.* to ‘root’@’%’ with grant option;


Query OK, 0 rows affected (0.00 sec)

Linux
 If you used the standard install locations, enter the following command to connect to
Infobright:

/usr/bin/mysql-ib

If you used a different install location, modify the above command to point to your socket
fle.
 When the Infobright server is frst installed, an administrator account with no password is
created. To connect to the administrator account, use the following command:

mysql-ib

 To run a script when connecting to the administrator account, use the following
command:

mysql-ib < input_script_name.txt

INFOBRIGHT 3.4.2 USER G UIDE


3 USING INFOBRIGHT 16

For example:
mysql-ib < /tmp/testing/input.txt

 To run a script when connecting to the administrator account and direct all output to a
text fle, use the following command:
mysql-ib < input_script_name.txt > output_results.txt

For example:
mysql-ib < /tmp/testing/input.txt > /tmp/testing/output.txt

During the Infobright server shutdown process, the server will not shut down until all
running commands are completed.
 To force the shutdown of the server:
Kill the mysqld process and all running bhloader processes.
Infobright can be used with most Business Intelligence tools and any MySQL GUI client tool
like Toad or Navicat. Simply point to the IP address and socket number for the Infobright
server, and logon using any user credentials that have been set up.

Checking the Infobright Version


You can use the following methods to check the version of the Infobright system.
 Enter the following command at the command prompt:
/usr/local/infobright/bin/mysqld –-version

 After connecting to the Infobright administrator account, enter the following command at
the mysql command prompt:
mysql> show variables like "version_comment";

The Infobright version will be shown. For example:


mysql> show variables like "version_comment";

+-----------------+-------------------------------------------------+
| Variable_name | Value |
+-----------------+-------------------------------------------------+
| version_comment | build number (revision)=IB_3.4.2_r5IB_3.2_GA_5316 |
+-----------------+-------------------------------------------------+
1 row in set (0.00 sec)

INFOBRIGHT 3.4.2 USER G UIDE


3 USING INFOBRIGHT 17

Infobright is the default Storage Engine


 The Infobright storage engine (named “Brighthouse”) should always be used when
working with Infobright data. This is the default setting created when using the installer.
 To view all available engines, enter the following command:
mysql> show engines;
The following information is displayed at the command prompt. In this example, MyISAM is
shown as the default storage engine. You can combine the usage of different storage engines
but you should avoid joining across storage engines as this can result in sub-optimal
performance due to the use of the MySQL query engine. However it can be quite useful in
some cases to store query results in Memory or MyISAM tables and do further manipulations
of results.
mysql> show engines;
+-----------+---------+----------------------------------------------------------+-------------+---+-----------+
|Engine |Support |Comment |Transactions |XA |Savepoints |
+-----------+---------+----------------------------------------------------------+-------------+---+-----------+
|BRIGHTHOUSE|DEFAULT |Infobright storage engine |YES |NO | NO |
|MRG_MYISAM |YES |Collection of identical MyISAM tables |NO |NO | NO |
|CSV |YES |CSV storage engine |NO |NO | NO |
|MEMORY |YES |Hash based, stored in memory, useful for temporary tables |NO |NO | NO |
|MyISAM |YES |Default engine as of MySQL 3.23 with great performance |NO |NO | NO |
+-----------+---------+----------------------------------------------------------+-------------+---+-----------+
5 rows in set (0.00 sec)

About Log Files


Infobright uses the MySQL server logs and also creates several new logs. For more
information about MySQL logs, see “MySQL Server Logs” in the MySQL 5.1 Reference Manual.

INFOBRIGHT LOG FILES

Log Type Information Written to Log

Error log Errors starting, stopping and running the Infobright server
(mysqld). To generate this log, add the following lines to my.cnf:
log-error=<flename>
log-output=FILE
General query log Connection and statement information received from clients.
Infobright log Server start and stop information. Also contains missing
confguration settings.

It is possible to turn on the display of diagnostic information. By default this information is


redirected to Infobright's console, unless an error log is specifed (see table above). To turn on
diagnostic messages you have to modify your brighthouse.ini confguration fle (see
“Confguring Infobright” in Chapter 2) and set parameter ControlMessages to 1 (log actions),
2 (to add a time stamp to each message), or 3 (to add memory and CPU resource
information).
Note: In general more detail in the log may have an impact on performance so its
recommended users fnd and use the setting that strikes the best balance for them in
terms of performance versus log details.

INFOBRIGHT 3.4.2 USER G UIDE


3 USING INFOBRIGHT 18

About Errors
Infobright reports the same errors as the standard MySQL server. For more information, see
“Appendix B. Errors, Error Codes, and Common Problems” in the MySQL 5.1 Reference
Manual.
There are a few additional errors specifc to Infobright import and export commands. For
more information, see “About Import Errors” and “About Export Errors” in Chapter 7.

About SQL Command Syntax


The syntax for Infobright SQL commands is exactly the same as the syntax for MySQL
commands. For more information, see “SQL Statement Syntax” in the MySQL 5.1 Reference
Manual.
There are special considerations when using the following commands with Infobright. All
other SQL commands can be used with Infobright as they are with the standard MySQL.

USING MYSQL COMMANDS WITH INFOBRIGHT

MySQL Command More Information

CREATE TABLE, “Creating and Dropping Tables” in Chapter 4


DROP TABLE
SHOW TABLE STATUS, “Viewing Table Information” in Chapter 4
SHOW FULL COLUMNS “Viewing Compression Ratio Statistics” in Chapter 4
INSERT, UPDATE, “Unsupported Data Manipulation Commands (INSERT,
DELETE UPDATE, DELETE)”, in Chapter 5.
Important: Do not use INSERT, UPDATE, and DELETE to
manipulate Infobright data.
LOAD DATA INFILE “Infobright Import/Export Syntax” in Chapter 7
SELECT Chapter 8, “Running Queries in Infobright”
VIEW “Creating VIEWs in Infobright”, in Chapter 8

About SQL ISO Standards


As mentioned in the previous section, Infobright uses the same syntax as the standard
MySQL commands. For information about the compliance of the MySQL language with ISO
SQL standards, see “MySQL Standards Compliance” in the MySQL 5.1 Reference Manual.
Infobright is approaching full ISO SQL compliance. However, certain sections of the ISO SQL
standard are open to interpretation and each DBMS, including Infobright, may implement
these sections slightly differently. Consequently, Infobright query results may differ from
those of other databases.
For example, the SQL standard does not defne a default collation for string comparisons,
which affects the ordering of query results. Different databases will implement different

INFOBRIGHT 3.4.2 USER G UIDE


3 USING INFOBRIGHT 19

collation approaches, thus displaying inconsistent results for such things as sorts.

INFOBRIGHT 3.4.2 USER G UIDE


4 MANAGING INFOBRIGHT TABLES 20

4. Managing Infobright Tables


About the Infobright Database Files
Infobright tables are located in the data subdirectory in your Infobright installation directory.
This is the same directory structure used for standard MySQL databases and tables. For more
information, see “Installation Layouts” in the MySQL 5.1 Reference Manual.
Within the data subdirectory, Infobright databases are stored in separate subdirectories.
Within each database subdirectory, data fles for each Infobright table are stored in separate
subdirectories.
Important: Do not manually copy a data table from one database to another by copying
the database fles—internal table numbering errors and Knowledge Grid
inconsistencies may occur. To copy a table, use import and export commands (see
Chapter 7, “Importing and Exporting Data in Infobright”) or back up the entire
database directory (see Chapter 9, “Infobright Backup and Recovery”).
The Infobright server uses additional directories to store temporary data, and optimization
information, such as Knowledge Nodes. The following shows the data directory, containing
the Infobright databases:

[root@ib03 data]# pwd


/usr/local/infobright/data
[root@ib03 data]# ls

BH_RSI_Repository
Infobright.log
Infobright.seq
ib03.corp.infobright.com.err
mysql
test

INFOBRIGHT 3.4.2 USER G UIDE


4 MANAGING INFOBRIGHT TABLES 21

About Supported Data Types


The following data types are supported in Infobright. Note that numeric data types ranges
are 1 less than the MySQL minimums and maximums.

NUMERIC TYPES

Data Type Minimum Maximum

TINYINT -127 127


BOOL, BOOLEAN -127 127
SMALLINT -32767 32767
MEDIUMINT -8388608 8388607
INT (INTEGER) -2147483647 2147483647
BIGINT -9223372036854775806 9223372036854775806
FLOAT -3.402823466E+38 3.402823466E+38
DOUBLE -1.7976931348623157E+308 1.7976931348623157E+308
(DOUBLE
PRECISION)
DEC(M, D) -(1E+M – 1) / (1E+D) (1E+M – 1) / (1E+D)
(DECIMAL(M, D))
where 0 < M <= 18
and
0 <= D <= M

DATE AND TIME TYPES

Data Type Minimum Maximum Format

DATE 100-01-01 9999-12-31 YYYY-mm-dd


DATETIME 100-01-01 00:00:00 9999-12-31 23:59:59 YYYY-mm-dd
HH:MM:SS
TIMESTAMP 1970-01-01 00:00:00 2038-01-01 00:59:59 YYYY-mm-dd
HH:MM:SS
TIME -838:59:59 838:59:59 HHH:MM:SS
YEAR (4-digit format 1901 2155 YYYY
only)

INFOBRIGHT 3.4.2 USER G UIDE


4 MANAGING INFOBRIGHT TABLES 22

STRING TYPES

Data Type Maximum Length

CHAR(N) 255
VARCHAR(N) 65532
BINARY(N) 255
VARBINARY(N) 65532
TINYTEXT 255
TEXT(N) 65535

Creating and Dropping Tables


Use the standard MySQL commands to create and drop tables in Infobright, the same as you
would with a MySQL table. For detailed syntax information, see “CREATE TABLE Syntax” and
“DROP TABLE Syntax” in the MySQL 5.1 Reference Manual.
Important: Do not manually copy a data table from one database to another by copying
the database fles—internal table numbering errors and Knowledge Grid
inconsistencies may occur. To copy a table from one database to another, export
from the source database and then import into the target database (see Chapter 7,
“Importing and Exporting Data in Infobright”) or back up the entire database
directory (see Chapter 9, “Infobright Backup and Recovery”). You can rename the
entire database by renaming the folder. However, you should not copy a
database folder from one active instance to another, or within the same active
instance.
 To create a table, enter the following command:

mysql> create table < table_name> (<column(s)>);

 To drop a table, enter the following command:

mysql> drop table table_name;

See the section below, “About Column Options,” for information on supported and
unsupported options when creating columns.
Note: When creating a table, as a matter of practice one should always use the ENGINE=
option to ensure that the correct database engine is used. Infobright is shipped with
DEFAULT ENGINE = BRIGHTHOUSE, but this can be changed. The name of the
engine can be specifed explicitly at the end of create table statement:

mysql> create table <table_name> (<column(s)>) engine=brighthouse;

INFOBRIGHT 3.4.2 USER G UIDE


4 MANAGING INFOBRIGHT TABLES 23

About Column Options


NULL and NOT NULL
Infobright supports NULL and NOT NULL specifcations for columns.
• NULL allows NULL values for the column.
• NOT NULL replaces the imported NULL values with default values such as 0 (zero) for
numeric columns and an empty string (‘’) for string columns.

Lookup C olumn s
Infobright provides an additional modifer for string data type columns, called a lookup
column. The lookup column utilizes an integer substitution for values. You can declare a
lookup column on a CHAR or VARCHAR column to increase its compression and performance in
queries. However, to use a lookup column, the CHAR or VARCHAR column must meet the
following criteria:
• The column must have less than 10,000 distinct values.
• The column must contain a large number of duplicate values: the ratio of total number of
records to distinct values should be greater than 10.
Typically, a lookup column is useful for felds like state, gender, category, and the like where
the number of instances is very high, but the number of unique values is very low. To
determine the ratio of records to distinct values, determine the number of distinct values
using SELECT COUNT (DISTINCT <COLUMN>) FROM… Then compare this to the
number of records using a SELECT COUNT(<COLUMN>) FROM…
Note: Using a lookup on a column where there are more than 10,000 distinct values will
result in greatly reduced load speeds.
 To declare a column as a lookup column, add the comment 'lookup' on the column. Enter
the following command:

mysql> create table …


(…
<<column name>> <<column type>> … comment 'lookup' …
…)
engine=brighthouse;

Unsupported C olumn Options


The following column options are ignored by Infobright:
• default values

INFOBRIGHT 3.4.2 USER G UIDE


4 MANAGING INFOBRIGHT TABLES 24

• references to other tables

Unsupported Indice s Options


Infobright uses Knowledge Grid technology instead of standard indices and does not
support explicit indices. The following elements of CREATE TABLE syntax related to indices are
not allowed:
• keys
• indices
• unique columns
• auto-increment columns

Converting Oracle DDL to Infobright


If you have an existing Oracle schema defnition, you should do the following steps to make
it work on Infobright:
• Convert MEDIUMTEXT to VARCHAR (N), where ‘N’ is only as large as necessary
• Convert LONGTEXT to VARCHAR (N), where ‘N’ is only as large as necessary
• Convert DOUBLE(A,B) to DECIMAL(A,B)
• INTEGER types may be converted to BIGINT
• Convert VARCHAR2/CHAR2 to VARCHAR/CHAR

Converting SQL Server to Infobright


If you have an existing SQL Server schema defnition, you should do the following steps to
make it work on Infobright:
• Convert MEDIUMTEXT to VARCHAR (N), where ‘N’ is only as large as necessary
• Convert LONGTEXT to VARCHAR (N), where ‘N’ is only as large as necessary
• Convert DOUBLE(A,B) to DECIMAL(A,B)
• Convert MONEY to DECIMAL(18,4)
• Convert SMALLMONEY to DECIMAL(6,4)
• INTEGER types may be converted to BIGINT
• NCHAR/NVARCHAR should be converted to CHAR/VARCHAR
• Convert NUMBER to INTEGER.
• Convert NUMBER(A,B) to DECIMAL(A,B)

INFOBRIGHT 3.4.2 USER G UIDE


4 MANAGING INFOBRIGHT TABLES 25

Converting MySQL (MyISAM) to Infobright


If you have an existing MyISAM schema defnition, you should do the following steps to
ensure compliance with Infobright:
• Convert MEDIUMTEXT to VARCHAR (N), where ‘N’ is only as large as necessary
• Convert LONGTEXT to VARCHAR (N), where ‘N’ is only as large as necessary
• Convert DOUBLE(A,B) to DECIMAL(A,B)

Viewing Table Information


You can use the standard MySQL commands to obtain information about a table.
 To view column information, enter the following command:

SHOW [FULL] COLUMNS FROM tbl_name [FROM db_name] [LIKE 'pattern'];

For more information, see “SHOW COLUMNS Syntax” in the MySQL 5.1 Reference
Manual.
Utilization of the FULL option will provide an estimate of the compression for each
column

mysql> show full columns from dim_cars;

+------------+---------------+-------------------+------+-----+---------+-------+-------------------+---------------------------------------+

| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |

+------------+---------------+-------------------+------+-----+---------+-------+-------------------+---------------------------------------+

| make_id | decimal(10,0) | NULL | YES | | NULL | | select,references | Size[MB]: 0.1; Ratio: 15.64; unique |

| make_name | varchar(25) | latin1_swedish_ci | YES | | NULL | | select,references | Size[MB]: 0.1; Ratio: 5.05 |

| model_name | varchar(25) | latin1_swedish_ci | YES | | NULL | | select,references | Size[MB]: 0.1; Ratio: 1.38 |

| record_dt | datetime | NULL | YES | | NULL | | select,references | Size[MB]: 0.1; Ratio: 3.86 |

+------------+---------------+-------------------+------+-----+---------+-------+-------------------+---------------------------------------+

4 rows in set (0.01 sec)

 To view the CREATE TABLE statement used to create a given table, enter the following
command:

SHOW CREATE TABLE tbl_name;

For more information, see “SHOW CREATE TABLE Syntax” in the MySQL 5.1 Reference
Manual.

mysql> show create table dim_cars;


+----------+--------------------------------------------------------------------+
| Table | Create Table |
+----------+--------------------------------------------------------------------+
| dim_cars | CREATE TABLE `dim_cars` (
`make_id` decimal(10,0) DEFAULT NULL,

INFOBRIGHT 3.4.2 USER G UIDE


4 MANAGING INFOBRIGHT TABLES 26

`make_name` varchar(25) DEFAULT NULL,


`model_name` varchar(25) DEFAULT NULL,
`record_dt` datetime DEFAULT NULL
) ENGINE=BRIGHTHOUSE DEFAULT CHARSET=latin1 |
+----------+--------------------------------------------------------------------+
1 row in set (0.00 sec)

 To view table information, enter the following command:

SHOW TABLE STATUS [FROM db_name] [LIKE 'pattern'];

For more information, see “SHOW TABLE STATUS Syntax”in the MySQL 5.1 Reference
Manual.

mysql> show table status like 'dim_cars';

+----------+-------------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+--

| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment |

+----------+-------------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+--

| dim_cars | BRIGHTHOUSE | 10 | Compressed | 400 | 11 | 4672 | 0 | 0 | 0 | NULL |

+----------+-------------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+--

---------------------+---------------------+------------+-------------------+----------+----------------+----------------------------------+

Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |

---------------------+---------------------+------------+-------------------+----------+----------------+----------------------------------+

2008-08-28 05:30:44 | 2008-04-23 14:17:13 | NULL | latin1_swedish_ci | NULL | | Overall compression ratio: 3.622 |

---------------------+---------------------+------------+-------------------+----------+----------------+----------------------------------+

1 row in set (0.01 sec)

Viewing Compression Ratio Statistics


Infobright provides specifc statistics on table and column compression. The compression
ratio is calculated in relation to the “natural size” of uncompressed data in the table or
column. The ratio equal to n means that the compressed data, including statistics and
technical description of a column, is n times smaller than its theoretical natural size.
The following natural sizes (in bytes) are defned for various data types. Note the following:
• For all data types, if the column is not declared as NOT NULL, add one bit per value for
NULL indicators.
• These data sizes take into account the typical format of data display, for example “yyyy-
mm-dd” for DATE or decimal point for DEC. The size also counts the bytes that store the
actual text length (VARCHAR).

INFOBRIGHT 3.4.2 USER G UIDE


4 MANAGING INFOBRIGHT TABLES 27

DATA TYPES AND NATURAL SIZES

Data Type Natural Size (in bytes)

CHAR(n), BINARY(n) n*(number of rows)


BIGINT, INT, MEDIUMINT, (8 or 4 or 3 or 2 or 1 or 1)*(number of rows)
SMALLINT, TINYINT, BOOL
YEAR 4*(number of rows)
DATE 10*(number of rows)
TIME 8*(number of rows)
TIMESTAMP / DATETIME 19*(number of rows)
DEC(x,y) (x+1)*(number of rows)
FLOAT 4*(number of rows)
REAL, DOUBLE 8*(number of rows)
VARCHAR(n), VARBINARY(n) (total number of bytes used—i.e., the total length of all
strings, excluding terminating characters) + 2*(number of
rows)

Viewing Table Level C ompre s sion Ratio Statistic s


 To view the compression ratio at the table level, enter the following command:
mysql> show table status [from db_name ] [like 'table_name'];
The optional like clause can be used to flter the tables. Note that the table name must be
provided in single quotes.
The compression statistics are provided in the table comment. For example:

mysql> show table status from test like ‘t1’ \G


*********************** 1. Row **********************
Name: t1
Engine: BRIGHTHOUSE
Version: 10
Row_format: Compressed
Rows: 3430387
Avg_row_length: 0
Data_length: 0
Max_data_length: 0
Index_length: 0
Data_free: 0
Auto_increment: NULL
Create_time: 2008-09-04 15:31:39
Check_time: NULL
Update_time: 2008-09-04 15:35:30
Collation: ascii_bin
Checksum: NULL
Create_options:

INFOBRIGHT 3.4.2 USER G UIDE


4 MANAGING INFOBRIGHT TABLES 28

Comment: Overall compression ratio 39.908


1 row in set (0.59 sec)

Viewing C olumn C ompre s sion Ratio Statistic s


 To view the compression ratio and the compressed size for a column, enter the following
command:

mysql> show full columns from table_name …;

A database name and a column flter can be specifed in optional clauses. For more
information, see “SHOW COLUMNS Syntax” in the MySQL 5.1 Reference Manual.
The compression statistics are provided in the column comment. In addition to the
compression information, the comment line may also contain a “unique” indicator , meaning
that the column has all unique values (except nulls).
For example:

C omparison of Calculated C o mpre s sion Ratio to Phy sical Size


The compression ratio calculated above will differ from the compression ratio calculated
from physical sizes of fles on disk. The compression ratio based on physical size will be
slightly smaller, due to extra fles that are generated containing statistics on the imported
data, such as Knowledge Nodes. Knowledge Nodes are used to optimize query execution
and are discussed further in “About the Knowledge Grid” in Chapter 8.

INFOBRIGHT 3.4.2 USER G UIDE


5 DATA MANIPULATION S TATEMENTS 29

5. Data Manipulation Statements


Unsupported Data Manipulation Commands (INSERT, UPDATE,
DELETE)
INSERT, UPDATE, and DELETE commands are not supported in Infobright and should not be
used to manipulate data in Infobright tables. Using these commands may result in errors.
When using GUI tools with Infobright, such as MySQL Browser, use these tools in read-only
mode only. Do not use these tools to insert, update, or delete data. This may result in errors
and the hanging of the GUI application.
To insert data into Infobright tables, use the MySQL import command. For more information,
see Chapter 7, “Importing and Exporting Data in Infobright”.

INFOBRIGHT 3.4.2 USER G UIDE


6 CHARACTER SET SUPPORT 30

6. Character Set Support


Supported Character Sets
Infobright storage supports all ANSI and UTF-8 character sets. This means that Infobright can
store and retrieve data encoded in 8-bit and multi-byte character sets.

Important: Queries that evaluate against UTF-8 character data columns will execute with
less performance than and equivalent query against ASCII character data, due to
ASCII support of Character Maps in the Knowledge Grid (see Chapter 8). UTF-8
specifc Knowledge Grid extensions will be available in an upcoming release.

Collations and Comparisons


Infobright supports all custom UTF-8 collations supported by MySQL 5.1:
• utf8_bin • utf8_polish_ci
• utf8_czech_ci • utf8_roman_ci
• utf8_danish_ci • utf8_romanian_ci
• utf8_esperanto_ci • utf8_slovak_ci
• utf8_estonian_ci • utf8_slovenian_ci
• utf8_general_ci (default) • utf8_spanish2_ci
• utf8_hungarian_ci • utf8_spanish_ci
• utf8_icelandic_ci • utf8_swedish_ci
• utf8_latvian_ci • utf8_turkish_ci
• utf8_lithuanian_ci • utf8_unicode_ci*
• utf8_persian_ci
*utf8_unicode_ci properly handles both French and German collation, so specifc collation
types for these languages are not necessary.
For more information, see “Unicode Support” in the MySQL 5.1 Reference Manual.
The SQL standard does not defne a default collation; therefore, many DBMS engines have
different default collations and produce different results. As a result, there are several
differences between Infobright and other DBMS engines.
• For Infobright, character data types are case-sensitive. For example, the condition
‘toronto’=‘Toronto’ is not true in Infobright. Similarly, the condition LIKE ‘Abc%’ is
not true for ‘abcde’.
• The Infobright sorting order is “A…Z a…z” (for example ‘Zeta’ < ‘alfa’ ), which is the
same sorting order as used by Oracle. The Infobright sorting order is different than the
default MySQL sorting order, which mixes lowercase and uppercase; the SQL Server
order, which is “aAbB…zZ”; and the DB2 order, which is “AaBb…Zz”.
• The Infobright sorting order affects ORDER BY results, GROUP BY results (which is the order
INFOBRIGHT 3.4.2 USER G UIDE
6 CHARACTER SET SUPPORT 31

of groups and their defnitions—for example, ‘aaa’ and ‘AAA’ defne different groups)
and DISTINCT results. WHERE conditions may also be affected if you are expecting a
different sorting order than the one used by Infobright.
• To simulate Infobright collation in the MySQL engine, set latin1_bin collation while
creating a table (for more information, see “Table Character Set and Collation” in the
MySQL 5.1 Reference Manual). Enter the following command:

mysql> create table … collate ascii_bin;

Padding
Infobright treats padding differently than other DBMS engines. Infobright assumes literal
comparisons of text felds, including all whitespace characters. Therefore, a string containing
two spaces is different than a string containing one space or an empty (0 length) string, which
is also different than the NULL value.
The Infobright padding defnition is compatible with the SQL standard. However, most
DBMS systems have defned less restricted, customizable rules regarding text comparison.
For example, ‘abc ’ = ‘abc’ may be true in some databases but is not true in Infobright.
Note: In CHAR columns, trailing spaces are trimmed on LOAD, whereas in VARCHAR
columns values are loaded with all spaces.

INFOBRIGHT 3.4.2 USER G UIDE


7 IMPORTING AND EXPORTING DATA IN INFOBRIGHT 32

7. Importing and Exporting Data in


Infobright
About Transactions
Using AUTOCOMMIT, COMMIT and ROLLBACK C ommands
By default, Infobright uses AUTOCOMMIT mode to fnalize transactions, meaning that every
transaction is either automatically committed or rolled back if an error occurs. However, you
can and should choose to disable AUTOCOMMIT and use COMMIT and ROLLBACK commands
instead.
A new transaction starts with the frst LOAD command or DML statement entered in a new
Infobright session. A new transaction also starts after each COMMIT or ROLLBACK command.
 To enable the use of COMMIT and ROLLBACK commands in Infobright, you must disable
AUTOCOMMIT. Enter the following command:
mysql> set autocommit=0;

You can disable AUTOCOMMIT by setting the parameter to 0 (zero) and enable AUTOCOMMIT by
setting the parameter to 1. If AUTOCOMMIT is set to 1, then when a LOAD is completed, the
transaction is automatically committed.
 To commit the current transaction, enter the following command:
mysql> commit;

 If you have not yet committed a LOAD DATA INFILE transaction, you can rollback the
transaction. This will restore the import tables to the state that existed before the current
transaction. Enter the following command:
mysql> rollback;

Using COMMIT and ROLLBACK makes it possible to check the load within the same session
before committing the data, as the loaded data is available (viewable) to the load session. For
instance, you could check something about the data (number of records load) before
committing.
After importing data using the LOAD DATA INFILE command, the status of the import and the
number of affected rows is shown. All uncommitted rows, including those from previous
imports, are shown; therefore, the number of affected rows may be greater than the number
of rows in the fle you just imported.

INFOBRIGHT 3.4.2 USER G UIDE


7 IMPORTING AND EXPORTING DATA IN INFOBRIGHT 33

About Transaction Behavior


While a write operation is being performed on a table, the following occurs:
• Queries to the table are not executed until the current import is complete and the
operation is committed.
• Until the current write operation is committed, all subsequent write commands to the
table are queued. They will wait for the write lock to be released before proceeding in the
order they were received.

While a read query is being executed on a table, the following occurs:


• All subsequent queries run concurrently with the current query.
In general, Infobright uses table level locking where only one LOAD operation can execute at
one time and after queries have completed.

Failure Handling
If AUTOCOMMIT is disabled and the Infobright server is terminated during an import session,
the following occurs:
• Infobright does not store the rows that were loaded during the failed import operation.
• The input fle and the database fles are not harmed. To load data from the input fle,
repeat the LOAD operation.
If AUTOCOMMIT is disabled and the Infobright server is terminated after an import session is
completed successfully but is not committed, the following occurs:
• The transaction is rolled back and the imported data is lost when the server restarts.
• The input fle and the database fles are not harmed by the failed import operation (the
database is unaffected, as if the import session did not occur). To re-import the data,
repeat the LOAD operation.
If the Infobright server is terminated during an export operation to a disk fle, the following
occurs:
• A non-empty fle is saved on disk; however, the last row in the saved fle is inconsistent.
• The database fles are not harmed by the failed export operation. To export the data,
repeat the export operation.
If Infobright tries to import data from a fle created during a failed export session, the
following occurs:
• No data is inserted because the input fle consists of corrupted table rows. No new
records are added to the database fles, so no harm is done.

About Export Differences in Infobright


There are several important differences between exporting data from Infobright and
exporting data from other DBMS engines.

INFOBRIGHT 3.4.2 USER G UIDE


7 IMPORTING AND EXPORTING DATA IN INFOBRIGHT 34

CHAR(n) Data Type Value s


In Infobright, when you export CHAR(n) data type values to a text fle, the extra spaces are
trimmed from the export.

Es cape Characters
The Infobright Loader supports escape character defnition and usage.

Exporting NULL Value s


Infobright recognizes the following representations of NULL values when loading data from a
text fle:

NULL, \N, <field delimiter><field delimiter>

However, Infobright only exports NULL values in the following representation:

<field delimiter><field delimiter>

Other DBMS systems may have different representations of the NULL value; for example,
MySQL only recognizes the representation \N for a NULL value. This can create issues if you
export data from Infobright and import the data into MySQL. Since MySQL will only look for
\N and will not recognize the Infobright representation of the NULL value, MySQL will change
the NULL value into the default values in numeric and string columns.

Infobright Import/Export Syntax


Importing Data
 To import data into an Infobright table, use the following MySQL loading command:

LOAD DATA INFILE ' file_name' INTO TABLE tbl_name


[FIELDS
[TERMINATED BY ' char']
[ENCLOSED BY ' char']
[ESCAPED BY ' char']
];
where:
file_name = path to the fle to be loaded
tbl_name = name of the table where the data will be loaded

INFOBRIGHT 3.4.2 USER G UIDE


7 IMPORTING AND EXPORTING DATA IN INFOBRIGHT 35

Exporting Data
 To export data from an Infobright table, use the following MySQL export command:

SELECT … INTO OUTFILE ' file_name'


[FIELDS
[TERMINATED BY ' string']
[ENCLOSED BY ' char']
[ESCAPED BY ' char']]
FROM 'tbl_name';
where:
file_name = path to the fle where data will be exported
tbl_name = name of the table from which the data will be retrieved
For more information on export syntax, see “SELECT Syntax” in the MySQL 5.1 Reference
Manual.

Optional FIELDS Clause


Several optional clauses exist for the MySQL LOAD command. All of these clauses are ignored
by Infobright, with the exception of the FIELDS clause. You can also use the FIELDS clause
when exporting data.
You can use the optional FIELDS clause to specify how values are provided in the input fle.
To use the FIELDS clause, the following must be true:
• The data import format must be defned as variable-length text.
Within the FIELDS clause, you can use the following sub clauses:
• Use the TERMINATED BY sub clause to specify the character recognized as the separator
(delimiter) between values. By default, a semicolon ';' is assumed to separate values.
• Use the ENCLOSED BY sub clause to specify the character that begins and ends each string
representing a text value. By default, a double quotation mark '"' is assumed to enclose
each value. If the text values in the input fle do not use any enclosing characters, use the
value 'NULL' in the ENCLOSED BY sub clause. Note that this is the same as using the empty
string '' option in standard MySQL.
• Use the ESCAPED BY sub clause to support special characters that may be imbedded
within text felds.

Importing Files with Invalid Value s


Infobright may abort a load when invalid values are found. Certain invalid values, however,
can be loaded in Infobright. The following rules are used with invalid data:
• If a numeric, date or time value is invalid, the value is replaced by 0.
• If a NULL value is imported into a column defned as NOT NULL (except for TIMESTAMP
columns), it is replaced by 0 (for numerical, date and time columns) or by an empty
string (for string columns).

INFOBRIGHT 3.4.2 USER G UIDE


7 IMPORTING AND EXPORTING DATA IN INFOBRIGHT 36

Importing Data Using Linux Pipes


You can use Linux pipes when importing data in Infobright. The same dataformat parameter
applies, see “Setting Import and Export Parameters.” You can also use the FIELDS clause
when exporting data. For more information, see “Optional FIELDS Clause" in Chapter 7.
To set up a Linux pipe, you need to run the mkfifo command from Linux, and ensure that the
pipe is accessible to Infobright. In the following example the pipe is setup as
/pipe_test/thepipe.pipe . You can use the directory and name of your choice.

mkfifo /pipe_test/thepipe.pipe
chmod 666 /pipe_test/thepipe.pipe

Once the pipe is set up, direct the data either by directing a fle or a process to the pipe:

cat /usr/tmp/jkvarload.txt > /pipe_test/thepipe.pipe &

Then execute a LOAD DATA INFILE statement using the pipe:

mysql> load data infile '/pipe_test/thepipe.pipe' into table pipe_test lines


terminated by '\n';

When fnished, remember to remove the pipe:

rm thepipe.pipe

About Import Errors


There are several possible Infobright-related errors that could occur when using the LOAD
DATA command on a Infobright table. These errors are described in the following table.
Standard MySQL errors may also occur (for more information, see “Appendix B. Errors, Error
Codes, and Common Problems” in the MySQL 5.1 Reference Manual).

INFOBRIGHT IMPORT ERRORS

Code Message Description Action

1 Cannot open fle or Cannot open a fle or a pipe Ensure the fle exists and the path is
pipe containing input data entered correctly
2 Wrong data or Format of data does not Ensure the data being imported is
column defnition comply with table defnition the correct data type and does not
exceed the size specifed
3 Syntax error Not used N/A

INFOBRIGHT 3.4.2 USER G UIDE


7 IMPORTING AND EXPORTING DATA IN INFOBRIGHT 37

INFOBRIGHT IMPORT ERRORS

Code Message Description Action


4 Cannot connect to Not used N/A
the database
5 Unknown error Unspecifed error occurred Contact customer support
6 Wrong parameter Wrong value for one of the Make sure the correct parameter is
loading parameters used (see “Setting Import and
Export Parameters”)
7 Data conversion A value in data cannot be Ensure the data is the correct
error converted to a column type column type

About Export Errors


There are several possible Infobright-related errors that could occur when exporting data
from a Infobright table. These errors are described in the following table. Standard MySQL
errors may also occur (for more information, see “Appendix B. Errors, Error Codes, and
Common Problems” in the MySQL 5.1 Reference Manual).

INFOBRIGHT EXPORT ERRORS

Code Message Description Action

1 Cannot open fle or Can not open a fle or a Ensure the fle exists and the path is
pipe pipe for output entered correctly
2 Wrong data or Not used Ensure the data being exported is the
column defnition correct data type and does not exceed
the size specifed
3 Syntax error Not used Check the export syntax
4 Cannot connect to the Not used Ensure database exists, the correct
database path is given and Infobright is
started
5 Unknown error Unspecifed error Contact customer support
occurred
6 Wrong parameter Wrong value for one of Make sure the correct parameter is
the export parameters used (see “Setting Import and Export
Parameters”)
7 Data conversion error Not used Ensure the data is the correct column
type

INFOBRIGHT 3.4.2 USER G UIDE


7 IMPORTING AND EXPORTING DATA IN INFOBRIGHT 38

Sample Script (Create Table, Import Data, Export Data)


The following sample script creates a table called customers, sets Infobright as the default
engine, imports data from an existing text fle and exports the data.

USE Northwind;

DROP TABLE IF EXISTS customers;


CREATE TABLE customers (
CustomerID varchar(5),
CompanyName varchar(40),
ContactName varchar(30),
ContactTitle varchar(30),
Address varchar(60),
City varchar(15)
Region char(15)
PostalCode char(10),
Country char(15),
Phone char(24),
Fax varchar(24),
CreditCard float(17,1),
FederalTaxes decimal(4,2)
) ENGINE=BRIGHTHOUSE;

-- Import the text file.


Set AUTOCOMMIT=0;

LOAD DATA INFILE "/tmp/Input/customers.txt" INTO TABLE customers FIELDS


TERMINATED BY ';' ENCLOSED BY 'NULL' LINES TERMINATED BY '\r\n';
COMMIT;

-- Export the data


-- Export the data into BINARY format.
-- Export the data into TEXT format.
SELECT * INTO OUTFILE "/tmp/output/customers.text" FIELDS TERMINATED BY ';'
ENCLOSED BY 'NULL' LINES TERMINATED BY '\r\n' FROM customers;

INFOBRIGHT 3.4.2 USER G UIDE


7 IMPORTING AND EXPORTING DATA IN INFOBRIGHT 39

Exporting and Importing Query Results


After exporting the results of a query to an output fle, you may not be able to import the fle
back into the same defnition of the accessed table. This is because the query may contain
aggregates that will produce values beyond the boundaries of the original data types. In
order to load the output fle, you may need to create a new table with the appropriate data
types for the values to be imported.
The following table shows the required data type conversions.

DATA TYPE CONVERSIONS

Operation Column Data Type Results Data Type


SUM Tinyint BigInt
Smallint
Mediumint
Int
BigInt
Float Double
Double
Decimal(N, M) Decimal(18, M)
AVG Tinyint Double
Smallint
Mediumint
Int
BigInt
Float
Double
Decimal(N, M)
COUNT Tinyint Decimal(18, 0)
Smallint
Mediumint
Int
BigInt
Float
Double
Decimal(N, M)

INFOBRIGHT 3.4.2 USER G UIDE


8 RUNNING QUERIES IN INFOBRIGHT 40

8. Running Queries in Infobright


About the Knowledge Grid
The Knowledge Grid is a set of Infobright metadata used by the Infobright storage engine
(named “Brighthouse”) to optimize query execution. The Knowledge Grid consists of
Knowledge Nodes, which are optimization data for particular tables and columns.
Knowledge Nodes are stored on disk in a special directory, specifed in the brighthouse.ini
confguration fle (see “Confguring Infobright” in Chapter 2). Knowledge Nodes can be lost
without losing data integrity.

About Knowledge Node s


There are four kinds of Knowledge Nodes:

INFOBRIGHT KNOWLEDGE NODES

Knowledge Node Type Description

Histogram Used by Infobright to enhance the speed of most queries


consisting of numerical conditions (including date/time,
decimal, etc.).
Histograms are created automatically during data load.
Character Map Used by Infobright to enhance the speed of most queries
consisting of text conditions.
Character Maps are created automatically during data load.
Pack/Pack Used to enhance joining of tables. Created or updated
automatically while executing user queries.
DPN (Data Pack Nodes) Statistical metadata that describes the content of the Data
Pack. Used to both assist in data access and in rough
operations.
DPNs are created automatically during data load.

Running Queries
 To run queries on Infobright tables, use the following standard MySQL syntax:

mysql> select …;

The Infobright Optimizer is the primary engine used to resolve queries. While signifcant
INFOBRIGHT 3.4.2 USER G UIDE
8 RUNNING QUERIES IN INFOBRIGHT 41

additions have been made to the library of supported SQL, there are cases where the query
will still be executed by the MySQL query engine instead of the Infobright engine. In this
event, query response time tends to suffer due to the fact that the MySQL engine is row-
oriented and therefore cannot make use of the Knowledge Grid information, and in some
cases it can be too slow to be usable. For best performance, ensure your queries (and VIEWs)
contain only syntax supported by the Infobright Optimizer. For more information, see
"Appendix A - Infobright Optimizer – Supported Functions and Operators" for select syntax
supported in Infobright"

Enabling Queries to be Redirected to the MySQL Engine


By default, executing queries in MySQL query engine is disabled. You can enable queries that
cannot be handled by the Infobright Optimizer to be redirected to the MySQL query engine
by editing the fle brighthouse.ini within the data directory:

AllowMySQLQueryPath=1

If the MySQL query path is disabled, then the following message will be returned if the query
would have otherwise been directed to MySQL for processing:
The query includes syntax that is not supported by the Infobright Optimizer.
Infobright suggests either restructure the query with supported syntax, or
enable the MySQL Query Path in the brighthouse.ini file to execute the query
with reduced performance.

Viewing Querie s Redirected to the MySQL Engine


When a query is redirected from the Infobright Optimizer to the MySQL query engine, a
warning is reported. For example:

400 rows in set, 1 warning (0.00 sec)

This will occur when functions not optimized in Infobright are used. If you get poor query
performance, you should execute the command below to identify if a query has been directed
to the MySQL query engine.
 After running a query, enter the following command to view any warnings:

mysql> show warnings;

The following message indicates that the query was directed to MySQL for processing:

1105 | Query syntax not implemented in Brighthouse, executed by MySQL engine.

Important: When queries are executed on Infobright tables by the standard MySQL engine,
performance can be signifcantly slower than when queries are executed by
Infobright .

INFOBRIGHT 3.4.2 USER G UIDE


8 RUNNING QUERIES IN INFOBRIGHT 42

Terminating a Query
If you want to terminate a query executed from a client session before the query is complete,
do the following:
1. Use the show [full] processlist command to determine the query’s process ID.
2. Use the kill <id> command to terminate the query.
OR
If you are using a command-line MySQL client, you can also use Ctrl+C to terminate the
query.

Creating VIEWs in Infobright


Infobright supports the creation of VIEWs. Please note that the VIEW must contain
Infobright optimized syntax, or the VIEW will be run in the MySQL query engine.

Create VIEW S yntax


The syntax to create a VIEW is as follows:

CREATE
[OR REPLACE]
VIEW view_name [(column_list)]
AS select_statement

A VIEW must contain unique column names. If you select two columns with the same name
from separate tables, at least one must be aliased or the column list option must be used.

If the View’s select statement contains functionality that is not supported in the Infobright
optimizer, then the VIEW will perform sub-optimally since it will always fip over to the
MySQL query engine.

Select Syntax Supported in Infobright


The following SELECT syntax is supported in Infobright.

S ele ct S yntax
For more information, see “SELECT Syntax” in the MySQL 5.1 Reference Manual.

SELECT [ ALL | DISTINCT | DISTINCTROW ]


Select_expr , …
[ FROM table_references
[ WHERE where_condition ]
[ GROUP BY {col_name | expr | position} ]
INFOBRIGHT 3.4.2 USER G UIDE
8 RUNNING QUERIES IN INFOBRIGHT 43

[ HAVING where_condition ]
[ ORDER BY {col_name | expr | position } [ ASC | DESC ], … ]
[ LIMIT { [ offset,] row_count | row_count OFFSET offset} ]
[ INTO OUTFILE ‘file_name’ export_options
- AS alias_name
- ORDER BY NULL ]

Join S yntax
For more information, see “JOIN Syntax” in the MySQL 5.1 Reference Manual.
Infobright supports the following JOIN syntax for the table_references part of SELECT
statements (as described in the previous section, “Select Syntax”):

table_references: table_reference [, table_references]


table_reference: table_factor | join_table

table_factor:
tbl_name [ [ AS ] alias]

join_table:
table_reference [ INNER | CROSS ] JOIN table_factor [join_condition]
| table_reference STRAIGHT_JOIN table_factor
| table_reference STRAIGHT_JOIN table_factor ON condition
| table_reference {LEFT|RIGHT} [OUTER] JOIN table_reference join_condition

Join_condition:
ON conditional_expr | USING (column_list)

Union S yntax
For more information, see “UNION Syntax” in the MySQL 5.1 Reference Manual.

SELECT ….
UNION [ ALL | DISTINCT ] SELECT …
[ UNION [ ALL | DISTINCT ] SELECT … ]

Subquerie s
For more information, see “Subquery Syntax” in the MySQL 5.1 Reference Manual.

SELECT * FROM t1 WHERE column1 = (SELECT max(column1) FROM t2);


The following functions are also supported:
• subquery as scalar operand
• subquery with ANY, IN, SOME and ALL

INFOBRIGHT 3.4.2 USER G UIDE


8 RUNNING QUERIES IN INFOBRIGHT 44

• EXISTS and NOT EXISTS


• correlated subqueries
• subqueries in the FROM clause
• VIEWs in the FROM clause

Query Performance
Due to Infobright’s column-oriented data organization and other Infobright-specifc features,
query optimization in Infobright is slightly different than in traditional DBMS approaches.
• Infobright works well with data tables containing many columns, where only necessary
columns are accessed by query (as opposed to SELECT *). The traditional approach
suggests keeping records as small as possible (e.g., using schema normalization and table
decomposition). However, in Infobright, only necessary columns are used in calculations.
Therefore, queries with many limiting conditions on many columns of the same table are
especially well optimized in Infobright.
• In traditional DBMS systems, better performance can be achieved by creating indices. In
Infobright, Knowledge Nodes are used instead of indices (Knowledge Nodes are created
automatically). To further enhance performance, you can try to infuence the data loading
procedure by keeping similar data (e.g., for similar time frames) close together. The order
in which data are loaded may infuence both compression ratio and query speed.
• Avoid using OR in queries and, if possible, use IN instead. In some cases ORs can be
translated to UNION ALL or IN, for example: “...WHERE a=1 OR a=2... “ could be replaced
by “...WHERE a IN (1,2)... ”.
• Try to replace correlated subqueries with joins and independent subqueries.
• Executing queries in steps may also help with missing function support. For instance,
execute the bulk of query in Infobright and export the data to MyISAM table. Then
execute the function query on the result set.
To optimize your query performance, avoid the following which will result in the query
being handled by the MySQL query engine:
• Using functions or type cast operators.
• Creating queries containing mixed Infobright and MySQL tables.
• Performing comparisons or arithmetical operations on two different data types (such as
numbers and text).
• Creating JOINs with the JOIN condition defned as NOT BETWEEN.

INFOBRIGHT 3.4.2 USER G UIDE


9 INFOBRIGHT BACKUP AND RECOVE RY 45

9. Infobright Backup and Recovery


Backup Procedure
• Use the following procedures to back up Infobright.
• To back up the Infobright databases, copy the entire directory containing the Infobright
databases (usually the data subdirectory in your Infobright installation directory).
• You can take advantage of incremental backups, since only some of the database fles are
updated when new data is imported. Be sure to do a full backup occasionally.
Important: Some fles in the KNFolder are updated when queries (using JOIN) are run so be
sure to back up the KNFolder on a regular basis.

Restore Procedure
To restore the Infobright databases from a backup copy, do the following:
1. Replace the entire data directory (usually the data subdirectory in your Infobright
installation directory) with the backup copy.
2. Replace the KNFolder with the backup copy (if the KNFolder is not inside the data
directory).
Important: Do not manually modify database fles or move them from one database to
another—this may lead to data corruption and unpredictable results.

INFOBRIGHT 3.4.2 USER G UIDE


A. FUNCTIONS AND OPE RA TORS
46

A. Functions and Operators

Infobright Optimizer – Supported Functions and Operators


C omparison Function s and Operators
Equal = YES
Null safe equal <= > No (MySQL engine)
Not equal <> , != YES
Less than or equal <= YES
Less than < YES
Greater than > YES
Greater than or equal >= YES
IS No (MySQL engine)
IS NOT No (MySQL engine)
IS NULL YES
IS NOT NULL YES
BETWEEN … AND … YES (except in join conditions)
NOT BETWEEN … AND ... YES
COALESCE YES
GREATEST No (MySQL engine)
IN YES
NOT IN YES
ISNULL YES
INTERVAL No (MySQL engine)
LEAST No (MySQL engine)

Logical Operators
NOT, ! YES (except in join conditions)
AND, && YES
OR, | | YES
XOR No (MySQL engine)

INFOBRIGHT 3.4.2 USER G UIDE


A. FUNCTIONS AND OPE RA TORS
47

C ontrol Flow Function s


CASE YES
IF YES
IFNULL YES
NULLIF YES

String Function s
ASCII YES
BIN YES
BIT_LENGTH YES
CHAR No (MySQL engine)
CHAR_LENGTH YES
CHARACTER_LENGTH YES
CONCAT YES
CONCAT_WS YES
CONV YES
ELT YES
EXPORT_SET YES
FIELD YES
FIND_IN_SET YES
FORMAT YES
HEX YES
INSTR YES
LCASE YES
LEFT YES
LENGTH YES
LOAD_FILE No (MySQL engine)
LOCATE YES
LOWER YES
LPAD YES
LTRIM YES
MAKE_SET YES
MID YES
OCT YES
OCTET_LENGTH YES
ORD YES
POSITION YES
QUOTE YES
REPEAT YES
REPLACE YES

INFOBRIGHT 3.4.2 USER G UIDE


A. FUNCTIONS AND OPE RA TORS
48

REVERSE YES
RIGHT YES
RPAD YES
RTRIM YES
SOUNDEX YES
SOUNDS LIKE No (MySQL engine)
SPACE YES
SUBSTR YES
SUBSTRING YES
SUBSTRING_INDEX YES
TRIM YES
UCASE YES
UNHEX No (MySQL engine)
UPPER YES

String C omparison Function s


LIKE YES
NOT LIKE YES
RLIKE YES
REGEXP YES
NOT REGEXP YES
STRCMP YES

Numeric Function s
Addition ( + ) YES
Subtraction ( - ) YES
Multiplication ( * ) YES
Division ( / ) YES
Modulo ( % ) YES

ABS YES
ACOS YES
ASIN YES
ATAN2, ATAN YES
ATAN YES
CEIL YES
CEILING YES
CONV YES
COS YES
COT YES

INFOBRIGHT 3.4.2 USER G UIDE


A. FUNCTIONS AND OPE RA TORS
49

DEGREES YES
EXP YES
FLOOR YES
LN YES
LOG10 YES
LOG2 YES
LOG YES
MOD YES
OCT YES
PI YES
POW YES
POWER YES
RADIANS YES
RAND YES
ROUND YES
SIGN YES
SIN YES
SQRT YES
TAN YES
TRUNCATE YES

Date and Time Functions


ADDDATE YES
ADDTIME YES
CURDATE YES
CURRENT_DATE YES
CURRENT_TIME YES
CURRENT_TIMESTAMP YES
CURTIME YES
DATE YES
DATEDIFF YES
DATE_ADD YES
DATE_FORMAT YES
DATE_SUB YES
DAY YES
DAYNAME YES
DAYOFMONTH YES
DAYOFWEEK YES
DAYOFYEAR YES
EXTRACT YES
FROM_DAYS No (MySQL engine)

INFOBRIGHT 3.4.2 USER G UIDE


A. FUNCTIONS AND OPE RA TORS
50

FROM_UNIXTIME YES
GET_FORMAT No (MySQL engine)
HOUR YES
LAST_DAY No (MySQL engine)
LOCALTIME YES
LOCALTIMESTAMP YES
MAKEDATE No (MySQL engine)
MAKETIME No (MySQL engine)
MICROSECOND No (MySQL engine)
MINUTE YES
MONTH YES
MONTHNAME YES
NOW YES
PERIOD_ADD YES
PERIOD_DIFF YES
QUARTER YES
SECOND YES
SEC_TO_TIME No (MySQL engine)
STR_TO_DATE No (MySQL engine)
SUBDATE YES
SUBTIME YES
SYSDATE YES
TIME YES
TIMEDIFF YES
TIMESTAMP No (MySQL engine)
TIMESTAMPADD No (MySQL engine)
TIMESTAMPDIFF No (MySQL engine)
TIME_FORMAT YES
TIME_TO_SEC No (MySQL engine)
TO_DAYS YES
UNIX_TIMESTAMP YES
UTC_DATE YES
UTC_TIME YES
UTC_TIMESTAMP No (MySQL engine)
WEEK YES
WEEKDAY No (MySQL engine)
WEEKOFYEAR No (MySQL engine)
YEAR No (MySQL engine)
YEARWEEK YES

INFOBRIGHT 3.4.2 USER G UIDE


A. FUNCTIONS AND OPE RA TORS
51

Text S earch and Other Functions


BINARY No (MySQL engine)
CAST YES
CONVERT YES
MATCH No (MySQL engine)

Bit Functions No (MySQL engine)


Encryption, Compression Functions No (MySQL engine)
Information Functions No (MySQL engine)

Group By Aggregate Functions


AVG YES
BIT_OR No (MySQL engine)
BIT_AND No (MySQL engine)
BIT_XOR No (MySQL engine)
COUNT YES
GROUP_CONCAT No (MySQL engine)
MIN YES
MAX YES
STD, STDDEV YES
STDDEV_POP YES
STDDEV_SAMP YES
SUM YES
VAR_POP YES
VAR_SAMP YES
VARIANCE YES

Group By Modifiers
ROLLUP No (error signalled)

INFOBRIGHT 3.4.2 USER G UIDE


B. LINUX TUNING SETTINGS
52

B. Infobright Data Tools

Infobright Configuration Manager


With ICE 3.4.2 GA, several tuning parameters are now confgured automatically and have
been deprecated from the brighthouse.ini fle. These parameters as well as several new
parameters for managing multi-core query processing have been moved to a different
system-only fle.
The deprecated parameters include:
ServerCompressedHeapSize
LoaderSaveThreadNumber
BufferingLevel
CachingLevel
ClusterSize
HugefileDir
The additional query processing parameters include:
Threads
QueueLength
Depth
Setting ControlMessages=4 in the brighthouse.ini will print the confguration settings to log
fle.

Running the Infobright C onfiguration Manager


To run the Infobright Confguration Manager, use the following command:
confman.sh --defaults-file=/etc/my-ib.cnf --autoconfigure=yes

Charset Migration Tool


Installations prior to ICE 3.3.1 may have charsets and collations defned for tables/columns
that do not match the actual Infobright storage of ascii charset and ascii_bin collation. When
you upgrade to ICE 3.4.2 GA these settings will be respected.
Infobright includes a standalone application to adapt existing tables created prior to ICE 3.3.1
GA to UTF-8 capable structures. The Charset Migration Tool (CHMT) is in the Infobright bin
directory.

Running the Charset Migration To ol


CHMT requires a text fle containing a mapping between collations used for conversion.
chmt --help // help message
Executing CHMT:

INFOBRIGHT 3.4.2 USER G UIDE


B. LINUX TUNING SETTINGS
53

chmt --datadir=/absolute/path/to/data/directory [other parameters]

INFOBRIGHT CHARSET MIGRATION TOOL PARAMETERS


Parameter Type Description Details

datadir Mandatory Absolute path to


data directory

conv-map Optional Absolute path to fle If not specifed CHMT would try
with collations to use fle: chmt-binary-
conversions folder/../support-
fles/collations.txt ; if not found
there it would search for: chmt-
binary-folder/collations.txt

database Optional Name of database If specifed, tables from no other


for migrating databases would be migrated

table Optional Name of table for If specifed, database must be also


migrating specifed; no other tables but
specifed will be migrated

log-fle Optional Absolute path to If not specifed, logs will be


output log fle printed to the console

Log Structure
The logs detail information about every considered table found in a specifed datadir. Each
conversion fnishes with [NOT NEEDED], [PASS] or [FAILED] status.

C ollations-c o nversion-file Structure


Each conversion directive is stored in one line of fle:
collation_from_name;collation_from_id;collation_to_name;collation_to_id
For example:
big5_chinese_ci;1;binary;63
where both felds containing names are only informative (all conversions will be done using
only ids).
Example collations-conversion-fle (with conversion directives described above) can be
obtained by running the following SQL:
use information_schema; select a.collation_name a_n, a.id a_id,
b.collation_name b_n, b.id b_id from information_schema.collations a,
information_schema.collations b, character_sets c where
substr(a.collation_name, 1, locate('_',a.collation_name)-
1)=c.character_set_name and substr(a.collation_name, 1,
locate('_',a.collation_name)) = substr(b.collation_name, 1,
locate('_',b.collation_name)) and b.collation_name like '%bin' and c.maxlen=1
UNION select a.collation_name a_n, a.id a_id, 'binary' b_n, 63 b_id from
information_schema.collations a, character_sets c where
((substr(a.collation_name, 1, locate('_',a.collation_name)-
INFOBRIGHT 3.4.2 USER G UIDE
B. LINUX TUNING SETTINGS
54

1)=c.character_set_name) or (locate('_',a.collation_name)=0)) and (c.maxlen>1


or c.character_set_name ='binary') order by a_id into outfile
'/some/path/my_collations.txt' fields terminated by ';';

INFOBRIGHT 3.4.2 USER G UIDE


B. LINUX TUNING SETTINGS
55

C. Linux Tuning Settings

System Settings for Red Hat Enterprise Linux and CentOS


Disable SElinux
SElinux is intended to protect Linux servers on the public internet such as Web Servers. It
provides an extra layer of security that isn't really required for a back-end database server.
In /etc/sysconfig/selinux add:

SELINUX=disabled

S wappines s
Set low swappiness to avoid unnecessary paging. This only helps for machines with low
levels of memory (say 4GB with 3GB allocated for Infobright).
In /etc/rc.local add:

echo "7" > /proc/sys/vm/swappiness

Disable Unused Proc e s s e s


Run system-confg-services (or edit /etc/initd.d directory) and leave ssh running.

File System Settings


Ensure Ca cheFolder is on a Fast Lo cal Disk
See “Infobright Tuning Parameters” in Chapter 2.

Larger Readahead
In /etc/rc.local add:

blockdev --setra 2048 /dev/sd<x>

Replace sd<x> with a proper device symbol, e.g. sdc, it should be the drive(s) on which
datadir and/or CacheFolder resides

INFOBRIGHT 3.4.2 USER G UIDE


B. LINUX TUNING SETTINGS
56

Use XFS File S y stem for Data Directorie s


For XFS (may need to install kmod-xfs and xfsprogs):

mkfs.xfs -b size=4096 /dev/sdc1

In /etc/fstab add:

/dev/sdc1 /bha xfs noatime 1 2

Note: This is for data folders only. Linux boot partition can be ext3.

noatime
Use noatime options for mounting database and cache volumes (see below for details).
Otherwise the system will update the access time for fles and directories (which degrades
performance).

Deadline Elevator
The default scheduler - CFQ - is 1% faster than elevator for a single user. However, in multi-
user test with 4 users, elevator had 20% better performance.
In /etc/rc.local add:

echo "deadline" > /sys/block/sd<x>/queue/scheduler

Replace sd<x> with a proper device symbol, e.g. sdc, it should be the drive(s) on which
datadir and/or CacheFolder resides

Increase ulimit to Support Large Data Volume or Users


This will not change performance, but may avoid errors.
Ulimit determines the maximum number of fles a user can have open.
Increase ulimit to unlimited or 32,768 since the default fle limit is 1024. This is insuffcient for
large databases (lot's of columns) or servers with multiple Infobright databases.
To view current settings, run command:

# ulimit -a

To set it to a new value for this running session, which takes effect immediately, run
command:

# ulimit -n 8800

INFOBRIGHT 3.4.2 USER G UIDE


B. LINUX TUNING SETTINGS
57

# ulimit -n -1 // for unlimited; recommended if server isn't shared,


reportedly doesn't work on IB03

Alternatively, if you want the changes to survive reboot, do the following:


• Exit all shell sessions for the user you want to change limits on.
• As root, edit the fle /etc/security/limits.conf and add these two lines toward the end:

user1 soft nofile 16000


user1 hard nofile 20000

The two lines above change the max number of fle handles - nofle - to new settings.
• Save the fle.
• Login as user1 again. The new changes will be in effect.

Note on how to detect ulimit problem


If you are noticing crashes during multi-user use cases, please check the console log for the
following error:

what(): FileSystem Error : Bad file descriptor


mysqld got signal 6;

To fx this increase ulimit (see above).

INFOBRIGHT 3.4.2 USER G UIDE

You might also like