Odoo Deployment & Performance

Goal
The aim of this training is to give good practice to deploy and maintain an odoo deployment, using a Linux system,
for small business to bigger ones, from one system with multiple odoo to one odoo on multiple systems.

We’ll also deploy tools to monitor server, track performance bottleneck, …

There’s many ways to deploy odoo, from a .deb on a virtual machine to multiple docker containers,
from a self hosted server to a cloud instance. We’ll use the recommended way to use a less layers as possible,
as every layer you add require to master it.

Schedule

Day 1 Day 2
Install PostgreSQL Adanced NGinx
Install Odoo PostgreSQL Configuration
Install NGinx Data Replication
Odoo Configuration Load balancing
Odoo command line Etherpad
Exercices Backup
Monitor

The Odoo architecture

From the web browser to PostgreSQL, many layers interact with other ones to process datas.
Inline-style:

One server
Easy to understand and deploy, it’s the most common case. 1 instance or multiple instances
Multiple servers
Harder to deploy and maintain, require higher sysadmin skills, recommended for fault tolerance and larger business.

And a mix
Between those two configurations, there’s plenty of deployment scenarios, depending on the needs.
Always remember that you’ll need a production system, and a testing system.

Testing system does not have to be the same that the production one, but must use same architecture (OS, …).

Don’t use a shared server between test and production env, to avoid performance bottleneck while testing,
permit a restart on test without stopping production, …

KISS
Keep it stupid and simple

Remember that you need a stable and maintainable system. Every layer you add require knowledge.
Don’t add unnecessary layers. En example can be a replicated database with nobody able to monitor it
or easy recover from a failure, causing a bigger downtime in case of failure than in a single server scenario.

Small system requirements

You’ve received many virtual machines, with Ubuntu 16.04 installed with a running SSH server, like you’ve received
if you had ordered a machine in a data center. All thoses machines are configured to connect each others using a
192.168.56.x private network.
You should configure this private network within VirtualBox and set 192.168.56.1 as your machine ip address within
this private network.

SSH
Make server available by ssh with key authenticationMake server available by ssh with key authentication. You should :

Allow users to use key authentification

permit dsa keyspermit dsa keys

1 $ sudo vi /etc/ssh/sshd_config

1 AuthorizedKeysFile %h/.ssh/authorized_keys
2 PubkeyAcceptedKeyTypes ssh-dss

Have SSH keys

Github require user authentication for private repositories (enterprise). As using https require to enter
user/password for every fetch, it’s easier to use ssh auth with github.

You can choose to use dsa or rsa, the key must be added to your github account, and must be password protected.

1 $ ssh-keygen -t rsa -N odoo!

Some required tools

We’ll need some tools to check system, like htop for process checking, git for cloning odoo sources, iotop
and systat to check I/O. Others will be installed later when necessary.

1 $ sudo apt-get install git htop iotop sysstat

Fix locale issue

You’ve maybe seen a locale issue when performing some commands like apt. This can simply be fixed by adding the required ones.

1 $ sudo locale-gen "en_US.UTF-8"

2 $ sudo dpkg-reconfigure locales
3 $ export LC_ALL="en_US.UTF-8"
4 $ sudo vi /etc/environment

1 PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
2 LC_ALL=en_US.UTF-8
3 LANG=en_US.UTF-8

Install PostgreSQL
Use apt.postgresql.org
You can choose to use the distribution repo to install PostgreSQL, or the official PostgreSQL apt repository to install
a more recent version of PostgreSQL. Ubuntu has only one PostgreSQL version per release in official repo.
We’ll use the PostgreSQL one here.

1 $ wget https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/plain/pgdg/apt.postgresql.org.sh
2 $ sudo bash apt.postgresql.org.sh

Install PostgreSQL
You’ll need the postgresql server itself, the client lib, and pg-activity to check running queries on a real-time basis.
 1 $ sudo apt-get install postgresql-9.6 postgresql-client-9.6 \
2 postgresql-client-common postgresql-common \
3 postgresql-server-dev-9.6 pg-activity

A PostgreSQL user is required by Odoo to connect. For now, we’ll use same PostgreSQL username as the unix one to use unix socket.

1 $ sudo su - postgres
2 postgres> psql
3 postgres=# CREATE USER odoo WITH LOGIN CREATEDB;
4 postgres=# \q
5 postgres> exit
6 $

PostgreSQL is now up and running

Install Odoo
We’ll deploy Odoo from git sources

Why using git instead of .deb?

Better control on which version to deploy (per commit basis)

Easier to download than the enterprise deb file
No need to wait the nigthly build for a bug fix
Other packages, like community ones, are easily available this way

git clone
We’ll use /opt/odoo10 as base dir.

1 $ cd /opt
2 $ sudo mkdir odoo10
3 $ sudo chown odoo odoo10
4 $ cd odoo10
5 $ git clone [email protected]:odoo/odoo.git
6 OR
7 $ git clone -b 10.0 --single-branch [email protected]:odoo/odoo.git

The next one require enterprise access

1 $ git clone [email protected]:odoo/enterprise.git

Odoo prerequisites
Odoo have some required python packages (like the web server, db connection library, …). Those python packages needs
some C/C++ headers to compile. We install the required C/C++ dev libraries from distribution repo, then we use pip
to install the Odoo python required packages.

** Odoo is running on CPython 2.7! **

1 $ sudo apt install python-pip libxml2-dev libxslt-dev libjpeg-dev \

2 libjpeg8-dev libpng-dev libldap2-dev libsasl2-dev node-less
3 $ cd /opt/odoo10/odoo
4 $ sudo pip install pip --upgrade
5 $ sudo pip install -r requirements.txt --upgrade

First launch
We can now launch odoo to check if all is ok and create a first database. We’ve deployed all required dependencies,
we’ll just use one option on the command line to explain where we can find addons. Enterprise addons are set before
the community ones!

1 $ cd /opt/odoo10/odoo
2 $ ./odoo.py --addons-path=/opt/odoo10/enterprise,/opt/odoo10/odoo/addons

Hit CTRL+C twice to stop the server.

WKHtmlToPdf
To print reports, odoo generates an html, send it to wkhtmltopdf which is in charge of the html->pdf transformation.

1 $ sudo apt install libxrender1 fontconfig

2 $ wget http://nightly.odoo.com/deb/xenial/wkhtmltox-0.12.1_linux-trusty-amd64.deb
3 $ sudo dpkg -i wkhtmltox-0.12.1_linux-trusty-amd64.deb

NGinx
Install NGINX
1 $ sudo apt-get install nginx

We’ll assume that our server name is odoo.training.internal, and that our DNS is well configured to handle this.

We’ll also need other aliases pad.training.internal and community.training.internal

Locally, for training purpose, we can simply modify our hosts file.

A small config later

 1 $ sudo vi /etc/nginx/sites-available/odoo

1 #odoo server
2 upstream odoo {
3 server 127.0.0.1:8069;
4 }
5
6 server {
7 listen 80;
8 server_name odoo.training.internal;
9
10 # Add Headers for odoo proxy mode
11 proxy_set_header X-Forwarded-Host $host;
12 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
13 proxy_set_header X-Forwarded-Proto $scheme;
14 proxy_set_header X-Real-IP $remote_addr;
15
16 #log
17 access_log /var/log/nginx/odoo.access.log;
18 error_log /var/log/nginx/odoo.error.log;
19
20 # Redirect requests to odoo backend server
21 location / {
22 proxy_redirect off;
23 proxy_pass http://odoo;
24 }
25 }

Re-Launch NGINX

1 $ sudo ln -s /etc/nginx/sites-available/odoo /etc/nginx/sites-enabled/odoo

2 $ sudo /etc/init.d/nginx restart
3 $ ./odoo.py --proxy-mode \
4 --addons-path=/opt/odoo10/enterprise,/opt/odoo10/odoo/addons

Odoo Seems to run behind NGINX. Proxy-mode because odoo is behing nginx. Forwarded headers are very important because
Odoo only see 1 client : nginx!

Check the logs and you’ll see only 1 ip : 127.0.0.1

Odoo as a service
using Systemd, we’ll start odoo at system staretup.

1 $ sudo vi /lib/systemd/system/odoo.service

1 [Unit]
2 Description=odoo
3 Requires=postgresql.service
4 After=postgresql.service
 5 Wants=nginx.service
6
7 [Install]
8 Alias=odoo.service
9
10 [Service]
11 Type=simple
12 User=odoo
13 Group=odoo
14 SyslogIdentifier=odoo-server
15 PIDFile=/run/odoo/odoo-server.pid
16 ExecStart=/opt/odoo10/odoo/odoo.py --config=/etc/odoo/odoo10.conf
17
18 [Install]
19 WantedBy=multi-user.target

If PostgreSQL is on another host

1 [Unit]
2 After=network.target

1 $ sudo mkdir /etc/odoo

2 $ sudo chown odoo /etc/odoo
3 $ sudo mkdir /var/log/odoo
4 $ sudo chown odoo /var/log/odoo
5 $ vi /etc/odoo/odoo10.conf

1 [options]
2 addons_path = /opt/odoo10/enterprise,/opt/odoo10/odoo/addons
3 proxy_mode = True

1 $ sudo systemctl start odoo

2 $ sudo systemctl enable odoo.service

Now it seems working, even after a system restart.

Configure Odoo
Add logging
By default, Odoo displays all logging of level info except for workflow logging (warning only), and log output is sent to stdout

1 $ sudo chown odoo /var/log/odoo

2 $ sudo chgrp odoo /var/log/odoo

1 [options]
2 addons_path = /opt/odoo10/enterprise,/opt/odoo10/odoo/addons
3 proxy_mode = True
 4 logfile = /var/log/odoo/odoo.log
5 syslog = False
6 log_db = False

Log rotation
In some cases, it’s not a good idea to use the odoo log rotation internal mechanism, because 2 workers can try to rotate logs at the same time.

To achieve such a goal, we use the logrotate daemon. We’ll also need the pidfile for odoo

1 [options]
2 ...
3 logrotate = False
4 pidfile = /var/run/odoo.pid

1 $ sudo apt install logrotate

2 $ sudo vi /etc/logrotate.d/odoo

1 /var/log/odoo/odoo.log {
2 rotate 10
3 daily
4
5 # Do not rotate the log if it is empty
6 notifempty
7
8 # if the log file is missing, go on to the next one without
9 # issuing an error message.
10 missingok
11
12 compress
13 delaycompress
14
15 postrotate
16 if [ -f /var/run/odoo.pid ]; then
17 kill -HUP `cat /var/run/odoo.pid`
18 fi
19 endscript
20 }

Check logs
As logs are sent to a file, here’s how to display them in real-time

1 $ tail -f /var/log/odoo/odoo.log

To search within the log, for a regex, and display line number, 2 lines after and 5 lines before the result

1 $ grep -n Worker.*unregistered /var/log/odoo/odoo.log -A2 -B5

Log handler
the log handler is a list of logger/level combination. If logger is omitted, the root logger is used. Level is a
choice between INFO, WARNING, CRITICAL and DEBUG

1 [options]
2 ...
3 log_handler = openerp.sql_db:DEBUG,werkzeug:DEBUG,openerp.fields:WARNING,openerp.http.rpc.request:DEBUG,:WARNING

This handler will

Debug sql queries

Debug werkzeug queries
Warn openerp.fields module
Debug RPC queries
Warn on others

Log level
log levels are shortcut for log handlers, to ease configuration of most commonly used handlers combination : critical, error, warn, debug,
debug_sql, debug_rpc, debug_rpc_answer

1 [options]
2 ...
3 log_level = debug_rpc

Odoo workers
Only on unix
Python GIL: in CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python
bytecodes at once.
Add workers
1 worker ~= 6 users
# workers = (#cpus * 2 ) + 1

1 [options]
2 ...
3 workers = 4
4 max_cron_threads = 1

Check in log :

openerp.service.server: Evented Service (longpolling) running on 0.0.0.0:8072

Exception: bus.Bus unavailable

Nginx with workers

As the longpolling is on another tcp port, we must forward /longpoll/ request to the correct port.
Don’t forward all requests to this port, or you’ll never user the workers process

1 [...]
2 upstream odoo_longpoll {
3 server 127.0.0.1:8072;
4 }
5 [...]
6 location /longpoll {
7 proxy_redirect off;
8 proxy_pass http://odoo_longpoll;
9 }
10 [...]

Check process list with/without workers

Workers limits
Odoo process are recycled, on a parametrized basis.

CPU

limit_time_cpu = 60
limit_time_real = 120
limit_time_real_cron = limit_time_real New since V10

If a worker reach the limit, the process is killed, without sending the response to the client.

It’s important to set to a high enough value.

Memory

limit_memory_soft = 2 Gb

A worker which reach this memory limit during the processing of the request is recycled after sending the response.
limit_memory_hard = 2.5 Gb

A worker which reach this memory limit during the processing of the request is recycled immediatly, without sending the response.

Requests count

limit_request = 8192

A worker which process this # of request is recycled at the end of the request response

Set workers limits

1 [options]
2 ...
3 limit_time_cpu = 60
4 limit_time_real = 240
5 limit_time_real_cron = 3600
6 limit_memory_soft = 2147483648
7 limit_memory_hard = 4294967296
8 limit_request = 8192

Test workers recycling, soft and hard… You can set for example, limit_request to 2 or any other stupid value

1 INFO tst1 openerp.service.server: Worker (3120) max request (8192) reached.

2 INFO tst1 openerp.service.server: Worker (3120) exiting. request_count: 8192, registry count: 1.
3 DEBUG ? openerp.service.server: Worker (3120) unregistered
4 INFO ? openerp.service.server: Worker WorkerHTTP (3129) alive

Wizard limits
Wizard are object defined as not-persistent within Odoo. Those objects are in fact stored within the database and
cleaned with a cron. By default, cron clean those records after one hour. You can choose to clean faster/slower and/or
limit the number of such records within the database.

osv_memory_count_limit is a per table setting, while osv_memory_age_limit (expressed in hours) is for all wizards.

1 [options]
2 ...
3 osv_memory_count_limit = 100000
4 osv_memory_age_limit = 4

DB Filtering
When using website, it’s required to use db filtering. dbfilter is a regex, which can contain %d and %h special values, representing respectively
the server name and the FQDN.

dbfilter
 regex
%h
%d

1 ...
2 dbfilter = ^%d[0-9]*$

DB admin
Avoid anyone to drop your db’s, stole your datas,… via /web/database/manager

admin_passwd

1 $ python -c 'import base64, os; print(base64.b64encode(os.urandom(24)))'

1 ...
2 admin_passwd = ApRmoGy+DcFREAVCkD7PbtznKc1KDsyU

DB Connection
Manage connection to the server. Using uniw sockets by default (or localhost on windows)

db_maxconn is a per-worker limit

1 [options]
2 ...
3 db_name = False
4 db_host = False
5 db_port = 5432
6 db_user = False
7 db_password = False
8 db_template = template1
9 db_maxconn = 4

List DB
By default, a user can choose db to connect to. list_db can disable this feature.

1 [options]
2 ...
3 list_db = False

Unaccent & geoip_database

Use the unaccent function provided by the database when available.
geoip_database is a free database of IP geolocalisation.

1 [options]
2 ...
3 unaccent = False
4 geoip_database = /usr/share/GeoIP/GeoLiteCity.dat

Data dir
Some binaries are stored on the file system, instead of storing them in DB. Default is
home_drive/.local/share/odoo/version, C:\Documents and Settings\\Application Data\… on Windows, …

1 [options]
2 ...
3 data_dir = /home/odoo/datas

Don’t forget to backup it!

IP ports
You can change the ip ports used

1 [options]
2 ...
3 xmlrpc = True
4 xmlrpc_interface = 127.0.0.1
5 xmlrpc_port = 8069
6 longpolling_port = 8072

SMTP
You can configure a default mail server. By default, Odoo use localhost on port 25, without SSL nor authentication

1 [options]
2 ...
3 email_from = [email protected]
4 smtp_server = 127.0.0.1
5 smtp_port = 25
6 smtp_ssl = False
7 smtp_user = False
8 smtp_password = False

Demo data
It’s better on production server to avoid deploying demo datas… (comma-separated, use \“all\” for all modules) Used with -d and -i
 1 [options]
2 ...
3 without_demo = all

Debug mode
Known from Odoo developers, this mode alter a bit the datadriven mode of Odoo, by reading files from disks instead of database

1 [options]
2 ...
3 debug_mode = False

Big imports
Use this for big data importation, if it crashes you will be able to continue at the current state. Provide a filename to store intermediate
importation states.

1 [options]
2 ...
3 import_partial = /user/odoo/tempimport

Pid File
Store the pid within a file

1 [options]
2 ...
3 pidfile = /run/odoo/odoo-server.pid

Let systemd storing pidfile

1 [options]
2 ...
3 pidfile = False

Server wide modules

server-wide modules are supposed to provide features not necessarily tied to a particular database. By default, only /web

1 [options]
2 ...
3 server_wide_modules = web,mysupermodule

A use case for those kind of modules are the hw_xxx modules of the posbox
Per-DB Odoo Configuration
ir.config_parameter
It’s possible to change some settings on a per-database basis. Those parameters permit to solve some issues
depending on the deployment senario.

The most frequent ones are web.base.url.freeze and report.url

web.base.url
web.base.url is the base url of your Odoo deployment. You can change it in the ir.config_parameter table.
The default behaviour of Odoo is to reset the value of this parameter each time the admin logs in.
Logging in with admin using an ssh tunnel can broke some features…

If parameter web.base.url.freeze is set, this behaviour is disabled.

pdf reports
Here are the steps when odoo prints a pdf:

Odoo generates an html

Odoo send the html to wkhtmltopdf
wkhtmltopdf request assets to Odoo
wkhtmltopdf transform html into pdf
wkhtmltopdf send pdf to Odoo
Odoo sends pdf to the browser

report.url
wkhtmltopdf can request informations (assets) directly from Odoo, without contacting reverse proxy.
In some scenarios, when web.base.url is resolved as the firewall ip for example, the firewall detects a software in the
lan requesting an url in the lan, then drop the request. This makes the wkhtmltopdf request for assets fails, and thus
the generated pdf lacks of css.

the report.url can be set to _ http://localhost:8069 _ to enable direct connection from wkhtmltopdf to Odoo.

Other Odoo command line operations

Perform a full update
 1 $ cd /opt/odoo10/odoo
2 $ git pull
3 $ cd /opt/odoo10/enterprise
4 $ git pull
5 $ sudo service odoo stop
6 $ cd /opt/odoo10/odoo
7 #comment out logfile in config
8 $ ./odoo.py -c /etc/odoo/odoo10.conf -d mydb -u all --stop-after-init
9 $ sudo service odoo start

Avoid lack of log

The idea is to remove the logfile from config, add it to startup script, to ease the update process

in /etc/odoo/odoo10.conf :

1 #logfile = /var/log/odoo/odoo.log

in /lib/systemd/system/odoo.service :

1 ExecStart=/opt/odoo10/odoo/odoo.py --config=/etc/odoo/odoo10.conf --logfile /var/log/odoo/odoo.log

1 $ sudo systemctl daemon-reload

2 $ sudo systemctl stop odoo
3 $ sudo systemctl start odoo

Install a module in command line

1 $ ./odoo.py -c /etc/odoo/odoo10.conf -d mydb -i mymodule --stop-after-init

Exercices
2 instances
Without altering the running server, it’s asked to deploy as a service, and only accessible through NGinx:

Workers, cron FQDN DB filter Ports Edition

3, 2 odoo10.training.internal odoo10*c 9069, 9072 community

2, 2 odoo10.training.internal odoo10*c 9069, 9072 community

1 instance, 2 customers
Without altering the running servers, it’s asked to deploy as a service, and only accessible through NGinx, one odoo 10 enterprise serving two
customers, with 3 workers and 1 cron, on ports 8079 and 8082:

FQDN DB

cust1.training.internal cust1
cust2.training.internal cust2

Logging
Set logging on cust1 to debug the rpc responses.

Set logging on cust2 to debug the rpc queries, debug sql queries, and only critical on others.

Set logging on the odoo9 instance to debug only the crm module, and info on others.

New module
On the Odoo9 instance, deploy one OCA module, accessible through github, and install it on database.

Advanced NGINX
SSL
For now, Odoo is running in clear http. We need to redirect http requests to https ones.

1 proxy_set_header X-Forwarded-Host $host;

2 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
3 proxy_set_header X-Forwarded-Proto $scheme;
4 proxy_set_header X-Real-IP $remote_addr;
5 proxy_set_header Host $http_host;
6 add_header X-Content-Type-Options nosniff;
7
8 # http -> https
9 server {
10 listen 80;
11 server_name odoo.training.internal;
12
13 rewrite ^(.*)$ https://$host$1 permanent;
14 }

And we need to handle https requests

1 #SSL
2 server {
3 listen 443;
 4 server_name odoo.training.internal;
5
6 ssl on;
7 ssl_certificate /etc/ssl/nginx/server.crt;
8 ssl_certificate_key /etc/ssl/nginx/server.key;
9 ssl_session_timeout 30m;
10 ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
11 ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-
12 ssl_prefer_server_ciphers on;
13
14 [...]

Pos
If you’re using posbox, as the browser connect to the posbox using http, you’ve to publish the pos using http
to avoid a mixed content error within the browser.

1 server {
2 listen 80;
3 [...]
4 #Redirect non-POS and related content in https to allow POS in http
5 location ~ ^(?!/pos/|/web/binary/company_logo|/?[^/]*/website/action/|/web/webclient/|/web/content/|/web_editor/s
6 proxy_redirect off;
7 if ($request_method != POST) {
8 # rewriting a POST request has no sense as the browser will do the request as GET
9 rewrite ^(.*)$ https://$host$1 permanent;
10 }
11 if ($request_method = POST) { proxy_pass http://odoo; }
12 }
13
14 location = /longpolling/poll { proxy_redirect off; proxy_pass http://odoo_longpoll;}
15 location / { proxy_redirect off; proxy_pass http://odoo; }
16 }
17 [...]
18 server {
19 listen 443;
20 [...]
21 # Redirect POS to http
22 location ~ ^/pos/web {
23 rewrite ^(.*) http://$host:80$1 permanent;
24 }
25 # Redirect requests to odoo backend server
26 location / {
27 proxy_redirect off;
28 proxy_pass http://odoo;
29 }
30 location /longpolling {
31 proxy_redirect off;
32 proxy_pass http://odoo_longpoll;
33 }
34 }
Full NGinx config
1 #odoo server
2 upstream odoo {
3 server 127.0.0.1:8069;
4 }
5 upstream odoo_longpoll {
6 server 127.0.0.1:8072;
7 }
8
9 proxy_set_header X-Forwarded-Host $host;
10 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
11 proxy_set_header X-Forwarded-Proto $scheme;
12 proxy_set_header X-Real-IP $remote_addr;
13 proxy_set_header Host $http_host;
14 add_header X-Content-Type-Options nosniff;
15
16 server {
17 listen 80;
18 server_name odoo.training.internal;
19
20 access_log /var/log/nginx/odoo.access.log;
21 error_log /var/log/nginx/odoo.error.log;
22
23 #Redirect non-POS and related content in https to allow POS in http
24 location ~ ^(?!/pos/|/web/binary/company_logo|/?[^/]*/website/action/|/web/webclient/|/web/content/|/web_editor/s
25 proxy_redirect off;
26
27 if ($request_method != POST) {
28 # rewriting a POST request has no sense as the browser will do the request as GET
29 rewrite ^(.*)$ https://$host$1 permanent;
30 }
31 if ($request_method = POST) {
32 proxy_pass http://odoo;
33 }
34 }
35 location = /longpolling/poll {
36 proxy_redirect off;
37 proxy_pass http://odoo_longpoll;
38 }
39
40 location / {
41 proxy_redirect off;
42 proxy_pass http://odoo;
43 }
44 }
45
46 server {
47 listen 443;
48 server_name odoo.training.internal
49
50 access_log /var/log/nginx/odoossl.access.log;
51 error_log /var/log/nginx/odoossl.error.log;
52
53 # SSL parameters
54 ssl on;
 55 ssl_certificate /etc/ssl/nginx/server.crt;
56 ssl_certificate_key /etc/ssl/nginx/server.key;
57 ssl_session_timeout 30m;
58 ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
59 ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES
60 ssl_prefer_server_ciphers on;
61
62 # Redirect POS to http
63 location ~ ^/pos/web {
64 rewrite ^(.*) http://$host:80$1 permanent;
65 }
66
67 # Redirect requests to odoo backend server
68 location / {
69 proxy_redirect off;
70 proxy_pass http://odoo;
71 }
72 location /longpolling {
73 proxy_redirect off;
74 proxy_pass http://odoo_longpoll;
75 }
76 }

PostgreSQL configuration
Until now, we’ve used to deploy odoo and postgresql on the same server. We will now deploy PostgreSQL on his own server, and configure it.

Use VM odoo-pg1
We will use a new VM and the private network.

Copy private/public ssh keys from odoo server, alter config, add hosts, …

1 $ sudo vi /etc/ssh/sshd_config

1 AuthorizedKeysFile %h/.ssh/authorized_keys
2 PubkeyAcceptedKeyTypes ssh-dss
 1 $ sudo vi /etc/hosts

1 192.168.56.10 odoo-training
2 192.168.56.22 odoo-pg2
3 192.168.56.23 odoo-pg3

Use apt.postgresql.org
Deploy it like in the single server scenario

1 $ wget https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/plain/pgdg/apt.postgresql.org.sh
2 $ sudo bash apt.postgresql.org.sh
3 $ sudo apt-get install postgresql-9.6 postgresql-client-9.6 \
4 postgresql-client-common postgresql-common \
5 postgresql-server-dev-9.6 pg-activity
6 $ sudo su - postgres

We also need to set a password for the Odoo PostgreSQL user.

1 postgres> psql
2 postgres=# CREATE USER odoo WITH LOGIN CREATEDB PASSWORD 'OdooPostgrePWD';
3 postgres=# \q
4 postgres> exit
5 $

Allow network connections

1 $ sudo vi /etc/postgresql/9.6/main/pg_hba.conf

1 ...
2 # IPv4 local connections:
3 host all all 127.0.0.1/32 md5
4 host all all 192.168.56.0/24 md5

1 $ sudo vi /etc/postgresql/9.6/main/postgresql.conf

1 ...
2 listen_addresses = '192.168.56.21'

1 $ sudo service postgresql restart

STOP postgre on odoo

We need to stop and disable postgresql service on the odoo server, and connect odoo to our new postgresql server.
 1 $ sudo systemctl stop postgresql
2 $ sudo systemctl disable postgresql
3 $ sudo vi /lib/systemd/system/odoo.service

1 ...
2 After=network.target

1 $ sudo systemctl daemon-reload

2 $ sudo vi /etc/odoo/odoo10.conf

1 db_host = odoo-pg1
2 db_port = 5432
3 db_user = odoo
4 db_password = OdooPostgrePWD

# connections
This is very important to some of the below parameters (particularly work_mem) because there are some memory resources that are or can be
allocated on a per-client basis, so the maximum number of clients suggests the maximum possible memory use

set db_maxconn in odoo config

~4 connections per worker + 3 for admin

1 $ sudo vi /etc/postgresql/9.6/main/postgresql.conf

1 ...
2 max_connections = 80

shared_buffers
The shared_buffers configuration parameter determines how much memory is dedicated to PostgreSQL to use for caching data

25% of RAM on physical server

recommended when in a VM: more than 55% VM RAM (VMWare doc)

1 $ sudo vi /etc/postgresql/9.6/main/postgresql.conf

1 ...
2 shared_buffers = 1GB

You can retrieve the value the OS can allocate to PostgreSQL in monitoring systems like Munin
effective_cache_size
effective_cache_size should be set to an estimate of how much memory is available for disk caching[…]

Should be 50% of the RAM

75% of memory is a more aggressive but still reasonable amount

1 $ sudo vi /etc/postgresql/9.6/main/postgresql.conf

1 ...
2 effective_cache_size = 1GB

work_mem
If you do a lot of complex sorts, and have a lot of memory, then increasing the work_mem parameter allows PostgreSQL
to do larger in-memory sorts which, unsurprisingly, will be faster than disk-based equivalents. This parameter is a
per connection value!

1 $ sudo vi /etc/postgresql/9.6/main/postgresql.conf

1 ...
2 work_mem = 64MB
autovacuum
Required to:

To recover or reuse disk space.

To update data statistics used by the PostgreSQL query planner.
To update the visibility map, which speeds up index-only scans.
To protect against loss of very old data due to transaction ID wraparound or multixact ID wraparound.

Should vacuum more than less

WAL size
Prior to 9.5, parameter was checkpoint_segments. It has been replaced in 9.5 with min_wal_size and max_wal_size

max_wal_size = (3*checkpoint_segments) * 16MB

Note that the default setting for max_wal_size is much higher than the default checkpoint_segments used to be,
so adjusting it might no longer be necessary.

Good ideas only at first sight

Avoid to do this, you’re probably not in a situation where it’s required.

fsync = false : Can cause data corruption

autovacuum = false : Transaction ID Wraparound Failures
track_counts = false : Needed by autovacuum

What about file store?

We now have PostgreSQL datas stored on the DB server, we also need Filestore on the datas server

Even if NFS is not designed for performance, it’s an easy way to achieve our goal, and due to the way Odoo handles
files I/O, this will not be an issue.

NFS server
On odoo-pg1, we deploy NFS server and allow connections from odoo servers.

1 $ sudo apt install nfs-kernel-server

2 $ sudo mkdir -p /export/odoo
3 $ sudo chown -R odoo /export
4 $ sudo vi /etc/exports
 1 /export/odoo 192.168.56.0/255.255.255.0(rw,sync,no_subtree_check)

1 $ sudo exportfs -ra

2 $ sudo service nfs-kernel-server restart

NFS client
On odoo-training, we deploy NFS client.

1 $ sudo apt install nfs-common

2 $ sudo mkdir /filestore
3 $ sudo chown -R odoo /filestore
4 $ sudo vi /etc/fstab

1 ...
2 # Odoo filestore
3 odoo-pg1:/export/odoo /filestore nfs rw,hard,intr 0 0

1 $ sudo mount /filestore

2 $ cd ~/.local/share/Odoo
3 $ cp -R ./ /filestore/
4 $ sudo vi /etc/odoo/odoo10.conf

1 ...
2 data_dir = /filestore

1 $ sudo service odoo restart

Datas Replication
Rebuilding an Odoo server from scratch when all datas are stored on another server is quite fast and easy,
rebuilding a data server (PostgreSQL and filestore) is a bit harder.

Use VM odoo-pg2
We will use a new VM.

Copy private/public ssh keys from odoo server, alter config, add hosts, …

1 $ sudo vi /etc/ssh/sshd_config

1 AuthorizedKeysFile %h/.ssh/authorized_keys
2 PubkeyAcceptedKeyTypes ssh-dss
 1 $ sudo vi /etc/hosts

1 192.168.56.10 odoo-training
2 192.168.56.21 odoo-pg1
3 192.168.56.23 odoo-pg3

Use apt.postgresql.org
Deploy it like on odoo-pg1

1 $ wget https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/plain/pgdg/apt.postgresql.org.sh
2 $ sudo bash apt.postgresql.org.sh
3 $ sudo apt-get install postgresql-9.6 postgresql-client-9.6 \
4 postgresql-client-common postgresql-common \
5 postgresql-server-dev-9.6 pg-activity

Promote odoo-pg1
First, we’ll promote odoo-pg1 as replication master.

1 $ python -c 'import base64, os; print(base64.b64encode(os.urandom(24)))'

2 $ sudo -u postgres psql -c "CREATE USER odoo_rep REPLICATION LOGIN
3 PASSWORD 'ByrXga8IbnL5hNTeB46tj41e3B7edPs';"
4 $ sudo vi /etc/postgresql/9.6/main/postgresql.conf

1 ...
2 wal_level = hot_standby
3 max_wal_senders = 3
4 wal_keep_segments = 8
5 archive_mode = on
6 full_page_writes = on
7 archive_command = 'cp "%p" "/var/lib/postgresql/9.6/shipped/%f"'

Allow replication connections

1 $ sudo vi /etc/postgresql/9.6/main/pg_hba.conf

1 ...
2 hostssl replication odoo_rep 192.168.56.0/24 md5

1 $ sudo su - postgres
2 $ mkdir /var/lib/postgresql/9.6/shipped/
3 $ exit
Initiate replication
It’s now time to initiate replication from odoo-pg1 to odoo-pg2. We do the same config as on odoo-pg1, the main difference is the hot_standby
parameter.

1 $ sudo vi /etc/postgresql/9.6/main/postgresql.conf

1 ...
2 listen_addresses = '192.168.56.22'
3 max_connections = 80
4 shared_buffers = 1GB
5 effective_cache_size = 1GB
6 work_mem = 64MB
7 wal_level = hot_standby
8 max_wal_senders = 3
9 wal_keep_segments = 8
10 full_page_writes = on
11 archive_command = 'cp "%p" "/var/lib/postgresql/9.6/shipped/%f"'
12 hot_standby = on

Allow connections
Even if not required, it’s a good idea to permit connections to this slave the same way it is on the master. It will be easier in case of failover.

1 $ sudo vi /etc/postgresql/9.6/main/pg_hba.conf

1 ...
2 # IPv4 local connections:
3 host all all 127.0.0.1/32 md5
4 host all all 192.168.56.0/24 md5
5 ...
6 hostssl replication odoo_rep 192.168.56.0/24 md5

recovery.conf
We now configure odoo-pg2 To connect to odoo-pg1 to stream replicate.

1 $ sudo vi /etc/postgresql/9.6/main/recovery.conf

1 standby_mode = 'on'
2 primary_conninfo = 'host=odoo-pg1 port=5432 user=odoo_rep password=ByrXga8IbnL5hNTeB46tj41e3B7edPs sslmode=require'
3 trigger_file = '/tmp/postgresql.failover'

1 $ sudo su - postgres
2 $ mkdir /var/lib/postgresql/9.6/shipped/
3 $ exit
Replication script
We will now destroy cluster on odoo-pg2 and dump the one from odoo-pg1 to start replication.

Keep care on what you’re doing now, we destroy a db cluster!

1 $ sudo vi ~/replicate.sh

1 echo Stopping PostgreSQL Slave node

2 sudo service postgresql stop
3
4 echo Drop old cluster
5 sudo -u postgres rm -rf /var/lib/postgresql/9.6/main
6
7 echo Performing base backup
8 sudo -u postgres pg_basebackup -h 192.168.56.21 -D /var/lib/postgresql/9.6/main -U odoo_rep -v -P
9 sudo -u postgres cp /etc/postgresql/9.6/main/recovery.conf /var/lib/postgresql/9.6/main/
10
11 echo Startging PostgreSQL Slave node
12 sudo service postgresql start

1 $ sudo chmod a+x ~/replicate.sh

2 $ ~/replicate.sh

and Check that it’s working!

1 $ sudo -u postgres psql -x -c "select * from pg_stat_replication;"

1 -[ RECORD 1 ]----+------------------------------
2 pid | 8721
3 usesysid | 24029
4 usename | odoo_rep
5 application_name | walreceiver
6 client_addr | 192.168.56.22
7 client_hostname |
8 client_port | 38268
9 backend_start | 2016-09-01 17:01:38.184454+02
10 backend_xmin |
11 state | streaming
12 sent_location | 0/23000060
13 write_location | 0/23000060
14 flush_location | 0/23000060
15 replay_location | 0/23000000
16 sync_priority | 0
17 sync_state | async

Replicate filestore
We will now replicate filestore to have a fully redundant data store. We’ll use rsync, but other solutions based on clustered file systems, like
drbd, are possible. On odoo-pg2:

1 $ sudo mkdir /export

2 $ sudo chown odoo /export
3 $ vi ~/syncfilestore.sh

1 #!/bin/bash
2 # Replicate odoo file store
3 rsync -az odoo@odoo-pg1:/export/ /export

1 $ crontab -e

1 */5 * * * * /home/odoo/syncfilestore.sh

Deploy odoo-pg3
As an exercice, deploy another slave postgresql server. Use odoo-pg3 VM for this purpose.

Load balancing
Before adding more Odoo server to handle a higher load, we need something to distribute requests across the different odoo servers :
HAProxy

Deploy HAProxy VM
Deploy VM as usual :

Network card
/etc/hosts
SSH
Locale

1 $ sudo apt install haproxy

SSL certificate
Use at least let’s encrypt… Here we’ll use a self signed as no public ip routing.

1 $ sudo mkdir /etc/ssl/odoo

2 $ sudo openssl genrsa -out /etc/ssl/odoo/odoo.key 1024
3 $ sudo openssl req -new -key /etc/ssl/odoo/odoo.key \
4 -out /etc/ssl/odoo/odoo.csr
5 > Country Name (2 letter code) [AU]:BE
6 > State or Province Name (full name) [Some-State]:Wal-Brabant
7 > Locality Name (eg, city) []:LLN
8 > Organization Name (eg, company) [Internet Widgits Pty Ltd]: PSTech
9 > Organizational Unit Name (eg, section) []:
10 > Common Name (e.g. server FQDN or YOUR name) []:odoo.training.internal
11 > Email Address []:
12
13 > Please enter the following 'extra' attributes to be sent with your certificate request
14 > A challenge password []:
15 > An optional company name []:
16 $ sudo openssl x509 -req -days 365 -in /etc/ssl/odoo/odoo.csr \
17 -signkey /etc/ssl/odoo/odoo.key \
18 -out /etc/ssl/odoo/odoo.crt
19 $ sudo cat /etc/ssl/odoo/odoo.crt /etc/ssl/odoo/odoo.key \
20 | sudo tee /etc/haproxy/odoo.pem

Configure HAProxy
1 $ sudo vi /etc/haproxy/haproxy.cfg

1 global
2 log /dev/log local1 warning
3 stats socket /run/haproxy/admin.sock mode 660 level admin
4 stats timeout 30s
5 user haproxy
6 group haproxy
7 daemon
8 maxconn 3000
9 ca-base /etc/ssl/certs
10 crt-base /etc/ssl/private
11 ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+A
12 ssl-default-bind-options no-sslv3 no-tls-tickets
13 tune.ssl.default-dh-param 2048
14
15 defaults
16 log global
17 mode http
18 option httplog
19 option dontlognull
20 timeout connect 5000ms
21 timeout client 900s
22 timeout server 900s
23 timeout check 90s
24
25 listen stats
26 bind *:1936
27 mode http
28 stats enable
29 stats hide-version
30 stats realm Haproxy\ Statistics
31 stats auth odoo:odoo
32 stats uri /haproxy-status
33
34 frontend odoo_no_ssl
35 bind *:80
36 maxconn 3000
37 option http-server-close
38 option forwardfor
39 reqadd X-Forwarded-Proto:\ http
40 reqadd X-Forwarded-Port:\ 80
41 default_backend backend_odoo_no_ssl
42
43 frontend odoo_ssl
44 bind *:443 ssl crt /etc/haproxy/odoo.pem
45 maxconn 3000
46 option http-server-close
47 option forwardfor
48 reqadd X-Forwarded-Proto:\ https
49 reqadd X-Forwarded-Port:\ 443
50 default_backend backend_odoo_ssl
51
52 backend backend_odoo_no_ssl
53 mode http
54 balance roundrobin
55 option httpchk HEAD / HTTP/1.0\r\nHost:\ odoo.training.internal
56 default-server inter 10s rise 1 fall 3
57 http-check send-state
58 server odoo1 192.168.56.10:80 check
59 server odoo2 192.168.56.11:80 check
60
61 backend backend_odoo_ssl
62 mode http
63 balance roundrobin
64 option httpchk HEAD / HTTP/1.0\r\nHost:\ odoo.training.internal
65 default-server inter 10s rise 1 fall 3
66 http-check send-state
67 server odoo1 192.168.56.10:443 ssl verify none check check-ssl
68 server odoo2 192.168.56.11:443 ssl verify none check check-ssl
Test HAProxy
Change IP in your host file…

Sessions cleaning
Avoid having odoo itself cleaning sessions
Every ~1000 requests, Odoo list sessions, and remove the too old ones.
As it’s not a problem on a local filesystem, this can be problematic on a shared filesystem, like the NFS one we’re using.
To avoid this performance bottleneck, we’ll stop to check this with Odoo itself, and use a linux cron to achieve this.

Stop cleaning session in Python code

It’s a bit tricky to change this behaviour. The method consist in a monkey patch of the method. Beware that this code
will reside in a module, and even if the module is not installed, due to the monkey patch nature of the module, the code
will be executed.

in session.py:

1 from openerp import http

2
3 def session_gc(session_store):
4 pass
5
6 http.session_gc = session_gc
in init.py:

1 import session

Creating the session cleaning cron

1 $ crontab -e

1 48 * * * * find /filestore/sessions -mtime +8 -delete

Improve admissible load

Simply duplicate odoo server, change ip and hostname…

clone VM in virtualbox
sudo vi /etc/hostname
sudo vi /etc/hosts
sudo vi /etc/network/interfaces

That’s it, you’ve a new server able to respond to odoo queries.

As an exercice, deploy an odoo-3 vm and add it to the haproxy pool

Update source on a HA cluster

Some process, like the bundle generation, rely on the last modified date of the files contained in the bundle.
If those dates are different on the two servers, the bundle will be re-generated continously, and you’ll face 404 errors.

To ensure a proper bundle generation, all dates must be identical between the servers. To achieve this, can have two
solutions:

A share drive
A rsync between the servers

As we already have deployed a share drive for filestore, it’s easy to achieve this for code.

We’ll explain here the rsync method

Sync code base

To update source code on servers, we will do a git checkout on one server, or a git pull, and sync the code
on the other ones.

On server odoo-1:
 1 $ cd /opt/odoo10/odoo
2 $ git pull
3 $ cd /opt/odoo10/enterprise
4 $ git pull
5 [...]
6 $ cd /opt/odoo10/my_modules
7 $ git pull
8 [...]

On server odoo-2 (and odoo-3, odoo-xxx, …):

1 $ rsync -az odoo@odoo-1:/opt/odoo10/ /opt/odoo10

Etherpad
node.js
1 $ sudo apt install -y build-essential
2 $ cd
3 $ curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
4 $ sudo apt-get install -y nodejs

Etherpad
1 $ sudo apt install libssl-dev pkg-config
2 $ cd /opt
3 $ sudo mkdir etherpad
4 $ sudo mkdir /var/log/etherpad
5 $ sudo chown odoo etherpad
6 $ sudo chgrp odoo etherpad
7 $ cd etherpad
8 $ git clone git://github.com/ether/etherpad-lite.git
9 $ sudo adduser etherpad --system --group --home /var/etherpad
10 $ sudo chown -R etherpad /var/log/etherpad
11 $ sudo chown -R etherpad etherpad-lite
12 $ cd etherpad-lite
13 $ ./bin/run.sh

1 $ sudo vi /opt/etherpad/etherpad-lite/settings.json

1 "trustProxy" : true,
2 "users": {
3 "admin": {
4 "password": "admin",
5 "is_admin": true
6 },
7 "user": {
8 "password": "user",
 9 "is_admin": false
10 }
11 },
12 "defaultPadText" : "",
13 "suppressErrorsInPadText" : true,
14 "logconfig" :
15 { "appenders": [
16 /*{ "type": "console"
17 //, "category": "access"// only logs pad access
18 }*/
19 { "type": "file"
20 , "filename": "/var/log/etherpad/etherpad.log"
21 , "maxLogSize": 1024
22 , "backups": 3 // how many log files there're gonna be at max
23 //, "category": "test" // only log a specific category
24 }

1 $ sudo vi /lib/systemd/system/etherpad.service

1 [Unit]
2 Description=Etherpad-lite, the collaborative editor.
3 After=syslog.target network.target
4
5 [Service]
6 Type=simple
7 User=etherpad
8 Group=etherpad
9 WorkingDirectory=/opt/etherpad/etherpad-lite
10 ExecStart=/opt/etherpad/etherpad-lite/bin/run.sh
11 Restart=always
12
13 [Install]
14 WantedBy=multi-user.target

1 $ sudo chown -R etherpad /opt/etherpad/etherpad-lite

2 $ sudo systemctl daemon-reload
3 $ sudo systemctl start etherpad
4 $ sudo systemctl enable etherpad

If etherpad doesn’t start, check with

1 $ journalctl

Proxiing etherpad
1 $ sudo vi /etc/nginx/sites-available/pad

1 server {
2 listen 80;
3 server_name pad.training.internal;
 4
5 #log
6 access_log /var/log/nginx/etherpad.access.log;
7 error_log /var/log/nginx/etherpad.error.log;
8
9 # Redirect requests to etherpad backend server
10 location / {
11 proxy_redirect off;
12 proxy_set_header Host $host;
13 proxy_pass http://127.0.0.1:9001;
14 }
15 }

1 $ sudo ln -s /etc/nginx/sites-available/pad /etc/nginx/sites-enabled/pad

2 $ sudo service nginx restart

Test it
Install module Pad on tasks

1 $ cat /opt/etherpad/etherpad-lite/APIKEY.txt

Settings->General settings->Configure your company data

In configuration tab:
pad server = http://pad.training.internal
Pad Api Key = content of APIKEY.txt

If etherpad is stopped
Test by stopping it

If description is empty
If task description is empty after filling it within the pad, that’s because odoo server cannot contact pad server

check name resolution on the odoo server. In this training, edit /etc/hosts on the odoo server

1 127.0.0.1 localhost pad.training.internal

2 127.0.1.1 odoo-training
3
4 # The following lines are desirable for IPv6 capable hosts
5 ::1 localhost ip6-localhost ip6-loopback
6 ff02::1 ip6-allnodes
7 ff02::2 ip6-allrouters

Backup
PostgreSQL
You’ve mainly 2 ways to do backups:

pg_dump
pg_basebackup
Each one have pros and cons. There’s no one size fits all solution, mainly dependant on :

RTO: Recovery Time Objective

RPO: Recovery Point Objective

pg_dump
The most used solution, permit to backup and restore databases one at a time.

1 $ pg_dump -Fc -d odoo > odoo.backup

1 $ pg_restore -d odoo -j8 odoo.backup

Can also use text format, easy to migrate from one postgresql version to another (>=)

1 $ pg_dump -d odoo > odoo.backup

1 $ psql -d odoo < odoo.backup

Can also be used to copy to a test server

1 $ pg_dump -Fc -d odoo | pg_restore -d odoo -h my_other_server

pg_basebackup
Backup the whole database cluster. Already used in training for starting streaming replication. Can be used to backup a database, and restore it
with WAL files to minimise RPO.

1 $ pg_basebackup --xlog --format=t -D /backups/`date +%Y%m%d`

This create a tar with the whole database storage. To restore, simply stop the cluster, remove the file system directory and replace with the
backup content.

To incluide WAL files, create a recovery.conf file in the cluster data directory, containing:

1 restore_command = 'cp /mnt/server/archivedir/%f %p'

Adapt the command to your configuration. If you choose this backup/restore method, train yourself to be ready!

Monitor
Munin require 1 master server which collects datas, and plenty of node where to collect datas from. We will use a new VM for the main munin
server.

Copy private/public ssh keys from odoo server, alter config, add hosts, …

1 $ sudo vi /etc/ssh/sshd_config

1 AuthorizedKeysFile %h/.ssh/authorized_keys
2 PubkeyAcceptedKeyTypes ssh-dss

1 $ sudo vi /etc/hosts

1 192.168.56.10 odoo-1
2 192.168.56.11 odoo-2
3 192.168.56.12 odoo-3
4 192.168.56.21 odoo-pg1
5 192.168.56.22 odoo-pg2
6 192.168.56.23 odoo-pg3
7 192.168.56.30 haodoo

Install Munin package

1 $ sudo apt install munin

This will install munin, as well as munin-node (for local server monitoring), some plugins, …

We need only one Munin, but we’ll need to deploy munin-node on each node to monitor.

Install NGinx
We’ll use NGinx to serve static files generated by munin. It’s also possible to use NGinx as a proxy to forward requests to munin-http.

1 $ sudo apt install nginx apache2-utils

2 $ sudo htpasswd -cb /etc/nginx/htpasswd admin admin
 3 $ sudo vi /etc/nginx/sites-enabled/default

1 server {
2 listen 80 default_server;
3 listen [::]:80 default_server;
4
5 location / {
6 rewrite ^/$ munin/ redirect; break;
7 }
8 location /munin/static/ {
9 alias /etc/munin/static/;
10 expires modified +1w;
11 }
12 location /munin/ {
13 auth_basic "Restricted";
14 # Create the htpasswd file with the htpasswd tool.
15 auth_basic_user_file /etc/nginx/htpasswd;
16
17 alias /var/cache/munin/www/;
18 expires modified +310s;
19 }
20 }

Munin is up and running, and published in http.

Monitor a first node

As a first node, we’ll use our HaProxy server. We’ll deploy on it standard monitoring as well as more specific ones.

1 $ sudo apt install munin-node

Allow munin server to monitor this node

1 $ sudo vi /etc/munin/munin-node.conf

1 ...
2 host_name haodoo
3 allow ^192\.168\.56\.40$
4 ...

Add this node in server configuration

1 $ sudo vi /etc/munin/munin.conf

1 ...
2 [ha;haodoo]
3 address 192.168.56.30
4 ...
Munin is able to monitor pretty anything. The node has for job to measure and send datas to master. By default, some plugins are already
deployed (ram, cpu, …) but we can add somes more relevant to our system. A way to determine which plugins are usefull is to use munin-node-
configure itself :

1 $ sudo munin-node-configure --suggest

1 Plugin | Used | Suggestions

2 ------ | ---- | -----------
3 acpi | no | no [cannot read /proc/acpi/thermal_zone/*/temperature]
4 amavis | no | no
5 apache_accesses | no | no [LWP::UserAgent not found]
6 apache_processes | no | no [LWP::UserAgent not found]
7 apache_volume | no | no [LWP::UserAgent not found]
8 apc_envunit_ | no | no [no units to monitor]
9 bonding_err_ | no | no [No /proc/net/bonding]
10 courier_mta_mailqueue | no | no [spooldir not found]
11 courier_mta_mailstats | no | no [could not find executable]
12 courier_mta_mailvolume | no | no [could not find executable]
13 cps_ | no | no
14 cpu | yes | yes
15 ...

Result show what’s installed and what should be.

It’s possible to add more plugins on node :

1 $ sudo apt install libwww-perl

2 $ ls /usr/share/munin/plugins/ha*
3 /usr/share/munin/plugins/haproxy_ /usr/share/munin/plugins/haproxy_ng

Those 2 plugins seems relevant to our particular node. Let’s deploy them

1 $ sudo ln -s /usr/share/munin/plugins/haproxy_ /etc/munin/plugins/haproxy_backend

2 $ sudo ln -s /usr/share/munin/plugins/haproxy_ng /etc/munin/plugins/haproxy_ng
3 $ sudo vi /etc/munin/plugin-conf.d/munin-node

1 ...
2 [haproxy_*]
3 env.url http://odoo:odoo@localhost:1936/haproxy-status?stats;csv

1 $ sudo service munin-node restart

Define alarm threshold

Monitoring is good, and is even better if I can define my own threshold for warning and critical level. Thoses levels can be defined on the node,
in the plugin configuration if plugin handle it. Or can be defined on the server. In the case of the haproxy_ng plugin, it doesn’t handle warning
and critical levels. So, on the munin server, not the node:
 1 $ sudo vi /etc/munin/munin.conf

1 [ha;haodoo]
2 address 192.168.56.30
3 HAPActive.backend_odoo_ssl_act.warning 2:
4 HAPActive.backend_odoo_ssl_act.critical 1:
5 HAPActive.backend_odoo_no_ssl_act.warning 2:
6 HAPActive.backend_odoo_no_ssl_act.critical 1:

It’s also possible to add more plugins on node, from github for example, like the ones from https://github.com/munin-monitoring/contrib

1 $ sudo rm /etc/munin/plugins/haproxy_ng
2 $ cd /opt
3 $ sudo git clone https://github.com/munin-monitoring/contrib.git
4 $ sudo ln -s /opt/contrib/plugins/haproxy/haproxy_active_backend /etc/munin/plugins/haproxy_active_backend
5 $ sudo ln -s /opt/contrib/plugins/haproxy/haproxy_abort_backend /etc/munin/plugins/haproxy_abort_backend
6 $ sudo vi /etc/munin/plugin-conf.d/munin-node

1 [haproxy_*]
2 env.url http://odoo:odoo@localhost:1936/haproxy-status?stats;csv
3 env.backend backend_odoo_no_ssl backend_odoo_ssl

1 $ sudo service munin-node restart

Monitor PostgreSQL
As datas resides on this server, it’s important to monitor this particular server. Depending on your configuration, some metrics are relevant or
not, like replication delay. We add postgresql server 1 in munin config then check plugins on the postgresql node.

Allow munin server to monitor this node

1 $ sudo apt install munin-node libdbd-pg-perl

2 $ sudo vi /etc/munin/munin-node.conf

1 ...
2 host_name odoo-pg1
3 allow ^192\.168\.56\.40$
4 ...

Configure munin server to monitor it

 1 $ sudo vi /etc/munin/munin.conf

1 [pg;odoo-pg1]
2 address 192.168.56.21
3 [pg;odoo-pg2]
4 address 192.168.56.22

Suggest PostgreSQL plugins

1 $ sudo munin-node-configure --suggest

1 ...
2 postgres_autovacuum | no | yes
3 postgres_bgwriter | no | yes
4 postgres_cache_ | no | yes (+ALL +odoo)
5 postgres_checkpoints | no | yes
6 postgres_connections_ | no | yes (+ALL +odoo)
7 postgres_connections_db | no | yes
8 postgres_locks_ | no | yes (+ALL +odoo)
9 postgres_oldest_prepared_xact_ | no | no [Prepared transactions not enabled]
10 postgres_prepared_xacts_ | no | no [Prepared transactions not enabled]
11 postgres_querylength_ | no | yes (+ALL +odoo)
12 postgres_scans_ | no | yes (+odoo)
13 postgres_size_ | no | yes (+ALL +odoo)
14 ...

As suggested, deploy the PostgreSQL plugins

1 $ sudo munin-node-configure --shell

2 ln -s '/usr/share/munin/plugins/postgres_autovacuum' '/etc/munin/plugins/postgres_autovacuum'
3 ln -s '/usr/share/munin/plugins/postgres_bgwriter' '/etc/munin/plugins/postgres_bgwriter'
4 ln -s '/usr/share/munin/plugins/postgres_cache_' '/etc/munin/plugins/postgres_cache_ALL'
5 ln -s '/usr/share/munin/plugins/postgres_cache_' '/etc/munin/plugins/postgres_cache_odoo'
6 ln -s '/usr/share/munin/plugins/postgres_checkpoints' '/etc/munin/plugins/postgres_checkpoints'
7 ln -s '/usr/share/munin/plugins/postgres_connections_' '/etc/munin/plugins/postgres_connections_ALL'
8 ln -s '/usr/share/munin/plugins/postgres_connections_' '/etc/munin/plugins/postgres_connections_odoo'
9 ln -s '/usr/share/munin/plugins/postgres_connections_db' '/etc/munin/plugins/postgres_connections_db'
10 ln -s '/usr/share/munin/plugins/postgres_locks_' '/etc/munin/plugins/postgres_locks_ALL'
11 ln -s '/usr/share/munin/plugins/postgres_locks_' '/etc/munin/plugins/postgres_locks_odoo'
12 ln -s '/usr/share/munin/plugins/postgres_querylength_' '/etc/munin/plugins/postgres_querylength_ALL'
13 ln -s '/usr/share/munin/plugins/postgres_querylength_' '/etc/munin/plugins/postgres_querylength_odoo'
14 ln -s '/usr/share/munin/plugins/postgres_scans_' '/etc/munin/plugins/postgres_scans_odoo'
15 ln -s '/usr/share/munin/plugins/postgres_size_' '/etc/munin/plugins/postgres_size_ALL'
16 ln -s '/usr/share/munin/plugins/postgres_size_' '/etc/munin/plugins/postgres_size_odoo'
17 ln -s '/usr/share/munin/plugins/postgres_transactions_' '/etc/munin/plugins/postgres_transactions_ALL'
18 ln -s '/usr/share/munin/plugins/postgres_transactions_' '/etc/munin/plugins/postgres_transactions_odoo'
19 ln -s '/usr/share/munin/plugins/postgres_tuples_' '/etc/munin/plugins/postgres_tuples_odoo'
20 ln -s '/usr/share/munin/plugins/postgres_users' '/etc/munin/plugins/postgres_users'
21 ln -s '/usr/share/munin/plugins/postgres_xlog' '/etc/munin/plugins/postgres_xlog'
22 $ sudo munin-node-configure --shell | sudo sh
23 $ sudo service munin-node restart
Monitor replication
All thoses plugins are cool, but I want to monitor my streaming replication!

1 $ sudo vi /etc/munin/plugins/pg_replication

1 #!/usr/bin/env bash
2 #%# family=manual
3
4 slave=`psql postgres -A -t -c "SELECT pg_is_in_recovery();"`
5 master=`psql postgres -A -t -c "SELECT count(client_addr) from pg_stat_replication;"`
6
7 if [[ $slave == "f" ]];
8 then
9 if [[ $master -eq 0 ]];
10 then
11 exit 1
12 else
13 client=`psql postgres -A -t -c "SELECT client_addr from pg_stat_replication;"`
14 client=`dig -x $client +short`
15 fi
16 fi
17
18 case $1 in
19 config)
20 echo graph_category postgres
21 echo graph_title Streaming Replication
22 if [[ $master -gt 0 ]]; then
23 echo delay_b.label Replication Lag \($client\)\(B\)
24 echo delay_b.warning 10000000
25 echo delay_b.critical 300000000
26 fi
27 if [[ $slave == "t" ]]; then
28 echo graph_args --logarithmic --units=si
29 echo delay_s.label Replication Delay \(s\)
30 echo delay_s.warning 60
31 echo delay_s.critical 600
32 echo delay_b.label Replication Lag \(B\)
33 echo delay_b.warning 10000000
34 echo delay_b.critical 300000000
35 fi
36 exit 0
37 ;;
38 esac
39
40 if [[ $master -gt 0 ]]; then
41 # master
42 echo -n "delay_b.value "
43 psql postgres -A -t -c "select pg_xlog_location_diff(sent_location, replay_location) from pg_stat_replication;"
44 fi
45
46 if [[ $slave == "t" ]]; then
47 # slave
48 echo -n "delay_s.value "
49 psql postgres -A -t -c "
 50 SELECT
51 CASE
52 WHEN pg_last_xlog_receive_location() = pg_last_xlog_replay_location() THEN 0
53 ELSE EXTRACT (EPOCH FROM now() - pg_last_xact_replay_timestamp())
54 END;"
55 echo -n "delay_b.value "
56 psql postgres -A -t -c "SELECT pg_xlog_location_diff(pg_last_xlog_receive_location(), pg_last_xlog_replay_locat
57 fi
58
59 exit 0

1 $ sudo chmod a+x /etc/munin/plugins/pg_replication

2 $ sudo vi /etc/munin/plugin-conf.d/munin-node

1 [pg_*]
2 user postgres
3 group postgres

Don’t forget ot restart munin-node on the node, and you can run munin-cron on the master as munin user to speed up a bit.
You can also run munin-run pg_replication on the node to check if everything is ok.

Other usefull graphs

Some graphs are activated by default, and used to diagnose performance issues, size postgresql memory, …

Disk IOs
Disk latency
CPU Usage (iowait, steal, …)
 Load average
Memory usage (cache, …)

Monitor Odoo node

As usual, deploy munin-node on your server and ask munin main server to monitor it.

1 $ sudo apt install munin-node

2 $ sudo vi /etc/munin/munin-node.conf

1 ...
2 host_name odoo-1
3 allow ^192\.168\.56\.40$
4 ...

Configure munin server to monitor it

1 $ sudo vi /etc/munin/munin.conf

1 [odoo;odoo-1]
2 address 192.168.56.10
3 [odoo;odoo-2]
4 address 192.168.56.11

Adjust log-level
We want to monitor response time within odoo. To achieve this, we need to get the response time from odoo, in the odoo logs. In your odoo
config file:

1 log_level = debug_rpc

This will generate logs like:

1 2016-09-22 12:23:11,205 30977 DEBUG 9dpm-test openerp.http.rpc.request: read_subscription_data: None

2 None: time:0.014s mem: 1005056k -> 1005056k (diff: 0k)
3 2016-09-22 12:23:11,206 30977 INFO 9dpm-test werkzeug: 127.0.0.1 - - [22/Sep/2016 12:23:11] "POST
4 /mail/read_subscription_data HTTP/1.1" 200 -
5 2016-09-22 12:23:11,485 30977 DEBUG 9dpm-test openerp.http.rpc.request: call_kw: order.category r
6 ead: time:0.235s mem: 1005056k -> 1006080k (diff: 1024k)
7 2016-09-22 12:23:11,486 30977 INFO 9dpm-test werkzeug: 127.0.0.1 - - [22/Sep/2016 12:23:11] "POST
8 /web/dataset/call_kw/order.category/read HTTP/1.1" 200 -
9 2016-09-22 12:23:11,750 30977 DEBUG 9dpm-test openerp.http.rpc.request: call_kw: sale.order.line
10 read: time:0.152s mem: 1006080k -> 1007360k (diff: 1280k)
11 2016-09-22 12:23:11,752 30977 INFO 9dpm-test werkzeug: 127.0.0.1 - - [22/Sep/2016 12:23:11] "POST
12 /web/dataset/call_kw/sale.order.line/read HTTP/1.1" 200 -
13 2016-09-22 12:23:12,074 30977 DEBUG 9dpm-test openerp.http.rpc.request: call_kw: sale.order.line
14 read: time:0.169s mem: 1007360k -> 1008128k (diff: 768k)
15 2016-09-22 12:23:12,075 30977 INFO 9dpm-test werkzeug: 127.0.0.1 - - [22/Sep/2016 12:23:12] "POST
16 /web/dataset/call kw/sale order line/read HTTP/1 1" 200 -
 16 /web/dataset/call_kw/sale.order.line/read HTTP/1.1" 200 -

How many users?

A usual question is “how many concurrent users do I have?”, and another one is “How many transactions/minute do they do?”. As the number
of workers and thus CPU is directly linked to this answer, it’s important to answer.

We can distinguish two kind of requests. The longpoll ones, and the others.

Number of longpoll requests per minute ~= number of users, even inactive with just an open browser tab.

Other requests are sent by active users.

1 $ sudo vi /etc/munin/plugins/odootrmin

1 #!/bin/sh
2 #%# family=manual
3 #%# capabilities=autoconf suggest
4 case $1 in
5 autoconf)
6 exit 0
7 ;;
8 suggest)
9 exit 0
10 ;;
11 config)
12 echo graph_category odoo
13 echo graph_title odoo rpc request count
14 echo graph_vlabel num requests/minute in last 5 minutes
15 echo requests.label num requests
16 echo pollrequests.label num poll requests
17 exit 0
18 ;;
19 esac
20 # watch out for the time zone of the logs => using date -u for UTC timestamps
21 result=$(tail -60000 /var/log/odoo/odoo.log | grep -a "odoo.http.rpc.request" | grep -a -v "poll" | awk "BEGIN{coun
22 pollresult=$(tail -60000 /var/log/odoo/odoo.log | grep -a "odoo.http.rpc.request" | grep -a "poll" | awk "BEGIN{cou
23 echo "requests.value ${result}"
24 echo "pollrequests.value ${pollresult}"
25 exit 0

Once munin node restarted, and munin cron ran, a new graph appears.
Measure response time
It’s possible to extract the rpc response time from odoo logs. in /etc/munin/plugins/odooresponse:

1 #!/bin/sh
2 #%# family=manual
3 #%# capabilities=autoconf suggest
4 case $1 in
5 config)
6 echo graph_category odoo
7 echo graph_title odoo rpc requests min/average response time
8 echo graph_vlabel seconds
9 echo graph_args --units-exponent -3
10 echo min.label min
11 echo min.warning 1
12 echo min.critical 5
13 echo avg.label average
14 echo avg.warning 1
15 echo avg.critical 5
16 exit 0
17 ;;
18 esac
19 # watch out for the time zone of the logs => using date -u for UTC timestamps
20 result=$(tail -60000 /var/log/odoo/odoo.log | grep -a "odoo.http.rpc.request" | grep -a -v "poll" | awk "BEGIN{sum=
21 echo -n "min.value "
22 echo ${result} | cut -d" " -f1
23 echo -n "avg.value "
24 echo ${result} | cut -d" " -f2
25 exit 0

Once munin node restarted, and munin cron ran, a new graph appears.