Oracle 11g RAC with ASM Checklist

Sudipta Ganguly

Oracle 11g RAC with ASM Page 1 of 7

 Operating System Pre-Requisites for Oracle 11g RAC with ASM

System Requirements

S. No Description Minimum Requirement Team

1 Memory 2.5 GB MIN SYSTEMS

2 Swap 1.5 times the memory. Max 16 GB SYSTEMS

3 Temp Space 1 GB MIN SYSTEMS

4 Disk Space 3 Disk MIN, (5 recommended) each of minimum 1gb for oracle clusterware files (ocr and SYSTEMS
vd) and Minimum 5gb for grid home.
5 Groups oinstall, dba, osasm (Create it on both nodes) SYSTEMS

6 Users grid, oracle, asm with dba, oinstall SYSTEMS

asm owner
Verify user nobody exists
7 Binary Staging Area /staging (Should be owned by the above users) SYSTEMS

8 NTP Settings Modify /etc/sysconfig/ntpd -> OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" SYSTEMS

9 RPMS binutils-2.17.50.0.6 SYSTEMS

compat-libstdc++-33-3.2.3
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
elfutils-libelf-devel-static-0.125
gcc-4.1.2
gcc-c++-4.1.2
glibc-2.5-24
glibc-common-2.5
glibc-devel-2.5
glibc-headers-2.5
kernel-headers-2.6.18
ksh-20060214
libaio-0.3.106
libaio-devel-0.3.106
libgcc-4.1.2
libgomp-4.1.2
libstdc++-4.1.2
libstdc++-devel-4.1.2
make-3.81
sysstat-7.0.2
unixODBC-2.2.11
unixODBC-devel-2.2.11
cvuqdisk -> To discover shared disk by clv utility
10 Shell Limits Add to /etc/security/limits.conf : oracle SYSTEMS
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

Add the following lines to the /etc/pam.d/login file, if they are not already present:
session required /lib/security/pam_limits.so
session required pam_limits.so

Oracle 11g RAC with ASM Page 2 of 7

 11 Kernel Parameters fs.aio-max-nr = 1048576 SYSTEMS
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

12 Installation Folders /u01/app/oracle –> for $ORACLE_BASE for Oracle Database instance SYSTEMS
/u01/app/grid –> for $ORACLE_BASE for ASM instance
/u01/app/11.2.0/grid –> for $ORACLE_HOME for ASM instance
13 Folder Permissions chown -R grid:dba /u01 SYSTEMS
chown -R oracle:dba /u01/app/oracle
14 SSH Configure SSH on each RAC Node SYSTEMS

15 VNC X-Windows Emulator SYSTEMS

16 Oracle User Profile ~oracle/.bash_profile DBA

~oracle/.bashrc
~oracle//grid_env
~oracle//db_env
Network Requirements

1 SCAN IP single client access name need to be configured at dns with 03 IP address (recommended) SYSTEMS

scan-129069.pdf

2 Node IP's IPs for 02 Node. This IP, names mapping should be updated in hosts file SYSTEMS

i. Public -> 02 IPs

ii. VIP -> 02 IPs from public subnet
iii. Private -> 02 IPs from private subnet. (Protocol –UDP). This subnet should be exclusive for
internode communication
3 Correct Entry The /etc/hosts on rac01 and rac02 entries have the following: NETWORK

Wrong Entry

10.177.30.4 rac01-vip.esrd.com rac02-vip <------------- why does this say rac02-vip

10.177.30.5 rac02-vip.esrd.com rac02-vip

Correct Entry

from
10.177.30.4 rac01-vip.esrd.com rac02-vip
to
10.177.30.4 rac01-vip.esrd.com rac01-vip

* If /etc/hostfile has an incorrect entry follow the steps mentioned in the document to resolve the
issue

Oracle 11g RAC with ASM Page 3 of 7

ASM Requirements

1 ASM RPMS These rpms need to be installed for oracle ASM. Need to check kernel version. Note each SYSTEMS
kernel version has corresponding oracleasm

oracleasm-support-2.1.8-1.el5.x86_64.rpm
oracleasmlib-2.0.4-1.el5.x86_64.rpm
oracleasm-2.6.18-194.17.4.el5-2.0.5-1.el5.x86_64.rpm
2 ASMLib Download ASMLib SYSTEMS

oracleasm-support-2.1.3-1.el5.x86_64.rpm
oracleasmlib-2.0.4-1.el5.x86_64.rpm
3 Multipath Device Check ASM is pointing to MULTIPATH device on both the nodes (NEED TO PERFORM AS SYSTEMS
ROOT USER)

Add/Edit following in "/etc/sysconfig/oracleasm" file on both the nodes and restart oracleasm on
both the nodes.

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER="dm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE="sd"

then issue

/etc/init.d/oracleasm stop
/etc/init.d/oracleasm start

Note:- Perform the steps mentioned below (Oracle Support Recommendations)

/usr/sbin/oracleasm exit
/usr/sbin/oracleasm configure -e -u grid -g dba -s y -o "dm" -x "sd"
/usr/sbin/oracleasm init
/usr/sbin/oracleasm scandisks

Issue the follwing command to check that the disks are same on both the nodes

[root@rac01 ~]# /etc/init.d/oracleasm listdisks | xargs /etc/init.d/oracleasm querydisk -d

[root@rac02 ~]# /etc/init.d/oracleasm listdisks | xargs /etc/init.d/oracleasm querydisk -d

*Note:

1. If we are supposed to build the environment as per clients requirement then check that the pre-
requisites are met as per clients specifications.
2. All root.sh for RAC nodes has to be run on a node at a time and not in PARALLEL. Follow the
attached document to resolve the issue if root.sh is run in PARALLEL.

Oracle 11g RAC with ASM Page 4 of 7

Commands to check basic pre-requisites

Please run the following commands to check that basic pre-requisites are met before installing Oracle

System Requirements

S. No Parameters Commands

1 Memory grep MemTotal /proc/meminfo

2 Swap grep MemTotal /proc/meminfo
3 Temp Space du –h
4 Disk Space du –h
5 Groups # /usr/sbin/groupadd -g 501 oinstall
# /usr/sbin/groupadd -g 502 crs
# /usr/sbin/groupadd -g 503 dba
# /usr/sbin/groupadd -g 505 oper
# /usr/sbin/groupadd -g 504 asm
# /usr/sbin/groupadd -g 506 asmdba

6 Users # /usr/sbin/useradd -u 501 -g oinstall oracle

# /usr/sbin/useradd -u 504 -g oinstall -G asm asm
# /usr/sbin/useradd -u 1100 -g oinstall -G dba grid
# /usr/sbin/useradd -u number nobody

7 Binary Staging Area /u01/app/oracle

8 NTP Settings ntpstat
9 RPMS Commands to check that all necessary RPMS are installed

rpm -q binutils compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel \

gcc gcc-c++ glibc glibc-common glibc-devel glibc-headers ksh \
libaio libaio-devel libgcc libstdc++- libstdc++-devel make \
sysstat unixODBC unixODBC-devel
10 Shell Limits cat /etc/security/limits.conf
cat /etc/pam.d/login
11 Kernel Params cat /etc/sysctl.conf
Dynamic update:/sbin/sysctl -p
12 Installation Folders # mkdir -p /u01/app/oracle
# chown -R oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/app/oracle
# chown -R grid:dba /u01
# chown -R oracle:dba /u01/app/oracle
13 Oracle User Profile This is a sample as a reference. Please create the file using exact values.

Oracle User
Profile.pdf
14 Check User On both the nodes issue:
equivalence Node 1:
$ ssh node1 date
$ ssh node 2 date

Node 2:
$ ssh node 1 date
$ ssh node 2 date

Oracle 11g RAC with ASM Page 5 of 7

 Or

$ cluvfy comp admprv -n node1,node2 -o user_equiv -sshonly -verbose

Network Requirements
1 SCAN IP Ask Systems/Network for SCAN IPs
2 Node IP's cat /etc/hosts file
*Each node must have at least two network adapters; one for the public network
interface and one for the private network interface (the RAC interconnect).

These must be the same on all database nodes:

• eth0
• eth1

*The interface names associated with the network adapters for each network must be
the same on all nodes

*For the public network, each network adapter must support TCP/IP.

*For the private network, the interconnect must support the user datagram protocol
(UDP) using high-speed network adapters and switches that support TCP/IP (Gigabit
Ethernet or better recommended).
ASM Requirements
1 ASMLib Check that ASMLib is installed and configured

2 ASM RPMS Verify ASM RPMs are installed: rpm -qa |grep oracle
3 Disks Run the following command to see that the disks are only be stamped by one node
# /etc/init.d/oracleasm scandisks
# /etc/init.d/oracleasm listdisks

4 Multipath Check that the disks are pointing to same multipath devices on both nodes

Node 1: ls -ltr /dev/oracleasm/disks

Node 2: ls -ltr /dev/oracleasm/disks

5 Disk Allocation (5 recommended) each of minimum 1gb for oracle clusterware files (ocr and vd) and
Minimum 5gb for grid home.

RAC Requirements

1 Cluster Health CRSCTL Utility : # ./crsctl check cluster –all

2 Cluster Nodes OLSNODES : # ./olsnodes
3 Installation Information ./runcluvfy.sh stage -pre crsinst -n psmlnlsrac01,psmlnlsrac02 –verbose
4 MULTICAST If you are installting Oracle 11.2.0.2, you need to install patch 9974223 to get past the
multicasting issue. The issue is worked upon from Oracle 11.2.0.3

Metalink-Multicasting Steps to apply patch

.pdf 9974223.pdf
Oracle Installation Requirements (Specially for Migration and Upgrade)
1 OS Version uname -a

2 Database Version Select * from v$version;

3 Patch Applied Run the following command to see that the disks are only be stamped by one node
$ opatch - lsinventory
Select * from dba_registry;
Select * from v$banner;

Oracle 11g RAC with ASM Page 6 of 7

 4 Oracle Binaries Check that oracle binaries are available on the server or
Download binaries from oracle OTN or EDelivery site
5 Oracle Patch Download Download all the necessary patches to be applied from MOS
6 Patch Readme Download the Readme files associated with all the patches and follow the steps
mentioned in the readme file to apply the patch

If any of the checks mentioned above fails raise an IssueTrak ticket for the Tricore Systems/Network
team.

Oracle 11g RAC with ASM Page 7 of 7