Database Administrator Blogs : RAC

1. Currently i am having one voting disk, Since the diskgroup data configured with External Redundancy .I could not add new voting disk.

[oragrid@node2 ~]$ crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 463612408dbd4f0cbf65955f3ac3ef9c (/dev/oracleasm/disks/ASMDISK5) [DATA]

Located 1 voting disk(s).

[oragrid@node2 ~]$ exit

2. So I need to create separate diskgroup with Normal redudancy ( It will have 2 mirror copy mean 2 failed group for each disk)

so we can have 3 Votedisk files.

3. Added a disk /dev/sdd in to my VM and created logical partion .

[root@node2 ~]# fdisk -l /dev/sdd

Disk /dev/sdd: 1073 MB, 1073741824 bytes, 2097152 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x539955a3

Device Boot Start End Blocks Id System

/dev/sdd1 2048 2097151 1047552 5 Extended

/dev/sdd5 4096 686079 340992 83 Linux

/dev/sdd6 688128 1370111 340992 83 Linux

/dev/sdd7 1372160 2054143 340992 83 Linux

4. Create asm disks.

[root@node2 ~]# oracleasm createdisk OCRVD1 /dev/sdd5

Writing disk header: done

Instantiating disk: done

[root@node2 ~]# oracleasm createdisk OCRVD2 /dev/sdd6

Writing disk header: done

Instantiating disk: done

[root@node2 ~]# oracleasm createdisk OCRVD3 /dev/sdd7

Writing disk header: done

Instantiating disk: done

[root@node2 ~]# oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

[root@node2 ~]# oracleasm listdisks

ASMDISK10

ASMDISK11

ASMDISK12

ASMDISK13

ASMDISK14

ASMDISK5

ASMDISK6

ASMDISK7

ASMDISK8

ASMDISK9

OCRVD1

OCRVD2

OCRVD3

[root@node2 ~]#

5. CREATE DISKGROUP OCRVD NORMAL REDUNDANCY DISK '/dev/oracleasm/disks/OCRVD1' SIZE 333M

DISK '/dev/oracleasm/disks/OCRVD2' SIZE 333M

DISK '/dev/oracleasm/disks/OCRVD3' SIZE 333M

SQL> select NAME,STATE,TYPE,TOTAL_MB from v$asm_diskgroup;

NAME STATE TYPE TOTAL_MB

------------------------------ ----------- ------ ----------

DATA MOUNTED EXTERN 18432

OCRVD MOUNTED NORMAL 996

SQL>

6. Replace new votedisk.

[oragrid@node2 trace]$ crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 463612408dbd4f0cbf65955f3ac3ef9c (/dev/oracleasm/disks/ASMDISK5) [DATA]

Located 1 voting disk(s).

[oragrid@node2 trace]$ crsctl replace votedisk +OCRVD

Successful addition of voting disk 4219d4f7a49b4fb3bf9601cc89f87794.

Successful addition of voting disk 95cb9d0d75384fc3bf0a5aae2752b422.

Successful addition of voting disk 715d2fbc57e44f62bf959e61915db38a.

Successful deletion of voting disk 463612408dbd4f0cbf65955f3ac3ef9c.

Successfully replaced voting disk group with +OCRVD.

CRS-4266: Voting file(s) successfully replaced

[oragrid@node2 trace]$ crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 4219d4f7a49b4fb3bf9601cc89f87794 (/dev/oracleasm/disks/OCRVD1) [OCRVD]

2. ONLINE 95cb9d0d75384fc3bf0a5aae2752b422 (/dev/oracleasm/disks/OCRVD2) [OCRVD]

3. ONLINE 715d2fbc57e44f62bf959e61915db38a (/dev/oracleasm/disks/OCRVD3) [OCRVD]

Located 3 voting disk(s).

[oragrid@node2 trace]$

7. OCR addition:

[oragrid@node2 trace]$ ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 4

Total space (kbytes) : 491684

Used space (kbytes) : 84440

Available space (kbytes) : 407244

ID : 939418504

Device/File Name : +DATA

Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

[oragrid@node2 trace]$ ocrconfig -add +OCRVD

PROT-20: Insufficient permission to proceed. Require privileged user

[oragrid@node2 trace]$ which ocrconfig

/opt/app/grid19c/bin/ocrconfig

[oragrid@node2 trace]$ exit

logout

[root@node2 ~]# /opt/app/grid19c/bin/ocrconfig -add +OCRVD

[root@node2 ~]# /opt/app/grid19c/bin/ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 4

Total space (kbytes) : 491684

Used space (kbytes) : 84440

Available space (kbytes) : 407244

ID : 939418504

Device/File Name : +DATA

Device/File integrity check succeeded

Device/File Name : +OCRVD

Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@node2 ~]#

[root@node2 ~]# /opt/app/grid19c/bin/ocrconfig -replace +DATA -replacement +OCRVD

PROT-34: The Oracle Cluster Registry location to be deleted is not configured.

[root@node2 ~]# /opt/app/grid19c/bin/ocrconfig -add +DATA

[root@node2 ~]# /opt/app/grid19c/bin/ocrconfig -delete +OCRVD

[root@node2 ~]# /opt/app/grid19c/bin/ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 4

Total space (kbytes) : 491684

Used space (kbytes) : 84440

Available space (kbytes) : 407244

ID : 939418504

Device/File Name : +DATA

Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@node2 ~]# /opt/app/grid19c/bin/ocrconfig -replace +DATA -replacement +OCRVD

PROT-28: Cannot delete or replace the only configured Oracle Cluster Registry location.

[root@node2 ~]# /opt/app/grid19c/bin/ocrconfig -add +DATA

PROT-29: The Oracle Cluster Registry location is already configured

[root@node2 ~]# /opt/app/grid19c/bin/ocrconfig -add +OCRVD

[root@node2 ~]# /opt/app/grid19c/bin/ocrconfig -delete +DATA

[root@node2 ~]# /opt/app/grid19c/bin/ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 4

Total space (kbytes) : 491684

Used space (kbytes) : 84440

Available space (kbytes) : 407244

ID : 939418504

Device/File Name : +OCRVD

Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@node2 ~]#

:Oracle 11g R2 RAC On Oracle Enterprise Linux 5:

Real Application Clusters (RAC) is the Oracle trade name for its database server cluster product and provides Load-Balancing, Scalability, Elasticity and High-Availability by keeping the Oracle database server product available and running across a set of multiple server nodes accessing a common database residing on Shared Storage.

Oracle Real Application Clusters (RAC) technology is Oracle's clustered version of its database server product and was introduced and popularized by Oracle with the release of 9i in 2001. The technology did exist before 9i under a different name: Oracle Parallel Server. However, Oracle Parallel Server was nowhere as fast and advanced as RAC in its architecture, application and deployment.

This Document shows the step by step of installing and setting up 3-Node 11gR2 RAC cluster. This setup uses IP Based iSCSI Openfiler SAN

as a shared storage subsystem. This setup does not have IPMI and Grid Naming Service (GNS) configured. The SCAN is resolved through DNS.

Hardware Used in setting up 2-node 11g R2 RAC using iSCSI SAN (Openfiler):

Total Machines: 4 (2 for RAC nodes + 1 for SAN + 1 for DNS)
Network Switches: 3 (for Public, Private and Shared Storage)
Extra Network Adaptors: 5 (4 for RAC nodes (2 for each node) and one for Storage Server)
Network cables: 9 (5 for RAC nodes (3 for each node), one for Shared Storage and 1 for DNS server)

Software Used for the 2-node RAC Setup using NAS (Openfiler):

SAN Storage Solution: Openfiler 2.3 (2.6.26.8-1.0.11.smp.pae.gcc3.4.x86.i686)
Operating System: Oracle Enterprise Linux 5.5 (2.6.18-194.el5PAE)
Clusterware: Oracle 11g R2 Grid Infrastructure (11.2.0.3)
Oracle RAC: Oracle RDBMS 11g R2 (11.2.0.3)

2-Node RAC Setup

Operating System: Oracle Enterprise Linux 5.5 (2.6.18-194.el5PAE):

Server: All the RAC Nodes + DNS server

Grid Infrastructure Software (Clusterware + ASM 11.2.0.1):

Server: All the RAC Nodes
ORACLE_BASE: /u/oracle/
ORACLE_HOME: /u/oracle/server/grid203
Owner: grid (Primary Group: oinstall, Secondary Group: dba)
Permissions: 755
OCR/Voting Disk Storage Type: ASM
Oracle Inventory Location: /u01/app/oraInventory

Oracle Database Software (RAC 11.2.0.1):

Server: All the RAC Nodes
ORACLE_BASE: /u/oracle/
ORACLE_HOME: /u/oracle/server/database203
Owner: oracle (Primary Group: oinstall, Secondary Group: dba)
Permissions: 755
Oracle Inventory Location: /u01/app/oraInventory

Database Name: labdb
Listener: LAB_LISTENER (TCP:1525)

Openfiler 2.3:
Server: single dedicated server acting as SAN
OS: Openfiler 2.3 (2.6.26.8-1.0.11.smp.pae.gcc3.4.x86.i686).

Server Hardware Requirements:

Each node in the Cluster must meet the below requirement.
At least 1024 x 768 display resolution, so that OUI displays correctly.
1 GB of space in the /tmp directory
5.5 GB space for Oracle Grid Infrastructure Home.
At least 2.5 GB of RAM and equivalent swap space (for 32 bit installation as in my case).
All the RAC nodes must share the same Instruction Set Architecture. For a testing RAC setup, it is possible to install RAC on servers with mixtures of Intel 32 and AMD 32 with differences in sizes of Memory/CPU speed.

Minimum Required RPMs for OEL 5.5 (All the 3 RAC Nodes):

binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
elfutils-libelf-devel-static-0.125
gcc-4.1.2
gcc-c++-4.1.2
glibc-2.5-24
glibc-common-2.5
glibc-devel-2.5
glibc-headers-2.5
kernel-headers-2.6.18
ksh-20060214
libaio-0.3.106
libaio-devel-0.3.106
libgcc-4.1.2
libgomp-4.1.2
libstdc++-4.1.2
libstdc++-devel-4.1.2
make-3.81
numactl-devel-0.9.8.i386
sysstat-7.0.2
unixODBC-2.2.11
unixODBC-devel-2.2.11

Set the below Kernel Parameters with recommended range in /etc/sysctl.conf
# Kernel sysctl configuration file for Oracle Enterprise Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 1

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 8192

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4294967295

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 1073741824
# For 11g, recommended value for file-max is 6815744
fs.file-max = 6815744
# For 10g, uncomment 'fs.file-max 327679', comment other entries for this parameter and re-run sysctl -p
# fs.file-max:327679
kernel.msgmni = 2878
kernel.sem = 250 32000 100 142
kernel.shmmni = 4096
net.core.rmem_default = 262144
# For 11g, recommended value for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304
# For 10g, uncomment 'net.core.rmem_max 2097152', comment other entries for this parameter and re-run sysctl -p
# net.core.rmem_max=2097152
net.core.wmem_default = 262144
# For 11g, recommended value for wmem_max is 1048576
net.core.wmem_max = 1048576
# For 10g, uncomment 'net.core.wmem_max 262144', comment other entries for this parameter and re-run sysctl -p
# net.core.wmem_max:262144
fs.aio-max-nr = 3145728
# For 11g, recommended value for ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500
# For 10g, uncomment 'net.ipv4.ip_local_port_range 1024 65000', comment other entries for this parameter and re-run sysctl -p
# net.ipv4.ip_local_port_range:1024 65000
# Added min_free_kbytes 50MB to

Grid Infrastructure installation Steps