Upgrade 11gR2 RAC to 12c RAC on RHEL

In this article we will see the steps to Upgrade 11gR2 RAC to 12c RAC on RHEL. We will be doing upgradation of Oracle version 11.2.0.3 to 12.1.0.2 which will include Grid and Database both.

The current 11gR2 RAC configuration is as below,
• OS: RHEL 5.8
• Oracle Software owner: oracle
• Grid Software owner: grid
• Oracle Home: /u01/app/oracle/products/11.2.0/db
• Grid Home: /u01/app/11.2.0/crs
• The OCR, Voting Disk and Database files are stored in ASM diskgroups.
• RAC Nodes – rac1,rac2

[root@rac2 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3128
         Available space (kbytes) :     258992
         ID                       :  100709723
         Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check succeeded
 
                                    Device/File not configured
 
                                    Device/File not configured
 
                                    Device/File not configured
 
                                    Device/File not configured
 
         Cluster registry integrity check succeeded
 
         Logical corruption check succeeded


[root@rac2 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   b1594051a8eb4f45bfdf58c241afced1 (/dev/raw/raw1) [OCR_VOTE]
Located 1 voting disk(s).

[grid@rac2 ~]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576      8999     8669                0            8669              0             N  CTRL_LOG1/
MOUNTED  EXTERN  N         512   4096  1048576      8999     8669                0            8669              0             N  CTRL_LOG2/
MOUNTED  EXTERN  N         512   4096  1048576     12999    11503                0           11503              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576     17999    17603                0           17603              0             Y  OCR_VOTE/
MOUNTED  EXTERN  N         512   4096  1048576     12999    12904

========================================================================

• Complete prerequisites

Complete the below checks before proceeding with the upgrade installation of 12c Clusterware

– First of all make sure you have downloaded the below software from Oracle Support site.

Grid binaries
• linuxamd64_12102_grid_1of2.zip
• linuxamd64_12102_grid_2of2.zip

Oracle binaries
• linuxamd64_12102_database_1of2.zip
• linuxamd64_12102_database_2of2.zip

Perform the below checks required for 12c installation.

– Stop the “avahi-daemon” process and disable it

# /etc/init.d/avahi-daemon stop
# chkconfig avahi-daemon off
# chkconfig –list avahi-daemon

– Add below kernel parameter in “sysctl.conf” file

# vi /etc/sysctl.conf
kernel.panic_on_oops = 1

# sysctl -p

– Add below lines “/etc/rc.local” file

chown grid:oinstall /dev/sdb1
chown grid:oinstall /dev/sdc1
chown grid:oinstall /dev/sdd1
chown grid:oinstall /dev/sde1
chown grid:oinstall /dev/sdf1
chmod 660 /dev/sdb1
chmod 660 /dev/sdc1
chmod 660 /dev/sdd1
chmod 660 /dev/sde1
chmod 660 /dev/sdf1

– Run the below command as “grid” user to check whether rolling upgrade prechecks have been completed or not. For this the 11gR2 RAC cluster should be up and running.

$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/11.2.0/crs -dest_crshome /u01/app/12.1.0/crs -dest_version 12.1.0.2.0 -fixup -verbose

Verify that all the requirements have “Passed”. If any prechecks failed, resolve it and again run the cluvfy utility.

=========================================================================

• Upgrade 11gR2 Clusterware and ASM

Once the pre-requisites are completed, you can proceed with the Grid upgradation.
[Note: The existing 11g RAC cluster and ASM should be up and running on both the nodes]

– Login as grid user, Unzip the grid software and start the installation.
$ cd grid
$ ./runInstaller

– SELECT INSTALLATION OPTION Screen

Upgrade 11gR2 RAC Grid Select Installation Option

The upgrade option will be automatically selected. Click on Next to continue.

– SELECT PRODUCT LANGUAGES Screen

Upgrade 11gR2 RAC Grid Select Product Languages

Click on Next to continue.

– GRID INFRASTRUCTURE NODE SELECTION Screen

By default, both nodes will be selected. Click on Next to continue.

– SPECIFY MANAGEMENT OPTIONS Screen

Upgrade 11gR2 RAC Grid Specify Management Options

You can specify the OMS details to register your database in OEM or leave it and proceed without registering.

– OPERATING SYSTEM GROUPS Screen

Check whether appropriate groups are shown. Click on Next to continue.

– INSTALLATION LOCATION Screen

Upgrade 11gR2 RAC Grid Installation Location

Enter the path for Oracle Software installation and Click on Next to continue.

– ROOT SCRIPT EXECUTION CONFIGURATION Screen

Upgrade 11gR2 RAC Grid Root Script Execution Cfg

You can provide root credentials here if you want to automate the execution of root scripts. If you want to execute root script manually then you can ignore and continue.

– SUMMARY Screen

Check whether all the information displayed is proper or not. To continue with the installation click on Install.

– INSTALL PRODUCT Screen

You can observe the installation process.

– EXECUTE CONFIGURATION SCRIPTS Screen

Upgrade 11gR2 RAC Grid Execute Configuration Scripts

Run the root.sh script on both nodes. I have attached output of root.sh script for both nodes.

root.sh script output of Node 1

[root@rac1 ~]# /u01/app/12.1.0/crs/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/crs

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/crs/crs/install/crsconfig_params
2016/03/24 11:11:42 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2016/03/24 11:12:24 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2016/03/24 11:12:33 CLSRSC-464: Starting retrieval of the cluster configuration data

2016/03/24 11:12:52 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2016/03/24 11:13:19 CLSRSC-515: Starting OCR manual backup.

2016/03/24 11:13:24 CLSRSC-516: OCR manual backup successful.

2016/03/24 11:13:35 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2016/03/24 11:13:36 CLSRSC-482: Running command: '/u01/app/12.1.0/crs/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/11.2.0/crs -oldCRSVersion 11.2.0.3.0 -nodeNumber 1 -firstNode true -startRolling true'

ASM configuration upgraded in local node successfully.

2016/03/24 11:13:49 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode

2016/03/24 11:13:49 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2016/03/24 11:15:51 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

OLR initialization - successful
2016/03/24 11:20:43 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/03/24 11:25:47 CLSRSC-472: Attempting to export the OCR

2016/03/24 11:25:47 CLSRSC-482: Running command: 'ocrconfig -upgrade grid oinstall'

2016/03/24 11:26:15 CLSRSC-473: Successfully exported the OCR

2016/03/24 11:26:23 CLSRSC-486:
At this stage of upgrade, the OCR has changed.
Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2016/03/24 11:26:23 CLSRSC-541:
To downgrade the cluster:
1. All nodes that have been upgraded must be downgraded.

2016/03/24 11:26:24 CLSRSC-542:
2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2016/03/24 11:26:25 CLSRSC-543:
3. The downgrade command must be run on the node rac1 with the '-lastnode' option to restore global configuration data.

2016/03/24 11:26:47 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2016/03/24 11:27:25 CLSRSC-474: Initiating upgrade of resource types

2016/03/24 11:27:57 CLSRSC-482: Running command: 'upgrade model -s 11.2.0.3.0 -d 12.1.0.2.0 -p first'

2016/03/24 11:27:57 CLSRSC-475: Upgrade of resource types successfully initiated.

2016/03/24 11:28:12 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

root.sh script output of Node 2

[root@rac2 ~]# /u01/app/12.1.0/crs/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/crs

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/crs/crs/install/crsconfig_params
2016/03/24 11:52:56 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2016/03/24 11:52:56 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.

2016/03/24 11:53:31 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.

2016/03/24 11:53:45 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2016/03/24 11:53:49 CLSRSC-464: Starting retrieval of the cluster configuration data

2016/03/24 11:54:04 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2016/03/24 11:54:29 CLSRSC-343: Successfully started Oracle Clusterware stack

ASM configuration upgraded in local node successfully.

2016/03/24 11:54:38 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2016/03/24 11:54:38 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

OLR initialization - successful
2016/03/24 11:54:56 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/03/24 11:55:05 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Start upgrade invoked..
2016/03/24 11:55:26 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded

2016/03/24 11:55:26 CLSRSC-482: Running command: '/u01/app/12.1.0/crs/bin/crsctl set crs activeversion'

Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the OCR.
Started to upgrade the CSS.
The CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Successfully upgraded the Oracle Clusterware.
Oracle Clusterware operating version was successfully set to 12.1.0.2.0
2016/03/24 11:55:41 CLSRSC-479: Successfully set Oracle Clusterware active version

2016/03/24 11:55:48 CLSRSC-476: Finishing upgrade of resource types

2016/03/24 11:56:08 CLSRSC-482: Running command: 'upgrade model -s 11.2.0.3.0 -d 12.1.0.2.0 -p last'

2016/03/24 11:56:08 CLSRSC-477: Successfully completed upgrade of resource types

2016/03/24 11:57:02 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Click on OK after execution of root.sh script is completed on both nodes. Once you click on OK, the installation will proceed further as shown in below screenshot.

Upgrade 11gR2 RAC Grid Install Product 2

Wait for the installation to complete.

– FINISH Screen

The last screen confirms that upgrade of Grid Infrastructure was successful.

=========================================================================

• Post Installation Sanity Checks

– You can confirm the status of 12c cluster using below commands,

$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.1.0.2.0]
 
 
$ crsctl query crs softwareversion
Oracle Clusterware version on node [rac1] is [12.1.0.2.0]
 
 
$ crsctl check cluster -all
**************************************************************
rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

– Check status of OCR and Voting disk

# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       9196
         Available space (kbytes) :     400372
         ID                       :  100709723
         Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check succeeded
 
                                    Device/File not configured
 
                                    Device/File not configured
 
                                    Device/File not configured
 
                                    Device/File not configured
 
         Cluster registry integrity check succeeded
 
         Logical corruption check succeeded
 
 
# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   b1594051a8eb4f45bfdf58c241afced1 (/dev/sdb1) [OCR_VOTE]
Located 1 voting disk(s).

– Detach old (11gR2) GRID home

If all post checks are completed and OK, then you can detach the old GRID home as shown below.

$ /u01/app/11.2.0/crs/deinstall/deinstall

If you get errors for some files that they could not be deleted then you can remove it manually after the above script finishes.

=========================================================================

• Install 12c Oracle Database Software

– Login as oracle user and start the installation.
$ cd database
$ ./runInstaller

– CONFIGURE SECURITY UPDATES Screen

Upgrade 11gR2 RAC DB Configure Security Updates

Uncheck the option “I wish to receive security updates….” and click on Next to continue the installation.

– INSTALLATION OPTION Screen

Upgrade 11gR2 RAC DB Installation Option

Select “Install Database Software Only” and click on Next to continue.

– GRID INSTALLATION OPTIONS Screen

Upgrade 11gR2 RAC DB Grid Installation Options

By default, the option “Oracle Real Application Clusters database installation” is selected. Make sure that all the nodes are also selected before proceeding with the installation.

– NODES SELECTION Screen