Tales From A Lazy Fat DBA

Recently as a part of solution i proposed for a new infrastructure for one of the customer, project team came with an error which they encountered during the GRID Infrastructure – Especially soon after executing the ‘Root.sh’ script. This was actually 11.2.0.4 Grid Infrastructure software installation on Oracle Enterprise 7 which is the latest release from Oracle Corp.

While troubleshooting the problem i experienced much pain getting it to install. The installation process fails when the root.sh script is run.
* Sometimes to configure Grid Infrastructure for a Stand-Alone Server system asks to run the following command as the root user:
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl

The error reported is:

[root@localhost /]# /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user ‘oracle’, privgrp ‘oinstall’..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
CRS-4664: Node localhost successfully pinned.
Adding Clusterware entries to inittab
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2016-01-01 02:14:46.806:
[client(11401)]CRS-2101:The OLR was formatted using version 3.
2016-01-01 02:14:49.572:
[client(11424)]CRS-1001:The OCR was formatted using version 3.

ohasd failed to start at /u01/app/11.2.0/grid/crs/install/roothas.pl line 377, line 4.

I hunted though various blog posts and even Oracle Metalink initially but all of them were was of little to no use.
Finally, I stumbled across an apparently poorly indexed (and titled) support note (1951613.1) that made reference to a RHEL 7 specific patch. The patch number is: 18370031.

So a Patch download and a new installation process later, I was finally able to get the GI installer to properly register the ohasd services. In the end, I was glad it was a patch, that resolved the issue since (in theory) Oracle will support it. I was surprised that the Oracle Support tech was not able to locate the patch 🙂

In applying the patch it is a little different. You have to run the GI installer to the point where it instructs you to run root.sh. Before you run root.sh, you then use OPatch to install the provided patch. Then finally you run root.sh.

Below provided are the steps performed during the fix.
1. First i had to deinstall previous GRID configuration (Where I’ve got that error message after roo.sh execution).
– During the deinstallation process it will ask you to execute few scripts which will ultimately help you to deinstall the entire Oracle Restart stack.
2. Download, Unzip and Apply the patch using OPATCH.
3. Execute the root.sh script once you applied the patch.
4. Check the services status using crs_stat.

 

Step 1:

Deinstall previous GRID configuration
[root@localhost deinstall]# su – oracle
Last login: Fri Jan 1 02:17:02 EST 2016 on pts/1
[oracle@localhost ~]$ cd /u01/app/11.2.0/grid/deinstall
[oracle@localhost deinstall]$ ./deinstall

Checking for required files and bootstrapping …
Please wait …
Location of logs /tmp/deinstall2016-01-01_02-30-16AM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############
######################### CHECK OPERATION START ###########################
[START] Install check configuration ##

Checking for existence of the Oracle home location /u01/app/11.2.0/grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Standalone Server
Oracle Base selected for deinstall is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0/grid
Checking for sufficient temp space availability on node(s) : ‘localhost.localdomain’

## [END] Install check configuration ##
Traces log file: /tmp/deinstall2016-01-01_02-30-16AM/logs//crsdc.log

Network Configuration check config START
Network de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16A/logs/netdc_check2016-01-01_02-30-40-AM.log

Network Configuration check config END
Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16AM/logs/asmcadc_check2016-01-01_02-30-41-AM.log

ASM configuration was not detected in this Oracle home. Was ASM configured in this Oracle home (y|n) [n]: n
ASM was not detected in the Oracle Home

######################### CHECK OPERATION END #########################
####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/11.2.0/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:null
Oracle Home selected for deinstall is: /u01/app/11.2.0/grid
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
ASM was not detected in the Oracle Home
Do you want to continue (y – yes, n – no)? [n]: y
A log of this session will be written to: ‘/tmp/deinstall2016-01-01_02-30-16AM/logs/deinstall_deconfig2016-01-01_02-30-35-AM.out’
Any error messages from this session will be written to: ‘/tmp/deinstall2016-01-01_02-30-16AM/logs/deinstall_deconfig2016-01-01_02-30-35-AM.err’

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16AM/logs/asmcadc_clean2016-01-01_02-30-57-AM.log
ASM Clean Configuration END

Network Configuration clean config START
Network de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16A/logs/netdc_clean2016-01-01_02-30-57-AM.log

De-configuring backup files…
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.
Network Configuration clean config END

—————————————->

Run the following command as the root user or the administrator on node “localhost”.

/tmp/deinstall2016-01-01_02-30-16AM/perl/bin/perl -I/tmp/deinstall2016-01-01_02-30-16AM/perl/lib -I/tmp/deinstall2016-01-01_02-30-16AM/crs/install /tmp/deinstall2016-01-01_02-30-16AM/crs/install/roothas.pl -force -deconfig -paramfile “/tmp/deinstall2016-01-01_02-30-16AM/response/deinstall_Ora11g_gridinfrahome1.rsp”

Press Enter after you finish running the above commands

<————————————-

This is the point where you have to execute the command highlighted and execute it from root user and wait for it to complete in order to deconfigure the Oracle Restart Stack.

[root@localhost ~]# /tmp/deinstall2016-01-01_02-30-16AM/perl/bin/perl -I/tmp/deinstall2016-01-01_02-30-16AM/perl/lib -I/tmp/deinstall2016-01-01_02-30-16AM/crs/install /tmp/deinstall2016-01-01_02-30-16AM/crs/install/roothas.pl -force -deconfig -paramfile “/tmp/deinstall2016-01-01_02-30-16AM/response/deinstall_Ora11g_gridinfrahome1.rsp”
Using configuration parameter file: /tmp/deinstall2016-01-01_02-30-16AM/response/deinstall_Ora11g_gridinfrahome1.rsp
CRS-2613: Could not find resource ‘ora.cssd’.
CRS-4000: Command Stop failed, or completed with errors.
CRS-2613: Could not find resource ‘ora.cssd’.
CRS-4000: Command Delete failed, or completed with errors.
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle Restart stack

#########################
Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home ‘/u01/app/11.2.0/grid’ from the central inventory on the local node : Done
Delete directory ‘/u01/app/11.2.0/grid’ on the local node : Done
Delete directory ‘/u01/app/oraInventory’ on the local node : Done
Delete directory ‘/u01/app/oracle’ on the local node : Done
Oracle Universal Installer cleanup was successful.
Oracle Universal Installer clean END

## [START] Oracle install clean ##
Clean install operation removing temporary directory ‘/tmp/deinstall2016-01-01_02-30-16AM’ on node ‘localhost’
## [END] Oracle install clean ##
######################### CLEAN OPERATION END #########################
####################### CLEAN OPERATION SUMMARY #######################
Oracle Restart was already stopped and de-configured on node “localhost”
Oracle Restart is stopped and de-configured successfully.
Successfully detached Oracle home ‘/u01/app/11.2.0/grid’ from the central inventory on the local node.
Successfully deleted directory ‘/u01/app/11.2.0/grid’ on the local node.
Successfully deleted directory ‘/u01/app/oraInventory’ on the local node.
Successfully deleted directory ‘/u01/app/oracle’ on the local node.
Oracle Universal Installer cleanup was successful.

Run ‘rm -rf /etc/oraInst.loc’ as root on node(s) ‘localhost’ at the end of the session.
Run ‘rm -rf /opt/ORCLfmap’ as root on node(s) ‘localhost’ at the end of the session.
Run ‘rm -rf /etc/oratab’ as root on node(s) ‘localhost’ at the end of the session.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################
############# ORACLE DEINSTALL & DECONFIG TOOL END #############

Step 2:
Set all required environment variables and call the opatch from the patch directory and follow steps.

[oracle@localhost OPatch]$ ./opatch napply -local /u01/app/11.2.0/grid/OPatch/18370031
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/11.2.0/grid/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2016-01-01_12-22-43PM_1.log

Verifying environment and performing prerequisite checks…
OPatch continues with these patches: 18370031

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = ‘/u01/app/11.2.0/grid’)

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files…
Applying interim patch ‘18370031’ to OH ‘/u01/app/11.2.0/grid’

Patching component oracle.crs, 11.2.0.4.0…

Verifying the update…
Patch 18370031 successfully applied.
Log file location: /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2016-01-01_12-22-43PM_1.log

OPatch succeeded.
[oracle@localhost OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/11.2.0/grid/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2016-01-01_12-24-54PM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2016-01-01_12-24-54PM.txt

——————————————————————————–
Installed Top-level Products (1):
Oracle Grid Infrastructure 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :
Patch 18370031 : applied on Fri Jan 01 12:24:15 EST 2016
Unique Patch ID: 17950825
Patch description: “Grid Infrastructure Patch Set Update : 11.2.0.4.x (gibugno)”
Created on 15 Aug 2014, 16:14:40 hrs PST8PDT
Bugs fixed:
18370031
——————————————————————————–

OPatch succeeded.

Step 3:
Execute the root.sh script once you applied the patch.

[root@localhost grid]# ./root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl

To configure Grid Infrastructure for a Cluster execute the following command:
/u01/app/11.2.0/grid/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

[root@localhost grid]# /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user ‘oracle’, privgrp ‘oinstall’..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
CRS-4664: Node localhost successfully pinned.
Adding Clusterware entries to oracle-ohasd.service

localhost 2016/01/01 12:28:47 /u01/app/11.2.0/grid/cdata/localhost/backup_20160101_122847.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server.

 

Step 4:
Check the services status using crs_stat.

[oracle@localhost bin]$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.cssd ora.cssd.type OFFLINE OFFLINE
ora.diskmon ora....on.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE localhost
ora.ons ora.ons.type OFFLINE OFFLINE

 

Hope That Helps
Prashant Dixit