Tales From A Lazy Fat DBA

Den of an Oracle DB Performance freak & a fan of Cassandra, MySQL and PostgreSQL …

  • Likes

    • 138,101
  • Archives

  • Categories

  • Subscribe

  • Advertisements
  • Prashant Dixit is the FatDBA

  • Follow Tales From A Lazy Fat DBA on WordPress.com
  • My Twitter Feeds

  • Disclaimer!

    FatDBA or Oracle ‘Ant’ is an independent web-blog/site.The experiences, Test cases, views, and opinions expressed in this website are my own and does not reflect the views or opinions of my employer.

    This site is independent of and does not represent Oracle Corporation in any way. Oracle does not officially sponsor, approve, or endorse this site or its content.
    Product and company names mentioned in this website may be the trademarks of their respective owners.

Oracle GI 11.2 Installation on RHEL 7 – Error: ohasd failed to start the Clusterware.

Posted by FatDBA on January 6, 2016

Recently as a part of solution i proposed for a new infrastructure for one of the customer, project team came with an error which they encountered during the GRID Infrastructure – Especially soon after executing the ‘Root.sh’ script. This was actually 11.2.0.4 Grid Infrastructure software installation on Oracle Enterprise 7 which is the latest release from Oracle Corp.

While troubleshooting the problem i experienced much pain getting it to install. The installation process fails when the root.sh script is run.
* Sometimes to configure Grid Infrastructure for a Stand-Alone Server system asks to run the following command as the root user:
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl

The error reported is:

[root@localhost /]# /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user ‘oracle’, privgrp ‘oinstall’..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
CRS-4664: Node localhost successfully pinned.
Adding Clusterware entries to inittab
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2016-01-01 02:14:46.806:
[client(11401)]CRS-2101:The OLR was formatted using version 3.
2016-01-01 02:14:49.572:
[client(11424)]CRS-1001:The OCR was formatted using version 3.

ohasd failed to start at /u01/app/11.2.0/grid/crs/install/roothas.pl line 377, line 4.

I hunted though various blog posts and even Oracle Metalink initially but all of them were was of little to no use.
Finally, I stumbled across an apparently poorly indexed (and titled) support note (1951613.1) that made reference to a RHEL 7 specific patch. The patch number is: 18370031.

So a Patch download and a new installation process later, I was finally able to get the GI installer to properly register the ohasd services. In the end, I was glad it was a patch, that resolved the issue since (in theory) Oracle will support it. I was surprised that the Oracle Support tech was not able to locate the patch 🙂

In applying the patch it is a little different. You have to run the GI installer to the point where it instructs you to run root.sh. Before you run root.sh, you then use OPatch to install the provided patch. Then finally you run root.sh.

Below provided are the steps performed during the fix.
1. First i had to deinstall previous GRID configuration (Where I’ve got that error message after roo.sh execution).
– During the deinstallation process it will ask you to execute few scripts which will ultimately help you to deinstall the entire Oracle Restart stack.
2. Download, Unzip and Apply the patch using OPATCH.
3. Execute the root.sh script once you applied the patch.
4. Check the services status using crs_stat.

 

Step 1:

Deinstall previous GRID configuration
[root@localhost deinstall]# su – oracle
Last login: Fri Jan 1 02:17:02 EST 2016 on pts/1
[oracle@localhost ~]$ cd /u01/app/11.2.0/grid/deinstall
[oracle@localhost deinstall]$ ./deinstall

Checking for required files and bootstrapping …
Please wait …
Location of logs /tmp/deinstall2016-01-01_02-30-16AM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############
######################### CHECK OPERATION START ###########################
[START] Install check configuration ##

Checking for existence of the Oracle home location /u01/app/11.2.0/grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Standalone Server
Oracle Base selected for deinstall is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0/grid
Checking for sufficient temp space availability on node(s) : ‘localhost.localdomain’

## [END] Install check configuration ##
Traces log file: /tmp/deinstall2016-01-01_02-30-16AM/logs//crsdc.log

Network Configuration check config START
Network de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16A/logs/netdc_check2016-01-01_02-30-40-AM.log

Network Configuration check config END
Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16AM/logs/asmcadc_check2016-01-01_02-30-41-AM.log

ASM configuration was not detected in this Oracle home. Was ASM configured in this Oracle home (y|n) [n]: n
ASM was not detected in the Oracle Home

######################### CHECK OPERATION END #########################
####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/11.2.0/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:null
Oracle Home selected for deinstall is: /u01/app/11.2.0/grid
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
ASM was not detected in the Oracle Home
Do you want to continue (y – yes, n – no)? [n]: y
A log of this session will be written to: ‘/tmp/deinstall2016-01-01_02-30-16AM/logs/deinstall_deconfig2016-01-01_02-30-35-AM.out’
Any error messages from this session will be written to: ‘/tmp/deinstall2016-01-01_02-30-16AM/logs/deinstall_deconfig2016-01-01_02-30-35-AM.err’

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16AM/logs/asmcadc_clean2016-01-01_02-30-57-AM.log
ASM Clean Configuration END

Network Configuration clean config START
Network de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16A/logs/netdc_clean2016-01-01_02-30-57-AM.log

De-configuring backup files…
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.
Network Configuration clean config END

—————————————->

Run the following command as the root user or the administrator on node “localhost”.

/tmp/deinstall2016-01-01_02-30-16AM/perl/bin/perl -I/tmp/deinstall2016-01-01_02-30-16AM/perl/lib -I/tmp/deinstall2016-01-01_02-30-16AM/crs/install /tmp/deinstall2016-01-01_02-30-16AM/crs/install/roothas.pl -force -deconfig -paramfile “/tmp/deinstall2016-01-01_02-30-16AM/response/deinstall_Ora11g_gridinfrahome1.rsp”

Press Enter after you finish running the above commands

<————————————-

This is the point where you have to execute the command highlighted and execute it from root user and wait for it to complete in order to deconfigure the Oracle Restart Stack.

[root@localhost ~]# /tmp/deinstall2016-01-01_02-30-16AM/perl/bin/perl -I/tmp/deinstall2016-01-01_02-30-16AM/perl/lib -I/tmp/deinstall2016-01-01_02-30-16AM/crs/install /tmp/deinstall2016-01-01_02-30-16AM/crs/install/roothas.pl -force -deconfig -paramfile “/tmp/deinstall2016-01-01_02-30-16AM/response/deinstall_Ora11g_gridinfrahome1.rsp”
Using configuration parameter file: /tmp/deinstall2016-01-01_02-30-16AM/response/deinstall_Ora11g_gridinfrahome1.rsp
CRS-2613: Could not find resource ‘ora.cssd’.
CRS-4000: Command Stop failed, or completed with errors.
CRS-2613: Could not find resource ‘ora.cssd’.
CRS-4000: Command Delete failed, or completed with errors.
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle Restart stack

#########################
Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home ‘/u01/app/11.2.0/grid’ from the central inventory on the local node : Done
Delete directory ‘/u01/app/11.2.0/grid’ on the local node : Done
Delete directory ‘/u01/app/oraInventory’ on the local node : Done
Delete directory ‘/u01/app/oracle’ on the local node : Done
Oracle Universal Installer cleanup was successful.
Oracle Universal Installer clean END

## [START] Oracle install clean ##
Clean install operation removing temporary directory ‘/tmp/deinstall2016-01-01_02-30-16AM’ on node ‘localhost’
## [END] Oracle install clean ##
######################### CLEAN OPERATION END #########################
####################### CLEAN OPERATION SUMMARY #######################
Oracle Restart was already stopped and de-configured on node “localhost”
Oracle Restart is stopped and de-configured successfully.
Successfully detached Oracle home ‘/u01/app/11.2.0/grid’ from the central inventory on the local node.
Successfully deleted directory ‘/u01/app/11.2.0/grid’ on the local node.
Successfully deleted directory ‘/u01/app/oraInventory’ on the local node.
Successfully deleted directory ‘/u01/app/oracle’ on the local node.
Oracle Universal Installer cleanup was successful.

Run ‘rm -rf /etc/oraInst.loc’ as root on node(s) ‘localhost’ at the end of the session.
Run ‘rm -rf /opt/ORCLfmap’ as root on node(s) ‘localhost’ at the end of the session.
Run ‘rm -rf /etc/oratab’ as root on node(s) ‘localhost’ at the end of the session.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################
############# ORACLE DEINSTALL & DECONFIG TOOL END #############

Step 2:
Set all required environment variables and call the opatch from the patch directory and follow steps.

[oracle@localhost OPatch]$ ./opatch napply -local /u01/app/11.2.0/grid/OPatch/18370031
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/11.2.0/grid/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2016-01-01_12-22-43PM_1.log

Verifying environment and performing prerequisite checks…
OPatch continues with these patches: 18370031

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = ‘/u01/app/11.2.0/grid’)

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files…
Applying interim patch ‘18370031’ to OH ‘/u01/app/11.2.0/grid’

Patching component oracle.crs, 11.2.0.4.0…

Verifying the update…
Patch 18370031 successfully applied.
Log file location: /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2016-01-01_12-22-43PM_1.log

OPatch succeeded.
[oracle@localhost OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/11.2.0/grid/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2016-01-01_12-24-54PM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2016-01-01_12-24-54PM.txt

——————————————————————————–
Installed Top-level Products (1):
Oracle Grid Infrastructure 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :
Patch 18370031 : applied on Fri Jan 01 12:24:15 EST 2016
Unique Patch ID: 17950825
Patch description: “Grid Infrastructure Patch Set Update : 11.2.0.4.x (gibugno)”
Created on 15 Aug 2014, 16:14:40 hrs PST8PDT
Bugs fixed:
18370031

——————————————————————————–

OPatch succeeded.

Step 3:
Execute the root.sh script once you applied the patch.

[root@localhost grid]# ./root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl

To configure Grid Infrastructure for a Cluster execute the following command:
/u01/app/11.2.0/grid/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

[root@localhost grid]# /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user ‘oracle’, privgrp ‘oinstall’..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
CRS-4664: Node localhost successfully pinned.
Adding Clusterware entries to oracle-ohasd.service

localhost 2016/01/01 12:28:47 /u01/app/11.2.0/grid/cdata/localhost/backup_20160101_122847.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server.

 

Step 4:
Check the services status using crs_stat.

[oracle@localhost bin]$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.cssd ora.cssd.type OFFLINE OFFLINE
ora.diskmon ora....on.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE localhost
ora.ons ora.ons.type OFFLINE OFFLINE

 

Hope That Helps
Prashant Dixit

Advertisements

20 Responses to “Oracle GI 11.2 Installation on RHEL 7 – Error: ohasd failed to start the Clusterware.”

  1. Fred said

    Hi,
    I am facing the same problem trying to install Oracle on a CentOS7 Virtual Machine (VirtualBox)
    Currently on holidays don’t have access to my MOS account though I can’t download 18370031 patch.
    Can you upload the patch and give me access to it?
    Tx in advance
    Fred

  2. Geraldo Peralta said

    Thanks!! 5 starts for you. 🙂

  3. Régis Vaz said

    Thank you very much ! 😉

  4. johnp said

    Hi FatDBA,

    I had faced the same issue on RHEL7. However, I have Exact same issue with AIX . Root.sh has failed with same message ” ohasd failed to start at “GRID_HOME/crs/install/roothas.pl line 377, line 4. ”

    I Checked for patch on support.oracle.com but as of now oracle has released patch for RHEL7 & IBM Linux on z system

    Has anyone faced this issue on AIX 7.1 ?

    Thank you!

  5. Nate Fuzi said

    Thank you very much for your step-by-step instructions. Maybe it was obvious to others, but I didn’t see where you ran through the GI installer after deinstalling the previously created Oracle GI Home, and then paused just prior to running root.sh to install the patch. It was the logical place to do it, but since you’re presumably providing all the steps, maybe you could put a comment in that spot to show that step is required. Thanks again!

  6. Pradipto said

    This post helps me alot, thanks!

  7. Zia said

    Hi Mate,

    Thanks a million for this post. It was really helpful specially when there is no article at Oracle support related to this particular issue and Oracle support engine sucks.

    Cheers.
    Zia

  8. Thanks very much. I was working with support and they failed to resolve this issue. I’ve asked them to update the title of the document to include RHEL7, but I’m pretty sure they won’t.

  9. @azzee said

    Can you please upload the patch? I am not able to download.

  10. Eliandro said

    Good job, thank you, it helped

  11. MrJansjo said

    Thanks for it…I’ll be trying.
    One question – in step 2 there is:
    “Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
    (Oracle Home = ‘/u01/app/11.2.0/grid’)

    Is the local system ready for patching? [y|n]
    y”

    Do we really have to shutdown something? Or we simply hit “y”?
    What to shutdown if we are installing yet?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

 
%d bloggers like this: