Tales From A Lazy Fat DBA

Fan of Oracle DB & Performance, PostgreSQL & Cassandra … \,,/

  • Likes

    • 236,634
  • Archives

  • ŗ•ź

  • Categories

  • Subscribe

  • Prashant Dixit is the FatDBA

  • Follow Tales From A Lazy Fat DBA on WordPress.com
  • My Twitter Feeds

  • Disclaimer!

    FatDBA or Oracle ‘Ant’ is an independent web-blog/site.The experiences, Test cases, views, and opinions expressed in this website are my own and does not reflect the views or opinions of my employer.

    This site is independent of and does not represent Oracle Corporation in any way. Oracle does not officially sponsor, approve, or endorse this site or its content.
    Product and company names mentioned in this website may be the trademarks of their respective owners.

Posts Tagged ‘Errors’

Something Strikingly odd with Oracle: DB with Two ‘Current’ log-files!!

Posted by FatDBA on January 8, 2014

Something real uncanny happened with one of our databases recently: There were 2 log-files with CURRENT status: Strange right!!!

SQL> select GROUP#,SEQUENCE#,BYTES/1024/1024 as SIZE_MB,MEMBERS,STATUS from v$log;
GROUP#  SEQUENCE#      SIZE_MB    MEMBERS STATUS
‚ÄĒ‚ÄĒ‚ÄĒ- ‚ÄĒ‚ÄĒ‚ÄĒ- ‚ÄĒ‚ÄĒ‚ÄĒ- ‚ÄĒ‚ÄĒ‚ÄĒ-¬†¬† ‚ÄĒ‚ÄĒ‚ÄĒ‚ÄĒ‚ÄĒ
1       15651       50          2 INACTIVE
3       15653       50          2 CURRENT
2       15652       50          2 CURRENT
4       15650       50          2 INACTIVE

Found the ‘Notorious’ Generic error ORA-600 in alert log :
ORA-00600: internal error code, arguments: [3705], [1], [3], [2], [8], [], [], []

We had to perform point-in-time recovery up to 15652 sequence and we were all good.

Hope That Helps
Prashant Dixit

Posted in Uncategorized | Tagged: | Leave a Comment »

RMAN-06900/RMAN-06901 and ORA-19921: maximum number of 64 rows exceeded

Posted by FatDBA on November 22, 2013

Received one of the odd and obscure RMAN/ORA errors of its kind While trying to connect with RMAN in Red Hat Enterprise Linux 5 and have wasted almost several minutes of my precious time, until i realized that there is one old RMAN session alive and was connected from the another terminal and was one day old before getting this error.

Okay so here is the case, while trying to connect with the target database in RMAN we have started recieving several RMAN and ORA errors …

[oracle@prashant1 ~]$ rman target /
Recovery Manager: Release 10.2.0.1.0 – Production on Fri Nov 22 21:30:38 2013
Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: QRCL (DBID=859215136)
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 64 rows exceeded

RMAN> host;

Let’s explore more aboput the ora error using my favorite utility OERR

[oracle@prashant1 ~]$ oerr ORA 19921
19921, 00000, “maximum number of %s rows exceeded”
// *Cause:  The maximum number of rows in the V$RMAN_STATUS or V$RMAN_OUTPUT
//          table has been exceeded.
// *Action: Close some of existing and unused RMAN connections and sessions.

Let’s check how many and how old the rman sessions exists in my database.

[oracle@prashant1 ~]$ ps -ef|grep rman
oracle   25970 25861  0 Nov21 pts/1    00:00:50 rman target /
oracle   29999 29863  0 21:30 pts/4    00:00:00 rman target /
oracle   30113 30088  0 21:33 pts/4    00:00:00 grep rman

[oracle@prashant1 ~]$ date
Fri Nov 22 21:34:05 IST 2013

Alright, so there is one of the session with OS ID – 25970 running from November 21 from Terminal (tty 1) –
Let’s kill the session and try to connect with the Recovery Manager once again.

[oracle@prashant1 ~]$ kill -9 25970

[oracle@prashant1 ~]$ ps -ef|grep rman
oracle   29999 29863  0 21:30 pts/4    00:00:00 rman target /
oracle   30115 30088  0 21:34 pts/4    00:00:00 grep rman

Okay, so it’s gone and no more under the process list.

[oracle@prashant1 ~]$ rman target /
Recovery Manager: Release 10.2.0.1.0 – Production on Fri Nov 22 21:34:15 2013
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
connected to target database: QRCL (DBID=859215136)
RMAN>

Okay and we have not getting list of those errors what we previously encountered and issue is F.I.X.E.D

Hope That Helps
Prashant Dixit

Posted in Advanced | Tagged: , | 1 Comment »

Standby/Dataguard:– RFS Process not working.

Posted by FatDBA on November 12, 2013

Not feeling well today!! ūüė¶ , but as¬† It’s been a long time since i wrote my last article on Oracle database issues … here i am back again with one of the problem that we faced in our production database of our erstwhile customer, some time ago.
It was a typical Physical Standby environment with Maximum Performance model set. This is a story happened on one fine day when we started receiving some issues after a small network outage which blocked the redo stream to transfer logs from Production to standby server. ** * The bad part was  Рthe issue had happened during peak hours where we had confined amount of time to rectify the case.

Issue: RFS Process not working
Problem Description:
The filesystem containing the archive destination in DR server was not accessible. As a result the log shipping got stopped. We deferred the log shipping in the production server. After the filesystem was back, we enabled the log shipping but the RFS process in DR server was not running…The problem occurred when the log 24717 was being shipped… When we queried

SQL> Select status, sequence# from v$managed_standby;

The status for MRP showed that it was waiting for gap…We then manually shipped the log file and then applied… When we enabled the shipping we found that the RFS process was still not started…..There were no error in the alert log of DR…We found a trace file in the production server with the following message..

tkcrrsarc: (WARN) Failed to find ARCH for message (message:0xa)
tkcrrpa: (WARN) Failed initial attempt to send ARCH message (message:0xa)

So we thought that the issue is with the archiver process….

Solution:
Check whether the archiver process is available for shipping log files. You can identify this by querying V$ARCHIVE_PROCESSES view.
SQL> Select * from v$archive_processes;

The output have the following columns:
Process: Indicates the process number.
Status: This should be ACTIVE
Log Sequence: Log sequence number of the log that is being shipped by the archiver. If it is not shipping any log then it should be 0.
State: This should be IDLE if the archiver is not shipping any log. If it is shipping any log then its state is BUSY.

In our case we had two archiver process running.

The status of both the arch process is ACTIVE.
The log sequence of First arch process is 0 and its state is IDLE. Hence it is healthy. However the log sequence of 2nd arch process is 24717 and its state is BUSY.

This was interesting because the problem occurred when the arch process was transferring the log 24717. This log was then manually shipped and applied. But the process still shows that it was shipping the 24717 log…
So we thought of increasing the arch processes. We increased the arch process from two to four.
SQL> alter system set log_archive_max_processes=4 scope=both;

We queried the v$archive_processes, the 3rd and 4th arch process was ready to ship the log files 24718 and 24719 logs respectively with their corresponding state as IDLE…
We enabled the log shipping and the RFS process in DR was started and the log shipping went smoothly.

However the 2nd arch process still was showing the same log sequence (24717) and state (BUSY)…. We then killed that archiver process…. And we were all good!

Hope That Helped
Prashant Dixit

Posted in Advanced | Tagged: , | 5 Comments »

PRVF-7531 – Oracle 11.2 Prerequisite Checks fail (MS Windows Error)

Posted by FatDBA on October 8, 2013

Today i got a call from one of my good friend who was trying to install Oracle 11g on Windows 7 platform (And was very furious about — Why oracle website don’t have version 10g available anymore, after the release of latest 12c … ūüôā … ) and was getting errors during pre checks during OUI calls.

Exact Error:
On  Step 7 of 10,

All Prerequisite Checks fail.

Physical Memory                        Failed
Available Physical Memory         Failed
Swap Size                                  Failed
Free Space                                 Failed
Architecture                                Failed
Environment variable PATH        Failed

Clicking on the more details link, shows:
– PRVF-7531 : Physical memory check cannot be performed on node “machine-name”
– Cause: Could not perform check of physical memory on the node indicated.
– Action: Ensure ability to access the node specified and view memory information.

 

image001

 

image002
With many errors related to Physical/Swap Size, initially i thought that i could be due to insufficient memory on the system, But he has 4GB of RAM available.
Next i asked if he is trying to install the software on any Admin controlled machine, and my guess was right – He was.

So, all of these errors left us bewildered and with very less details available. Ultimately, MOS came to rescue.
Found one of the Metalink note for his issue — Doc ID 1086063.1

Cause
The Workstation or Server services have not been started.

Solution
Make sure both the “WORKSTATION” and “SERVER” services are started.
Rerun the setup program.
Now all the Prerequisite Checks Succeed.

– The Workstation or Server services need to be started for Oracle to communicate with the machine, to make sure all the prerequisite checks are performed.
– It has been reported, the TCP/IP NetBIOS Helper service should also be started, to pass the Pre-Req checks.

Found SERVER service disabled on his machine and is the reason which is obstructing the pre checks to complete.

Posted in Advanced | Tagged: | 5 Comments »

ORA-00904: “POLTYP”: invalid identifier — During EXPORT

Posted by FatDBA on October 6, 2013

Error:
EXP-00008: ORACLE error 904 encountered
ORA-00904: “POLTYP”: invalid identifier

When:  During EXPORT operations.
This Scenario: During export of rman recovery catalog (Production).

$ exp rman/xxxxxx@rxxxxxx FILE=catalogexport.dmp OWNER=rman

Export: Release 11.2.0.3.0 – Production on Sun Oct 6 10:50:22 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 – 64bit Production
With the Partitioning, OLAP and Data Mining options
Export done in US7ASCII character set and AL16UTF16 NCHAR character set
server uses AL32UTF8 character set (possible charset conversion)
. exporting pre-schema procedural objects and actions
. exporting foreign function library names for user RMAN
. exporting PUBLIC type synonyms
. exporting private type synonyms
. exporting object type definitions for user RMAN
About to export RMAN’s objects …
. exporting database links
. exporting sequence numbers
. exporting cluster definitions
. about to export RMAN’s tables via Conventional Path …
EXP-00008: ORACLE error 904 encountered
ORA-00904: “POLTYP”: invalid identifier
EXP-00000: Export terminated unsuccessfully

This error occurs if you try the old export command from an 11g client against a database on version 10g or lower. The export command runs a query against a table called EXU9RLS in the SYS schema. On 11g this table was expanded with the column POLTYP and the export command (exp) expects to find this column. This should not be much of a problem since Data Pump export can be used.

SQL> desc EXU9RLS
Name                                                                                                  Null?    Type
—————————————————————————————————– ——– ——————————————————————–
OBJOWN                                                                                                NOT NULL VARCHAR2(30)
OBJNAM                                                                                                NOT NULL VARCHAR2(30)
POLGRP                                                                                                NOT NULL VARCHAR2(30)
POLICY                                                                                                NOT NULL VARCHAR2(30)
POLOWN                                                                                                NOT NULL VARCHAR2(30)
POLSCH                                                                                                         VARCHAR2(30)
POLFUN                                                                                                NOT NULL VARCHAR2(30)
STMT                                                                                                           VARCHAR2(34)
CHKOPT                                                                                                NOT NULL NUMBER
ENABLED                                                                                               NOT NULL NUMBER
SPOLICY                                                                                                        NUMBER
 POLTYP                                                                                                         VARCHAR2(33)

Resolution:
Since there was no code added in export to extract the RLS policy type, the fix for Bug 7568350 introduces a new column POLTYP in EXU9RLS view that is associated with the RLS policy type.

Fixed: Patch 7568350
Link:
https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?_afrLoop=15303088597334&patchId=7568350&_afrWindowMode=0&_adf.ctrl-state=x42u0k77b_206

Doc ID: Doc ID 784038.1

Posted in Advanced | Tagged: | Leave a Comment »

VMWare: Reason: “Cannot open the disk, Reason: Failed to lock the file”.

Posted by FatDBA on August 25, 2013

Today while starting one of my Test machine/instance which is a part of an VMWare (This Team includes a total of 3 virtual machines) has started giving an error message which reads:

“Cannot open the disk: ‘H:\Machine ………….’ or one of the snapshot disks it depends on.Reason: Failed to lock the file”.

errrrrrro
And this error restricts us to open the same machine and kept on throwing the same again and again.

Error:
After reading errors it seems to be related with the locks that machines acquire in VMWare Team environment in order to access resources.

Resolution:
While we start our virtual machines there are many different folders related to locks created which contains .lck (Lock Files) and are used by the machine during the operation i.e M30076.lck, Mx456.lck et.
These locks are dynamic in nature and auto refreshes after some time. So, i removed/renamed all of the lock files (.lck) existed under all of these folder and tried the machine again and as what i’ve expected, it started working back again.

* Note: Performing these steps on PROD systems could be dangerous and can cause serious performance/hang issues. Performing such steps always requires recommendations from the VMWARE support.

Thanks
Prashant Dixit
“Sharing is Good”

Posted in Advanced | Tagged: , , | Leave a Comment »

OPatch: ApplySession failed: Patch ID is null (OPatch failed with error code 73).

Posted by FatDBA on August 20, 2013

Going to discuss one of the most recent issue that I’ve faced while applying PSU patch for the month of July on one of our Pre-Prod Database. I’ve tried to reproduce the similar situation on one of my test machine.

We were trying to apply it (Patch) on our databse which is running on 11.2.0.1.0.  After sucessfully passed all prechecks of the patch we downloaded from Metalink

Prechecks:
1. Setting Env Variables (ORACLE_SID, ORACLE_HOME, PATH etc.)
2. Shutting down DB and Listener (No RAC Env)
3. Uncompressed zip file to the OPatch directory.

ORACLE_SID = [orcl] ? sairam
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle

[oracle@prashant ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Aug 20 22:00:32 2013
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> exit

[oracle@prashant ~]$ cd /u01/app/oracle/product/11.2.0/db_1/OPatch/
[oracle@prashant OPatch]$ cd 12419378/
[oracle@prashant 12419378]$ ls
custom  etc  files  patchmd.xml  README.html  README.txt

[oracle@prashant 12419378]$ opatch apply
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oraInventory
from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0/db_1/oui
Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2013-08-20_22-02-22PM.log

Patch history file: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt

ApplySession failed: Patch ID is null.
System intact, OPatch will not attempt to restore the system

OPatch failed with error code 73

* After Checking the current OPatch Version it Looks like we will need a higher version of OPatch utility to apply this patch.
[oracle@prashant 12419378]$ opatch version
Invoking OPatch 11.1.0.6.6
OPatch Version: 11.1.0.6.6

OPatch succeeded.

Hence downloaded latest version of Opatch from My Oracle Support (patch 6880880).
1. Downloaded Patch 6880880.
2. Took a backup of $ORACLE_HOME/OPatch into a dedicated backup location.
3. Removed the contents of $ORACLE_HOME/OPatch directory (Please do not
remove $ORACLE_HOME/OPatch directory itself).
4. Unzipped the OPatch downloaded zip into $ORACLE_HOME directory.

[oracle@prashant OPatch]$ ls
p6880880_112000_Linux-x86-64.zip
[oracle@prashant OPatch]$ unzip p6880880_112000_Linux-x86-64.zip
Archive:  p6880880_112000_Linux-x86-64.zip
creating: OPatch/
creating: OPatch/oplan/
……….

……….

[oracle@prashant OPatch]$ ls -ltr
total 32408
-rw-rw—- 1 oracle oinstall¬†¬†¬†¬†¬†¬† 26 Jun 26 11:54 version.txt
-rwxrwxr-x 1 oracle oinstall     2991 Jun 26 11:54 README.txt
-rw-rw-rР1 oracle oinstall     2576 Jun 26 11:54 opatch.pl
-rw-rw-rР1 oracle oinstall       49 Jun 26 11:54 opatch.ini
-rwxrwxr-x 1 oracle oinstall     9352 Jun 26 11:54 opatchdiag.bat
-rwxrwxr-x 1 oracle oinstall    10056 Jun 26 11:54 opatchdiag
-rwxrwxr-x 1 oracle oinstall    19867 Jun 26 11:54 opatch.bat
-rwx–x— 1 oracle oinstall¬†¬†¬† 31493 Jun 26 11:54 opatch
-rw-rw-rР1 oracle oinstall    23695 Jun 26 11:54 emdpatch.pl
drwxrwxr-x 2 oracle oinstall     4096 Jun 26 11:54 docs
drwxrwxr-x 3 oracle oinstall     4096 Jun 26 11:54 oplan
drwxrwxr-x 3 oracle oinstall     4096 Jun 26 11:54 jlib
drwxrwxr-x 3 oracle oinstall     4096 Jun 26 11:54 crs
drwxrwxr-x 4 oracle oinstall     4096 Jun 26 11:54 opatchprereqs
drwxrwxr-x 5 oracle oinstall     4096 Jun 26 11:54 ocm
-rw-rw-rР1 oracle oinstall     2147 Aug 20 02:22 PatchSearch.xml
-rw-r–r– 1 oracle oinstall 32996451 Aug 20 22:07 p6880880_112000_Linux-x86-64.zip

And finally after deploying last patch, we have the latest OPatch version available.

[oracle@prashant OPatch]$ opatch version
OPatch Version: 11.2.0.3.5

OPatch succeeded.

Now, we can go and deploy our PSU Patch.

[oracle@prashant OPatch]$ cd 12419378/
[oracle@prashant 12419378]$ opatch apply

Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oraInventory
from           : /u01/app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.1.0
Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/12419

Applying interim patch ‘12419378’ to OH ‘/u01/app/oracle/product/11.2.0/db_1’
Verifying environment and performing prerequisite checks…
Patch 12419378: Optional component(s) missing : [ oracle.client, 11.2.0.1.0 ]
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = ‘/u01/app/oracle/product/11.2.0/db_1’)

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files…

Patching component oracle.rdbms.rsf, 11.2.0.1.0…

Patching component oracle.rdbms.dbscripts, 11.2.0.1.0…

Patching component oracle.rdbms, 11.2.0.1.0…

Patching component oracle.oraolap, 11.2.0.1.0…

Patching component oracle.rdbms.deconfig, 11.2.0.1.0…

Patching component oracle.javavm.server, 11.2.0.1.0…

Patching component oracle.precomp.common, 11.2.0.1.0…

Patching component oracle.network.rsf, 11.2.0.1.0…

Patching component oracle.network.listener, 11.2.0.1.0…

Patching component oracle.rdbms.dv.oc4j, 11.2.0.1.0…

Patching component oracle.sdo.locator, 11.2.0.1.0…

Patching component oracle.sysman.console.db, 11.2.0.1.0…

Patching component oracle.sysman.oms.core, 10.2.0.4.2…

Patching component oracle.rdbms.dv, 11.2.0.1.0…

Patching component oracle.xdk.rsf, 11.2.0.1.0…

Patching component oracle.ldap.rsf.ic, 11.2.0.1.0…

Patching component oracle.ldap.rsf, 11.2.0.1.0…

Patching component oracle.sysman.plugin.db.main.repository, 11.2.0.1.0…

Verifying the update…
Patch 12419378 successfully applied
Log file location: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/12419378_Aug_20_2013_22_38_06/apply2013-08-20_22-38-06PM_1.log

OPatch succeeded.

Let’s Check the status of patch applied.
[oracle@prashant OPatch]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oraInventory
from           : /u01/app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.1.0
Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2013-08-20_22-56-30PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2013-08-20_22-56-30PM.txt

——————————————————————————–
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :

Patch  12419378     : applied on Tue Aug 20 22:41:38 IST 2013
Unique Patch ID:  13710328
Created on 8 Jul 2011, 02:48:54 hrs PST8PDT
Bugs fixed:
9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
9093300, 8909984, 8755082, 8780372, 9952216, 8664189, 8769569, 7519406
9302343, 9471411, 8822531, 7705591, 8650719, 10205230, 9637033, 8883722
8639114, 8723477, 8729793, 8919682, 8856478, 9001453, 8733749, 8565708
8735201, 8684517, 8870559, 8773383, 8981059, 8812705, 9488887, 12534742
8813366, 12534743, 9242411, 12534745, 12534746, 12534747, 8822832
12534748, 8897784, 8760714, 12534749, 8775569, 8671349, 8898589, 9714832
8642202, 9011088, 9369797, 9170608, 9165206, 8834636, 8891037, 8431487
8570322, 8685253, 8872096, 8718952, 8799099, 12534750, 9032717, 9399090
12534751, 12534752, 9713537, 9546223, 12534753, 12534754, 8588519
8783738, 12534755, 12534756, 8834425, 9454385, 8856497, 8890026, 8721315
10248516, 8818175, 8674263, 10249532, 9145541, 8720447, 9272086, 9467635
9010222, 9102860, 9197917, 8991997, 8661168, 8803762, 12419378, 8769239
9654983, 8706590, 8546356, 10408903, 8778277, 9058865, 8815639, 11724991
9971778, 9971779, 9027691, 9454036, 9454037, 9454038, 8761974, 9255542
9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668, 8891929
8798317, 9971780, 8782959, 8774868, 8820324, 8544696, 8702535, 9952260
9406607, 8268775, 9036013, 9363145, 8933870, 8405205, 9467727, 8822365
9676419, 11724930, 8761260, 8790767, 8795418, 8913269, 8717461, 8861700
9531984, 8607693, 8780281, 8330783, 8784929, 8780711, 9341448, 9015983
10323077, 8828328, 9119194, 10323079, 8832205, 8717031, 8665189, 9482399
9676420, 9399991, 8821286, 8633358, 9321701, 9655013, 9231605, 8796511
9167285, 8782971, 8756598, 8703064, 9390484, 9066116, 9007102, 9461782
10323080, 10323081, 10323082, 8753903, 8505803, 9382101, 9352237, 9216806
8918433, 11794163, 9057443, 8790561, 11794164, 8733225, 8795792, 11794165
11794167, 9067282, 8928276, 8837736, 9210925

——————————————————————————–

OPatch succeeded.

 

MOSC Note: 1066937.1

Thanks
Prashant Dixit
“Sharing is Good”

Posted in Advanced | Tagged: , | Leave a Comment »

EMCTL Error (OC4J Configuration Issue)

Posted by FatDBA on April 12, 2013

Noticed that some of my test databases monitored in OEM Grid Control started showing status of ‘Agent Unreachable’. This is what we are getting when we tried checking status of the Agent on the servers …

[oracle@prashant ~]$ emctl status dbconsole
OC4J Configuration issue. /u01/app/oracle/product/11.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_prashant_orcl not found.

Below provided steps are sequence of steps that we should adopt in such scenario and will help us to fix the issue most of the times.

  • Set the SID
  • Drop/Deconfig current EM repository.
  • Recreate EM repository for the same DB.
  • Re-check the Status using EMCTL commands for both Dbconsole & Agent

1. Set the SID
[oracle@prashant ~]$ echo $ORACLE_SID
orcl

2. Drop/Deconfig current EM repository.
This step will require you to feed deconfig process following details.
SID, Listener Port Number, SYS and SYSMAN Password.

[oracle@prashant ~]$ emca -deconfig dbcontrol db -repos drop;

STARTED EMCA at Apr 12, 2013 10:40:14 PM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database SID: orcl
Listener port number: 1521
Password for SYS user:
Password for SYSMAN user:

Do you wish to continue? [yes(Y)/no(N)]: Y

Apr 12, 2013 10:41:14 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/REDANT/emca_2013_04_12_22_40_14.log.
Apr 12, 2013 10:41:14 PM oracle.sysman.emcp.EMDBPreConfig performDeconfiguration
WARNING: EM is not configured for this database. No EM-specific actions can be performed.
Apr 12, 2013 10:41:14 PM oracle.sysman.emcp.ParamsManager checkListenerStatusForDBControl
WARNING: Error initializing SQL connection. SQL operations cannot be performed
Apr 12, 2013 10:41:14 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) …

Apr 12, 2013 10:45:42 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 12, 2013 10:45:43 PM

3. Recreate EM repository for the same DB.
This step will require you to feed config process following details.
SID, Listener Port Number, Email address (Optional), SMTP server details (Optional) along with SYS and DBSNMP and SYSMAN Password.

[oracle@prashant ~]$ emca -config dbcontrol db -repos create;

STARTED EMCA at Apr 12, 2013 10:46:17 PM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database SID: orcl
Listener port number: 1521
Listener ORACLE_HOME [ /u01/app/oracle/product/11.2.0/db_1 ]:
Password for SYS user:
Password for DBSNMP user:
Password for SYSMAN user:
Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
—————————————————————–

You have specified the following settings

Database ORACLE_HOME ……………. /u01/app/oracle/product/11.2.0/db_1

Local hostname ……………. prashant
Listener ORACLE_HOME ……………. /u01/app/oracle/product/11.2.0/db_1
Listener port number ……………. 1521
Database SID ……………. orcl
Email address for notifications ……………
Outgoing Mail (SMTP) server for notifications ……………

—————————————————————–
Do you wish to continue? [yes(Y)/no(N)]:

Apr 12, 2013 10:46:59 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/REDANT/emca_2013_04_12_22_46_16.log.
Apr 12, 2013 10:47:00 PM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) …

Apr 12, 2013 10:54:15 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully created
Apr 12, 2013 10:54:28 PM oracle.sysman.emcp.EMReposConfig uploadConfigDataToRepository
INFO: Uploading configuration data to EM repository (this may take a while) …
Apr 12, 2013 10:55:43 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Uploaded configuration data successfully
Apr 12, 2013 10:55:47 PM oracle.sysman.emcp.util.DBControlUtil configureSoftwareLib
INFO: Software library configured successfully.
Apr 12, 2013 10:55:47 PM oracle.sysman.emcp.EMDBPostConfig configureSoftwareLibrary
INFO: Deploying Provisioning archives …
Apr 12, 2013 10:56:06 PM oracle.sysman.emcp.EMDBPostConfig configureSoftwareLibrary
INFO: Provisioning archives deployed successfully.
Apr 12, 2013 10:56:06 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Securing Database Control (this may take a while) …
Apr 12, 2013 10:56:40 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Database Control secured successfully.
Apr 12, 2013 10:56:40 PM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) …

Apr 12, 2013 10:57:23 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
Apr 12, 2013 10:57:23 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is https://prashant:5500/em <<<<<<<<<<<
Apr 12, 2013 10:57:32 PM oracle.sysman.emcp.EMDBPostConfig invoke
WARNING:
************************  WARNING  ************************

Management Repository has been placed in secure mode wherein Enterprise Manager data will be encrypted.  The encryption key has been placed in the file: /u01/app/oracle/product/11.2.0/db_1/prashant_REDANT/sysman/config/emkey.ora.   Please ensure this file is backed up as the encrypted data will become unusable if this file is lost.

***********************************************************
Enterprise Manager configuration completed successfully

4. Re-check the Status using EMCTL commands for both Dbconsole & Agent.

[oracle@prashant ~]$ emctl status dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.1.0
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
https://prashant:5500/em/console/aboutApplication
Oracle Enterprise Manager 11g is running.
——————————————————————
Logs are generated in directory /u01/app/oracle/product/11.2.0/db_1/prashant_REDANT/sysman/log

[oracle@prashant ~]$ emctl status agent
Oracle Enterprise Manager 11g Database Control Release 11.2.0.1.0
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 10.2.0.4.2
OMS Version       : 10.2.0.4.2
Protocol Version  : 10.2.0.4.2
Agent Home        : /u01/app/oracle/product/11.2.0/db_1/prashant_REDANT
Agent binaries    : /u01/app/oracle/product/11.2.0/db_1
Agent Process ID  : 26794
Parent Process ID : 26757
Agent URL         : https://prashant:3938/emd/main
Repository URL    : https://prashant:5500/em/upload/
Started at        : 2013-04-12 22:56:42
Started by user   : oracle
Last Reload       : 2013-04-12 22:56:42
Last successful upload                       : 2013-04-12 22:58:34
Total Megabytes of XML files uploaded so far :    10.04
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    26.87%
Data channel upload directory                : /u01/app/oracle/product/11.2.0/db_1/prashant_REDANT/sysman/recv
Last successful heartbeat to OMS             : 2013-04-12 22:58:24
—————————————————————
Agent is Running and Ready

 

OEM Final screen is up!

Untitled

Posted in Advanced | Tagged: , | Leave a Comment »

Recovery Error: ORA-01207: file is more recent than control file ‚Äď old control file

Posted by FatDBA on January 21, 2013

Solution steps when you started receiving ORA-01207.

Reason: The control file change sequence number in the data file is greater than the number in the control file. This implies that the wrong control file is being used. Note that repeatedly causing this error can make it stop happening without correcting the real problem. Every attempt to open the database will advance the control file change sequence number until it is great enough.

Scenario: In my scenario one of our SIT box lost all the copies of multiplexed ControlFiles from the system and after couple of minutes Database got crashed. While trying fixing the problem we discovered that we did’nt have the backup copies of Control Files Available, but we had a mount point backup available which holds all backup information but due to issues with our SMARTS (Monitoring Tool) backup script got failed to update the control file backup. So we didn’t knew that the ControlFile backup available was too old and have a very old sequence number registered.

So, after restoring the controlfile from that outdated backup we started recieving this error message during Databse OPEN:

SQL> alter database open;

ORA-00283: recovery session canceled due to errors
ORA-01122: database file 1 failed verification check
ORA-01110: data file 1: ‚Äė/app/oracle/oradata/rac10g/system01.dbf‚Äô
ORA-01207: file is more recent than control file ‚Äď old control file

Solution: recreate the control file.

1) dump controlfile to trace
SQL> alter database backup controlfile to trace as ‚Äė/tmp/ccc.sql‚Äô;

2) startup database nomount
SQL> startup nomount

3) create the control file
SQL> @/tmp/ccc.sql
Control file created.

sys@SIT4435Z89> recover database using backup controlfile;

SQL> recover database using backup controlfile ;
ORA-00279: change 1904510 generated at 12/11/2012 15:46:23 needed for thread 1
ORA-00289: suggestion : +FRA
ORA-00280: change 1904510 for thread 1 is in sequence #62

Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+DATA/orcl/onlinelog/group_1.261.801074115
ORA-00310: archived log contains sequence 61; sequence 62 required
ORA-00334: archived log: ‘+DATA/orcl/onlinelog/group_1.261.801074115’

Now it is asking for Log sequence 62 which is not available under archive dest
ASMCMD> ls
thread_1_seq_52.314.801739289
thread_1_seq_53.294.801743225
thread_1_seq_54.295.801743259
thread_1_seq_55.285.801743403
thread_1_seq_56.283.801757235
thread_1_seq_57.282.801758303
thread_1_seq_58.281.801759635
thread_1_seq_59.286.801761231
thread_1_seq_60.305.801762327
thread_1_seq_61.296.801762385

We only have seq:61 available.

Hint: It could be still inside REDO Log groups and is not archived before the crash or Control file loss.
Hence we’ll check both of the two avialable redo groups to find seq:62
+DATA/orcl/onlinelog/group_1.261.801074115
+DATA/orcl/onlinelog/group_2.262.801074117

SQL> startup  mount
ORACLE instance started.

Total System Global Area 1636814848 bytes
Fixed Size                  2213856 bytes
Variable Size            1140852768 bytes
Database Buffers          486539264 bytes
Redo Buffers                7208960 bytes
Database mounted.
SQL> recover database using backup controlfile ;
ORA-00279: change 1904510 generated at 12/11/2012 15:46:23 needed for thread 1
ORA-00289: suggestion : +FRA
ORA-00280: change 1904510 for thread 1 is in sequence #62

Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+DATA/orcl/onlinelog/group_1.261.801074115
ORA-00310: archived log contains sequence 61; sequence 62 required
ORA-00334: archived log: ‘+DATA/orcl/onlinelog/group_1.261.801074115’

SQL> recover database using backup controlfile ;
ORA-00279: change 1904510 generated at 12/11/2012 15:46:23 needed for thread 1
ORA-00289: suggestion : +FRA
ORA-00280: change 1904510 for thread 1 is in sequence #62

Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+DATA/orcl/onlinelog/group_2.262.801074117
Log applied.
Media recovery complete.

Shows it was available under redo group: 2 (could be Current or Active) and it is successfully applied.

Finally we’d open DB in reset log mode.

SQL> alter database open resetlogs ;
Database altered.

Posted in Advanced | Tagged: , | Leave a Comment »

KEEP Clause * Error: ORA-19811: cannot have files in DB_RECOVERY_FILE_DEST with keep attributes*

Posted by FatDBA on January 13, 2013

KEEP option with RMAN:
If you have any backup criterion set and want to override settings then there is ‘KEEP FOREVER | UNTIL TIME’ clause from recovery manager which will help you to achieve required. There are certain conditions that’s needs to be fulfilled before using FOREVER clause.

Conditions:
1. You should be connected with the CATALOG server  (If using KEEP FOREVER Clause. No need if using KEEP UNTIL TIME Clause).
2. Oracle will not allow you to save backup pieces/set’s inside the FRA. You have to change the backup directory.

Let’s take an example:

[oracle@localhost ~]$ rman target / catalog rman/oracle90@orcl;
RMAN> backup current controlfile keep forever;

Starting backup at 13-JAN-13
current log archived

using channel ORA_DISK_1
backup will never be obsolete
archived logs required to recover from this backup will be backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 13-JAN-13
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 01/13/2013 18:35:23
ORA-19811: cannot have files in DB_RECOVERY_FILE_DEST with keep attributes

Solution:
RMAN> backup current controlfile keep forever format ‘/u01/%U’;

Starting backup at 13-JAN-13
current log archived

using channel ORA_DISK_1
backup will never be obsolete
archived logs required to recover from this backup will be backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 13-JAN-13
channel ORA_DISK_1: finished piece 1 at 13-JAN-13
piece handle=/u01/20nvb5gk_1_1 tag=TAG20130113T183811 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01

using channel ORA_DISK_1
backup will never be obsolete
archived logs required to recover from this backup will be backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 13-JAN-13
channel ORA_DISK_1: finished piece 1 at 13-JAN-13
piece handle=/u01/21nvb5gm_1_1 tag=TAG20130113T183811 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01

current log archived
using channel ORA_DISK_1
backup will never be obsolete
archived logs required to recover from this backup will be backed up
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=66 RECID=39 STAMP=804623897
channel ORA_DISK_1: starting piece 1 at 13-JAN-13
channel ORA_DISK_1: finished piece 1 at 13-JAN-13
piece handle=/u01/22nvb5gq_1_1 tag=TAG20130113T183811 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01

using channel ORA_DISK_1
backup will never be obsolete
archived logs required to recover from this backup will be backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 13-JAN-13
channel ORA_DISK_1: finished piece 1 at 13-JAN-13
piece handle=/u01/23nvb5gr_1_1 tag=TAG20130113T183811 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 13-JAN-13

Posted in Advanced | Tagged: , , | Leave a Comment »

 
%d bloggers like this: