Tales From A Lazy Fat DBA

Its all about Databases & their performance, troubleshooting & much more …. ¯\_(ツ)_/¯

«
»

Contol File Missing – Recovery (ORA-00205: error in identifying control file)

Posted by FatDBA on September 24, 2012

Today i intentionaly deleted one of three available Control File of one of the instance ‘orcl’. Below are the steps i performed to recover the lost DB Controlfile to bring back the Database up.

This is what i recieved in SQL command line when tried to start the effected the instance.
SQL> startup
ORACLE instance started.

Total System Global Area  306184192 bytes
Fixed Size                  1219112 bytes
Variable Size             109053400 bytes
Database Buffers          192937984 bytes
Redo Buffers                2973696 bytes
ORA-00205: error in identifying control file, check alert log for more info

Pre – Checks:
* First check if the entry is available in PFILE and check status of control_files paramter.
* Veify the Control Files entries in PFILE and supply changes to the spfile if discrepancies found by using create spfile/pfile statement.

Example Entry in an PFILE (INIT.ORA)
*.control_files=’/u01/app/oracle/oradata/orcl/control01.ctl’,’/u01/app/oracle/oradata/orcl/control02.ctl’,’/u01/app/oracle/oradata/orcl/control03.ctl’

SQL> show parameter control_files

NAME                                 TYPE        VALUE
———————————— ———– ——————————
control_file_record_keep_time        integer     7
control_files                        string      /u01/app/oracle/oradata/orcl/c
ontrol01.ctl, /u01/app/oracle/
oradata/orcl/control02.ctl, /u
01/app/oracle/oradata/orcl/con
trol03.ctl

* If you have ASM setup on the same machine, it is recommended that you should always check ASM state. It should be always running without any errors during startup (i.e. DISK Groups errors etc.) otherwise would cause issues while restoring/recovering files that are created using ASM signatures.

Scenario 1: Corrupt or Missing control file

When attempting to start the database you see the following error.

1
2
3
4
5
6
7
8
9
10
11
12
SQL> startup
ORACLE instance started.
Total System Global Area  849530880 bytes
Fixed Size                  1339824 bytes
Variable Size             528485968 bytes
Database Buffers          314572800 bytes
Redo Buffers                5132288 bytes
ORA-00205: error in identifying control file, check alert log for more info
SQL>

Looking the alert log you see that the /u02/app/oracle/oradata/orcl/ctl/control01.ctl is corrupt.

1
2
3
4
5
6
7
8
ALTER DATABASE   MOUNT
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/u02/app/oracle/oradata/orcl/ctl/control01.ctl'
ORA-27048: skgfifi: file header information is invalid
ORA-205 signalled during: ALTER DATABASE   MOUNT...
Mon Apr 26 10:11:23 2010
Checker run found 1 new persistent data failures
ORA-205 signalled during: ALTER DATABASE   MOUNT...

Or you may see the file is missing when looking in the alert log.

1
2
3
4
5
6
7
8
9
ALTER DATABASE   MOUNT
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/u02/app/oracle/oradata/orcl/ctl/control01.ctl'
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
Mon Apr 26 10:28:28 2010
Checker run found 1 new persistent data failures
ORA-205 signalled during: ALTER DATABASE   MOUNT...

In either case, to recover from this problem is simple and does not require going to the last backup provided your control files are multiplexed. To resolve this problem we simply copy a control file from one of the other locations.

Identify the control files:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
SQL> startup
ORACLE instance started.
Total System Global Area  849530880 bytes
Fixed Size                  1339824 bytes
Variable Size             528485968 bytes
Database Buffers          314572800 bytes
Redo Buffers                5132288 bytes
ORA-00205: error in identifying control file, check alert log for more info
SQL> show parameter control_files
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
control_files                        string      /u02/app/oracle/oradata/orcl/c
                                                 tl/control01.ctl, /u01/app/ora
                                                 cle/flash_recovery_area/orcl/c
                                                 ontrol02.ctl, /u03/app/oracle/
                                                 oradata/orcl/ctl/control03.ctl
SQL>

We know from the alert log that /u02/app/oracle/oradata/orcl/ctl/control01.ctl is the control file that is having problems so we will copy the one from the /u01/app/oracle/flash_recovery_area/orcl/control02.ctl and then mount and open the database.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
SQL> host
[oracle@ora1 ~]$ cp /u01/app/oracle/flash_recovery_area/orcl/control02.ctl /u02/app/oracle/oradata/orcl/ctl/control01.ctl
[oracle@ora1 ~]$ exit
exit
SQL> alter database mount;
Database altered.
SQL> alter database open;
Database altered.
SQL>

By having a multiplexed copy of the control file we were able to recover from a corrupt or missing control file without having to go to a database backup.

Scenario 2: All control files are corrupt or missing

When starting the database we see the following error.

1
2
3
4
5
6
7
8
9
10
11
12
SQL> startup
ORACLE instance started.
Total System Global Area  849530880 bytes
Fixed Size                  1339824 bytes
Variable Size             528485968 bytes
Database Buffers          314572800 bytes
Redo Buffers                5132288 bytes
ORA-00205: error in identifying control file, check alert log for more info
SQL>

In anticipation of restoring a control we get a list of the control file locations.

1
2
3
4
5
6
7
8
9
10
SQL> show parameter control_files
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
control_files                        string      /u02/app/oracle/oradata/orcl/c
                                                 tl/control01.ctl, /u01/app/ora
                                                 cle/flash_recovery_area/orcl/c
                                                 ontrol02.ctl, /u03/app/oracle/
                                                 oradata/orcl/ctl/control03.ctl
SQL>

Looking at the alert log we see that none of the control files are available.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
ALTER DATABASE   MOUNT
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/u03/app/oracle/oradata/orcl/ctl/control03.ctl'
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/u01/app/oracle/flash_recovery_area/orcl/control02.ctl'
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/u02/app/oracle/oradata/orcl/ctl/control01.ctl'
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
ORA-205 signalled during: ALTER DATABASE   MOUNT...
Mon Apr 26 11:07:53 2010
Checker run found 2 new persistent data failures

Since all of the control files are missing we will need to restore them from a backup using RMAN.

1
2
3
4
5
6
7
8
9
10
11
[oracle@ora1 ctl]$ rman
Recovery Manager: Release 11.2.0.1.0 - Production on Mon Apr 26 11:24:32 2010
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
RMAN> connect target /
connected to target database: ORCL (not mounted)
RMAN>

Currently none of the control files are mounted so RMAN does not know about the backups or any pre-configured RMAN settings. All settings are at their default.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
RMAN> show all;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
RMAN>

In order to use the backups we will need to tell RMAN the Database ID. If you do not have or know the database you have two options available.

Option 1: Shutdown the database and re-start it in RMAN

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
SQL> shutdown immediate
ORA-01507: database not mounted
ORACLE instance shut down.
SQL> host
[oracle@ora1 ctl]$ rman
Recovery Manager: Release 11.2.0.1.0 - Production on Mon Apr 26 11:30:49 2010
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
RMAN> connect target /
connected to target database (not started)
RMAN> startup
Oracle instance started
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of startup command at 04/26/2010 11:31:00
ORA-00205: error in identifying control file, check alert log for more info

You will again receive an error stating that the control files could not be found. That is OK because in the next step we are going to restore them. Because the RMAN configuration is back to the default we are going to have to pass the entire path along with the file name of the latest control file autobackup to the restore control file command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
RMAN> restore controlfile from '/u03/app/oracle/oradata/orcl/backup/c-1239150297-20100426-00';
Starting restore at 26-APR-10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=20 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
output file name=/u02/app/oracle/oradata/orcl/ctl/control01.ctl
output file name=/u01/app/oracle/flash_recovery_area/orcl/control02.ctl
output file name=/u03/app/oracle/oradata/orcl/ctl/control03.ctl
Finished restore at 26-APR-10
RMAN>

After restoring the control files we can see mount the database.

1
2
3
4
5
6
RMAN> alter database mount;
database mounted
released channel: ORA_DISK_1
RMAN>

Now that the database has been mounted we can see that our custom RMAN settings are back.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
RMAN> show all;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1;
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/u03/app/oracle/oradata/orcl/backup/%F';
CONFIGURE DEVICE TYPE DISK PARALLELISM 2 BACKUP TYPE TO BACKUPSET;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT   '/u03/app/oracle/oradata/orcl/backup/%U' MAXPIECESIZE 2 G;
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE';
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/11.2.0/dbhome_1/dbs/snapcf_orcl.f'; # default
RMAN>

Now it would appear we are ready to open the database.

1
2
3
4
5
6
7
8
9
RMAN> alter database open;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of alter db command at 04/26/2010 11:38:40
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
RMAN>

We just restored the control files. In order to open the database we have to use the RESETLOGS option.

1
2
3
4
5
6
7
8
9
10
RMAN> alter database open resetlogs;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of alter db command at 04/26/2010 11:39:08
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '/u02/app/oracle/oradata/orcl/system01.dbf'
RMAN>

The restored control files are older than the data files. Hence in this case we need to restore the database.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
RMAN> restore database;
Starting restore at 26-APR-10
using channel ORA_DISK_1
using channel ORA_DISK_2
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00003 to /u02/app/oracle/oradata/orcl/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00005 to /u02/app/oracle/oradata/orcl/example01.dbf
channel ORA_DISK_1: restoring datafile 00006 to /u02/app/oracle/oradata/orcl/test.dbf
channel ORA_DISK_1: reading from backup piece /u03/app/oracle/oradata/orcl/backup/1olc324t_1_1
channel ORA_DISK_2: starting datafile backup set restore
channel ORA_DISK_2: specifying datafile(s) to restore from backup set
channel ORA_DISK_2: restoring datafile 00004 to /u02/app/oracle/oradata/orcl/users01.dbf
channel ORA_DISK_2: reading from backup piece /u03/app/oracle/oradata/orcl/backup/1plc326b_1_1
channel ORA_DISK_2: piece handle=/u03/app/oracle/oradata/orcl/backup/1plc326b_1_1 tag=TAG20100426T092133
channel ORA_DISK_2: restored backup piece 1
channel ORA_DISK_2: restore complete, elapsed time: 00:00:15
channel ORA_DISK_2: starting datafile backup set restore
channel ORA_DISK_2: specifying datafile(s) to restore from backup set
channel ORA_DISK_2: restoring datafile 00001 to /u02/app/oracle/oradata/orcl/system01.dbf
channel ORA_DISK_2: restoring datafile 00002 to /u02/app/oracle/oradata/orcl/sysaux01.dbf
channel ORA_DISK_2: restoring datafile 00007 to /u03/app/oracle/oradata/orcl/dbfs01.dbf
channel ORA_DISK_2: reading from backup piece /u03/app/oracle/oradata/orcl/backup/1nlc324t_1_1
channel ORA_DISK_1: piece handle=/u03/app/oracle/oradata/orcl/backup/1olc324t_1_1 tag=TAG20100426T092133
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:34
channel ORA_DISK_2: piece handle=/u03/app/oracle/oradata/orcl/backup/1nlc324t_1_1 tag=TAG20100426T092133
channel ORA_DISK_2: restored backup piece 1
channel ORA_DISK_2: restore complete, elapsed time: 00:02:09
Finished restore at 26-APR-10
RMAN>

Next we recover the database.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
RMAN> recover database;
Starting recover at 26-APR-10
using channel ORA_DISK_1
using channel ORA_DISK_2
starting media recovery
archived log for thread 1 with sequence 13 is already on disk as file /u03/app/oracle/oradata/orcl/redo/red01_b.log
archived log for thread 1 with sequence 14 is already on disk as file /u02/app/oracle/oradata/orcl/redo/redo02.log
archived log file name=/u03/app/oracle/oradata/orcl/redo/red01_b.log thread=1 sequence=13
archived log file name=/u02/app/oracle/oradata/orcl/redo/redo02.log thread=1 sequence=14
media recovery complete, elapsed time: 00:00:02
Finished recover at 26-APR-10
RMAN>

Now that the database has been restored and recovered we can finally open the database, again with the RESETLOGS option.

1
2
3
4
5
RMAN> alter database open resetlogs;
database opened
RMAN>

I think after seeing what is required to restore the control files you will agree that multiplexing the control files is a good thing to do.

* Thanks Dilip Kumar Singh for highlighting the errors in the post and helped me to rectify 🙂

This entry was posted on September 24, 2012 at 1:42 PM and is filed under Advanced. Tagged: . You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a comment

«
»