Tales From A Lazy Fat DBA

Fan of Oracle DB & Performance, PostgreSQL & Cassandra … \,,/

  • Likes

    • 236,075
  • Archives

  • Categories

  • Subscribe

Posts Tagged ‘linux’

File is DELETED but didn’t reclaim space in filesystem – LINUX

Posted by FatDBA on March 9, 2015

There are times when even after you deleted a file in Linux but that didn’t reclaim space in filesystem.
Below command shows that there are few of the files which are deleted from the system but are still active and still taking the space on the disk.

[oracle@dixitdb053 /proc]# /usr/sbin/lsof |grep -i deleted
oracle 11441 oracle 19w REG 253,2 1810 3129788 /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_diag_11441.trc (deleted)
oracle 11441 oracle 20w REG 253,2 126 3129795 /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_diag_11441.trm (deleted)
oracle 30157 oracle 19w REG 253,2 14182 3129684 /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_vkrm_30157.trc (deleted)
dd 32592 oracle 1w REG 253,2 24238593024 3129587 /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_ora_31778.trm (deleted)

Here the file is available under mount point — /opt/oracle and is still 86% full.

[oracle@dixitdb053 /proc/11441/fd]# df -hk
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
31614888 6127104 23856156 21% /
/dev/mapper/VolGroup02-LogVol05
36124288 29165480 5123800 86% /opt/oracle
/dev/sda1 101086 15711 80156 17% /boot
tmpfs 1956756 0 1956756 0% /dev/shm
113.11.88.199:/vol/dixitdb053_BO
110100480 58354656 51745824 54% /db01
113.11.88.199:/vol/vol_dixitdb053_backup
104857600 55011648 49845952 53% /backup

Well, the space will be automatically reclaimed depending on the size of the file deleted.
Below are the steps in order to remove it instantly from the file system


Steps:
=============

1. Go to /proc directory which contains PID of all active processes and files.
2. login to respective PID

[oracle@dixitdb053 /proc/11441/fd]# cd /proc/32592
[oracle@dixitdb053 /proc/32592]# cd fd
[oracle@dixitdb053 /proc/32592/fd]# pwd
/proc/32592/fd

3. Check if there is a link and in the end it says deleted.

[oracle@dixitdb053 /proc/32592/fd]# ls -ltrh
total 0
lrwx—— 1 oracle oinstall 64 Mar 7 07:57 2 -> /dev/pts/1
l-wx—— 1 oracle oinstall 64 Mar 7 08:25 1 -> /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_ora_31778.trm (deleted)
lr-x—— 1 oracle oinstall 64 Mar 7 08:25 0 -> /dev/zero

4. Type > in the number that was shown in that line and it will release the space instantly.

[oracle@dixitdb053 /proc/32592/fd]# > 1

[oracle@dixitdb053 /proc/32592/fd]# ls -ltrh
total 0
lrwx—— 1 oracle oinstall 64 Mar 7 07:57 2 -> /dev/pts/1
l-wx—— 1 oracle oinstall 64 Mar 7 08:25 1 -> /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_ora_31778.trm (deleted)
lr-x—— 1 oracle oinstall 64 Mar 7 08:25 0 -> /dev/zero

5. verify it once again by listing the open files in the filesystem to see if there is still an open file with the deleted status.

Now the space is reclaimed and has been brought down to 0.

[oracle@dixitdb053 /proc/32592/fd]# /usr/sbin/lsof |grep -i deleted
oracle 11441 oracle 19w REG 253,2 0 3129788 /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_diag_11441.trc (deleted)
oracle 11441 oracle 20w REG 253,2 0 3129795 /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_diag_11441.trm (deleted)
oracle 30157 oracle 19w REG 253,2 14792 3129684 /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_vkrm_30157.trc (deleted)
dd 32592 oracle 1w REG 253,2 0 3129587 /opt/oracle/diag/rdbms/dixitdb/dixitdb/trace/dixitdb_ora_31778.trm (deleted)

And the mount point /opt/oracle usage went down too and reached to 17%.

[oracle@dixitdb053 /proc/32592/fd]# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
31G 5.9G 23G 21% /
/dev/mapper/VolGroup02-LogVol05
35G 5.3G 28G 17% /opt/oracle
/dev/sda1 99M 16M 79M 17% /boot
tmpfs 1.9G 0 1.9G 0% /dev/shm
113.11.88.199:/vol/dixitdb053_BO
105G 56G 50G 54% /db01
113.11.88.199:/vol/vol_dixitdb053_backup
100G 53G 48G 53% /backup

Thanks
Prashant Dixit

Posted in Advanced | Tagged: , | Leave a Comment »

Large Pages & use_large_pages parameter in Oracle — BMC Patrol Alerts global parameter ”MEMUsedMemPerc” triggered on ”MEMORY.MEMORY”

Posted by FatDBA on December 31, 2014

All of sudden one morning we started constant alerts from one of our production system used for SAP and other  BI/BO applications which reads about Memory Used Percentage threshold breached. Being one of the very rarely used and least bust production system out of all, this was never expected from the server. I have previously written about the benefits and usage of large pages in Linux.

Summary: prashantdb053 global parameter ”MEMUsedMemPerc” triggered on ”MEMORY.MEMORY”.  95 <= 95.03 <= 100
Notes: prashantdb053 global parameter ”MEMUsedMemPerc” triggered on ”MEMORY.MEMORY”.  95 <= 95.03 <= 100 Object=MEMORY Object Class=MEMORY Parameter=MEMUsedMemPerc
BMC Impact Manager Initiated Incident

The database I have running doesn’t use large pages, as shown in the alert.log:
****************** Large Pages Information *****************
Total Shared Global Region in Large Pages = 0 KB (0%)
Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB) (alloc incr 16 MB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB

RECOMMENDATION:
Total Shared Global Region size is 2514 MB. For optimal performance,
prior to the next instance restart increase the number
of unused Large Pages by atleast 1257 2048 KB Large Pages (2514 MB)
system wide to get 100% of the Shared
Global Region allocated with Large pages
***********************************************************

Load statistics on the server at the time when we got the error message.

top – 07:26:24 up 30 days, 34 min,  3 users,  load average: 0.07, 0.05, 0.01
Tasks: 183 total,   1 running, 182 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3913512k total,  3886112k used,    27400k free,   149432k buffers
Swap:  4947960k total,   284392k used,  4663568k free,  3036064k cached

Memory Statistics on the server.

[oracle@prashantdb053 /backup/dixit/scripts]# free -m
total       used       free     shared    buffers     cached
Mem:          3821       3795         26          0        146       2965
-/+ buffers/cache:        684       3137
Swap:         4831        277       4554

[oracle@prashantdb053 /backup/dixit/scripts]# cat /proc/meminfo
MemTotal:      3913512 kB
MemFree:         26504 kB
Buffers:        149548 kB
Cached:        3036664 kB
SwapCached:       4620 kB
Active:        2250272 kB
Inactive:      1377312 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      3913512 kB
LowFree:         26504 kB
SwapTotal:     4947960 kB
SwapFree:      4663568 kB
Dirty:             132 kB
Writeback:           0 kB
AnonPages:      437440 kB
Mapped:        1668000 kB
Slab:            86592 kB
PageTables:     143008 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   6904716 kB
Committed_AS:  4125228 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    265480 kB
VmallocChunk: 34359472519 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Below stats shows that Large Page are not utlized or used.
[oracle@prashantdb053 /backup/dixit/scripts]# grep Huge /proc/meminfo
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

[oracle@prashantdb053 /backup/dixit/scripts]# vmstat -s
3913512  total memory
3884496  used memory
2248032  active memory
1376892  inactive memory
29016  free memory
149560  buffer memory
3036676  swap cache
4947960  total swap
284392  used swap
4663568  free swap
4399893 non-nice user cpu ticks
123135 nice user cpu ticks
1173193 system cpu ticks
497863320 idle cpu ticks
9827196 IO-wait cpu ticks
156731 IRQ cpu ticks
337956 softirq cpu ticks
0 stolen cpu ticks
14560315 pages paged in
286499680 pages paged out
1086328 pages swapped in
1030486 pages swapped out
2808993403 interrupts
2839798366 CPU context switches
1417413111 boot time
333424 forks

As you can see the parameter is static and requires an instance restart, so this is what I did next. Here is an interesting side effect of setting the parameter to “auto”: it doesn’t have an effect if you didn’t prepare the system for use of large pages in /etc/security/limits.conf.

07:33:48 SQL> select value,isdefault from V$PARAMETER_VALID_VALUES where name = ‘use_large_pages’;

VALUE                ISDEFAULT
——————– —————
TRUE                 TRUE
AUTO                 FALSE
ONLY                 FALSE
FALSE                FALSE

07:44:05 SQL> show parameter use_large_pages

NAME                                 TYPE        VALUE
———————————— ———– ——————————
use_large_pages                      string      TRUE

So let’s change that, but dynamically and not manually.
07:57:32 SQL> alter system set use_large_pages=auto scope=spfile;
System altered.

HugePagesTotal is still 0, which means system is still not using Large Pages, To allow oracle to lock memory you need to grant it the privilege. I had to edit /etc/security/limits.conf and set the memlock parameters.

[oracle@prashantdb053 /backup/dixit/scripts]# grep Huge /proc/meminfo
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Being a static parameter we need to bounce the database to make changes persistent.
07:58:00 SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

07:58:18 SQL> startup
ORACLE instance started.

07:58:54 SQL> show parameter use_large_pages

NAME                                 TYPE        VALUE
———————————— ———– ——————————
use_large_pages                      string      AUTO

Entries from alert log which shows that it has been changed to ‘AUTO’ values and started using large pages.

****************** Large Pages Information *****************
Parameter use_large_pages = AUTO
Total Shared Global Region in Large Pages = 2048 KB (0%)
Large Pages used by this instance: 1 (2048 KB)
Large Pages unused system wide = 3 (6144 KB) (alloc incr 16 MB)
Large Pages configured system wide = 4 (8192 KB)
Large Page size = 2048 KB
Time taken to allocate Large Pages = 0.033721 sec

RECOMMENDATION:
Total Shared Global Region size is 2514 MB. For optimal performance,
prior to the next instance restart increase the number
of unused Large Pages by atleast 1253 2048 KB Large Pages (2506 MB)
system wide to get 100% of the Shared
Global Region allocated with Large pages
***********************************************************

Finally after the change of parameter and hard coding limits system finally started using large pages and alerts coming from the Patrol Agents.

[oracle@prashantdb053 /opt/oracle/diag/rdbms/prashantdb/prashantdb/trace]# grep Huge /proc/meminfo
HugePages_Total:     4
HugePages_Free:      3
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Thanks
Prashant Dixit

Posted in Advanced | Tagged: , | Leave a Comment »

VMWare: Reason: “Cannot open the disk, Reason: Failed to lock the file”.

Posted by FatDBA on August 25, 2013

Today while starting one of my Test machine/instance which is a part of an VMWare (This Team includes a total of 3 virtual machines) has started giving an error message which reads:

“Cannot open the disk: ‘H:\Machine ………….’ or one of the snapshot disks it depends on.Reason: Failed to lock the file”.

errrrrrro
And this error restricts us to open the same machine and kept on throwing the same again and again.

Error:
After reading errors it seems to be related with the locks that machines acquire in VMWare Team environment in order to access resources.

Resolution:
While we start our virtual machines there are many different folders related to locks created which contains .lck (Lock Files) and are used by the machine during the operation i.e M30076.lck, Mx456.lck et.
These locks are dynamic in nature and auto refreshes after some time. So, i removed/renamed all of the lock files (.lck) existed under all of these folder and tried the machine again and as what i’ve expected, it started working back again.

* Note: Performing these steps on PROD systems could be dangerous and can cause serious performance/hang issues. Performing such steps always requires recommendations from the VMWARE support.

Thanks
Prashant Dixit
“Sharing is Good”

Posted in Advanced | Tagged: , , | Leave a Comment »

What is Yum in Linux and how to configure ?

Posted by FatDBA on October 1, 2012

Why to use: Many times while installing Database on Linux/Linux most of us encounters problems with the pre-installations which requires some of the Packages to be available to Oracle before the installation starts.  Searching required RPM’s online is a tedious job and needs lot of patience which could ultimately lead to errors about “Dependencies Failed”.  A best resolution of this problem is YUM, which resolves dependency part by itself  and allow the package search and install easily from the repository created.

The Yellow dog Updater Modified (YUM) is a package management application for computers running Linux operating systems. yum is a standard method of managing the installation and removal of software.

Steps to Configure YUM Repository on RED HAT:

1) Install vsftpd (FTP Server) & createrepo Packages
Insert RHEL 5 Installation DVD

# mkdir ~/Desktop/rhel_cd
# mount /dev/cdrom ~/Desktop/rhel_cd
# cd ~/Desktop/rhel_cd/Server
# rpm –ivh vsftpd*
# rpm –ivh createrepo*

2) Copy all packages from DVD to LINUX filesystem

# cp ~/Desktop/rhel_cd/Server/ /var/ftp/pub
# cd /
# eject

3) Configure Local Yum Repository Server

# cd /var/ftp/pub/Server/repodata
# cp comps-rhel5-server-core.xml /var/ftp/pub/Server/
# cd /var/ftp/pub/Server
# createrepo -vg comps-rhel5-server-core.xml /var/ftp/pub/Server/
# service vsftpd start
# chkconfig –add vsftpd

4) Configure Clients
Run following commands on clients

# vi /etc/yum.repos.d/yum_server.repo

Add:
[Server]
name=YUM_SERVER
baseurl=ftp://192.168.2.100/pub/Server
gpgcheck=0

# vi /etc/vsftpd/vsftpd.conf

Set the following parameter (if not already set)
anonymous_enable = yes

# service vsftpd restart

Now your clients can use this yum server to download & install packages & resolve
dependencies.

Posted in Advanced | Tagged: | Leave a Comment »

Alias (Unix)

Posted by FatDBA on July 23, 2012

Alias:

The alias command allows you to make new shortcuts and synonyms for commonly used commands.

Alias settings resets every time you reboots or switch users in Unix. There are number of ways/methods to permanently save Alias in your machine. Below discussed is one of the most common way to fix alias settings to make them global and always available.

1. Edit your bash profile (If using Bash Shell) and add all alias you want to set.

[root@localhost /]# gedit /etc/bashrc
(Below are excerpts from a live bashrc profile from one of my machine)

# /etc/bashrc

# System wide functions and aliases
# Environment stuff goes in /etc/profile

alias base=’cd /u01/app/oracle’
alias c=’clear’
alias home=’cd /u01/app/oracle/product/10.2.0/db_1′
alias sql=’sqlplus / as sysdba’
alias sid=”echo $ORACLE_SID”

# By default, we want this to get set.
# Even for non-interactive, non-login shells.

2. Add source /etc/bashrc to your ~/.bashrc profile.

[root@localhost /]# gedit ~/.bashrc

Will look like —

# .bashrc
# User specific aliases and functions

alias rm=’rm -i’
alias cp=’cp -i’
alias mv=’mv -i’
source /etc/bashrc

# Source global definitions

 

Enjoy The Shortcuts using ‘alias’ — Below is one of alias used to reach directory – ORACLE_BASE using alias named ‘base’.

[root@localhost ~]# base

[root@localhost oracle]# pwd
/u01/app/oracle
[root@localhost oracle]#

Posted in Advanced | Tagged: , | 1 Comment »

 
%d bloggers like this: