Handling OOM issues gracefully on linux

I figured out that there is almost no simple way to handle situation when your system is running out of memory. There is a subsystem called OOM killer implemented in the kernel, but that thing is truly dangerous. You probably never want to get your system to point when it is being used, because it might leave the system in unusable state.

For this reason I created a tool that allows everyone who uses linux to handle this kind of situation gracefully, it’s called terminator daemon, and it watches the system and eventually kills specific kinds of processes, which can be easily defined by an administrator.

You can find more about this tool on

Basic idea is, that when your system is running out of memory, this daemon will pick processes that can be “safely” killed without any impact on system and takes them out according to your settings. You can even prevent it from killing certain processes, for example: If you were running mysql server, you probably wouldn’t care about interactive shells being killed, but you would care if mysql was killed.

In such situation you can tell terminatord to never kill mysql, but other “user processes” can be safely killed.

In addition you can even set terminatord to execute some command on every kill, for example it can send you a mail that something wrong is going on with your system.

There are several examples on GitHub, here I copy pasted some of them:

Kill all processes that use more than 400mb of ram, except for user apache and root

# get uid of apache
grep apache /etc/passwd
# let's say apache is user 20, now we test it
terminatord -dvvv --soft 400 --hard 420 --ignore 20 --dry
# if everything is ok
terminatord -d --soft 400 --hard 420 --ignore 20 --kill

Kill random processes in case that system has less than 100mb of free ram, except for root

# test it
terminatord -dvvv --ssoft 100 --shard 60 --quiet --dry

Combine both examples, in this example unlike the previous one, apache processes will not be killed, when system go OOM

# let's say apache is user 20, now we test it
terminatord -dvvv --soft 400 --hard 420 --ignore 20 --shard 60 --ssoft 100 --dry
# if everything is ok
terminatord -d --soft 400 --hard 420 --ignore 20 --kill --shard 60 --ssoft 100

Installing oracle 11i (12c) on debian sid (6.0)

Somehow it happened that I decided to try out oracle on latest linux kernel and debian system.

I horribly failed.

In order to make you not fail, here is a guide what to do in order to install it

1. Install necessary packages

apt-get install build-essential libaio1 gawk ksh libmotif3 alien libtool lsb-rpm make gcc libsc-dev libxp-dev

This is probably not needed:
2. Update /dev/shm

Oracle requires shared pool to be present there, but latest linux is using /run/ to store it – this is dangerous and you should reboot your server once you finish (you only need to do that if /dev/shm points to /run/shm)

sudo rm /dev/shm
sudo mkdir /dev/shm
sudo chmod 1777 /dev/shm 

3. Change the system limits

Write the following options to the /etc/security/limits.confce file:

oracle soft nproc 2047        
oracle hard nproc 16383
oracle soft nofile 1023
oracle hard nofile 65535

Add the following to /etc/pam.d/login to validate parameters /etc/security/limits.conf:

session required /lib/security/        
session required

Make sure you have this in your /etc/sysctl.conf file:

sysctl -p:         
# sysctl -p
kernel.printk = 4 4 1 7
kernel.maps_protect = 1
fs.inotify.max_user_watches = 524288
vm.mmap_min_addr = 65536
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
fs.file-max = 65535
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 1024 65535
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 262144
net.core.wmem_max = 262144

4. Change some more links

ln -s /usr/bin/awk /bin/awk        
ln -s /usr/bin/rpm /bin/rpm
ln -s /usr/bin/basename /bin/basename
mkdir /etc/rc.d
for i in 0 1 2 3 4 5 6 S ; do ln -s /etc/rc$i.d /etc/rc.d/rc$i.d ; done

5. Create user credentials

addgroup oinstall        
Adding group 'oinstall' (GID 1001) ...
# addgroup dba
Adding group 'dba' (GID 1002) ...
# addgroup nobody
Adding group 'nobody' (GID 1003) ...
# usermod -g nobody nobody
useradd -g oinstall -G dba -p password -d /home/oracle -s /bin/bash oracle
# passwd oracle        
Password changed.
mkdir /home/oracle 
chown -R oracle:dba /home/oracle

6. Change dash to bash

# if you see this:
ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Mar  2 18:07 /bin/sh -> dash
# do this:
sudo su
cd /bin
rm sh
ln -s bash sh
# These links needs to be created as well to fix problems with makefiles
sudo su
mkdir /usr/lib64
ln -s /lib/x86_64-linux-gnu/ /usr/lib64/
ln -s /usr/lib/x86_64-linux-gnu/libc_nonshared.a /usr/lib64/
ln -s /usr/lib/x86_64-linux-gnu/libpthread_nonshared.a /usr/lib64/
ln -s /usr/lib/x86_64-linux-gnu/ /usr/lib64/

Now you can install oracle using their universal installer. If you don’t have GUI, you need to create a response file. Don’t forget to bypass system checks, because the version of libraries used by debian is too new for oracle

About me

Hi all,

So I decided to start writing some interesting articles occasionally, and this one is my first, so I will introduce myself a bit.

My name is Petr Bena, I am an open source developer and in real life I work as a sysadmin.

I am heavily involved in wikimedia project, where I serve as a “community  developer” (non-paid guy who is helping out with many software projects) and I am also volunteering on many other wikimedia projects.

Beside wikimedia, I am working on Pidgeon – open source irc client which I decided to create because I just couldn’t find a good cross-platform irc client 🙂

If you want to find out more personal information about me, have a look on this wikipedia page:

If you are lazy to click the link, here you have some personal information:

I live in Europe, I like coca-cola and czech beer. I don’t like czech politicians. I have 1 cat and 2 tomcats 🙂 and I like them more than they like me. And the most important thing – I like star wars (who doesn’t? :P)

Thank you for reading my blog