Linux (9)


How to install mdadm to XenServer 7

Based on https://discussions.citrix.com/topic/378478-xenserver-7-raid1-mdadm-after-install-running-system/

This post is basically just a backup of that forum post in case it become dead link




Letsencrypt kung-fu

Let’s encrypt CLI client is by far the most shittiest software ever invented, there is probably no doubt about it, but sadly, it’s the only interface that is supported, and unless you want to pay money for SSL certificate you need to live with that.

First of all – yes, their client (without asking or telling you) WILL run sudo and WILL use root and most likely WILL install garbage on your server that you don’t want to have there. If you never used letsencrypt client before, run it on testing VM first, before it desecrates your favorite web server with random garbage you don’t want there.

The letsencrypt client is written for dumb people, and it is based on undocumented black magic that I will try to uncover here a bit. The client basically works with a component called “certbot” which is a software that run on your server and does something to prove that you really own the domains for which you want to generate your SSL certificate. Because letsencrypt staff doesn’t want to bother you with technicalities they created this crap of a software to deal with them for you, in their own way, like it or not. It uses so called ACME (Automatic Certificate Management Environment) protocol to verify that you are owner. This thing is not a rocket science, and in a nutshell all it does is publish some data used to prove your ownership through your webserver, usually located on webroot/.well_known, their counter-party server will try to locate these by accessing your.domain/.well_known and in order to make it possible to verify your domain without modifications to your webserver, all you need to do is to create a central webroot and then make a symlink from all domain webroots to this one (just ln -s /var/www/letsencryptshite/.well_known /var/www/your.uber.tld/.well_known).

Once you do that, always pass these 2 parameters to their “software”:

I also strongly recommend you to maintain a comma separated list of all domains for which you want to get your certificate and store it somewhere like /etc/letsencrypt/domains because you will need to provide this list very often.

Now a little cheat sheet:

Renewing all domains

This can even be in your cron

You may need to restart / reload your web server after doing this, since the certificate will be overwritten, and Apache seems to be caching it somehow.

Adding or remove a domain and regenerate certificate

Modify your /etc/letsencrypt/domains list and run

Common locations:

/etc/letsencrypt – root of this thing’s config

/etc/letsencrypt/live – symlinks to current certificates, that’s where you can find chains for your domains

Example apache config that uses letsencrypt cert

 




Gentoo quick setup (for advanced gentoo users)

  • May 8, 2016
  • Linux

This is an excerpt from gentoo handbook containing only the stuff that really matters, with no extra stuff:

Prepare your disks

Do I need to explain how? 🙂 if yes, this is not for you

Mount them

Prepare stage3

Chroot

Emerge setup

Kernel

Initramfs

Pick one

Filesystems

Just edit /etc/fstab

Networking

Grub

 




How to create login info in motd similar to ubuntu server

Ever wondered how could you get this cool login screen you can see when you login to ubuntu server on other distros?

This is a part of proprietary system called landscape. But this thing is too cool to remain proprietary, so I created a similar login info screen here: http://github.com/benapetr/system-info

All you need to do is to append it to your login scripts and here we go:

 




How to setup debian build environment for both amd64 and x86 platform

This article describes how to setup environment in which you can build both amd64 as well x86 .deb packages. It’s assuming that you already have working debian on amd64 kernel.

First step: install debootstrap

Now you should have fully working x86 debian in /opt/jessie-386

Second step: make a script to switch into i386 version

Create a script anywhere you like, for example /bin/switch_32 with following content:

Then make it executable. Now running it would switch you into fully working debian x86 in which you can install all required packages using apt-get and build your x86 packages.




How to install mdadm on citrix xen 6.5

For some reason citrix doesn’t like mdadm so they make everything possible to stop it from working on their xen server.

Here is a guide that would make it work there, but it may not survive system patching

Setup

Connect at least 2 disks to your box. Install a xen server without local storage on first disk.

Installing mdadm

The default install contains mdadm but it doesn’t load raid modules to kernel. In order to enable it, following needs to be done:

Partitioning the disks

Now we create a final schema we want to use on our server on disk /dev/sdb, xen needs to have at least 3 partitions, 1 is for boot loader, second is for OS, I recommend 20gb or more, because this disk is pretty much impossible to extend, although citrix defaults it to 4GB, last partition is for local storage and it should take all remaining space on disk.

Note: Citrix by default creates 3 partitions, 1 for OS, second is empty, same sized as first one and probably used for system upgrade. Third is used for local storage LVM. You don’t have to create second partition for it to work, but system upgrades may not be available if you don’t create it. On other hand system upgrades will likely not work anyway as citrix doesn’t support mdadm installations.

In this guide I will use old MS-DOS partition table because although it’s old, it’s much better supported and it just works. You can also use GPT partitions if you want, but I had some issues getting them work with mdadm and syslinux.

We will have a separate /boot partition for boot loader, because syslinux shipped with xen is having troubles booting from raid device for some reason.

So this is how the layout of sdb should look after we finish the partitioning:

  • /dev/sdb1 (2 GB) for bootloader
  • /dev/sdb2 (20 GB) for OS
  • /dev/sdb3 (rest) for LVM

Now you should be able to boot from /dev/sdb if you are not there is something wrong with the setup, you need to figure out if your problem is with

  • MBR (No bootable device)
  • Boot loader (Missing operating system.)
  • /boot (Linux will start booting but die in progress – try removing quiet and splash from parameters)

 Syncing the disks

Now if you were able to boot up you need to setup the sda disk

Create the same 3 partitions as you did on sdb on sda and then

 




How to insert ubuntu PPA’s to debian

Ubuntu created a very interesting service called PPA – personal package archives.

If anyone of you ever used debian (or ubuntu) you may be wondering how cool it could be, if your software could be in official repository so that people could just type:

sudo apt-get install blah

These of you who managed to get through all the bureaucracy and got the packages there, might be wondering how cool it would be, if that package which got build and published there months ago, could be ever updated to latest version of your software 🙂

Today, it’s both possible and very easy, thanks to PPA.

I am not going to describe the process of how to submit stuff to PPA, because that is explained on many places, but I am going to explain how PPA work and how you can use this powerful service on debian.

What is PPA

PPA is basically a repository server, where each user can create their own personal aptitude remote repository, using unlimited number of own GPG keys. There is basically no difference between PPA repository and any other apt repository.

How do I make it work on debian

There is no command apt-add-repository on debian, because PPA is ubuntu thing. That makes 2 problems:

* You can’t easily add PPA
* You may have problems with dependencies, as some debian packages are named differently than ubuntu packages

First problem can be overriden easily. You can either get the source code of apt-add-repository and install it localy, OR you can just insert the url of PPA in format of

to your /etc/apt/sources.list where USER is username on PPA and REPOSITORY is PPA. You also need to replace ubuntu version with any ubuntu version which is most close to your debian version, typically some LTS (precise, or lucid).
The other thing you need to do, is installation of GPG key of the user you want to download packages from:

Now you should be able to download and use the PPA repository.

You may be however facing the other problem with dependecies, and there is currently no other solution for it, than rebuilding the source package yourself with correct debian dependencies.




Handling OOM issues gracefully on linux

I figured out that there is almost no simple way to handle situation when your system is running out of memory. There is a subsystem called OOM killer implemented in the kernel, but that thing is truly dangerous. You probably never want to get your system to point when it is being used, because it might leave the system in unusable state.

For this reason I created a tool that allows everyone who uses linux to handle this kind of situation gracefully, it’s called terminator daemon, and it watches the system and eventually kills specific kinds of processes, which can be easily defined by an administrator.

You can find more about this tool on https://github.com/benapetr/terminator

Basic idea is, that when your system is running out of memory, this daemon will pick processes that can be “safely” killed without any impact on system and takes them out according to your settings. You can even prevent it from killing certain processes, for example: If you were running mysql server, you probably wouldn’t care about interactive shells being killed, but you would care if mysql was killed.

In such situation you can tell terminatord to never kill mysql, but other “user processes” can be safely killed.

In addition you can even set terminatord to execute some command on every kill, for example it can send you a mail that something wrong is going on with your system.

There are several examples on GitHub, here I copy pasted some of them:

Kill all processes that use more than 400mb of ram, except for user apache and root

Kill random processes in case that system has less than 100mb of free ram, except for root

Combine both examples, in this example unlike the previous one, apache processes will not be killed, when system go OOM




Installing oracle 11i (12c) on debian sid (6.0)

Somehow it happened that I decided to try out oracle on latest linux kernel and debian system.

I horribly failed.

In order to make you not fail, here is a guide what to do in order to install it

1. Install necessary packages

This is probably not needed:
2. Update /dev/shm

Oracle requires shared pool to be present there, but latest linux is using /run/ to store it – this is dangerous and you should reboot your server once you finish (you only need to do that if /dev/shm points to /run/shm)


3. Change the system limits

Write the following options to the /etc/security/limits.confce file:

Add the following to /etc/pam.d/login to validate parameters /etc/security/limits.conf:

Make sure you have this in your /etc/sysctl.conf file:

4. Change some more links

5. Create user credentials

6. Change dash to bash

Now you can install oracle using their universal installer. If you don’t have GUI, you need to create a response file. Don’t forget to bypass system checks, because the version of libraries used by debian is too new for oracle