Blog


How to easily enable automatic DNSSEC in BIND9

I find most of DNSSEC howto’s unnecessarilly complicated and inefficient. Here I am going to desribe how to enable DNSSEC in BIND9 using its native automatic signing that doesn’t require any extra scripting.

This manual requires the zone to be dynamic – it will not work with static zone.

Step 1 – create the zone keys

Create a new directory in named / bind config folder, for example /etc/bind/dnssec and change its owner to user who is running bind daemon.

Enter the folder and execute these 2 commands to create zone and record signing keypairs:

dnssec-keygen -a NSEC3RSASHA1 -b 2048 -n ZONE example.org
dnssec-keygen -f KSK -a NSEC3RSASHA1 -b 4096 -n ZONE example.org

Step 2 – configure zone

Now edit the zone configuration file and add 2 lines with key-directory and auto-dnssec, example of whole zone config follows:

    zone "bena.rocks" {
        type master;
        notify yes;
        file "/etc/bind/db.bena.rocks.conf";
        key-directory "/etc/bind/dnssec";
        auto-dnssec maintain;
    };

Now run named-checkconf && rndc reload. It’s almost done.

Step 3 – enable signing

Finally tell BIND to load the keys and sign your zone:

rndc loadkeys example.org
rndc sign example.org

Unless anything goes wrong (bind logs are your friend) you just enabled automatic zone signing, you can check your zone – it should contain signed records now. All new records will be automatically signed too.




How to disable spotlight on MacOS

So, few weeks after having this MacBook Air with horrid screen, I figured out that there is a small issue with RAM. 2 tasks permanently eating 2GB of RAM, effectively reducing 8GB of RAM this machine has, to only 6GB (which sadly isn’t enough these days, especially not for a power user, who wants to use Firefox and also some other programs – Mozilla expects that firefox is all you need, so it deliberately uses all free RAM you have :)).

The problem was with these 2:

  • Kernel task – obviously a kernel, but why does it need 1GB? I need to figure that out later
  • mdm_store – process belonging to “spotlight” which is some searching / suggestion feature I will never use

So I found some information about spotlight and people who wanted to disable to protect privacy. While I completely don’t care about that, I do care about my RAM, so here is how you can kill it permanently. Open a terminal and put there:

sudo mdutil -a -i off
ps -ef | grep mdm
# then kill the pids of mdm processes

Here we go, 1GB of RAM back for use!




Why I hate wireless

There is this new ongoing trend, forced by Apple and companies that follow Apple, that everything must be wireless. Your headphones, your charger, phone, everything. No more cables.

Sure this is a step forward, no more cables, sounds really cool – but is the technology we have now ready for it? I don’t think so, for multiple reasons:

Batteries

Everything that is wireless must have a battery inside, otherwise it will not operate since it needs power. Can we create batteries that are eco-friendly and last forever? Nope.

For each wireless gadget we need to create a battery, which is actively harming environment, the battery adds to complexity and weight of the item and it is soon (within 5 years) going to wear out and whole thing becomes unusable. Not a big deal for large corporations like Apple that wish their consumers renewed (buy new) gadgets every year or two. The battery waste (old batteries) are also not very eco-friendly.

Energy efficiency

Wireless charging is least efficient charging that was ever invented. Not that typical charging was efficient, even that is wasting lot of energy, from total energy produced by power plant that is taken by a charger, only few percent is actual energy stored in a battery. But it’s even worse with wireless, the charger consumes huge amount of power just to store tiny amount of it in battery and it’s extremely slow.

 

These 2 things combined make wireless one of most disastrous innovation from ecological point of view – just image what would happen if everyone switched to wireless gadgets, the vast amount of used batteries and energy overhead from wireless charging would put huge impact on environment. Wireless may be a future, but technology is not yet there to use it in eco-friendly manner.




Optimizing Ansible for high performance

So, I have made a custom Ansible setup for more than 4000 servers in 12 different countries across the planet, and that gave me some insight into how to make it perform better.

First of all, sadly Ansible doesn’t yet support “proxy / caching servers” as in servers that you could use to execute playbook through. You can configure SSH proxy server, but that won’t help with performance. Only way to execute playbook from another server is to install Ansible there as well, sync the playbooks somehow and execute from this host.

Anyway, now for the performance hacks.

Redis caching

Major boost in performance. Simply install redis server on same host as Ansible and put this to configuration of ansible:

[defaults]
fact_caching = redis
fact_caching_timeout = 86400
fact_caching_connection = localhost:6379:0
gathering = smart

This will put all facts of every server you connect to into redis cache and next time you execute anything on that server (within 1 day), ansible will not gather facts again, but it would take them from redis cache.

Pipelining

Minor boost. But slightly helps:

[ssh_connection]
retries=3
pipelining=True

Multithreading

Major boost, but not very stable, often causes troubles. Putting more than 20 makes Ansible quite unstable.

[default]
forks = 10

Example config

This config works pretty well to me:

[defaults]
fact_caching = redis
fact_caching_timeout = 86400
fact_caching_connection = localhost:6379:0
gathering = smart
host_key_checking = False
timeout = 20
retry_files_save_path = /home/ansible/retry/
forks = 10
log_path=/var/log/ansible.log

[ssh_connection]
retries=3
pipelining=True

 




How to install mdadm to XenServer 7

Based on https://discussions.citrix.com/topic/378478-xenserver-7-raid1-mdadm-after-install-running-system/

# 1. Install Xenserver 7 with normal single disk configuration, don't create SR storage
# 2. copy partition talbe from sda to sdb
 
# !!! important don't write the order wrongly, from sda to sdb is like the following
sgdisk /dev/sda -R /dev/sdb


# Important! The partition layout may differ with XenServer version, basically there are 2 partitions with same size for OS and backup
# and at least 1 for GRUB and 1 for swap

parted /dev/sdb
# print
# quit

# You should see a list of partitions, one of them will be flagged as legacy_boot, grub, let's call it BOOT

## flag them as raid disks for sdb partitions
sgdisk --typecode=1:fd00 /dev/sdb # OS
sgdisk --typecode=2:fd00 /dev/sdb # Backup OS
sgdisk --typecode=3:fd00 /dev/sdb # ??
sgdisk --typecode=4:ef02 /dev/sdb # BOOT
sgdisk --typecode=5:fd00 /dev/sdb # LOGS
sgdisk --typecode=6:fd00 /dev/sdb # SWAP

## note that BOOT partition is not the same like the others. Because in the new disk configuration they changed the boot partition.

# 5. create the software raid partitions

mdadm --create /dev/md0 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb1 missing
mdadm --create /dev/md1 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb2 missing
mdadm --create /dev/md2 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb3 missing
mdadm --create /dev/md3 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb4 missing
mdadm --create /dev/md4 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb5 missing
mdadm --create /dev/md5 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb6 missing
mkswap /dev/md5

# 6. copy the contents of / and /var/log directories to the new partitions

mkfs.ext3 /dev/md0
mkfs.ext3 /dev/md4

# 7. mount newly created/formatted partitions
mount /dev/md0 /mnt
mkdir -p /mnt/var/log
mount /dev/md4 /mnt/var/log

# 8. copy contents to the newly mounted directory

cp -xR --preserve=all / /mnt

# 9. create a mdadm file for boot process (!!!if you forget the file the MD devices will have different names)

### the head of the file should include these lines
echo "MAILADDR root" > /mnt/etc/mdadm.conf
echo "auto +imsm +1.x -all" >> /mnt/etc/mdadm.conf
echo "DEVICE /dev/sd*[a-z][1-9]" >> /mnt/etc/mdadm.conf
mdadm --detail --scan >> /mnt/etc/mdadm.conf

# 10. copy the contents to the root folder
cp /mnt/etc/mdadm.conf /etc

# 11. configure mount points
sed -i 's/LABEL=root-[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab
sed -i 's/LABEL=swap-[a-zA-Z\-]*/\/dev\/md5/' /mnt/etc/fstab
sed -i 's/LABEL=logs-[a-zA-Z\-]*/\/dev\/md4/' /mnt/etc/fstab
sed -i '/md5/ a\/dev/md5          swap      swap   defaults   0  0 ' /mnt/etc/fstab
cp /mnt/etc/fstab /etc

# 12. change the label name for /dev/sdb1 partition
e2label /dev/sda1 |xargs -t e2label /dev/sdb1

# 13. bind mount dev sys proc to the mnt folder
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys
mount --bind /proc /mnt/proc
chroot /mnt  /bin/bash

# 14. install grub on /dev/sdb
grub-install /dev/sdb

# 15. backup initrd
cp /boot/initrd-$(uname -r).img /boot/initrd-$(uname -r).img.bck

# 16. create new initrd for raid
dracut --mdadmconf --fstab --add="mdraid" --filesystems "ext3 tmpfs devpts sysfs proc" --add-drivers="raid1 raid456 mdraid1x mdraid09" --force /boot/initrd-$(uname -r).img $(uname -r) -M

###never change the boot configuration via grub-mkconfig.. it will kill xenserver.. change the GRUB configuation, by hand inside the files

# 17. change grub configuration
sed -i 's/quiet/rd.auto rd.auto=1 rhgb quiet/' /boot/grub/grub.cfg
sed -i 's/LABEL=root-[a-zA-Z\-]*/\/dev\/md0/' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod gzio' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod part_msdos' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod diskfilter mdraid09' /boot/grub/grub.cfg
sed -i '/search/ c\   set root=(hd0,gpt1)' /boot/grub/grub.cfg

# 18. exit from chroot
exit

# 19. change the same things in sda1 partition so after reboot you don't need to boot from second disk
cp /mnt/boot/initrd-3.10.0+10.img /boot/

sed -i 's/quiet/rd.auto rd.auto=1 rhgb quiet/' /boot/grub/grub.cfg
sed -i 's/LABEL=root-[a-zA-Z\-]*/\/dev\/md0/' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod gzio' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod part_msdos' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod diskfilter mdraid09' /boot/grub/grub.cfg
sed -i '/search/ c\   set root=(hd0,gpt1)' /boot/grub/grub.cfg

# 20. reboot

# !!!!!!reboot the server the server will boot from software raid..!!!!!
# 21. After the reboot add the /dev/sda to the new MD disks.

sgdisk /dev/sdb -R /dev/sda

mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md1 /dev/sda2
mdadm -a /dev/md2 /dev/sda3
mdadm -a /dev/md3 /dev/sda4
mdadm -a /dev/md4 /dev/sda5

# This will take a while for resync of all disks

grub-install /dev/sda

# Create SR

xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=<host-uuid> name-label=”SRRaid1-Local” shared=false type=lvm

This post is basically just a backup of that forum post in case it become dead link




Letsencrypt kung-fu

Let’s encrypt CLI client is by far the most shittiest software ever invented, there is probably no doubt about it, but sadly, it’s the only interface that is supported, and unless you want to pay money for SSL certificate you need to live with that.

First of all – yes, their client (without asking or telling you) WILL run sudo and WILL use root and most likely WILL install garbage on your server that you don’t want to have there. If you never used letsencrypt client before, run it on testing VM first, before it desecrates your favorite web server with random garbage you don’t want there.

The letsencrypt client is written for dumb people, and it is based on undocumented black magic that I will try to uncover here a bit. The client basically works with a component called “certbot” which is a software that run on your server and does something to prove that you really own the domains for which you want to generate your SSL certificate. Because letsencrypt staff doesn’t want to bother you with technicalities they created this crap of a software to deal with them for you, in their own way, like it or not. It uses so called ACME (Automatic Certificate Management Environment) protocol to verify that you are owner. This thing is not a rocket science, and in a nutshell all it does is publish some data used to prove your ownership through your webserver, usually located on webroot/.well_known, their counter-party server will try to locate these by accessing your.domain/.well_known and in order to make it possible to verify your domain without modifications to your webserver, all you need to do is to create a central webroot and then make a symlink from all domain webroots to this one (just ln -s /var/www/letsencryptshite/.well_known /var/www/your.uber.tld/.well_known).

Once you do that, always pass these 2 parameters to their “software”:

--webroot --webroot-path /var/www/letsencrypt_shite

I also strongly recommend you to maintain a comma separated list of all domains for which you want to get your certificate and store it somewhere like /etc/letsencrypt/domains because you will need to provide this list very often.

Now a little cheat sheet:

Renewing all domains

This can even be in your cron

./letsencrypt-auto renew --webroot --webroot-path /var/www/letsencrypt_shite

You may need to restart / reload your web server after doing this, since the certificate will be overwritten, and Apache seems to be caching it somehow.

Adding or remove a domain and regenerate certificate

Modify your /etc/letsencrypt/domains list and run

./certbot-auto certonly --webroot --webroot-path /var/www/letsencrypt_shite/ --agree-tos --expand -d `cat /etc/letsencrypt/domains`

Common locations:

/etc/letsencrypt – root of this thing’s config

/etc/letsencrypt/live – symlinks to current certificates, that’s where you can find chains for your domains

Example apache config that uses letsencrypt cert

<VirtualHost *:80>
    ServerName bena.rocks
    DocumentRoot /var/www/bena.rocks
</VirtualHost>

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/insw.cz/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/insw.cz/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/insw.cz/chain.pem
    ServerName bena.rocks
    DocumentRoot /var/www/bena.rocks
</VirtualHost>

 




Why SPF doesn’t solve anything in regards of e-mail spoofing

If you ever thought that having an SPF record would effectively prevent anyone from spoofing a delivery of e-mail from your domain, you were terribly wrong. It wouldn’t. This whole “sender policy framework” is built on top of flawed ancient SMTP protocol that never was designed with slightest security in mind.

Majority of all e-mail clients will display value stored in “From” of e-mail header as a sender of your e-mail. That is exactly the “sender” that SPF doesn’t check. SPF is checking the “envelope sender” which is something entirely else, and sadly, almost never displayed anywhere in e-mail clients.

Basically, delivering an e-mail on behalf of anyone that would pass SPF is as easy as spoofing the From in the mail header and sending this from a mail server that has its own IP in its own SPF record.

This 3rd domain would be a part of “envelope sender” which is however not visible anywhere in the mail client (unless you open original source) and SPF check would check against this domain. The spoofed domain that would be visible in mail client wouldn’t be checked anywhere though and most of users would be easily tricked by this.

There are ways to make this harder to do, such as implementing DMARC policies or DKIM, but again, most of regular users would easily be tricked. Nobody cares if e-mail was digitally signed or not, unless you explicitly tell them to do that.

As of now there is only one working protection when it comes to e-mails and that is: DO NOT TRUST THEM. Whatever e-mail you received, from whoever, may be a fake. Even GPG signature could be a fake if someone steals the private key of victim somehow. So yes, if someone is asking you for money in e-mail, or whatever else, better call them, or meet them in person. It can save you lot of troubles.




Gentoo quick setup (for advanced gentoo users)

  • May 8, 2016
  • Linux

This is an excerpt from gentoo handbook containing only the stuff that really matters, with no extra stuff:

Prepare your disks

Do I need to explain how? 🙂 if yes, this is not for you

Mount them

mkdir /mnt/gentoo
mount root /mnt/gentoo
mount boot /mnt/gentoo/boot

Prepare stage3

cd /mnt/gentoo
lynx http://distfiles.gentoo.org/releases/amd64/autobuilds/current-install-amd64-minimal/
tar xvjpf stage3-*.tar.bz2 --xattrs

Chroot

mirrorselect -i -o >> /mnt/gentoo/etc/portage/make.conf
mkdir /mnt/gentoo/etc/portage/repos.conf
cp /mnt/gentoo/usr/share/portage/config/repos.conf /mnt/gentoo/etc/portage/repos.conf/gentoo.conf
cp -L /etc/resolv.conf /mnt/gentoo/etc/
mount -t proc proc /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --make-rslave /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
mount --make-rslave /mnt/gentoo/dev
chroot /mnt/gentoo /bin/bash
emerge-webrsync
emerge --sync

Emerge setup

eselect profile list
eselect profile set XXX
emerge --ask --update --deep --newuse @world
echo "Europe/Prague" > /etc/timezone
emerge --config sys-libs/timezone-data
emerge vim
vi /etc/locale.gen
locale-gen
eselect locale list
eselect locale set 5

Kernel

emerge sys-kernel/gentoo-sources sys-apps/pciutils 
cd /usr/src/linux 
# Build as you like
emerge sys-kernel/linux-firmware

Initramfs

Pick one

# Genkernel
emerge sys-kernel/genkernel
genkernel --install initramfs

# Dracut
emerge dracut
cd /boot
dracut

Filesystems

Just edit /etc/fstab

/dev/sda2   /boot        ext2    defaults,noatime     0 2
/dev/sda3   none         swap    sw                   0 0
/dev/sda4   /            ext4    noatime              0 1

Networking

emerge net-misc/dhcpcd ntpd net-misc/netifrc

cd /etc/init.d
ln -s net.lo net.eth0
rc-update add net.eth0 default

Grub

emerge --ask sys-boot/grub:2
grub2-install /dev/sda
grub2-mkconfig -o /boot/grub/grub.cfg

 




systemd cheat sheet

Services overview and unit files

# Show running units
systemctl

# Show status of OS
systemctl status

# List all units
systemctl list-units

# Display failed units:
systemctl --failed

The available unit files can be seen in /usr/lib/systemd/system/ and /etc/systemd/system/

Start / stop

systemctl start unit
# samples
systemctl start apache2.service

# stop
systemctl stop apache2.service

# restart
systemctl restart unit

# reload
systemctl reload unit

# Check whether a unit is already enabled or not:
systemctl is-enabled unit

# Show the status of a unit, including whether it is running or not:
systemctl status unit

Checking status

By default it should be possible to view the output of every unit using journal – journalctl

# Display logs of unit
journalctl -u unit

Enable / disable service

# Enable a unit to be started on bootup:
systemctl enable unit

# Disable a unit to not start during bootup:
systemctl disable unit

# Mask a unit to make it impossible to start it:
systemctl mask unit

# Unmask a unit:
systemctl unmask unit

Power management

# Reboot
systemctl reboot

# Shut down and power-off the system:
systemctl poweroff

# Suspend the system:
systemctl suspend

# Put the system into hibernation:
systemctl hibernate

# Put the system into hybrid-sleep state (or suspend-to-both):
systemctl hybrid-sleep

Modifying system

  • /usr/lib/systemd/system/: units provided by installed packages
  • /etc/systemd/system/: units installed by the system administrator

When you modify the unit file, you always need to run:

systemctl daemon-reload

See https://wiki.archlinux.org/index.php/systemd for more information




New blog

Heya, I have decided to move from blogger to my own blog for couple of reasons. The biggest one was that blogger doesn’t support custom code formatters and I often need that when I am posting source codes.

I also got my own domain now 🙂

The contents of previous blog were imported back here and I will post new stuff here only.

Thank you for using my blog, I hope you will find it useful!