Pattern for replacable pet servers on AWS

So, here is a problem:

You have a server that you want to be able to replace.  Usually this would be done by having some kind of configuration management, packed image, or cloud-init startup scripts that set it back up.  However, it’s a pet.. it’s way too much work to setup full configuration management, and it’s config files get updated by people logging in to it anyway.

Now, sure, you could just create an AMI of it and restore the AMI if something goes wrong.. but this has some downsides.   The AMI gets out of date if it is not created frequently.  You also have to patch in place and it’s a fairly complex job to go from a major OS release to another, or to switch distro.

Here is a pattern I use that seems to make this much less painful, where I have a single pet.  This is basically like etckeeper, but is even more basic.

1. I setup a user data (cloud-init) script to do the basics of setting host name, installing packages, configuring bind mounts, starting services, installing cron jobs, and other random bits.
2. I create an EFS share, iam role and mount that on the server as /config
3. I bind mount the bits of the OS config/data I actually care about into the filesystem, using a cloud-init shell script.

mkdir -p /config
echo ${efs-fs-id} /config    efs >> /etc/fstab
mount /config
for i in /etc/httpd/conf/httpd.conf /etc/named.conf /var/spool/cron /usr/local /root; do
echo /config$i $i none defaults,bind >> /etc/fstab
mount $i
done

This gives me a server that I can taint and replace with Terraform, I can replace the AMI with minimal effort, and the EFS data is of course backed up with AWS backup.  It means a normal server admin can easily see what is going to stay if they edit it, and can go edit config files without having to fight configuration management.

OS Configuration management saves times, if you have 3 or more servers that need to be configured the same / similar -  but for a single host, it sometimes takes longer than just configuring it.

Automated DNS and SSL certificates with Terraform and AWS

When building web infrastructure it used to be a really slow and often expensive process to get an SSL certificate, along with DNS records and overhead to keep the certificate from expiring.

AWS has made SSL certificates a whole lot easier, cheaper and faster than it used to be, by becoming a certificate authority and issuing SSL certificates with Amazon Certificate Manager (ACM). Certificates AWS generates can ONLY be used on AWS services, such as AWS ALB, but they have got some compelling reasons to use them, such as that the certificates are publicly signed, free and renewed automatically. AWS have also made DNS quick and easy to automate with AWS Route53 providing the ability to manage DNS records via automation even if they are just a short lived server such as a demo spun up for an hour.

Prior to 2018, we could use Terraform to create DNS records, but there was a manual step to get ACM to work, which was in order to use ACM, we used to have to prove the domain ownership manually by creating the certificate in AWS ACM, then adding the DNS record or responding to the email they send.

Turns out, that in early 2018 Hashicorp started to work on adding a Terraform feature to enable full automation of ACM: aws_acm_certificate_validation, Using this, we can now automate the entire process of creating a free publicly signed SSL certificate, even when the route53 zone is in a different AWS account.

I've created a little bit of example code in github that anyone is welcome to use as a basis for setting up AWS ACM. It is demonstrating Terraform connecting to two different AWS accounts and also defines a minimal ALB to demo the certificate. If your route53 zone is in the same account, you can just remove the lines "provider = aws.account_route53" in every resource to use the example for a single AWS account.

The demo includes two providers, as it assumes that the route53 zone is in a different AWS account. If you just use one account, then you only need one provider.  

Public Github project URL: https://github.com/nelg/terraform-aws-acmdemo  

In this SSL ACM Terraform demo we:

1. look up the route53 zone
2. create an ACM certificate
3. create a route53 record to validate the ACM certificate
4. get AWS ACM to check route53 for the validation certificate
5. create a route53 record that matches the certificate name and points at an ALB.
6. print out the URL to test it with

The remainder of the github project is just minimal configuration needed to demo this working, so it creates a minimal ALB in a VPC.

Google authenticator TFA for Android - Backup and OATH

I’ve been a fan of using multi/two factor authentication for anything that matters.

Thankfully, many sites these days are beginning to support using MFA, and many of them have standardized on OATH,
Google Authenticator, is one such OATH client app, implimenting TOTP (time based on time passsword).

OATH is a reasonaly good method of providing MFA, because it’s easy for the user to setup and pretty secure, and open, both for the client and server.
You can read all about how it works in RFC 6238, or wikipedia.
So, in a nut shell, we now have method that a client can generate a key, and a server than can authenticate that key.

Google Authenticator, being the client I use, as it supports adding the share key by simply reading a QR code, is great.
But, what if I loose my phone.. or want to use a second device.. or my computer? MFA of course, pretty much locks you out if you
loose your way to generate your TOTP..

Well, google provides you a number of static keys.. you can use.. but that’s not good enough for me.

So, I thought I’d see if I could backup google authenticator, and read the shared key from it. The answer to this is yes.

Here’s the technical details:
Backup Google Authenticator using Titanium Backup. This will generate 3 files on your SD card:
The file of intrest is:
sdcard/TitaniumBackup/com.google.android.apps.authenticator2-DATE-TIME.tar.gz

In this tar.gz, you will find:
data/data/com.google.android.apps.authenticator2/./databases/databases

This is an SQLlite3 database, that contains each account you have added to google authenticator.
So, after opening it with sqllite3, IE:

tar -zxvf sdcard/TitaniumBackup/com.google.android.apps.authenticator2-DATE-TIME.tar.gz data/data/com.google.android.apps.authenticator2/./databases/databases
sqlite3 data/data/com.google.android.apps.authenticator2/./databases/databases
sqlite> select * from accounts;


to get a list of your keys.
Each key is base32 encoded.

So, to decode your key, you use:

$ python
>>> import base64
>>> base64.b16encode(base64.b32decode('THEKEYFROMTHESELECT', True))

Then this will output the key into base 16, which is the format that oathtool

You can then generate the token form your linux, computer.
Ensure you have the package: oath-toolkit

Then

$ oathtool --totp BASE16KEY


will generate you the same key as google authentcator, provided the time is correct on your Linux system.
Note: Make sure you clear your bash history, if you don’t want your MFA key in your history. And of course,
only store it on encrypted storage.. including make sure your sdcard is secure/erased in some way.

Fritz!Box Telephony

 

In New Zealand,  VDSL internet is available from a number of providers.  Snap! is the provider I use, and they offer some very cool Fritz!Box VDSL modems.   I have the Fritz!Box 7390, and the other one on offer is the cheaper Fritz!Box 7360.

These routers are far more than just a basic VDSL router, offering a range of awesome features, including NAS, IPV6 (standard with Snap), good WIFI, DECT, VPNs.   This blog posts are about my experiences with setting up the Telephony component.

The first thing to note is, Snap! does not ship the right cable for connecting the Fritz!Box to a standard telephone line.  This is a special Y cable, and they will ship it to you if you ask them.   Note however, you’ll still need to make up an adapter, as this cable is for RJ545 telephone plugs, not NZ BT plugs.  My setup, is that I have a monitored alarm, so I need a telephone line, theirfore, this is for VDSL + standard phone line, rather than VDSL + VOIP phone.

Hardware:

Step 1:

Get the Y cable from Snap!.    It will have 1 plug at the end that connects to the Fritz!Box and a split end, one for your VDSL plug, and one for the telephone line.  They will change you $5 postage.
Here is the description of it, it’s the grey cable, first one on the left.

Here is the email I sent to Snap!   It took a while, but Michael Wadman did agree to ship it to me, case number #BOA-920-53402  if you need a reference case.

Hi,

I purchased a Fritz!Box 7390 from you, along with my internet subscription. It looks to me, from looking at their website, that it should come with some
cables that I don’t have. See:

http://www.fritzbox.eu/en/products/FRITZBox_Fon_WLAN_7390/index.php?tab=5

The Fritz!Box 7390 you supplied came with two Ethernet cables and power, but no other cables.
In the above link, you can see it should come with:

• 4.25 m-long ADSL/fixed-line network connection cable (Y cable)
• 1.5 m-long LAN cable
• RJ45-RJ11 adapter for connection to the ADSL line
• RJ45-RJ11 adapter for connection to the analog telephone line

I would like to plug my Analog PSTN phone line into the Fritz!Box so I can use it’s telephony features with my fixed line.     To do this, I need the cable that goes between the ISDN/analog and the phone jack on the wall.
I am aware that the Y cable does not actually fit NZ phone plugs.
This post discusses the matter on geekzone.
http://www.geekzone.co.nz/forums.asp?forumid=90&topicid=115999

Ref:

http://service.avm.de/support/en/skb/FRITZ-Box-7390-int/56:Pin-assignment-of-cables-adapters-and-ports-for-telephony-devices

Step 2:

Build an adapter for the telephone end.  This is easier than you might think.  What you need:

• RF45  Crimping tool
• RJ45 plug
• Any old phone cable with a BT11 plug on it
• A CAT5 RJ45 Network Cable Extender Plug Coupler Joiner, you can get this off ebay for < $2
• A multi-meter

Then, simply cut off the end of the old phone cable that had the end the plugs into the phone.  It will probably be a 4 wire cable.  Now, use your multimeter to identify which two wires are connected to the outer two BT-11 pins.  Then plug it into your phone jack and check you get around 50 volts DC from those two pins..   Once you’ve got the two wires that have power, cut off the other 2 wires and crimp the two powered wires to the two outside pins on the RJ45 plug.   See the above diagram.

Then, plug the RJ45 plug you crimped into the joiner you got off ebay, and label it, as you don’t want to be plugging in normal networking equipment by accident to this plug.

Step 3:

1. Connect the Y cable to your Fritz!Box and to your VDSL.  Check it works.
2. Connect the Y cable to your Analog line, using the adapter you made.  Your Fritz!Box is now able answer and make calls with your landline.

Now, your ready to configure Telephony on the FRITZ!Box.

Manual: http://www.avm.de/en/service/manuals/FRITZBox/Manual_FRITZBox_Fon_WLAN_7390.pdf

Configuration:

Start with Telephone numbers.  Here you should configure your fixed line, plus any SIP providers you want. I have added ippi and comfytel.   Also to note, that Snap! provides a SIP service, if you want.

Next, connect some phones to your Fritz!Box.  You can plugin your standard analoge phones into the FON1 and FON2 plugs.  You can connect your ISDN phones, you can connect any DECT wireless phones to the Fritz!Box, as their base station (your luck my vary), and you can connect your mobile phones to it, using the FRITZ!Box Fon app.  These will appear under telephone devices.  You should now be able to make a call.

Each device can have a default outgoing telephone number connected to it, and you can pre-select which phone number to make outgoing calls with, by dialing the ** prefix code.

Things you can do

1. Use a number of devices as phones in your home, including normal phones and your mobiles
2. Answer calls from skype, sip, landlines and internal numbers
3. Make free calls to skype and global sip inum’s.
4. Make low cost calls to overseas landlines using a sip provider
5. Make calls to local numbers via your normal phone line
6. Answer machine
7. Click to dail
8. Telephone book, including calling internal numbers
9. Wakeup calls
10. Send and receive faxes
11. Block calls
12. Call Diversion / Call through
13. Automatically select different providers when dialing different numbers
14. Set a device to use a specific number, or only to ring for calls for a specific number.
15. Set do not disturb on a device, based on time of day.
16. Connect your wireless DECT phones directly to the FRITZ!Box as a base station
17. See a call log of the calls you’ve made

Sip providers

ippi – http://www.ippi.fr – They allow free outgoing and incoming skype calls, plus a free global sip number

Country	Number
SIP	glenogilvie@ippi.fr
SIP numeric	889507473
iNum	+883510012028558

For outgoing skype calls with ippi, if your phone cannot dial email addresses, you need to add the skype contacts to the phone book on the ippi.com website under your account, and assign a short code, which you can then use from your phone.

Comfytel – http://www.comfytel.com/ – They provide cheap calling, but you have to pay them manually with paypal, and currently their skype gateway does not work.

iNum: 883510001220681
Internal number: 99982009943

 

Since screen shots are much nicer than words, below is a collection from my config for your reference.

Call Log:

Answer Phone:

Telephone book:

Alarm:

Fax:

Call Blocking:

Call through

Dialing rules

Telephone Devices

Dect configuration (I don’t have any compatible phones)

Telephone Numbers (fixed and SIP)

ComfyTel configuration:

IPPI configuration:

IPPI phone book, for calling skype numbers:

Fixed line configuration:

Line Settings:

time to blog again.

I have been a bit slack with my blogging and not posted much for a long time. This has been due to both working on lots of things, buying a house and a busy lifestyle.

I do however have a few things to blog about. So, in the coming days i will blog about auto_inst os testing, corporate patching, android tools, aucklug, raspberry pi, rdiff-backup, mulitseat Linux, the local riverside community centre, getting 10 laptops, which will run mageia, my cat gorse, gps tracking, house automation, Amazon AMIs and maybe some other stuff.

a portable, offline, distributed wiki setup

I’ve found recently, that I keep having need of somewhere to write stuff down and that works off line, with updates from different computers.

As I’ve been using git for this for code, and it works fine, thought I would see if I could apply it to a wiki. Turns out, it’s easy. Dokuwiki is a wiki that is packaged with Mandriva and happens to store it’s data in plain text files. It also does not store history in these files, so they stay nice, clean and easy to read with a text editor. They are also conveniently in just one standard place. So, with a minimal amount of work, I turned /var/lib/dokuwiki/pages/ into a git repository, added the pages to the repo, and volla. Now, I can push my wiki up to any remote git repo I want, or pull it from another computer. In my case, I have a main remote repo for git, so push it up to that main repo when I am online, then pull it down to all the computers I use.

Keeping my wiki up to date is now quite simple and easy, with git pull/push and the odd git commit. Merges will happen if I end up editing a page on two different computers before they both get to sync with my master git, but that should be fine as the wiki format is clean and simple text.

Booting an lvm snapshot of /

When it becomes time to upgrade Mandriva, or in fact, any other distro, I always have the same problem. I don’t want to do an upgrade because I can’t easily go back if the upgrade fails. So I usually end up creating a fresh install with a different root volume in LVM, then installing the same packages and copying configuration across, as I see fit. Good, but could be better.

I’ve also found I do keep wanting to try stuff, which would be nice to be able to roll back to a known state if things turn bad. So, recently I investigated how to create LVM snapshots of the root filesystem (/).

The procedure below works for me.. but if you don’t understand it, don’t try this on an important system.

Here is my basic howto:
The example below assumes your root volume is /dev/vg/rootlv. It assumes your /boot partition is a normal partition, not part of /.

1. Create a snapshot of / (which I usually do while it’s running, which means it’s not 100% clean). I create this with the following command.
lvcreate -n rootsnap1 -L 5G -s /dev/vg/rootlv
2. Mount the snapshot in /mnt/rootsnap1
mkdir /mnt/rootsnap1; mount /dev/vg/rootsnap1 /mnt/rootsnap1
3. Adjust snapshot to be able to boot:
perl -pi -e "s#/dev/vg/rootlv#/dev/vg/rootsnap1#" /mnt/rootsnap1/etc/fstab
rm /mnt/rootsnap1/etc/mtab
umount /mnt/rootsnap1
4. Create yourself a copy of initrd that has the new root volume in it. I do this by just editing the current one.
mkdir /boot/tt
cd /boot/tt
cat ../initrd.img | gunzip | cpio -i
perl -pi -e "s#/dev/vg/rootlv#/dev/vg/rootsnap1#" init
find . | cpio -H newc -o | gzip -9 > ../initrd-rootsnap1.img
cd /boot
rm -rf /boot/tt
5. Add the new root snapshot1 to grub’s menu.1st file, by copying your existing entry and changing the copied entry:
initrd.img to initrd-rootsnap1.img
root=/dev/vg/rootlv to root=/dev/vg/rootsnap1
vi /boot/grub/menu.1st
6. Reboot and choose new entry from grub.

Notes:
1. If the snapshot gets full, you’ll loose access to it and the volume it is a snapshot of, until you fix it. You can however, make it bigger at any time before it is full. It will never use more than the size of the original volume.
2. Snapshots can be merged* (fairly new feature) back into the base for that snapshot. This means that changes on a snapshot can be tried and if successful, can be merged back into the base. See lvconvert.
(*note, you don’t want to have the snapshot, or the base mounted at the time. lvconvert will delay until next reboot if you do. Reboot twice for it to completely free up the snapshot space used.)
3. If you take multiple snapshots, it is best to not do large amounts of writes to the base for the snapshots, as every block changed on the base will first have to be copied to every snapshot which affects write performance.

Readers: Comments are welcome, however I do moderate them, so they don’t appear on my blog immediately.