Skip to content

TaskCreatingNewContainers

Creating containers

LXD allows us to create lightweight virtual machines, and combined with filesystems such as ZFS, provides several mechanisms to easily configure, backup, replicate and update them. Adding ansible to the mix makes this process even more simple.

To create a container in our current environment simply add the hostname, ip_address, and purpose to /etc/ansible/hosts and run the create-lxd-containers.yml

root@kb2018:/etc/ansible# nano hosts
...
redshirt ip_address=198.202.31.200 purpose="Disposable Ubuntu"
...
root@kb2018:/etc/ansible# ansible-playbook /etc/ansible/playbooks/create-lxd-containers.yml

The containers created have admin accounts and ssh keys installed. They have Isolated static ip addresses which can not reach the server directly. Getting onto the containers requires an ssh key and a password to escalate privileges. By default the containers are unprivileged which should minimize the security risks to the main server. They also have an os agnostic script to perform periodic updates.

Mechanisms

LXD provides images and profiles which define the disk storage, network and other configuration used to create the container. The profiles include cloud config however only the ubuntu image implements it. Most things work well except some don't or some things change and the updates or the updated images require some tweaking (resolved for instance because it wasn't broken and you had to work around what was already)

Container !Image/Profile notes

The fragments below are from my work to create other images that would work as well.

Ubuntu LTS (20.04)

My initial forays into the new LTS release are [StartUsingTheFwords here].

Previous Ubuntu LTS (18.04)

Ubuntu 18.04 is really well suited for LXD in that it comes stock with cloud init. This means that with a simple profile you can create a usable container pre seeded with admin accounts, static networking and an update script.

Creating a new ubuntu container using lxd

In our environment hosts do not tend to be temporary so they are not dynamically allocated. Containers are built at the static ip address at redshirt.suspectdevices.com and then reconfigured before being used.

root@bs2020:~# lxc image list kb2018:
+------------+--------------+--------+---------------------------------------------+--------+----------+-------------------------------+
|   ALIAS    | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 |  ARCH  |   SIZE   |          UPLOAD DATE          |
+------------+--------------+--------+---------------------------------------------+--------+----------+-------------------------------+
| ubuntu-lts | ae465acff89b | no     | ubuntu 18.04 LTS amd64 (release) (20180613) | x86_64 | 173.14MB | Jun 16, 2018 at 10:07pm (UTC) |
+------------+--------------+--------+---------------------------------------------+--------+----------+-------------------------------+
root@bs2020:~# lxc init kb2018:ubuntu-lts test18 -p susdev19 -p default
Creating test18
root@bs2020:~# lxc start test18 
root@bs2020:~# lxc exec test18 bash
root@test18:~# nano /etc/netplan/50-cloud-init.yaml 
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: no
      addresses: [198.202.31.216/25]
      gateway4: 198.202.31.129
      nameservers:
        search: [suspectdevices.com fromhell.com vpn]
        addresses: [198.202.31.141]
root@test18:~# netplan apply
root@test18:~# update.sh
root@test18:~# reboot
root@test18:~# root@bs2020:~# lxc list
+----------+---------+--------------------------------+------+------------+-----------+
|   NAME   |  STATE  |              IPV4              | IPV6 |    TYPE    | SNAPSHOTS |
+----------+---------+--------------------------------+------+------------+-----------+
....
+----------+---------+--------------------------------+------+------------+-----------+
| test18   | RUNNING | 198.202.31.216 (eth0)          |      | PERSISTENT | 0         |
+----------+---------+--------------------------------+------+------------+-----------+
root@bs2020:~#

lxd profile for suspectdevices

The profile for suspect devices is broken into three major parts. * Network configuration * System, User and Security Configuration * Disk Pools and Network Devices

Overriding the profiles network configuration

.... do this by hand and discuss the ansible update in progress....

    version: 1
    config:
      - type: physical
        name: eth0
        subnets:
          - type: static
            ipv4: true
            address: 198.202.31.200
            netmask: 255.255.255.128
            gateway: 198.202.31.129
            control: auto
      - type: nameserver
        address: 198.202.31.141

System, User and Security Configuration

(Network Configuration is Stubbed in)

root@kb2018:~# lxc profile show susdev19
config:
  user.network-config: |
    version: 1
    config:
      - type: physical
        name: eth0
        subnets:
          - type: static
            ipv4: true
            address: 198.202.31.200
            netmask: 255.255.255.128
            gateway: 198.202.31.129
            control: auto
      - type: nameserver
        address: 198.202.31.141
  user.user-data: |
    #cloud-config
    timezone: America/Vancouver
    users:
      - name: feurig
        passwd: "$6$2Pf0ittl$nl....VdI/FyCXtu."
        gecos: Donald Delmar Davis
        ssh-authorized-keys:
          - ssh-rsa AAAA....uj4SL don@annie
          - ssh-rsa AAA..... FMNNn don@haifisch.local
        groups: sudo,root,wheel
        shell: /bin/bash
      - name: joe
        passwd: "$6$o14Dp3u...pD3vLrS1vX."
        gecos: Joseph Wayne Dumoulin
        ssh-authorized-keys:
          - ssh-rsa AAAA...r6Y/ZePpr jdumoulin@nextit.com
        groups: sudo,root,wheel
        shell: /bin/bash
    manage_resolv_conf: false
    packages:
    - python
    package_update: true
    package_upgrade: true
    write_files:
    - path: /etc/systemd/resolved.conf
      permissions: '0644'
      owner: root:root
      content: |
        # resolved because that wasnt broken either
        [Resolve]
        DNS= 198.202.31.141 198.202.31.132 8.8.4.4
    - path: /usr/local/bin/update.sh
      permissions: '0774'
      owner: root:root
      content: |
        #!/bin/bash
        # update.sh for debian/ubuntu/centos  (copyleft) don@suspecdevices.com
        echo --------------------- begin updating `uname -n` ----------------------
        if [ -x "$(command -v apt-get)" ]; then
          apt-get update
          apt-get -y dist-upgrade
          apt-get -y autoremove
        fi
        if  [ -x "$(command -v yum)" ]; then
          echo yum upgrade.
          yum -y upgrade
        fi
        if  [ -x "$(command -v zypper)" ]; then
          echo zypper dist-upgrade.
          zypper -y dist-upgrade
        fi
        echo ======================#### done==========================
    runcmd:
    # fix stupid subtle things
    # sudo needs to be able to resolve itself to authenticate users
    # and the users are locked by default
    # cloud cart blanch accounts are inexcusable
    - sed -i "s/^127.0.0.1/#127.0.0.1/" /etc/hosts
    - echo 127.0.0.1 `hostname` localhost >>/etc/hosts
    - passwd joe -u
    - passwd feurig -u
    - userdel -f ubuntu
    - userdel -f centos
    - userdel -f opensuse
    #- netplan apply
    power_state:
       mode: reboot
       message: See You Soon...
       condition: True
description: Try to create a sane environment for cloud-init based operating systems
devices: {}
name: susdev19
used_by:

cloud - init and ubuntu but no where else.

When I ran up the lxc containers for operating systems that aren't ubuntu the magic profile doesn't work. And in fact cloud-init and the utilities that are native on Ubuntu are not installed on those images. (WTF??) So my first attempt (using the Debian 9 container was to add cloud-init cloud-utils and the other packages and then get export and reimport the container) which more or less failed miserably (because creating new containers from scratch isn't as simple as they say it is. :).

debian 9 (Works!)

It turns out I didn't need to export or import the image. I just needed to copy the lxc templates from a working ubuntu image and then modify metadata.yaml on the image while its running and publish the result. (this method is buried in the discussion | here)

... lxc create using images:debian/9 ....
... lxc start image and add cloud init and cloud utils ...
... copy templates and metadata data from working ubuntu ....
... link /etc/network/interfaces.d/50... -> /etc/network/interfaces ...
... delete /var/log/cloud cruft ...
... shutdown and lxc publish ....
root@bs2020:~# lxc publish kernigan --alias debian/9c
root@bs2020:~# lxc init debian/9c redshirt -p susdev19

This works well. More better documentation to follow.

Centos 7 (works)

[root@keynes ~]# cd /etc/sysconfig/
[root@keynes sysconfig]# cat network
NETWORKING=yes
HOSTNAME=LXC_NAME
[root@keynes sysconfig]# cd network-scripts/
[root@keynes network-scripts]# vi ifcfg-eth0 
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
HOSTNAME=LXC_NAME
NM_CONTROLLED=no
TYPE=Ethernet
PREFIX=25
IPADDR=198.202.31.220
MTU=
GATEWAY=198.202.31.129
[root@keynes network-scripts]# systemctl restart network
[root@keynes network-scripts]# nano /etc/resolv.conf 
bash: nano: command not found
[root@keynes network-scripts]# vi /etc/resolv.conf 
...
nameserver 198.202.31.141
search suspectdevices.com
...
[root@keynes network-scripts]# ping digithink.com
PING digithink.com (198.202.31.230) 56(84) bytes of data.
64 bytes from 198.202.31.230 (198.202.31.230): icmp_seq=1 ttl=64 time=0.441 ms
...
[root@keynes network-scripts]# cd
[root@keynes ~]# yum update
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
...
updates                                                                                                                                                       | 3.4 kB  00:00:00     
(1/4): extras/7/x86_64/primary_db                                                                                                                             | 156 kB  00:00:00     
(2/4): updates/7/x86_64/primary_db                                                                                                                            | 1.3 MB  00:00:00     
(3/4): base/7/x86_64/group_gz                                                                                                                                 | 166 kB  00:00:00     
(4/4): base/7/x86_64/primary_db                                                                                                                               | 6.0 MB  00:00:01     
No packages marked for update
[root@keynes ~]# yum install -y nano less
[root@keynes ~]# yum install -y cloud-init
[root@keynes ~]# yum install -y cloud-utils
[root@keynes ~]# yum install -y openssh-server
[root@keynes ~]# yum install -y sudo
[root@keynes ~]# cat >>/etc/sudoers.d/9_fix-centos-sudo  <<EOD
%sudo ALL=(ALL) ALL
centos  ALL = /usr/bin/su nobody
EOD
[root@keynes ~]# exit
... modify metadata.yaml ...
... copy templates ....
root@bs2020:~# lxc image delete centos/7c
root@bs2020:~# lxc publish keynes --alias centos/7c
Container published with fingerprint: a27609a23021f4577dfea987176fa942635d349b2e3be0e046118db88af4c56a
root@bs2020:~# 
root@bs2020:~# lxc launch centos/7c redshirt -p susdev19
Creating redshirt
Starting redshirt

Check the work....

haifisch:~ don$ ssh feurig@redshirt.suspectdevices.com
The authenticity of host 'redshirt.suspectdevices.com (198.202.31.200)' can't be established.
ECDSA key fingerprint is SHA256:ad0/DY7qDl9XKSl4lnjSq9jv63e18Nrr4IZjT0yu7Og.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'redshirt.suspectdevices.com,198.202.31.200' (ECDSA) to the list of known hosts.
[feurig@redshirt ~]$ sudo bash

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for feurig: 
[root@redshirt feurig]#

todo: look deeper into right way to sudo and cloud config. (ubuntu:ubuntu problem)

Fedora 29 (not picking up local cloud-init)

[root@kundara ~]# cat .bash_history 
ip address add 198.202.31.200/25 dev eth0
ip route add default via 198.202.31.129
ping digithink.com
dnf upgrade
dnf upgrade cloud-init
dnf install cloud-init
dnf install -y cloud-utils
dnf install -y nano less sudo
dnf install -y openssh-server
cat >>/etc/sudoers.d/9_fix-fedora-sudo  <<EOD
%sudo ALL=(ALL) ALL
fedora  ALL = /usr/bin/su nobody
EOD
dnf install -y network-scripts
echo "NOZEROCONF=yes" >> /etc/sysconfig/network
systemctl enable cloud-init
chkconfig --levels 2345 sshd on
chkconfig --levels 2345 network on
journalctl --vacuum-time=`date +%s`
shutdown -h now

OpenSuse 15.0 (Close -- some oddities)

Image comes up with no network.

ip address add 198.202.31.200/25 dev eth0
ip route add default via 198.202.31.129
cat >/etc/resolv.conf<<EOD
nameserver 198.202.31.141
nameserver 198.202.31.132
nameserver 8.8.8.8
search suspectdevices.com
EOD

Install the packages needed to work here.

zypper -y install nano sudo cloud-init
zypper -y install openssh
zypper -y dist-upgrade

systemctl enable cloud-init

sshd install masks the service as disabled.

systemctl unmask sshd
systemctl enable sshd

Sudo comes out of the box pretty insecurely configured.

cat> /etc/sudoers.d/9_fix_opensuse_sudo<<EOD
Defaults !targetpw
%sudo ALL=(ALL) ALL
opensuse  ALL = /usr/bin/su nobody
EOD

todo: (cloud init bugs) * figure out why hashed passwords don't work. (or is it just my long complicated password). * figure out why the default route isn't getting propagated.

updating running containers

The update script created by the profile can be easily executed on all running containers on both hosts with the following 2 lines of bash.

root@kb2018:~# for h in `lxc list bs2020: -c n --format csv ` ;do echo $h ;lxc exec bs2020:$h update.sh; done
root@kb2018:~# for h in `lxc list local: -c n --format csv ` ;do echo $h ;lxc exec local:$h update.sh; done