TaskCreatingNewContainers
Creating containers
LXD allows us to create lightweight virtual machines, and combined with filesystems such as ZFS, provides several mechanisms to easily configure, backup, replicate and update them. Adding ansible to the mix makes this process even more simple.
To create a container in our current environment simply add the hostname, ip_address, and purpose to /etc/ansible/hosts and run the create-lxd-containers.yml
root@kb2018:/etc/ansible# nano hosts
...
redshirt ip_address=198.202.31.200 purpose="Disposable Ubuntu"
...
root@kb2018:/etc/ansible# ansible-playbook /etc/ansible/playbooks/create-lxd-containers.yml
The containers created have admin accounts and ssh keys installed. They have Isolated static ip addresses which can not reach the server directly. Getting onto the containers requires an ssh key and a password to escalate privileges. By default the containers are unprivileged which should minimize the security risks to the main server. They also have an os agnostic script to perform periodic updates.
Mechanisms
LXD provides images and profiles which define the disk storage, network and other configuration used to create the container. The profiles include cloud config however only the ubuntu image implements it. Most things work well except some don't or some things change and the updates or the updated images require some tweaking (resolved for instance because it wasn't broken and you had to work around what was already)
Container !Image/Profile notes
The fragments below are from my work to create other images that would work as well.
Ubuntu LTS (20.04)
My initial forays into the new LTS release are [StartUsingTheFwords here].
Previous Ubuntu LTS (18.04)
Ubuntu 18.04 is really well suited for LXD in that it comes stock with cloud init. This means that with a simple profile you can create a usable container pre seeded with admin accounts, static networking and an update script.
Creating a new ubuntu container using lxd
In our environment hosts do not tend to be temporary so they are not dynamically allocated. Containers are built at the static ip address at redshirt.suspectdevices.com and then reconfigured before being used.
root@bs2020:~# lxc image list kb2018:
+------------+--------------+--------+---------------------------------------------+--------+----------+-------------------------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | SIZE | UPLOAD DATE |
+------------+--------------+--------+---------------------------------------------+--------+----------+-------------------------------+
| ubuntu-lts | ae465acff89b | no | ubuntu 18.04 LTS amd64 (release) (20180613) | x86_64 | 173.14MB | Jun 16, 2018 at 10:07pm (UTC) |
+------------+--------------+--------+---------------------------------------------+--------+----------+-------------------------------+
root@bs2020:~# lxc init kb2018:ubuntu-lts test18 -p susdev19 -p default
Creating test18
root@bs2020:~# lxc start test18
root@bs2020:~# lxc exec test18 bash
root@test18:~# nano /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
eth0:
dhcp4: no
addresses: [198.202.31.216/25]
gateway4: 198.202.31.129
nameservers:
search: [suspectdevices.com fromhell.com vpn]
addresses: [198.202.31.141]
root@test18:~# netplan apply
root@test18:~# update.sh
root@test18:~# reboot
root@test18:~# root@bs2020:~# lxc list
+----------+---------+--------------------------------+------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------+---------+--------------------------------+------+------------+-----------+
....
+----------+---------+--------------------------------+------+------------+-----------+
| test18 | RUNNING | 198.202.31.216 (eth0) | | PERSISTENT | 0 |
+----------+---------+--------------------------------+------+------------+-----------+
root@bs2020:~#
lxd profile for suspectdevices
The profile for suspect devices is broken into three major parts. * Network configuration * System, User and Security Configuration * Disk Pools and Network Devices
Overriding the profiles network configuration
.... do this by hand and discuss the ansible update in progress....
version: 1
config:
- type: physical
name: eth0
subnets:
- type: static
ipv4: true
address: 198.202.31.200
netmask: 255.255.255.128
gateway: 198.202.31.129
control: auto
- type: nameserver
address: 198.202.31.141
System, User and Security Configuration
(Network Configuration is Stubbed in)
root@kb2018:~# lxc profile show susdev19
config:
user.network-config: |
version: 1
config:
- type: physical
name: eth0
subnets:
- type: static
ipv4: true
address: 198.202.31.200
netmask: 255.255.255.128
gateway: 198.202.31.129
control: auto
- type: nameserver
address: 198.202.31.141
user.user-data: |
#cloud-config
timezone: America/Vancouver
users:
- name: feurig
passwd: "$6$2Pf0ittl$nl....VdI/FyCXtu."
gecos: Donald Delmar Davis
ssh-authorized-keys:
- ssh-rsa AAAA....uj4SL don@annie
- ssh-rsa AAA..... FMNNn don@haifisch.local
groups: sudo,root,wheel
shell: /bin/bash
- name: joe
passwd: "$6$o14Dp3u...pD3vLrS1vX."
gecos: Joseph Wayne Dumoulin
ssh-authorized-keys:
- ssh-rsa AAAA...r6Y/ZePpr jdumoulin@nextit.com
groups: sudo,root,wheel
shell: /bin/bash
manage_resolv_conf: false
packages:
- python
package_update: true
package_upgrade: true
write_files:
- path: /etc/systemd/resolved.conf
permissions: '0644'
owner: root:root
content: |
# resolved because that wasnt broken either
[Resolve]
DNS= 198.202.31.141 198.202.31.132 8.8.4.4
- path: /usr/local/bin/update.sh
permissions: '0774'
owner: root:root
content: |
#!/bin/bash
# update.sh for debian/ubuntu/centos (copyleft) don@suspecdevices.com
echo --------------------- begin updating `uname -n` ----------------------
if [ -x "$(command -v apt-get)" ]; then
apt-get update
apt-get -y dist-upgrade
apt-get -y autoremove
fi
if [ -x "$(command -v yum)" ]; then
echo yum upgrade.
yum -y upgrade
fi
if [ -x "$(command -v zypper)" ]; then
echo zypper dist-upgrade.
zypper -y dist-upgrade
fi
echo ======================#### done==========================
runcmd:
# fix stupid subtle things
# sudo needs to be able to resolve itself to authenticate users
# and the users are locked by default
# cloud cart blanch accounts are inexcusable
- sed -i "s/^127.0.0.1/#127.0.0.1/" /etc/hosts
- echo 127.0.0.1 `hostname` localhost >>/etc/hosts
- passwd joe -u
- passwd feurig -u
- userdel -f ubuntu
- userdel -f centos
- userdel -f opensuse
#- netplan apply
power_state:
mode: reboot
message: See You Soon...
condition: True
description: Try to create a sane environment for cloud-init based operating systems
devices: {}
name: susdev19
used_by:
cloud - init and ubuntu but no where else.
When I ran up the lxc containers for operating systems that aren't ubuntu the magic profile doesn't work. And in fact cloud-init and the utilities that are native on Ubuntu are not installed on those images. (WTF??) So my first attempt (using the Debian 9 container was to add cloud-init cloud-utils and the other packages and then get export and reimport the container) which more or less failed miserably (because creating new containers from scratch isn't as simple as they say it is. :).
debian 9 (Works!)
It turns out I didn't need to export or import the image. I just needed to copy the lxc templates from a working ubuntu image and then modify metadata.yaml on the image while its running and publish the result. (this method is buried in the discussion | here)
... lxc create using images:debian/9 ....
... lxc start image and add cloud init and cloud utils ...
... copy templates and metadata data from working ubuntu ....
... link /etc/network/interfaces.d/50... -> /etc/network/interfaces ...
... delete /var/log/cloud cruft ...
... shutdown and lxc publish ....
root@bs2020:~# lxc publish kernigan --alias debian/9c
root@bs2020:~# lxc init debian/9c redshirt -p susdev19
This works well. More better documentation to follow.
Centos 7 (works)
[root@keynes ~]# cd /etc/sysconfig/
[root@keynes sysconfig]# cat network
NETWORKING=yes
HOSTNAME=LXC_NAME
[root@keynes sysconfig]# cd network-scripts/
[root@keynes network-scripts]# vi ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
HOSTNAME=LXC_NAME
NM_CONTROLLED=no
TYPE=Ethernet
PREFIX=25
IPADDR=198.202.31.220
MTU=
GATEWAY=198.202.31.129
[root@keynes network-scripts]# systemctl restart network
[root@keynes network-scripts]# nano /etc/resolv.conf
bash: nano: command not found
[root@keynes network-scripts]# vi /etc/resolv.conf
...
nameserver 198.202.31.141
search suspectdevices.com
...
[root@keynes network-scripts]# ping digithink.com
PING digithink.com (198.202.31.230) 56(84) bytes of data.
64 bytes from 198.202.31.230 (198.202.31.230): icmp_seq=1 ttl=64 time=0.441 ms
...
[root@keynes network-scripts]# cd
[root@keynes ~]# yum update
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
...
updates | 3.4 kB 00:00:00
(1/4): extras/7/x86_64/primary_db | 156 kB 00:00:00
(2/4): updates/7/x86_64/primary_db | 1.3 MB 00:00:00
(3/4): base/7/x86_64/group_gz | 166 kB 00:00:00
(4/4): base/7/x86_64/primary_db | 6.0 MB 00:00:01
No packages marked for update
[root@keynes ~]# yum install -y nano less
[root@keynes ~]# yum install -y cloud-init
[root@keynes ~]# yum install -y cloud-utils
[root@keynes ~]# yum install -y openssh-server
[root@keynes ~]# yum install -y sudo
[root@keynes ~]# cat >>/etc/sudoers.d/9_fix-centos-sudo <<EOD
%sudo ALL=(ALL) ALL
centos ALL = /usr/bin/su nobody
EOD
[root@keynes ~]# exit
... modify metadata.yaml ...
... copy templates ....
root@bs2020:~# lxc image delete centos/7c
root@bs2020:~# lxc publish keynes --alias centos/7c
Container published with fingerprint: a27609a23021f4577dfea987176fa942635d349b2e3be0e046118db88af4c56a
root@bs2020:~#
root@bs2020:~# lxc launch centos/7c redshirt -p susdev19
Creating redshirt
Starting redshirt
Check the work....
haifisch:~ don$ ssh feurig@redshirt.suspectdevices.com
The authenticity of host 'redshirt.suspectdevices.com (198.202.31.200)' can't be established.
ECDSA key fingerprint is SHA256:ad0/DY7qDl9XKSl4lnjSq9jv63e18Nrr4IZjT0yu7Og.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'redshirt.suspectdevices.com,198.202.31.200' (ECDSA) to the list of known hosts.
[feurig@redshirt ~]$ sudo bash
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
[sudo] password for feurig:
[root@redshirt feurig]#
todo: look deeper into right way to sudo and cloud config. (ubuntu:ubuntu problem)
Fedora 29 (not picking up local cloud-init)
[root@kundara ~]# cat .bash_history
ip address add 198.202.31.200/25 dev eth0
ip route add default via 198.202.31.129
ping digithink.com
dnf upgrade
dnf upgrade cloud-init
dnf install cloud-init
dnf install -y cloud-utils
dnf install -y nano less sudo
dnf install -y openssh-server
cat >>/etc/sudoers.d/9_fix-fedora-sudo <<EOD
%sudo ALL=(ALL) ALL
fedora ALL = /usr/bin/su nobody
EOD
dnf install -y network-scripts
echo "NOZEROCONF=yes" >> /etc/sysconfig/network
systemctl enable cloud-init
chkconfig --levels 2345 sshd on
chkconfig --levels 2345 network on
journalctl --vacuum-time=`date +%s`
shutdown -h now
OpenSuse 15.0 (Close -- some oddities)
Image comes up with no network.
ip address add 198.202.31.200/25 dev eth0
ip route add default via 198.202.31.129
cat >/etc/resolv.conf<<EOD
nameserver 198.202.31.141
nameserver 198.202.31.132
nameserver 8.8.8.8
search suspectdevices.com
EOD
Install the packages needed to work here.
zypper -y install nano sudo cloud-init
zypper -y install openssh
zypper -y dist-upgrade
systemctl enable cloud-init
sshd install masks the service as disabled.
systemctl unmask sshd
systemctl enable sshd
Sudo comes out of the box pretty insecurely configured.
cat> /etc/sudoers.d/9_fix_opensuse_sudo<<EOD
Defaults !targetpw
%sudo ALL=(ALL) ALL
opensuse ALL = /usr/bin/su nobody
EOD
todo: (cloud init bugs) * figure out why hashed passwords don't work. (or is it just my long complicated password). * figure out why the default route isn't getting propagated.
updating running containers
The update script created by the profile can be easily executed on all running containers on both hosts with the following 2 lines of bash.
root@kb2018:~# for h in `lxc list bs2020: -c n --format csv ` ;do echo $h ;lxc exec bs2020:$h update.sh; done
root@kb2018:~# for h in `lxc list local: -c n --format csv ` ;do echo $h ;lxc exec local:$h update.sh; done