Skip to content

FIRST IMPRESSIONS:

Creating LXD Container with Static IP (and Docker Profile)We want to create a docker capable LXD container using an existing bridge with a static ip and zfs. Then we want to install docker and test it. We will make a copy of this container once the admin users have been added so that we wont have to replicate these tasks. Our security model requires ssh keys to log in AND passwords to escalate privileges.

The first thing we learned is that LXC and LXD are pretty different beasts and that while lxc with lxc-templates is a straightforward way to create containers that act a lot like regular old hardware LXD brings on all of its we love the mother fucking cloud baggage. Major differences had to do with user mapping on the containers files created by root on the host were mapped to nobody on the container, making it really difficult to set up home directories etc. (for a workaround to this see https://stackoverflow.com/questions/33377916/migrating-lxc-to-lxd) The second was the way that the network is initialized with the assumption that LXD would be providing the bridge and the context.

First Attempt and zfs/bridge setup

  • create zfs container and bridge as before

    root@bs2020:~# lxd init ... create new zfs pool and use all of /dev/sdd1 do not configure bridge ... root@bs2020:~# dpkg-reconfigure -p medium lxd ... no yes br1 ... use existing bridge... root@bs2020:~# lxc launch ubuntu:16.04 franklin -p default -p docker root@bs2020:~# lxc stop franklin root@bs2020:~# passwd -l ubuntu -R /var/lib/lxd/containers/franklin.zfs/rootfs passwd: user 'ubuntu' does not exist root@bs2020:~# cd /var/lib/lxd/containers/franklin.zfs/rootfs/ root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# cat ~feurig/passwd.add>>etc/passwd root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# cat ~feurig/shadow.add>>etc/shadow root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# tar -xzvf ~feurig/fnj.tgz home/feurig ... home/joe/.ssh/authorized_keys root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# usermod -R /var/lib/lxd/containers/franklin.zfs/rootfs -G sudo,root joe root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# usermod -R /var/lib/lxd/containers/franklin.zfs/rootfs -G sudo,root feurig root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# groupadd -R /var/lib/lxd/containers/franklin.zfs/rootfs -g 1001 feurig root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# groupadd -R /var/lib/lxd/containers/franklin.zfs/rootfs -g 1002 joe root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# cat <>/var/lib/lxd/containers/franklin.zfs/rootfs/etc/resolvconf/resolv.conf.d/base

    dns-nameserver 198.202.31.132 198.202.31.141 8.8.8.8 nameserver 198.202.31.132 198.202.31.141 8.8.8.8 eod root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# sed -i 's/^iface eth0/#iface eth0/' /var/lib/lxd/containers/franklin.zfs/rootfs/etc/network/interfaces root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# cat <>/var/lib/lxd/containers/franklin.zfs/rootfs/etc/network/interfaces iface eth0 inet static address 198.202.31.201/25 gateway 198.202.31.129 dns-nameservers 198.202.31.132 198.202.31.141 8.8.8.8 dns-search suspectdevices.com digithink.com eod2 root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# lxc start franklin

  • try to log in to instance over the network..... FAIL

  • unlike lxc's ubuntu:16.04, lxd's ubuntu:16.04 has all of the cloud cruft . That and all of the modifications to the containers directory was rootsquashed (rendering it useless).

    root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# lxc exec franklin bash root@franklin:~# nano /etc/network/interfaces root@franklin:~# cat /etc/network/interfaces

    This file describes the network interfaces available on your system

    and how to activate them. For more information, see interfaces(5).

    The loopback network interface

    auto lo iface lo inet loopback

    Source interfaces

    Please check /etc/network/interfaces.d before changing this file

    as interfaces may have been defined in /etc/network/interfaces.d

    See LP: #1262951

    source /etc/network/interfaces.d/*.cfg iface eth0 inet static address 198.202.31.201/25 gateway 198.202.31.129 dns-nameservers 198.202.31.132 198.202.31.141 8.8.8.8 dns-search suspectdevices.com digithink.com

  • first thought: remove all of the cloud crap...

    root@franklin:~# apt-get remove cloud* ... The following packages will be REMOVED: cloud-guest-utils cloud-init cloud-initramfs-copymods cloud-initramfs-dyn-netconf 0 upgraded, 0 newly installed, 4 to remove and 0 not upgraded. After this operation, 1682 kB disk space will be freed. Do you want to continue? [Y/n] Y ... root@franklin:~# nano /etc/network/interfaces ... add auto eth0 root@franklin:~# reboot root@franklin:~# root@bs2020:/var/lib/lxd/containers/franklin.zfs/rootfs# lxc list +----------+---------+-----------------------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +----------+---------+-----------------------+------+------------+-----------+ | franklin | RUNNING | 198.202.31.201 (eth0) | | PERSISTENT | 0 | +----------+---------+-----------------------+------+------------+-----------+

  • second thought: Fuck that! Make it work!

Second attempt

(create LXD profile for suspect devices development).

root@bs2020:~# lxc stop franklin 
root@bs2020:~# lxc delete franklin
root@bs2020:~# lxc profile create susdev
root@bs2020:~# lxc profile edit susdev
...
  • repeat until you have a working system that can be logged into remotely
  • create docker container container

    root@bs2020:~# lxc profile show susdev config: user.network_mode: link-local user.user-data: | #cloud-config timezone: America/Vancouver users: - name: feurig passwd: "... SUBSTITUTE REAL PASSWORD HASH HERE ...." gecos: Donald Delmar Davis ssh-authorized-keys: - ssh-rss ... SUBSTITUTE REAL KEY HERE ... don@viscious groups: sudo,root shell: /bin/bash - name: joe passwd: "... SUBSTITUTE REALPASSWORD HASH HERE ...." gecos: Joseph Wayne Dumoulin ssh-authorized-keys: - ssh-rss ...SUBSTITUTE REAL KEY HERE... jdumoulin@joeslaptop groups: sudo,root shell: /bin/bash manage_resolv_conf: true resolv_conf: nameservers: ['198.202.31.141', '198.202.31.132', '8.8.8.8'] searchdomains: - suspectdevices.com - digithink.com domain: suspectdevices.com options: rotate: true timeout: 1 write_files: # Set static IP address could not get this to work the "right" way - path: /etc/network/interfaces permissions: '0644' owner: root:root content: | auto lo iface lo inet loopback auto eth0 # change this after first instantiation iface eth0 inet static address 198.202.31.200 broadcast 198.202.31.255 netmask 255.255.255.128 gateway 198.202.31.129 dns-nameservers 198.202.31.141 198.202.31.132 8.8.8.8 runcmd: # sudo needs to be able to resolve itself to authenticate users # and the users are locked by default - sed -i "s/^127.0.0.1/#127.0.0.1/" /etc/hosts - echo 127.0.0.1 hostname localhost >>/etc/hosts - passwd joe -u - passwd feurig -u description: Try to create a sane environment for cloud-init based operating systems devices: eth0: name: eth0 nictype: bridged parent: br1 type: nic name: susdev root@bs2020:~#

    root@bs2020:~# lxc list +--------+---------+-----------------------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +--------+---------+-----------------------+------+------------+-----------+ | test13 | RUNNING | 198.202.31.200 (eth0) | | PERSISTENT | 0 | +--------+---------+-----------------------+------+------------+-----------+ root@bs2020:~# lxc init ubuntults16 franklin -p susdev -p docker Creating franklin root@bs2020:~# lxc start franklin root@bs2020:~# lxc exec franklin bash root@franklin:~# tail -2 /etc/shadow feurig:<>:17453:0:99999:7::: joe:<>:17453:0:99999:7::: root@franklin:~# nano /etc/network/interfaces root@franklin:~# cat /etc/network/interfaces auto lo iface lo inet loopback auto eth0

    change this after first instantiation

    iface eth0 inet static address 198.202.31.201 broadcast 198.202.31.255 netmask 255.255.255.128 gateway 198.202.31.129 dns-nameservers 198.202.31.141 198.202.31.132 8.8.8.8 root@franklin:~# cat /etc/hosts

    127.0.0.1 localhost

    The following lines are desirable for IPv6 capable hosts

    ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts 127.0.0.1 franklin localhost root@franklin:~# reboot root@bs2020:~# lxc list +----------+---------+-----------------------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +----------+---------+-----------------------+------+------------+-----------+ | franklin | RUNNING | 198.202.31.201 (eth0) | | PERSISTENT | 0 | +----------+---------+-----------------------+------+------------+-----------+ | test13 | RUNNING | 198.202.31.200 (eth0) | | PERSISTENT | 0 | +----------+---------+-----------------------+------+------------+-----------+ root@bs2020:~#

References

  • http://www.whiteboardcoder.com/2016/04/cloud-init-nocloud-with-url-for-meta.html
  • https://stgraber.org/2016/03/11/lxd-2-0-blog-post-series-012/
  • https://github.com/lxc/lxd/blob/master/doc/cloud-init.md
  • http://www.mattjarvis.org.uk/post/lxd-openstack-cloudinit-pt1/
  • https://sdgsystems.com/blog/understanding-and-using-lxc-and-lxd
  • http://cloudinit.readthedocs.io/en/latest/topics/examples.html
  • http://cloudinit.readthedocs.io/en/latest/topics/debugging.html