Ubuntu 16.04: Run containers with LXD/LXC

This article will describe installing LXD/LXC and running containers.

 

1 Install packages for LXD/LXC and ZFS

Install packages for LXD/LXC and ZFS with apt.

$ sudo apt install -y lxd lxd-client lxd-tools criu
$ sudo apt install -y zfsutils-linux
$ sudo modprobe zfs # or reboot machine

A criu package is for live migration. ZFS is used for containers store. ZFS should be used when there is a block device/volume for ZFS. If not using ZFS, a directory on filesystem is used.

2 Initialize LXD

Initialize LXD and lxd-bridge with interactive mode. "Address to bind LXD to" means which NIC is listened by LXD daemon.

$ sudo lxd init
Name of the storage backend to use (dir or zfs): zfs
Create a new ZFS pool (yes/no)? yes
Name of the new ZFS pool: tank
Would you like to use an existing block device (yes/no)? no
Size in GB of the new loop device (1GB minimum): 20
Would you like LXD to be available over the network (yes/no)? yes
Address to bind LXD to (not including port): 0.0.0.0
Port to bind LXD to (8443 recommended): 8443
Trust password for new clients:
Again:
Do you want to configure the LXD bridge (yes/no)? yes
Warning: Stopping lxd.service, but it can still be activated by:
  lxd.socket
LXD has been successfully configured.

Using –auto option can initialize LXD without interactive mode.

$ sudo lxd init --auto \
--storage-backend zfs \
--storage-pool tank \
--storage-create-loop 20 \
--network-address 0.0.0.0 \
--network-port 8443 \
--trust-password password

A case of dir is as below.

$sudo lxd init --auto \
     --storage-backend dir \
     --network-address 0.0.0.0 \
     --network-port 8443 \
     --trust-password password

lxd-bridge can be initialized without interactive mode.

$ sudo systemctl stop lxd-bridge
$ sudo systemctl --system daemon-reload
$ sudo su -c 'cat <<EOF > /etc/default/lxd-bridge
USE_LXD_BRIDGE="true"
LXD_BRIDGE="lxdbr0"
UPDATE_PROFILE="true"
LXD_CONFILE=""
LXD_DOMAIN="lxd"
LXD_IPV4_ADDR="10.202.80.1"
LXD_IPV4_NETMASK="255.255.255.0"
LXD_IPV4_NETWORK="10.202.80.1/24"
LXD_IPV4_DHCP_RANGE="10.202.80.2,10.202.80.254"
LXD_IPV4_DHCP_MAX="252"
LXD_IPV4_NAT="true"
LXD_IPV6_ADDR=""
LXD_IPV6_MASK=""
LXD_IPV6_NETWORK=""
LXD_IPV6_NAT="false"
LXD_IPV6_PROXY="false"
EOF
'
$ sudo systemctl enable lxd-bridge
$ sudo systemctl start lxd-bridge

3 Create containers

Create containers with lxc launch. '.' and '_' cannot be used to containers name.

$ # lxc launch <image> <countainer>
$ lxc launch ubuntu:16.04 ubuntu-16-04

A image of Debian 8 in images.linuxcontainers.org is as below.

images:debian/jessie/amd64

You can get image list with below command.

$ lxc image list <remote>:

The remote servers are images, ubuntu and ubuntu-daily by default. Image size is about 100MByte.

$ lxc image list images:
$ lxc image list ubuntu:
$ lxc image list ubuntu-daily:

3.1 Fedora 22/23 dnf causes 'Inappropriate ioctl for device'

A gpgcheck of dnf on Fedora 22/23 containers does not work.

# dnf install -y <pkg>
gpgme.GpgmeError: (7, 32870, 'Inappropriate ioctl for device')

A –nogpgcheck option can avoid this error.

# dnf install --nogpgcheck -y <pkg>

The tee command will fix this error according to RedHat bugzilla.

$ lxc exec images-fedora-23-amd64 -- dnf reinstall -y gzip | tee

Or run below command on containers.

$ lxc exec images-fedora-23-amd64 -- /bin/bash
# dnf reinstall -y gzip | tee

3.2 Debian jessie's installation of policykit-1 will be freezed

Because LXC or Debian jessie has a problem, installation of policykit-1 will be freezed.

# apt install -y policykit-1
<snip>
Setting up policykit-1 (0.105-8) ...
<hung>

Debian strech has no this problem. Replacing systemd to Debian stretch's systemd will avoid this problem temporarily but some package dependencies will be broken. So Debian jessie cannot be used till this problem is fixed.

4 Network configration

A default profile is used for containers network by default.

$ lxc profile list
default
docker
$ lxc profile show default
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic

4.1 bridged (Not recommended)

A bridged will construct private network. A lxd-bridge provides NAT with dnsmasq. If you use multiple nameservers for private network and internet, dnsmasq will be problem.

4.1.1 Multiple nameserver cannot resolve name

A below /etc/resolv.conf provide nameserver 192.168.11.2 for private network and nameserver 192.168.11.1 for internet.

$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by
# resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE
# OVERWRITTEN
nameserver 192.168.11.2
nameserver 192.168.11.1
search hiroom2.com

The dnsmasq will use /etc/resolv.conf, but one nameserver return REFUSE which means cannot resolve name, dnsmasq will not request next nameserver.

For avoiding this problem, server directive and resolv-file directive must be used in dnsmasq configuration.

$ cat /var/lib/lxd-bridge/dnsmasq.conf
server=/hiroom2.com/192.168.11.2
resolv-file=/var/lib/lxd-bridge/resolv.conf

A nameserver 192.168.11.1 for internet is in resolv.conf.

$ cat /var/lib/lxd-bridge/resolv.conf
nameserver 192.168.11.1

This dnsmasq.conf is set to LXD_CONFILE.

$ diff -uprN /etc/default/lxd-bridge{.org,}
--- /etc/default/lxd-bridge.org 2016-06-18 09:16:05.533082034 +0900
+++ /etc/default/lxd-bridge     2016-06-18 09:16:18.037062209 +0900
@@ -1,7 +1,7 @@
 USE_LXD_BRIDGE="true"
 LXD_BRIDGE="lxdbr0"
 UPDATE_PROFILE="true"
-LXD_CONFILE=""
+LXD_CONFILE="/var/lib/lxd-bridge/dnsmasq.conf"
 LXD_DOMAIN="lxd"
 LXD_IPV4_ADDR="10.202.80.1"
 LXD_IPV4_NETMASK="255.255.255.0"

If /etc/resolv.conf is generated by dhcp server information, you need to change dnsmasq.conf and resolv.conf when dhcp server information is changed.

4.1.2 Applying patch for auto generating LXD_CONFILE

I create patch. Sorry but this is not enough. After lxd init, recreate deb file with below command.

Install packages for building lxd.

$ sudo apt install -y devscripts
$ sudo apt-get build-dep -y lxd

Download lxd source code. If you could not download source code, please erase # before deb-src in /etc/apt/sources.list and run "sudo apt update".

$ mkdir lxd.ubuntu-16.04
$ cd lxd.ubuntu-16.04/
$ apt source lxd
$ cd lxd-2.0.2

Apply patch.

$ GITHUB=https://github.com/hiroom2/lxd-pkg-ubuntu/commit
$ wget ${GITHUB}/f9c429c8c7177ad1484cb2f6b3a17e5eeacf2442.patch
$ patch -p1 < f9c429c8c7177ad1484cb2f6b3a17e5eeacf2442.patch
$ echo "a.patch" | EDITOR=true dpkg-source --commit

Build and install lxd.

$ dpkg-buildpackage -us -uc
$ sudo dpkg -i ../*.deb

Commant out LXD_CONFILE and add LXD_GEN_CONFILE="true" in /etc/default/lxd-bridge. /var/run/lxd-bridge/dnsmasq.conf and /var/run/lxd-bridge/resolv.conf will be generate and used by lxd-bridge.

$ sudo sed -i 's/^LXD_CONFILE/#LXD_CONFILE/g' /etc/default/lxd-bridge
$ sudo su -c 'echo LXD_GEN_CONFILE="true" >> /etc/default/lxd-bridge'
$ sudo systemctl restart lxd-bridge

4.2 macvlan (Not recommended)

macvlan provides external access. But the connection between host machine and containers cannot be enabled.

$ lxc profile create vlan
$ cat <<EOF | lxc profile edit vlan
name: vlan
config: {}
description: VLAN profile
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: ens3
    type: nic
EOF

Apply this profile to containers which names ubuntu-16-04.

$ lxc profile apply ubuntu-16-04 vlan

MAC address can be checkd by lxc config show.

$ lxc config show ubuntu-16-04 | \
grep volatile.eth0.hwaddr: | \
awk '{ print $2 }'
00:16:3e:0e:f6:38

DHCP server can provide IP address to this MAC address.

$ lxc exec ubuntu-16-04 -- ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:16:3e:0e:f6:38
          inet addr:192.168.11.86  Bcast:192.168.11.255 Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe0e:f638/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:70 errors:0 dropped:0 overruns:0 frame:0
          TX packets:77 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:7192 (7.1 KB)  TX bytes:9830 (9.8 KB)

4.3 bridged (Recommended)

Create bridge interface before creating profile. This provides external access. And the connection between host machine and containers can be enabled.

Create profile which using br0.

$ lxc profile create bridge
$ cat <<EOF | lxc profile edit bridge
name: bridge
config: {}
description: Bridge profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
EOF

5 Export containers to image

Stop containers and export to image.

$ lxc stop ubuntu-16-04
$ lxc publish ubuntu-16-04 --alias ubuntu-16-04
Container published with fingerprint:
fdd934c336bbd506c2f4f068a872ccd7504564b50d27d3a168a4835b6cfe4e96

Image can be export to file.

$ lxc image export fdd934c336bbd506c2f4f068a872ccd7504564b50d27d3a168a4835b6cfe4e96

Exported image can be import on other machine.

$ lxc import fdd934c336bbd506c2f4f068a872ccd7504564b50d27d3a168a4835b6cfe4e96.tar.gz

6 Remote server configuration

NIC and port for LXD daemon listening is set when sudo lxc init. You can change NIC and port with below command.

$ lxc config set core.https_address 0.0.0.0:8443

You can change password with below command.

$ lxc config set core.trust_password password

Other machine can add this machine as remote server.

$ lxc remote add myimages 192.168.11.77:8443
Certificate fingerprint:
c2e967e5d788db7fbbd19e1f22b789b74313083094bf5585334fe3d15192ab6a
ok (y/n)? y
Admin password for myimages: # type password
Client certificate stored at server:  myimages

6.1 Download image from remote server

Download image from remote server like images, ubuntu, ubuntu-daily.

$ lxc image list myimages:
<snip>
...  b39919dbaf79 ...
<snip>
$ lxc launch myimages:b39919dbaf79 container-from-myimages
Creating container-from-myimages

6.2 Container migration

Install criu package local machine and remote server.

$ lxc move images-centos-7-amd64 myimages:
$ lxc list myimages:
<snip>
... images-centos-7-amd64 ...
<snip>