Docker on LFS

abril 12, 2015

Docker on LFS - The hard way

The hard way

I know, docker is to make things simple, not hard. BUT I'm supposed to be the guy that solve problems, then I use LFS. This way I'm forced to learn if I want to make any thing work in my environment.

About LFS

LFS (www.linuxfromscratch.org) is not a linux distro, it is a book that instruct you how to build your own linux. I've using it for too many years as my O.S. After configured it's fully functional, but it's not for all people. It's for ones that want to know how things works inside, and are ready to pay the price for it. What price ?... well, let's go ahead with docker instalation and you will see.

My docker knowledge

I'm really new to docker game, until now I've no knowledge of go lang, and only searched a little about docker. After some reading I decided it's a nice tool and I need to know it more and test it in some of my projects. The docker project seems to be well documented, and the README file in source says some good tips to go ahead. so let's move on.

Installing docker

In some distros docker seems to be too simple to install, just:

wget -qO- https://get.docker.com/ | sh

Do not work for me, but give me tips.

One is the reference of instalation documents:
https://docs.docker.com/en/latest/installation/

It can help you, for me I did select "Binaries".
It is a nice tutorial, but lacks important info like kernel compilation features and real dependencies.
so, let's resume:

Dependency: linux kernel options:

Docker needs linux kernel 3.10+
Docker needs kernel with:
CONFIG_CGROUP_*
CONFIG_BRIDGE=m
CONFIG_AUFS_FS=m (patch from aufs3)
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_VETH=m
bridge-utils ( for CONFIG_BRIDGE kernel options)
DO NOT use built-in(eg: y), use modules(m) in kernel.

If you are using kernel option "CONFIG_IKCONFIG=y", you can see the current kernel options using:

cat /proc/config.gz | gunzip

zcat /proc/config.gz

Dependency: AUFS and AUFS-

AUFS(http://aufs.sf.net) is not a option in kernel, so you need to download and patch the kernel, it's not a hard task, only git clone it:
git clone git://git.code.sf.net/p/aufs/aufs3-standalone
and see the README file. It shows to you how to patch you kernel.
(use the command "patch -Np1 -i file.patch" in kernel source dir to apply path)

You will need the aufs-utils too:

git clone git://git.code.sf.net/p/aufs/aufs-util

If you do not make things right, this happens(about aufs):

$docker -D -d
...
WARN[0000] WARNING: Udev sync is not supported. This will lead to unexpected behavior, data loss and errors
DEBU[0000] devicemapper: udev sync support: false
...
FATA[0000] Error running DeviceCreate (CreatePool) dm_task_run failed

NOTE: do not forget aufs-util or you will fall in error (explained later)

Dependency: BRIDGE

kernel options CONFIG_BRIDGE and bridge-utils is mandatory.
If your bridge module (bridge.ko) is not correctly installed, the docker daemon will not start:

$ docker -D -d
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
WARN[0000] graphdriver aufs selected. Warning: your graphdriver directory /var/lib/docker already contains data managed by other graphdrivers: devicemapper
DEBU[0000] Using graph driver aufs
DEBU[0000] Migrating existing containers
DEBU[0000] Creating images graph
DEBU[0000] Restored 0 elements
DEBU[0000] Creating repository list
INFO[0000] +job init_networkdriver()
DEBU[0000] Creating bridge docker0 with network 172.17.42.1/16
DEBU[0000] setting bridge mac address = true
package not installed
INFO[0000] -job init_networkdriver() = ERR (1)
FATA[0000] package not installed

iptables failure:

You problally forget "CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m".

Success on run docker daemon

So, after this we can start de docker daemon:

INFO[0000] +job serveapi(unix:///var/run/docker.sock)
DEBU[0000] Using graph driver aufs
DEBU[0000] Migrating existing containers
DEBU[0000] Creating images graph
DEBU[0000] Restored 0 elements
DEBU[0000] Creating repository list
INFO[0000] +job init_networkdriver()
DEBU[0000] /sbin/iptables, [-C POSTROUTING -t nat -s 172.17.42.1/16 ! -o docker0 -j MASQUERADE]
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
DEBU[0000] Registering GET, /images/search
DEBU[0000] Registering GET, /containers/json
DEBU[0000] Registering GET, /containers/{name:.*}/export
DEBU[0000] Registering GET, /containers/{name:.*}/json
DEBU[0000] Registering GET, /containers/{name:.*}/attach/ws
DEBU[0000] Registering GET, /containers/{name:.*}/stats
DEBU[0000] Registering GET, /info
DEBU[0000] Registering GET, /version
DEBU[0000] Registering GET, /images/viz
DEBU[0000] Registering GET, /containers/ps
DEBU[0000] Registering GET, /containers/{name:.*}/top
DEBU[0000] Registering GET, /images/{name:.*}/history
DEBU[0000] Registering GET, /images/{name:.*}/json
DEBU[0000] Registering GET, /_ping
DEBU[0000] Registering GET, /events
DEBU[0000] Registering GET, /images/json
DEBU[0000] Registering GET, /images/get
DEBU[0000] Registering GET, /images/{name:.*}/get
DEBU[0000] Registering GET, /containers/{name:.*}/changes
DEBU[0000] Registering GET, /containers/{name:.*}/logs
DEBU[0000] Registering GET, /exec/{id:.*}/json
DEBU[0000] Registering POST, /auth
DEBU[0000] Registering POST, /exec/{name:.*}/start
DEBU[0000] Registering POST, /exec/{name:.*}/resize
DEBU[0000] Registering POST, /images/create
DEBU[0000] Registering POST, /images/load
DEBU[0000] Registering POST, /images/{name:.*}/push
DEBU[0000] Registering POST, /containers/{name:.*}/start
DEBU[0000] Registering POST, /containers/{name:.*}/rename
DEBU[0000] Registering POST, /build
DEBU[0000] Registering POST, /containers/{name:.*}/unpause
DEBU[0000] Registering POST, /containers/{name:.*}/restart
DEBU[0000] Registering POST, /containers/{name:.*}/wait
DEBU[0000] Registering POST, /containers/{name:.*}/attach
DEBU[0000] Registering POST, /containers/{name:.*}/copy
DEBU[0000] Registering POST, /containers/{name:.*}/exec
DEBU[0000] Registering POST, /commit
DEBU[0000] Registering POST, /images/{name:.*}/tag
DEBU[0000] Registering POST, /containers/create
DEBU[0000] Registering POST, /containers/{name:.*}/kill
DEBU[0000] Registering POST, /containers/{name:.*}/pause
DEBU[0000] Registering POST, /containers/{name:.*}/stop
DEBU[0000] Registering POST, /containers/{name:.*}/resize
DEBU[0000] /sbin/iptables, [-D FORWARD -i docker0 -o docker0 -j DROP]
DEBU[0000] /sbin/iptables, [-C FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[0000] /sbin/iptables, [-C FORWARD -i docker0 ! -o docker0 -j ACCEPT]
DEBU[0000] /sbin/iptables, [-C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]
DEBU[0000] /sbin/iptables, [-t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[0000] /sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [-t nat -D PREROUTING -j DOCKER]
DEBU[0000] /sbin/iptables, [-t nat -D OUTPUT -j DOCKER]
DEBU[0000] /sbin/iptables, [-t nat -F DOCKER]
DEBU[0000] /sbin/iptables, [-t nat -X DOCKER]
DEBU[0000] /sbin/iptables, [-t nat -n -L DOCKER]
DEBU[0000] /sbin/iptables, [-t nat -N DOCKER]
DEBU[0000] /sbin/iptables, [-C -m addrtype --dst-type LOCAL]
DEBU[0000] /sbin/iptables, [-t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [-C -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8]
DEBU[0000] /sbin/iptables, [-t nat -A OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[0000] /sbin/iptables, [-t filter -n -L DOCKER]
DEBU[0000] Registering DELETE, /containers/{name:.*}
DEBU[0000] Registering DELETE, /images/{name:.*}
DEBU[0000] Registering OPTIONS,
DEBU[0000] docker group found. gid: 1004
DEBU[0000] /sbin/iptables, [-C FORWARD -o docker0 -j DOCKER]
INFO[0000] -job init_networkdriver() = OK (0)
INFO[0000] WARNING: mountpoint for memory not found

DEBU[0000] Restarting containers...
INFO[0000] docker daemon: 1.5.0 a8a31ef; execdriver: native-0.2; graphdriver: aufs
INFO[0000] +job acceptconnections()
INFO[0000] -job acceptconnections() = OK (0)
Note: if you have the warning:
WARN[0000] graphdriver aufs selected. Warning: your graphdriver directory /var/lib/docker already contains data managed by other graphdrivers: devicemapper

just erase the dir contents:
rm -fr /var/lib/docker/*

Using docker group

Just a remember, you need to create a group named docker, and add your user to this docker group, then re-login(can use sudo su - youruser), just something like(as root):

$ groupadd docker
$ usermod -aG docker youruser

then redo your login, or as normal user do:

$ sudo su - youruser

You can use groups to see if you currently on docker group

$ groups
users ... your other groups ... docker

If you don't do it, only can run docker as root, that is not a good thing. or as normal user will get the message:

FATA[0000] Post http:///var/run/docker.sock/v1.17/containers/create: dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?

HTTPS Problem

Then I follow the tutorial until it fails again :

INFO[0005] Get https://index.docker.io/v1/repositories/library/centos/images: x509: failed to load system roots and no roots provided

NOTE: I did solve the problem, but I'm not sure yet of how, so here I'm posting the steps I took.

After some research, I understand that the docker error is because of problems of ca-certificates.
based on tianon commend on post:
https://github.com/docker/docker/issues/3825
I did run the commands in:
https://github.com/dram/crux/blob/master/ca-certificates/Pkgfile
that are:

# Description: Common CA certificates
# URL: http://packages.debian.org/sid/ca-certificates
# Packager: Xin Wang, dram dot wang at gmail dot com

name=ca-certificates
version=20111025
release=1
source=(http://ftp.us.debian.org/debian/pool/main/c/$name/${name}_${version}_all.deb)

build() {
    ar -x ${name}_${version}_all.deb
    tar zxvf ./data.tar.gz
    install -D -m 0755 usr/sbin/update-ca-certificates $PKG/usr/sbin/update-ca-certificates
    cp -a etc $PKG/
    mkdir -p $PKG/usr/share/ca-certificates/
    cp -a usr/share/ca-certificates/* $PKG/usr/share/ca-certificates/
    (
        cd $PKG/usr/share/ca-certificates/
        find . -name '*.crt' | sort | cut -b3-
    ) >$PKG/etc/ca-certificates.conf
}

But aparently still no success, anyway this post was closed in favor of issue 5157:

https://github.com/docker/docker/issues/5157
geokala user posted:
"For SLES11 the following seems to work (at least it gives me a new error that hg isn't present):

cat /etc/ssl/certs/*.pem > /etc/ssl/certs/ca-certificates.crt
sed -i -r '/^#.+/d' /etc/ssl/certs/ca-certificates.crt"

I did run and again(I think) it do not solved my problem. I'm not sure, it was 3A.M. so give me a break if it's not true.

There are many others programs in my LFS using ca-certificates and I don't have any other ca-certificates problems. So, maybe it is something related with go.

then I deep into go instalation from source(of couse):
$ git clone https://go.googlesource.com/go
read the README and docs in doc folder... nicely documented, then I have a functional go environment.
BUT despite the docker error is from "golang" my "go binary" do not have this https problem, only docker.

Then research again, and I found this:
http://blog.devtable.com/2014/01/using-docker-on-osx-with-private.html
It's exactly my problem! but on OSX, not LFS. anyway he explains how to fix, then let's try it.

We need some adaptations but it not hard.

After this docker works (at least the error changed)

Success on run

$ docker run -i -t ubuntu /bin/bash
Unable to find image 'ubuntu:latest' locally
511136ea3c5a: Pull complete
511136ea3c5a: Download complete
f3c84ac3a053: Download complete
a1a958a24818: Download complete
9fec74352904: Download complete
d0955f21bf24: Download complete
Status: Downloaded newer image for ubuntu:latest
FATA[0244] Error response from daemon: Cannot start container ac28ce91b6ce51da7b5b49c8a9e3711eed7ad5e25ceef3cb1f189c7b56a09486: failed to find the cgroup root

ooops... I forgot the correct cgroup mount(but it is in tutorial):https://github.com/tianon/cgroupfs-mount/blob/master/cgroupfs-mount

After correctly mount cgroups, restart the docker daemon but it do not solved:

ERRO[0010] [warning]: couldn't run auplink before unmount: exec: "auplink": executable file not found in $PATH ERRO[0010] [warning]: couldn't run auplink before unmount: exec: "auplink": executable file not found in $PATH

auplink ??? a new player ? researching...

$ locate auplink
/sources/aufs-util.git/auplink.

So, auplink is just part of aufs-util, that I forgot to install... let's do it.
but a make was not so ease:

$ make
cc -O -Wall -D_GNU_SOURCE -I./libau -DAUFHSM_CMD=\"/usr/bin/aufhsm\" -DMOUNT_CMD=\"/bin/mount\" -DUMOUNT_CMD=\"/bin/umount\" ver.c -o ver
ver.c:19:29: fatal error: linux/aufs_type.h: No such file or directory
#include
^
compilation terminated.
make: *** [ver] Error 1

after manually copy the aufs_type.h to /usr/include/linux it fails too:

$ make
cc -O -Wall -D_GNU_SOURCE -I./libau -DAUFHSM_CMD=\"/usr/bin/aufhsm\" -DMOUNT_CMD=\"/bin/mount\" -DUMOUNT_CMD=\"/bin/umount\" ver.c -o ver
In file included from ver.c:19:0:
/usr/include/linux/aufs_type.h:289:27: error: expected ‘:’, ‘,’, ‘;’, ‘}’ or ‘__attribute__’ before ‘*’ token
struct au_rdu_ent __user *e;
^
make: ** [ver] Erro 1

research...
http://comments.gmane.org/gmane.linux.file-systems.aufs.user/3241
"...No, don't copy the header files manually.
"make headers_install" modifies the header files for userspace and copy
them where you specify..."
...
"...Okay, got it. I didn't realize "make headers_install" puts them in
usr/include in the kernel tree..."
What ? make headers_install do not install headers? lets see:

$ find -name aufs_type.h
./usr/include/linux/aufs_type.h
./include/uapi/linux/aufs_type.h
$ diff ./usr/include/linux/aufs_type.h ./include/uapi/linux/aufs_type.h -q
Files ./usr/include/linux/aufs_type.h and ./include/uapi/linux/aufs_type.h differ
$ sudo cp usr/include/linux/aufs_type.h /usr/include/linux/ -v
“usr/include/linux/aufs_type.h” -> “/usr/include/linux/aufs_type.h”

then back to aufs-util dir:

$ cd -
/sources/aufs-util.git
$ make
...
$ sudo make install

Now it's done =D

restarting docker daemon

I'm worried about this, but let's go ahead:

...

INFO[0000] WARNING: mountpoint for memory not found
...

try to run:

$ docker -D run -i -t ubuntu /bin/bash
DEBU[0000] [hijack] End of stdout
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
FATA[0000] Error response from daemon: Cannot start container 4e0c441cc067a59ef769b98cbce99f983b734f30354d8cc2ce6c8aebf4f1ce66: mountpoint for devices not found

Docker daemon says:

Cannot start container 4e0c441cc067a59ef769b98cbce99f983b734f30354d8cc2ce6c8aebf4f1ce66: mountpoint for devices not found
INFO[0195] -job start(4e0c441cc067a59ef769b98cbce99f983b734f30354d8cc2ce6c8aebf4f1ce66) = ERR (1)
ERRO[0195] Handler for POST /containers/{name:.*}/start returned error: Cannot start container 4e0c441cc067a59ef769b98cbce99f983b734f30354d8cc2ce6c8aebf4f1ce66: mountpoint for devices not found
ERRO[0195] HTTP Error: statusCode=500 Cannot start container 4e0c441cc067a59ef769b98cbce99f983b734f30354d8cc2ce6c8aebf4f1ce66: mountpoint for devices not found

Let's look at CGROUP kernel config:

$ zcat /proc/config.gz | grep CONFIG_CGROUP_DEVICE
# CONFIG_CGROUP_DEVICE is not set

Another kernel change needed. This time I did recompile the kernel with all CONFIG_CGGROUP* and some CONFIG*CG* checked. This way the WARNING message is gone.

What next ?

FATA[0003] Error response from daemon: Cannot start container 7f031a5eeaa76518a8bb4c0ccbbdb3143280b52d99f65aaca3bc0db463838620: operation not supported

daemon:

ERRO[0305] Handler for POST /containers/{name:.*}/start returned error: Cannot start container 7f031a5eeaa76518a8bb4c0ccbbdb3143280b52d99f65aaca3bc0db463838620: operation not supported
ERRO[0305] HTTP Error: statusCode=500 Cannot start container 7f031a5eeaa76518a8bb4c0ccbbdb3143280b52d99f65aaca3bc0db463838620: operation not supported

some more research
https://github.com/docker/docker/issues/7246
Once more time:

$ zcat /proc/config.gz | grep VETH
# CONFIG_VETH is not set

again... change kernel config, recompile and try again.

this time I'm installing LXC too, but I don't know if it is really required in this context. I will search later:

sudo git clone https://github.com/lxc/lxc.git

VICTORY!

I'm exausted but the docker is running:

$ docker -D run -i -t ubuntu /bin/bash
root@d877dd7aed57:/#
root@12b316d2ff64:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"
root@d877dd7aed57:/# exit
exit
DEBU[0200] [hijack] End of stdout
DEBU[0200] End of CmdRun(), Waiting for hijack to finish.
$

Usefull links:

https://docs.docker.com/installation/binaries/
http://blog.tremily.us/posts/Docker/
https://github.com/dram/crux/blob/master/ca-certificates/Pkgfile
https://byteoutalife.wordpress.com/2013/12/30/building-docker-from-source-on-the-raspberry-pi/

Pesquisar este blog

Glaudiston Blog