Golang w/SQLite3 + Docker Scratch Image

June 4th at 10:11am

While deploying a containerized application I made my first foray into docker scratch images. The application is written in Golang and leverages CGO to interact with SQLite databases which posed a small complication.

References

Dockerfile multi-stage build

Since a 'scratch' image is completely empty I used a multi-stage build to setup the environment for my Golang application. This allows me to leverage tools found in typical linux distributions to configure necessary files/directories while keeping the resulting application image trim and slim.

This is what my Dockerfile looks like:

# Stage 1: Use tools in Ubuntu to create an unprivileged user, get ca-certificates and make folders
FROM ubuntu:latest
RUN useradd -u 10001 scratchuser
RUN apt update && apt -y install ca-certificates && mkdir /persist && chown scratchuser:scratchuser /persist && mkdir /keys && chown scratchuser:scratchuser /keys && chmod -R 0700 /keys

# Stage 2: Setup my application image
FROM scratch
# Ensure the right directory structure exists with the right permissions
# Also ensure that ca-certificates are available for TLS
COPY --from=0 /etc/passwd /etc/passwd
COPY --chown=10001:10001 --from=0 /persist /persist
COPY --chown=10001:10001 --from=0 /keys /keys
COPY --from=0 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

ADD test-app /

USER 10001
CMD ["/test-app"]

With this dockerfile I can build the app, then build/tag/push the image to my registry:

go build && sudo docker build -t gitlab.domain.tld:4000/apps/test-app:latest . && sudo docker push gitlab.domain.tld:4000/apps/test-app:latest

A Problem...

While this builds successfully, when I try to run the application I get a strange error:

standard_init_linux.go:207: exec user process caused "no such file or directory"

There are plenty of examples of people running into this issue online, but none of the answers I found really helped me address the issue.

The Solution

After a seemingly never-ending troubleshooting cycle I found that my Golang application was not compiled correctly. In order to get a true 'static build' when using CGO (in my case to leverage the exceptional github.com/mattn/go-sqlite3 library) you need to tweak the build command a bit. This causes the build process to slow down substantially yet results in a more truly static binary than what you get with your typical go build:

CGO_ENABLED=1 GOOS=linux go build -a -ldflags '-linkmode external -extldflags "-static"' .

Using this command I get output that looks a little concerning, yet in my testing I have not run into issues:

# test-app
/tmp/go-link-732605171/000015.o: In function `unixDlOpen':
/home/user/go/pkg/mod/github.com/mattn/go-sqlite3@v1.10.0/sqlite3-binding.c:38461: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-732605171/000004.o: In function `_cgo_7e1b3c2abc8d_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:57: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

A Note on Golang Static Builds

When looking into how to statically build go binaries I found examples that worked without CGO that looked like this:

CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' .

This (of course) would not work for me since I needed to leverage CGO. My first attempt here was to simply change CGO_ENABLED=1 and see what happened. Unfortunately, this was not enough to get the application to run successfully in my production environment. Instead I had to add additional flags to -ldflags in the form of -linkmode external. This helped me successfully deploy my appliction to production:

CGO_ENABLED=1 GOOS=linux go build -a -ldflags '-linkmode external -extldflags "-static"' .