4–5 Minute Detailed Overview: Docker-lite / Conductor

Target Duration: 4–5 minutes (~650–850 spoken words)
Goal: Deliver an end-to-end architectural explanation covering low-level kernel mechanics, storage layering, and with a heavy primary focus on virtual networking and inter-container communication. Plant deep-dive hooks along the way.

🎙️ Spoken Script & Section Walkthrough

To explain the complete architecture, I like to break it down into four stages: how images are built, how containers are launched and isolated, how networking is set up, and how a real user request flows through the entire system.

Stage 1: Image Building — Build Time

Everything starts with a build file containing instructions like FROM, COPY, and RUN.

When I run conductor build, it first processes the base OS from the FROM instruction, such as Debian, and downloads a minimal base root filesystem if it isn't already available.

Each subsequent instruction creates a new filesystem layer:

• For a COPY command, I copy the required files from the build context into the new layer.
• For a RUN command, such as installing packages with apt, I mount the previous image layers as read-only lower layers using OverlayFS and create a writable upper layer. I then start a temporary isolated container using that filesystem and execute the command inside it.
• Any files created or modified by the command are written to the OverlayFS upper layer. Once the command finishes, I preserve those changes as the new image layer. This ensures that the build command doesn't modify the host filesystem or the previous immutable layers.

I also use content-based caching for the layers. So when I rebuild an image, if a build step and its inputs haven't changed, Conductor reuses the existing layer instead of executing the step again.

(Hook: Deep dive into Topic 03: OverlayFS & Layers)

Stage 2: Container Launch — Run Time

When I run conductor run <image-name> <container-name>, Conductor takes the immutable image layers created during the build and mounts them as read-only lower layers using OverlayFS.

It then creates a fresh writable upper layer for that particular container. This means multiple containers can share the same underlying image layers while each container gets its own filesystem changes.

Next, Conductor starts the container using Linux namespaces:

• I use namespaces to isolate process IDs, mount points, networking, and the hostname, so the process inside the container sees an environment that is separated from the host.
• For PID isolation, I create a new PID namespace and fork a child process into it. That child becomes PID 1 inside the container. I also mount a private /proc so processes inside the container see the appropriate process view.
• For resource control, I use cgroups v2. For example, I can place the container into a cgroup with a CPU limit corresponding to 50% of a CPU core via cpu.max.

(Hook: Deep dive into Topic 01: Namespaces and Topic 02: cgroups v2)

Stage 3: Networking — Plumbing

Initially, the container's network namespace is isolated from the host, so I connect it to the host network using a virtual Ethernet pair (veth pair), which behaves like a virtual network cable:

• One end (veth-host) stays in the host network namespace.
• The other end (veth-guest) is moved into the container's network namespace.

I assign private IP addresses to the two ends and configure the default routing gateway.

To give the container internet access:

1. I enable IP forwarding (net.ipv4.ip_forward=1) in the Linux kernel.
2. I configure an iptables NAT MASQUERADE rule on the host, allowing outbound packets from the container to be translated and forwarded through the host's physical network interface.

I also support port forwarding for services running inside the container. For example, an incoming connection to host port 3000 is translated using DNAT and forwarded to port 8080 inside the container.

(Hook: Deep dive into Topic 05: Virtual Networking & veth and Topic 06: iptables Routing)

Stage 4: End-to-End Request Flow (Microservice Validation)

To test the complete architecture, I built a two-container microservice:

• Container 1 runs a Python Flask frontend.
• Container 2 runs a C-based counter service.

The request flow:

1. An external user sends a request to http://<host-ip>:3000.
2. The packet reaches the host, where the iptables DNAT rule translates the destination and forwards it through the veth interface to the Flask container.
3. The Flask application receives the request on its internal port.
4. To fulfill the page load, Flask sends an internal HTTP request over the private container network to the C counter service.
5. The C service receives the request, increments its counter, and sends the updated value back to Flask.
6. Flask constructs the HTML response and returns it through the network stack to the host and back to the user's browser.

This verifies the complete stack: container networking, iptables, network namespaces, the application layer, and communication between two isolated containers.