Podman

Podman is a daemonless container engine alternative to Docker. We don't support Docker in many of our environments as it grants trivial administrative control over the host the Docker daemon runs on. Podman on the other hand has the ability to run containers in user namespaces. This means that for every user name space in the kernel you create the processes within it will map to a new uid/gid range. For example, if you are root in your container, you will not be uid 0 outside the container, but instead you will be uid 4294000000.

We still believe that Singularity is the best option for running containerized workloads on our clustered based resources. Podman is a good option for developing the containers to be run via Singularity or building a deliverable for a funding agency. Please contact staff if you would like podman installed on a workstation or standalone server. More information on Podman running rootless.

Getting Started

To get started there are a few things that you need to configure.

First run the podman command. If it says command not found or that you get an ERRO like the one below about no subuid ranges please contact staff with the error and the host that you are using. We will need to do some steps to setup the host you want ready.

[username@zerus:~ ] $ podman
ERRO[0000] cannot find mappings for user username: No subuid ranges found for user "username" in /etc/subuid
Error: missing command 'podman COMMAND'
Try 'podman --help' for more information.

Storage

Containers are made up of layers for the image and these are stored in the graphroot setting of ~/.config/containers/storage.conf which by default will be in your home directory. With our home directories being available over NFS there is an issue[1] that due to the user name space mapping described above you will not be able to access your home directory when you are building the layers.

You need to update the graphroot setting to a local directory on the host. The file ~/.config/containers/storage.conf may not exist until you run podman the first time.

[storage]
  driver = "vfs"
  runroot = "/tmp/run-2174"
  graphroot = "/scratch1/username/.local/share/containers/storage"
...

GPUs

Running Podman with the local Nvidia GPUs requires some additional configuration steps that staff has to add to any individual host that runs Podman. This includes ensuring the nvidia-container-runtime package is installed.

For example you can run nvidia-smi from within the official Nvidia CUDA containers with a command like this:

$ podman run --rm --hooks-dir=/usr/share/containers/oci/hooks.d docker.io/nvidia/cuda nvidia-smi
Thu Apr 16 18:47:04 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.44       Driver Version: 440.44       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:03:00.0 Off |                  N/A |
| 22%   40C    P8    14W / 250W |    142MiB / 12212MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 00000000:04:00.0 Off |                  N/A |
| 22%   34C    P8    15W / 250W |      1MiB / 12212MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Example

To build your own image you can start from an example we have https://gitlab.umiacs.umd.edu/derek/gpudocker.

First clone the repository, change directory and build the image with podman.

git clone https://gitlab.umiacs.umd.edu/derek/gpudocker.git
cd gpudocker
podman build -t gpudocker .

Then you can run the test script to verify. Notice that we pass the local directory test as a path into the image so we can run a script. This can also be useful for your data output data as well as if you write anywhere else in the container it will not be available outside the container.

$ podman run --volume `pwd`/test:/mnt --hooks-dir=/usr/share/containers/oci/hooks.d gpudocker python3 /mnt/test_torch.py
GPU found 0: GeForce GTX 1080 Ti
tensor([[0.3479, 0.6594, 0.5791],
        [0.6065, 0.3415, 0.9328],
        [0.9117, 0.3541, 0.9050],
        [0.6611, 0.5361, 0.3212],
        [0.8574, 0.5116, 0.7021]])