Podman

Podman is a daemonless container engine alternative to Docker. We don't support Docker in many of our environments as it grants trivial administrative control over the host the Docker daemon runs on. Podman on the other hand has the ability to run containers in user namespaces. This means that for every user name space in the kernel you create the processes within it will map to a new uid/gid range. For example, if you are root in your container, you will not be uid 0 outside the container, but instead you will be uid 4294000000.

We still believe that Apptainer is the best option for running containerized workloads on our clustered based resources. Podman is a good option for developing the containers to be run via Apptainer or building a deliverable for a funding agency. Please contact staff if you would like Podman installed on a workstation or standalone server. More information on Podman running rootless.

Getting Started

To get started there are a few things that you need to configure.

First, run the podman command. If it says command not found or you get an ERRO like the one below about no subuid ranges, and you are on a workstation or standalone (non-cluster) server, please contact staff with the error and the host that you are using. We will need to do some steps to setup the host you want ready.

$ podman
ERRO[0000] cannot find mappings for user username: No subuid ranges found for user "username" in /etc/subuid
Error: missing command 'podman COMMAND'
Try 'podman --help' for more information.

Storage

Containers are made up of layers for the image and these are stored in the graphroot setting of ~/.config/containers/storage.conf which by default will be in your home directory. With our home directories being available over NFS there is an issue that due to the user name space mapping described above you will not be able to access your home directory when you are building the layers.

You need to update the graphroot setting to a local directory on the host. The file ~/.config/containers/storage.conf may not exist until you run podman the first time, however you can manually create it.

[storage]
  driver = "vfs"
  graphroot = "/scratch0/username/.local/share/containers/storage"
...

When building larger images, it may fill up the default directory for imageCopyTmpDir (/var/tmp). If this happens, you will need to specify a different directory using the environment variable TMPDIR. For example:

export TMPDIR="/scratch0/example_tmp_directory"

GPUs

Running Podman with the local Nvidia GPUs requires some additional configuration steps that staff has to add to any individual workstation or standalone (non-cluster) server that runs Podman. This includes ensuring the nvidia-container-runtime package is installed.

For example you can run nvidia-smi from within the official Nvidia CUDA containers with a command like this, optionally replacing the tag for different CUDA versions/OS images:

$ podman run --rm --hooks-dir=/usr/share/containers/oci/hooks.d docker.io/nvidia/cuda:12.2.2-base-ubi8 nvidia-smi
Thu Apr 16 18:47:04 2020
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
+---------------------------------------------------------------------------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A6000               Off | 00000000:01:00.0 Off |                  Off |
| 30%   28C    P8               6W / 300W |      2MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

The full list of tags can be found at https://hub.docker.com/r/nvidia/cuda/tags.

Example

To build your own image you can start from an example we have https://gitlab.umiacs.umd.edu/derek/gpudocker.

First clone the repository, change directory and build the image with podman.

git clone https://gitlab.umiacs.umd.edu/derek/gpudocker.git
cd gpudocker
podman build -t gpudocker .

Then you can run the test script to verify. Notice that we pass the local directory test as a path into the image so we can run a script. This can also be useful for your data output data as well as if you write anywhere else in the container it will not be available outside the container.

$ podman run --volume `pwd`/test:/mnt --hooks-dir=/usr/share/containers/oci/hooks.d gpudocker python3 /mnt/test_torch.py
GPU found 0: GeForce GTX 1080 Ti
tensor([[0.3479, 0.6594, 0.5791],
        [0.6065, 0.3415, 0.9328],
        [0.9117, 0.3541, 0.9050],
        [0.6611, 0.5361, 0.3212],
        [0.8574, 0.5116, 0.7021]])

If you instead want to push modifications to this example to your own container registry such that you can pull the container image down later, please see the README.md located in the example repository itself.