Nexus/Apptainer - Revision history

Mbaney: Redirected page to Apptainer

2024-08-13T17:47:21Z

Redirected page to Apptainer

← Older revision		Revision as of 17:47, 13 August 2024
Line 1:		Line 1:
	Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular, the most typical setups require a daemon that has administrative level privileges that makes it not tenable. There has been a lot of work in this area but ultimately for HPC environments, [[Apptainer]] ~~is a solution that enables the capabilities of container workloads in multi-tenant environments.~~		#REDIRECT [[Apptainer]]

	The one consideration is that to create an image from a definition file you need to have administrative rights on the machine. For this reason you can't directly create Apptainer images on our supported systems. You can download or pull images from other repositories including the Docker repositories. If you do need to create images on our supported systems, please see [[Podman]].

	~~=Bind Mounts=~~
	~~Apptainer containers will not automatically mount data from the outside operating system other than your home directory. Users need to manually bind mounts for other file paths.~~

	~~<code>--bind /fs/nexus-scratch/username/project1:/mnt</code>~~

	~~In this scenario we are binding the directory outside the container <code>/fs/nexus-scratch/username/project1</code> to exist in the path <code>/mnt</code> inside the container.~~

	~~=Shared Containers=~~
	Portable images called Singularity Image Format or .sif files can be copied and shared. Nexus maintains some shared containers in <code>/fs/nexus-containers</code>. These are arranged by the application(s) that are installed.

	~~=GPUs=~~
	Nvidia has a very specific driver and libraries that are required to run CUDA programs. To ensure that all appropriate devices are created inside the container and that these libraries are made available in the container users need to use the <code>--nv</code> flag when instantiating their container(s).

	~~=Example=~~
	~~If you have the following example file in <code>/fs/nexus-scratch/username/apptainer</code>.~~

	~~<pre>~~
	~~#!/usr/bin/env python~~

	~~import torch;~~

	~~print(f'Torch cuda is available: {torch.cuda.is_available()}')~~
	~~print(f'Torch cuda number of devices: {torch.cuda.device_count()}')~~
	~~for g in range(torch.cuda.device_count()):~~
	~~print(f'Torch cuda device {g}: {torch.cuda.get_device_name(0)}')~~
	~~</pre>~~

	~~<pre>~~
	~~$ apptainer exec --bind /fs/nexus-scratch/username/apptainer:/mnt --nv /fs/nexus-containers/pytorch/pytorch_1.10.2+cu113.sif python3 /mnt/test.py~~
	~~Torch cuda is available: True~~
	~~Torch cuda number of devices: 1~~
	~~Torch cuda device 0: NVIDIA RTX A4000~~
	~~</pre>~~

Mbaney at 16:56, 13 August 2024

2024-08-13T16:56:05Z

← Older revision		Revision as of 16:56, 13 August 2024
Line 1:		Line 1:
	Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular, the most typical setups require a daemon that has administrative level privileges that makes it not tenable. There has been a lot of work in this area but ultimately for HPC environments, [[Apptainer]] is a solution that enables the capabilities of container workloads in multi-tenant environments.		Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular, the most typical setups require a daemon that has administrative level privileges that makes it not tenable. There has been a lot of work in this area but ultimately for HPC environments, [[Apptainer]] is a solution that enables the capabilities of container workloads in multi-tenant environments.

	The one consideration is that to create an image from a definition file you need to have administrative rights on the machine. For this reason you can't directly create Apptainer images on our supported systems. You can download or pull images from other repositories including the Docker repositories.		The one consideration is that to create an image from a definition file you need to have administrative rights on the machine. For this reason you can't directly create Apptainer images on our supported systems. You can download or pull images from other repositories including the Docker repositories. If you do need to create images on our supported systems, please see [[Podman]].

	=Bind Mounts=		=Bind Mounts=

Mbaney at 18:07, 29 January 2024

2024-01-29T18:07:02Z

← Older revision		Revision as of 18:07, 29 January 2024
Line 6:		Line 6:
	Apptainer containers will not automatically mount data from the outside operating system other than your home directory. Users need to manually bind mounts for other file paths.		Apptainer containers will not automatically mount data from the outside operating system other than your home directory. Users need to manually bind mounts for other file paths.

	<code>--bind /fs/nexus-scratch/~~derek~~/project1:/mnt</code>		<code>--bind /fs/nexus-scratch/username/project1:/mnt</code>

	In this scenario we are binding the directory outside the container <code>/fs/nexus-scratch/~~derek~~/project1</code> to exist in the path <code>/mnt</code> inside the container.		In this scenario we are binding the directory outside the container <code>/fs/nexus-scratch/username/project1</code> to exist in the path <code>/mnt</code> inside the container.

	=Shared Containers=		=Shared Containers=
Line 17:		Line 17:

	=Example=		=Example=
	If you have the following example file in <code>/fs/nexus-scratch/~~derek~~/apptainer</code>.		If you have the following example file in <code>/fs/nexus-scratch/username/apptainer</code>.

	<pre>		<pre>
Line 31:		Line 31:

	<pre>		<pre>
	$ apptainer exec --bind /fs/nexus-scratch/~~derek~~/apptainer:/mnt --nv /fs/nexus-containers/pytorch/pytorch_1.10.2+cu113.sif python3 /mnt/test.py		$ apptainer exec --bind /fs/nexus-scratch/username/apptainer:/mnt --nv /fs/nexus-containers/pytorch/pytorch_1.10.2+cu113.sif python3 /mnt/test.py
	Torch cuda is available: True		Torch cuda is available: True
	Torch cuda number of devices: 1		Torch cuda number of devices: 1
	Torch cuda device 0: NVIDIA RTX A4000		Torch cuda device 0: NVIDIA RTX A4000
	</pre>		</pre>

Mbaney at 19:25, 8 September 2023

2023-09-08T19:25:17Z

← Older revision		Revision as of 19:25, 8 September 2023
Line 1:		Line 1:
	Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular the most typical setups require a daemon that has administrative level privileges that makes it not tenable. There has been a lot of work in this area but ultimately for HPC environments, [[Apptainer]] is a solution that enables the capabilities of container workloads in multi-tenant environments.		Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular, the most typical setups require a daemon that has administrative level privileges that makes it not tenable. There has been a lot of work in this area but ultimately for HPC environments, [[Apptainer]] is a solution that enables the capabilities of container workloads in multi-tenant environments.

	The one consideration is that to create an image you need to have administrative rights on the machine. For this reason you can't directly create Apptainer images on our supported systems. You can download or pull images from other repositories including the Docker repositories.		The one consideration is that to create an image from a definition file you need to have administrative rights on the machine. For this reason you can't directly create Apptainer images on our supported systems. You can download or pull images from other repositories including the Docker repositories.

	=Bind Mounts=		=Bind Mounts=
Line 14:		Line 14:

	=GPUs=		=GPUs=

	Nvidia has a very specific driver and libraries that are required to run CUDA programs. To ensure that all appropriate devices are created inside the container and that these libraries are made available in the container users need to use the <code>--nv</code> flag when instantiating their container(s).		Nvidia has a very specific driver and libraries that are required to run CUDA programs. To ensure that all appropriate devices are created inside the container and that these libraries are made available in the container users need to use the <code>--nv</code> flag when instantiating their container(s).

	=Example=		=Example=

	If you have the following example file in <code>/fs/nexus-scratch/derek/apptainer</code>.		If you have the following example file in <code>/fs/nexus-scratch/derek/apptainer</code>.

Mbaney at 18:58, 8 September 2023

2023-09-08T18:58:12Z

← Older revision		Revision as of 18:58, 8 September 2023
Line 1:		Line 1:
	Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular the most typical setups require a daemon that has administrative level privileges that makes it not tenable. There has been a lot of work in this area but ultimately for HPC environments ~~Singularity or as it is now known~~ Apptainer is a solution that enables the capabilities of container workloads in multi-tenant environments.		Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular the most typical setups require a daemon that has administrative level privileges that makes it not tenable. There has been a lot of work in this area but ultimately for HPC environments, [[Apptainer]] is a solution that enables the capabilities of container workloads in multi-tenant environments.

	The one consideration is that to create an image you need to have administrative rights on the machine. For this reason you can't directly create Apptainer images on our supported systems. You can download or pull images from other repositories including the Docker repositories.		The one consideration is that to create an image you need to have administrative rights on the machine. For this reason you can't directly create Apptainer images on our supported systems. You can download or pull images from other repositories including the Docker repositories.
Line 11:		Line 11:

	=Shared Containers=		=Shared Containers=

	Portable images called Singularity Image Format or .sif files can be copied and shared. Nexus maintains some shared containers in <code>/fs/nexus-containers</code>. These are arranged by the application(s) that are installed.		Portable images called Singularity Image Format or .sif files can be copied and shared. Nexus maintains some shared containers in <code>/fs/nexus-containers</code>. These are arranged by the application(s) that are installed.

Line 20:		Line 19:
	=Example=		=Example=

	If you have the following example file in <code>/fs/nexus-scratch/derek/~~singularity~~</code>.		If you have the following example file in <code>/fs/nexus-scratch/derek/apptainer</code>.

	<pre>		<pre>
Line 34:		Line 33:

	<pre>		<pre>
	$ ~~singularity~~ exec --bind /fs/nexus-scratch/derek/~~singularity~~:/mnt --nv /fs/nexus-containers/pytorch/pytorch_1.10.2+cu113.sif python3 /mnt/test.py		$ apptainer exec --bind /fs/nexus-scratch/derek/apptainer:/mnt --nv /fs/nexus-containers/pytorch/pytorch_1.10.2+cu113.sif python3 /mnt/test.py
	Torch cuda is available: True		Torch cuda is available: True
	Torch cuda number of devices: 1		Torch cuda number of devices: 1
	Torch cuda device 0: NVIDIA RTX A4000		Torch cuda device 0: NVIDIA RTX A4000
	</pre>		</pre>

Derek at 14:15, 24 August 2022

2022-08-24T14:15:05Z

← Older revision		Revision as of 14:15, 24 August 2022
Line 13:		Line 13:

	Portable images called Singularity Image Format or .sif files can be copied and shared. Nexus maintains some shared containers in <code>/fs/nexus-containers</code>. These are arranged by the application(s) that are installed.		Portable images called Singularity Image Format or .sif files can be copied and shared. Nexus maintains some shared containers in <code>/fs/nexus-containers</code>. These are arranged by the application(s) that are installed.

			=GPUs=

			Nvidia has a very specific driver and libraries that are required to run CUDA programs. To ensure that all appropriate devices are created inside the container and that these libraries are made available in the container users need to use the <code>--nv</code> flag when instantiating their container(s).

			=Example=

			If you have the following example file in <code>/fs/nexus-scratch/derek/singularity</code>.

			<pre>
			#!/usr/bin/env python

			import torch;

			print(f'Torch cuda is available: {torch.cuda.is_available()}')
			print(f'Torch cuda number of devices: {torch.cuda.device_count()}')
			for g in range(torch.cuda.device_count()):
			print(f'Torch cuda device {g}: {torch.cuda.get_device_name(0)}')
			</pre>

			<pre>
			$ singularity exec --bind /fs/nexus-scratch/derek/singularity:/mnt --nv /fs/nexus-containers/pytorch/pytorch_1.10.2+cu113.sif python3 /mnt/test.py
			Torch cuda is available: True
			Torch cuda number of devices: 1
			Torch cuda device 0: NVIDIA RTX A4000
			</pre>

Derek: Created page with "Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular the most typical setups require a daemon that has administra..."

2022-08-24T14:09:01Z

Created page with "Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular the most typical setups require a daemon that has administra..."

New page

Running containers in a multi-tenant environment has a number of security considerations. While Docker is popular the most typical setups require a daemon that has administrative level privileges that makes it not tenable. There has been a lot of work in this area but ultimately for HPC environments Singularity or as it is now known Apptainer is a solution that enables the capabilities of container workloads in multi-tenant environments.

The one consideration is that to create an image you need to have administrative rights on the machine. For this reason you can't directly create Apptainer images on our supported systems. You can download or pull images from other repositories including the Docker repositories.

=Bind Mounts=
Apptainer containers will not automatically mount data from the outside operating system other than your home directory. Users need to manually bind mounts for other file paths.

<code>--bind /fs/nexus-scratch/derek/project1:/mnt</code>

In this scenario we are binding the directory outside the container <code>/fs/nexus-scratch/derek/project1</code> to exist in the path <code>/mnt</code> inside the container.

=Shared Containers=

Portable images called Singularity Image Format or .sif files can be copied and shared. Nexus maintains some shared containers in <code>/fs/nexus-containers</code>. These are arranged by the application(s) that are installed.