CBCB Software Modules: Difference between revisions

From Cbcb
Jump to navigation Jump to search
No edit summary
(→‎Setup for non-interactive shells: -- Remove Torque reference)
(18 intermediate revisions by 7 users not shown)
Line 1: Line 1:
Starting in the Spring of 2015, communal CBCB Bioinformatics Software has been installed using GNU Modules.
Starting in the Spring of 2015, communal CBCB Bioinformatics Software has been installed using GNU Modules.
== Common Modules ==
== Common Modules ==


Line 7: Line 6:


To see see what modules are available:
To see see what modules are available:
  $ module avail
  bash$ module avail 2>&1 | less
or
tcsh> module avail |& less


To add a module to your environment, use <code>module add</code>:
To add a module to your environment, use <code>module add</code>:
Line 19: Line 20:
  /cbcb/sw/RedHat-7-x86_64/common/local/samtools/0.1.19/bin/samtools
  /cbcb/sw/RedHat-7-x86_64/common/local/samtools/0.1.19/bin/samtools


== Personal Modules ==
== User Modules ==


We have created some scripts to assist you with installing software and modules to your personal directory in <code>/cbcb/sw.</code>. We recommend that you use these scripts because this will help us share installed software among other CBCB users.
We have created some scripts to assist you with installing software and modules to your user directory in <code>/cbcb/sw</code>. We recommend that you use these scripts because this will help us share installed software among other CBCB users.


Run this initialization script to set up your directory structure:
Run this initialization script to set up your directory structure:
Line 56: Line 57:
=== Installing Software ===
=== Installing Software ===


Use the <code>install_package.sh</code> script to automatically compile software which uses <code>./configure; make; make install</code>. The advantage to using this install script is that it will:
Use the <code>install_package.sh</code> script to automatically compile software that uses <code>./configure; make; make install</code>. The advantage to using this install script is that it will:
* use the standardized directory structure for personal software installation, and
* use the standardized directory structure for personal software installation, and
* '''''automatically''''' create a module file for the software in your <code>modules</code> directory!
* '''''automatically''''' create a module file for the software in your <code>modules</code> directory!




For full details, read the <code>README</code> placed in your home directory. It's also available in the [https://gitlab.umiacs.umd.edu/cbcb/cbcb-sw/blob/master/templates/user_readme.md  cbcb-sw Gitlab repository].
For full details, read the <code>README</code> placed in your personal directory (<code>/cbcb/sw/RedHat-7-x86_64/users/<username></code>). It's also available in the [https://gitlab.umiacs.umd.edu/cbcb/cbcb-sw/blob/master/templates/user_readme.md  cbcb-sw Gitlab repository].
 
=== Listing All Modules ===
 
The command <code>module avail</code> only lists modulefiles that appear in directories that have been added to your <code>$MODULEPATH</code> environment variable (either by hand or via <code>module use</code> command).
To see all modulefiles that are available for your use in both the Common Modules and the User Modules:


<pre>
cat /cbcb/sw/RedHat-7-x86_64/common/all_modules.txt
</pre>
or see [http://cbcb.umd.edu/~lmendelo/cbcb_modules/all_modules.txt here].


== Setup for non-interactive shells ==
== Setup for non-interactive shells ==


Modules are already configured for interactive shells, but to use modules with non-interactive shells, add the following to your ~/.bashrc:
Modules are already configured for interactive shells, but to use modules with non-interactive shells, add the following to your ~/.bashrc:
  . /usr/share/Modules/init/bash
  $ source /usr/share/Modules/init/${SHELL}
  . /etc/profile.d/ummodules.sh
  bash$ source /etc/profile.d/ummodules.sh


For more information, see the [https://wiki.umiacs.umd.edu/umiacs/index.php/Modules UMIACS wiki].
For more information, see the [https://wiki.umiacs.umd.edu/umiacs/index.php/Modules UMIACS wiki].
Line 78: Line 89:


List available modules:
List available modules:
  module avail
  $ module avail
 
Read the description of a module:
$ module whatis blast
 
Read the help text for a module:
$ module help blast


Add a module to your environment:
Add a module to your environment:
  module add blast
  $ module add blast
 
Add a specific version to your environment:
$ module add blast/2.2.31


List loaded modules:
List loaded modules:
  module list
  $ module list


Remove a module from your environment:
Remove a module from your environment:
  module rm blast
  $ module rm blast


Remove all modules from your environment:
Remove all modules from your environment:
  module purge
  $ module purge


Add a path to search for available modules:
Add a path to search for available modules:
  module use /cbcb/sw/RedHat-7-x86_64/users/lmendelo/modules
  $ module use /cbcb/sw/RedHat-7-x86_64/users/lmendelo/modules


See what changes a module makes to your environment:
See what changes a module makes to your environment:
  module show Python2/common/2.7.9
  $ module show Python2/common/2.7.9
 
== Maintainers ==
 
=== Creating a new release ===
 
=== Upgrading R ===
 
There are a couple scripts to hopefully make upgrading R easier.
Let us say that you are installing a new version of R and R libraries.
If that is (for example) version 3.2.1, then a set of commands might include:
 
> cd $MOD/R/common
> cp 3.2.0 3.2.1
 
Now edit 3.2.1 to include the new version and change anything of interest.
 
> mkdir $STOW/Rext/3.2.1
> module add R/common/3.2.1
 
Assuming that you edited the $MOD 3.2.1 file to tell it to load R version 3.2.1, then
$(which R) will show the new version and the module command will set the R_LIBS
variable to a new, empty directory.
> cd $STOW/R/common
> ./update_R.sh 3.2.0_list
 
This little shell script should call Bioconductor's biocLite() once for each
R library which was installed in 3.2.0.
 
(Adapted from /cbcb/sw/RedHat-7-x86_64/common/local/Rext/README.md)


== Contact ==
== Contact ==
Line 102: Line 152:
Brought to you by:
Brought to you by:


* [mailto:abelew@umiacs.umd.edu Trey Abelew]
* [mailto:abelew@umiacs.umd.edu Trey Belew]
* [mailto:keith@umiacs.umd.edu Keith Hughitt]
* [mailto:keith@umiacs.umd.edu Keith Hughitt]
* [mailto:lmendelo@umiacs.umd.edu Lee Mendelowitz]
* [mailto:welles@umiacs.umd.edu Welles Robinson]
* [mailto:hiren@umiacs.umd.edu Hiren  Karathia]
* [mailto:schrist1@umd.edu Steve Christensen]
* [mailto:spatkar@umiacs.umd.edu Sushant Patkar]
 


== See Also ==
== See Also ==
[https://wiki.umiacs.umd.edu/umiacs/index.php/Modules UMIACS wiki]
* [https://wiki.umiacs.umd.edu/umiacs/index.php/Modules UMIACS wiki]
[https://gitlab.umiacs.umd.edu/cbcb/cbcb-sw Project on GitLab]
* [https://gitlab.umiacs.umd.edu/cbcb/cbcb-sw Repository on GitLab]
[https://docs.google.com/presentation/d/1UgKtnjcqHlpLZU79hGXfkgvNtse-ksMQ_gJMFCVLKRw/edit?usp=sharing Slides on Google Drive]
* [https://docs.google.com/presentation/d/1UgKtnjcqHlpLZU79hGXfkgvNtse-ksMQ_gJMFCVLKRw/edit?usp=sharing CBCB Software slides on Google Drive]

Revision as of 19:45, 16 November 2018

Starting in the Spring of 2015, communal CBCB Bioinformatics Software has been installed using GNU Modules.

Common Modules

CBCB Software modules are already configured for interactive shells on Red Hat 7 machines - no additional setup is required. The module files are installed in the following location:

/cbcb/sw/RedHat-7-x86_64/common/modules/release/latest

To see see what modules are available:

bash$ module avail 2>&1 | less

or

tcsh> module avail |& less

To add a module to your environment, use module add:

$ module add samtools/0.1.19

Note that you can also specify the software name without the version:

$ module add samtools

Now samtools has been added to your environment:

$ which samtools
/cbcb/sw/RedHat-7-x86_64/common/local/samtools/0.1.19/bin/samtools

User Modules

We have created some scripts to assist you with installing software and modules to your user directory in /cbcb/sw. We recommend that you use these scripts because this will help us share installed software among other CBCB users.

Run this initialization script to set up your directory structure:

$ /cbcb/sw/RedHat-7-x86_64/common/scripts/init_cbcb_sw_user.sh

This will create a directory for you:

$ tree /cbcb/sw/RedHat-7-x86_64/users/<username>
/cbcb/sw/RedHat-7-x86_64/users/<username>
├── local
├── modules
│   └── <username>
│       └── env
├── module_template
├── README
├── scripts
│   ├── copy_module_template.sh
│   ├── init_install_vars.sh
│   └── install_package.sh
└── src
  • Use the src directory for storing and compiling source code.
  • Use the local directory as the installation prefix.
  • Use the modules directory to store your modulefiles.

See #Installing Software below.

To make use of your personal module file directory, add the following to your ~/.bashrc:

module use /cbcb/sw/RedHat-7-x86_64/users/<username>/modules


Installing Software

Use the install_package.sh script to automatically compile software that uses ./configure; make; make install. The advantage to using this install script is that it will:

  • use the standardized directory structure for personal software installation, and
  • automatically create a module file for the software in your modules directory!


For full details, read the README placed in your personal directory (/cbcb/sw/RedHat-7-x86_64/users/<username>). It's also available in the cbcb-sw Gitlab repository.

Listing All Modules

The command module avail only lists modulefiles that appear in directories that have been added to your $MODULEPATH environment variable (either by hand or via module use command). To see all modulefiles that are available for your use in both the Common Modules and the User Modules:

cat /cbcb/sw/RedHat-7-x86_64/common/all_modules.txt

or see here.

Setup for non-interactive shells

Modules are already configured for interactive shells, but to use modules with non-interactive shells, add the following to your ~/.bashrc:

$ source /usr/share/Modules/init/${SHELL}
bash$ source /etc/profile.d/ummodules.sh

For more information, see the UMIACS wiki.

GNU Modules Cheatsheet

Ask for help:

module --help

List available modules:

$ module avail

Read the description of a module:

$ module whatis blast

Read the help text for a module:

$ module help blast

Add a module to your environment:

$ module add blast

Add a specific version to your environment:

$ module add blast/2.2.31

List loaded modules:

$ module list

Remove a module from your environment:

$ module rm blast

Remove all modules from your environment:

$ module purge

Add a path to search for available modules:

$ module use /cbcb/sw/RedHat-7-x86_64/users/lmendelo/modules

See what changes a module makes to your environment:

$ module show Python2/common/2.7.9

Maintainers

Creating a new release

Upgrading R

There are a couple scripts to hopefully make upgrading R easier. Let us say that you are installing a new version of R and R libraries. If that is (for example) version 3.2.1, then a set of commands might include:

> cd $MOD/R/common
> cp 3.2.0 3.2.1

Now edit 3.2.1 to include the new version and change anything of interest.

> mkdir $STOW/Rext/3.2.1
> module add R/common/3.2.1

Assuming that you edited the $MOD 3.2.1 file to tell it to load R version 3.2.1, then $(which R) will show the new version and the module command will set the R_LIBS variable to a new, empty directory.

> cd $STOW/R/common
> ./update_R.sh 3.2.0_list

This little shell script should call Bioconductor's biocLite() once for each R library which was installed in 3.2.0.

(Adapted from /cbcb/sw/RedHat-7-x86_64/common/local/Rext/README.md)

Contact

Brought to you by:


See Also