Toaster:UMD Condor Configuration

Disk Config

Disk space on terpcondor was snagged by umounting /tmp and setting it under the management of lvm.

Create physical volume and volume group.

[root@terpcondor root]# vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume group

[root@terpcondor root]# pvcreate /dev/sda5
pvcreate -- physical volume "/dev/sda5" successfully created

[root@terpcondor root]# vgcreate internal-disk /dev/sda5
vgcreate -- INFO: using default physical extent size 32 MB
vgcreate -- INFO: maximum logical volume size is 2 Terabyte
vgcreate -- doing automatic backup of volume group "internal-disk"
vgcreate -- volume group "internal-disk" successfully created and activated

Re-create tmp space.

[root@terpcondor root]# lvcreate -L2G -ntmp internal-disk
lvcreate -- doing automatic backup of "internal-disk"
lvcreate -- logical volume "/dev/internal-disk/tmp" successfully created

[root@terpcondor root]# mke2fs -j /dev/internal-disk/tmp 
mke2fs 1.32 (09-Nov-2002)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
262144 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
16 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Create condor volume, this will store software and configs

[root@terpcondor root]# lvcreate -L2G -ncondor internal-disk
lvcreate -- doing automatic backup of "internal-disk"
lvcreate -- logical volume "/dev/internal-disk/condor" successfully created

[root@terpcondor root]# mke2fs -j /dev/internal-disk/condor 
mke2fs 1.32 (09-Nov-2002)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
262144 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
16 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 31 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Now config home space in /export

[root@terpcondor root]# lvcreate -L2G -nhomes internal-disk
lvcreate -- doing automatic backup of "internal-disk"
lvcreate -- logical volume "/dev/internal-disk/homes" successfully created

[root@terpcondor root]# mke2fs -j /dev/internal-disk/homes 
mke2fs 1.32 (09-Nov-2002)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
262144 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
16 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Condor work directory (normally this just goes in /var/condor, but since this install is 100% relocatable, it's going in /export

[root@terpcondor root]# lvcreate -ncondor-var -L2G internal-disk
lvcreate -- doing automatic backup of "internal-disk"
lvcreate -- logical volume "/dev/internal-disk/condor-var" successfully created

[root@terpcondor root]# mke2fs -j /dev/internal-disk/condor-var
mke2fs 1.32 (09-Nov-2002)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
262144 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
16 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Add to /etc/fstab

/dev/internal-disk/tmp     /tmp                 ext3    defaults        1 2
/dev/internal-disk/condor  /export/condor       ext3    defaults        1 2
/dev/internal-disk/condor-var    /export/condor-var     ext3    defaults        1 2
/dev/internal-disk/homes    /export/homes       ext3    defaults        1 2

Condor Base Install

Condor will be installed in /export/condor/condor-6.6.10 with a symlink created to /export/condor/condor. All scripts and such will use the symlink to aid in an easier upgrade. To make upgrades a truly copy binaries, the condor_config sits in /export/condor/condor_config. The environment CONDOR_CONFIG should be set so that condor can find this file.

• Installation Log

In ~condor/condor-6.6.10

./condor_install
Would you like to do a full installation of Condor? [yes] 
...
...
Are you planning to setup Condor on multiple machines? [yes] 
Will all the machines share files via a file server? [yes] no
...
...
Have you installed a release directory already? [no] 

Where would you like to install the Condor release directory?
[/usr/local/condor] /export/condor/condor-6.6.10
That directory doesn't exist, should I create it now? [yes] 
Installing a release directory into /export/condor/condor-6.6.10 ... 
...
...
[root@terpcondor.umiacs.umd.edu] root@umiacs.umd.edu

What is the full path to a mail program that understands "-s" means
you want to specify a subject? [/bin/mail] 
...
...
Do all of the machines in your pool from your domain ("umiacs.umd.edu")
share a common filesystem? [no] 

Configuring each machine to be in its own filesystem domain.

Do all of the users across all the machines in your domain have a unique
UID (in other words, do they all share a common passwd file)? [no] 

Configuring each machine to be in its own uid domain.
...
...
Enable Java Universe support? [yes] 

I wasn't able to find a valid JVM.

Please enter the full path to the JVM, or "none" to leave unconfigured:
/opt/j2sdk1.4.2/bin/java

You entered: /opt/j2sdk1.4.2/bin/java
Is that right? [no] yes

Checking to see if you have a Sun JVM...yes.

Using JVM /opt/j2sdk1.4.2/bin/java for Java universe support.
...
...
Shall I create links in some other directory? [yes] 

Where should I install these files?
[/usr/local/bin] /usr/bin
...
...
What is the full hostname of the central manager?
[terpcondor.umiacs.umd.edu] 
...
...
You have a "condor" user on this machine.  Do you want to put all the
Condor directories in /export/homes/condor? [yes] no

Do you want to put all the Condor directories in
/export/condor/condor-6.6.10/home? [yes] no

Where do you want the Condor directories?
/export/condor-var

Creating all necessary Condor directories ... done.
...
...
Should I put a "condor_config.local" file in /export/condor-var?
[yes] 
Creating config files in "/export/condor-var" ... done.

Configuring global condor config file ... done.
Created /export/condor/condor-6.6.10/etc/condor_config.
...
...
Setting up terpcondor.umiacs.umd.edu as your central manager

What name would you like to use for this pool?  This should be a
short description (20 characters or so) that describes your site.
For example, the name for the UW-Madison Computer Science Condor
Pool is: "UW-Madison CS".  This value is stored in your central
manager's local config file as "COLLECTOR_NAME", if you decide to
change it later.  (This shouldn't include any " marks).
UM College Park
...
...
Should I put in a soft link from /export/homes/condor/condor_config to
/export/condor/condor-6.6.10/etc/condor_config [yes] 

Installing links for public binaries into /usr/bin ... done.

Post-Install cleanup

To make condor a little more machine and install independant, we moved the condor_config to /export/condor/condor_config

[root@terpcondor condor]# cd ~condor
[root@terpcondor condor]# ln -s /export/condor/condor_config 
[root@terpcondor condor]# cd /export/condor
[root@terpcondor condor]# ln -s condor-6.6.10 condor
[root@terpcondor condor]# mv condor-6.6.10/etc/condor_config .

First Start

Now, we should have a working condor install. This should be able to startup using the UMcondor init script. You'll notice that a few too many daemons startup. We'll fix this next.

(in ~condor)

[root@terpcondor condor]# ./UMcondor start
[root@terpcondor condor]# ps -ef|grep cond
squid    12859     1  0 12:48 ?        00:00:00 /opt/stow/condor/sbin/condor_master
squid    12860 12859  0 12:48 ?        00:00:00 condor_collector -f
squid    12861 12859  0 12:48 ?        00:00:00 condor_negotiator -f
squid    12862 12859 22 12:48 ?        00:00:05 condor_startd -f
squid    12864 12859  0 12:48 ?        00:00:00 condor_schedd -f
root     12890  6380  0 12:48 pts/0    00:00:00 grep cond
[root@terpcondor condor]# ./UMcondor stop
Shutting down Condor (fast-shutdown mode)

Local Configuration

In /export/condor/condor_config you'll need to make the following changes

# version independant release dir
RELEASE_DIR             = /export/condor/condor

# allow umiacs jobs to flow to this box
FLOCK_FROM = *.umiacs.umd.edu

# allow us to admin the box remotely
HOSTALLOW_ADMINISTRATOR = $(CONDOR_HOST), loach.umiacs.umd.edu

# only allow umd hosts to join this pool
HOSTALLOW_READ = *.umd.edu
HOSTALLOW_WRITE = *.umd.edu

now, for the local config in /export/condor-var/condor_config.local

DAEMON_LIST   = MASTER, COLLECTOR, NEGOTIATOR, SCHEDD

Set Condor to start on boot

Copy init script and start service.

[root@terpcondor condor]# cp UMcondor /etc/init.d
[root@terpcondor condor]# /sbin/chkconfig UMcondor on
[root@terpcondor condor]# /sbin/service UMcondor start
[root@terpcondor condor]# ps -ef|grep cond
squid    13393     1  0 13:04 ?        00:00:00 /export/condor/condor/sbin/condor_master
squid    13394 13393  0 13:04 ?        00:00:00 condor_collector -f
squid    13396 13393  0 13:04 ?        00:00:00 condor_negotiator -f
squid    13397 13393  0 13:04 ?        00:00:00 condor_schedd -f

Test Condor

On a standard umiacs host, add in /var/condor/condor_config.local

CONDOR_HOST = terpcondor.umiacs.umd.edu FLOCK_FROM = rogueleader.umiacs.umd.edu

In the global umiacs config, we added a 'FLOCK_TO = termcondor...' line to allow jobs to flow to the new condor pool

-- Main.MikeSmorul - 17 Aug 2005