Some Useful Scripts

Alternatieve WMS

# Connect proxy with alternative wms service
WMS=https://wms.grid.sara.nl:7443/glite_wms_wmproxy_server
glite-wms-job-delegate-proxy -e $WMS -d $USER

# Submit job with alternative wms service
glite-wms-job-submit -e $WMS -d $USER -o myjobs myprogram.jdl

Testing if a module is available

testModule.sh is a script that checks whether a given module is installed on the computer, assuming that the the software packages are installed using the modules environment: If available, it loads the module (using module load) and displays the new PATH and LD_LIBRARY_PATH variables.

This is useful to test if a given software package is installed on clusters (see more here).

Download: testModule.sh

Run:

testModule.sh <module name>

Example:

testModule.sh matlab

Submitting a given job to all available clusters

submit-jobs-all-ces.sh is a script that lists all the available queues (CE) for a given VO and submits the same job to all of them. The job ids and CE names are stored in two files ( jobdis.txt and ces.txt). All input and output files are stored in the current directory.

NOTE: the first line in both files (with jobIDs and names) are ignored.

Download: submit-jobs-all-ces.sh

Run:

submit-jobs-all-ces.sh <vo> <jdl>

Retrieving the outputs of many jobs

get-outputs-jobs.sh is a script that gets the outputs of all jobs and stores them in directories with names given by a "name" file, both in text format (e.g., as generated by submit-jobs-all-ces.sh.

The script first tests if all jobs have reached the status Done. The user can decide to proceed or stop.

A new directory ./outputs is created in the current dir and all the files are stored in subdirectories with the given names. A file is created ./outputs/summary.txt

Download: get-outputs-jobs.sh

Run:

get-outputs-jobs.sh <jobs file> <names file>

Example (all together)

Goal: to run the testModule.sh script as a job (using testMCR.jdl) on each node to check the presence of the MCR software (runtime libs for matlab):

submit-jobs-all-ces.sh vlemed testMCR.jdl

This will generate the files ./jobids.txt and ./ces.txt, which can be used to retrieve the job outputs with

get-outputs-jobs.sh jobids.txt ces.txt

The output files will be stored as illustrated below:

silviado$ ll -R outputs/
outputs/:
total 56
drwxr-xr-x 2 silviado silviado 4096 May 26 19:09 trekker.nikhef.nl:2119-jobmanager-pbs-qshort/
-rw-r--r-- 1 silviado silviado 1546 May 26 19:08 summary.txt

outputs/trekker.nikhef.nl:2119-jobmanager-pbs-qshort:
total 4
-rw-r--r-- 1 silviado silviado 0 May 26 19:09 std.err
-rw-r--r-- 1 silviado silviado 373 May 26 19:09 std.out >

Test if Matlab is correctly running on all CEs

The script above can be used to check if MCR is installed and matlab is correctly running.

Download files here: testMCRMatlab.zip

Similar to above, say:

submit-jobs-all-ces.sh vlemed testMCRMatlab.jdl

Wait for all jobs to end, check via glite-wms-job-status -i jobids.txt

cd to outputs directory and retrieve and parse results:

get-outputs-jobs.sh jobids.txt ces.txt
cd outputs
results-jobs-all-ces.sh

Example output:

deimos.htc.biggrid.nl:2119-jobmanager-pbs-long/mcaan_44EY4-YQ3BKLBsRYreT63Q/std.out:****** ERROR: mcr/7.11 not available

ce.grid.rug.nl:2119-jobmanager-pbs-long/mcaan_5goGiBNUlmZ2ejWCRC4Hpg/std.out:****** ERROR: can't run matlab binary

ce.gina.sara.nl:2119-jobmanager-lcgpbs-medium/mcaan_hgN0FKc-gkvMrAUBE6xM6w/std.out:######################### Job completed successfully (0)

First CE has MCR/7.11 not installed. Second has broken MCR installation. Third is doing fine.

Change permissions on SRM level

Usage: ./lcg-rec-lr -vo vlemed -cp /grid/vlemed/mydirectory | ./change-permissions.sh

Input: list of srm file locations

Description: Finds out on which SE the file is located. Sets the DPNS_HOST variable to this SE. Changes the permission with the command dpns-setacl. At the end of the script the original value for the DPNS_HOST variable is restored.

Download: change-permissions.sh

Replicating to a list of SEs

lcg-infosites --vo vlemed se -v 1 > se.txt

Usage:

./lcg-rep-all.sh vlemed se.txt lfn:/grid/vlemed/AMC-e-BioScience/freesurfer/freesurfer.tar.gz

Download: lcg-rep-all.sh