Using Grid-Ireland

• Introduction
• Portals
• Migrating Desktop
• P-Grade Portal
• Command Line (on a UI)
• Logging On
• Globus
• gLite
  • EGEE User Guide
  • Useful Links
  • gLite Workload Management
  • gLite Data Management
  • Job Description Language
    • Things to be aware of in JDLs
    • OutputData syntax
    • Job without data requirements
    • Job with data requirements
    • Job with output data requirements
    • Job with input data requirements
    • Job that stores data in the Replica Catalog using the lcg-* commands
    • Job that updates data in the Replica Catalog using the lcg-* commands
• Appendices
  • SSH Key Pair Generation
  • X11 Forwarding with SSH
  • GLUE Schema

Introduction

You will also need to apply for a Grid certificate. For more information see the guide to getting a Grid-Ireland User Certificate. You will also need to register with one of the Virtual Organizations. All Grid-Ireland users must accept the Grid-Ireland and Virtual Organization usage rules.

Grid-Ireland currently provides Globus 2.4 and EGEE gLite software for Grid computing. For an overview of the EGEE gLite software and usage instructions consult the EGEE gLite User Guide.

There are two generic ways to use the grid: through a web portal or via the command line.

Web Portals

If you need to contact the portal from outside your site, see the appendix for information on X11 forwarding.

Migrating Desktop

This is a portal that you can run from your usual browser on your own desktop. Browse to the Migrating Desktop. From there it is possible to create proxy credentials and submit jobs, all from your usual browser.

You need to have Java Web Start (JWS) installed. You can download it from java.com

You must have your cert key pair on your desktop machine, and you must point the Migrating Desktop at cagraidsvr26.cs.tcd.ie port 8443. You should save the configuration on the initial panel (but remember not to save your password!), and you should start by opening a new Migrating Desktop window and saving that as your default window - this will then be the window you get whenever you start up, no matter where you start from. For a tutorial, see MD Tutorial.

P-Grade Portal

The P-Grade Portal allows you to develop, execute and monitor workflows and workflow based parameter studies from a browser on your own desktop. For more detailed information on P-Grade click here. N.B. There are some client requirements for using the portal (a supported browser & Java plug-in) which are specified here

Grid-Ireland operates a P-Grade portal providing transparent access to Grid-Ireland resources no matter where the user is located. To use the portal first browse to the Grid-Ireland P-Grade portal server here. If you are a Grid-Ireland user and would like to try P-Grade you may apply for an account from your browser by clicking on the "Create new account" link and filling in the required details.

Once your account has been approved, browse to the P-Grade Portal - if this is your first time logging onto the portal you should then ensure that you are a member of the szupergrid group by checking the checkbox for this group under the "Configure group membership" panel which is located on the Welcome --> Settings tab (szupergrid is a p-grade portal group that allows access to the installed portlets). When this has been done the grid portal tabs for creating workflows, uploading certificates and accessing the information system will be displayed.

The first thing you need to do before jobs can be submitted is to obtain a proxy cert for your p-grade session - click on the "Certificates" tab. There are two options (i) download an existing proxy cert from a MyProxy server, or (ii) Upload credentials to MyProxy & then download a proxy cert - in either case you will be asked for the hostname and port number of the MyProxy service [Grid-Ireland's MyProxy server is cagraidsvr20.cs.tcd.ie and the port is 7512]. Once you have a proxy cert then you need to enable this proxy for the use with any (or all) virtual organisation (VO) that you intend to submit workflows under. Please choose the vo_LCG_2_BROKER options (with the '_LCG_2_BROKER' trailing text) rather than the VO (without the '_LCG_2_BROKER' trailing text). E.g. to submit workflows as a cosmo VO user you should choose the cosmo_LCG_2_BROKER entry.

Useful hints

There is an very informative Online User's Manual available. Alternatively see the PDF User's Manual.

Remember that p-grade uploads a binary from your local machine by default rather than executing something hosted on the remote machine. If your local machine is FC6 and the execute node is SL3, the uploaded hostname binary will not run. So if you want to run the ''hostname'' program on the remote host, you will have to upload a script which runs /bin/hostname. Also the "Name" of a workflow node is automatically converted to a script file "Name.sh" so don't call your script by the same name.

Also remember that to adding input files to the job is done by adding "ports" to the job node you define. Everything in pgrade is a workflow, so files are ports that allow data to flow between nodes (or from the local machine). It takes a little while to get used to this approach.

For MPI or PVM jobs, see Building your own Workflow. For parameter studies, see Parameter Studies (NB: upload the input files as replicas into an LFC directory; P-Grade automatically generates a job per each input file, then the sub-jobs are displayed with status information).

If you are using a Windows system remember that Windows represent text file end-of-lines with "CRLF" rather than "LF" as in UNIX - this will cause errors of the form "/bin/bash: bad interpreter: No such file or directory". You can overcome this by creating text files in UNIX format (or converting existing text files to UNIX format) using a Windows editor such as Cream (select Format -> File Format).

Command Line (on a UI)

Logging On to the UI

In order to make use of Grid-Ireland via the cmdline you will need an account on a Grid User Interface (UI) machine. A UI is installed at each Grid-Ireland site. If you would like an account please contact the Grid-Ireland helpdesk (grid-ireland-help at cs.tcd.ie) and we will check if you are eligible.

Once you have a UI account, you will need to log on to the Grid UI at your site with the username that has been assigned to you:

ssh <username>@gridui.<DNS domain>

After logging in, you must get a proxy credential which allows authentication with grid services. Note: these instructions assume you have already got a grid certificate and have installed successfully on the UI according to the instructions.

To use the Grid you must contact the VO membership service to get a credential that verifies your membership of the VO. Such a VOMS proxy is created with the voms-proxy-init command. In the following example, the --voms argument is the alias of the VO that you belong to. The aliases for Grid-Ireland VOs are listed here.

$ voms-proxy-init --voms test
Your identity: /C=IE/O=Grid-Ireland/OU=grid.ie/L=RA-TCD/CN=Grid User
Enter GRID pass phrase for this identity:
Creating temporary proxy ............................................. Done
/C=IE/O=Grid-Ireland/OU=cs.tcd.ie/L=RA-TCD/CN=host/cagraidsvr10.cs.tcd.ie
/C=IE/O=Grid-Ireland/CN=Grid-Ireland Certification Authority
Creating proxy ............................. Done

Globus Middleware

Globus provides a number of useful commands that act in the same way as Operating System commands. For example, to run /echo/bin "hello" on host gridigate.cs.tcd.ie, enter:

globus-job-run gridgate.cs.tcd.ie:2119/jobmanager-fork /bin/echo "hello"

gridgate should respond with hello.

EGEE gLite Middleware

Useful Links

• Glite Users Guide (Workload Management, Data Management, Information System, GLUE)
• EGEE User Guide (nice guide for new users)
• ClassAds syntax
• JDL/ClassAds HOWTO (a very useful reference document)
• lcg-utils (Replica Catalog commands)
• GFAL library (used by lcg-utils)

gLite Workload Management

From the gridui command line, one can submit a job:

glite-wms-job-submit -a myJDLfile.jdl

This will print out a job identifier that acts as a handle to the job. Then one can monitor the status of the job:

      glite-wms-job-status <job identifier>

When the job status is Done then one can retrieve the results:

      glite-wms-job-output <job identifier>

When submitting a job you can save the job identifier (referred to as edg_jobId in LCG2) to a file and use this file when checking the job's status and retrieving output. In the example below the job identifier is stored in the file ~/myJobIdFile:

      glite-wms-job-submit -a -o myJobIdFile myJDLfile

The job identifier is then saved saved in the file ~/myJobIdFile.Then one can monitor the status of the job as follows:

      glite-wms-job-status -i myJobIdFile

And one can retrieve the results as follows:

      glite-wms-job-output -i myJobIdFile --dir .

gLite Data Management

The Replica Catalog can be considered to be temporary storage that is available at each site on the gridstore node (the Storage Element) of the site's grid gateway. All large data sets should be input and output to/from grid jobs via the Replica Catalog, rather than via the InputSandbox or OutputSandbox. Once registered with the Replica Catalog, files may be referred to via grid logical filenames (LFNs) that the user associates with the file at the time of registration.

1. Set your environment to refer to the Replica Catalog:

        export LCG_CATALOG_TYPE=lfc
        export LFC_HOST=cagraidsvr22.cs.tcd.ie
        export LCG_GFAL_INFOSYS=cagraidsvr19.cs.tcd.ie:2170
   

2. To create a directory within the Replica Catalog:

        lfc-mkdir /grid/<VO name>/<user name>/<directory path>
   

   e.g:

        lfc-mkdir /grid/cosmo/coghlan/heatsim/2D/datafiles
   

   This only creates the rightmost subdirectory 'datafiles' - add the '-p' option to create non-existing parent subdirectories:

        lfc-mkdir -p /grid/cosmo/coghlan/heatsim/2D/datafiles
   

3. To register a file with the Replica Catalog:

        lcg-cr --vo <VO name> \
               -d <storage element> \
               -l <logical file name> \
               file:<file name>
   

   e.g:

        lcg-cr --vo webcom \
               -d gridstore.cs.tcd.ie \
               -l lfn:/grid/webcom/coghlan/aixgraphs/graph1.cg \
               file:`pwd`/graph1.cg
   

4. To retrieve a file from the Replica Catalog:

        lcg-cp --vo <VO name> \
               lfn:<logical file name> \
               file:<file to create>
   

   e.g.

        lcg-cp --vo webcom \
               lfn:/grid/webcom/coghlan/aixgraphs/graph1.cg \
               file:`pwd`/InputGraph.cg
   

5. To delete a registered file from the Replica Catalog:

        lfc-rm <logical file name> \
   

   e.g.:

        lfc-rm /grid/webcom/coghlan/aixgraphs/graph1.cg
   

6. To delete a directory from the Replica Catalog use the 'lfc-rm' command but with an extra '-r' option (this recursively removes subdirectories of and including the rightmost subdirectory 'aixgraphs'):

        lfc-rm -r /grid/webcom/coghlan/aixgraphs/
   

7. To list the LFNs on a storage element:

        lfc-ls /grid/<VO name>/<user name>/<directory path>
   

   e.g.

        lfc-ls /grid/cosmo/coghlan/heatsim/2D/datafiles
   

8. To input data from the Replica Catalog to within the job, the following items can be included within the JDL:

        InputData = "lfn:/grid/webcom/coghlan/aixgraphs/graph1.cg";
   

9. Similarly, to register the job output data with the Replica Catalog, i.e. to send it to the catalog, add to the JDL:

       OutputData = {
         [
           Outputfile = "OutputGraph.cg";
           LogicalName = "lfn:/grid/webcom/coghlan/aixgraphs/graph2.cg";
           StorageElement = "gridstore.cs.tcd.ie";
         ]};
   

10. The data access protocol defaults to "gridftp", but you can specify it in the JDL if you wish:

         DataAccessProtocol = {"gridftp","rfio"};
    

11. the VO structure is enforced using your proxy certificate, but you do not have to switch proxy certificates each time you wish to navigate a different VO's subtrees - once you own a filesystem then you can use it no matter what VO is specified in your proxy. In theory you can specify more than one VO when creating your proxy certificate, such as below, but thus far most gLite software (including the lcg-*, lfc-* and edg-* commands) only recognizes the first VO specified:

         voms-proxy-init --voms cosmo --voms webcom --voms solovo
    

Once you have registered a file in the Catalog you have two ways to get it to the job:

1. by retrieving it as in 4; or
2. by getting the system to retrieve it as in 8.

Similarly, when the job has generated an output file, you have two ways to register it in the Catalog:

1. by registering it as in 3; or
2. by getting the system to register it as in 9.

Job Description Language

WARNING!!! Information in this section is in development and is not guaranteed to be correct.

Things to be aware of in JDLs

If the arguments contains quoted strings, the quotes must be escaped with a backslash:

      Arguments = "\"Hello World!\" 10";

Special characters such as &, |, >, < are only allowed if specified inside a quoted string or preceded by triple '\\\':

      Arguments = "-f file1\\\&file2";)

The user can provide a local executable name, which will be staged from the UI to the WN, but the InputSandbox must then have the full path:

      Executable = {"egeode.sh"};
      InputSandbox = {"/home/bloggsj/egeode/egeode.sh"};

OutputData syntax

Optionally one can register the job output with the Register Catalog, i.e. send it to the catalog. The filename is mandatory, but the user can optionally specify the LFN and the Storage Element:

OutputData = {
  [
    OutputFile = "dataset1.out";            # filename is mandatory
  ],[
    OutputFile = "dataset2.out";            # filename is mandatory
    LogicalFileName = "lfn:test-result1";   # optional LFN
  ],[
    OutputFile = "dataset3.out";            # filename is mandatory
    LogicalFileName = "lfn:test-result2";   # optional LFN
    StorageElement = "gridstore.cs.tcd.ie"; # optional Storage Element
  ]
};

Job without data requirements

# Assumes script.sh is:
#   #!/bin/sh
#   /bin/echo Hello $1 and Welcome to the Grid-Ireland Tutorial!
# script.jdl
[
  Type = "job";
  JobType = "Normal";
  Executable = "script.sh";
  Arguments = "Brian";
  StdOutput = "sim.out";
  StdError = "sim.err";
  InputSandbox = {"script.sh"};
  OutputSandbox = {"sim.err","sim.out"};
  # A site with more than 4 CPU is required.
  Requirements=(other.GlueCEInfoTotalCPUs>4);
  # If more than one resource matches, the resource with the largest
  # number of CPU is chosen.
  Rank = (other.GlueCEStateFreeCPUs);   
]

Job with data requirements

# gridTest.jdl
[
  Executable = "gridTest";
  StdError = "stderr.log";
  StdOutput = "stdout.log";
  InputSandbox = {"/home/coghlan/test/gridTest"};
  OutputSandbox = {"stderr.log", "stdout.log"};
  InputData = "lfn:testbed0-00019";
  DataAccessProtocol = "gridftp";
  Requirements = other.Architecture=="INTEL" && \
                 other.OpSys=="LINUX" && other.FreeCpus >=4;
  Rank = "other.GlueHostBenchmarkSF00";
]

Job with output data requirements

# scriptOutput.jdl
[
  Type = "job";
  JobType = "Normal";
  Executable = "scriptOutput.sh";
  Arguments = "Gabriele";
  VirtualOrganisation = "webcom";
  StdOutput = "sim.out";
  StdError = "sim.err";
  InputSandbox = {"scriptOutput.sh"};
  OutputSandbox = {"sim.err", "sim.out"};
  OutputData = {
    [
      OutputFile = "Trinity-2007-06-20.out";
      LogicalFileName = "lfn:/grid/webcom/Trinity-2007-06-20.out";
      StorageElement = "gridstore.cs.tcd.ie";
    ]};           
  Requirements=(other.GlueCEInfoTotalCPUs>4);
  Rank=(other.GlueCEStateFreeCPUs);
  RetryCount = 0;
]

#!/bin/sh
fileOut="Trinity-2007-06-20.out"
/bin/echo Hello $1 and Welcome to the Grid-Ireland Tutorial! > $fileOut

Job with input data requirements

# Assumes scriptInput.sh is:
#   #!/bin/sh
#   lcg-cp vo marine lfn:myoutdata.1 file:`pwd`/dataset1.out
#   echo "Before updating.."
#   cat dataset1.out
#   #Adding new entry on the dataset1.out file.
#   /bin/echo Hello $1 and Welcome to the Grid-Ireland Tutorial! >> dataset1.out
#   echo "After updating.."
#   cat dataset1.out
# scriptInput.jdl
[
  Type = "job";
  JobType = "Normal";
  Executable = "scriptInput.sh";
  Arguments = "Andy";
  VirtualOrganisation = "MarineGrid";
  StdOutput = "std.out";
  StdError = "std.err";
  InputSandbox = {"scriptInput.sh"};
  OutputSandbox = {"std.err", "std.out"};
  InputData = "lfn:myoutdata.1";
  DataAccessProtocol = {"gridftp","rfio"};
  Requirements=(other.GlueCEInfoTotalCPUs>4);
  Rank=(other.GlueCEStateFreeCPUs);
  RetryCount = 0;
]

Job that stores data in the Replica Catalog using the lcg-* commands

# StoreData.jdl
[
  Type="Job";
  JobType="Normal";
  Executable="StoreData.sh";
  Arguments="myfile myfile gridstore.cp.dias.ie";
  VirtualOrganisation="cosmo";
  StdOutput="std.out";
  StdError="std.err";
  InputSandbox={"StoreData.sh","myfile"};
  OutputSandbox={"std.out","std.err"};
  Requirements=(other.GlueCEInfoTotalCPUs > 4);
  Rank=other.GlueCEStateFreeCPUs;
]

Assumes StoreData.sh is:

   #!/bin/sh
   # StoreData.sh
   ARGS=3
   ERROR_BADARGS=10        # Bad arguments error
   ERROR_FILE=20           # File to copy on RLS doesn't exist
   ERROR_STORAGE=30        # Storage Element doesn't exist in this catalog
   # Check for proper no. of command line args.
   if [ $# -ne $ARGS ]
   then
     echo "`basename $0.` Stores data into Replica Catalog."
     echo "Usage: `basename $0` [text-file2copy] [lfn-file2copy] [se-name]"
     exit $ERROR_BADARGS
   else
     # Set the variables for the script.
     FILE2COPY=$1
     LFN2COPY=$2
     SENAME=$3
   fi
   # Check if the ${FILE2COPY} exists on the path.
   if [ ! -e `pwd`/${FILE2COPY} ]
   then
     echo "Error: `basename $0.` ${FILE2COPY} does not exist on the path."
     exit ${ERROR_FILE}
   fi
   # Check if the SENAME is a good SE where to store data.
   lcg-infosites --vo cosmo closeSE | grep ${SENAME}
   if [ $? -eq 0 ]
   then
     # Upload the data to the RLS.
     lcg-cr --vo cosmo -d ${SENAME} -l ${LFN2COPY} file:`pwd`/${FILE2COPY}
     # Should really check for errors here
   else
     echo Sorry, but you have to specify a closer Storage Element.
     exit $ERROR_STORAGE
   fi

Job that updates data in the Replica Catalog using the lcg-* commands

# UpdateData.jdl
[
  Type="Job";
  JobType="Normal";
  Executable="UpdateData.sh";
  Arguments="myfile Gareth gridstore.cp.dias.ie";
  VirtualOrganisation="cosmo";
  StdOutput="std.out";
  StdError="std.err";
  InputSandbox={"UpdateData.sh"};
  OutputSandbox={"std.out","std.err"};
  Requirements=(other.GlueCEInfoTotalCPUs > 4);
  Rank=other.GlueCEStateFreeCPUs;
]

Assumes UpdateData.sh is:

   #!/bin/sh
   # UpdateData.sh
   ARGS=3
   ERROR_BADARGS=10        # Bad arguments error.
   ERROR_BADLFN=20         # LFN doesn't exist on the RLS.
   ERROR_BADSTORAGE=30     # Storage Element doesn't exist in this Catalog.
   # Check for proper no. of command line args.
   if [ $# -ne $ARGS ]
   then
     echo "`basename $0.` Updates data in Replica Catalog."
     echo "Usage: `basename $0` [lfn-file2copy] [argument] [se-name]"
     exit ${ERROR_BADARGS}
   else
     # Set the arguments for the script.
     LFN2RETRIEVE=$1
     ARGUMENT=$2
     SENAME=$3
   fi
   # Check if the ${LFN2RETRIEVE} exists on the RLS.
   lcg-lr --vo cosmo lfn:${LFN2RETRIEVE}
   if [ $? -eq 1 ]
   then
     echo "Sorry, but the LFN you specified does not exists on the RLS".
     exit ${ERROR_BADLFN}
   else
     # Retrieve the file from the catalog...
     lcg-cp --vo cosmo lfn:${LFN2RETRIEVE} file:`pwd`/myfile
     # ...adding new data to myfile.
     echo Hello ${ARGUMENT} and Welcome to the Grid-Ireland Tutorial!! >> `pwd`/myfile
   fi
   # Check if the SENAME is a good SE where to store data.
   lcg-infosites --vo cosmo closeSE | grep ${SENAME}
   if [ $? -eq 1 ]
   then
     echo Sorry, but you have to specify a closer Storage Element.
     exit ${ERROR_BADSTORAGE}
   else
     # Set the variables for the new copy.
     FILE2COPY=`pwd`/myfile
     LFN2COPY=${LFN2RETRIEVE}
     # Removing the old version from the RLS...
     # Should really rename to a temp file here
     lcg-del --vo cosmo -a lfn:${LFN2COPY}
     # ... and stage the new data onto the RLS.
     lcg-cr --vo cosmo -d ${SENAME} -l ${LFN2COPY} file:${FILE2COPY}
     # Should really check for errors here,
     #   and then if no errors then delete temp file
     echo "Your data has been correctly updated in the RLS.
     echo Have a nice day!!"
   fi

Appendices

SSH Key Pair Generation

See How To Generate SSH Keys for use on the Grid User Interface.

X11 forwarding with SSH

The UI at a site is only directly accessible from within that site. If you wish run a graphical program remotely on your site from home or when travelling, for example, it may be possible to use X11 forwarding with SSH. What follows are a few examples. It will be necssary to alter the details to suit your particular situation.

To run the JSUI from home, first SSH into an externally accessible host at your institution, and enable X11 forwarding:

ssh -X <username>@<external host>

Then, you can run the JSUI command normally, as specified above:

ssh -X <username>@gridui.<DNS domain> jsui

To run a browser remotely at your site - to access the UI portal - first SSH into an externally accessibly host at your institution, and enable X11 forwarding:

ssh -X <username>@<external host>

It may be possible to run the browser from this externally accessible host directly:

mozilla -browser &

Or it may be necessary to SSH again to another host that has the browser installed:

ssh -X <workstation> firefox -browser &

Windows users can set up X11 using Cygwin/X.

The GLUE Grid Information Schema

The Grid-Ireland website is hosted on cagraidsvr06.cs.tcd.ie in the Department of Computer Science, Trinity College Dublin.