Treating urgent or pseudo-interactive jobs requires low-latency scheduling to minimize the response time. The "Short Deadline Job" (SDJ) configuration provides a mechanism for low-latency scheduling on the EGEE grid infrastructure. The trade-off for the low scheduling is that the associated queues usually have low limits on the CPU time and access restricted to a few Virtual Organizations.
On the EGEE grid infrastructure, the scheduling policy for jobs is controlled by the owners of the resources. Most sites implement either a first-in, first-out (FIFO) policy or fair-share policy. Sites schedule submitted jobs as quickly as possible, but make no guarantees as to how long a particular job will sit in a queue. For people submitting large numbers of similar jobs or long jobs, these best-effort policies are reasonable. However, they are not sufficient for urgent tasks or tasks requiring pseudo-interactive responses.
For jobs that require low-latency scheduling, the "Short Deadline Job" (SDJ) configuration was developed. This configuration provides a higher Quality-of-Service guarantee on how submitted jobs are handled. In particular a Computing Element (CE) that advertises the SDJ configuration will immediately start executing the job, if there are available resources. If there are no available resources, then the CE will immediately kill the job. This means that the job will never sit idle in a queue on the CE. Having the job killed immediately allows the controlling application to submit the job to another available resource.
The trade-off for providing the higher quality-of-service is that access to these resources is usually more restricted and the maximum CPU/wallclock limits are shorter than other resources.
The resources that provide the SDJ configuration are identified by computing element (CE) unique IDs that end with "sdj". One can use the standard JDL requirements clause to limit eligible resources to those with the SDJ configuration. An example JDL is:
Executable = "/bin/hostname";
StdOutput = "std.out";
StdError = "std.err";
OutputSandbox = {"std.out","std.err"};
Requirements = RegExp(".*sdj$",other.GlueCEUniqueID)
&& other.GlueCEStateFreeCPUs>0
&& other.GlueCEPolicyMaxCPUTime>5
&& other.GlueCEPolicyMaxWallClockTime>10;
Rank = other.GlueCEStateFreeCPUs;
The most important part of the Requirements statement is the RegExp clause; this selects only SDJ resources. The SDJ configuration will kill jobs that are submitted to saturated resources, so it is also a good idea to select resources that are advertising some free CPUs. Doing this will reduce the number of job "failures" that your application must handle.
As for all jobs, it is a good idea to identify the amount of CPU time and real (wallclock) time that your job requires. This is especially important for urgent jobs because you do not want to risk having the job killed because it exceeded published limits.
The Rank expression is entirely optional. The idea is to preferentially select resources that are not near capacity to minimize the number of job "failures" the application must treat.
Aside from the special requirements in the job description, submitting
a SDJ job, following its progress, and collecting the output is the same
for any other job. Use the commands glite-wms-job-submit,
glite-wms-job-status, and glite-wms-job-output,
respectively.
As the SDJ configuration will kill jobs that arrive on a saturated resource, your application should be prepared to handle rejected (killed) jobs. You can allow the WMS system to handle this with automatic resubmissions or you can let the application handle this by turning the automatic resubmission off by setting the retry count to zero.
RetryCount = 0;
By using the WMS you benefit from its matchmaking capabilities, however, the WMS does not treat SDJ jobs specially. Because of this, you may experience significant scheduling latencies at the WMS level, especially if the WMS server you use is heavily loaded.
For jobs that must run with the absolute minimum latency, one must use direct submission. This minimizes the number of servers treating the job and hence the latency.
First, you must first identify appropriate resources for your job. The easiest way to do this is to use the WMS system to list them. Take, for example, the JDL file above and save it as Hostname.jdl. Then run the following command:
$ glite-wms-job-list-match -a Hostname.jdl
This should return a list of CEs appropriate for your job, similar to the following output:
==========================================================================
COMPUTING ELEMENT IDs LIST
The following CE(s) matching your job requirements have been found:
*CEId*
- ipnls2001.in2p3.fr:2119/jobmanager-pbs-sdj
- grid10.lal.in2p3.fr:2119/jobmanager-pbs-sdj
==========================================================================
You could also directly search the information system with the
ldapsearch command; unless you are very familar with the GLUE
schema, this is not the recommended approach.
Second, you must submit your job directly to the chosen resource.
Currently, the CEs on the EGEE grid infrastructure support the
non-web-service GRAM protocol. Any client able to speak that protocol can
be used for direct submission. In particular, the Globus client commands
globus-job-submit, globus-job-status, and
globus-job-get-output are usually installed on EGEE User
Interface machines.
Presumably you will want to run a script that is not already installed
on the remote machine. As an example, create the file
hostname.sh with the following contents:
#!/bin/sh hostname
To submit this script to one of the identified SDJ resources, use the
globus-job-submit command:
$ globus-job-submit grid10.lal.in2p3.fr:2119/jobmanager-pbs -q sdj -s hostname.sh
Note that the queue name is 'sdj' and that this has been specified with
the "-q sdj" option. The '-s' option indicates
that this is a script to be staged. This command will return a job
identifier:
https://grid10.lal.in2p3.fr:20059/6983/1195051143/
Use this job identifier to query for the status of the job; continue until the status is "DONE".
$ globus-job-status https://grid10.lal.in2p3.fr:20059/6983/1195051143/ DONE
Once this has finished, the output can be recovered using the globus-job-get-output command.
$ globus-job-get-output https://grid10.lal.in2p3.fr:20059/6983/1195051143/ grid29.lal.in2p3.fr
You can write this to a file by redirection on the command line.
If you have many jobs to submit and to manage, then you should consider a job submission framework. With these you will have the benefits of automated matchmaking and also of direct submission. Two notable frameworks that permit this are Ganga and GridWay. See the documentation of these products to learn how to use them for job submission.