qcg-pilotjobs
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Last revision Both sides next revision | ||
|
qcg-pilotjobs [2019/01/23 14:17] bbosak@man.poznan.pl |
qcg-pilotjobs [2019/01/24 13:01] bbosak@man.poznan.pl [Static QCG PilotJob] |
||
|---|---|---|---|
| Line 6: | Line 6: | ||
| ===== What is QCG PilotJob? ===== | ===== What is QCG PilotJob? ===== | ||
| - | The QCG PilotJob is a computing job playing a role of a container over a number of subordinate computing jobs. It allows to execute many subordinate jobs in a single scheduling system allocation. Direct submission of a large group of jobs to a scheduling system can result in long aggregated time to finish as each single job is scheduled independently and waits in a queue. On the other hand the submission of a group of jobs can be restricted or even forbidden by administrative policies defined on clusters. One can argue that there are available job array mechanisms in many systems, however the traditional job array mechanism allows to run only bunch of jobs having the same resource requirements while jobs being parts of a multiscale simulation by nature vary in requirements and therefore need more flexible solutions. On a technical level QCG PilotJob is exposed as a lightweight service - QCG PilotJob Manager. | + | QCG PilotJob is a computing job playing a role of a container over a number of subordinate computing jobs. It allows to execute many subordinate jobs in a single scheduling system allocation. Direct submission of a large group of jobs to a scheduling system can result in long aggregated time to finish as each single job is scheduled independently and waits in a queue. On the other hand the submission of a group of jobs can be restricted or even forbidden by administrative policies defined on clusters. One can argue that there are available job array mechanisms in many systems, however the traditional job array mechanism allows to run only bunch of jobs having the same resource requirements while jobs being parts of a multiscale simulation by nature vary in requirements and therefore need more flexible solutions. On a technical level QCG PilotJob is exposed as a lightweight service - QCG PilotJob Manager. |
| ===== How to use QCG PilotJob ? ===== | ===== How to use QCG PilotJob ? ===== | ||
| - | QCG PilotJob Manager is implemented in Python and as so may be run with the pure Python interpreter. Typically it is started on computing nodes utilising a small part of allocation. This may be quite tedious to run it by hand, thus, for user convenience, QCG PilotJob Manager has been integrated with the QCG middleware. Currently it may be easily used with the help of QCG-Client tool, what we recommend and describe later. | + | QCG PilotJob Manager is implemented in Python and as so may be run with the pure Python interpreter. Typically it is started on computing nodes utilising a small part of allocation. This may be quite tedious to run it by hand, thus, for user convenience, QCG PilotJob Manager has been integrated with the QCG middleware. Currently it may be easily used with the help of [[ http://www.qoscosgrid.org/trac/qcg-broker/wiki/client_user_guide | QCG-Client tool ]], what we recommend and describe later. |
| QCG PilotJob Manager offers two basic modes of its usage: static and dynamic. In both modes a user starts the QCG PilotJob Manager service, however while in the static mode the way of execution of subordinate jobs is known in advance, in the dynamic mode, the subordinate jobs are added to the QCG PilotJob Manager service on-demand programmatically, using predefined network API (e.g. from the python program). | QCG PilotJob Manager offers two basic modes of its usage: static and dynamic. In both modes a user starts the QCG PilotJob Manager service, however while in the static mode the way of execution of subordinate jobs is known in advance, in the dynamic mode, the subordinate jobs are added to the QCG PilotJob Manager service on-demand programmatically, using predefined network API (e.g. from the python program). | ||
| Line 106: | Line 106: | ||
| ] | ] | ||
| </code> | </code> | ||
| + | |||
| + | In this example, the first executed job will be ''date1'', then ''sleep'' and then ''date2''. The thing to note is the fact that the order of specification of jobs is important when there are dependencies between tasks applied. Thus, the ''sleep'' job has to be defined after ''date1'' and ''date2'' after ''sleep''. | ||
| This is very basic scenario, but in a similar way QCG PilotJob system supports definition of more advanced use cases, e.g scenarios including loops and/or parallel processing. | This is very basic scenario, but in a similar way QCG PilotJob system supports definition of more advanced use cases, e.g scenarios including loops and/or parallel processing. | ||
qcg-pilotjobs.txt ยท Last modified: 2019/01/31 11:47 by bbosak@man.poznan.pl
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
