English 中文(简体)
how to automatically run a bash script when my qsub jobs are finished on a server?
原标题:

I would like to run a script when all of the jobs that I have sent to a server are done.

for example, I send

ssh server "for i in config*; do qsub ./run 1 $i; done"

And I get back a list of the jobs that were started. I would like to automatically start another script on the server to process the output from these jobs once all are completed.

I would appreciate any advice that would help me avoid the following inelegant solution:

If I save each of the 1000 job id s from the above call in a separate file, I could check the contents of each file against the current list of running jobs, i.e. output from a call to:

ssh qstat

I would only need to check every half hour, but I would imagine that there is a better way.

最佳回答

It depends a bit on what job scheduler you are using and what version, but there s another approach that can be taken too if your results-processing can also be done on the same queue as the job.

One very handy way of managing lots of related job in more recent versions of torque (and with grid engine, and others) is to launch the any individual jobs as a job array (cf. http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm#-t). This requires mapping the individual runs to numbers somehow, which may or may not be convenient; but if you can do it for your jobs, it does greatly simplify managing the jobs; you can qsub them all in one line, you can qdel or qhold them all at once (while still having the capability to deal with jobs individually).

If you do this, then you could submit an analysis job which had a dependency on the array of jobs which would only run once all of the jobs in the array were complete: (cf. http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm#dependencyExamples). Submitting the job would look like:

qsub analyze.sh -W depend=afterokarray:427[]

where analyze.sh had the script to do the analysis, and 427 would be the job id of the array of jobs you launched. (The [] means only run after all are completed). The syntax differs for other schedulers (eg, SGE/OGE) but the ideas are the same.

Getting this right can take some doing, and certainly Tristan s approach has the advantage of being simple, and working with any scheduler; but learning to use job arrays in this situation if you ll be doing alot of this may be worth your time.

问题回答

Something you might consider is having each job script just touch a filename in a dedicated folder like $i.jobdone, and in your master script, you could simply use ls *.jobdone | wc -l to test for the right number of jobs done.

You can use wait to stop execution until all your jobs are done. You can even collect all the exit statuses and other running statistics (time it took, count of jobs done at the time, whatever) if you cycle around waiting for specific ids.

I d write a small C program to do the waiting and collecting (if you have permissions to upload and run executables), but you can easily use the bash wait built-in for roughly the same purpose, albeit with less flexibility.

Edit: small example.

#!/bin/bash

...
waitfor=  

for i in tasks; do
    task &
    waitfor="$waitfor $!"
done

wait $waitfor
...

If you run this script in background, It won t bother you and whatever comes after the wait line will run when your jobs are over.





相关问题
Parse players currently in lobby

I m attempting to write a bash script to parse out the following log file and give me a list of CURRENT players in the room (so ignoring players that left, but including players that may have rejoined)...

encoding of file shell script

How can I check the file encoding in a shell script? I need to know if a file is encoded in utf-8 or iso-8859-1. Thanks

Bash usage of vi or emacs

From a programming standpoint, when you set the bash shell to use vi or emacs via set -o vi or set -o emacs What is actually going on here? I ve been reading a book where it claims the bash shell ...

Dynamically building a command in bash

I am construcing a command in bash dynamically. This works fine: COMMAND="java myclass" ${COMMAND} Now I want to dynamically construct a command that redirectes the output: LOG=">> myfile.log ...

Perform OR on two hash outputs of sha1sum

I want perform sha1sum file1 and sha1sum file2 and perform bitwise OR operation with them using bash. Output should be printable i.e 53a23bc2e24d039 ... (160 bit) How can I do this? I know echo $(( ...

Set screen-title from shellscript

Is it possible to set the Screen Title using a shell script? I thought about something like sending the key commands ctrl+A shift-A Name enter I searched for about an hour on how to emulate ...

热门标签