Ticket 1685

By default sbatch submits a script and returns immediately.

Currently, there are two simple ways to know when a job has finished:
1. Notify by email.
2. Run a depended job with afterany,afternotok,afterok.

#1 is mostly useful when humans run a batch script.

#2 could be used by a workflow manager, but has issues. First afterok and afternotok cannot be used in practice, because if a batch finishes successfully, then the afternotok job blocks indefinitely, and vice versa if the batch finishes with an error. afterany works well, but then the command invoked does not know what was the exit status of the batch.

There are other ways to identify that a batch has finished. All of them use some sort of polling (poll over the output files of the job and wait for a signal that each job has completed, or poll calling scontrol show job until the job status changes, etc.). All of these approaches are error prone, and impose a heavy burden on the user.

But there is a simple way to fix this problem. Introduce an sbatch parameter that instructs sbatch to block and wait until the batch has finished. The exit code of sbatch can be the exit code of the batch script or for job arrays the largest exit code across all jobs.