Bug 1978 - PMIx plugin for SLURM
Summary: PMIx plugin for SLURM
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: Other (show other bugs)
Version: 15.08.1
Hardware: Linux Linux
: --- 5 - Enhancement
Assignee: David Bigagli
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2015-09-25 21:16 MDT by Artem Polyakov
Modified: 2017-08-24 09:00 MDT (History)
3 users (show)

See Also:
Site: SchedMD
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 16.05.0pre1
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments
PMIx patch (33.19 KB, application/x-bzip)
2015-09-25 21:16 MDT, Artem Polyakov
Details
Batch script for blocking test (4 nodex 2 ppn) (476 bytes, text/plain)
2015-09-25 21:22 MDT, Artem Polyakov
Details
Batch script for non-blocking test (4 nodex 2 ppn) (473 bytes, text/plain)
2015-09-25 21:23 MDT, Artem Polyakov
Details
Batch script for osu bcast (484 bytes, application/x-shellscript)
2015-09-25 21:23 MDT, Artem Polyakov
Details
Batch script for OSU alltoall (484 bytes, text/plain)
2015-09-25 21:24 MDT, Artem Polyakov
Details
result of test slurm_4x2p.job (41.57 KB, text/plain)
2015-10-07 21:24 MDT, David Bigagli
Details
results of test slurm_4x2p_nb.job (505.47 KB, text/plain)
2015-10-07 21:25 MDT, David Bigagli
Details
mpi/pmix patch2 (1.67 KB, patch)
2015-10-16 04:21 MDT, igor ivanov
Details | Diff
mpi/pmix patch2 (1.67 KB, patch)
2015-10-16 04:22 MDT, igor ivanov
Details | Diff
3rd patchset (17.27 KB, patch)
2015-11-19 20:14 MST, Artem Polyakov
Details | Diff
git apply --check (3.07 KB, text/plain)
2015-11-20 00:34 MST, David Bigagli
Details
all-in-one patch set (51.61 KB, patch)
2015-11-22 22:25 MST, Artem Polyakov
Details | Diff
Fix the autoconf subsystem to support "--without-PACKAGE" option. (3.82 KB, patch)
2015-12-14 12:01 MST, Artem Polyakov
Details | Diff
Documentation update (2.87 KB, patch)
2015-12-21 19:02 MST, Artem Polyakov
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Artem Polyakov 2015-09-25 21:16:29 MDT
Created attachment 2252 [details]
PMIx patch

Hello,
here is the patch for SLURM to add PMIx support. It was made against master however it applies to 15.08 as well.

Installation instructions. To configure with PMIx plugin you need:

1. Install libevent:
- download it from http://libevent.org/ (I used libevent-2.0.21-stable)
- configure and install it at some prefix <levent-pref>.

2. Install PMIx:
- clone it from https://github.com/open-mpi/pmix
- configure with libevent:
./configure --with-libevent=<levent-pref> --prefix=<pmix-pref>
- "make && make install" it.

3. Build SLURM with PMIx support:
./configure --with-pmix=<pmix-pref>

Testing. I used 2 tests to check the plugin. I'll provide sbatch scripts for both of them.

1. The first test is a program named pmix_client from the PMIx test suite:
https://github.com/open-mpi/pmix/tree/master/test
(Note that after compilation you'll need to check in <srcdir>/test/.libs because pmix_client is a script that resolves various linking issues).
Here is what you need to get:

$ sbatch slurm_4x2p_nb.job
Silent program output (just dates) and a set of files ot the format out.0.<rank> with "OK" in each identifying that all is OK.
$ ls out.*
out.0.0  out.0.1  out.0.2  out.0.3  out.0.4  out.0.5  out.0.6  out.0.7
$ cat out.0.*
OK
...
OK

2. The second test I've done was OSU benchmark (I've done bcast and alltoall) compiled with MPICH 3.1.4. The batch script is attached. NOTE that since PMIx is not currently supported by MPICH you'll need to use LD_PRELOAD to force it to use PMIx. This will be resolved in future on the PMIx library side.
Comment 1 Artem Polyakov 2015-09-25 21:22:38 MDT
Created attachment 2253 [details]
Batch script for blocking test (4 nodex 2 ppn)
Comment 2 Artem Polyakov 2015-09-25 21:23:14 MDT
Created attachment 2254 [details]
Batch script for non-blocking test (4 nodex 2 ppn)
Comment 3 Artem Polyakov 2015-09-25 21:23:47 MDT
Created attachment 2255 [details]
Batch script for osu bcast
Comment 4 Artem Polyakov 2015-09-25 21:24:25 MDT
Created attachment 2256 [details]
Batch script for OSU alltoall
Comment 5 David Bigagli 2015-10-05 04:15:45 MDT
Assigned to David.

David
Comment 6 David Bigagli 2015-10-05 21:31:01 MDT
Hi Artem, I am reviewing this code. I would like to commit it to the master
branch so the next major release, this will give more time to test and get
familiar with the overall plugin functionality including libevent which
is new for us.

David
Comment 7 Artem Polyakov 2015-10-05 23:01:46 MDT
Hi, David.

Thank you for the update. I think it's fine for now. 
Note however that libevent is _NOT_ used in the plugin. You need to install it for pmix library but the PMIx plugin in SLURM uses eoi subsystem. PMIx lib uses libevent internally to talk to clients. So at runtime libevent is loaded into stepd but the plugin has nothing to do with it.

Also would it be possible that we will backport it to 15.08.xx at some point (when you'll be sure that it works OK)? My understanding is that this was original plan.
Comment 8 David Bigagli 2015-10-06 01:50:32 MDT
Hi,
   let me get familiar with it than we can discuss. Is there any user waiting for
the plugin to be usable in Slurm?

David
Comment 9 Artem Polyakov 2015-10-06 02:25:02 MDT
Thank you!

We have OEM partners that eager to evaluate the plugin.
Comment 10 David Bigagli 2015-10-06 20:47:40 MDT
Hi Artem,
        is there a way to build pmix on centos 6?

./autogen.sh 
==> Running autoreconf
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I ./config
configure.ac:38: error: Autoconf version 2.69 or higher is required
configure.ac:38: the top level
autom4te: /usr/bin/m4 failed with exit status: 63
aclocal: autom4te failed with exit status: 63
autoreconf: aclocal failed with exit status: 63

David
Comment 11 Artem Polyakov 2015-10-06 21:09:20 MDT
Yes, sure.
Give me 20 min. and I'll provide you with instructions.
Comment 12 David Bigagli 2015-10-06 21:10:26 MDT
Thanks. Meanwhile I am trying to upgrade the autoconf to 2.69.

David
Comment 13 David Bigagli 2015-10-06 21:16:58 MDT
It needs to update autconf and automake to 

$ autoconf --version
autoconf (GNU Autoconf) 2.69

$ automake --version
automake (GNU automake) 1.15

then pmix builds fine, automake can be 1.12. I build theose from the source.

David
Comment 14 Artem Polyakov 2015-10-06 21:20:16 MDT
Here they are:
1. git clone git@github.com:artpol84/ompi_trunk_build.git
to get the set of scripts that I usually use.
2. cd ompi_trunk_build/scripts/
3. vi pmix_env.sh
Set PMIX_BASE variable to the dir containing cloned pmix directory
PMIX_BASE="<path-to-dir-with-PMIx>"
NOTE that you don't need to include pmix dir itself here!
4. ./download_tools.sh && ./prepare.sh
to download, compile and install proper versions of autotools and other tools.
5. ./pmix_sources_prepare.sh
to run autogen.sh with those tools.

Let me know if you'll have any problems with that. We can talk on Skype to speedup the process.
Comment 15 Artem Polyakov 2015-10-06 21:20:40 MDT
Here they are:
1. git clone git@github.com:artpol84/ompi_trunk_build.git
to get the set of scripts that I usually use.
2. cd ompi_trunk_build/scripts/
3. vi pmix_env.sh
Set PMIX_BASE variable to the dir containing cloned pmix directory
PMIX_BASE="<path-to-dir-with-PMIx>"
NOTE that you don't need to include pmix dir itself here!
4. ./download_tools.sh && ./prepare.sh
to download, compile and install proper versions of autotools and other tools.
5. ./pmix_sources_prepare.sh
to run autogen.sh with those tools.

Let me know if you'll have any problems with that. We can talk on Skype to speedup the process.
Comment 16 David Bigagli 2015-10-07 00:43:16 MDT
After working with Artem I did checkin the code for the PMIX plugin in the
master branch.

The code is clean and appear to be well written. The plugin is in a separate
code base so it should not impact other development.

I have to run autogen to recreate the Makefiles.in. The code compiles
with and without the pmix plugin.

We are still testing as we found some issues when mutiple-slurmds are defined
so I will continue to run some tests provided by Artem tomorrow.

I see two open issues. One is the documentation, we need to update the man page
for srun, perhaps have a separate man page for pmix? and update the html
document. Slurm testing suite needs to get updated to test the new functionality.

David
Comment 17 David Bigagli 2015-10-07 21:24:29 MDT
Created attachment 2282 [details]
result of test slurm_4x2p.job

result of test slurm_4x2p.job
Comment 18 David Bigagli 2015-10-07 21:25:52 MDT
Created attachment 2283 [details]
results of test slurm_4x2p_nb.job

results of test slurm_4x2p_nb.job
Comment 19 David Bigagli 2015-10-07 21:28:26 MDT
The test srun_pmix_a2a.job does not run as I don't have the osu_bcast binary.

I have appended the result of the tests but I am not able to judge if
they ran correctly or not.

I think we would need some simple mpi tests to verify the functionality
works as expected.

For now I leave it as it is for later evaluation, the code is in the
master branch.

David
Comment 20 Artem Polyakov 2015-10-07 22:01:52 MDT
Hi, David.

1. I think you can disable PMIX_DEBUG now.
2. See my explanation of the test. The output should be empty in the good case (when you run with disabled debug of course)! And you must see number of files created for each rank and have OK in each of them. Could you also provide the result of the following commands:
$ ls out.*
out.0.0  out.0.1  out.0.2  out.0.3  out.0.4  out.0.5  out.0.6  out.0.7
$ cat out.0.*
OK
...
OK
Comment 21 David Bigagli 2015-10-07 22:04:18 MDT
Thanks Artem, in that case it works:

$ cat out.0.*
OK
OK
OK
OK
OK
OK
OK
OK

David
Comment 22 Artem Polyakov 2015-10-07 22:11:56 MDT
Both blocking and non-blocking?
Comment 23 David Bigagli 2015-10-07 22:16:36 MDT
Yes.
Comment 24 Artem Polyakov 2015-10-07 22:35:15 MDT
Great!

If you want to try OSU you'll need MPICH with SLURM support (PMI1) as described here:
http://slurm.schedmd.com/mpi_guide.html#mpich2
(see section "MPICH2 with srun and PMI version 1").

And then you'll need to download and build OSU benchmarks with this MPICH.
http://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.0.tar.gz

Then you'll have a binary and will be able to use those scripts.
I can help you to setup this if you want (lets chat in Skype).
Comment 25 Artem Polyakov 2015-10-08 18:50:42 MDT
One correction, the actual MPICH building instructions for this case can be found here:
https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Note_that_the_default_build_of_MPICH_will_work_fine_in_SLURM_environments._No_extra_steps_are_needed.
Comment 26 igor ivanov 2015-10-16 04:21:10 MDT
Created attachment 2311 [details]
mpi/pmix patch2

Hello,
I am attaching two patches in addition as:
0001-mpi-pmix-Fix-resource-collection.patch
0002-mpi-pmix-Add-pmix-option-to-rpm-configuration-file.patch

also feel free to use our jenkins script for testing mpi/pmix locally.
It is located at https://github.com/mellanox-hpc/jenkins_scripts
May be you find it useful.
How to use:
1. go to jenkins/slurm folder
2. launch 
env WORKSPACE=<path to slurm sources> JENKINS_RUN_TESTS=1 ./pmix-slurm-jenkins.sh
Comment 27 igor ivanov 2015-10-16 04:22:59 MDT
Created attachment 2312 [details]
mpi/pmix patch2

Hello,
I am attaching two patches in addition as:
0001-mpi-pmix-Fix-resource-collection.patch
0002-mpi-pmix-Add-pmix-option-to-rpm-configuration-file.patch

also feel free to use our jenkins script for testing mpi/pmix locally.
It is located at https://github.com/mellanox-hpc/jenkins_scripts
May be you find it useful.
How to use:
1. go to jenkins/slurm folder
2. launch 
env WORKSPACE=<path to slurm sources> JENKINS_RUN_TESTS=1 ./pmix-slurm-jenkins.sh
Comment 28 David Bigagli 2015-10-20 02:29:24 MDT
This has been checked in in the master branch.

commit 7be9fabca5c97c530a53997db1b2cf6f5556ddce
Author: Igor Ivanov <igor.ivanov.va@gmail.com>
Date:   Tue Oct 20 16:28:52 2015 +0200

    Additional patches for pmix.

David
Comment 29 David Bigagli 2015-10-22 03:55:00 MDT
Hi Artem.
         do you foresee further changes to the plugin or we can consider the coding phase complete? If so we would need some documentation, for sure
we need to update http://slurm.schedmd.com/mpi_guide.html and the srun 
man page.

David
Comment 30 Artem Polyakov 2015-10-22 03:56:40 MDT
Hi, Dave.
Yes well have more fixes in near future.

четверг, 22 октября 2015 г. пользователь написал:

> *Comment # 29 <http://bugs.schedmd.com/show_bug.cgi?id=1978#c29> on bug
> 1978 <http://bugs.schedmd.com/show_bug.cgi?id=1978> from David Bigagli
> <javascript:_e(%7B%7D,'cvml','david@schedmd.com');> *
>
> Hi Artem.
>          do you foresee further changes to the plugin or we can consider the
> coding phase complete? If so we would need some documentation, for sure
> we need to update http://slurm.schedmd.com/mpi_guide.html and the srun
> man page.
>
> David
>
> ------------------------------
> You are receiving this mail because:
>
>    - You reported the bug.
>
>
Comment 31 Artem Polyakov 2015-11-19 20:14:00 MST
Created attachment 2433 [details]
3rd patchset

3rd set of patches that contains changes made during plugin verification at medium scale (upto 3584 cores)
Comment 32 David Bigagli 2015-11-20 00:34:14 MST
Created attachment 2435 [details]
git apply --check


Hi, 
   the patch does not apply against the current master branch. I am attaching
the result of 'git apply --check' could please review it.

David
Comment 33 Artem Polyakov 2015-11-20 01:10:49 MST
Sure. I've rebased against master and was expecting no problems. Looks like
I was wrong. Sorry. Will fix that.

пятница, 20 ноября 2015 г. пользователь написал:

> *Comment # 32 <http://bugs.schedmd.com/show_bug.cgi?id=1978#c32> on bug
> 1978 <http://bugs.schedmd.com/show_bug.cgi?id=1978> from David Bigagli
> <javascript:_e(%7B%7D,'cvml','david@schedmd.com');> *
>
> Created attachment 2435 [details] <http://bugs.schedmd.com/attachment.cgi?id=2435> [details] <http://bugs.schedmd.com/attachment.cgi?id=2435&action=edit>
> git apply --check
>
>
> Hi,
>    the patch does not apply against the current master branch. I am attaching
> the result of 'git apply --check' could please review it.
>
> David
>
> ------------------------------
> You are receiving this mail because:
>
>    - You reported the bug.
>
>
Comment 34 David Bigagli 2015-11-20 01:13:15 MST
No worries. Master branch is very fluid so this type of things happens all 
the time.

David
Comment 35 David Bigagli 2015-11-20 01:26:48 MST
Actually I was wrong. git apply --check does not consider dependencies among
patches so the patches have to be actually applied in the right order.
I did it and they applied correctly.

David
Comment 36 David Bigagli 2015-11-20 01:34:50 MST
commit 9ea3e728f8f7a2db8a701203fb76b2413dde7f2c
Author: Artem Polyakov <artpol84@gmail.com>
Date:   Fri Nov 20 16:34:59 2015 +0100

    New set of patches for pmix plugin.


Thanks, David
Comment 37 David Bigagli 2015-11-22 20:14:00 MST
Should be the default build be without pmix? I have a build failure if I don't
specify --with-pmix=/usr/local/pmix.

David
Comment 38 David Bigagli 2015-11-22 21:18:19 MST
The first error that happened here was that I committed the patche to 15.08.
I have reverted it since. So now 15.08 is clean and it build correctly.
So this is fixed.

The second error is that now 16.05 does not compile without --with--pmix
which is not right. I have reverted the patches in 16.05 as well.

Sorry this patching has been done too hastily.

David
Comment 39 Artem Polyakov 2015-11-22 22:25:06 MST
Created attachment 2438 [details]
all-in-one patch set

1. This patchset has nothing to do with the build system. See attached all-in-one.patch for convinience.
2. I've tried to do configure without "--with-pmix" and it works well. Here is the sequence:

$ ./configure.sh

# To make sure that we didn't configured with pmix:
$ cat config.h | grep -A 2 -B 2 PMIX
/* Define to 1 if pmix library found */
/* #undef HAVE_PMIX */

/* Define if you have Posix semaphores. */

$ make

And I'm getting successful build.
And no build was done in pmix directory:

$ ls src/plugins/mpi/pmi2/*.o
src/plugins/mpi/pmi2/agent.o   src/plugins/mpi/pmi2/kvs.o	src/plugins/mpi/pmi2/pmi1.o  src/plugins/mpi/pmi2/setup.o
src/plugins/mpi/pmi2/client.o  src/plugins/mpi/pmi2/mpi_pmi2.o	src/plugins/mpi/pmi2/pmi2.o  src/plugins/mpi/pmi2/spawn.o
src/plugins/mpi/pmi2/info.o    src/plugins/mpi/pmi2/nameserv.o	src/plugins/mpi/pmi2/ring.o  src/plugins/mpi/pmi2/tree.o
$ ls src/plugins/mpi/pmix/*.o
ls: Cannot access to src/plugins/mpi/pmix/*.o: No such file or directory
Comment 40 Artem Polyakov 2015-11-22 22:27:28 MST
P.S. 
1. what compilation error do you get?
2. Could you use:
for i in `ls -1 *.patch`; do git am $i; done
this time to keep the history?
Comment 41 Artem Polyakov 2015-11-22 22:27:53 MST
P.S. 
1. what compilation error do you get?
2. Could you use:
for i in `ls -1 *.patch`; do git am $i; done
this time to keep the history?
Comment 42 Artem Polyakov 2015-11-22 23:08:44 MST
P.P.S
First non-pmix build success I reported on my Linux Mint laptop. Now I tried the same on our RHEL cluster node. It was successful too. Again no build was performed inside pmix directory and config.h contains:

/* Define to 1 if pmix library found */                                                                                                               
/* #undef HAVE_PMIX */    

so PMIx wasn't configured. 

I ran ./configure without any options.
Comment 43 Artem Polyakov 2015-12-14 12:01:19 MST
Created attachment 2504 [details]
Fix the autoconf subsystem to support "--without-PACKAGE" option.

This is the fix for the case where PACKAGE (i.e. pmix) is installed in the standard paths ("/usr/" or "/usr/local"), however user doesn't want to configure with it and sets --without-Package option.
Comment 44 Artem Polyakov 2015-12-14 12:05:11 MST
David, after applying the patch that fixes "--without-PACKAGE" option you can run

./configure --without-pmix to fix the problem on your computer.
But better remove the old pmix library from /usr/ or /usr/local/.
Comment 45 David Bigagli 2015-12-14 21:37:56 MST
Committed the PMIx patch and the patch that implements the configure
--without=package option. Please do not forget to update he relevant
documentation, the srun man page and the mpi html page.

Great job thanks!

David
Comment 46 Artem Polyakov 2015-12-21 19:02:26 MST
Created attachment 2531 [details]
Documentation update

This is an update of Slurm documentation.
Comment 47 David Bigagli 2015-12-21 19:27:12 MST
This is now done. This is the latest commit:

commit 843a1257f406c08a1d32f9652ca874bc867d637f
Author: Artem Polyakov <artpol84@gmail.com>
Date:   Tue Dec 22 14:44:25 2015 +0600

  Update html documentation with information related to PMIx.

David
Comment 48 Alejandro Sanchez 2017-08-24 08:27:02 MDT
I see this comment as well:

        /* save client temp directory if requested
         * TODO: We want to get TmpFS value as well if exists.
         * Need to sync with SLURM developers.
         */
        p = getenvp(*env, PMIXP_TMPDIR_CLI);

alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi/pmi2$ grep -r "slurm_get_[slurmd|tmp]"
setup.c:			tree_width = slurm_get_tree_width();
setup.c:		tree_width = slurm_get_tree_width();
setup.c:	spool = slurm_get_slurmd_spooldir(NULL);
setup.c:	spool = slurm_get_slurmd_spooldir(NULL);
alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi/pmi2$ cd ..
alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi$ cd pmix
alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi/pmix$ grep -r "slurm_get_[slurmd|tmp]"
pmixp_utils.c:	timeout = slurm_get_msg_timeout() * 1000;
pmixp_utils.c:		int temp_rc = slurm_get_return_code(ret_data_info->type,
pmixp_info.c:	_pmixp_job_info.server_addr_unfmt = slurm_get_slurmd_spooldir(NULL);
pmixp_info.c:		_pmixp_job_info.cli_tmpdir_base = slurm_get_tmp_fs(
pmixp_coll.c:	width = slurm_get_tree_width();
alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi/pmix$

I'm not sure if we coordinated in the past with Artem how should we specify the pmix directory from Slurm side, if it should be TmpFS or SlurmdSpoolDir or what and/or what PMIx_server_init should interpret the information stored in the pmix_info_t struct members.
Comment 49 Danny Auble 2017-08-24 08:56:56 MDT
Wrong bug Alex?
Comment 50 Alejandro Sanchez 2017-08-24 09:00:04 MDT
(In reply to Danny Auble from comment #49)
> Wrong bug Alex?

Yes, wanted to comment in bug 4097. Please, ignore this one.