Created attachment 2252 [details] PMIx patch Hello, here is the patch for SLURM to add PMIx support. It was made against master however it applies to 15.08 as well. Installation instructions. To configure with PMIx plugin you need: 1. Install libevent: - download it from http://libevent.org/ (I used libevent-2.0.21-stable) - configure and install it at some prefix <levent-pref>. 2. Install PMIx: - clone it from https://github.com/open-mpi/pmix - configure with libevent: ./configure --with-libevent=<levent-pref> --prefix=<pmix-pref> - "make && make install" it. 3. Build SLURM with PMIx support: ./configure --with-pmix=<pmix-pref> Testing. I used 2 tests to check the plugin. I'll provide sbatch scripts for both of them. 1. The first test is a program named pmix_client from the PMIx test suite: https://github.com/open-mpi/pmix/tree/master/test (Note that after compilation you'll need to check in <srcdir>/test/.libs because pmix_client is a script that resolves various linking issues). Here is what you need to get: $ sbatch slurm_4x2p_nb.job Silent program output (just dates) and a set of files ot the format out.0.<rank> with "OK" in each identifying that all is OK. $ ls out.* out.0.0 out.0.1 out.0.2 out.0.3 out.0.4 out.0.5 out.0.6 out.0.7 $ cat out.0.* OK ... OK 2. The second test I've done was OSU benchmark (I've done bcast and alltoall) compiled with MPICH 3.1.4. The batch script is attached. NOTE that since PMIx is not currently supported by MPICH you'll need to use LD_PRELOAD to force it to use PMIx. This will be resolved in future on the PMIx library side.
Created attachment 2253 [details] Batch script for blocking test (4 nodex 2 ppn)
Created attachment 2254 [details] Batch script for non-blocking test (4 nodex 2 ppn)
Created attachment 2255 [details] Batch script for osu bcast
Created attachment 2256 [details] Batch script for OSU alltoall
Assigned to David. David
Hi Artem, I am reviewing this code. I would like to commit it to the master branch so the next major release, this will give more time to test and get familiar with the overall plugin functionality including libevent which is new for us. David
Hi, David. Thank you for the update. I think it's fine for now. Note however that libevent is _NOT_ used in the plugin. You need to install it for pmix library but the PMIx plugin in SLURM uses eoi subsystem. PMIx lib uses libevent internally to talk to clients. So at runtime libevent is loaded into stepd but the plugin has nothing to do with it. Also would it be possible that we will backport it to 15.08.xx at some point (when you'll be sure that it works OK)? My understanding is that this was original plan.
Hi, let me get familiar with it than we can discuss. Is there any user waiting for the plugin to be usable in Slurm? David
Thank you! We have OEM partners that eager to evaluate the plugin.
Hi Artem, is there a way to build pmix on centos 6? ./autogen.sh ==> Running autoreconf autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal --force -I ./config configure.ac:38: error: Autoconf version 2.69 or higher is required configure.ac:38: the top level autom4te: /usr/bin/m4 failed with exit status: 63 aclocal: autom4te failed with exit status: 63 autoreconf: aclocal failed with exit status: 63 David
Yes, sure. Give me 20 min. and I'll provide you with instructions.
Thanks. Meanwhile I am trying to upgrade the autoconf to 2.69. David
It needs to update autconf and automake to $ autoconf --version autoconf (GNU Autoconf) 2.69 $ automake --version automake (GNU automake) 1.15 then pmix builds fine, automake can be 1.12. I build theose from the source. David
Here they are: 1. git clone git@github.com:artpol84/ompi_trunk_build.git to get the set of scripts that I usually use. 2. cd ompi_trunk_build/scripts/ 3. vi pmix_env.sh Set PMIX_BASE variable to the dir containing cloned pmix directory PMIX_BASE="<path-to-dir-with-PMIx>" NOTE that you don't need to include pmix dir itself here! 4. ./download_tools.sh && ./prepare.sh to download, compile and install proper versions of autotools and other tools. 5. ./pmix_sources_prepare.sh to run autogen.sh with those tools. Let me know if you'll have any problems with that. We can talk on Skype to speedup the process.
After working with Artem I did checkin the code for the PMIX plugin in the master branch. The code is clean and appear to be well written. The plugin is in a separate code base so it should not impact other development. I have to run autogen to recreate the Makefiles.in. The code compiles with and without the pmix plugin. We are still testing as we found some issues when mutiple-slurmds are defined so I will continue to run some tests provided by Artem tomorrow. I see two open issues. One is the documentation, we need to update the man page for srun, perhaps have a separate man page for pmix? and update the html document. Slurm testing suite needs to get updated to test the new functionality. David
Created attachment 2282 [details] result of test slurm_4x2p.job result of test slurm_4x2p.job
Created attachment 2283 [details] results of test slurm_4x2p_nb.job results of test slurm_4x2p_nb.job
The test srun_pmix_a2a.job does not run as I don't have the osu_bcast binary. I have appended the result of the tests but I am not able to judge if they ran correctly or not. I think we would need some simple mpi tests to verify the functionality works as expected. For now I leave it as it is for later evaluation, the code is in the master branch. David
Hi, David. 1. I think you can disable PMIX_DEBUG now. 2. See my explanation of the test. The output should be empty in the good case (when you run with disabled debug of course)! And you must see number of files created for each rank and have OK in each of them. Could you also provide the result of the following commands: $ ls out.* out.0.0 out.0.1 out.0.2 out.0.3 out.0.4 out.0.5 out.0.6 out.0.7 $ cat out.0.* OK ... OK
Thanks Artem, in that case it works: $ cat out.0.* OK OK OK OK OK OK OK OK David
Both blocking and non-blocking?
Yes.
Great! If you want to try OSU you'll need MPICH with SLURM support (PMI1) as described here: http://slurm.schedmd.com/mpi_guide.html#mpich2 (see section "MPICH2 with srun and PMI version 1"). And then you'll need to download and build OSU benchmarks with this MPICH. http://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.0.tar.gz Then you'll have a binary and will be able to use those scripts. I can help you to setup this if you want (lets chat in Skype).
One correction, the actual MPICH building instructions for this case can be found here: https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Note_that_the_default_build_of_MPICH_will_work_fine_in_SLURM_environments._No_extra_steps_are_needed.
Created attachment 2311 [details] mpi/pmix patch2 Hello, I am attaching two patches in addition as: 0001-mpi-pmix-Fix-resource-collection.patch 0002-mpi-pmix-Add-pmix-option-to-rpm-configuration-file.patch also feel free to use our jenkins script for testing mpi/pmix locally. It is located at https://github.com/mellanox-hpc/jenkins_scripts May be you find it useful. How to use: 1. go to jenkins/slurm folder 2. launch env WORKSPACE=<path to slurm sources> JENKINS_RUN_TESTS=1 ./pmix-slurm-jenkins.sh
Created attachment 2312 [details] mpi/pmix patch2 Hello, I am attaching two patches in addition as: 0001-mpi-pmix-Fix-resource-collection.patch 0002-mpi-pmix-Add-pmix-option-to-rpm-configuration-file.patch also feel free to use our jenkins script for testing mpi/pmix locally. It is located at https://github.com/mellanox-hpc/jenkins_scripts May be you find it useful. How to use: 1. go to jenkins/slurm folder 2. launch env WORKSPACE=<path to slurm sources> JENKINS_RUN_TESTS=1 ./pmix-slurm-jenkins.sh
This has been checked in in the master branch. commit 7be9fabca5c97c530a53997db1b2cf6f5556ddce Author: Igor Ivanov <igor.ivanov.va@gmail.com> Date: Tue Oct 20 16:28:52 2015 +0200 Additional patches for pmix. David
Hi Artem. do you foresee further changes to the plugin or we can consider the coding phase complete? If so we would need some documentation, for sure we need to update http://slurm.schedmd.com/mpi_guide.html and the srun man page. David
Hi, Dave. Yes well have more fixes in near future. четверг, 22 октября 2015 г. пользователь написал: > *Comment # 29 <http://bugs.schedmd.com/show_bug.cgi?id=1978#c29> on bug > 1978 <http://bugs.schedmd.com/show_bug.cgi?id=1978> from David Bigagli > <javascript:_e(%7B%7D,'cvml','david@schedmd.com');> * > > Hi Artem. > do you foresee further changes to the plugin or we can consider the > coding phase complete? If so we would need some documentation, for sure > we need to update http://slurm.schedmd.com/mpi_guide.html and the srun > man page. > > David > > ------------------------------ > You are receiving this mail because: > > - You reported the bug. > >
Created attachment 2433 [details] 3rd patchset 3rd set of patches that contains changes made during plugin verification at medium scale (upto 3584 cores)
Created attachment 2435 [details] git apply --check Hi, the patch does not apply against the current master branch. I am attaching the result of 'git apply --check' could please review it. David
Sure. I've rebased against master and was expecting no problems. Looks like I was wrong. Sorry. Will fix that. пятница, 20 ноября 2015 г. пользователь написал: > *Comment # 32 <http://bugs.schedmd.com/show_bug.cgi?id=1978#c32> on bug > 1978 <http://bugs.schedmd.com/show_bug.cgi?id=1978> from David Bigagli > <javascript:_e(%7B%7D,'cvml','david@schedmd.com');> * > > Created attachment 2435 [details] <http://bugs.schedmd.com/attachment.cgi?id=2435> [details] <http://bugs.schedmd.com/attachment.cgi?id=2435&action=edit> > git apply --check > > > Hi, > the patch does not apply against the current master branch. I am attaching > the result of 'git apply --check' could please review it. > > David > > ------------------------------ > You are receiving this mail because: > > - You reported the bug. > >
No worries. Master branch is very fluid so this type of things happens all the time. David
Actually I was wrong. git apply --check does not consider dependencies among patches so the patches have to be actually applied in the right order. I did it and they applied correctly. David
commit 9ea3e728f8f7a2db8a701203fb76b2413dde7f2c Author: Artem Polyakov <artpol84@gmail.com> Date: Fri Nov 20 16:34:59 2015 +0100 New set of patches for pmix plugin. Thanks, David
Should be the default build be without pmix? I have a build failure if I don't specify --with-pmix=/usr/local/pmix. David
The first error that happened here was that I committed the patche to 15.08. I have reverted it since. So now 15.08 is clean and it build correctly. So this is fixed. The second error is that now 16.05 does not compile without --with--pmix which is not right. I have reverted the patches in 16.05 as well. Sorry this patching has been done too hastily. David
Created attachment 2438 [details] all-in-one patch set 1. This patchset has nothing to do with the build system. See attached all-in-one.patch for convinience. 2. I've tried to do configure without "--with-pmix" and it works well. Here is the sequence: $ ./configure.sh # To make sure that we didn't configured with pmix: $ cat config.h | grep -A 2 -B 2 PMIX /* Define to 1 if pmix library found */ /* #undef HAVE_PMIX */ /* Define if you have Posix semaphores. */ $ make And I'm getting successful build. And no build was done in pmix directory: $ ls src/plugins/mpi/pmi2/*.o src/plugins/mpi/pmi2/agent.o src/plugins/mpi/pmi2/kvs.o src/plugins/mpi/pmi2/pmi1.o src/plugins/mpi/pmi2/setup.o src/plugins/mpi/pmi2/client.o src/plugins/mpi/pmi2/mpi_pmi2.o src/plugins/mpi/pmi2/pmi2.o src/plugins/mpi/pmi2/spawn.o src/plugins/mpi/pmi2/info.o src/plugins/mpi/pmi2/nameserv.o src/plugins/mpi/pmi2/ring.o src/plugins/mpi/pmi2/tree.o $ ls src/plugins/mpi/pmix/*.o ls: Cannot access to src/plugins/mpi/pmix/*.o: No such file or directory
P.S. 1. what compilation error do you get? 2. Could you use: for i in `ls -1 *.patch`; do git am $i; done this time to keep the history?
P.P.S First non-pmix build success I reported on my Linux Mint laptop. Now I tried the same on our RHEL cluster node. It was successful too. Again no build was performed inside pmix directory and config.h contains: /* Define to 1 if pmix library found */ /* #undef HAVE_PMIX */ so PMIx wasn't configured. I ran ./configure without any options.
Created attachment 2504 [details] Fix the autoconf subsystem to support "--without-PACKAGE" option. This is the fix for the case where PACKAGE (i.e. pmix) is installed in the standard paths ("/usr/" or "/usr/local"), however user doesn't want to configure with it and sets --without-Package option.
David, after applying the patch that fixes "--without-PACKAGE" option you can run ./configure --without-pmix to fix the problem on your computer. But better remove the old pmix library from /usr/ or /usr/local/.
Committed the PMIx patch and the patch that implements the configure --without=package option. Please do not forget to update he relevant documentation, the srun man page and the mpi html page. Great job thanks! David
Created attachment 2531 [details] Documentation update This is an update of Slurm documentation.
This is now done. This is the latest commit: commit 843a1257f406c08a1d32f9652ca874bc867d637f Author: Artem Polyakov <artpol84@gmail.com> Date: Tue Dec 22 14:44:25 2015 +0600 Update html documentation with information related to PMIx. David
I see this comment as well: /* save client temp directory if requested * TODO: We want to get TmpFS value as well if exists. * Need to sync with SLURM developers. */ p = getenvp(*env, PMIXP_TMPDIR_CLI); alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi/pmi2$ grep -r "slurm_get_[slurmd|tmp]" setup.c: tree_width = slurm_get_tree_width(); setup.c: tree_width = slurm_get_tree_width(); setup.c: spool = slurm_get_slurmd_spooldir(NULL); setup.c: spool = slurm_get_slurmd_spooldir(NULL); alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi/pmi2$ cd .. alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi$ cd pmix alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi/pmix$ grep -r "slurm_get_[slurmd|tmp]" pmixp_utils.c: timeout = slurm_get_msg_timeout() * 1000; pmixp_utils.c: int temp_rc = slurm_get_return_code(ret_data_info->type, pmixp_info.c: _pmixp_job_info.server_addr_unfmt = slurm_get_slurmd_spooldir(NULL); pmixp_info.c: _pmixp_job_info.cli_tmpdir_base = slurm_get_tmp_fs( pmixp_coll.c: width = slurm_get_tree_width(); alex@ibiza:~/slurm/17.02/slurm/src/plugins/mpi/pmix$ I'm not sure if we coordinated in the past with Artem how should we specify the pmix directory from Slurm side, if it should be TmpFS or SlurmdSpoolDir or what and/or what PMIx_server_init should interpret the information stored in the pmix_info_t struct members.
Wrong bug Alex?
(In reply to Danny Auble from comment #49) > Wrong bug Alex? Yes, wanted to comment in bug 4097. Please, ignore this one.