I think we are not getting quite the right combination of build flags to enable full support of Intel MPI and OpenMPI with SLURM. I have installed Intel Parallel Studio 2018, SP 1, and we would like to build OpenMPI 1.10.7, 2.1.2, and 3.0. The actual symptoms vary, depending on the build options given when building SLURM. All of them indicate that somewhere along the line one component or another is not finding the right pmi library. In the closest to success we had, IMPI would run with srun but not with mpirun, and OMPI would run with mpirun but not with srun. This is on CentOS 7.3, soon to be 7.4. Do you have a standard .spec file and build options for SLURM that we should use to enable the Intel MPI to build and run with both srun and mpirun? Sorry to be vague, but perhaps it would be best to start from scratch rather than trying to disentangle all the combinations we have tried. -- bennet
Hi - When you get a chance, can you please review the severity level descriptions at https://www.schedmd.com/support.php ? I do not believe this merits Sev2, and I'm retagging as Sev3 instead. I do not have specific notes on hand for Intel MPI, but https://slurm.schedmd.com/mpi_guide.html has a general overview of getting the PMI2 interface working with a variety of installations. Are you using the slurm.spec file we provide to install? That should be placing our PMI library as /usr/lib64/libpmi.so, which is what you'd need to build OpenPMI against. Alex may be able to provide more specific pointers as well. - tIM
Tim, Sorry about the severity level. This is our development system, and we cannot continue to installing user software until we get the situation with SLURM and MPI compatibility squared away. Prior to trying to enable pmix, I believe that IMPI would run with srun but not with mpirun, and OMPI would run with mpirun but not with srun. We got to that situation by following the instructions on https://slurm.schedmd.com/mpi_guide.html We are specifically trying to get pmix support enabled, though. We are using the .spec file that comes with SLURM. Which version of pmix should we install, and where? We tried installing into the default system directories, but then the .rpm package for SLURM and the .rpm package for pmix conflict: Transaction check error: file /usr/lib64/libpmi.so from install of pmix-2.1.0-1.el7.centos.x86_64 conflicts with file from package slurm-17.11.0-1.el7.centos.x86_64 file /usr/lib64/libpmi2.so from install of pmix-2.1.0-1.el7.centos.x86_64 conflicts with file from package slurm-17.11.0-1.el7.centos.x86_64 Thanks, -- bennet
Hi, I've been testing today and managed to make Slurm work with PMIx and OMPI. These are the versions I've used as well as the procedure to build the different components: CentOS 7.3 Slurm 17.11.0 PMIx v1.2 OMPI v3.1.x You'll need these requirement packages: libevent-devel, hwloc-devel, libhwloc. PMIx v1.2: alex@ibiza:~/git$ git clone git@github.com:pmix/master.git pmix alex@ibiza:~/git$ cd pmix alex@ibiza:~/git/pmix$ git branch -a alex@ibiza:~/git/pmix$ git checkout v1.2 alex@ibiza:~/git/pmix$ ./autogen.sh alex@ibiza:~/git/pmix$ cd .. alex@ibiza:~/git$ mkdir pmix_build pmix_install alex@ibiza:~/git$ cd pmix_build alex@ibiza:~/git/pmix_build$ ../pmix/configure --prefix=/home/alex/git/pmix_install alex@ibiza:~/git/pmix_build$ make -j install >/dev/null alex@ibiza:~/git/pmix_build$ cd ../pmix_install alex@ibiza:~/git/pmix_install$ ls include lib share alex@ibiza:~/git/pmix_install$ Slurm v17.11.0: alex@ibiza:~/slurm/17.11/ibiza/slurm$ ../../slurm/configure --prefix=/home/alex/slurm/17.11/ibiza --with-pmix=/home/alex/git/pmix_install _IFF_ you need Slurm with multiple slurmd support: 1. Add --enable-multiple-slurmd to the configure line 2. Configure TmpFS in slurm.conf, i.e: TmpFS=/home/alex/slurm/17.02/ibiza/spool/slurmd-tmpfs-%n 3. Manually mkdir the corresponding TmpFS directories: /home/alex/slurm/17.02/ibiza/spool/slurmd-tmpfs-compute1, /home/alex/slurm/17.02/ibiza/spool/slurmd-tmpfs-compute2, ... ... checking for hwloc installation... /usr checking for pmix installation... /home/alex/git/pmix_install <-- you should see the provided --with-pmix=PATH found and usable ... alex@ibiza:~/slurm/17.02/ibiza/slurm$ make -j install > /dev/null alex@ibiza:~/slurm/17.11/ibiza/slurm$ ls -l ../lib/slurm | grep pmi -rw-r--r-- 1 alex alex 5602568 Dec 4 16:03 mpi_pmi2.a -rwxr-xr-x 1 alex alex 962 Dec 4 16:03 mpi_pmi2.la -rwxr-xr-x 1 alex alex 784848 Dec 4 16:03 mpi_pmi2.so lrwxrwxrwx 1 alex alex 16 Dec 4 16:03 mpi_pmix.so -> ./mpi_pmix_v1.so -rw-r--r-- 1 alex alex 7124622 Dec 4 16:03 mpi_pmix_v1.a -rwxr-xr-x 1 alex alex 1036 Dec 4 16:03 mpi_pmix_v1.la -rwxr-xr-x 1 alex alex 1079504 Dec 4 16:03 mpi_pmix_v1.so alex@ibiza:~/slurm/17.11/ibiza/slurm$ cd contribs/pmi2 alex@ibiza:~/slurm/17.11/ibiza/slurm/contribs/pmi2$ make install alex@ibiza:~/slurm/17.11/ibiza/slurm/contribs/pmi2$ cd ../../../lib alex@ibiza:~/slurm/17.11/ibiza/lib$ ls -l | grep pmi2 -rw-r--r-- 1 alex alex 498144 Dec 4 16:08 libpmi2.a -rwxr-xr-x 1 alex alex 958 Dec 4 16:08 libpmi2.la lrwxrwxrwx 1 alex alex 16 Dec 4 16:08 libpmi2.so -> libpmi2.so.0.0.0 lrwxrwxrwx 1 alex alex 16 Dec 4 16:08 libpmi2.so.0 -> libpmi2.so.0.0.0 -rwxr-xr-x 1 alex alex 214512 Dec 4 16:08 libpmi2.so.0.0.0 alex@ibiza:~/slurm/17.11/ibiza/lib$ OMPI v3.1.x: alex@ibiza:~/git$ git clone https://github.com/open-mpi/ompi.git ompi alex@ibiza:~/git$ cd ompi alex@ibiza:~/git/ompi$ git branch -a alex@ibiza:~/git/ompi$ git checkout v3.1.x alex@ibiza:~/git/ompi$ git pull alex@ibiza:~/git/ompi$ ./autogen.pl alex@ibiza:~/git/ompi$ mkdir ../ompi_build ../ompi_install alex@ibiza:~/git/ompi$ cd ../ompi_build alex@ibiza:~/git/ompi_build$ ../ompi/configure --prefix=/home/alex/git/ompi_install --with-pmix=/home/alex/git/pmix_install --with-pmi=/home/alex/slurm/17.11/ibiza --with-libevent=/usr --with-hwloc=/usr alex@ibiza:~/git/ompi_build$ make -j install > /dev/null Testing: Then you can compile and test a mpi_hello.c program by using the mpicc wrapper installed under your ompi installation. This combination is working for me: alex@ibiza:~/t$ srun --mpi=pmix -N2 mpi/xthi2 Hello from rank 0, thread 0, on ibiza. (core affinity = 0,4) Hello from rank 0, thread 1, on ibiza. (core affinity = 0,4) Hello from rank 1, thread 0, on ibiza. (core affinity = 0,4) Hello from rank 1, thread 1, on ibiza. (core affinity = 0,4) alex@ibiza:~/t$ Some notes: 1. I'll come back to you confirming whether pmix v2.1 can already be used with Slurm 17.11.0. 2. The slurm.spec was overhauled in Slurm 17.11.0 (a copy of the old slurm.spec can still be found under contribs/slurm.spec-legacy). I'll also come back to you saying how to specify the pmix installation path to the slurm.spec, since things have changed and I've to test further on that. But for now the configure && make && make install way is working for me as specified above. Please, let us know if you have further questions. Thanks.
If you see paths with 17.02 they are copy paste errors and should be 17.11, since I've been trying with both versions today and wrote the procedure for 17.02, then wrote it here in the bug changing 17.02 by 17.11 but have a few lines missing the replacement.
Totally excellent, Alejandro! Thank you. I will read carefully to make sure versions match when I try to recreate. Just to clarify, PMI is not used in any communication between the client and the SLURM scheduler/manager when requesting jobs, correct? I want to confirm my thinking that we can build the client separately from the server components, so long as the versions agree. I did some further testing on Fri, just with the client and PMI building, and this combination seemed to work. Get PMIx 2.0.2 .src.rpm from GitHub. $ rpmbuild --define '_prefix /opt/pmix/2.0.2' -ba pmix-2.0.2.spec $ rpmbuild --define '_prefix /opt/slurm' --define '_with-pmix --with-pmix=/opt/pmix/2.0.2' -ta slurm-17.11.0.tar.bz2 However, the server was not configured, so I was unable to test that with any MPI programs. I believe that the server is configured and running, and that I can install clients independently. I hope to confirm that today. Thanks very much for the detailed reply!
We can confirm that slurm 17.11.0 will work with pmix v2.0, but not with pmix 2.1. Commit b61fe2ac584c fixes the problem and will be available since slurm 17.11.1 when released. (In reply to ARCTS Admins from comment #5) > I believe that the server is configured and running, and that I can install > clients independently. I hope to confirm that today. They should be independent, yes. But better to try on a test system just in case you find any issues. I still have to dig further on the slurm.spec switch for specifying the pmix installation path, and how to provide it with rpmbuild, since $ rpmbuild --define '_prefix /opt/slurm' --define '_with-pmix --with-pmix=/opt/pmix/2.0.2' -ta slurm-17.11.0.tar.bz2 --define '_with-pmix --with-pmix=/opt/pmix/2.0.2' <-- this doesn't look ok to me at first glance.
Alejandro, I tried your non-rpm installation instructions. $ git clone git@github.com:pmix/master.git pmix $ git checkout v1.2 Branch v1.2 set up to track remote branch v1.2 from origin. Switched to a new branch 'v1.2' $ ./autogen.sh ==> Running autoreconf . . . . $ cd pmix_build/ $ ../pmix/configure --prefix=/sw/arcts/centos7/pmix/1.2 . . . . $ make -j 4 . . . . $ make install . . . . Libraries have been installed in: /sw/arcts/centos7/pmix/1.2/lib . . . . $ ls /sw/arcts/centos7/pmix/1.2 include lib share $ wget https://download.schedmd.com/slurm/slurm-17.11.0.tar.bz2 $ md5sum slurm-17.11.0.tar.bz2 d19d1fea67848dc923edde9c565b6b3a slurm-17.11.0.tar.bz2 ### From https://www.schedmd.com/downloads.php ### slurm-17.11.0.tar.bz2 ### md5: d19d1fea67848dc923edde9c565b6b3a $ tar xjf slurm-17.11.0.tar.bz2 $ mkdir slurm_build $ cd slurm_build/ $ ../slurm-17.11.0/configure --prefix=/sw/arcts/centos7/slurm/17.11.0 --with-pmix=/sw/arcts/centos7/pmix/1.2 . . . . $ grep -B4 -A4 pmix config.log It was created by slurm configure 17.11, which was generated by GNU Autoconf 2.69. Invocation command line was $ ../slurm-17.11.0/configure --prefix=/sw/arcts/centos7/slurm/17.11.0 --with-pmix=/sw/arcts/centos7/pmix/1.2 . . . . configure:21524: checking for pmix installation configure:21559: gcc -o conftest -g -O2 -pthread -I/sw/arcts/centos7/pmix/1.2/include conftest.c -L/sw/arcts/centos7/pmix/1.2/lib -lpmix >&5 configure:21559: $? = 0 configure:21590: gcc -E -I/sw/arcts/centos7/pmix/1.2/include conftest.c conftest.c:114:18: error: #error "not version 2" #error "not version 2" ^ configure:21590: $? = 1 -- | #define HAVE_LIBZ 1 | #define HAVE_HWLOC 1 | /* end confdefs.h. */ | | #include<pmix_server.h> | #if (PMIX_VERSION_MAJOR != 2L) | #error "not version 2" | #endif | -- | | ; | return 0; | } configure:21642: result: /sw/arcts/centos7/pmix/1.2 configure:21703: checking for freeipmi installation configure:21746: result: configure:21750: WARNING: unable to locate freeipmi installation It doesn't appear that pmix v1.2 is usable. Unless I am missing something. With regard to your Comment 9: I used this --define '_with-pmix --with-pmix=/opt/pmix/2.0.2' with the rpmbuild for slurm because I did a non-standard installation of pmix-2.0.2 such that $ ls /opt/pmix/2.0.2/ include lib lib64 share I tried it using just --define '_with-pmix --with-pmix=/opt/pmix' and it did not find pmix because the 'include|bin|lib' directories are not in /opt/pmix. -- bennet On Tue, Dec 5, 2017 at 12:25 PM, <bugs@schedmd.com> wrote: > Comment # 9 on bug 4464 from Alejandro Sanchez > > We can confirm that slurm 17.11.0 will work with pmix v2.0, but not with > pmix > 2.1. Commit b61fe2ac584c fixes the problem and will be available since slurm > 17.11.1 when released. > > (In reply to ARCTS Admins from comment #5) >> I believe that the server is configured and running, and that I can >> install >> clients independently. I hope to confirm that today. > > They should be independent, yes. But better to try on a test system just in > case you find any issues. > > I still have to dig further on the slurm.spec switch for specifying the pmix > installation path, and how to provide it with rpmbuild, since > > $ rpmbuild --define '_prefix /opt/slurm' --define '_with-pmix > --with-pmix=/opt/pmix/2.0.2' -ta slurm-17.11.0.tar.bz2 > > --define '_with-pmix --with-pmix=/opt/pmix/2.0.2' <-- this doesn't look ok > to > me at first glance. > > ________________________________ > You are receiving this mail because: > > You reported the bug.
Some clarifications on which PMIx versions are supported by which Slurm ones: Slurm 17.02 does not support PMIx v2.x, but supports PMIx v1.x series. Slurm 17.11.0 supports PMIx v1.x and v2.0. Slurm 17.11.1 supports PMIx v1.x, v2.0 and v2.1. But Slurm 17.11.1 is still not released at the time of writing this comment, so stick with Slurm 17.11.0 and PMIx v2.0. I know in comment 3 I installed a combination of Slurm 17.11.0 and PMIx v1.2 (which is perfectly valid), but I'd encourage you to rebuild PMIx v2.0 and make Slurm point to that PMIx v2.0 installation path with --with-pmix=<PATH>. Your grep for 'pmix' to the Slurm config.log is case sensitive, but still it shows this line: configure:21642: result: /sw/arcts/centos7/pmix/1.2 so it seems the path to your PMIx v1.2 installation was found and should be usable. If you rebuild again PMIx v2.0 and then rebuild Slurm, and grep case insensitive for pmix in config.log, you'll see these variables set: PMIX_V2_CPPFLAGS='-I/path/to/your/pmix/install/2/include' PMIX_V2_LDFLAGS='-Wl,-rpath -Wl,/path/to/your/pmix/install/2/lib -L/path/to/your/pmix/install/2/lib' After make install you should be able to see: alex@ibiza:~/slurm/17.11/ibiza/lib/slurm$ ls -l | grep -i pmix lrwxrwxrwx 1 alex alex 16 Dec 6 13:30 mpi_pmix.so -> ./mpi_pmix_v2.so -rw-r--r-- 1 alex alex 7192246 Dec 6 13:30 mpi_pmix_v2.a -rwxr-xr-x 1 alex alex 1038 Dec 6 13:30 mpi_pmix_v2.la -rwxr-xr-x 1 alex alex 1103216 Dec 6 13:30 mpi_pmix_v2.so alex@ibiza:~/slurm/17.11/ibiza/lib/slurm$ Please, rebuid PMIx with v2.0 and reinstall Slurm accordingly to that and let me know how it goes. Sorry for the confusion, I'm writing for another bug a web guide so that everybody has a better reference for this build process.
Alejandro, Just to be totally clear, PMIx 2.0.2 counts as PMIx 2.0, correct? I don't need to go all the way back to 2.0.0. It also appears that OMPI v3 will need to be configured with an external PMIx, as it comes with its own version of PMIx included, which might not be fully compatible with an externally installed PMIx. Thanks, -- bennet On Wed, Dec 6, 2017 at 9:10 AM, <bugs@schedmd.com> wrote: > > Slurm 17.02 does not support PMIx v2.x, but supports PMIx v1.x series. > Slurm 17.11.0 supports PMIx v1.x and v2.0. > Slurm 17.11.1 supports PMIx v1.x, v2.0 and v2.1. >
(In reply to ARCTS Admins from comment #12) > Alejandro, > > Just to be totally clear, PMIx 2.0.2 counts as PMIx 2.0, correct? No, that is not correct. After cloning pmix, you have this: alex@ibiza:~/git/pmix/source$ git branch -a * master remotes/origin/HEAD -> origin/master remotes/origin/gh-pages remotes/origin/master remotes/origin/v1.1 remotes/origin/v1.2 remotes/origin/v2.0 remotes/origin/v2.1 alex@ibiza:~/git/pmix/source$ Then if you git checkout v2.0 (branch compatible with slurm 17.11.0), you'll see this: alex@ibiza:~/git/pmix/source$ git checkout v2.0 Branch v2.0 set up to track remote branch v2.0 from origin by rebasing. Switched to a new branch 'v2.0' alex@ibiza:~/git/pmix/source$ git log -1 commit 82264872dc987b7fd9d61d46f99a91743794fc3f (HEAD -> v2.0, origin/v2.0) ... alex@ibiza:~/git/pmix/source$ git branch -r --contains 82264872dc origin/v2.0 alex@ibiza:~/git/pmix/source$ If you configure this above, you'll see this in the output of pmix configure: checking for pmix version... 2.0.3a1 And if you make install this above, you'll see this version installed in the pmix_version.h header: alex@ibiza:~/git/pmix/install/2$ grep PMIX_VERSION include/pmix_version.h #define PMIX_VERSION_MAJOR 2L #define PMIX_VERSION_MINOR 0L alex@ibiza:~/git/pmix/install/2$ Now, if you go back to the source, and instead of git checkout v2.0 (branch), you git checkout v2.0.2 (tag): alex@ibiza:~/git/pmix/source$ git checkout v2.0.2 Note: checking out 'v2.0.2'. You are in 'detached HEAD' state. ... alex@ibiza:~/git/pmix/source$ git log -1 commit 3dcd5bb873fb9b616abdb12846545f750ef3e3c3 (HEAD, tag: v2.1.0rc1, tag: v2.0.2) alex@ibiza:~/git/pmix/source$ git branch * (HEAD detached at v2.0.2) master v2.0 v2.1 alex@ibiza:~/git/pmix/source$ git branch -r --contains 3dcd5bb873 origin/v2.1 alex@ibiza:~/git/pmix/source$ If you configure with HEAD detached at v2.0.2 tag, you'll see this: checking for pmix version... 2.1.0rc1 alex@ibiza:~/git/pmix/install/2/include$ grep PMIX_VERSION pmix_version.h #define PMIX_VERSION_MAJOR 2L #define PMIX_VERSION_MINOR 1L alex@ibiza:~/git/pmix/install/2/include$ And that is not supported by slurm 17.11.0, but it is supported by 17.11.1. Does it make sense? > I don't need to go all the way back to 2.0.0. Slurm 17.11.0 supports PMIx v2.0 (pmix major 2 and minor 0). You can accomplish that by git checkout v2.0 (branch) on the PMIx source directory. I know... v2.0.2 tag sounds like it should be contained in the v2.0 branch, but inspecting with git that is not the case. Remember that a tag is just a pointer to a specific commit. > It also appears that OMPI v3 will need to be configured with an > external PMIx, as it comes with its own version of PMIx included, > which might not be fully compatible with an externally installed PMIx. That's correct. I use something like this when configuring OpenMPI v3.1.x: alex@ibiza:~/git/ompi_build$ ../ompi/configure \ --prefix=/home/alex/git/ompi_install \ --with-pmix=/home/alex/git/pmix_install \ <-- path to external PMIx install --with-pmi=/home/alex/slurm/17.02/ibiza \ <-- path to PMI install* --with-libevent=/usr \ --with-hwloc=/usr * (Optional) You can install by chdir to Slurm build contribs/pmi2, then make install there.
(In reply to Alejandro Sanchez from comment #13) > I know... v2.0.2 tag sounds like it should be contained in the v2.0 branch, > but inspecting with git that is not the case. Remember that a tag is just a > pointer to a specific commit. It looks like if you installed the PMIx v2.0.2 tag from this link: https://github.com/pmix/pmix/releases/tag/v2.0.2 that installed major 2 and minor 0. But if instead you installed tag v2.0.2 from doing git checkout v2.0.2, you would end up with major 2 and minor 1. We reported that to Artem (PMIx developer), and he is on the way of fixing that incoherence. Once it is fixed, either the tarball or git way should both end up with the same major=2 and minor=0. In the meantime, you can use the v2.0.2 from that link above, which works correct with slurm 17.11.0. Sorry for all this confusion, I also was surprised at first sight.
How is the build going? Did you manage to make all software components work together? I've updated in bug 4222 the mpi_guide web page with the following commit: https://github.com/SchedMD/slurm/commit/34209c471a29aeb5cf44e35 Please let me know if you have any more questions or if we can go ahead and close this bug. Thanks.
We have a working version that meets our criteria, which was + Both IMPI and OMPI should run at the command line outside a job using mpirun/mpiexec + Both IMPI and OMPI should run from within a job using mpirun/mipexec + Both IMPI and OMPI should run from within a job using srun The instructions you provided for building with `configure` were critical to getting there, but in the end we used `rpmbuild` to build both. I will ask my collegaue, Dan Barker, to update the ticket with the `rpmbuild` options, as I am on vacation for two weeks. -- bennet
Feel free to close it out. Your updated documentation in that commit looks great. Your rpmbuild instructions look like what we ended up using. rpmbuild --define '_slurm_sysconfdir /opt/slurm/etc' --define '_prefix /opt/slurm' --define '_with-pmix --with-pmix=/opt/pmix/2.0.2' -ta slurm-17.11.0.tar.bz2 Thanks for your help with this one. -Dan > On Dec 19, 2017, at 6:02 AM, bugs@schedmd.com wrote: > > > Comment # 15 <https://bugs.schedmd.com/show_bug.cgi?id=4464#c15> on bug 4464 <https://bugs.schedmd.com/show_bug.cgi?id=4464> from Alejandro Sanchez <mailto:alex@schedmd.com> > How is the build going? Did you manage to make all software components work > together? > > I've updated in bug 4222 <x-msg://3/show_bug.cgi?id=4222> the mpi_guide web page with the following commit: > > https://github.com/SchedMD/slurm/commit/34209c471a29aeb5cf44e35 <https://github.com/SchedMD/slurm/commit/34209c471a29aeb5cf44e35> > > Please let me know if you have any more questions or if we can go ahead and > close this bug. Thanks. > > You are receiving this mail because: > You reported the bug.
Cool, thanks for your feedback. Closing.
