Ticket 2602 - Upgrade instructions from slurmdbd 14.11.4 to 15.08.9
Summary: Upgrade instructions from slurmdbd 14.11.4 to 15.08.9
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmdbd (show other tickets)
Version: 14.11.4
Hardware: Linux Linux
: --- 4 - Minor Issue
Assignee: Tim Wickberg
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2016-03-31 06:37 MDT by David Richardson
Modified: 2016-04-26 06:06 MDT (History)
0 users

See Also:
Site: University of Utah
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description David Richardson 2016-03-31 06:37:39 MDT 
Hi,

We're preparing to update slurm here at UofU CHPC from 14.11.4 to 15.08.9.

I'm responsible for updating slurmdbd, but I'm new to this responsiblility. I understand that slurmdbd has to be upgraded before updating slurm.

Are there any special instructions or considerations or caveats in making this upgrade?

Thanks,
DR
Comment 1 Tim Wickberg 2016-03-31 06:58:00 MDT 
(In reply to David Richardson from comment #0)
> Hi,
> 
> We're preparing to update slurm here at UofU CHPC from 14.11.4 to 15.08.9.
> 
> I'm responsible for updating slurmdbd, but I'm new to this responsiblility.
> I understand that slurmdbd has to be upgraded before updating slurm.

Yep. The order is slurmdbd, then slurmctld, then slurmd's and user commands.

> Are there any special instructions or considerations or caveats in making
> this upgrade?

14.11 to 15.08 does affect some of the MySQL table structure, and can take a while to complete depending on how many job steps you have stored and how many associations there are.

I don't have an exact formula, but can say that when I upgraded GW it took around 15 minutes for ~2 million jobs steps, and 500 associations, and that was with the database on a modestly performing set of 10k disks. YMMV.

Assuming you're not doing anything especially stressful to the system (running less than 10k jobs per day, nodes are generally stable and unlikely to fail) the upgrade should be relatively straightforward. Just stop slurmdbd, install 15.08, and start slurmdbd. Once started it handles the database migration automatically, and will resume normal service once complete. The slurmdbd.log file will have some information related to the conversion, but its generally just best to run it and be patient.

A few caveats - you should (as a general best practice at least) make sure to have a recent backup of the slurm database in case something goes catastrophically wrong (although nothing has ever been reported, and we're very careful about handling data during the migration), and you cannot restart any dependent slurmctld until slurmdbd has finished.

Any running slurmctld's will be fine - any job completion messages or other data destined for slurmdbd will be cached until it is able to connect later (for a reasonable duration).

- Tim
Comment 2 Tim Wickberg 2016-04-26 06:06:23 MDT 
Marking this as resolved/infogiven. If you have further questions please reopen.

- Tim