5882 – Error when running seff: undefined symbol: node_record_count

Ticket 5882 - Error when running seff: undefined symbol: node_record_count

Summary: Error when running seff: undefined symbol: node_record_count

Status:	RESOLVED FIXED

Alias:	None

Product:	Slurm
Classification:	Unclassified
Component:	User Commands (show other tickets)
Version:	18.08.1
Hardware:	Linux Linux

Importance:	--- 3 - Medium Impact
Assignee:	Nate Rini
QA Contact:

URL:

Duplicates (2):	5796 5892 (view as ticket list)
Depends on:
Blocks:

Reported:	2018-10-18 13:23 MDT by Steve Ford
Modified:	2018-12-18 22:09 MST (History)
CC List:	4 users (show)

See Also:	5796 6266
Site:	MSU
Alineos Sites:	---
Atos/Eviden Sites:	---
Confidential Site:	---
Coreweave sites:	---
Cray Sites:	---
DS9 clusters:	---
HPCnow Sites:	---
HPE Sites:	---
IBM Sites:	---
NOAA SIte:	---
OCF Sites:	---
Recursion Pharma Sites:	---
SFW Sites:	---
SNIC sites:	---
Linux Distro:	---
Machine Name:
CLE Version:
Version Fixed:	18.08.3 19.05.0pre2
Target Release:	---
DevPrio:	---
Emory-Cloud Sites:	---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.

Description Steve Ford 2018-10-18 13:23:02 MDT

We are seeing the following error when calling seff in 18.08.1:

$ seff 312333
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd

We have an admin node with 17.11.7 clients that does not have this issue.

Comment 1 Nate Rini 2018-10-18 13:44:47 MDT

Steve

Can you provide the output of:

ldd /usr/lib64/slurm/accounting_storage_slurmdbd.so
nm o /usr/lib64/slurm/accounting_storage_slurmdbd.so

Thanks
--Nate

Comment 2 Steve Ford 2018-10-18 13:53:53 MDT

# ldd /usr/lib64/slurm/accounting_storage_slurmdbd.so 
	linux-vdso.so.1 =>  (0x00007fffad36d000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b70fda15000)
	libc.so.6 => /lib64/libc.so.6 (0x00002b70fdc31000)
	/lib64/ld-linux-x86-64.so.2 (0x0000563174fad000)    

# nm -o /usr/lib64/slurm/accounting_storage_slurmdbd.so 
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U abort@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005403 T acct_storage_p_add_accts
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000055ce T acct_storage_p_add_assocs
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005477 T acct_storage_p_add_clusters
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005393 T acct_storage_p_add_coord
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000054eb T acct_storage_p_add_federations
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005642 T acct_storage_p_add_qos
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000056b6 T acct_storage_p_add_res
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000579e T acct_storage_p_add_reservation
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000555f T acct_storage_p_add_tres
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000531f T acct_storage_p_add_users
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000572a T acct_storage_p_add_wckeys
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008f35 T acct_storage_p_clear_stats
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000528a T acct_storage_p_close_connection
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000052ae T acct_storage_p_commit
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000800f T acct_storage_p_fix_runaway_jobs
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008d4a T acct_storage_p_flush_jobs_on_cluster
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000006f2e T acct_storage_p_get_accts
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007497 T acct_storage_p_get_assocs
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007041 T acct_storage_p_get_clusters
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007267 T acct_storage_p_get_config
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000524a T acct_storage_p_get_connection
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008f9c T acct_storage_p_get_data
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000075aa T acct_storage_p_get_events
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007154 T acct_storage_p_get_federations
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000076bd T acct_storage_p_get_problems
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000077d0 T acct_storage_p_get_qos
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000078fa T acct_storage_p_get_res
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007b4e T acct_storage_p_get_reservations
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008e32 T acct_storage_p_get_stats
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007384 T acct_storage_p_get_tres
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007c78 T acct_storage_p_get_txn
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007d8b T acct_storage_p_get_usage
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000006e1b T acct_storage_p_get_users
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007a24 T acct_storage_p_get_wckeys
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005984 T acct_storage_p_modify_accts
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005bba T acct_storage_p_modify_assocs
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005a9f T acct_storage_p_modify_clusters
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005cd5 T acct_storage_p_modify_federations
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005df0 T acct_storage_p_modify_job
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005f2d T acct_storage_p_modify_qos
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000006048 T acct_storage_p_modify_res
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000627e T acct_storage_p_modify_reservation
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005869 T acct_storage_p_modify_users
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000006163 T acct_storage_p_modify_wckeys
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008dc0 T acct_storage_p_reconfig
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000065a3 T acct_storage_p_remove_accts
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000067dd T acct_storage_p_remove_assocs
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000066c0 T acct_storage_p_remove_clusters
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000006488 T acct_storage_p_remove_coord
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000068fa T acct_storage_p_remove_federations
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000006a17 T acct_storage_p_remove_qos
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000006b2a T acct_storage_p_remove_res
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000006d50 T acct_storage_p_remove_reservation
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000636b T acct_storage_p_remove_users
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000006c3d T acct_storage_p_remove_wckeys
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008e2c T acct_storage_p_reset_lft_rgt
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000007f82 T acct_storage_p_roll_usage
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008ff0 T acct_storage_p_shutdown
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008d44 T acct_storage_p_update_shares_used
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000009a4d t _agent
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f5a0 b agent_cond
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f590 b agent_list
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f5e0 b agent_lock
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f588 b agent_tid
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U __assert_fail@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f620 B assoc_cache_cond
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020fc20 B assoc_cache_mutex
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f41c B __bss_start
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U build_array_str
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U close@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000aadd T close_slurmdbd_conn
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008190 T clusteracct_storage_p_cluster_tres
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000082d4 T clusteracct_storage_p_fini_ctld
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008083 T clusteracct_storage_p_node_down
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008122 T clusteracct_storage_p_node_up
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000822a T clusteracct_storage_p_register_ctld
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000082ce T clusteracct_storage_p_register_disconn_ctld
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f420 b completed.6344
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000097ab t _create_agent
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 w __cxa_finalize@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f460 b db_inx_cond
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f4c8 b db_inx_handler_thread
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f4a0 b db_inx_lock
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U debug
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U debug3
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U debug4
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000004280 t deregister_tm_clones
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U difftime@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000042f0 t __do_global_dtors_aux
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020eda0 t __do_global_dtors_aux_fini_array_entry
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020edb0 d __dso_handle
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020edb8 d _DYNAMIC
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f41c D _edata
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020fc48 B _end
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U __errno_location@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U error@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f4f0 b fail_time.17250
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U fatal
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U find_job_record
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000005135 T fini
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000b52c T _fini
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f418 d first
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000004330 t frame_dummy
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020ed98 t __frame_dummy_init_array_entry
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000e230 r __FRAME_END__
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U free_buf
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cf10 r __func__.17139
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cf30 r __func__.17163
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000ced0 r __func__.17197
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cef0 r __func__.17232
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cf80 r __func__.17238
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cf64 r __func__.17255
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cf6b r __func__.17289
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cea0 r __func__.17317
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cfa0 r __func__.17327
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000ceb0 r __func__.17335
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000ce80 r __func__.17345
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000ce60 r __func__.17368
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000c680 r __func__.18088
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000c6a0 r __func__.18093
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000c6c0 r __func__.18108
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000c6d3 r __func__.18140
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000c678 r __func__.18146
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000c660 r __func__.18878
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U gettimeofday@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f000 d _GLOBAL_OFFSET_TABLE_
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 w __gmon_start__
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f580 b halt_agent
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U info
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000004f94 T init
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000003a48 T _init
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U init_buf
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 w _ITM_deregisterTMCloneTable
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 w _ITM_registerTMCloneTable
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020eda8 d __JCR_END__
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020eda8 d __JCR_LIST__
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008b99 T jobacct_storage_p_archive
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008c72 T jobacct_storage_p_archive_load
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008a4e T jobacct_storage_p_get_jobs_cond
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000841f T jobacct_storage_p_job_complete
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000082da T jobacct_storage_p_job_start
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000087f4 T jobacct_storage_p_step_complete
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000085d2 T jobacct_storage_p_step_start
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000008986 T jobacct_storage_p_suspend
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f448 B job_list
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 w _Jv_RegisterClasses
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_count
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_create
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_dequeue
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_destroy
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_enqueue
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_iterator_create
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_iterator_destroy
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_next
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_peek
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U list_remove
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000009402 t _load_dbd_rec
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000009556 t _load_dbd_state
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U lock_slurmctld
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f4e8 b max_agent_queue.17366
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U node_record_count
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U open@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000009062 t _open_slurmdbd_conn
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000a8d7 T open_slurmdbd_conn
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pack16
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U packmem
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pack_slurmdbd_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000004534 t _partial_destroy_dbd_job_start
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000004370 t _partial_free_dbd_job_start
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000c700 R plugin_name
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f450 b plugin_shutdown
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000c6e0 R plugin_type
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000c6d8 R plugin_version
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U prctl@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cf50 r __PRETTY_FUNCTION__.17158
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cee0 r __PRETTY_FUNCTION__.17195
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000cf00 r __PRETTY_FUNCTION__.17227
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_attr_destroy@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_attr_init@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_attr_setscope@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_attr_setstacksize@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_cancel@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_cond_broadcast@@GLIBC_2.3.2
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_cond_signal@@GLIBC_2.3.2
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_cond_timedwait@@GLIBC_2.3.2
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_cond_wait@@GLIBC_2.3.2
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_create@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_join@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_kill@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_mutex_lock@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_mutex_unlock@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_setcancelstate@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U pthread_setcanceltype@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U read@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000042b0 t register_tm_clones
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U rpc_num2string
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f440 B running_cache
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f458 b running_db_inx
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000009302 t _save_dbd_rec
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U select_get_plugin_id
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000ad3e T send_recv_slurmdbd_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000b0ca T send_slurmdbd_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000af36 T send_slurmdbd_recv_rc_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000490d t _set_db_inx_thread
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000458b t _setup_job_start_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:0000000000009060 t _sig_handler
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U sleep@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_bit_fmt
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f660 B slurmctld_conf
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000b50b T slurmdbd_agent_queue_count
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f570 b slurmdbd_cluster
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f500 b slurmdbd_cond
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f568 b slurmdbd_conn
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000000b0af T slurmdbd_conn_active
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurmdbd_free_buffer
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurmdbd_free_id_rc_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurmdbd_free_list_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f540 b slurmdbd_lock
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurmdbd_msg_type_2_str
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurmdbd_pack_fini_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f578 b slurmdbd_shutdown
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurmdbd_unpack_list_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurmdb_setup_cluster_flags
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_debug
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_debug2
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_debug4
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_error
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_fatal
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_get_accounting_storage_backup_host
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_get_accounting_storage_enforce
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_get_accounting_storage_host
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_get_accounting_storage_port
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_get_cluster_name
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_get_msg_timeout
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_get_state_save_location
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_info
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_list_append
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_list_count
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_list_create
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_list_destroy
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_list_iterator_create
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_list_iterator_destroy
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_list_next
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_persist_conn_close
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_persist_conn_destroy
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_persist_conn_open
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_persist_conn_writeable
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_persist_free_rc_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_persist_msg_pack
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_persist_recv_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_persist_send_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_persist_unpack_rc_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_set_accounting_storage_host
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_set_accounting_storage_port
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_seterrno
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_slurmdbd_free_id_rc_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_slurmdbd_free_list_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_slurmdbd_free_usage_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_strerror
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_verbose
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_xfree
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_xmalloc
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U slurm_xstrdup
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U __snprintf_chk@@GLIBC_2.3.4
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U snprintf@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U __stack_chk_fail@@GLIBC_2.4
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U strerror@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U strtoul@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U __syslog_chk@@GLIBC_2.4
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f4e0 b syslog_time.17365
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U time@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:000000000020f420 d __TMC_END__
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U unlink@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U unlock_slurmctld
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U unpack16
/usr/lib64/slurm/accounting_storage_slurmdbd.so:00000000000098c9 t _unpack_return_code
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U unpack_slurmdbd_msg
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U unpackstr_xmalloc_chooser
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U usleep@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U verbose
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U write@@GLIBC_2.2.5
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U xsignal
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U xsignal_unblock
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U _xstrcat
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U xstrdup

Comment 4 Nate Rini 2018-10-18 14:19:34 MDT

Steve

Please verify your LD_LIBRARY_PATH (or rpath) is pointing to the correct libslurm.so:

> "U" The symbol is undefined.
/usr/lib64/slurm/accounting_storage_slurmdbd.so:                 U node_record_count

node_record_count should be getting exported from libslurm:
>nate@spheron:~/slurm/slurm-18-08-1-1/build$ nm ./src/api/.libs/libslurm.so|grep node_record_count
>00000000003ed120 b node_record_count

If changing the order LD_LIBRARY_PATH doesn't work, then please send the output of this command:

> env LD_DEBUG=libs seff 


Thanks
--Nate

Comment 5 Steve Ford 2018-10-18 14:45:15 MDT

LD_LIBRARY_PATH is unset when I'm running seff.

# env LD_DEBUG=libs seff 40000
     15974:	find library=libperl.so [0]; searching
     15974:	 search path=/usr/lib64/perl5/CORE/tls/x86_64:/usr/lib64/perl5/CORE/tls:/usr/lib64/perl5/CORE/x86_64:/usr/lib64/perl5/CORE		(RUNPATH from file /usr/bin/perl)
     15974:	  trying file=/usr/lib64/perl5/CORE/tls/x86_64/libperl.so
     15974:	  trying file=/usr/lib64/perl5/CORE/tls/libperl.so
     15974:	  trying file=/usr/lib64/perl5/CORE/x86_64/libperl.so
     15974:	  trying file=/usr/lib64/perl5/CORE/libperl.so
     15974:	
     15974:	find library=libresolv.so.2 [0]; searching
     15974:	 search path=/usr/lib64/perl5/CORE		(RUNPATH from file /usr/bin/perl)
     15974:	  trying file=/usr/lib64/perl5/CORE/libresolv.so.2
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libresolv.so.2
     15974:	
     15974:	find library=libnsl.so.1 [0]; searching
     15974:	 search path=/usr/lib64/perl5/CORE		(RUNPATH from file /usr/bin/perl)
     15974:	  trying file=/usr/lib64/perl5/CORE/libnsl.so.1
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libnsl.so.1
     15974:	
     15974:	find library=libdl.so.2 [0]; searching
     15974:	 search path=/usr/lib64/perl5/CORE		(RUNPATH from file /usr/bin/perl)
     15974:	  trying file=/usr/lib64/perl5/CORE/libdl.so.2
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libdl.so.2
     15974:	
     15974:	find library=libm.so.6 [0]; searching
     15974:	 search path=/usr/lib64/perl5/CORE		(RUNPATH from file /usr/bin/perl)
     15974:	  trying file=/usr/lib64/perl5/CORE/libm.so.6
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libm.so.6
     15974:	
     15974:	find library=libcrypt.so.1 [0]; searching
     15974:	 search path=/usr/lib64/perl5/CORE		(RUNPATH from file /usr/bin/perl)
     15974:	  trying file=/usr/lib64/perl5/CORE/libcrypt.so.1
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libcrypt.so.1
     15974:	
     15974:	find library=libutil.so.1 [0]; searching
     15974:	 search path=/usr/lib64/perl5/CORE		(RUNPATH from file /usr/bin/perl)
     15974:	  trying file=/usr/lib64/perl5/CORE/libutil.so.1
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libutil.so.1
     15974:	
     15974:	find library=libpthread.so.0 [0]; searching
     15974:	 search path=/usr/lib64/perl5/CORE		(RUNPATH from file /usr/bin/perl)
     15974:	  trying file=/usr/lib64/perl5/CORE/libpthread.so.0
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libpthread.so.0
     15974:	
     15974:	find library=libc.so.6 [0]; searching
     15974:	 search path=/usr/lib64/perl5/CORE		(RUNPATH from file /usr/bin/perl)
     15974:	  trying file=/usr/lib64/perl5/CORE/libc.so.6
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libc.so.6
     15974:	
     15974:	find library=libfreebl3.so [0]; searching
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libfreebl3.so
     15974:	
     15974:	
     15974:	calling init: /lib64/libpthread.so.0
     15974:	
     15974:	
     15974:	calling init: /lib64/libc.so.6
     15974:	
     15974:	
     15974:	calling init: /lib64/libdl.so.2
     15974:	
     15974:	
     15974:	calling init: /lib64/libfreebl3.so
     15974:	
     15974:	
     15974:	calling init: /lib64/libutil.so.1
     15974:	
     15974:	
     15974:	calling init: /lib64/libcrypt.so.1
     15974:	
     15974:	
     15974:	calling init: /lib64/libm.so.6
     15974:	
     15974:	
     15974:	calling init: /lib64/libnsl.so.1
     15974:	
     15974:	
     15974:	calling init: /lib64/libresolv.so.2
     15974:	
     15974:	
     15974:	calling init: /usr/lib64/perl5/CORE/libperl.so
     15974:	
     15974:	
     15974:	initialize program: /usr/bin/perl
     15974:	
     15974:	
     15974:	transferring control: /usr/bin/perl
     15974:	
     15974:	
     15974:	calling init: /usr/lib64/perl5/auto/Fcntl/Fcntl.so
     15974:	
     15974:	
     15974:	calling init: /usr/lib64/perl5/auto/POSIX/POSIX.so
     15974:	
     15974:	
     15974:	calling init: /usr/lib64/perl5/auto/Sys/Hostname/Hostname.so
     15974:	
     15974:	find library=libslurmdb.so.33 [0]; searching
     15974:	 search path=../../../../src/db_api/.libs/tls/x86_64:../../../../src/db_api/.libs/tls:../../../../src/db_api/.libs/x86_64:../../../../src/db_api/.libs:/usr/lib/tls/x86_64:/usr/lib/tls:/usr/lib/x86_64:/usr/lib		(RPATH from file /usr/lib64/perl5/auto/Slurmdb/Slurmdb.so)
     15974:	  trying file=../../../../src/db_api/.libs/tls/x86_64/libslurmdb.so.33
     15974:	  trying file=../../../../src/db_api/.libs/tls/libslurmdb.so.33
     15974:	  trying file=../../../../src/db_api/.libs/x86_64/libslurmdb.so.33
     15974:	  trying file=../../../../src/db_api/.libs/libslurmdb.so.33
     15974:	  trying file=/usr/lib/tls/x86_64/libslurmdb.so.33
     15974:	  trying file=/usr/lib/tls/libslurmdb.so.33
     15974:	  trying file=/usr/lib/x86_64/libslurmdb.so.33
     15974:	  trying file=/usr/lib/libslurmdb.so.33
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libslurmdb.so.33
     15974:	
     15974:	
     15974:	calling init: /lib64/libslurmdb.so.33
     15974:	
     15974:	
     15974:	calling init: /usr/lib64/perl5/auto/Slurmdb/Slurmdb.so
     15974:	
     15974:	find library=libslurm.so.33 [0]; searching
     15974:	 search path=../../../../src/api/.libs/tls/x86_64:../../../../src/api/.libs/tls:../../../../src/api/.libs/x86_64:../../../../src/api/.libs:/usr/lib		(RPATH from file /usr/lib64/perl5/auto/Slurm/Slurm.so)
     15974:	  trying file=../../../../src/api/.libs/tls/x86_64/libslurm.so.33
     15974:	  trying file=../../../../src/api/.libs/tls/libslurm.so.33
     15974:	  trying file=../../../../src/api/.libs/x86_64/libslurm.so.33
     15974:	  trying file=../../../../src/api/.libs/libslurm.so.33
     15974:	  trying file=/usr/lib/libslurm.so.33
     15974:	 search cache=/etc/ld.so.cache
     15974:	  trying file=/lib64/libslurm.so.33
     15974:	
     15974:	
     15974:	calling init: /lib64/libslurm.so.33
     15974:	
     15974:	
     15974:	calling init: /usr/lib64/perl5/auto/Slurm/Slurm.so
     15974:	
     15974:	/usr/lib64/slurm/accounting_storage_slurmdbd.so: error: symbol lookup error: undefined symbol: node_record_count (fatal)
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd
     15974:	find library=libmunge.so.2 [0]; searching
     15974:	 search path=/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64:/usr/lib64		(system search path)
     15974:	  trying file=/usr/lib64/tls/x86_64/libmunge.so.2
     15974:	  trying file=/usr/lib64/tls/libmunge.so.2
     15974:	  trying file=/usr/lib64/x86_64/libmunge.so.2
     15974:	  trying file=/usr/lib64/libmunge.so.2
     15974:	
     15974:	
     15974:	calling init: /usr/lib64/libmunge.so.2
     15974:	
     15974:	
     15974:	calling init: /usr/lib64/slurm/auth_munge.so
     15974:	
     15974:	/usr/lib64/slurm/accounting_storage_slurmdbd.so: error: symbol lookup error: undefined symbol: node_record_count (fatal)
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd
Job not found.
     15974:	
     15974:	calling fini: /usr/bin/perl [0]
     15974:	
     15974:	
     15974:	calling fini: /usr/lib64/perl5/CORE/libperl.so [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libresolv.so.2 [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libnsl.so.1 [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libcrypt.so.1 [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libutil.so.1 [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libfreebl3.so [0]
     15974:	
     15974:	
     15974:	calling fini: /usr/lib64/perl5/auto/Fcntl/Fcntl.so [0]
     15974:	
     15974:	
     15974:	calling fini: /usr/lib64/perl5/auto/POSIX/POSIX.so [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libm.so.6 [0]
     15974:	
     15974:	
     15974:	calling fini: /usr/lib64/perl5/auto/Sys/Hostname/Hostname.so [0]
     15974:	
     15974:	
     15974:	calling fini: /usr/lib64/perl5/auto/Slurmdb/Slurmdb.so [0]
     15974:	
     15974:	
     15974:	calling fini: /usr/lib64/perl5/auto/Slurm/Slurm.so [0]
     15974:	
     15974:	
     15974:	calling fini: /usr/lib64/slurm/auth_munge.so [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libslurm.so.33 [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libslurmdb.so.33 [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libdl.so.2 [0]
     15974:	
     15974:	
     15974:	calling fini: /lib64/libpthread.so.0 [0]
     15974:	
     15974:	
     15974:	calling fini: /usr/lib64/libmunge.so.2 [0]
     15974:

Comment 6 Steve Ford 2018-10-18 14:46:40 MDT

For what it's worth:

#nm /usr/lib64/libslurm.so | grep record_count
00000000003a9420 b node_record_count

Comment 8 Nate Rini 2018-10-18 15:42:29 MDT

Steve

Based on your output, LD is not trying to link against /usr/lib64/libslurm.so (or the tagged version /usr/lib64/libslurm.so.33) but is instead linking against /lib64/libslurm.so.33

Please provide the output to the following:
ldd /usr/lib64/perl5/auto/Slurmdb/Slurmdb.so
namei /usr/lib64/libslurm.so
ldd /usr/lib64/libslurm.so
ldd /lib64/libslurm.so.33

Please try:
env LD_LIBRARY_PATH=/usr/lib64/:$LD_LIBRARY_PATH ldd $(which seff)
env LD_LIBRARY_PATH=/usr/lib64/:$LD_LIBRARY_PATH seff

Thanks
--Nate

Comment 9 Steve Ford 2018-10-18 17:04:04 MDT

In CentOS 7, /lib64 is symlinked to /usr/lib64

$ ll -d /lib64
lrwxrwxrwx. 1 root root 9 Jul  9 09:43 /lib64 -> usr/lib64

$ ldd /usr/lib64/perl5/auto/Slurmdb/Slurmdb.so 
	linux-vdso.so.1 =>  (0x00007ffd8c7f4000)
	libslurmdb.so.33 => /lib64/libslurmdb.so.33 (0x00002b5317c7d000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b531801e000)
	libc.so.6 => /lib64/libc.so.6 (0x00002b531823a000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00002b53185fe000)
	/lib64/ld-linux-x86-64.so.2 (0x000055818dc6d000)

$ namei /usr/lib64/libslurm.so
f: /usr/lib64/libslurm.so
 d /
 d usr
 d lib64
 l libslurm.so -> libslurm.so.33.0.0
   - libslurm.so.33.0.0

$ ldd /usr/lib64/libslurm.so
	linux-vdso.so.1 =>  (0x00007fff8cfe2000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00002ad3bb32e000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002ad3bb532000)
	libc.so.6 => /lib64/libc.so.6 (0x00002ad3bb74e000)
	/lib64/ld-linux-x86-64.so.2 (0x000055997f84f000)

$ ldd /lib64/libslurm.so.33
	linux-vdso.so.1 =>  (0x00002b7cfb888000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00002b7cfbc55000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b7cfbe59000)
	libc.so.6 => /lib64/libc.so.6 (0x00002b7cfc075000)
	/lib64/ld-linux-x86-64.so.2 (0x000055bcac9e4000)

$ env LD_LIBRARY_PATH=/usr/lib64/:$LD_LIBRARY_PATH ldd $(which seff)
	not a dynamic executable

The seff executable is a perl script.

$ env LD_LIBRARY_PATH=/usr/lib64/:$LD_LIBRARY_PATH seff 40000
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd
Job not found.

(In reply to Nate Rini from comment #8)
> Steve
> 
> Based on your output, LD is not trying to link against
> /usr/lib64/libslurm.so (or the tagged version /usr/lib64/libslurm.so.33) but
> is instead linking against /lib64/libslurm.so.33
> 
> Please provide the output to the following:
> ldd /usr/lib64/perl5/auto/Slurmdb/Slurmdb.so
> namei /usr/lib64/libslurm.so
> ldd /usr/lib64/libslurm.so
> ldd /lib64/libslurm.so.33
> 
> Please try:
> env LD_LIBRARY_PATH=/usr/lib64/:$LD_LIBRARY_PATH ldd $(which seff)
> env LD_LIBRARY_PATH=/usr/lib64/:$LD_LIBRARY_PATH seff
> 
> Thanks
> --Nate

Comment 20 Nate Rini 2018-10-19 15:53:06 MDT

I can confirm the issue and we are working on a solution.

Can you please provide the output of the following of your cluster:
lsb_release -a
grep . /etc/*release*

--Nate

Comment 21 Steve Ford 2018-10-20 13:04:17 MDT

# lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.4.1708 (Core) 
Release:	7.4.1708
Codename:	Core
# grep . /etc/*release*
/etc/centos-release:CentOS Linux release 7.4.1708 (Core) 
/etc/centos-release-upstream:Derived from Red Hat Enterprise Linux 7.4 (Source)
grep: /etc/lsb-release.d: Is a directory
/etc/os-release:NAME="CentOS Linux"
/etc/os-release:VERSION="7 (Core)"
/etc/os-release:ID="centos"
/etc/os-release:ID_LIKE="rhel fedora"
/etc/os-release:VERSION_ID="7"
/etc/os-release:PRETTY_NAME="CentOS Linux 7 (Core)"
/etc/os-release:ANSI_COLOR="0;31"
/etc/os-release:CPE_NAME="cpe:/o:centos:centos:7"
/etc/os-release:HOME_URL="https://www.centos.org/"
/etc/os-release:BUG_REPORT_URL="https://bugs.centos.org/"
/etc/os-release:CENTOS_MANTISBT_PROJECT="CentOS-7"
/etc/os-release:CENTOS_MANTISBT_PROJECT_VERSION="7"
/etc/os-release:REDHAT_SUPPORT_PRODUCT="centos"
/etc/os-release:REDHAT_SUPPORT_PRODUCT_VERSION="7"
/etc/redhat-release:CentOS Linux release 7.4.1708 (Core) 
/etc/system-release:CentOS Linux release 7.4.1708 (Core) 
/etc/system-release-cpe:cpe:/o:centos:centos:7

Comment 25 Nate Rini 2018-10-22 14:07:14 MDT

Steve

Can we get an output of the following:

rpm -qa
env LD_DEBUG=all seff

Thanks
--Nate

Comment 35 Nate Rini 2018-10-23 16:10:57 MDT

Steve

Please disregard the last request for information. We are working on verifying a patch to correct the issue.

Thanks
--Nate

Comment 40 Nate Rini 2018-10-24 17:02:49 MDT

Steve

This is fixed by commit 3d85c8f92 in 18.08.3 release that came out today. 

Please re-open this ticket if there are any further issues.

Thanks
--Nate

Comment 41 Felip Moll 2018-10-25 04:28:50 MDT

*** Ticket 5796 has been marked as a duplicate of this ticket. ***

Comment 42 Nate Rini 2018-11-01 14:13:45 MDT

*** Ticket 5892 has been marked as a duplicate of this ticket. ***