forked from SchedMD/slurm
-
Notifications
You must be signed in to change notification settings - Fork 0
/
NEWS
14253 lines (13923 loc) · 808 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and administrators.
* Changes in Slurm 24.08.0rc1
=============================
-- Make slurmstepd retry REQUEST_COMPLETE_BATCH_SCRIPT indefinitely.
-- Always load serializer/json when using any data_parser plugins.
-- slurmrestd - Reject single http query with multiple path requests.
-- slurmrestd - Add time/planned field to slurmdb/v0.0.41/job/{job_id}.
-- Improve Power Save's Resume/Suspend rate limiting.
-- slurmrestd - Improve reliablity under high memory pressure by closing
connections instead of forcing a fatal exit due to lack of memory.
-- data_parser/v0.0.41 - Avoid aborting when invoking a not implemented
parser.
-- data_parser/v0.0.41 - Fix how nice values are parsed for job submissions.
-- data_parser/v0.0.41 - Fix regression where parsing error did not result in
invalid request being rejected.
-- Print an error message in 'scontrol reboot' when a node reboot request
is ignored due to the current node state.
-- squeue - Add "--notme" option.
-- data_parser/v0.0.41 - change "association.id" to just include the int "id"
rather than include redundant assoc info (cluster, user, partition, account)
that's already included in the "association" object.
-- data_parser/v0.0.41 - Improve parsing of numeric user id.
-- data_parser/v0.0.41 - Improve parsing of numeric group id.
* Changes in Slurm 23.11.2
==========================
-- slurmrestd - Reject single http query with multiple path requests.
-- Fix launching Singularity v4.x containers with srun --container by setting
.process.terminal to true in generated config.json when step has
pseudoterminal (--pty) requested.
-- Fix loading in dyanmic/cloud node jobs after net_cred expired.
-- Fix cgroup null path error on slurmd/slurmstepd tear down.
-- data_parser/v0.0.40 - Prevent failure if accounting is disabled, instead
issue a warning if needed data from the database can not be retrieved.
-- openapi/slurmctld - Prevent failure if accounting is disabled.
-- Prevent slurmscriptd processing delays from blocking other threads in
slurmctld while trying to launch various scripts. This is additional work
for a fix in 23.02.6.
-- Fix memory leak when receiving alias addrs from controller.
-- scontrol - Accept `scontrol token lifespan=infinite` to create tokens that
effectively do not expire.
-- Avoid errors when Slurmdb accounting disabled when '--json' or '--yaml' is
invoked with CLI commands and slurmrestd. Add warnings when query would
have populated data from Slurmdb instead of errors.
-- Fix slurmctld memory leak when running job with --tres-per-task=gres:shard:#
-- Fix backfill trying to start jobs outside of backfill window.
-- Fix oversubscription on partitions with PreemptMode=OFF.
-- Preserve node reason on power up if the node is downed or drained.
-- data_parser/v0.0.40 - Avoid aborting when invoking a not implemented
parser.
-- data_parser/v0.0.40 - Fix how nice values are parsed for job submissions.
-- data_parser/v0.0.40 - Fix regression where parsing error did not result in
invalid request being rejected.
-- Fix segfault in front-end node registration.
-- Prevent jobs using none typed gpus from being killed by the controller after
a reconfig or restart.
-- Fix deadlock situation in the dbd when adding associations.
-- Update default values of text/blob columns when updating from old mysql
versions in more situations. This improves a previous fix to handle an
uncommon case when upgrading mysql/mariadb.
-- Fix rpmbuild in openSUSE/SLES due to incorrect mariadb dependency.
* Changes in Slurm 23.11.1
==========================
-- Fix scontrol update job=... TimeLimit+=/-= when used with a raw JobId of job
array element.
-- Reject TimeLimit increment/decrement when called on job with
TimeLimit=UNLIMITED.
-- Fix slurmctld segfault when reconfiguring after a job resize.
-- Fix compilation on FreeBSD.
-- Fix issue with requesting a job with --licenses as well as
--tres-per-task=license.
-- slurmctld - Prevent segfault in getopt_long() with an invalid long option.
-- Switch to man2html-base in Build-Depends for Debian package.
-- slurmrestd - Added /meta/slurm/cluster field to responses.
-- Adjust systemd service files to start daemons after remote-fs.target.
-- Add "--with selinux" option to slurm.spec.
-- Fix task/cgroup indexing tasks in cgroup plugins, which caused
jobacct/gather to match the gathered stats with the wrong task id.
-- select/linear - Fix regression in 23.11 in which jobs that requested
--cpus-per-task were rejected.
-- Fix crash in slurmstepd that can occur when launching tasks via mpi using
the pmi2 plugin and using the route/topology plugin.
-- Fix sgather not gathering from all nodes when using CR_PACK_NODES/--m pack.
-- Fix mysql query syntax error when getting jobs with private data.
-- Fix sanity check to prevent deleting default account of users.
-- data_parser/v0.0.40 - Fix the parsing for /slurmdb/v0.0.40/jobs exit_code
query parameter.
-- Fix issue where TRES for energy wasn't always set before sending it to the
jobcomp plugin.
-- jobcomp/[kafka|elastisearch] Print raw TRES values along with the
formatted versions as tres_[req|alloc]_raw.
-- Fix inconsistencies with --cpu-bind/SLURM_CPU_BIND and --hint/SLURM_HINT.
-- Fix ignoring invalid json in various subsystems.
-- Remove shebang from bash completion script.
-- Fix elapsed time in JobComp being set from invalid start and end times.
-- Update service files to start slurmd, slurmctld, and slurmdbd after sssd.
-- data_parser/v0.0.40 - Fix output of DefMemPerCpu, MaxMemPerCpu, and
max_shares.
-- When determining a jobs index in the database don't wait if there are more
jobs waiting.
-- If a job requests more shards which would allocate more than one sharing
GRES (gpu) per node refuse it unless SelectTypeparameters has
MULTIPLE_SHARING_GRES_PJ.
-- Avoid refreshing the hwloc xml file when slurmd is reconfigured. This fixes
an issue seen with CoreSpecCount used on nodes with Intel E-cores.
-- Trigger fatal exit when Slurm API function is called before slurm_init() is
called.
-- slurmd - Fix issue with 'scontrol reconfigure' when started with '-c'.
-- data_parser/v0.0.40 - Fix handling of negative job nice values.
-- data_parser/v0.0.40 - Fill the "id" object for associations with the
cluster, account, partition, and user in addition to the assoc id.
-- data_parser/v0.0.40 - Remove unusable cpu_binding_flags enums from
v00.0.40_job_desc_msg.
-- Improve performance and resiliency of slurmscriptd shutdown on
'scontrol reconfigure'.
-- slurmrestd - Job submissions that result in the following error codes
will be considered as successfully submitted (with a warning), instead
of returning an HTTP 500 error back:
ESLURM_NODES_BUSY, ESLURM_RESERVATION_BUSY, ESLURM_JOB_HELD,
ESLURM_NODE_NOT_AVAIL, ESLURM_QOS_THRES, ESLURM_ACCOUNTING_POLICY,
ESLURM_RESERVATION_NOT_USABLE, ESLURM_REQUESTED_PART_CONFIG_UNAVAILABLE,
ESLURM_BURST_BUFFER_WAIT, ESLURM_PARTITION_DOWN,
ESLURM_LICENSES_UNAVAILABLE.
-- Fix issue with node appearing to reboot on every "scontrol reconfigure"
when slurmd was started with the '-b' flag.
-- Fix a slurmctld fatal error when upgrading to 23.11 and changing from
select/cons_res to select/cons_tres at the same time.
-- slurmctld - Fix subsequent reconfigure hanging after a failed reconfigure.
-- slurmctld - Reject arbitrary distribution jobs that have a minimum node
count that differs from the number of unique nodes in the hostlist.
-- Prevent slurmdbd errors when updating reservations with names containing
apostrophes.
-- Prevent message extension attacks that could bypass the message hash.
CVE-2023-49933.
-- Prevent SQL injection attacks in slurmdbd. CVE-2023-49934.
-- Prevent message hash bypass in slurmd which can allow an attacker to reuse
root-level MUNGE tokens and escalate permissions. CVE-2023-49935.
-- Prevent NULL pointer dereference on size_valp overflow. CVE-2023-49936.
-- Prevent double-xfree() on error in _unpack_node_reg_resp().
CVE-2023-49937.
* Changes in Slurm 23.11.0
==========================
-- For jobs that request --cpus-per-gpu, ensure that the --cpus-per-gpu request
is honored on every node in the and not just for the job as a whole.
-- Fix "srun -Z" for cred/munge and cred/none.
-- Fix listing available data_parser plugins for json and yaml when giving no
commands to scontrol or sacctmgr.
-- data_parser/v0.0.40 - Fixed how deleted QOS and associations for jobs are
dumped.
-- data_parser/v0.0.40 - Fix how errors and warnings are dumped.
-- Print consistent errors when serializer plugin fails to load.
-- data_parser/v0.0.40 - Fix parsing of flag arrays to allow muliple flags to
be set.
-- slurmctld - Rework 'scontrol reconfigure' to avoid race conditions that
can result in stray jobs.
-- slurmctld - Shave ~1 second off average reconfigure time by terminating
internal processing threads faster.
-- Skip running slurmdbd -R if the connected cluster is 23.11 or newer.
This operation is nolonger relevant for 23.11.
-- Fix segfault when updating node instance id/type without accounting enabled.
-- Ensure slurmscriptd shuts down before slurmctld is stopped / reconfigured.
-- Improve error handling and error messages in slurmctld to slurmscriptd
communications. This includes avoiding potential deadlock in slurmctld if
slurmscript dies unexpectedly.
-- Do not hold batch jobs whose extra constraints cannot be immediately
satisfied, and set the state reason to "Constraints" instead of
"BadConstraints".
-- Fix verbose log message printing a hex number instead of a job id.
-- Upgrade rate limit parameters message from debug to info.
-- Fix missing symbols for those linking to libslurm.
-- Fix memory leak when getting and forwarding addrs from client.
-- Fix xassert when forwarding to non-addressable nodes.
-- Fix regression in 23.11.0rc1 where assocs were created with an incorrect
hierarchy for non-23.11 clusters.
-- For SchedulerParameters=extra_constraints, prevent slurmctld segfault when
starting a slurmd with --extra for a node that did not previously set this.
This also ensures the extra constraints model works off the current node
state, not the prior state.
-- Fix regression in 23.11.0rc1 where data_t would not decode negative float
values correctly, instead the absolute value was always returned.
-- Fix issue where 'scontrol reconfigure' right as the controller was started
or reconfigured could lead to it shutting down completely.
-- Fix --tres-per-task assertion.
-- Fix a few issues when creating reservations.
-- Fix slurmctld segfault when packing a job step loaded from a < 23.11 state.
-- Fix a 32-bit compile issue.
-- Add SchedulerParameters=time_min_as_soft_limit option.
* Changes in Slurm 23.11.0rc1
=============================
-- task/affinity - remove Power7 cpu-specific workarounds.
-- Remove SLURM_WORKING_CLUSTER env from batch and srun environments.
-- cli_filter/lua - return nil for unset time options rather than the string
"2982616-04:14:00" (which is the internal macro "NO_VAL" represented as
time string).
-- Remove 'none' plugins for all but auth and cred. scontrol show config
will report (null) now.
-- Removed select/cons_res. Please update your configuration to
select/cons_tres.
-- mpi/pmix - When aborted with status 0, avoid marking job/step as failed.
-- Fixed typo on "initialized" for the description of ESLURM_PLUGIN_NOT_LOADED.
-- Added max_submit_line_size to SchedulerParameters.
-- Change TreeWidth default from 50 to 16.
-- cgroup.conf - Removed deprecated parameters AllowedKmemSpace,
ConstrainKmemSpace, MaxKmemPercent, and MinKmemSpace.
-- proctrack/cgroup - Add "SignalChildrenProcesses=<yes|no>" option to
cgroup.conf. This allows signals for cancelling, suspending, resuming, etc.
to be sent to children processes in a step/job rather than just the parent.
-- Add PreemptParameters=suspend_grace_time parameter to control amount of
time between SIGTSTP and SIGSTOP signals when suspending jobs.
-- job_submit/throttle - improve reset of submitted job counts per user in
order to better honor SchedulerParameters=jobs_per_user_per_hour=#.
-- Load the user environment into a private pid namespace to avoid user scripts
leaving background processes on a node.
-- scontrol show assoc_mgr will display Lineage instead of Lft for
associations.
-- Add SlurmctldParameters=no_quick_restart to avoid a new slurmctld taking
over the old slurmctld on accedent.
-- Fix --cpus-per-gpu for step allocations, which was previously ignored for
job steps. --cpus-per-gpu implies --exact.
-- Fix mutual exclusivity of --cpus-per-gpu and --cpus-per-task: fatal if both
options are requested in the commandline or both are requested in the
environment. If one option is requested in the command line, it will
override the other option in the environment.
-- slurmrestd - openapi/dbv0.0.37 and openapi/v0.0.37 plugins have been
removed.
-- slurmrestd - openapi/dbv0.0.38 and openapi/v0.0.38 plugins have been tagged
as deprecated.
-- openapi/slurmctld - forked from openapi/v0.0.38.
-- openapi/slurmdbd - forked from openapi/dbv0.0.38.
-- data_parser/v0.0.40 - forked from data_parser/v0.0.39 plugin.
-- data_parser/v0.0.40 - added OpenAPI schema generation of path parameters
and OperationIds.
-- slurmrestd - added auto population of info/version field.
-- openapi/slurmctld - convert to using data_parser plugins for all input and
output formatting.
-- openapi/slurmdbd - convert to using data_parser plugins for all input and
output formatting.
-- data_parser/v0.0.39 - skip empty string when parsing QOS ids.
-- data_parser/v0.0.40 - skip empty string when parsing QOS ids.
-- data_parser/v0.0.40 - log errors on every level of parsing on failure.
-- sdiag - add --yaml and --json arg support to specify data_parser plugin.
-- sacct - add --yaml and --json arg support to specify data_parser plugin.
-- scontrol - add --yaml and --json arg support to specify data_parser plugin.
-- sinfo - add --yaml and --json arg support to specify data_parser plugin.
-- squeue - add --yaml and --json arg support to specify data_parser plugin.
-- data_parser/v0.0.40 - add warnings for unknown fields during parsing.
-- data_parser/v0.0.40 - add FAST parameter to allow requestor to skip more
time intensive warning checks.
-- Changed the default SelectType to select/cons_tres (from select/linear).
-- Allow SlurmUser/root to use reservations without specific permissions.
-- Fix sending step signals to nodes not allocated by the step.
-- Remove CgroupAutomount= option from cgroup.conf.
-- Add TopologyRoute=RoutePart to route communications based on partition node
lists.
-- SPANK - added new spank_prepend_task_argv() function.
-- slurmd - improve error logging at job startup during transition from slurmd
to slurmstepd.
-- Added ability for configless to push Prolog and Epilog scripts to slurmds.
-- Prolog and Epilog do not have to be fully qualified pathnames.
-- Changed default value of PriorityType from priority/basic to
priority/multifactor.
-- torque/mpiexec - Propogate exit code from launched process.
-- slurmrestd - Add new rlimits fields for job submission.
-- data_parser/v0.0.40 - convert job state field to flag array to provide
enumeration of values.
-- sbatch - removed --export-file option (used with defunct Moab integration).
-- Define SPANK options environment variables when --export=[NIL|NONE] is
specified.
-- slurmrestd - Numeric input fields provided with a null formatted value will
now convert to zero (0) where it can be a valid value. This is expected to
be only be notable with job submission against v0.0.38 versioned endpoints
with job requests with fields provided with null values. These fields were
already rejected by v0.0.39+ endpoints, unless +complex parser value is
provided to v0.0.40+ endpoints.
-- slurmrestd - Improve parsing of integers and floating point numbers when
handling incoming user provided numeric fields. Fields that would have not
rejected a number for a numeric field followed by other non-numeric
characters will now get rejected. This is expected to be only be notable
with job submission against v0.0.38 versioned endpoints with malformed job
requests.
-- Reject reservation update if it will result in previously submitted
jobs losing access to the reservation.
-- data_parser/v0.0.40 - output partition state when dumping partitions.
-- Allow for a shared suffix to be used with the hostlist format. E.g.,
"node[0001-0010]-int".
-- Fix perlapi build when using non-default libdir.
-- Replace SRUN_CPUS_PER_TASK with SLURM_CPUS_PER_TASK and get back the
previous behavior before Slurm 22.05 since now we have the new external
launcher step.
-- data_parser/v0.0.40 - Change v0.0.40_job_info response to tag exit_code
field as verbose job exit code object.
-- data_parser/v0.0.40 - Change v0.0.40_job_info response to tag
derived_exit_code field as verbose job exit code object.
-- Avoid database upgrade failures with galera by enabling streaming
replication for Galera 4 clusters during the upgrade process.
-- job_container/tmpfs - disable plugin for nodes that are not listed in
job_container.conf when there is no global BasePath set.
-- job_container/tmpfs - Add "BasePath=none" option to disable plugin on node
subsets when there is a global setting.
-- Remove cloud_reg_addrs and make it default behavior.
-- Remove NoAddrCache CommunicationParameter.
-- Add QOS flag 'Relative'. If set the QOS limits will be treated as
percentages of a cluster/partition instead of absolutes.
-- Remove FIRST_CORES flag from reservations.
-- scontrol/sview - Remove comma separated CoreCnt option from reservations.
-- scontrol/sview - Remove comma separated NodeCnt option from reservations.
-- Add cloud instance id and instance type to node records. Can be viewed/
updated with scontrol.
-- slurmd - add "instance-id", "instance-type", and "extra" options to allow
them to be set on startup.
-- Add cloud instance accounting to database that can be viewed with 'sacctmgr
show instance'.
-- openapi/v0.0.40 - add /instance and /instances endpoints.
-- SelectTypeParameters=cr_cpu - Fix a log error "CPU underflow" at step
completion for steps that request --threads-per-core or
--hint=nomultithread.
-- select/linear - fix task launch failure that sometimes occurred when
requesting --threads-per-core or --hint=nomultithread. This also fixes
memory calculation with one of these options and --mem-per-cpu:
Previously, memory = mem-per-cpu * all cpus including unusable threads.
Now, memory = mem-per-cpu * only usuable threads. This behavior matches
the documentation and select/cons_tres.
-- gpu/nvml - Reduce chances of NVML_ERROR_INSUFFICIENT_SIZE error when getting
gpu memory information.
-- slurmrestd - Convert to generating OperationIDs based on path for all
v0.0.40 tagged paths.
-- slurmrestd - Reduce memory used while dumping a job's stdio paths.
-- slurmrestd - Jobs queried from data_parser/v0.0.40 from slurmdb will have
'step/id' field given as a string to match CLI formatting instead of an
object.
-- sacct - Output in JSON or YAML output will will have the 'step/id' field
given as a string instead of an object.
-- scontrol/squeue - Step output in JSON or YAML output will will have the
'id' field given as a string instead of an object.
-- slurmrestd - For 'GET /slurmdb/v0.0.40/jobs' mimick default behavior for
handling of job start and end times as sacct when one or both fields are
not provided as a query parameter.
-- openapi/slurmctld - Add 'GET /slurm/v0.0.40/shares' endpoint to dump same
output as sshare.
-- sshare - add JSON/YAML support.
-- data_parser/v0.0.40 - Remove "required/memory" output in json. It is
replaced by "required/memory_per_cpu" and "required/memory_per_node".
-- slurmrestd - Add numeric id to all association identifiers to allow unique
identification where association has been deleted but is still referenced by
accounting record.
-- slurmrestd - Add accounting, id, and comment fields to association dumps.
-- slurmrestd - Removed usage field from association dumps which was never
populated. See accounting field for accounting usage records.
-- slurmrestd - Default to not query assocations or coordinators with
'GET /slurmdb/v0.0.40/accounts'.
-- slurmrestd - Default to not query assocations, wckeys or coordinators with
'GET /slurmdb/v0.0.40/user'.
-- slurmrestd - Enforce user's default wckey on supplied list of user wckeys in
'POST /slurmdb/v0.0.40/user' queries to avoid conflicting or changed default
wckey from being ignored.
-- slurmrestd - Enforce user's default wckey on supplied list of user wckeys in
'POST /slurmdb/v0.0.40/user' queries to avoid conflicting or changed default
wckey from being ignored.
-- slurmrestd - 'POST /slurm/v0.0.40/job/submit' will return "step_id" as
string to provide descriptive step names (batch, extern, interactive, TBD)
for non-numeric steps.
-- slurmrestd - Tagged "result" field from 'POST /slurm/v0.0.40/job/submit'
as deprecated.
-- slurmrestd - Warning will be added for rejected job submissions with
submissions to 'POST /slurm/v0.0.40/job/submit'.
-- slurmrestd - Tagged "job_id", "step_id", and "job_submit_user_msg" fields
from 'POST /slurm/v0.0.40/job/{job_id}' response as deprecated due their
only being valid for the first entry in the "result" field array.
-- slurmrestd - Warning will be added for rejected job updates with
queries to 'POST /slurm/v0.0.40/job/{job_id}'.
-- slurmrestd - Add SLURMRESTD_JSON and SLURMRESTD_YAML input environment
variables.
-- slurmrestd - Correct issue where field and $ref description fields were not
getting populated for OpenAPI specification generation for queries to
'GET /openapi/v3' for v0.0.40 endpoints.
-- slurmdbd - Check for innodb_redo_log_capacity instead of
innodb_log_file_size in MySQL 8.0.30+.
-- slurmrestd - Fix log level requests SLURMRESTD_DEBUG and -v applying on top
of each other and the default logging level (info). -v now applies on top
of the default log level, and SLURMRESTD_DEBUG sets the log level if -v is
not given.
-- slurmrestd - Allow SLURMRESTD_DEBUG=quiet or 0, which was previously denied.
Also deny negative values for SLURMRESTD_DEBUG, which previously set the
debug level to debug5.
-- The backup slurmctld now checks that the heartbeat file exists in
StateSaveLocation before starting and attempting to assert control.
This avoids issues with misconfiguration, or the shared filesystem being
unavailable, that could previously have lead to all jobs being cancelled.
-- The warning printed when using configure --without-PACKAGE has been changed
to a notice.
-- Fix --cpu-freq with userspace governor and frequency ranges behavior.
-- Fix --cpu-freq parsing with incorrect frequencies.
-- PMIx support is nolonger built by default. Passing --with-pmix option is
now required to build with PMIx.
-- Use memory.current in cgroup/v2 instead of manually calculating RSS. This
makes accounting consistent with OOM Killer.
-- Update slurmstepd processes with current SlurmctldHost settings, allowing
for controller changes without draining all compute jobs.
-- Add format_stderr to LogTimeFormat of slurm.conf and slurmdbd.conf.
-- slurmrestd - add `GET /slurm/v0.0.40/reconfigure` endpoint to allow
equivalent requests of `scontrol reconfigure`.
-- sreport - cluster Utilization PlannedDown field now includes the time that
all nodes were in the POWERED_DOWN state instead of just cloud nodes.
-- scontrol update partition now allows Nodes+=<node-list> and
Nodes-=<node-list> to add/delete nodes from the existing partition node
list. Nodes=+host1,-host2 is also allowed.
-- sacctmgr - add --yaml and --json arg support to specify data_parser plugin.
-- slurmrestd - Add last_update and last_backfill fields to response to
`GET /slurm/v0.0.40/job` and `GET /slurm/v0.0.40/jobs` queries.
-- slurmrestd - Add last_update fields to response to
`GET /slurm/v0.0.40/node` and `GET /slurm/v0.0.40/nodes` queries.
-- slurmrestd - Add last_update fields to response to
`GET /slurm/v0.0.40/partition` and `GET /slurm/v0.0.40/partitions` queries.
-- slurmrestd - Add last_update fields to response to
`GET /slurm/v0.0.40/licenses` query.
-- auth/jwt - fatal when jwt or jwks key files are writable by other.
-- sacctmgr can now modify QOS's RawUsage to zero or a positive value.
-- sdiag - Added statistics on why the main and backfill schedulers have
stopped evaluation on each scheduling cycle.
-- openapi/v0.0.40 - add /{accounts,users}_association endpoints.
-- slurm.spec - Add `--with yaml` argument to require YAML support.
-- Add new rl_log_freq option to SlurmctldParameters to allow sites to limit
the number of 'RPC limit exceeded...' messages that are logged.
-- Rename sbcast --fanout to --treewidth.
-- Remove SLURM_NODE_ALIASES env variable.
-- Enable fanout for dynamic and unaddresable cloud nodes.
-- Fix how steps are dealloced in an allocation if the last step of an srun
never completes due to a node failure.
-- Remove redundant database indexes.
-- Add database index to suspend table to speed up archive/purges.
-- When requesting --tres-per-task alter incorrect request for TRES,
it should be TRESType/TRESName not TRESType:TRESName.
-- Make it so reservations can reserve GRES.
-- switch/hpe_slingshot - Add disable_rdzv_get flag to disable rendezvous gets.
-- slurmrestd - Avoid matching query URLs based on partial matches to an
existing endpoint, e.g. POST request to /slurm/v0.0.40/job was routed
to the handler of POST /slurm/v0.0.40/job/submit.
-- Don't display old job_arrays/het_jobs in sacct if Job ID was reused.
-- sbcast - use the specified --fanout value on all hops in message
forwarding; previously the specified fanout was only used on the first hop,
and additional hops used TreeWidth in slurm.conf.
-- slurmrestd - remove logger prefix from '-s/-a list' options outputs.
-- Fix fd socket name resolution which could flood log files at debug level.
-- The rpmbuild "--with mysql" option has been removed. The rpm has long
required sql development libraries to build and the existence of this option
was confusing. The default behavior now is to always require one of the sql
development libraries.
-- Add support for Debian packaging.
-- switch/hpe_slingshot - Add support for collectives.
-- Nodes with suspended jobs can now be displayed as MIXED.
-- sview - Fix search by node state returning incorrect node list.
-- Fix inconsistent handling of using cli and/or environment options for
tres_per_task=cpu:# and cpus_per_gpu.
-- Requesting --cpus-per-task will now set SLURM_TRES_PER_TASK=cpu:# in the
environment.
-- For some tres related environment variables such as SLURM_TRES_PER_TASK,
when srun requests a different value for that option, set these environment
variables to the value requested by srun. Previously these environment
variables were unchanged from the job allocation. This bug only affected the
output environment variables, not the actual step resource allocation.
-- RoutePlugin=route/topology has been replaced with TopologyParam=RouteTree.
-- If ThreadsPerCore in slurm.conf is configured with less
than the number of hardware threads, fix a bug where the task plugins used
fewer cores instead of using fewer threads per core.
-- Fix arbitrary distribution allowing it to be used with salloc and sbatch and
fix how cpus are allocated to nodes.
-- Allow nodes to reboot while node is drained or in a maintenance state.
-- Allow scontrol reboot to use nodesets to filter nodes to reboot.
-- Fix how the topology of typed gres gets updated.
-- Changes to the Type option in gres.conf now can be applied with scontrol
reconfig.
-- Allow for jobs that request a newly configured gres type to be queued
even when the needed slurmds have not yet registered.
-- Kill recovered jobs that require unconfigured gres types.
-- If keepalives are configured, enable them on all persistent connections.
-- data_parser/v0.0.40 - add parsers for main/backfill cycle exit reasons.
-- Configless - Also send Includes from configuration files not parsed by the
controller (i.e. from plugstack.conf).
-- Add gpu/nrt plugin for nodes using Trainium/Inferentia devices.
-- data_parser/v0.0.40 - Add START_RECEIVED to job flags in dumped output.
-- SPANK - Failures from most spank functions (not epilog or exit) will now
cause the step to be marked as failed and the command (srun, salloc,
sbatch --wait) to return 1.
-- data_parser/v0.0.40 - Fix how the "INVALID" nodes state is dumped.
-- Add SchedulerParameters=extra_constraints. This enables various node
filtering options in the --extra flag of salloc, sbatch, and srun.
-- Improve scontrol show node -d information of used shared gres (shard/mps)
topology.
* Changes in Slurm 23.02.8
==========================
* Changes in Slurm 23.02.7
==========================
-- libslurm_nss - Avoid causing glibc to assert due to an unexpected return
from slurm_nss due to an error during lookup.
-- Fix job requests with --tres-per-task sometimes resulting in bad allocations
that cannot run subsequent job steps.
-- Fix issue with slurmd where srun fails to be warned when a node prolog
script runs beyond MsgTimeout set in slurm.conf.
-- gres/shard - Fix plugin functions to have matching parameter orders.
-- gpu/nvml - Fix issue that resulted in the wrong MIG devices being
constrained to a job
-- gpu/nvml - Fix linking issue with MIGs that prevented multiple MIGs being
used in a single job for certain MIG configurations
-- Add JobAcctGatherParams=DisableGPUAcct to disable gpu accounting.
-- Fix file descriptor leak in slurmd when using acct_gather_energy/ipmi with
DCMI devices.
-- sview - avoid crash when job has a node list string > 49 characters.
-- Prevent slurmctld crash during reconfigure when packing job start messages.
-- Preserve reason uid on reconfig.
-- Update node reason with updated INVAL state reason if different from last
registration.
-- acct_gather_energy/ipmi - Improve logging of DCMI issues.
-- conmgr - Avoid NULL dereference when using auth/none.
-- data_parser/v0.0.39 - Fixed how deleted QOS and associations for jobs are
dumped.
-- burst_buffer/lua - fix stage in counter not decrementing when a job is
cancelled during stage in. This counter is used to enforce the limit of 128
scripts per stage.
-- gpu/oneapi - Add support for new env vars ZE_FLAT_DEVICE_HIERARCHY and
ZE_ENABLE_PCI_ID_DEVICE_ORDER.
-- data_parser/v0.0.39 - Fix how the "INVALID" nodes state is dumped.
-- data_parser/v0.0.39 - Fix parsing of flag arrays to allow muliple flags to
be set.
-- Avoid leaking sockets when an x11 application is closed in an allocation.
-- Fix missing mutex unlock in group cache code which could cause slurmctld to
freeze.
-- Fix scrontab monthly jobs possibly skipping a month if added near the end of
the month.
-- Fix loading of the gpu account gather energy plugin.
-- Fix slurmctld segfault when reconfiguring after a job resize.
-- Fix crash in slurmstepd that can occur when launching tasks via mpi using
the pmi2 plugin and using the route/topology plugin.
-- data_parser/v0.0.39 - skip empty string when parsing QOS ids.
-- Fix "qos <id> doesn't exist" error message in assoc_mgr_update_assocs to
print the attempted new default qos, rather than the current default qos.
-- Remove error message from assoc_mgr_update_assocs when purposefully
resetting the default qos.
-- data_parser/v0.0.39 - Fix segfault when POSTing data with association usage.
-- Prevent message extension attacks that could bypass the message hash.
CVE-2023-49933.
-- Prevent message hash bypass in slurmd which can allow an attacker to reuse
root-level MUNGE tokens and escalate permissions. CVE-2023-49935.
-- Prevent NULL pointer dereference on size_valp overflow. CVE-2023-49936.
-- Prevent double-xfree() on error in _unpack_node_reg_resp(). CVE-2023-49937.
-- Prevent modified sbcast RPCs from opening a file with the wrong group
permissions. CVE-2023-49938.
* Changes in Slurm 23.02.6
==========================
-- Fix CpusPerTres= not upgreadable with scontrol update
-- Fix unintentional gres removal when validating the gres job state.
-- Fix --without-hpe-slingshot configure option.
-- Fix cgroup v2 memory calculations when transparent huge pages are used.
-- Fix parsing of sgather --timeout option.
-- Fix regression from 22.05.0 that caused srun --cpu-bind "=verbose" and "=v"
options give different CPU bind masks.
-- Fix "_find_node_record: lookup failure for node" error message appearing
for all dynamic nodes during reconfigure.
-- Avoid segfault if loading serializer plugin fails.
-- slurmrestd - Correct OpenAPI format for 'GET /slurm/v0.0.39/licenses'.
-- slurmrestd - Correct OpenAPI format for 'GET /slurm/v0.0.39/job/{job_id}'.
-- slurmrestd - Change format to multiple fields in 'GET
/slurmdb/v0.0.39/assocations' and 'GET /slurmdb/v0.0.39/qos' to handle
infinite and unset states.
-- When a node fails in a job with --no-kill, preserve the extern step on the
remaining nodes to avoid breaking features that rely on the extern step
such as pam_slurm_adopt, x11, and job_container/tmpfs.
-- auth/jwt - Ignore 'x5c' field in JWKS files.
-- auth/jwt - Treat 'alg' field as optional in JWKS files.
-- Allow job_desc.selinux_context to be read from the job_submit.lua script.
-- Skip check in slurmstepd that causes a large number of errors in the munge
log: "Unauthorized credential for client UID=0 GID=0". This error will
still appear on slurmd/slurmctld/slurmdbd start up and is not a cause for
concern.
-- slurmctld - Allow startup with zero partitions.
-- Fix some mig profile names in slurm not matching nvidia mig profiles.
-- Prevent slurmscriptd processing delays from blocking other threads in
slurmctld while trying to launch {Prolog|Epilog}Slurmctld.
-- Fix sacct printing ReqMem field when memory doesn't exist in requested TRES.
-- Fix how heterogenous steps in an allocation with CR_PACK_NODE or -mpack are
created.
-- Fix slurmctld crash from race condition within job_submit_throttle plugin.
-- Fix --with-systemdsystemunitdir when requesting a default location.
-- Fix not being able to cancel an array task by the jobid (i.e. not
<jobid>_<taskid>) through scancel, job launch failure or prolog failure.
-- Fix cancelling the whole array job when the array task is the meta job and
it fails job or prolog launch and is not requeable. Cancel only the
specific task instead.
-- Fix regression in 21.08.2 where MailProg did not run for mail-type=end for
jobs with non-zero exit codes.
-- Fix incorrect setting of memory.swap.max in cgroup/v2.
-- Fix jobacctgather/cgroup collection of disk/io, gpumem, gpuutil TRES values.
-- Fix -d singleton for heterogeneous jobs.
-- Downgrade info logs about a job meeting a "maximum node limit" in the
select plugin to DebugFlags=SelectType. These info logs could spam the
slurmctld log file under certain circumstances.
-- prep/script - Fix [Srun|Task]<Prolog|Epilog> missing SLURM_JOB_NODELIST.
-- gres - Rebuild GRES core bitmap for nodes at startup. This fixes error:
"Core bitmaps size mismatch on node [HOSTNAME]", which causes jobs to enter
state "Requested node configuration is not available".
-- slurmctd - Allow startup with zero nodes.
-- Fix filesystem handling race conditions that could lead to an attacker
taking control of an arbitrary file, or removing entire directories'
contents. CVE-2023-41914.
-- Return error when updating ArrayTaskThrottle on a non-ArrayJobId.
* Changes in Slurm 23.02.5
==========================
-- Add the JobId to debug() messages indicating when cpus_per_task/mem_per_cpu
or pn_min_cpus are being automatically adjusted.
-- Fix regression in 23.02.2 that caused slurmctld -R to crash on startup if
a node features plugin is configured.
-- Fix and prevent reoccurring reservations from overlapping.
-- job_container/tmpfs - Avoid attempts to share BasePath between nodes.
-- Change the log message warning for rate limited users from verbose to info.
-- With CR_Cpu_Memory, fix node selection for jobs that request gres and
--mem-per-cpu.
-- Fix a regression from 22.05.7 in which some jobs were allocated too few
nodes, thus overcommitting cpus to some tasks.
-- Fix a job being stuck in the completing state if the job ends while the
primary controller is down or unresponsive and the backup controller has
not yet taken over.
-- Fix slurmctld segfault when a node registers with a configured CpuSpecList
while slurmctld configuration has the node without CpuSpecList.
-- Fix cloud nodes getting stuck in POWERED_DOWN+NO_RESPOND state after not
registering by ResumeTimeout.
-- slurmstepd - Avoid cleanup of config.json-less containers spooldir getting
skipped.
-- slurmstepd - Cleanup per task generated environment for containers in
spooldir.
-- Fix scontrol segfault when 'completing' command requested repeatedly in
interactive mode.
-- Properly handle a race condition between bind() and listen() calls in the
network stack when running with SrunPortRange set.
-- Federation - Fix revoked jobs being returned regardless of the -a/--all
option for privileged users.
-- Federation - Fix canceling pending federated jobs from non-origin clusters
which could leave federated jobs orphaned from the origin cluster.
-- Fix sinfo segfault when printing multiple clusters with --noheader option.
-- Federation - fix clusters not syncing if clusters are added to a federation
before they have registered with the dbd.
-- Change pmi2 plugin to honor the SrunPortRange option. This matches the new
behavior of the pmix plugin in 23.02.0. Note that neither of these plugins
makes use of the "MpiParams=ports=" option, and previously were only limited
by the systems ephemeral port range.
-- node_features/helpers - Fix node selection for jobs requesting changeable
features with the '|' operator, which could prevent jobs from running on
some valid nodes.
-- node_features/helpers - Fix inconsistent handling of '&' and '|', where an
AND'd feature was sometimes AND'd to all sets of features instead of just
the current set. E.g. "foo|bar&baz" was interpreted as {foo,baz} or
{bar,baz} instead of how it is documented: "{foo} or {bar,baz}".
-- Fix job accounting so that when a job is requeued its allocated node count
is cleared. After the requeue, sacct will correctly show that the job has
0 AllocNodes while it is pending or if it is canceled before restarting.
-- sacct - AllocCPUS now correctly shows 0 if a job has not yet received an
allocation or if the job was canceled before getting one.
-- Fix intel oneapi autodetect: detect the /dev/dri/renderD[0-9]+ gpus, and do
not detect /dev/dri/card[0-9]+.
-- Format batch, extern, interactive, and pending step ids into strings that
are human readable.
-- Fix node selection for jobs that request --gpus and a number of tasks fewer
than gpus, which resulted in incorrectly rejecting these jobs.
-- Remove MYSQL_OPT_RECONNECT completely.
-- Fix cloud nodes in POWERING_UP state disappearing (getting set to FUTURE)
when an `scontrol reconfigure` happens.
-- openapi/dbv0.0.39 - Avoid assert / segfault on missing coordinators list.
-- slurmrestd - Correct memory leak while parsing OpenAPI specification
templates with server overrides.
-- slurmrestd - Reduce memory usage when printing out job CPU frequency.
-- Fix overwriting user node reason with system message.
-- Remove --uid / --gid options from salloc and srun commands.
-- Prevent deadlock when rpc_queue is enabled.
-- slurmrestd - Correct OpenAPI specification generation bug where fields with
overlapping parent paths would not get generated.
-- Fix memory leak as a result of a partition info query.
-- Fix memory leak as a result of a job info query.
-- slurmrestd - For 'GET /slurm/v0.0.39/node[s]', change format of node's
energy field "current_watts" to a dictionary to account for unset value
instead of dumping 4294967294.
-- slurmrestd - For 'GET /slurm/v0.0.39/qos', change format of QOS's
field "priority" to a dictionary to account for unset value instead of
dumping 4294967294.
-- slurmrestd - For 'GET /slurm/v0.0.39/job[s]', the 'return code' code field
in v0.0.39_job_exit_code will be set to -127 instead of being left unset
where job does not have a relevant return code.
-- data_parser/v0.0.39 - Add required/memory_per_cpu and
required/memory_per_node to `sacct --json` and `sacct --yaml` and
'GET /slurmdb/v0.0.39/jobs' from slurmrestd.
-- For step allocations, fix --gres=none sometimes not ignoring gres from the
job.
-- Fix --exclusive jobs incorrectly gang-scheduling where they shouldn't.
-- Fix allocations with CR_SOCKET, gres not assigned to a specific socket, and
block core distribion potentially allocating more sockets than required.
-- gpu/oneapi - Store cores correctly so CPU affinity is tracked.
-- Revert a change in 23.02.3 where Slurm would kill a script's process group
as soon as the script ended instead of waiting as long as any process in
that process group held the stdout/stderr file descriptors open. That change
broke some scripts that relied on the previous behavior. Setting time limits
for scripts (such as PrologEpilogTimeout) is strongly encouraged to avoid
Slurm waiting indefinitely for scripts to finish.
-- Allow slurmdbd -R to work if the root assoc id is not 1.
-- Fix slurmdbd -R not returning an error under certain conditions.
-- slurmdbd - Avoid potential NULL pointer dereference in the mysql plugin.
-- Revert a change in 23.02 where SLURM_NTASKS was no longer set in the job's
environment when --ntasks-per-node was requested.
-- Limit periodic node registrations to 50 instead of the full TreeWidth.
Since unresolvable cloud/dynamic nodes must disable fanout by setting
TreeWidth to a large number, this would cause all nodes to register at
once.
-- Fix regression in 23.02.3 which broken x11 forwarding for hosts when
MUNGE sends a localhost address in the encode host field. This is caused
when the node hostname is mapped to 127.0.0.1 (or similar) in /etc/hosts.
-- openapi/[db]v0.0.39 - fix memory leak on parsing error.
-- data_parser/v0.0.39 - fix updating qos for associations.
-- openapi/dbv0.0.39 - fix updating values for associations with null users.
-- Fix minor memory leak with --tres-per-task and licenses.
-- Fix cyclic socket cpu distribution for tasks in a step where
--cpus-per-task < usable threads per core.
* Changes in Slurm 23.02.4
==========================
-- Fix sbatch return code when --wait is requested on a job array.
-- switch/hpe_slingshot - avoid segfault when running with old libcxi.
-- Avoid slurmctld segfault when specifying AccountingStorageExternalHost.
-- Fix collected GPUUtilization values for acct_gather_profile plugins.
-- Fix slurmrestd handling of job hold/release operations.
-- Make spank S_JOB_ARGV item value hold the requested command argv instead of
the srun --bcast value when --bcast requested (only in local context).
-- Fix step running indefinitely when slurmctld takes more than MessageTimeout
to respond. Now, slurmctld will cancel the step when detected, preventing
following steps from getting stuck waiting for resources to be released.
-- Fix regression to make job_desc.min_cpus accurate again in job_submit when
requesting a job with --ntasks-per-node.
-- scontrol - Permit changes to StdErr and StdIn for pending jobs.
-- scontrol - Reset std{err,in,out} when set to empty string.
-- slurmrestd - mark environment as a required field for job submission
descriptions.
-- slurmrestd - avoid dumping null in OpenAPI schema required fields.
-- data_parser/v0.0.39 - avoid rejecting valid memory_per_node formatted as
dictionary provided with a job description.
-- data_parser/v0.0.39 - avoid rejecting valid memory_per_cpu formatted as
dictionary provided with a job description.
-- slurmrestd - Return HTTP error code 404 when job query fails.
-- slurmrestd - Add return schema to error response to job and license query.
-- Fix handling of ArrayTaskThrottle in backfill.
-- Fix regression in 23.02.2 when checking gres state on slurmctld startup or
reconfigure. Gres changes in the configuration were not updated on slurmctld
startup. On startup or reconfigure, these messages were present in the log:
"error: Attempt to change gres/gpu Count".
-- Fix potential double count of gres when dealing with limits.
-- switch/hpe_slingshot - support alternate traffic class names with "TC_"
prefix.
-- scrontab - Fix cutting off the final character of quoted variables.
-- Fix slurmstepd segfault when ContainerPath is not set in oci.conf
-- Change the log message warning for rate limited users from debug to verbose.
-- Fixed an issue where jobs requesting licenses were incorrectly rejected.
-- smail - Fix issues where e-mails at job completion were not being sent.
-- scontrol/slurmctld - fix comma parsing when updating a reservation's nodes.
-- cgroup/v2 - Avoid capturing log output for ebpf when constraining devices,
as this can lead to inadvertent failure if the log buffer is too small.
-- Fix --gpu-bind=single binding tasks to wrong gpus, leading to some gpus
having more tasks than they should and other gpus being unused.
-- Fix main scheduler loop not starting after failover to backup controller.
-- Added error message when attempting to use sattach on batch or extern steps.
-- Fix regression in 23.02 that causes slurmstepd to crash when srun requests
more than TreeWidth nodes in a step and uses the pmi2 or pmix plugin.
-- Reject job ArrayTaskThrottle update requests from unprivileged users.
-- data_parser/v0.0.39 - populate description fields of property objects in
generated OpenAPI specifications where defined.
-- slurmstepd - Avoid segfault caused by ContainerPath not being terminated by
'/' in oci.conf.
-- data_parser/v0.0.39 - Change v0.0.39_job_info response to tag exit_code
field as being complex instead of only an unsigned integer.
-- job_container/tmpfs - Fix %h and %n substitution in BasePath where %h was
substituted as the NodeName instead of the hostname, and %n was substituted
as an empty string.
-- Fix regression where --cpu-bind=verbose would override TaskPluginParam.
-- scancel - Fix --clusters/-M for federations. Only filtered jobs (e.g. -A,
-u, -p, etc.) from the specified clusters will be canceled, rather than all
jobs in the federation. Specific jobids will still be routed to the origin
cluster for cancellation.
-- Add SelectTypeParameters=LL_SHARED_GRES.
* Changes in Slurm 23.02.3
==========================
-- Fix regression in 23.02.2 that ignored the partition DefCpuPerGPU setting
on the first pass of scheduling a job requesting --gpus --ntasks.
-- openapi/dbv0.0.39/users - If a default account update failed, resulting in a
no-op, the query returned success without any warning. Now a warning is sent
back to the client that the default account wasn't modified.
-- srun - fix issue creating regular and interactive steps because
*_PACK_GROUP* environment variables were incorrectly set on non-HetSteps.
-- Fix dynamic nodes getting stuck in allocated states when reconfiguring.
-- Avoid job write lock when nodes are dynamically added/removed.
-- burst_buffer/lua - allow jobs to get scheduled sooner after
slurm_bb_data_in completes.
-- mpi/pmix - fix regression introduced in 23.02.2 which caused PMIx shmem
backed files permissions to be incorrect.
-- api/submit - fix memory leaks when submission of batch regular jobs or batch
HetJobs fails (response data is a return code).
-- openapi/v0.0.39 - fix memory leak in _job_post_het_submit().
-- Fix regression in 23.02.2 that set the SLURM_NTASKS environment variable
in sbatch jobs from --ntasks-per-node when --ntasks was not requested.
-- Fix regression in 23.02 that caused sbatch jobs to set the wrong number
of tasks when requesting --ntasks-per-node without --ntasks, and also
requesting one of the following options: --sockets-per-node,
--cores-per-socket, --threads-per-core (or --hint=nomultithread), or
-B,--extra-node-info.
-- Fix double counting suspended job counts on nodes when reconfiguring, which
prevented nodes with suspended jobs from being powered down or rebooted
once the jobs completed.
-- Fix backfill not scheduling jobs submitted with --prefer and --constraint
properly.
-- Avoid possible slurmctld segfault caused by race condition with already
completed slurmdbd_conn connections.
-- Slurmdbd.conf checks included conf files for 0600 permissions
-- slurmrestd - fix regression "oversubscribe" fields were removed from job
descriptions and submissions from v0.0.39 end points.
-- accounting_storage/mysql - Query for indiviual QOS correctly when you have
more than 10.
-- Add warning message about ignoring --tres-per-tasks=license when used
on a step.
-- sshare - Fix command to work when using priority/basic.
-- Avoid loading cli_filter plugins outside of salloc/sbatch/scron/srun. This
fixes a number of missing symbol problems that can manifest for executables
linked against libslurm (and not libslurmfull).
-- Allow cloud_reg_addrs to update dynamically registered node's addrs on
subsequent registrations.
-- switch/hpe_slingshot - Fix hetjob components being assigned different vnis.
-- Revert a change in 22.05.5 that prevented tasks from sharing a core if
--cpus-per-task > threads per core, but caused incorrect accounting and cpu
binding. Instead, --ntasks-per-core=1 may be requested to prevent tasks from
sharing a core.
-- Correctly send assoc_mgr lock to mcs plugin.
-- Fix regression in 23.02 leading to error() messages being sent at INFO
instead of ERR in syslog.
-- switch/hpe_slingshot - Fix bad instant-on data due to incorrect parsing of
data from jackaloped.
-- Fix TresUsageIn[Tot|Ave] calculation for gres/gpumem and gres/gpuutil.
-- Avoid unnecessary gres/gpumem and gres/gpuutil TRES position lookups.
-- Fix issue in the gpu plugins where gpu frequencies would only be set if both
gpu memory and gpu frequencies were set, while one or the other suffices.
-- Fix reservations group ACL's not working with the root group.
-- slurmctld - Fix backup slurmctld crash when it takes control multiple times.
-- Fix updating a job with a ReqNodeList greater than the job's node count.
-- Fix inadvertent permission denied error for --task-prolog and --task-epilog
with filesystems mounted with root_squash.
-- switch/hpe_slingshot - remove the unused vni_pids option.
-- Fix missing detailed cpu and gres information in json/yaml output from
scontrol, squeue and sinfo.
-- Fix regression in 23.02 that causes a failure to allocate job steps that
request --cpus-per-gpu and gpus with types.
-- sacct - when printing PLANNED time, use end time instead of start time for
jobs cancelled before they started.
-- Fix potentially waiting indefinitely for a defunct process to finish,
which affects various scripts including Prolog and Epilog. This could have
various symptoms, such as jobs getting stuck in a completing state.
-- Hold the job with "(Reservation ... invalid)" state reason if the
reservation is not usable by the job.
-- Fix losing list of reservations on job when updating job with list of
reservations and restarting the controller.
-- Fix nodes resuming after down and drain state update requests from
clients older than 23.02.
-- Fix advanced reservation creation/update when an association that should
have access to it is composed with partition(s).
-- auth/jwt - Fix memory leak.
-- sbatch - Added new --export=NIL option.
-- Fix job layout calculations with --ntasks-per-gpu, especially when --nodes
has not been explicitly provided.
-- Fix X11 forwarding for jobs submitted from the slurmctld host.
-- When a job requests --no-kill and one or more nodes fail during the job,
fix subsequent job steps unable to use some of the remaining resources
allocated to the job.
-- Fix shared gres allocation when using --tres-per-task with tasks that span
multiple sockets.
* Changes in Slurm 23.02.2
==========================
-- Fix regression introduced with the migration to interfaces which caused
sshare to core dump. Sshare now initialized the priority context correctly
when calculating with PriorityFlags=NO_FAIR_TREE.
-- Fix IPMI DCMI sensor initialization.
-- For the select/cons_tres plugin, improve the best effort GPU to core
binding, for requests with per job task count (-n) and GPU (--gpus)
specification.
-- scrontab - don't update the cron job tasks if the whole crontab file is
left untouched after opening it with "scrontab -e".
-- mpi/pmix - avoid crashing when running PMIx v5.0 branch with shmem support.
-- Fix building switch topology after a reconfig with the correct nodes.
-- Allow a dynamic node to register with a reason, using --conf, when the
state is DOWN or DRAIN.
-- Fix slurmd running tasks before RPC Prolog is run.
-- Fix slurmd deadlock iff the controller were to give a bad alias_list.
-- slurmrestd - correctly process job submission field "exclusive" with boolean
True or False.
-- slurmrestd - correctly process job submission field "exclusive" with strings
"true" or "false".
-- slurmctld/step_mgr - prevent non-allocatable steps from decrementing values
that weren't previously incremented when trying to allocate them.
-- auth/jwt - Fix memory leak in slurmctld with 'scontrol token'.
-- Fix shared gres (shard/mps) leak when using --tres-per-task
-- Fix sacctmgr segfault when listing accounts with coordinators.
-- slurmrestd - improve error logging when client connections experience
polling errors.
-- slurmrestd - improve handling of sockets in different states of shutdown to
avoid infinite poll() loop causing a thread to max CPU usage until process
is killed.
-- slurmrestd - avoid possible segfault caused by race condition of already
completed connections.
-- mpi/cray_shasta - Fix PMI shared secret for hetjobs.
-- gpu/oneapi - Fix CPU affinity handling.
-- Fix dynamic nodes powering up when already up after adding/deleting nodes
when using power_save logic.
-- slurmrestd - Add support for setting max connections.
-- data_parser/v0.0.39 - fix sacct --json matching associations from a
different cluster.
-- Fix segfault when clearing reqnodelist of a pending job.
-- Fix memory leak of argv when submitting jobs via slurmrestd or CLI commands.
-- slurmrestd - correct miscalculation of job argument count that could cause
memory leak when job submission fails.
-- slurmdbd - add warning on startup if max_allowed_packet is too small.
-- gpu/nvml - Remove E-cores from NVML's cpu affinity bitmap when
"allow_ecores" is not set in SlurmdParameters.
-- Fix regression from 23.02.0rc1 causing a FrontEnd slurmd to assert fail on
startup and don't be configured with the appropriate port.
-- Fix dynamic nodes not being sorted and not being included in topology,
which resulted in suboptimal dynamic node selection for jobs.
-- Fix slurmstepd crash due to potential division by zero (SIGFPE) in certain
edge-cases using the PMIx plugin.
-- Fix issue with PMIx HetJob requests where certain use-cases would end up
with communication errors due to incorrect PMIx hostname info setup.
-- openapi/v0.0.39 - revert regression in job update requests to accept job
description for changes instead of requiring job description in "job" field.
-- Fix regression in 23.02.0rc1 that caused a step to crash with a bad
--gpu-bind=single request.
-- job_container/tmpfs - skip more in-depth attempt to clean up the base path
when not required. This prevents unhelpful, and possibly misleading, debug2
messages when not using the new "shared" mode.
-- gpu/nvml - Fix gpu usage when graphics processes are running on the gpu.
-- slurmrestd - fix regression where "exclusive" field was removed from job
descriptions and submissions.
-- Fix issue where requeued jobs had bad gres allocations leading to gres not
being deallocated at the end of the job, preventing other jobs from using
those resources.
-- Fix regression in 23.02.0rc1 which caused incorrect values for
SLURM_TASKS_PER_NODE when the job requests --ntasks-per-node and --exclusive
or --ntasks-per-core=1 (or CR_ONE_TASK_PER_CORE) and without requesting
--ntasks. SLURM_TASKS_PER_NODE is used by mpirun, so this regression
caused mpirun to launch the wrong number of tasks and to sometimes fail to
launch tasks.
-- Prevent jobs running on shards from being canceled on slurmctld restart.
-- Fix SPANK prolog and epilog hooks that rely on slurm_init() for access to
internal Slurm API calls.
-- oci.conf - Populate %m pattern with ContainerPath or SlurmdSpoolDir if
ContainerPath is not configured.
-- Removed zero padding for numeric values in container spool directory names.
-- Avoid creating an unused task-4294967295 directory in container spooldir.
-- Cleanup container step directories at step completion.
-- sacctmgr - Fix segfault when printing empty tres.
-- srun - fix communication issue that prevented slurmctld from connecting to
an srun running outside of a compute node.
* Changes in Slurm 23.02.1
==========================
-- job_container/tmpfs - cleanup job container even if namespace mount is
already unmounted.
-- When cluster specific tables are be removed also remove the job_env_table
and job_script_table.
-- Fix the way bf_max_job_test is applied to job arrays in backfill.
-- data_parser/v0.0.39 - Avoid dumping -1 value or NULL when step's
consumed_energy is unset.
-- scontrol - Fix showing Array Job Steps.
-- scontrol - Fix showing Job HetStep.
-- openapi/dbv0.0.38 - Fix not displaying an error when updating QOS or
associations fails.
-- data_parser/v0.0.39 - Avoid crash while parsing composite structures.
-- sched/backfill - fix deleted planned node staying in planned node bitmap.
-- Fix nodes remaining as PLANNED after slurmctld save state recovery.
-- Fix parsing of cgroup.controllers file with a blank line at the end.
-- Add cgroup.conf EnableControllers option for cgroup/v2.
-- Get correct cgroup root to allow slurmd to run in containers like Docker.
-- Fix "(null)" cluster name in SLURM_WORKING_CLUSTER env.
-- slurmctld - add missing PrivateData=jobs check to step ContainerID lookup
requests originated from 'scontrol show step container-id=<id>' or certain
scrun operations when container state can't be directly queried.
-- Automatically sort the TaskPlugin list reverse-alphabetically. This
addresses an issue where cpu masks were reset if task/affinity was listed
before task/cgroup on cgroup/v2 systems with Linux kernel < 6.2.
-- Fix some failed terminate job requests from a 23.02 slurmctld to a 22.05 or
21.08 slurmd.
-- Fix compile issues on 32-bit systems.
-- Fix nodes un-draining after being drained due to unkillable step.