Skip to content

Commit

Permalink
[SPARK-32476][CORE] ResourceAllocator.availableAddrs should be determ…
Browse files Browse the repository at this point in the history
…inistic

### What changes were proposed in this pull request?

This PR aims to make `ResourceAllocator.availableAddrs` deterministic.

### Why are the changes needed?

Currently, this function returns indeterministically due to the underlying `HashMap`. So, the test case itself is creating a list `[0, 1, 2]` initially, but ends up with comparing `[2, 1, 0]`.

Not only this happens in the 3.0.0, but also this causes UT failures on Scala 2.13 environment.

### Does this PR introduce _any_ user-facing change?

Yes, but this fixes the in-deterministic behavior.

### How was this patch tested?

- Scala 2.12: This should pass the UT with the modified test case.
- Scala 2.13: This can be tested like the following (at least `JsonProtocolSuite`)

```
$ dev/change-scala-version.sh 2.13
$ build/mvn test -pl core --am -Pscala-2.13 -Dtest=none -DwildcardSuites=org.apache.spark.deploy.JsonProtocolSuite
```

**BEFORE**
```
*** 2 TESTS FAILED ***
```

**AFTER**
```
All tests passed.
```

Closes apache#29281 from dongjoon-hyun/SPARK-32476.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
dongjoon-hyun committed Jul 29, 2020
1 parent d897825 commit 9dc0237
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ trait ResourceAllocator {
def availableAddrs: Seq[String] = addressAvailabilityMap
.flatMap { case (addr, available) =>
(0 until available).map(_ => addr)
}.toSeq
}.toSeq.sorted

/**
* Sequence of currently assigned resource addresses.
Expand All @@ -68,7 +68,7 @@ trait ResourceAllocator {
private[spark] def assignedAddrs: Seq[String] = addressAvailabilityMap
.flatMap { case (addr, available) =>
(0 until slotsPerAddress - available).map(_ => addr)
}.toSeq
}.toSeq.sorted

/**
* Acquire a sequence of resource addresses (to a launched task), these addresses must be
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,8 @@ object JsonConstants {
|:["3","4","5"]}},"resourcesused":{"gpu":
|{"name":"gpu","addresses":[]},"fpga":
|{"name":"fpga","addresses":[]}},"resourcesfree":
|{"gpu":{"name":"gpu","addresses":["2","1","0"]},
|"fpga":{"name":"fpga","addresses":["5","4","3"]}},
|{"gpu":{"name":"gpu","addresses":["0","1","2"]},
|"fpga":{"name":"fpga","addresses":["3","4","5"]}},
|"state":"ALIVE","lastheartbeat":%d}
""".format(currTimeInMillis).stripMargin

Expand Down

0 comments on commit 9dc0237

Please sign in to comment.