Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEVPROD-9923: remove Evergreen client on MacOS task hosts #8263

Merged

Conversation

Kimchelly
Copy link
Contributor

@Kimchelly Kimchelly commented Sep 3, 2024

DEVPROD-9923

Description

The MacOS hosts have recently had trouble deploying new agent binaries. The suspected issue is that some of the MacOS hosts have SIP enabled, and those hosts are SIGKILL'ing the binary if it has some issue (such as running with CGO enabled or running an Evergreen binary that hasn't been signed by Apple). This problem persists even if a new version is downloaded that fixes the original issue - the binary still gets SIGKILL'd after redownloading a valid Evergreen agent to the file path. The workaround we found (that was found long ago during an identical outage) was to rm the binary, which seemed to fix the issue.

  • Remove the Evergreen binary when downloading the agent monitor onto MacOS hosts.
  • Remove the Evergreen binary when downloading the agent onto MacOS hosts.

Testing

  • Added unit tests.
  • I can't test this in staging because there are no longer any MacOS static hosts available in staging (there used to be, but they don't work anymore). Instead, I'll be rolling this out incrementally to the MacOS hosts and manually checking that they're still able to run tasks.

Documentation

N/A

@Kimchelly Kimchelly requested a review from a team September 3, 2024 19:02
func (d *Distro) IsWindows() bool {
// XXX: if this is-windows check is updated, make sure to also update
// public/static/js/spawned_hosts.js as well
return strings.Contains(d.Arch, "windows")
}

// IsWindows returns whether or not the distro's hosts run on Linux.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: IsLinux

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch, fixed.


require.NoError(t, m.removeMacOSClient())
_, err = os.Stat(m.clientPath)
if runtime.GOOS == "darwin" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slightly tangential but our MacOS variant is inactive. Should we enable it for the sake of affirming these sorts of tests?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, I activated it.

@Kimchelly Kimchelly merged commit 4b32fac into evergreen-ci:main Sep 5, 2024
11 checks passed
@Kimchelly Kimchelly deleted the DEVPROD-9923_remove_macos_binary branch September 5, 2024 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants