RFC - Compile in a box

[Nuc1eoN]

RFC - Compile in a box

Abstract

This RFC suggests new concept of "bulletproofing" the GoboLinux build system, while keeping it simple on the user.

In order to increase the reliability recipes, we need better reproducibility. Reproducible builds require a stable build environment. Read: strictly defined.

Diverging build environments appear to be the major reason, why compiling recipes succeeds for one user, while they fail for someone else.

Context

GoboLinux already has a mechanism to accomplish this: It is called Runner - a GoboLinux exclusive sandboxing tool. It can be accessed from within Compile by attaching the --pure argument.

--pure put the build into a sandbox and it exclusively includes programs, which are defined in Dependecies and BuildDependencies, including the runtime dependencies of Compile itself as well as inherited dependencies. It recursively scans all Dependency files.

The big drawback, and the reason why --pure is not used broadly in GoboLinux packaging today, is that is is extremely tedious in practice. It requires a lot of effort as well as going through cryptic error messages. Sometimes a --pure build will lack basic tools that are even required to start the build.

To counter that, over time multiple new dependencies have been added to the Compile runtime Dependencies file (eg #48, #49, #50). However this has been more of a kludge, in order to make --pure feasible and working at all for certain builds. Also it is cluttering up Compile's dependency tree.

Proposed solution:

In order to achieve (more) reproducible builds, we can re-use GoboLinux' existing mechanisms.

--pure has everything we need but it is tedious to use. Therefore we have to streamline the packaging experience.

In order to achieve this we need to have predefined sets of Dependencies, which can be passed over to Compile --pure. We could also call these sets of dependencies "build profiles".
These profiles will be standardized among GoboLinux users in order to achieve reproducibility.

A basic profile has to represent a GoboLinux "base" environment - a stripped down environment of a standard GoboLinux install, which everybody can be expected to have on his/her PC.

Other profiles could be created for more specific appliances, eg a build profiles based on a musl build environment etc... they could even depend on one another (see irc log below).

Implementation

I am suggesting the following implementation:

Build profiles could be stored under a new location in GoboLinux file system hierarchy:

/System/Settings/Compile/Profiles/base/017.01.conf

As above, a base build profile could be versioned after it's corresponding GoboLinux release.

The 017.01.conf file would be a simple list of dependencies, just as in a Dependencies file.

This profile could be passed to Compile explicitly like this (exact implementation is up for discussion):

# Compile bash --pure --profile=base/017.01

Since the 017.01 base build profile is supposed to be a default for building new packages with Compile, we could define a Current default profile, re-using the symlinking mechanism we are already using in /Programs/<program_name>/Current.

For instance:

# SymlinkProfile base 017.01
/S/S/Compile/Profiles/base/Current -> /S/S/Compile/Profiles/base/017.01.conf

Further considerations

After a successful build, the BuildInformation file could contain information about the Profile used during a build.

What has to be evaluated is, whether it makes sense to add eg a Profile= field to a Recipe file, meaning that this recipe has been tested or is meant to be built using a certain profile.

Sources & Inspiration

I did not come up with this alone, this idea is based on previous discussions. Credit goes to @ermo and @lucasvr 🙏. I have simply distilled this into an RFC.

Relevant discussion:

[...]
<--- 2021-02-08 --->
<ermo> But I'd do a BaseSystem per version of Gobo
<ermo> that versions the Bash, the Glibc, the GCC etc. that each release uses
<ermo> and then set that for Compile for each release
<ermo> i.e. put that Meta package into the Compile Dependencies
<ermo> so that Compile 017.1 depends on DevBase 017.1
<ermo> and then DevBase 017.1 Depends on set of '=' versions of Python, Perl, Lua, Glibc, GCC, etc. etc.
       etc.
<ermo> so you can have a range of DevBase Recipes for each release
<ermo> so you can develop a new Release image via a new DevBase Recipe and cook a chroot or a VM for
       testing
<ermo> without wrecking your existing install
<ermo> over time, you could add other *Base Recipes
<ermo> so you can have runner build a complete, minimal LAMP (or similar) stack and run it
<ermo> The gobo base system is super flexible and with Runner, it suddenly has a killer app
[...]
<--- 2021-02-13 --->
<ermo> but, principally, this (for me) is about Glibc (and its ancillaries), GCC and the Kernel and
       building an idea of a "Development Gobo Baselayout"
<ermo> where you have the GNU toolchain deps all in once place
[...]
<--- 2021-03-16 --->
<ermo> paranoidd: so, what I need from you and hisham is some idea of how you view my suggestion for
       having a "base system set"
<ermo> on many distributions, there's a 'system.base' and a 'system.devel' set (or something equivalent)
<ermo> because, frankly, having to specify the same set of components over and over again for compile
       --pure is just a shit packaging experience.
<ermo> so I'd aim for a base GNU devel set to be specified and either implied or specified via a single
       dep
<ermo> paranoidd: how does that thought sit with you?
<--- 2021-03-17 --->
<paranoidd> ermo: how about having a bunch of flat files under, say, /S/Settings/Compile/Profiles/, with
            a default profile file set under /S/S/Compile/Compile.conf?
<paranoidd> each of these flat files would be a  collection of dependency files that would be
            automatically picked by Compile
<ermo> "automatically" how?
<paranoidd> actually, we could also use a 'Current' symlink to determine the default one, too (so we
            don't have to mess with parsing of Compile.conf)
<ermo> as in "every single .conf file in /S/S/Compile/Profiles/"?
<ermo> I'm trying to understand the use case
<ermo> Having profiles like that is not a bad idea
<paranoidd> "automatically" in the sense that it's just a matter of providing that file to Runner (Runner
            -d /path/to/profile.conf)
<ermo> so a "Profile" is a set?
<paranoidd> Runner -d /path/to/profile.conf -d /Data/Compile/Recipes/.../Dependencies.conf  Compile ...
<paranoidd> yes, a profile is a set
<ermo> can a profile contain other profiles?
<ermo> i.e. a set of sets?
<ermo> and if so, how do you break recursive cycles?
<paranoidd> if we were to keep it simple, then it would have the same syntax as a Dependencies file
<ermo> oh, I *definitely* agree
<ermo> my point being that you could have a 0.17.0 and a 0.17.1 file that specifies the same dep *names*
       but different *versions* for instance.
<ermo> and you could also have a completely different profile for musl purposes or llvm purposes if you
       wanted to
<paranoidd> so we could have something like Profiles/017.conf (which would have the standard "base system
            set"), Profiles/017-Musl.conf, etc
<ermo> Yep
<paranoidd> yes, I think  we're on the same page
<paranoidd> that's doable, for sure
<paranoidd> and it's likely a 1-line change on bin/Compile
<ermo> the question is if you want a 'Current' symlink or if you want it in a config file
<ermo> i.e. Compile.conf, you could specify the default set per release
<ermo> *in Compile.conf
<ermo> I can see advantages to both
<paranoidd> we just need to have a way to specify a default profile 
<ermo> Yes, but say you want a non-standard addition to the default profile
<paranoidd> we'll actually need a way to let the user override the default one on the command-line too,
            when invoking Compile
<ermo> or you want to inherit it
<ermo> yes, we do
<ermo> It needs some more thought
<ermo> because there's a difference between build sets and build profiles imo
<ermo> (profiles could contain additional info and/or flags)
<ermo> perhaps the musl profile needs different C/CXX/LDFLAGS for instance
<paranoidd> another way to see this is to have a dummy package with entries such as
            /Programs/CompileProfiles/{017,017-Musl,...}/Resources/Dependencies
<paranoidd> we could have different profiles depending on one another to achieve what you suggest
<ermo> so there needs to be both a way to have build deps and run deps as well as default settings
<ermo> both gentoo and exherbo have extensive experience with profiles
<paranoidd> in the case of having CFLAGS/etc we could just honor Resources/Environment
<ermo> but I feel like the existing approach that you outline might be the most useful
<ermo> because then it suddenly becomes trivial to create new profiles (you just copy the current one and
       make the changes you need to make and off you go)
<ermo> I think it's more of a question to make it intuitive and to reuse the existing tooling
<ermo> and then work through the caveats
<ermo> (easier said than done because it's time consuming)
<paranoidd> exactly.. both /S/Settings/Compile/Profiles and /Programs/CompileProfiles are valid
            approaches
<ermo> what does hisham say?
<paranoidd> no idea, I haven't asked him yet
<ermo> or, rather, we probably ought to do this in a forum thread
<ermo> to get input and preserve the discussion
<paranoidd> yes, sounds like a better place to discuss this (so others can chime in too)
<paranoidd> let me know if you'd like to create this post 
<paranoidd> otherwise I can look into posting it later tonight
<ermo> the points that need to be considered is a) default profile (when you don't specify an explicit
       profile) and b) custom profile (I'd suggest only being able to specify one for simplicity)
<ermo> Then the one you specify can be carefully tailored.
<paranoidd> agreed
[...]
<ermo> I feel like upstream profiles shouldn't be changed
<ermo> if you want to change something, you copy the profile you want to change and then set it as the
       default
<paranoidd> yes, I think that's the way to go
<ermo> but there also needs to be a way to update upstream profiles seamlessly
<ermo> i.e. to sync them
<ermo> such that users doing nothing will benefit from them
<ermo> i.e. if you add a change, you want users to be able to benefit from it as soon as they get it
<paranoidd> then, Compile & Scripts should have a way to be seamlessly updated as well
<ermo> yeah
<paranoidd> Compile already attempts to update /D/C/Recipes with a call to 'git pull'
<paranoidd> it could do something similar to keep the profiles up to date
<ermo> yep
<ermo> one of the things I have a slight issue with is editing /D/C/Recipes directly. It's not entirely
       to profiles
<ermo> I've noticed that if I edit a recipe and contribute it, that recipe disappears from my local
       visible tree
<ermo> (possibly because it's moved to a branch)
<paranoidd> as a regular user, yes, that's because of the branching thing
<paranoidd> with 'ContributeRecipe -g' that not an issue
<paranoidd> that's*
<ermo> I think I would prefer that contributors are told to maintain their updated recipes in a
       well-define directory structure under ${HOME}
<ermo> sure, but to users contributing recipes, having them "disappear" is disconcerting?
<ermo> as in, it's not a nice user experince
<ermo> *experience
<paranoidd> I think that's fixable
<paranoidd> it's more a matter of running a git merge
<ermo> yes, but you can't do that as long as it's not accepted upstream, can you?
<ermo> i.e. you'll have diverging histories
<paranoidd> (and then to watch for conflicts when it comes the time to git pull from upstream, yes)
<ermo> on the main branch
<ermo> anyway, I haven't looked into local repositories yet, so I have some studying to do on that front
<ermo> but for now, I need to head to bed.
<paranoidd> all right, take care!
<paranoidd> have a good  one
<ermo> if you get a brigt idea in terms of how to handle profiles, do ping me
[...]

Source (archived irc log)

Other related discussion: gobolinux/Freshen#5 (comment)

Please share and comment your thoughts.