Skip to content

Imagine build system concept in preparation for choosing one #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Imagine build system concept in preparation for choosing one #13

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions doc/concept/build_system.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
_Note: This document is designed to render nicely with GitHub-flavored Markdown. My apologies in advance to everyone else._

Build System Concepts
=====================

Goals of this document
----------------------

Be deleberate about choosing a build system; explain criteria, options, choices, and decisions.

Strategy to choose a build system is:
1. Enumerate good experiences to retain
2. Enumerate bad experiences to avoid
3. Be judiciously terse; the longer this is, the fewer people will read it
4. Enumerate & experiment with options
5. Make decision


Definitions
-----------

**Contributor:** Who is a contributor? Is there some base-level knowledge we can expect?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contributor is someone who is sending a pull request.

Maintainer is a set of people that have had a track record to contribution, has volunteered to help with the project maintenance (tagging releases, updating the website, managing infrastructure, etc.).

Code Owner is a person who has responsibility for a sub-part of the overall project, and has approval rights to merge changes from non-code-owner contributors.



Requirements
------------

* Minimal dependencies as practical, to facilitate easy building by others
* Cross-platform
* Any external dependencies must be freely installable (i.e., no registration, no license codes, no money; see also `CONTRIBUTING.md`)
* Any internally satisfied dependencies must be freely runable (i.e., no license codes; see also `CONTRIBUTING.md`)
* Is maintainable by contributors given a reasonable learning curve


Good experiences to retain
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think one experience we might want to support is meeting the users where they are. If users are already using a particular package manager, it would be good for us to support them there.

In particular, if CMake is the overwhelming majority, we should at least support CMake. If Bazel is next in the list, we should have some support for it too. Whether we test and/or keep both in-sync is another question, but I think we can support more than one if we're clear about what the priorities are and who will be owning which ones we're supporting.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like you think a survey is a good idea. I'll put one together.

--------------------------

_Disclaimer: A good or bad experience in any particular build system is not an implicit vote for or against that build system; these days, how you use build systems is just as important as which one you choose._

**Single build system for everything:** (ABK) Used Waf to create a single build system for the entire project; fetching dependencies, configuring, building, testing, packaging, all from just `waf configure --platform=<...> && waf`.

**Fully internal dependencies:** (ABK) Built an ad-hoc package manager (thin wrapper on top of `git clone`) into the project's configure stage so that the project can compile on any computer for any platform (within reason). This includes required 3rd party libraries, required 3rd party tools, you name it. Setting up a developer machine no longer requires any special work past SSH keys, a working compiler and such.


Bad experiences to avoid
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this section -- but I think there's a few more I could add. At a high level:

  • Installing to a system location is hard.
  • Embedding project in another project is also hard. Given for example a user would like to add the library to an existing project, doing this currently is super-hard.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What characteristics do you see as being conducive to embedding inside another project?

In my organization, we have cpp-netlib pre-compiled by hand, and then committed in our package manager. The downside is that when we upgrade boost, we also need to recompile cpp-netlib. We don't upgrade boost often, so it's not a big deal.

What sort of interface do you imagine between cpp-netlib and other projects? Compile first and interface the compiled results with a dependent project, or interface cpp-netlib's source with a dependent project, and both are built at the same time?

------------------------

**Boost version difference broke build:** (ABK) While onboarding a new developer, a difference in a Boost version caused an obscure build failure. Takeaway: when practical, always freeze dependencies using some sort of package manager. (Is this takeaway actually practical in an open-source project where the dependencies are almost by definition non-frozen?)


Available build systems
-----------------------

| Name | Namesake | Overview | Notable Advantages | Notable Disadvantages |
| ---- | -------- | -------- | ------------------ | --------------------- |
| [A-A-P](https://en.wikipedia.org/wiki/A-A-P) | | Python-based package manager written by the author of vim. | Augmentable via scripting (python), supports uploading and downloading files | Not designed for building software
| [Ant](https://en.wikipedia.org/wiki/Apache_Ant) | "Another Neat Tool" | Java+XML-based build system | Widely used, straight-forward, relatively easy to read if you don't know Ant. | Build specification file is not as powerful as other options here; extensions required if you want to improve things.<br />Non-Java languages feel second-class<br />Ant extensions are developed using a different SDK than the main build specification file.<br />
| [Autoconf](https://en.wikipedia.org/wiki/Autoconf) & [Automake](https://en.wikipedia.org/wiki/Automake) | auto wrappers on top of make | Automates the creation of Makefiles for C-based projects | Widely used | Complexity, difficult to debug
| [Bazel](https://en.wikipedia.org/wiki/Bazel_(software)) | Anagram of Blaze, Google's internal build system | Java-based Build system using Skylark (flavor of Python)-based configuration files for describing the build | Parallel by default, supported by Google, (computer) language-agnostic. Repeatable, scalable builds<br />Xcode support: bypasses Xcode to an extent; builds various Apple components manually
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preference is this would be a secondary system to support.

| [Buck](https://en.wikipedia.org/wiki/Buck_(software)) | Facebook's internal build system | | Native integration with Xcode and IntelliJ | Native integration in only Xcode and IntelliJ
| [BuildAMation](https://en.wikipedia.org/wiki/BuildAMation) | | A modular build system using C#-like syntax
| [Buildr](https://en.wikipedia.org/wiki/Apache_Buildr) | | A Ruby-based competitor to Ant. | | | Intended primarily for Java projects
| [CMake](https://en.wikipedia.org/wiki/CMake) | | A cross-platform, modular wrapper around Make
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preference is this (CMake) would be the primary supported build system.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I agree. I don't have a preference of a secondary build system.

| [Gradle](https://en.wikipedia.org/wiki/Gradle) | | | | Intended primarily for Java, Groovy, and Scala
| [GYP](https://en.wikipedia.org/wiki/GYP_(software)) | Written by Google; used by Chrome and NodeJS | Native IDE project file generator. Used by the V8 Javascript engine, Google's web browser (Chrome), Dart (programming language), Node.js, WebRTC Telegram, and Electron.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GYP is now deprecated (I think) in favour of GN -- (link) which personally is tied with Bazel support as ranked in my head.

| [make](https://en.wikipedia.org/wiki/Make_(software)) |
| [Meson](https://en.wikipedia.org/wiki/Meson_build_system) | | | Good support of clerical tasks (Xcode integration, VS integration, Apple installer packages, etc) | Cross-platform insulation layer for compiler parameters seems to be a bit naive at first glance
| [NixOS](https://github.com/NixOS/nixpkgs) | | | Idempotent package management and build system with atomic releases/rollbacks and human readable configs (with integrated bash scripting) that are easily source controlled. Comes with integrated release distribution, and can even be used as our server OS, which would give us the same benefits with user management and system state (including replacing cron). | Linux only
| [Premake]( https://en.wikipedia.org/wiki/Premake) | | An Xcode/VS/etc project file generator | | Only a project file generator
| [Rake](https://en.wikipedia.org/wiki/Rake_(software)) | | Ruby-based competitor of SCons | | Lacking strong IDE integration
| [SCons](https://en.wikipedia.org/wiki/SCons) | | Predecessor of Waf | | KDE used SCons for a while but decided to switch to CMake for scalability reasons.
| [Tweaker](https://en.wikipedia.org/wiki/Tweaker_(build_tool)) | | Competitor of SCons
| [Waf](https://en.wikipedia.org/wiki/Waf) | | Successor of SCons (written by same author); Waf is a python-based general-purpose parallel task execution engine; includes "tools" for common tasks, such as C/C++ compiling. Relatively easy to write generic tasks. | Highly modularized; easily extensible | Highly modularized; easily extensible (bad for complexity/maintenance; easy to shoot yourself in the foot)