civ build system and pkg manager

Civstack is a full developer tech stack, and the bedrock of that is the build system -- which is this package (civ).

A "build system" is fundamentally a mechanism to convert a set of files and configuration (a package) into a runnable (and testable) software. A package manager is then a way to upload and depend on other packages.

A good build system has two other primary features:

• ergonomics: package specification is complete and easy to both read and write.
• hermetic: each built component can be isolated from the rest of the system and deterministically built.

PKG.luk

The PKG.luk file is is how you specify how to build a package and what is exported from that package. Below is an example one which we will walk though. Before we do, a few notes:

• This file is 100% pure lua except with different builtin globals, hence the different file extension (.luk)
• Noteablly very few of lua's builtin functions or modules are available. Instead there are several functions for importing non-PKG luk files.

Example PKG.luk:

civ output directory

By default, civ build puts files in .out/civ/. Wherever it puts it's output (whether during build or installation), the output structure under that directory is as follows:

• bin/: executable binaries/scripts segregated by pkg, i.e. bin/pkgname_foo. Also contains soft-links for binaries with unique names to conform with PATH, i.e. bin/foo.
• lua/: contains all mod.lua files, and the sub-module files under mod/sub.lua. This conforms with package.searchpath, which allows the normal lua loader to load these packages with LUA_PATH set to .../civ/lua/.
• data/: contains arbitrary data files defined by targets.
• lib/: all library files in a flat directory, all of which must start with lib i.e. libfoo.so. This is to conform with the envornment variables LUA_CPATH, LD_LIBRARY_PATH, etc as well as the cc -l flag.
• include/: header files, both .h and .iH (iA header) for linking dynamic libraries during compilation.
• log/: contains metadata about what each target installed, i.e. log/mypkg/foo.files used by the build system for incremental builds as well as by installation system to remove files during uninstall.

Mod civ

The civ build system command.
Types: Init Base Build Test Run Install
Functions

• fn build(cv, tgtnames)
• fn test(cv, tgtnames)
• fn main(args)

Record Init

civ init arguments. Fields:

• .out =".civconfig.lua"path to output config.lua
• .basebase config to copy from

Record Base

Fields:

• .config =".civconfig.lua"path to civ config.

Record Build

Usage: civ build hub:tgt#name Fields:

• .config =".civconfig.lua"path to civ config.

Record Test

Usage: civ test hub:tgt#name Fields:

• .config =".civconfig.lua"path to civ config.

Record Run

Usage: civ run hub:tgt#name -- ...args Build+run a single build target which has a single bin output. Fields:

• .config =".civconfig.lua"path to civ config.

Record Install

Usage: civ install hub:tgt#name Fields:

• .config =".civconfig.lua"path to civ config.
• .forcedo not confirm deletion of files.

Mod civ.core

Core civ (build system) types and functions.
Types: TargetName Target CfgBuilder Cfg Civ
Functions

• fn tgtnameSplit
• fn tgtname(tgtname)

Record TargetName

Represents a pkgname.target parsed from a string Fields:

• .pkgname
• .name

Methods

• fn parse(T, tgtname)

Record Target

A build target, the result of compiling a package. Fields:

• .pkgnamename of package target is in.
• .namethe name of the target.
• .kindthe kind of target: build, test, executable
• .dirdirectory of src files
• .srclist of input source files (strings).
• .deplist of input Target objects (dependencies).
• .extraarbitrary value, used by run command
• .apithe lang-specific exported import paths.
• .depIdslist of dependency target ids. Populated for Worker.
• .outPOD table output segregated by language.
  • t: PoD in a k/v table, can be used to configure downstream targets.
  • data: list of raw files.
  • include: header file paths for native code (C/iA).
  • lib: dynamic library files (i.e. libfoo.so)
  • bin: executable binaries
  • lua: lua files
• .linklink outputs from -> to
• .tagarbitrary attributes like test, testonly, etc.
• .ENVthe environment that scripts run in.
• .runexecutable script which performs the operation kind.

Methods

• fn:tgtname()

Record CfgBuilder

Fields:

• .directprefer building directly (running build scripts w/ dofile).

Record Cfg

The user configuration, typically at ./.civconfig.lua Fields:

• .paththe path to this config file.
• .host_osthe operating system of this computer.' Typically equal to civix.OS
• .hubstable of hubname -> /absolute/dir/path
• .buildDirdirectory to put build/test files.
• .installDirdirectory to install files to.
• .builderbuilder settings

Methods

• fn load(T, path)

Record Civ

Holds top-level data structures and algorithms for processing civ build graphs (pkgname graphs). Fields:

• .cfgthe user config
• .hubstable of hub -> dir (cfg.hubs)
• .pkgstable of pkgname -> pkg
• .luk
• .cycle
• .workerdirect worker
• .ENVenvironment for running workers

Methods

• fn:expand(pat) -> targets
  Expand a pattern to it's targets. This has the side effect of loading all related packages.
• fn:expandAll(tgtpats) -> tgtnames
  Given a list of hub:foo/.*#.* patterns, expand them.
• fn:getPkgname(dep) -> pkgname
• fn:abspath(pkgpath) -> abspath
  Given a pkg:path/to/file convert to an abspath (used for Luk).
• fn:pkgDir(pkgname) -> dir/
  Get pkgname's full directory.
• fn:tgtDir(tgt) -> dir/
• fn:target(tgt) -> Target?, errmsg
• fn:loadPkg(pkgname)
• fn:load(pkgnames)
  Load the pkgs and update self.pkgs with values.
• fn:targetDepMap(tgts, depMap)
  recursively find all deps
• fn:run(stage, tgtname, script, ids)
• fn:build(tgts) -> ordered, tgtsCache
  Build the target.
• fn:test(tgtnames, ordered, tgtsCache)
  Test the targets.

Record civ.Worker

Fields:

• .idsthe target ids to work on.
• .cfg
• .tgtsDbindexed line-file of targets by id
• .tgtsCachealready loaded targets

Methods

• fn get(T, args)
  Usage: worker = Worker:get()
• fn:target(id) -> Target
• fn:copyOut(tgt, outKey)
  Copy output files from tgt.out[outKey

. * fn:link(tgt) * fn:set()
Make self the singleton (future calls to Worker.get will return) * fn:close()
Remove this as singleton.