civ build system and pkg manager

Civstack is a full developer tech stack, and the bedrock of that is the build system -- which is this command (civ).

A "build system" is fundamentally a mechanism to convert a set of files and configuration (a package) into runnable (and testable) software. A package manager is then a way to depend on other packages.

A good build system has two other primary features:

ergonomics: package specification is complete and easy to both read and write.
hermetic: each built component can be isolated from the rest of the system and deterministically built.

PKG.luk

The PKG.luk file is is how you specify how to build a package and what is exported from that package. Below is an example one which we will walk though. Before we do, a few notes:

This file is 100% pure lua except with different builtin globals, hence the different file extension (.luk)
Noteablly very few of lua's builtin functions or modules are available. Instead there are several functions for importing non-PKG luk files.

Example PKG.luk:

local P = { summary = "Does foo and some bar" }

--- Imports a function to help us create Target

local lua = import'sys:lua.luk' -- Lua build targets

-- Set up metadata for this PKG.luk file, this must be

-- returned at the end with the targets and PoD to provide

-- clients.

--

-- This also creates the global `P` value, which is the data

-- (build information) exported by this file.

P.foo = lua {

  mod = 'foo',

  src = { 'foo.lua' },

  dep = {

    'civ:lib',           -- depend on entire civ lib

    'myhub:foo#libfoo',  -- depend on next target

  },

}

-- test rule

P.test = lua.test {

  src = 'test.lua',

  dep = { 'myhub:foo' },

}

-- C build target with lua linked.

P.libfoo = lua.cc {

  hdr = { 'foo.h' },  -- C headers

  src = { 'foo.c' },  -- C source files

  dep = {

    'myhub:bar#libbar', -- depend on another target

  },

}

return P

Every value assigned to P must be either a concrete type or a #civ.core.Target.

civ output directory

By default, civ build puts files in .civ/. Wherever it puts it's output (whether during build or installation), the output structure under that directory is as follows:

bin/: executables (and links to them) go here and must not conflict.
lua/: contains all mod.lua files, and the sub-module files under mod/sub.lua. This conforms with package.searchpath, which allows the normal lua loader to load these packages with LUA_PATH set to .../civ/lua/?.lua.
data/: contains arbitrary data files defined by targets.
lib/: all library files in a flat directory, all of which must start with lib i.e. libfoo.so. This is to conform with the envornment variables LUA_CPATH, LD_LIBRARY_PATH, etc as well as the cc -l flag.
include/: header files, both .h and .iH (iA header) for linking dynamic libraries during compilation.
log/: contains metadata about what each target installed, i.e. log/mypkg/foo.files used by the build system for incremental builds as well as by installation system to remove files during uninstall.

Command civ

civ command

Subcmd init

Usage: civ init
Initialize the repository. This should be run when starting a new repo. Arguments:

config =".civconfig.luk" path to config.lua to output
llua ="lua" the lua library name override for pkg-config, i.e. lua5.3

Subcmd build

Usage: civ build hub:tgt#name
Build targets which match the pattern. Arguments:

config path to civ config.
clean

Subcmd test

Usage: civ test hub:tgt#name
Test targets which match the pattern. Arguments:

config path to civ config.
clean

Subcmd run

Usage: civ run hub:tgt#name -- ...args
Build+run a single build target which has a single bin or link output. Arguments:

config path to civ config.
clean

Subcmd install

Usage: civ install hub:tgt#name
Install targets which match the pattern. Arguments:

config path to civ config.
clean
force do not confirm deletion of files.

Mod civ.core

civ.core contains types used by the civ build system.

If you are just a user of civ this is likely not useful. This library is

useful primarily for those who want to extend civ and/or write their own

build/test macros.

Types: TargetName Target Hub Cfg BuilderCfg Civ

Functions

fn tgtnameSplit
fn tgtname(tgtname)
fn loadPrevTargets(jsonPath)

Record TargetName

Represents a pkgname.target parsed from a string

Fields:

pkgname
name

Methods

fn parse(T, tgtname)

Record Target

A build target, the result of compiling a package.

Fields:

pkgname name of package target is in.
name the name of the target.
kind the kind of target: build, test, executable
dir directory of src files
src list of input source files (strings).
extra arbitrary value, used by run command
dep list of input Target objects (dependencies).
api the lang-specific exported import paths.
out POD table output segregated by language.
- t: PoD in a k/v table, can be used to configure downstream targets.
- data: list of raw files.
- include: header file paths for native code (C/iA).
- lib: dynamic library files (i.e. libfoo.so)
- bin: executable binaries
- lua: lua files
link link outputs from -> to
tag arbitrary attributes like test, testonly, etc.
run executable script which performs the operation kind.
mtime modified time of target and deps
id id of this target. Populated for Worker.
depIds list of dependency target ids. Populated for Worker.

Methods

fn:tgtname()
fn:outPaths(dir)
Fill to with output path strings.

Record Hub

A hub configuration

Fields:

name ="this" the name of the hub

Record Cfg

The user configuration, typically at ./.civconfig.lua

Fields:

path the path to this config file.
basePath the base path of this config file.
host_os the operating system of this computer.' Typically equal to civix.OS
hubs table of hubname -> /absolute/dir/path
buildDir directory to put build/test files.
installDir directory to install files to.
builder builder settings

Methods

fn load(T, path)

Record BuilderCfg

Cfg.builder settings

Fields:

direct prefer building directly (running build scripts w/ dofile).

Record Civ

Holds top-level data structures and algorithms for processing civ build graphs (pkgname graphs).

Fields:

cfg the user config
thisHub
thisPkg
hubs table of hub -> dir (cfg.hubs)
pkgs table of pkgname -> pkg
luk
cycle
worker direct worker
lastTargets the last targets
ENV environment for running workers

Methods

fn load(T, cfgPath)
fn:expand(pat) -> targets
Expand a pattern to it's targets. This has the side effect of loading all related packages.
fn:expandAll(tgtpats) -> tgtnames
Given a list of hub:foo/.*#.* patterns, expand them.
fn:getPkgname(dep) -> pkgname
fn:abspath(pkgpath) -> abspath
Given a pkg:path/to/file convert to an abspath (used for Luk).
fn:pkgDir(pkgname) -> dir/
Get pkgname's full directory.
fn:tgtDir(tgt) -> dir/
fn:target(tgt) -> Target?, errmsg
fn:tgtMod(tgt) -> ds.time.Epoch
Get the "modtime" of the whole target.
fn:loadPkg(pkgname)
fn:loadPkgs(pkgnames)
Load the pkgs and update self.pkgs with values.
fn:targetDepMap(tgts, depMap)
recursively find all deps
fn:run(stage, tgtname, script, ids)
fn:prebuild(prevTgts, tgts) -> toBuild, ordered
fn:build(tgts) -> ordered, tgtsCache
Build the target.
fn:test(tgtnames, ordered, tgtsCache)
Test the targets.

Record civ.Worker

civ.Worker encapsulates a single civ worker, i.e. the process which actually performs build/test actions.

If you are just a user of civ this is likely not useful. This library is useful primarily for those who want to extend civ and/or write their own build/test macros.

Fields:

ids the target ids to work on.
cfg
tgtsDb indexed line-file of targets by id
tgtsCache already loaded targets

Methods

fn get(T, args)
Usage: worker = Worker:get()
fn:target(id) -> Target
fn:copyOut(tgt)
Copy output files from tgt.out[outKey].
fn:link(tgt)
fn:set()
Make self the singleton (future calls to Worker.get will return)
fn:close()
Remove this as singleton.