pod: plain-old-data library and serialization

A Lua library for specifying and converting types to/from "plain old data" and methods to serialize/deserialize those types to/from bytes. In Lua, the 5 supported types considered "plain old data" are: nil, boolean, integer, string and any table of only these types.

local M = mod'mymod'

local mty = require'metaty'

local pod = require'pod'

-- enums are already plain-old-data.

M.Job = mty.enum'Job' { BOSS = 1, PEON = 2 }

--- records need to have pod() called on them.

--- This implements __toPod and __fromPod for the metaty record.

M.Worker = pod(mty'Worker' {

  'name [string] #1',    -- must specify type and #id

  'job  [mymod.Job] #2', -- can lookup any type in PKG_LOOKUP

  'salary [int] #3',

  'schedule {int: mymod.Schedule} #4', -- map of weekday[1,7] -> scheduled time

})

M.Schedule = pod(mty'Schedule' {

  'start [int]: start time in seconds since midnight',

  'stop  [int]: stop time in seconds since midnight',

})

local serialized = pod.ser(M.Worker{...})   -- convert to string

local worker = pod.de(serialized, M.Worker) -- convert from string

Explanation of above:

metaty.enum are already plain-old-data (no need to call pod() on them).
metaty record types can be made plain old data by calling the pod() module on them.
This simply implements the __toPod and __fromPod methods on them -- you can do this yourself if you prefer!
pod parses the metaty field type stanza as [singleType], {listType} or {map: type} repsecitively. The collection types are converted to pod.List and pod.Map respectively.

Custom Podder

For making your own podder impl, the signatures are

function MyT.__toPod(T, pod, value) ... end --> p

function MyT.__fromPod(T, pod, p) ... end   --> value

Where:

value is the value (instance of T).
p is a builtin plain-old-data (concrete)
T will equal MyT

Serialization Best Practices

Only make a type pod if it is actually plain-old-data -- the type must be always convert to/from Lua concrete types even if called on a different machine or on a future date. For instance don't implement pod on a type with a file inside it (even if you're using a path to reload it) since another machine might not have that specific path!
If a type may be serialized anywhere (even by an older version of the software), do not change or reuse the #id of a field (changing the name is okay if you can fix your dependencies).
If you need to delete the field then add it to __fieldIds={...} to ensure future you doesn't use the field id.

Usecases

Usecases of pod are:

database schema: The primary intended usecase of pod is to be used in civdb -- each "row" is simply a pod record.
This is likely the only usecase civstack will use
general data storage: use fn load and fn dump to de/serialize values to/from a file.
cross-language communication, see Cross Language Tooling.
RPC framework: pod could be used in a similar capacity as JSON RPC to communicate with other services (i.e. frontend <-> backend).

library support

In addition to providing methods to de/serialize data to a compact binary form, pod exports the toPod() and fromPod() functions to help other libraries (i.e. lson) de/serialize arbitrary lua types.

Cross Language Tooling

pod is designed to (eventually) serve the same need as protobuf: it can and will generate code for other languages to read/write pod's binary serialization format.

At this time, support for other languages has not started -- but the design of pod is meant to mimick protobuf as much as possible so that such a goal can be met in the future. Work on supporting multiple languages will likely not be part of the civstack project but the civstack project will support such work.

Mod pod

pod: plain old data

Types: Pod Podder key builtin List Map

Functions

fn isConcrete(v)
Return whether v is a nil, boolean, number or string.
fn isBuiltin(obj)
Return whether v is concrete or a table with no metatable.
fn serRaw
fn deserRaw
fn isPod(v, isPodFn)
return true if the value is "plain old data".
Plain old data is defined as any native type or a table with no metatable and who's pairs() are only POD.
The isPodFn fn takes v and should return true if it is pod.
fn isPodder(P) -> isPodder, whyNot?
fn tableToPod(T, pod, t)
fn toPod(v, podder, pod)
fn fromPod(v, poder, pod)
fn mty_toPod(T, pod, t)
Default __toPod for metatype records
fn mty_fromPod(T, pod, p)
Default __fromPod for metatype records
fn implPod(T, tys)
Make metaty type convertable to/from plain-old-data
Typically this is called by calling the module itself, i.e. pod(mty'myType'{'field [int]#1'})
fn ser(value) -> string
Serialize value, converting it to a compact string. Note: this function first calls toPod on the value.
fn deser(str, P, index) -> value, lenUsed
Deserialize value from a compact string (and call fromPod on it) index (default=1) is where to start in str
fn dump(f, ...)
dump ser(...) to f, which can be a path or file.
fn load(f, ...)
load deser(f:read'a', ...), f can be a path or file.