Cleanup and update documentation

Signed-off-by: David Oberhollenzer <david.oberhollenzer@tele2.at>

7 files changed, 510 insertions, 132 deletions

diff --git a/docs/bootup.md b/docs/bootup.md
deleted file mode 100644
index bd2b80e..0000000
--- a/docs/bootup.md
+++ /dev/null
@@ -1,74 +0,0 @@
-# System Bootup Process
-
-## Initial Ram Disk to Rootfs transition
-
-After mounting the root filesystem, either the kernel or the initial ram disk
-startup process is expected to exec the init program from the root filesystem.
-
-At the current time, there is no support for re-scanning the service files
-*yet*, so when init is started, the final configuration in `/etc/init.d` has to
-be present. As a result, we currently cannot perform mounting of `/etc/` or
-packing init into the initial ram disk and doing the rootfs transition.
-
-Also, as a result of this, changing the service configuration requires a system
-reboot to be effective.
-
-This _will_ change in the future.
-
-
-## Processing Service Descriptions
-
-The init process reads service description files from `/etc/init.d` which are
-usually symlinks to actual files in `/usr/share/init`.
-
-The exact locations may be changed through configure flags when compiling init.
-
-Service files specify a *target* which is basically like a SystemV runlevel and
-can be one of the following:
-
-* boot
-* reboot
-* shutdown
-* ctrlaltdel
-
-After parsing the configuration files, the init process starts running the
-services for the `boot` target in a topological order as determined by their
-*before* and *after* dependencies.
-
-Services can be of one of the following *types*:
-
-* wait
-* once
-* respawn
-
-Services of type `wait` are started exactly once and the init process waits
-until they terminate before continuing with other services.
-
-The type `once` also only runs services once, but immediately continues
-starting other services in the mean time without waiting.
-
-Services of type `respawn` also don't stall the init process and are re-started
-whenever they terminate.
-
-## Service Process Setup
-
-If a service description contains only a single `exec` line, the init process
-forks and then execs the command directly in the child process.
-
-If the service description contains a `tty` field, the specified device file
-is opened in the child process and standard I/O is redirected to it before
-calling exec. Also, a new session is created.
-
-
-If a service description contains multiple `exec` lines, the init process forks
-off to a single child process that does the same setup as above, and then runs
-the command lines sequentially by forking a second time for each one, followed
-by an exec in the grand child and a wait in the original child.
-
-If a single command line returns something other than `EXIT_SUCCESS`,
-processing of multiple command lines is immediately stopped and the offending
-exit status is returned to init.
-
-
-The init process reads environment variables from `/etc/initd.env`.
-
diff --git a/docs/cmdline.md b/docs/cmdline.md
new file mode 100644
index 0000000..c3945ec
--- /dev/null
+++ b/docs/cmdline.md
@@ -0,0 +1,43 @@
+# Available Command Line Tools
+
+## The service command
+
+The `service` utility can be used for control and administration of services.
+
+It is composed of several sub commands run by issuing
+`service <command> [arguments..]`.
+
+Currently available service commands are:
+
+ * enable - enable a service with the given list of arguments.
+ * disable - disable a service. If the service is parameterized, requires the
+   same arguments used for enabling, to disable the specific instance of the
+   service.
+ * dumpscript - generate an equivalent shell script from the `exec` lines of
+   a service after applying all parameter substitutions.
+ * list - list all enabled service. A target can be specified to only list
+   services for the specified target.
+ * help - display a short help text and a list of available commands.
+
+
+## shutdown and reboot
+
+The programs `shudown` and `reboot` can be used to signal to the `init` program
+that it should transition to the `shutdown` or `reboot` target respectively.
+
+The option `-f` or `--force` can be used to by pass the init system entirely
+and force a hard reset or power off by directly signalling the kernel.
+
+Running any one of those programs requires superuser privileges.
+
+
+## syslog
+
+If the `usyslogd` service is built as part of this package, a program called
+`syslog` is built that can be used from the command line to send syslog
+messages.
+
+This can for instance be used to produce log messages from shell scripts.
+
+The log level, facility and identity string can be specified.
+See `syslog --help` for more information.
diff --git a/docs/defconfig.md b/docs/defconfig.md
new file mode 100644
index 0000000..af01375
--- /dev/null
+++ b/docs/defconfig.md
@@ -0,0 +1,106 @@
+# Default Service Configuration
+
+## Pseudo Services
+
+The default configuration contains a number of "pseudo services" in the boot
+target that don't actually do anything but are merely used as anchors in
+service dependencies, i.e. they indicate that some sort of milestone in the
+boot sequence has been reached. Everything that is part of that milestone
+specifies that it should be run *before* that pseudo service and everything
+that requires that this milestone has been reached, specifies that it wants
+to run afterwards.
+
+The pseudo targets are (in the order that they are executed):
+
+ * vfs
+
+   All services that do mount point setup go before this, all service that
+   depend on the fully mounted rootfs go after this.
+
+ * sysinit
+
+   The system has reached a sane state, i.e. the hostname is set, the system
+   clock has a sane value, modules and kernel parameters are loaded, some
+   very basic, fundamental services are running (e.g. syslog).
+   Everything that is part of that setup process goes between `vfs` and
+   `sysinit`, everything that requires a sane setup goes *after* `sysinit`.
+
+ * network
+
+   Network configuration is done. All services that do network configuration
+   should position themselves between `sysinit` and `network`. Everything that
+   requires a fully configured networking setup should go *after* `network`.
+
+## Default Bootup Services
+
+This section outlines the services for the boot target that are enabled by
+default.
+
+
+The following services are enabled by default and run *before* the `vfs` target
+for filesystem setup:
+
+ * procfs - mount `procfs` to `/proc` and try to mount additional pseudo
+   filesystems in `/proc` such as `binfmt_misc`
+ * tmpfs - mount a `tmpfs` to `/tmp`
+ * sysfs - mount `sysfs` to `/sys` and try to mount additional pseudo
+   filesystems in `/sys` (e.g. `securityfs`, `configfs`, ...)
+ * devfs - mount `devtmpfs` to `/dev`, try to mount additional pseudo
+   filesystems in `/dev` (e.g. `devpts`, `mqueue`, ...) and try to create
+   some additional device nodes and symlinks.
+
+
+The following services are enabled by default and configured to run *after*
+the `vfs` target and *before* the `sysinit` target:
+
+ * hostname - reload hostname `/etc/hostname`
+ * loopback - bring the loopback device up
+ * usyslogd - if the `usyslogd` services is compiled with this package, this
+   service is enabled by default and starts `usyslogd`.
+ * modules - iterate over the file `/etc/modules` and try to load each module
+   using modprobe.
+ * sysctl - restore kernel parameters using `sysctl --system`. See `sysctl(8)`
+   for a list of possible locations that the parameters are read from.
+
+
+The following services are enabled by default and configured to run *after*
+the `sysinit` target and *before* the `network` target:
+
+ * ifcfg - static network configuration
+   Does the static network configuration outlined in [network.md](network.md)
+
+
+## Default Shutdown and Reboot Services
+
+For the shutdown and reboot targets, the following services are executed:
+
+ * sigterm - send the SIGTERM signal to all processes and wait for 5 seconds
+ * sigkill - send the SIGKILL signal to all remaining processes
+ * ifdown - bring all network interfaces down
+ * sync - run the sync command
+
+
+## Additional Services not Enabled by Default
+
+ * agetty - A parameterizeable, respawn type `agetty` service. The first
+   parameter is the terminal device that the getty should run on.
+ * dhcpcdmaster - If one or more network interfaces should be configured using
+   dhcpcd, this service starts a central `dhcpcd` master instance.
+ * dhcpcd - A parameterizeable single shot service that signals the `dhcpcd`
+   master that it should configure a specific interface. The first parameter
+   is the interface that should be configured by `dhcpcd`.
+ * dnsmasq - A respawn type service for the `dnsmasq` DNS and DHCP server.
+ * hostapd - If the system should operate a WIFI access point, this respawn
+   type service can be enabled to manage an instace of the `hostapd` program.
+ * unbound - A respawn type service that manages an instance of the `unbound`
+   name resolver.
+ * usyslogd - A respawn type service that manages an instance of the `usyslogd`
+   syslogd implementation that is part of this package.
+ * hwclock - If the system has a hardware clock, this service can restore the
+   kernels clock from the hardware at bootup, between the `vfs` and `sysinit`
+   targets.
+ * nft - If enabled, restores net filter table rules during boot.
+ * swclock - For systems that don't have a hardware clock, this service
+   restores a somewhat usable time from a file during boot.
+ * swclocksave - For systems that don't have a hardware clock, this service
+   saves the current time to a file during shutdown or reboot.
diff --git a/docs/init.md b/docs/init.md
new file mode 100644
index 0000000..e76a4b4
--- /dev/null
+++ b/docs/init.md
@@ -0,0 +1,71 @@
+# The Init Daemon
+
+## Initial Ram Disk to Rootfs transition
+
+After mounting the root filesystem, either the kernel or the initial ram disk
+startup process is expected to exec the init program from the root filesystem.
+
+At the current time, there is no support for re-scanning the service files
+*yet*, so when init is started, the final configuration in `/etc/init.d` has to
+be present. As a result, we currently cannot perform mounting of `/etc/` or
+packing init into the initial ram disk and doing the rootfs transition.
+
+Also, as a result of this, changing the service configuration requires a system
+reboot to be effective.
+
+This _will_ change in the future.
+
+
+## Service Types and Targets
+
+A service file has a *type*, specifying whether the service should be run once
+or restarted when it terminates and a *target*, specifying when the service
+should be run.
+
+The *target* is similar to a runlevel in System V init. The init daemon
+currently knows about the following targets:
+
+* boot
+* reboot
+* shutdown
+
+When `init` is run, it starts all the services for the `boot` target. From the
+`boot` target it can transition to any other target and execute the services
+for the specified target.
+
+The `reboot` and `shutdown` targets cannot transition to any other target and
+when invoked, cause initd to drop everything else it intended to do.
+
+For the `reboot` and `shutdown` targets, respawn type processes are no longer
+restarted when they terminate and once all services have been executed, the
+`init` program performs a hard system reboot or power off.
+
+The init program tries to capture the `CTRL+ALT+DELETE` key sequence (or its
+local equivalent) and transitions to the reboot target if pressed.
+
+
+## Service Configuration Rescan
+
+TBD
+
+
+## Control Socket and Signals
+
+The `init` program catches the following signals:
+
+* `SIGCHLD`
+* `SIGINT`
+* `SIGTERM`
+
+The `SIGCHLD` handler implements standard process reaping. If a terminated
+process belongs to one of the supervised services, the configured action is
+taken (e.g. restarting it).
+
+When `SIGINT` is caugth, `init` transitions to the `reboot` target. Similarly,
+`SIGTERM` causes `init` to transition to the `shutdown` target.
+
+
+For more complex tasks, `init` creates a control socket that the command line
+tools included in this package can use. For the time being, the control socket
+can only tell the init daemon to transition to the `reboot` or `shutdown`
+target.
diff --git a/docs/network.md b/docs/network.md
new file mode 100644
index 0000000..f111a8f
--- /dev/null
+++ b/docs/network.md
@@ -0,0 +1,83 @@
+# Static Network Configuration
+
+The default configuration provides multiple services that perform network
+initialization and static configuration using helper scripts that require
+programs from the `iproute2` package.
+
+Configuration files are typically stored in `/etc/netcfg/` (depending on
+configure options).
+
+Please note that the loopback device is treated specially and not included in
+any of the network configuration outlined below. The loopback device is brought
+up and configured by a dedicated service long before the network configuration
+is done.
+
+
+## Interface Renaming
+
+If the `ifrename` service is enabled (it is disabled by default), network
+interfaces are renamed based on a rule set stored in the file `ifrename`.
+The file contains comma separated shell globing patterns for the current
+interface name, MAC address and a prefix for the new interface name.
+
+For each network interface, rules are processed top to bottom. If the first two
+globing patterns apply, the interface is renamed. Interfaces with the same
+prefix are sorted by mac address and a running index is appended to the prefix.
+
+If none of the rules apply, the interface name is left unchanged.
+
+
+The intent is, to provide a way to configure persistent, deterministic names for
+at least all network interfaces that are permanently installed on a board.
+
+Extension cards or external network adapters should be given a different prefix
+to avoid changes in the order as they come and go.
+
+
+## Interface Configuration
+
+After interface renaming, for each network interface, the configuration path is
+scanned for files with the same name as the interface.
+
+Each successfully found configuration file is processed line by line, top to
+bottom. Each line may contain a keyword, followed by multiple arguments.
+
+The following keywords can be used to add IPv4 or IPv6 network addresses to
+an interface:
+
+ * address
+ * addr
+ * ip
+ * ip6
+ * ipv6
+
+Those commands are expected to be followed by an IPv4 or IPv6 address and
+network mask.
+
+
+Furthermore, the following commands can be used for configuring interface
+parameters:
+
+ * `arp {on|off}`
+ * `multicast {on|off}`
+ * `mtu <value>`
+ * `offload [rx {on|off}] [tx {on|off}] [sg {on|off}] [tso {on|off}]`
+ * `offload [gso {on|off}] [gro {on|off}] [lro {on|off}] [rxvlan {on|off}]`
+ * `offload [txvlan {on|off}] [ntuple {on|off}] [rxhash {on|off}]`
+ * `offload [ufo {on|off}]`
+
+
+## Route Configuration
+
+After interface configuration is done, routes and rules are restored from a
+file named `routes` in the same configuration path.
+
+The file may contain lines starting with `route` or `rule`. Everything that
+follows is passed on to `ip route add` or `ip rule add` respectively.
+
+
+## Net Filter Tables
+
+
+An additional service is provided that restores the nft rule set from
+`/etc/nftables.rules`.
diff --git a/docs/services.md b/docs/services.md
index 223cfb5..de0bd21 100644
--- a/docs/services.md
+++ b/docs/services.md
@@ -1,22 +1,121 @@
 # Service Files
 
-Services that can be started and managed by init are described by service
-description files stored in `/usr/share/init`.
+The init process reads service descriptions from `/etc/init.d` which usually
+contains symlinks to the actual service files which can be conveniently removed
+or added to disable or enable services.
 
-The init process actually reads from `/etc/init.d` which contains symlinks to
-the actual service files.
+Default services provided by this package are installed in `/usr/share/init`,
+i.e. this is where the symlinks point to.
+
+
+Note that the actual locations may be different, depending on the configure
+flags used.
 
-Enabling a service means adding a symlink, disabling means removing a symlink.
 
 Service descriptions can be parameterized. The arguments are extracted from the
 name of the symlink. Currently only 1 parameter is supported. The argument
 value is separated from the service name by an '@' character in the symlink
 name.
 
+The file name of the sysmlink, excluding any parameters, is accepted as the
+canonical name of a service and used when referring to it via command line
+utilities or when injecting dependencies from a service file.
+
+
+## Syntax
+
+Each line in a service description is made up of a keyword, followed by one
+or more arguments and terminated by a line break.
+
+Blank lines are ignored and shell-style comments can be used.
+
+Arguments are separated by space. Quotation marks can be used to treat
+something containing spaces or comment character as a single argument.
+
+In between quotation marks, C-style escape sequences can be used.
+
+Argument substitution (arguments derived from the symlink name) can be
+done using a '%' sign, followed by the argument index. A '%' sign can be
+escaped by writing '%%'.
+
+
+## Targets and Types
+
+Service files specify a *target* which is basically like a SystemV runlevel
+and can be one of the following:
+
+* boot
+* reboot
+* shutdown
+
+After parsing the configuration files, the init process starts running the
+services for the `boot` target in a topological order as determined by their
+dependencies.
+
+Services can be of one of the following *types*:
+
+* wait
+* once
+* respawn
+
+Services of type `wait` are started exactly once and the init process waits
+until they terminate before continuing with other services.
+
+The type `once` also only runs services once, but immediately continues
+starting other services in the mean time without waiting. The init process
+only waits for `once` types when transitioning to another target.
+
+Services of type `respawn` also don't stall the init process and are re-started
+whenever they terminate.
+
+The keyword `limit` can be used a after `respawn` to specify how often a service
+may be restarted before giving up.
+
+
+## Dependencies
+
+A service description file can specify dependencies using the keywords `after`
+and `before`, each followed by a space separated list of service names.
+
+The init program executes a service after all the services specified by the
+`after` keyword have been started (type once or respawn) or have been completed
+(type wait).
+
+The `before` keyword injects dependencies in reverse, i.e. all services
+specified after the `before` keyword are only executed after the service in
+question has been started.
+
+If a service specified by `after` or `before` does not exist, it is simply
+ignored. This can occur for instance if the specified service is not enabled
+at all in the current configuration.
+
 
+## Running Services
+
+If a service contains an `exec` line, the init process attempts to run it
+using the `runsvc` helper program that sets up the environment, attempts to
+execute the specified command line and passes the exit status back to the init
+process.
+
+If multiple exec lines are specified, `runsvc` executes them sequentially and
+stops if any one returns a non-zero exit status.
+
+The environment variables visible to the service processes are read
+from `/etc/initd.env`.
+
+If the service description contains a `tty` field, the specified device file
+is opened by runsvc and standard I/O is redirected to it and a new session
+is created. The keyword `truncate` can be used to make `runsvc` truncate the
+file to zero size first.
+
+For convenience, multiple exec lines can be wrapped into braces, as can be
+seen in one of the examples below.
+
+
+## Example
 
 Below is an annotated example for a simple, service description for a
-generic, parameterized agetty service:
+generic, parameterized getty service:
 
     #
     # The text that init should print out when the status of the
@@ -27,70 +126,27 @@ generic, parameterized agetty service:
     #
     description "agetty on %0"
 
-    #
-    # How to run the service. 'respawn' means restart the service when it
-    # terminates, 'once' means run it only once and continue with other
-    # services in the mean while, 'wait' means run it once, but block until
-    # it exits.
-    #
+    # Restart the getty when it terminates.
     type respawn
 
-    #
-    # When to start the service. 'boot' means when booting the system. Other
-    # options are 'reboot', 'shutdown' and 'ctrlaltdel'. The system always
-    # starts into the 'boot' target and then later transitions to one of the
-    # others.
-    #
+    # We want to spawn gettys when booting the system
     target boot
 
-    #
-    # A list of service names that must be started before this service can
-    # be run, i.e. this services needs to be started after those.
-    #
-    # This can only refer to generic names, not specific instances. For
-    # instance, you can say "after getty" to make sure a service comes up after
-    # all gettys are started, but you cannot specify "after agetty@tty1".
-    #
-    # Similar to 'after', there is also a 'before' keyword for specifying
-    # dependencies.
-    #
+    # Only run this service after the 'sysinit' service is done
     after sysinit
 
     #
-    # The 'tty' directive specifies a file to which all I/O of the process is
-    # redirected. The specified device file is used as a controlling tty for
-    # the process and a new session is created with the service process as
-    # session leader.
+    # Redirect all I/O of the process to this file. The specified device file
+	# is used as a controlling tty for the process and a new session is created
+	# with the service process as session leader.
     #
     # In this example, we derive the controlling tty from the service
     # description argument.
     #
     tty "/dev/%0"
-    
-    #
-    # The 'exec' directive specifies the command to execute in order to start
-    # the service. See in the example below on how to run multiple commands.
-    #
-    # Again we use the argument to specify what terminal our getty
-    # should run on.
-    #
-    exec agetty %0 linux
-
-As can be seen in this simple example, each line in a service description is
-made up of a keyword, followed by one or more arguments and terminated by a
-line break.
-
-Blank lines are ignored and shell-style comments can be used.
-
-Arguments are separated by space. Quotation marks can be used to treat
-something containing spaces or comment character as a single argument.
-
-In between quotation marks, C-style escape sequences can be used.
-
-Argument substitution (arguments derived from the symlink name) can be
-done using a '%' sign, followed by the argument index. A '%' sign can be
-escaped by writing '%%'.
 
+    # In order to run the service, simply fire up the agetty program
+    exec agetty %0 linux
 
 If a service should sequentially run multiple commands, they can be grouped
 inside braces as can be seen in the following, abbreviated example:
@@ -107,4 +163,3 @@ inside braces as can be seen in the following, abbreviated example:
         mkdir /var/tmp -m 0755
         mount --bind /cfg/preserve/var_lib /var/lib
     }
-
diff --git a/docs/usyslogd.md b/docs/usyslogd.md
new file mode 100644
index 0000000..96f5755
--- /dev/null
+++ b/docs/usyslogd.md
@@ -0,0 +1,94 @@
+# Syslogd Implementation
+
+A tiny syslogd implementation `usyslogd` is provided as part of this package.
+
+It opens a socket in `/dev/log`, processes syslog messages and forwards the
+parsed message to a modular backend interface.
+
+Currently, there is only one implementation of the backend interface that dumps
+the log messages into files in the processes working directory (by default
+`/var/log`).
+
+A simple log rotation scheme has been implemented.
+
+
+## Security Considerations
+
+By default, the daemon switches its working directory to `/var/log`. The
+directory is created if it doesn't exist and the daemon always tries to
+change its mode to one that doesn't allow other users (except group members)
+to access the directory.
+
+If told to so on the command line, the daemon chroots to the log directory.
+
+By default, the daemon then tries to drop privileges by switching to user and
+group named `syslogd` if they exist (any other user or group can be specified
+on the command line; doing so causes syslogd to fail if they don't exist).
+
+
+On a system that hosts accounts for multiple users that may be more or less
+trusted, one may consider only giving system services access to the syslog
+socket and not allowing regular users. Otherwise, a user may flood the syslog
+daemon with messages, possibly leading to resource starvation, or (in the case
+of size limited log rotation outlined below) to the loss of otherwise critical
+log messages. Since this is not the primary target of the Pygos system, such
+a mechanism is not yet implemented.
+
+In case of a system where only daemons are running, the above mentioned
+security measure is useless. If a remote attacker manages to get regular user
+privileges, you already have a different, much greater problem. Also, a remote
+attacker would have to compromise a local daemon that already has special
+access to the syslog socket, which is again your least concern in this
+scenario.
+
+
+## Logrotation
+
+The backend can be configured to do log rotation in a continuous fashion (i.e.
+in a way that log messages aren't lost), or in a way where it drops old
+messages. Furthermore, the backend can be configured to automatically do a log
+rotation if a certain size threshold is hit.
+
+If the `usyslogd` receives a `SIGHUP`, it tells the backend to do log rotation.
+
+In the case of the size threshold, the backend is expected to do the rotation
+on its own if the predetermined limit is hit.
+
+
+## File Based Backend
+
+The file based backend writes log messages to files in the current working
+directory (by default `/var/log`), named either after the ident string (if
+specified) or the facility name.
+
+Log messages are prefixed with an ISO 8601 time stamp, optionally the facility
+name (unless part of the file name), the log level and the senders PID. Each
+of those fields is enclosed in brackets.
+
+Log rotation in a continuous fashion means renaming the existing log file to
+one suffixed with the current time stamp. Overwriting old messages renaming
+the log file by appending a constant `.1` suffix.
+
+
+## Default Configuration
+
+The default service configuration limits the log file size to 8 KiB and
+configures the daemon to overwrite old messages when rotating log files,
+effectively limiting the amount of log data to 16 KiB per source or facility.
+
+The intended use case in the Pygos system is logging to a ramdisk without
+exhausting available memory.
+
+
+## Possible Future Directions
+
+In the near term future, the daemon probably requires more fine grained control
+over logging such as setting a minimum log level or a way to configure limits
+per facility or service.
+
+In the medium term future, extended resource control using c-groups might be
+a possibility.
+
+Future directions may include adding other backends, such as forwarding the
+log messages to a central server, for instance using syslog over UDP/TCP or
+using the front end of some time series database.