Deft - The Dylan CLI#

The deft command-line tool is intended to make Dylan development easier by taking care of some of the drudgery for you, including:

managing Dylan workspaces and package dependencies
creating boilerplate for new projects
downloading and installing dependencies (no need for git submodules)
generating “registry” files
publishing packages to the Dylan package catalog

Terminology#

package

A blob of files that can be unpacked into a directory and which has a dylan-package.json file in the top-level directory. A package currently corresponds to a single Git repository. A package has a set of versioned releases. A package may contain zero or more Dylan libraries.

workspace

The directory in which deft operates. Effectively this means a workspace is where the _build and registry directories are generated. In most cases, a workspace is the directory containing the dylan-package.json file, but the ultimate arbiter is the workspace.json file, if it exists. See workspaces for details.

active package

A package checked out into the top-level of a workspace. In most cases a workspace is the same as a package directory so there is only one active package. See Workspaces for discussion of multi-package workspaces.

deft update scans active packages when creating the registry.

release

A specific version of a package. A release has a Semantic Version associated with it, such as 0.5.0.

Requirements#

Make sure git is on your PATH so it can be found by the package manager, which currently exec’s git clone to install packages. (This dependency will be removed in a future release.)

Where are Packages Installed?#

deft installs package dependencies in the _packages directory at the root of your workspace by default. However, you may choose to install them globally with deft update --global.

The package manager always caches its catalog in the global location (per user).

The global _packages directory location is chosen based on environment variables, in this order:

${DYLAN}/_packages if DYLAN is set.
${XDG_STATE_HOME}/dylan/_packages (Unix) if XDG_STATE_HOME is set, or ${CSIDL_LOCAL_APPDATA}/dylan/_packages (Windows) if CSIDL_LOCAL_APPDATA is set.
${HOME}/.local/state/dylan/_packages otherwise.

See also: XDG spec

Building From Source#

If you have Open Dylan 2024.2 or later, deft is already installed as part of that release. (In all 2022 and 2023 releases and in 2024.1 the tool was named dylan but has now been renamed to deft.) But deft is still under active development so you may want to build the latest version. Here’s how….

Read the Requirements section, above.

Clone and build the “deft” project:

$ git clone --recursive https://github.com/dylan-lang/deft
$ cd deft
$ make
$ make test      # optional
$ make install

Make sure that $DYLAN/bin is on your $PATH. If you prefer not to set $DYLAN, make sure that $HOME/dylan/bin is on your $PATH, as that is where the Makefile installs the executable.

You should now be able to run deft-app help and go through the Hello World example below.

Quick Start#

This section shows how to

create a hello-world application and its test suite,
generate a registry for the compiler to locate libraries,
build hello-world and its test suite, and
add a new dependency to your package file.

First, create a place to put all your Dylan workspaces (usually one per project), and change to that directory:

$ mkdir -p ${HOME}/dylan/workspaces
$ cd ${HOME}/dylan/workspaces

Note

The above is a typical setup, but you can put your workspaces anywhere, and they don’t need to be together in a “workspaces” directory.

Now generate a new application library called “hello-world”:

$ deft new application hello-world
Created library hello-world.
Created library hello-world-test-suite.
Created library hello-world-app.
Downloaded strings@1.1.0 to /home/you/dylan/workspaces/hello-world/_packages/strings/1.1.0/src/
Downloaded command-line-parser@3.1.1 to /home/you/dylan/workspaces/hello-world/_packages/command-line-parser/3.1.1/src/
Downloaded json@1.0.0 to /home/you/dylan/workspaces/hello-world/_packages/json/1.0.0/src/
Downloaded testworks@2.3.1 to /home/you/dylan/workspaces/hello-world/_packages/testworks/2.3.1/src/
Workspace directory is /home/you/dylan/workspaces/hello-world/.
Updated 18 files in /home/you/dylan/workspaces/hello-world/registry/.

What did this do?

It created files with some initial code for your application, hello-world.
It created a test suite.
It ran deft update, which downloaded all the packages your application depends on.
It created a “registry” directory, which dylan-compiler will use to locate each used library.

Take a look at the generated files in the “hello-world” subdirectory. In particular, hello-world/dylan-package.json describes a Dylan package, which you could eventually publish for others to use.

Also look at one or two registry files and you’ll see that they simply contain a pointer to the build file (a “.lid” file) for a library.

Now let’s build!

$ cd hello-world
$ deft build --all
...compiler output...

$ _build/bin/hello-world
Hello world!

Note

On the initial build there are compiler warnings for the “dylan” library. These are due to a known (harmless) bug and can be ignored. Subsequent builds will not show them, and will go much faster since they’ll use cached build products.

Since we used the --all flag above, hello-world, hello-world-app, and hello-world-test-suite were built. Run the test suite:

$ _build/bin/hello-world-test-suite
Running suite hello-world-test-suite:
Running test test-greeting: PASSED in 0.000065s and 7KiB
Completed suite hello-world-test-suite: PASSED in 0.000065s

Ran 1 check: PASSED
Ran 1 test: PASSED
PASSED in 0.000065 seconds

Now let’s add a new dependency to our library. Let’s say we want to use base64 in our library.dylan file. The compiler finds libraries via the registry, but there is no “base64” registry file so the compiler won’t find it. To fix this, edit hello-world/dylan-package.json to add the dependency. Change this:

"dependencies": [  ],

to this:

"dependencies": [ "base64" ],

and then run deft update again:

$ deft update
Workspace directory is /home/you/dylan/workspaces/hello-world/.
Updated 1 file in /home/you/dylan/workspaces/hello-world/registry/.

Note that we didn’t specify a version for “base64”, so the latest version is downloaded. Usually it’s a good idea to specify a particular version, like “base64@0.1”. Take a look at “registry/<your-platform>/base64” to see where it was installed.

We also haven’t actually changed the hello-world code to use base64. That is left as an exercise. (Modify library.dylan and run deft build -a again.)

Now that you’ve got a working project, try some other deft subcommands, the most useful ones are:

deft status tells you the status of the active packages. It will find the hello-world package but will complain that it’s not a Git repository. Run git init in it if you like.
deft list with --all lists all the packages in the catalog. (Note that many libraries are still included with Open Dylan. They’ll be moved to packages eventually.)

Workspaces#

A workspace is a directory in which you work on a Dylan package, or multiple interrelated packages. deft often needs to find the root of the workspace, for example to decide where to write the “registry” directory or to invoke dylan-compiler. It does this by looking for one of the following files, in the order shown, and by using the directory containing the file:

workspace.json – A place to put workspace configuration settings. If this file exists, it takes precedence over the following two options in determining the workspace root.
dylan-package.json – The package definition file, required for projects that will be published to the package catalog.
The current working directory is used if neither of the above are found.

Usually, the workspace root is just the package directory (i.e., the directory containing dylan-package.json), because most of the time you will be working on one package at a time. In this case there is no need for a workspace.json file unless you need to provide workspace settings not contained in the package file.

In the less common case of working on multiple, interrelated Dylan packages at the same time, the workspace.json file is necessary in order to put the workspace root above the level of the package directories. For example, your multi-package workspace might look like this:

my-workspace/_build               // created by dylan-compiler
my-workspace/package-1/*.dylan
my-workspace/package-1/*.lid
my-workspace/package-1/dylan-package.json
my-workspace/package-2/*.dylan
my-workspace/package-2/*.lid
my-workspace/package-2/dylan-package.json
my-workspace/registry             // created by deft
my-workspace/workspace.json       // created by you

Most deft subcommands need to be run inside a workspace so that they can

find or create the “registry” directory,
invoke dylan-compiler in the workspace root directory, so that compiler always uses the same _build subdirectory,
find the “active packages” in the workspace, and
find settings in the workspace.json file.

If you create a workspace.json file it must contain at least an empty JSON dictionary, {}.

{
    "default-library": "cool-app-test-suite"
}

The "default-library" attribute is currently the only valid attribute and is used by the deft build command to decide which library to build when no other library is specified. A good choice would be your main test suite library. It may also be left unspecified.

The Registry#

Open Dylan uses “registries” to locate used libraries. Setting up a development workspace historically involved a lot of manual Git cloning, creating registry files for each used library, and adding Git submodules.

deft update takes care of that for you. It scans each active package and its dependencies for “.lid” files and writes a registry file for each one (but see below for platform-specific libraries), and it downloads and installs package dependencies for you.

Note

If you use the same workspace directory on multiple platforms (e.g., a network mounted directory or shared by a virtual machine) you will need to run deft update on each platform so that the correct platform-specific registry entries are created. deft makes no attempt to figure out which packages are “generic” and which are platform-specific, so it always writes registry files specifically for the current platform, e.g., x86_64-linux.

Platform-specific Libraries#

Note

If you’re new to Dylan you may want to skip this section as it’s likely you won’t need to worry about it yet.

Open Dylan supports multi-platform libraries via the registry and per-platform LID files. Among other things, LID files tell the compiler which files to compile, and in which order. To write platform-specific code, put it in a separate Dylan source file and only include it in that platform’s LID file.

To complicate matters, one LID file may include another LID file via the LID header.

In order for deft update to generate the registry it must figure out which LID files match the current platform. For example, when on Linux it shouldn’t generate a registry file for a Windows-only library.

To accomplish this the Platforms LID header was introduced. In your LID file you may specify the platforms on which the library runs:

Platforms: x86_64-linux
           riscv64-linux

If the current platform matches one of the platforms listed in the LID file, a registry file is generated for the library. (If there is no Platforms header, the library is assumed to run on all platforms.)

If a LID is included in another LID file and does not explicitly match the current platform via the Platforms keyword, then no registry entry is written for that LID file. The assumption being that the included LID file only contains shared data and isn’t a complete LID file on its own.

This effectively means that if you include a LID file in one platform-specific LID file then you must either create one LID file per platform for that library, or you must use the Platforms header in the included LID file to specify all platforms that don’t have a platform-specific LID file.

For example, the base “dylan” library itself has a dylan-win32.lid file so that it can specify some Windows resource files. “dylan-win32.lid” includes “dylan.lid” and has Platforms: x86-win32. Since there’s nothing platform-specific for any other platform, creating 8 other platform-specific LID files would be cumbersome. Instead, “dylan.lid” just needs to say which platforms it explicitly applies to by adding this:

Platforms: aarch-64-linux
           arm-linux
           x86_64-freebsd
           ...etc, but not x86-win32...

Package Manager#

deft relies on pacman, the Dylan package manager (unrelated to the Arch Linux tool by the same name), to install dependencies. See the pacman documentation for information on how to define a package, version syntax, and how dependency resolution works.

Global `deft` Options#

Note that global command line options must be specified between “deft” and the first subcommand name. Example: deft --debug build --all

--debug

Disables error handling so that when an error occurs the debugger will be entered, or if not running under a debugger a stack trace will be printed. When used with the --verbose flag this also enables tracing of dependency resolution, which can be fun! :-).

--verbose

Enables more verbose output, such as displaying which packages are downloaded, which registry files are written, etc.

When used with the --debug flag this also enables tracing of dependency resolution.

Subcommands#

deft help#

Displays overall help or help for a specific subcommand.

Synopsis:

deft help

deft help <subcommand> [<sub-subcommand> ...]

deft <subcommand> [<sub-subcommand> ...] --help

deft build#

Build the configured default library or the specified libraries.

Synopsis:: deft build [options] [--all | lib1 lib2 ...]

deft build is essentially a wrapper around dylan-compiler that has a few advantages:

Invoke it from any directory inside your workspace and it will run the build in the top-level workspace directory so that the _build and registry directories are used.
Configure a set of libraries to build by default, in dylan-package.json.
Use the --all flag to build all libraries in the workspace. For example, normally this builds both the main library and the test suite.
Specify multiple libraries on one command line, unlike with dylan-compiler.

deft build exits after the first library that generates serious compiler warnings, i.e., if dylan-compiler exits with an error status. (Requires an Open Dylan release later than 2020.1.)

Note

This subcommand is purely a convenience; it is perfectly valid to run dylan-compiler directly instead, after changing to the workspace top-level directory.

Options:

--all: Build all libraries found in the active packages of the current workspace. This option is ignored if specific libraries are requested on the command line also.
--clean: Do not use cached build products; rebuild from scratch.
--link: Link the executable or shared library. Defaults to true. Use --no-link for faster builds when iterating through compiler warnings.
--unify: Combine all used libraries into a single executable. Note that dylan-compiler puts the generated executable in _build/sbin instead of _build/bin when this flag is used. (Requires Open Dylan 2022.1 or later.)

deft install#

Install packages.

Synopsis: deft install <package> [<package> ...]

This command is primarily useful if you want to browse the source code in a package locally without having to worry about where to clone it from. If you are in a workspace directory the packages are installed in the workspace’s “_packages” subdirectory. Otherwise, see Where are Packages Installed?.

deft list#

Display a list of installed packages along with the installed version number and the latest version available in the catalog, plus a short description. With the --all option, list all packages in the catalog whether installed or not.

An exclamation point is displayed next to packages for which the latest installed version is lower than the latest published version.

Example:

$ deft list
     Inst.   Latest  Package               Description
     0.1.0    0.1.0  base64                Base64 encoding
   ! 3.1.0    3.2.0  command-line-parser   Parse command line flags and subcommands
     0.1.0    0.1.0  concurrency           Concurrency utilities
     0.6.0    0.6.0  deft                  Manage Dylan workspaces, packages, and registries
     ...

deft new application#

Generate the boilerplate for a new executable application.

Synopsis: deft new application [options] <name> [<dependency> ...]

This command is the same as deft new library except that in addition to the <name> library it also generates a <name>-app executable library with a main function.

Here’s an example of creating an executable named “killer-app” which depends on http version 1.0 and the latest version of logging.

$ deft new application killer http@1.0 logging
$ deft build --all
$ _build/bin/killer-test-suite
$ _build/bin/killer-app

You must run deft update whenever dependencies are changed, to install the new dependencies and update the registry files.

See also: deft new library

deft new library#

Generate code for a new shared library.

Synopsis: deft new library [options] <name> [<dependency> ...]

This command is the same as deft new application except that it doesn’t generate the corresponding <name>-app executable library.

Specifying dependencies is optional. They should be in the same form as specified in the dylan-package.json file. For example, “strings@1.0”.

This command generates the following code:

A main library and module definition and initial source files.
A corresponding test suite library and initial source files.
A dylan-package.json file (unless this new library is being added to an existing package).

You must run deft update whenever dependencies are changed, to install the new dependencies and update the registry files.

See also: deft new application

Options:

--force-package, -p: Create dylan-package.json even if already inside a package. This is intended for testing and continuous integration use.

Here’s an example of creating a library named “http” which depends on “strings” version 1.0 and the latest version of “logging”.

$ deft new library http strings@1.0 logging
$ deft build --all
$ _build/bin/killer-app-test-suite

Edit the generated dylan-package.json file to set the repository URL, description, and other attributes for your package.

deft new workspace#

Create a new workspace.

Synopsis: deft new workspace [options] <name>

Note

In most cases there is no need to explicitly create a workspace since the package directory (the directory containing dylan-package.json) will be used as the workspace by deft subcommands if no workspace.json file is found. Explicit workspaces are mainly needed when working on multiple interrelated packages at the same time.

Options:

--directory: Create the workspace under this directory instead of in the current working directory.

deft new workspace creates a new workspace directory and initializes it with a workspace.json file. The workspace name is the only required argument. Example:

$ deft new workspace my-app
$ cd my-app
$ ls -l
total 8
-rw-r--r-- 1 you you   28 Dec 29 18:03 workspace.json

Clone repositories in the top-level workspace directory to create active packages (or create them with deft new library and deft new application), then run deft update.

See also: Workspaces

deft publish#

The “publish” subcommand adds a new release of a package to the package catalog.

Synopsis: deft publish <pacman-catalog-directory>

Note

For now, until a fully automated solution is implemented, the publish command works by modifying a local copy of the catalog so that you can manually submit a pull request. This eliminates a lot of possibilities for making mistakes while editing the catalog by hand.

This command publishes a package associated with the current workspace. It searches up from the current directory to find dylan-package.json. Note that this means you can’t be in the root of a multi-package workspace. Once you’re satisfied that you’re ready to release a new version of your package (tests pass, doc updated, etc.) follow these steps:

Update the "version" attribute in dylan-package.json to be the new release’s version.

Also update any dependencies as needed. Normally this will happen naturally during development as you discover you need newer package versions, but this is a good time to review deps and update to get bug fixes if desired. Remember to deft update and re-run your tests if you change deps!

Push the above changes (if any) to your main branch.
Make a new release on GitHub with a tag that matches the release version. For example, if the "version" attribute in dylan-package.json is "0.5.0" the GitHub release should be tagged v0.5.0.
Clone https://github.com/dylan-lang/pacman-catalog somewhere and create a new branch named after your package.
```
$ cd /tmp
$ git clone https://github.com/dylan-lang/pacman-catalog
$ cd pacman-catalog
$ git switch -t -c my-package
```
In the next step the deft publish command will make changes there for you.
Run deft publish /tmp/pacman-catalog, pointing to where you just cloned the pacman catalog.
Commit the changes to pacman-catalog and submit a pull request. The tests to verify the catalog will be run automatically by the GitHub CI.
Once your PR has been merged, verify that the package is available in the catalog by running deft install my-package@0.5.0, substituting your new package name and release version.

deft status#

Display the status of the current workspace.

Synopsis: deft status

Options:

--directory: Only show the workspace directory and skip showing the active packages. This is intended for use by tooling.

Example:

$ deft status
Workspace: /home/cgay/dylan/workspaces/dt/
Active packages:
  http                     : ## master...origin/master (dirty)
  deft                     : ## dev...master [ahead 2] (dirty)
  pacman-catalog           : ## publish...master [ahead 1] (dirty)

deft update#

Update the workspace based on the current set of active packages.

Synopsis: deft update

The “update” command may be run from anywhere inside a workspace directory and performs two actions:

Installs all active package dependencies, as specified in their dylan-package.json files. Any time these dependencies are changed you should run deft update again.
Updates the registry to have an entry for each library in the workspace’s active packages or their dependencies.

The registry directory is created in the root of the workspace and all registry files are written to a subdirectory named after the local platform.

If a dependency is also an active package in this workspace, the active package is preferred over the specific version listed as a dependency.

Note

Registry files are only created if they apply to the platform of the local machine. For example, on the x86_64-linux platform LID files that specify Platforms: win32 will not cause a registry file to be generated.

Example:

Create a workspace named dt, with one active package, “deft”, update it, and build the test suite:

$ deft new workspace dt
$ cd dt
$ git clone --recursive https://github.com/dylan-lang/deft
$ deft update
$ deft build deft-test-suite

deft version#

Show the version of the deft command you are using. This is the Git version from which deft was compiled.

Synopsis: deft version

Deft - The Dylan CLI#

Terminology#

Requirements#

Where are Packages Installed?#

Building From Source#

Quick Start#

Workspaces#

The Registry#

Platform-specific Libraries#

Package Manager#

Global deft Options#

Subcommands#

deft help#

deft build#

deft install#

deft list#

deft new application#

deft new library#

deft new workspace#

deft publish#

deft status#

deft update#

deft version#

Index and Search#

Global `deft` Options#