nunavut (library)

nunavut

Code generator built on top of pydsdl.

Nunavut uses pydsdl to generate text files using templates. While these text files are often source code this module could also be used to generate documentation or data interchange formats like JSON or XML.

The input to the nunavut library is a list of templates and a list of pydsdl.pydsdl.CompositeType objects. The latter is typically obtained by calling pydsdl:

from pydsdl import read_namespace

compound_types = read_namespace(root_namespace, include_paths)

nunavut.generators.AbstractGenerator objects require a nunavut.Namespace tree which can be built from the pydsdl type map using nunavut.build_namespace_tree():

from nunavut import build_namespace_tree

root_namespace = build_namespace_tree(compound_types,
                                      root_ns_folder,
                                      out_dir,
                                      '.hpp',
                                      'Types')

Putting this all together, the typical use of this library looks something like this:

from pydsdl import read_namespace
from nunavut import build_namespace_tree
from nunavut.jinja import Generator

# parse the dsdl
compound_types = read_namespace(root_namespace, include_paths)

# build the namespace tree
root_namespace = build_namespace_tree(compound_types,
                                      root_ns_folder,
                                      out_dir,
                                      '.hpp',
                                      'Types')

# give the root namespace to the generator and...
generator = Generator(root_namespace, False, templates_dir)

# generate all the code!
generator.generate_all()
class nunavut.Namespace(full_namespace: str, root_namespace_dir: pathlib.Path, base_output_path: pathlib.PurePath, extension: str, namespace_file_stem: str)[source]

K-ary tree (where K is the largest set of data types in a single dsdl namespace) where the nodes represent dsdl namespaces and the children are the datatypes and other nested namespaces (with datatypes always being leaf nodes). This structure extends pydsdl.Any and is a pydsdl.pydsdl.CompositeType via duck typing.

Parameters:
  • full_namespace (str) – The full, dot-separated name of the namepace. This is expected to be a unique identifier.
  • root_namespace_dir (pathlib.Path) – The directory representing the dsdl namespace and containing the namespaces’s datatypes and nested namespaces.
  • base_output_path (pathlib.PurePath) – The base path under which all namespaces and datatypes should be generated.
  • extension (str) – The file suffix to give to generated files.
  • namespace_file_stem (str) – The file stem (name) to give to files generated for namespaces.
output_folder

The folder where this namespace’s output file and datatypes are generated.

get_root_namespace() → nunavut.Namespace[source]

Traverses the namespace tree up to the root and returns the root node.

Returns:The root namepace object.
get_nested_namespaces() → Iterator[nunavut.Namespace][source]

Get an iterator over all the nested namespaces within this namespace. This is a shallow iterator that only provides directly nested namespaces.

get_nested_types() → ItemsView[pydsdl._serializable.CompositeType, pathlib.Path][source]

Get a view of a tuple relating datatypes in this namepace to the path for the type’s generated output. This is a shallow view including only the types directly within this namespace.

get_all_datatypes() → Generator[Tuple[pydsdl._serializable.CompositeType, pathlib.Path], None, None][source]

Generates tuples relating datatypes at and below this namepace to the path for each type’s generated output.

get_all_namespaces() → Generator[Tuple[nunavut.Namespace, pathlib.Path], None, None][source]

Generates tuples relating nested namespaces at and below this namepace to the path for each namespace’s generated output.

get_all_types() → Generator[Tuple[pydsdl._expression.Any, pathlib.Path], None, None][source]

Generates tuples relating datatypes and nested namespaces at and below this namepace to the path for each type’s generated output.

find_output_path_for_type(any_type: pydsdl._expression.Any) → pathlib.Path[source]

Searches the entire namespace tree to find a mapping of the type to an output file path.

Parameters:any_type (Any) – Either a Namespace or pydsdl.CompositeType to find the output path for.
Returns:The path where a file will be generated for a given type.
Raises:KeyError – If the type was not found in this namespace tree.
nunavut.build_namespace_tree(types: List[pydsdl._serializable.CompositeType], root_namespace_dir: str, output_dir: str, extension: str, namespace_output_stem: str) → nunavut.Namespace[source]

Generates a nunavut.Namespace tree.

Given a list of pydsdl types, this method returns a root nunavut.Namespace. The root nunavut.Namespace is the top of a tree where each node contains references to nested nunavut.Namespace and to any pydsdl.CompositeType instances contained within the namespace.

Parameters:
  • types (list) – A list of pydsdl types.
  • root_namespace_dir (str) – A path to the folder which is the root namespace.
  • output_dir (str) – The base directory under which all generated files will be created.
  • extension (str) – The extension to use for generated file types. All paths and filenames are built using pathlib. See pathlib documentation for platform differences when forming paths, filenames, and extensions.
  • namespace_output_stem (str) – The filename stem to give to Namespace output files if emitted.
Returns:

The root nunavut.Namespace.

nunavut.generators

Module containing types and utilities for building generator objects. Generators abstract the code generation technology used to transform pydsdl AST into source code.

class nunavut.generators.AbstractGenerator(namespace: nunavut.Namespace, generate_namespace_types: bool)[source]

Abstract base class for classes that generate source file output from a given pydsdl parser result.

Parameters:
  • namespace (nunavut.Namespace) – The top-level namespace to generates types at and from.
  • generate_namespace_types (bool) – Set to true to emit files for namespaces. False will only generate files for datatypes.
namespace

The root nunavut.Namespace for this generator.

generate_namespace_types

If true then the generator is set to emit files for nunavut.Namespace in addition to the pydsdl datatypes. If false then only files for pydsdl datatypes will be generated.

generate_all(is_dryrun: bool = False, allow_overwrite: bool = True, post_processors: Optional[List[nunavut.postprocessors.PostProcessor]] = None) → int[source]

Generates all output for a given nunavut.Namespace and using the templates found by this object.

Parameters:
  • is_dryrun (bool) – If True then no output files will actually be written but all other operations will be performed.
  • allow_overwrite (bool) – If True then the generator will attempt to overwrite any existing files it encounters. If False then the generator will raise an error if the output file exists and the generation is not a dry-run.
  • post_processors – A list of nunavut.postprocessors.PostProcessor

nunavut.postprocessors

Module containing post processing logic to run on generated files.

class nunavut.postprocessors.PostProcessor[source]

Abstract base class for all post processor functors.

__call__(generated: Any) → Optional[Any][source]

Call self as a function.

class nunavut.postprocessors.FilePostProcessor[source]

Abstract base class for all post processor functors that are invoked after a file is written.

All file post processors are callable with the generated file pathlib.Path as the sole argument.

Example Usage:

class ClangFormat(FilePostProcessor):
    """
    Invoke clang-format on each file after it is generated.
    """
    def __init__(self, clang_format_path: str):
        self._clang_format_args = [clang_format_path, '-i']

    def __call__(self, generated: pathlib.Path) -> pathlib.Path:
        subprocess.run(self._clang_format_args + [str(generated)])
        return generated

...

my_generator.generate_all(False, True, [ClangFormat('clang-format')])
__call__(generated: pathlib.Path) → pathlib.Path[source]

Performs the post-processing action on the generated file.

Parameters:generated (pathlib.Path) – The path of the generated file.
Returns:A path to the generated file. This may be a modified path for some post-processors.
class nunavut.postprocessors.LinePostProcessor[source]

Abstract base class for all post processor functors that are invoked after a line is generated from a template but before it is written to the output file.

All line post processors are callable with a 2-tuple containing the contents of the line as the first item and any newline characters as the second item. Note that if there are no newlines generated or if the last line generated does not end with a newline then this post-processor will be invoked at least once with the second item in the tuple as an empty string.

Important

Providing even a single LinePostProcessor to a generator may have a significant impact on generation performance. Some underlying generators (e.g. Jinja) are optimized to stream output based on internal buffer sizes and are not line oriented. For such implementations nunavut will have to create an intermediate line buffer which may impact performance.

Example Usage:

class CommentItAllOut(nunavut.postprocessors.LinePostProcessor):

    def __init__(self, open_line_comment: str, close_line_comment: str):
        self._line_comment_pattern = open_line_comment + ' {} ' + close_line_comment

    def __call__(self, line_and_lineend: typing.Tuple[str, str]) -> typing.Tuple[str, str]:
        if len(line_and_lineend[0]) > 0:
            return (self._line_comment_pattern.format(line_and_lineend[0]), line_and_lineend[1])
        else:
            return ('', '')

...

c_style = CommentItAllOut('/*', '*/')
my_generator.generate_all(False, True, [c_style])
__call__(line_and_lineend: Tuple[str, str]) → Tuple[str, str][source]

Performs a post-processing action on a generated line of text.

Parameters:line_and_lineend (str) – A tuple where the first argument is the line generated from the template and the second are any new line characters.
Returns:A tuple with the first item being the content of the line and the second being any newline characters to append. It is reccommended that the newline characters are treated as opaque since these tend to be different on various platforms. Line post-processors are encouraged to either pass alone the line endings provided as the second item the line_and_lineend argument or to return an empty string to elide any newline characters for this line. Returning a 2-tuple of empty strings is the same as eliding the entire line.
class nunavut.postprocessors.SetFileMode(file_mode: int)[source]

Set the file mode after a file is generated using the pathlib.Path.chmod(mode) API.

Parameters:file_mode (int) – The file permissions to set for the file.
__call__(generated: pathlib.Path) → pathlib.Path[source]

Performs the post-processing action on the generated file.

Parameters:generated (pathlib.Path) – The path of the generated file.
Returns:A path to the generated file. This may be a modified path for some post-processors.
class nunavut.postprocessors.ExternalProgramEditInPlace(command_line: List[str], check: bool = True)[source]

Run an external program after generating a file. This version expects the program to either not modify the file or to modify it in-place (e.g. the functor always returns the same path it was provided).

Parameters:
  • command_line (typing.List[str]) – The command and arguments to pass to the external program using subprocess.run. The file to be processed will be appended as the last positional argument in the command before it is invoked.
  • check (bool) – By default, if the external program returns a non-zero exit status a subprocess.CalledProcessError is raised. Set this argument to False to ignore external program errors.
__call__(generated: pathlib.Path) → pathlib.Path[source]

Performs the post-processing action on the generated file.

Parameters:generated (pathlib.Path) – The path of the generated file.
Returns:A path to the generated file. This may be a modified path for some post-processors.
class nunavut.postprocessors.TrimTrailingWhitespace[source]

Remove all trailing whitespace from each line.

Important

See performance note in LinePostProcessor documentation. Consider invoking a code formatter from a FilePostProcessor instead.

__call__(line_and_lineend: Tuple[str, str]) → Tuple[str, str][source]

Performs a post-processing action on a generated line of text.

Parameters:line_and_lineend (str) – A tuple where the first argument is the line generated from the template and the second are any new line characters.
Returns:A tuple with the first item being the content of the line and the second being any newline characters to append. It is reccommended that the newline characters are treated as opaque since these tend to be different on various platforms. Line post-processors are encouraged to either pass alone the line endings provided as the second item the line_and_lineend argument or to return an empty string to elide any newline characters for this line. Returning a 2-tuple of empty strings is the same as eliding the entire line.
class nunavut.postprocessors.LimitEmptyLines(max_empty_lines: int)[source]

Set a limit to the number of consecutive empty lines to allow.

Important

See performance note in LinePostProcessor documentation. Consider invoking a code formatter from a FilePostProcessor instead.

__call__(line_and_lineend: Tuple[str, str]) → Tuple[str, str][source]

Performs a post-processing action on a generated line of text.

Parameters:line_and_lineend (str) – A tuple where the first argument is the line generated from the template and the second are any new line characters.
Returns:A tuple with the first item being the content of the line and the second being any newline characters to append. It is reccommended that the newline characters are treated as opaque since these tend to be different on various platforms. Line post-processors are encouraged to either pass alone the line endings provided as the second item the line_and_lineend argument or to return an empty string to elide any newline characters for this line. Returning a 2-tuple of empty strings is the same as eliding the entire line.

nunavut.jinja

jinja-based AbstractGenerator implementation.

class nunavut.jinja.JinjaAssert(environment: nunavut.jinja.jinja2.environment.Environment)[source]

Jinja2 extension that allows {% assert T.attribute %} statements. Templates should uses these statements where False values would result in malformed source code.

parse(parser: nunavut.jinja.jinja2.parser.Parser) → nunavut.jinja.jinja2.nodes.Node[source]

See http://jinja.pocoo.org/docs/2.10/extensions/ for help writing extensions.

class nunavut.jinja.Generator(namespace: nunavut.Namespace, generate_namespace_types: bool, templates_dir: pathlib.Path, followlinks: bool = False, trim_blocks: bool = False, lstrip_blocks: bool = False, additional_filters: Optional[Dict[str, Callable]] = None, additional_tests: Optional[Dict[str, Callable]] = None)[source]

AbstractGenerator implementation that uses Jinja2 templates to generate source code.

Parameters:
  • namespace (nunavut.Namespace) – The top-level namespace to generates types at and from.
  • generate_namespace_types (bool) – typing.Set to true to emit files for namespaces. False will only generate files for datatypes.
  • templates_dir (pathlib.Path) – The directory containing the jinja templates.
  • followlinks (bool) – If True then symbolic links will be followed when searching for templates.
  • trim_blocks (bool) – If this is set to True the first newline after a block is removed (block, not variable tag!).
  • lstrip_blocks (bool) – If this is set to True leading spaces and tabs are stripped from the start of a line to a block. Defaults to False.
  • typing.Callable] additional_filters (typing.Dict[str,) – typing.Optional jinja filters to add to the global environment using the key as the filter name and the callable as the filter.
  • typing.Callable] additional_tests (typing.Dict[str,) – typing.Optional jinja tests to add to the global environment using the key as the test name and the callable as the test.
Raises:

RuntimeError – If any additional filter or test attempts to replace a built-in or otherwise already defined filter or test.

TEMPLATE_SUFFIX = '.j2'

The suffix expected for Jinja templates.

static filter_yamlfy(value: Any) → str[source]

Filter to, optionally, emit a dump of the dsdl input as a yaml document. Available as yamlfy in all template environments.

Example:

/*
{{ T | yamlfy }}
*/

Result Example (truncated for brevity):

/*
!!python/object:pydsdl.StructureType
_attributes:
- !!python/object:pydsdl.Field
_serializable: !!python/object:pydsdl.UnsignedIntegerType
    _bit_length: 16
    _cast_mode: &id001 !!python/object/apply:pydsdl.CastMode
    - 0
_name: value
*/
Parameters:value – The input value to parse as yaml.
Returns:If pyyaml is available, a pretty dump of the given value as yaml. If pyyaml is not available then an empty string is returned.
filter_type_to_template(value: Any) → str[source]

Template for type resolution as a filter. Available as type_to_template in all template environments.

Example:

{%- for attribute in T.attributes %}
    {%* include attribute.data_type | type_to_template %}
    {%- if not loop.last %},{% endif %}
{%- endfor %}
Parameters:value – The input value to change into a template include path.
Returns:A path to a template named for the type with Generator.TEMPLATE_SUFFIX
filter_type_to_include_path(value: Any, resolve: bool = False) → str[source]

Emits and include path to the output target for a given type.

Example:

#include "{{ T.my_type | type_to_include_path }}"

Result Example:

#include “foo/bar/my_type.h”
Parameters:
  • value (typing.Any) – The type to emit an include for.
  • resolve (bool) – If True the path returned will be absolute else the path will be relative to the folder of the root namepace.
Returns:

A string path to output file for the type.

static filter_typename(value: Any) → str[source]

Filters a given token as its type name. Available as typename in all template environments.

This example supposes that T.some_value == "some string"

Example:

{{ T.some_value | typename }}

Result Example:

str
Parameters:value – The input value to filter into a type name.
Returns:The __name__ of the python type.
static is_primitive(value: pydsdl._expression.Any) → bool[source]

Tests if a given dsdl instance is a pydsdl.PrimitiveType. Available in all template environments as is primitive.

Example:

{% if field.data_type is primitive %}
    {{ field.data_type | c.type_from_primitive }} {{ field.name }};
{% endif -%}
Parameters:value – The instance to test.
Returns:True if value is an instance of pydsdl.PrimitiveType.
static is_constant(value: pydsdl._expression.Any) → bool[source]

Tests if a given dsdl instance is a pydsdl.Constant. Available in all template environments as is constant.

Example:

{%- if attribute is constant %}
    const {{ attribute.data_type | c.type_from_primitive(use_standard_types=True) }} {{ attribute.name }} = {{ attribute.initialization_expression }};
{% endif %}
Parameters:value – The instance to test.
Returns:True if value is an instance of pydsdl.Constant.
static is_serializable(value: pydsdl._expression.Any) → bool[source]

Tests if a given dsdl instance is a pydsdl.SerializableType. Available in all template environments as is serializable.

Example:

{%- if attribute is serializable %}
    // Yup, this is serializable
{% endif %}
Parameters:value – The instance to test.
Returns:True if value is an instance of pydsdl.SerializableType.
get_templates() → List[pathlib.Path][source]

Enumerate all templates found in the templates path. TEMPLATE_SUFFIX as the suffix for the filename.

Returns:A list of paths to all templates found by this Generator object.
generate_all(is_dryrun: bool = False, allow_overwrite: bool = True, post_processors: Optional[List[nunavut.postprocessors.PostProcessor]] = None) → int[source]

Generates all output for a given nunavut.Namespace and using the templates found by this object.

Parameters:
  • is_dryrun (bool) – If True then no output files will actually be written but all other operations will be performed.
  • allow_overwrite (bool) – If True then the generator will attempt to overwrite any existing files it encounters. If False then the generator will raise an error if the output file exists and the generation is not a dry-run.
  • post_processors – A list of nunavut.postprocessors.PostProcessor

nunavut.jinja.lang

Language-specific support in jinja templates.

This package contains modules that provide specific support for generating source for various languages using jinja templates.

nunavut.jinja.lang.get_supported_languages() → Iterable[str][source]

Get a list of languages this module supports.

Returns:An iterable of strings which are language names accepted by the add_language_support() function.
nunavut.jinja.lang.add_language_support(language_name: str, environment: nunavut.jinja.jinja2.environment.Environment) → None[source]

Inspects a given language support module and adds all functions found whose name starts with “filter_” to the provided environment as “[language_name].[function name minus ‘filter_’]”.

For example, if a language module foo.py has a filter function “filter_bar” and this method is called with a prefix of “foo” then the environment will have a filter name “foo.bar” added to it.

Parameters:
  • language_name (str) – The language to add support for.
  • environment (Environment) – The jinja2 environment to inject language support into.
Raises:

KeyError – If language_name is not a supported language.