nunavut

Code generator built on top of pydsdl.

Nunavut uses pydsdl to generate text files using templates. While these text files are often source code this module could also be used to generate documentation or data interchange formats like JSON or XML.

The input to the nunavut library is a list of templates and a list of pydsdl.pydsdl.CompositeType objects. The latter is typically obtained by calling pydsdl:

from pydsdl import read_namespace

compound_types = read_namespace(root_namespace, include_paths)

Next a nunavut.LanguageContext is needed which is used to configure all Nunavut objects for a specific target language

from nunavut.lang import LanguageContext

# Here we are going to generate C headers.
language_context = LanguageContext('c')

nunavut.generators.AbstractGenerator objects require a nunavut.Namespace tree which can be built from the pydsdl type map using nunavut.build_namespace_tree():

from nunavut import build_namespace_tree

root_namespace = build_namespace_tree(compound_types,
                                      root_ns_folder,
                                      out_dir,
                                      language_context)

Putting this all together, the typical use of this library looks something like this:

from pydsdl import read_namespace
from nunavut import build_namespace_tree
from nunavut.lang import LanguageContext
from nunavut.jinja import DSDLCodeGenerator

# parse the dsdl
compound_types = read_namespace(root_namespace, include_paths)

# select a target language
language_context = LanguageContext('c')

# build the namespace tree
root_namespace = build_namespace_tree(compound_types,
                                      root_ns_folder,
                                      out_dir,
                                      language_context)

# give the root namespace to the generator and...
generator = DSDLCodeGenerator(root_namespace)

# generate all the code!
generator.generate_all()
class nunavut.Namespace(full_namespace: str, root_namespace_dir: pathlib.Path, base_output_path: pathlib.PurePath, language_context: nunavut.lang.LanguageContext)[source]

Bases: pydsdl._expression._any.Any

K-ary tree (where K is the largest set of data types in a single dsdl namespace) where the nodes represent dsdl namespaces and the children are the datatypes and other nested namespaces (with datatypes always being leaf nodes). This structure extends pydsdl.Any and is a pydsdl.pydsdl.CompositeType via duck typing.

Parameters:
  • full_namespace (str) – The full, dot-separated name of the namepace. This is expected to be a unique identifier.
  • root_namespace_dir (pathlib.Path) – The directory representing the dsdl namespace and containing the namespaces’s datatypes and nested namespaces.
  • base_output_path (pathlib.PurePath) – The base path under which all namespaces and datatypes should be generated.
  • language_context (LanguageContext) – The generated software language context the namespace is within.
DefaultOutputStem = '_'
output_folder

The folder where this namespace’s output file and datatypes are generated.

get_support_output_folder() → pathlib.PurePath[source]

The folder under which support artifacts are generated.

get_language_context() → nunavut.lang.LanguageContext[source]

The generated software language context the namespace is within.

get_root_namespace() → nunavut.Namespace[source]

Traverses the namespace tree up to the root and returns the root node.

Returns:The root namespace object.
get_nested_namespaces() → Iterator[nunavut.Namespace][source]

Get an iterator over all the nested namespaces within this namespace. This is a shallow iterator that only provides directly nested namespaces.

get_nested_types() → ItemsView[pydsdl._serializable._composite.CompositeType, pathlib.Path][source]

Get a view of a tuple relating datatypes in this namespace to the path for the type’s generated output. This is a shallow view including only the types directly within this namespace.

get_all_datatypes() → Generator[Tuple[pydsdl._serializable._composite.CompositeType, pathlib.Path], None, None][source]

Generates tuples relating datatypes at and below this namespace to the path for each type’s generated output.

get_all_namespaces() → Generator[Tuple[nunavut.Namespace, pathlib.Path], None, None][source]

Generates tuples relating nested namespaces at and below this namespace to the path for each namespace’s generated output.

get_all_types() → Generator[Tuple[pydsdl._expression._any.Any, pathlib.Path], None, None][source]

Generates tuples relating datatypes and nested namespaces at and below this namespace to the path for each type’s generated output.

find_output_path_for_type(any_type: pydsdl._expression._any.Any) → pathlib.Path[source]

Searches the entire namespace tree to find a mapping of the type to an output file path.

Parameters:any_type (Any) – Either a Namespace or pydsdl.CompositeType to find the output path for.
Returns:The path where a file will be generated for a given type.
Raises:KeyError – If the type was not found in this namespace tree.
full_name
full_namespace
source_file_path
data_types
attributes
nunavut.build_namespace_tree(types: List[pydsdl._serializable._composite.CompositeType], root_namespace_dir: str, output_dir: str, language_context: nunavut.lang.LanguageContext) → nunavut.Namespace[source]

Generates a nunavut.Namespace tree.

Given a list of pydsdl types, this method returns a root nunavut.Namespace. The root nunavut.Namespace is the top of a tree where each node contains references to nested nunavut.Namespace and to any pydsdl.CompositeType instances contained within the namespace.

Parameters:
  • types (list) – A list of pydsdl types.
  • root_namespace_dir (str) – A path to the folder which is the root namespace.
  • output_dir (str) – The base directory under which all generated files will be created.
  • language_context (nunavut.lang.LanguageContext) – The language context to use when building nunavut.Namespace objects.
Returns:

The root nunavut.Namespace.

nunavut.generate_types(language_key: str, root_namespace_dir: pathlib.Path, out_dir: pathlib.Path, omit_serialization_support: bool = True, is_dryrun: bool = False, allow_overwrite: bool = True, lookup_directories: Optional[Iterable[str]] = None, allow_unregulated_fixed_port_id: bool = False, language_options: Optional[Mapping[str, Any]] = None) → None[source]

Helper method that uses default settings and built-in templates to generate types for a given language. This method is the most direct way to generate code using Nunavut.

Parameters:
  • language_key (str) – The name of the language to generate source for. See the Template Language Guide for details on available language support.
  • root_namespace_dir (pathlib.Path) – The path to the root of the DSDL types to generate code for.
  • out_dir (pathlib.Path) – The path to generate code at and under.
  • omit_serialization_support (bool) – If True then logic used to serialize and deserialize data is omitted.
  • is_dryrun (bool) – If True then nothing is generated but all other activity is performed and any errors that would have occurred are reported.
  • allow_overwrite (bool) – If True then generated files are allowed to overwrite existing files under the out_dir path.
  • lookup_directories (typing.Optional[typing.Iterable[str]]) – Additional directories to search for dependent types referenced by the types provided under the root_namespace_dir. Types will not be generated for these unless they are used by a type in the root namespace.
  • allow_unregulated_fixed_port_id (bool) – If True then errors will become warning when using fixed port identifiers for unregulated datatypes.
  • typing.Any]] language_options (typing.Optional[typing.Mapping[str,) – Opaque arguments passed through to the language objects. The supported arguments and valid values are different depending on the language specified by the language_key parameter.
class nunavut.YesNoDefault[source]

Bases: enum.Enum

Trinary type for decisions that allow a default behavior to be requested that can be different based on other contexts. For example:

def should_we_order_pizza(answer: YesNoDefault) -> bool:
    if answer == YesNoDefault.YES or (
       answer == YesNoDefault.DEFAULT and
       datetime.today().isoweekday() == 5):
        # if yes or if we are taking the default action which is to
        # order pizza on Friday, and today is Friday, then we order pizza
        return True
    else:
        return False
test_truth = <bound method YesNoDefault.test_truth of <enum 'YesNoDefault'>>[source]
NO = 0
YES = 1
DEFAULT = 2

Submodules

nunavut.dependencies

Objects and utilities for handling DSDL dependencies when generating code for a given type.

class nunavut.dependencies.Dependencies[source]

Bases: object

Data structure that contains a set of composite types and annotations (bool flags) which constitute a set of dependencies for a set of DSDL objects.

class nunavut.dependencies.DependencyBuilder(*dependant_types)[source]

Bases: object

Given a list of DSDL types this object builds a set of types that the given types use.

Parameters:dependant_types (typing.Iterable[pydsdl.Any]) – A list of types to build dependencies for.
transitive() → nunavut.dependencies.Dependencies[source]

Build a set of all transitive dependencies for the dependent types set for this builder.

direct() → nunavut.dependencies.Dependencies[source]

Build a set of all first-order dependencies for the dependent types set for this builder.

nunavut.generators

Module containing types and utilities for building generator objects. Generators abstract the code generation technology used to transform pydsdl AST into source code.

class nunavut.generators.AbstractGenerator(namespace: nunavut.Namespace, generate_namespace_types: nunavut._utilities.YesNoDefault = <YesNoDefault.DEFAULT: 2>)[source]

Bases: object

Abstract base class for classes that generate source file output from a given pydsdl parser result.

Parameters:
  • namespace (nunavut.Namespace) – The top-level namespace to generates types at and from.
  • generate_namespace_types (YesNoDefault) – Set to YES to force generation files for namespaces and NO to suppress. DEFAULT will generate namespace files based on the language preference.
namespace

The root nunavut.Namespace for this generator.

generate_namespace_types

If true then the generator is set to emit files for nunavut.Namespace in addition to the pydsdl datatypes. If false then only files for pydsdl datatypes will be generated.

get_templates() → Iterable[pathlib.Path][source]

Enumerate all templates found in the templates path. :return: A list of paths to all templates found by this Generator object.

generate_all(is_dryrun: bool = False, allow_overwrite: bool = True) → Iterable[pathlib.Path][source]

Generates all output for a given nunavut.Namespace and using the templates found by this object.

Parameters:
  • is_dryrun (bool) – If True then no output files will actually be written but all other operations will be performed.
  • allow_overwrite (bool) – If True then the generator will attempt to overwrite any existing files it encounters. If False then the generator will raise an error if the output file exists and the generation is not a dry-run.
Returns:

0 for success. Non-zero for errors.

Raises:

PermissionError if allow_overwrite is False and the file exists.

nunavut.generators.create_generators(namespace: nunavut.Namespace, **kwargs) → Tuple[nunavut.generators.AbstractGenerator, nunavut.generators.AbstractGenerator][source]

Create the two generators used by Nunavut; a code-generator and a support-library generator.

Parameters:
  • namespace (nunavut.Namespace) – The namespace to generate code within.
  • kwargs – A list of arguments that are forwarded to the generator constructors.
Returns:

Tuple with the first item being the code-generator and the second the support-library generator.

nunavut.postprocessors

Module containing post processing logic to run on generated files.

class nunavut.postprocessors.PostProcessor[source]

Bases: object

Abstract base class for all post processor functors.

class nunavut.postprocessors.FilePostProcessor[source]

Bases: nunavut.postprocessors.PostProcessor

Abstract base class for all post processor functors that are invoked after a file is written.

All file post processors are callable with the generated file pathlib.Path as the sole argument.

Example Usage:

class ClangFormat(FilePostProcessor):
    """
    Invoke clang-format on each file after it is generated.
    """
    def __init__(self, clang_format_path: str):
        self._clang_format_args = [clang_format_path, '-i']

    def __call__(self, generated: pathlib.Path) -> pathlib.Path:
        subprocess.run(self._clang_format_args + [str(generated)])
        return generated

...
class nunavut.postprocessors.LinePostProcessor[source]

Bases: nunavut.postprocessors.PostProcessor

Abstract base class for all post processor functors that are invoked after a line is generated from a template but before it is written to the output file.

All line post processors are callable with a 2-tuple containing the contents of the line as the first item and any newline characters as the second item. Note that if there are no newlines generated or if the last line generated does not end with a newline then this post-processor will be invoked at least once with the second item in the tuple as an empty string.

Important

Providing even a single LinePostProcessor to a generator may have a significant impact on generation performance. Some underlying generators (e.g. Jinja) are optimized to stream output based on internal buffer sizes and are not line oriented. For such implementations nunavut will have to create an intermediate line buffer which may impact performance.

Example Usage:

class CommentItAllOut(nunavut.postprocessors.LinePostProcessor):

    def __init__(self, open_line_comment: str, close_line_comment: str):
        self._line_comment_pattern = open_line_comment + ' {} ' + close_line_comment

    def __call__(self, line_and_lineend: typing.Tuple[str, str]) -> typing.Tuple[str, str]:
        if len(line_and_lineend[0]) > 0:
            return (self._line_comment_pattern.format(line_and_lineend[0]), line_and_lineend[1])
        else:
            return ('', '')

...

c_style = CommentItAllOut('/*', '*/')
my_generator.generate_all(False, True, [c_style])
class nunavut.postprocessors.SetFileMode(file_mode: int)[source]

Bases: nunavut.postprocessors.FilePostProcessor

Set the file mode after a file is generated using the pathlib.Path.chmod(mode) API.

Parameters:file_mode (int) – The file permissions to set for the file.
class nunavut.postprocessors.ExternalProgramEditInPlace(command_line: List[str], check: bool = True)[source]

Bases: nunavut.postprocessors.FilePostProcessor

Run an external program after generating a file. This version expects the program to either not modify the file or to modify it in-place (e.g. the functor always returns the same path it was provided).

Parameters:
  • command_line (typing.List[str]) – The command and arguments to pass to the external program using subprocess.run. The file to be processed will be appended as the last positional argument in the command before it is invoked.
  • check (bool) – By default, if the external program returns a non-zero exit status a subprocess.CalledProcessError is raised. Set this argument to False to ignore external program errors.
class nunavut.postprocessors.TrimTrailingWhitespace[source]

Bases: nunavut.postprocessors.LinePostProcessor

Remove all trailing whitespace from each line.

Important

See performance note in LinePostProcessor documentation. Consider invoking a code formatter from a FilePostProcessor instead.

class nunavut.postprocessors.LimitEmptyLines(max_empty_lines: int)[source]

Bases: nunavut.postprocessors.LinePostProcessor

Set a limit to the number of consecutive empty lines to allow.

Important

See performance note in LinePostProcessor documentation. Consider invoking a code formatter from a FilePostProcessor instead.

nunavut.templates

Abstractions around template engine internals.

nunavut.templates.ENVIRONMENT_FILTER_ATTRIBUTE_NAME = 'environmentfilter'

For now this is set to a value that is internally compatible with the embedded version of jinja2 we use in Nunavut. If you use this variable instead of its current value you will be insulated from this.

nunavut.templates.CONTEXT_FILTER_ATTRIBUTE_NAME = 'contextfilter'

For now this is set to a value that is internally compatible with the embedded version of jinja2 we use in Nunavut. If you use this variable instead of its current value you will be insulated from this.

nunavut.templates.LANGUAGE_FILTER_ATTRIBUTE_NAME = 'nv_languagefilter'

Nunavut-specific attribute for filters or tests that take their nunavut.lang.Language as the first argument.

class nunavut.templates.SupportsTemplateContext[source]

Bases: object

Provided as a pseudo protocol. (in anticipation of that becoming part of the core Python typing someday).

nunavut.templates.template_environment_filter(filter_func: Callable) → Callable[[...], str][source]

Decorator for marking environment dependent filters. An object supporting the SupportsTemplateEnv protocol will be passed to the filter as the first argument.

nunavut.templates.template_context_filter(filter_func: Callable) → Callable[[...], str][source]

Decorator for marking context dependent filters. An object supporting the SupportsTemplateContext protocol will be passed to the filter as the first argument.

Note that any template that uses such a filter will make the jinja “frame” the filter appears within volatile and therefore unable to be optimized.

nunavut.templates.template_volatile_filter(filter_func: Callable) → Callable[[...], str][source]

Decorator for marking a filter as volatile therefore disabling optimizations for the frame it appears within. An opaque object will be passed to the filter as the first argument.

class nunavut.templates.GenericTemplateLanguageFilter(language_name_or_module: str)[source]

Bases: typing.Generic

Decorator for marking template filters that take a nunavut.lang.Language object as the first argument with a generic return type.

class nunavut.templates.template_language_filter(language_name_or_module: str)[source]

Bases: nunavut.templates.GenericTemplateLanguageFilter

Decorator for marking template filters that take a nunavut.lang.Language object as the first argument.

class nunavut.templates.template_language_list_filter(language_name_or_module: str)[source]

Bases: nunavut.templates.GenericTemplateLanguageFilter

Decorator for marking template filters that take a nunavut.lang.Language object as the first argument and return a list of strings.

class nunavut.templates.template_language_int_filter(language_name_or_module: str)[source]

Bases: nunavut.templates.GenericTemplateLanguageFilter

Decorator for marking template filters that take a nunavut.lang.Language object as the first argument and return an integer.

class nunavut.templates.template_language_test(language_name_or_module: str)[source]

Bases: nunavut.templates.GenericTemplateLanguageFilter

Decorator for marking template tests that take a nunavut.lang.Language object as the first argument.

class nunavut.templates.LanguageEnvironment(language_name: str)[source]

Bases: object

Data structure defining stuff contributed to a template environment for a given Language.

TEST_NAME_PREFIX = 'is_'
FILTER_NAME_PREFIX = 'filter_'
USES_QUERY_PREFIX = 'uses_'
classmethod is_test_name(callable_name: Optional[str]) → bool[source]
classmethod is_filter_name(callable_name: Optional[str]) → bool[source]
classmethod is_uses_query_name(callable_name: Optional[str]) → bool[source]
LanguageListT = ~LanguageListT
language_name
tests
filters
uses_queries
classmethod handle_conventional_methods(callable: Callable, callable_name: Optional[str] = None, supported_languages: Optional[LanguageListT] = None) → Tuple[Optional[str], str, Callable][source]

Processes method objects that utilize the nunavut convention of is_, filter_, or uses_ prefixes. Also wraps the method in a partial if it requested the language as the first argument.

Parameters:callable_name (str) – If provided this is the name used to process the callable otherwise the __name__ property is used from the callable itself.
Returns:A 3-tuple with the prefix, method name without prefix, and the method or partial. If the first element is None then the callable was not a conventional method.
classmethod find_all_conventional_methods_in_language_module(language: nunavut.lang.Language, all_languages: LanguageListT, language_module: module) → nunavut.templates.LanguageEnvironment[source]

nunavut.version

nunavut.version.__version__ = '1.8.3'

The version number used in the release of nunavut to pypi.