nunavut.jinja

jinja-based AbstractGenerator implementation.

class nunavut.jinja.CodeGenerator(namespace: nunavut.Namespace, generate_namespace_types: nunavut._utilities.YesNoDefault = <YesNoDefault.DEFAULT: 2>, templates_dir: Union[pathlib.Path, List[pathlib.Path], None] = None, followlinks: bool = False, trim_blocks: bool = False, lstrip_blocks: bool = False, additional_filters: Optional[Dict[str, Callable]] = None, additional_tests: Optional[Dict[str, Callable]] = None, additional_globals: Optional[Dict[str, Any]] = None, post_processors: Optional[List[nunavut.postprocessors.PostProcessor]] = None, builtin_template_path: str = 'templates')[source]

Bases: nunavut.generators.AbstractGenerator

Abstract base class for all Generators that build source code using Jinja templates.

Parameters:
  • namespace (nunavut.Namespace) – The top-level namespace to generates code at and from.
  • generate_namespace_types (YesNoDefault) – Set to YES to emit files for namespaces. NO will suppress namespace file generation and DEFAULT will use the language’s preference.
  • templates_dir (typing.Optional[typing.Union[pathlib.Path,typing.List[pathlib.Path]]]) – Directories containing jinja templates. These will be available along with any built-in templates provided by the target language. The templates at these paths will take precedence masking any built-in templates where the names are the same. See jinja2.ChoiceLoader for rules on the lookup hierarchy.
  • followlinks (bool) – If True then symbolic links will be followed when searching for templates.
  • trim_blocks (bool) – If this is set to True the first newline after a block is removed (block, not variable tag!).
  • lstrip_blocks (bool) – If this is set to True leading spaces and tabs are stripped from the start of a line to a block. Defaults to False.
  • typing.Callable] additional_filters (typing.Dict[str,) – typing.Optional jinja filters to add to the global environment using the key as the filter name and the callable as the filter.
  • typing.Callable] additional_tests (typing.Dict[str,) – typing.Optional jinja tests to add to the global environment using the key as the test name and the callable as the test.
  • typing.Any] additional_globals (typing.Dict[str,) – typing.Optional objects to add to the template environment globals collection.
  • post_processors (typing.Optional[typing.List[nunavut.postprocessors.PostProcessor]]) – A list of nunavut.postprocessors.PostProcessor
  • builtin_template_path – If provided overrides the folder name under which built-in templates are loaded from within a target language’s package (i.e. ignored if no target language is specified). For example, if the target language is c and this parameter was set to foo then built-in templates would be loaded from nunavut.lang.c.foo.
Raises:

RuntimeError – If any additional filter or test attempts to replace a built-in or otherwise already defined filter or test.

dsdl_loader
language_context
get_templates() → Iterable[pathlib.Path][source]

Enumerate all templates found in the templates path. TEMPLATE_SUFFIX as the suffix for the filename.

Returns:A list of paths to all templates found by this Generator object.
class nunavut.jinja.DSDLCodeGenerator(namespace: nunavut.Namespace, **kwargs)[source]

Bases: nunavut.jinja.CodeGenerator

CodeGenerator implementation that generates code for a given set of DSDL types.

static filter_yamlfy(value: Any) → str[source]

Filter to, optionally, emit a dump of the dsdl input as a yaml document. Available as yamlfy in all template environments.

Example:

/*
{{ T | yamlfy }}
*/

Result Example (truncated for brevity):

/*
!!python/object:pydsdl.StructureType
_attributes:
- !!python/object:pydsdl.Field
_serializable: !!python/object:pydsdl.UnsignedIntegerType
    _bit_length: 16
    _cast_mode: &id001 !!python/object/apply:pydsdl.CastMode
    - 0
_name: value
*/
Parameters:value – The input value to parse as yaml.
Returns:If a yaml parser is available, a pretty dump of the given value as yaml. If a yaml parser is not available then an empty string is returned.
filter_type_to_template(value: Any) → str[source]

Template for type resolution as a filter. Available as type_to_template in all template environments.

Example:

{%- for attribute in T.attributes %}
    {%* include attribute.data_type | type_to_template %}
    {%- if not loop.last %},{% endif %}
{%- endfor %}
Parameters:value – The input value to change into a template include path.
Returns:A path to a template named for the type with TEMPLATE_SUFFIX
filter_type_to_include_path(value: Any, resolve: bool = False) → str[source]

Emits an include path to the output target for a given type.

Example:

# include "{{ T.my_type | type_to_include_path }}"

Result Example:

# include “foo/bar/my_type.h”
Parameters:
  • value (typing.Any) – The type to emit an include for.
  • resolve (bool) – If True the path returned will be absolute else the path will be relative to the folder of the root namespace.
Returns:

A string path to output file for the type.

static filter_typename(value: Any) → str[source]

Filters a given token as its type name. Available as typename in all template environments.

This example supposes that T.some_value == "some string"

Example:

{{ T.some_value | typename }}

Result Example:

str
Parameters:value – The input value to filter into a type name.
Returns:The __name__ of the python type.
static filter_alignment_prefix(offset: pydsdl._bit_length_set._bit_length_set.BitLengthSet) → str[source]

Provides a string prefix based on a given pydsdl.BitLengthSet.

# Given
B = pydsdl.BitLengthSet(32)

# and
template = '{{ B | alignment_prefix }}'

# then ('str' is stropped to 'str_' before the version is suffixed)
rendered = 'aligned'
# Given
B = pydsdl.BitLengthSet(32)
B += 1

# and
template = '{{ B | alignment_prefix }}'

# then ('str' is stropped to 'str_' before the version is suffixed)
rendered = 'unaligned'
Parameters:offset (pydsdl.BitLengthSet) – A bit length set to test for alignment.
Returns:‘aligned’ or ‘unaligned’ based on the state of the offset argument.
static filter_bit_length_set(values: Union[Iterable[int], int, None]) → pydsdl._bit_length_set._bit_length_set.BitLengthSet[source]

Convert an integer or a list of integers into a pydsdl.BitLengthSet.

static filter_remove_blank_lines(text: str) → str[source]
Remove blank lines from the supplied string. Lines that contain only whitespace characters are also considered blank.

456

789’) == ‘123 456 789’

static filter_bits2bytes_ceil(n_bits: int) → int[source]

Implements int(ceil(x/8)) | x >= 0.

static is_None(value: Any) → bool[source]

Tests if a value is None

static is_saturated(t: pydsdl._serializable._primitive.PrimitiveType) → bool[source]

Tests if a type is a saturated type or not.

static is_service_request(instance: pydsdl._expression._any.Any) → bool[source]

Tests if a type is request type of a service type.

static is_service_response(instance: pydsdl._expression._any.Any) → bool[source]

Tests if a type is response type of a service type.

static is_deprecated(instance: pydsdl._expression._any.Any) → bool[source]

Tests if a type is marked as deprecated

generate_all(is_dryrun: bool = False, allow_overwrite: bool = True) → Iterable[pathlib.Path][source]

Generates all output for a given nunavut.Namespace and using the templates found by this object.

Parameters:
  • is_dryrun (bool) – If True then no output files will actually be written but all other operations will be performed.
  • allow_overwrite (bool) – If True then the generator will attempt to overwrite any existing files it encounters. If False then the generator will raise an error if the output file exists and the generation is not a dry-run.
Returns:

0 for success. Non-zero for errors.

Raises:

PermissionError if allow_overwrite is False and the file exists.

class nunavut.jinja.SupportGenerator(namespace: nunavut.Namespace, **kwargs)[source]

Bases: nunavut.jinja.CodeGenerator

Generates output files by copying them from within the Nunavut package itself for non templates but uses jinja to generate headers from templates with the language environment provided but no T (DSDL type) global set. This generator always copies files from those returned by the file_iterator to locations under nunavut.Namespace.get_support_output_folder()

get_templates() → Iterable[pathlib.Path][source]

Enumerate all templates found in the templates path. TEMPLATE_SUFFIX as the suffix for the filename.

Returns:A list of paths to all templates found by this Generator object.
generate_all(is_dryrun: bool = False, allow_overwrite: bool = True) → Iterable[pathlib.Path][source]

Generates all output for a given nunavut.Namespace and using the templates found by this object.

Parameters:
  • is_dryrun (bool) – If True then no output files will actually be written but all other operations will be performed.
  • allow_overwrite (bool) – If True then the generator will attempt to overwrite any existing files it encounters. If False then the generator will raise an error if the output file exists and the generation is not a dry-run.
Returns:

0 for success. Non-zero for errors.

Raises:

PermissionError if allow_overwrite is False and the file exists.

Submodules

nunavut.jinja.environment

class nunavut.jinja.environment.LanguageTemplateNamespace(**kwargs)[source]

Bases: object

Generic namespace object used to create reserved namespaces in the global environment.

ns = LanguageTemplateNamespace()

# any property can be set at any time.
ns.foo = 'foo'
assert ns.foo == 'foo'

# repr of the ns enables cloning using exec
exec('ns2={}'.format(repr(ns)))
assert ns2.foo == 'foo'

# clones will be equal
assert ns2 == ns

# but not the same object
assert ns2 is not ns

In addition to the namespace behavior this object exposed some dictionary-like methods:

ns = LanguageTemplateNamespace()
ns.update({'foo':'bar'})

assert ns.foo == 'bar'
update(update_from: Mapping[str, Any]) → None[source]
items() → ItemsView[str, Any][source]
values() → ValuesView[Any][source]
class nunavut.jinja.environment.CodeGenEnvironment(loader: nunavut.jinja.jinja2.loaders.BaseLoader, lctx: Optional[nunavut.lang.LanguageContext] = None, trim_blocks: bool = False, lstrip_blocks: bool = False, additional_filters: Optional[Dict[str, Callable]] = None, additional_tests: Optional[Dict[str, Callable]] = None, additional_globals: Optional[Dict[str, Any]] = None, extensions: List[nunavut.jinja.jinja2.ext.Extension] = [<class 'nunavut.jinja.jinja2.ext.ExprStmtExtension'>, <class 'nunavut.jinja.jinja2.ext.LoopControlExtension'>, <class 'nunavut.jinja.extensions.JinjaAssert'>, <class 'nunavut.jinja.extensions.UseQuery'>], allow_filter_test_or_use_query_overwrite: bool = False)[source]

Bases: nunavut.jinja.jinja2.environment.Environment

Jinja Environment optimized for compile-time generation of source code (i.e. as opposed to dynamically generating webpages).

template = 'Hello World'

e = CodeGenEnvironment(loader=DictLoader({'test': template}))
assert 'Hello World' ==  e.get_template('test').render()

Warning

The RESERVED_GLOBAL_NAMESPACES and RESERVED_GLOBAL_NAMES collections contain names in the global namespace reserved by this environment. Attempting to override one of these reserved names will cause the constructor to raise an error.

try:
    CodeGenEnvironment(loader=DictLoader({'test': template}), additional_globals={'ln': 'bad_ln'})
    assert False
except RuntimeError:
    pass

Other safe-guards include checks that Jinja built-ins aren’t accidentally overridden…

try:
    CodeGenEnvironment(loader=DictLoader({'test': template}),
                                         additional_filters={'indent': lambda x: x})
    assert False
except RuntimeError:
    pass

# You can allow overwrite of built-ins using the ``allow_filter_test_or_use_query_overwrite``
# argument.
e = CodeGenEnvironment(loader=DictLoader({'test': template}),
                                         additional_filters={'indent': lambda x: x},
                                         allow_filter_test_or_use_query_overwrite=True)
assert 'foo' == e.filters['indent']('foo')

…or that user-defined filters or redefined.

class MyFilters:

    @staticmethod
    def filter_misnamed(name: str) -> str:
        return name

e = CodeGenEnvironment(loader=DictLoader({'test': template}),
                       additional_filters={'filter_misnamed': lambda x: x})

try:
    e.add_conventional_methods_to_environment(MyFilters())
    assert False
except RuntimeError:
    pass

Note

Maintainer’s Note This class should remain DSDL agnostic. It is, theoretically, applicable using Jinja with any compiler front-end input although, in practice, it will only ever be used with pydsdl AST. Pydsdl-specific logic should live in the CodeGenerator (nunavut.jinja.DSDLCodeGenerator).

RESERVED_GLOBAL_NAMESPACES = {'ln', 'nunavut', 'options', 'uses_queries'}
RESERVED_GLOBAL_NAMES = {'now_utc'}
NUNAVUT_NAMESPACE_PREFIX = 'nunavut.lang.'
add_conventional_methods_to_environment(obj: Any) → None[source]
supported_languages
nunavut_global
target_language_uses_queries
language_options
language_support
target_language
now_utc
add_test(test_name: str, test_callable: Callable) → None[source]

nunavut.jinja.extensions

class nunavut.jinja.extensions.JinjaAssert(environment: nunavut.jinja.jinja2.environment.Environment)[source]

Bases: nunavut.jinja.jinja2.ext.Extension

Jinja2 extension that allows {% assert T.attribute %} statements. Templates should use these statements where False values would result in malformed source code :

template = '''{% assert False %}'''

This extension also support provding an assertion message:

template = '''{% assert (1 + 1 == 4) and (2 + 2 == 5), "this was truly false" %}'''
tags = {'assert'}
parse(parser: nunavut.jinja.jinja2.parser.Parser) → nunavut.jinja.jinja2.nodes.Node[source]

See http://jinja.pocoo.org/docs/2.10/extensions/ for help writing extensions.

identifier = 'nunavut.jinja.extensions.JinjaAssert'
class nunavut.jinja.extensions.UseQuery(environment: nunavut.jinja.jinja2.environment.Environment)[source]

Bases: nunavut.jinja.jinja2.ext.Extension

Jinja2 extension that allows conditional blocks like {% ifuses "std_variant" %} or {% ifnuses "std_variant" %}. These are defined by the nunavut.lang.Language object based on the values returned from nunavut.lang.Language.get_uses_queries().

template  = ''' {%- ifuses "some_language_key" -%}
                    #include "header 0"
                {%- elifuses "some_other_language_key" -%}
                    #include "header 1"
                {%- else -%}
                    #include "header 2"
                {%- endifuses -%}
            '''

For “not uses” replace all “uses” tokens with “nuses”:

template  = ''' {%- ifnuses "some_language_key" -%}
                    #include "header 1"
                {%- elifnuses "some_other_language_key" -%}
                    #include "header 0"
                {%- elifuses "yet_another_language_key" -%}
                    #include "header 2"
                {%- else -%}
                    #include "header 3"
                {%- endifnuses -%}
            '''
tags = {'ifnuses', 'ifuses'}
parse(parser: nunavut.jinja.jinja2.parser.Parser) → nunavut.jinja.jinja2.nodes.Node[source]

See http://jinja.pocoo.org/docs/2.10/extensions/ for help writing extensions.

identifier = 'nunavut.jinja.extensions.UseQuery'

nunavut.jinja.loaders

nunavut.jinja.loaders.TEMPLATE_SUFFIX = '.j2'

The suffix expected for Jinja templates.

class nunavut.jinja.loaders.DSDLTemplateLoader(templates_dirs: Optional[List[pathlib.Path]] = None, followlinks: bool = False, package_name_for_templates: Optional[str] = None, builtin_template_path: str = 'templates', **kwargs)[source]

Bases: nunavut.jinja.jinja2.loaders.BaseLoader

Nunavut’s DSDL template loader is similar to a choice loader with a file-system loader first and a package loader as a fallback. The major difference is a DFS is performed on the type hierarchy of the type a template is being loaded for. So, for example, if no StructureType.j2 template is found then this loader will look for a CompositeType.j2 and so on.

Parameters:
  • templates_dirs (Optional[List[Path]]) – A list of directories to load templates from using a nunavut.jinja.jinja2.FileSystemLoader. If None no filesystem loader is created.
  • followlinks (bool) – Argument passed on to the nunavut.jinja.jinja2.FileSystemLoader instance.
  • package_name_for_templates (Optional[str]) – The name of the package to load templates from. If None then no nunavut.jinja.jinja2.PackageLoader is created.
  • builtin_template_path (str) – The name of the package under the package_name_for_templates package to load templates from. This is ignored if package_name_for_templates is None.
  • kwargs (Any) – Arguments forwarded to the jinja.jinja2.BaseLoader.
get_source(environment: nunavut.jinja.jinja2.environment.Environment, template: str) → Tuple[Any, str, Callable[[...], bool]][source]

Get the template source, filename and reload helper for a template. It’s passed the environment and template name and has to return a tuple in the form (source, filename, uptodate) or raise a TemplateNotFound error if it can’t locate the template.

The source part of the returned tuple must be the source of the template as unicode string or a ASCII bytestring. The filename should be the name of the file on the filesystem if it was loaded from there, otherwise None. The filename is used by python for the tracebacks if no loader extension is used.

The last item in the tuple is the uptodate function. If auto reloading is enabled it’s always called to check if the template changed. No arguments are passed so the function must store the old state somewhere (for example in a closure). If it returns False the template will be reloaded.

list_templates() → Iterable[str][source]

Override of BaseLoader.list_templates() that returns an aggregate of the filesystem loader and package loader templates.

Returns:A list of templates names (i.e. file stems) found by this Generator object.
get_template_sets() → List[Tuple[str, str, Tuple[int, int, int]]][source]
get_templates() → Iterable[pathlib.Path][source]

Enumerate all templates found in the templates path. TEMPLATE_SUFFIX as the suffix for the filename. This method differs from the BaseLoader override of BaseLoader.list_templates() in that it returns paths instead of just file name stems.

Returns:A list of paths to all templates found by this Generator object.
type_to_template(value_type: Type[CT_co]) → Optional[pathlib.Path][source]

Given a type object, return a template used to generate code for the type.

Returns:a template or None if no template could be found for the given type.