Software Language Generation Guide¶
Note
This is a placeholder for documentation this project owes you, the user, for how to integrate nnvg with build systems and how to tune and optimize source code generation for each supported language.
C++ (experimental)¶
See Template Language Guide until this section is more complete.
Using a Different Variable-Length Array Type¶
For now this tip is important for people using the experimental C++ support. To use std::vector
instead of the
minimal build-in variable_length_array
type create a properties override yaml file and pass it to nnvg.
vector.yaml¶
nunavut.lang.cpp:
options:
variable_array_type_include: <vector>
variable_array_type_template: std::vector<{TYPE}>
nnvg command¶
nnvg --configuration=vector.yaml \
-l cpp \
--experimental-languages \
-I path/to/public_regulated_data_types/uavcan \
/path/to/my_types
Template Language Guide¶
For now we have only a jinja generator for code generation. As such this guide will only discuss using nunavut with jinja templates. There are no immediate plans to support any other template syntax.
Environment¶
Note
See nunavut.jinja
and the language support modules within this one for detailed
documentation on available filters and tests provided by nunavut.
The Jinja templates documentation is indispensible as nunavut embeds a full-featured version of jinja 2.
Each template has in its global environment the following:
T¶
The T global contains the type for the given template. For example:
{{T.full_name}}
{%- for attr in T.attributes %}
{{ attr.data_type }}
{%- endfor %}
now_utc¶
The time UTC as a Python datetime.datetime
object. This is the system time right before
the file generation step for the current template began. This will be the same time for included
templates and parent templates.
ln¶
Options and other language-specific facilities for all supported languages. This namespace provides access to other languages that are not the target language for the current template. This allows the development of mixed language templates.
Filters and Tests¶
In addition to the built-in Jinja filters and tests (again, see the
Jinja templates documentation for details)
pydsdl adds several global tests and filters to the template environment.
See nunavut.jinja
for full documentation on these. For example:
# typename filter returns the name of the value's type.
{{ field | typename }}
Also, for every pydsdl type there is a test automatically appended to the global environment. This means you can do:
{% if field is IntegerType %}
// stuff for integer fields
{% endif %}
Finally, for every pydsdl object that ends in “Type” or “Field” a lower-case name ommitting these suffixes is made available. For example:
template = '{% if some_type is integer %}This is a pydsdl.IntegerType{% endif %}'
Named Types¶
Some language provide named types to allow templates to use a type without making concrete decisions about the headers and conventions in use. For example, when using C it is common to use size_t as an unsigned, integer length type. To avoid hard-coding this type a C template can use the named type:
{{ typename_unsigned_length }} array_len;
Named Types by Language¶
Type name | Language(s) | Use |
---|---|---|
typename_unsigned_length | C, C++ | An unsigned integer type suitable for expressing the length of any valid type on the local system. |
typename_byte | C | An unsigned integer type used to represent a single byte (8-bits). |
Named Values¶
Some languages can use different values to represent certain data like null references or boolean values. Named values allow templates to insert a token appropriate for the language and configurable by the generator in use. For example:
MyType* p = {{ valuetoken_null }};
Named Values by Language¶
Value name | Language(s) |
---|---|
valuetoken_true | C |
valuetoken_false | C |
valuetoken_null | C |
Language Options¶
The target language for a template contributes options to the template globals. These options can be invented by users of the Nunavut library but a built-in set of defaults exists.
All language options are made available as globals within the options namespace. For example, a language option “target_arch” would be available as the “options.target_arch” global in templates.
For options that do not come with built-in defaults you’ll need to test if the option is available before you use it. For example:
# This will throw an exception
template = '{% if options.foo %}bar{% endif %}'
Use the built-in test defined
to avoid these exceptions:
# Avoid the exception
template = '{% if options.foo is defined and options.foo %}bar{% endif %}'
Language Options with Built-in Defaults¶
The following options have built-in defaults for certain languages. These options will always be defined in templates targeting their languages.
options.target_endianness¶
This option is currently defined for C and C++; the possible values are as follows:
any
— generate endianness-agnostic code that is compatible with big-endian and little-endian machines alike.big
— generate code optimized for big-endian platforms only. Implementations may treat this option likeany
when no such optimizations are possible.little
— generate code optimized for little-endian platforms only. Little-endian optimizations are made possible by the fact that DSDL is a little-endian format.
template = '{{ options.target_endianness }}'
# then
rendered = 'any'
Filters¶
Common Filters¶
-
nunavut.jinja.DSDLCodeGenerator.
filter_yamlfy
(value: Any) → str Filter to, optionally, emit a dump of the dsdl input as a yaml document. Available as
yamlfy
in all template environments.Example:
/* {{ T | yamlfy }} */
Result Example (truncated for brevity):
/* !!python/object:pydsdl.StructureType _attributes: - !!python/object:pydsdl.Field _serializable: !!python/object:pydsdl.UnsignedIntegerType _bit_length: 16 _cast_mode: &id001 !!python/object/apply:pydsdl.CastMode - 0 _name: value */
Parameters: value – The input value to parse as yaml. Returns: If a yaml parser is available, a pretty dump of the given value as yaml. If a yaml parser is not available then an empty string is returned.
-
nunavut.jinja.DSDLCodeGenerator.
filter_type_to_template
(self, value: Any) → str Template for type resolution as a filter. Available as
type_to_template
in all template environments.Example:
{%- for attribute in T.attributes %} {%* include attribute.data_type | type_to_template %} {%- if not loop.last %},{% endif %} {%- endfor %}
Parameters: value – The input value to change into a template include path. Returns: A path to a template named for the type with TEMPLATE_SUFFIX
-
nunavut.jinja.DSDLCodeGenerator.
filter_type_to_include_path
(self, value: Any, resolve: bool = False) → str Emits an include path to the output target for a given type.
Example:
# include "{{ T.my_type | type_to_include_path }}"
Result Example:
# include “foo/bar/my_type.h”Parameters: - value (typing.Any) – The type to emit an include for.
- resolve (bool) – If True the path returned will be absolute else the path will be relative to the folder of the root namespace.
Returns: A string path to output file for the type.
-
nunavut.jinja.DSDLCodeGenerator.
filter_typename
(value: Any) → str Filters a given token as its type name. Available as
typename
in all template environments.This example supposes that
T.some_value == "some string"
Example:
{{ T.some_value | typename }}
Result Example:
str
Parameters: value – The input value to filter into a type name. Returns: The __name__
of the python type.
-
nunavut.jinja.DSDLCodeGenerator.
filter_alignment_prefix
(offset: pydsdl._bit_length_set._bit_length_set.BitLengthSet) → str Provides a string prefix based on a given
pydsdl.BitLengthSet
.# Given B = pydsdl.BitLengthSet(32) # and template = '{{ B | alignment_prefix }}' # then ('str' is stropped to 'str_' before the version is suffixed) rendered = 'aligned'
# Given B = pydsdl.BitLengthSet(32) B += 1 # and template = '{{ B | alignment_prefix }}' # then ('str' is stropped to 'str_' before the version is suffixed) rendered = 'unaligned'
Parameters: offset (pydsdl.BitLengthSet) – A bit length set to test for alignment. Returns: ‘aligned’ or ‘unaligned’ based on the state of the offset
argument.
-
nunavut.jinja.DSDLCodeGenerator.
filter_bit_length_set
(values: Union[Iterable[int], int, None]) → pydsdl._bit_length_set._bit_length_set.BitLengthSet Convert an integer or a list of integers into a
pydsdl.BitLengthSet
.
-
nunavut.jinja.DSDLCodeGenerator.
filter_remove_blank_lines
(text: str) → str - Remove blank lines from the supplied string. Lines that contain only whitespace characters are also considered blank.
456
789’) == ‘123 456 789’
-
nunavut.jinja.DSDLCodeGenerator.
filter_bits2bytes_ceil
(n_bits: int) → int Implements
int(ceil(x/8)) | x >= 0
.
Common Tests¶
-
nunavut.jinja.DSDLCodeGenerator.
is_None
(value: Any) → bool Tests if a value is
None
-
nunavut.jinja.DSDLCodeGenerator.
is_saturated
(t: pydsdl._serializable._primitive.PrimitiveType) → bool Tests if a type is a saturated type or not.
-
nunavut.jinja.DSDLCodeGenerator.
is_service_request
(instance: pydsdl._expression._any.Any) → bool Tests if a type is request type of a service type.
-
nunavut.jinja.DSDLCodeGenerator.
is_service_response
(instance: pydsdl._expression._any.Any) → bool Tests if a type is response type of a service type.
-
nunavut.jinja.DSDLCodeGenerator.
is_deprecated
(instance: pydsdl._expression._any.Any) → bool Tests if a type is marked as deprecated
C Filters¶
-
nunavut.lang.c.
filter_id
(language: nunavut.lang.c.Language, instance: Any, id_type: str = 'any') → str[source] Filter that produces a valid C identifier for a given object. The encoding may not be reversible.
# Given I = 'I ❤ c' # and template = '{{ I | id }}' # then rendered = 'I_zX2764_c'
# Given I = 'if' # and template = '{{ I | id }}' # then rendered = '_if'
# Given I = '_Reserved' # and template = '{{ I | id }}' # then rendered = '_reserved'
# Given I = 'EMACRO_TOKEN' # and template = '{{ I | id("macro") }}' # then rendered = '_eMACRO_TOKEN'
Parameters: - instance (any) – Any object or data that either has a name property or can be converted to a string.
- id_type (str) – A type of identifier. For C this value can be ‘typedef’, ‘macro’, ‘function’, or ‘enum’. use ‘any’ to apply stropping rules for all identifier types to the instance.
Returns: A token that is a valid identifier for C, is not a reserved keyword, and is transformed in a deterministic manner based on the provided instance.
-
nunavut.lang.c.
filter_macrofy
(language: nunavut.lang.c.Language, value: str) → str[source] Filter to transform an input into a valid C preprocessor identifier token.
# Given template = '#ifndef {{ "my full name" | macrofy }}' # then rendered = '#ifndef MY_FULL_NAME'
Note that individual tokens are not stropped so the appearance of an identifier in the
SCREAMING_SNAKE_CASE
output my be different then the token as it appears on its own. For example:# "register" is reserved so it will be stropped if it appears as an # identifier... template = '''#ifndef {{ "namespaced.Type.register" | macrofy }} {{ "register" | id }} ''' # ...but it will not be stropped within the macro. rendered = '''#ifndef NAMESPACED_TYPE_REGISTER _register '''
If stropping is enabled, however, the entire token generated by this filter will be stropped:
# Given template = '#ifndef {{ "_starts_with_underscore" | macrofy }}' # then rendered = '#ifndef _sTARTS_WITH_UNDERSCORE'
And again with stropping disabled:
# Given template = '#ifndef {{ "_starts_with_underscore" | macrofy }}' # then with stropping disabled rendered = '#ifndef _STARTS_WITH_UNDERSCORE'
Parameters: value (str) – The value to transform. Returns: A valid C preprocessor identifier token.
-
nunavut.lang.c.
filter_type_from_primitive
(language: nunavut.lang.c.Language, value: pydsdl._serializable._primitive.PrimitiveType) → str[source] Filter to transform a pydsdl
PrimitiveType
into a valid C type.# Given template = '{{ unsigned_int_32_type | type_from_primitive }}' # then rendered = 'uint32_t'
# Given template = '{{ int_64_type | type_from_primitive }}' # then rendered = 'int64_t'
Parameters: value (str) – The dsdl primitive to transform. Returns: A valid C99 type name. Raises: RuntimeError – If the primitive cannot be represented as a standard C type.
-
nunavut.lang.c.
filter_to_snake_case
(value: str) → str[source] Filter to transform a string into a snake-case token.
# Given template = '{{ "scotec.mcu.Timer" | to_snake_case }} a();' # then rendered = 'scotec_mcu_timer a();'
# Given template = '{{ "scotec.mcu.TimerHelper" | to_snake_case }} b();' # then rendered = 'scotec_mcu_timer_helper b();'
# and Given template = '{{ "SCOTEC_MCU_TimerHelper" | to_snake_case }} b();' # then rendered = 'scotec_mcu_timer_helper b();'
Parameters: value (str) – The string to transform into C snake-case. Returns: A valid C99 token using the snake-case convention.
-
nunavut.lang.c.
filter_to_screaming_snake_case
(value: str) → str[source] Filter to transform a string into a SCREAMING_SNAKE_CASE token.
# Given template = '{{ "scotec.mcu.Timer" | to_screaming_snake_case }} a();' # then rendered = 'SCOTEC_MCU_TIMER a();'
-
nunavut.lang.c.
filter_to_template_unique_name
(_: Any, base_token: str) → str[source] Filter that takes a base token and forms a name that is very likely to be unique within the template the filter is invoked. This name is also very likely to be a valid C identifier.
Important
The exact tokens generated may change between major or minor versions of this library. The only guarantee provided is that the tokens will be stable for the same version of this library given the same input.
Also note that name uniqueness is only likely within a given template. Between templates there is no guarantee of uniqueness and, since this library does not lex generated source, there is no guarantee that the generated name does not conflict with a name generated by another means.
# Given template = '{{ "foo" | to_template_unique_name }},{{ "Foo" | to_template_unique_name }},' template += '{{ "fOO" | to_template_unique_name }}' # then rendered = '_foo0_,_foo1_,_fOO0_'
# Given template = '{{ "i like coffee" | to_template_unique_name }}' # then rendered = '_i like coffee0_'
Parameters: base_token (str) – A token to include in the base name. Returns: A name that is likely to be valid C identifier and is likely to be unique within the file generated by the current template.
-
nunavut.lang.c.
filter_short_reference_name
(language: nunavut.lang.c.Language, t: pydsdl._serializable._composite.CompositeType) → str[source] Provides a string that is a shorted version of the full reference name.
# Given a type with illegal C characters my_type.short_name = '_Foo' my_type.version.major = 1 my_type.version.minor = 2 # and template = '{{ my_type | short_reference_name }}' # then, with stropping enabled rendered = '_foo_1_2'
With stropping disabled:
rendered = '_Foo_1_2'
Parameters: t (pydsdl.CompositeType) – The DSDL type to get the reference name for.
-
nunavut.lang.c.
filter_includes
(language: nunavut.lang.c.Language, env: nunavut.jinja.jinja2.environment.Environment, t: pydsdl._serializable._composite.CompositeType, sort: bool = True) → List[str][source] Returns a list of all include paths for a given type.
# Listing the includes for a union with only integer types: template = '''{% for include in my_type | includes -%} {{include}} {% endfor %}''' # stdint.h will normally be generated rendered = '''<stdint.h> <stdlib.h> '''
# You can suppress std includes by setting use_standard_types to False under # nunavut.lang.c rendered = ''
Parameters: - t (pydsdl.CompositeType) – The type to scan for dependencies.
- sort (bool) – If true the returned list will be sorted.
Returns: a list of include headers needed for a given type.
-
nunavut.lang.c.
filter_to_static_assertion_value
(obj: Any) → int[source] Tries to convert a Python object into a value compatible with static comparisons in C. This allows stable comparison of static values in headers to promote consistency and version compatibility in generated code.
Will raise a ValueError if the object provided does not (yet) have an available conversion in this function.
Currently supported types are string:
# given template = '{{ "Any" | to_static_assertion_value }}' # then rendered = '1556001108'
int:
# given template = '{{ 123 | to_static_assertion_value }}' # then rendered = '123'
and bool:
# given template = '{{ True | to_static_assertion_value }}' # then rendered = '1'
-
nunavut.lang.c.
filter_constant_value
(language: nunavut.lang.c.Language, constant: pydsdl._serializable._attribute.Constant) → str[source] Renders the specified constant as a literal. This is a shorthand for
filter_literal()
.# given template = '{{ my_true_constant | constant_value }}' # then rendered = 'true'
Language configuration can control the output of some constant tokens. For example, to use non-standard true and false values in c:
# given template = '{{ my_true_constant | constant_value }}' # then, if true = 'NUNAVUT_TRUE' in the named_values for nunavut.lang.c rendered = 'NUNAVUT_TRUE'
Floating point values are converted as fractions to ensure no python-specific transformations are applied:
# given a float value using a fraction of 355/113 template = '{{ almost_pi | constant_value }}' # ...the rendered value with include that fraction as a division statement. rendered = '((float) (355.0 / 113.0))'
-
nunavut.lang.c.
filter_literal
(language: nunavut.lang.c.Language, value: Union[fractions.Fraction, bool, int], ty: pydsdl._expression._any.Any, cast_format: Optional[str] = None) → str[source] Renders the specified value of the specified type as a literal.
-
nunavut.lang.c.
filter_full_reference_name
(language: nunavut.lang.c.Language, t: pydsdl._serializable._composite.CompositeType) → str[source] Provides a string that is the full namespace, typename, major, and minor version for a given composite type.
# Given a type with illegal characters for C++ my_obj.full_name = 'any.int.2Foo' my_obj.full_namespace = 'any.int' my_obj.version.major = 1 my_obj.version.minor = 2 # and template = '{{ my_obj | full_reference_name }}' # then, with stropping enabled rendered = 'any_int_2Foo_1_2'
Parameters: t (pydsdl.CompositeType) – The DSDL type to get the fully-resolved reference name for.
-
nunavut.lang.c.
filter_to_standard_bit_length
(t: pydsdl._serializable._primitive.PrimitiveType) → int[source] Returns the nearest standard bit length of a type as an int.
# Given I = pydsdl.UnsignedIntegerType(7, pydsdl.PrimitiveType.CastMode.TRUNCATED) # and template = '{{ I | to_standard_bit_length }}' # then rendered = '8'
C Tests¶
-
nunavut.lang.c.
is_zero_cost_primitive
(language: nunavut.lang.c.Language, t: pydsdl._serializable._primitive.PrimitiveType) → bool[source] Assuming that the target platform is IEEE754-conformant detects whether the native in-memory representation of a value of the supplied primitive type is the same as its on-the-wire representation defined by the DSDL Specification.
For instance; all little-endian, IEEE754-conformant platforms have compatible in-memory representations of int8, int16, int32, int64, uint8, uint16, uint32, uint64, float32, float64. Values of other primitive types typically require some transformations (e.g., float16).
It follows that arrays, certain composite types, and some other entities composed of zero-cost composites are also zero-cost types, but such non-trivial conjectures are not recognized by this function.
Raises a
TypeError
if the argument is not a value of typepydsdl.PrimitiveType
.# Given i7 = pydsdl.SignedIntegerType(7, pydsdl.PrimitiveType.CastMode.SATURATED) u32 = pydsdl.UnsignedIntegerType(32, pydsdl.PrimitiveType.CastMode.TRUNCATED) f16 = pydsdl.FloatType(16, pydsdl.PrimitiveType.CastMode.TRUNCATED) f32 = pydsdl.FloatType(32, pydsdl.PrimitiveType.CastMode.SATURATED) bl = pydsdl.BooleanType(pydsdl.PrimitiveType.CastMode.SATURATED) # and template = ( '{{ i7 is zero_cost_primitive }} ' '{{ u32 is zero_cost_primitive }} ' '{{ f16 is zero_cost_primitive }} ' '{{ f32 is zero_cost_primitive }} ' '{{ bl is zero_cost_primitive }}' ) # then rendered = 'False True False True False'
C++ Filters¶
-
nunavut.lang.cpp.
filter_id
(language: nunavut.lang.cpp.Language, instance: Any, id_type: str = 'any') → str[source] Filter that produces a valid C and/or C++ identifier for a given object. The encoding may not be reversible.
# Given I = 'I like c++' # and template = '{{ I | id }}' # then rendered = 'I_like_czX002BzX002B'
# Given I = 'if' # and template = '{{ I | id }}' # then rendered = '_if'
# Given I = 'I really like coffee' # and template = '{{ I | id }}' # then rendered = 'I_really_like_coffee'
Parameters: instance (any) – Any object or data that either has a name property or can be converted to a string. Returns: A token that is a valid identifier for C and C++, is not a reserved keyword, and is transformed in a deterministic manner based on the provided instance.
-
nunavut.lang.cpp.
filter_open_namespace
(language: nunavut.lang.cpp.Language, full_namespace: str, bracket_on_next_line: bool = True, linesep: str = '\n') → str[source] Emits c++ opening namespace syntax parsed from a pydsdl “full_namespace”, dot-separated value.
# Given T.full_namespace = 'uavcan.foo' # and template = '{{ T.full_namespace | open_namespace }}' # then rendered = '''namespace uavcan { namespace foo {'''
Parameters: Returns: C++ namespace declarations with opening brackets.
-
nunavut.lang.cpp.
filter_close_namespace
(language: nunavut.lang.cpp.Language, full_namespace: str, omit_comments: bool = False, linesep: str = '\n') → str[source] Emits c++ closing namespace syntax parsed from a pydsdl “full_namespace”, dot-separated value.
# Given T.full_namespace = 'uavcan.foo' # and template = '{{ T.full_namespace | close_namespace }}' # then rendered = '''} // namespace foo } // namespace uavcan'''
Parameters: Returns: C++ namespace declarations with opening brackets.
-
nunavut.lang.cpp.
filter_full_reference_name
(language: nunavut.lang.cpp.Language, t: pydsdl._serializable._composite.CompositeType) → str[source] Provides a string that is the full namespace, typename, major, and minor version for a given composite type.
# Given a type with illegal characters for C++ my_obj.full_name = 'any.int.2Foo' my_obj.version.major = 1 my_obj.version.minor = 2 # and template = '{{ my_obj | full_reference_name }}' # then, with stropping enabled rendered = 'any::_int::_2Foo_1_2'
Parameters: t (pydsdl.CompositeType) – The DSDL type to get the fully-resolved reference name for.
-
nunavut.lang.cpp.
filter_short_reference_name
(language: nunavut.lang.cpp.Language, t: pydsdl._serializable._composite.CompositeType) → str[source] Provides a string that is a shorted version of the full reference name. This type is unique only within its namespace.
# Given a type with illegal C++ characters my_type.short_name = '2Foo' my_type.version.major = 1 my_type.version.minor = 2 # and template = '{{ my_type | short_reference_name }}' # then, with stropping enabled rendered = '_2Foo_1_2'
# Given a type with legal C++ characters my_type.short_name = 'Struct_' my_type.version.major = 0 my_type.version.minor = 1 # and template = '{{ my_type | short_reference_name }}' # then, with stropping enabled rendered = 'Struct__0_1'
# Given a service type my_service_type.short_name = 'Struct_' my_service_type.version.major = 0 my_service_type.version.minor = 1 # and template = ''' {{ my_service_type | short_reference_name }} {{ my_service_type.request_type | short_reference_name }} {{ my_service_type.response_type | short_reference_name }} ''' # then, with stropping enabled rendered = ''' Struct_ Request_0_1 Response_0_1 '''
Parameters: t (pydsdl.CompositeType) – The DSDL type to get the reference name for.
-
nunavut.lang.cpp.
filter_includes
(language: nunavut.lang.cpp.Language, env: nunavut.jinja.jinja2.environment.Environment, t: pydsdl._serializable._composite.CompositeType, sort: bool = True) → List[str][source] Returns a list of all include paths for a given type.
Parameters: - t (pydsdl.CompositeType) – The type to scan for dependencies.
- sort (bool) – If true the returned list will be sorted.
Returns: a list of include headers needed for a given type.
# Listing the includes for a union with only integer types: template = "{% for include in my_type | includes -%}{{include}}{%- endfor %}" # cstdint will normally be generated. limits is always generated. rendered = "<cstdint><limits>"
# You can suppress std includes by setting use_standard_types to False under # nunavut.lang.cpp rendered = "<limits>"
-
nunavut.lang.cpp.
filter_destructor_name
(language: nunavut.lang.cpp.Language, instance: pydsdl._expression._any.Any) → str[source] Returns a token that is the local destructor name. For example:
# Given a pydsdl.FixedLengthArrayType "my_type": my_type.short_name = 'Foo' my_type.capacity = 2 # and template = 'ptr->{{ my_type | destructor_name }}' # then rendered = 'ptr->~array<std::uint8_t,2>'
Parameters: t (pydsdl.CompositeType) – The type to generate a destructor template for. Returns: A destructor name token.
-
nunavut.lang.cpp.
filter_declaration
(language: nunavut.lang.cpp.Language, instance: pydsdl._expression._any.Any) → str[source] Emit a declaration statement for the given instance.
-
nunavut.lang.cpp.
filter_to_namespace_qualifier
(namespace_list: List[str]) → str[source] Converts a list of namespace names into a qualifier string. For example:
my_namespace = ['foo', 'bar'] template = '{{ my_namespace | to_namespace_qualifier }}myType()' expected = 'foo::bar::myType()'
This filter gracefully handles empty namespace lists:
my_namespace = [] template = '{{ my_namespace | to_namespace_qualifier }}myType()' expected = 'myType()'
-
nunavut.lang.cpp.
filter_type_from_primitive
(language: nunavut.lang.cpp.Language, value: pydsdl._serializable._primitive.PrimitiveType) → str[source] Filter to transform a pydsdl
PrimitiveType
into a valid C++ type.# Given template = '{{ unsigned_int_32_type | type_from_primitive }}' # then rendered = 'std::uint32_t'
Also note that this is sensitive to the
use_standard_types
configuration in the language properties:# rendered will be different if use_standard_types is False rendered = 'unsigned long'
Parameters: value (str) – The dsdl primitive to transform. Returns: A valid C++ type name. Raises: TemplateRuntimeError – If the primitive cannot be represented as a standard C++ type.
-
nunavut.lang.cpp.
filter_to_template_unique_name
(base_token: str) → str[source] Filter that takes a base token and forms a name that is very likely to be unique within the template the filter is invoked. This name is also very likely to be a valid C++ identifier.
Important
The exact tokens generated may change between major or minor versions of this library. The only guarantee provided is that the tokens will be stable for the same version of this library given the same input.
Also note that name uniqueness is only likely within a given template. Between templates there is no guarantee of uniqueness and, since this library does not lex generated source, there is no guarantee that the generated name does not conflict with a name generated by another means.
# Given template = '{{ "foo" | to_template_unique_name }},{{ "Foo" | to_template_unique_name }},' template += '{{ "fOO" | to_template_unique_name }}' # then rendered = '_foo0_,_foo1_,_fOO0_'
# Given template = '{{ "i like coffee" | to_template_unique_name }}' # then rendered = '_i like coffee0_'
Parameters: base_token (str) – A token to include in the base name. Returns: A name that is likely to be valid C++ identifier and is likely to be unique within the file generated by the current template.
-
nunavut.lang.cpp.
filter_as_boolean_value
(value: bool) → str[source] Filter a boolean expression to produce a valid C++ “true” or “false” token.
assert "true" == filter_as_boolean_value(True) assert "false" == filter_as_boolean_value(False)
-
nunavut.lang.cpp.
filter_indent_if_not
(language: nunavut.lang.cpp.Language, text: str, depth: int = 1) → str[source] Emit indent characters as configured for the language but only as needed. This is different from the built-in indent filter in that it may add or remove spaces based on the existing indent.
# Given a string with an existing indent of 4 spaces... template = '{{ " int a = 1;" | indent_if_not }}' # then if cpp.indent == 4 we expect no change. rendered = ' int a = 1;'
# If the indent is only 3 spaces... template = '{{ " int a = 1;" | indent_if_not }}' # then if cpp.indent == 4 we expect 4 spaces (i.e. not 7 spaces) rendered = ' int a = 1;'
# We can also specify multiple indents... template = '{{ "int a = 1;" | indent_if_not(2) }}' rendered = ' int a = 1;'
# ...or no indent template = '{{ " int a = 1;" | indent_if_not(0) }}' rendered = 'int a = 1;'
# Finally, note that blank lines are not indented. template = ''' {%- set block_text -%} int a = 1; {# empty line #} int b = 2; {%- endset -%}{{ block_text | indent_if_not(1) }}''' rendered = ' int a = 1;' rendered += '\n' rendered += '\n' # Nothing but spaces so this is stripped rendered += ' int b = 2;'
Parameters: - text – The text to indent.
- depth – The number of indents. For example, if depth is 2 and the indent for this language is 4 spaces then the text will be indented by 8 spaces.
-
nunavut.lang.cpp.
filter_minimum_required_capacity_bits
(t: pydsdl._serializable._serializable.SerializableType) → int[source] Returns the minimum number of bits required to store the deserialized value of a pydsdl
SerializableType
. This capacity may be too small for some instances of the value (e.g. variable length arrays).# Given template = '{{ unsigned_int_32_type | minimum_required_capacity_bits }}' # then rendered = '32'
Parameters: t (pydsdl.SerializableType) – The dsdl type. Returns: The minimum, required bits needed to store some values of the given type.
-
nunavut.lang.cpp.
filter_block_comment
(language: nunavut.lang.cpp.Language, text: str, style: str, indent: int = 0, line_length: int = 100) → str[source] Reformats text as a block comment using Python’s
textwrap.TextWrapper.wrap()
function.Parameters: - text – The text to emit as a block comment.
- style – Dictates the style of comments (see return documentation for valid style names).
- indent – The number of spaces to indent the comments by (tab indent is not supported. Sorry).
- line_length – The soft maximum width to wrap text at. Some violations may occur where long words are used.
Returns str: A comment block. Comment styles supported are:
javadoc
# Given a type with the following docstring text = 'This is a bunch of documentation.' # and template = ''' {{ text | block_comment('javadoc', 4, 24) }} void some_method(); ''' # the output will be rendered = ''' /** * This is a bunch * of documentation. */ void some_method(); '''
cpp-doxygen
# that same template using the cpp style of doxygen... template = ''' {{ text | block_comment('cpp-doxygen', 4, 24) }} void some_method(); ''' # ...will be rendered = ''' /// /// This is a bunch /// of /// documentation. /// void some_method(); '''
cpp
# also supported is cpp style... template = ''' {{ text | block_comment('cpp', 4, 24) }} void some_method(); ''' rendered = ''' // This is a bunch of // documentation. void some_method(); '''
c
# c style... template = ''' {{ text | block_comment('c', 4, 24) }} void some_method(); ''' rendered = ''' /* * This is a bunch * of documentation. */ void some_method(); '''
qt
# and Qt style... template = ''' {{ text | block_comment('qt', 4, 24) }} void some_method(); ''' rendered = ''' /*! * This is a bunch * of documentation. */ void some_method(); '''
C++ Use Queries¶
-
nunavut.lang.cpp.
uses_std_variant
(language: nunavut.lang.cpp.Language) → bool[source] Uses query for std variant.
If the language options contain an
std
entry for C++ and the specified standard includes thestd::variant
type added to the language at C++17 then this value is true. The logic included in this filter can be stated as “options has key std and the value for options.std evaluates to C++ version 17 or greater” but the implementation is able to parse out actual compiler flags likegnu++20
and is aware of any overrides to suppress use of the standard variant type even if available.Example:
template = ''' {%- ifuses "std_variant" -%} #include <variant> {%- else -%} #include "user_variant.h" {%- endifuses -%} '''
Python Filters¶
-
nunavut.lang.py.
filter_id
(language: nunavut.lang.py.Language, instance: Any, id_type: str = 'any') → str[source] Filter that produces a valid Python identifier for a given object. The encoding may not be reversible.
# Given I = 'I like python' # and template = '{{ I | id }}' # then rendered = 'I_like_python'
# Given I = '&because' # and template = '{{ I | id }}' # then rendered = 'zX0026because'
# Given I = 'if' # and template = '{{ I | id }}' # then rendered = 'if_'
Parameters: instance (any) – Any object or data that either has a name property or can be converted to a string. Returns: A token that is a valid Python identifier, is not a reserved keyword, and is transformed in a deterministic manner based on the provided instance.
-
nunavut.lang.py.
filter_to_template_unique_name
(context: nunavut._templates.SupportsTemplateContext, base_token: str) → str[source] Filter that takes a base token and forms a name that is very likely to be unique within the template the filter is invoked. This name is also very likely to be a valid Python identifier.
Important
The exact tokens generated may change between major or minor versions of this library. The only guarantee provided is that the tokens will be stable for the same version of this library given the same input.
Also note that name uniqueness is only likely within a given template. Between templates there is no guarantee of uniqueness and, since this library does not lex generated source, there is no guarantee that the generated name does not conflict with a name generated by another means.
# Given template = '{{ "f" | to_template_unique_name }},{{ "f" | to_template_unique_name }},' template += '{{ "f" | to_template_unique_name }},{{ "bar" | to_template_unique_name }}' # then rendered = '_f0_,_f1_,_f2_,_bar0_'
# Given template = '{{ "i like coffee" | to_template_unique_name }}' # then rendered = '_i like coffee0_'
Parameters: base_token (str) – A token to include in the base name. Returns: A name that is likely to be valid python identifier and is likely to be unique within the file generated by the current template.
-
nunavut.lang.py.
filter_full_reference_name
(language: nunavut.lang.py.Language, t: pydsdl._serializable._composite.CompositeType) → str[source] Provides a string that is the full namespace, typename, major, and minor version for a given composite type.
# Given full_name = 'any.str.2Foo' major = 1 minor = 2 # and template = '{{ my_obj | full_reference_name }}' # then rendered = 'any_.str_.zX0032Foo_1_2'
Parameters: t (pydsdl.CompositeType) – The DSDL type to get the fully-resolved reference name for.
-
nunavut.lang.py.
filter_short_reference_name
(language: nunavut.lang.py.Language, t: pydsdl._serializable._composite.CompositeType) → str[source] Provides a string that is a shorted version of the full reference name. This type is unique only within its namespace.
# Given short_name = '2Foo' major = 1 minor = 2 # and template = '{{ my_obj | short_reference_name }}' # then rendered = 'zX0032Foo_1_2'
Parameters: t (pydsdl.CompositeType) – The DSDL type to get the reference name for.
-
nunavut.lang.py.
filter_imports
(language: nunavut.lang.py.Language, t: pydsdl._serializable._composite.CompositeType, sort: bool = True) → List[str][source] Returns a list of all modules that must be imported to use a given type.
Parameters: - t (pydsdl.CompositeType) – The type to scan for dependencies.
- sort (bool) – If true the returned list will be sorted.
Returns: a list of python module names the provided type depends on.
-
nunavut.lang.py.
filter_longest_id_length
(language: nunavut.lang.py.Language, attributes: List[pydsdl._serializable._attribute.Attribute]) → int[source] Return the length of the longest identifier in a list of
pydsdl.Attribute
objects.# Given I = ['one.str.int.any', 'three.str.int.any'] # and template = '{{ I | longest_id_length }}' # then rendered = '32'
HTML Filters¶
-
nunavut.lang.html.
filter_extent
(instance: pydsdl._expression._any.Any) → int[source]
-
nunavut.lang.html.
filter_max_bit_length
(instance: pydsdl._expression._any.Any) → int[source]
-
nunavut.lang.html.
filter_tag_id
(instance: pydsdl._expression._any.Any) → str[source]
-
nunavut.lang.html.
filter_url_from_type
(instance: pydsdl._expression._any.Any) → str[source]
-
nunavut.lang.html.
filter_make_unique
(_: Any, base_token: str) → str[source] Filter that takes a base token and forms a name that is very likely to be unique within the template the filter is invoked.
Important
The exact tokens generated may change between major or minor versions of this library. The only guarantee provided is that the tokens will be stable for the same version of this library given the same input.
Also note that name uniqueness is only likely within a given template. Between templates there is no guarantee of uniqueness and, since this library does not lex generated source, there is no guarantee that the generated name does not conflict with a name generated by another means.
# Given template = '{{ "foo" | make_unique }},{{ "Foo" | make_unique }},' template += '{{ "fOO" | make_unique }}' # then rendered = 'foo0,foo1,fOO0'
# Given template = '{{ "coffee > tea" | make_unique }}' # then rendered = 'coffee > tea0'
Parameters: base_token (str) – A token to include in the base name. Returns: A name that is likely to be unique within the file generated by the current template.
-
nunavut.lang.html.
filter_namespace_doc
(ns: nunavut._namespace.Namespace) → str[source]
-
nunavut.lang.html.
filter_display_type
(instance: pydsdl._expression._any.Any) → str[source]
-
nunavut.lang.html.
filter_natural_sort_namespace
(instance: List[pydsdl._expression._any.Any]) → List[pydsdl._expression._any.Any][source] Namespaces come in plain lists; sort by name only.
-
nunavut.lang.html.
filter_natural_sort_type
(instance: pydsdl._expression._any.Any) → List[pydsdl._expression._any.Any][source] Types come in tuples (type, path). Sort by type name.
Template Mapping and Use¶
Templates are resolved as templates/path/[dsdl_typename].j2
This means you must, typically, start with four templates under the templates directory
given to the Generator
instance
ServiceType.j2
StructureType.j2
DelimitedType.j2
UnionType.j2
Note
You can chose to use a single template Any.j2
but this may lead to more complex
templates with many more control statements. By providing discreet templates named for top-level
data types and using jinja template inheritance and includes your templates will be smaller
and easier to maintain.
To share common formatting for these templates use Jinja template inheritance. For example,
given a template common_header.j2
:
/*
* Cyphal data structure definition for nunavut.
*
* Auto-generated, do not edit.
*
* Source file: {{T.source_file_path.as_posix()}}
* Generated at: {{now_utc}}
* Template: {{ self._TemplateReference__context.name }}
* deprecated: {{T.deprecated}}
* fixed_port_id: {{T.fixed_port_id}}
* full_name: {{T.full_name}}
* full_namespace: {{T.full_namespace}}
*/
#ifndef {{T.full_name | ln.c.macrofy}}
#define {{T.full_name | ln.c.macrofy}}
{%- block contents %}{% endblock -%}
#endif // {{T.full_name | ln.c.macrofy}}
/*
{{ T | yamlfy }}
*/
… your three top-level templates would each start out with something like this:
{% extends "common_header.j2" %}
{% block contents %}
// generate stuff here
{% endblock %}
Resolving Types to Templates¶
You can apply the same logic used by the top level generator to recursively include templates
by type if this seems useful for your project. Simply use the
nunavut.jinja.Generator.filter_type_to_template()
filter:
{%- for attribute in T.attributes %}
{%* include attribute.data_type | type_to_template %}
{%- endfor %}
Namespace Templates¶
If the generate_namespace_types
parameter of Generator
is
YES
then the generator will always invoke a template for the root namespace and all
nested namespaces regardless of language. NO
suppresses this behavior and DEFAULT
will choose the behavior based on the target language. For example:
root_namespace = build_namespace_tree(compound_types,
root_ns_folder,
out_dir,
language_context)
generator = Generator(root_namespace, YesNoDefault.DEFAULT)
Would generate python __init__.py
files to define each namespace as a python module but
would not generate any additional headers for C++.
The Generator
will use the same template name resolution logic as used
for pydsdl data types. For namespaces this will resolve first to a template named
Namespace.j2
and then, if not found, Any.j2
.
Internals¶
Nunavut reserves all global identifiers that start with _nv_ as private internal globals.
Built-in Template Guide¶
This section will contain more information as the project matures about the build-in language support for generating code. Nunavut is both a framework that allows users to write their own dsdl transformation templates but also works, out-of-the-box, as a transplier for C, and C++. More languages may be added in the future.
C++¶
Note
C++ support is currently experimental. You can only use this by setting the --experimental-languages
flag
when invoking nnvg.
C¶
Note
Documentation is provided in the generated source.
Manual Override of Array Capacity¶
By default, the C structures generated will utilize C arrays sized by the maximum size of a
variable-length array. To override this behavior you can pre-define the _ARRAY_CAPACITY_
for individual
fields. For example:
#include <stdio.h>
#define reg_drone_service_battery_Status_0_2_cell_voltages_ARRAY_CAPACITY_ 6
#include "inc/UAVCAN/reg/drone/service/battery/Status_0_2.h"
int main(int argc, char *argv[])
{
reg_drone_service_battery_Status_0_2 msg;
printf("Size of reg_drone_service_battery_Status_0_2 %li\n", sizeof(msg));
}
nnvg¶
Usage¶
Generate code from Cyphal DSDL using pydsdl and jinja2
usage: nnvg [-h] [--lookup-dir LOOKUP_DIR] [--verbose] [--version]
[--outdir OUTDIR] [--templates TEMPLATES]
[--target-language TARGET_LANGUAGE] [--experimental-languages]
[--output-extension OUTPUT_EXTENSION] [--dry-run] [--list-outputs]
[--generate-support {always,never,as-needed,only}] [--list-inputs]
[--generate-namespace-types] [--omit-serialization-support]
[--namespace-output-stem NAMESPACE_OUTPUT_STEM] [--no-overwrite]
[--file-mode FILE_MODE] [--trim-blocks] [--lstrip-blocks]
[--allow-unregulated-fixed-port-id]
[--pp-max-emptylines PP_MAX_EMPTYLINES]
[--pp-trim-trailing-whitespace] [-pp-rp PP_RUN_PROGRAM]
[-pp-rpa PP_RUN_PROGRAM_ARG]
[--target-endianness {any,big,little}]
[--omit-float-serialization-support]
[--enable-serialization-asserts]
[--enable-override-variable-array-capacity]
[--language-standard LANGUAGE_STANDARD]
[--configuration [CONFIGURATION [CONFIGURATION ...]]]
[--list-configuration]
[root_namespace]
Positional Arguments¶
root_namespace | A source directory with DSDL definitions. Default: “.” |
Named Arguments¶
--lookup-dir, -I | |
List of other namespace directories containing data type definitions that are referred to from the target root namespace. For example, if you are reading a vendor-specific namespace, the list of lookup directories should always include a path to the standard root namespace “uavcan”, otherwise the types defined in the vendor-specific namespace won’t be able to use data types from the standard namespace. Additional directories can also be specified through an environment variable DSDL_INCLUDE_PATH where the path entries are separated by colons “:” on posix systems and “;” on Windows. Default: [] | |
--verbose, -v | verbosity level (-v, -vv) |
--version | show program’s version number and exit |
--outdir, -O | output directory Default: “nunavut_out” |
--templates | Paths to a directory containing templates to use when generating code. Templates found under these paths will override the built-in templates for a given language. |
--target-language, -l | |
Language support to install into the templates. If provided then the output extension (–e) can be inferred otherwise the output extension must be provided. | |
--experimental-languages, -Xlang | |
Activate languages with unstable, experimental support. By default, target languages where support is not finalized are not enabled when running nunavut, to make it clear that the code output may change in a non-backwards-compatible way in future versions, or that it might not even work yet. Default: False | |
--output-extension, -e | |
The extension to use for generated files. | |
--dry-run, -d | If True then no files will be generated. Default: False |
--list-outputs | Emit a semicolon-separated list of files. (implies –dry-run) Emits files that would be generated if invoked without –dry-run. This command is useful for integrating with CMake and other build systems that need a list of targets to determine if a rebuild is necessary. Default: False |
--generate-support | |
Possible choices: always, never, as-needed, only Change the criteria used to enable or disable support code generation. as-needed (default) - generate support code if serialization is enabled. always - always generate support code. never - never generate support code. only - only generate support code. Default: “as-needed” | |
--list-inputs | Emit a semicolon-separated list of files. (implies –dry-run) A list of files that are resolved given input arguments like templates. This command is useful for integrating with CMake and other build systems that need a list of inputs to determine if a rebuild is necessary. Default: False |
--generate-namespace-types | |
If enabled this script will generate source for namespaces. All namespaces including and under the root namespace will be treated as a pseudo-type and the appropriate template will be used. The generator will first look for a template with the stem “Namespace” and will then use the “Any” template if that is available. The name of the output file will be the default value for the –namespace-output-stem argument and can be changed using that argument. Default: False | |
--omit-serialization-support, -pod | |
If provided then the types generated will be POD datatypes with no additional logic. By default types generated include serialization routines and additional support libraries, headers, or methods. Default: False | |
--namespace-output-stem | |
The name of the file generated when –generate-namespace-types is provided. | |
--no-overwrite | By default, generated files will be silently overwritten by subsequent invocations of the generator. If this argument is specified an error will be raised instead preventing overwrites. Default: False |
--file-mode | The file-mode each generated file is set to after it is created. Note that this value is interpreted using python auto base detection. Because of this, to provide an octal value, you’ll need to prefix your literal with ‘0o’ (e.g. –file-mode 0o664). Default: 292 |
--trim-blocks | If this is set to True the first newline after a block in a template is removed (block, not variable tag!). Default: False |
--lstrip-blocks | |
If this is set to True leading spaces and tabs are stripped from the start of a line to a block in templates. Default: False | |
--allow-unregulated-fixed-port-id | |
Do not reject unregulated fixed port identifiers. This is a dangerous feature that must not be used unless you understand the risks. The background information is provided in the Cyphal specification. Default: False | |
--pp-max-emptylines | |
If provided this will suppress generation of additional consecutive empty lines beyond the limit set by this argument. Note that this will insert a line post-processor which may reduce performance. Consider using a code formatter on the generated output to enforce whitespace rules instead. | |
--pp-trim-trailing-whitespace | |
Enables a line post-processor that will elide all whitespace at the end of each line. Note that this will insert a line post-processor which may reduce performance. Consider using a code formatter on the generated output to enforce whitespace rules instead. Default: False | |
-pp-rp, --pp-run-program | |
Runs a program after each file is generated but before the file is set to read-only. example # invokes clang-format with the "in-place" argument on each file after it is
# generated.
nnvg --outdir include --templates c_jinja -e .h -pp-rp clang-format -pp-rpa=-i dsdl
| |
-pp-rpa, --pp-run-program-arg | |
Additional arguments to provide to the program specified by –pp-run-program. The last argument will always be the path to the generated file. |
language options¶
Options passed through to templates as options on the target language.
Note that these arguments are passed though without validation, have no effect on the Nunavut library, and may or may not be appropriate based on the target language and generator templates in use.
--target-endianness | |
Possible choices: any, big, little Specify the endianness of the target hardware. This allows serialization logic to be optimized for different CPU architectures. | |
--omit-float-serialization-support | |
Instruct support header generators to omit support for floating point operations in serialization routines. This will result in errors if floating point types are used, however; if you are working on a platform without IEEE754 support and do not use floating point types in your message definitions this option will avoid dead code or compiler errors in generated serialization logic. Default: False | |
--enable-serialization-asserts | |
Instruct support header generators to generate language-specific assert statements as part of serialization routines. By default the serialization logic generated may make assumptions based on documented requirements for calling logic that could expose a system to undefined behavior. The alternative, for languages that do not support exception handling, is to use assertions designed to halt a program rather than execute undefined logic. Default: False | |
--enable-override-variable-array-capacity | |
Instruct support header generators to add the possibility to override max capacity of a variable length array in serialization routines. This option will disable serialization buffer checks and add conditional compilation statements which violates MISRA. Default: False | |
--language-standard, -std | |
For language generators that support different standards of their core language this option can be used to optimize the output. For example, C templates may generate slightly different code for the the c99 standard then for c11. For available support in Nunavut see the documentation for built-in templates (https://nunavut.readthedocs.io/en/latest/docs/templates.html#built-in-template-guide). | |
--configuration, -c | |
There is a set of built-in configuration for Nunvut that provides default falues for known languages as documented in the template guide. This argument lets you specify override configuration yamls. | |
--list-configuration, -lc | |
Lists all configuration values resolved for the given arguments. Default: False |
Example Usage:
# This would include j2 templates for a folder named 'c_jinja'
# and generate .h files into a directory named 'include' using
# dsdl root namespaces found under a folder named 'dsdl'.
nnvg --outdir include --templates c_jinja -e .h dsdl
Contributor Notes¶
👋 Thanks for contributing. This page contains all the details about getting your dev environment setup.
Note
This is documentation for contributors developing nunavut. If you are a user of this software you can ignore everything here.
- To ask questions about nunavut or Cyphal in general please see the OpenCyphal forum.
- See nunavut on read the docs for the full set of nunavut documentation.
- See the OpenCyphal website for documentation on the Cyphal protocol.
Warning
When committing to main you must bump at least the patch number in src/nunavut/_version.py
or the build will fail on the upload step.
Tools¶
tox -e local¶
I highly recommend using the local tox environment when doing python development. It’ll save you hours of lost productivity the first time it keeps you from pulling in an unexpected dependency from your global python environment. You can install tox from brew on osx or apt-get on GNU/Linux. I’d recommend the following environment for vscode:
git submodule update --init --recursive
tox -e local
source .tox/local/bin/activate
cmake¶
Our language generation verification suite uses CMake to build and run unit tests. If you are working with a native language see Nunavut Verification Suite for details on manually running these builds and tests.
Visual Studio Code¶
To use vscode you’ll need:
- vscode
- install vscode command line (Shell Command: Install)
- tox
- cmake (and an available GCC or Clang toolchain, or Docker to use our toolchain-as-container)
Do:
cd path/to/nunavut
git submodule update --init --recursive
tox -e local
source .tox/local/bin/activate
code .
Then install recommended extensions.
Running The Tests¶
To run the full suite of tox tests locally you’ll need docker. Once you have docker installed and running do:
git submodule update --init --recursive
docker pull ghcr.io/opencyphal/toxic:tx22.4.1
docker run --rm -v $PWD:/repo ghcr.io/opencyphal/toxic:tx22.4.1 tox
To run a limited suite using only locally available interpreters directly on your host machine,
skip the docker invocations and use tox -s
.
To run the language verification build you’ll need to use a different docker container:
docker pull ghcr.io/opencyphal/toolshed:ts20.4.1
docker run --rm -it -v $PWD:/workspace ghcr.io/opencyphal/toolshed:ts20.4.1
cd /workspace
./.github/verify.py -l c
./.github/verify.py -l cpp
The verify.py script is a simple commandline generator for our cmake scripts. Use help for details:
./.github/verify.py --help
If you get a “denied” response from ghcr your ip might be getting rate-limited. While these are public containers you’ll have to login to get around any rate-limiting for your local site. See [the github docs](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) for how to setup a docker client login.
Files Generated by the Tests¶
Given that Nunavut is a file generator our tests do have to write files. Normally these files are temporary and therefore automatically deleted after the test completes. If you want to keep the files so you can debug an issue provide a “keep-generated” argument.
example
pytest -k test_namespace_stropping --keep-generated
You will see each test’s output under “build/(test name}”.
Warning
Don’t use this option when running tests in parallel. You will get errors.
Sybil Doctest¶
This project makes extensive use of Sybil doctests. These take the form of docstrings with a structure like thus:
.. invisible-code-block: python
from nunavut.lang.c import filter_to_snake_case
.. code-block:: python
# an input like this:
input = "scotec.mcu.Timer"
# should yield:
filter_to_snake_case(input)
>>> scotec_mcu_timer
The invisible code block is executed but not displayed in the generated documentation and,
conversely, code-block
is both rendered using proper syntax formatting in the documentation
and executed. REPL works the same as it does for doctest
but assert
is also a valid
way to ensure the example is correct especially if used in a trailing invisible-code-block
:
.. invisible-code-block: python
assert 'scotec_mcu_timer' == filter_to_snake_case(input)
These tests are run as part of the regular pytest build. You can see the Sybil setup in the
conftest.py
found under the src
directory but otherwise shouldn’t need to worry about
it. The simple rule is; if the docstring ends up in the rendered documentation then your
code-block
tests will be executed as unit tests.
import file mismatch¶
If you get an error like the following:
_____ ERROR collecting test/gentest_dsdl/test_dsdl.py _______________________________________
import file mismatch:
imported module 'test_dsdl' has this __file__ attribute:
/my/workspace/nunavut/test/gentest_dsdl/test_dsdl.py
which is not the same as the test file we want to collect:
/repo/test/gentest_dsdl/test_dsdl.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
Then you are probably a wonderful developer that is running the unit-tests locally. Pytest’s cache is interfering with your docker test run. To work around this simply delete the pycache files. For example:
#! /usr/bin/env bash
clean_dirs="src test"
for clean_dir in $clean_dirs
do
find $clean_dir -name __pycache__ | xargs rm -rf
find $clean_dir -name \.coverage\* | xargs rm -f
done
Note that we also delete the .coverage intermediates since they may contain different paths between the container and the host build.
Alternatively just nuke everything temporary using git clean:
git clean -X -d -f
Building The Docs¶
We rely on read the docs to build our documentation from github but we also verify this build
as part of our tox build. This means you can view a local copy after completing a full, successful
test run (See Running The Tests) or do
docker run --rm -t -v $PWD:/repo ghcr.io/opencyphal/toxic:tx22.4.1 /bin/sh -c "tox -e docs"
to build
the docs target. You can open the index.html under .tox/docs/tmp/index.html or run a local
web-server:
python3 -m http.server --directory .tox/docs/tmp &
open http://localhost:8000/docs/index.html
Of course, you can just use Visual Studio Code to build and preview the docs using
> reStructuredText: Open Preview
.
apidoc¶
We manually generate the api doc using sphinx-apidoc
. To regenerate use tox -e gen-apidoc
.
Warning
tox -e gen-apidoc
will start by deleting the docs/api directory.
Coverage and Linting Reports¶
We publish the results of our coverage data to sonarcloud and the tox build will fail for any mypy
or black errors but you can view additional reports locally under the .tox
dir.
Coverage¶
We generate a local html coverage report. You can open the index.html under .tox/report/tmp or run a local web-server:
python -m http.server --directory .tox/report/tmp &
open http://localhost:8000/index.html
Mypy¶
At the end of the mypy run we generate the following summaries:
- .tox/mypy/tmp/mypy-report-lib/index.txt
- .tox/mypy/tmp/mypy-report-script/index.txt
Licenses¶
Licence¶
Jinja2 and Markupsafe (BSD 3 clause)¶
Copyright (c) 2009 by the Jinja Team, see AUTHORS for more details.
Some rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
- The names of the contributors may not be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.