MDL SDK API nvidia_logo_transpbg.gif Up
Example for Traversing Compiled Materials
[Previous] [Up] [Next]

This example demonstrates the traversal of compiled materials. It thereby allows investigating the structure of expression graphs based on an example application that reconstructs compilable MDL code.

New Topics

  • Traversal of a compiled material
  • Generating MDL code

Detailed Description

Basic traversal of a compiled material


With this example, we provide the compiled_material_traverser_base, a simple base class for iterating over the structure of a material in a recursive manner. During the traversal, virtual functions are called on certain occasions and while reaching particular elements in the expression graph. Deriving classes, like the compiled_material_traverser_print, implement those virtual methods in order to realize their intended logic. To keep track of the current state, the implementations can provide a custom context object that is passed through for each invocation without any changes by the base class.

Similar to the definition of an MDL material, a compiled material also consists of three sections: parameters, temporaries, and the body. Therefore, the traversal also has three corresponding stages: ES_PARAMETERS, ES_TEMPORARIES, and ES_BODY. There is one virtual method call at the beginning of each of those stages and one at the end, respectively. For convenience, there are two additional stages provided to the deriving classes: ES_NOT_STARTED to define the current state before a traversal has been started and ES_FINISHED to indicate that the traversal is done. Deriving classes are free to keep track of the current state using the context object that is passed through.

During the three main stages, the traversal visits the mi::neuraylib::IExpression and mi::neuraylib::IValue nodes of the tree recursively. When reaching the next node in the tree, the visit_begin(...) method is called. Then, the child elements are processed. In case of multiple children, there is an additional call to visit_child(...) for each child. This will happen while iterating over the arguments of a mi::neuraylib::IExpression_direct_call and while iterating over the components of a mi::neuraylib::IValue_compound. A call to visit_end(...) indicates that the processing of the current node is done.

Each of these three methods provides a pointer to the compiled material that is processed, the custom context object, and a Traversal_element with information about the element that is currently visited. This includes pointers to the current expression, value, parameter, or temporary. Only one of these four pointers is different from NULL during each invocation. Depending on the stage, not all cases are possible. During the ES_PARAMETERS stage, for instance, only parameters and values but no temporaries or expressions can occur. The Traversal_element also provides a sibling_count and a sibling_index that might be useful to derived classes like the compiled_material_traverser_print.

To start the actual traversal, the protected traverse(...) method is called by by the deriving class. At this point the custom context object is provided to the base class.

Reconstructing MDL code as example traversal


The example application contains an MDL printer. Based on the compiled_material_traverser_base this printer generates a serialized representation of the compiled material in MDL syntax.

Using the MDL printer


The input of the traversers is a compiled material. Therefore, we first need to load a module, create a material instance, and compile this instance to get a mi::neuraylib::ICompiled_material. Before starting the iteration of the expression tree, we setup the Compiled_material_traverser_print::Context to configure the behavior of the printer:

// Creates the context of a traversal, which includes required component and configurations.
Compiled_material_traverser_print::Context(
mi::neuraylib::ITransaction* transaction, // used to resolve resources
mi::neuraylib::IMdl_compiler* compiler, // used to resolve resources
bool keep_structure) // show compiler output vs. print valid MDL code

The constructor consumes three arguments. While the first two are required to resolve textures, measured BSDFs, and light profiles, the last one parameter requires more explanations.

The structures produced by the compiler do not always match the structure of the input material. The main reason for that is optimization and the simplification of expressions that are constant during compile time. Another example is the transformation of constants to parameters while in class compilation mode. Generating MDL code that reflects the latter directly may result in invalid MDL code. This can happen if keywords, like material_emission, appear as parameters. However, the discussed printer focuses on generating compilable MDL code but we also want to provide a tool to investigate the produced compiler output. To enable both, the example allows choosing between the two behaviors. Setting keep_structure to true preserves the structure of the compiler output even if this would result in printed MDL code that will not compile. Setting keep_structure to false, which is the default value, will inline the transformed constants back into the body. This results in MDL code that is easier to read and that can be loaded and compiled later.

After setting up the context, the traversal can be started by calling print_mdl(...):

// generates MDL code that can be saved as a module
std::string Compiled_material_traverser_print::print_mdl(
const mi::neuraylib::ICompiled_material* material, // the compiled material to process
Compiled_material_traverser_print::Context& context, // the context that is passed through
const std::string& original_module_name, // qualified name of the original module
const std::string& output_material_name) const // simple name of the output material
This interface represents a compiled material.
Definition: icompiled_material.h:97

Since the mi::neuraylib::ICompiled_material lacks some information to generate a valid module, the original module name has to be provided for referencing exported functions in the original module in which the material to process was defined in. The new material also requires a name that has to be provided, too.

The returned result can then be written into a file for later investigation or processing. Besides that, the context contains some further information about the material:

// indicates whether the output MDL code should be valid or not.
std::string Compiled_material_traverser_print::Context::get_is_valid_mdl()

If we do not inline transformed constants, we want to inform about invalid MDL after running the printer. The result will be false if an invalid case was encountered. Note, this can only be false if keep_structure of the context was set to true.

// gets modules that have been imported directly by the module and used by the input material
const std::set<std::string>& Compiled_material_traverser_print::Context::get_used_modules() const
// gets resources that have been imported directly by the module and used by the input material
const std::set<std::string>& Compiled_material_traverser_print::Context::get_used_resources() const

Since the printer needs to keep track of modules that have to be imported by the generated module, we also provide that information to the user. In addition to the modules, we also track the directly referenced resources.

Running the example from the command line


The example can be started from the command line. The module to process can be specified by passing a qualified name as an argument. Additionally, there are flags to choose between class and instance compilation: -class and -instance, respectively. If none of them is provided, the class compilation mode is used. Furthermore, passing the -keep flag will preserve the structure of the compiler output as described above. Here are some examples:

>> example_traversal ::nvidia::core_definitions -instance -keep

will traverse all materials in the core_definitions module in instance mode while keeping the structure produced by the compiler.

>> example_traversal ::example -class
>> example_traversal ::example
>> example_traversal

All three commands above will produce the same results as they use the default parameters -- the traversal of the materials defined in the example module. The class compilation mode will be used and the compiler generated parameters are inlined.

Restrictions


With the goal of demonstrating the traversal, this printer has some restrictions:

  • Since the original module of the compiled input material is simply imported in the generated MDL module, all local functions that are used by the material need to be exported by the original module. If this is not the case, these dependencies can not be resolved for the generated material.
  • Only MDL 1.4 compliant materials are supported.

The header of compiled_material_traverser_base is listed below. It contains the interface declarations for the basic traversal. Please also refer to the source files for more details on the printing and the formatting that is performed by the printer implementation.

Example Source

Source Code Location:
examples/mdl_sdk/traversal/example_traversal.cpp
examples/mdl_sdk/shared/compiled_material_traverser_base.h
examples/mdl_sdk/shared/compiled_material_traverser_base.cpp
examples/mdl_sdk/traversal/compiled_material_traverser_print.h
examples/mdl_sdk/traversal/compiled_material_traverser_print.cpp

/******************************************************************************
* Copyright 2024 NVIDIA Corporation. All rights reserved.
*****************************************************************************/
// examples/mdl_sdk/shared/compiled_material_traverser_base.h
//
// Utility class for traversing compiled materials.
// Derived classes override virtual functions to implement their logic.
#ifndef COMPILED_MATERIAL_TRAVERSER_BASE_H
#define COMPILED_MATERIAL_TRAVERSER_BASE_H
#include "example_shared.h"
#include <string>
#include <vector>
// A base class that implements a simple traversal logic for compiled materials.
class Compiled_material_traverser_base
{
protected:
// Possible stages of the traversal of an compiled material
enum Traveral_stage
{
// Traversal has not been started
ES_NOT_STARTED = 0,
// Indicates that the parameters of the material are currently traversed
ES_PARAMETERS,
// Indicates that the temporaries of the material are currently traversed
ES_TEMPORARIES,
// Indicates that the main body of the material are currently traversed
ES_BODY,
// Traversal is done
ES_FINISHED,
// For alignment only.
ES_FORCE_32_BIT = 0xffffffffU
};
// An internal structure that is passed to the user code while traversing.
// This struct is used while visiting the material parameters.
struct Parameter
{
explicit Parameter(const mi::neuraylib::IValue* value)
: value(value)
{ }
const mi::neuraylib::IValue* value;
};
// An internal structure that is passed to the user code while traversing.
// This struct is used while visiting the materials temporaries.
struct Temporary
{
explicit Temporary(const mi::neuraylib::IExpression* expression)
: expression(expression)
{ }
const mi::neuraylib::IExpression* expression;
};
// Encapsulated to current element that is visited during the traversal
// It contains either an IExpression, an IValue, a Parameter or a Temporary while the
// others are nullptr.
struct Traversal_element
{
explicit Traversal_element(const mi::neuraylib::IExpression* expression,
mi::Size sibling_count = 1, mi::Size sibling_index = 0)
: expression(expression)
, value(nullptr)
, parameter(nullptr)
, temporary(nullptr)
, sibling_count(sibling_count)
, sibling_index(sibling_index)
{ }
explicit Traversal_element(const mi::neuraylib::IValue* value,
mi::Size sibling_count = 1, mi::Size sibling_index = 0)
: expression(nullptr)
, value(value)
, parameter(nullptr)
, temporary(nullptr)
, sibling_count(sibling_count)
, sibling_index(sibling_index)
{ }
explicit Traversal_element(const Parameter* parameter,
mi::Size sibling_count = 1, mi::Size sibling_index = 0)
: expression(nullptr)
, value(nullptr)
, parameter(parameter)
, temporary(nullptr)
, sibling_count(sibling_count)
, sibling_index(sibling_index)
{ }
explicit Traversal_element(const Temporary* temporary,
mi::Size sibling_count = 1, mi::Size sibling_index = 0)
: expression(nullptr)
, value(nullptr)
, parameter(nullptr)
, temporary(temporary)
, sibling_count(sibling_count)
, sibling_index(sibling_index)
{ }
// Not nullptr if the current traversal element is an IExpression.
const mi::neuraylib::IExpression* expression;
// Not nullptr if the current traversal element is an IValue.
const mi::neuraylib::IValue* value;
// Not nullptr if the current traversal element is a Parameter.
// This can happen only in the ES_PARAMETERS stage.
const Parameter* parameter;
// Not nullptr if the current traversal element is a Parameter.
// This can happen only in the ES_TEMPORARAY stage.
const Temporary* temporary;
// Total number of children at the parent of the currently traversed element.
mi::Size sibling_count;
// Index of the currently traversed element in the list of children at the parent.
mi::Size sibling_index;
};
public:
// virtual destructor
virtual ~Compiled_material_traverser_base() {}; /* = default;*/
protected:
// Traverses a compiled material and calls the corresponding virtual visit methods.
//
// This method is meant to be called by deriving class to start the actual traversal.
//
// Param: material The material that is traversed.
// Param: [in,out] context User defined context that is passed through without changes.
void traverse(const mi::neuraylib::ICompiled_material* material, void* context);
// Called at the beginning of each traversal stage: Parameters, Temporaries and Body.
//
// Param: material The material that is traversed.
// Param: stage The stage that was entered.
// Param: [in,out] context User defined context that is passed through without changes.
virtual void stage_begin(const mi::neuraylib::ICompiled_material* material,
Traveral_stage stage, void* context) {};
// Called at the end of each traversal stage: Parameters, Temporaries and Body.
//
// Param: material The material that is traversed.
// Param: stage The stage that was finished.
// Param: [in,out] context User defined context that is passed through without changes.
virtual void stage_end(const mi::neuraylib::ICompiled_material* material,
Traveral_stage stage, void* context) {};
// Called when the traversal reaches a new element.
//
// Param: material The material that is traversed.
// Param: element The element that was reached.
// Param: [in,out] context User defined context that is passed through without changes.
virtual void visit_begin(const mi::neuraylib::ICompiled_material* material,
const Traversal_element& element, void* context) {};
// Occurs only if the current element has multiple child elements, e.g., a function call.
// In that case, the method is called before each of the children are traversed, e.g.,
// before each argument of a function call.
//
// Param: material The material that is traversed.
// Param: element The currently traversed element with multiple children.
// Param: children_count Number of children of the current element.
// Param: child_index The index of the child that will be traversed next.
// Param: [in,out] context User defined context that is passed through without changes.
virtual void visit_child(const mi::neuraylib::ICompiled_material* material,
const Traversal_element& element,
mi::Size children_count, mi::Size child_index,
void* context) {};
// Called when the traversal reaches finishes an element.
//
// Param: material The material that is traversed.
// Param: element The element that is finished.
// Param: [in,out] context User defined context that is passed through without changes.
virtual void visit_end(const mi::neuraylib::ICompiled_material* material,
const Traversal_element& element, void* context) {};
// Gets the name of a parameter of the traversed material.
//
// Param: material The material that is traversed.
// Param: index Index of the parameter in the materials parameter list.
// Param: out_generated Optional output parameter that indicates whether the parameter
// was generated by the compiler rather than defined in the
// material definition.
//
// Return: The parameter name.
std::string get_parameter_name(const mi::neuraylib::ICompiled_material* material,
mi::Size index, bool* out_generated = nullptr) const;
// Gets the name of a temporary of the traversed material.
// Since the name is usually unknown, due to optimization, a proper name is generated.
//
// Param: material The material that is traversed.
// Param: index Index of the parameter in the materials temporary list.
//
// Return: The temporary name.
std::string get_temporary_name(const mi::neuraylib::ICompiled_material* material,
mi::Size index) const;
private:
// Recursive function that is used for the actual traversal.
// The names of templates are lost during compilation. Therefore, we generate numbered ones.
//
// Param: material The material.
// Param: element The element that is currently visited.
// Param: [in,out] context User defined context that is passed through without changes.
void traverse(const mi::neuraylib::ICompiled_material* material,
const Traversal_element& element, void* context);
};
#endif // COMPILED_MATERIAL_TRAVERSER_BASE_H
The interface to MDL expressions.
Definition: iexpression.h:50
The interface to MDL values.
Definition: ivalue.h:33
Uint64 Size
Unsigned integral type that is large enough to hold the size of all types.
Definition: types.h:112
[Previous] [Up] [Next]