C++ Code gems: a XML persistency framework

Matthias

With this note I would like to start a series of discussions focussing on special aspects of the code involved in KLayout.

When the project was started, one specific intention was to explore the limits of coding techniques. Despite coming of age, C++ is still a wide and exciting field for explorations of that kind. In particular the metaprogramming techniques employing the subtle yet powerful capabilities of templates and template specialization are - at least in my opinion - still underestimated.

One of the projects that have earned credits in that respect is the Boost project. Started as a nightmare for compiler builders, this project now provides practical solutions for a wide field of applications which formerly were the domain of special preprocessors or code generators - by using the built-in features of modern C++ compilers only.

The XML persistency framework built into KLayout was inspired by the Boost philosophy and has proven extremely useful in the project.

The basic idea for the persistency framework is to provide a direct XML-to-Object mapping for the various files generated and read by KLayout, i.e. the layer properties files, bookmark files, session files and so on. The process of writing a object hierarchy to a file and retrieving it from there (with all the child objects) and summarized in the term "persistency".

The classical solution for this problem is to create a C++ object hierarchy to represent the data and to embed some "load" and "save" methods, which read or write for example from or to a XML DOM structure.

In that approach, the conversion of the XML DOM structure to the internal object representation has to be done manually for every persistency problem.

The process of generating this reader and writer code could be automated, for example by using a interface language like IDL and creating a code generator to produce the C++ code.

However, having read the introduction above, you are probably not surprised to learn that it is possible to delegate the code generation to the compiler.

To be more specific, what is delegated to the compiler is the generation of a table which is worked by a SAX-type parser to implement the reading and provides a dump function to write the contents to a XML file. The framework can attach XML nodes to C++ properties (represented by a getter and setter method). Property values can be itself objects with a XML structure. This way, arbitrary complex hierarchies of XML elements and substructure can be associated with a C++ object hierarchy.

The framework supports iterated properties using the usual "begin" .. "end" iterator pair concept as well as as the proprietary "iter=begin" .. "iter.at_end" construct. Properties can be objects (the getter can return a reference or a copy). Such objects can be bound to XML either with a substructure or to a single string value through a converter that converts the object to a string and back. Properties with a substructure are called "elements" and such without a structure are called "members". Following that naming, two functions are provided to create associations: "make_element" and "make_member".

Elements can be of the same type than the containing object. This way, the framework allows to create arbitrary hierarchies (trees).

Currently, the framework does not support references automatically: when two elements refer to the same object, it is written twice. A n-to-1 association can be emulated by explicitly using integer ID's for example. This is not a basic constraint. Instead, there was simply no need to implement that yet.

In the following example, I use the implementation of the layer properties list (.lyp files). The actual implementation can be found in layLayerProperties.cc and is somewhat more elaborate, because some crabby compilers need a little bit more explicitness. The example is already somewhat complex and I admit it's like being thrown in at the deep end. However, I promise to provide a simple example if there is demand ...

As a first step, we need to instantiate a tl::XMLElementList object statically. This represents the XML structure or parts of it. It is build from parts which are created with "tl::make_element" or "tl::make_member". Both associate getters and setters with a XML node. The first one in addition needs a specification of a sub-hierarchy structure, which is itself a tl::XMLElementList object:

// Format declaration for the LayerPropertiesNode class
// (= association of properties with XML nodes and a XML hierarchy)
static const tl::XMLElementList layer_element = tl::XMLElementList (
  // associate the frame_color_loc property of LayerPropertiesNode with a XML node
  tl::make_member (&LayerPropertiesNode::frame_color_loc,        // getter
                   &LayerPropertiesNode::set_frame_color_code,   // setter
                   "frame-color",                                // node name
                   UIntColorConverter ()) +                      // converter
  // associate the fill_color_loc property of LayerPropertiesNode with a XML node
  tl::make_member (&LayerPropertiesNode::fill_color_loc,         // getter
                   &LayerPropertiesNode::set_fill_color_code,    // setter
                   "fill-color",                                 // node name
                   UIntColorConverter ()) +                      // converter
  // associate the name property of LayerPropertiesNode with a XML node
  tl::make_member (&LayerPropertiesNode::name,                   // getter
                   &LayerPropertiesNode::set_name,               // setter
                   "name") +                                     // node name - no converter
  ...
  // associate the child elements (of the same type) with a XML node plus 
  // a hierarchy of child nodes. "make_element" creates a sub-hierarchy, in this case
  // using the same format than the LayerPropertiesNode.
  tl::make_element (&LayerPropertiesNode::begin_children,        // iterated property - begin
                    &LayerPropertiesNode::end_children,          // iterated property - end
                    &LayerPropertiesNode::add_child,             // "push" setter
                    "group-members",                             // parent node name
                    &layer_element)                              // format of list members
);

// Format declaration of the whole layer properties file
static const tl::XMLElementList layer_prop_list = tl::XMLElementList (
  // associate the top-level layer properties (a iterated property of LayerPropertieList) 
  // with a XML node plus several children
  tl::make_element (&LayerPropertiesList::begin_const,           // iterated property - begin
                    &LayerPropertiesList::end_const,             // iterated property - end
                    &LayerPropertiesList::push_back,             // "push" setter
                    "properties",                                // parent node name
                    &layer_element) +                            // format of list members
  // associate the custom dither pattern list with another XML node. This time the
  // format for each member of that list is declared in place:
  tl::make_element (&LayerPropertiesList::begin_custom_dither_pattern,  // begin
                    &LayerPropertiesList::end_custom_dither_pattern,    // end
                    &LayerPropertiesList::push_custom_dither_pattern,   // push
                    "custom-dither-pattern",                            // parent node name
    // each custom dither pattern is stored as a set of strings, 
    // each one in it's own <line> element.
    tl::make_element (&lay::DitherPatternInfo::to_strings,       // getter (a vector<string>)
                      &lay::DitherPatternInfo::from_strings,     // setter
                      "pattern",                                 // node name
      tl::make_member (&std::vector<std::string>::begin,         // begin
                       &std::vector<std::string>::end,           // end
                       &std::vector<std::string>::push_back,     // push
                       "line")                                   // node name
    ) +
    // now just add two simple scalar properties ..
    tl::make_member (&lay::DitherPatternInfo::order_index, 
                     &lay::DitherPatternInfo::set_order_index, 
                     "order") +
    tl::make_member (&lay::DitherPatternInfo::name, 
                     &lay::DitherPatternInfo::set_name, 
                     "name") 
  )
);

Having mastered that specification, the actual implementation of the reader and writer is a no-brainer:

// Association of the top-level structure with a class
static const tl::XMLStruct<LayerPropertiesList>
layer_prop_list_structure ("layer-properties", &layer_prop_list);

// Implementation of the load method
void LayerPropertiesList::load (tl::XMLSource &stream)
{
  layer_prop_list_structure.parse (stream, *this); 
}

// Implementation of the save method
void LayerPropertiesList::save (std::ostream &os) const
{
  layer_prop_list_structure.write (os, *this);
}

That's actually all! The framework will take care of all the details like tracking the SAX parser's status, performing the string conversions, instantiating objects, error handling and so on.

The formulation of the XML structure is straightforward, provided the C++ object hierarchy fits to the XML structure and proper setters and getters are present. The only major complication is mastering the waterfalls of C++ error messages upon typing errors, but that is a different issue ...