Writing a Native COLLADA Import

The weather this weekend was appalling, so I “muggled in” as they say around here and got a bit side-tracked. For half a year, I have been relying on Assimp – the open asset import library – for loading COLLADA files which have been the main source of test files right now. They can contain a complex scene tree, camera and light definitions, multiple UV sets and much more that was helpful to stress test a scene graph.

However, I noticed a couple of shortcomings with Assimp over the last months:

• Model unit: There is no way to access the model unit and it makes a huge difference for placing objects in surrounding or using depth of field shaders to have things down to a proper scale.
• Z-is-up: Assimp converts (i.e. inserts a 90 degree rotation on the root node) everything to Y-is-up which you usually need for rendering, but as part of a modeling application, my importer had to preserve the correct up-axis.
• Progress reporting: There actually is an interface to report progress, however it always returns 1!
• Cameras: Assimp assumes that cameras are defined on scene level and cannot parent them to a node in the scene tree. Therefore I ended up with lots of COLLADA files where I had the correct field of view camera parameters and empty, transformed group somewhere in the hierarchy called “Camera”.
• Multiple Materials per Mesh: Assimp automatically splits a mesh into multiple meshes when there are multiple materials assigned. While this makes sense for rendering, during the modeling process you want to have everything as one mesh.
• Development progress: Maybe this is a wrong perception, but there doesn’t seem to be that much progress anymore. A colleague of mine is talking to me about their FBX importer for months now but as far as I can see it, it’s not part of an official release yet.
• …

Don’t get me wrong, Assimp does a awesome job at what it does: Give you the data you need in a format that is ideal for rendering. But if you do want to do some geometry processing, I would suggest writing your own importer.

I only have ever tried once to write a COLLADA import (for use in Streetsoccer as a way to import my Blender3D assets) and it gave me so many headaches I promised myself to never try it again 🙂 There are just too many ways to express the same thing, with all the vertex-data-indirections, defining input semantics and so on.

Well, since I used BOOST to parse the XML and extract the model unit (as a post step of my assimp importer), I was wondering how long it would take to write an importer that get’s all the meshes, materials and so on (without animations). The answer: about a day if your base library is good! The architectural design of my scene tree nodes and all the math base code I collected really paid off.

Using BOOST property trees, it was actually quite easy to do. In my iOS code, I used the Apple streaming XML parser which was a headache to use. A stream parser has the advantage of using less memory as it does not allocate any data but just presents the user with the XML data of the current node. The disadvantage is that you have to process the information in the order it is written in the file.

With the new approach, I use BOOST to parse the complete XML into memory and then go over the data in the order I want, which simplifies things A LOT! Here are some findings for anyone else trying to write a COLLADA importer:

• Don’t use a streaming XML parser! Process stuff in the order you want. For example, it helps to first extract the vertex positions and face indices before processing all the input attributes (e.g tex coords, normals, etc).
• ColladaDOM and similar libraries that could be used all seem to be pretty outdated.
• Use lots of test data. Google’s 3D warehouse for SketchUp is a nice source for lots of test models. However …
• … don’t rely on files coming just from one program. SketchUp for example had a bug up to (including) version 7 which wrote out the transparency value incorrectly. And different programs have different ways of using the capabilities given by the spec. You might test 20 files and suddenly release “oh, you can have nodes in a library section and multi-reference them as well!”.
• Speaking of which, there are multiple “opaque modes” defined in the COLLADA specification which influence how transparency and transparent are combined to the final opacity value. Just something to be aware of…
• Don’t use std::istringstream to parse the float/p-arrays, it’s way too slow. I ended up writing a simple import code similar to what is mentioned on the linked page. However …
• … be aware that float values can be in E-notation (e.g. 8.3e-05)!
• Use the count attributes to check against your array parsing code.
• At least log a warning if you find a child tag that your import does not expect/support yet. This makes it way easier to figure out why a new test file doesn’t work when everything else worked before.
• A mesh can have multiple primitive-sets, each with its own material assignment. It’s not uncommon to have a single mesh with 3 triangle primitive sets, each using a different material
• Vertex attributes can be either shared or unshared. What this means is that each attribute can either be uniform for all faces connected to the vertex or you can have per-face-per-vertex normals/texcoords, etc.
• If you use the OSX preview of DAE files or some other source of visual reference, take into account how the other application is rendering. For example, in the Ferrari FF430 test file I often use, there are some duplicate faces with different materials. So a gray and an orange face lie exactly on top of each other. Since I render with GL_LEQUAL, the orange face shines through where the OSX finder seems to render GL_LESS where the gray surface is visible.
• Don’t attempt to implement the complete spec! The format got extended so often it is bloated and I no no-one who supports everything. Matter of fact, I was surprised that you actually express NURBS and parametric surfaces in there.

For reference, the importer code is roughly 1000 lines of code (of which approx. 150 are the fast float/int parsing code) right now and seems to handle meshes/materials very well, no support for animations yet.

Hope that helps

– Alex