USD: Introducing USD support

This commit introduces an exporter to Pixar's Universal Scene
Description (USD) format.

- The USD libraries are expected to be in /opt/usd/, not yet built by
  install_deps.sh. I'll work on that while this patch is under review.
- Only experimental support for instancing; by default all duplicated
  objects are made real in the USD file. This is fine for exporting a
  linked-in posed character, not so much for thousands of pebbles etc.
- This patch contains LazyDodo's fixes for building on Windows in D5359.

== Meshes ==

USD seems to support neither per-material nor per-face-group
double-sidedness, so we just use the flag from the first non-empty
material slot. If there is no material we default to double-sidedness.

Each UV map is stored on the mesh in a separate primvar. Materials can
refer to these UV maps, but this is not yet exported by Blender. The
primvar name is the same as the UV Map name. This is to allow the
standard name "st" for texture coordinates by naming the UV Map as such,
without having to guess which UV Map is the "standard" one.

Face-varying mesh normals are written to USD. When the mesh has custom
loop normals those are written. Otherwise the poly flag `ME_SMOOTH` is
inspected to determine the normals.

The UV maps and mesh normals take up a significant amount of space, so
exporting them is optional. They're still enabled by default, though.
For comparison: a shot of Spring (03_035_A) is 1.2 GiB when exported
with UVs and normals, and 262 MiB without. We probably have room for
optimisation of written UVs and normals.

The mesh subdivision scheme isn't using the default value 'Catmull
Clark', but uses 'None', indicating we're exporting a polygonal mesh.
This is necessary for USD to understand our normals; otherwise the mesh
is always rendered smooth. In the future we may want to expose this
choice of subdivision scheme to the user, or auto-detect it when we
actually support exporting pre-subdivision meshes.

A possible optimisation could be to inspect whether all polygons are
smooth or flat, and mark the USD mesh as such. This can be added when
needed.

== Animation ==

Mesh and transform animation are now written when passing
`animation=True` to the export operator. There is no inspection of
whether an object is actually animated or not; USD can handle
deduplication of static values for us.

The administration of which timecode to use for the export is left to
the file-format-specific concrete subclasses of
`AbstractHierarchyIterator`; the abstract iterator itself doesn't know
anything about the passage of time. This will allow subclasses for the
frame-based USD format and time-based Alembic format.

== support for simple preview materials ==

Very simple versions of the materials are now exported, using only the
viewport diffuse RGB, metallic, and roughness.

When there are multiple materials, the mesh faces are stored as geometry
subset and each material is assigned to the appropriate subset. If there
is only one material this is skipped.

The first material if any) is always applied to the mesh itself
(regardless of the existence of geometry subsets), because the Hydra
viewport doesn't support materials on subsets. See
PixarAnimationStudios/OpenUSD#542 for more info.

Note that the geometry subsets are not yet time-sampled, so it may break
when an animated mesh changes topology.

== Hair ==

Only the parent strands are exported, and only with a constant colour.
No UV coordinates, no information about the normals.

== Camera ==

Only perspective cameras are supported for now.

== Particles ==

Particles are only written when they are alive, which means that they
are always visible (there is currently no code that deals with marking
them as invisible outside their lifespan).

Particle-system-instanced objects are exported by suffixing the object
name with the particle's persistent ID, giving each particle XForm a
unique name.

== Instancing/referencing ==

This exporter has experimental support for instancing/referencing.

Dupli-object meshes are now written to USD as references to the original
mesh. This is still very limited in correctness, as there are issues
referencing to materials from a referenced mesh.

I am still committing this, as it gives us a place to start when
continuing the quest for proper instancing in USD.

== Lights ==

USD does not directly support spot lights, so those aren't exported yet.
It's possible to add this in the future via the UsdLuxShapingAPI. The
units used for the light intensity are also still a bit of a mystery.

== Fluid vertex velocities ==

Currently only fluid simulations (not meshes in general) have explicit
vertex velocities. This is the most important case for exporting
velocities, though, as the baked mesh changes topology all the time, and
thus computing the velocities at import time in a post-processing step
is hard.

CMakeLists.txt

@@ -335,6 +335,10 @@ option(WITH_CODEC_SNDFILE       "Enable libsndfile Support (http://www.mega-nerd
 option(WITH_ALEMBIC             "Enable Alembic Support" OFF)
 option(WITH_ALEMBIC_HDF5        "Enable Legacy Alembic Support (not officially supported)" OFF)
 
+# Universal Scene Description support
+option(WITH_USD                 "Enable Universal Scene Description (USD) Support" OFF)
+set(USD_PATH "/opt/usd" CACHE STRING "Universal Scene Description (USD) installation path")
+
 # 3D format support
 # Disable opencollada when we don't have precompiled libs
 option(WITH_OPENCOLLADA   "Enable OpenCollada Support (http://www.opencollada.org)" ${_init_OPENCOLLADA})
@@ -1791,6 +1795,7 @@ if(FIRST_RUN)
   info_cfg_option(WITH_OPENVDB)
   info_cfg_option(WITH_ALEMBIC)
   info_cfg_option(WITH_QUADRIFLOW)
+  info_cfg_option(WITH_USD)
 
   info_cfg_text("Compiler Options:")
   info_cfg_option(WITH_BUILDINFO)

build_files/cmake/Modules/FindUSD.cmake

@@ -0,0 +1,70 @@
+# - Find Universal Scene Description (USD) library
+# Find the native USD includes and libraries
+# This module defines
+#  USD_INCLUDE_DIRS, where to find USD headers, Set when
+#                        USD_INCLUDE_DIR is found.
+#  USD_LIBRARIES, libraries to link against to use USD.
+#  USD_ROOT_DIR, The base directory to search for USD.
+#                    This can also be an environment variable.
+#  USD_FOUND, If false, do not try to use USD.
+#
+
+#=============================================================================
+# Copyright 2019 Blender Foundation.
+#
+# Distributed under the OSI-approved BSD License (the "License");
+# see accompanying file Copyright.txt for details.
+#
+# This software is distributed WITHOUT ANY WARRANTY; without even the
+# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the License for more information.
+#=============================================================================
+
+# If USD_ROOT_DIR was defined in the environment, use it.
+IF(NOT USD_ROOT_DIR AND NOT $ENV{USD_ROOT_DIR} STREQUAL "")
+  SET(USD_ROOT_DIR $ENV{USD_ROOT_DIR})
+ENDIF()
+
+SET(_usd_SEARCH_DIRS
+  ${USD_ROOT_DIR}
+  /usr/local
+  /sw # Fink
+  /opt/local # DarwinPorts
+  /opt/lib/usd
+  /opt/usd
+)
+
+FIND_PATH(USD_INCLUDE_DIR
+  NAMES
+    pxr/usd/usd/api.h
+  HINTS
+    ${_usd_SEARCH_DIRS}
+  PATH_SUFFIXES
+    include
+)
+
+FIND_LIBRARY(USD_LIBRARY
+  NAMES
+  usd
+  HINTS
+    ${_usd_SEARCH_DIRS}
+  PATH_SUFFIXES
+    lib64 lib lib/static
+)
+
+# handle the QUIETLY and REQUIRED arguments and set USD_FOUND to TRUE if
+# all listed variables are TRUE
+INCLUDE(FindPackageHandleStandardArgs)
+FIND_PACKAGE_HANDLE_STANDARD_ARGS(USD DEFAULT_MSG USD_LIBRARY USD_INCLUDE_DIR)
+
+IF(USD_FOUND)
+  SET(USD_LIBRARIES ${USD_LIBRARY})
+  SET(USD_INCLUDE_DIRS ${USD_INCLUDE_DIR})
+ENDIF(USD_FOUND)
+
+MARK_AS_ADVANCED(
+  USD_INCLUDE_DIR
+  USD_LIBRARY
+)
+
+UNSET(_usd_SEARCH_DIRS)

release/scripts/startup/bl_ui/space_topbar.py

@@ -400,6 +400,8 @@ def draw(self, _context):
             self.layout.operator("wm.collada_export", text="Collada (Default) (.dae)")
         if bpy.app.build_options.alembic:
             self.layout.operator("wm.alembic_export", text="Alembic (.abc)")
+        if bpy.app.build_options.usd:
+            self.layout.operator("wm.usd_export", text="Universal Scene Description (.usdc, .usda)")
 
 
 class TOPBAR_MT_file_external_data(Menu):

source/blender/CMakeLists.txt

@@ -158,3 +158,6 @@ endif()
 if(WIN32)
   add_subdirectory(blendthumb)
 endif()
+if(WITH_USD)
+  add_subdirectory(usd)
+endif()

source/blender/depsgraph/DEG_depsgraph_query.h

@@ -26,6 +26,8 @@
 #ifndef __DEG_DEPSGRAPH_QUERY_H__
 #define __DEG_DEPSGRAPH_QUERY_H__
 
+#include "BLI_iterator.h"
+
 #include "DEG_depsgraph.h"
 #include "DEG_depsgraph_build.h"
 

source/blender/editors/io/CMakeLists.txt

@@ -26,6 +26,7 @@ set(INC
   ../../depsgraph
   ../../makesdna
   ../../makesrna
+  ../../usd
   ../../windowmanager
   ../../../../intern/guardedalloc
 )
@@ -39,11 +40,13 @@ set(SRC
   io_cache.c
   io_collada.c
   io_ops.c
+  io_usd.c
 
   io_alembic.h
   io_cache.h
   io_collada.h
   io_ops.h
+  io_usd.h
 )
 
 set(LIB
@@ -69,6 +72,13 @@ if(WITH_ALEMBIC)
   endif()
 endif()
 
+if(WITH_USD)
+  list(APPEND LIB
+    bf_usd
+  )
+  add_definitions(-DWITH_USD)
+endif()
+
 if(WITH_INTERNATIONAL)
   add_definitions(-DWITH_INTERNATIONAL)
 endif()

source/blender/editors/io/io_ops.c

@@ -33,6 +33,10 @@
 #  include "io_alembic.h"
 #endif
 
+#ifdef WITH_USD
+#  include "io_usd.h"
+#endif
+
 #include "io_cache.h"
 
 void ED_operatortypes_io(void)
@@ -46,6 +50,9 @@ void ED_operatortypes_io(void)
   WM_operatortype_append(WM_OT_alembic_import);
   WM_operatortype_append(WM_OT_alembic_export);
 #endif
+#ifdef WITH_USD
+  WM_operatortype_append(WM_OT_usd_export);
+#endif
 
   WM_operatortype_append(CACHEFILE_OT_open);
   WM_operatortype_append(CACHEFILE_OT_reload);

source/blender/editors/io/io_usd.c

@@ -0,0 +1,227 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software  Foundation,
+ * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * The Original Code is Copyright (C) 2016 Blender Foundation.
+ * All rights reserved.
+ */
+
+/** \file
+ * \ingroup editor/io
+ */
+
+#ifdef WITH_USD
+#  include "DNA_space_types.h"
+
+#  include "BKE_context.h"
+#  include "BKE_main.h"
+#  include "BKE_report.h"
+
+#  include "BLI_path_util.h"
+#  include "BLI_string.h"
+#  include "BLI_utildefines.h"
+
+#  include "RNA_access.h"
+#  include "RNA_define.h"
+
+#  include "UI_interface.h"
+#  include "UI_resources.h"
+
+#  include "WM_api.h"
+#  include "WM_types.h"
+
+#  include "DEG_depsgraph.h"
+
+#  include "io_usd.h"
+#  include "usd.h"
+
+const EnumPropertyItem rna_enum_usd_export_evaluation_mode_items[] = {
+    {DAG_EVAL_RENDER,
+     "RENDER",
+     0,
+     "Render",
+     "Use Render settings for object visibility, modifier settings, etc"},
+    {DAG_EVAL_VIEWPORT,
+     "VIEWPORT",
+     0,
+     "Viewport",
+     "Use Viewport settings for object visibility, modifier settings, etc"},
+    {0, NULL, 0, NULL, NULL},
+};
+
+/* Stored in the wmOperator's customdata field to indicate it should run as a background job.
+ * This is set when the operator is invoked, and not set when it is only executed. */
+#  define AS_BACKGROUND_JOB 1
+
+static int wm_usd_export_invoke(bContext *C, wmOperator *op, const wmEvent *event)
+{
+  /* Mark this operator call as 'invoked', so that it'll run as a background job. */
+  op->customdata = POINTER_FROM_INT(AS_BACKGROUND_JOB);
+
+  if (!RNA_struct_property_is_set(op->ptr, "filepath")) {
+    Main *bmain = CTX_data_main(C);
+    char filepath[FILE_MAX];
+
+    if (BKE_main_blendfile_path(bmain)[0] == '\0') {
+      BLI_strncpy(filepath, "untitled", sizeof(filepath));
+    }
+    else {
+      BLI_strncpy(filepath, BKE_main_blendfile_path(bmain), sizeof(filepath));
+    }
+
+    BLI_path_extension_replace(filepath, sizeof(filepath), ".usdc");
+    RNA_string_set(op->ptr, "filepath", filepath);
+  }
+
+  WM_event_add_fileselect(C, op);
+
+  return OPERATOR_RUNNING_MODAL;
+
+  UNUSED_VARS(event);
+}
+
+static int wm_usd_export_exec(bContext *C, wmOperator *op)
+{
+  if (!RNA_struct_property_is_set(op->ptr, "filepath")) {
+    BKE_report(op->reports, RPT_ERROR, "No filename given");
+    return OPERATOR_CANCELLED;
+  }
+
+  char filename[FILE_MAX];
+  RNA_string_get(op->ptr, "filepath", filename);
+
+  const bool as_background_job = (POINTER_AS_INT(op->customdata) & AS_BACKGROUND_JOB) != 0;
+  const bool selected_objects_only = RNA_boolean_get(op->ptr, "selected_objects_only");
+  const bool visible_objects_only = RNA_boolean_get(op->ptr, "visible_objects_only");
+  const bool export_animation = RNA_boolean_get(op->ptr, "export_animation");
+  const bool export_hair = RNA_boolean_get(op->ptr, "export_hair");
+  const bool export_uvmaps = RNA_boolean_get(op->ptr, "export_uvmaps");
+  const bool export_normals = RNA_boolean_get(op->ptr, "export_normals");
+  const bool export_materials = RNA_boolean_get(op->ptr, "export_materials");
+  const bool use_instancing = RNA_boolean_get(op->ptr, "use_instancing");
+  const bool evaluation_mode = RNA_enum_get(op->ptr, "evaluation_mode");
+
+  struct USDExportParams params = {
+      export_animation,
+      export_hair,
+      export_uvmaps,
+      export_normals,
+      export_materials,
+      selected_objects_only,
+      visible_objects_only,
+      use_instancing,
+      evaluation_mode,
+  };
+
+  bool ok = USD_export(C, filename, &params, as_background_job);
+
+  return as_background_job || ok ? OPERATOR_FINISHED : OPERATOR_CANCELLED;
+}
+
+static void wm_usd_export_draw(bContext *UNUSED(C), wmOperator *op)
+{
+  uiLayout *layout = op->layout;
+  uiLayout *col;
+  struct PointerRNA *ptr = op->ptr;
+
+  uiLayoutSetPropSep(layout, true);
+
+  col = uiLayoutColumn(layout, true);
+  uiItemR(col, ptr, "selected_objects_only", 0, NULL, ICON_NONE);
+  uiItemR(col, ptr, "visible_objects_only", 0, NULL, ICON_NONE);
+
+  col = uiLayoutColumn(layout, true);
+  uiItemR(col, ptr, "export_animation", 0, NULL, ICON_NONE);
+  uiItemR(col, ptr, "export_hair", 0, NULL, ICON_NONE);
+  uiItemR(col, ptr, "export_uvmaps", 0, NULL, ICON_NONE);
+  uiItemR(col, ptr, "export_normals", 0, NULL, ICON_NONE);
+  uiItemR(col, ptr, "export_materials", 0, NULL, ICON_NONE);
+  uiItemR(col, ptr, "use_instancing", 0, NULL, ICON_NONE);
+
+  col = uiLayoutColumn(layout, true);
+  uiItemR(col, ptr, "evaluation_mode", 0, NULL, ICON_NONE);
+}
+
+void WM_OT_usd_export(struct wmOperatorType *ot)
+{
+  ot->name = "Export USD";
+  ot->description = "Export current scene in a USD archive";
+  ot->idname = "WM_OT_usd_export";
+
+  ot->invoke = wm_usd_export_invoke;
+  ot->exec = wm_usd_export_exec;
+  ot->poll = WM_operator_winactive;
+  ot->ui = wm_usd_export_draw;
+
+  WM_operator_properties_filesel(
+      ot, 0, FILE_BLENDER, FILE_SAVE, WM_FILESEL_FILEPATH, FILE_DEFAULTDISPLAY, FILE_SORT_ALPHA);
+
+  RNA_def_boolean(ot->srna,
+                  "selected_objects_only",
+                  false,
+                  "Only Export Selected Objects",
+                  "Only selected objects are exported. Unselected parents of selected objects are "
+                  "exported as empty transform");
+
+  RNA_def_boolean(ot->srna,
+                  "visible_objects_only",
+                  true,
+                  "Only Export Visible Objects",
+                  "Only visible objects are exported. Invisible parents of visible objects are "
+                  "exported as empty transform");
+
+  RNA_def_boolean(ot->srna,
+                  "export_animation",
+                  false,
+                  "Export Animation",
+                  "When checked, the render frame range is exported. When false, only the current "
+                  "frame is exported");
+  RNA_def_boolean(ot->srna,
+                  "export_hair",
+                  false,
+                  "Export Hair",
+                  "When checked, hair is exported as USD curves");
+  RNA_def_boolean(ot->srna,
+                  "export_uvmaps",
+                  true,
+                  "Export UV Maps",
+                  "When checked, all UV maps of exported meshes are included in the export");
+  RNA_def_boolean(ot->srna,
+                  "export_normals",
+                  true,
+                  "Export Normals",
+                  "When checked, normals of exported meshes are included in the export");
+  RNA_def_boolean(ot->srna,
+                  "export_materials",
+                  true,
+                  "Export Materials",
+                  "When checked, the viewport settings of materials are exported as USD preview "
+                  "materials, and material assignments are exported as geometry subsets");
+
+  RNA_def_boolean(ot->srna,
+                  "use_instancing",
+                  false,
+                  "Use Instancing (EXPERIMENTAL)",
+                  "When true, dupli-objects are written as instances of the original in USD. "
+                  "Experimental feature, not working perfectly");
+
+  RNA_def_enum(ot->srna,
+               "evaluation_mode",
+               rna_enum_usd_export_evaluation_mode_items,
+               DAG_EVAL_RENDER,
+               "Evaluation Mode",
+               "Determines visibility of objects and modifier settings");
+}
+
+#endif /* WITH_USD */