Merge branch 'master' of https://github.com/eliemichel/LilySurfaceScr…

…apper

.gitignore

@@ -1,4 +1,5 @@
 __pycache__/
 *.pyc
 /Builds/
-/releases/
+/releases/
+.idea/

README.md

@@ -5,29 +5,29 @@ Lily Surface Scraper
 
 There are many sources for getting PBR textures on the Internet, but it is always a repetitive task to setup the shader once in Blender. Some sources provide an add-on to automatically handle this, but it remains painful to install a new add-on for each of them, and learn over how they slightly differ from the other one.
 
-LilySurfaceScrapper suggest a very intuitive and unified workflow: browse your favorite library in your browser. Once you have made your choice, just copy the URL of the page and paste it in Blender. The script will prompt for potential variants if some are detected, then download the maps and setup the material.
+LilySurfaceScraper suggest a very intuitive and unified workflow: browse your favorite library in your browser. Once you have made your choice, just copy the URL of the page and paste it in Blender. The script will prompt for potential variants if some are detected, then download the maps and setup the material.
 
 This add-on has been designed to make it very easy to add new sources. You can just add a script in the `Scrapers/` directory, it will be automatically detected. See below for more information.
 
 ## Links & Resources
 
 - Video demonstration on [YouTube](https://www.youtube.com/watch?v=KfEhjxvia0Q)
-- Support thread on [BlenderArtists](https://blenderartists.org/t/2-80-lily-surface-scrapper-import-material-from-a-simple-url/1157897)
-- [Downloads](https://github.com/eliemichel/LilySurfaceScrapper/releases)
+- Support thread on [BlenderArtists](https://blenderartists.org/t/2-80-lily-surface-scraper-import-material-from-a-simple-url/1157897)
+- [Downloads](https://github.com/eliemichel/LilySurfaceScraper/releases)
 
 ## Installation
 
-Download the [latest release](https://github.com/eliemichel/LilySurfaceScrapper/releases/latest), then in Blender, go to `Edit > Preferences`, `Add-on`, `Install`, browse to the zip file.
+Download the [latest release](https://github.com/eliemichel/LilySurfaceScraper/releases/latest), then in Blender, go to `Edit > Preferences`, `Add-on`, `Install`, browse to the zip file.
 
-Make sure that you have ticked the small checkbox next to "Import: Lily Surface Scrapper", otherwise the add-on will not be active.
+Make sure that you have ticked the small checkbox next to "Import: Lily Surface Scraper", otherwise the add-on will not be active.
 
 ![Add-on loaded in the User Preferences](doc/preferences.jpg)
 
 ### Preferences
 
 You can set a path to your texture library. If the path is in absolute form, like `C:\Users\Suzanne\Pictures`, you will not have to save your blend files before you can use the add-on.
 
-If a path is relative like `image-textures\lily` _LilySurfaceScrapper_ searches for a folder named _image-textures_ next to your .blend project file and saves the textures inside _image-textures_ in a subfolder named _lily_.
+If a path is relative like `image-textures\lily` _LilySurfaceScraper_ searches for a folder named _image-textures_ next to your .blend project file and saves the textures inside _image-textures_ in a subfolder named _lily_.
 
 ## Usage
 
@@ -59,7 +59,7 @@ To change where the textures are being stored on the drive, check [Preferences](
 
 **NB** The same process is available in the World panel:
 
-![Lily Surface Scrapper in world panel](doc/world.png)
+![Lily Surface Scraper in world panel](doc/world.png)
 
 ## Supported sources
 
@@ -68,36 +68,49 @@ The following sources are supported, feel free to suggest new ones.
 Materials:
 
  - cgbookcase: https://www.cgbookcase.com
- - CC0Textures: https://cc0textures.com
- - Texture Haven: https://texturehaven.com
+ - ambientCG: https://ambientcg.com/
+ - Poly Haven Texture: https://polyhaven.com/textures
  - [Your local filesystem](https://www.youtube.com/watch?v=BXbNA3nN_HI)
  - Search on Textures.one: If the "URL" contains just words, no slash, they are used as search keywords to randomly pick a result on https://textures.one. You can also perform the search yourself and provide a link to the result page on Textures.one. Make sure that the link is for a supported texture site.
 
 Worlds:
 
- - HDRI Haven: https://hdrihaven.com/
+ - Poly Haven HDRIs: https://polyhaven.com/hdris
  - Search on Textures.one (same as for materials)
 
+Lights:
+
+ - ies library: https://ieslibrary.com/
+
 ## Troubleshooting
 
-*Blender hangs forever when downloading the files* If you are using a VPN, try to disable it.
+**Blender hangs forever when downloading the files**  
+If you are using a VPN, try to disable it.
+
+**Cannot import name 'etree' from 'lxml'**
+We tried to bundle lxml into the add-on to avoid issues, but there are still some people having trouble with it. If you get such an error when activating the add-on, install lxml manually by running the following command line in admin mode (adapt the path to your version and installation location of Blender):
+
+    "C:\Program Files\Blender Foundation\Blender 2.93\2.93\python\bin\python.exe" -m pip install lxml -r "C:\Program Files\Blender Foundation\Blender 2.93\2.93\scripts\modules"
+
+Ideally, you could share the folder `C:\Program Files\Blender Foundation\Blender 2.93\2.93\scripts\modules\lxml` that this creates in an issue here so that I can add it to the repo.
 
-If you run into any sort of trouble running this add-on, please fill an [issue](https://github.com/eliemichel/LilySurfaceScrapper/issues/new) on this repository.
+**Other**  
+If you run into any sort of trouble running this add-on, please fill an [issue](https://github.com/eliemichel/LilySurfaceScraper/issues/new) on this repository.
 Please, do not attempt to report problems with the addon to the material and sky sources' websites as they are not involved in this project and I don't want them to receive undue "spam" because of me.
 
 Trouble is to be expected the source websites change their design. Please report it here and be patient so we can fix the addon, or try to propose your own changes (they should be pretty easy to do, see section about new sources below).
 
 ## Adding new sources
 
-I tried to make it as easy as possible to add new sources of data. The only thing to do is to add a python file in `Scrappers/` and define in it a class deriving from `AbstractScrapper`.
+I tried to make it as easy as possible to add new sources of data. The only thing to do is to add a python file in `Scrapers/` and define in it a class deriving from `AbstractScraper`.
 
-You can start from a copy of [`Cc0texturesScrapper.py`](https://github.com/eliemichel/LilySurfaceScrapper/blob/master/blender/LilySurfaceScrapper/Scrappers/Cc0texturesScrapper.py) or [`CgbookcaseScrapper.py`](https://github.com/eliemichel/LilySurfaceScrapper/blob/master/blender/LilySurfaceScrapper/Scrappers/CgbookcaseScrapper.py). The former loads a zip and extracts maps while the second looks for a different URL for each map (base color, normal, etc.).
+You can start from a copy of [`AmbientCgScraper.py`](https://github.com/eliemichel/LilySurfaceScraper/blob/master/blender/LilySurfaceScraper/Scrapers/AmbientCgScraper.py) or [`CgbookcaseScraper.py`](https://github.com/eliemichel/LilySurfaceScraper/blob/master/blender/LilySurfaceScraper/Scrapers/CgbookcaseScraper.py). The former loads a zip and extracts maps while the second looks for a different URL for each map (base color, normal, etc.).
 
 The following three methods are required:
 
 ### canHandleUrl(cls, url)
 
-A static method (just add `@staticmethod` before its definition) that returns `True` only if the scrapper recognizes the URL `url`.
+A static method (just add `@staticmethod` before its definition) that returns `True` only if the scraper recognizes the URL `url`.
 
 ### fetchVariantList(self, url)
 
@@ -125,7 +138,7 @@ Scrap the information of the variant numbered `variant_index`, and write it to `
  - `material_data.maps['height']`: The path to the height map, or None
  - `material_data.maps['vectorDisplacement']`: The path to the vector displacement map, or None
 
-You can define your own texture maps by adding them to `self.maps` in `MaterialData.py`. You can then assign a texture map that name in your scrapper (we, by convention, have a dictionary called `maps_tr` that maps the scraped name onto the internal naming defined in `MaterialData.py`) and translate it to a node setup in for example `CyclesMaterialData.py`.
+You can define your own texture maps by adding them to `self.maps` in `MaterialData.py`. You can then assign a texture map that name in your scraper (we, by convention, have a dictionary called `maps_tr` that maps the scraped name onto the internal naming defined in `MaterialData.py`) and translate it to a node setup in for example `CyclesMaterialData.py`.
 
 ## Utility functions
 
@@ -141,7 +154,7 @@ Get an image from the URL `url`, place it in a directory whose name is generated
 
 ### fetchZip(self, url, material_name, zip_name)
 
-Get a zip file from the URL `url`. This works like `fetchImage()`, returning the path to the zip file. You can then use the [zipfile](https://docs.python.org/3/library/zipfile.html) module, like [`Cc0texturesScrapper.py`](https://github.com/eliemichel/LilySurfaceScrapper/blob/master/blender/LilySurfaceScrapper/Scrappers/Cc0texturesScrapper.py) does.
+Get a zip file from the URL `url`. This works like `fetchImage()`, returning the path to the zip file. You can then use the [zipfile](https://docs.python.org/3/library/zipfile.html) module, like [`AmbientCgScraper.py`](https://github.com/eliemichel/LilySurfaceScraper/blob/master/blender/LilySurfaceScraper/Scrapprs/AmbientCgScraper.py) does.
 
 ### self.clearString(s)
 
@@ -157,18 +170,18 @@ These properties default to True, but can be turned off to prevent the operator
 
 ### callback_handle
 
-It can be useful to have operations run after the operator. Since it is always painful to do so with the vanilla bpy API, Lily Surface Scrapper features a simple callback mechanism. All operators can take a callback as property, a callback being a function called once the operator is done. It recieves one argument, namely the bpy context into which the operator was running.
+It can be useful to have operations run after the operator. Since it is always painful to do so with the vanilla bpy API, Lily Surface Scraper features a simple callback mechanism. All operators can take a callback as property, a callback being a function called once the operator is done. It recieves one argument, namely the bpy context into which the operator was running.
 
 Since Blender operators cannot take arbitrary values like callbacks as properties, a `register_callback()` utility function is provided to convert the callback into a numeric handle that can then be provided to the operator. The following snippet illustrates the process:
 
 ```python
-import LilySurfaceScrapper
+import LilySurfaceScraper
 
 def c(ctx):
     print("Callback running!")
     print(ctx)
 
-h = LilySurfaceScrapper.register_callback(c)
+h = LilySurfaceScraper.register_callback(c)
 bpy.ops.object.lily_surface_import(url="https://cc0textures.com/view.php?tex=Metal01", callback_handle=h)
 ```
 

blender/LilySurfaceScraper/CyclesLightData.py

@@ -15,36 +15,35 @@
 class CyclesLightData(LightData):
     def createLights(self):
         pref = getPreferences()
-        light = bpy.context.object.data
+
+        light = bpy.data.lights.new(self.name, "POINT")
+        bpy.context.object.data = light
 
         light.use_nodes = True
-        light.type = "POINT"
         light.shadow_soft_size = 0
 
-        nodes = light.node_tree.nodes
-        links = light.node_tree.links
+        tree = light.node_tree
+        nodes = tree.nodes
+        links = tree.links
 
         nodes.clear()
 
         ies = self.maps['ies']
-        energyPath = self.maps['energy']
-        with open(energyPath, "r") as f:
-            energy = float(f.read())
+        energy = self.maps['energy']
 
         out = nodes.new(type="ShaderNodeOutputLight")
-        emmision = nodes.new(type="ShaderNodeEmission")
-        links.new(out.inputs['Surface'], emmision.outputs['Emission'])
+        emission = nodes.new(type="ShaderNodeEmission")
+
+        links.new(out.inputs['Surface'], emission.outputs['Emission'])
 
         iesNode = nodes.new(type="ShaderNodeTexIES")
         if pref.ies_pack_files:
             bpy.ops.text.open(filepath=ies, internal=False)
-            name = f"{os.path.basename(os.path.dirname(ies))}.ies"
-            bpy.data.texts["lightData.ies"].name = name
-            iesNode.ies = bpy.data.texts[name]
+            iesNode.ies = bpy.data.texts[os.path.basename(ies)]
         else:
             iesNode.mode = "EXTERNAL"
             iesNode.filepath = ies
-        links.new(iesNode.outputs['Fac'], emmision.inputs["Strength"])
+        links.new(iesNode.outputs['Fac'], emission.inputs["Strength"])
 
         if pref.ies_use_strength:
             if pref.ies_light_strength:
@@ -53,5 +52,12 @@ def createLights(self):
                 links.new(value.outputs["Value"], iesNode.inputs["Strength"])
             else:
                 light.energy = energy
+
+        if pref.ies_add_blackbody:
+            blacbody = nodes.new(type="ShaderNodeBlackbody")
+            blacbody.inputs["Temperature"].default_value = 7000
+            links.new(blacbody.outputs["Color"], emission.inputs["Color"])
 
         autoAlignNodes(out)
+
+        return light

blender/LilySurfaceScraper/CyclesMaterialData.py

@@ -44,7 +44,8 @@ class CyclesMaterialData(MaterialData):
         'ambientOcclusion': '', # FIXME Handle this better https://github.com/KhronosGroup/glTF-Blender-IO/issues/123
         'ambientOcclusionRough': '', # TODO Do something with this
         'glossiness': '',
-    }    
+        'ARM': '',
+    }
 
     def loadImages(self):
         """This is not needed by createMaterial, but is called when
@@ -132,7 +133,43 @@ def mixFrontBackDicts(self):
                     mixed[name] = node
         return mixed
 
+    def armGroup(self):
+        name = "Separate ARM"
+        index = bpy.data.node_groups.find(name)
+        if index != -1:
+            return bpy.data.node_groups[index]
+
+        group = bpy.data.node_groups.new(name, "ShaderNodeTree")
+
+        # input
+        group_inputs = group.nodes.new("NodeGroupInput")
+        group_inputs.location = (-200, 0)
+        group.inputs.new("NodeSocketColor", "ARM map")
+
+        # output
+        group_outputs = group.nodes.new("NodeGroupOutput")
+        group_outputs.location = (200, 0)
+        group.outputs.new("NodeSocketFloat", "AO")
+        group.outputs.new("NodeSocketFloat", "Metalness")
+        group.outputs.new("NodeSocketFloat", "Roughness")
+
+        separate = group.nodes.new("ShaderNodeSeparateRGB")
+
+        # link
+        group.links.new(group_inputs.outputs['ARM map'], separate.inputs[0])
+
+        # Blue channel is metalness map
+        # Red channel is ambient occlusion
+        # Green is roughness
+        group.links.new(separate.outputs["R"], group_outputs.inputs['AO'])
+        group.links.new(separate.outputs["G"], group_outputs.inputs['Roughness'])
+        group.links.new(separate.outputs["B"], group_outputs.inputs['Metalness'])
+
+        return group
+
     def createMaterial(self):
+        pref = getPreferences()
+
         self.initMaterial()
         self.front = {}
         self.back = {}
@@ -144,11 +181,13 @@ def createMaterial(self):
         for map_name, img in self.maps.items():
             if img is None or map_name.split("_")[0] not in __class__.input_tr:
                 continue
-            
+
             self.makeTextureNode(img, map_name)
 
         mixed = self.mixFrontBackDicts()
 
+        aoOutput = None
+
         for name, node in mixed.items():
             if __class__.input_tr.get(name):
                 links.new(node.outputs["Color"], self.principled_node.inputs[__class__.input_tr[name]])
@@ -157,6 +196,7 @@ def createMaterial(self):
                     invert_node = nodes.new(type="ShaderNodeInvert")
                     links.new(node.outputs["Color"], invert_node.inputs["Color"])
                     links.new(invert_node.outputs["Color"], self.principled_node.inputs["Roughness"])
+
                 if name == "diffuse":
                     if not self.principled_node.inputs["Base Color"].is_linked:
                         links.new(node.outputs["Color"], self.principled_node.inputs["Base Color"])
@@ -187,20 +227,30 @@ def createMaterial(self):
                     links.new(combine_node.outputs["Image"], normal_node.inputs["Color"])
                     links.new(normal_node.outputs["Normal"], self.principled_node.inputs["Normal"])
 
-        # Second pass, inserting AO requires that everything else (base color) is wired
-        pref = getPreferences()
-        if pref.use_ao:
-            for name, node in mixed.items():
+                if name == "ARM":
+                    armSplitter = nodes.new(type="ShaderNodeGroup")
+                    armSplitter.node_tree = self.armGroup()
+                    links.new(node.outputs["Color"], armSplitter.inputs[0])
+
+                    links.new(armSplitter.outputs["Roughness"], self.principled_node.inputs["Roughness"])
+                    links.new(armSplitter.outputs["Metalness"], self.principled_node.inputs["Metallic"])
+                    aoOutput = armSplitter.outputs["AO"]
+
                 if name == "ambientOcclusion":
-                    basecolor_links = self.principled_node.inputs["Base Color"].links
-                    if len(basecolor_links) == 0:
-                        continue
-                    base_color_output = basecolor_links[0].from_socket
-                    mix_node = nodes.new(type="ShaderNodeMixRGB")
-                    mix_node.blend_type = 'MULTIPLY'
-                    links.new(base_color_output, mix_node.inputs[1])
-                    links.new(node.outputs["Color"], mix_node.inputs[2])
-                    links.new(mix_node.outputs["Color"], self.principled_node.inputs["Base Color"])
+                    aoOutput = node.outputs["Color"]
+
+            #  inserting AO requires that everything else (base color) is wired
+            if pref.use_ao and aoOutput is not None:
+                basecolor_links = self.principled_node.inputs["Base Color"].links
+                if len(basecolor_links) == 0:
+                    continue
+                base_color_output = basecolor_links[0].from_socket
+                mix_node = nodes.new(type="ShaderNodeMixRGB")
+                mix_node.blend_type = 'MULTIPLY'
+                mix_node.default_value = 1.
+                links.new(base_color_output, mix_node.inputs[1])
+                links.new(aoOutput, mix_node.inputs[2])
+                links.new(mix_node.outputs["Color"], self.principled_node.inputs["Base Color"])
 
         autoAlignNodes(self.mat_output)
 

blender/LilySurfaceScraper/CyclesWorldData.py

@@ -70,9 +70,6 @@ def createWorld(self):
             texcoord_node = nodes.new(type="ShaderNodeTexCoord")
             links.new(texcoord_node.outputs[0], mapping_node.inputs[0])
 
-
-
-
         autoAlignNodes(world_output)
 
         return world

blender/LilySurfaceScraper/LightData.py

@@ -7,14 +7,17 @@
 from .ScrapersManager import ScrapersManager
 from .ScrapedData import ScrapedData
 
+
 class LightData(ScrapedData):
     """Internal representation of world, responsible on one side for
     scrapping texture providers and on the other side to build blender materials.
     This class must not use the Blender API. Put Blender related stuff in subclasses
     like CyclesMaterialData."""
-
-    def reset(self):
-        self.name = "Lily World"
+
+    def __init__(self, url, texture_root="", asset_name=None):
+        super().__init__(url, texture_root=texture_root, asset_name=asset_name, scraping_type="LIGHT")
+
+        self.name = "Lily Light"
         self.maps = {
             'ies': None,
             'energy': None