I could run a full series of tests, for all models and for cpu and cuda mode.

[koinkoin-project]

I can pull up my console logs and test the same types of mini-prompts for all the default models offered in biniou, with the cpu / and biniou logs.

I don't yet know how to organise this, so to make sure it's useful to you, let me know if you want such logs to be archived here.

[Woolverine94]

Hello @koinkoin-project,

Thank you for this proposal. I've already done a first pass of metrics measurement on cpu-only with simple prompts (e.g. : "a cute kitten", but haven't published it yet.
I don't think you'll have slightly differents results in cpu mode, but it will be very interesting to redo a similar benchmark with CUDA to validate the benefits of it.

[koinkoin-project]

Hello,

I'll test to see and run a series of cpu/gpu processes that I'll try to classify by models tested. And try to post the endless terminal output as message attachments so as not to make it difficult to read the information displayed.

I'll start very soon,

[Woolverine94]

Thanks a lot. Take your time, there's no rush !

[koinkoin-project]

################################
TEST-3D-IMG2SHAPE-DEFAULT-CPU-1
################################
MODULE : 3D
MODEL : img2shape
QUALITY : default

RESULT :

INPUT:

!

`./webui.sh

[biniou 🧠]: Detected TCMalloc installation : using it.
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.1.0+cu121 with CUDA 1201 (you have 2.1.0+cpu)
Python 3.11.6 (you have 3.11.2)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
Running on local URL: https://0.0.0.0:7860

To create a public link, set share=True in launch().
text_config_dict is provided which will be used to initialize CLIPTextConfig. The value text_config["id2label"] will be overriden.
text_config_dict is provided which will be used to initialize CLIPTextConfig. The value text_config["bos_token_id"] will be overriden.
text_config_dict is provided which will be used to initialize CLIPTextConfig. The value text_config["eos_token_id"] will be overriden.
renderer/diffusion_pytorch_model.safetensors not found
Downloading (…)renderer/config.json: 100%|█████████████████████████████████████| 496/496 [00:00<00:00, 5.32MB/s]
Downloading (…)_encoder/config.json: 100%|█████████████████████████████████████| 560/560 [00:00<00:00, 1.22MB/s]
Downloading (…)rocessor_config.json: 100%|█████████████████████████████████████| 315/315 [00:00<00:00, 1.26MB/s]
Downloading (…)cheduler_config.json: 100%|█████████████████████████████████████| 276/276 [00:00<00:00, 4.01MB/s]
Downloading (…)9b/prior/config.json: 100%|█████████████████████████████████████| 493/493 [00:00<00:00, 7.52MB/s]
Downloading (…)renderer/config.json: 100%|█████████████████████████████████████| 551/551 [00:00<00:00, 4.82MB/s]
Downloading (…)on_pytorch_model.bin: 100%|███████████████████████████████████| 905M/905M [05:07<00:00, 2.94MB/s]
Downloading (…)on_pytorch_model.bin: 100%|███████████████████████████████████| 905M/905M [05:40<00:00, 2.66MB/s]
Downloading (…)on_pytorch_model.bin: 100%|█████████████████████████████████| 1.26G/1.26G [05:53<00:00, 3.58MB/s]
Downloading pytorch_model.bin: 100%|███████████████████████████████████████| 1.21G/1.21G [06:15<00:00, 3.23MB/s]
The config attributes {'renderer': ['shap_e', 'ShapERenderer']} were passed to ShapEImg2ImgPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.1G [06:15<00:00, 11.0MB/s]
Keyword arguments {'renderer': ['shap_e', 'ShapERenderer'], 'safety_checker': StableDiffusionSafetyChecker(
(vision_model): CLIPVisionModel(
(vision_model): CLIPVisionTransformer(
(embeddings): CLIPVisionEmbeddings(
(patch_embedding): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14), bias=False)
(position_embedding): Embedding(257, 1024)
)
(pre_layrnorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(encoder): CLIPEncoder(
(layers): ModuleList(
(0-23): 24 x CLIPEncoderLayer(
(self_attn): CLIPAttention(
(k_proj): Linear(in_features=1024, out_features=1024, bias=True)
(v_proj): Linear(in_features=1024, out_features=1024, bias=True)
(q_proj): Linear(in_features=1024, out_features=1024, bias=True)
(out_proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(layer_norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(mlp): CLIPMLP(
(activation_fn): QuickGELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(layer_norm2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
)
)
)
(post_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
)
)
(visual_projection): Linear(in_features=1024, out_features=768, bias=False)
), 'feature_extractor': CLIPFeatureExtractor {
"crop_size": {
"height": 224,
"width": 224
},
"do_center_crop": true,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"feature_extractor_type": "CLIPFeatureExtractor",
"image_mean": [
0.48145466,
0.4578275,
0.40821073
],
"image_processor_type": "CLIPFeatureExtractor",
"image_std": [
0.26862954,
0.26130258,
0.27577711
],
"resample": 3,
"rescale_factor": 0.00392156862745098,
"size": {
"shortest_edge": 224
},
"use_square_size": false
}
} are not expected by ShapEImg2ImgPipeline and will be ignored.

[biniou 🧠]: Generation finished in 700 seconds
`

################################
TEST-3D-IMG2SHAPE-DEFAULT-CUDA-1
################################
ERROR

`biniou@drpbiniou1test:~/biniou$ bash biniouCuda.sh
Found existing installation: torch 2.1.0+cpu
Uninstalling torch-2.1.0+cpu:
Would remove:
/home/biniou/biniou/env/bin/convert-caffe2-to-onnx
/home/biniou/biniou/env/bin/convert-onnx-to-caffe2
/home/biniou/biniou/env/bin/torchrun
/home/biniou/biniou/env/lib/python3.11/site-packages/functorch/*
/home/biniou/biniou/env/lib/python3.11/site-packages/torch-2.1.0+cpu.dist-info/*
/home/biniou/biniou/env/lib/python3.11/site-packages/torch/*
/home/biniou/biniou/env/lib/python3.11/site-packages/torchgen/*
Proceed (Y/n)? Y
Successfully uninstalled torch-2.1.0+cpu
Found existing installation: torchvision 0.16.0+cpu
Uninstalling torchvision-0.16.0+cpu:
Would remove:
/home/biniou/biniou/env/lib/python3.11/site-packages/torchvision-0.16.0+cpu.dist-info/*
/home/biniou/biniou/env/lib/python3.11/site-packages/torchvision.libs/libjpeg.ceea7512.so.62
/home/biniou/biniou/env/lib/python3.11/site-packages/torchvision.libs/libpng16.7f72a3c5.so.16
/home/biniou/biniou/env/lib/python3.11/site-packages/torchvision.libs/libz.76028ad6.so.1
/home/biniou/biniou/env/lib/python3.11/site-packages/torchvision/*
Proceed (Y/n)? Y
Successfully uninstalled torchvision-0.16.0+cpu
Found existing installation: torchaudio 2.1.0+cpu
Uninstalling torchaudio-2.1.0+cpu:
Would remove:
/home/biniou/biniou/env/lib/python3.11/site-packages/torchaudio-2.1.0+cpu.dist-info/*
/home/biniou/biniou/env/lib/python3.11/site-packages/torchaudio/*
Proceed (Y/n)? Y
Successfully uninstalled torchaudio-2.1.0+cpu
Collecting torch
Using cached torch-2.1.0-cp311-cp311-manylinux1_x86_64.whl.metadata (25 kB)
Collecting torchvision
Using cached torchvision-0.16.0-cp311-cp311-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting torchaudio
Using cached torchaudio-2.1.0-cp311-cp311-manylinux1_x86_64.whl.metadata (5.7 kB)
Requirement already satisfied: filelock in ./env/lib/python3.11/site-packages (from torch) (3.9.0)
Requirement already satisfied: typing-extensions in ./env/lib/python3.11/site-packages (from torch) (4.8.0)
Requirement already satisfied: sympy in ./env/lib/python3.11/site-packages (from torch) (1.12)
Requirement already satisfied: networkx in ./env/lib/python3.11/site-packages (from torch) (3.0)
Requirement already satisfied: jinja2 in ./env/lib/python3.11/site-packages (from torch) (3.1.2)
Requirement already satisfied: fsspec in ./env/lib/python3.11/site-packages (from torch) (2023.4.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in ./env/lib/python3.11/site-packages (from torch) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in ./env/lib/python3.11/site-packages (from torch) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in ./env/lib/python3.11/site-packages (from torch) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in ./env/lib/python3.11/site-packages (from torch) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in ./env/lib/python3.11/site-packages (from torch) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in ./env/lib/python3.11/site-packages (from torch) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in ./env/lib/python3.11/site-packages (from torch) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in ./env/lib/python3.11/site-packages (from torch) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in ./env/lib/python3.11/site-packages (from torch) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in ./env/lib/python3.11/site-packages (from torch) (2.18.1)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in ./env/lib/python3.11/site-packages (from torch) (12.1.105)
Requirement already satisfied: triton==2.1.0 in ./env/lib/python3.11/site-packages (from torch) (2.1.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in ./env/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch) (12.3.52)
Requirement already satisfied: numpy in ./env/lib/python3.11/site-packages (from torchvision) (1.24.2)
Requirement already satisfied: requests in ./env/lib/python3.11/site-packages (from torchvision) (2.28.1)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in ./env/lib/python3.11/site-packages (from torchvision) (9.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./env/lib/python3.11/site-packages (from jinja2->torch) (2.1.2)
Requirement already satisfied: charset-normalizer<3,>=2 in ./env/lib/python3.11/site-packages (from requests->torchvision) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in ./env/lib/python3.11/site-packages (from requests->torchvision) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./env/lib/python3.11/site-packages (from requests->torchvision) (1.26.13)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.11/site-packages (from requests->torchvision) (2022.12.7)
Requirement already satisfied: mpmath>=0.19 in ./env/lib/python3.11/site-packages (from sympy->torch) (1.3.0)
Using cached torch-2.1.0-cp311-cp311-manylinux1_x86_64.whl (670.2 MB)
Using cached torchvision-0.16.0-cp311-cp311-manylinux1_x86_64.whl (6.9 MB)
Using cached torchaudio-2.1.0-cp311-cp311-manylinux1_x86_64.whl (3.3 MB)
Installing collected packages: torch, torchvision, torchaudio
Successfully installed torch-2.1.0 torchaudio-2.1.0 torchvision-0.16.0

[biniou 🧠]: Detected TCMalloc installation : using it.
Running on local URL: https://0.0.0.0:7860

To create a public link, set share=True in launch().
text_config_dict is provided which will be used to initialize CLIPTextConfig. The value text_config["id2label"] will be overriden.
text_config_dict is provided which will be used to initialize CLIPTextConfig. The value text_config["bos_token_id"] will be overriden.
text_config_dict is provided which will be used to initialize CLIPTextConfig. The value text_config["eos_token_id"] will be overriden.
renderer/diffusion_pytorch_model.safetensors not found
The config attributes {'renderer': ['shap_e', 'ShapERenderer']} were passed to ShapEImg2ImgPipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
Keyword arguments {'renderer': ['shap_e', 'ShapERenderer'], 'safety_checker': StableDiffusionSafetyChecker(
(vision_model): CLIPVisionModel(
(vision_model): CLIPVisionTransformer(
(embeddings): CLIPVisionEmbeddings(
(patch_embedding): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14), bias=False)
(position_embedding): Embedding(257, 1024)
)
(pre_layrnorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(encoder): CLIPEncoder(
(layers): ModuleList(
(0-23): 24 x CLIPEncoderLayer(
(self_attn): CLIPAttention(
(k_proj): Linear(in_features=1024, out_features=1024, bias=True)
(v_proj): Linear(in_features=1024, out_features=1024, bias=True)
(q_proj): Linear(in_features=1024, out_features=1024, bias=True)
(out_proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(layer_norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
(mlp): CLIPMLP(
(activation_fn): QuickGELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(layer_norm2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
)
)
)
(post_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
)
)
(visual_projection): Linear(in_features=1024, out_features=768, bias=False)
), 'feature_extractor': CLIPFeatureExtractor {
"crop_size": {
"height": 224,
"width": 224
},
"do_center_crop": true,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"feature_extractor_type": "CLIPFeatureExtractor",
"image_mean": [
0.48145466,
0.4578275,
0.40821073
],
"image_processor_type": "CLIPFeatureExtractor",
"image_std": [
0.26862954,
0.26130258,
0.27577711
],
"resample": 3,
"rescale_factor": 0.00392156862745098,
"size": {
"shortest_edge": 224
},
"use_square_size": false
}
} are not expected by ShapEImg2ImgPipeline and will be ignored.
Traceback (most recent call last):
File "/home/biniou/biniou/env/lib/python3.11/site-packages/gradio/queueing.py", line 407, in call_prediction
output = await route_utils.call_process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/gradio/route_utils.py", line 226, in call_process_api
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/gradio/blocks.py", line 1550, in process_api
result = await self.call_function(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/gradio/blocks.py", line 1185, in call_function
prediction = await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/gradio/utils.py", line 661, in wrapper
response = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/gradio/utils.py", line 661, in wrapper
response = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/ressources/common.py", line 312, in wrap_func
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/ressources/img2shape.py", line 100, in image_img2shape
pipe_img2shape = pipe_img2shape.to(device_img2shape)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/diffusers/pipelines/pipeline_utils.py", line 848, in to
module.to(device, dtype)
File "/home/biniou/biniou/env/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2271, in to
return super().to(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1160, in to
return self._apply(convert)
^^^^^^^^^^^^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 810, in _apply
module._apply(fn)
File "/home/biniou/biniou/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 810, in _apply
module._apply(fn)
File "/home/biniou/biniou/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 810, in _apply
module._apply(fn)
[Previous line repeated 3 more times]
File "/home/biniou/biniou/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 833, in _apply
param_applied = fn(param)
^^^^^^^^^
File "/home/biniou/biniou/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1158, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 MiB. GPU 0 has a total capacty of 1.95 GiB of which 16.00 MiB is free. Including non-PyTorch memory, this process has 1.93 GiB memory in use. Of the allocated memory 1.57 GiB is allocated by PyTorch, and 12.37 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
img2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shapeimg2shape
`

[Woolverine94]

Thanks for your reporting. Render of the gif is very strange (should have more volume). 700 seconds is definitely not bad, as this module is incredibly slow (had to investigate why).
Don't bother doing test with CUDA on this hardware : at the very least 3GB VRAM is definitely required for diffusers, and there's no way a 2GB VRAM GPU could be useful on biniou.