Inference with pre-trained model and sample data is not working.

[jeongchanKim]

I tried to test the inference with ubuntu environment.
I used the pre-trained model and the sample data(HR_photo).
But I cannot generate well.
Some generated images are black, and some looks almost noise.

This is what I set from conda environment.
Is there any missing or mismatched library version?

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
_tflow_select 2.1.0 gpu
absl-py 1.4.0 pypi_0 pypi
astor 0.8.1 pypi_0 pypi
blas 1.0 mkl
bzip2 1.0.8 h7b6447c_0
c-ares 1.19.0 h5eee18b_0
ca-certificates 2023.05.30 h06a4308_0
cached-property 1.5.2 pypi_0 pypi
cairo 1.16.0 hb05425b_5
certifi 2021.5.30 py36h06a4308_0
colorama 0.4.4 pyhd3eb1b0_0
cudatoolkit 10.0.130 0
cudnn 7.6.5 cuda10.0_0
cupti 10.0.130 0
dataclasses 0.8 pypi_0 pypi
expat 2.5.0 h6a678d5_0
ffmpeg 4.0 hcdf2ecd_0
fontconfig 2.14.1 h4c34cd2_2
freeglut 3.0.0 hf484d3e_5
freetype 2.12.1 h4a9f257_0
gast 0.2.2 pypi_0 pypi
glib 2.69.1 h4ff587b_1
google-pasta 0.2.0 pyhd3eb1b0_0
graphite2 1.3.14 h295c915_1
grpcio 1.48.2 pypi_0 pypi
h5py 3.1.0 pypi_0 pypi
harfbuzz 1.8.8 hffaf4a1_0
hdf5 1.10.2 hba1933b_1
icu 58.2 he6710b0_3
importlib-metadata 4.8.3 pypi_0 pypi
intel-openmp 2022.1.0 h9e868ea_3769
jasper 2.0.14 hd8c5072_2
jpeg 9e h5eee18b_1
keras-applications 1.0.8 py_1
keras-preprocessing 1.1.2 pyhd3eb1b0_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libdeflate 1.17 h5eee18b_0
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgfortran-ng 7.5.0 ha8ba4b0_17
libgfortran4 7.5.0 ha8ba4b0_17
libglu 9.0.0 hf484d3e_1
libgomp 11.2.0 h1234567_1
libopencv 3.4.2 hb342d67_1
libopus 1.3.1 h7b6447c_0
libpng 1.6.39 h5eee18b_0
libprotobuf 3.17.2 h4ff587b_1
libstdcxx-ng 11.2.0 h1234567_1
libtiff 4.5.1 h6a678d5_0
libuuid 1.41.5 h5eee18b_0
libvpx 1.7.0 h439df22_0
libwebp-base 1.2.4 h5eee18b_1
libxcb 1.15 h7f8727e_0
libxml2 2.10.4 hcbfbd50_0
lz4-c 1.9.4 h6a678d5_0
markdown 3.3.7 pypi_0 pypi
mkl 2020.2 256
mkl-service 2.3.0 py36he8ac12f_0
mkl_fft 1.3.0 py36h54f3939_0
mkl_random 1.1.1 py36h0573a6f_0
ncurses 6.4 h6a678d5_0
numpy 1.19.5 pypi_0 pypi
numpy-base 1.19.2 py36hfa32c7d_0
opencv 3.4.2 py36h6fd60c2_1
openssl 1.1.1v h7f8727e_0
opt_einsum 3.3.0 pyhd3eb1b0_1
pcre 8.45 h295c915_0
pip 21.2.2 py36h06a4308_0
pixman 0.40.0 h7f8727e_1
protobuf 3.19.6 pypi_0 pypi
py-opencv 3.4.2 py36hb342d67_1
pyserial 3.5 py36h06a4308_0
python 3.6.13 h12debd9_1
readline 8.2 h5eee18b_0
scipy 1.5.2 py36h0b6359f_0
setuptools 58.0.4 py36h06a4308_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.41.2 h5eee18b_0
tensorboard 1.15.0 pyhb230dea_0
tensorflow 1.15.0 gpu_py36h5a509aa_0
tensorflow-base 1.15.0 gpu_py36h9dcbed7_0
tensorflow-estimator 1.15.1 pyh2649769_0
tensorflow-gpu 1.15.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tqdm 4.63.0 pyhd3eb1b0_0
typing-extensions 4.1.1 pypi_0 pypi
webencodings 0.5.1 py36_1
werkzeug 2.0.3 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
wrapt 1.15.0 pypi_0 pypi
xz 5.4.2 h5eee18b_0
zipp 3.6.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0
zstd 1.5.5 hc292b87_0

[jeongchanKim]

This is what I got from HR_photo 1(40).jpg

Most of the results are same.

[jeongchanKim]

This log is what I run for inference.

checkpoint/generator_Hayao_weight/
WARNING:tensorflow:From test.py:39: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From test.py:41: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons
• https://github.com/tensorflow/io (for I/O related ops)
  If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /home/jc-kim/anaconda3/envs/animeganv2/lib/python3.6/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /home/jc-kim/jckim/AnimeGANv2/net/generator.py:41: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:From /home/jc-kim/jckim/AnimeGANv2/net/generator.py:58: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From test.py:43: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From test.py:45: The name tf.GPUOptions is deprecated. Please use tf.compat.v1.GPUOptions instead.

WARNING:tensorflow:From test.py:46: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From test.py:46: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2023-09-12 10:56:09.931424: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2023-09-12 10:56:09.936578: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3699800000 Hz
2023-09-12 10:56:09.936918: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56446be0b050 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-09-12 10:56:09.936935: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2023-09-12 10:56:09.938045: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2023-09-12 10:56:10.001288: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-12 10:56:10.001452: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56446d886270 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-09-12 10:56:10.001463: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 3080, Compute Capability 8.6
2023-09-12 10:56:10.001551: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-12 10:56:10.001648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA GeForce RTX 3080 major: 8 minor: 6 memoryClockRate(GHz): 1.71
pciBusID: 0000:2b:00.0
2023-09-12 10:56:10.001836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2023-09-12 10:56:10.002618: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2023-09-12 10:56:10.003291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2023-09-12 10:56:10.003482: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2023-09-12 10:56:10.004285: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2023-09-12 10:56:10.004958: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2023-09-12 10:56:10.006899: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-09-12 10:56:10.006943: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-12 10:56:10.007048: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-12 10:56:10.007117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2023-09-12 10:56:10.007139: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2023-09-12 10:56:10.007300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-09-12 10:56:10.007307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2023-09-12 10:56:10.007310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2023-09-12 10:56:10.007362: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-12 10:56:10.007459: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-12 10:56:10.007544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9294 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3080, pci bus id: 0000:2b:00.0, compute capability: 8.6)
[*] Success to read checkpoint/generator_Hayao_weight/Hayao-99.ckpt
0%| | 0/45 [00:00<?, ?it/s]2023-09-12 10:57:17.122270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-09-12 10:58:11.942482: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Not found: ./bin/ptxas not found
Relying on driver to perform ptx compilation. This message will be only logged once.
2023-09-12 10:58:11.954097: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
60%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 27/45 [01:04<00:04, 4.09it/s]2023-09-12 10:58:21.565655: W tensorflow/core/common_runtime/bfc_allocator.cc:305] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
/home/jc-kim/jckim/AnimeGANv2/tools/adjust_brightness.py:25: RuntimeWarning: divide by zero encountered in double_scalars
brightness_difference = brightness1 / brightness2
/home/jc-kim/jckim/AnimeGANv2/tools/adjust_brightness.py:33: RuntimeWarning: invalid value encountered in multiply
dstf = dst * brightness_difference
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 45/45 [01:08<00:00, 1.52s/it]
test-time: 68.44015574455261 s
one image test time : 1.5208923498789468 s

[jeongchanKim]

Oh I got this log also, after uncomment the stats_graph(tf.get_default_graph()) in test.py

Use tf.compat.v1.graph_util.tensor_shape_from_node_def_name
125 ops no flops stats due to incomplete shapes.
Parsing Inputs...
Incomplete shape.

=========================Options=============================
-max_depth 10000
-min_bytes 0
-min_peak_bytes 0
-min_residual_bytes 0
-min_output_bytes 0
-min_micros 0
-min_accelerator_micros 0
-min_cpu_micros 0
-min_params 0
-min_float_ops 1
-min_occurrence 0
-step -1
-order_by float_ops
-account_type_regexes .*
-start_name_regexes .*
-trim_name_regexes
-show_name_regexes .*
-hide_name_regexes
-account_displayed_op_only true
-select float_ops
-output stdout:

==================Model Analysis Report======================
Incomplete shape.

[FanYans]

me too