Training on custom dataset

[abhigoku10]

@Tangshitao @iaoqian Thanks fofr sharing ur code base i had few queries

1. can we train on custom dataset if so what are the changes to be made into the code base??
2. can we use only multiview generation capability seperately for one use case and panoramic capability to other usecase ?? how tightly coupled are they
   Thansk in advance

[kkaiwwana]

@abhigoku10
thanks for your question, though I'm not the owner of this repo.

For (1) I would say, yes. just refer to the code in src/dataset and custom your dataset load data in certain format. (btw, I could be wrong, it seems that the code @Tangshitao shared in this repo do not support batchsize > 1 training. If you wanna do that so, you should modify that part of code by your own).
For (2) I didn't really understand what you exactly wanna do.

[abhigoku10]

@iaoqian thanks for ur reponse ...
Q1. the author has used scannet and matter3d plot dataset which has depth , promopts , K and other params so should my custom dataset consists of all these things ?? if so which are the minimal things required for training on the custom data
Q2.i Wanted to knw if can use the repo only for panoramic image generation only instead of multiview ?? how much tightly coupled these modules are ??

[kkaiwwana]

@abhigoku10
For Q1, Depth is not required for panorama generation. As for a minimal dataset implementation, Prompts, Image GT is required which are key inputs for SD. If you wanna use CAA, K, R (or just homography matrix for adjacent camera pairs. It requires code modification, though use K, R do not) is required.
For Q2, I'm afraid not. This repo will be an implementation of SD without CAA (as you said if you don't wanna use it for multiview generation). If I understand it right, you wanna generate panorama images directly (like end-to-end panorama generation)? If so, in this case, you don't need CAA to ensure consistency of different views. However, this might be challenging for SD (like SD 2.0 used in this repo) to generate a panorama image directly (which usually has a higher resolution and extreme ration).

[abhigoku10]

@iaoqian Thanks for the response
Q1. without CAA , K and R multiview data generation is not possbile rigth ?? my custom dataset has images and prompts but doesnot have the other params .
Q2. Yes for my first application 1 i need only multiview output and for application 2 i need just panoroma output . but i think CAA is required as it brings in the continutity with tthe image

[kkaiwwana]

@abhigoku10

1. yes. you have some multi-view data (like shot from different angle at on scene), right? in practice you can use some other method (for example, cv2 findHomography) to get homography matrix between 2 views and that's enough for caa.
2. it looks like multiview consistency is required in your application. for sure caa can improve it. for your application 2, maybe you can generate multi-views and then stitch them together.

[OrangeSodahub]

@abhigoku10 Hi, have you successfully trained your custom data ?