GitHub - Nanofortress/veryhiddennameidontknowthisnamecsc420projectbywendezhou: Occlusion Filling Team members Hugh Matsubara Wen De Zhou (1001713274) Problem When we are looking around our world, we many great things around us. However some of the things are blocked by others, making us unable to fully enjoy all the great things around us. We propose a solution that can solve this problem, were as everyone in the near distant future has access to intelligence eye wear (i.e. Google Glass) that can keeps track of what you are seeing (before time = now) and what others are seeing, we can use this as the database to generate/fill in the parts where the object is being occluded, allowing us to have the ability to see through objects.. Our solution involves recognize occlusion using segmentation. Figure out what parts of an image are being occluded by matching keypoints in images taken from multiple angles. Using pose estimates, we attempt to fill in the occluded image. Procedure 1. Generate data set by taking pictures of objects at different angles wherein some parts are occluded (i.e. by trees and/or buildings). (Done by Wen De) 2. Use segmentation to detect objects in the source image (the image with object that are occluded) and the data image(s) (the image(s) where the object is not occluded). (Done by Hugh) 3. Use pose estimation to figure out camera positions relative to each shot. (Done by Wen De) 4. Detect key points between those images using SIFT (and/or other techniques), and use those key points to determine the transformation/relation between the images. (Done by Hugh) 5. Use the transformation and segmentation found to transform the data image(s), and use them to reconstruct the object that is being occluded in the source image. (Done by Wen De) 6. Adjust and blend parts of the source image that has been filled in so that the filled out parts closely match the rest of the image even if the images used have different photometry. (Done by Hugh) Implementation This project is going to be done with a group of 2. Images used in this project is going to be taken by us. This assignment is going to be written in MATLAB, with the imaging libraries packaged inside. Since we have yet to learn the full scope of this course (recognition and parts of geometry) we will not be able to list the full libraries we will be using to implement this project. This project is completely original, with no prior on the market, thus we will be trying our best to implement and explore additional options. Inspired From Real-Time Monocular Segmentation and Pose Tracking of Multiple Objects https://www.youtube.com/watch?v=-nFkNPqf1LU Note: this is done with prior shape knowledge, but we do not use that in our project.

Nanofortress / veryhiddennameidontknowthisnamecsc420projectbywendezhou Public

Notifications You must be signed in to change notification settings
Fork 0
Star 3

Occlusion Filling Team members Hugh Matsubara Wen De Zhou (1001713274) Problem When we are looking around our world, we many great things around us. However some of the things are blocked by others, making us unable to fully enjoy all the great things around us. We propose a solution that can solve this problem, were as everyone in the near dist…

3 stars 0 forks Branches Tags Activity

Star

Notifications

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
code		code
data		data
.DS_Store		.DS_Store
Proposal.pdf		Proposal.pdf
README.md		README.md
Testing_results.pdf		Testing_results.pdf
csc420project.docx		csc420project.docx
project_report.pdf		project_report.pdf
testing_results.docx		testing_results.docx
~$sting_results.docx		~$sting_results.docx

Repository files navigation

Occlusion Filling

This is one approach to this project done by me. My partner has his own approach and uses his code sperately.

Note this uses the the vlfeat library, found in http://www.vlfeat.org/download/vlfeat-0.9.20-bin.tar.gz due to its size, it cannot be included in this zip file. Thus to run this code you will need to download vlsift from the link above, and place it in the ./code folder.

The occlusion filling algorithm is located in ./code/project.m Before running the script please set up the environment as this project uses SIFT and DPM.

To setup execute: run('vlfeat-0.9.20/toolbox/vl_setup') on folder /code
To setup DPM, execute: addpath(genpath(pwd)) on directory /code to add all paths. Then run compile.m inside folder /code/dmp

The pictures used is contained in the folder /code/data/project