Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update questions.md #55

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Update questions.md #55

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions analysis/questions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Open Questions
The best research work often leaves the reader with more questions than it answers. Here's some of the questions we thought of while reproducing this work.
* Waiting for long training runs to end is frustrating, and impacts research and development. How can we further improve the speed and scalability of our system?
* Sentiment is not just black or white, positive or negative, it's much more nuanced and a result of multiple factors. What are the other significant features our regression model is focusing on? What other discriminitive language tasks would these features do well on?
* Sentiment is not just black or white, positive or negative, it's much more nuanced and a result of multiple factors. What are the other significant features our regression model is focusing on? What other discriminative language tasks would these features do well on?
* More generally, what other tasks could unsupervised language modeling at scale be used for? Discriminative or not.
* The performance of the model on sentiment transfer seems to exhibit uncertainty dependent on the most recently modeled samples from the unsupervised task. Due to this apparent dependence on data, we'd ideally like to fine tune our model to a subset of data that is relevant to the task being transferred to. With this in mind, given a specific subset of data, how can we fine-tune our language model to the specified data?
* Our model learned sentiment because it was a relevant piece of information for reconstruction of product reviews, not because it was told to learn sentiment. In our case we even found that it learned negative sentiment, contrary to OpenAI's model. In an ideal scenario we should have the ability to supervise/force our model to learn about a specific characteristic, such as author frustration. Given a target transfer task, how can performance on that task be used to regularize what the model learns about language?
* Most importantly, can the above be achieved without adversely impacting performance? Will the model retain its understanding of general language properties and its ability to generalize and transfer well?

-----

[<- Data Parallel Scalability](./scale.md)
[<- Data Parallel Scalability](./scale.md)