Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implemented RSS reader and auto downloader in reference WebUI #12557

Merged
merged 1 commit into from
Jun 9, 2020
Merged

Implemented RSS reader and auto downloader in reference WebUI #12557

merged 1 commit into from
Jun 9, 2020

Conversation

seproDev
Copy link
Contributor

@seproDev seproDev commented Apr 20, 2020

I have implemented the RSS reader and auto downloader in the reference WebUI. Requires API defined in #12549 and closes #12355.

Screenshots:
image

image

 

Regarding localization: all used strings are in the same context as their desktop ui version. Is there an easy way to port the translations over?

Also the renameRule API seems to be broken. Since I couldn't figure out where the mistake lies I created an issue.

@FranciscoPombal FranciscoPombal added the RSS RSS-related issues/changes label Apr 20, 2020
@glassez
Copy link
Member

glassez commented Apr 20, 2020

Also the renameRule API seems to be broken. Since I couldn't figure out where the mistake lies I created an issue.

Well, I'll look into it ASAP.

@glassez glassez added the WebUI WebUI-related issues/changes label Apr 24, 2020
@Piccirello
Copy link
Member

Wow, someone actually implemented this! Major props @seprode - you're going to get a lot of love from the qBittorrent community for this. This is a big change so it may take a bit to review.

@glassez please include me in all web ui/api PRs

@glassez
Copy link
Member

glassez commented Apr 25, 2020

please include me in all web ui/api PRs

Sorry, what do you mean?

@Chocobo1
Copy link
Member

please include me in all web ui/api PRs

Sorry, what do you mean?

Please remind/ask him to review all web ui/api related PRs.

@glassez
Copy link
Member

glassez commented Apr 27, 2020

Please remind/ask him to review all web ui/api related PRs.

@Piccirello, please review this PR ASAP, since it's most wanted feature for many years.

Copy link
Member

@Piccirello Piccirello left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks pretty good, from the perspective of how it fits into the current web app's architecture. I haven't evaluated the correctness of the implementation in any way, as I've never used qBittorrent's RSS feature. Someone else will need to rigorously test the implementation.

new Event(e).stop();
let completionCount = 0;
rules.forEach(rule => {
window.parent.qBittorrent.RssDownloader.modifyRuleState(rule, "previouslyMatchedEpisodes", [], () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alas, we can't yet use arrow functions while we support IE11. I'd love to drop IE11 support (as Bootstrap 5 has now done), but we've collectively decided to support it for a bit longer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I could tell IE11 support had already (unintentionally) been dropped. Since the refactor from 4.2.0 IE doesn't load the page correctly.

If you still want I can replace all arrow functions.

Copy link
Member

@FranciscoPombal FranciscoPombal Apr 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for officially dropping support for IE 11. I think it has been long enough anyway.

src/webui/www/private/confirmruleclear.html Outdated Show resolved Hide resolved
src/webui/www/private/scripts/client.js Outdated Show resolved Hide resolved
src/webui/www/private/scripts/client.js Outdated Show resolved Hide resolved
src/webui/www/private/scripts/contextmenu.js Outdated Show resolved Hide resolved
src/webui/www/private/views/rss.html Outdated Show resolved Hide resolved
src/webui/www/private/views/rss.html Outdated Show resolved Hide resolved
src/webui/www/private/views/rss.html Outdated Show resolved Hide resolved
src/webui/www/private/views/rss.html Outdated Show resolved Hide resolved
src/webui/www/private/newfeed.html Outdated Show resolved Hide resolved
src/webui/www/private/newfeed.html Outdated Show resolved Hide resolved
src/webui/www/private/newfeed.html Show resolved Hide resolved
src/webui/www/webui.qrc Show resolved Hide resolved
$('showRssReaderLink').addEvent('click', function(e) {
showRssReader = !showRssReader;
LocalPreferences.set('show_rss_reader', showRssReader.toString());
initializeTabs();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, will this work? initializeTabs is defined later.

Copy link
Contributor Author

@seproDev seproDev Apr 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works because initializeTabs isn't actually called there. It's only called on click so after initialization.

Though since there was quite a bit of duplicate code I moved things around and renamed the function to updateTabDisplay to better reflect its purpose.

I also removed the adding and removing of the tabs event listeners , I think it's fine to have those three event listeners even when the buttons are not displayed. Please take a look.

src/webui/www/private/scripts/contextmenu.js Outdated Show resolved Hide resolved
src/webui/www/private/scripts/contextmenu.js Outdated Show resolved Hide resolved
src/webui/www/private/scripts/contextmenu.js Outdated Show resolved Hide resolved
src/webui/www/private/scripts/contextmenu.js Outdated Show resolved Hide resolved
src/webui/www/private/scripts/contextmenu.js Outdated Show resolved Hide resolved
@WolfganP
Copy link

Any news on this PR? Is there any showstopper?

@seproDev
Copy link
Contributor Author

@WolfganP

Any news on this PR? Is there any showstopper?

Still waiting for people to test the functionality.

Other then that #12558 should probably be fixed before merging.

Copy link
Member

@Chocobo1 Chocobo1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I barely scratch the surface, it would take ages for only one man to review it all...

src/webui/www/private/views/rssDownloader.html Outdated Show resolved Hide resolved
src/webui/www/private/views/rssDownloader.html Outdated Show resolved Hide resolved
src/webui/www/private/confirmfeeddeletion.html Outdated Show resolved Hide resolved
src/webui/www/private/views/rssDownloader.html Outdated Show resolved Hide resolved
src/webui/www/private/views/rss.html Outdated Show resolved Hide resolved
src/webui/www/private/views/rss.html Show resolved Hide resolved
@glassez
Copy link
Member

glassez commented May 15, 2020

Other then that #12558 should probably be fixed before merging.

#12843 should probably fix it. Can someone test it?

@seproDev
Copy link
Contributor Author

#12843 should probably fix it. Can someone test it?

Works!

Implemented RSS Reader and AutoDownloader in reference WebUI.
@seproDev
Copy link
Contributor Author

Rebased to resolve merge conflicts.

@Jylijs
Copy link

Jylijs commented Jun 5, 2020

Any update?

@RaduMemetea
Copy link

Nothing new on this?

@glassez
Copy link
Member

glassez commented Jun 8, 2020

@Chocobo1, if you don't have any serious comments in general, and you can't check it thoroughly, I would merge it as is and wait for user feedback. Judging by the related Issue, there is a fairly large user base.

@Chocobo1
Copy link
Member

Chocobo1 commented Jun 8, 2020

I would merge it as is and wait for user feedback.

I'm ok if Piccirello is fine with it too. @Piccirello What do you think?

@Piccirello
Copy link
Member

I tested very quickly with a public RSS feed I found online and everything seemed to work. I'm ok with merging as long as we have users thoroughly test it before it makes it into a release.

@Chocobo1 Chocobo1 added this to the 4.2.6 milestone Jun 9, 2020
@Chocobo1
Copy link
Member

Chocobo1 commented Jun 9, 2020

@seprode
Thank you!

@glassez
Copy link
Member

glassez commented Jun 9, 2020

It would be useful if someone provides test builds with this feature merged.

@seproDev seproDev deleted the webui-rss branch June 9, 2020 13:01
@WolfganP
Copy link

WolfganP commented Jun 9, 2020

It would be useful if someone provides test builds with this feature merged.

A quick test of a new master build worked perfectly (adding new subscription, updating it, selecting entry, getting details, double click to download). Couldn't test the rules system via the webUI, but will report later if any errors appear.

Thanks a lot @seprode !!!

@FranciscoPombal
Copy link
Member

@glassez 1 or 2 hours after this comment is posted, Windows builds with this feature will be available here: https://github.com/FranciscoPombal/qBittorrent/actions/runs/129842756

@jobrien2001
Copy link

jobrien2001 commented Jun 10, 2020

@seprode

I found a bug in the webui.

It truncates an rss url string. Max characters is set too low.

Tried a jacket rss url (which are somewhat long) and it cut the string at a certain point.

Try enter this and you will see. You wont get any results, but you can check in the feeds.json it has been cut.

Try "http://192.168.1.155:9117/api/v2.0/indexers/1337x/results/torznab/api?apikey=o15guuigf7jdy1qj8sdsa45uisermjl&t=search&cat=100070&limit=10"

Also not sure where to post feedback on this new feature so i posted it here. Should i have opened an issue?

UPDATE: edited manually with nano feeds.json with and its updating the feed. Now I have another issue. The RSS downloader popup containing all the forms to create new rules, has all the forms disabled. No error shows on log or browser console.

Could it be the linuxserv docker image that is not good?

@rmarble
Copy link

rmarble commented Jun 10, 2020

I'm happy to help test and give feedback after I get my new server set up.
BUT...I'm a bit of a novice, and have been doing my best to understand the technical part of all this.
Is this built in to a version of qbittorrent, or is there some extra wizardry involved to implement it?

I appreciate your patience with a new Linux user and right brainer. :)

@jobrien2001
Copy link

jobrien2001 commented Jun 10, 2020

setup docker on your environment then use this image:
docker pull linuxserver/qbittorrent:4.3.0202006091235-7009-1720580ubuntu18.04.1-ls50

Update: for more info on the variables go to https://hub.docker.com/r/linuxserver/qbittorrent

@seproDev
Copy link
Contributor Author

seproDev commented Jun 11, 2020

@jobrien2001

Max characters is set too low.

Yes, that's definitely an issue. I think it should also be removed for newfolder and rename_feed.

The RSS downloader popup containing all the forms to create new rules, has all the forms disabled.

The forms stay disabled when you select a rule?
What browser and OS are you using?

@jobrien2001
Copy link

jobrien2001 commented Jun 11, 2020

@seprode

The forms stay disabled when you select a rule?
What browser and OS are you using?

Disabled as soon as i hit the autodownloader button and the popup loads. I cant create any rules.

Unraid

This docker:

https://hub.docker.com/layers/linuxserver/qbittorrent/4.3.0202006091235-7009-1720580ubuntu18.04.1-ls50/images/sha256-aad4e3e8005a05f00c065c79c4c5edb3dd36a01cd941c825b802433d09e38b40?context=explore

Brave, tried edge aswel.

@seproDev
Copy link
Contributor Author

@jobrien2001

I cant create any rules.

So the context menu doesn't load? The Web UI is missing a dedicated add rule button and instead only has the context menu. Maybe that should be added.

@jobrien2001
Copy link

The rss popup for creating rules pops and i can see the forms, but the forms are disabled, so i cannot interact with them to create a new rule.

@seproDev
Copy link
Contributor Author

@jobrien2001
When you open the Rss Downloader window the forms should be disabled since no rules is selected. When you right click on the left hand side the context menu should open to create a new rule.
image

@jobrien2001
Copy link

I cannot test atm but i will report back and see if I can get past it.

I would agree the add rule button should be there. Maybe the forms hidden when they are in a disabled state.

@seproDev
Copy link
Contributor Author

I have started work on fixes in this branch. Should I create an issue for each fix/commit or squash all changes?

@FranciscoPombal
Copy link
Member

FranciscoPombal commented Jun 11, 2020

@seprode

I have started work on fixes in this branch. Should I create an issue for each fix/commit or squash all changes?

Fixes look good. I would say one PR for all these fixes, but with the 3 separate commits. Then others can decide whether or not they should be squashed (in which case the purpose of each change should be mentioned in the commit message body).

@jobrien2001
Copy link

Everything working properly from the UI, but there is still a problem with handling completed rss torrents.

A lot of us use jackett as a proxy to get rss magnets. Example rss item from jackett, note the id and torrenturl attributes (the id is a link to the page and the torrenturl a link that will redirect to a magnet).

    {
        "attr": "",
        "category": "100036",
        "comments": "https://1337x.to/torrent/4492307/The-First-Actress-A-Novel-of-Sarah-Bernhardt-by-C-W-Gortner-EPUB/",
        "date": "11 Jun 2020 22:38:25 +0000",
        "description": "",
        "id": "https://1337x.to/torrent/4492307/The-First-Actress-A-Novel-of-Sarah-Bernhardt-by-C-W-Gortner-EPUB/",
        "jackettindexer": "1337x",
        "link": "http://192.168.1.222:9117/dl/1337x/?jackett_apikey=w3mi0bdfocf7tkq39ex6opjdy1qj8qql&path=Q2ZESjhHUXlrRlJPNEtaR3VaOU5GWGJ1TUpXSmhjWHVULTF5V3NsaGZiOUlJQ0xieXBTZWwtVDJsSGkwRFc3Z3lSUzRZMkZPQ0o3Q1hLTkY5eS1xSEJEMU9VcUgtbV9CM200eUdXN1UyWUxtTXJDZlpseTVyMzliNmVPV1NkMFZNTDUxZDcxVEFLeTJUV1ROcVp2MV84dDJhcFFmeGtFUV9udVItOXJVUEtXazhvOVowcWxqeDNLejg4NFpPSE5yS1RhREs3U0pTVm5Ud1RKa242YmhmV2VHcUpQWThLNjhXMkJUUmxmU0xXcU9LNTlxcnZJS2R5UFV6Z0pHM0JrRGJ4NV9jZw&file=The+First+Actress+A+Novel+of+Sarah+Bernhardt+by+C+W+Gortner+EPUB",
        "size": "5033165",
        "title": "The First Actress A Novel of Sarah Bernhardt by C W Gortner EPUB",
        "torrentURL": "http://192.168.1.222:9117/dl/1337x/?jackett_apikey=w3mi0bdfocf7tkq39ex6opjdy1qj8qql&path=Q2ZESjhHUXlrRlJPNEtaR3VaOU5GWGJ1TUpXSmhjWHVULTF5V3NsaGZiOUlJQ0xieXBTZWwtVDJsSGkwRFc3Z3lSUzRZMkZPQ0o3Q1hLTkY5eS1xSEJEMU9VcUgtbV9CM200eUdXN1UyWUxtTXJDZlpseTVyMzliNmVPV1NkMFZNTDUxZDcxVEFLeTJUV1ROcVp2MV84dDJhcFFmeGtFUV9udVItOXJVUEtXazhvOVowcWxqeDNLejg4NFpPSE5yS1RhREs3U0pTVm5Ud1RKa242YmhmV2VHcUpQWThLNjhXMkJUUmxmU0xXcU9LNTlxcnZJS2R5UFV6Z0pHM0JrRGJ4NV9jZw&file=The+First+Actress+A+Novel+of+Sarah+Bernhardt+by+C+W+Gortner+EPUB"
    }

Jackett's torrentURL and link attributes change every single time an rss is requested. If i set an autodownloader rule and it downloads an item it will not be marked "isRead": true on completion. This causes the downloader to redownload the article when there is a QB restart. For some reason I do not understand this only happens on a QB restart, in other words, it remembers it has already downloaded the article in the current QB process.

I have 2 suspicions:

  1. It could be the id attribute, note that most rss lists use the hash for an id while jackett uses a url link (a jackett limitation). Maybe a variable type problem?

  2. QB uses more than the id attribute to determine if an article should be marked "isRead": true (which it should not). I suspect it could use the feedurl as well, and since the feedurl change constantly this could be interpreted as a new item. Also, jackett links cannot be magnets because of the way it works so this could also be the culprit.

I did some digging and these 2 blocks of code look where this is happening in rss_autodownloader.cpp

I do not know the language so please forgive any errors in my assessment.

void AutoDownloader::handleTorrentDownloadFinished(const QString &url)
{
    const auto job = m_waitingJobs.take(url);
    if (!job) return;

    if (Feed *feed = Session::instance()->feedByURL(job->feedURL))
        if (Article *article = feed->articleByGUID(job->articleData.value(Article::KeyId).toString()))
            article->markAsRead();
}

"if (Feed *feed = Session::instance()->feedByURL(job->feedURL))"
This may not pass because the feed url in the json may not be the same as the one obtained when accessing torrenturl.

        if (BitTorrent::MagnetUri(torrentURL).isValid()) {
            if (Feed *feed = Session::instance()->feedByURL(job->feedURL)) {
                if (Article *article = feed->articleByGUID(job->articleData.value(Article::KeyId).toString()))
                    article->markAsRead();
            }
        }
        else {
            // waiting for torrent file downloading
            // normalize URL string via QUrl since DownloadManager do it
            m_waitingJobs.insert(QUrl(torrentURL).toString(), job);
        }

        return;
    }

"if (BitTorrent::MagnetUri(torrentURL).isValid()) {"
torrentURL's provided by jacket are not magnets, so this may not pass.
"if (Feed *feed = Session::instance()->feedByURL(job->feedURL)) {"
And this line pops again.

@jobrien2001
Copy link

jobrien2001 commented Jun 12, 2020

Update: If I download manually the rss article in the webui by using the RSS tab and right clicking on a giving article and selecting download.... it will correctly mark the article with "isRead": true

Something must be different in the rss auto download code.

Here is confirmation that its working properly marking "isRead": true by downloading manually from rss list:

    {
        "attr": "",
        "category": "100036",
        "comments": "https://1337x.to/torrent/4492748/FreeCourseWeb-Spark-Joy-An-Illustrated-Master-Class-on-the-Art-of-Organizing-and-Tidying-Up-True-EPUB/",
        "date": "11 Jun 2020 10:51:00 +0000",
        "description": "",
        "id": "https://1337x.to/torrent/4492748/FreeCourseWeb-Spark-Joy-An-Illustrated-Master-Class-on-the-Art-of-Organizing-and-Tidying-Up-True-EPUB/",
        "isRead": true,
        "jackettindexer": "1337x",
        "link": "http://192.168.1.222:9117/dl/1337x/?jackett_apikey=w3mi0bdfocf7tkq39ex6opjdy1qj8qql&path=Q2ZESjhHUXlrRlJPNEtaR3VaOU5GWGJ1TUpYMjlvQzVsd2lyc1VHZ1RSRXFBaGhIcklEUFIxc0d4RzI3RGJqalE3WjdGT21Lek1Dd1Mza2h2TWFncjZPTm4xRHdZbGp6Tjk3bWdjamZocXZUX2pQc1Fxd0VrSU1na2R2RlRHS1p0cVJuNVNud1FYdmlyTHBfUzY0ZFAxd2JZeDZuVm1Pa015WmNBOGxBVjN6bW5QTXozYW0tWUtoZXFrNnUxOElyYk9FT2hQR1VfZGVXTThsZXBwUnJMaVROcnlnbGtZb1EtNTNvR1RabFdYcTlWeFBLSjhObHN2c3FBZHdLeGU0bUpQem1oYmdSdE43bU51ZEpsTXVKWlF1dXFKczRQQXV4cndKc0NDRnRHYU96eEJ4WA&file=FreeCourseWeb+Spark+Joy+An+Illustrated+Master+Class+on+the+Art+of+Organizing+and+Tidying+Up+True+EPUB",
        "size": "36280728",
        "title": "FreeCourseWeb Spark Joy An Illustrated Master Class on the Art of Organizing and Tidying Up True EPUB",
        "torrentURL": "http://192.168.1.222:9117/dl/1337x/?jackett_apikey=w3mi0bdfocf7tkq39ex6opjdy1qj8qql&path=Q2ZESjhHUXlrRlJPNEtaR3VaOU5GWGJ1TUpYMjlvQzVsd2lyc1VHZ1RSRXFBaGhIcklEUFIxc0d4RzI3RGJqalE3WjdGT21Lek1Dd1Mza2h2TWFncjZPTm4xRHdZbGp6Tjk3bWdjamZocXZUX2pQc1Fxd0VrSU1na2R2RlRHS1p0cVJuNVNud1FYdmlyTHBfUzY0ZFAxd2JZeDZuVm1Pa015WmNBOGxBVjN6bW5QTXozYW0tWUtoZXFrNnUxOElyYk9FT2hQR1VfZGVXTThsZXBwUnJMaVROcnlnbGtZb1EtNTNvR1RabFdYcTlWeFBLSjhObHN2c3FBZHdLeGU0bUpQem1oYmdSdE43bU51ZEpsTXVKWlF1dXFKczRQQXV4cndKc0NDRnRHYU96eEJ4WA&file=FreeCourseWeb+Spark+Joy+An+Illustrated+Master+Class+on+the+Art+of+Organizing+and+Tidying+Up+True+EPUB"
    }

@glassez
Copy link
Member

glassez commented Jun 12, 2020

@jobrien2001, your last comments are unrelated to both current topic and WebUI. If you want to report some problem in RSS core functionality you should create separate Issue (one Issue per problem).

@jobrien2001
Copy link

jobrien2001 commented Jun 14, 2020

Add rule button and feed max length is working now. Manual download button is working and so is Open news.

BTW the buttons looks great and the user interface looks more straight forward because of them.

@jobrien2001
Copy link

jobrien2001 commented Jun 14, 2020

Not sure if this is WebUI or another RSS core problem.

-add a feed
-add rule assigned to that feed.
-delete feed

Rule still exists assigned to that feed (which doesnt exist anymore). "affectedFeeds": [xxxxxxx]

This might be harmless or intended to preserve the rule settings.

The reference to the feed can be removed by simply opening the rule and hitting save again.

@jobrien2001
Copy link

3 Days running without a problem.

1 thought about the icons: I found myself clicking on the big trashcan icon on the main toolbar instead of the small trashcan when trying to delete a rule. Maybe a plus and minus would be better choice for icons to distinguish from the main toolbar.

@glassez Created a separate issue. Thank you.

@seproDev
Copy link
Contributor Author

@jobrien2001

Not sure if this is WebUI or another RSS core problem.

As long as this doesn't break anything I would keep it as intended behavior. Even if this were considered a bug I don't think the WebUI should deal with it. Instead that should be changed at API or Core level.

1 thought about the icons

Currently all shown icon are identical to the desktop UI. If we were to change them here they should also be changed in the desktop UI.

I thought about adding icons to the "New subscription", "Mark items read", "Update all" and "RSS Downloader..." buttons. The problem is that it the regular buttons are styled by the browser and thus the custom styling for the image buttons would only really be fitting for one browser.

@glassez
Copy link
Member

glassez commented Jun 16, 2020

Currently all shown icon are identical to the desktop UI. If we were to change them here they should also be changed in the desktop UI.

It's a disadvantage of currently used approach, IMO. I have repeatedly suggested that we separate Web UI from Desktop UI completely. Really they are separate applications.

@Piccirello
Copy link
Member

I have repeatedly suggested that we separate Web UI from Desktop UI completely.

I've yet to see this suggested by anyone that actually uses the Web UI. If this split were to happen, I predict that the Web UI and APIs would quickly fall behind in functionality.

@jobrien2001
Copy link

jobrien2001 commented Jun 17, 2020

May I suggest an enhancement?

A size column in the RSS list, if thats its available in the API. Some feeds do provide a size attribute.

@FranciscoPombal
Copy link
Member

@jobrien2001 please open new threads to discuss each new bug or feature.

IceCodeNew pushed a commit to IceCodeNew/qBittorrent-Enhanced-Edition that referenced this pull request Aug 28, 2020
Implemented RSS reader and auto downloader in reference WebUI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
RSS RSS-related issues/changes WebUI WebUI-related issues/changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

RSS functionality in the reference WebUI implementation