Optimize PROPFIND having CommentsApp enabled by default #27339

mrow4a · 2017-03-08T13:25:38Z

The idea here is to reduce number of O(n) queries coming from each PROPFIND in the web browser for instance with enabled comments app (this is by default).

Prefetching required data in one query instead of iterative queries should reduce number of queries to O(1), where N is number of files in the folder.

Query
SELECT object_id, COUNT(object_id) FROM oc_comments C WHERE object_id IN(?) AND object_id NOT IN(SELECT object_id FROM oc_comments_read_markers CRM WHERE C.object_id = CRM.object_id AND CRM.user_id = '?' AND CRM.marker_datetime > C.creation_timestamp) GROUP BY object_id;
does the job very well. Yes, I know it is more OLAP query, but this is what Web Front End is about. This is also most optimal query in our case https://sqlperformance.com/2012/12/t-sql-queries/left-anti-semi-join

Finish first optimization
Add chunking of IN() query for more NodeIDs to 100
Add unit tests for optimization
Inform that Comments app is now "usable" for bigger instances @felixboehm @cdamken

Case 1

- 16 queries - this is worst case because caching of getDirectoryListing is not yet done

Case 2
100 NON-SHARED/SHARED files, just someone uploaded >100 files to the folder

- 200 queries - this is best case

mention-bot · 2017-03-08T13:25:40Z

@mrow4a, thanks for your PR! By analyzing the history of the files in this pull request, we identified @DeepDiver1975, @PVince81 and @tcitworld to be potential reviewers.

mrow4a · 2017-03-08T13:30:00Z

EDIT: Updated in main post

mrow4a · 2017-03-08T13:32:00Z

apps/dav/lib/Connector/Sabre/CommentPropertiesPlugin.php

+			$this->numberOfCommentsForNodes[$folderNodeID] = 0;
+
+			// Get IDs for all children of the parent folder
+			$children = $folderNode->getDirectoryListing();


This gets exactly the same result as https://github.com/owncloud/core/blob/master/apps/dav/lib/Connector/Sabre/SharesPlugin.php#L157, so need to be cached, otherwise it is huge overhead

@DeepDiver1975 @PVince81

mrow4a · 2017-03-10T13:17:47Z

@DeepDiver1975 @PVince81 New Benchmark in the top post.. Seems comments app will be fixed now. However I want to continue with caching.. I cannot stand it like that :>

DeepDiver1975 · 2017-03-10T13:27:26Z

lib/private/Comments/Manager.php

+	 * 			AND CRM.marker_datetime > C.creation_timestamp)
+	 * GROUP BY C.object_id;
+	 *
+	 * @param $objectType string the object type, e.g. 'files'


should be

@param string $objectType the object type, e.g. 'files'

DeepDiver1975 · 2017-03-10T13:29:58Z

lib/private/Comments/Manager.php

+	 * GROUP BY C.object_id;
+	 *
+	 * @param $objectType string the object type, e.g. 'files'
+	 * @param int[] NodeIDs that may be returned


DeepDiver1975 · 2017-03-10T13:30:30Z

tests/lib/Comments/ManagerTest.php

@@ -293,6 +303,62 @@ public function testGetNumberOfCommentsForObject() {
 		$this->assertSame($amount, 4);
 	}

+	public function testGetNumberOfUnreadCommentsForNodes() {
+		$manager = $this->getManager();
+		$user1 = $this->createMock('\OCP\IUser');


better use

IUser::class

mrow4a · 2017-03-12T11:29:55Z

@butonic Jorn, do you know which part of the querybuilder for ORACLE could cause error below in

core/lib/private/Comments/Manager.php

Line 389 in 4e84cd2

    
           public function getNumberOfUnreadCommentsForNodes($objectType, $objectIds, IUser $user) {

? For other DBMS it is ok

I have problems spotting an issue, @DeepDiver1975 says it is the c and crm is not escaped, but not sure how to address that.

butonic · 2017-03-12T11:36:30Z

It seens object_id is not always escaped.

butonic · 2017-03-12T11:38:12Z

Try using an alias for c.object_id. Especially in The count function.

mrow4a · 2017-03-12T12:19:45Z

@PVince81 @DeepDiver1975 Covered will unit tests and checked manually. To review

butonic · 2017-03-12T12:22:01Z

C and CRM are not a problem, as long as they are consistently escaped or not escaped. In contrast to mysql, postgres or sqlite oracle uppercases unquoted identifiers.

butonic · 2017-03-12T12:53:48Z

lib/private/Comments/Manager.php

+		$objectIdChunks = array_chunk($objectIds, 100);
+		foreach ($objectIdChunks as $objectIdChunk) {
+			// Fetch only records from oc_comments which are in specified int[] NodeIDs array and satisfy specified $objectType
+			$qbMain->selectAlias('object_id', 'id')->selectAlias($qbMain->createFunction('COUNT(object_id)'), 'count')


>createFunction('COUNT(id)'), you created an alias. try using it in the count

mrow4a · 2017-03-12T18:13:18Z

Jenkins error?

butonic · 2017-03-12T19:03:18Z

The mysql docker seems dead?

mrow4a · 2017-03-13T11:45:02Z

@DeepDiver1975 @PVince81 Tests are passing, what do we do with it? Travis?

mrow4a · 2017-03-13T14:36:16Z

@DeepDiver1975 @PVince81 This PR is ready from my side. All checks passing.

jvillafanez · 2017-03-14T15:56:22Z

I haven't tested the code but it looks fine.

mrow4a · 2017-03-14T16:01:40Z

Do we have behavioural tests for comments? Better AI test it @jvillafanez @SamuAlfageme @DeepDiver1975

…quired data in one query instead of iterative queries.

mrow4a · 2017-03-20T08:28:40Z

@jvillafanez Any new input on this PR? Does CommentsApp have integration tests?

PVince81 · 2017-03-20T14:46:20Z

@mrow4a yes:

± % grep -lrie comment tests/integration 
tests/integration/config/behat.yml
tests/integration/features/bootstrap/CommentsContext.php
tests/integration/features/comments.feature
tests/integration/features/provisioning-v1.feature

PVince81 · 2017-03-20T14:52:42Z

Looks like the comments integration tests are only for the comments Webdav endpoint but not for the files endpoint.

Tests are missing for the property "oc:comments-count", "oc:comments-unread", "oc:comments-href".

@mrow4a please work with @SergioBertolinSG to get some tests for this added.
The tests should have users sharing a file, one user adding a comment and the other user retrieving the above properties on directly that file.
Then do the same with "Depth: 1" on the parent folder to test the pre-fetching code path.

You can run the integration tests with these commands:

% cd tests/integration
% sudo -u wwwrun ./run.sh features/comments.feature

(you can also add line number of the scenario "comments.feature:191".

SergioBertolinSG · 2017-03-21T08:05:53Z

@PVince81 @mrow4a Ok.

jvillafanez · 2017-03-22T10:20:54Z

No more changes from my side.

PVince81 · 2017-03-22T10:41:59Z

@SergioBertolinSG is working on comments tests separately, merging this.

👍

lock · 2019-08-03T17:02:18Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

mrow4a requested review from PVince81, butonic and DeepDiver1975 March 8, 2017 13:25

mrow4a commented Mar 8, 2017

View reviewed changes

mrow4a added the 2 - Developing label Mar 8, 2017

mrow4a force-pushed the comm_optimize_propfind branch 2 times, most recently from e5946ba to 4e4e05e Compare March 10, 2017 12:58

DeepDiver1975 requested changes Mar 10, 2017

View reviewed changes

mrow4a force-pushed the comm_optimize_propfind branch 2 times, most recently from 76d2686 to 4e84cd2 Compare March 10, 2017 23:35

mrow4a force-pushed the comm_optimize_propfind branch 2 times, most recently from 9488209 to 53bbd3e Compare March 12, 2017 12:00

mrow4a added 3 - To Review and removed 2 - Developing labels Mar 12, 2017

butonic reviewed Mar 12, 2017

View reviewed changes

mrow4a force-pushed the comm_optimize_propfind branch from 53bbd3e to 92516d6 Compare March 12, 2017 17:04

mrow4a force-pushed the comm_optimize_propfind branch from 92516d6 to 4689863 Compare March 13, 2017 09:50

mrow4a force-pushed the comm_optimize_propfind branch from 4689863 to 044f51f Compare March 13, 2017 12:37

mrow4a force-pushed the comm_optimize_propfind branch 5 times, most recently from de9fd29 to 260f724 Compare March 14, 2017 14:15

mrow4a added 2 - Developing and removed 3 - To Review labels Mar 15, 2017

mrow4a mentioned this pull request Mar 15, 2017

Depth infinity: Make recursive PROPFIND calls cheaper #14531

Closed

mrow4a force-pushed the comm_optimize_propfind branch 4 times, most recently from e807d9a to 9c7194c Compare March 16, 2017 11:51

Optimize PROPFIND having CommentsApp enabled by default - prefetch re…

be4309a

…quired data in one query instead of iterative queries.

mrow4a force-pushed the comm_optimize_propfind branch from 9c7194c to be4309a Compare March 16, 2017 11:58

mrow4a added 3 - To Review and removed 2 - Developing labels Mar 16, 2017

mrow4a mentioned this pull request Mar 16, 2017

Optimise getFileInfo - for mounts, dont fetch again filecache, but reuse sourceRootInfo #27403

Closed

2 tasks

PVince81 merged commit a50acb4 into master Mar 22, 2017

PVince81 deleted the comm_optimize_propfind branch March 22, 2017 10:42

SergioBertolinSG mentioned this pull request Mar 28, 2017

Modifying comments tests #27446

Merged

lock bot locked as resolved and limited conversation to collaborators Aug 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize PROPFIND having CommentsApp enabled by default #27339

Optimize PROPFIND having CommentsApp enabled by default #27339

mrow4a commented Mar 8, 2017 •

edited

Loading

mention-bot commented Mar 8, 2017

mrow4a commented Mar 8, 2017 •

edited

Loading

mrow4a Mar 8, 2017

mrow4a Mar 8, 2017

mrow4a commented Mar 10, 2017 •

edited

Loading

DeepDiver1975 Mar 10, 2017

DeepDiver1975 Mar 10, 2017

DeepDiver1975 Mar 10, 2017

mrow4a commented Mar 12, 2017 •

edited

Loading

butonic commented Mar 12, 2017

butonic commented Mar 12, 2017

mrow4a commented Mar 12, 2017

butonic commented Mar 12, 2017

butonic Mar 12, 2017

mrow4a commented Mar 12, 2017

butonic commented Mar 12, 2017

mrow4a commented Mar 13, 2017

mrow4a commented Mar 13, 2017

jvillafanez commented Mar 14, 2017

mrow4a commented Mar 14, 2017 •

edited

Loading

mrow4a commented Mar 20, 2017

PVince81 commented Mar 20, 2017

PVince81 commented Mar 20, 2017

SergioBertolinSG commented Mar 21, 2017

jvillafanez commented Mar 22, 2017

PVince81 commented Mar 22, 2017

lock bot commented Aug 3, 2019

Optimize PROPFIND having CommentsApp enabled by default #27339

Optimize PROPFIND having CommentsApp enabled by default #27339

Conversation

mrow4a commented Mar 8, 2017 • edited Loading

mention-bot commented Mar 8, 2017

mrow4a commented Mar 8, 2017 • edited Loading

mrow4a Mar 8, 2017

Choose a reason for hiding this comment

mrow4a Mar 8, 2017

Choose a reason for hiding this comment

mrow4a commented Mar 10, 2017 • edited Loading

DeepDiver1975 Mar 10, 2017

Choose a reason for hiding this comment

DeepDiver1975 Mar 10, 2017

Choose a reason for hiding this comment

DeepDiver1975 Mar 10, 2017

Choose a reason for hiding this comment

mrow4a commented Mar 12, 2017 • edited Loading

butonic commented Mar 12, 2017

butonic commented Mar 12, 2017

mrow4a commented Mar 12, 2017

butonic commented Mar 12, 2017

butonic Mar 12, 2017

Choose a reason for hiding this comment

mrow4a commented Mar 12, 2017

butonic commented Mar 12, 2017

mrow4a commented Mar 13, 2017

mrow4a commented Mar 13, 2017

jvillafanez commented Mar 14, 2017

mrow4a commented Mar 14, 2017 • edited Loading

mrow4a commented Mar 20, 2017

PVince81 commented Mar 20, 2017

PVince81 commented Mar 20, 2017

SergioBertolinSG commented Mar 21, 2017

jvillafanez commented Mar 22, 2017

PVince81 commented Mar 22, 2017

lock bot commented Aug 3, 2019

mrow4a commented Mar 8, 2017 •

edited

Loading

mrow4a commented Mar 8, 2017 •

edited

Loading

mrow4a commented Mar 10, 2017 •

edited

Loading

mrow4a commented Mar 12, 2017 •

edited

Loading

mrow4a commented Mar 14, 2017 •

edited

Loading