quota querying and tree accounting #1405

butonic · 2021-01-18T21:12:47Z

The ocs api now returns the user quota for the users home storage. Furthermore, the ocis storage driver now reads the quota from the extended attributes of the user home or root node and implements tree size accounting. Finally, ocdav PROPFINDS now handle the DAV:quota-used-bytes and DAV:quote-available-bytes properties.

~~check uploads against the quota~~ in a different PR, we need add delete home to the storage provider cs3apis#95 before continuing with anything quota related.

relvated: owncloud/ocis#1313

update-docs · 2021-01-18T21:12:50Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

lgtm-com · 2021-02-01T10:34:28Z

This pull request fixes 4 alerts when merging 99864e4 into d919d59 - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

butonic · 2021-02-02T19:47:10Z

the only api that end users can use to access the free and used space of a storage is through webdav rfc 4331

lgtm-com · 2021-02-03T12:55:22Z

This pull request fixes 4 alerts when merging 60e4c47 into ab3be6b - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

lgtm-com · 2021-02-03T15:40:16Z

This pull request fixes 4 alerts when merging 790a8c7 into ffb0fb4 - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

lgtm-com · 2021-02-03T16:17:22Z

This pull request fixes 4 alerts when merging bb1c3f9 into 82be569 - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

lgtm-com · 2021-02-03T17:46:29Z

This pull request fixes 4 alerts when merging 6f484a8 into 6f07395 - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

lgtm-com · 2021-02-03T19:37:15Z

This pull request fixes 4 alerts when merging 0336464 into 6f07395 - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

lgtm-com · 2021-02-03T20:23:48Z

This pull request fixes 4 alerts when merging 9326e3b into 6f07395 - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

lgtm-com · 2021-02-03T20:53:31Z

This pull request fixes 4 alerts when merging 0c6c3d4 into 6f07395 - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

butonic · 2021-02-09T11:01:58Z

To test this using ocis

replace reva in ocis/ocis/go.mod and
make the proxy forward all ocs requests to reva:

diff --git a/proxy/pkg/proxy/proxy.go b/proxy/pkg/proxy/proxy.go
index a6c06bc0..8e2b0c76 100644
--- a/proxy/pkg/proxy/proxy.go
+++ b/proxy/pkg/proxy/proxy.go
@@ -284,11 +284,13 @@ func defaultPolicies() []config.Policy {
                                        Endpoint: "/signin/",
                                        Backend:  "http://localhost:9130",
                                },
-                               {
-                                       Type:     config.RegexRoute,
-                                       Endpoint: "/ocs/v[12].php/cloud/(users?|groups)", // we have `user`, `users` and `groups` in ocis-ocs
-                                       Backend:  "http://localhost:9110",
-                               },
+                               /*
+                                       {
+                                               Type:     config.RegexRoute,
+                                               Endpoint: "/ocs/v[12].php/cloud/(users?|groups)", // we have `user`, `users` and `groups` in ocis-ocs
+                                               Backend:  "http://localhost:9110",
+                                       },
+                               */
                                {
                                        Endpoint: "/ocs/",
                                        Backend:  "http://localhost:9140",

set a quota on a users home by using this one weird extended attribute trick:
setfattr -n user.ocis.quota -v 2024000 /var/tmp/ocis/storage/users/nodes/root/f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c/ which should set the attribute on the linked directory node, not the symlink.

Now test fetching the quota using ocs:

curl --location --request GET 'https://demo.owncloud.com/ocs/v1.php/cloud/users/demo' \
--header 'ocs-apirequest: true' \
--header 'Authorization: Basic ZGVtbzpkZW1v'
<?xml version="1.0"?>
<ocs>
    <meta>
        <status>ok</status>
        <statuscode>100</statuscode>
        <message/>
    </meta>
    <data>
        <quota>
            <free>25953636352</free>
            <used>7831162</used>
            <total>25961467514</total>
            <relative>0.03</relative>
            <definition>default</definition>
        </quota>
        <email/>
        <displayname>demo</displayname>
        <home>/var/lib/owncloud/files/demo</home>
        <two_factor_auth_enabled>false</two_factor_auth_enabled>
    </data>
</ocs>

or using propfinds:

curl --location --request PROPFIND 'https://demo.owncloud.com/remote.php/dav/files/demo' \
--header 'Authorization: Basic ZGVtbzpkZW1v' \
--header 'Content-Type: text/plain' \
--header 'Cookie: cookie_test=test; oc_sessionPassphrase=9u%2FvsPAzRkzHqPl6tDokV6uFQiSn9MsSOonY1r0IsJhlym1d%2FOYxK1NOIHQ%2F%2FLtM%2BYV6N5fZuJg%2FyK40W4QEqrm%2BFLWeDA6ZZmbgvLuWKseRzZZ%2FT0VP6PXAa1kjV1hT; ocxahbm4e6a4=mmdhqh82puj3bcm38pus26ac3j' \
--data-raw '<?xml version="1.0"?>
<d:propfind  xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
<d:prop>
<oc:permissions />
<oc:favorite />
<oc:fileid />
<oc:owner-id />
<oc:owner-display-name />
<oc:share-types />
<oc:privatelink />
<d:getcontentlength />
<oc:size />
<d:getlastmodified />
<d:getetag />
<d:resourcetype />
<oc:public-link-item-type />
<oc:public-link-permission />
<oc:public-link-expiration />
<oc:public-link-share-datetime />
<oc:public-link-share-owner />
<oc:downloadURL />
<d:quota-used-bytes />
<d:quota-available-bytes />
  </d:prop>
</d:propfind>'

kulmann · 2021-02-10T06:26:10Z

For the sake of completeness, the command for setting the quota on a mac is:
xattr -w user.ocis.quota 2024000 /var/tmp/ocis/storage/users/nodes/root/f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c/
(that should write to the node, not the symlink. And additional -s would be required for writing to the symlink instead)

kulmann · 2021-02-10T06:34:13Z

I can confirm the results with the instructions from above.

<?xml version="1.0" encoding="utf-8"?>
<ocs>
  <meta>
    <status>ok</status>
    <statuscode>100</statuscode>
    <message>OK</message>
  </meta>
  <data>
    <quota>
      <free>1799840</free>
      <used>224160</used>
      <total>2024000</total>
      <relative>0.11075099</relative>
      <definition>default</definition>
    </quota>
    <email>einstein@example.org</email>
    <displayname>Albert Einstein</displayname>
    <two_factor_auth_enabled>false</two_factor_auth_enabled>
  </data>
</ocs>

kulmann · 2021-02-10T06:47:17Z

pkg/storage/fs/ocis/ocis.go

+		return 0, 0, err
+	}
+
+	total := stat.Blocks * uint64(stat.Bsize) // Total data blocks in filesystem


Shouldn't total be the available disk space? 🤔 at least from a user perspective it's misleading that with unlimited quota the response will tell that (on my machine) 495GB are free, while my actual free disk space is at 195GB.

uh ... that is weird. maybe the total is calculated differently on mac? what does this look like on mac:

> stat . File: . Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 820h/2080d Inode: 420580 Links: 29 Access: (0755/drwxr-xr-x) Uid: ( 1000/ vscode) Gid: ( 1000/ vscode) Access: 2021-02-09 20:43:26.250000000 +0000 Modify: 2021-02-09 20:43:26.100000000 +0000 Change: 2021-02-09 20:43:26.100000000 +0000 Birth: -

macOS is weird for stat... being in the ocis user home dir of einstein:

> stat -x . File: "." Size: 128 FileType: Directory Mode: (0700/drwx------) Uid: ( 501/ bene) Gid: ( 0/ wheel) Device: 1,5 Inode: 62850061 Links: 4 Access: Wed Feb 10 07:33:06 2021 Modify: Wed Feb 10 07:33:06 2021 Change: Wed Feb 10 09:56:26 2021

> stat . 16777221 62850061 drwx------ 4 bene wheel 0 128 "Feb 10 07:33:06 2021" "Feb 10 07:33:06 2021" "Feb 10 09:56:26 2021" "Feb 10 07:23:36 2021" 4096 0 0 .

ok, what about filesystem stat and df:

> stat -f . File: "." ID: fedc9aa3bd65bc57 Namelen: 255 Type: ext2/ext3 Block size: 4096 Fundamental block size: 4096 Blocks: Total: 65793553 Free: 58593039 Available: 55233500 Inodes: Total: 16777216 Free: 16011441 > df -B4K Filesystem 4K-blocks Used Available Use% Mounted on overlay 65793553 7200514 55233500 12% / tmpfs 16384 0 16384 0% /dev tmpfs 2046465 0 2046465 0% /sys/fs/cgroup shm 16384 0 16384 0% /dev/shm /dev/sdc 65793553 7200514 55233500 12% /etc/hosts drvfs 31096934 19848635 11248299 64% /home/vscode/.ssh

macOS doesn't have stat for the entire filesystem AFAICT.

> BLOCKSIZE=4K df Filesystem 4K-blocks Used Available Capacity iused ifree %iused Mounted on /dev/disk1s5s1 122061322 4756743 34314966 13% 568975 4881883905 0% / devfs 47 47 0 100% 658 0 100% /dev /dev/disk1s4 122061322 524298 34314966 2% 2 4882452878 0% /System/Volumes/VM /dev/disk1s2 122061322 80331 34314966 1% 792 4882452088 0% /System/Volumes/Preboot /dev/disk1s6 122061322 27269 34314966 1% 396 4882452484 0% /System/Volumes/Update /dev/disk1s1 122061322 82168442 34314966 71% 4246575 4878206305 0% /System/Volumes/Data map auto_home 0 0 0 100% 0 0 100% /System/Volumes/Data/home /dev/disk1s5 122061322 4756743 34314966 13% 568977 4881883903 0% /System/Volumes/Update/mnt1

ah now I get it ... the switch case in https://github.com/cs3org/reva/pull/1405/files#diff-62fe53c34a46cf0bf6e8f27409f0ba1ba6b0687387b8c9cde985299ba10b839fR229-R240 should limit the total to the quota if the latter is set. the free space is then calculated in the ocs / ocdav handler by calculating free = total-used. The problem is that we currently cannot return 'unlimited' quota. It would be a dedicated flag because currently quota is an unsigned int...

actually, if no quote is set we should return free+used as the total, because other users files also count against the total.

lgtm-com · 2021-02-10T20:02:37Z

This pull request fixes 4 alerts when merging d080ab3 into 7a9a5ef - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

kulmann

👍

butonic · 2021-02-18T08:44:52Z

@ishank011 can you test on eos?

ishank011

@butonic Looks good, a few minor nits. And works for EOS, nice!

internal/grpc/services/gateway/storageprovider.go

pkg/storage/utils/localfs/localfs.go

pkg/storage/fs/ocis/ocis.go

internal/http/services/owncloud/ocs/handlers/cloud/users/users.go

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

lgtm-com · 2021-02-18T12:56:10Z

This pull request fixes 4 alerts when merging 3f69dc2 into 8a29177 - view on LGTM.com

fixed alerts:

4 for Incorrect conversion between integer types

butonic requested a review from labkode as a code owner January 18, 2021 21:12

butonic marked this pull request as draft January 18, 2021 21:17

butonic force-pushed the quota branch from 7851535 to 99864e4 Compare February 1, 2021 10:15

butonic force-pushed the quota branch from 99864e4 to 60e4c47 Compare February 3, 2021 12:41

ishank011 force-pushed the master branch 2 times, most recently from 71c6a47 to 48f10db Compare February 3, 2021 14:23

butonic force-pushed the quota branch from 60e4c47 to 790a8c7 Compare February 3, 2021 15:27

butonic force-pushed the quota branch from 790a8c7 to bb1c3f9 Compare February 3, 2021 16:03

butonic force-pushed the quota branch from bb1c3f9 to 6f484a8 Compare February 3, 2021 17:33

exalate-issue-sync bot mentioned this pull request Feb 3, 2021

[ocis] Size Tree accounting - calculated folder sizes owncloud/ocis#1150

Closed

butonic force-pushed the quota branch from 6f484a8 to 0336464 Compare February 3, 2021 19:21

butonic marked this pull request as ready for review February 3, 2021 20:34

butonic force-pushed the quota branch from 9326e3b to 0c6c3d4 Compare February 3, 2021 20:40

butonic changed the title ~~initial quota support~~ quota querying and tree accounting Feb 3, 2021

butonic requested a review from ishank011 February 3, 2021 20:53

kulmann reviewed Feb 10, 2021

View reviewed changes

kulmann previously approved these changes Feb 10, 2021

View reviewed changes

ishank011 reviewed Feb 18, 2021

View reviewed changes

butonic dismissed kulmann’s stale review via b7564d3 February 18, 2021 12:36

butonic added 9 commits February 18, 2021 12:38

initial quota support

aa0e980

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

add quota-availabe/used-bytes property

03b3d2c

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

implement tree size propagation in ocis

3487a09

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

remove explicit treesize handling

22f383e

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

add changelog

2f1c97e

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

fix rebase

180095f

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

update expected failures, continue propagating on minor errors

0d49304

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

calculate total available as available + used

4795dca

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

apply feedback

3f69dc2

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic force-pushed the quota branch from b7564d3 to 3f69dc2 Compare February 18, 2021 12:38

ishank011 approved these changes Feb 18, 2021

View reviewed changes

ishank011 merged commit a954e72 into cs3org:master Feb 18, 2021

butonic deleted the quota branch February 18, 2021 19:46

C0rby mentioned this pull request Feb 19, 2021

update reva to v1.6.1-0.20210218130854-a954e7242e7a owncloud/ocis#1684

Merged

butonic mentioned this pull request Mar 1, 2021

Quota query owncloud/ocis#1313

Closed

micbar mentioned this pull request Mar 9, 2021

Release 1.3.0 Tech Preview owncloud/ocis#1775

Closed

16 tasks

C0rby mentioned this pull request Mar 26, 2021

update reva to v1.6.1-0.20210326165326-e8a00d9b2368 owncloud/ocis#1861

Merged

fschade mentioned this pull request Mar 29, 2021

Release 1.4.0 Tech Preview owncloud/ocis#1866

Closed

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

quota querying and tree accounting #1405

quota querying and tree accounting #1405

butonic commented Jan 18, 2021 •

edited

Loading

update-docs bot commented Jan 18, 2021

lgtm-com bot commented Feb 1, 2021

butonic commented Feb 2, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

butonic commented Feb 9, 2021 •

edited

Loading

kulmann commented Feb 10, 2021 •

edited

Loading

kulmann commented Feb 10, 2021

kulmann Feb 10, 2021 •

edited

Loading

butonic Feb 10, 2021 •

edited

Loading

kulmann Feb 10, 2021

butonic Feb 10, 2021 •

edited

Loading

kulmann Feb 10, 2021

butonic Feb 10, 2021

butonic Feb 10, 2021

lgtm-com bot commented Feb 10, 2021

kulmann left a comment

butonic commented Feb 18, 2021 •

edited

Loading

ishank011 left a comment

lgtm-com bot commented Feb 18, 2021

quota querying and tree accounting #1405

quota querying and tree accounting #1405

Conversation

butonic commented Jan 18, 2021 • edited Loading

update-docs bot commented Jan 18, 2021

lgtm-com bot commented Feb 1, 2021

butonic commented Feb 2, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

lgtm-com bot commented Feb 3, 2021

butonic commented Feb 9, 2021 • edited Loading

kulmann commented Feb 10, 2021 • edited Loading

kulmann commented Feb 10, 2021

kulmann Feb 10, 2021 • edited Loading

Choose a reason for hiding this comment

butonic Feb 10, 2021 • edited Loading

Choose a reason for hiding this comment

kulmann Feb 10, 2021

Choose a reason for hiding this comment

butonic Feb 10, 2021 • edited Loading

Choose a reason for hiding this comment

kulmann Feb 10, 2021

Choose a reason for hiding this comment

butonic Feb 10, 2021

Choose a reason for hiding this comment

butonic Feb 10, 2021

Choose a reason for hiding this comment

lgtm-com bot commented Feb 10, 2021

kulmann left a comment

Choose a reason for hiding this comment

butonic commented Feb 18, 2021 • edited Loading

ishank011 left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Feb 18, 2021

butonic commented Jan 18, 2021 •

edited

Loading

butonic commented Feb 9, 2021 •

edited

Loading

kulmann commented Feb 10, 2021 •

edited

Loading

kulmann Feb 10, 2021 •

edited

Loading

butonic Feb 10, 2021 •

edited

Loading

butonic Feb 10, 2021 •

edited

Loading

butonic commented Feb 18, 2021 •

edited

Loading