[spaces 2/2] /dav/spaces endpoint #1803

butonic · 2021-06-18T09:35:58Z

I split #1776 into two PRs:

the initial list spaces implementation
the new /dav/spaces endpoint to access storage with relative references

This is the second PR about spaces. It

adds a new ocdav handler for /dav/spaces
refactors the ocdav handlers to make them reusable for /dav/spaces and /(web)dav requests
adds logic to the gateway to forward relative References to the storage providers
adds logic to the storageprovider to only prefix mountpoints for absolute References
changes the CreateDir signature in the storage.FS interface to allow creating folders with relative References
adds a new datatx spaces implementation
update storage drivers to distinguish relative vs absolute References and at least handle absolute references for path and id only based access

Examples

When using ocis clients would use the /graph/v1.0/me/drives endpoint to ge a listing of the spaces/driles a user has access to (try using the /graph-explorer that comes with ocis):

{
    "value": [
        {
            "driveType": "personal",
            "id": "1284d238-aa92-42ce-bdc4-0b0000009157!1d2e72be-da48-42c5-b7b5-49eb8e5fa482",
            "lastModifiedDateTime": "2021-07-20T18:18:06.240248736Z",
            "name": "root",
            "owner": {
                "user": {
                    "id": "f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"
                }
            },
            "root": {
                "id": "1284d238-aa92-42ce-bdc4-0b0000009157!1d2e72be-da48-42c5-b7b5-49eb8e5fa482",
                "webDavUrl": "https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d2e72be-da48-42c5-b7b5-49eb8e5fa482"
            }
        },
        {
            "driveType": "share",
            "id": "1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142",
            "lastModifiedDateTime": "2021-07-21T08:40:41.887071968Z",
            "name": "from einstein",
            "owner": {
                "user": {
                    "id": "4c510ada-c86b-4815-8820-42cdf82c3d51"
                }
            },
            "root": {
                "id": "1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142",
                "webDavUrl": "https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142"
            }
        }
    ]
}

The webDavUrl properties can then be used to browse the space using normal webdav:

curl -X PROPFIND 'https://cloud.ocis.test/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142/' -u marie:radioactivity
<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns">
    <d:response>
        <d:href>/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142/</d:href>
        <d:propstat>
            <d:prop>
                <oc:id>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjFkODljMmY4LWE4MDAtNGUzYy05OTdkLWQ5NjlmMjlkZTE0Mg==</oc:id>
                <oc:fileid>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjFkODljMmY4LWE4MDAtNGUzYy05OTdkLWQ5NjlmMjlkZTE0Mg==</oc:fileid>
                <d:getetag>&#34;f8d8c4396d8ef37d5eaa0584cb4460ff&#34;</d:getetag>
                <oc:permissions>S</oc:permissions>
                <d:resourcetype>
                    <d:collection/>
                </d:resourcetype>
                <oc:size>345396172</oc:size>
                <d:getlastmodified>Wed, 21 Jul 2021 08:44:38 GMT</d:getlastmodified>
                <oc:favorite>0</oc:favorite>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
    <d:response>
        <d:href>/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142/ffmpeg-4.4-full_build/</d:href>
        <d:propstat>
            <d:prop>
                <oc:id>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjAxYzZkOWRkLThkMDgtNGIyNy1hMzZmLTEwMjIwZDdiOThlOA==</oc:id>
                <oc:fileid>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjAxYzZkOWRkLThkMDgtNGIyNy1hMzZmLTEwMjIwZDdiOThlOA==</oc:fileid>
                <d:getetag>&#34;7aadd18a32470de2d5371b7db813987f&#34;</d:getetag>
                <oc:permissions>S</oc:permissions>
                <d:resourcetype>
                    <d:collection/>
                </d:resourcetype>
                <oc:size>345396172</oc:size>
                <d:getlastmodified>Wed, 21 Jul 2021 08:44:38 GMT</d:getlastmodified>
                <oc:favorite>0</oc:favorite>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
</d:multistatus>

Navigating into subfolders happens using relative URLs:

curl -X PROPFIND 'https://cloud.ocis.test/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142/ffmpeg-4.4-full_build/bin/' -u marie:radioactivity
<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns">
    <d:response>
        <d:href>/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142/ffmpeg-4.4-full_build/bin/</d:href>
        <d:propstat>
            <d:prop>
                <oc:id>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjczNGM4NGQwLWNiYWItNGNhYS05YmQ3LWE1ZWYwYzM1ZTVhNA==</oc:id>
                <oc:fileid>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjczNGM4NGQwLWNiYWItNGNhYS05YmQ3LWE1ZWYwYzM1ZTVhNA==</oc:fileid>
                <d:getetag>&#34;5d7ef09c6250be66843b639ceb156349&#34;</d:getetag>
                <oc:permissions>S</oc:permissions>
                <d:resourcetype>
                    <d:collection/>
                </d:resourcetype>
                <oc:size>337036800</oc:size>
                <d:getlastmodified>Wed, 21 Jul 2021 08:44:38 GMT</d:getlastmodified>
                <oc:favorite>0</oc:favorite>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
    <d:response>
        <d:href>/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142/ffmpeg-4.4-full_build/bin/ffmpeg.exe</d:href>
        <d:propstat>
            <d:prop>
                <oc:id>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OmZhMjQzODVhLWVhM2UtNDQ3OC1hZTNlLWRlZTY2YjVlODcwMQ==</oc:id>
                <oc:fileid>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OmZhMjQzODVhLWVhM2UtNDQ3OC1hZTNlLWRlZTY2YjVlODcwMQ==</oc:fileid>
                <d:getetag>&#34;64259a52e69bcdd8ac8d707c4a234577&#34;</d:getetag>
                <oc:permissions>S</oc:permissions>
                <d:resourcetype></d:resourcetype>
                <d:getcontentlength>112424960</d:getcontentlength>
                <d:getcontenttype>application/x-msdos-program</d:getcontenttype>
                <d:getlastmodified>Fri, 09 Apr 2021 09:16:00 GMT</d:getlastmodified>
                <oc:checksums>
                    <oc:checksum>SHA1:82a787896f0d78a822b14dc3ec40dbf54d6fd049 MD5:544ff37775a43daaf30a7731fec87bfe ADLER32:abcf52ac</oc:checksum>
                </oc:checksums>
                <oc:favorite>0</oc:favorite>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
    <d:response>
        <d:href>/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142/ffmpeg-4.4-full_build/bin/ffprobe.exe</d:href>
        <d:propstat>
            <d:prop>
                <oc:id>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjA0MGY5Zjk0LWY3NjUtNDU1Ni1hYTI4LTk2NjdhNzUzNjA5NA==</oc:id>
                <oc:fileid>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OjA0MGY5Zjk0LWY3NjUtNDU1Ni1hYTI4LTk2NjdhNzUzNjA5NA==</oc:fileid>
                <d:getetag>&#34;c29362f5c52c4eb212c86714b13d2c16&#34;</d:getetag>
                <oc:permissions>S</oc:permissions>
                <d:resourcetype></d:resourcetype>
                <d:getcontentlength>112323072</d:getcontentlength>
                <d:getcontenttype>application/x-msdos-program</d:getcontenttype>
                <d:getlastmodified>Fri, 09 Apr 2021 09:16:00 GMT</d:getlastmodified>
                <oc:checksums>
                    <oc:checksum>SHA1:44104ae8b9872779566848a08370128ef21ea919 MD5:50877f3904a0ec73bb7fab5771527b2f ADLER32:ff40105e</oc:checksum>
                </oc:checksums>
                <oc:favorite>0</oc:favorite>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
    <d:response>
        <d:href>/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!1d89c2f8-a800-4e3c-997d-d969f29de142/ffmpeg-4.4-full_build/bin/ffplay.exe</d:href>
        <d:propstat>
            <d:prop>
                <oc:id>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OmVlYWMzMjlmLTU1NzYtNGY3Mi1iNzZlLWEwMmQzM2MzYzUzMA==</oc:id>
                <oc:fileid>MTI4NGQyMzgtYWE5Mi00MmNlLWJkYzQtMGIwMDAwMDA5MTU3OmVlYWMzMjlmLTU1NzYtNGY3Mi1iNzZlLWEwMmQzM2MzYzUzMA==</oc:fileid>
                <d:getetag>&#34;2fc643027b8cdaff07f0cd933bb188ff&#34;</d:getetag>
                <oc:permissions>S</oc:permissions>
                <d:resourcetype></d:resourcetype>
                <d:getcontentlength>112288768</d:getcontentlength>
                <d:getcontenttype>application/x-msdos-program</d:getcontenttype>
                <d:getlastmodified>Fri, 09 Apr 2021 09:16:00 GMT</d:getlastmodified>
                <oc:checksums>
                    <oc:checksum>SHA1:415cc678c4f24038f1b36d59e504611c743a940a MD5:19701fd1916d811072bbd72209fa8d72 ADLER32:7d7cd134</oc:checksum>
                </oc:checksums>
                <oc:favorite>0</oc:favorite>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
</d:multistatus>

TODO

move spaces migration code in decomposedfs tree to lis spaces PR
revert storageprovider wrap returning ctx, no longer needed -> move unwrapping and wrapping of paths to the gateway #1866
move unrelated ResourceEqual -> ResourceIDEqual changes to dedicated PR -> Simplify resource comparison #1876

update-docs · 2021-06-18T09:36:01Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

lgtm-com · 2021-06-18T10:16:15Z

This pull request introduces 1 alert and fixes 18 when merging a79de7f into 9740eed - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression

fixed alerts:

18 for Uncontrolled data used in path expression

lgtm-com · 2021-06-18T11:49:12Z

This pull request introduces 1 alert and fixes 18 when merging 9eb6ab6 into 9740eed - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression

fixed alerts:

18 for Uncontrolled data used in path expression

lgtm-com · 2021-06-22T07:43:53Z

This pull request introduces 1 alert and fixes 18 when merging a80e3f0 into 567ec47 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression

fixed alerts:

18 for Uncontrolled data used in path expression

lgtm-com · 2021-06-23T13:59:30Z

This pull request introduces 1 alert and fixes 18 when merging 683a307 into 3a9efaa - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression

fixed alerts:

18 for Uncontrolled data used in path expression

ishank011 · 2021-07-07T10:23:58Z

@butonic can you also add sample expected requests and responses in the description once this PR is finished?

lgtm-com · 2021-07-08T22:05:48Z

This pull request introduces 1 alert and fixes 18 when merging 7058dd1 into 7d21c54 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression

fixed alerts:

18 for Uncontrolled data used in path expression

pkg/rhttp/datatx/utils/download/download.go

lgtm-com · 2021-07-16T13:46:42Z

This pull request introduces 1 alert and fixes 18 when merging 490b0c8 into 7df477f - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression

fixed alerts:

18 for Uncontrolled data used in path expression

lgtm-com · 2021-07-19T09:55:49Z

This pull request introduces 1 alert and fixes 18 when merging ac604bc into 7df477f - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression

fixed alerts:

18 for Uncontrolled data used in path expression

butonic · 2021-07-26T15:36:13Z

@ishank011 I rebased, please review

lgtm-com · 2021-07-26T15:50:06Z

This pull request fixes 30 alerts when merging b26fa94 into e11a198 - view on LGTM.com

fixed alerts:

30 for Uncontrolled data used in path expression

ishank011 · 2021-07-27T10:33:33Z

@butonic I'm not sure about having the new protocol in datatx since it's not exactly a protocol. I think we can add the relavant checks in tus/simple and frame the requests accordingly. Also, won't spaces support TUS?

butonic · 2021-08-10T14:50:44Z

@ishank011 @labkode rebased and adressed review. Good to merge?

lgtm-com · 2021-08-10T15:05:00Z

This pull request fixes 30 alerts when merging 29c03b3 into b317948 - view on LGTM.com

fixed alerts:

30 for Uncontrolled data used in path expression

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

lgtm-com · 2021-08-11T18:48:02Z

This pull request fixes 30 alerts when merging c13d9b1 into ec4099d - view on LGTM.com

fixed alerts:

30 for Uncontrolled data used in path expression

ishank011

@butonic couple of minor comments, we can merge then.

internal/grpc/services/storageprovider/storageprovider.go

internal/http/services/owncloud/ocdav/put.go

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

lgtm-com · 2021-08-12T10:31:50Z

This pull request fixes 30 alerts when merging f0ecf47 into 04a1cf2 - view on LGTM.com

fixed alerts:

30 for Uncontrolled data used in path expression

butonic mentioned this pull request Jun 18, 2021

[spaces 1/2] initial ListStorageSpaces implementation #1802

Merged

butonic force-pushed the dav-spaces branch from 13c2c9d to 21f6901 Compare June 18, 2021 09:44

C0rby force-pushed the dav-spaces branch from ae68291 to a79de7f Compare June 18, 2021 09:53

butonic force-pushed the dav-spaces branch from a79de7f to 9eb6ab6 Compare June 18, 2021 11:30

butonic mentioned this pull request Jun 21, 2021

[WIP] introduce /dav/spaces endpoint && CS3 list spaces implementation (ocis driver only) #1776

Closed

C0rby force-pushed the dav-spaces branch from 9eb6ab6 to a80e3f0 Compare June 22, 2021 07:19

butonic force-pushed the dav-spaces branch from 683a307 to ce7845b Compare July 8, 2021 21:37

C0rby reviewed Jul 9, 2021

View reviewed changes

pkg/rhttp/datatx/utils/download/download.go Show resolved Hide resolved

butonic mentioned this pull request Jul 9, 2021

graph refactorings owncloud/ocis#2277

Merged

butonic force-pushed the dav-spaces branch from 02c46f3 to 171d142 Compare July 13, 2021 11:45

C0rby mentioned this pull request Jul 13, 2021

Refactor dav #1888

Merged

C0rby force-pushed the dav-spaces branch from 171d142 to 490b0c8 Compare July 16, 2021 13:28

C0rby force-pushed the dav-spaces branch 2 times, most recently from da7108b to ac604bc Compare July 19, 2021 09:37

butonic marked this pull request as ready for review July 21, 2021 08:48

butonic requested a review from labkode as a code owner July 21, 2021 08:48

butonic requested a review from ishank011 July 21, 2021 08:48

butonic force-pushed the dav-spaces branch from ac604bc to b26fa94 Compare July 26, 2021 15:35

butonic force-pushed the dav-spaces branch from 4385f05 to 29c03b3 Compare August 10, 2021 14:49

butonic requested a review from ishank011 August 10, 2021 21:41

butonic force-pushed the dav-spaces branch from 29c03b3 to c13d9b1 Compare August 11, 2021 18:33

butonic and others added 13 commits August 11, 2021 18:35

add /dav/spaces endpoint

f452a1e

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

remove context from unwrap response

1a1c3bf

fix relative reference handling in storageprovider.CreateContainer

bbb4e0e

fix dav spaces copy

66ebd99

add litmus tests for dav spaces api

5ae683c

add changelog

fb0486d

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

add util function to split storage space id

e729ace

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

new IsAbsolutePathReference check

f53bb0d

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

fix logic & logging

9c18618

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

resolve merge conflicts

33f5521

fix rebase

83975c2

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

incorporate review

3ee5e6f

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

call CreateDir without splitting name from ref

c13d9b1

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic requested review from C0rby, refs and dragotin August 11, 2021 20:37

ishank011 reviewed Aug 12, 2021

View reviewed changes

internal/grpc/services/storageprovider/storageprovider.go Outdated Show resolved Hide resolved

internal/http/services/owncloud/ocdav/put.go Show resolved Hide resolved

ishank011 removed the under review: don't merge label Aug 12, 2021

unwrap and spaces cleanup

f0ecf47

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

ishank011 approved these changes Aug 12, 2021

View reviewed changes

ishank011 merged commit 756bdce into cs3org:master Aug 12, 2021

butonic deleted the dav-spaces branch August 12, 2021 10:56

butonic mentioned this pull request Aug 12, 2021

[full-ci] update reva to 756bdced1d22 owncloud/ocis#2385

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[spaces 2/2] /dav/spaces endpoint #1803

[spaces 2/2] /dav/spaces endpoint #1803

butonic commented Jun 18, 2021 •

edited

Loading

update-docs bot commented Jun 18, 2021

lgtm-com bot commented Jun 18, 2021

lgtm-com bot commented Jun 18, 2021

lgtm-com bot commented Jun 22, 2021

lgtm-com bot commented Jun 23, 2021

ishank011 commented Jul 7, 2021

lgtm-com bot commented Jul 8, 2021

lgtm-com bot commented Jul 16, 2021

lgtm-com bot commented Jul 19, 2021

butonic commented Jul 26, 2021

lgtm-com bot commented Jul 26, 2021

ishank011 commented Jul 27, 2021

butonic commented Aug 10, 2021

lgtm-com bot commented Aug 10, 2021

lgtm-com bot commented Aug 11, 2021

ishank011 left a comment

lgtm-com bot commented Aug 12, 2021

[spaces 2/2] /dav/spaces endpoint #1803

[spaces 2/2] /dav/spaces endpoint #1803

Conversation

butonic commented Jun 18, 2021 • edited Loading

Examples

TODO

update-docs bot commented Jun 18, 2021

lgtm-com bot commented Jun 18, 2021

lgtm-com bot commented Jun 18, 2021

lgtm-com bot commented Jun 22, 2021

lgtm-com bot commented Jun 23, 2021

ishank011 commented Jul 7, 2021

lgtm-com bot commented Jul 8, 2021

lgtm-com bot commented Jul 16, 2021

lgtm-com bot commented Jul 19, 2021

butonic commented Jul 26, 2021

lgtm-com bot commented Jul 26, 2021

ishank011 commented Jul 27, 2021

butonic commented Aug 10, 2021

lgtm-com bot commented Aug 10, 2021

lgtm-com bot commented Aug 11, 2021

ishank011 left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Aug 12, 2021

butonic commented Jun 18, 2021 •

edited

Loading