Steps to install #14

tompinzler · 2018-10-21T20:32:43Z

Sorry if this is a stupid question!

I'm on Debian 9 and did
pip install git+https://github.com/matrix-org/synapse-s3-storage-provider.git

I add the following configuration to /etc/matrix-synapse/homeserver.yaml

media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
   bucket: matrix

Then I did a
systemctl restart matrix-synapse

In syslog I see

matrix python[6899]: ImportError: No module named s3_storage_provider

Tow questions:

How do I have to install synapse-s3-storage-provider so Matrix can find/use it?
How/Where do I configure my S3 credentials?

The text was updated successfully, but these errors were encountered:

abeluck · 2018-11-21T13:58:45Z

Also looking for an answer to this!

Sharparam · 2019-03-13T17:03:24Z

Late to the party but could be useful to others finding this issue: It seems it will pick up credentials that have previously been saved with the AWS CLI tool.

I installed it in the same virtual env I use for synapse and authenticated with the tokens for the synapse-specific IAM I made in the AWS console.

For system-installed Synapse I'm not sure what the process would be.

Edit: And also the steps outlined in the boto3 docs which I realised are actually linked in the readme of this project.

Edit 2: Also for Synapse to find the tool I just installed this project via pip while I had the virtual env for synapse active.

bradinator · 2019-03-15T15:10:04Z

Steps to install would be great.
I'm having trouble getting it all working.

rkfg · 2019-06-03T09:34:38Z

I think you've installed the storage provider for Python 2 while Synapse uses Python 3. I bumped into the same issue. Install pip for Python 3: apt install python3-pip then do pip3 install git+https://github.com/matrix-org/synapse-s3-storage-provider.git and proceed with the README instructions. I did that and it works fine it seems, I can see the new files in the bucket.

bradinator · 2019-06-04T14:40:54Z

@rkfg thanks for that. I'm probably a step closer now.
However, I'm still running into an issue anytime I update the config to include the storage provider and restart synapse, it just continually crashes until I remove those lines again.

rkfg · 2019-06-04T14:47:48Z

You should check the logs then (like /var/log/daemon.log), maybe you miss a module.

bradinator · 2019-07-02T12:27:25Z

I checked for logs, @rkfg i don't find a daemon.log but this is from my homeserver.log This seems to be the issues, but i don't know what to make of it.

`2019-07-02 11:13:25,582 - twisted - 242 - INFO - - Received SIGTERM, shutting down.
2019-07-02 11:13:25,582 - synapse.handlers.presence - 244 - INFO - presence.on_shutdown-0- Performing _on_shutdown. Persisting 25 unpersisted changes
2019-07-02 11:13:25,590 - synapse.handlers.presence - 253 - INFO - presence.on_shutdown-0- Finished _on_shutdown
2019-07-02 11:13:25,591 - synapse.http.site - 203 - WARNING - GET-8991- Error processing request <XForwardedForRequest at 0x7fc75566b8d0 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722238_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.
2019-07-02 11:13:25,592 - synapse.http.site - 203 - WARNING - GET-8977- Error processing request <XForwardedForRequest at 0x7fc748443cc0 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722238_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.

2019-07-02 11:13:25,597 - twisted - 242 - INFO - - (TCP Port 8008 Closed)
2019-07-02 11:13:25,597 - synapse.http.site - 203 - WARNING - GET-9003- Error processing request <XForwardedForRequest at 0x7fc729d3b6d8 method='GET' uri='/_matrix/client/r0/sync?filter=1&timeout=30000&since=s168684_2722261_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.
2019-07-02 11:13:25,598 - synapse.http.site - 203 - WARNING - GET-8979- Error processing request <XForwardedForRequest at 0x7fc729b02400 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722238_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.

2019-07-02 11:13:25,600 - twisted - 242 - INFO - - (TCP Port 8008 Closed)
2019-07-02 11:13:25,600 - synapse.http.site - 203 - WARNING - GET-9011- Error processing request <XForwardedForRequest at 0x7fc729a53668 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722261_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.

2019-07-02 11:13:25,606 - twisted - 242 - INFO - - (TCP Port 8448 Closed)
2019-07-02 11:13:25,607 - synapse.http.site - 203 - WARNING - GET-8969- Error processing request <XForwardedForRequest at 0x7fc74811dbe0 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722238_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.
2019-07-02 11:13:25,617 - twisted - 242 - INFO - - Main loop terminated.`

rkfg · 2019-07-02T12:29:14Z

Does your Synapse work fine without this S3 backend? Post the full log, this part only says it's got SIGTERM and terminates but it's not clear why.

bradinator · 2019-07-02T12:47:01Z

Yes, synapse works fine without the s3 backend. As soon as I change out the storage provider lines in the homeserver.yaml and restart, clients can't connect, the status indicates that it continues to attempt restarting.

The full log is 300k lines, so I was trying to only post from the timestamp I restarted, to when I restarted with no s3 backend. How much of the log should i provide?

rkfg · 2019-07-02T12:49:51Z

Just zip it up and attach here. Or post more lines because the first line in your snippet says that Synapse received SIGTERM (which is an external event like stopping the systemd unit) and the rest looks like cleaning up from that. I don't see anything related to its startup.

rkfg · 2019-07-02T13:46:28Z

I don't see anything unusual there. Can you produce a clean startup log by removing whatever log files you already have and starting Synapse with S3 enabled?

bradinator · 2019-07-02T14:42:34Z

Steps I tried to grab a better log:

Replace homeser.yaml with one pointing to S3
Clear Log
Restart Matrix
See that it isn't starting correctly,
Download logs provided below.

homeserver2.zip

rkfg · 2019-07-02T14:47:07Z

Failed to insert client IP

This is a weird error, I don't have it. Does it happen when Synapse works without S3? Is the only difference between those configs you swap S3 being enabled and not? Also, post that config (without passwords and such of course). Maybe the problem is there.

bradinator · 2019-07-02T15:03:28Z

Yes, I get the Failed to insert client IP while synapse is working properly without S3.

Everything else is identical.

# Directory where uploaded images and attachments are stored.
##
#media_store_path: "/var/lib/matrix-synapse/media"

media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
    bucket: riot

# Directory where in-progress uploads are stored.
#
uploads_path: "/var/lib/matrix-synapse/uploads"

rkfg · 2019-07-02T15:30:03Z

Here's mine:

media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_synchronous: False
  config:
    bucket: "synapse-bc"
    access_key_id: "key_id"
    secret_access_key: "access_key"
    storage_class: "INTELLIGENT_TIERING"

I suppose you have access_key_id and secret_access_key parameters set. How do you run Synapse? Do you use Systemd/SysV or anything like that (if yes, check the system logs not just Synapse's own logs; some errors are not visible there)? Have you tried to run it manually (using the same command the system uses + sudo to run as matrix-synapse user)?

bradinator · 2019-07-03T23:15:35Z

Does this mean anything useful? As the matrix-synapse user, I went in to python3 help, and found the module "s3_storage_provider". But when I give the command for help on that module, i see the below message.

help> s3_storage_provider
problem in s3_storage_provider - ModuleNotFoundError: No module named 'synapse'

rkfg · 2019-07-04T05:53:57Z

Could be a path problem then. Where's your Synapse installed to (path)? I use the official apt repository, it was a bit tricky to install this S3 module because they provide their own python and libraries with Synapse.

bradinator · 2019-07-09T02:50:07Z

I used this guide when installing, so I used apt to install. https://www.natrius.eu/dokuwiki/doku.php?id=digital:server:matrixsynapse

rkfg · 2019-07-09T05:49:04Z

Are you using matrix-synapse-py3 or matrix-synapse? The former comes with its own Python and libraries so you should use pip from there to install the S3 provider properly.

HelderFSFerreira · 2020-05-20T11:07:34Z

What about the install process in docker?
As the package is not installed on the official synapse docker image, there is an alternative or a PR to the official docker image should be made?

ShadowJonathan · 2020-09-02T13:50:41Z

What about the install process in docker?

I also really wanna know this, as I handle a personal synapse server right now, and after adding a whatsapp bridge, the media store started bloating significantly, i'd want to move that to b2, and do the same with other synapse servers in the future

ShadowJonathan · 2020-09-02T13:53:27Z

After I looked at the readme real quick, i noticed the only "installation" step:

The s3_storage_provider.py should be on the PYTHONPATH when starting synapse.

This means that doing something like mounting or placing it in the "/data" folder (if that's the working directory of synapse while running) would be enough to "install" it, though i think i'd be ignoring circumstances and dependencies at that point

iokiwi · 2020-09-28T08:31:29Z

@ShadowJonathan @CaptainYukinoshitaHachiman I had this question too. I think the best option is to build and image based on the official matrix/synapse image and install the synapse-s3-storage-provider during the image build. Doing a pip install . inside the repository should build, install and add synapse-s3-storage-provider to the expected path(s).

It should be almost as simple as either cloning or copying the repository into the image, entering the folder and doing a pip install .

Eg. Cloning

FROM matrixdotorg/synapse:latest

RUN git clone https://github.com/matrix-org/synapse-s3-storage-provider.git && \
        cd /synapse-s3-storage-provider && \
        pip install .

E.g COPY

FROM matrixdotorg/synapse:latest

COPY synapse-s3-storage-provider /synapse-s3-storage-provider

RUN cd /synapse-s3-storage-provider && \
        pip install .

In practice you may need to also install some missing libraries / dependencies in order for pip to be able to install synapse-s3-storage-provider..

Here's a Dockerfile i've been using to test an Openstack Swift Storage provider I am trying to build based off the AWS S3 storage provider.

FROM matrixdotorg/synapse:latest

# COPY synapse-swift-storage-provider /synapse-swift-storage-provider

RUN apk update && \
    apk add --no-cache \
      g++ \
      libffi-dev \
      git \
      python3-dev && \
    pip install --upgrade pip && \
    git clone https://github.com/iokiwi/synapse-swift-storage-provider.git && \
    cd /synapse-swift-storage-provider && \
    pip install . && \
    mkdir -p  /root/.config/openstack && \
    apk del \
        git \
        g++ \
        libffi-dev \
        python3-dev && \
    rm -rf /var/cache/apk/* && \
    rm -rf /tmp/*

# COPY clouds.yaml /root/.config/openstack/

Note I had to install a lot of extras (mostly for the openstacksdk). Unsure about dependencies for boto3 and botocore sorry.

Further installing extra depencies, I'd also recommend uninstalling as many of the dependencies as you can at the end of the build process to reduce the weight of the image and reduce the number of unnecessary binaries included.

jimmybrancaccio · 2020-11-10T22:31:26Z

I was able to get this working by building my own image.

FROM matrixdotorg/synapse:latest

RUN apt -y update \
    && apt -y upgrade \
    && apt -y install g++ libffi-dev git python3-dev python3-pip \
    && pip install --upgrade pip \
    && git clone https://github.com/iokiwi/synapse-swift-storage-provider.git \
    && cd /synapse-swift-storage-provider \
    && pip install . \
    && pip3 install git+https://github.com/matrix-org/synapse-s3-storage-provider.git

and my homeserver.yml:

media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
    bucket: my.bucketname.com
    region_name: us-east-1
    access_key_id: key-id-here
    secret_access_key: key-here

You may need to adjust the region_name depending on the region of your S3 bucket.

wioxjk · 2021-02-25T10:57:51Z

Why is this module not included in the .deb package yet?

inaniyants · 2021-05-24T16:00:18Z

Hi! In case you have missed, this package was published to pip registry on May 18, 2021
https://pypi.org/project/synapse-s3-storage-provider/
But this package publication is not automated (not in sync with this repo), so use next on your own risk.

So now you could install it by simply running
pip install synapse-s3-storage-provider.

This means, that starting from now you don't need to install git via apk, or apt
Here is working Dockerfile

FROM matrixdotorg/synapse:v1.34.0

RUN pip install synapse-s3-storage-provider

Ibrahim-AbouElseoud · 2021-07-21T20:14:56Z

For those using Yunohost to install this module, I would like to add the following steps which had worked for me:

SSH to your yunohost server
Login as root or dosudo -i if logged in with admin
Activate the python venv of the synapse app which is located in /opt/yunohost/matrix-synapse/ by the following command:
source /opt/yunohost/matrix-synapse/bin/activate
Install the module with pip:
pip install synapse-s3-storage-provider
Add the required settings in your homeserver.yaml which is located in /etc/matrix-synapse/homeserver.yaml, the new settings to insert should look like the following:

media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
    bucket: "your_bucket_name"
    # All of the below options are optional, for use with non-AWS S3-like
    # services, or to specify access tokens here instead of some external method.
    region_name: "your-region-name"
    endpoint_url: "https://your-storage.com"
    access_key_id: "your_key"
    secret_access_key: "your_key"

Co-authored-by: reivilibre <oliverw@matrix.org>

sorcer1122 · 2023-01-08T09:56:26Z

I am still getting No module named s3_storage_provider on Debian 11. I am using matrix-synapse-py3 and did pip3 install synapse-s3-storage-provider.

Any suggestions where to dig?

Ok for those who face the same issue - I copied all the necessary folders from /usr/local/lib/python3.9 to /opt/venvs/matrix-synapse/lib/python3.9 manually and all worked!

gityzon · 2023-02-15T05:31:40Z

If you use command apt install matrix-synapse-py3 to install synapse
then try /opt/venvs/matrix-synapse/bin/pip3 install synapse-s3-storage-provider to install synapse-s3-storage-provider.
Hope this may help someone.

ifedan-ed · 2023-06-18T15:08:37Z

If you use command apt install matrix-synapse-py3 to install synapse then try /opt/venvs/matrix-synapse/bin/pip3 install synapse-s3-storage-provider to install synapse-s3-storage-provider. Hope this may help someone.

This works for the DO app. Thanks!

richvdh mentioned this issue Aug 23, 2020

Installation Procedures? #42

Closed

ShadowJonathan mentioned this issue Oct 20, 2020

PyPI/PyPA package? #22

Closed

DMRobertson pushed a commit that referenced this issue Jan 10, 2022

Improvements around releasing modules on PyPI (#14)

231241f

Co-authored-by: reivilibre <oliverw@matrix.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Steps to install #14

Steps to install #14

tompinzler commented Oct 21, 2018 •

edited

Loading

abeluck commented Nov 21, 2018

Sharparam commented Mar 13, 2019 •

edited

Loading

bradinator commented Mar 15, 2019

rkfg commented Jun 3, 2019

bradinator commented Jun 4, 2019

rkfg commented Jun 4, 2019

bradinator commented Jul 2, 2019 •

edited

Loading

rkfg commented Jul 2, 2019

bradinator commented Jul 2, 2019

rkfg commented Jul 2, 2019

rkfg commented Jul 2, 2019

bradinator commented Jul 2, 2019

rkfg commented Jul 2, 2019

bradinator commented Jul 2, 2019

rkfg commented Jul 2, 2019

bradinator commented Jul 3, 2019

rkfg commented Jul 4, 2019

bradinator commented Jul 9, 2019

rkfg commented Jul 9, 2019

HelderFSFerreira commented May 20, 2020

ShadowJonathan commented Sep 2, 2020

ShadowJonathan commented Sep 2, 2020

iokiwi commented Sep 28, 2020 •

edited

Loading

jimmybrancaccio commented Nov 10, 2020 •

edited

Loading

wioxjk commented Feb 25, 2021

inaniyants commented May 24, 2021 •

edited

Loading

Ibrahim-AbouElseoud commented Jul 21, 2021

sorcer1122 commented Jan 8, 2023 •

edited

Loading

gityzon commented Feb 15, 2023

ifedan-ed commented Jun 18, 2023

Steps to install #14

Steps to install #14

Comments

tompinzler commented Oct 21, 2018 • edited Loading

abeluck commented Nov 21, 2018

Sharparam commented Mar 13, 2019 • edited Loading

bradinator commented Mar 15, 2019

rkfg commented Jun 3, 2019

bradinator commented Jun 4, 2019

rkfg commented Jun 4, 2019

bradinator commented Jul 2, 2019 • edited Loading

rkfg commented Jul 2, 2019

bradinator commented Jul 2, 2019

rkfg commented Jul 2, 2019

rkfg commented Jul 2, 2019

bradinator commented Jul 2, 2019

rkfg commented Jul 2, 2019

bradinator commented Jul 2, 2019

rkfg commented Jul 2, 2019

bradinator commented Jul 3, 2019

rkfg commented Jul 4, 2019

bradinator commented Jul 9, 2019

rkfg commented Jul 9, 2019

HelderFSFerreira commented May 20, 2020

ShadowJonathan commented Sep 2, 2020

ShadowJonathan commented Sep 2, 2020

iokiwi commented Sep 28, 2020 • edited Loading

jimmybrancaccio commented Nov 10, 2020 • edited Loading

wioxjk commented Feb 25, 2021

inaniyants commented May 24, 2021 • edited Loading

Ibrahim-AbouElseoud commented Jul 21, 2021

sorcer1122 commented Jan 8, 2023 • edited Loading

gityzon commented Feb 15, 2023

ifedan-ed commented Jun 18, 2023

tompinzler commented Oct 21, 2018 •

edited

Loading

Sharparam commented Mar 13, 2019 •

edited

Loading

bradinator commented Jul 2, 2019 •

edited

Loading

iokiwi commented Sep 28, 2020 •

edited

Loading

jimmybrancaccio commented Nov 10, 2020 •

edited

Loading

inaniyants commented May 24, 2021 •

edited

Loading

sorcer1122 commented Jan 8, 2023 •

edited

Loading