V3 API S3 Datasource root user not allowed

[chapmanjacobd]

Describe the bug
Jupyter doesn't allow running as root user even though we live in the container universe whether we like it or not

To Reproduce
Steps to reproduce the behavior:

# whoami
root
# great_expectations --v3-api datasource new
Using v3 (Batch Request) API

What data would you like Great Expectations to connect to?
    1. Files on a filesystem (for processing with Pandas or Spark)
    2. Relational database (SQL)
: 2

Which database backend are you using?
    1. MySQL
    2. Postgres
    3. Redshift
    4. Snowflake
    5. BigQuery
    6. other - Do you have a working SQLAlchemy connection string?
: 2
Because you requested to create a new Datasource, we'll open a notebook for you now to complete it!


[I 17:03:04.385 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[C 17:03:04.727 NotebookApp] Running as root is not recommended. Use --allow-root to bypass.

# great_expectations --v3-api datasource new --allow-root
Using v3 (Batch Request) API
Usage: great_expectations datasource new [OPTIONS]
Try 'great_expectations datasource new --help' for help.

Error: No such option: --allow-root
[2] # great_expectations --allow-root --v3-api datasource new
Usage: great_expectations [OPTIONS] COMMAND [ARGS]...
Try 'great_expectations --help' for help.

Error: No such option: --allow-root

Expected behavior
This is the first program I've run into where I've had to create another user account just to run it. Mildly annoying but maybe not worth fixing either. v2 Datasource ran fine but I wanted to try out v3.

Environment (please complete the following information):

• Operating System: WSL2 (Fedora)
• Great Expectations Version: 0.13.37

Additional context

Workaround:

# adduser ge_user
# su ge_user
$ pipenv install && pipenv shell
$ great_expectations --v3-api datasource new

[chapmanjacobd]

oh actually it doesn't even work as a new user

 PermissionError: [Errno 13] Permission denied: '/usr/local/share/jupyter/nbconvert/templates/conf.json'

https://github.com/jupyter/notebook/blob/ccd9665571107e02a325a738b8baebd6532b2d3d/notebook/notebookapp.py#L2291

I've already spent more time on this than I care to. Guess I'll stick with API v2 for now

[jasongrout]

PermissionError: [Errno 13] Permission denied: '/usr/local/share/jupyter/nbconvert/templates/conf.json'

FYI, the situation that caused this error was likely fixed in nbconvert 6.2 (see jupyter/nbconvert#1646), released a few weeks ago.

[chapmanjacobd]

Interesting! thanks for taking a look. but it seems like I was already using nbconvert 6.2

(my Pipfile.lock looks like this)

        "nbconvert": {
            "hashes": [
                "sha256:16ceecd0afaa8fd26c245fa32e2c52066c02f13aa73387fffafd84750baea863",
                "sha256:b1b9dc4f1ff6cafae0e6d91f42fb9046fdc32e6beb6d7e2fa2cd7191ad535240"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==6.2.0"
        },

maybe it is something else

[jasongrout]

You may need to reinstall after moving to nbconvert 6.2, or clean up the erroneous files, as a previous version might have created those files with the wrong permissions. See some of the steps people had to do in jupyter/nbconvert#1430. What 6.2 does is not create the problem in the future, but it won't fix the problems created by previous versions of nbconvert.

[talagluck]

Thanks @chapmanjacobd and @jasongrout! To confirm, is this still an issue?

[chapmanjacobd]

hmmm I still get the same error even with a new venv created under the normal user account with the same Pipfile. I'm going to switch to using my linux laptop for GE related things so not super interested in going down this rabbit hole to fix... if someone else has an issue maybe this stacktrace will help:

actually it wasn't the same Pipfile I just did pipenv install great_expectations and it used

        "nbconvert": {
            "hashes": [
                "sha256:16ceecd0afaa8fd26c245fa32e2c52066c02f13aa73387fffafd84750baea863",
                "sha256:b1b9dc4f1ff6cafae0e6d91f42fb9046fdc32e6beb6d7e2fa2cd7191ad535240"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==6.2.0"
        },

but same error so 🤷‍♂️

[I 19:18:12.935 NotebookApp] Jupyter Notebook 6.4.4 is running at:
[I 19:18:12.935 NotebookApp] http://localhost:8888/?token=18e26da8f63f0f850f6dc5a76f9a87341c643161cec64f13
[I 19:18:12.935 NotebookApp]  or http://127.0.0.1:8888/?token=18e26da8f63f0f850f6dc5a76f9a87341c643161cec64f13
[I 19:18:12.935 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 19:18:12.988 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/xk/.local/share/jupyter/runtime/nbserver-13742-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=18e26da8f63f0f850f6dc5a76f9a87341c643161cec64f13
     or http://127.0.0.1:8888/?token=18e26da8f63f0f850f6dc5a76f9a87341c643161cec64f13
[E 19:18:15.377 NotebookApp] Uncaught exception GET /notebooks/datasource_new.ipynb?token=18e26da8f63f0f850f6dc5a76f9a87341c643161cec64f13 (127.0.0.1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/notebooks/datasource_new.ipynb?token=18e26da8f63f0f850f6dc5a76f9a87341c643161cec64f13', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib64/python3.9/site-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib64/python3.9/site-packages/tornado/gen.py", line 775, in run
        yielded = self.gen.send(value)
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/notebook/notebook/handlers.py", line 95, in get
        self.write(self.render_template('notebook.html',
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/notebook/base/handlers.py", line 516, in render_template
        return template.render(**ns)
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/jinja2/environment.py", line 1304, in render
        self.environment.handle_exception()
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/jinja2/environment.py", line 925, in handle_exception
        raise rewrite_traceback_stack(source=source)
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/notebook/templates/notebook.html", line 1, in top-level template code
        {% extends "page.html" %}
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/notebook/templates/page.html", line 154, in top-level template code
        {% block header %}
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/notebook/templates/notebook.html", line 115, in block 'header'
        {% for exporter in get_frontend_exporters() %}
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/notebook/notebook/handlers.py", line 40, in get_frontend_exporters
        for name in get_export_names():
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/nbconvert/exporters/base.py", line 146, in get_export_names
        e = get_exporter(exporter_name)(config=config)
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/nbconvert/exporters/base.py", line 103, in get_exporter
        if getattr(exporter(config=config), 'enabled', True):
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/nbconvert/exporters/templateexporter.py", line 329, in __init__
        super().__init__(config=config, **kw)
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 114, in __init__
        self._init_preprocessors()
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/nbconvert/exporters/templateexporter.py", line 496, in _init_preprocessors
        conf = self._get_conf()
      File "/home/xk/.local/share/virtualenvs/testgev3-pljby3Z9/lib/python3.9/site-packages/nbconvert/exporters/templateexporter.py", line 516, in _get_conf
        if conf_path.exists():
      File "/usr/lib64/python3.9/pathlib.py", line 1424, in exists
        self.stat()
      File "/usr/lib64/python3.9/pathlib.py", line 1232, in stat
        return self._accessor.stat(self)
    PermissionError: [Errno 13] Permission denied: '/usr/local/share/jupyter/nbconvert/templates/conf.json'
[E 19:18:15.391 NotebookApp] {
      "Host": "localhost:8888",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept-Encoding": "gzip, deflate",
      "Dnt": "1",
      "Connection": "keep-alive",
      "Cookie": "username-localhost-8888=\"2|1:0|10:1634250046|23:username-localhost-8888|44:ZDlkMTI0OTEzNGQyNGFjOTgwNjc5M2UyNGRiODVmMWY=|eba3910fa82b82a8a4b61e0f6e9177b62f6eaec0550c19b40b0e44c79cd3943a\"; _xsrf=2|19049fa9|4980f22b21e25adf6055a455acb84bfa|1634250046; CSRF-Token-QEO36=FRpx3faYXWKunYQw7ApcFTbJv4YNuhEw",
      "Upgrade-Insecure-Requests": "1",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "cross-site",
      "Sec-Gpc": "1"
    }
[E 19:18:15.391 NotebookApp] 500 GET /notebooks/datasource_new.ipynb?token=18e26da8f63f0f850f6dc5a76f9a87341c643161cec64f13 (127.0.0.1) 152.770000ms referer=None