Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[macOS] command to backup fails as crontab only #43

Closed
dgraziotin opened this issue Aug 17, 2020 · 4 comments
Closed

[macOS] command to backup fails as crontab only #43

dgraziotin opened this issue Aug 17, 2020 · 4 comments
Assignees
@MatthieuBizien
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@dgraziotin
Copy link

Describe the bug
On macOS Catalina. The same command that I run from the console (and works) fails when run as crontab.

To Reproduce

  1. Run the backup command from the command line (I anonymized e-mail address and graph name):
/Users/daniel/.local/bin/roam-to-git /Users/daniel/BrainBackup
2020-08-17 10:40:18.015 | INFO     | roam_to_git.__main__:main:50 - Loading secrets from /Users/daniel/BrainBackup/.env
2020-08-17 10:40:18.020 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-17 10:40:18.400 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-17 10:40:19.052 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmphp6ahdp0
2020-08-17 10:40:19.091 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-17 10:40:19.147 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmptfrz3qf0
2020-08-17 10:40:19.186 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-17 10:40:23.731 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
2020-08-17 10:40:23.778 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
2020-08-17 10:40:28.943 | DEBUG    | roam_to_git.scrapping:signin:192 - Fill password
2020-08-17 10:40:28.994 | DEBUG    | roam_to_git.scrapping:signin:192 - Fill password
2020-08-17 10:40:30.103 | DEBUG    | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-17 10:40:30.154 | DEBUG    | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-17 10:40:32.363 | DEBUG    | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/myroam'
2020-08-17 10:40:32.407 | DEBUG    | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/myroam'
2020-08-17 10:40:33.415 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-17 10:40:33.438 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-17 10:40:39.683 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:130 - Launch download popup
2020-08-17 10:40:39.705 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:130 - Launch download popup
2020-08-17 10:40:42.043 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:144 - Checking download type
2020-08-17 10:40:42.072 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:144 - Checking download type
2020-08-17 10:40:42.091 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:159 - Downloading output of type markdown
2020-08-17 10:40:42.116 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:148 - Changing output type to json
2020-08-17 10:40:42.433 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:164 - Wait download of markdown to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmphp6ahdp0
2020-08-17 10:40:43.435 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:174 - File /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmphp6ahdp0/Roam-Export-1597653643328.zip found for markdown
2020-08-17 10:40:44.436 | DEBUG    | roam_to_git.scrapping:download_rr_archive:76 - Closing browser markdown
2020-08-17 10:40:44.489 | DEBUG    | roam_to_git.scrapping:download_rr_archive:78 - Closed browser markdown
2020-08-17 10:40:46.480 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:159 - Downloading output of type json
2020-08-17 10:40:46.816 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:164 - Wait download of json to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmptfrz3qf0
2020-08-17 10:40:47.818 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:174 - File /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmptfrz3qf0/Roam-Export-1597653647235.zip found for json
2020-08-17 10:40:48.821 | DEBUG    | roam_to_git.scrapping:download_rr_archive:76 - Closing browser json
2020-08-17 10:40:48.867 | DEBUG    | roam_to_git.scrapping:download_rr_archive:78 - Closed browser json
2020-08-17 10:40:48.867 | DEBUG    | roam_to_git.scrapping:scrap:253 - Scrapping finished
2020-08-17 10:40:48.872 | DEBUG    | roam_to_git.fs:save_markdowns:50 - Saving markdown to /Users/daniel/BrainBackup/markdown
2020-08-17 10:40:48.885 | DEBUG    | roam_to_git.fs:unzip_and_save_json_archive:62 - Saving json to /Users/daniel/BrainBackup/json
2020-08-17 10:40:48.945 | DEBUG    | roam_to_git.fs:save_markdowns:50 - Saving markdown to /Users/daniel/BrainBackup/formatted
2020-08-17 10:40:49.056 | DEBUG    | roam_to_git.fs:commit_git_directory:79 - Committing git repository /Users/daniel/BrainBackup/.git
2020-08-17 10:40:49.141 | DEBUG    | roam_to_git.fs:push_git_repository:85 - Pushing to origin
  1. Add the same command as crontab (crontab -e)
0 * * * * /Users/daniel/.local/bin/roam-to-git /Users/daniel/BrainBackup

Expected behavior
I would expect the same output and a successful backup.

What I get (I just take the latest one I received in my mailbox):

Subject: Cron <daniel@mymacbook> /Users/daniel/.local/bin/roam-to-git /Users/daniel/BrainBackup
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=daniel>
X-Cron-Env: <USER=daniel>
Date: Sun, 16 Aug 2020 22:00:01 +0200 (CEST)

2020-08-16 22:00:01.019 | INFO     | roam_to_git.__main__:main:50 - Loading secrets from /Users/daniel/BrainBackup/.env
2020-08-16 22:00:01.026 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-16 22:00:01.389 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-16 22:00:03.232 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /tmp/tmpij4r1sjm
2020-08-16 22:00:03.272 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /tmp/tmpcg12_9fi
2020-08-16 22:00:03.336 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-16 22:00:03.372 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-16 22:00:10.268 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
2020-08-16 22:00:10.572 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
2020-08-16 22:00:22.892 | DEBUG    | roam_to_git.scrapping:download_rr_archive:76 - Closing browser json
2020-08-16 22:00:22.903 | DEBUG    | roam_to_git.scrapping:download_rr_archive:76 - Closing browser markdown
2020-08-16 22:00:23.059 | DEBUG    | roam_to_git.scrapping:download_rr_archive:78 - Closed browser json
2020-08-16 22:00:23.060 | ERROR    | __main__:<module>:8 - An error has been caught in function '<module>', process 'MainProcess' (20024), thread 'MainThread' (4372716992):
Traceback (most recent call last):
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 85, in evaluateHandle
    'userGesture': True,

pyppeteer.errors.NetworkError: Protocol error (Runtime.evaluate): Cannot find context with specified id


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
> File "/Users/daniel/.local/bin/roam-to-git", line 8, in <module>
    sys.exit(main())
    │   │    └ <function main at 0x7fb9100ba290>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/__main__.py", line 76, in main
    scrap(markdown_zip_path, json_zip_path, config)
    │     │                  │              └ <roam_to_git.scrapping.Config object at 0x7fb928052b50>
    │     │                  └ PosixPath('/tmp/tmpcg12_9fi')
    │     └ PosixPath('/tmp/tmpij4r1sjm')
    └ <function scrap at 0x7fb9100ba0e0>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 252, in scrap
    asyncio.get_event_loop().run_until_complete(asyncio.gather(*tasks))
    │       │                                   │       │       └ [<coroutine object download_rr_archive at 0x7fb9100ba560>, <coroutine object download_rr_archive at 0x7fb9100ba7a0>]
    │       │                                   │       └ <function gather at 0x7fb9507d2c20>
    │       │                                   └ <module 'asyncio' from '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/__init__.py'>
    │       └ <built-in function get_event_loop>
    └ <module 'asyncio' from '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/__init__.py'>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
           │      └ <method 'result' of '_asyncio.Future' objects>
           └ <_GatheringFuture finished exception=NetworkError('Execution context was destroyed, most likely because of a navigation.')>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 69, in download_rr_archive
    return await _download_rr_archive(document, output_type, output_directory, config)
                 │                    │         │            │                 └ <roam_to_git.scrapping.Config object at 0x7fb928052b50>
                 │                    │         │            └ PosixPath('/tmp/tmpcg12_9fi')
                 │                    │         └ 'json'
                 │                    └ <pyppeteer.page.Page object at 0x7fb950b1a5d0>
                 └ <function _download_rr_archive at 0x7fb93009c9e0>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 97, in _download_rr_archive
    await signin(document, config, sleep_duration=config.sleep_duration)
          │      │         │                      │      └ 2.0
          │      │         │                      └ <roam_to_git.scrapping.Config object at 0x7fb928052b50>
          │      │         └ <roam_to_git.scrapping.Config object at 0x7fb928052b50>
          │      └ <pyppeteer.page.Page object at 0x7fb950b1a5d0>
          └ <function signin at 0x7fb93009c830>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 188, in signin
    email_elem = await document.querySelector("input[name='email']")
                       │        └ <function Page.querySelector at 0x7fb9100a49e0>
                       └ <pyppeteer.page.Page object at 0x7fb950b1a5d0>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/page.py", line 371, in querySelector
    return await frame.querySelector(selector)
                 │     │             └ "input[name='email']"
                 │     └ <function Frame.querySelector at 0x7fb910063320>
                 └ <pyppeteer.frame_manager.Frame object at 0x7fb8f0010d90>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/frame_manager.py", line 316, in querySelector
    document = await self._document()
                     │    └ <function Frame._document at 0x7fb9100633b0>
                     └ <pyppeteer.frame_manager.Frame object at 0x7fb8f0010d90>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/frame_manager.py", line 326, in _document
    document = (await context.evaluateHandle('document')).asElement()
                      │       └ <function ExecutionContext.evaluateHandle at 0x7fb928071710>
                      └ <pyppeteer.execution_context.ExecutionContext object at 0x7fb8f00218d0>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 88, in evaluateHandle
    _rewriteError(e)
    └ <function _rewriteError at 0x7fb92806b710>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 237, in _rewriteError
    raise type(error)(msg)
               │      └ 'Execution context was destroyed, most likely because of a navigation.'
               └ NetworkError('Protocol error (Runtime.evaluate): Cannot find context with specified id')

pyppeteer.errors.NetworkError: Execution context was destroyed, most likely because of a navigation.
Traceback (most recent call last):
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 85, in evaluateHandle
    'userGesture': True,
pyppeteer.errors.NetworkError: Protocol error (Runtime.evaluate): Cannot find context with specified id

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/daniel/.local/bin/roam-to-git", line 8, in <module>
    sys.exit(main())
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/loguru/_logger.py", line 1149, in catch_wrapper
    return function(*args, **kwargs)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/__main__.py", line 76, in main
    scrap(markdown_zip_path, json_zip_path, config)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 252, in scrap
    asyncio.get_event_loop().run_until_complete(asyncio.gather(*tasks))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 69, in download_rr_archive
    return await _download_rr_archive(document, output_type, output_directory, config)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 97, in _download_rr_archive
    await signin(document, config, sleep_duration=config.sleep_duration)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 188, in signin
    email_elem = await document.querySelector("input[name='email']")
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/page.py", line 371, in querySelector
    return await frame.querySelector(selector)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/frame_manager.py", line 316, in querySelector
    document = await self._document()
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/frame_manager.py", line 326, in _document
    document = (await context.evaluateHandle('document')).asElement()
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 88, in evaluateHandle
    _rewriteError(e)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 237, in _rewriteError
    raise type(error)(msg)
pyppeteer.errors.NetworkError: Execution context was destroyed, most likely because of a navigation.
2020-08-16 22:00:23.086 | DEBUG    | roam_to_git.scrapping:_kill_child_process:215 - Terminate child process [psutil.Process(pid=20032, name='Chromium', status='zombie', started='22:00:01')]

Traceback
Please use http://gist.github.com/ or similar, and report the last line here.

Run roam-to-git --debug notes/ and report what you get.

/Users/daniel/.local/bin/roam-to-git --debug /Users/daniel/BrainBackup
2020-08-17 10:51:15.974 | INFO     | roam_to_git.__main__:main:50 - Loading secrets from /Users/daniel/BrainBackup/.env
2020-08-17 10:51:15.977 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-17 10:51:17.654 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-17 10:51:22.112 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error (Runtime.releaseObject): Cannot find context with specified id')>
pyppeteer.errors.NetworkError: Protocol error (Runtime.releaseObject): Cannot find context with specified id
2020-08-17 10:51:27.398 | DEBUG    | roam_to_git.scrapping:signin:192 - Fill password
2020-08-17 10:51:28.572 | DEBUG    | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-17 10:51:30.821 | DEBUG    | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/myroam'
2020-08-17 10:51:31.919 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-17 10:51:38.225 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:130 - Launch download popup
2020-08-17 10:51:40.586 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:144 - Checking download type
2020-08-17 10:51:40.635 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:159 - Downloading output of type markdown
2020-08-17 10:51:40.970 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:164 - Wait download of markdown to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmpy8e_6hso
2020-08-17 10:51:40.970 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-17 10:51:42.647 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-17 10:51:46.838 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error (Runtime.releaseObject): Cannot find context with specified id')>
pyppeteer.errors.NetworkError: Protocol error (Runtime.releaseObject): Cannot find context with specified id
2020-08-17 10:51:52.303 | DEBUG    | roam_to_git.scrapping:signin:192 - Fill password
2020-08-17 10:51:53.458 | DEBUG    | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-17 10:51:55.713 | DEBUG    | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/myroam'
2020-08-17 10:51:56.786 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-17 10:52:03.071 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:130 - Launch download popup
2020-08-17 10:52:05.426 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:144 - Checking download type
2020-08-17 10:52:05.475 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:148 - Changing output type to json
2020-08-17 10:52:09.842 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:159 - Downloading output of type json
2020-08-17 10:52:10.183 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:164 - Wait download of json to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmp77q3gvgy
2020-08-17 10:52:10.183 | WARNING  | roam_to_git.scrapping:scrap:249 - Exiting without updating the git repository, because we can't get the downloads with the option --debug
2020-08-17 10:52:10.183 | DEBUG    | roam_to_git.__main__:main:78 - waiting for the download...

Please complete the following information:

  • OS: macOS Catalina, latest version
  • Do you use Github Action? No, a self-hosted Gitea server.
  • Do you use multiple Roam databases? No. My account has just one associated graph.
  • Does roam-to-git use to work for you? When precisely did it stopped to work? Works perfectly when triggered manually.
  • Does some backup runs are still working? I guess? They look fine.

Additional context
Add any other context about the problem here.
@dgraziotin dgraziotin added the bug Something isn't working label Aug 17, 2020
@MatthieuBizien
Copy link
Owner

Well I have the same error now. It uses to work. No idea where it's from, but Mac crontab is weird.

@MatthieuBizien MatthieuBizien added the help wanted Extra attention is needed label Aug 21, 2020
@sinoth
Copy link

sinoth commented Aug 25, 2020

I'm getting the same error running the script through a github action. The error is "Execution context was destroyed, most likely because of a navigation".

I'm only running once a day so I don't know how recurrent the problem is. My first error was last night.

cumtnp pushed a commit to cumtnp/roam-to-git that referenced this issue Oct 20, 2020
bug fix MatthieuBizien#43
9600830 
Signed-off-by: liang ding <liangding@liangdeMacBook-Air.local>
@cumtnp
Copy link

cumtnp commented Oct 20, 2020

@MatthieuBizien try my solution：#58
1，Wait for this element input[name='email'] on the webpage to be rendered。
2，Adjust the operating time slightly

@MatthieuBizien
Copy link
Owner

The backup errors should be fixed by #92 🍾.

I'm closing that issue, but feel free to re-open it if you get new errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MatthieuBizien MatthieuBizien

Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dgraziotin @MatthieuBizien @sinoth @cumtnp