gsync only create directories #69

richardh538 · 2014-06-23T09:16:32Z

Hello,

I have a directory that I wish to sync with google drive.
My directory, temp, contain one text file, with the name apan.txt and content "apan"

$ gsync -r --progress -v temp/ drive://temp
./
apan.txt
0 0% 0.00B/s 0:00:00
sent 0 bytes received 0 bytes 0.00 bytes/sec

However, only the directory temp is created on the google drive.

What do I do wrong?

/Richard

snowliondev · 2014-06-24T09:27:20Z

I get this same issue

ypmict · 2014-07-07T07:16:07Z

Hi,
I also get same issue, some of the progress output :

$ time gsync --owner --group --progress --perms -u -d -r -vi /opt/ict/monday/ drive://ict/

<f+++++++++ 2012/work-jp.docx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/~$noplanning.docx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/PLANNING (edit).xlsx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/Planned 2012.docx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/strategic (c).docx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/presentation LATEST.pptx
0 0% 0.00B/s 0:00:00

all shows 0.00B/s ..when checking through Google Drive webpage.. only folder is created but no content...

ypmict · 2014-07-08T00:30:24Z

Hi.. I fixed it by using method as this thread issue : #66
thanks to @GyroLand
...but some error occured while syncing :

return req.execute()
  File "/usr/lib/python2.6/site-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/apiclient/http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)

Error: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/xxxxxxxxxxxxxxxxxxxxxx?alt=json&setModifiedDate=false&newRevision=true returned "Internal Error">
sent 16005692534 bytes  received 0 bytes  4471186.31 bytes/sec
sent 0 bytes  received 0 bytes  0.00 bytes/sec
sent 0 bytes  received 0 bytes  0.00 bytes/sec

... also many files were not sync maybe because of above error?
it takes about 7 hours+ to sync document file total size of 8GB office files .doc, ppt, pdf, etc (this is by using 12 Mbps leased line dedicated Internet)

GyroLand · 2014-07-12T14:41:06Z

Yes, i have the same, i think this is some kind of a time out error or something. Anyway, if you don't use the -u flag and run the program several times, it will upload also the failed files. And it takes much less than the first time, as it uploads only the missing files.

plecavalier · 2014-07-12T14:59:34Z

I have the same issue and can't get files uploaded(with or without the -u
flag). I've tried many times and the files still get ignored(or seem to).

Thanks, Phil

On Sat, Jul 12, 2014 at 10:41 AM, GyroLand notifications@github.com wrote:

Yes, i have the same, i think this is some kind of a time out error or
something. Anyway, if you don't use the -u flag and run the program several
times, it will upload also the failed files. And it takes much less than
the first time, as it uploads only the missing files.

—
Reply to this email directly or view it on GitHub
#69 (comment).

nmatthews · 2014-07-23T03:25:36Z

Also having the directories but no files issue. I don't think it's a timeout error, running with -v shows that it's not spending much time on any given file.

plecavalier · 2014-07-23T12:43:51Z

I've yet to decipher what's happening myself. In my case I found a
workaround by uploading my files manually, killing the md5sum and running
gsync. It's been working well ever since. Must be done strange semi-rare
condition.
On Jul 22, 2014 11:25 PM, "nmatthews" notifications@github.com wrote:

Also having the directories but no files issue. I don't think it's a
timeout error, running with -v shows that it's not spending much time on
any given file.

—
Reply to this email directly or view it on GitHub
#69 (comment).

dmutters · 2014-09-19T16:52:39Z

$ gsync --version
0.1.14

I'm also having this problem. It seems to have started only recently; old files seem to update (I think), but new ones don't upload. Directories are created, but are unpopulated on Google Drive. I'm wondering if it has to do with having a directory path that's more than one level deep. Here's one example:

gsync -c -r -t -p -o -g -v --progress --delete -l -s /mnt/PERSONAL/Dane/Recipes/ drive://orchestrator/Recipes

I've tried this with just the "-r -v --progress" options, as well, and it still doesn't upload the files. (I refreshed the Google Drive page a few times, just to be sure.) My network activity applet reports that while (supposedly) uploading directories full of small text files, the download speed is ~12KiB/s, and the upload speed is ~14KiB/s. (My Internet connection is much, much faster than this, so I don't think it's an uplink issue.)

I've tried doing this sync multiple times, and the files haven't yet been uploaded, so the solution, above, about just trying it a few times seems not to be working for me.

Thanks for your work on this! It's truly an indespensible tool!

dmutters · 2014-09-19T17:05:30Z

It looks like editing /usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py, and changing:

body = {}
        for k, v in properties.iteritems():
                body[k] = _Drive.utf8(v)

...to:

body = {}
        for k, v in properties.iteritems():
            if v is not None:
                body[k] = _Drive.utf8(v)

...solves the problem, per #66.

It does, however, produce this when it runs into a file that ends in a ~ character:

Monsters/Lvl. 3 - Undead Horrid Housecat~
           0   0%     0.00B/s    0:00:00DEBUG: 'Exception':   File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 712, in update
    status, res = req.next_chunk()
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 874, in next_chunk
    return self._process_response(resp, content)
  File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 901, in _process_response
    raise HttpError(resp, content, uri=self.uri)

plecavalier · 2014-09-21T15:54:42Z

Worked for me.

Thanks, Phil

On Fri, Sep 19, 2014 at 1:05 PM, dmutters notifications@github.com wrote:

It looks like editing
/usr/local/lib/python2.7/dist-packages/libgsync/drive/init.py, and
changing:

body = {}
for k, v in properties.iteritems():
body[k] = _Drive.utf8(v)

...to:

body = {}
for k, v in properties.iteritems():
if v is not None:
body[k] = _Drive.utf8(v)

...solves the problem, per #66
#66.

—
Reply to this email directly or view it on GitHub
#69 (comment).

johanels · 2014-10-31T11:02:44Z

Solution not working for me. Making the change it will not run at all.

dmutters · 2014-10-31T18:54:33Z

It worked for a while, for me, and then broke. This is what I did to fix it (as root):

(Install pip using your distro's package manager.)
pip uninstall gsync
rm -vfr /usr/local/lib/python2.7/dist-packages/libgsync /usr/local/bin/gsync
pip install gsync
pip install --upgrade gsync

It's once again working for me with only the occasional hiccup.

plecavalier · 2014-10-31T19:42:00Z

the fix worked for me for quite some time and just the other I realized it
had stopped.
On Oct 31, 2014 2:54 PM, "dmutters" notifications@github.com wrote:

It worked for a while, for me, and then broke. This is what I did to fix
it (as root):

(Install pip using your distro's package manager.)

pip uninstall gsync

rm -vfr /usr/local/lib/python2.7/dist-packages/libgsync
/usr/local/bin/gsync

pip install gsync

pip install --upgrade gsync

It's once again working for me with only the occasional hiccup.

—
Reply to this email directly or view it on GitHub
#69 (comment).

johanels · 2014-11-01T01:23:33Z

I'm thinking this might be caused by something else.

Found a "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup." error in the cache files link for the files API.

If you

grep -R "selfLink" ~/.gsync/ | more

and look for the https://www.googleapis.com/drive/v2/files/.......... links and try them, I got the response from Google about a limit.

If you also get the key

https://www.googleapis.com/drive/v2/files/0B42Fqxu8L6YfYkZW.........

Then we are all using the same key.

Thank you,

plecavalier · 2014-11-01T02:32:06Z

bingo.

Thanks, Phil

On Fri, Oct 31, 2014 at 9:23 PM, johanels notifications@github.com wrote:

I'm thinking this might be caused by something else.

Found a "Daily Limit for Unauthenticated Use Exceeded. Continued use
requires signup." error in the cache files link for the files API.

If you

grep -R "selfLink" ~/.gsync/ | more

and look for the https://www.googleapis.com/drive/v2/files/..........
links and try them, I got the response from Google about a limit.

If you also get the key

https://www.googleapis.com/drive/v2/files/0B42Fqxu8L6YfYkZW.........

Then we are all using the same key.

Thank you,

—
Reply to this email directly or view it on GitHub
#69 (comment).

johanels · 2014-11-01T06:04:39Z

Ok, sow how do I fix this?

dmutters · 2014-11-01T06:28:17Z

My logs are full of this notice. Thanks for pointing it out.

How can we see what key we're using? That would be a pretty big problem if we're all using the same key, and there's a limit on how much data transfer can be ascribed to any one key.

johanels · 2014-11-01T06:50:02Z

Hi,

The Google JSON key information is loaded in /usr/local/lib/python2.7/dist-packages/libgsync/drive/client_json.py and are the same from the source, so everyone is using the same key.

https://github.com/iwonbigbro/gsync/blob/master/libgsync/drive/client_json.py

I created a new project on Google Developers site, created the JSON auth and changed the file on my machine. I see the requests in Google Developers Dashboard, but number of requests matches number of errors, and gsync still does not upload.

https://console.developers.google.com

Will keep trying.

dmutters · 2014-11-01T08:42:06Z

Thanks, johanels. Let us know when you get it sorted! Hopefully, such a fix can be integrated into the gsync code...

bondurantdt · 2014-11-04T19:22:46Z

Same issue here, only empty directories created.

fnkr · 2014-11-09T16:26:01Z

+1

kaidokert · 2014-12-04T05:19:16Z

#66 fix also worked here

jeztucker · 2015-01-06T13:38:48Z

As johanels indicated, the gsync utility is set to use pre-defined Google API key / credentials when shipped.

This means that by default all users will utilise the data limit (typically 10m API requests per day).

Therefore, you need to create your own Google API Project with associated key / credentials.

Method to resolve this:

Goto: https://console.developers.google.com/project/
Create a new project and go to that Projects Dashboard, then:
Enable an API -> Set Drive API to On
APIs & Auth -> Consent Screen. Configure the consent screen, set a project name 'gsync-someotherthing' (or suchlike) and any relevant information
APIs & Auth -> Credentials -> Create new client ID. Select 'installed application'. Type: Other
As root, edit /usr/local/lib/python2.7/dist-packages/libgsync/drive/client_json.py and modify the client_id and client_secret to reflect your newly generated client credentials
Also apply fix ID = None cause File id mismatch errors on first upload #66
As your required (suggest non-root..) user, run gsync --authenticate and follow the instructions as per the standard docs
Uploading of files will work, against your own personal API requests quota

All the best,

Jez

johanels · 2015-01-06T19:31:05Z

Thank you Jez. Seems to be running and I can see Response codes in the Development Console.

jeztucker · 2015-01-06T20:04:12Z

Great. Have fun.

The solution is clearly to move the client_json requirement to within a
per user .gsync folder.

I plan on submitting a patch for this in the near future.

Jez

On Tuesday, 6 January 2015, johanels notifications@github.com wrote:

Thank you Jez. Seems to be running and I can see Response codes in the
Development Console.

—
Reply to this email directly or view it on GitHub
#69 (comment).

This email is confidential in that it is intended for the exclusive
attention of the addressee(s) indicated. If you are not the intended
recipient, this email should not be read or disclosed to any other person.
Please notify the sender immediately and delete this email from your
computer system. Any opinions expressed are not necessarily those of the
company from which this email was sent and, whilst to the best of our
knowledge no viruses or defects exist, no responsibility can be accepted
for any loss or damage arising from its receipt or subsequent use of this
email.

Mr-Si · 2015-01-08T13:10:22Z

Hi Jez, I've been following this fix and am struggling as I don't seem to have the directory:

/usr/local/lib/python2.7/dist-packages/libgsync/drive/

I'm on Centos 6.6 server and have tried uninstalling and re-installing GSync following carrying out an installation of Python 2.7.

I would appreciate your guidance.

Many thanks,
Simon

jeztucker · 2015-01-08T16:41:56Z

Hi

So I don't have access to a CentOS 6.6 right this second.

However, the issue is highly likely to be just that libgsync is in a different place to the one on my system.

To find it, you could do either:

jtucker@elmo:~/Desktop$ pip show -f gsync
---
Name: gsync
Version: 0.1.14
Location: /usr/local/lib/python2.7/dist-packages
Requires: google-api-python-client, docopt, httplib2, oauth2client, python-dateutil, urllib3
Files:
Cannot locate installed-files.txt

So libgsync on mine is under: /usr/local/lib/python2.7/dist-packages

(I'm presuming you've used pip, if you've used the git version, you should know where you've installed it ;-)

or

updatedb
locate drive | grep libgsync

or

find / -name "drive" -type d -print | grep libgsync

All the best,

Jez

Mr-Si · 2015-01-08T16:46:28Z

Thank you oh so much. The first command you issued showed me that instead of dist-packages, it was within site-packages.

I shall progress now. Many many thanks.

phs · 2015-03-10T19:48:32Z

https://github.com/google/skicka already handles this, and appears to also meet my basic needs for syncing content up and down to google drive.

Mr-Si · 2015-10-14T12:14:31Z

Hi there, I'm still getting the exceeded 30GB issue, but my client_json.py file has the google API secret in it as followed by the steps above. It's giving me the following error:

DEBUG: 'Exception': File "/usr/local/lib/python2.7/site-packages/libgsync/drive/init.py", line 703, in update
return req.execute()
File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 142, in positional_wrappe r
return wrapped(_args, *_kwargs)
File "/usr/local/lib/python2.7/site-packages/googleapiclient/http.py", line 729, in execute
raise HttpError(resp, content, uri=self.uri)

DEBUG: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/0B2oFXY6ToU6fNE04aUY wclZFMTg?alt=json&setModifiedDate=true&newRevision=true returned "Internal Error">: File "/usr/loc al/lib/python2.7/site-packages/libgsync/crawler.py", line 150, in run
self._walk(srcpath, self._walkCallback, self._dev)
File "/usr/local/lib/python2.7/site-packages/libgsync/crawler.py", line 124, in _walk
self._sync(f)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/init.py", line 31, in call
self._sync(path)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/init.py", line 197, in _sync
self.dst.update(dstPath, srcFile)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/init.py", line 325, in update
self.updateFile(path, srcObj)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/__init.py", line 228, in updat eFile
self.__updateStats(path, src)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/__init.py", line 273, in updat eStats
self._updateStats(path, src, mode, uid, gid, mtime, atime)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/remote/__init.py", line 148, in _updateStats
'setModifiedDate': GsyncOptions.times
File "/usr/local/lib/python2.7/site-packages/libgsync/drive/init.py", line 703, in update
return req.execute()
File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 142, in positional_wrappe r
return wrapped(_args, *_kwargs)
File "/usr/local/lib/python2.7/site-packages/googleapiclient/http.py", line 729, in execute
raise HttpError(resp, content, uri=self.uri)

Error: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/0B2oFXY6ToU6fNE04aUY wclZFMTg?alt=json&setModifiedDate=true&newRevision=true returned "Internal Error">
sent 307341525 bytes received 0 bytes 60409.28 bytes/sec

When I go in to the link, it tells me the following:

{
"error": {
"errors": [
{
"domain": "usageLimits",
"reason": "dailyLimitExceededUnreg",
"message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.",
"extendedHelp": "https://code.google.com/apis/console"
}
],
"code": 403,
"message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup."
}
}

but the issue is that I already have a google API Secret with an account that has a 1TB limit.

Please can I have further assistance?

dmutters mentioned this issue Jan 20, 2015

gsync to use for backup entire system to Google Drive? #82

Open

dmutters mentioned this issue Feb 6, 2015

Only synching dirs. Not Files? #90

Closed

UbuntuRunner mentioned this issue Mar 3, 2016

Sync's Directories but none of the files within #97

Open

JSDragon mentioned this issue May 21, 2017

gsync create directory but sync noting #101

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gsync only create directories #69

gsync only create directories #69

richardh538 commented Jun 23, 2014

snowliondev commented Jun 24, 2014

ypmict commented Jul 7, 2014

ypmict commented Jul 8, 2014

GyroLand commented Jul 12, 2014

plecavalier commented Jul 12, 2014

nmatthews commented Jul 23, 2014

plecavalier commented Jul 23, 2014

dmutters commented Sep 19, 2014

dmutters commented Sep 19, 2014

plecavalier commented Sep 21, 2014

johanels commented Oct 31, 2014

dmutters commented Oct 31, 2014

plecavalier commented Oct 31, 2014

johanels commented Nov 1, 2014

plecavalier commented Nov 1, 2014

johanels commented Nov 1, 2014

dmutters commented Nov 1, 2014

johanels commented Nov 1, 2014

dmutters commented Nov 1, 2014

bondurantdt commented Nov 4, 2014

fnkr commented Nov 9, 2014

kaidokert commented Dec 4, 2014

jeztucker commented Jan 6, 2015

johanels commented Jan 6, 2015

jeztucker commented Jan 6, 2015

Mr-Si commented Jan 8, 2015

jeztucker commented Jan 8, 2015

Mr-Si commented Jan 8, 2015

phs commented Mar 10, 2015

Mr-Si commented Oct 14, 2015

gsync only create directories #69

gsync only create directories #69

Comments

richardh538 commented Jun 23, 2014

snowliondev commented Jun 24, 2014

ypmict commented Jul 7, 2014

ypmict commented Jul 8, 2014

GyroLand commented Jul 12, 2014

plecavalier commented Jul 12, 2014

nmatthews commented Jul 23, 2014

plecavalier commented Jul 23, 2014

dmutters commented Sep 19, 2014

dmutters commented Sep 19, 2014

plecavalier commented Sep 21, 2014

johanels commented Oct 31, 2014

dmutters commented Oct 31, 2014

plecavalier commented Oct 31, 2014

johanels commented Nov 1, 2014

plecavalier commented Nov 1, 2014

johanels commented Nov 1, 2014

dmutters commented Nov 1, 2014

johanels commented Nov 1, 2014

dmutters commented Nov 1, 2014

bondurantdt commented Nov 4, 2014

fnkr commented Nov 9, 2014

kaidokert commented Dec 4, 2014

jeztucker commented Jan 6, 2015

johanels commented Jan 6, 2015

jeztucker commented Jan 6, 2015

Mr-Si commented Jan 8, 2015

jeztucker commented Jan 8, 2015

Mr-Si commented Jan 8, 2015

phs commented Mar 10, 2015

Mr-Si commented Oct 14, 2015