"New Google Drive" and 2 Millions of files ? « Syncdocs Google Drive Forum

Syncdocs Google Drive Forum » General

"New Google Drive" and 2 Millions of files ?

(7 posts) (2 voices)

Started 11 years ago by er007
Latest reply from Phil
1 Members Subscribed To Topic

Tags:

er007
Member

Hi all,

Currently I have about 130 000 files in Google Drive. Syncdoc takes about 10 minutes to count them all.
But in fact I have like about 2 Millions more files to upload (not sync) to google drive (mainly very small txt files <10Mo each - that I have converted from html using imacro freeware in Firefox). By luck do you know if the "New Google Drive" API could just allow the same kind of things that the new gmail API ?

Example : "To keep in sync, the API allows you to query the inbox change history, thereby avoiding the need to do “archaeology” to figure out what changed."
http://googleappsdeveloper.blogspot.fr/2014/06/introducing-new-gmail-api.html

If so I think this could improve speed (I guess it should be 3.5 hours for 2 Million files!) for counting files and sync only a few other files, I have to sync.

Thanks in advance ;)

Posted 11 years ago #

Edit page
Phil
Syncdocs

Hi,

Thanks for the suggestion. Google Drive also has a change history, and Syncdocs does use it, but not on the first sync. That is one of the reasons it is best to keep Syncdocs running in the background. On the first sync needs to check locally what has happened since it was last run, so this takes a bit of time. Also, the Google change history is not 100% accurate, it sometimes misses files if they are stored across multiple Google server, so Syncdocs also takes a bit of time to double check when uploading.

The new version of Syncdocs that we are working on, has speed improvements, which are really visible for users with lots of files.

Posted 11 years ago #
er007
Member

Hi,

Thanks Phil.

> (...)it sometimes misses files if they are stored across multiple Google server (...)

So it is why I have some of my folders in my main google drive online (https://drive.google.com/?tab=mo&authuser=0#my-drive) that are appearing one or two seconds after the first ones!

>new version of Syncdoc

I cant' wait ! ;) Just kidding! Please take your time and do it right as you have the habit. ;)

Thanks

Posted 11 years ago #

Edit page
er007
Member

Hello,
We are about 1.5 year later. Now you have released the much awaited version 6. ;)
I am wondering if someone has already tested having about a few Millions of small txt files into Google Drive ? Does the Google Drive UI still responds fast ?
Thanks in advance ;)

Posted 9 years ago #
Phil
Syncdocs

The new version is much faster with many files, as it no longer re-checks for the full file list from Google at startup, only for changes. This can greatly reduce the startup time. It will still take a long time the first time you run it to find all the files on Google, but after that it will be much faster.

Posted 9 years ago #
er007
Member

Hello,
Thanks. In my small tests, if I run syndocs on 2 computers at the same time (for the same Google Drive account): let's call them : PC1 and PC2.
PC1 is busy uploading big zip files (often a few GB).
PC2 is uploading a few small files from time to time. I notice that PC2 does already takes a lot of time finding all the files (about 200,000) quite often (when I add manually some files to be uploaded from time to time) !
Is it normal behaviour ?

Also, each time I touch the preference options of syndocs, it list all the files on google drive. That one seems usual to me.
Thanks ;)

Posted 9 years ago #
Phil
Syncdocs

There is one option that can prevent Syncdocs doing a fast sync, and make it request the full file list from Google, instead of just the changes. This option is the "Allow duplicate filenames on Google" in the Files Tab of the Preferences. It is checked by default, and this is the fast option. If you un-check it, then you will slow the sync down. Perhaps check this option on PC2.

Posted 9 years ago #

RSS feed for this topic

Reply

You must log in to post.