occ scan:files Adding more details in the base print out (II) #21701

mmattel · 2016-01-13T20:17:40Z

This is a restart of #21445 because of #21504 (Why are some PR checks passing and why are some checks stuck).

This is a PR for adding more details in the base printout of ./occ files:scan

Reference issue #21412 (occ files:scan missing feedback while running)
@jancborchardt @karlitschek

All the other parameters stay the same. When adding eg --verbose, you get the details of each file/directory beeing scanned.

I have implemented the following:

sudo -uwww-data ./occ files:scan user1 -v

Starting scan for user 1 out of 1 (user1)
    File   /user1/
    Folder /user1/
    File   /user1/files
    Folder /user1/files
    File   /user1/files/welcome.txt

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 2       | 3     | 00:00:00     |
+---------+-------+--------------+

sudo -uwww-data ./occ files:scan --all -v

Scanning files for 3 users

Starting scan for user 1 out of 3 (user0)
    File   /user0/
    Folder /user0/
    File   /user0/files
    File   /user0/thumbnails
    File   /user0/cache
    Folder /user0/files
    File   /user0/files/welcome.txt
    File   /user0/files/folder2
    File   /user0/files/folder1
    Folder /user0/files/folder2
    Folder /user0/files/folder1
    File   /user0/files/folder1/hs_err_pid3522.log
    File   /user0/files/folder1/x11vnc-0.9.14-dev.tar.gz
    Folder /user0/thumbnails
    File   /user0/thumbnails/3
    Folder /user0/thumbnails/3
    File   /user0/thumbnails/3/2048-2048-max.png
    File   /user0/thumbnails/3/32-32.png
    Folder /user0/cache

Starting scan for user 2 out of 3 (user1)
    File   /user1/
    Folder /user1/
    File   /user1/files
    Folder /user1/files
    File   /user1/files/welcome.txt

Starting scan for user 3 out of 3 (user2)
    File   /user2/
    Folder /user2/
    File   /user2/files
    Folder /user2/files
    File   /user2/files/welcome.txt

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 11      | 18    | 00:00:00     |
+---------+-------+--------------+

sudo -uwww-data ./occ files:scan --all 

Scanning files for 3 users
Starting scan for user 1 out of 3 (user0)
Starting scan for user 2 out of 3 (user1)
Starting scan for user 3 out of 3 (user2)

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 11      | 18    | 00:00:00     |
+---------+-------+--------------+

There was a discussion in the original PR having the username in the middle or not.
I decided to stay with the username at the end because it then will be formatted properly.
Please see the examples below:

Scanning files of mm (1 of 3 users)
Scanning files of verylogusername (2 of 3 users)
Scanning files of user0 (3 of 3 users)

Starting scan for user 1 out of 3 (mm)
Starting scan for user 2 out of 3 (verylogusername)
Starting scan for user 3 out of 3 (user0)

The table printout at the end is a symfony function which renders automatically. Pls see http://symfony.com/doc/current/components/console/helpers/table.html

mention-bot · 2016-01-13T20:17:41Z

By analyzing the blame information on this pull request, we identified @PVince81, @butonic and @icewind1991 to be potential reviewers

icewind1991 · 2016-01-15T13:05:43Z

👍

DeepDiver1975 · 2016-01-15T13:32:06Z

👍

DeepDiver1975 · 2016-01-15T13:32:27Z

NOTE: the whole interruption logic doesn't seem to work for me ....

mmattel · 2016-01-16T21:44:15Z

@DeepDiver1975 fixed the interruption logic and squashed the commits

mmattel · 2016-01-20T11:43:53Z

@DeepDiver1975 would you mind to give this a try again and based on success merge it?

PVince81 · 2016-01-22T15:07:33Z

Doesn't work for me:

± % sudo -u wwwrun ./occ files:scan --all                                                                                                  !10380
PHP Fatal error:  Call to undefined function OCA\Files\Command\pcntl_signal() in /srv/www/htdocs/owncloud/apps/files/command/scan.php on line 226
PHP Stack trace:
PHP   1. {main}() /srv/www/htdocs/owncloud/occ:0
PHP   2. require_once() /srv/www/htdocs/owncloud/occ:11
PHP   3. OC\Console\Application->run($input = *uninitialized*, $output = *uninitialized*) /srv/www/htdocs/owncloud/console.php:81
PHP   4. Symfony\Component\Console\Application->run($input = *uninitialized*, $output = *uninitialized*) /srv/www/htdocs/owncloud/lib/private/console/application.php:110
PHP   5. Symfony\Component\Console\Application->doRun($input = *uninitialized*, $output = *uninitialized*) /srv/www/htdocs/owncloud/3rdparty/symfony/console/Application.php:123
PHP   6. Symfony\Component\Console\Application->doRunCommand($command = *uninitialized*, $input = *uninitialized*, $output = *uninitialized*) /srv/www/htdocs/owncloud/3rdparty/symfony/console/Application.php:192
PHP   7. Symfony\Component\Console\Command\Command->run($input = *uninitialized*, $output = *uninitialized*) /srv/www/htdocs/owncloud/3rdparty/symfony/console/Application.php:841
PHP   8. OCA\Files\Command\Scan->execute($input = *uninitialized*, $output = *uninitialized*) /srv/www/htdocs/owncloud/3rdparty/symfony/console/Command/Command.php:259
PHP   9. OCA\Files\Command\Scan->initTools() /srv/www/htdocs/owncloud/apps/files/command/scan.php:184

DeepDiver1975 · 2016-01-22T15:20:03Z

pcntl_signal

@mmattel you have to check if pcntl is available - http://php.net/manual/en/pcntl.installation.php

mmattel · 2016-01-22T17:55:07Z

@DeepDiver1975 thanks for bringing this up. I am running Ubuntu and I did not had problems with and never thought that this could be a problem. When I looked in http://php.net/manual/de/function.pcntl-signal.php I saw that it is "present" in our supported php version. But it seems that it is system dependent for beeing available or not (your link mentions that)...
I now added a check if the pcntl_signal function is available and if not, the process can´t be interrupted with ctrl-c anymore - it just runs thru.

@PVince81 you are running openSUSE, could that be that this OS (or others) need special treatement to enable pcntl_signal function calls for php?

@PVince81 @DeepDiver1975 how can we check that or describe in the documentation the need for php pcntl_signal functions? Maybe a info in the admin page at the top? That would at least give us a hint about possible system setup needs.

mmattel · 2016-01-25T13:48:46Z

@PVince81 May I ask you a favour:

Can you check if you get success when you run php -m | grep pcntl
(from my understanding means that it is available/compiled in when you get pcntl as answer)
And does your php5/cli/php.ini (this is the path for Ubuntu) has pcntl_xxx functions disabled in the directive: disable_functions =

I am trying to find a good way identifying if pcntl functions are available on the hosting system.

PVince81 · 2016-01-25T14:40:40Z

@mmattel

php -m | grep pcntl gives no results
no functions are disabled

I'm running openSUSE Tumbleweed

mmattel · 2016-01-25T15:41:33Z

@PVince81 Thanks, I like this answer 😄 ! because it means we can check against and give advices to enable it.

My system is on Ubuntu 14.04 which has pcntl already compiled in and shipped with php.
You can always disable a existing php module for any reason you might have with the disable_functions directive in any appropriate php.ini
Some systems like yours need to get the php module seperately, pls see as example http://rpmfind.net/linux/rpm2html/search.php?query=php5-pcntl

With the current code of this PR, the existance of the pcntl_system function is checked upfront and if not present, no error will be raised, the code can not be interrupted by ctrl-c. If present, you get the ability to clean interrupt a possible long running occ command. I have already informed @oparoz as he has also written a PR for a occ gallery command having the ability for user interruption.

My next steps would be to update the documentation with regards to the pcntl php module and to create a PR to show at the top of the admin page a note if the pcntl php module is missing (may take some time due to regular business...).

Would you mind to test this PR again?
In your current configuration, the code should not create an error and can´t be interrupted.

PVince81 · 2016-01-25T17:19:07Z

apps/files/command/scan.php

@@ -110,9 +120,15 @@ protected function scanFiles($user, $path, $verbose, OutputInterface $output) {
 		} else {
 			$scanner->listen('\OC\Files\Utils\Scanner', 'scanFile', function ($path) use ($output) {
 				$this->filesCounter += 1;
+				if ($this->hasBeenInterrupted()) {
+					throw new \Exception('crtl-c');


=> ctrl, not crtl

PVince81 · 2016-01-25T17:20:03Z

Apart from the copy-pasted typo, seems to work fine 👍 when the PHP module for interruption is not present

mmattel · 2016-01-26T17:19:03Z

Fixed the text typo (crtl-c --> ctrl-c)

mmattel · 2016-01-27T16:18:59Z

@DeepDiver1975 after the ok from @PVince81, merge?

PVince81 · 2016-01-29T07:33:44Z

@icewind1991 @DeepDiver1975 ready for merge ? (needs rereview for pnctl fallback)

Use proper method name Fixed the interruption logic Checks the availability of the pcntl_signal function Fixed typo crtl-c --> ctrl-c one overseen crtl-c typo

rullzer · 2016-01-29T07:43:14Z

Had to rebase since the files app has had a version bump.
But now everything is working fine over here with and without pnctl.

👍

occ scan:files Adding more details in the base print out (II)

DeepDiver1975 added the in progress label Jan 13, 2016

mmattel mentioned this pull request Jan 13, 2016

occ scan:files Adding more details in the base print out #21445

Closed

DeepDiver1975 added this to the 9.0-current milestone Jan 15, 2016

DeepDiver1975 added the 3 - To Review label Jan 15, 2016

PVince81 added the comp:filesystem label Jan 15, 2016

mmattel force-pushed the scan_files_more_details_II branch from 6e7cc78 to cd24ab3 Compare January 16, 2016 21:42

mmattel force-pushed the scan_files_more_details_II branch from cd24ab3 to a9a84b4 Compare January 22, 2016 17:30

mmattel mentioned this pull request Jan 22, 2016

New occ commands to manage thumbnails owncloud/gallery#387

Closed

15 tasks

PVince81 reviewed Jan 25, 2016
View reviewed changes

mmattel force-pushed the scan_files_more_details_II branch from a9a84b4 to ee8d470 Compare January 26, 2016 17:16

mmattel force-pushed the scan_files_more_details_II branch from ee8d470 to c3122ac Compare January 26, 2016 17:42

mmattel mentioned this pull request Jan 28, 2016

implement path_list|-l "..." option for batch scanning. This option t… #21924

Closed

PVince81 mentioned this pull request Jan 29, 2016

Cannot run file scan without pcntl module #22001

Closed

occ scan:files Adding more details in the base print out (II)

e05592d

Use proper method name Fixed the interruption logic Checks the availability of the pcntl_signal function Fixed typo crtl-c --> ctrl-c one overseen crtl-c typo

rullzer force-pushed the scan_files_more_details_II branch from c3122ac to e05592d Compare January 29, 2016 07:42

PVince81 added 4 - To release and removed 3 - To Review labels Jan 29, 2016

DeepDiver1975 added a commit that referenced this pull request Jan 29, 2016

Merge pull request #21701 from owncloud/scan_files_more_details_II

b65a23c

occ scan:files Adding more details in the base print out (II)

DeepDiver1975 merged commit b65a23c into master Jan 29, 2016

DeepDiver1975 deleted the scan_files_more_details_II branch January 29, 2016 10:41

DeepDiver1975 removed the in progress label Jan 29, 2016

This was referenced Jan 29, 2016

Check if the php module 'pcntl' is available for occ command processing #22009

Closed

Source installation: optional PHP module pcntl (II) owncloud-archive/documentation#2113

Merged

lock bot locked as resolved and limited conversation to collaborators Aug 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

occ scan:files Adding more details in the base print out (II) #21701

occ scan:files Adding more details in the base print out (II) #21701

mmattel commented Jan 13, 2016

mention-bot commented Jan 13, 2016

icewind1991 commented Jan 15, 2016

DeepDiver1975 commented Jan 15, 2016

DeepDiver1975 commented Jan 15, 2016

mmattel commented Jan 16, 2016

mmattel commented Jan 20, 2016

PVince81 commented Jan 22, 2016

DeepDiver1975 commented Jan 22, 2016

mmattel commented Jan 22, 2016

mmattel commented Jan 25, 2016

PVince81 commented Jan 25, 2016

mmattel commented Jan 25, 2016

PVince81 Jan 25, 2016

PVince81 commented Jan 25, 2016

mmattel commented Jan 26, 2016

mmattel commented Jan 27, 2016

PVince81 commented Jan 29, 2016

rullzer commented Jan 29, 2016

occ scan:files Adding more details in the base print out (II) #21701

occ scan:files Adding more details in the base print out (II) #21701

Conversation

mmattel commented Jan 13, 2016

mention-bot commented Jan 13, 2016

icewind1991 commented Jan 15, 2016

DeepDiver1975 commented Jan 15, 2016

DeepDiver1975 commented Jan 15, 2016

mmattel commented Jan 16, 2016

mmattel commented Jan 20, 2016

PVince81 commented Jan 22, 2016

DeepDiver1975 commented Jan 22, 2016

mmattel commented Jan 22, 2016

mmattel commented Jan 25, 2016

PVince81 commented Jan 25, 2016

mmattel commented Jan 25, 2016

PVince81 Jan 25, 2016

Choose a reason for hiding this comment

PVince81 commented Jan 25, 2016

mmattel commented Jan 26, 2016

mmattel commented Jan 27, 2016

PVince81 commented Jan 29, 2016

rullzer commented Jan 29, 2016