Enabled large file support (affects 32bit systems) #901

delfer · 2019-10-17T09:51:42Z

Problem: on 32-bit systems PHP can not read/write files over 2GiB

Solution: Add build flags: -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

Rationale: Official PHP manual https://www.php.net/manual/en/intro.filesystem.php

Test

Write 2GiB+ file

<?php
$f = fopen('largefile.tmp', 'wb');
for ($i = 0; $i < 2048+2; $i++) {
	$c = fwrite($f, pack('x1048576'));
	if ($c == 0) {
		trigger_error("Can not write large file", E_USER_ERROR);
	}
}
fclose ($f);

Performance test: https://github.com/rusoft/php-simple-benchmark-script

Test results

armhf = Raspberry Pi 3 model B
amd64 = DigitalOcean $5 droplet

Write 2GiB+ file

Build/arch	Result	Time
Vanilla/armhf	❌ Fail	4m29.433s
Largefile/armhf	✅ Pass	4m28.182s
Vanilla/amd64	✅ Pass	0m6,166s
Largefile/amd64	✅ Pass	0m6,152s

Performance test

Vanilla/armhf

-------------------------------------------------------------------------------------------
|                                  PHP BENCHMARK SCRIPT                                   |
-------------------------------------------------------------------------------------------
Start               : 2019-10-16 20:26:19
Server              : Linux/4.19.66-v7+ armv7l
Platform            : Linux
System              : Debian GNU/Linux 9 (stretch)
CPU                 :
              model : ARMv7 Processor rev 4 (v7l)
              cores : 4
                MHz : 1200MHz
Memory              : 256 Mb available
Benchmark version   : 1.0.34
PHP version         : 7.3.10
  available modules :
           mbstring : yes
               json : yes
             xmlrpc : no
               pcre : yes
Max execution time  : 600 sec
Crypt hash algo     : MD5
-------------------------------------------------------------------------------------------
TEST NAME                      :      SECONDS |       OP/SEC |      OP/SEC/MHz |    MEMORY
-------------------------------------------------------------------------------------------
01_math                        :    7.211 sec |  46.23 kOp/s |  38.52  Ops/MHz |      2 Mb
02_string_concat               :    1.016 sec |   2.53 MOp/s |   2.11 kOps/MHz |  45.61 Mb
03_1_string_number_concat      :    4.704 sec | 354.34 kOp/s | 295.29  Ops/MHz |      4 Mb
03_2_string_number_format      :    4.021 sec | 414.48 kOp/s | 345.40  Ops/MHz |      4 Mb
04_string_simple_functions     :    7.072 sec |  61.27 kOp/s |  51.06  Ops/MHz |      4 Mb
05_string_multibyte            :    2.869 sec |  15.10 kOp/s |  12.58  Ops/MHz |      4 Mb
06_string_manipulation         :   11.912 sec |  36.38 kOp/s |  30.31  Ops/MHz |      4 Mb
07_regex                       :    7.567 sec |  57.26 kOp/s |  47.72  Ops/MHz |      4 Mb
08_1_hashing                   :    5.539 sec |  78.24 kOp/s |  65.20  Ops/MHz |      4 Mb
08_2_crypt                     :   11.384 sec | 292.78  Op/s |   0.24  Ops/MHz |      4 Mb
09_json_encode                 :    8.243 sec |  52.57 kOp/s |  43.81  Ops/MHz |      4 Mb
10_json_decode                 :   12.823 sec |  33.79 kOp/s |  28.16  Ops/MHz |      4 Mb
11_serialize                   :    6.172 sec |  70.21 kOp/s |  58.51  Ops/MHz |      4 Mb
12_unserialize                 :    8.607 sec |  50.35 kOp/s |  41.95  Ops/MHz |      4 Mb
13_array_fill                  :    7.089 sec |   2.33 MOp/s |   1.94 kOps/MHz |     10 Mb
14_array_range                 :    0.638 sec |  51.72 kOp/s |  43.10  Ops/MHz |     10 Mb
14_array_unset                 :    7.262 sec |   2.27 MOp/s |   1.89 kOps/MHz |     10 Mb
15_loops                       :    4.476 sec |  14.89 MOp/s |  12.41 kOps/MHz |     10 Mb
16_loop_ifelse                 :    3.383 sec |   4.93 MOp/s |   4.11 kOps/MHz |     10 Mb
17_loop_ternary                :    6.759 sec |   2.47 MOp/s |   2.05 kOps/MHz |     10 Mb
18_1_loop_defined_access       :    1.973 sec |   3.38 MOp/s |   2.82 kOps/MHz |     10 Mb
18_2_loop_undefined_access     :   14.612 sec | 456.24 kOp/s | 380.20  Ops/MHz |     10 Mb
19_type_functions              :    3.459 sec | 289.09 kOp/s | 240.91  Ops/MHz |     10 Mb
20_type_conversion             :    3.489 sec | 286.63 kOp/s | 238.86  Ops/MHz |     10 Mb
21_0_loop_exception_none       :    0.167 sec |   7.97 MOp/s |   6.64 kOps/MHz |     10 Mb
21_1_loop_exception_try        :    0.191 sec |   6.99 MOp/s |   5.83 kOps/MHz |     10 Mb
21_2_loop_exception_catch      :    5.349 sec | 249.25 kOp/s | 207.70  Ops/MHz |     10 Mb
22_loop_null_op                :    5.375 sec |   3.10 MOp/s |   2.58 kOps/MHz |     10 Mb
23_loop_spaceship_op           :    5.236 sec |   3.18 MOp/s |   2.65 kOps/MHz |     10 Mb
24_xmlrpc_encode               :    -.--- sec |     -.--Op/s |     -.--Ops/MHz |         0
25_xmlrpc_decode               :    -.--- sec |     -.--Op/s |     -.--Ops/MHz |         0
26_1_class_public_properties   :    0.411 sec |   4.06 MOp/s |   3.38 kOps/MHz |     10 Mb
26_2_class_getter_setter       :    1.394 sec |   1.20 MOp/s | 996.36  Ops/MHz |     10 Mb
26_3_class_magic_methods       :    2.910 sec | 572.81 kOp/s | 477.34  Ops/MHz |     10 Mb
-------------------------------------------------------------------------------------------
Total time:                    :  173.314 sec |   1.16 MOp/s | 963.79  Ops/MHz |
Current PHP memory usage:      :       10 Mb
Peak PHP memory usage:         :     42.2 Mb

Largefile/armhf

-------------------------------------------------------------------------------------------
|                                  PHP BENCHMARK SCRIPT                                   |
-------------------------------------------------------------------------------------------
Start               : 2019-10-16 21:08:32
Server              : Linux/4.19.66-v7+ armv7l
Platform            : Linux
System              : Debian GNU/Linux 9 (stretch)
CPU                 :
              model : ARMv7 Processor rev 4 (v7l)
              cores : 4
                MHz : 1200MHz
Memory              : 256 Mb available
Benchmark version   : 1.0.34
PHP version         : 7.3.10
  available modules :
           mbstring : yes
               json : yes
             xmlrpc : no
               pcre : yes
Max execution time  : 600 sec
Crypt hash algo     : MD5
-------------------------------------------------------------------------------------------
TEST NAME                      :      SECONDS |       OP/SEC |      OP/SEC/MHz |    MEMORY
-------------------------------------------------------------------------------------------
01_math                        :    7.070 sec |  47.14 kOp/s |  39.29  Ops/MHz |      2 Mb
02_string_concat               :    1.017 sec |   2.52 MOp/s |   2.10 kOps/MHz |  45.61 Mb
03_1_string_number_concat      :    4.600 sec | 362.35 kOp/s | 301.96  Ops/MHz |      4 Mb
03_2_string_number_format      :    3.946 sec | 422.39 kOp/s | 351.99  Ops/MHz |      4 Mb
04_string_simple_functions     :    7.036 sec |  61.59 kOp/s |  51.32  Ops/MHz |      4 Mb
05_string_multibyte            :    2.858 sec |  15.16 kOp/s |  12.63  Ops/MHz |      4 Mb
06_string_manipulation         :   11.857 sec |  36.55 kOp/s |  30.46  Ops/MHz |      4 Mb
07_regex                       :    7.508 sec |  57.72 kOp/s |  48.10  Ops/MHz |      4 Mb
08_1_hashing                   :    5.552 sec |  78.05 kOp/s |  65.04  Ops/MHz |      4 Mb
08_2_crypt                     :   11.380 sec | 292.89  Op/s |   0.24  Ops/MHz |      4 Mb
09_json_encode                 :    8.108 sec |  53.45 kOp/s |  44.54  Ops/MHz |      4 Mb
10_json_decode                 :   12.741 sec |  34.01 kOp/s |  28.34  Ops/MHz |      4 Mb
11_serialize                   :    6.244 sec |  69.40 kOp/s |  57.83  Ops/MHz |      4 Mb
12_unserialize                 :    8.428 sec |  51.42 kOp/s |  42.85  Ops/MHz |      4 Mb
13_array_fill                  :    6.907 sec |   2.39 MOp/s |   1.99 kOps/MHz |     10 Mb
14_array_range                 :    0.636 sec |  51.87 kOp/s |  43.22  Ops/MHz |     10 Mb
14_array_unset                 :    6.979 sec |   2.36 MOp/s |   1.97 kOps/MHz |     10 Mb
15_loops                       :    4.463 sec |  14.94 MOp/s |  12.45 kOps/MHz |     10 Mb
16_loop_ifelse                 :    3.411 sec |   4.89 MOp/s |   4.07 kOps/MHz |     10 Mb
17_loop_ternary                :    6.734 sec |   2.48 MOp/s |   2.06 kOps/MHz |     10 Mb
18_1_loop_defined_access       :    1.954 sec |   3.41 MOp/s |   2.84 kOps/MHz |     10 Mb
18_2_loop_undefined_access     :   14.217 sec | 468.91 kOp/s | 390.76  Ops/MHz |     10 Mb
19_type_functions              :    3.476 sec | 287.71 kOp/s | 239.76  Ops/MHz |     10 Mb
20_type_conversion             :    3.484 sec | 287.00 kOp/s | 239.17  Ops/MHz |     10 Mb
21_0_loop_exception_none       :    0.167 sec |   7.98 MOp/s |   6.65 kOps/MHz |     10 Mb
21_1_loop_exception_try        :    0.191 sec |   6.99 MOp/s |   5.82 kOps/MHz |     10 Mb
21_2_loop_exception_catch      :    5.182 sec | 257.29 kOp/s | 214.41  Ops/MHz |     10 Mb
22_loop_null_op                :    5.388 sec |   3.09 MOp/s |   2.58 kOps/MHz |     10 Mb
23_loop_spaceship_op           :    4.793 sec |   3.48 MOp/s |   2.90 kOps/MHz |     10 Mb
24_xmlrpc_encode               :    -.--- sec |     -.--Op/s |     -.--Ops/MHz |         0
25_xmlrpc_decode               :    -.--- sec |     -.--Op/s |     -.--Ops/MHz |         0
26_1_class_public_properties   :    0.411 sec |   4.05 MOp/s |   3.38 kOps/MHz |     10 Mb
26_2_class_getter_setter       :    1.380 sec |   1.21 MOp/s |   1.01 kOps/MHz |     10 Mb
26_3_class_magic_methods       :    2.872 sec | 580.29 kOp/s | 483.57  Ops/MHz |     10 Mb
-------------------------------------------------------------------------------------------
Total time:                    :  170.989 sec |   1.17 MOp/s | 976.90  Ops/MHz |
Current PHP memory usage:      :       10 Mb
Peak PHP memory usage:         :     42.2 Mb

Vanilla/amd64

-------------------------------------------------------------------------------------------
|                                  PHP BENCHMARK SCRIPT                                   |
-------------------------------------------------------------------------------------------
Start               : 2019-10-17 08:48:45
Server              : Linux/4.16.3-301.fc28.x86_64 x86_64
Platform            : Linux
System              : Debian GNU/Linux 9 (stretch)
CPU                 :
              model : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
              cores : 1
                MHz : 2199.998MHz
Memory              : 256 Mb available
Benchmark version   : 1.0.34
PHP version         : 7.3.10
  available modules :
           mbstring : yes
               json : yes
             xmlrpc : no
               pcre : yes
Max execution time  : 600 sec
Crypt hash algo     : MD5
-------------------------------------------------------------------------------------------
TEST NAME                      :      SECONDS |       OP/SEC |      OP/SEC/MHz |    MEMORY
-------------------------------------------------------------------------------------------
01_math                        :    3.480 sec | 287.34 kOp/s | 130.61  Ops/MHz |      4 Mb
02_string_concat               :    0.455 sec |  16.93 MOp/s |   7.70 kOps/MHz | 128.84 Mb
03_1_string_number_concat      :    2.397 sec |   2.09 MOp/s | 948.18  Ops/MHz |      4 Mb
03_2_string_number_format      :    2.156 sec |   2.32 MOp/s |   1.05 kOps/MHz |      4 Mb
04_string_simple_functions     :    3.706 sec | 350.78 kOp/s | 159.45  Ops/MHz |      4 Mb
05_string_multibyte            :    3.260 sec |  39.87 kOp/s |  18.12  Ops/MHz |      4 Mb
06_string_manipulation         :    4.237 sec | 306.85 kOp/s | 139.48  Ops/MHz |      4 Mb
07_regex                       :    3.217 sec | 404.12 kOp/s | 183.69  Ops/MHz |      4 Mb
08_1_hashing                   :    4.143 sec | 313.79 kOp/s | 142.63  Ops/MHz |      4 Mb
08_2_crypt                     :   12.370 sec | 808.41  Op/s |   0.37  Ops/MHz |      4 Mb
09_json_encode                 :    3.824 sec | 339.96 kOp/s | 154.53  Ops/MHz |      4 Mb
10_json_decode                 :    6.482 sec | 200.56 kOp/s |  91.17  Ops/MHz |      4 Mb
11_serialize                   :    2.824 sec | 460.42 kOp/s | 209.28  Ops/MHz |      4 Mb
12_unserialize                 :    4.474 sec | 290.57 kOp/s | 132.08  Ops/MHz |      4 Mb
13_array_fill                  :    3.502 sec |  14.28 MOp/s |   6.49 kOps/MHz |     12 Mb
14_array_range                 :    0.841 sec | 118.91 kOp/s |  54.05  Ops/MHz |     12 Mb
14_array_unset                 :    2.586 sec |  19.34 MOp/s |   8.79 kOps/MHz |     12 Mb
15_loops                       :    2.041 sec |  97.99 MOp/s |  44.54 kOps/MHz |      4 Mb
16_loop_ifelse                 :    1.104 sec |  45.28 MOp/s |  20.58 kOps/MHz |      4 Mb
17_loop_ternary                :    2.841 sec |  17.60 MOp/s |   8.00 kOps/MHz |      4 Mb
18_1_loop_defined_access       :    0.797 sec |  25.10 MOp/s |  11.41 kOps/MHz |      4 Mb
18_2_loop_undefined_access     :    5.776 sec |   3.46 MOp/s |   1.57 kOps/MHz |      4 Mb
19_type_functions              :    1.344 sec |   2.23 MOp/s |   1.01 kOps/MHz |      4 Mb
20_type_conversion             :    1.528 sec |   1.96 MOp/s | 892.67  Ops/MHz |      4 Mb
21_0_loop_exception_none       :    0.059 sec |  67.26 MOp/s |  30.57 kOps/MHz |      4 Mb
21_1_loop_exception_try        :    0.068 sec |  58.74 MOp/s |  26.70 kOps/MHz |      4 Mb
21_2_loop_exception_catch      :    3.226 sec |   1.24 MOp/s | 563.62  Ops/MHz |      4 Mb
22_loop_null_op                :    2.088 sec |  23.95 MOp/s |  10.89 kOps/MHz |      4 Mb
23_loop_spaceship_op           :    2.112 sec |  23.67 MOp/s |  10.76 kOps/MHz |      4 Mb
24_xmlrpc_encode               :    -.--- sec |     -.--Op/s |     -.--Ops/MHz |         0
25_xmlrpc_decode               :    -.--- sec |     -.--Op/s |     -.--Ops/MHz |         0
26_1_class_public_properties   :    0.195 sec |  25.68 MOp/s |  11.67 kOps/MHz |      4 Mb
26_2_class_getter_setter       :    0.529 sec |   9.45 MOp/s |   4.30 kOps/MHz |      4 Mb
26_3_class_magic_methods       :    1.088 sec |   4.60 MOp/s |   2.09 kOps/MHz |      4 Mb
-------------------------------------------------------------------------------------------
Total time:                    :   88.748 sec |   6.79 MOp/s |   3.09 kOps/MHz |
Current PHP memory usage:      :        4 Mb
Peak PHP memory usage:         :   125.48 Mb

Largefile/amd64

-------------------------------------------------------------------------------------------
|                                  PHP BENCHMARK SCRIPT                                   |
-------------------------------------------------------------------------------------------
Start               : 2019-10-17 09:05:25
Server              : Linux/4.16.3-301.fc28.x86_64 x86_64
Platform            : Linux
System              : Debian GNU/Linux 9 (stretch)
CPU                 :
              model : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
              cores : 1
                MHz : 2199.998MHz
Memory              : 256 Mb available
Benchmark version   : 1.0.34
PHP version         : 7.3.10
  available modules :
           mbstring : yes
               json : yes
             xmlrpc : no
               pcre : yes
Max execution time  : 600 sec
Crypt hash algo     : MD5
-------------------------------------------------------------------------------------------
TEST NAME                      :      SECONDS |       OP/SEC |      OP/SEC/MHz |    MEMORY
-------------------------------------------------------------------------------------------
01_math                        :    3.341 sec | 299.29 kOp/s | 136.04  Ops/MHz |      4 Mb
02_string_concat               :    0.453 sec |  17.01 MOp/s |   7.73 kOps/MHz | 128.84 Mb
03_1_string_number_concat      :    2.533 sec |   1.97 MOp/s | 897.29  Ops/MHz |      4 Mb
03_2_string_number_format      :    2.264 sec |   2.21 MOp/s |   1.00 kOps/MHz |      4 Mb
04_string_simple_functions     :    2.594 sec | 501.20 kOp/s | 227.82  Ops/MHz |      4 Mb
05_string_multibyte            :    1.912 sec |  68.00 kOp/s |  30.91  Ops/MHz |      4 Mb
06_string_manipulation         :    4.433 sec | 293.24 kOp/s | 133.29  Ops/MHz |      4 Mb
07_regex                       :    3.673 sec | 353.92 kOp/s | 160.87  Ops/MHz |      4 Mb
08_1_hashing                   :    4.602 sec | 282.50 kOp/s | 128.41  Ops/MHz |      4 Mb
08_2_crypt                     :   12.689 sec | 788.08  Op/s |   0.36  Ops/MHz |      4 Mb
09_json_encode                 :    4.892 sec | 265.74 kOp/s | 120.79  Ops/MHz |      4 Mb
10_json_decode                 :    6.010 sec | 216.32 kOp/s |  98.33  Ops/MHz |      4 Mb
11_serialize                   :    2.941 sec | 441.99 kOp/s | 200.90  Ops/MHz |      4 Mb
12_unserialize                 :    4.958 sec | 262.20 kOp/s | 119.18  Ops/MHz |      4 Mb
13_array_fill                  :    3.644 sec |  13.72 MOp/s |   6.24 kOps/MHz |     12 Mb
14_array_range                 :    0.957 sec | 104.51 kOp/s |  47.51  Ops/MHz |     12 Mb
14_array_unset                 :    2.913 sec |  17.16 MOp/s |   7.80 kOps/MHz |     12 Mb
15_loops                       :    1.636 sec | 122.24 MOp/s |  55.57 kOps/MHz |      4 Mb
16_loop_ifelse                 :    1.490 sec |  33.56 MOp/s |  15.25 kOps/MHz |      4 Mb
17_loop_ternary                :    3.206 sec |  15.59 MOp/s |   7.09 kOps/MHz |      4 Mb
18_1_loop_defined_access       :    0.816 sec |  24.52 MOp/s |  11.14 kOps/MHz |      4 Mb
18_2_loop_undefined_access     :    6.671 sec |   3.00 MOp/s |   1.36 kOps/MHz |      4 Mb
19_type_functions              :    1.833 sec |   1.64 MOp/s | 743.86  Ops/MHz |      4 Mb
20_type_conversion             :    1.558 sec |   1.93 MOp/s | 875.51  Ops/MHz |      4 Mb
21_0_loop_exception_none       :    0.059 sec |  68.11 MOp/s |  30.96 kOps/MHz |      4 Mb
21_1_loop_exception_try        :    0.082 sec |  48.76 MOp/s |  22.16 kOps/MHz |      4 Mb
21_2_loop_exception_catch      :    2.293 sec |   1.74 MOp/s | 792.90  Ops/MHz |      4 Mb
22_loop_null_op                :    2.404 sec |  20.79 MOp/s |   9.45 kOps/MHz |      4 Mb
23_loop_spaceship_op           :    2.518 sec |  19.85 MOp/s |   9.02 kOps/MHz |      4 Mb
24_xmlrpc_encode               :    -.--- sec |     -.--Op/s |     -.--Ops/MHz |         0
25_xmlrpc_decode               :    -.--- sec |     -.--Op/s |     -.--Ops/MHz |         0
26_1_class_public_properties   :    0.176 sec |  28.47 MOp/s |  12.94 kOps/MHz |      4 Mb
26_2_class_getter_setter       :    0.539 sec |   9.28 MOp/s |   4.22 kOps/MHz |      4 Mb
26_3_class_magic_methods       :    1.150 sec |   4.35 MOp/s |   1.98 kOps/MHz |      4 Mb
-------------------------------------------------------------------------------------------
Total time:                    :   91.240 sec |   6.60 MOp/s |   3.00 kOps/MHz |
Current PHP memory usage:      :        4 Mb
Peak PHP memory usage:         :   125.48 Mb

Conclusion
This PR fixes problem, suggested by PHP authors and does not affect performance.
PR can be merged.

Build flags added: -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 According to https://www.php.net/manual/en/intro.filesystem.php

Changes: - docker-library/php@da77ea9: Merge pull request docker-library/php#901 from delfer/largefile - docker-library/php@88a8a1e: Add comment to remind us why we added largefile support - docker-library/php@168caf5: Enabled large file support (affects 32bit systems)

This was referenced Oct 17, 2019

NC16.0.3 Upload fails with files 2GiB in 32bit instances (again) nextcloud/server#16431

Closed

Enable large file support (on 32bit systems) #839

Closed

delfer and others added 2 commits October 21, 2019 10:33

Enabled large file support (affects 32bit systems)

168caf5

Build flags added: -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 According to https://www.php.net/manual/en/intro.filesystem.php

Add comment to remind us why we added largefile support

88a8a1e

tianon force-pushed the largefile branch from 77ea56a to 88a8a1e Compare October 21, 2019 17:36

tianon approved these changes Oct 21, 2019

View reviewed changes

tianon merged commit da77ea9 into docker-library:master Oct 21, 2019

yosifkit mentioned this pull request Oct 23, 2019

Update php docker-library/official-images#6851

Merged

StableNarwhal mentioned this pull request Mar 16, 2020

"Error while copying file to target location (copied bytes: 0, expected filesize: -1 )" nextcloud/server#18720

Closed

Vigasto mentioned this pull request Sep 21, 2020

Error assembling chunks from large uploads (>4GB) nextcloud/server#22936

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabled large file support (affects 32bit systems) #901

Enabled large file support (affects 32bit systems) #901

delfer commented Oct 17, 2019

Enabled large file support (affects 32bit systems) #901

Enabled large file support (affects 32bit systems) #901

Conversation

delfer commented Oct 17, 2019