Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducible builds: build_path showing variation #231

Closed
conorsch opened this issue Mar 17, 2021 · 3 comments
Closed

Reproducible builds: build_path showing variation #231

conorsch opened this issue Mar 17, 2021 · 3 comments

Comments

@conorsch
Copy link
Contributor

Discovered while debugging #228. See the diff from two package builds of securedrop-client, built from the 0.4.1 tag, but built in slightly different locations:

│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/direct_url.json
│ │ │ │   --- /tmp/diffoscope_ee2yspmx/tmpiinoiy6_/0/1050.json
│ │ │ ├── +++ /tmp/diffoscope_ee2yspmx/tmpfc5ilrvz/0/1050.json
│ │ │ │┄ Similarity: 0.0%
│ │ │ │┄ Differences: {
│ │ │ │┄   "replace": {
│ │ │ │┄     "dir_info": {},
│ │ │ │┄     "url": "file:///home/user/src/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │┄   }
│ │ │ │┄ }
│ │ │ │ @@ -1,4 +1,4 @@
│ │ │ │  {
│ │ │ │      "dir_info": {},
│ │ │ │ -    "url": "file:///home/user/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │ +    "url": "file:///home/user/src/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │  }

Depending on the buildroot for the package build, we're seeing artifacts within the deb package change. That's not good. A few questions:

  1. Can we catch this type of variations by enabling +build_path for reprotest in CI?
  2. Have we always had this problem, or did it start occurring recently? (We believe it's new.)
  3. Are there documented or unexplained differences in tooling at work here? For example, are newer versions of pip and/or setuptools being pulled in dynamically? Does the behavior change inside a virtualenv vs using dist-packages?

Let's do some research and see what's what.

full diffoscope output
--- debugging/conor/securedrop-client_0.4.1+buster_all.deb
+++ debugging/mickael/securedrop-client_0.4.1+buster_all.deb
├── file list
│ @@ -1,3 +1,3 @@
│  -rw-r--r--   0        0        0        4 2021-03-17 18:20:12.000000 debian-binary
│ --rw-r--r--   0        0        0    33320 2021-03-17 18:20:12.000000 control.tar.xz
│ --rw-r--r--   0        0        0  7870096 2021-03-17 18:20:12.000000 data.tar.xz
│ +-rw-r--r--   0        0        0    33328 2021-03-17 18:20:12.000000 control.tar.xz
│ +-rw-r--r--   0        0        0  7869464 2021-03-17 18:20:12.000000 data.tar.xz
├── control.tar.xz
│ ├── control.tar
│ │ ├── ./md5sums
│ │ │ ├── ./md5sums
│ │ │ │┄ Files differ
├── data.tar.xz
│ ├── data.tar
│ │ ├── file list
│ │ │ @@ -1044,15 +1044,15 @@
│ │ │  drwxr-xr-x   0 root         (0) root         (0)        0 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/
│ │ │  -rw-r--r--   0 root         (0) root         (0)        4 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/INSTALLER
│ │ │  -rw-r--r--   0 root         (0) root         (0)    34520 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/LICENSE
│ │ │  -rw-r--r--   0 root         (0) root         (0)    18867 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/METADATA
│ │ │  -rw-r--r--   0 root         (0) root         (0)    12536 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/RECORD
│ │ │  -rw-r--r--   0 root         (0) root         (0)        0 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/REQUESTED
│ │ │  -rw-r--r--   0 root         (0) root         (0)       92 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/WHEEL
│ │ │ --rw-r--r--   0 root         (0) root         (0)      115 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/direct_url.json
│ │ │ +-rw-r--r--   0 root         (0) root         (0)      119 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/direct_url.json
│ │ │  -rw-r--r--   0 root         (0) root         (0)       57 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/entry_points.txt
│ │ │  -rw-r--r--   0 root         (0) root         (0)       18 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/top_level.txt
│ │ │  drwxr-xr-x   0 root         (0) root         (0)        0 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/
│ │ │  -rw-r--r--   0 root         (0) root         (0)        4 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/INSTALLER
│ │ │  -rw-r--r--   0 root         (0) root         (0)    35149 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/LICENSE
│ │ │  -rw-r--r--   0 root         (0) root         (0)    10999 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/METADATA
│ │ │  -rw-r--r--   0 root         (0) root         (0)      787 2021-03-17 18:20:12.000000 ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_sdk-0.2.0.dist-info/RECORD
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/Mako-1.0.7.dist-info/RECORD
│ │ │ @@ -1,8 +1,8 @@
│ │ │ -../../../bin/mako-render,sha256=FkzHpLvY33u9MXaS4pmE_2yRxL6CsLwmAQudiHTnJ00,375
│ │ │ +../../../bin/mako-render,sha256=J_dJvEz-DX0bCdRoDjmoAIuLN1LLIcFa9I0XGVOTX_0,379
│ │ │  Mako-1.0.7.dist-info/AUTHORS,sha256=Io2Vw70mjYS7yFcUuJxhIGiMUQt8FWJuxiiwyUW1WRg,282
│ │ │  Mako-1.0.7.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  Mako-1.0.7.dist-info/LICENSE,sha256=hPb4eYmQI51pe21iYx3EApYtT7OJWVXqkm3OeVa69xs,1217
│ │ │  Mako-1.0.7.dist-info/METADATA,sha256=GjlrwIUAxPMmxGogZ0bavg5adWJ0m7cxPmQ-AiFBh_M,2233
│ │ │  Mako-1.0.7.dist-info/RECORD,,
│ │ │  Mako-1.0.7.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  Mako-1.0.7.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/alembic-1.0.2.dist-info/RECORD
│ │ │ @@ -1,8 +1,8 @@
│ │ │ -../../../bin/alembic,sha256=wbvkVmx93k_oUgj13pLFT-Bio7tnA2HxHK7ZVkCVJpg,375
│ │ │ +../../../bin/alembic,sha256=G--41PkEELrFREBOM_8CYfypz5WqdRFUWNaStvYVcm4,379
│ │ │  alembic-1.0.2.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  alembic-1.0.2.dist-info/LICENSE,sha256=hJsWUpYBfNjhGU91S2q9caPyPu1klJdvePG7esIrFAI,1184
│ │ │  alembic-1.0.2.dist-info/METADATA,sha256=ly9Z052_B4p_pxRop_DkrF0P70qePxD5tIu-1rHZivs,6038
│ │ │  alembic-1.0.2.dist-info/RECORD,,
│ │ │  alembic-1.0.2.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  alembic-1.0.2.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
│ │ │  alembic-1.0.2.dist-info/entry_points.txt,sha256=jOSnN_2fhU8xzDQ50rdNr425J8kf_exuY8GrAo1daz8,49
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/chardet-3.0.4.dist-info/RECORD
│ │ │ @@ -1,8 +1,8 @@
│ │ │ -../../../bin/chardetect,sha256=wlBmZ3qG8Rprbmz6bEZHndD6Zq0l0Y17oei7wj-6FkY,383
│ │ │ +../../../bin/chardetect,sha256=x7wS4T3NKiV9U3gkPqPi1KfZ-GQjhX5ORcMrwTYSgbA,387
│ │ │  chardet-3.0.4.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  chardet-3.0.4.dist-info/LICENSE,sha256=YJXp_6d33SKDn3gBqoRbMcntB_PWv4om3F0t7IzMDvM,26432
│ │ │  chardet-3.0.4.dist-info/METADATA,sha256=o6XNN41EUioeDnklH1-8haOSjI60AkAaI823ANFkOM4,3304
│ │ │  chardet-3.0.4.dist-info/RECORD,,
│ │ │  chardet-3.0.4.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  chardet-3.0.4.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
│ │ │  chardet-3.0.4.dist-info/entry_points.txt,sha256=fAMmhu5eJ-zAJ-smfqQwRClQ3-nozOCmvJ6-E8lgGJo,60
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/pip-21.0.1.dist-info/RECORD
│ │ │ @@ -1,10 +1,10 @@
│ │ │ -../../../bin/pip,sha256=Ot1LQ47VGHnVx424wHTA0HJPDJKDg7g_Enpt_qV3hA0,387
│ │ │ -../../../bin/pip3,sha256=Ot1LQ47VGHnVx424wHTA0HJPDJKDg7g_Enpt_qV3hA0,387
│ │ │ -../../../bin/pip3.7,sha256=Ot1LQ47VGHnVx424wHTA0HJPDJKDg7g_Enpt_qV3hA0,387
│ │ │ +../../../bin/pip,sha256=ISHOtLHBOkdso4f5K5O6sgYKb1Y2GQOsibXK4co4Emw,391
│ │ │ +../../../bin/pip3,sha256=ISHOtLHBOkdso4f5K5O6sgYKb1Y2GQOsibXK4co4Emw,391
│ │ │ +../../../bin/pip3.7,sha256=ISHOtLHBOkdso4f5K5O6sgYKb1Y2GQOsibXK4co4Emw,391
│ │ │  pip-21.0.1.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  pip-21.0.1.dist-info/LICENSE.txt,sha256=ejlw8iXn2TntLdOpADqlISSc1qhJJgiYAKMZmq713Gk,1110
│ │ │  pip-21.0.1.dist-info/METADATA,sha256=a6mCPyb1qd3cdVI5OorlrDhSN3HHYiN8feJrxmL4QgY,4168
│ │ │  pip-21.0.1.dist-info/RECORD,,
│ │ │  pip-21.0.1.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
│ │ │  pip-21.0.1.dist-info/entry_points.txt,sha256=5ExSa1s54zSPNA_1epJn5SX06786S8k5YHwskMvVYzw,125
│ │ │  pip-21.0.1.dist-info/top_level.txt,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/RECORD
│ │ │ @@ -1,15 +1,15 @@
│ │ │ -../../../bin/sd-client,sha256=A4hlyqylu8nTK6zfXDbsIF1XU_lFGbhOzJXF4gq2fnA,380
│ │ │ +../../../bin/sd-client,sha256=mqe-XAEPhq3yYJIkernAvpAcLjsjooNpliiVeeG5Eq0,384
│ │ │  securedrop_client-0.4.1.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  securedrop_client-0.4.1.dist-info/LICENSE,sha256=dql8h4yceoMhuzlcK0TT_i-NgTFNIZsgE47Q4t3dUYI,34520
│ │ │  securedrop_client-0.4.1.dist-info/METADATA,sha256=9v90HnwVWyOt3Q5jEMxY2FUJH9za1CUeMuXGOmalhlU,18867
│ │ │  securedrop_client-0.4.1.dist-info/RECORD,,
│ │ │  securedrop_client-0.4.1.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  securedrop_client-0.4.1.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
│ │ │ -securedrop_client-0.4.1.dist-info/direct_url.json,sha256=niQE_MymeZaFP2DXBdybmQCZ7xGJQ0Lmt3Q_tf1rGcg,115
│ │ │ +securedrop_client-0.4.1.dist-info/direct_url.json,sha256=XJy52Q1fgkeMNe_Dj6D5h-54lG8zwiELec8kUsgdRIo,119
│ │ │  securedrop_client-0.4.1.dist-info/entry_points.txt,sha256=Ky6pbwDDvJfpk4vu3wLB8MB9eSCxa_fzkAD898WY2fE,57
│ │ │  securedrop_client-0.4.1.dist-info/top_level.txt,sha256=-kikgNWO_1DaWx2DRA8yo8Ibcpu5A2Epjx49jAxmF7A,18
│ │ │  securedrop_client/__init__.py,sha256=pMtTmSUht-XtbR_7Doz6bsQqopJJd8rZ8I8zy2HwwoA,22
│ │ │  securedrop_client/__main__.py,sha256=K98BO7GszyT6TPeQmNevXJtJh8StR-HG0rr8GnOIaSc,28
│ │ │  securedrop_client/api_jobs/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
│ │ │  securedrop_client/api_jobs/base.py,sha256=Wg7NsH7eBo7dgcQL-r7Sfwc2PFuCs0e2z5uW_uzY9CM,3923
│ │ │  securedrop_client/api_jobs/downloads.py,sha256=siO9ANtRmx2qTLWWvnH7kUKiDq7UZmwnUxcseABJ2pM,14352
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/securedrop_client-0.4.1.dist-info/direct_url.json
│ │ │ │   --- /tmp/diffoscope_9q8otjqv/tmpvwaebm61/0/1050.json
│ │ │ ├── +++ /tmp/diffoscope_9q8otjqv/tmpdo8n3v94/0/1050.json
│ │ │ │┄ Similarity: 0.0%
│ │ │ │┄ Differences: {
│ │ │ │┄   "replace": {
│ │ │ │┄     "dir_info": {},
│ │ │ │┄     "url": "file:///home/user/src/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │┄   }
│ │ │ │┄ }
│ │ │ │ @@ -1,4 +1,4 @@
│ │ │ │  {
│ │ │ │      "dir_info": {},
│ │ │ │ -    "url": "file:///home/user/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │ +    "url": "file:///home/user/src/securedrop-debian-packaging/build/debbuild/packaging/securedrop-client"
│ │ │ │  }
│ │ ├── ./opt/venvs/securedrop-client/lib/python3.7/site-packages/wheel-0.36.2.dist-info/RECORD
│ │ │ @@ -1,8 +1,8 @@
│ │ │ -../../../bin/wheel,sha256=J_gtOIynaYEwTx71xxcMqAgMWxwI15CAex3b2zBN5XU,374
│ │ │ +../../../bin/wheel,sha256=spl-_M0tfI2y45a1gtPJi1O8MZmfEflLBpcTg6czOeo,378
│ │ │  wheel-0.36.2.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
│ │ │  wheel-0.36.2.dist-info/LICENSE.txt,sha256=zKniDGrx_Pv2lAjzd3aShsvuvN7TNhAMm0o_NfvmNeQ,1125
│ │ │  wheel-0.36.2.dist-info/METADATA,sha256=w96pAXMHZWzFy-dwqGZqXbxu5Hup3cb3OHujhCuWfDs,2277
│ │ │  wheel-0.36.2.dist-info/RECORD,,
│ │ │  wheel-0.36.2.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
│ │ │  wheel-0.36.2.dist-info/entry_points.txt,sha256=N8HbYFST3yrNQYeB2wXWBEPUhFsEtKNRPaCFGJPyqyc,108
│ │ │  wheel-0.36.2.dist-info/top_level.txt,sha256=HxSBIbgEstMPe4eFawhA66Mq-QYHMopXVoAncfjb_1c,6
conorsch pushed a commit that referenced this issue Apr 7, 2021
We were already checking that building a wheel twice yields the same
checksum both times, but we weren't confirming that the checksum matched
what's already committed to version control. Let's do that.

Still leaving the 'reprotest' check enabled because it supports multiple
methods for checking reproducible, which should help us shake out
problems like #231.
@conorsch
Copy link
Contributor Author

conorsch commented Apr 8, 2021

Dug into this a bit. The build path is embedded inside direct_url.json, as shown in the diffoscope output above. That's following the PEP 610 spec, which appears to be geared toward VCS URLs, not local filepaths. I don't see any reason for us to include that in our debian packages, since it's not required information for the packages to work correctly once installed.

Snipped out that file removes the specific variation reported above, but has a side-effect of altering the checksumming in the RECORD files, according to PEP 376 and PEP 427. Similarly, I don't see any value in shipping these files, given our use of Debian packages for installing Python dependencies. From PEP 427:

RECORD is a list of (almost) all the files in the wheel and their secure hashes. Unlike PEP 376, every file except RECORD, which cannot contain a hash of itself, must include its hash. The hash algorithm must be sha256 or better; specifically, md5 and sha1 are not permitted, as signed wheel files rely on the strong hashes in RECORD to validate the integrity of the archive.

@kushaldas It seems to me we can snip these files out of the Debian package, via the strip-nondeterminism step, to get better reproducibility. Do you see any problems with that?

conorsch pushed a commit that referenced this issue Apr 8, 2021
In #231 we noticed that the filesystem path used to build the debian
packages was breaking reproducibility. That's due to the filepath being
recorded inside `direct_url.json`, as stipulated by PEP610.

Also relevant is PEP427, which describes the `RECORD` file for wheels.
Here we remove that file, as well, to ensure full reproducibility
regardless of path.
@kushaldas
Copy link
Contributor

@kushaldas It seems to me we can snip these files out of the Debian package, via the strip-nondeterminism step, to get better reproducibility. Do you see any problems with that?

I am okay with removing them, I still asked the upstream to get some details (if any).

conorsch pushed a commit that referenced this issue Apr 8, 2021
In #231 we noticed that the filesystem path used to build the debian
packages was breaking reproducibility. That's due to the filepath being
recorded inside `direct_url.json`, as stipulated by PEP610.

Also relevant is PEP427, which describes the `RECORD` file for wheels.
Here we remove that file, as well, to ensure full reproducibility
regardless of path.
conorsch pushed a commit that referenced this issue Apr 8, 2021
We were already checking that building a wheel twice yields the same
checksum both times, but we weren't confirming that the checksum matched
what's already committed to version control. Let's do that.

Still leaving the 'reprotest' check enabled because it supports multiple
methods for checking reproducible, which should help us shake out
problems like #231.
@conorsch
Copy link
Contributor Author

conorsch commented Apr 9, 2021

Resolved by #244.

@conorsch conorsch closed this as completed Apr 9, 2021
conorsch pushed a commit that referenced this issue Apr 14, 2021
We were already checking that building a wheel twice yields the same
checksum both times, but we weren't confirming that the checksum matched
what's already committed to version control. Let's do that.

Still leaving the 'reprotest' check enabled because it supports multiple
methods for checking reproducible, which should help us shake out
problems like #231.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants