Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release v1.5.4 #3487

Merged
merged 668 commits into from
Feb 10, 2023
Merged

release v1.5.4 #3487

merged 668 commits into from
Feb 10, 2023
This pull request is big! We’re only showing the most recent 250 commits.

Commits on Dec 14, 2022

  1. Refactor progress bar & summary line logic

    * Centralize the logic about whether to print the progress bar or not in
      the `*_PROGRESS()` macros.
    * Centralize the logc about whether to print the summary line or not in
      `FIO_shouldDisplayFileSummary()` and
      `FIO_shouldDisplayMultipleFileSummary()`.
    * Make `--progress` work for non-zstd (de)compressors.
    * Clean up several edge cases in compression and decompression progress
      printing along the way. E.g. wrong log level, or missing summary line.
    
    One thing I don't like about stdout mode, which sets the display level
    to 1, is that warnings aren't displayed. After this PR, we could change
    stdout mode from lowering the display level, to defaulting to implied
    `--no-progress`. But, I think that deserves a separate PR.
    terrelln committed Dec 14, 2022
    Configuration menu
    Copy the full SHA
    fbff782 View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2022

  1. Use proper unaligned access attributes

    Instead of using packed attribute hack, just use aligned attribute. It
    improves code generation on armv6 and armv7, and slightly improves code
    generation on aarch64. GCC generates identical code to regular aligned
    access on ARMv6 for all versions between 4.5 and trunk, except GCC 5
    which is buggy and generates the same (bad) code as packed access:
    https://gcc.godbolt.org/z/hq37rz7sb
    Hello71 authored and terrelln committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    a78c91a View commit details
    Browse the repository at this point in the history
  2. [fileio] Separate parameter adaption from display update rate

    Split the logic for parameter adaption from the logic to update the display rate.
    This decouples the two updates, so changes to display updates don't affect
    parameter adaption.
    
    Also add a test case that checks that parameter adaption actually happens.
    
    This fixes Issue #3353, where --adapt is broken when --no-progress is passed.
    terrelln committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    15f32ad View commit details
    Browse the repository at this point in the history
  3. [decompress] Fix nullptr addition & improve fuzzer

    Fix an instance of `NULL + 0` in `ZSTD_decompressStream()`. Also, improve our
    `stream_decompress` fuzzer to pass `NULL` in/out buffers to
    `ZSTD_decompressStream()`, and fix 2 issues that were immediately surfaced.
    
    Fixes #3351
    terrelln committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    f31b83f View commit details
    Browse the repository at this point in the history
  4. [legacy] Remove FORCE_MEMORY_ACCESS and only use memcpy

    Delete unaligned memory access code from the legacy codebase by removing all the
    non-memcpy functions. We don't care about speed at all for this codebase, only
    simplicity.
    terrelln committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    728e73e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    e2fc933 View commit details
    Browse the repository at this point in the history
  6. Merge pull request #3352 from daniellerozenblit/http-to-https

    Convert references to https from http
    daniellerozenblit authored Dec 15, 2022
    Configuration menu
    Copy the full SHA
    6e3667a View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    6be3181 View commit details
    Browse the repository at this point in the history
  8. Fix corruption that rarely occurs in 32-bit mode with wlog=25

    Fix an off-by-one error in the compressor that emits corrupt blocks if:
    
    * Zstd is compiled in 32-bit mode
    * The windowLog == 25 exactly
    * An offset of 2^25-3, 2^25-2, 2^25-1, or 2^25 is emitted
    * The bitstream had 7 bits leftover before writing the offset
    
    This bug has been present since before v1.0, but wasn't able to easily
    be triggered, since until somewhat recently zstd wasn't able to find
    matches that were within 128KB of the window size.
    
    Add a test case, and fix 2 bugs in `ZSTD_compressSequences()`:
    * The `ZSTD_isRLE()` check was incorrect. It wouldn't produce
      corruption, but it could waste CPU and not emit RLE even if the block
      was RLE
    * One windowSize was `1 << windowLog`, not `1u << windowLog`
    
    Thanks to @tansy for finding the issue, and giving us a reproducer!
    
    Fixes Issue #3350.
    terrelln committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    a91e7ec View commit details
    Browse the repository at this point in the history
  9. check potential overflow of compressBound()

    fixed #3323, reported by @nigeltao
    
    Completed documentation around this risk
    (which is largely theoretical,
    I can't see that happening in any "real world" scenario,
    but an erroneous @srcSize value could indeed trigger it).
    Cyan4973 committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    45ed0df View commit details
    Browse the repository at this point in the history
  10. meson: partial fix for building pzstd on MSVC

    It uses non-portable compiler options unconditionally. Elsewhere, we
    check the compiler ID and only add the right ones, globally. Do the same
    here.
    
    NDEBUG can actually be handled by a core option, so while we are moving
    things around, do so.
    
    Unfortunately, this doesn't fix things entirely. The remaining issue is
    not Meson's issue though -- MSVC simply does not like this source code
    and somehow chokes on innocent code with the inscrutable "syntax error"
    and "illegal token".
    eli-schwartz authored and terrelln committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    e0ef09d View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    1bc9dfe View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    26e8c65 View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2022

  1. meson: fix broken commit that broke the build

    In commit 031de3c some code was added
    that returned a boolean, but was treated as if it returned a dependency
    object. This wasn't tested and could not work. Moreover, zstd no longer
    built at all unless the entire programs directory was disabled and not
    even evaluated.
    
    Fix the return type checking.
    eli-schwartz committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    e8401e9 View commit details
    Browse the repository at this point in the history
  2. meson: fix warning for using too-new features

    In commit 031de3c a feature of Meson
    0.50.0 was added, but the minimum specified version of Meson is 0.48.0.
    Meson therefore emitted a warning:
    
    WARNING: Project targets '>=0.48.0' but uses feature introduced in '0.50.0': required arg in compiler.has_header.
    
    And if anyone actually used Meson 0.48.0 to build with, it would error
    out with mysterious claims that the build file itself is invalid, rather
    than telling the user to install a newer version of Meson.
    
    Solve this by bumping the minimum version to align with reality. This
    e.g. drops support for Debian oldstable (buster)'s packaged version of
    Meson, but still works if backports are enabled, or if the user can
    `pip install` a newer version.
    eli-schwartz committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    626425d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ee6475c View commit details
    Browse the repository at this point in the history
  4. Merge pull request #3368 from eli-schwartz/meson-execinfo-bool

    meson: fix broken commit that broke the build
    yoniko authored Dec 16, 2022
    Configuration menu
    Copy the full SHA
    8745229 View commit details
    Browse the repository at this point in the history
  5. [circleci] Try to re-enable aarch64build

    Update CircleCI to focal and try to re-enable aarch64build.
    
    Fixes #2785 if it works.
    terrelln committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    ef566c8 View commit details
    Browse the repository at this point in the history
  6. meson: don't require valgrind tests

    It's entirely possible some people don't have valgrind installed, but
    still want to run the tests. If they don't have it installed, then they
    probably don't intend to run those precise test targets anyway.
    
    Also, this solves an error when running the tests in an automated
    environment. The valgrind tests have a hard dependency on behavior such
    as `./zstd` erroring out with the message "stdin is a console, aborting"
    which does not work if the automated environment doesn't have a console.
    As a rough heuristic, automated environments lacking a console will
    *probably* also not have valgrind, so avoiding that test definition
    neatly sidesteps the issue.
    
    Also, valgrind is not easily installable on macOS, at least homebrew
    says it isn't available there. This makes it needlessly hard to
    enable the testsuite on macOS.
    eli-schwartz committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    26134b4 View commit details
    Browse the repository at this point in the history
  7. meson: add support for running both fast and slow version of tests

    playTests.sh has an option to run really slow tests. This is enabled by
    default in Meson, but what we really want is to do like the Makefile,
    and run the fast ones by default, but with an option to run the slow
    ones instead.
    eli-schwartz committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    9c3e18f View commit details
    Browse the repository at this point in the history
  8. meson: add Linux CI

    Travis is no longer run, but this wasn't ported to something else.
    eli-schwartz committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    7f29c18 View commit details
    Browse the repository at this point in the history
  9. meson: add Windows CI

    There are a couple of oddities here. We don't attempt to build e.g.
    contrib, because that doesn't seem to work at the moment. Also notice
    that each command is its own step. This happens because github actions
    runs in powershell, which doesn't seem to let you abort on the first
    failure.
    eli-schwartz committed Dec 16, 2022
    4 Configuration menu
    Copy the full SHA
    937e9d3 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    6747ba4 View commit details
    Browse the repository at this point in the history
  11. [CI] Re-enable versions-test

    It seems like with the deletion of Travis CI we didn't successfully transfer the
    version compatibility test. Attempt to enable the version compatibility test.
    terrelln committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    5850839 View commit details
    Browse the repository at this point in the history
  12. added unit tests for compressBound()

    and rephrased the code documentation, as suggested by @terrelln
    Cyan4973 committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    97f63ce View commit details
    Browse the repository at this point in the history
  13. [api][visibility] Make the visibility macros more consistent

    1. Follow the scheme introduced in PR #2501 for both `zdict.h` and `zstd_errors.h`.
    2. If the `*_VISIBLE` macro isn't set, but the `*_VISIBILITY` macro is, use that.
       Also make this change for `zstd.h`, since we probably shouldn't have changed
       that macro name without backward compatibility in the first place.
    3. Change all references to `*_VISIBILITY` to `*_VISIBLE`.
    
    Fixes #3359.
    terrelln committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    358a237 View commit details
    Browse the repository at this point in the history
  14. [build] Fix ZSTD_LIB_MINIFY build option

    `ZSTD_LIB_MINIFY` broke in 8bf699a.
    
    This commit fixes the macro and the static library shrinks from ~600K to 324K
    with ZSTD_LIB_MINIFY set.
    
    Fixes #3066.
    terrelln committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    0c42424 View commit details
    Browse the repository at this point in the history
  15. [zdict] Fix static linking only include guards

    Fix `zdict.h` static linking only section so if you include it twice it
    still exposes the static linking only symbols. E.g. this pattern:
    
    ```
    ```
    
    This can easily happen when a header you include includes `zdict.h`.
    terrelln committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    2f7b8d4 View commit details
    Browse the repository at this point in the history
  16. Merge pull request #3362 from facebook/compressBound

    check potential overflow of compressBound()
    Cyan4973 authored Dec 16, 2022
    Configuration menu
    Copy the full SHA
    51355e1 View commit details
    Browse the repository at this point in the history
  17. decompressBound() tests

    fixed an overflow in an intermediate result on 32-bit platform.
    Checked that the new test catch this bug in 32-bit mode.
    Cyan4973 committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    ea24b88 View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2022

  1. make ZSTD_DECOMPRESSBOUND() compatible with input size 0

    for environments with stringent compilation warnings.
    Cyan4973 committed Dec 17, 2022
    Configuration menu
    Copy the full SHA
    2f4238e View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3373 from facebook/decompressBound

    decompressBound tests and fix
    Cyan4973 authored Dec 17, 2022
    Configuration menu
    Copy the full SHA
    3a484ef View commit details
    Browse the repository at this point in the history
  3. Merge pull request #3120 from eli-schwartz/meson-fixup

    Meson test fixups
    yoniko authored Dec 17, 2022
    Configuration menu
    Copy the full SHA
    ce61cb8 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    3867c41 View commit details
    Browse the repository at this point in the history
  5. Merge pull request #3374 from yoniko/fix-meson-ci-dependancy

    Fix Meson-Windows CI test by pulling a more recent version msvc-dev-cmd
    yoniko authored Dec 17, 2022
    Configuration menu
    Copy the full SHA
    fba704f View commit details
    Browse the repository at this point in the history
  6. Fixes two bugs in the Windows thread / pthread translation layer

    1. If threads are resized the threads' `ZSTD_pthread_t` might move
    while the worker still holds a pointer into it (see more details in #3120).
    2. The join operation was waiting for a thread and then return its `thread.arg`
    as a return value, but since the `ZSTD_pthread_t thread` was passed by value it
    would have a stale `arg` that wouldn't match the thread's actual return value.
    
    This fix changes the `ZSTD_pthread_join` API and removes support for returning
    a value. This means that we are diverging from the `pthread_join` API and this
    is no longer just an alias.
    In the future, if needed, we could return a Windows thread's return value using
    `GetExitCodeThread`, but as this path wouldn't be excised in any case, it's
    preferable to not add it right now.
    yoniko committed Dec 17, 2022
    Configuration menu
    Copy the full SHA
    500f02e View commit details
    Browse the repository at this point in the history
  7. Fix race condition in the Windows thread / pthread translation layer

    When spawning a Windows thread we have small worker wrapper function that translates
    between the interfaces of Windows and POSIX threads.
    This wrapper is given a pointer that might get stale before the worker starts running,
    resulting in UB and crashes.
    This commit adds synchronization so that we know the wrapper has finished reading the data
    it needs before we allow the main thread to resume execution.
    yoniko committed Dec 17, 2022
    Configuration menu
    Copy the full SHA
    ec42c92 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    aaa38b2 View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2022

  1. minor reformatting

    and minor reliability and maintenance changes
    Cyan4973 committed Dec 18, 2022
    Configuration menu
    Copy the full SHA
    832c1a6 View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2022

  1. Bump ossf/scorecard-action from 2.0.6 to 2.1.0

    Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.0.6 to 2.1.0.
    - [Release notes](https://github.com/ossf/scorecard-action/releases)
    - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md)
    - [Commits](ossf/scorecard-action@99c5375...937ffa9)
    
    ---
    updated-dependencies:
    - dependency-name: ossf/scorecard-action
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Dec 19, 2022
    Configuration menu
    Copy the full SHA
    6554596 View commit details
    Browse the repository at this point in the history
  2. Bump github/codeql-action from 1.0.26 to 2.1.37

    Bumps [github/codeql-action](https://github.com/github/codeql-action) from 1.0.26 to 2.1.37.
    - [Release notes](https://github.com/github/codeql-action/releases)
    - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
    - [Commits](github/codeql-action@5f53256...959cbb7)
    
    ---
    updated-dependencies:
    - dependency-name: github/codeql-action
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Dec 19, 2022
    Configuration menu
    Copy the full SHA
    79a00f8 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #3376 from facebook/split2

    Block splitter : minor reformatting
    Cyan4973 authored Dec 19, 2022
    Configuration menu
    Copy the full SHA
    9073fe0 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    3cee69a View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    67cd24b View commit details
    Browse the repository at this point in the history
  6. [pzstd] Fixes for Windows build

    * Add `Portability.h` to fix min/max issues.
    * Fix conversion warnings
    * Assert that windowLog <= 23, which is currently always the case.
      This could be loosened, but we aren't looking to add new functionality.
    
    Fixes on top of PR #3375 by @eli-schwartz, which added Windows CI for contrib & programs.
    terrelln committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    e9797b5 View commit details
    Browse the repository at this point in the history
  7. CR fixes

    yoniko committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    26f1bf7 View commit details
    Browse the repository at this point in the history
  8. Merge pull request #3364 from yoniko/fix-windows-mt-thread-resize-bug

    Windows MT layer bug fixes
    yoniko authored Dec 19, 2022
    Configuration menu
    Copy the full SHA
    a8add43 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2022

  1. Merge pull request #3289 from nmoinvaz/cmake/playtest-win

    Don't attempt playTests.sh cmake test if running on Windows.
    Cyan4973 authored Dec 20, 2022
    Configuration menu
    Copy the full SHA
    6a90c0f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a05cca6 View commit details
    Browse the repository at this point in the history
  3. Update Copyright Headers 'Facebook' -> 'Meta Platforms'

    ```
    for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
    do
      sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
    done
    ```
    felixhandte committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    8927f98 View commit details
    Browse the repository at this point in the history
  4. Update Copyright Year ('2021' -> 'present')

    ```
    for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f);
    do
      sed -i 's/\-2021/-present/' $f;
    done
    
    g co HEAD -- .github/workflows/dev-short-tests.yml # fix bad match
    ```
    felixhandte committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    36d5c2f View commit details
    Browse the repository at this point in the history
  5. Update test-license.py

    felixhandte committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    4c999cf View commit details
    Browse the repository at this point in the history
  6. Rewrite Copyright Date Ranges from -present to -2022

    Apparently it's better. Somehow.
    
    ```
    for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done
    
    g co HEAD -- build/meson/
    ```
    felixhandte committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    7f12f24 View commit details
    Browse the repository at this point in the history
  7. Coalesce Almost All Copyright Notices to Standard Phrasing

    ```
    for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done
    
    git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
    nano ./programs/windres/zstd.rc
    nano ./build/VS2010/zstd/zstd.rc
    nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
    ```
    felixhandte committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    5d693cc View commit details
    Browse the repository at this point in the history
  8. Merge pull request #3173 from felixhandte/update-copyright-company

    Update Copyright Comments
    felixhandte authored Dec 20, 2022
    Configuration menu
    Copy the full SHA
    f302ad8 View commit details
    Browse the repository at this point in the history
  9. [docs] Clarify dictionary loading documentation

    Reinforce that loading a new dictionary clears the current dictionary.
    Except for the multiple-ddict mode.
    terrelln committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    e4018c4 View commit details
    Browse the repository at this point in the history
  10. Merge pull request #3340 from facebook/dependabot/github_actions/acti…

    …ons/upload-artifact-3
    
    Bump actions/upload-artifact from 1 to 3
    felixhandte authored Dec 20, 2022
    Configuration menu
    Copy the full SHA
    3a7e2ee View commit details
    Browse the repository at this point in the history
  11. Merge pull request #3377 from facebook/dependabot/github_actions/ossf…

    …/scorecard-action-2.1.0
    
    Bump ossf/scorecard-action from 2.0.6 to 2.1.0
    felixhandte authored Dec 20, 2022
    Configuration menu
    Copy the full SHA
    5f0543a View commit details
    Browse the repository at this point in the history
  12. Merge pull request #3378 from facebook/dependabot/github_actions/gith…

    …ub/codeql-action-2.1.37
    
    Bump github/codeql-action from 1.0.26 to 2.1.37
    felixhandte authored Dec 20, 2022
    Configuration menu
    Copy the full SHA
    ea19987 View commit details
    Browse the repository at this point in the history
  13. Pin actions/checkout Dependency to Specific Commit Hash

    It's a bit silly, because if we can't trust GitHub, what are we doing here?
    But OSSF complains about it, so let's fix it.
    felixhandte committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    e3f2c8b View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    482689b View commit details
    Browse the repository at this point in the history
  15. Merge branch 'optimal-huff-depth-speed' of github.com:daniellerozenbl…

    …it/zstd into optimal-huff-depth-speed
    daniellerozenblit committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    2def93b View commit details
    Browse the repository at this point in the history
  16. fix CI errors

    daniellerozenblit committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    c26f348 View commit details
    Browse the repository at this point in the history
  17. [build][cmake] Fix cmake with custom assembler

    Tell CMake to explicitly compile our assembly as C code, because we
    require it is compiled by a C compiler, and it is only enabled for
    clang/gcc.
    
    Fixes #3193.
    terrelln committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    651a381 View commit details
    Browse the repository at this point in the history
  18. Merge pull request #3384 from felixhandte/pin-checkout-action-commit-…

    …hash
    
    Pin actions/checkout Dependency to Specific Commit Hash
    felixhandte authored Dec 20, 2022
    Configuration menu
    Copy the full SHA
    0790e86 View commit details
    Browse the repository at this point in the history
  19. Pin Remaining Action Dependencies (Except OSS-Fuzz)

    The one that isn't pinned is the OSS-Fuzz builder and runner. They don't
    offer tagged releases. I could pin to the current master commit, but I'm not
    sure how desirable that is.
    felixhandte committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    150aa23 View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2022

  1. Configuration menu
    Copy the full SHA
    678335c View commit details
    Browse the repository at this point in the history
  2. Clean up welcome message

    jonpalmisc committed Dec 21, 2022
    Configuration menu
    Copy the full SHA
    9c93dd7 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    125bcde View commit details
    Browse the repository at this point in the history
  4. Merge pull request #3386 from felixhandte/pin-other-action-commit-hashes

    Pin Remaining Action Dependencies (Except OSS-Fuzz)
    felixhandte authored Dec 21, 2022
    Configuration menu
    Copy the full SHA
    0694f14 View commit details
    Browse the repository at this point in the history
  5. Merge pull request #3385 from jonpalmisc/improve_help

    Improve help/usage (`-h`, `-H`) formatting
    Cyan4973 authored Dec 21, 2022
    Configuration menu
    Copy the full SHA
    7607b96 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2022

  1. [util] Add traces enabled by --trace-file-stat

    Print traces to stderr when --trace-file-stat is passed.
    We trace all functions that read and write file metadata.
    terrelln committed Dec 22, 2022
    Configuration menu
    Copy the full SHA
    b6e8112 View commit details
    Browse the repository at this point in the history
  2. [cli-tests] Add --set-exact-output to update the expected output

    `./run.py --set-exact-output` will update `stdout.expect` and
    `stderr.expect` to match the expected output. This doesn't apply to
    outputs which use `.glob` or `.ignore`.
    terrelln committed Dec 22, 2022
    Configuration menu
    Copy the full SHA
    7df6e25 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    31a703e View commit details
    Browse the repository at this point in the history
  4. Fix make clangbuild & add CI

    Fix the errors for:
    * `-Wdocumentation`
    * `-Wconversion` except `-Wsign-conversion`
    terrelln committed Dec 22, 2022
    Configuration menu
    Copy the full SHA
    40a7188 View commit details
    Browse the repository at this point in the history
  5. [tests] Remove deprecated function from longmatch.c test

    Thanks to @eli-schwartz for pointing it out!
    
    We should maybe consider adding a helper function for applying
    `ZSTD_parameters` and `ZSTD_compressionParameters` to a context.
    That would aid the transition to the new API in situations like this.
    terrelln committed Dec 22, 2022
    Configuration menu
    Copy the full SHA
    4b40e40 View commit details
    Browse the repository at this point in the history
  6. [cli-tests] Add tests that use --trace-file-stat

    Basic tests for (de)compressing in the following modes:
    * file to file
    * file to stdout
    * stdin to file
    * stdin to stdout
    
    These are basic tests, and aren't testing more advanced scenarios, but
    it adds the groundwork for more complex tests as needed.
    
    Fixes #3010.
    terrelln committed Dec 22, 2022
    Configuration menu
    Copy the full SHA
    7fe7a16 View commit details
    Browse the repository at this point in the history
  7. Man Page Tweaks, Edits, Formatting Fixes

    This started as an application of the edits suggested in #3201 and expanded
    from there.
    felixhandte committed Dec 22, 2022
    Configuration menu
    Copy the full SHA
    382026f View commit details
    Browse the repository at this point in the history
  8. make man

    felixhandte committed Dec 22, 2022
    Configuration menu
    Copy the full SHA
    11aba9b View commit details
    Browse the repository at this point in the history
  9. Merge pull request #3397 from felixhandte/man-page-tweaks

    Man Page Tweaks, Edits, Formatting Fixes
    felixhandte authored Dec 22, 2022
    Configuration menu
    Copy the full SHA
    f5ea3a1 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    ea2895c View commit details
    Browse the repository at this point in the history
  11. Merge pull request #3399 from facebook/fix2577

    Support decompression of compressed blocks of size ZSTD_BLOCKSIZE_MAX
    Cyan4973 authored Dec 22, 2022
    Configuration menu
    Copy the full SHA
    8209bfc View commit details
    Browse the repository at this point in the history

Commits on Dec 23, 2022

  1. spec update : require minimum nb of literals for 4-streams mode

    Reported by @shulib :
    the specification for 4-streams mode
    doesn't work when the amount of literals to compress is 5 bytes.
    Extending it, it also doesn't work for sizes 1 or 2.
    
    This patch updates the specification and the implementation
    to require a minimum of 6 literals to trigger or accept the 4-streams mode.
    
    The impact is expected to be a no-op :
    the 4-streams mode is never triggered for such small quantity of literals anyway,
    since it would be wasteful (it costs ~7.3 bytes more than single-stream mode).
    An informal lower limit is set at ~256 bytes,
    so the technical minimum is very far from this limit.
    
    This is just meant for completeness of the specification.
    Cyan4973 committed Dec 23, 2022
    Configuration menu
    Copy the full SHA
    6a9c525 View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3398 from facebook/fix3316

    spec update : require minimum nb of literals for 4-streams mode
    Cyan4973 authored Dec 23, 2022
    Configuration menu
    Copy the full SHA
    089b279 View commit details
    Browse the repository at this point in the history
  3. Move deprecated annotation before static to allow C++ compilation for…

    … clang
    
    This fixes last 2 instances of #3250
    danlark1 authored Dec 23, 2022
    Configuration menu
    Copy the full SHA
    48f4aa7 View commit details
    Browse the repository at this point in the history
  4. cmake build: fix nit

    reported by @jaimeMF in #3392 (comment)
    Cyan4973 committed Dec 23, 2022
    Configuration menu
    Copy the full SHA
    6640377 View commit details
    Browse the repository at this point in the history

Commits on Dec 26, 2022

  1. Bump ossf/scorecard-action from 2.1.0 to 2.1.2

    Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.1.0 to 2.1.2.
    - [Release notes](https://github.com/ossf/scorecard-action/releases)
    - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md)
    - [Commits](ossf/scorecard-action@937ffa9...e38b190)
    
    ---
    updated-dependencies:
    - dependency-name: ossf/scorecard-action
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Dec 26, 2022
    Configuration menu
    Copy the full SHA
    1f72dca View commit details
    Browse the repository at this point in the history

Commits on Dec 28, 2022

  1. Merge pull request #3394 from terrelln/issue-3010

    [cli-tests] Test file stat read/write
    Cyan4973 authored Dec 28, 2022
    Configuration menu
    Copy the full SHA
    90597d7 View commit details
    Browse the repository at this point in the history
  2. New xp library symbol : ZSTD_CCtx_setCParams()

    Inspired by #3395,
    offer a new capability to set all parameters defined in a ZSTD_compressionParameters structure
    with a single symbol invocation
    to improve user code brevity.
    Cyan4973 committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    89342d1 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b17743e View commit details
    Browse the repository at this point in the history
  4. External matchfinder API (#3333)

    * First building commit with sample matchfinder
    
    * Set up ZSTD_externalMatchCtx struct
    
    * move seqBuffer to ZSTD_Sequence*
    
    * support non-contiguous dictionary
    
    * clean up parens
    
    * add clearExternalMatchfinder, handle allocation errors
    
    * Add useExternalMatchfinder cParam
    
    * validate useExternalMatchfinder cParam
    
    * Disable LDM + external matchfinder
    
    * Check for static CCtx
    
    * Validate mState and mStateDestructor
    
    * Improve LDM check to cover both branches
    
    * Error API with optional fallback
    
    * handle RLE properly for external matchfinder
    
    * nit
    
    * Move to a CDict-like model for resource ownership
    
    * Add hidden useExternalMatchfinder bool to CCtx_params_s
    
    * Eliminate malloc, move to cwksp allocation
    
    * Handle CCtx reset properly
    
    * Ensure seqStore has enough space for external sequences
    
    * fix capitalization
    
    * Add DEBUGLOG statements
    
    * Add compressionLevel param to matchfinder API
    
    * fix c99 issues and add a param combination error code
    
    * nits
    
    * Test external matchfinder API
    
    * C90 compat for simpleExternalMatchFinder
    
    * Fix some @nocommits and an ASAN bug
    
    * nit
    
    * nit
    
    * nits
    
    * forward declare copySequencesToSeqStore functions in zstd_compress_internal.h
    
    * nit
    
    * nit
    
    * nits
    
    * Update copyright headers
    
    * Fix CMake zstreamtest build
    
    * Fix copyright headers (again)
    
    * typo
    
    * Add externalMatchfinder demo program to make contrib
    
    * Reduce memory consumption for small blockSize
    
    * ZSTD_postProcessExternalMatchFinderResult nits
    
    * test sum(matchlen) + sum(litlen) == srcSize in debug builds
    
    * refExternalMatchFinder -> registerExternalMatchFinder
    
    * C90 nit
    
    * zstreamtest nits
    
    * contrib nits
    
    * contrib nits
    
    * allow block splitter + external matchfinder, refactor
    
    * add windowSize param
    
    * add contrib/externalMatchfinder/README.md
    
    * docs
    
    * go back to old RLE heuristic because of the first block issue
    
    * fix initializer element is not a constant expression
    
    * ref contrib from zstd.h
    
    * extremely pedantic compiler warning fix, meson fix, typo fix
    
    * Additional docs on API limitations
    
    * minor nits
    
    * Refactor maxNbSeq calculation into a helper function
    
    * Fix copyright
    embg authored Dec 28, 2022
    Configuration menu
    Copy the full SHA
    2a40262 View commit details
    Browse the repository at this point in the history
  5. Merge pull request #3403 from facebook/setCParams

    ZSTD_CCtx_setCParams
    Cyan4973 authored Dec 28, 2022
    Configuration menu
    Copy the full SHA
    481a2e1 View commit details
    Browse the repository at this point in the history
  6. update ZSTD_CCts_setCParams() inline documentation

    specify behavior when changing compression parameters during MT compression,
    reported by @embg
    Cyan4973 committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    00c85b2 View commit details
    Browse the repository at this point in the history
  7. Merge pull request #3395 from terrelln/2022-12-21-deprecated-test

    [tests] Remove deprecated function from longmatch.c test
    Cyan4973 authored Dec 28, 2022
    Configuration menu
    Copy the full SHA
    bcbd395 View commit details
    Browse the repository at this point in the history
  8. Merge pull request #3400 from danlark1/dev

    Move deprecated annotation before static to allow C++ compilation for clang
    Cyan4973 authored Dec 28, 2022
    Configuration menu
    Copy the full SHA
    9fbbd74 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2022

  1. Configuration menu
    Copy the full SHA
    4a1a79a View commit details
    Browse the repository at this point in the history
  2. fixed incorrect assert

    commented Fweight instead
    Cyan4973 committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    d07e72b View commit details
    Browse the repository at this point in the history
  3. Merge pull request #3248 from facebook/opt_comments1

    [easy] add a few comments to the optimal parser code base for improved clarity
    Cyan4973 authored Dec 29, 2022
    Configuration menu
    Copy the full SHA
    3248910 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2023

  1. Configuration menu
    Copy the full SHA
    df714dd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    87becc5 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #3302 from daniellerozenblit/optimal-huff-depth-speed

    Optimal huff depth speed improvements
    daniellerozenblit authored Jan 3, 2023
    Configuration menu
    Copy the full SHA
    1c818e3 View commit details
    Browse the repository at this point in the history
  4. improve compression ratio of small alphabets

    fix #3328
    
    In situations where the alphabet size is very small,
    the evaluation of literal costs from the Optimal Parser is initially incorrect.
    It takes some time to converge, during which compression is less efficient.
    This is especially important for small files,
    because there will not be enough data to converge,
    so most of the parsing is selected based on incorrect metrics.
    
    After this patch, the scenario ##3328 gets fixed,
    delivering the expected 29 bytes compressed size (smallest known compressed size).
    Cyan4973 committed Jan 3, 2023
    Configuration menu
    Copy the full SHA
    5434de0 View commit details
    Browse the repository at this point in the history
  5. update regression results

    Cyan4973 committed Jan 3, 2023
    Configuration menu
    Copy the full SHA
    ebba9ff View commit details
    Browse the repository at this point in the history
  6. update levels.sh test

    comparing level 19 to level 22 and expecting a stricter better result from level 22
    is not that guaranteed,
    because level 19 and 22 are very close to each other,
    especially for small files,
    so any noise in the final compression result
    result in failing this test.
    
    Level 22 could be compared to something much lower, like level 15,
    But level 19 is required anyway, because there is a clamping test which depends on it.
    
    Removed level 22, kept level 19
    Cyan4973 committed Jan 3, 2023
    Configuration menu
    Copy the full SHA
    c79fb4d View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2023

  1. Merge pull request #3391 from facebook/fix3228

    improve compression ratio of small alphabets
    Cyan4973 authored Jan 4, 2023
    Configuration menu
    Copy the full SHA
    834fd07 View commit details
    Browse the repository at this point in the history
  2. initial commit

    daniellerozenblit committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    908e812 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d913417 View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2023

  1. Configuration menu
    Copy the full SHA
    71dbe8f View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2023

  1. minor simplification refactoring for timefn

    `UTIL_getSpanTimeMicro()` can be factored in a generic way,
    reducing OS-dependent code.
    Cyan4973 committed Jan 7, 2023
    Configuration menu
    Copy the full SHA
    8b13000 View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2023

  1. Bump actions/checkout from 3.2.0 to 3.3.0

    Bumps [actions/checkout](https://github.com/actions/checkout) from 3.2.0 to 3.3.0.
    - [Release notes](https://github.com/actions/checkout/releases)
    - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
    - [Commits](actions/checkout@755da8c...ac59398)
    
    ---
    updated-dependencies:
    - dependency-name: actions/checkout
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Jan 9, 2023
    Configuration menu
    Copy the full SHA
    6f17a5d View commit details
    Browse the repository at this point in the history
  2. Bump actions/upload-artifact from 3.1.1 to 3.1.2

    Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.1 to 3.1.2.
    - [Release notes](https://github.com/actions/upload-artifact/releases)
    - [Commits](actions/upload-artifact@83fd05a...0b7f8ab)
    
    ---
    updated-dependencies:
    - dependency-name: actions/upload-artifact
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Jan 9, 2023
    Configuration menu
    Copy the full SHA
    59a536a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fe08137 View commit details
    Browse the repository at this point in the history
  4. Merge pull request #3414 from facebook/dependabot/github_actions/acti…

    …ons/checkout-3.3.0
    
    Bump actions/checkout from 3.2.0 to 3.3.0
    Cyan4973 authored Jan 9, 2023
    Configuration menu
    Copy the full SHA
    98ca8b4 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2023

  1. Fix fuzzing with ZSTD_MULTITHREAD

    At Google we fuzz zstd without ZSTD_MULTITHREAD but we want inputs to be as much as reproducible. It allows us to test new fuzzing methods for our fuzz team internally and have more horsepower to find bugs
    danlark1 authored and terrelln committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    3ac0b91 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ca2ff78 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1fffcfe View commit details
    Browse the repository at this point in the history
  4. Completely overhaul Windows CI (#3410)

    * Overhaul windows CI
    
    * upgrade setup-msbuild from v1.1.3 to v1.3
    
    * remove cmake 2019 test
    
    * fix 32-bit gcc mingw test
    
    * merge conflict
    embg authored Jan 11, 2023
    Configuration menu
    Copy the full SHA
    4f7183d View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2023

  1. Configuration menu
    Copy the full SHA
    53eb5a7 View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3402 from facebook/dependabot/github_actions/ossf…

    …/scorecard-action-2.1.2
    
    Bump ossf/scorecard-action from 2.1.0 to 2.1.2
    felixhandte authored Jan 12, 2023
    Configuration menu
    Copy the full SHA
    fd2eb8a View commit details
    Browse the repository at this point in the history
  3. additional tests and documentation updates + allow maxBlockSize to be…

    … set to 0 (goes to default)
    daniellerozenblit committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    06b096d View commit details
    Browse the repository at this point in the history
  4. Merge pull request #3413 from facebook/timefn

    minor refactoring for timefn
    Cyan4973 authored Jan 12, 2023
    Configuration menu
    Copy the full SHA
    423500d View commit details
    Browse the repository at this point in the history
  5. fix root cause of #3416

    A minor change in 5434de0 changed a `<=` into a `<`,
    and as an indirect consequence allowed compression attempt of literals when there are only 6 literals to compress
    (previous limit was effectively 7 literals).
    
    This is not in itself a problem, as the threshold is merely an heuristic,
    but it emerged a bug that has always been there, and was just never triggered so far due to the previous limit.
    This bug would make the literal compressor believes that all literals are the same symbol,
    but for the exact case where nbLiterals==6, plus a pretty wild combination of other limit conditions,
    this outcome could be false, resulting in data corruption.
    
    Replaced the blind heuristic by an actual test for all limit cases,
    so that even if the threshold is changed again in the future,
    the detection of RLE mode will remain reliable.
    Cyan4973 committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    796699c View commit details
    Browse the repository at this point in the history
  6. add explanation about new test

    as requested by @terrelln
    Cyan4973 committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    ac45e07 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2023

  1. Add support for in-place decompression

    * Add a function and macro ZSTD_decompressionMargin() that computes the
      decompression margin for in-place decompression. The function computes
      a tight margin that works in all cases, and the macro computes an upper
      bound that will only work if flush isn't used.
    * When doing in-place decompression, make sure that our output buffer
      doesn't overlap with the input buffer. This ensures that we don't
      decide to use the portion of the output buffer that overlaps the input
      buffer for temporary memory, like for literals.
    * Add a simple unit test.
    * Add in-place decompression to the simple_round_trip and
      stream_round_trip fuzzers. This should help verify that our margin stays
      correct.
    terrelln committed Jan 13, 2023
    Configuration menu
    Copy the full SHA
    5b26619 View commit details
    Browse the repository at this point in the history
  2. refactor timefn

    The timer storage type is no longer dependent on OS.
    This will make it possible to re-enable posix precise timers
    since the timer storage type will no longer be sensible to #include order.
    See #3168 for details of pbs of previous interface.
    
    Suggestion by @terrelln
    Cyan4973 committed Jan 13, 2023
    Configuration menu
    Copy the full SHA
    bcfb7ad View commit details
    Browse the repository at this point in the history
  3. restore support of clock_gettime() for POSIX systems

    This should notably allow posix systems with timespec_get()
    to have access to a high resolution timer,
    instead of falling back to C90's clock_t.
    Cyan4973 committed Jan 13, 2023
    Configuration menu
    Copy the full SHA
    a2ef23d View commit details
    Browse the repository at this point in the history
  4. Merge pull request #3419 from facebook/fix3416

    fix root cause of #3416
    Cyan4973 authored Jan 13, 2023
    Configuration menu
    Copy the full SHA
    d550908 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    14b8def View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    f10922a View commit details
    Browse the repository at this point in the history
  7. Don't Even Declare Poisoning Functions if Poisoning is Disabled

    This guarantees that we won't accidentally forget to check the macro somewhere
    where we use these functions.
    felixhandte committed Jan 13, 2023
    Configuration menu
    Copy the full SHA
    d78fbed View commit details
    Browse the repository at this point in the history
  8. missing #include for Windows

    Cyan4973 committed Jan 13, 2023
    Configuration menu
    Copy the full SHA
    2086e73 View commit details
    Browse the repository at this point in the history
  9. Deprecate advanced streaming functions (#3408)

    * deprecate advanced streaming functions
    
    * remove internal usage of the deprecated functions
    
    * nit
    
    * suppress warnings in tests/zstreamtest.c
    
    * purge ZSTD_initDStream_usingDict
    
    * nits
    
    * c90 compat
    
    * zstreamtest.c already disables deprecation warnings!
    
    * fix initDStream() return value
    
    * fix typo
    
    * wasn't able to import private symbol properly, this commit works around that
    
    * new strategy for zbuff
    
    * undo zbuff deprecation warning changes
    
    * move ZSTD_DISABLE_DEPRECATE_WARNINGS from .h to .c
    embg authored Jan 13, 2023
    Configuration menu
    Copy the full SHA
    5d8cfa6 View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2023

  1. Bump github/codeql-action from 2.1.37 to 2.1.38 (#3428)

    Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.1.37 to 2.1.38.
    - [Release notes](https://github.com/github/codeql-action/releases)
    - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
    - [Commits](github/codeql-action@959cbb7...515828d)
    
    ---
    updated-dependencies:
    - dependency-name: github/codeql-action
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    
    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    dependabot[bot] authored Jan 16, 2023
    Configuration menu
    Copy the full SHA
    3add5ca View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2023

  1. Configuration menu
    Copy the full SHA
    018b68f View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3424 from felixhandte/disable-asan-msan-poison-mingw

    Disable Custom ASAN/MSAN Poisoning on MinGW Builds
    felixhandte authored Jan 17, 2023
    Configuration menu
    Copy the full SHA
    23a356c View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8353a4b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2ad6855 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5653f96 View commit details
    Browse the repository at this point in the history
  6. Use Existing Src File Stat in *_dstFile() Funcs

    One fewer `stat()` call to make per operation!
    felixhandte committed Jan 17, 2023
    Configuration menu
    Copy the full SHA
    a5ed28f View commit details
    Browse the repository at this point in the history
  7. [tests] Fix version test determinism

    The dictionary source files were taken from the `dev` branch before this
    commit, which could introduce non-determinism on PR jobs. Instead take
    the sources from the PR checkout.
    
    This PR also adds stderr logging, and verbose output for the jobs that
    are failing, to help catch the failure if it occurs again.
    terrelln committed Jan 17, 2023
    Configuration menu
    Copy the full SHA
    860548c View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2023

  1. Configuration menu
    Copy the full SHA
    0382076 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1e3eba6 View commit details
    Browse the repository at this point in the history
  3. Mimic gzip chown(gid), chmod(), chown(uid) Behavior

    Avoids a race condition in which we unintentionally open up permissions to
    the wrong group.
    felixhandte committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    0d2d460 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7a8c8f3 View commit details
    Browse the repository at this point in the history
  5. Merge pull request #3423 from facebook/ptime

    Refactor timefn, restore support for clock_gettime()
    Cyan4973 authored Jan 18, 2023
    Configuration menu
    Copy the full SHA
    bbe65d7 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2023

  1. Configuration menu
    Copy the full SHA
    dc1c6cc View commit details
    Browse the repository at this point in the history
  2. Bugfixes for the External Matchfinder API (#3433)

    * external matchfinder bugfixes + tests
    
    * small doc fix
    embg authored Jan 19, 2023
    Configuration menu
    Copy the full SHA
    bce0382 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3f9f568 View commit details
    Browse the repository at this point in the history
  4. added c89 build test to CI

    Cyan4973 committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    ea684c3 View commit details
    Browse the repository at this point in the history

Commits on Jan 20, 2023

  1. Configuration menu
    Copy the full SHA
    cd272d7 View commit details
    Browse the repository at this point in the history
  2. Enable if == 1 rather than if == 0

    Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
    embg and terrelln authored Jan 20, 2023
    Configuration menu
    Copy the full SHA
    f593e54 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #3437 from embg/fuzz_emf

    Fuzz the External Matchfinder API
    embg authored Jan 20, 2023
    Configuration menu
    Copy the full SHA
    64963dc View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    aa385ec View commit details
    Browse the repository at this point in the history
  5. Merge pull request #3415 from facebook/dependabot/github_actions/acti…

    …ons/upload-artifact-3.1.2
    
    Bump actions/upload-artifact from 3.1.1 to 3.1.2
    Cyan4973 authored Jan 20, 2023
    Configuration menu
    Copy the full SHA
    abf965c View commit details
    Browse the repository at this point in the history
  6. Cap hashLog & chainLog to ensure that we only use 32 bits of hash

    * Cap shortCache chainLog to 24
    * Cap row match finder hashLog so that rowLog <= 24
    * Add unit tests to expose all cases. The row match finder unit tests
      are only run in 64-bit mode, because they allocate ~1GB.
    
    Fixes #3336
    terrelln committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    666944f View commit details
    Browse the repository at this point in the history
  7. [versions-test] Work around bug in dictionary builder for older versions

    Older versions of zstandard have a bug in the dictionary builder, that
    can cause dictionary building to fail. The process still exits 0, but
    the dictionary is not created.
    
    For reference, the bug is that it creates a dictionary that starts with
    the zstd dictionary magic, in the process of writing the dictionary header,
    but the header isn't fully written yet, and zstd fails compressions in
    this case, because the dictionary is malformated. We fixed this later on
    by trying to load the dictionary as a zstd dictionary, but if that fails
    we fallback to content only (by default).
    
    The fix is to:
    1. Make the dictionary determinsitic by sorting the input files.
       Previously the bug would only sometimes occur, when the input files
       were in a particular order.
    2. If dictionary creation fails, fallback to the `head` dictionary.
    terrelln committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    667eb6d View commit details
    Browse the repository at this point in the history
  8. Merge pull request #3435 from facebook/c89build

    added c89 build test to CI
    Cyan4973 authored Jan 20, 2023
    Configuration menu
    Copy the full SHA
    6742f20 View commit details
    Browse the repository at this point in the history
  9. Merge pull request #3431 from facebook/cygwin

    added cygwin tests to Github Actions
    Cyan4973 authored Jan 20, 2023
    Configuration menu
    Copy the full SHA
    bb9b9bc View commit details
    Browse the repository at this point in the history
  10. Delete unused Huffman functions

    Remove all Huffman functions that aren't used by zstd.
    terrelln committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    0cc1b0c View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    3291691 View commit details
    Browse the repository at this point in the history
  12. Add Additional Flags to PGO Build

    In GCC, we can add a couple more flags to give us confidence that the profile
    data is actually being found and used.
    
    Also, my system for example doesn't have a binary installed under the name
    `llvm-profdata`, but it does have, e.g., `llvm-profdata-13`, etc. So this
    commit adds a variable that can be overridden.
    felixhandte committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    87e169d View commit details
    Browse the repository at this point in the history
  13. Add PGO Build Jobs to CI

    felixhandte committed Jan 20, 2023
    Configuration menu
    Copy the full SHA
    aab3dd4 View commit details
    Browse the repository at this point in the history

Commits on Jan 21, 2023

  1. Configuration menu
    Copy the full SHA
    b4467c1 View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3442 from felixhandte/pgo-tests

    Test PGO Builds
    felixhandte authored Jan 21, 2023
    Configuration menu
    Copy the full SHA
    772229a View commit details
    Browse the repository at this point in the history
  3. Merge pull request #3432 from felixhandte/fix-perms

    Fix CLI Handling of Permissions and Ownership (Again)
    felixhandte authored Jan 21, 2023
    Configuration menu
    Copy the full SHA
    3d25502 View commit details
    Browse the repository at this point in the history
  4. fixed minor c89 warning

    introduced due to parallel merges
    Cyan4973 committed Jan 21, 2023
    Configuration menu
    Copy the full SHA
    d9280af View commit details
    Browse the repository at this point in the history
  5. refactor : --rm is ignored with stdout

    `zstd` CLI has progressively moved to the policy of
    ignoring `--rm` command when the output is `stdout`.
    The primary drive is to feature a behavior more consistent with `gzip`,
    when `--rm` is the default, but is also ignored when output is `stdout`.
    Other policies are certainly possible, but would break from this `gzip` convention.
    
    The new policy was inconsistenly enforced, depending on the exact list of commands.
    For example, it was possible to circumvent it by using `-c --rm` in this order,
    which would re-establish source removal.
    
    - Update the CLI so that it necessarily catch these situations and ensure that `--rm` is always disabled when output is `stdout`.
    - Added a warning message in this case (for verbosity 3 `-v`).
    - Added an `assert()`, which controls that `--rm` is no longer active with `stdout`
    - Added tests, which control the behavior, even when `--rm` is added after `-c`
    - Removed some legacy code which where trying to apply a specific policy for the `stdout` + `--rm` case, which is no longer possible
    Cyan4973 committed Jan 21, 2023
    Configuration menu
    Copy the full SHA
    cee6bec View commit details
    Browse the repository at this point in the history
  6. update man

    Cyan4973 committed Jan 21, 2023
    Configuration menu
    Copy the full SHA
    b6fd91b View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2023

  1. Bump github/codeql-action from 2.1.38 to 2.1.39

    Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.1.38 to 2.1.39.
    - [Release notes](https://github.com/github/codeql-action/releases)
    - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
    - [Commits](github/codeql-action@515828d...a34ca99)
    
    ---
    updated-dependencies:
    - dependency-name: github/codeql-action
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Jan 23, 2023
    Configuration menu
    Copy the full SHA
    3663faa View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    638d502 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1b65727 View commit details
    Browse the repository at this point in the history
  4. merge dev

    daniellerozenblit committed Jan 23, 2023
    Configuration menu
    Copy the full SHA
    f75afb6 View commit details
    Browse the repository at this point in the history
  5. Merge pull request #3446 from facebook/dependabot/github_actions/gith…

    …ub/codeql-action-2.1.39
    
    Bump github/codeql-action from 2.1.38 to 2.1.39
    embg authored Jan 23, 2023
    Configuration menu
    Copy the full SHA
    6aee603 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    815d1d4 View commit details
    Browse the repository at this point in the history
  7. Fix -Wstringop-overflow warning

    Backported from kernel patch [0].
    
    I wasn't able to reproduce the warning locally, but could repro it in
    the kernel.
    
    [0] https://lore.kernel.org/lkml/20220330193352.GA119296@embeddedor/
    terrelln committed Jan 23, 2023
    Configuration menu
    Copy the full SHA
    dc2b3e8 View commit details
    Browse the repository at this point in the history
  8. Merge pull request #3443 from facebook/no_rm_w_stdout

    refactor : --rm ignored with stdout
    Cyan4973 authored Jan 23, 2023
    Configuration menu
    Copy the full SHA
    ced0882 View commit details
    Browse the repository at this point in the history
  9. Fix ZSTD_estimate* and ZSTD_initCStream() docs

    Fix the following documentation bugs:
    * Note that `ZSTD_estimate*` functions are not compatible with the external matchfinder API
    * Note that `ZSTD_estimateCStreamSize_usingCCtxParams()` is not compatible with `nbWorkers >= 1`
    * Remove incorrect warning that the legacy streaming API is incompatible with advanced parameters and/or dictionary compression
    * Note that `ZSTD_initCStream()` is incompatible with dictionary compression
    * Warn that
    embg authored Jan 23, 2023
    Configuration menu
    Copy the full SHA
    3bfd3be View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    7fc00c1 View commit details
    Browse the repository at this point in the history
  11. Merge pull request #3439 from daniellerozenblit/sequence-validation-b…

    …ug-fix
    
    Fix sequence validation and seqStore bounds check
    daniellerozenblit authored Jan 23, 2023
    Configuration menu
    Copy the full SHA
    9116000 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    0a91b31 View commit details
    Browse the repository at this point in the history
  13. Merge pull request #3448 from facebook/embg-doc-fix

    Fix ZSTD_estimate* and ZSTD_initCStream() docs
    embg authored Jan 23, 2023
    Configuration menu
    Copy the full SHA
    41682e6 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2023

  1. Configuration menu
    Copy the full SHA
    7d600c6 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1d636b4 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #3451 from yoniko/red-zones-bugfix

    Bugfix redzone unpoisoning
    yoniko authored Jan 24, 2023
    Configuration menu
    Copy the full SHA
    2baac04 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    63042f1 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2023

  1. Merge pull request #3452 from daniellerozenblit/fix-seekable-32bit

    Fix 32-bit build errors in zstd seekable format
    daniellerozenblit authored Jan 25, 2023
    Configuration menu
    Copy the full SHA
    29a4c8c View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3447 from daniellerozenblit/fuzz-sequence-compres…

    …sion
    
    Fuzz large offsets through sequence compression api
    daniellerozenblit authored Jan 25, 2023
    Configuration menu
    Copy the full SHA
    f3255bf View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    db18a62 View commit details
    Browse the repository at this point in the history
  4. [huf] Add generic C versions of the fast decoding loops

    Add generic C versions of the fast decoding loops to serve architectures
    that don't have an assembly implementation. Also allow selecting the C
    decoding loop over the assembly decoding loop through a zstd
    decompression parameter `ZSTD_d_disableHuffmanAssembly`.
    
    I benchmarked on my Intel i9-9900K and my Macbook Air with an M1 processor.
    The benchmark command forces zstd to compress without any matches, using
    only literals compression, and measures only Huffman decompression speed:
    
    ```
    zstd -b1e1 --compress-literals --zstd=tlen=131072 silesia.tar
    ```
    
    The new fast decoding loops outperform the previous implementation uniformly,
    but don't beat the x86-64 assembly. Additionally, the fast C decoding loops suffer
    from the same stability problems that we've seen in the past, where the assembly
    version doesn't. So even though clang gets close to assembly on x86-64, it still
    has stability issues.
    
    | Arch    | Function       | Compiler     | Default (MB/s) | Assembly (MB/s) | Fast (MB/s) |
    |---------|----------------|--------------|----------------|-----------------|-------------|
    | x86-64  | decompress 4X1 | gcc-12.2.0   |         1029.6 |          1308.1 |      1208.1 |
    | x86-64  | decompress 4X1 | clang-14.0.6 |         1019.3 |          1305.6 |      1276.3 |
    | x86-64  | decompress 4X2 | gcc-12.2.0   |         1348.5 |          1657.0 |      1374.1 |
    | x86-64  | decompress 4X2 | clang-14.0.6 |         1027.6 |          1659.9 |      1468.1 |
    | aarch64 | decompress 4X1 | clang-12.0.5 |         1081.0 |             N/A |      1234.9 |
    | aarch64 | decompress 4X2 | clang-12.0.5 |         1270.0 |             N/A |      1516.6 |
    terrelln committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    8957fef View commit details
    Browse the repository at this point in the history
  5. [version-test] Work around bugs in v0.7.3 dict builder

    Before calling a dictionary good, make sure that it can compress an
    input. If v0.7.3 rejects v0.7.3's dictionary, fall back to the v1.0
    dictionary. This is not the job of the verison test to test it, because
    we cannot fix this code.
    terrelln committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    321490c View commit details
    Browse the repository at this point in the history
  6. Merge pull request #3455 from facebook/fix3454

    Provide more accurate error codes for busy-loop scenarios
    Cyan4973 authored Jan 25, 2023
    Configuration menu
    Copy the full SHA
    efc9ae3 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2023

  1. disable --rm on -o command

    make it more similar to -c (aka `stdout`) convention.
    Cyan4973 committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    8c85b29 View commit details
    Browse the repository at this point in the history
  2. enforce a hard fail when input files are set to be erased

    in scenarios where it's supposed to not be possible.
    
    suggested by @terrelln
    Cyan4973 committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    02434e0 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #3450 from facebook/no_rm_on_o

    disable --rm on -o command
    Cyan4973 authored Jan 26, 2023
    Configuration menu
    Copy the full SHA
    a82e0aa View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6422d1d View commit details
    Browse the repository at this point in the history
  5. Merge pull request #3457 from yoniko/fix-rowhash-cli

    [Bugfix] CLI row hash flags set the wrong values
    
    
    `--[no-]row-match-finder` do the opposite of what they are supposed to.
    In effect the no option would activate row hash while the other option will disable it.
    This commit fixes the issue and changes the code to use the more readable enum values.
    yoniko authored Jan 26, 2023
    Configuration menu
    Copy the full SHA
    7b3f03b View commit details
    Browse the repository at this point in the history
  6. modify cli-test logic : ignore stderr message by default

    Previously, cli-test would, by default, check that a stderr output is strictly identical to a saved outcome.
    When there was no instructions on how to interpret stderr, it would default to requiring it to be empty.
    
    There are many tests cases though where stderr content doesn't matter, and we are mainly interested in the return code of the cli.
    For these cases, it was possible to set a .ignore document, which would instruct to ignore stderr content.
    
    This PR update the logic, to make .ignore the default.
    When willing to check that stderr content is empty, one must now add an empty .strict file.
    
    This will allow status message to evolve without triggering many cli-tests errors.
    This is especially important when some of these status include compression results, which may change as a result of compression optimizations.
    It also makes it easier to add new tests which only care about the CLI's return code.
    Cyan4973 committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    3c21522 View commit details
    Browse the repository at this point in the history
  7. change logic when stderr is not console : don't update progress status

    but keep warnings and final operation statement.
    
    updated tests/cli-tests/ accordingly
    Cyan4973 committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    82ca008 View commit details
    Browse the repository at this point in the history
  8. [huf] Fix bug in fast C decoders

    The input bounds checks were buggy because they were only breaking from
    the inner loop, not the outer loop. The fuzzers found this immediately.
    The fix is to use `goto _out` instead of `break`.
    
    This condition can happen on corrupted inputs.
    
    I've benchmarked before and after on x86-64 and there were small changes
    in performance, some positive, and some negative, and they end up about
    balacing out.
    
    Credit to  OSS-Fuzz
    terrelln committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    bda947e View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2023

  1. Merge pull request #3458 from facebook/stderr_finalStatus

    Update logic when `stderr` is not the console
    Cyan4973 authored Jan 27, 2023
    Configuration menu
    Copy the full SHA
    88b7088 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    814f4bf View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d210628 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    9e4c66b View commit details
    Browse the repository at this point in the history
  5. [fse] Delete unused functions

    Delete all unused FSE functions, now that we are no longer syncing
    to/from upstream.
    
    This avoids confusion about Zstd's stack usage like in Issue #3453.
    It also removes dead code, which is always a plus.
    terrelln committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    423a749 View commit details
    Browse the repository at this point in the history
  6. Update lib/compress/zstd_compress.c

    Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
    daniellerozenblit and terrelln authored Jan 27, 2023
    Configuration menu
    Copy the full SHA
    2bde9fb View commit details
    Browse the repository at this point in the history
  7. update CI

    daniellerozenblit committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    da589a1 View commit details
    Browse the repository at this point in the history
  8. Merge branch 'fix-long-offsets-resolution-pointer' of github.com:dani…

    …ellerozenblit/zstd into fix-long-offsets-resolution-pointer
    daniellerozenblit committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    5ec77ad View commit details
    Browse the repository at this point in the history
  9. Fix invalid assert in 32-bit decoding

    The assert is only correct for valid sequences, so disable it for
    everything execpt round trip fuzzers.
    terrelln committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    b3b43f2 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2023

  1. Update .github/workflows/dev-long-tests.yml

    Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
    daniellerozenblit and terrelln authored Jan 28, 2023
    Configuration menu
    Copy the full SHA
    295724b View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2023

  1. Bump github/codeql-action from 2.1.39 to 2.2.1

    Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.1.39 to 2.2.1.
    - [Release notes](https://github.com/github/codeql-action/releases)
    - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
    - [Commits](github/codeql-action@a34ca99...3ebbd71)
    
    ---
    updated-dependencies:
    - dependency-name: github/codeql-action
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Jan 30, 2023
    Configuration menu
    Copy the full SHA
    dd7fdc9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    66fae56 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'fix-long-offsets-resolution-pointer' of github.com:dani…

    …ellerozenblit/zstd into fix-long-offsets-resolution-pointer
    daniellerozenblit committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    0843d9b View commit details
    Browse the repository at this point in the history
  4. Merge pull request #3460 from daniellerozenblit/fix-long-offsets-reso…

    …lution-pointer
    
    fix long offset resolution
    daniellerozenblit authored Jan 30, 2023
    Configuration menu
    Copy the full SHA
    0017663 View commit details
    Browse the repository at this point in the history
  5. Simplify 32-bit long offsets decoding logic

    The previous code had an issue when `bitsConsumed == 32` it would read 0
    bits for the `ofBits` read, which violates the precondition of
    `BIT_readBitsFast()`. This can happen when the stream is corrupted.
    
    Fix thie issue by always reading the maximum possible number of extra
    bits. I've measured neutral decoding performance, likely because this
    branch is unlikely, but this should be faster anyways. And if not, it is
    only 32-bit decoding, so performance isn't as critical.
    
    Credit to OSS-Fuzz
    terrelln committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    2f74507 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2023

  1. bump version number to v1.5.4

    start preparation for release
    Cyan4973 committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    39ceef2 View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3464 from facebook/dependabot/github_actions/gith…

    …ub/codeql-action-2.2.1
    
    Bump github/codeql-action from 2.1.39 to 2.2.1
    Cyan4973 authored Jan 31, 2023
    Configuration menu
    Copy the full SHA
    71c911d View commit details
    Browse the repository at this point in the history
  3. Merge pull request #3469 from facebook/updateVersion

    bump version number to v1.5.4
    Cyan4973 authored Jan 31, 2023
    Configuration menu
    Copy the full SHA
    4794bbf View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    af09777 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    64052ef View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2023

  1. Merge pull request #3470 from facebook/bench_zstd_only

    ensure that benchmark mode can only be invoked with zstd format
    Cyan4973 authored Feb 1, 2023
    1 Configuration menu
    Copy the full SHA
    ac0746a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7f8189c View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2023

  1. Fix 32-bit decoding with large dictionary

    The 32-bit decoder could corrupt the regenerated data by using regular
    offset mode when there were actually long offsets. This is because we
    were only considering the window size in the calculation, not the
    dictionary size. So a large dictionary could allow longer offsets.
    
    Fix this in two ways:
    1. Instead of looking at the window size, look at the total referencable
       bytes in the history buffer. Use this in the comparison instead of
       the window size. Additionally, we were comparing against the wrong
       value, it was too low. Fix that by computing exactly the maximum
       offset for regular sequence decoding.
    2. If it is possible that we have long offsets due to (1), then check
       the offset code decoding table, and if the decoding table's maximum
       number of additional bits is no more than STREAM_ACCUMULATOR_MIN,
       then we can't have long offsets.
    
    This gates us to be using the long offsets decoder only when we are very
    likely to actually have long offsets.
    
    Note that this bug only affects the decoding of the data, and the
    original compressed data, if re-read with a patched decoder, will
    correctly regenerate the orginal data. Except that the encoder also had
    the same issue previously.
    
    This fixes both the open OSS-Fuzz issues.
    
    Credit to OSS-Fuzz
    terrelln committed Feb 2, 2023
    Configuration menu
    Copy the full SHA
    cc3e3ac View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3472 from terrelln/2023-02-01-fix-32-bit-decoding

    Fix 32-bit decoding with large dictionary
    Cyan4973 authored Feb 2, 2023
    Configuration menu
    Copy the full SHA
    c22c995 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3fe5f1f View commit details
    Browse the repository at this point in the history
  4. Merge pull request #3471 from embg/fast_seq_parse

    Reduce external matchfinder API overhead by 25%
    embg authored Feb 2, 2023
    Configuration menu
    Copy the full SHA
    31e41b3 View commit details
    Browse the repository at this point in the history
  5. Fix ZSTD_getOffsetInfo() when nbSeq == 0

    In 32-bit mode, ZSTD_getOffsetInfo() can be called when nbSeq == 0, and
    in this case the offset table is uninitialized. The function should just
    return 0 for both values, because there are no sequences.
    
    Credit to OSS-Fuzz
    terrelln committed Feb 2, 2023
    Configuration menu
    Copy the full SHA
    71a0259 View commit details
    Browse the repository at this point in the history
  6. AsyncIO performance regression for small files fix (#3474)

    - Do not use threaded AsyncIO when handling small files.
    - Some typo / doc fixes
    yoniko authored Feb 2, 2023
    Configuration menu
    Copy the full SHA
    79bdb8c View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2023

  1. Bump ilammy/msvc-dev-cmd from 1.12.0 to 1.12.1

    Bumps [ilammy/msvc-dev-cmd](https://github.com/ilammy/msvc-dev-cmd) from 1.12.0 to 1.12.1.
    - [Release notes](https://github.com/ilammy/msvc-dev-cmd/releases)
    - [Commits](ilammy/msvc-dev-cmd@7315a94...cec98b9)
    
    ---
    updated-dependencies:
    - dependency-name: ilammy/msvc-dev-cmd
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Feb 6, 2023
    Configuration menu
    Copy the full SHA
    35835f4 View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3477 from facebook/dependabot/github_actions/ilam…

    …my/msvc-dev-cmd-1.12.1
    
    Bump ilammy/msvc-dev-cmd from 1.12.0 to 1.12.1
    felixhandte authored Feb 6, 2023
    Configuration menu
    Copy the full SHA
    54a173e View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2023

  1. return error code when benchmark fails

    such scenario can happen, for example,
    when trying a decompression-only benchmark on invalid data.
    Other possibilities include an allocation error in an intermediate step.
    
    So far, the benchmark would return immediately, but still return 0.
    On command line, this would be confusing, as the program appears successful (though it does not display any successful message).
    
    Now it returns !0, which can be interpreted as an error by command line.
    Cyan4973 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    9cabd15 View commit details
    Browse the repository at this point in the history
  2. added more accurate error messages

    for the decompression-only benchmark mode.
    Cyan4973 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    58e7067 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6740f8f View commit details
    Browse the repository at this point in the history
  4. Merge pull request #3480 from facebook/benchFail

    return error code when benchmark fails
    Cyan4973 authored Feb 7, 2023
    Configuration menu
    Copy the full SHA
    df21ace View commit details
    Browse the repository at this point in the history
  5. fix oss-fuzz case 55714

    impacts legacy decoder v0.3 in 32-bit mode
    Cyan4973 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    e04706c View commit details
    Browse the repository at this point in the history
  6. fix for v0.3 blindly ported to v0.2

    in case it would be applicable here too.
    Cyan4973 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    cfec005 View commit details
    Browse the repository at this point in the history
  7. adapt v0.3 fix to v0.1

    slightly different constraints on end of buffer conditions
    Cyan4973 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    7eb4471 View commit details
    Browse the repository at this point in the history
  8. copy fix for v0.3 to v0.4

    in case it would be applicable for this legacy version too.
    Cyan4973 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    b20e4e9 View commit details
    Browse the repository at this point in the history
  9. port fix for v0.3 to v0.5

    in case it would be applicable for this version too
    Cyan4973 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    7a1a171 View commit details
    Browse the repository at this point in the history
  10. port fix for v0.3 to v0.6

    in case it would applicable for this version
    Cyan4973 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    67d7a65 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    9419747 View commit details
    Browse the repository at this point in the history
  12. add requested check for legacy decoder v0.1

    which uses a different technique to store literals,
    and therefore must check for potential overwrites.
    Cyan4973 committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    c5bf6b8 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2023

  1. Configuration menu
    Copy the full SHA
    c689310 View commit details
    Browse the repository at this point in the history
  2. Merge pull request #3476 from facebook/fix55714

    fix oss-fuzz case 55714
    Cyan4973 authored Feb 8, 2023
    Configuration menu
    Copy the full SHA
    488f7c0 View commit details
    Browse the repository at this point in the history
  3. Fix empty-block.zst golden decompression file

    This frame is invalid because the `Window_Size = 0`, and the
    `Block_Maximum_Size = min(128 KB, Window_Size) = 0`. But the empty
    compressed block has a `Block_Content` size of 2, which is invalid.
    
    The fix is to switch to using a `Window_Descriptor` instead of the
    `Single_Segment_Flag`. This sets the `Window_Size = 1024`.
    
    Hexdump before this PR: `28b5 2ffd 2000 1500 0000 00`
    
    Hexdump after this PR: `28b5 2ffd 0000 1500 0000 00`
    
    For issue #3482.
    terrelln committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    83f8a05 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4de9d63 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    67fb9d9 View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2023

  1. Configuration menu
    Copy the full SHA
    4aa3bc4 View commit details
    Browse the repository at this point in the history
  2. Rename "External Matchfinder" to "Block-Level Sequence Producer" (#3484)

    * change "external matchfinder" to "external sequence producer"
    
    * migrate contrib/ to new naming convention
    
    * fix contrib build
    
    * fix error message
    
    * update debug strings
    
    * fix def of invalid sequences in zstd.h
    
    * nit
    
    * update CHANGELOG
    
    * fix .gitignore
    embg authored Feb 9, 2023
    Configuration menu
    Copy the full SHA
    ff42ed1 View commit details
    Browse the repository at this point in the history
  3. updated man pages

    Cyan4973 committed Feb 9, 2023
    Configuration menu
    Copy the full SHA
    95ffc76 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    515266e View commit details
    Browse the repository at this point in the history