Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal compiler error triggered by preprocessor inspecting code comments #18609

Closed
sgbeal opened this issue Jan 27, 2023 · 11 comments
Closed
Assignees

Comments

@sgbeal
Copy link

sgbeal commented Jan 27, 2023

Please include the following in your bug report:

Version of emscripten/emsdk:

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.31 (e88336121cfe6da4a96c88e46f314552f07dfed0)

The build itself is extremely "involved", but the reason for the breakage is clear...

The message:

Internal compiler error in src/compiler.js!
Please create a bug report at https://github.com/emscripten-core/emscripten/issues/
with a log of the build and the input files used to run. Exception message: "Error: /home/stephan/f/s/lite/ext/wasm/bld/post-js.vanilla.js:8566: Unknown preprocessor directive #1",
    at preprocess (/home/stephan/src/emsdk/upstream/emscripten/src/parseTools.js:131:19)
    at includeFile (/home/stephan/src/emsdk/upstream/emscripten/src/jsifier.js:458:25)
    at finalCombiner (/home/stephan/src/emsdk/upstream/emscripten/src/jsifier.js:530:7)
    at runJSify (/home/stephan/src/emsdk/upstream/emscripten/src/jsifier.js:554:3)
    at Object.<anonymous> (/home/stephan/src/emsdk/upstream/emscripten/src/compiler.js:97:3)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)

The cause of that is a comment block with "#1" in it. About halfway down...

/*
  2022-07-22

  The author disclaims copyright to this source code.  In place of a
  legal notice, here is a blessing:

  *   May you do good and not evil.
  *   May you find forgiveness for yourself and forgive others.
  *   May you share freely, never taking more than you give.

  ***********************************************************************

  This file implements the initializer for the sqlite3 "Worker API
  #1", a very basic DB access API intended to be scripted from a main
  window thread via Worker-style messages. Because of limitations in
  that type of communication, this API is minimalistic and only
  capable of serving relatively basic DB requests (e.g. it cannot
  process nested query loops concurrently).

  This file requires that the core C-style sqlite3 API and OO API #1
  have been loaded.
*/

Reformatting the comment so that the line does not start with the hash sign works around it, but doing so requires rephrasing the documentation so that the auto-wrapper just happens to not place that token at the start of the line. Any given future edits and auto-wrapping might undo that kludge.

sqlite pushed a commit to sqlite/sqlite that referenced this issue Jan 27, 2023
…n#18609].

FossilOrigin-Name: fa784101775b795077a23c211b5b16f51ad5a13967c284511f310dfcbfa9f77a
mackyle referenced this issue in mackyle/sqlite Jan 27, 2023
@sbc100 sbc100 self-assigned this Jan 27, 2023
@sbc100
Copy link
Collaborator

sbc100 commented Jan 29, 2023

Yes, our pre-processor is a little basic I'm afraid. Sadly it looks like handling comments like this correctly might add quite a bit a complexity. If you want to take a look at the code it's here:

// Simple #if/else/endif preprocessing for a file. Checks if the
// ident checked is true in our global.
// Also handles #include x.js (similar to C #include <file>)
function preprocess(filename) {

@sgbeal
Copy link
Author

sgbeal commented Jan 30, 2023

Indeed, handling comments correctly is a pain in the butt, but generating a compilation error for perfectly legitimate client code which doesn't have any reliance on that preprocessor is "just wrong." Rather than add proper comment support, perhaps the preprocessor should be opt-in? Certainly only a minority of client code makes use of it?

@sbc100
Copy link
Collaborator

sbc100 commented Feb 15, 2023

What do you think would be good way to opt in? I'd rather avoid an entirely new --pre-js-preprocessed flag.

I'm think maybe we can just remove that error, and ignore unknown directives.

@sgbeal
Copy link
Author

sgbeal commented Feb 15, 2023

i was initially (and perhaps naively) thinking --preprocess would enable it globally, but making it opt-in may well break any number of client trees, so i'm no longer a fan of that approach. Silently skipping unknown directives sounds destined to cause more grief than incorrectly reading directives from comments. i'd currently rather see it stay as-is than take either of those approaches, on the grounds that the current code seems likely to break with the fewest clients and has a relatively painless (just annoying) client-side workaround.

Perhaps... rather than extending the preprocessor to support comments, add a comments-stripping-only preprocessor step before the current preprocessor. That would keep the comment-stripping complexity out of the core preprocessor. The only(?) tricky part about stripping comments is making sure not to strip them from inside string literals. As a real-world case of that, in the sqlite project we have some SQL strings which contain /*...*/ comments.

@sbc100
Copy link
Collaborator

sbc100 commented Feb 15, 2023

Sadly its not just you can has run into this issue We got another report recently from someone using ES6 private members: #18724. In this case there is no good workaround for the user, so I think the status quo cannot stand.

The way I see it that leave 2 choices:

  1. Make a smarter, context aware parser that can handle both comments a private member fields.
  2. Ignore unknown directives
  3. Make preprocessing optional, on a per-file bases (perhaps in combination with 2).

With the added complexity private members I don't think (1) is feasible.

The risks of (2) are that there are some rare cases were user might use #if or #else or #endif as private members or in comments. This seems exceeding unlikely, but also is pretty gross failure mode. Is this what you mean when you say "destined to cause grief", or did you have something else in mind?

So I think (3) is our best option. One questions is, do we make this opt-on or opt-out. It seems like if we ignore unknown directives, then making it opt-out could work, particularly if we could detect failure cases and recommend the opt-out? What about something like "Encountered unknown preprocessor directive. If this is not actually a directive for the pre-processor you can opt by doing "?

@sgbeal
Copy link
Author

sgbeal commented Feb 15, 2023

This seems exceeding unlikely, but also is pretty gross failure mode. Is this what you mean when you say "destined to cause grief", or did you have something else in mind?

That was precisely the kind of thing i had in mind: completely innocent docs/comments which cause it to choke.

Private members are something i had not considered at all. That's a whole other can of worms.

@sgbeal
Copy link
Author

sgbeal commented Feb 15, 2023

What if the preprocessor syntax was changed? e.g. in the sqlite project we preprocess some of our JS files but the CPP preprocessor just makes a mess of it, so we use use a custom preprocessor with a configurable directive marker. We use:

//#if ...
...
//#else
...
//#endif

they look like comments to JS but //# is the delimiter for the preprocessor. Perhaps using ## as a delimiter would suffice for emscripten?

@sbc100
Copy link
Collaborator

sbc100 commented Feb 15, 2023

Changing the pre-process syntax would be tricky since it would effect all the third-party JS library code that relies on it.

Have a different pre-processor for JS library code and for pre/post-js also seems like a bad outcome.

I'm still leaning towards the "explicit opt-out along with useful error/hint when opt out is required/needed"

@sgbeal
Copy link
Author

sgbeal commented Feb 15, 2023

I'm still leaning towards the "explicit opt-out along with useful error/hint when opt out is required/needed"

FWIW, that sounds like the least painful option all around.

@hostilefork
Copy link
Contributor

I have an error which looks suspiciously related. It started happening in SDK 3.1.31 (no problems in 3.1.30)

However it's an assertion failure:

Internal compiler error in src/compiler.js!
Please create a bug report at https://github.com/emscripten-core/emscripten/issues/
with a log of the build and the input files used to run. Exception message: "AssertionError [ERR_ASSERTION]: false == true
    at preprocess (/home/ae1020/emsdk/upstream/emscripten/src/parseTools.js:120:13)
    at includeFile (/home/ae1020/emsdk/upstream/emscripten/src/jsifier.js:458:25)
    at finalCombiner (/home/ae1020/emsdk/upstream/emscripten/src/jsifier.js:530:7)
    at runJSify (/home/ae1020/emsdk/upstream/emscripten/src/jsifier.js:554:3)
    at Object.<anonymous> (/home/ae1020/emsdk/upstream/emscripten/src/compiler.js:97:3)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)

My case is also extremely..."involved". 🙄 So if it's conceivable that this assert would trigger given knowledge already present I can use 3.1.30 until it is fixed.

hostilefork added a commit to metaeducation/ren-c that referenced this issue Feb 17, 2023
clrpackages pushed a commit to clearlinux-pkgs/sqlite-autoconf that referenced this issue Feb 23, 2023
…version 3.41.0

dan (81):
      Avoid returning SQLITE_SCHEMA if the first query run on a connection is "SELECT nosuchcolumn" or similar. Forum post <https://sqlite.org/forum/forumpost/c1798f77ef>.
      Add Makefile.in targets for sqlite3r.c and sqlite3r.h, versions of the amalgamation that include the recover extension.
      Update multiplex3.test to account for the fact that the multiplexor xDelete method may return an error even if it manages to delete the first chunk of a file.
      Fix a test case in fts3expr4.test to account for different locales.
      Ensure the RETURNING clause is honoured when a row of a temp table is updated by an ON CONFLICT clause.
      Add the SQLITE_FCNTL_RESET_CACHE verb. Use it to ensure that the page cache is purged before and after a the recovery extension is run.
      Fix legacy build system main.mk to include basexx.c in testfixture builds.
      Enhance the sqlite3_stmt_scanstatus() API and add sqlite3_stmt_scanstatus_v2(). For creation of easier to read query performance reports.
      Add CYCLES scanstat measurement to "USE TEMP B-TREE FOR ORDER BY" lines.
      Enhance SQLITE_SCANSTAT_NCYCLE so that it reports on virtual tables.
      Fix a problem in the memdb vfs xLock() function allowing clients to upgrade to EXCLUSIVE locks when other connections are holding SHARED.
      Add test case that should have been part of previous commit.
      Add loops and rows counters to "USE TEMP B-TREE FOR ORDER BY" records. Also fix the sqliteHwtime() function so that it returns a 64-bit value.
      Update comments in sqlite.h.in to account for sqlite3_stmt_scanstatus_v2().
      Support an SQLITE_SCANSTAT_NCYCLE statistic for "CO-ROUTINE" elements.
      Have sqlite3_stmt_scanstatus_v2() return an NCYCLE value for all loops, not just virtual tables ones. The value returned is the sum of the NCYCLE counts for the various opcodes that move or read data from the table or index cursor associated with the loop.
      Have sqlite3_stmt_scanstatus() report cycle, loop and row counts separately for creating an automatic index and using that automatic index.
      Reduce the overhead of SQLITE_ENABLE_STMT_SCANSTATUS some.
      Support ".scanstats est" to enable including planner estimates in query profiles.
      Ensure that the types and collation sequences of a view are taken from the leftmost component of compound SELECTs, even when column names are explicitly provided for the view. Possible fix for [679ed6a2].
      Add new logging code SQLITE_NOTICE_RBU and use it when logging for the spurious error that RBU injects into SQLite as part of applying an update.
      Add extra test cases for applying sessions module changesets to databases that have been modified using "ALTER TABLE .. ADD COLUMN".
      Prevent the sessions rebaser from generating changesets containing UPDATE records for which non-PK fields are present in the old.* but not the new.* record. Also update the changeset iterator to work around such changesets.
      Fix a comment in ext/session/test_session.c.
      Fix an unsafe use of sqlite3_mprintf() in sqlite3_overload_function() identified by forum post: <https://sqlite.org/forum/forumpost/95b338860d>.
      Update a test case in scanstatus2.test to account for recent planner enhancements.
      Fix an incompatibility between the Tcl interface and the "memdb" vfs by allowing memdb to accept filenames that begin with '\' characters.
      Fix an assert() in fts5. Simplify memdb xLock/xUnlock some.
      Fix a faulty assert() in fts5.
      Fix a broken assert() in fts5.
      Ensure that LIMIT clauses may be passed through to virtual table implementations even if the WHERE clause uses operators that may only be optimized by virtual, not built-in, tables (!=, functions, MATCH etc.).
      Fix a problem with applying integer affinity to a MEM_IntReal value. Forum post [forum:/forumpost/d270125fae|forum post d270125fae].
      Account for read-only connections on databases with non-default block or page sizes.
      Fix some problems with large, compressed, lsm databases.
      Fix handling of unix paths that contain ".." components such that "/" is considered its own parent directory.
      Fix another 64-bit offset problem in lsm.
      When deleting any old OAL file before starting an RBU update or vacuum, use the same VFS as will be used for the target database, even if this is not the system default.
      Avoid crashing when using an RBU VFS with a version 1 parent VFS.
      Do not attempt to run new test file rbupass.test if the "demo" VFS is not available.
      Fix another lsm compression-mode bug.
      Update virtual tables json_each and json_tree so that adding "ORDER BY rowid" to a query does not require an external sort.
      Fix another issue with very large compressed LSM databases.
      Fix numbering issue in and remove duplicate test from windowB.test.
      Add tests for changing the database encoding via RESET_DATABASE/VACUUM. And test that it is not possible to trick another connection with this.
      Ensure that the database encoding cannot be changed while there are statements running. And that the connection is left in a valid state after an obscure OOM within sqlite3_deserialize().
      Add experimental user function unhex().
      Fix a problem with using fts3 auxiliary functions with expressions like "E AND ...", where E is a NEAR expression that consists entirely of deferred tokens.
      Update unhex() to allow a second argument, specifying a set of characters that are permitted to appear between pairs of hexadecimal digits.
      Fix another problem with fts3/4 auxiliary functions and NEAR expressions that consist entirely of deferred tokens.
      Fix a problem with fts3 auxiliary functions and one or more NEAR expressions ORed together.
      Have some RBU tests run as part of veryquick.test/testrunner.tcl.
      Do not try to run rbu tests with builds that do not support rbu.
      Update sqllimits1.test to account for the fact that if an odd value is specified as the length in bytes of a utf-16 string, it is truncated to the largest smaller even number.
      Fix a problem causing "PRAGMA quick_check" to return spurious corruption errors for a WITHOUT ROWID for which all columns are either virtual or part of the primary key, and for which the order of the columns in the primary key definition is different from the order in the table.
      Do not assume that sub-queries that contain window functions are uncorrelated.
      Fix a race condition during hot-journal rollback that could theoretically cause spurious corruption errors.
      Fix a comment related to PENDING locks in os_unix.c. No code changes.
      Update testrunner.tcl to use a separate process for each test script. And to run some extra tests too.
      Add configuration data for osx and windows to testrunner_data.tcl
      Do not run test script pendingrace.test as part of the inmemory_journal permutation.
      Avoid using Tcl command [clock] in testrunner.tcl.
      Have testrunner.tcl run various builds as part of the 'release' command.
      Fix a problem where optimizing an fts5 table too often might cause it to become unreadable.
      Fixes for testrunner.tcl on windows.
      Add testrunner.tcl "njob" and "status" commands.
      Update testrunner.tcl usage message.
      Changes so that wapptest.tcl still works.
      Avoid running rbu tests under permutation "journaltest", which does not support wal. Fix some other test script problems.
      Fix some test scripts that were failing with SQLITE_OMIT_VIRTUALTABLE builds.
      Fix problems with test scripts preventing them from running with SQLITE_DEFAULT_MEMSTATUS=0 builds. Or builds without SQLITE_ENABLE_STAT4.
      Add the --fuzztest option to testrunner.tcl. Also, have it print whitespace over the top of the old report line before writing the new.
      Fix a formatting problem in scanstatus2.test introduced by [81c118d9].
      Update test scripts speed1.test and speed1p.test so they may be run by testrunner.tcl.
      Fix RBU test scripts to avoid attempting to copy or delete a locked file.
      Fix a test script problem causing "make test" to fail for non-RBU builds.
      Update an assert() in the stat4 code that is only true for a well-formed database.
      Better fix the problem where optimizing an fts5 table too often causes it to become unreadable (first attempt was [35bed981]).
      Fix a problem with the fts5 trigram tokenizer and LIKE or GLOB patterns for which contain runs of 2 or fewer non-wildcard characters that are 3 or more bytes when encoded as utf-8.
      Allow vector-IN expressions like "(a, b) IN ( (?,?), (?,?) )" to use an index.
      Fix compile time option SQLITE_DEFAULT_SYNCHRONOUS so that it works consistently.
      Update testrunner.tcl to run zipvfs test scripts on unix.

drh (206):
      Fix harmless typo in comment, reported by Debian in Fossil
      Raise an error if an attempt is made to compile the CLI with the
      Split out the documentation for sqlite3_value_encoding() into its own
      Use the log10() and log2() functions from the standard C library to implement
      Fix corner cases in UTF8 handling in the REGEXP extension.
      Update the version number to 3.41.0 to begin the next development cycle.
      Databases created using sqlite3_deserialize() should report their filename
      Small performance optimization in btree.c.
      Streamline the decodeFlags() routine in btree.c for a small performance
      Improved comments on the pageFreeArray() routine of btree.c.  No changes to
      Small performance improvement and size reduction in btree.
      Performance improvement and size reduction in balance_nonroot().
      Performance optimization in sqlite3BtreeInsert().
      New assert()s confirm that BTREE_PREFORMAT must be the same value as
      Small performance improvement in sqlite3BtreeTransferRow().
      Performance optimization to sqlite3BtreeInsert().
      Performance optimization and size reduction in insertCell() by omitting
      Change the name of the Parse.pIdxExpr field to pIdxEpr so that the name is
      Split out the debugging code that dumps an AggInfo object into a separate
      Convert an ALWAYS() in sqlite3DbSpanDup() into an assert(), for a performance
      This is the first in what is anticipated to be a long sequence of incremental
      Factor out the allocation of registers for aggregates into a separate
      Omit the unnecessary AggInfo.mnReg field.
      Since the memory registers used by the columns and functions of an
      Fix the error message in the CLI for "no such VFS".
      Rename the SELECTTRACE macro to TREETRACE, so that is corresponds to the new
      Include the treetrace bitmap comment accidentally omitted from the previous
      Add the stub function: optimizeAggregateUsingIndexedExpr().  The hope is that
      Further foundation prep work prior to starting to flesh-out the
      This attempt at modifying AggInfo to make use of indexed expressions does not
      Aggregates with GROUP BY now make use of expressions on indexes.  This code
      Add explanatory comment to the new optimization.  And add a test case.
      New test cases.  Fix the logic so that it works for GROUP BY aggregates
      Add NEVER() and ALWAYS() macros on branches that are believed to be
      Change a NEVER() into an assert().
      Take care not to try to add comments to a TK_AGG_COLUMN opcode that does
      Correctly deal with IF-NULL-ROW operators when dealing with
      Improved comments.  Add assert()s to verify that the AggInfo structure
      Improve the robustness of the updateAccumulator() routine against OOM.
      Remove a redundant assert() statement.
      Add restriction (9) to the push-down optimization:  If the subquery is
      Fix #ifdefs so that restrictions (8) and (9) of the push-down optimization
      Relax restriction (8) on the push-down optimization so that it only applies
      Experimental changes to help the query planner detect when an expression
      Fix an incorrect assertion-fault in the "TREETRACE" debugging logic on
      Show which AggInfo.aCol an Expr node represents in the tree trace debugging
      Better reporting of when the WHERE clause analysis thinks that an index
      Rework the covering index checking routine, whereIsCoveringIndex(), so that
      Test cases added derived from the enhancment ticket.
      Fix harmless compiler warnings.
      Check-in [8d5b76593d82b3a5] contained an error that was causing some obscure
      Change the handling of hwtime.h to make it easier to compile performance
      Always use nanosleep() (instead of usleep() or sleep) if the
      Attempt to rationalize the bits associated with ".wheretrace".  Provide
      Fix an over-zealous assert() reported by Yong Heng.
      Add a testcase() macro to verify that the case of a NOT NULL error message
      Use sqlite3_result_int64() instead of sqlite3_result_int() when returning
      For the sqlite3_bind and sqlite3_result interfaces for UTF16 strings, round
      Tuning the query planner by adjusting the weights that predict the relative
      Further improvements to the estimated cost of sorting.  Take into account
      Add a test case to show that ticket [e8b674241947eb3b] has been fixed.
      Add TOTAL rows to the output generated by tool/vdbe_profile.tcl.
      Increase the nominal row size for IPK index lookups slightly, for better
      Use the smaller estimated row size for searching IPK tables, but use the
      Remove obsolete "experimental" comments on the virtual-table interface
      In the unix backend, when implementing the defenses against small file
      Fix compiler warnings in the new dynamic continuation prompt logic of
      Simplified experimental changes to promote the use of co-routines.  Less
      Fix a (harmless) off-by-one error in code generation that comes up when
      Streamline and improve testing of the locking in the memdb VFS.
      Fix harmless compiler warning in the dynamic continuation prompt of the CLI.
      Do not use a co-routine on a subquery that is participating in a self-join.
      Improved comments on the new co-routine logic.  Fix a C++-ism in the code.
      Fix typo in the SQLITE_DBCONFIG_LOOKASIDE documentation.
      Do not allow OOM faults in EQP messages associated with automatic indexes
      Remove the SQLITE_PREPARE_SAFEOPT flag.  The name is obsolete and it is at the
      Rerun autoconf using version 2.69 to minimize configure script differences
      If a subquery has a result column of the form "CAST(... AS NUMERIC)" then
      Slightly faster implementation of sqlite3ExprAffinity().
      Refactor the sqlite3SelectAddColumnTypeAndCollation() routine.  Improved
      Use standard datatype names for the column datatypes in views and subqueries.
      Remove an unused variable from sqlite3SubqueryColumnTypes().
      Back out the part of the change in [88a05141c28e5ff1] that adds affinity to
      If the SELECT that implements a subquery or a VIEW or a CREATE TABLE AS is
      Fix minor problems in the new sqlite3ExprDataType() function.
      Fix harmless compiler warnings.
      Omit the columnTypeImpl() function from the build when SQLITE_OMIT_DECLTYPE
      Create a new affinity called FLEXNUM that works like NUMERIC except that it
      In the fuzzer invariant checker, do not add new WHERE clause terms that
      Disallow some of the query invariant tests on SQL statements that contain
      Add NEVER/ALWAYS to branches that have become unreachable due to recent
      Fix an unnecessarily restrictive assert() in the aggregate logic.
      Fix a minor problem with the printAggInfo() routine (used on during
      Remove an unused variable.
      Do not use indexed expressions to replace the expression being indexed on a
      Improvement to the dbsqlfuzz fix of [f113eebdbe68246f].
      Add ALWAYS() to always-true branches in the locking logic of the memdb VFS.
      Add about 150 new fuzzer cases to test/fuzzdata8.db.
      New WHERETRACE entries for adding and removing objects from the
      Ensure that the expression of a virtual column really is an expression and
      More efficient implementation of large precisions on the "%c" format for
      Small performance optimization on the OP_Insert opcode.
      Small performance optimization associated with shared cache in the
      Rename the Vdbe.pResultSet field to pResultRow in order to better distinguish
      Avoid having to reinitialize Vdbe.pResultRow upon each call to sqlite3_step()
      Asserts added to ensure that the iCompare variable in the bytecode engine
      Additional fixes for yet more completely harmless compiler warnings.
      Fix lots of harmless, nuisance compiler warnings, mostly unused parameter
      Fix an infinite loop in the MEMSYS5 auxiliary memory allocator that occurs
      Fix a missing 0 at the end of an integer literal in check-in [8da0f0c38a458c57]
      A call to sqlite3_declare_vtab() should not cause DML/DDL authorization
      Small performance improvement in the btreeOverwriteCell() routine.
      Avoid an unnecessary call to strlen() in the sqlite3VdbeMemStringify() routine,
      Small performance increase in the symbol hash table.
      Remove the unused "sqlite3StackAllocZero()" macro.
      Enhance PRAGMA integrity_check so that it verifies that the string values
      Adjustements to the tool/warnings.sh script to account for compiler
      In the zipvfile extension (which is not part of the SQLite amalgamation, but
      Changes a testcase() into an assert() due to the
      Fix JSON functions so that they work correctly under PRAGMA trusted_schema.
      Improvements to the SQLITE_DIRECTONLY documentation.
      When computing the datatypes for columns in a view, use the same datatype name
      Add a new sqlite3_is_interrupted() interface that can be used by long-running
      Improved progress-handler and interrupt detection during PRAGMA integrity_check.
      Add an assert() to the byte-code engine that goes off if the OP_Halt opcode
      If OP_Rewind has P2 of zero, that is an assertion that the table is never
      Fix duplicate semicolon in btreeInt.h.
      Attempt to provide a mechanism to do early termination of long-running
      Fix sqlite3_prepare() so that it only invokes the progress handler on every
      Reduce the frequency of calls to sqlite3ProgressCheck().
      Remove an incorrect legacy assert().
      Update documentation for sqlite3_progress_handler().
      Add missing "const" on variables in recent RBU fixes.
      In expression nodes of type REGISTER with sub-type COLUMN, ensure that the
      Omit the long-disused FTS1 and FTS2 implements from the active source tree.
      In the CLI, create our own private version of strncpy() to work around
      Add test cases to confirm that the schema parsing quirk in which an
      Fix harmless compiler warnings in the CLI.
      Change an now unreachable testcase() into an assert().
      The json_group_array(), json_group_object(), and fts5_source_id() functions
      Two branches associated with memdb are now always taken (I believe).  Tag
      Extend the carray extension to allow the use of BLOB values to be bound
      Fix harmless compiler warnings.
      Fix a memory leak in the TCL test harness associated with carray.
      Add the missing CARRAY_BLOB macro to the carray.h extension header.
      Due to a coding error, check-in [8efd61e8518594e3] did not actually use
      Suppress a harmless compiler warning.
      Enhance the sqlite3_vtab_in_first() and sqlite3_vtab_in_next() interfaces so
      Defer calling the destructor for the user data on a module until after the
      Fix a harmless compiler warning in FTS3.
      Improved fix to allow sqlite3_vtab_in_first() to reliably return SQLITE_ERROR.
      Add an assert() to help static analyzers.
      Fix compiler warning in base85.c.
      Show more details about the SrcItem.fg field in tree-trace output.
      Additional tweaks to the enhancement at [609fbb94b8f01d67] to further reduce
      Just because a CTE is used more than once, does not mean it should be
      New assert() statements to verify that sqlite3DeleteIndexSamples() is always
      Adjustments of assert() statement in STAT4 in order to give 100% MC/DC.
      Resolve all possible aliases and variations of the schema table names.
      Improved and simplified logic for resolving the various aliases of the schema
      Improved detection of invalid command-line arguments to the showdb and
      Attempt to fix harmless compiler warnings that reportedly appear in clang 15.
      Better error message when trying to do an INSERT on an sqlite_dbpage virtual
      Remove an ALWAYS() that can be false in some very rare cases.
      Add ALWAYS() and NEVER() macros to unreachable branchs in sqlite_dbpage.
      Changes to the sqlite_dbpage virtual table to tag or remove unreachable
      Modify the status line output from testrunner so that each line overwrites
      Restore a test for pBt NULL that was removed by [12a14711afa6cddb] but turned
      In fuzzcheck, only show the description of each database if the -v option
      Add the "devtest" makefile target that runs both fuzztest and testrunner.
      Do not use a Bloom filter if any outer loop lacks STAT1 data, since without
      Update "releasetest" makefile targets to use testrunner.tcl.
      Fix error in the releasetest makefile target from the previous check-in.
      Shorten the status line for testrunner.tcl so that it fits on an 80-character
      Remove the long obsolete "client/server" mode tests.
      Fix the build after the previous change
      Fix a typo in example documentation code for sqlite3_vtab_in_next().
      Add support for the 'txn' argument to date/time functions that works like
      Simplify the code and add test cases.
      Change the behavior of date-time functions without any arguments (and thus
      Revert the behavior of date/time functions with no arguments so that they
      Test cases to show that CURRENT_TIMESTAMP and similar hold the same value
      Fix an incorrect assert() in STAT4 logic added just a few days ago on
      Always use 64-bit integers for stats associated with STAT1 and STAT4.
      Back out the 'txn' enhancement to date/time functions.  The duration of a
      The "flexnum" affinity that was added by [44135d6ea84f7ba6] needs to also
      New test cases added to fuzzdata8.db.
      Disable the double-quoted string misfeature by default in CLI builds.
      Ensure that the valueFromFunction() routine does not clear a prior
      Do a better job of detecting when a WHERE clause term might be useful to
      Change a variable from 32 to 64-bits to avoid a harmless compiler warning
      In the LIKE optimization, do not analyze the new virtual WHERE clause terms
      Do not allow WHERE clause terms to match constant string index terms, which
      Ignore extra parentheses around a subquery on the RHS of an IN operator,
      Do not allow the COUNTOFVIEW optimization to run if the count() contains
      Fix a harmless typo in the test case added by [29fc06465efb948f].
      Fix a harmless UBSAN warning in debugging code of the new unhex() function.
      Fix stale requirement marks and fix a typo in the documentation for
      Fix a harmless compiler warning.
      Fix an incorrect #ifdef in the CLI.
      Update the TEA version number to 3.41.0
      Fix formatting and improved documentation on the various sqlite3_file_control()
      Fix a minor typo in a comment.  No changes to working code.
      Yet another comment typo fix.  No changes to working code.
      Disable DQS by default in the MSVC makefile for the amalgamation tarball.
      Version 3.41.0

larrybr (27):
      Create new branch named "base_convert"
      New extensions for base85 and base64 conversion UDFs
      Fix a base64 decode bug. Provide for convenient inclusion of extension(s) built into the CLI, to simplify testing and for its own sake. Improve comments. Cure collision between base64.c and base85.c when both are in the same translation unit.
      Get clean builds for MS tools. Add build examples for extension glommer (basexx.c)
      Fix last function pointer in sqlite3Apis init. Reported at https://sqlite.org/forum/info/eba0faa96d
      Speed up base64 conversions, and add test with more data for the baseNN conversion to grind.
      Conform CLI .trace arg handling to its help.
      Get ext/misc/basexx.c into the testfixture.exe build for MSC.
      Cause CLI .sha3sum to warn of text fields that do not survive CAST(CAST(t as BLOB) AS TEXT) due to invalid UTF encoding.
      Fix safe mode authorizer callback to reject disallowed UDFs. Reported at [forum:/forumpost/07beac8056151b2f|Forum post 07beac8056151b2f].
      Add optional feature: A CLI continuation prompt which reflects open lexemes and parens, similarly to PG shell.
      Clear a few more -Wall warnings and simplify dynaprompt feature keep/omit macros.
      For CLI .sha3sum, emit warning to stderr for any invalidly encoded text fields.
      Avoid several -Wall warnings in textfixture build.
      Omit CLI use of pragma_table_xinfo when it is not defined in the build.
      Remove slew of inadvertant tabs in favor of spaces.
      Add base64() and base85() text/blob conversions to the CLI.
      Add base64() and base85() to shell sources for the non-configured makefiles, too.
      Grammar fixup in comment re SQLITE_TRACE_PROFILE trace event.
      Employ deliberate_fall_through macro to quiet some compilers.
      Doc-only update, sqlite3_preupdate_hook() return
      Clarify help for .quit.
      Doc-only fix, per [forum:/forumpost/0cfaf6876b|forum post 0cfaf6876b]
      Add OOM check, per tip at [forum:/forumpost/933479b2d5|forum post 933479b2d5]
      Cause .clone to not trip over sequence table as reported at [forum:/forumpost/71ff9e6c4c|forum post 71ff9e6c4c].
      Give CLI .version a place in .help output.
      Cause gcc warning suppression in shell.c to be nice in other project(s).

mistachkin (1):
      Avoid naming collision between the sha1 and shathree extensions.

peter.d.reid (2):
      Avoid an infinite loop when an unexpected character is being decoded by kvvfs
      Remove redundant assignment in kvvfs's decoding.

stephan (218):
      Experimentally use clang's C preprocessor to filter #ifdef's out of the generated sqlite3-api.js, the goal being to see if we can feasibly use cpp to include ES6 module-specific code in the main code base and conditionally filter it out.
      shell.c.in: on non-Windows platforms, check for $XDG_CONFIG_HOME/sqlite3/sqliterc before ~/.sqliterc, per request in [forum:7a16582b1e403c81|forum post 7a16582b1e403c81].
      wasm build: include FTS5 instead of FTS4, per /chat and HN discussions.
      Change a self.X JS reference to X to account for a symbol resolution discrepancy between vanilla JS and ES6 modules, as explained in [forum:801d8f77e5115141|forum post 801d8f77e5115141].
      Replace use of cpp with the fit-to-purpose c-pp to avoid cpp's C-centric/JS-unfriendly quirks.
      More work towards creation of a ES6 JS module.
      Add build of sqlite3.mjs (ES6 module), add a test app for it, and include it in the dist build.
      Preliminary patches to get sqlite3.c building as-is in WASI environments.
      Account for lack of mmap(), getpid(), and shared memory APIs in wasi.
      Default to SQLITE_THREADSAFE=0 and SQLITE_OMIT_LOAD_EXTENSION for wasi builds.
      Elide a wasi-incompatible shell.c block in SQLITE_WASI builds.
      Get tester1.js working via an ES6 worker module and add that variant to the dist zipfile.
      Minor cleanups in the ESM-related preprocessor filtering.
      Add 'snapshot' target to create distinctly-named snapshot builds of the wasm deliverables zip file.
      Generic cleanups and doc additions in the wasm build files.
      Remove an obsolete reference to WASMFS from ext/wasm/index.html.
      Further minor cleanups in the JS build related to vanilla vs ESM.
      js dist: account for a file rename in the previous checkin.
      Remove check for WASM_WASI macro when detecting wasi compilation mode, as that macro is project-specific. Rely only on __wasi__ (exposed by clang) to detect wasi compilation mode.
      Add test app for experimenting with multi-worker OPFS concurrency. Tweak OPFS VFS to significantly improve the otherwise "unfortunate" concurrency situation.
      Resolve missing SQLITE_LOCKED result code which triggered a new (since last checkin) exception in the OPFS VFS. Improve output of the OPFS contention tester app.
      More tweaking of OPFS concurrency measures and the related test app.
      Add new JS tests dir to those pushed to the test server.
      OPFS contention test: give each worker a distinct recognizable name instead of a random one.
      OPFS concurrency tester: ensure that the work interval timer does not overlap with the work time.
      Remove a bit of over-cleverness which breaks loading of sqlite3.js in some main-thread cases.
      Initial infrastructure for adding a mode to the OPFS VFS which causes implicit locks to be released ASAP, which increases concurrency at the cost of performance.
      Add an experimental OPFS VFS-specific URI flag, opfs-unlock-asap, which tells the VFS to release implicit locks ASAP. This permits higher concurrency but hurts performance considerably. This may or may not be obsoleted by other concurrency-related experimentation.
      OPFS concurrency test: add a URL flag to enable/disable unlock-asap mode.
      Add optional zSchema argument to sqlite3_js_db_export().
      Minor JS doc updates.
      Add sqlite3.oo1.DB.prototype.checkRc() and tests for both that method and its class-level counterpart.
      JS documentation cleanups. No code changes.
      More work on the OPFS concurrency testing app.
      Document an OPFS API change in Chrome v108 which does not break our code but does change several formerly async methods to synchronous. No code changes.
      Add an explicit warning about the current API-instability of the sqlite3.opfs namespace, which may need to be eliminated based on re-thinking of how the OPFS sqlite3_vfs is registered. Comment changes only - no code.
      Internal restructuring of the OPFS sqlite3_vfs in order to facilitate certain experimentation and improve error reporting/hints if it cannot be activated. Deprecate the name sqlite3.opfs.OpfsDb, preferring sqlite3.oo1.OpfsDb for consistency with JsStorageDb and any future DB subclasses.
      Minor internal cleanups and docs in the OPFS sqlite3_vfs.
      Rename one instance of opfs.OpfsDb to oo1.OpfsDb, as per [0c5c51f4fb04].
      OPFS speedtest1: hide a currently-broken/to-fix WASMFS-build link.
      sqlite3.oo1.OpfsDb: default to journal_mode=persist, as current tests show it to be marginally faster than truncate/delete in Chrome v109. Also increase default busy_timeout from 2 seconds to 3, admittedly on a whim.
      Doc and logging text tweaks in the OPFS async proxy and test app.
      Refactor a significant chunk of the OPFS sqlite3_vfs init code into sqlite3.VfsHelper, and internal-use-only API encapsulating code relevant to creating new VFSes in JS. Intended to assist in pending experimentation with an alternative OPFS VFS.
      Rename some OPFS JS files. Prevent JS bindings of sqlite3_uri_...() from performing JS-to-C-string argument conversion on their first argument, as doing so is specifically illegal.
      OPFS VFS: remove an invalid TODO and fix a property name typo which caused xCheckReservedLock() to always report false.
      Simplify how the OPFS VFS async proxy copies initial state sent to it from the synchronous side of the connection. Make the lock-wait time a multiple of the wait-loop timeout interval.
      Install sqlite3_malloc/sqlite3_free() as the JS-side WASM allocator (as opposed to replacing C-level's malloc()/free() with them). All tests work and this eliminates the potential for allocator discrepancies when using the (de)serialize APIs.
      Rename some JS files from X.js to X.c-pp.js to keep the maintainer, and downstream build customizers, aware that those files contain constructs specific to the c-pp preprocessor and will not run as-is in JS.
      Expand "sqlite3_vfs*" JS-to-WASM function argument conversions to accept VFS names (JS strings) and capi.sqlite3_vfs instances. Implement sqlite3_js_vfs_create_file() to facilitate creation of file-upload features which store the file in VFS-specific storage (where possible, e.g. "unix" and "opfs" VFSes). Correct an argument type check in the SQLite3Error and WasmAllocError constructors.
      Reformulate and simplify some JS tests related to the previous checkin.
      sqlite3_js_create_file() now accepts an ArrayBuffer data source. Add test for OPFS-based export/re-import. The (sqlite3*) argument converter now optionally accepts sqlite3.oo1.DB instances.
      Remove extraneous/unused sqlite3.oo1.version object. Add httpd makefile target.
      Expand JS tests for db export/import and document reason it cannot currently work with kvvfs. Fix a minor JS build dependencies bug. Update page title with PASS/FAIL prefix for tester1.js to improve overview when launching multiple test tabs. Add ability of tester1 should-run-test predicates to report why a given test is disabled.
      sqlite3-wasm.c: code legibility and coding style tweaks. Increase SQLITE_DEFAULT_PAGE_SIZE from 4k to 8k, as that improves OPFS speedtest1 performance by roughly 12%.
      Roll back the SQLITE_DEFAULT_PAGE_SIZE part of [c260895faacb34] because kvvfs does not work at all with a page size of 8kb.
      Correct the problem which triggered the rollback in [7eec635562f6]: an incorrect default db page size (not a multiple of 512 bytes).
      wasm: after building snapshot zip file, emit instructions for pushing it to the test server.
      wasm builds: explicitly set a default stack size because emsdk 3.1.27 reduced it from 4MB to only 64kb, leading to memory corruption when kvvfs is used (it requires at least twice that for I/O).
      Minor touchups to the JS test index page and test server push rules.
      OPFS VFS: translate createSyncAccessHandle() exceptions which appear to be locking violations to SQLITE_BUSY. This seems to improve concurrency considerably even with a reduced retry count of 5 (was 6).
      Minor internal tweaks to the OPFS VFS. Resolve a missing result code which lead to a null deref in xFileSize().
      Rework the oo1.DB's distinct-per-VFS post-open() step to accept either a batch of SQL or a callback function. Increase OPFS's busy timeout to 10s.
      wasm snapshot and doc-related build automation tweaks.
      wasm build: rename the path to the wasm docs checkout, for clarity and consistency.
      JavaScript: add sqlite3.wasm.realloc(), sqlite3.capi.SQLITE_MAX_ALLOCATION_SIZE, and related tests.
      Move kvvfs xRead()/xWrite() buffer from function-local stack memory to heap memory to accommodate changes in emsdk 3.1.27.
      Doc typo fix for SQLITE_MAX_ALLOCATION_SIZE in malloc.c. No code changes.
      Rename wasm.xWrap.resultAdapter() X:free entries to X:dealloc for consistency with wasm.dealloc(). Add an undocumented feature to replace wasm.alloc/dealloc/realloc() with the C-standard allocators (after an allocator misuse led down a several-hour rabbit hole trying to discover a mis-free() violation). Related test updates.
      sqlite3.wasm.allocFromTypedArray() now optionally accepts an ArrayBuffer as its argument.
      Correct a memory leak in tester1.js.
      Export sqlite3_result_zeroblob/zeroblob64() to wasm.
      Initial infrastructure for adding virtual table/table-valued function support to WASM.
      Remove some dead code. Improve some error checks and comments.
      More work on the JS side of the virtual table APIs.
      Export sqlite3_bind/value/result_pointer() to wasm. Add 'static-string' argument converter to support the lifetime requirements of bind/result_pointer()'s string argument. Correct an endless loop in wasm.cstrlen() when passed a non-C-string argument.
      Rename 'static-string' argument adapter to 'string:static'. Replace JS unit tests which were lost via editing a generated copy of tester1.js instead of the original tester1.c-pp.js input file.
      Remove SQLITE_EXPERIMENTAL tag from sqlite3_vtab_collation() and fix a related doc typo.
      Export sqlite3_vtab_collation() to wasm. Rename 'flexible-string' JS argument adapter to 'string:flexible' for consistency.
      Export collation-related APIs and strncmp()/strnicmp() to wasm.
      Expose sqlite3_get/set_auxdata() to wasm. Minor test app CSS tweaks.
      Jaccwabyt (JS) doc updates.
      Remove two features of jaccwabyt which were fundamentally flawed, along with approx. 250 lines of unit tests which heavily relied on them. Thankfully, none of the sqlite3.js-level code used those bits.
      Add a demonstration sqlite3_vtab/module implemented in JS, based on ext/misc/templatevtab.c. Add oo1.selectArrays() and selectObjects().
      Minor test tweaks.
      Minor internal JS code/docs cleanups.
      Add wasm.cArgvToJs() to support sqlite3_module::xConnect().
      Rename wasm.cstringToJs() to wasm.cstrToJs() for consistency with other wasm.cstr... APIs.
      Remove deprecated symbol sqlite3.opfs.OpfsDb, which was renamed to sqlite3.oo1.OpfsDb on 2022-11-29.
      JS vtables: add infrastructure related to accessing and modifying sqlite3_index_info.
      Add addOnDispose() method to Jaccwabyt and code-adjacent minor internal cleanups.
      Work on an alternate (slightly simpler) approach to binding JS vtabs. Non-eponymous vtabs are not working, for reasons as yet unknown.
      Remove some dead JS code and tweak some docs.
      Got JS non-eponymous vtable working thanks to a hint from Dan.
      More work on the JS vtable tests.
      Describe the SQLITE_DBCONFIG_RESET_DATABASE flag's policy regarding virtual tables, per /chat discussion.
      Ease-of-use/legibility improvements in the virtual table JS helpers.
      Further docs and minor cleanups in the JS virtual table helper.
      Reorganization and renaming in the new VFS/vtab JS pieces.
      Expose sqlite3_db_status() and sqlite3_db_config() to wasm, noting that the latter requires several internal wrappers to account for the various varidic forms (C varargs cannot be bound to wasm).
      Expose sqlite3_table_column_metadata() to wasm.
      JS API doc updates.
      Export sqlite3_status() and sqlite3_stmt_status() to wasm. Expand the arg/return semantics of wasm.get/setPtrValue() and get/setMemValue() to simplify handling of multiple pointers.
      Correct a test bug which broke the previous checkin's wasm tests in higher optimization levels. Test bug - it should not have worked in -O0 mode.
      Micro-optimization in the oft-activated JS-to-WASM arguments conversion step.
      Rename the oft-used, verbose sqlite3.wasm.get/setMemValue() and get/setPtrValue() to peek/poke() and peek/pokePtr(). The old names are retained as aliases just in case any client code actually uses them, but they are now deprecated.
      Remove some unused sqlite3_status() codes from the JS API. Add custom JS wrappers for sqlite3_create_collation/_v2() which accept JS functions (plus tests). Expand the argument options for sqlite3_wasm_db_error() to enable it to translate exception objects to C-level errors.
      Refactor the sqlite3_value-to-JS conversion from an internal detail to sqlite3.capi.sqlite3_value_to_js() for use with routines like sqlite3_module::xFilter().
      Expose sqlite3_value_frombind/nochange/numeric_type() to WASM. Add a flag to sqlite3_value_to_js() to cause it to return undefined, instead of throwing, if no conversion can be found.
      Export sqlite3_result_subtype() and sqlite3_value_dup/free/subtype() to WASM.
      Run configure.ac through autoupdate, as required by autoconf for rebuilding configure. These (many) changes were made entirely by the autotools, except that some whitespace-only changes were reverted by hand to reduce noise, and are a preliminary step to adding a new flag to configure.ac.
      Add --with-wasi-sdk=DIR flag to configure.ac. This mode compiles but fails to link the DLL because libtool is apparently hard-coded to gcc for the DLL.
      Imply configure flag --disable-shared when --with-wasi-sdk is active because libtool is running gcc for linking the shared lib, which cannot work in a wasi build.
      Refactor the internal JS routines for converting UDF results and errors to JS into public APIs.
      Expose sqlite3_column_value() to WASM and add sqlite3_column_js().
      Remove two incorrect calls to structType.dipose() which prematurely freed objects in use by the virtual table test/demo code.
      Correct wasm heap corruption introduced in test code added in [e144fd5c88fa4] which led to unpredictable failures.
      Remove -flto from wasm build flags, as it increases build time and doesn't seem to have a measurable benefit.
      Minor build tweaks in ext/wasm to create a faster-running build for the common dev-mode cases. No code changes.
      Expose a number of infrequently-used sqlite3_...() functions to wasm whose absences were noticed while documenting.
      Expose sqlite3_txn_state() to wasm.
      Add sqlite3.mjs to the new 'quick' wasm build for the sake of the snapshot build.
      Rework the internals of how argument/result types are converted to/from wasm in prep for attempting to support automated conversions of function pointer argument types.
      ext/wasm/module-symbols.html: add a hyperlink to the API docs for each exported sqlite3_...() function.
      ext/wasm/module-symbols.html: for C APIs which have extended JS-side semantics, link to the JS-side API docs instead of the C docs.
      Move JS-to-C binding signatures from sqlite3-api-prologue.js to sqlite3-api-glue.js to allow for use of the new/experimental sqlite3.wasm.xWrap() feature which automatically binds JS functions to WASM/C as needed, which simplifies creation of bindings which take C function pointers. Reimplement sqlite3_exec(), sqlite3_create_collation(), sqlite3_progress_handler() to use this new feature.
      Revert part of [9386d6f63468] because the new automatic function pointer binding cannot properly track per-context function mappings when the context is more complex than a single context-type pointer. e.g. it is fine for sqlite3_trace_v2() but it breaks down with sqlite3_create_collation() because that one needs to use the collation name as part of the context key and we cannot sensibly do so with the current code.
      Extend the sqlite3.wasm function pointer argument converter to be able to handle the "two-layered context" of sqlite3_create_collation() and friends and make use of FuncPtrAdapter to perform JS-to-WASM function conversion for them.
      Document the role of the sqlite3.client JS API property.
      A micro-optimization in sqlite3.oo1.DB.exec(). Changed the rowMode option of that method to only accept $X instead of $X, @X, and :X, as the extra options are entirely superfluous and may lead to confusion.
      Add a test to demonstrate/verify which object acts as "this" in a oo1.DB.exec() callback.
      Add convenience variants of sqlite3.wasm.peek/poke() for each numeric type to help reduce errors related to typos in the final argument (type-name strings). If wasm.xWrap.FuncPtrAdapter is called as a function, instead of a constructor, it now behaves as if it were called as a constructor (previously it threw an exception).
      Use the new peek/poke() variants throughout tester1.js.
      Slight reformulation of some test code to test multi-arg wasm.peek() call form.
      Rename the new peek/pokeF32() and peek/pokeF64() to peek/poke32f() and peek/poke64f() for consistency with related APIs and because they're easier on both the eyes and the fingers.
      Remove an unnecessary/obsolete Emscripten-specific export.
      Internal refactoring of how sqlite3.wasm.xWrap() handles JS-to-C function pointer conversions, to enable similar conversions to be added more easily.
      Added a couple missing 'not part of the public API' tags on functions in sqlite3-wasm.c. No code changes.
      Add sqlite3_set_authorizer() support and related tests to JS.
      Add a test confirming that exceptions are not passed through C-space if a JS-side sqlite3_set_authorizer() callback throws.
      Expose a JS-friendly subset of sqlite3_config() to JS, with the notable caveats that (1) setting up the JS bindings requires starting the library, making sqlite3_config() illegal to call and (2) calling sqlite3_shutdown() in order to make it legal to call sqlite3_config() may undo certain JS-side library initialization. Move sqlite3_(de)serialize() into the int64-mode-only bindings because of their int64 args.
      Update wasmfs.make to get WASMFS building again, but changes made to OPFS-over-WASMFS since we last tested it have made it incompatible with how we used it. It can now only be used from worker threads, eliminating the one benefit it had over the sqlite3_vfs OPFS implementation. Remove/amend references to WASMFS in the docs and remove all WASMFS-specific test app links from index.html.
      JS namespace updates in ext/wasm/api/README.md.
      Update test-opfs-vfs.js to account for recent API changes. Reported in [forum:4a97813fcbd4f63e|forum post 4a97813fcbd4f63e].
      Minor sed script tweak in the wasm GNUmakefile for BSD portability, as reported in [forum:4a97813fcbd4f63e|forum post 4a97813fcbd4f63e].
      sessionfuzz.c: use lt/gt brackets instead of double-quotes when #including zlib.h, as reported in [forum:91a104bd65 | forum post 91a104bd65].
      Squelch a new (and, in this case, harmless) compiler warning.
      Initial pieces for binding the session API to JS. Far from complete. See [forum:210e36a1e3 | forum post 210e36a1e3] for the discussion.
      Add sqlite3.capi JS bindings for the sqlite3session_...(), sqlite3changeset_...() and sqlite3changegroup_...() APIs, noting that they are completely untested. Aside from missing tests, these bindings reveal a slight string-argument-type shortcoming in the callback function pointer "reverse binding" which should ideally be resolved before publishing them.
      Internal cleanups and minor speed optimizations in the sqlite3.wasm.xWrap() infrastructure.
      Cherry-pick [c4dab53b8ea3401abd] for sqlite3.wasm.xWrap() optimizations.
      Add SQLITE_ENABLE_MATH_FUNCTIONS to the list of feature flags in sqlite3-wasm.c.
      Explicitly omit threading and extension loading from the ext/wasm/c-pp binary build to avoid a link error on some systems.
      Consolidate/unify how the JS bindings of the create_function/collation family of functions react to a non-UTF8 encoding: they now treat a falsy value as SQLITE_UTF8 and fail with SQLITE_FORMAT for an invalid encoding.
      Internal JS cleanups. Correct part of [ac136925a645] to account for the eTextRep flag being able to hold flags other than the encoding.
      Reimplement JS's sqlite3_bind_text/blob() with hand-written bindings to permit more flexible inputs. Add automated JS-to-C function conversion to sqlite3_busy_handler(). sqlite3.wasm.xWrap()'s '*' argument conversion no longer treats JS strings as C-strings: those conversions require explicit opt-in via the 'string' converter (or equivalent).
      If sqlite3.oo1.DB.exec()'s callback returns a literal false, stop step()ing over results as if the end of the result set had been reached. Unrelated minor code-adjacent cleanups.
      oo1.DB.exec() rowMode="$columnName": a minor optimization and a preemtive fix for a hypothetical corner-case bug.
      Add a test for the (failure) case of client-level code calling the oo1.Stmt constructor directly.
      Extend oo1.Stmt.bind() to accept ArrayBuffer instances to bind as blobs.
      Replace JS-side use of SQLITE_TRANSIENT with the new SQLITE_WASM_DEALLOC, reducing the amount allocation/copying required by sqlite3_bind_blob/text() and sqlite3_result_blob/text(). Remove the 'experimental' log message from the virtual table tests.
      Incorporate wasi-sdk RANLIB configure script patch from the VMware OCTO team.
      Simplify the signature for JS functions, as opposed to function pointers, passed to sqlite3_exec(), eliminating the superfluous initial two arguments. Update related tests to demonstrate both function-passing approaches.
      Enhance sqlite3.wasm.xWrap.FuncPtrAdapter to be able to handle sqlite3_create_function() and friends and reimplement those bindings to use this feature (this will also simplify certain session API bindings). Interal API changes only with no client-side breakage.
      Add missing sqlite3_context_db_handle() JS binding. Reimplement sqlite3_set_authorizer() and sqlite3_set_auxdata() JS bindings to take advantage of [7f9ace1b11a67]. Teach FuncPtrAdapter to emit a console.warn() message if it is invoked after the library is bootstrapped, the goal being to inform users that it's an internal API and should not be invoked from client-side code.
      Update the session-related JS bindings to account for today's internal API changes.
      Add the address of the associated db handle to the sqlite3_trace_v2() output originating from sqlite3.oo1.DB's trace flag.
      Remove duplicated JS binding of sqlite3changeset_apply_v2().
      Replace the "manual" implementation of sqlite3.capi.sqlite3_exec() with a briefer "automated" one via the [7f9ace1b11a67] feature addition. Minor code-adjacent internal cleanups.
      Add tests confirming that JS sqlite3_exec()'s SQL argument participates in the 'string:flexible' type conversion.
      Add sqlite3.wasm.irSizeof() and extend certain allocation functions to make use of it.
      Add basic session API JS tests.
      Add sqlite3.oo1.DB.selectValues(). Correct a logic error which could cause DB.selectValue(), DB.selectArray(), and DB.selectObject() to fail to finalize a statement if a call to bind() failed. Add more session API tests.
      Minor internal API doc correction and extend a test for oo1.DB.selectValues().
      Rename the new wasm.irSizeof() to sizeofIR() because that seems clearer.
      Update JS worker #1 API and related tests for recent API changes.
      module-symbols.html: add some missing links to C-side API docs for recently-added JS counterparts.
      Reimplement sqlite3.capi.sqlite3_close_v2() and sqlite3session_delete() as a hand-written bindings so that they can attempt to clean up certain (potentially) FuncPtrAdapter-installed functions before closing. Correct the create-function family of JS-to-function-pointer automated conversions to include the UDF's arity as part of the mapping's key so that (un)binding a UDF to different functions for different arities works (and add tests confirming it). Correct a broken doc link in module-symbols.html.
      Add JS infrastructure which enables sqlite3.capi.sqlite3_close_v2() to clean up stale JS-to-WASM collation function conversions installed on behalf of a given db handle. The same for UDF mappings is TODO.
      Extend [0e69b2c379e618] to support uninstalling stale JS-to-WASM function pointers added on behalf of UDFs.
      Only add an on-db-close cleanup entry for collations if adding the collation succeeds and xCompare is-a JS function.
      Correct some internal-use JS docs and update the 'string:flexible' type conversion to accept an ArrayBuffer (as it was recently documented to).
      Document sqlite3.capi.sqlite3_prepare_v3() as accepting an ArrayBuffer and ensure that it can.
      Expose the auto-extension API to JS and reorganize some nearby code.
      Add an optional argument to oo1.DB.transaction() to specify an explicit BEGIN qualifier.
      Expose sqlite3_commit/rollback/update_hook() to JS API.
      Add two more tests to [f99f8e3ecfe20].
      Add links to module-symbols.html for newly-added APIs.
      Expose sqlite3_preupdate_hook() and friends to the JS API.
      Add sqlite3changeset_new/old_js(), which work like sqlite3_preupdate_new/old_js() but on changesets.
      An alternative solution to mapping SQLITE_WASM_DEALLOC to the proper function pointer in JS, to account for a Safari-specific quirk reported in [forum:e5b20e1feb|forum post e5b20e1feb].
      Another reformulation of SQLITE_WASM_DEALLOC to attempt to work around a Safari-specific quirk reported in [forum:5489305f601b8c3f|forum post 5489305f601b8c3f].
      Add some docs explaining a particular piece of [ae0196d86ee8]. No code changes.
      Remove the JS-side SQLITE_WASM_DEALLOC sanity check which triggers the problem mentioned in [688c5c13d156] and [ae0196d86ee8], for reasons covered in the code comments, per discussion in [forum:e5b20e1feb|forum post e5b20e1feb].
      Elaborate on the open-in-read-only fallback behavior of the SQLITE_OPEN_READWRITE flag, per user request. Unrelated trailing EOL whitespace cleanups.
      Emit sqlite3-api.(m)js during the JS build process, which are the JS APIs without the Emscripten/wasm-loading parts. They are hypothetically useful for arbitrary build environments/toolchains but have notable caveats related to the wasm imports, as elaborated on in the makefile.
      Minor API doc cleanups and JS code simplification.
      End-of-line whitespace cleanups and doc typo fixes. No code changes.
      Beginnings of a bundler-friendly build of sqlite3.mjs. Not yet ready for downstream testing.
      Resolve a nested if-block bug in ext/wasm/c-pp.c which caused output after a nested block to be unduly elided. Remove a kludge, added in the previous check-in, which worked around that bug.
      More work on creating a separate sqlite3.js build which is hopefully friendly to JS bundlers.
      Work around upstream emscripten 3.1.31 bug [https://github.com/emscripten-core/emscripten/issues/18609].
      Cherrypick [fa784101775b7|emscripten ticket #18609 workaround] into trunk.
      Work around a JS null pointer deref which could be triggered from the dev console, but not (it seems) from client-side code.
      Extract emcc version in JS build and use it to conditionally set build flags. Initially a workaround for [https://github.com/emscripten-core/emscripten/issues/18610|Emscripten ticket #18610] but may have other uses.
      Cherrypick [3773934e91c20ca243] into trunk.
      Add a feature idea note to DB.exec(), derived from a forum discussion.
      Update ext/wasm/README-dist.txt for the bundler-friendly build.
      Enhance oo1.DB.exec() to simplify returning whole result sets.
      Overhaul ext/wasm/GNUmakefile to consolidate what amounts to much copy/paste/slightly-edit duplication into a single function, called once per distinctive build mode (vanilla, ESM, bundler-friendly).
      Makefile doc touchups - no code/build changes.
      Correct the handling of the worker1 and promiser JS files in the face of the bundler-friendly changes. Those files require separate, bundler-friendly copies.
      Two JS file renames which got inadvertently undone while setting up [9062b31174618c0e]. Cosmetic cleanups in ext/wasm/dist.make.
      Remove automatic installation of JS-global S object for the sake of client libraries which embed this library, per [forum:9d4f722c6912799d|request in the forum].
      Add capability to override the JS's use of console.log/debug/warn/error() with client-provided versions via the bootstrap-time config object.
      Roll back part of [c54f29d8] which attempted to use symbols which that worker does not have access to.
      Squelch two harmless signedness comparison warnings in shell.c.in.
      Fix ext/wasm/fiddle build, which was silently broken by recent build refactoring.
      Minor text-only updates to wasm demo/test HTML and license header.
sbc100 added a commit that referenced this issue Mar 19, 2023
sbc100 added a commit that referenced this issue Mar 20, 2023
@sbc100
Copy link
Collaborator

sbc100 commented Mar 31, 2023

Fixed in #19006

@sbc100 sbc100 closed this as completed Mar 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants