`-safe-string` compilation for Batteries #673

gasche · 2016-04-20T13:26:27Z

In preparation for the upcoming 3.0 release, I have been working on switching Batteries to be usable with -safe-string. While the change is not compatibility-breaking for people that do not use -safe-string in their libraries yet (we are only replacing some instances of string by Bytes.t in the interfaces; and sometimes adding another function that works on the other type), it would break compatibility for people using Batteries and -safe-string together.

I would of course appreciate any review or comment, on the interface choices in particular.

The rest of this message discusses a detail point, String.Cap.

This module uses a phantom type discipline to distinguish read-only, write-only and read-write string, but the phantomness of it relies on defining all of these as string. I could try to turn this into a GADT that would hold either a string or a bytes depending on the intended usage, but this would change the interface slightly, as users would have to declare at string creation time what the usage mode is going to be (whereas this is inferred fully automatically). Also it would introduce some minimal amount of packing, against the current claim that using Cap has no performance impact whatsoever.

What I have done for now is to disable String.Cap module altogether. One could agree that it is less relevant in presence of the string/bytes distinction, and could be dropped from 3.0. I'm not convinced; I think that the users that were careful enough to use Cap in the past (we know that for example @jpdeplaix uses Exceptionless, which is similar in intended audience) would want to be able to write code polymorphic over the writeability of the string, and string/bytes does not allow this -- they would still feel a need for the more expressive interface.

An option we would have is to not have String.Cap in 3.0, and loudly advertise that we plan to add it back in future versions if users have a need for it and contact us about it. In particular, having a user at hand could help clarifying how the interface should evolve to meet their needs.

c-cube · 2016-04-20T13:38:17Z

I didn't look at the code, but couldn't String.Cap use bytes where it used to use string?

UnixJunkie · 2016-04-20T13:41:43Z

Not completely on the point, but to me it is fine to break compatibility
between major releases. Of course, if you break everything every-time there
is a major release, users will not stay.

gasche · 2016-04-20T13:44:17Z

@c-cube Yes, that's also an option. I think it's better than trying to dive into GADTs head-first.

@UnixJunkie indeed, given that we decided to break compatibility in the next release, it is fine to break compatibility. We should still understand and document well what we break and why. But on the question of -safe-string in particular I'm not worried about backward-compatibility (except maybe for the Cap thing). Today it is morally wrong to rely on the types of Batteries when using -safe-string anyway, as Batteries may mutate string parameters; so users of -safe-string will be glad that we break compatibility, as it makes their code safer. (And disabling -safe-string gives an easy way to remain backward-compatible with their pre-change codebases.)

kit-ty-kate · 2016-04-20T14:30:19Z

@gasche for the record, I don't use Batteries anymore. I replaced it with @c-cube's Containers.
But yes, apart for that, I have used the Exceptionless module but not the Cap module tho.

gasche · 2016-04-21T23:56:56Z

@c-cube I started working on String.Cap but it's still not-so-convincing and a lot of work for two reasons:

The only correct types you can give for going in and out of the cap interface are _ t -> bytes, bytes -> _ t, and string -> [<Read] t. Because of theread_only : _ t -> [Read] t function, there is no safe way to return a string from a Cap.t, which means that the to_string function will always have a counter-intuitive type -- really, it should be deprecated.
Most of the Cap implementations are not implemented for Bytes. I could hack my way through wrapping each operation in conversion function, but the "right" thing to do is to define a BatInnerByteString module that define operations for both datatypes and shares as much code as possible (plus, if all the code is internal to a single module it is safer to reason about ownership and use unsafe conversions), but that is a fair amount of work.

I'm hesitating between just having String.Cap.{of,to}_string just manipulate Bytes.t, and use conversion functions all over the place (the easiest option, still some work, kind of ugly for end-users), and just removing the module altogether as I initially proposed.

kit-ty-kate · 2017-09-11T14:34:12Z

Any news on this ?

gasche · 2017-09-11T14:59:20Z

Well I think there would be nothing wrong with rebasing the current PR and merging it (thanks for the ping!), but a bit of work remains to fix the -safe-string build.

gasche · 2017-09-23T11:20:57Z

I just rebased this PR on top of the current master. I hope to merge in the following week(s), and have a release before 4.06.0+beta and the release, to allow users to test their own packages.

gasche · 2017-10-06T13:05:14Z

Merged! Apologies for the delay. We have to prepare a release now to be 4.06-ready.

disable print_string_cap_ro/rw in battop to match #673

@gasche

* Add Changelog for #763 (BatVect.Labels addition) * Fix an implementation/specification mismatch in BatVect.insert `BatVect.insert : int -> 'a t -> 'a t` is specified so that `insert n r u` inserts the rope `r` between the elements of index `n` and `n+1` in `u`. Its implementation does something different, it inserts `r` strictly before the element of index `n` in `u`. The present commit changes the specification to match the implementation. First, we expect users to have tested their program instead of trusting (or, unfortunately, even reading) the documentation, so it is likely that the uses of BatVect.insert with the current implementation is correct for their needs; changing it would break user program. Second, we argue that the implemented behavior is actually far more natural: - the implementation (in terms of `sub` and `concat`) is nicer -- hear the code! - the ranges of integer values valid for `insert` is `0 .. (length u - 1)`, as one would expect, instead of `(-1) .. (length u - 2)` as with the current specification. - this gives a very natural invariant between `insert` and `BatVect.remove : (*start*)int -> (*len*)int -> 'a t -> 'a t`: `u` is equal to `u |> insert n r |> remove n (length r))`. fixes #766. * remove exceptions in BatString.starts_with * remove exceptions in BatString.ends_with * remove exceptions in BatString.find_from * remove exceptions in BatString.rfind_from * fix bug in batSubstring.equal * remove exceptions in BatSubstring.equal * change entry for the String, Substring changes * Changelog: add review information for #768 * remove exceptions in BatVect.{exists,for_all} * remove exceptions in BatVect.find Implementing BatVect.find_opt makes this more efficient (we have at most one allocation for the result, instead of using exception handlers for each left subtree), but the function is not exposed publicly. * BatVect: expose find_opt and harmonize variable names in .mli section * remove exceptions in BatVect.Make.{for_all,exists} * remove exceptions in BatVect.Make.find * tests for BatVect.Make functions * expose BatVect.Make.find_opt * less exceptions in batVect: changelog entry * Makefile: introduce a QTEST_SEED variable to set qtest random seed To reproduce a failure that may depend on the seed 1234, you may use make test QTEST_SEED=1345 * Fix some issues in the README * Fix compilation with OCaml 4.05.0 * Fixes based on @gasche's comments * Add support for new APIs * Update for compatibility with qcheck 0.6 Unfortunately this does break backwards compatibility with qcheck 0.5, so there is now an explicit opam dependency on qcheck >= 0.6. Fixes #756. * Simplify dependency restriction on qtest * batText: add tests and fix a bug in `rindex{,_from}` * batText: add tests for contains_from and fix documentation bug (exception name) * batText: add tests to rcontains_from and fix bug in passing * battext changes: ChangeLog entry * batNum: handle negatives in of_float_string * ChangeLog entry on BatNum.of_float_string * BatNum.of_float_string: extra testcase * Backport new 4.05 functions (#779) * Backport the new option-returning *_opt functions from 4.05.0 * Bigarray.Array0 cannot be backported from 4.05 (it depends on a runtime change) * batList: backport compare_length{s,_with} to <4.5 versions * Printexc.raise_with_backtrace cannot be backported from 4.05 This is a new primitive from the runtime. * Change entry for #777, #779 * batBytes: minor compatibility fix * release summary (#782) * add a "num" dependency to mirror upstream opam file (#783) See ocaml/opam-repository#9341 Note that, if/when "num" is split off the standard distribution, there will be the question of keeping it or not in Batteries. It could be come a separate package (batteries-num, depending on batteries) for example. * next release will be 2.7.0 * simplification to the opam file so that it is easier to compare with the one in the opam repository * added note explaining that BatX is also Batteries.X * substitued the for a * added explanation for case when Batteries is opened * added example illustrating the BatX -> Batteries.X mapping. For an example, I think it's better to use module that's not listed in the "Extensions to the Standard Library" section. That section has its own specialized message about opening Batteries. Using a module from this section as an illustration has a slight risk of suggesting to users that the BatX->Batteries.X point applies only to this section. * safe-string: make Lexing safe * safe-string: make Big_int safe * safe-string: make Buffer safe * safe-string: make Digest safe * safe-string: make Genlex safe * safe-string: make IO safe * safe-string: make Format safe * safe-string: make Int32 safe * safe-string: make Marshal safe * safe-string: make Pervasives safe * safe-string: make Printf safe * safe-string: make String safe (but String.Cap) * safe-string: adapt String.Cap * safe-string: make Unix safe * safe-string: make Base64 safe * safe-string: make BitSet safe * safe-string: make Substring safe * safe-string: make Text safe * _tags: actually enforce safe_string * safe-string: make BatBytes safe * -safe-string: Changelog * [minor] exclude compattest.ml from qtest (Otherwise you cannot run the tests for an OCaml version that does not pass the compat-test.) * add 4.06-only Array.Floatarray primitive module * add 4.06-only Uchar.t functions in Buffer It would be nicer to make sure that BatUChar is a superset of the stdlib Uchar module, and that the two types are compatible, but that requires more work. * 4.06 Bigarray functions * 4.06 Unix functions * 4.06: Changes entry * fix more FloatArray failures in the test targets * added BatString.count_char * added count_string * safe-string: fix compatibility with OCaml <=4.02 The safe-string compatibility fixes rely on the Bytes package, which we convincingly mock under OCaml<=4.02, but also on bytes-adapted functions or just convenience functions that are useful for code porting: Buffer.add_subbytes, String.init, etc. Before this commit, the codebase would simply not build on OCaml<=4.02 because of this. What the commit does is to introduce a new module, BatBytesCompat, meant to contain backward-compatibility implementation of those functions. * a prebaked patch to try our inline tests on older OCaml versions The following command should get Batteries (temporarily) testable with old qtest versions: git apply build/make-our-inline-tests-compatible-with-older-OCaml-versions.patch * mention count_string * Revert "added count_string" This reverts commit 1171462. The idea is to do the next minor release without the count_* functions that are not yet ready for inclusion. The commit should be merged back right after the release. * missing @SInCE tags * update scripts/replace_since.sh * refresh @SInCE NEXT_RELEASE tags with new version * bump _oasis version number * Update setup.ml based on _oasis * Revert "mention count_string" This reverts commit 0ae9a35. This has to be pushed again after v2.8.0, in the right Changelog section * batString: safe-string-ready map_first_char * first commit after v2.8.0 * Revert "Revert "added count_string"" This reverts commit 10e2dd1. * Revert "Revert "mention count_string"" This reverts commit bc10ba1. * move count_string's ChangeLog entry into the new section * BatInt.Safe_int.mul performance improvements The fast-path was made less expensive, by getting rid of absolute value computation (which incurs branching). The downside is that the fast-path supports only non-negative integers from now on. * Add ChangeLog entry * disable print_string_cap_ro/rw to match #673 * Fix: incorrect behaviour of Map.union In case of binding conflicts, [Map.union m1 m2] should prefer the value from [m2], as stated in documentation. * Add a test case for Map.union * Add ChangeLog entry * added BatSeq.{to_buffer|to_string|of_string} * Cosmetic tweaks in BatMap.mli * Document that Map.extract can raise Not_found. (#825) * added BatArray.split * updated ChangeLog * Clarified documentation for `next` in LazyList (#810) * Revised docs for seq, unfold, from_loop in LazyList (#806) seq: Revised basic description to fit first part of from_loop doc, which seems clearer. Also added note that you can create an infinite list by passing a function that constantly returns true. from_loop: Gave additional detail about what the next function is supposed to do, and noted that its result pair might contain different types. Added a simple example where the types are the same. (This example could be done more simpy using seq. I still don't understand the purpose of from_loop.) I moved seq before from_loop because it's a more basic version of this kind of operation. from_loop is more specialized, and complicated, but its name might suggest that it's the normal way of making a lazy list by repeated application of a function. Users should see seq first before trying to work out what from_loop is for. * improve MultiPMap documentation (#827) * simpler (and faster?) implementation of BatArray.partition (#829) * simpler and faster implementation of BatArray.partition (at the cost of using more memory) * updated changelog * removed previously added exception types * updated ocamldoc for BatSeq.of_string * Compile Batteries with -no-alias-deps The size cost of using batteries has always been a problem. OCaml 4.02 introduced --no-deps-libs for this exact purpose (see http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#s%3Amodule-alias). I couldn't find any discussion relative to this but I'm well aware this is unlikely to not have been considered already. Simple test: ``` % cat toto1.ml open Batteries let () = Printf.printf "hello\n" % cat toto2.ml let () = BatPrintf.printf "hello\n" % for f in toto1 toto2 ; do \ ocamlfind ocamlopt $f.ml -o $f.opt -package batteries -linkpkg ; done % ls -l toto*opt ``` Before the patch: ``` -rwxr-xr-x 1 rixed 5272428 Feb 7 05:43 toto1.opt* -rwxr-xr-x 1 rixed 2006588 Feb 7 05:43 toto2.opt* ``` After: ``` -rwxr-xr-x 1 rixed 3098452 Feb 7 05:38 toto1.opt* -rwxr-xr-x 1 rixed 2006588 Feb 7 05:38 toto2.opt* ``` That's a 40% reduction in size of a program using only BatPrintf from Batteries, still bigger than a program using explicitly only BatPrintf because of batPervasives, I assume. There are several ways i can imagine this patch would break other programs: - it might break the build of some programs relying on some of their direct dependencies to link in others of their direct dependencies, which I would qualify as a bug in those builds; - this could prevent the dynamic linkage of some plugins if it expects the whole of batteries to be available from the main program; - programs relying on any side effects of some otherwise unused batteries module initialization code, but that sounds very unlikely. Note that I'm not familiar with this option so there might be others, more obvious issues. I'm confident you will know :-) Note: I have no idea if/how the "pack" option that seems to be present in the setup.ml could be used for the same purpose. * Fix: Map.update was broken Searching for the node to update was broken, as the keys were compared in wrong order. * Fix: Map.update did not throw an exception on failure The docstring says "@raise Not_found if [k1] is not bound in [m]", but there was no raise if [k1] and [k2] were different. * Fix: Set.update did not throw an exception on failure The docstring says "@raise Not_found if [x] is not in [s]", but there was no raise if [x] and [y] were different. * Fix typo * Simplify Set.cartesian_product * Remove a redundant exception from BatOption.ml * Simplify Text.concat * Fix typos (#838) * Fix: Array.insert error message improvement The function now throws a more relevant message on invalid indices instead of the generic [invalid_arg "index out of bounds]. The assertion is now documented. * Implementation of Array.insert now uses unsafe_{get,set} * Do not depend upon bisect any longer Rational: bisect is not very actively maintained any more and code coverage have not been useful (or is not useful any more). * Corrected documentation for batList {last, hd} (#840) * Fix docstring of String.right (#849) * Fix docstring of String.right * BatInt.Safe_int.mul performance improvements The previous overflow test from CERT C is notable for avoiding an overflow while checking for overflow; hovewer, since overflows are perfectly defined in OCaml, a shorter test from Hacker's Delight is a better option. * Update ChangeLog * A clean-up of invalid_arg uses This patch makes uses of `invalid_arg` more uniform, fixing the following kinds of irregularities: * Lack of module name in a message (e.g. `Enum.arg_min` threw "arg_min"); * Lack of function name in a message (e.g. `List.reduce` threw "Empty list"); * Wrong module name in a message (e.g. `PathGen` functions threw messages with "Path"); * Wrong function name in a message (e.g. `List.fold_left2` threw a message with "fold_right2"); * Minor style issues: capitalization, punctuation, "Bat" prefixes; * Use of `raise (Invalid_argument "...")` instead of `invalid_arg "..."`. * Add ChangeLog entry * Restructuring of Heap.del_min Utility functions [find_min_tree] and [del_min_tree] are extended with an additional argument so as to make it more obvious to the reader that error messages are correct. * Optimization of List.unique_cmp A set should be more efficient than [bool PMap.t]. * Add ChangeLog entry * Fix docstrings of List.append and List.concat Stack usage estimates are invalid. * BatString.ml should use unsafe versions of set and get * batString: annotate unsafe_{get,set} with static reasoning in comments (partial) * BatString.ml: annotate unsafe_{get,set} with static assertions * Remove mention of IO.flush_out that does not exist. * List.mli: remove erroneous mentions of Different_list_size * List.ml: clean-up invalid_arg messages * Add ChangeLog entry * Add String.cut (#856) * Add `String.index_after_n : char -> int -> string -> int` * Add ChangeLog entry for `String.index_after_n` * fix index_after_n doc * Add String.cut * Update documentation for String.{index_after_n,cut} * Remove spurious empty line before BatString.index_after_n * Add Changelog entry for String.cut * Update #856 Changelog with PR number * Clarify documentation of String.{index_after_n,cut} * Rename String.cut to cut_on_char * better exception string in BatString.index_after_n * better ocamldoc for BatHashtbl.find_default (#859) * better ocamldoc for BatHashtbl.find_default so much better with an example * better ocamldoc for find_default * Rename cut to cut_on_char in Changelog * Fix build on OCaml < 4.05 * Fix a regression in BatInt.Safe_int.mul * Make doc of String.index_from a bit clearer (#864) * Map.equal used (=) instead of argument * Changelog entry for #865 * Adding ocamlfind plugin support (#867) * changelog update for PR 867 * Update batVect.mli Typo in description of sub. Ordering r m n required to be consistent with function definition * batInnerWeaktbl: separate interface from Hashtbl for compatibility * 4.07 compatibility fixes * testsuite: remove test_interface.ml, subsumed by 'make test-compat' * beef up Travis checking * prepare release 2.9.0 * Update setup.ml based on _oasis * prepare the 2.9.0 release * Update setup.ml based on _oasis * first commit after 2.9.0 * extend the Changelog (#877) * Fix Dllist.skip skip had the same behavior no matter the sign of the parameter * batNum: fix the testsuite * fix a batNum.of_float_sting bug on inputs in ]-1; 0[. (fixes #886) * added BatList.fold_while (#889) * added BatList.fold_while * Fixing basic typos (#890) * Gc: add new 4.08 fields * Array: reorder type and include to avoid error * re-export Bigarray.*.map_file functions removed in 4.08 * Int32: incorrect primitive names * Lexing: no support for disabling position-tracking * Unix: we don't support the new (link ?follow) parameter on older OCamls * add CI testing for 4.08 * Corrected BatDeque.is_empty documentation. * Hashtbl merge (#892) * Addition of Hashtbl.merge and merge_all Closes #891 * BatString.count_char uses a while loop instead of fold * Current module name seems Extlib, not Extlibcompat (#873) https://github.com/ocaml-batteries-team/batteries-included/blob/master/src/extlib.ml * Add lazylist equal predicate (#811) added LazyList.equal * updated ChangeLog * get rid of >=4.02 syntax in BatHashtbl.merge * BatHashtbl: avoid |> when possible * some more files need preprocessing Hashtbl.statistics is from 4.00.0 and was used in batInnerWeaktbl.ml{i} * BatArray: 4.07 compat * BatBuffer: 4.07 and 4.08 compat * BatBytes: 4.07 and 4.08 compat * BatInt32: 4.08 compat * BatInt64: 4.08 compat * BatLexing: 4.08 compat * BatList: 4.08 compat * BatNativeint: 4.08 compat * BatPrintexc: 4.08 compat * BatQueue: 4.07 and 4.08 compat Note to other maintainers: I am not sure take_opt and peek_opt should be introduced by preprocessing. Maybe we want them added to all OCaml versions? * BatString: 4.08 compat * BatSys: 4.08 compat * BatUnix: 4.08 compat * BatString: to_seq{i} and of_seq are from OCaml 4.07, not 4.08 also added NEXT_RELEASE tag for them * added missing NEXT_RELEASE tags in many places * next release will be 2.10.0 * Update setup.ml based on _oasis * updated opam file with bits from the one in opam-repository * repaired upload-docs target in Makefile and make it more readable too * start of next release * Fix compilation by BER MetaOCaml (#909) Set.Infix used to have operators `<.` and `>.` (for strict subset and strict superset), but `>.` is also used as a code block end-quote by BER. It's been decided to jettison the whole BatSet.Infix module, as a simple and future proof fix. Ref #908 * Add BatFilename (#910) Also added `Filename.split_extension` Closes #445 * s/RELASE/RELEASE (#911) typos * Improve toplevel printability of predefined sets (#912) Redefining the modules (instead of using those predefined in BatSet) was causing the printers for BatSet.Int and friends useless. Closes #750 * Remove BatOo (#915) As it is undocumented, unsupported and unsafe. Closes #848 * Make BatSeq.Exceptionless.combine really exceptionless (#917) Now, BatSeq.Exceptionless.combine returns the longest sequence possible (stopping when one of the input sequence ends). Closes #418 * BatString.split_on_char and nsplit must never return an empty list (#918) BatString.split_on_char behavior matches the one of Legacy.String.split_on_char, and that nsplit behavior matches the one of split_on_char. Closes #845 Closes #846 * Add Legacy.Result for OCaml >= 4.8.0 (#913) Now, users can pattern match Result.t returned from some library using Legacy.Result.Error while Batteries is opened. Ref #839 * List.sum [] is now 0 (#916) List.sum and List.fsum default to 0 (or 0.0) in case the list is empty. Previously, this was raising Invalid_argument. Closes #519

@gasche

* Changelog: add review information for #768 * remove exceptions in BatVect.{exists,for_all} * remove exceptions in BatVect.find Implementing BatVect.find_opt makes this more efficient (we have at most one allocation for the result, instead of using exception handlers for each left subtree), but the function is not exposed publicly. * BatVect: expose find_opt and harmonize variable names in .mli section * remove exceptions in BatVect.Make.{for_all,exists} * remove exceptions in BatVect.Make.find * tests for BatVect.Make functions * expose BatVect.Make.find_opt * less exceptions in batVect: changelog entry * Makefile: introduce a QTEST_SEED variable to set qtest random seed To reproduce a failure that may depend on the seed 1234, you may use make test QTEST_SEED=1345 * Fix some issues in the README * Fix compilation with OCaml 4.05.0 * Fixes based on @gasche's comments * Add support for new APIs * Update for compatibility with qcheck 0.6 Unfortunately this does break backwards compatibility with qcheck 0.5, so there is now an explicit opam dependency on qcheck >= 0.6. Fixes #756. * Simplify dependency restriction on qtest * batText: add tests and fix a bug in `rindex{,_from}` * batText: add tests for contains_from and fix documentation bug (exception name) * batText: add tests to rcontains_from and fix bug in passing * battext changes: ChangeLog entry * batNum: handle negatives in of_float_string * ChangeLog entry on BatNum.of_float_string * BatNum.of_float_string: extra testcase * Backport new 4.05 functions (#779) * Backport the new option-returning *_opt functions from 4.05.0 * Bigarray.Array0 cannot be backported from 4.05 (it depends on a runtime change) * batList: backport compare_length{s,_with} to <4.5 versions * Printexc.raise_with_backtrace cannot be backported from 4.05 This is a new primitive from the runtime. * Change entry for #777, #779 * batBytes: minor compatibility fix * release summary (#782) * add a "num" dependency to mirror upstream opam file (#783) See ocaml/opam-repository#9341 Note that, if/when "num" is split off the standard distribution, there will be the question of keeping it or not in Batteries. It could be come a separate package (batteries-num, depending on batteries) for example. * next release will be 2.7.0 * simplification to the opam file so that it is easier to compare with the one in the opam repository * added note explaining that BatX is also Batteries.X * substitued the for a * added explanation for case when Batteries is opened * added example illustrating the BatX -> Batteries.X mapping. For an example, I think it's better to use module that's not listed in the "Extensions to the Standard Library" section. That section has its own specialized message about opening Batteries. Using a module from this section as an illustration has a slight risk of suggesting to users that the BatX->Batteries.X point applies only to this section. * safe-string: make Lexing safe * safe-string: make Big_int safe * safe-string: make Buffer safe * safe-string: make Digest safe * safe-string: make Genlex safe * safe-string: make IO safe * safe-string: make Format safe * safe-string: make Int32 safe * safe-string: make Marshal safe * safe-string: make Pervasives safe * safe-string: make Printf safe * safe-string: make String safe (but String.Cap) * safe-string: adapt String.Cap * safe-string: make Unix safe * safe-string: make Base64 safe * safe-string: make BitSet safe * safe-string: make Substring safe * safe-string: make Text safe * _tags: actually enforce safe_string * safe-string: make BatBytes safe * -safe-string: Changelog * [minor] exclude compattest.ml from qtest (Otherwise you cannot run the tests for an OCaml version that does not pass the compat-test.) * add 4.06-only Array.Floatarray primitive module * add 4.06-only Uchar.t functions in Buffer It would be nicer to make sure that BatUChar is a superset of the stdlib Uchar module, and that the two types are compatible, but that requires more work. * 4.06 Bigarray functions * 4.06 Unix functions * 4.06: Changes entry * fix more FloatArray failures in the test targets * added BatString.count_char * added count_string * safe-string: fix compatibility with OCaml <=4.02 The safe-string compatibility fixes rely on the Bytes package, which we convincingly mock under OCaml<=4.02, but also on bytes-adapted functions or just convenience functions that are useful for code porting: Buffer.add_subbytes, String.init, etc. Before this commit, the codebase would simply not build on OCaml<=4.02 because of this. What the commit does is to introduce a new module, BatBytesCompat, meant to contain backward-compatibility implementation of those functions. * a prebaked patch to try our inline tests on older OCaml versions The following command should get Batteries (temporarily) testable with old qtest versions: git apply build/make-our-inline-tests-compatible-with-older-OCaml-versions.patch * mention count_string * Revert "added count_string" This reverts commit 1171462. The idea is to do the next minor release without the count_* functions that are not yet ready for inclusion. The commit should be merged back right after the release. * missing @SInCE tags * update scripts/replace_since.sh * refresh @SInCE NEXT_RELEASE tags with new version * bump _oasis version number * Update setup.ml based on _oasis * Revert "mention count_string" This reverts commit 0ae9a35. This has to be pushed again after v2.8.0, in the right Changelog section * batString: safe-string-ready map_first_char * first commit after v2.8.0 * Revert "Revert "added count_string"" This reverts commit 10e2dd1. * Revert "Revert "mention count_string"" This reverts commit bc10ba1. * move count_string's ChangeLog entry into the new section * BatInt.Safe_int.mul performance improvements The fast-path was made less expensive, by getting rid of absolute value computation (which incurs branching). The downside is that the fast-path supports only non-negative integers from now on. * Add ChangeLog entry * disable print_string_cap_ro/rw to match #673 * Fix: incorrect behaviour of Map.union In case of binding conflicts, [Map.union m1 m2] should prefer the value from [m2], as stated in documentation. * Add a test case for Map.union * Add ChangeLog entry * added BatSeq.{to_buffer|to_string|of_string} * Cosmetic tweaks in BatMap.mli * Document that Map.extract can raise Not_found. (#825) * added BatArray.split * updated ChangeLog * Clarified documentation for `next` in LazyList (#810) * Revised docs for seq, unfold, from_loop in LazyList (#806) seq: Revised basic description to fit first part of from_loop doc, which seems clearer. Also added note that you can create an infinite list by passing a function that constantly returns true. from_loop: Gave additional detail about what the next function is supposed to do, and noted that its result pair might contain different types. Added a simple example where the types are the same. (This example could be done more simpy using seq. I still don't understand the purpose of from_loop.) I moved seq before from_loop because it's a more basic version of this kind of operation. from_loop is more specialized, and complicated, but its name might suggest that it's the normal way of making a lazy list by repeated application of a function. Users should see seq first before trying to work out what from_loop is for. * improve MultiPMap documentation (#827) * simpler (and faster?) implementation of BatArray.partition (#829) * simpler and faster implementation of BatArray.partition (at the cost of using more memory) * updated changelog * removed previously added exception types * updated ocamldoc for BatSeq.of_string * Compile Batteries with -no-alias-deps The size cost of using batteries has always been a problem. OCaml 4.02 introduced --no-deps-libs for this exact purpose (see http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#s%3Amodule-alias). I couldn't find any discussion relative to this but I'm well aware this is unlikely to not have been considered already. Simple test: ``` % cat toto1.ml open Batteries let () = Printf.printf "hello\n" % cat toto2.ml let () = BatPrintf.printf "hello\n" % for f in toto1 toto2 ; do \ ocamlfind ocamlopt $f.ml -o $f.opt -package batteries -linkpkg ; done % ls -l toto*opt ``` Before the patch: ``` -rwxr-xr-x 1 rixed 5272428 Feb 7 05:43 toto1.opt* -rwxr-xr-x 1 rixed 2006588 Feb 7 05:43 toto2.opt* ``` After: ``` -rwxr-xr-x 1 rixed 3098452 Feb 7 05:38 toto1.opt* -rwxr-xr-x 1 rixed 2006588 Feb 7 05:38 toto2.opt* ``` That's a 40% reduction in size of a program using only BatPrintf from Batteries, still bigger than a program using explicitly only BatPrintf because of batPervasives, I assume. There are several ways i can imagine this patch would break other programs: - it might break the build of some programs relying on some of their direct dependencies to link in others of their direct dependencies, which I would qualify as a bug in those builds; - this could prevent the dynamic linkage of some plugins if it expects the whole of batteries to be available from the main program; - programs relying on any side effects of some otherwise unused batteries module initialization code, but that sounds very unlikely. Note that I'm not familiar with this option so there might be others, more obvious issues. I'm confident you will know :-) Note: I have no idea if/how the "pack" option that seems to be present in the setup.ml could be used for the same purpose. * Fix: Map.update was broken Searching for the node to update was broken, as the keys were compared in wrong order. * Fix: Map.update did not throw an exception on failure The docstring says "@raise Not_found if [k1] is not bound in [m]", but there was no raise if [k1] and [k2] were different. * Fix: Set.update did not throw an exception on failure The docstring says "@raise Not_found if [x] is not in [s]", but there was no raise if [x] and [y] were different. * Fix typo * Simplify Set.cartesian_product * Remove a redundant exception from BatOption.ml * Simplify Text.concat * Fix typos (#838) * Fix: Array.insert error message improvement The function now throws a more relevant message on invalid indices instead of the generic [invalid_arg "index out of bounds]. The assertion is now documented. * Implementation of Array.insert now uses unsafe_{get,set} * Do not depend upon bisect any longer Rational: bisect is not very actively maintained any more and code coverage have not been useful (or is not useful any more). * Corrected documentation for batList {last, hd} (#840) * Fix docstring of String.right (#849) * Fix docstring of String.right * BatInt.Safe_int.mul performance improvements The previous overflow test from CERT C is notable for avoiding an overflow while checking for overflow; hovewer, since overflows are perfectly defined in OCaml, a shorter test from Hacker's Delight is a better option. * Update ChangeLog * A clean-up of invalid_arg uses This patch makes uses of `invalid_arg` more uniform, fixing the following kinds of irregularities: * Lack of module name in a message (e.g. `Enum.arg_min` threw "arg_min"); * Lack of function name in a message (e.g. `List.reduce` threw "Empty list"); * Wrong module name in a message (e.g. `PathGen` functions threw messages with "Path"); * Wrong function name in a message (e.g. `List.fold_left2` threw a message with "fold_right2"); * Minor style issues: capitalization, punctuation, "Bat" prefixes; * Use of `raise (Invalid_argument "...")` instead of `invalid_arg "..."`. * Add ChangeLog entry * Restructuring of Heap.del_min Utility functions [find_min_tree] and [del_min_tree] are extended with an additional argument so as to make it more obvious to the reader that error messages are correct. * Optimization of List.unique_cmp A set should be more efficient than [bool PMap.t]. * Add ChangeLog entry * Fix docstrings of List.append and List.concat Stack usage estimates are invalid. * BatString.ml should use unsafe versions of set and get * batString: annotate unsafe_{get,set} with static reasoning in comments (partial) * BatString.ml: annotate unsafe_{get,set} with static assertions * Remove mention of IO.flush_out that does not exist. * List.mli: remove erroneous mentions of Different_list_size * List.ml: clean-up invalid_arg messages * Add ChangeLog entry * Add String.cut (#856) * Add `String.index_after_n : char -> int -> string -> int` * Add ChangeLog entry for `String.index_after_n` * fix index_after_n doc * Add String.cut * Update documentation for String.{index_after_n,cut} * Remove spurious empty line before BatString.index_after_n * Add Changelog entry for String.cut * Update #856 Changelog with PR number * Clarify documentation of String.{index_after_n,cut} * Rename String.cut to cut_on_char * better exception string in BatString.index_after_n * better ocamldoc for BatHashtbl.find_default (#859) * better ocamldoc for BatHashtbl.find_default so much better with an example * better ocamldoc for find_default * Rename cut to cut_on_char in Changelog * Fix build on OCaml < 4.05 * Fix a regression in BatInt.Safe_int.mul * Make doc of String.index_from a bit clearer (#864) * Map.equal used (=) instead of argument * Changelog entry for #865 * Adding ocamlfind plugin support (#867) * changelog update for PR 867 * Update batVect.mli Typo in description of sub. Ordering r m n required to be consistent with function definition * batInnerWeaktbl: separate interface from Hashtbl for compatibility * 4.07 compatibility fixes * testsuite: remove test_interface.ml, subsumed by 'make test-compat' * beef up Travis checking * prepare release 2.9.0 * Update setup.ml based on _oasis * prepare the 2.9.0 release * Update setup.ml based on _oasis * first commit after 2.9.0 * extend the Changelog (#877) * Fix Dllist.skip skip had the same behavior no matter the sign of the parameter * batNum: fix the testsuite * fix a batNum.of_float_sting bug on inputs in ]-1; 0[. (fixes #886) * added BatList.fold_while (#889) * added BatList.fold_while * Fixing basic typos (#890) * Gc: add new 4.08 fields * Array: reorder type and include to avoid error * re-export Bigarray.*.map_file functions removed in 4.08 * Int32: incorrect primitive names * Lexing: no support for disabling position-tracking * Unix: we don't support the new (link ?follow) parameter on older OCamls * add CI testing for 4.08 * Corrected BatDeque.is_empty documentation. * Hashtbl merge (#892) * Addition of Hashtbl.merge and merge_all Closes #891 * BatString.count_char uses a while loop instead of fold * Current module name seems Extlib, not Extlibcompat (#873) https://github.com/ocaml-batteries-team/batteries-included/blob/master/src/extlib.ml * Add lazylist equal predicate (#811) added LazyList.equal * updated ChangeLog * get rid of >=4.02 syntax in BatHashtbl.merge * BatHashtbl: avoid |> when possible * some more files need preprocessing Hashtbl.statistics is from 4.00.0 and was used in batInnerWeaktbl.ml{i} * BatArray: 4.07 compat * BatBuffer: 4.07 and 4.08 compat * BatBytes: 4.07 and 4.08 compat * BatInt32: 4.08 compat * BatInt64: 4.08 compat * BatLexing: 4.08 compat * BatList: 4.08 compat * BatNativeint: 4.08 compat * BatPrintexc: 4.08 compat * BatQueue: 4.07 and 4.08 compat Note to other maintainers: I am not sure take_opt and peek_opt should be introduced by preprocessing. Maybe we want them added to all OCaml versions? * BatString: 4.08 compat * BatSys: 4.08 compat * BatUnix: 4.08 compat * BatString: to_seq{i} and of_seq are from OCaml 4.07, not 4.08 also added NEXT_RELEASE tag for them * added missing NEXT_RELEASE tags in many places * next release will be 2.10.0 * Update setup.ml based on _oasis * updated opam file with bits from the one in opam-repository * repaired upload-docs target in Makefile and make it more readable too * start of next release * Fix compilation by BER MetaOCaml (#909) Set.Infix used to have operators `<.` and `>.` (for strict subset and strict superset), but `>.` is also used as a code block end-quote by BER. It's been decided to jettison the whole BatSet.Infix module, as a simple and future proof fix. Ref #908 * Add BatFilename (#910) Also added `Filename.split_extension` Closes #445 * s/RELASE/RELEASE (#911) typos * Improve toplevel printability of predefined sets (#912) Redefining the modules (instead of using those predefined in BatSet) was causing the printers for BatSet.Int and friends useless. Closes #750 * Remove BatOo (#915) As it is undocumented, unsupported and unsafe. Closes #848 * Make BatSeq.Exceptionless.combine really exceptionless (#917) Now, BatSeq.Exceptionless.combine returns the longest sequence possible (stopping when one of the input sequence ends). Closes #418 * BatString.split_on_char and nsplit must never return an empty list (#918) BatString.split_on_char behavior matches the one of Legacy.String.split_on_char, and that nsplit behavior matches the one of split_on_char. Closes #845 Closes #846 * Add Legacy.Result for OCaml >= 4.8.0 (#913) Now, users can pattern match Result.t returned from some library using Legacy.Result.Error while Batteries is opened. Ref #839 * List.sum [] is now 0 (#916) List.sum and List.fsum default to 0 (or 0.0) in case the list is empty. Previously, this was raising Invalid_argument. Closes #519 * Remove Legacy.Sort for OCaml >= 4.08 (#926) Closes #925 * Build with -strict-sequence (#927) * with_locked_file (#904) * high level file locking file interface * with_file_lock: now call user function with the opened file * Release v2.11.0 (#931) * Fix batFilename for various old OCaml versions split_extension was relying on remove_extension and extension functions that were introduced in OCaml 4.04.0. Also, an ocaml.deprecated annotation needed to be protected by some preprocessing. Tested with most of supported compiler versions. * Prepare Changelog and NEXT_RELEASE placeholder for v2.11.0 * Update setup.ml based on _oasis Closes #928 * make Unix.sleepf available under all OCaml versions (#930) * make Unix.sleepf available across all OCaml versions * the '|>' operator was not available in old versions of the stdlib (e.g. OCaml 4.0.0) * BatBytes was missing unsafe_blit_string's signature to compile on 4.09.0 * provide signatures of new functions from 4.09.0 in BatPrintexc * BatSys: change in argv declaration for ocaml >= 4.09.0 * updated opam file to reflect more the one from opam-repository * bits for next release Co-authored-by: Gabriel Scherer <gabriel.scherer@gmail.com> Co-authored-by: Tej Chajed <tchajed@mit.edu> Co-authored-by: Anton Yabchinskiy <arn@bestmx.net> Co-authored-by: Max Mouratov <mmouratov@gmail.com> Co-authored-by: Milo <iitalics@gmail.com> Co-authored-by: Thibault Suzanne <thi.suzanne@gmail.com> Co-authored-by: Marshall Abrams <mars0i@users.noreply.github.com> Co-authored-by: sapristi <megabigfoutoir@gmail.com> Co-authored-by: FkHina <36575050+FkHina@users.noreply.github.com> Co-authored-by: Ralf Vogler <ralf.vogler@gmail.com> Co-authored-by: Arlen Cox <arlencox@gmail.com> Co-authored-by: Francois Berenger <unixjunkie@sdf.org> Co-authored-by: yaseersheriff <yaseer.sheriff@gmail.com> Co-authored-by: Virgile Robles <virgile.robles@protonmail.ch> Co-authored-by: Fourchaux <jprodi04@gmail.com> Co-authored-by: Zachary Palmer <zepalmer@users.noreply.github.com> Co-authored-by: Catalin Hritcu <catalin.hritcu@gmail.com> Co-authored-by: Armaël Guéneau <Armael@users.noreply.github.com>

gasche added the for next major release label Apr 20, 2016

gasche mentioned this pull request Apr 20, 2016

move to -safe-string #656

Closed

gasche force-pushed the safe-string branch from 9caa5d3 to b2b2e24 Compare May 8, 2016 23:14

gasche force-pushed the master branch from 72b0741 to ed8dc65 Compare May 12, 2016 00:00

gasche force-pushed the master branch from 453817c to f36f4ff Compare October 28, 2016 19:03

gasche force-pushed the master branch from f36f4ff to f33b6e2 Compare December 6, 2016 14:39

gasche added 12 commits September 23, 2017 12:59

safe-string: make Lexing safe

8327cf0

safe-string: make Big_int safe

44a3a21

safe-string: make Buffer safe

180f9b9

safe-string: make Digest safe

d64443a

safe-string: make Genlex safe

8cbb5d5

safe-string: make IO safe

d12b5f1

safe-string: make Format safe

310259b

safe-string: make Int32 safe

6153fee

safe-string: make Marshal safe

1a9b980

safe-string: make Pervasives safe

4bbf690

safe-string: make Printf safe

1c5f96a

safe-string: make String safe (but String.Cap)

aed6515

gasche force-pushed the safe-string branch from b2b2e24 to 523dafc Compare September 23, 2017 11:17

gasche force-pushed the safe-string branch from 523dafc to f788c75 Compare September 23, 2017 11:25

gasche added 2 commits September 23, 2017 13:29

safe-string: adapt String.Cap

dd4dbb4

safe-string: make Unix safe

bcb538c

gasche added 7 commits September 23, 2017 13:29

safe-string: make Base64 safe

5d67375

safe-string: make BitSet safe

4b50dcc

safe-string: make Substring safe

b5cd62c

safe-string: make Text safe

7bc0550

_tags: actually enforce safe_string

ea80aa7

safe-string: make BatBytes safe

f788c75

-safe-string: Changelog

2b67824

gasche merged commit ce92e34 into ocaml-batteries-team:master Oct 6, 2017

gasche mentioned this pull request Oct 13, 2017

support for upcoming 4.06 release #797

Merged

gasche mentioned this pull request Nov 14, 2017

batteries 2.8.0 ocaml/opam-repository#10690

Merged

iitalics added a commit to iitalics/batteries-included that referenced this pull request Dec 2, 2017

disable print_string_cap_ro/rw to match ocaml-batteries-team#673

e86cc0f

gasche added a commit that referenced this pull request Dec 3, 2017

Merge pull request #813 from iitalics/battop_string_cap

af61097

disable print_string_cap_ro/rw in battop to match #673

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`-safe-string` compilation for Batteries #673

`-safe-string` compilation for Batteries #673

gasche commented Apr 20, 2016

c-cube commented Apr 20, 2016

UnixJunkie commented Apr 20, 2016

gasche commented Apr 20, 2016

kit-ty-kate commented Apr 20, 2016 •

edited

Loading

gasche commented Apr 21, 2016 •

edited

Loading

kit-ty-kate commented Sep 11, 2017

gasche commented Sep 11, 2017 •

edited

Loading

gasche commented Sep 23, 2017

gasche commented Oct 6, 2017

-safe-string compilation for Batteries #673

-safe-string compilation for Batteries #673

Conversation

gasche commented Apr 20, 2016

c-cube commented Apr 20, 2016

UnixJunkie commented Apr 20, 2016

gasche commented Apr 20, 2016

kit-ty-kate commented Apr 20, 2016 • edited Loading

gasche commented Apr 21, 2016 • edited Loading

kit-ty-kate commented Sep 11, 2017

gasche commented Sep 11, 2017 • edited Loading

gasche commented Sep 23, 2017

gasche commented Oct 6, 2017

`-safe-string` compilation for Batteries #673

`-safe-string` compilation for Batteries #673

kit-ty-kate commented Apr 20, 2016 •

edited

Loading

gasche commented Apr 21, 2016 •

edited

Loading

gasche commented Sep 11, 2017 •

edited

Loading