From a2dddd818c51df56e42ca2d79d3388ec756c2309 Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 19 Oct 2020 13:49:38 +0200
Subject: [PATCH 001/442] Initial commit

---
 LICENSE   | 373 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 README.md |   1 +
 2 files changed, 374 insertions(+)
 create mode 100644 LICENSE
 create mode 100644 README.md

diff --git a/LICENSE b/LICENSE
new file mode 100644
index 000000000..a612ad981
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,373 @@
+Mozilla Public License Version 2.0
+==================================
+
+1. Definitions
+--------------
+
+1.1. "Contributor"
+    means each individual or legal entity that creates, contributes to
+    the creation of, or owns Covered Software.
+
+1.2. "Contributor Version"
+    means the combination of the Contributions of others (if any) used
+    by a Contributor and that particular Contributor's Contribution.
+
+1.3. "Contribution"
+    means Covered Software of a particular Contributor.
+
+1.4. "Covered Software"
+    means Source Code Form to which the initial Contributor has attached
+    the notice in Exhibit A, the Executable Form of such Source Code
+    Form, and Modifications of such Source Code Form, in each case
+    including portions thereof.
+
+1.5. "Incompatible With Secondary Licenses"
+    means
+
+    (a) that the initial Contributor has attached the notice described
+        in Exhibit B to the Covered Software; or
+
+    (b) that the Covered Software was made available under the terms of
+        version 1.1 or earlier of the License, but not also under the
+        terms of a Secondary License.
+
+1.6. "Executable Form"
+    means any form of the work other than Source Code Form.
+
+1.7. "Larger Work"
+    means a work that combines Covered Software with other material, in
+    a separate file or files, that is not Covered Software.
+
+1.8. "License"
+    means this document.
+
+1.9. "Licensable"
+    means having the right to grant, to the maximum extent possible,
+    whether at the time of the initial grant or subsequently, any and
+    all of the rights conveyed by this License.
+
+1.10. "Modifications"
+    means any of the following:
+
+    (a) any file in Source Code Form that results from an addition to,
+        deletion from, or modification of the contents of Covered
+        Software; or
+
+    (b) any new file in Source Code Form that contains any Covered
+        Software.
+
+1.11. "Patent Claims" of a Contributor
+    means any patent claim(s), including without limitation, method,
+    process, and apparatus claims, in any patent Licensable by such
+    Contributor that would be infringed, but for the grant of the
+    License, by the making, using, selling, offering for sale, having
+    made, import, or transfer of either its Contributions or its
+    Contributor Version.
+
+1.12. "Secondary License"
+    means either the GNU General Public License, Version 2.0, the GNU
+    Lesser General Public License, Version 2.1, the GNU Affero General
+    Public License, Version 3.0, or any later versions of those
+    licenses.
+
+1.13. "Source Code Form"
+    means the form of the work preferred for making modifications.
+
+1.14. "You" (or "Your")
+    means an individual or a legal entity exercising rights under this
+    License. For legal entities, "You" includes any entity that
+    controls, is controlled by, or is under common control with You. For
+    purposes of this definition, "control" means (a) the power, direct
+    or indirect, to cause the direction or management of such entity,
+    whether by contract or otherwise, or (b) ownership of more than
+    fifty percent (50%) of the outstanding shares or beneficial
+    ownership of such entity.
+
+2. License Grants and Conditions
+--------------------------------
+
+2.1. Grants
+
+Each Contributor hereby grants You a world-wide, royalty-free,
+non-exclusive license:
+
+(a) under intellectual property rights (other than patent or trademark)
+    Licensable by such Contributor to use, reproduce, make available,
+    modify, display, perform, distribute, and otherwise exploit its
+    Contributions, either on an unmodified basis, with Modifications, or
+    as part of a Larger Work; and
+
+(b) under Patent Claims of such Contributor to make, use, sell, offer
+    for sale, have made, import, and otherwise transfer either its
+    Contributions or its Contributor Version.
+
+2.2. Effective Date
+
+The licenses granted in Section 2.1 with respect to any Contribution
+become effective for each Contribution on the date the Contributor first
+distributes such Contribution.
+
+2.3. Limitations on Grant Scope
+
+The licenses granted in this Section 2 are the only rights granted under
+this License. No additional rights or licenses will be implied from the
+distribution or licensing of Covered Software under this License.
+Notwithstanding Section 2.1(b) above, no patent license is granted by a
+Contributor:
+
+(a) for any code that a Contributor has removed from Covered Software;
+    or
+
+(b) for infringements caused by: (i) Your and any other third party's
+    modifications of Covered Software, or (ii) the combination of its
+    Contributions with other software (except as part of its Contributor
+    Version); or
+
+(c) under Patent Claims infringed by Covered Software in the absence of
+    its Contributions.
+
+This License does not grant any rights in the trademarks, service marks,
+or logos of any Contributor (except as may be necessary to comply with
+the notice requirements in Section 3.4).
+
+2.4. Subsequent Licenses
+
+No Contributor makes additional grants as a result of Your choice to
+distribute the Covered Software under a subsequent version of this
+License (see Section 10.2) or under the terms of a Secondary License (if
+permitted under the terms of Section 3.3).
+
+2.5. Representation
+
+Each Contributor represents that the Contributor believes its
+Contributions are its original creation(s) or it has sufficient rights
+to grant the rights to its Contributions conveyed by this License.
+
+2.6. Fair Use
+
+This License is not intended to limit any rights You have under
+applicable copyright doctrines of fair use, fair dealing, or other
+equivalents.
+
+2.7. Conditions
+
+Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
+in Section 2.1.
+
+3. Responsibilities
+-------------------
+
+3.1. Distribution of Source Form
+
+All distribution of Covered Software in Source Code Form, including any
+Modifications that You create or to which You contribute, must be under
+the terms of this License. You must inform recipients that the Source
+Code Form of the Covered Software is governed by the terms of this
+License, and how they can obtain a copy of this License. You may not
+attempt to alter or restrict the recipients' rights in the Source Code
+Form.
+
+3.2. Distribution of Executable Form
+
+If You distribute Covered Software in Executable Form then:
+
+(a) such Covered Software must also be made available in Source Code
+    Form, as described in Section 3.1, and You must inform recipients of
+    the Executable Form how they can obtain a copy of such Source Code
+    Form by reasonable means in a timely manner, at a charge no more
+    than the cost of distribution to the recipient; and
+
+(b) You may distribute such Executable Form under the terms of this
+    License, or sublicense it under different terms, provided that the
+    license for the Executable Form does not attempt to limit or alter
+    the recipients' rights in the Source Code Form under this License.
+
+3.3. Distribution of a Larger Work
+
+You may create and distribute a Larger Work under terms of Your choice,
+provided that You also comply with the requirements of this License for
+the Covered Software. If the Larger Work is a combination of Covered
+Software with a work governed by one or more Secondary Licenses, and the
+Covered Software is not Incompatible With Secondary Licenses, this
+License permits You to additionally distribute such Covered Software
+under the terms of such Secondary License(s), so that the recipient of
+the Larger Work may, at their option, further distribute the Covered
+Software under the terms of either this License or such Secondary
+License(s).
+
+3.4. Notices
+
+You may not remove or alter the substance of any license notices
+(including copyright notices, patent notices, disclaimers of warranty,
+or limitations of liability) contained within the Source Code Form of
+the Covered Software, except that You may alter any license notices to
+the extent required to remedy known factual inaccuracies.
+
+3.5. Application of Additional Terms
+
+You may choose to offer, and to charge a fee for, warranty, support,
+indemnity or liability obligations to one or more recipients of Covered
+Software. However, You may do so only on Your own behalf, and not on
+behalf of any Contributor. You must make it absolutely clear that any
+such warranty, support, indemnity, or liability obligation is offered by
+You alone, and You hereby agree to indemnify every Contributor for any
+liability incurred by such Contributor as a result of warranty, support,
+indemnity or liability terms You offer. You may include additional
+disclaimers of warranty and limitations of liability specific to any
+jurisdiction.
+
+4. Inability to Comply Due to Statute or Regulation
+---------------------------------------------------
+
+If it is impossible for You to comply with any of the terms of this
+License with respect to some or all of the Covered Software due to
+statute, judicial order, or regulation then You must: (a) comply with
+the terms of this License to the maximum extent possible; and (b)
+describe the limitations and the code they affect. Such description must
+be placed in a text file included with all distributions of the Covered
+Software under this License. Except to the extent prohibited by statute
+or regulation, such description must be sufficiently detailed for a
+recipient of ordinary skill to be able to understand it.
+
+5. Termination
+--------------
+
+5.1. The rights granted under this License will terminate automatically
+if You fail to comply with any of its terms. However, if You become
+compliant, then the rights granted under this License from a particular
+Contributor are reinstated (a) provisionally, unless and until such
+Contributor explicitly and finally terminates Your grants, and (b) on an
+ongoing basis, if such Contributor fails to notify You of the
+non-compliance by some reasonable means prior to 60 days after You have
+come back into compliance. Moreover, Your grants from a particular
+Contributor are reinstated on an ongoing basis if such Contributor
+notifies You of the non-compliance by some reasonable means, this is the
+first time You have received notice of non-compliance with this License
+from such Contributor, and You become compliant prior to 30 days after
+Your receipt of the notice.
+
+5.2. If You initiate litigation against any entity by asserting a patent
+infringement claim (excluding declaratory judgment actions,
+counter-claims, and cross-claims) alleging that a Contributor Version
+directly or indirectly infringes any patent, then the rights granted to
+You by any and all Contributors for the Covered Software under Section
+2.1 of this License shall terminate.
+
+5.3. In the event of termination under Sections 5.1 or 5.2 above, all
+end user license agreements (excluding distributors and resellers) which
+have been validly granted by You or Your distributors under this License
+prior to termination shall survive termination.
+
+************************************************************************
+*                                                                      *
+*  6. Disclaimer of Warranty                                           *
+*  -------------------------                                           *
+*                                                                      *
+*  Covered Software is provided under this License on an "as is"       *
+*  basis, without warranty of any kind, either expressed, implied, or  *
+*  statutory, including, without limitation, warranties that the       *
+*  Covered Software is free of defects, merchantable, fit for a        *
+*  particular purpose or non-infringing. The entire risk as to the     *
+*  quality and performance of the Covered Software is with You.        *
+*  Should any Covered Software prove defective in any respect, You     *
+*  (not any Contributor) assume the cost of any necessary servicing,   *
+*  repair, or correction. This disclaimer of warranty constitutes an   *
+*  essential part of this License. No use of any Covered Software is   *
+*  authorized under this License except under this disclaimer.         *
+*                                                                      *
+************************************************************************
+
+************************************************************************
+*                                                                      *
+*  7. Limitation of Liability                                          *
+*  --------------------------                                          *
+*                                                                      *
+*  Under no circumstances and under no legal theory, whether tort      *
+*  (including negligence), contract, or otherwise, shall any           *
+*  Contributor, or anyone who distributes Covered Software as          *
+*  permitted above, be liable to You for any direct, indirect,         *
+*  special, incidental, or consequential damages of any character      *
+*  including, without limitation, damages for lost profits, loss of    *
+*  goodwill, work stoppage, computer failure or malfunction, or any    *
+*  and all other commercial damages or losses, even if such party      *
+*  shall have been informed of the possibility of such damages. This   *
+*  limitation of liability shall not apply to liability for death or   *
+*  personal injury resulting from such party's negligence to the       *
+*  extent applicable law prohibits such limitation. Some               *
+*  jurisdictions do not allow the exclusion or limitation of           *
+*  incidental or consequential damages, so this exclusion and          *
+*  limitation may not apply to You.                                    *
+*                                                                      *
+************************************************************************
+
+8. Litigation
+-------------
+
+Any litigation relating to this License may be brought only in the
+courts of a jurisdiction where the defendant maintains its principal
+place of business and such litigation shall be governed by laws of that
+jurisdiction, without reference to its conflict-of-law provisions.
+Nothing in this Section shall prevent a party's ability to bring
+cross-claims or counter-claims.
+
+9. Miscellaneous
+----------------
+
+This License represents the complete agreement concerning the subject
+matter hereof. If any provision of this License is held to be
+unenforceable, such provision shall be reformed only to the extent
+necessary to make it enforceable. Any law or regulation which provides
+that the language of a contract shall be construed against the drafter
+shall not be used to construe this License against a Contributor.
+
+10. Versions of the License
+---------------------------
+
+10.1. New Versions
+
+Mozilla Foundation is the license steward. Except as provided in Section
+10.3, no one other than the license steward has the right to modify or
+publish new versions of this License. Each version will be given a
+distinguishing version number.
+
+10.2. Effect of New Versions
+
+You may distribute the Covered Software under the terms of the version
+of the License under which You originally received the Covered Software,
+or under the terms of any subsequent version published by the license
+steward.
+
+10.3. Modified Versions
+
+If you create software not governed by this License, and you want to
+create a new license for such software, you may create and use a
+modified version of this License if you rename the license and remove
+any references to the name of the license steward (except to note that
+such modified license differs from this License).
+
+10.4. Distributing Source Code Form that is Incompatible With Secondary
+Licenses
+
+If You choose to distribute Source Code Form that is Incompatible With
+Secondary Licenses under the terms of this version of the License, the
+notice described in Exhibit B of this License must be attached.
+
+Exhibit A - Source Code Form License Notice
+-------------------------------------------
+
+  This Source Code Form is subject to the terms of the Mozilla Public
+  License, v. 2.0. If a copy of the MPL was not distributed with this
+  file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+If it is not possible or desirable to put the notice in a particular
+file, then You may include the notice in a location (such as a LICENSE
+file in a relevant directory) where a recipient would be likely to look
+for such a notice.
+
+You may add additional accurate notices of copyright ownership.
+
+Exhibit B - "Incompatible With Secondary Licenses" Notice
+---------------------------------------------------------
+
+  This Source Code Form is "Incompatible With Secondary Licenses", as
+  defined by the Mozilla Public License, v. 2.0.
diff --git a/README.md b/README.md
new file mode 100644
index 000000000..254f2d790
--- /dev/null
+++ b/README.md
@@ -0,0 +1 @@
+# bergamot-translator
\ No newline at end of file

From ef2323c9520c8517f23399e373441044cf11787c Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 29 Oct 2020 09:17:32 +0100
Subject: [PATCH 002/442] Unified api draft (#1)

* Changed README file

 - Added a short introduction of this repository
 - More updates to come later

* First draft of the unified API
---
 README.md          |   4 +-
 doc/Unified_API.md | 212 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 215 insertions(+), 1 deletion(-)
 create mode 100644 doc/Unified_API.md

diff --git a/README.md b/README.md
index 254f2d790..dd3798232 100644
--- a/README.md
+++ b/README.md
@@ -1 +1,3 @@
-# bergamot-translator
\ No newline at end of file
+# Bergamot Translator
+
+Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.github.io/) framework based) neural machine translation functionality in accordance with the [Bergamot](https://browser.mt/) project that focuses on improving client-side machine translation in a web browser.
diff --git a/doc/Unified_API.md b/doc/Unified_API.md
new file mode 100644
index 000000000..e6a14301b
--- /dev/null
+++ b/doc/Unified_API.md
@@ -0,0 +1,212 @@
+# Unified (C++) API of Bergamot Translator
+
+/* A Translation model interface for translating a plain utf-8 encoded text (without any markups and emojis). The model supports translation from 1 source language to 1 target language. There can be different implementations of this interface. */
+
+class **AbstractTranslationModel** {
+
+    public:
+
+	AbstractTranslationModel();
+
+	virtual ~AbstractTranslationModel() {};
+
+	/* This method performs translation on a list of (utf-8) texts and returns a list of results in the same order. Each text entry can either be a word, a phrase, a sentence or a list of sentences and should contain plain text (without any markups or emojis). Additional information related to the translated text can be requested via TranslationRequest which is applied equally to each text entry. The translated text corresponding to each text entry and the additional information (as specified in the TranslationRequest) is encapsulated and returned in TranslationResult.
+	The API splits each text entry into sentences internally, which are then translated independent of each other. The translated sentences are then joined together and returned in TranslationResult.
+	Please refer to the TranslationRequest class to find out what additional information can be requested. The alignment information can only be requested if the model supports it (check isAlignmentSupported() API).
+	*/
+	virtual std::vector<std::future<TranslationResult>> translate(std::vector<std::string> texts, TranslationRequest request) = 0;
+
+	/* Check if the model can provide alignment information b/w original and translated text. */
+	virtual bool isAlignmentSupported() const = 0;
+}
+
+/* This class specifies the additional information related to the translated text (e.g. quality of the translation etc.) that can be requested to be included in the TranslationResult. These optional requests are set/unset independent of each other i.e. setting any one of them doesn’t have the side effect of setting any of the others. */
+
+class **TranslationRequest** {
+
+    private:
+
+	// Optional request. The granularity for which Quality scores of the translated text will be included in TranslationResult. By default (QualityScoreGranularity::NONE), scores are not included.
+	QualityScoreGranularity qualityScore = QualityScoreGranularity::NONE;
+
+	// Optional request. The type of the alignment b/w original and translated text that will be included in TranslationResult. By default (AlignmentType::NONE), alignment is not included.
+	AlignmentType alignmentType = AlignmentType::NONE;
+
+	// Optional request. A true/false value will include/exclude the original text in the TranslationResult. By default (false), the original text is not included.
+	bool includeOriginalText = false;
+
+	// Optional request. A true/false value will include/exclude the information regarding how individual sentences of original text map to corresponding translated sentences in joined translated text in the TranslationResult. By default (false), this information is not included.
+	bool includeSentenceMapping = false;
+
+    public:
+
+	explicit TranslationRequest();
+
+	~TranslationRequest();
+
+	/* Set the granularity for which the Quality scores of translated text should be included in the TranslationResult. By default (QualityScoreGranularity::NONE), scores are not included. */
+	void setQualityScoreGranularity(QualityScoreGranularity granularity);
+
+	/* Set the type of Alignment b/w original and translated text to be included in the TranslationResult. By default (AlignmentType::NONE), alignment is not included. */
+	void setAlignmentType(AlignmentType alignmentType);
+
+	/* Set to true/false to include/exclude the original text in the TranslationResult. By default (false), the original text is not included. */
+	void includeOriginalText(bool originalText);
+
+	/* Set to true/false to include/exclude the information regarding how individual sentences of original text map to corresponding translated sentences in joined translated text in the TranslationResult. By default (false), this information is not included. */
+	void includeSentenceMapping(bool sentenceMapping);
+
+	/* Return the granularity for which the Quality scores of the translated text will be included in TranslationResult. QualityScoreGranularity::NONE means the scores will not be included. */
+	QualityScoreGranularity getQualityScoreGranularity() const;
+
+	/* Return the type of Alignment b/w original and translated text that should be included in the TranslationResult. AlignmentType::NONE means the alignment will not be included. */
+	AlignmentType getAlignmentType() const;
+
+	/* Return whether the original text should be included in the TranslationResult. False means the original text will not be included. */
+	bool includeOriginalText() const;
+
+	/* Return whether the information regarding how individual sentences of original text map to corresponding translated sentences in joined translated text should be included in the TranslationResult. False means this information will not be included. */
+	bool includeSentenceMapping() const;
+}
+
+/* This class represents the result of translation on a TranslationRequest. */
+
+class **TranslationResult** {
+
+    private:
+
+	// Original text (utf-8) that was supposed to be translated; An optional result (it will be an empty string if not requested in TranslationRequest).
+	std::string originalText;
+
+	// Translation (in utf-8 format) of the originalText
+	std::string translatedText;
+
+	// Quality score of the translated text at the granularity specified in TranslationRequest; An optional result (it will have no information if not requested in TranslationRequest)
+	QualityScore qualityScore;
+
+	// Alignment information b/w original and translated text for AlignmentType specified in TranslationRequest; An optional result (it will have no information if not requested in TranslationRequest)
+	Alignment alignment;
+
+	// Information regarding how individual sentences of originalText map to corresponding translated sentences
+        // in joined translated text (translatedText); An optional result (it will be empty if not requested in TranslationRequest);
+        //       An example:
+        //       originalText (contains 2 sentences)              = "What is your name? My name is Abc."
+        //       translatedText (contains 2 translated sentences) = "Was ist dein Name? Mein Name ist Abc."
+        //       sentenceMappings = [
+        //                   {"What is your name?", "Was ist dein Name?"},  // A pair of Sentence 1 of originalText (originalText[0]) and corresponding translated sentence in translatedText (translatedText[0])
+        //                   {"My name is Abc", "Mein Name ist Abc."}       // A pair of Sentence 2 of originalText (originalText[1]) and corresponding translated sentence in translatedText (translatedText[1])
+        //                 ]
+        //
+	std::vector<std::pair<std::string_view, std::string_view>> sentenceMappings;
+
+    public:
+	// ToDo: Public Methods
+}
+
+/* This class encapsulates the configuration that is required by a translation model to perform translation. This configuration includes a path to the model file, source language vocabulary file, target language vocabulary file along with other options. */
+
+class **TranslationModelConfiguration** {
+
+    private:
+
+	// Path to the translation model file
+	const std::string modelPath;
+
+	// Path to the source vocabulary file to be used by the model
+	const std::string sourceLanguageVocabPath;
+
+	// Path to the target vocabulary file to be used by the model	
+	const std::string targetLanguageVocabPath;
+
+	// ToDo: Add all possible user configurable options (e.g. min batch size, max batch size) that are relevant for translation
+
+    public:
+
+	// Provide the path to the model file along with the source and target vocabulary files
+	TranslationModelConfiguration(const std::string& modelFilePath,
+					const std::string& sourceVocabPath,
+					const std::string& targetVocabPath);
+
+	// Return the path of the model file
+	const std::string& getModelFilePath() const;
+
+	// Return the path of the source language vocabulary file
+	const std::string& getSourceVocabularyPath() const;
+
+	// Return the path of the target language vocabulary file
+	const std::string& getSourceVocabularyPath() const;
+}
+
+// All possible granularities for which Quality Scores can be returned for translated (utf-8) text
+
+enum class QualityScoreGranularity {
+
+	WORD,
+	SENTENCE,
+	NONE,
+}
+
+// All possible supported alignment types between a text and its translation
+
+enum class AlignmentType {
+
+	SOFT,
+	NONE,
+}
+
+// This class represents the Quality Scores for various spans of the translated text at a specific granularity
+
+class QualityScore {
+
+      private:
+
+	// Sections of a text for the Quality Scores
+	std::vector<std::string_view> textViews;
+
+	// Quality Scores corresponding to each section of the text in textViews in the same order
+	std::vector<float> textScores;
+
+        // Granularity of the text for the Quality scores above
+	QualityScoreGranularity textGranularity;
+
+      public:
+	// ToDo: Public Methods
+}
+
+// This class encapsulates a translated text, all the sections of the original text that align to this translated text and the corresponding alignments for each of these sections of original text.
+
+class Alignment {
+
+      private:
+
+        // A list of sections of a translated text
+        // An example: originalText   = "What do you need"
+        //             translatedText = "Was brauchst du"
+        //             translatedTextViews = ["Was ", "brauchst", "du"]
+	std::vector<std::string_view> translatedTextViews;
+
+        // Each ith entry of this container corresponds to a list of all the sections of the original text that align to the ith entry of translatedTextView
+        // For the example above:
+        //             translatedTextViews = ["Was ", "brauchst", "du"]
+        //             originalTextViews   = [
+        //                                      ["What"],         // originalTextViews[0] = All sections of original text that align with translatedTextViews[0] i.e. "Was"
+        //                                      ["you", "need"],  // originalTextViews[1] = All sections of original text that align with translatedTextViews[1] i.e. "brauchst"
+        //                                      ["you"]           // originalTextViews[2] = All sections of original text that align with translatedTextViews[2] i.e. "du"
+        //                                   ]
+	std::vector<std::vector<std::string_view>> originalTextViews;
+
+        // Each ith entry of this container corresponds to the alignments of all the sections of the original text (ith entry of originalTextViews) that align to the ith entry of translatedTextViews
+        // For the example above:
+        //             alignments          = [
+        //                                      [0.90],         // alignments[0] = Alignments of all sections of original text (i.e. originalTextViews[0]) to translatedTextViews[0] i.e. "Was"
+        //                                      [0.3, 0.7],     // alignments[1] = Alignments of all sections of original text (i.e. originalTextViews[1]) to translatedTextViews[1] i.e. "brauchst"
+        //                                      [0.9]           // alignments[2] = Alignments of all sections of original text (i.e. originalTextViews[2]) to translatedTextViews[2] i.e. "du"
+        //                                   ]
+	std::vector<std::vector<float>> alignments;
+
+        // Type of the alignment b/w original and translated text above
+	AlignmentType alignmentType;
+
+      public:
+	// ToDo: Public Methods
+}

From e5f3d51effc37c21de9350124e1c354744694ffa Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 3 Nov 2020 09:00:33 +0100
Subject: [PATCH 003/442] Basic skeleton code for the Unified API specification

 - Contains classes for the API specification (doc/Unified_API.md)
 - Things to be changed/decided later:
     Use of std::string_view to represent ranges
     Adding Alignment information
     Basic Setters and Getters for some of the classes
---
 CMakeLists.txt                              | 13 ++++
 src/CMakeLists.txt                          |  1 +
 src/translator/AbstractTranslationModel.cpp |  8 +++
 src/translator/AbstractTranslationModel.h   | 52 +++++++++++++++
 src/translator/CMakeLists.txt               |  2 +
 src/translator/QualityScore.h               | 36 ++++++++++
 src/translator/TranslationRequest.h         | 69 +++++++++++++++++++
 src/translator/TranslationResult.h          | 74 +++++++++++++++++++++
 8 files changed, 255 insertions(+)
 create mode 100644 CMakeLists.txt
 create mode 100644 src/CMakeLists.txt
 create mode 100644 src/translator/AbstractTranslationModel.cpp
 create mode 100644 src/translator/AbstractTranslationModel.h
 create mode 100644 src/translator/CMakeLists.txt
 create mode 100644 src/translator/QualityScore.h
 create mode 100644 src/translator/TranslationRequest.h
 create mode 100644 src/translator/TranslationResult.h

diff --git a/CMakeLists.txt b/CMakeLists.txt
new file mode 100644
index 000000000..d4890299b
--- /dev/null
+++ b/CMakeLists.txt
@@ -0,0 +1,13 @@
+cmake_minimum_required(VERSION 3.5.1)
+
+if (POLICY CMP0074)
+  cmake_policy(SET CMP0074 NEW) # CMake 3.12
+endif ()
+
+project(bergamot_translator CXX C)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(BUILD_ARCH native CACHE STRING "Compile for this CPU architecture.")
+
+add_subdirectory(src)
\ No newline at end of file
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
new file mode 100644
index 000000000..27fecc4bc
--- /dev/null
+++ b/src/CMakeLists.txt
@@ -0,0 +1 @@
+add_subdirectory(translator)
\ No newline at end of file
diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
new file mode 100644
index 000000000..a180a710a
--- /dev/null
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -0,0 +1,8 @@
+/*
+ * AbstractTranslationModel.cpp
+ *
+ */
+
+#include "AbstractTranslationModel.h"
+
+AbstractTranslationModel::~AbstractTranslationModel() {}
diff --git a/src/translator/AbstractTranslationModel.h b/src/translator/AbstractTranslationModel.h
new file mode 100644
index 000000000..6f013afb0
--- /dev/null
+++ b/src/translator/AbstractTranslationModel.h
@@ -0,0 +1,52 @@
+/*
+ * AbstractTranslationModel.h
+ *
+ * An interface for a translation model for translating a plain (without any markups and emojis) UTF-8 encoded text.
+ * The model supports translation from 1 source language to 1 target language. There can be different implementations
+ * of this interface.
+ */
+
+#ifndef SRC_TRANSLATOR_ABSTRACTTRANSLATIONMODEL_H_
+#define SRC_TRANSLATOR_ABSTRACTTRANSLATIONMODEL_H_
+
+#include <vector>
+#include <string>
+#include <future>
+
+#include "TranslationRequest.h"
+#include "TranslationResult.h"
+
+/* An interface for a translation model for translating a plain (without any markups and emojis) UTF-8 encoded text.
+ * The model supports translation from 1 source language to 1 target language.
+ */
+class AbstractTranslationModel {
+public:
+
+	AbstractTranslationModel();
+
+	virtual ~AbstractTranslationModel();
+
+	/* This method performs translation on a list of (UTF-8 encoded) texts and returns a list of results in the same order.
+	 * Each text entry can either be a word, a phrase, a sentence or a list of sentences and should contain plain text
+	 * (without any markups or emojis). Additional information related to the translated text can be requested via
+	 * TranslationRequest which is applied equally to each text entry.
+	 *
+	 * The translated text corresponding to each text entry and the additional information (as specified in the
+	 * TranslationRequest) is encapsulated and returned in TranslationResult.
+	 *
+	 * The API splits each text entry into sentences internally, which are then translated independent of each other.
+	 * The translated sentences are then joined together and returned in TranslationResult.
+	 * Please refer to the TranslationRequest class to find out what additional information can be requested.
+	 * The alignment information can only be requested if the model supports it (check isAlignmentSupported() API).
+	 *
+	 * The texts argument will become empty after the execution of this API (each entry of texts list will be moved to its
+	 * corresponding TranslationResult object).
+	 */
+	virtual std::future<std::vector<TranslationResult>> translate(
+			std::vector<std::string> &&texts, TranslationRequest request) = 0;
+
+	/* Check if the model can provide alignment information b/w original and translated text. */
+	virtual bool isAlignmentSupported() const = 0;
+};
+
+#endif /* SRC_TRANSLATOR_ABSTRACTTRANSLATIONMODEL_H_ */
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
new file mode 100644
index 000000000..bcd34da33
--- /dev/null
+++ b/src/translator/CMakeLists.txt
@@ -0,0 +1,2 @@
+include_directories(.)
+add_library(bergamot-translator STATIC AbstractTranslationModel.cpp)
\ No newline at end of file
diff --git a/src/translator/QualityScore.h b/src/translator/QualityScore.h
new file mode 100644
index 000000000..020aebc8e
--- /dev/null
+++ b/src/translator/QualityScore.h
@@ -0,0 +1,36 @@
+/*
+ * QualityScore.h
+ *
+ */
+
+#ifndef SRC_TRANSLATOR_QUALITYSCORE_H_
+#define SRC_TRANSLATOR_QUALITYSCORE_H_
+
+#include <vector>
+#include <string>
+
+
+/* All possible Granularities for which Quality Scores can be returned for translated text. */
+enum class QualityScoreGranularity {
+	WORD, SENTENCE, NONE,
+};
+
+/* This class represents the Quality Scores for various spans of a translated text at a specific granularity. */
+class QualityScore {
+private:
+
+	// Sections of the translated text for the Quality Scores.
+	std::vector<std::string_view> textViews;
+
+	// Quality Scores corresponding to each entry of textViews in the same order
+	std::vector<float> textScores;
+
+	// Granularity of the text for the Quality scores above
+	QualityScoreGranularity textGranularity;
+
+public:
+	// ToDo: Public Methods
+};
+
+
+#endif /* SRC_TRANSLATOR_QUALITYSCORE_H_ */
diff --git a/src/translator/TranslationRequest.h b/src/translator/TranslationRequest.h
new file mode 100644
index 000000000..bdd56803a
--- /dev/null
+++ b/src/translator/TranslationRequest.h
@@ -0,0 +1,69 @@
+/*
+ * TranslationRequest.h
+ *
+ *  This file defines the translation request class to be used in AbstractTranslationModel::translate() API.
+ */
+
+#ifndef SRC_TRANSLATOR_TRANSLATIONREQUEST_H_
+#define SRC_TRANSLATOR_TRANSLATIONREQUEST_H_
+
+#include "QualityScore.h"
+
+/* This class specifies the information related to the translated text (e.g. quality of the translation etc.) that
+ * can be included in the TranslationResult. These optional requests are set/unset independent of each other i.e. setting
+ * any one of them doesn’t have the side effect of setting any of the others.
+ */
+class TranslationRequest {
+private:
+	// The granularity for which Quality scores of the translated text will be included in TranslationResult.
+	// QualityScoreGranularity::NONE means the scores are not included in TranslationResult.
+	QualityScoreGranularity qualityScoreGranularity = QualityScoreGranularity::NONE;
+
+	// A flag to include/exclude the information regarding how individual sentences of original text map to
+	// corresponding translated sentences in joined translated text in the TranslationResult.
+	// An example of sentence mappings:
+	//     originalText (containing 2 sentences)              = "What is your name? My name is Abc."
+	//     translatedText (containing 2 translated sentences) = "Was ist dein Name? Mein Name ist Abc."
+	//     sentenceMappings = [
+	//         {"What is your name?", "Was ist dein Name?"},  // Pair(originalText[0],translatedText[0])
+	//         {"My name is Abc", "Mein Name ist Abc."}       // Pair(originalText[1],translatedText[1])
+	//     ]
+	bool includeSentenceMapping = false;
+
+public:
+	explicit TranslationRequest();
+
+	~TranslationRequest();
+
+	/* Set the granularity for which the Quality scores of translated text should be included in the TranslationResult.
+	 * By default (QualityScoreGranularity::NONE), scores are not included.
+	 */
+	void setQualityScoreGranularity(QualityScoreGranularity granularity) {
+		qualityScoreGranularity = granularity;
+	}
+
+	/* Set to true/false to include/exclude the information regarding how individual sentences of original text map to
+	 * corresponding translated sentences in joined translated text in the TranslationResult. By default (false), this
+	 * information is not included.
+	 */
+	void sentenceMappingInResult(bool includeMapping) {
+		includeSentenceMapping = includeMapping;
+	}
+
+	/* Return the granularity for which the Quality scores of the translated text will be included in TranslationResult.
+	 * QualityScoreGranularity::NONE means the scores will not be included.
+	 */
+	QualityScoreGranularity getQualityScoreGranularity() const {
+		return qualityScoreGranularity;
+	}
+
+	/* Return whether the information regarding how individual sentences of original text map to corresponding translated
+	 * sentences in joined translated text will be included in the TranslationResult. By default (false) means this
+	 * information will not be included.
+	 */
+	bool sentenceMappingInResult() const {
+		return includeSentenceMapping;
+	}
+};
+
+#endif /* SRC_TRANSLATOR_TRANSLATIONREQUEST_H_ */
diff --git a/src/translator/TranslationResult.h b/src/translator/TranslationResult.h
new file mode 100644
index 000000000..33bad1b66
--- /dev/null
+++ b/src/translator/TranslationResult.h
@@ -0,0 +1,74 @@
+/*
+ * TranslationResult.h
+ *
+ * The class that represents the result of AbstractTranslationModel::translate() API for each of its text entry and
+ * TranslationRequest.
+ */
+
+#ifndef SRC_TRANSLATOR_TRANSLATIONRESULT_H_
+#define SRC_TRANSLATOR_TRANSLATIONRESULT_H_
+
+#include <vector>
+#include <string>
+
+#include "QualityScore.h"
+
+/* This class represents the result of AbstractTranslationModel::translate() API for each of its text entry and
+ * TranslationRequest.
+ */
+class TranslationResult {
+public:
+	typedef std::vector<std::pair<std::string_view, std::string_view>> SentenceMappings;
+
+	TranslationResult(const std::string &original, const std::string &translation);
+
+	TranslationResult(std::string &&original, std::string &&translation);
+
+	/* Return the original text. */
+	const std::string& getOriginalText() const {
+		return originalText;
+	}
+
+	/* Return the translated text. */
+	const std::string& getTranslatedText() const {
+		return translatedText;
+	}
+
+	/* Return the Quality scores of the translated text. */
+	const QualityScore& getQualityScore() const {
+		return qualityScore;
+	}
+
+	/* Return the Sentence mappings (information regarding how individual sentences of originalText map to
+	 * corresponding translated sentences in translatedText).
+	 */
+	const SentenceMappings& getSentenceMappings() const {
+		return sentenceMappings;
+	}
+
+private:
+	// Original text (in UTF-8 encoded format) that was supposed to be translated
+	std::string originalText;
+
+	// Translation (in UTF-8 encoded format) of the originalText
+	std::string translatedText;
+
+	// Quality score of the translated text at the granularity specified in TranslationRequest.
+	// It is an optional result (it will have no information if not requested in TranslationRequest)
+	QualityScore qualityScore;
+
+	// Information regarding how individual sentences of originalText map to corresponding translated sentences
+	// in joined translated text (translatedText)
+	// An example of sentence mapping:
+	//     originalText (contains 2 sentences)              = "What is your name? My name is Abc."
+	//     translatedText (contains 2 translated sentences) = "Was ist dein Name? Mein Name ist Abc."
+	//     sentenceMappings = [
+	//         {"What is your name?", "Was ist dein Name?"},  // Pair(originalText[0],translatedText[0])
+	//         {"My name is Abc", "Mein Name ist Abc."}       // Pair(originalText[1],translatedText[1])
+	//     ]
+	//
+	// It is an optional result (it will be empty if not requested in TranslationRequest).
+	SentenceMappings sentenceMappings;
+};
+
+#endif /* SRC_TRANSLATOR_TRANSLATIONRESULT_H_ */

From cd90f89126d3a7040ebb181caa294744bdfa2d05 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 3 Nov 2020 15:33:10 +0100
Subject: [PATCH 004/442] Added TranslationModel class

 - This class is an implementation of AbstractTranslationModel
   interface

 - This is the main class that will implement the translate API
 - Contains dummy responses for now
---
 src/translator/AbstractTranslationModel.cpp   |  7 ++
 src/translator/AbstractTranslationModel.h     |  7 ++
 src/translator/CMakeLists.txt                 |  2 +-
 src/translator/TranslationModel.cpp           | 29 ++++++++
 src/translator/TranslationModel.h             | 63 +++++++++++++++++
 .../TranslationModelConfiguration.h           | 68 +++++++++++++++++++
 6 files changed, 175 insertions(+), 1 deletion(-)
 create mode 100644 src/translator/TranslationModel.cpp
 create mode 100644 src/translator/TranslationModel.h
 create mode 100644 src/translator/TranslationModelConfiguration.h

diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
index a180a710a..2f4f05631 100644
--- a/src/translator/AbstractTranslationModel.cpp
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -2,7 +2,14 @@
  * AbstractTranslationModel.cpp
  *
  */
+#include <memory>
 
 #include "AbstractTranslationModel.h"
+#include "TranslationModel.h"
 
 AbstractTranslationModel::~AbstractTranslationModel() {}
+
+std::shared_ptr<AbstractTranslationModel>
+AbstractTranslationModel::createInstance(const TranslationModelConfiguration& config) {
+	return std::make_shared<TranslationModel>(config);
+}
diff --git a/src/translator/AbstractTranslationModel.h b/src/translator/AbstractTranslationModel.h
index 6f013afb0..77ad87e5b 100644
--- a/src/translator/AbstractTranslationModel.h
+++ b/src/translator/AbstractTranslationModel.h
@@ -12,7 +12,9 @@
 #include <vector>
 #include <string>
 #include <future>
+#include <memory>
 
+#include "TranslationModelConfiguration.h"
 #include "TranslationRequest.h"
 #include "TranslationResult.h"
 
@@ -22,6 +24,11 @@
 class AbstractTranslationModel {
 public:
 
+	/* A Factory method to create and return an instance of AbstractTranslationModel implementation. The instance is
+	 * created using translation model configuration (TranslationModelConfiguration).
+	 */
+	static std::shared_ptr<AbstractTranslationModel> createInstance(const TranslationModelConfiguration& config);
+
 	AbstractTranslationModel();
 
 	virtual ~AbstractTranslationModel();
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index bcd34da33..b227decb8 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,2 +1,2 @@
 include_directories(.)
-add_library(bergamot-translator STATIC AbstractTranslationModel.cpp)
\ No newline at end of file
+add_library(bergamot-translator STATIC AbstractTranslationModel.cpp TranslationModel.cpp)
\ No newline at end of file
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
new file mode 100644
index 000000000..a309cdd3d
--- /dev/null
+++ b/src/translator/TranslationModel.cpp
@@ -0,0 +1,29 @@
+/*
+ * TranslationModel.cpp
+ *
+ */
+
+#include <future>
+#include <vector>
+
+#include "TranslationModel.h"
+
+TranslationModel::TranslationModel(const TranslationModelConfiguration &configuration) :
+		modelConfiguration(configuration), AbstractTranslationModel() {
+}
+
+TranslationModel::~TranslationModel() {}
+
+std::future<std::vector<TranslationResult>> TranslationModel::translate(
+		std::vector<std::string> &&texts, TranslationRequest request) {
+	//ToDo: Replace this code with the actual implementation
+	return std::async([]() {
+		std::vector<TranslationResult> results;
+		results.emplace_back(TranslationResult{"a","d"});
+		return results;
+	});
+}
+
+bool TranslationModel::isAlignmentSupported() const {
+	return false;
+}
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
new file mode 100644
index 000000000..14cbcbd8b
--- /dev/null
+++ b/src/translator/TranslationModel.h
@@ -0,0 +1,63 @@
+/*
+ * TranslationModel.h
+ *
+ *  A implementation of AbstractTranslationModel interface.
+ */
+
+#ifndef SRC_TRANSLATOR_TRANSLATIONMODEL_H_
+#define SRC_TRANSLATOR_TRANSLATIONMODEL_H_
+
+#include <vector>
+#include <string>
+#include <future>
+
+#include "AbstractTranslationModel.h"
+#include "TranslationModelConfiguration.h"
+
+/* A Translation model that translates a plain (without any markups and emojis) UTF-8 encoded text.
+ * This implementation supports translation from 1 source language to 1 target language.
+ */
+class TranslationModel: public AbstractTranslationModel {
+public:
+	/* Construct the model using the model configuration. The model configuration specifies options
+	 * that are required by a translation model to perform translation. It stays constant during the
+	 * lifetime of the model instance. Please refer to TranslationModelConfiguration class
+	 * for details regarding configuration.
+	 */
+	TranslationModel(const TranslationModelConfiguration &modelConfiguration);
+
+	~TranslationModel();
+
+	/* This method performs translation on a list of UTF-8 encoded plain text (without any markups
+	 * or emojis) and returns a list of results in the same order. The model supports translation
+	 * from 1 source language to 1 target language.
+	 *
+	 * Each text entry can either be a word, a phrase, a sentence or a list of sentences. Additional
+	 * information related to the translated text can be requested via TranslationRequest which is
+	 * applied equally to each text entry. The translated text corresponding to each text entry and
+	 * the additional information (as specified in the TranslationRequest) is encapsulated and
+	 * returned in TranslationResult.
+	 *
+	 * The API splits each text entry into sentences internally, which are then translated
+	 * independent of each other. The translated sentences are then joined back together and returned
+	 * in TranslationResult.
+	 *
+	 * Please refer to the TranslationRequest class to find out what additional information can be
+	 * requested. The alignment information can only be requested if the model supports it (check
+	 * isAlignmentSupported() API).
+	 *
+	 * The texts argument will become empty after the execution of this API (each entry of texts list
+	 * will be moved to its corresponding TranslationResult object).
+	 */
+	std::future<std::vector<TranslationResult>> translate(
+			std::vector<std::string> &&texts, TranslationRequest request) override;
+
+	/* Check if the model can provide alignment information b/w original and translated text. */
+	bool isAlignmentSupported() const override;
+
+private:
+	// Model configuration
+	const TranslationModelConfiguration modelConfiguration;
+};
+
+#endif /* SRC_TRANSLATOR_TRANSLATIONMODEL_H_ */
diff --git a/src/translator/TranslationModelConfiguration.h b/src/translator/TranslationModelConfiguration.h
new file mode 100644
index 000000000..8c6582454
--- /dev/null
+++ b/src/translator/TranslationModelConfiguration.h
@@ -0,0 +1,68 @@
+/*
+ * TranslationModelConfiguration.h
+ *
+ */
+
+#ifndef SRC_TRANSLATOR_TRANSLATIONMODELCONFIGURATION_H_
+#define SRC_TRANSLATOR_TRANSLATIONMODELCONFIGURATION_H_
+
+#include <string>
+
+/* This class encapsulates the configuration that is required by a translation model to perform
+ * translation.
+ */
+class TranslationModelConfiguration {
+public:
+
+	// Constructor
+	TranslationModelConfiguration(const std::string &modelFilePath,
+			const std::string &sourceVocabPath,
+			const std::string &targetVocabPath) :
+				modelPath(modelFilePath),
+				sourceLanguageVocabPath(sourceVocabPath),
+				targetLanguageVocabPath(targetVocabPath) {
+	}
+
+	// Copy constructor
+	TranslationModelConfiguration(const TranslationModelConfiguration &rhs) :
+			modelPath(rhs.modelPath),
+			sourceLanguageVocabPath(rhs.sourceLanguageVocabPath),
+			targetLanguageVocabPath(rhs.targetLanguageVocabPath) {
+	}
+
+	// Move constructor
+	TranslationModelConfiguration(TranslationModelConfiguration &&rhs) :
+			modelPath(std::move(rhs.modelPath)),
+			sourceLanguageVocabPath(std::move(rhs.sourceLanguageVocabPath)),
+			targetLanguageVocabPath(std::move(rhs.targetLanguageVocabPath)) {
+	}
+
+	// Return the path of the model file
+	const std::string& getModelFilePath() const {
+		return modelPath;
+	}
+
+	// Return the path of the source language vocabulary file
+	const std::string& getSourceVocabularyPath() const {
+		return sourceLanguageVocabPath;
+	}
+
+	// Return the path of the target language vocabulary file
+	const std::string& getTargetVocabularyPath() const {
+		return targetLanguageVocabPath;
+	}
+
+private:
+	// Path to the translation model file
+	const std::string modelPath;
+
+	// Path to the source vocabulary file to be used by the model
+	const std::string sourceLanguageVocabPath;
+
+	// Path to the target vocabulary file to be used by the model
+	const std::string targetLanguageVocabPath;
+
+	// ToDo: Add other user configurable options (e.g. min batch size)
+};
+
+#endif /* SRC_TRANSLATOR_TRANSLATIONMODELCONFIGURATION_H_ */

From 468508d75d97ed628e5f9749bca00d195443a3ad Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 3 Nov 2020 18:09:07 +0100
Subject: [PATCH 005/442] Added constructor definitions

 - Added definitions that were absent in previous commits
   of unified API
---
 src/translator/AbstractTranslationModel.cpp |  1 -
 src/translator/AbstractTranslationModel.h   | 12 +++++++-----
 src/translator/TranslationModel.cpp         |  1 -
 src/translator/TranslationRequest.h         |  9 +++++++--
 src/translator/TranslationResult.h          |  6 ++++--
 5 files changed, 18 insertions(+), 11 deletions(-)

diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
index 2f4f05631..39b359af4 100644
--- a/src/translator/AbstractTranslationModel.cpp
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -7,7 +7,6 @@
 #include "AbstractTranslationModel.h"
 #include "TranslationModel.h"
 
-AbstractTranslationModel::~AbstractTranslationModel() {}
 
 std::shared_ptr<AbstractTranslationModel>
 AbstractTranslationModel::createInstance(const TranslationModelConfiguration& config) {
diff --git a/src/translator/AbstractTranslationModel.h b/src/translator/AbstractTranslationModel.h
index 77ad87e5b..ddadc07bf 100644
--- a/src/translator/AbstractTranslationModel.h
+++ b/src/translator/AbstractTranslationModel.h
@@ -24,14 +24,16 @@
 class AbstractTranslationModel {
 public:
 
-	/* A Factory method to create and return an instance of AbstractTranslationModel implementation. The instance is
-	 * created using translation model configuration (TranslationModelConfiguration).
+	/* A Factory method to create and return an instance of an implementation of
+	 * AbstractTranslationModel. The instance is created using translation model configuration
+	 * (TranslationModelConfiguration).
 	 */
-	static std::shared_ptr<AbstractTranslationModel> createInstance(const TranslationModelConfiguration& config);
+	static std::shared_ptr<AbstractTranslationModel>
+	createInstance(const TranslationModelConfiguration& config);
 
-	AbstractTranslationModel();
+	AbstractTranslationModel() = default;
 
-	virtual ~AbstractTranslationModel();
+	virtual ~AbstractTranslationModel() = default;
 
 	/* This method performs translation on a list of (UTF-8 encoded) texts and returns a list of results in the same order.
 	 * Each text entry can either be a word, a phrase, a sentence or a list of sentences and should contain plain text
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index a309cdd3d..ed894e567 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -19,7 +19,6 @@ std::future<std::vector<TranslationResult>> TranslationModel::translate(
 	//ToDo: Replace this code with the actual implementation
 	return std::async([]() {
 		std::vector<TranslationResult> results;
-		results.emplace_back(TranslationResult{"a","d"});
 		return results;
 	});
 }
diff --git a/src/translator/TranslationRequest.h b/src/translator/TranslationRequest.h
index bdd56803a..b19cc892d 100644
--- a/src/translator/TranslationRequest.h
+++ b/src/translator/TranslationRequest.h
@@ -31,9 +31,14 @@ class TranslationRequest {
 	bool includeSentenceMapping = false;
 
 public:
-	explicit TranslationRequest();
+	TranslationRequest() {}
 
-	~TranslationRequest();
+	TranslationRequest(const TranslationRequest& request) :
+		qualityScoreGranularity(request.qualityScoreGranularity),
+		includeSentenceMapping(request.includeSentenceMapping) {
+	}
+
+	~TranslationRequest() {}
 
 	/* Set the granularity for which the Quality scores of translated text should be included in the TranslationResult.
 	 * By default (QualityScoreGranularity::NONE), scores are not included.
diff --git a/src/translator/TranslationResult.h b/src/translator/TranslationResult.h
index 33bad1b66..4d231a89b 100644
--- a/src/translator/TranslationResult.h
+++ b/src/translator/TranslationResult.h
@@ -20,9 +20,11 @@ class TranslationResult {
 public:
 	typedef std::vector<std::pair<std::string_view, std::string_view>> SentenceMappings;
 
-	TranslationResult(const std::string &original, const std::string &translation);
+	TranslationResult(const std::string &original, const std::string &translation) :
+		originalText(original), translatedText(translation) {}
 
-	TranslationResult(std::string &&original, std::string &&translation);
+	TranslationResult(std::string &&original, std::string &&translation) :
+		originalText(std::move(original)), translatedText(std::move(translation)) {}
 
 	/* Return the original text. */
 	const std::string& getOriginalText() const {

From 7a695a08cbca1b2b7e8610e11492f099dcdc1991 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 9 Nov 2020 12:01:54 +0100
Subject: [PATCH 006/442] Added "ugermann/ssplit-cpp" as a submodule

---
 .gitmodules          | 3 +++
 3rd_party/ssplit-cpp | 1 +
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 3rd_party/ssplit-cpp

diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 000000000..c3d3b4dbb
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "3rd_party/ssplit-cpp"]
+	path = 3rd_party/ssplit-cpp
+	url = https://github.com/ugermann/ssplit-cpp
diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
new file mode 160000
index 000000000..f5d022992
--- /dev/null
+++ b/3rd_party/ssplit-cpp
@@ -0,0 +1 @@
+Subproject commit f5d022992f4a00c860eb809389748908bb85ffcf

From e8716f7fd1b0b1d68e9fab304da320f63880a957 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 9 Nov 2020 12:02:51 +0100
Subject: [PATCH 007/442] Added "browsermt/marian-dev" as submodule

---
 .gitmodules          | 3 +++
 3rd_party/marian-dev | 1 +
 2 files changed, 4 insertions(+)
 create mode 160000 3rd_party/marian-dev

diff --git a/.gitmodules b/.gitmodules
index c3d3b4dbb..d3bbf18d6 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,3 +1,6 @@
 [submodule "3rd_party/ssplit-cpp"]
 	path = 3rd_party/ssplit-cpp
 	url = https://github.com/ugermann/ssplit-cpp
+[submodule "3rd_party/marian-dev"]
+	path = 3rd_party/marian-dev
+	url = https://github.com/browsermt/marian-dev
diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
new file mode 160000
index 000000000..69894793e
--- /dev/null
+++ b/3rd_party/marian-dev
@@ -0,0 +1 @@
+Subproject commit 69894793ebd93256d824a1590924780a6d54cae8

From a220f915fc6915063b3ef5a2d3d3c6e8589df79d Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 11 Nov 2020 16:19:54 +0100
Subject: [PATCH 008/442] Compile marian submodule in the project

 - marian compiles successfully and is ready to be used
   in the project
---
 3rd_party/CMakeLists.txt | 6 ++++++
 CMakeLists.txt           | 6 ++++++
 2 files changed, 12 insertions(+)
 create mode 100644 3rd_party/CMakeLists.txt

diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
new file mode 100644
index 000000000..5a2b56e24
--- /dev/null
+++ b/3rd_party/CMakeLists.txt
@@ -0,0 +1,6 @@
+add_subdirectory(marian-dev)
+
+# Add include directories for marian target to be able to use it anywhere in the project without
+# explicitly specifying its include directories. Once marian fixes this problem, it can be removed.
+get_property(INCDIRS DIRECTORY marian-dev/src PROPERTY INCLUDE_DIRECTORIES)
+target_include_directories(marian PUBLIC ${INCDIRS})
\ No newline at end of file
diff --git a/CMakeLists.txt b/CMakeLists.txt
index d4890299b..2b868299c 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -10,4 +10,10 @@ set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 set(BUILD_ARCH native CACHE STRING "Compile for this CPU architecture.")
 
+# Custom CMake options to compile marian (a 3rd party submodule) for this project
+option(COMPILE_CUDA "Compile GPU version" OFF)
+option(USE_SENTENCEPIECE "Download and compile SentencePiece" ON)
+option(USE_STATIC_LIBS "Link statically against non-system libs" ON)
+
+add_subdirectory(3rd_party)
 add_subdirectory(src)
\ No newline at end of file

From 36911d39d5d155ac9a10d82c0e687caeb64895e1 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 11 Nov 2020 16:24:50 +0100
Subject: [PATCH 009/442] Link marian library in the project

---
 src/translator/CMakeLists.txt | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index b227decb8..c820c9309 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,2 +1,5 @@
-include_directories(.)
-add_library(bergamot-translator STATIC AbstractTranslationModel.cpp TranslationModel.cpp)
\ No newline at end of file
+add_library(bergamot-translator STATIC
+    AbstractTranslationModel.cpp
+    TranslationModel.cpp)
+
+target_link_libraries(bergamot-translator marian)
\ No newline at end of file

From 358d76871fe6dce602a70cfd7608bd43443451f8 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 11 Nov 2020 17:18:12 +0100
Subject: [PATCH 010/442] Small change: Added New line endings

---
 3rd_party/CMakeLists.txt      | 2 +-
 CMakeLists.txt                | 2 +-
 src/translator/CMakeLists.txt | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index 5a2b56e24..97bf94e05 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -3,4 +3,4 @@ add_subdirectory(marian-dev)
 # Add include directories for marian target to be able to use it anywhere in the project without
 # explicitly specifying its include directories. Once marian fixes this problem, it can be removed.
 get_property(INCDIRS DIRECTORY marian-dev/src PROPERTY INCLUDE_DIRECTORIES)
-target_include_directories(marian PUBLIC ${INCDIRS})
\ No newline at end of file
+target_include_directories(marian PUBLIC ${INCDIRS})
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 2b868299c..6aaff4ee6 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -16,4 +16,4 @@ option(USE_SENTENCEPIECE "Download and compile SentencePiece" ON)
 option(USE_STATIC_LIBS "Link statically against non-system libs" ON)
 
 add_subdirectory(3rd_party)
-add_subdirectory(src)
\ No newline at end of file
+add_subdirectory(src)
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index c820c9309..ac8936645 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -2,4 +2,4 @@ add_library(bergamot-translator STATIC
     AbstractTranslationModel.cpp
     TranslationModel.cpp)
 
-target_link_libraries(bergamot-translator marian)
\ No newline at end of file
+target_link_libraries(bergamot-translator marian)

From 210c5a466a7e57acf56a0bcb17bcaa2d94b28a99 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 11 Nov 2020 17:52:27 +0100
Subject: [PATCH 011/442] Separated the public includes of the project from
 implementation

 - All interfaces are present in ROOT/src
---
 src/{translator => }/AbstractTranslationModel.h      | 0
 src/{translator => }/QualityScore.h                  | 0
 src/{translator => }/TranslationModelConfiguration.h | 0
 src/{translator => }/TranslationRequest.h            | 0
 src/{translator => }/TranslationResult.h             | 0
 src/translator/CMakeLists.txt                        | 4 ++++
 6 files changed, 4 insertions(+)
 rename src/{translator => }/AbstractTranslationModel.h (100%)
 rename src/{translator => }/QualityScore.h (100%)
 rename src/{translator => }/TranslationModelConfiguration.h (100%)
 rename src/{translator => }/TranslationRequest.h (100%)
 rename src/{translator => }/TranslationResult.h (100%)

diff --git a/src/translator/AbstractTranslationModel.h b/src/AbstractTranslationModel.h
similarity index 100%
rename from src/translator/AbstractTranslationModel.h
rename to src/AbstractTranslationModel.h
diff --git a/src/translator/QualityScore.h b/src/QualityScore.h
similarity index 100%
rename from src/translator/QualityScore.h
rename to src/QualityScore.h
diff --git a/src/translator/TranslationModelConfiguration.h b/src/TranslationModelConfiguration.h
similarity index 100%
rename from src/translator/TranslationModelConfiguration.h
rename to src/TranslationModelConfiguration.h
diff --git a/src/translator/TranslationRequest.h b/src/TranslationRequest.h
similarity index 100%
rename from src/translator/TranslationRequest.h
rename to src/TranslationRequest.h
diff --git a/src/translator/TranslationResult.h b/src/TranslationResult.h
similarity index 100%
rename from src/translator/TranslationResult.h
rename to src/TranslationResult.h
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index ac8936645..5e2b4d6e3 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -3,3 +3,7 @@ add_library(bergamot-translator STATIC
     TranslationModel.cpp)
 
 target_link_libraries(bergamot-translator marian)
+
+target_include_directories(bergamot-translator
+    PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}
+    PUBLIC ${CMAKE_SOURCE_DIR}/src)

From 59c940090b9491874aae7e307953e09a8fc33eea Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 12 Nov 2020 10:23:47 +0100
Subject: [PATCH 012/442] Use marian::Options class internally for
 configuration options

 - Marian uses Options class everywhere as configuration options

 - Owing to this project's heavy dependency on Marian:
   -- Made the internal implementation files of the project work
      with marian::Options instead of TranslationModelConfiguration
   -- An Adaptor class to adapt TranslationModelConfiguration
      to marian::Options will be added in following commit
---
 src/translator/AbstractTranslationModel.cpp |  8 +++++++-
 src/translator/TranslationModel.cpp         |  4 ++--
 src/translator/TranslationModel.h           | 15 ++++++++-------
 3 files changed, 17 insertions(+), 10 deletions(-)

diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
index 39b359af4..0ad3971e6 100644
--- a/src/translator/AbstractTranslationModel.cpp
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -4,11 +4,17 @@
  */
 #include <memory>
 
+// All 3rd party includes
+#include "common/options.h"
+
+// All local includes
 #include "AbstractTranslationModel.h"
 #include "TranslationModel.h"
 
 
 std::shared_ptr<AbstractTranslationModel>
 AbstractTranslationModel::createInstance(const TranslationModelConfiguration& config) {
-	return std::make_shared<TranslationModel>(config);
+	// ToDo: Write an adaptor for adapting TranslationModelConfiguration to marian::Options
+	auto options = std::make_shared<marian::Options>();
+	return std::make_shared<TranslationModel>(options);
 }
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index ed894e567..099d930cd 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -8,8 +8,8 @@
 
 #include "TranslationModel.h"
 
-TranslationModel::TranslationModel(const TranslationModelConfiguration &configuration) :
-		modelConfiguration(configuration), AbstractTranslationModel() {
+TranslationModel::TranslationModel(std::shared_ptr<marian::Options> options) :
+		configOptions(std::move(options)), AbstractTranslationModel() {
 }
 
 TranslationModel::~TranslationModel() {}
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index 14cbcbd8b..5b75b8fec 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -11,6 +11,10 @@
 #include <string>
 #include <future>
 
+// All 3rd party includes
+#include "common/options.h"
+
+// All local project includes
 #include "AbstractTranslationModel.h"
 #include "TranslationModelConfiguration.h"
 
@@ -19,12 +23,9 @@
  */
 class TranslationModel: public AbstractTranslationModel {
 public:
-	/* Construct the model using the model configuration. The model configuration specifies options
-	 * that are required by a translation model to perform translation. It stays constant during the
-	 * lifetime of the model instance. Please refer to TranslationModelConfiguration class
-	 * for details regarding configuration.
+	/* Construct the model using the model configuration options.
 	 */
-	TranslationModel(const TranslationModelConfiguration &modelConfiguration);
+	TranslationModel(std::shared_ptr<marian::Options> options);
 
 	~TranslationModel();
 
@@ -56,8 +57,8 @@ class TranslationModel: public AbstractTranslationModel {
 	bool isAlignmentSupported() const override;
 
 private:
-	// Model configuration
-	const TranslationModelConfiguration modelConfiguration;
+	// Model configuration options
+	std::shared_ptr<marian::Options> configOptions;
 };
 
 #endif /* SRC_TRANSLATOR_TRANSLATIONMODEL_H_ */

From ce7312cfd4ff5f310abd649d7b57f6bf4ff109d5 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 12 Nov 2020 11:17:34 +0100
Subject: [PATCH 013/442] Added basic skeleton for Adaptor class

 - The class adapts the TranslationModelConfiguration to marian::Options
 - Returns a dummy marian::Options for now
---
 src/translator/AbstractTranslationModel.cpp   |  5 +--
 src/translator/CMakeLists.txt                 |  3 +-
 ...TranslationModelConfigToOptionsAdaptor.cpp | 17 ++++++++++
 .../TranslationModelConfigToOptionsAdaptor.h  | 32 +++++++++++++++++++
 4 files changed, 54 insertions(+), 3 deletions(-)
 create mode 100644 src/translator/TranslationModelConfigToOptionsAdaptor.cpp
 create mode 100644 src/translator/TranslationModelConfigToOptionsAdaptor.h

diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
index 0ad3971e6..afd62e7ec 100644
--- a/src/translator/AbstractTranslationModel.cpp
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -10,11 +10,12 @@
 // All local includes
 #include "AbstractTranslationModel.h"
 #include "TranslationModel.h"
+#include "TranslationModelConfigToOptionsAdaptor.h"
 
 
 std::shared_ptr<AbstractTranslationModel>
 AbstractTranslationModel::createInstance(const TranslationModelConfiguration& config) {
-	// ToDo: Write an adaptor for adapting TranslationModelConfiguration to marian::Options
-	auto options = std::make_shared<marian::Options>();
+	TranslationModelConfigToOptionsAdaptor adaptor;
+	auto options = adaptor.adapt(config);
 	return std::make_shared<TranslationModel>(options);
 }
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 5e2b4d6e3..c9a51df45 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,6 +1,7 @@
 add_library(bergamot-translator STATIC
     AbstractTranslationModel.cpp
-    TranslationModel.cpp)
+    TranslationModel.cpp
+    TranslationModelConfigToOptionsAdaptor.cpp)
 
 target_link_libraries(bergamot-translator marian)
 
diff --git a/src/translator/TranslationModelConfigToOptionsAdaptor.cpp b/src/translator/TranslationModelConfigToOptionsAdaptor.cpp
new file mode 100644
index 000000000..3405a5fcf
--- /dev/null
+++ b/src/translator/TranslationModelConfigToOptionsAdaptor.cpp
@@ -0,0 +1,17 @@
+/*
+ * TranslationModelConfigToOptionsAdaptor.cpp
+ *
+ */
+#include <memory>
+
+#include "TranslationModelConfigToOptionsAdaptor.h"
+
+TranslationModelConfigToOptionsAdaptor::TranslationModelConfigToOptionsAdaptor() {}
+
+TranslationModelConfigToOptionsAdaptor::~TranslationModelConfigToOptionsAdaptor() {}
+
+std::shared_ptr<marian::Options>
+TranslationModelConfigToOptionsAdaptor::adapt(const TranslationModelConfiguration& config) {
+	// ToDo: Add actual implementation
+	return std::make_shared<marian::Options>();
+}
diff --git a/src/translator/TranslationModelConfigToOptionsAdaptor.h b/src/translator/TranslationModelConfigToOptionsAdaptor.h
new file mode 100644
index 000000000..309ea69c8
--- /dev/null
+++ b/src/translator/TranslationModelConfigToOptionsAdaptor.h
@@ -0,0 +1,32 @@
+/*
+ * This class adapts the TranslationModelConfiguration object to marian::Options object.
+ * marian::Options is a class that is specific to Marian and is used heavily inside it
+ * as configuration options (even for translation workflow). It makes sense to work with
+ * this class internally in the implementation files.
+ */
+
+#ifndef SRC_TRANSLATOR_TRANSLATIONMODELCONFIGTOOPTIONSADAPTOR_H_
+#define SRC_TRANSLATOR_TRANSLATIONMODELCONFIGTOOPTIONSADAPTOR_H_
+
+#include <memory>
+
+// All 3rd party includes
+#include "common/options.h"
+
+// All local includes
+#include "TranslationModelConfiguration.h"
+
+
+class TranslationModelConfigToOptionsAdaptor {
+public:
+
+	TranslationModelConfigToOptionsAdaptor();
+
+	~TranslationModelConfigToOptionsAdaptor();
+
+	/* Create an Options object from the translation model configuration object.
+	 */
+	std::shared_ptr<marian::Options> adapt(const TranslationModelConfiguration& config);
+};
+
+#endif /* SRC_TRANSLATOR_TRANSLATIONMODELCONFIGTOOPTIONSADAPTOR_H_ */

From cd505c9286a8d48a2f5b5e91106b5073801b6b40 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 16 Nov 2020 13:09:42 +0100
Subject: [PATCH 014/442] Updated README with 'Build' and 'Use' instructions

---
 README.md | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/README.md b/README.md
index dd3798232..fbbbe7b46 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,17 @@
 # Bergamot Translator
 
 Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.github.io/) framework based) neural machine translation functionality in accordance with the [Bergamot](https://browser.mt/) project that focuses on improving client-side machine translation in a web browser.
+
+## Build Instructions
+```
+$ git clone https://github.com/browsermt/bergamot-translator
+$ cd bergamot-translator
+$ mkdir build
+$ cd build
+$ cmake ../
+$ make -j
+
+```
+
+## Using Bergamot Translator
+The build will generate the library that can be linked to any project. All the public header files are specified in `src` folder.

From 9478a54628eae05fc3abd4c7fcb1104ce424c713 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 16 Nov 2020 15:14:50 +0100
Subject: [PATCH 015/442] Improved 3rd party header inclusion

 - Inclusion now contains explicit names of the 3rd party
   libraries
---
 src/translator/AbstractTranslationModel.cpp             | 2 +-
 src/translator/CMakeLists.txt                           | 1 +
 src/translator/TranslationModel.h                       | 2 +-
 src/translator/TranslationModelConfigToOptionsAdaptor.h | 2 +-
 4 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
index afd62e7ec..597c592d3 100644
--- a/src/translator/AbstractTranslationModel.cpp
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -5,7 +5,7 @@
 #include <memory>
 
 // All 3rd party includes
-#include "common/options.h"
+#include "3rd_party/marian-dev/src/common/options.h"
 
 // All local includes
 #include "AbstractTranslationModel.h"
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index c9a51df45..08a82fcb5 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -7,4 +7,5 @@ target_link_libraries(bergamot-translator marian)
 
 target_include_directories(bergamot-translator
     PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}
+    PRIVATE ${CMAKE_SOURCE_DIR}
     PUBLIC ${CMAKE_SOURCE_DIR}/src)
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index 5b75b8fec..587926516 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -12,7 +12,7 @@
 #include <future>
 
 // All 3rd party includes
-#include "common/options.h"
+#include "3rd_party/marian-dev/src/common/options.h"
 
 // All local project includes
 #include "AbstractTranslationModel.h"
diff --git a/src/translator/TranslationModelConfigToOptionsAdaptor.h b/src/translator/TranslationModelConfigToOptionsAdaptor.h
index 309ea69c8..1eba4cced 100644
--- a/src/translator/TranslationModelConfigToOptionsAdaptor.h
+++ b/src/translator/TranslationModelConfigToOptionsAdaptor.h
@@ -11,7 +11,7 @@
 #include <memory>
 
 // All 3rd party includes
-#include "common/options.h"
+#include "3rd_party/marian-dev/src/common/options.h"
 
 // All local includes
 #include "TranslationModelConfiguration.h"

From f8c9a6b0cce845ab8b6e1fe929cb7a6b260c72a4 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 16 Nov 2020 13:17:53 +0100
Subject: [PATCH 016/442] Added an application showing usage of bergamot
 translator

 - 'app' folder contains the application
 - The application uses dummy requests and responses for now
---
 CMakeLists.txt     |  1 +
 app/CMakeLists.txt |  3 +++
 app/main.cpp       | 35 +++++++++++++++++++++++++++++++++++
 3 files changed, 39 insertions(+)
 create mode 100644 app/CMakeLists.txt
 create mode 100644 app/main.cpp

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 6aaff4ee6..68a075d5c 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -17,3 +17,4 @@ option(USE_STATIC_LIBS "Link statically against non-system libs" ON)
 
 add_subdirectory(3rd_party)
 add_subdirectory(src)
+add_subdirectory(app)
diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
new file mode 100644
index 000000000..f9698dc55
--- /dev/null
+++ b/app/CMakeLists.txt
@@ -0,0 +1,3 @@
+add_executable(bergamot-translator-app main.cpp)
+
+target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
diff --git a/app/main.cpp b/app/main.cpp
new file mode 100644
index 000000000..dc808228f
--- /dev/null
+++ b/app/main.cpp
@@ -0,0 +1,35 @@
+/*
+ * main.cpp
+ *
+ * An example application to demonstrate the use of Bergamot translator.
+ *
+ */
+
+#include <iostream>
+
+#include "TranslationModelConfiguration.h"
+#include "AbstractTranslationModel.h"
+#include "TranslationRequest.h"
+#include "TranslationResult.h"
+
+
+int main(int argc, char** argv) {
+
+	// Create an instance of AbstractTranslationModel with a dummy model configuration
+	TranslationModelConfiguration config("dummy_modelFilePath",
+				"dummy_sourceVocabPath",
+				"dummy_targetVocabPath");
+	std::shared_ptr<AbstractTranslationModel> model =
+			AbstractTranslationModel::createInstance(config);
+
+	// Call to translate a dummy (empty) texts with a dummy (empty) translation request
+	TranslationRequest req;
+	std::vector<std::string> texts;
+	auto result = model->translate(std::move(texts), req);
+
+	// Resolve the future and get the actual result
+	std::vector<TranslationResult> res = result.get();
+
+	std::cout << "Count is: " << res.size() << std::endl;
+	return 0;
+}

From 601bd527168d6f0cdef00b5226b912ffc59a5f69 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 20 Jan 2021 19:08:46 +0000
Subject: [PATCH 017/442] Import sources from mts adaptation

This first commit imports files from  mts which was repurposed for bergamot translator
from https://github.com/browsermt/mts/tree/nuke.
---
 src/translator/batch_translator.cpp     | 123 ++++++++++
 src/translator/batch_translator.h       |  51 ++++
 src/translator/batcher.cpp              |  54 +++++
 src/translator/batcher.h                |  35 +++
 src/translator/definitions.h            |  27 +++
 src/translator/main.cpp                 |  92 ++++++++
 src/translator/multifactor_priority.cpp |   7 +
 src/translator/multifactor_priority.h   |  20 ++
 src/translator/pcqueue.h                | 299 ++++++++++++++++++++++++
 src/translator/request.cpp              |  93 ++++++++
 src/translator/request.h                | 114 +++++++++
 src/translator/sanelogging.h            |  44 ++++
 src/translator/service.cpp              |  99 ++++++++
 src/translator/service.h                |  44 ++++
 src/translator/textops.cpp              | 135 +++++++++++
 src/translator/textops.h                | 102 ++++++++
 src/translator/timer.h                  |  32 +++
 src/translator/translation_result.cpp   |  97 ++++++++
 src/translator/translation_result.h     |  64 +++++
 src/translator/utils.cpp                |  31 +++
 src/translator/utils.h                  |  20 ++
 21 files changed, 1583 insertions(+)
 create mode 100644 src/translator/batch_translator.cpp
 create mode 100644 src/translator/batch_translator.h
 create mode 100644 src/translator/batcher.cpp
 create mode 100644 src/translator/batcher.h
 create mode 100644 src/translator/definitions.h
 create mode 100644 src/translator/main.cpp
 create mode 100644 src/translator/multifactor_priority.cpp
 create mode 100644 src/translator/multifactor_priority.h
 create mode 100644 src/translator/pcqueue.h
 create mode 100644 src/translator/request.cpp
 create mode 100644 src/translator/request.h
 create mode 100644 src/translator/sanelogging.h
 create mode 100644 src/translator/service.cpp
 create mode 100644 src/translator/service.h
 create mode 100644 src/translator/textops.cpp
 create mode 100644 src/translator/textops.h
 create mode 100644 src/translator/timer.h
 create mode 100644 src/translator/translation_result.cpp
 create mode 100644 src/translator/translation_result.h
 create mode 100644 src/translator/utils.cpp
 create mode 100644 src/translator/utils.h

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
new file mode 100644
index 000000000..f41fa590f
--- /dev/null
+++ b/src/translator/batch_translator.cpp
@@ -0,0 +1,123 @@
+#include "batch_translator.h"
+#include "common/logging.h"
+#include "data/corpus.h"
+#include "data/text_input.h"
+#include "sanelogging.h"
+#include "translator/beam_search.h"
+#include "utils.h"
+
+namespace marian {
+namespace bergamot {
+
+BatchTranslator::BatchTranslator(DeviceId const device,
+                                 PCQueue<PCItem> &pcqueue, Ptr<Options> options)
+    : device_(device), options_(options), pcqueue_(&pcqueue) {
+
+  thread_ = std::thread([this] { this->mainloop(); });
+}
+
+void BatchTranslator::initGraph() {
+  vocabs_ = loadVocabularies(options_);
+  if (options_->hasAndNotEmpty("shortlist")) {
+    Ptr<data::ShortlistGenerator const> slgen;
+    int srcIdx = 0, trgIdx = 1;
+    bool shared_vcb = vocabs_.front() == vocabs_.back();
+    slgen_ = New<data::LexicalShortlistGenerator>(
+        options_, vocabs_.front(), vocabs_.back(), srcIdx, trgIdx, shared_vcb);
+  }
+
+  graph_ = New<ExpressionGraph>(true); // always optimize
+  auto prec = options_->get<std::vector<std::string>>("precision", {"float32"});
+  graph_->setDefaultElementType(typeFromString(prec[0]));
+  graph_->setDevice(device_);
+  graph_->getBackend()->configureDevice(options_);
+  graph_->reserveWorkspaceMB(options_->get<size_t>("workspace"));
+  scorers_ = createScorers(options_);
+  for (auto scorer : scorers_) {
+    scorer->init(graph_);
+    if (slgen_) {
+      scorer->setShortlistGenerator(slgen_);
+    }
+  }
+
+  graph_->forward();
+}
+
+void BatchTranslator::translate(RequestSentences &requestSentences,
+                                Histories &histories) {
+  std::vector<data::SentenceTuple> batchVector;
+
+  for (auto &sentence : requestSentences) {
+    data::SentenceTuple sentence_tuple(sentence.lineNumber());
+    Segment segment = sentence.getUnderlyingSegment();
+    sentence_tuple.push_back(segment);
+    batchVector.push_back(sentence_tuple);
+  }
+
+  size_t batchSize = batchVector.size();
+  std::vector<size_t> sentenceIds;
+  std::vector<int> maxDims;
+  for (auto &ex : batchVector) {
+    if (maxDims.size() < ex.size())
+      maxDims.resize(ex.size(), 0);
+    for (size_t i = 0; i < ex.size(); ++i) {
+      if (ex[i].size() > (size_t)maxDims[i])
+        maxDims[i] = (int)ex[i].size();
+    }
+    sentenceIds.push_back(ex.getId());
+  }
+
+  typedef marian::data::SubBatch SubBatch;
+  typedef marian::data::CorpusBatch CorpusBatch;
+
+  std::vector<Ptr<SubBatch>> subBatches;
+  for (size_t j = 0; j < maxDims.size(); ++j) {
+    subBatches.emplace_back(New<SubBatch>(batchSize, maxDims[j], vocabs_[j]));
+  }
+
+  std::vector<size_t> words(maxDims.size(), 0);
+  for (size_t i = 0; i < batchSize; ++i) {
+    for (size_t j = 0; j < maxDims.size(); ++j) {
+      for (size_t k = 0; k < batchVector[i][j].size(); ++k) {
+        subBatches[j]->data()[k * batchSize + i] = batchVector[i][j][k];
+        subBatches[j]->mask()[k * batchSize + i] = 1.f;
+        words[j]++;
+      }
+    }
+  }
+
+  for (size_t j = 0; j < maxDims.size(); ++j)
+    subBatches[j]->setWords(words[j]);
+
+  auto batch = Ptr<CorpusBatch>(new CorpusBatch(subBatches));
+  batch->setSentenceIds(sentenceIds);
+
+  auto trgVocab = vocabs_.back();
+  auto search = New<BeamSearch>(options_, scorers_, trgVocab);
+
+  histories = std::move(search->search(graph_, batch));
+}
+
+void BatchTranslator::mainloop() {
+  initGraph();
+
+  PCItem pcitem;
+  Histories histories;
+
+  while (true) {
+    pcqueue_->ConsumeSwap(pcitem);
+    if (pcitem.isPoison()) {
+      return;
+    } else {
+      translate(pcitem.sentences, histories);
+      for (int i = 0; i < pcitem.sentences.size(); i++) {
+        pcitem.sentences[i].completeSentence(histories[i]);
+      }
+    }
+  }
+}
+
+void BatchTranslator::join() { thread_.join(); }
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
new file mode 100644
index 000000000..638a1a971
--- /dev/null
+++ b/src/translator/batch_translator.h
@@ -0,0 +1,51 @@
+#ifndef SRC_BERGAMOT_BATCH_TRANSLATOR_H_
+#define SRC_BERGAMOT_BATCH_TRANSLATOR_H_
+
+#include <string>
+#include <vector>
+
+#include "common/utils.h"
+#include "data/shortlist.h"
+#include "definitions.h"
+#include "pcqueue.h"
+#include "request.h"
+#include "translator/history.h"
+#include "translator/scorers.h"
+
+namespace marian {
+namespace bergamot {
+
+class BatchTranslator {
+  // Launches minimal marian-translation (only CPU at the moment) in individual
+  // threads. Constructor launches each worker thread running mainloop().
+  // mainloop runs until until it receives poison from the PCQueue. Threads are
+  // shut down in Service which calls join() on the threads.
+
+public:
+  BatchTranslator(DeviceId const device, PCQueue<PCItem> &pcqueue,
+                  Ptr<Options> options);
+  void join();
+
+  // convenience function for logging. TODO(jerin)
+  std::string _identifier() { return "worker" + std::to_string(device_.no); }
+
+private:
+  void initGraph();
+  void translate(RequestSentences &requestSentences, Histories &histories);
+  void mainloop();
+
+  Ptr<Options> options_;
+
+  DeviceId device_;
+  std::vector<Ptr<Vocab const>> vocabs_;
+  Ptr<ExpressionGraph> graph_;
+  std::vector<Ptr<Scorer>> scorers_;
+  Ptr<data::ShortlistGenerator const> slgen_;
+
+  PCQueue<PCItem> *pcqueue_;
+  std::thread thread_;
+};
+} // namespace bergamot
+} // namespace marian
+
+#endif //  SRC_BERGAMOT_BATCH_TRANSLATOR_H_
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
new file mode 100644
index 000000000..471263df9
--- /dev/null
+++ b/src/translator/batcher.cpp
@@ -0,0 +1,54 @@
+#include "batcher.h"
+#include "common/logging.h"
+#include "sanelogging.h"
+#include <cassert>
+
+namespace marian {
+namespace bergamot {
+
+Batcher::Batcher(Ptr<Options> options) {
+  max_input_tokens_ = options->get<int>("max-input-tokens");
+  bucket_.resize(options->get<int>("max-input-sentence-tokens") + 1);
+  ABORT_IF(max_input_tokens_ >= bucket_.size(),
+           "max-input-sentence-tokens cannot be greater than max-input-tokens, "
+           "batcher fail");
+}
+
+void Batcher::addSentenceWithPriority(RequestSentence &sentence) {
+  int bucket_id = sentence.numTokens();
+  assert(bucket_id < bucket_.size());
+  bucket_[bucket_id].insert(sentence);
+}
+
+void Batcher::cleaveBatch(RequestSentences &sentences) {
+  // For now simply iterates on buckets and converts batches greedily.  This
+  // has to be enhanced with optimizing over priority. The baseline
+  // implementation should at least be as fast as marian's maxi-batch with full
+  // corpus size as maxi-batch size.
+
+  int segments_added = 0;
+  int current_input_tokens = 0;
+  int padded_batch_size = 0;
+  int prev_padded_batch_size;
+
+  for (int i = 0; i < bucket_.size(); i++) {
+    auto p = bucket_[i].begin();
+    while (p != bucket_[i].end()) {
+      padded_batch_size = (segments_added + 1) * i;
+      if (padded_batch_size <= max_input_tokens_) {
+        auto q = p;
+        ++p;
+        current_input_tokens += i;
+        sentences.push_back(*q);
+        ++segments_added;
+        bucket_[i].erase(q);
+        prev_padded_batch_size = padded_batch_size;
+      } else {
+        return;
+      }
+    }
+  }
+}
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
new file mode 100644
index 000000000..b60b642c7
--- /dev/null
+++ b/src/translator/batcher.h
@@ -0,0 +1,35 @@
+#ifndef SRC_BERGAMOT_BATCHER_H_
+#define SRC_BERGAMOT_BATCHER_H_
+
+#include "common/options.h"
+#include "data/corpus_base.h"
+#include "definitions.h"
+#include "request.h"
+
+#include <set>
+#include <vector>
+
+namespace marian {
+namespace bergamot {
+class Batcher {
+public:
+  explicit Batcher(Ptr<Options> options);
+
+  // RequestSentence incorporates (tentative) notions of priority with each
+  // sentence. This method inserts the sentence into the internal data-structure
+  // which maintains priority among sentences from multiple concurrent requests.
+  void addSentenceWithPriority(RequestSentence &sentence);
+
+  // Loads sentences with sentences compiled from (tentatively) multiple
+  // requests optimizing for both padding and priority.
+  void cleaveBatch(RequestSentences &sentences);
+
+private:
+  unsigned int max_input_tokens_;
+  std::vector<std::set<RequestSentence>> bucket_;
+};
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_BATCHER_H_
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
new file mode 100644
index 000000000..35797a2b4
--- /dev/null
+++ b/src/translator/definitions.h
@@ -0,0 +1,27 @@
+#ifndef SRC_BERGAMOT_DEFINITIONS_H_
+#define SRC_BERGAMOT_DEFINITIONS_H_
+
+#include "data/types.h"
+#include "data/vocab_base.h"
+#include <vector>
+
+namespace marian {
+namespace bergamot {
+
+typedef marian::Words Segment;
+typedef std::vector<Segment> Segments;
+typedef std::vector<marian::string_view> TokenRanges;
+typedef std::vector<TokenRanges> SentenceTokenRanges;
+
+/** @brief Creates unique_ptr any type, passes all arguments to any available
+ *  * constructor */
+template <class T, typename... Args> UPtr<T> UNew(Args &&... args) {
+  return UPtr<T>(new T(std::forward<Args>(args)...));
+}
+
+template <class T> UPtr<T> UNew(UPtr<T> p) { return UPtr<T>(p); }
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_DEFINITIONS_H_
diff --git a/src/translator/main.cpp b/src/translator/main.cpp
new file mode 100644
index 000000000..b3fb3f116
--- /dev/null
+++ b/src/translator/main.cpp
@@ -0,0 +1,92 @@
+#include <cstdlib>
+#include <iostream>
+#include <sstream>
+
+#include "common/definitions.h"
+#include "common/timer.h"
+#include "common/utils.h"
+#include "marian.h"
+#include "translator/history.h"
+#include "translator/output_collector.h"
+#include "translator/output_printer.h"
+
+#include "service.h"
+
+void marian_decoder_minimal(const marian::Histories &histories,
+                            marian::Ptr<marian::Vocab const> targetVocab,
+                            marian::Ptr<marian::Options> options) {
+
+  bool doNbest = options->get<bool>("n-best");
+
+  auto collector =
+      marian::New<marian::OutputCollector>(options->get<std::string>("output"));
+
+  // There is a dependency of vocabs here.
+  auto printer = marian::New<marian::OutputPrinter>(options, targetVocab);
+  if (options->get<bool>("quiet-translation"))
+    collector->setPrintingStrategy(marian::New<marian::QuietPrinting>());
+
+  for (auto &history : histories) {
+    std::stringstream best1;
+    std::stringstream bestn;
+    printer->print(history, best1, bestn);
+    collector->Write((long)history->getLineNum(), best1.str(), bestn.str(),
+                     doNbest);
+  }
+}
+
+int main(int argc, char *argv[]) {
+  marian::ConfigParser cp(marian::cli::mode::translation);
+
+  cp.addOption<std::string>(
+      "--ssplit-prefix-file", "Bergamot Options",
+      "File with nonbreaking prefixes for sentence splitting.");
+
+  cp.addOption<std::string>("--ssplit-mode", "Server Options",
+                            "[paragraph, sentence, wrapped_text]");
+
+  cp.addOption<int>(
+      "--max-input-sentence-tokens", "Bergamot Options",
+      "Maximum input tokens to be processed in a single sentence.", 128);
+
+  cp.addOption<int>("--max-input-tokens", "Bergamot Options",
+                    "Maximum input tokens in a batch. control for"
+                    "Bergamot Queue",
+                    1024);
+
+  cp.addOption<int>("--nbest", "Bergamot Options",
+                    "NBest value used for decoding", 1);
+
+  cp.addOption<bool>("--marian-decoder-alpha", "Bergamot Options",
+                     "Run marian-decoder output printer code", false);
+
+  // TODO(jerin): Add QE later.
+  // marian::qe::QualityEstimator::addOptions(cp);
+
+  marian::timer::Timer decoderTimer;
+
+  auto options = cp.parseOptions(argc, argv, true);
+  marian::bergamot::Service service(options);
+
+  std::ostringstream std_input;
+  std_input << std::cin.rdbuf();
+  std::string input = std_input.str();
+
+  LOG(info, "IO complete Translating input");
+  auto translation_result_future = service.translate(std::move(input));
+  translation_result_future.wait();
+  auto translation_result = translation_result_future.get();
+  if (options->get<bool>("marian-decoder-alpha")) {
+    marian_decoder_minimal(translation_result.getHistories(),
+                           service.targetVocab(), options);
+    LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
+  } else {
+    for (auto &p : translation_result.getSentenceMappings()) {
+      std::cout << "[src] " << p.first << "\n";
+      std::cout << "[tgt] " << p.second << "\n";
+    }
+  }
+
+  service.stop();
+  return 0;
+}
diff --git a/src/translator/multifactor_priority.cpp b/src/translator/multifactor_priority.cpp
new file mode 100644
index 000000000..0f93a8148
--- /dev/null
+++ b/src/translator/multifactor_priority.cpp
@@ -0,0 +1,7 @@
+#include "multifactor_priority.h"
+
+namespace marian {
+namespace bergamot {
+
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/multifactor_priority.h b/src/translator/multifactor_priority.h
new file mode 100644
index 000000000..1e239f73b
--- /dev/null
+++ b/src/translator/multifactor_priority.h
@@ -0,0 +1,20 @@
+#ifndef SRC_BERGAMOT_MULTIFACTOR_PRIORITY_H_
+#define SRC_BERGAMOT_MULTIFACTOR_PRIORITY_H_
+
+#include "data/types.h"
+#include "definitions.h"
+#include "sys/time.h"
+
+namespace marian {
+namespace bergamot {
+
+struct MultiFactorPriority {
+  int nice; /* user configurable priority, at a request */
+  unsigned int Id;
+  /* What else should priority depend on? */
+  double priority() { return Id; }
+};
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_MULTIFACTOR_PRIORITY_H_
diff --git a/src/translator/pcqueue.h b/src/translator/pcqueue.h
new file mode 100644
index 000000000..512932560
--- /dev/null
+++ b/src/translator/pcqueue.h
@@ -0,0 +1,299 @@
+#ifndef SRC_BERGAMOT_PCQUEUE_H_
+#define SRC_BERGAMOT_PCQUEUE_H_
+
+#include "common/logging.h"
+
+#include <algorithm>
+#include <cerrno>
+#include <iostream>
+#include <memory>
+#include <mutex>
+
+#ifdef __APPLE__
+#include <mach/mach.h>
+#include <mach/mach_traps.h>
+#include <mach/semaphore.h>
+#include <mach/task.h>
+#elif defined(__linux)
+#include <semaphore.h>
+#else
+#include <boost/interprocess/sync/interprocess_semaphore.hpp>
+#endif
+
+#if __GNUC__ >= 3
+#define UTIL_UNLIKELY(x) __builtin_expect(!!(x), 0)
+#else
+#define UTIL_UNLIKELY(x) (x)
+#endif
+
+namespace marian {
+namespace bergamot {
+
+/* OS X Maverick and Boost interprocess were doing "Function not implemented."
+ * So this is my own wrapper around the mach kernel APIs.
+ */
+#ifdef __APPLE__
+
+class Semaphore {
+public:
+  explicit Semaphore(int value) : task_(mach_task_self()) {
+    ABORT_IF(KERN_SUCCESS !=
+                 semaphore_create(task_, &back_, SYNC_POLICY_FIFO, value),
+             "Could not create semaphore");
+  }
+
+  ~Semaphore() {
+    if (KERN_SUCCESS != semaphore_destroy(task_, back_)) {
+      std::cerr << "Could not destroy semaphore" << std::endl;
+      abort();
+    }
+  }
+
+  void wait() {
+    ABORT_IF(KERN_SUCCESS != semaphore_wait(back_), Exception,
+             "Wait for semaphore failed");
+  }
+
+  void post() {
+    ABORT_IF(KERN_SUCCESS != semaphore_signal(back_), Exception,
+             "Could not post to semaphore");
+  }
+
+private:
+  semaphore_t back_;
+  task_t task_;
+};
+
+inline void WaitSemaphore(Semaphore &semaphore) { semaphore.wait(); }
+
+#elif defined(__linux)
+
+class Semaphore {
+public:
+  explicit Semaphore(unsigned int value) {
+    ABORT_IF(sem_init(&sem_, 0, value), "Could not create semaphore");
+  }
+
+  ~Semaphore() {
+    if (-1 == sem_destroy(&sem_)) {
+      std::cerr << "Could not destroy semaphore " << std::endl;
+      abort();
+    }
+  }
+
+  void wait() {
+    while (UTIL_UNLIKELY(-1 == sem_wait(&sem_))) {
+      ABORT_IF(errno != EINTR, "Wait for semaphore failed");
+    }
+  }
+
+  void post() {
+    ABORT_IF(-1 == sem_post(&sem_), "Could not post to semaphore");
+  }
+
+private:
+  sem_t sem_;
+};
+
+inline void WaitSemaphore(Semaphore &semaphore) { semaphore.wait(); }
+
+#else
+typedef boost::interprocess::interprocess_semaphore Semaphore;
+
+inline void WaitSemaphore(Semaphore &on) {
+  while (1) {
+    try {
+      on.wait();
+      break;
+    } catch (boost::interprocess::interprocess_exception &e) {
+      if (e.get_native_error() != EINTR) {
+        throw;
+      }
+    }
+  }
+}
+
+#endif // Apple
+
+/**
+ * Producer consumer queue safe for multiple producers and multiple consumers.
+ * T must be default constructable and have operator=.
+ * The value is copied twice for Consume(T &out) or three times for Consume(),
+ * so larger objects should be passed via pointer.
+ * Strong exception guarantee if operator= throws.  Undefined if semaphores
+ * throw.
+ */
+template <class T> class PCQueue {
+public:
+  explicit PCQueue(size_t size)
+      : empty_(size), used_(0), storage_(new T[size]),
+        end_(storage_.get() + size), produce_at_(storage_.get()),
+        consume_at_(storage_.get()) {}
+
+  // Add a value to the queue.
+  void Produce(const T &val) {
+    WaitSemaphore(empty_);
+    {
+      std::lock_guard<std::mutex> produce_lock(produce_at_mutex_);
+      try {
+        *produce_at_ = val;
+      } catch (...) {
+        empty_.post();
+        throw;
+      }
+      if (++produce_at_ == end_)
+        produce_at_ = storage_.get();
+    }
+    used_.post();
+  }
+
+  // Add a value to the queue, but swap it into place.
+  void ProduceSwap(T &val) {
+    WaitSemaphore(empty_);
+    {
+      std::lock_guard<std::mutex> produce_lock(produce_at_mutex_);
+      try {
+        std::swap(*produce_at_, val);
+      } catch (...) {
+        empty_.post();
+        throw;
+      }
+      if (++produce_at_ == end_)
+        produce_at_ = storage_.get();
+    }
+    used_.post();
+  }
+
+  // Consume a value, assigning it to out.
+  T &Consume(T &out) {
+    WaitSemaphore(used_);
+    {
+      std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
+      try {
+        out = *consume_at_;
+      } catch (...) {
+        used_.post();
+        throw;
+      }
+      if (++consume_at_ == end_)
+        consume_at_ = storage_.get();
+    }
+    empty_.post();
+    return out;
+  }
+
+  // Consume a value, swapping it to out.
+  T &ConsumeSwap(T &out) {
+    WaitSemaphore(used_);
+    {
+      std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
+      try {
+        std::swap(out, *consume_at_);
+      } catch (...) {
+        used_.post();
+        throw;
+      }
+      if (++consume_at_ == end_)
+        consume_at_ = storage_.get();
+    }
+    empty_.post();
+    return out;
+  }
+
+  // Convenience version of Consume that copies the value to return.
+  // The other version is faster.
+  T Consume() {
+    T ret;
+    Consume(ret);
+    return ret;
+  }
+
+private:
+  // Number of empty spaces in storage_.
+  Semaphore empty_;
+  // Number of occupied spaces in storage_.
+  Semaphore used_;
+
+  std::unique_ptr<T[]> storage_;
+
+  T *const end_;
+
+  // Index for next write in storage_.
+  T *produce_at_;
+  std::mutex produce_at_mutex_;
+
+  // Index for next read from storage_.
+  T *consume_at_;
+  std::mutex consume_at_mutex_;
+};
+
+template <class T> struct UnboundedPage {
+  UnboundedPage() : next(nullptr) {}
+  UnboundedPage *next;
+  T entries[1023];
+};
+
+template <class T> class UnboundedSingleQueue {
+public:
+  UnboundedSingleQueue() : valid_(0) {
+    SetFilling(new UnboundedPage<T>());
+    SetReading(filling_);
+  }
+
+  void Produce(T &&val) {
+    if (filling_current_ == filling_end_) {
+      UnboundedPage<T> *next = new UnboundedPage<T>();
+      filling_->next = next;
+      SetFilling(next);
+    }
+    *(filling_current_++) = std::move(val);
+    valid_.post();
+  }
+
+  void Produce(const T &val) { Produce(T(val)); }
+
+  T &Consume(T &out) {
+    WaitSemaphore(valid_);
+    if (reading_current_ == reading_end_) {
+      SetReading(reading_->next);
+    }
+    out = std::move(*(reading_current_++));
+    return out;
+  }
+
+  // Warning: very much a no-guarantees race-condition-rich implementation!
+  // But sufficient for our specific purpose: The single thread that consumes
+  // is also the only one that checks Empty, and knows that it's racing.
+  bool Empty() const { return reading_current_ == filling_current_; }
+
+private:
+  void SetFilling(UnboundedPage<T> *to) {
+    filling_ = to;
+    filling_current_ = to->entries;
+    filling_end_ = filling_current_ + sizeof(to->entries) / sizeof(T);
+  }
+  void SetReading(UnboundedPage<T> *to) {
+    reading_.reset(to);
+    reading_current_ = to->entries;
+    reading_end_ = reading_current_ + sizeof(to->entries) / sizeof(T);
+  }
+
+  Semaphore valid_;
+
+  UnboundedPage<T> *filling_;
+
+  std::unique_ptr<UnboundedPage<T>> reading_;
+
+  T *filling_current_;
+  T *filling_end_;
+  T *reading_current_;
+  T *reading_end_;
+
+  UnboundedSingleQueue(const UnboundedSingleQueue &) = delete;
+  UnboundedSingleQueue &operator=(const UnboundedSingleQueue &) = delete;
+};
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_PCQUEUE_H_
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
new file mode 100644
index 000000000..0d02c03ac
--- /dev/null
+++ b/src/translator/request.cpp
@@ -0,0 +1,93 @@
+#include "request.h"
+
+#include "definitions.h"
+#include "translation_result.h"
+
+#include "common/logging.h"
+
+#include <string>
+
+namespace marian {
+namespace bergamot {
+
+Request::Request(unsigned int Id, int lineNumberBegin,
+                 std::vector<Ptr<Vocab const>> &vocabs, std::string &&source,
+                 Segments &&segments,
+                 std::vector<TokenRanges> &&sourceAlignments,
+                 std::promise<TranslationResult> translationResultPromise)
+    : Id_(Id), lineNumberBegin_(lineNumberBegin), vocabs_(&vocabs),
+      source_(std::move(source)), segments_(std::move(segments)),
+      sourceAlignments_(std::move(sourceAlignments)),
+      response_(std::move(translationResultPromise)) {
+
+  counter_ = segments_.size();
+  histories_.resize(segments_.size(), nullptr);
+}
+
+size_t Request::lineNumberBegin() const { return lineNumberBegin_; }
+size_t Request::numSegments() const { return segments_.size(); }
+
+size_t Request::segmentTokens(size_t index) const {
+  return (segments_[index].size());
+}
+
+Segment Request::getSegment(size_t index) const { return segments_[index]; }
+
+void Request::processHistory(size_t index, Ptr<History> history) {
+  // Concurrently called by multiple workers as a history from translation is
+  // ready. The container storing histories is set with the value obtained.
+  histories_[index] = history;
+
+  // In case this is last request in, completeRequest is called, which sets the
+  // value of the promise.
+  if (--counter_ == 0) {
+    completeRequest();
+  }
+}
+
+void Request::completeRequest() {
+  // Request no longer needs to hold the content, can transfer it to
+  // TranslationResult.
+  TranslationResult translation_result(std::move(source_), std::move(segments_),
+                                       std::move(sourceAlignments_),
+                                       std::move(histories_), *vocabs_);
+  LOG(info, "Last translation in. Closing request;");
+  response_.set_value(translation_result);
+}
+
+bool Request::operator<(const Request &b) const {
+  // Among Requests, only sequence id is used for obtaining priority.
+  return Id_ < b.Id_;
+}
+
+RequestSentence::RequestSentence(size_t index, Ptr<Request> request)
+    : index_(index), request_(request) {}
+
+size_t RequestSentence::numTokens() const {
+  return (request_->segmentTokens(index_));
+}
+
+size_t RequestSentence::lineNumber() const {
+  return (request_->lineNumberBegin() + index_);
+}
+
+void RequestSentence::completeSentence(Ptr<History> history) {
+  // Relays completeSentence into request's processHistory, using index
+  // information.
+  request_->processHistory(index_, history);
+}
+
+Segment RequestSentence::getUnderlyingSegment() const {
+  return request_->getSegment(index_);
+}
+
+bool operator<(const RequestSentence &a, const RequestSentence &b) {
+  // Operator overload for usage in priority-queue / set.
+  if (a.request_ == b.request_) {
+    return a.index_ < b.index_;
+  }
+  return a.request_ < b.request_;
+}
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/request.h b/src/translator/request.h
new file mode 100644
index 000000000..6f268ba1c
--- /dev/null
+++ b/src/translator/request.h
@@ -0,0 +1,114 @@
+//
+// Defines:
+//
+// Request: holds the input blob of a text, Segments (vector<Words>) which are
+// to go to the batching mechanism and alignments between the processed
+// segments and the input blob (sourceAlignments). In addition, Request takes
+// care of the barrier which fires when all the Segments in a request are done
+// translating by the workers (BatchTranslator). Request is to be extended with
+// notions of Priority (sequence, user-given).
+//
+// RequestSentence: is a tuple of (index, Request*). This provides the
+// batching mechanism access to the segment within the request. The backref to
+// Request allows event triggering the barrier upon completion of the last
+// sentence by a worker.
+//
+// PCItem: is a vector of RequestSentences and a batchNumber, which is what the
+// PCQueue holds. The batches are constructed from segments returned by a
+// RequestSentence. Can be enhanced with paddingSize, countTokens eventually for
+// logging.
+
+#ifndef SRC_BERGAMOT_REQUEST_H_
+#define SRC_BERGAMOT_REQUEST_H_
+
+#include "definitions.h"
+#include "translation_result.h"
+
+#include "data/types.h"
+#include "translator/beam_search.h"
+
+#include <future>
+#include <vector>
+
+namespace marian {
+namespace bergamot {
+
+class Request {
+private:
+  unsigned int Id_;
+  int lineNumberBegin_;
+  std::string source_;
+  std::atomic<int> counter_;
+  std::vector<Ptr<Vocab const>> *vocabs_;
+
+  Segments segments_;
+  std::vector<TokenRanges> sourceAlignments_;
+  std::vector<Ptr<History>> histories_;
+
+  std::promise<TranslationResult> response_;
+
+public:
+  Request(unsigned int Id, int lineNumberBegin,
+          std::vector<Ptr<Vocab const>> &vocabs_, std::string &&source,
+          Segments &&segments, std::vector<TokenRanges> &&sourceAlignments,
+          std::promise<TranslationResult> translationResultPromise);
+
+  // Obtain the count of tokens in the segment correponding to index. Used to
+  // insert sentence from multiple requests into the corresponding size bucket.
+  size_t segmentTokens(size_t index) const;
+
+  // Obtain number of segments in a request.
+  size_t numSegments() const;
+  size_t lineNumberBegin() const;
+
+  // Obtains segment corresponding to index  to create a batch of segments among
+  // several requests.
+  Segment getSegment(size_t index) const;
+
+  // For notions of priority among requests (used to enable <set> in Batcher).
+  bool operator<(const Request &request) const;
+
+  // Processes a history obtained after translating in a heterogenous batch
+  // compiled from requests.
+  void processHistory(size_t index, Ptr<History> history);
+
+  // On completion of last segment, sets value of the promise.
+  void completeRequest();
+};
+
+class RequestSentence {
+private:
+  size_t index_;
+  Ptr<Request> request_;
+
+public:
+  RequestSentence(size_t, Ptr<Request>);
+  size_t numTokens() const;
+  size_t lineNumber() const;
+  Segment getUnderlyingSegment() const;
+  void completeSentence(Ptr<History> history);
+  friend bool operator<(const RequestSentence &a, const RequestSentence &b);
+};
+
+typedef std::vector<RequestSentence> RequestSentences;
+
+struct PCItem {
+  int batchNumber;
+  RequestSentences sentences;
+
+  // PCItem should be default constructible for PCQueue. Default constructed
+  // element is poison.
+  PCItem() : batchNumber(-1) {}
+
+  // PCItem constructor to construct a legit PCItem.
+  explicit PCItem(int batchNumber, RequestSentences &&sentences)
+      : batchNumber(batchNumber), sentences(std::move(sentences)) {}
+
+  // Convenience function to determine poison.
+  bool isPoison() { return (batchNumber == -1); }
+};
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_REQUEST_H_
diff --git a/src/translator/sanelogging.h b/src/translator/sanelogging.h
new file mode 100644
index 000000000..21f70dda8
--- /dev/null
+++ b/src/translator/sanelogging.h
@@ -0,0 +1,44 @@
+#ifndef SRC_BERGAMOT_SANELOGGING_H_
+#define SRC_BERGAMOT_SANELOGGING_H_
+
+#include "spdlog/spdlog.h"
+#include <iostream>
+
+namespace marian {
+
+#define PLOG(worker, level, ...)
+#define _PLOG(worker, level, ...) checkedPLog(worker, #level, __VA_ARGS__)
+
+template <class... Args>
+void checkedPLog(std::string logger, std::string level, Args... args) {
+  Logger log = spdlog::get(logger);
+  if (!log) {
+    try {
+      log = spdlog::daily_logger_st(logger, "logs/" + logger + ".log");
+    } catch (const spdlog::spdlog_ex &ex) {
+      std::cout << "Log initialization failed: " << ex.what() << std::endl;
+    }
+  }
+
+  if (level == "trace")
+    log->trace(args...);
+  else if (level == "debug")
+    log->debug(args...);
+  else if (level == "info")
+    log->info(args...);
+  else if (level == "warn")
+    log->warn(args...);
+  else if (level == "error")
+    log->error(args...);
+  else if (level == "critical")
+    log->critical(args...);
+  else {
+    log->warn("Unknown log level '{}' for logger '{}'", level, logger);
+  }
+  // Not required when threads clean-exit.
+  log->flush();
+}
+
+} // namespace marian
+
+#endif // SRC_BERGAMOT_SANELOGGING_H_
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
new file mode 100644
index 000000000..c9260812d
--- /dev/null
+++ b/src/translator/service.cpp
@@ -0,0 +1,99 @@
+#include "service.h"
+#include "definitions.h"
+#include "sanelogging.h"
+
+#include "utils.h"
+#include <string>
+#include <utility>
+
+namespace marian {
+namespace bergamot {
+
+Service::Service(Ptr<Options> options)
+    : requestId_(0), batchNumber_(0),
+      numWorkers_(options->get<int>("cpu-threads")), text_processor_(options),
+      batcher_(options), pcqueue_(2 * options->get<int>("cpu-threads")) {
+
+  vocabs_ = loadVocabularies(options);
+  workers_.reserve(numWorkers_);
+
+  for (int i = 0; i < numWorkers_; i++) {
+    marian::DeviceId deviceId(i, DeviceType::cpu);
+    workers_.emplace_back(deviceId, pcqueue_, options);
+  }
+}
+
+std::future<TranslationResult> Service::translateWithCopy(std::string input) {
+  return translate(std::move(input));
+}
+
+std::future<TranslationResult> Service::translate(std::string &&input) {
+  // Takes in a blob of text. Segments and std::vector<TokenRanges> are
+  // extracted from the input (blob of text) and used to construct a Request
+  // along with a promise. promise value is set by the worker completing a
+  // request.
+  //
+  // Batcher, which currently runs on the main thread constructs batches out of
+  // a single request (at the moment) and adds them into a Producer-Consumer
+  // queue holding a bunch of requestSentences used to construct batches.
+  // TODO(jerin): Make asynchronous and compile from multiple requests.
+  //
+  // returns future corresponding to the promise.
+
+  Segments segments;
+  std::vector<TokenRanges> sourceAlignments;
+  text_processor_.query_to_segments(input, segments, sourceAlignments);
+
+  std::promise<TranslationResult> translationResultPromise;
+  auto future = translationResultPromise.get_future();
+
+  Ptr<Request> request = New<Request>(
+      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(input),
+      std::move(segments), std::move(sourceAlignments),
+      std::move(translationResultPromise));
+
+  for (int i = 0; i < request->numSegments(); i++) {
+    RequestSentence requestSentence(i, request);
+    batcher_.addSentenceWithPriority(requestSentence);
+  }
+
+  int numSentences;
+  do {
+    RequestSentences batchSentences;
+    batcher_.cleaveBatch(batchSentences);
+    numSentences = batchSentences.size();
+
+    if (numSentences > 0) {
+      PCItem pcitem(batchNumber_++, std::move(batchSentences));
+      pcqueue_.ProduceSwap(pcitem);
+    }
+
+    if (batchNumber_ % 500 == 0) {
+      LOG(info, "Queuing batch {}", batchNumber_);
+    }
+  } while (numSentences > 0);
+
+  return future;
+}
+
+void Service::stop() {
+  int counter = 0;
+  for (auto &worker : workers_) {
+    PCItem pcitem;
+    pcqueue_.ProduceSwap(pcitem);
+    ++counter;
+  }
+
+  counter = 0;
+  for (auto &worker : workers_) {
+    worker.join();
+    ++counter;
+  }
+
+  workers_.clear(); // Takes care of idempotency.
+}
+
+Service::~Service() { stop(); }
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/service.h b/src/translator/service.h
new file mode 100644
index 000000000..519975445
--- /dev/null
+++ b/src/translator/service.h
@@ -0,0 +1,44 @@
+#ifndef SRC_BERGAMOT_SERVICE_H_
+#define SRC_BERGAMOT_SERVICE_H_
+
+#include "batch_translator.h"
+#include "batcher.h"
+#include "pcqueue.h"
+#include "textops.h"
+#include "translation_result.h"
+
+#include <queue>
+#include <vector>
+
+#include "data/types.h"
+
+namespace marian {
+namespace bergamot {
+
+class Service {
+public:
+  explicit Service(Ptr<Options> options);
+  std::future<TranslationResult> translateWithCopy(std::string input);
+  std::future<TranslationResult> translate(std::string &&input);
+  void stop();
+  Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); };
+  Ptr<Vocab const> targetVocab() const { return vocabs_.back(); };
+  ;
+  ~Service();
+
+private:
+  unsigned int requestId_;
+  unsigned int batchNumber_;
+  int numWorkers_;
+
+  std::vector<Ptr<Vocab const>> vocabs_;
+  TextProcessor text_processor_;
+  Batcher batcher_;
+  PCQueue<PCItem> pcqueue_;
+  std::vector<BatchTranslator> workers_;
+};
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_SERVICE_H_
diff --git a/src/translator/textops.cpp b/src/translator/textops.cpp
new file mode 100644
index 000000000..55f22dab8
--- /dev/null
+++ b/src/translator/textops.cpp
@@ -0,0 +1,135 @@
+#include "textops.h"
+#include "common/timer.h"
+#include "utils.h"
+#include <pcrecpp.h>
+#include <string>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+namespace marian {
+namespace bergamot {
+
+SentenceSplitter::SentenceSplitter(marian::Ptr<marian::Options> options)
+    : options_(options) {
+
+  std::string smode_str = options_->get<std::string>("ssplit-mode", "");
+  mode_ = string2splitmode(smode_str);
+  std::string ssplit_prefix_file =
+      options_->get<std::string>("ssplit-prefix-file", "");
+
+  if (ssplit_prefix_file.size()) {
+    ssplit_prefix_file = marian::cli::interpolateEnvVars(ssplit_prefix_file);
+
+    LOG(info, "Loading protected prefixes for sentence splitting from {}",
+        ssplit_prefix_file);
+
+    ssplit_.load(ssplit_prefix_file);
+  } else {
+    LOG(warn, "Missing list of protected prefixes for sentence splitting. "
+              "Set with --ssplit-prefix-file.");
+  }
+}
+
+ug::ssplit::SentenceStream
+SentenceSplitter::createSentenceStream(const string_view &input) {
+  pcrecpp::StringPiece spiece(input.begin(), input.size());
+  return std::move(ug::ssplit::SentenceStream(spiece, this->ssplit_, mode_));
+}
+
+ug::ssplit::SentenceStream::splitmode
+SentenceSplitter::string2splitmode(const std::string &m) {
+  typedef ug::ssplit::SentenceStream::splitmode splitmode;
+  // @TODO: throw Exception on error
+  if (m == "sentence" || m == "Sentence")
+    return splitmode::one_sentence_per_line;
+  if (m == "paragraph" || m == "Paragraph")
+    return splitmode::one_paragraph_per_line;
+  if (m != "wrapped_text" && m != "WrappedText" && m != "wrappedText") {
+    LOG(warn, "Ignoring unknown text input format specification: {}.", m);
+  }
+  return splitmode::wrapped_text;
+}
+
+Tokenizer::Tokenizer(Ptr<Options> options) : inference_(true), addEOS_(false) {
+  vocabs_ = loadVocabularies(options);
+}
+
+Segment Tokenizer::tokenize(const string_view &snt, TokenRanges &tokenRanges) {
+  // TODO(jerin): Bunch of hardcode here, 1, 0, need to get rid off somehow.
+  return vocabs_[0]->encodePreservingSource(snt, tokenRanges, addEOS_,
+                                            inference_);
+}
+
+TextProcessor::TextProcessor(Ptr<Options> options)
+    : tokenizer_(options), sentence_splitter_(options) {
+  max_input_sentence_tokens_ = options->get<int>("max-input-sentence-tokens");
+  max_input_sentence_tokens_ =
+      max_input_sentence_tokens_ - 1; // Account for EOS
+  // Dirty assert, should do at configparse
+  assert(max_input_sentence_tokens_ > 0);
+}
+
+void TextProcessor::query_to_segments(const string_view &query,
+                                      Segments &segments,
+                                      std::vector<TokenRanges> &sourceRanges) {
+  auto buf = sentence_splitter_.createSentenceStream(query);
+  // pcrecpp::StringPiece snt;
+  string_view snt;
+
+  int sentencesProcessed{0};
+
+  while (buf >> snt) {
+    // LOG(info, "SNT: {}", snt);
+    string_view snt_string_view(snt.data(), snt.size());
+    TokenRanges snt_alignment;
+    timer::Timer spiece_timer;
+    Segment tokenized_sentence =
+        tokenizer_.tokenize(snt_string_view, snt_alignment);
+
+    // LOG(info, "Tokenization took {:.5f} seconds", spiece_timer.elapsed());
+    if (tokenized_sentence.size() > 0) {
+      if (tokenized_sentence.size() > max_input_sentence_tokens_) {
+        int offset;
+        for (offset = 0;
+             offset + max_input_sentence_tokens_ < tokenized_sentence.size();
+             offset += max_input_sentence_tokens_) {
+          auto start = tokenized_sentence.begin() + offset;
+          Segment segment(start, start + max_input_sentence_tokens_);
+          segment.push_back(tokenizer_.sourceEosId());
+          segments.push_back(segment);
+
+          auto astart = snt_alignment.begin() + offset;
+          TokenRanges segment_alignment(astart,
+                                        astart + max_input_sentence_tokens_);
+          sourceRanges.push_back(segment_alignment);
+        }
+
+        if (offset < max_input_sentence_tokens_) {
+          auto start = tokenized_sentence.begin() + offset;
+          Segment segment(start, tokenized_sentence.end());
+          segment.push_back(tokenizer_.sourceEosId());
+          segments.push_back(segment);
+
+          auto astart = snt_alignment.begin() + offset;
+          TokenRanges segment_alignment(astart, snt_alignment.end());
+          sourceRanges.push_back(segment_alignment);
+        }
+
+      } else {
+        timer::Timer push_timer;
+        tokenized_sentence.push_back(tokenizer_.sourceEosId());
+        segments.push_back(tokenized_sentence);
+        sourceRanges.push_back(snt_alignment);
+        // LOG(info, "Push took {:.5f} seconds", push_timer.elapsed());
+      }
+    }
+    ++sentencesProcessed;
+    if (sentencesProcessed % 10000 == 0) {
+      LOG(info, "Processed {}", sentencesProcessed);
+    }
+  }
+}
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/textops.h b/src/translator/textops.h
new file mode 100644
index 000000000..0b4ee6e5c
--- /dev/null
+++ b/src/translator/textops.h
@@ -0,0 +1,102 @@
+#ifndef SRC_BERGAMOT_TEXTOPS_H_
+#define SRC_BERGAMOT_TEXTOPS_H_
+
+#include "common/definitions.h"
+#include "common/logging.h"
+#include "common/options.h"
+#include "common/types.h" // missing in shortlist.h
+#include "common/utils.h"
+#include "data/sentencepiece_vocab.h"
+#include "data/shortlist.h"
+#include "definitions.h"
+#include "ssplit/ssplit.h"
+
+#include <cassert>
+#include <iostream>
+#include <string>
+#include <vector>
+
+namespace marian {
+namespace bergamot {
+
+class StringViewStream {
+private:
+  string_view text_;
+  string_view::iterator current_;
+
+public:
+  StringViewStream(const string_view &text) : text_(text) {
+    current_ = text_.begin();
+  }
+
+  bool operator>>(string_view &sentence_view) {
+    // Skip to the next non-newline; whitespaces, anything else are okay.
+    while (current_ != text_.end() &&
+           (*current_ == '\n' || *current_ == ' ' || *current_ == '\t')) {
+      ++current_;
+    }
+
+    string_view::iterator p = current_;
+    while (p != text_.end() && *p != '\n') {
+      ++p;
+    }
+
+    if (p == current_)
+      return false;
+
+    sentence_view = string_view(current_, p - current_);
+    current_ = p;
+    return true;
+  };
+};
+
+class SentenceSplitter {
+public:
+  explicit SentenceSplitter(Ptr<Options> options);
+  ug::ssplit::SentenceStream createSentenceStream(string_view const &input);
+
+private:
+  ug::ssplit::SentenceSplitter ssplit_;
+  Ptr<Options> options_;
+  ug::ssplit::SentenceStream::splitmode mode_;
+  ug::ssplit::SentenceStream::splitmode string2splitmode(const std::string &m);
+};
+
+class LineSplitter {
+public:
+  explicit LineSplitter(Ptr<Options> options){
+      // Do nothing.
+  };
+  StringViewStream createSentenceStream(string_view const &input) {
+    return std::move(StringViewStream(input));
+  }
+};
+
+class Tokenizer {
+private:
+  std::vector<Ptr<Vocab const>> vocabs_;
+  bool inference_;
+  bool addEOS_;
+
+public:
+  explicit Tokenizer(Ptr<Options>);
+  Segment tokenize(const string_view &input, TokenRanges &tokenRanges);
+  Word sourceEosId() { return vocabs_.front()->getEosId(); };
+};
+
+class TextProcessor {
+private:
+  Tokenizer tokenizer_;
+  LineSplitter sentence_splitter_;
+  unsigned int max_input_sentence_tokens_;
+
+public:
+  explicit TextProcessor(Ptr<Options>);
+  void query_to_segments(const string_view &query, Segments &segments,
+                         std::vector<TokenRanges> &sourceRanges);
+};
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_TEXTOPS_H_
diff --git a/src/translator/timer.h b/src/translator/timer.h
new file mode 100644
index 000000000..744038081
--- /dev/null
+++ b/src/translator/timer.h
@@ -0,0 +1,32 @@
+#ifndef __BERGAMOT_TIMER_H 
+#define __BERGAMOT_TIMER_H
+
+// https://stackoverflow.com/a/19800231/4565794
+//
+// Careful: This won't work if the user changes his time between Timer() and
+// the call to elapsed() if !std::chrono::high_resolution_clock::is_steady -
+// which is the case on Linux!
+
+#include <iostream>
+#include <chrono>
+
+namespace marian {
+namespace bergamot {
+class Timer {
+public:
+    Timer() : beg_(clock_::now()) {}
+    void reset() { beg_ = clock_::now(); }
+    double elapsed() const { 
+        return std::chrono::duration_cast<second_>
+            (clock_::now() - beg_).count(); }
+
+private:
+    typedef std::chrono::high_resolution_clock clock_;
+    typedef std::chrono::duration<double, std::ratio<1> > second_;
+    std::chrono::time_point<clock_> beg_;
+};
+
+} // namespace bergamot 
+} // namespace marian
+
+#endif // __BERGAMOT_TIMER_H
diff --git a/src/translator/translation_result.cpp b/src/translator/translation_result.cpp
new file mode 100644
index 000000000..43b233eed
--- /dev/null
+++ b/src/translator/translation_result.cpp
@@ -0,0 +1,97 @@
+#include "translation_result.h"
+#include "common/logging.h"
+#include "data/alignment.h"
+
+#include <utility>
+
+namespace marian {
+namespace bergamot {
+
+TranslationResult::TranslationResult(std::string &&source, Segments &&segments,
+                                     std::vector<TokenRanges> &&sourceRanges,
+                                     Histories &&histories,
+                                     std::vector<Ptr<Vocab const>> &vocabs)
+    : source_(std::move(source)), sourceRanges_(std::move(sourceRanges)),
+      segments_(std::move(segments)), histories_(std::move(histories)),
+      vocabs_(&vocabs) {
+
+  // Process sourceMappings into sourceMappings_.
+  LOG(info, "Creating sourcemappings");
+  sourceMappings_.reserve(segments_.size());
+  for (int i = 0; i < segments_.size(); i++) {
+    string_view first = sourceRanges_[i].front();
+    string_view last = sourceRanges_[i].back();
+    int size = last.end() - first.begin();
+    sourceMappings_.emplace_back(first.data(), size);
+  }
+
+  // Compiles translations into a single std::string translation_
+  // Current implementation uses += on std::string, multiple resizes.
+  // Stores ByterRanges as indices first, followed by conversion into
+  // string_views.
+  // TODO(jerin): Add token level string_views here as well.
+  LOG(info, "Decoding");
+  std::vector<std::pair<int, int>> translationRanges;
+  int offset{0}, end{0};
+  bool first{true};
+  for (auto &history : histories_) {
+    // TODO(jerin): Change hardcode of nBest = 1
+    NBestList onebest = history->nBest(1);
+
+    Result result = onebest[0]; // Expecting only one result;
+    Words words = std::get<0>(result);
+    std::string decoded = vocabs_->back()->decode(words);
+    if (first) {
+      first = false;
+    } else {
+      translation_ += " ";
+    }
+
+    translation_ += decoded;
+    end = offset + (first ? 0 : 1) /*space*/ + decoded.size();
+    translationRanges.emplace_back(offset, end);
+    offset = end;
+  }
+
+  // Converting ByteRanges as indices into string_views.
+  LOG(info, "generating targetMappings");
+  targetMappings_.reserve(translationRanges.size());
+  for (auto &p : translationRanges) {
+    targetMappings_.emplace_back(&translation_[p.first], p.second - p.first);
+  }
+
+  // Surely, let's add sentenceMappings_
+  LOG(info, "generating SentenceMappings");
+  for (auto p = sourceMappings_.begin(), q = targetMappings_.begin();
+       p != sourceMappings_.end() && q != targetMappings_.end(); ++p, ++q) {
+    sentenceMappings_.emplace_back(*p, *q);
+  }
+}
+
+std::vector<int> TranslationResult::getAlignment(unsigned int index) {
+  Ptr<History> history = histories_[index];
+  NBestList onebest = history->nBest(1);
+  Result &result = onebest[0]; // Expecting only one result;
+  Words &words = std::get<0>(result);
+  auto &hypothesis = std::get<1>(result);
+
+  // soft alignment = P(src pos|trg pos) for each beam and batch index, stored
+  // in a flattened CPU-side array
+  //
+  // Also used on QuickSAND boundary where beam and batch size is 1. Then it is
+  // simply [t][s] -> P(s|t)
+  //
+  // typedef std::vector<std::vector<float>> SoftAlignment;
+  // [trg pos][beam depth * max src length * batch size]
+
+  auto softAlignment = hypothesis->tracebackAlignment();
+  auto hardAlignment = data::ConvertSoftAlignToHardAlign(softAlignment);
+  std::vector<int> alignment(words.size(), -1);
+  for (auto &p : hardAlignment) {
+    alignment[p.tgtPos] = p.srcPos;
+  }
+  return alignment;
+}
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/translation_result.h b/src/translator/translation_result.h
new file mode 100644
index 000000000..b2cb393b9
--- /dev/null
+++ b/src/translator/translation_result.h
@@ -0,0 +1,64 @@
+#ifndef SRC_BERGAMOT_TRANSLATION_RESULT_H_
+#define SRC_BERGAMOT_TRANSLATION_RESULT_H_
+
+#include "data/types.h"
+#include "definitions.h"
+#include "translator/beam_search.h"
+
+#include <cassert>
+#include <string>
+#include <vector>
+
+namespace marian {
+namespace bergamot {
+class TranslationResult {
+public:
+  TranslationResult(std::string &&source, Segments &&segments,
+                    std::vector<TokenRanges> &&sourceRanges,
+                    Histories &&histories,
+                    std::vector<Ptr<Vocab const>> &vocabs);
+
+  const Histories &getHistories() const { return histories_; }
+
+  // https://github.com/browsermt/bergamot-translator/blob/0200843ed7e5366f4143422c64fcd1837d9baca7/src/TranslationResult.h
+  const std::string &getOriginalText() const { return source_; }
+  const std::string &getTranslatedText() const { return translation_; }
+  typedef std::vector<std::pair<string_view, string_view>> SentenceMappings;
+  const SentenceMappings &getSentenceMappings() const {
+    return sentenceMappings_;
+  }
+
+  // Return the Quality scores of the translated text.
+  // Not implemented currently, commenting out.
+  // const QualityScore &getQualityScore() const { return qualityScore; }
+
+  // Provides a hard alignment between source and target words.
+  std::vector<int> getAlignment(unsigned int index);
+
+private:
+  std::string source_;
+  std::string translation_;
+
+  // Histories are currently required for interoperability with OutputPrinter
+  // and OutputCollector and hence comparisons with marian-decoder.
+  Histories histories_;
+
+  // Can be removed eventually.
+  Segments segments_;
+  std::vector<Ptr<Vocab const>> *vocabs_;
+
+  // string_views at the token level.
+  std::vector<TokenRanges> sourceRanges_;
+
+  // string_views at the sentence-level.
+  std::vector<string_view> sourceMappings_;
+  std::vector<string_view> targetMappings_;
+
+  // Adding the following to complete bergamot-translator spec, redundant with
+  // sourceMappings_ and targetMappings_.
+  SentenceMappings sentenceMappings_;
+};
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_TRANSLATION_RESULT_H_
diff --git a/src/translator/utils.cpp b/src/translator/utils.cpp
new file mode 100644
index 000000000..ea4c5037c
--- /dev/null
+++ b/src/translator/utils.cpp
@@ -0,0 +1,31 @@
+#include "utils.h"
+
+#include <unordered_map>
+
+namespace marian {
+namespace bergamot {
+
+
+std::vector<Ptr<const Vocab>> loadVocabularies(
+    Ptr<Options> options) {
+  // @TODO: parallelize vocab loading for faster startup
+  auto vfiles = options->get<std::vector<std::string>>("vocabs");
+  // with the current setup, we need at least two vocabs: src and trg
+  ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
+  std::vector<Ptr<Vocab const>> vocabs(vfiles.size());
+  std::unordered_map<std::string, Ptr<Vocab>> vmap;
+  for (size_t i = 0; i < vocabs.size(); ++i) {
+    auto m = vmap.emplace(std::make_pair(vfiles[i], Ptr<Vocab>()));
+    if (m.second) {  // new: load the vocab
+      m.first->second = New<Vocab>(options, i);
+      m.first->second->load(vfiles[i]);
+    }
+    vocabs[i] = m.first->second;
+  }
+  return vocabs;
+}
+
+
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/utils.h b/src/translator/utils.h
new file mode 100644
index 000000000..594d0cabd
--- /dev/null
+++ b/src/translator/utils.h
@@ -0,0 +1,20 @@
+#ifndef __BERGAMOT_UTILS_H
+#define __BERGAMOT_UTILS_H
+
+#include "common/options.h"
+#include "common/types.h"  
+#include "data/vocab.h"
+#include "translator/history.h"
+
+#include <string>
+#include <vector>
+
+namespace marian {
+namespace bergamot {
+
+std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // __BERGAMOT_UTILS_H

From d786f2554ea8cf362211d4766231b53745a97840 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 20 Jan 2021 19:14:34 +0000
Subject: [PATCH 018/442] Bumping marian with sentencepiece capable fork

Modifications to SentencePiece are necessary to provide token level
string_views. This commit changes marian to an alternate branch which
has the feature incorporated.
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 69894793e..96d5a712d 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 69894793ebd93256d824a1590924780a6d54cae8
+Subproject commit 96d5a712d3b8bc56f120ba5220365f955719f4d4

From bde90947285db4cad7da50ce89ac590dfb89dea3 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 20 Jan 2021 19:52:34 +0000
Subject: [PATCH 019/442] Updating CMakeLists to build main

CMakeLists have been modified with the necessary includes to add
browsermt/mts@nuke files to the bergamot-translator library. In
addition, adds the ssplit dependency, corresponding includes.

Intel MKL fails on compilation, unable to find libraries. To solve this
3rd_party/CMakeLists.txt is modified with @ug's fixes to propogate
variables (EXT_LIBS, etc) at a library level.
---
 3rd_party/CMakeLists.txt      | 24 ++++++++++++++++++++++--
 CMakeLists.txt                |  5 +++++
 src/translator/CMakeLists.txt | 22 ++++++++++++++++++++--
 3 files changed, 47 insertions(+), 4 deletions(-)

diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index 97bf94e05..a5aed0689 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -1,6 +1,26 @@
 add_subdirectory(marian-dev)
+add_subdirectory(ssplit-cpp)
+
+include_directories(ssplit-cpp/src)
+
+# Add include directories for marian target to be able to use it anywhere in the
+# project without explicitly specifying its include directories. Once marian
+# fixes this problem, it can be removed.
 
-# Add include directories for marian target to be able to use it anywhere in the project without
-# explicitly specifying its include directories. Once marian fixes this problem, it can be removed.
 get_property(INCDIRS DIRECTORY marian-dev/src PROPERTY INCLUDE_DIRECTORIES)
 target_include_directories(marian PUBLIC ${INCDIRS})
+
+
+get_property(INCLUDE_DIRECTORIES DIRECTORY . PROPERTY INCLUDE_DIRECTORIES)
+set(INCLUDE_DIRECTORIES ${INCLUDE_DIRECTORIES} PARENT_SCOPE)
+
+# Required to enable MKL, at least
+get_directory_property(EXT_LIBS DIRECTORY marian-dev DEFINITION EXT_LIBS)
+set(EXT_LIBS ${EXT_LIBS} PARENT_SCOPE)
+
+# Compilation flags
+get_directory_property(CMAKE_C_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_C_FLAGS)
+get_directory_property(CMAKE_CXX_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_CXX_FLAGS)
+set(CMAKE_C_FLAGS ${CMAKE_C_FLAGS} PARENT_SCOPE)
+set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} PARENT_SCOPE)
+
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 68a075d5c..935cd1eab 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -14,7 +14,12 @@ set(BUILD_ARCH native CACHE STRING "Compile for this CPU architecture.")
 option(COMPILE_CUDA "Compile GPU version" OFF)
 option(USE_SENTENCEPIECE "Download and compile SentencePiece" ON)
 option(USE_STATIC_LIBS "Link statically against non-system libs" ON)
+option(USE_MKL "Compile with MKL support" ON)
 
 add_subdirectory(3rd_party)
+
+# Adds the include directories set inside 3rd_party.
+include_directories(${INCLUDE_DIRECTORIES})
+
 add_subdirectory(src)
 add_subdirectory(app)
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 08a82fcb5..16c99e7d6 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,11 +1,29 @@
 add_library(bergamot-translator STATIC
     AbstractTranslationModel.cpp
     TranslationModel.cpp
-    TranslationModelConfigToOptionsAdaptor.cpp)
+    TranslationModelConfigToOptionsAdaptor.cpp
 
-target_link_libraries(bergamot-translator marian)
+    # Following files added from browsermt/mts@nuke
+    textops.cpp
+    batch_translator.cpp 
+    multifactor_priority.cpp 
+    request.cpp 
+    service.cpp
+    batcher.cpp
+    utils.cpp
+    translation_result.cpp
+)
+
+# Replacement app for marian-decoder from browsermt/mts@nuke
+add_executable(main main.cpp)
+set_target_properties(main PROPERTIES OUTPUT bergamot-cli RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}")
+target_compile_options(main PUBLIC ${ALL_WARNINGS})
+set(EXECUTABLES ${EXECUTABLES} main)
+target_link_libraries(main bergamot-translator marian ${MARIAN_CUDA_LIB} ${EXT_LIBS} ssplit pcrecpp.a pcre.a)
 
+target_link_libraries(bergamot-translator marian)
 target_include_directories(bergamot-translator
     PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}
     PRIVATE ${CMAKE_SOURCE_DIR}
     PUBLIC ${CMAKE_SOURCE_DIR}/src)
+

From b25b2276e35cf7f0079a254793a042b714efdbcf Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 20 Jan 2021 20:10:19 +0000
Subject: [PATCH 020/442] Undoing LineSplitter, reverting SentenceSplitter.

A faster linesplitter added for benchmarks is removed in favour of @ug's
ssplit-cpp.
NOTE: ssplit-cpp's regex based implementation is slow for one-line
parses, which ideally needs to be improved in upstream ssplit-cpp to
trivially reduce to a faster newline character based split.
---
 src/translator/textops.cpp |  4 ++--
 src/translator/textops.h   | 43 +-------------------------------------
 2 files changed, 3 insertions(+), 44 deletions(-)

diff --git a/src/translator/textops.cpp b/src/translator/textops.cpp
index 55f22dab8..add3b1026 100644
--- a/src/translator/textops.cpp
+++ b/src/translator/textops.cpp
@@ -74,8 +74,8 @@ void TextProcessor::query_to_segments(const string_view &query,
                                       Segments &segments,
                                       std::vector<TokenRanges> &sourceRanges) {
   auto buf = sentence_splitter_.createSentenceStream(query);
-  // pcrecpp::StringPiece snt;
-  string_view snt;
+  pcrecpp::StringPiece snt;
+  // string_view snt;
 
   int sentencesProcessed{0};
 
diff --git a/src/translator/textops.h b/src/translator/textops.h
index 0b4ee6e5c..5de54fdd5 100644
--- a/src/translator/textops.h
+++ b/src/translator/textops.h
@@ -19,37 +19,6 @@
 namespace marian {
 namespace bergamot {
 
-class StringViewStream {
-private:
-  string_view text_;
-  string_view::iterator current_;
-
-public:
-  StringViewStream(const string_view &text) : text_(text) {
-    current_ = text_.begin();
-  }
-
-  bool operator>>(string_view &sentence_view) {
-    // Skip to the next non-newline; whitespaces, anything else are okay.
-    while (current_ != text_.end() &&
-           (*current_ == '\n' || *current_ == ' ' || *current_ == '\t')) {
-      ++current_;
-    }
-
-    string_view::iterator p = current_;
-    while (p != text_.end() && *p != '\n') {
-      ++p;
-    }
-
-    if (p == current_)
-      return false;
-
-    sentence_view = string_view(current_, p - current_);
-    current_ = p;
-    return true;
-  };
-};
-
 class SentenceSplitter {
 public:
   explicit SentenceSplitter(Ptr<Options> options);
@@ -62,16 +31,6 @@ class SentenceSplitter {
   ug::ssplit::SentenceStream::splitmode string2splitmode(const std::string &m);
 };
 
-class LineSplitter {
-public:
-  explicit LineSplitter(Ptr<Options> options){
-      // Do nothing.
-  };
-  StringViewStream createSentenceStream(string_view const &input) {
-    return std::move(StringViewStream(input));
-  }
-};
-
 class Tokenizer {
 private:
   std::vector<Ptr<Vocab const>> vocabs_;
@@ -87,7 +46,7 @@ class Tokenizer {
 class TextProcessor {
 private:
   Tokenizer tokenizer_;
-  LineSplitter sentence_splitter_;
+  SentenceSplitter sentence_splitter_;
   unsigned int max_input_sentence_tokens_;
 
 public:

From b3f1905a120caeff042faa4a7cc539e9fa495194 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 20 Jan 2021 20:56:50 +0000
Subject: [PATCH 021/442] Adding documentation and example to service.h

---
 src/translator/service.h | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/src/translator/service.h b/src/translator/service.h
index 519975445..8270a33a8 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -16,14 +16,27 @@ namespace marian {
 namespace bergamot {
 
 class Service {
+
+  // Service exposes methods to translate an incoming blob of text to the
+  // Consumer of bergamot API.
+  //
+  // An example use of this API looks as follows:
+  //
+  //  options = ...;
+  //  service = Service(options);
+  //  std::string input_blob = "Hello World";
+  //  std::future<TranslationResult>
+  //      response = service.translate(std::move(input)_blob);
+  //  response.wait();
+  //  TranslationResult result = response.get();
+
 public:
   explicit Service(Ptr<Options> options);
   std::future<TranslationResult> translateWithCopy(std::string input);
   std::future<TranslationResult> translate(std::string &&input);
   void stop();
-  Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); };
-  Ptr<Vocab const> targetVocab() const { return vocabs_.back(); };
-  ;
+  Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); }
+  Ptr<Vocab const> targetVocab() const { return vocabs_.back(); }
   ~Service();
 
 private:
@@ -31,6 +44,15 @@ class Service {
   unsigned int batchNumber_;
   int numWorkers_;
 
+  // Consists of:
+  // 1. an instance of text-processing class (TextProcessor),
+  // 2. a Batcher // class which handles efficient batching by minimizing
+  //    padding wasting compute.
+  // 3. Multiple workers - which are instances of BatchTranslators are
+  //    spawned in threads. The Batcher acts as a producer for a
+  //    producer-consumer queue, with idle BatchTranslators requesting batches
+  //    as they're ready.
+
   std::vector<Ptr<Vocab const>> vocabs_;
   TextProcessor text_processor_;
   Batcher batcher_;

From d3c707f73541879795940d113683ad72a1c2aa76 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 20 Jan 2021 21:11:27 +0000
Subject: [PATCH 022/442] Enhancing service.h further

---
 src/translator/service.h | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

diff --git a/src/translator/service.h b/src/translator/service.h
index 8270a33a8..982166eea 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -26,17 +26,22 @@ class Service {
   //  service = Service(options);
   //  std::string input_blob = "Hello World";
   //  std::future<TranslationResult>
-  //      response = service.translate(std::move(input)_blob);
+  //      response = service.translate(std::move(input_blob));
   //  response.wait();
   //  TranslationResult result = response.get();
 
 public:
   explicit Service(Ptr<Options> options);
+
+  // Constructs new string copying, calls translate internally.
   std::future<TranslationResult> translateWithCopy(std::string input);
   std::future<TranslationResult> translate(std::string &&input);
+
   void stop();
+
   Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); }
   Ptr<Vocab const> targetVocab() const { return vocabs_.back(); }
+
   ~Service();
 
 private:
@@ -44,16 +49,23 @@ class Service {
   unsigned int batchNumber_;
   int numWorkers_;
 
+  // vocabs are used to construct a Request, which later uses it to construct
+  // TranslationResult (decode from words to string).
+  std::vector<Ptr<Vocab const>> vocabs_;
+
   // Consists of:
-  // 1. an instance of text-processing class (TextProcessor),
-  // 2. a Batcher // class which handles efficient batching by minimizing
+  //
+  // 1. text-processing class (TextProcessor), which handles breaking a blob of
+  //    text into sentences and providing them representated by finite
+  //    vocabulary for further processing by hte neural machine translation.
+  // 2. a Batcher class which handles efficient batching by minimizing
   //    padding wasting compute.
   // 3. Multiple workers - which are instances of BatchTranslators are
-  //    spawned in threads. The Batcher acts as a producer for a
-  //    producer-consumer queue, with idle BatchTranslators requesting batches
-  //    as they're ready.
+  //    spawned in separate threads.
+  //
+  // Batcher acts as a producer for a producer-consumer queue (pcqueue_), with
+  // idle BatchTranslators being consumers requesting batches as they're ready.
 
-  std::vector<Ptr<Vocab const>> vocabs_;
   TextProcessor text_processor_;
   Batcher batcher_;
   PCQueue<PCItem> pcqueue_;

From 54a6c6ce8088ba1123d8f3e7a1518f367bad0cbb Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 20 Jan 2021 21:18:20 +0000
Subject: [PATCH 023/442] Moving main (mts) to app/

Commit modifies the example test-code main-mts into the app folder,
updating CMakeLists accordingly.
---
 app/CMakeLists.txt            |  8 ++-
 app/main-mts.cpp              | 58 ++++++++++++++++++++++
 src/translator/CMakeLists.txt |  7 ---
 src/translator/main.cpp       | 92 -----------------------------------
 4 files changed, 65 insertions(+), 100 deletions(-)
 create mode 100644 app/main-mts.cpp
 delete mode 100644 src/translator/main.cpp

diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index f9698dc55..fcc03237e 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -1,3 +1,9 @@
 add_executable(bergamot-translator-app main.cpp)
-
 target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
+
+# Replacement app for marian-decoder from browsermt/mts@nuke
+add_executable(main main-mts.cpp)
+set_target_properties(main PROPERTIES OUTPUT bergamot-cli RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}")
+target_compile_options(main PUBLIC ${ALL_WARNINGS})
+set(EXECUTABLES ${EXECUTABLES} main)
+target_link_libraries(main bergamot-translator marian ${MARIAN_CUDA_LIB} ${EXT_LIBS} ssplit pcrecpp.a pcre.a)
diff --git a/app/main-mts.cpp b/app/main-mts.cpp
new file mode 100644
index 000000000..3de57b074
--- /dev/null
+++ b/app/main-mts.cpp
@@ -0,0 +1,58 @@
+#include <cstdlib>
+#include <iostream>
+#include <sstream>
+
+#include "common/definitions.h"
+#include "common/timer.h"
+#include "common/utils.h"
+#include "marian.h"
+#include "translator/history.h"
+#include "translator/output_collector.h"
+#include "translator/output_printer.h"
+
+#include "translator/service.h"
+
+int main(int argc, char *argv[]) {
+  marian::ConfigParser cp(marian::cli::mode::translation);
+
+  cp.addOption<std::string>(
+      "--ssplit-prefix-file", "Bergamot Options",
+      "File with nonbreaking prefixes for sentence splitting.");
+
+  cp.addOption<std::string>("--ssplit-mode", "Server Options",
+                            "[paragraph, sentence, wrapped_text]");
+
+  cp.addOption<int>(
+      "--max-input-sentence-tokens", "Bergamot Options",
+      "Maximum input tokens to be processed in a single sentence.", 128);
+
+  cp.addOption<int>("--max-input-tokens", "Bergamot Options",
+                    "Maximum input tokens in a batch. control for"
+                    "Bergamot Queue",
+                    1024);
+
+  // Launch service.
+  auto options = cp.parseOptions(argc, argv, true);
+  marian::bergamot::Service service(options);
+
+  // Read a large input text blob from stdin
+  std::ostringstream std_input;
+  std_input << std::cin.rdbuf();
+  std::string input = std_input.str();
+
+  LOG(info, "IO complete Translating input");
+  // Wait on future until TranslationResult is complete
+  auto translation_result_future = service.translate(std::move(input));
+  translation_result_future.wait();
+  auto translation_result = translation_result_future.get();
+
+  // Obtain sentencemappings and print them as Proof of Concept.
+  for (auto &p : translation_result.getSentenceMappings()) {
+    std::cout << "[src] " << p.first << "\n";
+    std::cout << "[tgt] " << p.second << "\n";
+  }
+
+  // Stop Service.
+  service.stop();
+  return 0;
+}
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 16c99e7d6..ce3193d41 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -14,13 +14,6 @@ add_library(bergamot-translator STATIC
     translation_result.cpp
 )
 
-# Replacement app for marian-decoder from browsermt/mts@nuke
-add_executable(main main.cpp)
-set_target_properties(main PROPERTIES OUTPUT bergamot-cli RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}")
-target_compile_options(main PUBLIC ${ALL_WARNINGS})
-set(EXECUTABLES ${EXECUTABLES} main)
-target_link_libraries(main bergamot-translator marian ${MARIAN_CUDA_LIB} ${EXT_LIBS} ssplit pcrecpp.a pcre.a)
-
 target_link_libraries(bergamot-translator marian)
 target_include_directories(bergamot-translator
     PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}
diff --git a/src/translator/main.cpp b/src/translator/main.cpp
deleted file mode 100644
index b3fb3f116..000000000
--- a/src/translator/main.cpp
+++ /dev/null
@@ -1,92 +0,0 @@
-#include <cstdlib>
-#include <iostream>
-#include <sstream>
-
-#include "common/definitions.h"
-#include "common/timer.h"
-#include "common/utils.h"
-#include "marian.h"
-#include "translator/history.h"
-#include "translator/output_collector.h"
-#include "translator/output_printer.h"
-
-#include "service.h"
-
-void marian_decoder_minimal(const marian::Histories &histories,
-                            marian::Ptr<marian::Vocab const> targetVocab,
-                            marian::Ptr<marian::Options> options) {
-
-  bool doNbest = options->get<bool>("n-best");
-
-  auto collector =
-      marian::New<marian::OutputCollector>(options->get<std::string>("output"));
-
-  // There is a dependency of vocabs here.
-  auto printer = marian::New<marian::OutputPrinter>(options, targetVocab);
-  if (options->get<bool>("quiet-translation"))
-    collector->setPrintingStrategy(marian::New<marian::QuietPrinting>());
-
-  for (auto &history : histories) {
-    std::stringstream best1;
-    std::stringstream bestn;
-    printer->print(history, best1, bestn);
-    collector->Write((long)history->getLineNum(), best1.str(), bestn.str(),
-                     doNbest);
-  }
-}
-
-int main(int argc, char *argv[]) {
-  marian::ConfigParser cp(marian::cli::mode::translation);
-
-  cp.addOption<std::string>(
-      "--ssplit-prefix-file", "Bergamot Options",
-      "File with nonbreaking prefixes for sentence splitting.");
-
-  cp.addOption<std::string>("--ssplit-mode", "Server Options",
-                            "[paragraph, sentence, wrapped_text]");
-
-  cp.addOption<int>(
-      "--max-input-sentence-tokens", "Bergamot Options",
-      "Maximum input tokens to be processed in a single sentence.", 128);
-
-  cp.addOption<int>("--max-input-tokens", "Bergamot Options",
-                    "Maximum input tokens in a batch. control for"
-                    "Bergamot Queue",
-                    1024);
-
-  cp.addOption<int>("--nbest", "Bergamot Options",
-                    "NBest value used for decoding", 1);
-
-  cp.addOption<bool>("--marian-decoder-alpha", "Bergamot Options",
-                     "Run marian-decoder output printer code", false);
-
-  // TODO(jerin): Add QE later.
-  // marian::qe::QualityEstimator::addOptions(cp);
-
-  marian::timer::Timer decoderTimer;
-
-  auto options = cp.parseOptions(argc, argv, true);
-  marian::bergamot::Service service(options);
-
-  std::ostringstream std_input;
-  std_input << std::cin.rdbuf();
-  std::string input = std_input.str();
-
-  LOG(info, "IO complete Translating input");
-  auto translation_result_future = service.translate(std::move(input));
-  translation_result_future.wait();
-  auto translation_result = translation_result_future.get();
-  if (options->get<bool>("marian-decoder-alpha")) {
-    marian_decoder_minimal(translation_result.getHistories(),
-                           service.targetVocab(), options);
-    LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
-  } else {
-    for (auto &p : translation_result.getSentenceMappings()) {
-      std::cout << "[src] " << p.first << "\n";
-      std::cout << "[tgt] " << p.second << "\n";
-    }
-  }
-
-  service.stop();
-  return 0;
-}

From caa03e1d9fbc62fe4295798a0ac668139ad30451 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 20 Jan 2021 21:21:43 +0000
Subject: [PATCH 024/442] Removing unused timer.h

---
 src/translator/timer.h | 32 --------------------------------
 1 file changed, 32 deletions(-)
 delete mode 100644 src/translator/timer.h

diff --git a/src/translator/timer.h b/src/translator/timer.h
deleted file mode 100644
index 744038081..000000000
--- a/src/translator/timer.h
+++ /dev/null
@@ -1,32 +0,0 @@
-#ifndef __BERGAMOT_TIMER_H 
-#define __BERGAMOT_TIMER_H
-
-// https://stackoverflow.com/a/19800231/4565794
-//
-// Careful: This won't work if the user changes his time between Timer() and
-// the call to elapsed() if !std::chrono::high_resolution_clock::is_steady -
-// which is the case on Linux!
-
-#include <iostream>
-#include <chrono>
-
-namespace marian {
-namespace bergamot {
-class Timer {
-public:
-    Timer() : beg_(clock_::now()) {}
-    void reset() { beg_ = clock_::now(); }
-    double elapsed() const { 
-        return std::chrono::duration_cast<second_>
-            (clock_::now() - beg_).count(); }
-
-private:
-    typedef std::chrono::high_resolution_clock clock_;
-    typedef std::chrono::duration<double, std::ratio<1> > second_;
-    std::chrono::time_point<clock_> beg_;
-};
-
-} // namespace bergamot 
-} // namespace marian
-
-#endif // __BERGAMOT_TIMER_H

From d6ec007df93ffac6c40bd8adc4db861d960ee1c1 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 20 Jan 2021 21:58:13 +0000
Subject: [PATCH 025/442] TranslationResult Docs

Removed Alignments, too many questions and no concrete answers. Better
off removing unused code. History is kept for now, for internal use.
---
 src/translator/translation_result.cpp | 25 -------------------------
 src/translator/translation_result.h   |  9 ++++-----
 2 files changed, 4 insertions(+), 30 deletions(-)

diff --git a/src/translator/translation_result.cpp b/src/translator/translation_result.cpp
index 43b233eed..1c74314e3 100644
--- a/src/translator/translation_result.cpp
+++ b/src/translator/translation_result.cpp
@@ -68,30 +68,5 @@ TranslationResult::TranslationResult(std::string &&source, Segments &&segments,
   }
 }
 
-std::vector<int> TranslationResult::getAlignment(unsigned int index) {
-  Ptr<History> history = histories_[index];
-  NBestList onebest = history->nBest(1);
-  Result &result = onebest[0]; // Expecting only one result;
-  Words &words = std::get<0>(result);
-  auto &hypothesis = std::get<1>(result);
-
-  // soft alignment = P(src pos|trg pos) for each beam and batch index, stored
-  // in a flattened CPU-side array
-  //
-  // Also used on QuickSAND boundary where beam and batch size is 1. Then it is
-  // simply [t][s] -> P(s|t)
-  //
-  // typedef std::vector<std::vector<float>> SoftAlignment;
-  // [trg pos][beam depth * max src length * batch size]
-
-  auto softAlignment = hypothesis->tracebackAlignment();
-  auto hardAlignment = data::ConvertSoftAlignToHardAlign(softAlignment);
-  std::vector<int> alignment(words.size(), -1);
-  for (auto &p : hardAlignment) {
-    alignment[p.tgtPos] = p.srcPos;
-  }
-  return alignment;
-}
-
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/translation_result.h b/src/translator/translation_result.h
index b2cb393b9..27bfb370b 100644
--- a/src/translator/translation_result.h
+++ b/src/translator/translation_result.h
@@ -18,11 +18,9 @@ class TranslationResult {
                     Histories &&histories,
                     std::vector<Ptr<Vocab const>> &vocabs);
 
-  const Histories &getHistories() const { return histories_; }
-
-  // https://github.com/browsermt/bergamot-translator/blob/0200843ed7e5366f4143422c64fcd1837d9baca7/src/TranslationResult.h
   const std::string &getOriginalText() const { return source_; }
   const std::string &getTranslatedText() const { return translation_; }
+
   typedef std::vector<std::pair<string_view, string_view>> SentenceMappings;
   const SentenceMappings &getSentenceMappings() const {
     return sentenceMappings_;
@@ -32,8 +30,8 @@ class TranslationResult {
   // Not implemented currently, commenting out.
   // const QualityScore &getQualityScore() const { return qualityScore; }
 
-  // Provides a hard alignment between source and target words.
-  std::vector<int> getAlignment(unsigned int index);
+  // For development use to benchmark with marian-decoder.
+  const Histories &getHistories() const { return histories_; }
 
 private:
   std::string source_;
@@ -41,6 +39,7 @@ class TranslationResult {
 
   // Histories are currently required for interoperability with OutputPrinter
   // and OutputCollector and hence comparisons with marian-decoder.
+  // Future hook to gain alignments.
   Histories histories_;
 
   // Can be removed eventually.

From 4640ae409121bd01fb1f5eda3ee9764531ba6dc3 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 21 Jan 2021 00:29:53 +0000
Subject: [PATCH 026/442] Fixes copying around vocabs

Vocabs was earlier loaded in each thread and copied several times.
Modified this to be loaded only once in Service and reference used
consistently later on.

This change makes Tokenizer as a class rather moot, as there's only one
private member and a function. Moved this into TextProcessor.
SentenceSplitter, however remains a separate class.

utils.{h,cpp} had only a single loadVocabularies function, which
is at the moment required only in Service. Making loadVocabularies a
function inside Service and getting rid of utils.*.
---
 src/translator/CMakeLists.txt       |  1 -
 src/translator/batch_translator.cpp | 20 ++++++++++---------
 src/translator/batch_translator.h   |  4 ++--
 src/translator/service.cpp          | 29 ++++++++++++++++++++++-----
 src/translator/service.h            |  6 ++++--
 src/translator/textops.cpp          | 26 ++++++++++--------------
 src/translator/textops.h            | 25 ++++++++---------------
 src/translator/utils.cpp            | 31 -----------------------------
 src/translator/utils.h              | 20 -------------------
 9 files changed, 60 insertions(+), 102 deletions(-)
 delete mode 100644 src/translator/utils.cpp
 delete mode 100644 src/translator/utils.h

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index ce3193d41..025ef3d9c 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -10,7 +10,6 @@ add_library(bergamot-translator STATIC
     request.cpp 
     service.cpp
     batcher.cpp
-    utils.cpp
     translation_result.cpp
 )
 
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index f41fa590f..622162ca4 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -4,26 +4,27 @@
 #include "data/text_input.h"
 #include "sanelogging.h"
 #include "translator/beam_search.h"
-#include "utils.h"
 
 namespace marian {
 namespace bergamot {
 
 BatchTranslator::BatchTranslator(DeviceId const device,
-                                 PCQueue<PCItem> &pcqueue, Ptr<Options> options)
-    : device_(device), options_(options), pcqueue_(&pcqueue) {
+                                 PCQueue<PCItem> &pcqueue,
+                                 std::vector<Ptr<Vocab const>> &vocabs,
+                                 Ptr<Options> options)
+    : device_(device), options_(options), pcqueue_(&pcqueue), vocabs_(&vocabs) {
 
   thread_ = std::thread([this] { this->mainloop(); });
 }
 
 void BatchTranslator::initGraph() {
-  vocabs_ = loadVocabularies(options_);
   if (options_->hasAndNotEmpty("shortlist")) {
     Ptr<data::ShortlistGenerator const> slgen;
     int srcIdx = 0, trgIdx = 1;
-    bool shared_vcb = vocabs_.front() == vocabs_.back();
-    slgen_ = New<data::LexicalShortlistGenerator>(
-        options_, vocabs_.front(), vocabs_.back(), srcIdx, trgIdx, shared_vcb);
+    bool shared_vcb = vocabs_->front() == vocabs_->back();
+    slgen_ = New<data::LexicalShortlistGenerator>(options_, vocabs_->front(),
+                                                  vocabs_->back(), srcIdx,
+                                                  trgIdx, shared_vcb);
   }
 
   graph_ = New<ExpressionGraph>(true); // always optimize
@@ -72,7 +73,8 @@ void BatchTranslator::translate(RequestSentences &requestSentences,
 
   std::vector<Ptr<SubBatch>> subBatches;
   for (size_t j = 0; j < maxDims.size(); ++j) {
-    subBatches.emplace_back(New<SubBatch>(batchSize, maxDims[j], vocabs_[j]));
+    subBatches.emplace_back(
+        New<SubBatch>(batchSize, maxDims[j], vocabs_->at(j)));
   }
 
   std::vector<size_t> words(maxDims.size(), 0);
@@ -92,7 +94,7 @@ void BatchTranslator::translate(RequestSentences &requestSentences,
   auto batch = Ptr<CorpusBatch>(new CorpusBatch(subBatches));
   batch->setSentenceIds(sentenceIds);
 
-  auto trgVocab = vocabs_.back();
+  auto trgVocab = vocabs_->back();
   auto search = New<BeamSearch>(options_, scorers_, trgVocab);
 
   histories = std::move(search->search(graph_, batch));
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 638a1a971..069155efb 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -23,7 +23,7 @@ class BatchTranslator {
 
 public:
   BatchTranslator(DeviceId const device, PCQueue<PCItem> &pcqueue,
-                  Ptr<Options> options);
+                  std::vector<Ptr<Vocab const>> &vocabs, Ptr<Options> options);
   void join();
 
   // convenience function for logging. TODO(jerin)
@@ -37,7 +37,7 @@ class BatchTranslator {
   Ptr<Options> options_;
 
   DeviceId device_;
-  std::vector<Ptr<Vocab const>> vocabs_;
+  std::vector<Ptr<Vocab const>> *vocabs_;
   Ptr<ExpressionGraph> graph_;
   std::vector<Ptr<Scorer>> scorers_;
   Ptr<data::ShortlistGenerator const> slgen_;
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index c9260812d..fa6e59767 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -2,7 +2,6 @@
 #include "definitions.h"
 #include "sanelogging.h"
 
-#include "utils.h"
 #include <string>
 #include <utility>
 
@@ -11,15 +10,16 @@ namespace bergamot {
 
 Service::Service(Ptr<Options> options)
     : requestId_(0), batchNumber_(0),
-      numWorkers_(options->get<int>("cpu-threads")), text_processor_(options),
-      batcher_(options), pcqueue_(2 * options->get<int>("cpu-threads")) {
+      numWorkers_(options->get<int>("cpu-threads")),
+      vocabs_(std::move(loadVocabularies(options))),
+      text_processor_(vocabs_, options), batcher_(options),
+      pcqueue_(2 * options->get<int>("cpu-threads")) {
 
-  vocabs_ = loadVocabularies(options);
   workers_.reserve(numWorkers_);
 
   for (int i = 0; i < numWorkers_; i++) {
     marian::DeviceId deviceId(i, DeviceType::cpu);
-    workers_.emplace_back(deviceId, pcqueue_, options);
+    workers_.emplace_back(deviceId, pcqueue_, vocabs_, options);
   }
 }
 
@@ -95,5 +95,24 @@ void Service::stop() {
 
 Service::~Service() { stop(); }
 
+// Internal function nobody used, only within service.
+std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options) {
+  // @TODO: parallelize vocab loading for faster startup
+  auto vfiles = options->get<std::vector<std::string>>("vocabs");
+  // with the current setup, we need at least two vocabs: src and trg
+  ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
+  std::vector<Ptr<Vocab const>> vocabs(vfiles.size());
+  std::unordered_map<std::string, Ptr<Vocab>> vmap;
+  for (size_t i = 0; i < vocabs.size(); ++i) {
+    auto m = vmap.emplace(std::make_pair(vfiles[i], Ptr<Vocab>()));
+    if (m.second) { // new: load the vocab
+      m.first->second = New<Vocab>(options, i);
+      m.first->second->load(vfiles[i]);
+    }
+    vocabs[i] = m.first->second;
+  }
+  return vocabs;
+}
+
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/service.h b/src/translator/service.h
index 982166eea..4069d1392 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -51,7 +51,7 @@ class Service {
 
   // vocabs are used to construct a Request, which later uses it to construct
   // TranslationResult (decode from words to string).
-  std::vector<Ptr<Vocab const>> vocabs_;
+  std::vector<Ptr<Vocab const>> vocabs_; // ORDER DEPENDENCY
 
   // Consists of:
   //
@@ -66,12 +66,14 @@ class Service {
   // Batcher acts as a producer for a producer-consumer queue (pcqueue_), with
   // idle BatchTranslators being consumers requesting batches as they're ready.
 
-  TextProcessor text_processor_;
+  TextProcessor text_processor_; // ORDER DEPENDENCY
   Batcher batcher_;
   PCQueue<PCItem> pcqueue_;
   std::vector<BatchTranslator> workers_;
 };
 
+std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);
+
 } // namespace bergamot
 } // namespace marian
 
diff --git a/src/translator/textops.cpp b/src/translator/textops.cpp
index add3b1026..80d262edb 100644
--- a/src/translator/textops.cpp
+++ b/src/translator/textops.cpp
@@ -1,6 +1,5 @@
 #include "textops.h"
 #include "common/timer.h"
-#include "utils.h"
 #include <pcrecpp.h>
 #include <string>
 #include <unordered_map>
@@ -51,18 +50,16 @@ SentenceSplitter::string2splitmode(const std::string &m) {
   return splitmode::wrapped_text;
 }
 
-Tokenizer::Tokenizer(Ptr<Options> options) : inference_(true), addEOS_(false) {
-  vocabs_ = loadVocabularies(options);
-}
-
-Segment Tokenizer::tokenize(const string_view &snt, TokenRanges &tokenRanges) {
+Segment TextProcessor::tokenize(const string_view &snt,
+                                TokenRanges &tokenRanges) {
   // TODO(jerin): Bunch of hardcode here, 1, 0, need to get rid off somehow.
-  return vocabs_[0]->encodePreservingSource(snt, tokenRanges, addEOS_,
-                                            inference_);
+  return vocabs_->front()->encodePreservingSource(
+      snt, tokenRanges, /*addEOS=*/false, /*inference=*/true);
 }
 
-TextProcessor::TextProcessor(Ptr<Options> options)
-    : tokenizer_(options), sentence_splitter_(options) {
+TextProcessor::TextProcessor(std::vector<Ptr<Vocab const>> &vocabs,
+                             Ptr<Options> options)
+    : vocabs_(&vocabs), sentence_splitter_(options) {
   max_input_sentence_tokens_ = options->get<int>("max-input-sentence-tokens");
   max_input_sentence_tokens_ =
       max_input_sentence_tokens_ - 1; // Account for EOS
@@ -84,8 +81,7 @@ void TextProcessor::query_to_segments(const string_view &query,
     string_view snt_string_view(snt.data(), snt.size());
     TokenRanges snt_alignment;
     timer::Timer spiece_timer;
-    Segment tokenized_sentence =
-        tokenizer_.tokenize(snt_string_view, snt_alignment);
+    Segment tokenized_sentence = tokenize(snt_string_view, snt_alignment);
 
     // LOG(info, "Tokenization took {:.5f} seconds", spiece_timer.elapsed());
     if (tokenized_sentence.size() > 0) {
@@ -96,7 +92,7 @@ void TextProcessor::query_to_segments(const string_view &query,
              offset += max_input_sentence_tokens_) {
           auto start = tokenized_sentence.begin() + offset;
           Segment segment(start, start + max_input_sentence_tokens_);
-          segment.push_back(tokenizer_.sourceEosId());
+          segment.push_back(sourceEosId());
           segments.push_back(segment);
 
           auto astart = snt_alignment.begin() + offset;
@@ -108,7 +104,7 @@ void TextProcessor::query_to_segments(const string_view &query,
         if (offset < max_input_sentence_tokens_) {
           auto start = tokenized_sentence.begin() + offset;
           Segment segment(start, tokenized_sentence.end());
-          segment.push_back(tokenizer_.sourceEosId());
+          segment.push_back(sourceEosId());
           segments.push_back(segment);
 
           auto astart = snt_alignment.begin() + offset;
@@ -118,7 +114,7 @@ void TextProcessor::query_to_segments(const string_view &query,
 
       } else {
         timer::Timer push_timer;
-        tokenized_sentence.push_back(tokenizer_.sourceEosId());
+        tokenized_sentence.push_back(sourceEosId());
         segments.push_back(tokenized_sentence);
         sourceRanges.push_back(snt_alignment);
         // LOG(info, "Push took {:.5f} seconds", push_timer.elapsed());
diff --git a/src/translator/textops.h b/src/translator/textops.h
index 5de54fdd5..5202f1b0c 100644
--- a/src/translator/textops.h
+++ b/src/translator/textops.h
@@ -31,28 +31,19 @@ class SentenceSplitter {
   ug::ssplit::SentenceStream::splitmode string2splitmode(const std::string &m);
 };
 
-class Tokenizer {
-private:
-  std::vector<Ptr<Vocab const>> vocabs_;
-  bool inference_;
-  bool addEOS_;
-
+class TextProcessor {
 public:
-  explicit Tokenizer(Ptr<Options>);
-  Segment tokenize(const string_view &input, TokenRanges &tokenRanges);
-  Word sourceEosId() { return vocabs_.front()->getEosId(); };
-};
+  explicit TextProcessor(std::vector<Ptr<Vocab const>> &vocabs, Ptr<Options>);
+  void query_to_segments(const string_view &query, Segments &segments,
+                         std::vector<TokenRanges> &sourceRanges);
 
-class TextProcessor {
 private:
-  Tokenizer tokenizer_;
+  Segment tokenize(const string_view &input, TokenRanges &tokenRanges);
+  Word sourceEosId() { return vocabs_->front()->getEosId(); }
+
+  std::vector<Ptr<Vocab const>> *vocabs_;
   SentenceSplitter sentence_splitter_;
   unsigned int max_input_sentence_tokens_;
-
-public:
-  explicit TextProcessor(Ptr<Options>);
-  void query_to_segments(const string_view &query, Segments &segments,
-                         std::vector<TokenRanges> &sourceRanges);
 };
 
 } // namespace bergamot
diff --git a/src/translator/utils.cpp b/src/translator/utils.cpp
deleted file mode 100644
index ea4c5037c..000000000
--- a/src/translator/utils.cpp
+++ /dev/null
@@ -1,31 +0,0 @@
-#include "utils.h"
-
-#include <unordered_map>
-
-namespace marian {
-namespace bergamot {
-
-
-std::vector<Ptr<const Vocab>> loadVocabularies(
-    Ptr<Options> options) {
-  // @TODO: parallelize vocab loading for faster startup
-  auto vfiles = options->get<std::vector<std::string>>("vocabs");
-  // with the current setup, we need at least two vocabs: src and trg
-  ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
-  std::vector<Ptr<Vocab const>> vocabs(vfiles.size());
-  std::unordered_map<std::string, Ptr<Vocab>> vmap;
-  for (size_t i = 0; i < vocabs.size(); ++i) {
-    auto m = vmap.emplace(std::make_pair(vfiles[i], Ptr<Vocab>()));
-    if (m.second) {  // new: load the vocab
-      m.first->second = New<Vocab>(options, i);
-      m.first->second->load(vfiles[i]);
-    }
-    vocabs[i] = m.first->second;
-  }
-  return vocabs;
-}
-
-
-
-} // namespace bergamot
-} // namespace marian
diff --git a/src/translator/utils.h b/src/translator/utils.h
deleted file mode 100644
index 594d0cabd..000000000
--- a/src/translator/utils.h
+++ /dev/null
@@ -1,20 +0,0 @@
-#ifndef __BERGAMOT_UTILS_H
-#define __BERGAMOT_UTILS_H
-
-#include "common/options.h"
-#include "common/types.h"  
-#include "data/vocab.h"
-#include "translator/history.h"
-
-#include <string>
-#include <vector>
-
-namespace marian {
-namespace bergamot {
-
-std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);
-
-} // namespace bergamot
-} // namespace marian
-
-#endif // __BERGAMOT_UTILS_H

From ea1a628cd2a894de7eed9d3a3111120612b49d53 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 21 Jan 2021 01:31:29 +0000
Subject: [PATCH 027/442] Neaten TextProcessor, add a bit of docs.

- Truncating long sentences into those of a specified length for faster
  processing is now a separate function, for improved readability.
- Changes doing push_back -> emplace_back at places to avoid copy.
- query_to_segments is renamed as process.
- Comments are added in an attempt to bring some sanity.
---
 src/translator/service.cpp |   2 +-
 src/translator/textops.cpp | 113 +++++++++++++++++--------------------
 src/translator/textops.h   |  25 +++++++-
 3 files changed, 76 insertions(+), 64 deletions(-)

diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index fa6e59767..4a5af301c 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -42,7 +42,7 @@ std::future<TranslationResult> Service::translate(std::string &&input) {
 
   Segments segments;
   std::vector<TokenRanges> sourceAlignments;
-  text_processor_.query_to_segments(input, segments, sourceAlignments);
+  text_processor_.process(input, segments, sourceAlignments);
 
   std::promise<TranslationResult> translationResultPromise;
   auto future = translationResultPromise.get_future();
diff --git a/src/translator/textops.cpp b/src/translator/textops.cpp
index 80d262edb..22b65e9c7 100644
--- a/src/translator/textops.cpp
+++ b/src/translator/textops.cpp
@@ -52,7 +52,6 @@ SentenceSplitter::string2splitmode(const std::string &m) {
 
 Segment TextProcessor::tokenize(const string_view &snt,
                                 TokenRanges &tokenRanges) {
-  // TODO(jerin): Bunch of hardcode here, 1, 0, need to get rid off somehow.
   return vocabs_->front()->encodePreservingSource(
       snt, tokenRanges, /*addEOS=*/false, /*inference=*/true);
 }
@@ -60,70 +59,64 @@ Segment TextProcessor::tokenize(const string_view &snt,
 TextProcessor::TextProcessor(std::vector<Ptr<Vocab const>> &vocabs,
                              Ptr<Options> options)
     : vocabs_(&vocabs), sentence_splitter_(options) {
+
   max_input_sentence_tokens_ = options->get<int>("max-input-sentence-tokens");
-  max_input_sentence_tokens_ =
-      max_input_sentence_tokens_ - 1; // Account for EOS
-  // Dirty assert, should do at configparse
-  assert(max_input_sentence_tokens_ > 0);
+  max_input_sentence_tokens_ = max_input_sentence_tokens_ - 1;
+  ABORT_IF(max_input_sentence_tokens < 0,
+           "max-input-sentence-tokens cannot be < 0");
+}
+
+void TextProcessor::process(const string_view &query, Segments &segments,
+                            std::vector<TokenRanges> &sourceRanges) {
+
+  auto sentenceStream = sentence_splitter_.createSentenceStream(query);
+  pcrecpp::StringPiece sentenceStringPiece;
+
+  while (sentenceStream >> sentenceStringPiece) {
+    string_view sentence(sentenceStringPiece.data(),
+                         sentenceStringPiece.size());
+    TokenRanges tokenRanges;
+    Segment segment = tokenize(sentence, tokenRanges);
+
+    // There are some cases where SentencePiece or vocab returns no words
+    // after normalization. 0 prevents any empty entries from being added.
+    if (segment.size() > 0) {
+      // Truncate segment into max_input_size segments.
+      truncate(segment, tokenRanges, segments, sourceRanges);
+    }
+  }
 }
 
-void TextProcessor::query_to_segments(const string_view &query,
-                                      Segments &segments,
-                                      std::vector<TokenRanges> &sourceRanges) {
-  auto buf = sentence_splitter_.createSentenceStream(query);
-  pcrecpp::StringPiece snt;
-  // string_view snt;
-
-  int sentencesProcessed{0};
-
-  while (buf >> snt) {
-    // LOG(info, "SNT: {}", snt);
-    string_view snt_string_view(snt.data(), snt.size());
-    TokenRanges snt_alignment;
-    timer::Timer spiece_timer;
-    Segment tokenized_sentence = tokenize(snt_string_view, snt_alignment);
-
-    // LOG(info, "Tokenization took {:.5f} seconds", spiece_timer.elapsed());
-    if (tokenized_sentence.size() > 0) {
-      if (tokenized_sentence.size() > max_input_sentence_tokens_) {
-        int offset;
-        for (offset = 0;
-             offset + max_input_sentence_tokens_ < tokenized_sentence.size();
-             offset += max_input_sentence_tokens_) {
-          auto start = tokenized_sentence.begin() + offset;
-          Segment segment(start, start + max_input_sentence_tokens_);
-          segment.push_back(sourceEosId());
-          segments.push_back(segment);
-
-          auto astart = snt_alignment.begin() + offset;
-          TokenRanges segment_alignment(astart,
-                                        astart + max_input_sentence_tokens_);
-          sourceRanges.push_back(segment_alignment);
-        }
-
-        if (offset < max_input_sentence_tokens_) {
-          auto start = tokenized_sentence.begin() + offset;
-          Segment segment(start, tokenized_sentence.end());
-          segment.push_back(sourceEosId());
-          segments.push_back(segment);
-
-          auto astart = snt_alignment.begin() + offset;
-          TokenRanges segment_alignment(astart, snt_alignment.end());
-          sourceRanges.push_back(segment_alignment);
-        }
-
-      } else {
-        timer::Timer push_timer;
-        tokenized_sentence.push_back(sourceEosId());
-        segments.push_back(tokenized_sentence);
-        sourceRanges.push_back(snt_alignment);
-        // LOG(info, "Push took {:.5f} seconds", push_timer.elapsed());
-      }
+void TextProcessor::truncate(Segment &segment, TokenRanges &tokenRanges,
+                             Segments &segments,
+                             std::vector<TokenRanges> &sourceRanges) {
+  if (segment.size() > max_input_sentence_tokens_) {
+    int offset;
+    // Loop as long as I can grab max_input_sentence_tokens_
+    for (offset = 0; offset + max_input_sentence_tokens_ < segment.size();
+         offset += max_input_sentence_tokens_) {
+      auto start = segment.begin() + offset;
+
+      segments.emplace_back(start, start + max_input_sentence_tokens_);
+      segments.back().push_back(sourceEosId());
+
+      auto astart = tokenRanges.begin() + offset;
+      sourceRanges.emplace_back(astart, astart + max_input_sentence_tokens_);
     }
-    ++sentencesProcessed;
-    if (sentencesProcessed % 10000 == 0) {
-      LOG(info, "Processed {}", sentencesProcessed);
+
+    if (offset < max_input_sentence_tokens_) {
+      auto start = segment.begin() + offset;
+      segments.emplace_back(start, segment.end());
+      segments.back().push_back(sourceEosId());
+
+      auto astart = tokenRanges.begin() + offset;
+      sourceRanges.emplace_back(astart, tokenRanges.end());
     }
+
+  } else {
+    segments.emplace_back(segment);
+    segments.back().push_back(sourceEosId());
+    sourceRanges.emplace_back(tokenRanges);
   }
 }
 
diff --git a/src/translator/textops.h b/src/translator/textops.h
index 5202f1b0c..e5c07b6b7 100644
--- a/src/translator/textops.h
+++ b/src/translator/textops.h
@@ -20,6 +20,10 @@ namespace marian {
 namespace bergamot {
 
 class SentenceSplitter {
+  // A wrapper around @ugermann's ssplit-cpp compiled from several places in
+  // mts. Constructed based on options. Used in TextProcessor below to create
+  // sentence-streams, which provide access to one sentence from blob of text at
+  // a time.
 public:
   explicit SentenceSplitter(Ptr<Options> options);
   ug::ssplit::SentenceStream createSentenceStream(string_view const &input);
@@ -32,14 +36,29 @@ class SentenceSplitter {
 };
 
 class TextProcessor {
+  // TextProcessor handles loading the sentencepiece vocabulary and also
+  // contains an instance of sentence-splitter based on ssplit.
+  //
+  // Used in Service to convert an incoming blog of text to a vector of
+  // sentences (vector of words). In addition, the ByteRanges of the
+  // source-tokens in unnormalized text are provided as string_views.
 public:
   explicit TextProcessor(std::vector<Ptr<Vocab const>> &vocabs, Ptr<Options>);
-  void query_to_segments(const string_view &query, Segments &segments,
-                         std::vector<TokenRanges> &sourceRanges);
+
+  void process(const string_view &query, Segments &segments,
+               std::vector<TokenRanges> &sourceRanges);
 
 private:
+  // Tokenizes an input string, returns Words corresponding. Loads the
+  // corresponding byte-ranges into tokenRanges.
   Segment tokenize(const string_view &input, TokenRanges &tokenRanges);
-  Word sourceEosId() { return vocabs_->front()->getEosId(); }
+
+  // Truncate sentence into max_input_size segments.
+  void truncate(Segment &sentence, TokenRanges &tokenRanges, Segments &segments,
+                std::vector<TokenRanges> &sourceRanges);
+
+  // shorthand, used only in truncate()
+  const Word sourceEosId() const { return vocabs_->front()->getEosId(); }
 
   std::vector<Ptr<Vocab const>> *vocabs_;
   SentenceSplitter sentence_splitter_;

From 9b18bd9ffcfcc3b918ef7533f1de128e7396d36a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 21 Jan 2021 02:03:47 +0000
Subject: [PATCH 028/442] MTranslationResult, more comments

---
 src/translator/translation_result.h | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/src/translator/translation_result.h b/src/translator/translation_result.h
index 27bfb370b..fb5a42a09 100644
--- a/src/translator/translation_result.h
+++ b/src/translator/translation_result.h
@@ -18,9 +18,16 @@ class TranslationResult {
                     Histories &&histories,
                     std::vector<Ptr<Vocab const>> &vocabs);
 
+  // Returns const references to source and translated texts, for external
+  // consumption.
+
   const std::string &getOriginalText() const { return source_; }
   const std::string &getTranslatedText() const { return translation_; }
 
+  // A mapping of string_views in the source_ and translation_ are provide as a
+  // pair for external consumption. Each entry corresponding
+  // to a (source-sentence, target-sentence).
+
   typedef std::vector<std::pair<string_view, string_view>> SentenceMappings;
   const SentenceMappings &getSentenceMappings() const {
     return sentenceMappings_;
@@ -53,8 +60,9 @@ class TranslationResult {
   std::vector<string_view> sourceMappings_;
   std::vector<string_view> targetMappings_;
 
-  // Adding the following to complete bergamot-translator spec, redundant with
-  // sourceMappings_ and targetMappings_.
+  // Adding the following to complete bergamot-translator spec, redundant while
+  // sourceMappings_ and targetMappings_ exists or vice-versa.
+
   SentenceMappings sentenceMappings_;
 };
 } // namespace bergamot

From 12e7e2c650fdabfe9b815e2ad1e36a452d891318 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 21 Jan 2021 14:53:53 +0000
Subject: [PATCH 029/442] Fixing compile error, need tests, CI

---
 src/translator/textops.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/translator/textops.cpp b/src/translator/textops.cpp
index 22b65e9c7..837ea7226 100644
--- a/src/translator/textops.cpp
+++ b/src/translator/textops.cpp
@@ -62,7 +62,7 @@ TextProcessor::TextProcessor(std::vector<Ptr<Vocab const>> &vocabs,
 
   max_input_sentence_tokens_ = options->get<int>("max-input-sentence-tokens");
   max_input_sentence_tokens_ = max_input_sentence_tokens_ - 1;
-  ABORT_IF(max_input_sentence_tokens < 0,
+  ABORT_IF(max_input_sentence_tokens_ < 0,
            "max-input-sentence-tokens cannot be < 0");
 }
 

From 80125e2789825b8ea05e2c7a71f398d69a538034 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 21 Jan 2021 14:54:30 +0000
Subject: [PATCH 030/442] Removing unused variable in batch_translator

---
 src/translator/batch_translator.cpp | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 622162ca4..6380a00cc 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -19,7 +19,6 @@ BatchTranslator::BatchTranslator(DeviceId const device,
 
 void BatchTranslator::initGraph() {
   if (options_->hasAndNotEmpty("shortlist")) {
-    Ptr<data::ShortlistGenerator const> slgen;
     int srcIdx = 0, trgIdx = 1;
     bool shared_vcb = vocabs_->front() == vocabs_->back();
     slgen_ = New<data::LexicalShortlistGenerator>(options_, vocabs_->front(),

From 37143933a19c81cf3d2ce7785c77321aaab6e616 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 22 Jan 2021 11:29:32 +0000
Subject: [PATCH 031/442] CMakeLists improvements

Only the bergamot-translator library should be linked to main target
Any other library (marian ${MARIAN_CUDA_LIB} ${EXT_LIBS} ssplit
pcrecpp.a pcre.a) should be linked to bergamot-translator target inside
src/translator folder.
---
 app/CMakeLists.txt            | 8 ++------
 src/translator/CMakeLists.txt | 2 +-
 2 files changed, 3 insertions(+), 7 deletions(-)

diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index fcc03237e..6e71e9e27 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -1,9 +1,5 @@
 add_executable(bergamot-translator-app main.cpp)
 target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
 
-# Replacement app for marian-decoder from browsermt/mts@nuke
-add_executable(main main-mts.cpp)
-set_target_properties(main PROPERTIES OUTPUT bergamot-cli RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}")
-target_compile_options(main PUBLIC ${ALL_WARNINGS})
-set(EXECUTABLES ${EXECUTABLES} main)
-target_link_libraries(main bergamot-translator marian ${MARIAN_CUDA_LIB} ${EXT_LIBS} ssplit pcrecpp.a pcre.a)
+add_executable(service-cli main-mts.cpp)
+target_link_libraries(service-cli PRIVATE bergamot-translator)
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 025ef3d9c..24b9a7b85 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -13,7 +13,7 @@ add_library(bergamot-translator STATIC
     translation_result.cpp
 )
 
-target_link_libraries(bergamot-translator marian)
+target_link_libraries(bergamot-translator marian ${MARIAN_CUDA_LIB} ${EXT_LIBS} ssplit pcrecpp.a pcre.a)
 target_include_directories(bergamot-translator
     PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}
     PRIVATE ${CMAKE_SOURCE_DIR}

From e75bd7eb57da3d0c407184d531911e95c1d2c23c Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 22 Jan 2021 11:31:20 +0000
Subject: [PATCH 032/442] Adding vim temporary files to .gitignore

---
 .gitignore | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 .gitignore

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000000000..e63aee1e1
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,4 @@
+# vim temporary files
+*.swp
+*.swo
+

From 3b6b9cd2bf2328a397366faa2305737240b8c854 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 22 Jan 2021 11:51:49 +0000
Subject: [PATCH 033/442] Updating README.md with instructions to run
 service-cli

---
 README.md | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index fbbbe7b46..52f60b287 100644
--- a/README.md
+++ b/README.md
@@ -13,5 +13,48 @@ $ make -j
 
 ```
 
-## Using Bergamot Translator
+## Usage 
+
+### Bergamot Translator
+
 The build will generate the library that can be linked to any project. All the public header files are specified in `src` folder.
+
+### `service-cli`
+
+An executable `service-cli` is generated by the build in the `app` folder and
+provides command line interface to the underlying translator. The models
+required to run the command-line are available at
+[data.statmt.org/bergamot/models/](http://data.statmt.org/bergamot/models/).
+The following example uses an English to German tiny11 student model, available
+at:
+
+* [data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz](http://data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz)
+
+```bash
+MODEL_DIR=... # path to where the model-files are.
+ARGS=(
+    -m $MODEL_DIR/model.intgemm.alphas.bin # Path to model file.
+    --vocabs 
+        $MODEL_DIR/vocab.deen.spm # source-vocabulary
+        $MODEL_DIR/vocab.deen.spm # target-vocabulary
+
+    # The following increases speed through one-best-decoding, shortlist and quantization.
+    --beam-size 1 --skip-cost --shortlist $MODEL_DIR/lex.s2t.gz 50 50 --int8shiftAlphaAll 
+
+    # Number of CPU threads (workers to launch). Parallelizes over cores and improves speed.
+    --cpu-threads 4
+
+    # Hyperparameters of how many tokens to be accounted for in a batch and maximum tokens in a sentence.
+    --max-input-sentence-tokens 1024 --max-input-tokens 1024 
+
+    # Three modes are supported
+    #   - sentence: One sentence per line
+    #   - paragraph: One paragraph per line.
+    #   - wrapped text: Paragraphs are separated by empty line.
+
+    --ssplit-mode paragraph 
+
+)
+
+./app/service-cli "${ARGS[@]}" < path-to-input-file
+```

From c8fc004452d5a90fe9405fce65badb620080aa9e Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Fri, 22 Jan 2021 12:44:08 +0100
Subject: [PATCH 034/442] Improved 3rd party header inclusion and library
 linking

---
 3rd_party/CMakeLists.txt      | 25 +++++--------------------
 src/translator/CMakeLists.txt |  3 ++-
 src/translator/textops.h      |  2 +-
 3 files changed, 8 insertions(+), 22 deletions(-)

diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index a5aed0689..6d5a5c926 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -1,26 +1,11 @@
 add_subdirectory(marian-dev)
 add_subdirectory(ssplit-cpp)
 
-include_directories(ssplit-cpp/src)
-
-# Add include directories for marian target to be able to use it anywhere in the
-# project without explicitly specifying its include directories. Once marian
-# fixes this problem, it can be removed.
-
+# Add include directories for 3rd party targets to be able to use it anywhere in the
+# project without explicitly specifying their include directories. Once they
+# fixe this problem, it can be removed.
 get_property(INCDIRS DIRECTORY marian-dev/src PROPERTY INCLUDE_DIRECTORIES)
 target_include_directories(marian PUBLIC ${INCDIRS})
 
-
-get_property(INCLUDE_DIRECTORIES DIRECTORY . PROPERTY INCLUDE_DIRECTORIES)
-set(INCLUDE_DIRECTORIES ${INCLUDE_DIRECTORIES} PARENT_SCOPE)
-
-# Required to enable MKL, at least
-get_directory_property(EXT_LIBS DIRECTORY marian-dev DEFINITION EXT_LIBS)
-set(EXT_LIBS ${EXT_LIBS} PARENT_SCOPE)
-
-# Compilation flags
-get_directory_property(CMAKE_C_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_C_FLAGS)
-get_directory_property(CMAKE_CXX_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_CXX_FLAGS)
-set(CMAKE_C_FLAGS ${CMAKE_C_FLAGS} PARENT_SCOPE)
-set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} PARENT_SCOPE)
-
+get_property(INCLUDE_DIRECTORIES DIRECTORY ssplit-cpp/src PROPERTY INCLUDE_DIRECTORIES)
+target_include_directories(ssplit PUBLIC ${INCLUDE_DIRECTORIES})
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 24b9a7b85..25dc77210 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -13,7 +13,8 @@ add_library(bergamot-translator STATIC
     translation_result.cpp
 )
 
-target_link_libraries(bergamot-translator marian ${MARIAN_CUDA_LIB} ${EXT_LIBS} ssplit pcrecpp.a pcre.a)
+target_link_libraries(bergamot-translator marian ssplit)
+
 target_include_directories(bergamot-translator
     PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}
     PRIVATE ${CMAKE_SOURCE_DIR}
diff --git a/src/translator/textops.h b/src/translator/textops.h
index e5c07b6b7..79a504013 100644
--- a/src/translator/textops.h
+++ b/src/translator/textops.h
@@ -9,7 +9,7 @@
 #include "data/sentencepiece_vocab.h"
 #include "data/shortlist.h"
 #include "definitions.h"
-#include "ssplit/ssplit.h"
+#include "ssplit.h"
 
 #include <cassert>
 #include <iostream>

From 1c3b656852641457a2675ffd9aa1aa3fa3dcfb3a Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Fri, 22 Jan 2021 15:53:19 +0100
Subject: [PATCH 035/442] Removed a redundant directory inclusion in CMakeFile

---
 src/translator/CMakeLists.txt | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 25dc77210..27158a786 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -16,7 +16,6 @@ add_library(bergamot-translator STATIC
 target_link_libraries(bergamot-translator marian ssplit)
 
 target_include_directories(bergamot-translator
-    PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}
     PRIVATE ${CMAKE_SOURCE_DIR}
     PUBLIC ${CMAKE_SOURCE_DIR}/src)
 

From 988e76baf973723daf694a1eed768bd20a86233d Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 22 Jan 2021 15:13:30 +0000
Subject: [PATCH 036/442] Removing Exception to fix Apple compile

---
 src/translator/pcqueue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/translator/pcqueue.h b/src/translator/pcqueue.h
index 512932560..f0b354145 100644
--- a/src/translator/pcqueue.h
+++ b/src/translator/pcqueue.h
@@ -50,12 +50,12 @@ class Semaphore {
   }
 
   void wait() {
-    ABORT_IF(KERN_SUCCESS != semaphore_wait(back_), Exception,
+    ABORT_IF(KERN_SUCCESS != semaphore_wait(back_),
              "Wait for semaphore failed");
   }
 
   void post() {
-    ABORT_IF(KERN_SUCCESS != semaphore_signal(back_), Exception,
+    ABORT_IF(KERN_SUCCESS != semaphore_signal(back_),
              "Could not post to semaphore");
   }
 

From 7e2eb02e18cb029f599292f536b32964e854daf5 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 22 Jan 2021 18:17:10 +0000
Subject: [PATCH 037/442] CI and Associated Changes

Enables Mac and Ubuntu CPU only builds through GitHub CI. CI scripts are
copied from marian-dev with necessary changes.

3rd-party/marian-dev is modified to meet C++17 requirements modifying
for half_float.
---
 .github/workflows/macos.yml   |  59 +++++++++++++++
 .github/workflows/ubuntu.yml  | 124 ++++++++++++++++++++++++++++++++
 .github/workflows/windows.yml | 130 ++++++++++++++++++++++++++++++++++
 3rd_party/CMakeLists.txt      |   6 ++
 3rd_party/marian-dev          |   2 +-
 5 files changed, 320 insertions(+), 1 deletion(-)
 create mode 100644 .github/workflows/macos.yml
 create mode 100644 .github/workflows/ubuntu.yml
 create mode 100644 .github/workflows/windows.yml

diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml
new file mode 100644
index 000000000..4a34a3cd7
--- /dev/null
+++ b/.github/workflows/macos.yml
@@ -0,0 +1,59 @@
+name: MacOS
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  build-macos:
+    name: MacOS CPU-only
+    runs-on: macos-10.15
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Install dependencies
+      run: brew install openblas protobuf
+
+    # Openblas location is exported explicitly because openblas is keg-only,
+    # which means it was not symlinked into /usr/local/.
+    # CMake cannot find BLAS on GitHub runners if Marian is being compiled
+    # statically, hence USE_STATIC_LIBS=off
+    - name: Configure CMake
+      run: |
+        export LDFLAGS="-L/usr/local/opt/openblas/lib"
+        export CPPFLAGS="-I/usr/local/opt/openblas/include"
+        mkdir -p build
+        cd build
+        cmake .. \
+          -DCOMPILE_CPU=on \
+          -DCOMPILE_CUDA=off \
+          -DCOMPILE_EXAMPLES=on \
+          -DCOMPILE_SERVER=on \
+          -DCOMPILE_TESTS=on \
+          -DUSE_FBGEMM=on \
+          -DUSE_SENTENCEPIECE=on \
+          -DUSE_STATIC_LIBS=off
+
+    - name: Compile
+      working-directory: build
+      run: make -j2
+
+    # Removing unit-tests, taken care of in browsermt/marian-dev
+    # - name: Run unit tests
+    # - working-directory: build
+    # - run: make test
+
+    - name: Print versions
+      working-directory: build
+      run: |
+        ./marian --version
+        ./marian-decoder --version
+        ./marian-scorer --version
+        ./spm_encode --version
+
diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml
new file mode 100644
index 000000000..88e72f780
--- /dev/null
+++ b/.github/workflows/ubuntu.yml
@@ -0,0 +1,124 @@
+name: Ubuntu
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  build-ubuntu:
+    strategy:
+      matrix:
+        include:
+          # Ubuntu CPU-only build
+          - name: "Ubuntu CPU-only"
+            os: ubuntu-latest
+            cuda: ""
+            gcc: 7
+            cpu: true
+            gpu: false
+          # GPU Builds are commented out, for bergamot-translator CI runs.
+          # Ubuntu GPU-only build
+          # - name: "Ubuntu GPU-only"
+          #   os: ubuntu-latest
+          #   cuda: "10.2"
+          #   gcc: 7
+          #   cpu: false
+          #   gpu: true
+          # Ubuntu 20.04 supports CUDA 11+
+          #- name: "Ubuntu 20.04 CUDA 11.0 gcc-9"
+            #os: ubuntu-20.04
+            #cuda: "11.0"
+            #gcc: 9
+            #cpu: false
+            #gpu: true
+          # Ubuntu 18.04 supports CUDA 10.1+
+          # - name: "Ubuntu 18.04 CUDA 10.2 gcc-8"
+          #   os: ubuntu-18.04
+          #   cuda: "10.2"
+          #   gcc: 8
+          #   cpu: true
+          #   gpu: true
+          # Ubuntu 16.04 supports CUDA 8+
+          # - name: "Ubuntu 16.04 CUDA 9.2 gcc-7"
+          #   os: ubuntu-16.04
+          #   cuda: "9.2"
+          #   gcc: 7
+          #   cpu: true
+          #   gpu: true
+
+    runs-on: ${{ matrix.os }}
+    name: ${{ matrix.name }}
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    # The following packages are already installed on GitHub-hosted runners:
+    # build-essential openssl libssl-dev
+    # No need to install libprotobuf{17,10,9v5} on Ubuntu {20,18,16}.04 because
+    # it is installed together with libprotobuf-dev
+    - name: Install dependencies
+      run: sudo apt-get install -y libgoogle-perftools-dev libprotobuf-dev protobuf-compiler
+
+    # https://software.intel.com/content/www/us/en/develop/articles/installing-intel-free-libs-and-python-apt-repo.html
+    - name: Install MKL
+      run: |
+        wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB" | sudo apt-key add -
+        sudo sh -c "echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list"
+        sudo apt-get update -o Dir::Etc::sourcelist="/etc/apt/sources.list.d/intel-mkl.list"
+        sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
+      if: matrix.cpu == true
+
+    # The script simplifies installation of different versions of CUDA
+    - name: Install CUDA
+      run: ./3rd_party/marian-dev/scripts/ci/install_cuda_ubuntu.sh ${{ matrix.cuda }}
+      if: matrix.gpu == true
+
+    # Boost is installed on GitHub-hosted runners in a non-standard location
+    # https://github.com/actions/virtual-environments/issues/687#issuecomment-610471671
+    - name: Configure CMake
+      run: |
+        mkdir -p build
+        cd build
+        CC=/usr/bin/gcc-${{ matrix.gcc }} CXX=/usr/bin/g++-${{ matrix.gcc }} CUDAHOSTCXX=/usr/bin/g++-${{ matrix.gcc }} \
+        cmake .. \
+          -DBoost_ARCHITECTURE=-x64 \
+          -DBOOST_INCLUDEDIR=$BOOST_ROOT_1_72_0/include \
+          -DBOOST_LIBRARYDIR=$BOOST_ROOT_1_72_0/lib \
+          -DBOOST_ROOT=$BOOST_ROOT_1_72_0 \
+          -DCMAKE_BUILD_TYPE=Release \
+          -DCOMPILE_CPU=${{ matrix.cpu }} \
+          -DCOMPILE_CUDA=${{ matrix.gpu }} \
+          -DCOMPILE_EXAMPLES=on \
+          -DCOMPILE_SERVER=on \
+          -DCOMPILE_TESTS=on \
+          -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-${{ matrix.cuda }} \
+          -DUSE_FBGEMM=${{ matrix.cpu }} \
+          -DUSE_SENTENCEPIECE=on \
+          -DUSE_STATIC_LIBS=on \
+
+    - name: Compile
+      working-directory: build
+      run: make -j2
+
+    # Removing unit-tests, taken care of in browsermt/marian-dev
+    # TODO: add a flag to CMake to compile unit tests only on CPU
+    # - name: Run unit tests
+    #   working-directory: build
+    #   run: make test
+    #   # GitHub-hosted VMs do not have GPUs, so can not be run in CUDA builds
+    #   if: matrix.gpu == false
+
+    - name: Print versions
+      working-directory: build
+      run: |
+        ./marian --version
+        ./marian-decoder --version
+        ./marian-scorer --version
+        ./marian-server --version
+        ./spm_encode --version
+
diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
new file mode 100644
index 000000000..cc0b1bef5
--- /dev/null
+++ b/.github/workflows/windows.yml
@@ -0,0 +1,130 @@
+name: Windows
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+env:
+  MKL_URL: "https://romang.blob.core.windows.net/mariandev/ci/mkl-2020.1-windows-static.zip"
+
+jobs:
+  build-windows:
+    strategy:
+      matrix:
+        include:
+          # Windows CPU-only build
+          - name: "Windows CPU-only"
+            cuda: ""
+            gpu: false
+          # GPU Builds are commented out, for bergamot-translator CI runs.
+          # Windows CPU+GPU build
+          # - name: "Windows CPU+CUDA"
+          #   cuda: "10.2"
+          #   gpu: true
+
+    runs-on: windows-2019
+    name: ${{ matrix.name }}
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Download MKL
+      run: |
+        # Wget retries downloading files and is faster than Invoke-WebRequest
+        C:\msys64\usr\bin\wget.exe -nv ${{ env.MKL_URL }} -O mkl.zip
+        Expand-Archive -Force mkl.zip ${{ github.workspace }}\mkl
+        # Set MKLROOT environment variable so that CMake can find MKL
+        echo "MKLROOT=${{ github.workspace }}\mkl" | Out-File -FilePath $env:GITHUB_ENV  -Encoding utf8 -Append
+      shell: powershell
+
+    - name: Install CUDA
+      run: |
+        .\3rd_party\marian-dev\scripts\ci\install_cuda_windows.ps1 "10.2"
+        # Set CUDA_PATH environment variable so that CMake can find CUDA
+        echo "CUDA_PATH=$env:CUDA_PATH" | Out-File -FilePath $env:GITHUB_ENV  -Encoding utf8 -Append
+        echo "$env:CUDA_PATH/bin"       | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
+      shell: powershell
+      if: matrix.gpu == true
+
+    - name: Prepare vcpkg
+      uses: lukka/run-vcpkg@v4
+      with:
+        vcpkgArguments: protobuf
+        vcpkgGitCommitId: 6185aa76504a5025f36754324abf307cc776f3da
+        vcpkgDirectory: ${{ github.workspace }}/vcpkg/
+        vcpkgTriplet: x64-windows-static
+
+    # Windows CUDA builds use USE_NCCL=off due to compilation errors.
+    - name: Build Debug
+      uses: lukka/run-cmake@v3
+      with:
+        buildDirectory: ${{ github.workspace }}/build/Debug
+        cmakeAppendedArgs: '-G Ninja
+          -DCMAKE_BUILD_TYPE="Debug"
+          -DOPENSSL_USE_STATIC_LIBS="TRUE"
+          -DOPENSSL_MSVC_STATIC_RT="TRUE"
+          -DCOMPILE_CPU="TRUE"
+          -DCOMPILE_CUDA="${{ matrix.gpu }}"
+          -DCOMPILE_SERVER="FALSE"
+          -DCOMPILE_TESTS="TRUE"
+          -DUSE_FBGEMM="TRUE"
+          -DUSE_MPI="FALSE"
+          -DUSE_NCCL="FALSE"
+          -DUSE_SENTENCEPIECE="TRUE"
+          -DUSE_STATIC_LIBS="TRUE"'
+        cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
+        cmakeListsTxtPath: ${{ github.workspace }}/CMakeLists.txt
+        useVcpkgToolchainFile: true
+      # Building in Debug is sufficient for the all-in CPU+GPU compilation;
+      # its main purpose is to detect warnings that the Release build is not
+      # able to find sometimes.
+      if: matrix.gpu == true
+
+    # Windows CUDA builds use USE_NCCL=off due to compilation errors
+    # Boost is pre-installed on Azure/GitHub-hosted Windows runners
+    # https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#boost
+    # (not used yet)
+    - name: Build Release
+      uses: lukka/run-cmake@v3
+      with:
+        buildDirectory: ${{ github.workspace }}/build/
+        cmakeAppendedArgs: '-G Ninja
+          -DBOOST_ROOT="$(BOOST_ROOT_1_72_0)"
+          -DBOOST_INCLUDEDIR="$(BOOST_ROOT_1_72_0)/include"
+          -DBOOST_LIBRARYDIR="$(BOOST_ROOT_1_72_0)/lib"
+          -DCMAKE_BUILD_TYPE="Release"
+          -DOPENSSL_USE_STATIC_LIBS="TRUE"
+          -DOPENSSL_MSVC_STATIC_RT="TRUE"
+          -DCOMPILE_CPU="TRUE"
+          -DCOMPILE_CUDA="${{ matrix.gpu }}"
+          -DCOMPILE_SERVER="FALSE"
+          -DCOMPILE_TESTS="TRUE"
+          -DUSE_FBGEMM="TRUE"
+          -DUSE_MPI="FALSE"
+          -DUSE_NCCL="FALSE"
+          -DUSE_SENTENCEPIECE="TRUE"
+          -DUSE_STATIC_LIBS="TRUE"'
+        cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
+        cmakeListsTxtPath: ${{ github.workspace }}/CMakeLists.txt
+        useVcpkgToolchainFile: true
+
+    # Removing unit-tests, taken care of in browsermt/marian-dev
+    # - name: Run unit tests
+    #   working-directory: build/
+    #   run: ctest
+    #   # Not run in GPU builds because GitHub-hosted VMs do not have GPUs
+    #   if: matrix.gpu == false
+
+    - name: Print versions
+      working-directory: build/
+      run: |
+        .\marian.exe --version
+        .\marian-decoder.exe --version
+        .\marian-scorer.exe --version
+        dir *.exe
+      shell: cmd
diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index 6d5a5c926..644ac52de 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -9,3 +9,9 @@ target_include_directories(marian PUBLIC ${INCDIRS})
 
 get_property(INCLUDE_DIRECTORIES DIRECTORY ssplit-cpp/src PROPERTY INCLUDE_DIRECTORIES)
 target_include_directories(ssplit PUBLIC ${INCLUDE_DIRECTORIES})
+
+# Compilation flags 
+get_directory_property(CMAKE_C_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_C_FLAGS) 
+get_directory_property(CMAKE_CXX_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_CXX_FLAGS) 
+set(CMAKE_C_FLAGS ${CMAKE_C_FLAGS} PARENT_SCOPE)    
+set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} PARENT_SCOPE)    
diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 96d5a712d..ee56e02f0 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 96d5a712d3b8bc56f120ba5220365f955719f4d4
+Subproject commit ee56e02f0525a4651157a07f74b44f456db14c8c

From cd025e9f651bab6b901e0306690b8bed5625165a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sat, 23 Jan 2021 14:39:08 +0000
Subject: [PATCH 038/442] CI scripts: master -> main

---
 .github/workflows/macos.yml   | 4 ++--
 .github/workflows/ubuntu.yml  | 4 ++--
 .github/workflows/windows.yml | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml
index 4a34a3cd7..8ccdecaf5 100644
--- a/.github/workflows/macos.yml
+++ b/.github/workflows/macos.yml
@@ -2,9 +2,9 @@ name: MacOS
 
 on:
   push:
-    branches: [ master ]
+    branches: [ main ]
   pull_request:
-    branches: [ master ]
+    branches: [ main ]
 
 jobs:
   build-macos:
diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml
index 88e72f780..240efd2c3 100644
--- a/.github/workflows/ubuntu.yml
+++ b/.github/workflows/ubuntu.yml
@@ -2,9 +2,9 @@ name: Ubuntu
 
 on:
   push:
-    branches: [ master ]
+    branches: [ main ]
   pull_request:
-    branches: [ master ]
+    branches: [ main ]
 
 jobs:
   build-ubuntu:
diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index cc0b1bef5..ef9ad25d1 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -2,9 +2,9 @@ name: Windows
 
 on:
   push:
-    branches: [ master ]
+    branches: [ main ]
   pull_request:
-    branches: [ master ]
+    branches: [ main ]
 
 env:
   MKL_URL: "https://romang.blob.core.windows.net/mariandev/ci/mkl-2020.1-windows-static.zip"

From 69adc7af777b5e672d54345f6e7bec5d915faade Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sun, 24 Jan 2021 21:46:47 +0000
Subject: [PATCH 039/442] Changing code-style to clang-format-google

---
 app/main.cpp                                  |  34 +++---
 src/AbstractTranslationModel.h                |  84 ++++++++------
 src/QualityScore.h                            |  29 ++---
 src/TranslationModelConfiguration.h           |  87 +++++++-------
 src/TranslationRequest.h                      | 108 ++++++++++--------
 src/TranslationResult.h                       |  95 +++++++--------
 src/translator/AbstractTranslationModel.cpp   |  10 +-
 src/translator/TranslationModel.cpp           |  24 ++--
 src/translator/TranslationModel.h             |  78 +++++++------
 ...TranslationModelConfigToOptionsAdaptor.cpp |  14 ++-
 .../TranslationModelConfigToOptionsAdaptor.h  |  22 ++--
 11 files changed, 302 insertions(+), 283 deletions(-)

diff --git a/app/main.cpp b/app/main.cpp
index dc808228f..bb0fa34e2 100644
--- a/app/main.cpp
+++ b/app/main.cpp
@@ -7,29 +7,29 @@
 
 #include <iostream>
 
-#include "TranslationModelConfiguration.h"
 #include "AbstractTranslationModel.h"
+#include "TranslationModelConfiguration.h"
 #include "TranslationRequest.h"
 #include "TranslationResult.h"
 
+int main(int argc, char **argv) {
 
-int main(int argc, char** argv) {
-
-	// Create an instance of AbstractTranslationModel with a dummy model configuration
-	TranslationModelConfiguration config("dummy_modelFilePath",
-				"dummy_sourceVocabPath",
-				"dummy_targetVocabPath");
-	std::shared_ptr<AbstractTranslationModel> model =
-			AbstractTranslationModel::createInstance(config);
+  // Create an instance of AbstractTranslationModel with a dummy model
+  // configuration
+  TranslationModelConfiguration config(
+      "dummy_modelFilePath", "dummy_sourceVocabPath", "dummy_targetVocabPath");
+  std::shared_ptr<AbstractTranslationModel> model =
+      AbstractTranslationModel::createInstance(config);
 
-	// Call to translate a dummy (empty) texts with a dummy (empty) translation request
-	TranslationRequest req;
-	std::vector<std::string> texts;
-	auto result = model->translate(std::move(texts), req);
+  // Call to translate a dummy (empty) texts with a dummy (empty) translation
+  // request
+  TranslationRequest req;
+  std::vector<std::string> texts;
+  auto result = model->translate(std::move(texts), req);
 
-	// Resolve the future and get the actual result
-	std::vector<TranslationResult> res = result.get();
+  // Resolve the future and get the actual result
+  std::vector<TranslationResult> res = result.get();
 
-	std::cout << "Count is: " << res.size() << std::endl;
-	return 0;
+  std::cout << "Count is: " << res.size() << std::endl;
+  return 0;
 }
diff --git a/src/AbstractTranslationModel.h b/src/AbstractTranslationModel.h
index ddadc07bf..b76aeebed 100644
--- a/src/AbstractTranslationModel.h
+++ b/src/AbstractTranslationModel.h
@@ -1,61 +1,69 @@
 /*
  * AbstractTranslationModel.h
  *
- * An interface for a translation model for translating a plain (without any markups and emojis) UTF-8 encoded text.
- * The model supports translation from 1 source language to 1 target language. There can be different implementations
+ * An interface for a translation model for translating a plain (without any
+ * markups and emojis) UTF-8 encoded text. The model supports translation from 1
+ * source language to 1 target language. There can be different implementations
  * of this interface.
  */
 
 #ifndef SRC_TRANSLATOR_ABSTRACTTRANSLATIONMODEL_H_
 #define SRC_TRANSLATOR_ABSTRACTTRANSLATIONMODEL_H_
 
-#include <vector>
-#include <string>
 #include <future>
 #include <memory>
+#include <string>
+#include <vector>
 
 #include "TranslationModelConfiguration.h"
 #include "TranslationRequest.h"
 #include "TranslationResult.h"
 
-/* An interface for a translation model for translating a plain (without any markups and emojis) UTF-8 encoded text.
- * The model supports translation from 1 source language to 1 target language.
+/* An interface for a translation model for translating a plain (without any
+ * markups and emojis) UTF-8 encoded text. The model supports translation from 1
+ * source language to 1 target language.
  */
 class AbstractTranslationModel {
 public:
+  /* A Factory method to create and return an instance of an implementation of
+   * AbstractTranslationModel. The instance is created using translation model
+   * configuration (TranslationModelConfiguration).
+   */
+  static std::shared_ptr<AbstractTranslationModel>
+  createInstance(const TranslationModelConfiguration &config);
+
+  AbstractTranslationModel() = default;
+
+  virtual ~AbstractTranslationModel() = default;
+
+  /* This method performs translation on a list of (UTF-8 encoded) texts and
+   * returns a list of results in the same order. Each text entry can either be
+   * a word, a phrase, a sentence or a list of sentences and should contain
+   * plain text (without any markups or emojis). Additional information related
+   * to the translated text can be requested via TranslationRequest which is
+   * applied equally to each text entry.
+   *
+   * The translated text corresponding to each text entry and the additional
+   * information (as specified in the TranslationRequest) is encapsulated and
+   * returned in TranslationResult.
+   *
+   * The API splits each text entry into sentences internally, which are then
+   * translated independent of each other. The translated sentences are then
+   * joined together and returned in TranslationResult. Please refer to the
+   * TranslationRequest class to find out what additional information can be
+   * requested. The alignment information can only be requested if the model
+   * supports it (check isAlignmentSupported() API).
+   *
+   * The texts argument will become empty after the execution of this API (each
+   * entry of texts list will be moved to its corresponding TranslationResult
+   * object).
+   */
+  virtual std::future<std::vector<TranslationResult>>
+  translate(std::vector<std::string> &&texts, TranslationRequest request) = 0;
 
-	/* A Factory method to create and return an instance of an implementation of
-	 * AbstractTranslationModel. The instance is created using translation model configuration
-	 * (TranslationModelConfiguration).
-	 */
-	static std::shared_ptr<AbstractTranslationModel>
-	createInstance(const TranslationModelConfiguration& config);
-
-	AbstractTranslationModel() = default;
-
-	virtual ~AbstractTranslationModel() = default;
-
-	/* This method performs translation on a list of (UTF-8 encoded) texts and returns a list of results in the same order.
-	 * Each text entry can either be a word, a phrase, a sentence or a list of sentences and should contain plain text
-	 * (without any markups or emojis). Additional information related to the translated text can be requested via
-	 * TranslationRequest which is applied equally to each text entry.
-	 *
-	 * The translated text corresponding to each text entry and the additional information (as specified in the
-	 * TranslationRequest) is encapsulated and returned in TranslationResult.
-	 *
-	 * The API splits each text entry into sentences internally, which are then translated independent of each other.
-	 * The translated sentences are then joined together and returned in TranslationResult.
-	 * Please refer to the TranslationRequest class to find out what additional information can be requested.
-	 * The alignment information can only be requested if the model supports it (check isAlignmentSupported() API).
-	 *
-	 * The texts argument will become empty after the execution of this API (each entry of texts list will be moved to its
-	 * corresponding TranslationResult object).
-	 */
-	virtual std::future<std::vector<TranslationResult>> translate(
-			std::vector<std::string> &&texts, TranslationRequest request) = 0;
-
-	/* Check if the model can provide alignment information b/w original and translated text. */
-	virtual bool isAlignmentSupported() const = 0;
+  /* Check if the model can provide alignment information b/w original and
+   * translated text. */
+  virtual bool isAlignmentSupported() const = 0;
 };
 
 #endif /* SRC_TRANSLATOR_ABSTRACTTRANSLATIONMODEL_H_ */
diff --git a/src/QualityScore.h b/src/QualityScore.h
index 020aebc8e..3ad6349bd 100644
--- a/src/QualityScore.h
+++ b/src/QualityScore.h
@@ -6,31 +6,32 @@
 #ifndef SRC_TRANSLATOR_QUALITYSCORE_H_
 #define SRC_TRANSLATOR_QUALITYSCORE_H_
 
-#include <vector>
 #include <string>
+#include <vector>
 
-
-/* All possible Granularities for which Quality Scores can be returned for translated text. */
+/* All possible Granularities for which Quality Scores can be returned for
+ * translated text. */
 enum class QualityScoreGranularity {
-	WORD, SENTENCE, NONE,
+  WORD,
+  SENTENCE,
+  NONE,
 };
 
-/* This class represents the Quality Scores for various spans of a translated text at a specific granularity. */
+/* This class represents the Quality Scores for various spans of a translated
+ * text at a specific granularity. */
 class QualityScore {
 private:
+  // Sections of the translated text for the Quality Scores.
+  std::vector<std::string_view> textViews;
 
-	// Sections of the translated text for the Quality Scores.
-	std::vector<std::string_view> textViews;
+  // Quality Scores corresponding to each entry of textViews in the same order
+  std::vector<float> textScores;
 
-	// Quality Scores corresponding to each entry of textViews in the same order
-	std::vector<float> textScores;
-
-	// Granularity of the text for the Quality scores above
-	QualityScoreGranularity textGranularity;
+  // Granularity of the text for the Quality scores above
+  QualityScoreGranularity textGranularity;
 
 public:
-	// ToDo: Public Methods
+  // ToDo: Public Methods
 };
 
-
 #endif /* SRC_TRANSLATOR_QUALITYSCORE_H_ */
diff --git a/src/TranslationModelConfiguration.h b/src/TranslationModelConfiguration.h
index 8c6582454..f4a5572ea 100644
--- a/src/TranslationModelConfiguration.h
+++ b/src/TranslationModelConfiguration.h
@@ -8,61 +8,54 @@
 
 #include <string>
 
-/* This class encapsulates the configuration that is required by a translation model to perform
- * translation.
+/* This class encapsulates the configuration that is required by a translation
+ * model to perform translation.
  */
 class TranslationModelConfiguration {
 public:
-
-	// Constructor
-	TranslationModelConfiguration(const std::string &modelFilePath,
-			const std::string &sourceVocabPath,
-			const std::string &targetVocabPath) :
-				modelPath(modelFilePath),
-				sourceLanguageVocabPath(sourceVocabPath),
-				targetLanguageVocabPath(targetVocabPath) {
-	}
-
-	// Copy constructor
-	TranslationModelConfiguration(const TranslationModelConfiguration &rhs) :
-			modelPath(rhs.modelPath),
-			sourceLanguageVocabPath(rhs.sourceLanguageVocabPath),
-			targetLanguageVocabPath(rhs.targetLanguageVocabPath) {
-	}
-
-	// Move constructor
-	TranslationModelConfiguration(TranslationModelConfiguration &&rhs) :
-			modelPath(std::move(rhs.modelPath)),
-			sourceLanguageVocabPath(std::move(rhs.sourceLanguageVocabPath)),
-			targetLanguageVocabPath(std::move(rhs.targetLanguageVocabPath)) {
-	}
-
-	// Return the path of the model file
-	const std::string& getModelFilePath() const {
-		return modelPath;
-	}
-
-	// Return the path of the source language vocabulary file
-	const std::string& getSourceVocabularyPath() const {
-		return sourceLanguageVocabPath;
-	}
-
-	// Return the path of the target language vocabulary file
-	const std::string& getTargetVocabularyPath() const {
-		return targetLanguageVocabPath;
-	}
+  // Constructor
+  TranslationModelConfiguration(const std::string &modelFilePath,
+                                const std::string &sourceVocabPath,
+                                const std::string &targetVocabPath)
+      : modelPath(modelFilePath), sourceLanguageVocabPath(sourceVocabPath),
+        targetLanguageVocabPath(targetVocabPath) {}
+
+  // Copy constructor
+  TranslationModelConfiguration(const TranslationModelConfiguration &rhs)
+      : modelPath(rhs.modelPath),
+        sourceLanguageVocabPath(rhs.sourceLanguageVocabPath),
+        targetLanguageVocabPath(rhs.targetLanguageVocabPath) {}
+
+  // Move constructor
+  TranslationModelConfiguration(TranslationModelConfiguration &&rhs)
+      : modelPath(std::move(rhs.modelPath)),
+        sourceLanguageVocabPath(std::move(rhs.sourceLanguageVocabPath)),
+        targetLanguageVocabPath(std::move(rhs.targetLanguageVocabPath)) {}
+
+  // Return the path of the model file
+  const std::string &getModelFilePath() const { return modelPath; }
+
+  // Return the path of the source language vocabulary file
+  const std::string &getSourceVocabularyPath() const {
+    return sourceLanguageVocabPath;
+  }
+
+  // Return the path of the target language vocabulary file
+  const std::string &getTargetVocabularyPath() const {
+    return targetLanguageVocabPath;
+  }
 
 private:
-	// Path to the translation model file
-	const std::string modelPath;
+  // Path to the translation model file
+  const std::string modelPath;
 
-	// Path to the source vocabulary file to be used by the model
-	const std::string sourceLanguageVocabPath;
+  // Path to the source vocabulary file to be used by the model
+  const std::string sourceLanguageVocabPath;
 
-	// Path to the target vocabulary file to be used by the model
-	const std::string targetLanguageVocabPath;
+  // Path to the target vocabulary file to be used by the model
+  const std::string targetLanguageVocabPath;
 
-	// ToDo: Add other user configurable options (e.g. min batch size)
+  // ToDo: Add other user configurable options (e.g. min batch size)
 };
 
 #endif /* SRC_TRANSLATOR_TRANSLATIONMODELCONFIGURATION_H_ */
diff --git a/src/TranslationRequest.h b/src/TranslationRequest.h
index b19cc892d..6d449bbab 100644
--- a/src/TranslationRequest.h
+++ b/src/TranslationRequest.h
@@ -1,7 +1,8 @@
 /*
  * TranslationRequest.h
  *
- *  This file defines the translation request class to be used in AbstractTranslationModel::translate() API.
+ *  This file defines the translation request class to be used in
+ * AbstractTranslationModel::translate() API.
  */
 
 #ifndef SRC_TRANSLATOR_TRANSLATIONREQUEST_H_
@@ -9,66 +10,75 @@
 
 #include "QualityScore.h"
 
-/* This class specifies the information related to the translated text (e.g. quality of the translation etc.) that
- * can be included in the TranslationResult. These optional requests are set/unset independent of each other i.e. setting
- * any one of them doesn’t have the side effect of setting any of the others.
+/* This class specifies the information related to the translated text (e.g.
+ * quality of the translation etc.) that can be included in the
+ * TranslationResult. These optional requests are set/unset independent of each
+ * other i.e. setting any one of them doesn’t have the side effect of setting
+ * any of the others.
  */
 class TranslationRequest {
 private:
-	// The granularity for which Quality scores of the translated text will be included in TranslationResult.
-	// QualityScoreGranularity::NONE means the scores are not included in TranslationResult.
-	QualityScoreGranularity qualityScoreGranularity = QualityScoreGranularity::NONE;
+  // The granularity for which Quality scores of the translated text will be
+  // included in TranslationResult. QualityScoreGranularity::NONE means the
+  // scores are not included in TranslationResult.
+  QualityScoreGranularity qualityScoreGranularity =
+      QualityScoreGranularity::NONE;
 
-	// A flag to include/exclude the information regarding how individual sentences of original text map to
-	// corresponding translated sentences in joined translated text in the TranslationResult.
-	// An example of sentence mappings:
-	//     originalText (containing 2 sentences)              = "What is your name? My name is Abc."
-	//     translatedText (containing 2 translated sentences) = "Was ist dein Name? Mein Name ist Abc."
-	//     sentenceMappings = [
-	//         {"What is your name?", "Was ist dein Name?"},  // Pair(originalText[0],translatedText[0])
-	//         {"My name is Abc", "Mein Name ist Abc."}       // Pair(originalText[1],translatedText[1])
-	//     ]
-	bool includeSentenceMapping = false;
+  // A flag to include/exclude the information regarding how individual
+  // sentences of original text map to corresponding translated sentences in
+  // joined translated text in the TranslationResult. An example of sentence
+  // mappings:
+  //     originalText (containing 2 sentences)              = "What is your
+  //     name? My name is Abc." translatedText (containing 2 translated
+  //     sentences) = "Was ist dein Name? Mein Name ist Abc." sentenceMappings =
+  //     [
+  //         {"What is your name?", "Was ist dein Name?"},  //
+  //         Pair(originalText[0],translatedText[0])
+  //         {"My name is Abc", "Mein Name ist Abc."}       //
+  //         Pair(originalText[1],translatedText[1])
+  //     ]
+  bool includeSentenceMapping = false;
 
 public:
-	TranslationRequest() {}
+  TranslationRequest() {}
 
-	TranslationRequest(const TranslationRequest& request) :
-		qualityScoreGranularity(request.qualityScoreGranularity),
-		includeSentenceMapping(request.includeSentenceMapping) {
-	}
+  TranslationRequest(const TranslationRequest &request)
+      : qualityScoreGranularity(request.qualityScoreGranularity),
+        includeSentenceMapping(request.includeSentenceMapping) {}
 
-	~TranslationRequest() {}
+  ~TranslationRequest() {}
 
-	/* Set the granularity for which the Quality scores of translated text should be included in the TranslationResult.
-	 * By default (QualityScoreGranularity::NONE), scores are not included.
-	 */
-	void setQualityScoreGranularity(QualityScoreGranularity granularity) {
-		qualityScoreGranularity = granularity;
-	}
+  /* Set the granularity for which the Quality scores of translated text should
+   * be included in the TranslationResult. By default
+   * (QualityScoreGranularity::NONE), scores are not included.
+   */
+  void setQualityScoreGranularity(QualityScoreGranularity granularity) {
+    qualityScoreGranularity = granularity;
+  }
 
-	/* Set to true/false to include/exclude the information regarding how individual sentences of original text map to
-	 * corresponding translated sentences in joined translated text in the TranslationResult. By default (false), this
-	 * information is not included.
-	 */
-	void sentenceMappingInResult(bool includeMapping) {
-		includeSentenceMapping = includeMapping;
-	}
+  /* Set to true/false to include/exclude the information regarding how
+   * individual sentences of original text map to corresponding translated
+   * sentences in joined translated text in the TranslationResult. By default
+   * (false), this information is not included.
+   */
+  void sentenceMappingInResult(bool includeMapping) {
+    includeSentenceMapping = includeMapping;
+  }
 
-	/* Return the granularity for which the Quality scores of the translated text will be included in TranslationResult.
-	 * QualityScoreGranularity::NONE means the scores will not be included.
-	 */
-	QualityScoreGranularity getQualityScoreGranularity() const {
-		return qualityScoreGranularity;
-	}
+  /* Return the granularity for which the Quality scores of the translated text
+   * will be included in TranslationResult. QualityScoreGranularity::NONE means
+   * the scores will not be included.
+   */
+  QualityScoreGranularity getQualityScoreGranularity() const {
+    return qualityScoreGranularity;
+  }
 
-	/* Return whether the information regarding how individual sentences of original text map to corresponding translated
-	 * sentences in joined translated text will be included in the TranslationResult. By default (false) means this
-	 * information will not be included.
-	 */
-	bool sentenceMappingInResult() const {
-		return includeSentenceMapping;
-	}
+  /* Return whether the information regarding how individual sentences of
+   * original text map to corresponding translated sentences in joined
+   * translated text will be included in the TranslationResult. By default
+   * (false) means this information will not be included.
+   */
+  bool sentenceMappingInResult() const { return includeSentenceMapping; }
 };
 
 #endif /* SRC_TRANSLATOR_TRANSLATIONREQUEST_H_ */
diff --git a/src/TranslationResult.h b/src/TranslationResult.h
index 4d231a89b..34858f74c 100644
--- a/src/TranslationResult.h
+++ b/src/TranslationResult.h
@@ -1,76 +1,77 @@
 /*
  * TranslationResult.h
  *
- * The class that represents the result of AbstractTranslationModel::translate() API for each of its text entry and
- * TranslationRequest.
+ * The class that represents the result of AbstractTranslationModel::translate()
+ * API for each of its text entry and TranslationRequest.
  */
 
 #ifndef SRC_TRANSLATOR_TRANSLATIONRESULT_H_
 #define SRC_TRANSLATOR_TRANSLATIONRESULT_H_
 
-#include <vector>
 #include <string>
+#include <vector>
 
 #include "QualityScore.h"
 
-/* This class represents the result of AbstractTranslationModel::translate() API for each of its text entry and
- * TranslationRequest.
+/* This class represents the result of AbstractTranslationModel::translate() API
+ * for each of its text entry and TranslationRequest.
  */
 class TranslationResult {
 public:
-	typedef std::vector<std::pair<std::string_view, std::string_view>> SentenceMappings;
+  typedef std::vector<std::pair<std::string_view, std::string_view>>
+      SentenceMappings;
 
-	TranslationResult(const std::string &original, const std::string &translation) :
-		originalText(original), translatedText(translation) {}
+  TranslationResult(const std::string &original, const std::string &translation)
+      : originalText(original), translatedText(translation) {}
 
-	TranslationResult(std::string &&original, std::string &&translation) :
-		originalText(std::move(original)), translatedText(std::move(translation)) {}
+  TranslationResult(std::string &&original, std::string &&translation)
+      : originalText(std::move(original)),
+        translatedText(std::move(translation)) {}
 
-	/* Return the original text. */
-	const std::string& getOriginalText() const {
-		return originalText;
-	}
+  /* Return the original text. */
+  const std::string &getOriginalText() const { return originalText; }
 
-	/* Return the translated text. */
-	const std::string& getTranslatedText() const {
-		return translatedText;
-	}
+  /* Return the translated text. */
+  const std::string &getTranslatedText() const { return translatedText; }
 
-	/* Return the Quality scores of the translated text. */
-	const QualityScore& getQualityScore() const {
-		return qualityScore;
-	}
+  /* Return the Quality scores of the translated text. */
+  const QualityScore &getQualityScore() const { return qualityScore; }
 
-	/* Return the Sentence mappings (information regarding how individual sentences of originalText map to
-	 * corresponding translated sentences in translatedText).
-	 */
-	const SentenceMappings& getSentenceMappings() const {
-		return sentenceMappings;
-	}
+  /* Return the Sentence mappings (information regarding how individual
+   * sentences of originalText map to corresponding translated sentences in
+   * translatedText).
+   */
+  const SentenceMappings &getSentenceMappings() const {
+    return sentenceMappings;
+  }
 
 private:
-	// Original text (in UTF-8 encoded format) that was supposed to be translated
-	std::string originalText;
+  // Original text (in UTF-8 encoded format) that was supposed to be translated
+  std::string originalText;
 
-	// Translation (in UTF-8 encoded format) of the originalText
-	std::string translatedText;
+  // Translation (in UTF-8 encoded format) of the originalText
+  std::string translatedText;
 
-	// Quality score of the translated text at the granularity specified in TranslationRequest.
-	// It is an optional result (it will have no information if not requested in TranslationRequest)
-	QualityScore qualityScore;
+  // Quality score of the translated text at the granularity specified in
+  // TranslationRequest. It is an optional result (it will have no information
+  // if not requested in TranslationRequest)
+  QualityScore qualityScore;
 
-	// Information regarding how individual sentences of originalText map to corresponding translated sentences
-	// in joined translated text (translatedText)
-	// An example of sentence mapping:
-	//     originalText (contains 2 sentences)              = "What is your name? My name is Abc."
-	//     translatedText (contains 2 translated sentences) = "Was ist dein Name? Mein Name ist Abc."
-	//     sentenceMappings = [
-	//         {"What is your name?", "Was ist dein Name?"},  // Pair(originalText[0],translatedText[0])
-	//         {"My name is Abc", "Mein Name ist Abc."}       // Pair(originalText[1],translatedText[1])
-	//     ]
-	//
-	// It is an optional result (it will be empty if not requested in TranslationRequest).
-	SentenceMappings sentenceMappings;
+  // Information regarding how individual sentences of originalText map to
+  // corresponding translated sentences in joined translated text
+  // (translatedText) An example of sentence mapping:
+  //     originalText (contains 2 sentences)              = "What is your name?
+  //     My name is Abc." translatedText (contains 2 translated sentences) =
+  //     "Was ist dein Name? Mein Name ist Abc." sentenceMappings = [
+  //         {"What is your name?", "Was ist dein Name?"},  //
+  //         Pair(originalText[0],translatedText[0])
+  //         {"My name is Abc", "Mein Name ist Abc."}       //
+  //         Pair(originalText[1],translatedText[1])
+  //     ]
+  //
+  // It is an optional result (it will be empty if not requested in
+  // TranslationRequest).
+  SentenceMappings sentenceMappings;
 };
 
 #endif /* SRC_TRANSLATOR_TRANSLATIONRESULT_H_ */
diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
index 597c592d3..94782fa81 100644
--- a/src/translator/AbstractTranslationModel.cpp
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -12,10 +12,10 @@
 #include "TranslationModel.h"
 #include "TranslationModelConfigToOptionsAdaptor.h"
 
-
 std::shared_ptr<AbstractTranslationModel>
-AbstractTranslationModel::createInstance(const TranslationModelConfiguration& config) {
-	TranslationModelConfigToOptionsAdaptor adaptor;
-	auto options = adaptor.adapt(config);
-	return std::make_shared<TranslationModel>(options);
+AbstractTranslationModel::createInstance(
+    const TranslationModelConfiguration &config) {
+  TranslationModelConfigToOptionsAdaptor adaptor;
+  auto options = adaptor.adapt(config);
+  return std::make_shared<TranslationModel>(options);
 }
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 099d930cd..b3a8fec32 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -8,21 +8,19 @@
 
 #include "TranslationModel.h"
 
-TranslationModel::TranslationModel(std::shared_ptr<marian::Options> options) :
-		configOptions(std::move(options)), AbstractTranslationModel() {
-}
+TranslationModel::TranslationModel(std::shared_ptr<marian::Options> options)
+    : configOptions(std::move(options)), AbstractTranslationModel() {}
 
 TranslationModel::~TranslationModel() {}
 
-std::future<std::vector<TranslationResult>> TranslationModel::translate(
-		std::vector<std::string> &&texts, TranslationRequest request) {
-	//ToDo: Replace this code with the actual implementation
-	return std::async([]() {
-		std::vector<TranslationResult> results;
-		return results;
-	});
+std::future<std::vector<TranslationResult>>
+TranslationModel::translate(std::vector<std::string> &&texts,
+                            TranslationRequest request) {
+  // ToDo: Replace this code with the actual implementation
+  return std::async([]() {
+    std::vector<TranslationResult> results;
+    return results;
+  });
 }
 
-bool TranslationModel::isAlignmentSupported() const {
-	return false;
-}
+bool TranslationModel::isAlignmentSupported() const { return false; }
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index 587926516..ba58969d1 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -7,9 +7,9 @@
 #ifndef SRC_TRANSLATOR_TRANSLATIONMODEL_H_
 #define SRC_TRANSLATOR_TRANSLATIONMODEL_H_
 
-#include <vector>
-#include <string>
 #include <future>
+#include <string>
+#include <vector>
 
 // All 3rd party includes
 #include "3rd_party/marian-dev/src/common/options.h"
@@ -18,47 +18,53 @@
 #include "AbstractTranslationModel.h"
 #include "TranslationModelConfiguration.h"
 
-/* A Translation model that translates a plain (without any markups and emojis) UTF-8 encoded text.
- * This implementation supports translation from 1 source language to 1 target language.
+/* A Translation model that translates a plain (without any markups and emojis)
+ * UTF-8 encoded text. This implementation supports translation from 1 source
+ * language to 1 target language.
  */
-class TranslationModel: public AbstractTranslationModel {
+class TranslationModel : public AbstractTranslationModel {
 public:
-	/* Construct the model using the model configuration options.
-	 */
-	TranslationModel(std::shared_ptr<marian::Options> options);
+  /* Construct the model using the model configuration options.
+   */
+  TranslationModel(std::shared_ptr<marian::Options> options);
 
-	~TranslationModel();
+  ~TranslationModel();
 
-	/* This method performs translation on a list of UTF-8 encoded plain text (without any markups
-	 * or emojis) and returns a list of results in the same order. The model supports translation
-	 * from 1 source language to 1 target language.
-	 *
-	 * Each text entry can either be a word, a phrase, a sentence or a list of sentences. Additional
-	 * information related to the translated text can be requested via TranslationRequest which is
-	 * applied equally to each text entry. The translated text corresponding to each text entry and
-	 * the additional information (as specified in the TranslationRequest) is encapsulated and
-	 * returned in TranslationResult.
-	 *
-	 * The API splits each text entry into sentences internally, which are then translated
-	 * independent of each other. The translated sentences are then joined back together and returned
-	 * in TranslationResult.
-	 *
-	 * Please refer to the TranslationRequest class to find out what additional information can be
-	 * requested. The alignment information can only be requested if the model supports it (check
-	 * isAlignmentSupported() API).
-	 *
-	 * The texts argument will become empty after the execution of this API (each entry of texts list
-	 * will be moved to its corresponding TranslationResult object).
-	 */
-	std::future<std::vector<TranslationResult>> translate(
-			std::vector<std::string> &&texts, TranslationRequest request) override;
+  /* This method performs translation on a list of UTF-8 encoded plain text
+   * (without any markups or emojis) and returns a list of results in the same
+   * order. The model supports translation from 1 source language to 1 target
+   * language.
+   *
+   * Each text entry can either be a word, a phrase, a sentence or a list of
+   * sentences. Additional information related to the translated text can be
+   * requested via TranslationRequest which is applied equally to each text
+   * entry. The translated text corresponding to each text entry and the
+   * additional information (as specified in the TranslationRequest) is
+   * encapsulated and returned in TranslationResult.
+   *
+   * The API splits each text entry into sentences internally, which are then
+   * translated independent of each other. The translated sentences are then
+   * joined back together and returned in TranslationResult.
+   *
+   * Please refer to the TranslationRequest class to find out what additional
+   * information can be requested. The alignment information can only be
+   * requested if the model supports it (check isAlignmentSupported() API).
+   *
+   * The texts argument will become empty after the execution of this API (each
+   * entry of texts list will be moved to its corresponding TranslationResult
+   * object).
+   */
+  std::future<std::vector<TranslationResult>>
+  translate(std::vector<std::string> &&texts,
+            TranslationRequest request) override;
 
-	/* Check if the model can provide alignment information b/w original and translated text. */
-	bool isAlignmentSupported() const override;
+  /* Check if the model can provide alignment information b/w original and
+   * translated text. */
+  bool isAlignmentSupported() const override;
 
 private:
-	// Model configuration options
-	std::shared_ptr<marian::Options> configOptions;
+  // Model configuration options
+  std::shared_ptr<marian::Options> configOptions;
 };
 
 #endif /* SRC_TRANSLATOR_TRANSLATIONMODEL_H_ */
diff --git a/src/translator/TranslationModelConfigToOptionsAdaptor.cpp b/src/translator/TranslationModelConfigToOptionsAdaptor.cpp
index 3405a5fcf..00e37e0eb 100644
--- a/src/translator/TranslationModelConfigToOptionsAdaptor.cpp
+++ b/src/translator/TranslationModelConfigToOptionsAdaptor.cpp
@@ -6,12 +6,14 @@
 
 #include "TranslationModelConfigToOptionsAdaptor.h"
 
-TranslationModelConfigToOptionsAdaptor::TranslationModelConfigToOptionsAdaptor() {}
+TranslationModelConfigToOptionsAdaptor::
+    TranslationModelConfigToOptionsAdaptor() {}
 
-TranslationModelConfigToOptionsAdaptor::~TranslationModelConfigToOptionsAdaptor() {}
+TranslationModelConfigToOptionsAdaptor::
+    ~TranslationModelConfigToOptionsAdaptor() {}
 
-std::shared_ptr<marian::Options>
-TranslationModelConfigToOptionsAdaptor::adapt(const TranslationModelConfiguration& config) {
-	// ToDo: Add actual implementation
-	return std::make_shared<marian::Options>();
+std::shared_ptr<marian::Options> TranslationModelConfigToOptionsAdaptor::adapt(
+    const TranslationModelConfiguration &config) {
+  // ToDo: Add actual implementation
+  return std::make_shared<marian::Options>();
 }
diff --git a/src/translator/TranslationModelConfigToOptionsAdaptor.h b/src/translator/TranslationModelConfigToOptionsAdaptor.h
index 1eba4cced..49197b898 100644
--- a/src/translator/TranslationModelConfigToOptionsAdaptor.h
+++ b/src/translator/TranslationModelConfigToOptionsAdaptor.h
@@ -1,8 +1,9 @@
 /*
- * This class adapts the TranslationModelConfiguration object to marian::Options object.
- * marian::Options is a class that is specific to Marian and is used heavily inside it
- * as configuration options (even for translation workflow). It makes sense to work with
- * this class internally in the implementation files.
+ * This class adapts the TranslationModelConfiguration object to marian::Options
+ * object. marian::Options is a class that is specific to Marian and is used
+ * heavily inside it as configuration options (even for translation workflow).
+ * It makes sense to work with this class internally in the implementation
+ * files.
  */
 
 #ifndef SRC_TRANSLATOR_TRANSLATIONMODELCONFIGTOOPTIONSADAPTOR_H_
@@ -16,17 +17,16 @@
 // All local includes
 #include "TranslationModelConfiguration.h"
 
-
 class TranslationModelConfigToOptionsAdaptor {
 public:
+  TranslationModelConfigToOptionsAdaptor();
 
-	TranslationModelConfigToOptionsAdaptor();
-
-	~TranslationModelConfigToOptionsAdaptor();
+  ~TranslationModelConfigToOptionsAdaptor();
 
-	/* Create an Options object from the translation model configuration object.
-	 */
-	std::shared_ptr<marian::Options> adapt(const TranslationModelConfiguration& config);
+  /* Create an Options object from the translation model configuration object.
+   */
+  std::shared_ptr<marian::Options>
+  adapt(const TranslationModelConfiguration &config);
 };
 
 #endif /* SRC_TRANSLATOR_TRANSLATIONMODELCONFIGTOOPTIONSADAPTOR_H_ */

From 08a7358c3d6caf55a6eb38f24b9955f474cd9729 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 25 Jan 2021 18:15:22 +0000
Subject: [PATCH 040/442] Integrating marian-translator through API

Using std::string for config. Now capable of launching marian translator
through API interface. There's a sketchy workaround to convert a string
config to marian::Options, with an added note.
---
 app/main-mts.cpp                            | 26 +------
 app/main.cpp                                | 52 ++++++++++---
 src/AbstractTranslationModel.h              |  2 +-
 src/TranslationResult.h                     | 12 ++-
 src/translator/AbstractTranslationModel.cpp |  9 +--
 src/translator/TranslationModel.cpp         | 82 +++++++++++++++++++--
 src/translator/TranslationModel.h           |  6 +-
 src/translator/parser.h                     | 32 ++++++++
 src/translator/translation_result.h         | 11 ++-
 9 files changed, 173 insertions(+), 59 deletions(-)
 create mode 100644 src/translator/parser.h

diff --git a/app/main-mts.cpp b/app/main-mts.cpp
index 3de57b074..9a1e71c63 100644
--- a/app/main-mts.cpp
+++ b/app/main-mts.cpp
@@ -3,35 +3,13 @@
 #include <sstream>
 
 #include "common/definitions.h"
-#include "common/timer.h"
 #include "common/utils.h"
 #include "marian.h"
-#include "translator/history.h"
-#include "translator/output_collector.h"
-#include "translator/output_printer.h"
-
+#include "translator/parser.h"
 #include "translator/service.h"
 
 int main(int argc, char *argv[]) {
-  marian::ConfigParser cp(marian::cli::mode::translation);
-
-  cp.addOption<std::string>(
-      "--ssplit-prefix-file", "Bergamot Options",
-      "File with nonbreaking prefixes for sentence splitting.");
-
-  cp.addOption<std::string>("--ssplit-mode", "Server Options",
-                            "[paragraph, sentence, wrapped_text]");
-
-  cp.addOption<int>(
-      "--max-input-sentence-tokens", "Bergamot Options",
-      "Maximum input tokens to be processed in a single sentence.", 128);
-
-  cp.addOption<int>("--max-input-tokens", "Bergamot Options",
-                    "Maximum input tokens in a batch. control for"
-                    "Bergamot Queue",
-                    1024);
-
-  // Launch service.
+  auto cp = marian::bergamot::createConfigParser();
   auto options = cp.parseOptions(argc, argv, true);
   marian::bergamot::Service service(options);
 
diff --git a/app/main.cpp b/app/main.cpp
index bb0fa34e2..ec6ef6da0 100644
--- a/app/main.cpp
+++ b/app/main.cpp
@@ -11,25 +11,57 @@
 #include "TranslationModelConfiguration.h"
 #include "TranslationRequest.h"
 #include "TranslationResult.h"
+#include "translator/parser.h"
 
 int main(int argc, char **argv) {
 
-  // Create an instance of AbstractTranslationModel with a dummy model
-  // configuration
-  TranslationModelConfiguration config(
-      "dummy_modelFilePath", "dummy_sourceVocabPath", "dummy_targetVocabPath");
+  // Create a configParser and load command line parameters into a YAML config
+  // string.
+  auto configParser = marian::bergamot::createConfigParser();
+  auto options = configParser.parseOptions(argc, argv, true);
+  std::string config = options->asYamlString();
+  std::cout << config << std::endl;
+
+  // Route the config string to construct marian model through
+  // AbstractTranslationModel
   std::shared_ptr<AbstractTranslationModel> model =
       AbstractTranslationModel::createInstance(config);
 
-  // Call to translate a dummy (empty) texts with a dummy (empty) translation
-  // request
-  TranslationRequest req;
+  TranslationRequest translationRequest;
   std::vector<std::string> texts;
-  auto result = model->translate(std::move(texts), req);
+  for (int i = 0; i < 10; i++) {
+    texts.emplace_back(
+        "The Bergamot project will add and improve client-side machine"
+        "translation in a web browser.  Unlike current cloud-based"
+        "options, running directly on users’ machines empowers citizens to"
+        "preserve their privacy and increases the uptake of language"
+        "technologies in Europe in various sectors that require"
+        "confidentiality. Free software integrated with an open-source web"
+        "browser, such as Mozilla Firefox, will enable bottom-up adoption"
+        "by non-experts, resulting in cost savings for private and public"
+        "sector users who would otherwise procure translation or operate"
+        "monolingually.  Bergamot is a consortium coordinated by the"
+        "University of Edinburgh with partners Charles University in"
+        "Prague, the University of Sheffield, University of Tartu, and"
+        "Mozilla.");
+  }
+
+  auto result = model->translate(std::move(texts), translationRequest);
 
   // Resolve the future and get the actual result
-  std::vector<TranslationResult> res = result.get();
+  std::vector<TranslationResult> results = result.get();
+
+  for (auto &result : results) {
+    auto mappings = result.getSentenceMappings();
+    for (auto &p : mappings) {
+      std::string_view src = p.first;
+      std::string_view tgt = p.second;
+
+      std::cout << "[src]: " << src << std::endl;
+      std::cout << "[tgt]: " << tgt << std::endl;
+      std::cout << std::endl;
+    }
+  }
 
-  std::cout << "Count is: " << res.size() << std::endl;
   return 0;
 }
diff --git a/src/AbstractTranslationModel.h b/src/AbstractTranslationModel.h
index b76aeebed..69b72cf39 100644
--- a/src/AbstractTranslationModel.h
+++ b/src/AbstractTranslationModel.h
@@ -30,7 +30,7 @@ class AbstractTranslationModel {
    * configuration (TranslationModelConfiguration).
    */
   static std::shared_ptr<AbstractTranslationModel>
-  createInstance(const TranslationModelConfiguration &config);
+  createInstance(const std::string &config);
 
   AbstractTranslationModel() = default;
 
diff --git a/src/TranslationResult.h b/src/TranslationResult.h
index 34858f74c..6e5d801e1 100644
--- a/src/TranslationResult.h
+++ b/src/TranslationResult.h
@@ -21,12 +21,16 @@ class TranslationResult {
   typedef std::vector<std::pair<std::string_view, std::string_view>>
       SentenceMappings;
 
-  TranslationResult(const std::string &original, const std::string &translation)
-      : originalText(original), translatedText(translation) {}
+  TranslationResult(const std::string &original, const std::string &translation,
+                    SentenceMappings &sentenceMappings)
+      : originalText(original), translatedText(translation),
+        sentenceMappings(sentenceMappings) {}
 
-  TranslationResult(std::string &&original, std::string &&translation)
+  TranslationResult(std::string &&original, std::string &&translation,
+                    SentenceMappings &&sentenceMappings)
       : originalText(std::move(original)),
-        translatedText(std::move(translation)) {}
+        translatedText(std::move(translation)),
+        sentenceMappings(std::move(sentenceMappings)) {}
 
   /* Return the original text. */
   const std::string &getOriginalText() const { return originalText; }
diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
index 94782fa81..e7a917922 100644
--- a/src/translator/AbstractTranslationModel.cpp
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -13,9 +13,8 @@
 #include "TranslationModelConfigToOptionsAdaptor.h"
 
 std::shared_ptr<AbstractTranslationModel>
-AbstractTranslationModel::createInstance(
-    const TranslationModelConfiguration &config) {
-  TranslationModelConfigToOptionsAdaptor adaptor;
-  auto options = adaptor.adapt(config);
-  return std::make_shared<TranslationModel>(options);
+AbstractTranslationModel::createInstance(const std::string &config) {
+  // TranslationModelConfigToOptionsAdaptor adaptor;
+  // auto options = adaptor.adapt(config);
+  return std::make_shared<TranslationModel>(config);
 }
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index b3a8fec32..ce1310614 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -6,21 +6,89 @@
 #include <future>
 #include <vector>
 
+#include "3rd_party/marian-dev/src/3rd_party/yaml-cpp/yaml.h"
+#include "3rd_party/marian-dev/src/common/config_parser.h"
 #include "TranslationModel.h"
+#include "common/config_validator.h"
+#include "common/options.h"
+#include "translator/service.h"
 
-TranslationModel::TranslationModel(std::shared_ptr<marian::Options> options)
-    : configOptions(std::move(options)), AbstractTranslationModel() {}
+std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
+  marian::Options options;
+
+  // @TODO(jerinphilip) There's something off here, @XapaJIaMnu suggests
+  // that should not be using the defaultConfig. This function only has access
+  // to std::string config and needs to be able to construct Options from the
+  // same.
+
+  // Absent the following code-segment, there is a parsing exception thrown on
+  // rebuilding YAML.
+  //
+  // Error: Unhandled exception of type 'N4YAML11InvalidNodeE': invalid node;
+  // this may result from using a map iterator as a sequence iterator, or
+  // vice-versa
+  //
+  // Error: Aborted from void unhandledException() in
+  // 3rd_party/marian-dev/src/common/logging.cpp:113
+
+  marian::ConfigParser configParser(marian::cli::mode::translation);
+  const YAML::Node &defaultConfig = configParser.getConfig();
+
+  options.merge(defaultConfig);
+
+  // Parse configs onto defaultConfig.
+  options.parse(config);
+  YAML::Node configCopy = options.cloneToYamlNode();
+
+  marian::ConfigValidator validator(configCopy);
+  validator.validateOptions(marian::cli::mode::translation);
+
+  return std::make_shared<marian::Options>(options);
+}
+
+TranslationModel::TranslationModel(const std::string &config)
+    : configOptions_(std::move(parseOptions(config))),
+      AbstractTranslationModel(), service_(configOptions_) {}
 
 TranslationModel::~TranslationModel() {}
 
 std::future<std::vector<TranslationResult>>
 TranslationModel::translate(std::vector<std::string> &&texts,
                             TranslationRequest request) {
-  // ToDo: Replace this code with the actual implementation
-  return std::async([]() {
-    std::vector<TranslationResult> results;
-    return results;
-  });
+  // Implementing a non-async version first. Unpleasant, but should work.
+  std::promise<std::vector<TranslationResult>> promise;
+  auto future = promise.get_future();
+
+  auto convert = [](marian::bergamot::TranslationResult &mTranslationResult) {
+    // Change marian::string_view to std::string_view
+    TranslationResult::SentenceMappings sentenceMappings;
+    for (auto &p : mTranslationResult.getSentenceMappings()) {
+      std::string_view src(p.first.data(), p.first.size()),
+          tgt(p.second.data(), p.second.size());
+      sentenceMappings.emplace_back(src, tgt);
+    }
+
+    TranslationResult translationResult(
+        std::move(mTranslationResult.source_),
+        std::move(mTranslationResult.translation_),
+        std::move(sentenceMappings));
+
+    return translationResult;
+  };
+
+  // This code, move into async?
+  std::vector<TranslationResult> translationResults;
+  for (auto &text : texts) {
+    // Copying text, can also be replaced with move based function.
+    // translate(...)
+    auto intermediate = service_.translateWithCopy(text);
+    intermediate.wait();
+    marian::bergamot::TranslationResult result = intermediate.get();
+    translationResults.push_back(convert(result));
+  }
+
+  promise.set_value(translationResults);
+  return future;
 }
 
 bool TranslationModel::isAlignmentSupported() const { return false; }
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index ba58969d1..686ca0554 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -17,6 +17,7 @@
 // All local project includes
 #include "AbstractTranslationModel.h"
 #include "TranslationModelConfiguration.h"
+#include "translator/service.h"
 
 /* A Translation model that translates a plain (without any markups and emojis)
  * UTF-8 encoded text. This implementation supports translation from 1 source
@@ -26,7 +27,7 @@ class TranslationModel : public AbstractTranslationModel {
 public:
   /* Construct the model using the model configuration options.
    */
-  TranslationModel(std::shared_ptr<marian::Options> options);
+  TranslationModel(const std::string &config);
 
   ~TranslationModel();
 
@@ -64,7 +65,8 @@ class TranslationModel : public AbstractTranslationModel {
 
 private:
   // Model configuration options
-  std::shared_ptr<marian::Options> configOptions;
+  std::shared_ptr<marian::Options> configOptions_; // ORDER DEPENDECNY
+  marian::bergamot::Service service_;              // ORDER DEPENDENCY
 };
 
 #endif /* SRC_TRANSLATOR_TRANSLATIONMODEL_H_ */
diff --git a/src/translator/parser.h b/src/translator/parser.h
new file mode 100644
index 000000000..e273d6aea
--- /dev/null
+++ b/src/translator/parser.h
@@ -0,0 +1,32 @@
+#ifndef SRC_BERGAMOT_PARSER_H
+#define SRC_BERGAMOT_PARSER_H
+
+#include "marian.h"
+
+namespace marian {
+namespace bergamot {
+marian::ConfigParser createConfigParser() {
+  marian::ConfigParser cp(marian::cli::mode::translation);
+  cp.addOption<std::string>(
+      "--ssplit-prefix-file", "Bergamot Options",
+      "File with nonbreaking prefixes for sentence splitting.");
+
+  cp.addOption<std::string>("--ssplit-mode", "Server Options",
+                            "[paragraph, sentence, wrapped_text]", "paragraph");
+
+  cp.addOption<int>(
+      "--max-input-sentence-tokens", "Bergamot Options",
+      "Maximum input tokens to be processed in a single sentence.", 128);
+
+  cp.addOption<int>("--max-input-tokens", "Bergamot Options",
+                    "Maximum input tokens in a batch. control for"
+                    "Bergamot Queue",
+                    1024);
+
+  return cp;
+}
+
+} //  namespace bergamot
+} //  namespace marian
+
+#endif //  SRC_BERGAMOT_PARSER_H
diff --git a/src/translator/translation_result.h b/src/translator/translation_result.h
index fb5a42a09..3987ee27b 100644
--- a/src/translator/translation_result.h
+++ b/src/translator/translation_result.h
@@ -40,10 +40,14 @@ class TranslationResult {
   // For development use to benchmark with marian-decoder.
   const Histories &getHistories() const { return histories_; }
 
-private:
   std::string source_;
   std::string translation_;
+  // Adding the following to complete bergamot-translator spec, redundant while
+  // sourceMappings_ and targetMappings_ exists or vice-versa.
 
+  SentenceMappings sentenceMappings_;
+
+private:
   // Histories are currently required for interoperability with OutputPrinter
   // and OutputCollector and hence comparisons with marian-decoder.
   // Future hook to gain alignments.
@@ -59,11 +63,6 @@ class TranslationResult {
   // string_views at the sentence-level.
   std::vector<string_view> sourceMappings_;
   std::vector<string_view> targetMappings_;
-
-  // Adding the following to complete bergamot-translator spec, redundant while
-  // sourceMappings_ and targetMappings_ exists or vice-versa.
-
-  SentenceMappings sentenceMappings_;
 };
 } // namespace bergamot
 } // namespace marian

From 026f1af887bb4e6dc205207b6433598f0ce89114 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 25 Jan 2021 11:52:23 +0100
Subject: [PATCH 041/442] Removed redundant lines from CMakeFile

---
 CMakeLists.txt | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 935cd1eab..0a2005dc1 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -17,9 +17,5 @@ option(USE_STATIC_LIBS "Link statically against non-system libs" ON)
 option(USE_MKL "Compile with MKL support" ON)
 
 add_subdirectory(3rd_party)
-
-# Adds the include directories set inside 3rd_party.
-include_directories(${INCLUDE_DIRECTORIES})
-
 add_subdirectory(src)
 add_subdirectory(app)

From b49f2c1af3a9113fbdf426b4133c0587e799ffa0 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 25 Jan 2021 18:46:04 +0100
Subject: [PATCH 042/442] Cleanup TranslationModelConfiguration to std::string
 change in API

 - Provide yaml formatted string as model configuration
 - Remove redundant files
---
 app/main.cpp                                  |  1 -
 src/AbstractTranslationModel.h                |  3 +-
 src/TranslationModelConfiguration.h           | 61 -------------------
 src/translator/AbstractTranslationModel.cpp   |  6 --
 src/translator/CMakeLists.txt                 |  1 -
 src/translator/TranslationModel.cpp           |  5 +-
 src/translator/TranslationModel.h             |  3 +-
 ...TranslationModelConfigToOptionsAdaptor.cpp | 19 ------
 .../TranslationModelConfigToOptionsAdaptor.h  | 32 ----------
 9 files changed, 6 insertions(+), 125 deletions(-)
 delete mode 100644 src/TranslationModelConfiguration.h
 delete mode 100644 src/translator/TranslationModelConfigToOptionsAdaptor.cpp
 delete mode 100644 src/translator/TranslationModelConfigToOptionsAdaptor.h

diff --git a/app/main.cpp b/app/main.cpp
index ec6ef6da0..f5d65969d 100644
--- a/app/main.cpp
+++ b/app/main.cpp
@@ -8,7 +8,6 @@
 #include <iostream>
 
 #include "AbstractTranslationModel.h"
-#include "TranslationModelConfiguration.h"
 #include "TranslationRequest.h"
 #include "TranslationResult.h"
 #include "translator/parser.h"
diff --git a/src/AbstractTranslationModel.h b/src/AbstractTranslationModel.h
index 69b72cf39..6cb30c4a2 100644
--- a/src/AbstractTranslationModel.h
+++ b/src/AbstractTranslationModel.h
@@ -15,7 +15,6 @@
 #include <string>
 #include <vector>
 
-#include "TranslationModelConfiguration.h"
 #include "TranslationRequest.h"
 #include "TranslationResult.h"
 
@@ -27,7 +26,7 @@ class AbstractTranslationModel {
 public:
   /* A Factory method to create and return an instance of an implementation of
    * AbstractTranslationModel. The instance is created using translation model
-   * configuration (TranslationModelConfiguration).
+   * configuration provided as yaml-formatted string.
    */
   static std::shared_ptr<AbstractTranslationModel>
   createInstance(const std::string &config);
diff --git a/src/TranslationModelConfiguration.h b/src/TranslationModelConfiguration.h
deleted file mode 100644
index f4a5572ea..000000000
--- a/src/TranslationModelConfiguration.h
+++ /dev/null
@@ -1,61 +0,0 @@
-/*
- * TranslationModelConfiguration.h
- *
- */
-
-#ifndef SRC_TRANSLATOR_TRANSLATIONMODELCONFIGURATION_H_
-#define SRC_TRANSLATOR_TRANSLATIONMODELCONFIGURATION_H_
-
-#include <string>
-
-/* This class encapsulates the configuration that is required by a translation
- * model to perform translation.
- */
-class TranslationModelConfiguration {
-public:
-  // Constructor
-  TranslationModelConfiguration(const std::string &modelFilePath,
-                                const std::string &sourceVocabPath,
-                                const std::string &targetVocabPath)
-      : modelPath(modelFilePath), sourceLanguageVocabPath(sourceVocabPath),
-        targetLanguageVocabPath(targetVocabPath) {}
-
-  // Copy constructor
-  TranslationModelConfiguration(const TranslationModelConfiguration &rhs)
-      : modelPath(rhs.modelPath),
-        sourceLanguageVocabPath(rhs.sourceLanguageVocabPath),
-        targetLanguageVocabPath(rhs.targetLanguageVocabPath) {}
-
-  // Move constructor
-  TranslationModelConfiguration(TranslationModelConfiguration &&rhs)
-      : modelPath(std::move(rhs.modelPath)),
-        sourceLanguageVocabPath(std::move(rhs.sourceLanguageVocabPath)),
-        targetLanguageVocabPath(std::move(rhs.targetLanguageVocabPath)) {}
-
-  // Return the path of the model file
-  const std::string &getModelFilePath() const { return modelPath; }
-
-  // Return the path of the source language vocabulary file
-  const std::string &getSourceVocabularyPath() const {
-    return sourceLanguageVocabPath;
-  }
-
-  // Return the path of the target language vocabulary file
-  const std::string &getTargetVocabularyPath() const {
-    return targetLanguageVocabPath;
-  }
-
-private:
-  // Path to the translation model file
-  const std::string modelPath;
-
-  // Path to the source vocabulary file to be used by the model
-  const std::string sourceLanguageVocabPath;
-
-  // Path to the target vocabulary file to be used by the model
-  const std::string targetLanguageVocabPath;
-
-  // ToDo: Add other user configurable options (e.g. min batch size)
-};
-
-#endif /* SRC_TRANSLATOR_TRANSLATIONMODELCONFIGURATION_H_ */
diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
index e7a917922..1b2f2b104 100644
--- a/src/translator/AbstractTranslationModel.cpp
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -4,17 +4,11 @@
  */
 #include <memory>
 
-// All 3rd party includes
-#include "3rd_party/marian-dev/src/common/options.h"
-
 // All local includes
 #include "AbstractTranslationModel.h"
 #include "TranslationModel.h"
-#include "TranslationModelConfigToOptionsAdaptor.h"
 
 std::shared_ptr<AbstractTranslationModel>
 AbstractTranslationModel::createInstance(const std::string &config) {
-  // TranslationModelConfigToOptionsAdaptor adaptor;
-  // auto options = adaptor.adapt(config);
   return std::make_shared<TranslationModel>(config);
 }
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 27158a786..b6fcf69fc 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,7 +1,6 @@
 add_library(bergamot-translator STATIC
     AbstractTranslationModel.cpp
     TranslationModel.cpp
-    TranslationModelConfigToOptionsAdaptor.cpp
 
     # Following files added from browsermt/mts@nuke
     textops.cpp
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index ce1310614..9bfaf1bec 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -6,11 +6,14 @@
 #include <future>
 #include <vector>
 
+// All 3rd party includes
 #include "3rd_party/marian-dev/src/3rd_party/yaml-cpp/yaml.h"
 #include "3rd_party/marian-dev/src/common/config_parser.h"
-#include "TranslationModel.h"
 #include "common/config_validator.h"
 #include "common/options.h"
+
+// All local project includes
+#include "TranslationModel.h"
 #include "translator/service.h"
 
 std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index 686ca0554..c922538a3 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -16,7 +16,6 @@
 
 // All local project includes
 #include "AbstractTranslationModel.h"
-#include "TranslationModelConfiguration.h"
 #include "translator/service.h"
 
 /* A Translation model that translates a plain (without any markups and emojis)
@@ -25,7 +24,7 @@
  */
 class TranslationModel : public AbstractTranslationModel {
 public:
-  /* Construct the model using the model configuration options.
+  /* Construct the model using the model configuration options as yaml-formatted string
    */
   TranslationModel(const std::string &config);
 
diff --git a/src/translator/TranslationModelConfigToOptionsAdaptor.cpp b/src/translator/TranslationModelConfigToOptionsAdaptor.cpp
deleted file mode 100644
index 00e37e0eb..000000000
--- a/src/translator/TranslationModelConfigToOptionsAdaptor.cpp
+++ /dev/null
@@ -1,19 +0,0 @@
-/*
- * TranslationModelConfigToOptionsAdaptor.cpp
- *
- */
-#include <memory>
-
-#include "TranslationModelConfigToOptionsAdaptor.h"
-
-TranslationModelConfigToOptionsAdaptor::
-    TranslationModelConfigToOptionsAdaptor() {}
-
-TranslationModelConfigToOptionsAdaptor::
-    ~TranslationModelConfigToOptionsAdaptor() {}
-
-std::shared_ptr<marian::Options> TranslationModelConfigToOptionsAdaptor::adapt(
-    const TranslationModelConfiguration &config) {
-  // ToDo: Add actual implementation
-  return std::make_shared<marian::Options>();
-}
diff --git a/src/translator/TranslationModelConfigToOptionsAdaptor.h b/src/translator/TranslationModelConfigToOptionsAdaptor.h
deleted file mode 100644
index 49197b898..000000000
--- a/src/translator/TranslationModelConfigToOptionsAdaptor.h
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * This class adapts the TranslationModelConfiguration object to marian::Options
- * object. marian::Options is a class that is specific to Marian and is used
- * heavily inside it as configuration options (even for translation workflow).
- * It makes sense to work with this class internally in the implementation
- * files.
- */
-
-#ifndef SRC_TRANSLATOR_TRANSLATIONMODELCONFIGTOOPTIONSADAPTOR_H_
-#define SRC_TRANSLATOR_TRANSLATIONMODELCONFIGTOOPTIONSADAPTOR_H_
-
-#include <memory>
-
-// All 3rd party includes
-#include "3rd_party/marian-dev/src/common/options.h"
-
-// All local includes
-#include "TranslationModelConfiguration.h"
-
-class TranslationModelConfigToOptionsAdaptor {
-public:
-  TranslationModelConfigToOptionsAdaptor();
-
-  ~TranslationModelConfigToOptionsAdaptor();
-
-  /* Create an Options object from the translation model configuration object.
-   */
-  std::shared_ptr<marian::Options>
-  adapt(const TranslationModelConfiguration &config);
-};
-
-#endif /* SRC_TRANSLATOR_TRANSLATIONMODELCONFIGTOOPTIONSADAPTOR_H_ */

From 0d16b1957ff3bda44311cd48d267ed238cf1c594 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 26 Jan 2021 14:49:28 +0100
Subject: [PATCH 043/442] Improved main.cpp file

 - Print original and translated text
 - Just add 2 vector entries for texts
---
 app/main.cpp | 40 ++++++++++++++++++++--------------------
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/app/main.cpp b/app/main.cpp
index f5d65969d..8b7fe5390 100644
--- a/app/main.cpp
+++ b/app/main.cpp
@@ -28,38 +28,38 @@ int main(int argc, char **argv) {
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
-  for (int i = 0; i < 10; i++) {
-    texts.emplace_back(
-        "The Bergamot project will add and improve client-side machine"
-        "translation in a web browser.  Unlike current cloud-based"
-        "options, running directly on users’ machines empowers citizens to"
-        "preserve their privacy and increases the uptake of language"
-        "technologies in Europe in various sectors that require"
-        "confidentiality. Free software integrated with an open-source web"
-        "browser, such as Mozilla Firefox, will enable bottom-up adoption"
-        "by non-experts, resulting in cost savings for private and public"
-        "sector users who would otherwise procure translation or operate"
-        "monolingually.  Bergamot is a consortium coordinated by the"
-        "University of Edinburgh with partners Charles University in"
-        "Prague, the University of Sheffield, University of Tartu, and"
+  texts.emplace_back("The Bergamot project will add and improve client-side machine "
+        "translation in a web browser.  Unlike current cloud-based "
+        "options, running directly on users’ machines empowers citizens to "
+        "preserve their privacy and increases the uptake of language "
+        "technologies in Europe in various sectors that require "
+        "confidentiality.");
+  texts.emplace_back("Free software integrated with an open-source web "
+        "browser, such as Mozilla Firefox, will enable bottom-up adoption "
+        "by non-experts, resulting in cost savings for private and public "
+        "sector users who would otherwise procure translation or operate "
+        "monolingually.  Bergamot is a consortium coordinated by the "
+        "University of Edinburgh with partners Charles University in "
+        "Prague, the University of Sheffield, University of Tartu, and "
         "Mozilla.");
-  }
 
-  auto result = model->translate(std::move(texts), translationRequest);
+  auto futureResults = model->translate(std::move(texts), translationRequest);
 
   // Resolve the future and get the actual result
-  std::vector<TranslationResult> results = result.get();
+  std::vector<TranslationResult> results = futureResults.get();
 
   for (auto &result : results) {
+    std::cout << "[original]: " << result.getOriginalText() << std::endl;
+    std::cout << "[translated]: " << result.getTranslatedText() << std::endl;
     auto mappings = result.getSentenceMappings();
     for (auto &p : mappings) {
       std::string_view src = p.first;
       std::string_view tgt = p.second;
 
-      std::cout << "[src]: " << src << std::endl;
-      std::cout << "[tgt]: " << tgt << std::endl;
-      std::cout << std::endl;
+      std::cout << " [src Sentence]: " << src << std::endl;
+      std::cout << " [tgt Sentence]: " << tgt << std::endl;
     }
+    std::cout << std::endl;
   }
 
   return 0;

From 9a17f365c6c0161742af901e2ff0c93f75aa7593 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 26 Jan 2021 21:18:15 +0000
Subject: [PATCH 044/442] Fix for garbled output through cli.

Requirement for string_view is the original source string be transferred
all the way from input to service to back to TranslationResult. This
constraint was violated in several places by means of existence of a
copy-constructor. The issue is fixed by deleting copy and assignment
constructors in marian::bergamot::TranslationResult and
UnifiedAPI::TranslationResult, which demonstrated a few occurances of
the same. Replaced the same with move semantics.  In addition, future is
set and get using move semantics at the moment.  Default
move-constructor didn't seem to be working, so they're made explicit for
TranslationResults.

This commit additionally packs a few deletions and improvements made to
improve structure (textops.cpp, batcher.cpp) along the process of
inspecting and fixing the garbled outputs. They are choose to be kept,
in the interest of time, against a prettified atomic commit engineering.

Combinations of the following commits in jp/string-view-bug
[acfc92 78a588 12d91b 00a277 919e2f 9d3a46 b7e39b 18f67b bf667c]
---
 app/main-mts.cpp                      | 25 ++++++++++----
 src/TranslationResult.h               |  7 ++++
 src/translator/TranslationModel.cpp   | 34 ++++++++-----------
 src/translator/batcher.cpp            |  7 ++--
 src/translator/request.cpp            |  5 ++-
 src/translator/textops.cpp            | 37 ++++++---------------
 src/translator/translation_result.cpp | 48 +++++++++++++--------------
 src/translator/translation_result.h   | 26 +++++++++------
 8 files changed, 96 insertions(+), 93 deletions(-)

diff --git a/app/main-mts.cpp b/app/main-mts.cpp
index 9a1e71c63..44a019a0d 100644
--- a/app/main-mts.cpp
+++ b/app/main-mts.cpp
@@ -1,4 +1,5 @@
 #include <cstdlib>
+#include <future>
 #include <iostream>
 #include <sstream>
 
@@ -7,6 +8,7 @@
 #include "marian.h"
 #include "translator/parser.h"
 #include "translator/service.h"
+#include "translator/translation_result.h"
 
 int main(int argc, char *argv[]) {
   auto cp = marian::bergamot::createConfigParser();
@@ -17,17 +19,26 @@ int main(int argc, char *argv[]) {
   std::ostringstream std_input;
   std_input << std::cin.rdbuf();
   std::string input = std_input.str();
+  using marian::bergamot::TranslationResult;
 
-  LOG(info, "IO complete Translating input");
   // Wait on future until TranslationResult is complete
-  auto translation_result_future = service.translate(std::move(input));
+  std::future<TranslationResult> translation_result_future =
+      service.translate(std::move(input));
   translation_result_future.wait();
-  auto translation_result = translation_result_future.get();
+  const TranslationResult &translation_result = translation_result_future.get();
 
-  // Obtain sentencemappings and print them as Proof of Concept.
-  for (auto &p : translation_result.getSentenceMappings()) {
-    std::cout << "[src] " << p.first << "\n";
-    std::cout << "[tgt] " << p.second << "\n";
+  std::cout << "service-cli [Source text]: ";
+  std::cout << translation_result.getOriginalText() << std::endl;
+
+  std::cout << "service-cli [Translated text]: ";
+  std::cout << translation_result.getTranslatedText() << std::endl;
+
+  // Obtain sentenceMappings and print them as Proof of Concept.
+  const TranslationResult::SentenceMappings &sentenceMappings =
+      translation_result.getSentenceMappings();
+  for (auto &p : sentenceMappings) {
+    std::cout << "service-cli [src] " << p.first << "\n";
+    std::cout << "service-cli [tgt] " << p.second << "\n";
   }
 
   // Stop Service.
diff --git a/src/TranslationResult.h b/src/TranslationResult.h
index 6e5d801e1..d743ff5ff 100644
--- a/src/TranslationResult.h
+++ b/src/TranslationResult.h
@@ -26,12 +26,19 @@ class TranslationResult {
       : originalText(original), translatedText(translation),
         sentenceMappings(sentenceMappings) {}
 
+  TranslationResult(TranslationResult &&other)
+      : originalText(std::move(other.originalText)),
+        translatedText(std::move(other.translatedText)),
+        sentenceMappings(std::move(other.sentenceMappings)) {}
+
   TranslationResult(std::string &&original, std::string &&translation,
                     SentenceMappings &&sentenceMappings)
       : originalText(std::move(original)),
         translatedText(std::move(translation)),
         sentenceMappings(std::move(sentenceMappings)) {}
 
+  TranslationResult &operator=(const TranslationResult &) = delete;
+
   /* Return the original text. */
   const std::string &getOriginalText() const { return originalText; }
 
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 9bfaf1bec..f501678cf 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -62,8 +62,15 @@ TranslationModel::translate(std::vector<std::string> &&texts,
   std::promise<std::vector<TranslationResult>> promise;
   auto future = promise.get_future();
 
-  auto convert = [](marian::bergamot::TranslationResult &mTranslationResult) {
-    // Change marian::string_view to std::string_view
+  // This code, move into async?
+  std::vector<TranslationResult> translationResults;
+  for (auto &text : texts) {
+    // Collect future as marian::bergamot::TranslationResult
+    auto intermediate = service_.translate(std::move(text));
+    intermediate.wait();
+    auto mTranslationResult(std::move(intermediate.get()));
+
+    // Convert to UnifiedAPI::TranslationResult
     TranslationResult::SentenceMappings sentenceMappings;
     for (auto &p : mTranslationResult.getSentenceMappings()) {
       std::string_view src(p.first.data(), p.first.size()),
@@ -71,26 +78,13 @@ TranslationModel::translate(std::vector<std::string> &&texts,
       sentenceMappings.emplace_back(src, tgt);
     }
 
-    TranslationResult translationResult(
-        std::move(mTranslationResult.source_),
-        std::move(mTranslationResult.translation_),
-        std::move(sentenceMappings));
-
-    return translationResult;
-  };
-
-  // This code, move into async?
-  std::vector<TranslationResult> translationResults;
-  for (auto &text : texts) {
-    // Copying text, can also be replaced with move based function.
-    // translate(...)
-    auto intermediate = service_.translateWithCopy(text);
-    intermediate.wait();
-    marian::bergamot::TranslationResult result = intermediate.get();
-    translationResults.push_back(convert(result));
+    // In place construction.
+    translationResults.emplace_back(std::move(mTranslationResult.source_),
+                                    std::move(mTranslationResult.translation_),
+                                    std::move(sentenceMappings));
   }
 
-  promise.set_value(translationResults);
+  promise.set_value(std::move(translationResults));
   return future;
 }
 
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 471263df9..22ee46d2a 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -9,9 +9,10 @@ namespace bergamot {
 Batcher::Batcher(Ptr<Options> options) {
   max_input_tokens_ = options->get<int>("max-input-tokens");
   bucket_.resize(options->get<int>("max-input-sentence-tokens") + 1);
-  ABORT_IF(max_input_tokens_ >= bucket_.size(),
-           "max-input-sentence-tokens cannot be greater than max-input-tokens, "
-           "batcher fail");
+  ABORT_IF(
+      max_input_tokens_ < bucket_.size() - 1,
+      "max-input-tokens cannot be less than than max-input-sentence-tokens, "
+      "batcher fail");
 }
 
 void Batcher::addSentenceWithPriority(RequestSentence &sentence) {
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 0d02c03ac..a743389b4 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -48,11 +48,10 @@ void Request::processHistory(size_t index, Ptr<History> history) {
 void Request::completeRequest() {
   // Request no longer needs to hold the content, can transfer it to
   // TranslationResult.
-  TranslationResult translation_result(std::move(source_), std::move(segments_),
+  TranslationResult translation_result(std::move(source_),
                                        std::move(sourceAlignments_),
                                        std::move(histories_), *vocabs_);
-  LOG(info, "Last translation in. Closing request;");
-  response_.set_value(translation_result);
+  response_.set_value(std::move(translation_result));
 }
 
 bool Request::operator<(const Request &b) const {
diff --git a/src/translator/textops.cpp b/src/translator/textops.cpp
index 837ea7226..25e48f1fd 100644
--- a/src/translator/textops.cpp
+++ b/src/translator/textops.cpp
@@ -50,10 +50,10 @@ SentenceSplitter::string2splitmode(const std::string &m) {
   return splitmode::wrapped_text;
 }
 
-Segment TextProcessor::tokenize(const string_view &snt,
+Segment TextProcessor::tokenize(const string_view &segment,
                                 TokenRanges &tokenRanges) {
   return vocabs_->front()->encodePreservingSource(
-      snt, tokenRanges, /*addEOS=*/false, /*inference=*/true);
+      segment, tokenRanges, /*addEOS=*/false, /*inference=*/true);
 }
 
 TextProcessor::TextProcessor(std::vector<Ptr<Vocab const>> &vocabs,
@@ -90,33 +90,18 @@ void TextProcessor::process(const string_view &query, Segments &segments,
 void TextProcessor::truncate(Segment &segment, TokenRanges &tokenRanges,
                              Segments &segments,
                              std::vector<TokenRanges> &sourceRanges) {
-  if (segment.size() > max_input_sentence_tokens_) {
-    int offset;
-    // Loop as long as I can grab max_input_sentence_tokens_
-    for (offset = 0; offset + max_input_sentence_tokens_ < segment.size();
-         offset += max_input_sentence_tokens_) {
-      auto start = segment.begin() + offset;
-
-      segments.emplace_back(start, start + max_input_sentence_tokens_);
-      segments.back().push_back(sourceEosId());
-
-      auto astart = tokenRanges.begin() + offset;
-      sourceRanges.emplace_back(astart, astart + max_input_sentence_tokens_);
-    }
-
-    if (offset < max_input_sentence_tokens_) {
-      auto start = segment.begin() + offset;
-      segments.emplace_back(start, segment.end());
-      segments.back().push_back(sourceEosId());
+  for (int offset = 0; offset < segment.size();
+       offset += max_input_sentence_tokens_) {
+    auto start = segment.begin() + offset;
 
-      auto astart = tokenRanges.begin() + offset;
-      sourceRanges.emplace_back(astart, tokenRanges.end());
-    }
+    unsigned int left = segment.size() - offset;
+    unsigned int diff = std::min(max_input_sentence_tokens_, left);
 
-  } else {
-    segments.emplace_back(segment);
+    segments.emplace_back(start, start + diff);
     segments.back().push_back(sourceEosId());
-    sourceRanges.emplace_back(tokenRanges);
+
+    auto astart = tokenRanges.begin() + offset;
+    sourceRanges.emplace_back(astart, astart + diff);
   }
 }
 
diff --git a/src/translator/translation_result.cpp b/src/translator/translation_result.cpp
index 1c74314e3..d69259f84 100644
--- a/src/translator/translation_result.cpp
+++ b/src/translator/translation_result.cpp
@@ -7,32 +7,31 @@
 namespace marian {
 namespace bergamot {
 
-TranslationResult::TranslationResult(std::string &&source, Segments &&segments,
+TranslationResult::TranslationResult(std::string &&source,
                                      std::vector<TokenRanges> &&sourceRanges,
                                      Histories &&histories,
                                      std::vector<Ptr<Vocab const>> &vocabs)
     : source_(std::move(source)), sourceRanges_(std::move(sourceRanges)),
-      segments_(std::move(segments)), histories_(std::move(histories)),
-      vocabs_(&vocabs) {
+      histories_(std::move(histories)) {
 
-  // Process sourceMappings into sourceMappings_.
-  LOG(info, "Creating sourcemappings");
-  sourceMappings_.reserve(segments_.size());
-  for (int i = 0; i < segments_.size(); i++) {
+  std::vector<string_view> sourceMappings;
+  std::vector<string_view> targetMappings;
+
+  // Process sourceMappings into sourceMappings.
+  sourceMappings.reserve(sourceRanges_.size());
+  for (int i = 0; i < sourceRanges_.size(); i++) {
     string_view first = sourceRanges_[i].front();
     string_view last = sourceRanges_[i].back();
-    int size = last.end() - first.begin();
-    sourceMappings_.emplace_back(first.data(), size);
+    sourceMappings.emplace_back(first.data(), last.end() - first.begin());
   }
 
   // Compiles translations into a single std::string translation_
   // Current implementation uses += on std::string, multiple resizes.
-  // Stores ByterRanges as indices first, followed by conversion into
+  // Stores ByteRanges as indices first, followed by conversion into
   // string_views.
   // TODO(jerin): Add token level string_views here as well.
-  LOG(info, "Decoding");
   std::vector<std::pair<int, int>> translationRanges;
-  int offset{0}, end{0};
+  size_t offset{0};
   bool first{true};
   for (auto &history : histories_) {
     // TODO(jerin): Change hardcode of nBest = 1
@@ -40,31 +39,32 @@ TranslationResult::TranslationResult(std::string &&source, Segments &&segments,
 
     Result result = onebest[0]; // Expecting only one result;
     Words words = std::get<0>(result);
-    std::string decoded = vocabs_->back()->decode(words);
+    std::string decoded = (vocabs.back())->decode(words);
     if (first) {
       first = false;
     } else {
       translation_ += " ";
+      ++offset;
     }
 
     translation_ += decoded;
-    end = offset + (first ? 0 : 1) /*space*/ + decoded.size();
-    translationRanges.emplace_back(offset, end);
-    offset = end;
+    translationRanges.emplace_back(offset, decoded.size());
+    offset += decoded.size();
   }
 
   // Converting ByteRanges as indices into string_views.
-  LOG(info, "generating targetMappings");
-  targetMappings_.reserve(translationRanges.size());
-  for (auto &p : translationRanges) {
-    targetMappings_.emplace_back(&translation_[p.first], p.second - p.first);
+  targetMappings.reserve(translationRanges.size());
+  for (auto &range : translationRanges) {
+    const char *begin = &translation_[range.first];
+    targetMappings.emplace_back(begin, range.second);
   }
 
   // Surely, let's add sentenceMappings_
-  LOG(info, "generating SentenceMappings");
-  for (auto p = sourceMappings_.begin(), q = targetMappings_.begin();
-       p != sourceMappings_.end() && q != targetMappings_.end(); ++p, ++q) {
-    sentenceMappings_.emplace_back(*p, *q);
+  for (auto src = sourceMappings.begin(), tgt = targetMappings.begin();
+       src != sourceMappings.end() && tgt != targetMappings.end();
+       ++src, ++tgt) {
+    sentenceMappings_.emplace_back(*src, *tgt);
+    auto &t = sentenceMappings_.back();
   }
 }
 
diff --git a/src/translator/translation_result.h b/src/translator/translation_result.h
index 3987ee27b..edc9a8ddd 100644
--- a/src/translator/translation_result.h
+++ b/src/translator/translation_result.h
@@ -13,11 +13,21 @@ namespace marian {
 namespace bergamot {
 class TranslationResult {
 public:
-  TranslationResult(std::string &&source, Segments &&segments,
+  TranslationResult(std::string &&source,
                     std::vector<TokenRanges> &&sourceRanges,
                     Histories &&histories,
                     std::vector<Ptr<Vocab const>> &vocabs);
 
+  TranslationResult(TranslationResult &&other)
+      : source_(std::move(other.source_)),
+        translation_(std::move(other.translation_)),
+        sourceRanges_(std::move(other.sourceRanges_)),
+        sentenceMappings_(std::move(other.sentenceMappings_)),
+        histories_(std::move(other.histories_)){};
+
+  TranslationResult(const TranslationResult &) = delete;
+  TranslationResult &operator=(const TranslationResult &) = delete;
+
   // Returns const references to source and translated texts, for external
   // consumption.
 
@@ -28,7 +38,8 @@ class TranslationResult {
   // pair for external consumption. Each entry corresponding
   // to a (source-sentence, target-sentence).
 
-  typedef std::vector<std::pair<string_view, string_view>> SentenceMappings;
+  typedef std::vector<std::pair<const string_view, const string_view>>
+      SentenceMappings;
   const SentenceMappings &getSentenceMappings() const {
     return sentenceMappings_;
   }
@@ -40,6 +51,9 @@ class TranslationResult {
   // For development use to benchmark with marian-decoder.
   const Histories &getHistories() const { return histories_; }
 
+  // @jerinphilip: Why are these members no longer-private? For move-semantics
+  // with consistent string_views for bergamot-translator.
+
   std::string source_;
   std::string translation_;
   // Adding the following to complete bergamot-translator spec, redundant while
@@ -53,16 +67,8 @@ class TranslationResult {
   // Future hook to gain alignments.
   Histories histories_;
 
-  // Can be removed eventually.
-  Segments segments_;
-  std::vector<Ptr<Vocab const>> *vocabs_;
-
   // string_views at the token level.
   std::vector<TokenRanges> sourceRanges_;
-
-  // string_views at the sentence-level.
-  std::vector<string_view> sourceMappings_;
-  std::vector<string_view> targetMappings_;
 };
 } // namespace bergamot
 } // namespace marian

From e76a602dc7205567fdcb76820349bafa8f51bf51 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 28 Jan 2021 21:44:05 +0000
Subject: [PATCH 045/442] Removing config file printing

---
 app/main.cpp | 31 ++++++++++++++++---------------
 1 file changed, 16 insertions(+), 15 deletions(-)

diff --git a/app/main.cpp b/app/main.cpp
index 8b7fe5390..ef61eb2da 100644
--- a/app/main.cpp
+++ b/app/main.cpp
@@ -19,7 +19,6 @@ int main(int argc, char **argv) {
   auto configParser = marian::bergamot::createConfigParser();
   auto options = configParser.parseOptions(argc, argv, true);
   std::string config = options->asYamlString();
-  std::cout << config << std::endl;
 
   // Route the config string to construct marian model through
   // AbstractTranslationModel
@@ -28,20 +27,22 @@ int main(int argc, char **argv) {
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
-  texts.emplace_back("The Bergamot project will add and improve client-side machine "
-        "translation in a web browser.  Unlike current cloud-based "
-        "options, running directly on users’ machines empowers citizens to "
-        "preserve their privacy and increases the uptake of language "
-        "technologies in Europe in various sectors that require "
-        "confidentiality.");
-  texts.emplace_back("Free software integrated with an open-source web "
-        "browser, such as Mozilla Firefox, will enable bottom-up adoption "
-        "by non-experts, resulting in cost savings for private and public "
-        "sector users who would otherwise procure translation or operate "
-        "monolingually.  Bergamot is a consortium coordinated by the "
-        "University of Edinburgh with partners Charles University in "
-        "Prague, the University of Sheffield, University of Tartu, and "
-        "Mozilla.");
+  texts.emplace_back(
+      "The Bergamot project will add and improve client-side machine "
+      "translation in a web browser.  Unlike current cloud-based "
+      "options, running directly on users’ machines empowers citizens to "
+      "preserve their privacy and increases the uptake of language "
+      "technologies in Europe in various sectors that require "
+      "confidentiality.");
+  texts.emplace_back(
+      "Free software integrated with an open-source web "
+      "browser, such as Mozilla Firefox, will enable bottom-up adoption "
+      "by non-experts, resulting in cost savings for private and public "
+      "sector users who would otherwise procure translation or operate "
+      "monolingually.  Bergamot is a consortium coordinated by the "
+      "University of Edinburgh with partners Charles University in "
+      "Prague, the University of Sheffield, University of Tartu, and "
+      "Mozilla.");
 
   auto futureResults = model->translate(std::move(texts), translationRequest);
 

From 548c8880ff024104e46673107709dd3f9d2c67f9 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 2 Feb 2021 14:39:19 +0000
Subject: [PATCH 046/442] CMake updates submodules

---
 CMakeLists.txt | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 0a2005dc1..2341410d7 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -19,3 +19,7 @@ option(USE_MKL "Compile with MKL support" ON)
 add_subdirectory(3rd_party)
 add_subdirectory(src)
 add_subdirectory(app)
+
+execute_process(COMMAND git submodule update --init --recursive --no-fetch
+                WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
+

From 2929077324acbb4488eac615422394e2f42218b8 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 2 Feb 2021 14:41:26 +0000
Subject: [PATCH 047/442] Reordering git submodule update before includes

---
 CMakeLists.txt | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 2341410d7..ce48a9079 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -16,10 +16,11 @@ option(USE_SENTENCEPIECE "Download and compile SentencePiece" ON)
 option(USE_STATIC_LIBS "Link statically against non-system libs" ON)
 option(USE_MKL "Compile with MKL support" ON)
 
+execute_process(COMMAND git submodule update --init --recursive --no-fetch
+                WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
+
 add_subdirectory(3rd_party)
 add_subdirectory(src)
 add_subdirectory(app)
 
-execute_process(COMMAND git submodule update --init --recursive --no-fetch
-                WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
 

From 9a54d2116cc0b26fcc7582c0a99c7905c2d3be66 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 8 Feb 2021 13:46:59 +0100
Subject: [PATCH 048/442] Updated marian-dev submodule

 - Switch to "wasm" branch of browsermt/marian-dev
---
 3rd_party/marian-dev | 2 +-
 CMakeLists.txt       | 5 ++++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index ee56e02f0..a4e50b66b 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit ee56e02f0525a4651157a07f74b44f456db14c8c
+Subproject commit a4e50b66be38a94b90c46c4695d86de9932c34e8
diff --git a/CMakeLists.txt b/CMakeLists.txt
index ce48a9079..45551ea85 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -14,7 +14,10 @@ set(BUILD_ARCH native CACHE STRING "Compile for this CPU architecture.")
 option(COMPILE_CUDA "Compile GPU version" OFF)
 option(USE_SENTENCEPIECE "Download and compile SentencePiece" ON)
 option(USE_STATIC_LIBS "Link statically against non-system libs" ON)
-option(USE_MKL "Compile with MKL support" ON)
+option(USE_MKL "Compile with MKL support" OFF)
+option(COMPILE_DECODER_ONLY "Compile marian-decoder only" ON)
+option(COMPILE_WITH_PTHREADS "Compile with pthreads support" OFF)
+option(USE_WASM_COMPATIBLE_BLAS "Compile with a WASM compatible blas for decoder only builds" ON)
 
 execute_process(COMMAND git submodule update --init --recursive --no-fetch
                 WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})

From 47b4bae268bf98dd1fad70ce50731a5f74e09c3b Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 8 Feb 2021 14:31:12 +0100
Subject: [PATCH 049/442] Changed encodePreservingSource ->
 encodeWithByteRanges

 - This change happened because marian submodule changed
   this name

 - Native builds are working fine
   -- bergamot-translator-app output is consistent
---
 src/translator/textops.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/translator/textops.cpp b/src/translator/textops.cpp
index 25e48f1fd..ac93421ab 100644
--- a/src/translator/textops.cpp
+++ b/src/translator/textops.cpp
@@ -52,7 +52,7 @@ SentenceSplitter::string2splitmode(const std::string &m) {
 
 Segment TextProcessor::tokenize(const string_view &segment,
                                 TokenRanges &tokenRanges) {
-  return vocabs_->front()->encodePreservingSource(
+  return vocabs_->front()->encodeWithByteRanges(
       segment, tokenRanges, /*addEOS=*/false, /*inference=*/true);
 }
 

From 5683168a8d0011e7311ec62e13806b23bce52ec9 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 9 Feb 2021 15:42:02 +0100
Subject: [PATCH 050/442] Updated ssplit submodule to a different repository

 - Added abhi-agg/ssplit-cpp
 - Added its wasm branch in bergamot-translator
 - Native builds of bergamot-translator are successful
   -- Sentence splitting is NOT WORKING
   -- Only translation is working
---
 .gitmodules          | 2 +-
 3rd_party/ssplit-cpp | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/.gitmodules b/.gitmodules
index d3bbf18d6..e4feab500 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,6 +1,6 @@
 [submodule "3rd_party/ssplit-cpp"]
 	path = 3rd_party/ssplit-cpp
-	url = https://github.com/ugermann/ssplit-cpp
+	url = https://github.com/abhi-agg/ssplit-cpp
 [submodule "3rd_party/marian-dev"]
 	path = 3rd_party/marian-dev
 	url = https://github.com/browsermt/marian-dev
diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index f5d022992..4f5d1348a 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit f5d022992f4a00c860eb809389748908bb85ffcf
+Subproject commit 4f5d1348a3fba1a8cb70135f68470d613573f9f3

From 584700ce911de9da92489661c42a4ecc7c58d35e Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 10 Feb 2021 11:15:16 +0100
Subject: [PATCH 051/442] Changed translate() API from non-blocking to blocking

 - Can be changed back to non-blocking once blocking API
   becomes integrable via WASM port in browser
---
 app/main.cpp                        | 4 ++--
 src/AbstractTranslationModel.h      | 2 +-
 src/translator/TranslationModel.cpp | 5 ++---
 src/translator/TranslationModel.h   | 2 +-
 4 files changed, 6 insertions(+), 7 deletions(-)

diff --git a/app/main.cpp b/app/main.cpp
index ef61eb2da..2f67feb9c 100644
--- a/app/main.cpp
+++ b/app/main.cpp
@@ -44,10 +44,10 @@ int main(int argc, char **argv) {
       "Prague, the University of Sheffield, University of Tartu, and "
       "Mozilla.");
 
-  auto futureResults = model->translate(std::move(texts), translationRequest);
+  auto results = model->translate(std::move(texts), translationRequest);
 
   // Resolve the future and get the actual result
-  std::vector<TranslationResult> results = futureResults.get();
+  //std::vector<TranslationResult> results = futureResults.get();
 
   for (auto &result : results) {
     std::cout << "[original]: " << result.getOriginalText() << std::endl;
diff --git a/src/AbstractTranslationModel.h b/src/AbstractTranslationModel.h
index 6cb30c4a2..7562b0ad0 100644
--- a/src/AbstractTranslationModel.h
+++ b/src/AbstractTranslationModel.h
@@ -57,7 +57,7 @@ class AbstractTranslationModel {
    * entry of texts list will be moved to its corresponding TranslationResult
    * object).
    */
-  virtual std::future<std::vector<TranslationResult>>
+  virtual std::vector<TranslationResult>
   translate(std::vector<std::string> &&texts, TranslationRequest request) = 0;
 
   /* Check if the model can provide alignment information b/w original and
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index f501678cf..3d5ae2380 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -55,7 +55,7 @@ TranslationModel::TranslationModel(const std::string &config)
 
 TranslationModel::~TranslationModel() {}
 
-std::future<std::vector<TranslationResult>>
+std::vector<TranslationResult>
 TranslationModel::translate(std::vector<std::string> &&texts,
                             TranslationRequest request) {
   // Implementing a non-async version first. Unpleasant, but should work.
@@ -84,8 +84,7 @@ TranslationModel::translate(std::vector<std::string> &&texts,
                                     std::move(sentenceMappings));
   }
 
-  promise.set_value(std::move(translationResults));
-  return future;
+  return translationResults;
 }
 
 bool TranslationModel::isAlignmentSupported() const { return false; }
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index c922538a3..d468e2fb6 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -54,7 +54,7 @@ class TranslationModel : public AbstractTranslationModel {
    * entry of texts list will be moved to its corresponding TranslationResult
    * object).
    */
-  std::future<std::vector<TranslationResult>>
+  std::vector<TranslationResult>
   translate(std::vector<std::string> &&texts,
             TranslationRequest request) override;
 

From a2d32693448fbbc582efc0da1e05f6731e548845 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 10 Feb 2021 11:27:16 +0100
Subject: [PATCH 052/442] Updated ssplit submodule

---
 3rd_party/ssplit-cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 4f5d1348a..16864967b 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 4f5d1348a3fba1a8cb70135f68470d613573f9f3
+Subproject commit 16864967b7313e76e3b107d11ec39d8d5cedff1e

From 9747d9ba83e2eb6f7cf5edfee37a90592d2c220b Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 15:34:27 +0100
Subject: [PATCH 053/442] Add cmake option to compile project on WASM

 - Set cmake option COMPILE_WASM to ON to compile the project
   on WASM
---
 CMakeLists.txt | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 45551ea85..b662a7880 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -8,7 +8,9 @@ project(bergamot_translator CXX C)
 
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
-set(BUILD_ARCH native CACHE STRING "Compile for this CPU architecture.")
+
+# Project specific cmake options
+option(COMPILE_WASM "Compile for WASM" OFF)
 
 # Custom CMake options to compile marian (a 3rd party submodule) for this project
 option(COMPILE_CUDA "Compile GPU version" OFF)
@@ -22,8 +24,19 @@ option(USE_WASM_COMPATIBLE_BLAS "Compile with a WASM compatible blas for decoder
 execute_process(COMMAND git submodule update --init --recursive --no-fetch
                 WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
 
-add_subdirectory(3rd_party)
-add_subdirectory(src)
-add_subdirectory(app)
+if(NOT COMPILE_WASM)
+  # Set BUILD_ARCH to native only while compiling for non wasm platform
+  set(BUILD_ARCH native CACHE STRING "Compile for this CPU architecture.")
+endif()
 
+if(COMPILE_WASM)
+  add_compile_options(-pthread -O3 -g2 -fPIC -mssse3 -msimd128)
+  add_compile_options("SHELL:-s WASM=1" "SHELL:-s ASSERTIONS=1" "SHELL:-s DISABLE_EXCEPTION_CATCHING=0" "SHELL:-s LLD_REPORT_UNDEFINED" "SHELL:-s FORCE_FILESYSTEM=1" "SHELL:-s ALLOW_MEMORY_GROWTH=1")
+  add_compile_options(-Wno-error=pthreads-mem-growth)
+endif(COMPILE_WASM)
 
+add_subdirectory(3rd_party)
+add_subdirectory(src)
+if(NOT COMPILE_WASM)
+  add_subdirectory(app)
+endif()

From b73d4f4cc275277b35545af2a0d35ea7953166d4 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 15:37:38 +0100
Subject: [PATCH 054/442] Set cmake option to compile marian library only

 - Set COMPILE_LIBRARY_ONLY to ON for marian library
---
 CMakeLists.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index b662a7880..daea56074 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -20,6 +20,7 @@ option(USE_MKL "Compile with MKL support" OFF)
 option(COMPILE_DECODER_ONLY "Compile marian-decoder only" ON)
 option(COMPILE_WITH_PTHREADS "Compile with pthreads support" OFF)
 option(USE_WASM_COMPATIBLE_BLAS "Compile with a WASM compatible blas for decoder only builds" ON)
+SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
 
 execute_process(COMMAND git submodule update --init --recursive --no-fetch
                 WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})

From 838547e4d582089d6222aadf14e77732d8955d17 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 15:42:18 +0100
Subject: [PATCH 055/442] Set cmake options of marian properly for this project

---
 CMakeLists.txt | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index daea56074..09ac2fce3 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -12,14 +12,14 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
 
-# Custom CMake options to compile marian (a 3rd party submodule) for this project
-option(COMPILE_CUDA "Compile GPU version" OFF)
-option(USE_SENTENCEPIECE "Download and compile SentencePiece" ON)
-option(USE_STATIC_LIBS "Link statically against non-system libs" ON)
-option(USE_MKL "Compile with MKL support" OFF)
-option(COMPILE_DECODER_ONLY "Compile marian-decoder only" ON)
-option(COMPILE_WITH_PTHREADS "Compile with pthreads support" OFF)
-option(USE_WASM_COMPATIBLE_BLAS "Compile with a WASM compatible blas for decoder only builds" ON)
+# Set marian (3rd party submodule) cmake options to compile for this project
+SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
+SET(USE_SENTENCEPIECE ON CACHE BOOL "Download and compile SentencePiece")
+SET(USE_STATIC_LIBS ON CACHE BOOL "Link statically against non-system libs")
+SET(USE_MKL OFF CACHE BOOL "Compile with MKL support")
+SET(COMPILE_DECODER_ONLY ON CACHE BOOL "Compile marian-decoder only")
+SET(COMPILE_WITH_PTHREADS OFF CACHE BOOL "Compile with pthreads support")
+SET(USE_WASM_COMPATIBLE_BLAS ON CACHE BOOL "Compile with a WASM compatible blas for decoder only builds")
 SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
 
 execute_process(COMMAND git submodule update --init --recursive --no-fetch

From 9b896507e3860b5c3cf0e452659d336fe43958e1 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 15:53:38 +0100
Subject: [PATCH 056/442] cmake compile option changes

 - Make native builds successful with marian decoder
 - COMPILE_DECODER_ONLY flag requires importing some
   compile definitions from marian
---
 src/translator/CMakeLists.txt | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index b6fcf69fc..eab04abf3 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -11,6 +11,10 @@ add_library(bergamot-translator STATIC
     batcher.cpp
     translation_result.cpp
 )
+if (COMPILE_DECODER_ONLY)
+  # A dirty hack because of marian's bad cmake practices
+  target_compile_definitions(bergamot-translator PUBLIC DECODER_ONLY)
+endif()
 
 target_link_libraries(bergamot-translator marian ssplit)
 

From 79c445ae3a9c63fa68cd7687e5bdae7b76dc72b1 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 15:57:26 +0100
Subject: [PATCH 057/442] cmake compile option changes for wasm builds

  - Make WASM builds successful with marian decoder
  - Setting COMPILE_WASM to ON requires importing some
    compile definitions from marian
---
 src/translator/CMakeLists.txt | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index eab04abf3..b8ed19635 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -16,6 +16,11 @@ if (COMPILE_DECODER_ONLY)
   target_compile_definitions(bergamot-translator PUBLIC DECODER_ONLY)
 endif()
 
+if(COMPILE_WASM)
+  # A dirty hack because of marian's bad cmake practices
+  target_compile_definitions(bergamot-translator PUBLIC USE_SSE2 WASM)
+endif(COMPILE_WASM)
+
 target_link_libraries(bergamot-translator marian ssplit)
 
 target_include_directories(bergamot-translator

From a06530e92b6d16527487c8fa0ead4ae04f0ddbb5 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 16:14:03 +0100
Subject: [PATCH 058/442] Fixed a bug in TranslationModel class

 - Using bergamot-translator as a library fails at run time because
   necessary parser options are not set
---
 src/translator/TranslationModel.cpp | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 3d5ae2380..fd2db1d2d 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -15,6 +15,8 @@
 // All local project includes
 #include "TranslationModel.h"
 #include "translator/service.h"
+#include "translator/parser.h"
+
 
 std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
   marian::Options options;
@@ -34,7 +36,7 @@ std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
   // Error: Aborted from void unhandledException() in
   // 3rd_party/marian-dev/src/common/logging.cpp:113
 
-  marian::ConfigParser configParser(marian::cli::mode::translation);
+  marian::ConfigParser configParser = marian::bergamot::createConfigParser();
   const YAML::Node &defaultConfig = configParser.getConfig();
 
   options.merge(defaultConfig);

From 23a952782479401c4ac31bab6eccccb546c1f4ee Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 16:38:36 +0100
Subject: [PATCH 059/442] Source code changes to compile the project without
 threads

 - Set COMPILE_THREAD_VARIANT cmake option to ON to compile
   multithreaded variant of the project
---
 CMakeLists.txt                      |  4 ++++
 src/translator/CMakeLists.txt       |  4 ++++
 src/translator/batch_translator.cpp | 16 +++++++++++++++-
 src/translator/batch_translator.h   |  8 ++++++++
 src/translator/pcqueue.h            | 29 +++++++++++++++++++++++++++++
 src/translator/service.cpp          |  3 +++
 6 files changed, 63 insertions(+), 1 deletion(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 09ac2fce3..7327e1449 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -11,6 +11,7 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
+option(COMPILE_THREAD_VARIANT "Compile with thread support" OFF)
 
 # Set marian (3rd party submodule) cmake options to compile for this project
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
@@ -41,3 +42,6 @@ add_subdirectory(src)
 if(NOT COMPILE_WASM)
   add_subdirectory(app)
 endif()
+if(COMPILE_WASM)
+  add_subdirectory(app)
+endif(COMPILE_WASM)
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index b8ed19635..71bdd97f6 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -21,6 +21,10 @@ if(COMPILE_WASM)
   target_compile_definitions(bergamot-translator PUBLIC USE_SSE2 WASM)
 endif(COMPILE_WASM)
 
+if (COMPILE_THREAD_VARIANT)
+  target_compile_definitions(bergamot-translator PRIVATE WITH_PTHREADS)
+endif()
+
 target_link_libraries(bergamot-translator marian ssplit)
 
 target_include_directories(bergamot-translator
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 6380a00cc..6dc399321 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -14,7 +14,11 @@ BatchTranslator::BatchTranslator(DeviceId const device,
                                  Ptr<Options> options)
     : device_(device), options_(options), pcqueue_(&pcqueue), vocabs_(&vocabs) {
 
+#ifdef WITH_PTHREADS
   thread_ = std::thread([this] { this->mainloop(); });
+#else
+  this->initGraph();
+#endif
 }
 
 void BatchTranslator::initGraph() {
@@ -100,12 +104,16 @@ void BatchTranslator::translate(RequestSentences &requestSentences,
 }
 
 void BatchTranslator::mainloop() {
+#ifdef WITH_PTHREADS
   initGraph();
+#endif
 
   PCItem pcitem;
   Histories histories;
 
+#ifdef WITH_PTHREADS
   while (true) {
+#endif
     pcqueue_->ConsumeSwap(pcitem);
     if (pcitem.isPoison()) {
       return;
@@ -115,10 +123,16 @@ void BatchTranslator::mainloop() {
         pcitem.sentences[i].completeSentence(histories[i]);
       }
     }
+#ifdef WITH_PTHREADS
   }
+#endif
 }
 
-void BatchTranslator::join() { thread_.join(); }
+void BatchTranslator::join() {
+#ifdef WITH_PTHREADS
+  thread_.join();
+#endif
+}
 
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 069155efb..3f1d2e4bd 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -29,10 +29,16 @@ class BatchTranslator {
   // convenience function for logging. TODO(jerin)
   std::string _identifier() { return "worker" + std::to_string(device_.no); }
 
+#ifndef WITH_PTHREADS
+  void mainloop();
+#endif
+
 private:
   void initGraph();
   void translate(RequestSentences &requestSentences, Histories &histories);
+#ifdef WITH_PTHREADS
   void mainloop();
+#endif
 
   Ptr<Options> options_;
 
@@ -43,7 +49,9 @@ class BatchTranslator {
   Ptr<data::ShortlistGenerator const> slgen_;
 
   PCQueue<PCItem> *pcqueue_;
+#ifdef WITH_PTHREADS
   std::thread thread_;
+#endif
 };
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/pcqueue.h b/src/translator/pcqueue.h
index f0b354145..79d6b75e0 100644
--- a/src/translator/pcqueue.h
+++ b/src/translator/pcqueue.h
@@ -9,6 +9,7 @@
 #include <memory>
 #include <mutex>
 
+#ifdef WITH_PTHREADS
 #ifdef __APPLE__
 #include <mach/mach.h>
 #include <mach/mach_traps.h>
@@ -19,6 +20,7 @@
 #else
 #include <boost/interprocess/sync/interprocess_semaphore.hpp>
 #endif
+#endif // WITH_PTHREADS
 
 #if __GNUC__ >= 3
 #define UTIL_UNLIKELY(x) __builtin_expect(!!(x), 0)
@@ -29,6 +31,7 @@
 namespace marian {
 namespace bergamot {
 
+#ifdef WITH_PTHREADS
 /* OS X Maverick and Boost interprocess were doing "Function not implemented."
  * So this is my own wrapper around the mach kernel APIs.
  */
@@ -114,6 +117,20 @@ inline void WaitSemaphore(Semaphore &on) {
 }
 
 #endif // Apple
+#else // WITH_PTHREADS
+// A dummy Semaphore class that does nothing
+class Semaphore {
+public:
+  explicit Semaphore(unsigned int value) : count(value) {}
+  ~Semaphore() {}
+  void wait() {}
+  void post() {}
+private:
+  unsigned int count;
+};
+
+inline void WaitSemaphore(Semaphore &semaphore) { semaphore.wait(); }
+#endif // WITH_PTHREADS
 
 /**
  * Producer consumer queue safe for multiple producers and multiple consumers.
@@ -134,7 +151,9 @@ template <class T> class PCQueue {
   void Produce(const T &val) {
     WaitSemaphore(empty_);
     {
+    #ifdef WITH_PTHREADS
       std::lock_guard<std::mutex> produce_lock(produce_at_mutex_);
+    #endif
       try {
         *produce_at_ = val;
       } catch (...) {
@@ -151,7 +170,9 @@ template <class T> class PCQueue {
   void ProduceSwap(T &val) {
     WaitSemaphore(empty_);
     {
+    #ifdef WITH_PTHREADS
       std::lock_guard<std::mutex> produce_lock(produce_at_mutex_);
+    #endif
       try {
         std::swap(*produce_at_, val);
       } catch (...) {
@@ -168,7 +189,9 @@ template <class T> class PCQueue {
   T &Consume(T &out) {
     WaitSemaphore(used_);
     {
+    #ifdef WITH_PTHREADS
       std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
+    #endif
       try {
         out = *consume_at_;
       } catch (...) {
@@ -186,7 +209,9 @@ template <class T> class PCQueue {
   T &ConsumeSwap(T &out) {
     WaitSemaphore(used_);
     {
+    #ifdef WITH_PTHREADS
       std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
+    #endif
       try {
         std::swap(out, *consume_at_);
       } catch (...) {
@@ -220,11 +245,15 @@ template <class T> class PCQueue {
 
   // Index for next write in storage_.
   T *produce_at_;
+#ifdef WITH_PTHREADS
   std::mutex produce_at_mutex_;
+#endif
 
   // Index for next read from storage_.
   T *consume_at_;
+#ifdef WITH_PTHREADS
   std::mutex consume_at_mutex_;
+#endif
 };
 
 template <class T> struct UnboundedPage {
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 4a5af301c..f61ad4731 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -73,6 +73,9 @@ std::future<TranslationResult> Service::translate(std::string &&input) {
     }
   } while (numSentences > 0);
 
+#ifndef WITH_PTHREADS
+  workers_[0].mainloop();
+#endif
   return future;
 }
 

From 7b80003a5fd60d5e28beee74d8f45590390581f5 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 16:59:07 +0100
Subject: [PATCH 060/442] Added code to generate proper JS bindings of
 translator

 - COMPILE_WASM cmake option sets WASM_BINDINGS compile
   definition that enables code for generating proper JS
   bindings
---
 src/TranslationResult.h       | 22 +++++++++++++++++++++-
 src/translator/CMakeLists.txt |  2 ++
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/src/TranslationResult.h b/src/TranslationResult.h
index d743ff5ff..b4867af65 100644
--- a/src/TranslationResult.h
+++ b/src/TranslationResult.h
@@ -20,7 +20,11 @@ class TranslationResult {
 public:
   typedef std::vector<std::pair<std::string_view, std::string_view>>
       SentenceMappings;
-
+#ifdef WASM_BINDINGS
+  TranslationResult(const std::string &original, const std::string &translation)
+      : originalText(original), translatedText(translation),
+        sentenceMappings() {}
+#endif
   TranslationResult(const std::string &original, const std::string &translation,
                     SentenceMappings &sentenceMappings)
       : originalText(original), translatedText(translation),
@@ -31,13 +35,29 @@ class TranslationResult {
         translatedText(std::move(other.translatedText)),
         sentenceMappings(std::move(other.sentenceMappings)) {}
 
+#ifdef WASM_BINDINGS
+  TranslationResult(const TranslationResult &other)
+      : originalText(other.originalText),
+        translatedText(other.translatedText),
+        sentenceMappings(other.sentenceMappings) {}
+#endif
+
   TranslationResult(std::string &&original, std::string &&translation,
                     SentenceMappings &&sentenceMappings)
       : originalText(std::move(original)),
         translatedText(std::move(translation)),
         sentenceMappings(std::move(sentenceMappings)) {}
 
+#ifndef WASM_BINDINGS
   TranslationResult &operator=(const TranslationResult &) = delete;
+#else
+  TranslationResult &operator=(const TranslationResult &result) {
+    originalText = result.originalText;
+    translatedText = result.translatedText;
+    sentenceMappings = result.sentenceMappings;
+    return *this;
+  }
+#endif
 
   /* Return the original text. */
   const std::string &getOriginalText() const { return originalText; }
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 71bdd97f6..ba2c2e033 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -19,6 +19,8 @@ endif()
 if(COMPILE_WASM)
   # A dirty hack because of marian's bad cmake practices
   target_compile_definitions(bergamot-translator PUBLIC USE_SSE2 WASM)
+  # Enable code that is required for generating JS bindings
+  target_compile_definitions(bergamot-translator PRIVATE WASM_BINDINGS)
 endif(COMPILE_WASM)
 
 if (COMPILE_THREAD_VARIANT)

From 74b06d863ebbd0b0b59dfd7be1e541a338c8a3f8 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 19:09:30 +0100
Subject: [PATCH 061/442] Add wasm folder to compile JS bindings

---
 CMakeLists.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 7327e1449..4b6e2241b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -43,5 +43,5 @@ if(NOT COMPILE_WASM)
   add_subdirectory(app)
 endif()
 if(COMPILE_WASM)
-  add_subdirectory(app)
+  add_subdirectory(wasm)
 endif(COMPILE_WASM)

From de501e8f963b8fed6cc6f1799d55f2e20b325d3e Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 20:48:29 +0100
Subject: [PATCH 062/442] Added JS binding files and cmake infrastructure to
 build them

 - Added "wasm" folder
 - Contains README file as well
---
 CMakeLists.txt                               |  1 +
 wasm/CMakeLists.txt                          | 27 ++++++++++
 wasm/README.md                               | 52 +++++++++++++++++++
 wasm/bergamot.html                           | 54 ++++++++++++++++++++
 wasm/bindings/TranslationModelBindings.cpp   | 23 +++++++++
 wasm/bindings/TranslationRequestBindings.cpp | 17 ++++++
 wasm/bindings/TranslationResultBindings.cpp  | 20 ++++++++
 7 files changed, 194 insertions(+)
 create mode 100644 wasm/CMakeLists.txt
 create mode 100644 wasm/README.md
 create mode 100644 wasm/bergamot.html
 create mode 100644 wasm/bindings/TranslationModelBindings.cpp
 create mode 100644 wasm/bindings/TranslationRequestBindings.cpp
 create mode 100644 wasm/bindings/TranslationResultBindings.cpp

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 4b6e2241b..505d78549 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -12,6 +12,7 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
 option(COMPILE_THREAD_VARIANT "Compile with thread support" OFF)
+option(PACKAGE_DIR "Directory including all the files to be packaged (pre-loaded) in wasm builds" "")
 
 # Set marian (3rd party submodule) cmake options to compile for this project
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
new file mode 100644
index 000000000..9ede6a612
--- /dev/null
+++ b/wasm/CMakeLists.txt
@@ -0,0 +1,27 @@
+add_executable(bergamot-translator-worker
+    bindings/TranslationModelBindings.cpp
+    bindings/TranslationRequestBindings.cpp
+    bindings/TranslationResultBindings.cpp
+)
+
+# This header inclusion needs to go away later as path to public headers of bergamot
+# translator should be directly available from "bergamot-translator" target
+target_include_directories(bergamot-translator-worker
+    PRIVATE ${CMAKE_SOURCE_DIR}/src/translator
+    PRIVATE ${CMAKE_SOURCE_DIR}
+)
+# This compile definition is required for generating binding code properly
+target_compile_definitions(bergamot-translator-worker PRIVATE WASM_BINDINGS)
+
+set(LINKER_FLAGS "--bind -s ASSERTIONS=1 -s DISABLE_EXCEPTION_CATCHING=0 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1")
+if (NOT PACKAGE_DIR STREQUAL "")
+  set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${PACKAGE_DIR}@/")
+endif()
+
+set_target_properties(bergamot-translator-worker PROPERTIES
+                        SUFFIX ".js"
+                        LINK_FLAGS ${LINKER_FLAGS}
+                        )
+#target_link_options(bergamot-translator-worker --preload-file ${PACKAGE_DIR}@/)
+
+target_link_libraries(bergamot-translator-worker bergamot-translator)
diff --git a/wasm/README.md b/wasm/README.md
new file mode 100644
index 000000000..83d4738cd
--- /dev/null
+++ b/wasm/README.md
@@ -0,0 +1,52 @@
+## Using Bergamot Translator in JavaScript
+The example file `bergamot.html` in this folder demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
+This example assumes that files were packaged in wasm binary.
+
+A brief summary is here though:
+
+```js
+// The model configuration as YAML formatted string. For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
+// This example captures the most relevant options: model file, vocabulary files and shortlist file
+const modelConfig = "{\"models\":[\"/model.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"shortlist\":[\"/lex.s2t\"],\"beam-size\":1}";
+
+// Instantiate the TranslationModel
+const model = new Module.TranslationModel(modelConfig);
+
+// Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
+const request = new Module.TranslationRequest();
+const input = new Module.VectorString;
+
+// Initialize the input
+input.push_back("Hola"); input.push_back("Mundo");
+
+// translate the input; the result is a vector<TranslationResult>
+const result = model.translate(input, request);
+
+// Print original and translated text from each entry of vector<TranslationResult>
+for (let i = 0; i < result.size(); i++) {
+    console.log(' original=' + result.get(i).getOriginalText() + ', translation=' + result.get(i).getTranslatedText());
+}
+
+// Don't forget to clean up the instances
+model.delete();
+request.delete();
+input.delete();
+```
+
+You can also see everything in action by running `bergamot.html` file in the browser following these steps:
+* Copy bergamot.html to the directory where build artefacts of wasm compilation (i.e. .js and .wasm files) reside.
+* Start an http server locally
+* Open the link provided by the http server in any browser
+* Open the browser's console and you will see all the console messages there
+
+Assuming build artefacts are present in `$ROOT/build-wasm` folder where `ROOT` is repository's root.
+Above instructions would become:
+
+```bash
+cd $ROOT/build-wasm
+cp ../wasm/bergamot.html wasm/.
+python3 -m http.server -d wasm
+```
+
+Assuming it starts the http server on 8000 port,
+open `http://0.0.0.0:8000/bergamot.html` in any browser and see the console logs in browser's console.
diff --git a/wasm/bergamot.html b/wasm/bergamot.html
new file mode 100644
index 000000000..2bbad046b
--- /dev/null
+++ b/wasm/bergamot.html
@@ -0,0 +1,54 @@
+<!doctype html>
+<html>
+  <script>
+    var Module = {
+      onRuntimeInitialized: function() {
+        // Set the Model Configuration as YAML formatted string.
+        // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
+        // This example captures the most relevant options: model file, vocabulary files and shortlist file
+        var modelConfig = "{\"models\":[\"/model.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1}";//,\"shortlist\":[\"/lex.s2t\"]
+
+        // Instantiate the TranslationModel
+        var model = new Module.TranslationModel(modelConfig);
+        console.log('Alignment:', model.isAlignmentSupported());
+
+        // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
+        var request = new Module.TranslationRequest();
+        let input = new Module.VectorString;
+
+        // Initialize the input
+        input.push_back("¿Qué estás haciendo? ¿Por qué estás haciendo eso?");
+        input.push_back("Han pasado tres días de las elecciones presidenciales y los ecuatorianos siguen sin saber quién pasará a la segunda vuelta. El anuncio de Iván Duque del plan para regularizar al millón de migrantes venezolanos que están ilegales en Colombia ha generado esperanza a la vez que confusión.");
+
+        // Access input (just for debugging)
+        console.log('Input size=', input.size());
+        for (let i = 0; i < input.size(); i++) {
+          console.log(' val:' + input.get(i));
+        }
+
+        // Translate the input; the result is a vector<TranslationResult>
+        var result = model.translate(input, request);
+
+        // Access original and translated text from each entry of vector<TranslationResult>
+        console.log('Result size=', result.size());
+        for (let i = 0; i < result.size(); i++) {
+          console.log(' original=' + result.get(i).getOriginalText() + ', translation=' + result.get(i).getTranslatedText());
+        }
+
+        // Translate again more text
+        input.push_back("Luego de los cuestionamientos al gobierno chileno por la gestión de la pandemia, el país sudamericano lidera el ranking de vacunación en América Latina. Aquí te explicamos cómo lo ha logrado.");
+        result = model.translate(input, request);
+        console.log('Result size=', result.size());
+        for (let i = 0; i < result.size(); i++) {
+          console.log(' original=' + result.get(i).getOriginalText() + ', translation=' + result.get(i).getTranslatedText());
+        }
+
+        // Don't forget to clean up the instances
+        model.delete();
+        request.delete();
+        input.delete();
+      }
+    };
+  </script>
+  <script src="bergamot-translator-worker.js"></script>
+</html>
diff --git a/wasm/bindings/TranslationModelBindings.cpp b/wasm/bindings/TranslationModelBindings.cpp
new file mode 100644
index 000000000..245416c6a
--- /dev/null
+++ b/wasm/bindings/TranslationModelBindings.cpp
@@ -0,0 +1,23 @@
+/*
+ * TranslationModelBindings.cpp
+ *
+ * Bindings for TranslationModel class
+ */
+
+#include <emscripten/bind.h>
+
+#include "TranslationModel.h"
+
+using namespace emscripten;
+
+// Binding code
+EMSCRIPTEN_BINDINGS(translation_model) {
+  class_<TranslationModel>("TranslationModel")
+    .constructor<std::string>()
+    .function("translate", &TranslationModel::translate)
+	  .function("isAlignmentSupported", &TranslationModel::isAlignmentSupported)
+    ;
+
+  register_vector<std::string>("VectorString");
+  register_vector<TranslationResult>("VectorTranslationResult");
+}
diff --git a/wasm/bindings/TranslationRequestBindings.cpp b/wasm/bindings/TranslationRequestBindings.cpp
new file mode 100644
index 000000000..bb5ec9884
--- /dev/null
+++ b/wasm/bindings/TranslationRequestBindings.cpp
@@ -0,0 +1,17 @@
+/*
+ * Bindings for TranslationRequest class
+ *
+ */
+
+#include <emscripten/bind.h>
+
+#include "TranslationRequest.h"
+
+using namespace emscripten;
+
+// Binding code
+EMSCRIPTEN_BINDINGS(translation_request) {
+  class_<TranslationRequest>("TranslationRequest")
+    .constructor<>()
+    ;
+}
diff --git a/wasm/bindings/TranslationResultBindings.cpp b/wasm/bindings/TranslationResultBindings.cpp
new file mode 100644
index 000000000..a3713a130
--- /dev/null
+++ b/wasm/bindings/TranslationResultBindings.cpp
@@ -0,0 +1,20 @@
+/*
+ * Bindings for TranslationResult class
+ *
+ */
+
+#include <emscripten/bind.h>
+#include <vector>
+
+#include "TranslationResult.h"
+
+using namespace emscripten;
+
+// Binding code
+EMSCRIPTEN_BINDINGS(translation_result) {
+  class_<TranslationResult>("TranslationResult")
+    .constructor<std::string, std::string, TranslationResult::SentenceMappings>()
+	  .function("getOriginalText", &TranslationResult::getOriginalText)
+	  .function("getTranslatedText", &TranslationResult::getTranslatedText)
+    ;
+}

From e12647076c69c4e0355b598b16127d4112f662bd Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 23:27:16 +0100
Subject: [PATCH 063/442] Updated README with wasm build and use instructions

---
 README.md | 84 ++++++++++++++++++++++++++++++-------------------------
 1 file changed, 46 insertions(+), 38 deletions(-)

diff --git a/README.md b/README.md
index 52f60b287..e1ad9c37a 100644
--- a/README.md
+++ b/README.md
@@ -3,58 +3,66 @@
 Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.github.io/) framework based) neural machine translation functionality in accordance with the [Bergamot](https://browser.mt/) project that focuses on improving client-side machine translation in a web browser.
 
 ## Build Instructions
-```
-$ git clone https://github.com/browsermt/bergamot-translator
-$ cd bergamot-translator
-$ mkdir build
-$ cd build
-$ cmake ../
-$ make -j
 
+### Build Natively
+
+```bash
+git clone https://github.com/browsermt/bergamot-translator
+cd bergamot-translator
+mkdir build
+cd build
+cmake ../
+make -j
 ```
 
-## Usage 
+### Build WASM
 
-### Bergamot Translator
+To compile WASM, first download and Install Emscripten using following instructions:
 
-The build will generate the library that can be linked to any project. All the public header files are specified in `src` folder.
+1. Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
+2. Enter the cloned directory: `cd emsdk`
+3. Install the lastest sdk tools: `./emsdk install latest`
+4. Activate the latest sdk tools: `./emsdk activate latest`
+5. Activate path variables: `source ./emsdk_env.sh`
 
-### `service-cli`
+After the successful installation of Emscripten, perform these steps:
 
-An executable `service-cli` is generated by the build in the `app` folder and
-provides command line interface to the underlying translator. The models
-required to run the command-line are available at
-[data.statmt.org/bergamot/models/](http://data.statmt.org/bergamot/models/).
-The following example uses an English to German tiny11 student model, available
-at:
+```bash
+git clone https://github.com/browsermt/bergamot-translator
+cd bergamot-translator
+mkdir build-wasm
+cd build-wasm
+emcmake cmake -DCOMPILE_WASM=on ../
+emmake make -j
+```
 
-* [data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz](http://data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz)
+It should generate the artefacts (.js and .wasm files) in `wasm` folder inside build directory ("build-wasm" in this case).
 
+The build also allows packaging files into wasm binary (i.e. preloading in Emscripten’s virtual file system) using cmake
+option `PACKAGE_DIR`. The compile command below packages all the files in PATH directory into wasm binary.
 ```bash
-MODEL_DIR=... # path to where the model-files are.
-ARGS=(
-    -m $MODEL_DIR/model.intgemm.alphas.bin # Path to model file.
-    --vocabs 
-        $MODEL_DIR/vocab.deen.spm # source-vocabulary
-        $MODEL_DIR/vocab.deen.spm # target-vocabulary
+emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=<PATH> ../
+```
+Files packaged this way are preloaded in the root of the virtual file system.
 
-    # The following increases speed through one-best-decoding, shortlist and quantization.
-    --beam-size 1 --skip-cost --shortlist $MODEL_DIR/lex.s2t.gz 50 50 --int8shiftAlphaAll 
 
-    # Number of CPU threads (workers to launch). Parallelizes over cores and improves speed.
-    --cpu-threads 4
+After Editing Files:
 
-    # Hyperparameters of how many tokens to be accounted for in a batch and maximum tokens in a sentence.
-    --max-input-sentence-tokens 1024 --max-input-tokens 1024 
+```bash
+emmake make -j
+```
+
+After Adding/Removing Files:
+
+```bash
+emcmake cmake -DCOMPILE_WASM=on ../
+emmake make -j
+```
 
-    # Three modes are supported
-    #   - sentence: One sentence per line
-    #   - paragraph: One paragraph per line.
-    #   - wrapped text: Paragraphs are separated by empty line.
+### Using Native version
 
-    --ssplit-mode paragraph 
+The builds generate library that can be integrated to any project. All the public header files are specified in `src` folder. A short example of how to use the APIs is provided in `app/main.cpp` file
 
-)
+### Using WASM version
 
-./app/service-cli "${ARGS[@]}" < path-to-input-file
-```
+Please follow the `README` inside the `wasm` folder of this repository that demonstrates how to use the translator in JavaScript.

From ff95e37f89e2ed67e4a6420e6a3415bb8e794994 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Feb 2021 23:51:45 +0100
Subject: [PATCH 064/442] Improved cmake option PACKAGE_DIR

---
 CMakeLists.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 505d78549..10256c218 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -12,7 +12,7 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
 option(COMPILE_THREAD_VARIANT "Compile with thread support" OFF)
-option(PACKAGE_DIR "Directory including all the files to be packaged (pre-loaded) in wasm builds" "")
+SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be packaged (pre-loaded) in wasm builds")
 
 # Set marian (3rd party submodule) cmake options to compile for this project
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")

From 28dcf55b417549f1b5ba7ec739e416166ac93591 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Fri, 12 Feb 2021 11:35:47 +0100
Subject: [PATCH 065/442] Improved cmake to use wasm compilation flags across
 project

---
 3rd_party/CMakeLists.txt      | 6 ++++++
 CMakeLists.txt                | 6 +++---
 src/translator/CMakeLists.txt | 1 +
 wasm/CMakeLists.txt           | 2 +-
 4 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index 644ac52de..74ce906dd 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -1,4 +1,10 @@
 add_subdirectory(marian-dev)
+
+if(COMPILE_WASM)
+  # This is a bad way of adding compilation flags. Will be improved soon.
+  add_compile_options(${WASM_COMPILE_FLAGS})
+endif(COMPILE_WASM)
+
 add_subdirectory(ssplit-cpp)
 
 # Add include directories for 3rd party targets to be able to use it anywhere in the
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 10256c218..677963f12 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -33,9 +33,9 @@ if(NOT COMPILE_WASM)
 endif()
 
 if(COMPILE_WASM)
-  add_compile_options(-pthread -O3 -g2 -fPIC -mssse3 -msimd128)
-  add_compile_options("SHELL:-s WASM=1" "SHELL:-s ASSERTIONS=1" "SHELL:-s DISABLE_EXCEPTION_CATCHING=0" "SHELL:-s LLD_REPORT_UNDEFINED" "SHELL:-s FORCE_FILESYSTEM=1" "SHELL:-s ALLOW_MEMORY_GROWTH=1")
-  add_compile_options(-Wno-error=pthreads-mem-growth)
+  list(APPEND WASM_COMPILE_FLAGS -pthread -O3 -g2 -fPIC -mssse3 -msimd128)
+  list(APPEND WASM_COMPILE_FLAGS "SHELL:-s WASM=1" "SHELL:-s ASSERTIONS=1" "SHELL:-s DISABLE_EXCEPTION_CATCHING=0" "SHELL:-s LLD_REPORT_UNDEFINED" "SHELL:-s FORCE_FILESYSTEM=1" "SHELL:-s ALLOW_MEMORY_GROWTH=1")
+  list(APPEND WASM_COMPILE_FLAGS -Wno-error=pthreads-mem-growth)
 endif(COMPILE_WASM)
 
 add_subdirectory(3rd_party)
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index ba2c2e033..1a664b3ef 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -21,6 +21,7 @@ if(COMPILE_WASM)
   target_compile_definitions(bergamot-translator PUBLIC USE_SSE2 WASM)
   # Enable code that is required for generating JS bindings
   target_compile_definitions(bergamot-translator PRIVATE WASM_BINDINGS)
+  target_compile_options(bergamot-translator PRIVATE ${WASM_COMPILE_FLAGS})
 endif(COMPILE_WASM)
 
 if (COMPILE_THREAD_VARIANT)
diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 9ede6a612..40b08bf6a 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -12,6 +12,7 @@ target_include_directories(bergamot-translator-worker
 )
 # This compile definition is required for generating binding code properly
 target_compile_definitions(bergamot-translator-worker PRIVATE WASM_BINDINGS)
+target_compile_options(bergamot-translator-worker PRIVATE ${WASM_COMPILE_FLAGS})
 
 set(LINKER_FLAGS "--bind -s ASSERTIONS=1 -s DISABLE_EXCEPTION_CATCHING=0 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1")
 if (NOT PACKAGE_DIR STREQUAL "")
@@ -22,6 +23,5 @@ set_target_properties(bergamot-translator-worker PROPERTIES
                         SUFFIX ".js"
                         LINK_FLAGS ${LINKER_FLAGS}
                         )
-#target_link_options(bergamot-translator-worker --preload-file ${PACKAGE_DIR}@/)
 
 target_link_libraries(bergamot-translator-worker bergamot-translator)

From 3b7673bf15e9877f3cfc15c17a366db8a494a4d5 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Fri, 12 Feb 2021 14:38:16 +0100
Subject: [PATCH 066/442] Updated marian-dev submodule

 - This fixes the issue of sentencepiece not being able to checkout
   properly
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index a4e50b66b..29ecba1cb 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit a4e50b66be38a94b90c46c4695d86de9932c34e8
+Subproject commit 29ecba1cb1b8ea26ae582d3851e214769b89e566

From 38e8b3cd6d5a2db561ce201c3e69fb79c676389c Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 5 Feb 2021 12:55:57 +0000
Subject: [PATCH 067/442] Updates: marian-dev, ssplit for marian-decoder-new

Updates marian-dev and ssplit submodules to point to the upstream
commits which implements the following:

 - marian-dev: encodeWithByteRanges(...) to get source token byte-ranges
 - ssplit: Has a trivial sentencesplitter functionality implemented, and
   now is faster to benchmark with marian-decoder.

This enables a marian-decoder replacement written through ssplit in this
source to be benchmarked constantly with existing marian-decoder.

Nits: Removes logging introduced for multiple workers, and respective
log statements.
---
 .gitignore                                    | 14 +++++
 3rd_party/marian-dev                          |  2 +-
 3rd_party/ssplit-cpp                          |  2 +-
 app/CMakeLists.txt                            |  3 +
 app/main-mts.cpp                              | 13 ----
 app/marian-decoder-new.cpp                    | 63 +++++++++++++++++++
 src/translator/CMakeLists.txt                 |  4 +-
 src/translator/batch_translator.cpp           |  1 -
 src/translator/batcher.cpp                    |  1 -
 src/translator/sanelogging.h                  | 44 -------------
 src/translator/sentence_splitter.cpp          | 52 +++++++++++++++
 src/translator/sentence_splitter.h            | 31 +++++++++
 src/translator/service.cpp                    |  1 -
 src/translator/service.h                      |  2 +-
 .../{textops.cpp => text_processor.cpp}       | 61 +++---------------
 .../{textops.h => text_processor.h}           | 37 +++--------
 16 files changed, 186 insertions(+), 145 deletions(-)
 create mode 100644 app/marian-decoder-new.cpp
 delete mode 100644 src/translator/sanelogging.h
 create mode 100644 src/translator/sentence_splitter.cpp
 create mode 100644 src/translator/sentence_splitter.h
 rename src/translator/{textops.cpp => text_processor.cpp} (52%)
 rename src/translator/{textops.h => text_processor.h} (56%)

diff --git a/.gitignore b/.gitignore
index e63aee1e1..54493b911 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,3 +2,17 @@
 *.swp
 *.swo
 
+# CMake
+CMakeLists.txt.user
+CMakeCache.txt
+CMakeFiles
+CMakeScripts
+Testing
+Makefile
+cmake_install.cmake
+install_manifest.txt
+compile_commands.json
+CTestTestfile.cmake
+_deps
+
+
diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index ee56e02f0..2f6528045 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit ee56e02f0525a4651157a07f74b44f456db14c8c
+Subproject commit 2f65280459737c37c270e4ad0b6d41de215d11e0
diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index f5d022992..01e71b496 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit f5d022992f4a00c860eb809389748908bb85ffcf
+Subproject commit 01e71b4964fdc351f932a7a23cab4cb80b9698e8
diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index 6e71e9e27..24bd0b43e 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -3,3 +3,6 @@ target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
 
 add_executable(service-cli main-mts.cpp)
 target_link_libraries(service-cli PRIVATE bergamot-translator)
+
+add_executable(marian-decoder-new marian-decoder-new.cpp)
+target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)
diff --git a/app/main-mts.cpp b/app/main-mts.cpp
index 44a019a0d..c94ff306c 100644
--- a/app/main-mts.cpp
+++ b/app/main-mts.cpp
@@ -26,21 +26,8 @@ int main(int argc, char *argv[]) {
       service.translate(std::move(input));
   translation_result_future.wait();
   const TranslationResult &translation_result = translation_result_future.get();
-
-  std::cout << "service-cli [Source text]: ";
-  std::cout << translation_result.getOriginalText() << std::endl;
-
-  std::cout << "service-cli [Translated text]: ";
   std::cout << translation_result.getTranslatedText() << std::endl;
 
-  // Obtain sentenceMappings and print them as Proof of Concept.
-  const TranslationResult::SentenceMappings &sentenceMappings =
-      translation_result.getSentenceMappings();
-  for (auto &p : sentenceMappings) {
-    std::cout << "service-cli [src] " << p.first << "\n";
-    std::cout << "service-cli [tgt] " << p.second << "\n";
-  }
-
   // Stop Service.
   service.stop();
   return 0;
diff --git a/app/marian-decoder-new.cpp b/app/marian-decoder-new.cpp
new file mode 100644
index 000000000..62b1bb4b3
--- /dev/null
+++ b/app/marian-decoder-new.cpp
@@ -0,0 +1,63 @@
+#include <cstdlib>
+#include <future>
+#include <iostream>
+#include <sstream>
+
+#include "common/definitions.h"
+#include "common/timer.h"
+#include "common/utils.h"
+#include "marian.h"
+#include "translator/history.h"
+#include "translator/output_collector.h"
+#include "translator/output_printer.h"
+#include "translator/parser.h"
+#include "translator/service.h"
+#include "translator/translation_result.h"
+
+void marian_decoder_minimal(const marian::Histories &histories,
+                            marian::Ptr<marian::Vocab const> targetVocab,
+                            marian::Ptr<marian::Options> options) {
+
+  bool doNbest = options->get<bool>("n-best");
+  auto collector =
+      marian::New<marian::OutputCollector>(options->get<std::string>("output"));
+
+  // There is a dependency of vocabs here.
+  auto printer = marian::New<marian::OutputPrinter>(options, targetVocab);
+  if (options->get<bool>("quiet-translation"))
+    collector->setPrintingStrategy(marian::New<marian::QuietPrinting>());
+
+  for (auto &history : histories) {
+    std::stringstream best1;
+    std::stringstream bestn;
+    printer->print(history, best1, bestn);
+    collector->Write((long)history->getLineNum(), best1.str(), bestn.str(),
+                     doNbest);
+  }
+}
+
+int main(int argc, char *argv[]) {
+  auto cp = marian::bergamot::createConfigParser();
+  auto options = cp.parseOptions(argc, argv, true);
+  marian::timer::Timer decoderTimer;
+
+  marian::bergamot::Service service(options);
+  // Read a large input text blob from stdin
+  std::ostringstream std_input;
+  std_input << std::cin.rdbuf();
+  std::string input = std_input.str();
+  using marian::bergamot::TranslationResult;
+
+  // Wait on future until TranslationResult is complete
+  std::future<TranslationResult> translation_result_future =
+      service.translate(std::move(input));
+  translation_result_future.wait();
+  const TranslationResult &translation_result = translation_result_future.get();
+
+  marian_decoder_minimal(translation_result.getHistories(),
+                         service.targetVocab(), options);
+
+  LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
+  service.stop();
+  return 0;
+}
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index b6fcf69fc..16c3db962 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -3,7 +3,8 @@ add_library(bergamot-translator STATIC
     TranslationModel.cpp
 
     # Following files added from browsermt/mts@nuke
-    textops.cpp
+    text_processor.cpp
+    sentence_splitter.cpp
     batch_translator.cpp 
     multifactor_priority.cpp 
     request.cpp 
@@ -18,3 +19,4 @@ target_include_directories(bergamot-translator
     PRIVATE ${CMAKE_SOURCE_DIR}
     PUBLIC ${CMAKE_SOURCE_DIR}/src)
 
+
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 6380a00cc..860255cd4 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -2,7 +2,6 @@
 #include "common/logging.h"
 #include "data/corpus.h"
 #include "data/text_input.h"
-#include "sanelogging.h"
 #include "translator/beam_search.h"
 
 namespace marian {
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 22ee46d2a..2fa4eaf09 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -1,6 +1,5 @@
 #include "batcher.h"
 #include "common/logging.h"
-#include "sanelogging.h"
 #include <cassert>
 
 namespace marian {
diff --git a/src/translator/sanelogging.h b/src/translator/sanelogging.h
deleted file mode 100644
index 21f70dda8..000000000
--- a/src/translator/sanelogging.h
+++ /dev/null
@@ -1,44 +0,0 @@
-#ifndef SRC_BERGAMOT_SANELOGGING_H_
-#define SRC_BERGAMOT_SANELOGGING_H_
-
-#include "spdlog/spdlog.h"
-#include <iostream>
-
-namespace marian {
-
-#define PLOG(worker, level, ...)
-#define _PLOG(worker, level, ...) checkedPLog(worker, #level, __VA_ARGS__)
-
-template <class... Args>
-void checkedPLog(std::string logger, std::string level, Args... args) {
-  Logger log = spdlog::get(logger);
-  if (!log) {
-    try {
-      log = spdlog::daily_logger_st(logger, "logs/" + logger + ".log");
-    } catch (const spdlog::spdlog_ex &ex) {
-      std::cout << "Log initialization failed: " << ex.what() << std::endl;
-    }
-  }
-
-  if (level == "trace")
-    log->trace(args...);
-  else if (level == "debug")
-    log->debug(args...);
-  else if (level == "info")
-    log->info(args...);
-  else if (level == "warn")
-    log->warn(args...);
-  else if (level == "error")
-    log->error(args...);
-  else if (level == "critical")
-    log->critical(args...);
-  else {
-    log->warn("Unknown log level '{}' for logger '{}'", level, logger);
-  }
-  // Not required when threads clean-exit.
-  log->flush();
-}
-
-} // namespace marian
-
-#endif // SRC_BERGAMOT_SANELOGGING_H_
diff --git a/src/translator/sentence_splitter.cpp b/src/translator/sentence_splitter.cpp
new file mode 100644
index 000000000..0f9be019a
--- /dev/null
+++ b/src/translator/sentence_splitter.cpp
@@ -0,0 +1,52 @@
+#include "common/cli_helper.h"
+#include "common/logging.h"
+#include "common/options.h"
+#include "sentence_splitter.h"
+#include <string>
+
+namespace marian {
+namespace bergamot {
+
+SentenceSplitter::SentenceSplitter(marian::Ptr<marian::Options> options)
+    : options_(options) {
+
+  std::string smode_str = options_->get<std::string>("ssplit-mode", "");
+  mode_ = string2splitmode(smode_str);
+  std::string ssplit_prefix_file =
+      options_->get<std::string>("ssplit-prefix-file", "");
+
+  if (ssplit_prefix_file.size()) {
+    ssplit_prefix_file = marian::cli::interpolateEnvVars(ssplit_prefix_file);
+
+    LOG(info, "Loading protected prefixes for sentence splitting from {}",
+        ssplit_prefix_file);
+
+    ssplit_.load(ssplit_prefix_file);
+  } else {
+    LOG(warn, "Missing list of protected prefixes for sentence splitting. "
+              "Set with --ssplit-prefix-file.");
+  }
+}
+
+ug::ssplit::SentenceStream
+SentenceSplitter::createSentenceStream(const string_view &input) {
+  return std::move(ug::ssplit::SentenceStream(input.data(), input.size(),
+                                              this->ssplit_, mode_));
+}
+
+ug::ssplit::SentenceStream::splitmode
+SentenceSplitter::string2splitmode(const std::string &m) {
+  typedef ug::ssplit::SentenceStream::splitmode splitmode;
+  // @TODO: throw Exception on error
+  if (m == "sentence" || m == "Sentence")
+    return splitmode::one_sentence_per_line;
+  if (m == "paragraph" || m == "Paragraph")
+    return splitmode::one_paragraph_per_line;
+  if (m != "wrapped_text" && m != "WrappedText" && m != "wrappedText") {
+    LOG(warn, "Ignoring unknown text input format specification: {}.", m);
+  }
+  return splitmode::wrapped_text;
+}
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/sentence_splitter.h b/src/translator/sentence_splitter.h
new file mode 100644
index 000000000..5175176bf
--- /dev/null
+++ b/src/translator/sentence_splitter.h
@@ -0,0 +1,31 @@
+#ifndef SRC_BERGAMOT_SENTENCE_SPLITTER_H_
+#define SRC_BERGAMOT_SENTENCE_SPLITTER_H_
+
+#include "common/options.h"
+#include "data/types.h"
+#include "ssplit.h"
+#include <string>
+
+namespace marian {
+namespace bergamot {
+
+class SentenceSplitter {
+  // A wrapper around @ugermann's ssplit-cpp compiled from several places in
+  // mts. Constructed based on options. Used in TextProcessor below to create
+  // sentence-streams, which provide access to one sentence from blob of text at
+  // a time.
+public:
+  explicit SentenceSplitter(Ptr<Options> options);
+  ug::ssplit::SentenceStream createSentenceStream(string_view const &input);
+
+private:
+  ug::ssplit::SentenceSplitter ssplit_;
+  Ptr<Options> options_;
+  ug::ssplit::SentenceStream::splitmode mode_;
+  ug::ssplit::SentenceStream::splitmode string2splitmode(const std::string &m);
+};
+
+} // namespace bergamot
+} // namespace marian
+
+#endif //  SRC_BERGAMOT_SENTENCE_SPLITTER_H_
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 4a5af301c..2acbbdb1b 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -1,6 +1,5 @@
 #include "service.h"
 #include "definitions.h"
-#include "sanelogging.h"
 
 #include <string>
 #include <utility>
diff --git a/src/translator/service.h b/src/translator/service.h
index 4069d1392..0ed8d0c1e 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -4,7 +4,7 @@
 #include "batch_translator.h"
 #include "batcher.h"
 #include "pcqueue.h"
-#include "textops.h"
+#include "text_processor.h"
 #include "translation_result.h"
 
 #include <queue>
diff --git a/src/translator/textops.cpp b/src/translator/text_processor.cpp
similarity index 52%
rename from src/translator/textops.cpp
rename to src/translator/text_processor.cpp
index 25e48f1fd..8114855bb 100644
--- a/src/translator/textops.cpp
+++ b/src/translator/text_processor.cpp
@@ -1,58 +1,17 @@
-#include "textops.h"
-#include "common/timer.h"
-#include <pcrecpp.h>
-#include <string>
-#include <unordered_map>
-#include <utility>
+#include "text_processor.h"
+#include "data/types.h"
+#include "definitions.h"
+
+#include "common/options.h"
+#include "data/vocab.h"
 #include <vector>
 
 namespace marian {
 namespace bergamot {
 
-SentenceSplitter::SentenceSplitter(marian::Ptr<marian::Options> options)
-    : options_(options) {
-
-  std::string smode_str = options_->get<std::string>("ssplit-mode", "");
-  mode_ = string2splitmode(smode_str);
-  std::string ssplit_prefix_file =
-      options_->get<std::string>("ssplit-prefix-file", "");
-
-  if (ssplit_prefix_file.size()) {
-    ssplit_prefix_file = marian::cli::interpolateEnvVars(ssplit_prefix_file);
-
-    LOG(info, "Loading protected prefixes for sentence splitting from {}",
-        ssplit_prefix_file);
-
-    ssplit_.load(ssplit_prefix_file);
-  } else {
-    LOG(warn, "Missing list of protected prefixes for sentence splitting. "
-              "Set with --ssplit-prefix-file.");
-  }
-}
-
-ug::ssplit::SentenceStream
-SentenceSplitter::createSentenceStream(const string_view &input) {
-  pcrecpp::StringPiece spiece(input.begin(), input.size());
-  return std::move(ug::ssplit::SentenceStream(spiece, this->ssplit_, mode_));
-}
-
-ug::ssplit::SentenceStream::splitmode
-SentenceSplitter::string2splitmode(const std::string &m) {
-  typedef ug::ssplit::SentenceStream::splitmode splitmode;
-  // @TODO: throw Exception on error
-  if (m == "sentence" || m == "Sentence")
-    return splitmode::one_sentence_per_line;
-  if (m == "paragraph" || m == "Paragraph")
-    return splitmode::one_paragraph_per_line;
-  if (m != "wrapped_text" && m != "WrappedText" && m != "wrappedText") {
-    LOG(warn, "Ignoring unknown text input format specification: {}.", m);
-  }
-  return splitmode::wrapped_text;
-}
-
 Segment TextProcessor::tokenize(const string_view &segment,
                                 TokenRanges &tokenRanges) {
-  return vocabs_->front()->encodePreservingSource(
+  return vocabs_->front()->encodeWithByteRanges(
       segment, tokenRanges, /*addEOS=*/false, /*inference=*/true);
 }
 
@@ -70,11 +29,11 @@ void TextProcessor::process(const string_view &query, Segments &segments,
                             std::vector<TokenRanges> &sourceRanges) {
 
   auto sentenceStream = sentence_splitter_.createSentenceStream(query);
-  pcrecpp::StringPiece sentenceStringPiece;
+  std::string_view sentenceStringPiece;
 
   while (sentenceStream >> sentenceStringPiece) {
-    string_view sentence(sentenceStringPiece.data(),
-                         sentenceStringPiece.size());
+    marian::string_view sentence(sentenceStringPiece.data(),
+                                 sentenceStringPiece.size());
     TokenRanges tokenRanges;
     Segment segment = tokenize(sentence, tokenRanges);
 
diff --git a/src/translator/textops.h b/src/translator/text_processor.h
similarity index 56%
rename from src/translator/textops.h
rename to src/translator/text_processor.h
index 79a504013..111ae009b 100644
--- a/src/translator/textops.h
+++ b/src/translator/text_processor.h
@@ -1,40 +1,17 @@
-#ifndef SRC_BERGAMOT_TEXTOPS_H_
-#define SRC_BERGAMOT_TEXTOPS_H_
+#ifndef SRC_BERGAMOT_TEXT_PROCESSOR_H_
+#define SRC_BERGAMOT_TEXT_PROCESSOR_H_
 
-#include "common/definitions.h"
-#include "common/logging.h"
-#include "common/options.h"
-#include "common/types.h" // missing in shortlist.h
-#include "common/utils.h"
-#include "data/sentencepiece_vocab.h"
-#include "data/shortlist.h"
+#include "data/types.h"
+#include "data/vocab.h"
 #include "definitions.h"
-#include "ssplit.h"
 
-#include <cassert>
-#include <iostream>
-#include <string>
+#include "sentence_splitter.h"
+
 #include <vector>
 
 namespace marian {
 namespace bergamot {
 
-class SentenceSplitter {
-  // A wrapper around @ugermann's ssplit-cpp compiled from several places in
-  // mts. Constructed based on options. Used in TextProcessor below to create
-  // sentence-streams, which provide access to one sentence from blob of text at
-  // a time.
-public:
-  explicit SentenceSplitter(Ptr<Options> options);
-  ug::ssplit::SentenceStream createSentenceStream(string_view const &input);
-
-private:
-  ug::ssplit::SentenceSplitter ssplit_;
-  Ptr<Options> options_;
-  ug::ssplit::SentenceStream::splitmode mode_;
-  ug::ssplit::SentenceStream::splitmode string2splitmode(const std::string &m);
-};
-
 class TextProcessor {
   // TextProcessor handles loading the sentencepiece vocabulary and also
   // contains an instance of sentence-splitter based on ssplit.
@@ -68,4 +45,4 @@ class TextProcessor {
 } // namespace bergamot
 } // namespace marian
 
-#endif // SRC_BERGAMOT_TEXTOPS_H_
+#endif // SRC_BERGAMOT_TEXT_PROCESSOR_H_

From 9108d9f0b3e96c1890746ab740df1901b5cc2245 Mon Sep 17 00:00:00 2001
From: Andre Natal <andrenatal@users.noreply.github.com>
Date: Fri, 12 Feb 2021 15:25:40 -0800
Subject: [PATCH 068/442] Update README.md

Add  `--recursive` to `git clone` instructions
---
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.md b/README.md
index e1ad9c37a..e8adaba32 100644
--- a/README.md
+++ b/README.md
@@ -7,7 +7,7 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
 ### Build Natively
 
 ```bash
-git clone https://github.com/browsermt/bergamot-translator
+git clone  --recursive https://github.com/browsermt/bergamot-translator
 cd bergamot-translator
 mkdir build
 cd build

From 3a53a68444834aeb6e78bfdb35ae12570187acd7 Mon Sep 17 00:00:00 2001
From: Andre Natal <andrenatal@users.noreply.github.com>
Date: Fri, 12 Feb 2021 15:41:17 -0800
Subject: [PATCH 069/442] Update README.md

updating  `--recursive`  on wasm instructions too
---
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.md b/README.md
index e8adaba32..4b1094415 100644
--- a/README.md
+++ b/README.md
@@ -19,7 +19,7 @@ make -j
 
 To compile WASM, first download and Install Emscripten using following instructions:
 
-1. Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
+1. Get the latest sdk: `git clone  --recursive https://github.com/emscripten-core/emsdk.git`
 2. Enter the cloned directory: `cd emsdk`
 3. Install the lastest sdk tools: `./emsdk install latest`
 4. Activate the latest sdk tools: `./emsdk activate latest`

From a97bf7b504e151494d3206e8b2459e666482640b Mon Sep 17 00:00:00 2001
From: Andre Natal <andrenatal@users.noreply.github.com>
Date: Fri, 12 Feb 2021 17:00:12 -0800
Subject: [PATCH 070/442] Update README.md

---
 README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index 4b1094415..2791ebf96 100644
--- a/README.md
+++ b/README.md
@@ -19,7 +19,7 @@ make -j
 
 To compile WASM, first download and Install Emscripten using following instructions:
 
-1. Get the latest sdk: `git clone  --recursive https://github.com/emscripten-core/emsdk.git`
+1. Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
 2. Enter the cloned directory: `cd emsdk`
 3. Install the lastest sdk tools: `./emsdk install latest`
 4. Activate the latest sdk tools: `./emsdk activate latest`
@@ -28,7 +28,7 @@ To compile WASM, first download and Install Emscripten using following instructi
 After the successful installation of Emscripten, perform these steps:
 
 ```bash
-git clone https://github.com/browsermt/bergamot-translator
+git clone --recursive https://github.com/browsermt/bergamot-translator
 cd bergamot-translator
 mkdir build-wasm
 cd build-wasm

From 47db65972cd791cbb59b4ee9825e1d80a1e9d0f1 Mon Sep 17 00:00:00 2001
From: Andre Natal <andrenatal@users.noreply.github.com>
Date: Fri, 12 Feb 2021 17:18:57 -0800
Subject: [PATCH 071/442] Update README.md

---
 README.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/README.md b/README.md
index 2791ebf96..3e458dfe0 100644
--- a/README.md
+++ b/README.md
@@ -30,6 +30,8 @@ After the successful installation of Emscripten, perform these steps:
 ```bash
 git clone --recursive https://github.com/browsermt/bergamot-translator
 cd bergamot-translator
+git checkout wasm-integration
+git submodule update --recursive
 mkdir build-wasm
 cd build-wasm
 emcmake cmake -DCOMPILE_WASM=on ../

From 4764f11e95cb2ec3c2766949ba58a74ee0d2cc90 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sat, 13 Feb 2021 10:55:07 +0000
Subject: [PATCH 072/442] Move BatchTranslator::thread_ to Service (#10)

Service now holds an std::vector<std::thread> instead of
BatchTranslators.
---
 src/translator/batch_translator.cpp | 26 +++++++++++---------------
 src/translator/batch_translator.h   | 19 ++++++++-----------
 src/translator/service.cpp          |  8 +++++---
 src/translator/service.h            |  2 +-
 4 files changed, 25 insertions(+), 30 deletions(-)

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 860255cd4..7f801c97c 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -8,15 +8,10 @@ namespace marian {
 namespace bergamot {
 
 BatchTranslator::BatchTranslator(DeviceId const device,
-                                 PCQueue<PCItem> &pcqueue,
                                  std::vector<Ptr<Vocab const>> &vocabs,
                                  Ptr<Options> options)
-    : device_(device), options_(options), pcqueue_(&pcqueue), vocabs_(&vocabs) {
-
-  thread_ = std::thread([this] { this->mainloop(); });
-}
-
-void BatchTranslator::initGraph() {
+    : device_(device), options_(options), vocabs_(&vocabs) {
+  // Initializes the graph.
   if (options_->hasAndNotEmpty("shortlist")) {
     int srcIdx = 0, trgIdx = 1;
     bool shared_vcb = vocabs_->front() == vocabs_->back();
@@ -38,7 +33,6 @@ void BatchTranslator::initGraph() {
       scorer->setShortlistGenerator(slgen_);
     }
   }
-
   graph_->forward();
 }
 
@@ -98,18 +92,22 @@ void BatchTranslator::translate(RequestSentences &requestSentences,
   histories = std::move(search->search(graph_, batch));
 }
 
-void BatchTranslator::mainloop() {
-  initGraph();
+// void BatchTranslator::join() { thread_.join(); }
+
+void translation_loop(DeviceId const &device, PCQueue<PCItem> &pcqueue,
+                      std::vector<Ptr<Vocab const>> &vocabs,
+                      Ptr<Options> options) {
+
+  BatchTranslator translator(device, vocabs, options);
 
   PCItem pcitem;
   Histories histories;
-
   while (true) {
-    pcqueue_->ConsumeSwap(pcitem);
+    pcqueue.ConsumeSwap(pcitem);
     if (pcitem.isPoison()) {
       return;
     } else {
-      translate(pcitem.sentences, histories);
+      translator.translate(pcitem.sentences, histories);
       for (int i = 0; i < pcitem.sentences.size(); i++) {
         pcitem.sentences[i].completeSentence(histories[i]);
       }
@@ -117,7 +115,5 @@ void BatchTranslator::mainloop() {
   }
 }
 
-void BatchTranslator::join() { thread_.join(); }
-
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 069155efb..c718b32a0 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -22,29 +22,26 @@ class BatchTranslator {
   // shut down in Service which calls join() on the threads.
 
 public:
-  BatchTranslator(DeviceId const device, PCQueue<PCItem> &pcqueue,
-                  std::vector<Ptr<Vocab const>> &vocabs, Ptr<Options> options);
-  void join();
+  BatchTranslator(DeviceId const device, std::vector<Ptr<Vocab const>> &vocabs,
+                  Ptr<Options> options);
 
   // convenience function for logging. TODO(jerin)
   std::string _identifier() { return "worker" + std::to_string(device_.no); }
-
-private:
-  void initGraph();
   void translate(RequestSentences &requestSentences, Histories &histories);
-  void mainloop();
 
+private:
   Ptr<Options> options_;
-
   DeviceId device_;
   std::vector<Ptr<Vocab const>> *vocabs_;
   Ptr<ExpressionGraph> graph_;
   std::vector<Ptr<Scorer>> scorers_;
   Ptr<data::ShortlistGenerator const> slgen_;
-
-  PCQueue<PCItem> *pcqueue_;
-  std::thread thread_;
 };
+
+void translation_loop(DeviceId const &device, PCQueue<PCItem> &pcqueue,
+                      std::vector<Ptr<Vocab const>> &vocabs,
+                      Ptr<Options> options);
+
 } // namespace bergamot
 } // namespace marian
 
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 2acbbdb1b..62073f931 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -16,9 +16,11 @@ Service::Service(Ptr<Options> options)
 
   workers_.reserve(numWorkers_);
 
-  for (int i = 0; i < numWorkers_; i++) {
-    marian::DeviceId deviceId(i, DeviceType::cpu);
-    workers_.emplace_back(deviceId, pcqueue_, vocabs_, options);
+  for (int cpuId = 0; cpuId < numWorkers_; cpuId++) {
+    workers_.emplace_back([&] {
+      marian::DeviceId deviceId(cpuId, DeviceType::cpu);
+      translation_loop(deviceId, pcqueue_, vocabs_, options);
+    });
   }
 }
 
diff --git a/src/translator/service.h b/src/translator/service.h
index 0ed8d0c1e..e516bba60 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -69,7 +69,7 @@ class Service {
   TextProcessor text_processor_; // ORDER DEPENDENCY
   Batcher batcher_;
   PCQueue<PCItem> pcqueue_;
-  std::vector<BatchTranslator> workers_;
+  std::vector<std::thread> workers_;
 };
 
 std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);

From f1d9f67b56ed5d84f74236b166fd592c060bf8d2 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sat, 13 Feb 2021 11:42:57 +0000
Subject: [PATCH 073/442] single-threaded run with --cpu-threads 0 (#10)

---
 src/translator/batch_translator.cpp | 13 +++----
 src/translator/batch_translator.h   |  2 +-
 src/translator/batcher.cpp          | 25 +++++++++++++
 src/translator/batcher.h            |  4 ++
 src/translator/service.cpp          | 57 ++++++++++++++++-------------
 src/translator/service.h            |  3 ++
 6 files changed, 70 insertions(+), 34 deletions(-)

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 7f801c97c..3d2ec41c3 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -36,8 +36,7 @@ BatchTranslator::BatchTranslator(DeviceId const device,
   graph_->forward();
 }
 
-void BatchTranslator::translate(RequestSentences &requestSentences,
-                                Histories &histories) {
+void BatchTranslator::translate(RequestSentences &requestSentences) {
   std::vector<data::SentenceTuple> batchVector;
 
   for (auto &sentence : requestSentences) {
@@ -89,7 +88,10 @@ void BatchTranslator::translate(RequestSentences &requestSentences,
   auto trgVocab = vocabs_->back();
   auto search = New<BeamSearch>(options_, scorers_, trgVocab);
 
-  histories = std::move(search->search(graph_, batch));
+  auto histories = std::move(search->search(graph_, batch));
+  for (int i = 0; i < requestSentences.size(); i++) {
+    requestSentences[i].completeSentence(histories[i]);
+  }
 }
 
 // void BatchTranslator::join() { thread_.join(); }
@@ -107,10 +109,7 @@ void translation_loop(DeviceId const &device, PCQueue<PCItem> &pcqueue,
     if (pcitem.isPoison()) {
       return;
     } else {
-      translator.translate(pcitem.sentences, histories);
-      for (int i = 0; i < pcitem.sentences.size(); i++) {
-        pcitem.sentences[i].completeSentence(histories[i]);
-      }
+      translator.translate(pcitem.sentences);
     }
   }
 }
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index c718b32a0..4067e59a0 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -27,7 +27,7 @@ class BatchTranslator {
 
   // convenience function for logging. TODO(jerin)
   std::string _identifier() { return "worker" + std::to_string(device_.no); }
-  void translate(RequestSentences &requestSentences, Histories &histories);
+  void translate(RequestSentences &requestSentences);
 
 private:
   Ptr<Options> options_;
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 2fa4eaf09..18bf5fdc1 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -50,5 +50,30 @@ void Batcher::cleaveBatch(RequestSentences &sentences) {
   }
 }
 
+void Batcher::addWholeRequest(Ptr<Request> request) {
+  for (int i = 0; i < request->numSegments(); i++) {
+    RequestSentence requestSentence(i, request);
+    addSentenceWithPriority(requestSentence);
+  }
+}
+
+void Batcher::enqueue(PCQueue<PCItem> &pcqueue) {
+  int numSentences;
+  do {
+    RequestSentences batchSentences;
+    cleaveBatch(batchSentences);
+    numSentences = batchSentences.size();
+
+    if (numSentences > 0) {
+      PCItem pcitem(batchNumber_++, std::move(batchSentences));
+      pcqueue.ProduceSwap(pcitem);
+    }
+
+    if (batchNumber_ % 500 == 0) {
+      LOG(info, "Queuing batch {}", batchNumber_);
+    }
+  } while (numSentences > 0);
+}
+
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index b60b642c7..2499cd2ff 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -4,6 +4,7 @@
 #include "common/options.h"
 #include "data/corpus_base.h"
 #include "definitions.h"
+#include "pcqueue.h"
 #include "request.h"
 
 #include <set>
@@ -19,6 +20,8 @@ class Batcher {
   // sentence. This method inserts the sentence into the internal data-structure
   // which maintains priority among sentences from multiple concurrent requests.
   void addSentenceWithPriority(RequestSentence &sentence);
+  void addWholeRequest(Ptr<Request> request);
+  void enqueue(PCQueue<PCItem> &pcqueue);
 
   // Loads sentences with sentences compiled from (tentatively) multiple
   // requests optimizing for both padding and priority.
@@ -27,6 +30,7 @@ class Batcher {
 private:
   unsigned int max_input_tokens_;
   std::vector<std::set<RequestSentence>> bucket_;
+  unsigned int batchNumber_{0};
 };
 
 } // namespace bergamot
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 62073f931..fc713851e 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -14,13 +14,17 @@ Service::Service(Ptr<Options> options)
       text_processor_(vocabs_, options), batcher_(options),
       pcqueue_(2 * options->get<int>("cpu-threads")) {
 
-  workers_.reserve(numWorkers_);
-
-  for (int cpuId = 0; cpuId < numWorkers_; cpuId++) {
-    workers_.emplace_back([&] {
-      marian::DeviceId deviceId(cpuId, DeviceType::cpu);
-      translation_loop(deviceId, pcqueue_, vocabs_, options);
-    });
+  if (numWorkers_ > 0) {
+    workers_.reserve(numWorkers_);
+    for (int cpuId = 0; cpuId < numWorkers_; cpuId++) {
+      workers_.emplace_back([&] {
+        marian::DeviceId deviceId(cpuId, DeviceType::cpu);
+        translation_loop(deviceId, pcqueue_, vocabs_, options);
+      });
+    }
+  } else {
+    marian::DeviceId deviceId(/*cpuId=*/0, DeviceType::cpu);
+    translator = new BatchTranslator(deviceId, vocabs_, options);
   }
 }
 
@@ -53,27 +57,28 @@ std::future<TranslationResult> Service::translate(std::string &&input) {
       std::move(segments), std::move(sourceAlignments),
       std::move(translationResultPromise));
 
-  for (int i = 0; i < request->numSegments(); i++) {
-    RequestSentence requestSentence(i, request);
-    batcher_.addSentenceWithPriority(requestSentence);
+  batcher_.addWholeRequest(request);
+  if (numWorkers_ > 0) {
+    batcher_.enqueue(pcqueue_);
+  } else {
+    // Queue single-threaded
+    int numSentences;
+    do {
+      RequestSentences batchSentences;
+      batcher_.cleaveBatch(batchSentences);
+      numSentences = batchSentences.size();
+
+      if (numSentences > 0) {
+        translator->translate(batchSentences);
+        batchNumber_++;
+      }
+
+      if (batchNumber_ % 500 == 0) {
+        LOG(info, "Tranlsating batch {}", batchNumber_);
+      }
+    } while (numSentences > 0);
   }
 
-  int numSentences;
-  do {
-    RequestSentences batchSentences;
-    batcher_.cleaveBatch(batchSentences);
-    numSentences = batchSentences.size();
-
-    if (numSentences > 0) {
-      PCItem pcitem(batchNumber_++, std::move(batchSentences));
-      pcqueue_.ProduceSwap(pcitem);
-    }
-
-    if (batchNumber_ % 500 == 0) {
-      LOG(info, "Queuing batch {}", batchNumber_);
-    }
-  } while (numSentences > 0);
-
   return future;
 }
 
diff --git a/src/translator/service.h b/src/translator/service.h
index e516bba60..951398df5 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -70,6 +70,9 @@ class Service {
   Batcher batcher_;
   PCQueue<PCItem> pcqueue_;
   std::vector<std::thread> workers_;
+
+  // Optional
+  BatchTranslator *translator{nullptr};
 };
 
 std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);

From 77a600b637afd854a189f96b052f37896d37acb7 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sat, 13 Feb 2021 14:19:10 +0000
Subject: [PATCH 074/442] Removing join() (#10)

---
 src/translator/batch_translator.cpp | 2 --
 1 file changed, 2 deletions(-)

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 3d2ec41c3..b944bed32 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -94,8 +94,6 @@ void BatchTranslator::translate(RequestSentences &requestSentences) {
   }
 }
 
-// void BatchTranslator::join() { thread_.join(); }
-
 void translation_loop(DeviceId const &device, PCQueue<PCItem> &pcqueue,
                       std::vector<Ptr<Vocab const>> &vocabs,
                       Ptr<Options> options) {

From 73a56a8f4fa447fb58e230905c7c6e3d25c366da Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sat, 13 Feb 2021 15:48:23 +0000
Subject: [PATCH 075/442] Refactoring batching-mechanisms into Batcher

Guided by an objective to  move batching mechanism and queueing request
to generate batches into a diffenrent thread. This commit is in
preparation for this functionality.

First, PCItem from the looks of it is *Batch*. Renamed to reflect the
same. Fingers crossed, hopefully no naming conflicts with marian.
BatchTranslator translates a "Batch" now, instead of
vector<RequestSentence>. Additional data members are setup at Batch to
enable development.

Workflows previously in Service, but more adequate in Batcher are now
moved, preparing to move Batcher/enqueuing of a request into a new
thread making it non-blocking. This will allow service to queue requests
into the batcher thread and exit, without waiting until the full-request
is queued.

Batcher now has a path with and without pcqueue.
---
 src/translator/batch_translator.cpp | 25 +++++-----
 src/translator/batch_translator.h   |  4 +-
 src/translator/batcher.cpp          | 73 +++++++++++++++--------------
 src/translator/batcher.h            |  7 +--
 src/translator/request.h            | 22 +++++----
 src/translator/service.cpp          | 27 ++++-------
 src/translator/service.h            |  3 +-
 7 files changed, 78 insertions(+), 83 deletions(-)

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index b944bed32..a6e6b9347 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -36,10 +36,10 @@ BatchTranslator::BatchTranslator(DeviceId const device,
   graph_->forward();
 }
 
-void BatchTranslator::translate(RequestSentences &requestSentences) {
+void BatchTranslator::translate(Batch &batch) {
   std::vector<data::SentenceTuple> batchVector;
 
-  for (auto &sentence : requestSentences) {
+  for (auto &sentence : batch.sentences) {
     data::SentenceTuple sentence_tuple(sentence.lineNumber());
     Segment segment = sentence.getUnderlyingSegment();
     sentence_tuple.push_back(segment);
@@ -82,32 +82,31 @@ void BatchTranslator::translate(RequestSentences &requestSentences) {
   for (size_t j = 0; j < maxDims.size(); ++j)
     subBatches[j]->setWords(words[j]);
 
-  auto batch = Ptr<CorpusBatch>(new CorpusBatch(subBatches));
-  batch->setSentenceIds(sentenceIds);
+  auto corpus_batch = Ptr<CorpusBatch>(new CorpusBatch(subBatches));
+  corpus_batch->setSentenceIds(sentenceIds);
 
   auto trgVocab = vocabs_->back();
   auto search = New<BeamSearch>(options_, scorers_, trgVocab);
 
-  auto histories = std::move(search->search(graph_, batch));
-  for (int i = 0; i < requestSentences.size(); i++) {
-    requestSentences[i].completeSentence(histories[i]);
+  auto histories = std::move(search->search(graph_, corpus_batch));
+  for (int i = 0; i < batch.sentences.size(); i++) {
+    batch.sentences[i].completeSentence(histories[i]);
   }
 }
 
-void translation_loop(DeviceId const &device, PCQueue<PCItem> &pcqueue,
+void translation_loop(DeviceId const &device, PCQueue<Batch> &pcqueue,
                       std::vector<Ptr<Vocab const>> &vocabs,
                       Ptr<Options> options) {
 
   BatchTranslator translator(device, vocabs, options);
-
-  PCItem pcitem;
+  Batch batch;
   Histories histories;
   while (true) {
-    pcqueue.ConsumeSwap(pcitem);
-    if (pcitem.isPoison()) {
+    pcqueue.ConsumeSwap(batch);
+    if (batch.isPoison()) {
       return;
     } else {
-      translator.translate(pcitem.sentences);
+      translator.translate(batch);
     }
   }
 }
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 4067e59a0..2ee4e04ef 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -27,7 +27,7 @@ class BatchTranslator {
 
   // convenience function for logging. TODO(jerin)
   std::string _identifier() { return "worker" + std::to_string(device_.no); }
-  void translate(RequestSentences &requestSentences);
+  void translate(Batch &batch);
 
 private:
   Ptr<Options> options_;
@@ -38,7 +38,7 @@ class BatchTranslator {
   Ptr<data::ShortlistGenerator const> slgen_;
 };
 
-void translation_loop(DeviceId const &device, PCQueue<PCItem> &pcqueue,
+void translation_loop(DeviceId const &device, PCQueue<Batch> &pcqueue,
                       std::vector<Ptr<Vocab const>> &vocabs,
                       Ptr<Options> options);
 
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 18bf5fdc1..13b563542 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -6,10 +6,10 @@ namespace marian {
 namespace bergamot {
 
 Batcher::Batcher(Ptr<Options> options) {
-  max_input_tokens_ = options->get<int>("max-input-tokens");
+  miniBatchWords = options->get<int>("max-input-tokens");
   bucket_.resize(options->get<int>("max-input-sentence-tokens") + 1);
   ABORT_IF(
-      max_input_tokens_ < bucket_.size() - 1,
+      miniBatchWords < bucket_.size() - 1,
       "max-input-tokens cannot be less than than max-input-sentence-tokens, "
       "batcher fail");
 }
@@ -20,34 +20,48 @@ void Batcher::addSentenceWithPriority(RequestSentence &sentence) {
   bucket_[bucket_id].insert(sentence);
 }
 
-void Batcher::cleaveBatch(RequestSentences &sentences) {
+bool Batcher::operator>>(Batch &batch) { return cleaveBatch(batch); }
+
+bool Batcher::cleaveBatch(Batch &batch) {
   // For now simply iterates on buckets and converts batches greedily.  This
   // has to be enhanced with optimizing over priority. The baseline
   // implementation should at least be as fast as marian's maxi-batch with full
   // corpus size as maxi-batch size.
+  batch.reset();
+  int paddedBatchSize = 0;
 
-  int segments_added = 0;
-  int current_input_tokens = 0;
-  int padded_batch_size = 0;
-  int prev_padded_batch_size;
-
-  for (int i = 0; i < bucket_.size(); i++) {
-    auto p = bucket_[i].begin();
-    while (p != bucket_[i].end()) {
-      padded_batch_size = (segments_added + 1) * i;
-      if (padded_batch_size <= max_input_tokens_) {
+  for (int length = 0; length < bucket_.size(); length++) {
+    auto p = bucket_[length].begin();
+    while (p != bucket_[length].end()) {
+      paddedBatchSize = (batch.sentences.size() + 1) * length;
+      if (paddedBatchSize <= miniBatchWords) {
         auto q = p;
         ++p;
-        current_input_tokens += i;
-        sentences.push_back(*q);
-        ++segments_added;
-        bucket_[i].erase(q);
-        prev_padded_batch_size = padded_batch_size;
+
+        batch.numTokens += length;
+        batch.sentences.push_back(*q);
+        batch.maxLength = std::max(batch.maxLength, length);
+
+        bucket_[length].erase(q);
       } else {
-        return;
+        // Check if elements exist
+        assert(batch.sentences.size() > 0);
+        batch.Id = ++batchNumber_;
+        if (batchId % 500 == 0) {
+          batch.log();
+        }
+        return true;
       }
     }
   }
+
+  if (batch.sentences.size()) {
+    batch.Id = ++batchNumber_;
+    batch.log();
+    return true;
+  } else {
+    return false;
+  }
 }
 
 void Batcher::addWholeRequest(Ptr<Request> request) {
@@ -57,22 +71,11 @@ void Batcher::addWholeRequest(Ptr<Request> request) {
   }
 }
 
-void Batcher::enqueue(PCQueue<PCItem> &pcqueue) {
-  int numSentences;
-  do {
-    RequestSentences batchSentences;
-    cleaveBatch(batchSentences);
-    numSentences = batchSentences.size();
-
-    if (numSentences > 0) {
-      PCItem pcitem(batchNumber_++, std::move(batchSentences));
-      pcqueue.ProduceSwap(pcitem);
-    }
-
-    if (batchNumber_ % 500 == 0) {
-      LOG(info, "Queuing batch {}", batchNumber_);
-    }
-  } while (numSentences > 0);
+void Batcher::enqueue(PCQueue<Batch> &pcqueue) {
+  Batch batch;
+  while (cleaveBatch(batch)) {
+    pcqueue.ProduceSwap(batch);
+  }
 }
 
 } // namespace bergamot
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index 2499cd2ff..d6b85f3f3 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -21,14 +21,15 @@ class Batcher {
   // which maintains priority among sentences from multiple concurrent requests.
   void addSentenceWithPriority(RequestSentence &sentence);
   void addWholeRequest(Ptr<Request> request);
-  void enqueue(PCQueue<PCItem> &pcqueue);
+  void enqueue(PCQueue<Batch> &pcqueue);
 
   // Loads sentences with sentences compiled from (tentatively) multiple
   // requests optimizing for both padding and priority.
-  void cleaveBatch(RequestSentences &sentences);
+  bool cleaveBatch(Batch &batch);
+  bool operator>>(Batch &batch); // alias
 
 private:
-  unsigned int max_input_tokens_;
+  unsigned int miniBatchWords;
   std::vector<std::set<RequestSentence>> bucket_;
   unsigned int batchNumber_{0};
 };
diff --git a/src/translator/request.h b/src/translator/request.h
index 6f268ba1c..673f88ce3 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -24,6 +24,7 @@
 #include "definitions.h"
 #include "translation_result.h"
 
+#include "common/logging.h"
 #include "data/types.h"
 #include "translator/beam_search.h"
 
@@ -92,20 +93,23 @@ class RequestSentence {
 
 typedef std::vector<RequestSentence> RequestSentences;
 
-struct PCItem {
-  int batchNumber;
+struct Batch {
+  int Id;
+  int numTokens, maxLength;
   RequestSentences sentences;
 
-  // PCItem should be default constructible for PCQueue. Default constructed
+  // Batch should be default constructible for PCQueue. Default constructed
   // element is poison.
-  PCItem() : batchNumber(-1) {}
-
-  // PCItem constructor to construct a legit PCItem.
-  explicit PCItem(int batchNumber, RequestSentences &&sentences)
-      : batchNumber(batchNumber), sentences(std::move(sentences)) {}
+  Batch() { reset(); }
+  void reset() { Id = -1, numTokens = 0, maxLength = 0, sentences.clear(); }
 
   // Convenience function to determine poison.
-  bool isPoison() { return (batchNumber == -1); }
+  bool isPoison() { return (Id == -1); }
+
+  void log() {
+    LOG(info, "Batch(Id={}, tokens={}, max-length={}, sentences={})", Id,
+        numTokens, maxLength, sentences.size());
+  }
 };
 
 } // namespace bergamot
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index fc713851e..37019552c 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -8,8 +8,7 @@ namespace marian {
 namespace bergamot {
 
 Service::Service(Ptr<Options> options)
-    : requestId_(0), batchNumber_(0),
-      numWorkers_(options->get<int>("cpu-threads")),
+    : requestId_(0), numWorkers_(options->get<int>("cpu-threads")),
       vocabs_(std::move(loadVocabularies(options))),
       text_processor_(vocabs_, options), batcher_(options),
       pcqueue_(2 * options->get<int>("cpu-threads")) {
@@ -58,25 +57,15 @@ std::future<TranslationResult> Service::translate(std::string &&input) {
       std::move(translationResultPromise));
 
   batcher_.addWholeRequest(request);
+
   if (numWorkers_ > 0) {
     batcher_.enqueue(pcqueue_);
   } else {
     // Queue single-threaded
-    int numSentences;
-    do {
-      RequestSentences batchSentences;
-      batcher_.cleaveBatch(batchSentences);
-      numSentences = batchSentences.size();
-
-      if (numSentences > 0) {
-        translator->translate(batchSentences);
-        batchNumber_++;
-      }
-
-      if (batchNumber_ % 500 == 0) {
-        LOG(info, "Tranlsating batch {}", batchNumber_);
-      }
-    } while (numSentences > 0);
+    Batch batch;
+    while (batcher_ >> batch) {
+      translator->translate(batch);
+    }
   }
 
   return future;
@@ -85,8 +74,8 @@ std::future<TranslationResult> Service::translate(std::string &&input) {
 void Service::stop() {
   int counter = 0;
   for (auto &worker : workers_) {
-    PCItem pcitem;
-    pcqueue_.ProduceSwap(pcitem);
+    Batch batch;
+    pcqueue_.ProduceSwap(batch);
     ++counter;
   }
 
diff --git a/src/translator/service.h b/src/translator/service.h
index 951398df5..c57e609a7 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -46,7 +46,6 @@ class Service {
 
 private:
   unsigned int requestId_;
-  unsigned int batchNumber_;
   int numWorkers_;
 
   // vocabs are used to construct a Request, which later uses it to construct
@@ -68,7 +67,7 @@ class Service {
 
   TextProcessor text_processor_; // ORDER DEPENDENCY
   Batcher batcher_;
-  PCQueue<PCItem> pcqueue_;
+  PCQueue<Batch> pcqueue_;
   std::vector<std::thread> workers_;
 
   // Optional

From e585a9e7861934e40d3d4e2a5793724be3a9e3a6 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sat, 13 Feb 2021 16:31:30 +0000
Subject: [PATCH 076/442] Sanitizing Batch construction

Batch Ids cannot be set by outside classes to values < 0.

Batch.Id_ =
    -1 : Poison, for use in PCQueue
     0 : Default constructed, invalid batch.
    >0 : Legit batch.

Book-keeping for batch metrics (maxLength, numTokens, etc) and logging
are now moved to Batch. Batch is now a class instead of a struct with
accessors controlling how members can be modified to suit above.
---
 src/translator/batch_translator.cpp |  7 ++--
 src/translator/batcher.cpp          | 23 ++++---------
 src/translator/request.h            | 53 ++++++++++++++++++++++-------
 src/translator/service.cpp          |  4 +--
 4 files changed, 53 insertions(+), 34 deletions(-)

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index a6e6b9347..13eb58a21 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -39,7 +39,8 @@ BatchTranslator::BatchTranslator(DeviceId const device,
 void BatchTranslator::translate(Batch &batch) {
   std::vector<data::SentenceTuple> batchVector;
 
-  for (auto &sentence : batch.sentences) {
+  auto &sentences = batch.sentences();
+  for (auto &sentence : sentences) {
     data::SentenceTuple sentence_tuple(sentence.lineNumber());
     Segment segment = sentence.getUnderlyingSegment();
     sentence_tuple.push_back(segment);
@@ -89,9 +90,7 @@ void BatchTranslator::translate(Batch &batch) {
   auto search = New<BeamSearch>(options_, scorers_, trgVocab);
 
   auto histories = std::move(search->search(graph_, corpus_batch));
-  for (int i = 0; i < batch.sentences.size(); i++) {
-    batch.sentences[i].completeSentence(histories[i]);
-  }
+  batch.completeBatch(histories);
 }
 
 void translation_loop(DeviceId const &device, PCQueue<Batch> &pcqueue,
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 13b563542..5fdcc3ac6 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -33,31 +33,22 @@ bool Batcher::cleaveBatch(Batch &batch) {
   for (int length = 0; length < bucket_.size(); length++) {
     auto p = bucket_[length].begin();
     while (p != bucket_[length].end()) {
-      paddedBatchSize = (batch.sentences.size() + 1) * length;
+      paddedBatchSize = (batch.size() + 1) * length;
       if (paddedBatchSize <= miniBatchWords) {
-        auto q = p;
-        ++p;
-
-        batch.numTokens += length;
-        batch.sentences.push_back(*q);
-        batch.maxLength = std::max(batch.maxLength, length);
-
+        auto q = p++;
+        batch.add(*q);
         bucket_[length].erase(q);
       } else {
         // Check if elements exist
-        assert(batch.sentences.size() > 0);
-        batch.Id = ++batchNumber_;
-        if (batchId % 500 == 0) {
-          batch.log();
-        }
+        assert(batch.size() > 0);
+        batch.setId(++batchNumber_);
         return true;
       }
     }
   }
 
-  if (batch.sentences.size()) {
-    batch.Id = ++batchNumber_;
-    batch.log();
+  if (batch.size()) {
+    batch.setId(++batchNumber_);
     return true;
   } else {
     return false;
diff --git a/src/translator/request.h b/src/translator/request.h
index 673f88ce3..5fb9c3c5d 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -28,6 +28,8 @@
 #include "data/types.h"
 #include "translator/beam_search.h"
 
+#include <cassert>
+
 #include <future>
 #include <vector>
 
@@ -93,23 +95,50 @@ class RequestSentence {
 
 typedef std::vector<RequestSentence> RequestSentences;
 
-struct Batch {
-  int Id;
-  int numTokens, maxLength;
-  RequestSentences sentences;
-
-  // Batch should be default constructible for PCQueue. Default constructed
-  // element is poison.
+class Batch {
+public:
   Batch() { reset(); }
-  void reset() { Id = -1, numTokens = 0, maxLength = 0, sentences.clear(); }
-
+  void reset() { Id_ = 0, numTokens_ = 0, maxLength_ = 0, sentences_.clear(); }
   // Convenience function to determine poison.
-  bool isPoison() { return (Id == -1); }
+  bool isPoison() { return (Id_ == -1); }
+  static Batch poison() {
+    Batch poison_;
+    poison_.Id_ = -1;
+    return poison_;
+  }
 
   void log() {
-    LOG(info, "Batch(Id={}, tokens={}, max-length={}, sentences={})", Id,
-        numTokens, maxLength, sentences.size());
+    LOG(info, "Batch(Id_={}, tokens={}, max-length={}, sentences_={})", Id_,
+        numTokens_, maxLength_, sentences_.size());
+  }
+
+  void add(const RequestSentence &sentence) {
+    sentences_.push_back(sentence);
+    maxLength_ = std::max(sentence.numTokens(), maxLength_);
+    numTokens_ += sentence.numTokens();
   }
+
+  size_t size() { return sentences_.size(); }
+
+  void setId(int Id) {
+    assert(Id > 0);
+    Id_ = Id;
+    if (Id % 500 == 0) {
+      log();
+    }
+  }
+
+  const RequestSentences &sentences() { return sentences_; }
+  void completeBatch(const Histories &histories) {
+    for (int i = 0; i < sentences_.size(); i++) {
+      sentences_[i].completeSentence(histories[i]);
+    }
+  }
+
+private:
+  int Id_;
+  size_t numTokens_, maxLength_;
+  RequestSentences sentences_;
 };
 
 } // namespace bergamot
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 37019552c..c93aa5f00 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -74,8 +74,8 @@ std::future<TranslationResult> Service::translate(std::string &&input) {
 void Service::stop() {
   int counter = 0;
   for (auto &worker : workers_) {
-    Batch batch;
-    pcqueue_.ProduceSwap(batch);
+    Batch poison = Batch::poison();
+    pcqueue_.ProduceSwap(poison);
     ++counter;
   }
 

From 1e413f71cd583bba570af8ed8fde7f797174dd41 Mon Sep 17 00:00:00 2001
From: Andre Natal <anatal@gmail.com>
Date: Sat, 13 Feb 2021 14:54:36 -0800
Subject: [PATCH 077/442] Including a more elaborated test page, a node
 webserver containing the proper cors headers and wasm mimetype

---
 .gitignore                       |   2 +
 README.md                        |   6 +-
 wasm/README.md                   |  42 +-
 wasm/bergamot.html               |  54 --
 wasm/test_page/bergamot.html     | 140 +++++
 wasm/test_page/helper.js         |  40 ++
 wasm/test_page/package-lock.json | 904 +++++++++++++++++++++++++++++++
 wasm/test_page/package.json      |   7 +
 wasm/test_page/start_server.sh   |   8 +
 9 files changed, 1131 insertions(+), 72 deletions(-)
 delete mode 100644 wasm/bergamot.html
 create mode 100644 wasm/test_page/bergamot.html
 create mode 100644 wasm/test_page/helper.js
 create mode 100644 wasm/test_page/package-lock.json
 create mode 100644 wasm/test_page/package.json
 create mode 100644 wasm/test_page/start_server.sh

diff --git a/.gitignore b/.gitignore
index e63aee1e1..59363a81c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,3 +2,5 @@
 *.swp
 *.swo
 
+wasm/test_page/node_modules
+build-wasm
diff --git a/README.md b/README.md
index 3e458dfe0..333e758e3 100644
--- a/README.md
+++ b/README.md
@@ -40,10 +40,12 @@ emmake make -j
 
 It should generate the artefacts (.js and .wasm files) in `wasm` folder inside build directory ("build-wasm" in this case).
 
+Download the models from `https://github.com/mozilla-applied-ml/bergamot-models`, and place all the desired ones to package in a folder called `models`.
+
 The build also allows packaging files into wasm binary (i.e. preloading in Emscripten’s virtual file system) using cmake
-option `PACKAGE_DIR`. The compile command below packages all the files in PATH directory into wasm binary.
+option `PACKAGE_DIR`. The compile command below packages all the files in PATH directory (in these case, your models) into wasm binary.
 ```bash
-emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=<PATH> ../
+emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=<PATH> ./models
 ```
 Files packaged this way are preloaded in the root of the virtual file system.
 
diff --git a/wasm/README.md b/wasm/README.md
index 83d4738cd..6be620956 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -1,5 +1,5 @@
 ## Using Bergamot Translator in JavaScript
-The example file `bergamot.html` in this folder demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
+The example file `bergamot.html` in the folder `test_page` demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
 This example assumes that files were packaged in wasm binary.
 
 A brief summary is here though:
@@ -33,20 +33,30 @@ request.delete();
 input.delete();
 ```
 
-You can also see everything in action by running `bergamot.html` file in the browser following these steps:
-* Copy bergamot.html to the directory where build artefacts of wasm compilation (i.e. .js and .wasm files) reside.
-* Start an http server locally
-* Open the link provided by the http server in any browser
-* Open the browser's console and you will see all the console messages there
-
-Assuming build artefacts are present in `$ROOT/build-wasm` folder where `ROOT` is repository's root.
-Above instructions would become:
-
-```bash
-cd $ROOT/build-wasm
-cp ../wasm/bergamot.html wasm/.
-python3 -m http.server -d wasm
+You can also see everything in action by following the next steps:
+* Start the test webserver (ensure you have the latest nodejs installed)
+```
+cd test_page
+bash start_server
+```
+* Open any of the browsers below
+    * Firefox Nightly +87: make sure the following prefs are on (about:config)
+    ````
+    javascript.options.wasm_simd = true
+    javascript.options.wasm_simd_wormhole = true
+    ````
+
+    * Chrome Canary +90: start with the following argument
+    ```
+    --js-flags="--experimental-wasm-simd"
+    ```
+
+* Browse to the following page:
+```
+http://localhost:8000/bergamot.html
 ```
 
-Assuming it starts the http server on 8000 port,
-open `http://0.0.0.0:8000/bergamot.html` in any browser and see the console logs in browser's console.
+* Run some translations:
+    * Choose a model and press `Load Model`
+    * Type a sentence to be translated in the `From` textbox and press `Translate`
+    * See the results in the `To` and `Log` textboxes
\ No newline at end of file
diff --git a/wasm/bergamot.html b/wasm/bergamot.html
deleted file mode 100644
index 2bbad046b..000000000
--- a/wasm/bergamot.html
+++ /dev/null
@@ -1,54 +0,0 @@
-<!doctype html>
-<html>
-  <script>
-    var Module = {
-      onRuntimeInitialized: function() {
-        // Set the Model Configuration as YAML formatted string.
-        // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-        // This example captures the most relevant options: model file, vocabulary files and shortlist file
-        var modelConfig = "{\"models\":[\"/model.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1}";//,\"shortlist\":[\"/lex.s2t\"]
-
-        // Instantiate the TranslationModel
-        var model = new Module.TranslationModel(modelConfig);
-        console.log('Alignment:', model.isAlignmentSupported());
-
-        // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
-        var request = new Module.TranslationRequest();
-        let input = new Module.VectorString;
-
-        // Initialize the input
-        input.push_back("¿Qué estás haciendo? ¿Por qué estás haciendo eso?");
-        input.push_back("Han pasado tres días de las elecciones presidenciales y los ecuatorianos siguen sin saber quién pasará a la segunda vuelta. El anuncio de Iván Duque del plan para regularizar al millón de migrantes venezolanos que están ilegales en Colombia ha generado esperanza a la vez que confusión.");
-
-        // Access input (just for debugging)
-        console.log('Input size=', input.size());
-        for (let i = 0; i < input.size(); i++) {
-          console.log(' val:' + input.get(i));
-        }
-
-        // Translate the input; the result is a vector<TranslationResult>
-        var result = model.translate(input, request);
-
-        // Access original and translated text from each entry of vector<TranslationResult>
-        console.log('Result size=', result.size());
-        for (let i = 0; i < result.size(); i++) {
-          console.log(' original=' + result.get(i).getOriginalText() + ', translation=' + result.get(i).getTranslatedText());
-        }
-
-        // Translate again more text
-        input.push_back("Luego de los cuestionamientos al gobierno chileno por la gestión de la pandemia, el país sudamericano lidera el ranking de vacunación en América Latina. Aquí te explicamos cómo lo ha logrado.");
-        result = model.translate(input, request);
-        console.log('Result size=', result.size());
-        for (let i = 0; i < result.size(); i++) {
-          console.log(' original=' + result.get(i).getOriginalText() + ', translation=' + result.get(i).getTranslatedText());
-        }
-
-        // Don't forget to clean up the instances
-        model.delete();
-        request.delete();
-        input.delete();
-      }
-    };
-  </script>
-  <script src="bergamot-translator-worker.js"></script>
-</html>
diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
new file mode 100644
index 000000000..49ca50e96
--- /dev/null
+++ b/wasm/test_page/bergamot.html
@@ -0,0 +1,140 @@
+<!doctype html>
+<html>
+  <head>
+    <link rel="icon" href="data:,">
+    <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1">
+  </head>
+  <style>
+    body, html, div {
+      margin-left:1%;
+      margin-right:1%;
+      margin-bottom:1%;
+      margin-top:1%;
+      padding-left:1%;
+      padding-right:1%;
+      padding-bottom:1%;
+      padding-top:1%;
+      }
+
+      textarea, #to, #from {
+        width: 100%;
+        max-width: 100%;
+      }
+
+      div {
+        float: left;
+        width: 80%;
+      }
+  </style>
+  <body>
+
+  <div id="divradios">
+    <label for="radios">Choose the model to use</label>
+    <input type="radio" id="modellang" name="modellang" value="enes" checked><label for="es">English to Spanish</label>
+    <input type="radio" id="modellang" name="modellang" value="esen"><label for="en">Spanish to English</label>
+    <input type="button" id="load" value="Load Model"/>
+  </div>
+
+  <div id="divtranslation">
+    <label for="from">From</label>
+    <input type="text" id="from" name="from"/>
+    <br><br>
+    <label for="to">To</label>
+    <input type="text" id="to" name="to" readonly/>
+    <br><br>
+    <input type="button" id="translate" value="Translate"/>
+  </div>
+
+  <div id="divlog">
+    <label for="log">Log:</label><br>
+    <textarea id="log" name="log" rows="50" cols="75"></textarea>
+  </div>
+
+  <script>
+
+    var model, request, input = undefined;
+    const loadModel = (lang) => {
+        // Set the Model Configuration as YAML formatted string.
+        // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
+        // This example captures the most relevant options: model file, vocabulary files and shortlist file
+        //        var modelConfig = "{\"models\":[\"/model.enes.npz\"],\"vocabs\":[\"/vocab.esen.spm\"],\"beam-size\":1}";//,\"shortlist\":[\"/lex.s2t\"]
+        const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1} ,\"shortlist\":[\"/lex.s2t\"]`;
+
+        // Instantiate the TranslationModel
+        if (model) model.delete();
+        model = new Module.TranslationModel(modelConfig);
+    }
+
+    const translate = (sentence) => {
+
+        // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
+        var request = new Module.TranslationRequest();
+        let input = new Module.VectorString;
+
+        // Initialize the input
+        input.push_back(sentence);
+        /*
+        // Access input (just for debugging)
+        console.log('Input size=', input.size());
+        for (let i = 0; i < input.size(); i++) {
+          console.log(' val:' + input.get(i));
+        }
+        */
+
+        // Translate the input; the result is a vector<TranslationResult>
+        let result = model.translate(input, request);
+        // Access original and translated text from each entry of vector<TranslationResult>
+        //console.log('Result size=', result.size(), ' - TimeDiff - ', (Date.now() - start)/1000);
+        let translatedText = "";
+        for (let i = 0; i < result.size(); i++) {
+          translatedText += result.get(i).getTranslatedText() + " ";
+        }
+        console.log(translatedText);
+        request.delete();
+        input.delete();
+        return translatedText;
+  }
+
+    document.querySelector("#load").addEventListener("click", () => {
+      const lang = document.querySelector('input[name="modellang"]:checked').value;
+      let start = Date.now();
+      loadModel(lang)
+      log(`model ${lang} loaded in ${(Date.now() - start)/1000} secs`);
+      //log('Model Alignment:', model.isAlignmentSupported());
+    });
+
+    const translateCall = () => {
+      const text = document.querySelector('#from').value;
+      let start = Date.now();
+      const translate_text = translate(text);
+      log(`sentence translation time ${(Date.now() - start)/1000} secs`);
+      document.querySelector('#to').value = translate_text;
+    }
+
+    document.querySelector("#translate").addEventListener("click", () => {
+      translateCall();
+    });
+
+    document.querySelector("#from").addEventListener('keyup', function (event) {
+      if (event.keyCode === 13) {
+        translateCall();
+      }
+   });
+
+    const log = (message) => {
+      document.querySelector("#log").value += message + "\n";
+    }
+
+    let moduleLoadStart;
+    var Module = {
+      preRun: [function() {
+        moduleLoadStart = Date.now();
+      }],
+      onRuntimeInitialized: function() {
+        log(`Wasm Runtime initialized in ${(Date.now() - moduleLoadStart)/1000} secs`);
+      }
+    };
+  </script>
+  <script src="bergamot-translator-worker.js"></script>
+</body>
+</html>
diff --git a/wasm/test_page/helper.js b/wasm/test_page/helper.js
new file mode 100644
index 000000000..bff116ced
--- /dev/null
+++ b/wasm/test_page/helper.js
@@ -0,0 +1,40 @@
+/*
+ * @author - Based of a file from Gist here: https://gist.github.com/1757658
+ *
+ * @modified - Mike Newell - it was on Gist so I figure I can use it
+ *
+ * @Description -   Added support for a few more mime types including the new
+ *                  .ogv, .webm, and .mp4 file types for HTML5 video.
+ *
+ */
+
+/*
+* @modified - Andre Natal - removed unused types for the purpose of this use
+case
+*/
+
+Helper = {
+
+    types: {
+       "wasm" : "application/wasm"
+       , "js" : "application/javascript"
+       , "html" : "text/html"
+       , "htm" : "text/html"
+       , "ico" : "image/vnd.microsoft.icon",
+    },
+
+    getMime: function(u) {
+
+        var ext = this.getExt(u.pathname).replace('.', '');
+
+        return this.types[ext.toLowerCase()] || 'application/octet-stream';
+
+    },
+
+    getExt: function(path) {
+        var i = path.lastIndexOf('.');
+
+        return (i < 0) ? '' : path.substr(i);
+    }
+
+};
diff --git a/wasm/test_page/package-lock.json b/wasm/test_page/package-lock.json
new file mode 100644
index 000000000..065c92de8
--- /dev/null
+++ b/wasm/test_page/package-lock.json
@@ -0,0 +1,904 @@
+{
+  "name": "test_page",
+  "lockfileVersion": 2,
+  "requires": true,
+  "packages": {
+    "": {
+      "dependencies": {
+        "cors": "^2.8.5",
+        "express": "^4.17.1",
+        "nocache": "^2.1.0"
+      }
+    },
+    "node_modules/accepts": {
+      "version": "1.3.7",
+      "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.7.tgz",
+      "integrity": "sha512-Il80Qs2WjYlJIBNzNkK6KYqlVMTbZLXgHx2oT0pU/fjRHyEp+PEfEPY0R3WCwAGVOtauxh1hOxNgIf5bv7dQpA==",
+      "dependencies": {
+        "mime-types": "~2.1.24",
+        "negotiator": "0.6.2"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/array-flatten": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz",
+      "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI="
+    },
+    "node_modules/body-parser": {
+      "version": "1.19.0",
+      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.19.0.tgz",
+      "integrity": "sha512-dhEPs72UPbDnAQJ9ZKMNTP6ptJaionhP5cBb541nXPlW60Jepo9RV/a4fX4XWW9CuFNK22krhrj1+rgzifNCsw==",
+      "dependencies": {
+        "bytes": "3.1.0",
+        "content-type": "~1.0.4",
+        "debug": "2.6.9",
+        "depd": "~1.1.2",
+        "http-errors": "1.7.2",
+        "iconv-lite": "0.4.24",
+        "on-finished": "~2.3.0",
+        "qs": "6.7.0",
+        "raw-body": "2.4.0",
+        "type-is": "~1.6.17"
+      },
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/bytes": {
+      "version": "3.1.0",
+      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.0.tgz",
+      "integrity": "sha512-zauLjrfCG+xvoyaqLoV8bLVXXNGC4JqlxFCutSDWA6fJrTo2ZuvLYTqZ7aHBLZSMOopbzwv8f+wZcVzfVTI2Dg==",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/content-disposition": {
+      "version": "0.5.3",
+      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.3.tgz",
+      "integrity": "sha512-ExO0774ikEObIAEV9kDo50o+79VCUdEB6n6lzKgGwupcVeRlhrj3qGAfwq8G6uBJjkqLrhT0qEYFcWng8z1z0g==",
+      "dependencies": {
+        "safe-buffer": "5.1.2"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/content-type": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz",
+      "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/cookie": {
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.4.0.tgz",
+      "integrity": "sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/cookie-signature": {
+      "version": "1.0.6",
+      "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.6.tgz",
+      "integrity": "sha1-4wOogrNCzD7oylE6eZmXNNqzriw="
+    },
+    "node_modules/cors": {
+      "version": "2.8.5",
+      "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.5.tgz",
+      "integrity": "sha512-KIHbLJqu73RGr/hnbrO9uBeixNGuvSQjul/jdFvS/KFSIH1hWVd1ng7zOHx+YrEfInLG7q4n6GHQ9cDtxv/P6g==",
+      "dependencies": {
+        "object-assign": "^4",
+        "vary": "^1"
+      },
+      "engines": {
+        "node": ">= 0.10"
+      }
+    },
+    "node_modules/debug": {
+      "version": "2.6.9",
+      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
+      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
+      "dependencies": {
+        "ms": "2.0.0"
+      }
+    },
+    "node_modules/depd": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/depd/-/depd-1.1.2.tgz",
+      "integrity": "sha1-m81S4UwJd2PnSbJ0xDRu0uVgtak=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/destroy": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.0.4.tgz",
+      "integrity": "sha1-l4hXRCxEdJ5CBmE+N5RiBYJqvYA="
+    },
+    "node_modules/ee-first": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
+      "integrity": "sha1-WQxhFWsK4vTwJVcyoViyZrxWsh0="
+    },
+    "node_modules/encodeurl": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
+      "integrity": "sha1-rT/0yG7C0CkyL1oCw6mmBslbP1k=",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/escape-html": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
+      "integrity": "sha1-Aljq5NPQwJdN4cFpGI7wBR0dGYg="
+    },
+    "node_modules/etag": {
+      "version": "1.8.1",
+      "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
+      "integrity": "sha1-Qa4u62XvpiJorr/qg6x9eSmbCIc=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/express": {
+      "version": "4.17.1",
+      "resolved": "https://registry.npmjs.org/express/-/express-4.17.1.tgz",
+      "integrity": "sha512-mHJ9O79RqluphRrcw2X/GTh3k9tVv8YcoyY4Kkh4WDMUYKRZUq0h1o0w2rrrxBqM7VoeUVqgb27xlEMXTnYt4g==",
+      "dependencies": {
+        "accepts": "~1.3.7",
+        "array-flatten": "1.1.1",
+        "body-parser": "1.19.0",
+        "content-disposition": "0.5.3",
+        "content-type": "~1.0.4",
+        "cookie": "0.4.0",
+        "cookie-signature": "1.0.6",
+        "debug": "2.6.9",
+        "depd": "~1.1.2",
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "etag": "~1.8.1",
+        "finalhandler": "~1.1.2",
+        "fresh": "0.5.2",
+        "merge-descriptors": "1.0.1",
+        "methods": "~1.1.2",
+        "on-finished": "~2.3.0",
+        "parseurl": "~1.3.3",
+        "path-to-regexp": "0.1.7",
+        "proxy-addr": "~2.0.5",
+        "qs": "6.7.0",
+        "range-parser": "~1.2.1",
+        "safe-buffer": "5.1.2",
+        "send": "0.17.1",
+        "serve-static": "1.14.1",
+        "setprototypeof": "1.1.1",
+        "statuses": "~1.5.0",
+        "type-is": "~1.6.18",
+        "utils-merge": "1.0.1",
+        "vary": "~1.1.2"
+      },
+      "engines": {
+        "node": ">= 0.10.0"
+      }
+    },
+    "node_modules/finalhandler": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.2.tgz",
+      "integrity": "sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA==",
+      "dependencies": {
+        "debug": "2.6.9",
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "on-finished": "~2.3.0",
+        "parseurl": "~1.3.3",
+        "statuses": "~1.5.0",
+        "unpipe": "~1.0.0"
+      },
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/forwarded": {
+      "version": "0.1.2",
+      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.1.2.tgz",
+      "integrity": "sha1-mMI9qxF1ZXuMBXPozszZGw/xjIQ=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/fresh": {
+      "version": "0.5.2",
+      "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz",
+      "integrity": "sha1-PYyt2Q2XZWn6g1qx+OSyOhBWBac=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/http-errors": {
+      "version": "1.7.2",
+      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.7.2.tgz",
+      "integrity": "sha512-uUQBt3H/cSIVfch6i1EuPNy/YsRSOUBXTVfZ+yR7Zjez3qjBz6i9+i4zjNaoqcoFVI4lQJ5plg63TvGfRSDCRg==",
+      "dependencies": {
+        "depd": "~1.1.2",
+        "inherits": "2.0.3",
+        "setprototypeof": "1.1.1",
+        "statuses": ">= 1.5.0 < 2",
+        "toidentifier": "1.0.0"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/iconv-lite": {
+      "version": "0.4.24",
+      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
+      "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
+      "dependencies": {
+        "safer-buffer": ">= 2.1.2 < 3"
+      },
+      "engines": {
+        "node": ">=0.10.0"
+      }
+    },
+    "node_modules/inherits": {
+      "version": "2.0.3",
+      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz",
+      "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4="
+    },
+    "node_modules/ipaddr.js": {
+      "version": "1.9.1",
+      "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
+      "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",
+      "engines": {
+        "node": ">= 0.10"
+      }
+    },
+    "node_modules/media-typer": {
+      "version": "0.3.0",
+      "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
+      "integrity": "sha1-hxDXrwqmJvj/+hzgAWhUUmMlV0g=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/merge-descriptors": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz",
+      "integrity": "sha1-sAqqVW3YtEVoFQ7J0blT8/kMu2E="
+    },
+    "node_modules/methods": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz",
+      "integrity": "sha1-VSmk1nZUE07cxSZmVoNbD4Ua/O4=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/mime": {
+      "version": "1.6.0",
+      "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
+      "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==",
+      "bin": {
+        "mime": "cli.js"
+      },
+      "engines": {
+        "node": ">=4"
+      }
+    },
+    "node_modules/mime-db": {
+      "version": "1.45.0",
+      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.45.0.tgz",
+      "integrity": "sha512-CkqLUxUk15hofLoLyljJSrukZi8mAtgd+yE5uO4tqRZsdsAJKv0O+rFMhVDRJgozy+yG6md5KwuXhD4ocIoP+w==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/mime-types": {
+      "version": "2.1.28",
+      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.28.tgz",
+      "integrity": "sha512-0TO2yJ5YHYr7M2zzT7gDU1tbwHxEUWBCLt0lscSNpcdAfFyJOVEpRYNS7EXVcTLNj/25QO8gulHC5JtTzSE2UQ==",
+      "dependencies": {
+        "mime-db": "1.45.0"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/ms": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
+      "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g="
+    },
+    "node_modules/negotiator": {
+      "version": "0.6.2",
+      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.2.tgz",
+      "integrity": "sha512-hZXc7K2e+PgeI1eDBe/10Ard4ekbfrrqG8Ep+8Jmf4JID2bNg7NvCPOZN+kfF574pFQI7mum2AUqDidoKqcTOw==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/nocache": {
+      "version": "2.1.0",
+      "resolved": "https://registry.npmjs.org/nocache/-/nocache-2.1.0.tgz",
+      "integrity": "sha512-0L9FvHG3nfnnmaEQPjT9xhfN4ISk0A8/2j4M37Np4mcDesJjHgEUfgPhdCyZuFI954tjokaIj/A3NdpFNdEh4Q==",
+      "engines": {
+        "node": ">=4.0.0"
+      }
+    },
+    "node_modules/object-assign": {
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
+      "integrity": "sha1-IQmtx5ZYh8/AXLvUQsrIv7s2CGM=",
+      "engines": {
+        "node": ">=0.10.0"
+      }
+    },
+    "node_modules/on-finished": {
+      "version": "2.3.0",
+      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.3.0.tgz",
+      "integrity": "sha1-IPEzZIGwg811M3mSoWlxqi2QaUc=",
+      "dependencies": {
+        "ee-first": "1.1.1"
+      },
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/parseurl": {
+      "version": "1.3.3",
+      "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
+      "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/path-to-regexp": {
+      "version": "0.1.7",
+      "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz",
+      "integrity": "sha1-32BBeABfUi8V60SQ5yR6G/qmf4w="
+    },
+    "node_modules/proxy-addr": {
+      "version": "2.0.6",
+      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.6.tgz",
+      "integrity": "sha512-dh/frvCBVmSsDYzw6n926jv974gddhkFPfiN8hPOi30Wax25QZyZEGveluCgliBnqmuM+UJmBErbAUFIoDbjOw==",
+      "dependencies": {
+        "forwarded": "~0.1.2",
+        "ipaddr.js": "1.9.1"
+      },
+      "engines": {
+        "node": ">= 0.10"
+      }
+    },
+    "node_modules/qs": {
+      "version": "6.7.0",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.7.0.tgz",
+      "integrity": "sha512-VCdBRNFTX1fyE7Nb6FYoURo/SPe62QCaAyzJvUjwRaIsc+NePBEniHlvxFmmX56+HZphIGtV0XeCirBtpDrTyQ==",
+      "engines": {
+        "node": ">=0.6"
+      }
+    },
+    "node_modules/range-parser": {
+      "version": "1.2.1",
+      "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
+      "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/raw-body": {
+      "version": "2.4.0",
+      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.4.0.tgz",
+      "integrity": "sha512-4Oz8DUIwdvoa5qMJelxipzi/iJIi40O5cGV1wNYp5hvZP8ZN0T+jiNkL0QepXs+EsQ9XJ8ipEDoiH70ySUJP3Q==",
+      "dependencies": {
+        "bytes": "3.1.0",
+        "http-errors": "1.7.2",
+        "iconv-lite": "0.4.24",
+        "unpipe": "1.0.0"
+      },
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/safe-buffer": {
+      "version": "5.1.2",
+      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
+      "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="
+    },
+    "node_modules/safer-buffer": {
+      "version": "2.1.2",
+      "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
+      "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
+    },
+    "node_modules/send": {
+      "version": "0.17.1",
+      "resolved": "https://registry.npmjs.org/send/-/send-0.17.1.tgz",
+      "integrity": "sha512-BsVKsiGcQMFwT8UxypobUKyv7irCNRHk1T0G680vk88yf6LBByGcZJOTJCrTP2xVN6yI+XjPJcNuE3V4fT9sAg==",
+      "dependencies": {
+        "debug": "2.6.9",
+        "depd": "~1.1.2",
+        "destroy": "~1.0.4",
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "etag": "~1.8.1",
+        "fresh": "0.5.2",
+        "http-errors": "~1.7.2",
+        "mime": "1.6.0",
+        "ms": "2.1.1",
+        "on-finished": "~2.3.0",
+        "range-parser": "~1.2.1",
+        "statuses": "~1.5.0"
+      },
+      "engines": {
+        "node": ">= 0.8.0"
+      }
+    },
+    "node_modules/send/node_modules/ms": {
+      "version": "2.1.1",
+      "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz",
+      "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg=="
+    },
+    "node_modules/serve-static": {
+      "version": "1.14.1",
+      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.14.1.tgz",
+      "integrity": "sha512-JMrvUwE54emCYWlTI+hGrGv5I8dEwmco/00EvkzIIsR7MqrHonbD9pO2MOfFnpFntl7ecpZs+3mW+XbQZu9QCg==",
+      "dependencies": {
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "parseurl": "~1.3.3",
+        "send": "0.17.1"
+      },
+      "engines": {
+        "node": ">= 0.8.0"
+      }
+    },
+    "node_modules/setprototypeof": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.1.tgz",
+      "integrity": "sha512-JvdAWfbXeIGaZ9cILp38HntZSFSo3mWg6xGcJJsd+d4aRMOqauag1C63dJfDw7OaMYwEbHMOxEZ1lqVRYP2OAw=="
+    },
+    "node_modules/statuses": {
+      "version": "1.5.0",
+      "resolved": "https://registry.npmjs.org/statuses/-/statuses-1.5.0.tgz",
+      "integrity": "sha1-Fhx9rBd2Wf2YEfQ3cfqZOBR4Yow=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/toidentifier": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.0.tgz",
+      "integrity": "sha512-yaOH/Pk/VEhBWWTlhI+qXxDFXlejDGcQipMlyxda9nthulaxLZUNcUqFxokp0vcYnvteJln5FNQDRrxj3YcbVw==",
+      "engines": {
+        "node": ">=0.6"
+      }
+    },
+    "node_modules/type-is": {
+      "version": "1.6.18",
+      "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
+      "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",
+      "dependencies": {
+        "media-typer": "0.3.0",
+        "mime-types": "~2.1.24"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/unpipe": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
+      "integrity": "sha1-sr9O6FFKrmFltIF4KdIbLvSZBOw=",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/utils-merge": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",
+      "integrity": "sha1-n5VxD1CiZ5R7LMwSR0HBAoQn5xM=",
+      "engines": {
+        "node": ">= 0.4.0"
+      }
+    },
+    "node_modules/vary": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
+      "integrity": "sha1-IpnwLG3tMNSllhsLn3RSShj2NPw=",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    }
+  },
+  "dependencies": {
+    "accepts": {
+      "version": "1.3.7",
+      "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.7.tgz",
+      "integrity": "sha512-Il80Qs2WjYlJIBNzNkK6KYqlVMTbZLXgHx2oT0pU/fjRHyEp+PEfEPY0R3WCwAGVOtauxh1hOxNgIf5bv7dQpA==",
+      "requires": {
+        "mime-types": "~2.1.24",
+        "negotiator": "0.6.2"
+      }
+    },
+    "array-flatten": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz",
+      "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI="
+    },
+    "body-parser": {
+      "version": "1.19.0",
+      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.19.0.tgz",
+      "integrity": "sha512-dhEPs72UPbDnAQJ9ZKMNTP6ptJaionhP5cBb541nXPlW60Jepo9RV/a4fX4XWW9CuFNK22krhrj1+rgzifNCsw==",
+      "requires": {
+        "bytes": "3.1.0",
+        "content-type": "~1.0.4",
+        "debug": "2.6.9",
+        "depd": "~1.1.2",
+        "http-errors": "1.7.2",
+        "iconv-lite": "0.4.24",
+        "on-finished": "~2.3.0",
+        "qs": "6.7.0",
+        "raw-body": "2.4.0",
+        "type-is": "~1.6.17"
+      }
+    },
+    "bytes": {
+      "version": "3.1.0",
+      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.0.tgz",
+      "integrity": "sha512-zauLjrfCG+xvoyaqLoV8bLVXXNGC4JqlxFCutSDWA6fJrTo2ZuvLYTqZ7aHBLZSMOopbzwv8f+wZcVzfVTI2Dg=="
+    },
+    "content-disposition": {
+      "version": "0.5.3",
+      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.3.tgz",
+      "integrity": "sha512-ExO0774ikEObIAEV9kDo50o+79VCUdEB6n6lzKgGwupcVeRlhrj3qGAfwq8G6uBJjkqLrhT0qEYFcWng8z1z0g==",
+      "requires": {
+        "safe-buffer": "5.1.2"
+      }
+    },
+    "content-type": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz",
+      "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA=="
+    },
+    "cookie": {
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.4.0.tgz",
+      "integrity": "sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg=="
+    },
+    "cookie-signature": {
+      "version": "1.0.6",
+      "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.6.tgz",
+      "integrity": "sha1-4wOogrNCzD7oylE6eZmXNNqzriw="
+    },
+    "cors": {
+      "version": "2.8.5",
+      "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.5.tgz",
+      "integrity": "sha512-KIHbLJqu73RGr/hnbrO9uBeixNGuvSQjul/jdFvS/KFSIH1hWVd1ng7zOHx+YrEfInLG7q4n6GHQ9cDtxv/P6g==",
+      "requires": {
+        "object-assign": "^4",
+        "vary": "^1"
+      }
+    },
+    "debug": {
+      "version": "2.6.9",
+      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
+      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
+      "requires": {
+        "ms": "2.0.0"
+      }
+    },
+    "depd": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/depd/-/depd-1.1.2.tgz",
+      "integrity": "sha1-m81S4UwJd2PnSbJ0xDRu0uVgtak="
+    },
+    "destroy": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.0.4.tgz",
+      "integrity": "sha1-l4hXRCxEdJ5CBmE+N5RiBYJqvYA="
+    },
+    "ee-first": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
+      "integrity": "sha1-WQxhFWsK4vTwJVcyoViyZrxWsh0="
+    },
+    "encodeurl": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
+      "integrity": "sha1-rT/0yG7C0CkyL1oCw6mmBslbP1k="
+    },
+    "escape-html": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
+      "integrity": "sha1-Aljq5NPQwJdN4cFpGI7wBR0dGYg="
+    },
+    "etag": {
+      "version": "1.8.1",
+      "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
+      "integrity": "sha1-Qa4u62XvpiJorr/qg6x9eSmbCIc="
+    },
+    "express": {
+      "version": "4.17.1",
+      "resolved": "https://registry.npmjs.org/express/-/express-4.17.1.tgz",
+      "integrity": "sha512-mHJ9O79RqluphRrcw2X/GTh3k9tVv8YcoyY4Kkh4WDMUYKRZUq0h1o0w2rrrxBqM7VoeUVqgb27xlEMXTnYt4g==",
+      "requires": {
+        "accepts": "~1.3.7",
+        "array-flatten": "1.1.1",
+        "body-parser": "1.19.0",
+        "content-disposition": "0.5.3",
+        "content-type": "~1.0.4",
+        "cookie": "0.4.0",
+        "cookie-signature": "1.0.6",
+        "debug": "2.6.9",
+        "depd": "~1.1.2",
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "etag": "~1.8.1",
+        "finalhandler": "~1.1.2",
+        "fresh": "0.5.2",
+        "merge-descriptors": "1.0.1",
+        "methods": "~1.1.2",
+        "on-finished": "~2.3.0",
+        "parseurl": "~1.3.3",
+        "path-to-regexp": "0.1.7",
+        "proxy-addr": "~2.0.5",
+        "qs": "6.7.0",
+        "range-parser": "~1.2.1",
+        "safe-buffer": "5.1.2",
+        "send": "0.17.1",
+        "serve-static": "1.14.1",
+        "setprototypeof": "1.1.1",
+        "statuses": "~1.5.0",
+        "type-is": "~1.6.18",
+        "utils-merge": "1.0.1",
+        "vary": "~1.1.2"
+      }
+    },
+    "finalhandler": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.2.tgz",
+      "integrity": "sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA==",
+      "requires": {
+        "debug": "2.6.9",
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "on-finished": "~2.3.0",
+        "parseurl": "~1.3.3",
+        "statuses": "~1.5.0",
+        "unpipe": "~1.0.0"
+      }
+    },
+    "forwarded": {
+      "version": "0.1.2",
+      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.1.2.tgz",
+      "integrity": "sha1-mMI9qxF1ZXuMBXPozszZGw/xjIQ="
+    },
+    "fresh": {
+      "version": "0.5.2",
+      "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz",
+      "integrity": "sha1-PYyt2Q2XZWn6g1qx+OSyOhBWBac="
+    },
+    "http-errors": {
+      "version": "1.7.2",
+      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.7.2.tgz",
+      "integrity": "sha512-uUQBt3H/cSIVfch6i1EuPNy/YsRSOUBXTVfZ+yR7Zjez3qjBz6i9+i4zjNaoqcoFVI4lQJ5plg63TvGfRSDCRg==",
+      "requires": {
+        "depd": "~1.1.2",
+        "inherits": "2.0.3",
+        "setprototypeof": "1.1.1",
+        "statuses": ">= 1.5.0 < 2",
+        "toidentifier": "1.0.0"
+      }
+    },
+    "iconv-lite": {
+      "version": "0.4.24",
+      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
+      "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
+      "requires": {
+        "safer-buffer": ">= 2.1.2 < 3"
+      }
+    },
+    "inherits": {
+      "version": "2.0.3",
+      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz",
+      "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4="
+    },
+    "ipaddr.js": {
+      "version": "1.9.1",
+      "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
+      "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="
+    },
+    "media-typer": {
+      "version": "0.3.0",
+      "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
+      "integrity": "sha1-hxDXrwqmJvj/+hzgAWhUUmMlV0g="
+    },
+    "merge-descriptors": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz",
+      "integrity": "sha1-sAqqVW3YtEVoFQ7J0blT8/kMu2E="
+    },
+    "methods": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz",
+      "integrity": "sha1-VSmk1nZUE07cxSZmVoNbD4Ua/O4="
+    },
+    "mime": {
+      "version": "1.6.0",
+      "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
+      "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg=="
+    },
+    "mime-db": {
+      "version": "1.45.0",
+      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.45.0.tgz",
+      "integrity": "sha512-CkqLUxUk15hofLoLyljJSrukZi8mAtgd+yE5uO4tqRZsdsAJKv0O+rFMhVDRJgozy+yG6md5KwuXhD4ocIoP+w=="
+    },
+    "mime-types": {
+      "version": "2.1.28",
+      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.28.tgz",
+      "integrity": "sha512-0TO2yJ5YHYr7M2zzT7gDU1tbwHxEUWBCLt0lscSNpcdAfFyJOVEpRYNS7EXVcTLNj/25QO8gulHC5JtTzSE2UQ==",
+      "requires": {
+        "mime-db": "1.45.0"
+      }
+    },
+    "ms": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
+      "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g="
+    },
+    "negotiator": {
+      "version": "0.6.2",
+      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.2.tgz",
+      "integrity": "sha512-hZXc7K2e+PgeI1eDBe/10Ard4ekbfrrqG8Ep+8Jmf4JID2bNg7NvCPOZN+kfF574pFQI7mum2AUqDidoKqcTOw=="
+    },
+    "nocache": {
+      "version": "2.1.0",
+      "resolved": "https://registry.npmjs.org/nocache/-/nocache-2.1.0.tgz",
+      "integrity": "sha512-0L9FvHG3nfnnmaEQPjT9xhfN4ISk0A8/2j4M37Np4mcDesJjHgEUfgPhdCyZuFI954tjokaIj/A3NdpFNdEh4Q=="
+    },
+    "object-assign": {
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
+      "integrity": "sha1-IQmtx5ZYh8/AXLvUQsrIv7s2CGM="
+    },
+    "on-finished": {
+      "version": "2.3.0",
+      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.3.0.tgz",
+      "integrity": "sha1-IPEzZIGwg811M3mSoWlxqi2QaUc=",
+      "requires": {
+        "ee-first": "1.1.1"
+      }
+    },
+    "parseurl": {
+      "version": "1.3.3",
+      "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
+      "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ=="
+    },
+    "path-to-regexp": {
+      "version": "0.1.7",
+      "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz",
+      "integrity": "sha1-32BBeABfUi8V60SQ5yR6G/qmf4w="
+    },
+    "proxy-addr": {
+      "version": "2.0.6",
+      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.6.tgz",
+      "integrity": "sha512-dh/frvCBVmSsDYzw6n926jv974gddhkFPfiN8hPOi30Wax25QZyZEGveluCgliBnqmuM+UJmBErbAUFIoDbjOw==",
+      "requires": {
+        "forwarded": "~0.1.2",
+        "ipaddr.js": "1.9.1"
+      }
+    },
+    "qs": {
+      "version": "6.7.0",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.7.0.tgz",
+      "integrity": "sha512-VCdBRNFTX1fyE7Nb6FYoURo/SPe62QCaAyzJvUjwRaIsc+NePBEniHlvxFmmX56+HZphIGtV0XeCirBtpDrTyQ=="
+    },
+    "range-parser": {
+      "version": "1.2.1",
+      "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
+      "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="
+    },
+    "raw-body": {
+      "version": "2.4.0",
+      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.4.0.tgz",
+      "integrity": "sha512-4Oz8DUIwdvoa5qMJelxipzi/iJIi40O5cGV1wNYp5hvZP8ZN0T+jiNkL0QepXs+EsQ9XJ8ipEDoiH70ySUJP3Q==",
+      "requires": {
+        "bytes": "3.1.0",
+        "http-errors": "1.7.2",
+        "iconv-lite": "0.4.24",
+        "unpipe": "1.0.0"
+      }
+    },
+    "safe-buffer": {
+      "version": "5.1.2",
+      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
+      "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="
+    },
+    "safer-buffer": {
+      "version": "2.1.2",
+      "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
+      "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
+    },
+    "send": {
+      "version": "0.17.1",
+      "resolved": "https://registry.npmjs.org/send/-/send-0.17.1.tgz",
+      "integrity": "sha512-BsVKsiGcQMFwT8UxypobUKyv7irCNRHk1T0G680vk88yf6LBByGcZJOTJCrTP2xVN6yI+XjPJcNuE3V4fT9sAg==",
+      "requires": {
+        "debug": "2.6.9",
+        "depd": "~1.1.2",
+        "destroy": "~1.0.4",
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "etag": "~1.8.1",
+        "fresh": "0.5.2",
+        "http-errors": "~1.7.2",
+        "mime": "1.6.0",
+        "ms": "2.1.1",
+        "on-finished": "~2.3.0",
+        "range-parser": "~1.2.1",
+        "statuses": "~1.5.0"
+      },
+      "dependencies": {
+        "ms": {
+          "version": "2.1.1",
+          "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz",
+          "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg=="
+        }
+      }
+    },
+    "serve-static": {
+      "version": "1.14.1",
+      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.14.1.tgz",
+      "integrity": "sha512-JMrvUwE54emCYWlTI+hGrGv5I8dEwmco/00EvkzIIsR7MqrHonbD9pO2MOfFnpFntl7ecpZs+3mW+XbQZu9QCg==",
+      "requires": {
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "parseurl": "~1.3.3",
+        "send": "0.17.1"
+      }
+    },
+    "setprototypeof": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.1.tgz",
+      "integrity": "sha512-JvdAWfbXeIGaZ9cILp38HntZSFSo3mWg6xGcJJsd+d4aRMOqauag1C63dJfDw7OaMYwEbHMOxEZ1lqVRYP2OAw=="
+    },
+    "statuses": {
+      "version": "1.5.0",
+      "resolved": "https://registry.npmjs.org/statuses/-/statuses-1.5.0.tgz",
+      "integrity": "sha1-Fhx9rBd2Wf2YEfQ3cfqZOBR4Yow="
+    },
+    "toidentifier": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.0.tgz",
+      "integrity": "sha512-yaOH/Pk/VEhBWWTlhI+qXxDFXlejDGcQipMlyxda9nthulaxLZUNcUqFxokp0vcYnvteJln5FNQDRrxj3YcbVw=="
+    },
+    "type-is": {
+      "version": "1.6.18",
+      "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
+      "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",
+      "requires": {
+        "media-typer": "0.3.0",
+        "mime-types": "~2.1.24"
+      }
+    },
+    "unpipe": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
+      "integrity": "sha1-sr9O6FFKrmFltIF4KdIbLvSZBOw="
+    },
+    "utils-merge": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",
+      "integrity": "sha1-n5VxD1CiZ5R7LMwSR0HBAoQn5xM="
+    },
+    "vary": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
+      "integrity": "sha1-IpnwLG3tMNSllhsLn3RSShj2NPw="
+    }
+  }
+}
diff --git a/wasm/test_page/package.json b/wasm/test_page/package.json
new file mode 100644
index 000000000..20af6d2ab
--- /dev/null
+++ b/wasm/test_page/package.json
@@ -0,0 +1,7 @@
+{
+  "dependencies": {
+    "cors": "^2.8.5",
+    "express": "^4.17.1",
+    "nocache": "^2.1.0"
+  }
+}
diff --git a/wasm/test_page/start_server.sh b/wasm/test_page/start_server.sh
new file mode 100644
index 000000000..b83344b8a
--- /dev/null
+++ b/wasm/test_page/start_server.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+cp ../../build-wasm/wasm/bergamot-translator-worker.data .
+cp ../../build-wasm/wasm/bergamot-translator-worker.js .
+cp ../../build-wasm/wasm/bergamot-translator-worker.wasm .
+cp ../../build-wasm/wasm/bergamot-translator-worker.worker.js .
+npm install
+node bergamot-httpserver.js
\ No newline at end of file

From 47323d21b93795e19d82a499bfb13b71f7032c40 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sun, 14 Feb 2021 13:05:05 +0000
Subject: [PATCH 078/442] Getting rid of unused variables in Batch

---
 src/translator/request.h | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/src/translator/request.h b/src/translator/request.h
index 5fb9c3c5d..eab0d4b2a 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -98,7 +98,10 @@ typedef std::vector<RequestSentence> RequestSentences;
 class Batch {
 public:
   Batch() { reset(); }
-  void reset() { Id_ = 0, numTokens_ = 0, maxLength_ = 0, sentences_.clear(); }
+  void reset() {
+    Id_ = 0;
+    sentences_.clear();
+  }
   // Convenience function to determine poison.
   bool isPoison() { return (Id_ == -1); }
   static Batch poison() {
@@ -108,15 +111,17 @@ class Batch {
   }
 
   void log() {
+    int numTokens{0}, maxLength{0};
+    for (auto &sentence : sentences_) {
+      numTokens += sentence.numTokens();
+      maxLength = std::max(maxLength, static_cast<int>(sentence.numTokens()));
+    }
+
     LOG(info, "Batch(Id_={}, tokens={}, max-length={}, sentences_={})", Id_,
-        numTokens_, maxLength_, sentences_.size());
+        numTokens, maxLength, sentences_.size());
   }
 
-  void add(const RequestSentence &sentence) {
-    sentences_.push_back(sentence);
-    maxLength_ = std::max(sentence.numTokens(), maxLength_);
-    numTokens_ += sentence.numTokens();
-  }
+  void add(const RequestSentence &sentence) { sentences_.push_back(sentence); }
 
   size_t size() { return sentences_.size(); }
 
@@ -137,7 +142,6 @@ class Batch {
 
 private:
   int Id_;
-  size_t numTokens_, maxLength_;
   RequestSentences sentences_;
 };
 

From ecc91c51e3b439b32173e3e4a821fdfe1a538436 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sun, 14 Feb 2021 13:23:46 +0000
Subject: [PATCH 079/442] BatchTranslator* -> unique_ptr<BatchTranslator>

---
 src/translator/service.cpp | 3 ++-
 src/translator/service.h   | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index c93aa5f00..bdfb7e992 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -23,7 +23,8 @@ Service::Service(Ptr<Options> options)
     }
   } else {
     marian::DeviceId deviceId(/*cpuId=*/0, DeviceType::cpu);
-    translator = new BatchTranslator(deviceId, vocabs_, options);
+    translator =
+        UPtr<BatchTranslator>(new BatchTranslator(deviceId, vocabs_, options));
   }
 }
 
diff --git a/src/translator/service.h b/src/translator/service.h
index c57e609a7..db01468c7 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -71,7 +71,7 @@ class Service {
   std::vector<std::thread> workers_;
 
   // Optional
-  BatchTranslator *translator{nullptr};
+  UPtr<BatchTranslator> translator{nullptr};
 };
 
 std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);

From 0dbc8612c2431722152ca925f1bd7152187a399a Mon Sep 17 00:00:00 2001
From: Andre Natal <anatal@gmail.com>
Date: Sun, 14 Feb 2021 09:15:08 -0800
Subject: [PATCH 080/442] Adding missing bergamot-httpserver.js

---
 wasm/test_page/bergamot-httpserver.js | 39 +++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
 create mode 100644 wasm/test_page/bergamot-httpserver.js

diff --git a/wasm/test_page/bergamot-httpserver.js b/wasm/test_page/bergamot-httpserver.js
new file mode 100644
index 000000000..f23b3e750
--- /dev/null
+++ b/wasm/test_page/bergamot-httpserver.js
@@ -0,0 +1,39 @@
+require(__dirname  + '/helper.js');
+
+var http = require('http');
+var express = require('express');
+var app = express();
+var server = http.createServer(app);
+var fs = require('fs');
+var url = require('url');
+const nocache = require('nocache');
+const cors = require('cors');
+
+app.use(cors())
+app.use(nocache());
+app.get('/*.*' , cors(), function(req, res) {
+    var options = url.parse(req.url, true);
+    var mime = Helper.getMime(options);
+    serveFile(res, options.pathname, mime);
+});
+
+function serveFile(res, pathName, mime) {
+    mime = mime || 'text/html';
+    fs.readFile(__dirname + '/' + pathName, function (err, data) {
+        if (err) {
+            res.writeHead(500, {"Content-Type": "text/plain"});
+            return res.end('Error loading ' + pathName + " with Error: " + err);
+        }
+        res.header('Cross-Origin-Embedder-Policy','require-corp');
+        res.header('Cross-Origin-Opener-Policy','same-origin');
+        res.writeHead(200, {"Content-Type": mime});
+        res.end(data);
+    });
+}
+
+server.listen(8000);
+console.log('HTTP and BinaryJS server started on port 8000');
+
+
+
+

From 5bd4a1a3c0ef388249794298b5ed2c0b1cf92d05 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sun, 14 Feb 2021 19:58:29 +0000
Subject: [PATCH 081/442] Refactor: marian-TranslationResult and associated

marian-TranslationResult has more guards in place. Switching to a
construction on demand model for sentenceMappings. These changes
propogate to bergamot translation results.

Integration broke with the change in marian's internals, which are
updated accordingly to get back functionality.

Changes revealed a few bugs, which are fixed:

- ConfigParser already discovered in wasm-integration
  (https://github.com/browsermt/bergamot-translator/commit/a06530e92b6d16527487c8fa0ead4ae04f0ddbb5).
- Lambda captures and undefined values in DeviceId
---
 app/main-mts.cpp                      |  2 +-
 app/marian-decoder-new.cpp            |  4 +-
 src/translator/TranslationModel.cpp   | 18 +++++---
 src/translator/parser.h               |  3 +-
 src/translator/service.cpp            | 10 ++---
 src/translator/translation_result.cpp | 65 +++++++++++++++++----------
 src/translator/translation_result.h   | 42 +++++------------
 7 files changed, 75 insertions(+), 69 deletions(-)

diff --git a/app/main-mts.cpp b/app/main-mts.cpp
index c94ff306c..d8e756704 100644
--- a/app/main-mts.cpp
+++ b/app/main-mts.cpp
@@ -26,7 +26,7 @@ int main(int argc, char *argv[]) {
       service.translate(std::move(input));
   translation_result_future.wait();
   const TranslationResult &translation_result = translation_result_future.get();
-  std::cout << translation_result.getTranslatedText() << std::endl;
+  std::cout << translation_result.translation() << std::endl;
 
   // Stop Service.
   service.stop();
diff --git a/app/marian-decoder-new.cpp b/app/marian-decoder-new.cpp
index 62b1bb4b3..6e44fb777 100644
--- a/app/marian-decoder-new.cpp
+++ b/app/marian-decoder-new.cpp
@@ -54,8 +54,8 @@ int main(int argc, char *argv[]) {
   translation_result_future.wait();
   const TranslationResult &translation_result = translation_result_future.get();
 
-  marian_decoder_minimal(translation_result.getHistories(),
-                         service.targetVocab(), options);
+  marian_decoder_minimal(translation_result.histories(), service.targetVocab(),
+                         options);
 
   LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
   service.stop();
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index f501678cf..9c55422ef 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -14,6 +14,7 @@
 
 // All local project includes
 #include "TranslationModel.h"
+#include "translator/parser.h"
 #include "translator/service.h"
 
 std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
@@ -34,7 +35,7 @@ std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
   // Error: Aborted from void unhandledException() in
   // 3rd_party/marian-dev/src/common/logging.cpp:113
 
-  marian::ConfigParser configParser(marian::cli::mode::translation);
+  marian::ConfigParser configParser = marian::bergamot::createConfigParser();
   const YAML::Node &defaultConfig = configParser.getConfig();
 
   options.merge(defaultConfig);
@@ -70,18 +71,25 @@ TranslationModel::translate(std::vector<std::string> &&texts,
     intermediate.wait();
     auto mTranslationResult(std::move(intermediate.get()));
 
+    // This mess because marian::string_view != std::string_view
+    std::string source, translation;
+    marian::bergamot::TranslationResult::SentenceMappings mSentenceMappings;
+    mTranslationResult.move(source, translation, mSentenceMappings);
+
     // Convert to UnifiedAPI::TranslationResult
     TranslationResult::SentenceMappings sentenceMappings;
-    for (auto &p : mTranslationResult.getSentenceMappings()) {
+    for (auto &p : mSentenceMappings) {
       std::string_view src(p.first.data(), p.first.size()),
           tgt(p.second.data(), p.second.size());
       sentenceMappings.emplace_back(src, tgt);
     }
 
     // In place construction.
-    translationResults.emplace_back(std::move(mTranslationResult.source_),
-                                    std::move(mTranslationResult.translation_),
-                                    std::move(sentenceMappings));
+    translationResults.emplace_back(
+        std::move(source),          // &&mTranslationResult.source_
+        std::move(translation),     // &&mTranslationResult.translation_
+        std::move(sentenceMappings) // &&sentenceMappings
+    );
   }
 
   promise.set_value(std::move(translationResults));
diff --git a/src/translator/parser.h b/src/translator/parser.h
index e273d6aea..606b6a47b 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -5,7 +5,8 @@
 
 namespace marian {
 namespace bergamot {
-marian::ConfigParser createConfigParser() {
+
+inline marian::ConfigParser createConfigParser() {
   marian::ConfigParser cp(marian::cli::mode::translation);
   cp.addOption<std::string>(
       "--ssplit-prefix-file", "Bergamot Options",
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index bdfb7e992..ef2bacb64 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -15,11 +15,11 @@ Service::Service(Ptr<Options> options)
 
   if (numWorkers_ > 0) {
     workers_.reserve(numWorkers_);
-    for (int cpuId = 0; cpuId < numWorkers_; cpuId++) {
-      workers_.emplace_back([&] {
-        marian::DeviceId deviceId(cpuId, DeviceType::cpu);
-        translation_loop(deviceId, pcqueue_, vocabs_, options);
-      });
+    for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
+      marian::DeviceId deviceId(cpuId, DeviceType::cpu);
+      workers_.emplace_back(translation_loop, // Function
+                            deviceId, std::ref(pcqueue_), std::ref(vocabs_),
+                            options);
     }
   } else {
     marian::DeviceId deviceId(/*cpuId=*/0, DeviceType::cpu);
diff --git a/src/translator/translation_result.cpp b/src/translator/translation_result.cpp
index d69259f84..ee147be42 100644
--- a/src/translator/translation_result.cpp
+++ b/src/translator/translation_result.cpp
@@ -14,22 +14,26 @@ TranslationResult::TranslationResult(std::string &&source,
     : source_(std::move(source)), sourceRanges_(std::move(sourceRanges)),
       histories_(std::move(histories)) {
 
-  std::vector<string_view> sourceMappings;
-  std::vector<string_view> targetMappings;
+  constructTargetProperties(vocabs);
+}
 
-  // Process sourceMappings into sourceMappings.
-  sourceMappings.reserve(sourceRanges_.size());
-  for (int i = 0; i < sourceRanges_.size(); i++) {
-    string_view first = sourceRanges_[i].front();
-    string_view last = sourceRanges_[i].back();
-    sourceMappings.emplace_back(first.data(), last.end() - first.begin());
-  }
+void TranslationResult::move(std::string &source, std::string &translation,
+                             SentenceMappings &sentenceMappings) {
+
+  constructSentenceMappings(sentenceMappings);
+  // Totally illegal stuff.
+  source = std::move(source_);
+  translation = std::move(translation_);
 
-  // Compiles translations into a single std::string translation_
-  // Current implementation uses += on std::string, multiple resizes.
-  // Stores ByteRanges as indices first, followed by conversion into
-  // string_views.
-  // TODO(jerin): Add token level string_views here as well.
+  // The above assignment expects source, target be moved.
+  // which makes the following invalid, hence required to be cleared.
+  sourceRanges_.clear();
+  targetRanges_.clear();
+  histories_.clear();
+}
+
+void TranslationResult::constructTargetProperties(
+    std::vector<Ptr<Vocab const>> &vocabs) {
   std::vector<std::pair<int, int>> translationRanges;
   size_t offset{0};
   bool first{true};
@@ -52,21 +56,36 @@ TranslationResult::TranslationResult(std::string &&source,
     offset += decoded.size();
   }
 
-  // Converting ByteRanges as indices into string_views.
-  targetMappings.reserve(translationRanges.size());
+  // TODO(@jerinphilip):
+  // Currently considers target tokens as whole text. Needs
+  // to be further enhanced in marian-dev to extract alignments.
   for (auto &range : translationRanges) {
+    std::vector<string_view> targetMappings;
     const char *begin = &translation_[range.first];
     targetMappings.emplace_back(begin, range.second);
+    targetRanges_.push_back(std::move(targetMappings));
   }
+}
 
-  // Surely, let's add sentenceMappings_
-  for (auto src = sourceMappings.begin(), tgt = targetMappings.begin();
-       src != sourceMappings.end() && tgt != targetMappings.end();
-       ++src, ++tgt) {
-    sentenceMappings_.emplace_back(*src, *tgt);
-    auto &t = sentenceMappings_.back();
+void TranslationResult::constructSentenceMappings(
+    TranslationResult::SentenceMappings &sentenceMappings) {
+
+  for (int i = 0; i < sourceRanges_.size(); i++) {
+    string_view first, last;
+
+    // Handle source-sentence
+    first = sourceRanges_[i].front();
+    last = sourceRanges_[i].back();
+    string_view src_sentence(first.data(), last.end() - first.begin());
+
+    // Handle target-sentence
+    first = targetRanges_[i].front();
+    last = targetRanges_[i].back();
+    string_view tgt_sentence(first.data(), last.end() - first.begin());
+
+    // Add both into sentence-mappings
+    sentenceMappings.emplace_back(src_sentence, tgt_sentence);
   }
 }
-
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/translation_result.h b/src/translator/translation_result.h
index edc9a8ddd..5903145ad 100644
--- a/src/translator/translation_result.h
+++ b/src/translator/translation_result.h
@@ -22,53 +22,31 @@ class TranslationResult {
       : source_(std::move(other.source_)),
         translation_(std::move(other.translation_)),
         sourceRanges_(std::move(other.sourceRanges_)),
-        sentenceMappings_(std::move(other.sentenceMappings_)),
+        targetRanges_(std::move(other.targetRanges_)),
         histories_(std::move(other.histories_)){};
 
   TranslationResult(const TranslationResult &) = delete;
   TranslationResult &operator=(const TranslationResult &) = delete;
 
-  // Returns const references to source and translated texts, for external
-  // consumption.
-
-  const std::string &getOriginalText() const { return source_; }
-  const std::string &getTranslatedText() const { return translation_; }
-
-  // A mapping of string_views in the source_ and translation_ are provide as a
-  // pair for external consumption. Each entry corresponding
-  // to a (source-sentence, target-sentence).
-
   typedef std::vector<std::pair<const string_view, const string_view>>
       SentenceMappings;
-  const SentenceMappings &getSentenceMappings() const {
-    return sentenceMappings_;
-  }
 
-  // Return the Quality scores of the translated text.
-  // Not implemented currently, commenting out.
-  // const QualityScore &getQualityScore() const { return qualityScore; }
+  void move(std::string &source, std::string &target,
+            SentenceMappings &sentenceMappings);
 
-  // For development use to benchmark with marian-decoder.
-  const Histories &getHistories() const { return histories_; }
+  const Histories &histories() const { return histories_; }
+  const std::string &source() const { return source_; }
+  const std::string &translation() const { return translation_; }
 
-  // @jerinphilip: Why are these members no longer-private? For move-semantics
-  // with consistent string_views for bergamot-translator.
+private:
+  void constructTargetProperties(std::vector<Ptr<Vocab const>> &vocabs);
+  void constructSentenceMappings(SentenceMappings &);
 
   std::string source_;
   std::string translation_;
-  // Adding the following to complete bergamot-translator spec, redundant while
-  // sourceMappings_ and targetMappings_ exists or vice-versa.
-
-  SentenceMappings sentenceMappings_;
-
-private:
-  // Histories are currently required for interoperability with OutputPrinter
-  // and OutputCollector and hence comparisons with marian-decoder.
-  // Future hook to gain alignments.
   Histories histories_;
-
-  // string_views at the token level.
   std::vector<TokenRanges> sourceRanges_;
+  std::vector<TokenRanges> targetRanges_;
 };
 } // namespace bergamot
 } // namespace marian

From 0fc6105df49a4e0f05e1d382ea9909776ad3aeec Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sun, 14 Feb 2021 20:27:53 +0000
Subject: [PATCH 082/442] No more two TranslationResults (sort-of)

To avoid confusion, this commit renames
marian::bergamot::TranslationResult -> marian::bergamot::Response.
Usages of marian::bergamot::TranslationResults are updated across the
source to be consistent with the change and get source back working.
---
 app/main-mts.cpp                      | 13 ++++++-------
 app/marian-decoder-new.cpp            | 14 ++++++--------
 src/translator/TranslationModel.cpp   | 10 +++++-----
 src/translator/TranslationModel.h     |  3 ++-
 src/translator/request.cpp            | 11 +++++------
 src/translator/request.h              |  4 ++--
 src/translator/service.cpp            | 16 ++++++++--------
 src/translator/service.h              | 10 +++++-----
 src/translator/translation_result.cpp | 17 ++++++++---------
 src/translator/translation_result.h   | 14 ++++++--------
 10 files changed, 53 insertions(+), 59 deletions(-)

diff --git a/app/main-mts.cpp b/app/main-mts.cpp
index d8e756704..b5a4938b0 100644
--- a/app/main-mts.cpp
+++ b/app/main-mts.cpp
@@ -19,14 +19,13 @@ int main(int argc, char *argv[]) {
   std::ostringstream std_input;
   std_input << std::cin.rdbuf();
   std::string input = std_input.str();
-  using marian::bergamot::TranslationResult;
+  using marian::bergamot::Response;
 
-  // Wait on future until TranslationResult is complete
-  std::future<TranslationResult> translation_result_future =
-      service.translate(std::move(input));
-  translation_result_future.wait();
-  const TranslationResult &translation_result = translation_result_future.get();
-  std::cout << translation_result.translation() << std::endl;
+  // Wait on future until Response is complete
+  std::future<Response> responseFuture = service.translate(std::move(input));
+  responseFuture.wait();
+  const Response &response = responseFuture.get();
+  std::cout << response.translation() << std::endl;
 
   // Stop Service.
   service.stop();
diff --git a/app/marian-decoder-new.cpp b/app/marian-decoder-new.cpp
index 6e44fb777..8988310aa 100644
--- a/app/marian-decoder-new.cpp
+++ b/app/marian-decoder-new.cpp
@@ -46,16 +46,14 @@ int main(int argc, char *argv[]) {
   std::ostringstream std_input;
   std_input << std::cin.rdbuf();
   std::string input = std_input.str();
-  using marian::bergamot::TranslationResult;
+  using marian::bergamot::Response;
 
-  // Wait on future until TranslationResult is complete
-  std::future<TranslationResult> translation_result_future =
-      service.translate(std::move(input));
-  translation_result_future.wait();
-  const TranslationResult &translation_result = translation_result_future.get();
+  // Wait on future until Response is complete
+  std::future<Response> responseFuture = service.translate(std::move(input));
+  responseFuture.wait();
+  const Response &response = responseFuture.get();
 
-  marian_decoder_minimal(translation_result.histories(), service.targetVocab(),
-                         options);
+  marian_decoder_minimal(response.histories(), service.targetVocab(), options);
 
   LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
   service.stop();
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 9c55422ef..a5d396e36 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -69,12 +69,12 @@ TranslationModel::translate(std::vector<std::string> &&texts,
     // Collect future as marian::bergamot::TranslationResult
     auto intermediate = service_.translate(std::move(text));
     intermediate.wait();
-    auto mTranslationResult(std::move(intermediate.get()));
+    auto marianResponse(std::move(intermediate.get()));
 
     // This mess because marian::string_view != std::string_view
     std::string source, translation;
-    marian::bergamot::TranslationResult::SentenceMappings mSentenceMappings;
-    mTranslationResult.move(source, translation, mSentenceMappings);
+    marian::bergamot::Response::SentenceMappings mSentenceMappings;
+    marianResponse.move(source, translation, mSentenceMappings);
 
     // Convert to UnifiedAPI::TranslationResult
     TranslationResult::SentenceMappings sentenceMappings;
@@ -86,8 +86,8 @@ TranslationModel::translate(std::vector<std::string> &&texts,
 
     // In place construction.
     translationResults.emplace_back(
-        std::move(source),          // &&mTranslationResult.source_
-        std::move(translation),     // &&mTranslationResult.translation_
+        std::move(source),          // &&marianResponse.source_
+        std::move(translation),     // &&marianResponse.translation_
         std::move(sentenceMappings) // &&sentenceMappings
     );
   }
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index c922538a3..5f590d9e9 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -24,7 +24,8 @@
  */
 class TranslationModel : public AbstractTranslationModel {
 public:
-  /* Construct the model using the model configuration options as yaml-formatted string
+  /* Construct the model using the model configuration options as yaml-formatted
+   * string
    */
   TranslationModel(const std::string &config);
 
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index a743389b4..5433699f0 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -14,11 +14,11 @@ Request::Request(unsigned int Id, int lineNumberBegin,
                  std::vector<Ptr<Vocab const>> &vocabs, std::string &&source,
                  Segments &&segments,
                  std::vector<TokenRanges> &&sourceAlignments,
-                 std::promise<TranslationResult> translationResultPromise)
+                 std::promise<Response> responsePromise)
     : Id_(Id), lineNumberBegin_(lineNumberBegin), vocabs_(&vocabs),
       source_(std::move(source)), segments_(std::move(segments)),
       sourceAlignments_(std::move(sourceAlignments)),
-      response_(std::move(translationResultPromise)) {
+      response_(std::move(responsePromise)) {
 
   counter_ = segments_.size();
   histories_.resize(segments_.size(), nullptr);
@@ -47,10 +47,9 @@ void Request::processHistory(size_t index, Ptr<History> history) {
 
 void Request::completeRequest() {
   // Request no longer needs to hold the content, can transfer it to
-  // TranslationResult.
-  TranslationResult translation_result(std::move(source_),
-                                       std::move(sourceAlignments_),
-                                       std::move(histories_), *vocabs_);
+  // Response.
+  Response translation_result(std::move(source_), std::move(sourceAlignments_),
+                              std::move(histories_), *vocabs_);
   response_.set_value(std::move(translation_result));
 }
 
diff --git a/src/translator/request.h b/src/translator/request.h
index eab0d4b2a..ddd6cccc0 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -48,13 +48,13 @@ class Request {
   std::vector<TokenRanges> sourceAlignments_;
   std::vector<Ptr<History>> histories_;
 
-  std::promise<TranslationResult> response_;
+  std::promise<Response> response_;
 
 public:
   Request(unsigned int Id, int lineNumberBegin,
           std::vector<Ptr<Vocab const>> &vocabs_, std::string &&source,
           Segments &&segments, std::vector<TokenRanges> &&sourceAlignments,
-          std::promise<TranslationResult> translationResultPromise);
+          std::promise<Response> responsePromise);
 
   // Obtain the count of tokens in the segment correponding to index. Used to
   // insert sentence from multiple requests into the corresponding size bucket.
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index ef2bacb64..4ab539fa8 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -28,11 +28,11 @@ Service::Service(Ptr<Options> options)
   }
 }
 
-std::future<TranslationResult> Service::translateWithCopy(std::string input) {
+std::future<Response> Service::translateWithCopy(std::string input) {
   return translate(std::move(input));
 }
 
-std::future<TranslationResult> Service::translate(std::string &&input) {
+std::future<Response> Service::translate(std::string &&input) {
   // Takes in a blob of text. Segments and std::vector<TokenRanges> are
   // extracted from the input (blob of text) and used to construct a Request
   // along with a promise. promise value is set by the worker completing a
@@ -49,13 +49,13 @@ std::future<TranslationResult> Service::translate(std::string &&input) {
   std::vector<TokenRanges> sourceAlignments;
   text_processor_.process(input, segments, sourceAlignments);
 
-  std::promise<TranslationResult> translationResultPromise;
-  auto future = translationResultPromise.get_future();
+  std::promise<Response> responsePromise;
+  auto future = responsePromise.get_future();
 
-  Ptr<Request> request = New<Request>(
-      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(input),
-      std::move(segments), std::move(sourceAlignments),
-      std::move(translationResultPromise));
+  Ptr<Request> request =
+      New<Request>(requestId_++, /* lineNumberBegin = */ 0, vocabs_,
+                   std::move(input), std::move(segments),
+                   std::move(sourceAlignments), std::move(responsePromise));
 
   batcher_.addWholeRequest(request);
 
diff --git a/src/translator/service.h b/src/translator/service.h
index db01468c7..6f26bc8a6 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -25,17 +25,17 @@ class Service {
   //  options = ...;
   //  service = Service(options);
   //  std::string input_blob = "Hello World";
-  //  std::future<TranslationResult>
+  //  std::future<Response>
   //      response = service.translate(std::move(input_blob));
   //  response.wait();
-  //  TranslationResult result = response.get();
+  //  Response result = response.get();
 
 public:
   explicit Service(Ptr<Options> options);
 
   // Constructs new string copying, calls translate internally.
-  std::future<TranslationResult> translateWithCopy(std::string input);
-  std::future<TranslationResult> translate(std::string &&input);
+  std::future<Response> translateWithCopy(std::string input);
+  std::future<Response> translate(std::string &&input);
 
   void stop();
 
@@ -49,7 +49,7 @@ class Service {
   int numWorkers_;
 
   // vocabs are used to construct a Request, which later uses it to construct
-  // TranslationResult (decode from words to string).
+  // Response (decode from words to string).
   std::vector<Ptr<Vocab const>> vocabs_; // ORDER DEPENDENCY
 
   // Consists of:
diff --git a/src/translator/translation_result.cpp b/src/translator/translation_result.cpp
index ee147be42..58f092630 100644
--- a/src/translator/translation_result.cpp
+++ b/src/translator/translation_result.cpp
@@ -7,18 +7,17 @@
 namespace marian {
 namespace bergamot {
 
-TranslationResult::TranslationResult(std::string &&source,
-                                     std::vector<TokenRanges> &&sourceRanges,
-                                     Histories &&histories,
-                                     std::vector<Ptr<Vocab const>> &vocabs)
+Response::Response(std::string &&source,
+                   std::vector<TokenRanges> &&sourceRanges,
+                   Histories &&histories, std::vector<Ptr<Vocab const>> &vocabs)
     : source_(std::move(source)), sourceRanges_(std::move(sourceRanges)),
       histories_(std::move(histories)) {
 
   constructTargetProperties(vocabs);
 }
 
-void TranslationResult::move(std::string &source, std::string &translation,
-                             SentenceMappings &sentenceMappings) {
+void Response::move(std::string &source, std::string &translation,
+                    SentenceMappings &sentenceMappings) {
 
   constructSentenceMappings(sentenceMappings);
   // Totally illegal stuff.
@@ -32,7 +31,7 @@ void TranslationResult::move(std::string &source, std::string &translation,
   histories_.clear();
 }
 
-void TranslationResult::constructTargetProperties(
+void Response::constructTargetProperties(
     std::vector<Ptr<Vocab const>> &vocabs) {
   std::vector<std::pair<int, int>> translationRanges;
   size_t offset{0};
@@ -67,8 +66,8 @@ void TranslationResult::constructTargetProperties(
   }
 }
 
-void TranslationResult::constructSentenceMappings(
-    TranslationResult::SentenceMappings &sentenceMappings) {
+void Response::constructSentenceMappings(
+    Response::SentenceMappings &sentenceMappings) {
 
   for (int i = 0; i < sourceRanges_.size(); i++) {
     string_view first, last;
diff --git a/src/translator/translation_result.h b/src/translator/translation_result.h
index 5903145ad..6ed892732 100644
--- a/src/translator/translation_result.h
+++ b/src/translator/translation_result.h
@@ -11,22 +11,20 @@
 
 namespace marian {
 namespace bergamot {
-class TranslationResult {
+class Response {
 public:
-  TranslationResult(std::string &&source,
-                    std::vector<TokenRanges> &&sourceRanges,
-                    Histories &&histories,
-                    std::vector<Ptr<Vocab const>> &vocabs);
+  Response(std::string &&source, std::vector<TokenRanges> &&sourceRanges,
+           Histories &&histories, std::vector<Ptr<Vocab const>> &vocabs);
 
-  TranslationResult(TranslationResult &&other)
+  Response(Response &&other)
       : source_(std::move(other.source_)),
         translation_(std::move(other.translation_)),
         sourceRanges_(std::move(other.sourceRanges_)),
         targetRanges_(std::move(other.targetRanges_)),
         histories_(std::move(other.histories_)){};
 
-  TranslationResult(const TranslationResult &) = delete;
-  TranslationResult &operator=(const TranslationResult &) = delete;
+  Response(const Response &) = delete;
+  Response &operator=(const Response &) = delete;
 
   typedef std::vector<std::pair<const string_view, const string_view>>
       SentenceMappings;

From 370e9e2fb619b5f45693a3d4e6e3dac1442b6fed Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sun, 14 Feb 2021 20:35:41 +0000
Subject: [PATCH 083/442] {translation_result -> response}.h; propogates;

---
 app/main-mts.cpp                                        | 2 +-
 app/marian-decoder-new.cpp                              | 2 +-
 src/translator/CMakeLists.txt                           | 2 +-
 src/translator/request.cpp                              | 2 +-
 src/translator/request.h                                | 2 +-
 src/translator/{translation_result.cpp => response.cpp} | 2 +-
 src/translator/{translation_result.h => response.h}     | 6 +++---
 src/translator/service.h                                | 2 +-
 8 files changed, 10 insertions(+), 10 deletions(-)
 rename src/translator/{translation_result.cpp => response.cpp} (98%)
 rename src/translator/{translation_result.h => response.h} (91%)

diff --git a/app/main-mts.cpp b/app/main-mts.cpp
index b5a4938b0..78967be0e 100644
--- a/app/main-mts.cpp
+++ b/app/main-mts.cpp
@@ -7,8 +7,8 @@
 #include "common/utils.h"
 #include "marian.h"
 #include "translator/parser.h"
+#include "translator/response.h"
 #include "translator/service.h"
-#include "translator/translation_result.h"
 
 int main(int argc, char *argv[]) {
   auto cp = marian::bergamot::createConfigParser();
diff --git a/app/marian-decoder-new.cpp b/app/marian-decoder-new.cpp
index 8988310aa..f8079096d 100644
--- a/app/marian-decoder-new.cpp
+++ b/app/marian-decoder-new.cpp
@@ -11,8 +11,8 @@
 #include "translator/output_collector.h"
 #include "translator/output_printer.h"
 #include "translator/parser.h"
+#include "translator/response.h"
 #include "translator/service.h"
-#include "translator/translation_result.h"
 
 void marian_decoder_minimal(const marian::Histories &histories,
                             marian::Ptr<marian::Vocab const> targetVocab,
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 16c3db962..c279ab975 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -10,7 +10,7 @@ add_library(bergamot-translator STATIC
     request.cpp 
     service.cpp
     batcher.cpp
-    translation_result.cpp
+    response.cpp
 )
 
 target_link_libraries(bergamot-translator marian ssplit)
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 5433699f0..23bd67963 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -1,7 +1,7 @@
 #include "request.h"
 
 #include "definitions.h"
-#include "translation_result.h"
+#include "response.h"
 
 #include "common/logging.h"
 
diff --git a/src/translator/request.h b/src/translator/request.h
index ddd6cccc0..8912a497d 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -22,7 +22,7 @@
 #define SRC_BERGAMOT_REQUEST_H_
 
 #include "definitions.h"
-#include "translation_result.h"
+#include "response.h"
 
 #include "common/logging.h"
 #include "data/types.h"
diff --git a/src/translator/translation_result.cpp b/src/translator/response.cpp
similarity index 98%
rename from src/translator/translation_result.cpp
rename to src/translator/response.cpp
index 58f092630..d40f88da7 100644
--- a/src/translator/translation_result.cpp
+++ b/src/translator/response.cpp
@@ -1,4 +1,4 @@
-#include "translation_result.h"
+#include "response.h"
 #include "common/logging.h"
 #include "data/alignment.h"
 
diff --git a/src/translator/translation_result.h b/src/translator/response.h
similarity index 91%
rename from src/translator/translation_result.h
rename to src/translator/response.h
index 6ed892732..57377176d 100644
--- a/src/translator/translation_result.h
+++ b/src/translator/response.h
@@ -1,5 +1,5 @@
-#ifndef SRC_BERGAMOT_TRANSLATION_RESULT_H_
-#define SRC_BERGAMOT_TRANSLATION_RESULT_H_
+#ifndef SRC_BERGAMOT_RESPONSE_H_
+#define SRC_BERGAMOT_RESPONSE_H_
 
 #include "data/types.h"
 #include "definitions.h"
@@ -49,4 +49,4 @@ class Response {
 } // namespace bergamot
 } // namespace marian
 
-#endif // SRC_BERGAMOT_TRANSLATION_RESULT_H_
+#endif // SRC_BERGAMOT_RESPONSE_H_
diff --git a/src/translator/service.h b/src/translator/service.h
index 6f26bc8a6..38a45c6d0 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -4,8 +4,8 @@
 #include "batch_translator.h"
 #include "batcher.h"
 #include "pcqueue.h"
+#include "response.h"
 #include "text_processor.h"
-#include "translation_result.h"
 
 #include <queue>
 #include <vector>

From be455a3da101132c5d7c3a283b90cc1cffd8a119 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sun, 14 Feb 2021 22:08:17 +0000
Subject: [PATCH 084/442] Straightening multithreading in translator workers

BatchTranslators are now held in Service. Threads are separate, and
constructed via lambdas. Retaining BatchTranslator class and member
function (Probably a matter of taste I guess).

This should eliminate complaints in (#10), hopefully.
---
 src/translator/batch_translator.cpp | 12 +++++-------
 src/translator/batch_translator.h   |  6 ++----
 src/translator/service.cpp          | 28 +++++++++++++++++++---------
 src/translator/service.h            |  4 +---
 4 files changed, 27 insertions(+), 23 deletions(-)

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 13eb58a21..7da63cf57 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -10,7 +10,9 @@ namespace bergamot {
 BatchTranslator::BatchTranslator(DeviceId const device,
                                  std::vector<Ptr<Vocab const>> &vocabs,
                                  Ptr<Options> options)
-    : device_(device), options_(options), vocabs_(&vocabs) {
+    : device_(device), options_(options), vocabs_(&vocabs) {}
+
+void BatchTranslator::initialize() {
   // Initializes the graph.
   if (options_->hasAndNotEmpty("shortlist")) {
     int srcIdx = 0, trgIdx = 1;
@@ -93,11 +95,7 @@ void BatchTranslator::translate(Batch &batch) {
   batch.completeBatch(histories);
 }
 
-void translation_loop(DeviceId const &device, PCQueue<Batch> &pcqueue,
-                      std::vector<Ptr<Vocab const>> &vocabs,
-                      Ptr<Options> options) {
-
-  BatchTranslator translator(device, vocabs, options);
+void BatchTranslator::consumeFrom(PCQueue<Batch> &pcqueue) {
   Batch batch;
   Histories histories;
   while (true) {
@@ -105,7 +103,7 @@ void translation_loop(DeviceId const &device, PCQueue<Batch> &pcqueue,
     if (batch.isPoison()) {
       return;
     } else {
-      translator.translate(batch);
+      translate(batch);
     }
   }
 }
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 2ee4e04ef..83b911ceb 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -28,6 +28,8 @@ class BatchTranslator {
   // convenience function for logging. TODO(jerin)
   std::string _identifier() { return "worker" + std::to_string(device_.no); }
   void translate(Batch &batch);
+  void initialize();
+  void consumeFrom(PCQueue<Batch> &pcqueue);
 
 private:
   Ptr<Options> options_;
@@ -38,10 +40,6 @@ class BatchTranslator {
   Ptr<data::ShortlistGenerator const> slgen_;
 };
 
-void translation_loop(DeviceId const &device, PCQueue<Batch> &pcqueue,
-                      std::vector<Ptr<Vocab const>> &vocabs,
-                      Ptr<Options> options);
-
 } // namespace bergamot
 } // namespace marian
 
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 4ab539fa8..1b33558e7 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -13,18 +13,28 @@ Service::Service(Ptr<Options> options)
       text_processor_(vocabs_, options), batcher_(options),
       pcqueue_(2 * options->get<int>("cpu-threads")) {
 
-  if (numWorkers_ > 0) {
+  if (numWorkers_ == 0) {
+    // In case workers are 0, a single-translator is created and initialized
+    // in the main thread.
+    marian::DeviceId deviceId(/*cpuId=*/0, DeviceType::cpu);
+    translators_.emplace_back(deviceId, vocabs_, options);
+    translators_.back().initialize();
+  } else {
+    // If workers specified are greater than 0, translators_ are populated with
+    // unitialized instances. These are then initialized inside
+    // individual threads and set to consume from producer-consumer queue.
     workers_.reserve(numWorkers_);
+    translators_.reserve(numWorkers_);
     for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
       marian::DeviceId deviceId(cpuId, DeviceType::cpu);
-      workers_.emplace_back(translation_loop, // Function
-                            deviceId, std::ref(pcqueue_), std::ref(vocabs_),
-                            options);
+      translators_.emplace_back(deviceId, vocabs_, options);
+
+      auto &translator = translators_.back();
+      workers_.emplace_back([&translator, this] {
+        translator.initialize();
+        translator.consumeFrom(pcqueue_);
+      });
     }
-  } else {
-    marian::DeviceId deviceId(/*cpuId=*/0, DeviceType::cpu);
-    translator =
-        UPtr<BatchTranslator>(new BatchTranslator(deviceId, vocabs_, options));
   }
 }
 
@@ -65,7 +75,7 @@ std::future<Response> Service::translate(std::string &&input) {
     // Queue single-threaded
     Batch batch;
     while (batcher_ >> batch) {
-      translator->translate(batch);
+      translators_[0].translate(batch);
     }
   }
 
diff --git a/src/translator/service.h b/src/translator/service.h
index 38a45c6d0..55b754a2f 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -68,10 +68,8 @@ class Service {
   TextProcessor text_processor_; // ORDER DEPENDENCY
   Batcher batcher_;
   PCQueue<Batch> pcqueue_;
+  std::vector<BatchTranslator> translators_;
   std::vector<std::thread> workers_;
-
-  // Optional
-  UPtr<BatchTranslator> translator{nullptr};
 };
 
 std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);

From 45a8309c6972b121d62f1e9329267f752b8c796b Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sun, 14 Feb 2021 22:28:08 +0000
Subject: [PATCH 085/442] Missed translation_result -> response rename

---
 src/translator/request.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 23bd67963..9317f697e 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -48,9 +48,9 @@ void Request::processHistory(size_t index, Ptr<History> history) {
 void Request::completeRequest() {
   // Request no longer needs to hold the content, can transfer it to
   // Response.
-  Response translation_result(std::move(source_), std::move(sourceAlignments_),
-                              std::move(histories_), *vocabs_);
-  response_.set_value(std::move(translation_result));
+  Response response(std::move(source_), std::move(sourceAlignments_),
+                    std::move(histories_), *vocabs_);
+  response_.set_value(std::move(response));
 }
 
 bool Request::operator<(const Request &b) const {

From d27a96fc53add7b36d063aaf86c528bc03798eea Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 10:04:15 +0200
Subject: [PATCH 086/442] Updated wasm readme

---
 wasm/README.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/wasm/README.md b/wasm/README.md
index 6be620956..131f9eb06 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -37,11 +37,12 @@ You can also see everything in action by following the next steps:
 * Start the test webserver (ensure you have the latest nodejs installed)
 ```
 cd test_page
-bash start_server
+bash start_server.sh
 ```
 * Open any of the browsers below
     * Firefox Nightly +87: make sure the following prefs are on (about:config)
     ````
+    dom.postMessage.sharedArrayBuffer.bypassCOOP_COEP.insecure.enabled = true
     javascript.options.wasm_simd = true
     javascript.options.wasm_simd_wormhole = true
     ````

From f7c86518cfbe418ba9db6655a6e093de520c618d Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 10:04:49 +0200
Subject: [PATCH 087/442] Update test page package-lock.json

---
 wasm/test_page/package-lock.json | 515 +------------------------------
 1 file changed, 1 insertion(+), 514 deletions(-)

diff --git a/wasm/test_page/package-lock.json b/wasm/test_page/package-lock.json
index 065c92de8..ae4cb9dd6 100644
--- a/wasm/test_page/package-lock.json
+++ b/wasm/test_page/package-lock.json
@@ -1,519 +1,6 @@
 {
-  "name": "test_page",
-  "lockfileVersion": 2,
   "requires": true,
-  "packages": {
-    "": {
-      "dependencies": {
-        "cors": "^2.8.5",
-        "express": "^4.17.1",
-        "nocache": "^2.1.0"
-      }
-    },
-    "node_modules/accepts": {
-      "version": "1.3.7",
-      "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.7.tgz",
-      "integrity": "sha512-Il80Qs2WjYlJIBNzNkK6KYqlVMTbZLXgHx2oT0pU/fjRHyEp+PEfEPY0R3WCwAGVOtauxh1hOxNgIf5bv7dQpA==",
-      "dependencies": {
-        "mime-types": "~2.1.24",
-        "negotiator": "0.6.2"
-      },
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/array-flatten": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz",
-      "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI="
-    },
-    "node_modules/body-parser": {
-      "version": "1.19.0",
-      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.19.0.tgz",
-      "integrity": "sha512-dhEPs72UPbDnAQJ9ZKMNTP6ptJaionhP5cBb541nXPlW60Jepo9RV/a4fX4XWW9CuFNK22krhrj1+rgzifNCsw==",
-      "dependencies": {
-        "bytes": "3.1.0",
-        "content-type": "~1.0.4",
-        "debug": "2.6.9",
-        "depd": "~1.1.2",
-        "http-errors": "1.7.2",
-        "iconv-lite": "0.4.24",
-        "on-finished": "~2.3.0",
-        "qs": "6.7.0",
-        "raw-body": "2.4.0",
-        "type-is": "~1.6.17"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/bytes": {
-      "version": "3.1.0",
-      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.0.tgz",
-      "integrity": "sha512-zauLjrfCG+xvoyaqLoV8bLVXXNGC4JqlxFCutSDWA6fJrTo2ZuvLYTqZ7aHBLZSMOopbzwv8f+wZcVzfVTI2Dg==",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/content-disposition": {
-      "version": "0.5.3",
-      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.3.tgz",
-      "integrity": "sha512-ExO0774ikEObIAEV9kDo50o+79VCUdEB6n6lzKgGwupcVeRlhrj3qGAfwq8G6uBJjkqLrhT0qEYFcWng8z1z0g==",
-      "dependencies": {
-        "safe-buffer": "5.1.2"
-      },
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/content-type": {
-      "version": "1.0.4",
-      "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz",
-      "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/cookie": {
-      "version": "0.4.0",
-      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.4.0.tgz",
-      "integrity": "sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg==",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/cookie-signature": {
-      "version": "1.0.6",
-      "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.6.tgz",
-      "integrity": "sha1-4wOogrNCzD7oylE6eZmXNNqzriw="
-    },
-    "node_modules/cors": {
-      "version": "2.8.5",
-      "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.5.tgz",
-      "integrity": "sha512-KIHbLJqu73RGr/hnbrO9uBeixNGuvSQjul/jdFvS/KFSIH1hWVd1ng7zOHx+YrEfInLG7q4n6GHQ9cDtxv/P6g==",
-      "dependencies": {
-        "object-assign": "^4",
-        "vary": "^1"
-      },
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
-    "node_modules/debug": {
-      "version": "2.6.9",
-      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
-      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
-      "dependencies": {
-        "ms": "2.0.0"
-      }
-    },
-    "node_modules/depd": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/depd/-/depd-1.1.2.tgz",
-      "integrity": "sha1-m81S4UwJd2PnSbJ0xDRu0uVgtak=",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/destroy": {
-      "version": "1.0.4",
-      "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.0.4.tgz",
-      "integrity": "sha1-l4hXRCxEdJ5CBmE+N5RiBYJqvYA="
-    },
-    "node_modules/ee-first": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
-      "integrity": "sha1-WQxhFWsK4vTwJVcyoViyZrxWsh0="
-    },
-    "node_modules/encodeurl": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
-      "integrity": "sha1-rT/0yG7C0CkyL1oCw6mmBslbP1k=",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/escape-html": {
-      "version": "1.0.3",
-      "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
-      "integrity": "sha1-Aljq5NPQwJdN4cFpGI7wBR0dGYg="
-    },
-    "node_modules/etag": {
-      "version": "1.8.1",
-      "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
-      "integrity": "sha1-Qa4u62XvpiJorr/qg6x9eSmbCIc=",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/express": {
-      "version": "4.17.1",
-      "resolved": "https://registry.npmjs.org/express/-/express-4.17.1.tgz",
-      "integrity": "sha512-mHJ9O79RqluphRrcw2X/GTh3k9tVv8YcoyY4Kkh4WDMUYKRZUq0h1o0w2rrrxBqM7VoeUVqgb27xlEMXTnYt4g==",
-      "dependencies": {
-        "accepts": "~1.3.7",
-        "array-flatten": "1.1.1",
-        "body-parser": "1.19.0",
-        "content-disposition": "0.5.3",
-        "content-type": "~1.0.4",
-        "cookie": "0.4.0",
-        "cookie-signature": "1.0.6",
-        "debug": "2.6.9",
-        "depd": "~1.1.2",
-        "encodeurl": "~1.0.2",
-        "escape-html": "~1.0.3",
-        "etag": "~1.8.1",
-        "finalhandler": "~1.1.2",
-        "fresh": "0.5.2",
-        "merge-descriptors": "1.0.1",
-        "methods": "~1.1.2",
-        "on-finished": "~2.3.0",
-        "parseurl": "~1.3.3",
-        "path-to-regexp": "0.1.7",
-        "proxy-addr": "~2.0.5",
-        "qs": "6.7.0",
-        "range-parser": "~1.2.1",
-        "safe-buffer": "5.1.2",
-        "send": "0.17.1",
-        "serve-static": "1.14.1",
-        "setprototypeof": "1.1.1",
-        "statuses": "~1.5.0",
-        "type-is": "~1.6.18",
-        "utils-merge": "1.0.1",
-        "vary": "~1.1.2"
-      },
-      "engines": {
-        "node": ">= 0.10.0"
-      }
-    },
-    "node_modules/finalhandler": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.2.tgz",
-      "integrity": "sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA==",
-      "dependencies": {
-        "debug": "2.6.9",
-        "encodeurl": "~1.0.2",
-        "escape-html": "~1.0.3",
-        "on-finished": "~2.3.0",
-        "parseurl": "~1.3.3",
-        "statuses": "~1.5.0",
-        "unpipe": "~1.0.0"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/forwarded": {
-      "version": "0.1.2",
-      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.1.2.tgz",
-      "integrity": "sha1-mMI9qxF1ZXuMBXPozszZGw/xjIQ=",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/fresh": {
-      "version": "0.5.2",
-      "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz",
-      "integrity": "sha1-PYyt2Q2XZWn6g1qx+OSyOhBWBac=",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/http-errors": {
-      "version": "1.7.2",
-      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.7.2.tgz",
-      "integrity": "sha512-uUQBt3H/cSIVfch6i1EuPNy/YsRSOUBXTVfZ+yR7Zjez3qjBz6i9+i4zjNaoqcoFVI4lQJ5plg63TvGfRSDCRg==",
-      "dependencies": {
-        "depd": "~1.1.2",
-        "inherits": "2.0.3",
-        "setprototypeof": "1.1.1",
-        "statuses": ">= 1.5.0 < 2",
-        "toidentifier": "1.0.0"
-      },
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/iconv-lite": {
-      "version": "0.4.24",
-      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
-      "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
-      "dependencies": {
-        "safer-buffer": ">= 2.1.2 < 3"
-      },
-      "engines": {
-        "node": ">=0.10.0"
-      }
-    },
-    "node_modules/inherits": {
-      "version": "2.0.3",
-      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz",
-      "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4="
-    },
-    "node_modules/ipaddr.js": {
-      "version": "1.9.1",
-      "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
-      "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
-    "node_modules/media-typer": {
-      "version": "0.3.0",
-      "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
-      "integrity": "sha1-hxDXrwqmJvj/+hzgAWhUUmMlV0g=",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/merge-descriptors": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz",
-      "integrity": "sha1-sAqqVW3YtEVoFQ7J0blT8/kMu2E="
-    },
-    "node_modules/methods": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz",
-      "integrity": "sha1-VSmk1nZUE07cxSZmVoNbD4Ua/O4=",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/mime": {
-      "version": "1.6.0",
-      "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
-      "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==",
-      "bin": {
-        "mime": "cli.js"
-      },
-      "engines": {
-        "node": ">=4"
-      }
-    },
-    "node_modules/mime-db": {
-      "version": "1.45.0",
-      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.45.0.tgz",
-      "integrity": "sha512-CkqLUxUk15hofLoLyljJSrukZi8mAtgd+yE5uO4tqRZsdsAJKv0O+rFMhVDRJgozy+yG6md5KwuXhD4ocIoP+w==",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/mime-types": {
-      "version": "2.1.28",
-      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.28.tgz",
-      "integrity": "sha512-0TO2yJ5YHYr7M2zzT7gDU1tbwHxEUWBCLt0lscSNpcdAfFyJOVEpRYNS7EXVcTLNj/25QO8gulHC5JtTzSE2UQ==",
-      "dependencies": {
-        "mime-db": "1.45.0"
-      },
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/ms": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
-      "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g="
-    },
-    "node_modules/negotiator": {
-      "version": "0.6.2",
-      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.2.tgz",
-      "integrity": "sha512-hZXc7K2e+PgeI1eDBe/10Ard4ekbfrrqG8Ep+8Jmf4JID2bNg7NvCPOZN+kfF574pFQI7mum2AUqDidoKqcTOw==",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/nocache": {
-      "version": "2.1.0",
-      "resolved": "https://registry.npmjs.org/nocache/-/nocache-2.1.0.tgz",
-      "integrity": "sha512-0L9FvHG3nfnnmaEQPjT9xhfN4ISk0A8/2j4M37Np4mcDesJjHgEUfgPhdCyZuFI954tjokaIj/A3NdpFNdEh4Q==",
-      "engines": {
-        "node": ">=4.0.0"
-      }
-    },
-    "node_modules/object-assign": {
-      "version": "4.1.1",
-      "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
-      "integrity": "sha1-IQmtx5ZYh8/AXLvUQsrIv7s2CGM=",
-      "engines": {
-        "node": ">=0.10.0"
-      }
-    },
-    "node_modules/on-finished": {
-      "version": "2.3.0",
-      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.3.0.tgz",
-      "integrity": "sha1-IPEzZIGwg811M3mSoWlxqi2QaUc=",
-      "dependencies": {
-        "ee-first": "1.1.1"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/parseurl": {
-      "version": "1.3.3",
-      "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
-      "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/path-to-regexp": {
-      "version": "0.1.7",
-      "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz",
-      "integrity": "sha1-32BBeABfUi8V60SQ5yR6G/qmf4w="
-    },
-    "node_modules/proxy-addr": {
-      "version": "2.0.6",
-      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.6.tgz",
-      "integrity": "sha512-dh/frvCBVmSsDYzw6n926jv974gddhkFPfiN8hPOi30Wax25QZyZEGveluCgliBnqmuM+UJmBErbAUFIoDbjOw==",
-      "dependencies": {
-        "forwarded": "~0.1.2",
-        "ipaddr.js": "1.9.1"
-      },
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
-    "node_modules/qs": {
-      "version": "6.7.0",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.7.0.tgz",
-      "integrity": "sha512-VCdBRNFTX1fyE7Nb6FYoURo/SPe62QCaAyzJvUjwRaIsc+NePBEniHlvxFmmX56+HZphIGtV0XeCirBtpDrTyQ==",
-      "engines": {
-        "node": ">=0.6"
-      }
-    },
-    "node_modules/range-parser": {
-      "version": "1.2.1",
-      "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
-      "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/raw-body": {
-      "version": "2.4.0",
-      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.4.0.tgz",
-      "integrity": "sha512-4Oz8DUIwdvoa5qMJelxipzi/iJIi40O5cGV1wNYp5hvZP8ZN0T+jiNkL0QepXs+EsQ9XJ8ipEDoiH70ySUJP3Q==",
-      "dependencies": {
-        "bytes": "3.1.0",
-        "http-errors": "1.7.2",
-        "iconv-lite": "0.4.24",
-        "unpipe": "1.0.0"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/safe-buffer": {
-      "version": "5.1.2",
-      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
-      "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="
-    },
-    "node_modules/safer-buffer": {
-      "version": "2.1.2",
-      "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
-      "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
-    },
-    "node_modules/send": {
-      "version": "0.17.1",
-      "resolved": "https://registry.npmjs.org/send/-/send-0.17.1.tgz",
-      "integrity": "sha512-BsVKsiGcQMFwT8UxypobUKyv7irCNRHk1T0G680vk88yf6LBByGcZJOTJCrTP2xVN6yI+XjPJcNuE3V4fT9sAg==",
-      "dependencies": {
-        "debug": "2.6.9",
-        "depd": "~1.1.2",
-        "destroy": "~1.0.4",
-        "encodeurl": "~1.0.2",
-        "escape-html": "~1.0.3",
-        "etag": "~1.8.1",
-        "fresh": "0.5.2",
-        "http-errors": "~1.7.2",
-        "mime": "1.6.0",
-        "ms": "2.1.1",
-        "on-finished": "~2.3.0",
-        "range-parser": "~1.2.1",
-        "statuses": "~1.5.0"
-      },
-      "engines": {
-        "node": ">= 0.8.0"
-      }
-    },
-    "node_modules/send/node_modules/ms": {
-      "version": "2.1.1",
-      "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz",
-      "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg=="
-    },
-    "node_modules/serve-static": {
-      "version": "1.14.1",
-      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.14.1.tgz",
-      "integrity": "sha512-JMrvUwE54emCYWlTI+hGrGv5I8dEwmco/00EvkzIIsR7MqrHonbD9pO2MOfFnpFntl7ecpZs+3mW+XbQZu9QCg==",
-      "dependencies": {
-        "encodeurl": "~1.0.2",
-        "escape-html": "~1.0.3",
-        "parseurl": "~1.3.3",
-        "send": "0.17.1"
-      },
-      "engines": {
-        "node": ">= 0.8.0"
-      }
-    },
-    "node_modules/setprototypeof": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.1.tgz",
-      "integrity": "sha512-JvdAWfbXeIGaZ9cILp38HntZSFSo3mWg6xGcJJsd+d4aRMOqauag1C63dJfDw7OaMYwEbHMOxEZ1lqVRYP2OAw=="
-    },
-    "node_modules/statuses": {
-      "version": "1.5.0",
-      "resolved": "https://registry.npmjs.org/statuses/-/statuses-1.5.0.tgz",
-      "integrity": "sha1-Fhx9rBd2Wf2YEfQ3cfqZOBR4Yow=",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/toidentifier": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.0.tgz",
-      "integrity": "sha512-yaOH/Pk/VEhBWWTlhI+qXxDFXlejDGcQipMlyxda9nthulaxLZUNcUqFxokp0vcYnvteJln5FNQDRrxj3YcbVw==",
-      "engines": {
-        "node": ">=0.6"
-      }
-    },
-    "node_modules/type-is": {
-      "version": "1.6.18",
-      "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
-      "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",
-      "dependencies": {
-        "media-typer": "0.3.0",
-        "mime-types": "~2.1.24"
-      },
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/unpipe": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
-      "integrity": "sha1-sr9O6FFKrmFltIF4KdIbLvSZBOw=",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/utils-merge": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",
-      "integrity": "sha1-n5VxD1CiZ5R7LMwSR0HBAoQn5xM=",
-      "engines": {
-        "node": ">= 0.4.0"
-      }
-    },
-    "node_modules/vary": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
-      "integrity": "sha1-IpnwLG3tMNSllhsLn3RSShj2NPw=",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    }
-  },
+  "lockfileVersion": 1,
   "dependencies": {
     "accepts": {
       "version": "1.3.7",

From 26ea5bba7a0a37c5785d34be6586f154f1bebb0b Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 10:26:04 +0200
Subject: [PATCH 088/442] Some cleanup

---
 wasm/README.md                        | 6 +++---
 wasm/test_page/bergamot-httpserver.js | 4 ----
 wasm/test_page/bergamot.html          | 6 +++---
 3 files changed, 6 insertions(+), 10 deletions(-)

diff --git a/wasm/README.md b/wasm/README.md
index 131f9eb06..bb431447c 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -35,17 +35,17 @@ input.delete();
 
 You can also see everything in action by following the next steps:
 * Start the test webserver (ensure you have the latest nodejs installed)
-```
+```bash
 cd test_page
 bash start_server.sh
 ```
 * Open any of the browsers below
     * Firefox Nightly +87: make sure the following prefs are on (about:config)
-    ````
+    ```
     dom.postMessage.sharedArrayBuffer.bypassCOOP_COEP.insecure.enabled = true
     javascript.options.wasm_simd = true
     javascript.options.wasm_simd_wormhole = true
-    ````
+    ```
 
     * Chrome Canary +90: start with the following argument
     ```
diff --git a/wasm/test_page/bergamot-httpserver.js b/wasm/test_page/bergamot-httpserver.js
index f23b3e750..b28719fed 100644
--- a/wasm/test_page/bergamot-httpserver.js
+++ b/wasm/test_page/bergamot-httpserver.js
@@ -33,7 +33,3 @@ function serveFile(res, pathName, mime) {
 
 server.listen(8000);
 console.log('HTTP and BinaryJS server started on port 8000');
-
-
-
-
diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 49ca50e96..e7e1fe5b3 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -29,9 +29,9 @@
   <body>
 
   <div id="divradios">
-    <label for="radios">Choose the model to use</label>
-    <input type="radio" id="modellang" name="modellang" value="enes" checked><label for="es">English to Spanish</label>
-    <input type="radio" id="modellang" name="modellang" value="esen"><label for="en">Spanish to English</label>
+    <label>Choose the model to use</label>
+    <input type="radio" name="modellang" value="enes" checked/><label>English to Spanish</label>
+    <input type="radio" name="modellang" value="esen"/><label>Spanish to English</label>
     <input type="button" id="load" value="Load Model"/>
   </div>
 

From d3969bcd2d2430a4bf5f047d791eb768ba4cb013 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 10:34:57 +0200
Subject: [PATCH 089/442] Add support for translating multiple sentences on the
 test page + report words per second metric in the log

---
 wasm/test_page/bergamot.html | 42 +++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 13 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index e7e1fe5b3..d093208c2 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -37,10 +37,13 @@
 
   <div id="divtranslation">
     <label for="from">From</label>
-    <input type="text" id="from" name="from"/>
+    <textarea id="from" name="from">
+hello world
+the sky is blue
+    </textarea>
     <br><br>
     <label for="to">To</label>
-    <input type="text" id="to" name="to" readonly/>
+    <textarea id="to" name="to" readonly></textarea>
     <br><br>
     <input type="button" id="translate" value="Translate"/>
   </div>
@@ -65,17 +68,23 @@
         model = new Module.TranslationModel(modelConfig);
     }
 
-    const translate = (sentence) => {
+    const translate = (sentences) => {
 
         // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
         var request = new Module.TranslationRequest();
         let input = new Module.VectorString;
 
         // Initialize the input
-        input.push_back(sentence);
-        /*
+        sentences.forEach(sentence => {
+          // prevent empty sentences - it breaks the translation
+          if (sentence.trim() === "") {
+            return;
+          }
+          input.push_back(sentence.trim())
+        })
         // Access input (just for debugging)
         console.log('Input size=', input.size());
+        /*
         for (let i = 0; i < input.size(); i++) {
           console.log(' val:' + input.get(i));
         }
@@ -85,14 +94,14 @@
         let result = model.translate(input, request);
         // Access original and translated text from each entry of vector<TranslationResult>
         //console.log('Result size=', result.size(), ' - TimeDiff - ', (Date.now() - start)/1000);
-        let translatedText = "";
+        const translatedSentences = [];
         for (let i = 0; i < result.size(); i++) {
-          translatedText += result.get(i).getTranslatedText() + " ";
+          translatedSentences.push(result.get(i).getTranslatedText());
         }
-        console.log(translatedText);
+        console.log({translatedSentences});
         request.delete();
         input.delete();
-        return translatedText;
+        return translatedSentences;
   }
 
     document.querySelector("#load").addEventListener("click", () => {
@@ -105,10 +114,17 @@
 
     const translateCall = () => {
       const text = document.querySelector('#from').value;
-      let start = Date.now();
-      const translate_text = translate(text);
-      log(`sentence translation time ${(Date.now() - start)/1000} secs`);
-      document.querySelector('#to').value = translate_text;
+      const sentences = text.split("\n");
+      let wordCount = 0;
+      sentences.forEach(sentence => {
+        wordCount += sentence.trim().split(" ").length;
+      })
+      const start = Date.now();
+      const translatedSentences = translate(sentences);
+      const secs = (Date.now() - start) / 1000;
+      log(`Translation of ${translatedSentences.length} sentences (wordCount ${wordCount}) took ${secs} secs (${Math.round(wordCount / secs)} words per second)`);
+
+      document.querySelector('#to').value = translatedSentences.join("\n");
     }
 
     document.querySelector("#translate").addEventListener("click", () => {

From 28c0ab2e04f6e32b999aac0caa181cd914f92e30 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 10:37:37 +0200
Subject: [PATCH 090/442] Tweak words per second metric in the test page log

---
 wasm/test_page/bergamot.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index d093208c2..992d7585d 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -117,7 +117,7 @@
       const sentences = text.split("\n");
       let wordCount = 0;
       sentences.forEach(sentence => {
-        wordCount += sentence.trim().split(" ").length;
+        wordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
       })
       const start = Date.now();
       const translatedSentences = translate(sentences);

From a33b3a3bb5bcac9fe34135d671773a11554dce82 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 11:21:36 +0200
Subject: [PATCH 091/442] Add instructions on how to assemble and package the
 set of files expected by the test page

---
 README.md | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 333e758e3..4bff753d1 100644
--- a/README.md
+++ b/README.md
@@ -45,10 +45,25 @@ Download the models from `https://github.com/mozilla-applied-ml/bergamot-models`
 The build also allows packaging files into wasm binary (i.e. preloading in Emscripten’s virtual file system) using cmake
 option `PACKAGE_DIR`. The compile command below packages all the files in PATH directory (in these case, your models) into wasm binary.
 ```bash
-emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=<PATH> ./models
+emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=/repo/models ../
 ```
 Files packaged this way are preloaded in the root of the virtual file system.
 
+To package the set of files expected by the test page:
+
+```bash
+git clone https://github.com/browsermt/students
+cd students/esen/
+./download-models.sh
+cp esen.student.tiny11/lex.s2t ../../models/lex.esen.s2t
+cp esen.student.tiny11/model.npz ../../models/model.esen.npz
+cp esen.student.tiny11/vocab.esen.spm ../../models/vocab.esen.spm
+cd -
+cd students/enes/
+./download-models.sh
+cp enes.student.tiny11/lex.s2t ../../models/lex.enes.s2t
+cp enes.student.tiny11/model.npz ../../models/model.enes.npz
+```
 
 After Editing Files:
 

From 53e0b9fc5c219ae57d79be57acbec0dd580e89a8 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 11:22:23 +0200
Subject: [PATCH 092/442] Fix typo in lexical shortlist argument on test page

---
 wasm/test_page/bergamot.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 992d7585d..4ead87dbb 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -61,7 +61,7 @@
         // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
         // This example captures the most relevant options: model file, vocabulary files and shortlist file
         //        var modelConfig = "{\"models\":[\"/model.enes.npz\"],\"vocabs\":[\"/vocab.esen.spm\"],\"beam-size\":1}";//,\"shortlist\":[\"/lex.s2t\"]
-        const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1} ,\"shortlist\":[\"/lex.s2t\"]`;
+        const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1} ,\"shortlist\":[\"/lex.esen.s2t\"]`;
 
         // Instantiate the TranslationModel
         if (model) model.delete();

From e50dd0909f4709a6336b46a0baee175353ed0150 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 11:23:08 +0200
Subject: [PATCH 093/442] Ignore contents in models directory

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 59363a81c..6c301d661 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,3 +4,4 @@
 
 wasm/test_page/node_modules
 build-wasm
+models

From 7030fa015745070e0d7dc8ab6f0a5d25a1d95a78 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 11:25:13 +0200
Subject: [PATCH 094/442] Ignore test page bundled artifacts

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 6c301d661..d7d931f6e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -5,3 +5,4 @@
 wasm/test_page/node_modules
 build-wasm
 models
+wasm/test_page/bergamot-translator-worker.*

From 49ad6514aec6498e2a24a7dd96cff25d4e64ab5d Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 11:27:47 +0200
Subject: [PATCH 095/442] Add reproducible docker-based builds + let test page
 use these by default

---
 .gitignore                     |  2 +-
 docker/Makefile                | 55 ++++++++++++++++++++++++++++++++++
 docker/README.md               | 27 +++++++++++++++++
 docker/wasm/Dockerfile         | 36 ++++++++++++++++++++++
 wasm/test_page/start_server.sh |  8 ++---
 5 files changed, 123 insertions(+), 5 deletions(-)
 create mode 100644 docker/Makefile
 create mode 100644 docker/README.md
 create mode 100644 docker/wasm/Dockerfile

diff --git a/.gitignore b/.gitignore
index d7d931f6e..5a73aac90 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,6 +3,6 @@
 *.swo
 
 wasm/test_page/node_modules
-build-wasm
+build-*
 models
 wasm/test_page/bergamot-translator-worker.*
diff --git a/docker/Makefile b/docker/Makefile
new file mode 100644
index 000000000..583a58852
--- /dev/null
+++ b/docker/Makefile
@@ -0,0 +1,55 @@
+# -*- mode: makefile-gmake; indent-tabs-mode: true; tab-width: 4 -*-
+SHELL   		= bash
+PWD     		= $(shell pwd)
+WASM_IMAGE	    = local/bergamot-translator-build-wasm
+
+all: wasm-image compile-wasm
+
+# Build the Docker image for WASM builds
+wasm-image:
+	docker build -t local/bergamot-translator-build-wasm ./wasm/
+
+# Commands for compilation:
+cmake_cmd  = cmake
+
+wasm_cmake_cmd = ${cmake_cmd}
+wasm_cmake_cmd += -DCOMPILE_WASM=on
+wasm_cmake_cmd += -DProtobuf_INCLUDE_DIR=/usr/opt/protobuf-wasm-lib/dist/include
+wasm_cmake_cmd += -DProtobuf_LIBRARY=/usr/opt/protobuf-wasm-lib/dist/lib/libprotobuf.a
+wasm_cmake_cmd += -DPACKAGE_DIR=/repo/models
+
+make_cmd  = make
+#make_cmd += VERBOSE=1
+
+# ... and running things on Docker
+docker_mounts  = ${PWD}/..:/repo
+docker_mounts += ${HOME}/.ccache:/.ccache
+run_on_docker  = docker run --rm
+run_on_docker += $(addprefix -v, ${docker_mounts})
+run_on_docker += ${INTERACTIVE_DOCKER_SESSION}
+
+${HOME}/.ccache:
+	mkdir -p $@
+
+# Remove the bergamot-translator WASM build dir, forcing a clean compilation attempt
+clean-wasm: BUILD_DIR = /repo/build-wasm-docker
+clean-wasm: ${HOME}/.ccache
+	${run_on_docker} ${WASM_IMAGE} bash -c '(rm -rf ${BUILD_DIR} || true)'
+
+# Compile bergamot-translator to WASM
+compile-wasm: BUILD_DIR = /repo/build-wasm-docker
+compile-wasm: ${HOME}/.ccache
+	${run_on_docker} ${WASM_IMAGE} bash -c 'mkdir -p ${BUILD_DIR} && \
+cd ${BUILD_DIR} && \
+(emcmake ${wasm_cmake_cmd} .. && \
+(emmake ${make_cmd}) || \
+rm CMakeCache.txt)'
+
+# Start interactive shells for development / debugging purposes
+native-shell: INTERACTIVE_DOCKER_SESSION = -it
+native-shell:
+	${run_on_docker} ${NATIVE_IMAGE} bash
+
+wasm-shell: INTERACTIVE_DOCKER_SESSION = -it
+wasm-shell:
+	${run_on_docker} ${WASM_IMAGE} bash
diff --git a/docker/README.md b/docker/README.md
new file mode 100644
index 000000000..d98456a54
--- /dev/null
+++ b/docker/README.md
@@ -0,0 +1,27 @@
+## WASM
+
+Prepare docker image for WASM compilation:
+
+```bash
+make wasm-image
+```
+
+Compile to wasm:
+
+```bash
+make compile-wasm
+```
+
+## Debugging
+
+Remove the marian-decoder build dir, forcing the next compilation attempt to start from scratch:
+
+```bash
+make clean-wasm
+```
+
+Enter a docker container shell for manually running commands:
+
+```bash
+make wasm-shell
+```
diff --git a/docker/wasm/Dockerfile b/docker/wasm/Dockerfile
new file mode 100644
index 000000000..f309662a7
--- /dev/null
+++ b/docker/wasm/Dockerfile
@@ -0,0 +1,36 @@
+FROM emscripten/emsdk:2.0.9
+
+# Install specific version of CMake
+WORKDIR /usr
+RUN wget https://github.com/Kitware/CMake/releases/download/v3.17.2/cmake-3.17.2-Linux-x86_64.tar.gz -qO-\
+    | tar xzf - --strip-components 1
+
+# Install Python and Java (needed for Closure Compiler minification)
+RUN apt-get update \
+    && apt-get install -y \
+    python3 \
+    default-jre
+
+# Deps to compile protobuf from source + the protoc binary which we need natively
+RUN apt-get update -y  && apt-get --no-install-recommends -y install \
+    protobuf-compiler \
+    autoconf \
+    autotools-dev \
+    automake \
+    autogen \
+    libtool && ln -s /usr/bin/libtoolize /usr/bin/libtool \
+    && mkdir -p /usr/opt \
+    && cd /usr/opt \
+    && git clone https://github.com/menduz/protobuf-wasm-lib
+
+RUN cd /usr/opt/protobuf-wasm-lib \
+    && /bin/bash -c "BRANCH=v3.6.1 ./prepare.sh"
+RUN cd /usr/opt/protobuf-wasm-lib/protobuf \
+    && bash -x ../build.sh
+RUN cp /usr/bin/protoc /usr/opt/protobuf-wasm-lib/dist/bin/protoc
+
+RUN apt-get --no-install-recommends -y install \
+    libprotobuf-dev
+
+# Necessary for benchmarking
+RUN pip3 install sacrebleu
diff --git a/wasm/test_page/start_server.sh b/wasm/test_page/start_server.sh
index b83344b8a..b0b5be1b2 100644
--- a/wasm/test_page/start_server.sh
+++ b/wasm/test_page/start_server.sh
@@ -1,8 +1,8 @@
 #!/bin/bash
 
-cp ../../build-wasm/wasm/bergamot-translator-worker.data .
-cp ../../build-wasm/wasm/bergamot-translator-worker.js .
-cp ../../build-wasm/wasm/bergamot-translator-worker.wasm .
-cp ../../build-wasm/wasm/bergamot-translator-worker.worker.js .
+cp ../../build-wasm-docker/wasm/bergamot-translator-worker.data .
+cp ../../build-wasm-docker/wasm/bergamot-translator-worker.js .
+cp ../../build-wasm-docker/wasm/bergamot-translator-worker.wasm .
+cp ../../build-wasm-docker/wasm/bergamot-translator-worker.worker.js .
 npm install
 node bergamot-httpserver.js
\ No newline at end of file

From 77f39545f314c7a931c91aef0a11e871ff5a880c Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 11:30:45 +0200
Subject: [PATCH 096/442] Add time it takes to arrive to preRun to test page

---
 wasm/test_page/bergamot.html | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 4ead87dbb..7b38cc22f 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -141,13 +141,15 @@
       document.querySelector("#log").value += message + "\n";
     }
 
+    const start = Date.now();
     let moduleLoadStart;
     var Module = {
       preRun: [function() {
+        log(`Time until Module.preRun: ${(Date.now() - start)/1000} secs`);
         moduleLoadStart = Date.now();
       }],
       onRuntimeInitialized: function() {
-        log(`Wasm Runtime initialized in ${(Date.now() - moduleLoadStart)/1000} secs`);
+        log(`Wasm Runtime initialized (preRun -> onRuntimeInitialized) in ${(Date.now() - moduleLoadStart)/1000} secs`);
       }
     };
   </script>

From dbdcdab1153be9891e2a44aa308b29c0141349aa Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 11:59:03 +0200
Subject: [PATCH 097/442] Avoid use of unsafe eval in glue code

---
 wasm/CMakeLists.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 40b08bf6a..837515837 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -14,7 +14,7 @@ target_include_directories(bergamot-translator-worker
 target_compile_definitions(bergamot-translator-worker PRIVATE WASM_BINDINGS)
 target_compile_options(bergamot-translator-worker PRIVATE ${WASM_COMPILE_FLAGS})
 
-set(LINKER_FLAGS "--bind -s ASSERTIONS=1 -s DISABLE_EXCEPTION_CATCHING=0 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1")
+set(LINKER_FLAGS "--bind -s ASSERTIONS=1 -s DISABLE_EXCEPTION_CATCHING=0 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1")
 if (NOT PACKAGE_DIR STREQUAL "")
   set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${PACKAGE_DIR}@/")
 endif()

From 70bdcd436571de532ea202d95edf7cccf9505bb4 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 12:54:32 +0200
Subject: [PATCH 098/442] Fix typo from when fixing typo

---
 wasm/test_page/bergamot.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 7b38cc22f..e5d7a90b3 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -61,7 +61,7 @@
         // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
         // This example captures the most relevant options: model file, vocabulary files and shortlist file
         //        var modelConfig = "{\"models\":[\"/model.enes.npz\"],\"vocabs\":[\"/vocab.esen.spm\"],\"beam-size\":1}";//,\"shortlist\":[\"/lex.s2t\"]
-        const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1} ,\"shortlist\":[\"/lex.esen.s2t\"]`;
+        const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1} ,\"shortlist\":[\"/lex.${lang}.s2t\"]`;
 
         // Instantiate the TranslationModel
         if (model) model.delete();

From da56501c4f255d9bc57c2d244e0979c29676ad3f Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 13:10:10 +0200
Subject: [PATCH 099/442] Finally found the original typo that made it appear
 as if loading the model in the test page was faster than elsewhere - the
 lexical shortlist was not being included at the right place in the model
 config

---
 wasm/test_page/bergamot.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index e5d7a90b3..6985cee89 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -61,7 +61,7 @@
         // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
         // This example captures the most relevant options: model file, vocabulary files and shortlist file
         //        var modelConfig = "{\"models\":[\"/model.enes.npz\"],\"vocabs\":[\"/vocab.esen.spm\"],\"beam-size\":1}";//,\"shortlist\":[\"/lex.s2t\"]
-        const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1} ,\"shortlist\":[\"/lex.${lang}.s2t\"]`;
+        const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1,\"shortlist\":[\"/lex.${lang}.s2t\"]}`;
 
         // Instantiate the TranslationModel
         if (model) model.delete();

From 1e94d78c4d2b6bb9b763c16c59b0178a8458e18f Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 13:19:39 +0200
Subject: [PATCH 100/442] Formatting

---
 wasm/test_page/bergamot.html | 228 +++++++++++++++++------------------
 1 file changed, 114 insertions(+), 114 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 6985cee89..541da1580 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -1,41 +1,41 @@
 <!doctype html>
 <html>
-  <head>
+<head>
     <link rel="icon" href="data:,">
     <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1">
-  </head>
-  <style>
+</head>
+<style>
     body, html, div {
-      margin-left:1%;
-      margin-right:1%;
-      margin-bottom:1%;
-      margin-top:1%;
-      padding-left:1%;
-      padding-right:1%;
-      padding-bottom:1%;
-      padding-top:1%;
-      }
+        margin-left: 1%;
+        margin-right: 1%;
+        margin-bottom: 1%;
+        margin-top: 1%;
+        padding-left: 1%;
+        padding-right: 1%;
+        padding-bottom: 1%;
+        padding-top: 1%;
+    }
 
-      textarea, #to, #from {
+    textarea, #to, #from {
         width: 100%;
         max-width: 100%;
-      }
+    }
 
-      div {
+    div {
         float: left;
         width: 80%;
-      }
-  </style>
-  <body>
+    }
+</style>
+<body>
 
-  <div id="divradios">
+<div id="divradios">
     <label>Choose the model to use</label>
     <input type="radio" name="modellang" value="enes" checked/><label>English to Spanish</label>
     <input type="radio" name="modellang" value="esen"/><label>Spanish to English</label>
     <input type="button" id="load" value="Load Model"/>
-  </div>
+</div>
 
-  <div id="divtranslation">
+<div id="divtranslation">
     <label for="from">From</label>
     <textarea id="from" name="from">
 hello world
@@ -46,113 +46,113 @@
     <textarea id="to" name="to" readonly></textarea>
     <br><br>
     <input type="button" id="translate" value="Translate"/>
-  </div>
+</div>
 
-  <div id="divlog">
+<div id="divlog">
     <label for="log">Log:</label><br>
     <textarea id="log" name="log" rows="50" cols="75"></textarea>
-  </div>
+</div>
 
-  <script>
+<script>
 
-    var model, request, input = undefined;
-    const loadModel = (lang) => {
-        // Set the Model Configuration as YAML formatted string.
-        // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-        // This example captures the most relevant options: model file, vocabulary files and shortlist file
-        //        var modelConfig = "{\"models\":[\"/model.enes.npz\"],\"vocabs\":[\"/vocab.esen.spm\"],\"beam-size\":1}";//,\"shortlist\":[\"/lex.s2t\"]
-        const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1,\"shortlist\":[\"/lex.${lang}.s2t\"]}`;
+  var model, request, input = undefined;
+  const loadModel = (lang) => {
+    // Set the Model Configuration as YAML formatted string.
+    // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
+    // This example captures the most relevant options: model file, vocabulary files and shortlist file
+    //        var modelConfig = "{\"models\":[\"/model.enes.npz\"],\"vocabs\":[\"/vocab.esen.spm\"],\"beam-size\":1}";//,\"shortlist\":[\"/lex.s2t\"]
+    const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1,\"shortlist\":[\"/lex.${lang}.s2t\"]}`;
 
-        // Instantiate the TranslationModel
-        if (model) model.delete();
-        model = new Module.TranslationModel(modelConfig);
+    // Instantiate the TranslationModel
+    if (model) model.delete();
+    model = new Module.TranslationModel(modelConfig);
+  }
+
+  const translate = (sentences) => {
+
+    // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
+    var request = new Module.TranslationRequest();
+    let input = new Module.VectorString;
+
+    // Initialize the input
+    sentences.forEach(sentence => {
+      // prevent empty sentences - it breaks the translation
+      if (sentence.trim() === "") {
+        return;
+      }
+      input.push_back(sentence.trim())
+    })
+    // Access input (just for debugging)
+    console.log('Input size=', input.size());
+    /*
+    for (let i = 0; i < input.size(); i++) {
+      console.log(' val:' + input.get(i));
     }
+    */
+
+    // Translate the input; the result is a vector<TranslationResult>
+    let result = model.translate(input, request);
+    // Access original and translated text from each entry of vector<TranslationResult>
+    //console.log('Result size=', result.size(), ' - TimeDiff - ', (Date.now() - start)/1000);
+    const translatedSentences = [];
+    for (let i = 0; i < result.size(); i++) {
+      translatedSentences.push(result.get(i).getTranslatedText());
+    }
+    console.log({ translatedSentences });
+    request.delete();
+    input.delete();
+    return translatedSentences;
+  }
+
+  document.querySelector("#load").addEventListener("click", () => {
+    const lang = document.querySelector('input[name="modellang"]:checked').value;
+    let start = Date.now();
+    loadModel(lang)
+    log(`model ${lang} loaded in ${(Date.now() - start) / 1000} secs`);
+    //log('Model Alignment:', model.isAlignmentSupported());
+  });
+
+  const translateCall = () => {
+    const text = document.querySelector('#from').value;
+    const sentences = text.split("\n");
+    let wordCount = 0;
+    sentences.forEach(sentence => {
+      wordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
+    })
+    const start = Date.now();
+    const translatedSentences = translate(sentences);
+    const secs = (Date.now() - start) / 1000;
+    log(`Translation of ${translatedSentences.length} sentences (wordCount ${wordCount}) took ${secs} secs (${Math.round(wordCount / secs)} words per second)`);
 
-    const translate = (sentences) => {
-
-        // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
-        var request = new Module.TranslationRequest();
-        let input = new Module.VectorString;
-
-        // Initialize the input
-        sentences.forEach(sentence => {
-          // prevent empty sentences - it breaks the translation
-          if (sentence.trim() === "") {
-            return;
-          }
-          input.push_back(sentence.trim())
-        })
-        // Access input (just for debugging)
-        console.log('Input size=', input.size());
-        /*
-        for (let i = 0; i < input.size(); i++) {
-          console.log(' val:' + input.get(i));
-        }
-        */
-
-        // Translate the input; the result is a vector<TranslationResult>
-        let result = model.translate(input, request);
-        // Access original and translated text from each entry of vector<TranslationResult>
-        //console.log('Result size=', result.size(), ' - TimeDiff - ', (Date.now() - start)/1000);
-        const translatedSentences = [];
-        for (let i = 0; i < result.size(); i++) {
-          translatedSentences.push(result.get(i).getTranslatedText());
-        }
-        console.log({translatedSentences});
-        request.delete();
-        input.delete();
-        return translatedSentences;
+    document.querySelector('#to').value = translatedSentences.join("\n");
   }
 
-    document.querySelector("#load").addEventListener("click", () => {
-      const lang = document.querySelector('input[name="modellang"]:checked').value;
-      let start = Date.now();
-      loadModel(lang)
-      log(`model ${lang} loaded in ${(Date.now() - start)/1000} secs`);
-      //log('Model Alignment:', model.isAlignmentSupported());
-    });
-
-    const translateCall = () => {
-      const text = document.querySelector('#from').value;
-      const sentences = text.split("\n");
-      let wordCount = 0;
-      sentences.forEach(sentence => {
-        wordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
-      })
-      const start = Date.now();
-      const translatedSentences = translate(sentences);
-      const secs = (Date.now() - start) / 1000;
-      log(`Translation of ${translatedSentences.length} sentences (wordCount ${wordCount}) took ${secs} secs (${Math.round(wordCount / secs)} words per second)`);
-
-      document.querySelector('#to').value = translatedSentences.join("\n");
-    }
+  document.querySelector("#translate").addEventListener("click", () => {
+    translateCall();
+  });
 
-    document.querySelector("#translate").addEventListener("click", () => {
+  document.querySelector("#from").addEventListener('keyup', function(event) {
+    if (event.keyCode === 13) {
       translateCall();
-    });
+    }
+  });
 
-    document.querySelector("#from").addEventListener('keyup', function (event) {
-      if (event.keyCode === 13) {
-        translateCall();
-      }
-   });
+  const log = (message) => {
+    document.querySelector("#log").value += message + "\n";
+  }
 
-    const log = (message) => {
-      document.querySelector("#log").value += message + "\n";
+  const start = Date.now();
+  let moduleLoadStart;
+  var Module = {
+    preRun: [function() {
+      log(`Time until Module.preRun: ${(Date.now() - start) / 1000} secs`);
+      moduleLoadStart = Date.now();
+    }],
+    onRuntimeInitialized: function() {
+      log(`Wasm Runtime initialized (preRun -> onRuntimeInitialized) in ${(Date.now() - moduleLoadStart) / 1000} secs`);
     }
-
-    const start = Date.now();
-    let moduleLoadStart;
-    var Module = {
-      preRun: [function() {
-        log(`Time until Module.preRun: ${(Date.now() - start)/1000} secs`);
-        moduleLoadStart = Date.now();
-      }],
-      onRuntimeInitialized: function() {
-        log(`Wasm Runtime initialized (preRun -> onRuntimeInitialized) in ${(Date.now() - moduleLoadStart)/1000} secs`);
-      }
-    };
-  </script>
-  <script src="bergamot-translator-worker.js"></script>
+  };
+</script>
+<script src="bergamot-translator-worker.js"></script>
 </body>
 </html>

From fcc998ffa4c2468baed11889951685ff0b923cf7 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 13:30:07 +0200
Subject: [PATCH 101/442] Add 10 lines of esen benchmark sentences to test page

---
 wasm/test_page/bergamot.html | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 541da1580..cbd266567 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -30,16 +30,24 @@
 
 <div id="divradios">
     <label>Choose the model to use</label>
-    <input type="radio" name="modellang" value="enes" checked/><label>English to Spanish</label>
-    <input type="radio" name="modellang" value="esen"/><label>Spanish to English</label>
+    <input type="radio" name="modellang" value="enes"/><label>English to Spanish</label>
+    <input type="radio" name="modellang" value="esen" checked/><label>Spanish to English</label>
     <input type="button" id="load" value="Load Model"/>
 </div>
 
 <div id="divtranslation">
     <label for="from">From</label>
     <textarea id="from" name="from">
-hello world
-the sky is blue
+Una estrategia republicana para obstaculizar la reelecci�n de Obama
+Los dirigentes republicanos justificaron su pol�tica por la necesidad de luchar contra el fraude electoral.
+Ahora bien, el Centro Brennan considera esto �ltimo un mito y afirma que el fraude electoral es menos frecuente en los Estados Unidos que el n�mero de personas que mueren a causa de la ca�da de un rayo.
+De hecho, los abogados republicanos no han encontrado m�s que 300 casos de fraude electoral en los Estados Unidos en diez a�os.
+Una cosa es cierta: esas nuevas disposiciones afectar�n negativamente a la tasa de participaci�n.
+En ese sentido, estas medidas minar�n en parte el sistema democr�tico americano.
+Al contrario de lo que ocurre en Canad�, los estados americanos son responsables de la organizaci�n de las elecciones federales en los Estados Unidos.
+Y en esa misma l�nea una mayor�a de los gobiernos americanos promulgaron, a partir de 2009, nuevas leyes que dificultaban el proceso de inscripci�n o de votaci�n.
+Este fen�meno se ha extendido tras las elecciones de noviembre de 2010, que vieron el aumento de 675 nuevos representantes republicanos en 26 estados.
+En consecuencia, durante el a�o 2011 se introdujeron 180 proyectos de ley que restring�an el ejercicio del derecho de voto en 41 estados.
     </textarea>
     <br><br>
     <label for="to">To</label>

From f3ff1d29ae4d6d036f68bc993420c36015f10b09 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 13:30:46 +0200
Subject: [PATCH 102/442] Make modelConfig an object instead of string (less
 likelihood of typos)

---
 wasm/test_page/bergamot.html | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index cbd266567..0de9925e7 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -67,13 +67,25 @@
   const loadModel = (lang) => {
     // Set the Model Configuration as YAML formatted string.
     // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-    // This example captures the most relevant options: model file, vocabulary files and shortlist file
-    //        var modelConfig = "{\"models\":[\"/model.enes.npz\"],\"vocabs\":[\"/vocab.esen.spm\"],\"beam-size\":1}";//,\"shortlist\":[\"/lex.s2t\"]
-    const modelConfig = `{\"models\":[\"/model.${lang}.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"beam-size\":1,\"shortlist\":[\"/lex.${lang}.s2t\"]}`;
+
+    const modelConfig = {
+      "models": [
+        `/model.${lang}.npz`
+      ],
+      "vocabs": [
+        "/vocab.esen.spm",
+        "/vocab.esen.spm"
+      ],
+      "shortlist": [
+        `/lex.${lang}.s2t`,
+        50,
+        50,
+      ]
+    };
 
     // Instantiate the TranslationModel
     if (model) model.delete();
-    model = new Module.TranslationModel(modelConfig);
+    model = new Module.TranslationModel(JSON.stringify(modelConfig));
   }
 
   const translate = (sentences) => {

From 7d6346d3b0b000f281e99f972bb2fe663b93b27f Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 13:35:22 +0200
Subject: [PATCH 103/442] Add model config used in pr6 benchmarks

---
 wasm/test_page/bergamot.html | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 0de9925e7..9322368ef 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -76,11 +76,22 @@
         "/vocab.esen.spm",
         "/vocab.esen.spm"
       ],
+      "beam-size": 1,
+      "mini-batch": 32,
+      "maxi-batch": 100,
+      "maxi-batch-sort": "src",
+      "workspace": 128,
+      "skip-cost": true,
+      "cpu-threads": 1,
       "shortlist": [
         `/lex.${lang}.s2t`,
         50,
         50,
       ]
+      // TODO: Enable when wormhole is enabled
+      // "int8shift": true,
+      // TODO: Enable when loading of binary models is supported and we use model.intgemm.alphas.bin
+      // "int8shiftAlphaAll": true,
     };
 
     // Instantiate the TranslationModel

From 64d57d8aa089957f5c8ffe88f7ce805de0423e6e Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 13:50:59 +0200
Subject: [PATCH 104/442] Use yaml for modelConfig on test page

---
 wasm/test_page/bergamot.html | 51 +++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 27 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 9322368ef..04ff5aeb9 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -67,36 +67,33 @@
   const loadModel = (lang) => {
     // Set the Model Configuration as YAML formatted string.
     // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-
-    const modelConfig = {
-      "models": [
-        `/model.${lang}.npz`
-      ],
-      "vocabs": [
-        "/vocab.esen.spm",
-        "/vocab.esen.spm"
-      ],
-      "beam-size": 1,
-      "mini-batch": 32,
-      "maxi-batch": 100,
-      "maxi-batch-sort": "src",
-      "workspace": 128,
-      "skip-cost": true,
-      "cpu-threads": 1,
-      "shortlist": [
-        `/lex.${lang}.s2t`,
-        50,
-        50,
-      ]
-      // TODO: Enable when wormhole is enabled
-      // "int8shift": true,
-      // TODO: Enable when loading of binary models is supported and we use model.intgemm.alphas.bin
-      // "int8shiftAlphaAll": true,
-    };
+    const modelConfig = `models:
+  - /model.${lang}.npz
+vocabs:
+  - /vocab.esen.spm
+  - /vocab.esen.spm
+beam-size: 1
+normalize: 1.0
+word-penalty: 0
+mini-batch: 32
+maxi-batch: 100
+maxi-batch-sort: src
+workspace: 128
+max-length-factor: 2.0
+skip-cost: true
+shortlist:
+    - lex.${lang}.s2t
+    - 50
+    - 50
+`;
+// TODO: Use in model config when wormhole is enabled:
+// gemm-precision: int8shift
+// TODO: Use in model config when loading of binary models is supported and we use model.intgemm.alphas.bin:
+// gemm-precision: int8shiftAlphaAll
 
     // Instantiate the TranslationModel
     if (model) model.delete();
-    model = new Module.TranslationModel(JSON.stringify(modelConfig));
+    model = new Module.TranslationModel(modelConfig);
   }
 
   const translate = (sentences) => {

From 3dd7a60b3511e5ebc09169f33d37913834e83a1d Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 15 Feb 2021 12:50:40 +0100
Subject: [PATCH 105/442] Enabled simd shuffle pattern for intgemm compilation

 - WORMHOLE cmake option is set to ON when compiling for WASM
 - WASM module might not run on Chrome
---
 CMakeLists.txt | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 677963f12..ccaf65224 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -23,6 +23,10 @@ SET(COMPILE_DECODER_ONLY ON CACHE BOOL "Compile marian-decoder only")
 SET(COMPILE_WITH_PTHREADS OFF CACHE BOOL "Compile with pthreads support")
 SET(USE_WASM_COMPATIBLE_BLAS ON CACHE BOOL "Compile with a WASM compatible blas for decoder only builds")
 SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
+if(COMPILE_WASM)
+  # Set WORMHOLE to ON for marian whenever compiling for wasm platform
+  SET(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
+endif()
 
 execute_process(COMMAND git submodule update --init --recursive --no-fetch
                 WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})

From 91e45cb4f08a1b9f59757c82c61fbd5b86d88915 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 13:58:12 +0200
Subject: [PATCH 106/442] Prepend shortlist path with /

---
 wasm/test_page/bergamot.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 04ff5aeb9..8fc7824e1 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -82,7 +82,7 @@
 max-length-factor: 2.0
 skip-cost: true
 shortlist:
-    - lex.${lang}.s2t
+    - /lex.${lang}.s2t
     - 50
     - 50
 `;

From 9a5ae9568e50856d520839854dc00ee2662b2d04 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 14:24:59 +0200
Subject: [PATCH 107/442] Turn of assertions and disable exception catching for
 wasm builds

---
 CMakeLists.txt      | 2 +-
 wasm/CMakeLists.txt | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ccaf65224..8044cb08a 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -38,7 +38,7 @@ endif()
 
 if(COMPILE_WASM)
   list(APPEND WASM_COMPILE_FLAGS -pthread -O3 -g2 -fPIC -mssse3 -msimd128)
-  list(APPEND WASM_COMPILE_FLAGS "SHELL:-s WASM=1" "SHELL:-s ASSERTIONS=1" "SHELL:-s DISABLE_EXCEPTION_CATCHING=0" "SHELL:-s LLD_REPORT_UNDEFINED" "SHELL:-s FORCE_FILESYSTEM=1" "SHELL:-s ALLOW_MEMORY_GROWTH=1")
+  list(APPEND WASM_COMPILE_FLAGS "SHELL:-s WASM=1" "SHELL:-s ASSERTIONS=0" "SHELL:-s DISABLE_EXCEPTION_CATCHING=1" "SHELL:-s LLD_REPORT_UNDEFINED" "SHELL:-s FORCE_FILESYSTEM=1" "SHELL:-s ALLOW_MEMORY_GROWTH=1")
   list(APPEND WASM_COMPILE_FLAGS -Wno-error=pthreads-mem-growth)
 endif(COMPILE_WASM)
 
diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 837515837..748762d14 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -14,7 +14,7 @@ target_include_directories(bergamot-translator-worker
 target_compile_definitions(bergamot-translator-worker PRIVATE WASM_BINDINGS)
 target_compile_options(bergamot-translator-worker PRIVATE ${WASM_COMPILE_FLAGS})
 
-set(LINKER_FLAGS "--bind -s ASSERTIONS=1 -s DISABLE_EXCEPTION_CATCHING=0 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1")
+set(LINKER_FLAGS "--bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1")
 if (NOT PACKAGE_DIR STREQUAL "")
   set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${PACKAGE_DIR}@/")
 endif()

From 9a5cf30bbbdee83d98e933ee122aed00b26b161a Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Mon, 15 Feb 2021 15:03:00 +0200
Subject: [PATCH 108/442] Revert "Enabled simd shuffle pattern for intgemm
 compilation"

This reverts commit 3dd7a60b3511e5ebc09169f33d37913834e83a1d.
---
 CMakeLists.txt | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 8044cb08a..108338411 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -23,10 +23,6 @@ SET(COMPILE_DECODER_ONLY ON CACHE BOOL "Compile marian-decoder only")
 SET(COMPILE_WITH_PTHREADS OFF CACHE BOOL "Compile with pthreads support")
 SET(USE_WASM_COMPATIBLE_BLAS ON CACHE BOOL "Compile with a WASM compatible blas for decoder only builds")
 SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
-if(COMPILE_WASM)
-  # Set WORMHOLE to ON for marian whenever compiling for wasm platform
-  SET(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
-endif()
 
 execute_process(COMMAND git submodule update --init --recursive --no-fetch
                 WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})

From ca6ca154b9ee74899f1a801a8a3c91972ca10043 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 15 Feb 2021 15:22:31 +0000
Subject: [PATCH 109/442] Changing fn name from enqueue to produceTo(pcqueue)

---
 src/translator/batcher.cpp | 2 +-
 src/translator/batcher.h   | 2 +-
 src/translator/service.cpp | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 5fdcc3ac6..9ba0d035f 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -62,7 +62,7 @@ void Batcher::addWholeRequest(Ptr<Request> request) {
   }
 }
 
-void Batcher::enqueue(PCQueue<Batch> &pcqueue) {
+void Batcher::produceTo(PCQueue<Batch> &pcqueue) {
   Batch batch;
   while (cleaveBatch(batch)) {
     pcqueue.ProduceSwap(batch);
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index d6b85f3f3..342725708 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -21,7 +21,7 @@ class Batcher {
   // which maintains priority among sentences from multiple concurrent requests.
   void addSentenceWithPriority(RequestSentence &sentence);
   void addWholeRequest(Ptr<Request> request);
-  void enqueue(PCQueue<Batch> &pcqueue);
+  void produceTo(PCQueue<Batch> &pcqueue);
 
   // Loads sentences with sentences compiled from (tentatively) multiple
   // requests optimizing for both padding and priority.
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 1b33558e7..96f391c2d 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -70,7 +70,7 @@ std::future<Response> Service::translate(std::string &&input) {
   batcher_.addWholeRequest(request);
 
   if (numWorkers_ > 0) {
-    batcher_.enqueue(pcqueue_);
+    batcher_.produceTo(pcqueue_);
   } else {
     // Queue single-threaded
     Batch batch;

From 0374ac4696b124ed9e015325aef3c1501a514736 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 15 Feb 2021 14:28:06 +0100
Subject: [PATCH 110/442] Updated marian submodule

 - Includes try/catch free builds
 - Has ASSERTION=0 and DISABLE_EXCEPTION_CATCHING=1 for wasm builds
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 29ecba1cb..467c43a29 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 29ecba1cb1b8ea26ae582d3851e214769b89e566
+Subproject commit 467c43a292a68b7913af2a00d353de97c1740f92

From 3607523c24ca69fa3b195f1aae1aaf0c0bb44f65 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 15 Feb 2021 16:54:50 +0100
Subject: [PATCH 111/442] Enabled COMPILE_WITHOUT_EXCEPTIONS for marian
 submodule

---
 CMakeLists.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 108338411..a2aec07a3 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -23,6 +23,7 @@ SET(COMPILE_DECODER_ONLY ON CACHE BOOL "Compile marian-decoder only")
 SET(COMPILE_WITH_PTHREADS OFF CACHE BOOL "Compile with pthreads support")
 SET(USE_WASM_COMPATIBLE_BLAS ON CACHE BOOL "Compile with a WASM compatible blas for decoder only builds")
 SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
+SET(COMPILE_WITHOUT_EXCEPTIONS ON CACHE BOOL "Compile without exceptions")
 
 execute_process(COMMAND git submodule update --init --recursive --no-fetch
                 WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})

From c5c5339489d6d209271f76ac2f53ce7ac92fa7c0 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 15 Feb 2021 17:18:59 +0100
Subject: [PATCH 112/442] Re-enable simd shuffle pattern for intgemm
 compilation

---
 CMakeLists.txt | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index a2aec07a3..8d1ff1b52 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -24,6 +24,10 @@ SET(COMPILE_WITH_PTHREADS OFF CACHE BOOL "Compile with pthreads support")
 SET(USE_WASM_COMPATIBLE_BLAS ON CACHE BOOL "Compile with a WASM compatible blas for decoder only builds")
 SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
 SET(COMPILE_WITHOUT_EXCEPTIONS ON CACHE BOOL "Compile without exceptions")
+if(COMPILE_WASM)
+  # Set WORMHOLE to ON for marian whenever compiling for wasm platform
+  SET(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
+endif()
 
 execute_process(COMMAND git submodule update --init --recursive --no-fetch
                 WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})

From d5a5e754510aeb158fea3e82939426e4d29885ed Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 15 Feb 2021 20:21:10 +0000
Subject: [PATCH 113/442] Renaming variables; Enhancing documentation

---
 src/translator/request.cpp |  45 ++++++++++-
 src/translator/request.h   | 148 ++++++++++++++++++++++---------------
 src/translator/service.cpp |   6 +-
 3 files changed, 133 insertions(+), 66 deletions(-)

diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 9317f697e..303f9cc7d 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -10,14 +10,15 @@
 namespace marian {
 namespace bergamot {
 
+// -----------------------------------------------------------------
 Request::Request(unsigned int Id, int lineNumberBegin,
                  std::vector<Ptr<Vocab const>> &vocabs, std::string &&source,
                  Segments &&segments,
-                 std::vector<TokenRanges> &&sourceAlignments,
+                 std::vector<TokenRanges> &&sourceTokenRanges,
                  std::promise<Response> responsePromise)
     : Id_(Id), lineNumberBegin_(lineNumberBegin), vocabs_(&vocabs),
       source_(std::move(source)), segments_(std::move(segments)),
-      sourceAlignments_(std::move(sourceAlignments)),
+      sourceTokenRanges_(std::move(sourceTokenRanges)),
       response_(std::move(responsePromise)) {
 
   counter_ = segments_.size();
@@ -48,7 +49,7 @@ void Request::processHistory(size_t index, Ptr<History> history) {
 void Request::completeRequest() {
   // Request no longer needs to hold the content, can transfer it to
   // Response.
-  Response response(std::move(source_), std::move(sourceAlignments_),
+  Response response(std::move(source_), std::move(sourceTokenRanges_),
                     std::move(histories_), *vocabs_);
   response_.set_value(std::move(response));
 }
@@ -58,6 +59,8 @@ bool Request::operator<(const Request &b) const {
   return Id_ < b.Id_;
 }
 
+// ------------------------------------------------------------------
+
 RequestSentence::RequestSentence(size_t index, Ptr<Request> request)
     : index_(index), request_(request) {}
 
@@ -87,5 +90,41 @@ bool operator<(const RequestSentence &a, const RequestSentence &b) {
   return a.request_ < b.request_;
 }
 
+// ----------------------------------------------------------------------
+
+void Batch::reset() {
+  Id_ = 0;
+  sentences_.clear();
+}
+
+void Batch::log() {
+  int numTokens{0}, maxLength{0};
+  for (auto &sentence : sentences_) {
+    numTokens += sentence.numTokens();
+    maxLength = std::max(maxLength, static_cast<int>(sentence.numTokens()));
+  }
+
+  LOG(info, "Batch(Id_={}, tokens={}, max-length={}, sentences_={})", Id_,
+      numTokens, maxLength, sentences_.size());
+}
+
+void Batch::add(const RequestSentence &sentence) {
+  sentences_.push_back(sentence);
+}
+
+void Batch::setId(int Id) {
+  assert(Id > 0);
+  Id_ = Id;
+  if (Id % 500 == 0) {
+    log();
+  }
+}
+
+void Batch::completeBatch(const Histories &histories) {
+  for (int i = 0; i < sentences_.size(); i++) {
+    sentences_[i].completeSentence(histories[i]);
+  }
+}
+
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/request.h b/src/translator/request.h
index 8912a497d..095a03ccd 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -3,20 +3,19 @@
 //
 // Request: holds the input blob of a text, Segments (vector<Words>) which are
 // to go to the batching mechanism and alignments between the processed
-// segments and the input blob (sourceAlignments). In addition, Request takes
+// segments and the input blob (sourceTokenRanges). In addition, Request takes
 // care of the barrier which fires when all the Segments in a request are done
-// translating by the workers (BatchTranslator). Request is to be extended with
-// notions of Priority (sequence, user-given).
+// translating by the workers (BatchTranslator).
+// TODO(jerinphilip):  Extend Request with notions of Priority (sequence,
+// user-given).
 //
-// RequestSentence: is a tuple of (index, Request*). This provides the
+// RequestSentence: is a tuple of (index, Ptr<Request>). This provides the
 // batching mechanism access to the segment within the request. The backref to
 // Request allows event triggering the barrier upon completion of the last
 // sentence by a worker.
 //
-// PCItem: is a vector of RequestSentences and a batchNumber, which is what the
-// PCQueue holds. The batches are constructed from segments returned by a
-// RequestSentence. Can be enhanced with paddingSize, countTokens eventually for
-// logging.
+// Batch: is a vector of RequestSentences tagged with a batchNumber, which is
+// what the PCQueue holds. Batch is "produced" by the Batcher.
 
 #ifndef SRC_BERGAMOT_REQUEST_H_
 #define SRC_BERGAMOT_REQUEST_H_
@@ -37,23 +36,10 @@ namespace marian {
 namespace bergamot {
 
 class Request {
-private:
-  unsigned int Id_;
-  int lineNumberBegin_;
-  std::string source_;
-  std::atomic<int> counter_;
-  std::vector<Ptr<Vocab const>> *vocabs_;
-
-  Segments segments_;
-  std::vector<TokenRanges> sourceAlignments_;
-  std::vector<Ptr<History>> histories_;
-
-  std::promise<Response> response_;
-
 public:
   Request(unsigned int Id, int lineNumberBegin,
           std::vector<Ptr<Vocab const>> &vocabs_, std::string &&source,
-          Segments &&segments, std::vector<TokenRanges> &&sourceAlignments,
+          Segments &&segments, std::vector<TokenRanges> &&sourceTokenRanges,
           std::promise<Response> responsePromise);
 
   // Obtain the count of tokens in the segment correponding to index. Used to
@@ -68,7 +54,8 @@ class Request {
   // several requests.
   Segment getSegment(size_t index) const;
 
-  // For notions of priority among requests (used to enable <set> in Batcher).
+  // For notions of priority among requests, used to enable std::set in
+  // Batcher.
   bool operator<(const Request &request) const;
 
   // Processes a history obtained after translating in a heterogenous batch
@@ -77,20 +64,60 @@ class Request {
 
   // On completion of last segment, sets value of the promise.
   void completeRequest();
+
+private:
+  unsigned int Id_;
+  int lineNumberBegin_;
+
+  // Multiple translation-workers can concurrently access the same Request. The
+  // following atomic atomically operates on the variable holding sentences
+  // remaining to be translated.
+  std::atomic<int> counter_;
+
+  // source_ holds the source string to be translated. segments_ hold the
+  // sentences generated from source_ in vector<Words>. sourceTokenRanges_ are
+  // string_views of the text corresponding to these words, pointing to
+  // sequences in source_. histories_ is a buffer which eventually stores the
+  // translations of each segment in the corresponding index.
+  std::string source_;
+  Segments segments_;
+  std::vector<TokenRanges> sourceTokenRanges_;
+  std::vector<Ptr<History>> histories_;
+
+  // Members above are moved into newly constructed Response on completion
+  // of translation of all segments. The promise below is set to this Response
+  // value. future to this promise is made available to the user through
+  // Service.
+  std::promise<Response> response_;
+
+  // Constructing Response requires the vocabs_ used to generate Request.
+  std::vector<Ptr<Vocab const>> *vocabs_;
 };
 
 class RequestSentence {
-private:
-  size_t index_;
-  Ptr<Request> request_;
+  // A RequestSentence provides a view to a sentence within a Request. Existence
+  // of this class allows the sentences and associated information to be kept
+  // within Request.
 
 public:
   RequestSentence(size_t, Ptr<Request>);
   size_t numTokens() const;
+
+  // lineNumber in Request, used for matching marian-decoder. SentenceTuple
+  // requires lineNumber to be set for Corpus based batches.
   size_t lineNumber() const;
+
+  // Accessor to the segment represented by the RequestSentence.
   Segment getUnderlyingSegment() const;
+
+  // Forwards call to Request, checking for completion.
   void completeSentence(Ptr<History> history);
+
   friend bool operator<(const RequestSentence &a, const RequestSentence &b);
+
+private:
+  size_t index_;
+  Ptr<Request> request_;
 };
 
 typedef std::vector<RequestSentence> RequestSentences;
@@ -98,47 +125,48 @@ typedef std::vector<RequestSentence> RequestSentences;
 class Batch {
 public:
   Batch() { reset(); }
-  void reset() {
-    Id_ = 0;
-    sentences_.clear();
-  }
-  // Convenience function to determine poison.
-  bool isPoison() { return (Id_ == -1); }
+  // Reset is required to reuse the same batch by consumer.
+  void reset();
+
+  //  Methods to construct and determine poison.
   static Batch poison() {
     Batch poison_;
     poison_.Id_ = -1;
     return poison_;
   }
+  bool isPoison() const { return (Id_ == -1); }
+
+  size_t size() const { return sentences_.size(); }
+
+  // Accessors to load data into a batch. Use add(...) to add sentences into a
+  // batch. Once complete with a legal batch, use setId to set Id_ accordingly.
+  // setId only allows setting Id > 0. For use in Batcher, which acts as a
+  // producer to a PCQueue holding "Batch"es.
+  //
+  // Id_ =
+  //    -1 : Batch::Poison
+  //     0 : Empty Batch
+  //    >0 : Legal batch containing sentences
+
+  void add(const RequestSentence &sentence);
+  void setId(int Id);
+
+  // Accessors to read from a Batch. For use in BatchTranslator (consumer on a
+  // PCQueue holding batches).
+  //
+  // sentences() are used to access sentences to construct marian internal
+  // batch.
+  const RequestSentences &sentences() { return sentences_; }
 
-  void log() {
-    int numTokens{0}, maxLength{0};
-    for (auto &sentence : sentences_) {
-      numTokens += sentence.numTokens();
-      maxLength = std::max(maxLength, static_cast<int>(sentence.numTokens()));
-    }
-
-    LOG(info, "Batch(Id_={}, tokens={}, max-length={}, sentences_={})", Id_,
-        numTokens, maxLength, sentences_.size());
-  }
-
-  void add(const RequestSentence &sentence) { sentences_.push_back(sentence); }
-
-  size_t size() { return sentences_.size(); }
-
-  void setId(int Id) {
-    assert(Id > 0);
-    Id_ = Id;
-    if (Id % 500 == 0) {
-      log();
-    }
-  }
+  // On obtaining Histories after translating a batch, completeBatch can be
+  // called with Histories , which forwards the call to Request through
+  // RequestSentence and triggers completion, by setting the promised value to
+  // the future given to client.
+  void completeBatch(const Histories &histories);
 
-  const RequestSentences &sentences() { return sentences_; }
-  void completeBatch(const Histories &histories) {
-    for (int i = 0; i < sentences_.size(); i++) {
-      sentences_[i].completeSentence(histories[i]);
-    }
-  }
+  // Convenience function to log batch-statistics. numTokens, max-length.
+  // TODO(jerinphilip): Use to log and report packing efficiency.
+  void log();
 
 private:
   int Id_;
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 96f391c2d..2163eefb9 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -56,8 +56,8 @@ std::future<Response> Service::translate(std::string &&input) {
   // returns future corresponding to the promise.
 
   Segments segments;
-  std::vector<TokenRanges> sourceAlignments;
-  text_processor_.process(input, segments, sourceAlignments);
+  std::vector<TokenRanges> sourceTokenRanges;
+  text_processor_.process(input, segments, sourceTokenRanges);
 
   std::promise<Response> responsePromise;
   auto future = responsePromise.get_future();
@@ -65,7 +65,7 @@ std::future<Response> Service::translate(std::string &&input) {
   Ptr<Request> request =
       New<Request>(requestId_++, /* lineNumberBegin = */ 0, vocabs_,
                    std::move(input), std::move(segments),
-                   std::move(sourceAlignments), std::move(responsePromise));
+                   std::move(sourceTokenRanges), std::move(responsePromise));
 
   batcher_.addWholeRequest(request);
 

From 921c2eedf812b3304a06ebfad890fb025755c2a0 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 16 Feb 2021 14:21:46 +0100
Subject: [PATCH 114/442] Updated config for min inference time

 - This combination gives min inference time (~ 200 WPS)
   on local machine
---
 wasm/test_page/bergamot.html | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 8fc7824e1..d91a9a160 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -75,17 +75,25 @@
 beam-size: 1
 normalize: 1.0
 word-penalty: 0
-mini-batch: 32
-maxi-batch: 100
-maxi-batch-sort: src
+max-input-sentence-tokens: 128
+max-input-tokens: 1024
 workspace: 128
 max-length-factor: 2.0
 skip-cost: true
+cpu-threads: 1
+quiet: true
+quiet-translation: true
 shortlist:
     - /lex.${lang}.s2t
     - 50
     - 50
 `;
+/*
+This config is not valid anymore in new APIs
+mini-batch: 32
+maxi-batch: 100
+maxi-batch-sort: src
+*/
 // TODO: Use in model config when wormhole is enabled:
 // gemm-precision: int8shift
 // TODO: Use in model config when loading of binary models is supported and we use model.intgemm.alphas.bin:

From b1e72ce75e2bce611b6dee11408278f8b3e3e4ec Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Tue, 16 Feb 2021 15:46:15 +0200
Subject: [PATCH 115/442] Updated instructions on how to get all relevant
 models in place for the upcoming release

---
 README.md | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff --git a/README.md b/README.md
index 4bff753d1..0d55686ff 100644
--- a/README.md
+++ b/README.md
@@ -52,17 +52,10 @@ Files packaged this way are preloaded in the root of the virtual file system.
 To package the set of files expected by the test page:
 
 ```bash
-git clone https://github.com/browsermt/students
-cd students/esen/
-./download-models.sh
-cp esen.student.tiny11/lex.s2t ../../models/lex.esen.s2t
-cp esen.student.tiny11/model.npz ../../models/model.esen.npz
-cp esen.student.tiny11/vocab.esen.spm ../../models/vocab.esen.spm
-cd -
-cd students/enes/
-./download-models.sh
-cp enes.student.tiny11/lex.s2t ../../models/lex.enes.s2t
-cp enes.student.tiny11/model.npz ../../models/model.enes.npz
+mkdir models
+git clone https://github.com/motin/bergamot-models
+cp -r bergamot-models/* models
+gunzip models/*/*
 ```
 
 After Editing Files:

From d907400a80d59cac771dad7f31d67bcb67411270 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Tue, 16 Feb 2021 17:00:45 +0200
Subject: [PATCH 116/442] Updated test page to use the model structure from
 bergamot-models repo

---
 wasm/test_page/bergamot.html | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index d91a9a160..795654495 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -64,14 +64,20 @@
 <script>
 
   var model, request, input = undefined;
-  const loadModel = (lang) => {
+  const loadModel = (from, to) => {
+
+    const languagePair = `${from}${to}`;
+
+    // Vocab files are re-used in both translation directions
+    const vocabLanguagePair = from === "en" ? `${to}${from}` : languagePair;
+
     // Set the Model Configuration as YAML formatted string.
     // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
     const modelConfig = `models:
-  - /model.${lang}.npz
+  - /${languagePair}/model.${languagePair}.npz
 vocabs:
-  - /vocab.esen.spm
-  - /vocab.esen.spm
+  - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
+  - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
 beam-size: 1
 normalize: 1.0
 word-penalty: 0
@@ -84,7 +90,7 @@
 quiet: true
 quiet-translation: true
 shortlist:
-    - /lex.${lang}.s2t
+    - /${languagePair}/lex.${languagePair}.s2t
     - 50
     - 50
 `;
@@ -99,6 +105,8 @@
 // TODO: Use in model config when loading of binary models is supported and we use model.intgemm.alphas.bin:
 // gemm-precision: int8shiftAlphaAll
 
+    console.debug("modelConfig: ", modelConfig);
+
     // Instantiate the TranslationModel
     if (model) model.delete();
     model = new Module.TranslationModel(modelConfig);
@@ -142,9 +150,11 @@
 
   document.querySelector("#load").addEventListener("click", () => {
     const lang = document.querySelector('input[name="modellang"]:checked').value;
+    const from = lang.substring(0, 2);
+    const to = lang.substring(2, 4);
     let start = Date.now();
-    loadModel(lang)
-    log(`model ${lang} loaded in ${(Date.now() - start) / 1000} secs`);
+    loadModel(from, to)
+    log(`model ${from}${to} loaded in ${(Date.now() - start) / 1000} secs`);
     //log('Model Alignment:', model.isAlignmentSupported());
   });
 

From 65e74069709784614d6936a1ea55623ee001d04b Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 16 Feb 2021 17:00:53 +0000
Subject: [PATCH 117/442] Comments and lazy stuff to response

---
 app/main-mts.cpp            |  2 +-
 src/translator/response.cpp | 30 +++++++++++--------
 src/translator/response.h   | 59 +++++++++++++++++++++++++++++++++----
 3 files changed, 72 insertions(+), 19 deletions(-)

diff --git a/app/main-mts.cpp b/app/main-mts.cpp
index 78967be0e..a8382b28b 100644
--- a/app/main-mts.cpp
+++ b/app/main-mts.cpp
@@ -24,7 +24,7 @@ int main(int argc, char *argv[]) {
   // Wait on future until Response is complete
   std::future<Response> responseFuture = service.translate(std::move(input));
   responseFuture.wait();
-  const Response &response = responseFuture.get();
+  Response response = responseFuture.get();
   std::cout << response.translation() << std::endl;
 
   // Stop Service.
diff --git a/src/translator/response.cpp b/src/translator/response.cpp
index d40f88da7..6af089d9e 100644
--- a/src/translator/response.cpp
+++ b/src/translator/response.cpp
@@ -11,16 +11,12 @@ Response::Response(std::string &&source,
                    std::vector<TokenRanges> &&sourceRanges,
                    Histories &&histories, std::vector<Ptr<Vocab const>> &vocabs)
     : source_(std::move(source)), sourceRanges_(std::move(sourceRanges)),
-      histories_(std::move(histories)) {
-
-  constructTargetProperties(vocabs);
-}
+      histories_(std::move(histories)), vocabs_(&vocabs) {}
 
 void Response::move(std::string &source, std::string &translation,
                     SentenceMappings &sentenceMappings) {
 
   constructSentenceMappings(sentenceMappings);
-  // Totally illegal stuff.
   source = std::move(source_);
   translation = std::move(translation_);
 
@@ -31,18 +27,23 @@ void Response::move(std::string &source, std::string &translation,
   histories_.clear();
 }
 
-void Response::constructTargetProperties(
-    std::vector<Ptr<Vocab const>> &vocabs) {
-  std::vector<std::pair<int, int>> translationRanges;
+void Response::constructTranslation() {
+
+  // In a first step, the decoded units (individual senteneces) are compiled
+  // into a huge string. This is done by computing indices first and appending
+  // to the string as each sentences are decoded.
+  std::vector<std::pair<size_t, size_t>> translationRanges;
+
   size_t offset{0};
   bool first{true};
+
   for (auto &history : histories_) {
     // TODO(jerin): Change hardcode of nBest = 1
     NBestList onebest = history->nBest(1);
 
     Result result = onebest[0]; // Expecting only one result;
     Words words = std::get<0>(result);
-    std::string decoded = (vocabs.back())->decode(words);
+    std::string decoded = vocabs_->back()->decode(words);
     if (first) {
       first = false;
     } else {
@@ -55,13 +56,18 @@ void Response::constructTargetProperties(
     offset += decoded.size();
   }
 
-  // TODO(@jerinphilip):
-  // Currently considers target tokens as whole text. Needs
-  // to be further enhanced in marian-dev to extract alignments.
+  // Once the entire string is constructed, there are no further possibility of
+  // reallocation in the string's storage, the indices are converted into
+  // string_views.
+
   for (auto &range : translationRanges) {
+    // TODO(@jerinphilip):  Currently considers target tokens as whole text.
+    // Needs to be further enhanced in marian-dev to extract alignments.
     std::vector<string_view> targetMappings;
+
     const char *begin = &translation_[range.first];
     targetMappings.emplace_back(begin, range.second);
+
     targetRanges_.push_back(std::move(targetMappings));
   }
 }
diff --git a/src/translator/response.h b/src/translator/response.h
index 57377176d..66891cc6a 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -12,10 +12,32 @@
 namespace marian {
 namespace bergamot {
 class Response {
+  // Response is a marian internal class (not a bergamot-translator class)
+  // holding source blob of text, vector of TokenRanges corresponding to each
+  // sentence in the source text blob and histories obtained from translating
+  // these sentences.
+  //
+  // This class provides an API at a higher level in comparison to History to
+  // access translations and additionally use string_view manipulations to
+  // recover structure in translation from source-text's structure known through
+  // reference string and string_view. As many of these computations are not
+  // required until invoked, they are computed as required and stored in data
+  // members where it makes sense to do so (translation,translationTokenRanges).
+  //
+  // Examples of such use-cases are:
+  //    translation()
+  //    translationInSourceStructure() TODO(@jerinphilip)
+  //    alignment(idx) TODO(@jerinphilip)
+  //    sentenceMappings (for bergamot-translator)
+
 public:
   Response(std::string &&source, std::vector<TokenRanges> &&sourceRanges,
-           Histories &&histories, std::vector<Ptr<Vocab const>> &vocabs);
+           Histories &&histories,
+           // Required for constructing translation and TokenRanges within
+           // translation lazily.
+           std::vector<Ptr<Vocab const>> &vocabs);
 
+  // Move constructor.
   Response(Response &&other)
       : source_(std::move(other.source_)),
         translation_(std::move(other.translation_)),
@@ -23,27 +45,52 @@ class Response {
         targetRanges_(std::move(other.targetRanges_)),
         histories_(std::move(other.histories_)){};
 
+  // Prevents CopyConstruction and CopyAssignment. sourceRanges_ is constituted
+  // by string_view and copying invalidates the data member.
   Response(const Response &) = delete;
   Response &operator=(const Response &) = delete;
 
   typedef std::vector<std::pair<const string_view, const string_view>>
       SentenceMappings;
 
-  void move(std::string &source, std::string &target,
+  // Moves source sentence into source, translated text into translation.
+  // Pairs of string_views to corresponding sentences in
+  // source and translation are loaded into sentenceMappings. These string_views
+  // reference the new source and translation.
+  //
+  // Calling move() invalidates the Response object as ownership is transferred.
+  // Exists for moving strc
+  void move(std::string &source, std::string &translation,
             SentenceMappings &sentenceMappings);
 
   const Histories &histories() const { return histories_; }
   const std::string &source() const { return source_; }
-  const std::string &translation() const { return translation_; }
+  const std::string &translation() {
+    if (!translationConstructed) {
+      constructTranslation();
+    }
+    return translation_;
+  }
+
+  // A convenience function provided to return translated text placed within
+  // source's structure. This is useful when the source text is a multi-line
+  // paragraph or string_views extracted from structured text like HTML and it's
+  // desirable to place the individual sentences in the locations of the source
+  // sentences.
+  // const std::string translationInSourceStructure();
+  // const PendingAlignmentType alignment(size_t idx);
 
 private:
-  void constructTargetProperties(std::vector<Ptr<Vocab const>> &vocabs);
+  void constructTranslation();
   void constructSentenceMappings(SentenceMappings &);
 
   std::string source_;
-  std::string translation_;
-  Histories histories_;
   std::vector<TokenRanges> sourceRanges_;
+  Histories histories_;
+
+  std::vector<Ptr<Vocab const>> *vocabs_{nullptr};
+  bool translationConstructed{false};
+  std::string translation_;
   std::vector<TokenRanges> targetRanges_;
 };
 } // namespace bergamot

From 4c8b655ac5c46c6eaaafa6dd3eea32232866a182 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 16 Feb 2021 19:46:40 +0000
Subject: [PATCH 118/442] Batch cleanup

Moves Batch into batch.{h,cpp}.

- Id_ no longer used due to overflow concerns. (#27)
- size_t for places where signed integer is not preferred.
- Adjustments to response.{h,cpp}
---
 src/translator/CMakeLists.txt       |  1 +
 src/translator/batch.cpp            | 28 +++++++++++++++
 src/translator/batch.h              | 52 +++++++++++++++++++++++++++
 src/translator/batch_translator.cpp |  1 +
 src/translator/batch_translator.h   |  1 +
 src/translator/batcher.cpp          | 12 +++----
 src/translator/batcher.h            |  1 +
 src/translator/request.cpp          | 34 ------------------
 src/translator/request.h            | 54 -----------------------------
 src/translator/response.cpp         | 17 +++++++--
 src/translator/response.h           | 11 +++---
 src/translator/service.cpp          |  1 +
 12 files changed, 109 insertions(+), 104 deletions(-)
 create mode 100644 src/translator/batch.cpp
 create mode 100644 src/translator/batch.h

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index c279ab975..e83c6155c 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -11,6 +11,7 @@ add_library(bergamot-translator STATIC
     service.cpp
     batcher.cpp
     response.cpp
+    batch.cpp
 )
 
 target_link_libraries(bergamot-translator marian ssplit)
diff --git a/src/translator/batch.cpp b/src/translator/batch.cpp
new file mode 100644
index 000000000..973762fa4
--- /dev/null
+++ b/src/translator/batch.cpp
@@ -0,0 +1,28 @@
+#include "batch.h"
+#include "request.h"
+
+namespace marian {
+namespace bergamot {
+
+void Batch::log() {
+  size_t numTokens{0}, maxLength{0};
+  for (auto &sentence : sentences_) {
+    numTokens += sentence.numTokens();
+    maxLength = std::max(maxLength, static_cast<size_t>(sentence.numTokens()));
+  }
+
+  LOG(info, "Batch(tokens={}, max-length={}, sentences_={})", numTokens,
+      maxLength, sentences_.size());
+}
+
+void Batch::add(const RequestSentence &sentence) {
+  sentences_.push_back(sentence);
+}
+
+void Batch::completeBatch(const Histories &histories) {
+  for (int i = 0; i < sentences_.size(); i++) {
+    sentences_[i].completeSentence(histories[i]);
+  }
+}
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/batch.h b/src/translator/batch.h
new file mode 100644
index 000000000..5f86a2f5c
--- /dev/null
+++ b/src/translator/batch.h
@@ -0,0 +1,52 @@
+#ifndef SRC_BERGAMOT_BATCH_H
+#define SRC_BERGAMOT_BATCH_H
+
+#include "request.h"
+#include "translator/beam_search.h"
+
+namespace marian {
+namespace bergamot {
+
+class Batch {
+public:
+  Batch() {}
+  void clear() { sentences_.clear(); }
+
+  //  Methods to construct and determine poison.
+  static Batch poison() {
+    Batch batch;
+    batch.poison_ = true;
+    return batch;
+  }
+
+  bool isPoison() const { return poison_; }
+
+  size_t size() const { return sentences_.size(); }
+
+  void add(const RequestSentence &sentence);
+
+  // Accessors to read from a Batch. For use in BatchTranslator (consumer on a
+  // PCQueue holding batches).
+  //
+  // sentences() are used to access sentences to construct marian internal
+  // batch.
+  const RequestSentences &sentences() { return sentences_; }
+
+  // On obtaining Histories after translating a batch, completeBatch can be
+  // called with Histories , which forwards the call to Request through
+  // RequestSentence and triggers completion, by setting the promised value to
+  // the future given to client.
+  void completeBatch(const Histories &histories);
+
+  // Convenience function to log batch-statistics. numTokens, max-length.
+  void log();
+
+private:
+  bool poison_{false};
+  RequestSentences sentences_;
+};
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_BATCH_H_
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 7da63cf57..2679f02a3 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -1,4 +1,5 @@
 #include "batch_translator.h"
+#include "batch.h"
 #include "common/logging.h"
 #include "data/corpus.h"
 #include "data/text_input.h"
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 83b911ceb..721cee481 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -4,6 +4,7 @@
 #include <string>
 #include <vector>
 
+#include "batch.h"
 #include "common/utils.h"
 #include "data/shortlist.h"
 #include "definitions.h"
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 9ba0d035f..657a78ab1 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -1,4 +1,5 @@
 #include "batcher.h"
+#include "batch.h"
 #include "common/logging.h"
 #include <cassert>
 
@@ -27,7 +28,7 @@ bool Batcher::cleaveBatch(Batch &batch) {
   // has to be enhanced with optimizing over priority. The baseline
   // implementation should at least be as fast as marian's maxi-batch with full
   // corpus size as maxi-batch size.
-  batch.reset();
+  batch.clear();
   int paddedBatchSize = 0;
 
   for (int length = 0; length < bucket_.size(); length++) {
@@ -41,18 +42,13 @@ bool Batcher::cleaveBatch(Batch &batch) {
       } else {
         // Check if elements exist
         assert(batch.size() > 0);
-        batch.setId(++batchNumber_);
         return true;
       }
     }
   }
 
-  if (batch.size()) {
-    batch.setId(++batchNumber_);
-    return true;
-  } else {
-    return false;
-  }
+  bool isValidBatch = batch.size() > 0;
+  return isValidBatch;
 }
 
 void Batcher::addWholeRequest(Ptr<Request> request) {
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index 342725708..adf2dbe12 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -1,6 +1,7 @@
 #ifndef SRC_BERGAMOT_BATCHER_H_
 #define SRC_BERGAMOT_BATCHER_H_
 
+#include "batch.h"
 #include "common/options.h"
 #include "data/corpus_base.h"
 #include "definitions.h"
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 303f9cc7d..82c48683a 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -92,39 +92,5 @@ bool operator<(const RequestSentence &a, const RequestSentence &b) {
 
 // ----------------------------------------------------------------------
 
-void Batch::reset() {
-  Id_ = 0;
-  sentences_.clear();
-}
-
-void Batch::log() {
-  int numTokens{0}, maxLength{0};
-  for (auto &sentence : sentences_) {
-    numTokens += sentence.numTokens();
-    maxLength = std::max(maxLength, static_cast<int>(sentence.numTokens()));
-  }
-
-  LOG(info, "Batch(Id_={}, tokens={}, max-length={}, sentences_={})", Id_,
-      numTokens, maxLength, sentences_.size());
-}
-
-void Batch::add(const RequestSentence &sentence) {
-  sentences_.push_back(sentence);
-}
-
-void Batch::setId(int Id) {
-  assert(Id > 0);
-  Id_ = Id;
-  if (Id % 500 == 0) {
-    log();
-  }
-}
-
-void Batch::completeBatch(const Histories &histories) {
-  for (int i = 0; i < sentences_.size(); i++) {
-    sentences_[i].completeSentence(histories[i]);
-  }
-}
-
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/request.h b/src/translator/request.h
index 095a03ccd..8aea3d807 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -13,9 +13,6 @@
 // batching mechanism access to the segment within the request. The backref to
 // Request allows event triggering the barrier upon completion of the last
 // sentence by a worker.
-//
-// Batch: is a vector of RequestSentences tagged with a batchNumber, which is
-// what the PCQueue holds. Batch is "produced" by the Batcher.
 
 #ifndef SRC_BERGAMOT_REQUEST_H_
 #define SRC_BERGAMOT_REQUEST_H_
@@ -122,57 +119,6 @@ class RequestSentence {
 
 typedef std::vector<RequestSentence> RequestSentences;
 
-class Batch {
-public:
-  Batch() { reset(); }
-  // Reset is required to reuse the same batch by consumer.
-  void reset();
-
-  //  Methods to construct and determine poison.
-  static Batch poison() {
-    Batch poison_;
-    poison_.Id_ = -1;
-    return poison_;
-  }
-  bool isPoison() const { return (Id_ == -1); }
-
-  size_t size() const { return sentences_.size(); }
-
-  // Accessors to load data into a batch. Use add(...) to add sentences into a
-  // batch. Once complete with a legal batch, use setId to set Id_ accordingly.
-  // setId only allows setting Id > 0. For use in Batcher, which acts as a
-  // producer to a PCQueue holding "Batch"es.
-  //
-  // Id_ =
-  //    -1 : Batch::Poison
-  //     0 : Empty Batch
-  //    >0 : Legal batch containing sentences
-
-  void add(const RequestSentence &sentence);
-  void setId(int Id);
-
-  // Accessors to read from a Batch. For use in BatchTranslator (consumer on a
-  // PCQueue holding batches).
-  //
-  // sentences() are used to access sentences to construct marian internal
-  // batch.
-  const RequestSentences &sentences() { return sentences_; }
-
-  // On obtaining Histories after translating a batch, completeBatch can be
-  // called with Histories , which forwards the call to Request through
-  // RequestSentence and triggers completion, by setting the promised value to
-  // the future given to client.
-  void completeBatch(const Histories &histories);
-
-  // Convenience function to log batch-statistics. numTokens, max-length.
-  // TODO(jerinphilip): Use to log and report packing efficiency.
-  void log();
-
-private:
-  int Id_;
-  RequestSentences sentences_;
-};
-
 } // namespace bergamot
 } // namespace marian
 
diff --git a/src/translator/response.cpp b/src/translator/response.cpp
index 6af089d9e..97af6253f 100644
--- a/src/translator/response.cpp
+++ b/src/translator/response.cpp
@@ -16,7 +16,11 @@ Response::Response(std::string &&source,
 void Response::move(std::string &source, std::string &translation,
                     SentenceMappings &sentenceMappings) {
 
+  // Construct required stuff first.
+  constructTranslation();
   constructSentenceMappings(sentenceMappings);
+
+  // Move content out.
   source = std::move(source_);
   translation = std::move(translation_);
 
@@ -28,6 +32,13 @@ void Response::move(std::string &source, std::string &translation,
 }
 
 void Response::constructTranslation() {
+  if (translationConstructed_) {
+    return;
+  }
+
+  // Reserving length at least as much as source_ seems like a reasonable thing
+  // to do to avoid reallocations.
+  translation_.reserve(source_.size());
 
   // In a first step, the decoded units (individual senteneces) are compiled
   // into a huge string. This is done by computing indices first and appending
@@ -43,7 +54,8 @@ void Response::constructTranslation() {
 
     Result result = onebest[0]; // Expecting only one result;
     Words words = std::get<0>(result);
-    std::string decoded = vocabs_->back()->decode(words);
+    auto targetVocab = vocabs_->back();
+    std::string decoded = targetVocab->decode(words);
     if (first) {
       first = false;
     } else {
@@ -67,9 +79,10 @@ void Response::constructTranslation() {
 
     const char *begin = &translation_[range.first];
     targetMappings.emplace_back(begin, range.second);
-
     targetRanges_.push_back(std::move(targetMappings));
   }
+
+  translationConstructed_ = true;
 }
 
 void Response::constructSentenceMappings(
diff --git a/src/translator/response.h b/src/translator/response.h
index 66891cc6a..54e14fb26 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -43,7 +43,8 @@ class Response {
         translation_(std::move(other.translation_)),
         sourceRanges_(std::move(other.sourceRanges_)),
         targetRanges_(std::move(other.targetRanges_)),
-        histories_(std::move(other.histories_)){};
+        histories_(std::move(other.histories_)),
+        vocabs_(std::move(other.vocabs_)){};
 
   // Prevents CopyConstruction and CopyAssignment. sourceRanges_ is constituted
   // by string_view and copying invalidates the data member.
@@ -66,9 +67,7 @@ class Response {
   const Histories &histories() const { return histories_; }
   const std::string &source() const { return source_; }
   const std::string &translation() {
-    if (!translationConstructed) {
-      constructTranslation();
-    }
+    constructTranslation();
     return translation_;
   }
 
@@ -88,8 +87,8 @@ class Response {
   std::vector<TokenRanges> sourceRanges_;
   Histories histories_;
 
-  std::vector<Ptr<Vocab const>> *vocabs_{nullptr};
-  bool translationConstructed{false};
+  std::vector<Ptr<Vocab const>> *vocabs_;
+  bool translationConstructed_{false};
   std::string translation_;
   std::vector<TokenRanges> targetRanges_;
 };
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 2163eefb9..151fcb665 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -1,4 +1,5 @@
 #include "service.h"
+#include "batch.h"
 #include "definitions.h"
 
 #include <string>

From 9c907ea605af4a568fb945272247ada8e70a6a9c Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 16 Feb 2021 20:04:30 +0000
Subject: [PATCH 119/442] another int to size_t

---
 src/translator/batch.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/translator/batch.cpp b/src/translator/batch.cpp
index 973762fa4..82ebbfbf1 100644
--- a/src/translator/batch.cpp
+++ b/src/translator/batch.cpp
@@ -20,7 +20,7 @@ void Batch::add(const RequestSentence &sentence) {
 }
 
 void Batch::completeBatch(const Histories &histories) {
-  for (int i = 0; i < sentences_.size(); i++) {
+  for (size_t i = 0; i < sentences_.size(); i++) {
     sentences_[i].completeSentence(histories[i]);
   }
 }

From d7556bc1681048856e8e3e5f83087b2856221c0e Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 00:31:44 +0000
Subject: [PATCH 120/442] SentenceRanges: Class to work with string_views

Adds SentenceRanges in sentence_ranges.{h,cpp} and propogates use of the
class into the rest of the pipeline.

SentenceRanges previously a vector<vector<...>> is now converted into a
flat single vector<string_view>. Annotations marking sentence boundaries
are additionally stored in the class, enabling sentence string_view
access through methods.
---
 src/translator/CMakeLists.txt      |  1 +
 src/translator/request.cpp         |  9 +++---
 src/translator/request.h           |  7 ++--
 src/translator/response.cpp        | 25 ++++----------
 src/translator/response.h          |  7 ++--
 src/translator/sentence_ranges.cpp | 46 ++++++++++++++++++++++++++
 src/translator/sentence_ranges.h   | 52 ++++++++++++++++++++++++++++++
 src/translator/service.cpp         | 13 ++++----
 src/translator/text_processor.cpp  | 24 +++++++-------
 src/translator/text_processor.h    | 10 +++---
 10 files changed, 143 insertions(+), 51 deletions(-)
 create mode 100644 src/translator/sentence_ranges.cpp
 create mode 100644 src/translator/sentence_ranges.h

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index e83c6155c..07d7ef057 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -12,6 +12,7 @@ add_library(bergamot-translator STATIC
     batcher.cpp
     response.cpp
     batch.cpp
+    sentence_ranges.cpp
 )
 
 target_link_libraries(bergamot-translator marian ssplit)
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 82c48683a..8c858e6d6 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -1,5 +1,5 @@
 #include "request.h"
-
+#include "sentence_ranges.h"
 #include "definitions.h"
 #include "response.h"
 
@@ -13,12 +13,11 @@ namespace bergamot {
 // -----------------------------------------------------------------
 Request::Request(unsigned int Id, int lineNumberBegin,
                  std::vector<Ptr<Vocab const>> &vocabs, std::string &&source,
-                 Segments &&segments,
-                 std::vector<TokenRanges> &&sourceTokenRanges,
+                 Segments &&segments, SentenceRanges &&sourceRanges,
                  std::promise<Response> responsePromise)
     : Id_(Id), lineNumberBegin_(lineNumberBegin), vocabs_(&vocabs),
       source_(std::move(source)), segments_(std::move(segments)),
-      sourceTokenRanges_(std::move(sourceTokenRanges)),
+      sourceRanges_(std::move(sourceRanges)),
       response_(std::move(responsePromise)) {
 
   counter_ = segments_.size();
@@ -49,7 +48,7 @@ void Request::processHistory(size_t index, Ptr<History> history) {
 void Request::completeRequest() {
   // Request no longer needs to hold the content, can transfer it to
   // Response.
-  Response response(std::move(source_), std::move(sourceTokenRanges_),
+  Response response(std::move(source_), std::move(sourceRanges_),
                     std::move(histories_), *vocabs_);
   response_.set_value(std::move(response));
 }
diff --git a/src/translator/request.h b/src/translator/request.h
index 8aea3d807..0bc3c272a 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -17,6 +17,7 @@
 #ifndef SRC_BERGAMOT_REQUEST_H_
 #define SRC_BERGAMOT_REQUEST_H_
 
+#include "sentence_ranges.h"
 #include "definitions.h"
 #include "response.h"
 
@@ -36,7 +37,7 @@ class Request {
 public:
   Request(unsigned int Id, int lineNumberBegin,
           std::vector<Ptr<Vocab const>> &vocabs_, std::string &&source,
-          Segments &&segments, std::vector<TokenRanges> &&sourceTokenRanges,
+          Segments &&segments, SentenceRanges &&sourceTokenRanges,
           std::promise<Response> responsePromise);
 
   // Obtain the count of tokens in the segment correponding to index. Used to
@@ -72,13 +73,13 @@ class Request {
   std::atomic<int> counter_;
 
   // source_ holds the source string to be translated. segments_ hold the
-  // sentences generated from source_ in vector<Words>. sourceTokenRanges_ are
+  // sentences generated from source_ in vector<Words>. sourceRanges_ are
   // string_views of the text corresponding to these words, pointing to
   // sequences in source_. histories_ is a buffer which eventually stores the
   // translations of each segment in the corresponding index.
   std::string source_;
   Segments segments_;
-  std::vector<TokenRanges> sourceTokenRanges_;
+  SentenceRanges sourceRanges_;
   std::vector<Ptr<History>> histories_;
 
   // Members above are moved into newly constructed Response on completion
diff --git a/src/translator/response.cpp b/src/translator/response.cpp
index 97af6253f..b731755d1 100644
--- a/src/translator/response.cpp
+++ b/src/translator/response.cpp
@@ -1,4 +1,5 @@
 #include "response.h"
+#include "sentence_ranges.h"
 #include "common/logging.h"
 #include "data/alignment.h"
 
@@ -7,8 +8,7 @@
 namespace marian {
 namespace bergamot {
 
-Response::Response(std::string &&source,
-                   std::vector<TokenRanges> &&sourceRanges,
+Response::Response(std::string &&source, SentenceRanges &&sourceRanges,
                    Histories &&histories, std::vector<Ptr<Vocab const>> &vocabs)
     : source_(std::move(source)), sourceRanges_(std::move(sourceRanges)),
       histories_(std::move(histories)), vocabs_(&vocabs) {}
@@ -79,7 +79,7 @@ void Response::constructTranslation() {
 
     const char *begin = &translation_[range.first];
     targetMappings.emplace_back(begin, range.second);
-    targetRanges_.push_back(std::move(targetMappings));
+    targetRanges_.addSentence(targetMappings);
   }
 
   translationConstructed_ = true;
@@ -88,21 +88,10 @@ void Response::constructTranslation() {
 void Response::constructSentenceMappings(
     Response::SentenceMappings &sentenceMappings) {
 
-  for (int i = 0; i < sourceRanges_.size(); i++) {
-    string_view first, last;
-
-    // Handle source-sentence
-    first = sourceRanges_[i].front();
-    last = sourceRanges_[i].back();
-    string_view src_sentence(first.data(), last.end() - first.begin());
-
-    // Handle target-sentence
-    first = targetRanges_[i].front();
-    last = targetRanges_[i].back();
-    string_view tgt_sentence(first.data(), last.end() - first.begin());
-
-    // Add both into sentence-mappings
-    sentenceMappings.emplace_back(src_sentence, tgt_sentence);
+  for (size_t i = 0; i < sourceRanges_.numSentences(); i++) {
+    string_view src = sourceRanges_.sentence(i);
+    string_view tgt = targetRanges_.sentence(i);
+    sentenceMappings.emplace_back(src, tgt);
   }
 }
 } // namespace bergamot
diff --git a/src/translator/response.h b/src/translator/response.h
index 54e14fb26..17fee05ae 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -1,6 +1,7 @@
 #ifndef SRC_BERGAMOT_RESPONSE_H_
 #define SRC_BERGAMOT_RESPONSE_H_
 
+#include "sentence_ranges.h"
 #include "data/types.h"
 #include "definitions.h"
 #include "translator/beam_search.h"
@@ -31,7 +32,7 @@ class Response {
   //    sentenceMappings (for bergamot-translator)
 
 public:
-  Response(std::string &&source, std::vector<TokenRanges> &&sourceRanges,
+  Response(std::string &&source, SentenceRanges &&sourceRanges,
            Histories &&histories,
            // Required for constructing translation and TokenRanges within
            // translation lazily.
@@ -84,13 +85,13 @@ class Response {
   void constructSentenceMappings(SentenceMappings &);
 
   std::string source_;
-  std::vector<TokenRanges> sourceRanges_;
+  SentenceRanges sourceRanges_;
   Histories histories_;
 
   std::vector<Ptr<Vocab const>> *vocabs_;
   bool translationConstructed_{false};
   std::string translation_;
-  std::vector<TokenRanges> targetRanges_;
+  SentenceRanges targetRanges_;
 };
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/sentence_ranges.cpp b/src/translator/sentence_ranges.cpp
new file mode 100644
index 000000000..a9ee8c54e
--- /dev/null
+++ b/src/translator/sentence_ranges.cpp
@@ -0,0 +1,46 @@
+#include "sentence_ranges.h"
+#include <cassert>
+#include <iostream>
+
+namespace marian {
+namespace bergamot {
+
+void SentenceRanges::addSentence(std::vector<string_view> &wordRanges) {
+  addSentence(std::begin(wordRanges), std::end(wordRanges));
+}
+
+void SentenceRanges::addSentence(WordIterator begin, WordIterator end) {
+  size_t size = flatByteRanges_.size();
+  flatByteRanges_.insert(std::end(flatByteRanges_), begin, end);
+  sentenceBeginIds_.push_back(size);
+}
+
+string_view SentenceRanges::sentence(size_t index) const {
+  size_t bos_id;
+  string_view eos, bos;
+
+  bos_id = sentenceBeginIds_[index];
+  bos = flatByteRanges_[bos_id];
+
+  if (index + 1 == numSentences()) {
+    eos = flatByteRanges_.back();
+  } else {
+    assert(index < numSentences());
+    size_t eos_id = sentenceBeginIds_[index + 1];
+    --eos_id;
+    eos = flatByteRanges_[eos_id];
+  }
+
+  return sentenceBetween(bos, eos);
+}
+
+string_view SentenceRanges::sentenceBetween(string_view firstWord,
+                                            string_view lastWord) const {
+
+  const char *data = firstWord.data();
+  size_t size = lastWord.data() + lastWord.size() - firstWord.data();
+  return string_view(data, size);
+}
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/sentence_ranges.h b/src/translator/sentence_ranges.h
new file mode 100644
index 000000000..c6a077028
--- /dev/null
+++ b/src/translator/sentence_ranges.h
@@ -0,0 +1,52 @@
+#ifndef BERGAMOT_SENTENCE_RANGES_H_
+#define BERGAMOT_SENTENCE_RANGES_H_
+
+#include "data/types.h"
+#include <cassert>
+#include <vector>
+
+namespace marian {
+namespace bergamot {
+
+class SentenceRanges {
+  // SentenceRanges stores string_views into a source text, with additional
+  // annotations to mark sentence boundaries.
+  //
+  // Given the availability annotations, this container provides capabilty to
+  // add sentences, and access individual sentences.
+public:
+  typedef std::vector<string_view>::iterator WordIterator;
+
+  void addSentence(std::vector<string_view> &wordRanges);
+  void addSentence(WordIterator begin, WordIterator end);
+
+  void clear() {
+    flatByteRanges_.clear();
+    sentenceBeginIds_.clear();
+  }
+
+  size_t numSentences() const { return sentenceBeginIds_.size(); }
+
+  // Returns a string_view into the ith sentence.
+  string_view sentence(size_t index) const;
+
+private:
+  // A flat storage for string_views. Can be words or sentences.
+  std::vector<string_view> flatByteRanges_;
+
+  // The container grows dynamically with addSentence. size_t marking index is
+  // used to ensure the sentence boundaries stay same while underlying storage
+  // might be changed during reallocation.
+  std::vector<size_t> sentenceBeginIds_;
+
+  // Utility function to extract the string starting at firstWord and ending at
+  // lastWord as a single string-view.
+  string_view sentenceBetween(string_view firstWord,
+                              string_view lastWord) const;
+};
+
+} // namespace bergamot
+
+} // namespace marian
+
+#endif //  BERGAMOT_SENTENCE_RANGES_H_
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 151fcb665..6d68f18d9 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -44,7 +44,7 @@ std::future<Response> Service::translateWithCopy(std::string input) {
 }
 
 std::future<Response> Service::translate(std::string &&input) {
-  // Takes in a blob of text. Segments and std::vector<TokenRanges> are
+  // Takes in a blob of text. Segments and SentenceRanges are
   // extracted from the input (blob of text) and used to construct a Request
   // along with a promise. promise value is set by the worker completing a
   // request.
@@ -57,16 +57,15 @@ std::future<Response> Service::translate(std::string &&input) {
   // returns future corresponding to the promise.
 
   Segments segments;
-  std::vector<TokenRanges> sourceTokenRanges;
-  text_processor_.process(input, segments, sourceTokenRanges);
+  SentenceRanges sourceRanges;
+  text_processor_.process(input, segments, sourceRanges);
 
   std::promise<Response> responsePromise;
   auto future = responsePromise.get_future();
 
-  Ptr<Request> request =
-      New<Request>(requestId_++, /* lineNumberBegin = */ 0, vocabs_,
-                   std::move(input), std::move(segments),
-                   std::move(sourceTokenRanges), std::move(responsePromise));
+  Ptr<Request> request = New<Request>(
+      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(input),
+      std::move(segments), std::move(sourceRanges), std::move(responsePromise));
 
   batcher_.addWholeRequest(request);
 
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index 8114855bb..ee13dbdfa 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -1,4 +1,5 @@
 #include "text_processor.h"
+#include "sentence_ranges.h"
 #include "data/types.h"
 #include "definitions.h"
 
@@ -10,9 +11,9 @@ namespace marian {
 namespace bergamot {
 
 Segment TextProcessor::tokenize(const string_view &segment,
-                                TokenRanges &tokenRanges) {
+                                std::vector<string_view> &wordRanges) {
   return vocabs_->front()->encodeWithByteRanges(
-      segment, tokenRanges, /*addEOS=*/false, /*inference=*/true);
+      segment, wordRanges, /*addEOS=*/false, /*inference=*/true);
 }
 
 TextProcessor::TextProcessor(std::vector<Ptr<Vocab const>> &vocabs,
@@ -26,7 +27,7 @@ TextProcessor::TextProcessor(std::vector<Ptr<Vocab const>> &vocabs,
 }
 
 void TextProcessor::process(const string_view &query, Segments &segments,
-                            std::vector<TokenRanges> &sourceRanges) {
+                            SentenceRanges &sourceRanges) {
 
   auto sentenceStream = sentence_splitter_.createSentenceStream(query);
   std::string_view sentenceStringPiece;
@@ -34,21 +35,22 @@ void TextProcessor::process(const string_view &query, Segments &segments,
   while (sentenceStream >> sentenceStringPiece) {
     marian::string_view sentence(sentenceStringPiece.data(),
                                  sentenceStringPiece.size());
-    TokenRanges tokenRanges;
-    Segment segment = tokenize(sentence, tokenRanges);
+
+    std::vector<string_view> wordRanges;
+    Segment segment = tokenize(sentence, wordRanges);
 
     // There are some cases where SentencePiece or vocab returns no words
     // after normalization. 0 prevents any empty entries from being added.
     if (segment.size() > 0) {
       // Truncate segment into max_input_size segments.
-      truncate(segment, tokenRanges, segments, sourceRanges);
+      truncate(segment, wordRanges, segments, sourceRanges);
     }
   }
 }
 
-void TextProcessor::truncate(Segment &segment, TokenRanges &tokenRanges,
-                             Segments &segments,
-                             std::vector<TokenRanges> &sourceRanges) {
+void TextProcessor::truncate(Segment &segment,
+                             std::vector<string_view> &wordRanges,
+                             Segments &segments, SentenceRanges &sourceRanges) {
   for (int offset = 0; offset < segment.size();
        offset += max_input_sentence_tokens_) {
     auto start = segment.begin() + offset;
@@ -59,8 +61,8 @@ void TextProcessor::truncate(Segment &segment, TokenRanges &tokenRanges,
     segments.emplace_back(start, start + diff);
     segments.back().push_back(sourceEosId());
 
-    auto astart = tokenRanges.begin() + offset;
-    sourceRanges.emplace_back(astart, astart + diff);
+    auto astart = wordRanges.begin() + offset;
+    sourceRanges.addSentence(astart, astart + diff);
   }
 }
 
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index 111ae009b..d366bf278 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -1,6 +1,7 @@
 #ifndef SRC_BERGAMOT_TEXT_PROCESSOR_H_
 #define SRC_BERGAMOT_TEXT_PROCESSOR_H_
 
+#include "sentence_ranges.h"
 #include "data/types.h"
 #include "data/vocab.h"
 #include "definitions.h"
@@ -23,16 +24,17 @@ class TextProcessor {
   explicit TextProcessor(std::vector<Ptr<Vocab const>> &vocabs, Ptr<Options>);
 
   void process(const string_view &query, Segments &segments,
-               std::vector<TokenRanges> &sourceRanges);
+               SentenceRanges &sourceRanges);
 
 private:
   // Tokenizes an input string, returns Words corresponding. Loads the
   // corresponding byte-ranges into tokenRanges.
-  Segment tokenize(const string_view &input, TokenRanges &tokenRanges);
+  Segment tokenize(const string_view &input,
+                   std::vector<string_view> &tokenRanges);
 
   // Truncate sentence into max_input_size segments.
-  void truncate(Segment &sentence, TokenRanges &tokenRanges, Segments &segments,
-                std::vector<TokenRanges> &sourceRanges);
+  void truncate(Segment &sentence, std::vector<string_view> &tokenRanges,
+                Segments &segments, SentenceRanges &sourceRanges);
 
   // shorthand, used only in truncate()
   const Word sourceEosId() const { return vocabs_->front()->getEosId(); }

From 0296a38cd48970234753cf4f95e7d46269aa089f Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 00:45:19 +0000
Subject: [PATCH 121/442] Bunch of integers on containers to size_ts

---
 src/translator/batcher.cpp        | 8 ++++----
 src/translator/batcher.h          | 4 ++--
 src/translator/request.cpp        | 4 ++--
 src/translator/request.h          | 8 ++++----
 src/translator/service.cpp        | 4 ----
 src/translator/service.h          | 4 ++--
 src/translator/text_processor.cpp | 8 ++++----
 src/translator/text_processor.h   | 4 ++--
 8 files changed, 20 insertions(+), 24 deletions(-)

diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 657a78ab1..bb223c982 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -16,7 +16,7 @@ Batcher::Batcher(Ptr<Options> options) {
 }
 
 void Batcher::addSentenceWithPriority(RequestSentence &sentence) {
-  int bucket_id = sentence.numTokens();
+  size_t bucket_id = sentence.numTokens();
   assert(bucket_id < bucket_.size());
   bucket_[bucket_id].insert(sentence);
 }
@@ -29,9 +29,9 @@ bool Batcher::cleaveBatch(Batch &batch) {
   // implementation should at least be as fast as marian's maxi-batch with full
   // corpus size as maxi-batch size.
   batch.clear();
-  int paddedBatchSize = 0;
+  size_t paddedBatchSize = 0;
 
-  for (int length = 0; length < bucket_.size(); length++) {
+  for (size_t length = 0; length < bucket_.size(); length++) {
     auto p = bucket_[length].begin();
     while (p != bucket_[length].end()) {
       paddedBatchSize = (batch.size() + 1) * length;
@@ -52,7 +52,7 @@ bool Batcher::cleaveBatch(Batch &batch) {
 }
 
 void Batcher::addWholeRequest(Ptr<Request> request) {
-  for (int i = 0; i < request->numSegments(); i++) {
+  for (size_t i = 0; i < request->numSegments(); i++) {
     RequestSentence requestSentence(i, request);
     addSentenceWithPriority(requestSentence);
   }
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index adf2dbe12..1b1bc21f2 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -30,9 +30,9 @@ class Batcher {
   bool operator>>(Batch &batch); // alias
 
 private:
-  unsigned int miniBatchWords;
+  size_t miniBatchWords;
   std::vector<std::set<RequestSentence>> bucket_;
-  unsigned int batchNumber_{0};
+  size_t batchNumber_{0};
 };
 
 } // namespace bergamot
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 8c858e6d6..42dfb35f3 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -1,7 +1,7 @@
 #include "request.h"
-#include "sentence_ranges.h"
 #include "definitions.h"
 #include "response.h"
+#include "sentence_ranges.h"
 
 #include "common/logging.h"
 
@@ -11,7 +11,7 @@ namespace marian {
 namespace bergamot {
 
 // -----------------------------------------------------------------
-Request::Request(unsigned int Id, int lineNumberBegin,
+Request::Request(size_t Id, size_t lineNumberBegin,
                  std::vector<Ptr<Vocab const>> &vocabs, std::string &&source,
                  Segments &&segments, SentenceRanges &&sourceRanges,
                  std::promise<Response> responsePromise)
diff --git a/src/translator/request.h b/src/translator/request.h
index 0bc3c272a..3909019c7 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -17,9 +17,9 @@
 #ifndef SRC_BERGAMOT_REQUEST_H_
 #define SRC_BERGAMOT_REQUEST_H_
 
-#include "sentence_ranges.h"
 #include "definitions.h"
 #include "response.h"
+#include "sentence_ranges.h"
 
 #include "common/logging.h"
 #include "data/types.h"
@@ -35,7 +35,7 @@ namespace bergamot {
 
 class Request {
 public:
-  Request(unsigned int Id, int lineNumberBegin,
+  Request(size_t Id, size_t lineNumberBegin,
           std::vector<Ptr<Vocab const>> &vocabs_, std::string &&source,
           Segments &&segments, SentenceRanges &&sourceTokenRanges,
           std::promise<Response> responsePromise);
@@ -64,8 +64,8 @@ class Request {
   void completeRequest();
 
 private:
-  unsigned int Id_;
-  int lineNumberBegin_;
+  size_t Id_;
+  size_t lineNumberBegin_;
 
   // Multiple translation-workers can concurrently access the same Request. The
   // following atomic atomically operates on the variable holding sentences
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 6d68f18d9..c7d15af95 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -83,17 +83,13 @@ std::future<Response> Service::translate(std::string &&input) {
 }
 
 void Service::stop() {
-  int counter = 0;
   for (auto &worker : workers_) {
     Batch poison = Batch::poison();
     pcqueue_.ProduceSwap(poison);
-    ++counter;
   }
 
-  counter = 0;
   for (auto &worker : workers_) {
     worker.join();
-    ++counter;
   }
 
   workers_.clear(); // Takes care of idempotency.
diff --git a/src/translator/service.h b/src/translator/service.h
index 55b754a2f..6c1d79980 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -45,8 +45,8 @@ class Service {
   ~Service();
 
 private:
-  unsigned int requestId_;
-  int numWorkers_;
+  size_t requestId_;
+  size_t numWorkers_;
 
   // vocabs are used to construct a Request, which later uses it to construct
   // Response (decode from words to string).
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index ee13dbdfa..277b9ca19 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -1,7 +1,7 @@
 #include "text_processor.h"
-#include "sentence_ranges.h"
 #include "data/types.h"
 #include "definitions.h"
+#include "sentence_ranges.h"
 
 #include "common/options.h"
 #include "data/vocab.h"
@@ -51,12 +51,12 @@ void TextProcessor::process(const string_view &query, Segments &segments,
 void TextProcessor::truncate(Segment &segment,
                              std::vector<string_view> &wordRanges,
                              Segments &segments, SentenceRanges &sourceRanges) {
-  for (int offset = 0; offset < segment.size();
+  for (size_t offset = 0; offset < segment.size();
        offset += max_input_sentence_tokens_) {
     auto start = segment.begin() + offset;
 
-    unsigned int left = segment.size() - offset;
-    unsigned int diff = std::min(max_input_sentence_tokens_, left);
+    size_t left = segment.size() - offset;
+    size_t diff = std::min(max_input_sentence_tokens_, left);
 
     segments.emplace_back(start, start + diff);
     segments.back().push_back(sourceEosId());
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index d366bf278..ad88ab906 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -1,10 +1,10 @@
 #ifndef SRC_BERGAMOT_TEXT_PROCESSOR_H_
 #define SRC_BERGAMOT_TEXT_PROCESSOR_H_
 
-#include "sentence_ranges.h"
 #include "data/types.h"
 #include "data/vocab.h"
 #include "definitions.h"
+#include "sentence_ranges.h"
 
 #include "sentence_splitter.h"
 
@@ -41,7 +41,7 @@ class TextProcessor {
 
   std::vector<Ptr<Vocab const>> *vocabs_;
   SentenceSplitter sentence_splitter_;
-  unsigned int max_input_sentence_tokens_;
+  size_t max_input_sentence_tokens_;
 };
 
 } // namespace bergamot

From 69201ba44ccd1611fe03ce5a6d225cad118c2ba1 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 00:54:30 +0000
Subject: [PATCH 122/442] Unify options with marian

Service specific options are renamed to align with marian-option naming
as follows:

1. max-input-sentence-tokens -> max-length-break (There's a
   max-length-crop in marian, this is the same, except breaks into
   multiple sentences than truncate/crop).
2. max-input-tokens -> mini-batch-words.
---
 src/translator/batcher.cpp        |  4 ++--
 src/translator/parser.h           |  7 +------
 src/translator/text_processor.cpp | 11 +++++------
 src/translator/text_processor.h   |  2 +-
 4 files changed, 9 insertions(+), 15 deletions(-)

diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index bb223c982..3a384d441 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -7,8 +7,8 @@ namespace marian {
 namespace bergamot {
 
 Batcher::Batcher(Ptr<Options> options) {
-  miniBatchWords = options->get<int>("max-input-tokens");
-  bucket_.resize(options->get<int>("max-input-sentence-tokens") + 1);
+  miniBatchWords = options->get<int>("mini-batch-words");
+  bucket_.resize(options->get<int>("max-length-break") + 1);
   ABORT_IF(
       miniBatchWords < bucket_.size() - 1,
       "max-input-tokens cannot be less than than max-input-sentence-tokens, "
diff --git a/src/translator/parser.h b/src/translator/parser.h
index 606b6a47b..7fb53e1e1 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -16,14 +16,9 @@ inline marian::ConfigParser createConfigParser() {
                             "[paragraph, sentence, wrapped_text]", "paragraph");
 
   cp.addOption<int>(
-      "--max-input-sentence-tokens", "Bergamot Options",
+      "--max-length-break", "Bergamot Options",
       "Maximum input tokens to be processed in a single sentence.", 128);
 
-  cp.addOption<int>("--max-input-tokens", "Bergamot Options",
-                    "Maximum input tokens in a batch. control for"
-                    "Bergamot Queue",
-                    1024);
-
   return cp;
 }
 
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index 277b9ca19..9d6733e24 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -20,10 +20,9 @@ TextProcessor::TextProcessor(std::vector<Ptr<Vocab const>> &vocabs,
                              Ptr<Options> options)
     : vocabs_(&vocabs), sentence_splitter_(options) {
 
-  max_input_sentence_tokens_ = options->get<int>("max-input-sentence-tokens");
-  max_input_sentence_tokens_ = max_input_sentence_tokens_ - 1;
-  ABORT_IF(max_input_sentence_tokens_ < 0,
-           "max-input-sentence-tokens cannot be < 0");
+  max_length_break_ = options->get<int>("max-length-break");
+  max_length_break_ = max_length_break_ - 1;
+  ABORT_IF(max_length_break_ < 0, "max-length-break cannot be < 0");
 }
 
 void TextProcessor::process(const string_view &query, Segments &segments,
@@ -52,11 +51,11 @@ void TextProcessor::truncate(Segment &segment,
                              std::vector<string_view> &wordRanges,
                              Segments &segments, SentenceRanges &sourceRanges) {
   for (size_t offset = 0; offset < segment.size();
-       offset += max_input_sentence_tokens_) {
+       offset += max_length_break_) {
     auto start = segment.begin() + offset;
 
     size_t left = segment.size() - offset;
-    size_t diff = std::min(max_input_sentence_tokens_, left);
+    size_t diff = std::min(max_length_break_, left);
 
     segments.emplace_back(start, start + diff);
     segments.back().push_back(sourceEosId());
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index ad88ab906..4cd176126 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -41,7 +41,7 @@ class TextProcessor {
 
   std::vector<Ptr<Vocab const>> *vocabs_;
   SentenceSplitter sentence_splitter_;
-  size_t max_input_sentence_tokens_;
+  size_t max_length_break_;
 };
 
 } // namespace bergamot

From fba44bec8f754e0a17c7bbc76d50697121bf97a3 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 01:05:20 +0000
Subject: [PATCH 123/442] Improving Batcher error message with new option names

---
 src/translator/batcher.cpp | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 3a384d441..f671ffc58 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -9,10 +9,9 @@ namespace bergamot {
 Batcher::Batcher(Ptr<Options> options) {
   miniBatchWords = options->get<int>("mini-batch-words");
   bucket_.resize(options->get<int>("max-length-break") + 1);
-  ABORT_IF(
-      miniBatchWords < bucket_.size() - 1,
-      "max-input-tokens cannot be less than than max-input-sentence-tokens, "
-      "batcher fail");
+  ABORT_IF(bucket_.size() - 1 > miniBatchWords,
+           "Fatal: max-length-break > mini-batch-words  will lead to sentences "
+           "longer than what can fit in a batch.");
 }
 
 void Batcher::addSentenceWithPriority(RequestSentence &sentence) {

From c205c82585752b32f6f60191edcbb8bdebd687f7 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 01:12:30 +0000
Subject: [PATCH 124/442] Updates to README with option changes

---
 README.md | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/README.md b/README.md
index 52f60b287..2019fbfa5 100644
--- a/README.md
+++ b/README.md
@@ -42,18 +42,23 @@ ARGS=(
     --beam-size 1 --skip-cost --shortlist $MODEL_DIR/lex.s2t.gz 50 50 --int8shiftAlphaAll 
 
     # Number of CPU threads (workers to launch). Parallelizes over cores and improves speed.
+    # A value of 0 allows a path with no worker thread-launches and a single-thread.
     --cpu-threads 4
 
-    # Hyperparameters of how many tokens to be accounted for in a batch and maximum tokens in a sentence.
-    --max-input-sentence-tokens 1024 --max-input-tokens 1024 
+    # Maximum size of a sentence allowed. If a sentence is above this length,
+    # it's broken into pieces of less than or equal to this size.
+    --max-length-break 1024  
+
+    # Maximum number of tokens that can be fit in a batch. The optimal value 
+    # for the parameter is dependant on hardware and can be obtained by running
+    # with variations and benchmarking.
+    --mini-batch-words 1024 
 
     # Three modes are supported
     #   - sentence: One sentence per line
     #   - paragraph: One paragraph per line.
-    #   - wrapped text: Paragraphs are separated by empty line.
-
+    #   - wrapped_text: Paragraphs are separated by empty line.
     --ssplit-mode paragraph 
-
 )
 
 ./app/service-cli "${ARGS[@]}" < path-to-input-file

From 44a44fa1562319178e5246205ef6c5b5fbc49f10 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 11:48:00 +0000
Subject: [PATCH 125/442] CMake build with submodule recursive clones

---
 CMakeLists.txt | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ce48a9079..820251108 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -16,8 +16,22 @@ option(USE_SENTENCEPIECE "Download and compile SentencePiece" ON)
 option(USE_STATIC_LIBS "Link statically against non-system libs" ON)
 option(USE_MKL "Compile with MKL support" ON)
 
-execute_process(COMMAND git submodule update --init --recursive --no-fetch
-                WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
+# Documentation: https://cliutils.gitlab.io/modern-cmake/chapters/projects/submodule.html
+# Ensures the submodules are set correctly during a build.
+find_package(Git QUIET)
+if(GIT_FOUND AND EXISTS "${PROJECT_SOURCE_DIR}/.git")
+# Update submodules as needed
+    option(GIT_SUBMODULE "Check submodules during build" ON)
+    if(GIT_SUBMODULE)
+        message(STATUS "Submodule update")
+        execute_process(COMMAND ${GIT_EXECUTABLE} submodule update --init --recursive
+                        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+                        RESULT_VARIABLE GIT_SUBMOD_RESULT)
+        if(NOT GIT_SUBMOD_RESULT EQUAL "0")
+            message(FATAL_ERROR "git submodule update --init failed with ${GIT_SUBMOD_RESULT}, please checkout submodules")
+        endif()
+    endif()
+endif()
 
 add_subdirectory(3rd_party)
 add_subdirectory(src)

From d005f73cb9c67e39766d19130cb4499c71adb9d0 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 13:10:39 +0000
Subject: [PATCH 126/442] Reverting changes to PCQueue

---
 src/translator/pcqueue.h | 29 -----------------------------
 1 file changed, 29 deletions(-)

diff --git a/src/translator/pcqueue.h b/src/translator/pcqueue.h
index 79d6b75e0..f0b354145 100644
--- a/src/translator/pcqueue.h
+++ b/src/translator/pcqueue.h
@@ -9,7 +9,6 @@
 #include <memory>
 #include <mutex>
 
-#ifdef WITH_PTHREADS
 #ifdef __APPLE__
 #include <mach/mach.h>
 #include <mach/mach_traps.h>
@@ -20,7 +19,6 @@
 #else
 #include <boost/interprocess/sync/interprocess_semaphore.hpp>
 #endif
-#endif // WITH_PTHREADS
 
 #if __GNUC__ >= 3
 #define UTIL_UNLIKELY(x) __builtin_expect(!!(x), 0)
@@ -31,7 +29,6 @@
 namespace marian {
 namespace bergamot {
 
-#ifdef WITH_PTHREADS
 /* OS X Maverick and Boost interprocess were doing "Function not implemented."
  * So this is my own wrapper around the mach kernel APIs.
  */
@@ -117,20 +114,6 @@ inline void WaitSemaphore(Semaphore &on) {
 }
 
 #endif // Apple
-#else // WITH_PTHREADS
-// A dummy Semaphore class that does nothing
-class Semaphore {
-public:
-  explicit Semaphore(unsigned int value) : count(value) {}
-  ~Semaphore() {}
-  void wait() {}
-  void post() {}
-private:
-  unsigned int count;
-};
-
-inline void WaitSemaphore(Semaphore &semaphore) { semaphore.wait(); }
-#endif // WITH_PTHREADS
 
 /**
  * Producer consumer queue safe for multiple producers and multiple consumers.
@@ -151,9 +134,7 @@ template <class T> class PCQueue {
   void Produce(const T &val) {
     WaitSemaphore(empty_);
     {
-    #ifdef WITH_PTHREADS
       std::lock_guard<std::mutex> produce_lock(produce_at_mutex_);
-    #endif
       try {
         *produce_at_ = val;
       } catch (...) {
@@ -170,9 +151,7 @@ template <class T> class PCQueue {
   void ProduceSwap(T &val) {
     WaitSemaphore(empty_);
     {
-    #ifdef WITH_PTHREADS
       std::lock_guard<std::mutex> produce_lock(produce_at_mutex_);
-    #endif
       try {
         std::swap(*produce_at_, val);
       } catch (...) {
@@ -189,9 +168,7 @@ template <class T> class PCQueue {
   T &Consume(T &out) {
     WaitSemaphore(used_);
     {
-    #ifdef WITH_PTHREADS
       std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
-    #endif
       try {
         out = *consume_at_;
       } catch (...) {
@@ -209,9 +186,7 @@ template <class T> class PCQueue {
   T &ConsumeSwap(T &out) {
     WaitSemaphore(used_);
     {
-    #ifdef WITH_PTHREADS
       std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
-    #endif
       try {
         std::swap(out, *consume_at_);
       } catch (...) {
@@ -245,15 +220,11 @@ template <class T> class PCQueue {
 
   // Index for next write in storage_.
   T *produce_at_;
-#ifdef WITH_PTHREADS
   std::mutex produce_at_mutex_;
-#endif
 
   // Index for next read from storage_.
   T *consume_at_;
-#ifdef WITH_PTHREADS
   std::mutex consume_at_mutex_;
-#endif
 };
 
 template <class T> struct UnboundedPage {

From b86f8a7dc2f57b2ff0d21ca16be6a332126d4199 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 17 Feb 2021 12:55:31 +0100
Subject: [PATCH 127/442] Improved README

 - Clears up the spaghetti of model packaging
 - Usage instructions
 - Formatting changes
---
 README.md | 114 ++++++++++++++++++++++++++++++------------------------
 1 file changed, 63 insertions(+), 51 deletions(-)

diff --git a/README.md b/README.md
index 0d55686ff..34131a9a8 100644
--- a/README.md
+++ b/README.md
@@ -16,64 +16,76 @@ make -j
 ```
 
 ### Build WASM
-
-To compile WASM, first download and Install Emscripten using following instructions:
-
-1. Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
-2. Enter the cloned directory: `cd emsdk`
-3. Install the lastest sdk tools: `./emsdk install latest`
-4. Activate the latest sdk tools: `./emsdk activate latest`
-5. Activate path variables: `source ./emsdk_env.sh`
-
-After the successful installation of Emscripten, perform these steps:
-
-```bash
-git clone --recursive https://github.com/browsermt/bergamot-translator
-cd bergamot-translator
-git checkout wasm-integration
-git submodule update --recursive
-mkdir build-wasm
-cd build-wasm
-emcmake cmake -DCOMPILE_WASM=on ../
-emmake make -j
-```
-
-It should generate the artefacts (.js and .wasm files) in `wasm` folder inside build directory ("build-wasm" in this case).
-
-Download the models from `https://github.com/mozilla-applied-ml/bergamot-models`, and place all the desired ones to package in a folder called `models`.
-
-The build also allows packaging files into wasm binary (i.e. preloading in Emscripten’s virtual file system) using cmake
-option `PACKAGE_DIR`. The compile command below packages all the files in PATH directory (in these case, your models) into wasm binary.
+#### Compiling for the first time
+
+1. Download and Install Emscripten using following instructions
+    * Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
+    * Enter the cloned directory: `cd emsdk`
+    * Install the lastest sdk tools: `./emsdk install latest`
+    * Activate the latest sdk tools: `./emsdk activate latest`
+    * Activate path variables: `source ./emsdk_env.sh`
+
+2. Clone the repository and checkout the appropriate branch using these instructions:
+    ```bash
+    git clone https://github.com/browsermt/bergamot-translator
+    cd bergamot-translator
+    git checkout -b wasm-integration origin/wasm-integration
+    git submodule update --init --recursive
+    ```
+
+3. Download models (only required if you want to package files in wasm binary)
+
+    This step is only required if you want to package files (e.g. models, vocabularies etc.)
+    into wasm binary. If you don't then just skip this step.
+
+    The build preloads the files in Emscripten’s virtual file system.
+
+    If you want to use bergamot models, please follow these instructions:
+    ```bash
+    mkdir models
+    git clone https://github.com/mozilla-applied-ml/bergamot-models
+    cp -rf bergamot-models/* models
+    gunzip models/*/*
+    ```
+
+4. Compile
+    1. Create a folder where you want to build all the artefacts (`build-wasm` in this case)
+        ```bash
+        mkdir build-wasm
+        cd build-wasm
+        ```
+
+    2. Compile the artefacts
+        * If you want to package files into wasm binary then execute following commands (Replace `FILES_TO_PACKAGE` with the absolute path of the
+        directory containing the files to be packaged in wasm binary)
+
+            ```bash
+            emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=FILES_TO_PACKAGE ../
+            emmake make -j
+            ```
+
+        * If you don't want to package any file into wasm binary then execute following commands:
+            ```bash
+            emcmake cmake -DCOMPILE_WASM=on ../
+            emmake make -j
+            ```
+
+    The artefacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
+
+#### Recompiling
+As long as you don't update any submodule, just follow steps in `4.ii` to recompile.\
+If you update a submodule, execute following command before executing steps in `4.ii` to recompile.
 ```bash
-emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=/repo/models ../
+git submodule update --init --recursive
 ```
-Files packaged this way are preloaded in the root of the virtual file system.
 
-To package the set of files expected by the test page:
 
-```bash
-mkdir models
-git clone https://github.com/motin/bergamot-models
-cp -r bergamot-models/* models
-gunzip models/*/*
-```
-
-After Editing Files:
-
-```bash
-emmake make -j
-```
-
-After Adding/Removing Files:
-
-```bash
-emcmake cmake -DCOMPILE_WASM=on ../
-emmake make -j
-```
+## How to use
 
 ### Using Native version
 
-The builds generate library that can be integrated to any project. All the public header files are specified in `src` folder. A short example of how to use the APIs is provided in `app/main.cpp` file
+The builds generate library that can be integrated to any project. All the public header files are specified in `src` folder.\
+A short example of how to use the APIs is provided in `app/main.cpp` file.
 
 ### Using WASM version
 

From 72848ba0f6070be711cbf7f8bbbdaf5dd534e9fb Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 13:28:58 +0000
Subject: [PATCH 128/442] Fixes UEdin builds after wasm-integration merge

A bug which crept in during manual merge is now fixed. PCItem -> Batch
on a PCQueue.

docs/marian-integration.md provides instructions to compile successfully
for multithread.
---
 doc/marian-integration.md         | 4 +++-
 src/translator/batch_translator.h | 1 -
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index 8fce52f82..f72543ad3 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -10,7 +10,9 @@ $ git clone https://github.com/browsermt/bergamot-translator
 $ cd bergamot-translator
 $ mkdir build
 $ cd build
-$ cmake ../
+$ cmake .. -DCOMPILE_CUDA=off -DCMAKE_BUILD_TYPE=Release \
+    -DCOMPILE_DECODER_ONLY=off -DCOMPILE_LIBRARY_ONLY=off -DUSE_MKL=on \
+    -DCOMPILE_THREAD_VARIANT=on
 $ make -j
 
 
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index b53bb99f4..78027a2ca 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -47,7 +47,6 @@ class BatchTranslator {
   Ptr<data::ShortlistGenerator const> slgen_;
 
 #ifdef WITH_PTHREADS
-  PCQueue<PCItem> *pcqueue_;
   std::thread thread_;
 #endif
 };

From 47b9db0c455528984e868a03709e0fb4bedddb37 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 13:35:10 +0000
Subject: [PATCH 129/442] Documentation formatting/syntax fix

---
 doc/marian-integration.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index f72543ad3..6d603a332 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -14,6 +14,7 @@ $ cmake .. -DCOMPILE_CUDA=off -DCMAKE_BUILD_TYPE=Release \
     -DCOMPILE_DECODER_ONLY=off -DCOMPILE_LIBRARY_ONLY=off -DUSE_MKL=on \
     -DCOMPILE_THREAD_VARIANT=on
 $ make -j
+```
 
 
 The build will generate the library that can be linked to any project. All the

From 7b10c3548333d8c5b67b1c6c32b1e279647c5e68 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 13:50:42 +0000
Subject: [PATCH 130/442] Hard abort if multithread path launched without
 multithread-support

---
 src/translator/service.cpp | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index a864a8831..986d4a477 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -41,6 +41,11 @@ Service::Service(Ptr<Options> options)
         translator.consumeFrom(pcqueue_);
       });
     }
+#else // WITH_PTHREADS
+    ABORT(
+        "Fatal: Service started requesting multiple threadswhile compiled with "
+        "COMPILE_THREAD_VARIANT=off. Please check your cmake build "
+        "configuration");
 #endif
   }
 }

From 70b57ee3e7ed1f0569d6c838851354753fe88c8d Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 16:38:47 +0000
Subject: [PATCH 131/442] Redundant parser include fixed

---
 src/translator/TranslationModel.cpp | 2 --
 1 file changed, 2 deletions(-)

diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 65b0fc056..45b93d18d 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -16,8 +16,6 @@
 #include "TranslationModel.h"
 #include "translator/parser.h"
 #include "translator/service.h"
-#include "translator/parser.h"
-
 
 std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
   marian::Options options;

From d72343567c978d9a8605193f7027c23edff728b3 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 16:41:04 +0000
Subject: [PATCH 132/442] BatchTranslator doesn't do thread_, residue from
 merge removed

---
 src/translator/batch_translator.h | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 78027a2ca..b927c012b 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -45,10 +45,6 @@ class BatchTranslator {
   Ptr<ExpressionGraph> graph_;
   std::vector<Ptr<Scorer>> scorers_;
   Ptr<data::ShortlistGenerator const> slgen_;
-
-#ifdef WITH_PTHREADS
-  std::thread thread_;
-#endif
 };
 
 } // namespace bergamot

From 9feebe5cb2509e137c412a925a8c905ad0a51daa Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 17 Feb 2021 20:06:04 +0100
Subject: [PATCH 133/442] Allow using relative paths for packaging files

 - PACKAGE_DIR cmake option can now accept relative paths
---
 wasm/CMakeLists.txt | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 748762d14..c89e3939d 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -16,7 +16,8 @@ target_compile_options(bergamot-translator-worker PRIVATE ${WASM_COMPILE_FLAGS})
 
 set(LINKER_FLAGS "--bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1")
 if (NOT PACKAGE_DIR STREQUAL "")
-  set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${PACKAGE_DIR}@/")
+  get_filename_component(REALPATH_PACKAGE_DIR ${PACKAGE_DIR} REALPATH BASE_DIR ${CMAKE_BINARY_DIR})
+  set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${REALPATH_PACKAGE_DIR}@/")
 endif()
 
 set_target_properties(bergamot-translator-worker PROPERTIES

From b9d081dd4552ac8f6baf536046b1eebbfb7792ef Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 19:25:19 +0000
Subject: [PATCH 134/442] Temporary: Updating marian-dev to wasm branch

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 2f6528045..467c43a29 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 2f65280459737c37c270e4ad0b6d41de215d11e0
+Subproject commit 467c43a292a68b7913af2a00d353de97c1740f92

From d249dcbfaa45f44cda022482b314143a9b9e2b80 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 17 Feb 2021 21:15:35 +0000
Subject: [PATCH 135/442] Build doc updated with wasm-branch compatible command

---
 doc/marian-integration.md | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index 6d603a332..b7089c06a 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -11,8 +11,9 @@ $ cd bergamot-translator
 $ mkdir build
 $ cd build
 $ cmake .. -DCOMPILE_CUDA=off -DCMAKE_BUILD_TYPE=Release \
-    -DCOMPILE_DECODER_ONLY=off -DCOMPILE_LIBRARY_ONLY=off -DUSE_MKL=on \
-    -DCOMPILE_THREAD_VARIANT=on
+     -DCOMPILE_DECODER_ONLY=on -DUSE_MKL=on -DCOMPILE_THREAD_VARIANT=on \
+     -DUSE_WASM_COMPATIBLE_BLAS=off -DCOMPILE_MAIN_EIGEN=off
+
 $ make -j
 ```
 

From b75e72e65d55307ff5f4e3b499680609031c686d Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 18 Feb 2021 10:42:06 +0100
Subject: [PATCH 136/442] Added more explanation for FILES_TO_PACKAGE in README

---
 README.md | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/README.md b/README.md
index 34131a9a8..d9e071971 100644
--- a/README.md
+++ b/README.md
@@ -33,14 +33,14 @@ make -j
     git submodule update --init --recursive
     ```
 
-3. Download models (only required if you want to package files in wasm binary)
+3. Download files (only required if you want to package files in wasm binary)
 
     This step is only required if you want to package files (e.g. models, vocabularies etc.)
     into wasm binary. If you don't then just skip this step.
 
     The build preloads the files in Emscripten’s virtual file system.
 
-    If you want to use bergamot models, please follow these instructions:
+    If you want to package bergamot project specific models, please follow these instructions:
     ```bash
     mkdir models
     git clone https://github.com/mozilla-applied-ml/bergamot-models
@@ -56,13 +56,15 @@ make -j
         ```
 
     2. Compile the artefacts
-        * If you want to package files into wasm binary then execute following commands (Replace `FILES_TO_PACKAGE` with the absolute path of the
+        * If you want to package files into wasm binary then execute following commands (Replace `FILES_TO_PACKAGE` with the path of the
         directory containing the files to be packaged in wasm binary)
 
             ```bash
             emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=FILES_TO_PACKAGE ../
             emmake make -j
             ```
+            e.g. If you want to package bergamot project specific models (downloaded using step 3 above) then
+            replace `FILES_TO_PACKAGE` with `../models`
 
         * If you don't want to package any file into wasm binary then execute following commands:
             ```bash

From ca9aa64926684f41a36c29e4cba8b23ac9e550ec Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 18 Feb 2021 11:07:31 +0000
Subject: [PATCH 137/442] Switch to work with ssplit-cpp both pcre2 and pcrecpp

---
 src/translator/sentence_splitter.cpp | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/src/translator/sentence_splitter.cpp b/src/translator/sentence_splitter.cpp
index 0f9be019a..037012575 100644
--- a/src/translator/sentence_splitter.cpp
+++ b/src/translator/sentence_splitter.cpp
@@ -1,7 +1,7 @@
+#include "sentence_splitter.h"
 #include "common/cli_helper.h"
 #include "common/logging.h"
 #include "common/options.h"
-#include "sentence_splitter.h"
 #include <string>
 
 namespace marian {
@@ -30,8 +30,9 @@ SentenceSplitter::SentenceSplitter(marian::Ptr<marian::Options> options)
 
 ug::ssplit::SentenceStream
 SentenceSplitter::createSentenceStream(const string_view &input) {
-  return std::move(ug::ssplit::SentenceStream(input.data(), input.size(),
-                                              this->ssplit_, mode_));
+  std::string_view input_converted(input.data(), input.size());
+  return std::move(
+      ug::ssplit::SentenceStream(input_converted, this->ssplit_, mode_));
 }
 
 ug::ssplit::SentenceStream::splitmode

From fbff7389d1f735272133db68f5029772dabc78c0 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 18 Feb 2021 11:20:01 +0000
Subject: [PATCH 138/442] Temporary: Switch to abhi-agg/ssplit-cpp@wasm

---
 3rd_party/ssplit-cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 01e71b496..16864967b 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 01e71b4964fdc351f932a7a23cab4cb80b9698e8
+Subproject commit 16864967b7313e76e3b107d11ec39d8d5cedff1e

From c2371dd771b36be8396da5da156fadc4bde0e744 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 18 Feb 2021 12:20:06 +0100
Subject: [PATCH 139/442] Replaced "build-wasm-docker" with "build-wasm"

 - Now things are consistent with the top level README
   instructions that suggest to build in "build-wasm"
   folder
---
 wasm/test_page/start_server.sh | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/wasm/test_page/start_server.sh b/wasm/test_page/start_server.sh
index b0b5be1b2..b83344b8a 100644
--- a/wasm/test_page/start_server.sh
+++ b/wasm/test_page/start_server.sh
@@ -1,8 +1,8 @@
 #!/bin/bash
 
-cp ../../build-wasm-docker/wasm/bergamot-translator-worker.data .
-cp ../../build-wasm-docker/wasm/bergamot-translator-worker.js .
-cp ../../build-wasm-docker/wasm/bergamot-translator-worker.wasm .
-cp ../../build-wasm-docker/wasm/bergamot-translator-worker.worker.js .
+cp ../../build-wasm/wasm/bergamot-translator-worker.data .
+cp ../../build-wasm/wasm/bergamot-translator-worker.js .
+cp ../../build-wasm/wasm/bergamot-translator-worker.wasm .
+cp ../../build-wasm/wasm/bergamot-translator-worker.worker.js .
 npm install
 node bergamot-httpserver.js
\ No newline at end of file

From 79571bada5ef8fdd20ae623521f0f64ac42ed4ff Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 18 Feb 2021 12:48:45 +0100
Subject: [PATCH 140/442] Improved wasm/README

 - Clarified that the Demo and API usage section assumes
   bergamot models were packaged into wasm binary
 - Formatting changes
---
 wasm/README.md | 40 +++++++++++++++++++++-------------------
 1 file changed, 21 insertions(+), 19 deletions(-)

diff --git a/wasm/README.md b/wasm/README.md
index bb431447c..23564b971 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -1,13 +1,14 @@
 ## Using Bergamot Translator in JavaScript
 The example file `bergamot.html` in the folder `test_page` demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
-This example assumes that files were packaged in wasm binary.
 
-A brief summary is here though:
+Please note that everything below assumes that the [bergamot project specific model files](https://github.com/mozilla-applied-ml/bergamot-models) were packaged in wasm binary (using the compile instructions given in the top level README).
+
+### Using JS APIs
 
 ```js
 // The model configuration as YAML formatted string. For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
 // This example captures the most relevant options: model file, vocabulary files and shortlist file
-const modelConfig = "{\"models\":[\"/model.npz\"],\"vocabs\":[\"/vocab.esen.spm\",\"/vocab.esen.spm\"],\"shortlist\":[\"/lex.s2t\"],\"beam-size\":1}";
+const modelConfig = "{\"models\":[\"/esen/model.esen.npz\"],\"vocabs\":[\"/esen/vocab.esen.spm\",\"/esen/vocab.esen.spm\"],\"shortlist\":[\"/esen/lex.esen.s2t\"],\"beam-size\":1}";
 
 // Instantiate the TranslationModel
 const model = new Module.TranslationModel(modelConfig);
@@ -33,29 +34,30 @@ request.delete();
 input.delete();
 ```
 
-You can also see everything in action by following the next steps:
+### Demo (see everything in action)
+
 * Start the test webserver (ensure you have the latest nodejs installed)
-```bash
-cd test_page
-bash start_server.sh
-```
+    ```bash
+    cd test_page
+    bash start_server.sh
+    ```
 * Open any of the browsers below
     * Firefox Nightly +87: make sure the following prefs are on (about:config)
-    ```
-    dom.postMessage.sharedArrayBuffer.bypassCOOP_COEP.insecure.enabled = true
-    javascript.options.wasm_simd = true
-    javascript.options.wasm_simd_wormhole = true
-    ```
+        ```
+        dom.postMessage.sharedArrayBuffer.bypassCOOP_COEP.insecure.enabled = true
+        javascript.options.wasm_simd = true
+        javascript.options.wasm_simd_wormhole = true
+        ```
 
     * Chrome Canary +90: start with the following argument
-    ```
-    --js-flags="--experimental-wasm-simd"
-    ```
+        ```
+        --js-flags="--experimental-wasm-simd"
+        ```
 
 * Browse to the following page:
-```
-http://localhost:8000/bergamot.html
-```
+    ```
+    http://localhost:8000/bergamot.html
+    ```
 
 * Run some translations:
     * Choose a model and press `Load Model`

From 51f702ea6c849687488d6478b3bc428f0ad033ae Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Thu, 18 Feb 2021 15:02:44 +0200
Subject: [PATCH 141/442] Remove Docker-based builds since they are no more
 reproducible than metal builds. Fixes
 https://github.com/browsermt/bergamot-translator/issues/31

---
 docker/Makefile        | 55 ------------------------------------------
 docker/README.md       | 27 ---------------------
 docker/wasm/Dockerfile | 36 ---------------------------
 3 files changed, 118 deletions(-)
 delete mode 100644 docker/Makefile
 delete mode 100644 docker/README.md
 delete mode 100644 docker/wasm/Dockerfile

diff --git a/docker/Makefile b/docker/Makefile
deleted file mode 100644
index 583a58852..000000000
--- a/docker/Makefile
+++ /dev/null
@@ -1,55 +0,0 @@
-# -*- mode: makefile-gmake; indent-tabs-mode: true; tab-width: 4 -*-
-SHELL   		= bash
-PWD     		= $(shell pwd)
-WASM_IMAGE	    = local/bergamot-translator-build-wasm
-
-all: wasm-image compile-wasm
-
-# Build the Docker image for WASM builds
-wasm-image:
-	docker build -t local/bergamot-translator-build-wasm ./wasm/
-
-# Commands for compilation:
-cmake_cmd  = cmake
-
-wasm_cmake_cmd = ${cmake_cmd}
-wasm_cmake_cmd += -DCOMPILE_WASM=on
-wasm_cmake_cmd += -DProtobuf_INCLUDE_DIR=/usr/opt/protobuf-wasm-lib/dist/include
-wasm_cmake_cmd += -DProtobuf_LIBRARY=/usr/opt/protobuf-wasm-lib/dist/lib/libprotobuf.a
-wasm_cmake_cmd += -DPACKAGE_DIR=/repo/models
-
-make_cmd  = make
-#make_cmd += VERBOSE=1
-
-# ... and running things on Docker
-docker_mounts  = ${PWD}/..:/repo
-docker_mounts += ${HOME}/.ccache:/.ccache
-run_on_docker  = docker run --rm
-run_on_docker += $(addprefix -v, ${docker_mounts})
-run_on_docker += ${INTERACTIVE_DOCKER_SESSION}
-
-${HOME}/.ccache:
-	mkdir -p $@
-
-# Remove the bergamot-translator WASM build dir, forcing a clean compilation attempt
-clean-wasm: BUILD_DIR = /repo/build-wasm-docker
-clean-wasm: ${HOME}/.ccache
-	${run_on_docker} ${WASM_IMAGE} bash -c '(rm -rf ${BUILD_DIR} || true)'
-
-# Compile bergamot-translator to WASM
-compile-wasm: BUILD_DIR = /repo/build-wasm-docker
-compile-wasm: ${HOME}/.ccache
-	${run_on_docker} ${WASM_IMAGE} bash -c 'mkdir -p ${BUILD_DIR} && \
-cd ${BUILD_DIR} && \
-(emcmake ${wasm_cmake_cmd} .. && \
-(emmake ${make_cmd}) || \
-rm CMakeCache.txt)'
-
-# Start interactive shells for development / debugging purposes
-native-shell: INTERACTIVE_DOCKER_SESSION = -it
-native-shell:
-	${run_on_docker} ${NATIVE_IMAGE} bash
-
-wasm-shell: INTERACTIVE_DOCKER_SESSION = -it
-wasm-shell:
-	${run_on_docker} ${WASM_IMAGE} bash
diff --git a/docker/README.md b/docker/README.md
deleted file mode 100644
index d98456a54..000000000
--- a/docker/README.md
+++ /dev/null
@@ -1,27 +0,0 @@
-## WASM
-
-Prepare docker image for WASM compilation:
-
-```bash
-make wasm-image
-```
-
-Compile to wasm:
-
-```bash
-make compile-wasm
-```
-
-## Debugging
-
-Remove the marian-decoder build dir, forcing the next compilation attempt to start from scratch:
-
-```bash
-make clean-wasm
-```
-
-Enter a docker container shell for manually running commands:
-
-```bash
-make wasm-shell
-```
diff --git a/docker/wasm/Dockerfile b/docker/wasm/Dockerfile
deleted file mode 100644
index f309662a7..000000000
--- a/docker/wasm/Dockerfile
+++ /dev/null
@@ -1,36 +0,0 @@
-FROM emscripten/emsdk:2.0.9
-
-# Install specific version of CMake
-WORKDIR /usr
-RUN wget https://github.com/Kitware/CMake/releases/download/v3.17.2/cmake-3.17.2-Linux-x86_64.tar.gz -qO-\
-    | tar xzf - --strip-components 1
-
-# Install Python and Java (needed for Closure Compiler minification)
-RUN apt-get update \
-    && apt-get install -y \
-    python3 \
-    default-jre
-
-# Deps to compile protobuf from source + the protoc binary which we need natively
-RUN apt-get update -y  && apt-get --no-install-recommends -y install \
-    protobuf-compiler \
-    autoconf \
-    autotools-dev \
-    automake \
-    autogen \
-    libtool && ln -s /usr/bin/libtoolize /usr/bin/libtool \
-    && mkdir -p /usr/opt \
-    && cd /usr/opt \
-    && git clone https://github.com/menduz/protobuf-wasm-lib
-
-RUN cd /usr/opt/protobuf-wasm-lib \
-    && /bin/bash -c "BRANCH=v3.6.1 ./prepare.sh"
-RUN cd /usr/opt/protobuf-wasm-lib/protobuf \
-    && bash -x ../build.sh
-RUN cp /usr/bin/protoc /usr/opt/protobuf-wasm-lib/dist/bin/protoc
-
-RUN apt-get --no-install-recommends -y install \
-    libprotobuf-dev
-
-# Necessary for benchmarking
-RUN pip3 install sacrebleu

From 5dcbb721faadc55ded424b6df5b74854e54d600e Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 22 Feb 2021 18:03:53 +0100
Subject: [PATCH 142/442] Update ssplit submodule to master branch

 - This submodule brings pcre2 lib compiled from sources
---
 3rd_party/ssplit-cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 16864967b..432208826 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 16864967b7313e76e3b107d11ec39d8d5cedff1e
+Subproject commit 432208826ee27e7b3984b53774b1a16d74256d77

From fa4a1ed67d2313efc88d4ac7930da08ab5e882b7 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 22 Feb 2021 18:28:32 +0100
Subject: [PATCH 143/442] Adapted model config in test example of bergamot

 - Replaced deprecated names with new names
      mini-batch-words and max-length-break
 - Set cpu-threads to 0
---
 wasm/test_page/bergamot.html | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 795654495..8b3cfa83d 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -81,12 +81,12 @@
 beam-size: 1
 normalize: 1.0
 word-penalty: 0
-max-input-sentence-tokens: 128
-max-input-tokens: 1024
+max-length-break: 128
+mini-batch-words: 1024
 workspace: 128
 max-length-factor: 2.0
 skip-cost: true
-cpu-threads: 1
+cpu-threads: 0
 quiet: true
 quiet-translation: true
 shortlist:

From 462a850d8ae554e835f0e3e64aef897aa3f93dad Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 22 Feb 2021 18:48:59 +0100
Subject: [PATCH 144/442] Changed Sentences to Paragraphs in test page of WASM

 - Sentence Splitter works now => No more sentence splitting in
   test code
 - Changed example to include some paragraphs
---
 wasm/test_page/bergamot.html | 43 ++++++++++++++++--------------------
 1 file changed, 19 insertions(+), 24 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 8b3cfa83d..d529e42ce 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -38,16 +38,11 @@
 <div id="divtranslation">
     <label for="from">From</label>
     <textarea id="from" name="from">
-Una estrategia republicana para obstaculizar la reelecci�n de Obama
-Los dirigentes republicanos justificaron su pol�tica por la necesidad de luchar contra el fraude electoral.
-Ahora bien, el Centro Brennan considera esto �ltimo un mito y afirma que el fraude electoral es menos frecuente en los Estados Unidos que el n�mero de personas que mueren a causa de la ca�da de un rayo.
-De hecho, los abogados republicanos no han encontrado m�s que 300 casos de fraude electoral en los Estados Unidos en diez a�os.
-Una cosa es cierta: esas nuevas disposiciones afectar�n negativamente a la tasa de participaci�n.
-En ese sentido, estas medidas minar�n en parte el sistema democr�tico americano.
-Al contrario de lo que ocurre en Canad�, los estados americanos son responsables de la organizaci�n de las elecciones federales en los Estados Unidos.
-Y en esa misma l�nea una mayor�a de los gobiernos americanos promulgaron, a partir de 2009, nuevas leyes que dificultaban el proceso de inscripci�n o de votaci�n.
-Este fen�meno se ha extendido tras las elecciones de noviembre de 2010, que vieron el aumento de 675 nuevos representantes republicanos en 26 estados.
-En consecuencia, durante el a�o 2011 se introdujeron 180 proyectos de ley que restring�an el ejercicio del derecho de voto en 41 estados.
+Una estrategia republicana para obstaculizar la reelecci�n de Obama. Los dirigentes republicanos justificaron su pol�tica por la necesidad de luchar contra el fraude electoral.
+Ahora bien, el Centro Brennan considera esto �ltimo un mito y afirma que el fraude electoral es menos frecuente en los Estados Unidos que el n�mero de personas que mueren a causa de la ca�da de un rayo. De hecho, los abogados republicanos no han encontrado m�s que 300 casos de fraude electoral en los Estados Unidos en diez a�os. Una cosa es cierta: esas nuevas disposiciones afectar�n negativamente a la tasa de participaci�n.
+En ese sentido, estas medidas minar�n en parte el sistema democr�tico americano.
+Al contrario de lo que ocurre en Canad�, los estados americanos son responsables de la organizaci�n de las elecciones federales en los Estados Unidos. Y en esa misma l�nea una mayor�a de los gobiernos americanos promulgaron, a partir de 2009, nuevas leyes que dificultaban el proceso de inscripci�n o de votaci�n.
+Este fen�meno se ha extendido tras las elecciones de noviembre de 2010, que vieron el aumento de 675 nuevos representantes republicanos en 26 estados. En consecuencia, durante el a�o 2011 se introdujeron 180 proyectos de ley que restring�an el ejercicio del derecho de voto en 41 estados.
     </textarea>
     <br><br>
     <label for="to">To</label>
@@ -112,19 +107,19 @@
     model = new Module.TranslationModel(modelConfig);
   }
 
-  const translate = (sentences) => {
+  const translate = (paragraphs) => {
 
     // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
     var request = new Module.TranslationRequest();
     let input = new Module.VectorString;
 
     // Initialize the input
-    sentences.forEach(sentence => {
-      // prevent empty sentences - it breaks the translation
-      if (sentence.trim() === "") {
+    paragraphs.forEach(paragraph => {
+      // prevent empty paragraph - it breaks the translation
+      if (paragraph.trim() === "") {
         return;
       }
-      input.push_back(sentence.trim())
+      input.push_back(paragraph.trim())
     })
     // Access input (just for debugging)
     console.log('Input size=', input.size());
@@ -138,14 +133,14 @@
     let result = model.translate(input, request);
     // Access original and translated text from each entry of vector<TranslationResult>
     //console.log('Result size=', result.size(), ' - TimeDiff - ', (Date.now() - start)/1000);
-    const translatedSentences = [];
+    const translatedParagraphs = [];
     for (let i = 0; i < result.size(); i++) {
-      translatedSentences.push(result.get(i).getTranslatedText());
+      translatedParagraphs.push(result.get(i).getTranslatedText());
     }
-    console.log({ translatedSentences });
+    console.log({ translatedParagraphs });
     request.delete();
     input.delete();
-    return translatedSentences;
+    return translatedParagraphs;
   }
 
   document.querySelector("#load").addEventListener("click", () => {
@@ -160,17 +155,17 @@
 
   const translateCall = () => {
     const text = document.querySelector('#from').value;
-    const sentences = text.split("\n");
+    const paragraphs = text.split("\n");
     let wordCount = 0;
-    sentences.forEach(sentence => {
+    paragraphs.forEach(sentence => {
       wordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
     })
     const start = Date.now();
-    const translatedSentences = translate(sentences);
+    const translatedParagraphs = translate(paragraphs);
     const secs = (Date.now() - start) / 1000;
-    log(`Translation of ${translatedSentences.length} sentences (wordCount ${wordCount}) took ${secs} secs (${Math.round(wordCount / secs)} words per second)`);
+    log(`Translation of (${wordCount}) words took ${secs} secs (${Math.round(wordCount / secs)} words per second)`);
 
-    document.querySelector('#to').value = translatedSentences.join("\n");
+    document.querySelector('#to').value = translatedParagraphs.join("\n");
   }
 
   document.querySelector("#translate").addEventListener("click", () => {

From 458176c0501880e3f308cf90bfc8b8fb7e9504c6 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 22 Feb 2021 18:51:48 +0100
Subject: [PATCH 145/442] Enable building pcre2 from sources for ssplit
 submodule

 - USE_INTERNAL_PCRE2 is set to ON
 - Sentence splitting is working (tested it via wasm test page)
---
 CMakeLists.txt | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index cb027b780..aed7c81ac 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -28,6 +28,8 @@ if(COMPILE_WASM)
   # Set WORMHOLE to ON for marian whenever compiling for wasm platform
   SET(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
 endif()
+# Set ssplit (3rd party submodule) cmake options to compile for this project
+SET(USE_INTERNAL_PCRE2 ON CACHE BOOL "Use internal PCRE2 instead of system PCRE2")
 
 # Documentation: https://cliutils.gitlab.io/modern-cmake/chapters/projects/submodule.html
 # Ensures the submodules are set correctly during a build.

From 415d16bd1de06e67f46d02c54edabbf4e980bd9b Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 23 Feb 2021 15:53:05 +0100
Subject: [PATCH 146/442] Single cmake option to enable/disable wasm compatible
 marian compilation

 - USE_WASM_COMPATIBLE_MARIAN=off will start using vanilla Marian
   i.e. with full threading support, with exceptions, with MKL

 - Changed the relevant documentation
---
 CMakeLists.txt            | 25 ++++++++++++++++---------
 doc/marian-integration.md |  5 +----
 2 files changed, 17 insertions(+), 13 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index aed7c81ac..622a41be3 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -9,24 +9,31 @@ project(bergamot_translator CXX C)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
+include(CMakeDependentOption)
+
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
-option(COMPILE_THREAD_VARIANT "Compile with thread support" OFF)
+option(USE_WASM_COMPATIBLE_MARIAN "Use wasm compatible marian backend" ON)
+CMAKE_DEPENDENT_OPTION(COMPILE_THREAD_VARIANT "Compile the project with thread support" OFF
+                       "USE_WASM_COMPATIBLE_MARIAN" ON)
 SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be packaged (pre-loaded) in wasm builds")
 
 # Set marian (3rd party submodule) cmake options to compile for this project
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
 SET(USE_SENTENCEPIECE ON CACHE BOOL "Download and compile SentencePiece")
 SET(USE_STATIC_LIBS ON CACHE BOOL "Link statically against non-system libs")
-SET(USE_MKL OFF CACHE BOOL "Compile with MKL support")
-SET(COMPILE_DECODER_ONLY ON CACHE BOOL "Compile marian-decoder only")
-SET(COMPILE_WITH_PTHREADS OFF CACHE BOOL "Compile with pthreads support")
-SET(USE_WASM_COMPATIBLE_BLAS ON CACHE BOOL "Compile with a WASM compatible blas for decoder only builds")
 SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
-SET(COMPILE_WITHOUT_EXCEPTIONS ON CACHE BOOL "Compile without exceptions")
-if(COMPILE_WASM)
-  # Set WORMHOLE to ON for marian whenever compiling for wasm platform
-  SET(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
+if (USE_WASM_COMPATIBLE_MARIAN)
+  # If using wasm compatible marian then set following flags
+  SET(USE_MKL OFF CACHE BOOL "Compile with MKL support")
+  SET(COMPILE_DECODER_ONLY ON CACHE BOOL "Compile marian-decoder only")
+  SET(COMPILE_WITH_PTHREADS OFF CACHE BOOL "Compile with pthreads support")
+  SET(USE_WASM_COMPATIBLE_BLAS ON CACHE BOOL "Compile with a WASM compatible blas for decoder only builds")
+  SET(COMPILE_WITHOUT_EXCEPTIONS ON CACHE BOOL "Compile without exceptions")
+  if(COMPILE_WASM)
+    # Set WORMHOLE to ON for marian whenever compiling for wasm platform
+    SET(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
+  endif()
 endif()
 # Set ssplit (3rd party submodule) cmake options to compile for this project
 SET(USE_INTERNAL_PCRE2 ON CACHE BOOL "Use internal PCRE2 instead of system PCRE2")
diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index b7089c06a..762102483 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -10,10 +10,7 @@ $ git clone https://github.com/browsermt/bergamot-translator
 $ cd bergamot-translator
 $ mkdir build
 $ cd build
-$ cmake .. -DCOMPILE_CUDA=off -DCMAKE_BUILD_TYPE=Release \
-     -DCOMPILE_DECODER_ONLY=on -DUSE_MKL=on -DCOMPILE_THREAD_VARIANT=on \
-     -DUSE_WASM_COMPATIBLE_BLAS=off -DCOMPILE_MAIN_EIGEN=off
-
+$ cmake .. -DUSE_WASM_COMPATIBLE_MARIAN=off -DCMAKE_BUILD_TYPE=Release
 $ make -j
 ```
 

From 4369a56f9058704cd6a13126d10cb6b59fdaa9ea Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 23 Feb 2021 18:13:07 +0100
Subject: [PATCH 147/442] Enable building marian executables for vanilla marian
 builds

 - COMPILE_LIBRARY_ONLY is set to ON only for wasm compatible marian
   builds
---
 CMakeLists.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 622a41be3..674bd5049 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -22,9 +22,9 @@ SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be package
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
 SET(USE_SENTENCEPIECE ON CACHE BOOL "Download and compile SentencePiece")
 SET(USE_STATIC_LIBS ON CACHE BOOL "Link statically against non-system libs")
-SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
 if (USE_WASM_COMPATIBLE_MARIAN)
   # If using wasm compatible marian then set following flags
+  SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
   SET(USE_MKL OFF CACHE BOOL "Compile with MKL support")
   SET(COMPILE_DECODER_ONLY ON CACHE BOOL "Compile marian-decoder only")
   SET(COMPILE_WITH_PTHREADS OFF CACHE BOOL "Compile with pthreads support")

From eb5284fb208487b68e320f08e60a7366fc7a20f9 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 23 Feb 2021 18:41:47 +0100
Subject: [PATCH 148/442] Removed erroneous '?' in example text in wasm test
 page

---
 wasm/test_page/bergamot.html | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index d529e42ce..b4d2027b0 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -38,11 +38,12 @@
 <div id="divtranslation">
     <label for="from">From</label>
     <textarea id="from" name="from">
-Una estrategia republicana para obstaculizar la reelecci�n de Obama. Los dirigentes republicanos justificaron su pol�tica por la necesidad de luchar contra el fraude electoral.
-Ahora bien, el Centro Brennan considera esto �ltimo un mito y afirma que el fraude electoral es menos frecuente en los Estados Unidos que el n�mero de personas que mueren a causa de la ca�da de un rayo. De hecho, los abogados republicanos no han encontrado m�s que 300 casos de fraude electoral en los Estados Unidos en diez a�os. Una cosa es cierta: esas nuevas disposiciones afectar�n negativamente a la tasa de participaci�n.
-En ese sentido, estas medidas minar�n en parte el sistema democr�tico americano.
-Al contrario de lo que ocurre en Canad�, los estados americanos son responsables de la organizaci�n de las elecciones federales en los Estados Unidos. Y en esa misma l�nea una mayor�a de los gobiernos americanos promulgaron, a partir de 2009, nuevas leyes que dificultaban el proceso de inscripci�n o de votaci�n.
-Este fen�meno se ha extendido tras las elecciones de noviembre de 2010, que vieron el aumento de 675 nuevos representantes republicanos en 26 estados. En consecuencia, durante el a�o 2011 se introdujeron 180 proyectos de ley que restring�an el ejercicio del derecho de voto en 41 estados.
+Una estrategia republicana para obstaculizar la reelección de Obama. Los dirigentes republicanos justificaron su política por la necesidad de luchar contra el fraude electoral.
+Ahora bien, el Centro Brennan considera esto último un mito y afirma que el fraude electoral es menos frecuente en los Estados Unidos que el número de personas que mueren a causa de la caída de un rayo.
+De hecho, los abogados republicanos no han encontrado más que 300 casos de fraude electoral en los Estados Unidos en diez años. Una cosa es cierta: esas nuevas disposiciones afectarán negativamente a la tasa de participación.
+En ese sentido, estas medidas minarán en parte el sistema democrático americano. Al contrario de lo que ocurre en Canadá, los estados americanos son responsables de la organización de las elecciones federales en los Estados Unidos.
+Y en esa misma línea una mayoría de los gobiernos americanos promulgaron, a partir de 2009, nuevas leyes que dificultaban el proceso de inscripción o de votación. Este fenómeno se ha extendido tras las elecciones de noviembre de 2010, que vieron el aumento de 675 nuevos representantes republicanos en 26 estados.
+En consecuencia, durante el año 2011 se introdujeron 180 proyectos de ley que restringían el ejercicio del derecho de voto en 41 estados.
     </textarea>
     <br><br>
     <label for="to">To</label>

From b845ed36930e166aaf600433b4602ae10a9344b2 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 24 Feb 2021 19:53:16 +0100
Subject: [PATCH 149/442] Update marian submodule

 - Fixes the compilation while building with full blown marian
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 467c43a29..05f2517f5 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 467c43a292a68b7913af2a00d353de97c1740f92
+Subproject commit 05f2517f58de493d2f42236c2d23db95a9edbd8f

From c2b1c6eab484de4acf8d19fb334998927567978e Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 24 Feb 2021 16:49:21 +0100
Subject: [PATCH 150/442] Use system installed PCRE2 for builds using full
 blown marian

 - USE_INTERNAL_PCRE2 is ON for custom marian builds while OFF
   for full marian builds
---
 CMakeLists.txt | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 674bd5049..fa854543d 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -36,7 +36,8 @@ if (USE_WASM_COMPATIBLE_MARIAN)
   endif()
 endif()
 # Set ssplit (3rd party submodule) cmake options to compile for this project
-SET(USE_INTERNAL_PCRE2 ON CACHE BOOL "Use internal PCRE2 instead of system PCRE2")
+CMAKE_DEPENDENT_OPTION(USE_INTERNAL_PCRE2 "Use internal PCRE2 instead of system PCRE2" ON
+                       "USE_WASM_COMPATIBLE_MARIAN" OFF)
 
 # Documentation: https://cliutils.gitlab.io/modern-cmake/chapters/projects/submodule.html
 # Ensures the submodules are set correctly during a build.

From 2538fb60076ab507c09556e3ff46d9b3c221b0d2 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 24 Feb 2021 13:51:05 +0100
Subject: [PATCH 151/442] Added workflows for compilation with custom marian

 - Custom marian means only those marian features that
   are required for wasm

 - Added workflow for native builds
 - Added workflow for wasm builds
---
 .../workflows/macos-custom-marian-native.yml  | 32 +++++++++++++
 .../workflows/macos-custom-marian-wasm.yml    | 47 +++++++++++++++++++
 2 files changed, 79 insertions(+)
 create mode 100644 .github/workflows/macos-custom-marian-native.yml
 create mode 100644 .github/workflows/macos-custom-marian-wasm.yml

diff --git a/.github/workflows/macos-custom-marian-native.yml b/.github/workflows/macos-custom-marian-native.yml
new file mode 100644
index 000000000..1d0db7d44
--- /dev/null
+++ b/.github/workflows/macos-custom-marian-native.yml
@@ -0,0 +1,32 @@
+name: MacOS Native (Custom)
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build-macos:
+    name: Native (With Custom Marian)
+    runs-on: macos-10.15
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: recursive
+
+      - name: Configure CMake
+        run: |
+          mkdir -p build-native
+          cd build-native
+          cmake ..
+
+      - name: Compile
+        working-directory: build-native
+        run: make -j2
+
+      - name: Print versions
+        working-directory: build-native
+        run: ./app/bergamot-translator-app --version
diff --git a/.github/workflows/macos-custom-marian-wasm.yml b/.github/workflows/macos-custom-marian-wasm.yml
new file mode 100644
index 000000000..31725b517
--- /dev/null
+++ b/.github/workflows/macos-custom-marian-wasm.yml
@@ -0,0 +1,47 @@
+name: MacOS WASM (Custom)
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build-wasm:
+    name: WASM (With Custom Marian)
+    runs-on: macos-10.15
+
+    steps:
+      - name: Setup Emscripten toolchain
+        uses: mymindstorm/setup-emsdk@v8
+
+      - name: Verify Emscripten setup
+        run: emcc -v
+
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: recursive
+
+      - name: Configure builds
+        run: |
+          mkdir -p build-wasm
+          cd build-wasm
+          emcmake cmake -DCOMPILE_WASM=on ..
+
+      - name: Compile
+        working-directory: build-wasm
+        run: emmake make -j2
+
+      - name: Check artifacts
+        working-directory: build-wasm
+        run: |
+          export WASM_ARTIFACTS_DIR=wasm
+          ls -all ${WASM_ARTIFACTS_DIR}
+          if ls ${WASM_ARTIFACTS_DIR}/*.wasm &>/dev/null && ls ${WASM_ARTIFACTS_DIR}/*.js &>/dev/null
+          then
+            echo "Artifacts Successfully Generated"
+          else
+            echo "Failure: Artifacts Not Present"
+            exit 1
+          fi

From 18b4c7a01634c7e1887edd0bb240b9cdff0de9cd Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 25 Feb 2021 15:22:52 +0100
Subject: [PATCH 152/442] Improved README

 - Removed 'wasm-integration' branch from wasm build instructions
 - Improved native build instructions
---
 README.md | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/README.md b/README.md
index d9e071971..2b7f0a6c7 100644
--- a/README.md
+++ b/README.md
@@ -5,15 +5,20 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
 ## Build Instructions
 
 ### Build Natively
+1. Clone the repository using these instructions:
+    ```bash
+    git clone https://github.com/browsermt/bergamot-translator
+    cd bergamot-translator
+    ```
+2. Compile
 
-```bash
-git clone  --recursive https://github.com/browsermt/bergamot-translator
-cd bergamot-translator
-mkdir build
-cd build
-cmake ../
-make -j
-```
+    Create a folder where you want to build all the artifacts (`build-native` in this case) and compile in that folder
+    ```bash
+    mkdir build-native
+    cd build-native
+    cmake ../
+    make -j
+    ```
 
 ### Build WASM
 #### Compiling for the first time
@@ -25,12 +30,10 @@ make -j
     * Activate the latest sdk tools: `./emsdk activate latest`
     * Activate path variables: `source ./emsdk_env.sh`
 
-2. Clone the repository and checkout the appropriate branch using these instructions:
+2. Clone the repository using these instructions:
     ```bash
     git clone https://github.com/browsermt/bergamot-translator
     cd bergamot-translator
-    git checkout -b wasm-integration origin/wasm-integration
-    git submodule update --init --recursive
     ```
 
 3. Download files (only required if you want to package files in wasm binary)

From 51da121057b4434912dd5a9cb82fad6fed13d5ba Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 25 Feb 2021 15:13:44 +0000
Subject: [PATCH 153/442] CI Fixes: vanilla bergamot-translator builds

* Updating vanilla workflows with -DUSE_WASM_COMPATIBLE_MARIAN=off
* Boost: Replacing with OS Boost for Ubuntu Builds
---
 .github/workflows/macos.yml  | 3 ++-
 .github/workflows/ubuntu.yml | 6 ++----
 2 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml
index 8ccdecaf5..e61857ca4 100644
--- a/.github/workflows/macos.yml
+++ b/.github/workflows/macos.yml
@@ -38,7 +38,8 @@ jobs:
           -DCOMPILE_TESTS=on \
           -DUSE_FBGEMM=on \
           -DUSE_SENTENCEPIECE=on \
-          -DUSE_STATIC_LIBS=off
+          -DUSE_STATIC_LIBS=off \
+          -DUSE_WASM_COMPATIBLE_MARIAN=off
 
     - name: Compile
       working-directory: build
diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml
index 240efd2c3..c7a53d89e 100644
--- a/.github/workflows/ubuntu.yml
+++ b/.github/workflows/ubuntu.yml
@@ -62,7 +62,7 @@ jobs:
     # No need to install libprotobuf{17,10,9v5} on Ubuntu {20,18,16}.04 because
     # it is installed together with libprotobuf-dev
     - name: Install dependencies
-      run: sudo apt-get install -y libgoogle-perftools-dev libprotobuf-dev protobuf-compiler
+      run: sudo apt-get install -y libgoogle-perftools-dev libprotobuf-dev protobuf-compiler libboost-all-dev
 
     # https://software.intel.com/content/www/us/en/develop/articles/installing-intel-free-libs-and-python-apt-repo.html
     - name: Install MKL
@@ -87,9 +87,6 @@ jobs:
         CC=/usr/bin/gcc-${{ matrix.gcc }} CXX=/usr/bin/g++-${{ matrix.gcc }} CUDAHOSTCXX=/usr/bin/g++-${{ matrix.gcc }} \
         cmake .. \
           -DBoost_ARCHITECTURE=-x64 \
-          -DBOOST_INCLUDEDIR=$BOOST_ROOT_1_72_0/include \
-          -DBOOST_LIBRARYDIR=$BOOST_ROOT_1_72_0/lib \
-          -DBOOST_ROOT=$BOOST_ROOT_1_72_0 \
           -DCMAKE_BUILD_TYPE=Release \
           -DCOMPILE_CPU=${{ matrix.cpu }} \
           -DCOMPILE_CUDA=${{ matrix.gpu }} \
@@ -100,6 +97,7 @@ jobs:
           -DUSE_FBGEMM=${{ matrix.cpu }} \
           -DUSE_SENTENCEPIECE=on \
           -DUSE_STATIC_LIBS=on \
+          -DUSE_WASM_COMPATIBLE_MARIAN=off
 
     - name: Compile
       working-directory: build

From cd01d7552a7cec49b175efbf959dc397d36c9c70 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 25 Feb 2021 23:11:09 +0000
Subject: [PATCH 154/442] ServiceBase -> [NonThreadedService, Service]

Through inheritance, a non-threaded and multithreaded Service are
created, both derived of the same ServiceBase class which holds the
common elements.

In preparation to solve SIGSEGV in #41. First inspections gave aborts in
thread part, and repeated SIGSEGV's in lock-policy's of shared_pointers
even in non-threaded paths.

Solving this first, to avoid ifdef or tricky paths. The non-threaded
implementation is not included in WASM builds at all, by separating out
the single-threaded logic. DRY is achieved through inheritance and
operator overloading.
---
 app/CMakeLists.txt                  |  10 +--
 src/translator/CMakeLists.txt       |   7 +-
 src/translator/TranslationModel.cpp |   2 +-
 src/translator/TranslationModel.h   |   9 ++-
 src/translator/batcher.cpp          |   9 ---
 src/translator/batcher.h            |   9 +--
 src/translator/service.cpp          | 104 ++++++----------------------
 src/translator/service.h            |  54 +++------------
 src/translator/service_base.cpp     |  40 +++++++++++
 src/translator/service_base.h       |  50 +++++++++++++
 10 files changed, 143 insertions(+), 151 deletions(-)
 create mode 100644 src/translator/service_base.cpp
 create mode 100644 src/translator/service_base.h

diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index 24bd0b43e..6a5996fd7 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -1,8 +1,10 @@
 add_executable(bergamot-translator-app main.cpp)
 target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
 
-add_executable(service-cli main-mts.cpp)
-target_link_libraries(service-cli PRIVATE bergamot-translator)
+if (NOT COMPILE_WASM)
+    add_executable(service-cli main-mts.cpp)
+    target_link_libraries(service-cli PRIVATE bergamot-translator)
 
-add_executable(marian-decoder-new marian-decoder-new.cpp)
-target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)
+    add_executable(marian-decoder-new marian-decoder-new.cpp)
+    target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)
+endif()
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index e8dce2bb7..1e9cf02df 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,3 +1,7 @@
+if (NOT COMPILE_WASM)
+    set(SERVICE "service.cpp")
+endif()
+
 add_library(bergamot-translator STATIC
     AbstractTranslationModel.cpp
     TranslationModel.cpp
@@ -8,7 +12,8 @@ add_library(bergamot-translator STATIC
     batch_translator.cpp 
     multifactor_priority.cpp 
     request.cpp 
-    service.cpp
+    service_base.cpp
+    ${SERVICE}
     batcher.cpp
     response.cpp
     batch.cpp
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 45b93d18d..b3e0db6d8 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -15,7 +15,7 @@
 // All local project includes
 #include "TranslationModel.h"
 #include "translator/parser.h"
-#include "translator/service.h"
+#include "translator/service_base.h"
 
 std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
   marian::Options options;
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index ea80ca907..e4765965a 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -16,7 +16,7 @@
 
 // All local project includes
 #include "AbstractTranslationModel.h"
-#include "translator/service.h"
+#include "translator/service_base.h"
 
 /* A Translation model that translates a plain (without any markups and emojis)
  * UTF-8 encoded text. This implementation supports translation from 1 source
@@ -55,9 +55,8 @@ class TranslationModel : public AbstractTranslationModel {
    * entry of texts list will be moved to its corresponding TranslationResult
    * object).
    */
-  std::vector<TranslationResult>
-  translate(std::vector<std::string> &&texts,
-            TranslationRequest request) override;
+  std::vector<TranslationResult> translate(std::vector<std::string> &&texts,
+                                           TranslationRequest request) override;
 
   /* Check if the model can provide alignment information b/w original and
    * translated text. */
@@ -66,7 +65,7 @@ class TranslationModel : public AbstractTranslationModel {
 private:
   // Model configuration options
   std::shared_ptr<marian::Options> configOptions_; // ORDER DEPENDECNY
-  marian::bergamot::Service service_;              // ORDER DEPENDENCY
+  marian::bergamot::NonThreadedService service_;   // ORDER DEPENDENCY
 };
 
 #endif /* SRC_TRANSLATOR_TRANSLATIONMODEL_H_ */
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 56fe6860a..2b9c55168 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -57,14 +57,5 @@ void Batcher::addWholeRequest(Ptr<Request> request) {
   }
 }
 
-#ifdef WITH_PTHREADS
-void Batcher::produceTo(PCQueue<Batch> &pcqueue) {
-  Batch batch;
-  while (cleaveBatch(batch)) {
-    pcqueue.ProduceSwap(batch);
-  }
-}
-#endif
-
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index a2bebab25..81ed2d270 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -25,16 +25,13 @@ class Batcher {
   // which maintains priority among sentences from multiple concurrent requests.
   void addSentenceWithPriority(RequestSentence &sentence);
   void addWholeRequest(Ptr<Request> request);
-#ifdef WITH_PTHREADS
-  void produceTo(PCQueue<Batch> &pcqueue);
-#endif
 
+  bool operator>>(Batch &batch); // alias for cleaveBatch
+
+private:
   // Loads sentences with sentences compiled from (tentatively) multiple
   // requests optimizing for both padding and priority.
   bool cleaveBatch(Batch &batch);
-  bool operator>>(Batch &batch); // alias
-
-private:
   size_t miniBatchWords;
   std::vector<std::set<RequestSentence>> bucket_;
   size_t batchNumber_{0};
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 986d4a477..12b548f37 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -9,105 +9,47 @@ namespace marian {
 namespace bergamot {
 
 Service::Service(Ptr<Options> options)
-    : requestId_(0), numWorkers_(options->get<int>("cpu-threads")),
-      vocabs_(std::move(loadVocabularies(options))),
-      text_processor_(vocabs_, options), batcher_(options)
-#ifdef WITH_PTHREADS
-      ,
-      pcqueue_(2 * options->get<int>("cpu-threads"))
-#endif // WITH_PTHREADS
-{
+    : ServiceBase(options), numWorkers_(options->get<int>("cpu-threads")),
+      pcqueue_(numWorkers_) {
+  if (numWorkers_ <= 0) {
+    ABORT("Fatal: numWorkers should be greater than 1");
+  }
+
+  translators_.reserve(numWorkers_);
+  workers_.reserve(numWorkers_);
 
-  if (numWorkers_ == 0) {
-    // In case workers are 0, a single-translator is created and initialized
-    // in the main thread.
-    marian::DeviceId deviceId(/*cpuId=*/0, DeviceType::cpu);
+  for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
+    marian::DeviceId deviceId(cpuId, DeviceType::cpu);
     translators_.emplace_back(deviceId, vocabs_, options);
-    translators_.back().initialize();
-  } else {
-#ifdef WITH_PTHREADS
-    // If workers specified are greater than 0, translators_ are populated with
-    // unitialized instances. These are then initialized inside
-    // individual threads and set to consume from producer-consumer queue.
-    workers_.reserve(numWorkers_);
-    translators_.reserve(numWorkers_);
-    for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
-      marian::DeviceId deviceId(cpuId, DeviceType::cpu);
-      translators_.emplace_back(deviceId, vocabs_, options);
 
-      auto &translator = translators_.back();
-      workers_.emplace_back([&translator, this] {
-        translator.initialize();
-        translator.consumeFrom(pcqueue_);
-      });
-    }
-#else // WITH_PTHREADS
-    ABORT(
-        "Fatal: Service started requesting multiple threadswhile compiled with "
-        "COMPILE_THREAD_VARIANT=off. Please check your cmake build "
-        "configuration");
-#endif
+    auto &translator = translators_.back();
+    workers_.emplace_back([&translator, this] {
+      translator.initialize();
+      translator.consumeFrom(pcqueue_);
+    });
   }
 }
 
-std::future<Response> Service::translateWithCopy(std::string input) {
-  return translate(std::move(input));
-}
-
-std::future<Response> Service::translate(std::string &&input) {
-  // Takes in a blob of text. Segments and SentenceRanges are
-  // extracted from the input (blob of text) and used to construct a Request
-  // along with a promise. promise value is set by the worker completing a
-  // request.
-  //
-  // Batcher, which currently runs on the main thread constructs batches out of
-  // a single request (at the moment) and adds them into a Producer-Consumer
-  // queue holding a bunch of requestSentences used to construct batches.
-  // TODO(jerin): Make asynchronous and compile from multiple requests.
-  //
-  // returns future corresponding to the promise.
-
-  Segments segments;
-  SentenceRanges sourceRanges;
-  text_processor_.process(input, segments, sourceRanges);
-
-  std::promise<Response> responsePromise;
-  auto future = responsePromise.get_future();
-
-  Ptr<Request> request = New<Request>(
-      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(input),
-      std::move(segments), std::move(sourceRanges), std::move(responsePromise));
-
-  batcher_.addWholeRequest(request);
-
-  if (numWorkers_ > 0) {
-#ifdef WITH_PTHREADS
-    batcher_.produceTo(pcqueue_);
-#endif
-  } else {
-    // Queue single-threaded
-    Batch batch;
-    while (batcher_ >> batch) {
-      translators_[0].translate(batch);
-    }
+void Service::enqueue() {
+  Batch batch;
+  while (batcher_ >> batch) {
+    pcqueue_.ProduceSwap(batch);
   }
-
-  return future;
 }
 
 void Service::stop() {
-#ifdef WITH_PTHREADS
   for (auto &worker : workers_) {
     Batch poison = Batch::poison();
     pcqueue_.ProduceSwap(poison);
   }
 
   for (auto &worker : workers_) {
-    worker.join();
+    if (worker.joinable()) {
+      worker.join();
+    }
   }
 
-  workers_.clear(); // Takes care of idempotency.
-#endif
+  workers_.clear();
 }
 
 Service::~Service() { stop(); }
diff --git a/src/translator/service.h b/src/translator/service.h
index cd22ea9e5..de9c2e8c4 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -1,24 +1,22 @@
+
 #ifndef SRC_BERGAMOT_SERVICE_H_
 #define SRC_BERGAMOT_SERVICE_H_
 
 #include "batch_translator.h"
 #include "batcher.h"
+#include "data/types.h"
+#include "pcqueue.h"
 #include "response.h"
+#include "service_base.h"
 #include "text_processor.h"
 
 #include <queue>
 #include <vector>
 
-#include "data/types.h"
-
-#ifdef WITH_PTHREADS
-#include "pcqueue.h"
-#endif
-
 namespace marian {
 namespace bergamot {
 
-class Service {
+class Service : public ServiceBase {
 
   // Service exposes methods to translate an incoming blob of text to the
   // Consumer of bergamot API.
@@ -35,47 +33,15 @@ class Service {
 
 public:
   explicit Service(Ptr<Options> options);
-
-  // Constructs new string copying, calls translate internally.
-  std::future<Response> translateWithCopy(std::string input);
-  std::future<Response> translate(std::string &&input);
-
-  void stop();
-
-  Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); }
-  Ptr<Vocab const> targetVocab() const { return vocabs_.back(); }
-
+  void enqueue() override;
+  void stop() override;
   ~Service();
 
 private:
-  size_t requestId_;
-  size_t numWorkers_;
-
-  // vocabs are used to construct a Request, which later uses it to construct
-  // Response (decode from words to string).
-  std::vector<Ptr<Vocab const>> vocabs_; // ORDER DEPENDENCY
-
-  // Consists of:
-  //
-  // 1. text-processing class (TextProcessor), which handles breaking a blob of
-  //    text into sentences and providing them representated by finite
-  //    vocabulary for further processing by hte neural machine translation.
-  // 2. a Batcher class which handles efficient batching by minimizing
-  //    padding wasting compute.
-  // 3. Multiple workers - which are instances of BatchTranslators are
-  //    spawned in separate threads.
-  //
-  // Batcher acts as a producer for a producer-consumer queue (pcqueue_), with
-  // idle BatchTranslators being consumers requesting batches as they're ready.
-
-  TextProcessor text_processor_; // ORDER DEPENDENCY
-  Batcher batcher_;
-  std::vector<BatchTranslator> translators_;
-
-#ifdef WITH_PTHREADS
-  PCQueue<Batch> pcqueue_;
+  size_t numWorkers_;      // ORDER DEPENDENCY
+  PCQueue<Batch> pcqueue_; // ORDER DEPENDENCY
   std::vector<std::thread> workers_;
-#endif
+  std::vector<BatchTranslator> translators_;
 };
 
 std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);
diff --git a/src/translator/service_base.cpp b/src/translator/service_base.cpp
new file mode 100644
index 000000000..596759ac5
--- /dev/null
+++ b/src/translator/service_base.cpp
@@ -0,0 +1,40 @@
+#include "service_base.h"
+
+namespace marian {
+namespace bergamot {
+
+ServiceBase::ServiceBase(Ptr<Options> options)
+    : requestId_(0), vocabs_(std::move(loadVocabularies(options))),
+      text_processor_(vocabs_, options), batcher_(options) {}
+
+std::future<Response> ServiceBase::translate(std::string &&input) {
+  Segments segments;
+  SentenceRanges sourceRanges;
+  text_processor_.process(input, segments, sourceRanges);
+
+  std::promise<Response> responsePromise;
+  auto future = responsePromise.get_future();
+
+  Ptr<Request> request = New<Request>(
+      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(input),
+      std::move(segments), std::move(sourceRanges), std::move(responsePromise));
+
+  batcher_.addWholeRequest(request);
+  enqueue();
+  return future;
+}
+
+NonThreadedService::NonThreadedService(Ptr<Options> options)
+    : ServiceBase(options),
+      translator_(DeviceId(0, DeviceType::cpu), vocabs_, options) {}
+
+void NonThreadedService::enqueue() {
+  // Queue single-threaded
+  Batch batch;
+  while (batcher_ >> batch) {
+    translator_.translate(batch);
+  }
+}
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/service_base.h b/src/translator/service_base.h
new file mode 100644
index 000000000..c29fd98ac
--- /dev/null
+++ b/src/translator/service_base.h
@@ -0,0 +1,50 @@
+#ifndef SRC_BERGAMOT_SUBSTANDARD_SERVICE_H_
+#define SRC_BERGAMOT_SUBSTANDARD_SERVICE_H_
+#include "batch_translator.h"
+#include "batcher.h"
+#include "data/types.h"
+#include "response.h"
+#include "text_processor.h"
+
+#include <queue>
+#include <vector>
+
+namespace marian {
+namespace bergamot {
+
+class ServiceBase {
+public:
+  explicit ServiceBase(Ptr<Options> options);
+  std::future<Response> translateWithCopy(std::string input) {
+    return translate(std::move(input));
+  };
+
+  std::future<Response> translate(std::string &&input);
+  Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); }
+  Ptr<Vocab const> targetVocab() const { return vocabs_.back(); }
+  virtual void enqueue() = 0;
+  virtual void stop() = 0;
+
+protected:
+  size_t requestId_;
+  std::vector<Ptr<Vocab const>> vocabs_; // ORDER DEPENDENCY
+  TextProcessor text_processor_;         // ORDER DEPENDENCY
+  Batcher batcher_;
+};
+
+class NonThreadedService : public ServiceBase {
+public:
+  explicit NonThreadedService(Ptr<Options> options);
+  void enqueue();
+  void stop() override{};
+
+private:
+  BatchTranslator translator_;
+};
+
+std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_SUBSTANDARD_SERVICE_H_

From 570865e7992460a6543d09ddad00f8cc5b64ade8 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 25 Feb 2021 23:50:57 +0000
Subject: [PATCH 155/442] Getting rid of the ifdef from BatchTranslator as well

---
 src/translator/batch_translator.cpp | 17 -----------------
 src/translator/batch_translator.h   |  4 ----
 src/translator/service.cpp          | 15 +++++++++++++--
 src/translator/service_base.cpp     |  4 +++-
 4 files changed, 16 insertions(+), 24 deletions(-)

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index f938c40a2..6036c3799 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -96,22 +96,5 @@ void BatchTranslator::translate(Batch &batch) {
   batch.completeBatch(histories);
 }
 
-#ifdef WITH_PTHREADS
-
-void BatchTranslator::consumeFrom(PCQueue<Batch> &pcqueue) {
-  Batch batch;
-  Histories histories;
-  while (true) {
-    pcqueue.ConsumeSwap(batch);
-    if (batch.isPoison()) {
-      return;
-    } else {
-      translate(batch);
-    }
-  }
-}
-
-#endif
-
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index b927c012b..50a7b7999 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -34,10 +34,6 @@ class BatchTranslator {
   void translate(Batch &batch);
   void initialize();
 
-#ifdef WITH_PTHREADS
-  void consumeFrom(PCQueue<Batch> &pcqueue);
-#endif
-
 private:
   Ptr<Options> options_;
   DeviceId device_;
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 12b548f37..04880f792 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -21,11 +21,22 @@ Service::Service(Ptr<Options> options)
   for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
     marian::DeviceId deviceId(cpuId, DeviceType::cpu);
     translators_.emplace_back(deviceId, vocabs_, options);
-
     auto &translator = translators_.back();
+
     workers_.emplace_back([&translator, this] {
       translator.initialize();
-      translator.consumeFrom(pcqueue_);
+
+      // Run thread mainloop
+      Batch batch;
+      Histories histories;
+      while (true) {
+        pcqueue_.ConsumeSwap(batch);
+        if (batch.isPoison()) {
+          return;
+        } else {
+          translator.translate(batch);
+        }
+      }
     });
   }
 }
diff --git a/src/translator/service_base.cpp b/src/translator/service_base.cpp
index 596759ac5..1a5f51144 100644
--- a/src/translator/service_base.cpp
+++ b/src/translator/service_base.cpp
@@ -26,7 +26,9 @@ std::future<Response> ServiceBase::translate(std::string &&input) {
 
 NonThreadedService::NonThreadedService(Ptr<Options> options)
     : ServiceBase(options),
-      translator_(DeviceId(0, DeviceType::cpu), vocabs_, options) {}
+      translator_(DeviceId(0, DeviceType::cpu), vocabs_, options) {
+  translator_.initialize();
+}
 
 void NonThreadedService::enqueue() {
   // Queue single-threaded

From 2c57f4b498208244a4b3c161b597859d5145dc93 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 26 Feb 2021 00:11:58 +0000
Subject: [PATCH 156/442] Adding ci-sandbox branch for GitHub CI test-use

---
 .github/workflows/macos-custom-marian-native.yml | 4 ++--
 .github/workflows/macos-custom-marian-wasm.yml   | 4 ++--
 .github/workflows/macos.yml                      | 4 ++--
 .github/workflows/ubuntu.yml                     | 4 ++--
 .github/workflows/windows.yml                    | 4 ++--
 5 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/.github/workflows/macos-custom-marian-native.yml b/.github/workflows/macos-custom-marian-native.yml
index 1d0db7d44..9a919d3cb 100644
--- a/.github/workflows/macos-custom-marian-native.yml
+++ b/.github/workflows/macos-custom-marian-native.yml
@@ -2,9 +2,9 @@ name: MacOS Native (Custom)
 
 on:
   push:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
 
 jobs:
   build-macos:
diff --git a/.github/workflows/macos-custom-marian-wasm.yml b/.github/workflows/macos-custom-marian-wasm.yml
index 31725b517..3e49369b4 100644
--- a/.github/workflows/macos-custom-marian-wasm.yml
+++ b/.github/workflows/macos-custom-marian-wasm.yml
@@ -2,9 +2,9 @@ name: MacOS WASM (Custom)
 
 on:
   push:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
 
 jobs:
   build-wasm:
diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml
index e61857ca4..112732d7d 100644
--- a/.github/workflows/macos.yml
+++ b/.github/workflows/macos.yml
@@ -2,9 +2,9 @@ name: MacOS
 
 on:
   push:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
 
 jobs:
   build-macos:
diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml
index c7a53d89e..7f61d4612 100644
--- a/.github/workflows/ubuntu.yml
+++ b/.github/workflows/ubuntu.yml
@@ -2,9 +2,9 @@ name: Ubuntu
 
 on:
   push:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
 
 jobs:
   build-ubuntu:
diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index ef9ad25d1..fd1f21f2c 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -2,9 +2,9 @@ name: Windows
 
 on:
   push:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
 
 env:
   MKL_URL: "https://romang.blob.core.windows.net/mariandev/ci/mkl-2020.1-windows-static.zip"

From dad3a4088c7dba2a9fdbc3afe2961f71f426cd30 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 26 Feb 2021 00:39:38 +0000
Subject: [PATCH 157/442] Marking enqueue as override()

---
 src/translator/service_base.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/translator/service_base.h b/src/translator/service_base.h
index c29fd98ac..9bb60b131 100644
--- a/src/translator/service_base.h
+++ b/src/translator/service_base.h
@@ -35,7 +35,7 @@ class ServiceBase {
 class NonThreadedService : public ServiceBase {
 public:
   explicit NonThreadedService(Ptr<Options> options);
-  void enqueue();
+  void enqueue() override;
   void stop() override{};
 
 private:

From 66e3b4493eb69104b6f661f2ef13cf669d2a3729 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 26 Feb 2021 00:43:36 +0000
Subject: [PATCH 158/442] LoadVocabularies inlined in service_base.h

To fix WASM Mac builds on CI. loadVocabularies function is now inlines
and available through service_base.h, from where it seems to propogate
to all places of use.
---
 src/translator/service.cpp    | 19 -------------------
 src/translator/service.h      |  3 ---
 src/translator/service_base.h | 25 +++++++++++++++++++++----
 3 files changed, 21 insertions(+), 26 deletions(-)

diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 04880f792..4ab8089c1 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -65,24 +65,5 @@ void Service::stop() {
 
 Service::~Service() { stop(); }
 
-// Internal function nobody used, only within service.
-std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options) {
-  // @TODO: parallelize vocab loading for faster startup
-  auto vfiles = options->get<std::vector<std::string>>("vocabs");
-  // with the current setup, we need at least two vocabs: src and trg
-  ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
-  std::vector<Ptr<Vocab const>> vocabs(vfiles.size());
-  std::unordered_map<std::string, Ptr<Vocab>> vmap;
-  for (size_t i = 0; i < vocabs.size(); ++i) {
-    auto m = vmap.emplace(std::make_pair(vfiles[i], Ptr<Vocab>()));
-    if (m.second) { // new: load the vocab
-      m.first->second = New<Vocab>(options, i);
-      m.first->second->load(vfiles[i]);
-    }
-    vocabs[i] = m.first->second;
-  }
-  return vocabs;
-}
-
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/service.h b/src/translator/service.h
index de9c2e8c4..b2fb61779 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -1,4 +1,3 @@
-
 #ifndef SRC_BERGAMOT_SERVICE_H_
 #define SRC_BERGAMOT_SERVICE_H_
 
@@ -44,8 +43,6 @@ class Service : public ServiceBase {
   std::vector<BatchTranslator> translators_;
 };
 
-std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);
-
 } // namespace bergamot
 } // namespace marian
 
diff --git a/src/translator/service_base.h b/src/translator/service_base.h
index 9bb60b131..3482ec8d1 100644
--- a/src/translator/service_base.h
+++ b/src/translator/service_base.h
@@ -1,5 +1,5 @@
-#ifndef SRC_BERGAMOT_SUBSTANDARD_SERVICE_H_
-#define SRC_BERGAMOT_SUBSTANDARD_SERVICE_H_
+#ifndef SRC_BERGAMOT_SERVICE_BASE_H_
+#define SRC_BERGAMOT_SERVICE_BASE_H_
 #include "batch_translator.h"
 #include "batcher.h"
 #include "data/types.h"
@@ -42,9 +42,26 @@ class NonThreadedService : public ServiceBase {
   BatchTranslator translator_;
 };
 
-std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options);
+// Internal function nobody used, only within service.
+inline std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options) {
+  // @TODO: parallelize vocab loading for faster startup
+  auto vfiles = options->get<std::vector<std::string>>("vocabs");
+  // with the current setup, we need at least two vocabs: src and trg
+  ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
+  std::vector<Ptr<Vocab const>> vocabs(vfiles.size());
+  std::unordered_map<std::string, Ptr<Vocab>> vmap;
+  for (size_t i = 0; i < vocabs.size(); ++i) {
+    auto m = vmap.emplace(std::make_pair(vfiles[i], Ptr<Vocab>()));
+    if (m.second) { // new: load the vocab
+      m.first->second = New<Vocab>(options, i);
+      m.first->second->load(vfiles[i]);
+    }
+    vocabs[i] = m.first->second;
+  }
+  return vocabs;
+}
 
 } // namespace bergamot
 } // namespace marian
 
-#endif // SRC_BERGAMOT_SUBSTANDARD_SERVICE_H_
+#endif // SRC_BERGAMOT_SERVICE_BASE_H_

From 9ead41d87952a0e64c7b4bc7e6cd888766e7c6ce Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 26 Feb 2021 10:33:07 +0000
Subject: [PATCH 159/442] Adds documentation, makes enqueue() private

---
 src/translator/service.h      |  9 ++++++++-
 src/translator/service_base.h | 25 +++++++++++++++++++------
 2 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/src/translator/service.h b/src/translator/service.h
index b2fb61779..9a57996c5 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -32,11 +32,18 @@ class Service : public ServiceBase {
 
 public:
   explicit Service(Ptr<Options> options);
-  void enqueue() override;
+  // Implements enqueue and top through blocking methods.
   void stop() override;
   ~Service();
 
 private:
+  void enqueue() override;
+
+  // In addition to the common members (text_processor, requestId, vocabs_,
+  // batcher) extends with a producer-consumer queue, vector of translator
+  // instances owned by service each listening to the pcqueue in separate
+  // threads.
+
   size_t numWorkers_;      // ORDER DEPENDENCY
   PCQueue<Batch> pcqueue_; // ORDER DEPENDENCY
   std::vector<std::thread> workers_;
diff --git a/src/translator/service_base.h b/src/translator/service_base.h
index 3482ec8d1..25cc003e9 100644
--- a/src/translator/service_base.h
+++ b/src/translator/service_base.h
@@ -11,21 +11,31 @@
 
 namespace marian {
 namespace bergamot {
+// This file describes the base class ServiceBase, and a non-threaded subclass
+// implementing translation functionality called NonThreadedService.
 
 class ServiceBase {
 public:
   explicit ServiceBase(Ptr<Options> options);
-  std::future<Response> translateWithCopy(std::string input) {
-    return translate(std::move(input));
-  };
 
+  // Transfers ownership of input string to Service, returns a future containing
+  // an object which provides access to translations, other features like
+  // sentencemappings and (tentatively) alignments.
   std::future<Response> translate(std::string &&input);
+
+  // Convenience accessor methods to extract these vocabulary outside service.
+  // e.g: For use in decoding histories for marian-decoder replacement.
   Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); }
   Ptr<Vocab const> targetVocab() const { return vocabs_.back(); }
-  virtual void enqueue() = 0;
+
+  // Wraps up any thread related destruction code.
   virtual void stop() = 0;
 
 protected:
+  // Enqueue queues a request for translation, this can be synchronous, blocking
+  // or asynchronous and queued in the background.
+  virtual void enqueue() = 0;
+
   size_t requestId_;
   std::vector<Ptr<Vocab const>> vocabs_; // ORDER DEPENDENCY
   TextProcessor text_processor_;         // ORDER DEPENDENCY
@@ -35,14 +45,17 @@ class ServiceBase {
 class NonThreadedService : public ServiceBase {
 public:
   explicit NonThreadedService(Ptr<Options> options);
-  void enqueue() override;
   void stop() override{};
 
 private:
+  // NonThreaded service overrides unimplemented functions in base-class using
+  // blocking mechanisms.
+  void enqueue() override;
+  // There's a single translator, launched as part of the main process.
   BatchTranslator translator_;
 };
 
-// Internal function nobody used, only within service.
+// Used across Services
 inline std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options) {
   // @TODO: parallelize vocab loading for faster startup
   auto vfiles = options->get<std::vector<std::string>>("vocabs");

From 7c14b737a4202404d1f971960f7341cd0892bd0b Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 26 Feb 2021 11:41:10 +0000
Subject: [PATCH 160/442] Improving abort error message.

---
 src/translator/service.cpp | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 4ab8089c1..2101b8ea1 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -11,8 +11,9 @@ namespace bergamot {
 Service::Service(Ptr<Options> options)
     : ServiceBase(options), numWorkers_(options->get<int>("cpu-threads")),
       pcqueue_(numWorkers_) {
-  if (numWorkers_ <= 0) {
-    ABORT("Fatal: numWorkers should be greater than 1");
+  if (numWorkers_ == 0) {
+    ABORT("Fatal: Attempt to create multithreaded instance with --cpu-threads "
+          "0. ");
   }
 
   translators_.reserve(numWorkers_);

From e1b74bccab2ea32c1c02f92f22c38f170ec55ae6 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 26 Feb 2021 11:42:23 +0000
Subject: [PATCH 161/442] Reverting moot COMPILE_WASM guards in app folder

---
 app/CMakeLists.txt | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index 6a5996fd7..24bd0b43e 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -1,10 +1,8 @@
 add_executable(bergamot-translator-app main.cpp)
 target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
 
-if (NOT COMPILE_WASM)
-    add_executable(service-cli main-mts.cpp)
-    target_link_libraries(service-cli PRIVATE bergamot-translator)
+add_executable(service-cli main-mts.cpp)
+target_link_libraries(service-cli PRIVATE bergamot-translator)
 
-    add_executable(marian-decoder-new marian-decoder-new.cpp)
-    target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)
-endif()
+add_executable(marian-decoder-new marian-decoder-new.cpp)
+target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)

From 4d4acf6b8b1f2445f81d79916aeace90cc2d9838 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Fri, 26 Feb 2021 14:16:51 +0100
Subject: [PATCH 162/442] Cleanup CMakeFiles.txt

 - Renamed USE_WASM_COMPATIBLE_MARIAN to USE_WASM_COMPATIBLE_SOURCES
 - Removed COMPILE_THREAD_VARIANT cmake option and removed
   corresponding compile definition
 - Updated workflows and READMEs accordingly
---
 .github/workflows/macos.yml   |  2 +-
 .github/workflows/ubuntu.yml  |  2 +-
 CMakeLists.txt                |  8 +++-----
 doc/marian-integration.md     |  2 +-
 src/translator/CMakeLists.txt | 10 +++-------
 5 files changed, 9 insertions(+), 15 deletions(-)

diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml
index 112732d7d..6a9035398 100644
--- a/.github/workflows/macos.yml
+++ b/.github/workflows/macos.yml
@@ -39,7 +39,7 @@ jobs:
           -DUSE_FBGEMM=on \
           -DUSE_SENTENCEPIECE=on \
           -DUSE_STATIC_LIBS=off \
-          -DUSE_WASM_COMPATIBLE_MARIAN=off
+          -DUSE_WASM_COMPATIBLE_SOURCES=off
 
     - name: Compile
       working-directory: build
diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml
index 7f61d4612..940ecddb0 100644
--- a/.github/workflows/ubuntu.yml
+++ b/.github/workflows/ubuntu.yml
@@ -97,7 +97,7 @@ jobs:
           -DUSE_FBGEMM=${{ matrix.cpu }} \
           -DUSE_SENTENCEPIECE=on \
           -DUSE_STATIC_LIBS=on \
-          -DUSE_WASM_COMPATIBLE_MARIAN=off
+          -DUSE_WASM_COMPATIBLE_SOURCES=off
 
     - name: Compile
       working-directory: build
diff --git a/CMakeLists.txt b/CMakeLists.txt
index fa854543d..45a857f01 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -13,16 +13,14 @@ include(CMakeDependentOption)
 
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
-option(USE_WASM_COMPATIBLE_MARIAN "Use wasm compatible marian backend" ON)
-CMAKE_DEPENDENT_OPTION(COMPILE_THREAD_VARIANT "Compile the project with thread support" OFF
-                       "USE_WASM_COMPATIBLE_MARIAN" ON)
+option(USE_WASM_COMPATIBLE_SOURCES "Use wasm compatible sources" ON)
 SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be packaged (pre-loaded) in wasm builds")
 
 # Set marian (3rd party submodule) cmake options to compile for this project
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
 SET(USE_SENTENCEPIECE ON CACHE BOOL "Download and compile SentencePiece")
 SET(USE_STATIC_LIBS ON CACHE BOOL "Link statically against non-system libs")
-if (USE_WASM_COMPATIBLE_MARIAN)
+if (USE_WASM_COMPATIBLE_SOURCES)
   # If using wasm compatible marian then set following flags
   SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
   SET(USE_MKL OFF CACHE BOOL "Compile with MKL support")
@@ -37,7 +35,7 @@ if (USE_WASM_COMPATIBLE_MARIAN)
 endif()
 # Set ssplit (3rd party submodule) cmake options to compile for this project
 CMAKE_DEPENDENT_OPTION(USE_INTERNAL_PCRE2 "Use internal PCRE2 instead of system PCRE2" ON
-                       "USE_WASM_COMPATIBLE_MARIAN" OFF)
+                       "USE_WASM_COMPATIBLE_SOURCES" OFF)
 
 # Documentation: https://cliutils.gitlab.io/modern-cmake/chapters/projects/submodule.html
 # Ensures the submodules are set correctly during a build.
diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index 762102483..101dbb219 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -10,7 +10,7 @@ $ git clone https://github.com/browsermt/bergamot-translator
 $ cd bergamot-translator
 $ mkdir build
 $ cd build
-$ cmake .. -DUSE_WASM_COMPATIBLE_MARIAN=off -DCMAKE_BUILD_TYPE=Release
+$ cmake .. -DUSE_WASM_COMPATIBLE_SOURCES=off -DCMAKE_BUILD_TYPE=Release
 $ make -j
 ```
 
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 1e9cf02df..5c7269e07 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,5 +1,5 @@
-if (NOT COMPILE_WASM)
-    set(SERVICE "service.cpp")
+if (NOT USE_WASM_COMPATIBLE_SOURCES)
+    set(MULTITHREADED_SERVICE_SOURCE "service.cpp")
 endif()
 
 add_library(bergamot-translator STATIC
@@ -13,7 +13,7 @@ add_library(bergamot-translator STATIC
     multifactor_priority.cpp 
     request.cpp 
     service_base.cpp
-    ${SERVICE}
+    ${MULTITHREADED_SERVICE_SOURCE}
     batcher.cpp
     response.cpp
     batch.cpp
@@ -32,10 +32,6 @@ if(COMPILE_WASM)
   target_compile_options(bergamot-translator PRIVATE ${WASM_COMPILE_FLAGS})
 endif(COMPILE_WASM)
 
-if (COMPILE_THREAD_VARIANT)
-  target_compile_definitions(bergamot-translator PRIVATE WITH_PTHREADS)
-endif()
-
 target_link_libraries(bergamot-translator marian ssplit)
 
 target_include_directories(bergamot-translator

From 0be73705d902cb23cc01efc6f01d4492d5b198e2 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Fri, 26 Feb 2021 14:55:30 +0100
Subject: [PATCH 163/442] Fixed native builds while using wasm compatible
 sources

 - main-mts and marian-decoder-new can't be used because
   it uses multi-threaded variant of Service class
---
 CMakeLists.txt     |  6 +++---
 app/CMakeLists.txt | 10 ++++++----
 2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 45a857f01..a84b8e4ad 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -67,9 +67,9 @@ endif(COMPILE_WASM)
 
 add_subdirectory(3rd_party)
 add_subdirectory(src)
-if(NOT COMPILE_WASM)
-  add_subdirectory(app)
-endif()
+
 if(COMPILE_WASM)
   add_subdirectory(wasm)
+else()
+  add_subdirectory(app)
 endif(COMPILE_WASM)
diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index 24bd0b43e..41c5cd0a9 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -1,8 +1,10 @@
 add_executable(bergamot-translator-app main.cpp)
 target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
 
-add_executable(service-cli main-mts.cpp)
-target_link_libraries(service-cli PRIVATE bergamot-translator)
+if (NOT USE_WASM_COMPATIBLE_SOURCES)
+    add_executable(service-cli main-mts.cpp)
+    target_link_libraries(service-cli PRIVATE bergamot-translator)
 
-add_executable(marian-decoder-new marian-decoder-new.cpp)
-target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)
+    add_executable(marian-decoder-new marian-decoder-new.cpp)
+    target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)
+endif()

From f17f02a54431403956d1fa333f44d1144e66dd8f Mon Sep 17 00:00:00 2001
From: Ulrich Germann <ulrich.germann@gmail.com>
Date: Wed, 3 Mar 2021 10:48:56 +0000
Subject: [PATCH 164/442] Update submodule ssplit-cpp

---
 .gitmodules          | 6 +++---
 3rd_party/ssplit-cpp | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/.gitmodules b/.gitmodules
index e4feab500..cc40735d6 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,6 +1,6 @@
-[submodule "3rd_party/ssplit-cpp"]
-	path = 3rd_party/ssplit-cpp
-	url = https://github.com/abhi-agg/ssplit-cpp
 [submodule "3rd_party/marian-dev"]
 	path = 3rd_party/marian-dev
 	url = https://github.com/browsermt/marian-dev
+[submodule "3rd_party/ssplit-cpp"]
+	path = 3rd_party/ssplit-cpp
+	url = https://github.com/browsermt/ssplit-cpp
diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 432208826..dfefe3421 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 432208826ee27e7b3984b53774b1a16d74256d77
+Subproject commit dfefe34218fe3aced70266994b6557f029fcbdde

From d3ef1a9bc3dafefbe82edfb2da9ea078203acba5 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 10 Mar 2021 10:33:11 +0100
Subject: [PATCH 165/442] Updated marian submodule

 - This fixes the binary model loading problem for wasm
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 05f2517f5..8ddb73fad 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 05f2517f58de493d2f42236c2d23db95a9edbd8f
+Subproject commit 8ddb73fad1001ae4c1697d2514ac1e5bd43e2ed3

From c64deb50a888079bb11a579a99c4df317a93d21d Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 10 Mar 2021 18:30:39 +0100
Subject: [PATCH 166/442] Imported CI scripts from
 mozilla/bergamot-translator-old (#1)

* CircleCI config, docs and badge

* Increase CircleCI RAM from 4gb to 16gb

Co-authored-by: Motin <motin@motin.eu>
---
 .circleci/config.yml | 19 ++++++++++++
 .gitignore           |  2 +-
 README.md            |  2 ++
 build-wasm.sh        | 70 ++++++++++++++++++++++++++++++++++++++++++++
 doc/CI.md            | 22 ++++++++++++++
 5 files changed, 114 insertions(+), 1 deletion(-)
 create mode 100644 .circleci/config.yml
 create mode 100755 build-wasm.sh
 create mode 100644 doc/CI.md

diff --git a/.circleci/config.yml b/.circleci/config.yml
new file mode 100644
index 000000000..6bc00febb
--- /dev/null
+++ b/.circleci/config.yml
@@ -0,0 +1,19 @@
+version: 2.1
+jobs:
+  build:
+    docker:
+      - image: 'emscripten/emsdk:2.0.9'
+    resource_class: xlarge
+
+    working_directory: ~/checkout
+
+    steps:
+      - checkout
+
+      - run:
+          name: Build WASM
+          command: bash build-wasm.sh
+
+      - store_artifacts:
+          path: "build-wasm/wasm"
+          destination: "build-wasm/wasm"
diff --git a/.gitignore b/.gitignore
index 3acc4a57e..840e69ab8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -17,6 +17,6 @@ _deps
 
 
 wasm/test_page/node_modules
-build-*
+build-wasm
 models
 wasm/test_page/bergamot-translator-worker.*
diff --git a/README.md b/README.md
index 2b7f0a6c7..92fb874c8 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,7 @@
 # Bergamot Translator
 
+[![CircleCI badge](https://img.shields.io/circleci/project/github/mozilla/bergamot-translator/main.svg?label=CircleCI)](https://circleci.com/gh/mozilla/bergamot-translator/)
+
 Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.github.io/) framework based) neural machine translation functionality in accordance with the [Bergamot](https://browser.mt/) project that focuses on improving client-side machine translation in a web browser.
 
 ## Build Instructions
diff --git a/build-wasm.sh b/build-wasm.sh
new file mode 100755
index 000000000..a4ea8833a
--- /dev/null
+++ b/build-wasm.sh
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+
+# Usage: ./build-wasm.sh
+
+set -e
+set -x
+
+# Run script from the context of the script-containing directory
+cd "$(dirname $0)"
+
+# This file replicates the instructions found in ./README.md under "Build WASM"
+# with slight adjustments to be able to run the build script multiple times without having to clone all dependencies
+# as per "As long as you don't update any submodule, just follow steps in `4.ii` to recompile."
+
+# 1. Download and Install Emscripten using following instructions (unless the EMSDK env var is already set)
+if [ "$EMSDK" == "" ]; then
+  EMSDK_UPDATE_REQUIRED=0
+  if [ ! -d "emsdk" ]; then
+    git clone https://github.com/emscripten-core/emsdk.git
+    EMSDK_UPDATE_REQUIRED=1
+  else
+    cd emsdk
+    git fetch
+    # Only pull if necessary
+    if [ $(git rev-parse HEAD) != $(git rev-parse @{u}) ]; then
+      git pull --ff-only
+      EMSDK_UPDATE_REQUIRED=1
+    fi
+    cd -
+  fi
+  if [ "$EMSDK_UPDATE_REQUIRED" == "1" ]; then
+    cd emsdk
+    ./emsdk install latest
+    ./emsdk activate latest
+    cd -
+  fi
+  source ./emsdk/emsdk_env.sh
+fi
+
+# 3. Download models (only required if you want to package files in wasm binary)
+if [ ! -d "bergamot-models" ]; then
+  git clone https://github.com/mozilla-applied-ml/bergamot-models
+else
+  cd bergamot-models
+  git fetch
+  # Only pull if necessary
+  if [ $(git rev-parse HEAD) != $(git rev-parse @{u}) ]; then
+    git pull --ff-only
+  fi
+  cd -
+fi
+mkdir -p models
+rm -rf models/*
+cp -rf bergamot-models/* models
+gunzip models/*/*
+
+# 4. Compile
+#     1. Create a folder where you want to build all the artefacts (`build-wasm` in this case)
+if [ ! -d "build-wasm" ]; then
+  mkdir build-wasm
+fi
+cd build-wasm
+
+#     2. Compile the artefacts
+emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR="../models/" ../
+emmake make -j
+
+# The artefacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
+
+exit 0
diff --git a/doc/CI.md b/doc/CI.md
new file mode 100644
index 000000000..2f29b02c1
--- /dev/null
+++ b/doc/CI.md
@@ -0,0 +1,22 @@
+# Continuous Integration
+
+[Circle CI](https://circleci.com/) is used for continuous integration. Configured via `./.circleci/config.yml`.
+
+##  Run Circle CI locally (requires Docker)
+
+1. [Install the CircleCI local cli](https://circleci.com/docs/2.0/local-cli/#installation)
+2. Validate Circle CI configuration (useful exercise before pushing any changes to the configuration)
+
+```shell
+circleci config validate -c .circleci/config.yml
+```
+
+3. To better mimic the starting point for CI, commit your changes and clone your repository into a clean directory then run CircleCI inside that directory:
+
+```shell
+git clone . /tmp/$(basename $PWD)
+cd /tmp/$(basename $PWD)
+circleci build
+```
+
+Note: Steps related to caching and uploading/storing artifacts will report as failed locally. This is not necessarily a problem, they are designed to fail since the operations are not supported locally by the CircleCI build agent.

From f89c989b444841afd7458df97152ac1401f23750 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 10 Mar 2021 14:55:37 +0000
Subject: [PATCH 167/442] apt-update for ubuntu github actions

---
 .github/workflows/ubuntu.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml
index 940ecddb0..6ddd49ade 100644
--- a/.github/workflows/ubuntu.yml
+++ b/.github/workflows/ubuntu.yml
@@ -62,7 +62,7 @@ jobs:
     # No need to install libprotobuf{17,10,9v5} on Ubuntu {20,18,16}.04 because
     # it is installed together with libprotobuf-dev
     - name: Install dependencies
-      run: sudo apt-get install -y libgoogle-perftools-dev libprotobuf-dev protobuf-compiler libboost-all-dev
+      run: sudo apt-get update && sudo apt-get install -y libgoogle-perftools-dev libprotobuf-dev protobuf-compiler libboost-all-dev
 
     # https://software.intel.com/content/www/us/en/develop/articles/installing-intel-free-libs-and-python-apt-repo.html
     - name: Install MKL

From e96d7047a7042336908c06197180b4891e76a66d Mon Sep 17 00:00:00 2001
From: Andre Natal <anatal@gmail.com>
Date: Wed, 10 Mar 2021 19:57:13 -0800
Subject: [PATCH 168/442] Enable SIMD wormhole in circle ci build scripts

 - Modify the APIs that compile & instantiate WASM module
---
 build-wasm.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/build-wasm.sh b/build-wasm.sh
index a4ea8833a..d490730e2 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -65,6 +65,11 @@ cd build-wasm
 emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR="../models/" ../
 emmake make -j
 
+#     3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
+sed -i.bak 's/var result = WebAssembly.instantiateStreaming(response, info);/var result = WebAssembly.instantiateStreaming(response, info,{simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+sed -i.bak 's/return WebAssembly.instantiate(binary, info);/return WebAssembly.instantiate(binary, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+sed -i.bak 's/var module = new WebAssembly.Module(bytes);/var module = new WebAssembly.Module(bytes, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+
 # The artefacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
 
 exit 0

From 8c8913e2effa02b0e763fdb249d1074443cb44e1 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Mar 2021 11:07:21 +0100
Subject: [PATCH 169/442] Use intgemm models in wasm test_page

---
 wasm/test_page/bergamot.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index b4d2027b0..d7207105c 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -70,7 +70,7 @@
     // Set the Model Configuration as YAML formatted string.
     // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
     const modelConfig = `models:
-  - /${languagePair}/model.${languagePair}.npz
+  - /${languagePair}/model.${languagePair}.intgemm.bin
 vocabs:
   - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
   - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm

From 4f124e7976a163b5d7dbead2e70a17eec11d81ed Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Mar 2021 11:23:04 +0100
Subject: [PATCH 170/442] Enabled simdwormhole in github workflows

---
 .github/workflows/macos-custom-marian-wasm.yml | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/.github/workflows/macos-custom-marian-wasm.yml b/.github/workflows/macos-custom-marian-wasm.yml
index 3e49369b4..373a26268 100644
--- a/.github/workflows/macos-custom-marian-wasm.yml
+++ b/.github/workflows/macos-custom-marian-wasm.yml
@@ -33,6 +33,13 @@ jobs:
         working-directory: build-wasm
         run: emmake make -j2
 
+      - name: Instantiate simd wormhole
+        working-directory: build-wasm
+        run: |
+          sed -i.bak 's/var result = WebAssembly.instantiateStreaming(response, info);/var result = WebAssembly.instantiateStreaming(response, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+          sed -i.bak 's/return WebAssembly.instantiate(binary, info);/return WebAssembly.instantiate(binary, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+          sed -i.bak 's/var module = new WebAssembly.Module(bytes);/var module = new WebAssembly.Module(bytes, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+
       - name: Check artifacts
         working-directory: build-wasm
         run: |

From 6e7b7c71ec34883f7fdbf67ef7f56db0059ef328 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Mar 2021 11:41:07 +0100
Subject: [PATCH 171/442] Updates README for enabling simdwormhole in WASM APIs

---
 README.md | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/README.md b/README.md
index 92fb874c8..299cc877c 100644
--- a/README.md
+++ b/README.md
@@ -77,6 +77,12 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
             emmake make -j
             ```
 
+    3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
+        ```
+        sed -i.bak 's/var result = WebAssembly.instantiateStreaming(response, info);/var result = WebAssembly.instantiateStreaming(response, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+        sed -i.bak 's/return WebAssembly.instantiate(binary, info);/return WebAssembly.instantiate(binary, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+        sed -i.bak 's/var module = new WebAssembly.Module(bytes);/var module = new WebAssembly.Module(bytes, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+        ```
     The artefacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
 
 #### Recompiling

From a2d66500976fdbd2b586e240d920b096fcedd529 Mon Sep 17 00:00:00 2001
From: Andre Natal <anatal@gmail.com>
Date: Wed, 10 Mar 2021 17:59:07 -0800
Subject: [PATCH 172/442] Patch to load just production ready models

---
 README.md     | 2 +-
 build-wasm.sh | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index 299cc877c..cb1bc03aa 100644
--- a/README.md
+++ b/README.md
@@ -49,7 +49,7 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
     ```bash
     mkdir models
     git clone https://github.com/mozilla-applied-ml/bergamot-models
-    cp -rf bergamot-models/* models
+    cp -rf bergamot-models/prod/* models
     gunzip models/*/*
     ```
 
diff --git a/build-wasm.sh b/build-wasm.sh
index d490730e2..8349bf8c2 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -51,7 +51,7 @@ else
 fi
 mkdir -p models
 rm -rf models/*
-cp -rf bergamot-models/* models
+cp -rf bergamot-models/prod/* models
 gunzip models/*/*
 
 # 4. Compile

From d1ecd007a696c133112f3b88fa3cdd92bfc2e61d Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Thu, 11 Mar 2021 10:27:47 +0200
Subject: [PATCH 173/442] Shallow clone bergamot-models repo since it has such
 a large history

---
 README.md     | 2 +-
 build-wasm.sh | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index cb1bc03aa..765d7cb7c 100644
--- a/README.md
+++ b/README.md
@@ -48,7 +48,7 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
     If you want to package bergamot project specific models, please follow these instructions:
     ```bash
     mkdir models
-    git clone https://github.com/mozilla-applied-ml/bergamot-models
+    git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
     cp -rf bergamot-models/prod/* models
     gunzip models/*/*
     ```
diff --git a/build-wasm.sh b/build-wasm.sh
index 8349bf8c2..9b08aa77b 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -39,7 +39,7 @@ fi
 
 # 3. Download models (only required if you want to package files in wasm binary)
 if [ ! -d "bergamot-models" ]; then
-  git clone https://github.com/mozilla-applied-ml/bergamot-models
+  git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
 else
   cd bergamot-models
   git fetch

From bf28edad82660db6a9524ae0e7f05dab007f78ce Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 11 Mar 2021 14:25:42 +0100
Subject: [PATCH 174/442] Improved wasm test_page

 - test page can load all 5 language pairs
 - Use intgemm.aplha* models
 - start_server.sh script automatically enable simdwormhole via
   APIs that instantiate WASM module
---
 wasm/test_page/bergamot.html   |  5 ++++-
 wasm/test_page/start_server.sh | 11 ++++++++++-
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index d7207105c..4f1f2a0f7 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -32,6 +32,9 @@
     <label>Choose the model to use</label>
     <input type="radio" name="modellang" value="enes"/><label>English to Spanish</label>
     <input type="radio" name="modellang" value="esen" checked/><label>Spanish to English</label>
+    <input type="radio" name="modellang" value="eten" checked/><label>Estonian to English</label>
+    <input type="radio" name="modellang" value="enet" checked/><label>English to Estonian</label>
+    <input type="radio" name="modellang" value="ende" checked/><label>English to German</label>
     <input type="button" id="load" value="Load Model"/>
 </div>
 
@@ -70,7 +73,7 @@
     // Set the Model Configuration as YAML formatted string.
     // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
     const modelConfig = `models:
-  - /${languagePair}/model.${languagePair}.intgemm.bin
+  - /${languagePair}/model.${languagePair}.intgemm.alphas.bin
 vocabs:
   - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
   - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
diff --git a/wasm/test_page/start_server.sh b/wasm/test_page/start_server.sh
index b83344b8a..153624e21 100644
--- a/wasm/test_page/start_server.sh
+++ b/wasm/test_page/start_server.sh
@@ -1,8 +1,17 @@
 #!/bin/bash
-
+echo "Start: Copying artifacts in local folder------"
 cp ../../build-wasm/wasm/bergamot-translator-worker.data .
 cp ../../build-wasm/wasm/bergamot-translator-worker.js .
 cp ../../build-wasm/wasm/bergamot-translator-worker.wasm .
 cp ../../build-wasm/wasm/bergamot-translator-worker.worker.js .
+echo "Done----"
+
+echo "Start: Enabling wormhole via APIs that compile and instantiate wasm module-------"
+sed -i.bak 's/var result = WebAssembly.instantiateStreaming(response, info);/var result = WebAssembly.instantiateStreaming(response, info, {simdWormhole:true});/g' bergamot-translator-worker.js
+sed -i.bak 's/return WebAssembly.instantiate(binary, info);/return WebAssembly.instantiate(binary, info, {simdWormhole:true});/g' bergamot-translator-worker.js
+sed -i.bak 's/var module = new WebAssembly.Module(bytes);/var module = new WebAssembly.Module(bytes, {simdWormhole:true});/g' bergamot-translator-worker.js
+echo "Done: Enabling wormhole via APIs that compile and instantiate wasm module--------"
+
 npm install
+echo "Start httpserver"
 node bergamot-httpserver.js
\ No newline at end of file

From d75dd85def5f86edd7a6a4f5e35840f1654738c1 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Mon, 22 Mar 2021 14:22:56 +0000
Subject: [PATCH 175/442] Load mode as a byte array (#55)

* Switch to wasm branch for this example

* Load marian model from a byte array

* Sanitise executable names

* Change marian branch

* Update marian branch that loads binary models

* Example of loading model as a byte array

* Add the byte array loading files

* Die on misaligned memory

* Remove the unused argument

* Allow loading without a ptr parameter so that we don't break emc workflow
---
 3rd_party/marian-dev                          |  2 +-
 app/CMakeLists.txt                            | 10 ++-
 app/bergamot-translator-app-bytearray.cpp     | 72 +++++++++++++++++++
 app/{main.cpp => bergamot-translator-app.cpp} |  0
 app/service-cli-bytearray.cpp                 | 40 +++++++++++
 app/{main-mts.cpp => service-cli.cpp}         |  0
 src/AbstractTranslationModel.h                |  6 +-
 src/translator/AbstractTranslationModel.cpp   |  8 ++-
 src/translator/CMakeLists.txt                 |  1 +
 src/translator/TranslationModel.cpp           |  4 +-
 src/translator/TranslationModel.h             |  6 +-
 src/translator/batch_translator.cpp           | 16 ++++-
 src/translator/batch_translator.h             | 12 +++-
 src/translator/byteArrayExample.cpp           | 45 ++++++++++++
 src/translator/byteArrayExample.h             |  8 +++
 src/translator/service.cpp                    |  6 +-
 src/translator/service.h                      |  7 +-
 src/translator/service_base.cpp               |  4 +-
 src/translator/service_base.h                 |  9 ++-
 19 files changed, 235 insertions(+), 21 deletions(-)
 create mode 100644 app/bergamot-translator-app-bytearray.cpp
 rename app/{main.cpp => bergamot-translator-app.cpp} (100%)
 create mode 100644 app/service-cli-bytearray.cpp
 rename app/{main-mts.cpp => service-cli.cpp} (100%)
 create mode 100644 src/translator/byteArrayExample.cpp
 create mode 100644 src/translator/byteArrayExample.h

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 8ddb73fad..370fdb5a2 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 8ddb73fad1001ae4c1697d2514ac1e5bd43e2ed3
+Subproject commit 370fdb5a212cfcd2d1c5fca9fffc041d2787a432
diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index 41c5cd0a9..3f2b9de49 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -1,10 +1,16 @@
-add_executable(bergamot-translator-app main.cpp)
+add_executable(bergamot-translator-app bergamot-translator-app.cpp)
 target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
 
+add_executable(bergamot-translator-app-bytearray bergamot-translator-app-bytearray.cpp)
+target_link_libraries(bergamot-translator-app-bytearray PRIVATE bergamot-translator)
+
 if (NOT USE_WASM_COMPATIBLE_SOURCES)
-    add_executable(service-cli main-mts.cpp)
+    add_executable(service-cli service-cli.cpp)
     target_link_libraries(service-cli PRIVATE bergamot-translator)
 
+    add_executable(service-cli-bytearray service-cli-bytearray.cpp)
+    target_link_libraries(service-cli-bytearray PRIVATE bergamot-translator)
+
     add_executable(marian-decoder-new marian-decoder-new.cpp)
     target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)
 endif()
diff --git a/app/bergamot-translator-app-bytearray.cpp b/app/bergamot-translator-app-bytearray.cpp
new file mode 100644
index 000000000..961a24e2f
--- /dev/null
+++ b/app/bergamot-translator-app-bytearray.cpp
@@ -0,0 +1,72 @@
+/*
+ * main.cpp
+ *
+ * An example application to demonstrate the use of Bergamot translator.
+ *
+ */
+
+#include <iostream>
+
+#include "AbstractTranslationModel.h"
+#include "TranslationRequest.h"
+#include "TranslationResult.h"
+#include "translator/parser.h"
+#include "translator/byteArrayExample.h"
+
+int main(int argc, char **argv) {
+
+  // Create a configParser and load command line parameters into a YAML config
+  // string.
+  auto configParser = marian::bergamot::createConfigParser();
+  auto options = configParser.parseOptions(argc, argv, true);
+  std::string config = options->asYamlString();
+
+  // Route the config string to construct marian model through
+  // AbstractTranslationModel
+  void * model_bytes = bergamot::getBinaryModelFromConfig(options);
+  std::shared_ptr<AbstractTranslationModel> model =
+      AbstractTranslationModel::createInstance(config, model_bytes);
+
+  TranslationRequest translationRequest;
+  std::vector<std::string> texts;
+  texts.emplace_back(
+      "The Bergamot project will add and improve client-side machine "
+      "translation in a web browser.  Unlike current cloud-based "
+      "options, running directly on users’ machines empowers citizens to "
+      "preserve their privacy and increases the uptake of language "
+      "technologies in Europe in various sectors that require "
+      "confidentiality.");
+  texts.emplace_back(
+      "Free software integrated with an open-source web "
+      "browser, such as Mozilla Firefox, will enable bottom-up adoption "
+      "by non-experts, resulting in cost savings for private and public "
+      "sector users who would otherwise procure translation or operate "
+      "monolingually.  Bergamot is a consortium coordinated by the "
+      "University of Edinburgh with partners Charles University in "
+      "Prague, the University of Sheffield, University of Tartu, and "
+      "Mozilla.");
+
+  auto results = model->translate(std::move(texts), translationRequest);
+
+  // Resolve the future and get the actual result
+  //std::vector<TranslationResult> results = futureResults.get();
+
+  for (auto &result : results) {
+    std::cout << "[original]: " << result.getOriginalText() << std::endl;
+    std::cout << "[translated]: " << result.getTranslatedText() << std::endl;
+    auto mappings = result.getSentenceMappings();
+    for (auto &p : mappings) {
+      std::string_view src = p.first;
+      std::string_view tgt = p.second;
+
+      std::cout << " [src Sentence]: " << src << std::endl;
+      std::cout << " [tgt Sentence]: " << tgt << std::endl;
+    }
+    std::cout << std::endl;
+  }
+
+  // Clear the memory used for the byte array
+  free(model_bytes); // Ideally, this should be done after the translation model has been gracefully shut down.
+
+  return 0;
+}
diff --git a/app/main.cpp b/app/bergamot-translator-app.cpp
similarity index 100%
rename from app/main.cpp
rename to app/bergamot-translator-app.cpp
diff --git a/app/service-cli-bytearray.cpp b/app/service-cli-bytearray.cpp
new file mode 100644
index 000000000..42a2af3fb
--- /dev/null
+++ b/app/service-cli-bytearray.cpp
@@ -0,0 +1,40 @@
+#include <cstdlib>
+#include <future>
+#include <iostream>
+#include <sstream>
+
+#include "common/definitions.h"
+#include "common/utils.h"
+#include "marian.h"
+#include "translator/parser.h"
+#include "translator/response.h"
+#include "translator/service.h"
+#include "translator/byteArrayExample.h"
+
+int main(int argc, char *argv[]) {
+  auto cp = marian::bergamot::createConfigParser();
+  auto options = cp.parseOptions(argc, argv, true);
+
+  void * model_bytes = bergamot::getBinaryModelFromConfig(options);
+  marian::bergamot::Service service(options, model_bytes);
+
+  // Read a large input text blob from stdin
+  std::ostringstream std_input;
+  std_input << std::cin.rdbuf();
+  std::string input = std_input.str();
+  using marian::bergamot::Response;
+
+  // Wait on future until Response is complete
+  std::future<Response> responseFuture = service.translate(std::move(input));
+  responseFuture.wait();
+  Response response = responseFuture.get();
+  std::cout << response.translation() << std::endl;
+
+  // Stop Service.
+  service.stop();
+
+  // Clear the memory used for the byte array
+  free(model_bytes); // Ideally, this should be done after the translation model has been gracefully shut down.
+
+  return 0;
+}
diff --git a/app/main-mts.cpp b/app/service-cli.cpp
similarity index 100%
rename from app/main-mts.cpp
rename to app/service-cli.cpp
diff --git a/src/AbstractTranslationModel.h b/src/AbstractTranslationModel.h
index 7562b0ad0..82dcbfa09 100644
--- a/src/AbstractTranslationModel.h
+++ b/src/AbstractTranslationModel.h
@@ -28,8 +28,12 @@ class AbstractTranslationModel {
    * AbstractTranslationModel. The instance is created using translation model
    * configuration provided as yaml-formatted string.
    */
+  /**
+   * @param config Marian yml config file in the form of a string
+   * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Optional, defaults to nullptr when not used
+   */
   static std::shared_ptr<AbstractTranslationModel>
-  createInstance(const std::string &config);
+  createInstance(const std::string &config, const void * model_memory=nullptr);
 
   AbstractTranslationModel() = default;
 
diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
index 1b2f2b104..b731c60d2 100644
--- a/src/translator/AbstractTranslationModel.cpp
+++ b/src/translator/AbstractTranslationModel.cpp
@@ -8,7 +8,11 @@
 #include "AbstractTranslationModel.h"
 #include "TranslationModel.h"
 
+/**
+ * @param config Marian yml config file in the form of a string
+ * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Optional, defaults to nullptr when not used
+ */
 std::shared_ptr<AbstractTranslationModel>
-AbstractTranslationModel::createInstance(const std::string &config) {
-  return std::make_shared<TranslationModel>(config);
+AbstractTranslationModel::createInstance(const std::string &config, const void * model_memory) {
+  return std::make_shared<TranslationModel>(config, model_memory);
 }
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 5c7269e07..d149c0392 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -7,6 +7,7 @@ add_library(bergamot-translator STATIC
     TranslationModel.cpp
 
     # Following files added from browsermt/mts@nuke
+    byteArrayExample.cpp
     text_processor.cpp
     sentence_splitter.cpp
     batch_translator.cpp 
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index b3e0db6d8..94f53a80f 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -50,9 +50,9 @@ std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
   return std::make_shared<marian::Options>(options);
 }
 
-TranslationModel::TranslationModel(const std::string &config)
+TranslationModel::TranslationModel(const std::string &config, const void * model_memory)
     : configOptions_(std::move(parseOptions(config))),
-      AbstractTranslationModel(), service_(configOptions_) {}
+      AbstractTranslationModel(), service_(configOptions_, model_memory) {}
 
 TranslationModel::~TranslationModel() {}
 
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index e4765965a..92fca6336 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -27,7 +27,11 @@ class TranslationModel : public AbstractTranslationModel {
   /* Construct the model using the model configuration options as yaml-formatted
    * string
    */
-  TranslationModel(const std::string &config);
+  /**
+   * @param config Marian yml config file in the form of a string
+   * @param model_memory optional byte array (aligned to 64!!!) that contains the bytes of a model.bin.
+   */
+  TranslationModel(const std::string &config, const void * model_memory = nullptr);
 
   ~TranslationModel();
 
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 6036c3799..c83cf8cd6 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -10,8 +10,9 @@ namespace bergamot {
 
 BatchTranslator::BatchTranslator(DeviceId const device,
                                  std::vector<Ptr<Vocab const>> &vocabs,
-                                 Ptr<Options> options)
-    : device_(device), options_(options), vocabs_(&vocabs) {}
+                                 Ptr<Options> options,
+                                 const void * model_memory)
+    : device_(device), options_(options), vocabs_(&vocabs), model_memory_(model_memory) {}
 
 void BatchTranslator::initialize() {
   // Initializes the graph.
@@ -29,7 +30,16 @@ void BatchTranslator::initialize() {
   graph_->setDevice(device_);
   graph_->getBackend()->configureDevice(options_);
   graph_->reserveWorkspaceMB(options_->get<size_t>("workspace"));
-  scorers_ = createScorers(options_);
+  if (model_memory_) { // If we have provided a byte array that contains the model memory, we can initialise the model from there, as opposed to from reading in the config file
+    if ((uintptr_t)model_memory_ % 256 != 0) {
+      std::cerr << "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it." << std::endl;
+      exit(1);
+    }
+    const std::vector<const void *> container = {model_memory_}; // Marian supports multiple models initialised in this manner hence std::vector. However we will only ever use 1 during decoding.
+    scorers_ = createScorers(options_, container);
+  } else {
+    scorers_ = createScorers(options_);
+  }
   for (auto scorer : scorers_) {
     scorer->init(graph_);
     if (slgen_) {
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 50a7b7999..683c27922 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -26,8 +26,15 @@ class BatchTranslator {
   // shut down in Service which calls join() on the threads.
 
 public:
-  BatchTranslator(DeviceId const device, std::vector<Ptr<Vocab const>> &vocabs,
-                  Ptr<Options> options);
+  /**
+   * Initialise the marian translator.
+   * @param device DeviceId that performs translation. Could be CPU or GPU
+   * @param vocabs Vector that contains ptrs to two vocabs
+   * @param options Marian options object
+   * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Provide a nullptr if not used.
+   */
+  explicit BatchTranslator(DeviceId const device, std::vector<Ptr<Vocab const>> &vocabs,
+                  Ptr<Options> options, const void * model_memory);
 
   // convenience function for logging. TODO(jerin)
   std::string _identifier() { return "worker" + std::to_string(device_.no); }
@@ -41,6 +48,7 @@ class BatchTranslator {
   Ptr<ExpressionGraph> graph_;
   std::vector<Ptr<Scorer>> scorers_;
   Ptr<data::ShortlistGenerator const> slgen_;
+  const void * model_memory_;
 };
 
 } // namespace bergamot
diff --git a/src/translator/byteArrayExample.cpp b/src/translator/byteArrayExample.cpp
new file mode 100644
index 000000000..28f9d9ba4
--- /dev/null
+++ b/src/translator/byteArrayExample.cpp
@@ -0,0 +1,45 @@
+#include "byteArrayExample.h"
+#include <stdlib.h>
+#include <fstream>
+#include <iostream>
+
+namespace bergamot {
+
+void * getBinaryFile(std::string path) {
+    std::ifstream is (path, std::ifstream::binary);
+    uint64_t length = 0; // Determine the length of file in bytes
+    if (is) {
+        is.seekg(0, is.end);
+        length = is.tellg();
+        is.seekg(0, is.beg);
+    } else {
+        std::cerr << "Failed opening file stream: " << path << std::endl;
+        std::exit(1);
+    }
+    void *result;
+    int fail = posix_memalign(&result, 256, length);
+    if (fail) {
+        std::cerr << "Failed to allocate aligned memory." << std::endl;
+        std::exit(1);
+    }
+    is.read(static_cast<char *>(result), length);
+    return result;
+}
+
+void * getBinaryModelFromConfig(marian::Ptr<marian::Options> options) {
+    std::vector<std::string> models = options->get<std::vector<std::string>>("models");
+    if (models.size() != 1) {
+        std::cerr << "Loading multiple binary models is not supported for now as it is not necessary." << std::endl;
+        std::exit(1);
+        marian::filesystem::Path modelPath(models[0]);
+        if (modelPath.extension() != marian::filesystem::Path(".bin")) {
+            std::cerr << "Non binary models cannot be loaded as a byte array." << std::endl;
+            std::exit(1);
+        }
+        return nullptr;
+    } else {
+        return getBinaryFile(models[0]);
+    }
+}
+
+} // namespace bergamot
diff --git a/src/translator/byteArrayExample.h b/src/translator/byteArrayExample.h
new file mode 100644
index 000000000..321ea5d5f
--- /dev/null
+++ b/src/translator/byteArrayExample.h
@@ -0,0 +1,8 @@
+#include "marian.h"
+
+namespace bergamot {
+
+void * getBinaryFile(std::string path);
+void * getBinaryModelFromConfig(marian::Ptr<marian::Options> options);
+
+} // namespace bergamot
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 2101b8ea1..86e8c59eb 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -8,9 +8,9 @@
 namespace marian {
 namespace bergamot {
 
-Service::Service(Ptr<Options> options)
+Service::Service(Ptr<Options> options, const void * model_memory)
     : ServiceBase(options), numWorkers_(options->get<int>("cpu-threads")),
-      pcqueue_(numWorkers_) {
+      pcqueue_(numWorkers_), model_memory_{model_memory} {
   if (numWorkers_ == 0) {
     ABORT("Fatal: Attempt to create multithreaded instance with --cpu-threads "
           "0. ");
@@ -21,7 +21,7 @@ Service::Service(Ptr<Options> options)
 
   for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
     marian::DeviceId deviceId(cpuId, DeviceType::cpu);
-    translators_.emplace_back(deviceId, vocabs_, options);
+    translators_.emplace_back(deviceId, vocabs_, options, model_memory_);
     auto &translator = translators_.back();
 
     workers_.emplace_back([&translator, this] {
diff --git a/src/translator/service.h b/src/translator/service.h
index 9a57996c5..38de3ae99 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -31,7 +31,11 @@ class Service : public ServiceBase {
   //  Response result = response.get();
 
 public:
-  explicit Service(Ptr<Options> options);
+  /**
+   * @param options Marian options object
+   * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Optional, defaults to nullptr when not used
+   */
+  explicit Service(Ptr<Options> options, const void * model_memory=nullptr);
   // Implements enqueue and top through blocking methods.
   void stop() override;
   ~Service();
@@ -46,6 +50,7 @@ class Service : public ServiceBase {
 
   size_t numWorkers_;      // ORDER DEPENDENCY
   PCQueue<Batch> pcqueue_; // ORDER DEPENDENCY
+  const void * model_memory_;
   std::vector<std::thread> workers_;
   std::vector<BatchTranslator> translators_;
 };
diff --git a/src/translator/service_base.cpp b/src/translator/service_base.cpp
index 1a5f51144..dc0c05aaf 100644
--- a/src/translator/service_base.cpp
+++ b/src/translator/service_base.cpp
@@ -24,9 +24,9 @@ std::future<Response> ServiceBase::translate(std::string &&input) {
   return future;
 }
 
-NonThreadedService::NonThreadedService(Ptr<Options> options)
+NonThreadedService::NonThreadedService(Ptr<Options> options, const void * model_memory)
     : ServiceBase(options),
-      translator_(DeviceId(0, DeviceType::cpu), vocabs_, options) {
+      translator_(DeviceId(0, DeviceType::cpu), vocabs_, options, model_memory) {
   translator_.initialize();
 }
 
diff --git a/src/translator/service_base.h b/src/translator/service_base.h
index 25cc003e9..a44273fae 100644
--- a/src/translator/service_base.h
+++ b/src/translator/service_base.h
@@ -16,6 +16,9 @@ namespace bergamot {
 
 class ServiceBase {
 public:
+  /**
+   * @param options Marian options object
+   */
   explicit ServiceBase(Ptr<Options> options);
 
   // Transfers ownership of input string to Service, returns a future containing
@@ -44,7 +47,11 @@ class ServiceBase {
 
 class NonThreadedService : public ServiceBase {
 public:
-  explicit NonThreadedService(Ptr<Options> options);
+  /**
+   * @param options Marian options object
+   * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Provide a nullptr if not used.
+   */
+  explicit NonThreadedService(Ptr<Options> options, const void * model_memory);
   void stop() override{};
 
 private:

From 34228d37bfcc44e36bf01d8ea70fbc2fce9d9ae9 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 23 Mar 2021 16:36:13 +0000
Subject: [PATCH 176/442] Collapse Service into one class instead of three
 (#62)

* Merging two Services

* Moving stop() logic to destructor

* We have WITH_PTHREADS back

* string based constructor on Service

* Removing now empty service_base.* files

* Hiding away pcqueue_ construction

Ugliest ifdefs I have done in my life.

* Another ifdef to hide pcqueue header file

* Missing semicolons in WITH_PTHREADS path

* Fixing async_translate residue argument from copy

* Adding comments

* Initialize batchtranslator only at one place

To reduce tax for bytebuffer loads, initialize batchtranslator only at
one place.

* \#ifdef WITH_PTHREADS -> #ifndef WASM_HIDE_THREADS

Sane platform (non WASM) is default. This truly only hide-threads from
compilation path and not switch unswitch pthreads (-lpthread).

* Review comments: Rearranging destructor, fix wrong comment

* Move loadVocabularies to service.cpp and put in anonymous namespace

* Prettifying diff: Removing unwanted empty lines

* Indicate in comments multithreaded has numWorkers translators

* Typo fix: bergamot_translator -> bergamot-translator

* Safety guards to avoid pcqueue illegal init

* Add WASM_HIDE_THREADS as a global WASM_COMPILE_FLAG

* Compile Defs: WASM_HIDE_THREADS -> __EMSCRIPTEN__

* Removing dead CMakeLists.txt code following __EMSCRIPTEN__

* Compile defs: __EMSCRIPTEN__ -> WASM
---
 app/marian-decoder-new.cpp          |   1 -
 app/service-cli-bytearray.cpp       |   3 -
 app/service-cli.cpp                 |   2 -
 src/translator/CMakeLists.txt       |   8 +-
 src/translator/TranslationModel.cpp |  47 +----------
 src/translator/TranslationModel.h   |  10 ++-
 src/translator/batch_translator.h   |   2 +-
 src/translator/batcher.h            |   2 +-
 src/translator/parser.h             |  38 +++++++++
 src/translator/service.cpp          | 119 +++++++++++++++++++++-----
 src/translator/service.h            | 126 +++++++++++++++++++++-------
 src/translator/service_base.cpp     |  42 ----------
 src/translator/service_base.h       |  87 -------------------
 13 files changed, 245 insertions(+), 242 deletions(-)
 delete mode 100644 src/translator/service_base.cpp
 delete mode 100644 src/translator/service_base.h

diff --git a/app/marian-decoder-new.cpp b/app/marian-decoder-new.cpp
index f8079096d..dea49eb8b 100644
--- a/app/marian-decoder-new.cpp
+++ b/app/marian-decoder-new.cpp
@@ -56,6 +56,5 @@ int main(int argc, char *argv[]) {
   marian_decoder_minimal(response.histories(), service.targetVocab(), options);
 
   LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
-  service.stop();
   return 0;
 }
diff --git a/app/service-cli-bytearray.cpp b/app/service-cli-bytearray.cpp
index 42a2af3fb..d96781037 100644
--- a/app/service-cli-bytearray.cpp
+++ b/app/service-cli-bytearray.cpp
@@ -30,9 +30,6 @@ int main(int argc, char *argv[]) {
   Response response = responseFuture.get();
   std::cout << response.translation() << std::endl;
 
-  // Stop Service.
-  service.stop();
-
   // Clear the memory used for the byte array
   free(model_bytes); // Ideally, this should be done after the translation model has been gracefully shut down.
 
diff --git a/app/service-cli.cpp b/app/service-cli.cpp
index a8382b28b..2bb825c26 100644
--- a/app/service-cli.cpp
+++ b/app/service-cli.cpp
@@ -27,7 +27,5 @@ int main(int argc, char *argv[]) {
   Response response = responseFuture.get();
   std::cout << response.translation() << std::endl;
 
-  // Stop Service.
-  service.stop();
   return 0;
 }
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index d149c0392..ae38f9bb4 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,24 +1,18 @@
-if (NOT USE_WASM_COMPATIBLE_SOURCES)
-    set(MULTITHREADED_SERVICE_SOURCE "service.cpp")
-endif()
-
 add_library(bergamot-translator STATIC
     AbstractTranslationModel.cpp
     TranslationModel.cpp
 
-    # Following files added from browsermt/mts@nuke
     byteArrayExample.cpp
     text_processor.cpp
     sentence_splitter.cpp
     batch_translator.cpp 
     multifactor_priority.cpp 
     request.cpp 
-    service_base.cpp
-    ${MULTITHREADED_SERVICE_SOURCE}
     batcher.cpp
     response.cpp
     batch.cpp
     sentence_ranges.cpp
+    service.cpp
 )
 if (COMPILE_DECODER_ONLY)
   # A dirty hack because of marian's bad cmake practices
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 94f53a80f..715df4727 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -6,53 +6,14 @@
 #include <future>
 #include <vector>
 
-// All 3rd party includes
-#include "3rd_party/marian-dev/src/3rd_party/yaml-cpp/yaml.h"
-#include "3rd_party/marian-dev/src/common/config_parser.h"
-#include "common/config_validator.h"
-#include "common/options.h"
-
 // All local project includes
 #include "TranslationModel.h"
 #include "translator/parser.h"
-#include "translator/service_base.h"
-
-std::shared_ptr<marian::Options> parseOptions(const std::string &config) {
-  marian::Options options;
-
-  // @TODO(jerinphilip) There's something off here, @XapaJIaMnu suggests
-  // that should not be using the defaultConfig. This function only has access
-  // to std::string config and needs to be able to construct Options from the
-  // same.
-
-  // Absent the following code-segment, there is a parsing exception thrown on
-  // rebuilding YAML.
-  //
-  // Error: Unhandled exception of type 'N4YAML11InvalidNodeE': invalid node;
-  // this may result from using a map iterator as a sequence iterator, or
-  // vice-versa
-  //
-  // Error: Aborted from void unhandledException() in
-  // 3rd_party/marian-dev/src/common/logging.cpp:113
-
-  marian::ConfigParser configParser = marian::bergamot::createConfigParser();
-  const YAML::Node &defaultConfig = configParser.getConfig();
-
-  options.merge(defaultConfig);
-
-  // Parse configs onto defaultConfig.
-  options.parse(config);
-  YAML::Node configCopy = options.cloneToYamlNode();
-
-  marian::ConfigValidator validator(configCopy);
-  validator.validateOptions(marian::cli::mode::translation);
-
-  return std::make_shared<marian::Options>(options);
-}
+#include "translator/service.h"
 
-TranslationModel::TranslationModel(const std::string &config, const void * model_memory)
-    : configOptions_(std::move(parseOptions(config))),
-      AbstractTranslationModel(), service_(configOptions_, model_memory) {}
+TranslationModel::TranslationModel(const std::string &config,
+                                   const void *model_memory)
+    : AbstractTranslationModel(), service_(config, model_memory) {}
 
 TranslationModel::~TranslationModel() {}
 
diff --git a/src/translator/TranslationModel.h b/src/translator/TranslationModel.h
index 92fca6336..224d6b0a1 100644
--- a/src/translator/TranslationModel.h
+++ b/src/translator/TranslationModel.h
@@ -16,7 +16,7 @@
 
 // All local project includes
 #include "AbstractTranslationModel.h"
-#include "translator/service_base.h"
+#include "translator/service.h"
 
 /* A Translation model that translates a plain (without any markups and emojis)
  * UTF-8 encoded text. This implementation supports translation from 1 source
@@ -29,9 +29,11 @@ class TranslationModel : public AbstractTranslationModel {
    */
   /**
    * @param config Marian yml config file in the form of a string
-   * @param model_memory optional byte array (aligned to 64!!!) that contains the bytes of a model.bin.
+   * @param model_memory optional byte array (aligned to 64!!!) that contains
+   * the bytes of a model.bin.
    */
-  TranslationModel(const std::string &config, const void * model_memory = nullptr);
+  TranslationModel(const std::string &config,
+                   const void *model_memory = nullptr);
 
   ~TranslationModel();
 
@@ -69,7 +71,7 @@ class TranslationModel : public AbstractTranslationModel {
 private:
   // Model configuration options
   std::shared_ptr<marian::Options> configOptions_; // ORDER DEPENDECNY
-  marian::bergamot::NonThreadedService service_;   // ORDER DEPENDENCY
+  marian::bergamot::Service service_;              // ORDER DEPENDENCY
 };
 
 #endif /* SRC_TRANSLATOR_TRANSLATIONMODEL_H_ */
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 683c27922..c920945fa 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -12,7 +12,7 @@
 #include "translator/history.h"
 #include "translator/scorers.h"
 
-#ifdef WITH_PTHREADS
+#ifndef WASM
 #include "pcqueue.h"
 #endif
 
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index 81ed2d270..675cd5944 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -7,7 +7,7 @@
 #include "definitions.h"
 #include "request.h"
 
-#ifdef WITH_PTHREADS
+#ifndef WASM
 #include "pcqueue.h"
 #endif
 
diff --git a/src/translator/parser.h b/src/translator/parser.h
index 7fb53e1e1..4d93e3aa9 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -1,6 +1,10 @@
 #ifndef SRC_BERGAMOT_PARSER_H
 #define SRC_BERGAMOT_PARSER_H
 
+#include "3rd_party/yaml-cpp/yaml.h"
+#include "common/config_parser.h"
+#include "common/config_validator.h"
+#include "common/options.h"
 #include "marian.h"
 
 namespace marian {
@@ -22,6 +26,40 @@ inline marian::ConfigParser createConfigParser() {
   return cp;
 }
 
+inline std::shared_ptr<marian::Options>
+parseOptions(const std::string &config) {
+  marian::Options options;
+
+  // @TODO(jerinphilip) There's something off here, @XapaJIaMnu suggests
+  // that should not be using the defaultConfig. This function only has access
+  // to std::string config and needs to be able to construct Options from the
+  // same.
+
+  // Absent the following code-segment, there is a parsing exception thrown on
+  // rebuilding YAML.
+  //
+  // Error: Unhandled exception of type 'N4YAML11InvalidNodeE': invalid node;
+  // this may result from using a map iterator as a sequence iterator, or
+  // vice-versa
+  //
+  // Error: Aborted from void unhandledException() in
+  // 3rd_party/marian-dev/src/common/logging.cpp:113
+
+  marian::ConfigParser configParser = createConfigParser();
+  const YAML::Node &defaultConfig = configParser.getConfig();
+
+  options.merge(defaultConfig);
+
+  // Parse configs onto defaultConfig.
+  options.parse(config);
+  YAML::Node configCopy = options.cloneToYamlNode();
+
+  marian::ConfigValidator validator(configCopy);
+  validator.validateOptions(marian::cli::mode::translation);
+
+  return std::make_shared<marian::Options>(options);
+}
+
 } //  namespace bergamot
 } //  namespace marian
 
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 86e8c59eb..8c8c45312 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -5,25 +5,77 @@
 #include <string>
 #include <utility>
 
+inline std::vector<marian::Ptr<const marian::Vocab>>
+loadVocabularies(marian::Ptr<marian::Options> options) {
+  // @TODO: parallelize vocab loading for faster startup
+  auto vfiles = options->get<std::vector<std::string>>("vocabs");
+  // with the current setup, we need at least two vocabs: src and trg
+  ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
+  std::vector<marian::Ptr<marian::Vocab const>> vocabs(vfiles.size());
+  std::unordered_map<std::string, marian::Ptr<marian::Vocab>> vmap;
+  for (size_t i = 0; i < vocabs.size(); ++i) {
+    auto m =
+        vmap.emplace(std::make_pair(vfiles[i], marian::Ptr<marian::Vocab>()));
+    if (m.second) { // new: load the vocab
+      m.first->second = marian::New<marian::Vocab>(options, i);
+      m.first->second->load(vfiles[i]);
+    }
+    vocabs[i] = m.first->second;
+  }
+  return vocabs;
+}
+
 namespace marian {
 namespace bergamot {
 
-Service::Service(Ptr<Options> options, const void * model_memory)
-    : ServiceBase(options), numWorkers_(options->get<int>("cpu-threads")),
-      pcqueue_(numWorkers_), model_memory_{model_memory} {
+Service::Service(Ptr<Options> options, const void *model_memory)
+    : requestId_(0), vocabs_(std::move(loadVocabularies(options))),
+      text_processor_(vocabs_, options), batcher_(options),
+      numWorkers_(options->get<int>("cpu-threads")), model_memory_(model_memory)
+#ifndef WASM
+      // 0 elements in PCQueue is illegal and can lead to failures. Adding a
+      // guard to have at least one entry allocated. In the single-threaded
+      // case, while initialized pcqueue_ remains unused.
+      ,
+      pcqueue_(std::max<size_t>(1, numWorkers_))
+#endif
+{
+
   if (numWorkers_ == 0) {
-    ABORT("Fatal: Attempt to create multithreaded instance with --cpu-threads "
-          "0. ");
+    build_translators(options, /*numTranslators=*/1);
+    initialize_blocking_translator();
+  } else {
+    build_translators(options, numWorkers_);
+    initialize_async_translators();
   }
+}
 
-  translators_.reserve(numWorkers_);
-  workers_.reserve(numWorkers_);
-
-  for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
+void Service::build_translators(Ptr<Options> options, size_t numTranslators) {
+  translators_.reserve(numTranslators);
+  for (size_t cpuId = 0; cpuId < numTranslators; cpuId++) {
     marian::DeviceId deviceId(cpuId, DeviceType::cpu);
     translators_.emplace_back(deviceId, vocabs_, options, model_memory_);
+  }
+}
+
+void Service::initialize_blocking_translator() {
+  translators_.back().initialize();
+}
+
+void Service::blocking_translate() {
+  Batch batch;
+  while (batcher_ >> batch) {
     auto &translator = translators_.back();
+    translator.translate(batch);
+  }
+}
+
+#ifndef WASM
+void Service::initialize_async_translators() {
+  workers_.reserve(numWorkers_);
 
+  for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
+    auto &translator = translators_[cpuId];
     workers_.emplace_back([&translator, this] {
       translator.initialize();
 
@@ -42,29 +94,58 @@ Service::Service(Ptr<Options> options, const void * model_memory)
   }
 }
 
-void Service::enqueue() {
+void Service::async_translate() {
   Batch batch;
   while (batcher_ >> batch) {
     pcqueue_.ProduceSwap(batch);
   }
 }
+#else  // WASM
+void Service::initialize_async_translators() {
+  ABORT("Cannot run in async mode without multithreading.");
+}
+
+void Service::async_translate() {
+  ABORT("Cannot run in async mode without multithreading.");
+}
+#endif // WASM
+
+std::future<Response> Service::translate(std::string &&input) {
+  Segments segments;
+  SentenceRanges sourceRanges;
+  text_processor_.process(input, segments, sourceRanges);
+
+  std::promise<Response> responsePromise;
+  auto future = responsePromise.get_future();
+
+  Ptr<Request> request = New<Request>(
+      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(input),
+      std::move(segments), std::move(sourceRanges), std::move(responsePromise));
+
+  batcher_.addWholeRequest(request);
+  if (numWorkers_ == 0) {
+    blocking_translate();
+  } else {
+    async_translate();
+  }
+  return future;
+}
+
+Service::~Service() {
+#ifndef WASM
+  for (size_t workerId = 0; workerId < numWorkers_; workerId++) {
 
-void Service::stop() {
-  for (auto &worker : workers_) {
     Batch poison = Batch::poison();
     pcqueue_.ProduceSwap(poison);
   }
 
-  for (auto &worker : workers_) {
-    if (worker.joinable()) {
-      worker.join();
+  for (size_t workerId = 0; workerId < numWorkers_; workerId++) {
+    if (workers_[workerId].joinable()) {
+      workers_[workerId].join();
     }
   }
-
-  workers_.clear();
+#endif
 }
 
-Service::~Service() { stop(); }
-
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/service.h b/src/translator/service.h
index 38de3ae99..bb8dbe929 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -4,10 +4,13 @@
 #include "batch_translator.h"
 #include "batcher.h"
 #include "data/types.h"
-#include "pcqueue.h"
 #include "response.h"
-#include "service_base.h"
 #include "text_processor.h"
+#include "translator/parser.h"
+
+#ifndef WASM
+#include "pcqueue.h"
+#endif
 
 #include <queue>
 #include <vector>
@@ -15,44 +18,103 @@
 namespace marian {
 namespace bergamot {
 
-class Service : public ServiceBase {
-
-  // Service exposes methods to translate an incoming blob of text to the
-  // Consumer of bergamot API.
-  //
-  // An example use of this API looks as follows:
-  //
-  //  options = ...;
-  //  service = Service(options);
-  //  std::string input_blob = "Hello World";
-  //  std::future<Response>
-  //      response = service.translate(std::move(input_blob));
-  //  response.wait();
-  //  Response result = response.get();
+/// Service exposes methods to translate an incoming blob of text to the
+/// Consumer of bergamot API.
+///
+/// An example use of this API looks as follows:
+///
+///  options = ...;
+///  service = Service(options);
+///  std::string input_blob = "Hello World";
+///  std::future<Response>
+///      response = service.translate(std::move(input_blob));
+///  response.wait();
+///  Response result = response.get();
+///
+/// Optionally Service can be initialized by also passing model_memory for
+/// purposes of efficiency (which defaults to nullpointer and then reads from
+/// file supplied through config).
+class Service {
 
 public:
-  /**
-   * @param options Marian options object
-   * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Optional, defaults to nullptr when not used
-   */
-  explicit Service(Ptr<Options> options, const void * model_memory=nullptr);
-  // Implements enqueue and top through blocking methods.
-  void stop() override;
+  /// @param options Marian options object
+  /// @param model_memory byte array (aligned to 64!!!) that contains the bytes
+  /// of a model.bin. Optional, defaults to nullptr when not used
+  explicit Service(Ptr<Options> options, const void *model_memory = nullptr);
+
+  /// Construct Service from a string configuration.
+  /// @param [in] config string parsable as YAML expected to adhere with marian
+  /// config
+  /// @param [in] model_memory byte array (aligned to 64!!!) that contains the
+  /// bytes of a model.bin. Optional, defaults to nullptr when not used
+  explicit Service(const std::string &config,
+                   const void *model_memory = nullptr)
+      : Service(parseOptions(config), model_memory) {}
+
+  /// Explicit destructor to clean up after any threads initialized in
+  /// asynchronous operation mode.
   ~Service();
 
+  /// Shared pointer to source-vocabulary.
+  Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); }
+
+  /// Shared pointer to target vocabulary.
+  Ptr<Vocab const> targetVocab() const { return vocabs_.back(); }
+
+  /// To stay efficient and to refer to the string for alignments, expects
+  /// ownership be moved through std::move(..)
+  ///
+  ///  @param [in] rvalue reference of string to be translated.
+  std::future<Response> translate(std::string &&input);
+
 private:
-  void enqueue() override;
+  /// Build numTranslators number of translators with options from options
+  void build_translators(Ptr<Options> options, size_t numTranslators);
+  /// Initializes a blocking translator without using std::thread
+  void initialize_blocking_translator();
+  /// Translates through direct interaction between batcher_ and translators_
+  void blocking_translate();
 
-  // In addition to the common members (text_processor, requestId, vocabs_,
-  // batcher) extends with a producer-consumer queue, vector of translator
-  // instances owned by service each listening to the pcqueue in separate
-  // threads.
+  /// Launches multiple workers of translators using std::thread
+  /// Reduces to ABORT if called when not compiled WITH_PTHREAD
+  void initialize_async_translators();
+  /// Async translate produces to a producer-consumer queue as batches are
+  /// generated by Batcher. In another thread, the translators consume from
+  /// producer-consumer queue.
+  /// Reduces to ABORT if called when not compiled WITH_PTHREAD
+  void async_translate();
 
-  size_t numWorkers_;      // ORDER DEPENDENCY
-  PCQueue<Batch> pcqueue_; // ORDER DEPENDENCY
-  const void * model_memory_;
-  std::vector<std::thread> workers_;
+  /// Number of workers to launch.
+  size_t numWorkers_;        // ORDER DEPENDENCY (pcqueue_)
+  const void *model_memory_; /// Model memory to load model passed as bytes.
+
+  /// Holds instances of batch translators, just one in case
+  /// of single-threaded application, numWorkers_ in case of multithreaded
+  /// setting.
   std::vector<BatchTranslator> translators_;
+
+  /// Stores requestId of active request. Used to establish
+  /// ordering among requests and logging/book-keeping.
+
+  size_t requestId_;
+
+  /// Store vocabs representing source and target.
+  std::vector<Ptr<Vocab const>> vocabs_; // ORDER DEPENDENCY (text_processor_)
+
+  /// TextProcesser takes a blob of text and converts into format consumable by
+  /// the batch-translator and annotates sentences and words.
+  TextProcessor text_processor_; // ORDER DEPENDENCY (vocabs_)
+
+  /// Batcher handles generation of batches from a request, subject to
+  /// packing-efficiency and priority optimization heuristics.
+  Batcher batcher_;
+
+  // The following constructs are available providing full capabilities on a non
+  // WASM platform, where one does not have to hide threads.
+#ifndef WASM
+  PCQueue<Batch> pcqueue_; // ORDER DEPENDENCY (numWorkers_)
+  std::vector<std::thread> workers_;
+#endif // WASM
 };
 
 } // namespace bergamot
diff --git a/src/translator/service_base.cpp b/src/translator/service_base.cpp
deleted file mode 100644
index dc0c05aaf..000000000
--- a/src/translator/service_base.cpp
+++ /dev/null
@@ -1,42 +0,0 @@
-#include "service_base.h"
-
-namespace marian {
-namespace bergamot {
-
-ServiceBase::ServiceBase(Ptr<Options> options)
-    : requestId_(0), vocabs_(std::move(loadVocabularies(options))),
-      text_processor_(vocabs_, options), batcher_(options) {}
-
-std::future<Response> ServiceBase::translate(std::string &&input) {
-  Segments segments;
-  SentenceRanges sourceRanges;
-  text_processor_.process(input, segments, sourceRanges);
-
-  std::promise<Response> responsePromise;
-  auto future = responsePromise.get_future();
-
-  Ptr<Request> request = New<Request>(
-      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(input),
-      std::move(segments), std::move(sourceRanges), std::move(responsePromise));
-
-  batcher_.addWholeRequest(request);
-  enqueue();
-  return future;
-}
-
-NonThreadedService::NonThreadedService(Ptr<Options> options, const void * model_memory)
-    : ServiceBase(options),
-      translator_(DeviceId(0, DeviceType::cpu), vocabs_, options, model_memory) {
-  translator_.initialize();
-}
-
-void NonThreadedService::enqueue() {
-  // Queue single-threaded
-  Batch batch;
-  while (batcher_ >> batch) {
-    translator_.translate(batch);
-  }
-}
-
-} // namespace bergamot
-} // namespace marian
diff --git a/src/translator/service_base.h b/src/translator/service_base.h
deleted file mode 100644
index a44273fae..000000000
--- a/src/translator/service_base.h
+++ /dev/null
@@ -1,87 +0,0 @@
-#ifndef SRC_BERGAMOT_SERVICE_BASE_H_
-#define SRC_BERGAMOT_SERVICE_BASE_H_
-#include "batch_translator.h"
-#include "batcher.h"
-#include "data/types.h"
-#include "response.h"
-#include "text_processor.h"
-
-#include <queue>
-#include <vector>
-
-namespace marian {
-namespace bergamot {
-// This file describes the base class ServiceBase, and a non-threaded subclass
-// implementing translation functionality called NonThreadedService.
-
-class ServiceBase {
-public:
-  /**
-   * @param options Marian options object
-   */
-  explicit ServiceBase(Ptr<Options> options);
-
-  // Transfers ownership of input string to Service, returns a future containing
-  // an object which provides access to translations, other features like
-  // sentencemappings and (tentatively) alignments.
-  std::future<Response> translate(std::string &&input);
-
-  // Convenience accessor methods to extract these vocabulary outside service.
-  // e.g: For use in decoding histories for marian-decoder replacement.
-  Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); }
-  Ptr<Vocab const> targetVocab() const { return vocabs_.back(); }
-
-  // Wraps up any thread related destruction code.
-  virtual void stop() = 0;
-
-protected:
-  // Enqueue queues a request for translation, this can be synchronous, blocking
-  // or asynchronous and queued in the background.
-  virtual void enqueue() = 0;
-
-  size_t requestId_;
-  std::vector<Ptr<Vocab const>> vocabs_; // ORDER DEPENDENCY
-  TextProcessor text_processor_;         // ORDER DEPENDENCY
-  Batcher batcher_;
-};
-
-class NonThreadedService : public ServiceBase {
-public:
-  /**
-   * @param options Marian options object
-   * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Provide a nullptr if not used.
-   */
-  explicit NonThreadedService(Ptr<Options> options, const void * model_memory);
-  void stop() override{};
-
-private:
-  // NonThreaded service overrides unimplemented functions in base-class using
-  // blocking mechanisms.
-  void enqueue() override;
-  // There's a single translator, launched as part of the main process.
-  BatchTranslator translator_;
-};
-
-// Used across Services
-inline std::vector<Ptr<const Vocab>> loadVocabularies(Ptr<Options> options) {
-  // @TODO: parallelize vocab loading for faster startup
-  auto vfiles = options->get<std::vector<std::string>>("vocabs");
-  // with the current setup, we need at least two vocabs: src and trg
-  ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
-  std::vector<Ptr<Vocab const>> vocabs(vfiles.size());
-  std::unordered_map<std::string, Ptr<Vocab>> vmap;
-  for (size_t i = 0; i < vocabs.size(); ++i) {
-    auto m = vmap.emplace(std::make_pair(vfiles[i], Ptr<Vocab>()));
-    if (m.second) { // new: load the vocab
-      m.first->second = New<Vocab>(options, i);
-      m.first->second->load(vfiles[i]);
-    }
-    vocabs[i] = m.first->second;
-  }
-  return vocabs;
-}
-
-} // namespace bergamot
-} // namespace marian
-
-#endif // SRC_BERGAMOT_SERVICE_BASE_H_

From 12e9232066d074f65cdb8642a4eac96ee3978ead Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 24 Mar 2021 17:10:42 +0100
Subject: [PATCH 177/442] Patch WASM artifacts to run optimized (wormhole
 enabled) inference (#68)

* A script to patch the wasm artifacts to use wormhole via
   APIs that instantiate WASM module
* Updated README
* Load just production ready models
* Shallow clone bergamot-models repo since it has such a large history
* Improved wasm test_page
 - test page can load all 5 language pairs
 - Use intgemm.alpha* models
* Refactor the code that patches wasm artifacts to enable wormhole

Co-authored-by: Andre Natal <anatal@gmail.com>
Co-authored-by: Motin <motin@motin.eu>
---
 .github/workflows/macos-custom-marian-wasm.yml |  4 ++++
 README.md                                      | 15 ++++++++++-----
 wasm/patch-artifacts-enable-wormhole.sh        |  7 +++++++
 wasm/test_page/bergamot.html                   |  5 ++++-
 wasm/test_page/start_server.sh                 |  4 +++-
 5 files changed, 28 insertions(+), 7 deletions(-)
 create mode 100644 wasm/patch-artifacts-enable-wormhole.sh

diff --git a/.github/workflows/macos-custom-marian-wasm.yml b/.github/workflows/macos-custom-marian-wasm.yml
index 3e49369b4..00f5cf7af 100644
--- a/.github/workflows/macos-custom-marian-wasm.yml
+++ b/.github/workflows/macos-custom-marian-wasm.yml
@@ -33,6 +33,10 @@ jobs:
         working-directory: build-wasm
         run: emmake make -j2
 
+      - name: Instantiate simd wormhole
+        working-directory: build-wasm
+        run: bash ../wasm/patch-artifacts-enable-wormhole.sh
+
       - name: Check artifacts
         working-directory: build-wasm
         run: |
diff --git a/README.md b/README.md
index 2b7f0a6c7..e70e8cd09 100644
--- a/README.md
+++ b/README.md
@@ -46,8 +46,8 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
     If you want to package bergamot project specific models, please follow these instructions:
     ```bash
     mkdir models
-    git clone https://github.com/mozilla-applied-ml/bergamot-models
-    cp -rf bergamot-models/* models
+    git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
+    cp -rf bergamot-models/prod/* models
     gunzip models/*/*
     ```
 
@@ -75,11 +75,16 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
             emmake make -j
             ```
 
-    The artefacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
+        The wasm artifacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
+
+    3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
+        ```bash
+        bash ../wasm/patch-artifacts-enable-wormhole.sh
+        ```
 
 #### Recompiling
-As long as you don't update any submodule, just follow steps in `4.ii` to recompile.\
-If you update a submodule, execute following command before executing steps in `4.ii` to recompile.
+As long as you don't update any submodule, just follow steps in `4.ii` and `4.iii` to recompile.\
+If you update a submodule, execute following command before executing steps in `4.ii` and `4.iii` to recompile.
 ```bash
 git submodule update --init --recursive
 ```
diff --git a/wasm/patch-artifacts-enable-wormhole.sh b/wasm/patch-artifacts-enable-wormhole.sh
new file mode 100644
index 000000000..c16ba66b8
--- /dev/null
+++ b/wasm/patch-artifacts-enable-wormhole.sh
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+echo "Patching wasm artifacts to enable wormhole via APIs that compile and instantiate wasm module"
+sed -i.bak 's/var result = WebAssembly.instantiateStreaming(response, info);/var result = WebAssembly.instantiateStreaming(response, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+sed -i.bak 's/return WebAssembly.instantiate(binary, info);/return WebAssembly.instantiate(binary, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+sed -i.bak 's/var module = new WebAssembly.Module(bytes);/var module = new WebAssembly.Module(bytes, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+echo "Done"
diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index b4d2027b0..4f1f2a0f7 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -32,6 +32,9 @@
     <label>Choose the model to use</label>
     <input type="radio" name="modellang" value="enes"/><label>English to Spanish</label>
     <input type="radio" name="modellang" value="esen" checked/><label>Spanish to English</label>
+    <input type="radio" name="modellang" value="eten" checked/><label>Estonian to English</label>
+    <input type="radio" name="modellang" value="enet" checked/><label>English to Estonian</label>
+    <input type="radio" name="modellang" value="ende" checked/><label>English to German</label>
     <input type="button" id="load" value="Load Model"/>
 </div>
 
@@ -70,7 +73,7 @@
     // Set the Model Configuration as YAML formatted string.
     // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
     const modelConfig = `models:
-  - /${languagePair}/model.${languagePair}.npz
+  - /${languagePair}/model.${languagePair}.intgemm.alphas.bin
 vocabs:
   - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
   - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
diff --git a/wasm/test_page/start_server.sh b/wasm/test_page/start_server.sh
index b83344b8a..b6fc2a6eb 100644
--- a/wasm/test_page/start_server.sh
+++ b/wasm/test_page/start_server.sh
@@ -1,8 +1,10 @@
 #!/bin/bash
-
+echo "Start: Copying artifacts in local folder------"
 cp ../../build-wasm/wasm/bergamot-translator-worker.data .
 cp ../../build-wasm/wasm/bergamot-translator-worker.js .
 cp ../../build-wasm/wasm/bergamot-translator-worker.wasm .
 cp ../../build-wasm/wasm/bergamot-translator-worker.worker.js .
+
 npm install
+echo "Start httpserver"
 node bergamot-httpserver.js
\ No newline at end of file

From a3250b401f6bfac49ec941b46ddb49c792f7ddd2 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 24 Mar 2021 17:00:53 +0000
Subject: [PATCH 178/442] Marian compatible documentation tooling (#67)

Adds doxygen configurations, additional sphinx which consumes the doxygen files to generate developer API, compatible with marian-nmt/marian-dev.
---
 BERGAMOT_VERSION           |    1 +
 Doxyfile.in                | 2494 ++++++++++++++++++++++++++++++++++++
 doc/.gitignore             |    4 +
 doc/README.md              |   51 +
 doc/_static/css/custom.css |    4 +
 doc/conf.py                |  120 ++
 doc/index.rst              |   38 +
 doc/make.bat               |   35 +
 doc/marian-integration.md  |    2 +-
 doc/references.bib         |    0
 doc/requirements.txt       |    6 +
 11 files changed, 2754 insertions(+), 1 deletion(-)
 create mode 100644 BERGAMOT_VERSION
 create mode 100644 Doxyfile.in
 create mode 100644 doc/.gitignore
 create mode 100644 doc/README.md
 create mode 100644 doc/_static/css/custom.css
 create mode 100644 doc/conf.py
 create mode 100644 doc/index.rst
 create mode 100644 doc/make.bat
 create mode 100644 doc/references.bib
 create mode 100644 doc/requirements.txt

diff --git a/BERGAMOT_VERSION b/BERGAMOT_VERSION
new file mode 100644
index 000000000..ae39fab35
--- /dev/null
+++ b/BERGAMOT_VERSION
@@ -0,0 +1 @@
+v0.0.0
diff --git a/Doxyfile.in b/Doxyfile.in
new file mode 100644
index 000000000..88948e2ad
--- /dev/null
+++ b/Doxyfile.in
@@ -0,0 +1,2494 @@
+# Doxyfile 1.8.13
+
+# This file describes the settings to be used by the documentation system
+# doxygen (www.doxygen.org) for a project.
+#
+# All text after a double hash (##) is considered a comment and is placed in
+# front of the TAG it is preceding.
+#
+# All text after a single hash (#) is considered a comment and will be ignored.
+# The format is:
+# TAG = value [value, ...]
+# For lists, items can also be appended using:
+# TAG += value [value, ...]
+# Values that contain spaces should be placed between quotes (\" \").
+
+#---------------------------------------------------------------------------
+# Project related configuration options
+#---------------------------------------------------------------------------
+
+# This tag specifies the encoding used for all characters in the config file
+# that follow. The default is UTF-8 which is also the encoding used for all text
+# before the first occurrence of this tag. Doxygen uses libiconv (or the iconv
+# built into libc) for the transcoding. See http://www.gnu.org/software/libiconv
+# for the list of possible encodings.
+# The default value is: UTF-8.
+
+DOXYFILE_ENCODING      = UTF-8
+
+# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by
+# double-quotes, unless you are using Doxywizard) that should identify the
+# project for which the documentation is generated. This name is used in the
+# title of most generated pages and in a few other places.
+# The default value is: My Project.
+
+PROJECT_NAME           = "Bergamot Translator"
+
+# The PROJECT_NUMBER tag can be used to enter a project or revision number. This
+# could be handy for archiving the generated documentation or if some version
+# control system is used.
+
+PROJECT_NUMBER         =
+
+# Using the PROJECT_BRIEF tag one can provide an optional one line description
+# for a project that appears at the top of each page and should give viewer a
+# quick idea about the purpose of the project. Keep the description short.
+
+PROJECT_BRIEF          =
+
+# With the PROJECT_LOGO tag one can specify a logo or an icon that is included
+# in the documentation. The maximum height of the logo should not exceed 55
+# pixels and the maximum width should not exceed 200 pixels. Doxygen will copy
+# the logo to the output directory.
+
+PROJECT_LOGO           =
+
+# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path
+# into which the generated documentation will be written. If a relative path is
+# entered, it will be relative to the location where doxygen was started. If
+# left blank the current directory will be used.
+
+OUTPUT_DIRECTORY       = build/doc
+
+# If the CREATE_SUBDIRS tag is set to YES then doxygen will create 4096 sub-
+# directories (in 2 levels) under the output directory of each output format and
+# will distribute the generated files over these directories. Enabling this
+# option can be useful when feeding doxygen a huge amount of source files, where
+# putting all generated files in the same directory would otherwise causes
+# performance problems for the file system.
+# The default value is: NO.
+
+CREATE_SUBDIRS         = NO
+
+# If the ALLOW_UNICODE_NAMES tag is set to YES, doxygen will allow non-ASCII
+# characters to appear in the names of generated files. If set to NO, non-ASCII
+# characters will be escaped, for example _xE3_x81_x84 will be used for Unicode
+# U+3044.
+# The default value is: NO.
+
+ALLOW_UNICODE_NAMES    = NO
+
+# The OUTPUT_LANGUAGE tag is used to specify the language in which all
+# documentation generated by doxygen is written. Doxygen will use this
+# information to generate all constant output in the proper language.
+# Possible values are: Afrikaans, Arabic, Armenian, Brazilian, Catalan, Chinese,
+# Chinese-Traditional, Croatian, Czech, Danish, Dutch, English (United States),
+# Esperanto, Farsi (Persian), Finnish, French, German, Greek, Hungarian,
+# Indonesian, Italian, Japanese, Japanese-en (Japanese with English messages),
+# Korean, Korean-en (Korean with English messages), Latvian, Lithuanian,
+# Macedonian, Norwegian, Persian (Farsi), Polish, Portuguese, Romanian, Russian,
+# Serbian, Serbian-Cyrillic, Slovak, Slovene, Spanish, Swedish, Turkish,
+# Ukrainian and Vietnamese.
+# The default value is: English.
+
+OUTPUT_LANGUAGE        = English
+
+# If the BRIEF_MEMBER_DESC tag is set to YES, doxygen will include brief member
+# descriptions after the members that are listed in the file and class
+# documentation (similar to Javadoc). Set to NO to disable this.
+# The default value is: YES.
+
+BRIEF_MEMBER_DESC      = YES
+
+# If the REPEAT_BRIEF tag is set to YES, doxygen will prepend the brief
+# description of a member or function before the detailed description
+#
+# Note: If both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the
+# brief descriptions will be completely suppressed.
+# The default value is: YES.
+
+REPEAT_BRIEF           = YES
+
+# This tag implements a quasi-intelligent brief description abbreviator that is
+# used to form the text in various listings. Each string in this list, if found
+# as the leading text of the brief description, will be stripped from the text
+# and the result, after processing the whole list, is used as the annotated
+# text. Otherwise, the brief description is used as-is. If left blank, the
+# following values are used ($name is automatically replaced with the name of
+# the entity):The $name class, The $name widget, The $name file, is, provides,
+# specifies, contains, represents, a, an and the.
+
+ABBREVIATE_BRIEF       = "The $name class" \
+                         "The $name widget" \
+                         "The $name file" \
+                         is \
+                         provides \
+                         specifies \
+                         contains \
+                         represents \
+                         a \
+                         an \
+                         the
+
+# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then
+# doxygen will generate a detailed section even if there is only a brief
+# description.
+# The default value is: NO.
+
+ALWAYS_DETAILED_SEC    = NO
+
+# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all
+# inherited members of a class in the documentation of that class as if those
+# members were ordinary class members. Constructors, destructors and assignment
+# operators of the base classes will not be shown.
+# The default value is: NO.
+
+INLINE_INHERITED_MEMB  = NO
+
+# If the FULL_PATH_NAMES tag is set to YES, doxygen will prepend the full path
+# before files name in the file list and in the header files. If set to NO the
+# shortest path that makes the file name unique will be used
+# The default value is: YES.
+
+FULL_PATH_NAMES        = YES
+
+# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path.
+# Stripping is only done if one of the specified strings matches the left-hand
+# part of the path. The tag can be used to show relative paths in the file list.
+# If left blank the directory from which doxygen is run is used as the path to
+# strip.
+#
+# Note that you can specify absolute paths here, but also relative paths, which
+# will be relative from the directory where doxygen is started.
+# This tag requires that the tag FULL_PATH_NAMES is set to YES.
+
+STRIP_FROM_PATH        =
+
+# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of the
+# path mentioned in the documentation of a class, which tells the reader which
+# header file to include in order to use a class. If left blank only the name of
+# the header file containing the class definition is used. Otherwise one should
+# specify the list of include paths that are normally passed to the compiler
+# using the -I flag.
+
+STRIP_FROM_INC_PATH    =
+
+# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter (but
+# less readable) file names. This can be useful is your file systems doesn't
+# support long names like on DOS, Mac, or CD-ROM.
+# The default value is: NO.
+
+SHORT_NAMES            = NO
+
+# If the JAVADOC_AUTOBRIEF tag is set to YES then doxygen will interpret the
+# first line (until the first dot) of a Javadoc-style comment as the brief
+# description. If set to NO, the Javadoc-style will behave just like regular Qt-
+# style comments (thus requiring an explicit @brief command for a brief
+# description.)
+# The default value is: NO.
+
+JAVADOC_AUTOBRIEF      = NO
+
+# If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first
+# line (until the first dot) of a Qt-style comment as the brief description. If
+# set to NO, the Qt-style will behave just like regular Qt-style comments (thus
+# requiring an explicit \brief command for a brief description.)
+# The default value is: NO.
+
+QT_AUTOBRIEF           = NO
+
+# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make doxygen treat a
+# multi-line C++ special comment block (i.e. a block of //! or /// comments) as
+# a brief description. This used to be the default behavior. The new default is
+# to treat a multi-line C++ comment block as a detailed description. Set this
+# tag to YES if you prefer the old behavior instead.
+#
+# Note that setting this tag to YES also means that rational rose comments are
+# not recognized any more.
+# The default value is: NO.
+
+MULTILINE_CPP_IS_BRIEF = NO
+
+# If the INHERIT_DOCS tag is set to YES then an undocumented member inherits the
+# documentation from any documented member that it re-implements.
+# The default value is: YES.
+
+INHERIT_DOCS           = YES
+
+# If the SEPARATE_MEMBER_PAGES tag is set to YES then doxygen will produce a new
+# page for each member. If set to NO, the documentation of a member will be part
+# of the file/class/namespace that contains it.
+# The default value is: NO.
+
+SEPARATE_MEMBER_PAGES  = NO
+
+# The TAB_SIZE tag can be used to set the number of spaces in a tab. Doxygen
+# uses this value to replace tabs by spaces in code fragments.
+# Minimum value: 1, maximum value: 16, default value: 4.
+
+TAB_SIZE               = 2
+
+# This tag can be used to specify a number of aliases that act as commands in
+# the documentation. An alias has the form:
+# name=value
+# For example adding
+# "sideeffect=@par Side Effects:\n"
+# will allow you to put the command \sideeffect (or @sideeffect) in the
+# documentation, which will result in a user-defined paragraph with heading
+# "Side Effects:". You can put \n's in the value part of an alias to insert
+# newlines.
+
+ALIASES                =
+
+# This tag can be used to specify a number of word-keyword mappings (TCL only).
+# A mapping has the form "name=value". For example adding "class=itcl::class"
+# will allow you to use the command class in the itcl::class meaning.
+
+TCL_SUBST              =
+
+# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources
+# only. Doxygen will then generate output that is more tailored for C. For
+# instance, some of the names that are used will be different. The list of all
+# members will be omitted, etc.
+# The default value is: NO.
+
+OPTIMIZE_OUTPUT_FOR_C  = NO
+
+# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java or
+# Python sources only. Doxygen will then generate output that is more tailored
+# for that language. For instance, namespaces will be presented as packages,
+# qualified scopes will look different, etc.
+# The default value is: NO.
+
+OPTIMIZE_OUTPUT_JAVA   = NO
+
+# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran
+# sources. Doxygen will then generate output that is tailored for Fortran.
+# The default value is: NO.
+
+OPTIMIZE_FOR_FORTRAN   = NO
+
+# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL
+# sources. Doxygen will then generate output that is tailored for VHDL.
+# The default value is: NO.
+
+OPTIMIZE_OUTPUT_VHDL   = NO
+
+# Doxygen selects the parser to use depending on the extension of the files it
+# parses. With this tag you can assign which parser to use for a given
+# extension. Doxygen has a built-in mapping, but you can override or extend it
+# using this tag. The format is ext=language, where ext is a file extension, and
+# language is one of the parsers supported by doxygen: IDL, Java, Javascript,
+# C#, C, C++, D, PHP, Objective-C, Python, Fortran (fixed format Fortran:
+# FortranFixed, free formatted Fortran: FortranFree, unknown formatted Fortran:
+# Fortran. In the later case the parser tries to guess whether the code is fixed
+# or free formatted code, this is the default for Fortran type files), VHDL. For
+# instance to make doxygen treat .inc files as Fortran files (default is PHP),
+# and .f files as C (default is Fortran), use: inc=Fortran f=C.
+#
+# Note: For files without extension you can use no_extension as a placeholder.
+#
+# Note that for custom extensions you also need to set FILE_PATTERNS otherwise
+# the files are not read by doxygen.
+
+EXTENSION_MAPPING      =
+
+# If the MARKDOWN_SUPPORT tag is enabled then doxygen pre-processes all comments
+# according to the Markdown format, which allows for more readable
+# documentation. See http://daringfireball.net/projects/markdown/ for details.
+# The output of markdown processing is further processed by doxygen, so you can
+# mix doxygen, HTML, and XML commands with Markdown formatting. Disable only in
+# case of backward compatibilities issues.
+# The default value is: YES.
+
+MARKDOWN_SUPPORT       = YES
+
+# When the TOC_INCLUDE_HEADINGS tag is set to a non-zero value, all headings up
+# to that level are automatically included in the table of contents, even if
+# they do not have an id attribute.
+# Note: This feature currently applies only to Markdown headings.
+# Minimum value: 0, maximum value: 99, default value: 0.
+# This tag requires that the tag MARKDOWN_SUPPORT is set to YES.
+
+TOC_INCLUDE_HEADINGS   = 0
+
+# When enabled doxygen tries to link words that correspond to documented
+# classes, or namespaces to their corresponding documentation. Such a link can
+# be prevented in individual cases by putting a % sign in front of the word or
+# globally by setting AUTOLINK_SUPPORT to NO.
+# The default value is: YES.
+
+AUTOLINK_SUPPORT       = YES
+
+# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want
+# to include (a tag file for) the STL sources as input, then you should set this
+# tag to YES in order to let doxygen match functions declarations and
+# definitions whose arguments contain STL classes (e.g. func(std::string);
+# versus func(std::string) {}). This also make the inheritance and collaboration
+# diagrams that involve STL classes more complete and accurate.
+# The default value is: NO.
+
+BUILTIN_STL_SUPPORT    = NO
+
+# If you use Microsoft's C++/CLI language, you should set this option to YES to
+# enable parsing support.
+# The default value is: NO.
+
+CPP_CLI_SUPPORT        = NO
+
+# Set the SIP_SUPPORT tag to YES if your project consists of sip (see:
+# http://www.riverbankcomputing.co.uk/software/sip/intro) sources only. Doxygen
+# will parse them like normal C++ but will assume all classes use public instead
+# of private inheritance when no explicit protection keyword is present.
+# The default value is: NO.
+
+SIP_SUPPORT            = NO
+
+# For Microsoft's IDL there are propget and propput attributes to indicate
+# getter and setter methods for a property. Setting this option to YES will make
+# doxygen to replace the get and set methods by a property in the documentation.
+# This will only work if the methods are indeed getting or setting a simple
+# type. If this is not the case, or you want to show the methods anyway, you
+# should set this option to NO.
+# The default value is: YES.
+
+IDL_PROPERTY_SUPPORT   = YES
+
+# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC
+# tag is set to YES then doxygen will reuse the documentation of the first
+# member in the group (if any) for the other members of the group. By default
+# all members of a group must be documented explicitly.
+# The default value is: NO.
+
+DISTRIBUTE_GROUP_DOC   = NO
+
+# If one adds a struct or class to a group and this option is enabled, then also
+# any nested class or struct is added to the same group. By default this option
+# is disabled and one has to add nested compounds explicitly via \ingroup.
+# The default value is: NO.
+
+GROUP_NESTED_COMPOUNDS = NO
+
+# Set the SUBGROUPING tag to YES to allow class member groups of the same type
+# (for instance a group of public functions) to be put as a subgroup of that
+# type (e.g. under the Public Functions section). Set it to NO to prevent
+# subgrouping. Alternatively, this can be done per class using the
+# \nosubgrouping command.
+# The default value is: YES.
+
+SUBGROUPING            = YES
+
+# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions
+# are shown inside the group in which they are included (e.g. using \ingroup)
+# instead of on a separate page (for HTML and Man pages) or section (for LaTeX
+# and RTF).
+#
+# Note that this feature does not work in combination with
+# SEPARATE_MEMBER_PAGES.
+# The default value is: NO.
+
+INLINE_GROUPED_CLASSES = NO
+
+# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and unions
+# with only public data fields or simple typedef fields will be shown inline in
+# the documentation of the scope in which they are defined (i.e. file,
+# namespace, or group documentation), provided this scope is documented. If set
+# to NO, structs, classes, and unions are shown on a separate page (for HTML and
+# Man pages) or section (for LaTeX and RTF).
+# The default value is: NO.
+
+INLINE_SIMPLE_STRUCTS  = NO
+
+# When TYPEDEF_HIDES_STRUCT tag is enabled, a typedef of a struct, union, or
+# enum is documented as struct, union, or enum with the name of the typedef. So
+# typedef struct TypeS {} TypeT, will appear in the documentation as a struct
+# with name TypeT. When disabled the typedef will appear as a member of a file,
+# namespace, or class. And the struct will be named TypeS. This can typically be
+# useful for C code in case the coding convention dictates that all compound
+# types are typedef'ed and only the typedef is referenced, never the tag name.
+# The default value is: NO.
+
+TYPEDEF_HIDES_STRUCT   = NO
+
+# The size of the symbol lookup cache can be set using LOOKUP_CACHE_SIZE. This
+# cache is used to resolve symbols given their name and scope. Since this can be
+# an expensive process and often the same symbol appears multiple times in the
+# code, doxygen keeps a cache of pre-resolved symbols. If the cache is too small
+# doxygen will become slower. If the cache is too large, memory is wasted. The
+# cache size is given by this formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range
+# is 0..9, the default is 0, corresponding to a cache size of 2^16=65536
+# symbols. At the end of a run doxygen will report the cache usage and suggest
+# the optimal cache size from a speed point of view.
+# Minimum value: 0, maximum value: 9, default value: 0.
+
+LOOKUP_CACHE_SIZE      = 0
+
+#---------------------------------------------------------------------------
+# Build related configuration options
+#---------------------------------------------------------------------------
+
+# If the EXTRACT_ALL tag is set to YES, doxygen will assume all entities in
+# documentation are documented, even if no documentation was available. Private
+# class members and static file members will be hidden unless the
+# EXTRACT_PRIVATE respectively EXTRACT_STATIC tags are set to YES.
+# Note: This will also disable the warnings about undocumented members that are
+# normally produced when WARNINGS is set to YES.
+# The default value is: NO.
+
+EXTRACT_ALL            = NO
+
+# If the EXTRACT_PRIVATE tag is set to YES, all private members of a class will
+# be included in the documentation.
+# The default value is: NO.
+
+EXTRACT_PRIVATE        = YES
+
+# If the EXTRACT_PACKAGE tag is set to YES, all members with package or internal
+# scope will be included in the documentation.
+# The default value is: NO.
+
+EXTRACT_PACKAGE        = NO
+
+# If the EXTRACT_STATIC tag is set to YES, all static members of a file will be
+# included in the documentation.
+# The default value is: NO.
+
+EXTRACT_STATIC         = NO
+
+# If the EXTRACT_LOCAL_CLASSES tag is set to YES, classes (and structs) defined
+# locally in source files will be included in the documentation. If set to NO,
+# only classes defined in header files are included. Does not have any effect
+# for Java sources.
+# The default value is: YES.
+
+EXTRACT_LOCAL_CLASSES  = YES
+
+# This flag is only useful for Objective-C code. If set to YES, local methods,
+# which are defined in the implementation section but not in the interface are
+# included in the documentation. If set to NO, only methods in the interface are
+# included.
+# The default value is: NO.
+
+EXTRACT_LOCAL_METHODS  = NO
+
+# If this flag is set to YES, the members of anonymous namespaces will be
+# extracted and appear in the documentation as a namespace called
+# 'anonymous_namespace{file}', where file will be replaced with the base name of
+# the file that contains the anonymous namespace. By default anonymous namespace
+# are hidden.
+# The default value is: NO.
+
+EXTRACT_ANON_NSPACES   = NO
+
+# If the HIDE_UNDOC_MEMBERS tag is set to YES, doxygen will hide all
+# undocumented members inside documented classes or files. If set to NO these
+# members will be included in the various overviews, but no documentation
+# section is generated. This option has no effect if EXTRACT_ALL is enabled.
+# The default value is: NO.
+
+HIDE_UNDOC_MEMBERS     = NO
+
+# If the HIDE_UNDOC_CLASSES tag is set to YES, doxygen will hide all
+# undocumented classes that are normally visible in the class hierarchy. If set
+# to NO, these classes will be included in the various overviews. This option
+# has no effect if EXTRACT_ALL is enabled.
+# The default value is: NO.
+
+HIDE_UNDOC_CLASSES     = NO
+
+# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, doxygen will hide all friend
+# (class|struct|union) declarations. If set to NO, these declarations will be
+# included in the documentation.
+# The default value is: NO.
+
+HIDE_FRIEND_COMPOUNDS  = NO
+
+# If the HIDE_IN_BODY_DOCS tag is set to YES, doxygen will hide any
+# documentation blocks found inside the body of a function. If set to NO, these
+# blocks will be appended to the function's detailed documentation block.
+# The default value is: NO.
+
+HIDE_IN_BODY_DOCS      = NO
+
+# The INTERNAL_DOCS tag determines if documentation that is typed after a
+# \internal command is included. If the tag is set to NO then the documentation
+# will be excluded. Set it to YES to include the internal documentation.
+# The default value is: NO.
+
+INTERNAL_DOCS          = NO
+
+# If the CASE_SENSE_NAMES tag is set to NO then doxygen will only generate file
+# names in lower-case letters. If set to YES, upper-case letters are also
+# allowed. This is useful if you have classes or files whose names only differ
+# in case and if your file system supports case sensitive file names. Windows
+# and Mac users are advised to set this option to NO.
+# The default value is: system dependent.
+
+CASE_SENSE_NAMES       = YES
+
+# If the HIDE_SCOPE_NAMES tag is set to NO then doxygen will show members with
+# their full class and namespace scopes in the documentation. If set to YES, the
+# scope will be hidden.
+# The default value is: NO.
+
+HIDE_SCOPE_NAMES       = NO
+
+# If the HIDE_COMPOUND_REFERENCE tag is set to NO (default) then doxygen will
+# append additional text to a page's title, such as Class Reference. If set to
+# YES the compound reference will be hidden.
+# The default value is: NO.
+
+HIDE_COMPOUND_REFERENCE= NO
+
+# If the SHOW_INCLUDE_FILES tag is set to YES then doxygen will put a list of
+# the files that are included by a file in the documentation of that file.
+# The default value is: YES.
+
+SHOW_INCLUDE_FILES     = YES
+
+# If the SHOW_GROUPED_MEMB_INC tag is set to YES then Doxygen will add for each
+# grouped member an include statement to the documentation, telling the reader
+# which file to include in order to use the member.
+# The default value is: NO.
+
+SHOW_GROUPED_MEMB_INC  = NO
+
+# If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include
+# files with double quotes in the documentation rather than with sharp brackets.
+# The default value is: NO.
+
+FORCE_LOCAL_INCLUDES   = NO
+
+# If the INLINE_INFO tag is set to YES then a tag [inline] is inserted in the
+# documentation for inline members.
+# The default value is: YES.
+
+INLINE_INFO            = YES
+
+# If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the
+# (detailed) documentation of file and class members alphabetically by member
+# name. If set to NO, the members will appear in declaration order.
+# The default value is: YES.
+
+SORT_MEMBER_DOCS       = YES
+
+# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief
+# descriptions of file, namespace and class members alphabetically by member
+# name. If set to NO, the members will appear in declaration order. Note that
+# this will also influence the order of the classes in the class list.
+# The default value is: NO.
+
+SORT_BRIEF_DOCS        = NO
+
+# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the
+# (brief and detailed) documentation of class members so that constructors and
+# destructors are listed first. If set to NO the constructors will appear in the
+# respective orders defined by SORT_BRIEF_DOCS and SORT_MEMBER_DOCS.
+# Note: If SORT_BRIEF_DOCS is set to NO this option is ignored for sorting brief
+# member documentation.
+# Note: If SORT_MEMBER_DOCS is set to NO this option is ignored for sorting
+# detailed member documentation.
+# The default value is: NO.
+
+SORT_MEMBERS_CTORS_1ST = NO
+
+# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the hierarchy
+# of group names into alphabetical order. If set to NO the group names will
+# appear in their defined order.
+# The default value is: NO.
+
+SORT_GROUP_NAMES       = NO
+
+# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be sorted by
+# fully-qualified names, including namespaces. If set to NO, the class list will
+# be sorted only by class name, not including the namespace part.
+# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES.
+# Note: This option applies only to the class list, not to the alphabetical
+# list.
+# The default value is: NO.
+
+SORT_BY_SCOPE_NAME     = NO
+
+# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to do proper
+# type resolution of all parameters of a function it will reject a match between
+# the prototype and the implementation of a member function even if there is
+# only one candidate or it is obvious which candidate to choose by doing a
+# simple string match. By disabling STRICT_PROTO_MATCHING doxygen will still
+# accept a match between prototype and implementation in such cases.
+# The default value is: NO.
+
+STRICT_PROTO_MATCHING  = NO
+
+# The GENERATE_TODOLIST tag can be used to enable (YES) or disable (NO) the todo
+# list. This list is created by putting \todo commands in the documentation.
+# The default value is: YES.
+
+GENERATE_TODOLIST      = YES
+
+# The GENERATE_TESTLIST tag can be used to enable (YES) or disable (NO) the test
+# list. This list is created by putting \test commands in the documentation.
+# The default value is: YES.
+
+GENERATE_TESTLIST      = YES
+
+# The GENERATE_BUGLIST tag can be used to enable (YES) or disable (NO) the bug
+# list. This list is created by putting \bug commands in the documentation.
+# The default value is: YES.
+
+GENERATE_BUGLIST       = YES
+
+# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or disable (NO)
+# the deprecated list. This list is created by putting \deprecated commands in
+# the documentation.
+# The default value is: YES.
+
+GENERATE_DEPRECATEDLIST= YES
+
+# The ENABLED_SECTIONS tag can be used to enable conditional documentation
+# sections, marked by \if <section_label> ... \endif and \cond <section_label>
+# ... \endcond blocks.
+
+ENABLED_SECTIONS       =
+
+# The MAX_INITIALIZER_LINES tag determines the maximum number of lines that the
+# initial value of a variable or macro / define can have for it to appear in the
+# documentation. If the initializer consists of more lines than specified here
+# it will be hidden. Use a value of 0 to hide initializers completely. The
+# appearance of the value of individual variables and macros / defines can be
+# controlled using \showinitializer or \hideinitializer command in the
+# documentation regardless of this setting.
+# Minimum value: 0, maximum value: 10000, default value: 30.
+
+MAX_INITIALIZER_LINES  = 30
+
+# Set the SHOW_USED_FILES tag to NO to disable the list of files generated at
+# the bottom of the documentation of classes and structs. If set to YES, the
+# list will mention the files that were used to generate the documentation.
+# The default value is: YES.
+
+SHOW_USED_FILES        = YES
+
+# Set the SHOW_FILES tag to NO to disable the generation of the Files page. This
+# will remove the Files entry from the Quick Index and from the Folder Tree View
+# (if specified).
+# The default value is: YES.
+
+SHOW_FILES             = YES
+
+# Set the SHOW_NAMESPACES tag to NO to disable the generation of the Namespaces
+# page. This will remove the Namespaces entry from the Quick Index and from the
+# Folder Tree View (if specified).
+# The default value is: YES.
+
+SHOW_NAMESPACES        = YES
+
+# The FILE_VERSION_FILTER tag can be used to specify a program or script that
+# doxygen should invoke to get the current version for each file (typically from
+# the version control system). Doxygen will invoke the program by executing (via
+# popen()) the command command input-file, where command is the value of the
+# FILE_VERSION_FILTER tag, and input-file is the name of an input file provided
+# by doxygen. Whatever the program writes to standard output is used as the file
+# version. For an example see the documentation.
+
+FILE_VERSION_FILTER    =
+
+# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed
+# by doxygen. The layout file controls the global structure of the generated
+# output files in an output format independent way. To create the layout file
+# that represents doxygen's defaults, run doxygen with the -l option. You can
+# optionally specify a file name after the option, if omitted DoxygenLayout.xml
+# will be used as the name of the layout file.
+#
+# Note that if you run doxygen from a directory containing a file called
+# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE
+# tag is left empty.
+
+LAYOUT_FILE            =
+
+# The CITE_BIB_FILES tag can be used to specify one or more bib files containing
+# the reference definitions. This must be a list of .bib files. The .bib
+# extension is automatically appended if omitted. This requires the bibtex tool
+# to be installed. See also http://en.wikipedia.org/wiki/BibTeX for more info.
+# For LaTeX the style of the bibliography can be controlled using
+# LATEX_BIB_STYLE. To use this feature you need bibtex and perl available in the
+# search path. See also \cite for info how to create references.
+
+CITE_BIB_FILES         =
+
+#---------------------------------------------------------------------------
+# Configuration options related to warning and progress messages
+#---------------------------------------------------------------------------
+
+# The QUIET tag can be used to turn on/off the messages that are generated to
+# standard output by doxygen. If QUIET is set to YES this implies that the
+# messages are off.
+# The default value is: NO.
+
+QUIET                  = NO
+
+# The WARNINGS tag can be used to turn on/off the warning messages that are
+# generated to standard error (stderr) by doxygen. If WARNINGS is set to YES
+# this implies that the warnings are on.
+#
+# Tip: Turn warnings on while writing the documentation.
+# The default value is: YES.
+
+WARNINGS               = YES
+
+# If the WARN_IF_UNDOCUMENTED tag is set to YES then doxygen will generate
+# warnings for undocumented members. If EXTRACT_ALL is set to YES then this flag
+# will automatically be disabled.
+# The default value is: YES.
+
+WARN_IF_UNDOCUMENTED   = YES
+
+# If the WARN_IF_DOC_ERROR tag is set to YES, doxygen will generate warnings for
+# potential errors in the documentation, such as not documenting some parameters
+# in a documented function, or documenting parameters that don't exist or using
+# markup commands wrongly.
+# The default value is: YES.
+
+WARN_IF_DOC_ERROR      = YES
+
+# This WARN_NO_PARAMDOC option can be enabled to get warnings for functions that
+# are documented, but have no documentation for their parameters or return
+# value. If set to NO, doxygen will only warn about wrong or incomplete
+# parameter documentation, but not about the absence of documentation.
+# The default value is: NO.
+
+WARN_NO_PARAMDOC       = NO
+
+# If the WARN_AS_ERROR tag is set to YES then doxygen will immediately stop when
+# a warning is encountered.
+# The default value is: NO.
+
+WARN_AS_ERROR          = NO
+
+# The WARN_FORMAT tag determines the format of the warning messages that doxygen
+# can produce. The string should contain the $file, $line, and $text tags, which
+# will be replaced by the file and line number from which the warning originated
+# and the warning text. Optionally the format may contain $version, which will
+# be replaced by the version of the file (if it could be obtained via
+# FILE_VERSION_FILTER)
+# The default value is: $file:$line: $text.
+
+WARN_FORMAT            = "$file:$line: $text"
+
+# The WARN_LOGFILE tag can be used to specify a file to which warning and error
+# messages should be written. If left blank the output is written to standard
+# error (stderr).
+
+WARN_LOGFILE           =
+
+#---------------------------------------------------------------------------
+# Configuration options related to the input files
+#---------------------------------------------------------------------------
+
+# The INPUT tag is used to specify the files and/or directories that contain
+# documented source files. You may enter file names like myfile.cpp or
+# directories like /usr/src/myproject. Separate the files or directories with
+# spaces. See also FILE_PATTERNS and EXTENSION_MAPPING
+# Note: If this tag is empty the current directory is searched.
+
+INPUT                  = src app 
+
+# This tag can be used to specify the character encoding of the source files
+# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
+# libiconv (or the iconv built into libc) for the transcoding. See the libiconv
+# documentation (see: http://www.gnu.org/software/libiconv) for the list of
+# possible encodings.
+# The default value is: UTF-8.
+
+INPUT_ENCODING         = UTF-8
+
+# If the value of the INPUT tag contains directories, you can use the
+# FILE_PATTERNS tag to specify one or more wildcard patterns (like *.cpp and
+# *.h) to filter out the source-files in the directories.
+#
+# Note that for custom extensions or not directly supported extensions you also
+# need to set EXTENSION_MAPPING for the extension otherwise the files are not
+# read by doxygen.
+#
+# If left blank the following patterns are tested:*.c, *.cc, *.cxx, *.cpp,
+# *.c++, *.java, *.ii, *.ixx, *.ipp, *.i++, *.inl, *.idl, *.ddl, *.odl, *.h,
+# *.hh, *.hxx, *.hpp, *.h++, *.cs, *.d, *.php, *.php4, *.php5, *.phtml, *.inc,
+# *.m, *.markdown, *.md, *.mm, *.dox, *.py, *.pyw, *.f90, *.f95, *.f03, *.f08,
+# *.f, *.for, *.tcl, *.vhd, *.vhdl, *.ucf and *.qsf.
+
+FILE_PATTERNS          = *.c \
+                         *.cc \
+                         *.cxx \
+                         *.cpp \
+                         *.c++ \
+                         *.java \
+                         *.ii \
+                         *.ixx \
+                         *.ipp \
+                         *.i++ \
+                         *.inl \
+                         *.idl \
+                         *.ddl \
+                         *.odl \
+                         *.h \
+                         *.hh \
+                         *.hxx \
+                         *.hpp \
+                         *.h++ \
+                         *.cs \
+                         *.d \
+                         *.php \
+                         *.php4 \
+                         *.php5 \
+                         *.phtml \
+                         *.inc \
+                         *.m \
+                         *.markdown \
+                         *.md \
+                         *.mm \
+                         *.dox \
+                         *.py \
+                         *.pyw \
+                         *.f90 \
+                         *.f95 \
+                         *.f03 \
+                         *.f08 \
+                         *.f \
+                         *.for \
+                         *.tcl \
+                         *.vhd \
+                         *.vhdl \
+                         *.ucf \
+                         *.qsf
+
+# The RECURSIVE tag can be used to specify whether or not subdirectories should
+# be searched for input files as well.
+# The default value is: NO.
+
+RECURSIVE              = YES
+
+# The EXCLUDE tag can be used to specify files and/or directories that should be
+# excluded from the INPUT source files. This way you can easily exclude a
+# subdirectory from a directory tree whose root is specified with the INPUT tag.
+#
+# Note that relative paths are relative to the directory from which doxygen is
+# run.
+
+EXCLUDE                = 
+
+# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or
+# directories that are symbolic links (a Unix file system feature) are excluded
+# from the input.
+# The default value is: NO.
+
+EXCLUDE_SYMLINKS       = NO
+
+# If the value of the INPUT tag contains directories, you can use the
+# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude
+# certain files from those directories.
+#
+# Note that the wildcards are matched against the file with absolute path, so to
+# exclude all test directories for example use the pattern */test/*
+
+EXCLUDE_PATTERNS       =
+
+# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names
+# (namespaces, classes, functions, etc.) that should be excluded from the
+# output. The symbol name can be a fully qualified name, a word, or if the
+# wildcard * is used, a substring. Examples: ANamespace, AClass,
+# AClass::ANamespace, ANamespace::*Test
+#
+# Note that the wildcards are matched against the file with absolute path, so to
+# exclude all test directories use the pattern */test/*
+
+EXCLUDE_SYMBOLS        =
+
+# The EXAMPLE_PATH tag can be used to specify one or more files or directories
+# that contain example code fragments that are included (see the \include
+# command).
+
+EXAMPLE_PATH           =
+
+# If the value of the EXAMPLE_PATH tag contains directories, you can use the
+# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and
+# *.h) to filter out the source-files in the directories. If left blank all
+# files are included.
+
+EXAMPLE_PATTERNS       = *
+
+# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be
+# searched for input files to be used with the \include or \dontinclude commands
+# irrespective of the value of the RECURSIVE tag.
+# The default value is: NO.
+
+EXAMPLE_RECURSIVE      = NO
+
+# The IMAGE_PATH tag can be used to specify one or more files or directories
+# that contain images that are to be included in the documentation (see the
+# \image command).
+
+IMAGE_PATH             =
+
+# The INPUT_FILTER tag can be used to specify a program that doxygen should
+# invoke to filter for each input file. Doxygen will invoke the filter program
+# by executing (via popen()) the command:
+#
+# <filter> <input-file>
+#
+# where <filter> is the value of the INPUT_FILTER tag, and <input-file> is the
+# name of an input file. Doxygen will then use the output that the filter
+# program writes to standard output. If FILTER_PATTERNS is specified, this tag
+# will be ignored.
+#
+# Note that the filter must not add or remove lines; it is applied before the
+# code is scanned, but not when the output code is generated. If lines are added
+# or removed, the anchors will not be placed correctly.
+#
+# Note that for custom extensions or not directly supported extensions you also
+# need to set EXTENSION_MAPPING for the extension otherwise the files are not
+# properly processed by doxygen.
+
+INPUT_FILTER           =
+
+# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern
+# basis. Doxygen will compare the file name with each pattern and apply the
+# filter if there is a match. The filters are a list of the form: pattern=filter
+# (like *.cpp=my_cpp_filter). See INPUT_FILTER for further information on how
+# filters are used. If the FILTER_PATTERNS tag is empty or if none of the
+# patterns match the file name, INPUT_FILTER is applied.
+#
+# Note that for custom extensions or not directly supported extensions you also
+# need to set EXTENSION_MAPPING for the extension otherwise the files are not
+# properly processed by doxygen.
+
+FILTER_PATTERNS        =
+
+# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using
+# INPUT_FILTER) will also be used to filter the input files that are used for
+# producing the source files to browse (i.e. when SOURCE_BROWSER is set to YES).
+# The default value is: NO.
+
+FILTER_SOURCE_FILES    = NO
+
+# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file
+# pattern. A pattern will override the setting for FILTER_PATTERN (if any) and
+# it is also possible to disable source filtering for a specific pattern using
+# *.ext= (so without naming a filter).
+# This tag requires that the tag FILTER_SOURCE_FILES is set to YES.
+
+FILTER_SOURCE_PATTERNS =
+
+# If the USE_MDFILE_AS_MAINPAGE tag refers to the name of a markdown file that
+# is part of the input, its contents will be placed on the main page
+# (index.html). This can be useful if you have a project on for instance GitHub
+# and want to reuse the introduction page also for the doxygen output.
+
+USE_MDFILE_AS_MAINPAGE =
+
+#---------------------------------------------------------------------------
+# Configuration options related to source browsing
+#---------------------------------------------------------------------------
+
+# If the SOURCE_BROWSER tag is set to YES then a list of source files will be
+# generated. Documented entities will be cross-referenced with these sources.
+#
+# Note: To get rid of all source code in the generated output, make sure that
+# also VERBATIM_HEADERS is set to NO.
+# The default value is: NO.
+
+SOURCE_BROWSER         = NO
+
+# Setting the INLINE_SOURCES tag to YES will include the body of functions,
+# classes and enums directly into the documentation.
+# The default value is: NO.
+
+INLINE_SOURCES         = NO
+
+# Setting the STRIP_CODE_COMMENTS tag to YES will instruct doxygen to hide any
+# special comment blocks from generated source code fragments. Normal C, C++ and
+# Fortran comments will always remain visible.
+# The default value is: YES.
+
+STRIP_CODE_COMMENTS    = YES
+
+# If the REFERENCED_BY_RELATION tag is set to YES then for each documented
+# function all documented functions referencing it will be listed.
+# The default value is: NO.
+
+REFERENCED_BY_RELATION = NO
+
+# If the REFERENCES_RELATION tag is set to YES then for each documented function
+# all documented entities called/used by that function will be listed.
+# The default value is: NO.
+
+REFERENCES_RELATION    = NO
+
+# If the REFERENCES_LINK_SOURCE tag is set to YES and SOURCE_BROWSER tag is set
+# to YES then the hyperlinks from functions in REFERENCES_RELATION and
+# REFERENCED_BY_RELATION lists will link to the source code. Otherwise they will
+# link to the documentation.
+# The default value is: YES.
+
+REFERENCES_LINK_SOURCE = YES
+
+# If SOURCE_TOOLTIPS is enabled (the default) then hovering a hyperlink in the
+# source code will show a tooltip with additional information such as prototype,
+# brief description and links to the definition and documentation. Since this
+# will make the HTML file larger and loading of large files a bit slower, you
+# can opt to disable this feature.
+# The default value is: YES.
+# This tag requires that the tag SOURCE_BROWSER is set to YES.
+
+SOURCE_TOOLTIPS        = YES
+
+# If the USE_HTAGS tag is set to YES then the references to source code will
+# point to the HTML generated by the htags(1) tool instead of doxygen built-in
+# source browser. The htags tool is part of GNU's global source tagging system
+# (see http://www.gnu.org/software/global/global.html). You will need version
+# 4.8.6 or higher.
+#
+# To use it do the following:
+# - Install the latest version of global
+# - Enable SOURCE_BROWSER and USE_HTAGS in the config file
+# - Make sure the INPUT points to the root of the source tree
+# - Run doxygen as normal
+#
+# Doxygen will invoke htags (and that will in turn invoke gtags), so these
+# tools must be available from the command line (i.e. in the search path).
+#
+# The result: instead of the source browser generated by doxygen, the links to
+# source code will now point to the output of htags.
+# The default value is: NO.
+# This tag requires that the tag SOURCE_BROWSER is set to YES.
+
+USE_HTAGS              = NO
+
+# If the VERBATIM_HEADERS tag is set the YES then doxygen will generate a
+# verbatim copy of the header file for each class for which an include is
+# specified. Set to NO to disable this.
+# See also: Section \class.
+# The default value is: YES.
+
+VERBATIM_HEADERS       = YES
+
+# If the CLANG_ASSISTED_PARSING tag is set to YES then doxygen will use the
+# clang parser (see: http://clang.llvm.org/) for more accurate parsing at the
+# cost of reduced performance. This can be particularly helpful with template
+# rich C++ code for which doxygen's built-in parser lacks the necessary type
+# information.
+# Note: The availability of this option depends on whether or not doxygen was
+# generated with the -Duse-libclang=ON option for CMake.
+# The default value is: NO.
+
+CLANG_ASSISTED_PARSING = NO
+
+# If clang assisted parsing is enabled you can provide the compiler with command
+# line options that you would normally use when invoking the compiler. Note that
+# the include paths will already be set by doxygen for the files and directories
+# specified with INPUT and INCLUDE_PATH.
+# This tag requires that the tag CLANG_ASSISTED_PARSING is set to YES.
+
+CLANG_OPTIONS          =
+
+#---------------------------------------------------------------------------
+# Configuration options related to the alphabetical class index
+#---------------------------------------------------------------------------
+
+# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index of all
+# compounds will be generated. Enable this if the project contains a lot of
+# classes, structs, unions or interfaces.
+# The default value is: YES.
+
+ALPHABETICAL_INDEX     = YES
+
+# The COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns in
+# which the alphabetical index list will be split.
+# Minimum value: 1, maximum value: 20, default value: 5.
+# This tag requires that the tag ALPHABETICAL_INDEX is set to YES.
+
+COLS_IN_ALPHA_INDEX    = 5
+
+# In case all classes in a project start with a common prefix, all classes will
+# be put under the same header in the alphabetical index. The IGNORE_PREFIX tag
+# can be used to specify a prefix (or a list of prefixes) that should be ignored
+# while generating the index headers.
+# This tag requires that the tag ALPHABETICAL_INDEX is set to YES.
+
+IGNORE_PREFIX          =
+
+#---------------------------------------------------------------------------
+# Configuration options related to the HTML output
+#---------------------------------------------------------------------------
+
+# If the GENERATE_HTML tag is set to YES, doxygen will generate HTML output
+# The default value is: YES.
+
+GENERATE_HTML          = YES
+
+# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. If a
+# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
+# it.
+# The default directory is: html.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_OUTPUT            = html
+
+# The HTML_FILE_EXTENSION tag can be used to specify the file extension for each
+# generated HTML page (for example: .htm, .php, .asp).
+# The default value is: .html.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_FILE_EXTENSION    = .html
+
+# The HTML_HEADER tag can be used to specify a user-defined HTML header file for
+# each generated HTML page. If the tag is left blank doxygen will generate a
+# standard header.
+#
+# To get valid HTML the header file that includes any scripts and style sheets
+# that doxygen needs, which is dependent on the configuration options used (e.g.
+# the setting GENERATE_TREEVIEW). It is highly recommended to start with a
+# default header using
+# doxygen -w html new_header.html new_footer.html new_stylesheet.css
+# YourConfigFile
+# and then modify the file new_header.html. See also section "Doxygen usage"
+# for information on how to generate the default header that doxygen normally
+# uses.
+# Note: The header is subject to change so you typically have to regenerate the
+# default header when upgrading to a newer version of doxygen. For a description
+# of the possible markers and block names see the documentation.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_HEADER            =
+
+# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each
+# generated HTML page. If the tag is left blank doxygen will generate a standard
+# footer. See HTML_HEADER for more information on how to generate a default
+# footer and what special commands can be used inside the footer. See also
+# section "Doxygen usage" for information on how to generate the default footer
+# that doxygen normally uses.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_FOOTER            =
+
+# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style
+# sheet that is used by each HTML page. It can be used to fine-tune the look of
+# the HTML output. If left blank doxygen will generate a default style sheet.
+# See also section "Doxygen usage" for information on how to generate the style
+# sheet that doxygen normally uses.
+# Note: It is recommended to use HTML_EXTRA_STYLESHEET instead of this tag, as
+# it is more robust and this tag (HTML_STYLESHEET) will in the future become
+# obsolete.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_STYLESHEET        =
+
+# The HTML_EXTRA_STYLESHEET tag can be used to specify additional user-defined
+# cascading style sheets that are included after the standard style sheets
+# created by doxygen. Using this option one can overrule certain style aspects.
+# This is preferred over using HTML_STYLESHEET since it does not replace the
+# standard style sheet and is therefore more robust against future updates.
+# Doxygen will copy the style sheet files to the output directory.
+# Note: The order of the extra style sheet files is of importance (e.g. the last
+# style sheet in the list overrules the setting of the previous ones in the
+# list). For an example see the documentation.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_EXTRA_STYLESHEET  =
+
+# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or
+# other source files which should be copied to the HTML output directory. Note
+# that these files will be copied to the base HTML output directory. Use the
+# $relpath^ marker in the HTML_HEADER and/or HTML_FOOTER files to load these
+# files. In the HTML_STYLESHEET file, use the file name only. Also note that the
+# files will be copied as-is; there are no commands or markers available.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_EXTRA_FILES       =
+
+# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen
+# will adjust the colors in the style sheet and background images according to
+# this color. Hue is specified as an angle on a colorwheel, see
+# http://en.wikipedia.org/wiki/Hue for more information. For instance the value
+# 0 represents red, 60 is yellow, 120 is green, 180 is cyan, 240 is blue, 300
+# purple, and 360 is red again.
+# Minimum value: 0, maximum value: 359, default value: 220.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_COLORSTYLE_HUE    = 220
+
+# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors
+# in the HTML output. For a value of 0 the output will use grayscales only. A
+# value of 255 will produce the most vivid colors.
+# Minimum value: 0, maximum value: 255, default value: 100.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_COLORSTYLE_SAT    = 100
+
+# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the
+# luminance component of the colors in the HTML output. Values below 100
+# gradually make the output lighter, whereas values above 100 make the output
+# darker. The value divided by 100 is the actual gamma applied, so 80 represents
+# a gamma of 0.8, The value 220 represents a gamma of 2.2, and 100 does not
+# change the gamma.
+# Minimum value: 40, maximum value: 240, default value: 80.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_COLORSTYLE_GAMMA  = 80
+
+# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML
+# page will contain the date and time when the page was generated. Setting this
+# to YES can help to show when doxygen was last run and thus if the
+# documentation is up to date.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_TIMESTAMP         = NO
+
+# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML
+# documentation will contain sections that can be hidden and shown after the
+# page has loaded.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_DYNAMIC_SECTIONS  = NO
+
+# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries
+# shown in the various tree structured indices initially; the user can expand
+# and collapse entries dynamically later on. Doxygen will expand the tree to
+# such a level that at most the specified number of entries are visible (unless
+# a fully collapsed tree already exceeds this amount). So setting the number of
+# entries 1 will produce a full collapsed tree by default. 0 is a special value
+# representing an infinite number of entries and will result in a full expanded
+# tree by default.
+# Minimum value: 0, maximum value: 9999, default value: 100.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+HTML_INDEX_NUM_ENTRIES = 100
+
+# If the GENERATE_DOCSET tag is set to YES, additional index files will be
+# generated that can be used as input for Apple's Xcode 3 integrated development
+# environment (see: http://developer.apple.com/tools/xcode/), introduced with
+# OSX 10.5 (Leopard). To create a documentation set, doxygen will generate a
+# Makefile in the HTML output directory. Running make will produce the docset in
+# that directory and running make install will install the docset in
+# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find it at
+# startup. See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html
+# for more information.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+GENERATE_DOCSET        = NO
+
+# This tag determines the name of the docset feed. A documentation feed provides
+# an umbrella under which multiple documentation sets from a single provider
+# (such as a company or product suite) can be grouped.
+# The default value is: Doxygen generated docs.
+# This tag requires that the tag GENERATE_DOCSET is set to YES.
+
+DOCSET_FEEDNAME        = "Doxygen generated docs"
+
+# This tag specifies a string that should uniquely identify the documentation
+# set bundle. This should be a reverse domain-name style string, e.g.
+# com.mycompany.MyDocSet. Doxygen will append .docset to the name.
+# The default value is: org.doxygen.Project.
+# This tag requires that the tag GENERATE_DOCSET is set to YES.
+
+DOCSET_BUNDLE_ID       = org.doxygen.Project
+
+# The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify
+# the documentation publisher. This should be a reverse domain-name style
+# string, e.g. com.mycompany.MyDocSet.documentation.
+# The default value is: org.doxygen.Publisher.
+# This tag requires that the tag GENERATE_DOCSET is set to YES.
+
+DOCSET_PUBLISHER_ID    = org.doxygen.Publisher
+
+# The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher.
+# The default value is: Publisher.
+# This tag requires that the tag GENERATE_DOCSET is set to YES.
+
+DOCSET_PUBLISHER_NAME  = Publisher
+
+# If the GENERATE_HTMLHELP tag is set to YES then doxygen generates three
+# additional HTML index files: index.hhp, index.hhc, and index.hhk. The
+# index.hhp is a project file that can be read by Microsoft's HTML Help Workshop
+# (see: http://www.microsoft.com/en-us/download/details.aspx?id=21138) on
+# Windows.
+#
+# The HTML Help Workshop contains a compiler that can convert all HTML output
+# generated by doxygen into a single compiled HTML file (.chm). Compiled HTML
+# files are now used as the Windows 98 help format, and will replace the old
+# Windows help format (.hlp) on all Windows platforms in the future. Compressed
+# HTML files also contain an index, a table of contents, and you can search for
+# words in the documentation. The HTML workshop also contains a viewer for
+# compressed HTML files.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+GENERATE_HTMLHELP      = NO
+
+# The CHM_FILE tag can be used to specify the file name of the resulting .chm
+# file. You can add a path in front of the file if the result should not be
+# written to the html output directory.
+# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
+
+CHM_FILE               =
+
+# The HHC_LOCATION tag can be used to specify the location (absolute path
+# including file name) of the HTML help compiler (hhc.exe). If non-empty,
+# doxygen will try to run the HTML help compiler on the generated index.hhp.
+# The file has to be specified with full path.
+# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
+
+HHC_LOCATION           =
+
+# The GENERATE_CHI flag controls if a separate .chi index file is generated
+# (YES) or that it should be included in the master .chm file (NO).
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
+
+GENERATE_CHI           = NO
+
+# The CHM_INDEX_ENCODING is used to encode HtmlHelp index (hhk), content (hhc)
+# and project file content.
+# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
+
+CHM_INDEX_ENCODING     =
+
+# The BINARY_TOC flag controls whether a binary table of contents is generated
+# (YES) or a normal table of contents (NO) in the .chm file. Furthermore it
+# enables the Previous and Next buttons.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
+
+BINARY_TOC             = NO
+
+# The TOC_EXPAND flag can be set to YES to add extra items for group members to
+# the table of contents of the HTML help documentation and to the tree view.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
+
+TOC_EXPAND             = NO
+
+# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and
+# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated that
+# can be used as input for Qt's qhelpgenerator to generate a Qt Compressed Help
+# (.qch) of the generated HTML documentation.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+GENERATE_QHP           = NO
+
+# If the QHG_LOCATION tag is specified, the QCH_FILE tag can be used to specify
+# the file name of the resulting .qch file. The path specified is relative to
+# the HTML output folder.
+# This tag requires that the tag GENERATE_QHP is set to YES.
+
+QCH_FILE               =
+
+# The QHP_NAMESPACE tag specifies the namespace to use when generating Qt Help
+# Project output. For more information please see Qt Help Project / Namespace
+# (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#namespace).
+# The default value is: org.doxygen.Project.
+# This tag requires that the tag GENERATE_QHP is set to YES.
+
+QHP_NAMESPACE          = org.doxygen.Project
+
+# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating Qt
+# Help Project output. For more information please see Qt Help Project / Virtual
+# Folders (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#virtual-
+# folders).
+# The default value is: doc.
+# This tag requires that the tag GENERATE_QHP is set to YES.
+
+QHP_VIRTUAL_FOLDER     = doc
+
+# If the QHP_CUST_FILTER_NAME tag is set, it specifies the name of a custom
+# filter to add. For more information please see Qt Help Project / Custom
+# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-
+# filters).
+# This tag requires that the tag GENERATE_QHP is set to YES.
+
+QHP_CUST_FILTER_NAME   =
+
+# The QHP_CUST_FILTER_ATTRS tag specifies the list of the attributes of the
+# custom filter to add. For more information please see Qt Help Project / Custom
+# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-
+# filters).
+# This tag requires that the tag GENERATE_QHP is set to YES.
+
+QHP_CUST_FILTER_ATTRS  =
+
+# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this
+# project's filter section matches. Qt Help Project / Filter Attributes (see:
+# http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes).
+# This tag requires that the tag GENERATE_QHP is set to YES.
+
+QHP_SECT_FILTER_ATTRS  =
+
+# The QHG_LOCATION tag can be used to specify the location of Qt's
+# qhelpgenerator. If non-empty doxygen will try to run qhelpgenerator on the
+# generated .qhp file.
+# This tag requires that the tag GENERATE_QHP is set to YES.
+
+QHG_LOCATION           =
+
+# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files will be
+# generated, together with the HTML files, they form an Eclipse help plugin. To
+# install this plugin and make it available under the help contents menu in
+# Eclipse, the contents of the directory containing the HTML and XML files needs
+# to be copied into the plugins directory of eclipse. The name of the directory
+# within the plugins directory should be the same as the ECLIPSE_DOC_ID value.
+# After copying Eclipse needs to be restarted before the help appears.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+GENERATE_ECLIPSEHELP   = NO
+
+# A unique identifier for the Eclipse help plugin. When installing the plugin
+# the directory name containing the HTML and XML files should also have this
+# name. Each documentation set should have its own identifier.
+# The default value is: org.doxygen.Project.
+# This tag requires that the tag GENERATE_ECLIPSEHELP is set to YES.
+
+ECLIPSE_DOC_ID         = org.doxygen.Project
+
+# If you want full control over the layout of the generated HTML pages it might
+# be necessary to disable the index and replace it with your own. The
+# DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) at top
+# of each HTML page. A value of NO enables the index and the value YES disables
+# it. Since the tabs in the index contain the same information as the navigation
+# tree, you can set this option to YES if you also set GENERATE_TREEVIEW to YES.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+DISABLE_INDEX          = NO
+
+# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index
+# structure should be generated to display hierarchical information. If the tag
+# value is set to YES, a side panel will be generated containing a tree-like
+# index structure (just like the one that is generated for HTML Help). For this
+# to work a browser that supports JavaScript, DHTML, CSS and frames is required
+# (i.e. any modern browser). Windows users are probably better off using the
+# HTML help feature. Via custom style sheets (see HTML_EXTRA_STYLESHEET) one can
+# further fine-tune the look of the index. As an example, the default style
+# sheet generated by doxygen has an example that shows how to put an image at
+# the root of the tree instead of the PROJECT_NAME. Since the tree basically has
+# the same information as the tab index, you could consider setting
+# DISABLE_INDEX to YES when enabling this option.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+GENERATE_TREEVIEW      = NO
+
+# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that
+# doxygen will group on one line in the generated HTML documentation.
+#
+# Note that a value of 0 will completely suppress the enum values from appearing
+# in the overview section.
+# Minimum value: 0, maximum value: 20, default value: 4.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+ENUM_VALUES_PER_LINE   = 4
+
+# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used
+# to set the initial width (in pixels) of the frame in which the tree is shown.
+# Minimum value: 0, maximum value: 1500, default value: 250.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+TREEVIEW_WIDTH         = 250
+
+# If the EXT_LINKS_IN_WINDOW option is set to YES, doxygen will open links to
+# external symbols imported via tag files in a separate window.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+EXT_LINKS_IN_WINDOW    = NO
+
+# Use this tag to change the font size of LaTeX formulas included as images in
+# the HTML documentation. When you change the font size after a successful
+# doxygen run you need to manually remove any form_*.png images from the HTML
+# output directory to force them to be regenerated.
+# Minimum value: 8, maximum value: 50, default value: 10.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+FORMULA_FONTSIZE       = 10
+
+# Use the FORMULA_TRANPARENT tag to determine whether or not the images
+# generated for formulas are transparent PNGs. Transparent PNGs are not
+# supported properly for IE 6.0, but are supported on all modern browsers.
+#
+# Note that when changing this option you need to delete any form_*.png files in
+# the HTML output directory before the changes have effect.
+# The default value is: YES.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+FORMULA_TRANSPARENT    = YES
+
+# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax (see
+# http://www.mathjax.org) which uses client side Javascript for the rendering
+# instead of using pre-rendered bitmaps. Use this if you do not have LaTeX
+# installed or if you want to formulas look prettier in the HTML output. When
+# enabled you may also need to install MathJax separately and configure the path
+# to it using the MATHJAX_RELPATH option.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+USE_MATHJAX            = NO
+
+# When MathJax is enabled you can set the default output format to be used for
+# the MathJax output. See the MathJax site (see:
+# http://docs.mathjax.org/en/latest/output.html) for more details.
+# Possible values are: HTML-CSS (which is slower, but has the best
+# compatibility), NativeMML (i.e. MathML) and SVG.
+# The default value is: HTML-CSS.
+# This tag requires that the tag USE_MATHJAX is set to YES.
+
+MATHJAX_FORMAT         = HTML-CSS
+
+# When MathJax is enabled you need to specify the location relative to the HTML
+# output directory using the MATHJAX_RELPATH option. The destination directory
+# should contain the MathJax.js script. For instance, if the mathjax directory
+# is located at the same level as the HTML output directory, then
+# MATHJAX_RELPATH should be ../mathjax. The default value points to the MathJax
+# Content Delivery Network so you can quickly see the result without installing
+# MathJax. However, it is strongly recommended to install a local copy of
+# MathJax from http://www.mathjax.org before deployment.
+# The default value is: http://cdn.mathjax.org/mathjax/latest.
+# This tag requires that the tag USE_MATHJAX is set to YES.
+
+MATHJAX_RELPATH        = http://cdn.mathjax.org/mathjax/latest
+
+# The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax
+# extension names that should be enabled during MathJax rendering. For example
+# MATHJAX_EXTENSIONS = TeX/AMSmath TeX/AMSsymbols
+# This tag requires that the tag USE_MATHJAX is set to YES.
+
+MATHJAX_EXTENSIONS     =
+
+# The MATHJAX_CODEFILE tag can be used to specify a file with javascript pieces
+# of code that will be used on startup of the MathJax code. See the MathJax site
+# (see: http://docs.mathjax.org/en/latest/output.html) for more details. For an
+# example see the documentation.
+# This tag requires that the tag USE_MATHJAX is set to YES.
+
+MATHJAX_CODEFILE       =
+
+# When the SEARCHENGINE tag is enabled doxygen will generate a search box for
+# the HTML output. The underlying search engine uses javascript and DHTML and
+# should work on any modern browser. Note that when using HTML help
+# (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets (GENERATE_DOCSET)
+# there is already a search function so this one should typically be disabled.
+# For large projects the javascript based search engine can be slow, then
+# enabling SERVER_BASED_SEARCH may provide a better solution. It is possible to
+# search using the keyboard; to jump to the search box use <access key> + S
+# (what the <access key> is depends on the OS and browser, but it is typically
+# <CTRL>, <ALT>/<option>, or both). Inside the search box use the <cursor down
+# key> to jump into the search results window, the results can be navigated
+# using the <cursor keys>. Press <Enter> to select an item or <escape> to cancel
+# the search. The filter options can be selected when the cursor is inside the
+# search box by pressing <Shift>+<cursor down>. Also here use the <cursor keys>
+# to select a filter and <Enter> or <escape> to activate or cancel the filter
+# option.
+# The default value is: YES.
+# This tag requires that the tag GENERATE_HTML is set to YES.
+
+SEARCHENGINE           = YES
+
+# When the SERVER_BASED_SEARCH tag is enabled the search engine will be
+# implemented using a web server instead of a web client using Javascript. There
+# are two flavors of web server based searching depending on the EXTERNAL_SEARCH
+# setting. When disabled, doxygen will generate a PHP script for searching and
+# an index file used by the script. When EXTERNAL_SEARCH is enabled the indexing
+# and searching needs to be provided by external tools. See the section
+# "External Indexing and Searching" for details.
+# The default value is: NO.
+# This tag requires that the tag SEARCHENGINE is set to YES.
+
+SERVER_BASED_SEARCH    = NO
+
+# When EXTERNAL_SEARCH tag is enabled doxygen will no longer generate the PHP
+# script for searching. Instead the search results are written to an XML file
+# which needs to be processed by an external indexer. Doxygen will invoke an
+# external search engine pointed to by the SEARCHENGINE_URL option to obtain the
+# search results.
+#
+# Doxygen ships with an example indexer (doxyindexer) and search engine
+# (doxysearch.cgi) which are based on the open source search engine library
+# Xapian (see: http://xapian.org/).
+#
+# See the section "External Indexing and Searching" for details.
+# The default value is: NO.
+# This tag requires that the tag SEARCHENGINE is set to YES.
+
+EXTERNAL_SEARCH        = NO
+
+# The SEARCHENGINE_URL should point to a search engine hosted by a web server
+# which will return the search results when EXTERNAL_SEARCH is enabled.
+#
+# Doxygen ships with an example indexer (doxyindexer) and search engine
+# (doxysearch.cgi) which are based on the open source search engine library
+# Xapian (see: http://xapian.org/). See the section "External Indexing and
+# Searching" for details.
+# This tag requires that the tag SEARCHENGINE is set to YES.
+
+SEARCHENGINE_URL       =
+
+# When SERVER_BASED_SEARCH and EXTERNAL_SEARCH are both enabled the unindexed
+# search data is written to a file for indexing by an external tool. With the
+# SEARCHDATA_FILE tag the name of this file can be specified.
+# The default file is: searchdata.xml.
+# This tag requires that the tag SEARCHENGINE is set to YES.
+
+SEARCHDATA_FILE        = searchdata.xml
+
+# When SERVER_BASED_SEARCH and EXTERNAL_SEARCH are both enabled the
+# EXTERNAL_SEARCH_ID tag can be used as an identifier for the project. This is
+# useful in combination with EXTRA_SEARCH_MAPPINGS to search through multiple
+# projects and redirect the results back to the right project.
+# This tag requires that the tag SEARCHENGINE is set to YES.
+
+EXTERNAL_SEARCH_ID     =
+
+# The EXTRA_SEARCH_MAPPINGS tag can be used to enable searching through doxygen
+# projects other than the one defined by this configuration file, but that are
+# all added to the same external search index. Each project needs to have a
+# unique id set via EXTERNAL_SEARCH_ID. The search mapping then maps the id of
+# to a relative location where the documentation can be found. The format is:
+# EXTRA_SEARCH_MAPPINGS = tagname1=loc1 tagname2=loc2 ...
+# This tag requires that the tag SEARCHENGINE is set to YES.
+
+EXTRA_SEARCH_MAPPINGS  =
+
+#---------------------------------------------------------------------------
+# Configuration options related to the LaTeX output
+#---------------------------------------------------------------------------
+
+# If the GENERATE_LATEX tag is set to YES, doxygen will generate LaTeX output.
+# The default value is: YES.
+
+GENERATE_LATEX         = YES
+
+# The LATEX_OUTPUT tag is used to specify where the LaTeX docs will be put. If a
+# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
+# it.
+# The default directory is: latex.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_OUTPUT           = latex
+
+# The LATEX_CMD_NAME tag can be used to specify the LaTeX command name to be
+# invoked.
+#
+# Note that when enabling USE_PDFLATEX this option is only used for generating
+# bitmaps for formulas in the HTML output, but not in the Makefile that is
+# written to the output directory.
+# The default file is: latex.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_CMD_NAME         = latex
+
+# The MAKEINDEX_CMD_NAME tag can be used to specify the command name to generate
+# index for LaTeX.
+# The default file is: makeindex.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+MAKEINDEX_CMD_NAME     = makeindex
+
+# If the COMPACT_LATEX tag is set to YES, doxygen generates more compact LaTeX
+# documents. This may be useful for small projects and may help to save some
+# trees in general.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+COMPACT_LATEX          = NO
+
+# The PAPER_TYPE tag can be used to set the paper type that is used by the
+# printer.
+# Possible values are: a4 (210 x 297 mm), letter (8.5 x 11 inches), legal (8.5 x
+# 14 inches) and executive (7.25 x 10.5 inches).
+# The default value is: a4.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+PAPER_TYPE             = a4
+
+# The EXTRA_PACKAGES tag can be used to specify one or more LaTeX package names
+# that should be included in the LaTeX output. The package can be specified just
+# by its name or with the correct syntax as to be used with the LaTeX
+# \usepackage command. To get the times font for instance you can specify :
+# EXTRA_PACKAGES=times or EXTRA_PACKAGES={times}
+# To use the option intlimits with the amsmath package you can specify:
+# EXTRA_PACKAGES=[intlimits]{amsmath}
+# If left blank no extra packages will be included.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+EXTRA_PACKAGES         =
+
+# The LATEX_HEADER tag can be used to specify a personal LaTeX header for the
+# generated LaTeX document. The header should contain everything until the first
+# chapter. If it is left blank doxygen will generate a standard header. See
+# section "Doxygen usage" for information on how to let doxygen write the
+# default header to a separate file.
+#
+# Note: Only use a user-defined header if you know what you are doing! The
+# following commands have a special meaning inside the header: $title,
+# $datetime, $date, $doxygenversion, $projectname, $projectnumber,
+# $projectbrief, $projectlogo. Doxygen will replace $title with the empty
+# string, for the replacement values of the other commands the user is referred
+# to HTML_HEADER.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_HEADER           =
+
+# The LATEX_FOOTER tag can be used to specify a personal LaTeX footer for the
+# generated LaTeX document. The footer should contain everything after the last
+# chapter. If it is left blank doxygen will generate a standard footer. See
+# LATEX_HEADER for more information on how to generate a default footer and what
+# special commands can be used inside the footer.
+#
+# Note: Only use a user-defined footer if you know what you are doing!
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_FOOTER           =
+
+# The LATEX_EXTRA_STYLESHEET tag can be used to specify additional user-defined
+# LaTeX style sheets that are included after the standard style sheets created
+# by doxygen. Using this option one can overrule certain style aspects. Doxygen
+# will copy the style sheet files to the output directory.
+# Note: The order of the extra style sheet files is of importance (e.g. the last
+# style sheet in the list overrules the setting of the previous ones in the
+# list).
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_EXTRA_STYLESHEET =
+
+# The LATEX_EXTRA_FILES tag can be used to specify one or more extra images or
+# other source files which should be copied to the LATEX_OUTPUT output
+# directory. Note that the files will be copied as-is; there are no commands or
+# markers available.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_EXTRA_FILES      =
+
+# If the PDF_HYPERLINKS tag is set to YES, the LaTeX that is generated is
+# prepared for conversion to PDF (using ps2pdf or pdflatex). The PDF file will
+# contain links (just like the HTML output) instead of page references. This
+# makes the output suitable for online browsing using a PDF viewer.
+# The default value is: YES.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+PDF_HYPERLINKS         = YES
+
+# If the USE_PDFLATEX tag is set to YES, doxygen will use pdflatex to generate
+# the PDF file directly from the LaTeX files. Set this option to YES, to get a
+# higher quality PDF documentation.
+# The default value is: YES.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+USE_PDFLATEX           = YES
+
+# If the LATEX_BATCHMODE tag is set to YES, doxygen will add the \batchmode
+# command to the generated LaTeX files. This will instruct LaTeX to keep running
+# if errors occur, instead of asking the user for help. This option is also used
+# when generating formulas in HTML.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_BATCHMODE        = NO
+
+# If the LATEX_HIDE_INDICES tag is set to YES then doxygen will not include the
+# index chapters (such as File Index, Compound Index, etc.) in the output.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_HIDE_INDICES     = NO
+
+# If the LATEX_SOURCE_CODE tag is set to YES then doxygen will include source
+# code with syntax highlighting in the LaTeX output.
+#
+# Note that which sources are shown also depends on other settings such as
+# SOURCE_BROWSER.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_SOURCE_CODE      = NO
+
+# The LATEX_BIB_STYLE tag can be used to specify the style to use for the
+# bibliography, e.g. plainnat, or ieeetr. See
+# http://en.wikipedia.org/wiki/BibTeX and \cite for more info.
+# The default value is: plain.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_BIB_STYLE        = plain
+
+# If the LATEX_TIMESTAMP tag is set to YES then the footer of each generated
+# page will contain the date and time when the page was generated. Setting this
+# to NO can help when comparing the output of multiple runs.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_LATEX is set to YES.
+
+LATEX_TIMESTAMP        = NO
+
+#---------------------------------------------------------------------------
+# Configuration options related to the RTF output
+#---------------------------------------------------------------------------
+
+# If the GENERATE_RTF tag is set to YES, doxygen will generate RTF output. The
+# RTF output is optimized for Word 97 and may not look too pretty with other RTF
+# readers/editors.
+# The default value is: NO.
+
+GENERATE_RTF           = NO
+
+# The RTF_OUTPUT tag is used to specify where the RTF docs will be put. If a
+# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
+# it.
+# The default directory is: rtf.
+# This tag requires that the tag GENERATE_RTF is set to YES.
+
+RTF_OUTPUT             = rtf
+
+# If the COMPACT_RTF tag is set to YES, doxygen generates more compact RTF
+# documents. This may be useful for small projects and may help to save some
+# trees in general.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_RTF is set to YES.
+
+COMPACT_RTF            = NO
+
+# If the RTF_HYPERLINKS tag is set to YES, the RTF that is generated will
+# contain hyperlink fields. The RTF file will contain links (just like the HTML
+# output) instead of page references. This makes the output suitable for online
+# browsing using Word or some other Word compatible readers that support those
+# fields.
+#
+# Note: WordPad (write) and others do not support links.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_RTF is set to YES.
+
+RTF_HYPERLINKS         = NO
+
+# Load stylesheet definitions from file. Syntax is similar to doxygen's config
+# file, i.e. a series of assignments. You only have to provide replacements,
+# missing definitions are set to their default value.
+#
+# See also section "Doxygen usage" for information on how to generate the
+# default style sheet that doxygen normally uses.
+# This tag requires that the tag GENERATE_RTF is set to YES.
+
+RTF_STYLESHEET_FILE    =
+
+# Set optional variables used in the generation of an RTF document. Syntax is
+# similar to doxygen's config file. A template extensions file can be generated
+# using doxygen -e rtf extensionFile.
+# This tag requires that the tag GENERATE_RTF is set to YES.
+
+RTF_EXTENSIONS_FILE    =
+
+# If the RTF_SOURCE_CODE tag is set to YES then doxygen will include source code
+# with syntax highlighting in the RTF output.
+#
+# Note that which sources are shown also depends on other settings such as
+# SOURCE_BROWSER.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_RTF is set to YES.
+
+RTF_SOURCE_CODE        = NO
+
+#---------------------------------------------------------------------------
+# Configuration options related to the man page output
+#---------------------------------------------------------------------------
+
+# If the GENERATE_MAN tag is set to YES, doxygen will generate man pages for
+# classes and files.
+# The default value is: NO.
+
+GENERATE_MAN           = NO
+
+# The MAN_OUTPUT tag is used to specify where the man pages will be put. If a
+# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
+# it. A directory man3 will be created inside the directory specified by
+# MAN_OUTPUT.
+# The default directory is: man.
+# This tag requires that the tag GENERATE_MAN is set to YES.
+
+MAN_OUTPUT             = man
+
+# The MAN_EXTENSION tag determines the extension that is added to the generated
+# man pages. In case the manual section does not start with a number, the number
+# 3 is prepended. The dot (.) at the beginning of the MAN_EXTENSION tag is
+# optional.
+# The default value is: .3.
+# This tag requires that the tag GENERATE_MAN is set to YES.
+
+MAN_EXTENSION          = .3
+
+# The MAN_SUBDIR tag determines the name of the directory created within
+# MAN_OUTPUT in which the man pages are placed. If defaults to man followed by
+# MAN_EXTENSION with the initial . removed.
+# This tag requires that the tag GENERATE_MAN is set to YES.
+
+MAN_SUBDIR             =
+
+# If the MAN_LINKS tag is set to YES and doxygen generates man output, then it
+# will generate one additional man file for each entity documented in the real
+# man page(s). These additional files only source the real man page, but without
+# them the man command would be unable to find the correct page.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_MAN is set to YES.
+
+MAN_LINKS              = NO
+
+#---------------------------------------------------------------------------
+# Configuration options related to the XML output
+#---------------------------------------------------------------------------
+
+# If the GENERATE_XML tag is set to YES, doxygen will generate an XML file that
+# captures the structure of the code including all documentation.
+# The default value is: NO.
+
+GENERATE_XML           = NO
+
+# The XML_OUTPUT tag is used to specify where the XML pages will be put. If a
+# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
+# it.
+# The default directory is: xml.
+# This tag requires that the tag GENERATE_XML is set to YES.
+
+XML_OUTPUT             = xml
+
+# If the XML_PROGRAMLISTING tag is set to YES, doxygen will dump the program
+# listings (including syntax highlighting and cross-referencing information) to
+# the XML output. Note that enabling this will significantly increase the size
+# of the XML output.
+# The default value is: YES.
+# This tag requires that the tag GENERATE_XML is set to YES.
+
+XML_PROGRAMLISTING     = YES
+
+#---------------------------------------------------------------------------
+# Configuration options related to the DOCBOOK output
+#---------------------------------------------------------------------------
+
+# If the GENERATE_DOCBOOK tag is set to YES, doxygen will generate Docbook files
+# that can be used to generate PDF.
+# The default value is: NO.
+
+GENERATE_DOCBOOK       = NO
+
+# The DOCBOOK_OUTPUT tag is used to specify where the Docbook pages will be put.
+# If a relative path is entered the value of OUTPUT_DIRECTORY will be put in
+# front of it.
+# The default directory is: docbook.
+# This tag requires that the tag GENERATE_DOCBOOK is set to YES.
+
+DOCBOOK_OUTPUT         = docbook
+
+# If the DOCBOOK_PROGRAMLISTING tag is set to YES, doxygen will include the
+# program listings (including syntax highlighting and cross-referencing
+# information) to the DOCBOOK output. Note that enabling this will significantly
+# increase the size of the DOCBOOK output.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_DOCBOOK is set to YES.
+
+DOCBOOK_PROGRAMLISTING = NO
+
+#---------------------------------------------------------------------------
+# Configuration options for the AutoGen Definitions output
+#---------------------------------------------------------------------------
+
+# If the GENERATE_AUTOGEN_DEF tag is set to YES, doxygen will generate an
+# AutoGen Definitions (see http://autogen.sf.net) file that captures the
+# structure of the code including all documentation. Note that this feature is
+# still experimental and incomplete at the moment.
+# The default value is: NO.
+
+GENERATE_AUTOGEN_DEF   = NO
+
+#---------------------------------------------------------------------------
+# Configuration options related to the Perl module output
+#---------------------------------------------------------------------------
+
+# If the GENERATE_PERLMOD tag is set to YES, doxygen will generate a Perl module
+# file that captures the structure of the code including all documentation.
+#
+# Note that this feature is still experimental and incomplete at the moment.
+# The default value is: NO.
+
+GENERATE_PERLMOD       = NO
+
+# If the PERLMOD_LATEX tag is set to YES, doxygen will generate the necessary
+# Makefile rules, Perl scripts and LaTeX code to be able to generate PDF and DVI
+# output from the Perl module output.
+# The default value is: NO.
+# This tag requires that the tag GENERATE_PERLMOD is set to YES.
+
+PERLMOD_LATEX          = NO
+
+# If the PERLMOD_PRETTY tag is set to YES, the Perl module output will be nicely
+# formatted so it can be parsed by a human reader. This is useful if you want to
+# understand what is going on. On the other hand, if this tag is set to NO, the
+# size of the Perl module output will be much smaller and Perl will parse it
+# just the same.
+# The default value is: YES.
+# This tag requires that the tag GENERATE_PERLMOD is set to YES.
+
+PERLMOD_PRETTY         = YES
+
+# The names of the make variables in the generated doxyrules.make file are
+# prefixed with the string contained in PERLMOD_MAKEVAR_PREFIX. This is useful
+# so different doxyrules.make files included by the same Makefile don't
+# overwrite each other's variables.
+# This tag requires that the tag GENERATE_PERLMOD is set to YES.
+
+PERLMOD_MAKEVAR_PREFIX =
+
+#---------------------------------------------------------------------------
+# Configuration options related to the preprocessor
+#---------------------------------------------------------------------------
+
+# If the ENABLE_PREPROCESSING tag is set to YES, doxygen will evaluate all
+# C-preprocessor directives found in the sources and include files.
+# The default value is: YES.
+
+ENABLE_PREPROCESSING   = YES
+
+# If the MACRO_EXPANSION tag is set to YES, doxygen will expand all macro names
+# in the source code. If set to NO, only conditional compilation will be
+# performed. Macro expansion can be done in a controlled way by setting
+# EXPAND_ONLY_PREDEF to YES.
+# The default value is: NO.
+# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
+
+MACRO_EXPANSION        = NO
+
+# If the EXPAND_ONLY_PREDEF and MACRO_EXPANSION tags are both set to YES then
+# the macro expansion is limited to the macros specified with the PREDEFINED and
+# EXPAND_AS_DEFINED tags.
+# The default value is: NO.
+# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
+
+EXPAND_ONLY_PREDEF     = NO
+
+# If the SEARCH_INCLUDES tag is set to YES, the include files in the
+# INCLUDE_PATH will be searched if a #include is found.
+# The default value is: YES.
+# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
+
+SEARCH_INCLUDES        = YES
+
+# The INCLUDE_PATH tag can be used to specify one or more directories that
+# contain include files that are not input files but should be processed by the
+# preprocessor.
+# This tag requires that the tag SEARCH_INCLUDES is set to YES.
+
+INCLUDE_PATH           =
+
+# You can use the INCLUDE_FILE_PATTERNS tag to specify one or more wildcard
+# patterns (like *.h and *.hpp) to filter out the header-files in the
+# directories. If left blank, the patterns specified with FILE_PATTERNS will be
+# used.
+# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
+
+INCLUDE_FILE_PATTERNS  =
+
+# The PREDEFINED tag can be used to specify one or more macro names that are
+# defined before the preprocessor is started (similar to the -D option of e.g.
+# gcc). The argument of the tag is a list of macros of the form: name or
+# name=definition (no spaces). If the definition and the "=" are omitted, "=1"
+# is assumed. To prevent a macro definition from being undefined via #undef or
+# recursively expanded use the := operator instead of the = operator.
+# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
+
+PREDEFINED             =
+
+# If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then this
+# tag can be used to specify a list of macro names that should be expanded. The
+# macro definition that is found in the sources will be used. Use the PREDEFINED
+# tag if you want to use a different macro definition that overrules the
+# definition found in the source code.
+# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
+
+EXPAND_AS_DEFINED      =
+
+# If the SKIP_FUNCTION_MACROS tag is set to YES then doxygen's preprocessor will
+# remove all references to function-like macros that are alone on a line, have
+# an all uppercase name, and do not end with a semicolon. Such function macros
+# are typically used for boiler-plate code, and will confuse the parser if not
+# removed.
+# The default value is: YES.
+# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
+
+SKIP_FUNCTION_MACROS   = YES
+
+#---------------------------------------------------------------------------
+# Configuration options related to external references
+#---------------------------------------------------------------------------
+
+# The TAGFILES tag can be used to specify one or more tag files. For each tag
+# file the location of the external documentation should be added. The format of
+# a tag file without this location is as follows:
+# TAGFILES = file1 file2 ...
+# Adding location for the tag files is done as follows:
+# TAGFILES = file1=loc1 "file2 = loc2" ...
+# where loc1 and loc2 can be relative or absolute paths or URLs. See the
+# section "Linking to external documentation" for more information about the use
+# of tag files.
+# Note: Each tag file must have a unique name (where the name does NOT include
+# the path). If a tag file is not located in the directory in which doxygen is
+# run, you must also specify the path to the tagfile here.
+
+TAGFILES               =
+
+# When a file name is specified after GENERATE_TAGFILE, doxygen will create a
+# tag file that is based on the input files it reads. See section "Linking to
+# external documentation" for more information about the usage of tag files.
+
+GENERATE_TAGFILE       =
+
+# If the ALLEXTERNALS tag is set to YES, all external class will be listed in
+# the class index. If set to NO, only the inherited external classes will be
+# listed.
+# The default value is: NO.
+
+ALLEXTERNALS           = NO
+
+# If the EXTERNAL_GROUPS tag is set to YES, all external groups will be listed
+# in the modules index. If set to NO, only the current project's groups will be
+# listed.
+# The default value is: YES.
+
+EXTERNAL_GROUPS        = YES
+
+# If the EXTERNAL_PAGES tag is set to YES, all external pages will be listed in
+# the related pages index. If set to NO, only the current project's pages will
+# be listed.
+# The default value is: YES.
+
+EXTERNAL_PAGES         = YES
+
+# The PERL_PATH should be the absolute path and name of the perl script
+# interpreter (i.e. the result of 'which perl').
+# The default file (with absolute path) is: /usr/bin/perl.
+
+PERL_PATH              = /usr/bin/perl
+
+#---------------------------------------------------------------------------
+# Configuration options related to the dot tool
+#---------------------------------------------------------------------------
+
+# If the CLASS_DIAGRAMS tag is set to YES, doxygen will generate a class diagram
+# (in HTML and LaTeX) for classes with base or super classes. Setting the tag to
+# NO turns the diagrams off. Note that this option also works with HAVE_DOT
+# disabled, but it is recommended to install and use dot, since it yields more
+# powerful graphs.
+# The default value is: YES.
+
+CLASS_DIAGRAMS         = YES
+
+# You can define message sequence charts within doxygen comments using the \msc
+# command. Doxygen will then run the mscgen tool (see:
+# http://www.mcternan.me.uk/mscgen/)) to produce the chart and insert it in the
+# documentation. The MSCGEN_PATH tag allows you to specify the directory where
+# the mscgen tool resides. If left empty the tool is assumed to be found in the
+# default search path.
+
+MSCGEN_PATH            =
+
+# You can include diagrams made with dia in doxygen documentation. Doxygen will
+# then run dia to produce the diagram and insert it in the documentation. The
+# DIA_PATH tag allows you to specify the directory where the dia binary resides.
+# If left empty dia is assumed to be found in the default search path.
+
+DIA_PATH               =
+
+# If set to YES the inheritance and collaboration graphs will hide inheritance
+# and usage relations if the target is undocumented or is not a class.
+# The default value is: YES.
+
+HIDE_UNDOC_RELATIONS   = YES
+
+# If you set the HAVE_DOT tag to YES then doxygen will assume the dot tool is
+# available from the path. This tool is part of Graphviz (see:
+# http://www.graphviz.org/), a graph visualization toolkit from AT&T and Lucent
+# Bell Labs. The other options in this section have no effect if this option is
+# set to NO
+# The default value is: YES.
+
+HAVE_DOT               = YES
+
+# The DOT_NUM_THREADS specifies the number of dot invocations doxygen is allowed
+# to run in parallel. When set to 0 doxygen will base this on the number of
+# processors available in the system. You can set it explicitly to a value
+# larger than 0 to get control over the balance between CPU load and processing
+# speed.
+# Minimum value: 0, maximum value: 32, default value: 0.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_NUM_THREADS        = 0
+
+# When you want a differently looking font in the dot files that doxygen
+# generates you can specify the font name using DOT_FONTNAME. You need to make
+# sure dot is able to find the font, which can be done by putting it in a
+# standard location or by setting the DOTFONTPATH environment variable or by
+# setting DOT_FONTPATH to the directory containing the font.
+# The default value is: Helvetica.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_FONTNAME           = Helvetica
+
+# The DOT_FONTSIZE tag can be used to set the size (in points) of the font of
+# dot graphs.
+# Minimum value: 4, maximum value: 24, default value: 10.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_FONTSIZE           = 10
+
+# By default doxygen will tell dot to use the default font as specified with
+# DOT_FONTNAME. If you specify a different font using DOT_FONTNAME you can set
+# the path where dot can find it using this tag.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_FONTPATH           =
+
+# If the CLASS_GRAPH tag is set to YES then doxygen will generate a graph for
+# each documented class showing the direct and indirect inheritance relations.
+# Setting this tag to YES will force the CLASS_DIAGRAMS tag to NO.
+# The default value is: YES.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+CLASS_GRAPH            = YES
+
+# If the COLLABORATION_GRAPH tag is set to YES then doxygen will generate a
+# graph for each documented class showing the direct and indirect implementation
+# dependencies (inheritance, containment, and class references variables) of the
+# class with other documented classes.
+# The default value is: YES.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+COLLABORATION_GRAPH    = YES
+
+# If the GROUP_GRAPHS tag is set to YES then doxygen will generate a graph for
+# groups, showing the direct groups dependencies.
+# The default value is: YES.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+GROUP_GRAPHS           = YES
+
+# If the UML_LOOK tag is set to YES, doxygen will generate inheritance and
+# collaboration diagrams in a style similar to the OMG's Unified Modeling
+# Language.
+# The default value is: NO.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+UML_LOOK               = NO
+
+# If the UML_LOOK tag is enabled, the fields and methods are shown inside the
+# class node. If there are many fields or methods and many nodes the graph may
+# become too big to be useful. The UML_LIMIT_NUM_FIELDS threshold limits the
+# number of items for each type to make the size more manageable. Set this to 0
+# for no limit. Note that the threshold may be exceeded by 50% before the limit
+# is enforced. So when you set the threshold to 10, up to 15 fields may appear,
+# but if the number exceeds 15, the total amount of fields shown is limited to
+# 10.
+# Minimum value: 0, maximum value: 100, default value: 10.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+UML_LIMIT_NUM_FIELDS   = 10
+
+# If the TEMPLATE_RELATIONS tag is set to YES then the inheritance and
+# collaboration graphs will show the relations between templates and their
+# instances.
+# The default value is: NO.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+TEMPLATE_RELATIONS     = NO
+
+# If the INCLUDE_GRAPH, ENABLE_PREPROCESSING and SEARCH_INCLUDES tags are set to
+# YES then doxygen will generate a graph for each documented file showing the
+# direct and indirect include dependencies of the file with other documented
+# files.
+# The default value is: YES.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+INCLUDE_GRAPH          = YES
+
+# If the INCLUDED_BY_GRAPH, ENABLE_PREPROCESSING and SEARCH_INCLUDES tags are
+# set to YES then doxygen will generate a graph for each documented file showing
+# the direct and indirect include dependencies of the file with other documented
+# files.
+# The default value is: YES.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+INCLUDED_BY_GRAPH      = YES
+
+# If the CALL_GRAPH tag is set to YES then doxygen will generate a call
+# dependency graph for every global function or class method.
+#
+# Note that enabling this option will significantly increase the time of a run.
+# So in most cases it will be better to enable call graphs for selected
+# functions only using the \callgraph command. Disabling a call graph can be
+# accomplished by means of the command \hidecallgraph.
+# The default value is: NO.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+CALL_GRAPH             = NO
+
+# If the CALLER_GRAPH tag is set to YES then doxygen will generate a caller
+# dependency graph for every global function or class method.
+#
+# Note that enabling this option will significantly increase the time of a run.
+# So in most cases it will be better to enable caller graphs for selected
+# functions only using the \callergraph command. Disabling a caller graph can be
+# accomplished by means of the command \hidecallergraph.
+# The default value is: NO.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+CALLER_GRAPH           = NO
+
+# If the GRAPHICAL_HIERARCHY tag is set to YES then doxygen will graphical
+# hierarchy of all classes instead of a textual one.
+# The default value is: YES.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+GRAPHICAL_HIERARCHY    = YES
+
+# If the DIRECTORY_GRAPH tag is set to YES then doxygen will show the
+# dependencies a directory has on other directories in a graphical way. The
+# dependency relations are determined by the #include relations between the
+# files in the directories.
+# The default value is: YES.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DIRECTORY_GRAPH        = YES
+
+# The DOT_IMAGE_FORMAT tag can be used to set the image format of the images
+# generated by dot. For an explanation of the image formats see the section
+# output formats in the documentation of the dot tool (Graphviz (see:
+# http://www.graphviz.org/)).
+# Note: If you choose svg you need to set HTML_FILE_EXTENSION to xhtml in order
+# to make the SVG files visible in IE 9+ (other browsers do not have this
+# requirement).
+# Possible values are: png, png:cairo, png:cairo:cairo, png:cairo:gd, png:gd,
+# png:gd:gd, jpg, jpg:cairo, jpg:cairo:gd, jpg:gd, jpg:gd:gd, gif, gif:cairo,
+# gif:cairo:gd, gif:gd, gif:gd:gd, svg, png:gd, png:gd:gd, png:cairo,
+# png:cairo:gd, png:cairo:cairo, png:cairo:gdiplus, png:gdiplus and
+# png:gdiplus:gdiplus.
+# The default value is: png.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_IMAGE_FORMAT       = png
+
+# If DOT_IMAGE_FORMAT is set to svg, then this option can be set to YES to
+# enable generation of interactive SVG images that allow zooming and panning.
+#
+# Note that this requires a modern browser other than Internet Explorer. Tested
+# and working are Firefox, Chrome, Safari, and Opera.
+# Note: For IE 9+ you need to set HTML_FILE_EXTENSION to xhtml in order to make
+# the SVG files visible. Older versions of IE do not have SVG support.
+# The default value is: NO.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+INTERACTIVE_SVG        = NO
+
+# The DOT_PATH tag can be used to specify the path where the dot tool can be
+# found. If left blank, it is assumed the dot tool can be found in the path.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_PATH               =
+
+# The DOTFILE_DIRS tag can be used to specify one or more directories that
+# contain dot files that are included in the documentation (see the \dotfile
+# command).
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOTFILE_DIRS           =
+
+# The MSCFILE_DIRS tag can be used to specify one or more directories that
+# contain msc files that are included in the documentation (see the \mscfile
+# command).
+
+MSCFILE_DIRS           =
+
+# The DIAFILE_DIRS tag can be used to specify one or more directories that
+# contain dia files that are included in the documentation (see the \diafile
+# command).
+
+DIAFILE_DIRS           =
+
+# When using plantuml, the PLANTUML_JAR_PATH tag should be used to specify the
+# path where java can find the plantuml.jar file. If left blank, it is assumed
+# PlantUML is not used or called during a preprocessing step. Doxygen will
+# generate a warning when it encounters a \startuml command in this case and
+# will not generate output for the diagram.
+
+PLANTUML_JAR_PATH      =
+
+# When using plantuml, the PLANTUML_CFG_FILE tag can be used to specify a
+# configuration file for plantuml.
+
+PLANTUML_CFG_FILE      =
+
+# When using plantuml, the specified paths are searched for files specified by
+# the !include statement in a plantuml block.
+
+PLANTUML_INCLUDE_PATH  =
+
+# The DOT_GRAPH_MAX_NODES tag can be used to set the maximum number of nodes
+# that will be shown in the graph. If the number of nodes in a graph becomes
+# larger than this value, doxygen will truncate the graph, which is visualized
+# by representing a node as a red box. Note that doxygen if the number of direct
+# children of the root node in a graph is already larger than
+# DOT_GRAPH_MAX_NODES then the graph will not be shown at all. Also note that
+# the size of a graph can be further restricted by MAX_DOT_GRAPH_DEPTH.
+# Minimum value: 0, maximum value: 10000, default value: 50.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_GRAPH_MAX_NODES    = 50
+
+# The MAX_DOT_GRAPH_DEPTH tag can be used to set the maximum depth of the graphs
+# generated by dot. A depth value of 3 means that only nodes reachable from the
+# root by following a path via at most 3 edges will be shown. Nodes that lay
+# further from the root node will be omitted. Note that setting this option to 1
+# or 2 may greatly reduce the computation time needed for large code bases. Also
+# note that the size of a graph can be further restricted by
+# DOT_GRAPH_MAX_NODES. Using a depth of 0 means no depth restriction.
+# Minimum value: 0, maximum value: 1000, default value: 0.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+MAX_DOT_GRAPH_DEPTH    = 0
+
+# Set the DOT_TRANSPARENT tag to YES to generate images with a transparent
+# background. This is disabled by default, because dot on Windows does not seem
+# to support this out of the box.
+#
+# Warning: Depending on the platform used, enabling this option may lead to
+# badly anti-aliased labels on the edges of a graph (i.e. they become hard to
+# read).
+# The default value is: NO.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_TRANSPARENT        = NO
+
+# Set the DOT_MULTI_TARGETS tag to YES to allow dot to generate multiple output
+# files in one run (i.e. multiple -o and -T options on the command line). This
+# makes dot run faster, but since only newer versions of dot (>1.8.10) support
+# this, this feature is disabled by default.
+# The default value is: NO.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_MULTI_TARGETS      = NO
+
+# If the GENERATE_LEGEND tag is set to YES doxygen will generate a legend page
+# explaining the meaning of the various boxes and arrows in the dot generated
+# graphs.
+# The default value is: YES.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+GENERATE_LEGEND        = YES
+
+# If the DOT_CLEANUP tag is set to YES, doxygen will remove the intermediate dot
+# files that are used to generate the various graphs.
+# The default value is: YES.
+# This tag requires that the tag HAVE_DOT is set to YES.
+
+DOT_CLEANUP            = YES
diff --git a/doc/.gitignore b/doc/.gitignore
new file mode 100644
index 000000000..4d192b770
--- /dev/null
+++ b/doc/.gitignore
@@ -0,0 +1,4 @@
+api
+build
+doxygen
+venv
diff --git a/doc/README.md b/doc/README.md
new file mode 100644
index 000000000..87d86ba1c
--- /dev/null
+++ b/doc/README.md
@@ -0,0 +1,51 @@
+# Marian NMT code documentation and library API
+
+This directory contains code documentation and library API for developers of Marian NMT.
+
+The documentation is generated using
+[Sphinx](https://www.sphinx-doc.org/en/master/usage/quickstart.html) +
+[Breathe](https://breathe.readthedocs.io/en/latest/directives.html) +
+[Doxygen](http://www.doxygen.nl/manual/docblocks.html) +
+[Exhale](https://exhale.readthedocs.io/en/latest/usage.html).
+The documentation source code is written in `.rst` or `.md` files with special directives that allow
+to reference to C++ source code and documentation. The source documents are then build into static
+HTML pages.
+
+
+## Installation
+
+On Ubuntu 20.04, install the following packages:
+
+    sudo apt-get install python3 python3-pip python3-setuptools doxygen
+
+Then set up a Python environment and install modules:
+
+    pip3 install virtualenv
+    virtualenv venv -p python3
+    source venv/bin/activate
+    pip install -r requirements.txt
+
+Documentation building should also work on Windows, but it has not been tested.
+
+
+## Generation
+
+The documentation can be generated by running:
+
+    make html
+
+The website will be generated into `build/html` and accessible by opening _index.html_ in your
+browser.
+
+Directories:
+
+- `build` - automatically output directory for HTML documentation
+- `doxygen` - automatically generated Doxygen XML files
+- `api` - automatic library API generated with Exhale
+- `.rst` and `.md` files in this directory and its subdirectories are documentation source files
+- `_static` - custom CSS and JavaScript files
+
+
+## Writing documentation
+
+To be documented...
diff --git a/doc/_static/css/custom.css b/doc/_static/css/custom.css
new file mode 100644
index 000000000..8352655e1
--- /dev/null
+++ b/doc/_static/css/custom.css
@@ -0,0 +1,4 @@
+.wy-body-for-nav > .wy-grid-for-nav > .wy-nav-side {
+    border-bottom: 5px solid #28bbee;
+    /*background-color: #494d55;*/
+}
diff --git a/doc/conf.py b/doc/conf.py
new file mode 100644
index 000000000..a9863cfe5
--- /dev/null
+++ b/doc/conf.py
@@ -0,0 +1,120 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import datetime
+import sys
+
+sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'Bergamot Translator'
+copyright = '2021, Bergamot Translator Team'
+author = 'Bergamot Translator Team'
+
+# The full version, including alpha/beta/rc tags
+# TODO: add GitHub commit hash to the version
+version_file = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'BERGAMOT_VERSION')
+with open(os.path.abspath(version_file)) as f:
+    version = f.read().strip()
+release = version + ' ' + str(datetime.date.today())
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.imgmath',
+    'sphinx.ext.todo',
+    'breathe',
+    'exhale',
+    'recommonmark',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = [
+    'build',
+    'doxygen',
+    'venv',
+    'README.md',
+]
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_rtd_theme'
+htmlhelp_basename = 'bergamot-translator'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+html_css_files = ['css/custom.css']
+
+# The base URL which points to the root of the HTML documentation
+html_baseurl = 'http://jerinphilip.github.io/bergamot-translator'
+
+
+# -- Extension configuration -------------------------------------------------
+
+breathe_projects = { 'bergamot-translator': './doxygen/xml' }
+breathe_default_project = 'bergamot-translator'
+
+doxygen_config = """
+INPUT                = ../src
+EXCLUDE             += ../3rd_party
+EXCLUDE             += ../src/tests
+EXCLUDE_PATTERNS     = *.md *.txt
+FILE_PATTERNS       += *.cu
+EXTENSION_MAPPING   += cu=C++ inc=C++
+ENABLE_PREPROCESSING = YES
+JAVADOC_AUTOBRIEF    = YES
+WARN_IF_UNDOCUMENTED = NO
+"""
+
+exhale_args = {
+    'containmentFolder'     : './api',
+    'rootFileName'          : 'library_index.rst',
+    'rootFileTitle'         : 'Library API',
+    'doxygenStripFromPath'  : '..',
+    'createTreeView'        : True,
+    'exhaleExecutesDoxygen' : True,
+    'exhaleDoxygenStdin'    : doxygen_config.strip(),
+}
+
+primary_domain = 'cpp'
+highlight_language = 'cpp'
+
+# A trick to include markdown files from outside the source directory using
+# 'mdinclude'. Warning: all other markdown files not included via 'mdinclude'
+# will be rendered using recommonmark as recommended by Sphinx
+from m2r import MdInclude
+
+def setup(app):
+    # from m2r to make `mdinclude` work
+    app.add_config_value('no_underscore_emphasis', False, 'env')
+    app.add_config_value('m2r_parse_relative_links', False, 'env')
+    app.add_config_value('m2r_anonymous_references', False, 'env')
+    app.add_config_value('m2r_disable_inline_math', False, 'env')
+    app.add_directive('mdinclude', MdInclude)
diff --git a/doc/index.rst b/doc/index.rst
new file mode 100644
index 000000000..85bc7f109
--- /dev/null
+++ b/doc/index.rst
@@ -0,0 +1,38 @@
+Welcome to Bergamot Translator's documentation!
+===============================================
+
+|buildcpu| |tests| |release| |license|
+
+Bergamot translator provides a unified API for (Marian NMT framework based)
+neural machine translation functionality in accordance with the Bergamot
+project that focuses on improving client-side machine translation in a web
+browser.
+
+This is developer documentation. 
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   marian-integration
+   api/library_index
+
+
+
+Indices and tables
+------------------
+
+* :ref:`genindex`
+
+
+.. |buildcpu| image:: https://img.shields.io/jenkins/s/http/vali.inf.ed.ac.uk/jenkins/view/browsermt/job/bergamot-translator.svg?label=CPU%20Build
+   :target: http://vali.inf.ed.ac.uk/jenkins/job/bergamot-translator
+   :alt: CPU build status
+
+.. |tests| image:: https://img.shields.io/jenkins/s/http/vali.inf.ed.ac.uk/jenkins/view/marian/job/bergamot-translator-regression-tests.svg?label=Tests
+   :target: http://vali.inf.ed.ac.uk/jenkins/job/bergamot-translator-regression-tests/
+   :alt: Tests status
+
+.. |license| image:: https://img.shields.io/badge/License-MPL%202.0-brightgreen.svg
+   :target: https://opensource.org/licenses/MPL-2.0
+   :alt: License: MPL
diff --git a/doc/make.bat b/doc/make.bat
new file mode 100644
index 000000000..6247f7e23
--- /dev/null
+++ b/doc/make.bat
@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd
diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index 101dbb219..50e0648e9 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -1,4 +1,4 @@
-# Marian Integration
+# Building marian code for bergamot
 
 This document summarizes the minimal build instructions develop for the
 marian-code powering bergamot-translator.
diff --git a/doc/references.bib b/doc/references.bib
new file mode 100644
index 000000000..e69de29bb
diff --git a/doc/requirements.txt b/doc/requirements.txt
new file mode 100644
index 000000000..8d56e6839
--- /dev/null
+++ b/doc/requirements.txt
@@ -0,0 +1,6 @@
+sphinx==2.4.4
+breathe==4.13.0
+exhale
+sphinx_rtd_theme
+recommonmark
+m2r

From f38a0bfbcc91648c55909da705b359e08c805adf Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 24 Mar 2021 13:50:28 +0100
Subject: [PATCH 179/442] Remove AbstractTranslationModel class and its
 references

---
 app/bergamot-translator-app-bytearray.cpp   | 10 +--
 app/bergamot-translator-app.cpp             | 10 +--
 src/AbstractTranslationModel.h              | 72 ---------------------
 src/{translator => }/TranslationModel.h     | 11 ++--
 src/TranslationRequest.h                    |  2 +-
 src/TranslationResult.h                     |  4 +-
 src/translator/AbstractTranslationModel.cpp | 18 ------
 src/translator/CMakeLists.txt               |  3 +-
 src/translator/TranslationModel.cpp         |  2 +-
 9 files changed, 17 insertions(+), 115 deletions(-)
 delete mode 100644 src/AbstractTranslationModel.h
 rename src/{translator => }/TranslationModel.h (92%)
 delete mode 100644 src/translator/AbstractTranslationModel.cpp

diff --git a/app/bergamot-translator-app-bytearray.cpp b/app/bergamot-translator-app-bytearray.cpp
index 961a24e2f..b58c638a3 100644
--- a/app/bergamot-translator-app-bytearray.cpp
+++ b/app/bergamot-translator-app-bytearray.cpp
@@ -7,9 +7,7 @@
 
 #include <iostream>
 
-#include "AbstractTranslationModel.h"
-#include "TranslationRequest.h"
-#include "TranslationResult.h"
+#include "TranslationModel.h"
 #include "translator/parser.h"
 #include "translator/byteArrayExample.h"
 
@@ -21,11 +19,9 @@ int main(int argc, char **argv) {
   auto options = configParser.parseOptions(argc, argv, true);
   std::string config = options->asYamlString();
 
-  // Route the config string to construct marian model through
-  // AbstractTranslationModel
+  // Route the config string to construct marian model through TranslationModel
   void * model_bytes = bergamot::getBinaryModelFromConfig(options);
-  std::shared_ptr<AbstractTranslationModel> model =
-      AbstractTranslationModel::createInstance(config, model_bytes);
+  auto model = std::make_shared<TranslationModel>(config, model_bytes);
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
diff --git a/app/bergamot-translator-app.cpp b/app/bergamot-translator-app.cpp
index 2f67feb9c..a71abbcac 100644
--- a/app/bergamot-translator-app.cpp
+++ b/app/bergamot-translator-app.cpp
@@ -7,9 +7,7 @@
 
 #include <iostream>
 
-#include "AbstractTranslationModel.h"
-#include "TranslationRequest.h"
-#include "TranslationResult.h"
+#include "TranslationModel.h"
 #include "translator/parser.h"
 
 int main(int argc, char **argv) {
@@ -20,10 +18,8 @@ int main(int argc, char **argv) {
   auto options = configParser.parseOptions(argc, argv, true);
   std::string config = options->asYamlString();
 
-  // Route the config string to construct marian model through
-  // AbstractTranslationModel
-  std::shared_ptr<AbstractTranslationModel> model =
-      AbstractTranslationModel::createInstance(config);
+  // Route the config string to construct marian model through TranslationModel
+  auto model = std::make_shared<TranslationModel>(config);
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
diff --git a/src/AbstractTranslationModel.h b/src/AbstractTranslationModel.h
deleted file mode 100644
index 82dcbfa09..000000000
--- a/src/AbstractTranslationModel.h
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * AbstractTranslationModel.h
- *
- * An interface for a translation model for translating a plain (without any
- * markups and emojis) UTF-8 encoded text. The model supports translation from 1
- * source language to 1 target language. There can be different implementations
- * of this interface.
- */
-
-#ifndef SRC_TRANSLATOR_ABSTRACTTRANSLATIONMODEL_H_
-#define SRC_TRANSLATOR_ABSTRACTTRANSLATIONMODEL_H_
-
-#include <future>
-#include <memory>
-#include <string>
-#include <vector>
-
-#include "TranslationRequest.h"
-#include "TranslationResult.h"
-
-/* An interface for a translation model for translating a plain (without any
- * markups and emojis) UTF-8 encoded text. The model supports translation from 1
- * source language to 1 target language.
- */
-class AbstractTranslationModel {
-public:
-  /* A Factory method to create and return an instance of an implementation of
-   * AbstractTranslationModel. The instance is created using translation model
-   * configuration provided as yaml-formatted string.
-   */
-  /**
-   * @param config Marian yml config file in the form of a string
-   * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Optional, defaults to nullptr when not used
-   */
-  static std::shared_ptr<AbstractTranslationModel>
-  createInstance(const std::string &config, const void * model_memory=nullptr);
-
-  AbstractTranslationModel() = default;
-
-  virtual ~AbstractTranslationModel() = default;
-
-  /* This method performs translation on a list of (UTF-8 encoded) texts and
-   * returns a list of results in the same order. Each text entry can either be
-   * a word, a phrase, a sentence or a list of sentences and should contain
-   * plain text (without any markups or emojis). Additional information related
-   * to the translated text can be requested via TranslationRequest which is
-   * applied equally to each text entry.
-   *
-   * The translated text corresponding to each text entry and the additional
-   * information (as specified in the TranslationRequest) is encapsulated and
-   * returned in TranslationResult.
-   *
-   * The API splits each text entry into sentences internally, which are then
-   * translated independent of each other. The translated sentences are then
-   * joined together and returned in TranslationResult. Please refer to the
-   * TranslationRequest class to find out what additional information can be
-   * requested. The alignment information can only be requested if the model
-   * supports it (check isAlignmentSupported() API).
-   *
-   * The texts argument will become empty after the execution of this API (each
-   * entry of texts list will be moved to its corresponding TranslationResult
-   * object).
-   */
-  virtual std::vector<TranslationResult>
-  translate(std::vector<std::string> &&texts, TranslationRequest request) = 0;
-
-  /* Check if the model can provide alignment information b/w original and
-   * translated text. */
-  virtual bool isAlignmentSupported() const = 0;
-};
-
-#endif /* SRC_TRANSLATOR_ABSTRACTTRANSLATIONMODEL_H_ */
diff --git a/src/translator/TranslationModel.h b/src/TranslationModel.h
similarity index 92%
rename from src/translator/TranslationModel.h
rename to src/TranslationModel.h
index 224d6b0a1..db0bdbf5c 100644
--- a/src/translator/TranslationModel.h
+++ b/src/TranslationModel.h
@@ -1,7 +1,7 @@
 /*
  * TranslationModel.h
  *
- *  A implementation of AbstractTranslationModel interface.
+ * Main interface for translation API.
  */
 
 #ifndef SRC_TRANSLATOR_TRANSLATIONMODEL_H_
@@ -15,14 +15,15 @@
 #include "3rd_party/marian-dev/src/common/options.h"
 
 // All local project includes
-#include "AbstractTranslationModel.h"
+#include "TranslationRequest.h"
+#include "TranslationResult.h"
 #include "translator/service.h"
 
 /* A Translation model that translates a plain (without any markups and emojis)
  * UTF-8 encoded text. This implementation supports translation from 1 source
  * language to 1 target language.
  */
-class TranslationModel : public AbstractTranslationModel {
+class TranslationModel {
 public:
   /* Construct the model using the model configuration options as yaml-formatted
    * string
@@ -62,11 +63,11 @@ class TranslationModel : public AbstractTranslationModel {
    * object).
    */
   std::vector<TranslationResult> translate(std::vector<std::string> &&texts,
-                                           TranslationRequest request) override;
+                                           TranslationRequest request);
 
   /* Check if the model can provide alignment information b/w original and
    * translated text. */
-  bool isAlignmentSupported() const override;
+  bool isAlignmentSupported() const;
 
 private:
   // Model configuration options
diff --git a/src/TranslationRequest.h b/src/TranslationRequest.h
index 6d449bbab..95289dd1f 100644
--- a/src/TranslationRequest.h
+++ b/src/TranslationRequest.h
@@ -2,7 +2,7 @@
  * TranslationRequest.h
  *
  *  This file defines the translation request class to be used in
- * AbstractTranslationModel::translate() API.
+ *  TranslationModel::translate() API.
  */
 
 #ifndef SRC_TRANSLATOR_TRANSLATIONREQUEST_H_
diff --git a/src/TranslationResult.h b/src/TranslationResult.h
index b4867af65..8c6c806e4 100644
--- a/src/TranslationResult.h
+++ b/src/TranslationResult.h
@@ -1,7 +1,7 @@
 /*
  * TranslationResult.h
  *
- * The class that represents the result of AbstractTranslationModel::translate()
+ * The class that represents the result of TranslationModel::translate()
  * API for each of its text entry and TranslationRequest.
  */
 
@@ -13,7 +13,7 @@
 
 #include "QualityScore.h"
 
-/* This class represents the result of AbstractTranslationModel::translate() API
+/* This class represents the result of TranslationModel::translate() API
  * for each of its text entry and TranslationRequest.
  */
 class TranslationResult {
diff --git a/src/translator/AbstractTranslationModel.cpp b/src/translator/AbstractTranslationModel.cpp
deleted file mode 100644
index b731c60d2..000000000
--- a/src/translator/AbstractTranslationModel.cpp
+++ /dev/null
@@ -1,18 +0,0 @@
-/*
- * AbstractTranslationModel.cpp
- *
- */
-#include <memory>
-
-// All local includes
-#include "AbstractTranslationModel.h"
-#include "TranslationModel.h"
-
-/**
- * @param config Marian yml config file in the form of a string
- * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Optional, defaults to nullptr when not used
- */
-std::shared_ptr<AbstractTranslationModel>
-AbstractTranslationModel::createInstance(const std::string &config, const void * model_memory) {
-  return std::make_shared<TranslationModel>(config, model_memory);
-}
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index ae38f9bb4..34f3762e9 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,5 +1,4 @@
 add_library(bergamot-translator STATIC
-    AbstractTranslationModel.cpp
     TranslationModel.cpp
 
     byteArrayExample.cpp
@@ -30,7 +29,7 @@ endif(COMPILE_WASM)
 target_link_libraries(bergamot-translator marian ssplit)
 
 target_include_directories(bergamot-translator
-    PRIVATE ${CMAKE_SOURCE_DIR}
+    PUBLIC ${CMAKE_SOURCE_DIR}
     PUBLIC ${CMAKE_SOURCE_DIR}/src)
 
 
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 715df4727..72de208da 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -13,7 +13,7 @@
 
 TranslationModel::TranslationModel(const std::string &config,
                                    const void *model_memory)
-    : AbstractTranslationModel(), service_(config, model_memory) {}
+    : service_(config, model_memory) {}
 
 TranslationModel::~TranslationModel() {}
 

From fdbce5705b0dd88143c32e41bf03b835031300ac Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 25 Mar 2021 12:32:06 +0100
Subject: [PATCH 180/442] Update marian-dev submodule to master

 - Earlier it was using 'wasm' branch
 - CMakefile changes
 - Github workflow change
---
 .../workflows/macos-custom-marian-native.yml  | 32 ------------
 .../workflows/ubuntu-custom-marian-wasm.yml   | 51 +++++++++++++++++++
 3rd_party/marian-dev                          |  2 +-
 CMakeLists.txt                                | 17 ++-----
 src/translator/CMakeLists.txt                 |  4 --
 5 files changed, 56 insertions(+), 50 deletions(-)
 delete mode 100644 .github/workflows/macos-custom-marian-native.yml
 create mode 100644 .github/workflows/ubuntu-custom-marian-wasm.yml

diff --git a/.github/workflows/macos-custom-marian-native.yml b/.github/workflows/macos-custom-marian-native.yml
deleted file mode 100644
index 9a919d3cb..000000000
--- a/.github/workflows/macos-custom-marian-native.yml
+++ /dev/null
@@ -1,32 +0,0 @@
-name: MacOS Native (Custom)
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ main, ci-sandbox ]
-
-jobs:
-  build-macos:
-    name: Native (With Custom Marian)
-    runs-on: macos-10.15
-
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-        with:
-          submodules: recursive
-
-      - name: Configure CMake
-        run: |
-          mkdir -p build-native
-          cd build-native
-          cmake ..
-
-      - name: Compile
-        working-directory: build-native
-        run: make -j2
-
-      - name: Print versions
-        working-directory: build-native
-        run: ./app/bergamot-translator-app --version
diff --git a/.github/workflows/ubuntu-custom-marian-wasm.yml b/.github/workflows/ubuntu-custom-marian-wasm.yml
new file mode 100644
index 000000000..a6645d483
--- /dev/null
+++ b/.github/workflows/ubuntu-custom-marian-wasm.yml
@@ -0,0 +1,51 @@
+name: WASM on Ubuntu
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build-wasm:
+    name: WASM (With Custom Marian)
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Setup Emscripten toolchain
+        uses: mymindstorm/setup-emsdk@v8
+
+      - name: Verify Emscripten setup
+        run: emcc -v
+
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: recursive
+
+      - name: Configure builds
+        run: |
+          mkdir -p build-wasm
+          cd build-wasm
+          emcmake cmake -DCOMPILE_WASM=on ..
+
+      - name: Compile
+        working-directory: build-wasm
+        run: emmake make -j2
+
+      - name: Instantiate simd wormhole
+        working-directory: build-wasm
+        run: bash ../wasm/patch-artifacts-enable-wormhole.sh
+
+      - name: Check artifacts
+        working-directory: build-wasm
+        run: |
+          export WASM_ARTIFACTS_DIR=wasm
+          ls -all ${WASM_ARTIFACTS_DIR}
+          if ls ${WASM_ARTIFACTS_DIR}/*.wasm &>/dev/null && ls ${WASM_ARTIFACTS_DIR}/*.js &>/dev/null
+          then
+            echo "Artifacts Successfully Generated"
+          else
+            echo "Failure: Artifacts Not Present"
+            exit 1
+          fi
diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 370fdb5a2..9337105c9 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 370fdb5a212cfcd2d1c5fca9fffc041d2787a432
+Subproject commit 9337105c9f905e2e2b04ee4141a564af4523b96e
diff --git a/CMakeLists.txt b/CMakeLists.txt
index a84b8e4ad..6417bd677 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -16,26 +16,17 @@ option(COMPILE_WASM "Compile for WASM" OFF)
 option(USE_WASM_COMPATIBLE_SOURCES "Use wasm compatible sources" ON)
 SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be packaged (pre-loaded) in wasm builds")
 
-# Set marian (3rd party submodule) cmake options to compile for this project
+# Set 3rd party submodule specific cmake options for this project
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
 SET(USE_SENTENCEPIECE ON CACHE BOOL "Download and compile SentencePiece")
 SET(USE_STATIC_LIBS ON CACHE BOOL "Link statically against non-system libs")
 if (USE_WASM_COMPATIBLE_SOURCES)
-  # If using wasm compatible marian then set following flags
+  # Setting the marian submodule specific cmake options for wasm
   SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
   SET(USE_MKL OFF CACHE BOOL "Compile with MKL support")
-  SET(COMPILE_DECODER_ONLY ON CACHE BOOL "Compile marian-decoder only")
-  SET(COMPILE_WITH_PTHREADS OFF CACHE BOOL "Compile with pthreads support")
-  SET(USE_WASM_COMPATIBLE_BLAS ON CACHE BOOL "Compile with a WASM compatible blas for decoder only builds")
-  SET(COMPILE_WITHOUT_EXCEPTIONS ON CACHE BOOL "Compile without exceptions")
-  if(COMPILE_WASM)
-    # Set WORMHOLE to ON for marian whenever compiling for wasm platform
-    SET(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
-  endif()
+  # # Setting the ssplit-cpp submodule specific cmake options for wasm
+  SET(USE_INTERNAL_PCRE2 ON CACHE BOOL "Use internal PCRE2 instead of system PCRE2")
 endif()
-# Set ssplit (3rd party submodule) cmake options to compile for this project
-CMAKE_DEPENDENT_OPTION(USE_INTERNAL_PCRE2 "Use internal PCRE2 instead of system PCRE2" ON
-                       "USE_WASM_COMPATIBLE_SOURCES" OFF)
 
 # Documentation: https://cliutils.gitlab.io/modern-cmake/chapters/projects/submodule.html
 # Ensures the submodules are set correctly during a build.
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 34f3762e9..90e9f9cac 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -13,10 +13,6 @@ add_library(bergamot-translator STATIC
     sentence_ranges.cpp
     service.cpp
 )
-if (COMPILE_DECODER_ONLY)
-  # A dirty hack because of marian's bad cmake practices
-  target_compile_definitions(bergamot-translator PUBLIC DECODER_ONLY)
-endif()
 
 if(COMPILE_WASM)
   # A dirty hack because of marian's bad cmake practices

From e0dca1ba1b5d345a1c81a8932addd711f37b51c7 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Fri, 26 Mar 2021 09:15:51 +0100
Subject: [PATCH 181/442] Renamed github workflow files

 - Naming follows
   <target-arch>-<nature-of-marian>-<runner-os>

   (wasm|native)-(full_marian|custom_marian)-(ubuntu|mac)
---
 .github/workflows/{macos.yml => native-full_marian-mac.yml}   | 2 +-
 .../workflows/{ubuntu.yml => native-full_marian-ubuntu.yml}   | 2 +-
 ...acos-custom-marian-wasm.yml => wasm-custom_marian-mac.yml} | 4 ++--
 ...u-custom-marian-wasm.yml => wasm-custom_marian-ubuntu.yml} | 4 ++--
 4 files changed, 6 insertions(+), 6 deletions(-)
 rename .github/workflows/{macos.yml => native-full_marian-mac.yml} (97%)
 rename .github/workflows/{ubuntu.yml => native-full_marian-ubuntu.yml} (99%)
 rename .github/workflows/{macos-custom-marian-wasm.yml => wasm-custom_marian-mac.yml} (94%)
 rename .github/workflows/{ubuntu-custom-marian-wasm.yml => wasm-custom_marian-ubuntu.yml} (94%)

diff --git a/.github/workflows/macos.yml b/.github/workflows/native-full_marian-mac.yml
similarity index 97%
rename from .github/workflows/macos.yml
rename to .github/workflows/native-full_marian-mac.yml
index 6a9035398..44f1568bf 100644
--- a/.github/workflows/macos.yml
+++ b/.github/workflows/native-full_marian-mac.yml
@@ -1,4 +1,4 @@
-name: MacOS
+name: Native (Full Marian) MacOS
 
 on:
   push:
diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/native-full_marian-ubuntu.yml
similarity index 99%
rename from .github/workflows/ubuntu.yml
rename to .github/workflows/native-full_marian-ubuntu.yml
index 6ddd49ade..6ab8ea655 100644
--- a/.github/workflows/ubuntu.yml
+++ b/.github/workflows/native-full_marian-ubuntu.yml
@@ -1,4 +1,4 @@
-name: Ubuntu
+name: Native (Full Marian) Ubuntu
 
 on:
   push:
diff --git a/.github/workflows/macos-custom-marian-wasm.yml b/.github/workflows/wasm-custom_marian-mac.yml
similarity index 94%
rename from .github/workflows/macos-custom-marian-wasm.yml
rename to .github/workflows/wasm-custom_marian-mac.yml
index 00f5cf7af..87141c74e 100644
--- a/.github/workflows/macos-custom-marian-wasm.yml
+++ b/.github/workflows/wasm-custom_marian-mac.yml
@@ -1,4 +1,4 @@
-name: MacOS WASM (Custom)
+name: WASM (Custom Marian) MacOS
 
 on:
   push:
@@ -8,7 +8,7 @@ on:
 
 jobs:
   build-wasm:
-    name: WASM (With Custom Marian)
+    name: WASM (Custom Marian) MacOS
     runs-on: macos-10.15
 
     steps:
diff --git a/.github/workflows/ubuntu-custom-marian-wasm.yml b/.github/workflows/wasm-custom_marian-ubuntu.yml
similarity index 94%
rename from .github/workflows/ubuntu-custom-marian-wasm.yml
rename to .github/workflows/wasm-custom_marian-ubuntu.yml
index a6645d483..d1364dc50 100644
--- a/.github/workflows/ubuntu-custom-marian-wasm.yml
+++ b/.github/workflows/wasm-custom_marian-ubuntu.yml
@@ -1,4 +1,4 @@
-name: WASM on Ubuntu
+name: WASM (Custom Marian) Ubuntu
 
 on:
   push:
@@ -8,7 +8,7 @@ on:
 
 jobs:
   build-wasm:
-    name: WASM (With Custom Marian)
+    name: WASM (Custom Marian) Ubuntu
     runs-on: ubuntu-latest
 
     steps:

From bfb5e7860278582c5ba1b2ee3bac1d3062b12623 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 31 Mar 2021 17:41:36 +0100
Subject: [PATCH 182/442] Alignments + weak quality scores capability in
 Service (#46)

* Draft adjustments to API

* Adjustments to docs

* Let's call the word + sentence ranges annotations

* Editing confusing comment on size()

* Fixing compilation for template adjustments for SentenceRanges

* string_view template hacks

This commit shifts AnnotatedBlob into a templated type and gets the
troubled part to compile. All to manage absl::string_view and
std::string_view.

Objective: marian::bergamot stays C++ 11 to pluck and put in marian
code, bergamot-translator somehow flexes C++17. Simplify development in
one place.

* Fixing the wiring: Gets source to build

Runtime errors exist, but AnnotatedBlobs are consistent.

* Bugfix: Matching old-state after factoring AnnotatedBlob in

* Removing vocabs_ from Response.

(For the umpteenth time).

* Alignment API ready in marian::bergamot::Response

* Wiring alignments upto TranslationResult

* Adjustment to get alignments; bergamot-translator-app has alignments available

* Accessing words instead of Ids

This code sets up access of word string_views from annotations instead
of printing Ids. However, we have segfault. This is likely due to
targetRanges not being set, pending from
https://github.com/browsermt/bergamot-translator/issues/25.

Could also be a rogue EOS token which we're filtering for in string_view
annotations, but not so in alignments.

* Switching to browsermt/marian-dev@jp/decode-string-view for targetTokenRanges

* Target word byte range annotations available

Issues corresponding to #25 should be resolved. There is still a
segfault. Could be due to EOS. Pending investigation.

* Bugfix: Tokens for alignments are now through.

Was not EOS.

* browsermt/marian-dev@master

ByteRange changes work downstream and has been merged to master.
Updating submodule to point to master.

* Style and documentation enhancements: response.cpp

* Style and documentation enhancements: TranslationResult.h

* Descriptions for SentenceRanges templating

* Switching to marian-dev@wasm-sync

* AnnotatedBlob can be copy-ctord/copy-assigned

* TranslationResult: Empty ctor + WASM Bindings

Allows empty construction of TranslationResult. Using this empty
constructor, WASM bindings are adjusted. Unsure of the results, maybe
@abhi-agg can test.

* Cosmetic: SentenceRangesT -> Annotation

- SentenceRangesT is renamed to AnnotationT;
- Further comments to explain heavily templated files.

* Response: Cleaning up unused members and adding docs

* Adding quality scores - attempt

* Stub QualityScores

This adjustment adds capability to get "scores", which should
potentially indicate how confident (at least relative in a
target-sentence) should be. This enables writing the code forward for
TranslationResult, and an example quality-score people can be pointed
at.

- These are not between [0,1] yet.
- In addition, guards to check out-of-bounds access have been placed so
  illegal accesses are caught early on during development.

* Removing token debug statements

* Reworking Annotation without templates

https://github.com/mozilla/bergamot-translator/issues/8 provides
ByteRanges.

- This ByteRange data-type is used in Annotation and converted
  to marian::string_view(=absl::string-view) on demand.
- Since Annotation[using ByteRange] is not bound to anything else, it
  can be unit tested. A unit test is added (originally to test
  independently for integration after).
- Annotation with ByteRange is now propogated across marian::bergamot
  and functionality matched to how it was previously working.

This eliminates the string-view conversion and template code.

* Nit: Removing std::endl flushes

* Bring TranslationResult and Response closer

Helps https://github.com/browsermt/bergamot-translator/issues/53.

In preparation , the data-export types for Quality and Alignment are
pushed down to Response from TranslationResult and computed during
construction. This brings TranslationResult closer to Response, paving
way to avoid having two TranslationResults.

histories_ only remain for marian-decoder replacement usage, which can
be removed in a separate PR.

* Clean up hacks originally added for a unit-test to compile

* Moving Annotation functions to cpp and documenting header file

* Shifting alignments, qualityScore testing capability into main-mts

* Restore Unified API files to previous state

* Adaptations to fix Response with Quality, Alignments to connect to old Unified API

* Missing reset on TranslationResultBindings

* Cleaning up Response documentation to reflect newer code

* Minor adjustments to get build back after main sync

* Marian seems to make available Catch somehow

* Disable COMPILE_BERGAMOT_TESTS for WASM

* Add COMPILE_BERGAMOT_TESTS as a CMakeDependent option

* Use the COMPILE_TESTS flag instead to skip macos.yml

* Trigger unit-tests on GitHub runners for Annotation

* Reordering enable_testing() to before inclusion of test directory

* doc constructs required to operate with alignments

Documents with doxygen compatible documentation for Response,
AnnotatedBlob, Annotation, ByteRange.

Incorporates doxygen compatible documentation for

* Updates ByteRange consistent with general C++

Also little documentation enhancements in the process.

* Updating marian-dev@9337105

* Copy-paste documentation because lazy

* Turn off autoformat and manually edit to fix style changes

* AnnotatedBlob -> AnnotatedText; blob -> text

* text.text in test app renamed

* text of text -> blob of text in places of documentation
---
 .github/workflows/native-full_marian-mac.yml  |   7 +-
 .../workflows/native-full_marian-ubuntu.yml   |  12 +-
 CMakeLists.txt                                |   8 ++
 app/service-cli-bytearray.cpp                 |   2 +-
 app/service-cli.cpp                           |  51 ++++++-
 src/CMakeLists.txt                            |   8 +-
 src/tests/CMakeLists.txt                      |  22 +++
 src/tests/annotation_tests.cpp                |  73 ++++++++++
 src/tests/run_tests.cpp                       |   2 +
 src/translator/TranslationModel.cpp           |  21 ++-
 src/translator/request.cpp                    |   9 +-
 src/translator/request.h                      |  12 +-
 src/translator/response.cpp                   | 116 ++++++++-------
 src/translator/response.h                     | 132 ++++++++---------
 src/translator/sentence_ranges.cpp            |  93 +++++++++---
 src/translator/sentence_ranges.h              | 136 ++++++++++++++----
 src/translator/service.cpp                    |   8 +-
 src/translator/service.h                      |   4 +-
 src/translator/text_processor.cpp             |  10 +-
 src/translator/text_processor.h               |   5 +-
 20 files changed, 509 insertions(+), 222 deletions(-)
 create mode 100644 src/tests/CMakeLists.txt
 create mode 100644 src/tests/annotation_tests.cpp
 create mode 100644 src/tests/run_tests.cpp

diff --git a/.github/workflows/native-full_marian-mac.yml b/.github/workflows/native-full_marian-mac.yml
index 44f1568bf..a2b62d5c5 100644
--- a/.github/workflows/native-full_marian-mac.yml
+++ b/.github/workflows/native-full_marian-mac.yml
@@ -45,10 +45,9 @@ jobs:
       working-directory: build
       run: make -j2
 
-    # Removing unit-tests, taken care of in browsermt/marian-dev
-    # - name: Run unit tests
-    # - working-directory: build
-    # - run: make test
+    - name: Run unit tests
+      working-directory: build
+      run: make test
 
     - name: Print versions
       working-directory: build
diff --git a/.github/workflows/native-full_marian-ubuntu.yml b/.github/workflows/native-full_marian-ubuntu.yml
index 6ab8ea655..e67c3663f 100644
--- a/.github/workflows/native-full_marian-ubuntu.yml
+++ b/.github/workflows/native-full_marian-ubuntu.yml
@@ -103,13 +103,11 @@ jobs:
       working-directory: build
       run: make -j2
 
-    # Removing unit-tests, taken care of in browsermt/marian-dev
-    # TODO: add a flag to CMake to compile unit tests only on CPU
-    # - name: Run unit tests
-    #   working-directory: build
-    #   run: make test
-    #   # GitHub-hosted VMs do not have GPUs, so can not be run in CUDA builds
-    #   if: matrix.gpu == false
+    - name: Run unit tests
+      working-directory: build
+      run: make test
+      # GitHub-hosted VMs do not have GPUs, so can not be run in CUDA builds
+      if: matrix.gpu == false
 
     - name: Print versions
       working-directory: build
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 6417bd677..206a5b5c0 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -14,6 +14,8 @@ include(CMakeDependentOption)
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
 option(USE_WASM_COMPATIBLE_SOURCES "Use wasm compatible sources" ON)
+option(COMPILE_TESTS "Compile bergamot-tests" OFF)
+
 SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be packaged (pre-loaded) in wasm builds")
 
 # Set 3rd party submodule specific cmake options for this project
@@ -56,6 +58,11 @@ if(COMPILE_WASM)
   list(APPEND WASM_COMPILE_FLAGS -Wno-error=pthreads-mem-growth)
 endif(COMPILE_WASM)
 
+# Needs to be enabled before including the folder containing tests (src/tests)
+if(COMPILE_TESTS)
+    enable_testing()
+endif(COMPILE_TESTS)
+
 add_subdirectory(3rd_party)
 add_subdirectory(src)
 
@@ -64,3 +71,4 @@ if(COMPILE_WASM)
 else()
   add_subdirectory(app)
 endif(COMPILE_WASM)
+
diff --git a/app/service-cli-bytearray.cpp b/app/service-cli-bytearray.cpp
index d96781037..cb3b17ff0 100644
--- a/app/service-cli-bytearray.cpp
+++ b/app/service-cli-bytearray.cpp
@@ -28,7 +28,7 @@ int main(int argc, char *argv[]) {
   std::future<Response> responseFuture = service.translate(std::move(input));
   responseFuture.wait();
   Response response = responseFuture.get();
-  std::cout << response.translation() << std::endl;
+  std::cout << response.target.text << std::endl;
 
   // Clear the memory used for the byte array
   free(model_bytes); // Ideally, this should be done after the translation model has been gracefully shut down.
diff --git a/app/service-cli.cpp b/app/service-cli.cpp
index 2bb825c26..6ed4d81f6 100644
--- a/app/service-cli.cpp
+++ b/app/service-cli.cpp
@@ -25,7 +25,56 @@ int main(int argc, char *argv[]) {
   std::future<Response> responseFuture = service.translate(std::move(input));
   responseFuture.wait();
   Response response = responseFuture.get();
-  std::cout << response.translation() << std::endl;
+
+  std::cout << "[original]: " << response.source.text << '\n';
+  std::cout << "[translated]: " << response.target.text << '\n';
+  for (int sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
+    std::cout << " [src Sentence]: " << response.source.sentence(sentenceIdx)
+              << '\n';
+    std::cout << " [tgt Sentence]: " << response.target.sentence(sentenceIdx)
+              << '\n';
+    std::cout << "Alignments" << '\n';
+    typedef std::pair<size_t, float> Point;
+
+    // Initialize a point vector.
+    std::vector<std::vector<Point>> aggregate(
+        response.source.numWords(sentenceIdx));
+
+    // Handle alignments
+    auto &alignments = response.alignments[sentenceIdx];
+    for (auto &p : alignments) {
+      aggregate[p.src].emplace_back(p.tgt, p.prob);
+    }
+
+    for (size_t src = 0; src < aggregate.size(); src++) {
+      std::cout << response.source.word(sentenceIdx, src) << ": ";
+      for (auto &p : aggregate[src]) {
+        std::cout << response.target.word(sentenceIdx, p.first) << "("
+                  << p.second << ") ";
+      }
+      std::cout << '\n';
+    }
+
+    // Handle quality.
+    auto &quality = response.qualityScores[sentenceIdx];
+    std::cout << "Quality: whole(" << quality.sequence
+              << "), tokens below:" << '\n';
+    size_t wordIdx = 0;
+    bool first = true;
+    for (auto &p : quality.word) {
+      if (first) {
+        first = false;
+      } else {
+        std::cout << " ";
+      }
+      std::cout << response.target.word(sentenceIdx, wordIdx) << "(" << p
+                << ")";
+      wordIdx++;
+    }
+    std::cout << '\n';
+  }
+  std::cout << "--------------------------\n";
+  std::cout << '\n';
 
   return 0;
 }
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 27fecc4bc..c2d62ef83 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -1 +1,7 @@
-add_subdirectory(translator)
\ No newline at end of file
+add_subdirectory(translator)
+
+if(COMPILE_TESTS)
+  # Catch currently comes from marian sources.
+  add_subdirectory(tests)
+endif(COMPILE_TESTS)
+
diff --git a/src/tests/CMakeLists.txt b/src/tests/CMakeLists.txt
new file mode 100644
index 000000000..5c1bc003c
--- /dev/null
+++ b/src/tests/CMakeLists.txt
@@ -0,0 +1,22 @@
+# Unit tests
+set(UNIT_TESTS
+    annotation_tests
+)
+
+foreach(test ${UNIT_TESTS})
+  add_executable("run_${test}" run_tests.cpp "${test}.cpp")
+  target_include_directories("run_${test}" PRIVATE ${CATCH_INCLUDE_DIR} "${CMAKE_SOURCE_DIR}/src")
+
+  if(CUDA_FOUND)
+    target_link_libraries("run_${test}" ${EXT_LIBS} marian ${EXT_LIBS} marian_cuda ${EXT_LIBS} Catch bergamot-translator)
+  else(CUDA_FOUND)
+    target_link_libraries("run_${test}" marian ${EXT_LIBS} Catch bergamot-translator)
+  endif(CUDA_FOUND)
+
+  if(msvc)
+    # disable c4305: truncation from 'double' to '_ty'
+    target_compile_options("run_${test}" public /wd4305)
+  endif(msvc)
+
+  add_test(NAME ${test} COMMAND "run_${test}")
+endforeach(test)
diff --git a/src/tests/annotation_tests.cpp b/src/tests/annotation_tests.cpp
new file mode 100644
index 000000000..228401146
--- /dev/null
+++ b/src/tests/annotation_tests.cpp
@@ -0,0 +1,73 @@
+#include "catch.hpp"
+#include "translator/sentence_ranges.h"
+#include <random>
+#include <vector>
+
+using namespace marian::bergamot;
+
+TEST_CASE("Test Annotation API with random sentences") {
+  /// Objective here is to test insertion for sentences, and that whatever comes
+  /// out adheres to the way it was inserted. Towards this, we keep externally
+  /// which sentence went in where and try to use accessor methods on
+  /// AnnotatedText to check if what we have as ground-truth by construction is
+  /// consistent with what is returned.
+  size_t sentences = 20;
+  size_t maxWords = 40;
+
+  std::mt19937 randomIntGen_;
+  randomIntGen_.seed(42);
+
+  AnnotatedText testAnnotation;
+  std::vector<std::vector<ByteRange>> sentenceWords;
+  std::vector<ByteRange> Words;
+
+  for (size_t idx = 0; idx < sentences; idx++) {
+    if (idx != 0)
+      testAnnotation.text += "\n";
+
+    Words.clear();
+    size_t words = randomIntGen_() % maxWords + 1;
+    Words.reserve(words);
+    for (size_t idw = 0; idw < words; idw++) {
+      size_t before = testAnnotation.text.size();
+      std::string word = std::to_string(idx) + "-" + std::to_string(idw);
+      testAnnotation.text += word;
+      if (idw != 0)
+        testAnnotation.text += " ";
+      Words.push_back((ByteRange){before, before + word.size() - 1});
+    }
+    // std::cout << std::endl;
+
+    sentenceWords.push_back(Words);
+  }
+
+  // std::cout << "Inserting words:" << std::endl;
+  std::vector<std::vector<marian::string_view>> byteRanges;
+  for (auto &sentence : sentenceWords) {
+    std::vector<marian::string_view> wordByteRanges;
+    for (auto &word : sentence) {
+      marian::string_view wordView(&testAnnotation.text[word.begin],
+                                   word.end - word.begin);
+      wordByteRanges.push_back(wordView);
+      // std::cout << std::string(wordView) << " ";
+    }
+    testAnnotation.addSentence(wordByteRanges);
+    byteRanges.push_back(wordByteRanges);
+    // std::cout << std::endl;
+  }
+
+  // std::cout << "From container: " << std::endl;
+  for (int idx = 0; idx < sentenceWords.size(); idx++) {
+    for (int idw = 0; idw < sentenceWords[idx].size(); idw++) {
+      ByteRange expected = sentenceWords[idx][idw];
+      ByteRange obtained = testAnnotation.wordAsByteRange(idx, idw);
+      // std::cout << std::string(testAnnotation.word(idx, idw)) << " ";
+      CHECK(expected.begin == obtained.begin);
+      CHECK(expected.end == obtained.end);
+
+      std::string expected_string = std::string(byteRanges[idx][idw]);
+      CHECK(expected_string == std::string(testAnnotation.word(idx, idw)));
+    }
+    // std::cout << std::endl;
+  }
+}
diff --git a/src/tests/run_tests.cpp b/src/tests/run_tests.cpp
new file mode 100644
index 000000000..0c7c351f4
--- /dev/null
+++ b/src/tests/run_tests.cpp
@@ -0,0 +1,2 @@
+#define CATCH_CONFIG_MAIN
+#include "catch.hpp"
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 72de208da..13f949578 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -32,24 +32,19 @@ TranslationModel::translate(std::vector<std::string> &&texts,
     intermediate.wait();
     auto marianResponse(std::move(intermediate.get()));
 
-    // This mess because marian::string_view != std::string_view
-    std::string source, translation;
-    marian::bergamot::Response::SentenceMappings mSentenceMappings;
-    marianResponse.move(source, translation, mSentenceMappings);
-
-    // Convert to UnifiedAPI::TranslationResult
     TranslationResult::SentenceMappings sentenceMappings;
-    for (auto &p : mSentenceMappings) {
-      std::string_view src(p.first.data(), p.first.size()),
-          tgt(p.second.data(), p.second.size());
-      sentenceMappings.emplace_back(src, tgt);
+    for (size_t idx = 0; idx < marianResponse.size(); idx++) {
+      marian::string_view src = marianResponse.source.sentence(idx);
+      marian::string_view tgt = marianResponse.target.sentence(idx);
+      sentenceMappings.emplace_back(std::string_view(src.data(), src.size()),
+                                    std::string_view(tgt.data(), tgt.size()));
     }
 
     // In place construction.
     translationResults.emplace_back(
-        std::move(source),          // &&marianResponse.source_
-        std::move(translation),     // &&marianResponse.translation_
-        std::move(sentenceMappings) // &&sentenceMappings
+        std::move(marianResponse.source.text), // &&marianResponse.source_
+        std::move(marianResponse.target.text), // &&marianResponse.translation_
+        std::move(sentenceMappings)            // &&sentenceMappings
     );
   }
 
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 42dfb35f3..b6d243857 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -12,12 +12,10 @@ namespace bergamot {
 
 // -----------------------------------------------------------------
 Request::Request(size_t Id, size_t lineNumberBegin,
-                 std::vector<Ptr<Vocab const>> &vocabs, std::string &&source,
-                 Segments &&segments, SentenceRanges &&sourceRanges,
-                 std::promise<Response> responsePromise)
+                 std::vector<Ptr<Vocab const>> &vocabs, AnnotatedText &&source,
+                 Segments &&segments, std::promise<Response> responsePromise)
     : Id_(Id), lineNumberBegin_(lineNumberBegin), vocabs_(&vocabs),
       source_(std::move(source)), segments_(std::move(segments)),
-      sourceRanges_(std::move(sourceRanges)),
       response_(std::move(responsePromise)) {
 
   counter_ = segments_.size();
@@ -48,8 +46,7 @@ void Request::processHistory(size_t index, Ptr<History> history) {
 void Request::completeRequest() {
   // Request no longer needs to hold the content, can transfer it to
   // Response.
-  Response response(std::move(source_), std::move(sourceRanges_),
-                    std::move(histories_), *vocabs_);
+  Response response(std::move(source_), std::move(histories_), *vocabs_);
   response_.set_value(std::move(response));
 }
 
diff --git a/src/translator/request.h b/src/translator/request.h
index 3909019c7..605dea7db 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -1,9 +1,9 @@
 //
 // Defines:
 //
-// Request: holds the input blob of a text, Segments (vector<Words>) which are
+// Request: holds the input text of a text, Segments (vector<Words>) which are
 // to go to the batching mechanism and alignments between the processed
-// segments and the input blob (sourceTokenRanges). In addition, Request takes
+// segments and the input text (sourceTokenRanges). In addition, Request takes
 // care of the barrier which fires when all the Segments in a request are done
 // translating by the workers (BatchTranslator).
 // TODO(jerinphilip):  Extend Request with notions of Priority (sequence,
@@ -36,9 +36,8 @@ namespace bergamot {
 class Request {
 public:
   Request(size_t Id, size_t lineNumberBegin,
-          std::vector<Ptr<Vocab const>> &vocabs_, std::string &&source,
-          Segments &&segments, SentenceRanges &&sourceTokenRanges,
-          std::promise<Response> responsePromise);
+          std::vector<Ptr<Vocab const>> &vocabs_, AnnotatedText &&source,
+          Segments &&segments, std::promise<Response> responsePromise);
 
   // Obtain the count of tokens in the segment correponding to index. Used to
   // insert sentence from multiple requests into the corresponding size bucket.
@@ -77,9 +76,8 @@ class Request {
   // string_views of the text corresponding to these words, pointing to
   // sequences in source_. histories_ is a buffer which eventually stores the
   // translations of each segment in the corresponding index.
-  std::string source_;
+  AnnotatedText source_;
   Segments segments_;
-  SentenceRanges sourceRanges_;
   std::vector<Ptr<History>> histories_;
 
   // Members above are moved into newly constructed Response on completion
diff --git a/src/translator/response.cpp b/src/translator/response.cpp
index b731755d1..faa42da00 100644
--- a/src/translator/response.cpp
+++ b/src/translator/response.cpp
@@ -1,49 +1,25 @@
 #include "response.h"
-#include "sentence_ranges.h"
 #include "common/logging.h"
 #include "data/alignment.h"
+#include "sentence_ranges.h"
 
 #include <utility>
 
 namespace marian {
 namespace bergamot {
 
-Response::Response(std::string &&source, SentenceRanges &&sourceRanges,
-                   Histories &&histories, std::vector<Ptr<Vocab const>> &vocabs)
-    : source_(std::move(source)), sourceRanges_(std::move(sourceRanges)),
-      histories_(std::move(histories)), vocabs_(&vocabs) {}
-
-void Response::move(std::string &source, std::string &translation,
-                    SentenceMappings &sentenceMappings) {
-
-  // Construct required stuff first.
-  constructTranslation();
-  constructSentenceMappings(sentenceMappings);
-
-  // Move content out.
-  source = std::move(source_);
-  translation = std::move(translation_);
-
-  // The above assignment expects source, target be moved.
-  // which makes the following invalid, hence required to be cleared.
-  sourceRanges_.clear();
-  targetRanges_.clear();
-  histories_.clear();
-}
-
-void Response::constructTranslation() {
-  if (translationConstructed_) {
-    return;
-  }
-
+Response::Response(AnnotatedText &&source, Histories &&histories,
+                   std::vector<Ptr<Vocab const>> &vocabs)
+    : source(std::move(source)), histories_(std::move(histories)) {
   // Reserving length at least as much as source_ seems like a reasonable thing
   // to do to avoid reallocations.
-  translation_.reserve(source_.size());
+  target.text.reserve(source.text.size());
 
   // In a first step, the decoded units (individual senteneces) are compiled
   // into a huge string. This is done by computing indices first and appending
   // to the string as each sentences are decoded.
   std::vector<std::pair<size_t, size_t>> translationRanges;
+  std::vector<size_t> sentenceBegins;
 
   size_t offset{0};
   bool first{true};
@@ -54,44 +30,76 @@ void Response::constructTranslation() {
 
     Result result = onebest[0]; // Expecting only one result;
     Words words = std::get<0>(result);
-    auto targetVocab = vocabs_->back();
-    std::string decoded = targetVocab->decode(words);
+    auto targetVocab = vocabs.back();
+
+    std::string decoded;
+    std::vector<string_view> targetMappings;
+    targetVocab->decodeWithByteRanges(words, decoded, targetMappings);
+
     if (first) {
       first = false;
     } else {
-      translation_ += " ";
+      target.text += " ";
       ++offset;
     }
 
-    translation_ += decoded;
-    translationRanges.emplace_back(offset, decoded.size());
+    sentenceBegins.push_back(translationRanges.size());
+    target.text += decoded;
+    auto decodedStringBeginMarker = targetMappings.front().begin();
+    for (auto &sview : targetMappings) {
+      size_t startIdx = offset + sview.begin() - decodedStringBeginMarker;
+      translationRanges.emplace_back(startIdx, startIdx + sview.size());
+    }
+
     offset += decoded.size();
-  }
 
-  // Once the entire string is constructed, there are no further possibility of
-  // reallocation in the string's storage, the indices are converted into
-  // string_views.
+    // Alignments
+    // TODO(jerinphilip): The following double conversion might not be
+    // necessary. Hard alignment can directly be exported, but this would mean
+    // WASM bindings for a structure deep within marian source.
+    auto hyp = std::get<1>(result);
+    auto softAlignment = hyp->tracebackAlignment();
+    auto hardAlignment = data::ConvertSoftAlignToHardAlign(
+        softAlignment, /*threshold=*/0.2f); // TODO(jerinphilip): Make this a
+                                            // configurable parameter.
+
+    Alignment unified_alignment;
+    for (auto &p : hardAlignment) {
+      unified_alignment.emplace_back((Point){p.srcPos, p.tgtPos, p.prob});
+    }
 
-  for (auto &range : translationRanges) {
-    // TODO(@jerinphilip):  Currently considers target tokens as whole text.
-    // Needs to be further enhanced in marian-dev to extract alignments.
-    std::vector<string_view> targetMappings;
+    alignments.push_back(std::move(unified_alignment));
 
-    const char *begin = &translation_[range.first];
-    targetMappings.emplace_back(begin, range.second);
-    targetRanges_.addSentence(targetMappings);
+    // Quality scores: Sequence level is obtained as normalized path scores.
+    // Word level using hypothesis traceback. These are most-likely logprobs.
+    auto normalizedPathScore = std::get<2>(result);
+    auto wordQualities = hyp->tracebackWordScores();
+    wordQualities.pop_back();
+    qualityScores.push_back((Quality){normalizedPathScore, wordQualities});
   }
 
-  translationConstructed_ = true;
-}
+  // Once we have the indices in translation (which might be resized a few
+  // times) ready, we can prepare and store the string_view as annotations
+  // instead. This is accomplished by iterating over available sentences using
+  // sentenceBegin and using addSentence(...) API from Annotation.
 
-void Response::constructSentenceMappings(
-    Response::SentenceMappings &sentenceMappings) {
+  for (size_t i = 1; i <= sentenceBegins.size(); i++) {
+    std::vector<string_view> targetMappings;
+    size_t begin = sentenceBegins[i - 1];
+    size_t safe_end = (i == sentenceBegins.size()) ? translationRanges.size()
+                                                   : sentenceBegins[i];
+
+    for (size_t idx = begin; idx < safe_end; idx++) {
+      auto &p = translationRanges[idx];
+      size_t begin_idx = p.first;
+      size_t end_idx = p.second;
+
+      const char *data = &target.text[begin_idx];
+      size_t size = end_idx - begin_idx;
+      targetMappings.emplace_back(data, size);
+    }
 
-  for (size_t i = 0; i < sourceRanges_.numSentences(); i++) {
-    string_view src = sourceRanges_.sentence(i);
-    string_view tgt = targetRanges_.sentence(i);
-    sentenceMappings.emplace_back(src, tgt);
+    target.addSentence(targetMappings);
   }
 }
 } // namespace bergamot
diff --git a/src/translator/response.h b/src/translator/response.h
index 17fee05ae..385735c88 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -1,9 +1,10 @@
 #ifndef SRC_BERGAMOT_RESPONSE_H_
 #define SRC_BERGAMOT_RESPONSE_H_
 
-#include "sentence_ranges.h"
+#include "data/alignment.h"
 #include "data/types.h"
 #include "definitions.h"
+#include "sentence_ranges.h"
 #include "translator/beam_search.h"
 
 #include <cassert>
@@ -12,86 +13,87 @@
 
 namespace marian {
 namespace bergamot {
+
+/// Alignment is stored as a sparse matrix, this pretty much aligns with marian
+/// internals but is brought here to maintain translator
+/// agnosticism/independence.
+struct Point {
+  size_t src; ///< Index pointing to source ByteRange
+  size_t tgt; ///< Index pointing to target ByteRange
+  float prob; ///< Score between [0, 1] on indicating degree of alignment.
+};
+
+/// Alignment is a sparse matrix, where Points represent entries with values.
+typedef std::vector<Point> Alignment;
+
+/// -loglikelhoods of the sequence components as proxy to quality.
+struct Quality {
+  /// Certainty/uncertainty score for sequence.
+  float sequence;
+  /// Certainty/uncertainty for each word in the sequence.
+  std::vector<float> word;
+};
+
+/// Response holds AnnotatedText(s) of source-text and translated text,
+/// alignment information between source and target sub-words and sentences.
+///
+/// AnnotatedText provides an API to access markings of (sub)-word and
+/// sentences boundaries, which are required to interpret Quality and
+/// Alignment (s) at the moment.
 class Response {
-  // Response is a marian internal class (not a bergamot-translator class)
-  // holding source blob of text, vector of TokenRanges corresponding to each
-  // sentence in the source text blob and histories obtained from translating
-  // these sentences.
-  //
-  // This class provides an API at a higher level in comparison to History to
-  // access translations and additionally use string_view manipulations to
-  // recover structure in translation from source-text's structure known through
-  // reference string and string_view. As many of these computations are not
-  // required until invoked, they are computed as required and stored in data
-  // members where it makes sense to do so (translation,translationTokenRanges).
-  //
-  // Examples of such use-cases are:
-  //    translation()
-  //    translationInSourceStructure() TODO(@jerinphilip)
-  //    alignment(idx) TODO(@jerinphilip)
-  //    sentenceMappings (for bergamot-translator)
 
 public:
-  Response(std::string &&source, SentenceRanges &&sourceRanges,
-           Histories &&histories,
-           // Required for constructing translation and TokenRanges within
-           // translation lazily.
+  ///
+  Response(AnnotatedText &&source, Histories &&histories,
            std::vector<Ptr<Vocab const>> &vocabs);
 
+  /// \cond HIDDEN_PUBLIC
   // Move constructor.
   Response(Response &&other)
-      : source_(std::move(other.source_)),
-        translation_(std::move(other.translation_)),
-        sourceRanges_(std::move(other.sourceRanges_)),
-        targetRanges_(std::move(other.targetRanges_)),
-        histories_(std::move(other.histories_)),
-        vocabs_(std::move(other.vocabs_)){};
-
-  // Prevents CopyConstruction and CopyAssignment. sourceRanges_ is constituted
-  // by string_view and copying invalidates the data member.
+      : source(std::move(other.source)), target(std::move(other.target)),
+        alignments(std::move(other.alignments)),
+        qualityScores(std::move(other.qualityScores)),
+        histories_(std::move(other.histories_)){};
+
+  // The following copy bans are not stricitly required anymore since Annotation
+  // is composed of the ByteRange primitive (which was previously string_view
+  // and required to be bound to string), but makes movement efficient by
+  // banning these letting compiler complain about copies.
+
   Response(const Response &) = delete;
   Response &operator=(const Response &) = delete;
 
-  typedef std::vector<std::pair<const string_view, const string_view>>
-      SentenceMappings;
+  /// \endcond
+
+  /// Number of sentences translated. The processing of a text of into sentences
+  /// are handled internally, and this information can be used to iterate
+  /// through meaningful units of translation for which alignment and quality
+  /// information are available.
+  const size_t size() const { return source.numSentences(); }
+
+  /// source text and annotations of (sub-)words and sentences.
+  AnnotatedText source;
 
-  // Moves source sentence into source, translated text into translation.
-  // Pairs of string_views to corresponding sentences in
-  // source and translation are loaded into sentenceMappings. These string_views
-  // reference the new source and translation.
-  //
-  // Calling move() invalidates the Response object as ownership is transferred.
-  // Exists for moving strc
-  void move(std::string &source, std::string &translation,
-            SentenceMappings &sentenceMappings);
+  /// translated text and annotations of (sub-)words and sentences.
+  AnnotatedText target;
 
+  /// -logprob of each word and negative log likelihood of sequence (sentence)
+  /// normalized by length, for each sentence processed by the translator.
+  /// Indices correspond to ranges accessible through respective Annotation on
+  /// source or target.
+  std::vector<Quality> qualityScores;
+
+  /// Alignments between source and target. Each Alignment is a
+  /// sparse matrix representation with indices corresponding
+  /// to (sub-)words accessible through Annotation.
+  std::vector<Alignment> alignments;
+
+  /// Access to histories, which holds rich information on translated text.
+  /// Not recommended to use, will be removed in future.
   const Histories &histories() const { return histories_; }
-  const std::string &source() const { return source_; }
-  const std::string &translation() {
-    constructTranslation();
-    return translation_;
-  }
-
-  // A convenience function provided to return translated text placed within
-  // source's structure. This is useful when the source text is a multi-line
-  // paragraph or string_views extracted from structured text like HTML and it's
-  // desirable to place the individual sentences in the locations of the source
-  // sentences.
-  // const std::string translationInSourceStructure();
-  // const PendingAlignmentType alignment(size_t idx);
 
 private:
-  void constructTranslation();
-  void constructSentenceMappings(SentenceMappings &);
-
-  std::string source_;
-  SentenceRanges sourceRanges_;
   Histories histories_;
-
-  std::vector<Ptr<Vocab const>> *vocabs_;
-  bool translationConstructed_{false};
-  std::string translation_;
-  SentenceRanges targetRanges_;
 };
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/sentence_ranges.cpp b/src/translator/sentence_ranges.cpp
index a9ee8c54e..053eeaa48 100644
--- a/src/translator/sentence_ranges.cpp
+++ b/src/translator/sentence_ranges.cpp
@@ -5,40 +5,87 @@
 namespace marian {
 namespace bergamot {
 
-void SentenceRanges::addSentence(std::vector<string_view> &wordRanges) {
-  addSentence(std::begin(wordRanges), std::end(wordRanges));
-}
-
-void SentenceRanges::addSentence(WordIterator begin, WordIterator end) {
+void Annotation::addSentence(std::vector<ByteRange> &sentence) {
   size_t size = flatByteRanges_.size();
-  flatByteRanges_.insert(std::end(flatByteRanges_), begin, end);
+  flatByteRanges_.insert(std::end(flatByteRanges_), std::begin(sentence),
+                         std::end(sentence));
   sentenceBeginIds_.push_back(size);
 }
 
-string_view SentenceRanges::sentence(size_t index) const {
-  size_t bos_id;
-  string_view eos, bos;
+size_t Annotation::numWords(size_t sentenceIdx) const {
+  auto terminals = sentenceTerminalIds(sentenceIdx);
+  return terminals.second - terminals.first + 1;
+}
+
+std::pair<size_t, size_t>
+Annotation::sentenceTerminalIds(size_t sentenceIdx) const {
+  size_t bosId, eosId;
+  bosId = sentenceBeginIds_[sentenceIdx];
+  eosId = sentenceIdx + 1 < numSentences()
+              ? sentenceBeginIds_[sentenceIdx + 1] - 1
+              : flatByteRanges_.size() - 1;
 
-  bos_id = sentenceBeginIds_[index];
-  bos = flatByteRanges_[bos_id];
+  // Out of bound checks.
+  assert(bosId < flatByteRanges_.size());
+  assert(eosId < flatByteRanges_.size());
+  return std::make_pair(bosId, eosId);
+}
+
+std::pair<ByteRange, ByteRange>
+Annotation::sentenceTerminals(size_t sentenceIdx) const {
+  auto terminals = sentenceTerminalIds(sentenceIdx);
+  return std::make_pair(flatByteRanges_[terminals.first],
+                        flatByteRanges_[terminals.second]);
+}
+
+ByteRange Annotation::sentence(size_t sentenceIdx) const {
+  auto terminals = sentenceTerminals(sentenceIdx);
+  return (ByteRange){terminals.first.begin, terminals.second.end};
+}
 
-  if (index + 1 == numSentences()) {
-    eos = flatByteRanges_.back();
-  } else {
-    assert(index < numSentences());
-    size_t eos_id = sentenceBeginIds_[index + 1];
-    --eos_id;
-    eos = flatByteRanges_[eos_id];
+ByteRange Annotation::word(size_t sentenceIdx, size_t wordIdx) const {
+  size_t offset = sentenceBeginIds_[sentenceIdx];
+  // auto terminals = sentenceTerminals(sentenceIdx);
+  // assert(offset + wordIdx <= terminals.second);
+  return flatByteRanges_[offset + wordIdx];
+}
+
+string_view AnnotatedText::word(size_t sentenceIdx, size_t wordIdx) const {
+  auto terminals = annotation.word(sentenceIdx, wordIdx);
+  return string_view(&text[terminals.begin], terminals.size());
+}
+
+string_view AnnotatedText::sentence(size_t sentenceIdx) const {
+  auto sentenceAsByteRange = annotation.sentence(sentenceIdx);
+  return asStringView(sentenceAsByteRange);
+}
+
+void AnnotatedText::addSentence(std::vector<string_view> &wordRanges) {
+  addSentence(std::begin(wordRanges), std::end(wordRanges));
+};
+
+void AnnotatedText::addSentence(std::vector<string_view>::iterator begin,
+                                std::vector<string_view>::iterator end) {
+  std::vector<ByteRange> sentence;
+  for (auto p = begin; p != end; p++) {
+    size_t begin_offset = p->data() - &text[0];
+    sentence.push_back((ByteRange){begin_offset, begin_offset + p->size()});
   }
+  annotation.addSentence(sentence);
+};
 
-  return sentenceBetween(bos, eos);
+ByteRange AnnotatedText::wordAsByteRange(size_t sentenceIdx,
+                                         size_t wordIdx) const {
+  return annotation.word(sentenceIdx, wordIdx);
 }
 
-string_view SentenceRanges::sentenceBetween(string_view firstWord,
-                                            string_view lastWord) const {
+ByteRange AnnotatedText::sentenceAsByteRange(size_t sentenceIdx) const {
+  return annotation.sentence(sentenceIdx);
+}
 
-  const char *data = firstWord.data();
-  size_t size = lastWord.data() + lastWord.size() - firstWord.data();
+string_view AnnotatedText::asStringView(const ByteRange &byteRange) const {
+  const char *data = &text[byteRange.begin];
+  size_t size = byteRange.size();
   return string_view(data, size);
 }
 
diff --git a/src/translator/sentence_ranges.h b/src/translator/sentence_ranges.h
index c6a077028..a0dc8c9a9 100644
--- a/src/translator/sentence_ranges.h
+++ b/src/translator/sentence_ranges.h
@@ -3,50 +3,134 @@
 
 #include "data/types.h"
 #include <cassert>
+#include <utility>
 #include <vector>
 
 namespace marian {
 namespace bergamot {
 
-class SentenceRanges {
-  // SentenceRanges stores string_views into a source text, with additional
-  // annotations to mark sentence boundaries.
-  //
-  // Given the availability annotations, this container provides capabilty to
-  // add sentences, and access individual sentences.
+/// ByteRange stores indices for half-interval [begin, end) in a string. Can be
+/// used to represent a sentence, word.
+struct ByteRange {
+  size_t begin;
+  size_t end;
+  const size_t size() const { return end - begin; }
+};
+
+/// An Annotation is a collection of ByteRanges used to denote ancillary
+/// information of sentences and words on a text of string. Annotation is meant
+/// for consumption on platforms where string_view creates problems (eg: exports
+/// through WASM). See AnnotatedText for cases where this is a non-issue.
+class Annotation {
 public:
-  typedef std::vector<string_view>::iterator WordIterator;
+  /// Annotation is constructed empty. See addSentence to populate it with
+  /// annotations.
+  Annotation() {}
 
-  void addSentence(std::vector<string_view> &wordRanges);
-  void addSentence(WordIterator begin, WordIterator end);
+  /// Returns the number of sentences annotated in a text.
+  size_t numSentences() const { return sentenceBeginIds_.size(); }
 
-  void clear() {
-    flatByteRanges_.clear();
-    sentenceBeginIds_.clear();
-  }
+  /// Returns number of words in the sentece identified by sentenceIdx.
+  size_t numWords(size_t sentenceIdx) const;
 
-  size_t numSentences() const { return sentenceBeginIds_.size(); }
+  /// Adds a sentences from vector<ByteRange> representation, internally doing
+  /// extra book-keeping for the sentence terminal markings. Sentences are
+  /// expected to be added in order as they occur in text.
+  void addSentence(std::vector<ByteRange> &sentence);
+
+  /// Returns a ByteRange representing wordIdx in sentenceIdx
+  ByteRange word(size_t sentenceIdx, size_t wordIdx) const;
 
-  // Returns a string_view into the ith sentence.
-  string_view sentence(size_t index) const;
+  /// Returns a ByteRange representing sentence corresponding to sentenceIdx.
+  ByteRange sentence(size_t sentenceIdx) const;
 
 private:
-  // A flat storage for string_views. Can be words or sentences.
-  std::vector<string_view> flatByteRanges_;
+  /// A flat storage for ByteRanges. Composed of word ByteRanges, extra
+  /// information in sentenceBeginIds_ to denote sentence boundary markers as
+  /// indices.
+  std::vector<ByteRange> flatByteRanges_;
 
-  // The container grows dynamically with addSentence. size_t marking index is
-  // used to ensure the sentence boundaries stay same while underlying storage
-  // might be changed during reallocation.
+  /// Stores indices where sentences begin
   std::vector<size_t> sentenceBeginIds_;
 
-  // Utility function to extract the string starting at firstWord and ending at
-  // lastWord as a single string-view.
-  string_view sentenceBetween(string_view firstWord,
-                              string_view lastWord) const;
+  /// Returns ByteRanges corresponding to beginning and end words of sentence
+  /// corresponding to sentenceIdx. This is useful in using the information to
+  /// construct a ByteRange of a sentence taking the begin from the first and
+  /// end from the second.
+  std::pair<ByteRange, ByteRange> sentenceTerminals(size_t sentenceIdx) const;
+
+  /// Returns indices of terminal (word) ByteRanges in sentenceIds_ of a
+  /// sentence corresponding to sentenceIdx. The distance can be used to compute
+  /// number of words in a sentence (numWords) and also to construct the
+  /// terminal ByteRanges (sentenceTerminals).
+  std::pair<size_t, size_t> sentenceTerminalIds(size_t sentenceIdx) const;
 };
 
-} // namespace bergamot
+/// AnnotatedText is effectively std::string text + Annotation, providing the
+/// following additional desiderata.
+///
+/// 1. Access to processed string_views for convenience rather than ByteRanges
+/// (which only provides index information).
+///
+/// 2. Transparently convert string_views into ByteRanges for the Annotation
+/// referring to the text bound by this structure.
+///
+/// 3. Bind the text and annotations together, to move around as a meaningful
+/// unit.
+
+struct AnnotatedText {
+public:
+  std::string text;      ///< Blob of string elements in annotation refers to.
+  Annotation annotation; ///< sentence and (sub-) word annotations.
+
+  /// Construct an empty AnnotatedText. This is useful when the target string or
+  /// ByteRanges are not known yet, but the public members can be used to
+  /// populate it. One use-case, when translated-text is created decoding from
+  /// histories and the ByteRanges only known after the string has been
+  /// constructed.
+  AnnotatedText() {}
+
+  /// Construct moving in a string (for efficiency purposes, copying string
+  /// constructor is disallowed).
+  AnnotatedText(std::string &&text) : text(std::move(text)){};
+
+  AnnotatedText(AnnotatedText &&annotatedBlob)
+      : text(std::move(annotatedBlob.text)),
+        annotation(std::move(annotatedBlob.annotation)) {}
 
+  /// Returns the number of sentences in the annotation structure.
+  const size_t numSentences() const { return annotation.numSentences(); }
+
+  /// Returns number of words in the sentece identified by sentenceIdx.
+  const size_t numWords(size_t sentenceIdx) const {
+    return annotation.numWords(sentenceIdx);
+  }
+
+  /// Adds a sentence, used to load from SentencePiece annotations conveniently.
+  void addSentence(std::vector<string_view> &wordRanges);
+
+  /// Adds a sentence between two iterators, often useful while constructing
+  /// from parts of a container.
+  void addSentence(std::vector<string_view>::iterator begin,
+                   std::vector<string_view>::iterator end);
+
+  /// Returns a string_view representing wordIdx in sentenceIdx
+  string_view word(size_t sentenceIdx, size_t wordIdx) const;
+
+  /// Returns a string_view representing sentence corresponding to sentenceIdx.
+  string_view sentence(size_t sentenceIdx) const;
+
+  /// Returns a ByteRange representing wordIdx in sentenceIdx
+  ByteRange wordAsByteRange(size_t sentenceIdx, size_t wordIdx) const;
+
+  /// Returns a ByteRange representing sentence corresponding to sentenceIdx.
+  ByteRange sentenceAsByteRange(size_t sentenceIdx) const;
+
+private:
+  string_view asStringView(const ByteRange &byteRange) const;
+};
+
+} // namespace bergamot
 } // namespace marian
 
 #endif //  BERGAMOT_SENTENCE_RANGES_H_
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 8c8c45312..80b2a106b 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -112,15 +112,15 @@ void Service::async_translate() {
 
 std::future<Response> Service::translate(std::string &&input) {
   Segments segments;
-  SentenceRanges sourceRanges;
-  text_processor_.process(input, segments, sourceRanges);
+  AnnotatedText source(std::move(input));
+  text_processor_.process(source, segments);
 
   std::promise<Response> responsePromise;
   auto future = responsePromise.get_future();
 
   Ptr<Request> request = New<Request>(
-      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(input),
-      std::move(segments), std::move(sourceRanges), std::move(responsePromise));
+      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(source),
+      std::move(segments), std::move(responsePromise));
 
   batcher_.addWholeRequest(request);
   if (numWorkers_ == 0) {
diff --git a/src/translator/service.h b/src/translator/service.h
index bb8dbe929..04ec2b838 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -25,9 +25,9 @@ namespace bergamot {
 ///
 ///  options = ...;
 ///  service = Service(options);
-///  std::string input_blob = "Hello World";
+///  std::string input_text = "Hello World";
 ///  std::future<Response>
-///      response = service.translate(std::move(input_blob));
+///      response = service.translate(std::move(input_text));
 ///  response.wait();
 ///  Response result = response.get();
 ///
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index 9d6733e24..8d7f25cc9 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -25,9 +25,9 @@ TextProcessor::TextProcessor(std::vector<Ptr<Vocab const>> &vocabs,
   ABORT_IF(max_length_break_ < 0, "max-length-break cannot be < 0");
 }
 
-void TextProcessor::process(const string_view &query, Segments &segments,
-                            SentenceRanges &sourceRanges) {
+void TextProcessor::process(AnnotatedText &source, Segments &segments) {
 
+  string_view query = string_view(source.text);
   auto sentenceStream = sentence_splitter_.createSentenceStream(query);
   std::string_view sentenceStringPiece;
 
@@ -42,14 +42,14 @@ void TextProcessor::process(const string_view &query, Segments &segments,
     // after normalization. 0 prevents any empty entries from being added.
     if (segment.size() > 0) {
       // Truncate segment into max_input_size segments.
-      truncate(segment, wordRanges, segments, sourceRanges);
+      truncate(segment, wordRanges, segments, source);
     }
   }
 }
 
 void TextProcessor::truncate(Segment &segment,
                              std::vector<string_view> &wordRanges,
-                             Segments &segments, SentenceRanges &sourceRanges) {
+                             Segments &segments, AnnotatedText &source) {
   for (size_t offset = 0; offset < segment.size();
        offset += max_length_break_) {
     auto start = segment.begin() + offset;
@@ -61,7 +61,7 @@ void TextProcessor::truncate(Segment &segment,
     segments.back().push_back(sourceEosId());
 
     auto astart = wordRanges.begin() + offset;
-    sourceRanges.addSentence(astart, astart + diff);
+    source.addSentence(astart, astart + diff);
   }
 }
 
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index 4cd176126..ed3c7736b 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -23,8 +23,7 @@ class TextProcessor {
 public:
   explicit TextProcessor(std::vector<Ptr<Vocab const>> &vocabs, Ptr<Options>);
 
-  void process(const string_view &query, Segments &segments,
-               SentenceRanges &sourceRanges);
+  void process(AnnotatedText &source, Segments &segments);
 
 private:
   // Tokenizes an input string, returns Words corresponding. Loads the
@@ -34,7 +33,7 @@ class TextProcessor {
 
   // Truncate sentence into max_input_size segments.
   void truncate(Segment &sentence, std::vector<string_view> &tokenRanges,
-                Segments &segments, SentenceRanges &sourceRanges);
+                Segments &segments, AnnotatedText &source);
 
   // shorthand, used only in truncate()
   const Word sourceEosId() const { return vocabs_->front()->getEosId(); }

From 47db7e2b3ec25d8ada1f6dd0f3032f2f6f34a489 Mon Sep 17 00:00:00 2001
From: evgeny pavlov <epavlov@mozilla.com>
Date: Fri, 26 Mar 2021 18:47:42 -0700
Subject: [PATCH 183/442] Change bergamot app to process stdin texts

---
 app/bergamot-translator-app.cpp | 36 ++++++++-------------------------
 1 file changed, 8 insertions(+), 28 deletions(-)

diff --git a/app/bergamot-translator-app.cpp b/app/bergamot-translator-app.cpp
index a71abbcac..086b9ca20 100644
--- a/app/bergamot-translator-app.cpp
+++ b/app/bergamot-translator-app.cpp
@@ -1,11 +1,13 @@
 /*
  * main.cpp
  *
- * An example application to demonstrate the use of Bergamot translator.
+ * An application which accepts line separated texts in stdin and returns translated ones in stdout.
+ * It is convenient for batch processing and can be used with tools like SacreBLEU.
  *
  */
 
 #include <iostream>
+#include <string>
 
 #include "TranslationModel.h"
 #include "translator/parser.h"
@@ -23,22 +25,10 @@ int main(int argc, char **argv) {
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
-  texts.emplace_back(
-      "The Bergamot project will add and improve client-side machine "
-      "translation in a web browser.  Unlike current cloud-based "
-      "options, running directly on users’ machines empowers citizens to "
-      "preserve their privacy and increases the uptake of language "
-      "technologies in Europe in various sectors that require "
-      "confidentiality.");
-  texts.emplace_back(
-      "Free software integrated with an open-source web "
-      "browser, such as Mozilla Firefox, will enable bottom-up adoption "
-      "by non-experts, resulting in cost savings for private and public "
-      "sector users who would otherwise procure translation or operate "
-      "monolingually.  Bergamot is a consortium coordinated by the "
-      "University of Edinburgh with partners Charles University in "
-      "Prague, the University of Sheffield, University of Tartu, and "
-      "Mozilla.");
+
+  for (std::string line; std::getline(std::cin, line);) {
+        texts.emplace_back(line);
+  }
 
   auto results = model->translate(std::move(texts), translationRequest);
 
@@ -46,17 +36,7 @@ int main(int argc, char **argv) {
   //std::vector<TranslationResult> results = futureResults.get();
 
   for (auto &result : results) {
-    std::cout << "[original]: " << result.getOriginalText() << std::endl;
-    std::cout << "[translated]: " << result.getTranslatedText() << std::endl;
-    auto mappings = result.getSentenceMappings();
-    for (auto &p : mappings) {
-      std::string_view src = p.first;
-      std::string_view tgt = p.second;
-
-      std::cout << " [src Sentence]: " << src << std::endl;
-      std::cout << " [tgt Sentence]: " << tgt << std::endl;
-    }
-    std::cout << std::endl;
+    std::cout << result.getTranslatedText() << std::endl;
   }
 
   return 0;

From 3068ed58ff23dc4ebb8d3ee45911549e41082f60 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Thu, 1 Apr 2021 13:56:16 +0100
Subject: [PATCH 184/442] Explictly install gcc version and use 8 (#81)

* Try to fix gcc missingness in CI
---
 .github/workflows/native-full_marian-ubuntu.yml | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/native-full_marian-ubuntu.yml b/.github/workflows/native-full_marian-ubuntu.yml
index e67c3663f..db3fa0d08 100644
--- a/.github/workflows/native-full_marian-ubuntu.yml
+++ b/.github/workflows/native-full_marian-ubuntu.yml
@@ -2,9 +2,9 @@ name: Native (Full Marian) Ubuntu
 
 on:
   push:
-    branches: [ main, ci-sandbox ]
+    branches: [ main, ci-test ]
   pull_request:
-    branches: [ main, ci-sandbox ]
+    branches: [ main, ci-test ]
 
 jobs:
   build-ubuntu:
@@ -15,7 +15,7 @@ jobs:
           - name: "Ubuntu CPU-only"
             os: ubuntu-latest
             cuda: ""
-            gcc: 7
+            gcc: 8
             cpu: true
             gpu: false
           # GPU Builds are commented out, for bergamot-translator CI runs.
@@ -62,7 +62,7 @@ jobs:
     # No need to install libprotobuf{17,10,9v5} on Ubuntu {20,18,16}.04 because
     # it is installed together with libprotobuf-dev
     - name: Install dependencies
-      run: sudo apt-get update && sudo apt-get install -y libgoogle-perftools-dev libprotobuf-dev protobuf-compiler libboost-all-dev
+      run: sudo apt-get update && sudo apt-get install -y libgoogle-perftools-dev libprotobuf-dev protobuf-compiler libboost-all-dev g++-8
 
     # https://software.intel.com/content/www/us/en/develop/articles/installing-intel-free-libs-and-python-apt-repo.html
     - name: Install MKL

From 2e5daac978f535b14bf32997e563f3bf430efbfb Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 1 Apr 2021 17:29:02 +0200
Subject: [PATCH 185/442] Marian submodule update (#74)

* Updated marian-dev submodule

 - cmake changes required after the submodule update

* Added workflows for building custom marian on mac and ubuntu

* Renamed cmake option

 - Renamed USE_WASM_COMPATIBLE_SOURCES to USE_WASM_COMPATIBLE_SOURCE
 - Use proper compile defnitions
---
 .../workflows/native-custom_marian-mac.yml    | 33 +++++++++++++++++++
 .../workflows/native-custom_marian-ubuntu.yml | 33 +++++++++++++++++++
 .github/workflows/native-full_marian-mac.yml  |  2 +-
 .../workflows/native-full_marian-ubuntu.yml   |  2 +-
 3rd_party/marian-dev                          |  2 +-
 CMakeLists.txt                                |  6 ++--
 app/CMakeLists.txt                            |  2 +-
 doc/marian-integration.md                     |  2 +-
 src/translator/CMakeLists.txt                 | 11 ++++---
 src/translator/batch_translator.h             |  2 +-
 src/translator/batcher.h                      |  2 +-
 src/translator/service.cpp                    | 10 +++---
 src/translator/service.h                      |  6 ++--
 13 files changed, 91 insertions(+), 22 deletions(-)
 create mode 100644 .github/workflows/native-custom_marian-mac.yml
 create mode 100644 .github/workflows/native-custom_marian-ubuntu.yml

diff --git a/.github/workflows/native-custom_marian-mac.yml b/.github/workflows/native-custom_marian-mac.yml
new file mode 100644
index 000000000..1aae7e590
--- /dev/null
+++ b/.github/workflows/native-custom_marian-mac.yml
@@ -0,0 +1,33 @@
+name: Native (Custom Marian) MacOS
+
+on:
+  push:
+    branches: [ main, ci-sandbox ]
+  pull_request:
+    branches: [ main, ci-sandbox ]
+
+jobs:
+  build-macos:
+    name: MacOS
+    runs-on: macos-10.15
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Configure CMake
+      run: |
+        mkdir -p build
+        cd build
+        cmake ..
+
+    - name: Compile
+      working-directory: build
+      run: make -j2
+
+    - name: Print versions
+      working-directory: build
+      run: |
+        ./app/bergamot-translator-app --version
diff --git a/.github/workflows/native-custom_marian-ubuntu.yml b/.github/workflows/native-custom_marian-ubuntu.yml
new file mode 100644
index 000000000..f0518718a
--- /dev/null
+++ b/.github/workflows/native-custom_marian-ubuntu.yml
@@ -0,0 +1,33 @@
+name: Native (Custom Marian) Ubuntu
+
+on:
+  push:
+    branches: [ main, ci-sandbox ]
+  pull_request:
+    branches: [ main, ci-sandbox ]
+
+jobs:
+  build-macos:
+    name: Ubuntu
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Configure CMake
+      run: |
+        mkdir -p build
+        cd build
+        cmake ..
+
+    - name: Compile
+      working-directory: build
+      run: make -j2
+
+    - name: Print versions
+      working-directory: build
+      run: |
+        ./app/bergamot-translator-app --version
diff --git a/.github/workflows/native-full_marian-mac.yml b/.github/workflows/native-full_marian-mac.yml
index a2b62d5c5..1928c5c03 100644
--- a/.github/workflows/native-full_marian-mac.yml
+++ b/.github/workflows/native-full_marian-mac.yml
@@ -39,7 +39,7 @@ jobs:
           -DUSE_FBGEMM=on \
           -DUSE_SENTENCEPIECE=on \
           -DUSE_STATIC_LIBS=off \
-          -DUSE_WASM_COMPATIBLE_SOURCES=off
+          -DUSE_WASM_COMPATIBLE_SOURCE=off
 
     - name: Compile
       working-directory: build
diff --git a/.github/workflows/native-full_marian-ubuntu.yml b/.github/workflows/native-full_marian-ubuntu.yml
index db3fa0d08..e414f6477 100644
--- a/.github/workflows/native-full_marian-ubuntu.yml
+++ b/.github/workflows/native-full_marian-ubuntu.yml
@@ -97,7 +97,7 @@ jobs:
           -DUSE_FBGEMM=${{ matrix.cpu }} \
           -DUSE_SENTENCEPIECE=on \
           -DUSE_STATIC_LIBS=on \
-          -DUSE_WASM_COMPATIBLE_SOURCES=off
+          -DUSE_WASM_COMPATIBLE_SOURCE=off
 
     - name: Compile
       working-directory: build
diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 9337105c9..0f0bcf996 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 9337105c9f905e2e2b04ee4141a564af4523b96e
+Subproject commit 0f0bcf99626c660227bb68b76267a8d2451e7172
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 206a5b5c0..412b3863e 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -13,7 +13,7 @@ include(CMakeDependentOption)
 
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
-option(USE_WASM_COMPATIBLE_SOURCES "Use wasm compatible sources" ON)
+option(USE_WASM_COMPATIBLE_SOURCE "Use wasm compatible sources" ON)
 option(COMPILE_TESTS "Compile bergamot-tests" OFF)
 
 SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be packaged (pre-loaded) in wasm builds")
@@ -22,8 +22,7 @@ SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be package
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
 SET(USE_SENTENCEPIECE ON CACHE BOOL "Download and compile SentencePiece")
 SET(USE_STATIC_LIBS ON CACHE BOOL "Link statically against non-system libs")
-if (USE_WASM_COMPATIBLE_SOURCES)
-  # Setting the marian submodule specific cmake options for wasm
+if (USE_WASM_COMPATIBLE_SOURCE)
   SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
   SET(USE_MKL OFF CACHE BOOL "Compile with MKL support")
   # # Setting the ssplit-cpp submodule specific cmake options for wasm
@@ -53,6 +52,7 @@ if(NOT COMPILE_WASM)
 endif()
 
 if(COMPILE_WASM)
+  set(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
   list(APPEND WASM_COMPILE_FLAGS -pthread -O3 -g2 -fPIC -mssse3 -msimd128)
   list(APPEND WASM_COMPILE_FLAGS "SHELL:-s WASM=1" "SHELL:-s ASSERTIONS=0" "SHELL:-s DISABLE_EXCEPTION_CATCHING=1" "SHELL:-s LLD_REPORT_UNDEFINED" "SHELL:-s FORCE_FILESYSTEM=1" "SHELL:-s ALLOW_MEMORY_GROWTH=1")
   list(APPEND WASM_COMPILE_FLAGS -Wno-error=pthreads-mem-growth)
diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index 3f2b9de49..e23a10720 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -4,7 +4,7 @@ target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
 add_executable(bergamot-translator-app-bytearray bergamot-translator-app-bytearray.cpp)
 target_link_libraries(bergamot-translator-app-bytearray PRIVATE bergamot-translator)
 
-if (NOT USE_WASM_COMPATIBLE_SOURCES)
+if (NOT USE_WASM_COMPATIBLE_SOURCE)
     add_executable(service-cli service-cli.cpp)
     target_link_libraries(service-cli PRIVATE bergamot-translator)
 
diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index 50e0648e9..411b55422 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -10,7 +10,7 @@ $ git clone https://github.com/browsermt/bergamot-translator
 $ cd bergamot-translator
 $ mkdir build
 $ cd build
-$ cmake .. -DUSE_WASM_COMPATIBLE_SOURCES=off -DCMAKE_BUILD_TYPE=Release
+$ cmake .. -DUSE_WASM_COMPATIBLE_SOURCE=off -DCMAKE_BUILD_TYPE=Release
 $ make -j
 ```
 
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 90e9f9cac..a34142787 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -13,10 +13,15 @@ add_library(bergamot-translator STATIC
     sentence_ranges.cpp
     service.cpp
 )
+if (USE_WASM_COMPATIBLE_SOURCE)
+  # Using wasm compatible sources should include this compile definition;
+  # Has to be done here because we are including marian headers + some sources
+  # in local repository use these definitions
+  target_compile_definitions(bergamot-translator PUBLIC USE_SSE2 WASM_COMPATIBLE_SOURCE)
+endif()
 
 if(COMPILE_WASM)
-  # A dirty hack because of marian's bad cmake practices
-  target_compile_definitions(bergamot-translator PUBLIC USE_SSE2 WASM)
+  target_compile_definitions(bergamot-translator PUBLIC WASM)
   # Enable code that is required for generating JS bindings
   target_compile_definitions(bergamot-translator PRIVATE WASM_BINDINGS)
   target_compile_options(bergamot-translator PRIVATE ${WASM_COMPILE_FLAGS})
@@ -27,5 +32,3 @@ target_link_libraries(bergamot-translator marian ssplit)
 target_include_directories(bergamot-translator
     PUBLIC ${CMAKE_SOURCE_DIR}
     PUBLIC ${CMAKE_SOURCE_DIR}/src)
-
-
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index c920945fa..4e17b6514 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -12,7 +12,7 @@
 #include "translator/history.h"
 #include "translator/scorers.h"
 
-#ifndef WASM
+#ifndef WASM_COMPATIBLE_SOURCE
 #include "pcqueue.h"
 #endif
 
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index 675cd5944..e5ac08604 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -7,7 +7,7 @@
 #include "definitions.h"
 #include "request.h"
 
-#ifndef WASM
+#ifndef WASM_COMPATIBLE_SOURCE
 #include "pcqueue.h"
 #endif
 
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 80b2a106b..718bc4dda 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -32,7 +32,7 @@ Service::Service(Ptr<Options> options, const void *model_memory)
     : requestId_(0), vocabs_(std::move(loadVocabularies(options))),
       text_processor_(vocabs_, options), batcher_(options),
       numWorkers_(options->get<int>("cpu-threads")), model_memory_(model_memory)
-#ifndef WASM
+#ifndef WASM_COMPATIBLE_SOURCE
       // 0 elements in PCQueue is illegal and can lead to failures. Adding a
       // guard to have at least one entry allocated. In the single-threaded
       // case, while initialized pcqueue_ remains unused.
@@ -70,7 +70,7 @@ void Service::blocking_translate() {
   }
 }
 
-#ifndef WASM
+#ifndef WASM_COMPATIBLE_SOURCE
 void Service::initialize_async_translators() {
   workers_.reserve(numWorkers_);
 
@@ -100,7 +100,7 @@ void Service::async_translate() {
     pcqueue_.ProduceSwap(batch);
   }
 }
-#else  // WASM
+#else  // WASM_COMPATIBLE_SOURCE
 void Service::initialize_async_translators() {
   ABORT("Cannot run in async mode without multithreading.");
 }
@@ -108,7 +108,7 @@ void Service::initialize_async_translators() {
 void Service::async_translate() {
   ABORT("Cannot run in async mode without multithreading.");
 }
-#endif // WASM
+#endif // WASM_COMPATIBLE_SOURCE
 
 std::future<Response> Service::translate(std::string &&input) {
   Segments segments;
@@ -132,7 +132,7 @@ std::future<Response> Service::translate(std::string &&input) {
 }
 
 Service::~Service() {
-#ifndef WASM
+#ifndef WASM_COMPATIBLE_SOURCE
   for (size_t workerId = 0; workerId < numWorkers_; workerId++) {
 
     Batch poison = Batch::poison();
diff --git a/src/translator/service.h b/src/translator/service.h
index 04ec2b838..59f2f4d86 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -8,7 +8,7 @@
 #include "text_processor.h"
 #include "translator/parser.h"
 
-#ifndef WASM
+#ifndef WASM_COMPATIBLE_SOURCE
 #include "pcqueue.h"
 #endif
 
@@ -111,10 +111,10 @@ class Service {
 
   // The following constructs are available providing full capabilities on a non
   // WASM platform, where one does not have to hide threads.
-#ifndef WASM
+#ifndef WASM_COMPATIBLE_SOURCE
   PCQueue<Batch> pcqueue_; // ORDER DEPENDENCY (numWorkers_)
   std::vector<std::thread> workers_;
-#endif // WASM
+#endif // WASM_COMPATIBLE_SOURCE
 };
 
 } // namespace bergamot

From f654ab0f715451d1104c4ba5d3a5a76cb2545b29 Mon Sep 17 00:00:00 2001
From: Qianqian Zhu <qianqian.zhu@hotmail.com>
Date: Thu, 1 Apr 2021 19:36:07 +0100
Subject: [PATCH 186/442] Enable binary shortlist loading from bytebuffer (#69)

Contains "hack" that must go immediately by editing TranslationModel, to come in following commit.

* add shortlist_memory and update service-cli-bytearray test

* update marian-dev

* address review comments

* fix ccompliation and tests failures and further address review comments

* small update on marian-dev (based on browsermt/marian-dev PR#28)

* update marian-dev with upstream

* code refactoring according to review

* fix marian-dev submodule conflicts

* switch MemoryGift to AlignedVector

* copy aligned.h from kpu/intgemm for AlignedVector

* changes based on memory ownership and AlignedVector

* fix BatchTranslator inits

* small fixes according to review comments

* update submodule marian-dev to master

* update submodule marian-dev with upstream

Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
---
 app/bergamot-translator-app-bytearray.cpp | 13 ++---
 app/service-cli-bytearray.cpp             | 12 ++--
 src/translator/CMakeLists.txt             |  3 +-
 src/translator/aligned.h                  | 71 +++++++++++++++++++++++
 src/translator/batch_translator.cpp       | 32 ++++++----
 src/translator/batch_translator.h         |  8 ++-
 src/translator/byteArrayExample.cpp       | 45 --------------
 src/translator/byteArrayExample.h         |  8 ---
 src/translator/byte_array_util.cpp        | 33 +++++++++++
 src/translator/byte_array_util.h          | 12 ++++
 src/translator/definitions.h              |  4 ++
 src/translator/parser.h                   |  6 +-
 src/translator/service.cpp                |  7 ++-
 src/translator/service.h                  | 52 ++++++++++++++---
 14 files changed, 211 insertions(+), 95 deletions(-)
 create mode 100644 src/translator/aligned.h
 delete mode 100644 src/translator/byteArrayExample.cpp
 delete mode 100644 src/translator/byteArrayExample.h
 create mode 100644 src/translator/byte_array_util.cpp
 create mode 100644 src/translator/byte_array_util.h

diff --git a/app/bergamot-translator-app-bytearray.cpp b/app/bergamot-translator-app-bytearray.cpp
index b58c638a3..215b57360 100644
--- a/app/bergamot-translator-app-bytearray.cpp
+++ b/app/bergamot-translator-app-bytearray.cpp
@@ -9,7 +9,7 @@
 
 #include "TranslationModel.h"
 #include "translator/parser.h"
-#include "translator/byteArrayExample.h"
+#include "translator/byte_array_util.h"
 
 int main(int argc, char **argv) {
 
@@ -19,9 +19,11 @@ int main(int argc, char **argv) {
   auto options = configParser.parseOptions(argc, argv, true);
   std::string config = options->asYamlString();
 
+  // Prepare model byte array
+  marian::bergamot::AlignedMemory modelBytes = marian::bergamot::getModelMemoryFromConfig(options);
+
   // Route the config string to construct marian model through TranslationModel
-  void * model_bytes = bergamot::getBinaryModelFromConfig(options);
-  auto model = std::make_shared<TranslationModel>(config, model_bytes);
+  TranslationModel model(config, modelBytes.begin());
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
@@ -42,7 +44,7 @@ int main(int argc, char **argv) {
       "Prague, the University of Sheffield, University of Tartu, and "
       "Mozilla.");
 
-  auto results = model->translate(std::move(texts), translationRequest);
+  auto results = model.translate(std::move(texts), translationRequest);
 
   // Resolve the future and get the actual result
   //std::vector<TranslationResult> results = futureResults.get();
@@ -61,8 +63,5 @@ int main(int argc, char **argv) {
     std::cout << std::endl;
   }
 
-  // Clear the memory used for the byte array
-  free(model_bytes); // Ideally, this should be done after the translation model has been gracefully shut down.
-
   return 0;
 }
diff --git a/app/service-cli-bytearray.cpp b/app/service-cli-bytearray.cpp
index cb3b17ff0..6b3948b7a 100644
--- a/app/service-cli-bytearray.cpp
+++ b/app/service-cli-bytearray.cpp
@@ -9,14 +9,17 @@
 #include "translator/parser.h"
 #include "translator/response.h"
 #include "translator/service.h"
-#include "translator/byteArrayExample.h"
+#include "translator/byte_array_util.h"
 
 int main(int argc, char *argv[]) {
   auto cp = marian::bergamot::createConfigParser();
   auto options = cp.parseOptions(argc, argv, true);
 
-  void * model_bytes = bergamot::getBinaryModelFromConfig(options);
-  marian::bergamot::Service service(options, model_bytes);
+  // Prepare memories for model and shortlist
+  marian::bergamot::AlignedMemory modelBytes = marian::bergamot::getModelMemoryFromConfig(options);
+  marian::bergamot::AlignedMemory shortlistBytes = marian::bergamot::getShortlistMemoryFromConfig(options);
+
+  marian::bergamot::Service service(options, std::move(modelBytes), std::move(shortlistBytes));
 
   // Read a large input text blob from stdin
   std::ostringstream std_input;
@@ -30,8 +33,5 @@ int main(int argc, char *argv[]) {
   Response response = responseFuture.get();
   std::cout << response.target.text << std::endl;
 
-  // Clear the memory used for the byte array
-  free(model_bytes); // Ideally, this should be done after the translation model has been gracefully shut down.
-
   return 0;
 }
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index a34142787..3ddfa7956 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,7 +1,6 @@
 add_library(bergamot-translator STATIC
     TranslationModel.cpp
-
-    byteArrayExample.cpp
+    byte_array_util.cpp
     text_processor.cpp
     sentence_splitter.cpp
     batch_translator.cpp 
diff --git a/src/translator/aligned.h b/src/translator/aligned.h
new file mode 100644
index 000000000..6edb84e35
--- /dev/null
+++ b/src/translator/aligned.h
@@ -0,0 +1,71 @@
+#pragma once
+#include <cstdlib>
+#include <new>
+#ifdef _MSC_VER
+#include <malloc.h>
+#endif
+
+// Aligned simple vector.
+
+namespace marian {
+namespace bergamot {
+
+template <class T> class AlignedVector {
+public:
+  AlignedVector() : mem_(nullptr), size_(0) {}
+
+  explicit AlignedVector(std::size_t size, std::size_t alignment = 64 /* CPU cares about this */)
+          : size_(size) {
+#ifdef _MSC_VER
+    mem_ = static_cast<T*>(_aligned_malloc(size * sizeof(T), alignment));
+      if (!mem_) throw std::bad_alloc();
+#else
+    if (posix_memalign(reinterpret_cast<void **>(&mem_), alignment, size * sizeof(T))) {
+      throw std::bad_alloc();
+    }
+#endif
+  }
+
+  AlignedVector(AlignedVector &&from) : mem_(from.mem_), size_(from.size_) {
+    from.mem_ = nullptr;
+    from.size_ = 0;
+  }
+
+  AlignedVector &operator=(AlignedVector &&from) {
+    mem_ = from.mem_;
+    size_ = from.size_;
+    from.mem_ = nullptr;
+    from.size_ = 0;
+    return *this;
+  }
+
+  AlignedVector(const AlignedVector&) = delete;
+  AlignedVector& operator=(const AlignedVector&) = delete;
+
+  ~AlignedVector() {
+#ifdef _MSC_VER
+    _aligned_free(mem_);
+#else
+    std::free(mem_);
+#endif
+  }
+
+  std::size_t size() const { return size_; }
+
+  T &operator[](std::size_t offset) { return mem_[offset]; }
+  const T &operator[](std::size_t offset) const { return mem_[offset]; }
+
+  T *begin() { return mem_; }
+  const T *begin() const { return mem_; }
+  T *end() { return mem_ + size_; }
+  const T *end() const { return mem_ + size_; }
+
+  template <typename ReturnType>
+  ReturnType *as() { return reinterpret_cast<ReturnType*>(mem_); }
+
+private:
+  T *mem_;
+  std::size_t size_;
+};
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index c83cf8cd6..8278ca729 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -11,17 +11,29 @@ namespace bergamot {
 BatchTranslator::BatchTranslator(DeviceId const device,
                                  std::vector<Ptr<Vocab const>> &vocabs,
                                  Ptr<Options> options,
-                                 const void * model_memory)
-    : device_(device), options_(options), vocabs_(&vocabs), model_memory_(model_memory) {}
+                                 const AlignedMemory* modelMemory,
+                                 const AlignedMemory* shortlistMemory)
+    : device_(device), options_(options), vocabs_(&vocabs),
+    modelMemory_(modelMemory), shortlistMemory_(shortlistMemory) {}
 
 void BatchTranslator::initialize() {
   // Initializes the graph.
   if (options_->hasAndNotEmpty("shortlist")) {
     int srcIdx = 0, trgIdx = 1;
     bool shared_vcb = vocabs_->front() == vocabs_->back();
-    slgen_ = New<data::LexicalShortlistGenerator>(options_, vocabs_->front(),
-                                                  vocabs_->back(), srcIdx,
-                                                  trgIdx, shared_vcb);
+    if (shortlistMemory_->size() > 0 && shortlistMemory_->begin() != nullptr) {
+      bool check = options_->get<bool>("check-bytearray",true);
+      slgen_ = New<data::BinaryShortlistGenerator>(shortlistMemory_->begin(), shortlistMemory_->size(),
+                                                     vocabs_->front(), vocabs_->back(),
+                                                     srcIdx, trgIdx, shared_vcb, check);
+    }
+    else {
+      // Changed to BinaryShortlistGenerator to enable loading binary shortlist file
+      // This class also supports text shortlist file
+      slgen_ = New<data::BinaryShortlistGenerator>(options_, vocabs_->front(),
+                                                    vocabs_->back(), srcIdx,
+                                                    trgIdx, shared_vcb);
+    }
   }
 
   graph_ = New<ExpressionGraph>(true); // always optimize
@@ -30,12 +42,10 @@ void BatchTranslator::initialize() {
   graph_->setDevice(device_);
   graph_->getBackend()->configureDevice(options_);
   graph_->reserveWorkspaceMB(options_->get<size_t>("workspace"));
-  if (model_memory_) { // If we have provided a byte array that contains the model memory, we can initialise the model from there, as opposed to from reading in the config file
-    if ((uintptr_t)model_memory_ % 256 != 0) {
-      std::cerr << "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it." << std::endl;
-      exit(1);
-    }
-    const std::vector<const void *> container = {model_memory_}; // Marian supports multiple models initialised in this manner hence std::vector. However we will only ever use 1 during decoding.
+  if (modelMemory_->size() > 0 && modelMemory_->begin() != nullptr) { // If we have provided a byte array that contains the model memory, we can initialise the model from there, as opposed to from reading in the config file
+    ABORT_IF((uintptr_t)modelMemory_->begin() % 256 != 0,
+             "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
+    const std::vector<const void *> container = {modelMemory_->begin()}; // Marian supports multiple models initialised in this manner hence std::vector. However we will only ever use 1 during decoding.
     scorers_ = createScorers(options_, container);
   } else {
     scorers_ = createScorers(options_);
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 4e17b6514..761a53449 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -31,10 +31,11 @@ class BatchTranslator {
    * @param device DeviceId that performs translation. Could be CPU or GPU
    * @param vocabs Vector that contains ptrs to two vocabs
    * @param options Marian options object
-   * @param model_memory byte array (aligned to 64!!!) that contains the bytes of a model.bin. Provide a nullptr if not used.
+   * @param modelMemory byte array (aligned to 256!!!) that contains the bytes of a model.bin. Provide a nullptr if not used.
+   * @param shortlistMemory byte array of shortlist (aligned to 64)
    */
   explicit BatchTranslator(DeviceId const device, std::vector<Ptr<Vocab const>> &vocabs,
-                  Ptr<Options> options, const void * model_memory);
+                  Ptr<Options> options, const AlignedMemory* modelMemory, const AlignedMemory* shortlistMemory);
 
   // convenience function for logging. TODO(jerin)
   std::string _identifier() { return "worker" + std::to_string(device_.no); }
@@ -48,7 +49,8 @@ class BatchTranslator {
   Ptr<ExpressionGraph> graph_;
   std::vector<Ptr<Scorer>> scorers_;
   Ptr<data::ShortlistGenerator const> slgen_;
-  const void * model_memory_;
+  const AlignedMemory* modelMemory_{nullptr};
+  const AlignedMemory* shortlistMemory_{nullptr};
 };
 
 } // namespace bergamot
diff --git a/src/translator/byteArrayExample.cpp b/src/translator/byteArrayExample.cpp
deleted file mode 100644
index 28f9d9ba4..000000000
--- a/src/translator/byteArrayExample.cpp
+++ /dev/null
@@ -1,45 +0,0 @@
-#include "byteArrayExample.h"
-#include <stdlib.h>
-#include <fstream>
-#include <iostream>
-
-namespace bergamot {
-
-void * getBinaryFile(std::string path) {
-    std::ifstream is (path, std::ifstream::binary);
-    uint64_t length = 0; // Determine the length of file in bytes
-    if (is) {
-        is.seekg(0, is.end);
-        length = is.tellg();
-        is.seekg(0, is.beg);
-    } else {
-        std::cerr << "Failed opening file stream: " << path << std::endl;
-        std::exit(1);
-    }
-    void *result;
-    int fail = posix_memalign(&result, 256, length);
-    if (fail) {
-        std::cerr << "Failed to allocate aligned memory." << std::endl;
-        std::exit(1);
-    }
-    is.read(static_cast<char *>(result), length);
-    return result;
-}
-
-void * getBinaryModelFromConfig(marian::Ptr<marian::Options> options) {
-    std::vector<std::string> models = options->get<std::vector<std::string>>("models");
-    if (models.size() != 1) {
-        std::cerr << "Loading multiple binary models is not supported for now as it is not necessary." << std::endl;
-        std::exit(1);
-        marian::filesystem::Path modelPath(models[0]);
-        if (modelPath.extension() != marian::filesystem::Path(".bin")) {
-            std::cerr << "Non binary models cannot be loaded as a byte array." << std::endl;
-            std::exit(1);
-        }
-        return nullptr;
-    } else {
-        return getBinaryFile(models[0]);
-    }
-}
-
-} // namespace bergamot
diff --git a/src/translator/byteArrayExample.h b/src/translator/byteArrayExample.h
deleted file mode 100644
index 321ea5d5f..000000000
--- a/src/translator/byteArrayExample.h
+++ /dev/null
@@ -1,8 +0,0 @@
-#include "marian.h"
-
-namespace bergamot {
-
-void * getBinaryFile(std::string path);
-void * getBinaryModelFromConfig(marian::Ptr<marian::Options> options);
-
-} // namespace bergamot
diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
new file mode 100644
index 000000000..b46461241
--- /dev/null
+++ b/src/translator/byte_array_util.cpp
@@ -0,0 +1,33 @@
+#include "byte_array_util.h"
+#include <stdlib.h>
+#include <iostream>
+
+namespace marian {
+namespace bergamot {
+
+AlignedMemory loadFileToMemory(const std::string& path, size_t alignment){
+  uint64_t fileSize = filesystem::fileSize(path);
+  io::InputFileStream in(path);
+  ABORT_IF(in.bad(), "Failed opening file stream: {}", path);
+  AlignedMemory alignedMemory(fileSize, alignment);
+  in.read(reinterpret_cast<char *>(alignedMemory.begin()), fileSize);
+  ABORT_IF(alignedMemory.size() != fileSize, "Error reading file {}", path);
+  return alignedMemory;
+}
+
+AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options){
+    auto models = options->get<std::vector<std::string>>("models");
+    ABORT_IF(models.size() != 1, "Loading multiple binary models is not supported for now as it is not necessary.");
+    marian::filesystem::Path modelPath(models[0]);
+    ABORT_IF(modelPath.extension() != marian::filesystem::Path(".bin"), "The file of binary model should end with .bin");
+    return loadFileToMemory(models[0], 256);
+}
+
+AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options){
+  auto shortlist = options->get<std::vector<std::string>>("shortlist");
+  ABORT_IF(shortlist.empty(), "No path to shortlist file is given.");
+  return loadFileToMemory(shortlist[0], 64);
+}
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/byte_array_util.h b/src/translator/byte_array_util.h
new file mode 100644
index 000000000..a8df1cbb0
--- /dev/null
+++ b/src/translator/byte_array_util.h
@@ -0,0 +1,12 @@
+#include "marian.h"
+#include "definitions.h"
+
+namespace marian {
+namespace bergamot {
+
+AlignedMemory loadFileToMemory(const std::string& path, size_t alignment);
+AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options);
+AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options);
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index 35797a2b4..32998b9e9 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -3,6 +3,7 @@
 
 #include "data/types.h"
 #include "data/vocab_base.h"
+#include "aligned.h"
 #include <vector>
 
 namespace marian {
@@ -21,6 +22,9 @@ template <class T, typename... Args> UPtr<T> UNew(Args &&... args) {
 
 template <class T> UPtr<T> UNew(UPtr<T> p) { return UPtr<T>(p); }
 
+/// Shortcut to AlignedVector<const void*> for byte arrays
+typedef AlignedVector<const void*> AlignedMemory;
+
 } // namespace bergamot
 } // namespace marian
 
diff --git a/src/translator/parser.h b/src/translator/parser.h
index 4d93e3aa9..fa4e7bbc8 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -23,7 +23,11 @@ inline marian::ConfigParser createConfigParser() {
       "--max-length-break", "Bergamot Options",
       "Maximum input tokens to be processed in a single sentence.", 128);
 
-  return cp;
+  cp.addOption<bool>(
+      "--check-bytearray", "Bergamot Options",
+      "Flag holds whether to check the content of the bytearray (true by default)", true);
+
+    return cp;
 }
 
 inline std::shared_ptr<marian::Options>
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 718bc4dda..76bcba296 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -28,10 +28,11 @@ loadVocabularies(marian::Ptr<marian::Options> options) {
 namespace marian {
 namespace bergamot {
 
-Service::Service(Ptr<Options> options, const void *model_memory)
+Service::Service(Ptr<Options> options, AlignedMemory modelMemory, AlignedMemory shortlistMemory)
     : requestId_(0), vocabs_(std::move(loadVocabularies(options))),
       text_processor_(vocabs_, options), batcher_(options),
-      numWorkers_(options->get<int>("cpu-threads")), model_memory_(model_memory)
+      numWorkers_(options->get<int>("cpu-threads")),
+      modelMemory_(std::move(modelMemory)), shortlistMemory_(std::move(shortlistMemory))
 #ifndef WASM_COMPATIBLE_SOURCE
       // 0 elements in PCQueue is illegal and can lead to failures. Adding a
       // guard to have at least one entry allocated. In the single-threaded
@@ -54,7 +55,7 @@ void Service::build_translators(Ptr<Options> options, size_t numTranslators) {
   translators_.reserve(numTranslators);
   for (size_t cpuId = 0; cpuId < numTranslators; cpuId++) {
     marian::DeviceId deviceId(cpuId, DeviceType::cpu);
-    translators_.emplace_back(deviceId, vocabs_, options, model_memory_);
+    translators_.emplace_back(deviceId, vocabs_, options, &modelMemory_, &shortlistMemory_);
   }
 }
 
diff --git a/src/translator/service.h b/src/translator/service.h
index 59f2f4d86..8fc1de736 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -18,6 +18,33 @@
 namespace marian {
 namespace bergamot {
 
+// Hack code to construct AlignedMemory* from void*
+inline AlignedMemory hackModel(const void* modelMemory) {
+  if(modelMemory != nullptr){
+    // Here is a hack to make TranslationModel works
+    size_t modelMemorySize = 73837568;   // Hack: model memory size should be changed to actual model size
+    AlignedMemory alignedMemory(modelMemorySize);
+    memcpy(alignedMemory.begin(), modelMemory, modelMemorySize);
+    return alignedMemory;
+  } else {
+    return AlignedMemory();
+  }
+}
+
+inline AlignedMemory hackShortLis(const void* shortlistMemory) {
+  if(shortlistMemory!= nullptr) {
+    // Hacks to obtain shortlist memory size as this will be checked during construction
+    size_t shortlistMemorySize = sizeof(uint64_t) * (6 + *((uint64_t*)shortlistMemory+4))
+                                 + sizeof(uint32_t) * *((uint64_t*)shortlistMemory+5);
+    // Here is a hack to make TranslationModel works
+    AlignedMemory alignedMemory(shortlistMemorySize);
+    memcpy(alignedMemory.begin(), shortlistMemory, shortlistMemorySize);
+    return alignedMemory;
+  }else {
+    return AlignedMemory();
+  }
+}
+
 /// Service exposes methods to translate an incoming blob of text to the
 /// Consumer of bergamot API.
 ///
@@ -38,18 +65,22 @@ class Service {
 
 public:
   /// @param options Marian options object
-  /// @param model_memory byte array (aligned to 64!!!) that contains the bytes
+  /// @param modelMemory byte array (aligned to 256!!!) that contains the bytes
   /// of a model.bin. Optional, defaults to nullptr when not used
-  explicit Service(Ptr<Options> options, const void *model_memory = nullptr);
+  /// @param shortlistMemory byte array of shortlist (aligned to 64)
+  explicit Service(Ptr<Options> options, AlignedMemory modelMemory, AlignedMemory shortlistMemory);
+
+  explicit Service(Ptr<Options> options) : Service(options, AlignedMemory(), AlignedMemory()){}
 
-  /// Construct Service from a string configuration.
+/// Construct Service from a string configuration.
   /// @param [in] config string parsable as YAML expected to adhere with marian
   /// config
-  /// @param [in] model_memory byte array (aligned to 64!!!) that contains the
+  /// @param [in] model_memory byte array (aligned to 256!!!) that contains the
   /// bytes of a model.bin. Optional, defaults to nullptr when not used
+  /// @param [in] shortlistMemory byte array of shortlist (aligned to 64)
   explicit Service(const std::string &config,
-                   const void *model_memory = nullptr)
-      : Service(parseOptions(config), model_memory) {}
+                   const void* modelMemory = nullptr, const void* shortlistMemory = nullptr)
+      : Service(parseOptions(config), hackModel(modelMemory), hackShortLis(shortlistMemory)) {}
 
   /// Explicit destructor to clean up after any threads initialized in
   /// asynchronous operation mode.
@@ -85,13 +116,16 @@ class Service {
   void async_translate();
 
   /// Number of workers to launch.
-  size_t numWorkers_;        // ORDER DEPENDENCY (pcqueue_)
-  const void *model_memory_; /// Model memory to load model passed as bytes.
+  size_t numWorkers_;              // ORDER DEPENDENCY (pcqueue_)
+  /// Model memory to load model passed as bytes.
+  AlignedMemory modelMemory_;      // ORDER DEPENDENCY (translators_)
+  /// Shortlist memory passed as bytes.
+  AlignedMemory shortlistMemory_;  // ORDER DEPENDENCY (translators_)
 
   /// Holds instances of batch translators, just one in case
   /// of single-threaded application, numWorkers_ in case of multithreaded
   /// setting.
-  std::vector<BatchTranslator> translators_;
+  std::vector<BatchTranslator> translators_;  // ORDER DEPENDENCY (modelMemory_, shortlistMemory_)
 
   /// Stores requestId of active request. Used to establish
   /// ordering among requests and logging/book-keeping.

From 27a3a3253f5c1d83a5c45dc43d8ec65afdaeb459 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Tue, 6 Apr 2021 13:23:55 +0100
Subject: [PATCH 187/442] Make AlignedMemory the means of passing in memory
 (#86)

---
 app/bergamot-translator-app-bytearray.cpp |  5 +---
 src/TranslationModel.h                    |  4 ++-
 src/translator/TranslationModel.cpp       |  5 ++--
 src/translator/service.h                  | 35 +++--------------------
 4 files changed, 11 insertions(+), 38 deletions(-)

diff --git a/app/bergamot-translator-app-bytearray.cpp b/app/bergamot-translator-app-bytearray.cpp
index 215b57360..518624203 100644
--- a/app/bergamot-translator-app-bytearray.cpp
+++ b/app/bergamot-translator-app-bytearray.cpp
@@ -19,11 +19,8 @@ int main(int argc, char **argv) {
   auto options = configParser.parseOptions(argc, argv, true);
   std::string config = options->asYamlString();
 
-  // Prepare model byte array
-  marian::bergamot::AlignedMemory modelBytes = marian::bergamot::getModelMemoryFromConfig(options);
-
   // Route the config string to construct marian model through TranslationModel
-  TranslationModel model(config, modelBytes.begin());
+  TranslationModel model(config, marian::bergamot::getModelMemoryFromConfig(options));
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
diff --git a/src/TranslationModel.h b/src/TranslationModel.h
index db0bdbf5c..4b1be2348 100644
--- a/src/TranslationModel.h
+++ b/src/TranslationModel.h
@@ -17,6 +17,7 @@
 // All local project includes
 #include "TranslationRequest.h"
 #include "TranslationResult.h"
+#include "translator/definitions.h"
 #include "translator/service.h"
 
 /* A Translation model that translates a plain (without any markups and emojis)
@@ -34,7 +35,8 @@ class TranslationModel {
    * the bytes of a model.bin.
    */
   TranslationModel(const std::string &config,
-                   const void *model_memory = nullptr);
+                   marian::bergamot::AlignedMemory modelMemory = marian::bergamot::AlignedMemory(),
+                   marian::bergamot::AlignedMemory shortlistMemory = marian::bergamot::AlignedMemory());
 
   ~TranslationModel();
 
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 13f949578..06b04eb5a 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -12,8 +12,9 @@
 #include "translator/service.h"
 
 TranslationModel::TranslationModel(const std::string &config,
-                                   const void *model_memory)
-    : service_(config, model_memory) {}
+                                   marian::bergamot::AlignedMemory model_memory,
+                                   marian::bergamot::AlignedMemory lexical_memory)
+    : service_(config, std::move(model_memory), std::move(lexical_memory)) {}
 
 TranslationModel::~TranslationModel() {}
 
diff --git a/src/translator/service.h b/src/translator/service.h
index 8fc1de736..f3502daae 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -18,33 +18,6 @@
 namespace marian {
 namespace bergamot {
 
-// Hack code to construct AlignedMemory* from void*
-inline AlignedMemory hackModel(const void* modelMemory) {
-  if(modelMemory != nullptr){
-    // Here is a hack to make TranslationModel works
-    size_t modelMemorySize = 73837568;   // Hack: model memory size should be changed to actual model size
-    AlignedMemory alignedMemory(modelMemorySize);
-    memcpy(alignedMemory.begin(), modelMemory, modelMemorySize);
-    return alignedMemory;
-  } else {
-    return AlignedMemory();
-  }
-}
-
-inline AlignedMemory hackShortLis(const void* shortlistMemory) {
-  if(shortlistMemory!= nullptr) {
-    // Hacks to obtain shortlist memory size as this will be checked during construction
-    size_t shortlistMemorySize = sizeof(uint64_t) * (6 + *((uint64_t*)shortlistMemory+4))
-                                 + sizeof(uint32_t) * *((uint64_t*)shortlistMemory+5);
-    // Here is a hack to make TranslationModel works
-    AlignedMemory alignedMemory(shortlistMemorySize);
-    memcpy(alignedMemory.begin(), shortlistMemory, shortlistMemorySize);
-    return alignedMemory;
-  }else {
-    return AlignedMemory();
-  }
-}
-
 /// Service exposes methods to translate an incoming blob of text to the
 /// Consumer of bergamot API.
 ///
@@ -72,15 +45,15 @@ class Service {
 
   explicit Service(Ptr<Options> options) : Service(options, AlignedMemory(), AlignedMemory()){}
 
-/// Construct Service from a string configuration.
+  /// Construct Service from a string configuration.
   /// @param [in] config string parsable as YAML expected to adhere with marian
   /// config
   /// @param [in] model_memory byte array (aligned to 256!!!) that contains the
-  /// bytes of a model.bin. Optional, defaults to nullptr when not used
+  /// bytes of a model.bin. Optional.
   /// @param [in] shortlistMemory byte array of shortlist (aligned to 64)
   explicit Service(const std::string &config,
-                   const void* modelMemory = nullptr, const void* shortlistMemory = nullptr)
-      : Service(parseOptions(config), hackModel(modelMemory), hackShortLis(shortlistMemory)) {}
+                   AlignedMemory modelMemory = AlignedMemory(), AlignedMemory shortlistMemory = AlignedMemory())
+      : Service(parseOptions(config), std::move(modelMemory), std::move(shortlistMemory)) {}
 
   /// Explicit destructor to clean up after any threads initialized in
   /// asynchronous operation mode.

From b71b3a18d815de21adcf4b222e1b76f23bf50ee7 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 7 Apr 2021 12:15:46 +0100
Subject: [PATCH 188/442] Removes vocabs and propogates fixes for breaks (#79)

* Removes vocabs and propogates fixes for breaks

* Prettify diff: Undoing comment shuffles due to merge conflict edits

* 20% of time actual work, 80% prettifying diff

* Histories members -> poof!

We however have Histories in constructor, which we will remove out of the way
soon.

Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
---
 app/marian-decoder-new.cpp  | 24 +++++-------------------
 src/translator/response.cpp |  4 ++--
 src/translator/response.h   | 10 +---------
 src/translator/service.h    |  6 ------
 4 files changed, 8 insertions(+), 36 deletions(-)

diff --git a/app/marian-decoder-new.cpp b/app/marian-decoder-new.cpp
index dea49eb8b..a2516d9c2 100644
--- a/app/marian-decoder-new.cpp
+++ b/app/marian-decoder-new.cpp
@@ -14,25 +14,11 @@
 #include "translator/response.h"
 #include "translator/service.h"
 
-void marian_decoder_minimal(const marian::Histories &histories,
-                            marian::Ptr<marian::Vocab const> targetVocab,
+void marian_decoder_minimal(const marian::bergamot::Response &response,
                             marian::Ptr<marian::Options> options) {
-
-  bool doNbest = options->get<bool>("n-best");
-  auto collector =
-      marian::New<marian::OutputCollector>(options->get<std::string>("output"));
-
-  // There is a dependency of vocabs here.
-  auto printer = marian::New<marian::OutputPrinter>(options, targetVocab);
-  if (options->get<bool>("quiet-translation"))
-    collector->setPrintingStrategy(marian::New<marian::QuietPrinting>());
-
-  for (auto &history : histories) {
-    std::stringstream best1;
-    std::stringstream bestn;
-    printer->print(history, best1, bestn);
-    collector->Write((long)history->getLineNum(), best1.str(), bestn.str(),
-                     doNbest);
+  // We are no longer marian-decoder compatible. Server ideas are on hold.
+  for (size_t sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
+    std::cout << response.target.sentence(sentenceIdx) << "\n";
   }
 }
 
@@ -53,7 +39,7 @@ int main(int argc, char *argv[]) {
   responseFuture.wait();
   const Response &response = responseFuture.get();
 
-  marian_decoder_minimal(response.histories(), service.targetVocab(), options);
+  marian_decoder_minimal(response, options);
 
   LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
   return 0;
diff --git a/src/translator/response.cpp b/src/translator/response.cpp
index faa42da00..e5bc38ff9 100644
--- a/src/translator/response.cpp
+++ b/src/translator/response.cpp
@@ -10,7 +10,7 @@ namespace bergamot {
 
 Response::Response(AnnotatedText &&source, Histories &&histories,
                    std::vector<Ptr<Vocab const>> &vocabs)
-    : source(std::move(source)), histories_(std::move(histories)) {
+    : source(std::move(source)) {
   // Reserving length at least as much as source_ seems like a reasonable thing
   // to do to avoid reallocations.
   target.text.reserve(source.text.size());
@@ -24,7 +24,7 @@ Response::Response(AnnotatedText &&source, Histories &&histories,
   size_t offset{0};
   bool first{true};
 
-  for (auto &history : histories_) {
+  for (auto &history : histories) {
     // TODO(jerin): Change hardcode of nBest = 1
     NBestList onebest = history->nBest(1);
 
diff --git a/src/translator/response.h b/src/translator/response.h
index 385735c88..4f87b8dac 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -52,8 +52,7 @@ class Response {
   Response(Response &&other)
       : source(std::move(other.source)), target(std::move(other.target)),
         alignments(std::move(other.alignments)),
-        qualityScores(std::move(other.qualityScores)),
-        histories_(std::move(other.histories_)){};
+        qualityScores(std::move(other.qualityScores)){};
 
   // The following copy bans are not stricitly required anymore since Annotation
   // is composed of the ByteRange primitive (which was previously string_view
@@ -87,13 +86,6 @@ class Response {
   /// sparse matrix representation with indices corresponding
   /// to (sub-)words accessible through Annotation.
   std::vector<Alignment> alignments;
-
-  /// Access to histories, which holds rich information on translated text.
-  /// Not recommended to use, will be removed in future.
-  const Histories &histories() const { return histories_; }
-
-private:
-  Histories histories_;
 };
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/service.h b/src/translator/service.h
index f3502daae..72f6d92ae 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -59,12 +59,6 @@ class Service {
   /// asynchronous operation mode.
   ~Service();
 
-  /// Shared pointer to source-vocabulary.
-  Ptr<Vocab const> sourceVocab() const { return vocabs_.front(); }
-
-  /// Shared pointer to target vocabulary.
-  Ptr<Vocab const> targetVocab() const { return vocabs_.back(); }
-
   /// To stay efficient and to refer to the string for alignments, expects
   /// ownership be moved through std::move(..)
   ///

From 5e15d73b7eaa6df79b74b4e7be2e5390687ecfd8 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Fri, 9 Apr 2021 09:34:24 +0100
Subject: [PATCH 189/442] Consistent api usage (#91)

* Consistent api between the two versions of the executables in app folder

* Remove shared ptrs
---
 app/bergamot-translator-app-bytearray.cpp | 32 +++-----------
 app/bergamot-translator-app.cpp           |  4 +-
 app/service-cli-bytearray.cpp             | 51 ++++++++++++++++++++++-
 3 files changed, 57 insertions(+), 30 deletions(-)

diff --git a/app/bergamot-translator-app-bytearray.cpp b/app/bergamot-translator-app-bytearray.cpp
index 518624203..1fa574839 100644
--- a/app/bergamot-translator-app-bytearray.cpp
+++ b/app/bergamot-translator-app-bytearray.cpp
@@ -24,22 +24,10 @@ int main(int argc, char **argv) {
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
-  texts.emplace_back(
-      "The Bergamot project will add and improve client-side machine "
-      "translation in a web browser.  Unlike current cloud-based "
-      "options, running directly on users’ machines empowers citizens to "
-      "preserve their privacy and increases the uptake of language "
-      "technologies in Europe in various sectors that require "
-      "confidentiality.");
-  texts.emplace_back(
-      "Free software integrated with an open-source web "
-      "browser, such as Mozilla Firefox, will enable bottom-up adoption "
-      "by non-experts, resulting in cost savings for private and public "
-      "sector users who would otherwise procure translation or operate "
-      "monolingually.  Bergamot is a consortium coordinated by the "
-      "University of Edinburgh with partners Charles University in "
-      "Prague, the University of Sheffield, University of Tartu, and "
-      "Mozilla.");
+
+  for (std::string line; std::getline(std::cin, line);) {
+        texts.emplace_back(line);
+  }
 
   auto results = model.translate(std::move(texts), translationRequest);
 
@@ -47,17 +35,7 @@ int main(int argc, char **argv) {
   //std::vector<TranslationResult> results = futureResults.get();
 
   for (auto &result : results) {
-    std::cout << "[original]: " << result.getOriginalText() << std::endl;
-    std::cout << "[translated]: " << result.getTranslatedText() << std::endl;
-    auto mappings = result.getSentenceMappings();
-    for (auto &p : mappings) {
-      std::string_view src = p.first;
-      std::string_view tgt = p.second;
-
-      std::cout << " [src Sentence]: " << src << std::endl;
-      std::cout << " [tgt Sentence]: " << tgt << std::endl;
-    }
-    std::cout << std::endl;
+    std::cout << result.getTranslatedText() << std::endl;
   }
 
   return 0;
diff --git a/app/bergamot-translator-app.cpp b/app/bergamot-translator-app.cpp
index 086b9ca20..4fba00b46 100644
--- a/app/bergamot-translator-app.cpp
+++ b/app/bergamot-translator-app.cpp
@@ -21,7 +21,7 @@ int main(int argc, char **argv) {
   std::string config = options->asYamlString();
 
   // Route the config string to construct marian model through TranslationModel
-  auto model = std::make_shared<TranslationModel>(config);
+  TranslationModel model(config);
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
@@ -30,7 +30,7 @@ int main(int argc, char **argv) {
         texts.emplace_back(line);
   }
 
-  auto results = model->translate(std::move(texts), translationRequest);
+  auto results = model.translate(std::move(texts), translationRequest);
 
   // Resolve the future and get the actual result
   //std::vector<TranslationResult> results = futureResults.get();
diff --git a/app/service-cli-bytearray.cpp b/app/service-cli-bytearray.cpp
index 6b3948b7a..f868d4def 100644
--- a/app/service-cli-bytearray.cpp
+++ b/app/service-cli-bytearray.cpp
@@ -31,7 +31,56 @@ int main(int argc, char *argv[]) {
   std::future<Response> responseFuture = service.translate(std::move(input));
   responseFuture.wait();
   Response response = responseFuture.get();
-  std::cout << response.target.text << std::endl;
+
+  std::cout << "[original]: " << response.source.text << '\n';
+  std::cout << "[translated]: " << response.target.text << '\n';
+  for (int sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
+    std::cout << " [src Sentence]: " << response.source.sentence(sentenceIdx)
+              << '\n';
+    std::cout << " [tgt Sentence]: " << response.target.sentence(sentenceIdx)
+              << '\n';
+    std::cout << "Alignments" << '\n';
+    typedef std::pair<size_t, float> Point;
+
+    // Initialize a point vector.
+    std::vector<std::vector<Point>> aggregate(
+        response.source.numWords(sentenceIdx));
+
+    // Handle alignments
+    auto &alignments = response.alignments[sentenceIdx];
+    for (auto &p : alignments) {
+      aggregate[p.src].emplace_back(p.tgt, p.prob);
+    }
+
+    for (size_t src = 0; src < aggregate.size(); src++) {
+      std::cout << response.source.word(sentenceIdx, src) << ": ";
+      for (auto &p : aggregate[src]) {
+        std::cout << response.target.word(sentenceIdx, p.first) << "("
+                  << p.second << ") ";
+      }
+      std::cout << '\n';
+    }
+
+    // Handle quality.
+    auto &quality = response.qualityScores[sentenceIdx];
+    std::cout << "Quality: whole(" << quality.sequence
+              << "), tokens below:" << '\n';
+    size_t wordIdx = 0;
+    bool first = true;
+    for (auto &p : quality.word) {
+      if (first) {
+        first = false;
+      } else {
+        std::cout << " ";
+      }
+      std::cout << response.target.word(sentenceIdx, wordIdx) << "(" << p
+                << ")";
+      wordIdx++;
+    }
+    std::cout << '\n';
+  }
+  std::cout << "--------------------------\n";
+  std::cout << '\n';
 
   return 0;
 }

From b345b0e0354873515f474f16bb1346bddefeda9d Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Mon, 12 Apr 2021 12:46:47 +0100
Subject: [PATCH 190/442] Rudimentary validator for binary files (#94)

* Rudimentary validator for binary files
---
 src/translator/byte_array_util.cpp | 77 +++++++++++++++++++++++++++++-
 1 file changed, 76 insertions(+), 1 deletion(-)

diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index b46461241..c3bf7cc69 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -5,6 +5,79 @@
 namespace marian {
 namespace bergamot {
 
+namespace {
+
+// This is a basic validator that checks if the file has not been truncated
+// it basically loads up the header and checks
+
+// This struct and the getter are copied from the marian source, because it's located
+// inside src/common/binary.cpp:15 and we can't include it.
+struct Header {
+  uint64_t nameLength;
+  uint64_t type;
+  uint64_t shapeLength;
+  uint64_t dataLength;
+};
+
+// cast current void pointer to T pointer and move forward by num elements
+template <typename T>
+const T* get(const void*& current, uint64_t num = 1) {
+  const T* ptr = (const T*)current;
+  current = (const T*)current + num;
+  return ptr;
+}
+
+bool validateBinaryModel(AlignedMemory& model, uint64_t fileSize) {
+  const void * current = &model[0];
+  uint64_t memoryNeeded = sizeof(uint64_t)*2; // We keep track of how much memory we would need if we have a complete file
+  uint64_t numHeaders;
+  if (fileSize >= memoryNeeded) { // We have enough filesize to fetch the headers.
+    uint64_t binaryFileVersion = *get<uint64_t>(current);
+    numHeaders = *get<uint64_t>(current); // number of item headers that follow
+  } else {
+    return false;
+  }
+  memoryNeeded += numHeaders*sizeof(Header);
+  const Header* headers;
+  if (fileSize >= memoryNeeded) {
+    headers = get<Header>(current, numHeaders); // read that many headers
+  } else {
+    return false;
+  }
+
+  // Calculate how many bytes we are going to for reading just the names and the shape
+  for (uint64_t i = 0; i < numHeaders; i++) {
+    memoryNeeded += headers[i].nameLength + headers[i].shapeLength*sizeof(int);
+    // Advance the pointers.
+    get<char>(current, headers[i].nameLength);
+    get<int>(current, headers[i].shapeLength);
+  }
+
+  // Before we start reading the data, there is a small padding to ensure alignment
+  // Read that in, before calculating the actual tensor memory requirements.
+  uint64_t aligned_offset;
+  if (fileSize >= memoryNeeded) {
+    aligned_offset = *get<uint64_t>(current); // Offset to align memory to 256 size
+    memoryNeeded += aligned_offset + sizeof(uint64_t);
+  } else {
+    return false;
+  }
+
+  // Finally the tensor size:
+  for (uint64_t i = 0; i < numHeaders; i++) {
+    memoryNeeded += headers[i].dataLength;
+  }
+
+  // If this final check passes, the file is at least big enough to contain the model
+  if (fileSize >= memoryNeeded) {
+    return true;
+  } else {
+    return false;
+  }
+}
+
+} // Anonymous namespace
+
 AlignedMemory loadFileToMemory(const std::string& path, size_t alignment){
   uint64_t fileSize = filesystem::fileSize(path);
   io::InputFileStream in(path);
@@ -20,7 +93,9 @@ AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options){
     ABORT_IF(models.size() != 1, "Loading multiple binary models is not supported for now as it is not necessary.");
     marian::filesystem::Path modelPath(models[0]);
     ABORT_IF(modelPath.extension() != marian::filesystem::Path(".bin"), "The file of binary model should end with .bin");
-    return loadFileToMemory(models[0], 256);
+    AlignedMemory alignedMemory = loadFileToMemory(models[0], 256);
+    ABORT_IF(!validateBinaryModel(alignedMemory, alignedMemory.size()), "The binary file is invalid. Incomplete or corrupted download?");
+    return alignedMemory;
 }
 
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options){

From 3daa024eb303b52e483f326b26d6c976b42b2fe9 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 12 Apr 2021 17:05:23 +0100
Subject: [PATCH 191/442] Strengthen the Annotation class: Handle empty
 sentences and tests (#85)

* Changing Annotation to adhere to [begin, end)

* Stronger unit tests on sentences + num words, num sentences

* Hotfix with empty string view from EOS

* No more absolving empty-sentence; Added tests now defined behaviour

* Uncommenting important section in unit test

* Ensure empty string view default, beginning at end so marker points

* Further strengthen and comment unit-tests, mark exactly where empty sentence is happening

* Review comments: Dummy sentence + docs

- What should be a simple fast accessor is turning into compute.
  Normally the way to deal with this, for better or worse, is to put 0 at
  the beginning of sentenceEndIds_. (Putting 0 at the beginning of
  sentenceEndIds_)

- Indices into what? Mentioned to be flatByteRanges_.

* Documentation updates

* More changes to docs

Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
---
 src/tests/annotation_tests.cpp     | 205 +++++++++++++++++++++++++----
 src/translator/sentence_ranges.cpp |  56 ++++----
 src/translator/sentence_ranges.h   |  79 +++++++----
 3 files changed, 257 insertions(+), 83 deletions(-)

diff --git a/src/tests/annotation_tests.cpp b/src/tests/annotation_tests.cpp
index 228401146..3e42baedb 100644
--- a/src/tests/annotation_tests.cpp
+++ b/src/tests/annotation_tests.cpp
@@ -11,63 +11,210 @@ TEST_CASE("Test Annotation API with random sentences") {
   /// which sentence went in where and try to use accessor methods on
   /// AnnotatedText to check if what we have as ground-truth by construction is
   /// consistent with what is returned.
-  size_t sentences = 20;
+  size_t sentences = 500;
   size_t maxWords = 40;
 
+  // Set in case needed to see output. The output is in lines of #sentences +
+  // header, which can be split and compared for easy understanding. The ideal
+  // way to inspect what is going wrong is to redirect output and use to split
+  // the different stages by sentences + 1 lines and check the diff.
+  bool debug{false};
+
   std::mt19937 randomIntGen_;
   randomIntGen_.seed(42);
 
-  AnnotatedText testAnnotation;
-  std::vector<std::vector<ByteRange>> sentenceWords;
-  std::vector<ByteRange> Words;
+  AnnotatedText testAnnotation; // This the container we add through API and
+                                // check if the access is correct.
+
+  // External book-keeping so we have ground truths. Each element represents a
+  // sentence.
+
+  // word byte ranges - for testAnnotation.word(sId, wId)
+  std::vector<std::vector<ByteRange>> groundTruthWords;
+  // sentence byte ranges - for testAnnotation.sentence(sId, wId)
+  std::vector<ByteRange> groundTruthSentences;
 
+  // Prepare the text and construct ByteRanges as intended for sentences and
+  // words. The ByteRanges we construct here are expected to be the
+  // ground-truths for words and sentences. The string being constructed is like
+  // as follows:
+  //
+  //     0-0 0-1 0-2 0-3
+  //     1-0 1-1 1-2 1-3 1-4
+  //     2-0 2-1
+  //
+  //     4-0 4-1 4-2 4-3
+  //
+  // Words are separated by space units.
+  //
+  // Below, we accumulate the text with intended structure as above, and
+  // ground-truth tables populated to be aware of the ByteRanges where they are
+  // meant to be.
+  if (debug) {
+    std::cout << "Preparing text and ground truth-tables" << std::endl;
+  }
   for (size_t idx = 0; idx < sentences; idx++) {
     if (idx != 0)
       testAnnotation.text += "\n";
 
-    Words.clear();
-    size_t words = randomIntGen_() % maxWords + 1;
-    Words.reserve(words);
-    for (size_t idw = 0; idw < words; idw++) {
-      size_t before = testAnnotation.text.size();
+    // Words can be zero, we need to support empty word sentences as well.
+    size_t numWords = randomIntGen_() % maxWords;
+
+    std::vector<ByteRange> wordByteRanges;
+    wordByteRanges.reserve(numWords);
+
+    // For empty sentence, we expect it to be empty and marked in position where
+    // the existing string is if needed to be pointed out.
+    size_t before = testAnnotation.text.size() - 1;
+    size_t sentenceBegin{before}, sentenceEnd{before};
+
+    for (size_t idw = 0; idw < numWords; idw++) {
+      if (idw != 0) {
+        testAnnotation.text += " ";
+        if (debug) {
+          std::cout << " ";
+        }
+      }
+
+      // Get new beginning, accounting for space above.
+      before = testAnnotation.text.size();
+
+      // Add the word
       std::string word = std::to_string(idx) + "-" + std::to_string(idw);
       testAnnotation.text += word;
-      if (idw != 0)
-        testAnnotation.text += " ";
-      Words.push_back((ByteRange){before, before + word.size() - 1});
+
+      // Do math, before, before + new-word's size.
+      wordByteRanges.push_back((ByteRange){before, before + word.size()});
+
+      if (debug) {
+        std::cout << word;
+      }
+
+      if (idw == 0) {
+        sentenceBegin = before;
+      }
+      if (idw == numWords - 1) {
+        sentenceEnd = before + word.size();
+      }
     }
-    // std::cout << std::endl;
+    if (debug) {
+      std::cout << std::endl;
+    }
+
+    groundTruthWords.push_back(wordByteRanges);
+    groundTruthSentences.push_back((ByteRange){sentenceBegin, sentenceEnd});
+  }
+
+  // We prepare string_views now with the known ByteRanges and use the
+  // string_view based AnnotatedText.addSentence(...) API to add sentences to
+  // transparently convert from string_views to ByteRanges, rebasing/working out
+  // the math underneath.
 
-    sentenceWords.push_back(Words);
+  if (debug) {
+    std::cout << "Inserting words onto container and save ground-truth-table:"
+              << std::endl;
   }
 
-  // std::cout << "Inserting words:" << std::endl;
-  std::vector<std::vector<marian::string_view>> byteRanges;
-  for (auto &sentence : sentenceWords) {
+  std::vector<std::vector<marian::string_view>> wordStringViews;
+  for (auto &sentence : groundTruthWords) {
     std::vector<marian::string_view> wordByteRanges;
+    bool first{true};
     for (auto &word : sentence) {
       marian::string_view wordView(&testAnnotation.text[word.begin],
-                                   word.end - word.begin);
+                                   word.size());
       wordByteRanges.push_back(wordView);
-      // std::cout << std::string(wordView) << " ";
+      if (debug) {
+        if (first) {
+          first = false;
+        } else {
+          std::cout << " ";
+        }
+        std::cout << std::string(wordView);
+      }
     }
     testAnnotation.addSentence(wordByteRanges);
-    byteRanges.push_back(wordByteRanges);
-    // std::cout << std::endl;
+    wordStringViews.push_back(wordByteRanges);
+    if (debug) {
+      std::cout << std::endl;
+    }
+  }
+
+  if (debug) {
+    std::cout
+        << "Inserting sentences onto container and save ground-truth-table"
+        << std::endl;
+  }
+  std::vector<marian::string_view> sentenceStringViews;
+  for (auto &sentenceByteRange : groundTruthSentences) {
+    char *data = &(testAnnotation.text[sentenceByteRange.begin]);
+    marian::string_view sentenceView(data, sentenceByteRange.size());
+    sentenceStringViews.push_back(sentenceView);
+
+    if (debug) {
+      std::cout << sentenceView << std::endl;
+    }
+  }
+
+  // Access from the sentence(sentenceIdx) API and confirm that the ground truth
+  // we expect is same as what comes out of the container.
+  if (debug) {
+    std::cout << "From container: Sentences" << std::endl;
+  }
+  for (int idx = 0; idx < groundTruthSentences.size(); idx++) {
+    ByteRange expected = groundTruthSentences[idx];
+    ByteRange obtained = testAnnotation.sentenceAsByteRange(idx);
+    if (debug) {
+      std::cout << std::string(testAnnotation.sentence(idx)) << std::endl;
+    }
+    CHECK(expected.begin == obtained.begin);
+    CHECK(expected.end == obtained.end);
+    std::string expected_string = std::string(sentenceStringViews[idx]);
+    std::string obtained_string = std::string(testAnnotation.sentence(idx));
+    CHECK(expected_string == obtained_string);
+  }
+
+  /// Access the word(sentenceIdx, wordIdx) API and confirm what we hold as
+  /// expected words are the same as those obtained from the container.
+  if (debug) {
+    std::cout << "From container: Words" << std::endl;
+  }
+
+  CHECK(groundTruthWords.size() == testAnnotation.numSentences());
+  for (int idx = 0; idx < groundTruthWords.size(); idx++) {
+    CHECK(groundTruthWords[idx].size() == testAnnotation.numWords(idx));
   }
 
-  // std::cout << "From container: " << std::endl;
-  for (int idx = 0; idx < sentenceWords.size(); idx++) {
-    for (int idw = 0; idw < sentenceWords[idx].size(); idw++) {
-      ByteRange expected = sentenceWords[idx][idw];
+  for (int idx = 0; idx < groundTruthWords.size(); idx++) {
+    for (int idw = 0; idw < groundTruthWords[idx].size(); idw++) {
+      ByteRange expected = groundTruthWords[idx][idw];
       ByteRange obtained = testAnnotation.wordAsByteRange(idx, idw);
-      // std::cout << std::string(testAnnotation.word(idx, idw)) << " ";
+      if (debug) {
+        std::cout << std::string(testAnnotation.word(idx, idw)) << " ";
+      }
       CHECK(expected.begin == obtained.begin);
       CHECK(expected.end == obtained.end);
 
-      std::string expected_string = std::string(byteRanges[idx][idw]);
-      CHECK(expected_string == std::string(testAnnotation.word(idx, idw)));
+      std::string expected_string = std::string(wordStringViews[idx][idw]);
+      std::string obtained_string = std::string(testAnnotation.word(idx, idw));
+      CHECK(expected_string == obtained_string);
+    }
+    if (debug) {
+      std::cout << std::endl;
     }
-    // std::cout << std::endl;
   }
+
+  // Try inserting an empty Sentence. This is ensuring we check for empty
+  // Sentence if the random test above does not cover it for some reason.
+  int emptySentenceIdx = sentences;
+  std::vector<marian::string_view> emptySentence;
+  testAnnotation.addSentence(emptySentence);
+
+  // There are no words.
+  CHECK(testAnnotation.numWords(emptySentenceIdx) == 0);
+
+  // Empty sentence expected at output.
+  std::string expectedEmptyString = "";
+  marian::string_view emptyView = testAnnotation.sentence(emptySentenceIdx);
+  std::string obtainedString = std::string(emptyView.data(), emptyView.size());
+  CHECK(expectedEmptyString == obtainedString);
 }
diff --git a/src/translator/sentence_ranges.cpp b/src/translator/sentence_ranges.cpp
index 053eeaa48..aae9dd346 100644
--- a/src/translator/sentence_ranges.cpp
+++ b/src/translator/sentence_ranges.cpp
@@ -6,48 +6,44 @@ namespace marian {
 namespace bergamot {
 
 void Annotation::addSentence(std::vector<ByteRange> &sentence) {
-  size_t size = flatByteRanges_.size();
   flatByteRanges_.insert(std::end(flatByteRanges_), std::begin(sentence),
                          std::end(sentence));
-  sentenceBeginIds_.push_back(size);
+  size_t size = flatByteRanges_.size();
+  sentenceEndIds_.push_back(size);
 }
 
 size_t Annotation::numWords(size_t sentenceIdx) const {
-  auto terminals = sentenceTerminalIds(sentenceIdx);
-  return terminals.second - terminals.first + 1;
-}
-
-std::pair<size_t, size_t>
-Annotation::sentenceTerminalIds(size_t sentenceIdx) const {
   size_t bosId, eosId;
-  bosId = sentenceBeginIds_[sentenceIdx];
-  eosId = sentenceIdx + 1 < numSentences()
-              ? sentenceBeginIds_[sentenceIdx + 1] - 1
-              : flatByteRanges_.size() - 1;
-
-  // Out of bound checks.
-  assert(bosId < flatByteRanges_.size());
-  assert(eosId < flatByteRanges_.size());
-  return std::make_pair(bosId, eosId);
-}
-
-std::pair<ByteRange, ByteRange>
-Annotation::sentenceTerminals(size_t sentenceIdx) const {
-  auto terminals = sentenceTerminalIds(sentenceIdx);
-  return std::make_pair(flatByteRanges_[terminals.first],
-                        flatByteRanges_[terminals.second]);
+  bosId = sentenceEndIds_[sentenceIdx]; // Half interval, so;
+  eosId = sentenceEndIds_[sentenceIdx + 1];
+  // Difference between eosId and bosId is the number of words.
+  return eosId - bosId;
 }
 
 ByteRange Annotation::sentence(size_t sentenceIdx) const {
-  auto terminals = sentenceTerminals(sentenceIdx);
-  return (ByteRange){terminals.first.begin, terminals.second.end};
+  size_t bosId, eosId;
+  bosId = sentenceEndIds_[sentenceIdx]; // Half interval, so;
+  eosId = sentenceEndIds_[sentenceIdx + 1];
+  ByteRange sentenceByteRange;
+
+  if (bosId == eosId) {
+    // We have an empty sentence. However, we want to be able to point where in
+    // target this happened through the ranges. We are looking for the end of
+    // the flatByteRange and non-empty sentence before this happened and
+    // construct empty string-view equivalent ByteRange.
+    ByteRange eos = flatByteRanges_[eosId - 1];
+    sentenceByteRange = (ByteRange){eos.end, eos.end};
+  } else {
+    ByteRange bos = flatByteRanges_[bosId];
+    ByteRange eos = flatByteRanges_[eosId - 1];
+    sentenceByteRange = (ByteRange){bos.begin, eos.end};
+  }
+  return sentenceByteRange;
 }
 
 ByteRange Annotation::word(size_t sentenceIdx, size_t wordIdx) const {
-  size_t offset = sentenceBeginIds_[sentenceIdx];
-  // auto terminals = sentenceTerminals(sentenceIdx);
-  // assert(offset + wordIdx <= terminals.second);
-  return flatByteRanges_[offset + wordIdx];
+  size_t bosOffset = sentenceEndIds_[sentenceIdx];
+  return flatByteRanges_[bosOffset + wordIdx];
 }
 
 string_view AnnotatedText::word(size_t sentenceIdx, size_t wordIdx) const {
diff --git a/src/translator/sentence_ranges.h b/src/translator/sentence_ranges.h
index a0dc8c9a9..b3986e30a 100644
--- a/src/translator/sentence_ranges.h
+++ b/src/translator/sentence_ranges.h
@@ -19,51 +19,82 @@ struct ByteRange {
 
 /// An Annotation is a collection of ByteRanges used to denote ancillary
 /// information of sentences and words on a text of string. Annotation is meant
-/// for consumption on platforms where string_view creates problems (eg: exports
-/// through WASM). See AnnotatedText for cases where this is a non-issue.
+/// for consumption on platforms where `string_view` creates problems (eg:
+/// exports through WASM) conveniently rebasing them as required into
+/// ByteRanges. See AnnotatedText for cases where this is a non-issue.
+///
+/// **Usage**
+///
+/// To ensure rebasing is consistent during creation and updation, use
+/// `Annotation` best through `AnnotatedText`, which also holds the reference
+/// string and can work with `string_views`.
+///
+/// If used separately, it is on the user to ensure the reference string
+/// is the same as what the Annotation refers to. For best results, an instance
+/// is expected to be read only in this mode of operation.
+///
+/// **Idea**
+///
+/// Annotation is intended to be the same structure conceptually as below,
+/// except the `std::vector<std::vector<ByteRange>>` hammered into a flat
+/// structure to avoid multiple reallocs keeping efficiency in mind. This is
+/// achieved by having markers of where sentence ends in the flat container
+/// storing word ByteRanges.
+///
+/// ```cpp
+/// typedef ByteRange Word;
+/// // std::vector<ByteRange>, a single sentence
+/// typedef std::vector<Word> Sentence;
+/// std::vector<std::vector<ByteRange> // multiple sentences
+/// typedef std::vector<Sentence> Annotation;
+///
+/// Annotation example;
+/// ```
+/// This structure exists to provide a consistent API to access the nested
+/// sentences of varying lengths, which occur in source-text processed into
+/// multiple sentences, and target-text translated from source as multiple
+/// sentences, both composed of (sub)-words, providing a List[List] like access
+/// while storing it in a compact and efficient manner.
 class Annotation {
 public:
-  /// Annotation is constructed empty. See addSentence to populate it with
+  /// Annotation is constructed empty. See `addSentence()` to populate it with
   /// annotations.
-  Annotation() {}
+  Annotation() {
+    // The -1-th sentence ends at 0.
+    sentenceEndIds_.push_back(0);
+  }
 
   /// Returns the number of sentences annotated in a text.
-  size_t numSentences() const { return sentenceBeginIds_.size(); }
+  size_t numSentences() const { return sentenceEndIds_.size() - 1; }
 
-  /// Returns number of words in the sentece identified by sentenceIdx.
+  /// Returns number of words in the sentence identified by `sentenceIdx`.
   size_t numWords(size_t sentenceIdx) const;
 
-  /// Adds a sentences from vector<ByteRange> representation, internally doing
+  /// Adds a sentences from `vector<ByteRange>` representation, internally doing
   /// extra book-keeping for the sentence terminal markings. Sentences are
   /// expected to be added in order as they occur in text.
   void addSentence(std::vector<ByteRange> &sentence);
 
-  /// Returns a ByteRange representing wordIdx in sentenceIdx
+  /// Returns a ByteRange representing `wordIdx` in sentence indexed by
+  /// `sentenceIdx`. `wordIdx` follows 0-based indexing, and should be less than
+  /// `.numWords()` for `sentenceIdx` for defined behaviour.
   ByteRange word(size_t sentenceIdx, size_t wordIdx) const;
 
-  /// Returns a ByteRange representing sentence corresponding to sentenceIdx.
+  /// Returns a ByteRange representing sentence corresponding to `sentenceIdx`.
+  /// `sentenceIdx` follows 0-based indexing, and behaviour is defined only when
+  /// less than `.numSentences()`.
   ByteRange sentence(size_t sentenceIdx) const;
 
 private:
   /// A flat storage for ByteRanges. Composed of word ByteRanges, extra
-  /// information in sentenceBeginIds_ to denote sentence boundary markers as
+  /// information in sentenceEndIds_ to denote sentence boundary markers as
   /// indices.
   std::vector<ByteRange> flatByteRanges_;
 
-  /// Stores indices where sentences begin
-  std::vector<size_t> sentenceBeginIds_;
-
-  /// Returns ByteRanges corresponding to beginning and end words of sentence
-  /// corresponding to sentenceIdx. This is useful in using the information to
-  /// construct a ByteRange of a sentence taking the begin from the first and
-  /// end from the second.
-  std::pair<ByteRange, ByteRange> sentenceTerminals(size_t sentenceIdx) const;
-
-  /// Returns indices of terminal (word) ByteRanges in sentenceIds_ of a
-  /// sentence corresponding to sentenceIdx. The distance can be used to compute
-  /// number of words in a sentence (numWords) and also to construct the
-  /// terminal ByteRanges (sentenceTerminals).
-  std::pair<size_t, size_t> sentenceTerminalIds(size_t sentenceIdx) const;
+  /// Stores indices onto flatByteRanges_ of where sentences end (not inclusive,
+  /// aligned with C++ half interval notions). There is a 0 marker to simplify
+  /// sources, indicating where the -1-th sentence ends.
+  std::vector<size_t> sentenceEndIds_;
 };
 
 /// AnnotatedText is effectively std::string text + Annotation, providing the

From e4b58357db44751d38100d73a31e0e10c03d88a1 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Wed, 14 Apr 2021 09:56:07 +0100
Subject: [PATCH 192/442] Clarify misleading comment (#99)

---
 src/translator/batch_translator.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 8278ca729..19cbaf9d1 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -36,7 +36,7 @@ void BatchTranslator::initialize() {
     }
   }
 
-  graph_ = New<ExpressionGraph>(true); // always optimize
+  graph_ = New<ExpressionGraph>(true); // set the graph to be inference only
   auto prec = options_->get<std::vector<std::string>>("precision", {"float32"});
   graph_->setDefaultElementType(typeFromString(prec[0]));
   graph_->setDevice(device_);

From f5dffeb5ca276886b7754644ea7f51d3e1c7d860 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 14 Apr 2021 12:03:33 +0200
Subject: [PATCH 193/442] Downgraded resource class to 'medium' for circle ci

 - Also restricted parallel make compilation to 3
---
 .circleci/config.yml | 2 +-
 build-wasm.sh        | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 6bc00febb..6818bc231 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -3,7 +3,7 @@ jobs:
   build:
     docker:
       - image: 'emscripten/emsdk:2.0.9'
-    resource_class: xlarge
+    resource_class: medium
 
     working_directory: ~/checkout
 
diff --git a/build-wasm.sh b/build-wasm.sh
index 9b08aa77b..f92d8ba20 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -63,7 +63,7 @@ cd build-wasm
 
 #     2. Compile the artefacts
 emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR="../models/" ../
-emmake make -j
+emmake make -j3
 
 #     3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
 sed -i.bak 's/var result = WebAssembly.instantiateStreaming(response, info);/var result = WebAssembly.instantiateStreaming(response, info,{simdWormhole:true});/g' wasm/bergamot-translator-worker.js

From a7f6bb51d980d4dd7b95e577933750bbe0bef62f Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 14 Apr 2021 14:51:08 +0200
Subject: [PATCH 194/442] Minor cleanup in build-wasm.sh file

---
 build-wasm.sh | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/build-wasm.sh b/build-wasm.sh
index f92d8ba20..5348fe994 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -66,10 +66,8 @@ emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR="../models/" ../
 emmake make -j3
 
 #     3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
-sed -i.bak 's/var result = WebAssembly.instantiateStreaming(response, info);/var result = WebAssembly.instantiateStreaming(response, info,{simdWormhole:true});/g' wasm/bergamot-translator-worker.js
-sed -i.bak 's/return WebAssembly.instantiate(binary, info);/return WebAssembly.instantiate(binary, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
-sed -i.bak 's/var module = new WebAssembly.Module(bytes);/var module = new WebAssembly.Module(bytes, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+bash ../wasm/patch-artifacts-enable-wormhole.sh
 
-# The artefacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
+# The artifacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
 
 exit 0

From f1fc4f8041ae321d2304a115810e702b9ca6be4f Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Wed, 14 Apr 2021 13:53:35 +0100
Subject: [PATCH 195/442] Fix the target_include_directories (#98)

---
 src/translator/CMakeLists.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 3ddfa7956..9bf7dca00 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -29,5 +29,5 @@ endif(COMPILE_WASM)
 target_link_libraries(bergamot-translator marian ssplit)
 
 target_include_directories(bergamot-translator
-    PUBLIC ${CMAKE_SOURCE_DIR}
-    PUBLIC ${CMAKE_SOURCE_DIR}/src)
+    PUBLIC ${PROJECT_SOURCE_DIR}
+           ${PROJECT_SOURCE_DIR}/src)

From c00c263f8f8e1eb02ecbac3c59acdbe591f4fe0a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 16 Apr 2021 11:58:53 +0100
Subject: [PATCH 196/442] Moving small tests to GitHub CI (#93)

Adds regression-tests to the workflow for native minimal/custom marian and full builds.

Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
---
 .../workflows/native-custom_marian-mac.yml    |  33 -----
 .../workflows/native-custom_marian-ubuntu.yml |  33 -----
 .github/workflows/native-full_marian-mac.yml  |  59 ---------
 .../workflows/native-full_marian-ubuntu.yml   | 120 ------------------
 .github/workflows/native-mac.yml              | 108 ++++++++++++++++
 .github/workflows/native-ubuntu.yml           | 117 +++++++++++++++++
 .../workflows/wasm-custom_marian-ubuntu.yml   |   4 +-
 .gitmodules                                   |   3 +
 bergamot-translator-tests                     |   1 +
 9 files changed, 231 insertions(+), 247 deletions(-)
 delete mode 100644 .github/workflows/native-custom_marian-mac.yml
 delete mode 100644 .github/workflows/native-custom_marian-ubuntu.yml
 delete mode 100644 .github/workflows/native-full_marian-mac.yml
 delete mode 100644 .github/workflows/native-full_marian-ubuntu.yml
 create mode 100644 .github/workflows/native-mac.yml
 create mode 100644 .github/workflows/native-ubuntu.yml
 create mode 160000 bergamot-translator-tests

diff --git a/.github/workflows/native-custom_marian-mac.yml b/.github/workflows/native-custom_marian-mac.yml
deleted file mode 100644
index 1aae7e590..000000000
--- a/.github/workflows/native-custom_marian-mac.yml
+++ /dev/null
@@ -1,33 +0,0 @@
-name: Native (Custom Marian) MacOS
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ main, ci-sandbox ]
-
-jobs:
-  build-macos:
-    name: MacOS
-    runs-on: macos-10.15
-
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-
-    - name: Configure CMake
-      run: |
-        mkdir -p build
-        cd build
-        cmake ..
-
-    - name: Compile
-      working-directory: build
-      run: make -j2
-
-    - name: Print versions
-      working-directory: build
-      run: |
-        ./app/bergamot-translator-app --version
diff --git a/.github/workflows/native-custom_marian-ubuntu.yml b/.github/workflows/native-custom_marian-ubuntu.yml
deleted file mode 100644
index f0518718a..000000000
--- a/.github/workflows/native-custom_marian-ubuntu.yml
+++ /dev/null
@@ -1,33 +0,0 @@
-name: Native (Custom Marian) Ubuntu
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ main, ci-sandbox ]
-
-jobs:
-  build-macos:
-    name: Ubuntu
-    runs-on: ubuntu-latest
-
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-
-    - name: Configure CMake
-      run: |
-        mkdir -p build
-        cd build
-        cmake ..
-
-    - name: Compile
-      working-directory: build
-      run: make -j2
-
-    - name: Print versions
-      working-directory: build
-      run: |
-        ./app/bergamot-translator-app --version
diff --git a/.github/workflows/native-full_marian-mac.yml b/.github/workflows/native-full_marian-mac.yml
deleted file mode 100644
index 1928c5c03..000000000
--- a/.github/workflows/native-full_marian-mac.yml
+++ /dev/null
@@ -1,59 +0,0 @@
-name: Native (Full Marian) MacOS
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ main, ci-sandbox ]
-
-jobs:
-  build-macos:
-    name: MacOS CPU-only
-    runs-on: macos-10.15
-
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-
-    - name: Install dependencies
-      run: brew install openblas protobuf
-
-    # Openblas location is exported explicitly because openblas is keg-only,
-    # which means it was not symlinked into /usr/local/.
-    # CMake cannot find BLAS on GitHub runners if Marian is being compiled
-    # statically, hence USE_STATIC_LIBS=off
-    - name: Configure CMake
-      run: |
-        export LDFLAGS="-L/usr/local/opt/openblas/lib"
-        export CPPFLAGS="-I/usr/local/opt/openblas/include"
-        mkdir -p build
-        cd build
-        cmake .. \
-          -DCOMPILE_CPU=on \
-          -DCOMPILE_CUDA=off \
-          -DCOMPILE_EXAMPLES=on \
-          -DCOMPILE_SERVER=on \
-          -DCOMPILE_TESTS=on \
-          -DUSE_FBGEMM=on \
-          -DUSE_SENTENCEPIECE=on \
-          -DUSE_STATIC_LIBS=off \
-          -DUSE_WASM_COMPATIBLE_SOURCE=off
-
-    - name: Compile
-      working-directory: build
-      run: make -j2
-
-    - name: Run unit tests
-      working-directory: build
-      run: make test
-
-    - name: Print versions
-      working-directory: build
-      run: |
-        ./marian --version
-        ./marian-decoder --version
-        ./marian-scorer --version
-        ./spm_encode --version
-
diff --git a/.github/workflows/native-full_marian-ubuntu.yml b/.github/workflows/native-full_marian-ubuntu.yml
deleted file mode 100644
index e414f6477..000000000
--- a/.github/workflows/native-full_marian-ubuntu.yml
+++ /dev/null
@@ -1,120 +0,0 @@
-name: Native (Full Marian) Ubuntu
-
-on:
-  push:
-    branches: [ main, ci-test ]
-  pull_request:
-    branches: [ main, ci-test ]
-
-jobs:
-  build-ubuntu:
-    strategy:
-      matrix:
-        include:
-          # Ubuntu CPU-only build
-          - name: "Ubuntu CPU-only"
-            os: ubuntu-latest
-            cuda: ""
-            gcc: 8
-            cpu: true
-            gpu: false
-          # GPU Builds are commented out, for bergamot-translator CI runs.
-          # Ubuntu GPU-only build
-          # - name: "Ubuntu GPU-only"
-          #   os: ubuntu-latest
-          #   cuda: "10.2"
-          #   gcc: 7
-          #   cpu: false
-          #   gpu: true
-          # Ubuntu 20.04 supports CUDA 11+
-          #- name: "Ubuntu 20.04 CUDA 11.0 gcc-9"
-            #os: ubuntu-20.04
-            #cuda: "11.0"
-            #gcc: 9
-            #cpu: false
-            #gpu: true
-          # Ubuntu 18.04 supports CUDA 10.1+
-          # - name: "Ubuntu 18.04 CUDA 10.2 gcc-8"
-          #   os: ubuntu-18.04
-          #   cuda: "10.2"
-          #   gcc: 8
-          #   cpu: true
-          #   gpu: true
-          # Ubuntu 16.04 supports CUDA 8+
-          # - name: "Ubuntu 16.04 CUDA 9.2 gcc-7"
-          #   os: ubuntu-16.04
-          #   cuda: "9.2"
-          #   gcc: 7
-          #   cpu: true
-          #   gpu: true
-
-    runs-on: ${{ matrix.os }}
-    name: ${{ matrix.name }}
-
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-
-    # The following packages are already installed on GitHub-hosted runners:
-    # build-essential openssl libssl-dev
-    # No need to install libprotobuf{17,10,9v5} on Ubuntu {20,18,16}.04 because
-    # it is installed together with libprotobuf-dev
-    - name: Install dependencies
-      run: sudo apt-get update && sudo apt-get install -y libgoogle-perftools-dev libprotobuf-dev protobuf-compiler libboost-all-dev g++-8
-
-    # https://software.intel.com/content/www/us/en/develop/articles/installing-intel-free-libs-and-python-apt-repo.html
-    - name: Install MKL
-      run: |
-        wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB" | sudo apt-key add -
-        sudo sh -c "echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list"
-        sudo apt-get update -o Dir::Etc::sourcelist="/etc/apt/sources.list.d/intel-mkl.list"
-        sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
-      if: matrix.cpu == true
-
-    # The script simplifies installation of different versions of CUDA
-    - name: Install CUDA
-      run: ./3rd_party/marian-dev/scripts/ci/install_cuda_ubuntu.sh ${{ matrix.cuda }}
-      if: matrix.gpu == true
-
-    # Boost is installed on GitHub-hosted runners in a non-standard location
-    # https://github.com/actions/virtual-environments/issues/687#issuecomment-610471671
-    - name: Configure CMake
-      run: |
-        mkdir -p build
-        cd build
-        CC=/usr/bin/gcc-${{ matrix.gcc }} CXX=/usr/bin/g++-${{ matrix.gcc }} CUDAHOSTCXX=/usr/bin/g++-${{ matrix.gcc }} \
-        cmake .. \
-          -DBoost_ARCHITECTURE=-x64 \
-          -DCMAKE_BUILD_TYPE=Release \
-          -DCOMPILE_CPU=${{ matrix.cpu }} \
-          -DCOMPILE_CUDA=${{ matrix.gpu }} \
-          -DCOMPILE_EXAMPLES=on \
-          -DCOMPILE_SERVER=on \
-          -DCOMPILE_TESTS=on \
-          -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-${{ matrix.cuda }} \
-          -DUSE_FBGEMM=${{ matrix.cpu }} \
-          -DUSE_SENTENCEPIECE=on \
-          -DUSE_STATIC_LIBS=on \
-          -DUSE_WASM_COMPATIBLE_SOURCE=off
-
-    - name: Compile
-      working-directory: build
-      run: make -j2
-
-    - name: Run unit tests
-      working-directory: build
-      run: make test
-      # GitHub-hosted VMs do not have GPUs, so can not be run in CUDA builds
-      if: matrix.gpu == false
-
-    - name: Print versions
-      working-directory: build
-      run: |
-        ./marian --version
-        ./marian-decoder --version
-        ./marian-scorer --version
-        ./marian-server --version
-        ./spm_encode --version
-
diff --git a/.github/workflows/native-mac.yml b/.github/workflows/native-mac.yml
new file mode 100644
index 000000000..8df203d5d
--- /dev/null
+++ b/.github/workflows/native-mac.yml
@@ -0,0 +1,108 @@
+name: Native MacOS
+
+on:
+  push:
+    branches: [ main, ci-sandbox ]
+  pull_request:
+    branches: [ main, ci-sandbox ]
+
+jobs:
+  build-macos:
+    strategy: 
+      fail-fast: false
+      matrix:
+        include:
+          - name: "full-marian"
+            os: macos-10.15
+            test_tags: ""
+            cmake: 
+              CMAKE_BUILD_TYPE: "Release"
+              COMPILE_TESTS: "ON"
+              USE_WASM_COMPATIBLE_SOURCE: "OFF"
+              USE_FBGEMM: "OFF"
+              USE_STATIC_LIBS: "OFF"
+              COMPILE_SERVER: "OFF"
+              COMPILE_EXAMPLES: "OFF"
+
+          - name: "minimal-marian"
+            os: macos-10.15
+            test_tags: "'#wasm'"
+            cmake: 
+              CMAKE_BUILD_TYPE: "Release"
+              COMPILE_TESTS: "OFF" # Minimal marian has no sqlite support and compile tests fail
+              USE_WASM_COMPATIBLE_SOURCE: "ON"
+              USE_FBGEMM: "OFF"
+              # explicitly set due to requirement of minimal marian being used
+              # within WASM. This is some yaml ugliness, but issok.
+              USE_STATIC_LIBS: "ON" 
+              COMPILE_SERVER: "OFF"
+              COMPILE_EXAMPLES: "OFF"
+        
+    name: ${{ matrix.name }}
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Install dependencies
+      run: |
+          brew update
+          brew install openblas protobuf coreutils
+
+    # Openblas location is exported explicitly because openblas is keg-only,
+    # which means it was not symlinked into /usr/local/.
+    - name: Set BLAS Environment variables
+      run: |
+          echo "LDFLAGS=-L/usr/local/opt/openblas/lib" >> $GITHUB_ENV
+          echo "CPPFLAGS=-I/usr/local/opt/openblas/include" >> $GITHUB_ENV
+      if: matrix.cmake.USE_WASM_COMPATIBLE_SOURCE == 'OFF'
+
+    # CMake cannot find BLAS on GitHub runners if Marian is being compiled
+    # statically, hence USE_STATIC_LIBS=off
+    - name: Configure CMake
+      run: |
+        mkdir -p build
+        cd build
+        cmake .. \
+          -DCMAKE_BUILD_TYPE=${{ matrix.cmake.CMAKE_BUILD_TYPE }}\
+          -DCOMPILE_TESTS=${{ matrix.cmake.COMPILE_TESTS }}\
+          -DCOMPILE_EXAMPLES=${{ matrix.cmake.COMPILE_EXAMPLES }} \
+          -DCOMPILE_SERVER=${{ matrix.cmake.COMPILE_SERVER }} \
+          -DUSE_STATIC_LIBS=${{ matrix.cmake.USE_STATIC_LIBS }} \
+          -DUSE_WASM_COMPATIBLE_SOURCE=${{ matrix.cmake.USE_WASM_COMPATIBLE_SOURCE }} \
+          -DUSE_FBGEMM=${{ matrix.cmake.USE_FBGEMM }}
+
+    - name: Compile
+      working-directory: build
+      run: make -j2
+
+    - name: Run unit tests
+      working-directory: build
+      run: make test
+      if: matrix.cmake.COMPILE_TESTS == 'ON'
+
+    - name: Print versions
+      working-directory: build
+      run: |
+        ./app/bergamot-translator-app --version
+
+    - name: Install regression-test framework (BRT)
+      working-directory: bergamot-translator-tests
+      run : make install
+
+    - name: Run regression-tests (BRT)
+      working-directory: bergamot-translator-tests
+      run : MARIAN=../build ./run_brt.sh ${{ matrix.test_tags }}
+
+    - name: Upload regression-tests artifacts
+      uses: actions/upload-artifact@v2
+      if: ${{ always() }}
+      with: 
+        name: brt-artifacts-${{ matrix.name }}
+        path: |
+            bergamot-translator-tests/**/*.expected
+            bergamot-translator-tests/**/*.log
+            bergamot-translator-tests/**/*.out
diff --git a/.github/workflows/native-ubuntu.yml b/.github/workflows/native-ubuntu.yml
new file mode 100644
index 000000000..dc8016bae
--- /dev/null
+++ b/.github/workflows/native-ubuntu.yml
@@ -0,0 +1,117 @@
+name: Native Ubuntu
+
+on:
+  push:
+    branches: [ main, ci-sandbox ]
+  pull_request:
+    branches: [ main, ci-sandbox ]
+
+jobs:
+  build-ubuntu:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - name: "full-marian"
+            os: ubuntu-latest
+            gcc: 8
+            cpu: 'ON'
+            gpu: 'OFF'
+            test_tags: ""
+            cmake: 
+              CMAKE_BUILD_TYPE: "Release"
+              COMPILE_TESTS: "ON"
+              USE_WASM_COMPATIBLE_SOURCE: "OFF"
+              COMPILE_SERVER: "OFF"
+              COMPILE_EXAMPLES: "OFF"
+
+          - name: "minimal-marian"
+            os: ubuntu-latest
+            gcc: 8
+            cpu: 'ON'
+            gpu: 'OFF'
+            test_tags: "'#wasm'"
+            cmake:
+              CMAKE_BUILD_TYPE: "Release"
+              COMPILE_TESTS: "OFF" # Minimal marian has no sqlite support and COMPILE_TEST=ON fails.
+              USE_WASM_COMPATIBLE_SOURCE: "ON"
+              COMPILE_SERVER: "OFF"
+              COMPILE_EXAMPLES: "OFF"
+
+
+    runs-on: ${{ matrix.os }}
+    name: ${{ matrix.name }}
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    # The following packages are already installed on GitHub-hosted runners:
+    # build-essential openssl libssl-dev
+    # No need to install libprotobuf{17,10,9v5} on Ubuntu {20,18,16}.04 because
+    # it is installed together with libprotobuf-dev
+    - name: Install dependencies
+      run: |
+        sudo apt-get update 
+        sudo apt-get install -y \
+            libgoogle-perftools-dev libprotobuf-dev protobuf-compiler  \
+            libboost-all-dev g++-${{ matrix.gcc }} 
+
+    # https://software.intel.com/content/www/us/en/develop/articles/installing-intel-free-libs-and-python-apt-repo.html
+    - name: Install MKL
+      run: |
+        wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB" | sudo apt-key add -
+        sudo sh -c "echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list"
+        sudo apt-get update -o Dir::Etc::sourcelist="/etc/apt/sources.list.d/intel-mkl.list"
+        sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
+      if: matrix.cmake.USE_WASM_COMPATIBLE_SOURCE == 'OFF'
+
+    # Boost is installed on GitHub-hosted runners in a non-standard location
+    # https://github.com/actions/virtual-environments/issues/687#issuecomment-610471671
+    - name: Configure CMake
+      run: |
+        mkdir -p build
+        cd build
+        CC=/usr/bin/gcc-${{ matrix.gcc }} CXX=/usr/bin/g++-${{ matrix.gcc }} CUDAHOSTCXX=/usr/bin/g++-${{ matrix.gcc }} \
+        cmake .. \
+          -DCMAKE_BUILD_TYPE=${{ matrix.cmake.CMAKE_BUILD_TYPE }}\
+          -DCOMPILE_TESTS=${{ matrix.cmake.COMPILE_TESTS }}\
+          -DCOMPILE_EXAMPLES=${{ matrix.cmake.COMPILE_EXAMPLES }} \
+          -DCOMPILE_SERVER=${{ matrix.cmake.COMPILE_SERVER }} \
+          -DUSE_WASM_COMPATIBLE_SOURCE=${{ matrix.cmake.USE_WASM_COMPATIBLE_SOURCE }} \
+
+    - name: Compile bergamot-translator
+      working-directory: build
+      run: make -j2
+
+    - name: Run unit tests
+      working-directory: build
+      run: make test
+      # GitHub-hosted VMs do not have GPUs, so can not be run in CUDA builds
+      if: matrix.gpu == 'OFF' && matrix.cmake.COMPILE_TESTS == 'ON'
+
+    - name: Print versions
+      working-directory: build
+      run: |
+        ./app/bergamot-translator-app --version
+
+
+    - name: Install regression-test framework (BRT)
+      working-directory: bergamot-translator-tests
+      run : make install
+
+    - name: Run regression-tests (BRT)
+      working-directory: bergamot-translator-tests
+      run : MARIAN=../build ./run_brt.sh ${{ matrix.test_tags }}
+
+    - name: Upload regression-tests artifacts
+      uses: actions/upload-artifact@v2
+      if: ${{ always() }}
+      with: 
+        name: brt-artifacts-${{ matrix.name }}
+        path: |
+            bergamot-translator-tests/**/*.expected
+            bergamot-translator-tests/**/*.log
+            bergamot-translator-tests/**/*.out
diff --git a/.github/workflows/wasm-custom_marian-ubuntu.yml b/.github/workflows/wasm-custom_marian-ubuntu.yml
index d1364dc50..7dfc83903 100644
--- a/.github/workflows/wasm-custom_marian-ubuntu.yml
+++ b/.github/workflows/wasm-custom_marian-ubuntu.yml
@@ -2,9 +2,9 @@ name: WASM (Custom Marian) Ubuntu
 
 on:
   push:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main ]
+    branches: [ main, ci-sandbox ]
 
 jobs:
   build-wasm:
diff --git a/.gitmodules b/.gitmodules
index cc40735d6..8aa101494 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -4,3 +4,6 @@
 [submodule "3rd_party/ssplit-cpp"]
 	path = 3rd_party/ssplit-cpp
 	url = https://github.com/browsermt/ssplit-cpp
+[submodule "bergamot-translator-tests"]
+	path = bergamot-translator-tests
+	url = https://github.com/browsermt/bergamot-translator-tests
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
new file mode 160000
index 000000000..377100172
--- /dev/null
+++ b/bergamot-translator-tests
@@ -0,0 +1 @@
+Subproject commit 3771001720a8f01bba185ee5d5d908b7c266ef31

From 1184875cc9d75d56596736b0487968fdc7a35bb3 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Thu, 22 Apr 2021 16:01:39 +0100
Subject: [PATCH 197/442] Windows PCQueue support without Boost (#106)

---
 src/translator/pcqueue.h | 256 +++++++++++++++++++++++----------------
 1 file changed, 151 insertions(+), 105 deletions(-)

diff --git a/src/translator/pcqueue.h b/src/translator/pcqueue.h
index f0b354145..d6f458275 100644
--- a/src/translator/pcqueue.h
+++ b/src/translator/pcqueue.h
@@ -10,12 +10,14 @@
 #include <mutex>
 
 #ifdef __APPLE__
-#include <mach/mach.h>
-#include <mach/mach_traps.h>
 #include <mach/semaphore.h>
 #include <mach/task.h>
+#include <mach/mach_traps.h>
+#include <mach/mach.h>
 #elif defined(__linux)
 #include <semaphore.h>
+#elif defined(_WIN32) || defined(_WIN64)
+#include <windows.h>
 #else
 #include <boost/interprocess/sync/interprocess_semaphore.hpp>
 #endif
@@ -35,67 +37,107 @@ namespace bergamot {
 #ifdef __APPLE__
 
 class Semaphore {
-public:
-  explicit Semaphore(int value) : task_(mach_task_self()) {
-    ABORT_IF(KERN_SUCCESS !=
-                 semaphore_create(task_, &back_, SYNC_POLICY_FIFO, value),
-             "Could not create semaphore");
-  }
+  public:
+    explicit Semaphore(int value) : task_(mach_task_self()) {
+      ABORT_IF(KERN_SUCCESS != semaphore_create(task_, &back_, SYNC_POLICY_FIFO, value), "Could not create semaphore");
+    }
 
-  ~Semaphore() {
-    if (KERN_SUCCESS != semaphore_destroy(task_, back_)) {
-      std::cerr << "Could not destroy semaphore" << std::endl;
-      abort();
+    ~Semaphore() {
+      if (KERN_SUCCESS != semaphore_destroy(task_, back_)) {
+        std::cerr << "Could not destroy semaphore" << std::endl;
+        abort();
+      }
     }
-  }
 
-  void wait() {
-    ABORT_IF(KERN_SUCCESS != semaphore_wait(back_),
-             "Wait for semaphore failed");
-  }
+    void wait() {
+      ABORT_IF(KERN_SUCCESS != semaphore_wait(back_), "Wait for semaphore failed");
+    }
 
-  void post() {
-    ABORT_IF(KERN_SUCCESS != semaphore_signal(back_),
-             "Could not post to semaphore");
-  }
+    void post() {
+      ABORT_IF(KERN_SUCCESS != semaphore_signal(back_), "Could not post to semaphore");
+    }
 
-private:
-  semaphore_t back_;
-  task_t task_;
+  private:
+    semaphore_t back_;
+    task_t task_;
 };
 
-inline void WaitSemaphore(Semaphore &semaphore) { semaphore.wait(); }
+inline void WaitSemaphore(Semaphore &semaphore) {
+  semaphore.wait();
+}
 
 #elif defined(__linux)
 
 class Semaphore {
-public:
-  explicit Semaphore(unsigned int value) {
-    ABORT_IF(sem_init(&sem_, 0, value), "Could not create semaphore");
-  }
+  public:
+    explicit Semaphore(unsigned int value) {
+      ABORT_IF(sem_init(&sem_, 0, value), "Could not create semaphore");
+    }
 
-  ~Semaphore() {
-    if (-1 == sem_destroy(&sem_)) {
-      std::cerr << "Could not destroy semaphore " << std::endl;
-      abort();
+    ~Semaphore() {
+      if (-1 == sem_destroy(&sem_)) {
+        std::cerr << "Could not destroy semaphore" << std::endl;
+        abort();
+      }
     }
-  }
 
-  void wait() {
-    while (UTIL_UNLIKELY(-1 == sem_wait(&sem_))) {
-      ABORT_IF(errno != EINTR, "Wait for semaphore failed");
+    void wait() {
+      while (-1 == sem_wait(&sem_)) {
+        ABORT_IF(errno != EINTR, "Wait for semaphore failed");
+      }
     }
-  }
 
-  void post() {
-    ABORT_IF(-1 == sem_post(&sem_), "Could not post to semaphore");
-  }
+    void post() {
+      ABORT_IF(-1 == sem_post(&sem_), "Could not post to semaphore");
+    }
 
-private:
-  sem_t sem_;
+  private:
+    sem_t sem_;
 };
 
-inline void WaitSemaphore(Semaphore &semaphore) { semaphore.wait(); }
+inline void WaitSemaphore(Semaphore &semaphore) {
+  semaphore.wait();
+}
+
+#elif defined(_WIN32) || defined(_WIN64)
+
+class Semaphore {
+  public:
+    explicit Semaphore(LONG value) : sem_(CreateSemaphoreA(NULL, value, 2147483647, NULL)) {
+      ABORT_IF(!sem_, "Could not CreateSemaphore {}", GetLastError());
+    }
+
+    ~Semaphore() {
+      CloseHandle(sem_);
+    }
+
+
+    void wait() {
+      while (true) {
+        switch (WaitForSingleObject(sem_, 0L)) {
+          case WAIT_OBJECT_0:
+            return;
+          case WAIT_ABANDONED:
+            ABORT("A semaphore can't be abandoned, confused by Windows");
+          case WAIT_TIMEOUT:
+            continue;
+          case WAIT_FAILED:
+            ABORT("Waiting on Semaphore failed {}", GetLastError());
+        }
+      }
+    }
+
+    void post() {
+      ABORT_IF(!ReleaseSemaphore(sem_, 1, NULL), "Failed to release Semaphore {}", GetLastError());
+    }
+
+  private:
+    HANDLE sem_;
+};
+
+inline void WaitSemaphore(Semaphore &semaphore) {
+  semaphore.wait();
+}
 
 #else
 typedef boost::interprocess::interprocess_semaphore Semaphore;
@@ -113,7 +155,7 @@ inline void WaitSemaphore(Semaphore &on) {
   }
 }
 
-#endif // Apple
+#endif // Cases for semaphore support
 
 /**
  * Producer consumer queue safe for multiple producers and multiple consumers.
@@ -124,11 +166,13 @@ inline void WaitSemaphore(Semaphore &on) {
  * throw.
  */
 template <class T> class PCQueue {
-public:
+ public:
   explicit PCQueue(size_t size)
-      : empty_(size), used_(0), storage_(new T[size]),
-        end_(storage_.get() + size), produce_at_(storage_.get()),
-        consume_at_(storage_.get()) {}
+   : empty_(size), used_(0),
+     storage_(new T[size]),
+     end_(storage_.get() + size),
+     produce_at_(storage_.get()),
+     consume_at_(storage_.get()) {}
 
   // Add a value to the queue.
   void Produce(const T &val) {
@@ -141,8 +185,7 @@ template <class T> class PCQueue {
         empty_.post();
         throw;
       }
-      if (++produce_at_ == end_)
-        produce_at_ = storage_.get();
+      if (++produce_at_ == end_) produce_at_ = storage_.get();
     }
     used_.post();
   }
@@ -158,14 +201,14 @@ template <class T> class PCQueue {
         empty_.post();
         throw;
       }
-      if (++produce_at_ == end_)
-        produce_at_ = storage_.get();
+      if (++produce_at_ == end_) produce_at_ = storage_.get();
     }
     used_.post();
   }
 
+
   // Consume a value, assigning it to out.
-  T &Consume(T &out) {
+  T& Consume(T &out) {
     WaitSemaphore(used_);
     {
       std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
@@ -175,15 +218,14 @@ template <class T> class PCQueue {
         used_.post();
         throw;
       }
-      if (++consume_at_ == end_)
-        consume_at_ = storage_.get();
+      if (++consume_at_ == end_) consume_at_ = storage_.get();
     }
     empty_.post();
     return out;
   }
 
   // Consume a value, swapping it to out.
-  T &ConsumeSwap(T &out) {
+  T& ConsumeSwap(T &out) {
     WaitSemaphore(used_);
     {
       std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
@@ -193,13 +235,13 @@ template <class T> class PCQueue {
         used_.post();
         throw;
       }
-      if (++consume_at_ == end_)
-        consume_at_ = storage_.get();
+      if (++consume_at_ == end_) consume_at_ = storage_.get();
     }
     empty_.post();
     return out;
   }
 
+
   // Convenience version of Consume that copies the value to return.
   // The other version is faster.
   T Consume() {
@@ -208,7 +250,7 @@ template <class T> class PCQueue {
     return ret;
   }
 
-private:
+ private:
   // Number of empty spaces in storage_.
   Semaphore empty_;
   // Number of occupied spaces in storage_.
@@ -234,63 +276,67 @@ template <class T> struct UnboundedPage {
 };
 
 template <class T> class UnboundedSingleQueue {
-public:
-  UnboundedSingleQueue() : valid_(0) {
-    SetFilling(new UnboundedPage<T>());
-    SetReading(filling_);
-  }
+  public:
+    UnboundedSingleQueue() : valid_(0) {
+      SetFilling(new UnboundedPage<T>());
+      SetReading(filling_);
+    }
 
-  void Produce(T &&val) {
-    if (filling_current_ == filling_end_) {
-      UnboundedPage<T> *next = new UnboundedPage<T>();
-      filling_->next = next;
-      SetFilling(next);
+    void Produce(T &&val) {
+      if (filling_current_ == filling_end_) {
+        UnboundedPage<T> *next = new UnboundedPage<T>();
+        filling_->next = next;
+        SetFilling(next);
+      }
+      *(filling_current_++) = std::move(val);
+      valid_.post();
     }
-    *(filling_current_++) = std::move(val);
-    valid_.post();
-  }
 
-  void Produce(const T &val) { Produce(T(val)); }
+    void Produce(const T &val) {
+      Produce(T(val));
+    }
 
-  T &Consume(T &out) {
-    WaitSemaphore(valid_);
-    if (reading_current_ == reading_end_) {
-      SetReading(reading_->next);
+    T& Consume(T &out) {
+      WaitSemaphore(valid_);
+      if (reading_current_ == reading_end_) {
+        SetReading(reading_->next);
+      }
+      out = std::move(*(reading_current_++));
+      return out;
     }
-    out = std::move(*(reading_current_++));
-    return out;
-  }
 
-  // Warning: very much a no-guarantees race-condition-rich implementation!
-  // But sufficient for our specific purpose: The single thread that consumes
-  // is also the only one that checks Empty, and knows that it's racing.
-  bool Empty() const { return reading_current_ == filling_current_; }
+    // Warning: very much a no-guarantees race-condition-rich implementation!
+    // But sufficient for our specific purpose: The single thread that consumes
+    // is also the only one that checks Empty, and knows that it's racing.
+    bool Empty() const {
+      return reading_current_ == filling_current_;
+    }
 
-private:
-  void SetFilling(UnboundedPage<T> *to) {
-    filling_ = to;
-    filling_current_ = to->entries;
-    filling_end_ = filling_current_ + sizeof(to->entries) / sizeof(T);
-  }
-  void SetReading(UnboundedPage<T> *to) {
-    reading_.reset(to);
-    reading_current_ = to->entries;
-    reading_end_ = reading_current_ + sizeof(to->entries) / sizeof(T);
-  }
+  private:
+    void SetFilling(UnboundedPage<T> *to) {
+      filling_ = to;
+      filling_current_ = to->entries;
+      filling_end_ = filling_current_ + sizeof(to->entries) / sizeof(T);
+    }
+    void SetReading(UnboundedPage<T> *to) {
+      reading_.reset(to);
+      reading_current_ = to->entries;
+      reading_end_ = reading_current_ + sizeof(to->entries) / sizeof(T);
+    }
 
-  Semaphore valid_;
+    Semaphore valid_;
 
-  UnboundedPage<T> *filling_;
+    UnboundedPage<T> *filling_;
 
-  std::unique_ptr<UnboundedPage<T>> reading_;
+    std::unique_ptr<UnboundedPage<T> > reading_;
 
-  T *filling_current_;
-  T *filling_end_;
-  T *reading_current_;
-  T *reading_end_;
+    T *filling_current_;
+    T *filling_end_;
+    T *reading_current_;
+    T *reading_end_;
 
-  UnboundedSingleQueue(const UnboundedSingleQueue &) = delete;
-  UnboundedSingleQueue &operator=(const UnboundedSingleQueue &) = delete;
+    UnboundedSingleQueue(const UnboundedSingleQueue &) = delete;
+    UnboundedSingleQueue &operator=(const UnboundedSingleQueue &) = delete;
 };
 
 } // namespace bergamot

From fc6976ae297fe0fe17c9c7179201f915121783e7 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Thu, 22 Apr 2021 17:29:22 +0100
Subject: [PATCH 198/442] Remove dead code (#107)

Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
---
 src/translator/CMakeLists.txt           |  1 -
 src/translator/multifactor_priority.cpp |  7 -------
 src/translator/multifactor_priority.h   | 20 --------------------
 3 files changed, 28 deletions(-)
 delete mode 100644 src/translator/multifactor_priority.cpp
 delete mode 100644 src/translator/multifactor_priority.h

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 9bf7dca00..cbb8369c6 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -4,7 +4,6 @@ add_library(bergamot-translator STATIC
     text_processor.cpp
     sentence_splitter.cpp
     batch_translator.cpp 
-    multifactor_priority.cpp 
     request.cpp 
     batcher.cpp
     response.cpp
diff --git a/src/translator/multifactor_priority.cpp b/src/translator/multifactor_priority.cpp
deleted file mode 100644
index 0f93a8148..000000000
--- a/src/translator/multifactor_priority.cpp
+++ /dev/null
@@ -1,7 +0,0 @@
-#include "multifactor_priority.h"
-
-namespace marian {
-namespace bergamot {
-
-}  // namespace bergamot
-}  // namespace marian
diff --git a/src/translator/multifactor_priority.h b/src/translator/multifactor_priority.h
deleted file mode 100644
index 1e239f73b..000000000
--- a/src/translator/multifactor_priority.h
+++ /dev/null
@@ -1,20 +0,0 @@
-#ifndef SRC_BERGAMOT_MULTIFACTOR_PRIORITY_H_
-#define SRC_BERGAMOT_MULTIFACTOR_PRIORITY_H_
-
-#include "data/types.h"
-#include "definitions.h"
-#include "sys/time.h"
-
-namespace marian {
-namespace bergamot {
-
-struct MultiFactorPriority {
-  int nice; /* user configurable priority, at a request */
-  unsigned int Id;
-  /* What else should priority depend on? */
-  double priority() { return Id; }
-};
-} // namespace bergamot
-} // namespace marian
-
-#endif // SRC_BERGAMOT_MULTIFACTOR_PRIORITY_H_

From 7d2e74f3c0bbcc9e3b153783c7e0db63550cf0bb Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 26 Apr 2021 17:26:27 +0200
Subject: [PATCH 199/442] Changed underlying template parameter of
 AlignedMemory class (#111)

- AlignedMemory is AlignedVector<char> now instead of AlignedVector<const void*>
 - This solves the issue of allocating 8x of the actual required memory for
   loading files as bytes
---
 src/translator/definitions.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index 32998b9e9..73d83203b 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -22,8 +22,8 @@ template <class T, typename... Args> UPtr<T> UNew(Args &&... args) {
 
 template <class T> UPtr<T> UNew(UPtr<T> p) { return UPtr<T>(p); }
 
-/// Shortcut to AlignedVector<const void*> for byte arrays
-typedef AlignedVector<const void*> AlignedMemory;
+/// Shortcut to AlignedVector<char> for byte arrays
+typedef AlignedVector<char> AlignedMemory;
 
 } // namespace bergamot
 } // namespace marian

From fdf9e66cef172cc240cb2c1ad1bbf78e964dcc2a Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Mon, 26 Apr 2021 18:59:20 +0100
Subject: [PATCH 200/442] Windows workflows and mac framework accelerate (#108)

Windows still failing but getting closer
---
 .github/workflows/windows.yml | 74 +++--------------------------------
 3rd_party/marian-dev          |  2 +-
 2 files changed, 6 insertions(+), 70 deletions(-)

diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index fd1f21f2c..6c3a05f8b 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -17,12 +17,6 @@ jobs:
           # Windows CPU-only build
           - name: "Windows CPU-only"
             cuda: ""
-            gpu: false
-          # GPU Builds are commented out, for bergamot-translator CI runs.
-          # Windows CPU+GPU build
-          # - name: "Windows CPU+CUDA"
-          #   cuda: "10.2"
-          #   gpu: true
 
     runs-on: windows-2019
     name: ${{ matrix.name }}
@@ -42,89 +36,31 @@ jobs:
         echo "MKLROOT=${{ github.workspace }}\mkl" | Out-File -FilePath $env:GITHUB_ENV  -Encoding utf8 -Append
       shell: powershell
 
-    - name: Install CUDA
-      run: |
-        .\3rd_party\marian-dev\scripts\ci\install_cuda_windows.ps1 "10.2"
-        # Set CUDA_PATH environment variable so that CMake can find CUDA
-        echo "CUDA_PATH=$env:CUDA_PATH" | Out-File -FilePath $env:GITHUB_ENV  -Encoding utf8 -Append
-        echo "$env:CUDA_PATH/bin"       | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
-      shell: powershell
-      if: matrix.gpu == true
-
     - name: Prepare vcpkg
       uses: lukka/run-vcpkg@v4
       with:
-        vcpkgArguments: protobuf
-        vcpkgGitCommitId: 6185aa76504a5025f36754324abf307cc776f3da
+        vcpkgArguments: protobuf pcre2
+        vcpkgGitCommitId: 6185aa76504a5025f36754324abf307cc776f3da 
         vcpkgDirectory: ${{ github.workspace }}/vcpkg/
         vcpkgTriplet: x64-windows-static
 
-    # Windows CUDA builds use USE_NCCL=off due to compilation errors.
+    # Windows CPU only minimal build
     - name: Build Debug
       uses: lukka/run-cmake@v3
       with:
         buildDirectory: ${{ github.workspace }}/build/Debug
         cmakeAppendedArgs: '-G Ninja
           -DCMAKE_BUILD_TYPE="Debug"
-          -DOPENSSL_USE_STATIC_LIBS="TRUE"
-          -DOPENSSL_MSVC_STATIC_RT="TRUE"
-          -DCOMPILE_CPU="TRUE"
-          -DCOMPILE_CUDA="${{ matrix.gpu }}"
-          -DCOMPILE_SERVER="FALSE"
-          -DCOMPILE_TESTS="TRUE"
-          -DUSE_FBGEMM="TRUE"
-          -DUSE_MPI="FALSE"
-          -DUSE_NCCL="FALSE"
-          -DUSE_SENTENCEPIECE="TRUE"
-          -DUSE_STATIC_LIBS="TRUE"'
-        cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
-        cmakeListsTxtPath: ${{ github.workspace }}/CMakeLists.txt
-        useVcpkgToolchainFile: true
-      # Building in Debug is sufficient for the all-in CPU+GPU compilation;
-      # its main purpose is to detect warnings that the Release build is not
-      # able to find sometimes.
-      if: matrix.gpu == true
-
-    # Windows CUDA builds use USE_NCCL=off due to compilation errors
-    # Boost is pre-installed on Azure/GitHub-hosted Windows runners
-    # https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#boost
-    # (not used yet)
-    - name: Build Release
-      uses: lukka/run-cmake@v3
-      with:
-        buildDirectory: ${{ github.workspace }}/build/
-        cmakeAppendedArgs: '-G Ninja
-          -DBOOST_ROOT="$(BOOST_ROOT_1_72_0)"
-          -DBOOST_INCLUDEDIR="$(BOOST_ROOT_1_72_0)/include"
-          -DBOOST_LIBRARYDIR="$(BOOST_ROOT_1_72_0)/lib"
-          -DCMAKE_BUILD_TYPE="Release"
-          -DOPENSSL_USE_STATIC_LIBS="TRUE"
-          -DOPENSSL_MSVC_STATIC_RT="TRUE"
-          -DCOMPILE_CPU="TRUE"
-          -DCOMPILE_CUDA="${{ matrix.gpu }}"
-          -DCOMPILE_SERVER="FALSE"
-          -DCOMPILE_TESTS="TRUE"
-          -DUSE_FBGEMM="TRUE"
-          -DUSE_MPI="FALSE"
-          -DUSE_NCCL="FALSE"
-          -DUSE_SENTENCEPIECE="TRUE"
+          -DUSE_WASM_COMPATIBLE_SOURCE="OFF"
           -DUSE_STATIC_LIBS="TRUE"'
         cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
         cmakeListsTxtPath: ${{ github.workspace }}/CMakeLists.txt
         useVcpkgToolchainFile: true
 
-    # Removing unit-tests, taken care of in browsermt/marian-dev
-    # - name: Run unit tests
-    #   working-directory: build/
-    #   run: ctest
-    #   # Not run in GPU builds because GitHub-hosted VMs do not have GPUs
-    #   if: matrix.gpu == false
 
     - name: Print versions
       working-directory: build/
       run: |
-        .\marian.exe --version
-        .\marian-decoder.exe --version
-        .\marian-scorer.exe --version
+        .\app\bergamot-translator-app.exe --version
         dir *.exe
       shell: cmd
diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 0f0bcf996..46a221873 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 0f0bcf99626c660227bb68b76267a8d2451e7172
+Subproject commit 46a22187341ff51b3f11a8cb1edf51c995e583ca

From fa2003e70d0baf88a5db6f51086ff899c7192e63 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 27 Apr 2021 15:56:39 +0100
Subject: [PATCH 201/442] Cleanup API: Refactor request on-complete transition
 (#80)

---
 app/service-cli-bytearray.cpp       |   8 +-
 app/service-cli.cpp                 |   9 ++-
 src/translator/CMakeLists.txt       |   2 +-
 src/translator/TranslationModel.cpp |  25 +++---
 src/translator/batch_translator.cpp |   5 +-
 src/translator/request.cpp          |  26 ++----
 src/translator/request.h            | 121 ++++++++++++++--------------
 src/translator/response.cpp         | 106 ------------------------
 src/translator/response.h           |  34 ++------
 src/translator/response_builder.cpp |  87 ++++++++++++++++++++
 src/translator/response_builder.h   |  93 +++++++++++++++++++++
 src/translator/response_options.h   |  50 ++++++++++++
 src/translator/sentence_ranges.cpp  |  20 ++++-
 src/translator/sentence_ranges.h    |  10 +--
 src/translator/service.cpp          |  58 ++++++++++++-
 src/translator/service.h            | 100 +++++++++++++++++++----
 16 files changed, 493 insertions(+), 261 deletions(-)
 delete mode 100644 src/translator/response.cpp
 create mode 100644 src/translator/response_builder.cpp
 create mode 100644 src/translator/response_builder.h
 create mode 100644 src/translator/response_options.h

diff --git a/app/service-cli-bytearray.cpp b/app/service-cli-bytearray.cpp
index f868d4def..d8c7059ef 100644
--- a/app/service-cli-bytearray.cpp
+++ b/app/service-cli-bytearray.cpp
@@ -27,8 +27,14 @@ int main(int argc, char *argv[]) {
   std::string input = std_input.str();
   using marian::bergamot::Response;
 
+  marian::bergamot::ResponseOptions responseOptions;
+  responseOptions.qualityScores = true;
+  responseOptions.alignment = true;
+  responseOptions.alignmentThreshold = 0.2f;
+
   // Wait on future until Response is complete
-  std::future<Response> responseFuture = service.translate(std::move(input));
+  std::future<Response> responseFuture =
+      service.translate(std::move(input), responseOptions);
   responseFuture.wait();
   Response response = responseFuture.get();
 
diff --git a/app/service-cli.cpp b/app/service-cli.cpp
index 6ed4d81f6..d7c72e604 100644
--- a/app/service-cli.cpp
+++ b/app/service-cli.cpp
@@ -8,6 +8,7 @@
 #include "marian.h"
 #include "translator/parser.h"
 #include "translator/response.h"
+#include "translator/response_options.h"
 #include "translator/service.h"
 
 int main(int argc, char *argv[]) {
@@ -21,8 +22,14 @@ int main(int argc, char *argv[]) {
   std::string input = std_input.str();
   using marian::bergamot::Response;
 
+  marian::bergamot::ResponseOptions responseOptions;
+  responseOptions.qualityScores = true;
+  responseOptions.alignment = true;
+  responseOptions.alignmentThreshold = 0.2f;
+
   // Wait on future until Response is complete
-  std::future<Response> responseFuture = service.translate(std::move(input));
+  std::future<Response> responseFuture =
+      service.translate(std::move(input), responseOptions);
   responseFuture.wait();
   Response response = responseFuture.get();
 
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index cbb8369c6..d7c8e3cfe 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -6,7 +6,7 @@ add_library(bergamot-translator STATIC
     batch_translator.cpp 
     request.cpp 
     batcher.cpp
-    response.cpp
+    response_builder.cpp
     batch.cpp
     sentence_ranges.cpp
     service.cpp
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
index 06b04eb5a..026a126ea 100644
--- a/src/translator/TranslationModel.cpp
+++ b/src/translator/TranslationModel.cpp
@@ -9,6 +9,7 @@
 // All local project includes
 #include "TranslationModel.h"
 #include "translator/parser.h"
+#include "translator/response.h"
 #include "translator/service.h"
 
 TranslationModel::TranslationModel(const std::string &config,
@@ -21,31 +22,25 @@ TranslationModel::~TranslationModel() {}
 std::vector<TranslationResult>
 TranslationModel::translate(std::vector<std::string> &&texts,
                             TranslationRequest request) {
-  // Implementing a non-async version first. Unpleasant, but should work.
-  std::promise<std::vector<TranslationResult>> promise;
-  auto future = promise.get_future();
 
   // This code, move into async?
   std::vector<TranslationResult> translationResults;
-  for (auto &text : texts) {
-    // Collect future as marian::bergamot::TranslationResult
-    auto intermediate = service_.translate(std::move(text));
-    intermediate.wait();
-    auto marianResponse(std::move(intermediate.get()));
-
+  std::vector<marian::bergamot::Response> responses =
+      service_.translateMultiple(std::move(texts), request);
+  for (auto &response : responses) {
     TranslationResult::SentenceMappings sentenceMappings;
-    for (size_t idx = 0; idx < marianResponse.size(); idx++) {
-      marian::string_view src = marianResponse.source.sentence(idx);
-      marian::string_view tgt = marianResponse.target.sentence(idx);
+    for (size_t idx = 0; idx < response.size(); idx++) {
+      marian::string_view src = response.source.sentence(idx);
+      marian::string_view tgt = response.target.sentence(idx);
       sentenceMappings.emplace_back(std::string_view(src.data(), src.size()),
                                     std::string_view(tgt.data(), tgt.size()));
     }
 
     // In place construction.
     translationResults.emplace_back(
-        std::move(marianResponse.source.text), // &&marianResponse.source_
-        std::move(marianResponse.target.text), // &&marianResponse.translation_
-        std::move(sentenceMappings)            // &&sentenceMappings
+        std::move(response.source.text), // &&response.source_
+        std::move(response.target.text), // &&response.translation_
+        std::move(sentenceMappings)      // &&sentenceMappings
     );
   }
 
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 19cbaf9d1..6b2425d26 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -63,11 +63,14 @@ void BatchTranslator::translate(Batch &batch) {
   std::vector<data::SentenceTuple> batchVector;
 
   auto &sentences = batch.sentences();
+  size_t batchSequenceNumber{0};
   for (auto &sentence : sentences) {
-    data::SentenceTuple sentence_tuple(sentence.lineNumber());
+    data::SentenceTuple sentence_tuple(batchSequenceNumber);
     Segment segment = sentence.getUnderlyingSegment();
     sentence_tuple.push_back(segment);
     batchVector.push_back(sentence_tuple);
+
+    ++batchSequenceNumber;
   }
 
   size_t batchSize = batchVector.size();
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index b6d243857..7e9b73977 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -11,18 +11,17 @@ namespace marian {
 namespace bergamot {
 
 // -----------------------------------------------------------------
-Request::Request(size_t Id, size_t lineNumberBegin,
-                 std::vector<Ptr<Vocab const>> &vocabs, AnnotatedText &&source,
-                 Segments &&segments, std::promise<Response> responsePromise)
-    : Id_(Id), lineNumberBegin_(lineNumberBegin), vocabs_(&vocabs),
-      source_(std::move(source)), segments_(std::move(segments)),
-      response_(std::move(responsePromise)) {
+Request::Request(size_t Id, Segments &&segments,
+                 ResponseBuilder &&responseBuilder)
+    : Id_(Id), segments_(std::move(segments)),
+      responseBuilder_(std::move(responseBuilder))
+
+{
 
   counter_ = segments_.size();
   histories_.resize(segments_.size(), nullptr);
 }
 
-size_t Request::lineNumberBegin() const { return lineNumberBegin_; }
 size_t Request::numSegments() const { return segments_.size(); }
 
 size_t Request::segmentTokens(size_t index) const {
@@ -39,17 +38,10 @@ void Request::processHistory(size_t index, Ptr<History> history) {
   // In case this is last request in, completeRequest is called, which sets the
   // value of the promise.
   if (--counter_ == 0) {
-    completeRequest();
+    responseBuilder_(std::move(histories_));
   }
 }
 
-void Request::completeRequest() {
-  // Request no longer needs to hold the content, can transfer it to
-  // Response.
-  Response response(std::move(source_), std::move(histories_), *vocabs_);
-  response_.set_value(std::move(response));
-}
-
 bool Request::operator<(const Request &b) const {
   // Among Requests, only sequence id is used for obtaining priority.
   return Id_ < b.Id_;
@@ -64,10 +56,6 @@ size_t RequestSentence::numTokens() const {
   return (request_->segmentTokens(index_));
 }
 
-size_t RequestSentence::lineNumber() const {
-  return (request_->lineNumberBegin() + index_);
-}
-
 void RequestSentence::completeSentence(Ptr<History> history) {
   // Relays completeSentence into request's processHistory, using index
   // information.
diff --git a/src/translator/request.h b/src/translator/request.h
index 605dea7db..e2188cdbe 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -1,24 +1,9 @@
-//
-// Defines:
-//
-// Request: holds the input text of a text, Segments (vector<Words>) which are
-// to go to the batching mechanism and alignments between the processed
-// segments and the input text (sourceTokenRanges). In addition, Request takes
-// care of the barrier which fires when all the Segments in a request are done
-// translating by the workers (BatchTranslator).
-// TODO(jerinphilip):  Extend Request with notions of Priority (sequence,
-// user-given).
-//
-// RequestSentence: is a tuple of (index, Ptr<Request>). This provides the
-// batching mechanism access to the segment within the request. The backref to
-// Request allows event triggering the barrier upon completion of the last
-// sentence by a worker.
-
 #ifndef SRC_BERGAMOT_REQUEST_H_
 #define SRC_BERGAMOT_REQUEST_H_
 
 #include "definitions.h"
 #include "response.h"
+#include "response_builder.h"
 #include "sentence_ranges.h"
 
 #include "common/logging.h"
@@ -33,80 +18,96 @@
 namespace marian {
 namespace bergamot {
 
+/// A Request is an internal representation used to represent a request after
+/// processed by TextProcessor into sentences constituted by marian::Words.
+///
+/// The batching mechanism (Batcher) draws from multiple Requests and compiles
+/// sentences into a batch. When a batch completes translation (at
+/// BatchTranslator, intended in a different thread), backward propogation
+/// happens through:
+///
+/// ```cpp
+///   Batch::completeBatch(...)
+///       -> RequestSentence::completeSentence(..)
+///          -> Request::processHistory(...)
+/// ```
+///
+/// When all sentences in a Request are completed, responseBuilder is
+/// triggered with the compiled Histories, to construct the Response
+/// corresponding to the Request and set value of the promise which triggers the
+/// future at client.
 class Request {
 public:
-  Request(size_t Id, size_t lineNumberBegin,
-          std::vector<Ptr<Vocab const>> &vocabs_, AnnotatedText &&source,
-          Segments &&segments, std::promise<Response> responsePromise);
-
-  // Obtain the count of tokens in the segment correponding to index. Used to
-  // insert sentence from multiple requests into the corresponding size bucket.
+  /// Constructs an internal representation of the Request identified by Id,
+  /// processed Segments and accepts a callback (ResponseBuilder) which builds
+  /// the Response upon completion of the Request.
+  ///
+  ///
+  /// @param [in] Id: Identifier assigned to Request by Service.
+  /// @param [in] segments: Each segment is a unit to be translated.
+  /// @param [in] responseBuilder: Callback function (of ResponseBuilder type)
+  /// to be triggered upon the completion of translation of all units in a
+  /// Request.
+  Request(size_t Id, Segments &&segments, ResponseBuilder &&responseBuilder);
+
+  /// Obtain the count of tokens in the segment correponding to index. Used to
+  /// insert sentence from multiple requests into the corresponding size bucket.
   size_t segmentTokens(size_t index) const;
 
-  // Obtain number of segments in a request.
+  /// Obtain number of segments in a request.
   size_t numSegments() const;
-  size_t lineNumberBegin() const;
 
-  // Obtains segment corresponding to index  to create a batch of segments among
-  // several requests.
+  /// Obtains segment corresponding to index  to create a batch of segments
+  /// among several requests.
   Segment getSegment(size_t index) const;
 
-  // For notions of priority among requests, used to enable std::set in
-  // Batcher.
+  /// For notions of priority among requests, used to enable std::set in
+  /// Batcher.
   bool operator<(const Request &request) const;
 
-  // Processes a history obtained after translating in a heterogenous batch
-  // compiled from requests.
+  /// Processes a history obtained after translating in a heterogenous batch
+  /// compiled from requests.
   void processHistory(size_t index, Ptr<History> history);
 
-  // On completion of last segment, sets value of the promise.
-  void completeRequest();
-
 private:
   size_t Id_;
-  size_t lineNumberBegin_;
 
-  // Multiple translation-workers can concurrently access the same Request. The
-  // following atomic atomically operates on the variable holding sentences
-  // remaining to be translated.
+  /// Multiple translation-workers can concurrently access the same Request. The
+  /// following atomic atomically operates on the variable holding sentences
+  /// remaining to be translated.
   std::atomic<int> counter_;
 
-  // source_ holds the source string to be translated. segments_ hold the
-  // sentences generated from source_ in vector<Words>. sourceRanges_ are
-  // string_views of the text corresponding to these words, pointing to
-  // sequences in source_. histories_ is a buffer which eventually stores the
-  // translations of each segment in the corresponding index.
-  AnnotatedText source_;
+  /// segments_ hold the sentences processed into Words which generated from
+  /// input string.
   Segments segments_;
-  std::vector<Ptr<History>> histories_;
 
-  // Members above are moved into newly constructed Response on completion
-  // of translation of all segments. The promise below is set to this Response
-  // value. future to this promise is made available to the user through
-  // Service.
-  std::promise<Response> response_;
+  /// histories_ is a buffer which eventually stores the translations of each
+  /// segment in the corresponding index.
+  std::vector<Ptr<History>> histories_;
 
-  // Constructing Response requires the vocabs_ used to generate Request.
-  std::vector<Ptr<Vocab const>> *vocabs_;
+  /// Constructing Response requires the vocabs_ used to generate Request.
+  /// std::vector<Ptr<Vocab const>> *vocabs_;
+  ResponseBuilder responseBuilder_;
 };
 
+/// A RequestSentence provides a view to a sentence within a Request. Existence
+/// of this class allows the sentences and associated information to be kept
+/// within Request, while batching mechanism (Batcher) compiles Batch from
+/// RequestSentence-s coming from different Requests.
 class RequestSentence {
-  // A RequestSentence provides a view to a sentence within a Request. Existence
-  // of this class allows the sentences and associated information to be kept
-  // within Request.
 
 public:
   RequestSentence(size_t, Ptr<Request>);
-  size_t numTokens() const;
 
-  // lineNumber in Request, used for matching marian-decoder. SentenceTuple
-  // requires lineNumber to be set for Corpus based batches.
-  size_t lineNumber() const;
+  /// Number of tokens in the segment this RequestSentence represents. Used to
+  /// order by length in batching.
+  size_t numTokens() const;
 
-  // Accessor to the segment represented by the RequestSentence.
+  /// Accessor to the segment represented by the RequestSentence.
   Segment getUnderlyingSegment() const;
 
-  // Forwards call to Request, checking for completion.
+  /// Forwards history to Request to set history corresponding to this
+  /// RequestSentence.
   void completeSentence(Ptr<History> history);
 
   friend bool operator<(const RequestSentence &a, const RequestSentence &b);
diff --git a/src/translator/response.cpp b/src/translator/response.cpp
deleted file mode 100644
index e5bc38ff9..000000000
--- a/src/translator/response.cpp
+++ /dev/null
@@ -1,106 +0,0 @@
-#include "response.h"
-#include "common/logging.h"
-#include "data/alignment.h"
-#include "sentence_ranges.h"
-
-#include <utility>
-
-namespace marian {
-namespace bergamot {
-
-Response::Response(AnnotatedText &&source, Histories &&histories,
-                   std::vector<Ptr<Vocab const>> &vocabs)
-    : source(std::move(source)) {
-  // Reserving length at least as much as source_ seems like a reasonable thing
-  // to do to avoid reallocations.
-  target.text.reserve(source.text.size());
-
-  // In a first step, the decoded units (individual senteneces) are compiled
-  // into a huge string. This is done by computing indices first and appending
-  // to the string as each sentences are decoded.
-  std::vector<std::pair<size_t, size_t>> translationRanges;
-  std::vector<size_t> sentenceBegins;
-
-  size_t offset{0};
-  bool first{true};
-
-  for (auto &history : histories) {
-    // TODO(jerin): Change hardcode of nBest = 1
-    NBestList onebest = history->nBest(1);
-
-    Result result = onebest[0]; // Expecting only one result;
-    Words words = std::get<0>(result);
-    auto targetVocab = vocabs.back();
-
-    std::string decoded;
-    std::vector<string_view> targetMappings;
-    targetVocab->decodeWithByteRanges(words, decoded, targetMappings);
-
-    if (first) {
-      first = false;
-    } else {
-      target.text += " ";
-      ++offset;
-    }
-
-    sentenceBegins.push_back(translationRanges.size());
-    target.text += decoded;
-    auto decodedStringBeginMarker = targetMappings.front().begin();
-    for (auto &sview : targetMappings) {
-      size_t startIdx = offset + sview.begin() - decodedStringBeginMarker;
-      translationRanges.emplace_back(startIdx, startIdx + sview.size());
-    }
-
-    offset += decoded.size();
-
-    // Alignments
-    // TODO(jerinphilip): The following double conversion might not be
-    // necessary. Hard alignment can directly be exported, but this would mean
-    // WASM bindings for a structure deep within marian source.
-    auto hyp = std::get<1>(result);
-    auto softAlignment = hyp->tracebackAlignment();
-    auto hardAlignment = data::ConvertSoftAlignToHardAlign(
-        softAlignment, /*threshold=*/0.2f); // TODO(jerinphilip): Make this a
-                                            // configurable parameter.
-
-    Alignment unified_alignment;
-    for (auto &p : hardAlignment) {
-      unified_alignment.emplace_back((Point){p.srcPos, p.tgtPos, p.prob});
-    }
-
-    alignments.push_back(std::move(unified_alignment));
-
-    // Quality scores: Sequence level is obtained as normalized path scores.
-    // Word level using hypothesis traceback. These are most-likely logprobs.
-    auto normalizedPathScore = std::get<2>(result);
-    auto wordQualities = hyp->tracebackWordScores();
-    wordQualities.pop_back();
-    qualityScores.push_back((Quality){normalizedPathScore, wordQualities});
-  }
-
-  // Once we have the indices in translation (which might be resized a few
-  // times) ready, we can prepare and store the string_view as annotations
-  // instead. This is accomplished by iterating over available sentences using
-  // sentenceBegin and using addSentence(...) API from Annotation.
-
-  for (size_t i = 1; i <= sentenceBegins.size(); i++) {
-    std::vector<string_view> targetMappings;
-    size_t begin = sentenceBegins[i - 1];
-    size_t safe_end = (i == sentenceBegins.size()) ? translationRanges.size()
-                                                   : sentenceBegins[i];
-
-    for (size_t idx = begin; idx < safe_end; idx++) {
-      auto &p = translationRanges[idx];
-      size_t begin_idx = p.first;
-      size_t end_idx = p.second;
-
-      const char *data = &target.text[begin_idx];
-      size_t size = end_idx - begin_idx;
-      targetMappings.emplace_back(data, size);
-    }
-
-    target.addSentence(targetMappings);
-  }
-}
-} // namespace bergamot
-} // namespace marian
diff --git a/src/translator/response.h b/src/translator/response.h
index 4f87b8dac..3b1f48da8 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -40,34 +40,12 @@ struct Quality {
 /// AnnotatedText provides an API to access markings of (sub)-word and
 /// sentences boundaries, which are required to interpret Quality and
 /// Alignment (s) at the moment.
-class Response {
-
-public:
-  ///
-  Response(AnnotatedText &&source, Histories &&histories,
-           std::vector<Ptr<Vocab const>> &vocabs);
-
-  /// \cond HIDDEN_PUBLIC
-  // Move constructor.
-  Response(Response &&other)
-      : source(std::move(other.source)), target(std::move(other.target)),
-        alignments(std::move(other.alignments)),
-        qualityScores(std::move(other.qualityScores)){};
-
-  // The following copy bans are not stricitly required anymore since Annotation
-  // is composed of the ByteRange primitive (which was previously string_view
-  // and required to be bound to string), but makes movement efficient by
-  // banning these letting compiler complain about copies.
-
-  Response(const Response &) = delete;
-  Response &operator=(const Response &) = delete;
-
-  /// \endcond
-
-  /// Number of sentences translated. The processing of a text of into sentences
-  /// are handled internally, and this information can be used to iterate
-  /// through meaningful units of translation for which alignment and quality
-  /// information are available.
+struct Response {
+  /// Convenience function to obtain number of units translated. Same as
+  /// `.source.numSentences()` and `.target.numSentences().` The processing of a
+  /// text of into sentences are handled internally, and this information can be
+  /// used to iterate through meaningful units of translation for which
+  /// alignment and quality information are available.
   const size_t size() const { return source.numSentences(); }
 
   /// source text and annotations of (sub-)words and sentences.
diff --git a/src/translator/response_builder.cpp b/src/translator/response_builder.cpp
new file mode 100644
index 000000000..c62470789
--- /dev/null
+++ b/src/translator/response_builder.cpp
@@ -0,0 +1,87 @@
+#include "response_builder.h"
+
+namespace marian {
+namespace bergamot {
+
+void ResponseBuilder::buildQualityScores(Histories &histories,
+                                         Response &response) {
+  std::vector<Quality> qualityScores;
+  for (auto &history : histories) {
+    // TODO(jerin): Change hardcode of nBest = 1
+    NBestList onebest = history->nBest(1);
+
+    Result result = onebest[0]; // Expecting only one result;
+    Words words = std::get<0>(result);
+    auto hyp = std::get<1>(result);
+    // Quality scores: Sequence level is obtained as normalized path scores.
+    // Word level using hypothesis traceback. These are most-likely
+    // logprobs.
+    auto normalizedPathScore = std::get<2>(result);
+    auto wordQualities = hyp->tracebackWordScores();
+    wordQualities.pop_back();
+    response.qualityScores.push_back(
+        Quality{normalizedPathScore, wordQualities});
+  }
+}
+
+void ResponseBuilder::buildAlignments(Histories &histories,
+                                      Response &response) {
+  for (auto &history : histories) {
+    // TODO(jerin): Change hardcode of nBest = 1
+    NBestList onebest = history->nBest(1);
+
+    Result result = onebest[0]; // Expecting only one result;
+    Words words = std::get<0>(result);
+    // Alignments
+    // TODO(jerinphilip): The following double conversion might not be
+    // necessary. Hard alignment can directly be exported, but this would
+    // mean WASM bindings for a structure deep within marian source.
+    auto hyp = std::get<1>(result);
+    auto softAlignment = hyp->tracebackAlignment();
+    auto threshold = responseOptions_.alignmentThreshold;
+    auto hardAlignment =
+        data::ConvertSoftAlignToHardAlign(softAlignment, threshold);
+    Alignment unified_alignment;
+    for (auto &p : hardAlignment) {
+      unified_alignment.emplace_back(Point{p.srcPos, p.tgtPos, p.prob});
+    }
+
+    response.alignments.push_back(std::move(unified_alignment));
+  }
+}
+
+void ResponseBuilder::buildTranslatedText(Histories &histories,
+                                          Response &response) {
+  // Reserving length at least as much as source_ seems like a reasonable
+  // thing to do to avoid reallocations.
+  response.target.text.reserve(response.source.text.size());
+
+  size_t offset{0};
+  bool first{true};
+
+  for (auto &history : histories) {
+    // TODO(jerin): Change hardcode of nBest = 1
+    NBestList onebest = history->nBest(1);
+
+    Result result = onebest[0]; // Expecting only one result;
+    Words words = std::get<0>(result);
+    auto targetVocab = vocabs_->back();
+
+    std::string decoded;
+    std::vector<string_view> targetSentenceMappings;
+    targetVocab->decodeWithByteRanges(words, decoded, targetSentenceMappings);
+
+    // delimiter can be used to fill in the blanks from source as well.
+    std::string delimiter;
+    if (first) {
+      first = false;
+    } else {
+      delimiter = " ";
+    }
+
+    response.target.appendSentence(delimiter, decoded, targetSentenceMappings);
+  }
+}
+
+} // namespace bergamot
+} // namespace marian
diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
new file mode 100644
index 000000000..85caffb6c
--- /dev/null
+++ b/src/translator/response_builder.h
@@ -0,0 +1,93 @@
+#ifndef SRC_BERGAMOT_RESPONSE_BUILDER_H_
+#define SRC_BERGAMOT_RESPONSE_BUILDER_H_
+
+#include "data/types.h"
+#include "response.h"
+#include "response_options.h"
+
+// For now we will work with this, to avoid complaints another structure is hard
+// to operate with.
+
+namespace marian {
+namespace bergamot {
+
+/// ResponseBuilder is a callback functor. It is expected to be bound to a
+/// Request after giving it the context of options, vocabs and promise to set.
+/// It constructs the Response and it's members based on options
+/// (quality=on|off, alignments=on|off, mappings=on|off, splitmode=sentence |
+/// paragraph).
+
+class ResponseBuilder {
+public:
+  /// @param [in] responseOptions: ResponseOptions, indicating what to include
+  /// or not in the response and any additional configurable parameters.
+  /// @param [in] vocabs: marian vocab object (used in decoding)
+  /// @param [in] promise: promise to set with the constructed Response.
+  ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source,
+                  std::vector<Ptr<Vocab const>> &vocabs,
+                  std::promise<Response> &&promise)
+      : responseOptions_(responseOptions), source_(std::move(source)),
+        vocabs_(&vocabs), promise_(std::move(promise)) {}
+
+  /// Constructs and sets the promise of a Response object from obtained
+  /// histories after translating.
+  /// @param [in] histories: Histories obtained after translating the Request
+  /// from which this functor is called.
+  void operator()(Histories &&histories) {
+    // TODO(jerinphilip) load ResponseOptions into options and turn build
+    // functions on or off.
+    // responseOptions_ is unused, but we can try something here.
+    ABORT_IF(source_.numSentences() != histories.size(),
+             "Mismatch in source and translated sentences");
+    Response response;
+
+    // Move source_ into response.
+    response.source = std::move(source_);
+
+    // Should be after source is set
+    buildTranslatedText(histories, response);
+
+    // Should always be after buildTranslatedText
+    if (responseOptions_.qualityScores) {
+      buildQualityScores(histories, response);
+    }
+
+    if (responseOptions_.alignment) {
+      buildAlignments(histories, response);
+    }
+
+    // Once complete, set promise.
+    promise_.set_value(std::move(response));
+  }
+
+private:
+  /// Builds qualityScores from histories and writes to response. expects
+  /// buildTranslatedText to be run before to be able to obtain target text and
+  /// subword information.
+  /// @param histories [in]
+  /// @param response [out]
+  void buildQualityScores(Histories &histories, Response &response);
+
+  /// Builds alignments from histories and writes onto response.
+  /// @param histories [in]
+  /// @param response [out]
+  void buildAlignments(Histories &histories, Response &response);
+
+  /// Builds translated text and subword annotations and writes onto response.
+  /// @param histories [in]
+  /// @param response [out]
+  void buildTranslatedText(Histories &histories, Response &response);
+
+  // Data members are context/curried args for the functor.
+
+  ResponseOptions responseOptions_;
+  std::vector<Ptr<Vocab const>> *vocabs_; // vocabs are required for decoding
+                                          // and any source validation checks.
+  std::promise<Response> promise_; //  To be set when callback triggered and
+                                   //  after Response constructed.
+  AnnotatedText source_;
+};
+} // namespace bergamot
+} // namespace marian
+
+#endif //  SRC_BERGAMOT_RESPONSE_BUILDER_H_
diff --git a/src/translator/response_options.h b/src/translator/response_options.h
new file mode 100644
index 000000000..ed3cce3a5
--- /dev/null
+++ b/src/translator/response_options.h
@@ -0,0 +1,50 @@
+#ifndef SRC_BERGAMOT_RESPONSE_OPTIONS_H_
+#define SRC_BERGAMOT_RESPONSE_OPTIONS_H_
+#include <string>
+
+namespace marian {
+namespace bergamot {
+
+enum ConcatStrategy {
+  /// Target text is constructed faithful to the source-text  structure.
+  FAITHFUL,
+
+  /// Target text is concatenated by a space.
+  SPACE
+};
+
+enum QualityScoreType {
+  /// Provide a free quality-score that comes with the machine-translation model
+  /// itself.
+  FREE,
+
+  /// An expensive quality-score that runs additional computations to determine
+  /// quality of an output.
+  EXPENSIVE
+};
+
+/// ResponseOptions dictate how to construct a Response for an input string of
+/// text to be translated.
+struct ResponseOptions {
+  bool qualityScores{false}; ///< Include quality-scores or not.
+  bool alignment{false};     ///< Include alignments or not.
+
+  /// Whether to include sentenceMappings or not. Alignments require
+  /// sentenceMappings and are available irrespective of this option if
+  /// `alignment=true`.
+  bool sentenceMappings{false};
+
+  /// Threshold between `[0.0f, 1.0f]` to filter alignments into a sparse
+  /// matrix. Higher value implies stronger filtering leading to provision of
+  /// higher-confidence matches. `1.0f` gives argmax (not the full-dense
+  /// matrix).
+  float alignmentThreshold{0.2f};
+
+  QualityScoreType qualityScoreType{QualityScoreType::FREE};
+  ConcatStrategy concatStrategy{ConcatStrategy::FAITHFUL};
+};
+
+} // namespace bergamot
+} // namespace marian
+
+#endif //  SRC_BERGAMOT_RESPONSE_OPTIONS_H_
diff --git a/src/translator/sentence_ranges.cpp b/src/translator/sentence_ranges.cpp
index aae9dd346..da9d3eee0 100644
--- a/src/translator/sentence_ranges.cpp
+++ b/src/translator/sentence_ranges.cpp
@@ -32,11 +32,11 @@ ByteRange Annotation::sentence(size_t sentenceIdx) const {
     // the flatByteRange and non-empty sentence before this happened and
     // construct empty string-view equivalent ByteRange.
     ByteRange eos = flatByteRanges_[eosId - 1];
-    sentenceByteRange = (ByteRange){eos.end, eos.end};
+    sentenceByteRange = ByteRange{eos.end, eos.end};
   } else {
     ByteRange bos = flatByteRanges_[bosId];
     ByteRange eos = flatByteRanges_[eosId - 1];
-    sentenceByteRange = (ByteRange){bos.begin, eos.end};
+    sentenceByteRange = ByteRange{bos.begin, eos.end};
   }
   return sentenceByteRange;
 }
@@ -56,6 +56,20 @@ string_view AnnotatedText::sentence(size_t sentenceIdx) const {
   return asStringView(sentenceAsByteRange);
 }
 
+void AnnotatedText::appendSentence(std::string prefix, std::string &reference,
+                                   std::vector<string_view> &wordRanges) {
+  text += prefix;
+  size_t offset = text.size(); // Get size before to do ByteRange arithmetic
+  text += reference;           // Append reference to text
+  std::vector<ByteRange> sentence;
+  for (auto &wordView : wordRanges) {
+    size_t thisWordBegin = offset + wordView.data() - &reference[0];
+    sentence.push_back(
+        ByteRange{thisWordBegin, thisWordBegin + wordView.size()});
+  }
+  annotation.addSentence(sentence);
+}
+
 void AnnotatedText::addSentence(std::vector<string_view> &wordRanges) {
   addSentence(std::begin(wordRanges), std::end(wordRanges));
 };
@@ -65,7 +79,7 @@ void AnnotatedText::addSentence(std::vector<string_view>::iterator begin,
   std::vector<ByteRange> sentence;
   for (auto p = begin; p != end; p++) {
     size_t begin_offset = p->data() - &text[0];
-    sentence.push_back((ByteRange){begin_offset, begin_offset + p->size()});
+    sentence.push_back(ByteRange{begin_offset, begin_offset + p->size()});
   }
   annotation.addSentence(sentence);
 };
diff --git a/src/translator/sentence_ranges.h b/src/translator/sentence_ranges.h
index b3986e30a..f9c881e17 100644
--- a/src/translator/sentence_ranges.h
+++ b/src/translator/sentence_ranges.h
@@ -64,7 +64,6 @@ class Annotation {
     sentenceEndIds_.push_back(0);
   }
 
-  /// Returns the number of sentences annotated in a text.
   size_t numSentences() const { return sentenceEndIds_.size() - 1; }
 
   /// Returns number of words in the sentence identified by `sentenceIdx`.
@@ -125,10 +124,6 @@ struct AnnotatedText {
   /// constructor is disallowed).
   AnnotatedText(std::string &&text) : text(std::move(text)){};
 
-  AnnotatedText(AnnotatedText &&annotatedBlob)
-      : text(std::move(annotatedBlob.text)),
-        annotation(std::move(annotatedBlob.annotation)) {}
-
   /// Returns the number of sentences in the annotation structure.
   const size_t numSentences() const { return annotation.numSentences(); }
 
@@ -137,6 +132,11 @@ struct AnnotatedText {
     return annotation.numWords(sentenceIdx);
   }
 
+  /// Appends a sentence to the existing text and transparently rebases
+  /// string_views
+  void appendSentence(std::string prefix, std::string &reference,
+                      std::vector<string_view> &wordRanges);
+
   /// Adds a sentence, used to load from SentencePiece annotations conveniently.
   void addSentence(std::vector<string_view> &wordRanges);
 
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 76bcba296..f676797ac 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -112,6 +112,44 @@ void Service::async_translate() {
 #endif // WASM_COMPATIBLE_SOURCE
 
 std::future<Response> Service::translate(std::string &&input) {
+  ResponseOptions responseOptions;  // Hardcode responseOptions for now
+  return translate(std::move(input), responseOptions);
+}
+
+std::vector<Response>
+Service::translateMultiple(std::vector<std::string> &&inputs,
+                           TranslationRequest translationRequest) {
+  ResponseOptions responseOptions;
+
+  // TODO(jerinphilip) Set options based on TranslationRequest, if and when it
+  // becomes non-dummy.
+
+  // We queue the individual Requests so they get compiled at batches to be
+  // efficiently translated.
+  std::vector<std::future<Response>> responseFutures;
+  for (auto &input : inputs) {
+    std::future<Response> inputResponse =
+        queueRequest(std::move(input), responseOptions);
+    responseFutures.push_back(std::move(inputResponse));
+  }
+
+  // Dispatch is called once per request so compilation of sentences from
+  // multiple Requests happen.
+  dispatchTranslate();
+
+  // Now wait for all Requests to complete, the future to fire and return the
+  // compiled Responses, we can probably return the future, but WASM quirks(?).
+  std::vector<Response> responses;
+  for (auto &future : responseFutures) {
+    future.wait();
+    responses.push_back(std::move(future.get()));
+  }
+
+  return responses;
+}
+
+std::future<Response> Service::queueRequest(std::string &&input,
+                                            ResponseOptions responseOptions) {
   Segments segments;
   AnnotatedText source(std::move(input));
   text_processor_.process(source, segments);
@@ -119,17 +157,29 @@ std::future<Response> Service::translate(std::string &&input) {
   std::promise<Response> responsePromise;
   auto future = responsePromise.get_future();
 
-  Ptr<Request> request = New<Request>(
-      requestId_++, /* lineNumberBegin = */ 0, vocabs_, std::move(source),
-      std::move(segments), std::move(responsePromise));
+  ResponseBuilder responseBuilder(responseOptions, std::move(source), vocabs_,
+                                  std::move(responsePromise));
+  Ptr<Request> request = New<Request>(requestId_++, std::move(segments),
+                                      std::move(responseBuilder));
 
   batcher_.addWholeRequest(request);
+  return future;
+}
+
+std::future<Response> Service::translate(std::string &&input,
+                                         ResponseOptions responseOptions) {
+  std::future<Response> future =
+      queueRequest(std::move(input), responseOptions);
+  dispatchTranslate();
+  return future;
+}
+
+void Service::dispatchTranslate() {
   if (numWorkers_ == 0) {
     blocking_translate();
   } else {
     async_translate();
   }
-  return future;
 }
 
 Service::~Service() {
diff --git a/src/translator/service.h b/src/translator/service.h
index 72f6d92ae..476be287b 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -1,10 +1,12 @@
 #ifndef SRC_BERGAMOT_SERVICE_H_
 #define SRC_BERGAMOT_SERVICE_H_
 
+#include "TranslationRequest.h"
 #include "batch_translator.h"
 #include "batcher.h"
 #include "data/types.h"
 #include "response.h"
+#include "response_builder.h"
 #include "text_processor.h"
 #include "translator/parser.h"
 
@@ -18,18 +20,33 @@
 namespace marian {
 namespace bergamot {
 
-/// Service exposes methods to translate an incoming blob of text to the
-/// Consumer of bergamot API.
+/// Service offers methods create an asynchronous translation service. This is
+/// intended to be similar to the ones provided for training or decoding in ML
+/// pipelines with the following additional capabilities:
+///
+///  1. Provision of a request -> response based translation flow unlike the
+///  usual a line based translation or decoding provided in most ML frameworks.
+///  2. Internal handling of normalization etc which changes source text to
+///  provide to client translation meta-information like alignments consistent
+///  with the unnormalized input text.
+///
+/// Service exposes methods to instantiate the service from a string
+/// configuration (which can cover most translators) and to translate an
+/// incoming blob of text.
 ///
-/// An example use of this API looks as follows:
 ///
+/// An example use of this API looks as follows:
+/// ```cpp
 ///  options = ...;
 ///  service = Service(options);
 ///  std::string input_text = "Hello World";
 ///  std::future<Response>
-///      response = service.translate(std::move(input_text));
-///  response.wait();
-///  Response result = response.get();
+///      responseFuture = service.translate(std::move(input_text));
+///  responseFuture.wait(); // Wait until translation has completed.
+///  Response response(std::move(response.get());
+///
+/// // Do things with response.
+/// ```
 ///
 /// Optionally Service can be initialized by also passing model_memory for
 /// purposes of efficiency (which defaults to nullpointer and then reads from
@@ -41,9 +58,22 @@ class Service {
   /// @param modelMemory byte array (aligned to 256!!!) that contains the bytes
   /// of a model.bin. Optional, defaults to nullptr when not used
   /// @param shortlistMemory byte array of shortlist (aligned to 64)
-  explicit Service(Ptr<Options> options, AlignedMemory modelMemory, AlignedMemory shortlistMemory);
+  explicit Service(Ptr<Options> options, AlignedMemory modelMemory,
+                   AlignedMemory shortlistMemory);
 
-  explicit Service(Ptr<Options> options) : Service(options, AlignedMemory(), AlignedMemory()){}
+  /// Construct Service purely from Options. This expects options which
+  /// marian-decoder expects to be set for loading model shortlist and
+  /// vocabularies from files in addition to parameters that set unset desired
+  /// features (e.g: alignments, quality-scores).
+  ///
+  /// This is equivalent to a call to:
+  /// ```cpp
+  ///    Service(options, AlignedMemory(),  AlignedMemory())
+  /// ```
+  /// wherein empty memory is passed and internal flow defaults to file-based
+  /// model, shortlist loading.
+  explicit Service(Ptr<Options> options)
+      : Service(options, AlignedMemory(), AlignedMemory()) {}
 
   /// Construct Service from a string configuration.
   /// @param [in] config string parsable as YAML expected to adhere with marian
@@ -52,20 +82,55 @@ class Service {
   /// bytes of a model.bin. Optional.
   /// @param [in] shortlistMemory byte array of shortlist (aligned to 64)
   explicit Service(const std::string &config,
-                   AlignedMemory modelMemory = AlignedMemory(), AlignedMemory shortlistMemory = AlignedMemory())
-      : Service(parseOptions(config), std::move(modelMemory), std::move(shortlistMemory)) {}
+                   AlignedMemory modelMemory = AlignedMemory(),
+                   AlignedMemory shortlistMemory = AlignedMemory())
+      : Service(parseOptions(config), std::move(modelMemory),
+                std::move(shortlistMemory)) {}
 
   /// Explicit destructor to clean up after any threads initialized in
   /// asynchronous operation mode.
   ~Service();
 
   /// To stay efficient and to refer to the string for alignments, expects
-  /// ownership be moved through std::move(..)
+  /// ownership be moved through `std::move(..)`
+  ///
+  ///  @param [in] source: rvalue reference of string to be translated.
+  std::future<Response> translate(std::string &&source);
+
+  /// Translate an input, providing Options to construct Response. This is
+  /// useful when one has to set/unset alignments or quality in the Response to
+  /// save compute spent in constructing these objects.
   ///
-  ///  @param [in] rvalue reference of string to be translated.
-  std::future<Response> translate(std::string &&input);
+  /// @param [in] source: rvalue reference of the string to be translated
+  /// @param [in] responseOptions: Options indicating whether or not to include
+  /// some member in the Response, also specify any additional configurable
+  /// parameters.
+  std::future<Response> translate(std::string &&source,
+                                  ResponseOptions options);
+
+  /// Translate an input, providing TranslationRequest across all texts to
+  /// construct Response. Provides the browser with the ability to break texts
+  /// into multiple Request keeping gains from efficiently batching internally.
+  /// Also useful when one has to set/unset alignments or quality in the
+  /// Response to save compute spent in constructing these objects.
+
+  /// @param [in] source: rvalue reference of the string to be translated
+  /// @param [in] translationRequest: TranslationRequest (Unified API)
+  /// indicating whether or not to include some member in the Response, also
+  /// specify any additional configurable parameters.
+
+  std::vector<Response>
+  translateMultiple(std::vector<std::string> &&source,
+                    TranslationRequest translationRequest);
 
 private:
+  /// Queue an input for translation.
+  std::future<Response> queueRequest(std::string &&input,
+                                     ResponseOptions responseOptions);
+
+  /// Dispatch call to translate after inserting in queue
+  void dispatchTranslate();
+
   /// Build numTranslators number of translators with options from options
   void build_translators(Ptr<Options> options, size_t numTranslators);
   /// Initializes a blocking translator without using std::thread
@@ -83,16 +148,17 @@ class Service {
   void async_translate();
 
   /// Number of workers to launch.
-  size_t numWorkers_;              // ORDER DEPENDENCY (pcqueue_)
+  size_t numWorkers_; // ORDER DEPENDENCY (pcqueue_)
   /// Model memory to load model passed as bytes.
-  AlignedMemory modelMemory_;      // ORDER DEPENDENCY (translators_)
+  AlignedMemory modelMemory_; // ORDER DEPENDENCY (translators_)
   /// Shortlist memory passed as bytes.
-  AlignedMemory shortlistMemory_;  // ORDER DEPENDENCY (translators_)
+  AlignedMemory shortlistMemory_; // ORDER DEPENDENCY (translators_)
 
   /// Holds instances of batch translators, just one in case
   /// of single-threaded application, numWorkers_ in case of multithreaded
   /// setting.
-  std::vector<BatchTranslator> translators_;  // ORDER DEPENDENCY (modelMemory_, shortlistMemory_)
+  std::vector<BatchTranslator>
+      translators_; // ORDER DEPENDENCY (modelMemory_, shortlistMemory_)
 
   /// Stores requestId of active request. Used to establish
   /// ordering among requests and logging/book-keeping.

From 4be96a97d727b500b3ff4cc3f40a3434a3d5afd9 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 27 Apr 2021 15:04:23 +0000
Subject: [PATCH 202/442] Handle empty translation requests

Fixes https://github.com/browsermt/bergamot-translator/issues/101.
ResponseBuilder is called with empty histories to trigger a valid but
mostly-empty response.
---
 src/translator/request.cpp | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 7e9b73977..8e46533fd 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -20,6 +20,13 @@ Request::Request(size_t Id, Segments &&segments,
 
   counter_ = segments_.size();
   histories_.resize(segments_.size(), nullptr);
+
+  // If there are no segments_, we are never able to trigger the responseBuilder
+  // calls from a different thread. However, in this case we want an empty valid
+  // response.
+  if (segments_.size() == 0) {
+    responseBuilder_(std::move(histories_));
+  }
 }
 
 size_t Request::numSegments() const { return segments_.size(); }

From e5ec5bdd330137febd28369447073eaf00b374d9 Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 29 Apr 2021 10:38:09 +0200
Subject: [PATCH 203/442] Control validating the config options via a boolean
 flag (#116)

* Control validating the config options via a boolean flag

 - parseOptions() function now validates the parsed options
   based on the validate argument

* Minor syntactic fix
---
 src/translator/parser.h  | 9 ++++++---
 src/translator/service.h | 2 +-
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/src/translator/parser.h b/src/translator/parser.h
index fa4e7bbc8..207890c84 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -31,7 +31,7 @@ inline marian::ConfigParser createConfigParser() {
 }
 
 inline std::shared_ptr<marian::Options>
-parseOptions(const std::string &config) {
+parseOptions(const std::string &config, bool validate = true) {
   marian::Options options;
 
   // @TODO(jerinphilip) There's something off here, @XapaJIaMnu suggests
@@ -58,8 +58,11 @@ parseOptions(const std::string &config) {
   options.parse(config);
   YAML::Node configCopy = options.cloneToYamlNode();
 
-  marian::ConfigValidator validator(configCopy);
-  validator.validateOptions(marian::cli::mode::translation);
+  if (validate) {
+    // Perform validation on parsed options only when requested
+    marian::ConfigValidator validator(configCopy);
+    validator.validateOptions(marian::cli::mode::translation);
+  }
 
   return std::make_shared<marian::Options>(options);
 }
diff --git a/src/translator/service.h b/src/translator/service.h
index 476be287b..a731653bc 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -84,7 +84,7 @@ class Service {
   explicit Service(const std::string &config,
                    AlignedMemory modelMemory = AlignedMemory(),
                    AlignedMemory shortlistMemory = AlignedMemory())
-      : Service(parseOptions(config), std::move(modelMemory),
+      : Service(parseOptions(config, /*validate=*/false), std::move(modelMemory),
                 std::move(shortlistMemory)) {}
 
   /// Explicit destructor to clean up after any threads initialized in

From de0abfd795f0160f948f9b84ad550a9485561814 Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 29 Apr 2021 12:04:04 +0200
Subject: [PATCH 204/442] JS bindings for loading model and shortlist files as
 bytes (#117)

* Bindings to load model and shortlist files as bytes
* Modified wasm test page for byte based loading of files
* Updates wasm README for byte loading based usage of TranslationModel
---
 wasm/README.md                             | 19 ++++-
 wasm/bindings/TranslationModelBindings.cpp | 21 ++++-
 wasm/test_page/bergamot.html               | 96 ++++++++++++++++------
 3 files changed, 105 insertions(+), 31 deletions(-)

diff --git a/wasm/README.md b/wasm/README.md
index 23564b971..e9ef13241 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -1,9 +1,19 @@
 ## Using Bergamot Translator in JavaScript
 The example file `bergamot.html` in the folder `test_page` demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
 
-Please note that everything below assumes that the [bergamot project specific model files](https://github.com/mozilla-applied-ml/bergamot-models) were packaged in wasm binary (using the compile instructions given in the top level README).
+### <a name="Pre-requisite"></a> Pre-requisite: Download files required for translation
 
-### Using JS APIs
+Please note that [Using JS APIs](#Using-JS-APIs) and [Demo](#Demo) section below assumes that the [bergamot project specific model files](https://github.com/mozilla-applied-ml/bergamot-models) are already downloaded and present in the `test_page` folder. If this is not done then use following instructions to do so:
+
+```bash
+cd test_page
+mkdir models
+git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
+cp -rf bergamot-models/prod/* models
+gunzip models/*/*
+```
+
+### <a name="Using-JS-APIs"></a> Using JS APIs
 
 ```js
 // The model configuration as YAML formatted string. For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
@@ -34,13 +44,16 @@ request.delete();
 input.delete();
 ```
 
-### Demo (see everything in action)
+### <a name="Demo"></a> Demo (see everything in action)
+
+* Make sure that you followed [Pre-requisite](#Pre-requisite) instructions before moving forward.
 
 * Start the test webserver (ensure you have the latest nodejs installed)
     ```bash
     cd test_page
     bash start_server.sh
     ```
+
 * Open any of the browsers below
     * Firefox Nightly +87: make sure the following prefs are on (about:config)
         ```
diff --git a/wasm/bindings/TranslationModelBindings.cpp b/wasm/bindings/TranslationModelBindings.cpp
index 245416c6a..cb9cf4bf8 100644
--- a/wasm/bindings/TranslationModelBindings.cpp
+++ b/wasm/bindings/TranslationModelBindings.cpp
@@ -10,10 +10,27 @@
 
 using namespace emscripten;
 
-// Binding code
+val getByteArrayView(marian::bergamot::AlignedMemory& alignedMemory) {
+  return val(typed_memory_view(alignedMemory.size(), alignedMemory.as<char>()));
+}
+
+EMSCRIPTEN_BINDINGS(aligned_memory) {
+  class_<marian::bergamot::AlignedMemory>("AlignedMemory")
+    .constructor<std::size_t, std::size_t>()
+    .function("size", &marian::bergamot::AlignedMemory::size)
+	  .function("getByteArrayView", &getByteArrayView)
+    ;
+}
+
+TranslationModel* TranslationModelFactory(const std::string &config,
+                                          marian::bergamot::AlignedMemory* modelMemory,
+                                          marian::bergamot::AlignedMemory* shortlistMemory) {
+  return new TranslationModel(config, std::move(*modelMemory), std::move(*shortlistMemory));
+}
+
 EMSCRIPTEN_BINDINGS(translation_model) {
   class_<TranslationModel>("TranslationModel")
-    .constructor<std::string>()
+    .constructor(&TranslationModelFactory, allow_raw_pointers())
     .function("translate", &TranslationModel::translate)
 	  .function("isAlignmentSupported", &TranslationModel::isAlignmentSupported)
     ;
diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 4f1f2a0f7..86a40bb81 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -2,7 +2,7 @@
 <html>
 <head>
     <link rel="icon" href="data:,">
-    <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1">
+    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
 </head>
 <style>
     body, html, div {
@@ -61,9 +61,27 @@
 </div>
 
 <script>
+  // This function downloads file from a url and returns the array buffer
+  const downloadAsArrayBuffer = async(url) => {
+    const response = await fetch(url);
+    if (!response.ok) {
+      throw Error(`HTTP ${response.status} - ${response.statusText}`);
+    }
+    return response.arrayBuffer();
+  }
+
+  // This function constructs the AlignedMemory from the array buffer and the alignment size
+  function constructAlignedMemoryFromBuffer(buffer, alignmentSize) {
+    var byteArray = new Int8Array(buffer);
+    console.debug("byteArray size: ", byteArray.byteLength);
+    var alignedMemory = new Module.AlignedMemory(byteArray.byteLength, alignmentSize);
+    const alignedByteArrayView = alignedMemory.getByteArrayView();
+    alignedByteArrayView.set(byteArray);
+    return alignedMemory;
+  }
 
-  var model, request, input = undefined;
-  const loadModel = (from, to) => {
+  var translationModel, request, input = undefined;
+  const constructTranslationModel = async (from, to) => {
 
     const languagePair = `${from}${to}`;
 
@@ -72,7 +90,7 @@
 
     // Set the Model Configuration as YAML formatted string.
     // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-    const modelConfig = `models:
+    /*const modelConfig = `models:
   - /${languagePair}/model.${languagePair}.intgemm.alphas.bin
 vocabs:
   - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
@@ -93,22 +111,53 @@
     - 50
     - 50
 `;
-/*
-This config is not valid anymore in new APIs
-mini-batch: 32
-maxi-batch: 100
-maxi-batch-sort: src
 */
+
+const modelConfigWithoutModelAndShortList = `vocabs:
+  - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
+  - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
+beam-size: 1
+normalize: 1.0
+word-penalty: 0
+max-length-break: 128
+mini-batch-words: 1024
+workspace: 128
+max-length-factor: 2.0
+skip-cost: true
+cpu-threads: 0
+quiet: true
+quiet-translation: true
+`;
+
 // TODO: Use in model config when wormhole is enabled:
 // gemm-precision: int8shift
 // TODO: Use in model config when loading of binary models is supported and we use model.intgemm.alphas.bin:
 // gemm-precision: int8shiftAlphaAll
 
-    console.debug("modelConfig: ", modelConfig);
-
-    // Instantiate the TranslationModel
-    if (model) model.delete();
-    model = new Module.TranslationModel(modelConfig);
+    const modelFile = `${languagePair}/model.${languagePair}.intgemm.alphas.bin`;
+    console.debug("modelFile: ", modelFile);
+    const shortlistFile = `${languagePair}/lex.${languagePair}.s2t.bin`;
+    console.debug("shortlistFile: ", shortlistFile);
+
+    try {
+      // Download the files as buffers from the given urls
+      let start = Date.now();
+      const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]);
+      const modelBuffer = downloadedBuffers[0];
+      const shortListBuffer = downloadedBuffers[1];
+      log(`${languagePair} file download took ${(Date.now() - start) / 1000} secs`);
+
+      // Construct AlignedMemory objects with downloaded buffers
+      var alignedModelMemory = constructAlignedMemoryFromBuffer(modelBuffer, 256);
+      var alignedShortlistMemory = constructAlignedMemoryFromBuffer(shortListBuffer, 64);
+
+      // Instantiate the TranslationModel
+      if (translationModel) translationModel.delete();
+      console.debug("Creating TranslationModel with config:", modelConfigWithoutModelAndShortList);
+      translationModel = new Module.TranslationModel(modelConfigWithoutModelAndShortList, alignedModelMemory, alignedShortlistMemory);
+    } catch (error) {
+      console.error(error);
+    }
   }
 
   const translate = (paragraphs) => {
@@ -127,16 +176,9 @@
     })
     // Access input (just for debugging)
     console.log('Input size=', input.size());
-    /*
-    for (let i = 0; i < input.size(); i++) {
-      console.log(' val:' + input.get(i));
-    }
-    */
 
     // Translate the input; the result is a vector<TranslationResult>
-    let result = model.translate(input, request);
-    // Access original and translated text from each entry of vector<TranslationResult>
-    //console.log('Result size=', result.size(), ' - TimeDiff - ', (Date.now() - start)/1000);
+    let result = translationModel.translate(input, request);
     const translatedParagraphs = [];
     for (let i = 0; i < result.size(); i++) {
       translatedParagraphs.push(result.get(i).getTranslatedText());
@@ -147,14 +189,16 @@
     return translatedParagraphs;
   }
 
-  document.querySelector("#load").addEventListener("click", () => {
+  document.querySelector("#load").addEventListener("click", async() => {
+    document.querySelector("#load").disabled = true;
     const lang = document.querySelector('input[name="modellang"]:checked').value;
     const from = lang.substring(0, 2);
     const to = lang.substring(2, 4);
     let start = Date.now();
-    loadModel(from, to)
-    log(`model ${from}${to} loaded in ${(Date.now() - start) / 1000} secs`);
-    //log('Model Alignment:', model.isAlignmentSupported());
+    await constructTranslationModel(from, to);
+    log(`translation model ${from}${to} construction took ${(Date.now() - start) / 1000} secs`);
+    document.querySelector("#load").disabled = false;
+    //log('Model Alignment:', translationModel.isAlignmentSupported());
   });
 
   const translateCall = () => {

From 3525af6a45371850efd379a04e07e61991654fe8 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 29 Apr 2021 16:20:11 +0200
Subject: [PATCH 205/442] Make wasm test page work with bergamot-models
 repository

 - bergamot-models now contains lexical shortlist bin files as well
---
 wasm/test_page/bergamot.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 86a40bb81..d52b8e478 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -134,9 +134,9 @@
 // TODO: Use in model config when loading of binary models is supported and we use model.intgemm.alphas.bin:
 // gemm-precision: int8shiftAlphaAll
 
-    const modelFile = `${languagePair}/model.${languagePair}.intgemm.alphas.bin`;
+    const modelFile = `models/${languagePair}/model.${languagePair}.intgemm.alphas.bin`;
     console.debug("modelFile: ", modelFile);
-    const shortlistFile = `${languagePair}/lex.${languagePair}.s2t.bin`;
+    const shortlistFile = `models/${languagePair}/lex.50.50.${languagePair}.s2t.bin`;
     console.debug("shortlistFile: ", shortlistFile);
 
     try {

From 2788116f8b10f7edd2168ae7fb0be0f059681c8b Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 29 Apr 2021 16:51:13 +0200
Subject: [PATCH 206/442] Better error logging for wasm test page

---
 wasm/test_page/bergamot.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index d52b8e478..717a09ee4 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -65,7 +65,7 @@
   const downloadAsArrayBuffer = async(url) => {
     const response = await fetch(url);
     if (!response.ok) {
-      throw Error(`HTTP ${response.status} - ${response.statusText}`);
+      throw Error(`Downloading ${url} failed: HTTP ${response.status} - ${response.statusText}`);
     }
     return response.arrayBuffer();
   }
@@ -156,7 +156,7 @@
       console.debug("Creating TranslationModel with config:", modelConfigWithoutModelAndShortList);
       translationModel = new Module.TranslationModel(modelConfigWithoutModelAndShortList, alignedModelMemory, alignedShortlistMemory);
     } catch (error) {
-      console.error(error);
+      log(error);
     }
   }
 

From e2865331648e19cfee315844f65b757815be70b8 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Fri, 30 Apr 2021 22:34:44 +0100
Subject: [PATCH 207/442] Update to marian-dev master

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 46a221873..94aeaa461 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 46a22187341ff51b3f11a8cb1edf51c995e583ca
+Subproject commit 94aeaa4616a0fb01ac95a23f0e74a214a94e7609

From d82e01eda4c40b3fe0a0693066e8b3d310c4e82d Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Sat, 1 May 2021 00:29:23 +0100
Subject: [PATCH 208/442] Full windows support with ssplit from browsermt, not
 a fork (#109)

* Update marian-dev to the newest mac version

* Attempt windows workflow

* force workflow rerun

* Separate id

* Attempt 3 at github action

* Marian dev submodule now compiles with apple clang

* Updated ssplit version to something more recent

* Attempt to fix compile on wasm

* Do not compile subproject tests

* Fix emscripten compilation on Mac

* 99% on the way to windows compile

* Try with a different generator

* Build release not debug

* Revert CMakeLists.txt hacks

* Fix sse2 compilation failure

* MSVC settings for WIN32

* Add nodefaultlib LIBCMT

* Do not compile ssplit.cpp as it contains sys/mman.h

* Revert ab56b9aa4f4360b0ab98d5806658d4302f31db1d

* Update paths

* Set the build type to release if not set previously

* Attempt to build release with the windows workflow

* Attempt 5 at VS studio release build

* Attempt 6 at getting release build on MSVC generator

* The windows build is debug at the moment...

* fix ssplit for ubuntu 16.04

* Fix compilation with clang

* Compile on ubuntu16.04

* Explain what is going on

* Updated ssplit and workflow
---
 .github/workflows/windows.yml      | 13 +++++++------
 3rd_party/ssplit-cpp               |  2 +-
 CMakeLists.txt                     | 25 ++++++++++++++++++++++++-
 src/QualityScore.h                 |  1 +
 src/translator/definitions.h       | 17 +++++++++++++++++
 src/translator/sentence_splitter.h |  1 +
 6 files changed, 51 insertions(+), 8 deletions(-)

diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 6c3a05f8b..00e9cfafb 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -37,7 +37,7 @@ jobs:
       shell: powershell
 
     - name: Prepare vcpkg
-      uses: lukka/run-vcpkg@v4
+      uses: lukka/run-vcpkg@v7.3
       with:
         vcpkgArguments: protobuf pcre2
         vcpkgGitCommitId: 6185aa76504a5025f36754324abf307cc776f3da 
@@ -45,22 +45,23 @@ jobs:
         vcpkgTriplet: x64-windows-static
 
     # Windows CPU only minimal build
-    - name: Build Debug
+    - name: Build Release # @TODO this is actually a debug build until the ninja generator gets fixed
       uses: lukka/run-cmake@v3
       with:
-        buildDirectory: ${{ github.workspace }}/build/Debug
+        buildDirectory: ${{ github.workspace }}/build
         cmakeAppendedArgs: '-G Ninja
-          -DCMAKE_BUILD_TYPE="Debug"
+          -DCMAKE_BUILD_TYPE="Release"
           -DUSE_WASM_COMPATIBLE_SOURCE="OFF"
           -DUSE_STATIC_LIBS="TRUE"'
         cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
         cmakeListsTxtPath: ${{ github.workspace }}/CMakeLists.txt
         useVcpkgToolchainFile: true
+        cmakeBuildType: Release
 
 
     - name: Print versions
-      working-directory: build/
+      working-directory: build
       run: |
-        .\app\bergamot-translator-app.exe --version
+        .\app\service-cli.exe --version
         dir *.exe
       shell: cmd
diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index dfefe3421..8d338ed5c 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit dfefe34218fe3aced70266994b6557f029fcbdde
+Subproject commit 8d338ed5c77d22f8c86f60554596fa57bf5091e6
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 412b3863e..3fe03c992 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -9,6 +9,28 @@ project(bergamot_translator CXX C)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
+# Note that with CMake MSVC build, the option CMAKE_BUILD_TYPE is automatically derived from the key
+# 'configurationType' in CMakeSettings.json configurations
+if(NOT CMAKE_BUILD_TYPE)
+  message(WARNING "CMAKE_BUILD_TYPE not set; setting to Release")
+  set(CMAKE_BUILD_TYPE "Release")
+endif()
+#MSVC can't seem to pick up correct flags otherwise:
+if(MSVC)
+  add_definitions(-DUSE_SSE2=1) # Supposed to fix something in the sse_mathfun.h but not sure it does
+  set(INTRINSICS "/arch:AVX2") # ARCH we're targetting on win32. @TODO variable
+  
+  set(CMAKE_CXX_FLAGS           "/EHsc /DWIN32 /D_WINDOWS /DUNICODE /D_UNICODE /D_CRT_NONSTDC_NO_WARNINGS /D_CRT_SECURE_NO_WARNINGS /bigobj")
+  set(CMAKE_CXX_FLAGS_RELEASE   "${CMAKE_CXX_FLAGS} /MT /O2 ${INTRINSICS} /Zi /MP /GL /DNDEBUG")
+  set(CMAKE_CXX_FLAGS_DEBUG     "${CMAKE_CXX_FLAGS} /MTd /Od /Ob0 ${INTRINSICS} /RTC1 /Zi /D_DEBUG")
+
+  # ignores warning LNK4049: locally defined symbol free imported - this comes from zlib
+  set(CMAKE_EXE_LINKER_FLAGS         "${CMAKE_EXE_LINKER_FLAGS} /DEBUG /LTCG:incremental /INCREMENTAL:NO /ignore:4049")
+  set(CMAKE_EXE_LINKER_FLAGS_RELEASE "${CMAKE_EXE_LINKER_FLAGS} /NODEFAULTLIB:MSVCRT")
+  set(CMAKE_EXE_LINKER_FLAGS_DEBUG   "${CMAKE_EXE_LINKER_FLAGS} /NODEFAULTLIB:MSVCRTD")
+  set(CMAKE_STATIC_LINKER_FLAGS      "${CMAKE_STATIC_LINKER_FLAGS} /LTCG:incremental")
+endif(MSVC)
+
 include(CMakeDependentOption)
 
 # Project specific cmake options
@@ -22,11 +44,12 @@ SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be package
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
 SET(USE_SENTENCEPIECE ON CACHE BOOL "Download and compile SentencePiece")
 SET(USE_STATIC_LIBS ON CACHE BOOL "Link statically against non-system libs")
+SET(SSPLIT_COMPILE_LIBRARY_ONLY ON CACHE BOOL "Do not compile ssplit tests")
 if (USE_WASM_COMPATIBLE_SOURCE)
   SET(COMPILE_LIBRARY_ONLY ON CACHE BOOL "Build only the Marian library and exclude all executables.")
   SET(USE_MKL OFF CACHE BOOL "Compile with MKL support")
   # # Setting the ssplit-cpp submodule specific cmake options for wasm
-  SET(USE_INTERNAL_PCRE2 ON CACHE BOOL "Use internal PCRE2 instead of system PCRE2")
+  SET(SSPLIT_USE_INTERNAL_PCRE2 ON CACHE BOOL "Use internal PCRE2 instead of system PCRE2")
 endif()
 
 # Documentation: https://cliutils.gitlab.io/modern-cmake/chapters/projects/submodule.html
diff --git a/src/QualityScore.h b/src/QualityScore.h
index 3ad6349bd..a6beb4edc 100644
--- a/src/QualityScore.h
+++ b/src/QualityScore.h
@@ -8,6 +8,7 @@
 
 #include <string>
 #include <vector>
+#include "translator/definitions.h"
 
 /* All possible Granularities for which Quality Scores can be returned for
  * translated text. */
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index 73d83203b..18b5fcab0 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -28,4 +28,21 @@ typedef AlignedVector<char> AlignedMemory;
 } // namespace bergamot
 } // namespace marian
 
+// @TODO at the moment the usage of string_view in this repository is a hot mess and a disaster waiting to happen.
+// ssplit uses std::string_view if the compiler supports c++17, else falls back to c++11 and absl::string_view
+// bergamot-translator uses, depending on the source file std::string_view (which will break if ssplit-cpp uses
+// absl::string_view) and marian::string_view which is an export of (confusingly) the sentencepiece module that
+// marian has. marian::string_view is our addition to the marian fork, which will make merging even nicer. Not.
+// This is just an ugly patchwork that allos gcc5, our lowest targetted gcc to run. We don't actually try to run
+// on older compilers.
+
+#if defined(__GNUC__) && __GNUC__ < 6 && !defined(__clang__)
+#include <experimental/string_view>
+namespace std {
+  using string_view = std::experimental::string_view;
+} // namespace std
+#else
+#include <string_view>
+#endif
+
 #endif // SRC_BERGAMOT_DEFINITIONS_H_
diff --git a/src/translator/sentence_splitter.h b/src/translator/sentence_splitter.h
index 5175176bf..1c4742e05 100644
--- a/src/translator/sentence_splitter.h
+++ b/src/translator/sentence_splitter.h
@@ -4,6 +4,7 @@
 #include "common/options.h"
 #include "data/types.h"
 #include "ssplit.h"
+#include "definitions.h"
 #include <string>
 
 namespace marian {

From f3a257d40b247a955738197f46ff3334d616c9cb Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 29 Apr 2021 16:52:05 +0200
Subject: [PATCH 209/442] Enabled gemm-precision in wasm test page

 - This increases the inference speed while providing
   models as bytes to the translation engine
   (it wasn't needed while providing models as files)
---
 wasm/test_page/bergamot.html | 1 +
 1 file changed, 1 insertion(+)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 717a09ee4..6bab6061d 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -127,6 +127,7 @@
 cpu-threads: 0
 quiet: true
 quiet-translation: true
+gemm-precision: int8shift
 `;
 
 // TODO: Use in model config when wormhole is enabled:

From 4908e4019e668d20c61996119a41999f575cf1b9 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Thu, 29 Apr 2021 17:08:23 +0200
Subject: [PATCH 210/442] Updated wasm/README file with instructions for byte
 loading APIs

---
 wasm/README.md | 32 +++++++++++++++++++++++++++++---
 1 file changed, 29 insertions(+), 3 deletions(-)

diff --git a/wasm/README.md b/wasm/README.md
index e9ef13241..952fe0748 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -17,11 +17,37 @@ gunzip models/*/*
 
 ```js
 // The model configuration as YAML formatted string. For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-// This example captures the most relevant options: model file, vocabulary files and shortlist file
-const modelConfig = "{\"models\":[\"/esen/model.esen.npz\"],\"vocabs\":[\"/esen/vocab.esen.spm\",\"/esen/vocab.esen.spm\"],\"shortlist\":[\"/esen/lex.esen.s2t\"],\"beam-size\":1}";
+// This example captures some of the most relevant options
+const modelConfig = `vocabs:
+  - /esen/vocab.esen.spm
+  - /esen/vocab.esen.spm
+beam-size: 1
+normalize: 1.0
+word-penalty: 0
+max-length-break: 128
+mini-batch-words: 1024
+workspace: 128
+max-length-factor: 2.0
+skip-cost: true
+cpu-threads: 0
+quiet: true
+quiet-translation: true
+gemm-precision: int8shift
+`;
+
+// Download model and shortlist files and read them into buffers
+const modelFile = `models/esen/model.esen.intgemm.alphas.bin`;
+const shortlistFile = `models/esen/lex.50.50.esen.s2t.bin`;
+const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]); // Please refer to bergamot.html in test_page folder for this function
+const modelBuffer = downloadedBuffers[0];
+const shortListBuffer = downloadedBuffers[1];
+
+// Construct AlignedMemory instances from the buffers
+var alignedModelMemory = constructAlignedMemoryFromBuffer(modelBuffer, 256); // Please refer to bergamot.html in test_page folder for this function
+var alignedShortlistMemory = constructAlignedMemoryFromBuffer(shortListBuffer, 64); // Please refer to bergamot.html in test_page folder for this function
 
 // Instantiate the TranslationModel
-const model = new Module.TranslationModel(modelConfig);
+const model = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory);
 
 // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
 const request = new Module.TranslationRequest();

From 36b3c7291a5818ffb23e07834724258f6e49d501 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 3 May 2021 13:41:37 +0100
Subject: [PATCH 211/442] WASM Bindings collapse (#87)

* Safe transfer of bindings through typedefs

* Removing Translation* files and bringing in counterparts

* Remove previously commented out code

* Removing commented out include

* Absorb Translation* documentation

Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
---
 app/bergamot-translator-app-bytearray.cpp   |  14 ++-
 app/bergamot-translator-app.cpp             |  16 ++-
 src/TranslationModel.h                      |  80 ---------------
 src/TranslationResult.h                     | 108 --------------------
 src/translator/CMakeLists.txt               |   1 -
 src/translator/TranslationModel.cpp         |  50 ---------
 src/translator/response.h                   |   4 +
 src/translator/service.cpp                  |   4 +-
 src/translator/service.h                    |  47 ++++++---
 wasm/bindings/TranslationModelBindings.cpp  |  10 +-
 wasm/bindings/TranslationResultBindings.cpp |  11 +-
 11 files changed, 68 insertions(+), 277 deletions(-)
 delete mode 100644 src/TranslationModel.h
 delete mode 100644 src/TranslationResult.h
 delete mode 100644 src/translator/TranslationModel.cpp

diff --git a/app/bergamot-translator-app-bytearray.cpp b/app/bergamot-translator-app-bytearray.cpp
index 1fa574839..91353c0a3 100644
--- a/app/bergamot-translator-app-bytearray.cpp
+++ b/app/bergamot-translator-app-bytearray.cpp
@@ -7,9 +7,9 @@
 
 #include <iostream>
 
-#include "TranslationModel.h"
-#include "translator/parser.h"
 #include "translator/byte_array_util.h"
+#include "translator/parser.h"
+#include "translator/service.h"
 
 int main(int argc, char **argv) {
 
@@ -20,19 +20,17 @@ int main(int argc, char **argv) {
   std::string config = options->asYamlString();
 
   // Route the config string to construct marian model through TranslationModel
-  TranslationModel model(config, marian::bergamot::getModelMemoryFromConfig(options));
+  marian::bergamot::Service model(
+      config, marian::bergamot::getModelMemoryFromConfig(options));
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
 
   for (std::string line; std::getline(std::cin, line);) {
-        texts.emplace_back(line);
+    texts.emplace_back(line);
   }
 
-  auto results = model.translate(std::move(texts), translationRequest);
-
-  // Resolve the future and get the actual result
-  //std::vector<TranslationResult> results = futureResults.get();
+  auto results = model.translateMultiple(std::move(texts), translationRequest);
 
   for (auto &result : results) {
     std::cout << result.getTranslatedText() << std::endl;
diff --git a/app/bergamot-translator-app.cpp b/app/bergamot-translator-app.cpp
index 4fba00b46..c48796990 100644
--- a/app/bergamot-translator-app.cpp
+++ b/app/bergamot-translator-app.cpp
@@ -1,16 +1,17 @@
 /*
  * main.cpp
  *
- * An application which accepts line separated texts in stdin and returns translated ones in stdout.
- * It is convenient for batch processing and can be used with tools like SacreBLEU.
+ * An application which accepts line separated texts in stdin and returns
+ * translated ones in stdout. It is convenient for batch processing and can be
+ * used with tools like SacreBLEU.
  *
  */
 
 #include <iostream>
 #include <string>
 
-#include "TranslationModel.h"
 #include "translator/parser.h"
+#include "translator/service.h"
 
 int main(int argc, char **argv) {
 
@@ -21,19 +22,16 @@ int main(int argc, char **argv) {
   std::string config = options->asYamlString();
 
   // Route the config string to construct marian model through TranslationModel
-  TranslationModel model(config);
+  marian::bergamot::Service model(config);
 
   TranslationRequest translationRequest;
   std::vector<std::string> texts;
 
   for (std::string line; std::getline(std::cin, line);) {
-        texts.emplace_back(line);
+    texts.emplace_back(line);
   }
 
-  auto results = model.translate(std::move(texts), translationRequest);
-
-  // Resolve the future and get the actual result
-  //std::vector<TranslationResult> results = futureResults.get();
+  auto results = model.translateMultiple(std::move(texts), translationRequest);
 
   for (auto &result : results) {
     std::cout << result.getTranslatedText() << std::endl;
diff --git a/src/TranslationModel.h b/src/TranslationModel.h
deleted file mode 100644
index 4b1be2348..000000000
--- a/src/TranslationModel.h
+++ /dev/null
@@ -1,80 +0,0 @@
-/*
- * TranslationModel.h
- *
- * Main interface for translation API.
- */
-
-#ifndef SRC_TRANSLATOR_TRANSLATIONMODEL_H_
-#define SRC_TRANSLATOR_TRANSLATIONMODEL_H_
-
-#include <future>
-#include <string>
-#include <vector>
-
-// All 3rd party includes
-#include "3rd_party/marian-dev/src/common/options.h"
-
-// All local project includes
-#include "TranslationRequest.h"
-#include "TranslationResult.h"
-#include "translator/definitions.h"
-#include "translator/service.h"
-
-/* A Translation model that translates a plain (without any markups and emojis)
- * UTF-8 encoded text. This implementation supports translation from 1 source
- * language to 1 target language.
- */
-class TranslationModel {
-public:
-  /* Construct the model using the model configuration options as yaml-formatted
-   * string
-   */
-  /**
-   * @param config Marian yml config file in the form of a string
-   * @param model_memory optional byte array (aligned to 64!!!) that contains
-   * the bytes of a model.bin.
-   */
-  TranslationModel(const std::string &config,
-                   marian::bergamot::AlignedMemory modelMemory = marian::bergamot::AlignedMemory(),
-                   marian::bergamot::AlignedMemory shortlistMemory = marian::bergamot::AlignedMemory());
-
-  ~TranslationModel();
-
-  /* This method performs translation on a list of UTF-8 encoded plain text
-   * (without any markups or emojis) and returns a list of results in the same
-   * order. The model supports translation from 1 source language to 1 target
-   * language.
-   *
-   * Each text entry can either be a word, a phrase, a sentence or a list of
-   * sentences. Additional information related to the translated text can be
-   * requested via TranslationRequest which is applied equally to each text
-   * entry. The translated text corresponding to each text entry and the
-   * additional information (as specified in the TranslationRequest) is
-   * encapsulated and returned in TranslationResult.
-   *
-   * The API splits each text entry into sentences internally, which are then
-   * translated independent of each other. The translated sentences are then
-   * joined back together and returned in TranslationResult.
-   *
-   * Please refer to the TranslationRequest class to find out what additional
-   * information can be requested. The alignment information can only be
-   * requested if the model supports it (check isAlignmentSupported() API).
-   *
-   * The texts argument will become empty after the execution of this API (each
-   * entry of texts list will be moved to its corresponding TranslationResult
-   * object).
-   */
-  std::vector<TranslationResult> translate(std::vector<std::string> &&texts,
-                                           TranslationRequest request);
-
-  /* Check if the model can provide alignment information b/w original and
-   * translated text. */
-  bool isAlignmentSupported() const;
-
-private:
-  // Model configuration options
-  std::shared_ptr<marian::Options> configOptions_; // ORDER DEPENDECNY
-  marian::bergamot::Service service_;              // ORDER DEPENDENCY
-};
-
-#endif /* SRC_TRANSLATOR_TRANSLATIONMODEL_H_ */
diff --git a/src/TranslationResult.h b/src/TranslationResult.h
deleted file mode 100644
index 8c6c806e4..000000000
--- a/src/TranslationResult.h
+++ /dev/null
@@ -1,108 +0,0 @@
-/*
- * TranslationResult.h
- *
- * The class that represents the result of TranslationModel::translate()
- * API for each of its text entry and TranslationRequest.
- */
-
-#ifndef SRC_TRANSLATOR_TRANSLATIONRESULT_H_
-#define SRC_TRANSLATOR_TRANSLATIONRESULT_H_
-
-#include <string>
-#include <vector>
-
-#include "QualityScore.h"
-
-/* This class represents the result of TranslationModel::translate() API
- * for each of its text entry and TranslationRequest.
- */
-class TranslationResult {
-public:
-  typedef std::vector<std::pair<std::string_view, std::string_view>>
-      SentenceMappings;
-#ifdef WASM_BINDINGS
-  TranslationResult(const std::string &original, const std::string &translation)
-      : originalText(original), translatedText(translation),
-        sentenceMappings() {}
-#endif
-  TranslationResult(const std::string &original, const std::string &translation,
-                    SentenceMappings &sentenceMappings)
-      : originalText(original), translatedText(translation),
-        sentenceMappings(sentenceMappings) {}
-
-  TranslationResult(TranslationResult &&other)
-      : originalText(std::move(other.originalText)),
-        translatedText(std::move(other.translatedText)),
-        sentenceMappings(std::move(other.sentenceMappings)) {}
-
-#ifdef WASM_BINDINGS
-  TranslationResult(const TranslationResult &other)
-      : originalText(other.originalText),
-        translatedText(other.translatedText),
-        sentenceMappings(other.sentenceMappings) {}
-#endif
-
-  TranslationResult(std::string &&original, std::string &&translation,
-                    SentenceMappings &&sentenceMappings)
-      : originalText(std::move(original)),
-        translatedText(std::move(translation)),
-        sentenceMappings(std::move(sentenceMappings)) {}
-
-#ifndef WASM_BINDINGS
-  TranslationResult &operator=(const TranslationResult &) = delete;
-#else
-  TranslationResult &operator=(const TranslationResult &result) {
-    originalText = result.originalText;
-    translatedText = result.translatedText;
-    sentenceMappings = result.sentenceMappings;
-    return *this;
-  }
-#endif
-
-  /* Return the original text. */
-  const std::string &getOriginalText() const { return originalText; }
-
-  /* Return the translated text. */
-  const std::string &getTranslatedText() const { return translatedText; }
-
-  /* Return the Quality scores of the translated text. */
-  const QualityScore &getQualityScore() const { return qualityScore; }
-
-  /* Return the Sentence mappings (information regarding how individual
-   * sentences of originalText map to corresponding translated sentences in
-   * translatedText).
-   */
-  const SentenceMappings &getSentenceMappings() const {
-    return sentenceMappings;
-  }
-
-private:
-  // Original text (in UTF-8 encoded format) that was supposed to be translated
-  std::string originalText;
-
-  // Translation (in UTF-8 encoded format) of the originalText
-  std::string translatedText;
-
-  // Quality score of the translated text at the granularity specified in
-  // TranslationRequest. It is an optional result (it will have no information
-  // if not requested in TranslationRequest)
-  QualityScore qualityScore;
-
-  // Information regarding how individual sentences of originalText map to
-  // corresponding translated sentences in joined translated text
-  // (translatedText) An example of sentence mapping:
-  //     originalText (contains 2 sentences)              = "What is your name?
-  //     My name is Abc." translatedText (contains 2 translated sentences) =
-  //     "Was ist dein Name? Mein Name ist Abc." sentenceMappings = [
-  //         {"What is your name?", "Was ist dein Name?"},  //
-  //         Pair(originalText[0],translatedText[0])
-  //         {"My name is Abc", "Mein Name ist Abc."}       //
-  //         Pair(originalText[1],translatedText[1])
-  //     ]
-  //
-  // It is an optional result (it will be empty if not requested in
-  // TranslationRequest).
-  SentenceMappings sentenceMappings;
-};
-
-#endif /* SRC_TRANSLATOR_TRANSLATIONRESULT_H_ */
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index d7c8e3cfe..25ca91638 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,5 +1,4 @@
 add_library(bergamot-translator STATIC
-    TranslationModel.cpp
     byte_array_util.cpp
     text_processor.cpp
     sentence_splitter.cpp
diff --git a/src/translator/TranslationModel.cpp b/src/translator/TranslationModel.cpp
deleted file mode 100644
index 026a126ea..000000000
--- a/src/translator/TranslationModel.cpp
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * TranslationModel.cpp
- *
- */
-
-#include <future>
-#include <vector>
-
-// All local project includes
-#include "TranslationModel.h"
-#include "translator/parser.h"
-#include "translator/response.h"
-#include "translator/service.h"
-
-TranslationModel::TranslationModel(const std::string &config,
-                                   marian::bergamot::AlignedMemory model_memory,
-                                   marian::bergamot::AlignedMemory lexical_memory)
-    : service_(config, std::move(model_memory), std::move(lexical_memory)) {}
-
-TranslationModel::~TranslationModel() {}
-
-std::vector<TranslationResult>
-TranslationModel::translate(std::vector<std::string> &&texts,
-                            TranslationRequest request) {
-
-  // This code, move into async?
-  std::vector<TranslationResult> translationResults;
-  std::vector<marian::bergamot::Response> responses =
-      service_.translateMultiple(std::move(texts), request);
-  for (auto &response : responses) {
-    TranslationResult::SentenceMappings sentenceMappings;
-    for (size_t idx = 0; idx < response.size(); idx++) {
-      marian::string_view src = response.source.sentence(idx);
-      marian::string_view tgt = response.target.sentence(idx);
-      sentenceMappings.emplace_back(std::string_view(src.data(), src.size()),
-                                    std::string_view(tgt.data(), tgt.size()));
-    }
-
-    // In place construction.
-    translationResults.emplace_back(
-        std::move(response.source.text), // &&response.source_
-        std::move(response.target.text), // &&response.translation_
-        std::move(sentenceMappings)      // &&sentenceMappings
-    );
-  }
-
-  return translationResults;
-}
-
-bool TranslationModel::isAlignmentSupported() const { return false; }
diff --git a/src/translator/response.h b/src/translator/response.h
index 3b1f48da8..0f7ecb592 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -64,6 +64,10 @@ struct Response {
   /// sparse matrix representation with indices corresponding
   /// to (sub-)words accessible through Annotation.
   std::vector<Alignment> alignments;
+
+  const std::string &getOriginalText() const { return source.text; }
+
+  const std::string &getTranslatedText() const { return target.text; }
 };
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index f676797ac..3d19f5e66 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -28,8 +28,8 @@ loadVocabularies(marian::Ptr<marian::Options> options) {
 namespace marian {
 namespace bergamot {
 
-Service::Service(Ptr<Options> options, AlignedMemory modelMemory, AlignedMemory shortlistMemory)
-    : requestId_(0), vocabs_(std::move(loadVocabularies(options))),
+Service::Service(Ptr<Options> options, AlignedMemory modelMemory, AlignedMemory shortlistMemory) 
+    : requestId_(0), options_(options), vocabs_(std::move(loadVocabularies(options))),
       text_processor_(vocabs_, options), batcher_(options),
       numWorkers_(options->get<int>("cpu-threads")),
       modelMemory_(std::move(modelMemory)), shortlistMemory_(std::move(shortlistMemory))
diff --git a/src/translator/service.h b/src/translator/service.h
index a731653bc..288c6495a 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -20,15 +20,22 @@
 namespace marian {
 namespace bergamot {
 
-/// Service offers methods create an asynchronous translation service. This is
-/// intended to be similar to the ones provided for training or decoding in ML
-/// pipelines with the following additional capabilities:
+/// Service offers methods create an asynchronous translation service that
+/// translates a plain (without any markups and emojis)  UTF-8 encoded text.
+/// This implementation supports translation from 1 source language to 1 target
+/// language.
+///
+///  This is intended to be similar to the ones  provided for training or
+///  decoding in ML pipelines with the following  additional capabilities:
 ///
 ///  1. Provision of a request -> response based translation flow unlike the
 ///  usual a line based translation or decoding provided in most ML frameworks.
 ///  2. Internal handling of normalization etc which changes source text to
 ///  provide to client translation meta-information like alignments consistent
 ///  with the unnormalized input text.
+///  3. The API splits each text entry into sentences internally, which are then
+///  translated independent of each other. The translated sentences are then
+///  joined back together and returned in Response.
 ///
 /// Service exposes methods to instantiate the service from a string
 /// configuration (which can cover most translators) and to translate an
@@ -48,9 +55,10 @@ namespace bergamot {
 /// // Do things with response.
 /// ```
 ///
-/// Optionally Service can be initialized by also passing model_memory for
+/// Optionally Service can be initialized by also passing model memory for
 /// purposes of efficiency (which defaults to nullpointer and then reads from
 /// file supplied through config).
+///
 class Service {
 
 public:
@@ -84,8 +92,8 @@ class Service {
   explicit Service(const std::string &config,
                    AlignedMemory modelMemory = AlignedMemory(),
                    AlignedMemory shortlistMemory = AlignedMemory())
-      : Service(parseOptions(config, /*validate=*/false), std::move(modelMemory),
-                std::move(shortlistMemory)) {}
+      : Service(parseOptions(config, /*validate=*/false),
+                std::move(modelMemory), std::move(shortlistMemory)) {}
 
   /// Explicit destructor to clean up after any threads initialized in
   /// asynchronous operation mode.
@@ -108,12 +116,18 @@ class Service {
   std::future<Response> translate(std::string &&source,
                                   ResponseOptions options);
 
-  /// Translate an input, providing TranslationRequest across all texts to
-  /// construct Response. Provides the browser with the ability to break texts
-  /// into multiple Request keeping gains from efficiently batching internally.
-  /// Also useful when one has to set/unset alignments or quality in the
-  /// Response to save compute spent in constructing these objects.
-
+  /// Translate multiple text-blobs in a single *blocking* API call, providing
+  /// TranslationRequest which applies across all text-blobs dictating how to
+  /// construct Response. TranslationRequest can be used to enable/disable
+  /// additional information like quality-scores, alignments etc.
+  ///
+  /// All texts are combined to efficiently construct batches together providing
+  /// speedups compared to calling translate() indepdently on individual
+  /// text-blob. Note that there will be minor differences in output when
+  /// text-blobs are individually translated due to approximations but similar
+  /// quality nonetheless. If you have async/multithread capabilities, it is
+  /// recommended to work with futures and translate() API.
+  ///
   /// @param [in] source: rvalue reference of the string to be translated
   /// @param [in] translationRequest: TranslationRequest (Unified API)
   /// indicating whether or not to include some member in the Response, also
@@ -123,6 +137,11 @@ class Service {
   translateMultiple(std::vector<std::string> &&source,
                     TranslationRequest translationRequest);
 
+  /// Returns if model is alignment capable or not.
+  bool isAlignmentSupported() const {
+    return options_->hasAndNotEmpty("alignment");
+  }
+
 private:
   /// Queue an input for translation.
   std::future<Response> queueRequest(std::string &&input,
@@ -149,6 +168,10 @@ class Service {
 
   /// Number of workers to launch.
   size_t numWorkers_; // ORDER DEPENDENCY (pcqueue_)
+
+  /// Options object holding the options Service was instantiated with.
+  Ptr<Options> options_;
+
   /// Model memory to load model passed as bytes.
   AlignedMemory modelMemory_; // ORDER DEPENDENCY (translators_)
   /// Shortlist memory passed as bytes.
diff --git a/wasm/bindings/TranslationModelBindings.cpp b/wasm/bindings/TranslationModelBindings.cpp
index cb9cf4bf8..41b9c2e61 100644
--- a/wasm/bindings/TranslationModelBindings.cpp
+++ b/wasm/bindings/TranslationModelBindings.cpp
@@ -6,10 +6,14 @@
 
 #include <emscripten/bind.h>
 
-#include "TranslationModel.h"
+#include "response.h"
+#include "service.h"
 
 using namespace emscripten;
 
+typedef marian::bergamot::Service TranslationModel;
+typedef marian::bergamot::Response TranslationResult;
+
 val getByteArrayView(marian::bergamot::AlignedMemory& alignedMemory) {
   return val(typed_memory_view(alignedMemory.size(), alignedMemory.as<char>()));
 }
@@ -31,9 +35,11 @@ TranslationModel* TranslationModelFactory(const std::string &config,
 EMSCRIPTEN_BINDINGS(translation_model) {
   class_<TranslationModel>("TranslationModel")
     .constructor(&TranslationModelFactory, allow_raw_pointers())
-    .function("translate", &TranslationModel::translate)
+    .function("translate", &TranslationModel::translateMultiple)
 	  .function("isAlignmentSupported", &TranslationModel::isAlignmentSupported)
     ;
+  // ^ We redirect Service::translateMultiple to WASMBound::translate instead. Sane API is
+  // translate. If and when async comes, we can be done with this inconsistency.
 
   register_vector<std::string>("VectorString");
   register_vector<TranslationResult>("VectorTranslationResult");
diff --git a/wasm/bindings/TranslationResultBindings.cpp b/wasm/bindings/TranslationResultBindings.cpp
index a3713a130..c1c0ca8ae 100644
--- a/wasm/bindings/TranslationResultBindings.cpp
+++ b/wasm/bindings/TranslationResultBindings.cpp
@@ -6,15 +6,16 @@
 #include <emscripten/bind.h>
 #include <vector>
 
-#include "TranslationResult.h"
+#include "response.h"
+
+typedef marian::bergamot::Response TranslationResult;
 
 using namespace emscripten;
 
 // Binding code
 EMSCRIPTEN_BINDINGS(translation_result) {
   class_<TranslationResult>("TranslationResult")
-    .constructor<std::string, std::string, TranslationResult::SentenceMappings>()
-	  .function("getOriginalText", &TranslationResult::getOriginalText)
-	  .function("getTranslatedText", &TranslationResult::getTranslatedText)
-    ;
+      .constructor<>()
+      .function("getOriginalText", &TranslationResult::getOriginalText)
+      .function("getTranslatedText", &TranslationResult::getTranslatedText);
 }

From 1a4add19da211e31c60c1e4ca6b27f3de5235020 Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 3 May 2021 17:13:43 +0200
Subject: [PATCH 212/442] Improve script to patch wasm artifacts and load
 EN->DE vocabulary in wasm test (#125)

* Improved script that patches wasm artifacts to enable wormhole

 - Made the regex pattern ignore multiple whitespaces b/w words of
   the matching pattern

* Fix for loading EN->DE vocabularies in wasm test page

 - Loading vocabularies for EN->DE was failing because of
   the new structure of bergamot-models
---
 wasm/patch-artifacts-enable-wormhole.sh | 6 +++---
 wasm/test_page/bergamot.html            | 8 ++++----
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/wasm/patch-artifacts-enable-wormhole.sh b/wasm/patch-artifacts-enable-wormhole.sh
index c16ba66b8..d6360567c 100644
--- a/wasm/patch-artifacts-enable-wormhole.sh
+++ b/wasm/patch-artifacts-enable-wormhole.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 
 echo "Patching wasm artifacts to enable wormhole via APIs that compile and instantiate wasm module"
-sed -i.bak 's/var result = WebAssembly.instantiateStreaming(response, info);/var result = WebAssembly.instantiateStreaming(response, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
-sed -i.bak 's/return WebAssembly.instantiate(binary, info);/return WebAssembly.instantiate(binary, info, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
-sed -i.bak 's/var module = new WebAssembly.Module(bytes);/var module = new WebAssembly.Module(bytes, {simdWormhole:true});/g' wasm/bergamot-translator-worker.js
+sed -i.bak 's/WebAssembly.instantiateStreaming[[:space:]]*([[:space:]]*response[[:space:]]*,[[:space:]]*info[[:space:]]*)/WebAssembly.instantiateStreaming(response, info, {simdWormhole:true})/g' wasm/bergamot-translator-worker.js
+sed -i.bak 's/WebAssembly.instantiate[[:space:]]*([[:space:]]*binary[[:space:]]*,[[:space:]]*info[[:space:]]*)/WebAssembly.instantiate(binary, info, {simdWormhole:true})/g' wasm/bergamot-translator-worker.js
+sed -i.bak 's/WebAssembly.Module[[:space:]]*([[:space:]]*bytes[[:space:]]*)/WebAssembly.Module(bytes, {simdWormhole:true})/g' wasm/bergamot-translator-worker.js
 echo "Done"
diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 6bab6061d..95ae325b2 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -93,8 +93,8 @@
     /*const modelConfig = `models:
   - /${languagePair}/model.${languagePair}.intgemm.alphas.bin
 vocabs:
-  - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
-  - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
+  - /${languagePair}/vocab.${vocabLanguagePair}.spm
+  - /${languagePair}/vocab.${vocabLanguagePair}.spm
 beam-size: 1
 normalize: 1.0
 word-penalty: 0
@@ -114,8 +114,8 @@
 */
 
 const modelConfigWithoutModelAndShortList = `vocabs:
-  - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
-  - /${vocabLanguagePair}/vocab.${vocabLanguagePair}.spm
+  - /${languagePair}/vocab.${vocabLanguagePair}.spm
+  - /${languagePair}/vocab.${vocabLanguagePair}.spm
 beam-size: 1
 normalize: 1.0
 word-penalty: 0

From 8de368c16674b378ad30135e3b3e664ff5ecf7a8 Mon Sep 17 00:00:00 2001
From: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
Date: Tue, 4 May 2021 11:18:45 +0200
Subject: [PATCH 213/442] Improved wasm scripts and README (#128)

---
 .github/workflows/wasm-custom_marian-mac.yml  |  5 +--
 .../workflows/wasm-custom_marian-ubuntu.yml   |  5 +--
 README.md                                     | 21 +++++------
 wasm/CMakeLists.txt                           |  3 +-
 wasm/README.md                                |  4 +-
 wasm/patch-artifacts-enable-wormhole.sh       | 37 +++++++++++++++++--
 wasm/test_page/start_server.sh                | 31 +++++++++++++---
 7 files changed, 78 insertions(+), 28 deletions(-)

diff --git a/.github/workflows/wasm-custom_marian-mac.yml b/.github/workflows/wasm-custom_marian-mac.yml
index 87141c74e..c275f3cd8 100644
--- a/.github/workflows/wasm-custom_marian-mac.yml
+++ b/.github/workflows/wasm-custom_marian-mac.yml
@@ -40,9 +40,8 @@ jobs:
       - name: Check artifacts
         working-directory: build-wasm
         run: |
-          export WASM_ARTIFACTS_DIR=wasm
-          ls -all ${WASM_ARTIFACTS_DIR}
-          if ls ${WASM_ARTIFACTS_DIR}/*.wasm &>/dev/null && ls ${WASM_ARTIFACTS_DIR}/*.js &>/dev/null
+          ls -all bergamot*
+          if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
           then
             echo "Artifacts Successfully Generated"
           else
diff --git a/.github/workflows/wasm-custom_marian-ubuntu.yml b/.github/workflows/wasm-custom_marian-ubuntu.yml
index 7dfc83903..44835465a 100644
--- a/.github/workflows/wasm-custom_marian-ubuntu.yml
+++ b/.github/workflows/wasm-custom_marian-ubuntu.yml
@@ -40,9 +40,8 @@ jobs:
       - name: Check artifacts
         working-directory: build-wasm
         run: |
-          export WASM_ARTIFACTS_DIR=wasm
-          ls -all ${WASM_ARTIFACTS_DIR}
-          if ls ${WASM_ARTIFACTS_DIR}/*.wasm &>/dev/null && ls ${WASM_ARTIFACTS_DIR}/*.js &>/dev/null
+          ls -all bergamot*
+          if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
           then
             echo "Artifacts Successfully Generated"
           else
diff --git a/README.md b/README.md
index e70e8cd09..456af70ce 100644
--- a/README.md
+++ b/README.md
@@ -36,19 +36,18 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
     cd bergamot-translator
     ```
 
-3. Download files (only required if you want to package files in wasm binary)
+3. Download files (only required if you want to perform inference using build artifacts)
 
-    This step is only required if you want to package files (e.g. models, vocabularies etc.)
-    into wasm binary. If you don't then just skip this step.
+    It packages the vocabulary files into wasm binary, which is required only if you want to perform inference.
+    The compilation commands will preload these files in Emscripten’s virtual file system.
 
-    The build preloads the files in Emscripten’s virtual file system.
-
-    If you want to package bergamot project specific models, please follow these instructions:
+    If you want to package bergamot project specific files, please follow these instructions:
     ```bash
-    mkdir models
     git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
+    mkdir models
     cp -rf bergamot-models/prod/* models
     gunzip models/*/*
+    find models \( -type f -name "model*" -or -type f -name "lex*" \) -delete
     ```
 
 4. Compile
@@ -59,14 +58,14 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
         ```
 
     2. Compile the artefacts
-        * If you want to package files into wasm binary then execute following commands (Replace `FILES_TO_PACKAGE` with the path of the
-        directory containing the files to be packaged in wasm binary)
+        * If you want to package files into wasm binary then execute following commands (Replace `FILES_TO_PACKAGE` with the
+        directory containing all the files to be packaged)
 
             ```bash
             emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=FILES_TO_PACKAGE ../
             emmake make -j
             ```
-            e.g. If you want to package bergamot project specific models (downloaded using step 3 above) then
+            e.g. If you want to package bergamot project specific files (downloaded using step 3 above) then
             replace `FILES_TO_PACKAGE` with `../models`
 
         * If you don't want to package any file into wasm binary then execute following commands:
@@ -75,7 +74,7 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
             emmake make -j
             ```
 
-        The wasm artifacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
+        The wasm artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
 
     3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
         ```bash
diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index c89e3939d..a785ba6c0 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -23,6 +23,7 @@ endif()
 set_target_properties(bergamot-translator-worker PROPERTIES
                         SUFFIX ".js"
                         LINK_FLAGS ${LINKER_FLAGS}
-                        )
+                        RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}
+                      )
 
 target_link_libraries(bergamot-translator-worker bergamot-translator)
diff --git a/wasm/README.md b/wasm/README.md
index 952fe0748..337ae1b28 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -77,9 +77,11 @@ input.delete();
 * Start the test webserver (ensure you have the latest nodejs installed)
     ```bash
     cd test_page
-    bash start_server.sh
+    bash start_server.sh ../../build-wasm
     ```
 
+    Provide the folder containing the wasm artifacts as the first argument of `start_server.sh` script (`../../build-wasm` in this case).
+
 * Open any of the browsers below
     * Firefox Nightly +87: make sure the following prefs are on (about:config)
         ```
diff --git a/wasm/patch-artifacts-enable-wormhole.sh b/wasm/patch-artifacts-enable-wormhole.sh
index d6360567c..e39988b4e 100644
--- a/wasm/patch-artifacts-enable-wormhole.sh
+++ b/wasm/patch-artifacts-enable-wormhole.sh
@@ -1,7 +1,36 @@
 #!/bin/bash
+usage="Patch wasm artifacts to enable wormhole via APIs that compile and instantiate wasm module.
 
-echo "Patching wasm artifacts to enable wormhole via APIs that compile and instantiate wasm module"
-sed -i.bak 's/WebAssembly.instantiateStreaming[[:space:]]*([[:space:]]*response[[:space:]]*,[[:space:]]*info[[:space:]]*)/WebAssembly.instantiateStreaming(response, info, {simdWormhole:true})/g' wasm/bergamot-translator-worker.js
-sed -i.bak 's/WebAssembly.instantiate[[:space:]]*([[:space:]]*binary[[:space:]]*,[[:space:]]*info[[:space:]]*)/WebAssembly.instantiate(binary, info, {simdWormhole:true})/g' wasm/bergamot-translator-worker.js
-sed -i.bak 's/WebAssembly.Module[[:space:]]*([[:space:]]*bytes[[:space:]]*)/WebAssembly.Module(bytes, {simdWormhole:true})/g' wasm/bergamot-translator-worker.js
+Usage: $(basename "$0") [WASM_ARTIFACTS_FOLDER]
+
+    where:
+    WASM_ARTIFACTS_FOLDER    Folder containing wasm artifacts
+                             (An optional argument, if unspecified the default is: current folder)"
+
+if [ "$#" -gt 1 ]; then
+    echo "Illegal number of parameters passed"
+    echo "$usage"
+    exit
+fi
+
+# Parse wasm artifacts folder if provided via script argument or set it to default
+WASM_ARTIFACTS_FOLDER=$PWD
+if [ "$#" -eq 1 ]; then
+    if [ ! -e "$1" ]; then
+        echo "Error: Folder \""$1"\" doesn't exist"
+        exit
+    fi
+    WASM_ARTIFACTS_FOLDER="$1"
+fi
+
+WASM_ARTIFACTS="$WASM_ARTIFACTS_FOLDER/bergamot-translator-worker.js"
+if [ ! -e "$WASM_ARTIFACTS" ]; then
+    echo "Error: Artifact \"$WASM_ARTIFACTS\" doesn't exist"
+    exit
+fi
+
+echo "Patching \"$WASM_ARTIFACTS\" to enable wormhole via APIs that compile and instantiate wasm module"
+sed -i.bak 's/WebAssembly.instantiateStreaming[[:space:]]*([[:space:]]*response[[:space:]]*,[[:space:]]*info[[:space:]]*)/WebAssembly.instantiateStreaming(response, info, {simdWormhole:true})/g' $WASM_ARTIFACTS
+sed -i.bak 's/WebAssembly.instantiate[[:space:]]*([[:space:]]*binary[[:space:]]*,[[:space:]]*info[[:space:]]*)/WebAssembly.instantiate(binary, info, {simdWormhole:true})/g' $WASM_ARTIFACTS
+sed -i.bak 's/WebAssembly.Module[[:space:]]*([[:space:]]*bytes[[:space:]]*)/WebAssembly.Module(bytes, {simdWormhole:true})/g' $WASM_ARTIFACTS
 echo "Done"
diff --git a/wasm/test_page/start_server.sh b/wasm/test_page/start_server.sh
index b6fc2a6eb..911364665 100644
--- a/wasm/test_page/start_server.sh
+++ b/wasm/test_page/start_server.sh
@@ -1,9 +1,30 @@
 #!/bin/bash
-echo "Start: Copying artifacts in local folder------"
-cp ../../build-wasm/wasm/bergamot-translator-worker.data .
-cp ../../build-wasm/wasm/bergamot-translator-worker.js .
-cp ../../build-wasm/wasm/bergamot-translator-worker.wasm .
-cp ../../build-wasm/wasm/bergamot-translator-worker.worker.js .
+
+usage="Copy wasm artifacts from build directory and start httpserver
+
+Usage: $(basename "$0") [WASM_ARTIFACTS_FOLDER]
+
+    where:
+    WASM_ARTIFACTS_FOLDER    Folder containing pre-built wasm artifacts"
+
+if [ "$#" -ne 1 ]; then
+    echo "Illegal number of parameters passed"
+    echo "$usage"
+    exit
+fi
+
+# Check if WASM_ARTIFACTS_FOLDER is valid or not
+if [ ! -e "$1" ]; then
+    echo "Error: Folder \""$1"\" doesn't exist"
+    exit
+fi
+
+WASM_ARTIFACTS="$1/bergamot-translator-worker.*"
+for i in $WASM_ARTIFACTS; do
+    [ -f "$i" ] || breaks
+    cp $i .
+    echo "Copied \"$i\""
+done
 
 npm install
 echo "Start httpserver"

From d8f7e51792fab8b34418734a5a8e55ab19e66ecb Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 4 May 2021 12:32:41 +0200
Subject: [PATCH 214/442] Minor README change

 - Changed "browsermt" to "mozilla"
---
 README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index 9dd47c434..eaf4abed7 100644
--- a/README.md
+++ b/README.md
@@ -9,7 +9,7 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
 ### Build Natively
 1. Clone the repository using these instructions:
     ```bash
-    git clone https://github.com/browsermt/bergamot-translator
+    git clone https://github.com/mozilla/bergamot-translator
     cd bergamot-translator
     ```
 2. Compile
@@ -34,7 +34,7 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
 
 2. Clone the repository using these instructions:
     ```bash
-    git clone https://github.com/browsermt/bergamot-translator
+    git clone https://github.com/mozilla/bergamot-translator
     cd bergamot-translator
     ```
 

From c478a626a8084f0e7b210a25b4645a5a79b4d83d Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 4 May 2021 11:00:01 +0200
Subject: [PATCH 215/442] Updating ci scripts for the latest upstream changes

 - The upstream browsermt/bergamot-translator builds the wasm artifacts
   in top level build folder now
---
 .circleci/config.yml | 17 +++++++++++++++--
 build-wasm.sh        |  9 +++++----
 2 files changed, 20 insertions(+), 6 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 6818bc231..6465469b1 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -14,6 +14,19 @@ jobs:
           name: Build WASM
           command: bash build-wasm.sh
 
+      - run:
+          name: Check artifacts
+          working_directory: build-wasm
+          command: |
+            ls -all bergamot*
+            if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null && ls bergamot*.data &>/dev/null
+            then
+              echo "Artifacts Successfully Generated"
+            else
+              echo "Failure: Artifacts Not Present"
+              exit 1
+            fi
+
       - store_artifacts:
-          path: "build-wasm/wasm"
-          destination: "build-wasm/wasm"
+          path: "build-wasm"
+          destination: "build-wasm"
diff --git a/build-wasm.sh b/build-wasm.sh
index 5348fe994..fff92a4b2 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -37,7 +37,7 @@ if [ "$EMSDK" == "" ]; then
   source ./emsdk/emsdk_env.sh
 fi
 
-# 3. Download models (only required if you want to package files in wasm binary)
+# 3. Download models (required to perform inference using build artifacts)
 if [ ! -d "bergamot-models" ]; then
   git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
 else
@@ -53,21 +53,22 @@ mkdir -p models
 rm -rf models/*
 cp -rf bergamot-models/prod/* models
 gunzip models/*/*
+find models \( -type f -name "model*" -or -type f -name "lex*" \) -delete
 
 # 4. Compile
-#     1. Create a folder where you want to build all the artefacts (`build-wasm` in this case)
+#     1. Create a folder where you want to build all the artifacts (`build-wasm` in this case)
 if [ ! -d "build-wasm" ]; then
   mkdir build-wasm
 fi
 cd build-wasm
 
-#     2. Compile the artefacts
+#     2. Compile the artifacts
 emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR="../models/" ../
 emmake make -j3
 
 #     3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
 bash ../wasm/patch-artifacts-enable-wormhole.sh
 
-# The artifacts (.js and .wasm files) will be available in `wasm` folder of build directory ("build-wasm" in this case).
+# The artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
 
 exit 0

From a63533b241e19cf316edf1c1627568c278e01419 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Tue, 4 May 2021 15:25:12 +0300
Subject: [PATCH 216/442] Extension desired changes (#129)

* Enable worker file system
* Avoid node.js-code in emscripten glue-code
---
 wasm/CMakeLists.txt | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index a785ba6c0..ba13f5c43 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -20,6 +20,12 @@ if (NOT PACKAGE_DIR STREQUAL "")
   set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${REALPATH_PACKAGE_DIR}@/")
 endif()
 
+# Enable worker file system
+set(LINKER_FLAGS "${LINKER_FLAGS} -lworkerfs.js")
+
+# Avoid node.js-code in emscripten glue-code
+set(LINKER_FLAGS "${LINKER_FLAGS} -s ENVIRONMENT=web,worker")
+
 set_target_properties(bergamot-translator-worker PROPERTIES
                         SUFFIX ".js"
                         LINK_FLAGS ${LINKER_FLAGS}

From 743ebcd3bcfdf5e4b141a9ed3478eb79de19dece Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Tue, 4 May 2021 15:25:12 +0300
Subject: [PATCH 217/442] Extension desired changes (#129)

* Enable worker file system
* Avoid node.js-code in emscripten glue-code
---
 wasm/CMakeLists.txt | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index a785ba6c0..ba13f5c43 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -20,6 +20,12 @@ if (NOT PACKAGE_DIR STREQUAL "")
   set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${REALPATH_PACKAGE_DIR}@/")
 endif()
 
+# Enable worker file system
+set(LINKER_FLAGS "${LINKER_FLAGS} -lworkerfs.js")
+
+# Avoid node.js-code in emscripten glue-code
+set(LINKER_FLAGS "${LINKER_FLAGS} -s ENVIRONMENT=web,worker")
+
 set_target_properties(bergamot-translator-worker PROPERTIES
                         SUFFIX ".js"
                         LINK_FLAGS ${LINKER_FLAGS}

From c61b2bdd1071c976635c9ce722f4dcff4b76289d Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Thu, 6 May 2021 00:21:50 +0100
Subject: [PATCH 218/442] Fix busy loop in windows (#131)

* Fix busy loop in windows

* Nick wants the while loop gone

* Fix continue leftover

Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
---
 src/translator/pcqueue.h | 20 +++++++++-----------
 1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/src/translator/pcqueue.h b/src/translator/pcqueue.h
index d6f458275..8d5d6e2a0 100644
--- a/src/translator/pcqueue.h
+++ b/src/translator/pcqueue.h
@@ -113,17 +113,15 @@ class Semaphore {
 
 
     void wait() {
-      while (true) {
-        switch (WaitForSingleObject(sem_, 0L)) {
-          case WAIT_OBJECT_0:
-            return;
-          case WAIT_ABANDONED:
-            ABORT("A semaphore can't be abandoned, confused by Windows");
-          case WAIT_TIMEOUT:
-            continue;
-          case WAIT_FAILED:
-            ABORT("Waiting on Semaphore failed {}", GetLastError());
-        }
+      switch (WaitForSingleObject(sem_, INFINITE)) {
+        case WAIT_OBJECT_0:
+          return;
+        case WAIT_ABANDONED:
+          ABORT("A semaphore can't be abandoned, confused by Windows");
+        case WAIT_TIMEOUT:
+          ABORT("Timeout on an infinite wait?");
+        case WAIT_FAILED:
+          ABORT("Waiting on Semaphore failed {}", GetLastError());
       }
     }
 

From bc2e4eee5ccb7bfc0158edb7bd7b7d12090703cb Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 6 May 2021 00:26:03 +0100
Subject: [PATCH 219/442] Making bytearray a commandline switch (#127)

* Adding bytearray option

* collapse intermediate for bytearray apps

* Removing service-cli-bytearray

* Removing the bergamot bytearray app

* Bumping updates to brt collapsing apps

* Reasonable defaults and hard check when cmd enabled

* Update documentation for flags

* Bump brt with MKL check and skip

* Bumping BRT with MKL_FOUND instead of USE_MKL

* Bumping BRT with no mkl enforce

* Bumping BRT with ssse3 output

* Let's try disabling OpenBLAS

* Trying to disable apple accelerate

* Using WASM compatible BLAS can enable intgemm

* Adding a CMake -L to see what exactly is the diff

* Revert "Let's try disabling OpenBLAS"

This reverts commit 9a6b9bc53bf7dec956889f6e0b7047e5388e1b7e.

* Revert "Using WASM compatible BLAS can enable intgemm"

This reverts commit 936a592e18431c279e6c5952a278d012d18ff295.

* Restricting mac tests through tags and on GitHub CI

* Using only check-bytearray

* Bumping BRT with change of default behaviour
---
 .github/workflows/native-mac.yml          |  8 +-
 app/CMakeLists.txt                        |  6 --
 app/bergamot-translator-app-bytearray.cpp | 40 ----------
 app/service-cli-bytearray.cpp             | 92 -----------------------
 app/service-cli.cpp                       | 14 +++-
 bergamot-translator-tests                 |  2 +-
 6 files changed, 20 insertions(+), 142 deletions(-)
 delete mode 100644 app/bergamot-translator-app-bytearray.cpp
 delete mode 100644 app/service-cli-bytearray.cpp

diff --git a/.github/workflows/native-mac.yml b/.github/workflows/native-mac.yml
index 8df203d5d..b1fe18fe4 100644
--- a/.github/workflows/native-mac.yml
+++ b/.github/workflows/native-mac.yml
@@ -14,7 +14,7 @@ jobs:
         include:
           - name: "full-marian"
             os: macos-10.15
-            test_tags: ""
+            test_tags: "'#mac'"
             cmake: 
               CMAKE_BUILD_TYPE: "Release"
               COMPILE_TESTS: "ON"
@@ -23,6 +23,7 @@ jobs:
               USE_STATIC_LIBS: "OFF"
               COMPILE_SERVER: "OFF"
               COMPILE_EXAMPLES: "OFF"
+              USE_APPLE_ACCELERATE: "OFF"
 
           - name: "minimal-marian"
             os: macos-10.15
@@ -37,6 +38,8 @@ jobs:
               USE_STATIC_LIBS: "ON" 
               COMPILE_SERVER: "OFF"
               COMPILE_EXAMPLES: "OFF"
+              USE_APPLE_ACCELERATE: "OFF"
+
         
     name: ${{ matrix.name }}
     runs-on: ${{ matrix.os }}
@@ -66,13 +69,14 @@ jobs:
       run: |
         mkdir -p build
         cd build
-        cmake .. \
+        cmake -L .. \
           -DCMAKE_BUILD_TYPE=${{ matrix.cmake.CMAKE_BUILD_TYPE }}\
           -DCOMPILE_TESTS=${{ matrix.cmake.COMPILE_TESTS }}\
           -DCOMPILE_EXAMPLES=${{ matrix.cmake.COMPILE_EXAMPLES }} \
           -DCOMPILE_SERVER=${{ matrix.cmake.COMPILE_SERVER }} \
           -DUSE_STATIC_LIBS=${{ matrix.cmake.USE_STATIC_LIBS }} \
           -DUSE_WASM_COMPATIBLE_SOURCE=${{ matrix.cmake.USE_WASM_COMPATIBLE_SOURCE }} \
+          -DUSE_APPLE_ACCELERATE=${{ matrix.cmake.USE_APPLE_ACCELERATE }} \
           -DUSE_FBGEMM=${{ matrix.cmake.USE_FBGEMM }}
 
     - name: Compile
diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index e23a10720..a55a571b7 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -1,16 +1,10 @@
 add_executable(bergamot-translator-app bergamot-translator-app.cpp)
 target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
 
-add_executable(bergamot-translator-app-bytearray bergamot-translator-app-bytearray.cpp)
-target_link_libraries(bergamot-translator-app-bytearray PRIVATE bergamot-translator)
-
 if (NOT USE_WASM_COMPATIBLE_SOURCE)
     add_executable(service-cli service-cli.cpp)
     target_link_libraries(service-cli PRIVATE bergamot-translator)
 
-    add_executable(service-cli-bytearray service-cli-bytearray.cpp)
-    target_link_libraries(service-cli-bytearray PRIVATE bergamot-translator)
-
     add_executable(marian-decoder-new marian-decoder-new.cpp)
     target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)
 endif()
diff --git a/app/bergamot-translator-app-bytearray.cpp b/app/bergamot-translator-app-bytearray.cpp
deleted file mode 100644
index 91353c0a3..000000000
--- a/app/bergamot-translator-app-bytearray.cpp
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * main.cpp
- *
- * An example application to demonstrate the use of Bergamot translator.
- *
- */
-
-#include <iostream>
-
-#include "translator/byte_array_util.h"
-#include "translator/parser.h"
-#include "translator/service.h"
-
-int main(int argc, char **argv) {
-
-  // Create a configParser and load command line parameters into a YAML config
-  // string.
-  auto configParser = marian::bergamot::createConfigParser();
-  auto options = configParser.parseOptions(argc, argv, true);
-  std::string config = options->asYamlString();
-
-  // Route the config string to construct marian model through TranslationModel
-  marian::bergamot::Service model(
-      config, marian::bergamot::getModelMemoryFromConfig(options));
-
-  TranslationRequest translationRequest;
-  std::vector<std::string> texts;
-
-  for (std::string line; std::getline(std::cin, line);) {
-    texts.emplace_back(line);
-  }
-
-  auto results = model.translateMultiple(std::move(texts), translationRequest);
-
-  for (auto &result : results) {
-    std::cout << result.getTranslatedText() << std::endl;
-  }
-
-  return 0;
-}
diff --git a/app/service-cli-bytearray.cpp b/app/service-cli-bytearray.cpp
deleted file mode 100644
index d8c7059ef..000000000
--- a/app/service-cli-bytearray.cpp
+++ /dev/null
@@ -1,92 +0,0 @@
-#include <cstdlib>
-#include <future>
-#include <iostream>
-#include <sstream>
-
-#include "common/definitions.h"
-#include "common/utils.h"
-#include "marian.h"
-#include "translator/parser.h"
-#include "translator/response.h"
-#include "translator/service.h"
-#include "translator/byte_array_util.h"
-
-int main(int argc, char *argv[]) {
-  auto cp = marian::bergamot::createConfigParser();
-  auto options = cp.parseOptions(argc, argv, true);
-
-  // Prepare memories for model and shortlist
-  marian::bergamot::AlignedMemory modelBytes = marian::bergamot::getModelMemoryFromConfig(options);
-  marian::bergamot::AlignedMemory shortlistBytes = marian::bergamot::getShortlistMemoryFromConfig(options);
-
-  marian::bergamot::Service service(options, std::move(modelBytes), std::move(shortlistBytes));
-
-  // Read a large input text blob from stdin
-  std::ostringstream std_input;
-  std_input << std::cin.rdbuf();
-  std::string input = std_input.str();
-  using marian::bergamot::Response;
-
-  marian::bergamot::ResponseOptions responseOptions;
-  responseOptions.qualityScores = true;
-  responseOptions.alignment = true;
-  responseOptions.alignmentThreshold = 0.2f;
-
-  // Wait on future until Response is complete
-  std::future<Response> responseFuture =
-      service.translate(std::move(input), responseOptions);
-  responseFuture.wait();
-  Response response = responseFuture.get();
-
-  std::cout << "[original]: " << response.source.text << '\n';
-  std::cout << "[translated]: " << response.target.text << '\n';
-  for (int sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
-    std::cout << " [src Sentence]: " << response.source.sentence(sentenceIdx)
-              << '\n';
-    std::cout << " [tgt Sentence]: " << response.target.sentence(sentenceIdx)
-              << '\n';
-    std::cout << "Alignments" << '\n';
-    typedef std::pair<size_t, float> Point;
-
-    // Initialize a point vector.
-    std::vector<std::vector<Point>> aggregate(
-        response.source.numWords(sentenceIdx));
-
-    // Handle alignments
-    auto &alignments = response.alignments[sentenceIdx];
-    for (auto &p : alignments) {
-      aggregate[p.src].emplace_back(p.tgt, p.prob);
-    }
-
-    for (size_t src = 0; src < aggregate.size(); src++) {
-      std::cout << response.source.word(sentenceIdx, src) << ": ";
-      for (auto &p : aggregate[src]) {
-        std::cout << response.target.word(sentenceIdx, p.first) << "("
-                  << p.second << ") ";
-      }
-      std::cout << '\n';
-    }
-
-    // Handle quality.
-    auto &quality = response.qualityScores[sentenceIdx];
-    std::cout << "Quality: whole(" << quality.sequence
-              << "), tokens below:" << '\n';
-    size_t wordIdx = 0;
-    bool first = true;
-    for (auto &p : quality.word) {
-      if (first) {
-        first = false;
-      } else {
-        std::cout << " ";
-      }
-      std::cout << response.target.word(sentenceIdx, wordIdx) << "(" << p
-                << ")";
-      wordIdx++;
-    }
-    std::cout << '\n';
-  }
-  std::cout << "--------------------------\n";
-  std::cout << '\n';
-
-  return 0;
-}
diff --git a/app/service-cli.cpp b/app/service-cli.cpp
index d7c72e604..e9dc4332d 100644
--- a/app/service-cli.cpp
+++ b/app/service-cli.cpp
@@ -6,6 +6,7 @@
 #include "common/definitions.h"
 #include "common/utils.h"
 #include "marian.h"
+#include "translator/byte_array_util.h"
 #include "translator/parser.h"
 #include "translator/response.h"
 #include "translator/response_options.h"
@@ -14,7 +15,18 @@
 int main(int argc, char *argv[]) {
   auto cp = marian::bergamot::createConfigParser();
   auto options = cp.parseOptions(argc, argv, true);
-  marian::bergamot::Service service(options);
+
+  // Prepare memories for model and shortlist
+  marian::bergamot::AlignedMemory modelBytes, shortlistBytes;
+
+  if (options->get<bool>("check-bytearray")) {
+    // Load legit values into bytearrays.
+    modelBytes = marian::bergamot::getModelMemoryFromConfig(options);
+    shortlistBytes = marian::bergamot::getShortlistMemoryFromConfig(options);
+  }
+
+  marian::bergamot::Service service(options, std::move(modelBytes),
+                                    std::move(shortlistBytes));
 
   // Read a large input text blob from stdin
   std::ostringstream std_input;
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 377100172..442edcfea 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 3771001720a8f01bba185ee5d5d908b7c266ef31
+Subproject commit 442edcfea34dc1c53c90b5775347958fba1ffd08

From b86c76b0041693a88cd18507e4dac94f6a89aecb Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 6 May 2021 16:19:27 +0100
Subject: [PATCH 220/442] Faithful to source-structure translation (#115)

* First draft of faithful translation

* Comments explaining pre and post

* Comments on response_builder

* Updating bergamot-translator-tests with new outputs

* Cosmetic changes in response target text construction

* Replacing &(x[0]) -> x.data() to avoid illegal indices

* Removing nullptr given both branches init pointer with legal values

* pre, post -> gap(i) addressing review comments

Functions which were pre and post before are subsumed by gap(i), and the
algorithm in ResponseBuilder adjusted to fix.

`x = nullptr` is back, should be harmless.

* Updating brt with paragraph outputs

* Bumping brt with updated outputs, buffer text at begin as well

* Bumping BRT with sync after bytearray collapse merge

* Pointing BRT to main after merge

Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
---
 bergamot-translator-tests           |  2 +-
 src/translator/response_builder.cpp | 40 +++++++++++++++++++++--------
 src/translator/sentence_ranges.cpp  | 32 +++++++++++++++++++++--
 src/translator/sentence_ranges.h    | 13 ++++++++++
 4 files changed, 73 insertions(+), 14 deletions(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 442edcfea..9209aa51e 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 442edcfea34dc1c53c90b5775347958fba1ffd08
+Subproject commit 9209aa51e71f57b90172ffd259cf3021c4890bcf
diff --git a/src/translator/response_builder.cpp b/src/translator/response_builder.cpp
index c62470789..f68bd31f7 100644
--- a/src/translator/response_builder.cpp
+++ b/src/translator/response_builder.cpp
@@ -1,4 +1,5 @@
 #include "response_builder.h"
+#include "response_options.h"
 
 namespace marian {
 namespace bergamot {
@@ -56,11 +57,10 @@ void ResponseBuilder::buildTranslatedText(Histories &histories,
   // thing to do to avoid reallocations.
   response.target.text.reserve(response.source.text.size());
 
-  size_t offset{0};
-  bool first{true};
-
-  for (auto &history : histories) {
+  for (size_t sentenceIdx = 0; sentenceIdx < histories.size(); sentenceIdx++) {
     // TODO(jerin): Change hardcode of nBest = 1
+
+    auto &history = histories[sentenceIdx];
     NBestList onebest = history->nBest(1);
 
     Result result = onebest[0]; // Expecting only one result;
@@ -71,15 +71,33 @@ void ResponseBuilder::buildTranslatedText(Histories &histories,
     std::vector<string_view> targetSentenceMappings;
     targetVocab->decodeWithByteRanges(words, decoded, targetSentenceMappings);
 
-    // delimiter can be used to fill in the blanks from source as well.
-    std::string delimiter;
-    if (first) {
-      first = false;
-    } else {
-      delimiter = " ";
+    switch (responseOptions_.concatStrategy) {
+    case ConcatStrategy::FAITHFUL: {
+      // For each sentence, prepend the filler text between the corresponding
+      // source-sentence and the source-sentence before.
+      string_view pre = response.source.gap(sentenceIdx);
+      response.target.appendSentence(std::string(pre.data(), pre.size()),
+                                     decoded, targetSentenceMappings);
+
+      // If this is the last history to be decoded and translated-text
+      // constructed, append the text till the end, which could be spaces or
+      // empty.
+      if (sentenceIdx + 1 == histories.size()) {
+        string_view post = response.source.gap(sentenceIdx + 1);
+        response.target.text += std::string(post.data(), post.size());
+      }
+      break;
+    }
+    case ConcatStrategy::SPACE: {
+      std::string delimiter = (sentenceIdx == 0) ? "" : " ";
+      response.target.appendSentence(delimiter, decoded,
+                                     targetSentenceMappings);
+      break;
     }
 
-    response.target.appendSentence(delimiter, decoded, targetSentenceMappings);
+    default:
+      ABORT("Unknown concat-strategy");
+    }
   }
 }
 
diff --git a/src/translator/sentence_ranges.cpp b/src/translator/sentence_ranges.cpp
index da9d3eee0..603d9bf9c 100644
--- a/src/translator/sentence_ranges.cpp
+++ b/src/translator/sentence_ranges.cpp
@@ -63,7 +63,7 @@ void AnnotatedText::appendSentence(std::string prefix, std::string &reference,
   text += reference;           // Append reference to text
   std::vector<ByteRange> sentence;
   for (auto &wordView : wordRanges) {
-    size_t thisWordBegin = offset + wordView.data() - &reference[0];
+    size_t thisWordBegin = offset + wordView.data() - reference.data();
     sentence.push_back(
         ByteRange{thisWordBegin, thisWordBegin + wordView.size()});
   }
@@ -78,7 +78,7 @@ void AnnotatedText::addSentence(std::vector<string_view>::iterator begin,
                                 std::vector<string_view>::iterator end) {
   std::vector<ByteRange> sentence;
   for (auto p = begin; p != end; p++) {
-    size_t begin_offset = p->data() - &text[0];
+    size_t begin_offset = p->data() - text.data();
     sentence.push_back(ByteRange{begin_offset, begin_offset + p->size()});
   }
   annotation.addSentence(sentence);
@@ -99,5 +99,33 @@ string_view AnnotatedText::asStringView(const ByteRange &byteRange) const {
   return string_view(data, size);
 }
 
+string_view AnnotatedText::gap(size_t sentenceIdx) const {
+  // Find start of filler-text before, there's a corner case when there's no
+  // sentence before.
+  const char *start = nullptr;
+  if (sentenceIdx == 0) {
+    // If first sentence, filler begins at start of whole-text.
+    start = text.data();
+  } else {
+    // Otherwise, filler begins at end of previous sentence.
+    string_view sentenceBefore = sentence(sentenceIdx - 1);
+    start = sentenceBefore.data() + sentenceBefore.size();
+  }
+
+  // Find end of filler-text, but there is a corner-case to handle.
+  const char *end = nullptr;
+  if (sentenceIdx == numSentences()) {
+    // If last sentence, manually find end of whole-text.
+    const char *begin = text.data();
+    end = begin + text.size();
+  } else {
+    // Otherwise, the filler ends at the start of next sentence.
+    string_view sentenceAfter = sentence(sentenceIdx);
+    end = sentenceAfter.data();
+  }
+
+  return string_view(start, end - start);
+}
+
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/sentence_ranges.h b/src/translator/sentence_ranges.h
index f9c881e17..8cb7caf0c 100644
--- a/src/translator/sentence_ranges.h
+++ b/src/translator/sentence_ranges.h
@@ -151,6 +151,19 @@ struct AnnotatedText {
   /// Returns a string_view representing sentence corresponding to sentenceIdx.
   string_view sentence(size_t sentenceIdx) const;
 
+  /// Returns the string_view of the gap between two sentences in the container.
+  ///
+  /// More precisely where `i = sentenceIdx, N = numSentences()` for brevity:
+  ///
+  /// * For `i = 0`: The gap between the start of text and the first sentence.
+  /// * For `i = 1...N-1`, returns the text comprising of the gap
+  ///   between the `i-1`-th and `i`-th sentence.
+  /// * For `i = N`, the gap between the last sentence and end of
+  ///   text.
+
+  /// @param sentenceIdx: Can be between `[0, numSentences()]`.
+  string_view gap(size_t sentenceIdx) const;
+
   /// Returns a ByteRange representing wordIdx in sentenceIdx
   ByteRange wordAsByteRange(size_t sentenceIdx, size_t wordIdx) const;
 

From 5b02008a97e7fe626abca3f78e617fbe3f23a9b7 Mon Sep 17 00:00:00 2001
From: Qianqian Zhu <qianqian.zhu@hotmail.com>
Date: Fri, 7 May 2021 14:54:48 +0100
Subject: [PATCH 221/442] Enable vocabs pass as byte arrays (#122)

* first attempt to enable vocabs pass as byte arrays

* pass vocabs bytes as AlignedMemory

* add vocabIndices to avoid double loading

* small fix on parameter names and documentation

* fix windows build plus tiny update on documentation

* update marian-dev submodule

* move validate model bytearray in BatchTranslator

* small refactors on validateBinaryModel()

* switch vocab memories to std::vector<marian::Ptr<AlignedMemory>>

* update marian-dev submodule

* replace marian::Ptr to std::shared_ptr for vocab memories

* add note for vocab memories
---
 3rd_party/marian-dev                |  2 +-
 app/service-cli.cpp                 |  4 ++-
 src/translator/batch_translator.cpp |  7 ++++-
 src/translator/byte_array_util.cpp  | 37 ++++++++++++++--------
 src/translator/byte_array_util.h    |  4 ++-
 src/translator/service.cpp          | 48 +++++++++++++++++++----------
 src/translator/service.h            | 29 ++++++++++-------
 7 files changed, 88 insertions(+), 43 deletions(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 94aeaa461..ca15d61c8 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 94aeaa4616a0fb01ac95a23f0e74a214a94e7609
+Subproject commit ca15d61c87ef2f8f2c290b75a5da6236eb9833d2
diff --git a/app/service-cli.cpp b/app/service-cli.cpp
index e9dc4332d..0e958d6a7 100644
--- a/app/service-cli.cpp
+++ b/app/service-cli.cpp
@@ -18,15 +18,17 @@ int main(int argc, char *argv[]) {
 
   // Prepare memories for model and shortlist
   marian::bergamot::AlignedMemory modelBytes, shortlistBytes;
+  std::vector<std::shared_ptr<marian::bergamot::AlignedMemory>> vocabsBytes;
 
   if (options->get<bool>("check-bytearray")) {
     // Load legit values into bytearrays.
     modelBytes = marian::bergamot::getModelMemoryFromConfig(options);
     shortlistBytes = marian::bergamot::getShortlistMemoryFromConfig(options);
+    marian::bergamot::getVocabsMemoryFromConfig(options, vocabsBytes);
   }
 
   marian::bergamot::Service service(options, std::move(modelBytes),
-                                    std::move(shortlistBytes));
+                                    std::move(shortlistBytes), std::move(vocabsBytes));
 
   // Read a large input text blob from stdin
   std::ostringstream std_input;
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index 6b2425d26..c6271726a 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -4,6 +4,7 @@
 #include "data/corpus.h"
 #include "data/text_input.h"
 #include "translator/beam_search.h"
+#include "byte_array_util.h"
 
 namespace marian {
 namespace bergamot {
@@ -18,11 +19,11 @@ BatchTranslator::BatchTranslator(DeviceId const device,
 
 void BatchTranslator::initialize() {
   // Initializes the graph.
+  bool check = options_->get<bool>("check-bytearray",false); // Flag holds whether validate the bytearray (model and shortlist)
   if (options_->hasAndNotEmpty("shortlist")) {
     int srcIdx = 0, trgIdx = 1;
     bool shared_vcb = vocabs_->front() == vocabs_->back();
     if (shortlistMemory_->size() > 0 && shortlistMemory_->begin() != nullptr) {
-      bool check = options_->get<bool>("check-bytearray",true);
       slgen_ = New<data::BinaryShortlistGenerator>(shortlistMemory_->begin(), shortlistMemory_->size(),
                                                      vocabs_->front(), vocabs_->back(),
                                                      srcIdx, trgIdx, shared_vcb, check);
@@ -45,6 +46,10 @@ void BatchTranslator::initialize() {
   if (modelMemory_->size() > 0 && modelMemory_->begin() != nullptr) { // If we have provided a byte array that contains the model memory, we can initialise the model from there, as opposed to from reading in the config file
     ABORT_IF((uintptr_t)modelMemory_->begin() % 256 != 0,
              "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
+    if (check) {
+      ABORT_IF(!validateBinaryModel(*modelMemory_, modelMemory_->size()),
+               "The binary file is invalid. Incomplete or corrupted download?");
+    }
     const std::vector<const void *> container = {modelMemory_->begin()}; // Marian supports multiple models initialised in this manner hence std::vector. However we will only ever use 1 during decoding.
     scorers_ = createScorers(options_, container);
   } else {
diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index c3bf7cc69..00beaa62e 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -1,12 +1,12 @@
 #include "byte_array_util.h"
 #include <stdlib.h>
 #include <iostream>
+#include <memory>
 
 namespace marian {
 namespace bergamot {
 
 namespace {
-
 // This is a basic validator that checks if the file has not been truncated
 // it basically loads up the header and checks
 
@@ -26,9 +26,10 @@ const T* get(const void*& current, uint64_t num = 1) {
   current = (const T*)current + num;
   return ptr;
 }
+} // Anonymous namespace
 
-bool validateBinaryModel(AlignedMemory& model, uint64_t fileSize) {
-  const void * current = &model[0];
+bool validateBinaryModel(const AlignedMemory& model, uint64_t fileSize) {
+  const void * current = model.begin();
   uint64_t memoryNeeded = sizeof(uint64_t)*2; // We keep track of how much memory we would need if we have a complete file
   uint64_t numHeaders;
   if (fileSize >= memoryNeeded) { // We have enough filesize to fetch the headers.
@@ -76,8 +77,6 @@ bool validateBinaryModel(AlignedMemory& model, uint64_t fileSize) {
   }
 }
 
-} // Anonymous namespace
-
 AlignedMemory loadFileToMemory(const std::string& path, size_t alignment){
   uint64_t fileSize = filesystem::fileSize(path);
   io::InputFileStream in(path);
@@ -89,13 +88,12 @@ AlignedMemory loadFileToMemory(const std::string& path, size_t alignment){
 }
 
 AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options){
-    auto models = options->get<std::vector<std::string>>("models");
-    ABORT_IF(models.size() != 1, "Loading multiple binary models is not supported for now as it is not necessary.");
-    marian::filesystem::Path modelPath(models[0]);
-    ABORT_IF(modelPath.extension() != marian::filesystem::Path(".bin"), "The file of binary model should end with .bin");
-    AlignedMemory alignedMemory = loadFileToMemory(models[0], 256);
-    ABORT_IF(!validateBinaryModel(alignedMemory, alignedMemory.size()), "The binary file is invalid. Incomplete or corrupted download?");
-    return alignedMemory;
+  auto models = options->get<std::vector<std::string>>("models");
+  ABORT_IF(models.size() != 1, "Loading multiple binary models is not supported for now as it is not necessary.");
+  marian::filesystem::Path modelPath(models[0]);
+  ABORT_IF(modelPath.extension() != marian::filesystem::Path(".bin"), "The file of binary model should end with .bin");
+  AlignedMemory alignedMemory = loadFileToMemory(models[0], 256);
+  return alignedMemory;
 }
 
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options){
@@ -104,5 +102,20 @@ AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options)
   return loadFileToMemory(shortlist[0], 64);
 }
 
+void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
+                               std::vector<std::shared_ptr<AlignedMemory>>& vocabMemories){
+  auto vfiles = options->get<std::vector<std::string>>("vocabs");
+  ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
+  vocabMemories.resize(vfiles.size());
+  std::unordered_map<std::string, std::shared_ptr<AlignedMemory>> vocabMap;
+  for (size_t i = 0; i < vfiles.size(); ++i) {
+    auto m = vocabMap.emplace(std::make_pair(vfiles[i], std::shared_ptr<AlignedMemory>()));
+    if (m.second) {
+      m.first->second = std::make_shared<AlignedMemory>(loadFileToMemory(vfiles[i], 64));
+    }
+    vocabMemories[i] = m.first->second;
+  }
+}
+
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/byte_array_util.h b/src/translator/byte_array_util.h
index a8df1cbb0..3cbf3d339 100644
--- a/src/translator/byte_array_util.h
+++ b/src/translator/byte_array_util.h
@@ -7,6 +7,8 @@ namespace bergamot {
 AlignedMemory loadFileToMemory(const std::string& path, size_t alignment);
 AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options);
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options);
-
+void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
+                               std::vector<std::shared_ptr<AlignedMemory>>& vocabMemories);
+bool validateBinaryModel(const AlignedMemory& model, uint64_t fileSize);
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 3d19f5e66..385a2a598 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -6,21 +6,34 @@
 #include <utility>
 
 inline std::vector<marian::Ptr<const marian::Vocab>>
-loadVocabularies(marian::Ptr<marian::Options> options) {
+loadVocabularies(marian::Ptr<marian::Options> options,
+                 std::vector<std::shared_ptr<marian::bergamot::AlignedMemory>>&& vocabMemories) {
   // @TODO: parallelize vocab loading for faster startup
-  auto vfiles = options->get<std::vector<std::string>>("vocabs");
-  // with the current setup, we need at least two vocabs: src and trg
-  ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
-  std::vector<marian::Ptr<marian::Vocab const>> vocabs(vfiles.size());
-  std::unordered_map<std::string, marian::Ptr<marian::Vocab>> vmap;
-  for (size_t i = 0; i < vocabs.size(); ++i) {
-    auto m =
-        vmap.emplace(std::make_pair(vfiles[i], marian::Ptr<marian::Vocab>()));
-    if (m.second) { // new: load the vocab
-      m.first->second = marian::New<marian::Vocab>(options, i);
-      m.first->second->load(vfiles[i]);
+  std::vector<marian::Ptr<marian::Vocab const>> vocabs;
+  if(!vocabMemories.empty()){
+    // load vocabs from buffer
+    ABORT_IF(vocabMemories.size() < 2, "Insufficient number of vocabularies.");
+    vocabs.resize(vocabMemories.size());
+    for (size_t i = 0; i < vocabs.size(); i++) {
+      marian::Ptr<marian::Vocab> vocab = marian::New<marian::Vocab>(options, i);
+      vocab->loadFromSerialized(absl::string_view(vocabMemories[i]->begin(), vocabMemories[i]->size()));
+      vocabs[i] = vocab;
+    }
+  } else {
+    // load vocabs from file
+    auto vfiles = options->get<std::vector<std::string>>("vocabs");
+    // with the current setup, we need at least two vocabs: src and trg
+    ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
+    vocabs.resize(vfiles.size());
+    std::unordered_map<std::string, marian::Ptr<marian::Vocab>> vmap;
+    for (size_t i = 0; i < vocabs.size(); ++i) {
+      auto m = vmap.emplace(std::make_pair(vfiles[i], marian::Ptr<marian::Vocab>()));
+      if (m.second) { // new: load the vocab
+        m.first->second = marian::New<marian::Vocab>(options, i);
+        m.first->second->load(vfiles[i]);
+      }
+      vocabs[i] = m.first->second;
     }
-    vocabs[i] = m.first->second;
   }
   return vocabs;
 }
@@ -28,11 +41,14 @@ loadVocabularies(marian::Ptr<marian::Options> options) {
 namespace marian {
 namespace bergamot {
 
-Service::Service(Ptr<Options> options, AlignedMemory modelMemory, AlignedMemory shortlistMemory) 
-    : requestId_(0), options_(options), vocabs_(std::move(loadVocabularies(options))),
+Service::Service(Ptr<Options> options, AlignedMemory modelMemory, AlignedMemory shortlistMemory,
+                 std::vector<std::shared_ptr<AlignedMemory>> vocabMemories)
+    : requestId_(0), options_(options),
+      vocabs_(std::move(loadVocabularies(options, std::move(vocabMemories)))),
       text_processor_(vocabs_, options), batcher_(options),
       numWorkers_(options->get<int>("cpu-threads")),
-      modelMemory_(std::move(modelMemory)), shortlistMemory_(std::move(shortlistMemory))
+      modelMemory_(std::move(modelMemory)),
+      shortlistMemory_(std::move(shortlistMemory))
 #ifndef WASM_COMPATIBLE_SOURCE
       // 0 elements in PCQueue is illegal and can lead to failures. Adding a
       // guard to have at least one entry allocated. In the single-threaded
diff --git a/src/translator/service.h b/src/translator/service.h
index 288c6495a..721d436a1 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -64,10 +64,12 @@ class Service {
 public:
   /// @param options Marian options object
   /// @param modelMemory byte array (aligned to 256!!!) that contains the bytes
-  /// of a model.bin. Optional, defaults to nullptr when not used
+  /// of a model.bin.
   /// @param shortlistMemory byte array of shortlist (aligned to 64)
+  /// @param vocabMemories vector of vocabulary memories (aligned to 64)
   explicit Service(Ptr<Options> options, AlignedMemory modelMemory,
-                   AlignedMemory shortlistMemory);
+                   AlignedMemory shortlistMemory,
+                   std::vector<std::shared_ptr<AlignedMemory>> vocabMemories);
 
   /// Construct Service purely from Options. This expects options which
   /// marian-decoder expects to be set for loading model shortlist and
@@ -76,24 +78,30 @@ class Service {
   ///
   /// This is equivalent to a call to:
   /// ```cpp
-  ///    Service(options, AlignedMemory(),  AlignedMemory())
+  ///    Service(options, AlignedMemory(), AlignedMemory(), {})
   /// ```
   /// wherein empty memory is passed and internal flow defaults to file-based
-  /// model, shortlist loading.
+  /// model, shortlist loading. AlignedMemory() corresponds to empty memory
   explicit Service(Ptr<Options> options)
-      : Service(options, AlignedMemory(), AlignedMemory()) {}
+      : Service(options, AlignedMemory(), AlignedMemory(), {}) {}
 
   /// Construct Service from a string configuration.
   /// @param [in] config string parsable as YAML expected to adhere with marian
   /// config
-  /// @param [in] model_memory byte array (aligned to 256!!!) that contains the
-  /// bytes of a model.bin. Optional.
-  /// @param [in] shortlistMemory byte array of shortlist (aligned to 64)
+  /// @param [in] modelMemory byte array (aligned to 256!!!) that contains the
+  /// bytes of a model.bin. Optional. AlignedMemory() corresponds to empty memory
+  /// @param [in] shortlistMemory byte array of shortlist (aligned to 64). Optional.
+  /// @param [in] vocabMemories vector of vocabulary memories (aligned to 64). Optional.
+  /// If two vocabularies are the same (based on the filenames), two entries (shared
+  /// pointers) will be generated which share the same AlignedMemory object.
   explicit Service(const std::string &config,
                    AlignedMemory modelMemory = AlignedMemory(),
-                   AlignedMemory shortlistMemory = AlignedMemory())
+                   AlignedMemory shortlistMemory = AlignedMemory(),
+                   std::vector<std::shared_ptr<AlignedMemory>> vocabsMemories = {})
       : Service(parseOptions(config, /*validate=*/false),
-                std::move(modelMemory), std::move(shortlistMemory)) {}
+                std::move(modelMemory),
+                std::move(shortlistMemory),
+                std::move(vocabsMemories)) {}
 
   /// Explicit destructor to clean up after any threads initialized in
   /// asynchronous operation mode.
@@ -187,7 +195,6 @@ class Service {
   /// ordering among requests and logging/book-keeping.
 
   size_t requestId_;
-
   /// Store vocabs representing source and target.
   std::vector<Ptr<Vocab const>> vocabs_; // ORDER DEPENDENCY (text_processor_)
 

From 21c1cae4724727e1b0cbfa397db7a92986c2ac62 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Fri, 7 May 2021 17:58:58 +0100
Subject: [PATCH 222/442] Update ssplit submodule, removing absl (#132)

* Update ssplit submodule, removing absl

* Fix ssplit variables

* Update ssplit branch

* Fix emscripten compilaiton

* Update tests
---
 3rd_party/ssplit-cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 8d338ed5c..f928904b0 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 8d338ed5c77d22f8c86f60554596fa57bf5091e6
+Subproject commit f928904b03e7f80d3942d38e5ca62c64a21c0b4b

From bef12765adb454b649fd4c54a041f6c7736ec146 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 7 May 2021 18:38:27 +0100
Subject: [PATCH 223/442] Minor rename: sentence_ranges -> annotation (#134)

---
 src/tests/annotation_tests.cpp                         | 2 +-
 src/translator/CMakeLists.txt                          | 2 +-
 src/translator/{sentence_ranges.cpp => annotation.cpp} | 2 +-
 src/translator/{sentence_ranges.h => annotation.h}     | 0
 src/translator/request.cpp                             | 2 +-
 src/translator/request.h                               | 2 +-
 src/translator/response.h                              | 2 +-
 src/translator/text_processor.cpp                      | 2 +-
 src/translator/text_processor.h                        | 2 +-
 9 files changed, 8 insertions(+), 8 deletions(-)
 rename src/translator/{sentence_ranges.cpp => annotation.cpp} (99%)
 rename src/translator/{sentence_ranges.h => annotation.h} (100%)

diff --git a/src/tests/annotation_tests.cpp b/src/tests/annotation_tests.cpp
index 3e42baedb..d323b9db0 100644
--- a/src/tests/annotation_tests.cpp
+++ b/src/tests/annotation_tests.cpp
@@ -1,5 +1,5 @@
 #include "catch.hpp"
-#include "translator/sentence_ranges.h"
+#include "translator/annotation.h"
 #include <random>
 #include <vector>
 
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 25ca91638..a7ba0d343 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -7,7 +7,7 @@ add_library(bergamot-translator STATIC
     batcher.cpp
     response_builder.cpp
     batch.cpp
-    sentence_ranges.cpp
+    annotation.cpp
     service.cpp
 )
 if (USE_WASM_COMPATIBLE_SOURCE)
diff --git a/src/translator/sentence_ranges.cpp b/src/translator/annotation.cpp
similarity index 99%
rename from src/translator/sentence_ranges.cpp
rename to src/translator/annotation.cpp
index 603d9bf9c..c27d7849c 100644
--- a/src/translator/sentence_ranges.cpp
+++ b/src/translator/annotation.cpp
@@ -1,4 +1,4 @@
-#include "sentence_ranges.h"
+#include "annotation.h"
 #include <cassert>
 #include <iostream>
 
diff --git a/src/translator/sentence_ranges.h b/src/translator/annotation.h
similarity index 100%
rename from src/translator/sentence_ranges.h
rename to src/translator/annotation.h
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 8e46533fd..37b164f8b 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -1,7 +1,7 @@
 #include "request.h"
 #include "definitions.h"
 #include "response.h"
-#include "sentence_ranges.h"
+#include "annotation.h"
 
 #include "common/logging.h"
 
diff --git a/src/translator/request.h b/src/translator/request.h
index e2188cdbe..798368099 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -4,7 +4,7 @@
 #include "definitions.h"
 #include "response.h"
 #include "response_builder.h"
-#include "sentence_ranges.h"
+#include "annotation.h"
 
 #include "common/logging.h"
 #include "data/types.h"
diff --git a/src/translator/response.h b/src/translator/response.h
index 0f7ecb592..5d4ea7860 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -4,7 +4,7 @@
 #include "data/alignment.h"
 #include "data/types.h"
 #include "definitions.h"
-#include "sentence_ranges.h"
+#include "annotation.h"
 #include "translator/beam_search.h"
 
 #include <cassert>
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index 8d7f25cc9..fb6690125 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -1,7 +1,7 @@
 #include "text_processor.h"
 #include "data/types.h"
 #include "definitions.h"
-#include "sentence_ranges.h"
+#include "annotation.h"
 
 #include "common/options.h"
 #include "data/vocab.h"
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index ed3c7736b..698e36ea1 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -4,7 +4,7 @@
 #include "data/types.h"
 #include "data/vocab.h"
 #include "definitions.h"
-#include "sentence_ranges.h"
+#include "annotation.h"
 
 #include "sentence_splitter.h"
 

From 87adb5d60a19c9e5d4ce39f1afa4a08a0ac7edaf Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Fri, 7 May 2021 18:41:08 +0100
Subject: [PATCH 224/442] Target master of ssplit-cpp

---
 3rd_party/ssplit-cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index f928904b0..177ee2a32 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit f928904b03e7f80d3942d38e5ca62c64a21c0b4b
+Subproject commit 177ee2a326b733f8395842f01b197637047de9f6

From 354e7ac6bebe4facca76d715312c62fee064df9c Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sun, 9 May 2021 13:42:57 +0100
Subject: [PATCH 225/442] Remove unused used types TokenRanges,
 SentenceTokenRanges, UPtr (#137)

---
 src/translator/definitions.h | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index 18b5fcab0..58fd4b386 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -11,16 +11,6 @@ namespace bergamot {
 
 typedef marian::Words Segment;
 typedef std::vector<Segment> Segments;
-typedef std::vector<marian::string_view> TokenRanges;
-typedef std::vector<TokenRanges> SentenceTokenRanges;
-
-/** @brief Creates unique_ptr any type, passes all arguments to any available
- *  * constructor */
-template <class T, typename... Args> UPtr<T> UNew(Args &&... args) {
-  return UPtr<T>(new T(std::forward<Args>(args)...));
-}
-
-template <class T> UPtr<T> UNew(UPtr<T> p) { return UPtr<T>(p); }
 
 /// Shortcut to AlignedVector<char> for byte arrays
 typedef AlignedVector<char> AlignedMemory;

From ce01de939d8da7b054169e3a8b9fa20a89c9b8e5 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Mon, 10 May 2021 11:28:37 +0100
Subject: [PATCH 226/442] Change USE_WASM_COMPATIBLE_SOURCE =OFF by default on
 native, force on for WASM (#138)

* Change WASM_COMPATIBLE_SOURCE=OFF by default

The default was WASN_COMPATIBLE_SOURCE=ON COMPILE_WASM=OFF which is a
testing configuration, not a sensible default for native or wasm.

* Always USE_WASM_COMPATIBLE_SOURCE with COMPILE_WASM

* Set CMP0077 to fix variable handling
---
 3rd_party/marian-dev | 2 +-
 CMakeLists.txt       | 6 +++++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index ca15d61c8..03db505fd 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit ca15d61c87ef2f8f2c290b75a5da6236eb9833d2
+Subproject commit 03db505fda750fdecf8000d7ef7dd78dae65861c
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 3fe03c992..ef64863ca 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -4,6 +4,10 @@ if (POLICY CMP0074)
   cmake_policy(SET CMP0074 NEW) # CMake 3.12
 endif ()
 
+if (POLICY CMP0077)
+  cmake_policy(SET CMP0077 NEW)
+endif()
+
 project(bergamot_translator CXX C)
 
 set(CMAKE_CXX_STANDARD 17)
@@ -35,7 +39,7 @@ include(CMakeDependentOption)
 
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
-option(USE_WASM_COMPATIBLE_SOURCE "Use wasm compatible sources" ON)
+cmake_dependent_option(USE_WASM_COMPATIBLE_SOURCE "Use wasm compatible sources" OFF "NOT COMPILE_WASM" ON)
 option(COMPILE_TESTS "Compile bergamot-tests" OFF)
 
 SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be packaged (pre-loaded) in wasm builds")

From ce576c27f18b7db94029ae4dacf045d10195d405 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 10 May 2021 15:22:35 +0200
Subject: [PATCH 227/442] Export "addOnPreMain" function from wasm module

 - This is required in the extension while using wasm module in a worker environment
---
 wasm/CMakeLists.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index ba13f5c43..d98705b7c 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -14,7 +14,7 @@ target_include_directories(bergamot-translator-worker
 target_compile_definitions(bergamot-translator-worker PRIVATE WASM_BINDINGS)
 target_compile_options(bergamot-translator-worker PRIVATE ${WASM_COMPILE_FLAGS})
 
-set(LINKER_FLAGS "--bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1")
+set(LINKER_FLAGS "--bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1 -s EXPORTED_RUNTIME_METHODS=[addOnPreMain]")
 if (NOT PACKAGE_DIR STREQUAL "")
   get_filename_component(REALPATH_PACKAGE_DIR ${PACKAGE_DIR} REALPATH BASE_DIR ${CMAKE_BINARY_DIR})
   set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${REALPATH_PACKAGE_DIR}@/")

From 331216e01794e44a4258297a3ee76c6e7799fd86 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 10 May 2021 15:24:27 +0200
Subject: [PATCH 228/442] Enable Debugging information in wasm module builds

 - Added "-g2" flag furing linking step
---
 wasm/CMakeLists.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index d98705b7c..4962da045 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -14,7 +14,7 @@ target_include_directories(bergamot-translator-worker
 target_compile_definitions(bergamot-translator-worker PRIVATE WASM_BINDINGS)
 target_compile_options(bergamot-translator-worker PRIVATE ${WASM_COMPILE_FLAGS})
 
-set(LINKER_FLAGS "--bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1 -s EXPORTED_RUNTIME_METHODS=[addOnPreMain]")
+set(LINKER_FLAGS "-g2 --bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1 -s EXPORTED_RUNTIME_METHODS=[addOnPreMain]")
 if (NOT PACKAGE_DIR STREQUAL "")
   get_filename_component(REALPATH_PACKAGE_DIR ${PACKAGE_DIR} REALPATH BASE_DIR ${CMAKE_BINARY_DIR})
   set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${REALPATH_PACKAGE_DIR}@/")

From 9f78985e4567ff2c5d300c6bc21353d8d7f94501 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 10 May 2021 19:09:18 +0200
Subject: [PATCH 229/442] JS bindings for vocabularies as bytes

---
 wasm/bindings/TranslationModelBindings.cpp | 28 ++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/wasm/bindings/TranslationModelBindings.cpp b/wasm/bindings/TranslationModelBindings.cpp
index 41b9c2e61..3fd2c8bf6 100644
--- a/wasm/bindings/TranslationModelBindings.cpp
+++ b/wasm/bindings/TranslationModelBindings.cpp
@@ -24,12 +24,36 @@ EMSCRIPTEN_BINDINGS(aligned_memory) {
     .function("size", &marian::bergamot::AlignedMemory::size)
 	  .function("getByteArrayView", &getByteArrayView)
     ;
+
+    register_vector<marian::bergamot::AlignedMemory*>("AlignedMemoryList");
+}
+
+std::vector<std::shared_ptr<marian::bergamot::AlignedMemory>>
+prepareVocabsSmartMemories(std::vector<marian::bergamot::AlignedMemory*>& vocabsMemories) {
+  auto sourceVocabMemory = std::make_shared<marian::bergamot::AlignedMemory>(std::move(*(vocabsMemories[0])));
+  std::vector<std::shared_ptr<marian::bergamot::AlignedMemory>> vocabsSmartMemories;
+  vocabsSmartMemories.push_back(sourceVocabMemory);
+  // When source and target vocab files are same, only one memory object is passed in vocabsMemories
+  // to avoid double memory allocation for the same file. However, the constructor of the TranslationModel
+  // class still expects 2 entries where each entry has the shared ownership of a single AlignedMemory object.
+  if (vocabsMemories.size() == 2) {
+    auto targetVocabMemory = std::make_shared<marian::bergamot::AlignedMemory>(std::move(*(vocabsMemories[1])));
+    vocabsSmartMemories.push_back(std::move(targetVocabMemory));
+  }
+  else {
+    vocabsSmartMemories.push_back(sourceVocabMemory);
+  }
+  return vocabsSmartMemories;
 }
 
 TranslationModel* TranslationModelFactory(const std::string &config,
                                           marian::bergamot::AlignedMemory* modelMemory,
-                                          marian::bergamot::AlignedMemory* shortlistMemory) {
-  return new TranslationModel(config, std::move(*modelMemory), std::move(*shortlistMemory));
+                                          marian::bergamot::AlignedMemory* shortlistMemory,
+                                          std::vector<marian::bergamot::AlignedMemory*> uniqueVocabsMemories) {
+  return new TranslationModel(config,
+                              std::move(*modelMemory),
+                              std::move(*shortlistMemory),
+                              std::move(prepareVocabsSmartMemories(uniqueVocabsMemories)));
 }
 
 EMSCRIPTEN_BINDINGS(translation_model) {

From 5025285e5c311e70dcce0a5261b0912785447563 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 10 May 2021 19:09:44 +0200
Subject: [PATCH 230/442] Updated wasm test page to pass vocabulary files as
 bytes

---
 wasm/test_page/bergamot.html | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index 95ae325b2..d150af618 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -113,10 +113,7 @@
 `;
 */
 
-const modelConfigWithoutModelAndShortList = `vocabs:
-  - /${languagePair}/vocab.${vocabLanguagePair}.spm
-  - /${languagePair}/vocab.${vocabLanguagePair}.spm
-beam-size: 1
+const modelConfig = `beam-size: 1
 normalize: 1.0
 word-penalty: 0
 max-length-break: 128
@@ -136,9 +133,15 @@
 // gemm-precision: int8shiftAlphaAll
 
     const modelFile = `models/${languagePair}/model.${languagePair}.intgemm.alphas.bin`;
-    console.debug("modelFile: ", modelFile);
     const shortlistFile = `models/${languagePair}/lex.50.50.${languagePair}.s2t.bin`;
+    const vocabFiles = [`models/${languagePair}/vocab.${vocabLanguagePair}.spm`,
+                        `models/${languagePair}/vocab.${vocabLanguagePair}.spm`];
+
+    const uniqueVocabFiles = new Set(vocabFiles);
+    console.debug("modelFile: ", modelFile);
     console.debug("shortlistFile: ", shortlistFile);
+    console.debug("No. of unique vocabs: ", uniqueVocabFiles.size);
+    uniqueVocabFiles.forEach(item => console.debug("unique vocabFile: ", item));
 
     try {
       // Download the files as buffers from the given urls
@@ -146,16 +149,23 @@
       const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]);
       const modelBuffer = downloadedBuffers[0];
       const shortListBuffer = downloadedBuffers[1];
+
+      const downloadedVocabBuffers = [];
+      for (let item of uniqueVocabFiles.values()) {
+        downloadedVocabBuffers.push(await downloadAsArrayBuffer(item));
+      }
       log(`${languagePair} file download took ${(Date.now() - start) / 1000} secs`);
 
       // Construct AlignedMemory objects with downloaded buffers
       var alignedModelMemory = constructAlignedMemoryFromBuffer(modelBuffer, 256);
       var alignedShortlistMemory = constructAlignedMemoryFromBuffer(shortListBuffer, 64);
+      var alignedVocabsMemoryList = new Module.AlignedMemoryList;
+      downloadedVocabBuffers.forEach(item => alignedVocabsMemoryList.push_back(constructAlignedMemoryFromBuffer(item, 64)));
 
       // Instantiate the TranslationModel
       if (translationModel) translationModel.delete();
-      console.debug("Creating TranslationModel with config:", modelConfigWithoutModelAndShortList);
-      translationModel = new Module.TranslationModel(modelConfigWithoutModelAndShortList, alignedModelMemory, alignedShortlistMemory);
+      console.debug("Creating TranslationModel with config:", modelConfig);
+      translationModel = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
     } catch (error) {
       log(error);
     }

From d7cb859ab758dd0b04f1dc8ddeda03216a319c2b Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 11 May 2021 15:20:16 +0200
Subject: [PATCH 231/442] Refactoring TranslationModelBindings class

 - typdef AlignedMemory for code readability

 - Added documentation for one of the binding function
---
 wasm/bindings/TranslationModelBindings.cpp | 32 ++++++++++++----------
 1 file changed, 17 insertions(+), 15 deletions(-)

diff --git a/wasm/bindings/TranslationModelBindings.cpp b/wasm/bindings/TranslationModelBindings.cpp
index 3fd2c8bf6..4ee926559 100644
--- a/wasm/bindings/TranslationModelBindings.cpp
+++ b/wasm/bindings/TranslationModelBindings.cpp
@@ -13,31 +13,33 @@ using namespace emscripten;
 
 typedef marian::bergamot::Service TranslationModel;
 typedef marian::bergamot::Response TranslationResult;
+typedef marian::bergamot::AlignedMemory AlignedMemory;
 
-val getByteArrayView(marian::bergamot::AlignedMemory& alignedMemory) {
+val getByteArrayView(AlignedMemory& alignedMemory) {
   return val(typed_memory_view(alignedMemory.size(), alignedMemory.as<char>()));
 }
 
 EMSCRIPTEN_BINDINGS(aligned_memory) {
-  class_<marian::bergamot::AlignedMemory>("AlignedMemory")
+  class_<AlignedMemory>("AlignedMemory")
     .constructor<std::size_t, std::size_t>()
-    .function("size", &marian::bergamot::AlignedMemory::size)
+    .function("size", &AlignedMemory::size)
 	  .function("getByteArrayView", &getByteArrayView)
     ;
 
-    register_vector<marian::bergamot::AlignedMemory*>("AlignedMemoryList");
+    register_vector<AlignedMemory*>("AlignedMemoryList");
 }
 
-std::vector<std::shared_ptr<marian::bergamot::AlignedMemory>>
-prepareVocabsSmartMemories(std::vector<marian::bergamot::AlignedMemory*>& vocabsMemories) {
-  auto sourceVocabMemory = std::make_shared<marian::bergamot::AlignedMemory>(std::move(*(vocabsMemories[0])));
-  std::vector<std::shared_ptr<marian::bergamot::AlignedMemory>> vocabsSmartMemories;
+// When source and target vocab files are same, only one memory object is passed from JS to
+// avoid allocating memory twice for the same file. However, the constructor of the TranslationModel
+// class still expects 2 entries in this case, where each entry has the shared ownership of the
+// same AlignedMemory object. This function prepares these smart pointer based AlignedMemory objects
+// for unique AlignedMemory objects passed from JS.
+std::vector<std::shared_ptr<AlignedMemory>> prepareVocabsSmartMemories(std::vector<AlignedMemory*>& vocabsMemories) {
+  auto sourceVocabMemory = std::make_shared<AlignedMemory>(std::move(*(vocabsMemories[0])));
+  std::vector<std::shared_ptr<AlignedMemory>> vocabsSmartMemories;
   vocabsSmartMemories.push_back(sourceVocabMemory);
-  // When source and target vocab files are same, only one memory object is passed in vocabsMemories
-  // to avoid double memory allocation for the same file. However, the constructor of the TranslationModel
-  // class still expects 2 entries where each entry has the shared ownership of a single AlignedMemory object.
   if (vocabsMemories.size() == 2) {
-    auto targetVocabMemory = std::make_shared<marian::bergamot::AlignedMemory>(std::move(*(vocabsMemories[1])));
+    auto targetVocabMemory = std::make_shared<AlignedMemory>(std::move(*(vocabsMemories[1])));
     vocabsSmartMemories.push_back(std::move(targetVocabMemory));
   }
   else {
@@ -47,9 +49,9 @@ prepareVocabsSmartMemories(std::vector<marian::bergamot::AlignedMemory*>& vocabs
 }
 
 TranslationModel* TranslationModelFactory(const std::string &config,
-                                          marian::bergamot::AlignedMemory* modelMemory,
-                                          marian::bergamot::AlignedMemory* shortlistMemory,
-                                          std::vector<marian::bergamot::AlignedMemory*> uniqueVocabsMemories) {
+                                          AlignedMemory* modelMemory,
+                                          AlignedMemory* shortlistMemory,
+                                          std::vector<AlignedMemory*> uniqueVocabsMemories) {
   return new TranslationModel(config,
                               std::move(*modelMemory),
                               std::move(*shortlistMemory),

From 8a6c7b44a30f4458a2fb414a1c98f4cf068c95f3 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 12 May 2021 08:59:32 +0200
Subject: [PATCH 232/442] Avoid packaging vocab files into wasm binary in CI
 builds

 - We don't need to package vocab files into wasm binary any more
   as a sync with upstream enabled passing vocabs as bytes
---
 .circleci/config.yml |  2 +-
 build-wasm.sh        | 20 +-------------------
 2 files changed, 2 insertions(+), 20 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 6465469b1..69ae35686 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -19,7 +19,7 @@ jobs:
           working_directory: build-wasm
           command: |
             ls -all bergamot*
-            if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null && ls bergamot*.data &>/dev/null
+            if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
             then
               echo "Artifacts Successfully Generated"
             else
diff --git a/build-wasm.sh b/build-wasm.sh
index fff92a4b2..bdc00d501 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -37,24 +37,6 @@ if [ "$EMSDK" == "" ]; then
   source ./emsdk/emsdk_env.sh
 fi
 
-# 3. Download models (required to perform inference using build artifacts)
-if [ ! -d "bergamot-models" ]; then
-  git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
-else
-  cd bergamot-models
-  git fetch
-  # Only pull if necessary
-  if [ $(git rev-parse HEAD) != $(git rev-parse @{u}) ]; then
-    git pull --ff-only
-  fi
-  cd -
-fi
-mkdir -p models
-rm -rf models/*
-cp -rf bergamot-models/prod/* models
-gunzip models/*/*
-find models \( -type f -name "model*" -or -type f -name "lex*" \) -delete
-
 # 4. Compile
 #     1. Create a folder where you want to build all the artifacts (`build-wasm` in this case)
 if [ ! -d "build-wasm" ]; then
@@ -63,7 +45,7 @@ fi
 cd build-wasm
 
 #     2. Compile the artifacts
-emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR="../models/" ../
+emcmake cmake -DCOMPILE_WASM=on ../
 emmake make -j3
 
 #     3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts

From e0b9bad0581963f4ae1f953f1b52de5619672384 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 12 May 2021 14:39:23 +0200
Subject: [PATCH 233/442] Updated wasm README to update for passing vocabs as
 bytes

 - Updated Using JS APIs section to pass vocabs as bytes
---
 wasm/README.md | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/wasm/README.md b/wasm/README.md
index 337ae1b28..eceac152b 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -7,8 +7,8 @@ Please note that [Using JS APIs](#Using-JS-APIs) and [Demo](#Demo) section below
 
 ```bash
 cd test_page
-mkdir models
 git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
+mkdir models
 cp -rf bergamot-models/prod/* models
 gunzip models/*/*
 ```
@@ -18,10 +18,7 @@ gunzip models/*/*
 ```js
 // The model configuration as YAML formatted string. For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
 // This example captures some of the most relevant options
-const modelConfig = `vocabs:
-  - /esen/vocab.esen.spm
-  - /esen/vocab.esen.spm
-beam-size: 1
+const modelConfig = `beam-size: 1
 normalize: 1.0
 word-penalty: 0
 max-length-break: 128
@@ -35,19 +32,31 @@ quiet-translation: true
 gemm-precision: int8shift
 `;
 
-// Download model and shortlist files and read them into buffers
+// Download model, shortlist and vocabulary files and read them into buffers
 const modelFile = `models/esen/model.esen.intgemm.alphas.bin`;
 const shortlistFile = `models/esen/lex.50.50.esen.s2t.bin`;
-const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]); // Please refer to bergamot.html in test_page folder for this function
+const vocabFiles = [`models/${languagePair}/vocab.${vocabLanguagePair}.spm`,
+                    `models/${languagePair}/vocab.${vocabLanguagePair}.spm`];
+const uniqueVocabFiles = new Set(vocabFiles);
+
+// Please refer to bergamot.html in test_page folder for downloadAsArrayBuffer function
+const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]);
+const downloadedVocabBuffers = [];
+for (let item of uniqueVocabFiles.values()) {
+  downloadedVocabBuffers.push(await downloadAsArrayBuffer(item));
+}
+
 const modelBuffer = downloadedBuffers[0];
 const shortListBuffer = downloadedBuffers[1];
 
 // Construct AlignedMemory instances from the buffers
 var alignedModelMemory = constructAlignedMemoryFromBuffer(modelBuffer, 256); // Please refer to bergamot.html in test_page folder for this function
 var alignedShortlistMemory = constructAlignedMemoryFromBuffer(shortListBuffer, 64); // Please refer to bergamot.html in test_page folder for this function
+var alignedVocabsMemoryList = new Module.AlignedMemoryList;
+downloadedVocabBuffers.forEach(item => alignedVocabsMemoryList.push_back(constructAlignedMemoryFromBuffer(item, 64)));
 
 // Instantiate the TranslationModel
-const model = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory);
+const model = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
 
 // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
 const request = new Module.TranslationRequest();

From 0189500160eaac8ae7e8a7b8d03ae91c98063684 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 12 May 2021 14:44:33 +0200
Subject: [PATCH 234/442] Updated README to remove packaging steps for wasm
 compilation

 - We don't need to package model, shortlist or vocab files into wasm
   binary at build time
---
 README.md | 94 +++++++++++++++++--------------------------------------
 1 file changed, 28 insertions(+), 66 deletions(-)

diff --git a/README.md b/README.md
index 456af70ce..f48c981d3 100644
--- a/README.md
+++ b/README.md
@@ -5,85 +5,47 @@ Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.
 ## Build Instructions
 
 ### Build Natively
-1. Clone the repository using these instructions:
-    ```bash
-    git clone https://github.com/browsermt/bergamot-translator
-    cd bergamot-translator
-    ```
-2. Compile
+Create a folder where you want to build all the artifacts (`build-native` in this case) and compile
 
-    Create a folder where you want to build all the artifacts (`build-native` in this case) and compile in that folder
-    ```bash
-    mkdir build-native
-    cd build-native
-    cmake ../
-    make -j
-    ```
+```bash
+mkdir build-native
+cd build-native
+cmake ../
+make -j3
+```
 
 ### Build WASM
-#### Compiling for the first time
+#### Prerequisite
+
+Building on wasm requires Emscripten toolchain. It can be downloaded and installed using following instructions:
+
+* Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
+* Enter the cloned directory: `cd emsdk`
+* Install the lastest sdk tools: `./emsdk install latest`
+* Activate the latest sdk tools: `./emsdk activate latest`
+* Activate path variables: `source ./emsdk_env.sh`
 
-1. Download and Install Emscripten using following instructions
-    * Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
-    * Enter the cloned directory: `cd emsdk`
-    * Install the lastest sdk tools: `./emsdk install latest`
-    * Activate the latest sdk tools: `./emsdk activate latest`
-    * Activate path variables: `source ./emsdk_env.sh`
+#### <a name="Compile"></a> Compile
 
-2. Clone the repository using these instructions:
+1. Create a folder where you want to build all the artifacts (`build-wasm` in this case) and compile
     ```bash
-    git clone https://github.com/browsermt/bergamot-translator
-    cd bergamot-translator
+    mkdir build-wasm
+    cd build-wasm
+    emcmake cmake -DCOMPILE_WASM=on ../
+    emmake make -j3
     ```
 
-3. Download files (only required if you want to perform inference using build artifacts)
+    The wasm artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
 
-    It packages the vocabulary files into wasm binary, which is required only if you want to perform inference.
-    The compilation commands will preload these files in Emscripten’s virtual file system.
-
-    If you want to package bergamot project specific files, please follow these instructions:
+2. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
     ```bash
-    git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
-    mkdir models
-    cp -rf bergamot-models/prod/* models
-    gunzip models/*/*
-    find models \( -type f -name "model*" -or -type f -name "lex*" \) -delete
+    bash ../wasm/patch-artifacts-enable-wormhole.sh
     ```
 
-4. Compile
-    1. Create a folder where you want to build all the artefacts (`build-wasm` in this case)
-        ```bash
-        mkdir build-wasm
-        cd build-wasm
-        ```
-
-    2. Compile the artefacts
-        * If you want to package files into wasm binary then execute following commands (Replace `FILES_TO_PACKAGE` with the
-        directory containing all the files to be packaged)
-
-            ```bash
-            emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=FILES_TO_PACKAGE ../
-            emmake make -j
-            ```
-            e.g. If you want to package bergamot project specific files (downloaded using step 3 above) then
-            replace `FILES_TO_PACKAGE` with `../models`
-
-        * If you don't want to package any file into wasm binary then execute following commands:
-            ```bash
-            emcmake cmake -DCOMPILE_WASM=on ../
-            emmake make -j
-            ```
-
-        The wasm artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
-
-    3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
-        ```bash
-        bash ../wasm/patch-artifacts-enable-wormhole.sh
-        ```
-
 #### Recompiling
-As long as you don't update any submodule, just follow steps in `4.ii` and `4.iii` to recompile.\
-If you update a submodule, execute following command before executing steps in `4.ii` and `4.iii` to recompile.
+As long as you don't update any submodule, just follow [Compile](#Compile) steps.\
+If you update a submodule, execute following command in repository root folder before executing
+[Compile](#Compile) steps.
 ```bash
 git submodule update --init --recursive
 ```

From 6c063c607ee5a5ffc00a2fe64a6c32164699ceab Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 12 May 2021 14:46:02 +0200
Subject: [PATCH 235/442] Updated CMakeLists.txt to remove packaging steps for
 wasm compilation

 - Removed PACKAGE_DIR cmake option
 - Removed Workerfs, FORCE_FILESYSTEM=1 in wasm builds
   -- File system support is not needed any more (since model,
     shortlist and vocabs are being passed as bytes now)
---
 CMakeLists.txt      | 2 --
 wasm/CMakeLists.txt | 9 +--------
 2 files changed, 1 insertion(+), 10 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ef64863ca..332aed16e 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -42,8 +42,6 @@ option(COMPILE_WASM "Compile for WASM" OFF)
 cmake_dependent_option(USE_WASM_COMPATIBLE_SOURCE "Use wasm compatible sources" OFF "NOT COMPILE_WASM" ON)
 option(COMPILE_TESTS "Compile bergamot-tests" OFF)
 
-SET(PACKAGE_DIR "" CACHE STRING "Directory including all the files to be packaged (pre-loaded) in wasm builds")
-
 # Set 3rd party submodule specific cmake options for this project
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
 SET(USE_SENTENCEPIECE ON CACHE BOOL "Download and compile SentencePiece")
diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 4962da045..7feef7512 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -14,14 +14,7 @@ target_include_directories(bergamot-translator-worker
 target_compile_definitions(bergamot-translator-worker PRIVATE WASM_BINDINGS)
 target_compile_options(bergamot-translator-worker PRIVATE ${WASM_COMPILE_FLAGS})
 
-set(LINKER_FLAGS "-g2 --bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s FORCE_FILESYSTEM=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1 -s EXPORTED_RUNTIME_METHODS=[addOnPreMain]")
-if (NOT PACKAGE_DIR STREQUAL "")
-  get_filename_component(REALPATH_PACKAGE_DIR ${PACKAGE_DIR} REALPATH BASE_DIR ${CMAKE_BINARY_DIR})
-  set(LINKER_FLAGS "${LINKER_FLAGS} --preload-file ${REALPATH_PACKAGE_DIR}@/")
-endif()
-
-# Enable worker file system
-set(LINKER_FLAGS "${LINKER_FLAGS} -lworkerfs.js")
+set(LINKER_FLAGS "-g2 --bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1 -s EXPORTED_RUNTIME_METHODS=[addOnPreMain]")
 
 # Avoid node.js-code in emscripten glue-code
 set(LINKER_FLAGS "${LINKER_FLAGS} -s ENVIRONMENT=web,worker")

From 6c7e6156ab2816310d90e5f4d9489f96f10decb2 Mon Sep 17 00:00:00 2001
From: Qianqian Zhu <qianqian.zhu@hotmail.com>
Date: Thu, 13 May 2021 13:18:08 +0100
Subject: [PATCH 236/442] Bundle AlignedMemory inputs with MemoryBundle (#147)

---
 app/service-cli.cpp                        | 12 ++---
 src/translator/byte_array_util.cpp         |  8 +++
 src/translator/byte_array_util.h           |  1 +
 src/translator/definitions.h               | 36 ++++++++++++++
 src/translator/service.cpp                 |  9 ++--
 src/translator/service.h                   | 58 +++++++---------------
 wasm/bindings/TranslationModelBindings.cpp | 16 ++++--
 7 files changed, 82 insertions(+), 58 deletions(-)

diff --git a/app/service-cli.cpp b/app/service-cli.cpp
index 0e958d6a7..fbf013161 100644
--- a/app/service-cli.cpp
+++ b/app/service-cli.cpp
@@ -16,19 +16,15 @@ int main(int argc, char *argv[]) {
   auto cp = marian::bergamot::createConfigParser();
   auto options = cp.parseOptions(argc, argv, true);
 
-  // Prepare memories for model and shortlist
-  marian::bergamot::AlignedMemory modelBytes, shortlistBytes;
-  std::vector<std::shared_ptr<marian::bergamot::AlignedMemory>> vocabsBytes;
+  // Prepare memories for bytearrays (including model, shortlist and vocabs)
+  marian::bergamot::MemoryBundle memoryBundle;
 
   if (options->get<bool>("check-bytearray")) {
     // Load legit values into bytearrays.
-    modelBytes = marian::bergamot::getModelMemoryFromConfig(options);
-    shortlistBytes = marian::bergamot::getShortlistMemoryFromConfig(options);
-    marian::bergamot::getVocabsMemoryFromConfig(options, vocabsBytes);
+    memoryBundle = marian::bergamot::getMemoryBundleFromConfig(options);
   }
 
-  marian::bergamot::Service service(options, std::move(modelBytes),
-                                    std::move(shortlistBytes), std::move(vocabsBytes));
+  marian::bergamot::Service service(options, std::move(memoryBundle));
 
   // Read a large input text blob from stdin
   std::ostringstream std_input;
diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index 00beaa62e..69564d21b 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -117,5 +117,13 @@ void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
   }
 }
 
+MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options){
+  MemoryBundle memoryBundle;
+  memoryBundle.model = getModelMemoryFromConfig(options);
+  memoryBundle.shortlist = getShortlistMemoryFromConfig(options);
+  getVocabsMemoryFromConfig(options, memoryBundle.vocabs);
+  return memoryBundle;
+}
+
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/byte_array_util.h b/src/translator/byte_array_util.h
index 3cbf3d339..14c79b3b5 100644
--- a/src/translator/byte_array_util.h
+++ b/src/translator/byte_array_util.h
@@ -10,5 +10,6 @@ AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options)
 void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
                                std::vector<std::shared_ptr<AlignedMemory>>& vocabMemories);
 bool validateBinaryModel(const AlignedMemory& model, uint64_t fileSize);
+MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options);
 } // namespace bergamot
 } // namespace marian
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index 58fd4b386..175397d5f 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -15,6 +15,42 @@ typedef std::vector<Segment> Segments;
 /// Shortcut to AlignedVector<char> for byte arrays
 typedef AlignedVector<char> AlignedMemory;
 
+/// Memory bundle for all byte-arrays.
+/// Can be a set/subset of model, shortlist, vocabs and ssplitPrefixFile bytes.
+struct MemoryBundle {
+  AlignedMemory model;  ///< Byte-array of model (aligned to 256)
+  AlignedMemory shortlist;  ///< Byte-array of shortlist (aligned to 64)
+
+  /// Vector of vocabulary memories (aligned to 64).
+  /// If two vocabularies are the same (based on the filenames), two entries (shared
+  /// pointers) will be generated which share the same AlignedMemory object.
+  std::vector<std::shared_ptr<AlignedMemory>> vocabs;
+
+  /// @todo Not implemented yet
+  AlignedMemory ssplitPrefixFile;
+
+  MemoryBundle() = default;
+
+  MemoryBundle(MemoryBundle &&from){
+    model = std::move(from.model);
+    shortlist = std::move(from.shortlist);
+    vocabs = std::move(vocabs);
+    ssplitPrefixFile = std::move(from.ssplitPrefixFile);
+  }
+
+  MemoryBundle &operator=(MemoryBundle &&from) {
+    model = std::move(from.model);
+    shortlist = std::move(from.shortlist);
+    vocabs = std::move(vocabs);
+    ssplitPrefixFile = std::move(from.ssplitPrefixFile);
+    return *this;
+  }
+
+  // Delete copy constructors
+  MemoryBundle(const MemoryBundle&) = delete;
+  MemoryBundle& operator=(const MemoryBundle&) = delete;
+};
+
 } // namespace bergamot
 } // namespace marian
 
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 385a2a598..16c474393 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -41,14 +41,13 @@ loadVocabularies(marian::Ptr<marian::Options> options,
 namespace marian {
 namespace bergamot {
 
-Service::Service(Ptr<Options> options, AlignedMemory modelMemory, AlignedMemory shortlistMemory,
-                 std::vector<std::shared_ptr<AlignedMemory>> vocabMemories)
+Service::Service(Ptr<Options> options, MemoryBundle memoryBundle)
     : requestId_(0), options_(options),
-      vocabs_(std::move(loadVocabularies(options, std::move(vocabMemories)))),
+      vocabs_(std::move(loadVocabularies(options, std::move(memoryBundle.vocabs)))),
       text_processor_(vocabs_, options), batcher_(options),
       numWorkers_(options->get<int>("cpu-threads")),
-      modelMemory_(std::move(modelMemory)),
-      shortlistMemory_(std::move(shortlistMemory))
+      modelMemory_(std::move(memoryBundle.model)),
+      shortlistMemory_(std::move(memoryBundle.shortlist))
 #ifndef WASM_COMPATIBLE_SOURCE
       // 0 elements in PCQueue is illegal and can lead to failures. Adding a
       // guard to have at least one entry allocated. In the single-threaded
diff --git a/src/translator/service.h b/src/translator/service.h
index 721d436a1..9d0a67d4c 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -55,53 +55,29 @@ namespace bergamot {
 /// // Do things with response.
 /// ```
 ///
-/// Optionally Service can be initialized by also passing model memory for
-/// purposes of efficiency (which defaults to nullpointer and then reads from
+/// Optionally Service can be initialized by also passing bytearray memories
+/// for purposes of efficiency (which defaults to empty and then reads from
 /// file supplied through config).
 ///
 class Service {
 
 public:
+  /// Construct Service from Marian options. If memoryBundle is empty, Service is
+  /// initialized from file-based loading. Otherwise, Service is initialized from
+  /// the given bytearray memories.
   /// @param options Marian options object
-  /// @param modelMemory byte array (aligned to 256!!!) that contains the bytes
-  /// of a model.bin.
-  /// @param shortlistMemory byte array of shortlist (aligned to 64)
-  /// @param vocabMemories vector of vocabulary memories (aligned to 64)
-  explicit Service(Ptr<Options> options, AlignedMemory modelMemory,
-                   AlignedMemory shortlistMemory,
-                   std::vector<std::shared_ptr<AlignedMemory>> vocabMemories);
-
-  /// Construct Service purely from Options. This expects options which
-  /// marian-decoder expects to be set for loading model shortlist and
-  /// vocabularies from files in addition to parameters that set unset desired
-  /// features (e.g: alignments, quality-scores).
-  ///
-  /// This is equivalent to a call to:
-  /// ```cpp
-  ///    Service(options, AlignedMemory(), AlignedMemory(), {})
-  /// ```
-  /// wherein empty memory is passed and internal flow defaults to file-based
-  /// model, shortlist loading. AlignedMemory() corresponds to empty memory
-  explicit Service(Ptr<Options> options)
-      : Service(options, AlignedMemory(), AlignedMemory(), {}) {}
-
-  /// Construct Service from a string configuration.
-  /// @param [in] config string parsable as YAML expected to adhere with marian
-  /// config
-  /// @param [in] modelMemory byte array (aligned to 256!!!) that contains the
-  /// bytes of a model.bin. Optional. AlignedMemory() corresponds to empty memory
-  /// @param [in] shortlistMemory byte array of shortlist (aligned to 64). Optional.
-  /// @param [in] vocabMemories vector of vocabulary memories (aligned to 64). Optional.
-  /// If two vocabularies are the same (based on the filenames), two entries (shared
-  /// pointers) will be generated which share the same AlignedMemory object.
-  explicit Service(const std::string &config,
-                   AlignedMemory modelMemory = AlignedMemory(),
-                   AlignedMemory shortlistMemory = AlignedMemory(),
-                   std::vector<std::shared_ptr<AlignedMemory>> vocabsMemories = {})
-      : Service(parseOptions(config, /*validate=*/false),
-                std::move(modelMemory),
-                std::move(shortlistMemory),
-                std::move(vocabsMemories)) {}
+  /// @param memoryBundle holds all byte-array memories. Can be a set/subset of
+  /// model, shortlist, vocabs and ssplitPrefixFile bytes. Optional.
+  explicit Service(Ptr<Options> options, MemoryBundle memoryBundle={});
+
+  /// Construct Service from a string configuration. If memoryBundle is empty, Service is
+  /// initialized from file-based loading. Otherwise, Service is initialized from
+  /// the given bytearray memories.
+  /// @param [in] config string parsable as YAML expected to adhere with marian config
+  /// @param [in] memoryBundle holds all byte-array memories. Can be a set/subset of
+  /// model, shortlist, vocabs and ssplitPrefixFile bytes. Optional.
+  explicit Service(const std::string &config, MemoryBundle memoryBundle={})
+      : Service(parseOptions(config, /*validate=*/false), std::move(memoryBundle)) {}
 
   /// Explicit destructor to clean up after any threads initialized in
   /// asynchronous operation mode.
diff --git a/wasm/bindings/TranslationModelBindings.cpp b/wasm/bindings/TranslationModelBindings.cpp
index 4ee926559..1db740149 100644
--- a/wasm/bindings/TranslationModelBindings.cpp
+++ b/wasm/bindings/TranslationModelBindings.cpp
@@ -48,14 +48,22 @@ std::vector<std::shared_ptr<AlignedMemory>> prepareVocabsSmartMemories(std::vect
   return vocabsSmartMemories;
 }
 
+marian::bergamot::MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory,
+                                                   AlignedMemory* shortlistMemory,
+                                                   std::vector<AlignedMemory*> uniqueVocabsMemories){
+  marian::bergamot::MemoryBundle memoryBundle;
+  memoryBundle.model = std::move(*modelMemory);
+  memoryBundle.shortlist = std::move(*shortlistMemory);
+  memoryBundle.vocabs = std::move(prepareVocabsSmartMemories(uniqueVocabsMemories));
+
+  return memoryBundle;
+}
+
 TranslationModel* TranslationModelFactory(const std::string &config,
                                           AlignedMemory* modelMemory,
                                           AlignedMemory* shortlistMemory,
                                           std::vector<AlignedMemory*> uniqueVocabsMemories) {
-  return new TranslationModel(config,
-                              std::move(*modelMemory),
-                              std::move(*shortlistMemory),
-                              std::move(prepareVocabsSmartMemories(uniqueVocabsMemories)));
+  return new TranslationModel(config, std::move(prepareMemoryBundle(modelMemory, shortlistMemory, uniqueVocabsMemories)));
 }
 
 EMSCRIPTEN_BINDINGS(translation_model) {

From 77424a3df155a03c61338287a11b3c6c28815681 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 17 May 2021 11:42:47 +0100
Subject: [PATCH 237/442] Enabling ccache on github builds for Ubuntu (#95)

* CI Changes to add tiny regression tests

* Adding an inspect cache step

* Removing ccache, pursue in another

* Incorporating Nick's changes through submodule merge

* Submodule now points to master

* Restoring ccache enabled workflow file

* Restoring ccache enabled CMakeLists

* cache -> ccache typo fix

* Moving CCACHE setup to GitHub runner file

* Find also uses CCACHE dir

* Updating CMakeLists not to override env

* Cache compiler binary's contents

* Changing a few names to trigger new build; Testing cache looks fun

* USE_CCACHE=on, -L for inspection

* Adding a ccache_cmd, but will only use in next commit

* Using ccache_cmd

* Removing "

* Adding compiler hash script

* Bunch of absolute paths

* GITHUB_WORKSPACE typo

* Nah, I'll keep -L and trigger another build

* Trying something with compiler hash on cache key backup as well

* builtin, bash it seems

* Empty commit #1

* Move ccache stats to after compile

* Reshuffling ccache vars

* No comments

* Updates to Github output set syntax

* Empty Commit 1

* Empty Commit 2

* Empty commit 3

* /bin/bash -> bash; ccache_cmd for consistency

* Adding ccache -s before and after build

* Adding comments to compiler-hash script

* Let's build cached and non-cached variants together for comparison

* Fixing quotes, /bin/bash -> bash

* Minor var/env adjustment

* Adding ccache -z before the job

* Reverting CMakeLists.txt without CCACHE

* Switching to CMAKE_LANG_COMPILER_LAUNCHER instead of CMakeLists.txt rule

* 5G -> 1G cache size

* 1G -> 2G; Hyperparameter tuning
---
 .github/workflows/native-ubuntu.yml | 89 ++++++++++++++++++++++++++++-
 scripts/ci/compiler-hash.sh         | 35 ++++++++++++
 2 files changed, 122 insertions(+), 2 deletions(-)
 create mode 100644 scripts/ci/compiler-hash.sh

diff --git a/.github/workflows/native-ubuntu.yml b/.github/workflows/native-ubuntu.yml
index dc8016bae..563daf7d0 100644
--- a/.github/workflows/native-ubuntu.yml
+++ b/.github/workflows/native-ubuntu.yml
@@ -15,6 +15,8 @@ jobs:
           - name: "full-marian"
             os: ubuntu-latest
             gcc: 8
+            force_recache: false
+            ccache_cmd: "bash ${GITHUB_WORKSPACE}/scripts/ci/compiler-hash.sh %compiler%"
             cpu: 'ON'
             gpu: 'OFF'
             test_tags: ""
@@ -24,10 +26,14 @@ jobs:
               USE_WASM_COMPATIBLE_SOURCE: "OFF"
               COMPILE_SERVER: "OFF"
               COMPILE_EXAMPLES: "OFF"
+              CMAKE_C_COMPILER_LAUNCHER: "ccache"
+              CMAKE_CXX_COMPILER_LAUNCHER: "ccache"
 
           - name: "minimal-marian"
             os: ubuntu-latest
             gcc: 8
+            force_recache: false
+            ccache_cmd: "bash ${GITHUB_WORKSPACE}/scripts/ci/compiler-hash.sh %compiler%"
             cpu: 'ON'
             gpu: 'OFF'
             test_tags: "'#wasm'"
@@ -37,6 +43,42 @@ jobs:
               USE_WASM_COMPATIBLE_SOURCE: "ON"
               COMPILE_SERVER: "OFF"
               COMPILE_EXAMPLES: "OFF"
+              CMAKE_C_COMPILER_LAUNCHER: "ccache"
+              CMAKE_CXX_COMPILER_LAUNCHER: "ccache"
+
+          - name: "full-marian-force-recache"
+            os: ubuntu-latest
+            gcc: 8
+            force_recache: true
+            ccache_cmd: "bash ${GITHUB_WORKSPACE}/scripts/ci/compiler-hash.sh %compiler%"
+            cpu: 'ON'
+            gpu: 'OFF'
+            test_tags: ""
+            cmake: 
+              CMAKE_BUILD_TYPE: "Release"
+              COMPILE_TESTS: "ON"
+              USE_WASM_COMPATIBLE_SOURCE: "OFF"
+              COMPILE_SERVER: "OFF"
+              COMPILE_EXAMPLES: "OFF"
+              CMAKE_C_COMPILER_LAUNCHER: "ccache"
+              CMAKE_CXX_COMPILER_LAUNCHER: "ccache"
+
+          - name: "minimal-marian-force-recache"
+            os: ubuntu-latest
+            gcc: 8
+            force_recache: true
+            ccache_cmd: "bash ${GITHUB_WORKSPACE}/scripts/ci/compiler-hash.sh %compiler%"
+            cpu: 'ON'
+            gpu: 'OFF'
+            test_tags: "'#wasm'"
+            cmake:
+              CMAKE_BUILD_TYPE: "Release"
+              COMPILE_TESTS: "OFF" # Minimal marian has no sqlite support and COMPILE_TEST=ON fails.
+              USE_WASM_COMPATIBLE_SOURCE: "ON"
+              COMPILE_SERVER: "OFF"
+              COMPILE_EXAMPLES: "OFF"
+              CMAKE_C_COMPILER_LAUNCHER: "ccache"
+              CMAKE_CXX_COMPILER_LAUNCHER: "ccache"
 
 
     runs-on: ${{ matrix.os }}
@@ -57,7 +99,7 @@ jobs:
         sudo apt-get update 
         sudo apt-get install -y \
             libgoogle-perftools-dev libprotobuf-dev protobuf-compiler  \
-            libboost-all-dev g++-${{ matrix.gcc }} 
+            libboost-all-dev g++-${{ matrix.gcc }} ccache
 
     # https://software.intel.com/content/www/us/en/develop/articles/installing-intel-free-libs-and-python-apt-repo.html
     - name: Install MKL
@@ -68,6 +110,42 @@ jobs:
         sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
       if: matrix.cmake.USE_WASM_COMPATIBLE_SOURCE == 'OFF'
 
+    - name: Generate ccache_vars
+      id: ccache_vars
+      shell: bash
+      run: |
+          echo "::set-output name=hash::$(${{ matrix.ccache_cmd }})"
+          echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
+
+    - name: Setup ccache environment variables
+      run: | 
+        echo "CCACHE_COMPILERCHECK=${{ matrix.ccache_cmd }}" >> $GITHUB_ENV 
+        echo "CCACHE_BASE_DIR=${{ github.workspace }}" >> $GITHUB_ENV
+        echo "CCACHE_DIR=${{ github.workspace }}/.ccache" >> $GITHUB_ENV
+        echo "CCACHE_COMPRESS=true" >> $GITHUB_ENV
+        echo "CCACHE_COMPRESSLEVEL=6" >> $GITHUB_ENV
+        echo "CCACHE_MAXSIZE=2G" >> $GITHUB_ENV
+
+    - name: Setup ccache recache on
+      run: |
+        echo "CCACHE_RECACHE=" >> $GITHUB_ENV 
+      if: matrix.force_recache == true
+
+    - name: Cache-op for build-cache through ccache 
+      uses: actions/cache@v2
+      with:
+        path: ${{ env.CCACHE_DIR }}
+        key: ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+        restore-keys: |
+           ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}- 
+           ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}- 
+           ccache-${{ matrix.name }}- 
+
+    - name: Cache stats before build
+      run: |
+          ccache -s
+          ccache -z
+
     # Boost is installed on GitHub-hosted runners in a non-standard location
     # https://github.com/actions/virtual-environments/issues/687#issuecomment-610471671
     - name: Configure CMake
@@ -75,17 +153,24 @@ jobs:
         mkdir -p build
         cd build
         CC=/usr/bin/gcc-${{ matrix.gcc }} CXX=/usr/bin/g++-${{ matrix.gcc }} CUDAHOSTCXX=/usr/bin/g++-${{ matrix.gcc }} \
-        cmake .. \
+        cmake -L .. \
           -DCMAKE_BUILD_TYPE=${{ matrix.cmake.CMAKE_BUILD_TYPE }}\
           -DCOMPILE_TESTS=${{ matrix.cmake.COMPILE_TESTS }}\
           -DCOMPILE_EXAMPLES=${{ matrix.cmake.COMPILE_EXAMPLES }} \
           -DCOMPILE_SERVER=${{ matrix.cmake.COMPILE_SERVER }} \
           -DUSE_WASM_COMPATIBLE_SOURCE=${{ matrix.cmake.USE_WASM_COMPATIBLE_SOURCE }} \
+          -DCMAKE_C_COMPILER_LAUNCHER=${{ matrix.cmake.CMAKE_C_COMPILER_LAUNCHER}} \
+          -DCMAKE_CXX_COMPILER_LAUNCHER=${{ matrix.cmake.CMAKE_CXX_COMPILER_LAUNCHER}} 
+
 
     - name: Compile bergamot-translator
       working-directory: build
       run: make -j2
 
+    - name: Cache stats after build
+      run: |
+          ccache -s
+
     - name: Run unit tests
       working-directory: build
       run: make test
diff --git a/scripts/ci/compiler-hash.sh b/scripts/ci/compiler-hash.sh
new file mode 100644
index 000000000..a770dfd49
--- /dev/null
+++ b/scripts/ci/compiler-hash.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Uses the command from https://stackoverflow.com/a/9355840/4565794.
+# -v displays the commands executed to run compilation. Of this cc1 additional
+#    flags which contain the flags triggered by -march=native is what we need.
+# -E stop after preprocessing stage.
+
+# Output on a linux machine with gcc-8 looks as follows:
+
+# $ gcc -march=native -E -v - </dev/null 2>&1 | grep cc1
+#       /usr/lib/gcc/x86_64-linux-gnu/8/cc1 -E -quiet -v -imultiarch x86_64-linux-gnu
+#       - -march=skylake-avx512 -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3
+#       -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm
+#       -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mno-sgx -mbmi2 -mno-pconfig
+#       -mno-wbnoinvd -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm
+#       -mno-hle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave
+#       -mxsaveopt -mavx512f -mno-avx512er -mavx512cd -mno-avx512pf -mno-prefetchwt1
+#       -mclflushopt -mxsavec -mxsaves -mavx512dq -mavx512bw -mavx512vl
+#       -mno-avx512ifma -mno-avx512vbmi -mno-avx5124fmaps -mno-avx5124vnniw -mclwb
+#       -mno-mwaitx -mno-clzero -mpku -mno-rdpid -mno-gfni -mno-shstk
+#       -mno-avx512vbmi2 -mavx512vnni -mno-vaes -mno-vpclmulqdq -mno-avx512bitalg
+#       -mno-movdiri -mno-movdir64b --param l1-cache-size=32 --param
+#       l1-cache-line-size=64 --param l2-cache-size=28160 -mtune=skylake-avx512
+#       -fstack-protector-strong -Wformat -Wformat-security
+
+# The sha256sum of the output is computed, and stripped to the first 8
+# characters for use in ccache and github cache store key. Can effectively be
+# considered as a hash of the compiler version and the flags activated by
+# -march=native.
+
+COMPILER=$1
+
+$COMPILER -march=native -E -v - < /dev/null 2>&1 | grep cc1 \
+    | sha256sum | cut -c1-8
+

From 5bd1fc6b83d934198d6eed3b9bf50e751fa3d950 Mon Sep 17 00:00:00 2001
From: Qianqian Zhu <qianqian.zhu@hotmail.com>
Date: Mon, 17 May 2021 13:09:03 +0100
Subject: [PATCH 238/442] Refactor vocabs in Service (#143)

Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
---
 src/translator/batch_translator.cpp | 21 ++++----
 src/translator/batch_translator.h   |  5 +-
 src/translator/definitions.h        | 21 --------
 src/translator/response_builder.cpp |  3 +-
 src/translator/response_builder.h   |  7 +--
 src/translator/service.cpp          | 35 +------------
 src/translator/service.h            |  3 +-
 src/translator/text_processor.cpp   |  8 +--
 src/translator/text_processor.h     |  8 +--
 src/translator/vocabs.h             | 81 +++++++++++++++++++++++++++++
 10 files changed, 111 insertions(+), 81 deletions(-)
 create mode 100644 src/translator/vocabs.h

diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index c6271726a..b35c4cedc 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -10,11 +10,11 @@ namespace marian {
 namespace bergamot {
 
 BatchTranslator::BatchTranslator(DeviceId const device,
-                                 std::vector<Ptr<Vocab const>> &vocabs,
+                                 Vocabs &vocabs,
                                  Ptr<Options> options,
                                  const AlignedMemory* modelMemory,
                                  const AlignedMemory* shortlistMemory)
-    : device_(device), options_(options), vocabs_(&vocabs),
+    : device_(device), options_(options), vocabs_(vocabs),
     modelMemory_(modelMemory), shortlistMemory_(shortlistMemory) {}
 
 void BatchTranslator::initialize() {
@@ -22,17 +22,17 @@ void BatchTranslator::initialize() {
   bool check = options_->get<bool>("check-bytearray",false); // Flag holds whether validate the bytearray (model and shortlist)
   if (options_->hasAndNotEmpty("shortlist")) {
     int srcIdx = 0, trgIdx = 1;
-    bool shared_vcb = vocabs_->front() == vocabs_->back();
+    bool shared_vcb = vocabs_.sources().front() == vocabs_.target(); // vocabs_->sources().front() is invoked as we currently only support one source vocab
     if (shortlistMemory_->size() > 0 && shortlistMemory_->begin() != nullptr) {
       slgen_ = New<data::BinaryShortlistGenerator>(shortlistMemory_->begin(), shortlistMemory_->size(),
-                                                     vocabs_->front(), vocabs_->back(),
-                                                     srcIdx, trgIdx, shared_vcb, check);
+                                                   vocabs_.sources().front(), vocabs_.target(),
+                                                   srcIdx, trgIdx, shared_vcb, check);
     }
     else {
       // Changed to BinaryShortlistGenerator to enable loading binary shortlist file
       // This class also supports text shortlist file
-      slgen_ = New<data::BinaryShortlistGenerator>(options_, vocabs_->front(),
-                                                    vocabs_->back(), srcIdx,
+      slgen_ = New<data::BinaryShortlistGenerator>(options_, vocabs_.sources().front(),
+                                                    vocabs_.target(), srcIdx,
                                                     trgIdx, shared_vcb);
     }
   }
@@ -97,7 +97,7 @@ void BatchTranslator::translate(Batch &batch) {
   std::vector<Ptr<SubBatch>> subBatches;
   for (size_t j = 0; j < maxDims.size(); ++j) {
     subBatches.emplace_back(
-        New<SubBatch>(batchSize, maxDims[j], vocabs_->at(j)));
+        New<SubBatch>(batchSize, maxDims[j], vocabs_.sources().at(j)));
   }
 
   std::vector<size_t> words(maxDims.size(), 0);
@@ -116,9 +116,8 @@ void BatchTranslator::translate(Batch &batch) {
 
   auto corpus_batch = Ptr<CorpusBatch>(new CorpusBatch(subBatches));
   corpus_batch->setSentenceIds(sentenceIds);
-
-  auto trgVocab = vocabs_->back();
-  auto search = New<BeamSearch>(options_, scorers_, trgVocab);
+  
+  auto search = New<BeamSearch>(options_, scorers_, vocabs_.target());
 
   auto histories = std::move(search->search(graph_, corpus_batch));
   batch.completeBatch(histories);
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 761a53449..048ba7749 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -11,6 +11,7 @@
 #include "request.h"
 #include "translator/history.h"
 #include "translator/scorers.h"
+#include "vocabs.h"
 
 #ifndef WASM_COMPATIBLE_SOURCE
 #include "pcqueue.h"
@@ -34,7 +35,7 @@ class BatchTranslator {
    * @param modelMemory byte array (aligned to 256!!!) that contains the bytes of a model.bin. Provide a nullptr if not used.
    * @param shortlistMemory byte array of shortlist (aligned to 64)
    */
-  explicit BatchTranslator(DeviceId const device, std::vector<Ptr<Vocab const>> &vocabs,
+  explicit BatchTranslator(DeviceId const device, Vocabs &vocabs,
                   Ptr<Options> options, const AlignedMemory* modelMemory, const AlignedMemory* shortlistMemory);
 
   // convenience function for logging. TODO(jerin)
@@ -45,7 +46,7 @@ class BatchTranslator {
 private:
   Ptr<Options> options_;
   DeviceId device_;
-  std::vector<Ptr<Vocab const>> *vocabs_;
+  const Vocabs&  vocabs_;
   Ptr<ExpressionGraph> graph_;
   std::vector<Ptr<Scorer>> scorers_;
   Ptr<data::ShortlistGenerator const> slgen_;
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index 175397d5f..bf1cb572b 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -28,27 +28,6 @@ struct MemoryBundle {
 
   /// @todo Not implemented yet
   AlignedMemory ssplitPrefixFile;
-
-  MemoryBundle() = default;
-
-  MemoryBundle(MemoryBundle &&from){
-    model = std::move(from.model);
-    shortlist = std::move(from.shortlist);
-    vocabs = std::move(vocabs);
-    ssplitPrefixFile = std::move(from.ssplitPrefixFile);
-  }
-
-  MemoryBundle &operator=(MemoryBundle &&from) {
-    model = std::move(from.model);
-    shortlist = std::move(from.shortlist);
-    vocabs = std::move(vocabs);
-    ssplitPrefixFile = std::move(from.ssplitPrefixFile);
-    return *this;
-  }
-
-  // Delete copy constructors
-  MemoryBundle(const MemoryBundle&) = delete;
-  MemoryBundle& operator=(const MemoryBundle&) = delete;
 };
 
 } // namespace bergamot
diff --git a/src/translator/response_builder.cpp b/src/translator/response_builder.cpp
index f68bd31f7..037d4567b 100644
--- a/src/translator/response_builder.cpp
+++ b/src/translator/response_builder.cpp
@@ -65,11 +65,10 @@ void ResponseBuilder::buildTranslatedText(Histories &histories,
 
     Result result = onebest[0]; // Expecting only one result;
     Words words = std::get<0>(result);
-    auto targetVocab = vocabs_->back();
 
     std::string decoded;
     std::vector<string_view> targetSentenceMappings;
-    targetVocab->decodeWithByteRanges(words, decoded, targetSentenceMappings);
+    vocabs_.target()->decodeWithByteRanges(words, decoded, targetSentenceMappings);
 
     switch (responseOptions_.concatStrategy) {
     case ConcatStrategy::FAITHFUL: {
diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
index 85caffb6c..b8a8dd40d 100644
--- a/src/translator/response_builder.h
+++ b/src/translator/response_builder.h
@@ -4,6 +4,7 @@
 #include "data/types.h"
 #include "response.h"
 #include "response_options.h"
+#include "vocabs.h"
 
 // For now we will work with this, to avoid complaints another structure is hard
 // to operate with.
@@ -24,10 +25,10 @@ class ResponseBuilder {
   /// @param [in] vocabs: marian vocab object (used in decoding)
   /// @param [in] promise: promise to set with the constructed Response.
   ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source,
-                  std::vector<Ptr<Vocab const>> &vocabs,
+                  Vocabs &vocabs,
                   std::promise<Response> &&promise)
       : responseOptions_(responseOptions), source_(std::move(source)),
-        vocabs_(&vocabs), promise_(std::move(promise)) {}
+        vocabs_(vocabs), promise_(std::move(promise)) {}
 
   /// Constructs and sets the promise of a Response object from obtained
   /// histories after translating.
@@ -81,7 +82,7 @@ class ResponseBuilder {
   // Data members are context/curried args for the functor.
 
   ResponseOptions responseOptions_;
-  std::vector<Ptr<Vocab const>> *vocabs_; // vocabs are required for decoding
+  const Vocabs& vocabs_; // vocabs are required for decoding
                                           // and any source validation checks.
   std::promise<Response> promise_; //  To be set when callback triggered and
                                    //  after Response constructed.
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 16c474393..54396677e 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -5,45 +5,12 @@
 #include <string>
 #include <utility>
 
-inline std::vector<marian::Ptr<const marian::Vocab>>
-loadVocabularies(marian::Ptr<marian::Options> options,
-                 std::vector<std::shared_ptr<marian::bergamot::AlignedMemory>>&& vocabMemories) {
-  // @TODO: parallelize vocab loading for faster startup
-  std::vector<marian::Ptr<marian::Vocab const>> vocabs;
-  if(!vocabMemories.empty()){
-    // load vocabs from buffer
-    ABORT_IF(vocabMemories.size() < 2, "Insufficient number of vocabularies.");
-    vocabs.resize(vocabMemories.size());
-    for (size_t i = 0; i < vocabs.size(); i++) {
-      marian::Ptr<marian::Vocab> vocab = marian::New<marian::Vocab>(options, i);
-      vocab->loadFromSerialized(absl::string_view(vocabMemories[i]->begin(), vocabMemories[i]->size()));
-      vocabs[i] = vocab;
-    }
-  } else {
-    // load vocabs from file
-    auto vfiles = options->get<std::vector<std::string>>("vocabs");
-    // with the current setup, we need at least two vocabs: src and trg
-    ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
-    vocabs.resize(vfiles.size());
-    std::unordered_map<std::string, marian::Ptr<marian::Vocab>> vmap;
-    for (size_t i = 0; i < vocabs.size(); ++i) {
-      auto m = vmap.emplace(std::make_pair(vfiles[i], marian::Ptr<marian::Vocab>()));
-      if (m.second) { // new: load the vocab
-        m.first->second = marian::New<marian::Vocab>(options, i);
-        m.first->second->load(vfiles[i]);
-      }
-      vocabs[i] = m.first->second;
-    }
-  }
-  return vocabs;
-}
-
 namespace marian {
 namespace bergamot {
 
 Service::Service(Ptr<Options> options, MemoryBundle memoryBundle)
     : requestId_(0), options_(options),
-      vocabs_(std::move(loadVocabularies(options, std::move(memoryBundle.vocabs)))),
+      vocabs_(options, std::move(memoryBundle.vocabs)),
       text_processor_(vocabs_, options), batcher_(options),
       numWorkers_(options->get<int>("cpu-threads")),
       modelMemory_(std::move(memoryBundle.model)),
diff --git a/src/translator/service.h b/src/translator/service.h
index 9d0a67d4c..6c60888a1 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -9,6 +9,7 @@
 #include "response_builder.h"
 #include "text_processor.h"
 #include "translator/parser.h"
+#include "vocabs.h"
 
 #ifndef WASM_COMPATIBLE_SOURCE
 #include "pcqueue.h"
@@ -172,7 +173,7 @@ class Service {
 
   size_t requestId_;
   /// Store vocabs representing source and target.
-  std::vector<Ptr<Vocab const>> vocabs_; // ORDER DEPENDENCY (text_processor_)
+  Vocabs vocabs_; // ORDER DEPENDENCY (text_processor_)
 
   /// TextProcesser takes a blob of text and converts into format consumable by
   /// the batch-translator and annotates sentences and words.
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index fb6690125..457e2b9c9 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -4,7 +4,6 @@
 #include "annotation.h"
 
 #include "common/options.h"
-#include "data/vocab.h"
 #include <vector>
 
 namespace marian {
@@ -12,13 +11,14 @@ namespace bergamot {
 
 Segment TextProcessor::tokenize(const string_view &segment,
                                 std::vector<string_view> &wordRanges) {
-  return vocabs_->front()->encodeWithByteRanges(
+  // vocabs_->sources().front() is invoked as we currently only support one source vocab
+  return vocabs_.sources().front()->encodeWithByteRanges(
       segment, wordRanges, /*addEOS=*/false, /*inference=*/true);
 }
 
-TextProcessor::TextProcessor(std::vector<Ptr<Vocab const>> &vocabs,
+TextProcessor::TextProcessor(Vocabs &vocabs,
                              Ptr<Options> options)
-    : vocabs_(&vocabs), sentence_splitter_(options) {
+    : vocabs_(vocabs), sentence_splitter_(options) {
 
   max_length_break_ = options->get<int>("max-length-break");
   max_length_break_ = max_length_break_ - 1;
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index 698e36ea1..f5d4d88ea 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -7,6 +7,7 @@
 #include "annotation.h"
 
 #include "sentence_splitter.h"
+#include "vocabs.h"
 
 #include <vector>
 
@@ -21,7 +22,7 @@ class TextProcessor {
   // sentences (vector of words). In addition, the ByteRanges of the
   // source-tokens in unnormalized text are provided as string_views.
 public:
-  explicit TextProcessor(std::vector<Ptr<Vocab const>> &vocabs, Ptr<Options>);
+  explicit TextProcessor(Vocabs &vocabs, Ptr<Options>);
 
   void process(AnnotatedText &source, Segments &segments);
 
@@ -36,9 +37,10 @@ class TextProcessor {
                 Segments &segments, AnnotatedText &source);
 
   // shorthand, used only in truncate()
-  const Word sourceEosId() const { return vocabs_->front()->getEosId(); }
+  // vocabs_->sources().front() is invoked as we currently only support one source vocab
+  const Word sourceEosId() const { return vocabs_.sources().front()->getEosId(); }
 
-  std::vector<Ptr<Vocab const>> *vocabs_;
+  const Vocabs& vocabs_;
   SentenceSplitter sentence_splitter_;
   size_t max_length_break_;
 };
diff --git a/src/translator/vocabs.h b/src/translator/vocabs.h
new file mode 100644
index 000000000..89aed4b05
--- /dev/null
+++ b/src/translator/vocabs.h
@@ -0,0 +1,81 @@
+#pragma once
+
+namespace marian {
+namespace bergamot {
+
+/// Wrapper of Marian Vocab objects needed for translator.
+/// Holds multiple source vocabularies and one target vocabulary
+class Vocabs {
+public:
+  /// Construct vocabs object from either byte-arrays or files
+  Vocabs(Ptr<Options> options, std::vector<std::shared_ptr<AlignedMemory>>&& vocabMemories): options_(options){
+    if (!vocabMemories.empty()){
+      // load vocabs from buffer
+      load(std::move(vocabMemories));
+    }
+    else{
+      // load vocabs from file
+      auto vocabPaths = options->get<std::vector<std::string>>("vocabs");
+      load(vocabPaths);
+    }
+  }
+
+  /// Get all source vocabularies (as a vector)
+  const std::vector<Ptr<Vocab const>>& sources() const {
+    return srcVocabs_;
+  }
+
+  /// Get the target vocabulary
+  const Ptr<Vocab const>& target() const {
+    return trgVocab_;
+  }
+
+private:
+  std::vector<Ptr<Vocab const>> srcVocabs_;  // source vocabularies
+  Ptr<Vocab const> trgVocab_;                // target vocabulary
+  Ptr<Options> options_;
+
+  // load from buffer
+  void load(std::vector<std::shared_ptr<AlignedMemory>>&& vocabMemories) {
+    // At least two vocabs: src and trg
+    ABORT_IF(vocabMemories.size() < 2, "Insufficient number of vocabularies.");
+    srcVocabs_.resize(vocabMemories.size());
+    // hashMap is introduced to avoid double loading the same vocab
+    // loading vocabs (either from buffers or files) is the biggest bottleneck of the speed
+    // uintptr_t holds unique keys (address) for share_ptr<AlignedMemory>
+    std::unordered_map<uintptr_t, Ptr<Vocab>> vmap;
+    for (size_t i = 0; i < srcVocabs_.size(); i++) {
+      auto m = vmap.emplace(std::make_pair(reinterpret_cast<uintptr_t>(vocabMemories[i].get()), Ptr<Vocab>()));
+      if (m.second) { // new: load the vocab
+        m.first->second = New<Vocab>(options_, i);
+        m.first->second->loadFromSerialized(absl::string_view(vocabMemories[i]->begin(), vocabMemories[i]->size()));
+      }
+      srcVocabs_[i] = m.first->second;
+    }
+    // Initialize target vocab
+    trgVocab_ = srcVocabs_.back();
+    srcVocabs_.pop_back();
+  }
+
+  // load from file
+  void load(const std::vector<std::string>& vocabPaths){
+    // with the current setup, we need at least two vocabs: src and trg
+    ABORT_IF(vocabPaths.size() < 2, "Insufficient number of vocabularies.");
+    srcVocabs_.resize(vocabPaths.size());
+    std::unordered_map<std::string, Ptr<Vocab>> vmap;
+    for (size_t i = 0; i < srcVocabs_.size(); ++i) {
+      auto m = vmap.emplace(std::make_pair(vocabPaths[i], Ptr<Vocab>()));
+      if (m.second) { // new: load the vocab
+        m.first->second = New<Vocab>(options_, i);
+        m.first->second->load(vocabPaths[i]);
+      }
+      srcVocabs_[i] = m.first->second;
+    }
+    // Initialize target vocab
+    trgVocab_ = srcVocabs_.back();
+    srcVocabs_.pop_back();
+  }
+};
+
+} // namespace bergamot
+} // namespace marian

From 3e7058767222a8add4a505518c09ff0e7c6a2810 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Mon, 17 May 2021 16:42:18 +0100
Subject: [PATCH 239/442] Rewrite annotation class to remove corner cases
 (#135)

---
 src/tests/annotation_tests.cpp      |  30 ++---
 src/translator/annotation.cpp       | 152 +++++++--------------
 src/translator/annotation.h         | 199 +++++++++++++++++-----------
 src/translator/response_builder.cpp |  11 +-
 src/translator/text_processor.cpp   |  14 +-
 src/translator/text_processor.h     |   6 +-
 6 files changed, 191 insertions(+), 221 deletions(-)

diff --git a/src/tests/annotation_tests.cpp b/src/tests/annotation_tests.cpp
index d323b9db0..0f02a7ad3 100644
--- a/src/tests/annotation_tests.cpp
+++ b/src/tests/annotation_tests.cpp
@@ -23,9 +23,6 @@ TEST_CASE("Test Annotation API with random sentences") {
   std::mt19937 randomIntGen_;
   randomIntGen_.seed(42);
 
-  AnnotatedText testAnnotation; // This the container we add through API and
-                                // check if the access is correct.
-
   // External book-keeping so we have ground truths. Each element represents a
   // sentence.
 
@@ -45,7 +42,7 @@ TEST_CASE("Test Annotation API with random sentences") {
   //
   //     4-0 4-1 4-2 4-3
   //
-  // Words are separated by space units.
+  // Tokens are contiguous because that's how SentencePiece works.
   //
   // Below, we accumulate the text with intended structure as above, and
   // ground-truth tables populated to be aware of the ByteRanges where they are
@@ -53,9 +50,10 @@ TEST_CASE("Test Annotation API with random sentences") {
   if (debug) {
     std::cout << "Preparing text and ground truth-tables" << std::endl;
   }
+  std::string text;
   for (size_t idx = 0; idx < sentences; idx++) {
     if (idx != 0)
-      testAnnotation.text += "\n";
+      text += "\n";
 
     // Words can be zero, we need to support empty word sentences as well.
     size_t numWords = randomIntGen_() % maxWords;
@@ -65,23 +63,16 @@ TEST_CASE("Test Annotation API with random sentences") {
 
     // For empty sentence, we expect it to be empty and marked in position where
     // the existing string is if needed to be pointed out.
-    size_t before = testAnnotation.text.size() - 1;
+    size_t before = text.size() - 1;
     size_t sentenceBegin{before}, sentenceEnd{before};
 
     for (size_t idw = 0; idw < numWords; idw++) {
-      if (idw != 0) {
-        testAnnotation.text += " ";
-        if (debug) {
-          std::cout << " ";
-        }
-      }
-
       // Get new beginning, accounting for space above.
-      before = testAnnotation.text.size();
+      before = text.size();
 
       // Add the word
       std::string word = std::to_string(idx) + "-" + std::to_string(idw);
-      testAnnotation.text += word;
+      text += word;
 
       // Do math, before, before + new-word's size.
       wordByteRanges.push_back((ByteRange){before, before + word.size()});
@@ -105,6 +96,9 @@ TEST_CASE("Test Annotation API with random sentences") {
     groundTruthSentences.push_back((ByteRange){sentenceBegin, sentenceEnd});
   }
 
+  AnnotatedText testAnnotation(std::move(text)); // This the container we add through API and
+                                                 // check if the access is correct.
+
   // We prepare string_views now with the known ByteRanges and use the
   // string_view based AnnotatedText.addSentence(...) API to add sentences to
   // transparently convert from string_views to ByteRanges, rebasing/working out
@@ -116,6 +110,7 @@ TEST_CASE("Test Annotation API with random sentences") {
   }
 
   std::vector<std::vector<marian::string_view>> wordStringViews;
+  std::vector<ByteRange>::const_iterator sentence_iter = groundTruthSentences.begin();
   for (auto &sentence : groundTruthWords) {
     std::vector<marian::string_view> wordByteRanges;
     bool first{true};
@@ -132,7 +127,8 @@ TEST_CASE("Test Annotation API with random sentences") {
         std::cout << std::string(wordView);
       }
     }
-    testAnnotation.addSentence(wordByteRanges);
+    testAnnotation.recordExistingSentence(wordByteRanges.begin(), wordByteRanges.end(), testAnnotation.text.data() + sentence_iter->begin);
+    ++sentence_iter;
     wordStringViews.push_back(wordByteRanges);
     if (debug) {
       std::cout << std::endl;
@@ -207,7 +203,7 @@ TEST_CASE("Test Annotation API with random sentences") {
   // Sentence if the random test above does not cover it for some reason.
   int emptySentenceIdx = sentences;
   std::vector<marian::string_view> emptySentence;
-  testAnnotation.addSentence(emptySentence);
+  testAnnotation.recordExistingSentence(emptySentence.begin(), emptySentence.end(), testAnnotation.text.data() + testAnnotation.text.size());
 
   // There are no words.
   CHECK(testAnnotation.numWords(emptySentenceIdx) == 0);
diff --git a/src/translator/annotation.cpp b/src/translator/annotation.cpp
index c27d7849c..90e02e00f 100644
--- a/src/translator/annotation.cpp
+++ b/src/translator/annotation.cpp
@@ -1,130 +1,68 @@
 #include "annotation.h"
 #include <cassert>
-#include <iostream>
 
 namespace marian {
 namespace bergamot {
 
-void Annotation::addSentence(std::vector<ByteRange> &sentence) {
-  flatByteRanges_.insert(std::end(flatByteRanges_), std::begin(sentence),
-                         std::end(sentence));
-  size_t size = flatByteRanges_.size();
-  sentenceEndIds_.push_back(size);
+AnnotatedText::AnnotatedText(std::string &&t) : text(std::move(t)) {
+  // Treat the entire text as a gap that recordExistingSentence will break.
+  annotation.token_begin_.back() = text.size();
 }
 
-size_t Annotation::numWords(size_t sentenceIdx) const {
-  size_t bosId, eosId;
-  bosId = sentenceEndIds_[sentenceIdx]; // Half interval, so;
-  eosId = sentenceEndIds_[sentenceIdx + 1];
-  // Difference between eosId and bosId is the number of words.
-  return eosId - bosId;
-}
+void AnnotatedText::appendSentence(string_view prefix, std::vector<string_view>::iterator begin, std::vector<string_view>::iterator end) {
+  assert(annotation.token_begin_.back() == text.size());
+  // We'll be adding tokens from the sentence and another gap.
+  annotation.token_begin_.reserve(annotation.token_begin_.size() + (end - begin) + 1);
 
-ByteRange Annotation::sentence(size_t sentenceIdx) const {
-  size_t bosId, eosId;
-  bosId = sentenceEndIds_[sentenceIdx]; // Half interval, so;
-  eosId = sentenceEndIds_[sentenceIdx + 1];
-  ByteRange sentenceByteRange;
+  // prefix is just end of the previous one.
+  appendEndingWhitespace(prefix);
 
-  if (bosId == eosId) {
-    // We have an empty sentence. However, we want to be able to point where in
-    // target this happened through the ranges. We are looking for the end of
-    // the flatByteRange and non-empty sentence before this happened and
-    // construct empty string-view equivalent ByteRange.
-    ByteRange eos = flatByteRanges_[eosId - 1];
-    sentenceByteRange = ByteRange{eos.end, eos.end};
-  } else {
-    ByteRange bos = flatByteRanges_[bosId];
-    ByteRange eos = flatByteRanges_[eosId - 1];
-    sentenceByteRange = ByteRange{bos.begin, eos.end};
+  // Appending sentence text.
+  std::size_t offset = text.size();
+  for (std::vector<string_view>::iterator token = begin; token != end; ++token) {
+    offset += token->size();
+    annotation.token_begin_.push_back(offset);
   }
-  return sentenceByteRange;
-}
-
-ByteRange Annotation::word(size_t sentenceIdx, size_t wordIdx) const {
-  size_t bosOffset = sentenceEndIds_[sentenceIdx];
-  return flatByteRanges_[bosOffset + wordIdx];
-}
-
-string_view AnnotatedText::word(size_t sentenceIdx, size_t wordIdx) const {
-  auto terminals = annotation.word(sentenceIdx, wordIdx);
-  return string_view(&text[terminals.begin], terminals.size());
-}
-
-string_view AnnotatedText::sentence(size_t sentenceIdx) const {
-  auto sentenceAsByteRange = annotation.sentence(sentenceIdx);
-  return asStringView(sentenceAsByteRange);
-}
-
-void AnnotatedText::appendSentence(std::string prefix, std::string &reference,
-                                   std::vector<string_view> &wordRanges) {
-  text += prefix;
-  size_t offset = text.size(); // Get size before to do ByteRange arithmetic
-  text += reference;           // Append reference to text
-  std::vector<ByteRange> sentence;
-  for (auto &wordView : wordRanges) {
-    size_t thisWordBegin = offset + wordView.data() - reference.data();
-    sentence.push_back(
-        ByteRange{thisWordBegin, thisWordBegin + wordView.size()});
+  if (begin != end) {
+    text.append(begin->data(), (end - 1)->data() + (end - 1)->size());
+    assert(offset == text.size()); // Tokens should be contiguous.
   }
-  annotation.addSentence(sentence);
-}
-
-void AnnotatedText::addSentence(std::vector<string_view> &wordRanges) {
-  addSentence(std::begin(wordRanges), std::end(wordRanges));
-};
 
-void AnnotatedText::addSentence(std::vector<string_view>::iterator begin,
-                                std::vector<string_view>::iterator end) {
-  std::vector<ByteRange> sentence;
-  for (auto p = begin; p != end; p++) {
-    size_t begin_offset = p->data() - text.data();
-    sentence.push_back(ByteRange{begin_offset, begin_offset + p->size()});
-  }
-  annotation.addSentence(sentence);
-};
-
-ByteRange AnnotatedText::wordAsByteRange(size_t sentenceIdx,
-                                         size_t wordIdx) const {
-  return annotation.word(sentenceIdx, wordIdx);
+  // Add the gap after the sentence.  This is empty for now, but will be
+  // extended with appendEndingWhitespace or another appendSentence.
+  annotation.gap_.push_back(annotation.token_begin_.size() - 1);
+  annotation.token_begin_.push_back(offset);
 }
 
-ByteRange AnnotatedText::sentenceAsByteRange(size_t sentenceIdx) const {
-  return annotation.sentence(sentenceIdx);
+void AnnotatedText::appendEndingWhitespace(string_view whitespace) {
+  text.append(whitespace.data(), whitespace.size());
+  annotation.token_begin_.back() = text.size();
 }
 
-string_view AnnotatedText::asStringView(const ByteRange &byteRange) const {
-  const char *data = &text[byteRange.begin];
-  size_t size = byteRange.size();
-  return string_view(data, size);
-}
-
-string_view AnnotatedText::gap(size_t sentenceIdx) const {
-  // Find start of filler-text before, there's a corner case when there's no
-  // sentence before.
-  const char *start = nullptr;
-  if (sentenceIdx == 0) {
-    // If first sentence, filler begins at start of whole-text.
-    start = text.data();
-  } else {
-    // Otherwise, filler begins at end of previous sentence.
-    string_view sentenceBefore = sentence(sentenceIdx - 1);
-    start = sentenceBefore.data() + sentenceBefore.size();
+void AnnotatedText::recordExistingSentence(std::vector<string_view>::iterator begin, std::vector<string_view>::iterator end, const char *sentence_begin) {
+  assert(sentence_begin >= text.data());
+  assert(sentence_begin <= text.data() + text.size());
+  assert(begin == end || sentence_begin == begin->data());
+  assert(!annotation.token_begin_.empty());
+  assert(annotation.token_begin_.back() == text.size());
+  // Clip off size token ending.
+  annotation.token_begin_.resize(annotation.token_begin_.size() - 1);
+  for (std::vector<string_view>::iterator i = begin; i != end; ++i) {
+    assert(i->data() >= text.data()); // In range.
+    assert(i->data() + i->size() <= text.data() + text.size()); // In range
+    assert(i + 1 == end || i->data() + i->size() == (i+1)->data()); // Contiguous
+    annotation.token_begin_.push_back(i->data() - text.data());
   }
-
-  // Find end of filler-text, but there is a corner-case to handle.
-  const char *end = nullptr;
-  if (sentenceIdx == numSentences()) {
-    // If last sentence, manually find end of whole-text.
-    const char *begin = text.data();
-    end = begin + text.size();
+  // Gap token after sentence.
+  annotation.gap_.push_back(annotation.token_begin_.size());
+  if (begin != end) {
+    annotation.token_begin_.push_back((end - 1)->data() + (end - 1)->size() - text.data());
   } else {
-    // Otherwise, the filler ends at the start of next sentence.
-    string_view sentenceAfter = sentence(sentenceIdx);
-    end = sentenceAfter.data();
+    // empty sentence.
+    annotation.token_begin_.push_back(sentence_begin - text.data());
   }
-
-  return string_view(start, end - start);
+  // Add back size token ending.
+  annotation.token_begin_.push_back(text.size());
 }
 
 } // namespace bergamot
diff --git a/src/translator/annotation.h b/src/translator/annotation.h
index 8cb7caf0c..555ab53ae 100644
--- a/src/translator/annotation.h
+++ b/src/translator/annotation.h
@@ -17,83 +17,99 @@ struct ByteRange {
   const size_t size() const { return end - begin; }
 };
 
-/// An Annotation is a collection of ByteRanges used to denote ancillary
-/// information of sentences and words on a text of string. Annotation is meant
-/// for consumption on platforms where `string_view` creates problems (eg:
-/// exports through WASM) conveniently rebasing them as required into
-/// ByteRanges. See AnnotatedText for cases where this is a non-issue.
+/// Annotation expresses sentence and token boundary information as ranges of
+/// bytes in a string, but does not itself own the string.  
+/// 
+/// See also AnnotatedText, which owns Annotation and the string. AnnotatedText
+/// wraps these ByteRange functions to provide a string_view interface.
 ///
-/// **Usage**
+/// Text is divided into gaps (whitespace between sentences) and sentences like
+/// so:
+///   gap sentence gap sentence gap
+/// Because gaps appear at the beginning and end of the text, there's always
+/// one more gap than there are sentences.
 ///
-/// To ensure rebasing is consistent during creation and updation, use
-/// `Annotation` best through `AnnotatedText`, which also holds the reference
-/// string and can work with `string_views`.
+/// The entire text is a unbroken sequence of tokens (i.e. the end of a token
+/// is the beginning of the next token).  A gap is exactly one token containing
+/// whatever whitespace is between the sentences.  A sentence is a sequence of
+/// tokens.
 ///
-/// If used separately, it is on the user to ensure the reference string
-/// is the same as what the Annotation refers to. For best results, an instance
-/// is expected to be read only in this mode of operation.
+/// Since we are using SentencePiece, a token can include whitespace.  The term
+/// "word" is used, somewhat incorrectly, as a synonym of token.
 ///
-/// **Idea**
-///
-/// Annotation is intended to be the same structure conceptually as below,
-/// except the `std::vector<std::vector<ByteRange>>` hammered into a flat
-/// structure to avoid multiple reallocs keeping efficiency in mind. This is
-/// achieved by having markers of where sentence ends in the flat container
-/// storing word ByteRanges.
-///
-/// ```cpp
-/// typedef ByteRange Word;
-/// // std::vector<ByteRange>, a single sentence
-/// typedef std::vector<Word> Sentence;
-/// std::vector<std::vector<ByteRange> // multiple sentences
-/// typedef std::vector<Sentence> Annotation;
-///
-/// Annotation example;
-/// ```
-/// This structure exists to provide a consistent API to access the nested
-/// sentences of varying lengths, which occur in source-text processed into
-/// multiple sentences, and target-text translated from source as multiple
-/// sentences, both composed of (sub)-words, providing a List[List] like access
-/// while storing it in a compact and efficient manner.
+/// A gap can be empty (for example there may not have been whitespace at the
+/// beginning).  A sentence can also be empty (typically the translation system
+/// produced empty output).  That's fine, these are just empty ranges as you
+/// would expect.
 class Annotation {
 public:
-  /// Annotation is constructed empty. See `addSentence()` to populate it with
-  /// annotations.
+  /// Initially an empty string.  Populated by AnnotatedText.
   Annotation() {
-    // The -1-th sentence ends at 0.
-    sentenceEndIds_.push_back(0);
+    token_begin_.push_back(0);
+    token_begin_.push_back(0);
+    gap_.push_back(0);
   }
 
-  size_t numSentences() const { return sentenceEndIds_.size() - 1; }
+  size_t numSentences() const { return gap_.size() - 1; }
 
   /// Returns number of words in the sentence identified by `sentenceIdx`.
-  size_t numWords(size_t sentenceIdx) const;
-
-  /// Adds a sentences from `vector<ByteRange>` representation, internally doing
-  /// extra book-keeping for the sentence terminal markings. Sentences are
-  /// expected to be added in order as they occur in text.
-  void addSentence(std::vector<ByteRange> &sentence);
+  size_t numWords(size_t sentenceIdx) const {
+    return gap_[sentenceIdx + 1] - gap_[sentenceIdx] - 1 /* minus the gap */;
+  }
 
   /// Returns a ByteRange representing `wordIdx` in sentence indexed by
   /// `sentenceIdx`. `wordIdx` follows 0-based indexing, and should be less than
   /// `.numWords()` for `sentenceIdx` for defined behaviour.
-  ByteRange word(size_t sentenceIdx, size_t wordIdx) const;
+  ByteRange word(size_t sentenceIdx, size_t wordIdx) const {
+    size_t tokenIdx = gap_[sentenceIdx] + 1 + wordIdx;
+    return ByteRange {token_begin_[tokenIdx], token_begin_[tokenIdx + 1]};
+  }
 
   /// Returns a ByteRange representing sentence corresponding to `sentenceIdx`.
   /// `sentenceIdx` follows 0-based indexing, and behaviour is defined only when
   /// less than `.numSentences()`.
-  ByteRange sentence(size_t sentenceIdx) const;
+  ByteRange sentence(size_t sentenceIdx) const {
+    return ByteRange {
+      token_begin_[gap_[sentenceIdx] + 1], /*end of whitespace before */
+      token_begin_[gap_[sentenceIdx + 1]] /*beginning of whitespace after */
+    };
+  }
+
+  ByteRange gap(size_t gapIdx) const {
+    size_t tokenIdx = gap_[gapIdx];
+    return ByteRange {token_begin_[tokenIdx], token_begin_[tokenIdx + 1]};
+  }
 
 private:
-  /// A flat storage for ByteRanges. Composed of word ByteRanges, extra
-  /// information in sentenceEndIds_ to denote sentence boundary markers as
-  /// indices.
-  std::vector<ByteRange> flatByteRanges_;
-
-  /// Stores indices onto flatByteRanges_ of where sentences end (not inclusive,
-  /// aligned with C++ half interval notions). There is a 0 marker to simplify
-  /// sources, indicating where the -1-th sentence ends.
-  std::vector<size_t> sentenceEndIds_;
+  friend class AnnotatedText;
+  /// Map from token index to byte offset at which it begins.  Token i is:
+  ///   [token_begin_[i], token_begin_[i+1])
+  /// The vector is padded so that these indices are always valid, even at the
+  /// end.  So tokens_begin_.size() is the number of tokens plus 1.
+  std::vector<size_t> token_begin_;
+
+  /// Indices of tokens that correspond to gaps between sentences.  These are
+  /// indices into token_begin_.
+  /// Gap g is byte range:
+  ///   [token_begin_[gap_[w]], token_begin_[gap_[w]+1])
+  /// Sentence s is byte range:
+  ///   [token_begin_[gap_[s]+1], token_begin_[gap_[s+1]])
+  /// A sentence does not include whitespace at the beginning or end.
+  ///
+  /// gap_.size() == numSentences() + 1.
+  ///
+  /// Example: empty text "" -> just an empty gap.
+  /// token_begin_ = {0, 0};
+  /// gap_ = {0};
+  ///
+  /// Example: only space " " -> just a gap containing the space.
+  /// token_begin_ = {0, 1};
+  /// gap_ = {0};
+  ///
+  /// Example: one token "hi" -> empty gap, sentence with one token, empty gap
+  /// token_begin_ = {0, 0, 2, 2};
+  /// gap_ = {0, 2};
+  std::vector<size_t> gap_;
 };
 
 /// AnnotatedText is effectively std::string text + Annotation, providing the
@@ -107,7 +123,6 @@ class Annotation {
 ///
 /// 3. Bind the text and annotations together, to move around as a meaningful
 /// unit.
-
 struct AnnotatedText {
 public:
   std::string text;      ///< Blob of string elements in annotation refers to.
@@ -122,7 +137,31 @@ struct AnnotatedText {
 
   /// Construct moving in a string (for efficiency purposes, copying string
   /// constructor is disallowed).
-  AnnotatedText(std::string &&text) : text(std::move(text)){};
+  AnnotatedText(std::string &&text);
+
+  /// Appends a sentence to the existing text and transparently rebases
+  /// string_views.  Since this tracks only prefix, remember
+  /// appendEndingWhitespace.
+  /// The string_views must not already be in text.
+  void appendSentence(
+      string_view prefix,
+      std::vector<string_view>::iterator tokens_begin,
+      std::vector<string_view>::iterator tokens_end);
+
+  /// Append the whitespace at the end of input. string_view must not be in
+  /// text.
+  void appendEndingWhitespace(string_view whitespace);
+
+  /// Record the existence of a sentence that is already in text.  The
+  /// iterators are over string_views for each token that must be in text
+  /// already.  This function must be called to record sentences in order.
+  /// Normally the beginning of the sentence can be inferred from
+  /// tokens_begin->data() but the tokens could be empty, so sentence_begin is
+  /// required to know where the sentence is.
+  void recordExistingSentence(
+      std::vector<string_view>::iterator tokens_begin,
+      std::vector<string_view>::iterator tokens_end,
+      const char *sentence_begin);
 
   /// Returns the number of sentences in the annotation structure.
   const size_t numSentences() const { return annotation.numSentences(); }
@@ -132,46 +171,44 @@ struct AnnotatedText {
     return annotation.numWords(sentenceIdx);
   }
 
-  /// Appends a sentence to the existing text and transparently rebases
-  /// string_views
-  void appendSentence(std::string prefix, std::string &reference,
-                      std::vector<string_view> &wordRanges);
-
-  /// Adds a sentence, used to load from SentencePiece annotations conveniently.
-  void addSentence(std::vector<string_view> &wordRanges);
-
-  /// Adds a sentence between two iterators, often useful while constructing
-  /// from parts of a container.
-  void addSentence(std::vector<string_view>::iterator begin,
-                   std::vector<string_view>::iterator end);
-
   /// Returns a string_view representing wordIdx in sentenceIdx
-  string_view word(size_t sentenceIdx, size_t wordIdx) const;
+  string_view word(size_t sentenceIdx, size_t wordIdx) const {
+    return asStringView(annotation.word(sentenceIdx, wordIdx));
+  }
 
   /// Returns a string_view representing sentence corresponding to sentenceIdx.
-  string_view sentence(size_t sentenceIdx) const;
+  string_view sentence(size_t sentenceIdx) const {
+    return asStringView(annotation.sentence(sentenceIdx));
+  }
 
   /// Returns the string_view of the gap between two sentences in the container.
   ///
   /// More precisely where `i = sentenceIdx, N = numSentences()` for brevity:
   ///
-  /// * For `i = 0`: The gap between the start of text and the first sentence.
+  /// * For `i = 0`: The gap between the start of text and the 0th sentence.
   /// * For `i = 1...N-1`, returns the text comprising of the gap
-  ///   between the `i-1`-th and `i`-th sentence.
-  /// * For `i = N`, the gap between the last sentence and end of
+  ///   between the `i`-th and `i+1`-th sentence.
+  /// * For `i = N`, the gap between the last (N-1th) sentence and end of
   ///   text.
-
   /// @param sentenceIdx: Can be between `[0, numSentences()]`.
-  string_view gap(size_t sentenceIdx) const;
+  string_view gap(size_t sentenceIdx) const {
+    return asStringView(annotation.gap(sentenceIdx));
+  }
 
   /// Returns a ByteRange representing wordIdx in sentenceIdx
-  ByteRange wordAsByteRange(size_t sentenceIdx, size_t wordIdx) const;
+  ByteRange wordAsByteRange(size_t sentenceIdx, size_t wordIdx) const {
+    return annotation.word(sentenceIdx, wordIdx);
+  }
 
   /// Returns a ByteRange representing sentence corresponding to sentenceIdx.
-  ByteRange sentenceAsByteRange(size_t sentenceIdx) const;
+  ByteRange sentenceAsByteRange(size_t sentenceIdx) const {
+    return annotation.sentence(sentenceIdx);
+  }
 
 private:
-  string_view asStringView(const ByteRange &byteRange) const;
+  string_view asStringView(const ByteRange &byteRange) const {
+    return string_view(text.data() + byteRange.begin, byteRange.size());
+  }
 };
 
 } // namespace bergamot
diff --git a/src/translator/response_builder.cpp b/src/translator/response_builder.cpp
index 037d4567b..b2f561b6d 100644
--- a/src/translator/response_builder.cpp
+++ b/src/translator/response_builder.cpp
@@ -75,22 +75,19 @@ void ResponseBuilder::buildTranslatedText(Histories &histories,
       // For each sentence, prepend the filler text between the corresponding
       // source-sentence and the source-sentence before.
       string_view pre = response.source.gap(sentenceIdx);
-      response.target.appendSentence(std::string(pre.data(), pre.size()),
-                                     decoded, targetSentenceMappings);
+      response.target.appendSentence(pre, targetSentenceMappings.begin(), targetSentenceMappings.end());
 
       // If this is the last history to be decoded and translated-text
       // constructed, append the text till the end, which could be spaces or
       // empty.
       if (sentenceIdx + 1 == histories.size()) {
-        string_view post = response.source.gap(sentenceIdx + 1);
-        response.target.text += std::string(post.data(), post.size());
+        response.target.appendEndingWhitespace(response.source.gap(sentenceIdx + 1));
       }
       break;
     }
     case ConcatStrategy::SPACE: {
-      std::string delimiter = (sentenceIdx == 0) ? "" : " ";
-      response.target.appendSentence(delimiter, decoded,
-                                     targetSentenceMappings);
+      string_view delimiter = (sentenceIdx == 0) ? "" : " ";
+      response.target.appendSentence(delimiter, targetSentenceMappings.begin(), targetSentenceMappings.end());
       break;
     }
 
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index 457e2b9c9..bca5fd157 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -41,15 +41,16 @@ void TextProcessor::process(AnnotatedText &source, Segments &segments) {
     // There are some cases where SentencePiece or vocab returns no words
     // after normalization. 0 prevents any empty entries from being added.
     if (segment.size() > 0) {
-      // Truncate segment into max_input_size segments.
-      truncate(segment, wordRanges, segments, source);
+      // Wrap segment into sentences of at most max_length_break_ tokens and
+      // tell source about them.
+      wrap(segment, wordRanges, segments, source);
     }
   }
 }
 
-void TextProcessor::truncate(Segment &segment,
-                             std::vector<string_view> &wordRanges,
-                             Segments &segments, AnnotatedText &source) {
+void TextProcessor::wrap(Segment &segment,
+                         std::vector<string_view> &wordRanges,
+                         Segments &segments, AnnotatedText &source) {
   for (size_t offset = 0; offset < segment.size();
        offset += max_length_break_) {
     auto start = segment.begin() + offset;
@@ -61,7 +62,8 @@ void TextProcessor::truncate(Segment &segment,
     segments.back().push_back(sourceEosId());
 
     auto astart = wordRanges.begin() + offset;
-    source.addSentence(astart, astart + diff);
+    // diff > 0
+    source.recordExistingSentence(astart, astart + diff, astart->data());
   }
 }
 
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index f5d4d88ea..732887757 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -32,9 +32,9 @@ class TextProcessor {
   Segment tokenize(const string_view &input,
                    std::vector<string_view> &tokenRanges);
 
-  // Truncate sentence into max_input_size segments.
-  void truncate(Segment &sentence, std::vector<string_view> &tokenRanges,
-                Segments &segments, AnnotatedText &source);
+  // Wrap into sentences of at most max_length_break_ tokens and add to source.
+  void wrap(Segment &sentence, std::vector<string_view> &tokenRanges,
+            Segments &segments, AnnotatedText &source);
 
   // shorthand, used only in truncate()
   // vocabs_->sources().front() is invoked as we currently only support one source vocab

From c1ef6f2bcb08fcd4f9e2432ae443bb7a81813594 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 17 May 2021 17:33:23 +0200
Subject: [PATCH 240/442] Added cmake file to compute version information

 - Reads BERGAMOT_VERSION file for generating various strings
   for versioning
---
 cmake/GetVersionFromFile.cmake | 60 ++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
 create mode 100644 cmake/GetVersionFromFile.cmake

diff --git a/cmake/GetVersionFromFile.cmake b/cmake/GetVersionFromFile.cmake
new file mode 100644
index 000000000..2eadb427e
--- /dev/null
+++ b/cmake/GetVersionFromFile.cmake
@@ -0,0 +1,60 @@
+##
+# This CMake modules sets the project version from a version file.
+#
+# The module sets the following variables:
+#
+# * PROJECT_VERSION_STRING
+# * PROJECT_VERSION_STRING_FULL
+# * PROJECT_VERSION_MAJOR
+# * PROJECT_VERSION_MINOR
+# * PROJECT_VERSION_PATCH
+# * PROJECT_VERSION_TWEAK
+# * PROJECT_VERSION_GIT_SHA
+#
+# This module is public domain, use it as it fits you best.
+##
+
+# Get full string version from file
+if(PROJECT_VERSION_FILE)
+  file(STRINGS ${PROJECT_VERSION_FILE} PROJECT_VERSION_STRING)
+else()
+  file(STRINGS ${CMAKE_CURRENT_SOURCE_DIR}/BERGAMOT_VERSION PROJECT_VERSION_STRING)
+endif()
+
+# Get current commit SHA from git
+execute_process(COMMAND ${GIT_EXECUTABLE} rev-parse --short HEAD
+  WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
+  OUTPUT_VARIABLE PROJECT_VERSION_GIT_SHA
+  OUTPUT_STRIP_TRAILING_WHITESPACE)
+
+# Get partial versions into a list
+string(REGEX MATCHALL "-.*$|[0-9]+" PROJECT_PARTIAL_VERSION_LIST
+  ${PROJECT_VERSION_STRING})
+
+# Set the version numbers
+list(GET PROJECT_PARTIAL_VERSION_LIST 0 PROJECT_VERSION_MAJOR)
+list(GET PROJECT_PARTIAL_VERSION_LIST 1 PROJECT_VERSION_MINOR)
+list(GET PROJECT_PARTIAL_VERSION_LIST 2 PROJECT_VERSION_PATCH)
+
+# The tweak part is optional, so check if the list contains it
+list(LENGTH PROJECT_PARTIAL_VERSION_LIST PROJECT_PARTIAL_VERSION_LIST_LEN)
+if(PROJECT_PARTIAL_VERSION_LIST_LEN GREATER 3)
+  list(GET PROJECT_PARTIAL_VERSION_LIST 3 PROJECT_VERSION_TWEAK)
+  string(SUBSTRING ${PROJECT_VERSION_TWEAK} 1 -1 PROJECT_VERSION_TWEAK)
+endif()
+
+# Unset the list
+unset(PROJECT_PARTIAL_VERSION_LIST)
+
+# Set full project version string
+set(PROJECT_VERSION_STRING_FULL
+  ${PROJECT_VERSION_STRING}+${PROJECT_VERSION_GIT_SHA})
+
+# Print all variables for debugging
+#message(STATUS ${PROJECT_VERSION_STRING_FULL})
+#message(STATUS ${PROJECT_VERSION_STRING})
+#message(STATUS ${PROJECT_VERSION_MAJOR})
+#message(STATUS ${PROJECT_VERSION_MINOR})
+#message(STATUS ${PROJECT_VERSION_PATCH})
+#message(STATUS ${PROJECT_VERSION_TWEAK})
+#message(STATUS ${PROJECT_VERSION_GIT_SHA})

From c44868e1fdd56e1562afee18773a9ee1d08d7689 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 17 May 2021 17:34:57 +0200
Subject: [PATCH 241/442] Import GetVersionFromFile cmake file in root level
 CMakeLists.txt

---
 CMakeLists.txt | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 332aed16e..e561ed938 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1,4 +1,5 @@
 cmake_minimum_required(VERSION 3.5.1)
+set(CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake)
 
 if (POLICY CMP0074)
   cmake_policy(SET CMP0074 NEW) # CMake 3.12
@@ -71,6 +72,11 @@ if(GIT_FOUND AND EXISTS "${PROJECT_SOURCE_DIR}/.git")
     endif()
 endif()
 
+# Project versioning
+include(GetVersionFromFile)
+message(STATUS "Project name: ${PROJECT_NAME}")
+message(STATUS "Project version: ${PROJECT_VERSION_STRING_FULL}")
+
 if(NOT COMPILE_WASM)
   # Set BUILD_ARCH to native only while compiling for non wasm platform
   set(BUILD_ARCH native CACHE STRING "Compile for this CPU architecture.")

From 2e5880d3d499094e80eaae235fabf86068eb2f00 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 17 May 2021 17:37:10 +0200
Subject: [PATCH 242/442] Modified wasm cmake file to include version
 information in built artifacts

---
 wasm/CMakeLists.txt        | 7 +++++++
 wasm/project_version.js.in | 1 +
 2 files changed, 8 insertions(+)
 create mode 100644 wasm/project_version.js.in

diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 7feef7512..72b22c169 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -4,6 +4,10 @@ add_executable(bergamot-translator-worker
     bindings/TranslationResultBindings.cpp
 )
 
+# Generate version file that can be included in the wasm artifacts
+configure_file(${CMAKE_CURRENT_SOURCE_DIR}/project_version.js.in
+               ${CMAKE_CURRENT_SOURCE_DIR}/project_version.js @ONLY)
+
 # This header inclusion needs to go away later as path to public headers of bergamot
 # translator should be directly available from "bergamot-translator" target
 target_include_directories(bergamot-translator-worker
@@ -19,6 +23,9 @@ set(LINKER_FLAGS "-g2 --bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s
 # Avoid node.js-code in emscripten glue-code
 set(LINKER_FLAGS "${LINKER_FLAGS} -s ENVIRONMENT=web,worker")
 
+# Append version information in the Javascript artifact
+set(LINKER_FLAGS "${LINKER_FLAGS} --extern-pre-js ${CMAKE_CURRENT_SOURCE_DIR}/project_version.js")
+
 set_target_properties(bergamot-translator-worker PROPERTIES
                         SUFFIX ".js"
                         LINK_FLAGS ${LINKER_FLAGS}
diff --git a/wasm/project_version.js.in b/wasm/project_version.js.in
new file mode 100644
index 000000000..9a4095f11
--- /dev/null
+++ b/wasm/project_version.js.in
@@ -0,0 +1 @@
+var BERGAMOT_VERSION_FULL = "@PROJECT_VERSION_STRING_FULL@";
\ No newline at end of file

From 0ad583cc34affac322e36ab0e4b5d4310309db74 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 17 May 2021 17:41:59 +0200
Subject: [PATCH 243/442] Generate project version file for native builds

 - The header file exposes a function that provides version information
   for native binaries
---
 src/translator/CMakeLists.txt       |  4 ++++
 src/translator/project_version.h.in | 19 +++++++++++++++++++
 2 files changed, 23 insertions(+)
 create mode 100644 src/translator/project_version.h.in

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index a7ba0d343..1d48d5912 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,3 +1,7 @@
+# Generate version file
+configure_file(${CMAKE_CURRENT_SOURCE_DIR}/project_version.h.in
+               ${CMAKE_CURRENT_SOURCE_DIR}/project_version.h @ONLY)
+
 add_library(bergamot-translator STATIC
     byte_array_util.cpp
     text_processor.cpp
diff --git a/src/translator/project_version.h.in b/src/translator/project_version.h.in
new file mode 100644
index 000000000..b7a0d04b3
--- /dev/null
+++ b/src/translator/project_version.h.in
@@ -0,0 +1,19 @@
+#pragma once
+
+/*
+ * File project_version.h is generated using CMake. Do not modify project_version.h manually!
+ * Edit project_version.h.in file instead.
+ */
+
+#include <string>
+
+namespace marian {
+namespace bergamot {
+
+std::string bergamotBuildVersion() {
+    // e.g. v1.2.3-alpha.1.1+abc123d
+    return "@PROJECT_VERSION_STRING_FULL@";
+}
+
+} // namespace bergamot
+} // namespace marian

From 067076fbc180bac04492252eadda8497c22065c3 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 17 May 2021 17:38:00 +0200
Subject: [PATCH 244/442] Bumped version to 0.3.0

 - This brings the version info in sync with the various releases
   of extension
---
 BERGAMOT_VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BERGAMOT_VERSION b/BERGAMOT_VERSION
index ae39fab35..268b0334e 100644
--- a/BERGAMOT_VERSION
+++ b/BERGAMOT_VERSION
@@ -1 +1 @@
-v0.0.0
+v0.3.0

From 7a973df74d6a58904926192e163162497c2027ca Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 18 May 2021 11:47:32 +0200
Subject: [PATCH 245/442] Corrected the version number

 - To be in sync with versioning in mozilla/bergamot-translator repo
---
 BERGAMOT_VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BERGAMOT_VERSION b/BERGAMOT_VERSION
index 268b0334e..937cd7846 100644
--- a/BERGAMOT_VERSION
+++ b/BERGAMOT_VERSION
@@ -1 +1 @@
-v0.3.0
+v0.3.1

From 10131c731ad2ed510b76ab8286e14cce64f7be32 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Tue, 18 May 2021 12:45:22 +0100
Subject: [PATCH 246/442] Marian submodule with unified loading (#157)

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 03db505fd..80e474247 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 03db505fda750fdecf8000d7ef7dd78dae65861c
+Subproject commit 80e474247909a67231249d3598ac5ad248c9aff5

From 269edc7ce5a42dfe497020106aed081b01c5c283 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 18 May 2021 14:25:25 +0100
Subject: [PATCH 247/442] Collapsing TranslationRequest -> ResponseOptions
 (#139)

---
 app/bergamot-translator-app.cpp              |  4 +-
 src/QualityScore.h                           | 38 ---------
 src/TranslationRequest.h                     | 84 --------------------
 src/translator/service.cpp                   |  6 +-
 src/translator/service.h                     | 13 ++-
 wasm/bindings/TranslationRequestBindings.cpp |  4 +-
 6 files changed, 12 insertions(+), 137 deletions(-)
 delete mode 100644 src/QualityScore.h
 delete mode 100644 src/TranslationRequest.h

diff --git a/app/bergamot-translator-app.cpp b/app/bergamot-translator-app.cpp
index c48796990..77bc99875 100644
--- a/app/bergamot-translator-app.cpp
+++ b/app/bergamot-translator-app.cpp
@@ -24,14 +24,14 @@ int main(int argc, char **argv) {
   // Route the config string to construct marian model through TranslationModel
   marian::bergamot::Service model(config);
 
-  TranslationRequest translationRequest;
+  marian::bergamot::ResponseOptions responseOptions;
   std::vector<std::string> texts;
 
   for (std::string line; std::getline(std::cin, line);) {
     texts.emplace_back(line);
   }
 
-  auto results = model.translateMultiple(std::move(texts), translationRequest);
+  auto results = model.translateMultiple(std::move(texts), responseOptions);
 
   for (auto &result : results) {
     std::cout << result.getTranslatedText() << std::endl;
diff --git a/src/QualityScore.h b/src/QualityScore.h
deleted file mode 100644
index a6beb4edc..000000000
--- a/src/QualityScore.h
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * QualityScore.h
- *
- */
-
-#ifndef SRC_TRANSLATOR_QUALITYSCORE_H_
-#define SRC_TRANSLATOR_QUALITYSCORE_H_
-
-#include <string>
-#include <vector>
-#include "translator/definitions.h"
-
-/* All possible Granularities for which Quality Scores can be returned for
- * translated text. */
-enum class QualityScoreGranularity {
-  WORD,
-  SENTENCE,
-  NONE,
-};
-
-/* This class represents the Quality Scores for various spans of a translated
- * text at a specific granularity. */
-class QualityScore {
-private:
-  // Sections of the translated text for the Quality Scores.
-  std::vector<std::string_view> textViews;
-
-  // Quality Scores corresponding to each entry of textViews in the same order
-  std::vector<float> textScores;
-
-  // Granularity of the text for the Quality scores above
-  QualityScoreGranularity textGranularity;
-
-public:
-  // ToDo: Public Methods
-};
-
-#endif /* SRC_TRANSLATOR_QUALITYSCORE_H_ */
diff --git a/src/TranslationRequest.h b/src/TranslationRequest.h
deleted file mode 100644
index 95289dd1f..000000000
--- a/src/TranslationRequest.h
+++ /dev/null
@@ -1,84 +0,0 @@
-/*
- * TranslationRequest.h
- *
- *  This file defines the translation request class to be used in
- *  TranslationModel::translate() API.
- */
-
-#ifndef SRC_TRANSLATOR_TRANSLATIONREQUEST_H_
-#define SRC_TRANSLATOR_TRANSLATIONREQUEST_H_
-
-#include "QualityScore.h"
-
-/* This class specifies the information related to the translated text (e.g.
- * quality of the translation etc.) that can be included in the
- * TranslationResult. These optional requests are set/unset independent of each
- * other i.e. setting any one of them doesn’t have the side effect of setting
- * any of the others.
- */
-class TranslationRequest {
-private:
-  // The granularity for which Quality scores of the translated text will be
-  // included in TranslationResult. QualityScoreGranularity::NONE means the
-  // scores are not included in TranslationResult.
-  QualityScoreGranularity qualityScoreGranularity =
-      QualityScoreGranularity::NONE;
-
-  // A flag to include/exclude the information regarding how individual
-  // sentences of original text map to corresponding translated sentences in
-  // joined translated text in the TranslationResult. An example of sentence
-  // mappings:
-  //     originalText (containing 2 sentences)              = "What is your
-  //     name? My name is Abc." translatedText (containing 2 translated
-  //     sentences) = "Was ist dein Name? Mein Name ist Abc." sentenceMappings =
-  //     [
-  //         {"What is your name?", "Was ist dein Name?"},  //
-  //         Pair(originalText[0],translatedText[0])
-  //         {"My name is Abc", "Mein Name ist Abc."}       //
-  //         Pair(originalText[1],translatedText[1])
-  //     ]
-  bool includeSentenceMapping = false;
-
-public:
-  TranslationRequest() {}
-
-  TranslationRequest(const TranslationRequest &request)
-      : qualityScoreGranularity(request.qualityScoreGranularity),
-        includeSentenceMapping(request.includeSentenceMapping) {}
-
-  ~TranslationRequest() {}
-
-  /* Set the granularity for which the Quality scores of translated text should
-   * be included in the TranslationResult. By default
-   * (QualityScoreGranularity::NONE), scores are not included.
-   */
-  void setQualityScoreGranularity(QualityScoreGranularity granularity) {
-    qualityScoreGranularity = granularity;
-  }
-
-  /* Set to true/false to include/exclude the information regarding how
-   * individual sentences of original text map to corresponding translated
-   * sentences in joined translated text in the TranslationResult. By default
-   * (false), this information is not included.
-   */
-  void sentenceMappingInResult(bool includeMapping) {
-    includeSentenceMapping = includeMapping;
-  }
-
-  /* Return the granularity for which the Quality scores of the translated text
-   * will be included in TranslationResult. QualityScoreGranularity::NONE means
-   * the scores will not be included.
-   */
-  QualityScoreGranularity getQualityScoreGranularity() const {
-    return qualityScoreGranularity;
-  }
-
-  /* Return whether the information regarding how individual sentences of
-   * original text map to corresponding translated sentences in joined
-   * translated text will be included in the TranslationResult. By default
-   * (false) means this information will not be included.
-   */
-  bool sentenceMappingInResult() const { return includeSentenceMapping; }
-};
-
-#endif /* SRC_TRANSLATOR_TRANSLATIONREQUEST_H_ */
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 54396677e..265695a38 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -100,11 +100,7 @@ std::future<Response> Service::translate(std::string &&input) {
 
 std::vector<Response>
 Service::translateMultiple(std::vector<std::string> &&inputs,
-                           TranslationRequest translationRequest) {
-  ResponseOptions responseOptions;
-
-  // TODO(jerinphilip) Set options based on TranslationRequest, if and when it
-  // becomes non-dummy.
+                           ResponseOptions responseOptions) {
 
   // We queue the individual Requests so they get compiled at batches to be
   // efficiently translated.
diff --git a/src/translator/service.h b/src/translator/service.h
index 6c60888a1..a678d537c 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -1,7 +1,6 @@
 #ifndef SRC_BERGAMOT_SERVICE_H_
 #define SRC_BERGAMOT_SERVICE_H_
 
-#include "TranslationRequest.h"
 #include "batch_translator.h"
 #include "batcher.h"
 #include "data/types.h"
@@ -102,8 +101,8 @@ class Service {
                                   ResponseOptions options);
 
   /// Translate multiple text-blobs in a single *blocking* API call, providing
-  /// TranslationRequest which applies across all text-blobs dictating how to
-  /// construct Response. TranslationRequest can be used to enable/disable
+  /// ResponseOptions which applies across all text-blobs dictating how to
+  /// construct Response. ResponseOptions can be used to enable/disable
   /// additional information like quality-scores, alignments etc.
   ///
   /// All texts are combined to efficiently construct batches together providing
@@ -114,13 +113,13 @@ class Service {
   /// recommended to work with futures and translate() API.
   ///
   /// @param [in] source: rvalue reference of the string to be translated
-  /// @param [in] translationRequest: TranslationRequest (Unified API)
-  /// indicating whether or not to include some member in the Response, also
-  /// specify any additional configurable parameters.
+  /// @param [in] translationRequest: ResponseOptions indicating whether or not
+  /// to include some member in the Response, also specify any additional
+  /// configurable parameters.
 
   std::vector<Response>
   translateMultiple(std::vector<std::string> &&source,
-                    TranslationRequest translationRequest);
+                    ResponseOptions responseOptions);
 
   /// Returns if model is alignment capable or not.
   bool isAlignmentSupported() const {
diff --git a/wasm/bindings/TranslationRequestBindings.cpp b/wasm/bindings/TranslationRequestBindings.cpp
index bb5ec9884..7d5cd1e6f 100644
--- a/wasm/bindings/TranslationRequestBindings.cpp
+++ b/wasm/bindings/TranslationRequestBindings.cpp
@@ -5,7 +5,9 @@
 
 #include <emscripten/bind.h>
 
-#include "TranslationRequest.h"
+#include "response_options.h"
+
+typedef marian::bergamot::ResponseOptions TranslationRequest;
 
 using namespace emscripten;
 

From b25f223fe49c3bfc79c40f36a6fcd00d0d95bbd7 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Tue, 18 May 2021 16:11:14 +0100
Subject: [PATCH 248/442] Rewriting batching for threadsafety (#155)

This does make the batcher a critical section across job submission and
cleaving though.  If that becomes a problem, we should go back to
incoming and outgoing queues with a batcher thread.

Also removes blocking mode from native compiles.

Note that translateMultiple no longer guarantees great batching.  Guess
we could lease the mutex from ThreadsafeBatcher and create a session.

There is the risk that one sentence comes in at a time and each thread
grabs one sentence at a time instead of better batching.  Not sure what
to do about that other than some sort of Nagle algorithm.

Due to non-deterministic batching, even with one thread, the regression
tests will go haywire.
---
 src/translator/CMakeLists.txt         |   1 +
 src/translator/batch.h                |  11 +-
 src/translator/batch_translator.h     |   4 -
 src/translator/batcher.cpp            |   2 -
 src/translator/batcher.h              |   9 +-
 src/translator/pcqueue.h              | 343 --------------------------
 src/translator/service.cpp            | 115 ++-------
 src/translator/service.h              |  43 +---
 src/translator/threadsafe_batcher.cpp |  49 ++++
 src/translator/threadsafe_batcher.h   |  58 +++++
 10 files changed, 148 insertions(+), 487 deletions(-)
 delete mode 100644 src/translator/pcqueue.h
 create mode 100644 src/translator/threadsafe_batcher.cpp
 create mode 100644 src/translator/threadsafe_batcher.h

diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 1d48d5912..e007eadea 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -13,6 +13,7 @@ add_library(bergamot-translator STATIC
     batch.cpp
     annotation.cpp
     service.cpp
+    threadsafe_batcher.cpp
 )
 if (USE_WASM_COMPATIBLE_SOURCE)
   # Using wasm compatible sources should include this compile definition;
diff --git a/src/translator/batch.h b/src/translator/batch.h
index 5f86a2f5c..cfe5850f4 100644
--- a/src/translator/batch.h
+++ b/src/translator/batch.h
@@ -7,20 +7,12 @@
 namespace marian {
 namespace bergamot {
 
+// An empty batch is poison.
 class Batch {
 public:
   Batch() {}
   void clear() { sentences_.clear(); }
 
-  //  Methods to construct and determine poison.
-  static Batch poison() {
-    Batch batch;
-    batch.poison_ = true;
-    return batch;
-  }
-
-  bool isPoison() const { return poison_; }
-
   size_t size() const { return sentences_.size(); }
 
   void add(const RequestSentence &sentence);
@@ -42,7 +34,6 @@ class Batch {
   void log();
 
 private:
-  bool poison_{false};
   RequestSentences sentences_;
 };
 
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 048ba7749..395f29c94 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -13,10 +13,6 @@
 #include "translator/scorers.h"
 #include "vocabs.h"
 
-#ifndef WASM_COMPATIBLE_SOURCE
-#include "pcqueue.h"
-#endif
-
 namespace marian {
 namespace bergamot {
 
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 2b9c55168..678a81efe 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -20,8 +20,6 @@ void Batcher::addSentenceWithPriority(RequestSentence &sentence) {
   bucket_[bucket_id].insert(sentence);
 }
 
-bool Batcher::operator>>(Batch &batch) { return cleaveBatch(batch); }
-
 bool Batcher::cleaveBatch(Batch &batch) {
   // For now simply iterates on buckets and converts batches greedily.  This
   // has to be enhanced with optimizing over priority. The baseline
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index e5ac08604..5626b8e8d 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -7,10 +7,6 @@
 #include "definitions.h"
 #include "request.h"
 
-#ifndef WASM_COMPATIBLE_SOURCE
-#include "pcqueue.h"
-#endif
-
 #include <set>
 #include <vector>
 
@@ -26,7 +22,10 @@ class Batcher {
   void addSentenceWithPriority(RequestSentence &sentence);
   void addWholeRequest(Ptr<Request> request);
 
-  bool operator>>(Batch &batch); // alias for cleaveBatch
+  // indicate no more sentences will be added.  Does nothing here, for parity to threadsafe version.
+  void shutdown() {}
+
+  bool operator>>(Batch &batch) { return cleaveBatch(batch); }
 
 private:
   // Loads sentences with sentences compiled from (tentatively) multiple
diff --git a/src/translator/pcqueue.h b/src/translator/pcqueue.h
deleted file mode 100644
index 8d5d6e2a0..000000000
--- a/src/translator/pcqueue.h
+++ /dev/null
@@ -1,343 +0,0 @@
-#ifndef SRC_BERGAMOT_PCQUEUE_H_
-#define SRC_BERGAMOT_PCQUEUE_H_
-
-#include "common/logging.h"
-
-#include <algorithm>
-#include <cerrno>
-#include <iostream>
-#include <memory>
-#include <mutex>
-
-#ifdef __APPLE__
-#include <mach/semaphore.h>
-#include <mach/task.h>
-#include <mach/mach_traps.h>
-#include <mach/mach.h>
-#elif defined(__linux)
-#include <semaphore.h>
-#elif defined(_WIN32) || defined(_WIN64)
-#include <windows.h>
-#else
-#include <boost/interprocess/sync/interprocess_semaphore.hpp>
-#endif
-
-#if __GNUC__ >= 3
-#define UTIL_UNLIKELY(x) __builtin_expect(!!(x), 0)
-#else
-#define UTIL_UNLIKELY(x) (x)
-#endif
-
-namespace marian {
-namespace bergamot {
-
-/* OS X Maverick and Boost interprocess were doing "Function not implemented."
- * So this is my own wrapper around the mach kernel APIs.
- */
-#ifdef __APPLE__
-
-class Semaphore {
-  public:
-    explicit Semaphore(int value) : task_(mach_task_self()) {
-      ABORT_IF(KERN_SUCCESS != semaphore_create(task_, &back_, SYNC_POLICY_FIFO, value), "Could not create semaphore");
-    }
-
-    ~Semaphore() {
-      if (KERN_SUCCESS != semaphore_destroy(task_, back_)) {
-        std::cerr << "Could not destroy semaphore" << std::endl;
-        abort();
-      }
-    }
-
-    void wait() {
-      ABORT_IF(KERN_SUCCESS != semaphore_wait(back_), "Wait for semaphore failed");
-    }
-
-    void post() {
-      ABORT_IF(KERN_SUCCESS != semaphore_signal(back_), "Could not post to semaphore");
-    }
-
-  private:
-    semaphore_t back_;
-    task_t task_;
-};
-
-inline void WaitSemaphore(Semaphore &semaphore) {
-  semaphore.wait();
-}
-
-#elif defined(__linux)
-
-class Semaphore {
-  public:
-    explicit Semaphore(unsigned int value) {
-      ABORT_IF(sem_init(&sem_, 0, value), "Could not create semaphore");
-    }
-
-    ~Semaphore() {
-      if (-1 == sem_destroy(&sem_)) {
-        std::cerr << "Could not destroy semaphore" << std::endl;
-        abort();
-      }
-    }
-
-    void wait() {
-      while (-1 == sem_wait(&sem_)) {
-        ABORT_IF(errno != EINTR, "Wait for semaphore failed");
-      }
-    }
-
-    void post() {
-      ABORT_IF(-1 == sem_post(&sem_), "Could not post to semaphore");
-    }
-
-  private:
-    sem_t sem_;
-};
-
-inline void WaitSemaphore(Semaphore &semaphore) {
-  semaphore.wait();
-}
-
-#elif defined(_WIN32) || defined(_WIN64)
-
-class Semaphore {
-  public:
-    explicit Semaphore(LONG value) : sem_(CreateSemaphoreA(NULL, value, 2147483647, NULL)) {
-      ABORT_IF(!sem_, "Could not CreateSemaphore {}", GetLastError());
-    }
-
-    ~Semaphore() {
-      CloseHandle(sem_);
-    }
-
-
-    void wait() {
-      switch (WaitForSingleObject(sem_, INFINITE)) {
-        case WAIT_OBJECT_0:
-          return;
-        case WAIT_ABANDONED:
-          ABORT("A semaphore can't be abandoned, confused by Windows");
-        case WAIT_TIMEOUT:
-          ABORT("Timeout on an infinite wait?");
-        case WAIT_FAILED:
-          ABORT("Waiting on Semaphore failed {}", GetLastError());
-      }
-    }
-
-    void post() {
-      ABORT_IF(!ReleaseSemaphore(sem_, 1, NULL), "Failed to release Semaphore {}", GetLastError());
-    }
-
-  private:
-    HANDLE sem_;
-};
-
-inline void WaitSemaphore(Semaphore &semaphore) {
-  semaphore.wait();
-}
-
-#else
-typedef boost::interprocess::interprocess_semaphore Semaphore;
-
-inline void WaitSemaphore(Semaphore &on) {
-  while (1) {
-    try {
-      on.wait();
-      break;
-    } catch (boost::interprocess::interprocess_exception &e) {
-      if (e.get_native_error() != EINTR) {
-        throw;
-      }
-    }
-  }
-}
-
-#endif // Cases for semaphore support
-
-/**
- * Producer consumer queue safe for multiple producers and multiple consumers.
- * T must be default constructable and have operator=.
- * The value is copied twice for Consume(T &out) or three times for Consume(),
- * so larger objects should be passed via pointer.
- * Strong exception guarantee if operator= throws.  Undefined if semaphores
- * throw.
- */
-template <class T> class PCQueue {
- public:
-  explicit PCQueue(size_t size)
-   : empty_(size), used_(0),
-     storage_(new T[size]),
-     end_(storage_.get() + size),
-     produce_at_(storage_.get()),
-     consume_at_(storage_.get()) {}
-
-  // Add a value to the queue.
-  void Produce(const T &val) {
-    WaitSemaphore(empty_);
-    {
-      std::lock_guard<std::mutex> produce_lock(produce_at_mutex_);
-      try {
-        *produce_at_ = val;
-      } catch (...) {
-        empty_.post();
-        throw;
-      }
-      if (++produce_at_ == end_) produce_at_ = storage_.get();
-    }
-    used_.post();
-  }
-
-  // Add a value to the queue, but swap it into place.
-  void ProduceSwap(T &val) {
-    WaitSemaphore(empty_);
-    {
-      std::lock_guard<std::mutex> produce_lock(produce_at_mutex_);
-      try {
-        std::swap(*produce_at_, val);
-      } catch (...) {
-        empty_.post();
-        throw;
-      }
-      if (++produce_at_ == end_) produce_at_ = storage_.get();
-    }
-    used_.post();
-  }
-
-
-  // Consume a value, assigning it to out.
-  T& Consume(T &out) {
-    WaitSemaphore(used_);
-    {
-      std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
-      try {
-        out = *consume_at_;
-      } catch (...) {
-        used_.post();
-        throw;
-      }
-      if (++consume_at_ == end_) consume_at_ = storage_.get();
-    }
-    empty_.post();
-    return out;
-  }
-
-  // Consume a value, swapping it to out.
-  T& ConsumeSwap(T &out) {
-    WaitSemaphore(used_);
-    {
-      std::lock_guard<std::mutex> consume_lock(consume_at_mutex_);
-      try {
-        std::swap(out, *consume_at_);
-      } catch (...) {
-        used_.post();
-        throw;
-      }
-      if (++consume_at_ == end_) consume_at_ = storage_.get();
-    }
-    empty_.post();
-    return out;
-  }
-
-
-  // Convenience version of Consume that copies the value to return.
-  // The other version is faster.
-  T Consume() {
-    T ret;
-    Consume(ret);
-    return ret;
-  }
-
- private:
-  // Number of empty spaces in storage_.
-  Semaphore empty_;
-  // Number of occupied spaces in storage_.
-  Semaphore used_;
-
-  std::unique_ptr<T[]> storage_;
-
-  T *const end_;
-
-  // Index for next write in storage_.
-  T *produce_at_;
-  std::mutex produce_at_mutex_;
-
-  // Index for next read from storage_.
-  T *consume_at_;
-  std::mutex consume_at_mutex_;
-};
-
-template <class T> struct UnboundedPage {
-  UnboundedPage() : next(nullptr) {}
-  UnboundedPage *next;
-  T entries[1023];
-};
-
-template <class T> class UnboundedSingleQueue {
-  public:
-    UnboundedSingleQueue() : valid_(0) {
-      SetFilling(new UnboundedPage<T>());
-      SetReading(filling_);
-    }
-
-    void Produce(T &&val) {
-      if (filling_current_ == filling_end_) {
-        UnboundedPage<T> *next = new UnboundedPage<T>();
-        filling_->next = next;
-        SetFilling(next);
-      }
-      *(filling_current_++) = std::move(val);
-      valid_.post();
-    }
-
-    void Produce(const T &val) {
-      Produce(T(val));
-    }
-
-    T& Consume(T &out) {
-      WaitSemaphore(valid_);
-      if (reading_current_ == reading_end_) {
-        SetReading(reading_->next);
-      }
-      out = std::move(*(reading_current_++));
-      return out;
-    }
-
-    // Warning: very much a no-guarantees race-condition-rich implementation!
-    // But sufficient for our specific purpose: The single thread that consumes
-    // is also the only one that checks Empty, and knows that it's racing.
-    bool Empty() const {
-      return reading_current_ == filling_current_;
-    }
-
-  private:
-    void SetFilling(UnboundedPage<T> *to) {
-      filling_ = to;
-      filling_current_ = to->entries;
-      filling_end_ = filling_current_ + sizeof(to->entries) / sizeof(T);
-    }
-    void SetReading(UnboundedPage<T> *to) {
-      reading_.reset(to);
-      reading_current_ = to->entries;
-      reading_end_ = reading_current_ + sizeof(to->entries) / sizeof(T);
-    }
-
-    Semaphore valid_;
-
-    UnboundedPage<T> *filling_;
-
-    std::unique_ptr<UnboundedPage<T> > reading_;
-
-    T *filling_current_;
-    T *filling_end_;
-    T *reading_current_;
-    T *reading_end_;
-
-    UnboundedSingleQueue(const UnboundedSingleQueue &) = delete;
-    UnboundedSingleQueue &operator=(const UnboundedSingleQueue &) = delete;
-};
-
-} // namespace bergamot
-} // namespace marian
-
-#endif // SRC_BERGAMOT_PCQUEUE_H_
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 265695a38..6a459abab 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -12,91 +12,42 @@ Service::Service(Ptr<Options> options, MemoryBundle memoryBundle)
     : requestId_(0), options_(options),
       vocabs_(options, std::move(memoryBundle.vocabs)),
       text_processor_(vocabs_, options), batcher_(options),
-      numWorkers_(options->get<int>("cpu-threads")),
+      numWorkers_(std::max<int>(1, options->get<int>("cpu-threads"))),
       modelMemory_(std::move(memoryBundle.model)),
-      shortlistMemory_(std::move(memoryBundle.shortlist))
-#ifndef WASM_COMPATIBLE_SOURCE
-      // 0 elements in PCQueue is illegal and can lead to failures. Adding a
-      // guard to have at least one entry allocated. In the single-threaded
-      // case, while initialized pcqueue_ remains unused.
-      ,
-      pcqueue_(std::max<size_t>(1, numWorkers_))
+      shortlistMemory_(std::move(memoryBundle.shortlist)) 
+#ifdef WASM_COMPATIBLE_SOURCE
+      , blocking_translator_(DeviceId(0, DeviceType::cpu), vocabs_, options_, &modelMemory_, &shortlistMemory_)
 #endif
-{
-
-  if (numWorkers_ == 0) {
-    build_translators(options, /*numTranslators=*/1);
-    initialize_blocking_translator();
-  } else {
-    build_translators(options, numWorkers_);
-    initialize_async_translators();
-  }
-}
-
-void Service::build_translators(Ptr<Options> options, size_t numTranslators) {
-  translators_.reserve(numTranslators);
-  for (size_t cpuId = 0; cpuId < numTranslators; cpuId++) {
-    marian::DeviceId deviceId(cpuId, DeviceType::cpu);
-    translators_.emplace_back(deviceId, vocabs_, options, &modelMemory_, &shortlistMemory_);
-  }
-}
-
-void Service::initialize_blocking_translator() {
-  translators_.back().initialize();
-}
-
-void Service::blocking_translate() {
-  Batch batch;
-  while (batcher_ >> batch) {
-    auto &translator = translators_.back();
-    translator.translate(batch);
-  }
-}
-
-#ifndef WASM_COMPATIBLE_SOURCE
-void Service::initialize_async_translators() {
+  {
+#ifdef WASM_COMPATIBLE_SOURCE
+    blocking_translator_.initialize();
+#else
   workers_.reserve(numWorkers_);
-
   for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
-    auto &translator = translators_[cpuId];
-    workers_.emplace_back([&translator, this] {
+    workers_.emplace_back([cpuId, this] {
+      marian::DeviceId deviceId(cpuId, DeviceType::cpu);
+      BatchTranslator translator(deviceId, vocabs_, options_, &modelMemory_, &shortlistMemory_);
       translator.initialize();
-
-      // Run thread mainloop
       Batch batch;
-      Histories histories;
-      while (true) {
-        pcqueue_.ConsumeSwap(batch);
-        if (batch.isPoison()) {
-          return;
-        } else {
-          translator.translate(batch);
-        }
+      // Run thread mainloop
+      while (batcher_ >> batch) {
+        translator.translate(batch);
       }
     });
   }
+#endif
 }
 
-void Service::async_translate() {
+void Service::blockIfWASM() {
+#ifdef WASM_COMPATIBLE_SOURCE
   Batch batch;
+  // There's no need to do shutdown here because it's single threaded.
   while (batcher_ >> batch) {
-    pcqueue_.ProduceSwap(batch);
+    blocking_translator_.translate(batch);
   }
-}
-#else  // WASM_COMPATIBLE_SOURCE
-void Service::initialize_async_translators() {
-  ABORT("Cannot run in async mode without multithreading.");
+#endif
 }
 
-void Service::async_translate() {
-  ABORT("Cannot run in async mode without multithreading.");
-}
-#endif // WASM_COMPATIBLE_SOURCE
-
-std::future<Response> Service::translate(std::string &&input) {
-  ResponseOptions responseOptions;  // Hardcode responseOptions for now
-  return translate(std::move(input), responseOptions);
-}
 
 std::vector<Response>
 Service::translateMultiple(std::vector<std::string> &&inputs,
@@ -113,7 +64,7 @@ Service::translateMultiple(std::vector<std::string> &&inputs,
 
   // Dispatch is called once per request so compilation of sentences from
   // multiple Requests happen.
-  dispatchTranslate();
+  blockIfWASM();
 
   // Now wait for all Requests to complete, the future to fire and return the
   // compiled Responses, we can probably return the future, but WASM quirks(?).
@@ -148,30 +99,16 @@ std::future<Response> Service::translate(std::string &&input,
                                          ResponseOptions responseOptions) {
   std::future<Response> future =
       queueRequest(std::move(input), responseOptions);
-  dispatchTranslate();
+  blockIfWASM();
   return future;
 }
 
-void Service::dispatchTranslate() {
-  if (numWorkers_ == 0) {
-    blocking_translate();
-  } else {
-    async_translate();
-  }
-}
-
 Service::~Service() {
+  batcher_.shutdown();
 #ifndef WASM_COMPATIBLE_SOURCE
-  for (size_t workerId = 0; workerId < numWorkers_; workerId++) {
-
-    Batch poison = Batch::poison();
-    pcqueue_.ProduceSwap(poison);
-  }
-
-  for (size_t workerId = 0; workerId < numWorkers_; workerId++) {
-    if (workers_[workerId].joinable()) {
-      workers_[workerId].join();
-    }
+  for (std::thread &worker : workers_) {
+    assert(worker.joinable());
+    worker.join();
   }
 #endif
 }
diff --git a/src/translator/service.h b/src/translator/service.h
index a678d537c..8e2bc3f88 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -2,16 +2,16 @@
 #define SRC_BERGAMOT_SERVICE_H_
 
 #include "batch_translator.h"
-#include "batcher.h"
 #include "data/types.h"
 #include "response.h"
 #include "response_builder.h"
 #include "text_processor.h"
+#include "threadsafe_batcher.h"
 #include "translator/parser.h"
 #include "vocabs.h"
 
 #ifndef WASM_COMPATIBLE_SOURCE
-#include "pcqueue.h"
+#include <thread>
 #endif
 
 #include <queue>
@@ -83,12 +83,6 @@ class Service {
   /// asynchronous operation mode.
   ~Service();
 
-  /// To stay efficient and to refer to the string for alignments, expects
-  /// ownership be moved through `std::move(..)`
-  ///
-  ///  @param [in] source: rvalue reference of string to be translated.
-  std::future<Response> translate(std::string &&source);
-
   /// Translate an input, providing Options to construct Response. This is
   /// useful when one has to set/unset alignments or quality in the Response to
   /// save compute spent in constructing these objects.
@@ -98,7 +92,7 @@ class Service {
   /// some member in the Response, also specify any additional configurable
   /// parameters.
   std::future<Response> translate(std::string &&source,
-                                  ResponseOptions options);
+                                  ResponseOptions options = ResponseOptions());
 
   /// Translate multiple text-blobs in a single *blocking* API call, providing
   /// ResponseOptions which applies across all text-blobs dictating how to
@@ -116,7 +110,6 @@ class Service {
   /// @param [in] translationRequest: ResponseOptions indicating whether or not
   /// to include some member in the Response, also specify any additional
   /// configurable parameters.
-
   std::vector<Response>
   translateMultiple(std::vector<std::string> &&source,
                     ResponseOptions responseOptions);
@@ -134,24 +127,11 @@ class Service {
   /// Dispatch call to translate after inserting in queue
   void dispatchTranslate();
 
-  /// Build numTranslators number of translators with options from options
-  void build_translators(Ptr<Options> options, size_t numTranslators);
-  /// Initializes a blocking translator without using std::thread
-  void initialize_blocking_translator();
   /// Translates through direct interaction between batcher_ and translators_
-  void blocking_translate();
-
-  /// Launches multiple workers of translators using std::thread
-  /// Reduces to ABORT if called when not compiled WITH_PTHREAD
-  void initialize_async_translators();
-  /// Async translate produces to a producer-consumer queue as batches are
-  /// generated by Batcher. In another thread, the translators consume from
-  /// producer-consumer queue.
-  /// Reduces to ABORT if called when not compiled WITH_PTHREAD
-  void async_translate();
+  void blockIfWASM();
 
   /// Number of workers to launch.
-  size_t numWorkers_; // ORDER DEPENDENCY (pcqueue_)
+  size_t numWorkers_;
 
   /// Options object holding the options Service was instantiated with.
   Ptr<Options> options_;
@@ -161,12 +141,6 @@ class Service {
   /// Shortlist memory passed as bytes.
   AlignedMemory shortlistMemory_; // ORDER DEPENDENCY (translators_)
 
-  /// Holds instances of batch translators, just one in case
-  /// of single-threaded application, numWorkers_ in case of multithreaded
-  /// setting.
-  std::vector<BatchTranslator>
-      translators_; // ORDER DEPENDENCY (modelMemory_, shortlistMemory_)
-
   /// Stores requestId of active request. Used to establish
   /// ordering among requests and logging/book-keeping.
 
@@ -180,12 +154,13 @@ class Service {
 
   /// Batcher handles generation of batches from a request, subject to
   /// packing-efficiency and priority optimization heuristics.
-  Batcher batcher_;
+  ThreadsafeBatcher batcher_;
 
   // The following constructs are available providing full capabilities on a non
   // WASM platform, where one does not have to hide threads.
-#ifndef WASM_COMPATIBLE_SOURCE
-  PCQueue<Batch> pcqueue_; // ORDER DEPENDENCY (numWorkers_)
+#ifdef WASM_COMPATIBLE_SOURCE
+  BatchTranslator blocking_translator_; // ORDER DEPENDENCY (modelMemory_, shortlistMemory_)
+#else
   std::vector<std::thread> workers_;
 #endif // WASM_COMPATIBLE_SOURCE
 };
diff --git a/src/translator/threadsafe_batcher.cpp b/src/translator/threadsafe_batcher.cpp
new file mode 100644
index 000000000..e6b6a4668
--- /dev/null
+++ b/src/translator/threadsafe_batcher.cpp
@@ -0,0 +1,49 @@
+#ifndef WASM_COMPATIBLE_SOURCE
+#include "threadsafe_batcher.h"
+
+#include <cassert>
+
+namespace marian {
+namespace bergamot {
+
+ThreadsafeBatcher::ThreadsafeBatcher(Ptr<Options> options)
+  : backend_(options), enqueued_(0), shutdown_(false) {}
+
+ThreadsafeBatcher::~ThreadsafeBatcher() {
+  shutdown();
+}
+
+void ThreadsafeBatcher::addSentenceWithPriority(RequestSentence &sentence) {
+  std::unique_lock<std::mutex> lock(mutex_);
+  assert(!shutdown_);
+  backend_.addSentenceWithPriority(sentence);
+  ++enqueued_;
+  work_.notify_one();
+}
+
+void ThreadsafeBatcher::addWholeRequest(Ptr<Request> request) {
+  std::unique_lock<std::mutex> lock(mutex_);
+  assert(!shutdown_);
+  backend_.addWholeRequest(request);
+  enqueued_ += request->numSegments();
+  work_.notify_all();
+}
+
+void ThreadsafeBatcher::shutdown() {
+  std::unique_lock<std::mutex> lock(mutex_);
+  shutdown_ = true;
+  work_.notify_all();
+}
+
+bool ThreadsafeBatcher::operator>>(Batch &batch) {
+  std::unique_lock<std::mutex> lock(mutex_);
+  work_.wait(lock, [this](){ return enqueued_ || shutdown_; });
+  bool ret = backend_ >> batch;
+  assert(ret || shutdown_);
+  enqueued_ -= batch.size();
+  return ret;
+}
+
+} // namespace bergamot
+} // namespace marian
+#endif // WASM_COMPATIBLE_SOURCE
diff --git a/src/translator/threadsafe_batcher.h b/src/translator/threadsafe_batcher.h
new file mode 100644
index 000000000..994b13fa1
--- /dev/null
+++ b/src/translator/threadsafe_batcher.h
@@ -0,0 +1,58 @@
+/* Thread-safe wrapper around batcher. */
+#ifndef SRC_BERGAMOT_THREADSAFE_BATCHER_H_
+#define SRC_BERGAMOT_THREADSAFE_BATCHER_H_
+
+#include "batcher.h"
+#include "common/options.h"
+#include "definitions.h"
+
+#ifndef WASM_COMPATIBLE_SOURCE
+#include <condition_variable>
+#include <mutex>
+#endif
+
+namespace marian {
+namespace bergamot {
+
+#ifdef WASM_COMPATIBLE_SOURCE
+// No threads, no locks.
+typedef Batcher ThreadsafeBatcher;
+#else
+
+class ThreadsafeBatcher {
+  public:
+    explicit ThreadsafeBatcher(Ptr<Options> options);
+
+    ~ThreadsafeBatcher();
+
+    // Add sentences to be translated by calling these (see Batcher).  When
+    // done, call shutdown.
+    void addSentenceWithPriority(RequestSentence &sentence);
+    void addWholeRequest(Ptr<Request> request);
+    void shutdown();
+
+    // Get a batch out of the batcher.  Return false to shutdown worker.
+    bool operator>>(Batch &batch);
+
+  private:
+    Batcher backend_;
+
+    // Number of sentences in backend_;
+    size_t enqueued_;
+
+    // Are we shutting down?
+    bool shutdown_;
+
+    // Lock on this object.
+    std::mutex mutex_;
+
+    // Signaled when there are sentences to translate.
+    std::condition_variable work_;
+};
+
+#endif
+
+} // namespace bergamot
+} // namespace marian
+
+#endif // SRC_BERGAMOT_THREADSAFE_BATCHER_H_

From 89bd47342bda213490a7282bb1a71c27dd63d96d Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Wed, 19 May 2021 10:44:32 +0100
Subject: [PATCH 249/442] Use binary lexical shortlist in documentation (#152)

* Use binary lexical shortlist in documentation

* MKL/AppleAccelerate note

Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
Co-authored-by: Jerin Philip <jphilip@ed.ac.uk>
---
 doc/marian-integration.md | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index 411b55422..9e06e42f2 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -1,10 +1,20 @@
 # Building marian code for bergamot
 
 This document summarizes the minimal build instructions develop for the
-marian-code powering bergamot-translator.
+marian machine translation toolkit powering bergamot-translator.
 
 ## Build Instructions
 
+Marian CPU version requires Intel MKL or OpenBLAS. Both are free, but MKL is not open-sourced. Intel MKL is strongly recommended as it is faster. On Ubuntu 16.04 and newer it can be installed from the APT repositories.
+
+```bash
+wget -qO- 'https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB' | sudo apt-key add -
+sudo sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'
+sudo apt-get update
+sudo apt-get install intel-mkl-64bit-2020.0-088
+```
+On MacOS, apple accelerate framework will be used instead of MKL/OpenBLAS.
+
 ```
 $ git clone https://github.com/browsermt/bergamot-translator
 $ cd bergamot-translator
@@ -52,7 +62,7 @@ ARGS=(
         $MODEL_DIR/vocab.deen.spm # target-vocabulary
 
     # The following increases speed through one-best-decoding, shortlist and quantization.
-    --beam-size 1 --skip-cost --shortlist $MODEL_DIR/lex.s2t.gz 50 50 --int8shiftAlphaAll 
+    --beam-size 1 --skip-cost --shortlist $MODEL_DIR/lex.s2t.bin false --int8shiftAlphaAll 
 
     # Number of CPU threads (workers to launch). Parallelizes over cores and improves speed.
     # A value of 0 allows a path with no worker thread-launches and a single-thread.

From 7ad8d0a04d8e9d6c47436973fc79b0d8229598de Mon Sep 17 00:00:00 2001
From: Qianqian Zhu <qianqian.zhu@hotmail.com>
Date: Wed, 19 May 2021 20:11:20 +0100
Subject: [PATCH 250/442] initialise MemoryBundle members (#167)

---
 src/translator/definitions.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index bf1cb572b..fe434e74d 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -18,16 +18,16 @@ typedef AlignedVector<char> AlignedMemory;
 /// Memory bundle for all byte-arrays.
 /// Can be a set/subset of model, shortlist, vocabs and ssplitPrefixFile bytes.
 struct MemoryBundle {
-  AlignedMemory model;  ///< Byte-array of model (aligned to 256)
-  AlignedMemory shortlist;  ///< Byte-array of shortlist (aligned to 64)
+  AlignedMemory model{};  ///< Byte-array of model (aligned to 256)
+  AlignedMemory shortlist{};  ///< Byte-array of shortlist (aligned to 64)
 
   /// Vector of vocabulary memories (aligned to 64).
   /// If two vocabularies are the same (based on the filenames), two entries (shared
   /// pointers) will be generated which share the same AlignedMemory object.
-  std::vector<std::shared_ptr<AlignedMemory>> vocabs;
+  std::vector<std::shared_ptr<AlignedMemory>> vocabs{};
 
   /// @todo Not implemented yet
-  AlignedMemory ssplitPrefixFile;
+  AlignedMemory ssplitPrefixFile{};
 };
 
 } // namespace bergamot

From 9dcf6ab665511fbe62d7ef327a367d13693c0330 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 19 May 2021 21:50:21 +0100
Subject: [PATCH 251/442] Adding clang-format and updating existing sources to
 adhere (#151)

* Adding a first version of clang-format

* Adding run-clang-format.py

* Adding coding styles to workflow

* Fix indentation on coding-styles workflow

* run-clang-format.'py'

* -style -> --style in python

* Updating ColumnLimit: 120

* Format update with clang-format

* Revert "Format update with clang-format"

This reverts commit 5340b19eae8fcc91a2a79205e0b3dd65ad61ad4c.

* Apply update after sync

* Removing a few empty lines

* Removing one more empty line

* Removing empty in workflow file

* Updating README with coding style instructions

* clang-format-* provided in this repository doc update

Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
---
 .clang-format                                |   5 +
 .clang-format-ignore                         |   4 +
 .github/workflows/coding-styles.yml          |  26 ++
 doc/marian-integration.md                    |  16 +
 run-clang-format.py                          | 408 +++++++++++++++++++
 src/tests/annotation_tests.cpp               |  28 +-
 src/translator/annotation.cpp                |  19 +-
 src/translator/annotation.h                  |  67 ++-
 src/translator/batch.cpp                     |  12 +-
 src/translator/batch.h                       |  10 +-
 src/translator/batch_translator.cpp          |  65 +--
 src/translator/batch_translator.h            |  21 +-
 src/translator/batcher.cpp                   |   8 +-
 src/translator/batcher.h                     |  16 +-
 src/translator/byte_array_util.cpp           |  37 +-
 src/translator/byte_array_util.h             |   6 +-
 src/translator/definitions.h                 |  17 +-
 src/translator/parser.h                      |  29 +-
 src/translator/request.cpp                   |  36 +-
 src/translator/request.h                     |  31 +-
 src/translator/response.h                    |  22 +-
 src/translator/response_builder.cpp          |  66 ++-
 src/translator/response_builder.h            |  27 +-
 src/translator/response_options.h            |  10 +-
 src/translator/sentence_splitter.cpp         |  38 +-
 src/translator/sentence_splitter.h           |  15 +-
 src/translator/service.cpp                   |  48 +--
 src/translator/service.h                     |  41 +-
 src/translator/text_processor.cpp            |  36 +-
 src/translator/text_processor.h              |  25 +-
 src/translator/threadsafe_batcher.cpp        |  15 +-
 src/translator/threadsafe_batcher.h          |  46 +--
 src/translator/vocabs.h                      |  29 +-
 wasm/bindings/TranslationModelBindings.cpp   |  30 +-
 wasm/bindings/TranslationRequestBindings.cpp |   6 +-
 wasm/bindings/TranslationResultBindings.cpp  |   1 +
 36 files changed, 852 insertions(+), 464 deletions(-)
 create mode 100644 .clang-format
 create mode 100644 .clang-format-ignore
 create mode 100644 .github/workflows/coding-styles.yml
 create mode 100644 run-clang-format.py

diff --git a/.clang-format b/.clang-format
new file mode 100644
index 000000000..b7f746ef1
--- /dev/null
+++ b/.clang-format
@@ -0,0 +1,5 @@
+BasedOnStyle: Google
+
+# Maximum line length 80 is too low even for 1080p monitor.  @XapaJIaMnu
+# personally would like 120.
+ColumnLimit: 120
diff --git a/.clang-format-ignore b/.clang-format-ignore
new file mode 100644
index 000000000..50795cacb
--- /dev/null
+++ b/.clang-format-ignore
@@ -0,0 +1,4 @@
+3rd_party
+wasm/test_page
+src/translator/aligned.h
+src/translator/pcqueue.h
diff --git a/.github/workflows/coding-styles.yml b/.github/workflows/coding-styles.yml
new file mode 100644
index 000000000..176e8c7bf
--- /dev/null
+++ b/.github/workflows/coding-styles.yml
@@ -0,0 +1,26 @@
+name: "Coding Style"
+
+on: 
+  push:
+    branches: [ main, ci-sandbox ]
+  pull_request:
+    branches: [ main, ci-sandbox ]
+
+jobs:
+  clang-format:
+      name: "clang-format"
+      runs-on: ubuntu-latest
+      steps:
+        - name: Checkout
+          uses: actions/checkout@v2
+          with:
+            submodules: recursive
+
+        - name: Install dependencies
+          run: |
+            sudo apt-get update 
+            sudo apt-get install -y clang-format
+
+        - name: Run clang-format
+          run:
+              python3 run-clang-format.py --style file -r src wasm
diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index 9e06e42f2..d2957cbc9 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -93,3 +93,19 @@ ARGS=(
 </summary>
 </details>
 
+## Coding Style
+
+This repository contains C++ and JS source-files, of which C++ should adhere to
+the clang-format based style guidelines. You may configure your development
+environment to use the `.clang-format` and `.clang-format-ignore` files
+provided in the root folder of this repository with your preferred choice of
+editor/tooling.
+
+One simple and recommended method to get your code to adhere to this style is
+to issue the following command in the source-root of this repository, which is
+used to also check for the coding style in the CI.
+
+```bash
+python3 run-clang-format.py -i --style file -r src wasm
+```
+
diff --git a/run-clang-format.py b/run-clang-format.py
new file mode 100644
index 000000000..dcabaf1ec
--- /dev/null
+++ b/run-clang-format.py
@@ -0,0 +1,408 @@
+#!/usr/bin/env python
+"""A wrapper script around clang-format, suitable for linting multiple files
+and to use for continuous integration.
+
+This is an alternative API for the clang-format command line.
+It runs over multiple files and directories in parallel.
+A diff output is produced and a sensible exit code is returned.
+
+"""
+
+from __future__ import print_function, unicode_literals
+
+import argparse
+import codecs
+import difflib
+import fnmatch
+import io
+import errno
+import multiprocessing
+import os
+import signal
+import subprocess
+import sys
+import traceback
+
+from functools import partial
+
+try:
+    from subprocess import DEVNULL  # py3k
+except ImportError:
+    DEVNULL = open(os.devnull, "wb")
+
+
+DEFAULT_EXTENSIONS = 'c,h,C,H,cpp,hpp,cc,hh,c++,h++,cxx,hxx'
+DEFAULT_CLANG_FORMAT_IGNORE = '.clang-format-ignore'
+
+
+class ExitStatus:
+    SUCCESS = 0
+    DIFF = 1
+    TROUBLE = 2
+
+def excludes_from_file(ignore_file):
+    excludes = []
+    try:
+        with io.open(ignore_file, 'r', encoding='utf-8') as f:
+            for line in f:
+                if line.startswith('#'):
+                    # ignore comments
+                    continue
+                pattern = line.rstrip()
+                if not pattern:
+                    # allow empty lines
+                    continue
+                excludes.append(pattern)
+    except EnvironmentError as e:
+        if e.errno != errno.ENOENT:
+            raise
+    return excludes;
+
+def list_files(files, recursive=False, extensions=None, exclude=None):
+    if extensions is None:
+        extensions = []
+    if exclude is None:
+        exclude = []
+
+    out = []
+    for file in files:
+        if recursive and os.path.isdir(file):
+            for dirpath, dnames, fnames in os.walk(file):
+                fpaths = [os.path.join(dirpath, fname) for fname in fnames]
+                for pattern in exclude:
+                    # os.walk() supports trimming down the dnames list
+                    # by modifying it in-place,
+                    # to avoid unnecessary directory listings.
+                    dnames[:] = [
+                        x for x in dnames
+                        if
+                        not fnmatch.fnmatch(os.path.join(dirpath, x), pattern)
+                    ]
+                    fpaths = [
+                        x for x in fpaths if not fnmatch.fnmatch(x, pattern)
+                    ]
+                for f in fpaths:
+                    ext = os.path.splitext(f)[1][1:]
+                    if ext in extensions:
+                        out.append(f)
+        else:
+            out.append(file)
+    return out
+
+
+def make_diff(file, original, reformatted):
+    return list(
+        difflib.unified_diff(
+            original,
+            reformatted,
+            fromfile='{}\t(original)'.format(file),
+            tofile='{}\t(reformatted)'.format(file),
+            n=3))
+
+
+class DiffError(Exception):
+    def __init__(self, message, errs=None):
+        super(DiffError, self).__init__(message)
+        self.errs = errs or []
+
+
+class UnexpectedError(Exception):
+    def __init__(self, message, exc=None):
+        super(UnexpectedError, self).__init__(message)
+        self.formatted_traceback = traceback.format_exc()
+        self.exc = exc
+
+
+def run_clang_format_diff_wrapper(args, file):
+    try:
+        ret = run_clang_format_diff(args, file)
+        return ret
+    except DiffError:
+        raise
+    except Exception as e:
+        raise UnexpectedError('{}: {}: {}'.format(file, e.__class__.__name__,
+                                                  e), e)
+
+
+def run_clang_format_diff(args, file):
+    try:
+        with io.open(file, 'r', encoding='utf-8') as f:
+            original = f.readlines()
+    except IOError as exc:
+        raise DiffError(str(exc))
+    
+    if args.in_place:
+        invocation = [args.clang_format_executable, '-i', file]
+    else:
+        invocation = [args.clang_format_executable, file]
+
+    if args.style:
+        invocation.extend(['--style', args.style])
+
+    if args.dry_run:
+        print(" ".join(invocation))
+        return [], []
+
+    # Use of utf-8 to decode the process output.
+    #
+    # Hopefully, this is the correct thing to do.
+    #
+    # It's done due to the following assumptions (which may be incorrect):
+    # - clang-format will returns the bytes read from the files as-is,
+    #   without conversion, and it is already assumed that the files use utf-8.
+    # - if the diagnostics were internationalized, they would use utf-8:
+    #   > Adding Translations to Clang
+    #   >
+    #   > Not possible yet!
+    #   > Diagnostic strings should be written in UTF-8,
+    #   > the client can translate to the relevant code page if needed.
+    #   > Each translation completely replaces the format string
+    #   > for the diagnostic.
+    #   > -- http://clang.llvm.org/docs/InternalsManual.html#internals-diag-translation
+    #
+    # It's not pretty, due to Python 2 & 3 compatibility.
+    encoding_py3 = {}
+    if sys.version_info[0] >= 3:
+        encoding_py3['encoding'] = 'utf-8'
+
+    try:
+        proc = subprocess.Popen(
+            invocation,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            universal_newlines=True,
+            **encoding_py3)
+    except OSError as exc:
+        raise DiffError(
+            "Command '{}' failed to start: {}".format(
+                subprocess.list2cmdline(invocation), exc
+            )
+        )
+    proc_stdout = proc.stdout
+    proc_stderr = proc.stderr
+    if sys.version_info[0] < 3:
+        # make the pipes compatible with Python 3,
+        # reading lines should output unicode
+        encoding = 'utf-8'
+        proc_stdout = codecs.getreader(encoding)(proc_stdout)
+        proc_stderr = codecs.getreader(encoding)(proc_stderr)
+    # hopefully the stderr pipe won't get full and block the process
+    outs = list(proc_stdout.readlines())
+    errs = list(proc_stderr.readlines())
+    proc.wait()
+    if proc.returncode:
+        raise DiffError(
+            "Command '{}' returned non-zero exit status {}".format(
+                subprocess.list2cmdline(invocation), proc.returncode
+            ),
+            errs,
+        )
+    if args.in_place:
+        return [], errs
+    return make_diff(file, original, outs), errs
+
+
+def bold_red(s):
+    return '\x1b[1m\x1b[31m' + s + '\x1b[0m'
+
+
+def colorize(diff_lines):
+    def bold(s):
+        return '\x1b[1m' + s + '\x1b[0m'
+
+    def cyan(s):
+        return '\x1b[36m' + s + '\x1b[0m'
+
+    def green(s):
+        return '\x1b[32m' + s + '\x1b[0m'
+
+    def red(s):
+        return '\x1b[31m' + s + '\x1b[0m'
+
+    for line in diff_lines:
+        if line[:4] in ['--- ', '+++ ']:
+            yield bold(line)
+        elif line.startswith('@@ '):
+            yield cyan(line)
+        elif line.startswith('+'):
+            yield green(line)
+        elif line.startswith('-'):
+            yield red(line)
+        else:
+            yield line
+
+
+def print_diff(diff_lines, use_color):
+    if use_color:
+        diff_lines = colorize(diff_lines)
+    if sys.version_info[0] < 3:
+        sys.stdout.writelines((l.encode('utf-8') for l in diff_lines))
+    else:
+        sys.stdout.writelines(diff_lines)
+
+
+def print_trouble(prog, message, use_colors):
+    error_text = 'error:'
+    if use_colors:
+        error_text = bold_red(error_text)
+    print("{}: {} {}".format(prog, error_text, message), file=sys.stderr)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        '--clang-format-executable',
+        metavar='EXECUTABLE',
+        help='path to the clang-format executable',
+        default='clang-format')
+    parser.add_argument(
+        '--extensions',
+        help='comma separated list of file extensions (default: {})'.format(
+            DEFAULT_EXTENSIONS),
+        default=DEFAULT_EXTENSIONS)
+    parser.add_argument(
+        '-r',
+        '--recursive',
+        action='store_true',
+        help='run recursively over directories')
+    parser.add_argument(
+        '-d',
+        '--dry-run',
+        action='store_true',
+        help='just print the list of files')
+    parser.add_argument(
+        '-i',
+        '--in-place',
+        action='store_true',
+        help='format file instead of printing differences')
+    parser.add_argument('files', metavar='file', nargs='+')
+    parser.add_argument(
+        '-q',
+        '--quiet',
+        action='store_true',
+        help="disable output, useful for the exit code")
+    parser.add_argument(
+        '-j',
+        metavar='N',
+        type=int,
+        default=0,
+        help='run N clang-format jobs in parallel'
+        ' (default number of cpus + 1)')
+    parser.add_argument(
+        '--color',
+        default='auto',
+        choices=['auto', 'always', 'never'],
+        help='show colored diff (default: auto)')
+    parser.add_argument(
+        '-e',
+        '--exclude',
+        metavar='PATTERN',
+        action='append',
+        default=[],
+        help='exclude paths matching the given glob-like pattern(s)'
+        ' from recursive search')
+    parser.add_argument(
+        '--style',
+        help='formatting style to apply (LLVM, Google, Chromium, Mozilla, WebKit)')
+
+    args = parser.parse_args()
+
+    # use default signal handling, like diff return SIGINT value on ^C
+    # https://bugs.python.org/issue14229#msg156446
+    signal.signal(signal.SIGINT, signal.SIG_DFL)
+    try:
+        signal.SIGPIPE
+    except AttributeError:
+        # compatibility, SIGPIPE does not exist on Windows
+        pass
+    else:
+        signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+    colored_stdout = False
+    colored_stderr = False
+    if args.color == 'always':
+        colored_stdout = True
+        colored_stderr = True
+    elif args.color == 'auto':
+        colored_stdout = sys.stdout.isatty()
+        colored_stderr = sys.stderr.isatty()
+
+    version_invocation = [args.clang_format_executable, str("--version")]
+    try:
+        subprocess.check_call(version_invocation, stdout=DEVNULL)
+    except subprocess.CalledProcessError as e:
+        print_trouble(parser.prog, str(e), use_colors=colored_stderr)
+        return ExitStatus.TROUBLE
+    except OSError as e:
+        print_trouble(
+            parser.prog,
+            "Command '{}' failed to start: {}".format(
+                subprocess.list2cmdline(version_invocation), e
+            ),
+            use_colors=colored_stderr,
+        )
+        return ExitStatus.TROUBLE
+
+    retcode = ExitStatus.SUCCESS
+
+    excludes = excludes_from_file(DEFAULT_CLANG_FORMAT_IGNORE)
+    excludes.extend(args.exclude)
+
+    files = list_files(
+        args.files,
+        recursive=args.recursive,
+        exclude=excludes,
+        extensions=args.extensions.split(','))
+
+    if not files:
+        return
+
+    njobs = args.j
+    if njobs == 0:
+        njobs = multiprocessing.cpu_count() + 1
+    njobs = min(len(files), njobs)
+
+    if njobs == 1:
+        # execute directly instead of in a pool,
+        # less overhead, simpler stacktraces
+        it = (run_clang_format_diff_wrapper(args, file) for file in files)
+        pool = None
+    else:
+        pool = multiprocessing.Pool(njobs)
+        it = pool.imap_unordered(
+            partial(run_clang_format_diff_wrapper, args), files)
+        pool.close()
+    while True:
+        try:
+            outs, errs = next(it)
+        except StopIteration:
+            break
+        except DiffError as e:
+            print_trouble(parser.prog, str(e), use_colors=colored_stderr)
+            retcode = ExitStatus.TROUBLE
+            sys.stderr.writelines(e.errs)
+        except UnexpectedError as e:
+            print_trouble(parser.prog, str(e), use_colors=colored_stderr)
+            sys.stderr.write(e.formatted_traceback)
+            retcode = ExitStatus.TROUBLE
+            # stop at the first unexpected error,
+            # something could be very wrong,
+            # don't process all files unnecessarily
+            if pool:
+                pool.terminate()
+            break
+        else:
+            sys.stderr.writelines(errs)
+            if outs == []:
+                continue
+            if not args.quiet:
+                print_diff(outs, use_color=colored_stdout)
+            if retcode == ExitStatus.SUCCESS:
+                retcode = ExitStatus.DIFF
+    if pool:
+        pool.join()
+    return retcode
+
+
+if __name__ == '__main__':
+    sys.exit(main())
diff --git a/src/tests/annotation_tests.cpp b/src/tests/annotation_tests.cpp
index 0f02a7ad3..d7178f4df 100644
--- a/src/tests/annotation_tests.cpp
+++ b/src/tests/annotation_tests.cpp
@@ -1,8 +1,9 @@
-#include "catch.hpp"
-#include "translator/annotation.h"
 #include <random>
 #include <vector>
 
+#include "catch.hpp"
+#include "translator/annotation.h"
+
 using namespace marian::bergamot;
 
 TEST_CASE("Test Annotation API with random sentences") {
@@ -52,8 +53,7 @@ TEST_CASE("Test Annotation API with random sentences") {
   }
   std::string text;
   for (size_t idx = 0; idx < sentences; idx++) {
-    if (idx != 0)
-      text += "\n";
+    if (idx != 0) text += "\n";
 
     // Words can be zero, we need to support empty word sentences as well.
     size_t numWords = randomIntGen_() % maxWords;
@@ -96,8 +96,8 @@ TEST_CASE("Test Annotation API with random sentences") {
     groundTruthSentences.push_back((ByteRange){sentenceBegin, sentenceEnd});
   }
 
-  AnnotatedText testAnnotation(std::move(text)); // This the container we add through API and
-                                                 // check if the access is correct.
+  AnnotatedText testAnnotation(std::move(text));  // This the container we add through API and
+                                                  // check if the access is correct.
 
   // We prepare string_views now with the known ByteRanges and use the
   // string_view based AnnotatedText.addSentence(...) API to add sentences to
@@ -105,8 +105,7 @@ TEST_CASE("Test Annotation API with random sentences") {
   // the math underneath.
 
   if (debug) {
-    std::cout << "Inserting words onto container and save ground-truth-table:"
-              << std::endl;
+    std::cout << "Inserting words onto container and save ground-truth-table:" << std::endl;
   }
 
   std::vector<std::vector<marian::string_view>> wordStringViews;
@@ -115,8 +114,7 @@ TEST_CASE("Test Annotation API with random sentences") {
     std::vector<marian::string_view> wordByteRanges;
     bool first{true};
     for (auto &word : sentence) {
-      marian::string_view wordView(&testAnnotation.text[word.begin],
-                                   word.size());
+      marian::string_view wordView(&testAnnotation.text[word.begin], word.size());
       wordByteRanges.push_back(wordView);
       if (debug) {
         if (first) {
@@ -127,7 +125,8 @@ TEST_CASE("Test Annotation API with random sentences") {
         std::cout << std::string(wordView);
       }
     }
-    testAnnotation.recordExistingSentence(wordByteRanges.begin(), wordByteRanges.end(), testAnnotation.text.data() + sentence_iter->begin);
+    testAnnotation.recordExistingSentence(wordByteRanges.begin(), wordByteRanges.end(),
+                                          testAnnotation.text.data() + sentence_iter->begin);
     ++sentence_iter;
     wordStringViews.push_back(wordByteRanges);
     if (debug) {
@@ -136,9 +135,7 @@ TEST_CASE("Test Annotation API with random sentences") {
   }
 
   if (debug) {
-    std::cout
-        << "Inserting sentences onto container and save ground-truth-table"
-        << std::endl;
+    std::cout << "Inserting sentences onto container and save ground-truth-table" << std::endl;
   }
   std::vector<marian::string_view> sentenceStringViews;
   for (auto &sentenceByteRange : groundTruthSentences) {
@@ -203,7 +200,8 @@ TEST_CASE("Test Annotation API with random sentences") {
   // Sentence if the random test above does not cover it for some reason.
   int emptySentenceIdx = sentences;
   std::vector<marian::string_view> emptySentence;
-  testAnnotation.recordExistingSentence(emptySentence.begin(), emptySentence.end(), testAnnotation.text.data() + testAnnotation.text.size());
+  testAnnotation.recordExistingSentence(emptySentence.begin(), emptySentence.end(),
+                                        testAnnotation.text.data() + testAnnotation.text.size());
 
   // There are no words.
   CHECK(testAnnotation.numWords(emptySentenceIdx) == 0);
diff --git a/src/translator/annotation.cpp b/src/translator/annotation.cpp
index 90e02e00f..35f5c6916 100644
--- a/src/translator/annotation.cpp
+++ b/src/translator/annotation.cpp
@@ -1,4 +1,5 @@
 #include "annotation.h"
+
 #include <cassert>
 
 namespace marian {
@@ -9,7 +10,8 @@ AnnotatedText::AnnotatedText(std::string &&t) : text(std::move(t)) {
   annotation.token_begin_.back() = text.size();
 }
 
-void AnnotatedText::appendSentence(string_view prefix, std::vector<string_view>::iterator begin, std::vector<string_view>::iterator end) {
+void AnnotatedText::appendSentence(string_view prefix, std::vector<string_view>::iterator begin,
+                                   std::vector<string_view>::iterator end) {
   assert(annotation.token_begin_.back() == text.size());
   // We'll be adding tokens from the sentence and another gap.
   annotation.token_begin_.reserve(annotation.token_begin_.size() + (end - begin) + 1);
@@ -25,7 +27,7 @@ void AnnotatedText::appendSentence(string_view prefix, std::vector<string_view>:
   }
   if (begin != end) {
     text.append(begin->data(), (end - 1)->data() + (end - 1)->size());
-    assert(offset == text.size()); // Tokens should be contiguous.
+    assert(offset == text.size());  // Tokens should be contiguous.
   }
 
   // Add the gap after the sentence.  This is empty for now, but will be
@@ -39,7 +41,8 @@ void AnnotatedText::appendEndingWhitespace(string_view whitespace) {
   annotation.token_begin_.back() = text.size();
 }
 
-void AnnotatedText::recordExistingSentence(std::vector<string_view>::iterator begin, std::vector<string_view>::iterator end, const char *sentence_begin) {
+void AnnotatedText::recordExistingSentence(std::vector<string_view>::iterator begin,
+                                           std::vector<string_view>::iterator end, const char *sentence_begin) {
   assert(sentence_begin >= text.data());
   assert(sentence_begin <= text.data() + text.size());
   assert(begin == end || sentence_begin == begin->data());
@@ -48,9 +51,9 @@ void AnnotatedText::recordExistingSentence(std::vector<string_view>::iterator be
   // Clip off size token ending.
   annotation.token_begin_.resize(annotation.token_begin_.size() - 1);
   for (std::vector<string_view>::iterator i = begin; i != end; ++i) {
-    assert(i->data() >= text.data()); // In range.
-    assert(i->data() + i->size() <= text.data() + text.size()); // In range
-    assert(i + 1 == end || i->data() + i->size() == (i+1)->data()); // Contiguous
+    assert(i->data() >= text.data());                                  // In range.
+    assert(i->data() + i->size() <= text.data() + text.size());        // In range
+    assert(i + 1 == end || i->data() + i->size() == (i + 1)->data());  // Contiguous
     annotation.token_begin_.push_back(i->data() - text.data());
   }
   // Gap token after sentence.
@@ -65,5 +68,5 @@ void AnnotatedText::recordExistingSentence(std::vector<string_view>::iterator be
   annotation.token_begin_.push_back(text.size());
 }
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/annotation.h b/src/translator/annotation.h
index 555ab53ae..dde340ba3 100644
--- a/src/translator/annotation.h
+++ b/src/translator/annotation.h
@@ -1,11 +1,12 @@
 #ifndef BERGAMOT_SENTENCE_RANGES_H_
 #define BERGAMOT_SENTENCE_RANGES_H_
 
-#include "data/types.h"
 #include <cassert>
 #include <utility>
 #include <vector>
 
+#include "data/types.h"
+
 namespace marian {
 namespace bergamot {
 
@@ -18,8 +19,8 @@ struct ByteRange {
 };
 
 /// Annotation expresses sentence and token boundary information as ranges of
-/// bytes in a string, but does not itself own the string.  
-/// 
+/// bytes in a string, but does not itself own the string.
+///
 /// See also AnnotatedText, which owns Annotation and the string. AnnotatedText
 /// wraps these ByteRange functions to provide a string_view interface.
 ///
@@ -42,7 +43,7 @@ struct ByteRange {
 /// produced empty output).  That's fine, these are just empty ranges as you
 /// would expect.
 class Annotation {
-public:
+ public:
   /// Initially an empty string.  Populated by AnnotatedText.
   Annotation() {
     token_begin_.push_back(0);
@@ -62,25 +63,25 @@ class Annotation {
   /// `.numWords()` for `sentenceIdx` for defined behaviour.
   ByteRange word(size_t sentenceIdx, size_t wordIdx) const {
     size_t tokenIdx = gap_[sentenceIdx] + 1 + wordIdx;
-    return ByteRange {token_begin_[tokenIdx], token_begin_[tokenIdx + 1]};
+    return ByteRange{token_begin_[tokenIdx], token_begin_[tokenIdx + 1]};
   }
 
   /// Returns a ByteRange representing sentence corresponding to `sentenceIdx`.
   /// `sentenceIdx` follows 0-based indexing, and behaviour is defined only when
   /// less than `.numSentences()`.
   ByteRange sentence(size_t sentenceIdx) const {
-    return ByteRange {
-      token_begin_[gap_[sentenceIdx] + 1], /*end of whitespace before */
-      token_begin_[gap_[sentenceIdx + 1]] /*beginning of whitespace after */
+    return ByteRange{
+        token_begin_[gap_[sentenceIdx] + 1], /*end of whitespace before */
+        token_begin_[gap_[sentenceIdx + 1]]  /*beginning of whitespace after */
     };
   }
 
   ByteRange gap(size_t gapIdx) const {
     size_t tokenIdx = gap_[gapIdx];
-    return ByteRange {token_begin_[tokenIdx], token_begin_[tokenIdx + 1]};
+    return ByteRange{token_begin_[tokenIdx], token_begin_[tokenIdx + 1]};
   }
 
-private:
+ private:
   friend class AnnotatedText;
   /// Map from token index to byte offset at which it begins.  Token i is:
   ///   [token_begin_[i], token_begin_[i+1])
@@ -124,9 +125,9 @@ class Annotation {
 /// 3. Bind the text and annotations together, to move around as a meaningful
 /// unit.
 struct AnnotatedText {
-public:
-  std::string text;      ///< Blob of string elements in annotation refers to.
-  Annotation annotation; ///< sentence and (sub-) word annotations.
+ public:
+  std::string text;       ///< Blob of string elements in annotation refers to.
+  Annotation annotation;  ///< sentence and (sub-) word annotations.
 
   /// Construct an empty AnnotatedText. This is useful when the target string or
   /// ByteRanges are not known yet, but the public members can be used to
@@ -143,10 +144,8 @@ struct AnnotatedText {
   /// string_views.  Since this tracks only prefix, remember
   /// appendEndingWhitespace.
   /// The string_views must not already be in text.
-  void appendSentence(
-      string_view prefix,
-      std::vector<string_view>::iterator tokens_begin,
-      std::vector<string_view>::iterator tokens_end);
+  void appendSentence(string_view prefix, std::vector<string_view>::iterator tokens_begin,
+                      std::vector<string_view>::iterator tokens_end);
 
   /// Append the whitespace at the end of input. string_view must not be in
   /// text.
@@ -158,18 +157,14 @@ struct AnnotatedText {
   /// Normally the beginning of the sentence can be inferred from
   /// tokens_begin->data() but the tokens could be empty, so sentence_begin is
   /// required to know where the sentence is.
-  void recordExistingSentence(
-      std::vector<string_view>::iterator tokens_begin,
-      std::vector<string_view>::iterator tokens_end,
-      const char *sentence_begin);
+  void recordExistingSentence(std::vector<string_view>::iterator tokens_begin,
+                              std::vector<string_view>::iterator tokens_end, const char *sentence_begin);
 
   /// Returns the number of sentences in the annotation structure.
   const size_t numSentences() const { return annotation.numSentences(); }
 
   /// Returns number of words in the sentece identified by sentenceIdx.
-  const size_t numWords(size_t sentenceIdx) const {
-    return annotation.numWords(sentenceIdx);
-  }
+  const size_t numWords(size_t sentenceIdx) const { return annotation.numWords(sentenceIdx); }
 
   /// Returns a string_view representing wordIdx in sentenceIdx
   string_view word(size_t sentenceIdx, size_t wordIdx) const {
@@ -177,9 +172,7 @@ struct AnnotatedText {
   }
 
   /// Returns a string_view representing sentence corresponding to sentenceIdx.
-  string_view sentence(size_t sentenceIdx) const {
-    return asStringView(annotation.sentence(sentenceIdx));
-  }
+  string_view sentence(size_t sentenceIdx) const { return asStringView(annotation.sentence(sentenceIdx)); }
 
   /// Returns the string_view of the gap between two sentences in the container.
   ///
@@ -191,27 +184,21 @@ struct AnnotatedText {
   /// * For `i = N`, the gap between the last (N-1th) sentence and end of
   ///   text.
   /// @param sentenceIdx: Can be between `[0, numSentences()]`.
-  string_view gap(size_t sentenceIdx) const {
-    return asStringView(annotation.gap(sentenceIdx));
-  }
+  string_view gap(size_t sentenceIdx) const { return asStringView(annotation.gap(sentenceIdx)); }
 
   /// Returns a ByteRange representing wordIdx in sentenceIdx
-  ByteRange wordAsByteRange(size_t sentenceIdx, size_t wordIdx) const {
-    return annotation.word(sentenceIdx, wordIdx);
-  }
+  ByteRange wordAsByteRange(size_t sentenceIdx, size_t wordIdx) const { return annotation.word(sentenceIdx, wordIdx); }
 
   /// Returns a ByteRange representing sentence corresponding to sentenceIdx.
-  ByteRange sentenceAsByteRange(size_t sentenceIdx) const {
-    return annotation.sentence(sentenceIdx);
-  }
+  ByteRange sentenceAsByteRange(size_t sentenceIdx) const { return annotation.sentence(sentenceIdx); }
 
-private:
+ private:
   string_view asStringView(const ByteRange &byteRange) const {
     return string_view(text.data() + byteRange.begin, byteRange.size());
   }
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif //  BERGAMOT_SENTENCE_RANGES_H_
+#endif  //  BERGAMOT_SENTENCE_RANGES_H_
diff --git a/src/translator/batch.cpp b/src/translator/batch.cpp
index 82ebbfbf1..08d3d02c6 100644
--- a/src/translator/batch.cpp
+++ b/src/translator/batch.cpp
@@ -1,4 +1,5 @@
 #include "batch.h"
+
 #include "request.h"
 
 namespace marian {
@@ -11,18 +12,15 @@ void Batch::log() {
     maxLength = std::max(maxLength, static_cast<size_t>(sentence.numTokens()));
   }
 
-  LOG(info, "Batch(tokens={}, max-length={}, sentences_={})", numTokens,
-      maxLength, sentences_.size());
+  LOG(info, "Batch(tokens={}, max-length={}, sentences_={})", numTokens, maxLength, sentences_.size());
 }
 
-void Batch::add(const RequestSentence &sentence) {
-  sentences_.push_back(sentence);
-}
+void Batch::add(const RequestSentence &sentence) { sentences_.push_back(sentence); }
 
 void Batch::completeBatch(const Histories &histories) {
   for (size_t i = 0; i < sentences_.size(); i++) {
     sentences_[i].completeSentence(histories[i]);
   }
 }
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/batch.h b/src/translator/batch.h
index cfe5850f4..2f67252be 100644
--- a/src/translator/batch.h
+++ b/src/translator/batch.h
@@ -9,7 +9,7 @@ namespace bergamot {
 
 // An empty batch is poison.
 class Batch {
-public:
+ public:
   Batch() {}
   void clear() { sentences_.clear(); }
 
@@ -33,11 +33,11 @@ class Batch {
   // Convenience function to log batch-statistics. numTokens, max-length.
   void log();
 
-private:
+ private:
   RequestSentences sentences_;
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif // SRC_BERGAMOT_BATCH_H_
+#endif  // SRC_BERGAMOT_BATCH_H_
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index b35c4cedc..c27edc104 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -1,56 +1,63 @@
 #include "batch_translator.h"
+
 #include "batch.h"
+#include "byte_array_util.h"
 #include "common/logging.h"
 #include "data/corpus.h"
 #include "data/text_input.h"
 #include "translator/beam_search.h"
-#include "byte_array_util.h"
 
 namespace marian {
 namespace bergamot {
 
-BatchTranslator::BatchTranslator(DeviceId const device,
-                                 Vocabs &vocabs,
-                                 Ptr<Options> options,
-                                 const AlignedMemory* modelMemory,
-                                 const AlignedMemory* shortlistMemory)
-    : device_(device), options_(options), vocabs_(vocabs),
-    modelMemory_(modelMemory), shortlistMemory_(shortlistMemory) {}
+BatchTranslator::BatchTranslator(DeviceId const device, Vocabs &vocabs, Ptr<Options> options,
+                                 const AlignedMemory *modelMemory, const AlignedMemory *shortlistMemory)
+    : device_(device),
+      options_(options),
+      vocabs_(vocabs),
+      modelMemory_(modelMemory),
+      shortlistMemory_(shortlistMemory) {}
 
 void BatchTranslator::initialize() {
   // Initializes the graph.
-  bool check = options_->get<bool>("check-bytearray",false); // Flag holds whether validate the bytearray (model and shortlist)
+  bool check =
+      options_->get<bool>("check-bytearray", false);  // Flag holds whether validate the bytearray (model and shortlist)
   if (options_->hasAndNotEmpty("shortlist")) {
     int srcIdx = 0, trgIdx = 1;
-    bool shared_vcb = vocabs_.sources().front() == vocabs_.target(); // vocabs_->sources().front() is invoked as we currently only support one source vocab
+    bool shared_vcb =
+        vocabs_.sources().front() ==
+        vocabs_.target();  // vocabs_->sources().front() is invoked as we currently only support one source vocab
     if (shortlistMemory_->size() > 0 && shortlistMemory_->begin() != nullptr) {
       slgen_ = New<data::BinaryShortlistGenerator>(shortlistMemory_->begin(), shortlistMemory_->size(),
-                                                   vocabs_.sources().front(), vocabs_.target(),
-                                                   srcIdx, trgIdx, shared_vcb, check);
-    }
-    else {
+                                                   vocabs_.sources().front(), vocabs_.target(), srcIdx, trgIdx,
+                                                   shared_vcb, check);
+    } else {
       // Changed to BinaryShortlistGenerator to enable loading binary shortlist file
       // This class also supports text shortlist file
-      slgen_ = New<data::BinaryShortlistGenerator>(options_, vocabs_.sources().front(),
-                                                    vocabs_.target(), srcIdx,
-                                                    trgIdx, shared_vcb);
+      slgen_ = New<data::BinaryShortlistGenerator>(options_, vocabs_.sources().front(), vocabs_.target(), srcIdx,
+                                                   trgIdx, shared_vcb);
     }
   }
 
-  graph_ = New<ExpressionGraph>(true); // set the graph to be inference only
+  graph_ = New<ExpressionGraph>(true);  // set the graph to be inference only
   auto prec = options_->get<std::vector<std::string>>("precision", {"float32"});
   graph_->setDefaultElementType(typeFromString(prec[0]));
   graph_->setDevice(device_);
   graph_->getBackend()->configureDevice(options_);
   graph_->reserveWorkspaceMB(options_->get<size_t>("workspace"));
-  if (modelMemory_->size() > 0 && modelMemory_->begin() != nullptr) { // If we have provided a byte array that contains the model memory, we can initialise the model from there, as opposed to from reading in the config file
+  if (modelMemory_->size() > 0 &&
+      modelMemory_->begin() !=
+          nullptr) {  // If we have provided a byte array that contains the model memory, we can initialise the model
+                      // from there, as opposed to from reading in the config file
     ABORT_IF((uintptr_t)modelMemory_->begin() % 256 != 0,
              "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
     if (check) {
       ABORT_IF(!validateBinaryModel(*modelMemory_, modelMemory_->size()),
                "The binary file is invalid. Incomplete or corrupted download?");
     }
-    const std::vector<const void *> container = {modelMemory_->begin()}; // Marian supports multiple models initialised in this manner hence std::vector. However we will only ever use 1 during decoding.
+    const std::vector<const void *> container = {
+        modelMemory_->begin()};  // Marian supports multiple models initialised in this manner hence std::vector.
+                                 // However we will only ever use 1 during decoding.
     scorers_ = createScorers(options_, container);
   } else {
     scorers_ = createScorers(options_);
@@ -82,11 +89,9 @@ void BatchTranslator::translate(Batch &batch) {
   std::vector<size_t> sentenceIds;
   std::vector<int> maxDims;
   for (auto &ex : batchVector) {
-    if (maxDims.size() < ex.size())
-      maxDims.resize(ex.size(), 0);
+    if (maxDims.size() < ex.size()) maxDims.resize(ex.size(), 0);
     for (size_t i = 0; i < ex.size(); ++i) {
-      if (ex[i].size() > (size_t)maxDims[i])
-        maxDims[i] = (int)ex[i].size();
+      if (ex[i].size() > (size_t)maxDims[i]) maxDims[i] = (int)ex[i].size();
     }
     sentenceIds.push_back(ex.getId());
   }
@@ -96,8 +101,7 @@ void BatchTranslator::translate(Batch &batch) {
 
   std::vector<Ptr<SubBatch>> subBatches;
   for (size_t j = 0; j < maxDims.size(); ++j) {
-    subBatches.emplace_back(
-        New<SubBatch>(batchSize, maxDims[j], vocabs_.sources().at(j)));
+    subBatches.emplace_back(New<SubBatch>(batchSize, maxDims[j], vocabs_.sources().at(j)));
   }
 
   std::vector<size_t> words(maxDims.size(), 0);
@@ -111,17 +115,16 @@ void BatchTranslator::translate(Batch &batch) {
     }
   }
 
-  for (size_t j = 0; j < maxDims.size(); ++j)
-    subBatches[j]->setWords(words[j]);
+  for (size_t j = 0; j < maxDims.size(); ++j) subBatches[j]->setWords(words[j]);
 
   auto corpus_batch = Ptr<CorpusBatch>(new CorpusBatch(subBatches));
   corpus_batch->setSentenceIds(sentenceIds);
-  
+
   auto search = New<BeamSearch>(options_, scorers_, vocabs_.target());
 
   auto histories = std::move(search->search(graph_, corpus_batch));
   batch.completeBatch(histories);
 }
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
index 395f29c94..6a7fa9842 100644
--- a/src/translator/batch_translator.h
+++ b/src/translator/batch_translator.h
@@ -22,27 +22,28 @@ class BatchTranslator {
   // mainloop runs until until it receives poison from the PCQueue. Threads are
   // shut down in Service which calls join() on the threads.
 
-public:
+ public:
   /**
    * Initialise the marian translator.
    * @param device DeviceId that performs translation. Could be CPU or GPU
    * @param vocabs Vector that contains ptrs to two vocabs
    * @param options Marian options object
-   * @param modelMemory byte array (aligned to 256!!!) that contains the bytes of a model.bin. Provide a nullptr if not used.
+   * @param modelMemory byte array (aligned to 256!!!) that contains the bytes of a model.bin. Provide a nullptr if not
+   * used.
    * @param shortlistMemory byte array of shortlist (aligned to 64)
    */
-  explicit BatchTranslator(DeviceId const device, Vocabs &vocabs,
-                  Ptr<Options> options, const AlignedMemory* modelMemory, const AlignedMemory* shortlistMemory);
+  explicit BatchTranslator(DeviceId const device, Vocabs& vocabs, Ptr<Options> options,
+                           const AlignedMemory* modelMemory, const AlignedMemory* shortlistMemory);
 
   // convenience function for logging. TODO(jerin)
   std::string _identifier() { return "worker" + std::to_string(device_.no); }
-  void translate(Batch &batch);
+  void translate(Batch& batch);
   void initialize();
 
-private:
+ private:
   Ptr<Options> options_;
   DeviceId device_;
-  const Vocabs&  vocabs_;
+  const Vocabs& vocabs_;
   Ptr<ExpressionGraph> graph_;
   std::vector<Ptr<Scorer>> scorers_;
   Ptr<data::ShortlistGenerator const> slgen_;
@@ -50,7 +51,7 @@ class BatchTranslator {
   const AlignedMemory* shortlistMemory_{nullptr};
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif //  SRC_BERGAMOT_BATCH_TRANSLATOR_H_
+#endif  //  SRC_BERGAMOT_BATCH_TRANSLATOR_H_
diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 678a81efe..8735c4ba8 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -1,7 +1,9 @@
 #include "batcher.h"
+
+#include <cassert>
+
 #include "batch.h"
 #include "common/logging.h"
-#include <cassert>
 
 namespace marian {
 namespace bergamot {
@@ -55,5 +57,5 @@ void Batcher::addWholeRequest(Ptr<Request> request) {
   }
 }
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index 5626b8e8d..7735a0895 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -1,19 +1,19 @@
 #ifndef SRC_BERGAMOT_BATCHER_H_
 #define SRC_BERGAMOT_BATCHER_H_
 
+#include <set>
+#include <vector>
+
 #include "batch.h"
 #include "common/options.h"
 #include "data/corpus_base.h"
 #include "definitions.h"
 #include "request.h"
 
-#include <set>
-#include <vector>
-
 namespace marian {
 namespace bergamot {
 class Batcher {
-public:
+ public:
   explicit Batcher(Ptr<Options> options);
 
   // RequestSentence incorporates (tentative) notions of priority with each
@@ -27,7 +27,7 @@ class Batcher {
 
   bool operator>>(Batch &batch) { return cleaveBatch(batch); }
 
-private:
+ private:
   // Loads sentences with sentences compiled from (tentatively) multiple
   // requests optimizing for both padding and priority.
   bool cleaveBatch(Batch &batch);
@@ -36,7 +36,7 @@ class Batcher {
   size_t batchNumber_{0};
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif // SRC_BERGAMOT_BATCHER_H_
+#endif  // SRC_BERGAMOT_BATCHER_H_
diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index 69564d21b..247d1645b 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -1,5 +1,7 @@
 #include "byte_array_util.h"
+
 #include <stdlib.h>
+
 #include <iostream>
 #include <memory>
 
@@ -26,29 +28,30 @@ const T* get(const void*& current, uint64_t num = 1) {
   current = (const T*)current + num;
   return ptr;
 }
-} // Anonymous namespace
+}  // Anonymous namespace
 
 bool validateBinaryModel(const AlignedMemory& model, uint64_t fileSize) {
-  const void * current = model.begin();
-  uint64_t memoryNeeded = sizeof(uint64_t)*2; // We keep track of how much memory we would need if we have a complete file
+  const void* current = model.begin();
+  uint64_t memoryNeeded =
+      sizeof(uint64_t) * 2;  // We keep track of how much memory we would need if we have a complete file
   uint64_t numHeaders;
-  if (fileSize >= memoryNeeded) { // We have enough filesize to fetch the headers.
+  if (fileSize >= memoryNeeded) {  // We have enough filesize to fetch the headers.
     uint64_t binaryFileVersion = *get<uint64_t>(current);
-    numHeaders = *get<uint64_t>(current); // number of item headers that follow
+    numHeaders = *get<uint64_t>(current);  // number of item headers that follow
   } else {
     return false;
   }
-  memoryNeeded += numHeaders*sizeof(Header);
+  memoryNeeded += numHeaders * sizeof(Header);
   const Header* headers;
   if (fileSize >= memoryNeeded) {
-    headers = get<Header>(current, numHeaders); // read that many headers
+    headers = get<Header>(current, numHeaders);  // read that many headers
   } else {
     return false;
   }
 
   // Calculate how many bytes we are going to for reading just the names and the shape
   for (uint64_t i = 0; i < numHeaders; i++) {
-    memoryNeeded += headers[i].nameLength + headers[i].shapeLength*sizeof(int);
+    memoryNeeded += headers[i].nameLength + headers[i].shapeLength * sizeof(int);
     // Advance the pointers.
     get<char>(current, headers[i].nameLength);
     get<int>(current, headers[i].shapeLength);
@@ -58,7 +61,7 @@ bool validateBinaryModel(const AlignedMemory& model, uint64_t fileSize) {
   // Read that in, before calculating the actual tensor memory requirements.
   uint64_t aligned_offset;
   if (fileSize >= memoryNeeded) {
-    aligned_offset = *get<uint64_t>(current); // Offset to align memory to 256 size
+    aligned_offset = *get<uint64_t>(current);  // Offset to align memory to 256 size
     memoryNeeded += aligned_offset + sizeof(uint64_t);
   } else {
     return false;
@@ -77,17 +80,17 @@ bool validateBinaryModel(const AlignedMemory& model, uint64_t fileSize) {
   }
 }
 
-AlignedMemory loadFileToMemory(const std::string& path, size_t alignment){
+AlignedMemory loadFileToMemory(const std::string& path, size_t alignment) {
   uint64_t fileSize = filesystem::fileSize(path);
   io::InputFileStream in(path);
   ABORT_IF(in.bad(), "Failed opening file stream: {}", path);
   AlignedMemory alignedMemory(fileSize, alignment);
-  in.read(reinterpret_cast<char *>(alignedMemory.begin()), fileSize);
+  in.read(reinterpret_cast<char*>(alignedMemory.begin()), fileSize);
   ABORT_IF(alignedMemory.size() != fileSize, "Error reading file {}", path);
   return alignedMemory;
 }
 
-AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options){
+AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options) {
   auto models = options->get<std::vector<std::string>>("models");
   ABORT_IF(models.size() != 1, "Loading multiple binary models is not supported for now as it is not necessary.");
   marian::filesystem::Path modelPath(models[0]);
@@ -96,14 +99,14 @@ AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options){
   return alignedMemory;
 }
 
-AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options){
+AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options) {
   auto shortlist = options->get<std::vector<std::string>>("shortlist");
   ABORT_IF(shortlist.empty(), "No path to shortlist file is given.");
   return loadFileToMemory(shortlist[0], 64);
 }
 
 void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
-                               std::vector<std::shared_ptr<AlignedMemory>>& vocabMemories){
+                               std::vector<std::shared_ptr<AlignedMemory>>& vocabMemories) {
   auto vfiles = options->get<std::vector<std::string>>("vocabs");
   ABORT_IF(vfiles.size() < 2, "Insufficient number of vocabularies.");
   vocabMemories.resize(vfiles.size());
@@ -117,7 +120,7 @@ void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
   }
 }
 
-MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options){
+MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options) {
   MemoryBundle memoryBundle;
   memoryBundle.model = getModelMemoryFromConfig(options);
   memoryBundle.shortlist = getShortlistMemoryFromConfig(options);
@@ -125,5 +128,5 @@ MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options){
   return memoryBundle;
 }
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/byte_array_util.h b/src/translator/byte_array_util.h
index 14c79b3b5..63aa3a6ce 100644
--- a/src/translator/byte_array_util.h
+++ b/src/translator/byte_array_util.h
@@ -1,5 +1,5 @@
-#include "marian.h"
 #include "definitions.h"
+#include "marian.h"
 
 namespace marian {
 namespace bergamot {
@@ -11,5 +11,5 @@ void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
                                std::vector<std::shared_ptr<AlignedMemory>>& vocabMemories);
 bool validateBinaryModel(const AlignedMemory& model, uint64_t fileSize);
 MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options);
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index fe434e74d..14552a3c0 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -1,10 +1,11 @@
 #ifndef SRC_BERGAMOT_DEFINITIONS_H_
 #define SRC_BERGAMOT_DEFINITIONS_H_
 
+#include <vector>
+
+#include "aligned.h"
 #include "data/types.h"
 #include "data/vocab_base.h"
-#include "aligned.h"
-#include <vector>
 
 namespace marian {
 namespace bergamot {
@@ -18,7 +19,7 @@ typedef AlignedVector<char> AlignedMemory;
 /// Memory bundle for all byte-arrays.
 /// Can be a set/subset of model, shortlist, vocabs and ssplitPrefixFile bytes.
 struct MemoryBundle {
-  AlignedMemory model{};  ///< Byte-array of model (aligned to 256)
+  AlignedMemory model{};      ///< Byte-array of model (aligned to 256)
   AlignedMemory shortlist{};  ///< Byte-array of shortlist (aligned to 64)
 
   /// Vector of vocabulary memories (aligned to 64).
@@ -30,8 +31,8 @@ struct MemoryBundle {
   AlignedMemory ssplitPrefixFile{};
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
 // @TODO at the moment the usage of string_view in this repository is a hot mess and a disaster waiting to happen.
 // ssplit uses std::string_view if the compiler supports c++17, else falls back to c++11 and absl::string_view
@@ -44,10 +45,10 @@ struct MemoryBundle {
 #if defined(__GNUC__) && __GNUC__ < 6 && !defined(__clang__)
 #include <experimental/string_view>
 namespace std {
-  using string_view = std::experimental::string_view;
-} // namespace std
+using string_view = std::experimental::string_view;
+}  // namespace std
 #else
 #include <string_view>
 #endif
 
-#endif // SRC_BERGAMOT_DEFINITIONS_H_
+#endif  // SRC_BERGAMOT_DEFINITIONS_H_
diff --git a/src/translator/parser.h b/src/translator/parser.h
index 207890c84..b2b0a80d4 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -12,26 +12,21 @@ namespace bergamot {
 
 inline marian::ConfigParser createConfigParser() {
   marian::ConfigParser cp(marian::cli::mode::translation);
-  cp.addOption<std::string>(
-      "--ssplit-prefix-file", "Bergamot Options",
-      "File with nonbreaking prefixes for sentence splitting.");
+  cp.addOption<std::string>("--ssplit-prefix-file", "Bergamot Options",
+                            "File with nonbreaking prefixes for sentence splitting.");
 
-  cp.addOption<std::string>("--ssplit-mode", "Server Options",
-                            "[paragraph, sentence, wrapped_text]", "paragraph");
+  cp.addOption<std::string>("--ssplit-mode", "Server Options", "[paragraph, sentence, wrapped_text]", "paragraph");
 
-  cp.addOption<int>(
-      "--max-length-break", "Bergamot Options",
-      "Maximum input tokens to be processed in a single sentence.", 128);
+  cp.addOption<int>("--max-length-break", "Bergamot Options",
+                    "Maximum input tokens to be processed in a single sentence.", 128);
 
-  cp.addOption<bool>(
-      "--check-bytearray", "Bergamot Options",
-      "Flag holds whether to check the content of the bytearray (true by default)", true);
+  cp.addOption<bool>("--check-bytearray", "Bergamot Options",
+                     "Flag holds whether to check the content of the bytearray (true by default)", true);
 
-    return cp;
+  return cp;
 }
 
-inline std::shared_ptr<marian::Options>
-parseOptions(const std::string &config, bool validate = true) {
+inline std::shared_ptr<marian::Options> parseOptions(const std::string &config, bool validate = true) {
   marian::Options options;
 
   // @TODO(jerinphilip) There's something off here, @XapaJIaMnu suggests
@@ -67,7 +62,7 @@ parseOptions(const std::string &config, bool validate = true) {
   return std::make_shared<marian::Options>(options);
 }
 
-} //  namespace bergamot
-} //  namespace marian
+}  //  namespace bergamot
+}  //  namespace marian
 
-#endif //  SRC_BERGAMOT_PARSER_H
+#endif  //  SRC_BERGAMOT_PARSER_H
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 37b164f8b..9bdae9f74 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -1,23 +1,22 @@
 #include "request.h"
-#include "definitions.h"
-#include "response.h"
-#include "annotation.h"
-
-#include "common/logging.h"
 
 #include <string>
 
+#include "annotation.h"
+#include "common/logging.h"
+#include "definitions.h"
+#include "response.h"
+
 namespace marian {
 namespace bergamot {
 
 // -----------------------------------------------------------------
-Request::Request(size_t Id, Segments &&segments,
-                 ResponseBuilder &&responseBuilder)
-    : Id_(Id), segments_(std::move(segments)),
+Request::Request(size_t Id, Segments &&segments, ResponseBuilder &&responseBuilder)
+    : Id_(Id),
+      segments_(std::move(segments)),
       responseBuilder_(std::move(responseBuilder))
 
 {
-
   counter_ = segments_.size();
   histories_.resize(segments_.size(), nullptr);
 
@@ -31,9 +30,7 @@ Request::Request(size_t Id, Segments &&segments,
 
 size_t Request::numSegments() const { return segments_.size(); }
 
-size_t Request::segmentTokens(size_t index) const {
-  return (segments_[index].size());
-}
+size_t Request::segmentTokens(size_t index) const { return (segments_[index].size()); }
 
 Segment Request::getSegment(size_t index) const { return segments_[index]; }
 
@@ -56,12 +53,9 @@ bool Request::operator<(const Request &b) const {
 
 // ------------------------------------------------------------------
 
-RequestSentence::RequestSentence(size_t index, Ptr<Request> request)
-    : index_(index), request_(request) {}
+RequestSentence::RequestSentence(size_t index, Ptr<Request> request) : index_(index), request_(request) {}
 
-size_t RequestSentence::numTokens() const {
-  return (request_->segmentTokens(index_));
-}
+size_t RequestSentence::numTokens() const { return (request_->segmentTokens(index_)); }
 
 void RequestSentence::completeSentence(Ptr<History> history) {
   // Relays completeSentence into request's processHistory, using index
@@ -69,9 +63,7 @@ void RequestSentence::completeSentence(Ptr<History> history) {
   request_->processHistory(index_, history);
 }
 
-Segment RequestSentence::getUnderlyingSegment() const {
-  return request_->getSegment(index_);
-}
+Segment RequestSentence::getUnderlyingSegment() const { return request_->getSegment(index_); }
 
 bool operator<(const RequestSentence &a, const RequestSentence &b) {
   // Operator overload for usage in priority-queue / set.
@@ -83,5 +75,5 @@ bool operator<(const RequestSentence &a, const RequestSentence &b) {
 
 // ----------------------------------------------------------------------
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/request.h b/src/translator/request.h
index 798368099..a2ea1af86 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -1,20 +1,18 @@
 #ifndef SRC_BERGAMOT_REQUEST_H_
 #define SRC_BERGAMOT_REQUEST_H_
 
-#include "definitions.h"
-#include "response.h"
-#include "response_builder.h"
-#include "annotation.h"
+#include <cassert>
+#include <future>
+#include <vector>
 
+#include "annotation.h"
 #include "common/logging.h"
 #include "data/types.h"
+#include "definitions.h"
+#include "response.h"
+#include "response_builder.h"
 #include "translator/beam_search.h"
 
-#include <cassert>
-
-#include <future>
-#include <vector>
-
 namespace marian {
 namespace bergamot {
 
@@ -37,7 +35,7 @@ namespace bergamot {
 /// corresponding to the Request and set value of the promise which triggers the
 /// future at client.
 class Request {
-public:
+ public:
   /// Constructs an internal representation of the Request identified by Id,
   /// processed Segments and accepts a callback (ResponseBuilder) which builds
   /// the Response upon completion of the Request.
@@ -69,7 +67,7 @@ class Request {
   /// compiled from requests.
   void processHistory(size_t index, Ptr<History> history);
 
-private:
+ private:
   size_t Id_;
 
   /// Multiple translation-workers can concurrently access the same Request. The
@@ -95,8 +93,7 @@ class Request {
 /// within Request, while batching mechanism (Batcher) compiles Batch from
 /// RequestSentence-s coming from different Requests.
 class RequestSentence {
-
-public:
+ public:
   RequestSentence(size_t, Ptr<Request>);
 
   /// Number of tokens in the segment this RequestSentence represents. Used to
@@ -112,14 +109,14 @@ class RequestSentence {
 
   friend bool operator<(const RequestSentence &a, const RequestSentence &b);
 
-private:
+ private:
   size_t index_;
   Ptr<Request> request_;
 };
 
 typedef std::vector<RequestSentence> RequestSentences;
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif // SRC_BERGAMOT_REQUEST_H_
+#endif  // SRC_BERGAMOT_REQUEST_H_
diff --git a/src/translator/response.h b/src/translator/response.h
index 5d4ea7860..5c92441fe 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -1,16 +1,16 @@
 #ifndef SRC_BERGAMOT_RESPONSE_H_
 #define SRC_BERGAMOT_RESPONSE_H_
 
+#include <cassert>
+#include <string>
+#include <vector>
+
+#include "annotation.h"
 #include "data/alignment.h"
 #include "data/types.h"
 #include "definitions.h"
-#include "annotation.h"
 #include "translator/beam_search.h"
 
-#include <cassert>
-#include <string>
-#include <vector>
-
 namespace marian {
 namespace bergamot {
 
@@ -18,9 +18,9 @@ namespace bergamot {
 /// internals but is brought here to maintain translator
 /// agnosticism/independence.
 struct Point {
-  size_t src; ///< Index pointing to source ByteRange
-  size_t tgt; ///< Index pointing to target ByteRange
-  float prob; ///< Score between [0, 1] on indicating degree of alignment.
+  size_t src;  ///< Index pointing to source ByteRange
+  size_t tgt;  ///< Index pointing to target ByteRange
+  float prob;  ///< Score between [0, 1] on indicating degree of alignment.
 };
 
 /// Alignment is a sparse matrix, where Points represent entries with values.
@@ -69,7 +69,7 @@ struct Response {
 
   const std::string &getTranslatedText() const { return target.text; }
 };
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif // SRC_BERGAMOT_RESPONSE_H_
+#endif  // SRC_BERGAMOT_RESPONSE_H_
diff --git a/src/translator/response_builder.cpp b/src/translator/response_builder.cpp
index b2f561b6d..6d60b6c48 100644
--- a/src/translator/response_builder.cpp
+++ b/src/translator/response_builder.cpp
@@ -1,17 +1,17 @@
 #include "response_builder.h"
+
 #include "response_options.h"
 
 namespace marian {
 namespace bergamot {
 
-void ResponseBuilder::buildQualityScores(Histories &histories,
-                                         Response &response) {
+void ResponseBuilder::buildQualityScores(Histories &histories, Response &response) {
   std::vector<Quality> qualityScores;
   for (auto &history : histories) {
     // TODO(jerin): Change hardcode of nBest = 1
     NBestList onebest = history->nBest(1);
 
-    Result result = onebest[0]; // Expecting only one result;
+    Result result = onebest[0];  // Expecting only one result;
     Words words = std::get<0>(result);
     auto hyp = std::get<1>(result);
     // Quality scores: Sequence level is obtained as normalized path scores.
@@ -20,18 +20,16 @@ void ResponseBuilder::buildQualityScores(Histories &histories,
     auto normalizedPathScore = std::get<2>(result);
     auto wordQualities = hyp->tracebackWordScores();
     wordQualities.pop_back();
-    response.qualityScores.push_back(
-        Quality{normalizedPathScore, wordQualities});
+    response.qualityScores.push_back(Quality{normalizedPathScore, wordQualities});
   }
 }
 
-void ResponseBuilder::buildAlignments(Histories &histories,
-                                      Response &response) {
+void ResponseBuilder::buildAlignments(Histories &histories, Response &response) {
   for (auto &history : histories) {
     // TODO(jerin): Change hardcode of nBest = 1
     NBestList onebest = history->nBest(1);
 
-    Result result = onebest[0]; // Expecting only one result;
+    Result result = onebest[0];  // Expecting only one result;
     Words words = std::get<0>(result);
     // Alignments
     // TODO(jerinphilip): The following double conversion might not be
@@ -40,8 +38,7 @@ void ResponseBuilder::buildAlignments(Histories &histories,
     auto hyp = std::get<1>(result);
     auto softAlignment = hyp->tracebackAlignment();
     auto threshold = responseOptions_.alignmentThreshold;
-    auto hardAlignment =
-        data::ConvertSoftAlignToHardAlign(softAlignment, threshold);
+    auto hardAlignment = data::ConvertSoftAlignToHardAlign(softAlignment, threshold);
     Alignment unified_alignment;
     for (auto &p : hardAlignment) {
       unified_alignment.emplace_back(Point{p.srcPos, p.tgtPos, p.prob});
@@ -51,8 +48,7 @@ void ResponseBuilder::buildAlignments(Histories &histories,
   }
 }
 
-void ResponseBuilder::buildTranslatedText(Histories &histories,
-                                          Response &response) {
+void ResponseBuilder::buildTranslatedText(Histories &histories, Response &response) {
   // Reserving length at least as much as source_ seems like a reasonable
   // thing to do to avoid reallocations.
   response.target.text.reserve(response.source.text.size());
@@ -63,7 +59,7 @@ void ResponseBuilder::buildTranslatedText(Histories &histories,
     auto &history = histories[sentenceIdx];
     NBestList onebest = history->nBest(1);
 
-    Result result = onebest[0]; // Expecting only one result;
+    Result result = onebest[0];  // Expecting only one result;
     Words words = std::get<0>(result);
 
     std::string decoded;
@@ -71,31 +67,31 @@ void ResponseBuilder::buildTranslatedText(Histories &histories,
     vocabs_.target()->decodeWithByteRanges(words, decoded, targetSentenceMappings);
 
     switch (responseOptions_.concatStrategy) {
-    case ConcatStrategy::FAITHFUL: {
-      // For each sentence, prepend the filler text between the corresponding
-      // source-sentence and the source-sentence before.
-      string_view pre = response.source.gap(sentenceIdx);
-      response.target.appendSentence(pre, targetSentenceMappings.begin(), targetSentenceMappings.end());
-
-      // If this is the last history to be decoded and translated-text
-      // constructed, append the text till the end, which could be spaces or
-      // empty.
-      if (sentenceIdx + 1 == histories.size()) {
-        response.target.appendEndingWhitespace(response.source.gap(sentenceIdx + 1));
+      case ConcatStrategy::FAITHFUL: {
+        // For each sentence, prepend the filler text between the corresponding
+        // source-sentence and the source-sentence before.
+        string_view pre = response.source.gap(sentenceIdx);
+        response.target.appendSentence(pre, targetSentenceMappings.begin(), targetSentenceMappings.end());
+
+        // If this is the last history to be decoded and translated-text
+        // constructed, append the text till the end, which could be spaces or
+        // empty.
+        if (sentenceIdx + 1 == histories.size()) {
+          response.target.appendEndingWhitespace(response.source.gap(sentenceIdx + 1));
+        }
+        break;
+      }
+      case ConcatStrategy::SPACE: {
+        string_view delimiter = (sentenceIdx == 0) ? "" : " ";
+        response.target.appendSentence(delimiter, targetSentenceMappings.begin(), targetSentenceMappings.end());
+        break;
       }
-      break;
-    }
-    case ConcatStrategy::SPACE: {
-      string_view delimiter = (sentenceIdx == 0) ? "" : " ";
-      response.target.appendSentence(delimiter, targetSentenceMappings.begin(), targetSentenceMappings.end());
-      break;
-    }
 
-    default:
-      ABORT("Unknown concat-strategy");
+      default:
+        ABORT("Unknown concat-strategy");
     }
   }
 }
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
index b8a8dd40d..84ea09ad5 100644
--- a/src/translator/response_builder.h
+++ b/src/translator/response_builder.h
@@ -19,16 +19,14 @@ namespace bergamot {
 /// paragraph).
 
 class ResponseBuilder {
-public:
+ public:
   /// @param [in] responseOptions: ResponseOptions, indicating what to include
   /// or not in the response and any additional configurable parameters.
   /// @param [in] vocabs: marian vocab object (used in decoding)
   /// @param [in] promise: promise to set with the constructed Response.
-  ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source,
-                  Vocabs &vocabs,
+  ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source, Vocabs &vocabs,
                   std::promise<Response> &&promise)
-      : responseOptions_(responseOptions), source_(std::move(source)),
-        vocabs_(vocabs), promise_(std::move(promise)) {}
+      : responseOptions_(responseOptions), source_(std::move(source)), vocabs_(vocabs), promise_(std::move(promise)) {}
 
   /// Constructs and sets the promise of a Response object from obtained
   /// histories after translating.
@@ -38,8 +36,7 @@ class ResponseBuilder {
     // TODO(jerinphilip) load ResponseOptions into options and turn build
     // functions on or off.
     // responseOptions_ is unused, but we can try something here.
-    ABORT_IF(source_.numSentences() != histories.size(),
-             "Mismatch in source and translated sentences");
+    ABORT_IF(source_.numSentences() != histories.size(), "Mismatch in source and translated sentences");
     Response response;
 
     // Move source_ into response.
@@ -61,7 +58,7 @@ class ResponseBuilder {
     promise_.set_value(std::move(response));
   }
 
-private:
+ private:
   /// Builds qualityScores from histories and writes to response. expects
   /// buildTranslatedText to be run before to be able to obtain target text and
   /// subword information.
@@ -82,13 +79,13 @@ class ResponseBuilder {
   // Data members are context/curried args for the functor.
 
   ResponseOptions responseOptions_;
-  const Vocabs& vocabs_; // vocabs are required for decoding
-                                          // and any source validation checks.
-  std::promise<Response> promise_; //  To be set when callback triggered and
-                                   //  after Response constructed.
+  const Vocabs &vocabs_;            // vocabs are required for decoding
+                                    // and any source validation checks.
+  std::promise<Response> promise_;  //  To be set when callback triggered and
+                                    //  after Response constructed.
   AnnotatedText source_;
 };
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif //  SRC_BERGAMOT_RESPONSE_BUILDER_H_
+#endif  //  SRC_BERGAMOT_RESPONSE_BUILDER_H_
diff --git a/src/translator/response_options.h b/src/translator/response_options.h
index ed3cce3a5..b74f5782a 100644
--- a/src/translator/response_options.h
+++ b/src/translator/response_options.h
@@ -26,8 +26,8 @@ enum QualityScoreType {
 /// ResponseOptions dictate how to construct a Response for an input string of
 /// text to be translated.
 struct ResponseOptions {
-  bool qualityScores{false}; ///< Include quality-scores or not.
-  bool alignment{false};     ///< Include alignments or not.
+  bool qualityScores{false};  ///< Include quality-scores or not.
+  bool alignment{false};      ///< Include alignments or not.
 
   /// Whether to include sentenceMappings or not. Alignments require
   /// sentenceMappings and are available irrespective of this option if
@@ -44,7 +44,7 @@ struct ResponseOptions {
   ConcatStrategy concatStrategy{ConcatStrategy::FAITHFUL};
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif //  SRC_BERGAMOT_RESPONSE_OPTIONS_H_
+#endif  //  SRC_BERGAMOT_RESPONSE_OPTIONS_H_
diff --git a/src/translator/sentence_splitter.cpp b/src/translator/sentence_splitter.cpp
index 037012575..d0ada81e7 100644
--- a/src/translator/sentence_splitter.cpp
+++ b/src/translator/sentence_splitter.cpp
@@ -1,53 +1,47 @@
 #include "sentence_splitter.h"
+
+#include <string>
+
 #include "common/cli_helper.h"
 #include "common/logging.h"
 #include "common/options.h"
-#include <string>
 
 namespace marian {
 namespace bergamot {
 
-SentenceSplitter::SentenceSplitter(marian::Ptr<marian::Options> options)
-    : options_(options) {
-
+SentenceSplitter::SentenceSplitter(marian::Ptr<marian::Options> options) : options_(options) {
   std::string smode_str = options_->get<std::string>("ssplit-mode", "");
   mode_ = string2splitmode(smode_str);
-  std::string ssplit_prefix_file =
-      options_->get<std::string>("ssplit-prefix-file", "");
+  std::string ssplit_prefix_file = options_->get<std::string>("ssplit-prefix-file", "");
 
   if (ssplit_prefix_file.size()) {
     ssplit_prefix_file = marian::cli::interpolateEnvVars(ssplit_prefix_file);
 
-    LOG(info, "Loading protected prefixes for sentence splitting from {}",
-        ssplit_prefix_file);
+    LOG(info, "Loading protected prefixes for sentence splitting from {}", ssplit_prefix_file);
 
     ssplit_.load(ssplit_prefix_file);
   } else {
-    LOG(warn, "Missing list of protected prefixes for sentence splitting. "
-              "Set with --ssplit-prefix-file.");
+    LOG(warn,
+        "Missing list of protected prefixes for sentence splitting. "
+        "Set with --ssplit-prefix-file.");
   }
 }
 
-ug::ssplit::SentenceStream
-SentenceSplitter::createSentenceStream(const string_view &input) {
+ug::ssplit::SentenceStream SentenceSplitter::createSentenceStream(const string_view &input) {
   std::string_view input_converted(input.data(), input.size());
-  return std::move(
-      ug::ssplit::SentenceStream(input_converted, this->ssplit_, mode_));
+  return std::move(ug::ssplit::SentenceStream(input_converted, this->ssplit_, mode_));
 }
 
-ug::ssplit::SentenceStream::splitmode
-SentenceSplitter::string2splitmode(const std::string &m) {
+ug::ssplit::SentenceStream::splitmode SentenceSplitter::string2splitmode(const std::string &m) {
   typedef ug::ssplit::SentenceStream::splitmode splitmode;
   // @TODO: throw Exception on error
-  if (m == "sentence" || m == "Sentence")
-    return splitmode::one_sentence_per_line;
-  if (m == "paragraph" || m == "Paragraph")
-    return splitmode::one_paragraph_per_line;
+  if (m == "sentence" || m == "Sentence") return splitmode::one_sentence_per_line;
+  if (m == "paragraph" || m == "Paragraph") return splitmode::one_paragraph_per_line;
   if (m != "wrapped_text" && m != "WrappedText" && m != "wrappedText") {
     LOG(warn, "Ignoring unknown text input format specification: {}.", m);
   }
   return splitmode::wrapped_text;
 }
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/sentence_splitter.h b/src/translator/sentence_splitter.h
index 1c4742e05..9c5816574 100644
--- a/src/translator/sentence_splitter.h
+++ b/src/translator/sentence_splitter.h
@@ -1,11 +1,12 @@
 #ifndef SRC_BERGAMOT_SENTENCE_SPLITTER_H_
 #define SRC_BERGAMOT_SENTENCE_SPLITTER_H_
 
+#include <string>
+
 #include "common/options.h"
 #include "data/types.h"
-#include "ssplit.h"
 #include "definitions.h"
-#include <string>
+#include "ssplit.h"
 
 namespace marian {
 namespace bergamot {
@@ -15,18 +16,18 @@ class SentenceSplitter {
   // mts. Constructed based on options. Used in TextProcessor below to create
   // sentence-streams, which provide access to one sentence from blob of text at
   // a time.
-public:
+ public:
   explicit SentenceSplitter(Ptr<Options> options);
   ug::ssplit::SentenceStream createSentenceStream(string_view const &input);
 
-private:
+ private:
   ug::ssplit::SentenceSplitter ssplit_;
   Ptr<Options> options_;
   ug::ssplit::SentenceStream::splitmode mode_;
   ug::ssplit::SentenceStream::splitmode string2splitmode(const std::string &m);
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif //  SRC_BERGAMOT_SENTENCE_SPLITTER_H_
+#endif  //  SRC_BERGAMOT_SENTENCE_SPLITTER_H_
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 6a459abab..9a0370602 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -1,26 +1,30 @@
 #include "service.h"
-#include "batch.h"
-#include "definitions.h"
 
 #include <string>
 #include <utility>
 
+#include "batch.h"
+#include "definitions.h"
+
 namespace marian {
 namespace bergamot {
 
 Service::Service(Ptr<Options> options, MemoryBundle memoryBundle)
-    : requestId_(0), options_(options),
+    : requestId_(0),
+      options_(options),
       vocabs_(options, std::move(memoryBundle.vocabs)),
-      text_processor_(vocabs_, options), batcher_(options),
+      text_processor_(vocabs_, options),
+      batcher_(options),
       numWorkers_(std::max<int>(1, options->get<int>("cpu-threads"))),
       modelMemory_(std::move(memoryBundle.model)),
-      shortlistMemory_(std::move(memoryBundle.shortlist)) 
+      shortlistMemory_(std::move(memoryBundle.shortlist))
 #ifdef WASM_COMPATIBLE_SOURCE
-      , blocking_translator_(DeviceId(0, DeviceType::cpu), vocabs_, options_, &modelMemory_, &shortlistMemory_)
+      ,
+      blocking_translator_(DeviceId(0, DeviceType::cpu), vocabs_, options_, &modelMemory_, &shortlistMemory_)
 #endif
-  {
+{
 #ifdef WASM_COMPATIBLE_SOURCE
-    blocking_translator_.initialize();
+  blocking_translator_.initialize();
 #else
   workers_.reserve(numWorkers_);
   for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
@@ -48,17 +52,12 @@ void Service::blockIfWASM() {
 #endif
 }
 
-
-std::vector<Response>
-Service::translateMultiple(std::vector<std::string> &&inputs,
-                           ResponseOptions responseOptions) {
-
+std::vector<Response> Service::translateMultiple(std::vector<std::string> &&inputs, ResponseOptions responseOptions) {
   // We queue the individual Requests so they get compiled at batches to be
   // efficiently translated.
   std::vector<std::future<Response>> responseFutures;
   for (auto &input : inputs) {
-    std::future<Response> inputResponse =
-        queueRequest(std::move(input), responseOptions);
+    std::future<Response> inputResponse = queueRequest(std::move(input), responseOptions);
     responseFutures.push_back(std::move(inputResponse));
   }
 
@@ -77,8 +76,7 @@ Service::translateMultiple(std::vector<std::string> &&inputs,
   return responses;
 }
 
-std::future<Response> Service::queueRequest(std::string &&input,
-                                            ResponseOptions responseOptions) {
+std::future<Response> Service::queueRequest(std::string &&input, ResponseOptions responseOptions) {
   Segments segments;
   AnnotatedText source(std::move(input));
   text_processor_.process(source, segments);
@@ -86,19 +84,15 @@ std::future<Response> Service::queueRequest(std::string &&input,
   std::promise<Response> responsePromise;
   auto future = responsePromise.get_future();
 
-  ResponseBuilder responseBuilder(responseOptions, std::move(source), vocabs_,
-                                  std::move(responsePromise));
-  Ptr<Request> request = New<Request>(requestId_++, std::move(segments),
-                                      std::move(responseBuilder));
+  ResponseBuilder responseBuilder(responseOptions, std::move(source), vocabs_, std::move(responsePromise));
+  Ptr<Request> request = New<Request>(requestId_++, std::move(segments), std::move(responseBuilder));
 
   batcher_.addWholeRequest(request);
   return future;
 }
 
-std::future<Response> Service::translate(std::string &&input,
-                                         ResponseOptions responseOptions) {
-  std::future<Response> future =
-      queueRequest(std::move(input), responseOptions);
+std::future<Response> Service::translate(std::string &&input, ResponseOptions responseOptions) {
+  std::future<Response> future = queueRequest(std::move(input), responseOptions);
   blockIfWASM();
   return future;
 }
@@ -113,5 +107,5 @@ Service::~Service() {
 #endif
 }
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/service.h b/src/translator/service.h
index 8e2bc3f88..26ea83104 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -60,15 +60,14 @@ namespace bergamot {
 /// file supplied through config).
 ///
 class Service {
-
-public:
+ public:
   /// Construct Service from Marian options. If memoryBundle is empty, Service is
   /// initialized from file-based loading. Otherwise, Service is initialized from
   /// the given bytearray memories.
   /// @param options Marian options object
   /// @param memoryBundle holds all byte-array memories. Can be a set/subset of
   /// model, shortlist, vocabs and ssplitPrefixFile bytes. Optional.
-  explicit Service(Ptr<Options> options, MemoryBundle memoryBundle={});
+  explicit Service(Ptr<Options> options, MemoryBundle memoryBundle = {});
 
   /// Construct Service from a string configuration. If memoryBundle is empty, Service is
   /// initialized from file-based loading. Otherwise, Service is initialized from
@@ -76,7 +75,7 @@ class Service {
   /// @param [in] config string parsable as YAML expected to adhere with marian config
   /// @param [in] memoryBundle holds all byte-array memories. Can be a set/subset of
   /// model, shortlist, vocabs and ssplitPrefixFile bytes. Optional.
-  explicit Service(const std::string &config, MemoryBundle memoryBundle={})
+  explicit Service(const std::string &config, MemoryBundle memoryBundle = {})
       : Service(parseOptions(config, /*validate=*/false), std::move(memoryBundle)) {}
 
   /// Explicit destructor to clean up after any threads initialized in
@@ -91,8 +90,7 @@ class Service {
   /// @param [in] responseOptions: Options indicating whether or not to include
   /// some member in the Response, also specify any additional configurable
   /// parameters.
-  std::future<Response> translate(std::string &&source,
-                                  ResponseOptions options = ResponseOptions());
+  std::future<Response> translate(std::string &&source, ResponseOptions options = ResponseOptions());
 
   /// Translate multiple text-blobs in a single *blocking* API call, providing
   /// ResponseOptions which applies across all text-blobs dictating how to
@@ -110,19 +108,14 @@ class Service {
   /// @param [in] translationRequest: ResponseOptions indicating whether or not
   /// to include some member in the Response, also specify any additional
   /// configurable parameters.
-  std::vector<Response>
-  translateMultiple(std::vector<std::string> &&source,
-                    ResponseOptions responseOptions);
+  std::vector<Response> translateMultiple(std::vector<std::string> &&source, ResponseOptions responseOptions);
 
   /// Returns if model is alignment capable or not.
-  bool isAlignmentSupported() const {
-    return options_->hasAndNotEmpty("alignment");
-  }
+  bool isAlignmentSupported() const { return options_->hasAndNotEmpty("alignment"); }
 
-private:
+ private:
   /// Queue an input for translation.
-  std::future<Response> queueRequest(std::string &&input,
-                                     ResponseOptions responseOptions);
+  std::future<Response> queueRequest(std::string &&input, ResponseOptions responseOptions);
 
   /// Dispatch call to translate after inserting in queue
   void dispatchTranslate();
@@ -137,20 +130,20 @@ class Service {
   Ptr<Options> options_;
 
   /// Model memory to load model passed as bytes.
-  AlignedMemory modelMemory_; // ORDER DEPENDENCY (translators_)
+  AlignedMemory modelMemory_;  // ORDER DEPENDENCY (translators_)
   /// Shortlist memory passed as bytes.
-  AlignedMemory shortlistMemory_; // ORDER DEPENDENCY (translators_)
+  AlignedMemory shortlistMemory_;  // ORDER DEPENDENCY (translators_)
 
   /// Stores requestId of active request. Used to establish
   /// ordering among requests and logging/book-keeping.
 
   size_t requestId_;
   /// Store vocabs representing source and target.
-  Vocabs vocabs_; // ORDER DEPENDENCY (text_processor_)
+  Vocabs vocabs_;  // ORDER DEPENDENCY (text_processor_)
 
   /// TextProcesser takes a blob of text and converts into format consumable by
   /// the batch-translator and annotates sentences and words.
-  TextProcessor text_processor_; // ORDER DEPENDENCY (vocabs_)
+  TextProcessor text_processor_;  // ORDER DEPENDENCY (vocabs_)
 
   /// Batcher handles generation of batches from a request, subject to
   /// packing-efficiency and priority optimization heuristics.
@@ -159,13 +152,13 @@ class Service {
   // The following constructs are available providing full capabilities on a non
   // WASM platform, where one does not have to hide threads.
 #ifdef WASM_COMPATIBLE_SOURCE
-  BatchTranslator blocking_translator_; // ORDER DEPENDENCY (modelMemory_, shortlistMemory_)
+  BatchTranslator blocking_translator_;  // ORDER DEPENDENCY (modelMemory_, shortlistMemory_)
 #else
   std::vector<std::thread> workers_;
-#endif // WASM_COMPATIBLE_SOURCE
+#endif  // WASM_COMPATIBLE_SOURCE
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif // SRC_BERGAMOT_SERVICE_H_
+#endif  // SRC_BERGAMOT_SERVICE_H_
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index bca5fd157..234c48445 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -1,39 +1,33 @@
 #include "text_processor.h"
-#include "data/types.h"
-#include "definitions.h"
-#include "annotation.h"
 
-#include "common/options.h"
 #include <vector>
 
+#include "annotation.h"
+#include "common/options.h"
+#include "data/types.h"
+#include "definitions.h"
+
 namespace marian {
 namespace bergamot {
 
-Segment TextProcessor::tokenize(const string_view &segment,
-                                std::vector<string_view> &wordRanges) {
+Segment TextProcessor::tokenize(const string_view &segment, std::vector<string_view> &wordRanges) {
   // vocabs_->sources().front() is invoked as we currently only support one source vocab
-  return vocabs_.sources().front()->encodeWithByteRanges(
-      segment, wordRanges, /*addEOS=*/false, /*inference=*/true);
+  return vocabs_.sources().front()->encodeWithByteRanges(segment, wordRanges, /*addEOS=*/false, /*inference=*/true);
 }
 
-TextProcessor::TextProcessor(Vocabs &vocabs,
-                             Ptr<Options> options)
-    : vocabs_(vocabs), sentence_splitter_(options) {
-
+TextProcessor::TextProcessor(Vocabs &vocabs, Ptr<Options> options) : vocabs_(vocabs), sentence_splitter_(options) {
   max_length_break_ = options->get<int>("max-length-break");
   max_length_break_ = max_length_break_ - 1;
   ABORT_IF(max_length_break_ < 0, "max-length-break cannot be < 0");
 }
 
 void TextProcessor::process(AnnotatedText &source, Segments &segments) {
-
   string_view query = string_view(source.text);
   auto sentenceStream = sentence_splitter_.createSentenceStream(query);
   std::string_view sentenceStringPiece;
 
   while (sentenceStream >> sentenceStringPiece) {
-    marian::string_view sentence(sentenceStringPiece.data(),
-                                 sentenceStringPiece.size());
+    marian::string_view sentence(sentenceStringPiece.data(), sentenceStringPiece.size());
 
     std::vector<string_view> wordRanges;
     Segment segment = tokenize(sentence, wordRanges);
@@ -48,11 +42,9 @@ void TextProcessor::process(AnnotatedText &source, Segments &segments) {
   }
 }
 
-void TextProcessor::wrap(Segment &segment,
-                         std::vector<string_view> &wordRanges,
-                         Segments &segments, AnnotatedText &source) {
-  for (size_t offset = 0; offset < segment.size();
-       offset += max_length_break_) {
+void TextProcessor::wrap(Segment &segment, std::vector<string_view> &wordRanges, Segments &segments,
+                         AnnotatedText &source) {
+  for (size_t offset = 0; offset < segment.size(); offset += max_length_break_) {
     auto start = segment.begin() + offset;
 
     size_t left = segment.size() - offset;
@@ -67,5 +59,5 @@ void TextProcessor::wrap(Segment &segment,
   }
 }
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index 732887757..be37c3de6 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -1,16 +1,15 @@
 #ifndef SRC_BERGAMOT_TEXT_PROCESSOR_H_
 #define SRC_BERGAMOT_TEXT_PROCESSOR_H_
 
+#include <vector>
+
+#include "annotation.h"
 #include "data/types.h"
 #include "data/vocab.h"
 #include "definitions.h"
-#include "annotation.h"
-
 #include "sentence_splitter.h"
 #include "vocabs.h"
 
-#include <vector>
-
 namespace marian {
 namespace bergamot {
 
@@ -21,31 +20,29 @@ class TextProcessor {
   // Used in Service to convert an incoming blog of text to a vector of
   // sentences (vector of words). In addition, the ByteRanges of the
   // source-tokens in unnormalized text are provided as string_views.
-public:
+ public:
   explicit TextProcessor(Vocabs &vocabs, Ptr<Options>);
 
   void process(AnnotatedText &source, Segments &segments);
 
-private:
+ private:
   // Tokenizes an input string, returns Words corresponding. Loads the
   // corresponding byte-ranges into tokenRanges.
-  Segment tokenize(const string_view &input,
-                   std::vector<string_view> &tokenRanges);
+  Segment tokenize(const string_view &input, std::vector<string_view> &tokenRanges);
 
   // Wrap into sentences of at most max_length_break_ tokens and add to source.
-  void wrap(Segment &sentence, std::vector<string_view> &tokenRanges,
-            Segments &segments, AnnotatedText &source);
+  void wrap(Segment &sentence, std::vector<string_view> &tokenRanges, Segments &segments, AnnotatedText &source);
 
   // shorthand, used only in truncate()
   // vocabs_->sources().front() is invoked as we currently only support one source vocab
   const Word sourceEosId() const { return vocabs_.sources().front()->getEosId(); }
 
-  const Vocabs& vocabs_;
+  const Vocabs &vocabs_;
   SentenceSplitter sentence_splitter_;
   size_t max_length_break_;
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif // SRC_BERGAMOT_TEXT_PROCESSOR_H_
+#endif  // SRC_BERGAMOT_TEXT_PROCESSOR_H_
diff --git a/src/translator/threadsafe_batcher.cpp b/src/translator/threadsafe_batcher.cpp
index e6b6a4668..7e03d67ce 100644
--- a/src/translator/threadsafe_batcher.cpp
+++ b/src/translator/threadsafe_batcher.cpp
@@ -6,12 +6,9 @@
 namespace marian {
 namespace bergamot {
 
-ThreadsafeBatcher::ThreadsafeBatcher(Ptr<Options> options)
-  : backend_(options), enqueued_(0), shutdown_(false) {}
+ThreadsafeBatcher::ThreadsafeBatcher(Ptr<Options> options) : backend_(options), enqueued_(0), shutdown_(false) {}
 
-ThreadsafeBatcher::~ThreadsafeBatcher() {
-  shutdown();
-}
+ThreadsafeBatcher::~ThreadsafeBatcher() { shutdown(); }
 
 void ThreadsafeBatcher::addSentenceWithPriority(RequestSentence &sentence) {
   std::unique_lock<std::mutex> lock(mutex_);
@@ -37,13 +34,13 @@ void ThreadsafeBatcher::shutdown() {
 
 bool ThreadsafeBatcher::operator>>(Batch &batch) {
   std::unique_lock<std::mutex> lock(mutex_);
-  work_.wait(lock, [this](){ return enqueued_ || shutdown_; });
+  work_.wait(lock, [this]() { return enqueued_ || shutdown_; });
   bool ret = backend_ >> batch;
   assert(ret || shutdown_);
   enqueued_ -= batch.size();
   return ret;
 }
 
-} // namespace bergamot
-} // namespace marian
-#endif // WASM_COMPATIBLE_SOURCE
+}  // namespace bergamot
+}  // namespace marian
+#endif  // WASM_COMPATIBLE_SOURCE
diff --git a/src/translator/threadsafe_batcher.h b/src/translator/threadsafe_batcher.h
index 994b13fa1..195c3cd8e 100644
--- a/src/translator/threadsafe_batcher.h
+++ b/src/translator/threadsafe_batcher.h
@@ -20,39 +20,39 @@ typedef Batcher ThreadsafeBatcher;
 #else
 
 class ThreadsafeBatcher {
-  public:
-    explicit ThreadsafeBatcher(Ptr<Options> options);
+ public:
+  explicit ThreadsafeBatcher(Ptr<Options> options);
 
-    ~ThreadsafeBatcher();
+  ~ThreadsafeBatcher();
 
-    // Add sentences to be translated by calling these (see Batcher).  When
-    // done, call shutdown.
-    void addSentenceWithPriority(RequestSentence &sentence);
-    void addWholeRequest(Ptr<Request> request);
-    void shutdown();
+  // Add sentences to be translated by calling these (see Batcher).  When
+  // done, call shutdown.
+  void addSentenceWithPriority(RequestSentence &sentence);
+  void addWholeRequest(Ptr<Request> request);
+  void shutdown();
 
-    // Get a batch out of the batcher.  Return false to shutdown worker.
-    bool operator>>(Batch &batch);
+  // Get a batch out of the batcher.  Return false to shutdown worker.
+  bool operator>>(Batch &batch);
 
-  private:
-    Batcher backend_;
+ private:
+  Batcher backend_;
 
-    // Number of sentences in backend_;
-    size_t enqueued_;
+  // Number of sentences in backend_;
+  size_t enqueued_;
 
-    // Are we shutting down?
-    bool shutdown_;
+  // Are we shutting down?
+  bool shutdown_;
 
-    // Lock on this object.
-    std::mutex mutex_;
+  // Lock on this object.
+  std::mutex mutex_;
 
-    // Signaled when there are sentences to translate.
-    std::condition_variable work_;
+  // Signaled when there are sentences to translate.
+  std::condition_variable work_;
 };
 
 #endif
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
 
-#endif // SRC_BERGAMOT_THREADSAFE_BATCHER_H_
+#endif  // SRC_BERGAMOT_THREADSAFE_BATCHER_H_
diff --git a/src/translator/vocabs.h b/src/translator/vocabs.h
index 89aed4b05..bc5ef1406 100644
--- a/src/translator/vocabs.h
+++ b/src/translator/vocabs.h
@@ -6,14 +6,13 @@ namespace bergamot {
 /// Wrapper of Marian Vocab objects needed for translator.
 /// Holds multiple source vocabularies and one target vocabulary
 class Vocabs {
-public:
+ public:
   /// Construct vocabs object from either byte-arrays or files
-  Vocabs(Ptr<Options> options, std::vector<std::shared_ptr<AlignedMemory>>&& vocabMemories): options_(options){
-    if (!vocabMemories.empty()){
+  Vocabs(Ptr<Options> options, std::vector<std::shared_ptr<AlignedMemory>>&& vocabMemories) : options_(options) {
+    if (!vocabMemories.empty()) {
       // load vocabs from buffer
       load(std::move(vocabMemories));
-    }
-    else{
+    } else {
       // load vocabs from file
       auto vocabPaths = options->get<std::vector<std::string>>("vocabs");
       load(vocabPaths);
@@ -21,16 +20,12 @@ class Vocabs {
   }
 
   /// Get all source vocabularies (as a vector)
-  const std::vector<Ptr<Vocab const>>& sources() const {
-    return srcVocabs_;
-  }
+  const std::vector<Ptr<Vocab const>>& sources() const { return srcVocabs_; }
 
   /// Get the target vocabulary
-  const Ptr<Vocab const>& target() const {
-    return trgVocab_;
-  }
+  const Ptr<Vocab const>& target() const { return trgVocab_; }
 
-private:
+ private:
   std::vector<Ptr<Vocab const>> srcVocabs_;  // source vocabularies
   Ptr<Vocab const> trgVocab_;                // target vocabulary
   Ptr<Options> options_;
@@ -46,7 +41,7 @@ class Vocabs {
     std::unordered_map<uintptr_t, Ptr<Vocab>> vmap;
     for (size_t i = 0; i < srcVocabs_.size(); i++) {
       auto m = vmap.emplace(std::make_pair(reinterpret_cast<uintptr_t>(vocabMemories[i].get()), Ptr<Vocab>()));
-      if (m.second) { // new: load the vocab
+      if (m.second) {  // new: load the vocab
         m.first->second = New<Vocab>(options_, i);
         m.first->second->loadFromSerialized(absl::string_view(vocabMemories[i]->begin(), vocabMemories[i]->size()));
       }
@@ -58,14 +53,14 @@ class Vocabs {
   }
 
   // load from file
-  void load(const std::vector<std::string>& vocabPaths){
+  void load(const std::vector<std::string>& vocabPaths) {
     // with the current setup, we need at least two vocabs: src and trg
     ABORT_IF(vocabPaths.size() < 2, "Insufficient number of vocabularies.");
     srcVocabs_.resize(vocabPaths.size());
     std::unordered_map<std::string, Ptr<Vocab>> vmap;
     for (size_t i = 0; i < srcVocabs_.size(); ++i) {
       auto m = vmap.emplace(std::make_pair(vocabPaths[i], Ptr<Vocab>()));
-      if (m.second) { // new: load the vocab
+      if (m.second) {  // new: load the vocab
         m.first->second = New<Vocab>(options_, i);
         m.first->second->load(vocabPaths[i]);
       }
@@ -77,5 +72,5 @@ class Vocabs {
   }
 };
 
-} // namespace bergamot
-} // namespace marian
+}  // namespace bergamot
+}  // namespace marian
diff --git a/wasm/bindings/TranslationModelBindings.cpp b/wasm/bindings/TranslationModelBindings.cpp
index 1db740149..64203a16d 100644
--- a/wasm/bindings/TranslationModelBindings.cpp
+++ b/wasm/bindings/TranslationModelBindings.cpp
@@ -21,12 +21,11 @@ val getByteArrayView(AlignedMemory& alignedMemory) {
 
 EMSCRIPTEN_BINDINGS(aligned_memory) {
   class_<AlignedMemory>("AlignedMemory")
-    .constructor<std::size_t, std::size_t>()
-    .function("size", &AlignedMemory::size)
-	  .function("getByteArrayView", &getByteArrayView)
-    ;
+      .constructor<std::size_t, std::size_t>()
+      .function("size", &AlignedMemory::size)
+      .function("getByteArrayView", &getByteArrayView);
 
-    register_vector<AlignedMemory*>("AlignedMemoryList");
+  register_vector<AlignedMemory*>("AlignedMemoryList");
 }
 
 // When source and target vocab files are same, only one memory object is passed from JS to
@@ -41,16 +40,14 @@ std::vector<std::shared_ptr<AlignedMemory>> prepareVocabsSmartMemories(std::vect
   if (vocabsMemories.size() == 2) {
     auto targetVocabMemory = std::make_shared<AlignedMemory>(std::move(*(vocabsMemories[1])));
     vocabsSmartMemories.push_back(std::move(targetVocabMemory));
-  }
-  else {
+  } else {
     vocabsSmartMemories.push_back(sourceVocabMemory);
   }
   return vocabsSmartMemories;
 }
 
-marian::bergamot::MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory,
-                                                   AlignedMemory* shortlistMemory,
-                                                   std::vector<AlignedMemory*> uniqueVocabsMemories){
+marian::bergamot::MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory, AlignedMemory* shortlistMemory,
+                                                   std::vector<AlignedMemory*> uniqueVocabsMemories) {
   marian::bergamot::MemoryBundle memoryBundle;
   memoryBundle.model = std::move(*modelMemory);
   memoryBundle.shortlist = std::move(*shortlistMemory);
@@ -59,19 +56,18 @@ marian::bergamot::MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory,
   return memoryBundle;
 }
 
-TranslationModel* TranslationModelFactory(const std::string &config,
-                                          AlignedMemory* modelMemory,
+TranslationModel* TranslationModelFactory(const std::string& config, AlignedMemory* modelMemory,
                                           AlignedMemory* shortlistMemory,
                                           std::vector<AlignedMemory*> uniqueVocabsMemories) {
-  return new TranslationModel(config, std::move(prepareMemoryBundle(modelMemory, shortlistMemory, uniqueVocabsMemories)));
+  return new TranslationModel(config,
+                              std::move(prepareMemoryBundle(modelMemory, shortlistMemory, uniqueVocabsMemories)));
 }
 
 EMSCRIPTEN_BINDINGS(translation_model) {
   class_<TranslationModel>("TranslationModel")
-    .constructor(&TranslationModelFactory, allow_raw_pointers())
-    .function("translate", &TranslationModel::translateMultiple)
-	  .function("isAlignmentSupported", &TranslationModel::isAlignmentSupported)
-    ;
+      .constructor(&TranslationModelFactory, allow_raw_pointers())
+      .function("translate", &TranslationModel::translateMultiple)
+      .function("isAlignmentSupported", &TranslationModel::isAlignmentSupported);
   // ^ We redirect Service::translateMultiple to WASMBound::translate instead. Sane API is
   // translate. If and when async comes, we can be done with this inconsistency.
 
diff --git a/wasm/bindings/TranslationRequestBindings.cpp b/wasm/bindings/TranslationRequestBindings.cpp
index 7d5cd1e6f..42ac6c698 100644
--- a/wasm/bindings/TranslationRequestBindings.cpp
+++ b/wasm/bindings/TranslationRequestBindings.cpp
@@ -12,8 +12,4 @@ typedef marian::bergamot::ResponseOptions TranslationRequest;
 using namespace emscripten;
 
 // Binding code
-EMSCRIPTEN_BINDINGS(translation_request) {
-  class_<TranslationRequest>("TranslationRequest")
-    .constructor<>()
-    ;
-}
+EMSCRIPTEN_BINDINGS(translation_request) { class_<TranslationRequest>("TranslationRequest").constructor<>(); }
diff --git a/wasm/bindings/TranslationResultBindings.cpp b/wasm/bindings/TranslationResultBindings.cpp
index c1c0ca8ae..f02bef902 100644
--- a/wasm/bindings/TranslationResultBindings.cpp
+++ b/wasm/bindings/TranslationResultBindings.cpp
@@ -4,6 +4,7 @@
  */
 
 #include <emscripten/bind.h>
+
 #include <vector>
 
 #include "response.h"

From 0f8f8e026a034633e5df6119b73a35aa960539ec Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Thu, 20 May 2021 08:59:30 +0300
Subject: [PATCH 252/442] Pin emsdk version to the same one used in Circle CI
 (#165)

---
 build-wasm.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/build-wasm.sh b/build-wasm.sh
index bdc00d501..6a48f5bad 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -30,8 +30,8 @@ if [ "$EMSDK" == "" ]; then
   fi
   if [ "$EMSDK_UPDATE_REQUIRED" == "1" ]; then
     cd emsdk
-    ./emsdk install latest
-    ./emsdk activate latest
+    ./emsdk install 2.0.9
+    ./emsdk activate 2.0.9
     cd -
   fi
   source ./emsdk/emsdk_env.sh

From 4b177d57e414d9eca6066e9c913c79b5f3643f49 Mon Sep 17 00:00:00 2001
From: Motin <motin@motin.eu>
Date: Thu, 20 May 2021 09:33:58 +0300
Subject: [PATCH 253/442] GitHub action to push browsermt/main branch to
 mozilla/bergamot-translator every hour (#160)

* Create push-browsermt-main-to-mozilla-main.yml

* Update .github/workflows/push-browsermt-main-to-mozilla-main.yml

Co-authored-by: Graeme <graemenail@gmail.com>

* Tweaks

* Fix yaml syntax

* Parametrized the workflow based on @jerinphilip's example

Co-authored-by: Graeme <graemenail@gmail.com>
---
 .../push-browsermt-main-to-mozilla-main.yml   | 27 +++++++++++++++++++
 1 file changed, 27 insertions(+)
 create mode 100644 .github/workflows/push-browsermt-main-to-mozilla-main.yml

diff --git a/.github/workflows/push-browsermt-main-to-mozilla-main.yml b/.github/workflows/push-browsermt-main-to-mozilla-main.yml
new file mode 100644
index 000000000..bc0caf619
--- /dev/null
+++ b/.github/workflows/push-browsermt-main-to-mozilla-main.yml
@@ -0,0 +1,27 @@
+name: Push browsermt/main branch to mozilla/bergamot-translator
+
+on:
+  schedule:
+  # Run every hour
+  - cron: "0 * * * *"
+
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    name: Mirror a branch from a remote repo into this one
+    env:
+      source_repository: 'browsermt/bergamot-translator'
+      this_repository: 'mozilla/bergamot-translator'
+      branch: 'main'
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Mirror a branch from a remote repo into this one
+        if: github.repository == '${{ env.this_repository }}'
+        run: |
+          git clone -b ${{ env.branch }} https://github.com/${{ env.source_repository }}.git source
+          cd source
+          git remote add mirror https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ env.this_repository }}.git
+          git push mirror ${{ env.branch }}

From 4f8050be646e560496b86924deaeb92d4c9b96ef Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Thu, 20 May 2021 11:03:10 +0100
Subject: [PATCH 254/442] Update tests

---
 bergamot-translator-tests | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 9209aa51e..c914595b9 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 9209aa51e71f57b90172ffd259cf3021c4890bcf
+Subproject commit c914595b98b804ea29979a50551b6218e7a33429

From f1253720a8aa9e022975fa1e58689e13ac33f36e Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 20 May 2021 12:49:44 +0100
Subject: [PATCH 255/442] Bumping BRT for hotfixes (#169)

* Bumping BRT for hotfixes

* updating brt to point to main
---
 bergamot-translator-tests | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index c914595b9..636af01c6 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit c914595b98b804ea29979a50551b6218e7a33429
+Subproject commit 636af01c63f2f080a9e59e99b15ac4bfdaec76e1

From 22a1b9113ec983aac12c6ddb48a78d7ef8ebfe57 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sat, 22 May 2021 00:04:49 +0100
Subject: [PATCH 256/442] Remove O(N^2) reallocation (#171)

---
 src/translator/annotation.cpp | 2 --
 1 file changed, 2 deletions(-)

diff --git a/src/translator/annotation.cpp b/src/translator/annotation.cpp
index 35f5c6916..94ceb54d7 100644
--- a/src/translator/annotation.cpp
+++ b/src/translator/annotation.cpp
@@ -13,8 +13,6 @@ AnnotatedText::AnnotatedText(std::string &&t) : text(std::move(t)) {
 void AnnotatedText::appendSentence(string_view prefix, std::vector<string_view>::iterator begin,
                                    std::vector<string_view>::iterator end) {
   assert(annotation.token_begin_.back() == text.size());
-  // We'll be adding tokens from the sentence and another gap.
-  annotation.token_begin_.reserve(annotation.token_begin_.size() + (end - begin) + 1);
 
   // prefix is just end of the previous one.
   appendEndingWhitespace(prefix);

From 576afae6b3a87d2980ad8bbe3dfb1da8a73eb99a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 25 May 2021 11:10:56 +0100
Subject: [PATCH 257/442] Adding documentation action (#168)

Adds a GitHub workflow that builds documentation from sources through doxygen through sphinx on push to the main branch or on push of any semantic version tags. The built documentation is deployed at https://github.com/browsermt/docs@gh-pages, which is rendered at https://browser.mt/docs/<suffix>, where <suffix> is 'main' or a tag vM.m.p corresponding to a semantic version.

On pull request artifacts are uploaded for reviewers to inspect if need be.
---
 .github/workflows/doc.yml | 89 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 89 insertions(+)
 create mode 100644 .github/workflows/doc.yml

diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml
new file mode 100644
index 000000000..77307d695
--- /dev/null
+++ b/.github/workflows/doc.yml
@@ -0,0 +1,89 @@
+name: Documentation
+
+on:
+  push:
+    branches: [ main, ci-sandbox ]
+    tags: ['v[0-9]+.[0-9]+.[0-9]+']
+  pull_request: 
+    branches: [ main ]
+
+jobs:
+  api-documentation:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: recursive
+
+      # Runs javascript to extract push events from both tags and branch (only main, due to workflow trigger)
+      # converts refs/<>/<name> -> <name>
+      # eg:
+      #     refs/head/main   -> main
+      #     refs/tags/v0.1.0 -> v0.1.0
+      #
+      - name: Extract tag name
+        id: tag
+        uses: actions/github-script@0.2.0
+        if: ${{ github.event_name == 'push' }}
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          script: |
+            const args = context.payload.ref.split("/");
+            [refs, category, ...rest] = args;
+            return rest.join("/");
+
+      # Patches the BERGAMOT_VERSION file used by sphinx-docs at run time to
+      # obtain names like 'main' or 'ci-sandbox' to not confuse with version
+      # based documentation built separately.
+      - name: Deploy-time patch version 
+        run: | 
+            echo ${{steps.tag.outputs.result }} > BERGAMOT_VERSION
+
+      - name: Set up Doxygen
+        run: sudo apt-get install -y doxygen
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.7
+
+      - name: Set up dependency cache
+        uses: actions/cache@v2
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ hashFiles('doc/requirements.txt') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+
+      - name: Install dependencies
+        working-directory: ./doc
+        run: python3 -m pip install -r requirements.txt
+
+      - name: Build documentation
+        working-directory: ./doc
+        run: sphinx-build -b html ./ build/
+
+
+      - name: Deploy 🚀
+        uses: JamesIves/github-pages-deploy-action@4.1.3
+        if: ${{ github.event_name == 'push' }}
+        with:
+          repository-name: 'browsermt/docs' 
+          branch: gh-pages # The branch the action should deploy to.
+          folder: './doc/build/' # The folder the action should deploy.
+          target-folder: '${{ steps.tag.outputs.result }}' 
+          ssh-key: ${{ secrets.BERGAMOT_SSH_PRIVATE_KEY }}
+
+      # This artifact contains the HTML output of Sphinx only.
+      # With index.html at the root of the produced zip file.
+      # For use for maintainers to download the zip and check render of
+      # documentation while generated at pull-request. 
+      - name: Upload documentation
+        uses: actions/upload-artifact@v2
+        if: ${{ github.event_name == 'pull_request'}}
+        with:
+          name: api-docs
+          path: ./doc/build/
+          if-no-files-found: error
+

From 8bec1b7b6ba30c522c102621e55b8cbe4bd051da Mon Sep 17 00:00:00 2001
From: Qianqian Zhu <qianqian.zhu@hotmail.com>
Date: Tue, 25 May 2021 12:05:16 +0100
Subject: [PATCH 258/442] Fix failures when loading text shortlist (#154)

---
 app/service-cli.cpp                 |  2 +-
 bergamot-translator-tests           |  2 +-
 src/translator/batch_translator.cpp |  6 ++----
 src/translator/byte_array_util.cpp  | 10 +++++++---
 src/translator/parser.h             |  5 ++++-
 5 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/app/service-cli.cpp b/app/service-cli.cpp
index fbf013161..a29e71bb2 100644
--- a/app/service-cli.cpp
+++ b/app/service-cli.cpp
@@ -19,7 +19,7 @@ int main(int argc, char *argv[]) {
   // Prepare memories for bytearrays (including model, shortlist and vocabs)
   marian::bergamot::MemoryBundle memoryBundle;
 
-  if (options->get<bool>("check-bytearray")) {
+  if (options->get<bool>("bytearray")) {
     // Load legit values into bytearrays.
     memoryBundle = marian::bergamot::getMemoryBundleFromConfig(options);
   }
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 636af01c6..1b20a62f6 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 636af01c63f2f080a9e59e99b15ac4bfdaec76e1
+Subproject commit 1b20a62f6614db371f59b97ff83262b8ebd235de
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
index c27edc104..889ff0073 100644
--- a/src/translator/batch_translator.cpp
+++ b/src/translator/batch_translator.cpp
@@ -20,8 +20,6 @@ BatchTranslator::BatchTranslator(DeviceId const device, Vocabs &vocabs, Ptr<Opti
 
 void BatchTranslator::initialize() {
   // Initializes the graph.
-  bool check =
-      options_->get<bool>("check-bytearray", false);  // Flag holds whether validate the bytearray (model and shortlist)
   if (options_->hasAndNotEmpty("shortlist")) {
     int srcIdx = 0, trgIdx = 1;
     bool shared_vcb =
@@ -30,7 +28,7 @@ void BatchTranslator::initialize() {
     if (shortlistMemory_->size() > 0 && shortlistMemory_->begin() != nullptr) {
       slgen_ = New<data::BinaryShortlistGenerator>(shortlistMemory_->begin(), shortlistMemory_->size(),
                                                    vocabs_.sources().front(), vocabs_.target(), srcIdx, trgIdx,
-                                                   shared_vcb, check);
+                                                   shared_vcb, options_->get<bool>("check-bytearray"));
     } else {
       // Changed to BinaryShortlistGenerator to enable loading binary shortlist file
       // This class also supports text shortlist file
@@ -51,7 +49,7 @@ void BatchTranslator::initialize() {
                       // from there, as opposed to from reading in the config file
     ABORT_IF((uintptr_t)modelMemory_->begin() % 256 != 0,
              "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
-    if (check) {
+    if (options_->get<bool>("check-bytearray")) {
       ABORT_IF(!validateBinaryModel(*modelMemory_, modelMemory_->size()),
                "The binary file is invalid. Incomplete or corrupted download?");
     }
diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index 247d1645b..8c3d60837 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -1,10 +1,10 @@
 #include "byte_array_util.h"
 
-#include <stdlib.h>
-
-#include <iostream>
+#include <cstdlib>
 #include <memory>
 
+#include "data/shortlist.h"
+
 namespace marian {
 namespace bergamot {
 
@@ -102,6 +102,8 @@ AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options) {
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options) {
   auto shortlist = options->get<std::vector<std::string>>("shortlist");
   ABORT_IF(shortlist.empty(), "No path to shortlist file is given.");
+  ABORT_IF(!marian::data::isBinaryShortlist(shortlist[0]),
+           "Loading non-binary shortlist file into memory is not supported");
   return loadFileToMemory(shortlist[0], 64);
 }
 
@@ -112,6 +114,8 @@ void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
   vocabMemories.resize(vfiles.size());
   std::unordered_map<std::string, std::shared_ptr<AlignedMemory>> vocabMap;
   for (size_t i = 0; i < vfiles.size(); ++i) {
+    ABORT_IF(marian::filesystem::Path(vfiles[i]).extension() != marian::filesystem::Path(".spm"),
+             "Loading non-SentencePiece vocab files into memory is not supported");
     auto m = vocabMap.emplace(std::make_pair(vfiles[i], std::shared_ptr<AlignedMemory>()));
     if (m.second) {
       m.first->second = std::make_shared<AlignedMemory>(loadFileToMemory(vfiles[i], 64));
diff --git a/src/translator/parser.h b/src/translator/parser.h
index b2b0a80d4..790717c24 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -20,8 +20,11 @@ inline marian::ConfigParser createConfigParser() {
   cp.addOption<int>("--max-length-break", "Bergamot Options",
                     "Maximum input tokens to be processed in a single sentence.", 128);
 
+  cp.addOption<bool>("--bytearray", "Bergamot Options",
+                     "Flag holds whether to construct service from bytearrays, only for testing purpose", false);
+
   cp.addOption<bool>("--check-bytearray", "Bergamot Options",
-                     "Flag holds whether to check the content of the bytearray (true by default)", true);
+                     "Flag holds whether to check the content of the bytearrays (true by default)", true);
 
   return cp;
 }

From eb579ed26f4ffcd6e36afb9956d423b5424f8ccc Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 27 May 2021 10:51:53 +0100
Subject: [PATCH 259/442] Updating marian dev RelwithDebInfo -> Release (#178)

* Updating marian dev RelwithDebInfo -> Release

* Updating submodule to point to master
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 80e474247..2a6fdd4cc 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 80e474247909a67231249d3598ac5ad248c9aff5
+Subproject commit 2a6fdd4cc41b0976d33165cec04062d192de9c40

From 5d3ec9c0a94a1390d6a115a5015db023a90badb8 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 31 May 2021 14:44:59 +0100
Subject: [PATCH 260/442] Single executable (#175)

* Collapsing executables

* Adding new test executable

* Deleting old executable sources

* Updating brt to operate with modes

* cli-framework -> cli

* Updating workflows to check for bergamot instead of bergamot-translator-app

* Adding documentation

* Making fn pure virtual

* Shuffling apps into app namespace, alongside class documentation

* Include app folder in documentation

* BRT update service-cli -> native

* parser.h: service-cli -> native

* Updates to marian-integration.md

* Cleanup: Remove templates, interface proper

* change 4 to 2 cores for build instructions

* service-cli -> native

* Commenting the string constructor explanation

* Not doing halfway interface / inheritance

* Nick hates state, let's try this one

* Revert "Nick hates state, let's try this one"

This reverts commit e56db9f474b1906e62af0b06afb7c7d9e08ea9c8.

* class -> struct before trying std::function stuff

* oop -> functional?

* Hints on what is happening

* app::ftable -> app::REGISTRY

* We have if-else and functions now.

And we won't have test apps.

* Doc linking to usage examples in brt

* Remove unordered_map

* Documentation updates

* Fix warning
---
 .github/workflows/native-mac.yml    |   2 +-
 .github/workflows/native-ubuntu.yml |   2 +-
 app/CMakeLists.txt                  |  12 +-
 app/bergamot-translator-app.cpp     |  41 ------
 app/bergamot.cpp                    |  18 +++
 app/cli.h                           | 186 ++++++++++++++++++++++++++++
 app/marian-decoder-new.cpp          |  46 -------
 app/service-cli.cpp                 |  97 ---------------
 bergamot-translator-tests           |   2 +-
 doc/conf.py                         |   2 +-
 doc/marian-integration.md           |  72 +++++++----
 src/translator/parser.h             |   3 +
 12 files changed, 259 insertions(+), 224 deletions(-)
 delete mode 100644 app/bergamot-translator-app.cpp
 create mode 100644 app/bergamot.cpp
 create mode 100644 app/cli.h
 delete mode 100644 app/marian-decoder-new.cpp
 delete mode 100644 app/service-cli.cpp

diff --git a/.github/workflows/native-mac.yml b/.github/workflows/native-mac.yml
index b1fe18fe4..bbd3651c1 100644
--- a/.github/workflows/native-mac.yml
+++ b/.github/workflows/native-mac.yml
@@ -91,7 +91,7 @@ jobs:
     - name: Print versions
       working-directory: build
       run: |
-        ./app/bergamot-translator-app --version
+        ./app/bergamot --version
 
     - name: Install regression-test framework (BRT)
       working-directory: bergamot-translator-tests
diff --git a/.github/workflows/native-ubuntu.yml b/.github/workflows/native-ubuntu.yml
index 563daf7d0..0617201e5 100644
--- a/.github/workflows/native-ubuntu.yml
+++ b/.github/workflows/native-ubuntu.yml
@@ -180,7 +180,7 @@ jobs:
     - name: Print versions
       working-directory: build
       run: |
-        ./app/bergamot-translator-app --version
+        ./app/bergamot --version
 
 
     - name: Install regression-test framework (BRT)
diff --git a/app/CMakeLists.txt b/app/CMakeLists.txt
index a55a571b7..b5c6a433b 100644
--- a/app/CMakeLists.txt
+++ b/app/CMakeLists.txt
@@ -1,10 +1,2 @@
-add_executable(bergamot-translator-app bergamot-translator-app.cpp)
-target_link_libraries(bergamot-translator-app PRIVATE bergamot-translator)
-
-if (NOT USE_WASM_COMPATIBLE_SOURCE)
-    add_executable(service-cli service-cli.cpp)
-    target_link_libraries(service-cli PRIVATE bergamot-translator)
-
-    add_executable(marian-decoder-new marian-decoder-new.cpp)
-    target_link_libraries(marian-decoder-new PRIVATE bergamot-translator)
-endif()
+add_executable(bergamot bergamot.cpp)
+target_link_libraries(bergamot PRIVATE bergamot-translator)
diff --git a/app/bergamot-translator-app.cpp b/app/bergamot-translator-app.cpp
deleted file mode 100644
index 77bc99875..000000000
--- a/app/bergamot-translator-app.cpp
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * main.cpp
- *
- * An application which accepts line separated texts in stdin and returns
- * translated ones in stdout. It is convenient for batch processing and can be
- * used with tools like SacreBLEU.
- *
- */
-
-#include <iostream>
-#include <string>
-
-#include "translator/parser.h"
-#include "translator/service.h"
-
-int main(int argc, char **argv) {
-
-  // Create a configParser and load command line parameters into a YAML config
-  // string.
-  auto configParser = marian::bergamot::createConfigParser();
-  auto options = configParser.parseOptions(argc, argv, true);
-  std::string config = options->asYamlString();
-
-  // Route the config string to construct marian model through TranslationModel
-  marian::bergamot::Service model(config);
-
-  marian::bergamot::ResponseOptions responseOptions;
-  std::vector<std::string> texts;
-
-  for (std::string line; std::getline(std::cin, line);) {
-    texts.emplace_back(line);
-  }
-
-  auto results = model.translateMultiple(std::move(texts), responseOptions);
-
-  for (auto &result : results) {
-    std::cout << result.getTranslatedText() << std::endl;
-  }
-
-  return 0;
-}
diff --git a/app/bergamot.cpp b/app/bergamot.cpp
new file mode 100644
index 000000000..19dea1fcf
--- /dev/null
+++ b/app/bergamot.cpp
@@ -0,0 +1,18 @@
+#include "cli.h"
+
+int main(int argc, char *argv[]) {
+  auto cp = marian::bergamot::createConfigParser();
+  auto options = cp.parseOptions(argc, argv, true);
+  const std::string mode = options->get<std::string>("bergamot-mode");
+  using namespace marian::bergamot;
+  if (mode == "wasm") {
+    app::wasm(options);
+  } else if (mode == "native") {
+    app::native(options);
+  } else if (mode == "decoder") {
+    app::decoder(options);
+  } else {
+    ABORT("Unknown --mode {}. Use one of: {wasm,native,decoder}", mode);
+  }
+  return 0;
+}
diff --git a/app/cli.h b/app/cli.h
new file mode 100644
index 000000000..292d21c2e
--- /dev/null
+++ b/app/cli.h
@@ -0,0 +1,186 @@
+#ifndef BERGAMOT_APP_CLI_H
+#define BERGAMOT_APP_CLI_H
+#include <cstdlib>
+#include <future>
+#include <iostream>
+#include <sstream>
+
+#include "common/definitions.h"
+#include "common/timer.h"
+#include "common/utils.h"
+#include "marian.h"
+#include "translator/byte_array_util.h"
+#include "translator/parser.h"
+#include "translator/response.h"
+#include "translator/response_options.h"
+#include "translator/service.h"
+
+namespace marian {
+namespace bergamot {
+
+// marian::bergamot:: makes life easier, won't need to prefix it everywhere and these classes plenty use constructs.
+
+namespace app {
+
+/// Previously bergamot-translator-app. Provides a command-line app on native which executes the code-path used by Web
+/// Assembly. Expected to be maintained consistent with how the browser (Mozilla through WebAssembly) dictates its API
+/// and tests be intact. Also used in [bergamot-evaluation](https://github.com/mozilla/bergamot-evaluation).
+///
+/// Usage example:
+/// [brt/tests/basic/test_bergamot_translator_app_intgemm_8bit.cpu-threads.0.sh](https://github.com/browsermt/bergamot-translator-tests/blob/main/tests/basic/test_bergamot_translator_app_intgemm_8bit.cpu-threads.0.sh)
+///
+/// * Input : read from stdin as sentences as lines of text.
+/// * Output: written to stdout as translations for the sentences supplied in corresponding lines
+///
+/// @param [options]: Options to translate passed down to marian through Options.
+void wasm(Ptr<Options> options) {
+  // Here, we take the command-line interface which is uniform across all apps. This is parsed into Ptr<Options> by
+  // marian. However, mozilla does not allow a Ptr<Options> constructor and demands an std::string constructor since
+  // std::string isn't marian internal unlike Ptr<Options>. Since this std::string path needs to be tested for mozilla
+  // and since this class/CLI is intended at testing mozilla's path, we go from:
+  //
+  // cmdline -> Ptr<Options> -> std::string -> Service(std::string)
+  //
+  // Overkill, yes.
+
+  std::string config = options->asYamlString();
+  Service model(config);
+
+  ResponseOptions responseOptions;
+  std::vector<std::string> texts;
+
+  for (std::string line; std::getline(std::cin, line);) {
+    texts.emplace_back(line);
+  }
+
+  auto results = model.translateMultiple(std::move(texts), responseOptions);
+
+  for (auto &result : results) {
+    std::cout << result.getTranslatedText() << std::endl;
+  }
+}
+
+/// Application used to benchmark with marian-decoder from time-to-time. The implementation in this repository follows a
+/// different route than marian-decoder  and routinely needs to be checked that the speeds while operating similar to
+/// marian-decoder are not affected during the course of development.
+///
+/// Example usage:
+/// [brt/speed-tests/test_wngt20_perf.sh](https://github.com/browsermt/bergamot-translator-tests/blob/main/speed-tests/test_wngt20_perf.sh).
+///
+/// Expected to be compatible with Translator[1] and marian-decoder[2].
+///
+/// - [1]
+/// [marian-dev/../src/translator/translator.h](https://github.com/marian-nmt/marian-dev/blob/master/src/translator/translator.h)
+/// - [2]
+/// [marian-dev/../src/command/marian_decoder.cpp](https://github.com/marian-nmt/marian/blob/master/src/command/marian_decoder.cpp)
+///
+/// * Input: stdin, lines containing sentences, same as marian-decoder.
+/// * Output: to stdout, translations of the sentences supplied via stdin in corresponding lines
+///
+/// @param [in] options: constructed from command-line supplied arguments
+void decoder(Ptr<Options> options) {
+  marian::timer::Timer decoderTimer;
+  Service service(options);
+  // Read a large input text blob from stdin
+  std::ostringstream std_input;
+  std_input << std::cin.rdbuf();
+  std::string input = std_input.str();
+
+  // Wait on future until Response is complete
+  std::future<Response> responseFuture = service.translate(std::move(input));
+  responseFuture.wait();
+  const Response &response = responseFuture.get();
+
+  for (size_t sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
+    std::cout << response.target.sentence(sentenceIdx) << "\n";
+  }
+  LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
+}
+
+/// Command line interface to the test the features being developed as part of bergamot C++ library on native platform.
+///
+/// Usage example:
+/// [brt/tests/basic/test_service-cli_intgemm_8bit.cpu-threads.4.sh](https://github.com/browsermt/bergamot-translator-tests/blob/main/tests/basic/test_service-cli_intgemm_8bit.cpu-threads.4.sh)
+///
+/// * Input: reads from stdin, blob of text, read as a whole ; sentence-splitting etc handled internally.
+/// * Output: to stdout, translation of the source text and additional information like sentences, alignments between
+/// source and target tokens and quality scores.
+///
+/// @param [in] options: options to build translator
+void native(Ptr<Options> options) {
+  // Prepare memories for bytearrays (including model, shortlist and vocabs)
+  MemoryBundle memoryBundle;
+
+  if (options->get<bool>("bytearray")) {
+    // Load legit values into bytearrays.
+    memoryBundle = getMemoryBundleFromConfig(options);
+  }
+
+  Service service(options, std::move(memoryBundle));
+
+  // Read a large input text blob from stdin
+  std::ostringstream std_input;
+  std_input << std::cin.rdbuf();
+  std::string input = std_input.str();
+
+  ResponseOptions responseOptions;
+  responseOptions.qualityScores = true;
+  responseOptions.alignment = true;
+  responseOptions.alignmentThreshold = 0.2f;
+
+  // Wait on future until Response is complete
+  std::future<Response> responseFuture = service.translate(std::move(input), responseOptions);
+  responseFuture.wait();
+  Response response = responseFuture.get();
+
+  std::cout << "[original]: " << response.source.text << '\n';
+  std::cout << "[translated]: " << response.target.text << '\n';
+  for (int sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
+    std::cout << " [src Sentence]: " << response.source.sentence(sentenceIdx) << '\n';
+    std::cout << " [tgt Sentence]: " << response.target.sentence(sentenceIdx) << '\n';
+    std::cout << "Alignments" << '\n';
+    typedef std::pair<size_t, float> Point;
+
+    // Initialize a point vector.
+    std::vector<std::vector<Point>> aggregate(response.source.numWords(sentenceIdx));
+
+    // Handle alignments
+    auto &alignments = response.alignments[sentenceIdx];
+    for (auto &p : alignments) {
+      aggregate[p.src].emplace_back(p.tgt, p.prob);
+    }
+
+    for (size_t src = 0; src < aggregate.size(); src++) {
+      std::cout << response.source.word(sentenceIdx, src) << ": ";
+      for (auto &p : aggregate[src]) {
+        std::cout << response.target.word(sentenceIdx, p.first) << "(" << p.second << ") ";
+      }
+      std::cout << '\n';
+    }
+
+    // Handle quality.
+    auto &quality = response.qualityScores[sentenceIdx];
+    std::cout << "Quality: whole(" << quality.sequence << "), tokens below:" << '\n';
+    size_t wordIdx = 0;
+    bool first = true;
+    for (auto &p : quality.word) {
+      if (first) {
+        first = false;
+      } else {
+        std::cout << " ";
+      }
+      std::cout << response.target.word(sentenceIdx, wordIdx) << "(" << p << ")";
+      wordIdx++;
+    }
+    std::cout << '\n';
+  }
+  std::cout << "--------------------------\n";
+  std::cout << '\n';
+}
+
+}  // namespace app
+
+}  // namespace bergamot
+}  // namespace marian
+
+#endif  // BERGAMOT_APP_CLI_H
diff --git a/app/marian-decoder-new.cpp b/app/marian-decoder-new.cpp
deleted file mode 100644
index a2516d9c2..000000000
--- a/app/marian-decoder-new.cpp
+++ /dev/null
@@ -1,46 +0,0 @@
-#include <cstdlib>
-#include <future>
-#include <iostream>
-#include <sstream>
-
-#include "common/definitions.h"
-#include "common/timer.h"
-#include "common/utils.h"
-#include "marian.h"
-#include "translator/history.h"
-#include "translator/output_collector.h"
-#include "translator/output_printer.h"
-#include "translator/parser.h"
-#include "translator/response.h"
-#include "translator/service.h"
-
-void marian_decoder_minimal(const marian::bergamot::Response &response,
-                            marian::Ptr<marian::Options> options) {
-  // We are no longer marian-decoder compatible. Server ideas are on hold.
-  for (size_t sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
-    std::cout << response.target.sentence(sentenceIdx) << "\n";
-  }
-}
-
-int main(int argc, char *argv[]) {
-  auto cp = marian::bergamot::createConfigParser();
-  auto options = cp.parseOptions(argc, argv, true);
-  marian::timer::Timer decoderTimer;
-
-  marian::bergamot::Service service(options);
-  // Read a large input text blob from stdin
-  std::ostringstream std_input;
-  std_input << std::cin.rdbuf();
-  std::string input = std_input.str();
-  using marian::bergamot::Response;
-
-  // Wait on future until Response is complete
-  std::future<Response> responseFuture = service.translate(std::move(input));
-  responseFuture.wait();
-  const Response &response = responseFuture.get();
-
-  marian_decoder_minimal(response, options);
-
-  LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
-  return 0;
-}
diff --git a/app/service-cli.cpp b/app/service-cli.cpp
deleted file mode 100644
index a29e71bb2..000000000
--- a/app/service-cli.cpp
+++ /dev/null
@@ -1,97 +0,0 @@
-#include <cstdlib>
-#include <future>
-#include <iostream>
-#include <sstream>
-
-#include "common/definitions.h"
-#include "common/utils.h"
-#include "marian.h"
-#include "translator/byte_array_util.h"
-#include "translator/parser.h"
-#include "translator/response.h"
-#include "translator/response_options.h"
-#include "translator/service.h"
-
-int main(int argc, char *argv[]) {
-  auto cp = marian::bergamot::createConfigParser();
-  auto options = cp.parseOptions(argc, argv, true);
-
-  // Prepare memories for bytearrays (including model, shortlist and vocabs)
-  marian::bergamot::MemoryBundle memoryBundle;
-
-  if (options->get<bool>("bytearray")) {
-    // Load legit values into bytearrays.
-    memoryBundle = marian::bergamot::getMemoryBundleFromConfig(options);
-  }
-
-  marian::bergamot::Service service(options, std::move(memoryBundle));
-
-  // Read a large input text blob from stdin
-  std::ostringstream std_input;
-  std_input << std::cin.rdbuf();
-  std::string input = std_input.str();
-  using marian::bergamot::Response;
-
-  marian::bergamot::ResponseOptions responseOptions;
-  responseOptions.qualityScores = true;
-  responseOptions.alignment = true;
-  responseOptions.alignmentThreshold = 0.2f;
-
-  // Wait on future until Response is complete
-  std::future<Response> responseFuture =
-      service.translate(std::move(input), responseOptions);
-  responseFuture.wait();
-  Response response = responseFuture.get();
-
-  std::cout << "[original]: " << response.source.text << '\n';
-  std::cout << "[translated]: " << response.target.text << '\n';
-  for (int sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
-    std::cout << " [src Sentence]: " << response.source.sentence(sentenceIdx)
-              << '\n';
-    std::cout << " [tgt Sentence]: " << response.target.sentence(sentenceIdx)
-              << '\n';
-    std::cout << "Alignments" << '\n';
-    typedef std::pair<size_t, float> Point;
-
-    // Initialize a point vector.
-    std::vector<std::vector<Point>> aggregate(
-        response.source.numWords(sentenceIdx));
-
-    // Handle alignments
-    auto &alignments = response.alignments[sentenceIdx];
-    for (auto &p : alignments) {
-      aggregate[p.src].emplace_back(p.tgt, p.prob);
-    }
-
-    for (size_t src = 0; src < aggregate.size(); src++) {
-      std::cout << response.source.word(sentenceIdx, src) << ": ";
-      for (auto &p : aggregate[src]) {
-        std::cout << response.target.word(sentenceIdx, p.first) << "("
-                  << p.second << ") ";
-      }
-      std::cout << '\n';
-    }
-
-    // Handle quality.
-    auto &quality = response.qualityScores[sentenceIdx];
-    std::cout << "Quality: whole(" << quality.sequence
-              << "), tokens below:" << '\n';
-    size_t wordIdx = 0;
-    bool first = true;
-    for (auto &p : quality.word) {
-      if (first) {
-        first = false;
-      } else {
-        std::cout << " ";
-      }
-      std::cout << response.target.word(sentenceIdx, wordIdx) << "(" << p
-                << ")";
-      wordIdx++;
-    }
-    std::cout << '\n';
-  }
-  std::cout << "--------------------------\n";
-  std::cout << '\n';
-
-  return 0;
-}
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 1b20a62f6..020135af1 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 1b20a62f6614db371f59b97ff83262b8ebd235de
+Subproject commit 020135af1b620caa27929c1403c50ec3299e5bff
diff --git a/doc/conf.py b/doc/conf.py
index a9863cfe5..bffcda0cd 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -82,7 +82,7 @@
 breathe_default_project = 'bergamot-translator'
 
 doxygen_config = """
-INPUT                = ../src
+INPUT                = ../src ../app
 EXCLUDE             += ../3rd_party
 EXCLUDE             += ../src/tests
 EXCLUDE_PATTERNS     = *.md *.txt
diff --git a/doc/marian-integration.md b/doc/marian-integration.md
index d2957cbc9..d4ba82475 100644
--- a/doc/marian-integration.md
+++ b/doc/marian-integration.md
@@ -1,11 +1,23 @@
-# Building marian code for bergamot
+# Bergamot C++ Library
 
-This document summarizes the minimal build instructions develop for the
-marian machine translation toolkit powering bergamot-translator.
+This document contains instructions to develop for modifications on top of the
+marian machine translation toolkit powering bergamot-translator. The library is
+optimized towards fast and efficient translation of a given input.
 
 ## Build Instructions
 
-Marian CPU version requires Intel MKL or OpenBLAS. Both are free, but MKL is not open-sourced. Intel MKL is strongly recommended as it is faster. On Ubuntu 16.04 and newer it can be installed from the APT repositories.
+Note: You are strongly advised to refer to the continuous integration on this
+repository, which builds bergamot-translator and associated applications from
+scratch. Examples to run these command line-applications are available in the
+[bergamot-translator-tests](https://github.com/browsermt/bergamot-translator-tests)
+repository. Builds take about 30 mins on a consumer grade machine, so using a
+tool like ccache is highly recommended.
+
+### Dependencies 
+
+Marian CPU version requires Intel MKL or OpenBLAS. Both are free, but MKL is
+not open-sourced. Intel MKL is strongly recommended as it is faster. On Ubuntu
+16.04 and newer it can be installed from the APT repositories.
 
 ```bash
 wget -qO- 'https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB' | sudo apt-key add -
@@ -15,32 +27,47 @@ sudo apt-get install intel-mkl-64bit-2020.0-088
 ```
 On MacOS, apple accelerate framework will be used instead of MKL/OpenBLAS.
 
+
+### Building bergamot-translator
+
+Web Assembly (WASM) reduces building to only using a subset of functionalities
+of marian, the translation library powering bergamot-translator. When
+developing bergamot-translator it is important that the sources added be
+compatible with marian.  Therefore, it is required to set
+`-DUSE_WASM_COMPATIBLE_SOURCE=on`.
+
 ```
 $ git clone https://github.com/browsermt/bergamot-translator
 $ cd bergamot-translator
 $ mkdir build
 $ cd build
 $ cmake .. -DUSE_WASM_COMPATIBLE_SOURCE=off -DCMAKE_BUILD_TYPE=Release
-$ make -j
+$ make -j2 
 ```
 
-
 The build will generate the library that can be linked to any project. All the
 public header files are specified in `src` folder.
 
 ## Command line apps
 
-The following executables are created by the build:
+Bergamot-translator is intended to be used as a library. However, we provide a
+command-line application which is capable of translating text provided on
+standard-input. During development this application is used to perform
+regression-tests.
 
-1. `app/service-cli`: Extends marian to capability to work with string_views.
-   `service-cli` exists to check if the underlying code, without the
-   integration works or not.
-2. `app/bergamot-translator-app`: App which integreates service-cli's
-   functionality into the translator agnostic API specified as part of the
-   project. Integration failures are detected if same arguments work with
-   `service-cli` and does not with `bergamot-translator-app`.
-3. `app/marian-decoder-new`: Helper executable to conveniently benchmark new
-   implementation with the optimized upstream marian-decoder.
+There are effectively multiple CLIs subclassed from a unified interface all
+provided in `app/cli.h`. These are packed into a single executable named
+`bergamot` by means of a `--bergamot-mode BERGAMOT_MODE` switch. 
+
+The following modes are available:
+
+* `--bergamot-mode native` 
+* `--bergamot-mode wasm`    
+* `--bergamot-mode decoder` 
+
+Find documentation on these modes with the API documentation for apps [here](./api/namespace_marian__bergamot__app.html#functions).
+
+## Example command line run
 
 The models required to run the command-line are available at
 [data.statmt.org/bergamot/models/](http://data.statmt.org/bergamot/models/).
@@ -49,13 +76,11 @@ at:
 
 * [data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz](http://data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz)
 
-<details>
-<summary> Example run of commandline: Click to expand </summary>
-<p>
-
 ```bash
 MODEL_DIR=... # path to where the model-files are.
+BERGAMOT_MODE='native'
 ARGS=(
+    --bergamot-mode $BERGAMOT_MODE
     -m $MODEL_DIR/model.intgemm.alphas.bin # Path to model file.
     --vocabs 
         $MODEL_DIR/vocab.deen.spm # source-vocabulary
@@ -84,14 +109,10 @@ ARGS=(
     --ssplit-mode paragraph 
 )
 
-./app/service-cli "${ARGS[@]}" < path-to-input-file
-./app/bergamot-translator-app "${ARGS[@]}" < path-to-input-file
+./app/bergamot "${ARGS[@]}" < path-to-input-file
 
 ```
-</p>
 
-</summary>
-</details>
 
 ## Coding Style
 
@@ -108,4 +129,3 @@ used to also check for the coding style in the CI.
 ```bash
 python3 run-clang-format.py -i --style file -r src wasm
 ```
-
diff --git a/src/translator/parser.h b/src/translator/parser.h
index 790717c24..cd7096531 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -26,6 +26,9 @@ inline marian::ConfigParser createConfigParser() {
   cp.addOption<bool>("--check-bytearray", "Bergamot Options",
                      "Flag holds whether to check the content of the bytearrays (true by default)", true);
 
+  cp.addOption<std::string>("--bergamot-mode", "Bergamot Options",
+                            "Operating mode for bergamot: [wasm, native, decoder]", "native");
+
   return cp;
 }
 

From ceaf21a532ff4cb3f1e2016edbca0d71a92cefc3 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 1 Jun 2021 11:00:53 +0100
Subject: [PATCH 261/442] Deploy generated documentation only if browsermt
 (#179)

---
 .github/workflows/doc.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml
index 77307d695..706465e39 100644
--- a/.github/workflows/doc.yml
+++ b/.github/workflows/doc.yml
@@ -67,7 +67,7 @@ jobs:
 
       - name: Deploy 🚀
         uses: JamesIves/github-pages-deploy-action@4.1.3
-        if: ${{ github.event_name == 'push' }}
+        if: ${{ github.event_name == 'push' && github.repository == 'browsermt/bergamot-translator' }}
         with:
           repository-name: 'browsermt/docs' 
           branch: gh-pages # The branch the action should deploy to.

From 330840338cd0c45f4083f0dc3140bd7e465da23a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 1 Jun 2021 12:39:28 +0100
Subject: [PATCH 262/442] Including WASM documentation in sphinx build toc
 (#176)

---
 doc/index.rst       |  1 +
 doc/wasm-example.md |  1 +
 wasm/README.md      | 24 ++++++++++++++++--------
 3 files changed, 18 insertions(+), 8 deletions(-)
 create mode 120000 doc/wasm-example.md

diff --git a/doc/index.rst b/doc/index.rst
index 85bc7f109..5be3857a3 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -15,6 +15,7 @@ This is developer documentation.
    :caption: Contents:
 
    marian-integration
+   wasm-example
    api/library_index
 
 
diff --git a/doc/wasm-example.md b/doc/wasm-example.md
new file mode 120000
index 000000000..9188e9356
--- /dev/null
+++ b/doc/wasm-example.md
@@ -0,0 +1 @@
+../wasm/README.md
\ No newline at end of file
diff --git a/wasm/README.md b/wasm/README.md
index eceac152b..01fea18b0 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -1,9 +1,17 @@
-## Using Bergamot Translator in JavaScript
-The example file `bergamot.html` in the folder `test_page` demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
+# Using Bergamot Translator in JavaScript
 
-### <a name="Pre-requisite"></a> Pre-requisite: Download files required for translation
+Instructions in this document assume current-directory to be
+[wasm](https://github.com/browsermt/bergamot-translator/tree/main/wasm) within
+bergamot-translator source.
 
-Please note that [Using JS APIs](#Using-JS-APIs) and [Demo](#Demo) section below assumes that the [bergamot project specific model files](https://github.com/mozilla-applied-ml/bergamot-models) are already downloaded and present in the `test_page` folder. If this is not done then use following instructions to do so:
+The example file `bergamot.html` in the folder `test_page` demonstrates how to
+use the bergamot translator in JavaScript via a `<script>` tag.
+
+## Pre-requisites
+
+**Download files required for translation**
+
+Please note that [Using JS APIs](#using-js-apis) and [Demo](#demo) section below assumes that the [bergamot project specific model files](https://github.com/mozilla-applied-ml/bergamot-models) are already downloaded and present in the `test_page` folder. If this is not done then use following instructions to do so:
 
 ```bash
 cd test_page
@@ -13,7 +21,7 @@ cp -rf bergamot-models/prod/* models
 gunzip models/*/*
 ```
 
-### <a name="Using-JS-APIs"></a> Using JS APIs
+## Using JS APIs
 
 ```js
 // The model configuration as YAML formatted string. For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
@@ -79,9 +87,9 @@ request.delete();
 input.delete();
 ```
 
-### <a name="Demo"></a> Demo (see everything in action)
+## Demo 
 
-* Make sure that you followed [Pre-requisite](#Pre-requisite) instructions before moving forward.
+* Make sure that you followed [Pre-requisites](#pre-requisites) instructions before moving forward.
 
 * Start the test webserver (ensure you have the latest nodejs installed)
     ```bash
@@ -112,4 +120,4 @@ input.delete();
 * Run some translations:
     * Choose a model and press `Load Model`
     * Type a sentence to be translated in the `From` textbox and press `Translate`
-    * See the results in the `To` and `Log` textboxes
\ No newline at end of file
+    * See the results in the `To` and `Log` textboxes

From 73228bbb4a63c87203ef31bea2bee134f1c1c140 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Thu, 3 Jun 2021 21:01:26 +0100
Subject: [PATCH 263/442] Updating marian-dev: intgemm with env variable matmul
 switches (#187)

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 2a6fdd4cc..ad92307e0 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 2a6fdd4cc41b0976d33165cec04062d192de9c40
+Subproject commit ad92307e0f4ab66f75b03ddc87c48d28d07757f1

From 5f0d3963e228461e37b99f2394250174183fc007 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Fri, 4 Jun 2021 00:09:20 +0100
Subject: [PATCH 264/442] Remove addSentenceWithPriority (#186)

---
 src/translator/batcher.cpp            | 12 ++++--------
 src/translator/batcher.h              |  1 -
 src/translator/threadsafe_batcher.cpp |  8 --------
 src/translator/threadsafe_batcher.h   |  1 -
 4 files changed, 4 insertions(+), 18 deletions(-)

diff --git a/src/translator/batcher.cpp b/src/translator/batcher.cpp
index 8735c4ba8..0a14459f1 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batcher.cpp
@@ -16,12 +16,6 @@ Batcher::Batcher(Ptr<Options> options) {
            "longer than what can fit in a batch.");
 }
 
-void Batcher::addSentenceWithPriority(RequestSentence &sentence) {
-  size_t bucket_id = sentence.numTokens();
-  assert(bucket_id < bucket_.size());
-  bucket_[bucket_id].insert(sentence);
-}
-
 bool Batcher::cleaveBatch(Batch &batch) {
   // For now simply iterates on buckets and converts batches greedily.  This
   // has to be enhanced with optimizing over priority. The baseline
@@ -52,8 +46,10 @@ bool Batcher::cleaveBatch(Batch &batch) {
 
 void Batcher::addWholeRequest(Ptr<Request> request) {
   for (size_t i = 0; i < request->numSegments(); i++) {
-    RequestSentence requestSentence(i, request);
-    addSentenceWithPriority(requestSentence);
+    RequestSentence sentence(i, request);
+    size_t bucket_id = sentence.numTokens();
+    assert(bucket_id < bucket_.size());
+    bucket_[bucket_id].insert(sentence);
   }
 }
 
diff --git a/src/translator/batcher.h b/src/translator/batcher.h
index 7735a0895..277bfc934 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batcher.h
@@ -19,7 +19,6 @@ class Batcher {
   // RequestSentence incorporates (tentative) notions of priority with each
   // sentence. This method inserts the sentence into the internal data-structure
   // which maintains priority among sentences from multiple concurrent requests.
-  void addSentenceWithPriority(RequestSentence &sentence);
   void addWholeRequest(Ptr<Request> request);
 
   // indicate no more sentences will be added.  Does nothing here, for parity to threadsafe version.
diff --git a/src/translator/threadsafe_batcher.cpp b/src/translator/threadsafe_batcher.cpp
index 7e03d67ce..38b6681a9 100644
--- a/src/translator/threadsafe_batcher.cpp
+++ b/src/translator/threadsafe_batcher.cpp
@@ -10,14 +10,6 @@ ThreadsafeBatcher::ThreadsafeBatcher(Ptr<Options> options) : backend_(options),
 
 ThreadsafeBatcher::~ThreadsafeBatcher() { shutdown(); }
 
-void ThreadsafeBatcher::addSentenceWithPriority(RequestSentence &sentence) {
-  std::unique_lock<std::mutex> lock(mutex_);
-  assert(!shutdown_);
-  backend_.addSentenceWithPriority(sentence);
-  ++enqueued_;
-  work_.notify_one();
-}
-
 void ThreadsafeBatcher::addWholeRequest(Ptr<Request> request) {
   std::unique_lock<std::mutex> lock(mutex_);
   assert(!shutdown_);
diff --git a/src/translator/threadsafe_batcher.h b/src/translator/threadsafe_batcher.h
index 195c3cd8e..d0ab7b1cc 100644
--- a/src/translator/threadsafe_batcher.h
+++ b/src/translator/threadsafe_batcher.h
@@ -27,7 +27,6 @@ class ThreadsafeBatcher {
 
   // Add sentences to be translated by calling these (see Batcher).  When
   // done, call shutdown.
-  void addSentenceWithPriority(RequestSentence &sentence);
   void addWholeRequest(Ptr<Request> request);
   void shutdown();
 

From 71a62405e750bdc2012b5794d2c421e86b710000 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Fri, 4 Jun 2021 11:52:36 +0100
Subject: [PATCH 265/442] Update native (ubuntu, mac) workflows with ccache
 (#181)

* Matrix is now more organized, Ubuntu 20.04-gcc9.3, Ubuntu-18.04-gcc7.5 is added.
* ccache is extended to MacOS, and brings down CI run times to <5m when
  ccache works.
* The compiler hash scripts are gone, ccache already covers most ground
  by default. The shell script is unnecessary. Cache works by preprocessor
  mode output of running the compiler with -E, which includes the
  necessary information. ccache-docs:How the cache works.
* BRT if failed prints the final 20 lines of the test*.log to inspect
  what's going wrong without having to artifact download.
* Pull request on any branch triggers workflow.
* Push on main and ci-sandbox triggers workflow.
---
 .github/workflows/native-mac.yml    | 112 -------------
 .github/workflows/native-ubuntu.yml | 202 -----------------------
 .github/workflows/native.yml        | 238 ++++++++++++++++++++++++++++
 3 files changed, 238 insertions(+), 314 deletions(-)
 delete mode 100644 .github/workflows/native-mac.yml
 delete mode 100644 .github/workflows/native-ubuntu.yml
 create mode 100644 .github/workflows/native.yml

diff --git a/.github/workflows/native-mac.yml b/.github/workflows/native-mac.yml
deleted file mode 100644
index bbd3651c1..000000000
--- a/.github/workflows/native-mac.yml
+++ /dev/null
@@ -1,112 +0,0 @@
-name: Native MacOS
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ main, ci-sandbox ]
-
-jobs:
-  build-macos:
-    strategy: 
-      fail-fast: false
-      matrix:
-        include:
-          - name: "full-marian"
-            os: macos-10.15
-            test_tags: "'#mac'"
-            cmake: 
-              CMAKE_BUILD_TYPE: "Release"
-              COMPILE_TESTS: "ON"
-              USE_WASM_COMPATIBLE_SOURCE: "OFF"
-              USE_FBGEMM: "OFF"
-              USE_STATIC_LIBS: "OFF"
-              COMPILE_SERVER: "OFF"
-              COMPILE_EXAMPLES: "OFF"
-              USE_APPLE_ACCELERATE: "OFF"
-
-          - name: "minimal-marian"
-            os: macos-10.15
-            test_tags: "'#wasm'"
-            cmake: 
-              CMAKE_BUILD_TYPE: "Release"
-              COMPILE_TESTS: "OFF" # Minimal marian has no sqlite support and compile tests fail
-              USE_WASM_COMPATIBLE_SOURCE: "ON"
-              USE_FBGEMM: "OFF"
-              # explicitly set due to requirement of minimal marian being used
-              # within WASM. This is some yaml ugliness, but issok.
-              USE_STATIC_LIBS: "ON" 
-              COMPILE_SERVER: "OFF"
-              COMPILE_EXAMPLES: "OFF"
-              USE_APPLE_ACCELERATE: "OFF"
-
-        
-    name: ${{ matrix.name }}
-    runs-on: ${{ matrix.os }}
-
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-
-    - name: Install dependencies
-      run: |
-          brew update
-          brew install openblas protobuf coreutils
-
-    # Openblas location is exported explicitly because openblas is keg-only,
-    # which means it was not symlinked into /usr/local/.
-    - name: Set BLAS Environment variables
-      run: |
-          echo "LDFLAGS=-L/usr/local/opt/openblas/lib" >> $GITHUB_ENV
-          echo "CPPFLAGS=-I/usr/local/opt/openblas/include" >> $GITHUB_ENV
-      if: matrix.cmake.USE_WASM_COMPATIBLE_SOURCE == 'OFF'
-
-    # CMake cannot find BLAS on GitHub runners if Marian is being compiled
-    # statically, hence USE_STATIC_LIBS=off
-    - name: Configure CMake
-      run: |
-        mkdir -p build
-        cd build
-        cmake -L .. \
-          -DCMAKE_BUILD_TYPE=${{ matrix.cmake.CMAKE_BUILD_TYPE }}\
-          -DCOMPILE_TESTS=${{ matrix.cmake.COMPILE_TESTS }}\
-          -DCOMPILE_EXAMPLES=${{ matrix.cmake.COMPILE_EXAMPLES }} \
-          -DCOMPILE_SERVER=${{ matrix.cmake.COMPILE_SERVER }} \
-          -DUSE_STATIC_LIBS=${{ matrix.cmake.USE_STATIC_LIBS }} \
-          -DUSE_WASM_COMPATIBLE_SOURCE=${{ matrix.cmake.USE_WASM_COMPATIBLE_SOURCE }} \
-          -DUSE_APPLE_ACCELERATE=${{ matrix.cmake.USE_APPLE_ACCELERATE }} \
-          -DUSE_FBGEMM=${{ matrix.cmake.USE_FBGEMM }}
-
-    - name: Compile
-      working-directory: build
-      run: make -j2
-
-    - name: Run unit tests
-      working-directory: build
-      run: make test
-      if: matrix.cmake.COMPILE_TESTS == 'ON'
-
-    - name: Print versions
-      working-directory: build
-      run: |
-        ./app/bergamot --version
-
-    - name: Install regression-test framework (BRT)
-      working-directory: bergamot-translator-tests
-      run : make install
-
-    - name: Run regression-tests (BRT)
-      working-directory: bergamot-translator-tests
-      run : MARIAN=../build ./run_brt.sh ${{ matrix.test_tags }}
-
-    - name: Upload regression-tests artifacts
-      uses: actions/upload-artifact@v2
-      if: ${{ always() }}
-      with: 
-        name: brt-artifacts-${{ matrix.name }}
-        path: |
-            bergamot-translator-tests/**/*.expected
-            bergamot-translator-tests/**/*.log
-            bergamot-translator-tests/**/*.out
diff --git a/.github/workflows/native-ubuntu.yml b/.github/workflows/native-ubuntu.yml
deleted file mode 100644
index 0617201e5..000000000
--- a/.github/workflows/native-ubuntu.yml
+++ /dev/null
@@ -1,202 +0,0 @@
-name: Native Ubuntu
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ main, ci-sandbox ]
-
-jobs:
-  build-ubuntu:
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          - name: "full-marian"
-            os: ubuntu-latest
-            gcc: 8
-            force_recache: false
-            ccache_cmd: "bash ${GITHUB_WORKSPACE}/scripts/ci/compiler-hash.sh %compiler%"
-            cpu: 'ON'
-            gpu: 'OFF'
-            test_tags: ""
-            cmake: 
-              CMAKE_BUILD_TYPE: "Release"
-              COMPILE_TESTS: "ON"
-              USE_WASM_COMPATIBLE_SOURCE: "OFF"
-              COMPILE_SERVER: "OFF"
-              COMPILE_EXAMPLES: "OFF"
-              CMAKE_C_COMPILER_LAUNCHER: "ccache"
-              CMAKE_CXX_COMPILER_LAUNCHER: "ccache"
-
-          - name: "minimal-marian"
-            os: ubuntu-latest
-            gcc: 8
-            force_recache: false
-            ccache_cmd: "bash ${GITHUB_WORKSPACE}/scripts/ci/compiler-hash.sh %compiler%"
-            cpu: 'ON'
-            gpu: 'OFF'
-            test_tags: "'#wasm'"
-            cmake:
-              CMAKE_BUILD_TYPE: "Release"
-              COMPILE_TESTS: "OFF" # Minimal marian has no sqlite support and COMPILE_TEST=ON fails.
-              USE_WASM_COMPATIBLE_SOURCE: "ON"
-              COMPILE_SERVER: "OFF"
-              COMPILE_EXAMPLES: "OFF"
-              CMAKE_C_COMPILER_LAUNCHER: "ccache"
-              CMAKE_CXX_COMPILER_LAUNCHER: "ccache"
-
-          - name: "full-marian-force-recache"
-            os: ubuntu-latest
-            gcc: 8
-            force_recache: true
-            ccache_cmd: "bash ${GITHUB_WORKSPACE}/scripts/ci/compiler-hash.sh %compiler%"
-            cpu: 'ON'
-            gpu: 'OFF'
-            test_tags: ""
-            cmake: 
-              CMAKE_BUILD_TYPE: "Release"
-              COMPILE_TESTS: "ON"
-              USE_WASM_COMPATIBLE_SOURCE: "OFF"
-              COMPILE_SERVER: "OFF"
-              COMPILE_EXAMPLES: "OFF"
-              CMAKE_C_COMPILER_LAUNCHER: "ccache"
-              CMAKE_CXX_COMPILER_LAUNCHER: "ccache"
-
-          - name: "minimal-marian-force-recache"
-            os: ubuntu-latest
-            gcc: 8
-            force_recache: true
-            ccache_cmd: "bash ${GITHUB_WORKSPACE}/scripts/ci/compiler-hash.sh %compiler%"
-            cpu: 'ON'
-            gpu: 'OFF'
-            test_tags: "'#wasm'"
-            cmake:
-              CMAKE_BUILD_TYPE: "Release"
-              COMPILE_TESTS: "OFF" # Minimal marian has no sqlite support and COMPILE_TEST=ON fails.
-              USE_WASM_COMPATIBLE_SOURCE: "ON"
-              COMPILE_SERVER: "OFF"
-              COMPILE_EXAMPLES: "OFF"
-              CMAKE_C_COMPILER_LAUNCHER: "ccache"
-              CMAKE_CXX_COMPILER_LAUNCHER: "ccache"
-
-
-    runs-on: ${{ matrix.os }}
-    name: ${{ matrix.name }}
-
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-
-    # The following packages are already installed on GitHub-hosted runners:
-    # build-essential openssl libssl-dev
-    # No need to install libprotobuf{17,10,9v5} on Ubuntu {20,18,16}.04 because
-    # it is installed together with libprotobuf-dev
-    - name: Install dependencies
-      run: |
-        sudo apt-get update 
-        sudo apt-get install -y \
-            libgoogle-perftools-dev libprotobuf-dev protobuf-compiler  \
-            libboost-all-dev g++-${{ matrix.gcc }} ccache
-
-    # https://software.intel.com/content/www/us/en/develop/articles/installing-intel-free-libs-and-python-apt-repo.html
-    - name: Install MKL
-      run: |
-        wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB" | sudo apt-key add -
-        sudo sh -c "echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list"
-        sudo apt-get update -o Dir::Etc::sourcelist="/etc/apt/sources.list.d/intel-mkl.list"
-        sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
-      if: matrix.cmake.USE_WASM_COMPATIBLE_SOURCE == 'OFF'
-
-    - name: Generate ccache_vars
-      id: ccache_vars
-      shell: bash
-      run: |
-          echo "::set-output name=hash::$(${{ matrix.ccache_cmd }})"
-          echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
-
-    - name: Setup ccache environment variables
-      run: | 
-        echo "CCACHE_COMPILERCHECK=${{ matrix.ccache_cmd }}" >> $GITHUB_ENV 
-        echo "CCACHE_BASE_DIR=${{ github.workspace }}" >> $GITHUB_ENV
-        echo "CCACHE_DIR=${{ github.workspace }}/.ccache" >> $GITHUB_ENV
-        echo "CCACHE_COMPRESS=true" >> $GITHUB_ENV
-        echo "CCACHE_COMPRESSLEVEL=6" >> $GITHUB_ENV
-        echo "CCACHE_MAXSIZE=2G" >> $GITHUB_ENV
-
-    - name: Setup ccache recache on
-      run: |
-        echo "CCACHE_RECACHE=" >> $GITHUB_ENV 
-      if: matrix.force_recache == true
-
-    - name: Cache-op for build-cache through ccache 
-      uses: actions/cache@v2
-      with:
-        path: ${{ env.CCACHE_DIR }}
-        key: ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-        restore-keys: |
-           ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}- 
-           ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}- 
-           ccache-${{ matrix.name }}- 
-
-    - name: Cache stats before build
-      run: |
-          ccache -s
-          ccache -z
-
-    # Boost is installed on GitHub-hosted runners in a non-standard location
-    # https://github.com/actions/virtual-environments/issues/687#issuecomment-610471671
-    - name: Configure CMake
-      run: |
-        mkdir -p build
-        cd build
-        CC=/usr/bin/gcc-${{ matrix.gcc }} CXX=/usr/bin/g++-${{ matrix.gcc }} CUDAHOSTCXX=/usr/bin/g++-${{ matrix.gcc }} \
-        cmake -L .. \
-          -DCMAKE_BUILD_TYPE=${{ matrix.cmake.CMAKE_BUILD_TYPE }}\
-          -DCOMPILE_TESTS=${{ matrix.cmake.COMPILE_TESTS }}\
-          -DCOMPILE_EXAMPLES=${{ matrix.cmake.COMPILE_EXAMPLES }} \
-          -DCOMPILE_SERVER=${{ matrix.cmake.COMPILE_SERVER }} \
-          -DUSE_WASM_COMPATIBLE_SOURCE=${{ matrix.cmake.USE_WASM_COMPATIBLE_SOURCE }} \
-          -DCMAKE_C_COMPILER_LAUNCHER=${{ matrix.cmake.CMAKE_C_COMPILER_LAUNCHER}} \
-          -DCMAKE_CXX_COMPILER_LAUNCHER=${{ matrix.cmake.CMAKE_CXX_COMPILER_LAUNCHER}} 
-
-
-    - name: Compile bergamot-translator
-      working-directory: build
-      run: make -j2
-
-    - name: Cache stats after build
-      run: |
-          ccache -s
-
-    - name: Run unit tests
-      working-directory: build
-      run: make test
-      # GitHub-hosted VMs do not have GPUs, so can not be run in CUDA builds
-      if: matrix.gpu == 'OFF' && matrix.cmake.COMPILE_TESTS == 'ON'
-
-    - name: Print versions
-      working-directory: build
-      run: |
-        ./app/bergamot --version
-
-
-    - name: Install regression-test framework (BRT)
-      working-directory: bergamot-translator-tests
-      run : make install
-
-    - name: Run regression-tests (BRT)
-      working-directory: bergamot-translator-tests
-      run : MARIAN=../build ./run_brt.sh ${{ matrix.test_tags }}
-
-    - name: Upload regression-tests artifacts
-      uses: actions/upload-artifact@v2
-      if: ${{ always() }}
-      with: 
-        name: brt-artifacts-${{ matrix.name }}
-        path: |
-            bergamot-translator-tests/**/*.expected
-            bergamot-translator-tests/**/*.log
-            bergamot-translator-tests/**/*.out
diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
new file mode 100644
index 000000000..eb3f4d47e
--- /dev/null
+++ b/.github/workflows/native.yml
@@ -0,0 +1,238 @@
+name: native
+'on':
+  push:
+    branches:
+    - main
+    - ci-sandbox
+  pull_request:
+    branches:
+    - '**'
+env:
+  ccache_basedir: ${{ github.workspace }}
+  ccache_dir: "${{ github.workspace }}/.ccache"
+  ccache_compilercheck: content
+  ccache_compress: 'true'
+  ccache_compresslevel: 9
+  ccache_maxsize: 200M
+  ccache_cmake: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
+jobs:
+  ubuntu:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+        - name: Ubuntu 18.04 full
+          os: ubuntu-18.04
+          identifier: ubuntu_1804_full
+          cmake: -DCOMPILE_TESTS=on
+          brt_tags: ''
+          unittests: 'true'
+        - name: Ubuntu 18.04 minimal
+          os: ubuntu-18.04
+          identifier: ubuntu_1804_minimal
+          cmake: -DCOMPILE_TESTS=off -DUSE_WASM_COMPATIBLE_SOURCE=on
+          brt_tags: "'#wasm'"
+          unittests: 'false'
+        - name: Ubuntu 20.04 full
+          os: ubuntu-20.04
+          identifier: ubuntu_2004_full
+          cmake: -DCOMPILE_TESTS=on
+          brt_tags: ''
+          unittests: 'true'
+        - name: Ubuntu 20.04 minimal
+          os: ubuntu-20.04
+          identifier: ubuntu_2004_minimal
+          cmake: -DCOMPILE_TESTS=off -DUSE_WASM_COMPATIBLE_SOURCE=on
+          brt_tags: "'#wasm'"
+          unittests: 'false'
+    name: ${{ matrix.name }}
+    runs-on: ${{ matrix.os }}
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+    - name: Install Dependencies
+      run: |-
+        sudo apt-get update
+        sudo apt-get install -y \
+          libgoogle-perftools-dev libprotobuf-dev protobuf-compiler \
+          libboost-all-dev ccache
+    - name: Install MKL
+      run: |-
+        wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB" | sudo apt-key add -
+        sudo sh -c "echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list"
+        sudo apt-get update -o Dir::Etc::sourcelist="/etc/apt/sources.list.d/intel-mkl.list"
+        sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
+    - name: Generate ccache_vars for ccache based on machine
+      shell: bash
+      id: ccache_vars
+      run: |-
+        echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
+        echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
+    - name: Cache-op for build-cache through ccache
+      uses: actions/cache@v2
+      with:
+        path: ${{ env.ccache_dir }}
+        key: ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+        restore-keys: |-
+          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
+          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}
+          ccache-${{ matrix.identifier }}
+    - name: ccache environment setup
+      run: |-
+        echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
+        echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
+        echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
+        echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
+        echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
+        echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
+    - name: ccache prolog
+      run: |-
+        ccache -s # Print current cache stats
+        ccache -z # Zero cache entry
+    - name: cmake
+      run: |-
+        mkdir -p build
+        cd build
+        cmake -L .. ${{ matrix.cmake }} ${{ env.ccache_cmake }}
+    - name: Build from source
+      working-directory: build
+      run: make -j2
+    - name: ccache epilog
+      run: 'ccache -s # Print current cache stats'
+    - name: Print Versions
+      working-directory: build
+      run: ./app/bergamot --version
+    - name: Run unit tests
+      working-directory: build
+      run: make test
+      if: ${{ matrix.unittests == 'true' }}
+    - name: Install regression-test framework (BRT)
+      working-directory: bergamot-translator-tests
+      run: make install
+    - name: Run regression-tests (BRT)
+      working-directory: bergamot-translator-tests
+      id: brt_run
+      run: MARIAN=../build ./run_brt.sh ${{ matrix.brt_tags }}
+    - name: Print logs of unsuccessful BRTs
+      working-directory: bergamot-translator-tests
+      run: |-
+        grep "tests.*.sh" previous.log \
+          | sed 's/^\s*-\s*//' \
+          | xargs -I% bash -c 'echo %; tail -n20 %.log'
+      if: ${{ always() && steps.brt_run.outcome == 'failure' }}
+    - name: Upload regression-tests artifacts
+      uses: actions/upload-artifact@v2
+      if: ${{ always() && steps.brt_run.outcome != 'skipped' }}
+      with:
+        name: brt-${{ matrix.identifier }}
+        path: |-
+          bergamot-translator-tests/**/*.expected
+          bergamot-translator-tests/**/*.log
+          bergamot-translator-tests/**/*.out
+  mac:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+        - name: MacOS 10.15 full
+          os: macos-10.15
+          identifier: mac_1015_full
+          cmake: -DCOMPILE_TESTS=on -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=off
+          brt_tags: "'#mac'"
+          unittests: 'true'
+        - name: MacOS 10.15 minimal
+          os: macos-10.15
+          identifier: mac_1015_minimal
+          cmake: -DCOMPILE_TESTS=off -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
+          brt_tags: "'#wasm'"
+          unittests: 'false'
+    name: ${{ matrix.name }}
+    runs-on: ${{ matrix.os }}
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+    - name: Install Dependencies
+      run: |-
+        brew update
+        brew install openblas protobuf ccache
+        brew install coreutils findutils
+    - name: Setup path with gnu
+      run: |-
+        echo "/usr/local/opt/coreutils/libexec/gnubin" >> $GITHUB_PATH
+        echo "/usr/local/opt/findutils/libexec/gnubin" >> $GITHUB_PATH
+    - name: Setup BLAS
+      run: |-
+        echo "LDFLAGS=-L/usr/local/opt/openblas/lib" >> $GITHUB_ENV
+        echo "CPPFLAGS=-I/usr/local/opt/openblas/include" >> $GITHUB_ENV
+    - name: Generate ccache_vars for ccache based on machine
+      shell: bash
+      id: ccache_vars
+      run: |-
+        echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
+        echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
+    - name: Cache-op for build-cache through ccache
+      uses: actions/cache@v2
+      with:
+        path: ${{ env.ccache_dir }}
+        key: ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+        restore-keys: |-
+          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
+          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}
+          ccache-${{ matrix.identifier }}
+    - name: ccache environment setup
+      run: |-
+        echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
+        echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
+        echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
+        echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
+        echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
+        echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
+    - name: ccache prolog
+      run: |-
+        ccache -s # Print current cache stats
+        ccache -z # Zero cache entry
+    - name: cmake
+      run: |-
+        mkdir -p build
+        cd build
+        cmake -L .. ${{ matrix.cmake }} ${{ env.ccache_cmake }}
+    - name: Build from source
+      working-directory: build
+      run: make -j2
+    - name: ccache epilog
+      run: 'ccache -s # Print current cache stats'
+    - name: Print Versions
+      working-directory: build
+      run: ./app/bergamot --version
+    - name: Run unit tests
+      working-directory: build
+      run: make test
+      if: ${{ matrix.unittests == 'true' }}
+    - name: Install regression-test framework (BRT)
+      working-directory: bergamot-translator-tests
+      run: make install
+    - name: Run regression-tests (BRT)
+      working-directory: bergamot-translator-tests
+      id: brt_run
+      run: MARIAN=../build ./run_brt.sh ${{ matrix.brt_tags }}
+    - name: Print logs of unsuccessful BRTs
+      working-directory: bergamot-translator-tests
+      run: |-
+        grep "tests.*.sh" previous.log \
+          | sed 's/^\s*-\s*//' \
+          | xargs -I% bash -c 'echo %; tail -n20 %.log'
+      if: ${{ always() && steps.brt_run.outcome == 'failure' }}
+    - name: Upload regression-tests artifacts
+      uses: actions/upload-artifact@v2
+      if: ${{ always() && steps.brt_run.outcome != 'skipped' }}
+      with:
+        name: brt-${{ matrix.identifier }}
+        path: |-
+          bergamot-translator-tests/**/*.expected
+          bergamot-translator-tests/**/*.log
+          bergamot-translator-tests/**/*.out
+

From d39e0277c6d19e86401ed599000b08db9c721a95 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Sat, 5 Jun 2021 00:28:53 +0100
Subject: [PATCH 266/442] Replace resize with possible negative range with
 pop_back() (#189)

---
 src/translator/annotation.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/translator/annotation.cpp b/src/translator/annotation.cpp
index 94ceb54d7..e05a6a77d 100644
--- a/src/translator/annotation.cpp
+++ b/src/translator/annotation.cpp
@@ -47,7 +47,7 @@ void AnnotatedText::recordExistingSentence(std::vector<string_view>::iterator be
   assert(!annotation.token_begin_.empty());
   assert(annotation.token_begin_.back() == text.size());
   // Clip off size token ending.
-  annotation.token_begin_.resize(annotation.token_begin_.size() - 1);
+  annotation.token_begin_.pop_back();
   for (std::vector<string_view>::iterator i = begin; i != end; ++i) {
     assert(i->data() >= text.data());                                  // In range.
     assert(i->data() + i->size() <= text.data() + text.size());        // In range

From 3e46e3391c1ef767752af0977cbcc9c6ee9757f5 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 8 Jun 2021 15:36:22 +0200
Subject: [PATCH 267/442] Consistent EMSDK version and parallel make jobs in
 README and github actions

 - Set EMSDK version to 2.0.9 to make it consistent
   everywhere in repo
 - Set parallel make jobs to 2
---
 .github/workflows/wasm-custom_marian-mac.yml    |  4 +++-
 .github/workflows/wasm-custom_marian-ubuntu.yml |  4 +++-
 README.md                                       |  8 ++++----
 build-wasm.sh                                   | 14 +++++---------
 4 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/.github/workflows/wasm-custom_marian-mac.yml b/.github/workflows/wasm-custom_marian-mac.yml
index c275f3cd8..746fb9cdd 100644
--- a/.github/workflows/wasm-custom_marian-mac.yml
+++ b/.github/workflows/wasm-custom_marian-mac.yml
@@ -13,7 +13,9 @@ jobs:
 
     steps:
       - name: Setup Emscripten toolchain
-        uses: mymindstorm/setup-emsdk@v8
+        uses: mymindstorm/setup-emsdk@v9
+        with:
+          version: 2.0.9
 
       - name: Verify Emscripten setup
         run: emcc -v
diff --git a/.github/workflows/wasm-custom_marian-ubuntu.yml b/.github/workflows/wasm-custom_marian-ubuntu.yml
index 44835465a..dcea92850 100644
--- a/.github/workflows/wasm-custom_marian-ubuntu.yml
+++ b/.github/workflows/wasm-custom_marian-ubuntu.yml
@@ -13,7 +13,9 @@ jobs:
 
     steps:
       - name: Setup Emscripten toolchain
-        uses: mymindstorm/setup-emsdk@v8
+        uses: mymindstorm/setup-emsdk@v9
+        with:
+          version: 2.0.9
 
       - name: Verify Emscripten setup
         run: emcc -v
diff --git a/README.md b/README.md
index 6564f67fa..0fbf58efd 100644
--- a/README.md
+++ b/README.md
@@ -13,7 +13,7 @@ Create a folder where you want to build all the artifacts (`build-native` in thi
 mkdir build-native
 cd build-native
 cmake ../
-make -j3
+make -j2
 ```
 
 ### Build WASM
@@ -23,8 +23,8 @@ Building on wasm requires Emscripten toolchain. It can be downloaded and install
 
 * Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
 * Enter the cloned directory: `cd emsdk`
-* Install the lastest sdk tools: `./emsdk install latest`
-* Activate the latest sdk tools: `./emsdk activate latest`
+* Install the lastest sdk tools: `./emsdk install 2.0.9`
+* Activate the latest sdk tools: `./emsdk activate 2.0.9`
 * Activate path variables: `source ./emsdk_env.sh`
 
 #### <a name="Compile"></a> Compile
@@ -34,7 +34,7 @@ Building on wasm requires Emscripten toolchain. It can be downloaded and install
     mkdir build-wasm
     cd build-wasm
     emcmake cmake -DCOMPILE_WASM=on ../
-    emmake make -j3
+    emmake make -j2
     ```
 
     The wasm artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
diff --git a/build-wasm.sh b/build-wasm.sh
index 6a48f5bad..d3cd9d1db 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -9,10 +9,8 @@ set -x
 cd "$(dirname $0)"
 
 # This file replicates the instructions found in ./README.md under "Build WASM"
-# with slight adjustments to be able to run the build script multiple times without having to clone all dependencies
-# as per "As long as you don't update any submodule, just follow steps in `4.ii` to recompile."
 
-# 1. Download and Install Emscripten using following instructions (unless the EMSDK env var is already set)
+# Prerequisite: Download and Install Emscripten using following instructions (unless the EMSDK env var is already set)
 if [ "$EMSDK" == "" ]; then
   EMSDK_UPDATE_REQUIRED=0
   if [ ! -d "emsdk" ]; then
@@ -37,18 +35,16 @@ if [ "$EMSDK" == "" ]; then
   source ./emsdk/emsdk_env.sh
 fi
 
-# 4. Compile
-#     1. Create a folder where you want to build all the artifacts (`build-wasm` in this case)
+# Compile
+#    1. Create a folder where you want to build all the artifacts (`build-wasm` in this case) and compile
 if [ ! -d "build-wasm" ]; then
   mkdir build-wasm
 fi
 cd build-wasm
-
-#     2. Compile the artifacts
 emcmake cmake -DCOMPILE_WASM=on ../
-emmake make -j3
+emmake make -j2
 
-#     3. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
+#     2. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
 bash ../wasm/patch-artifacts-enable-wormhole.sh
 
 # The artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).

From dc2fb3d64e91414c2ed7ab24adfd7e230a7c4514 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 9 Jun 2021 10:12:00 +0100
Subject: [PATCH 268/442] CMake fixes: Generate project.h in binary dir, fix
 GetVersionFromFile for use as submodule. (#193)

* Use CMAKE_CURRENT_SOURCE_DIR instead of CMAKE_SOURCE_DIR for project bound version string

* marian-dev cmake fix

* Generate project.h in binary dir

* We don't want people asking about extra spaces
---
 3rd_party/marian-dev           | 2 +-
 cmake/GetVersionFromFile.cmake | 2 +-
 src/translator/CMakeLists.txt  | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index ad92307e0..6087379f2 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit ad92307e0f4ab66f75b03ddc87c48d28d07757f1
+Subproject commit 6087379f2ee7fb3062a82a6129ff81ca5fe56eed
diff --git a/cmake/GetVersionFromFile.cmake b/cmake/GetVersionFromFile.cmake
index 2eadb427e..47c35bc23 100644
--- a/cmake/GetVersionFromFile.cmake
+++ b/cmake/GetVersionFromFile.cmake
@@ -23,7 +23,7 @@ endif()
 
 # Get current commit SHA from git
 execute_process(COMMAND ${GIT_EXECUTABLE} rev-parse --short HEAD
-  WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
+  WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
   OUTPUT_VARIABLE PROJECT_VERSION_GIT_SHA
   OUTPUT_STRIP_TRAILING_WHITESPACE)
 
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index e007eadea..cea523e56 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -1,6 +1,6 @@
 # Generate version file
 configure_file(${CMAKE_CURRENT_SOURCE_DIR}/project_version.h.in
-               ${CMAKE_CURRENT_SOURCE_DIR}/project_version.h @ONLY)
+               ${CMAKE_CURRENT_BINARY_DIR}/project_version.h @ONLY)
 
 add_library(bergamot-translator STATIC
     byte_array_util.cpp

From 3039dea34bbc86e4b91178415054a27e3363718a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Wed, 9 Jun 2021 10:21:23 +0100
Subject: [PATCH 269/442] Fixing if syntax with YAML var subsitution (#188)

---
 .github/workflows/push-browsermt-main-to-mozilla-main.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/push-browsermt-main-to-mozilla-main.yml b/.github/workflows/push-browsermt-main-to-mozilla-main.yml
index bc0caf619..cf02fe3e9 100644
--- a/.github/workflows/push-browsermt-main-to-mozilla-main.yml
+++ b/.github/workflows/push-browsermt-main-to-mozilla-main.yml
@@ -19,7 +19,7 @@ jobs:
       - uses: actions/checkout@v2
 
       - name: Mirror a branch from a remote repo into this one
-        if: github.repository == '${{ env.this_repository }}'
+        if: ${{ github.repository == env.this_repository }}
         run: |
           git clone -b ${{ env.branch }} https://github.com/${{ env.source_repository }}.git source
           cd source

From 16eb47f47e5ebf241e6ecf552334227b7ab49851 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 9 Jun 2021 14:57:23 +0200
Subject: [PATCH 270/442] Generating cmake configured project version (.js)
 file in build folder (#194)

- Earlier this file was being generated in folder containing
   actual sources

 - Fixes https://github.com/browsermt/bergamot-translator/issues/161
---
 wasm/CMakeLists.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 72b22c169..602dc5d33 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -6,7 +6,7 @@ add_executable(bergamot-translator-worker
 
 # Generate version file that can be included in the wasm artifacts
 configure_file(${CMAKE_CURRENT_SOURCE_DIR}/project_version.js.in
-               ${CMAKE_CURRENT_SOURCE_DIR}/project_version.js @ONLY)
+               ${CMAKE_CURRENT_BINARY_DIR}/project_version.js @ONLY)
 
 # This header inclusion needs to go away later as path to public headers of bergamot
 # translator should be directly available from "bergamot-translator" target
@@ -24,7 +24,7 @@ set(LINKER_FLAGS "-g2 --bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s
 set(LINKER_FLAGS "${LINKER_FLAGS} -s ENVIRONMENT=web,worker")
 
 # Append version information in the Javascript artifact
-set(LINKER_FLAGS "${LINKER_FLAGS} --extern-pre-js ${CMAKE_CURRENT_SOURCE_DIR}/project_version.js")
+set(LINKER_FLAGS "${LINKER_FLAGS} --extern-pre-js ${CMAKE_CURRENT_BINARY_DIR}/project_version.js")
 
 set_target_properties(bergamot-translator-worker PROPERTIES
                         SUFFIX ".js"

From e9e5ac6782a57b9ac8eb98afd9c988870fb4c798 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 14 Jun 2021 15:02:42 +0100
Subject: [PATCH 271/442] Partial test-apps and tolerance in evaluations (#184)

* Partial test applications

Previously service-cli was used to generate output and accomplish
regression testing for all of: (1) translated-text (2) alignment tokens
+ scores (3) quality scores (4) indirectly annotation and tokenizations.

The --mode native now only outputs a faithful to source translated text
of the input source on stdin.

Test apps are separated into testing only individual functionalities.
This can help in independently testing ssplit-cpp, quality-scores for
the quality estimation implementation etc.

Separating numbers and text have the advantage of being able to compare
one with tolerance using BLEU (text) and some allowed error-rates
(numbers).

* Removing #mac tag

* Moving test apps to src/tests

* Tests are always on for CI

Unit tests are turned off looking for WASM_COMPATIBLE_SOURCES.

* Fixing WASM_COMPATIBLE_SOURCE -> USE_WASM_COMPATIBLE_SOURCE

* Workaround for now; CMakeLists.txt horrors are starting to bite

* BRT: use bergamot-test instead of bergamot now

* This should fix issues: CMakeLists.txt has so many paths

* Casing to camelCase and removing legacyServiceCli

* removing leftover service-cli declaration, some doc updates

* #pragma once is starting to look easier

* All the more reasons to do #pragma once

* Updating marian-dev with intgemm::kCPU print, resolved from INTGEMM_CPUID

* BRT: Use --gemm-highest-arch instead of python script

* Adding intgemm resolve here, where always(?) have intgemm on?

* intgemm-resolve in default binary directory

* BRT: Update to use intgemm-resolve

* marian-dev: Reset to without --gemm-highest-precision

Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
---
 .github/workflows/native.yml               |   8 +-
 3rd_party/CMakeLists.txt                   |   2 +
 CMakeLists.txt                             |   3 +
 app/cli.h                                  |  51 +--------
 bergamot-translator-tests                  |   2 +-
 src/CMakeLists.txt                         |   5 +-
 src/tests/CMakeLists.txt                   |  38 ++++---
 src/tests/apps.cpp                         | 116 +++++++++++++++++++++
 src/tests/apps.h                           |  49 +++++++++
 src/tests/cli.cpp                          |  27 +++++
 src/tests/intgemm_resolve.cpp              |   8 ++
 src/tests/units/CMakeLists.txt             |  22 ++++
 src/tests/{ => units}/annotation_tests.cpp |   0
 src/tests/{ => units}/run_tests.cpp        |   0
 14 files changed, 261 insertions(+), 70 deletions(-)
 create mode 100644 src/tests/apps.cpp
 create mode 100644 src/tests/apps.h
 create mode 100644 src/tests/cli.cpp
 create mode 100644 src/tests/intgemm_resolve.cpp
 create mode 100644 src/tests/units/CMakeLists.txt
 rename src/tests/{ => units}/annotation_tests.cpp (100%)
 rename src/tests/{ => units}/run_tests.cpp (100%)

diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
index eb3f4d47e..c572cb5d8 100644
--- a/.github/workflows/native.yml
+++ b/.github/workflows/native.yml
@@ -30,7 +30,7 @@ jobs:
         - name: Ubuntu 18.04 minimal
           os: ubuntu-18.04
           identifier: ubuntu_1804_minimal
-          cmake: -DCOMPILE_TESTS=off -DUSE_WASM_COMPATIBLE_SOURCE=on
+          cmake: -DCOMPILE_TESTS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
           brt_tags: "'#wasm'"
           unittests: 'false'
         - name: Ubuntu 20.04 full
@@ -42,7 +42,7 @@ jobs:
         - name: Ubuntu 20.04 minimal
           os: ubuntu-20.04
           identifier: ubuntu_2004_minimal
-          cmake: -DCOMPILE_TESTS=off -DUSE_WASM_COMPATIBLE_SOURCE=on
+          cmake: -DCOMPILE_TESTS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
           brt_tags: "'#wasm'"
           unittests: 'false'
     name: ${{ matrix.name }}
@@ -140,12 +140,12 @@ jobs:
           os: macos-10.15
           identifier: mac_1015_full
           cmake: -DCOMPILE_TESTS=on -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=off
-          brt_tags: "'#mac'"
+          brt_tags: ""
           unittests: 'true'
         - name: MacOS 10.15 minimal
           os: macos-10.15
           identifier: mac_1015_minimal
-          cmake: -DCOMPILE_TESTS=off -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
+          cmake: -DCOMPILE_TESTS=on -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
           brt_tags: "'#wasm'"
           unittests: 'false'
     name: ${{ matrix.name }}
diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index 74ce906dd..70e50d663 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -1,3 +1,5 @@
+# marian-dev is tested elsewhere in both paths, turning off here.
+set(COMPILE_TESTS OFF)
 add_subdirectory(marian-dev)
 
 if(COMPILE_WASM)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index e561ed938..c58ddd4ff 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -41,6 +41,9 @@ include(CMakeDependentOption)
 # Project specific cmake options
 option(COMPILE_WASM "Compile for WASM" OFF)
 cmake_dependent_option(USE_WASM_COMPATIBLE_SOURCE "Use wasm compatible sources" OFF "NOT COMPILE_WASM" ON)
+
+# WASM disables a million libraries, which also includes the unit test-library.
+cmake_dependent_option(COMPILE_UNIT_TESTS "Compile unit tests" OFF "USE_WASM_COMPATIBLE_SOURCE" ON)
 option(COMPILE_TESTS "Compile bergamot-tests" OFF)
 
 # Set 3rd party submodule specific cmake options for this project
diff --git a/app/cli.h b/app/cli.h
index 292d21c2e..d6e930fed 100644
--- a/app/cli.h
+++ b/app/cli.h
@@ -1,5 +1,6 @@
 #ifndef BERGAMOT_APP_CLI_H
 #define BERGAMOT_APP_CLI_H
+#include <algorithm>
 #include <cstdlib>
 #include <future>
 #include <iostream>
@@ -103,8 +104,7 @@ void decoder(Ptr<Options> options) {
 /// [brt/tests/basic/test_service-cli_intgemm_8bit.cpu-threads.4.sh](https://github.com/browsermt/bergamot-translator-tests/blob/main/tests/basic/test_service-cli_intgemm_8bit.cpu-threads.4.sh)
 ///
 /// * Input: reads from stdin, blob of text, read as a whole ; sentence-splitting etc handled internally.
-/// * Output: to stdout, translation of the source text and additional information like sentences, alignments between
-/// source and target tokens and quality scores.
+/// * Output: to stdout, translation of the source text faithful to source structure.
 ///
 /// @param [in] options: options to build translator
 void native(Ptr<Options> options) {
@@ -124,58 +124,13 @@ void native(Ptr<Options> options) {
   std::string input = std_input.str();
 
   ResponseOptions responseOptions;
-  responseOptions.qualityScores = true;
-  responseOptions.alignment = true;
-  responseOptions.alignmentThreshold = 0.2f;
 
   // Wait on future until Response is complete
   std::future<Response> responseFuture = service.translate(std::move(input), responseOptions);
   responseFuture.wait();
   Response response = responseFuture.get();
 
-  std::cout << "[original]: " << response.source.text << '\n';
-  std::cout << "[translated]: " << response.target.text << '\n';
-  for (int sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
-    std::cout << " [src Sentence]: " << response.source.sentence(sentenceIdx) << '\n';
-    std::cout << " [tgt Sentence]: " << response.target.sentence(sentenceIdx) << '\n';
-    std::cout << "Alignments" << '\n';
-    typedef std::pair<size_t, float> Point;
-
-    // Initialize a point vector.
-    std::vector<std::vector<Point>> aggregate(response.source.numWords(sentenceIdx));
-
-    // Handle alignments
-    auto &alignments = response.alignments[sentenceIdx];
-    for (auto &p : alignments) {
-      aggregate[p.src].emplace_back(p.tgt, p.prob);
-    }
-
-    for (size_t src = 0; src < aggregate.size(); src++) {
-      std::cout << response.source.word(sentenceIdx, src) << ": ";
-      for (auto &p : aggregate[src]) {
-        std::cout << response.target.word(sentenceIdx, p.first) << "(" << p.second << ") ";
-      }
-      std::cout << '\n';
-    }
-
-    // Handle quality.
-    auto &quality = response.qualityScores[sentenceIdx];
-    std::cout << "Quality: whole(" << quality.sequence << "), tokens below:" << '\n';
-    size_t wordIdx = 0;
-    bool first = true;
-    for (auto &p : quality.word) {
-      if (first) {
-        first = false;
-      } else {
-        std::cout << " ";
-      }
-      std::cout << response.target.word(sentenceIdx, wordIdx) << "(" << p << ")";
-      wordIdx++;
-    }
-    std::cout << '\n';
-  }
-  std::cout << "--------------------------\n";
-  std::cout << '\n';
+  std::cout << response.target.text;
 }
 
 }  // namespace app
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 020135af1..b0ba62ead 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 020135af1b620caa27929c1403c50ec3299e5bff
+Subproject commit b0ba62eade4af7752c65c76cb17eab421ea02445
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index c2d62ef83..856831be9 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -1,7 +1,6 @@
 add_subdirectory(translator)
 
-if(COMPILE_TESTS)
-  # Catch currently comes from marian sources.
-  add_subdirectory(tests)
+if (COMPILE_TESTS)
+    add_subdirectory(tests)
 endif(COMPILE_TESTS)
 
diff --git a/src/tests/CMakeLists.txt b/src/tests/CMakeLists.txt
index 5c1bc003c..483bd075f 100644
--- a/src/tests/CMakeLists.txt
+++ b/src/tests/CMakeLists.txt
@@ -1,22 +1,32 @@
 # Unit tests
-set(UNIT_TESTS
-    annotation_tests
-)
 
-foreach(test ${UNIT_TESTS})
-  add_executable("run_${test}" run_tests.cpp "${test}.cpp")
-  target_include_directories("run_${test}" PRIVATE ${CATCH_INCLUDE_DIR} "${CMAKE_SOURCE_DIR}/src")
+# Include Catch explicitly from marian.
+set(CATCH_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/3rd_party/marian-dev/3rd-party)
+add_library(Catch INTERFACE)
+target_include_directories(Catch INTERFACE ${CATCH_INCLUDE_DIR})
 
+if (COMPILE_UNIT_TESTS)
+    add_subdirectory(units)
+endif (COMPILE_UNIT_TESTS)
+
+
+
+if(NOT MSVC)
+  # Testing apps
+  set(APP_TESTS)
+  add_executable("bergamot-test" "cli.cpp" "apps.cpp")
+  
   if(CUDA_FOUND)
-    target_link_libraries("run_${test}" ${EXT_LIBS} marian ${EXT_LIBS} marian_cuda ${EXT_LIBS} Catch bergamot-translator)
+    target_link_libraries("bergamot-test" bergamot-translator)
   else(CUDA_FOUND)
-    target_link_libraries("run_${test}" marian ${EXT_LIBS} Catch bergamot-translator)
+    target_link_libraries("bergamot-test" bergamot-translator)
   endif(CUDA_FOUND)
+  
+  set_target_properties("bergamot-test" PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}")
 
-  if(msvc)
-    # disable c4305: truncation from 'double' to '_ty'
-    target_compile_options("run_${test}" public /wd4305)
-  endif(msvc)
+  # Adding an intgemm_resolve cmdline
+  add_executable(intgemm-resolve intgemm_resolve.cpp)
+  target_link_libraries(intgemm-resolve PRIVATE bergamot-translator)
+  set_target_properties(intgemm-resolve PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}")
+endif(NOT MSVC)
 
-  add_test(NAME ${test} COMMAND "run_${test}")
-endforeach(test)
diff --git a/src/tests/apps.cpp b/src/tests/apps.cpp
new file mode 100644
index 000000000..9c00bffab
--- /dev/null
+++ b/src/tests/apps.cpp
@@ -0,0 +1,116 @@
+#include "apps.h"
+
+namespace marian {
+namespace bergamot {
+namespace testapp {
+
+// Utility function, common for all testapps.
+Response translateFromStdin(Ptr<Options> options, ResponseOptions responseOptions) {
+  // Prepare memories for bytearrays (including model, shortlist and vocabs)
+  MemoryBundle memoryBundle;
+
+  if (options->get<bool>("bytearray")) {
+    // Load legit values into bytearrays.
+    memoryBundle = getMemoryBundleFromConfig(options);
+  }
+
+  Service service(options, std::move(memoryBundle));
+
+  // Read a large input text blob from stdin
+  std::ostringstream inputStream;
+  inputStream << std::cin.rdbuf();
+  std::string input = inputStream.str();
+
+  // Wait on future until Response is complete
+  std::future<Response> responseFuture = service.translate(std::move(input), responseOptions);
+  responseFuture.wait();
+  Response response = responseFuture.get();
+  return response;
+}
+
+void qualityScores(Ptr<Options> options) {
+  ResponseOptions responseOptions;
+  responseOptions.qualityScores = true;
+
+  Response response = translateFromStdin(options, responseOptions);
+  for (int sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
+    auto &quality = response.qualityScores[sentenceIdx];
+    std::cout << ((sentenceIdx == 0) ? "" : "\n") << quality.sequence << '\n';
+    for (int wordIdx = 0; wordIdx < quality.word.size(); wordIdx++) {
+      std::cout << ((wordIdx == 0) ? "" : " ");
+      std::cout << quality.word[wordIdx];
+    }
+    std::cout << '\n';
+  }
+}
+
+void alignmentAggregatedToSource(Ptr<Options> options, bool numeric) {
+  ResponseOptions responseOptions;
+  responseOptions.alignment = true;
+  responseOptions.alignmentThreshold = 0.2f;
+  Response response = translateFromStdin(options, responseOptions);
+
+  for (size_t sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
+    std::cout << (sentenceIdx == 0 ? "" : "\n");
+
+    // We are aggregating at source, which does not depend on matrix-multiplications and printing only target so we can
+    // do BLEU based stuff on the text.
+    //
+    typedef std::pair<size_t, float> Point;
+
+    std::vector<std::vector<Point>> aggregate(response.source.numWords(sentenceIdx));
+    auto &alignments = response.alignments[sentenceIdx];
+    for (auto &p : alignments) {
+      aggregate[p.src].emplace_back(p.tgt, p.prob);
+    }
+
+    for (size_t sourceIdx = 0; sourceIdx < aggregate.size(); sourceIdx++) {
+      // Sort in order of target tokens.
+      auto cmp = [](const Point &p, const Point &q) { return p.first < q.first; };
+      std::sort(aggregate[sourceIdx].begin(), aggregate[sourceIdx].end(), cmp);
+
+      if (!numeric) {
+        std::cout << response.source.word(sentenceIdx, sourceIdx) << ": ";
+      }
+
+      for (size_t j = 0; j < aggregate[sourceIdx].size(); j++) {
+        if (numeric) {
+          float alignmentScore = aggregate[sourceIdx][j].second;
+          std::cout << (j == 0 ? "" : " ");
+          std::cout << alignmentScore;
+        } else {
+          std::cout << " ";
+          size_t targetIdx = aggregate[sourceIdx][j].first;
+          std::cout << response.target.word(sentenceIdx, targetIdx);
+        }
+      }
+      std::cout << '\n';
+    }
+  }
+}
+
+void annotatedTextWords(Ptr<Options> options, bool source) {
+  ResponseOptions responseOptions;
+  Response response = translateFromStdin(options, responseOptions);
+  AnnotatedText &annotatedText = source ? response.source : response.target;
+  for (size_t s = 0; s < annotatedText.numSentences(); s++) {
+    for (size_t w = 0; w < annotatedText.numWords(s); w++) {
+      std::cout << (w == 0 ? "" : "\t");
+      std::cout << annotatedText.word(s, w);
+    }
+    std::cout << "\n";
+  }
+}
+
+void annotatedTextSentences(Ptr<Options> options, bool source) {
+  ResponseOptions responseOptions;
+  Response response = translateFromStdin(options, responseOptions);
+  AnnotatedText &annotatedText = source ? response.source : response.target;
+  for (size_t s = 0; s < annotatedText.numSentences(); s++) {
+    std::cout << annotatedText.sentence(s) << "\n";
+  }
+}
+
+}  // namespace testapp
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/tests/apps.h b/src/tests/apps.h
new file mode 100644
index 000000000..2ccf2c483
--- /dev/null
+++ b/src/tests/apps.h
@@ -0,0 +1,49 @@
+#ifndef BERGAMOT_SRC_TESTS_APPS_H
+#define BERGAMOT_SRC_TESTS_APPS_H
+#include <algorithm>
+#include <cstdlib>
+#include <future>
+#include <iostream>
+#include <sstream>
+
+#include "common/definitions.h"
+#include "common/timer.h"
+#include "common/utils.h"
+#include "marian.h"
+#include "translator/byte_array_util.h"
+#include "translator/parser.h"
+#include "translator/response.h"
+#include "translator/response_options.h"
+#include "translator/service.h"
+
+namespace marian {
+namespace bergamot {
+
+namespace testapp {
+
+// Utility function, common for all testapps. Reads content from stdin, builds a Service based on options and constructs
+// a response containing translation data according responseOptions.
+Response translateFromStdin(Ptr<Options> options, ResponseOptions responseOptions);
+
+// Reads from stdin and translates. The quality score for the translations (each sentence) are printed separated by
+// empty-lines. The first line contains whole quality scores and the second line word quality scores, for each entry.
+void qualityScores(Ptr<Options> options);
+
+// Reads from stdin and translates. Alignments are printed aligned to the source-tokens, following format src-token:
+// [possible-target-alignments], if numeric is false. If numeric is true, only alignment probabilities are printed
+// instead of the tokens.
+void alignmentAggregatedToSource(Ptr<Options> options, bool numeric = false);
+
+// Reads from stdin and translates.  Prints the tokens separated by space for each sentence. Prints words from source
+// side text annotation if source=true, target annotation otherwise.
+void annotatedTextWords(Ptr<Options> options, bool source = true);
+
+// Reads from stdin and translates the read content. Prints the sentences in source or target in constructed response
+// in each line, depending on source = true or false respectively.
+void annotatedTextSentences(Ptr<Options> options, bool source = true);
+
+}  // namespace testapp
+}  // namespace bergamot
+}  // namespace marian
+
+#endif  // BERGAMOT_SRC_TESTS_APPS_H
diff --git a/src/tests/cli.cpp b/src/tests/cli.cpp
new file mode 100644
index 000000000..f2f0218c2
--- /dev/null
+++ b/src/tests/cli.cpp
@@ -0,0 +1,27 @@
+
+#include "apps.h"
+
+int main(int argc, char *argv[]) {
+  auto cp = marian::bergamot::createConfigParser();
+  auto options = cp.parseOptions(argc, argv, true);
+  const std::string mode = options->get<std::string>("bergamot-mode");
+  using namespace marian::bergamot;
+  if (mode == "test-quality-scores") {
+    testapp::qualityScores(options);
+  } else if (mode == "test-alignment-scores") {
+    testapp::alignmentAggregatedToSource(options, /*numeric=*/true);
+  } else if (mode == "test-alignment-words") {
+    testapp::alignmentAggregatedToSource(options, /*numeric=*/false);
+  } else if (mode == "test-response-source-sentences") {
+    testapp::annotatedTextSentences(options, /*source=*/true);
+  } else if (mode == "test-response-target-sentences") {
+    testapp::annotatedTextSentences(options, /*source=*/false);
+  } else if (mode == "test-response-source-words") {
+    testapp::annotatedTextWords(options, /*source=*/true);
+  } else if (mode == "test-response-target-words") {
+    testapp::annotatedTextWords(options, /*source=*/false);
+  } else {
+    ABORT("Unknown --mode {}. Please run a valid test", mode);
+  }
+  return 0;
+}
diff --git a/src/tests/intgemm_resolve.cpp b/src/tests/intgemm_resolve.cpp
new file mode 100644
index 000000000..f95d0c449
--- /dev/null
+++ b/src/tests/intgemm_resolve.cpp
@@ -0,0 +1,8 @@
+#include <iostream>
+
+#include "intgemm/intgemm.h"
+
+int main() {
+  std::cout << static_cast<int>(intgemm::kCPU) << "\n";
+  return 0;
+}
diff --git a/src/tests/units/CMakeLists.txt b/src/tests/units/CMakeLists.txt
new file mode 100644
index 000000000..5c1bc003c
--- /dev/null
+++ b/src/tests/units/CMakeLists.txt
@@ -0,0 +1,22 @@
+# Unit tests
+set(UNIT_TESTS
+    annotation_tests
+)
+
+foreach(test ${UNIT_TESTS})
+  add_executable("run_${test}" run_tests.cpp "${test}.cpp")
+  target_include_directories("run_${test}" PRIVATE ${CATCH_INCLUDE_DIR} "${CMAKE_SOURCE_DIR}/src")
+
+  if(CUDA_FOUND)
+    target_link_libraries("run_${test}" ${EXT_LIBS} marian ${EXT_LIBS} marian_cuda ${EXT_LIBS} Catch bergamot-translator)
+  else(CUDA_FOUND)
+    target_link_libraries("run_${test}" marian ${EXT_LIBS} Catch bergamot-translator)
+  endif(CUDA_FOUND)
+
+  if(msvc)
+    # disable c4305: truncation from 'double' to '_ty'
+    target_compile_options("run_${test}" public /wd4305)
+  endif(msvc)
+
+  add_test(NAME ${test} COMMAND "run_${test}")
+endforeach(test)
diff --git a/src/tests/annotation_tests.cpp b/src/tests/units/annotation_tests.cpp
similarity index 100%
rename from src/tests/annotation_tests.cpp
rename to src/tests/units/annotation_tests.cpp
diff --git a/src/tests/run_tests.cpp b/src/tests/units/run_tests.cpp
similarity index 100%
rename from src/tests/run_tests.cpp
rename to src/tests/units/run_tests.cpp

From 4b014665ba54eef8f8ab68a1929d2ede9a012902 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 14 Jun 2021 18:40:41 +0100
Subject: [PATCH 272/442] Removing alignments and quality-scores test-code
 (#196)

* Removing alignments and quality-scores test-code
* BRT: Update to main
---
 bergamot-translator-tests |  2 +-
 src/tests/apps.cpp        | 61 ---------------------------------------
 src/tests/apps.h          |  9 ------
 src/tests/cli.cpp         |  8 +----
 4 files changed, 2 insertions(+), 78 deletions(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index b0ba62ead..e1ae3b58a 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit b0ba62eade4af7752c65c76cb17eab421ea02445
+Subproject commit e1ae3b58a6e6c25856a5f279fedb567bbe695c49
diff --git a/src/tests/apps.cpp b/src/tests/apps.cpp
index 9c00bffab..c57f5f571 100644
--- a/src/tests/apps.cpp
+++ b/src/tests/apps.cpp
@@ -28,67 +28,6 @@ Response translateFromStdin(Ptr<Options> options, ResponseOptions responseOption
   return response;
 }
 
-void qualityScores(Ptr<Options> options) {
-  ResponseOptions responseOptions;
-  responseOptions.qualityScores = true;
-
-  Response response = translateFromStdin(options, responseOptions);
-  for (int sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
-    auto &quality = response.qualityScores[sentenceIdx];
-    std::cout << ((sentenceIdx == 0) ? "" : "\n") << quality.sequence << '\n';
-    for (int wordIdx = 0; wordIdx < quality.word.size(); wordIdx++) {
-      std::cout << ((wordIdx == 0) ? "" : " ");
-      std::cout << quality.word[wordIdx];
-    }
-    std::cout << '\n';
-  }
-}
-
-void alignmentAggregatedToSource(Ptr<Options> options, bool numeric) {
-  ResponseOptions responseOptions;
-  responseOptions.alignment = true;
-  responseOptions.alignmentThreshold = 0.2f;
-  Response response = translateFromStdin(options, responseOptions);
-
-  for (size_t sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
-    std::cout << (sentenceIdx == 0 ? "" : "\n");
-
-    // We are aggregating at source, which does not depend on matrix-multiplications and printing only target so we can
-    // do BLEU based stuff on the text.
-    //
-    typedef std::pair<size_t, float> Point;
-
-    std::vector<std::vector<Point>> aggregate(response.source.numWords(sentenceIdx));
-    auto &alignments = response.alignments[sentenceIdx];
-    for (auto &p : alignments) {
-      aggregate[p.src].emplace_back(p.tgt, p.prob);
-    }
-
-    for (size_t sourceIdx = 0; sourceIdx < aggregate.size(); sourceIdx++) {
-      // Sort in order of target tokens.
-      auto cmp = [](const Point &p, const Point &q) { return p.first < q.first; };
-      std::sort(aggregate[sourceIdx].begin(), aggregate[sourceIdx].end(), cmp);
-
-      if (!numeric) {
-        std::cout << response.source.word(sentenceIdx, sourceIdx) << ": ";
-      }
-
-      for (size_t j = 0; j < aggregate[sourceIdx].size(); j++) {
-        if (numeric) {
-          float alignmentScore = aggregate[sourceIdx][j].second;
-          std::cout << (j == 0 ? "" : " ");
-          std::cout << alignmentScore;
-        } else {
-          std::cout << " ";
-          size_t targetIdx = aggregate[sourceIdx][j].first;
-          std::cout << response.target.word(sentenceIdx, targetIdx);
-        }
-      }
-      std::cout << '\n';
-    }
-  }
-}
-
 void annotatedTextWords(Ptr<Options> options, bool source) {
   ResponseOptions responseOptions;
   Response response = translateFromStdin(options, responseOptions);
diff --git a/src/tests/apps.h b/src/tests/apps.h
index 2ccf2c483..b380b5782 100644
--- a/src/tests/apps.h
+++ b/src/tests/apps.h
@@ -25,15 +25,6 @@ namespace testapp {
 // a response containing translation data according responseOptions.
 Response translateFromStdin(Ptr<Options> options, ResponseOptions responseOptions);
 
-// Reads from stdin and translates. The quality score for the translations (each sentence) are printed separated by
-// empty-lines. The first line contains whole quality scores and the second line word quality scores, for each entry.
-void qualityScores(Ptr<Options> options);
-
-// Reads from stdin and translates. Alignments are printed aligned to the source-tokens, following format src-token:
-// [possible-target-alignments], if numeric is false. If numeric is true, only alignment probabilities are printed
-// instead of the tokens.
-void alignmentAggregatedToSource(Ptr<Options> options, bool numeric = false);
-
 // Reads from stdin and translates.  Prints the tokens separated by space for each sentence. Prints words from source
 // side text annotation if source=true, target annotation otherwise.
 void annotatedTextWords(Ptr<Options> options, bool source = true);
diff --git a/src/tests/cli.cpp b/src/tests/cli.cpp
index f2f0218c2..4ecb24e02 100644
--- a/src/tests/cli.cpp
+++ b/src/tests/cli.cpp
@@ -6,13 +6,7 @@ int main(int argc, char *argv[]) {
   auto options = cp.parseOptions(argc, argv, true);
   const std::string mode = options->get<std::string>("bergamot-mode");
   using namespace marian::bergamot;
-  if (mode == "test-quality-scores") {
-    testapp::qualityScores(options);
-  } else if (mode == "test-alignment-scores") {
-    testapp::alignmentAggregatedToSource(options, /*numeric=*/true);
-  } else if (mode == "test-alignment-words") {
-    testapp::alignmentAggregatedToSource(options, /*numeric=*/false);
-  } else if (mode == "test-response-source-sentences") {
+  if (mode == "test-response-source-sentences") {
     testapp::annotatedTextSentences(options, /*source=*/true);
   } else if (mode == "test-response-target-sentences") {
     testapp::annotatedTextSentences(options, /*source=*/false);

From b00116cb9485689fd43521ad3cbcbe939f650f84 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Tue, 15 Jun 2021 16:02:14 +0200
Subject: [PATCH 273/442] Refactor wasm bindings to use consistent interface
 names as in native (#195)

* Refactored wasm bindings code
 - Replaced TranslationModel, TranslationRequest and TranslationResult
    with Service, ResponseOptions and Response
 - Corresponding documentation changes
 - Names of the bindings files changed
 - Moved Vector<Response> definition in Response specific bindings
   file
---
 src/translator/service.h                      |  2 +-
 wasm/CMakeLists.txt                           |  6 ++--
 wasm/README.md                                | 18 ++++++------
 wasm/bindings/TranslationRequestBindings.cpp  | 15 ----------
 wasm/bindings/TranslationResultBindings.cpp   | 22 --------------
 wasm/bindings/response_bindings.cpp           | 24 +++++++++++++++
 wasm/bindings/response_options_bindings.cpp   | 15 ++++++++++
 ...ModelBindings.cpp => service_bindings.cpp} | 29 +++++++------------
 wasm/test_page/bergamot.html                  | 28 +++++++++---------
 9 files changed, 77 insertions(+), 82 deletions(-)
 delete mode 100644 wasm/bindings/TranslationRequestBindings.cpp
 delete mode 100644 wasm/bindings/TranslationResultBindings.cpp
 create mode 100644 wasm/bindings/response_bindings.cpp
 create mode 100644 wasm/bindings/response_options_bindings.cpp
 rename wasm/bindings/{TranslationModelBindings.cpp => service_bindings.cpp} (69%)

diff --git a/src/translator/service.h b/src/translator/service.h
index 26ea83104..6c7ea9a7d 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -105,7 +105,7 @@ class Service {
   /// recommended to work with futures and translate() API.
   ///
   /// @param [in] source: rvalue reference of the string to be translated
-  /// @param [in] translationRequest: ResponseOptions indicating whether or not
+  /// @param [in] responseOptions: ResponseOptions indicating whether or not
   /// to include some member in the Response, also specify any additional
   /// configurable parameters.
   std::vector<Response> translateMultiple(std::vector<std::string> &&source, ResponseOptions responseOptions);
diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 602dc5d33..1580defa1 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -1,7 +1,7 @@
 add_executable(bergamot-translator-worker
-    bindings/TranslationModelBindings.cpp
-    bindings/TranslationRequestBindings.cpp
-    bindings/TranslationResultBindings.cpp
+    bindings/service_bindings.cpp
+    bindings/response_options_bindings.cpp
+    bindings/response_bindings.cpp
 )
 
 # Generate version file that can be included in the wasm artifacts
diff --git a/wasm/README.md b/wasm/README.md
index 01fea18b0..728b0a364 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -63,27 +63,27 @@ var alignedShortlistMemory = constructAlignedMemoryFromBuffer(shortListBuffer, 6
 var alignedVocabsMemoryList = new Module.AlignedMemoryList;
 downloadedVocabBuffers.forEach(item => alignedVocabsMemoryList.push_back(constructAlignedMemoryFromBuffer(item, 64)));
 
-// Instantiate the TranslationModel
-const model = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
+// Instantiate the Translation Service
+const translationService = new Module.Service(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
 
-// Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
-const request = new Module.TranslationRequest();
+// Instantiate the arguments of translate() API i.e. ResponseOptions and input (vector<string>)
+const responseOptions = new Module.ResponseOptions();
 const input = new Module.VectorString;
 
 // Initialize the input
 input.push_back("Hola"); input.push_back("Mundo");
 
-// translate the input; the result is a vector<TranslationResult>
-const result = model.translate(input, request);
+// translate the input; the result is a vector<Response>
+const result = translationService.translate(input, responseOptions);
 
-// Print original and translated text from each entry of vector<TranslationResult>
+// Print original and translated text from each entry of vector<Response>
 for (let i = 0; i < result.size(); i++) {
     console.log(' original=' + result.get(i).getOriginalText() + ', translation=' + result.get(i).getTranslatedText());
 }
 
 // Don't forget to clean up the instances
-model.delete();
-request.delete();
+translationService.delete();
+responseOptions.delete();
 input.delete();
 ```
 
diff --git a/wasm/bindings/TranslationRequestBindings.cpp b/wasm/bindings/TranslationRequestBindings.cpp
deleted file mode 100644
index 42ac6c698..000000000
--- a/wasm/bindings/TranslationRequestBindings.cpp
+++ /dev/null
@@ -1,15 +0,0 @@
-/*
- * Bindings for TranslationRequest class
- *
- */
-
-#include <emscripten/bind.h>
-
-#include "response_options.h"
-
-typedef marian::bergamot::ResponseOptions TranslationRequest;
-
-using namespace emscripten;
-
-// Binding code
-EMSCRIPTEN_BINDINGS(translation_request) { class_<TranslationRequest>("TranslationRequest").constructor<>(); }
diff --git a/wasm/bindings/TranslationResultBindings.cpp b/wasm/bindings/TranslationResultBindings.cpp
deleted file mode 100644
index f02bef902..000000000
--- a/wasm/bindings/TranslationResultBindings.cpp
+++ /dev/null
@@ -1,22 +0,0 @@
-/*
- * Bindings for TranslationResult class
- *
- */
-
-#include <emscripten/bind.h>
-
-#include <vector>
-
-#include "response.h"
-
-typedef marian::bergamot::Response TranslationResult;
-
-using namespace emscripten;
-
-// Binding code
-EMSCRIPTEN_BINDINGS(translation_result) {
-  class_<TranslationResult>("TranslationResult")
-      .constructor<>()
-      .function("getOriginalText", &TranslationResult::getOriginalText)
-      .function("getTranslatedText", &TranslationResult::getTranslatedText);
-}
diff --git a/wasm/bindings/response_bindings.cpp b/wasm/bindings/response_bindings.cpp
new file mode 100644
index 000000000..561911925
--- /dev/null
+++ b/wasm/bindings/response_bindings.cpp
@@ -0,0 +1,24 @@
+/*
+ * Bindings for Response class
+ *
+ */
+
+#include <emscripten/bind.h>
+
+#include <vector>
+
+#include "response.h"
+
+typedef marian::bergamot::Response Response;
+
+using namespace emscripten;
+
+// Binding code
+EMSCRIPTEN_BINDINGS(response) {
+  class_<Response>("Response")
+      .constructor<>()
+      .function("getOriginalText", &Response::getOriginalText)
+      .function("getTranslatedText", &Response::getTranslatedText);
+
+  register_vector<Response>("VectorResponse");
+}
diff --git a/wasm/bindings/response_options_bindings.cpp b/wasm/bindings/response_options_bindings.cpp
new file mode 100644
index 000000000..e2bf8e1f5
--- /dev/null
+++ b/wasm/bindings/response_options_bindings.cpp
@@ -0,0 +1,15 @@
+/*
+ * Bindings for ResponseOptions class
+ *
+ */
+
+#include <emscripten/bind.h>
+
+#include "response_options.h"
+
+typedef marian::bergamot::ResponseOptions ResponseOptions;
+
+using namespace emscripten;
+
+// Binding code
+EMSCRIPTEN_BINDINGS(response_options) { class_<ResponseOptions>("ResponseOptions").constructor<>(); }
diff --git a/wasm/bindings/TranslationModelBindings.cpp b/wasm/bindings/service_bindings.cpp
similarity index 69%
rename from wasm/bindings/TranslationModelBindings.cpp
rename to wasm/bindings/service_bindings.cpp
index 64203a16d..416a318ad 100644
--- a/wasm/bindings/TranslationModelBindings.cpp
+++ b/wasm/bindings/service_bindings.cpp
@@ -1,18 +1,14 @@
 /*
- * TranslationModelBindings.cpp
- *
- * Bindings for TranslationModel class
+ * Bindings for Service class
  */
 
 #include <emscripten/bind.h>
 
-#include "response.h"
 #include "service.h"
 
 using namespace emscripten;
 
-typedef marian::bergamot::Service TranslationModel;
-typedef marian::bergamot::Response TranslationResult;
+typedef marian::bergamot::Service Service;
 typedef marian::bergamot::AlignedMemory AlignedMemory;
 
 val getByteArrayView(AlignedMemory& alignedMemory) {
@@ -29,7 +25,7 @@ EMSCRIPTEN_BINDINGS(aligned_memory) {
 }
 
 // When source and target vocab files are same, only one memory object is passed from JS to
-// avoid allocating memory twice for the same file. However, the constructor of the TranslationModel
+// avoid allocating memory twice for the same file. However, the constructor of the Service
 // class still expects 2 entries in this case, where each entry has the shared ownership of the
 // same AlignedMemory object. This function prepares these smart pointer based AlignedMemory objects
 // for unique AlignedMemory objects passed from JS.
@@ -56,21 +52,18 @@ marian::bergamot::MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory, A
   return memoryBundle;
 }
 
-TranslationModel* TranslationModelFactory(const std::string& config, AlignedMemory* modelMemory,
-                                          AlignedMemory* shortlistMemory,
-                                          std::vector<AlignedMemory*> uniqueVocabsMemories) {
-  return new TranslationModel(config,
-                              std::move(prepareMemoryBundle(modelMemory, shortlistMemory, uniqueVocabsMemories)));
+Service* ServiceFactory(const std::string& config, AlignedMemory* modelMemory, AlignedMemory* shortlistMemory,
+                        std::vector<AlignedMemory*> uniqueVocabsMemories) {
+  return new Service(config, std::move(prepareMemoryBundle(modelMemory, shortlistMemory, uniqueVocabsMemories)));
 }
 
-EMSCRIPTEN_BINDINGS(translation_model) {
-  class_<TranslationModel>("TranslationModel")
-      .constructor(&TranslationModelFactory, allow_raw_pointers())
-      .function("translate", &TranslationModel::translateMultiple)
-      .function("isAlignmentSupported", &TranslationModel::isAlignmentSupported);
+EMSCRIPTEN_BINDINGS(translation_service) {
+  class_<Service>("Service")
+      .constructor(&ServiceFactory, allow_raw_pointers())
+      .function("translate", &Service::translateMultiple)
+      .function("isAlignmentSupported", &Service::isAlignmentSupported);
   // ^ We redirect Service::translateMultiple to WASMBound::translate instead. Sane API is
   // translate. If and when async comes, we can be done with this inconsistency.
 
   register_vector<std::string>("VectorString");
-  register_vector<TranslationResult>("VectorTranslationResult");
 }
diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index d150af618..c69c950a3 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -80,8 +80,8 @@
     return alignedMemory;
   }
 
-  var translationModel, request, input = undefined;
-  const constructTranslationModel = async (from, to) => {
+  var translationService, responseOptions, input = undefined;
+  const constructTranslationService = async (from, to) => {
 
     const languagePair = `${from}${to}`;
 
@@ -162,10 +162,10 @@
       var alignedVocabsMemoryList = new Module.AlignedMemoryList;
       downloadedVocabBuffers.forEach(item => alignedVocabsMemoryList.push_back(constructAlignedMemoryFromBuffer(item, 64)));
 
-      // Instantiate the TranslationModel
-      if (translationModel) translationModel.delete();
-      console.debug("Creating TranslationModel with config:", modelConfig);
-      translationModel = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
+      // Instantiate the Translation Service
+      if (translationService) translationService.delete();
+      console.debug("Creating Translation Service with config:", modelConfig);
+      translationService = new Module.Service(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
     } catch (error) {
       log(error);
     }
@@ -173,8 +173,8 @@
 
   const translate = (paragraphs) => {
 
-    // Instantiate the arguments of translate() API i.e. TranslationRequest and input (vector<string>)
-    var request = new Module.TranslationRequest();
+    // Instantiate the arguments of translate() API i.e. ResponseOptions and input (vector<string>)
+    var responseOptions = new Module.ResponseOptions();
     let input = new Module.VectorString;
 
     // Initialize the input
@@ -188,14 +188,14 @@
     // Access input (just for debugging)
     console.log('Input size=', input.size());
 
-    // Translate the input; the result is a vector<TranslationResult>
-    let result = translationModel.translate(input, request);
+    // Translate the input; the result is a vector<Response>
+    let result = translationService.translate(input, responseOptions);
     const translatedParagraphs = [];
     for (let i = 0; i < result.size(); i++) {
       translatedParagraphs.push(result.get(i).getTranslatedText());
     }
     console.log({ translatedParagraphs });
-    request.delete();
+    responseOptions.delete();
     input.delete();
     return translatedParagraphs;
   }
@@ -206,10 +206,10 @@
     const from = lang.substring(0, 2);
     const to = lang.substring(2, 4);
     let start = Date.now();
-    await constructTranslationModel(from, to);
-    log(`translation model ${from}${to} construction took ${(Date.now() - start) / 1000} secs`);
+    await constructTranslationService(from, to);
+    log(`translation service ${from}${to} construction took ${(Date.now() - start) / 1000} secs`);
     document.querySelector("#load").disabled = false;
-    //log('Model Alignment:', translationModel.isAlignmentSupported());
+    //log('Model Alignment:', translationService.isAlignmentSupported());
   });
 
   const translateCall = () => {

From 44aa70a0647a911df165d770c04fd68fa730d59a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Tue, 15 Jun 2021 18:59:51 +0100
Subject: [PATCH 274/442] Account for EOS in both source and target annotations
 (#190)

---
 src/translator/response_builder.cpp | 3 +--
 src/translator/text_processor.cpp   | 9 ++++++++-
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/src/translator/response_builder.cpp b/src/translator/response_builder.cpp
index 6d60b6c48..2944de53a 100644
--- a/src/translator/response_builder.cpp
+++ b/src/translator/response_builder.cpp
@@ -19,7 +19,6 @@ void ResponseBuilder::buildQualityScores(Histories &histories, Response &respons
     // logprobs.
     auto normalizedPathScore = std::get<2>(result);
     auto wordQualities = hyp->tracebackWordScores();
-    wordQualities.pop_back();
     response.qualityScores.push_back(Quality{normalizedPathScore, wordQualities});
   }
 }
@@ -64,7 +63,7 @@ void ResponseBuilder::buildTranslatedText(Histories &histories, Response &respon
 
     std::string decoded;
     std::vector<string_view> targetSentenceMappings;
-    vocabs_.target()->decodeWithByteRanges(words, decoded, targetSentenceMappings);
+    vocabs_.target()->decodeWithByteRanges(words, decoded, targetSentenceMappings, /*ignoreEOS=*/false);
 
     switch (responseOptions_.concatStrategy) {
       case ConcatStrategy::FAITHFUL: {
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index 234c48445..2d75f3348 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -54,8 +54,15 @@ void TextProcessor::wrap(Segment &segment, std::vector<string_view> &wordRanges,
     segments.back().push_back(sourceEosId());
 
     auto astart = wordRanges.begin() + offset;
+
+    // Construct a part vector of string_view representing wrapped segment, use the last string_view to create an EOS
+    // string_view manually.
+    std::vector<string_view> partWordRanges(astart, astart + diff);
+    string_view &last = partWordRanges.back();
+    const char *end = last.data() + last.size();
+    partWordRanges.emplace_back(end, 0);
     // diff > 0
-    source.recordExistingSentence(astart, astart + diff, astart->data());
+    source.recordExistingSentence(partWordRanges.begin(), partWordRanges.end(), astart->data());
   }
 }
 

From 13a1fe870f34b3290732d74fe7656bb0ce76aca0 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 21 Jun 2021 18:53:30 +0100
Subject: [PATCH 275/442] Load sentence-splitter (non-breaking prefixes) from
 ByteArray

Service now allows loading Sentence-Splitter (non-breaking prefix file) from ByteArray. Behaviour is consistent with the rest of the ByteArray loads (model, shortlist), where first the ByteArray is checked if empty, if not fall back to loading from file-path.

Adds regression test to check if source-sentences in constructed Response match expected behaviour when the non-breaking-prefixes file is provided.

Bonus refactoring to remove an extra layer that existed for no reason.
---
 3rd_party/ssplit-cpp                 |  2 +-
 bergamot-translator-tests            |  2 +-
 src/translator/CMakeLists.txt        |  1 -
 src/translator/byte_array_util.cpp   | 10 +++
 src/translator/byte_array_util.h     |  1 +
 src/translator/sentence_splitter.cpp | 47 --------------
 src/translator/sentence_splitter.h   | 33 ----------
 src/translator/service.cpp           |  7 ++-
 src/translator/text_processor.cpp    | 94 ++++++++++++++++++++++++----
 src/translator/text_processor.h      | 63 +++++++++++++------
 10 files changed, 145 insertions(+), 115 deletions(-)
 delete mode 100644 src/translator/sentence_splitter.cpp
 delete mode 100644 src/translator/sentence_splitter.h

diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 177ee2a32..f0fe09765 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 177ee2a326b733f8395842f01b197637047de9f6
+Subproject commit f0fe09765ce22c6db79b15123c6599b2b419d240
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index e1ae3b58a..a6d01f878 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit e1ae3b58a6e6c25856a5f279fedb567bbe695c49
+Subproject commit a6d01f8781e75043a9032913e9c9c2cb9ed24c64
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index cea523e56..34e599ba6 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -5,7 +5,6 @@ configure_file(${CMAKE_CURRENT_SOURCE_DIR}/project_version.h.in
 add_library(bergamot-translator STATIC
     byte_array_util.cpp
     text_processor.cpp
-    sentence_splitter.cpp
     batch_translator.cpp 
     request.cpp 
     batcher.cpp
diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index 8c3d60837..3790a01a9 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -129,8 +129,18 @@ MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options) {
   memoryBundle.model = getModelMemoryFromConfig(options);
   memoryBundle.shortlist = getShortlistMemoryFromConfig(options);
   getVocabsMemoryFromConfig(options, memoryBundle.vocabs);
+  memoryBundle.ssplitPrefixFile = getSsplitPrefixFileMemoryFromConfig(options);
   return memoryBundle;
 }
 
+AlignedMemory getSsplitPrefixFileMemoryFromConfig(marian::Ptr<marian::Options> options) {
+  std::string fpath = options->get<std::string>("ssplit-prefix-file", "");
+  if (!fpath.empty()) {
+    return loadFileToMemory(fpath, 64);
+  }
+  // Return empty AlignedMemory
+  return AlignedMemory();
+}
+
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/translator/byte_array_util.h b/src/translator/byte_array_util.h
index 63aa3a6ce..04cbf9ee9 100644
--- a/src/translator/byte_array_util.h
+++ b/src/translator/byte_array_util.h
@@ -7,6 +7,7 @@ namespace bergamot {
 AlignedMemory loadFileToMemory(const std::string& path, size_t alignment);
 AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options);
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options);
+AlignedMemory getSsplitPrefixFileMemoryFromConfig(marian::Ptr<marian::Options> options);
 void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
                                std::vector<std::shared_ptr<AlignedMemory>>& vocabMemories);
 bool validateBinaryModel(const AlignedMemory& model, uint64_t fileSize);
diff --git a/src/translator/sentence_splitter.cpp b/src/translator/sentence_splitter.cpp
deleted file mode 100644
index d0ada81e7..000000000
--- a/src/translator/sentence_splitter.cpp
+++ /dev/null
@@ -1,47 +0,0 @@
-#include "sentence_splitter.h"
-
-#include <string>
-
-#include "common/cli_helper.h"
-#include "common/logging.h"
-#include "common/options.h"
-
-namespace marian {
-namespace bergamot {
-
-SentenceSplitter::SentenceSplitter(marian::Ptr<marian::Options> options) : options_(options) {
-  std::string smode_str = options_->get<std::string>("ssplit-mode", "");
-  mode_ = string2splitmode(smode_str);
-  std::string ssplit_prefix_file = options_->get<std::string>("ssplit-prefix-file", "");
-
-  if (ssplit_prefix_file.size()) {
-    ssplit_prefix_file = marian::cli::interpolateEnvVars(ssplit_prefix_file);
-
-    LOG(info, "Loading protected prefixes for sentence splitting from {}", ssplit_prefix_file);
-
-    ssplit_.load(ssplit_prefix_file);
-  } else {
-    LOG(warn,
-        "Missing list of protected prefixes for sentence splitting. "
-        "Set with --ssplit-prefix-file.");
-  }
-}
-
-ug::ssplit::SentenceStream SentenceSplitter::createSentenceStream(const string_view &input) {
-  std::string_view input_converted(input.data(), input.size());
-  return std::move(ug::ssplit::SentenceStream(input_converted, this->ssplit_, mode_));
-}
-
-ug::ssplit::SentenceStream::splitmode SentenceSplitter::string2splitmode(const std::string &m) {
-  typedef ug::ssplit::SentenceStream::splitmode splitmode;
-  // @TODO: throw Exception on error
-  if (m == "sentence" || m == "Sentence") return splitmode::one_sentence_per_line;
-  if (m == "paragraph" || m == "Paragraph") return splitmode::one_paragraph_per_line;
-  if (m != "wrapped_text" && m != "WrappedText" && m != "wrappedText") {
-    LOG(warn, "Ignoring unknown text input format specification: {}.", m);
-  }
-  return splitmode::wrapped_text;
-}
-
-}  // namespace bergamot
-}  // namespace marian
diff --git a/src/translator/sentence_splitter.h b/src/translator/sentence_splitter.h
deleted file mode 100644
index 9c5816574..000000000
--- a/src/translator/sentence_splitter.h
+++ /dev/null
@@ -1,33 +0,0 @@
-#ifndef SRC_BERGAMOT_SENTENCE_SPLITTER_H_
-#define SRC_BERGAMOT_SENTENCE_SPLITTER_H_
-
-#include <string>
-
-#include "common/options.h"
-#include "data/types.h"
-#include "definitions.h"
-#include "ssplit.h"
-
-namespace marian {
-namespace bergamot {
-
-class SentenceSplitter {
-  // A wrapper around @ugermann's ssplit-cpp compiled from several places in
-  // mts. Constructed based on options. Used in TextProcessor below to create
-  // sentence-streams, which provide access to one sentence from blob of text at
-  // a time.
- public:
-  explicit SentenceSplitter(Ptr<Options> options);
-  ug::ssplit::SentenceStream createSentenceStream(string_view const &input);
-
- private:
-  ug::ssplit::SentenceSplitter ssplit_;
-  Ptr<Options> options_;
-  ug::ssplit::SentenceStream::splitmode mode_;
-  ug::ssplit::SentenceStream::splitmode string2splitmode(const std::string &m);
-};
-
-}  // namespace bergamot
-}  // namespace marian
-
-#endif  //  SRC_BERGAMOT_SENTENCE_SPLITTER_H_
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 9a0370602..b3695a542 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -13,7 +13,7 @@ Service::Service(Ptr<Options> options, MemoryBundle memoryBundle)
     : requestId_(0),
       options_(options),
       vocabs_(options, std::move(memoryBundle.vocabs)),
-      text_processor_(vocabs_, options),
+      text_processor_(options, vocabs_, std::move(memoryBundle.ssplitPrefixFile)),
       batcher_(options),
       numWorkers_(std::max<int>(1, options->get<int>("cpu-threads"))),
       modelMemory_(std::move(memoryBundle.model)),
@@ -78,8 +78,9 @@ std::vector<Response> Service::translateMultiple(std::vector<std::string> &&inpu
 
 std::future<Response> Service::queueRequest(std::string &&input, ResponseOptions responseOptions) {
   Segments segments;
-  AnnotatedText source(std::move(input));
-  text_processor_.process(source, segments);
+  AnnotatedText source;
+
+  text_processor_.process(std::move(input), source, segments);
 
   std::promise<Response> responsePromise;
   auto future = responsePromise.get_future();
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index 2d75f3348..cd8a14340 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -3,6 +3,7 @@
 #include <vector>
 
 #include "annotation.h"
+#include "common/cli_helper.h"
 #include "common/options.h"
 #include "data/types.h"
 #include "definitions.h"
@@ -10,20 +11,84 @@
 namespace marian {
 namespace bergamot {
 
+namespace {
+ug::ssplit::SentenceStream::splitmode string2splitmode(const std::string &m) {
+  typedef ug::ssplit::SentenceStream::splitmode splitmode;
+  if (m == "sentence") {
+    return splitmode::one_sentence_per_line;
+  } else if (m == "paragraph") {
+    return splitmode::one_paragraph_per_line;
+  } else if (m == "wrapped_text") {
+    return splitmode::wrapped_text;
+  } else {
+    ABORT("Unknown ssplitmode {}, Please choose one of {sentence,paragraph,wrapped_text}");
+  }
+}
+
+ug::ssplit::SentenceSplitter loadSplitter(const std::string &ssplitPrefixFile) {
+  // Temporarily supports empty, will be removed when mozilla passes ssplitPrefixFile
+  ug::ssplit::SentenceSplitter splitter;
+  if (ssplitPrefixFile.size()) {
+    std::string interpSsplitPrefixFile = marian::cli::interpolateEnvVars(ssplitPrefixFile);
+    LOG(info, "Loading protected prefixes for sentence splitting from {}", interpSsplitPrefixFile);
+    splitter.load(interpSsplitPrefixFile);
+  } else {
+    LOG(warn,
+        "Missing list of protected prefixes for sentence splitting. "
+        "Set with --ssplit-prefix-file.");
+  }
+  return splitter;
+}
+
+ug::ssplit::SentenceSplitter loadSplitter(const AlignedMemory &memory) {
+  // Temporarily supports empty, will be removed when mozilla passes memory
+  ug::ssplit::SentenceSplitter splitter;
+  if (memory.size()) {
+    std::string_view serialized(memory.begin(), memory.size());
+    splitter.loadFromSerialized(serialized);
+  }
+  return splitter;
+}
+
+}  // namespace
+
 Segment TextProcessor::tokenize(const string_view &segment, std::vector<string_view> &wordRanges) {
   // vocabs_->sources().front() is invoked as we currently only support one source vocab
   return vocabs_.sources().front()->encodeWithByteRanges(segment, wordRanges, /*addEOS=*/false, /*inference=*/true);
 }
 
-TextProcessor::TextProcessor(Vocabs &vocabs, Ptr<Options> options) : vocabs_(vocabs), sentence_splitter_(options) {
-  max_length_break_ = options->get<int>("max-length-break");
-  max_length_break_ = max_length_break_ - 1;
-  ABORT_IF(max_length_break_ < 0, "max-length-break cannot be < 0");
+TextProcessor::TextProcessor(Ptr<Options> options, const Vocabs &vocabs, const std::string &ssplit_prefix_file)
+    : vocabs_(vocabs), ssplit_(loadSplitter(ssplit_prefix_file)) {
+  parseCommonOptions(options);
+}
+
+TextProcessor::TextProcessor(Ptr<Options> options, const Vocabs &vocabs, const AlignedMemory &memory)
+    : vocabs_(vocabs) {
+  // This is not the best of the solutions at the moment, but is consistent with what happens among other structures
+  // like model, vocabulary or shortlist. First, we check if the bytearray is empty. If not, we load from ByteArray. In
+  // case empty, the string based loader which reads from file is called. However, ssplit allows for not supplying
+  // ssplit-prefix-file where-in the purely regular expression based splitter is activated.
+  //
+  // For now, we allow not supplying an ssplit-prefix-file.
+
+  if (memory.begin() == nullptr && memory.size()) {
+    ssplit_ = loadSplitter(memory);
+  } else {
+    ssplit_ = loadSplitter(options->get<std::string>("ssplit-prefix-file", ""));
+  }
+  parseCommonOptions(options);
 }
 
-void TextProcessor::process(AnnotatedText &source, Segments &segments) {
-  string_view query = string_view(source.text);
-  auto sentenceStream = sentence_splitter_.createSentenceStream(query);
+void TextProcessor::parseCommonOptions(Ptr<Options> options) {
+  maxLengthBreak_ = options->get<size_t>("max-length-break");
+  ssplitMode_ = string2splitmode(options->get<std::string>("ssplit-mode", "paragraph"));
+}
+
+void TextProcessor::process(std::string &&input, AnnotatedText &source, Segments &segments) {
+  source = std::move(AnnotatedText(std::move(input)));
+  std::string_view input_converted(source.text.data(), source.text.size());
+  auto sentenceStream = ug::ssplit::SentenceStream(input_converted, ssplit_, ssplitMode_);
+
   std::string_view sentenceStringPiece;
 
   while (sentenceStream >> sentenceStringPiece) {
@@ -35,7 +100,7 @@ void TextProcessor::process(AnnotatedText &source, Segments &segments) {
     // There are some cases where SentencePiece or vocab returns no words
     // after normalization. 0 prevents any empty entries from being added.
     if (segment.size() > 0) {
-      // Wrap segment into sentences of at most max_length_break_ tokens and
+      // Wrap segment into sentences of at most maxLengthBreak_ tokens and
       // tell source about them.
       wrap(segment, wordRanges, segments, source);
     }
@@ -44,14 +109,21 @@ void TextProcessor::process(AnnotatedText &source, Segments &segments) {
 
 void TextProcessor::wrap(Segment &segment, std::vector<string_view> &wordRanges, Segments &segments,
                          AnnotatedText &source) {
-  for (size_t offset = 0; offset < segment.size(); offset += max_length_break_) {
+  // There's an EOS token added to the words, manually. SentencePiece/marian-vocab is set to not append EOS. Marian
+  // requires EOS to be at the end as a marker to start translating. So while we're supplied maxLengthBreak_ from
+  // outside, we need to ensure there's space for EOS in each wrapped segment.
+  Word sourceEosId = vocabs_.sources().front()->getEosId();
+  size_t wrapStep = maxLengthBreak_ - 1;
+
+  for (size_t offset = 0; offset < segment.size(); offset += wrapStep) {
     auto start = segment.begin() + offset;
 
+    // Restrict the range within bounds.
     size_t left = segment.size() - offset;
-    size_t diff = std::min(max_length_break_, left);
+    size_t diff = std::min(maxLengthBreak_, left);
 
     segments.emplace_back(start, start + diff);
-    segments.back().push_back(sourceEosId());
+    segments.back().push_back(sourceEosId);
 
     auto astart = wordRanges.begin() + offset;
 
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index be37c3de6..1dc5a4fa7 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -3,43 +3,70 @@
 
 #include <vector>
 
+#include "aligned.h"
 #include "annotation.h"
 #include "data/types.h"
 #include "data/vocab.h"
 #include "definitions.h"
-#include "sentence_splitter.h"
+#include "ssplit.h"
 #include "vocabs.h"
 
 namespace marian {
 namespace bergamot {
 
 class TextProcessor {
-  // TextProcessor handles loading the sentencepiece vocabulary and also
-  // contains an instance of sentence-splitter based on ssplit.
-  //
-  // Used in Service to convert an incoming blog of text to a vector of
-  // sentences (vector of words). In addition, the ByteRanges of the
-  // source-tokens in unnormalized text are provided as string_views.
+  /// TextProcessor handles loading the sentencepiece vocabulary and also
+  /// contains an instance of sentence-splitter based on ssplit.
+  ///
+  /// Used in Service to convert an incoming blob of text to a vector of
+  /// sentences (vector of words). In addition, the ByteRanges of the
+  /// source-tokens in unnormalized text are provided as string_views.
  public:
-  explicit TextProcessor(Vocabs &vocabs, Ptr<Options>);
+  // There are two ways to construct text-processor, different in a file-system
+  // based prefix file load and a memory based prefix file store. @jerinphilip
+  // is not doing magic inference inside to determine file-based or memory
+  // based on one being empty or not.
 
-  void process(AnnotatedText &source, Segments &segments);
+  /// Construct TextProcessor from options, vocabs and prefix-file.
+  /// @param [in] options: expected to contain `max-length-break`, `ssplit-mode`.
+  /// @param [in] vocabs: Vocabularies used to process text into sentences to marian::Words and corresponding ByteRange
+  /// information in AnnotatedText.
+  /// @param [in] ssplit_prefix_file: Path to ssplit-prefix file compatible with moses-tokenizer.
+  TextProcessor(Ptr<Options>, const Vocabs &vocabs, const std::string &ssplit_prefix_file);
+
+  /// Construct TextProcessor from options, vocabs and prefix-file supplied as a bytearray. For other parameters, see
+  /// the path based constructor.
+  /// Note: This falls back to string based loads if memory is null, this behaviour will be deprecated in the future.
+  ///
+  /// @param [in] memory: ssplit-prefix-file contents in memory, passed as a bytearray.
+  TextProcessor(Ptr<Options>, const Vocabs &vocabs, const AlignedMemory &memory);
+
+  /// Wrap into sentences of at most maxLengthBreak_ tokens and add to source.
+  /// @param [in] blob: Input blob, will be bound to source and annotations on it stored.
+  /// @param [out] source: AnnotatedText instance holding input and annotations of sentences and pieces
+  /// @param [out] segments: marian::Word equivalents of the sentences processed and stored in AnnotatedText for
+  /// consumption of marian translation pipeline.
+
+  void process(std::string &&blob, AnnotatedText &source, Segments &segments);
 
  private:
-  // Tokenizes an input string, returns Words corresponding. Loads the
-  // corresponding byte-ranges into tokenRanges.
+  void parseCommonOptions(Ptr<Options> options);
+
+  /// Tokenizes an input string, returns Words corresponding. Loads the
+  /// corresponding byte-ranges into tokenRanges.
   Segment tokenize(const string_view &input, std::vector<string_view> &tokenRanges);
 
-  // Wrap into sentences of at most max_length_break_ tokens and add to source.
+  /// Wrap into sentences of at most maxLengthBreak_ tokens and add to source.
   void wrap(Segment &sentence, std::vector<string_view> &tokenRanges, Segments &segments, AnnotatedText &source);
 
-  // shorthand, used only in truncate()
-  // vocabs_->sources().front() is invoked as we currently only support one source vocab
-  const Word sourceEosId() const { return vocabs_.sources().front()->getEosId(); }
+  const Vocabs &vocabs_;   ///< Vocabularies used to tokenize a sentence
+  size_t maxLengthBreak_;  ///< Parameter used to wrap sentences to a maximum number of tokens
+
+  /// SentenceSplitter compatible with moses sentence-splitter
+  ug::ssplit::SentenceSplitter ssplit_;
 
-  const Vocabs &vocabs_;
-  SentenceSplitter sentence_splitter_;
-  size_t max_length_break_;
+  /// Mode of splitting, can be line ('\n') based, paragraph based, also supports a wrapped mode.
+  ug::ssplit::SentenceStream::splitmode ssplitMode_;
 };
 
 }  // namespace bergamot

From cb855be1a7f3d362995b9528119549517a81a5eb Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 28 Jun 2021 14:54:39 +0100
Subject: [PATCH 276/442] maxLengthBreak_ -> wrapStep bugfix (#200)

---
 src/translator/text_processor.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index cd8a14340..249ce8cda 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -120,7 +120,7 @@ void TextProcessor::wrap(Segment &segment, std::vector<string_view> &wordRanges,
 
     // Restrict the range within bounds.
     size_t left = segment.size() - offset;
-    size_t diff = std::min(maxLengthBreak_, left);
+    size_t diff = std::min(wrapStep, left);
 
     segments.emplace_back(start, start + diff);
     segments.back().push_back(sourceEosId);

From a202e350c71220301bfddfdd264f1bd4a14e26bd Mon Sep 17 00:00:00 2001
From: Jerin Philip <jphilip@ed.ac.uk>
Date: Mon, 5 Jul 2021 14:51:01 +0100
Subject: [PATCH 277/442] Change ResponseBuilder to accept callback instead of
 future (#142)

* Change ResponseBuilder to accept callback

Breaks things everywhere, now we follow the compiler to fix and convert
the std::future -> callback.

* More std::future -> callback

* std::future out of service.{h,cpp}

* compile is working, so is callback

* Some reshuffling of args

* Fixing merge error

* Fixing signature conflicts out of merge

* Fixing that test duct-taping future

* Minor adjustment to get that future back

* Add documentation for the new callback function

* Applying clang-format after update

* Using default responseOptions

* Remove future references from documentation

* translateMultiple only for WASM (#177)

* BRT: update to main; fresh-failures hopefully

* Converting test translateFromStdin to use callback

* BRT: Add fresh #native and #wasm tags

* future from promise, fix error

* Adding #native to GitHub CI

Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
---
 .github/workflows/native.yml      |  6 ++--
 app/cli.h                         | 15 ++++++++--
 bergamot-translator-tests         |  2 +-
 src/tests/apps.cpp                |  9 ++++--
 src/translator/response_builder.h | 20 +++++++------
 src/translator/service.cpp        | 50 +++++++++++--------------------
 src/translator/service.h          | 40 +++++++------------------
 7 files changed, 63 insertions(+), 79 deletions(-)

diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
index c572cb5d8..e27cbe1b0 100644
--- a/.github/workflows/native.yml
+++ b/.github/workflows/native.yml
@@ -25,7 +25,7 @@ jobs:
           os: ubuntu-18.04
           identifier: ubuntu_1804_full
           cmake: -DCOMPILE_TESTS=on
-          brt_tags: ''
+          brt_tags: "'#native'"
           unittests: 'true'
         - name: Ubuntu 18.04 minimal
           os: ubuntu-18.04
@@ -37,7 +37,7 @@ jobs:
           os: ubuntu-20.04
           identifier: ubuntu_2004_full
           cmake: -DCOMPILE_TESTS=on
-          brt_tags: ''
+          brt_tags: "'#native'"
           unittests: 'true'
         - name: Ubuntu 20.04 minimal
           os: ubuntu-20.04
@@ -140,7 +140,7 @@ jobs:
           os: macos-10.15
           identifier: mac_1015_full
           cmake: -DCOMPILE_TESTS=on -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=off
-          brt_tags: ""
+          brt_tags: "'#native'"
           unittests: 'true'
         - name: MacOS 10.15 minimal
           os: macos-10.15
diff --git a/app/cli.h b/app/cli.h
index d6e930fed..4afe8b9aa 100644
--- a/app/cli.h
+++ b/app/cli.h
@@ -50,6 +50,8 @@ void wasm(Ptr<Options> options) {
   ResponseOptions responseOptions;
   std::vector<std::string> texts;
 
+#ifdef WASM_COMPATIBLE_SOURCE
+  // Hide the translateMultiple operation
   for (std::string line; std::getline(std::cin, line);) {
     texts.emplace_back(line);
   }
@@ -59,6 +61,7 @@ void wasm(Ptr<Options> options) {
   for (auto &result : results) {
     std::cout << result.getTranslatedText() << std::endl;
   }
+#endif
 }
 
 /// Application used to benchmark with marian-decoder from time-to-time. The implementation in this repository follows a
@@ -88,7 +91,11 @@ void decoder(Ptr<Options> options) {
   std::string input = std_input.str();
 
   // Wait on future until Response is complete
-  std::future<Response> responseFuture = service.translate(std::move(input));
+  std::promise<Response> responsePromise;
+  std::future<Response> responseFuture = responsePromise.get_future();
+  auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
+
+  service.translate(std::move(input), std::move(callback));
   responseFuture.wait();
   const Response &response = responseFuture.get();
 
@@ -126,7 +133,11 @@ void native(Ptr<Options> options) {
   ResponseOptions responseOptions;
 
   // Wait on future until Response is complete
-  std::future<Response> responseFuture = service.translate(std::move(input), responseOptions);
+  std::promise<Response> responsePromise;
+  std::future<Response> responseFuture = responsePromise.get_future();
+  auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
+
+  service.translate(std::move(input), std::move(callback), responseOptions);
   responseFuture.wait();
   Response response = responseFuture.get();
 
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index a6d01f878..ee534f750 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit a6d01f8781e75043a9032913e9c9c2cb9ed24c64
+Subproject commit ee534f7507966efe3199ac84e56bdd4b3950b736
diff --git a/src/tests/apps.cpp b/src/tests/apps.cpp
index c57f5f571..b42f7a495 100644
--- a/src/tests/apps.cpp
+++ b/src/tests/apps.cpp
@@ -21,9 +21,14 @@ Response translateFromStdin(Ptr<Options> options, ResponseOptions responseOption
   inputStream << std::cin.rdbuf();
   std::string input = inputStream.str();
 
-  // Wait on future until Response is complete
-  std::future<Response> responseFuture = service.translate(std::move(input), responseOptions);
+  std::promise<Response> responsePromise;
+  std::future<Response> responseFuture = responsePromise.get_future();
+
+  auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
+  service.translate(std::move(input), callback, responseOptions);
+
   responseFuture.wait();
+
   Response response = responseFuture.get();
   return response;
 }
diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
index 84ea09ad5..bee189516 100644
--- a/src/translator/response_builder.h
+++ b/src/translator/response_builder.h
@@ -23,10 +23,13 @@ class ResponseBuilder {
   /// @param [in] responseOptions: ResponseOptions, indicating what to include
   /// or not in the response and any additional configurable parameters.
   /// @param [in] vocabs: marian vocab object (used in decoding)
-  /// @param [in] promise: promise to set with the constructed Response.
+  /// @param [in] callback: callback with operates on the constructed Response.
   ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source, Vocabs &vocabs,
-                  std::promise<Response> &&promise)
-      : responseOptions_(responseOptions), source_(std::move(source)), vocabs_(vocabs), promise_(std::move(promise)) {}
+                  std::function<void(Response &&)> callback)
+      : responseOptions_(responseOptions),
+        source_(std::move(source)),
+        vocabs_(vocabs),
+        callback_(std::move(callback)) {}
 
   /// Constructs and sets the promise of a Response object from obtained
   /// histories after translating.
@@ -54,8 +57,7 @@ class ResponseBuilder {
       buildAlignments(histories, response);
     }
 
-    // Once complete, set promise.
-    promise_.set_value(std::move(response));
+    callback_(std::move(response));
   }
 
  private:
@@ -79,10 +81,10 @@ class ResponseBuilder {
   // Data members are context/curried args for the functor.
 
   ResponseOptions responseOptions_;
-  const Vocabs &vocabs_;            // vocabs are required for decoding
-                                    // and any source validation checks.
-  std::promise<Response> promise_;  //  To be set when callback triggered and
-                                    //  after Response constructed.
+  const Vocabs &vocabs_;                       // vocabs are required for decoding
+                                               // and any source validation checks.
+  std::function<void(Response &&)> callback_;  //  To be set when callback triggered and
+                                               //  after Response constructed.
   AnnotatedText source_;
 };
 }  // namespace bergamot
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index b3695a542..26901debc 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -42,60 +42,44 @@ Service::Service(Ptr<Options> options, MemoryBundle memoryBundle)
 #endif
 }
 
-void Service::blockIfWASM() {
 #ifdef WASM_COMPATIBLE_SOURCE
-  Batch batch;
-  // There's no need to do shutdown here because it's single threaded.
-  while (batcher_ >> batch) {
-    blocking_translator_.translate(batch);
-  }
-#endif
-}
-
 std::vector<Response> Service::translateMultiple(std::vector<std::string> &&inputs, ResponseOptions responseOptions) {
   // We queue the individual Requests so they get compiled at batches to be
   // efficiently translated.
-  std::vector<std::future<Response>> responseFutures;
-  for (auto &input : inputs) {
-    std::future<Response> inputResponse = queueRequest(std::move(input), responseOptions);
-    responseFutures.push_back(std::move(inputResponse));
-  }
+  std::vector<Response> responses;
+  responses.resize(inputs.size());
 
-  // Dispatch is called once per request so compilation of sentences from
-  // multiple Requests happen.
-  blockIfWASM();
+  for (size_t i = 0; i < inputs.size(); i++) {
+    auto callback = [i, &responses](Response &&response) { responses[i] = std::move(response); };  //
+    queueRequest(std::move(inputs[i]), std::move(callback), responseOptions);
+  }
 
-  // Now wait for all Requests to complete, the future to fire and return the
-  // compiled Responses, we can probably return the future, but WASM quirks(?).
-  std::vector<Response> responses;
-  for (auto &future : responseFutures) {
-    future.wait();
-    responses.push_back(std::move(future.get()));
+  Batch batch;
+  // There's no need to do shutdown here because it's single threaded.
+  while (batcher_ >> batch) {
+    blocking_translator_.translate(batch);
   }
 
   return responses;
 }
+#endif
 
-std::future<Response> Service::queueRequest(std::string &&input, ResponseOptions responseOptions) {
+void Service::queueRequest(std::string &&input, std::function<void(Response &&)> &&callback,
+                           ResponseOptions responseOptions) {
   Segments segments;
   AnnotatedText source;
 
   text_processor_.process(std::move(input), source, segments);
 
-  std::promise<Response> responsePromise;
-  auto future = responsePromise.get_future();
-
-  ResponseBuilder responseBuilder(responseOptions, std::move(source), vocabs_, std::move(responsePromise));
+  ResponseBuilder responseBuilder(responseOptions, std::move(source), vocabs_, std::move(callback));
   Ptr<Request> request = New<Request>(requestId_++, std::move(segments), std::move(responseBuilder));
 
   batcher_.addWholeRequest(request);
-  return future;
 }
 
-std::future<Response> Service::translate(std::string &&input, ResponseOptions responseOptions) {
-  std::future<Response> future = queueRequest(std::move(input), responseOptions);
-  blockIfWASM();
-  return future;
+void Service::translate(std::string &&input, std::function<void(Response &&)> &&callback,
+                        ResponseOptions responseOptions) {
+  queueRequest(std::move(input), std::move(callback), responseOptions);
 }
 
 Service::~Service() {
diff --git a/src/translator/service.h b/src/translator/service.h
index 6c7ea9a7d..0a3658048 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -20,11 +20,6 @@
 namespace marian {
 namespace bergamot {
 
-/// Service offers methods create an asynchronous translation service that
-/// translates a plain (without any markups and emojis)  UTF-8 encoded text.
-/// This implementation supports translation from 1 source language to 1 target
-/// language.
-///
 ///  This is intended to be similar to the ones  provided for training or
 ///  decoding in ML pipelines with the following  additional capabilities:
 ///
@@ -37,23 +32,8 @@ namespace bergamot {
 ///  translated independent of each other. The translated sentences are then
 ///  joined back together and returned in Response.
 ///
-/// Service exposes methods to instantiate the service from a string
-/// configuration (which can cover most translators) and to translate an
-/// incoming blob of text.
-///
-///
-/// An example use of this API looks as follows:
-/// ```cpp
-///  options = ...;
-///  service = Service(options);
-///  std::string input_text = "Hello World";
-///  std::future<Response>
-///      responseFuture = service.translate(std::move(input_text));
-///  responseFuture.wait(); // Wait until translation has completed.
-///  Response response(std::move(response.get());
-///
-/// // Do things with response.
-/// ```
+/// Service exposes methods to instantiate from a string configuration (which
+/// can cover most translators) and to translate an incoming blob of text.
 ///
 /// Optionally Service can be initialized by also passing bytearray memories
 /// for purposes of efficiency (which defaults to empty and then reads from
@@ -87,11 +67,16 @@ class Service {
   /// save compute spent in constructing these objects.
   ///
   /// @param [in] source: rvalue reference of the string to be translated
+  /// @param [in] callback: A callback function provided by the client which
+  /// accepts an rvalue of a Response. Called on successful construction of a
+  /// Response following completion of translation of source by worker threads.
   /// @param [in] responseOptions: Options indicating whether or not to include
   /// some member in the Response, also specify any additional configurable
   /// parameters.
-  std::future<Response> translate(std::string &&source, ResponseOptions options = ResponseOptions());
+  void translate(std::string &&source, std::function<void(Response &&)> &&callback,
+                 ResponseOptions options = ResponseOptions());
 
+#ifdef WASM_COMPATIBLE_SOURCE
   /// Translate multiple text-blobs in a single *blocking* API call, providing
   /// ResponseOptions which applies across all text-blobs dictating how to
   /// construct Response. ResponseOptions can be used to enable/disable
@@ -102,26 +87,23 @@ class Service {
   /// text-blob. Note that there will be minor differences in output when
   /// text-blobs are individually translated due to approximations but similar
   /// quality nonetheless. If you have async/multithread capabilities, it is
-  /// recommended to work with futures and translate() API.
+  /// recommended to work with callbacks and translate() API.
   ///
   /// @param [in] source: rvalue reference of the string to be translated
   /// @param [in] responseOptions: ResponseOptions indicating whether or not
   /// to include some member in the Response, also specify any additional
   /// configurable parameters.
   std::vector<Response> translateMultiple(std::vector<std::string> &&source, ResponseOptions responseOptions);
+#endif
 
   /// Returns if model is alignment capable or not.
   bool isAlignmentSupported() const { return options_->hasAndNotEmpty("alignment"); }
 
  private:
   /// Queue an input for translation.
-  std::future<Response> queueRequest(std::string &&input, ResponseOptions responseOptions);
-
-  /// Dispatch call to translate after inserting in queue
-  void dispatchTranslate();
+  void queueRequest(std::string &&input, std::function<void(Response &&)> &&callback, ResponseOptions responseOptions);
 
   /// Translates through direct interaction between batcher_ and translators_
-  void blockIfWASM();
 
   /// Number of workers to launch.
   size_t numWorkers_;

From 6ad794fcef8438d4e1c2b66778a3c08e95982606 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Mon, 5 Jul 2021 11:38:39 +0200
Subject: [PATCH 278/442] Added public methods in Response class to return
 sentences

 - Refactored ByteRange struct and moved it to definition.h
---
 src/translator/annotation.h  |  9 +--------
 src/translator/definitions.h |  8 ++++++++
 src/translator/response.h    | 10 ++++++++++
 3 files changed, 19 insertions(+), 8 deletions(-)

diff --git a/src/translator/annotation.h b/src/translator/annotation.h
index dde340ba3..785d49dfe 100644
--- a/src/translator/annotation.h
+++ b/src/translator/annotation.h
@@ -6,18 +6,11 @@
 #include <vector>
 
 #include "data/types.h"
+#include "definitions.h"
 
 namespace marian {
 namespace bergamot {
 
-/// ByteRange stores indices for half-interval [begin, end) in a string. Can be
-/// used to represent a sentence, word.
-struct ByteRange {
-  size_t begin;
-  size_t end;
-  const size_t size() const { return end - begin; }
-};
-
 /// Annotation expresses sentence and token boundary information as ranges of
 /// bytes in a string, but does not itself own the string.
 ///
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index 14552a3c0..d5b874353 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -31,6 +31,14 @@ struct MemoryBundle {
   AlignedMemory ssplitPrefixFile{};
 };
 
+/// ByteRange stores indices for half-interval [begin, end) in a string. Can be
+/// used to represent a sentence, word.
+struct ByteRange {
+  size_t begin;
+  size_t end;
+  const size_t size() const { return end - begin; }
+};
+
 }  // namespace bergamot
 }  // namespace marian
 
diff --git a/src/translator/response.h b/src/translator/response.h
index 5c92441fe..2355f5225 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -65,6 +65,16 @@ struct Response {
   /// to (sub-)words accessible through Annotation.
   std::vector<Alignment> alignments;
 
+  /// Returns the source sentence (in terms of byte range) corresponding to sentenceIdx.
+  ///
+  /// @param [in] sentenceIdx: The index representing the sentence where 0 <= sentenceIdx < Response::size()
+  ByteRange getSourceSentenceAsByteRange(size_t sentenceIdx) const { return source.sentenceAsByteRange(sentenceIdx); }
+
+  /// Returns the translated sentence (in terms of byte range) corresponding to sentenceIdx.
+  ///
+  /// @param [in] sentenceIdx: The index representing the sentence where 0 <= sentenceIdx < Response::size()
+  ByteRange getTargetSentenceAsByteRange(size_t sentenceIdx) const { return target.sentenceAsByteRange(sentenceIdx); }
+
   const std::string &getOriginalText() const { return source.text; }
 
   const std::string &getTranslatedText() const { return target.text; }

From 7052722cd28f8016a5fd5781d126b04eab8257ef Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 6 Jul 2021 15:01:54 +0200
Subject: [PATCH 279/442] JS bindings to return sentence byte ranges

---
 wasm/bindings/response_bindings.cpp | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/wasm/bindings/response_bindings.cpp b/wasm/bindings/response_bindings.cpp
index 561911925..ca688249c 100644
--- a/wasm/bindings/response_bindings.cpp
+++ b/wasm/bindings/response_bindings.cpp
@@ -14,11 +14,20 @@ typedef marian::bergamot::Response Response;
 using namespace emscripten;
 
 // Binding code
+EMSCRIPTEN_BINDINGS(byte_range) {
+  value_object<marian::bergamot::ByteRange>("ByteRange")
+      .field("begin", &marian::bergamot::ByteRange::begin)
+      .field("end", &marian::bergamot::ByteRange::end);
+}
+
 EMSCRIPTEN_BINDINGS(response) {
   class_<Response>("Response")
       .constructor<>()
+      .function("size", &Response::size)
       .function("getOriginalText", &Response::getOriginalText)
-      .function("getTranslatedText", &Response::getTranslatedText);
+      .function("getTranslatedText", &Response::getTranslatedText)
+      .function("getSourceSentence", &Response::getSourceSentenceAsByteRange)
+      .function("getTranslatedSentence", &Response::getTargetSentenceAsByteRange);
 
   register_vector<Response>("VectorResponse");
 }

From 5a8fe209ce68137e5baa3b01e6e5971037dba6a1 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 6 Jul 2021 16:32:39 +0200
Subject: [PATCH 280/442] Wasm: Enabled sentence byte ranges in the wasm test
 page

 - Use JS bindings to print all sentences individually on
   console
---
 wasm/test_page/bergamot.html | 42 +++++++++++++++++++++++++++++++++++-
 1 file changed, 41 insertions(+), 1 deletion(-)

diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index c69c950a3..acc5bcd6d 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -81,6 +81,8 @@
   }
 
   var translationService, responseOptions, input = undefined;
+  const encoder = new TextEncoder(); // string to utf-8 converter
+  const decoder = new TextDecoder(); // utf-8 to string converter
   const constructTranslationService = async (from, to) => {
 
     const languagePair = `${from}${to}`;
@@ -188,18 +190,56 @@
     // Access input (just for debugging)
     console.log('Input size=', input.size());
 
-    // Translate the input; the result is a vector<Response>
+    // Translate the input, which is a vector<String>; the result is a vector<Response>
     let result = translationService.translate(input, responseOptions);
+
     const translatedParagraphs = [];
+    const translatedSentencesOfParagraphs = [];
+    const sourceSentencesOfParagraphs = [];
     for (let i = 0; i < result.size(); i++) {
       translatedParagraphs.push(result.get(i).getTranslatedText());
+      translatedSentencesOfParagraphs.push(getAllTranslatedSentencesOfParagraph(result.get(i)));
+      sourceSentencesOfParagraphs.push(getAllSourceSentencesOfParagraph(result.get(i)));
     }
     console.log({ translatedParagraphs });
+    console.log({ translatedSentencesOfParagraphs });
+    console.log({ sourceSentencesOfParagraphs });
+
     responseOptions.delete();
     input.delete();
     return translatedParagraphs;
   }
 
+  // This function extracts all the translated sentences from the Response and returns them.
+  const getAllTranslatedSentencesOfParagraph = (response) => {
+    const sentences = [];
+    const text = response.getTranslatedText();
+    for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
+      const utf8SentenceByteRange = response.getTranslatedSentence(sentenceIndex);
+      sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
+    }
+    return sentences;
+  }
+
+  // This function extracts all the source sentences from the Response and returns them.
+  const getAllSourceSentencesOfParagraph = (response) => {
+    const sentences = [];
+    const text = response.getOriginalText();
+    for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
+      const utf8SentenceByteRange = response.getSourceSentence(sentenceIndex);
+      sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
+    }
+    return sentences;
+  }
+
+  // This function returns a substring of text (a string). The substring is represented by
+  // byteRange (begin and end endices) within the utf-8 encoded version of the text.
+  const _getSentenceFromByteRange = (text, byteRange) => {
+    const utf8BytesView = encoder.encode(text);
+    const utf8SentenceBytes = utf8BytesView.subarray(byteRange.begin, byteRange.end);
+    return decoder.decode(utf8SentenceBytes);
+  }
+
   document.querySelector("#load").addEventListener("click", async() => {
     document.querySelector("#load").disabled = true;
     const lang = document.querySelector('input[name="modellang"]:checked').value;

From d31f96381b238b3987aa71c35864944d63919641 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Thu, 29 Jul 2021 12:25:09 +0100
Subject: [PATCH 281/442] Windows workflow: run-vcpkg7.{3->4}; vcpkg master
 (#208)

A cmake change has caused vcpkg to fail without much error message,
which is causing windows workflow runs to fail. Details in the following
link:

* https://github.com/microsoft/vcpkg/issues/18718

To fix, we're going with a version bump in vcpkg. Seeing that run-vcpkg
also seems to have gotten an update, updating run-vcpkg from 7.3 to 7.4
Playing with fire: vcpkg master commit
---
 .github/workflows/windows.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 00e9cfafb..7d1aca9d5 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -37,10 +37,10 @@ jobs:
       shell: powershell
 
     - name: Prepare vcpkg
-      uses: lukka/run-vcpkg@v7.3
+      uses: lukka/run-vcpkg@v7.4
       with:
         vcpkgArguments: protobuf pcre2
-        vcpkgGitCommitId: 6185aa76504a5025f36754324abf307cc776f3da 
+        vcpkgGitCommitId: 8dddc6c899ce6fdbeab38b525a31e7f23cb2d5bb
         vcpkgDirectory: ${{ github.workspace }}/vcpkg/
         vcpkgTriplet: x64-windows-static
 

From f3e00ae657bb99fe47e4c3592b83442662be2307 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Wed, 11 Aug 2021 13:28:15 +0200
Subject: [PATCH 282/442] Added build instructions to run on other browsers

 - Disabled compiling with wormhole which is Firefox specific feature
---
 README.md | 40 ++++++++++++++++++++++++++--------------
 1 file changed, 26 insertions(+), 14 deletions(-)

diff --git a/README.md b/README.md
index 0fbf58efd..3ba11f026 100644
--- a/README.md
+++ b/README.md
@@ -29,20 +29,32 @@ Building on wasm requires Emscripten toolchain. It can be downloaded and install
 
 #### <a name="Compile"></a> Compile
 
-1. Create a folder where you want to build all the artifacts (`build-wasm` in this case) and compile
-    ```bash
-    mkdir build-wasm
-    cd build-wasm
-    emcmake cmake -DCOMPILE_WASM=on ../
-    emmake make -j2
-    ```
-
-    The wasm artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
-
-2. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
-    ```bash
-    bash ../wasm/patch-artifacts-enable-wormhole.sh
-    ```
+To build a version that translates with higher speeds on Firefox Nightly browser, follow these instructions:
+
+   1. Create a folder where you want to build all the artifacts (`build-wasm` in this case) and compile
+       ```bash
+       mkdir build-wasm
+       cd build-wasm
+       emcmake cmake -DCOMPILE_WASM=on ../
+       emmake make -j2
+       ```
+
+       The wasm artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
+
+   2. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
+       ```bash
+       bash ../wasm/patch-artifacts-enable-wormhole.sh
+       ```
+
+To build a version that runs on all browsers (including Firefox Nightly) but translates slowly, follow these instructions:
+
+  1. Create a folder where you want to build all the artifacts (`build-wasm` in this case) and compile
+      ```bash
+      mkdir build-wasm
+      cd build-wasm
+      emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=off ../
+      emmake make -j2
+      ```
 
 #### Recompiling
 As long as you don't update any submodule, just follow [Compile](#Compile) steps.\

From 972d8560b576ce2991996a57c13f0c62ce825728 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Fri, 13 Aug 2021 16:26:44 +0100
Subject: [PATCH 283/442] Add a clang-tidy run (#214)

Adds a clang-tidy run in addition to the existing clang-format checks.
The clang-tidy checks are not enforced, but is potentially useful to
point to during review.
---
 .clang-tidy                         | 32 +++++++++++++++++++++++++++++
 .github/workflows/coding-styles.yml | 20 +++++++++++++++++-
 2 files changed, 51 insertions(+), 1 deletion(-)
 create mode 100644 .clang-tidy

diff --git a/.clang-tidy b/.clang-tidy
new file mode 100644
index 000000000..bdbadb624
--- /dev/null
+++ b/.clang-tidy
@@ -0,0 +1,32 @@
+Checks: >
+  .*,
+  bugprone-*,  
+  concurrency-*,
+  google-*,
+  portability-*,
+  performance-*,
+  clang-analyzer-*,
+  readability-*,
+  -readability-implicit-bool-conversion,
+  -readability-isolate-declaration,
+  -readability-uppercase-literal-suffix,
+  misc-*,
+  -misc-noexcept*,
+  modernize-*,
+  -modernize-deprecated-headers,
+  -modernize-use-nodiscard,
+  -modernize-raw-string-literal,
+  -modernize-return-braced-init-list,
+  -modernize-use-equals-delete,
+  -modernize-use-trailing-return-type,
+
+
+
+CheckOptions:
+  - { key: readability-identifier-naming.ClassCase,     value: CamelCase  }
+  - { key: readability-identifier-naming.ClassMethodCase, value: camelBack  }
+  - { key: readability-identifier-naming.VariableCase,  value: camelBack  }
+  - { key: readability-identifier-naming.FunctionCase,  value: camelBack  }
+  - { key: readability-identifier-naming.PrivateMemberSuffix,  value: _   }
+  - { key: readability-identifier-naming.ParameterCase, value: camelBack  }
+
diff --git a/.github/workflows/coding-styles.yml b/.github/workflows/coding-styles.yml
index 176e8c7bf..330790e88 100644
--- a/.github/workflows/coding-styles.yml
+++ b/.github/workflows/coding-styles.yml
@@ -1,5 +1,7 @@
 name: "Coding Style"
 
+env:
+    clang_version: 10
 on: 
   push:
     branches: [ main, ci-sandbox ]
@@ -19,8 +21,24 @@ jobs:
         - name: Install dependencies
           run: |
             sudo apt-get update 
-            sudo apt-get install -y clang-format
+            sudo apt-get install -y build-essential cmake
+            sudo apt-get install -y clang-format clang-tidy-${{ env.clang_version }}
 
         - name: Run clang-format
           run:
               python3 run-clang-format.py --style file -r src wasm
+
+
+        - name: Prepare build, compilation database etc.
+          run: |
+              mkdir -p build
+              cd build 
+              cmake \
+                -DUSE_WASM_COMPATIBLE_SOURCE=off -DCMAKE_EXPORT_COMPILE_COMMANDS=on \
+                -DCMAKE_C_COMPILER=clang-${{ env.clang_version }} -DCMAKE_CXX_COMPILER=clang++-${{ env.clang_version }} \
+                ..
+
+        - name: Run clang-tidy
+          run: |
+              run-clang-tidy-${{ env.clang_version }} -p build "$PWD/src/.*"
+              run-clang-tidy-${{ env.clang_version }} -p build "$PWD/app/.*"

From b64ffce496b195c2cbfa4707801dd8d209f3d48c Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 26 Aug 2021 15:22:52 +0200
Subject: [PATCH 284/442] Wasm test page using web workers now (#218)

---
 wasm/test_page/bergamot-httpserver.js |  19 +-
 wasm/test_page/bergamot.html          | 234 +------------------------
 wasm/test_page/bergamot.js            |  48 +++++
 wasm/test_page/worker.js              | 243 ++++++++++++++++++++++++++
 4 files changed, 303 insertions(+), 241 deletions(-)
 create mode 100644 wasm/test_page/bergamot.js
 create mode 100644 wasm/test_page/worker.js

diff --git a/wasm/test_page/bergamot-httpserver.js b/wasm/test_page/bergamot-httpserver.js
index b28719fed..f657de3e5 100644
--- a/wasm/test_page/bergamot-httpserver.js
+++ b/wasm/test_page/bergamot-httpserver.js
@@ -1,14 +1,16 @@
 require(__dirname  + '/helper.js');
 
-var http = require('http');
-var express = require('express');
-var app = express();
-var server = http.createServer(app);
-var fs = require('fs');
-var url = require('url');
+const http = require('http');
+const express = require('express');
+const app = express();
+const server = http.createServer(app);
+const fs = require('fs');
+const url = require('url');
 const nocache = require('nocache');
 const cors = require('cors');
 
+let port = 8000;
+
 app.use(cors())
 app.use(nocache());
 app.get('/*.*' , cors(), function(req, res) {
@@ -26,10 +28,11 @@ function serveFile(res, pathName, mime) {
         }
         res.header('Cross-Origin-Embedder-Policy','require-corp');
         res.header('Cross-Origin-Opener-Policy','same-origin');
+        res.header('Cross-Origin-Resource-Policy','same-origin');
         res.writeHead(200, {"Content-Type": mime});
         res.end(data);
     });
 }
 
-server.listen(8000);
-console.log('HTTP and BinaryJS server started on port 8000');
+server.listen(port);
+console.log(`HTTP and BinaryJS server started on port ${port}`);
diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
index acc5bcd6d..8c80ed2cd 100644
--- a/wasm/test_page/bergamot.html
+++ b/wasm/test_page/bergamot.html
@@ -60,239 +60,7 @@
     <textarea id="log" name="log" rows="50" cols="75"></textarea>
 </div>
 
-<script>
-  // This function downloads file from a url and returns the array buffer
-  const downloadAsArrayBuffer = async(url) => {
-    const response = await fetch(url);
-    if (!response.ok) {
-      throw Error(`Downloading ${url} failed: HTTP ${response.status} - ${response.statusText}`);
-    }
-    return response.arrayBuffer();
-  }
-
-  // This function constructs the AlignedMemory from the array buffer and the alignment size
-  function constructAlignedMemoryFromBuffer(buffer, alignmentSize) {
-    var byteArray = new Int8Array(buffer);
-    console.debug("byteArray size: ", byteArray.byteLength);
-    var alignedMemory = new Module.AlignedMemory(byteArray.byteLength, alignmentSize);
-    const alignedByteArrayView = alignedMemory.getByteArrayView();
-    alignedByteArrayView.set(byteArray);
-    return alignedMemory;
-  }
-
-  var translationService, responseOptions, input = undefined;
-  const encoder = new TextEncoder(); // string to utf-8 converter
-  const decoder = new TextDecoder(); // utf-8 to string converter
-  const constructTranslationService = async (from, to) => {
-
-    const languagePair = `${from}${to}`;
-
-    // Vocab files are re-used in both translation directions
-    const vocabLanguagePair = from === "en" ? `${to}${from}` : languagePair;
-
-    // Set the Model Configuration as YAML formatted string.
-    // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-    /*const modelConfig = `models:
-  - /${languagePair}/model.${languagePair}.intgemm.alphas.bin
-vocabs:
-  - /${languagePair}/vocab.${vocabLanguagePair}.spm
-  - /${languagePair}/vocab.${vocabLanguagePair}.spm
-beam-size: 1
-normalize: 1.0
-word-penalty: 0
-max-length-break: 128
-mini-batch-words: 1024
-workspace: 128
-max-length-factor: 2.0
-skip-cost: true
-cpu-threads: 0
-quiet: true
-quiet-translation: true
-shortlist:
-    - /${languagePair}/lex.${languagePair}.s2t
-    - 50
-    - 50
-`;
-*/
-
-const modelConfig = `beam-size: 1
-normalize: 1.0
-word-penalty: 0
-max-length-break: 128
-mini-batch-words: 1024
-workspace: 128
-max-length-factor: 2.0
-skip-cost: true
-cpu-threads: 0
-quiet: true
-quiet-translation: true
-gemm-precision: int8shift
-`;
-
-// TODO: Use in model config when wormhole is enabled:
-// gemm-precision: int8shift
-// TODO: Use in model config when loading of binary models is supported and we use model.intgemm.alphas.bin:
-// gemm-precision: int8shiftAlphaAll
-
-    const modelFile = `models/${languagePair}/model.${languagePair}.intgemm.alphas.bin`;
-    const shortlistFile = `models/${languagePair}/lex.50.50.${languagePair}.s2t.bin`;
-    const vocabFiles = [`models/${languagePair}/vocab.${vocabLanguagePair}.spm`,
-                        `models/${languagePair}/vocab.${vocabLanguagePair}.spm`];
-
-    const uniqueVocabFiles = new Set(vocabFiles);
-    console.debug("modelFile: ", modelFile);
-    console.debug("shortlistFile: ", shortlistFile);
-    console.debug("No. of unique vocabs: ", uniqueVocabFiles.size);
-    uniqueVocabFiles.forEach(item => console.debug("unique vocabFile: ", item));
-
-    try {
-      // Download the files as buffers from the given urls
-      let start = Date.now();
-      const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]);
-      const modelBuffer = downloadedBuffers[0];
-      const shortListBuffer = downloadedBuffers[1];
-
-      const downloadedVocabBuffers = [];
-      for (let item of uniqueVocabFiles.values()) {
-        downloadedVocabBuffers.push(await downloadAsArrayBuffer(item));
-      }
-      log(`${languagePair} file download took ${(Date.now() - start) / 1000} secs`);
-
-      // Construct AlignedMemory objects with downloaded buffers
-      var alignedModelMemory = constructAlignedMemoryFromBuffer(modelBuffer, 256);
-      var alignedShortlistMemory = constructAlignedMemoryFromBuffer(shortListBuffer, 64);
-      var alignedVocabsMemoryList = new Module.AlignedMemoryList;
-      downloadedVocabBuffers.forEach(item => alignedVocabsMemoryList.push_back(constructAlignedMemoryFromBuffer(item, 64)));
-
-      // Instantiate the Translation Service
-      if (translationService) translationService.delete();
-      console.debug("Creating Translation Service with config:", modelConfig);
-      translationService = new Module.Service(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
-    } catch (error) {
-      log(error);
-    }
-  }
-
-  const translate = (paragraphs) => {
-
-    // Instantiate the arguments of translate() API i.e. ResponseOptions and input (vector<string>)
-    var responseOptions = new Module.ResponseOptions();
-    let input = new Module.VectorString;
-
-    // Initialize the input
-    paragraphs.forEach(paragraph => {
-      // prevent empty paragraph - it breaks the translation
-      if (paragraph.trim() === "") {
-        return;
-      }
-      input.push_back(paragraph.trim())
-    })
-    // Access input (just for debugging)
-    console.log('Input size=', input.size());
-
-    // Translate the input, which is a vector<String>; the result is a vector<Response>
-    let result = translationService.translate(input, responseOptions);
-
-    const translatedParagraphs = [];
-    const translatedSentencesOfParagraphs = [];
-    const sourceSentencesOfParagraphs = [];
-    for (let i = 0; i < result.size(); i++) {
-      translatedParagraphs.push(result.get(i).getTranslatedText());
-      translatedSentencesOfParagraphs.push(getAllTranslatedSentencesOfParagraph(result.get(i)));
-      sourceSentencesOfParagraphs.push(getAllSourceSentencesOfParagraph(result.get(i)));
-    }
-    console.log({ translatedParagraphs });
-    console.log({ translatedSentencesOfParagraphs });
-    console.log({ sourceSentencesOfParagraphs });
-
-    responseOptions.delete();
-    input.delete();
-    return translatedParagraphs;
-  }
-
-  // This function extracts all the translated sentences from the Response and returns them.
-  const getAllTranslatedSentencesOfParagraph = (response) => {
-    const sentences = [];
-    const text = response.getTranslatedText();
-    for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
-      const utf8SentenceByteRange = response.getTranslatedSentence(sentenceIndex);
-      sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
-    }
-    return sentences;
-  }
-
-  // This function extracts all the source sentences from the Response and returns them.
-  const getAllSourceSentencesOfParagraph = (response) => {
-    const sentences = [];
-    const text = response.getOriginalText();
-    for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
-      const utf8SentenceByteRange = response.getSourceSentence(sentenceIndex);
-      sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
-    }
-    return sentences;
-  }
-
-  // This function returns a substring of text (a string). The substring is represented by
-  // byteRange (begin and end endices) within the utf-8 encoded version of the text.
-  const _getSentenceFromByteRange = (text, byteRange) => {
-    const utf8BytesView = encoder.encode(text);
-    const utf8SentenceBytes = utf8BytesView.subarray(byteRange.begin, byteRange.end);
-    return decoder.decode(utf8SentenceBytes);
-  }
-
-  document.querySelector("#load").addEventListener("click", async() => {
-    document.querySelector("#load").disabled = true;
-    const lang = document.querySelector('input[name="modellang"]:checked').value;
-    const from = lang.substring(0, 2);
-    const to = lang.substring(2, 4);
-    let start = Date.now();
-    await constructTranslationService(from, to);
-    log(`translation service ${from}${to} construction took ${(Date.now() - start) / 1000} secs`);
-    document.querySelector("#load").disabled = false;
-    //log('Model Alignment:', translationService.isAlignmentSupported());
-  });
-
-  const translateCall = () => {
-    const text = document.querySelector('#from').value;
-    const paragraphs = text.split("\n");
-    let wordCount = 0;
-    paragraphs.forEach(sentence => {
-      wordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
-    })
-    const start = Date.now();
-    const translatedParagraphs = translate(paragraphs);
-    const secs = (Date.now() - start) / 1000;
-    log(`Translation of (${wordCount}) words took ${secs} secs (${Math.round(wordCount / secs)} words per second)`);
-
-    document.querySelector('#to').value = translatedParagraphs.join("\n");
-  }
-
-  document.querySelector("#translate").addEventListener("click", () => {
-    translateCall();
-  });
-
-  document.querySelector("#from").addEventListener('keyup', function(event) {
-    if (event.keyCode === 13) {
-      translateCall();
-    }
-  });
-
-  const log = (message) => {
-    document.querySelector("#log").value += message + "\n";
-  }
-
-  const start = Date.now();
-  let moduleLoadStart;
-  var Module = {
-    preRun: [function() {
-      log(`Time until Module.preRun: ${(Date.now() - start) / 1000} secs`);
-      moduleLoadStart = Date.now();
-    }],
-    onRuntimeInitialized: function() {
-      log(`Wasm Runtime initialized (preRun -> onRuntimeInitialized) in ${(Date.now() - moduleLoadStart) / 1000} secs`);
-    }
-  };
-</script>
+<script src="bergamot.js"></script>
 <script src="bergamot-translator-worker.js"></script>
 </body>
 </html>
diff --git a/wasm/test_page/bergamot.js b/wasm/test_page/bergamot.js
new file mode 100644
index 000000000..e586b213c
--- /dev/null
+++ b/wasm/test_page/bergamot.js
@@ -0,0 +1,48 @@
+var worker;
+
+if (window.Worker) {
+    var worker = new Worker('worker.js');
+    worker.postMessage(["load_module"]);
+}
+
+const log = (message) => {
+    document.querySelector("#log").value += message + "\n";
+}
+
+document.querySelector("#translate").addEventListener("click", () => {
+    translateCall();
+});
+
+document.querySelector("#from").addEventListener('keyup', function(event) {
+    if (event.keyCode === 13) {
+        translateCall();
+    }
+});
+
+document.querySelector("#load").addEventListener("click", async() => {
+    document.querySelector("#load").disabled = true;
+    const lang = document.querySelector('input[name="modellang"]:checked').value;
+    const from = lang.substring(0, 2);
+    const to = lang.substring(2, 4);
+    let start = Date.now();
+    worker.postMessage(["load_model", from, to]);
+    document.querySelector("#load").disabled = false;
+});
+
+const translateCall = () => {
+    const text = document.querySelector('#from').value;
+    const paragraphs = text.split("\n");
+
+    worker.postMessage(["translate", paragraphs]);
+}
+
+worker.onmessage = function(e) {
+    console.debug(`Message received from worker`);
+    if (e.data[0] === 'translated_result') {
+        document.querySelector('#to').value = e.data[1].join("\n");
+        log(e.data[2]);
+    }
+    if ((e.data[0] === 'module_loaded') || (e.data[0] === 'model_loaded')) {
+        log(e.data[1]);
+    }
+}
\ No newline at end of file
diff --git a/wasm/test_page/worker.js b/wasm/test_page/worker.js
new file mode 100644
index 000000000..329081011
--- /dev/null
+++ b/wasm/test_page/worker.js
@@ -0,0 +1,243 @@
+var translationService, responseOptions, input = undefined;
+const BERGAMOT_TRANSLATOR_MODULE = "bergamot-translator-worker.js";
+
+const encoder = new TextEncoder(); // string to utf-8 converter
+const decoder = new TextDecoder(); // utf-8 to string converter
+
+const start = Date.now();
+let moduleLoadStart;
+var Module = {
+    preRun: [function() {
+        log(`Time until Module.preRun: ${(Date.now() - start) / 1000} secs`);
+        moduleLoadStart = Date.now();
+    }],
+    onRuntimeInitialized: function() {
+        log(`Wasm Runtime initialized (preRun -> onRuntimeInitialized) in ${(Date.now() - moduleLoadStart) / 1000} secs`);
+    }
+};
+
+const log = (message) => {
+  console.debug(message);
+}
+
+onmessage = async function(e) {
+    let command = e.data[0];
+    log(`Message '${command}' received from main script`);
+    let result = "";
+    if (command === 'load_module') {
+        importScripts(BERGAMOT_TRANSLATOR_MODULE);
+        result = `Translator wasm module successfully loaded`;
+        log(result);
+        log('Posting message back to main script');
+        postMessage(['module_loaded', result]);
+    }
+    else if (command === 'load_model') {
+        let start = Date.now();
+        await constructTranslationService(e.data[1], e.data[2]);
+        result = `translation model '${e.data[1]}${e.data[2]}' successfully loaded; took ${(Date.now() - start) / 1000} secs`;
+        log(result);
+        log('Posting message back to main script');
+        postMessage(['model_loaded', result]);
+    }
+    else if (command === 'translate') {
+        const inputParagraphs = e.data[1];
+        let inputWordCount = 0;
+        inputParagraphs.forEach(sentence => {
+            inputWordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
+        })
+
+        let start = Date.now();
+        const translatedParagraphs = translate(e.data[1]);
+        const secs = (Date.now() - start) / 1000;
+        result = `Translation of (${inputWordCount}) words took ${secs} secs (${Math.round(inputWordCount / secs)} words per second)`;
+        log(result);
+        log('Posting message back to main script');
+        postMessage(['translated_result', translatedParagraphs, result]);
+    }
+}
+
+// This function downloads file from a url and returns the array buffer
+const downloadAsArrayBuffer = async(url) => {
+    const response = await fetch(url);
+    if (!response.ok) {
+        throw Error(`Downloading ${url} failed: HTTP ${response.status} - ${response.statusText}`);
+    }
+    return response.arrayBuffer();
+}
+
+// This function constructs and initializes the AlignedMemory from the array buffer and alignment size
+const prepareAlignedMemoryFromBuffer = async (buffer, alignmentSize) => {
+    var byteArray = new Int8Array(buffer);
+    log(`Constructing Aligned memory with size: ${byteArray.byteLength} bytes with alignment: ${alignmentSize}`);
+    var alignedMemory = new Module.AlignedMemory(byteArray.byteLength, alignmentSize);
+    log(`Aligned memory construction done`);
+    const alignedByteArrayView = alignedMemory.getByteArrayView();
+    alignedByteArrayView.set(byteArray);
+    log(`Aligned memory initialized`);
+    return alignedMemory;
+}
+
+const constructTranslationService = async (from, to) => {
+    const languagePair = `${from}${to}`;
+
+    // Vocab files are re-used in both translation directions
+    const vocabLanguagePair = from === "en" ? `${to}${from}` : languagePair;
+
+    // Set the Model Configuration as YAML formatted string.
+    // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
+    /*const modelConfig = `models:
+        - /${languagePair}/model.${languagePair}.intgemm.alphas.bin
+        vocabs:
+        - /${languagePair}/vocab.${vocabLanguagePair}.spm
+        - /${languagePair}/vocab.${vocabLanguagePair}.spm
+        beam-size: 1
+        normalize: 1.0
+        word-penalty: 0
+        max-length-break: 128
+        mini-batch-words: 1024
+        workspace: 128
+        max-length-factor: 2.0
+        skip-cost: true
+        cpu-threads: 0
+        quiet: true
+        quiet-translation: true
+        shortlist:
+            - /${languagePair}/lex.${languagePair}.s2t
+            - 50
+            - 50
+        `;
+        */
+
+    // TODO: gemm-precision: int8shiftAlphaAll (for the models that support this)
+    // DONOT CHANGE THE SPACES BETWEEN EACH ENTRY OF CONFIG
+    const modelConfig = `beam-size: 1
+normalize: 1.0
+word-penalty: 0
+max-length-break: 128
+mini-batch-words: 1024
+workspace: 128
+max-length-factor: 2.0
+skip-cost: true
+cpu-threads: 0
+quiet: true
+quiet-translation: true
+gemm-precision: int8shift
+`;
+
+    const modelFile = `models/${languagePair}/model.${languagePair}.intgemm.alphas.bin`;
+    const shortlistFile = `models/${languagePair}/lex.50.50.${languagePair}.s2t.bin`;
+    const vocabFiles = [`models/${languagePair}/vocab.${vocabLanguagePair}.spm`,
+                        `models/${languagePair}/vocab.${vocabLanguagePair}.spm`];
+
+    const uniqueVocabFiles = new Set(vocabFiles);
+    log(`modelFile: ${modelFile}\nshortlistFile: ${shortlistFile}\nNo. of unique vocabs: ${uniqueVocabFiles.size}`);
+    uniqueVocabFiles.forEach(item => log(`unique vocabFile: ${item}`));
+
+    try {
+      // Download the files as buffers from the given urls
+        let start = Date.now();
+        const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]);
+        const modelBuffer = downloadedBuffers[0];
+        const shortListBuffer = downloadedBuffers[1];
+
+        const downloadedVocabBuffers = [];
+        for (let item of uniqueVocabFiles.values()) {
+            downloadedVocabBuffers.push(await downloadAsArrayBuffer(item));
+        }
+        log(`All files for ${languagePair} language pair took ${(Date.now() - start) / 1000} secs to download`);
+
+        // Construct AlignedMemory objects with downloaded buffers
+        let constructedAlignedMemories = await Promise.all([prepareAlignedMemoryFromBuffer(modelBuffer, 256),
+                                                                prepareAlignedMemoryFromBuffer(shortListBuffer, 64)]);
+        let alignedModelMemory = constructedAlignedMemories[0];
+        let alignedShortlistMemory = constructedAlignedMemories[1];
+        let alignedVocabsMemoryList = new Module.AlignedMemoryList;
+        for(let item of downloadedVocabBuffers) {
+            let alignedMemory = await prepareAlignedMemoryFromBuffer(item, 64);
+            alignedVocabsMemoryList.push_back(alignedMemory);
+        }
+        log(`Aligned vocab memories: ${alignedVocabsMemoryList.get(0).size()}`);
+        log(`Aligned model memory: ${alignedModelMemory.size()}`);
+        log(`Aligned shortlist memory: ${alignedShortlistMemory.size()}`);
+
+        // Instantiate the Translation Service
+        if (translationService) {
+            translationService.delete();
+            translationService = undefined;
+        }
+
+        log(`Creating Translation Service with config: ${modelConfig}`);
+        translationService = new Module.Service(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
+        if (typeof translationService === 'undefined') {
+            throw Error(`Translation Service construction failed`);
+        }
+    } catch (error) {
+        log(error);
+    }
+  }
+
+const translate = (paragraphs) => {
+    // Instantiate the arguments of translate() API i.e. ResponseOptions and input (vector<string>)
+    var responseOptions = new Module.ResponseOptions();
+    let input = new Module.VectorString;
+
+    // Initialize the input
+    paragraphs.forEach(paragraph => {
+      // prevent empty paragraph - it breaks the translation
+        if (paragraph.trim() === "") {
+           return;
+        }
+        input.push_back(paragraph.trim())
+    })
+    // Access input (just for debugging)
+    log(`Input size: ${input.size()}`);
+
+    // Translate the input, which is a vector<String>; the result is a vector<Response>
+    let result = translationService.translate(input, responseOptions);
+
+    const translatedParagraphs = [];
+    const translatedSentencesOfParagraphs = [];
+    const sourceSentencesOfParagraphs = [];
+    for (let i = 0; i < result.size(); i++) {
+        translatedParagraphs.push(result.get(i).getTranslatedText());
+        translatedSentencesOfParagraphs.push(getAllTranslatedSentencesOfParagraph(result.get(i)));
+        sourceSentencesOfParagraphs.push(getAllSourceSentencesOfParagraph(result.get(i)));
+    }
+    log({ translatedParagraphs });
+    log({ translatedSentencesOfParagraphs });
+    log({ sourceSentencesOfParagraphs });
+
+    responseOptions.delete();
+    input.delete();
+    return translatedParagraphs;
+}
+
+// This function extracts all the translated sentences from the Response and returns them.
+const getAllTranslatedSentencesOfParagraph = (response) => {
+    const sentences = [];
+    const text = response.getTranslatedText();
+    for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
+        const utf8SentenceByteRange = response.getTranslatedSentence(sentenceIndex);
+        sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
+    }
+    return sentences;
+}
+
+// This function extracts all the source sentences from the Response and returns them.
+const getAllSourceSentencesOfParagraph = (response) => {
+    const sentences = [];
+    const text = response.getOriginalText();
+    for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
+        const utf8SentenceByteRange = response.getSourceSentence(sentenceIndex);
+        sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
+    }
+    return sentences;
+}
+
+// This function returns a substring of text (a string). The substring is represented by
+// byteRange (begin and end endices) within the utf-8 encoded version of the text.
+const _getSentenceFromByteRange = (text, byteRange) => {
+    const utf8BytesView = encoder.encode(text);
+    const utf8SentenceBytes = utf8BytesView.subarray(byteRange.begin, byteRange.end);
+    return decoder.decode(utf8SentenceBytes);
+}

From ff391c6f0052c1fda54c77c6bab39ddfc9377455 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 24 Aug 2021 12:35:21 +0200
Subject: [PATCH 285/442] Updated marian submodule to latest commit of master

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 6087379f2..62bac858b 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 6087379f2ee7fb3062a82a6129ff81ca5fe56eed
+Subproject commit 62bac858bfd37060beb707d12eb9711649ea4cf6

From cafb65e0b5df4b48be10b1788c308fb827dffdb3 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 24 Aug 2021 18:03:38 +0200
Subject: [PATCH 286/442] Wasm builds without SharedArrayBuffer

---
 CMakeLists.txt | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index c58ddd4ff..a9586d8e5 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -87,9 +87,8 @@ endif()
 
 if(COMPILE_WASM)
   set(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
-  list(APPEND WASM_COMPILE_FLAGS -pthread -O3 -g2 -fPIC -mssse3 -msimd128)
+  list(APPEND WASM_COMPILE_FLAGS -O3 -g2 -fPIC -mssse3 -msimd128)
   list(APPEND WASM_COMPILE_FLAGS "SHELL:-s WASM=1" "SHELL:-s ASSERTIONS=0" "SHELL:-s DISABLE_EXCEPTION_CATCHING=1" "SHELL:-s LLD_REPORT_UNDEFINED" "SHELL:-s FORCE_FILESYSTEM=1" "SHELL:-s ALLOW_MEMORY_GROWTH=1")
-  list(APPEND WASM_COMPILE_FLAGS -Wno-error=pthreads-mem-growth)
 endif(COMPILE_WASM)
 
 # Needs to be enabled before including the folder containing tests (src/tests)

From 8e4374282a720c605bb9856dcd564d2fcec09baf Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 31 Aug 2021 15:45:14 +0200
Subject: [PATCH 287/442] Circle CI wasm artifacts for non-wormhole builds

---
 .circleci/config.yml | 41 ++++++++++++++++++++++++++++++--
 build-wasm.sh        | 56 ++++++++++++++++++++++++++++++++++----------
 2 files changed, 82 insertions(+), 15 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 69ae35686..9b14ed154 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -1,6 +1,37 @@
 version: 2.1
 jobs:
-  build:
+  build-with-wormhole:
+    docker:
+      - image: 'emscripten/emsdk:2.0.9'
+    resource_class: medium
+
+    working_directory: ~/checkout
+
+    steps:
+      - checkout
+
+      - run:
+          name: Build WASM
+          command: bash build-wasm.sh WORMHOLE
+
+      - run:
+          name: Check artifacts
+          working_directory: build-wasm
+          command: |
+            ls -all bergamot*
+            if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
+            then
+              echo "Artifacts Successfully Generated"
+            else
+              echo "Failure: Artifacts Not Present"
+              exit 1
+            fi
+
+      - store_artifacts:
+          path: "build-wasm"
+          destination: "wasm-wormhole"
+
+  build-without-wormhole:
     docker:
       - image: 'emscripten/emsdk:2.0.9'
     resource_class: medium
@@ -29,4 +60,10 @@ jobs:
 
       - store_artifacts:
           path: "build-wasm"
-          destination: "build-wasm"
+          destination: "wasm-without-wormhole"
+
+workflows:
+  build:
+      jobs:
+          - build-with-wormhole
+          - build-without-wormhole
\ No newline at end of file
diff --git a/build-wasm.sh b/build-wasm.sh
index d3cd9d1db..7da2685cf 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -1,15 +1,38 @@
 #!/usr/bin/env bash
-
-# Usage: ./build-wasm.sh
-
 set -e
 set -x
 
+# Usage
+Usage="Build translator to wasm (with/without wormhole).
+
+Usage: $(basename "$0") [WORMHOLE]
+
+    where:
+    WORMHOLE      An optional string argument
+                  - when specified on command line, builds wasm artifacts with wormhole
+                  - when not specified (the default behaviour), builds wasm artifacts without wormhole."
+
+if [ "$#" -gt 1 ]; then
+  echo "Illegal number of parameters passed"
+  echo "$Usage"
+  exit
+fi
+
+WORMHOLE=false
+
+if [ "$#" -eq 1 ]; then
+  if [ "$1" = "WORMHOLE" ]; then
+    WORMHOLE=true
+  else
+    echo "Illegal parameter passed"
+    echo "$Usage"
+    exit
+  fi
+fi
+
 # Run script from the context of the script-containing directory
 cd "$(dirname $0)"
 
-# This file replicates the instructions found in ./README.md under "Build WASM"
-
 # Prerequisite: Download and Install Emscripten using following instructions (unless the EMSDK env var is already set)
 if [ "$EMSDK" == "" ]; then
   EMSDK_UPDATE_REQUIRED=0
@@ -36,17 +59,24 @@ if [ "$EMSDK" == "" ]; then
 fi
 
 # Compile
-#    1. Create a folder where you want to build all the artifacts (`build-wasm` in this case) and compile
-if [ ! -d "build-wasm" ]; then
-  mkdir build-wasm
+#    1. Create a folder where you want to build all the artifacts and compile
+BUILD_DIRECTORY="build-wasm"
+if [ ! -d ${BUILD_DIRECTORY} ]; then
+  mkdir ${BUILD_DIRECTORY}
+fi
+cd ${BUILD_DIRECTORY}
+
+if [ "$WORMHOLE" = true ]; then
+  emcmake cmake -DCOMPILE_WASM=on ../
+else
+  emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=off ../
 fi
-cd build-wasm
-emcmake cmake -DCOMPILE_WASM=on ../
 emmake make -j2
 
 #     2. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
-bash ../wasm/patch-artifacts-enable-wormhole.sh
-
-# The artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
+if [ "$WORMHOLE" = true ]; then
+  bash ../wasm/patch-artifacts-enable-wormhole.sh
+fi
 
+# The artifacts (.js and .wasm files) will be available in the build directory
 exit 0

From 48e955c4685c6a244626416e4f0c061e4bc8ce7e Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Tue, 7 Sep 2021 19:10:41 +0100
Subject: [PATCH 288/442] BRT: Update sacrebleu to get tests back working
 (#217)

Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
---
 bergamot-translator-tests | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index ee534f750..2b1a1700e 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit ee534f7507966efe3199ac84e56bdd4b3950b736
+Subproject commit 2b1a1700e397934ba68746cb8ff9b251681d9eac

From 63120c174e3edfd664175d4a2be095d8b50a112f Mon Sep 17 00:00:00 2001
From: Andre Barbosa <abarbosa0494@gmail.com>
Date: Thu, 16 Sep 2021 12:28:40 -0300
Subject: [PATCH 289/442] QualityEstimation: Preliminary Implementation (#197)

Unifies quality estimation with an interface, refactors previously available
quality scores to fit this interface. Adds a new class of  model with Logistic
Regression powering the predictions as an implementation of said interface.
QE now provides annotations on words using subwords to word rule-based
algorithms working with space characters.

QualityEstimation
-----------------

Implementations of QE are bound together by a `QualityEstimator`
Interface.

1. The log-probabilities from the machine-translation model re-interpreted
   as quality scores are crafted as an implementation of QualityEstimator.

2. A Logistic-Regression based model is added. This class of models is
   trained supervised with scores labeled by a human annotator.
   Handcrafted features - number of words, log probs from MT model and
   statistics over the sequence are used to generate the numeric features.
   LogisticRegressor, Matrix (to hold features) are added.

The creation of an instance is switched by the `AlignedMemory` supplied
(be it loaded from the file-system or supplied as a parameter). An empty
AlignedMemory leads to quality scores from NMT while supplying weights
of a trained logistic-regression model in binary format as the contents
lead to an additional pass through the said model to provide more
refined scores.

Both the above now transform subwords into "words" using a heuristic
algorithm, scanning for spaces. This allows the client to work with "words"
to denote quality instead of subwords, as the former is more sensible to
the user.

Testing
-------

1. BRT now has two new test apps to check the QE outputs in text
  (covers subword to words) and numbers domain (covers quality scores).
  These are tested with en-et models for which QualityEstimation is
  available now, on a new input to avoid architecture/compiler issues.
2. Unit test for LogisticRegression model is added.


Docs
----

Doxygen now supports MathJax properly to render explanations for
Logistic Regressions' reductions in place to make computation more
efficient correctly.

Co-authored-by: Felipe C. Dos Santos <felipe.santos.k@gmail.com>
Co-authored-by: Jerin Philip <jerinphilip@live.in>
---
 .gitignore                                  |   3 +
 Doxyfile.in                                 |   4 +-
 bergamot-translator-tests                   |   2 +-
 doc/conf.py                                 |   2 +-
 src/tests/apps.cpp                          |  31 +++
 src/tests/apps.h                            |   6 +
 src/tests/cli.cpp                           |   6 +-
 src/tests/units/CMakeLists.txt              |   2 +-
 src/tests/units/quality_estimator_tests.cpp |  62 +++++
 src/tests/units/quality_estimator_tests.h   |   5 +
 src/translator/CMakeLists.txt               |   1 +
 src/translator/byte_array_util.cpp          |  18 ++
 src/translator/byte_array_util.h            |   2 +
 src/translator/definitions.h                |   2 +
 src/translator/parser.h                     |   2 +
 src/translator/quality_estimator.cpp        | 288 ++++++++++++++++++++
 src/translator/quality_estimator.h          | 222 +++++++++++++++
 src/translator/response.h                   |  25 +-
 src/translator/response_builder.cpp         |  16 +-
 src/translator/response_builder.h           |  12 +-
 src/translator/response_options.h           |  11 -
 src/translator/service.cpp                  |   6 +-
 src/translator/service.h                    |   7 +-
 23 files changed, 686 insertions(+), 49 deletions(-)
 create mode 100644 src/tests/units/quality_estimator_tests.cpp
 create mode 100644 src/tests/units/quality_estimator_tests.h
 create mode 100644 src/translator/quality_estimator.cpp
 create mode 100644 src/translator/quality_estimator.h

diff --git a/.gitignore b/.gitignore
index 840e69ab8..49093ba25 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,3 +20,6 @@ wasm/test_page/node_modules
 build-wasm
 models
 wasm/test_page/bergamot-translator-worker.*
+
+# VSCode
+.vscode
diff --git a/Doxyfile.in b/Doxyfile.in
index 88948e2ad..7b69eb8c5 100644
--- a/Doxyfile.in
+++ b/Doxyfile.in
@@ -1533,7 +1533,7 @@ FORMULA_TRANSPARENT    = YES
 # The default value is: NO.
 # This tag requires that the tag GENERATE_HTML is set to YES.
 
-USE_MATHJAX            = NO
+USE_MATHJAX            = YES
 
 # When MathJax is enabled you can set the default output format to be used for
 # the MathJax output. See the MathJax site (see:
@@ -1556,7 +1556,7 @@ MATHJAX_FORMAT         = HTML-CSS
 # The default value is: http://cdn.mathjax.org/mathjax/latest.
 # This tag requires that the tag USE_MATHJAX is set to YES.
 
-MATHJAX_RELPATH        = http://cdn.mathjax.org/mathjax/latest
+MATHJAX_RELPATH        = https://cdn.jsdelivr.net/npm/mathjax@3
 
 # The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax
 # extension names that should be enabled during MathJax rendering. For example
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 2b1a1700e..53c6e42a9 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 2b1a1700e397934ba68746cb8ff9b251681d9eac
+Subproject commit 53c6e42a97e512698711068d0be3c208359b1801
diff --git a/doc/conf.py b/doc/conf.py
index bffcda0cd..8a8f4224c 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -37,7 +37,7 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.imgmath',
+    'sphinx.ext.mathjax',
     'sphinx.ext.todo',
     'breathe',
     'exhale',
diff --git a/src/tests/apps.cpp b/src/tests/apps.cpp
index b42f7a495..991d3c3fd 100644
--- a/src/tests/apps.cpp
+++ b/src/tests/apps.cpp
@@ -55,6 +55,37 @@ void annotatedTextSentences(Ptr<Options> options, bool source) {
   }
 }
 
+void qualityEstimatorWords(const Ptr<Options> &options) {
+  ResponseOptions responseOptions;
+  responseOptions.qualityScores = true;
+  const Response response = translateFromStdin(options, responseOptions);
+
+  for (const auto &sentenceQualityEstimate : response.qualityScores) {
+    std::cout << "[SentenceBegin]\n";
+
+    for (const auto &wordByteRange : sentenceQualityEstimate.wordByteRanges) {
+      const string_view word(response.target.text.data() + wordByteRange.begin, wordByteRange.size());
+      std::cout << word << "\n";
+    }
+    std::cout << "[SentenceEnd]\n\n";
+  }
+}
+
+void qualityEstimatorScores(const Ptr<Options> &options) {
+  ResponseOptions responseOptions;
+  responseOptions.qualityScores = true;
+  const Response response = translateFromStdin(options, responseOptions);
+
+  for (const auto &sentenceQualityEstimate : response.qualityScores) {
+    std::cout << std::fixed << std::setprecision(3) << sentenceQualityEstimate.sentenceScore << "\n";
+
+    for (const float &wordScore : sentenceQualityEstimate.wordScores) {
+      std::cout << std::fixed << std::setprecision(3) << wordScore << "\n";
+    }
+    std::cout << "\n";
+  }
+}
+
 }  // namespace testapp
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/tests/apps.h b/src/tests/apps.h
index b380b5782..deb6a12dc 100644
--- a/src/tests/apps.h
+++ b/src/tests/apps.h
@@ -33,6 +33,12 @@ void annotatedTextWords(Ptr<Options> options, bool source = true);
 // in each line, depending on source = true or false respectively.
 void annotatedTextSentences(Ptr<Options> options, bool source = true);
 
+// Reads from stdin and translates the read content. Prints the quality words for each sentence.
+void qualityEstimatorWords(const Ptr<Options>& options);
+
+// Reads from stdin and translates the read content. Prints the quality scores for each sentence.
+void qualityEstimatorScores(const Ptr<Options>& options);
+
 }  // namespace testapp
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/tests/cli.cpp b/src/tests/cli.cpp
index 4ecb24e02..0e9469ab0 100644
--- a/src/tests/cli.cpp
+++ b/src/tests/cli.cpp
@@ -12,8 +12,10 @@ int main(int argc, char *argv[]) {
     testapp::annotatedTextSentences(options, /*source=*/false);
   } else if (mode == "test-response-source-words") {
     testapp::annotatedTextWords(options, /*source=*/true);
-  } else if (mode == "test-response-target-words") {
-    testapp::annotatedTextWords(options, /*source=*/false);
+  } else if (mode == std::string("test-quality-estimator-words")) {
+    testapp::qualityEstimatorWords(options);
+  } else if (mode == std::string("test-quality-estimator-scores")) {
+    testapp::qualityEstimatorScores(options);
   } else {
     ABORT("Unknown --mode {}. Please run a valid test", mode);
   }
diff --git a/src/tests/units/CMakeLists.txt b/src/tests/units/CMakeLists.txt
index 5c1bc003c..4794badcd 100644
--- a/src/tests/units/CMakeLists.txt
+++ b/src/tests/units/CMakeLists.txt
@@ -1,7 +1,7 @@
 # Unit tests
 set(UNIT_TESTS
     annotation_tests
-)
+    quality_estimator_tests)
 
 foreach(test ${UNIT_TESTS})
   add_executable("run_${test}" run_tests.cpp "${test}.cpp")
diff --git a/src/tests/units/quality_estimator_tests.cpp b/src/tests/units/quality_estimator_tests.cpp
new file mode 100644
index 000000000..e11c07a7b
--- /dev/null
+++ b/src/tests/units/quality_estimator_tests.cpp
@@ -0,0 +1,62 @@
+#include "quality_estimator_tests.h"
+
+#include "catch.hpp"
+#include "translator/quality_estimator.h"
+
+using namespace marian::bergamot;
+
+SCENARIO("Logistic Regressor test", "[QualityEstimator]") {
+  GIVEN("A feature matrix") {
+    const std::vector<std::vector<float> > features = {{-0.3, -0.3, 1.0, -0.183683336},
+                                                       {-0.0001, -0.0001, 1.0, -0.183683336},
+                                                       {-0.002, -0.002, 1.0, -0.183683336},
+                                                       {-0.5, -0.5, 1.0, -0.183683336},
+                                                       {-0.15, -0.2, 2.0, -0.183683336}};
+
+    LogisticRegressorQualityEstimator::Matrix featureMatrix(features.size(), features.begin()->size());
+
+    for (int i = 0; i < features.size(); ++i) {
+      for (int j = 0; j < features.begin()->size(); ++j) {
+        featureMatrix.at(i, j) = features[i][j];
+      }
+    }
+
+    AND_GIVEN("A LogistRegressor") {
+      LogisticRegressorQualityEstimator::Array coefficients = {0.99000001, 0.899999976, -0.200000003, 0.5};
+      const float intercept = {-0.300000012};
+
+      LogisticRegressorQualityEstimator::Scale scale;
+      scale.stds = {0.200000003, 0.300000012, 2.5, 0.100000001};
+      scale.means = {-0.100000001, -0.769999981, 5, -0.5};
+
+      LogisticRegressorQualityEstimator lrQE(std::move(scale), std::move(coefficients), intercept);
+
+      WHEN("It's call predict") {
+        const std::vector<float> prediction = lrQE.predict(featureMatrix);
+
+        THEN("return the prediction") {
+          CHECK(prediction == std::vector<float>{-2.14596, -4.41793, -4.403, -0.93204, -3.03343});
+        }
+      }
+
+      WHEN("LR is construct by aligned memory") {
+        const auto lrQEAlignedMemory = LogisticRegressorQualityEstimator::fromAlignedMemory(lrQE.toAlignedMemory());
+
+        WHEN("It's call predict") {
+          const std::vector<float> prediction = lrQEAlignedMemory.predict(featureMatrix);
+
+          THEN("return the prediction") {
+            CHECK(prediction == std::vector<float>{-2.14596, -4.41793, -4.403, -0.93204, -3.03343});
+          }
+        }
+      }
+    }
+  }
+}
+
+bool operator==(const std::vector<float>& value1, const std::vector<float>& value2) {
+  return std::equal(value1.begin(), value1.end(), value2.begin(), value2.end(), [](const auto& a, const auto& b) {
+    auto value = Approx(b).epsilon(0.001);
+    return a == value;
+  });
+}
diff --git a/src/tests/units/quality_estimator_tests.h b/src/tests/units/quality_estimator_tests.h
new file mode 100644
index 000000000..37cba3ef3
--- /dev/null
+++ b/src/tests/units/quality_estimator_tests.h
@@ -0,0 +1,5 @@
+#pragma once
+
+#include <vector>
+
+bool operator==(const std::vector<float>& value1, const std::vector<float>& value2);
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 34e599ba6..c0ee6be7a 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -9,6 +9,7 @@ add_library(bergamot-translator STATIC
     request.cpp 
     batcher.cpp
     response_builder.cpp
+    quality_estimator.cpp
     batch.cpp
     annotation.cpp
     service.cpp
diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index 3790a01a9..83d06acb9 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -124,12 +124,30 @@ void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
   }
 }
 
+AlignedMemory getQualityEstimatorModel(const marian::Ptr<marian::Options>& options) {
+  const auto qualityEstimatorPath = options->get<std::string>("quality", "");
+  if (qualityEstimatorPath.empty()) {
+    return {};
+  }
+  return loadFileToMemory(qualityEstimatorPath, 64);
+}
+
+AlignedMemory getQualityEstimatorModel(MemoryBundle& memoryBundle, const marian::Ptr<marian::Options>& options) {
+  if (memoryBundle.qualityEstimatorMemory.size() == 0) {
+    return getQualityEstimatorModel(options);
+  }
+
+  return std::move(memoryBundle.qualityEstimatorMemory);
+}
+
 MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options) {
   MemoryBundle memoryBundle;
   memoryBundle.model = getModelMemoryFromConfig(options);
   memoryBundle.shortlist = getShortlistMemoryFromConfig(options);
   getVocabsMemoryFromConfig(options, memoryBundle.vocabs);
   memoryBundle.ssplitPrefixFile = getSsplitPrefixFileMemoryFromConfig(options);
+  memoryBundle.qualityEstimatorMemory = getQualityEstimatorModel(options);
+
   return memoryBundle;
 }
 
diff --git a/src/translator/byte_array_util.h b/src/translator/byte_array_util.h
index 04cbf9ee9..b445b3dec 100644
--- a/src/translator/byte_array_util.h
+++ b/src/translator/byte_array_util.h
@@ -6,6 +6,8 @@ namespace bergamot {
 
 AlignedMemory loadFileToMemory(const std::string& path, size_t alignment);
 AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options);
+AlignedMemory getQualityEstimatorModel(const marian::Ptr<marian::Options>& options);
+AlignedMemory getQualityEstimatorModel(MemoryBundle& memoryBundle, const marian::Ptr<marian::Options>& options);
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options);
 AlignedMemory getSsplitPrefixFileMemoryFromConfig(marian::Ptr<marian::Options> options);
 void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index d5b874353..a0f544ded 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -29,6 +29,8 @@ struct MemoryBundle {
 
   /// @todo Not implemented yet
   AlignedMemory ssplitPrefixFile{};
+
+  AlignedMemory qualityEstimatorMemory;  ///< Byte-array of qe model (aligned to 64)
 };
 
 /// ByteRange stores indices for half-interval [begin, end) in a string. Can be
diff --git a/src/translator/parser.h b/src/translator/parser.h
index cd7096531..54aaaf86a 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -29,6 +29,8 @@ inline marian::ConfigParser createConfigParser() {
   cp.addOption<std::string>("--bergamot-mode", "Bergamot Options",
                             "Operating mode for bergamot: [wasm, native, decoder]", "native");
 
+  cp.addOption<std::string>("--quality", "Bergamot Options", "File considering Quality Estimation model");
+
   return cp;
 }
 
diff --git a/src/translator/quality_estimator.cpp b/src/translator/quality_estimator.cpp
new file mode 100644
index 000000000..936d293a4
--- /dev/null
+++ b/src/translator/quality_estimator.cpp
@@ -0,0 +1,288 @@
+#include "quality_estimator.h"
+
+namespace marian::bergamot {
+
+void UnsupervisedQualityEstimator::computeQualityScores(const Histories& histories, Response& response) const {
+  for (size_t i = 0; i < histories.size(); ++i) {
+    const Result result = histories[i]->top();
+    const Hypothesis::PtrType& hypothesis = std::get<1>(result);
+    const std::vector<float> logProbs = hypothesis->tracebackWordScores();
+    response.qualityScores.push_back(std::move(computeSentenceScores(logProbs, response.target, i)));
+  }
+}
+
+Response::SentenceQualityScore UnsupervisedQualityEstimator::computeSentenceScores(const std::vector<float>& logProbs,
+                                                                                   const AnnotatedText& target,
+                                                                                   const size_t sentenceIdx) const {
+  const std::vector<SubwordRange> wordIndices = mapWords(logProbs, target, sentenceIdx);
+
+  std::vector<float> wordScores;
+
+  for (const SubwordRange& wordIndice : wordIndices) {
+    wordScores.push_back(
+        std::accumulate(logProbs.begin() + wordIndice.begin, logProbs.begin() + wordIndice.end, float(0.0)) /
+        wordIndice.size());
+  }
+
+  const float sentenceScore =
+      std::accumulate(std::begin(wordScores), std::end(wordScores), float(0.0)) / wordScores.size();
+
+  return {wordScores, subwordToWords(wordIndices, target, sentenceIdx), sentenceScore};
+}
+
+LogisticRegressorQualityEstimator::Matrix::Matrix(const size_t rowsParam, const size_t colsParam)
+    : rows(rowsParam), cols(colsParam), data_(rowsParam * colsParam) {}
+
+LogisticRegressorQualityEstimator::Matrix::Matrix(Matrix&& other)
+    : rows(other.rows), cols(other.cols), data_(std::move(other.data_)) {}
+
+const float& LogisticRegressorQualityEstimator::Matrix::at(const size_t row, const size_t col) const {
+  return data_[row * cols + col];
+}
+
+float& LogisticRegressorQualityEstimator::Matrix::at(const size_t row, const size_t col) {
+  return data_[row * cols + col];
+}
+
+LogisticRegressorQualityEstimator::LogisticRegressorQualityEstimator(Scale&& scale, Array&& coefficients,
+                                                                     const float intercept)
+    : scale_(std::move(scale)), coefficients_(std::move(coefficients)), intercept_(intercept), coefficientsByStds_() {
+  // Pre-compute the scale operations for the linear model
+  for (int i = 0; i < coefficients_.size(); ++i) {
+    coefficientsByStds_[i] = coefficients_[i] / scale_.stds[i];
+    constantFactor_ += coefficientsByStds_[i] * scale_.means[i];
+  }
+}
+
+LogisticRegressorQualityEstimator::LogisticRegressorQualityEstimator(LogisticRegressorQualityEstimator&& other)
+    : scale_(std::move(other.scale_)),
+      coefficients_(std::move(other.coefficients_)),
+      intercept_(std::move(other.intercept_)),
+      coefficientsByStds_(std::move(other.coefficientsByStds_)),
+      constantFactor_(std::move(other.constantFactor_)) {}
+
+LogisticRegressorQualityEstimator LogisticRegressorQualityEstimator::fromAlignedMemory(
+    const AlignedMemory& alignedMemory) {
+  LOG(info, "[data] Loading Quality Estimator model from buffer");
+
+  const char* ptr = alignedMemory.begin();
+  const size_t blobSize = alignedMemory.size();
+
+  ABORT_IF(blobSize < sizeof(Header), "Quality estimation file too small");
+  const Header& header = *reinterpret_cast<const Header*>(ptr);
+
+  ABORT_IF(header.magic != BINARY_QE_MODEL_MAGIC, "Incorrect magic bytes for quality estimation file");
+  ABORT_IF(header.lrParametersDims <= 0, "The number of lr parameter dimension cannot be equal or less than zero");
+
+  const uint64_t expectedSize =
+      sizeof(Header) + (numLrParamsWithDimension_ * header.lrParametersDims + numIntercept_) * sizeof(float);
+  ABORT_IF(expectedSize != blobSize, "QE header claims file size should be {} bytes but file is {} bytes", expectedSize,
+           blobSize);
+
+  ptr += sizeof(Header);
+  const float* memoryIndex = reinterpret_cast<const float*>(ptr);
+
+  const float* stds = memoryIndex;
+  const float* means = memoryIndex += header.lrParametersDims;
+  const float* coefficientsMemory = memoryIndex += header.lrParametersDims;
+  const float intercept = *(memoryIndex += header.lrParametersDims);
+
+  Scale scale;
+
+  Array coefficients;
+
+  for (int i = 0; i < header.lrParametersDims; ++i) {
+    scale.stds[i] = *(stds + i);
+
+    ABORT_IF(scale.stds[i] == 0.0, "Invalid stds");
+
+    scale.means[i] = *(means + i);
+    coefficients[i] = *(coefficientsMemory + i);
+  }
+
+  return LogisticRegressorQualityEstimator(std::move(scale), std::move(coefficients), intercept);
+}
+
+AlignedMemory LogisticRegressorQualityEstimator::toAlignedMemory() const {
+  const size_t lrParametersDims = scale_.means.size();
+
+  const size_t lrSize =
+      (scale_.means.size() + scale_.stds.size() + coefficients_.size()) * sizeof(float) + sizeof(intercept_);
+
+  Header header = {BINARY_QE_MODEL_MAGIC, lrParametersDims};
+  marian::bergamot::AlignedMemory memory(sizeof(header) + lrSize);
+
+  char* buffer = memory.begin();
+
+  memcpy(buffer, &header, sizeof(header));
+  buffer += sizeof(header);
+
+  for (const float std : scale_.stds) {
+    memcpy(buffer, &std, sizeof(std));
+    buffer += sizeof(std);
+  }
+
+  for (const float mean : scale_.means) {
+    memcpy(buffer, &mean, sizeof(mean));
+    buffer += sizeof(mean);
+  }
+
+  for (size_t i = 0; i < lrParametersDims; ++i) {
+    const float coefficient = coefficients_[i];
+    memcpy(buffer, &coefficient, sizeof(coefficient));
+    buffer += sizeof(coefficient);
+  }
+
+  memcpy(buffer, &intercept_, sizeof(intercept_));
+  buffer += sizeof(intercept_);
+
+  return memory;
+}
+
+void LogisticRegressorQualityEstimator::computeQualityScores(const Histories& histories, Response& response) const {
+  for (size_t i = 0; i < histories.size(); ++i) {
+    const Result result = histories[i]->top();
+    const Hypothesis::PtrType& hypothesis = std::get<1>(result);
+    const std::vector<float> logProbs = hypothesis->tracebackWordScores();
+
+    response.qualityScores.push_back(std::move(computeSentenceScores(logProbs, response.target, i)));
+  }
+}
+
+Response::SentenceQualityScore LogisticRegressorQualityEstimator::computeSentenceScores(
+    const std::vector<float>& logProbs, const AnnotatedText& target, const size_t sentenceIdx) const
+
+{
+  const std::vector<SubwordRange> wordIndices = mapWords(logProbs, target, sentenceIdx);
+
+  const std::vector<float> wordScores = predict(extractFeatures(wordIndices, logProbs));
+
+  const float sentenceScore =
+      std::accumulate(std::begin(wordScores), std::end(wordScores), float(0.0)) / wordScores.size();
+
+  return {wordScores, subwordToWords(wordIndices, target, sentenceIdx), sentenceScore};
+}
+
+std::vector<float> LogisticRegressorQualityEstimator::predict(const Matrix& features) const {
+  std::vector<float> scores(features.rows);
+
+  for (int i = 0; i < features.rows; ++i) {
+    for (int j = 0; j < features.cols; ++j) {
+      scores[i] += features.at(i, j) * coefficientsByStds_[j];
+    }
+  }
+
+  /// Applies the linear model followed by a sigmoid function to each element
+
+  for (int i = 0; i < features.rows; ++i) {
+    scores[i] = std::log(1 - (1 / (1 + std::exp(-(scores[i] - constantFactor_ + intercept_)))));
+  }
+
+  return scores;
+}
+// Preprocess input data to provide correct features for the LogisticRegression model. Currently, there are
+// four features: mean of the log probability for a given word (remember that a word is made of a few subword tokens);
+// the minimum log probability of the subword level tokens that a given word is made of; the number of subword level
+// tokens that a word is made of and the overall log probability mean of the entire sequence
+LogisticRegressorQualityEstimator::Matrix LogisticRegressorQualityEstimator::extractFeatures(
+    const std::vector<SubwordRange>& wordIndices, const std::vector<float>& logProbs) const {
+  if (wordIndices.empty()) {
+    return std::move(Matrix(0, 0));
+  }
+  // The number of features (numFeatures), which is currently must be 4
+  Matrix features(wordIndices.size(), /*numFeatures =*/4);
+  size_t featureRow = 0;
+  // I_MEAN = index position in the feature vector hat represents the mean of log probability of a given word
+  // I_MIN = index position  in the feature vector that represents the minimum of log probability of a given word
+  // I_NUM_SUBWORDS = index position in the feature vector that represents the number of subwords that compose a given
+  // I_OVERALL_MEAN = index position in the feature vector that represents the overall log probability score in the
+  // entire sequence
+  const size_t I_MEAN{0}, I_MIN{1}, I_NUM_SUBWORDS{2}, I_OVERALL_MEAN{3};
+
+  float overallMean = 0.0;
+  size_t numlogProbs = 0;
+
+  for (const SubwordRange& wordIndice : wordIndices) {
+    if (wordIndice.begin == wordIndice.end) {
+      ++featureRow;
+      continue;
+    }
+
+    float minScore = std::numeric_limits<float>::max();
+
+    for (size_t i = wordIndice.begin; i < wordIndice.end; ++i) {
+      ++numlogProbs;
+      overallMean += logProbs[i];
+      features.at(featureRow, I_MEAN) += logProbs[i];
+
+      minScore = std::min<float>(logProbs[i], minScore);
+    }
+
+    features.at(featureRow, I_MEAN) /= static_cast<float>(wordIndice.size());
+    features.at(featureRow, I_MIN) = minScore;
+    features.at(featureRow, I_NUM_SUBWORDS) = wordIndice.size();
+
+    ++featureRow;
+  }
+
+  if (numlogProbs == 0) {
+    return std::move(Matrix(0, 0));
+  }
+
+  overallMean /= wordIndices.rbegin()->end;
+
+  for (int i = 0; i < features.rows; ++i) {
+    features.at(i, I_OVERALL_MEAN) = overallMean;
+  }
+
+  return std::move(features);
+}
+
+std::vector<SubwordRange> mapWords(const std::vector<float>& logProbs, const AnnotatedText& target,
+                                   const size_t sentenceIdx) {
+  // Ignore empty target
+  if ((logProbs.size() < 2) || (target.numWords(sentenceIdx) == 0)) {
+    return {};
+  }
+  // It is expected that translated words will have at least one word
+  std::vector<SubwordRange> wordIndices(/*numWords=*/1);
+
+  /// The LogisticRegressorQualityEstimator model ignores the presence of the EOS token, and hence we only need to
+  /// iterate n-1 positions.
+  for (size_t subwordIdx = 0; subwordIdx < (logProbs.size() - 1); ++subwordIdx) {
+    ByteRange subword = target.wordAsByteRange(sentenceIdx, subwordIdx);
+
+    const char firstLetter = target.text.at(subword.begin);
+
+    // if the first character is whitespace, it's a beginning of a new word
+    if (isspace(firstLetter)) {
+      wordIndices.back().end = subwordIdx;
+      wordIndices.emplace_back();
+      wordIndices.back().begin = subwordIdx;
+    }
+  }
+
+  wordIndices.back().end = logProbs.size() - 1;
+
+  return wordIndices;
+}
+
+std::vector<ByteRange> subwordToWords(const std::vector<SubwordRange>& wordIndices, const AnnotatedText& target,
+                                      const size_t sentenceIdx) {
+  std::vector<ByteRange> words;
+
+  for (const SubwordRange& wordIndice : wordIndices) {
+    size_t wordBegin = target.wordAsByteRange(sentenceIdx, wordIndice.begin).begin;
+    size_t wordEnd = target.wordAsByteRange(sentenceIdx, wordIndice.end).begin;
+
+    if (isspace(target.text.at(wordBegin))) {
+      ++wordBegin;
+    }
+
+    words.emplace_back(ByteRange{wordBegin, wordEnd});
+  }
+
+  return words;
+}
+
+}  // namespace marian::bergamot
diff --git a/src/translator/quality_estimator.h b/src/translator/quality_estimator.h
new file mode 100644
index 000000000..3d2fd68ea
--- /dev/null
+++ b/src/translator/quality_estimator.h
@@ -0,0 +1,222 @@
+#pragma once
+
+#include <array>
+#include <vector>
+
+#include "annotation.h"
+#include "response.h"
+#include "translator/history.h"
+
+namespace marian::bergamot {
+
+class QualityEstimator {
+ public:
+  /// Computes quality-scores using values from Histories and subword tokens which comes from Response
+  ///
+  ///
+  /// @param [in] histories: Histories obtained from translating a blob of source-text
+  /// @param [in] response: Partially constructed response, holding tokenization info
+  /// for source and target. The quality-scores for each sentence obtained from source-text blob
+  /// are written out as SentenceQualityEstimate into response.
+  virtual void computeQualityScores(const Histories &histories, Response &response) const = 0;
+};
+
+using SubwordRange = ByteRange;
+
+/// Unsupervised Quality Estimator model. It uses the translator model's log probabilities (log probs) as a proxy for
+/// quality scores. Then, for a given word, its quality score is computed by taking the mean of the log probs of the
+/// tokens that make it up. The sentence score is the mean of all word's log probs.
+class UnsupervisedQualityEstimator : public QualityEstimator {
+ public:
+  void computeQualityScores(const Histories &histories, Response &response) const override;
+
+ private:
+  Response::SentenceQualityScore computeSentenceScores(const std::vector<float> &logProbs, const AnnotatedText &target,
+                                                       const size_t sentenceIdx) const;
+};
+
+// ASCII and Unicode text files never start with the following 64 bits
+// It serves as a signature for quality estimator binary files
+constexpr std::uint64_t BINARY_QE_MODEL_MAGIC = 0x78cc336f1d54b180;
+
+/// LogisticRegressorQualityEstimator model implementation through a linear regressor + sigmoid function. Simply
+/// speaking, an LR model depends on features to be scaled, so it contains four elements of data: a vector of
+/// coefficients and an intercept (which represents the linear model) and a vector of means and stds (which are
+/// necessary for feature scaling). These variables are firstly initialized by parsing a file (which comes from
+/// `fromAlignedMemory`), and then they are used to build a model representation
+class LogisticRegressorQualityEstimator : public QualityEstimator {
+ public:
+  using Array = std::array<float, /*LRParamsDims = */ 4>;
+
+  struct Header {
+    /// Binary QE File magic number
+    uint64_t magic;
+    /// Length of lr parameters stds, means and coefficients.
+    uint64_t lrParametersDims;
+  };
+  /// Struct that contains information for applying standard scaling
+  struct Scale {
+    /// Array of standard deviations of feature values. Its length will be equals as featureDims
+    Array stds;
+    /// Array of mean of feature values. Its length will be equals as featureDims
+    Array means;
+  };
+  /// Matrix is an internal data structure that was created only to be used in LogisticRegressorQualityEstimator
+  /// methods. It intends to represent a matrix, so it receives row and column values as a constructor. Furthermore, the
+  /// method `at` can access specific data given a row and col position.
+  class Matrix {
+   public:
+    /// Number of rows
+    const size_t rows;
+    /// Number of columns
+    const size_t cols;
+
+    /// @param [in] rowsParam: number of rows in the Matrix
+    /// @param [in] colsParam: number of columns in the Matrix
+    Matrix(const size_t rowsParam, const size_t colsParam);
+    /// Move constructor
+    Matrix(Matrix &&other);
+
+    /// Return data value given a row and col position
+    /// @param [in] row: row position
+    /// @param [in] col: col position
+    const float &at(const size_t row, const size_t col) const;
+    float &at(const size_t row, const size_t col);
+
+   private:
+    std::vector<float> data_;
+  };
+  /// Logistic Regressor constructor. It creates a LR model that fits proper for the QualityEstimator use.
+  ///
+  ///
+  /// @param [in] scale: Array of stds and means that can be used to apply standard scaling in the features
+  /// @param [in] coefficients: coefficient values of linear part of LR model
+  /// @param [in] intercept: intercept value of the linear part of LR model
+  LogisticRegressorQualityEstimator(Scale &&scale, Array &&coefficients, const float intercept);
+
+  /// Move constructor
+  LogisticRegressorQualityEstimator(LogisticRegressorQualityEstimator &&other);
+
+  /// Binary file parser which came from AlignedMemory
+  /// It's expected from AlignedMemory the following structure:
+  /// - -a header with the number of parameters dimensions
+  /// - -a vector of standard deviations of features
+  /// - -a vector of means of features
+  /// - -a vector of coefficients
+  /// - -a intercept value
+  static LogisticRegressorQualityEstimator fromAlignedMemory(const AlignedMemory &alignedMemory);
+  AlignedMemory toAlignedMemory() const;
+
+  void computeQualityScores(const Histories &histories, Response &response) const override;
+  /// Given an input matrix \f$\mathbf{X}\f$, the usual Logistic Regression calculus can be seen as the following:
+  ///
+  /// 1) Standardize it, returning in \f$\mathbf{Z} = \frac{(\mathbf{X}-\mu)}{\sigma}\f$, where \f$\mu\f$ stands for the
+  /// mean vector and \f$\sigma\f$ represents the standard deviation
+  ///
+  /// 2) Then, we apply \f$\sum_{i=1}^{D}{ w_i z_i}\f$, where \f$D\f$ is the dimension (i.e. the number of features) and
+  /// \f$w\f$ is the model vector with learnt weights
+  ///
+  /// 3) We apply the sigmoid function to the result
+  ///
+  /// Notice, however, that for the first two steps we can do the following:
+  ///
+  /// \f{align*}{
+  /// \sum_{i=1}^{D}{ w_i z_i} &= \mathbf{w^T}\left(\mathbf{\sigma^{-1}} \odot (\mathbf{x} - \mathbf{\mu})\right) \text{
+  /// //
+  /// we are just vectoring step 1}\\
+  ///      &= \sum_{i=1}^{D}{\sigma_i^{-1} w_i (x_i - \mu_i)} \\
+  ///      &= \sum_{i=1}^{D}{\sigma_i^{-1} w_ix_i - \sigma_i^{-1} w_i \mu_i} \\
+  ///      &= \sum_{i=1}^{D}{\left(\sigma_i^{-1} w_i\right)x_i - \left(\sigma_i^{-1} w_i \mu_i\right)}
+  /// \f}
+  /// Then, \f$(\sigma_i^{-1} w_i \mu_i)\f$ can be precomputed without any dependence on inference data. This is done by
+  /// the variable \f$\textit{constantFactor_}\f$ and \f$\textit{intercept_}\f$ in the code.
+  ///
+  /// @param [in] features: A Matrix struct of features. For a defintion what features currently means, please refer to
+  /// `extractFeatures` method in `quality_estimator.cpp`
+  std::vector<float> predict(const Matrix &features) const;
+
+ private:
+  Scale scale_;
+  Array coefficients_;
+  float intercept_;
+  Array coefficientsByStds_;
+  float constantFactor_ = 0.0;
+
+  // Number of parameters with dimension - Scale(stds, means) and coefficients
+  static constexpr const size_t numLrParamsWithDimension_ = 3;
+  // Number of intercept values
+  static constexpr const size_t numIntercept_ = 1;
+
+  /// construct the struct SentenceQualityEstimate
+  /// @param [in] logProbs: the log probabilities given by an translation model
+  /// @param [in] target: AnnotatedText target value
+  /// @param [in] sentenceIdx: the id of a candidate sentence
+  Response::SentenceQualityScore computeSentenceScores(const std::vector<float> &logProbs, const AnnotatedText &target,
+                                                       const size_t sentenceIdx) const;
+
+  Matrix extractFeatures(const std::vector<SubwordRange> &wordIndices, const std::vector<float> &logProbs) const;
+};
+
+/// createQualityEstimator model takes an `AlignedMemory`, which is the return from `getQualityEstimatorModel`.
+///
+/// `getQualityEstimatorModel` contains two different implementations, one when the `quality` argument has some value as
+/// a possible `Options` and where it does not.
+///
+/// If a non `quality` option is provided, then by default, it uses the UnsupervisedQualityEstimator implementation.
+///
+/// If a value is passed to the `quality` argument, the model file is read and converted into an `AlignedMemory`
+/// structure, which instantiates a QualityEstimator object.
+
+/// @param [in] qualityFileMemory: An `AlignedMemory` which is created by parsing a QE model binary file through
+/// getQualityEstimatorModel
+inline std::shared_ptr<QualityEstimator> createQualityEstimator(const AlignedMemory &qualityFileMemory) {
+  // If no quality file return simple model
+  if (qualityFileMemory.size() == 0) {
+    return std::make_shared<UnsupervisedQualityEstimator>();
+  }
+
+  return std::make_shared<LogisticRegressorQualityEstimator>(
+      LogisticRegressorQualityEstimator::fromAlignedMemory(qualityFileMemory));
+}
+
+/// A word is composed of multiple subtokens. Entire words are tokens splitted by whitespace.
+/// This method takes a sequence of sublevel tokens (given by AnnotatedText) as well aligned with their log
+/// probabilities and conflate them to their respective words
+/// The return of this function is a SubwordRange (an alias of ByteRange) vector where each value corresponds to a word
+/// id and its content represent the range of subword value that compose a given word
+///
+/// If a translated sentence does not contain any alphanumeric character (therefore, it is made basically of the EOS
+/// token), this method ignores it and returns an empty ByteRange vector of words.
+///
+/// Examples:
+/// Suppose that you have the following source target (A): marian is a good translation service and the translate
+/// service gives you the following sentence (B):
+/// service gives you the following sentence (B):
+///
+/// ma(0.15) ri(0.15) an(0.2) es(0.3) un(0.1) bu(0.3) en(0.2) ser(0.1) vi(0.2) cio(0.4) de(0.1) tra(0.4) du(0.2)
+/// cción(0.1)
+///
+/// The numbers that the words follow represent the logProb of each BPE token.
+///
+/// Then, the result would be something like:
+/// a vector where each position corresponds to the SubwordRange of the following words: marian
+/// es un buen servicio de traducción. Hence, its length is 7. The value of the first element would be [0,3)
+
+/// @param [in] logProbs: the log probabilities of byte pair encodings (BPE) that comes from the tracebackWordScores
+/// method (which belongs to hypothesis.h in Marian)
+/// @param [in] target: AnnotatedText target value
+/// @param [in] sentenceIdx: the id of a candidate sentence
+std::vector<SubwordRange> mapWords(const std::vector<float> &logProbs, const AnnotatedText &target,
+                                   const size_t sentenceIdx);
+
+/// Given a vector of subwordRanges, it maps the elements to be real words rather than sublevel tokens. The words are
+/// represented through ByteRanges.
+
+/// @param [in] wordIndices: A vector where each element correspond to the index of a real word and its values are
+/// represented by the SubwordRanges (which are aliases of ByteRanges) which represents sublevel token positions
+/// @param [in] target: AnnotatedText target value
+/// @param [in] sentenceIdx: the id of a candidate sentence
+std::vector<ByteRange> subwordToWords(const std::vector<SubwordRange> &wordIndices, const AnnotatedText &target,
+                                      const size_t sentenceIdx);
+
+}  // namespace marian::bergamot
diff --git a/src/translator/response.h b/src/translator/response.h
index 2355f5225..b77fbb633 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -26,14 +26,6 @@ struct Point {
 /// Alignment is a sparse matrix, where Points represent entries with values.
 typedef std::vector<Point> Alignment;
 
-/// -loglikelhoods of the sequence components as proxy to quality.
-struct Quality {
-  /// Certainty/uncertainty score for sequence.
-  float sequence;
-  /// Certainty/uncertainty for each word in the sequence.
-  std::vector<float> word;
-};
-
 /// Response holds AnnotatedText(s) of source-text and translated text,
 /// alignment information between source and target sub-words and sentences.
 ///
@@ -41,6 +33,19 @@ struct Quality {
 /// sentences boundaries, which are required to interpret Quality and
 /// Alignment (s) at the moment.
 struct Response {
+  /// SentenceQualityScore  contains the quality data of a given translated sentence.
+  /// It includes the confidence (proxied by log probabilities) of each decoded word
+  /// (higher logprobs imply better-translated words), the ByteRanges of each term,
+  /// and logprobs of the whole sentence, represented as the mean word scores.
+  struct SentenceQualityScore {
+    /// Quality score of each translated word
+    std::vector<float> wordScores;
+    /// Each word position in the translated text
+    std::vector<ByteRange> wordByteRanges;
+    /// Whole sentence quality score (it is composed by the mean of its words)
+    float sentenceScore = 0.0;
+  };
+
   /// Convenience function to obtain number of units translated. Same as
   /// `.source.numSentences()` and `.target.numSentences().` The processing of a
   /// text of into sentences are handled internally, and this information can be
@@ -54,11 +59,11 @@ struct Response {
   /// translated text and annotations of (sub-)words and sentences.
   AnnotatedText target;
 
-  /// -logprob of each word and negative log likelihood of sequence (sentence)
+  /// logprob of each word and  the total sequence (sentence)
   /// normalized by length, for each sentence processed by the translator.
   /// Indices correspond to ranges accessible through respective Annotation on
   /// source or target.
-  std::vector<Quality> qualityScores;
+  std::vector<SentenceQualityScore> qualityScores;
 
   /// Alignments between source and target. Each Alignment is a
   /// sparse matrix representation with indices corresponding
diff --git a/src/translator/response_builder.cpp b/src/translator/response_builder.cpp
index 2944de53a..d51fbbf57 100644
--- a/src/translator/response_builder.cpp
+++ b/src/translator/response_builder.cpp
@@ -6,21 +6,7 @@ namespace marian {
 namespace bergamot {
 
 void ResponseBuilder::buildQualityScores(Histories &histories, Response &response) {
-  std::vector<Quality> qualityScores;
-  for (auto &history : histories) {
-    // TODO(jerin): Change hardcode of nBest = 1
-    NBestList onebest = history->nBest(1);
-
-    Result result = onebest[0];  // Expecting only one result;
-    Words words = std::get<0>(result);
-    auto hyp = std::get<1>(result);
-    // Quality scores: Sequence level is obtained as normalized path scores.
-    // Word level using hypothesis traceback. These are most-likely
-    // logprobs.
-    auto normalizedPathScore = std::get<2>(result);
-    auto wordQualities = hyp->tracebackWordScores();
-    response.qualityScores.push_back(Quality{normalizedPathScore, wordQualities});
-  }
+  qualityEstimator_.computeQualityScores(histories, response);
 }
 
 void ResponseBuilder::buildAlignments(Histories &histories, Response &response) {
diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
index bee189516..614c7c282 100644
--- a/src/translator/response_builder.h
+++ b/src/translator/response_builder.h
@@ -1,7 +1,10 @@
 #ifndef SRC_BERGAMOT_RESPONSE_BUILDER_H_
 #define SRC_BERGAMOT_RESPONSE_BUILDER_H_
 
+#include <optional>
+
 #include "data/types.h"
+#include "quality_estimator.h"
 #include "response.h"
 #include "response_options.h"
 #include "vocabs.h"
@@ -24,12 +27,15 @@ class ResponseBuilder {
   /// or not in the response and any additional configurable parameters.
   /// @param [in] vocabs: marian vocab object (used in decoding)
   /// @param [in] callback: callback with operates on the constructed Response.
+  /// @param [in] qualityEstimator: the QualityEstimator model that can be used
+  /// to provide translation quality probability.
   ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source, Vocabs &vocabs,
-                  std::function<void(Response &&)> callback)
+                  std::function<void(Response &&)> callback, const QualityEstimator &qualityEstimator)
       : responseOptions_(responseOptions),
         source_(std::move(source)),
         vocabs_(vocabs),
-        callback_(std::move(callback)) {}
+        callback_(std::move(callback)),
+        qualityEstimator_(qualityEstimator) {}
 
   /// Constructs and sets the promise of a Response object from obtained
   /// histories after translating.
@@ -86,6 +92,8 @@ class ResponseBuilder {
   std::function<void(Response &&)> callback_;  //  To be set when callback triggered and
                                                //  after Response constructed.
   AnnotatedText source_;
+
+  const QualityEstimator &qualityEstimator_;
 };
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/translator/response_options.h b/src/translator/response_options.h
index b74f5782a..92737a414 100644
--- a/src/translator/response_options.h
+++ b/src/translator/response_options.h
@@ -13,16 +13,6 @@ enum ConcatStrategy {
   SPACE
 };
 
-enum QualityScoreType {
-  /// Provide a free quality-score that comes with the machine-translation model
-  /// itself.
-  FREE,
-
-  /// An expensive quality-score that runs additional computations to determine
-  /// quality of an output.
-  EXPENSIVE
-};
-
 /// ResponseOptions dictate how to construct a Response for an input string of
 /// text to be translated.
 struct ResponseOptions {
@@ -40,7 +30,6 @@ struct ResponseOptions {
   /// matrix).
   float alignmentThreshold{0.2f};
 
-  QualityScoreType qualityScoreType{QualityScoreType::FREE};
   ConcatStrategy concatStrategy{ConcatStrategy::FAITHFUL};
 };
 
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 26901debc..f5996aa45 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -4,6 +4,7 @@
 #include <utility>
 
 #include "batch.h"
+#include "byte_array_util.h"
 #include "definitions.h"
 
 namespace marian {
@@ -17,7 +18,8 @@ Service::Service(Ptr<Options> options, MemoryBundle memoryBundle)
       batcher_(options),
       numWorkers_(std::max<int>(1, options->get<int>("cpu-threads"))),
       modelMemory_(std::move(memoryBundle.model)),
-      shortlistMemory_(std::move(memoryBundle.shortlist))
+      shortlistMemory_(std::move(memoryBundle.shortlist)),
+      qualityEstimator_(createQualityEstimator(getQualityEstimatorModel(memoryBundle, options)))
 #ifdef WASM_COMPATIBLE_SOURCE
       ,
       blocking_translator_(DeviceId(0, DeviceType::cpu), vocabs_, options_, &modelMemory_, &shortlistMemory_)
@@ -71,7 +73,7 @@ void Service::queueRequest(std::string &&input, std::function<void(Response &&)>
 
   text_processor_.process(std::move(input), source, segments);
 
-  ResponseBuilder responseBuilder(responseOptions, std::move(source), vocabs_, std::move(callback));
+  ResponseBuilder responseBuilder(responseOptions, std::move(source), vocabs_, std::move(callback), *qualityEstimator_);
   Ptr<Request> request = New<Request>(requestId_++, std::move(segments), std::move(responseBuilder));
 
   batcher_.addWholeRequest(request);
diff --git a/src/translator/service.h b/src/translator/service.h
index 0a3658048..3a3d616fc 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -3,6 +3,7 @@
 
 #include "batch_translator.h"
 #include "data/types.h"
+#include "quality_estimator.h"
 #include "response.h"
 #include "response_builder.h"
 #include "text_processor.h"
@@ -46,7 +47,7 @@ class Service {
   /// the given bytearray memories.
   /// @param options Marian options object
   /// @param memoryBundle holds all byte-array memories. Can be a set/subset of
-  /// model, shortlist, vocabs and ssplitPrefixFile bytes. Optional.
+  /// model, shortlist, vocabs and ssplitPrefixFile or QualityEstimation bytes. Optional.
   explicit Service(Ptr<Options> options, MemoryBundle memoryBundle = {});
 
   /// Construct Service from a string configuration. If memoryBundle is empty, Service is
@@ -54,7 +55,7 @@ class Service {
   /// the given bytearray memories.
   /// @param [in] config string parsable as YAML expected to adhere with marian config
   /// @param [in] memoryBundle holds all byte-array memories. Can be a set/subset of
-  /// model, shortlist, vocabs and ssplitPrefixFile bytes. Optional.
+  /// model, shortlist, vocabs and ssplitPrefixFile or qualityEstimation bytes. Optional.
   explicit Service(const std::string &config, MemoryBundle memoryBundle = {})
       : Service(parseOptions(config, /*validate=*/false), std::move(memoryBundle)) {}
 
@@ -116,6 +117,8 @@ class Service {
   /// Shortlist memory passed as bytes.
   AlignedMemory shortlistMemory_;  // ORDER DEPENDENCY (translators_)
 
+  std::shared_ptr<QualityEstimator> qualityEstimator_;
+
   /// Stores requestId of active request. Used to establish
   /// ordering among requests and logging/book-keeping.
 

From cf541c68f9b43bce8c68e2292007a1573cfaa38e Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Tue, 21 Sep 2021 18:10:40 +0100
Subject: [PATCH 290/442] Multiple TranslationModels Implementation (#210)

For outbound translation, we require having multiple models in the
inventory at the same time and abstracting the "how-to-translate"
using a model out.

Reorganization: TranslationModel + Service. The new entity which
contains everything required to translate in one direction is
`TranslationModel`. The how-to-translate blocking single-threaded mode
of operation or async multi-threaded mode of operation is decoupled as
`BlockingService` and `AsyncService`. There is a new regression-test
using multiple models in conjunction added, also serving as
a demonstration for using multiple models in Outbound Translation.

WASM: WebAssembly due to the inability to use threads uses
`BlockingService.  Bindings are provided with a new API to work with a
Service, and multiple TranslationModels which the client (JS extension)
can inventory and maintain.  Ownership of a given `TranslationModel` is
shared while translations using the model are active in the internal
mechanism.

Config-Parsing: So far bergamot-translator has been hijacking marian's
config-parsing mechanisms. However, in order to support multiple models,
it has become impractical to continue this approach and a new
config-parsing that is bergamot specific is provisioned for
command-line applications constituting tests. The original marian
config-parsing tooling is only associated with a subset of
`TranslationModel` now. The new config-parsing for the library manages
workers and other common options (tentatively).

There is a known issue of: Inefficient placing of workspaces, leading to
more memory usage than what's necessary. This is to be fixed trickling
down from marian-dev in a later pull request.

This PR also brings in BRT changes which fix speed-tests that were
broken and also fixes some QE outputs which were different due to not
using shortlist.
---
 app/bergamot.cpp                              |  26 ++-
 app/cli.h                                     |  46 ++--
 bergamot-translator-tests                     |   2 +-
 src/tests/apps.cpp                            |  68 ++++--
 src/tests/apps.h                              |  14 +-
 src/tests/cli.cpp                             |  54 +++--
 src/translator/CMakeLists.txt                 |   7 +-
 src/translator/aggregate_batching_pool.cpp    |  34 +++
 src/translator/aggregate_batching_pool.h      |  68 ++++++
 src/translator/batch_translator.cpp           | 128 -----------
 src/translator/batch_translator.h             |  57 -----
 .../{batcher.cpp => batching_pool.cpp}        |  15 +-
 src/translator/{batcher.h => batching_pool.h} |  23 +-
 src/translator/definitions.h                  |   3 +
 src/translator/parser.cpp                     | 170 +++++++++++++++
 src/translator/parser.h                       | 120 +++++------
 src/translator/request.h                      |   6 +-
 src/translator/response_builder.h             |   2 +-
 src/translator/service.cpp                    |  99 ++++-----
 src/translator/service.h                      | 198 ++++++++----------
 src/translator/text_processor.cpp             |   8 +-
 src/translator/text_processor.h               |   6 +-
 src/translator/threadsafe_batcher.cpp         |  38 ----
 src/translator/threadsafe_batcher.h           |  57 -----
 src/translator/threadsafe_batching_pool.cpp   |  49 +++++
 src/translator/threadsafe_batching_pool.h     |  71 +++++++
 src/translator/translation_model.cpp          | 173 +++++++++++++++
 src/translator/translation_model.h            | 122 +++++++++++
 wasm/bindings/service_bindings.cpp            |  45 ++--
 29 files changed, 1068 insertions(+), 641 deletions(-)
 create mode 100644 src/translator/aggregate_batching_pool.cpp
 create mode 100644 src/translator/aggregate_batching_pool.h
 delete mode 100644 src/translator/batch_translator.cpp
 delete mode 100644 src/translator/batch_translator.h
 rename src/translator/{batcher.cpp => batching_pool.cpp} (83%)
 rename src/translator/{batcher.h => batching_pool.h} (63%)
 create mode 100644 src/translator/parser.cpp
 delete mode 100644 src/translator/threadsafe_batcher.cpp
 delete mode 100644 src/translator/threadsafe_batcher.h
 create mode 100644 src/translator/threadsafe_batching_pool.cpp
 create mode 100644 src/translator/threadsafe_batching_pool.h
 create mode 100644 src/translator/translation_model.cpp
 create mode 100644 src/translator/translation_model.h

diff --git a/app/bergamot.cpp b/app/bergamot.cpp
index 19dea1fcf..bffbbb112 100644
--- a/app/bergamot.cpp
+++ b/app/bergamot.cpp
@@ -1,18 +1,22 @@
 #include "cli.h"
 
 int main(int argc, char *argv[]) {
-  auto cp = marian::bergamot::createConfigParser();
-  auto options = cp.parseOptions(argc, argv, true);
-  const std::string mode = options->get<std::string>("bergamot-mode");
+  marian::bergamot::ConfigParser configParser;
+  configParser.parseArgs(argc, argv);
+  auto &config = configParser.getConfig();
   using namespace marian::bergamot;
-  if (mode == "wasm") {
-    app::wasm(options);
-  } else if (mode == "native") {
-    app::native(options);
-  } else if (mode == "decoder") {
-    app::decoder(options);
-  } else {
-    ABORT("Unknown --mode {}. Use one of: {wasm,native,decoder}", mode);
+  switch (config.opMode) {
+    case OpMode::APP_WASM:
+      app::wasm(config);
+      break;
+    case OpMode::APP_NATIVE:
+      app::native(config);
+      break;
+    case OpMode::APP_DECODER:
+      app::decoder(config);
+      break;
+    default:
+      break;
   }
   return 0;
 }
diff --git a/app/cli.h b/app/cli.h
index 4afe8b9aa..9cb12dd28 100644
--- a/app/cli.h
+++ b/app/cli.h
@@ -34,34 +34,40 @@ namespace app {
 /// * Output: written to stdout as translations for the sentences supplied in corresponding lines
 ///
 /// @param [options]: Options to translate passed down to marian through Options.
-void wasm(Ptr<Options> options) {
+void wasm(const CLIConfig &config) {
   // Here, we take the command-line interface which is uniform across all apps. This is parsed into Ptr<Options> by
   // marian. However, mozilla does not allow a Ptr<Options> constructor and demands an std::string constructor since
   // std::string isn't marian internal unlike Ptr<Options>. Since this std::string path needs to be tested for mozilla
   // and since this class/CLI is intended at testing mozilla's path, we go from:
   //
-  // cmdline -> Ptr<Options> -> std::string -> Service(std::string)
+  // cmdline -> Ptr<Options> -> std::string -> TranslationModel(std::string)
   //
   // Overkill, yes.
 
-  std::string config = options->asYamlString();
-  Service model(config);
+  const std::string &modelConfigPath = config.modelConfigPaths.front();
+
+  Ptr<Options> options = parseOptionsFromFilePath(modelConfigPath);
+  MemoryBundle memoryBundle = getMemoryBundleFromConfig(options);
+
+  BlockingService::Config serviceConfig;
+  BlockingService service(serviceConfig);
+
+  std::shared_ptr<TranslationModel> translationModel =
+      std::make_shared<TranslationModel>(options->asYamlString(), std::move(memoryBundle));
 
   ResponseOptions responseOptions;
   std::vector<std::string> texts;
 
-#ifdef WASM_COMPATIBLE_SOURCE
   // Hide the translateMultiple operation
   for (std::string line; std::getline(std::cin, line);) {
     texts.emplace_back(line);
   }
 
-  auto results = model.translateMultiple(std::move(texts), responseOptions);
+  auto results = service.translateMultiple(translationModel, std::move(texts), responseOptions);
 
   for (auto &result : results) {
     std::cout << result.getTranslatedText() << std::endl;
   }
-#endif
 }
 
 /// Application used to benchmark with marian-decoder from time-to-time. The implementation in this repository follows a
@@ -82,9 +88,13 @@ void wasm(Ptr<Options> options) {
 /// * Output: to stdout, translations of the sentences supplied via stdin in corresponding lines
 ///
 /// @param [in] options: constructed from command-line supplied arguments
-void decoder(Ptr<Options> options) {
+void decoder(const CLIConfig &config) {
   marian::timer::Timer decoderTimer;
-  Service service(options);
+  AsyncService::Config asyncConfig{config.numWorkers};
+  AsyncService service(asyncConfig);
+  auto options = parseOptionsFromFilePath(config.modelConfigPaths.front());
+  MemoryBundle memoryBundle;
+  Ptr<TranslationModel> translationModel = service.createCompatibleModel(options, std::move(memoryBundle));
   // Read a large input text blob from stdin
   std::ostringstream std_input;
   std_input << std::cin.rdbuf();
@@ -95,14 +105,15 @@ void decoder(Ptr<Options> options) {
   std::future<Response> responseFuture = responsePromise.get_future();
   auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
 
-  service.translate(std::move(input), std::move(callback));
+  service.translate(translationModel, std::move(input), std::move(callback));
   responseFuture.wait();
   const Response &response = responseFuture.get();
 
   for (size_t sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
     std::cout << response.target.sentence(sentenceIdx) << "\n";
   }
-  LOG(info, "Total time: {:.5f}s wall", decoderTimer.elapsed());
+
+  std::cerr << "Total time: " << std::setprecision(5) << decoderTimer.elapsed() << "s wall" << std::endl;
 }
 
 /// Command line interface to the test the features being developed as part of bergamot C++ library on native platform.
@@ -114,16 +125,19 @@ void decoder(Ptr<Options> options) {
 /// * Output: to stdout, translation of the source text faithful to source structure.
 ///
 /// @param [in] options: options to build translator
-void native(Ptr<Options> options) {
+void native(const CLIConfig &config) {
+  AsyncService::Config asyncConfig{config.numWorkers};
+  AsyncService service(asyncConfig);
+
+  auto options = parseOptionsFromFilePath(config.modelConfigPaths.front());
   // Prepare memories for bytearrays (including model, shortlist and vocabs)
   MemoryBundle memoryBundle;
-
-  if (options->get<bool>("bytearray")) {
+  if (config.byteArray) {
     // Load legit values into bytearrays.
     memoryBundle = getMemoryBundleFromConfig(options);
   }
 
-  Service service(options, std::move(memoryBundle));
+  Ptr<TranslationModel> translationModel = service.createCompatibleModel(options, std::move(memoryBundle));
 
   // Read a large input text blob from stdin
   std::ostringstream std_input;
@@ -137,7 +151,7 @@ void native(Ptr<Options> options) {
   std::future<Response> responseFuture = responsePromise.get_future();
   auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
 
-  service.translate(std::move(input), std::move(callback), responseOptions);
+  service.translate(translationModel, std::move(input), std::move(callback), responseOptions);
   responseFuture.wait();
   Response response = responseFuture.get();
 
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 53c6e42a9..9dc3c5e9a 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 53c6e42a97e512698711068d0be3c208359b1801
+Subproject commit 9dc3c5e9a1027c1d6b4a467a27bdff16d0d6a006
diff --git a/src/tests/apps.cpp b/src/tests/apps.cpp
index 991d3c3fd..63febfaf0 100644
--- a/src/tests/apps.cpp
+++ b/src/tests/apps.cpp
@@ -2,30 +2,25 @@
 
 namespace marian {
 namespace bergamot {
-namespace testapp {
-
-// Utility function, common for all testapps.
-Response translateFromStdin(Ptr<Options> options, ResponseOptions responseOptions) {
-  // Prepare memories for bytearrays (including model, shortlist and vocabs)
-  MemoryBundle memoryBundle;
 
-  if (options->get<bool>("bytearray")) {
-    // Load legit values into bytearrays.
-    memoryBundle = getMemoryBundleFromConfig(options);
-  }
-
-  Service service(options, std::move(memoryBundle));
+namespace {
 
+std::string readFromStdin() {
   // Read a large input text blob from stdin
   std::ostringstream inputStream;
   inputStream << std::cin.rdbuf();
   std::string input = inputStream.str();
+  return input;
+}
 
+// Utility function, common for all testapps.
+Response translateForResponse(AsyncService &service, Ptr<TranslationModel> model, std::string &&source,
+                              ResponseOptions responseOptions) {
   std::promise<Response> responsePromise;
   std::future<Response> responseFuture = responsePromise.get_future();
 
   auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
-  service.translate(std::move(input), callback, responseOptions);
+  service.translate(model, std::move(source), callback, responseOptions);
 
   responseFuture.wait();
 
@@ -33,10 +28,15 @@ Response translateFromStdin(Ptr<Options> options, ResponseOptions responseOption
   return response;
 }
 
-void annotatedTextWords(Ptr<Options> options, bool source) {
+}  // namespace
+
+namespace testapp {
+
+void annotatedTextWords(AsyncService &service, Ptr<TranslationModel> model, bool sourceSide) {
   ResponseOptions responseOptions;
-  Response response = translateFromStdin(options, responseOptions);
-  AnnotatedText &annotatedText = source ? response.source : response.target;
+  std::string source = readFromStdin();
+  Response response = translateForResponse(service, model, std::move(source), responseOptions);
+  AnnotatedText &annotatedText = sourceSide ? response.source : response.target;
   for (size_t s = 0; s < annotatedText.numSentences(); s++) {
     for (size_t w = 0; w < annotatedText.numWords(s); w++) {
       std::cout << (w == 0 ? "" : "\t");
@@ -46,19 +46,39 @@ void annotatedTextWords(Ptr<Options> options, bool source) {
   }
 }
 
-void annotatedTextSentences(Ptr<Options> options, bool source) {
+void annotatedTextSentences(AsyncService &service, Ptr<TranslationModel> model, bool sourceSide) {
   ResponseOptions responseOptions;
-  Response response = translateFromStdin(options, responseOptions);
-  AnnotatedText &annotatedText = source ? response.source : response.target;
+  std::string source = readFromStdin();
+  Response response = translateForResponse(service, model, std::move(source), responseOptions);
+  AnnotatedText &annotatedText = sourceSide ? response.source : response.target;
   for (size_t s = 0; s < annotatedText.numSentences(); s++) {
     std::cout << annotatedText.sentence(s) << "\n";
   }
 }
 
-void qualityEstimatorWords(const Ptr<Options> &options) {
+void forwardAndBackward(AsyncService &service, std::vector<Ptr<TranslationModel>> &models) {
+  ABORT_IF(models.size() != 2, "Forward and backward test needs two models.");
+  ResponseOptions responseOptions;
+  std::string source = readFromStdin();
+  Response forwardResponse = translateForResponse(service, models.front(), std::move(source), responseOptions);
+
+  // Make a copy of target
+  std::string target = forwardResponse.target.text;
+  Response backwardResponse = translateForResponse(service, models.back(), std::move(target), responseOptions);
+
+  // Print both onto the command-line
+  std::cout << forwardResponse.source.text;
+  std::cout << "----------------\n";
+  std::cout << forwardResponse.target.text;
+  std::cout << "----------------\n";
+  std::cout << backwardResponse.target.text;
+}
+
+void qualityEstimatorWords(AsyncService &service, Ptr<TranslationModel> model) {
   ResponseOptions responseOptions;
   responseOptions.qualityScores = true;
-  const Response response = translateFromStdin(options, responseOptions);
+  std::string source = readFromStdin();
+  const Response response = translateForResponse(service, model, std::move(source), responseOptions);
 
   for (const auto &sentenceQualityEstimate : response.qualityScores) {
     std::cout << "[SentenceBegin]\n";
@@ -71,10 +91,12 @@ void qualityEstimatorWords(const Ptr<Options> &options) {
   }
 }
 
-void qualityEstimatorScores(const Ptr<Options> &options) {
+void qualityEstimatorScores(AsyncService &service, Ptr<TranslationModel> model) {
   ResponseOptions responseOptions;
   responseOptions.qualityScores = true;
-  const Response response = translateFromStdin(options, responseOptions);
+
+  std::string source = readFromStdin();
+  const Response response = translateForResponse(service, model, std::move(source), responseOptions);
 
   for (const auto &sentenceQualityEstimate : response.qualityScores) {
     std::cout << std::fixed << std::setprecision(3) << sentenceQualityEstimate.sentenceScore << "\n";
diff --git a/src/tests/apps.h b/src/tests/apps.h
index deb6a12dc..dee77a9be 100644
--- a/src/tests/apps.h
+++ b/src/tests/apps.h
@@ -21,23 +21,21 @@ namespace bergamot {
 
 namespace testapp {
 
-// Utility function, common for all testapps. Reads content from stdin, builds a Service based on options and constructs
-// a response containing translation data according responseOptions.
-Response translateFromStdin(Ptr<Options> options, ResponseOptions responseOptions);
-
 // Reads from stdin and translates.  Prints the tokens separated by space for each sentence. Prints words from source
 // side text annotation if source=true, target annotation otherwise.
-void annotatedTextWords(Ptr<Options> options, bool source = true);
+void annotatedTextWords(AsyncService &service, Ptr<TranslationModel> model, bool source = true);
 
 // Reads from stdin and translates the read content. Prints the sentences in source or target in constructed response
 // in each line, depending on source = true or false respectively.
-void annotatedTextSentences(Ptr<Options> options, bool source = true);
+void annotatedTextSentences(AsyncService &service, Ptr<TranslationModel> model, bool source = true);
+
+void forwardAndBackward(AsyncService &service, std::vector<Ptr<TranslationModel>> &models);
 
 // Reads from stdin and translates the read content. Prints the quality words for each sentence.
-void qualityEstimatorWords(const Ptr<Options>& options);
+void qualityEstimatorWords(AsyncService &service, Ptr<TranslationModel> model);
 
 // Reads from stdin and translates the read content. Prints the quality scores for each sentence.
-void qualityEstimatorScores(const Ptr<Options>& options);
+void qualityEstimatorScores(AsyncService &service, Ptr<TranslationModel> model);
 
 }  // namespace testapp
 }  // namespace bergamot
diff --git a/src/tests/cli.cpp b/src/tests/cli.cpp
index 0e9469ab0..90c386c84 100644
--- a/src/tests/cli.cpp
+++ b/src/tests/cli.cpp
@@ -1,23 +1,45 @@
-
 #include "apps.h"
 
 int main(int argc, char *argv[]) {
-  auto cp = marian::bergamot::createConfigParser();
-  auto options = cp.parseOptions(argc, argv, true);
-  const std::string mode = options->get<std::string>("bergamot-mode");
   using namespace marian::bergamot;
-  if (mode == "test-response-source-sentences") {
-    testapp::annotatedTextSentences(options, /*source=*/true);
-  } else if (mode == "test-response-target-sentences") {
-    testapp::annotatedTextSentences(options, /*source=*/false);
-  } else if (mode == "test-response-source-words") {
-    testapp::annotatedTextWords(options, /*source=*/true);
-  } else if (mode == std::string("test-quality-estimator-words")) {
-    testapp::qualityEstimatorWords(options);
-  } else if (mode == std::string("test-quality-estimator-scores")) {
-    testapp::qualityEstimatorScores(options);
-  } else {
-    ABORT("Unknown --mode {}. Please run a valid test", mode);
+  marian::bergamot::ConfigParser configParser;
+  configParser.parseArgs(argc, argv);
+  auto &config = configParser.getConfig();
+  AsyncService::Config serviceConfig{config.numWorkers};
+  AsyncService service(serviceConfig);
+  std::vector<std::shared_ptr<TranslationModel>> models;
+
+  for (auto &modelConfigPath : config.modelConfigPaths) {
+    TranslationModel::Config modelConfig = parseOptionsFromFilePath(modelConfigPath);
+    std::shared_ptr<TranslationModel> model = service.createCompatibleModel(modelConfig);
+    models.push_back(model);
+  }
+
+  switch (config.opMode) {
+    case OpMode::TEST_SOURCE_SENTENCES:
+      testapp::annotatedTextSentences(service, models.front(), /*source=*/true);
+      break;
+    case OpMode::TEST_TARGET_SENTENCES:
+      testapp::annotatedTextSentences(service, models.front(), /*source=*/false);
+      break;
+    case OpMode::TEST_SOURCE_WORDS:
+      testapp::annotatedTextWords(service, models.front(), /*source=*/true);
+      break;
+    case OpMode::TEST_TARGET_WORDS:
+      testapp::annotatedTextWords(service, models.front(), /*source=*/false);
+      break;
+    case OpMode::TEST_FORWARD_BACKWARD_FOR_OUTBOUND:
+      testapp::forwardAndBackward(service, models);
+      break;
+    case OpMode::TEST_QUALITY_ESTIMATOR_WORDS:
+      testapp::qualityEstimatorWords(service, models.front());
+      break;
+    case OpMode::TEST_QUALITY_ESTIMATOR_SCORES:
+      testapp::qualityEstimatorScores(service, models.front());
+      break;
+    default:
+      ABORT("Incompatible op-mode. Choose one of the test modes.");
+      break;
   }
   return 0;
 }
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index c0ee6be7a..ab1448800 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -5,15 +5,16 @@ configure_file(${CMAKE_CURRENT_SOURCE_DIR}/project_version.h.in
 add_library(bergamot-translator STATIC
     byte_array_util.cpp
     text_processor.cpp
-    batch_translator.cpp 
+    translation_model.cpp 
     request.cpp 
-    batcher.cpp
+    batching_pool.cpp
+    aggregate_batching_pool.cpp
     response_builder.cpp
     quality_estimator.cpp
     batch.cpp
     annotation.cpp
     service.cpp
-    threadsafe_batcher.cpp
+    parser.cpp
 )
 if (USE_WASM_COMPATIBLE_SOURCE)
   # Using wasm compatible sources should include this compile definition;
diff --git a/src/translator/aggregate_batching_pool.cpp b/src/translator/aggregate_batching_pool.cpp
new file mode 100644
index 000000000..38c55f1c4
--- /dev/null
+++ b/src/translator/aggregate_batching_pool.cpp
@@ -0,0 +1,34 @@
+
+#include "aggregate_batching_pool.h"
+
+namespace marian {
+namespace bergamot {
+
+AggregateBatchingPool::AggregateBatchingPool() {
+  // TODO(@jerinphilip): Set aggregate limits
+}
+
+size_t AggregateBatchingPool::enqueueRequest(Ptr<TranslationModel> model, Ptr<Request> request) {
+  model->enqueueRequest(request);
+  aggregateQueue_.insert(model);
+  return request->numSegments();
+}
+
+size_t AggregateBatchingPool::generateBatch(Ptr<TranslationModel>& model, Batch& batch) {
+  while (!aggregateQueue_.empty()) {
+    auto candidateItr = aggregateQueue_.begin();
+    Ptr<TranslationModel> candidate = *candidateItr;
+    size_t numSentences = candidate->generateBatch(batch);
+    if (numSentences > 0) {
+      model = candidate;
+      return numSentences;
+    } else {
+      // Try the next model's batching pool.
+      aggregateQueue_.erase(candidateItr);
+    }
+  }
+  return /*numSentences=*/0;
+}
+
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/aggregate_batching_pool.h b/src/translator/aggregate_batching_pool.h
new file mode 100644
index 000000000..5b5d4b17a
--- /dev/null
+++ b/src/translator/aggregate_batching_pool.h
@@ -0,0 +1,68 @@
+#ifndef SRC_BERGAMOT_AGGREGATE_BATCHING_POOL_H_
+#define SRC_BERGAMOT_AGGREGATE_BATCHING_POOL_H_
+
+#include <memory>
+#include <queue>
+
+#include "data/types.h"
+#include "translation_model.h"
+
+namespace marian {
+namespace bergamot {
+
+/// Hashes a pointer to an object using the address the pointer points to. If two pointers point to the same address,
+/// they hash to the same value.  Useful to put widely shared_ptrs of entities (eg: TranslationModel, Vocab, Shortlist)
+/// etc into containers which require the members to be hashable (std::unordered_set, std::unordered_map).
+template <class T>
+struct HashPtr {
+  size_t operator()(const std::shared_ptr<T>& t) const {
+    size_t address = reinterpret_cast<size_t>(t.get());
+    return std::hash<size_t>()(address);
+  }
+};
+
+/// Aggregates request queueing and generation of batches from multiple TranslationModels (BatchingPools within,
+/// specifically), thereby acting as an intermediary to enable multiple translation model capability in BlockingService
+/// and AsyncService.
+///
+/// A simple queue containing shared owning references to TranslationModels are held here from which batches are
+/// generated on demand. Since a queue is involved, the ordering is first-come first serve on requests except there are
+/// leaks effectively doing priority inversion if an earlier request with the same TranslationModel is pending
+/// to be consumed for translation.
+//
+/// Actual storage for the request and batch generation are within the respective TranslationModels, which owns its own
+/// BatchingPool.
+///
+/// Matches API provided by BatchingPool except arguments additionally parameterized by TranslationModel.
+///
+/// Note: This class is not thread-safe. You may use this class wrapped with ThreadsafeBatchingPool for a thread-safe
+/// equivalent of this class, if needed.
+class AggregateBatchingPool {
+ public:
+  /// Create an AggregateBatchingPool with (tentatively) global (across all BatchingPools) limits
+  /// imposed here.
+  AggregateBatchingPool();
+
+  /// Enqueue an existing request onto model, also keep account of that this model and request are now pending.
+  ///
+  /// @param [in] model: Model to use in translation. A shared ownership to this model is accepted by this object to
+  /// keep the model alive until translation is complete.
+  /// @param [in] request: A request to be enqueued to model.
+  /// @returns number of sentences added for translation.
+  size_t enqueueRequest(Ptr<TranslationModel> model, Ptr<Request> request);
+
+  /// Generate a batch from pending requests, obtained from available TranslationModels.
+  ///
+  /// @param [out] model: TranslationModel
+  /// @param [out] batch: Batch to write onto, which is consumed at translation elsewhere.
+  /// @returns Number of sentences in the generated batch.
+  size_t generateBatch(Ptr<TranslationModel>& model, Batch& batch);
+
+ private:
+  std::unordered_set<std::shared_ptr<TranslationModel>, HashPtr<TranslationModel>> aggregateQueue_;
+};
+
+}  // namespace bergamot
+}  // namespace marian
+
+#endif  //  SRC_BERGAMOT_AGGREGATE_BATCHING_POOL_H_
diff --git a/src/translator/batch_translator.cpp b/src/translator/batch_translator.cpp
deleted file mode 100644
index 889ff0073..000000000
--- a/src/translator/batch_translator.cpp
+++ /dev/null
@@ -1,128 +0,0 @@
-#include "batch_translator.h"
-
-#include "batch.h"
-#include "byte_array_util.h"
-#include "common/logging.h"
-#include "data/corpus.h"
-#include "data/text_input.h"
-#include "translator/beam_search.h"
-
-namespace marian {
-namespace bergamot {
-
-BatchTranslator::BatchTranslator(DeviceId const device, Vocabs &vocabs, Ptr<Options> options,
-                                 const AlignedMemory *modelMemory, const AlignedMemory *shortlistMemory)
-    : device_(device),
-      options_(options),
-      vocabs_(vocabs),
-      modelMemory_(modelMemory),
-      shortlistMemory_(shortlistMemory) {}
-
-void BatchTranslator::initialize() {
-  // Initializes the graph.
-  if (options_->hasAndNotEmpty("shortlist")) {
-    int srcIdx = 0, trgIdx = 1;
-    bool shared_vcb =
-        vocabs_.sources().front() ==
-        vocabs_.target();  // vocabs_->sources().front() is invoked as we currently only support one source vocab
-    if (shortlistMemory_->size() > 0 && shortlistMemory_->begin() != nullptr) {
-      slgen_ = New<data::BinaryShortlistGenerator>(shortlistMemory_->begin(), shortlistMemory_->size(),
-                                                   vocabs_.sources().front(), vocabs_.target(), srcIdx, trgIdx,
-                                                   shared_vcb, options_->get<bool>("check-bytearray"));
-    } else {
-      // Changed to BinaryShortlistGenerator to enable loading binary shortlist file
-      // This class also supports text shortlist file
-      slgen_ = New<data::BinaryShortlistGenerator>(options_, vocabs_.sources().front(), vocabs_.target(), srcIdx,
-                                                   trgIdx, shared_vcb);
-    }
-  }
-
-  graph_ = New<ExpressionGraph>(true);  // set the graph to be inference only
-  auto prec = options_->get<std::vector<std::string>>("precision", {"float32"});
-  graph_->setDefaultElementType(typeFromString(prec[0]));
-  graph_->setDevice(device_);
-  graph_->getBackend()->configureDevice(options_);
-  graph_->reserveWorkspaceMB(options_->get<size_t>("workspace"));
-  if (modelMemory_->size() > 0 &&
-      modelMemory_->begin() !=
-          nullptr) {  // If we have provided a byte array that contains the model memory, we can initialise the model
-                      // from there, as opposed to from reading in the config file
-    ABORT_IF((uintptr_t)modelMemory_->begin() % 256 != 0,
-             "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
-    if (options_->get<bool>("check-bytearray")) {
-      ABORT_IF(!validateBinaryModel(*modelMemory_, modelMemory_->size()),
-               "The binary file is invalid. Incomplete or corrupted download?");
-    }
-    const std::vector<const void *> container = {
-        modelMemory_->begin()};  // Marian supports multiple models initialised in this manner hence std::vector.
-                                 // However we will only ever use 1 during decoding.
-    scorers_ = createScorers(options_, container);
-  } else {
-    scorers_ = createScorers(options_);
-  }
-  for (auto scorer : scorers_) {
-    scorer->init(graph_);
-    if (slgen_) {
-      scorer->setShortlistGenerator(slgen_);
-    }
-  }
-  graph_->forward();
-}
-
-void BatchTranslator::translate(Batch &batch) {
-  std::vector<data::SentenceTuple> batchVector;
-
-  auto &sentences = batch.sentences();
-  size_t batchSequenceNumber{0};
-  for (auto &sentence : sentences) {
-    data::SentenceTuple sentence_tuple(batchSequenceNumber);
-    Segment segment = sentence.getUnderlyingSegment();
-    sentence_tuple.push_back(segment);
-    batchVector.push_back(sentence_tuple);
-
-    ++batchSequenceNumber;
-  }
-
-  size_t batchSize = batchVector.size();
-  std::vector<size_t> sentenceIds;
-  std::vector<int> maxDims;
-  for (auto &ex : batchVector) {
-    if (maxDims.size() < ex.size()) maxDims.resize(ex.size(), 0);
-    for (size_t i = 0; i < ex.size(); ++i) {
-      if (ex[i].size() > (size_t)maxDims[i]) maxDims[i] = (int)ex[i].size();
-    }
-    sentenceIds.push_back(ex.getId());
-  }
-
-  typedef marian::data::SubBatch SubBatch;
-  typedef marian::data::CorpusBatch CorpusBatch;
-
-  std::vector<Ptr<SubBatch>> subBatches;
-  for (size_t j = 0; j < maxDims.size(); ++j) {
-    subBatches.emplace_back(New<SubBatch>(batchSize, maxDims[j], vocabs_.sources().at(j)));
-  }
-
-  std::vector<size_t> words(maxDims.size(), 0);
-  for (size_t i = 0; i < batchSize; ++i) {
-    for (size_t j = 0; j < maxDims.size(); ++j) {
-      for (size_t k = 0; k < batchVector[i][j].size(); ++k) {
-        subBatches[j]->data()[k * batchSize + i] = batchVector[i][j][k];
-        subBatches[j]->mask()[k * batchSize + i] = 1.f;
-        words[j]++;
-      }
-    }
-  }
-
-  for (size_t j = 0; j < maxDims.size(); ++j) subBatches[j]->setWords(words[j]);
-
-  auto corpus_batch = Ptr<CorpusBatch>(new CorpusBatch(subBatches));
-  corpus_batch->setSentenceIds(sentenceIds);
-
-  auto search = New<BeamSearch>(options_, scorers_, vocabs_.target());
-
-  auto histories = std::move(search->search(graph_, corpus_batch));
-  batch.completeBatch(histories);
-}
-
-}  // namespace bergamot
-}  // namespace marian
diff --git a/src/translator/batch_translator.h b/src/translator/batch_translator.h
deleted file mode 100644
index 6a7fa9842..000000000
--- a/src/translator/batch_translator.h
+++ /dev/null
@@ -1,57 +0,0 @@
-#ifndef SRC_BERGAMOT_BATCH_TRANSLATOR_H_
-#define SRC_BERGAMOT_BATCH_TRANSLATOR_H_
-
-#include <string>
-#include <vector>
-
-#include "batch.h"
-#include "common/utils.h"
-#include "data/shortlist.h"
-#include "definitions.h"
-#include "request.h"
-#include "translator/history.h"
-#include "translator/scorers.h"
-#include "vocabs.h"
-
-namespace marian {
-namespace bergamot {
-
-class BatchTranslator {
-  // Launches minimal marian-translation (only CPU at the moment) in individual
-  // threads. Constructor launches each worker thread running mainloop().
-  // mainloop runs until until it receives poison from the PCQueue. Threads are
-  // shut down in Service which calls join() on the threads.
-
- public:
-  /**
-   * Initialise the marian translator.
-   * @param device DeviceId that performs translation. Could be CPU or GPU
-   * @param vocabs Vector that contains ptrs to two vocabs
-   * @param options Marian options object
-   * @param modelMemory byte array (aligned to 256!!!) that contains the bytes of a model.bin. Provide a nullptr if not
-   * used.
-   * @param shortlistMemory byte array of shortlist (aligned to 64)
-   */
-  explicit BatchTranslator(DeviceId const device, Vocabs& vocabs, Ptr<Options> options,
-                           const AlignedMemory* modelMemory, const AlignedMemory* shortlistMemory);
-
-  // convenience function for logging. TODO(jerin)
-  std::string _identifier() { return "worker" + std::to_string(device_.no); }
-  void translate(Batch& batch);
-  void initialize();
-
- private:
-  Ptr<Options> options_;
-  DeviceId device_;
-  const Vocabs& vocabs_;
-  Ptr<ExpressionGraph> graph_;
-  std::vector<Ptr<Scorer>> scorers_;
-  Ptr<data::ShortlistGenerator const> slgen_;
-  const AlignedMemory* modelMemory_{nullptr};
-  const AlignedMemory* shortlistMemory_{nullptr};
-};
-
-}  // namespace bergamot
-}  // namespace marian
-
-#endif  //  SRC_BERGAMOT_BATCH_TRANSLATOR_H_
diff --git a/src/translator/batcher.cpp b/src/translator/batching_pool.cpp
similarity index 83%
rename from src/translator/batcher.cpp
rename to src/translator/batching_pool.cpp
index 0a14459f1..83b5e00ab 100644
--- a/src/translator/batcher.cpp
+++ b/src/translator/batching_pool.cpp
@@ -1,4 +1,4 @@
-#include "batcher.h"
+#include "batching_pool.h"
 
 #include <cassert>
 
@@ -8,7 +8,7 @@
 namespace marian {
 namespace bergamot {
 
-Batcher::Batcher(Ptr<Options> options) {
+BatchingPool::BatchingPool(Ptr<Options> options) {
   miniBatchWords = options->get<int>("mini-batch-words");
   bucket_.resize(options->get<int>("max-length-break") + 1);
   ABORT_IF(bucket_.size() - 1 > miniBatchWords,
@@ -16,7 +16,7 @@ Batcher::Batcher(Ptr<Options> options) {
            "longer than what can fit in a batch.");
 }
 
-bool Batcher::cleaveBatch(Batch &batch) {
+size_t BatchingPool::generateBatch(Batch &batch) {
   // For now simply iterates on buckets and converts batches greedily.  This
   // has to be enhanced with optimizing over priority. The baseline
   // implementation should at least be as fast as marian's maxi-batch with full
@@ -35,22 +35,23 @@ bool Batcher::cleaveBatch(Batch &batch) {
       } else {
         // Check if elements exist
         assert(batch.size() > 0);
-        return true;
+        return batch.size();
       }
     }
   }
 
-  bool isValidBatch = batch.size() > 0;
-  return isValidBatch;
+  return batch.size();
 }
 
-void Batcher::addWholeRequest(Ptr<Request> request) {
+size_t BatchingPool::enqueueRequest(Ptr<Request> request) {
   for (size_t i = 0; i < request->numSegments(); i++) {
     RequestSentence sentence(i, request);
     size_t bucket_id = sentence.numTokens();
     assert(bucket_id < bucket_.size());
     bucket_[bucket_id].insert(sentence);
   }
+
+  return request->numSegments();
 }
 
 }  // namespace bergamot
diff --git a/src/translator/batcher.h b/src/translator/batching_pool.h
similarity index 63%
rename from src/translator/batcher.h
rename to src/translator/batching_pool.h
index 277bfc934..68b2cf0d0 100644
--- a/src/translator/batcher.h
+++ b/src/translator/batching_pool.h
@@ -1,5 +1,5 @@
-#ifndef SRC_BERGAMOT_BATCHER_H_
-#define SRC_BERGAMOT_BATCHER_H_
+#ifndef SRC_BERGAMOT_BATCHING_POOL_H_
+#define SRC_BERGAMOT_BATCHING_POOL_H_
 
 #include <set>
 #include <vector>
@@ -12,24 +12,21 @@
 
 namespace marian {
 namespace bergamot {
-class Batcher {
+
+class BatchingPool {
  public:
-  explicit Batcher(Ptr<Options> options);
+  explicit BatchingPool(Ptr<Options> options);
 
   // RequestSentence incorporates (tentative) notions of priority with each
   // sentence. This method inserts the sentence into the internal data-structure
   // which maintains priority among sentences from multiple concurrent requests.
-  void addWholeRequest(Ptr<Request> request);
-
-  // indicate no more sentences will be added.  Does nothing here, for parity to threadsafe version.
-  void shutdown() {}
-
-  bool operator>>(Batch &batch) { return cleaveBatch(batch); }
+  size_t enqueueRequest(Ptr<Request> request);
 
- private:
   // Loads sentences with sentences compiled from (tentatively) multiple
   // requests optimizing for both padding and priority.
-  bool cleaveBatch(Batch &batch);
+  size_t generateBatch(Batch &batch);
+
+ private:
   size_t miniBatchWords;
   std::vector<std::set<RequestSentence>> bucket_;
   size_t batchNumber_{0};
@@ -38,4 +35,4 @@ class Batcher {
 }  // namespace bergamot
 }  // namespace marian
 
-#endif  // SRC_BERGAMOT_BATCHER_H_
+#endif  // SRC_BERGAMOT_BATCHING_POOL_H_
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index a0f544ded..66ebb03b4 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -41,6 +41,9 @@ struct ByteRange {
   const size_t size() const { return end - begin; }
 };
 
+class Response;
+using CallbackType = std::function<void(Response&&)>;
+
 }  // namespace bergamot
 }  // namespace marian
 
diff --git a/src/translator/parser.cpp b/src/translator/parser.cpp
new file mode 100644
index 000000000..d927409b5
--- /dev/null
+++ b/src/translator/parser.cpp
@@ -0,0 +1,170 @@
+#include "parser.h"
+
+#include <unordered_map>
+
+#include "common/build_info.h"
+#include "common/config.h"
+#include "common/regex.h"
+#include "common/version.h"
+
+namespace marian {
+namespace bergamot {
+
+std::istringstream &operator>>(std::istringstream &in, OpMode &mode) {
+  std::string modeString;
+  in >> modeString;
+  std::unordered_map<std::string, OpMode> table = {
+      {"wasm", OpMode::APP_WASM},
+      {"native", OpMode::APP_NATIVE},
+      {"decoder", OpMode::APP_DECODER},
+      {"test-response-source-sentences", OpMode::TEST_SOURCE_SENTENCES},
+      {"test-response-target-sentences", OpMode::TEST_TARGET_SENTENCES},
+      {"test-response-source-words", OpMode::TEST_SOURCE_WORDS},
+      {"test-response-target-words", OpMode::TEST_TARGET_WORDS},
+      {"test-quality-estimator-words", OpMode::TEST_QUALITY_ESTIMATOR_WORDS},
+      {"test-quality-estimator-scores", OpMode::TEST_QUALITY_ESTIMATOR_SCORES},
+      {"test-forward-backward", OpMode::TEST_FORWARD_BACKWARD_FOR_OUTBOUND},
+  };
+
+  auto query = table.find(modeString);
+  if (query != table.end()) {
+    mode = query->second;
+  } else {
+    ABORT("Unknown mode {}", modeString);
+  }
+
+  return in;
+}
+
+ConfigParser::ConfigParser() : app_{"Bergamot Options"} {
+  addSpecialOptions(app_);
+  addOptionsBoundToConfig(app_, config_);
+};
+
+void ConfigParser::parseArgs(int argc, char *argv[]) {
+  try {
+    app_.parse(argc, argv);
+    handleSpecialOptions();
+  } catch (const CLI::ParseError &e) {
+    exit(app_.exit(e));
+  }
+}
+
+void ConfigParser::addSpecialOptions(CLI::App &app) {
+  app.add_flag("--build-info", build_info_, "Print build-info and exit");
+  app.add_flag("--version", version_, "Print version-info and exit");
+}
+
+void ConfigParser::handleSpecialOptions() {
+  if (build_info_) {
+#ifndef _MSC_VER  // cmake build options are not available on MSVC based build.
+    std::cerr << cmakeBuildOptionsAdvanced() << std::endl;
+    exit(0);
+#else   // _MSC_VER
+    ABORT("build-info is not available on MSVC based build.");
+#endif  // _MSC_VER
+  }
+
+  if (version_) {
+    std::cerr << buildVersion() << std::endl;
+    exit(0);
+  }
+}
+
+void ConfigParser::addOptionsBoundToConfig(CLI::App &app, CLIConfig &config) {
+  app.add_option("--model-config-paths", config.modelConfigPaths,
+                 "Configuration files list, can be used for pivoting multiple models or multiple model workflows");
+
+  app.add_flag("--bytearray", config.byteArray,
+               "Flag holds whether to construct service from bytearrays, only for testing purpose");
+
+  app.add_flag("--check-bytearray", config.validateByteArray,
+               "Flag holds whether to check the content of the bytearrays (true by default)");
+
+  app.add_option("--cpu-threads", config.numWorkers, "Number of worker threads to use for translation");
+
+  app_.add_option("--bergamot-mode", config.opMode, "Operating mode for bergamot: [wasm, native, decoder]");
+}
+
+std::shared_ptr<marian::Options> parseOptionsFromFilePath(const std::string &configPath, bool validate /*= true*/) {
+  // Read entire string and redirect to parseOptionsFromString
+  std::ifstream readStream(configPath);
+  std::stringstream buffer;
+  buffer << readStream.rdbuf();
+  return parseOptionsFromString(buffer.str(), validate, /*pathsInSameDirAs=*/configPath);
+};
+
+std::shared_ptr<marian::Options> parseOptionsFromString(const std::string &configAsString, bool validate /*= true*/,
+                                                        std::string pathsInSameDirAs /*=""*/) {
+  marian::Options options;
+
+  marian::ConfigParser configParser(cli::mode::translation);
+
+  // These are additional options we use to hijack for our own marian-replacement layer (for batching,
+  // multi-request-compile etc) and hence goes into Ptr<Options>.
+  configParser.addOption<size_t>("--max-length-break", "Bergamot Options",
+                                 "Maximum input tokens to be processed in a single sentence.", 128);
+
+  // The following is a complete hijack of an existing option, so no need to add explicitly.
+  // configParser.addOption<size_t>("--mini-batch-words", "Bergamot Options",
+  //                                "Maximum input tokens to be processed in a single sentence.", 1024);
+
+  configParser.addOption<std::string>("--ssplit-prefix-file", "Bergamot Options",
+                                      "File with nonbreaking prefixes for sentence splitting.");
+
+  configParser.addOption<std::string>("--ssplit-mode", "Bergamot Options", "[paragraph, sentence, wrapped_text]",
+                                      "paragraph");
+
+  configParser.addOption<std::string>("--quality", "Bergamot Options", "File considering Quality Estimation model");
+
+  // Parse configs onto defaultConfig. The preliminary merge sets the YAML internal representation with legal values.
+  const YAML::Node &defaultConfig = configParser.getConfig();
+  options.merge(defaultConfig);
+  options.parse(configAsString);
+
+  // This is in a marian `.cpp` as of now, and requires explicit copy-here.
+  // https://github.com/marian-nmt/marian-dev/blob/9fa166be885b025711f27b35453e0f2c00c9933e/src/common/config_parser.cpp#L28
+
+  // clang-format off
+  const std::set<std::string> PATHS = {
+      "model",
+      "models",
+      "train-sets",
+      "vocabs",
+      "embedding-vectors",
+      "valid-sets",
+      "valid-script-path",
+      "valid-script-args",
+      "valid-log",
+      "valid-translation-output",
+      "input",   // except: 'stdin', handled in makeAbsolutePaths and interpolateEnvVars
+      "output",  // except: 'stdout', handled in makeAbsolutePaths and interpolateEnvVars
+      "pretrained-model",
+      "data-weighting",
+      "log",
+      "sqlite",     // except: 'temporary', handled in the processPaths function
+      "shortlist",  // except: only the first element in the sequence is a path, handled in the
+                    //  processPaths function
+      "ssplit-prefix-file", // added for bergamot
+      "quality", // added for bergamot
+  };
+  // clang-format on
+
+  if (!pathsInSameDirAs.empty()) {
+    YAML::Node configYAML = options.cloneToYamlNode();
+    marian::cli::makeAbsolutePaths(configYAML, pathsInSameDirAs, PATHS);
+    options.merge(configYAML, /*overwrite=*/true);
+  }
+
+  // Perform validation on parsed options only when requested
+  if (validate) {
+    YAML::Node configYAML = options.cloneToYamlNode();
+    marian::ConfigValidator validator(configYAML);
+    validator.validateOptions(marian::cli::mode::translation);
+  }
+
+  return std::make_shared<marian::Options>(options);
+}
+
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/parser.h b/src/translator/parser.h
index 54aaaf86a..c9fffcebf 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -1,6 +1,10 @@
 #ifndef SRC_BERGAMOT_PARSER_H
 #define SRC_BERGAMOT_PARSER_H
 
+#include <fstream>
+#include <sstream>
+
+#include "3rd_party/marian-dev/src/3rd_party/CLI/CLI.hpp"
 #include "3rd_party/yaml-cpp/yaml.h"
 #include "common/config_parser.h"
 #include "common/config_validator.h"
@@ -10,65 +14,63 @@
 namespace marian {
 namespace bergamot {
 
-inline marian::ConfigParser createConfigParser() {
-  marian::ConfigParser cp(marian::cli::mode::translation);
-  cp.addOption<std::string>("--ssplit-prefix-file", "Bergamot Options",
-                            "File with nonbreaking prefixes for sentence splitting.");
-
-  cp.addOption<std::string>("--ssplit-mode", "Server Options", "[paragraph, sentence, wrapped_text]", "paragraph");
-
-  cp.addOption<int>("--max-length-break", "Bergamot Options",
-                    "Maximum input tokens to be processed in a single sentence.", 128);
-
-  cp.addOption<bool>("--bytearray", "Bergamot Options",
-                     "Flag holds whether to construct service from bytearrays, only for testing purpose", false);
-
-  cp.addOption<bool>("--check-bytearray", "Bergamot Options",
-                     "Flag holds whether to check the content of the bytearrays (true by default)", true);
-
-  cp.addOption<std::string>("--bergamot-mode", "Bergamot Options",
-                            "Operating mode for bergamot: [wasm, native, decoder]", "native");
-
-  cp.addOption<std::string>("--quality", "Bergamot Options", "File considering Quality Estimation model");
-
-  return cp;
-}
-
-inline std::shared_ptr<marian::Options> parseOptions(const std::string &config, bool validate = true) {
-  marian::Options options;
-
-  // @TODO(jerinphilip) There's something off here, @XapaJIaMnu suggests
-  // that should not be using the defaultConfig. This function only has access
-  // to std::string config and needs to be able to construct Options from the
-  // same.
-
-  // Absent the following code-segment, there is a parsing exception thrown on
-  // rebuilding YAML.
-  //
-  // Error: Unhandled exception of type 'N4YAML11InvalidNodeE': invalid node;
-  // this may result from using a map iterator as a sequence iterator, or
-  // vice-versa
-  //
-  // Error: Aborted from void unhandledException() in
-  // 3rd_party/marian-dev/src/common/logging.cpp:113
-
-  marian::ConfigParser configParser = createConfigParser();
-  const YAML::Node &defaultConfig = configParser.getConfig();
-
-  options.merge(defaultConfig);
-
-  // Parse configs onto defaultConfig.
-  options.parse(config);
-  YAML::Node configCopy = options.cloneToYamlNode();
-
-  if (validate) {
-    // Perform validation on parsed options only when requested
-    marian::ConfigValidator validator(configCopy);
-    validator.validateOptions(marian::cli::mode::translation);
-  }
-
-  return std::make_shared<marian::Options>(options);
-}
+enum OpMode {
+  APP_WASM,
+  APP_NATIVE,
+  APP_DECODER,
+  TEST_SOURCE_SENTENCES,
+  TEST_TARGET_SENTENCES,
+  TEST_SOURCE_WORDS,
+  TEST_TARGET_WORDS,
+  TEST_QUALITY_ESTIMATOR_WORDS,
+  TEST_QUALITY_ESTIMATOR_SCORES,
+  TEST_FORWARD_BACKWARD_FOR_OUTBOUND,
+};
+
+/// Overload for CL11, convert a read from a stringstream into opmode.
+std::istringstream &operator>>(std::istringstream &in, OpMode &mode);
+
+struct CLIConfig {
+  using ModelConfigPaths = std::vector<std::string>;
+  ModelConfigPaths modelConfigPaths;
+  bool byteArray;
+  bool validateByteArray;
+  size_t numWorkers;
+  OpMode opMode;
+};
+
+/// ConfigParser for bergamot. Internally stores config options with CLIConfig. CLI11 parsing binds the parsing code to
+/// write to the members of the CLIConfig instance owned by this class. Usage:
+///
+/// ```cpp
+/// ConfigParser configParser;
+/// configParser.parseArgs(argc, argv);
+/// auto &config = configParser.getConfig();
+/// ```
+class ConfigParser {
+ public:
+  ConfigParser();
+  void parseArgs(int argc, char *argv[]);
+  const CLIConfig &getConfig() { return config_; }
+
+ private:
+  // Special Options: build-info and version. These are not taken down further, the respective logic executed and
+  // program exits after.
+  void addSpecialOptions(CLI::App &app);
+  void handleSpecialOptions();
+
+  void addOptionsBoundToConfig(CLI::App &app, CLIConfig &config);
+
+  CLIConfig config_;
+  CLI::App app_;
+
+  bool build_info_{false};
+  bool version_{false};
+};
+
+std::shared_ptr<marian::Options> parseOptionsFromString(const std::string &config, bool validate = true,
+                                                        std::string pathsInSameDirAs = "");
+std::shared_ptr<marian::Options> parseOptionsFromFilePath(const std::string &config, bool validate = true);
 
 }  //  namespace bergamot
 }  //  namespace marian
diff --git a/src/translator/request.h b/src/translator/request.h
index a2ea1af86..d2645f6d8 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -19,7 +19,7 @@ namespace bergamot {
 /// A Request is an internal representation used to represent a request after
 /// processed by TextProcessor into sentences constituted by marian::Words.
 ///
-/// The batching mechanism (Batcher) draws from multiple Requests and compiles
+/// The batching mechanism (BatchingPool) draws from multiple Requests and compiles
 /// sentences into a batch. When a batch completes translation (at
 /// BatchTranslator, intended in a different thread), backward propogation
 /// happens through:
@@ -60,7 +60,7 @@ class Request {
   Segment getSegment(size_t index) const;
 
   /// For notions of priority among requests, used to enable std::set in
-  /// Batcher.
+  /// BatchingPool.
   bool operator<(const Request &request) const;
 
   /// Processes a history obtained after translating in a heterogenous batch
@@ -90,7 +90,7 @@ class Request {
 
 /// A RequestSentence provides a view to a sentence within a Request. Existence
 /// of this class allows the sentences and associated information to be kept
-/// within Request, while batching mechanism (Batcher) compiles Batch from
+/// within Request, while batching mechanism (BatchingPool) compiles Batch from
 /// RequestSentence-s coming from different Requests.
 class RequestSentence {
  public:
diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
index 614c7c282..36bae1e9e 100644
--- a/src/translator/response_builder.h
+++ b/src/translator/response_builder.h
@@ -29,7 +29,7 @@ class ResponseBuilder {
   /// @param [in] callback: callback with operates on the constructed Response.
   /// @param [in] qualityEstimator: the QualityEstimator model that can be used
   /// to provide translation quality probability.
-  ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source, Vocabs &vocabs,
+  ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source, const Vocabs &vocabs,
                   std::function<void(Response &&)> callback, const QualityEstimator &qualityEstimator)
       : responseOptions_(responseOptions),
         source_(std::move(source)),
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index f5996aa45..9de69ba8a 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -10,88 +10,59 @@
 namespace marian {
 namespace bergamot {
 
-Service::Service(Ptr<Options> options, MemoryBundle memoryBundle)
-    : requestId_(0),
-      options_(options),
-      vocabs_(options, std::move(memoryBundle.vocabs)),
-      text_processor_(options, vocabs_, std::move(memoryBundle.ssplitPrefixFile)),
-      batcher_(options),
-      numWorkers_(std::max<int>(1, options->get<int>("cpu-threads"))),
-      modelMemory_(std::move(memoryBundle.model)),
-      shortlistMemory_(std::move(memoryBundle.shortlist)),
-      qualityEstimator_(createQualityEstimator(getQualityEstimatorModel(memoryBundle, options)))
-#ifdef WASM_COMPATIBLE_SOURCE
-      ,
-      blocking_translator_(DeviceId(0, DeviceType::cpu), vocabs_, options_, &modelMemory_, &shortlistMemory_)
-#endif
-{
-#ifdef WASM_COMPATIBLE_SOURCE
-  blocking_translator_.initialize();
-#else
-  workers_.reserve(numWorkers_);
-  for (size_t cpuId = 0; cpuId < numWorkers_; cpuId++) {
-    workers_.emplace_back([cpuId, this] {
-      marian::DeviceId deviceId(cpuId, DeviceType::cpu);
-      BatchTranslator translator(deviceId, vocabs_, options_, &modelMemory_, &shortlistMemory_);
-      translator.initialize();
-      Batch batch;
-      // Run thread mainloop
-      while (batcher_ >> batch) {
-        translator.translate(batch);
-      }
-    });
-  }
-#endif
-}
+BlockingService::BlockingService(const BlockingService::Config &config) : requestId_(0), batchingPool_() {}
 
-#ifdef WASM_COMPATIBLE_SOURCE
-std::vector<Response> Service::translateMultiple(std::vector<std::string> &&inputs, ResponseOptions responseOptions) {
-  // We queue the individual Requests so they get compiled at batches to be
-  // efficiently translated.
+std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<TranslationModel> translationModel,
+                                                         std::vector<std::string> &&sources,
+                                                         const ResponseOptions &responseOptions) {
   std::vector<Response> responses;
-  responses.resize(inputs.size());
+  responses.resize(sources.size());
 
-  for (size_t i = 0; i < inputs.size(); i++) {
+  for (size_t i = 0; i < sources.size(); i++) {
     auto callback = [i, &responses](Response &&response) { responses[i] = std::move(response); };  //
-    queueRequest(std::move(inputs[i]), std::move(callback), responseOptions);
+    Ptr<Request> request =
+        translationModel->makeRequest(requestId_++, std::move(sources[i]), callback, responseOptions);
+    batchingPool_.enqueueRequest(translationModel, request);
   }
 
   Batch batch;
-  // There's no need to do shutdown here because it's single threaded.
-  while (batcher_ >> batch) {
-    blocking_translator_.translate(batch);
+  Ptr<TranslationModel> model{nullptr};
+  while (batchingPool_.generateBatch(model, batch)) {
+    model->translateBatch(/*deviceId=*/0, batch);
   }
 
   return responses;
 }
-#endif
-
-void Service::queueRequest(std::string &&input, std::function<void(Response &&)> &&callback,
-                           ResponseOptions responseOptions) {
-  Segments segments;
-  AnnotatedText source;
-
-  text_processor_.process(std::move(input), source, segments);
-
-  ResponseBuilder responseBuilder(responseOptions, std::move(source), vocabs_, std::move(callback), *qualityEstimator_);
-  Ptr<Request> request = New<Request>(requestId_++, std::move(segments), std::move(responseBuilder));
-
-  batcher_.addWholeRequest(request);
-}
 
-void Service::translate(std::string &&input, std::function<void(Response &&)> &&callback,
-                        ResponseOptions responseOptions) {
-  queueRequest(std::move(input), std::move(callback), responseOptions);
+AsyncService::AsyncService(const AsyncService::Config &config) : requestId_(0), config_(config), safeBatchingPool_() {
+  ABORT_IF(config_.numWorkers == 0, "Number of workers should be at least 1 in a threaded workflow");
+  workers_.reserve(config_.numWorkers);
+  for (size_t cpuId = 0; cpuId < config_.numWorkers; cpuId++) {
+    workers_.emplace_back([cpuId, this] {
+      // Consumer thread main-loop. Note that this is an infinite-loop unless the monitor is explicitly told to
+      // shutdown, which happens in the destructor for this class.
+      Batch batch;
+      Ptr<TranslationModel> translationModel{nullptr};
+      while (safeBatchingPool_.generateBatch(translationModel, batch)) {
+        translationModel->translateBatch(cpuId, batch);
+      }
+    });
+  }
 }
 
-Service::~Service() {
-  batcher_.shutdown();
-#ifndef WASM_COMPATIBLE_SOURCE
+AsyncService::~AsyncService() {
+  safeBatchingPool_.shutdown();
   for (std::thread &worker : workers_) {
     assert(worker.joinable());
     worker.join();
   }
-#endif
+}
+
+void AsyncService::translate(std::shared_ptr<TranslationModel> translationModel, std::string &&source,
+                             CallbackType callback, const ResponseOptions &responseOptions) {
+  // Producer thread, a call to this function adds new work items. If batches are available, notifies workers waiting.
+  Ptr<Request> request = translationModel->makeRequest(requestId_++, std::move(source), callback, responseOptions);
+  safeBatchingPool_.enqueueRequest(translationModel, request);
 }
 
 }  // namespace bergamot
diff --git a/src/translator/service.h b/src/translator/service.h
index 3a3d616fc..d37f5c262 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -1,146 +1,116 @@
 #ifndef SRC_BERGAMOT_SERVICE_H_
 #define SRC_BERGAMOT_SERVICE_H_
 
-#include "batch_translator.h"
+#include <queue>
+#include <thread>
+#include <vector>
+
 #include "data/types.h"
 #include "quality_estimator.h"
 #include "response.h"
 #include "response_builder.h"
 #include "text_processor.h"
-#include "threadsafe_batcher.h"
+#include "threadsafe_batching_pool.h"
+#include "translation_model.h"
 #include "translator/parser.h"
 #include "vocabs.h"
 
-#ifndef WASM_COMPATIBLE_SOURCE
-#include <thread>
-#endif
-
-#include <queue>
-#include <vector>
-
 namespace marian {
 namespace bergamot {
 
-///  This is intended to be similar to the ones  provided for training or
-///  decoding in ML pipelines with the following  additional capabilities:
-///
-///  1. Provision of a request -> response based translation flow unlike the
-///  usual a line based translation or decoding provided in most ML frameworks.
-///  2. Internal handling of normalization etc which changes source text to
-///  provide to client translation meta-information like alignments consistent
-///  with the unnormalized input text.
-///  3. The API splits each text entry into sentences internally, which are then
-///  translated independent of each other. The translated sentences are then
-///  joined back together and returned in Response.
-///
-/// Service exposes methods to instantiate from a string configuration (which
-/// can cover most translators) and to translate an incoming blob of text.
-///
-/// Optionally Service can be initialized by also passing bytearray memories
-/// for purposes of efficiency (which defaults to empty and then reads from
-/// file supplied through config).
+class BlockingService;
+class AsyncService;
+
+/// See AsyncService.
 ///
-class Service {
+/// BlockingService is a not-threaded counterpart of AsyncService which can operate only in a blocking workflow (queue a
+/// bunch of texts and optional args to translate, wait till the translation finishes).
+class BlockingService {
  public:
-  /// Construct Service from Marian options. If memoryBundle is empty, Service is
-  /// initialized from file-based loading. Otherwise, Service is initialized from
-  /// the given bytearray memories.
-  /// @param options Marian options object
-  /// @param memoryBundle holds all byte-array memories. Can be a set/subset of
-  /// model, shortlist, vocabs and ssplitPrefixFile or QualityEstimation bytes. Optional.
-  explicit Service(Ptr<Options> options, MemoryBundle memoryBundle = {});
-
-  /// Construct Service from a string configuration. If memoryBundle is empty, Service is
-  /// initialized from file-based loading. Otherwise, Service is initialized from
-  /// the given bytearray memories.
-  /// @param [in] config string parsable as YAML expected to adhere with marian config
-  /// @param [in] memoryBundle holds all byte-array memories. Can be a set/subset of
-  /// model, shortlist, vocabs and ssplitPrefixFile or qualityEstimation bytes. Optional.
-  explicit Service(const std::string &config, MemoryBundle memoryBundle = {})
-      : Service(parseOptions(config, /*validate=*/false), std::move(memoryBundle)) {}
-
-  /// Explicit destructor to clean up after any threads initialized in
-  /// asynchronous operation mode.
-  ~Service();
-
-  /// Translate an input, providing Options to construct Response. This is
-  /// useful when one has to set/unset alignments or quality in the Response to
-  /// save compute spent in constructing these objects.
-  ///
-  /// @param [in] source: rvalue reference of the string to be translated
-  /// @param [in] callback: A callback function provided by the client which
-  /// accepts an rvalue of a Response. Called on successful construction of a
-  /// Response following completion of translation of source by worker threads.
-  /// @param [in] responseOptions: Options indicating whether or not to include
-  /// some member in the Response, also specify any additional configurable
-  /// parameters.
-  void translate(std::string &&source, std::function<void(Response &&)> &&callback,
-                 ResponseOptions options = ResponseOptions());
-
-#ifdef WASM_COMPATIBLE_SOURCE
-  /// Translate multiple text-blobs in a single *blocking* API call, providing
-  /// ResponseOptions which applies across all text-blobs dictating how to
-  /// construct Response. ResponseOptions can be used to enable/disable
-  /// additional information like quality-scores, alignments etc.
-  ///
-  /// All texts are combined to efficiently construct batches together providing
-  /// speedups compared to calling translate() indepdently on individual
-  /// text-blob. Note that there will be minor differences in output when
-  /// text-blobs are individually translated due to approximations but similar
-  /// quality nonetheless. If you have async/multithread capabilities, it is
-  /// recommended to work with callbacks and translate() API.
-  ///
-  /// @param [in] source: rvalue reference of the string to be translated
-  /// @param [in] responseOptions: ResponseOptions indicating whether or not
-  /// to include some member in the Response, also specify any additional
-  /// configurable parameters.
-  std::vector<Response> translateMultiple(std::vector<std::string> &&source, ResponseOptions responseOptions);
-#endif
+  struct Config {};
+  /// Construct a BlockingService with configuration loaded from an Options object. Does not require any keys, values to
+  /// be set.
+  BlockingService(const BlockingService::Config &config);
+
+  /// Translate multiple text-blobs in a single *blocking* API call, providing ResponseOptions which applies across all
+  /// text-blobs dictating how to construct Response. ResponseOptions can be used to enable/disable additional
+  /// information like quality-scores, alignments etc.
 
-  /// Returns if model is alignment capable or not.
-  bool isAlignmentSupported() const { return options_->hasAndNotEmpty("alignment"); }
+  /// If you have async/multithread capabilities, it is recommended to work with AsyncService instead of this class.
+  /// Note that due to batching differences and consequent floating-point rounding differences, this is not guaranteed
+  /// to have the same output as AsyncService.
+
+  /// @param [in] translationModel: TranslationModel to use for the request.
+  /// @param [in] source: rvalue reference of the string to be translated
+  /// @param [in] responseOptions: ResponseOptions indicating whether or not to include some member in the Response,
+  /// also specify any additional configurable parameters.
+  std::vector<Response> translateMultiple(std::shared_ptr<TranslationModel> translationModel,
+                                          std::vector<std::string> &&source, const ResponseOptions &responseOptions);
 
  private:
-  /// Queue an input for translation.
-  void queueRequest(std::string &&input, std::function<void(Response &&)> &&callback, ResponseOptions responseOptions);
+  ///  Numbering requests processed through this instance. Used to keep account of arrival times of the request. This
+  ///  allows for using this quantity in priority based ordering.
+  size_t requestId_;
 
-  /// Translates through direct interaction between batcher_ and translators_
+  /// An aggregate batching pool associated with an async translating instance, which maintains an aggregate queue of
+  /// requests compiled from  batching-pools of multiple translation models. Not thread-safe.
+  AggregateBatchingPool batchingPool_;
 
-  /// Number of workers to launch.
-  size_t numWorkers_;
+  Config config_;
+};
 
-  /// Options object holding the options Service was instantiated with.
-  Ptr<Options> options_;
+/// Effectively a threadpool, providing an API to take a translation request of a source-text, paramaterized by
+/// TranslationModel to be used for translation. Configurability on optional items for the Response corresponding to a
+/// request is provisioned through ResponseOptions.
+class AsyncService {
+ public:
+  struct Config {
+    size_t numWorkers;
+  };
+  /// Construct an AsyncService with configuration loaded from Options. Expects positive integer value for
+  /// `cpu-threads`. Additionally requires options which configure AggregateBatchingPool.
+  AsyncService(const AsyncService::Config &config);
+
+  /// Create a TranslationModel compatible with this instance of Service. Internally assigns how many replicas of
+  /// backend needed based on worker threads set. See TranslationModel for documentation on other params.
+  template <class ConfigType>
+  Ptr<TranslationModel> createCompatibleModel(const ConfigType &config, MemoryBundle &&memory = MemoryBundle{}) {
+    // @TODO: Remove this remove this dependency/coupling.
+    return New<TranslationModel>(config, std::move(memory), /*replicas=*/config_.numWorkers);
+  }
+
+  /// With the supplied TranslationModel, translate an input. A Response is constructed with optional items set/unset
+  /// indicated via ResponseOptions. Upon completion translation of the input, the client supplied callback is triggered
+  /// with the constructed Response. Concurrent-calls to this function are safe.
+  ///
+  /// @param [in] translationModel: TranslationModel to use for the request.
+  /// @param [in] source: rvalue reference of the string to be translated. This is available as-is to the client later
+  /// in the Response corresponding to this call along with the translated-text and meta-data.
+  /// @param [in] callback: A callback function provided by the client which accepts an rvalue of a Response.
+  /// @param [in] responseOptions: Options indicating whether or not to include some member in the Response, also
+  /// specify any additional configurable parameters.
+  void translate(std::shared_ptr<TranslationModel> translationModel, std::string &&source, CallbackType callback,
+                 const ResponseOptions &options = ResponseOptions());
+
+  /// Thread joins and proper shutdown are required to be handled explicitly.
+  ~AsyncService();
 
-  /// Model memory to load model passed as bytes.
-  AlignedMemory modelMemory_;  // ORDER DEPENDENCY (translators_)
-  /// Shortlist memory passed as bytes.
-  AlignedMemory shortlistMemory_;  // ORDER DEPENDENCY (translators_)
+ private:
+  AsyncService::Config config_;
 
-  std::shared_ptr<QualityEstimator> qualityEstimator_;
+  std::vector<std::thread> workers_;
 
   /// Stores requestId of active request. Used to establish
   /// ordering among requests and logging/book-keeping.
 
+  /// Numbering requests processed through this instance. Used to keep account of arrival times of the request. This
+  /// allows for using this quantity in priority based ordering.
   size_t requestId_;
-  /// Store vocabs representing source and target.
-  Vocabs vocabs_;  // ORDER DEPENDENCY (text_processor_)
-
-  /// TextProcesser takes a blob of text and converts into format consumable by
-  /// the batch-translator and annotates sentences and words.
-  TextProcessor text_processor_;  // ORDER DEPENDENCY (vocabs_)
-
-  /// Batcher handles generation of batches from a request, subject to
-  /// packing-efficiency and priority optimization heuristics.
-  ThreadsafeBatcher batcher_;
-
-  // The following constructs are available providing full capabilities on a non
-  // WASM platform, where one does not have to hide threads.
-#ifdef WASM_COMPATIBLE_SOURCE
-  BatchTranslator blocking_translator_;  // ORDER DEPENDENCY (modelMemory_, shortlistMemory_)
-#else
-  std::vector<std::thread> workers_;
-#endif  // WASM_COMPATIBLE_SOURCE
+
+  /// An aggregate batching pool associated with an async translating instance, which maintains an aggregate queue of
+  /// requests compiled from  batching-pools of multiple translation models. The batching pool is wrapped around one
+  /// object for thread-safety.
+  ThreadsafeBatchingPool<AggregateBatchingPool> safeBatchingPool_;
 };
 
 }  // namespace bergamot
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index 249ce8cda..b747f79a5 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -52,7 +52,7 @@ ug::ssplit::SentenceSplitter loadSplitter(const AlignedMemory &memory) {
 
 }  // namespace
 
-Segment TextProcessor::tokenize(const string_view &segment, std::vector<string_view> &wordRanges) {
+Segment TextProcessor::tokenize(const string_view &segment, std::vector<string_view> &wordRanges) const {
   // vocabs_->sources().front() is invoked as we currently only support one source vocab
   return vocabs_.sources().front()->encodeWithByteRanges(segment, wordRanges, /*addEOS=*/false, /*inference=*/true);
 }
@@ -81,10 +81,10 @@ TextProcessor::TextProcessor(Ptr<Options> options, const Vocabs &vocabs, const A
 
 void TextProcessor::parseCommonOptions(Ptr<Options> options) {
   maxLengthBreak_ = options->get<size_t>("max-length-break");
-  ssplitMode_ = string2splitmode(options->get<std::string>("ssplit-mode", "paragraph"));
+  ssplitMode_ = string2splitmode(options->get<std::string>("ssplit-mode"));
 }
 
-void TextProcessor::process(std::string &&input, AnnotatedText &source, Segments &segments) {
+void TextProcessor::process(std::string &&input, AnnotatedText &source, Segments &segments) const {
   source = std::move(AnnotatedText(std::move(input)));
   std::string_view input_converted(source.text.data(), source.text.size());
   auto sentenceStream = ug::ssplit::SentenceStream(input_converted, ssplit_, ssplitMode_);
@@ -108,7 +108,7 @@ void TextProcessor::process(std::string &&input, AnnotatedText &source, Segments
 }
 
 void TextProcessor::wrap(Segment &segment, std::vector<string_view> &wordRanges, Segments &segments,
-                         AnnotatedText &source) {
+                         AnnotatedText &source) const {
   // There's an EOS token added to the words, manually. SentencePiece/marian-vocab is set to not append EOS. Marian
   // requires EOS to be at the end as a marker to start translating. So while we're supplied maxLengthBreak_ from
   // outside, we need to ensure there's space for EOS in each wrapped segment.
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index 1dc5a4fa7..a6c918c0e 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -47,17 +47,17 @@ class TextProcessor {
   /// @param [out] segments: marian::Word equivalents of the sentences processed and stored in AnnotatedText for
   /// consumption of marian translation pipeline.
 
-  void process(std::string &&blob, AnnotatedText &source, Segments &segments);
+  void process(std::string &&blob, AnnotatedText &source, Segments &segments) const;
 
  private:
   void parseCommonOptions(Ptr<Options> options);
 
   /// Tokenizes an input string, returns Words corresponding. Loads the
   /// corresponding byte-ranges into tokenRanges.
-  Segment tokenize(const string_view &input, std::vector<string_view> &tokenRanges);
+  Segment tokenize(const string_view &input, std::vector<string_view> &tokenRanges) const;
 
   /// Wrap into sentences of at most maxLengthBreak_ tokens and add to source.
-  void wrap(Segment &sentence, std::vector<string_view> &tokenRanges, Segments &segments, AnnotatedText &source);
+  void wrap(Segment &sentence, std::vector<string_view> &tokenRanges, Segments &segments, AnnotatedText &source) const;
 
   const Vocabs &vocabs_;   ///< Vocabularies used to tokenize a sentence
   size_t maxLengthBreak_;  ///< Parameter used to wrap sentences to a maximum number of tokens
diff --git a/src/translator/threadsafe_batcher.cpp b/src/translator/threadsafe_batcher.cpp
deleted file mode 100644
index 38b6681a9..000000000
--- a/src/translator/threadsafe_batcher.cpp
+++ /dev/null
@@ -1,38 +0,0 @@
-#ifndef WASM_COMPATIBLE_SOURCE
-#include "threadsafe_batcher.h"
-
-#include <cassert>
-
-namespace marian {
-namespace bergamot {
-
-ThreadsafeBatcher::ThreadsafeBatcher(Ptr<Options> options) : backend_(options), enqueued_(0), shutdown_(false) {}
-
-ThreadsafeBatcher::~ThreadsafeBatcher() { shutdown(); }
-
-void ThreadsafeBatcher::addWholeRequest(Ptr<Request> request) {
-  std::unique_lock<std::mutex> lock(mutex_);
-  assert(!shutdown_);
-  backend_.addWholeRequest(request);
-  enqueued_ += request->numSegments();
-  work_.notify_all();
-}
-
-void ThreadsafeBatcher::shutdown() {
-  std::unique_lock<std::mutex> lock(mutex_);
-  shutdown_ = true;
-  work_.notify_all();
-}
-
-bool ThreadsafeBatcher::operator>>(Batch &batch) {
-  std::unique_lock<std::mutex> lock(mutex_);
-  work_.wait(lock, [this]() { return enqueued_ || shutdown_; });
-  bool ret = backend_ >> batch;
-  assert(ret || shutdown_);
-  enqueued_ -= batch.size();
-  return ret;
-}
-
-}  // namespace bergamot
-}  // namespace marian
-#endif  // WASM_COMPATIBLE_SOURCE
diff --git a/src/translator/threadsafe_batcher.h b/src/translator/threadsafe_batcher.h
deleted file mode 100644
index d0ab7b1cc..000000000
--- a/src/translator/threadsafe_batcher.h
+++ /dev/null
@@ -1,57 +0,0 @@
-/* Thread-safe wrapper around batcher. */
-#ifndef SRC_BERGAMOT_THREADSAFE_BATCHER_H_
-#define SRC_BERGAMOT_THREADSAFE_BATCHER_H_
-
-#include "batcher.h"
-#include "common/options.h"
-#include "definitions.h"
-
-#ifndef WASM_COMPATIBLE_SOURCE
-#include <condition_variable>
-#include <mutex>
-#endif
-
-namespace marian {
-namespace bergamot {
-
-#ifdef WASM_COMPATIBLE_SOURCE
-// No threads, no locks.
-typedef Batcher ThreadsafeBatcher;
-#else
-
-class ThreadsafeBatcher {
- public:
-  explicit ThreadsafeBatcher(Ptr<Options> options);
-
-  ~ThreadsafeBatcher();
-
-  // Add sentences to be translated by calling these (see Batcher).  When
-  // done, call shutdown.
-  void addWholeRequest(Ptr<Request> request);
-  void shutdown();
-
-  // Get a batch out of the batcher.  Return false to shutdown worker.
-  bool operator>>(Batch &batch);
-
- private:
-  Batcher backend_;
-
-  // Number of sentences in backend_;
-  size_t enqueued_;
-
-  // Are we shutting down?
-  bool shutdown_;
-
-  // Lock on this object.
-  std::mutex mutex_;
-
-  // Signaled when there are sentences to translate.
-  std::condition_variable work_;
-};
-
-#endif
-
-}  // namespace bergamot
-}  // namespace marian
-
-#endif  // SRC_BERGAMOT_THREADSAFE_BATCHER_H_
diff --git a/src/translator/threadsafe_batching_pool.cpp b/src/translator/threadsafe_batching_pool.cpp
new file mode 100644
index 000000000..0c0d8d85a
--- /dev/null
+++ b/src/translator/threadsafe_batching_pool.cpp
@@ -0,0 +1,49 @@
+
+#ifndef SRC_BERGAMOT_THREADSAFE_BATCHING_POOL_IMPL
+#error "This is an impl file and must not be included directly!"
+#endif
+
+#include <cassert>
+
+namespace marian {
+namespace bergamot {
+
+template <class BatchingPoolType>
+template <class... Args>
+ThreadsafeBatchingPool<BatchingPoolType>::ThreadsafeBatchingPool(Args &&... args)
+    : backend_(std::forward<Args>(args)...), enqueued_(0), shutdown_(false) {}
+
+template <class BatchingPoolType>
+ThreadsafeBatchingPool<BatchingPoolType>::~ThreadsafeBatchingPool() {
+  shutdown();
+}
+
+template <class BatchingPoolType>
+template <class... Args>
+void ThreadsafeBatchingPool<BatchingPoolType>::enqueueRequest(Args &&... args) {
+  std::unique_lock<std::mutex> lock(mutex_);
+  assert(!shutdown_);
+  enqueued_ += backend_.enqueueRequest(std::forward<Args>(args)...);
+  work_.notify_all();
+}
+
+template <class BatchingPoolType>
+void ThreadsafeBatchingPool<BatchingPoolType>::shutdown() {
+  std::unique_lock<std::mutex> lock(mutex_);
+  shutdown_ = true;
+  work_.notify_all();
+}
+
+template <class BatchingPoolType>
+template <class... Args>
+size_t ThreadsafeBatchingPool<BatchingPoolType>::generateBatch(Args &&... args) {
+  std::unique_lock<std::mutex> lock(mutex_);
+  work_.wait(lock, [this]() { return enqueued_ || shutdown_; });
+  size_t sentencesInBatch = backend_.generateBatch(std::forward<Args>(args)...);
+  assert(sentencesInBatch > 0 || shutdown_);
+  enqueued_ -= sentencesInBatch;
+  return sentencesInBatch;
+}
+
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/threadsafe_batching_pool.h b/src/translator/threadsafe_batching_pool.h
new file mode 100644
index 000000000..96896eab3
--- /dev/null
+++ b/src/translator/threadsafe_batching_pool.h
@@ -0,0 +1,71 @@
+/* Thread-safe wrapper around BatchingPool or AggregateBatchingPool, made generic with templates. */
+#ifndef SRC_BERGAMOT_THREADSAFE_BATCHING_POOL_H_
+#define SRC_BERGAMOT_THREADSAFE_BATCHING_POOL_H_
+
+#include <condition_variable>
+#include <mutex>
+
+#include "aggregate_batching_pool.h"
+#include "batching_pool.h"
+#include "common/options.h"
+#include "definitions.h"
+#include "translation_model.h"
+
+namespace marian {
+namespace bergamot {
+
+/// The following mechanism operates in a multithreaded async-workflow guarding access to the pushes to the structure
+/// keeping sentences bucketed by length and sorted by priority.
+///
+/// This is a wrap of a producer-consumer queue implemented as a monitor, where there is a mutex guarding the
+/// underlying data structure (BatchingPoolType) and (worker/consumer) threads waiting on a condition variable and the
+/// queuing thread producing and notifying waiting threads (consumers) through the same condition variable.
+///
+/// Originally written by for a single model (where items are produce: Request, consume: Batch), converted to
+/// also work for multiple models where items are produce: (TranslationModel, Request), consume: (TranlsationModel,
+/// Batch). This is accomplished by template parameter packs.
+///
+/// Requires BatchingPoolType to implement the following:
+///
+/// * produce: `size_t enqueueRequest(...)` (returns number elements produced)
+/// * consume: `size_t generateBatch(...)` (returns number of elements available to be consumed)
+
+template <class BatchingPoolType>
+class ThreadsafeBatchingPool {
+ public:
+  template <class... Args>
+  ThreadsafeBatchingPool(Args &&... args);
+  ~ThreadsafeBatchingPool();
+
+  template <class... Args>
+  void enqueueRequest(Args &&... args);
+
+  template <class... Args>
+  size_t generateBatch(Args &&... args);
+
+  void shutdown();
+
+ private:
+  BatchingPoolType backend_;
+
+  // Number of sentences in backend_;
+  size_t enqueued_;
+
+  // Are we shutting down?
+  bool shutdown_;
+
+  // Lock on this object.
+  std::mutex mutex_;
+
+  // Signaled when there are sentences to translate.
+  std::condition_variable work_;
+};
+
+}  // namespace bergamot
+}  // namespace marian
+
+#define SRC_BERGAMOT_THREADSAFE_BATCHING_POOL_IMPL
+#include "threadsafe_batching_pool.cpp"
+#undef SRC_BERGAMOT_THREADSAFE_BATCHING_POOL_IMPL
+
+#endif  // SRC_BERGAMOT_THREADSAFE_BATCHING_POOL_H_
diff --git a/src/translator/translation_model.cpp b/src/translator/translation_model.cpp
new file mode 100644
index 000000000..5a2739542
--- /dev/null
+++ b/src/translator/translation_model.cpp
@@ -0,0 +1,173 @@
+#include "translation_model.h"
+
+#include "batch.h"
+#include "byte_array_util.h"
+#include "common/logging.h"
+#include "data/corpus.h"
+#include "data/text_input.h"
+#include "parser.h"
+#include "translator/beam_search.h"
+
+namespace marian {
+namespace bergamot {
+
+TranslationModel::TranslationModel(const Config &options, MemoryBundle &&memory /*=MemoryBundle{}*/,
+                                   size_t replicas /*=1*/)
+    : options_(options),
+      memory_(std::move(memory)),
+      vocabs_(options, std::move(memory_.vocabs)),
+      textProcessor_(options, vocabs_, std::move(memory_.ssplitPrefixFile)),
+      batchingPool_(options),
+      qualityEstimator_(createQualityEstimator(getQualityEstimatorModel(memory, options))) {
+  ABORT_IF(replicas == 0, "At least one replica needs to be created.");
+  backend_.resize(replicas);
+
+  if (options_->hasAndNotEmpty("shortlist")) {
+    int srcIdx = 0, trgIdx = 1;
+    bool shared_vcb =
+        vocabs_.sources().front() ==
+        vocabs_.target();  // vocabs_->sources().front() is invoked as we currently only support one source vocab
+    if (memory_.shortlist.size() > 0 && memory_.shortlist.begin() != nullptr) {
+      bool check = options_->get<bool>("check-bytearray", false);
+      shortlistGenerator_ = New<data::BinaryShortlistGenerator>(memory_.shortlist.begin(), memory_.shortlist.size(),
+                                                                vocabs_.sources().front(), vocabs_.target(), srcIdx,
+                                                                trgIdx, shared_vcb, check);
+    } else {
+      // Changed to BinaryShortlistGenerator to enable loading binary shortlist file
+      // This class also supports text shortlist file
+      shortlistGenerator_ = New<data::BinaryShortlistGenerator>(options_, vocabs_.sources().front(), vocabs_.target(),
+                                                                srcIdx, trgIdx, shared_vcb);
+    }
+  }
+
+  for (size_t idx = 0; idx < replicas; idx++) {
+    loadBackend(idx);
+  }
+}
+
+void TranslationModel::loadBackend(size_t idx) {
+  auto &graph = backend_[idx].graph;
+  auto &scorerEnsemble = backend_[idx].scorerEnsemble;
+
+  marian::DeviceId device_(idx, DeviceType::cpu);
+  graph = New<ExpressionGraph>(/*inference=*/true);  // set the graph to be inference only
+  auto prec = options_->get<std::vector<std::string>>("precision", {"float32"});
+  graph->setDefaultElementType(typeFromString(prec[0]));
+  graph->setDevice(device_);
+  graph->getBackend()->configureDevice(options_);
+  graph->reserveWorkspaceMB(options_->get<size_t>("workspace"));
+
+  // Marian Model: Load from memoryBundle or shortList
+  if (memory_.model.size() > 0 &&
+      memory_.model.begin() !=
+          nullptr) {  // If we have provided a byte array that contains the model memory, we can initialise the
+                      // model from there, as opposed to from reading in the config file
+    ABORT_IF((uintptr_t)memory_.model.begin() % 256 != 0,
+             "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
+    if (options_->get<bool>("check-bytearray", false)) {
+      ABORT_IF(!validateBinaryModel(memory_.model, memory_.model.size()),
+               "The binary file is invalid. Incomplete or corrupted download?");
+    }
+    const std::vector<const void *> container = {
+        memory_.model.begin()};  // Marian supports multiple models initialised in this manner hence std::vector.
+                                 // However we will only ever use 1 during decoding.
+    scorerEnsemble = createScorers(options_, container);
+  } else {
+    scorerEnsemble = createScorers(options_);
+  }
+  for (auto scorer : scorerEnsemble) {
+    scorer->init(graph);
+    if (shortlistGenerator_) {
+      scorer->setShortlistGenerator(shortlistGenerator_);
+    }
+  }
+  graph->forward();
+}
+
+// Make request process is shared between Async and Blocking workflow of translating.
+Ptr<Request> TranslationModel::makeRequest(size_t requestId, std::string &&source, CallbackType callback,
+                                           const ResponseOptions &responseOptions) {
+  Segments segments;
+  AnnotatedText annotatedSource;
+
+  textProcessor_.process(std::move(source), annotatedSource, segments);
+  ResponseBuilder responseBuilder(responseOptions, std::move(annotatedSource), vocabs_, callback, *qualityEstimator_);
+
+  Ptr<Request> request = New<Request>(requestId, std::move(segments), std::move(responseBuilder));
+  return request;
+}
+
+Ptr<marian::data::CorpusBatch> TranslationModel::convertToMarianBatch(Batch &batch) {
+  std::vector<data::SentenceTuple> batchVector;
+  auto &sentences = batch.sentences();
+
+  size_t batchSequenceNumber{0};
+  for (auto &sentence : sentences) {
+    data::SentenceTuple sentence_tuple(batchSequenceNumber);
+    Segment segment = sentence.getUnderlyingSegment();
+    sentence_tuple.push_back(segment);
+    batchVector.push_back(sentence_tuple);
+
+    ++batchSequenceNumber;
+  }
+
+  // Usually one would expect inputs to be [B x T], where B = batch-size and T = max seq-len among the sentences in the
+  // batch. However, marian's library supports multi-source and ensembling through different source-vocabulary but same
+  // target vocabulary. This means the inputs are 3 dimensional when converted into marian's library formatted batches.
+  //
+  // Consequently B x T projects to N x B x T, where N = ensemble size. This adaptation does not fully force the idea of
+  // N = 1 (the code remains general, but N iterates only from 0-1 in the nested loop).
+
+  size_t batchSize = batchVector.size();
+
+  std::vector<size_t> sentenceIds;
+  std::vector<int> maxDims;
+
+  for (auto &example : batchVector) {
+    if (maxDims.size() < example.size()) {
+      maxDims.resize(example.size(), 0);
+    }
+    for (size_t i = 0; i < example.size(); ++i) {
+      if (example[i].size() > static_cast<size_t>(maxDims[i])) {
+        maxDims[i] = static_cast<int>(example[i].size());
+      }
+    }
+    sentenceIds.push_back(example.getId());
+  }
+
+  using SubBatch = marian::data::SubBatch;
+  std::vector<Ptr<SubBatch>> subBatches;
+  for (size_t j = 0; j < maxDims.size(); ++j) {
+    subBatches.emplace_back(New<SubBatch>(batchSize, maxDims[j], vocabs_.sources().at(j)));
+  }
+
+  std::vector<size_t> words(maxDims.size(), 0);
+  for (size_t i = 0; i < batchSize; ++i) {
+    for (size_t j = 0; j < maxDims.size(); ++j) {
+      for (size_t k = 0; k < batchVector[i][j].size(); ++k) {
+        subBatches[j]->data()[k * batchSize + i] = batchVector[i][j][k];
+        subBatches[j]->mask()[k * batchSize + i] = 1.f;
+        words[j]++;
+      }
+    }
+  }
+
+  for (size_t j = 0; j < maxDims.size(); ++j) {
+    subBatches[j]->setWords(words[j]);
+  }
+
+  using CorpusBatch = marian::data::CorpusBatch;
+  Ptr<CorpusBatch> corpusBatch = New<CorpusBatch>(subBatches);
+  corpusBatch->setSentenceIds(sentenceIds);
+  return corpusBatch;
+}
+
+void TranslationModel::translateBatch(size_t deviceId, Batch &batch) {
+  auto &backend = backend_[deviceId];
+  BeamSearch search(options_, backend.scorerEnsemble, vocabs_.target());
+  Histories histories = search.search(backend.graph, convertToMarianBatch(batch));
+  batch.completeBatch(histories);
+}
+
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/translation_model.h b/src/translator/translation_model.h
new file mode 100644
index 000000000..599e6c707
--- /dev/null
+++ b/src/translator/translation_model.h
@@ -0,0 +1,122 @@
+#ifndef SRC_BERGAMOT_TRANSLATION_MODEL_H_
+#define SRC_BERGAMOT_TRANSLATION_MODEL_H_
+
+#include <string>
+#include <vector>
+
+#include "batch.h"
+#include "batching_pool.h"
+#include "common/utils.h"
+#include "data/shortlist.h"
+#include "definitions.h"
+#include "parser.h"
+#include "request.h"
+#include "text_processor.h"
+#include "translator/history.h"
+#include "translator/scorers.h"
+#include "vocabs.h"
+
+namespace marian {
+namespace bergamot {
+
+/// A TranslationModel is associated with the translation of a single language direction. Holds the graph and other
+/// structures required to run the forward pass of the neural network, along with preprocessing logic (TextProcessor)
+/// and a BatchingPool to create batches that are to be used in conjuction with an instance.
+///
+/// Thread-safety is not handled here, but the methods are available at granularity enough to be used in threaded async
+/// workflow for translation.
+
+class TranslationModel {
+ public:
+  using Config = Ptr<Options>;
+  using ShortlistGenerator = Ptr<data::ShortlistGenerator const>;
+
+  /// Equivalent to options based constructor, where `options` is parsed from string configuration. Configuration can be
+  /// JSON or YAML. Keys expected correspond to those of `marian-decoder`, available at
+  /// https://marian-nmt.github.io/docs/cmd/marian-decoder/
+  ///
+  /// Note that `replicas` is not stable. This is a temporary workaround while a more daunting task of separating
+  /// workspace from TranslationModel and binding it to threads is to be undertaken separately. Until the separation is
+  /// achieved, both TranslationModel and Service will need to be aware of workers. This is expected to be resolved
+  /// eventually, with only Service having the knowledge of how many workers are active.
+  ///
+  /// WebAssembly uses only single-thread, and we can hardcode replicas = 1 and use it anywhere and (client) needn't be
+  /// aware of this ugliness at the moment, thus providing a stable API solely for WebAssembly single-threaded modus
+  /// operandi.
+  ///
+  /// TODO(@jerinphilip): Clean this up.
+  TranslationModel(const std::string& config, MemoryBundle&& memory, size_t replicas = 1)
+      : TranslationModel(parseOptionsFromString(config, /*validate=*/false), std::move(memory), replicas){};
+
+  /// Construct TranslationModel from marian-options. If memory is empty, TranslationModel is initialized from
+  /// paths available in the options object, backed by filesystem. Otherwise, TranslationModel is initialized from the
+  /// given MemoryBundle composed of AlignedMemory holding equivalent parameters.
+  ///
+  /// @param [in] options: Marian options object.
+  /// @param [in] memory: MemoryBundle object holding memory buffers containing parameters to build MarianBackend,
+  /// ShortlistGenerator, Vocabs and SentenceSplitter.
+  TranslationModel(const Config& options, MemoryBundle&& memory = MemoryBundle{}, size_t replicas = 1);
+
+  /// Make a Request to be translated by this TranslationModel instance.
+  /// @param [in] requestId: Unique identifier associated with this request, available from Service.
+  /// @param [in] source: Source text to be translated. Ownership is accepted and eventually returned to the client in
+  /// Response corresponding to the Request created here.
+  /// @param [in] callback: Callback (from client) to be issued upon completion of translation of all sentences in the
+  /// created Request.
+  /// @param [in] responseOptions: Configuration used to prepare the Response corresponding to the created request.
+  //  @returns Request created from the query parameters wrapped within a shared-pointer.
+  Ptr<Request> makeRequest(size_t requestId, std::string&& source, CallbackType callback,
+                           const ResponseOptions& responseOptions);
+
+  /// Relays a request to the batching-pool specific to this translation model.
+  /// @param [in] request: Request constructed through makeRequest
+  void enqueueRequest(Ptr<Request> request) { batchingPool_.enqueueRequest(request); };
+
+  /// Generates a batch from the batching-pool for this translation model, compiling from several active requests. Note
+  /// that it is possible that calls to this method can give empty-batches.
+  ///
+  /// @param [out] batch: Batch to write a generated batch on to.
+  /// @returns number of sentences that constitute the Batch.
+  size_t generateBatch(Batch& batch) { return batchingPool_.generateBatch(batch); }
+
+  /// Translate a batch generated with generateBatch
+  ///
+  /// @param [in] deviceId: There are replicas of backend created for use in each worker thread. deviceId indicates
+  /// which replica to use.
+  /// @param [in] batch: A batch generated from generateBatch from the same TranslationModel instance.
+  void translateBatch(size_t deviceId, Batch& batch);
+
+ private:
+  Config options_;
+  MemoryBundle memory_;
+  Vocabs vocabs_;
+  TextProcessor textProcessor_;
+
+  /// Maintains sentences from multiple requests bucketed by length and sorted by priority in each bucket.
+  BatchingPool batchingPool_;
+
+  /// A package of marian-entities which form a backend to translate.
+  struct MarianBackend {
+    using Graph = Ptr<ExpressionGraph>;
+    using ScorerEnsemble = std::vector<Ptr<Scorer>>;
+
+    Graph graph;
+    ScorerEnsemble scorerEnsemble;
+  };
+
+  // ShortlistGenerator is purely const, we don't need one per thread.
+  ShortlistGenerator shortlistGenerator_;
+
+  /// Hold replicas of the backend (graph, scorers, shortlist) for use in each thread.
+  /// Controlled and consistent external access via graph(id), scorerEnsemble(id),
+  std::vector<MarianBackend> backend_;
+  std::shared_ptr<QualityEstimator> qualityEstimator_;
+
+  void loadBackend(size_t idx);
+  Ptr<marian::data::CorpusBatch> convertToMarianBatch(Batch& batch);
+};
+
+}  // namespace bergamot
+}  // namespace marian
+
+#endif  //  SRC_BERGAMOT_TRANSLATION_MODEL_H_
diff --git a/wasm/bindings/service_bindings.cpp b/wasm/bindings/service_bindings.cpp
index 416a318ad..d05cf57cf 100644
--- a/wasm/bindings/service_bindings.cpp
+++ b/wasm/bindings/service_bindings.cpp
@@ -8,8 +8,10 @@
 
 using namespace emscripten;
 
-typedef marian::bergamot::Service Service;
-typedef marian::bergamot::AlignedMemory AlignedMemory;
+using BlockingService = marian::bergamot::BlockingService;
+using TranslationModel = marian::bergamot::TranslationModel;
+using AlignedMemory = marian::bergamot::AlignedMemory;
+using MemoryBundle = marian::bergamot::MemoryBundle;
 
 val getByteArrayView(AlignedMemory& alignedMemory) {
   return val(typed_memory_view(alignedMemory.size(), alignedMemory.as<char>()));
@@ -42,9 +44,9 @@ std::vector<std::shared_ptr<AlignedMemory>> prepareVocabsSmartMemories(std::vect
   return vocabsSmartMemories;
 }
 
-marian::bergamot::MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory, AlignedMemory* shortlistMemory,
-                                                   std::vector<AlignedMemory*> uniqueVocabsMemories) {
-  marian::bergamot::MemoryBundle memoryBundle;
+MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory, AlignedMemory* shortlistMemory,
+                                 std::vector<AlignedMemory*> uniqueVocabsMemories) {
+  MemoryBundle memoryBundle;
   memoryBundle.model = std::move(*modelMemory);
   memoryBundle.shortlist = std::move(*shortlistMemory);
   memoryBundle.vocabs = std::move(prepareVocabsSmartMemories(uniqueVocabsMemories));
@@ -52,18 +54,31 @@ marian::bergamot::MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory, A
   return memoryBundle;
 }
 
-Service* ServiceFactory(const std::string& config, AlignedMemory* modelMemory, AlignedMemory* shortlistMemory,
-                        std::vector<AlignedMemory*> uniqueVocabsMemories) {
-  return new Service(config, std::move(prepareMemoryBundle(modelMemory, shortlistMemory, uniqueVocabsMemories)));
+// This allows only shared_ptrs to be operational in JavaScript, according to emscripten.
+// https://emscripten.org/docs/porting/connecting_cpp_and_javascript/embind.html#smart-pointers
+std::shared_ptr<TranslationModel> TranslationModelFactory(const std::string& config, AlignedMemory* model,
+                                                          AlignedMemory* shortlist,
+                                                          std::vector<AlignedMemory*> vocabs) {
+  MemoryBundle memoryBundle = prepareMemoryBundle(model, shortlist, vocabs);
+  return std::make_shared<TranslationModel>(config, std::move(memoryBundle));
 }
 
-EMSCRIPTEN_BINDINGS(translation_service) {
-  class_<Service>("Service")
-      .constructor(&ServiceFactory, allow_raw_pointers())
-      .function("translate", &Service::translateMultiple)
-      .function("isAlignmentSupported", &Service::isAlignmentSupported);
-  // ^ We redirect Service::translateMultiple to WASMBound::translate instead. Sane API is
-  // translate. If and when async comes, we can be done with this inconsistency.
+EMSCRIPTEN_BINDINGS(translation_model) {
+  class_<TranslationModel>("TranslationModel")
+      .smart_ptr_constructor("TranslationModel", &TranslationModelFactory, allow_raw_pointers());
+}
+
+EMSCRIPTEN_BINDINGS(blocking_service_config) {
+  value_object<BlockingService::Config>("BlockingServiceConfig");
+  // .field("name", &BlockingService::Config::name")
+  // The above is a future hook. Note that more will come - for cache, for workspace-size or graph details  limits on
+  // aggregate-batching etc.
+}
+
+EMSCRIPTEN_BINDINGS(blocking_service) {
+  class_<BlockingService>("BlockingService")
+      .constructor<BlockingService::Config>()
+      .function("translate", &BlockingService::translateMultiple);
 
   register_vector<std::string>("VectorString");
 }

From c7b626dfd0217471db5f034b03fe68e6ac933d0f Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Tue, 28 Sep 2021 15:53:02 +0530
Subject: [PATCH 291/442] Adapted wasm test page for new Service interface
 (#224)

- The new interface now supports running multiple TranslationModels
---
 wasm/test_page/bergamot.js |  12 +++-
 wasm/test_page/worker.js   | 120 ++++++++++++++++++++++---------------
 2 files changed, 81 insertions(+), 51 deletions(-)

diff --git a/wasm/test_page/bergamot.js b/wasm/test_page/bergamot.js
index e586b213c..848fba177 100644
--- a/wasm/test_page/bergamot.js
+++ b/wasm/test_page/bergamot.js
@@ -32,14 +32,20 @@ document.querySelector("#load").addEventListener("click", async() => {
 const translateCall = () => {
     const text = document.querySelector('#from').value;
     const paragraphs = text.split("\n");
-
-    worker.postMessage(["translate", paragraphs]);
+    document.querySelector("#load").disabled = true;
+    const lang = document.querySelector('input[name="modellang"]:checked').value;
+    const from = lang.substring(0, 2);
+    const to = lang.substring(2, 4);
+    worker.postMessage(["translate", from, to, paragraphs]);
+    document.querySelector("#load").disabled = false;
 }
 
 worker.onmessage = function(e) {
     console.debug(`Message received from worker`);
     if (e.data[0] === 'translated_result') {
-        document.querySelector('#to').value = e.data[1].join("\n");
+        if (e.data[1]) {
+            document.querySelector('#to').value = e.data[1].join("\n");
+        }
         log(e.data[2]);
     }
     if ((e.data[0] === 'module_loaded') || (e.data[0] === 'model_loaded')) {
diff --git a/wasm/test_page/worker.js b/wasm/test_page/worker.js
index 329081011..8b53a271a 100644
--- a/wasm/test_page/worker.js
+++ b/wasm/test_page/worker.js
@@ -1,4 +1,6 @@
 var translationService, responseOptions, input = undefined;
+// A map of language-pair to TranslationModel object
+var translationModels = new Map();
 const BERGAMOT_TRANSLATOR_MODULE = "bergamot-translator-worker.js";
 
 const encoder = new TextEncoder(); // string to utf-8 converter
@@ -33,23 +35,35 @@ onmessage = async function(e) {
     }
     else if (command === 'load_model') {
         let start = Date.now();
-        await constructTranslationService(e.data[1], e.data[2]);
-        result = `translation model '${e.data[1]}${e.data[2]}' successfully loaded; took ${(Date.now() - start) / 1000} secs`;
+        try {
+            await constructTranslationService();
+            await constructTranslationModel(e.data[1], e.data[2]);
+            result = `translation model '${e.data[1]}${e.data[2]}' successfully loaded; took ${(Date.now() - start) / 1000} secs`;
+        } catch (error) {
+            result = `translation model '${e.data[1]}${e.data[2]}' loading failed: '${error.message}'`;
+        }
         log(result);
         log('Posting message back to main script');
         postMessage(['model_loaded', result]);
     }
     else if (command === 'translate') {
-        const inputParagraphs = e.data[1];
+        const from = e.data[1];
+        const to = e.data[2];
+        const inputParagraphs = e.data[3];
         let inputWordCount = 0;
         inputParagraphs.forEach(sentence => {
             inputWordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
         })
 
         let start = Date.now();
-        const translatedParagraphs = translate(e.data[1]);
-        const secs = (Date.now() - start) / 1000;
-        result = `Translation of (${inputWordCount}) words took ${secs} secs (${Math.round(inputWordCount / secs)} words per second)`;
+        var translatedParagraphs;
+        try {
+            translatedParagraphs  = translate(from, to, inputParagraphs);
+            const secs = (Date.now() - start) / 1000;
+            result = `Translation '${from}${to}' Successful. Speed: ${Math.round(inputWordCount / secs)} Words per second (${inputWordCount} words in ${secs} secs)`;
+        } catch (error) {
+            result = `Error: ${error.message}`;
+        }
         log(result);
         log('Posting message back to main script');
         postMessage(['translated_result', translatedParagraphs, result]);
@@ -77,8 +91,24 @@ const prepareAlignedMemoryFromBuffer = async (buffer, alignmentSize) => {
     return alignedMemory;
 }
 
-const constructTranslationService = async (from, to) => {
+// Instantiate the Translation Service
+const constructTranslationService = async () => {
+    if (!translationService) {
+        var translationServiceConfig = {};
+        log(`Creating Translation Service with config: ${translationServiceConfig}`);
+        translationService = new Module.BlockingService(translationServiceConfig);
+        log(`Translation Service created successfully`);
+    }
+}
+
+const constructTranslationModel = async (from, to) => {
     const languagePair = `${from}${to}`;
+    if (translationModels.has(languagePair)) {
+        var oldModel = translationModels.get(languagePair);
+        // Destruct the old TranslationModel explicitly and Remove its entry from the map
+        oldModel.delete();
+        translationModels.delete(languagePair);
+    }
 
     // Vocab files are re-used in both translation directions
     const vocabLanguagePair = from === "en" ? `${to}${from}` : languagePair;
@@ -133,50 +163,44 @@ gemm-precision: int8shift
     log(`modelFile: ${modelFile}\nshortlistFile: ${shortlistFile}\nNo. of unique vocabs: ${uniqueVocabFiles.size}`);
     uniqueVocabFiles.forEach(item => log(`unique vocabFile: ${item}`));
 
-    try {
-      // Download the files as buffers from the given urls
-        let start = Date.now();
-        const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]);
-        const modelBuffer = downloadedBuffers[0];
-        const shortListBuffer = downloadedBuffers[1];
+    // Download the files as buffers from the given urls
+    let start = Date.now();
+    const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]);
+    const modelBuffer = downloadedBuffers[0];
+    const shortListBuffer = downloadedBuffers[1];
 
-        const downloadedVocabBuffers = [];
-        for (let item of uniqueVocabFiles.values()) {
-            downloadedVocabBuffers.push(await downloadAsArrayBuffer(item));
-        }
-        log(`All files for ${languagePair} language pair took ${(Date.now() - start) / 1000} secs to download`);
-
-        // Construct AlignedMemory objects with downloaded buffers
-        let constructedAlignedMemories = await Promise.all([prepareAlignedMemoryFromBuffer(modelBuffer, 256),
-                                                                prepareAlignedMemoryFromBuffer(shortListBuffer, 64)]);
-        let alignedModelMemory = constructedAlignedMemories[0];
-        let alignedShortlistMemory = constructedAlignedMemories[1];
-        let alignedVocabsMemoryList = new Module.AlignedMemoryList;
-        for(let item of downloadedVocabBuffers) {
-            let alignedMemory = await prepareAlignedMemoryFromBuffer(item, 64);
-            alignedVocabsMemoryList.push_back(alignedMemory);
-        }
-        log(`Aligned vocab memories: ${alignedVocabsMemoryList.get(0).size()}`);
-        log(`Aligned model memory: ${alignedModelMemory.size()}`);
-        log(`Aligned shortlist memory: ${alignedShortlistMemory.size()}`);
-
-        // Instantiate the Translation Service
-        if (translationService) {
-            translationService.delete();
-            translationService = undefined;
-        }
+    const downloadedVocabBuffers = [];
+    for (let item of uniqueVocabFiles.values()) {
+        downloadedVocabBuffers.push(await downloadAsArrayBuffer(item));
+    }
+    log(`All files for ${languagePair} language pair took ${(Date.now() - start) / 1000} secs to download`);
 
-        log(`Creating Translation Service with config: ${modelConfig}`);
-        translationService = new Module.Service(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
-        if (typeof translationService === 'undefined') {
-            throw Error(`Translation Service construction failed`);
-        }
-    } catch (error) {
-        log(error);
+    // Construct AlignedMemory objects with downloaded buffers
+    let constructedAlignedMemories = await Promise.all([prepareAlignedMemoryFromBuffer(modelBuffer, 256),
+                                                            prepareAlignedMemoryFromBuffer(shortListBuffer, 64)]);
+    let alignedModelMemory = constructedAlignedMemories[0];
+    let alignedShortlistMemory = constructedAlignedMemories[1];
+    let alignedVocabsMemoryList = new Module.AlignedMemoryList;
+    for(let item of downloadedVocabBuffers) {
+        let alignedMemory = await prepareAlignedMemoryFromBuffer(item, 64);
+        alignedVocabsMemoryList.push_back(alignedMemory);
+    }
+    log(`Aligned vocab memories: ${alignedVocabsMemoryList.get(0).size()}`);
+    log(`Aligned model memory: ${alignedModelMemory.size()}`);
+    log(`Aligned shortlist memory: ${alignedShortlistMemory.size()}`);
+
+    log(`Creating Translation Model with config: ${modelConfig}`);
+    var translationModel = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
+    translationModels.set(languagePair, translationModel);
+}
+
+const translate = (from, to, paragraphs) => {
+    const languagePair = `${from}${to}`;
+    if (!translationModels.has(languagePair)) {
+        throw Error(`Please load translation model '${languagePair}' before translating`);
     }
-  }
+    translationModel = translationModels.get(languagePair);
 
-const translate = (paragraphs) => {
     // Instantiate the arguments of translate() API i.e. ResponseOptions and input (vector<string>)
     var responseOptions = new Module.ResponseOptions();
     let input = new Module.VectorString;
@@ -193,7 +217,7 @@ const translate = (paragraphs) => {
     log(`Input size: ${input.size()}`);
 
     // Translate the input, which is a vector<String>; the result is a vector<Response>
-    let result = translationService.translate(input, responseOptions);
+    let result = translationService.translate(translationModel, input, responseOptions);
 
     const translatedParagraphs = [];
     const translatedSentencesOfParagraphs = [];

From a0cb1e4b3d2e06027a7f979b0c66f1336a6688e9 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Tue, 19 Oct 2021 14:40:54 +0200
Subject: [PATCH 292/442] Wasm test page UI for translating b/w non-English
 language pairs (#231)

* Updated Wasm test page UI for translating b/w non-English language pairs
* Both "from" and "to" language dropdowns now allow non-English languages
---
 wasm/README.md                        | 106 +-----
 wasm/test_page/bergamot-httpserver.js |  61 ++-
 wasm/test_page/bergamot.html          |  66 ----
 wasm/test_page/bergamot.js            |  54 ---
 wasm/test_page/css/index.css          |  99 +++++
 wasm/test_page/helper.js              |  40 --
 wasm/test_page/index.html             |  33 ++
 wasm/test_page/js/index.js            | 101 +++++
 wasm/test_page/js/modelRegistry.js    | 328 ++++++++++++++++
 wasm/test_page/js/worker.js           | 298 +++++++++++++++
 wasm/test_page/package-lock.json      | 515 +++++++++++++++++++++++++-
 wasm/test_page/start_server.sh        |   6 +-
 wasm/test_page/worker.js              | 267 -------------
 13 files changed, 1452 insertions(+), 522 deletions(-)
 delete mode 100644 wasm/test_page/bergamot.html
 delete mode 100644 wasm/test_page/bergamot.js
 create mode 100644 wasm/test_page/css/index.css
 delete mode 100644 wasm/test_page/helper.js
 create mode 100644 wasm/test_page/index.html
 create mode 100644 wasm/test_page/js/index.js
 create mode 100644 wasm/test_page/js/modelRegistry.js
 create mode 100644 wasm/test_page/js/worker.js
 delete mode 100644 wasm/test_page/worker.js

diff --git a/wasm/README.md b/wasm/README.md
index 728b0a364..a0b3d7820 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -1,95 +1,25 @@
 # Using Bergamot Translator in JavaScript
 
-Instructions in this document assume current-directory to be
-[wasm](https://github.com/browsermt/bergamot-translator/tree/main/wasm) within
-bergamot-translator source.
-
-The example file `bergamot.html` in the folder `test_page` demonstrates how to
-use the bergamot translator in JavaScript via a `<script>` tag.
-
-## Pre-requisites
-
-**Download files required for translation**
-
-Please note that [Using JS APIs](#using-js-apis) and [Demo](#demo) section below assumes that the [bergamot project specific model files](https://github.com/mozilla-applied-ml/bergamot-models) are already downloaded and present in the `test_page` folder. If this is not done then use following instructions to do so:
-
-```bash
-cd test_page
-git clone --depth 1 --branch main --single-branch https://github.com/mozilla-applied-ml/bergamot-models
-mkdir models
-cp -rf bergamot-models/prod/* models
-gunzip models/*/*
-```
+All the instructions below are meant to run from the current directory.
 
 ## Using JS APIs
 
-```js
-// The model configuration as YAML formatted string. For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-// This example captures some of the most relevant options
-const modelConfig = `beam-size: 1
-normalize: 1.0
-word-penalty: 0
-max-length-break: 128
-mini-batch-words: 1024
-workspace: 128
-max-length-factor: 2.0
-skip-cost: true
-cpu-threads: 0
-quiet: true
-quiet-translation: true
-gemm-precision: int8shift
-`;
-
-// Download model, shortlist and vocabulary files and read them into buffers
-const modelFile = `models/esen/model.esen.intgemm.alphas.bin`;
-const shortlistFile = `models/esen/lex.50.50.esen.s2t.bin`;
-const vocabFiles = [`models/${languagePair}/vocab.${vocabLanguagePair}.spm`,
-                    `models/${languagePair}/vocab.${vocabLanguagePair}.spm`];
-const uniqueVocabFiles = new Set(vocabFiles);
-
-// Please refer to bergamot.html in test_page folder for downloadAsArrayBuffer function
-const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]);
-const downloadedVocabBuffers = [];
-for (let item of uniqueVocabFiles.values()) {
-  downloadedVocabBuffers.push(await downloadAsArrayBuffer(item));
-}
-
-const modelBuffer = downloadedBuffers[0];
-const shortListBuffer = downloadedBuffers[1];
+Please refer to the file `test_page/js/worker.js` that demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
 
-// Construct AlignedMemory instances from the buffers
-var alignedModelMemory = constructAlignedMemoryFromBuffer(modelBuffer, 256); // Please refer to bergamot.html in test_page folder for this function
-var alignedShortlistMemory = constructAlignedMemoryFromBuffer(shortListBuffer, 64); // Please refer to bergamot.html in test_page folder for this function
-var alignedVocabsMemoryList = new Module.AlignedMemoryList;
-downloadedVocabBuffers.forEach(item => alignedVocabsMemoryList.push_back(constructAlignedMemoryFromBuffer(item, 64)));
+## Demo
 
-// Instantiate the Translation Service
-const translationService = new Module.Service(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
+* Download bergamot model files required for translation
 
-// Instantiate the arguments of translate() API i.e. ResponseOptions and input (vector<string>)
-const responseOptions = new Module.ResponseOptions();
-const input = new Module.VectorString;
+    Use following instructions to download [model files](https://github.com/mozilla/firefox-translations-models/) (make sure that `git-lfs` is installed and initialized before running these instructions):
 
-// Initialize the input
-input.push_back("Hola"); input.push_back("Mundo");
-
-// translate the input; the result is a vector<Response>
-const result = translationService.translate(input, responseOptions);
-
-// Print original and translated text from each entry of vector<Response>
-for (let i = 0; i < result.size(); i++) {
-    console.log(' original=' + result.get(i).getOriginalText() + ', translation=' + result.get(i).getTranslatedText());
-}
-
-// Don't forget to clean up the instances
-translationService.delete();
-responseOptions.delete();
-input.delete();
-```
-
-## Demo 
-
-* Make sure that you followed [Pre-requisites](#pre-requisites) instructions before moving forward.
+    ```bash
+    cd test_page
+    git clone --depth 1 --branch main --single-branch https://github.com/mozilla/firefox-translations-models/
+    mkdir models
+    cp -rf firefox-translations-models/models/prod/* models
+    cp -rf firefox-translations-models/models/dev/* models
+    gunzip models/*/*
+    ```
 
 * Start the test webserver (ensure you have the latest nodejs installed)
     ```bash
@@ -114,10 +44,10 @@ input.delete();
 
 * Browse to the following page:
     ```
-    http://localhost:8000/bergamot.html
+    http://localhost:80
     ```
 
-* Run some translations:
-    * Choose a model and press `Load Model`
-    * Type a sentence to be translated in the `From` textbox and press `Translate`
-    * See the results in the `To` and `Log` textboxes
+* Perform translations:
+    * Choose the source and target languages using `From` and `To` dropdowns.
+    * Type a sentence to be translated in the `From` textbox.
+    * See the result in the `To` textbox.
diff --git a/wasm/test_page/bergamot-httpserver.js b/wasm/test_page/bergamot-httpserver.js
index f657de3e5..273235e11 100644
--- a/wasm/test_page/bergamot-httpserver.js
+++ b/wasm/test_page/bergamot-httpserver.js
@@ -1,6 +1,5 @@
-require(__dirname  + '/helper.js');
-
 const http = require('http');
+const https = require('https')
 const express = require('express');
 const app = express();
 const server = http.createServer(app);
@@ -8,11 +7,36 @@ const fs = require('fs');
 const url = require('url');
 const nocache = require('nocache');
 const cors = require('cors');
+const path = require('path');
 
 let port = 8000;
+if (process.argv[2]) {
+    port = process.argv[2];
+}
+
+let skipssl = 0;
+if (process.argv[3]) {
+    skipssl = process.argv[3];
+}
+
+let certpath = "/etc/letsencrypt";
+if (process.argv[4]) {
+    certpath = process.argv[4];
+}
 
 app.use(cors())
 app.use(nocache());
+
+app.get('/', cors(), function(req, res) {
+    if (!req.secure && skipssl != 1) {
+        return res.redirect("https://" + req.headers.host + req.url);
+    }
+    res.sendFile(path.join(__dirname + '/index.html'));
+    res.header('Cross-Origin-Embedder-Policy','require-corp');
+    res.header('Cross-Origin-Opener-Policy','same-origin');
+    res.header('Cross-Origin-Resource-Policy','same-origin');
+});
+
 app.get('/*.*' , cors(), function(req, res) {
     var options = url.parse(req.url, true);
     var mime = Helper.getMime(options);
@@ -34,5 +58,36 @@ function serveFile(res, pathName, mime) {
     });
 }
 
+if (skipssl != 1){
+    https.createServer({
+            key: fs.readFileSync(`${certpath}/privkey.pem`),
+            cert: fs.readFileSync(`${certpath}/cert.pem`),
+            ca: fs.readFileSync(`${certpath}/chain.pem`),
+        },
+        app
+    ).listen(443, () => {
+        console.log('Listening https port 443')
+    })
+}
+
+const Helper = {
+    types: {
+       "wasm" : "application/wasm"
+       , "js" : "application/javascript"
+       , "html" : "text/html"
+       , "htm" : "text/html"
+       , "ico" : "image/vnd.microsoft.icon"
+       , "css" : "text/css"
+    },
+    getMime: function(u) {
+        var ext = this.getExt(u.pathname).replace('.', '');
+        return this.types[ext.toLowerCase()] || 'application/octet-stream';
+    },
+    getExt: function(path) {
+        var i = path.lastIndexOf('.');
+        return (i < 0) ? '' : path.substr(i);
+    }
+};
+
 server.listen(port);
-console.log(`HTTP and BinaryJS server started on port ${port}`);
+console.log(`HTTP and BinaryJS server started on port ${port}`);
\ No newline at end of file
diff --git a/wasm/test_page/bergamot.html b/wasm/test_page/bergamot.html
deleted file mode 100644
index 8c80ed2cd..000000000
--- a/wasm/test_page/bergamot.html
+++ /dev/null
@@ -1,66 +0,0 @@
-<!doctype html>
-<html>
-<head>
-    <link rel="icon" href="data:,">
-    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
-</head>
-<style>
-    body, html, div {
-        margin-left: 1%;
-        margin-right: 1%;
-        margin-bottom: 1%;
-        margin-top: 1%;
-        padding-left: 1%;
-        padding-right: 1%;
-        padding-bottom: 1%;
-        padding-top: 1%;
-    }
-
-    textarea, #to, #from {
-        width: 100%;
-        max-width: 100%;
-    }
-
-    div {
-        float: left;
-        width: 80%;
-    }
-</style>
-<body>
-
-<div id="divradios">
-    <label>Choose the model to use</label>
-    <input type="radio" name="modellang" value="enes"/><label>English to Spanish</label>
-    <input type="radio" name="modellang" value="esen" checked/><label>Spanish to English</label>
-    <input type="radio" name="modellang" value="eten" checked/><label>Estonian to English</label>
-    <input type="radio" name="modellang" value="enet" checked/><label>English to Estonian</label>
-    <input type="radio" name="modellang" value="ende" checked/><label>English to German</label>
-    <input type="button" id="load" value="Load Model"/>
-</div>
-
-<div id="divtranslation">
-    <label for="from">From</label>
-    <textarea id="from" name="from">
-Una estrategia republicana para obstaculizar la reelección de Obama. Los dirigentes republicanos justificaron su política por la necesidad de luchar contra el fraude electoral.
-Ahora bien, el Centro Brennan considera esto último un mito y afirma que el fraude electoral es menos frecuente en los Estados Unidos que el número de personas que mueren a causa de la caída de un rayo.
-De hecho, los abogados republicanos no han encontrado más que 300 casos de fraude electoral en los Estados Unidos en diez años. Una cosa es cierta: esas nuevas disposiciones afectarán negativamente a la tasa de participación.
-En ese sentido, estas medidas minarán en parte el sistema democrático americano. Al contrario de lo que ocurre en Canadá, los estados americanos son responsables de la organización de las elecciones federales en los Estados Unidos.
-Y en esa misma línea una mayoría de los gobiernos americanos promulgaron, a partir de 2009, nuevas leyes que dificultaban el proceso de inscripción o de votación. Este fenómeno se ha extendido tras las elecciones de noviembre de 2010, que vieron el aumento de 675 nuevos representantes republicanos en 26 estados.
-En consecuencia, durante el año 2011 se introdujeron 180 proyectos de ley que restringían el ejercicio del derecho de voto en 41 estados.
-    </textarea>
-    <br><br>
-    <label for="to">To</label>
-    <textarea id="to" name="to" readonly></textarea>
-    <br><br>
-    <input type="button" id="translate" value="Translate"/>
-</div>
-
-<div id="divlog">
-    <label for="log">Log:</label><br>
-    <textarea id="log" name="log" rows="50" cols="75"></textarea>
-</div>
-
-<script src="bergamot.js"></script>
-<script src="bergamot-translator-worker.js"></script>
-</body>
-</html>
diff --git a/wasm/test_page/bergamot.js b/wasm/test_page/bergamot.js
deleted file mode 100644
index 848fba177..000000000
--- a/wasm/test_page/bergamot.js
+++ /dev/null
@@ -1,54 +0,0 @@
-var worker;
-
-if (window.Worker) {
-    var worker = new Worker('worker.js');
-    worker.postMessage(["load_module"]);
-}
-
-const log = (message) => {
-    document.querySelector("#log").value += message + "\n";
-}
-
-document.querySelector("#translate").addEventListener("click", () => {
-    translateCall();
-});
-
-document.querySelector("#from").addEventListener('keyup', function(event) {
-    if (event.keyCode === 13) {
-        translateCall();
-    }
-});
-
-document.querySelector("#load").addEventListener("click", async() => {
-    document.querySelector("#load").disabled = true;
-    const lang = document.querySelector('input[name="modellang"]:checked').value;
-    const from = lang.substring(0, 2);
-    const to = lang.substring(2, 4);
-    let start = Date.now();
-    worker.postMessage(["load_model", from, to]);
-    document.querySelector("#load").disabled = false;
-});
-
-const translateCall = () => {
-    const text = document.querySelector('#from').value;
-    const paragraphs = text.split("\n");
-    document.querySelector("#load").disabled = true;
-    const lang = document.querySelector('input[name="modellang"]:checked').value;
-    const from = lang.substring(0, 2);
-    const to = lang.substring(2, 4);
-    worker.postMessage(["translate", from, to, paragraphs]);
-    document.querySelector("#load").disabled = false;
-}
-
-worker.onmessage = function(e) {
-    console.debug(`Message received from worker`);
-    if (e.data[0] === 'translated_result') {
-        if (e.data[1]) {
-            document.querySelector('#to').value = e.data[1].join("\n");
-        }
-        log(e.data[2]);
-    }
-    if ((e.data[0] === 'module_loaded') || (e.data[0] === 'model_loaded')) {
-        log(e.data[1]);
-    }
-}
\ No newline at end of file
diff --git a/wasm/test_page/css/index.css b/wasm/test_page/css/index.css
new file mode 100644
index 000000000..bbc5bf147
--- /dev/null
+++ b/wasm/test_page/css/index.css
@@ -0,0 +1,99 @@
+* {
+  box-sizing: border-box;
+}
+
+html,
+body {
+  height: 100%;
+  margin: 0;
+  font-size: 18px;
+  font-family: Optima, Helvetica, Arial;
+}
+
+body {
+  padding: 1rem;
+}
+
+.app {
+  padding: 1rem;
+  display: grid;
+  grid: "from swap to" 1fr "status status status" auto / 1fr auto 1fr;
+  grid-gap: 1rem;
+  overflow: hidden;
+  min-height: 400px;
+  max-width: 1024px;
+  margin: 1em auto;
+}
+
+@media screen and (max-width: 640px) {
+  .app {
+    grid: "from from" auto "status swap" auto "to to" auto / 1fr;
+  }
+}
+
+.panel {
+  display: grid;
+  grid-template-rows: auto 1fr;
+  grid-gap: 1rem;
+}
+
+label {
+  padding: 0 0.5em;
+  display: flex;
+  align-items: center;
+}
+
+.lang-select {
+  padding: 0.25rem 0.5rem;
+  margin-left: 1rem;
+  background: #f4f4f4;
+  font-size: 0.9rem;
+  border: 1px solid #ccc;
+  border-radius: 0.25rem;
+  cursor: pointer;
+}
+
+.panel--from {
+  grid-area: from;
+}
+
+.panel--to {
+  grid-area: to;
+}
+
+.swap {
+  align-self: center;
+  grid-area: swap;
+  font-size: 1.1rem;
+}
+
+#status {
+  grid-area: status;
+  text-align: center;
+  align-self: center;
+}
+
+textarea {
+  padding: 1rem;
+  font-family: sans-serif;
+  font-size: 1rem;
+  resize: none;
+  border-radius: 2px;
+  border: 1px solid #ccc;
+}
+
+button {
+  cursor: pointer;
+  border: 1px solid #88c;
+  border-radius: 4px;
+  background: #eef;
+  padding: 0;
+  padding: 0.25rem 0.5rem;
+}
+button:hover {
+  background: #cce;
+}
+
+#output {
+  background-color: #f4f4f4;
+}
diff --git a/wasm/test_page/helper.js b/wasm/test_page/helper.js
deleted file mode 100644
index bff116ced..000000000
--- a/wasm/test_page/helper.js
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * @author - Based of a file from Gist here: https://gist.github.com/1757658
- *
- * @modified - Mike Newell - it was on Gist so I figure I can use it
- *
- * @Description -   Added support for a few more mime types including the new
- *                  .ogv, .webm, and .mp4 file types for HTML5 video.
- *
- */
-
-/*
-* @modified - Andre Natal - removed unused types for the purpose of this use
-case
-*/
-
-Helper = {
-
-    types: {
-       "wasm" : "application/wasm"
-       , "js" : "application/javascript"
-       , "html" : "text/html"
-       , "htm" : "text/html"
-       , "ico" : "image/vnd.microsoft.icon",
-    },
-
-    getMime: function(u) {
-
-        var ext = this.getExt(u.pathname).replace('.', '');
-
-        return this.types[ext.toLowerCase()] || 'application/octet-stream';
-
-    },
-
-    getExt: function(path) {
-        var i = path.lastIndexOf('.');
-
-        return (i < 0) ? '' : path.substr(i);
-    }
-
-};
diff --git a/wasm/test_page/index.html b/wasm/test_page/index.html
new file mode 100644
index 000000000..86eae4637
--- /dev/null
+++ b/wasm/test_page/index.html
@@ -0,0 +1,33 @@
+<!DOCTYPE html>
+<html>
+  <head>
+    <title>Mozilla Translations</title>
+    <link rel="stylesheet" href="css/index.css" />
+    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8" />
+    <meta
+      name="viewport"
+      content="width=device-width, initial-scale=1.0, viewport-fit=cover"
+    />
+  </head>
+  <body>
+    <div class="app">
+      <div class="panel panel--from">
+        <label>
+          From
+          <select id="lang-from" name="from" class="lang-select"></select>
+        </label>
+        <textarea id="input" name="input"></textarea>
+      </div>
+      <button class="swap" title="swap">↔️</button>
+      <div class="panel panel--to">
+        <label>
+          To
+          <select id="lang-to" name="to" class="lang-select"></select>
+        </label>
+        <textarea id="output" name="output" readonly></textarea>
+      </div>
+      <div class="footer" id="status"></div>
+    </div>
+    <script src="js/index.js"></script>
+  </body>
+</html>
diff --git a/wasm/test_page/js/index.js b/wasm/test_page/js/index.js
new file mode 100644
index 000000000..6b580415f
--- /dev/null
+++ b/wasm/test_page/js/index.js
@@ -0,0 +1,101 @@
+let worker;
+let modelRegistry;
+
+const $ = selector => document.querySelector(selector);
+const status = message => ($("#status").innerText = message);
+
+const langFrom = $("#lang-from");
+const langTo = $("#lang-to");
+
+const langs = [
+  ["en", "English"],
+  ["it", "Italian"],
+  ["pt", "Portuguese"],
+  ["ru", "Russian"],
+  ["cs", "Czech"],
+  ["de", "German"],
+  ["es", "Spanish"],
+  ["et", "Estonian"],
+];
+
+if (window.Worker) {
+  worker = new Worker("js/worker.js");
+  worker.postMessage(["import"]);
+}
+
+document.querySelector("#input").addEventListener("keyup", function (event) {
+  translateCall();
+});
+
+const translateCall = () => {
+  const text = document.querySelector("#input").value + "  ";
+  if (!text.trim().length) return;
+  const paragraphs = text.split("\n");
+  $("#output").setAttribute("disabled", true);
+  const lngFrom = langFrom.value;
+  const lngTo = langTo.value;
+  worker.postMessage(["translate", lngFrom, lngTo, paragraphs]);
+};
+
+worker.onmessage = function (e) {
+  if (e.data[0] === "translate_reply" && e.data[1]) {
+    document.querySelector("#output").value = e.data[1].join("\n\n");
+    $("#output").removeAttribute("disabled");
+  } else if (e.data[0] === "load_model_reply" && e.data[1]) {
+    status(e.data[1]);
+    translateCall();
+  } else if (e.data[0] === "import_reply" && e.data[1]) {
+    modelRegistry = e.data[1];
+    init();
+  }
+};
+
+langs.forEach(([code, name]) => {
+  langFrom.innerHTML += `<option value="${code}">${name}</option>`;
+  langTo.innerHTML += `<option value="${code}">${name}</option>`;
+});
+
+const loadModel = () => {
+  const lngFrom = langFrom.value;
+  const lngTo = langTo.value;
+  if (lngFrom !== lngTo) {
+    status(`Installing model...`);
+    console.log(`Loading model '${lngFrom}${lngTo}'`);
+    worker.postMessage(["load_model", lngFrom, lngTo]);
+  } else {
+    const input = document.querySelector("#input").value;
+    document.querySelector("#output").value = input;
+  }
+};
+
+langFrom.addEventListener("change", e => {
+  loadModel();
+});
+
+langTo.addEventListener("change", e => {
+  loadModel();
+});
+
+$(".swap").addEventListener("click", e => {
+  [langFrom.value, langTo.value] = [langTo.value, langFrom.value];
+  $("#input").value = $("#output").value;
+  loadModel();
+});
+
+function init() {
+  // try to guess input language from user agent
+  let myLang = navigator.language;
+  if (myLang) {
+    myLang = myLang.split("-")[0];
+    let langIndex = langs.findIndex(([code]) => code === myLang);
+    if (langIndex > -1) {
+      console.log("guessing input language is", myLang);
+      langFrom.value = myLang;
+    }
+  }
+
+  // find first output lang that *isn't* input language
+  langTo.value = langs.find(([code]) => code !== langFrom.value)[0];
+  // load this model
+  loadModel();
+}
diff --git a/wasm/test_page/js/modelRegistry.js b/wasm/test_page/js/modelRegistry.js
new file mode 100644
index 000000000..c8d6eda5e
--- /dev/null
+++ b/wasm/test_page/js/modelRegistry.js
@@ -0,0 +1,328 @@
+
+//const rootURL = "https://storage.googleapis.com/bergamot-models-sandbox/0.2.10";
+const rootURL = "../models";
+
+const modelRegistry = {
+  enit: {
+    vocab: {
+      name: "vocab.enit.spm",
+      size: 814128,
+      estimatedCompressedSize: 405338,
+      expectedSha256Hash:
+        "de8cbeb79e0139304bfa47e8559f2447016bf9906225a97d3df1baed4de8f3a3",
+    },
+    lex: {
+      name: "lex.50.50.enit.s2t.bin",
+      size: 4489920,
+      estimatedCompressedSize: 2409986,
+      expectedSha256Hash:
+        "bb1fad3b3f6a13ebce1698cf7f39ca736c4dea4525f3dab5e1a78436f07445e6",
+    },
+    model: {
+      name: "model.enit.intgemm.alphas.bin",
+      size: 17140836,
+      estimatedCompressedSize: 13283223,
+      expectedSha256Hash:
+        "a5ce3723f62ead92a0e0373b6df0ad8e3e6d22963adb1333984206e33b8b6c61",
+    },
+  },
+  enpt: {
+    vocab: {
+      name: "vocab.enpt.spm",
+      size: 812781,
+      estimatedCompressedSize: 406524,
+      expectedSha256Hash:
+        "633a3d782c79f7d5e4b94ab96848f47c2fdf8ba82dd99efd1742b8a696bbd0cc",
+    },
+    lex: {
+      name: "lex.50.50.enpt.s2t.bin",
+      size: 4472528,
+      estimatedCompressedSize: 2411984,
+      expectedSha256Hash:
+        "1e96599123d275afa37353dfe84677a4070f013494fbdc9c52a28445cc9bc38d",
+    },
+    model: {
+      name: "model.enpt.intgemm.alphas.bin",
+      size: 17140836,
+      estimatedCompressedSize: 13429592,
+      expectedSha256Hash:
+        "d968735704c75e33c2e183b9241f14c0b2a560d01d88a2728e5c0119a4d7fb22",
+    },
+  },
+  enru: {
+    vocab: {
+      name: "vocab.enru.spm",
+      size: 937157,
+      estimatedCompressedSize: 435776,
+      expectedSha256Hash:
+        "feca2d44f01b946c85faba3b15b5eb53344bec84cd14a1a4d4a82ddd774c5edd",
+    },
+    lex: {
+      name: "lex.50.50.enru.s2t.bin",
+      size: 3049096,
+      estimatedCompressedSize: 1579779,
+      expectedSha256Hash:
+        "7bd3e2c0a72286fe1f3da65c56c49a7cd77efa5f1d1a444e2a9e769480b96ff3",
+    },
+    model: {
+      name: "model.enru.intgemm.alphas.bin",
+      size: 17140836,
+      estimatedCompressedSize: 12853987,
+      expectedSha256Hash:
+        "4a45186a93b8a2dd9301c66a3b3dad580b1bcfa74aadda583ca383f9fe0dea93",
+    },
+  },
+  iten: {
+    vocab: {
+      name: "vocab.iten.spm",
+      size: 814151,
+      estimatedCompressedSize: 405416,
+      expectedSha256Hash:
+        "22d5ce6973be5360a921103acbe984a9bfca952a1f6c55c9cb5ef7de4fd58266",
+    },
+    lex: {
+      name: "lex.50.50.iten.s2t.bin",
+      size: 5238420,
+      estimatedCompressedSize: 2860178,
+      expectedSha256Hash:
+        "357d362373022b029ee9965975a133e6f36fdb0fed749202ff578365cf0111f8",
+    },
+    model: {
+      name: "model.iten.intgemm.alphas.bin",
+      size: 17140836,
+      estimatedCompressedSize: 13423308,
+      expectedSha256Hash:
+        "1fae546faeb9046f80b1b7e940b37b660974ce72902778181d6cd1c30b717f35",
+    },
+  },
+  pten: {
+    vocab: {
+      name: "vocab.pten.spm",
+      size: 812889,
+      estimatedCompressedSize: 406730,
+      expectedSha256Hash:
+        "8389979e3c965688b07aeb712a7e44406e5dcdb2b84087229d26fcc71448c4ed",
+    },
+    lex: {
+      name: "lex.50.50.pten.s2t.bin",
+      size: 5001420,
+      estimatedCompressedSize: 2733800,
+      expectedSha256Hash:
+        "212ed0ae44a6f920cd6d17ca02f0a523ba6c4b0ef5078ae310c20bc4c51484c5",
+    },
+    model: {
+      name: "model.pten.intgemm.alphas.bin",
+      size: 17140836,
+      estimatedCompressedSize: 13584764,
+      expectedSha256Hash:
+        "6c3b7af01772022a19712410c63342ba581468c2f1aac34d7488409c4043e697",
+    },
+  },
+  ruen: {
+    vocab: {
+      name: "vocab.ruen.spm",
+      size: 936576,
+      estimatedCompressedSize: 435801,
+      expectedSha256Hash:
+        "aaf9a325c0a988c507d0312cb6ba1a02bac7a370bcd879aedee626a40bfbda78",
+    },
+    lex: {
+      name: "lex.50.50.ruen.s2t.bin",
+      size: 5090836,
+      estimatedCompressedSize: 2684919,
+      expectedSha256Hash:
+        "e6667e22f5f86be4872e3768b7184727f5dd8c9f2ccfb0639baabcb1176f5d11",
+    },
+    model: {
+      name: "model.ruen.intgemm.alphas.bin",
+      size: 17140836,
+      estimatedCompressedSize: 13108893,
+      expectedSha256Hash:
+        "3b6a0305e3d232fadd54f5a765365b7b96ad6d8f2e818cba594b02fbd8fadb3d",
+    },
+  },
+  csen: {
+    vocab: {
+      name: "vocab.csen.spm",
+      size: 769763,
+      estimatedCompressedSize: 366392,
+      expectedSha256Hash:
+        "f71cc5d045e479607078e079884f44032f5a0b82547fb96eefa29cd1eb47c6f3",
+    },
+    lex: {
+      name: "lex.50.50.csen.s2t.bin",
+      size: 4535788,
+      estimatedCompressedSize: 2418488,
+      expectedSha256Hash:
+        "8228a3c3f7887759a62b7d7c674a7bef9b70161913f9b0939ab58f71186835c2",
+    },
+    model: {
+      name: "model.csen.intgemm.alphas.bin",
+      size: 17140756,
+      estimatedCompressedSize: 13045032,
+      expectedSha256Hash:
+        "5b16661e2864dc50b2f4091a16bdd4ec8d8283e04271e602159ba348df5d6e2d",
+    },
+  },
+  deen: {
+    vocab: {
+      name: "vocab.deen.spm",
+      size: 784269,
+      estimatedCompressedSize: 410738,
+      expectedSha256Hash:
+        "417668f2ed297970febafb5b079a9d5ebc4ed0b3550ac8386d67a90473a09bd7",
+    },
+    lex: {
+      name: "lex.50.50.deen.s2t.bin",
+      size: 5047568,
+      estimatedCompressedSize: 2657472,
+      expectedSha256Hash:
+        "2f7c0f7bbce97ae5b52454074a892ba7b7610fb98e3c5d341e4ca79f0850c4de",
+    },
+    model: {
+      name: "model.deen.intgemm.alphas.bin",
+      size: 17140837,
+      estimatedCompressedSize: 13091214,
+      expectedSha256Hash:
+        "dda44d87ab0d8ad3b3871122fd3ee385f37878183a8b4ec139cd909531ec5009",
+    },
+  },
+  encs: {
+    vocab: {
+      name: "vocab.csen.spm",
+      size: 769763,
+      estimatedCompressedSize: 366392,
+      expectedSha256Hash:
+        "f71cc5d045e479607078e079884f44032f5a0b82547fb96eefa29cd1eb47c6f3",
+    },
+    lex: {
+      name: "lex.50.50.encs.s2t.bin",
+      size: 3556124,
+      estimatedCompressedSize: 1913246,
+      expectedSha256Hash:
+        "e19c77231bf977988e31ff8db15fe79966b5170564bd3e10613f239e7f461d97",
+    },
+    model: {
+      name: "model.encs.intgemm.alphas.bin",
+      size: 17140756,
+      estimatedCompressedSize: 12630325,
+      expectedSha256Hash:
+        "9a2fe0588bd972accfc801e2f31c945de0557804a91666ae5ab43b94fb74ac4b",
+    },
+  },
+  ende: {
+    vocab: {
+      name: "vocab.deen.spm",
+      size: 797501,
+      estimatedCompressedSize: 412505,
+      expectedSha256Hash:
+        "bc8f8229933d8294c727f3eab12f6f064e7082b929f2d29494c8a1e619ba174c",
+    },
+    lex: {
+      name: "lex.50.50.ende.s2t.bin",
+      size: 3062492,
+      estimatedCompressedSize: 1575385,
+      expectedSha256Hash:
+        "764797d075f0642c0b079cce6547348d65fe4e92ac69fa6a8605cd8b53dacb3f",
+    },
+    model: {
+      name: "model.ende.intgemm.alphas.bin",
+      size: 17140498,
+      estimatedCompressedSize: 13207068,
+      expectedSha256Hash:
+        "f0946515c6645304f0706fa66a051c3b7b7c507f12d0c850f276c18165a10c14",
+    },
+  },
+  enes: {
+    vocab: {
+      name: "vocab.esen.spm",
+      size: 825463,
+      estimatedCompressedSize: 414566,
+      expectedSha256Hash:
+        "909b1eea1face0d7f90a474fe29a8c0fef8d104b6e41e65616f864c964ba8845",
+    },
+    lex: {
+      name: "lex.50.50.enes.s2t.bin",
+      size: 3347104,
+      estimatedCompressedSize: 1720700,
+      expectedSha256Hash:
+        "3a113d713dec3cf1d12bba5b138ae616e28bba4bbc7fe7fd39ba145e26b86d7f",
+    },
+    model: {
+      name: "model.enes.intgemm.alphas.bin",
+      size: 17140755,
+      estimatedCompressedSize: 12602853,
+      expectedSha256Hash:
+        "fa7460037a3163e03fe1d23602f964bff2331da6ee813637e092ddf37156ef53",
+    },
+  },
+  enet: {
+    vocab: {
+      name: "vocab.eten.spm",
+      size: 828426,
+      estimatedCompressedSize: 416995,
+      expectedSha256Hash:
+        "e3b66bc141f6123cd40746e2fb9b8ee4f89cbf324ab27d6bbf3782e52f15fa2d",
+    },
+    lex: {
+      name: "lex.50.50.enet.s2t.bin",
+      size: 2700780,
+      estimatedCompressedSize: 1336443,
+      expectedSha256Hash:
+        "3d1b40ff43ebef82cf98d416a88a1ea19eb325a85785eef102f59878a63a829d",
+    },
+    model: {
+      name: "model.enet.intgemm.alphas.bin",
+      size: 17140754,
+      estimatedCompressedSize: 12543318,
+      expectedSha256Hash:
+        "a28874a8b702a519a14dc71bcee726a5cb4b539eeaada2d06492f751469a1fd6",
+    },
+  },
+  esen: {
+    vocab: {
+      name: "vocab.esen.spm",
+      size: 825463,
+      estimatedCompressedSize: 414566,
+      expectedSha256Hash:
+        "909b1eea1face0d7f90a474fe29a8c0fef8d104b6e41e65616f864c964ba8845",
+    },
+    lex: {
+      name: "lex.50.50.esen.s2t.bin",
+      size: 3860888,
+      estimatedCompressedSize: 1978538,
+      expectedSha256Hash:
+        "f11a2c23ef85ab1fee1c412b908d69bc20d66fd59faa8f7da5a5f0347eddf969",
+    },
+    model: {
+      name: "model.esen.intgemm.alphas.bin",
+      size: 17140755,
+      estimatedCompressedSize: 13215960,
+      expectedSha256Hash:
+        "4b6b7f451094aaa447d012658af158ffc708fc8842dde2f871a58404f5457fe0",
+    },
+  },
+  eten: {
+    vocab: {
+      name: "vocab.eten.spm",
+      size: 828426,
+      estimatedCompressedSize: 416995,
+      expectedSha256Hash:
+        "e3b66bc141f6123cd40746e2fb9b8ee4f89cbf324ab27d6bbf3782e52f15fa2d",
+    },
+    lex: {
+      name: "lex.50.50.eten.s2t.bin",
+      size: 3974944,
+      estimatedCompressedSize: 1920655,
+      expectedSha256Hash:
+        "6992bedc590e60e610a28129c80746fe5f33144a4520e2c5508d87db14ca54f8",
+    },
+    model: {
+      name: "model.eten.intgemm.alphas.bin",
+      size: 17140754,
+      estimatedCompressedSize: 12222624,
+      expectedSha256Hash:
+        "aac98a2371e216ee2d4843cbe896c617f6687501e17225ac83482eba52fd0028",
+    },
+  },
+};
\ No newline at end of file
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
new file mode 100644
index 000000000..1cf3a1461
--- /dev/null
+++ b/wasm/test_page/js/worker.js
@@ -0,0 +1,298 @@
+// All variables specific to translation service
+var translationService, responseOptions, input = undefined;
+// A map of language-pair to TranslationModel object
+var languagePairToTranslationModels = new Map();
+
+const BERGAMOT_TRANSLATOR_MODULE = "bergamot-translator-worker.js";
+const MODEL_REGISTRY = "modelRegistry.js";
+
+const encoder = new TextEncoder(); // string to utf-8 converter
+const decoder = new TextDecoder(); // utf-8 to string converter
+
+const start = Date.now();
+let moduleLoadStart;
+var Module = {
+  preRun: [function() {
+    log(`Time until Module.preRun: ${(Date.now() - start) / 1000} secs`);
+    moduleLoadStart = Date.now();
+  }],
+  onRuntimeInitialized: function() {
+    log(`Wasm Runtime initialized Successfully (preRun -> onRuntimeInitialized) in ${(Date.now() - moduleLoadStart) / 1000} secs`);
+    importScripts(MODEL_REGISTRY);
+    postMessage([`import_reply`, modelRegistry]);
+  }
+};
+
+const log = (message) => {
+  console.debug(message);
+}
+
+onmessage = async function(e) {
+  const command = e.data[0];
+  log(`Message '${command}' received from main script`);
+  let result = "";
+  if (command === 'import') {
+      importScripts(BERGAMOT_TRANSLATOR_MODULE);
+  } else if (command === 'load_model') {
+      let start = Date.now();
+      let from = e.data[1];
+      let to = e.data[2];
+      try {
+        await constructTranslationService();
+        await constructTranslationModel(from, to);
+        log(`Model '${from}${to}' successfully constructed. Time taken: ${(Date.now() - start) / 1000} secs`);
+        result = "Model successfully loaded";
+      } catch (error) {
+        log(`Model '${from}${to}' construction failed: '${error.message}'`);
+        result = "Model loading failed";
+      }
+      log(`'${command}' command done, Posting message back to main script`);
+      postMessage([`${command}_reply`, result]);
+  } else if (command === 'translate') {
+      const from = e.data[1];
+      const to = e.data[2];
+      const inputParagraphs = e.data[3];
+      let inputWordCount = 0;
+      inputParagraphs.forEach(sentence => {
+        inputWordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
+      })
+      let start = Date.now();
+      try {
+        result = translate(from, to, inputParagraphs);
+        const secs = (Date.now() - start) / 1000;
+        log(`Translation '${from}${to}' Successful. Speed: ${Math.round(inputWordCount / secs)} WPS (${inputWordCount} words in ${secs} secs)`);
+      } catch (error) {
+        log(`Error: ${error.message}`);
+      }
+      log(`'${command}' command done, Posting message back to main script`);
+      postMessage([`${command}_reply`, result]);
+  }
+}
+
+// Instantiates the Translation Service
+const constructTranslationService = async () => {
+  if (!translationService) {
+    var translationServiceConfig = {};
+    log(`Creating Translation Service with config: ${translationServiceConfig}`);
+    translationService = new Module.BlockingService(translationServiceConfig);
+    log(`Translation Service created successfully`);
+  }
+}
+
+// Constructs a translation model object for the source and target language pair
+const constructTranslationModel = async (from, to) => {
+  // Delete all previously constructed translation models and clear the map
+  languagePairToTranslationModels.forEach((value, key) => {
+    log(`Destructing model '${key}'`);
+    value.delete();
+  });
+  languagePairToTranslationModels.clear();
+
+  // If none of the languages is English then construct multiple models with
+  // English as a pivot language.
+  if (from !== 'en' && to !== 'en') {
+    log(`Constructing model '${from}${to}' via pivoting: '${from}en' and 'en${to}'`);
+    await Promise.all([_constructTranslationModelInvolvingEnglish(from, 'en'),
+                        _constructTranslationModelInvolvingEnglish('en', to)]);
+  }
+  else {
+    log(`Constructing model '${from}${to}'`);
+    await _constructTranslationModelInvolvingEnglish(from, to);
+  }
+}
+
+// Translates text from source language to target language.
+const translate = (from, to, paragraphs) => {
+  // If none of the languages is English then perform translation with
+  // English as a pivot language.
+  if (from !== 'en' && to !== 'en') {
+    log(`Translating '${from}${to}' via pivoting: '${from}en' -> 'en${to}'`);
+    let translatedParagraphsInEnglish = _translateInvolvingEnglish(from, 'en', paragraphs);
+    return _translateInvolvingEnglish('en', to, translatedParagraphsInEnglish);
+  }
+  else {
+    log(`Translating '${from}${to}'`);
+    return _translateInvolvingEnglish(from, to, paragraphs);
+  }
+}
+
+// Downloads file from a url and returns the array buffer
+const _downloadAsArrayBuffer = async(url) => {
+  const response = await fetch(url);
+  if (!response.ok) {
+    throw Error(`Downloading ${url} failed: HTTP ${response.status} - ${response.statusText}`);
+  }
+  return response.arrayBuffer();
+}
+
+// Constructs and initializes the AlignedMemory from the array buffer and alignment size
+const _prepareAlignedMemoryFromBuffer = async (buffer, alignmentSize) => {
+  var byteArray = new Int8Array(buffer);
+  log(`Constructing Aligned memory. Size: ${byteArray.byteLength} bytes, Alignment: ${alignmentSize}`);
+  var alignedMemory = new Module.AlignedMemory(byteArray.byteLength, alignmentSize);
+  log(`Aligned memory construction done`);
+  const alignedByteArrayView = alignedMemory.getByteArrayView();
+  alignedByteArrayView.set(byteArray);
+  log(`Aligned memory initialized`);
+  return alignedMemory;
+}
+
+const _constructTranslationModelInvolvingEnglish = async (from, to) => {
+  const languagePair = `${from}${to}`;
+
+  /*Set the Model Configuration as YAML formatted string.
+    For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
+    Vocab files are re-used in both translation directions
+    const vocabLanguagePair = from === "en" ? `${to}${from}` : languagePair;
+    const modelConfig = `models:
+      - /${languagePair}/model.${languagePair}.intgemm.alphas.bin
+      vocabs:
+      - /${languagePair}/vocab.${vocabLanguagePair}.spm
+      - /${languagePair}/vocab.${vocabLanguagePair}.spm
+      beam-size: 1
+      normalize: 1.0
+      word-penalty: 0
+      max-length-break: 128
+      mini-batch-words: 1024
+      workspace: 128
+      max-length-factor: 2.0
+      skip-cost: true
+      cpu-threads: 0
+      quiet: true
+      quiet-translation: true
+      shortlist:
+          - /${languagePair}/lex.${languagePair}.s2t
+          - 50
+          - 50
+      `;
+      */
+
+  // TODO: gemm-precision: int8shiftAlphaAll (for the models that support this)
+  // DONOT CHANGE THE SPACES BETWEEN EACH ENTRY OF CONFIG
+  const modelConfig = `beam-size: 1
+normalize: 1.0
+word-penalty: 0
+max-length-break: 128
+mini-batch-words: 1024
+workspace: 128
+max-length-factor: 2.0
+skip-cost: true
+cpu-threads: 0
+quiet: true
+quiet-translation: true
+gemm-precision: int8shiftAll
+`;
+
+  const modelFile = `${rootURL}/${languagePair}/${modelRegistry[languagePair]["model"].name}`;
+  const shortlistFile = `${rootURL}/${languagePair}/${modelRegistry[languagePair]["lex"].name}`;
+  const vocabFiles = [`${rootURL}/${languagePair}/${modelRegistry[languagePair]["vocab"].name}`,
+                      `${rootURL}/${languagePair}/${modelRegistry[languagePair]["vocab"].name}`];
+
+  const uniqueVocabFiles = new Set(vocabFiles);
+  log(`modelFile: ${modelFile}\nshortlistFile: ${shortlistFile}\nNo. of unique vocabs: ${uniqueVocabFiles.size}`);
+  uniqueVocabFiles.forEach(item => log(`unique vocabFile: ${item}`));
+
+  // Download the files as buffers from the given urls
+  let start = Date.now();
+  const downloadedBuffers = await Promise.all([_downloadAsArrayBuffer(modelFile), _downloadAsArrayBuffer(shortlistFile)]);
+  const modelBuffer = downloadedBuffers[0];
+  const shortListBuffer = downloadedBuffers[1];
+
+  const downloadedVocabBuffers = [];
+  for (let item of uniqueVocabFiles.values()) {
+    downloadedVocabBuffers.push(await _downloadAsArrayBuffer(item));
+  }
+  log(`Total Download time for all files of '${languagePair}': ${(Date.now() - start) / 1000} secs`);
+
+  // Construct AlignedMemory objects with downloaded buffers
+  let constructedAlignedMemories = await Promise.all([_prepareAlignedMemoryFromBuffer(modelBuffer, 256),
+                                                      _prepareAlignedMemoryFromBuffer(shortListBuffer, 64)]);
+  let alignedModelMemory = constructedAlignedMemories[0];
+  let alignedShortlistMemory = constructedAlignedMemories[1];
+  let alignedVocabsMemoryList = new Module.AlignedMemoryList;
+  for(let item of downloadedVocabBuffers) {
+    let alignedMemory = await _prepareAlignedMemoryFromBuffer(item, 64);
+    alignedVocabsMemoryList.push_back(alignedMemory);
+  }
+  for (let vocabs=0; vocabs < alignedVocabsMemoryList.size(); vocabs++) {
+    log(`Aligned vocab memory${vocabs+1} size: ${alignedVocabsMemoryList.get(vocabs).size()}`);
+  }
+  log(`Aligned model memory size: ${alignedModelMemory.size()}`);
+  log(`Aligned shortlist memory size: ${alignedShortlistMemory.size()}`);
+
+  log(`Translation Model config: ${modelConfig}`);
+  var translationModel = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
+  languagePairToTranslationModels.set(languagePair, translationModel);
+}
+
+const _translateInvolvingEnglish = (from, to, paragraphs) => {
+  const languagePair = `${from}${to}`;
+  if (!languagePairToTranslationModels.has(languagePair)) {
+    throw Error(`Please load translation model '${languagePair}' before translating`);
+  }
+  translationModel = languagePairToTranslationModels.get(languagePair);
+
+  // Instantiate the arguments of translate() API i.e. ResponseOptions and input (vector<string>)
+  var responseOptions = new Module.ResponseOptions();
+  let input = new Module.VectorString;
+
+  // Initialize the input
+  paragraphs.forEach(paragraph => {
+    // prevent empty paragraph - it breaks the translation
+    if (paragraph.trim() === "") {
+      return;
+    }
+    input.push_back(paragraph.trim())
+  })
+
+  // Access input (just for debugging)
+  log(`Input size: ${input.size()}`);
+
+  // Translate the input, which is a vector<String>; the result is a vector<Response>
+  let result = translationService.translate(translationModel, input, responseOptions);
+
+  const translatedParagraphs = [];
+  const translatedSentencesOfParagraphs = [];
+  const sourceSentencesOfParagraphs = [];
+  for (let i = 0; i < result.size(); i++) {
+    translatedParagraphs.push(result.get(i).getTranslatedText());
+    translatedSentencesOfParagraphs.push(_getAllTranslatedSentencesOfParagraph(result.get(i)));
+    sourceSentencesOfParagraphs.push(_getAllSourceSentencesOfParagraph(result.get(i)));
+  }
+
+  responseOptions.delete();
+  input.delete();
+  return translatedParagraphs;
+}
+
+// Extracts all the translated sentences from the Response and returns them.
+const _getAllTranslatedSentencesOfParagraph = (response) => {
+  const sentences = [];
+  const text = response.getTranslatedText();
+  for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
+    const utf8SentenceByteRange = response.getTranslatedSentence(sentenceIndex);
+    sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
+  }
+  return sentences;
+}
+
+// Extracts all the source sentences from the Response and returns them.
+const _getAllSourceSentencesOfParagraph = (response) => {
+  const sentences = [];
+  const text = response.getOriginalText();
+  for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
+    const utf8SentenceByteRange = response.getSourceSentence(sentenceIndex);
+    sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
+  }
+  return sentences;
+}
+
+/*
+ * Returns a substring of text (a string). The substring is represented by
+ * byteRange (begin and end endices) within the utf-8 encoded version of the text.
+ */
+const _getSentenceFromByteRange = (text, byteRange) => {
+  const utf8BytesView = encoder.encode(text);
+  const utf8SentenceBytes = utf8BytesView.subarray(byteRange.begin, byteRange.end);
+  return decoder.decode(utf8SentenceBytes);
+}
diff --git a/wasm/test_page/package-lock.json b/wasm/test_page/package-lock.json
index ae4cb9dd6..065c92de8 100644
--- a/wasm/test_page/package-lock.json
+++ b/wasm/test_page/package-lock.json
@@ -1,6 +1,519 @@
 {
+  "name": "test_page",
+  "lockfileVersion": 2,
   "requires": true,
-  "lockfileVersion": 1,
+  "packages": {
+    "": {
+      "dependencies": {
+        "cors": "^2.8.5",
+        "express": "^4.17.1",
+        "nocache": "^2.1.0"
+      }
+    },
+    "node_modules/accepts": {
+      "version": "1.3.7",
+      "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.7.tgz",
+      "integrity": "sha512-Il80Qs2WjYlJIBNzNkK6KYqlVMTbZLXgHx2oT0pU/fjRHyEp+PEfEPY0R3WCwAGVOtauxh1hOxNgIf5bv7dQpA==",
+      "dependencies": {
+        "mime-types": "~2.1.24",
+        "negotiator": "0.6.2"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/array-flatten": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz",
+      "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI="
+    },
+    "node_modules/body-parser": {
+      "version": "1.19.0",
+      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.19.0.tgz",
+      "integrity": "sha512-dhEPs72UPbDnAQJ9ZKMNTP6ptJaionhP5cBb541nXPlW60Jepo9RV/a4fX4XWW9CuFNK22krhrj1+rgzifNCsw==",
+      "dependencies": {
+        "bytes": "3.1.0",
+        "content-type": "~1.0.4",
+        "debug": "2.6.9",
+        "depd": "~1.1.2",
+        "http-errors": "1.7.2",
+        "iconv-lite": "0.4.24",
+        "on-finished": "~2.3.0",
+        "qs": "6.7.0",
+        "raw-body": "2.4.0",
+        "type-is": "~1.6.17"
+      },
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/bytes": {
+      "version": "3.1.0",
+      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.0.tgz",
+      "integrity": "sha512-zauLjrfCG+xvoyaqLoV8bLVXXNGC4JqlxFCutSDWA6fJrTo2ZuvLYTqZ7aHBLZSMOopbzwv8f+wZcVzfVTI2Dg==",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/content-disposition": {
+      "version": "0.5.3",
+      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.3.tgz",
+      "integrity": "sha512-ExO0774ikEObIAEV9kDo50o+79VCUdEB6n6lzKgGwupcVeRlhrj3qGAfwq8G6uBJjkqLrhT0qEYFcWng8z1z0g==",
+      "dependencies": {
+        "safe-buffer": "5.1.2"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/content-type": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz",
+      "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/cookie": {
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.4.0.tgz",
+      "integrity": "sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/cookie-signature": {
+      "version": "1.0.6",
+      "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.6.tgz",
+      "integrity": "sha1-4wOogrNCzD7oylE6eZmXNNqzriw="
+    },
+    "node_modules/cors": {
+      "version": "2.8.5",
+      "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.5.tgz",
+      "integrity": "sha512-KIHbLJqu73RGr/hnbrO9uBeixNGuvSQjul/jdFvS/KFSIH1hWVd1ng7zOHx+YrEfInLG7q4n6GHQ9cDtxv/P6g==",
+      "dependencies": {
+        "object-assign": "^4",
+        "vary": "^1"
+      },
+      "engines": {
+        "node": ">= 0.10"
+      }
+    },
+    "node_modules/debug": {
+      "version": "2.6.9",
+      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
+      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
+      "dependencies": {
+        "ms": "2.0.0"
+      }
+    },
+    "node_modules/depd": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/depd/-/depd-1.1.2.tgz",
+      "integrity": "sha1-m81S4UwJd2PnSbJ0xDRu0uVgtak=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/destroy": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.0.4.tgz",
+      "integrity": "sha1-l4hXRCxEdJ5CBmE+N5RiBYJqvYA="
+    },
+    "node_modules/ee-first": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
+      "integrity": "sha1-WQxhFWsK4vTwJVcyoViyZrxWsh0="
+    },
+    "node_modules/encodeurl": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
+      "integrity": "sha1-rT/0yG7C0CkyL1oCw6mmBslbP1k=",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/escape-html": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
+      "integrity": "sha1-Aljq5NPQwJdN4cFpGI7wBR0dGYg="
+    },
+    "node_modules/etag": {
+      "version": "1.8.1",
+      "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
+      "integrity": "sha1-Qa4u62XvpiJorr/qg6x9eSmbCIc=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/express": {
+      "version": "4.17.1",
+      "resolved": "https://registry.npmjs.org/express/-/express-4.17.1.tgz",
+      "integrity": "sha512-mHJ9O79RqluphRrcw2X/GTh3k9tVv8YcoyY4Kkh4WDMUYKRZUq0h1o0w2rrrxBqM7VoeUVqgb27xlEMXTnYt4g==",
+      "dependencies": {
+        "accepts": "~1.3.7",
+        "array-flatten": "1.1.1",
+        "body-parser": "1.19.0",
+        "content-disposition": "0.5.3",
+        "content-type": "~1.0.4",
+        "cookie": "0.4.0",
+        "cookie-signature": "1.0.6",
+        "debug": "2.6.9",
+        "depd": "~1.1.2",
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "etag": "~1.8.1",
+        "finalhandler": "~1.1.2",
+        "fresh": "0.5.2",
+        "merge-descriptors": "1.0.1",
+        "methods": "~1.1.2",
+        "on-finished": "~2.3.0",
+        "parseurl": "~1.3.3",
+        "path-to-regexp": "0.1.7",
+        "proxy-addr": "~2.0.5",
+        "qs": "6.7.0",
+        "range-parser": "~1.2.1",
+        "safe-buffer": "5.1.2",
+        "send": "0.17.1",
+        "serve-static": "1.14.1",
+        "setprototypeof": "1.1.1",
+        "statuses": "~1.5.0",
+        "type-is": "~1.6.18",
+        "utils-merge": "1.0.1",
+        "vary": "~1.1.2"
+      },
+      "engines": {
+        "node": ">= 0.10.0"
+      }
+    },
+    "node_modules/finalhandler": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.2.tgz",
+      "integrity": "sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA==",
+      "dependencies": {
+        "debug": "2.6.9",
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "on-finished": "~2.3.0",
+        "parseurl": "~1.3.3",
+        "statuses": "~1.5.0",
+        "unpipe": "~1.0.0"
+      },
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/forwarded": {
+      "version": "0.1.2",
+      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.1.2.tgz",
+      "integrity": "sha1-mMI9qxF1ZXuMBXPozszZGw/xjIQ=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/fresh": {
+      "version": "0.5.2",
+      "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz",
+      "integrity": "sha1-PYyt2Q2XZWn6g1qx+OSyOhBWBac=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/http-errors": {
+      "version": "1.7.2",
+      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.7.2.tgz",
+      "integrity": "sha512-uUQBt3H/cSIVfch6i1EuPNy/YsRSOUBXTVfZ+yR7Zjez3qjBz6i9+i4zjNaoqcoFVI4lQJ5plg63TvGfRSDCRg==",
+      "dependencies": {
+        "depd": "~1.1.2",
+        "inherits": "2.0.3",
+        "setprototypeof": "1.1.1",
+        "statuses": ">= 1.5.0 < 2",
+        "toidentifier": "1.0.0"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/iconv-lite": {
+      "version": "0.4.24",
+      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
+      "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
+      "dependencies": {
+        "safer-buffer": ">= 2.1.2 < 3"
+      },
+      "engines": {
+        "node": ">=0.10.0"
+      }
+    },
+    "node_modules/inherits": {
+      "version": "2.0.3",
+      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz",
+      "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4="
+    },
+    "node_modules/ipaddr.js": {
+      "version": "1.9.1",
+      "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
+      "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",
+      "engines": {
+        "node": ">= 0.10"
+      }
+    },
+    "node_modules/media-typer": {
+      "version": "0.3.0",
+      "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
+      "integrity": "sha1-hxDXrwqmJvj/+hzgAWhUUmMlV0g=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/merge-descriptors": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz",
+      "integrity": "sha1-sAqqVW3YtEVoFQ7J0blT8/kMu2E="
+    },
+    "node_modules/methods": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz",
+      "integrity": "sha1-VSmk1nZUE07cxSZmVoNbD4Ua/O4=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/mime": {
+      "version": "1.6.0",
+      "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
+      "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==",
+      "bin": {
+        "mime": "cli.js"
+      },
+      "engines": {
+        "node": ">=4"
+      }
+    },
+    "node_modules/mime-db": {
+      "version": "1.45.0",
+      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.45.0.tgz",
+      "integrity": "sha512-CkqLUxUk15hofLoLyljJSrukZi8mAtgd+yE5uO4tqRZsdsAJKv0O+rFMhVDRJgozy+yG6md5KwuXhD4ocIoP+w==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/mime-types": {
+      "version": "2.1.28",
+      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.28.tgz",
+      "integrity": "sha512-0TO2yJ5YHYr7M2zzT7gDU1tbwHxEUWBCLt0lscSNpcdAfFyJOVEpRYNS7EXVcTLNj/25QO8gulHC5JtTzSE2UQ==",
+      "dependencies": {
+        "mime-db": "1.45.0"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/ms": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
+      "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g="
+    },
+    "node_modules/negotiator": {
+      "version": "0.6.2",
+      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.2.tgz",
+      "integrity": "sha512-hZXc7K2e+PgeI1eDBe/10Ard4ekbfrrqG8Ep+8Jmf4JID2bNg7NvCPOZN+kfF574pFQI7mum2AUqDidoKqcTOw==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/nocache": {
+      "version": "2.1.0",
+      "resolved": "https://registry.npmjs.org/nocache/-/nocache-2.1.0.tgz",
+      "integrity": "sha512-0L9FvHG3nfnnmaEQPjT9xhfN4ISk0A8/2j4M37Np4mcDesJjHgEUfgPhdCyZuFI954tjokaIj/A3NdpFNdEh4Q==",
+      "engines": {
+        "node": ">=4.0.0"
+      }
+    },
+    "node_modules/object-assign": {
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
+      "integrity": "sha1-IQmtx5ZYh8/AXLvUQsrIv7s2CGM=",
+      "engines": {
+        "node": ">=0.10.0"
+      }
+    },
+    "node_modules/on-finished": {
+      "version": "2.3.0",
+      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.3.0.tgz",
+      "integrity": "sha1-IPEzZIGwg811M3mSoWlxqi2QaUc=",
+      "dependencies": {
+        "ee-first": "1.1.1"
+      },
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/parseurl": {
+      "version": "1.3.3",
+      "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
+      "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/path-to-regexp": {
+      "version": "0.1.7",
+      "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz",
+      "integrity": "sha1-32BBeABfUi8V60SQ5yR6G/qmf4w="
+    },
+    "node_modules/proxy-addr": {
+      "version": "2.0.6",
+      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.6.tgz",
+      "integrity": "sha512-dh/frvCBVmSsDYzw6n926jv974gddhkFPfiN8hPOi30Wax25QZyZEGveluCgliBnqmuM+UJmBErbAUFIoDbjOw==",
+      "dependencies": {
+        "forwarded": "~0.1.2",
+        "ipaddr.js": "1.9.1"
+      },
+      "engines": {
+        "node": ">= 0.10"
+      }
+    },
+    "node_modules/qs": {
+      "version": "6.7.0",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.7.0.tgz",
+      "integrity": "sha512-VCdBRNFTX1fyE7Nb6FYoURo/SPe62QCaAyzJvUjwRaIsc+NePBEniHlvxFmmX56+HZphIGtV0XeCirBtpDrTyQ==",
+      "engines": {
+        "node": ">=0.6"
+      }
+    },
+    "node_modules/range-parser": {
+      "version": "1.2.1",
+      "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
+      "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/raw-body": {
+      "version": "2.4.0",
+      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.4.0.tgz",
+      "integrity": "sha512-4Oz8DUIwdvoa5qMJelxipzi/iJIi40O5cGV1wNYp5hvZP8ZN0T+jiNkL0QepXs+EsQ9XJ8ipEDoiH70ySUJP3Q==",
+      "dependencies": {
+        "bytes": "3.1.0",
+        "http-errors": "1.7.2",
+        "iconv-lite": "0.4.24",
+        "unpipe": "1.0.0"
+      },
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/safe-buffer": {
+      "version": "5.1.2",
+      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
+      "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="
+    },
+    "node_modules/safer-buffer": {
+      "version": "2.1.2",
+      "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
+      "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
+    },
+    "node_modules/send": {
+      "version": "0.17.1",
+      "resolved": "https://registry.npmjs.org/send/-/send-0.17.1.tgz",
+      "integrity": "sha512-BsVKsiGcQMFwT8UxypobUKyv7irCNRHk1T0G680vk88yf6LBByGcZJOTJCrTP2xVN6yI+XjPJcNuE3V4fT9sAg==",
+      "dependencies": {
+        "debug": "2.6.9",
+        "depd": "~1.1.2",
+        "destroy": "~1.0.4",
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "etag": "~1.8.1",
+        "fresh": "0.5.2",
+        "http-errors": "~1.7.2",
+        "mime": "1.6.0",
+        "ms": "2.1.1",
+        "on-finished": "~2.3.0",
+        "range-parser": "~1.2.1",
+        "statuses": "~1.5.0"
+      },
+      "engines": {
+        "node": ">= 0.8.0"
+      }
+    },
+    "node_modules/send/node_modules/ms": {
+      "version": "2.1.1",
+      "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz",
+      "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg=="
+    },
+    "node_modules/serve-static": {
+      "version": "1.14.1",
+      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.14.1.tgz",
+      "integrity": "sha512-JMrvUwE54emCYWlTI+hGrGv5I8dEwmco/00EvkzIIsR7MqrHonbD9pO2MOfFnpFntl7ecpZs+3mW+XbQZu9QCg==",
+      "dependencies": {
+        "encodeurl": "~1.0.2",
+        "escape-html": "~1.0.3",
+        "parseurl": "~1.3.3",
+        "send": "0.17.1"
+      },
+      "engines": {
+        "node": ">= 0.8.0"
+      }
+    },
+    "node_modules/setprototypeof": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.1.tgz",
+      "integrity": "sha512-JvdAWfbXeIGaZ9cILp38HntZSFSo3mWg6xGcJJsd+d4aRMOqauag1C63dJfDw7OaMYwEbHMOxEZ1lqVRYP2OAw=="
+    },
+    "node_modules/statuses": {
+      "version": "1.5.0",
+      "resolved": "https://registry.npmjs.org/statuses/-/statuses-1.5.0.tgz",
+      "integrity": "sha1-Fhx9rBd2Wf2YEfQ3cfqZOBR4Yow=",
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/toidentifier": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.0.tgz",
+      "integrity": "sha512-yaOH/Pk/VEhBWWTlhI+qXxDFXlejDGcQipMlyxda9nthulaxLZUNcUqFxokp0vcYnvteJln5FNQDRrxj3YcbVw==",
+      "engines": {
+        "node": ">=0.6"
+      }
+    },
+    "node_modules/type-is": {
+      "version": "1.6.18",
+      "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
+      "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",
+      "dependencies": {
+        "media-typer": "0.3.0",
+        "mime-types": "~2.1.24"
+      },
+      "engines": {
+        "node": ">= 0.6"
+      }
+    },
+    "node_modules/unpipe": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
+      "integrity": "sha1-sr9O6FFKrmFltIF4KdIbLvSZBOw=",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    },
+    "node_modules/utils-merge": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",
+      "integrity": "sha1-n5VxD1CiZ5R7LMwSR0HBAoQn5xM=",
+      "engines": {
+        "node": ">= 0.4.0"
+      }
+    },
+    "node_modules/vary": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
+      "integrity": "sha1-IpnwLG3tMNSllhsLn3RSShj2NPw=",
+      "engines": {
+        "node": ">= 0.8"
+      }
+    }
+  },
   "dependencies": {
     "accepts": {
       "version": "1.3.7",
diff --git a/wasm/test_page/start_server.sh b/wasm/test_page/start_server.sh
index 911364665..8cb90071c 100644
--- a/wasm/test_page/start_server.sh
+++ b/wasm/test_page/start_server.sh
@@ -19,13 +19,13 @@ if [ ! -e "$1" ]; then
     exit
 fi
 
-WASM_ARTIFACTS="$1/bergamot-translator-worker.*"
+WASM_ARTIFACTS="$1/bergamot-translator-worker.js $1/bergamot-translator-worker.wasm"
 for i in $WASM_ARTIFACTS; do
     [ -f "$i" ] || breaks
-    cp $i .
+    cp $i js/.
     echo "Copied \"$i\""
 done
 
 npm install
 echo "Start httpserver"
-node bergamot-httpserver.js
\ No newline at end of file
+node bergamot-httpserver.js 80 1 0
\ No newline at end of file
diff --git a/wasm/test_page/worker.js b/wasm/test_page/worker.js
deleted file mode 100644
index 8b53a271a..000000000
--- a/wasm/test_page/worker.js
+++ /dev/null
@@ -1,267 +0,0 @@
-var translationService, responseOptions, input = undefined;
-// A map of language-pair to TranslationModel object
-var translationModels = new Map();
-const BERGAMOT_TRANSLATOR_MODULE = "bergamot-translator-worker.js";
-
-const encoder = new TextEncoder(); // string to utf-8 converter
-const decoder = new TextDecoder(); // utf-8 to string converter
-
-const start = Date.now();
-let moduleLoadStart;
-var Module = {
-    preRun: [function() {
-        log(`Time until Module.preRun: ${(Date.now() - start) / 1000} secs`);
-        moduleLoadStart = Date.now();
-    }],
-    onRuntimeInitialized: function() {
-        log(`Wasm Runtime initialized (preRun -> onRuntimeInitialized) in ${(Date.now() - moduleLoadStart) / 1000} secs`);
-    }
-};
-
-const log = (message) => {
-  console.debug(message);
-}
-
-onmessage = async function(e) {
-    let command = e.data[0];
-    log(`Message '${command}' received from main script`);
-    let result = "";
-    if (command === 'load_module') {
-        importScripts(BERGAMOT_TRANSLATOR_MODULE);
-        result = `Translator wasm module successfully loaded`;
-        log(result);
-        log('Posting message back to main script');
-        postMessage(['module_loaded', result]);
-    }
-    else if (command === 'load_model') {
-        let start = Date.now();
-        try {
-            await constructTranslationService();
-            await constructTranslationModel(e.data[1], e.data[2]);
-            result = `translation model '${e.data[1]}${e.data[2]}' successfully loaded; took ${(Date.now() - start) / 1000} secs`;
-        } catch (error) {
-            result = `translation model '${e.data[1]}${e.data[2]}' loading failed: '${error.message}'`;
-        }
-        log(result);
-        log('Posting message back to main script');
-        postMessage(['model_loaded', result]);
-    }
-    else if (command === 'translate') {
-        const from = e.data[1];
-        const to = e.data[2];
-        const inputParagraphs = e.data[3];
-        let inputWordCount = 0;
-        inputParagraphs.forEach(sentence => {
-            inputWordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
-        })
-
-        let start = Date.now();
-        var translatedParagraphs;
-        try {
-            translatedParagraphs  = translate(from, to, inputParagraphs);
-            const secs = (Date.now() - start) / 1000;
-            result = `Translation '${from}${to}' Successful. Speed: ${Math.round(inputWordCount / secs)} Words per second (${inputWordCount} words in ${secs} secs)`;
-        } catch (error) {
-            result = `Error: ${error.message}`;
-        }
-        log(result);
-        log('Posting message back to main script');
-        postMessage(['translated_result', translatedParagraphs, result]);
-    }
-}
-
-// This function downloads file from a url and returns the array buffer
-const downloadAsArrayBuffer = async(url) => {
-    const response = await fetch(url);
-    if (!response.ok) {
-        throw Error(`Downloading ${url} failed: HTTP ${response.status} - ${response.statusText}`);
-    }
-    return response.arrayBuffer();
-}
-
-// This function constructs and initializes the AlignedMemory from the array buffer and alignment size
-const prepareAlignedMemoryFromBuffer = async (buffer, alignmentSize) => {
-    var byteArray = new Int8Array(buffer);
-    log(`Constructing Aligned memory with size: ${byteArray.byteLength} bytes with alignment: ${alignmentSize}`);
-    var alignedMemory = new Module.AlignedMemory(byteArray.byteLength, alignmentSize);
-    log(`Aligned memory construction done`);
-    const alignedByteArrayView = alignedMemory.getByteArrayView();
-    alignedByteArrayView.set(byteArray);
-    log(`Aligned memory initialized`);
-    return alignedMemory;
-}
-
-// Instantiate the Translation Service
-const constructTranslationService = async () => {
-    if (!translationService) {
-        var translationServiceConfig = {};
-        log(`Creating Translation Service with config: ${translationServiceConfig}`);
-        translationService = new Module.BlockingService(translationServiceConfig);
-        log(`Translation Service created successfully`);
-    }
-}
-
-const constructTranslationModel = async (from, to) => {
-    const languagePair = `${from}${to}`;
-    if (translationModels.has(languagePair)) {
-        var oldModel = translationModels.get(languagePair);
-        // Destruct the old TranslationModel explicitly and Remove its entry from the map
-        oldModel.delete();
-        translationModels.delete(languagePair);
-    }
-
-    // Vocab files are re-used in both translation directions
-    const vocabLanguagePair = from === "en" ? `${to}${from}` : languagePair;
-
-    // Set the Model Configuration as YAML formatted string.
-    // For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-    /*const modelConfig = `models:
-        - /${languagePair}/model.${languagePair}.intgemm.alphas.bin
-        vocabs:
-        - /${languagePair}/vocab.${vocabLanguagePair}.spm
-        - /${languagePair}/vocab.${vocabLanguagePair}.spm
-        beam-size: 1
-        normalize: 1.0
-        word-penalty: 0
-        max-length-break: 128
-        mini-batch-words: 1024
-        workspace: 128
-        max-length-factor: 2.0
-        skip-cost: true
-        cpu-threads: 0
-        quiet: true
-        quiet-translation: true
-        shortlist:
-            - /${languagePair}/lex.${languagePair}.s2t
-            - 50
-            - 50
-        `;
-        */
-
-    // TODO: gemm-precision: int8shiftAlphaAll (for the models that support this)
-    // DONOT CHANGE THE SPACES BETWEEN EACH ENTRY OF CONFIG
-    const modelConfig = `beam-size: 1
-normalize: 1.0
-word-penalty: 0
-max-length-break: 128
-mini-batch-words: 1024
-workspace: 128
-max-length-factor: 2.0
-skip-cost: true
-cpu-threads: 0
-quiet: true
-quiet-translation: true
-gemm-precision: int8shift
-`;
-
-    const modelFile = `models/${languagePair}/model.${languagePair}.intgemm.alphas.bin`;
-    const shortlistFile = `models/${languagePair}/lex.50.50.${languagePair}.s2t.bin`;
-    const vocabFiles = [`models/${languagePair}/vocab.${vocabLanguagePair}.spm`,
-                        `models/${languagePair}/vocab.${vocabLanguagePair}.spm`];
-
-    const uniqueVocabFiles = new Set(vocabFiles);
-    log(`modelFile: ${modelFile}\nshortlistFile: ${shortlistFile}\nNo. of unique vocabs: ${uniqueVocabFiles.size}`);
-    uniqueVocabFiles.forEach(item => log(`unique vocabFile: ${item}`));
-
-    // Download the files as buffers from the given urls
-    let start = Date.now();
-    const downloadedBuffers = await Promise.all([downloadAsArrayBuffer(modelFile), downloadAsArrayBuffer(shortlistFile)]);
-    const modelBuffer = downloadedBuffers[0];
-    const shortListBuffer = downloadedBuffers[1];
-
-    const downloadedVocabBuffers = [];
-    for (let item of uniqueVocabFiles.values()) {
-        downloadedVocabBuffers.push(await downloadAsArrayBuffer(item));
-    }
-    log(`All files for ${languagePair} language pair took ${(Date.now() - start) / 1000} secs to download`);
-
-    // Construct AlignedMemory objects with downloaded buffers
-    let constructedAlignedMemories = await Promise.all([prepareAlignedMemoryFromBuffer(modelBuffer, 256),
-                                                            prepareAlignedMemoryFromBuffer(shortListBuffer, 64)]);
-    let alignedModelMemory = constructedAlignedMemories[0];
-    let alignedShortlistMemory = constructedAlignedMemories[1];
-    let alignedVocabsMemoryList = new Module.AlignedMemoryList;
-    for(let item of downloadedVocabBuffers) {
-        let alignedMemory = await prepareAlignedMemoryFromBuffer(item, 64);
-        alignedVocabsMemoryList.push_back(alignedMemory);
-    }
-    log(`Aligned vocab memories: ${alignedVocabsMemoryList.get(0).size()}`);
-    log(`Aligned model memory: ${alignedModelMemory.size()}`);
-    log(`Aligned shortlist memory: ${alignedShortlistMemory.size()}`);
-
-    log(`Creating Translation Model with config: ${modelConfig}`);
-    var translationModel = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
-    translationModels.set(languagePair, translationModel);
-}
-
-const translate = (from, to, paragraphs) => {
-    const languagePair = `${from}${to}`;
-    if (!translationModels.has(languagePair)) {
-        throw Error(`Please load translation model '${languagePair}' before translating`);
-    }
-    translationModel = translationModels.get(languagePair);
-
-    // Instantiate the arguments of translate() API i.e. ResponseOptions and input (vector<string>)
-    var responseOptions = new Module.ResponseOptions();
-    let input = new Module.VectorString;
-
-    // Initialize the input
-    paragraphs.forEach(paragraph => {
-      // prevent empty paragraph - it breaks the translation
-        if (paragraph.trim() === "") {
-           return;
-        }
-        input.push_back(paragraph.trim())
-    })
-    // Access input (just for debugging)
-    log(`Input size: ${input.size()}`);
-
-    // Translate the input, which is a vector<String>; the result is a vector<Response>
-    let result = translationService.translate(translationModel, input, responseOptions);
-
-    const translatedParagraphs = [];
-    const translatedSentencesOfParagraphs = [];
-    const sourceSentencesOfParagraphs = [];
-    for (let i = 0; i < result.size(); i++) {
-        translatedParagraphs.push(result.get(i).getTranslatedText());
-        translatedSentencesOfParagraphs.push(getAllTranslatedSentencesOfParagraph(result.get(i)));
-        sourceSentencesOfParagraphs.push(getAllSourceSentencesOfParagraph(result.get(i)));
-    }
-    log({ translatedParagraphs });
-    log({ translatedSentencesOfParagraphs });
-    log({ sourceSentencesOfParagraphs });
-
-    responseOptions.delete();
-    input.delete();
-    return translatedParagraphs;
-}
-
-// This function extracts all the translated sentences from the Response and returns them.
-const getAllTranslatedSentencesOfParagraph = (response) => {
-    const sentences = [];
-    const text = response.getTranslatedText();
-    for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
-        const utf8SentenceByteRange = response.getTranslatedSentence(sentenceIndex);
-        sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
-    }
-    return sentences;
-}
-
-// This function extracts all the source sentences from the Response and returns them.
-const getAllSourceSentencesOfParagraph = (response) => {
-    const sentences = [];
-    const text = response.getOriginalText();
-    for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
-        const utf8SentenceByteRange = response.getSourceSentence(sentenceIndex);
-        sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
-    }
-    return sentences;
-}
-
-// This function returns a substring of text (a string). The substring is represented by
-// byteRange (begin and end endices) within the utf-8 encoded version of the text.
-const _getSentenceFromByteRange = (text, byteRange) => {
-    const utf8BytesView = encoder.encode(text);
-    const utf8SentenceBytes = utf8BytesView.subarray(byteRange.begin, byteRange.end);
-    return decoder.decode(utf8SentenceBytes);
-}

From c5167b3d8cda016f305d192f438a54e841cbc46c Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 27 Oct 2021 11:54:39 +0200
Subject: [PATCH 293/442] Import matrix-multiply from a separate wasm module
 (#232)

* Updated marian-dev submodule
* Import wasm gemm from a separate wasm module
 - The fallback implementation of gemm is currently being imported dynamically
   for wasm target
* Updated CI scripts and README to import GEMM from a separate wasm module
* Setting model config to int8shiftAlphaAll in wasm test page
---
 .github/workflows/wasm-custom_marian-mac.yml  |  4 ++
 .../workflows/wasm-custom_marian-ubuntu.yml   |  4 ++
 3rd_party/marian-dev                          |  2 +-
 README.md                                     | 10 +++++
 build-wasm.sh                                 |  3 ++
 wasm/CMakeLists.txt                           |  6 +++
 wasm/patch-artifacts-import-gemm-module.sh    | 44 +++++++++++++++++++
 wasm/test_page/js/worker.js                   |  2 +-
 8 files changed, 73 insertions(+), 2 deletions(-)
 create mode 100644 wasm/patch-artifacts-import-gemm-module.sh

diff --git a/.github/workflows/wasm-custom_marian-mac.yml b/.github/workflows/wasm-custom_marian-mac.yml
index 746fb9cdd..636323581 100644
--- a/.github/workflows/wasm-custom_marian-mac.yml
+++ b/.github/workflows/wasm-custom_marian-mac.yml
@@ -39,6 +39,10 @@ jobs:
         working-directory: build-wasm
         run: bash ../wasm/patch-artifacts-enable-wormhole.sh
 
+      - name: Import GEMM library from a separate wasm module
+        working-directory: build-wasm
+        run: bash ../wasm/patch-artifacts-import-gemm-module.sh
+
       - name: Check artifacts
         working-directory: build-wasm
         run: |
diff --git a/.github/workflows/wasm-custom_marian-ubuntu.yml b/.github/workflows/wasm-custom_marian-ubuntu.yml
index dcea92850..b644d9763 100644
--- a/.github/workflows/wasm-custom_marian-ubuntu.yml
+++ b/.github/workflows/wasm-custom_marian-ubuntu.yml
@@ -39,6 +39,10 @@ jobs:
         working-directory: build-wasm
         run: bash ../wasm/patch-artifacts-enable-wormhole.sh
 
+      - name: Import GEMM library from a separate wasm module
+        working-directory: build-wasm
+        run: bash ../wasm/patch-artifacts-import-gemm-module.sh
+
       - name: Check artifacts
         working-directory: build-wasm
         run: |
diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 62bac858b..a1a82ff64 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 62bac858bfd37060beb707d12eb9711649ea4cf6
+Subproject commit a1a82ff64910dc066d64d631cd7a8212df9f88cd
diff --git a/README.md b/README.md
index 3ba11f026..156d12875 100644
--- a/README.md
+++ b/README.md
@@ -46,6 +46,11 @@ To build a version that translates with higher speeds on Firefox Nightly browser
        bash ../wasm/patch-artifacts-enable-wormhole.sh
        ```
 
+   3. Patch generated artifacts to import GEMM library from a separate wasm module
+       ```bash
+       bash ../wasm/patch-artifacts-import-gemm-module.sh
+       ```
+
 To build a version that runs on all browsers (including Firefox Nightly) but translates slowly, follow these instructions:
 
   1. Create a folder where you want to build all the artifacts (`build-wasm` in this case) and compile
@@ -56,6 +61,11 @@ To build a version that runs on all browsers (including Firefox Nightly) but tra
       emmake make -j2
       ```
 
+  2. Patch generated artifacts to import GEMM library from a separate wasm module
+       ```bash
+       bash ../wasm/patch-artifacts-import-gemm-module.sh
+       ```
+
 #### Recompiling
 As long as you don't update any submodule, just follow [Compile](#Compile) steps.\
 If you update a submodule, execute following command in repository root folder before executing
diff --git a/build-wasm.sh b/build-wasm.sh
index 7da2685cf..adc6556c3 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -78,5 +78,8 @@ if [ "$WORMHOLE" = true ]; then
   bash ../wasm/patch-artifacts-enable-wormhole.sh
 fi
 
+#     3. Import GEMM library from a separate wasm module
+bash ../wasm/patch-artifacts-import-gemm-module.sh
+
 # The artifacts (.js and .wasm files) will be available in the build directory
 exit 0
diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 1580defa1..92c9e1698 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -26,6 +26,12 @@ set(LINKER_FLAGS "${LINKER_FLAGS} -s ENVIRONMENT=web,worker")
 # Append version information in the Javascript artifact
 set(LINKER_FLAGS "${LINKER_FLAGS} --extern-pre-js ${CMAKE_CURRENT_BINARY_DIR}/project_version.js")
 
+# Allow importing undefined symbols dynamically
+set(LINKER_FLAGS "${LINKER_FLAGS} -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s DECLARE_ASM_MODULE_EXPORTS=0")
+
+# Export all the functions of fallback implementation of GEMM for wasm target
+set(LINKER_FLAGS "${LINKER_FLAGS} -s EXPORTED_FUNCTIONS=[_int8PrepareAFallback,_int8PrepareBFallback,_int8PrepareBFromTransposedFallback,_int8PrepareBFromQuantizedTransposedFallback,_int8PrepareBiasFallback,_int8MultiplyAndAddBiasFallback,_int8SelectColumnsOfBFallback]")
+
 set_target_properties(bergamot-translator-worker PROPERTIES
                         SUFFIX ".js"
                         LINK_FLAGS ${LINKER_FLAGS}
diff --git a/wasm/patch-artifacts-import-gemm-module.sh b/wasm/patch-artifacts-import-gemm-module.sh
new file mode 100644
index 000000000..2f2e29afd
--- /dev/null
+++ b/wasm/patch-artifacts-import-gemm-module.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+usage="Patch wasm artifacts to import fallback implementation of gemm for wasm.
+
+Usage: $(basename "$0") [WASM_ARTIFACTS_FOLDER]
+
+    where:
+    WASM_ARTIFACTS_FOLDER    Folder containing wasm artifacts
+                             (An optional argument, if unspecified the default is: current folder)"
+
+if [ "$#" -gt 1 ]; then
+    echo "Illegal number of parameters passed"
+    echo "$usage"
+    exit
+fi
+
+# Parse wasm artifacts folder if provided via script argument or set it to default
+WASM_ARTIFACTS_FOLDER=$PWD
+if [ "$#" -eq 1 ]; then
+    if [ ! -e "$1" ]; then
+        echo "Error: Folder \""$1"\" doesn't exist"
+        exit
+    fi
+    WASM_ARTIFACTS_FOLDER="$1"
+fi
+
+WASM_ARTIFACTS_JAVASCRIPT_FILE="bergamot-translator-worker.js"
+WASM_ARTIFACTS="$WASM_ARTIFACTS_FOLDER/${WASM_ARTIFACTS_JAVASCRIPT_FILE}"
+if [ ! -e "$WASM_ARTIFACTS" ]; then
+    echo "Error: Artifact \"$WASM_ARTIFACTS\" doesn't exist"
+    exit
+fi
+
+echo "Polyfill the fallback integer (8-bit) gemm implementation from the main module"
+sed -i.bak 's/"env"[[:space:]]*:[[:space:]]*asmLibraryArg,/"env": asmLibraryArg,\
+    "wasm_gemm":{\
+    "int8_prepare_a": (...a) => Module["asm"].int8PrepareAFallback(...a),\
+    "int8_prepare_b": (...a) => Module["asm"].int8PrepareBFallback(...a),\
+    "int8_prepare_b_from_transposed": (...a) => Module["asm"].int8PrepareBFromTransposedFallback(...a),\
+    "int8_prepare_b_from_quantized_transposed": (...a) => Module["asm"].int8PrepareBFromQuantizedTransposedFallback(...a),\
+    "int8_prepare_bias": (...a) => Module["asm"].int8PrepareBiasFallback(...a),\
+    "int8_multiply_and_add_bias": (...a) => Module["asm"].int8MultiplyAndAddBiasFallback(...a),\
+    "int8_select_columns_of_b": (...a) => Module["asm"].int8SelectColumnsOfBFallback(...a),\
+    },/g' ${WASM_ARTIFACTS_JAVASCRIPT_FILE}
+echo "SUCCESS"
\ No newline at end of file
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index 1cf3a1461..189658903 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -180,7 +180,7 @@ skip-cost: true
 cpu-threads: 0
 quiet: true
 quiet-translation: true
-gemm-precision: int8shiftAll
+gemm-precision: int8shiftAlphaAll
 `;
 
   const modelFile = `${rootURL}/${languagePair}/${modelRegistry[languagePair]["model"].name}`;

From d0d08c0f54b12868717c510d4118c52d4687bfa0 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 27 Oct 2021 19:26:55 +0200
Subject: [PATCH 294/442] JS bindings for Quality Estimation (#239)

* Quality Score bindings complete
* Updated wasm test page to test the bindings
  - Word and sentence scores can be seen in browser console
---
 wasm/bindings/response_bindings.cpp         |  19 ++-
 wasm/bindings/response_options_bindings.cpp |   9 +-
 wasm/test_page/js/worker.js                 | 159 ++++++++++++++------
 3 files changed, 137 insertions(+), 50 deletions(-)

diff --git a/wasm/bindings/response_bindings.cpp b/wasm/bindings/response_bindings.cpp
index ca688249c..11bc4cabb 100644
--- a/wasm/bindings/response_bindings.cpp
+++ b/wasm/bindings/response_bindings.cpp
@@ -9,25 +9,36 @@
 
 #include "response.h"
 
-typedef marian::bergamot::Response Response;
+using Response = marian::bergamot::Response;
+using SentenceQualityScore = marian::bergamot::Response::SentenceQualityScore;
+using ByteRange = marian::bergamot::ByteRange;
 
 using namespace emscripten;
 
 // Binding code
 EMSCRIPTEN_BINDINGS(byte_range) {
-  value_object<marian::bergamot::ByteRange>("ByteRange")
-      .field("begin", &marian::bergamot::ByteRange::begin)
-      .field("end", &marian::bergamot::ByteRange::end);
+  value_object<ByteRange>("ByteRange").field("begin", &ByteRange::begin).field("end", &ByteRange::end);
 }
 
+std::vector<SentenceQualityScore> getQualityScores(const Response& response) { return response.qualityScores; }
+
 EMSCRIPTEN_BINDINGS(response) {
   class_<Response>("Response")
       .constructor<>()
       .function("size", &Response::size)
+      .function("getQualityScores", &getQualityScores)
       .function("getOriginalText", &Response::getOriginalText)
       .function("getTranslatedText", &Response::getTranslatedText)
       .function("getSourceSentence", &Response::getSourceSentenceAsByteRange)
       .function("getTranslatedSentence", &Response::getTargetSentenceAsByteRange);
 
+  value_object<SentenceQualityScore>("SentenceQualityScore")
+      .field("wordScores", &SentenceQualityScore::wordScores)
+      .field("wordByteRanges", &SentenceQualityScore::wordByteRanges)
+      .field("sentenceScore", &SentenceQualityScore::sentenceScore);
+
   register_vector<Response>("VectorResponse");
+  register_vector<SentenceQualityScore>("VectorSentenceQualityScore");
+  register_vector<float>("VectorFloat");
+  register_vector<ByteRange>("VectorByteRange");
 }
diff --git a/wasm/bindings/response_options_bindings.cpp b/wasm/bindings/response_options_bindings.cpp
index e2bf8e1f5..4addbcbfc 100644
--- a/wasm/bindings/response_options_bindings.cpp
+++ b/wasm/bindings/response_options_bindings.cpp
@@ -7,9 +7,14 @@
 
 #include "response_options.h"
 
-typedef marian::bergamot::ResponseOptions ResponseOptions;
+using ResponseOptions = marian::bergamot::ResponseOptions;
 
 using namespace emscripten;
 
 // Binding code
-EMSCRIPTEN_BINDINGS(response_options) { class_<ResponseOptions>("ResponseOptions").constructor<>(); }
+EMSCRIPTEN_BINDINGS(response_options) {
+  value_object<ResponseOptions>("ResponseOptions")
+      .field("qualityScores", &ResponseOptions::qualityScores)
+      .field("alignment", &ResponseOptions::alignment)
+      .field("alignmentThreshold", &ResponseOptions::alignmentThreshold);
+}
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index 189658903..f6dc83623 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -1,5 +1,5 @@
 // All variables specific to translation service
-var translationService, responseOptions, input = undefined;
+var translationService = undefined;
 // A map of language-pair to TranslationModel object
 var languagePairToTranslationModels = new Map();
 
@@ -51,14 +51,14 @@ onmessage = async function(e) {
   } else if (command === 'translate') {
       const from = e.data[1];
       const to = e.data[2];
-      const inputParagraphs = e.data[3];
+      const input = e.data[3];
       let inputWordCount = 0;
-      inputParagraphs.forEach(sentence => {
+      input.forEach(sentence => {
         inputWordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
       })
       let start = Date.now();
       try {
-        result = translate(from, to, inputParagraphs);
+        result = translate(from, to, input);
         const secs = (Date.now() - start) / 1000;
         log(`Translation '${from}${to}' Successful. Speed: ${Math.round(inputWordCount / secs)} WPS (${inputWordCount} words in ${secs} secs)`);
       } catch (error) {
@@ -102,17 +102,17 @@ const constructTranslationModel = async (from, to) => {
 }
 
 // Translates text from source language to target language.
-const translate = (from, to, paragraphs) => {
+const translate = (from, to, input) => {
   // If none of the languages is English then perform translation with
   // English as a pivot language.
   if (from !== 'en' && to !== 'en') {
     log(`Translating '${from}${to}' via pivoting: '${from}en' -> 'en${to}'`);
-    let translatedParagraphsInEnglish = _translateInvolvingEnglish(from, 'en', paragraphs);
-    return _translateInvolvingEnglish('en', to, translatedParagraphsInEnglish);
+    const translatedTextInEnglish = _translateInvolvingEnglish(from, 'en', input);
+    return _translateInvolvingEnglish('en', to, translatedTextInEnglish);
   }
   else {
     log(`Translating '${from}${to}'`);
-    return _translateInvolvingEnglish(from, to, paragraphs);
+    return _translateInvolvingEnglish(from, to, input);
   }
 }
 
@@ -225,64 +225,135 @@ gemm-precision: int8shiftAlphaAll
   languagePairToTranslationModels.set(languagePair, translationModel);
 }
 
-const _translateInvolvingEnglish = (from, to, paragraphs) => {
+const _translateInvolvingEnglish = (from, to, input) => {
   const languagePair = `${from}${to}`;
   if (!languagePairToTranslationModels.has(languagePair)) {
     throw Error(`Please load translation model '${languagePair}' before translating`);
   }
   translationModel = languagePairToTranslationModels.get(languagePair);
 
-  // Instantiate the arguments of translate() API i.e. ResponseOptions and input (vector<string>)
-  var responseOptions = new Module.ResponseOptions();
-  let input = new Module.VectorString;
+  // Prepare the arguments of translate() API i.e. ResponseOptions and vectorSourceText (i.e. a vector<string>)
+  const responseOptions = _prepareResponseOptions();
+  let vectorSourceText = _prepareSourceText(input);
 
-  // Initialize the input
-  paragraphs.forEach(paragraph => {
-    // prevent empty paragraph - it breaks the translation
-    if (paragraph.trim() === "") {
-      return;
-    }
-    input.push_back(paragraph.trim())
-  })
+  // Call translate() API; result is vector<Response> where every item of vector<Response> corresponds
+  // to an item of vectorSourceText in the same order
+  const vectorResponse = translationService.translate(translationModel, vectorSourceText, responseOptions);
+
+  // Parse all relevant information from vectorResponse
+  const listTranslatedText = _parseTranslatedText(vectorResponse);
+  const listTranslatedTextSentences = _parseTranslatedTextSentences(vectorResponse);
+  const listSourceTextSentences = _parseSourceTextSentences(vectorResponse);
+  const listTranslatedTextSentenceQualityScores = _parseTranslatedTextSentenceQualityScores(vectorResponse);
 
-  // Access input (just for debugging)
-  log(`Input size: ${input.size()}`);
+  log(`Translated text: ${listTranslatedText}`);
+  log(`Translated sentences: ${JSON.stringify(listTranslatedTextSentences)}`);
+  log(`Source sentences: ${JSON.stringify(listSourceTextSentences)}`);
+  log(`Translated sentence quality scores: ${JSON.stringify(listTranslatedTextSentenceQualityScores)}`);
 
-  // Translate the input, which is a vector<String>; the result is a vector<Response>
-  let result = translationService.translate(translationModel, input, responseOptions);
+  // Delete prepared SourceText to avoid memory leak
+  vectorSourceText.delete();
+
+  return listTranslatedText;
+}
 
-  const translatedParagraphs = [];
-  const translatedSentencesOfParagraphs = [];
-  const sourceSentencesOfParagraphs = [];
-  for (let i = 0; i < result.size(); i++) {
-    translatedParagraphs.push(result.get(i).getTranslatedText());
-    translatedSentencesOfParagraphs.push(_getAllTranslatedSentencesOfParagraph(result.get(i)));
-    sourceSentencesOfParagraphs.push(_getAllSourceSentencesOfParagraph(result.get(i)));
+const _parseTranslatedText = (vectorResponse) => {
+  const result = [];
+  for (let i = 0; i < vectorResponse.size(); i++) {
+    const response = vectorResponse.get(i);
+    result.push(response.getTranslatedText());
   }
+  return result;
+}
+
+const _parseTranslatedTextSentences = (vectorResponse) => {
+  const result = [];
+  for (let i = 0; i < vectorResponse.size(); i++) {
+    const response = vectorResponse.get(i);
+    result.push(_getTranslatedSentences(response));
+  }
+  return result;
+}
+
+const _parseSourceTextSentences = (vectorResponse) => {
+  const result = [];
+  for (let i = 0; i < vectorResponse.size(); i++) {
+    const response = vectorResponse.get(i);
+    result.push(_getSourceSentences(response));
+  }
+  return result;
+}
 
-  responseOptions.delete();
-  input.delete();
-  return translatedParagraphs;
+const _parseTranslatedTextSentenceQualityScores = (vectorResponse) => {
+  const result = [];
+  for (let i = 0; i < vectorResponse.size(); i++) {
+    const response = vectorResponse.get(i);
+    const translatedText = response.getTranslatedText();
+    const vectorSentenceQualityScore = response.getQualityScores();
+    log(`No. of sentences: "${vectorSentenceQualityScore.size()}"`);
+    const sentenceQualityScores = [];
+    for (let sentenceIndex=0; sentenceIndex < vectorSentenceQualityScore.size(); sentenceIndex++) {
+      const sentenceQualityScoreObject = vectorSentenceQualityScore.get(sentenceIndex);
+      const wordByteRangeList = [];
+      const wordList = [];
+      const wordScoreList = [];
+      const vectorWordScore = sentenceQualityScoreObject.wordScores;
+      const vectorWordByteRange = sentenceQualityScoreObject.wordByteRanges;
+
+      for (let wordIndex = 0; wordIndex < vectorWordScore.size(); wordIndex++) {
+        const wordScore = vectorWordScore.get(wordIndex);
+        const wordByteRange = vectorWordByteRange.get(wordIndex);
+        wordScoreList.push(wordScore);
+        wordByteRangeList.push(wordByteRange);
+        const word = _getSubString(translatedText, wordByteRange);
+        wordList.push(word);
+      }
+
+      const sentenceQualityScore = {
+        wordByteRanges: wordByteRangeList,
+        words: wordList,
+        wordScores: wordScoreList,
+        sentenceScore: sentenceQualityScoreObject.sentenceScore
+      };
+      sentenceQualityScores.push(sentenceQualityScore);
+    }
+    result.push(sentenceQualityScores);
+  }
+  return result;
+}
+
+const _prepareResponseOptions = () => {
+  return {qualityScores: true, alignment: false, alignmentThreshold: 0.2};
+}
+
+const _prepareSourceText = (input) => {
+  let vectorSourceText = new Module.VectorString;
+  input.forEach(paragraph => {
+    // prevent empty paragraph - it breaks the translation
+    if (paragraph.trim() === "") {
+      return;
+    }
+    vectorSourceText.push_back(paragraph.trim())
+  })
+  return vectorSourceText;
 }
 
-// Extracts all the translated sentences from the Response and returns them.
-const _getAllTranslatedSentencesOfParagraph = (response) => {
+const _getTranslatedSentences = (response) => {
   const sentences = [];
   const text = response.getTranslatedText();
   for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
     const utf8SentenceByteRange = response.getTranslatedSentence(sentenceIndex);
-    sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
+    sentences.push(_getSubString(text, utf8SentenceByteRange));
   }
   return sentences;
 }
 
-// Extracts all the source sentences from the Response and returns them.
-const _getAllSourceSentencesOfParagraph = (response) => {
+const _getSourceSentences = (response) => {
   const sentences = [];
   const text = response.getOriginalText();
   for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
     const utf8SentenceByteRange = response.getSourceSentence(sentenceIndex);
-    sentences.push(_getSentenceFromByteRange(text, utf8SentenceByteRange));
+    sentences.push(_getSubString(text, utf8SentenceByteRange));
   }
   return sentences;
 }
@@ -291,8 +362,8 @@ const _getAllSourceSentencesOfParagraph = (response) => {
  * Returns a substring of text (a string). The substring is represented by
  * byteRange (begin and end endices) within the utf-8 encoded version of the text.
  */
-const _getSentenceFromByteRange = (text, byteRange) => {
-  const utf8BytesView = encoder.encode(text);
-  const utf8SentenceBytes = utf8BytesView.subarray(byteRange.begin, byteRange.end);
-  return decoder.decode(utf8SentenceBytes);
+const _getSubString = (text, utf8ByteRange) => {
+  const textUtf8ByteView = encoder.encode(text);
+  const substringUtf8ByteView = textUtf8ByteView.subarray(utf8ByteRange.begin, utf8ByteRange.end);
+  return decoder.decode(substringUtf8ByteView);
 }

From 2b98c67996eb2df7f3233c293eeb640e3b0b2fa3 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Wed, 27 Oct 2021 20:37:05 +0100
Subject: [PATCH 295/442] Cache for translations (#227)

Sets a cache to operate for each sentence that a TranslationModel process
caching the corresponding marian::History for a {TranslationModel::Id, marian::Words}
key.  Cache is thus shared across multiple TranslationModels bound to the lifetime
of a Service. Cache gracefully downgrades in the case of WebAssembly.
---
 bergamot-translator-tests                  |  2 +-
 src/tests/apps.cpp                         | 33 ++++++++++
 src/tests/apps.h                           |  2 +
 src/tests/cli.cpp                          | 10 ++-
 src/tests/units/CMakeLists.txt             |  1 +
 src/tests/units/cache_tests.cpp            | 56 ++++++++++++++++
 src/translator/aggregate_batching_pool.cpp |  4 +-
 src/translator/batching_pool.cpp           | 15 +++--
 src/translator/cache.h                     | 75 ++++++++++++++++++++++
 src/translator/parser.cpp                  |  6 ++
 src/translator/parser.h                    |  6 ++
 src/translator/request.cpp                 | 56 ++++++++++++++--
 src/translator/request.h                   | 18 +++++-
 src/translator/service.cpp                 | 13 ++--
 src/translator/service.h                   | 23 ++++++-
 src/translator/translation_model.cpp       | 11 +++-
 src/translator/translation_model.h         | 11 +++-
 17 files changed, 314 insertions(+), 28 deletions(-)
 create mode 100644 src/tests/units/cache_tests.cpp
 create mode 100644 src/translator/cache.h

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 9dc3c5e9a..6bd396922 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 9dc3c5e9a1027c1d6b4a467a27bdff16d0d6a006
+Subproject commit 6bd396922b2159b62c55530cb3ee6a40323d4171
diff --git a/src/tests/apps.cpp b/src/tests/apps.cpp
index 63febfaf0..20c6d2acb 100644
--- a/src/tests/apps.cpp
+++ b/src/tests/apps.cpp
@@ -108,6 +108,39 @@ void qualityEstimatorScores(AsyncService &service, Ptr<TranslationModel> model)
   }
 }
 
+void translationCache(AsyncService &service, Ptr<TranslationModel> model) {
+  ResponseOptions responseOptions;
+
+  // Read a large input text blob from stdin
+  const std::string source = readFromStdin();
+
+  // Round 1
+  std::string buffer = source;
+  Response firstResponse = translateForResponse(service, model, std::move(buffer), responseOptions);
+
+  auto statsFirstRun = service.cacheStats();
+  LOG(info, "Cache Hits/Misses = {}/{}", statsFirstRun.hits, statsFirstRun.misses);
+  ABORT_IF(statsFirstRun.hits != 0, "Expecting no cache hits, but hits found.");
+
+  // Round 2; There should be cache hits
+  buffer = source;
+  Response secondResponse = translateForResponse(service, model, std::move(buffer), responseOptions);
+
+  auto statsSecondRun = service.cacheStats();
+  LOG(info, "Cache Hits/Misses = {}/{}", statsSecondRun.hits, statsSecondRun.misses);
+  ABORT_IF(statsSecondRun.hits <= 0, "At least one hit expected, none found.");
+  if (statsSecondRun.hits != statsFirstRun.misses) {
+    std::cerr << "Mismatch in expected hits (Hits, Misses = " << statsSecondRun.hits << ", " << statsSecondRun.misses
+              << "). This can happen due to random eviction." << std::endl;
+  }
+
+  ABORT_IF(firstResponse.target.text != secondResponse.target.text,
+           "Recompiled string provided different output when operated with cache. On the same hardware while using "
+           "same path, this is expected to be same.");
+
+  std::cout << firstResponse.target.text;
+}
+
 }  // namespace testapp
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/tests/apps.h b/src/tests/apps.h
index dee77a9be..9e45a1caa 100644
--- a/src/tests/apps.h
+++ b/src/tests/apps.h
@@ -37,6 +37,8 @@ void qualityEstimatorWords(AsyncService &service, Ptr<TranslationModel> model);
 // Reads from stdin and translates the read content. Prints the quality scores for each sentence.
 void qualityEstimatorScores(AsyncService &service, Ptr<TranslationModel> model);
 
+// Tests if cache is active and functional
+void translationCache(AsyncService &service, Ptr<TranslationModel> model);
 }  // namespace testapp
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/tests/cli.cpp b/src/tests/cli.cpp
index 90c386c84..ba4d73218 100644
--- a/src/tests/cli.cpp
+++ b/src/tests/cli.cpp
@@ -5,7 +5,11 @@ int main(int argc, char *argv[]) {
   marian::bergamot::ConfigParser configParser;
   configParser.parseArgs(argc, argv);
   auto &config = configParser.getConfig();
-  AsyncService::Config serviceConfig{config.numWorkers};
+  AsyncService::Config serviceConfig;
+  serviceConfig.numWorkers = config.numWorkers;
+  serviceConfig.cacheEnabled = config.cacheEnabled;
+  serviceConfig.cacheMutexBuckets = config.cacheMutexBuckets;
+  serviceConfig.cacheSize = config.cacheSize;
   AsyncService service(serviceConfig);
   std::vector<std::shared_ptr<TranslationModel>> models;
 
@@ -37,6 +41,10 @@ int main(int argc, char *argv[]) {
     case OpMode::TEST_QUALITY_ESTIMATOR_SCORES:
       testapp::qualityEstimatorScores(service, models.front());
       break;
+    case OpMode::TEST_TRANSLATION_CACHE:
+      testapp::translationCache(service, models.front());
+      break;
+
     default:
       ABORT("Incompatible op-mode. Choose one of the test modes.");
       break;
diff --git a/src/tests/units/CMakeLists.txt b/src/tests/units/CMakeLists.txt
index 4794badcd..2570e05e7 100644
--- a/src/tests/units/CMakeLists.txt
+++ b/src/tests/units/CMakeLists.txt
@@ -1,6 +1,7 @@
 # Unit tests
 set(UNIT_TESTS
     annotation_tests
+    cache_tests
     quality_estimator_tests)
 
 foreach(test ${UNIT_TESTS})
diff --git a/src/tests/units/cache_tests.cpp b/src/tests/units/cache_tests.cpp
new file mode 100644
index 000000000..f2f1b19ed
--- /dev/null
+++ b/src/tests/units/cache_tests.cpp
@@ -0,0 +1,56 @@
+
+#include <random>
+#include <thread>
+
+#include "catch.hpp"
+#include "translator/cache.h"
+#include "translator/history.h"
+
+using namespace marian::bergamot;
+
+TEST_CASE("Test Cache in a threaded setting") {
+  size_t numThreads = 100;
+  size_t numIters = 10000;
+  using Key = int;
+  using Value = int;
+  using TestCache = AtomicCache<Key, Value>;
+
+  TestCache cache(/*size=*/300, /*mutexBuckets=*/16);
+
+  auto op = [numIters, &cache]() {
+    std::mt19937_64 randomGenerator;
+    randomGenerator.seed(42);  // reproducible outputs
+    Value randMax = 2000;
+
+    for (size_t i = 0; i < numIters; i++) {
+      Key query = randomGenerator() % randMax;
+      std::pair<bool, Value> result = cache.find(query);
+      if (result.first) {
+        REQUIRE(result.second == query);
+      }
+
+      Value value = query;
+      cache.store(/*key=*/query, std::move(value));
+    }
+  };
+
+  std::vector<std::thread> workers;
+  for (size_t t = 0; t < numThreads; t++) {
+    workers.emplace_back(op);
+  }
+
+  for (size_t t = 0; t < numThreads; t++) {
+    workers[t].join();
+  }
+
+  TestCache::Stats stats = cache.stats();
+  float hitRate = static_cast<float>(stats.hits) / static_cast<float>(stats.hits + stats.misses);
+
+  // This is non-deterministic due to threads.
+  std::cout << "Hit-Rate:" << hitRate << "\n";
+  std::cout << "(Hits, Misses) = " << stats.hits << " " << stats.misses << "\n";
+
+  // Can we create a specialization of the actual cache-type we want? Does it compile, at least?
+  // We already have Ptr<History>, it's easier to move Ptr<History> to cache.
+  TranslationCache translationCache(/*size=*/300, /*mutexBuckets=*/16);
+}
diff --git a/src/translator/aggregate_batching_pool.cpp b/src/translator/aggregate_batching_pool.cpp
index 38c55f1c4..60f5fcd2e 100644
--- a/src/translator/aggregate_batching_pool.cpp
+++ b/src/translator/aggregate_batching_pool.cpp
@@ -9,9 +9,9 @@ AggregateBatchingPool::AggregateBatchingPool() {
 }
 
 size_t AggregateBatchingPool::enqueueRequest(Ptr<TranslationModel> model, Ptr<Request> request) {
-  model->enqueueRequest(request);
+  size_t sentencesEnqueued = model->enqueueRequest(request);
   aggregateQueue_.insert(model);
-  return request->numSegments();
+  return sentencesEnqueued;
 }
 
 size_t AggregateBatchingPool::generateBatch(Ptr<TranslationModel>& model, Batch& batch) {
diff --git a/src/translator/batching_pool.cpp b/src/translator/batching_pool.cpp
index 83b5e00ab..1033e80cc 100644
--- a/src/translator/batching_pool.cpp
+++ b/src/translator/batching_pool.cpp
@@ -44,14 +44,19 @@ size_t BatchingPool::generateBatch(Batch &batch) {
 }
 
 size_t BatchingPool::enqueueRequest(Ptr<Request> request) {
+  size_t toBeFreshlyTranslated = 0;
   for (size_t i = 0; i < request->numSegments(); i++) {
-    RequestSentence sentence(i, request);
-    size_t bucket_id = sentence.numTokens();
-    assert(bucket_id < bucket_.size());
-    bucket_[bucket_id].insert(sentence);
+    if (!request->cacheHitPrefilled(i)) {
+      RequestSentence sentence(i, request);
+      size_t bucket_id = sentence.numTokens();
+      assert(bucket_id < bucket_.size());
+      bucket_[bucket_id].insert(sentence);
+
+      toBeFreshlyTranslated += 1;
+    }
   }
 
-  return request->numSegments();
+  return toBeFreshlyTranslated;
 }
 
 }  // namespace bergamot
diff --git a/src/translator/cache.h b/src/translator/cache.h
new file mode 100644
index 000000000..ba68e4e93
--- /dev/null
+++ b/src/translator/cache.h
@@ -0,0 +1,75 @@
+#pragma once
+#include <memory>
+#include <mutex>
+#include <vector>
+
+#include "definitions.h"
+#include "translator/history.h"
+
+namespace marian::bergamot {
+
+template <class Key, class Value, class Hash = std::hash<Key>, class Equals = std::equal_to<Key>>
+class AtomicCache {
+ public:
+  struct Stats {
+    size_t hits{0};
+    size_t misses{0};
+  };
+
+  explicit AtomicCache(size_t size, size_t buckets) : records_(size), mutexBuckets_(buckets) {}
+
+  std::pair<bool, Value> find(const Key &key) const {
+    Value value;
+    bool found = atomicLoad(key, value);
+    return std::make_pair(found, value);
+  }
+
+  void store(const Key &key, Value value) { atomicStore(key, value); }
+
+  const Stats stats() const { return stats_; }
+
+ private:
+  using Record = std::pair<Key, Value>;
+
+  bool atomicLoad(const Key &key, Value &value) const {
+    // No probing, direct map onto records_
+    size_t index = hash_(key) % records_.size();
+    size_t mutexId = index % mutexBuckets_.size();
+
+    std::lock_guard<std::mutex> lock(mutexBuckets_[mutexId]);
+    const Record &candidate = records_[index];
+    if (equals_(key, candidate.first)) {
+      value = candidate.second;
+      stats_.hits += 1;
+      return true;
+    } else {
+      stats_.misses += 1;
+    }
+
+    return false;
+  }
+
+  void atomicStore(const Key &key, Value value) {
+    // No probing, direct map onto records_
+    size_t index = hash_(key) % records_.size();
+    size_t mutexId = index % mutexBuckets_.size();
+
+    std::lock_guard<std::mutex> lock(mutexBuckets_[mutexId]);
+    Record &candidate = records_[index];
+
+    candidate.first = key;
+    candidate.second = value;
+  }
+
+  std::vector<Record> records_;
+
+  mutable std::vector<std::mutex> mutexBuckets_;
+  mutable Stats stats_;
+
+  Hash hash_;
+  Equals equals_;
+};
+
+typedef AtomicCache<size_t, Ptr<History>> TranslationCache;
+
+}  // namespace marian::bergamot
diff --git a/src/translator/parser.cpp b/src/translator/parser.cpp
index d927409b5..2295fd6c9 100644
--- a/src/translator/parser.cpp
+++ b/src/translator/parser.cpp
@@ -24,6 +24,7 @@ std::istringstream &operator>>(std::istringstream &in, OpMode &mode) {
       {"test-quality-estimator-words", OpMode::TEST_QUALITY_ESTIMATOR_WORDS},
       {"test-quality-estimator-scores", OpMode::TEST_QUALITY_ESTIMATOR_SCORES},
       {"test-forward-backward", OpMode::TEST_FORWARD_BACKWARD_FOR_OUTBOUND},
+      {"test-translation-cache", OpMode::TEST_TRANSLATION_CACHE},
   };
 
   auto query = table.find(modeString);
@@ -84,6 +85,11 @@ void ConfigParser::addOptionsBoundToConfig(CLI::App &app, CLIConfig &config) {
   app.add_option("--cpu-threads", config.numWorkers, "Number of worker threads to use for translation");
 
   app_.add_option("--bergamot-mode", config.opMode, "Operating mode for bergamot: [wasm, native, decoder]");
+
+  app_.add_option("--cache-translations", config.cacheEnabled, "Whether to cache translations or not.");
+  app_.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
+  app_.add_option("--cache-mutex-buckets", config.cacheMutexBuckets,
+                  "Number of mutex buckets to control locking granularity");
 }
 
 std::shared_ptr<marian::Options> parseOptionsFromFilePath(const std::string &configPath, bool validate /*= true*/) {
diff --git a/src/translator/parser.h b/src/translator/parser.h
index c9fffcebf..80006f3b0 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -25,6 +25,7 @@ enum OpMode {
   TEST_QUALITY_ESTIMATOR_WORDS,
   TEST_QUALITY_ESTIMATOR_SCORES,
   TEST_FORWARD_BACKWARD_FOR_OUTBOUND,
+  TEST_TRANSLATION_CACHE,
 };
 
 /// Overload for CL11, convert a read from a stringstream into opmode.
@@ -37,6 +38,11 @@ struct CLIConfig {
   bool validateByteArray;
   size_t numWorkers;
   OpMode opMode;
+
+  // Cache parameters
+  bool cacheEnabled{false};
+  size_t cacheSize{20};
+  size_t cacheMutexBuckets{4};
 };
 
 /// ConfigParser for bergamot. Internally stores config options with CLIConfig. CLI11 parsing binds the parsing code to
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index 9bdae9f74..feba62a4a 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -3,28 +3,63 @@
 #include <string>
 
 #include "annotation.h"
+#include "cache.h"
 #include "common/logging.h"
 #include "definitions.h"
 #include "response.h"
+#include "translation_model.h"
 
 namespace marian {
 namespace bergamot {
 
+size_t hashForCache(const TranslationModel &model, const marian::Words &words) {
+  size_t seed = model.modelId();
+  for (auto &word : words) {
+    size_t hashWord = static_cast<size_t>(word.toWordIndex());
+    util::hash_combine<size_t>(seed, hashWord);
+  }
+  return seed;
+}
+
 // -----------------------------------------------------------------
-Request::Request(size_t Id, Segments &&segments, ResponseBuilder &&responseBuilder)
+Request::Request(size_t Id, const TranslationModel &model, Segments &&segments, ResponseBuilder &&responseBuilder,
+                 TranslationCache *cache)
     : Id_(Id),
+      model_(model),
       segments_(std::move(segments)),
-      responseBuilder_(std::move(responseBuilder))
-
-{
+      responseBuilder_(std::move(responseBuilder)),
+      cache_(cache) {
   counter_ = segments_.size();
   histories_.resize(segments_.size(), nullptr);
 
-  // If there are no segments_, we are never able to trigger the responseBuilder
-  // calls from a different thread. However, in this case we want an empty valid
-  // response.
+  // 1. If there are no segments_, we are never able to trigger the responseBuilder calls from a different thread. This
+  // happens when the use provides empty input, or the sentence and subword preprocessing deems no translatable units
+  // present. However, in this case we want an empty valid response. There's no need to do any additional processing
+  // here.
   if (segments_.size() == 0) {
     responseBuilder_(std::move(histories_));
+  } else {
+    counter_ = segments_.size();
+    histories_.resize(segments_.size());
+
+    if (cache_ != nullptr) {
+      // Iterate through segments, see if any can be prefilled from cache. If prefilled, mark the particular segments as
+      // complete (non-empty ProcessedRequestSentence). Also update accounting used elsewhere (counter_) to reflect one
+      // less segment to translate.
+      for (size_t idx = 0; idx < segments_.size(); idx++) {
+        size_t key = hashForCache(model_, getSegment(idx));
+        auto [found, history] = cache_->find(key);
+        if (found) {
+          histories_[idx] = history;
+          --counter_;
+        }
+      }
+      // 2. Also, if cache somehow manages to decrease all counter prefilling histories, then we'd have to trigger
+      // ResponseBuilder as well. No segments go into batching and therefore no processHistory triggers.
+      if (counter_.load() == 0) {
+        responseBuilder_(std::move(histories_));
+      }
+    }
   }
 }
 
@@ -37,7 +72,14 @@ Segment Request::getSegment(size_t index) const { return segments_[index]; }
 void Request::processHistory(size_t index, Ptr<History> history) {
   // Concurrently called by multiple workers as a history from translation is
   // ready. The container storing histories is set with the value obtained.
+
+  // Fill in placeholder from History obtained by freshly translating. Since this was a cache-miss to have got through,
+  // update cache if available to store the result.
   histories_[index] = history;
+  if (cache_ != nullptr) {
+    size_t key = hashForCache(model_, getSegment(index));
+    cache_->store(key, histories_[index]);
+  }
 
   // In case this is last request in, completeRequest is called, which sets the
   // value of the promise.
diff --git a/src/translator/request.h b/src/translator/request.h
index d2645f6d8..8415e3233 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -6,6 +6,7 @@
 #include <vector>
 
 #include "annotation.h"
+#include "cache.h"
 #include "common/logging.h"
 #include "data/types.h"
 #include "definitions.h"
@@ -16,6 +17,8 @@
 namespace marian {
 namespace bergamot {
 
+class TranslationModel;
+
 /// A Request is an internal representation used to represent a request after
 /// processed by TextProcessor into sentences constituted by marian::Words.
 ///
@@ -42,11 +45,16 @@ class Request {
   ///
   ///
   /// @param [in] Id: Identifier assigned to Request by Service.
+  /// @param [in] model: TranslationModel for identifying a unique translation unit key (model, words in a sentence) for
+  /// cache.
   /// @param [in] segments: Each segment is a unit to be translated.
   /// @param [in] responseBuilder: Callback function (of ResponseBuilder type)
   /// to be triggered upon the completion of translation of all units in a
   /// Request.
-  Request(size_t Id, Segments &&segments, ResponseBuilder &&responseBuilder);
+  /// @param [in] cache: Cache supplied externally to attempt to fetch translations or store them after completion for
+  /// reuse later.
+  Request(size_t Id, const TranslationModel &model, Segments &&segments, ResponseBuilder &&responseBuilder,
+          TranslationCache *cache);
 
   /// Obtain the count of tokens in the segment correponding to index. Used to
   /// insert sentence from multiple requests into the corresponding size bucket.
@@ -67,9 +75,14 @@ class Request {
   /// compiled from requests.
   void processHistory(size_t index, Ptr<History> history);
 
+  bool cacheHitPrefilled(size_t index) const { return histories_[index] != nullptr; }
+
  private:
   size_t Id_;
 
+  /// TranslationModel associated with this request
+  const TranslationModel &model_;
+
   /// Multiple translation-workers can concurrently access the same Request. The
   /// following atomic atomically operates on the variable holding sentences
   /// remaining to be translated.
@@ -86,6 +99,9 @@ class Request {
   /// Constructing Response requires the vocabs_ used to generate Request.
   /// std::vector<Ptr<Vocab const>> *vocabs_;
   ResponseBuilder responseBuilder_;
+
+  /// Cache used to hold unit translations. If nullptr, means no-caching.
+  TranslationCache *cache_;
 };
 
 /// A RequestSentence provides a view to a sentence within a Request. Existence
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 9de69ba8a..ca92721da 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -10,7 +10,8 @@
 namespace marian {
 namespace bergamot {
 
-BlockingService::BlockingService(const BlockingService::Config &config) : requestId_(0), batchingPool_() {}
+BlockingService::BlockingService(const BlockingService::Config &config)
+    : config_(config), requestId_(0), batchingPool_(), cache_(config.cacheSize, /*mutexBuckets=*/1) {}
 
 std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<TranslationModel> translationModel,
                                                          std::vector<std::string> &&sources,
@@ -20,8 +21,9 @@ std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<Transla
 
   for (size_t i = 0; i < sources.size(); i++) {
     auto callback = [i, &responses](Response &&response) { responses[i] = std::move(response); };  //
+    TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
     Ptr<Request> request =
-        translationModel->makeRequest(requestId_++, std::move(sources[i]), callback, responseOptions);
+        translationModel->makeRequest(requestId_++, std::move(sources[i]), callback, responseOptions, cache);
     batchingPool_.enqueueRequest(translationModel, request);
   }
 
@@ -34,7 +36,8 @@ std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<Transla
   return responses;
 }
 
-AsyncService::AsyncService(const AsyncService::Config &config) : requestId_(0), config_(config), safeBatchingPool_() {
+AsyncService::AsyncService(const AsyncService::Config &config)
+    : requestId_(0), config_(config), safeBatchingPool_(), cache_(config_.cacheSize, config_.cacheMutexBuckets) {
   ABORT_IF(config_.numWorkers == 0, "Number of workers should be at least 1 in a threaded workflow");
   workers_.reserve(config_.numWorkers);
   for (size_t cpuId = 0; cpuId < config_.numWorkers; cpuId++) {
@@ -61,7 +64,9 @@ AsyncService::~AsyncService() {
 void AsyncService::translate(std::shared_ptr<TranslationModel> translationModel, std::string &&source,
                              CallbackType callback, const ResponseOptions &responseOptions) {
   // Producer thread, a call to this function adds new work items. If batches are available, notifies workers waiting.
-  Ptr<Request> request = translationModel->makeRequest(requestId_++, std::move(source), callback, responseOptions);
+  TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
+  Ptr<Request> request =
+      translationModel->makeRequest(requestId_++, std::move(source), callback, responseOptions, cache);
   safeBatchingPool_.enqueueRequest(translationModel, request);
 }
 
diff --git a/src/translator/service.h b/src/translator/service.h
index d37f5c262..fae9dbffc 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -5,6 +5,7 @@
 #include <thread>
 #include <vector>
 
+#include "cache.h"
 #include "data/types.h"
 #include "quality_estimator.h"
 #include "response.h"
@@ -27,7 +28,11 @@ class AsyncService;
 /// bunch of texts and optional args to translate, wait till the translation finishes).
 class BlockingService {
  public:
-  struct Config {};
+  struct Config {
+    bool cacheEnabled{false};  ///< Whether to enable cache or not.
+    size_t cacheSize{2000};    ///< Size in History items to be stored in the cache. Loosely corresponds to sentences to
+                               /// cache in the real world.
+  };
   /// Construct a BlockingService with configuration loaded from an Options object. Does not require any keys, values to
   /// be set.
   BlockingService(const BlockingService::Config &config);
@@ -47,6 +52,8 @@ class BlockingService {
   std::vector<Response> translateMultiple(std::shared_ptr<TranslationModel> translationModel,
                                           std::vector<std::string> &&source, const ResponseOptions &responseOptions);
 
+  TranslationCache::Stats cacheStats() { return cache_.stats(); }
+
  private:
   ///  Numbering requests processed through this instance. Used to keep account of arrival times of the request. This
   ///  allows for using this quantity in priority based ordering.
@@ -57,6 +64,8 @@ class BlockingService {
   AggregateBatchingPool batchingPool_;
 
   Config config_;
+
+  TranslationCache cache_;
 };
 
 /// Effectively a threadpool, providing an API to take a translation request of a source-text, paramaterized by
@@ -65,7 +74,13 @@ class BlockingService {
 class AsyncService {
  public:
   struct Config {
-    size_t numWorkers;
+    size_t numWorkers;         ///< How many worker translation threads to spawn.
+    bool cacheEnabled{false};  ///< Whether to enable cache or not.
+    size_t cacheSize{2000};    ///< Size in History items to be stored in the cache. Loosely corresponds to sentences to
+                               /// cache in the real world.
+    size_t cacheMutexBuckets;  ///< Controls the granularity of locking to reduce contention by bucketing mutexes
+                               ///< guarding cache entry read write. Optimal at min(core, numWorkers) assuming a
+                               ///< reasonably large cache-size.
   };
   /// Construct an AsyncService with configuration loaded from Options. Expects positive integer value for
   /// `cpu-threads`. Additionally requires options which configure AggregateBatchingPool.
@@ -95,6 +110,8 @@ class AsyncService {
   /// Thread joins and proper shutdown are required to be handled explicitly.
   ~AsyncService();
 
+  TranslationCache::Stats cacheStats() { return cache_.stats(); }
+
  private:
   AsyncService::Config config_;
 
@@ -111,6 +128,8 @@ class AsyncService {
   /// requests compiled from  batching-pools of multiple translation models. The batching pool is wrapped around one
   /// object for thread-safety.
   ThreadsafeBatchingPool<AggregateBatchingPool> safeBatchingPool_;
+
+  TranslationCache cache_;
 };
 
 }  // namespace bergamot
diff --git a/src/translator/translation_model.cpp b/src/translator/translation_model.cpp
index 5a2739542..5cf2b85f4 100644
--- a/src/translator/translation_model.cpp
+++ b/src/translator/translation_model.cpp
@@ -2,6 +2,7 @@
 
 #include "batch.h"
 #include "byte_array_util.h"
+#include "cache.h"
 #include "common/logging.h"
 #include "data/corpus.h"
 #include "data/text_input.h"
@@ -11,9 +12,12 @@
 namespace marian {
 namespace bergamot {
 
+std::atomic<size_t> TranslationModel::modelCounter_ = 0;
+
 TranslationModel::TranslationModel(const Config &options, MemoryBundle &&memory /*=MemoryBundle{}*/,
                                    size_t replicas /*=1*/)
-    : options_(options),
+    : modelId_(modelCounter_++),
+      options_(options),
       memory_(std::move(memory)),
       vocabs_(options, std::move(memory_.vocabs)),
       textProcessor_(options, vocabs_, std::move(memory_.ssplitPrefixFile)),
@@ -86,14 +90,15 @@ void TranslationModel::loadBackend(size_t idx) {
 
 // Make request process is shared between Async and Blocking workflow of translating.
 Ptr<Request> TranslationModel::makeRequest(size_t requestId, std::string &&source, CallbackType callback,
-                                           const ResponseOptions &responseOptions) {
+                                           const ResponseOptions &responseOptions, TranslationCache *cache) {
   Segments segments;
   AnnotatedText annotatedSource;
 
   textProcessor_.process(std::move(source), annotatedSource, segments);
   ResponseBuilder responseBuilder(responseOptions, std::move(annotatedSource), vocabs_, callback, *qualityEstimator_);
 
-  Ptr<Request> request = New<Request>(requestId, std::move(segments), std::move(responseBuilder));
+  Ptr<Request> request =
+      New<Request>(requestId, /*model=*/*this, std::move(segments), std::move(responseBuilder), cache);
   return request;
 }
 
diff --git a/src/translator/translation_model.h b/src/translator/translation_model.h
index 599e6c707..6d2169494 100644
--- a/src/translator/translation_model.h
+++ b/src/translator/translation_model.h
@@ -6,6 +6,7 @@
 
 #include "batch.h"
 #include "batching_pool.h"
+#include "cache.h"
 #include "common/utils.h"
 #include "data/shortlist.h"
 #include "definitions.h"
@@ -66,11 +67,11 @@ class TranslationModel {
   /// @param [in] responseOptions: Configuration used to prepare the Response corresponding to the created request.
   //  @returns Request created from the query parameters wrapped within a shared-pointer.
   Ptr<Request> makeRequest(size_t requestId, std::string&& source, CallbackType callback,
-                           const ResponseOptions& responseOptions);
+                           const ResponseOptions& responseOptions, TranslationCache* cache);
 
   /// Relays a request to the batching-pool specific to this translation model.
   /// @param [in] request: Request constructed through makeRequest
-  void enqueueRequest(Ptr<Request> request) { batchingPool_.enqueueRequest(request); };
+  size_t enqueueRequest(Ptr<Request> request) { return batchingPool_.enqueueRequest(request); };
 
   /// Generates a batch from the batching-pool for this translation model, compiling from several active requests. Note
   /// that it is possible that calls to this method can give empty-batches.
@@ -86,7 +87,11 @@ class TranslationModel {
   /// @param [in] batch: A batch generated from generateBatch from the same TranslationModel instance.
   void translateBatch(size_t deviceId, Batch& batch);
 
+  /// Returns a unique-identifier for the model.
+  size_t modelId() const { return modelId_; }
+
  private:
+  size_t modelId_;
   Config options_;
   MemoryBundle memory_;
   Vocabs vocabs_;
@@ -114,6 +119,8 @@ class TranslationModel {
 
   void loadBackend(size_t idx);
   Ptr<marian::data::CorpusBatch> convertToMarianBatch(Batch& batch);
+
+  static std::atomic<size_t> modelCounter_;
 };
 
 }  // namespace bergamot

From 45412ce7de0ba000bae96ce376421f4ef3250c85 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Thu, 28 Oct 2021 09:30:02 +0100
Subject: [PATCH 296/442] Set PR to any branch to trigger workflows (#230)

---
 .github/workflows/coding-styles.yml             | 2 +-
 .github/workflows/doc.yml                       | 2 +-
 .github/workflows/wasm-custom_marian-mac.yml    | 2 +-
 .github/workflows/wasm-custom_marian-ubuntu.yml | 2 +-
 .github/workflows/windows.yml                   | 2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/coding-styles.yml b/.github/workflows/coding-styles.yml
index 330790e88..0bff2ec79 100644
--- a/.github/workflows/coding-styles.yml
+++ b/.github/workflows/coding-styles.yml
@@ -6,7 +6,7 @@ on:
   push:
     branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main, ci-sandbox ]
+    branches: [ '**' ]
 
 jobs:
   clang-format:
diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml
index 706465e39..3874822b8 100644
--- a/.github/workflows/doc.yml
+++ b/.github/workflows/doc.yml
@@ -5,7 +5,7 @@ on:
     branches: [ main, ci-sandbox ]
     tags: ['v[0-9]+.[0-9]+.[0-9]+']
   pull_request: 
-    branches: [ main ]
+    branches: [ '**' ]
 
 jobs:
   api-documentation:
diff --git a/.github/workflows/wasm-custom_marian-mac.yml b/.github/workflows/wasm-custom_marian-mac.yml
index 636323581..a27f6b8de 100644
--- a/.github/workflows/wasm-custom_marian-mac.yml
+++ b/.github/workflows/wasm-custom_marian-mac.yml
@@ -4,7 +4,7 @@ on:
   push:
     branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main, ci-sandbox ]
+    branches: [ '**' ]
 
 jobs:
   build-wasm:
diff --git a/.github/workflows/wasm-custom_marian-ubuntu.yml b/.github/workflows/wasm-custom_marian-ubuntu.yml
index b644d9763..80d083fb8 100644
--- a/.github/workflows/wasm-custom_marian-ubuntu.yml
+++ b/.github/workflows/wasm-custom_marian-ubuntu.yml
@@ -4,7 +4,7 @@ on:
   push:
     branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main, ci-sandbox ]
+    branches: [ '**' ]
 
 jobs:
   build-wasm:
diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 7d1aca9d5..0933835de 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -4,7 +4,7 @@ on:
   push:
     branches: [ main, ci-sandbox ]
   pull_request:
-    branches: [ main, ci-sandbox ]
+    branches: [ '**' ]
 
 env:
   MKL_URL: "https://romang.blob.core.windows.net/mariandev/ci/mkl-2020.1-windows-static.zip"

From 47e57c95a6eb4e8c3d1f6ef9cd0cbdccd04b84a6 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Fri, 29 Oct 2021 13:40:28 +0100
Subject: [PATCH 297/442] [ssplit-cpp] Enable position independent library when
 compiled from sources (#240)

---
 3rd_party/ssplit-cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index f0fe09765..72dbd9346 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit f0fe09765ce22c6db79b15123c6599b2b419d240
+Subproject commit 72dbd9346b9f0eede4444922c4e3fcfdc0d16abb

From 9b443997e2c36d34679975a3ebddb374c9740b68 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Sun, 31 Oct 2021 12:33:42 +0000
Subject: [PATCH 298/442] EXCLUDE_FROM_ALL for marian and ssplit-cpp 3rd-party
 libraries (#243)

---
 .github/workflows/windows.yml | 3 +--
 3rd_party/CMakeLists.txt      | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 0933835de..74d2439f5 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -62,6 +62,5 @@ jobs:
     - name: Print versions
       working-directory: build
       run: |
-        .\app\service-cli.exe --version
-        dir *.exe
+        .\app\bergamot.exe --version
       shell: cmd
diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index 70e50d663..b84a37b80 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -1,13 +1,13 @@
 # marian-dev is tested elsewhere in both paths, turning off here.
 set(COMPILE_TESTS OFF)
-add_subdirectory(marian-dev)
+add_subdirectory(marian-dev EXCLUDE_FROM_ALL)
 
 if(COMPILE_WASM)
   # This is a bad way of adding compilation flags. Will be improved soon.
   add_compile_options(${WASM_COMPILE_FLAGS})
 endif(COMPILE_WASM)
 
-add_subdirectory(ssplit-cpp)
+add_subdirectory(ssplit-cpp EXCLUDE_FROM_ALL)
 
 # Add include directories for 3rd party targets to be able to use it anywhere in the
 # project without explicitly specifying their include directories. Once they

From c5bc3f5191c7d733f9d836a3bf007d58e4b71d96 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 1 Nov 2021 13:06:23 +0100
Subject: [PATCH 299/442] Update config "skip-cost" to enable log probabilities
 for QE scores (#247)

- Updated wasm test page
---
 wasm/test_page/js/worker.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index f6dc83623..912cc87d9 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -176,7 +176,7 @@ max-length-break: 128
 mini-batch-words: 1024
 workspace: 128
 max-length-factor: 2.0
-skip-cost: true
+skip-cost: false
 cpu-threads: 0
 quiet: true
 quiet-translation: true

From 806169c822c7d240d88ba17dc2c236e560fdc4dc Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Mon, 1 Nov 2021 16:31:01 +0000
Subject: [PATCH 300/442] Recover logging (#226)

---
 3rd_party/marian-dev      |  2 +-
 bergamot-translator-tests |  2 +-
 src/translator/logging.h  | 35 +++++++++++++++++++++++++++++++++++
 src/translator/service.h  |  5 +++++
 4 files changed, 42 insertions(+), 2 deletions(-)
 create mode 100644 src/translator/logging.h

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index a1a82ff64..87643a4e3 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit a1a82ff64910dc066d64d631cd7a8212df9f88cd
+Subproject commit 87643a4e3b121c74d3b0a4f048e9f6836ad11078
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 6bd396922..9dc3c5e9a 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 6bd396922b2159b62c55530cb3ee6a40323d4171
+Subproject commit 9dc3c5e9a1027c1d6b4a467a27bdff16d0d6a006
diff --git a/src/translator/logging.h b/src/translator/logging.h
new file mode 100644
index 000000000..bd5b17a45
--- /dev/null
+++ b/src/translator/logging.h
@@ -0,0 +1,35 @@
+#include "3rd_party/marian-dev/src/3rd_party/spdlog/spdlog.h"
+#include "common/logging.h"
+
+namespace marian {
+namespace bergamot {
+
+// RAII Wrap around logging, to clean up after the object on stack.
+class Logger {
+ public:
+  Logger() : marianLoggers_(createLoggers()) {
+    // We are manually creating loggers, because this is usually created in marian as a side-effect of
+    // config-parsing.
+  }
+
+  ~Logger() {
+    // We need to manually destroy the loggers, as marian doesn't do
+    // that but will complain when a new marian::Config tries to
+    // initialise loggers with the same name.
+    for (auto &logger : marianLoggers_) {
+      if (logger) {
+        spdlog::drop(logger->name());
+      }
+    }
+  }
+
+  // Explicit destructor above is an indicator we should not allow this class to copy-construct.
+  Logger &operator=(const Logger &) = delete;
+  Logger(const Logger &) = delete;
+
+ private:
+  using MarianLogger = std::shared_ptr<spdlog::logger>;
+  std::vector<MarianLogger> marianLoggers_;
+};
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/service.h b/src/translator/service.h
index fae9dbffc..d58a759da 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -7,6 +7,7 @@
 
 #include "cache.h"
 #include "data/types.h"
+#include "logging.h"
 #include "quality_estimator.h"
 #include "response.h"
 #include "response_builder.h"
@@ -65,6 +66,8 @@ class BlockingService {
 
   Config config_;
 
+  // Logger which shuts down cleanly with service.
+  Logger logger_;
   TranslationCache cache_;
 };
 
@@ -129,6 +132,8 @@ class AsyncService {
   /// object for thread-safety.
   ThreadsafeBatchingPool<AggregateBatchingPool> safeBatchingPool_;
 
+  // Logger which shuts down cleanly with service.
+  Logger logger_;
   TranslationCache cache_;
 };
 

From 0bb8095bca166d765ab837d2a155e54048994006 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Mon, 1 Nov 2021 19:21:28 +0000
Subject: [PATCH 301/442] Deprecate hardAlignment in favour of softAlignment
 (#250)

---
 bergamot-translator-tests                   |  2 +-
 src/translator/response.h                   | 20 ++++----------------
 src/translator/response_builder.cpp         |  9 +--------
 src/translator/response_options.h           |  6 ------
 wasm/bindings/response_options_bindings.cpp |  3 +--
 wasm/test_page/js/worker.js                 |  2 +-
 6 files changed, 8 insertions(+), 34 deletions(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 9dc3c5e9a..9344b9835 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 9dc3c5e9a1027c1d6b4a467a27bdff16d0d6a006
+Subproject commit 9344b9835797f7c19ee49d30bff134b74a1a336e
diff --git a/src/translator/response.h b/src/translator/response.h
index b77fbb633..49ac80392 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -14,18 +14,6 @@
 namespace marian {
 namespace bergamot {
 
-/// Alignment is stored as a sparse matrix, this pretty much aligns with marian
-/// internals but is brought here to maintain translator
-/// agnosticism/independence.
-struct Point {
-  size_t src;  ///< Index pointing to source ByteRange
-  size_t tgt;  ///< Index pointing to target ByteRange
-  float prob;  ///< Score between [0, 1] on indicating degree of alignment.
-};
-
-/// Alignment is a sparse matrix, where Points represent entries with values.
-typedef std::vector<Point> Alignment;
-
 /// Response holds AnnotatedText(s) of source-text and translated text,
 /// alignment information between source and target sub-words and sentences.
 ///
@@ -65,10 +53,10 @@ struct Response {
   /// source or target.
   std::vector<SentenceQualityScore> qualityScores;
 
-  /// Alignments between source and target. Each Alignment is a
-  /// sparse matrix representation with indices corresponding
-  /// to (sub-)words accessible through Annotation.
-  std::vector<Alignment> alignments;
+  /// Alignments between source and target. This is a collection of dense matrices providing
+  ///    P[t][s] = p(source-token s  | target token t)
+  /// with an alignment matrix for each sentence.
+  std::vector<std::vector<std::vector<float>>> alignments;
 
   /// Returns the source sentence (in terms of byte range) corresponding to sentenceIdx.
   ///
diff --git a/src/translator/response_builder.cpp b/src/translator/response_builder.cpp
index d51fbbf57..f1bb773e0 100644
--- a/src/translator/response_builder.cpp
+++ b/src/translator/response_builder.cpp
@@ -22,14 +22,7 @@ void ResponseBuilder::buildAlignments(Histories &histories, Response &response)
     // mean WASM bindings for a structure deep within marian source.
     auto hyp = std::get<1>(result);
     auto softAlignment = hyp->tracebackAlignment();
-    auto threshold = responseOptions_.alignmentThreshold;
-    auto hardAlignment = data::ConvertSoftAlignToHardAlign(softAlignment, threshold);
-    Alignment unified_alignment;
-    for (auto &p : hardAlignment) {
-      unified_alignment.emplace_back(Point{p.srcPos, p.tgtPos, p.prob});
-    }
-
-    response.alignments.push_back(std::move(unified_alignment));
+    response.alignments.push_back(std::move(softAlignment));
   }
 }
 
diff --git a/src/translator/response_options.h b/src/translator/response_options.h
index 92737a414..43b1c433b 100644
--- a/src/translator/response_options.h
+++ b/src/translator/response_options.h
@@ -24,12 +24,6 @@ struct ResponseOptions {
   /// `alignment=true`.
   bool sentenceMappings{false};
 
-  /// Threshold between `[0.0f, 1.0f]` to filter alignments into a sparse
-  /// matrix. Higher value implies stronger filtering leading to provision of
-  /// higher-confidence matches. `1.0f` gives argmax (not the full-dense
-  /// matrix).
-  float alignmentThreshold{0.2f};
-
   ConcatStrategy concatStrategy{ConcatStrategy::FAITHFUL};
 };
 
diff --git a/wasm/bindings/response_options_bindings.cpp b/wasm/bindings/response_options_bindings.cpp
index 4addbcbfc..deafe1e0a 100644
--- a/wasm/bindings/response_options_bindings.cpp
+++ b/wasm/bindings/response_options_bindings.cpp
@@ -15,6 +15,5 @@ using namespace emscripten;
 EMSCRIPTEN_BINDINGS(response_options) {
   value_object<ResponseOptions>("ResponseOptions")
       .field("qualityScores", &ResponseOptions::qualityScores)
-      .field("alignment", &ResponseOptions::alignment)
-      .field("alignmentThreshold", &ResponseOptions::alignmentThreshold);
+      .field("alignment", &ResponseOptions::alignment);
 }
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index 912cc87d9..7fbaea8d2 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -323,7 +323,7 @@ const _parseTranslatedTextSentenceQualityScores = (vectorResponse) => {
 }
 
 const _prepareResponseOptions = () => {
-  return {qualityScores: true, alignment: false, alignmentThreshold: 0.2};
+  return {qualityScores: true, alignment: false};
 }
 
 const _prepareSourceText = (input) => {

From 7693a1d0076929a57ba11a809932548234c82595 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 3 Nov 2021 13:54:48 +0100
Subject: [PATCH 302/442] Updated marian submodule (#256)

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 87643a4e3..200e81c0c 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 87643a4e3b121c74d3b0a4f048e9f6836ad11078
+Subproject commit 200e81c0cc88259c540b96afc6e0867cb05570b0

From fa4efb483ba4f5f4e3ac98bc8c3f14b2e87541f1 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Fri, 5 Nov 2021 16:46:03 +0000
Subject: [PATCH 303/442] Update ssplit cpp, pcre2 source compile to fix broken
 builds (#258)

* Update ssplit cpp, pcre2 source compile to fix tests

* Syncing with browsermt/ssplit-cpp

* Removing accidental binary inclusion

* Removing brt accidental update by git add -u

* Fix windows workflow, vcpkg is broken use our cmake route

* [ssplit-cpp] Try searching different library names for Windows
---
 .github/workflows/windows.yml | 3 ++-
 3rd_party/ssplit-cpp          | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 74d2439f5..434c947cb 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -39,7 +39,7 @@ jobs:
     - name: Prepare vcpkg
       uses: lukka/run-vcpkg@v7.4
       with:
-        vcpkgArguments: protobuf pcre2
+        vcpkgArguments: protobuf 
         vcpkgGitCommitId: 8dddc6c899ce6fdbeab38b525a31e7f23cb2d5bb
         vcpkgDirectory: ${{ github.workspace }}/vcpkg/
         vcpkgTriplet: x64-windows-static
@@ -51,6 +51,7 @@ jobs:
         buildDirectory: ${{ github.workspace }}/build
         cmakeAppendedArgs: '-G Ninja
           -DCMAKE_BUILD_TYPE="Release"
+          -DSSPLIT_USE_INTERNAL_PCRE2="ON"
           -DUSE_WASM_COMPATIBLE_SOURCE="OFF"
           -DUSE_STATIC_LIBS="TRUE"'
         cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 72dbd9346..36beacd1e 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 72dbd9346b9f0eede4444922c4e3fcfdc0d16abb
+Subproject commit 36beacd1ee4d9d591346d8e0f7f7700c7a91eb9f

From 5a693b7eecda96100a9f9397a16d04737fe6d7f7 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Fri, 5 Nov 2021 20:48:28 +0000
Subject: [PATCH 304/442] Fixes windows workflow for PCRE2  (#260)

---
 .github/workflows/windows.yml                 |  3 +-
 3rd_party/ssplit-cpp                          |  2 +-
 .../ports/pcre2/pcre2-10.35_fix-uwp.patch     | 10 +++
 vcpkg-override/ports/pcre2/portfile.cmake     | 72 +++++++++++++++++++
 vcpkg-override/ports/pcre2/vcpkg.json         |  6 ++
 5 files changed, 90 insertions(+), 3 deletions(-)
 create mode 100644 vcpkg-override/ports/pcre2/pcre2-10.35_fix-uwp.patch
 create mode 100644 vcpkg-override/ports/pcre2/portfile.cmake
 create mode 100644 vcpkg-override/ports/pcre2/vcpkg.json

diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 434c947cb..66ac2c413 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -39,7 +39,7 @@ jobs:
     - name: Prepare vcpkg
       uses: lukka/run-vcpkg@v7.4
       with:
-        vcpkgArguments: protobuf 
+        vcpkgArguments: protobuf pcre2 --overlay-ports="${{ github.workspace }}\vcpkg-override\ports\pcre2"
         vcpkgGitCommitId: 8dddc6c899ce6fdbeab38b525a31e7f23cb2d5bb
         vcpkgDirectory: ${{ github.workspace }}/vcpkg/
         vcpkgTriplet: x64-windows-static
@@ -51,7 +51,6 @@ jobs:
         buildDirectory: ${{ github.workspace }}/build
         cmakeAppendedArgs: '-G Ninja
           -DCMAKE_BUILD_TYPE="Release"
-          -DSSPLIT_USE_INTERNAL_PCRE2="ON"
           -DUSE_WASM_COMPATIBLE_SOURCE="OFF"
           -DUSE_STATIC_LIBS="TRUE"'
         cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 36beacd1e..a08d6bce2 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 36beacd1ee4d9d591346d8e0f7f7700c7a91eb9f
+Subproject commit a08d6bce20619a8475736832d5418458c14db9d4
diff --git a/vcpkg-override/ports/pcre2/pcre2-10.35_fix-uwp.patch b/vcpkg-override/ports/pcre2/pcre2-10.35_fix-uwp.patch
new file mode 100644
index 000000000..476dde0f6
--- /dev/null
+++ b/vcpkg-override/ports/pcre2/pcre2-10.35_fix-uwp.patch
@@ -0,0 +1,10 @@
+--- a/CMakeLists.txt	2020-05-09 16:43:10.000000000 +0200
++++ b/CMakeLists.txt	2020-06-03 20:57:17.026182500 +0200
+@@ -619,6 +619,7 @@
+ 
+ IF(MSVC)
+   ADD_DEFINITIONS(-D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS)
++  add_compile_options(/wd4146)
+ ENDIF(MSVC)
+ 
+ SET(CMAKE_INCLUDE_CURRENT_DIR 1)
diff --git a/vcpkg-override/ports/pcre2/portfile.cmake b/vcpkg-override/ports/pcre2/portfile.cmake
new file mode 100644
index 000000000..641af1cd1
--- /dev/null
+++ b/vcpkg-override/ports/pcre2/portfile.cmake
@@ -0,0 +1,72 @@
+set(PCRE2_VERSION 10.37)
+set(EXPECTED_SHA f91760a8e0747f52211612fb0e134d685e224d16bd884eb574718d077a586b1fd7b6435d4e3b75c879b12e02b252467ecc28cdc4bc2903c783dacab089f99c99)
+set(PATCHES
+        pcre2-10.35_fix-uwp.patch
+)
+
+vcpkg_download_distfile(ARCHIVE
+    URLS "https://sourceforge.net/projects/pcre/files/pcre2/${PCRE2_VERSION}/pcre2-${PCRE2_VERSION}.zip"
+    FILENAME "pcre2-${PCRE2_VERSION}.zip"
+    SHA512 ${EXPECTED_SHA}
+    SILENT_EXIT
+)
+
+if (EXISTS "${ARCHIVE}")
+    vcpkg_extract_source_archive_ex(
+        OUT_SOURCE_PATH SOURCE_PATH
+        ARCHIVE ${ARCHIVE}
+        PATCHES ${PATCHES}
+    )
+else()
+    vcpkg_from_sourceforge(
+        OUT_SOURCE_PATH SOURCE_PATH
+        REPO pcre/pcre2
+        REF ${PCRE2_VERSION}
+        FILENAME "pcre2-${PCRE2_VERSION}.zip"
+        SHA512 ${EXPECTED_SHA}
+        PATCHES ${PATCHES}
+    )
+endif()
+
+if(VCPKG_CMAKE_SYSTEM_NAME STREQUAL "Emscripten" OR VCPKG_CMAKE_SYSTEM_NAME STREQUAL "iOS")
+    set(JIT OFF)
+else()
+    set(JIT ON)
+endif()
+
+vcpkg_configure_cmake(
+    SOURCE_PATH ${SOURCE_PATH}
+    PREFER_NINJA
+    OPTIONS
+        -DPCRE2_BUILD_PCRE2_8=ON
+        -DPCRE2_BUILD_PCRE2_16=ON
+        -DPCRE2_BUILD_PCRE2_32=ON
+        -DPCRE2_SUPPORT_JIT=${JIT}
+        -DPCRE2_SUPPORT_UNICODE=ON
+        -DPCRE2_BUILD_TESTS=OFF
+        -DPCRE2_BUILD_PCRE2GREP=OFF)
+
+vcpkg_install_cmake()
+
+file(READ ${CURRENT_PACKAGES_DIR}/include/pcre2.h PCRE2_H)
+if(VCPKG_LIBRARY_LINKAGE STREQUAL "static")
+    string(REPLACE "defined(PCRE2_STATIC)" "1" PCRE2_H "${PCRE2_H}")
+else()
+    string(REPLACE "defined(PCRE2_STATIC)" "0" PCRE2_H "${PCRE2_H}")
+endif()
+file(WRITE ${CURRENT_PACKAGES_DIR}/include/pcre2.h "${PCRE2_H}")
+
+vcpkg_fixup_pkgconfig()
+
+vcpkg_copy_pdbs()
+
+file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/man)
+file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/share/doc)
+file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/include)
+file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/man)
+file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/share)
+if(VCPKG_LIBRARY_LINKAGE STREQUAL "static")
+    file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/bin" "${CURRENT_PACKAGES_DIR}/debug/bin")
+endif()
+
+file(INSTALL ${SOURCE_PATH}/COPYING DESTINATION ${CURRENT_PACKAGES_DIR}/share/${PORT} RENAME copyright)
diff --git a/vcpkg-override/ports/pcre2/vcpkg.json b/vcpkg-override/ports/pcre2/vcpkg.json
new file mode 100644
index 000000000..80d87e8fe
--- /dev/null
+++ b/vcpkg-override/ports/pcre2/vcpkg.json
@@ -0,0 +1,6 @@
+{
+  "name": "pcre2",
+  "version-string": "10.37",
+  "description": "PCRE2 is a re-working of the original Perl Compatible Regular Expressions library",
+  "homepage": "https://pcre.org/"
+}

From d6a14b1d6ff65ddd52780dee2798c5477cec2a62 Mon Sep 17 00:00:00 2001
From: Andre Natal <andrenatal@users.noreply.github.com>
Date: Mon, 15 Nov 2021 00:14:21 -0800
Subject: [PATCH 305/442] Fix badge to point to this repo instead mozilla's
 (#261)

---
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 156d12875..11f144cce 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 # Bergamot Translator
 
-[![CircleCI badge](https://img.shields.io/circleci/project/github/mozilla/bergamot-translator/main.svg?label=CircleCI)](https://circleci.com/gh/mozilla/bergamot-translator/)
+[![CircleCI badge](https://img.shields.io/circleci/project/github/browsermt/bergamot-translator/main.svg?label=CircleCI)](https://circleci.com/gh/browsermt/bergamot-translator/)
 
 Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.github.io/) framework based) neural machine translation functionality in accordance with the [Bergamot](https://browser.mt/) project that focuses on improving client-side machine translation in a web browser.
 

From f9e55b3cd845478f8cc84b795f0a1e5720991100 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 15 Nov 2021 22:30:52 +0100
Subject: [PATCH 306/442] Make script run from any directory (#262)

* Make script run from any directory
---
 wasm/test_page/start_server.sh | 30 +++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)

diff --git a/wasm/test_page/start_server.sh b/wasm/test_page/start_server.sh
index 8cb90071c..59d455d14 100644
--- a/wasm/test_page/start_server.sh
+++ b/wasm/test_page/start_server.sh
@@ -1,11 +1,13 @@
 #!/bin/bash
 
-usage="Copy wasm artifacts from build directory and start httpserver
+usage="Copy wasm artifacts from the given folder and start httpserver
 
-Usage: $(basename "$0") [WASM_ARTIFACTS_FOLDER]
+Usage: $(basename "$0") [ARTIFACTS_SOURCE_FOLDER]
 
     where:
-    WASM_ARTIFACTS_FOLDER    Folder containing pre-built wasm artifacts"
+    ARTIFACTS_SOURCE_FOLDER    Directory containing pre-built wasm artifacts"
+
+SCRIPT_ABSOLUTE_PATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
 
 if [ "$#" -ne 1 ]; then
     echo "Illegal number of parameters passed"
@@ -13,19 +15,25 @@ if [ "$#" -ne 1 ]; then
     exit
 fi
 
-# Check if WASM_ARTIFACTS_FOLDER is valid or not
+# Check if ARTIFACTS_SOURCE_FOLDER is valid or not
 if [ ! -e "$1" ]; then
     echo "Error: Folder \""$1"\" doesn't exist"
     exit
 fi
 
-WASM_ARTIFACTS="$1/bergamot-translator-worker.js $1/bergamot-translator-worker.wasm"
-for i in $WASM_ARTIFACTS; do
+# Prepare a list all wasm artifacts to be copied and copy them to the destination folder
+ARTIFACTS_BASE_NAME="bergamot-translator-worker"
+ARTIFACTS="$1/$ARTIFACTS_BASE_NAME.js $1/$ARTIFACTS_BASE_NAME.wasm"
+ARTIFACTS_DESTINATION_FOLDER=$SCRIPT_ABSOLUTE_PATH/js
+
+for i in $ARTIFACTS; do
     [ -f "$i" ] || breaks
-    cp $i js/.
-    echo "Copied \"$i\""
+    cp $i $ARTIFACTS_DESTINATION_FOLDER
+    echo "Copied \"$i\" to \"$ARTIFACTS_DESTINATION_FOLDER\""
 done
 
-npm install
-echo "Start httpserver"
-node bergamot-httpserver.js 80 1 0
\ No newline at end of file
+# Start http server
+(cd $SCRIPT_ABSOLUTE_PATH;
+npm install;
+echo "Start httpserver";
+node bergamot-httpserver.js 80 1 0)

From 2b1b0531ff359c00685f8ef750f83edbcb7bd578 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 17 Nov 2021 09:18:55 +0100
Subject: [PATCH 307/442] Import optimized gemm implementation (when available)
 for wasm target (#265)

* Enable importing optimized gemm module for wasm

 - Updated emscripten generated JS code to
   -- import and use the optimized gemm module when available, otherwise
     use fallback gemm implementation

* Added logging for gemm implementation being used for wasm target
---
 wasm/import-gemm-module.js                 | 25 ++++++++++++++++
 wasm/patch-artifacts-import-gemm-module.sh | 33 +++++++++-------------
 2 files changed, 38 insertions(+), 20 deletions(-)
 create mode 100644 wasm/import-gemm-module.js

diff --git a/wasm/import-gemm-module.js b/wasm/import-gemm-module.js
new file mode 100644
index 000000000..e23a69d7f
--- /dev/null
+++ b/wasm/import-gemm-module.js
@@ -0,0 +1,25 @@
+
+/* Use an optimized gemm implementation if available, otherwise use the fallback
+ * implementation.
+ */
+function createWasmGemm() {
+    const OPTIMIZED_GEMM = "mozIntGemm";
+    const FALLBACK_GEMM =  "asm";
+
+    if (WebAssembly[OPTIMIZED_GEMM]) {
+        console.log(`Using optimized gemm (${OPTIMIZED_GEMM}) implementation`);
+        return new WebAssembly.Instance(WebAssembly[OPTIMIZED_GEMM](), {"": {memory: wasmMemory}}).exports;
+    }
+    else {
+        console.log(`Using fallback gemm implementation`);
+        return {
+            "int8_prepare_a": (...a) => Module[FALLBACK_GEMM]["int8PrepareAFallback"](...a),
+            "int8_prepare_b": (...a) => Module[FALLBACK_GEMM]["int8PrepareBFallback"](...a),
+            "int8_prepare_b_from_transposed": (...a) => Module[FALLBACK_GEMM]["int8PrepareBFromTransposedFallback"](...a),
+            "int8_prepare_b_from_quantized_transposed": (...a) => Module[FALLBACK_GEMM]["int8PrepareBFromQuantizedTransposedFallback"](...a),
+            "int8_prepare_bias": (...a) => Module[FALLBACK_GEMM]["int8PrepareBiasFallback"](...a),
+            "int8_multiply_and_add_bias": (...a) => Module[FALLBACK_GEMM]["int8MultiplyAndAddBiasFallback"](...a),
+            "int8_select_columns_of_b": (...a) => Module[FALLBACK_GEMM]["int8SelectColumnsOfBFallback"](...a)
+        }
+    }
+}
diff --git a/wasm/patch-artifacts-import-gemm-module.sh b/wasm/patch-artifacts-import-gemm-module.sh
index 2f2e29afd..d9fa648fe 100644
--- a/wasm/patch-artifacts-import-gemm-module.sh
+++ b/wasm/patch-artifacts-import-gemm-module.sh
@@ -1,10 +1,10 @@
 #!/bin/bash
-usage="Patch wasm artifacts to import fallback implementation of gemm for wasm.
+usage="Patch wasm artifacts to import gemm implementation for wasm.
 
-Usage: $(basename "$0") [WASM_ARTIFACTS_FOLDER]
+Usage: $(basename "$0") [ARTIFACTS_FOLDER]
 
     where:
-    WASM_ARTIFACTS_FOLDER    Folder containing wasm artifacts
+    ARTIFACTS_FOLDER    Folder containing wasm artifacts
                              (An optional argument, if unspecified the default is: current folder)"
 
 if [ "$#" -gt 1 ]; then
@@ -14,31 +14,24 @@ if [ "$#" -gt 1 ]; then
 fi
 
 # Parse wasm artifacts folder if provided via script argument or set it to default
-WASM_ARTIFACTS_FOLDER=$PWD
+ARTIFACTS_FOLDER=$PWD
 if [ "$#" -eq 1 ]; then
     if [ ! -e "$1" ]; then
         echo "Error: Folder \""$1"\" doesn't exist"
         exit
     fi
-    WASM_ARTIFACTS_FOLDER="$1"
+    ARTIFACTS_FOLDER="$1"
 fi
 
-WASM_ARTIFACTS_JAVASCRIPT_FILE="bergamot-translator-worker.js"
-WASM_ARTIFACTS="$WASM_ARTIFACTS_FOLDER/${WASM_ARTIFACTS_JAVASCRIPT_FILE}"
-if [ ! -e "$WASM_ARTIFACTS" ]; then
-    echo "Error: Artifact \"$WASM_ARTIFACTS\" doesn't exist"
+ARTIFACT="$ARTIFACTS_FOLDER/bergamot-translator-worker.js"
+if [ ! -e "$ARTIFACT" ]; then
+    echo "Error: Artifact \"$ARTIFACT\" doesn't exist"
     exit
 fi
 
-echo "Polyfill the fallback integer (8-bit) gemm implementation from the main module"
+echo "Importing integer (8-bit) gemm implementation"
+SCRIPT_ABSOLUTE_PATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
 sed -i.bak 's/"env"[[:space:]]*:[[:space:]]*asmLibraryArg,/"env": asmLibraryArg,\
-    "wasm_gemm":{\
-    "int8_prepare_a": (...a) => Module["asm"].int8PrepareAFallback(...a),\
-    "int8_prepare_b": (...a) => Module["asm"].int8PrepareBFallback(...a),\
-    "int8_prepare_b_from_transposed": (...a) => Module["asm"].int8PrepareBFromTransposedFallback(...a),\
-    "int8_prepare_b_from_quantized_transposed": (...a) => Module["asm"].int8PrepareBFromQuantizedTransposedFallback(...a),\
-    "int8_prepare_bias": (...a) => Module["asm"].int8PrepareBiasFallback(...a),\
-    "int8_multiply_and_add_bias": (...a) => Module["asm"].int8MultiplyAndAddBiasFallback(...a),\
-    "int8_select_columns_of_b": (...a) => Module["asm"].int8SelectColumnsOfBFallback(...a),\
-    },/g' ${WASM_ARTIFACTS_JAVASCRIPT_FILE}
-echo "SUCCESS"
\ No newline at end of file
+    "wasm_gemm": createWasmGemm(),/g' ${ARTIFACT}
+cat $SCRIPT_ABSOLUTE_PATH/import-gemm-module.js >> ${ARTIFACT}
+echo "SUCCESS"

From 40366162d82e8ddfbb9023da039670ad3d616ecb Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Thu, 25 Nov 2021 13:57:50 +0000
Subject: [PATCH 308/442] HTML input (#253)

Co-authored-by: Jelmer van der Linde <jelmer@ikhoefgeen.nl>
Co-authored-by: Abhishek Aggarwal <aaggarwal@mozilla.com>
---
 src/tests/units/CMakeLists.txt              |   3 +-
 src/tests/units/html_tests.cpp              | 519 +++++++++++++++++++
 src/tests/units/html_tests.h                |   9 +
 src/translator/CMakeLists.txt               |   2 +
 src/translator/definitions.h                |   1 +
 src/translator/html.cpp                     | 538 ++++++++++++++++++++
 src/translator/html.h                       |  50 ++
 src/translator/response_builder.h           |   9 +-
 src/translator/response_options.h           |   2 +
 src/translator/translation_model.cpp        |   5 +-
 src/translator/xh_scanner.cpp               | 454 +++++++++++++++++
 src/translator/xh_scanner.h                 | 130 +++++
 wasm/bindings/response_options_bindings.cpp |   3 +-
 wasm/test_page/js/worker.js                 |   2 +-
 14 files changed, 1721 insertions(+), 6 deletions(-)
 create mode 100644 src/tests/units/html_tests.cpp
 create mode 100644 src/tests/units/html_tests.h
 create mode 100644 src/translator/html.cpp
 create mode 100644 src/translator/html.h
 create mode 100644 src/translator/xh_scanner.cpp
 create mode 100644 src/translator/xh_scanner.h

diff --git a/src/tests/units/CMakeLists.txt b/src/tests/units/CMakeLists.txt
index 2570e05e7..8c29ab397 100644
--- a/src/tests/units/CMakeLists.txt
+++ b/src/tests/units/CMakeLists.txt
@@ -2,7 +2,8 @@
 set(UNIT_TESTS
     annotation_tests
     cache_tests
-    quality_estimator_tests)
+    quality_estimator_tests
+    html_tests)
 
 foreach(test ${UNIT_TESTS})
   add_executable("run_${test}" run_tests.cpp "${test}.cpp")
diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
new file mode 100644
index 000000000..258847970
--- /dev/null
+++ b/src/tests/units/html_tests.cpp
@@ -0,0 +1,519 @@
+#include "html_tests.h"
+
+#include <vector>
+
+#include "catch.hpp"
+#include "data/types.h"  // for marian::string_view
+#include "translator/html.h"
+#include "translator/response.h"
+
+using namespace marian::bergamot;
+using marian::string_view;
+
+std::ostream &operator<<(std::ostream &out, std::pair<ByteRange, ByteRange> const &b) {
+  return out << '(' << b.first << ',' << b.second << ')';
+}
+
+std::ostream &operator<<(std::ostream &out, ByteRange const &b) { return out << '{' << b.begin << ',' << b.end << '}'; }
+
+std::vector<ByteRange> AsByteRanges(AnnotatedText const &annotation) {
+  std::vector<ByteRange> words;
+  words.emplace_back(annotation.annotation.gap(0));
+  for (size_t sentenceIdx = 0; sentenceIdx < annotation.numSentences(); ++sentenceIdx) {
+    for (size_t wordIdx = 0; wordIdx < annotation.numWords(sentenceIdx); ++wordIdx)
+      words.emplace_back(annotation.wordAsByteRange(sentenceIdx, wordIdx));
+    words.emplace_back(annotation.annotation.gap(sentenceIdx + 1));
+  }
+  return words;
+}
+
+std::vector<std::string> AsTokens(AnnotatedText const &annotation) {
+  std::vector<std::string> words;
+  words.emplace_back(annotation.gap(0));
+  for (size_t sentenceIdx = 0; sentenceIdx < annotation.numSentences(); ++sentenceIdx) {
+    for (size_t wordIdx = 0; wordIdx < annotation.numWords(sentenceIdx); ++wordIdx)
+      words.emplace_back(annotation.word(sentenceIdx, wordIdx));
+    words.emplace_back(annotation.gap(sentenceIdx + 1));
+  }
+  return words;
+}
+
+void RecordSentenceFromByteRange(AnnotatedText &text, std::vector<ByteRange> const &ranges) {
+  assert(ranges.size() > 0);
+
+  std::vector<string_view> tokens;
+  tokens.reserve(ranges.size());
+
+  for (auto &&range : ranges) tokens.emplace_back(text.text.data() + range.begin, range.size());
+
+  text.recordExistingSentence(tokens.begin(), tokens.end(), text.text.data() + ranges[0].begin);
+}
+
+TEST_CASE("Ignore HTML if process_markup is false") {
+  std::string html_code("<p>This text &amp; has <b>HTML</b> in it</p>");
+
+  std::string input(html_code);
+  HTML html(std::move(input), false);
+  CHECK(input == html_code);
+
+  Response response;
+  response.source.text = html_code;
+  response.target.text = html_code;
+  html.Restore(response);
+
+  // Assert that Restore() does not mess with my HTML code
+  CHECK(response.source.text == html_code);
+}
+
+TEST_CASE("Test reconstruction") {
+  std::string input("<p><input>H<u>e</u>llo <b>world</b> how <u>are you</u>?</p>\n");
+
+  std::string text(input);
+  HTML html(std::move(text), true);  // TODO: move, but really a reference?
+  CHECK(text == "Hello world how are you?\n");
+
+  AnnotatedText source(std::move(text));
+  std::vector<string_view> tokens{
+      string_view(source.text.data() + 0, 4),   // Hell
+      string_view(source.text.data() + 4, 1),   // o
+      string_view(source.text.data() + 5, 6),   // _world
+      string_view(source.text.data() + 11, 4),  // _how
+      string_view(source.text.data() + 15, 4),  // _are
+      string_view(source.text.data() + 19, 4),  // _you
+      string_view(source.text.data() + 23, 1),  // ?
+      string_view(source.text.data() + 24, 0),  // "\n" (but 0 length?)
+  };
+
+  source.recordExistingSentence(tokens.begin(), tokens.end(), source.text.data());
+
+  Response response;
+  response.source = source;
+
+  html.Restore(response);
+  // CHECK(response.source.text == input);  // fails because <u></u> has been moved to the front of the token
+  CHECK(response.source.text == "<p><input><u></u>Hello <b>world</b> how <u>are you</u>?</p>\n");
+
+  std::vector<ByteRange> restored_tokens{
+      ByteRange{0, 0 + 0},    // (start of sentence)
+      ByteRange{0, 0 + 21},   // <p><input>H<u>e</u>ll
+      ByteRange{21, 21 + 1},  // o
+      ByteRange{22, 22 + 9},  // _<b>world
+      ByteRange{31, 31 + 8},  // </b>_how
+      ByteRange{39, 39 + 7},  // _<u>are
+      ByteRange{46, 46 + 4},  // _you
+      ByteRange{50, 50 + 5},  // </u>?
+      ByteRange{55, 55 + 0},  // ""
+      ByteRange{55, 55 + 5},  // </p>\n
+  };
+  CHECK(response.source.text.size() == restored_tokens.back().end);
+  CHECK(AsByteRanges(response.source) == restored_tokens);
+
+  // Same test as above, but easier to read. Will use this further down.
+  std::vector<std::string> restored_tokens_str{"",
+                                               "<p><input><u></u>Hell",  // Should really be "<p><input>H<u>e</u>ll"
+                                               "o",
+                                               " <b>world",
+                                               "</b> how",
+                                               " <u>are",
+                                               " you",
+                                               "</u>?",
+                                               "",  // end of sentence
+                                               "</p>\n"};
+
+  CHECK(AsTokens(response.source) == restored_tokens_str);
+}
+
+TEST_CASE("Test reconstruction of multiple sentences") {
+  std::string input("<p>This <em>is a sentence. And so is</em> this.</p>\n");
+
+  HTML html(std::move(input), true);
+  CHECK(input == "This is a sentence. And so is this.\n");
+
+  Response response;
+  response.source = AnnotatedText(std::move(input));
+
+  RecordSentenceFromByteRange(response.source, {
+                                                   ByteRange{0, 4},    // 0.0 "This"
+                                                   ByteRange{4, 7},    // 0.1 " is"
+                                                   ByteRange{7, 9},    // 0.2 " a"
+                                                   ByteRange{9, 18},   // 0.3 " sentence"
+                                                   ByteRange{18, 19},  // 0.4 "."
+                                               });
+
+  RecordSentenceFromByteRange(response.source, {
+                                                   ByteRange{20, 23},  // 1.0 "And"
+                                                   ByteRange{23, 26},  // 1.1 " so"
+                                                   ByteRange{26, 29},  // 1.2 " is"
+                                                   ByteRange{29, 34},  // 1.3 " this"
+                                                   ByteRange{34, 35},  // 1.4 "."
+                                               });
+
+  std::vector<std::string> tokens{"",    "This", " is", " a",    " sentence", ".", " ",
+                                  "And", " so",  " is", " this", ".",         "\n"};
+
+  CHECK(AsTokens(response.source) == tokens);
+
+  html.Restore(response);
+
+  std::vector<std::string> html_tokens{
+      "",       "<p>This", " <em>is", " a", " sentence", ".", " ", "And", " so", " is", "</em> this", ".",
+      "</p>\n",  // </p> got moved into post-sentence gap
+  };
+
+  CHECK(AsTokens(response.source) == html_tokens);
+}
+
+TEST_CASE("Test case html entities") {
+  // These are all entities I would expect in innerHTML, since all other entities
+  // can be encoded as UTF-8 so there's no need to encode them through &...; when
+  // innerHTML encodes the DOM as HTML.
+  std::string input("<p data-attr=\"&quot;&apos;\">This is a sentence &lt;with&gt; named &amp; entities</p>\n");
+  HTML html(std::move(input), true);
+  CHECK(input == "This is a sentence <with> named & entities\n");
+
+  Response response;
+  response.source = AnnotatedText(std::move(input));
+
+  RecordSentenceFromByteRange(response.source, {
+                                                   ByteRange{0, 4},    // 0.0 "This"
+                                                   ByteRange{4, 7},    // 0.1 " is"
+                                                   ByteRange{7, 9},    // 0.2 " a"
+                                                   ByteRange{9, 18},   // 0.3 " sentence"
+                                                   ByteRange{18, 20},  // 0.4 " <"
+                                                   ByteRange{20, 24},  // 0.5 "with"
+                                                   ByteRange{24, 25},  // 0.6 ">"
+                                                   ByteRange{25, 31},  // 0.7 " named"
+                                                   ByteRange{31, 33},  // 0.8 " &"
+                                                   ByteRange{33, 42},  // 0.9 " entities"
+                                                   ByteRange{42, 42}   // 0.10 ""
+                                               });
+
+  html.Restore(response);
+
+  std::vector<std::string> html_tokens{"",          "<p data-attr=\"&quot;&apos;\">This",
+                                       " is",       " a",
+                                       " sentence",
+                                       " &lt;",  // Oh trouble! The < is completely 'consumed'
+                                       "with",      "&gt;",
+                                       " named",    " &amp;",
+                                       " entities", "",
+                                       "</p>\n"};
+
+  CHECK(AsTokens(response.source) == html_tokens);
+}
+
+TEST_CASE("Test self-closing tags should be treated as spaces") {
+  std::string input("<p>Space<br>please?</p>\n");
+
+  HTML html(std::move(input), true);
+  CHECK(input == "Space please?\n");
+}
+
+TEST_CASE("Test reconstruction of target sentence") {
+  std::string input("<p>hello <b>world</b></p>\n");
+  HTML html(std::move(input), true);
+  CHECK(input == "hello world\n");
+
+  AnnotatedText source("hello world\n");
+  RecordSentenceFromByteRange(source, {
+                                          ByteRange{0, 4},   // 0.0 "hell"
+                                          ByteRange{4, 5},   // 0.1 "o"
+                                          ByteRange{5, 11},  // 0.2 " world"
+                                          ByteRange{11, 11}  // 0.3 ""
+                                      });
+
+  AnnotatedText target("hallo Welt\n");
+  RecordSentenceFromByteRange(target, {
+                                          ByteRange{0, 4},   // 0.0 "hall"
+                                          ByteRange{4, 5},   // 0.1 "o"
+                                          ByteRange{5, 10},  // 0.2 " Welt"
+                                          ByteRange{10, 10}  // 0.3 ""
+                                      });
+
+  Response response;
+  response.source = source;
+  response.target = target;
+
+  html.Restore(response);
+
+  std::vector<std::string> html_tokens_source{"", "<p>hell", "o", " <b>world", "", "</b></p>\n"};
+
+  std::vector<std::string> html_tokens_target{"", "<p>hall", "o", " <b>Welt", "", "</b></p>\n"};
+
+  CHECK(AsTokens(response.source) == html_tokens_source);
+  CHECK(AsTokens(response.target) == html_tokens_target);
+}
+
+TEST_CASE("Test reconstruction of target sentence with entities") {
+  std::string input("<p>hello <b>world &amp; friends!</b></p>\n");
+  HTML html(std::move(input), true);
+  CHECK(input == "hello world & friends!\n");
+
+  AnnotatedText source("hello world & friends!\n");
+  RecordSentenceFromByteRange(source, {
+                                          ByteRange{0, 4},    // 0.0 "hell"
+                                          ByteRange{4, 5},    // 0.1 "o"
+                                          ByteRange{5, 11},   // 0.2 " world"
+                                          ByteRange{11, 13},  // 0.3 " &"
+                                          ByteRange{13, 21},  // 0.4 " friends"
+                                          ByteRange{21, 22},  // 0.5 "!"
+                                          ByteRange{22, 22}   // 0.6 ""
+                                      });
+
+  AnnotatedText target("hallo Welt & Freunde!\n");
+  RecordSentenceFromByteRange(target, {
+                                          ByteRange{0, 4},    // 0.0 "hall"
+                                          ByteRange{4, 5},    // 0.1 "o"
+                                          ByteRange{5, 10},   // 0.2 " Welt"
+                                          ByteRange{10, 12},  // 0.3 " &"
+                                          ByteRange{12, 20},  // 0.4 " Freunde"
+                                          ByteRange{20, 21},  // 0.5 "!"
+                                          ByteRange{21, 21}   // 0.6 ""
+                                      });
+
+  Response response;
+  response.source = source;
+  response.target = target;
+
+  html.Restore(response);
+
+  std::vector<std::string> html_tokens_source{"",         "<p>hell", "o", " <b>world", " &amp;",
+                                              " friends", "!",       "",  "</b></p>\n"};
+
+  std::vector<std::string> html_tokens_target{"",         "<p>hall", "o", " <b>Welt",  " &amp;",
+
+                                              " Freunde", "!",       "",  "</b></p>\n"};
+
+  CHECK(AsTokens(response.source) == html_tokens_source);
+  CHECK(AsTokens(response.target) == html_tokens_target);
+}
+
+TEST_CASE("Test reconstruction of target with multiple sentences") {
+  std::string input(
+      "<p>hello <b>world!</b> How does this <img> <b>deal <u>with multiple sentences?</u></b> Will it work?</p>\n");
+  HTML html(std::move(input), true);
+
+  AnnotatedText source("hello world! How does this  deal with multiple sentences? Will it work?\n");
+  CHECK(source.text == input);
+
+  RecordSentenceFromByteRange(source, {
+                                          ByteRange{0, 4},    // 0.0 "hell"
+                                          ByteRange{4, 5},    // 0.1 "o"
+                                          ByteRange{5, 11},   // 0.2 " world"
+                                          ByteRange{11, 12},  // 0.3 "!"
+                                          ByteRange{12, 12}   // 0.4 ""
+                                      });
+  RecordSentenceFromByteRange(source, {
+                                          ByteRange{13, 16},  // 1.0 "How"
+                                          ByteRange{16, 21},  // 1.1 " does"
+                                          ByteRange{21, 26},  // 1.2 " this"
+                                          ByteRange{26, 32},  // 1.3 "  deal"
+                                          ByteRange{32, 37},  // 1.4 " with"
+                                          ByteRange{37, 46},  // 1.5 " multiple"
+                                          ByteRange{46, 55},  // 1.6 " sentence"
+                                          ByteRange{55, 56},  // 1.7 "s"
+                                          ByteRange{56, 57},  // 1.8 "?"
+                                          ByteRange{57, 57}   // 1.9 ""
+                                      });
+  RecordSentenceFromByteRange(source, {
+                                          ByteRange{58, 62},  // 2.0 "Will"
+                                          ByteRange{62, 65},  // 2.1 " it"
+                                          ByteRange{65, 70},  // 2.2 " work"
+                                          ByteRange{70, 71},  // 2.3 "?"
+                                          ByteRange{71, 71}   // 2.4 ""
+                                      });
+
+  AnnotatedText target("hallo Welt! Wie geht das mit mehreren Sätzen um? Wird es funktionieren?\n");
+  RecordSentenceFromByteRange(target, {
+                                          ByteRange{0, 4},    // 0.0 "hall"
+                                          ByteRange{4, 5},    // 0.1 "o"
+                                          ByteRange{5, 10},   // 0.2 " Welt"
+                                          ByteRange{10, 11},  // 0.3 "!"
+                                          ByteRange{11, 11},  // 0.4 ""
+                                      });
+  RecordSentenceFromByteRange(target, {
+                                          ByteRange{12, 15},  // 1.0 "Wie"
+                                          ByteRange{15, 20},  // 1.1 " geht"
+                                          ByteRange{20, 24},  // 1.2 " das"
+                                          ByteRange{24, 28},  // 1.3 " mit"
+                                          ByteRange{28, 37},  // 1.4 " mehreren"
+                                          ByteRange{37, 44},  // 1.5 " Sätze"
+                                          ByteRange{44, 45},  // 1.6 "n"
+                                          ByteRange{45, 48},  // 1.7 " um"
+                                          ByteRange{48, 49},  // 1.8 "?"
+                                          ByteRange{49, 49},  // 1.9 ""
+                                      });
+  RecordSentenceFromByteRange(target, {
+                                          ByteRange{50, 54},  // 2.0 "Wird"
+                                          ByteRange{54, 57},  // 2.1 " es"
+                                          ByteRange{57, 71},  // 2.2 " funktionieren"
+                                          ByteRange{71, 72},  // 2.3 "?"
+                                          ByteRange{72, 72},  // 2.4 ""
+                                      });
+
+  std::vector<std::string> text_tokens_source{
+      "",       "hall", "o",   " Welt", "!", "",  " ",    "Wie", " geht",          " das", " mit", " mehreren",
+      " Sätze", "n",    " um", "?",     "",  " ", "Wird", " es", " funktionieren", "?",    "",     "\n"};
+
+  CHECK(AsTokens(target) == text_tokens_source);
+
+  Response response;
+  response.source = source;
+  response.target = target;
+  html.Restore(response);
+
+  std::vector<std::string> html_tokens_source{"",
+                                              "<p>hell",
+                                              "o",
+                                              " <b>world",
+                                              "!",
+                                              "",
+                                              "</b> ",
+                                              "How",
+                                              " does",
+                                              " this",
+                                              "  <img><b>deal",  // note how both spaces moved to __deal
+                                              " <u>with",
+                                              " multiple",
+                                              " sentence",
+                                              "s",
+                                              "?",
+                                              "",
+                                              "</u></b> ",
+                                              "Will",
+                                              " it",
+                                              " work",
+                                              "?",
+                                              "",
+                                              "</p>\n"};
+  CHECK(AsTokens(response.source) == html_tokens_source);
+}
+
+TEST_CASE("Test self-closing tag (HTML5)") {
+  std::string input("<p>hello <img> <b>world</b> <u>and other <a href=\"#\">creatures</a></u></p>\n");
+  HTML html(std::move(input), true);
+  CHECK(input == "hello  world and other creatures\n");  // Note double space between "hello" and "world"
+}
+
+TEST_CASE("Test empty tag", "[!mayfail]") {
+  std::string input(
+      "<p id=\"1\">hello <img id=\"1.1\"><span id=\"1.2\"><u id=\"1.2.1\"></u><b id=\"1.2.2\"></b><img "
+      "id=\"1.2.3\">world</span></p>\n");
+  HTML html(std::move(input), true);
+  CHECK(input == "hello world\n");
+
+  Response response;
+
+  std::string sentence_str("hello world");
+  std::vector<string_view> sentence{
+      string_view(sentence_str.data() + 0, 4),   // 0.0 hell
+      string_view(sentence_str.data() + 4, 1),   // 0.1 o
+      string_view(sentence_str.data() + 5, 6),   // 0.2 _world
+      string_view(sentence_str.data() + 11, 0),  // 0.3 ""
+  };
+  response.source.appendSentence("", sentence.begin(), sentence.end());
+  response.source.appendEndingWhitespace("\n");
+
+  html.Restore(response);
+  CHECK(response.source.text ==
+        "<p id=\"1\">hello <img id=\"1.1\"><span id=\"1.2\"><u id=\"1.2.1\"></u><b id=\"1.2.2\"></b><img "
+        "id=\"1.2.3\">world</span></p>\n");
+}
+
+TEST_CASE("End-to-end translation") {
+  std::string input("<p>I <b>like</b> to <u>drive</u> this car.</p>\n");
+  HTML html(std::move(input), true);
+  CHECK(input == "I like to drive this car.\n");
+
+  Response response;
+
+  // clang-format off
+  response.alignments = std::vector<std::vector<std::vector<float>>>{{
+    {0.982376,  0.00742467, 0.00682965, 0.00121767, 0.000848056,6.51436e-05,7.53791e-06,0.00123162},
+    {0.165639,  0.368694,   0.230394,   0.222476,   0.00349563, 0.00105052, 0.000603092,0.00764845},
+    {0.00493271,0.0805876,  0.0139988,  0.89116,    0.000928116,0.00200724, 0.000512013,0.00587302},
+    {0.0194648, 0.411029,   0.087059,   0.0477847,  0.26596,    0.111161,   0.000392092,0.0571499},
+    {0.00879706,0.492504,   0.0448291,  0.007779,   0.423114,   0.0125523,  0.00119587, 0.00922804},
+    {0.00181909,0.00603626, 0.0335758,  0.037193,   0.747266,   0.102497,   0.0585782,  0.0130341},
+    {4.1348e-06,0.000156165,2.16369e-05,0.00275059, 0.00183456, 0.992357,   0.0023765,  0.000499018},
+    {0.00149043,0.000719392,0.0168534,  0.00430164, 0.00200343, 0.0106381,  0.948566,   0.0154279},
+    {0.0903136, 0.0550843,  0.0699474,  0.0792285,  0.223006,   0.207565,   0.129241,   0.145614},
+  }};
+  // clang-format on
+
+  {
+    std::string sentence_str("I like to drive this car.");
+    std::vector<string_view> sentence{
+        string_view(sentence_str.data() + 0, 1),   // 0.0 "I"
+        string_view(sentence_str.data() + 1, 5),   // 0.1 " like"
+        string_view(sentence_str.data() + 6, 3),   // 0.2 " to"
+        string_view(sentence_str.data() + 9, 6),   // 0.3 " drive"
+        string_view(sentence_str.data() + 15, 5),  // 0.4 " this"
+        string_view(sentence_str.data() + 20, 4),  // 0.5 " car"
+        string_view(sentence_str.data() + 24, 1),  // 0.6 "."
+        string_view(sentence_str.data() + 25, 0),  // 0.7 ""
+    };
+    response.source.appendSentence("", sentence.begin(), sentence.end());
+    response.source.appendEndingWhitespace("\n");
+  }
+
+  {
+    std::string sentence_str("Ich fahre gerne dieses Auto.");
+    std::vector<string_view> sentence{
+        string_view(sentence_str.data() + 0, 3),   // 0.0 "Ich"
+        string_view(sentence_str.data() + 3, 1),   // 0.1 " "
+        string_view(sentence_str.data() + 4, 4),   // 0.2 "fahr"
+        string_view(sentence_str.data() + 8, 1),   // 0.3 "e"
+        string_view(sentence_str.data() + 9, 6),   // 0.4 " gerne"
+        string_view(sentence_str.data() + 15, 7),  // 0.5 " dieses"
+        string_view(sentence_str.data() + 22, 5),  // 0.6 " Auto"
+        string_view(sentence_str.data() + 27, 1),  // 0.7 "."
+        string_view(sentence_str.data() + 28, 0),  // 0.8 ""
+    };
+    response.target.appendSentence("", sentence.begin(), sentence.end());
+    response.target.appendEndingWhitespace("\n");
+  }
+
+  html.Restore(response);
+
+  {
+    AnnotatedText source;
+    std::string sentence_str("<p>I <b>like</b> to <u>drive</u> this car.");
+    std::vector<string_view> sentence{
+        string_view(sentence_str.data() + 0, 4),   // 0.0 "<p>I"
+        string_view(sentence_str.data() + 4, 8),   // 0.1 " <b>like"
+        string_view(sentence_str.data() + 12, 7),  // 0.2 "</b> to"
+        string_view(sentence_str.data() + 19, 9),  // 0.3 " <u>drive"
+        string_view(sentence_str.data() + 28, 9),  // 0.4 "</u> this"
+        string_view(sentence_str.data() + 37, 4),  // 0.5 " car"
+        string_view(sentence_str.data() + 41, 1),  // 0.6 "."
+        string_view(sentence_str.data() + 42, 0),  // 0.7 ""
+    };
+    source.appendSentence("", sentence.begin(), sentence.end());
+    source.appendEndingWhitespace("</p>\n");
+
+    CHECK(AsTokens(response.source) == AsTokens(source));
+  }
+
+  {
+    AnnotatedText target;
+    std::string sentence_str("<p>Ich <u>fahre</u> <b>gerne</b> dieses Auto.");
+    std::vector<string_view> sentence{
+        string_view(sentence_str.data() + 0, 6),    // 0.0 "<p>Ich"
+        string_view(sentence_str.data() + 6, 4),    // 0.1 " <u>"
+        string_view(sentence_str.data() + 10, 4),   // 0.2 "fahr"
+        string_view(sentence_str.data() + 14, 1),   // 0.3 "e"
+        string_view(sentence_str.data() + 15, 13),  // 0.4 "</u> <b>gerne"
+        string_view(sentence_str.data() + 28, 11),  // 0.5 "</b> dieses"
+        string_view(sentence_str.data() + 39, 5),   // 0.6 " Auto"
+        string_view(sentence_str.data() + 44, 1),   // 0.7 "."
+        string_view(sentence_str.data() + 45, 0),   // 0.8 ""
+    };
+    target.appendSentence("", sentence.begin(), sentence.end());
+    target.appendEndingWhitespace("</p>\n");
+
+    CHECK(AsTokens(response.target) == AsTokens(target));
+  }
+}
+
+// TEST_CASE("")
\ No newline at end of file
diff --git a/src/tests/units/html_tests.h b/src/tests/units/html_tests.h
new file mode 100644
index 000000000..0407b65b2
--- /dev/null
+++ b/src/tests/units/html_tests.h
@@ -0,0 +1,9 @@
+#pragma once
+#include <ostream>
+
+#include "translator/definitions.h"
+
+std::ostream &operator<<(std::ostream &out, marian::bergamot::ByteRange const &b);
+
+std::ostream &operator<<(std::ostream &out,
+                         std::pair<marian::bergamot::ByteRange, marian::bergamot::ByteRange> const &b);
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index ab1448800..6779b0fa4 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -15,6 +15,8 @@ add_library(bergamot-translator STATIC
     annotation.cpp
     service.cpp
     parser.cpp
+    html.cpp
+    xh_scanner.cpp
 )
 if (USE_WASM_COMPATIBLE_SOURCE)
   # Using wasm compatible sources should include this compile definition;
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index 66ebb03b4..2ac6bf0ef 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -39,6 +39,7 @@ struct ByteRange {
   size_t begin;
   size_t end;
   const size_t size() const { return end - begin; }
+  bool operator==(ByteRange other) const { return begin == other.begin && end == other.end; }
 };
 
 class Response;
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
new file mode 100644
index 000000000..0614c37e5
--- /dev/null
+++ b/src/translator/html.cpp
@@ -0,0 +1,538 @@
+#include "html.h"
+
+#include "response.h"
+#include "xh_scanner.h"
+
+namespace {
+using marian::string_view;
+using marian::bergamot::AnnotatedText;
+using marian::bergamot::ByteRange;
+using marian::bergamot::HTML;
+using marian::bergamot::Response;
+
+void EncodeEntities(string_view const &input, std::string &output) {
+  output.clear();
+  output.reserve(input.size());
+
+  for (auto it = input.begin(); it != input.end(); ++it) {
+    switch (*it) {
+      case '&':
+        output.append("&amp;");
+        break;
+      case '<':
+        output.append("&lt;");
+        break;
+      case '>':
+        output.append("&gt;");
+        break;
+      // case ???:
+      //   output.append("&nbsp;");
+      //   break;
+      // case '"':
+      //   output.append("&quot;");
+      //   break;
+      // case '\'':
+      //   output.append("&apos;");
+      //   break;
+      default:
+        output.push_back(*it);
+        break;
+    }
+  }
+}
+
+size_t CountPrefixWhitespaces(string_view const &input) {
+  size_t size = 0;
+  while (size < input.size() && input[size] == ' ') ++size;
+  return size;
+}
+
+std::ostream &operator<<(std::ostream &out, HTML::Tag const *tag) {
+  if (tag == nullptr) return out << "[nullptr]";
+  out << '<' << tag->name << tag->attributes;
+  if (tag->empty) out << '/';
+  return out << '>';
+}
+
+std::ostream &operator<<(std::ostream &out, HTML::Taint const &tags) {
+  for (auto it = tags.begin(); it != tags.end(); ++it) {
+    if (it != tags.begin()) out << ' ';
+    out << *it;
+  }
+  return out;
+}
+
+// Very simple replacement for std::format introduced in C++20
+std::string format(std::string const &format_str) { return format_str; }
+
+template <typename Arg>
+std::string format(std::string const &format_str, Arg arg) {
+  std::ostringstream os;
+  auto index = format_str.find("{}");
+  assert(index != std::string::npos);
+  os << format_str.substr(0, index) << arg << format_str.substr(index + 2);
+  return os.str();
+}
+
+template <typename Arg, typename... Args>
+std::string format(std::string const &format_str, Arg arg, Args... args) {
+  std::ostringstream os;
+  auto index = format_str.find("{}");
+  assert(index != std::string::npos);
+  os << format_str.substr(0, index) << arg << format(format_str.substr(index + 2), std::forward<Args>(args)...);
+  return os.str();
+}
+
+bool IsBlockElement(std::string const &name) {
+  // List of elements that we expect might occur inside words, and that should
+  // not introduce spacings around them. Not strictly inline elements, nor flow
+  // elements. See also https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Content_categories
+  static std::unordered_set<std::string> inline_ish_elements{
+      "abbr",  "a",    "b",      "em",  "i",   "kbd",  "mark", "math", "output", "q",   "ruby",
+      "small", "span", "strong", "sub", "sup", "time", "u",    "var",  "wbr",    "ins", "del"};
+
+  return inline_ish_elements.find(name) == inline_ish_elements.end();
+}
+
+bool IsEmtpyElement(std::string const &name) {
+  // List of elements for which we do not expect a closing tag, or self-closing
+  // elements in XHTML. See also https://developer.mozilla.org/en-US/docs/Glossary/Empty_element
+  static std::unordered_set<std::string> empty_elements{"area",  "base", "br",   "col",   "embed",  "hr",    "img",
+                                                        "input", "link", "meta", "param", "source", "track", "wbr"};
+
+  return empty_elements.find(name) != empty_elements.end();
+}
+
+void DiffTags(HTML::Taint const &prev, HTML::Taint const &curr, HTML::Taint &opening, HTML::Taint &closing) {
+  opening.clear();
+  closing.clear();
+
+  size_t i = 0;
+
+  // Find first difference
+  for (; i < prev.size(); ++i)
+    if (i >= curr.size() || prev[i] != curr[i]) break;
+
+  std::copy_if(prev.begin() + i, prev.end(), std::back_inserter(closing), [&](HTML::Tag *tag) { return !tag->empty; });
+
+  opening.insert(opening.end(), curr.begin() + i, curr.end());
+}
+
+bool Intersects(ByteRange const &range, HTML::Span const &span) {
+  return range.begin <= span.end && range.end >= span.begin;
+};
+
+void FilterEmpty(HTML::Taint &stack) {
+  auto src = stack.begin();
+  auto dst = stack.begin();
+
+  for (auto src = stack.begin(); src != stack.end(); ++src)
+    if (!(*src)->empty) *(dst++) = *src;
+
+  stack.resize(dst - stack.begin());
+}
+
+template <typename Fun>
+AnnotatedText Apply(AnnotatedText const &in, Fun fun) {
+  AnnotatedText out;
+
+  for (size_t sentenceIdx = 0; sentenceIdx < in.numSentences(); ++sentenceIdx) {
+    std::string sentence;
+    std::vector<ByteRange> tokens;
+
+    std::string prefix = fun(in.annotation.gap(sentenceIdx), in.gap(sentenceIdx), false);
+
+    for (size_t wordIdx = 0; wordIdx < in.numWords(sentenceIdx); ++wordIdx) {
+      std::string token = fun(in.wordAsByteRange(sentenceIdx, wordIdx), in.word(sentenceIdx, wordIdx), false);
+      tokens.push_back(ByteRange{sentence.size(), sentence.size() + token.size()});
+      sentence += token;
+    }
+
+    // Convert our ByteRanges to string_views since that's what appendSentence
+    // expects
+    // TODO: extend AnnotatedText::appendSentence to accept str + ByteRanges
+    // directly
+    std::vector<string_view> token_views(tokens.size());
+    std::transform(tokens.begin(), tokens.end(), token_views.begin(),
+                   [&](ByteRange const &range) { return string_view(sentence.data() + range.begin, range.size()); });
+
+    out.appendSentence(prefix, token_views.begin(), token_views.end());
+  }
+
+  out.appendEndingWhitespace(fun(in.annotation.gap(in.numSentences()), in.gap(in.numSentences()), true));
+
+  return out;
+}
+
+bool IsContinuation(string_view str) { return !str.empty() && str.compare(0, 1, " ", 1) != 0; }
+
+void HardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
+  // For each sentence...
+  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
+    alignments.emplace_back();
+    assert(response.alignments[sentenceIdx].size() == response.target.numWords(sentenceIdx));
+
+    // Hard-align: find for each target token the most prevalent source token
+    for (size_t t = 0; t < response.alignments[sentenceIdx].size(); ++t) {
+      size_t s_max = 0;
+      for (size_t s = 1; s < response.alignments[sentenceIdx][t].size(); ++s) {
+        if (response.alignments[sentenceIdx][t][s] > response.alignments[sentenceIdx][t][s_max]) {
+          s_max = s;
+        }
+      }
+
+      alignments.back().push_back(s_max);
+    }
+
+    // Next, we try to smooth out these selected alignments with a few heuristics
+    for (size_t t = 0; t < response.target.numWords(sentenceIdx); ++t) {
+      // If this token is a continuation of a previous token, pick the tags from the most
+      // prevalent token for the whole word.
+      if (t > 0 && IsContinuation(response.target.word(sentenceIdx, t))) {
+        // Note: only looking at the previous token since that will already
+        // have this treatment applied to it.
+        size_t s_curr = alignments.back()[t];
+        size_t s_prev = alignments.back()[t - 1];
+        float score_curr = response.alignments[sentenceIdx][t][s_curr];
+        float score_prev = response.alignments[sentenceIdx][t - 1][s_prev];
+
+        size_t s_max = score_curr > score_prev ? s_curr : s_prev;
+
+        // Apply this to all previous tokens in the word
+        for (size_t i = t; i >= 0; --i) {
+          alignments.back()[i] = s_max;
+
+          // Stop if this was the beginning of the word
+          if (!IsContinuation(response.target.word(sentenceIdx, i))) break;
+        }
+      }
+    }
+  }
+}
+
+void InterpolateAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
+  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
+    alignments.emplace_back();
+    double ratio = (double)response.source.numWords(sentenceIdx) / response.target.numWords(sentenceIdx);
+
+    for (size_t wordIdx = 0; wordIdx < response.target.numWords(sentenceIdx); ++wordIdx) {
+      size_t source_token_idx = static_cast<size_t>(ratio * wordIdx);
+      assert(source_token_idx < response.source.numWords(sentenceIdx));
+      alignments.back().push_back(source_token_idx);
+    }
+  }
+}
+
+void CopyTaint(Response const &response, std::vector<std::vector<size_t>> const &alignments,
+               std::vector<HTML::Taint> const &token_tags, std::vector<HTML::Taint> &token_tags_target) {
+  size_t token_offset = 0;
+
+  // Fill token_tags_target based on the alignments we just made up.
+  // NOTE: this should match the exact order of Apply()
+  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
+    token_tags_target.push_back(token_tags[token_offset]);  // token_tag for sentence ending gap
+    for (size_t t = 0; t < response.target.numWords(sentenceIdx); ++t) {
+      size_t s = alignments[sentenceIdx][t];
+      assert(s < response.source.numWords(sentenceIdx));
+      token_tags_target.push_back(token_tags[token_offset + 1 + s]);  // +1 for prefix gap
+    }
+
+    token_offset += response.source.numWords(sentenceIdx) + 1;  // +1 for prefix gap
+  }
+
+  assert(token_offset < token_tags.size());
+  token_tags_target.push_back(token_tags[token_offset]);  // token_tag for ending whitespace
+}
+
+AnnotatedText RestoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &token_tags,
+                            std::vector<HTML::Span>::const_iterator span_it,
+                            std::vector<HTML::Span>::const_iterator span_end) {
+  auto prev_it = span_it;  // safe because first span is always empty span, and
+                           // and the while-loop below will do the rest
+
+  // workspace variables for lambda
+  std::string html;
+  HTML::Taint opening, closing;
+
+  return Apply(in, [&](ByteRange range, string_view token, bool last) {
+    // Do encoding of any entities that popped up in the translation
+    // (Also effectively clears html from previous call)
+    EncodeEntities(token, html);
+
+    size_t offset = 0;  // Size added by prepending HTML
+    size_t whitespace_size = CountPrefixWhitespaces(token);
+
+    // Potential issue: spans and tokens can intersect, e.g.
+    //
+    //    text  <p> h <u> e </u> ll o </p>
+    //   spans     |1|   |2|    |3333| (so only 2 is tainted with <p><u>, others only <p>)
+    //  tokens     |111111111111111|2|
+    //
+    // Now 1 covers span 1 to 3, so what taint should it get? Just <p>, or <p><u>?
+
+    // Seek to the last span that overlaps with this token
+    while (true) {
+      DiffTags(prev_it->tags, span_it->tags, opening, closing);
+      prev_it = span_it;
+
+      for (auto cit = closing.crbegin(); cit != closing.crend(); ++cit) {
+        std::string close_tag = format("</{}>", (*cit)->name);
+        html.insert(offset, close_tag);
+        offset += close_tag.size();
+      }
+
+      for (HTML::Tag const *tag : opening) {
+        std::string open_tag = format("<{}{}>", tag->name, tag->attributes);
+        html.insert(offset + whitespace_size, open_tag);
+        offset += open_tag.size();
+      }
+
+      if (span_it + 1 != span_end && ((span_it + 1)->begin < range.end || last)) {
+        span_it++;
+        continue;
+      }
+
+      break;
+    }
+
+    // TODO: This is just the taint of the last span, not the ones in between
+    // I don't know if that is okay for transferring taints. We'll need to test.
+    token_tags.push_back(prev_it->tags);
+
+    return html;
+  });
+}
+
+AnnotatedText RestoreTarget(AnnotatedText const &in, std::vector<HTML::Taint> const &token_tags_target) {
+  auto token_prev_it = token_tags_target.begin();
+  auto token_tags_it = token_tags_target.begin() + 1;
+
+  // workspace for lambda
+  std::string html;
+  HTML::Taint opening, closing;
+
+  AnnotatedText out = Apply(in, [&](ByteRange range, string_view token, bool last) {
+    // Do encoding of any entities that popped up in the translation
+    // (Also effectively clears html from previous call)
+    EncodeEntities(token, html);
+
+    size_t offset = 0;  // Size added by prepending HTML
+    size_t whitespace_size = CountPrefixWhitespaces(token);
+
+    assert(token_tags_it != token_tags_target.end());
+    DiffTags(*token_prev_it, *token_tags_it, opening, closing);
+
+    for (auto cit = closing.crbegin(); cit != closing.crend(); ++cit) {
+      std::string close_tag = format("</{}>", (*cit)->name);
+      html.insert(offset, close_tag);
+      offset += close_tag.size();
+    }
+
+    for (HTML::Tag const *tag : opening) {
+      std::string open_tag = format("<{}{}>", tag->name, tag->attributes);
+      html.insert(offset + whitespace_size, open_tag);
+      offset += open_tag.size();
+    }
+
+    // If this is the last token of the response, close all open tags.
+    if (last) {
+      for (auto cit = token_tags_it->crbegin(); cit != token_tags_it->crend(); ++cit) {
+        html += format("</{}>", (*cit)->name);
+      }
+    }
+
+    ++token_prev_it;
+    ++token_tags_it;
+
+    return html;
+  });
+
+  // Assert that we did in fact use all our taints
+  assert(token_tags_it == token_tags_target.end());
+
+  return out;
+}
+
+std::ostream &DebugPrintMapping(std::ostream &out, Response const &response,
+                                std::vector<std::vector<size_t>> const &alignments,
+                                std::vector<HTML::Taint> const &token_tags_target) {
+  auto taints = token_tags_target.begin();
+  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
+    out << "Mapped sentence prefix with tags: ";
+    for (auto &&taint : *(++taints)) out << '/' << taint->name;
+    out << '\n';
+
+    for (size_t wordIdx = 0; wordIdx < response.target.numWords(sentenceIdx); ++wordIdx) {
+      assert(sentenceIdx < alignments.size());
+      assert(wordIdx < alignments[sentenceIdx].size());
+
+      out << "Mapped ";
+      out << std::setw(10) << std::setfill(' ') << response.target.word(sentenceIdx, wordIdx);
+      out << " to ";
+      out << std::setw(10) << std::setfill(' ') << response.source.word(sentenceIdx, alignments[sentenceIdx][wordIdx]);
+      out << " with tags: ";
+      for (auto &&taint : *(++taints)) out << '/' << taint->name;
+      out << '\n';
+    }
+  }
+
+  out << "Mapped end-of-input with tags: ";
+  for (auto &&taint : *(++taints)) out << '/' << taint->name;
+  out << '\n';
+
+  assert(++taints == token_tags_target.end());
+  return out;
+}
+
+std::ostream &DebugPrintAlignmentScores(std::ostream &out, Response const &response) {
+  out << "std::vector<std::vector<std::vector<float>>> alignments{\n";
+  for (size_t sentenceIdx = 0; sentenceIdx < response.source.numSentences(); ++sentenceIdx) {
+    out << "  {\n";
+    for (size_t t = 0; t < response.alignments[sentenceIdx].size(); ++t) {
+      out << "    {";
+      for (size_t s = 0; s < response.alignments[sentenceIdx][t].size(); ++s) {
+        out << std::fixed << std::setw(8) << std::setprecision(8) << std::setfill(' ')
+            << response.alignments[sentenceIdx][t][s];
+        out << ", ";
+      }
+      out << "},\n";
+    }
+    out << "  },\n";
+  }
+  return out << "};\n";
+}
+
+size_t DebugCountTokens(AnnotatedText const &text) {
+  size_t tokens = 1;  // for the ending gap
+  for (size_t sentenceIdx = 0; sentenceIdx < text.numSentences(); ++sentenceIdx) {
+    tokens += 1 + text.numWords(sentenceIdx);  // pre-sentence prefix/gap + each word
+  }
+  return tokens;
+}
+
+}  // namespace
+
+namespace marian {
+namespace bergamot {
+
+HTML::HTML(std::string &&source, bool process_markup) {
+  if (!process_markup) return;
+  std::string original = std::move(source);
+  markup::instream in(original.data(), original.data() + original.size());
+  markup::scanner scanner(in);
+  source.clear();  // source is moved out of, so should be clear anyway
+
+  Taint stack;
+  spans_.push_back(Span{0, 0, {}});
+
+  bool stop = false;
+  while (!stop) {
+    switch (scanner.get_token()) {
+      case markup::scanner::TT_ERROR:
+        throw BadHTML("HTML parse error");
+
+      case markup::scanner::TT_EOF:
+        stop = true;
+        break;
+
+      case markup::scanner::TT_TEXT: {
+        auto begin = source.size();
+        source.append(scanner.get_value());
+        spans_.push_back(Span{begin, source.size(), stack});
+        FilterEmpty(stack);
+      } break;
+
+      case markup::scanner::TT_TAG_START:
+        // If it makes sense to treat this element as a break in a word (e.g.
+        // <br>, <img>, <li>) make sure it does so in this text as well.
+        // TODO: Strong assumption here that the language uses spaces to
+        // separate words
+        if (IsBlockElement(scanner.get_tag_name()) && !source.empty() && source.back() != ' ') source.push_back(' ');
+
+        pool_.emplace_back(new Tag{
+            scanner.get_tag_name(), std::string(),
+            IsEmtpyElement(scanner.get_tag_name())  // TODO: detect empty elements by doing a second pass and detecting
+                                                    // non-closed elements?
+        });
+
+        stack.push_back(pool_.back().get());
+        break;
+
+      case markup::scanner::TT_TAG_END:
+        // Note: self-closing tags emit TT_TAG_END immediately after TT_TAG_START
+        // but since we're parsing HTML5, a sole <img> will never emit a TT_TAG_END
+        if (stack.empty())
+          throw BadHTML(format("Encountered more closing tags ({}) than opening tags", scanner.get_tag_name()));
+
+        // TODO: what to do with "<u></u>" case, where tag is immediately closed
+        // so it never makes it into the taint of any of the spans? Add it as
+        // an empty tag to the previous/following?
+        if (stack.back()->name != scanner.get_tag_name())
+          throw BadHTML(format("Encountered unexpected closing tag </{}>, stack is {}", scanner.get_tag_name(), stack));
+        stack.pop_back();
+        break;
+
+      case markup::scanner::TT_ATTR:
+        // TODO could be more efficient if format() accepted a destination, i.e. format_to?
+        stack.back()->attributes += format(" {}=\"{}\"", scanner.get_attr_name(), scanner.get_value());
+        break;
+
+      default:
+        break;
+    }
+  }
+
+  if (!stack.empty()) throw BadHTML(format("Not all tags were closed: {}", stack));
+
+  // Add a trailing span (that's empty) to signify all closed tags.
+  spans_.emplace_back(Span{source.size() + 1, source.size() + 1, stack});
+}
+
+void HTML::Restore(Response &response) {
+  if (spans_.empty()) return;
+
+  // Reconstruction of HTML tags:
+  // 1. Map each token to a Span
+  // 2. Apply the taint of that span to the token
+  // 3. Reconstruct the source HTML with these tainted tokens
+  // 4. Transfer the taint from the source tokens to the target tokens using alignment information
+  // 5. Reconstruct the target HTML with these tainted tokens
+
+  std::vector<Taint> token_tags;  // List of HTML tags active per token in source
+                                  // Calculating these is a side-effect of restoring
+                                  // the HTML in response.source.
+
+  AnnotatedText source = RestoreSource(response.source, token_tags, spans_.cbegin(), spans_.cend());
+  assert(token_tags.size() == DebugCountTokens(response.source));
+
+  // Find for every token in target the token in source that best matches.
+  std::vector<std::vector<size_t>> alignments;
+
+  // If we do have alignment information from the model, we use that to taint
+  // tokens with the tags from their source token counterpart. If there is no
+  // alignment information available, we just interpolate based on sentence
+  // length (badly).
+  if (!response.alignments.empty()) {
+    // DebugPrintAlignmentScores(std::cerr, response);
+    HardAlignments(response, alignments);
+  } else {
+    InterpolateAlignments(response, alignments);
+  }
+
+  std::vector<Taint> token_tags_target;
+  token_tags_target.emplace_back();  // add empty one to the beginning for easy
+                                     // life later on (we start iterating at 1,
+                                     // and can then do i - 1 for empty.
+  CopyTaint(response, alignments, token_tags, token_tags_target);
+  assert(token_tags_target.size() == DebugCountTokens(response.target) + 1);
+
+  // DebugPrintMapping(std::cerr, response, alignments, token_tags_target);
+
+  AnnotatedText target = RestoreTarget(response.target, token_tags_target);
+
+  response.source = source;
+  response.target = target;
+}
+
+}  // namespace bergamot
+}  // namespace marian
diff --git a/src/translator/html.h b/src/translator/html.h
new file mode 100644
index 000000000..ba4691541
--- /dev/null
+++ b/src/translator/html.h
@@ -0,0 +1,50 @@
+#ifndef SRC_BERGAMOT_HTML_H_
+#define SRC_BERGAMOT_HTML_H_
+
+#include <stdexcept>
+#include <string>
+
+#include "definitions.h"
+
+namespace marian {
+namespace bergamot {
+
+struct Response;
+
+class BadHTML : public std::runtime_error {
+ public:
+  explicit BadHTML(std::string const &what) : std::runtime_error(what) {}
+};
+
+class HTML {
+ public:
+  struct Tag {
+    std::string name;
+    std::string attributes;
+    bool empty;
+  };
+
+  typedef std::vector<Tag *> Taint;
+
+  struct Span {
+    size_t begin;
+    size_t end;
+    Taint tags;  // Note: free pointer! Lifetime of tags is managed by pool_
+    inline size_t size() const { return end - begin; }
+  };
+
+  explicit HTML(std::string &&source, bool process_markup);
+  void Restore(Response &response);
+
+ private:
+  // List of text spans, and which tags are applied to them
+  std::vector<Span> spans_;
+
+  // a pool of tags that we free when HTML goes out of scope
+  std::vector<std::unique_ptr<Tag>> pool_;
+};
+
+}  // namespace bergamot
+}  // namespace marian
+
+#endif  // SRC_BERGAMOT_HTML_H_
diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
index 36bae1e9e..b9d163a2e 100644
--- a/src/translator/response_builder.h
+++ b/src/translator/response_builder.h
@@ -4,6 +4,7 @@
 #include <optional>
 
 #include "data/types.h"
+#include "html.h"
 #include "quality_estimator.h"
 #include "response.h"
 #include "response_options.h"
@@ -30,12 +31,13 @@ class ResponseBuilder {
   /// @param [in] qualityEstimator: the QualityEstimator model that can be used
   /// to provide translation quality probability.
   ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source, const Vocabs &vocabs,
-                  std::function<void(Response &&)> callback, const QualityEstimator &qualityEstimator)
+                  std::function<void(Response &&)> callback, const QualityEstimator &qualityEstimator, HTML &&html)
       : responseOptions_(responseOptions),
         source_(std::move(source)),
         vocabs_(vocabs),
         callback_(std::move(callback)),
-        qualityEstimator_(qualityEstimator) {}
+        qualityEstimator_(qualityEstimator),
+        html_(std::move(html)) {}
 
   /// Constructs and sets the promise of a Response object from obtained
   /// histories after translating.
@@ -62,6 +64,7 @@ class ResponseBuilder {
     if (responseOptions_.alignment) {
       buildAlignments(histories, response);
     }
+    html_.Restore(response);
 
     callback_(std::move(response));
   }
@@ -94,6 +97,8 @@ class ResponseBuilder {
   AnnotatedText source_;
 
   const QualityEstimator &qualityEstimator_;
+
+  HTML html_;
 };
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/translator/response_options.h b/src/translator/response_options.h
index 43b1c433b..b5867d00d 100644
--- a/src/translator/response_options.h
+++ b/src/translator/response_options.h
@@ -19,6 +19,8 @@ struct ResponseOptions {
   bool qualityScores{false};  ///< Include quality-scores or not.
   bool alignment{false};      ///< Include alignments or not.
 
+  bool HTML{false};  /// Remove HTML tags from text and (TODO) insert in output.
+
   /// Whether to include sentenceMappings or not. Alignments require
   /// sentenceMappings and are available irrespective of this option if
   /// `alignment=true`.
diff --git a/src/translator/translation_model.cpp b/src/translator/translation_model.cpp
index 5cf2b85f4..9d2eb0cdb 100644
--- a/src/translator/translation_model.cpp
+++ b/src/translator/translation_model.cpp
@@ -6,6 +6,7 @@
 #include "common/logging.h"
 #include "data/corpus.h"
 #include "data/text_input.h"
+#include "html.h"
 #include "parser.h"
 #include "translator/beam_search.h"
 
@@ -94,8 +95,10 @@ Ptr<Request> TranslationModel::makeRequest(size_t requestId, std::string &&sourc
   Segments segments;
   AnnotatedText annotatedSource;
 
+  HTML html(std::move(source), responseOptions.HTML);
   textProcessor_.process(std::move(source), annotatedSource, segments);
-  ResponseBuilder responseBuilder(responseOptions, std::move(annotatedSource), vocabs_, callback, *qualityEstimator_);
+  ResponseBuilder responseBuilder(responseOptions, std::move(annotatedSource), vocabs_, callback, *qualityEstimator_,
+                                  std::move(html));
 
   Ptr<Request> request =
       New<Request>(requestId, /*model=*/*this, std::move(segments), std::move(responseBuilder), cache);
diff --git a/src/translator/xh_scanner.cpp b/src/translator/xh_scanner.cpp
new file mode 100644
index 000000000..78ae13526
--- /dev/null
+++ b/src/translator/xh_scanner.cpp
@@ -0,0 +1,454 @@
+// https://www.codeproject.com/Articles/14076/Fast-and-Compact-HTML-XML-Scanner-Tokenizer
+// BSD license
+
+#include "xh_scanner.h"
+
+#include <cctype>
+#include <cstring>
+
+namespace markup {
+
+// case sensitive string equality test
+// s_lowcase shall be lowercase string
+inline bool equal(const char *s, const char *s1, size_t length) { return strncmp(s, s1, length) == 0; }
+
+const char *scanner::get_value() {
+  value[value_length] = 0;
+  return value;
+}
+
+const char *scanner::get_attr_name() {
+  attr_name[attr_name_length] = 0;
+  return attr_name;
+}
+
+const char *scanner::get_tag_name() {
+  tag_name[tag_name_length] = 0;
+  return tag_name;
+}
+
+scanner::token_type scanner::scan_body() {
+  text_begin = input.p;
+  if (input_char) {
+    --text_begin;
+  }
+  text_end = text_begin;
+  value_length = 0;
+  char c = get_char();
+
+  if (c == 0)
+    return TT_EOF;
+  else if (c == '<')
+    return scan_tag();
+  else if (c == '&')
+    return scan_entity();
+
+  while (true) {
+    append_value(c);
+    ++text_end;
+
+    c = get_char();
+
+    if (c == 0) {
+      push_back(c);
+      break;
+    }
+    if (c == '<') {
+      push_back(c);
+      break;
+    }
+    if (c == '&') {
+      push_back(c);
+      break;
+    }
+  }
+  return TT_TEXT;
+}
+
+scanner::token_type scanner::scan_head() {
+  char c = skip_whitespace();
+
+  if (c == '>') {
+    if (equal(tag_name, "script", 6)) {
+      // script is special because we want to parse the attributes,
+      // but not the content
+      c_scan = &scanner::scan_special;
+      return scan_special();
+    } else if (equal(tag_name, "style", 5)) {
+      // same with style
+      c_scan = &scanner::scan_special;
+      return scan_special();
+    }
+    c_scan = &scanner::scan_body;
+    return scan_body();
+  }
+  if (c == '/') {
+    char t = get_char();
+    if (t == '>') {
+      // self closing tag
+      c_scan = &scanner::scan_body;
+      return TT_TAG_END;
+    } else {
+      push_back(t);
+      return TT_ERROR;
+    }  // erroneous situtation - standalone '/'
+  }
+
+  attr_name_length = 0;
+  value_length = 0;
+
+  // attribute name...
+  while (c != '=') {
+    if (c == 0) return TT_EOF;
+    if (c == '>') {
+      push_back(c);
+      return TT_ATTR;
+    }  // attribute without value (HTML style)
+    if (is_whitespace(c)) {
+      c = skip_whitespace();
+      if (c != '=') {
+        push_back(c);
+        return TT_ATTR;
+      }  // attribute without value (HTML style)
+      else
+        break;
+    }
+    if (c == '<') return TT_ERROR;
+    append_attr_name(c);
+    c = get_char();
+  }
+
+  c = skip_whitespace();
+  // attribute value...
+
+  if (c == '\"') {
+    c = get_char();
+    while (c) {
+      if (c == '\"') return TT_ATTR;
+      // if (c == '&') c = scan_entity();
+      append_value(c);
+      c = get_char();
+    }
+  } else if (c == '\'')  // allowed in html
+  {
+    c = get_char();
+    while (c) {
+      if (c == '\'') return TT_ATTR;
+      // if (c == '&') c = scan_entity();
+      append_value(c);
+      c = get_char();
+    }
+  } else  // scan token, allowed in html: e.g. align=center
+  {
+    c = get_char();
+    do {
+      if (is_whitespace(c)) return TT_ATTR;
+      /* these two removed in favour of better html support:
+      if( c == '/' || c == '>' ) { push_back(c); return TT_ATTR; }
+      if( c == '&' ) c = scan_entity();*/
+      if (c == '>') {
+        push_back(c);
+        return TT_ATTR;
+      }
+      append_value(c);
+      c = get_char();
+    } while (c);
+  }
+
+  return TT_ERROR;
+}
+
+// caller already consumed '<'
+// scan header start or tag tail
+scanner::token_type scanner::scan_tag() {
+  tag_name_length = 0;
+
+  char c = get_char();
+
+  bool is_tail = c == '/';
+  if (is_tail) c = get_char();
+
+  while (c) {
+    if (is_whitespace(c)) {
+      c = skip_whitespace();
+      break;
+    }
+    if (c == '/' || c == '>') break;
+    append_tag_name(c);
+
+    switch (tag_name_length) {
+      case 3:
+        if (equal(tag_name, "!--", 3)) {
+          c_scan = &scanner::scan_comment;
+          return TT_COMMENT_START;
+        }
+        break;
+      case 8:
+        if (equal(tag_name, "![CDATA[", 8)) {
+          c_scan = &scanner::scan_cdata;
+          return TT_CDATA_START;
+        }
+        break;
+      case 7:
+        if (equal(tag_name, "!ENTITY", 8)) {
+          c_scan = &scanner::scan_entity_decl;
+          return TT_ENTITY_START;
+        }
+        break;
+    }
+
+    c = get_char();
+  }
+
+  if (c == 0) return TT_ERROR;
+
+  if (is_tail) {
+    if (c == '>') return TT_TAG_END;
+    return TT_ERROR;
+  } else
+    push_back(c);
+
+  c_scan = &scanner::scan_head;
+  return TT_TAG_START;
+}
+
+scanner::token_type scanner::scan_entity() {
+  // note that when scan_entity() is called, & is already consumed.
+
+  char buffer[8];
+  unsigned int buflen = 0;
+  buffer[buflen++] = '&';  // (just makes resolve_entity and append_value(buffer) easier)
+
+  bool has_end = false;
+
+  while (true) {
+    char c = get_char();
+    buffer[buflen++] = c;
+
+    // Found end of entity
+    if (c == ';') break;
+
+    // Too long to be entity
+    if (buflen == sizeof(buffer)) break;
+
+    // Not a character we'd expect in an entity (esp '&' or '<')
+    if (!isalpha(c)) break;
+  }
+
+  // Keep the text_end that scanner::scan_body uses similarly up-to-date. Since
+  // scan_entity() is only called from scan_body we assume text_begin is already
+  // set correctly by it.
+  text_end += buflen;
+
+  // If we found the end of the entity, and we can identify it, then
+  // resolve_entity() will emit the char it encoded.
+  if (buffer[buflen - 1] == ';' && resolve_entity(buffer, buflen)) {
+    return TT_TEXT;
+  }
+
+  // Otherwise, we just emit whatever we read as text, except for the last
+  // character that caused us to break. That may be another &, or a <, which we
+  // would want to scan properly.
+  for (unsigned int i = 0; i < buflen - 1; ++i) append_value(buffer[i]);
+  push_back(buffer[buflen - 1]);
+  --text_end;  // because push_back()
+  return TT_TEXT;
+}
+
+bool scanner::resolve_entity(char *buffer, unsigned int len) {
+  switch (len) {
+    case 4:
+      if (equal(buffer, "&lt;", 4)) {
+        append_value('<');
+        return true;
+      }
+      if (equal(buffer, "&gt;", 4)) {
+        append_value('>');
+        return true;
+      }
+      break;
+
+    case 5:
+      if (equal(buffer, "&amp;", 5)) {
+        append_value('&');
+        return true;
+      }
+      break;
+
+    case 6:
+      if (equal(buffer, "&quot;", 6)) {
+        append_value('"');
+        return true;
+      }
+      if (equal(buffer, "&apos;", 6)) {
+        append_value('\'');
+        return true;
+      }
+      if (equal(buffer, "&nbsp;", 6)) {
+        append_value(' ');  // TODO: handle non-breaking spaces better than just converting them to spaces
+        return true;
+      }
+      break;
+  }
+  return false;
+}
+
+// skip whitespaces.
+// returns first non-whitespace char
+char scanner::skip_whitespace() {
+  while (char c = get_char()) {
+    if (!is_whitespace(c)) return c;
+  }
+  return 0;
+}
+
+void scanner::push_back(char c) { input_char = c; }
+
+char scanner::get_char() {
+  if (input_char) {
+    char t(input_char);
+    input_char = 0;
+    return t;
+  }
+  return input.get_char();
+}
+
+bool scanner::is_whitespace(char c) {
+  return c <= ' ' && (c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\f');
+}
+
+void scanner::append_value(char c) {
+  if (value_length < (MAX_TOKEN_SIZE - 1)) value[value_length++] = c;
+}
+
+void scanner::append_attr_name(char c) {
+  if (attr_name_length < (MAX_NAME_SIZE - 1)) attr_name[attr_name_length++] = char(c);
+}
+
+void scanner::append_tag_name(char c) {
+  if (tag_name_length < (MAX_NAME_SIZE - 1))
+    tag_name[tag_name_length++] =
+        std::tolower(static_cast<unsigned char>(c));  // cast because std::tolower has undefined behaviour otherwise
+}
+
+scanner::token_type scanner::scan_comment() {
+  if (got_tail) {
+    c_scan = &scanner::scan_body;
+    got_tail = false;
+    return TT_COMMENT_END;
+  }
+  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
+    char c = get_char();
+    if (c == 0) return TT_EOF;
+    value[value_length] = c;
+
+    if (value_length >= 2 && value[value_length] == '>' && value[value_length - 1] == '-' &&
+        value[value_length - 2] == '-') {
+      got_tail = true;
+      value_length -= 2;
+      break;
+    }
+  }
+  return TT_DATA;
+}
+
+scanner::token_type scanner::scan_special() {
+  if (got_tail) {
+    c_scan = &scanner::scan_body;
+    got_tail = false;
+    return TT_TAG_END;
+  }
+  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
+    char c = get_char();
+    if (c == 0) return TT_EOF;
+
+    // in case MAX_TOKEN_SIZE limit breaks up the end tag
+    if (c == '<' && value_length + tag_name_length + 3 >= MAX_TOKEN_SIZE) {
+      push_back(c);
+      break;
+    }
+
+    value[value_length] = c;
+
+    if (c == '>' && value_length >= tag_name_length + 2) {
+      unsigned int i = tag_name_length - 1;
+      do {
+        if (value[value_length + i - tag_name_length] != tag_name[i]) break;
+        --i;
+      } while (i > 0);
+      if (i > 0) continue;
+      if (value[value_length - tag_name_length - 1] != '/') continue;
+      if (value[value_length - tag_name_length - 2] != '<') continue;
+
+      got_tail = true;
+      value_length = value_length - tag_name_length - 2;
+      break;
+    }
+  }
+  return TT_DATA;
+}
+
+scanner::token_type scanner::scan_cdata() {
+  if (got_tail) {
+    c_scan = &scanner::scan_body;
+    got_tail = false;
+    return TT_CDATA_END;
+  }
+  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
+    char c = get_char();
+    if (c == 0) return TT_EOF;
+    value[value_length] = c;
+
+    if (value_length >= 2 && value[value_length] == '>' && value[value_length - 1] == ']' &&
+        value[value_length - 2] == ']') {
+      got_tail = true;
+      value_length -= 2;
+      break;
+    }
+  }
+  return TT_DATA;
+}
+
+scanner::token_type scanner::scan_pi() {
+  if (got_tail) {
+    c_scan = &scanner::scan_body;
+    got_tail = false;
+    return TT_PI_END;
+  }
+  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
+    char c = get_char();
+    if (c == 0) return TT_EOF;
+    value[value_length] = c;
+
+    if (value_length >= 1 && value[value_length] == '>' && value[value_length - 1] == '?') {
+      got_tail = true;
+      value_length -= 1;
+      break;
+    }
+  }
+  return TT_DATA;
+}
+
+scanner::token_type scanner::scan_entity_decl() {
+  if (got_tail) {
+    c_scan = &scanner::scan_body;
+    got_tail = false;
+    return TT_ENTITY_END;
+  }
+  char t;
+  unsigned int tc = 0;
+  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
+    t = get_char();
+    if (t == 0) return TT_EOF;
+    value[value_length] = t;
+    if (t == '\"')
+      tc++;
+    else if (t == '>' && (tc & 1u) == 0) {
+      got_tail = true;
+      break;
+    }
+  }
+  return TT_DATA;
+}
+
+}  // namespace markup
diff --git a/src/translator/xh_scanner.h b/src/translator/xh_scanner.h
new file mode 100644
index 000000000..0b2dd2be2
--- /dev/null
+++ b/src/translator/xh_scanner.h
@@ -0,0 +1,130 @@
+// https://www.codeproject.com/Articles/14076/Fast-and-Compact-HTML-XML-Scanner-Tokenizer
+// BSD license
+//|
+//| simple and fast XML/HTML scanner/tokenizer
+//|
+//| (C) Andrew Fedoniouk @ terrainformatica.com
+//|
+#include <string.h>
+
+namespace markup {
+struct instream {
+  const char *p;
+  const char *end;
+  explicit instream(const char *src) : p(src), end(src + strlen(src)) {}
+  instream(const char *begin, const char *end) : p(begin), end(end) {}
+  char get_char() { return p < end ? *p++ : 0; }
+};
+
+class scanner {
+ public:
+  enum token_type {
+    TT_ERROR = -1,
+    TT_EOF = 0,
+
+    TT_TAG_START,  // <tag ...
+    //     ^-- happens here
+    TT_TAG_END,  // </tag>
+    //       ^-- happens here
+    // <tag ... />
+    //            ^-- or here
+    TT_ATTR,  // <tag attr="value" >
+    //                  ^-- happens here
+    TT_TEXT,
+
+    TT_DATA,  // content of followings:
+    // (also content of TT_TAG_START and TT_TAG_END, if the tag is 'script' or 'style')
+
+    TT_COMMENT_START,
+    TT_COMMENT_END,  // after "<!--" and "-->"
+    TT_CDATA_START,
+    TT_CDATA_END,  // after "<![CDATA[" and "]]>"
+    TT_PI_START,
+    TT_PI_END,  // after "<?" and "?>"
+    TT_ENTITY_START,
+    TT_ENTITY_END,  // after "<!ENTITY" and ">"
+
+  };
+
+  enum $ { MAX_TOKEN_SIZE = 1024, MAX_NAME_SIZE = 128 };
+
+ public:
+  explicit scanner(instream &is)
+      : value_length(0), tag_name_length(0), attr_name_length(0), input(is), input_char(0), got_tail(false) {
+    c_scan = &scanner::scan_body;
+  }
+
+  // get next token
+  token_type get_token() { return (this->*c_scan)(); }
+
+  // get text span backed by original input.
+  const char *get_text_begin() { return text_begin; }
+  const char *get_text_end() { return text_end; }
+
+  // get value of TT_TEXT, TT_ATTR and TT_DATA
+  const char *get_value();
+
+  // get attribute name
+  const char *get_attr_name();
+
+  // get tag name (always lowercase)
+  const char *get_tag_name();
+
+ private: /* methods */
+  typedef token_type (scanner::*scan)();
+
+  scan c_scan;  // current 'reader'
+
+  // content 'readers'
+  token_type scan_body();
+
+  token_type scan_head();
+
+  token_type scan_comment();
+
+  token_type scan_cdata();
+
+  token_type scan_special();
+
+  token_type scan_pi();
+
+  token_type scan_tag();
+
+  token_type scan_entity();
+
+  token_type scan_entity_decl();
+
+  char skip_whitespace();
+
+  void push_back(char c);
+
+  char get_char();
+
+  bool resolve_entity(char *buffer, unsigned int len);
+
+  static bool is_whitespace(char c);
+
+  void append_value(char c);
+
+  void append_attr_name(char c);
+
+  void append_tag_name(char c);
+
+ private: /* data */
+  char value[MAX_TOKEN_SIZE]{};
+  unsigned int value_length;
+
+  char tag_name[MAX_NAME_SIZE]{};
+  unsigned int tag_name_length;
+
+  char attr_name[MAX_NAME_SIZE]{};
+  unsigned int attr_name_length;
+
+  instream &input;
+  char input_char;
+
+  bool got_tail;  // aux flag used in scan_comment, etc.
+
+  const char *text_begin, *text_end;
+};
+}  // namespace markup
diff --git a/wasm/bindings/response_options_bindings.cpp b/wasm/bindings/response_options_bindings.cpp
index deafe1e0a..c58d24c64 100644
--- a/wasm/bindings/response_options_bindings.cpp
+++ b/wasm/bindings/response_options_bindings.cpp
@@ -15,5 +15,6 @@ using namespace emscripten;
 EMSCRIPTEN_BINDINGS(response_options) {
   value_object<ResponseOptions>("ResponseOptions")
       .field("qualityScores", &ResponseOptions::qualityScores)
-      .field("alignment", &ResponseOptions::alignment);
+      .field("alignment", &ResponseOptions::alignment)
+      .field("html", &ResponseOptions::HTML);
 }
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index 7fbaea8d2..f252a9b3c 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -323,7 +323,7 @@ const _parseTranslatedTextSentenceQualityScores = (vectorResponse) => {
 }
 
 const _prepareResponseOptions = () => {
-  return {qualityScores: true, alignment: false};
+  return {qualityScores: true, alignment: false, html: true};
 }
 
 const _prepareSourceText = (input) => {

From eea5554b91dceea0e51a628adc844d7fc1e7ae85 Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Mon, 29 Nov 2021 08:41:24 +0000
Subject: [PATCH 309/442] HTML handling improvements (#266)

* Fix out-of-bounds error when determining alignment for whole word

If token at offset 0 was a continuation (which it always is, since the first word of a sentence does not start with a space) it would jump to (unsigned) -1 which is probably out of bounds.

* Don't segfault if alignment info is not available

When alignment info is requested, but model is missing `alignment: soft` you'd get empty alignment info for every target token.

* Partial fix for handling empty elements

This fixes a parse error when dealing with something like `<p>...<br></p>` or `...<br>` where there is no text after the last empty element. This also prevents losing empty elements in the source side of the translation. Empty elements are not yet transferred correctly to the target side.

* Fix formatting
---
 src/tests/units/html_tests.cpp | 18 ++++++++++
 src/translator/html.cpp        | 64 ++++++++++++++++++++++++----------
 2 files changed, 64 insertions(+), 18 deletions(-)

diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
index 258847970..59244a1b5 100644
--- a/src/tests/units/html_tests.cpp
+++ b/src/tests/units/html_tests.cpp
@@ -395,6 +395,24 @@ TEST_CASE("Test self-closing tag (HTML5)") {
   CHECK(input == "hello  world and other creatures\n");  // Note double space between "hello" and "world"
 }
 
+TEST_CASE("Test empty self-closing tag at end of input") {
+  std::string input("hello <br>");
+  HTML html(std::move(input), true);
+  CHECK(input == "hello ");
+}
+
+TEST_CASE("Test empty tag pair at end of input") {
+  std::string input("hello <u></u>");
+  HTML html(std::move(input), true);
+  CHECK(input == "hello ");
+}
+
+TEST_CASE("Test empty self-closing pair at end of input in parent") {
+  std::string input("<p>hello <br></p>");
+  HTML html(std::move(input), true);
+  CHECK(input == "hello ");
+}
+
 TEST_CASE("Test empty tag", "[!mayfail]") {
   std::string input(
       "<p id=\"1\">hello <img id=\"1.1\"><span id=\"1.2\"><u id=\"1.2.1\"></u><b id=\"1.2.2\"></b><img "
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index 0614c37e5..348b65ca3 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -94,7 +94,7 @@ bool IsBlockElement(std::string const &name) {
   return inline_ish_elements.find(name) == inline_ish_elements.end();
 }
 
-bool IsEmtpyElement(std::string const &name) {
+bool IsEmptyElement(std::string const &name) {
   // List of elements for which we do not expect a closing tag, or self-closing
   // elements in XHTML. See also https://developer.mozilla.org/en-US/docs/Glossary/Empty_element
   static std::unordered_set<std::string> empty_elements{"area",  "base", "br",   "col",   "embed",  "hr",    "img",
@@ -132,6 +132,10 @@ void FilterEmpty(HTML::Taint &stack) {
   stack.resize(dst - stack.begin());
 }
 
+bool ContainsTag(HTML::Taint const &stack, HTML::Tag const *tag) {
+  return std::find(stack.rbegin(), stack.rend(), tag) != stack.rend();
+}
+
 template <typename Fun>
 AnnotatedText Apply(AnnotatedText const &in, Fun fun) {
   AnnotatedText out;
@@ -166,6 +170,10 @@ AnnotatedText Apply(AnnotatedText const &in, Fun fun) {
 
 bool IsContinuation(string_view str) { return !str.empty() && str.compare(0, 1, " ", 1) != 0; }
 
+bool HasAlignments(Response const &response) {
+  return !response.alignments.empty() && !response.alignments[0][0].empty();
+}
+
 void HardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
   // For each sentence...
   for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
@@ -199,11 +207,11 @@ void HardAlignments(Response const &response, std::vector<std::vector<size_t>> &
         size_t s_max = score_curr > score_prev ? s_curr : s_prev;
 
         // Apply this to all previous tokens in the word
-        for (size_t i = t; i >= 0; --i) {
+        for (size_t i = t;; --i) {
           alignments.back()[i] = s_max;
 
-          // Stop if this was the beginning of the word
-          if (!IsContinuation(response.target.word(sentenceIdx, i))) break;
+          // Stop if this was the first token or the beginning of the word
+          if (i == 0 || !IsContinuation(response.target.word(sentenceIdx, i))) break;
         }
       }
     }
@@ -262,6 +270,14 @@ AnnotatedText RestoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &t
     size_t offset = 0;  // Size added by prepending HTML
     size_t whitespace_size = CountPrefixWhitespaces(token);
 
+    // Close tags we want to show up left (before) the token, but open tags
+    // ideally come directly after any prefix whitespace. However, some tokens
+    // match multiple spans. If a previous span has added an open tag, after any
+    // whitespace, and the next span closes said tag again, we need to close
+    // it after the whitespace. So after the first open tag, any closing tag
+    // should also align right, after whitespace, not before. Hence this bool.
+    bool close_left = true;
+
     // Potential issue: spans and tokens can intersect, e.g.
     //
     //    text  <p> h <u> e </u> ll o </p>
@@ -277,7 +293,7 @@ AnnotatedText RestoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &t
 
       for (auto cit = closing.crbegin(); cit != closing.crend(); ++cit) {
         std::string close_tag = format("</{}>", (*cit)->name);
-        html.insert(offset, close_tag);
+        html.insert(offset + (close_left ? 0 : whitespace_size), close_tag);
         offset += close_tag.size();
       }
 
@@ -285,6 +301,7 @@ AnnotatedText RestoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &t
         std::string open_tag = format("<{}{}>", tag->name, tag->attributes);
         html.insert(offset + whitespace_size, open_tag);
         offset += open_tag.size();
+        close_left = false;
       }
 
       if (span_it + 1 != span_end && ((span_it + 1)->begin < range.end || last)) {
@@ -295,8 +312,9 @@ AnnotatedText RestoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &t
       break;
     }
 
-    // TODO: This is just the taint of the last span, not the ones in between
-    // I don't know if that is okay for transferring taints. We'll need to test.
+    // TODO: This is just the taint of the last span, not the ones in between.
+    // This makes us lose empty tags, and maybe some markup as well, in the
+    // response target HTML restoration.
     token_tags.push_back(prev_it->tags);
 
     return html;
@@ -422,6 +440,7 @@ HTML::HTML(std::string &&source, bool process_markup) {
   markup::scanner scanner(in);
   source.clear();  // source is moved out of, so should be clear anyway
 
+  Tag *tag;
   Taint stack;
   spans_.push_back(Span{0, 0, {}});
 
@@ -449,13 +468,19 @@ HTML::HTML(std::string &&source, bool process_markup) {
         // separate words
         if (IsBlockElement(scanner.get_tag_name()) && !source.empty() && source.back() != ' ') source.push_back(' ');
 
-        pool_.emplace_back(new Tag{
-            scanner.get_tag_name(), std::string(),
-            IsEmtpyElement(scanner.get_tag_name())  // TODO: detect empty elements by doing a second pass and detecting
-                                                    // non-closed elements?
-        });
+        tag = new Tag{scanner.get_tag_name(), std::string(), IsEmptyElement(scanner.get_tag_name())};
+        pool_.emplace_back(tag);  // pool_ takes ownership of our tag
 
         stack.push_back(pool_.back().get());
+
+        // Empty elements (e.g. <img>) are not applicable to a span of text
+        // so instead we "apply" them to an empty span in between, and then
+        // immediately remove them again from the stack.
+        if (tag->empty) {
+          spans_.push_back(Span{source.size(), source.size(), stack});
+          stack.pop_back();
+        }
+
         break;
 
       case markup::scanner::TT_TAG_END:
@@ -464,17 +489,20 @@ HTML::HTML(std::string &&source, bool process_markup) {
         if (stack.empty())
           throw BadHTML(format("Encountered more closing tags ({}) than opening tags", scanner.get_tag_name()));
 
-        // TODO: what to do with "<u></u>" case, where tag is immediately closed
-        // so it never makes it into the taint of any of the spans? Add it as
-        // an empty tag to the previous/following?
         if (stack.back()->name != scanner.get_tag_name())
           throw BadHTML(format("Encountered unexpected closing tag </{}>, stack is {}", scanner.get_tag_name(), stack));
+
+        // What to do with "<u></u>" case, where tag is immediately closed
+        // so it never makes it into the taint of any of the spans? This adds
+        // an empty span so it still lives.
+        if (spans_.empty() || !ContainsTag(spans_.back().tags, stack.back()))
+          spans_.push_back(Span{source.size(), source.size(), stack});
+
         stack.pop_back();
         break;
 
       case markup::scanner::TT_ATTR:
-        // TODO could be more efficient if format() accepted a destination, i.e. format_to?
-        stack.back()->attributes += format(" {}=\"{}\"", scanner.get_attr_name(), scanner.get_value());
+        tag->attributes += format(" {}=\"{}\"", scanner.get_attr_name(), scanner.get_value());
         break;
 
       default:
@@ -512,7 +540,7 @@ void HTML::Restore(Response &response) {
   // tokens with the tags from their source token counterpart. If there is no
   // alignment information available, we just interpolate based on sentence
   // length (badly).
-  if (!response.alignments.empty()) {
+  if (HasAlignments(response)) {
     // DebugPrintAlignmentScores(std::cerr, response);
     HardAlignments(response, alignments);
   } else {

From e8fd01e9f4c28d3acdd49485cd4b9395b87aa631 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <aaggarwal@mozilla.com>
Date: Tue, 30 Nov 2021 14:31:01 +0100
Subject: [PATCH 310/442] Updated marian-dev submodule

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 200e81c0c..a284a05a1 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 200e81c0cc88259c540b96afc6e0867cb05570b0
+Subproject commit a284a05a12bdc6fdf72223c0120838b26d3a977c

From 8e79897f30a3948621e95b657fda2bfc6f69bc76 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 1 Dec 2021 11:32:51 +0100
Subject: [PATCH 311/442] Updated configuration for html text translation to
 work in wasm test page (#269)

* Updated translator configuration in wasm test page
 - Added alignment: soft

* Set ResponseOptions::alignment to "true"
 - Had to be set for html text translation to work
---
 wasm/test_page/js/worker.js | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index f252a9b3c..2711fe1c6 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -181,6 +181,7 @@ cpu-threads: 0
 quiet: true
 quiet-translation: true
 gemm-precision: int8shiftAlphaAll
+alignment: soft
 `;
 
   const modelFile = `${rootURL}/${languagePair}/${modelRegistry[languagePair]["model"].name}`;
@@ -323,7 +324,7 @@ const _parseTranslatedTextSentenceQualityScores = (vectorResponse) => {
 }
 
 const _prepareResponseOptions = () => {
-  return {qualityScores: true, alignment: false, html: true};
+  return {qualityScores: true, alignment: true, html: true};
 }
 
 const _prepareSourceText = (input) => {

From e75a9e1da3ecaace48b8cf41c191c1920b3cd3ac Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Tue, 14 Dec 2021 16:39:19 +0100
Subject: [PATCH 312/442] More robust logic to import wasm gemm (#276)

- Import optimized gemm implementation only if all the necessary functions
   are provided by it, othewise use the fallback gemm
---
 wasm/import-gemm-module.js | 49 +++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 14 deletions(-)

diff --git a/wasm/import-gemm-module.js b/wasm/import-gemm-module.js
index e23a69d7f..369c551cc 100644
--- a/wasm/import-gemm-module.js
+++ b/wasm/import-gemm-module.js
@@ -3,23 +3,44 @@
  * implementation.
  */
 function createWasmGemm() {
+    // Name of the optimized gemm implementation.
     const OPTIMIZED_GEMM = "mozIntGemm";
-    const FALLBACK_GEMM =  "asm";
 
-    if (WebAssembly[OPTIMIZED_GEMM]) {
-        console.log(`Using optimized gemm (${OPTIMIZED_GEMM}) implementation`);
-        return new WebAssembly.Instance(WebAssembly[OPTIMIZED_GEMM](), {"": {memory: wasmMemory}}).exports;
+    // A map of expected gemm function to the corresponding fallback gemm function names.
+    const GEMM_TO_FALLBACK_FUNCTIONS_MAP = {
+        "int8_prepare_a": "int8PrepareAFallback",
+        "int8_prepare_b": "int8PrepareBFallback",
+        "int8_prepare_b_from_transposed": "int8PrepareBFromTransposedFallback",
+        "int8_prepare_b_from_quantized_transposed": "int8PrepareBFromQuantizedTransposedFallback",
+        "int8_prepare_bias": "int8PrepareBiasFallback",
+        "int8_multiply_and_add_bias": "int8MultiplyAndAddBiasFallback",
+        "int8_select_columns_of_b": "int8SelectColumnsOfBFallback"
+    };
+
+    const optimizedGemmModule = WebAssembly[OPTIMIZED_GEMM];
+    if (!optimizedGemmModule) {
+        return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
     }
-    else {
-        console.log(`Using fallback gemm implementation`);
-        return {
-            "int8_prepare_a": (...a) => Module[FALLBACK_GEMM]["int8PrepareAFallback"](...a),
-            "int8_prepare_b": (...a) => Module[FALLBACK_GEMM]["int8PrepareBFallback"](...a),
-            "int8_prepare_b_from_transposed": (...a) => Module[FALLBACK_GEMM]["int8PrepareBFromTransposedFallback"](...a),
-            "int8_prepare_b_from_quantized_transposed": (...a) => Module[FALLBACK_GEMM]["int8PrepareBFromQuantizedTransposedFallback"](...a),
-            "int8_prepare_bias": (...a) => Module[FALLBACK_GEMM]["int8PrepareBiasFallback"](...a),
-            "int8_multiply_and_add_bias": (...a) => Module[FALLBACK_GEMM]["int8MultiplyAndAddBiasFallback"](...a),
-            "int8_select_columns_of_b": (...a) => Module[FALLBACK_GEMM]["int8SelectColumnsOfBFallback"](...a)
+
+    const optimizedGemmModuleExports = new WebAssembly.Instance(optimizedGemmModule(), {"": {memory: wasmMemory}}).exports;
+    for (let key in GEMM_TO_FALLBACK_FUNCTIONS_MAP) {
+        if (!optimizedGemmModuleExports[key]) {
+            return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
         }
     }
+    console.log(`Using optimized gemm (${OPTIMIZED_GEMM}) implementation`);
+    return optimizedGemmModuleExports;
+}
+
+// Return the fallback gemm implementation.
+function fallbackGemm(gemmToFallbackFunctionsMap) {
+    // The fallback gemm implementation
+    const FALLBACK_GEMM = "asm";
+
+    let fallbackGemmModuleExports = {};
+    for (let key in gemmToFallbackFunctionsMap) {
+        fallbackGemmModuleExports[key] = (...a) => Module[FALLBACK_GEMM][gemmToFallbackFunctionsMap[key]](...a)
+    }
+    console.log(`Using fallback gemm implementation`);
+    return fallbackGemmModuleExports;
 }

From 571d312930374d834f3dbdc4cd611f4be1fc820e Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Tue, 14 Dec 2021 16:34:30 +0000
Subject: [PATCH 313/442] Constrain mistune to fix docs CI (#278)

---
 doc/requirements.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/requirements.txt b/doc/requirements.txt
index 8d56e6839..28e6e70ca 100644
--- a/doc/requirements.txt
+++ b/doc/requirements.txt
@@ -2,5 +2,6 @@ sphinx==2.4.4
 breathe==4.13.0
 exhale
 sphinx_rtd_theme
+mistune<2.0.0
 recommonmark
 m2r

From feb9c90429fe23423dc23c56e8d0ee19d85acec7 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Tue, 14 Dec 2021 21:52:00 +0100
Subject: [PATCH 314/442] Additional logs in JS translation worker (#277)

- Print source text received in the response
 - Print no. of block elements in the input
---
 wasm/test_page/js/worker.js | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index 2711fe1c6..3bc89bef5 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -53,11 +53,14 @@ onmessage = async function(e) {
       const to = e.data[2];
       const input = e.data[3];
       let inputWordCount = 0;
+      let inputBlockElements = 0;
       input.forEach(sentence => {
         inputWordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
+        inputBlockElements++;
       })
       let start = Date.now();
       try {
+        log(`Blocks to translate: ${inputBlockElements}`);
         result = translate(from, to, input);
         const secs = (Date.now() - start) / 1000;
         log(`Translation '${from}${to}' Successful. Speed: ${Math.round(inputWordCount / secs)} WPS (${inputWordCount} words in ${secs} secs)`);
@@ -243,10 +246,12 @@ const _translateInvolvingEnglish = (from, to, input) => {
 
   // Parse all relevant information from vectorResponse
   const listTranslatedText = _parseTranslatedText(vectorResponse);
+  const listSourceText = _parseSourceText(vectorResponse);
   const listTranslatedTextSentences = _parseTranslatedTextSentences(vectorResponse);
   const listSourceTextSentences = _parseSourceTextSentences(vectorResponse);
   const listTranslatedTextSentenceQualityScores = _parseTranslatedTextSentenceQualityScores(vectorResponse);
 
+  log(`Source text: ${listSourceText}`);
   log(`Translated text: ${listTranslatedText}`);
   log(`Translated sentences: ${JSON.stringify(listTranslatedTextSentences)}`);
   log(`Source sentences: ${JSON.stringify(listSourceTextSentences)}`);
@@ -276,6 +281,15 @@ const _parseTranslatedTextSentences = (vectorResponse) => {
   return result;
 }
 
+const _parseSourceText = (vectorResponse) => {
+  const result = [];
+  for (let i = 0; i < vectorResponse.size(); i++) {
+    const response = vectorResponse.get(i);
+    result.push(response.getOriginalText());
+  }
+  return result;
+}
+
 const _parseSourceTextSentences = (vectorResponse) => {
   const result = [];
   for (let i = 0; i < vectorResponse.size(); i++) {

From 8563f0856f6dada0d6ce4037e727033dda97cfd1 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Tue, 14 Dec 2021 23:53:53 +0000
Subject: [PATCH 315/442] Proper arch setting on win32 (#275)

* Proper arch detection on win32

* Whoops
---
 3rd_party/marian-dev |  2 +-
 CMakeLists.txt       | 36 ++++++++++++++++++++++++++++++------
 2 files changed, 31 insertions(+), 7 deletions(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index a284a05a1..08b154463 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit a284a05a12bdc6fdf72223c0120838b26d3a977c
+Subproject commit 08b1544636fe13eaf1fbacb17c6fb050abfb8d42
diff --git a/CMakeLists.txt b/CMakeLists.txt
index a9586d8e5..006e9521d 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -20,10 +20,39 @@ if(NOT CMAKE_BUILD_TYPE)
   message(WARNING "CMAKE_BUILD_TYPE not set; setting to Release")
   set(CMAKE_BUILD_TYPE "Release")
 endif()
+
+if(NOT COMPILE_WASM)
+  # Setting BUILD_ARCH to native invokes CPU intrinsic detection logic below.
+  # Prevent invoking that logic for WASM builds.
+  set(BUILD_ARCH native CACHE STRING "Compile for this CPU architecture.")
+
+  # Unfortunately MSVC supports a limited subset of BUILD_ARCH flags. Instead try to guess
+  # what architecture we can compile to reading BUILD_ARCH and mapping it to MSVC values
+  # references: https://clang.llvm.org/docs/UsersManual.html https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html https://gcc.gnu.org/onlinedocs/gcc-4.8.5/gcc/i386-and-x86-64-Options.html
+  # https://docs.microsoft.com/en-us/cpp/build/reference/arch-x86?redirectedfrom=MSDN&amp;amp;view=vs-2019&view=msvc-170 https://devblogs.microsoft.com/oldnewthing/20201026-00/?p=104397
+  # This is by no means an exhaustive list but should match the most common flags Linux programmers expect to parse to MSVC
+  if(MSVC)
+    if(BUILD_ARCH STREQUAL "native") # avx2 is good default for native. Very few desktop systems support avx512
+      set(MSVC_BUILD_ARCH "/arch:AVX2")
+    elseif(BUILD_ARCH STREQUAL "skylake-avx512" OR BUILD_ARCH STREQUAL "cannonlake" OR BUILD_ARCH STREQUAL "x86-64-v4" OR BUILD_ARCH STREQUAL "tigerlake" OR BUILD_ARCH STREQUAL "cooperlake" OR BUILD_ARCH STREQUAL "cascadelake")
+      set(MSVC_BUILD_ARCH "/arch:AVX512")
+    elseif(BUILD_ARCH STREQUAL "core-avx2" OR BUILD_ARCH STREQUAL "haswell" OR BUILD_ARCH STREQUAL "x86-64-v3" OR BUILD_ARCH STREQUAL "broadwell" OR BUILD_ARCH STREQUAL "skylake")
+      set(MSVC_BUILD_ARCH "/arch:AVX2")
+    elseif(BUILD_ARCH STREQUAL "sandybridge" OR BUILD_ARCH STREQUAL "corei7-avx" OR BUILD_ARCH STREQUAL "core-avx-i" OR BUILD_ARCH STREQUAL "ivybridge")
+      set(MSVC_BUILD_ARCH "/arch:AVX")
+    elseif(BUILD_ARCH STREQUAL "nehalem" OR BUILD_ARCH STREQUAL "westmere" OR BUILD_ARCH STREQUAL "x86-64-v2" OR BUILD_ARCH STREQUAL "corei7" OR BUILD_ARCH STREQUAL "core2")
+      set(MSVC_BUILD_ARCH "/arch:SSE2") # This is MSVC default. We won't go down to SSE because we don't support that hardware at all with intgemm. Marian recommends to only go down to SSE4.1 at most
+    else()
+      message(WARNING "Unknown BUILD_ARCH ${BUILD_ARCH} provided. Default to SSE2 for Windows build")
+      set(MSVC_BUILD_ARCH "/arch:SSE2")
+    endif()
+  endif(MSVC)
+endif()
+
 #MSVC can't seem to pick up correct flags otherwise:
 if(MSVC)
   add_definitions(-DUSE_SSE2=1) # Supposed to fix something in the sse_mathfun.h but not sure it does
-  set(INTRINSICS "/arch:AVX2") # ARCH we're targetting on win32. @TODO variable
+  set(INTRINSICS ${MSVC_BUILD_ARCH}) # ARCH we're targetting on win32. @TODO variable
   
   set(CMAKE_CXX_FLAGS           "/EHsc /DWIN32 /D_WINDOWS /DUNICODE /D_UNICODE /D_CRT_NONSTDC_NO_WARNINGS /D_CRT_SECURE_NO_WARNINGS /bigobj")
   set(CMAKE_CXX_FLAGS_RELEASE   "${CMAKE_CXX_FLAGS} /MT /O2 ${INTRINSICS} /Zi /MP /GL /DNDEBUG")
@@ -80,11 +109,6 @@ include(GetVersionFromFile)
 message(STATUS "Project name: ${PROJECT_NAME}")
 message(STATUS "Project version: ${PROJECT_VERSION_STRING_FULL}")
 
-if(NOT COMPILE_WASM)
-  # Set BUILD_ARCH to native only while compiling for non wasm platform
-  set(BUILD_ARCH native CACHE STRING "Compile for this CPU architecture.")
-endif()
-
 if(COMPILE_WASM)
   set(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
   list(APPEND WASM_COMPILE_FLAGS -O3 -g2 -fPIC -mssse3 -msimd128)

From 420f12b3ff9eb854a8e09be975d4deddf60712c2 Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Wed, 15 Dec 2021 23:01:49 +0100
Subject: [PATCH 316/442] Remove value length limit from HTML parser &
 interpolated alignments (#274)

* Remove InterpolateAlignment

And some code improvements

* Replace the fixed value buffer with a std::string backing

* Fix tests that had no alignment info

These depended on the linear interpolation that I removed

* Remove arbitrary limits on tag and attribute names

This might also fix a bug caused by the eager lower casing of tag names, which could break <![CDATA , <style> and <script>

* Remove equals() in favour of operator==()

I trust the compiler can come up with better optimisations than I can.

* Expose std::strings instead of their data

Should save us some std::strlen() calls

* Add & remove headers and no-longer-defined functions from header files

* Remove all string buffers from xh_scanner

It now directly refers to either the input stream or constant strings

* Replace custom string_view with even lighter struct that's only used internally

To the outside world we just expose std::string_view

* Remove __builtin_sub_overflow for MSVC

* ABORT if trying to restore HTML when no alignment info is available

* Add test cases specifically for xh_scanner

Both good for testing regression, and as a little example/reference for what behaviour to expect from it.

* Add --html option to bergamot for tests

This should make it easier to have some integration tests for HTML input

* Add test and fix for empty inputs failing due to alignment check

Co-authored-by: Jerin Philip <jerinphilip@live.in>
---
 app/cli.h                            |   8 +
 src/tests/units/CMakeLists.txt       |   3 +-
 src/tests/units/html_tests.cpp       | 181 +++-----
 src/tests/units/xh_scanner_tests.cpp | 125 ++++++
 src/translator/html.cpp              | 104 ++---
 src/translator/parser.cpp            |   2 +
 src/translator/parser.h              |   1 +
 src/translator/xh_scanner.cpp        | 594 +++++++++++----------------
 src/translator/xh_scanner.h          |  97 ++---
 9 files changed, 539 insertions(+), 576 deletions(-)
 create mode 100644 src/tests/units/xh_scanner_tests.cpp

diff --git a/app/cli.h b/app/cli.h
index 9cb12dd28..08f203466 100644
--- a/app/cli.h
+++ b/app/cli.h
@@ -56,6 +56,10 @@ void wasm(const CLIConfig &config) {
       std::make_shared<TranslationModel>(options->asYamlString(), std::move(memoryBundle));
 
   ResponseOptions responseOptions;
+  if (config.html) {
+    responseOptions.HTML = true;
+    responseOptions.alignment = true;  // Necessary for HTML
+  }
   std::vector<std::string> texts;
 
   // Hide the translateMultiple operation
@@ -145,6 +149,10 @@ void native(const CLIConfig &config) {
   std::string input = std_input.str();
 
   ResponseOptions responseOptions;
+  if (config.html) {
+    responseOptions.HTML = true;
+    responseOptions.alignment = true;  // Necessary for HTML
+  }
 
   // Wait on future until Response is complete
   std::promise<Response> responsePromise;
diff --git a/src/tests/units/CMakeLists.txt b/src/tests/units/CMakeLists.txt
index 8c29ab397..9cfb50006 100644
--- a/src/tests/units/CMakeLists.txt
+++ b/src/tests/units/CMakeLists.txt
@@ -3,7 +3,8 @@ set(UNIT_TESTS
     annotation_tests
     cache_tests
     quality_estimator_tests
-    html_tests)
+    html_tests
+    xh_scanner_tests)
 
 foreach(test ${UNIT_TESTS})
   add_executable("run_${test}" run_tests.cpp "${test}.cpp")
diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
index 59244a1b5..8f6ca8216 100644
--- a/src/tests/units/html_tests.cpp
+++ b/src/tests/units/html_tests.cpp
@@ -49,6 +49,16 @@ void RecordSentenceFromByteRange(AnnotatedText &text, std::vector<ByteRange> con
   text.recordExistingSentence(tokens.begin(), tokens.end(), text.text.data() + ranges[0].begin);
 }
 
+template <typename T>
+std::vector<std::vector<T>> identity_matrix(size_t size) {
+  std::vector<std::vector<T>> rows(size);
+  for (size_t row = 0; row < size; ++row) {
+    rows[row].resize(size, T(0));
+    rows[row][row] = T(1);
+  }
+  return rows;
+}
+
 TEST_CASE("Ignore HTML if process_markup is false") {
   std::string html_code("<p>This text &amp; has <b>HTML</b> in it</p>");
 
@@ -59,108 +69,66 @@ TEST_CASE("Ignore HTML if process_markup is false") {
   Response response;
   response.source.text = html_code;
   response.target.text = html_code;
+  // Note: response.alignments is empty, which is allowed in this case
   html.Restore(response);
 
   // Assert that Restore() does not mess with my HTML code
   CHECK(response.source.text == html_code);
 }
 
-TEST_CASE("Test reconstruction") {
-  std::string input("<p><input>H<u>e</u>llo <b>world</b> how <u>are you</u>?</p>\n");
-
-  std::string text(input);
-  HTML html(std::move(text), true);  // TODO: move, but really a reference?
-  CHECK(text == "Hello world how are you?\n");
-
-  AnnotatedText source(std::move(text));
-  std::vector<string_view> tokens{
-      string_view(source.text.data() + 0, 4),   // Hell
-      string_view(source.text.data() + 4, 1),   // o
-      string_view(source.text.data() + 5, 6),   // _world
-      string_view(source.text.data() + 11, 4),  // _how
-      string_view(source.text.data() + 15, 4),  // _are
-      string_view(source.text.data() + 19, 4),  // _you
-      string_view(source.text.data() + 23, 1),  // ?
-      string_view(source.text.data() + 24, 0),  // "\n" (but 0 length?)
-  };
+// TODO: is there a better way to test for correct abort() calls than [!shouldfail]
+TEST_CASE("Abort if alignments are missing") {
+  marian::setThrowExceptionOnAbort(true);
 
-  source.recordExistingSentence(tokens.begin(), tokens.end(), source.text.data());
+  std::string input("<p>hello <b>world</b></p>\n");
+  HTML html(std::move(input), true);
+
+  AnnotatedText source("hello world\n");
+  RecordSentenceFromByteRange(source, {
+                                          ByteRange{0, 4},   // 0.0 "hell"
+                                          ByteRange{4, 5},   // 0.1 "o"
+                                          ByteRange{5, 11},  // 0.2 " world"
+                                          ByteRange{11, 11}  // 0.3 ""
+                                      });
+
+  AnnotatedText target("hallo Welt\n");
+  RecordSentenceFromByteRange(target, {
+                                          ByteRange{0, 4},   // 0.0 "hall"
+                                          ByteRange{4, 5},   // 0.1 "o"
+                                          ByteRange{5, 10},  // 0.2 " Welt"
+                                          ByteRange{10, 10}  // 0.3 ""
+                                      });
 
   Response response;
   response.source = source;
+  response.target = target;
+  // Note: explicitly not setting response.alignments
 
-  html.Restore(response);
-  // CHECK(response.source.text == input);  // fails because <u></u> has been moved to the front of the token
-  CHECK(response.source.text == "<p><input><u></u>Hello <b>world</b> how <u>are you</u>?</p>\n");
-
-  std::vector<ByteRange> restored_tokens{
-      ByteRange{0, 0 + 0},    // (start of sentence)
-      ByteRange{0, 0 + 21},   // <p><input>H<u>e</u>ll
-      ByteRange{21, 21 + 1},  // o
-      ByteRange{22, 22 + 9},  // _<b>world
-      ByteRange{31, 31 + 8},  // </b>_how
-      ByteRange{39, 39 + 7},  // _<u>are
-      ByteRange{46, 46 + 4},  // _you
-      ByteRange{50, 50 + 5},  // </u>?
-      ByteRange{55, 55 + 0},  // ""
-      ByteRange{55, 55 + 5},  // </p>\n
-  };
-  CHECK(response.source.text.size() == restored_tokens.back().end);
-  CHECK(AsByteRanges(response.source) == restored_tokens);
-
-  // Same test as above, but easier to read. Will use this further down.
-  std::vector<std::string> restored_tokens_str{"",
-                                               "<p><input><u></u>Hell",  // Should really be "<p><input>H<u>e</u>ll"
-                                               "o",
-                                               " <b>world",
-                                               "</b> how",
-                                               " <u>are",
-                                               " you",
-                                               "</u>?",
-                                               "",  // end of sentence
-                                               "</p>\n"};
-
-  CHECK(AsTokens(response.source) == restored_tokens_str);
+  CHECK_THROWS_WITH(
+      html.Restore(response),
+      "Response object does not contain alignments. TranslationModel or ResponseOptions is misconfigured?");
 }
 
-TEST_CASE("Test reconstruction of multiple sentences") {
-  std::string input("<p>This <em>is a sentence. And so is</em> this.</p>\n");
-
+TEST_CASE("Do not abort if the input is just empty") {
+  std::string input("");
   HTML html(std::move(input), true);
-  CHECK(input == "This is a sentence. And so is this.\n");
+  CHECK(input == "");
 
   Response response;
-  response.source = AnnotatedText(std::move(input));
-
-  RecordSentenceFromByteRange(response.source, {
-                                                   ByteRange{0, 4},    // 0.0 "This"
-                                                   ByteRange{4, 7},    // 0.1 " is"
-                                                   ByteRange{7, 9},    // 0.2 " a"
-                                                   ByteRange{9, 18},   // 0.3 " sentence"
-                                                   ByteRange{18, 19},  // 0.4 "."
-                                               });
-
-  RecordSentenceFromByteRange(response.source, {
-                                                   ByteRange{20, 23},  // 1.0 "And"
-                                                   ByteRange{23, 26},  // 1.1 " so"
-                                                   ByteRange{26, 29},  // 1.2 " is"
-                                                   ByteRange{29, 34},  // 1.3 " this"
-                                                   ByteRange{34, 35},  // 1.4 "."
-                                               });
-
-  std::vector<std::string> tokens{"",    "This", " is", " a",    " sentence", ".", " ",
-                                  "And", " so",  " is", " this", ".",         "\n"};
-
-  CHECK(AsTokens(response.source) == tokens);
-
   html.Restore(response);
+  CHECK(response.source.text == "");
+  CHECK(response.target.text == "");
+}
 
-  std::vector<std::string> html_tokens{
-      "",       "<p>This", " <em>is", " a", " sentence", ".", " ", "And", " so", " is", "</em> this", ".",
-      "</p>\n",  // </p> got moved into post-sentence gap
-  };
+TEST_CASE("Do not abort if the input is just empty element") {
+  std::string input("<p></p>");
+  HTML html(std::move(input), true);
+  CHECK(input == "");
 
-  CHECK(AsTokens(response.source) == html_tokens);
+  Response response;
+  html.Restore(response);
+  CHECK(response.source.text == "<p></p>");
+  CHECK(response.target.text == "");  // Should be <p></p> but hey not there yet.
 }
 
 TEST_CASE("Test case html entities") {
@@ -170,36 +138,6 @@ TEST_CASE("Test case html entities") {
   std::string input("<p data-attr=\"&quot;&apos;\">This is a sentence &lt;with&gt; named &amp; entities</p>\n");
   HTML html(std::move(input), true);
   CHECK(input == "This is a sentence <with> named & entities\n");
-
-  Response response;
-  response.source = AnnotatedText(std::move(input));
-
-  RecordSentenceFromByteRange(response.source, {
-                                                   ByteRange{0, 4},    // 0.0 "This"
-                                                   ByteRange{4, 7},    // 0.1 " is"
-                                                   ByteRange{7, 9},    // 0.2 " a"
-                                                   ByteRange{9, 18},   // 0.3 " sentence"
-                                                   ByteRange{18, 20},  // 0.4 " <"
-                                                   ByteRange{20, 24},  // 0.5 "with"
-                                                   ByteRange{24, 25},  // 0.6 ">"
-                                                   ByteRange{25, 31},  // 0.7 " named"
-                                                   ByteRange{31, 33},  // 0.8 " &"
-                                                   ByteRange{33, 42},  // 0.9 " entities"
-                                                   ByteRange{42, 42}   // 0.10 ""
-                                               });
-
-  html.Restore(response);
-
-  std::vector<std::string> html_tokens{"",          "<p data-attr=\"&quot;&apos;\">This",
-                                       " is",       " a",
-                                       " sentence",
-                                       " &lt;",  // Oh trouble! The < is completely 'consumed'
-                                       "with",      "&gt;",
-                                       " named",    " &amp;",
-                                       " entities", "",
-                                       "</p>\n"};
-
-  CHECK(AsTokens(response.source) == html_tokens);
 }
 
 TEST_CASE("Test self-closing tags should be treated as spaces") {
@@ -233,6 +171,7 @@ TEST_CASE("Test reconstruction of target sentence") {
   Response response;
   response.source = source;
   response.target = target;
+  response.alignments = {identity_matrix<float>(4)};
 
   html.Restore(response);
 
@@ -274,6 +213,7 @@ TEST_CASE("Test reconstruction of target sentence with entities") {
   Response response;
   response.source = source;
   response.target = target;
+  response.alignments = {identity_matrix<float>(7)};
 
   html.Restore(response);
 
@@ -360,6 +300,7 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
   Response response;
   response.source = source;
   response.target = target;
+  response.alignments = {identity_matrix<float>(5), identity_matrix<float>(10), identity_matrix<float>(5)};
   html.Restore(response);
 
   std::vector<std::string> html_tokens_source{"",
@@ -414,9 +355,11 @@ TEST_CASE("Test empty self-closing pair at end of input in parent") {
 }
 
 TEST_CASE("Test empty tag", "[!mayfail]") {
-  std::string input(
+  std::string test_str(
       "<p id=\"1\">hello <img id=\"1.1\"><span id=\"1.2\"><u id=\"1.2.1\"></u><b id=\"1.2.2\"></b><img "
       "id=\"1.2.3\">world</span></p>\n");
+
+  std::string input(test_str);
   HTML html(std::move(input), true);
   CHECK(input == "hello world\n");
 
@@ -432,10 +375,14 @@ TEST_CASE("Test empty tag", "[!mayfail]") {
   response.source.appendSentence("", sentence.begin(), sentence.end());
   response.source.appendEndingWhitespace("\n");
 
+  response.target.appendSentence("", sentence.begin(), sentence.end());
+  response.target.appendEndingWhitespace("\n");
+
+  response.alignments = {identity_matrix<float>(4)};
+
   html.Restore(response);
-  CHECK(response.source.text ==
-        "<p id=\"1\">hello <img id=\"1.1\"><span id=\"1.2\"><u id=\"1.2.1\"></u><b id=\"1.2.2\"></b><img "
-        "id=\"1.2.3\">world</span></p>\n");
+  CHECK(response.source.text == test_str);
+  CHECK(response.target.text == test_str);
 }
 
 TEST_CASE("End-to-end translation") {
diff --git a/src/tests/units/xh_scanner_tests.cpp b/src/tests/units/xh_scanner_tests.cpp
new file mode 100644
index 000000000..f01e47ddc
--- /dev/null
+++ b/src/tests/units/xh_scanner_tests.cpp
@@ -0,0 +1,125 @@
+#include <string>
+
+#include "catch.hpp"
+#include "translator/xh_scanner.h"
+
+TEST_CASE("scan element with attributes") {
+  markup::instream in("<div id=\"test\" class=\"a b c \" hidden>");
+  markup::scanner scanner(in);
+
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
+  CHECK(scanner.tag_name() == "div");
+  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
+  CHECK(scanner.attr_name() == "id");
+  CHECK(scanner.value() == "test");
+
+  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
+  CHECK(scanner.attr_name() == "class");
+  CHECK(scanner.value() == "a b c ");
+
+  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
+  CHECK(scanner.attr_name() == "hidden");
+  CHECK(scanner.value() == "");
+
+  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+}
+
+TEST_CASE("scan element with text") {
+  markup::instream in("<span>Hello world</span>");
+  markup::scanner scanner(in);
+
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == "Hello world");
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
+  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+}
+
+TEST_CASE("scan html entities") {
+  markup::instream in("Hello &amp; &apos;world&apos;");
+  markup::scanner scanner(in);
+
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == "Hello ");
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == "&");
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == " ");
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == "'");
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == "world");
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == "'");
+  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+}
+
+TEST_CASE("scan nested elements") {
+  markup::instream in("<div><p><img></p></div>");
+  markup::scanner scanner(in);
+
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
+  CHECK(scanner.tag_name() == "div");
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
+  CHECK(scanner.tag_name() == "p");
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
+  CHECK(scanner.tag_name() == "img");
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
+  CHECK(scanner.tag_name() == "p");
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
+  CHECK(scanner.tag_name() == "div");
+  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+}
+
+TEST_CASE("scan kitchen sink") {
+  std::string html_str =
+      "<div id=\"test-id\" class=\"a b c \">\n"
+      "<span x-custom-attribute=\"Hello &quot;world&quot;\"><!--\n"
+      "this is a comment -->this is &amp; text\n"
+      "</span></div>";
+  markup::instream in(html_str.data());
+  markup::scanner scanner(in);
+
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
+  CHECK(scanner.tag_name() == "div");
+  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
+  CHECK(scanner.attr_name() == "id");
+  CHECK(scanner.value() == "test-id");
+  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
+  CHECK(scanner.attr_name() == "class");
+  CHECK(scanner.value() == "a b c ");
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == "\n");
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
+  CHECK(scanner.tag_name() == "span");
+  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
+  CHECK(scanner.attr_name() == "x-custom-attribute");
+  CHECK(scanner.value() == "Hello &quot;world&quot;");  // We do not decode entities in attributes
+  CHECK(scanner.next_token() == markup::scanner::TT_COMMENT_START);
+  CHECK(scanner.next_token() == markup::scanner::TT_DATA);
+  CHECK(scanner.value() == "\nthis is a comment ");
+  CHECK(scanner.next_token() == markup::scanner::TT_COMMENT_END);
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == "this is ");
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == "&");
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == " text\n");
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
+  CHECK(scanner.tag_name() == "span");
+  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
+  CHECK(scanner.tag_name() == "div");
+  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+}
+
+TEST_CASE("test long text (#273)") {
+  std::string test_str;
+  for (size_t i = 0; i < 1024; ++i) test_str.append("testing ");
+
+  markup::instream in(test_str.data());
+  markup::scanner scanner(in);
+
+  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.value() == test_str);
+  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+}
\ No newline at end of file
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index 348b65ca3..efe7969f6 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -83,7 +83,7 @@ std::string format(std::string const &format_str, Arg arg, Args... args) {
   return os.str();
 }
 
-bool IsBlockElement(std::string const &name) {
+bool IsBlockElement(std::string_view const &name) {
   // List of elements that we expect might occur inside words, and that should
   // not introduce spacings around them. Not strictly inline elements, nor flow
   // elements. See also https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Content_categories
@@ -91,16 +91,16 @@ bool IsBlockElement(std::string const &name) {
       "abbr",  "a",    "b",      "em",  "i",   "kbd",  "mark", "math", "output", "q",   "ruby",
       "small", "span", "strong", "sub", "sup", "time", "u",    "var",  "wbr",    "ins", "del"};
 
-  return inline_ish_elements.find(name) == inline_ish_elements.end();
+  return inline_ish_elements.find(std::string(name)) == inline_ish_elements.end();
 }
 
-bool IsEmptyElement(std::string const &name) {
+bool IsEmptyElement(std::string_view const &name) {
   // List of elements for which we do not expect a closing tag, or self-closing
   // elements in XHTML. See also https://developer.mozilla.org/en-US/docs/Glossary/Empty_element
   static std::unordered_set<std::string> empty_elements{"area",  "base", "br",   "col",   "embed",  "hr",    "img",
                                                         "input", "link", "meta", "param", "source", "track", "wbr"};
 
-  return empty_elements.find(name) != empty_elements.end();
+  return empty_elements.find(std::string(name)) != empty_elements.end();
 }
 
 void DiffTags(HTML::Taint const &prev, HTML::Taint const &curr, HTML::Taint &opening, HTML::Taint &closing) {
@@ -171,19 +171,27 @@ AnnotatedText Apply(AnnotatedText const &in, Fun fun) {
 bool IsContinuation(string_view str) { return !str.empty() && str.compare(0, 1, " ", 1) != 0; }
 
 bool HasAlignments(Response const &response) {
-  return !response.alignments.empty() && !response.alignments[0][0].empty();
+  // Test for each sentence individually as a sentence may be empty (or there)
+  // might be no sentences, so just testing for alignments.empty() would not be
+  // sufficient.
+  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx)
+    if (response.alignments.size() <= sentenceIdx ||
+        response.alignments[sentenceIdx].size() != response.target.numWords(sentenceIdx))
+      return false;
+  return true;
 }
 
 void HardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
   // For each sentence...
   for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
     alignments.emplace_back();
-    assert(response.alignments[sentenceIdx].size() == response.target.numWords(sentenceIdx));
 
     // Hard-align: find for each target token the most prevalent source token
-    for (size_t t = 0; t < response.alignments[sentenceIdx].size(); ++t) {
+    // Note: only search from 0 to N-1 because token N is end-of-sentence token
+    // that can only align with the end-of-sentence token of the target
+    for (size_t t = 0; t + 1 < response.target.numWords(sentenceIdx); ++t) {
       size_t s_max = 0;
-      for (size_t s = 1; s < response.alignments[sentenceIdx][t].size(); ++s) {
+      for (size_t s = 1; s + 1 < response.source.numWords(sentenceIdx); ++s) {
         if (response.alignments[sentenceIdx][t][s] > response.alignments[sentenceIdx][t][s_max]) {
           s_max = s;
         }
@@ -193,10 +201,10 @@ void HardAlignments(Response const &response, std::vector<std::vector<size_t>> &
     }
 
     // Next, we try to smooth out these selected alignments with a few heuristics
-    for (size_t t = 0; t < response.target.numWords(sentenceIdx); ++t) {
+    for (size_t t = 1; t + 1 < response.target.numWords(sentenceIdx); ++t) {
       // If this token is a continuation of a previous token, pick the tags from the most
       // prevalent token for the whole word.
-      if (t > 0 && IsContinuation(response.target.word(sentenceIdx, t))) {
+      if (IsContinuation(response.target.word(sentenceIdx, t))) {
         // Note: only looking at the previous token since that will already
         // have this treatment applied to it.
         size_t s_curr = alignments.back()[t];
@@ -204,30 +212,22 @@ void HardAlignments(Response const &response, std::vector<std::vector<size_t>> &
         float score_curr = response.alignments[sentenceIdx][t][s_curr];
         float score_prev = response.alignments[sentenceIdx][t - 1][s_prev];
 
-        size_t s_max = score_curr > score_prev ? s_curr : s_prev;
+        if (score_curr > score_prev) {
+          // Apply this to all previous tokens in the word
+          for (size_t i = t;; --i) {
+            alignments.back()[i] = s_curr;
 
-        // Apply this to all previous tokens in the word
-        for (size_t i = t;; --i) {
-          alignments.back()[i] = s_max;
-
-          // Stop if this was the first token or the beginning of the word
-          if (i == 0 || !IsContinuation(response.target.word(sentenceIdx, i))) break;
+            // Stop if this was the first token or the beginning of the word
+            if (i == 0 || !IsContinuation(response.target.word(sentenceIdx, i))) break;
+          }
+        } else {
+          alignments.back()[t] = s_prev;
         }
       }
     }
-  }
-}
-
-void InterpolateAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
-  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
-    alignments.emplace_back();
-    double ratio = (double)response.source.numWords(sentenceIdx) / response.target.numWords(sentenceIdx);
 
-    for (size_t wordIdx = 0; wordIdx < response.target.numWords(sentenceIdx); ++wordIdx) {
-      size_t source_token_idx = static_cast<size_t>(ratio * wordIdx);
-      assert(source_token_idx < response.source.numWords(sentenceIdx));
-      alignments.back().push_back(source_token_idx);
-    }
+    // Always align target end with source end
+    alignments.back().push_back(response.source.numWords(sentenceIdx) - 1);
   }
 }
 
@@ -446,7 +446,7 @@ HTML::HTML(std::string &&source, bool process_markup) {
 
   bool stop = false;
   while (!stop) {
-    switch (scanner.get_token()) {
+    switch (scanner.next_token()) {
       case markup::scanner::TT_ERROR:
         throw BadHTML("HTML parse error");
 
@@ -456,7 +456,7 @@ HTML::HTML(std::string &&source, bool process_markup) {
 
       case markup::scanner::TT_TEXT: {
         auto begin = source.size();
-        source.append(scanner.get_value());
+        source.append(scanner.value());
         spans_.push_back(Span{begin, source.size(), stack});
         FilterEmpty(stack);
       } break;
@@ -466,12 +466,15 @@ HTML::HTML(std::string &&source, bool process_markup) {
         // <br>, <img>, <li>) make sure it does so in this text as well.
         // TODO: Strong assumption here that the language uses spaces to
         // separate words
-        if (IsBlockElement(scanner.get_tag_name()) && !source.empty() && source.back() != ' ') source.push_back(' ');
+        if (IsBlockElement(scanner.tag_name()) && !source.empty() && source.back() != ' ') source.push_back(' ');
 
-        tag = new Tag{scanner.get_tag_name(), std::string(), IsEmptyElement(scanner.get_tag_name())};
-        pool_.emplace_back(tag);  // pool_ takes ownership of our tag
+        // pool_ takes ownership of our tag, makes sure it's freed when necessary
+        pool_.emplace_back(new Tag{std::string(scanner.tag_name()), std::string(), IsEmptyElement(scanner.tag_name())});
 
-        stack.push_back(pool_.back().get());
+        // Tag *tag is used by attribute parsing
+        tag = pool_.back().get();
+
+        stack.push_back(tag);
 
         // Empty elements (e.g. <img>) are not applicable to a span of text
         // so instead we "apply" them to an empty span in between, and then
@@ -480,21 +483,20 @@ HTML::HTML(std::string &&source, bool process_markup) {
           spans_.push_back(Span{source.size(), source.size(), stack});
           stack.pop_back();
         }
-
         break;
 
       case markup::scanner::TT_TAG_END:
         // Note: self-closing tags emit TT_TAG_END immediately after TT_TAG_START
         // but since we're parsing HTML5, a sole <img> will never emit a TT_TAG_END
         if (stack.empty())
-          throw BadHTML(format("Encountered more closing tags ({}) than opening tags", scanner.get_tag_name()));
+          throw BadHTML(format("Encountered more closing tags ({}) than opening tags", scanner.tag_name()));
 
-        if (stack.back()->name != scanner.get_tag_name())
-          throw BadHTML(format("Encountered unexpected closing tag </{}>, stack is {}", scanner.get_tag_name(), stack));
+        if (stack.back()->name != scanner.tag_name())
+          throw BadHTML(format("Encountered unexpected closing tag </{}>, stack is {}", scanner.tag_name(), stack));
 
         // What to do with "<u></u>" case, where tag is immediately closed
         // so it never makes it into the taint of any of the spans? This adds
-        // an empty span so it still lives.
+        // an empty span so it still gets recorded in spans_.
         if (spans_.empty() || !ContainsTag(spans_.back().tags, stack.back()))
           spans_.push_back(Span{source.size(), source.size(), stack});
 
@@ -502,7 +504,8 @@ HTML::HTML(std::string &&source, bool process_markup) {
         break;
 
       case markup::scanner::TT_ATTR:
-        tag->attributes += format(" {}=\"{}\"", scanner.get_attr_name(), scanner.get_value());
+        assert(tag != nullptr);
+        tag->attributes += format(" {}=\"{}\"", scanner.attr_name(), scanner.value());
         break;
 
       default:
@@ -517,8 +520,15 @@ HTML::HTML(std::string &&source, bool process_markup) {
 }
 
 void HTML::Restore(Response &response) {
+  // No-op if process_markup was false (and thus spans_ is empty)
+  // TODO: replace this with optional<HTML> at a higher level
   if (spans_.empty()) return;
 
+  // We need alignment info to transfer the HTML tags from the input to the
+  // translation. If those are not available, no HTML in translations for you.
+  ABORT_UNLESS(HasAlignments(response),
+               "Response object does not contain alignments. TranslationModel or ResponseOptions is misconfigured?");
+
   // Reconstruction of HTML tags:
   // 1. Map each token to a Span
   // 2. Apply the taint of that span to the token
@@ -535,17 +545,7 @@ void HTML::Restore(Response &response) {
 
   // Find for every token in target the token in source that best matches.
   std::vector<std::vector<size_t>> alignments;
-
-  // If we do have alignment information from the model, we use that to taint
-  // tokens with the tags from their source token counterpart. If there is no
-  // alignment information available, we just interpolate based on sentence
-  // length (badly).
-  if (HasAlignments(response)) {
-    // DebugPrintAlignmentScores(std::cerr, response);
-    HardAlignments(response, alignments);
-  } else {
-    InterpolateAlignments(response, alignments);
-  }
+  HardAlignments(response, alignments);
 
   std::vector<Taint> token_tags_target;
   token_tags_target.emplace_back();  // add empty one to the beginning for easy
diff --git a/src/translator/parser.cpp b/src/translator/parser.cpp
index 2295fd6c9..e875d97e0 100644
--- a/src/translator/parser.cpp
+++ b/src/translator/parser.cpp
@@ -90,6 +90,8 @@ void ConfigParser::addOptionsBoundToConfig(CLI::App &app, CLIConfig &config) {
   app_.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
   app_.add_option("--cache-mutex-buckets", config.cacheMutexBuckets,
                   "Number of mutex buckets to control locking granularity");
+
+  app_.add_flag("--html", config.html, "Whether input and output should be HTML");
 }
 
 std::shared_ptr<marian::Options> parseOptionsFromFilePath(const std::string &configPath, bool validate /*= true*/) {
diff --git a/src/translator/parser.h b/src/translator/parser.h
index 80006f3b0..1aff5dba7 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -36,6 +36,7 @@ struct CLIConfig {
   ModelConfigPaths modelConfigPaths;
   bool byteArray;
   bool validateByteArray;
+  bool html;
   size_t numWorkers;
   OpMode opMode;
 
diff --git a/src/translator/xh_scanner.cpp b/src/translator/xh_scanner.cpp
index 78ae13526..bb72f8020 100644
--- a/src/translator/xh_scanner.cpp
+++ b/src/translator/xh_scanner.cpp
@@ -3,349 +3,317 @@
 
 #include "xh_scanner.h"
 
+#include <cassert>
 #include <cctype>
 #include <cstring>
 
-namespace markup {
+namespace {
 
-// case sensitive string equality test
-// s_lowcase shall be lowercase string
-inline bool equal(const char *s, const char *s1, size_t length) { return strncmp(s, s1, length) == 0; }
+// Simple replacement for str.ends_with(compile-time C string)
+template <typename Char_t, size_t Len>
+inline bool ends_with(markup::string_ref &str, const Char_t (&suffix)[Len]) {
+  size_t offset = str.size - (Len - 1);
+  return offset <= str.size && std::memcmp(str.data + offset, suffix, Len - 1) == 0;
+}
 
-const char *scanner::get_value() {
-  value[value_length] = 0;
-  return value;
+inline bool equals_case_insensitive(const char *lhs, const char *rhs, size_t len) {
+  for (size_t i = 0; i < len; ++i) {
+    // cast to unsigned char otherwise std::tolower has undefined behaviour
+    if (std::tolower(static_cast<unsigned char>(lhs[i])) != std::tolower(static_cast<unsigned char>(rhs[i])))
+      return false;
+  }
+
+  return true;
 }
 
-const char *scanner::get_attr_name() {
-  attr_name[attr_name_length] = 0;
-  return attr_name;
+// Alias for the above, but with compile-time known C string
+template <size_t Len>
+inline bool equals_case_insensitive(markup::string_ref &lhs, const char (&rhs)[Len]) {
+  return lhs.size == Len - 1 && equals_case_insensitive(lhs.data, rhs, Len);
 }
 
-const char *scanner::get_tag_name() {
-  tag_name[tag_name_length] = 0;
-  return tag_name;
+template <typename Char_t, size_t Len>
+bool operator==(markup::string_ref const &str, const Char_t (&str2)[Len]) {
+  return str.size == Len - 1 && std::memcmp(str.data, str2, Len - 1) == 0;
 }
 
-scanner::token_type scanner::scan_body() {
-  text_begin = input.p;
-  if (input_char) {
-    --text_begin;
-  }
-  text_end = text_begin;
-  value_length = 0;
-  char c = get_char();
+}  // end namespace
 
-  if (c == 0)
-    return TT_EOF;
-  else if (c == '<')
-    return scan_tag();
-  else if (c == '&')
-    return scan_entity();
+namespace markup {
 
-  while (true) {
-    append_value(c);
-    ++text_end;
+// case sensitive string equality test
+// s_lowcase shall be lowercase string
+std::string_view scanner::value() const { return std::string_view(value_.data, value_.size); }
 
-    c = get_char();
+std::string_view scanner::attr_name() const { return std::string_view(attr_name_.data, attr_name_.size); }
 
-    if (c == 0) {
-      push_back(c);
-      break;
-    }
-    if (c == '<') {
-      push_back(c);
-      break;
-    }
-    if (c == '&') {
-      push_back(c);
-      break;
-    }
+std::string_view scanner::tag_name() const { return std::string_view(tag_name_.data, tag_name_.size); }
+
+scanner::token_type scanner::scan_body() {
+  value_ = string_ref{input_.pos(), 0};
+
+  switch (input_.peek()) {
+    case '\0':
+      return TT_EOF;
+    case '<':
+      return scan_tag();
+    case '&':
+      return scan_entity(TT_TEXT);
   }
-  return TT_TEXT;
-}
 
-scanner::token_type scanner::scan_head() {
-  char c = skip_whitespace();
-
-  if (c == '>') {
-    if (equal(tag_name, "script", 6)) {
-      // script is special because we want to parse the attributes,
-      // but not the content
-      c_scan = &scanner::scan_special;
-      return scan_special();
-    } else if (equal(tag_name, "style", 5)) {
-      // same with style
-      c_scan = &scanner::scan_special;
-      return scan_special();
+  while (true) {
+    switch (input_.peek()) {
+      case '\0':
+      case '<':
+      case '&':
+        return TT_TEXT;
+      default:
+        input_.consume();
+        ++value_.size;
+        break;
     }
-    c_scan = &scanner::scan_body;
-    return scan_body();
   }
-  if (c == '/') {
-    char t = get_char();
-    if (t == '>') {
-      // self closing tag
-      c_scan = &scanner::scan_body;
-      return TT_TAG_END;
-    } else {
-      push_back(t);
-      return TT_ERROR;
-    }  // erroneous situtation - standalone '/'
+}
+
+// Consumes one or closing bit of a tag:
+//   <tag attr="value">...</tag>
+//       |------------|
+// Followed by:
+// - scan_special if <script> or <style>
+// - scan_body
+// - another scan_head for the next attribute or end of open tag
+// Returns:
+// - TT_ATTR if attribute is read
+// - TT_TAG_END if self-closing tag
+// - TT_ERROR if wrong character encountered
+// - TT_EOF if unexpected end of input (will not return TT_ATTR if attribute value wasn't finished yet)
+// - TT_TAG_END through scan_special
+// - TT_TEXT through scan_body
+scanner::token_type scanner::scan_attr() {
+  // Skip all whitespace between tag name or last attribute and next attribute or '>'
+  skip_whitespace();
+
+  // Find end of tag name
+  switch (input_.peek()) {
+    case '>':
+      input_.consume();
+      if (equals_case_insensitive(tag_name_, "script")) {
+        // script is special because we want to parse the attributes,
+        // but not the content
+        c_scan = &scanner::scan_special;
+        return scan_special();
+      } else if (equals_case_insensitive(tag_name_, "style")) {
+        // same with style
+        c_scan = &scanner::scan_special;
+        return scan_special();
+      } else {
+        c_scan = &scanner::scan_body;
+        return scan_body();
+      }
+    case '/':
+      input_.consume();
+      if (input_.peek() == '>') {
+        // self closing tag
+        input_.consume();
+        c_scan = &scanner::scan_body;
+        return TT_TAG_END;
+      } else {
+        return TT_ERROR;
+      }
   }
 
-  attr_name_length = 0;
-  value_length = 0;
+  attr_name_ = string_ref{input_.pos(), 0};
+  value_ = string_ref{nullptr, 0};
 
   // attribute name...
-  while (c != '=') {
-    if (c == 0) return TT_EOF;
-    if (c == '>') {
-      push_back(c);
-      return TT_ATTR;
-    }  // attribute without value (HTML style)
-    if (is_whitespace(c)) {
-      c = skip_whitespace();
-      if (c != '=') {
-        push_back(c);
-        return TT_ATTR;
-      }  // attribute without value (HTML style)
-      else
+  while (input_.peek() != '=') {
+    switch (input_.peek()) {
+      case '\0':
+        return TT_EOF;
+      case '>':
+        return TT_ATTR;  // attribute without value (HTML style)
+      case '<':
+        return TT_ERROR;
+      default:
+        if (skip_whitespace()) continue;
+        input_.consume();
+        ++attr_name_.size;
         break;
     }
-    if (c == '<') return TT_ERROR;
-    append_attr_name(c);
-    c = get_char();
   }
 
-  c = skip_whitespace();
+  // consume '=' and any following whitespace
+  input_.consume();
+  skip_whitespace();
   // attribute value...
 
-  if (c == '\"') {
-    c = get_char();
-    while (c) {
-      if (c == '\"') return TT_ATTR;
-      // if (c == '&') c = scan_entity();
-      append_value(c);
-      c = get_char();
-    }
-  } else if (c == '\'')  // allowed in html
-  {
-    c = get_char();
-    while (c) {
-      if (c == '\'') return TT_ATTR;
-      // if (c == '&') c = scan_entity();
-      append_value(c);
-      c = get_char();
-    }
-  } else  // scan token, allowed in html: e.g. align=center
-  {
-    c = get_char();
-    do {
-      if (is_whitespace(c)) return TT_ATTR;
-      /* these two removed in favour of better html support:
-      if( c == '/' || c == '>' ) { push_back(c); return TT_ATTR; }
-      if( c == '&' ) c = scan_entity();*/
-      if (c == '>') {
-        push_back(c);
-        return TT_ATTR;
+  char quote;  // Either '"' or '\'' depending on which quote we're searching for
+  switch (input_.peek()) {
+    case '"':
+    case '\'':
+      quote = input_.consume();
+      value_ = string_ref{input_.pos(), 0};
+      while (true) {
+        if (input_.peek() == '\0') {
+          return TT_ERROR;
+        } else if (input_.peek() == quote) {
+          input_.consume();
+          return TT_ATTR;
+        } else {
+          input_.consume();
+          ++value_.size;
+        }
+      }
+      break;
+    default:
+      value_ = string_ref{input_.pos(), 0};
+
+      while (true) {
+        if (is_whitespace(input_.peek())) return TT_ATTR;
+        if (input_.peek() == '>') return TT_ATTR;  // '>' will be consumed next round
+        input_.consume();
+        ++value_.size;
       }
-      append_value(c);
-      c = get_char();
-    } while (c);
+      break;
   }
 
+  // How did we end up here?!
   return TT_ERROR;
 }
 
-// caller already consumed '<'
-// scan header start or tag tail
+// scans tag name of open or closing tag
+//   <tag attr="value">...</tag>
+//   |--|                 |----|
+// Emits:
+// - TT_TAG_START if tag head is read
+// - TT_COMMENT_START
+// - TT_CDATA_START
+// - TT_ENTITY_START
+// - TT_ERROR if unexpected character or end
 scanner::token_type scanner::scan_tag() {
-  tag_name_length = 0;
+  if (input_.consume() != '<') return TT_ERROR;
 
-  char c = get_char();
+  bool is_tail = input_.peek() == '/';
+  if (is_tail) input_.consume();
 
-  bool is_tail = c == '/';
-  if (is_tail) c = get_char();
+  tag_name_ = string_ref{input_.pos(), 0};
 
-  while (c) {
-    if (is_whitespace(c)) {
-      c = skip_whitespace();
-      break;
-    }
-    if (c == '/' || c == '>') break;
-    append_tag_name(c);
-
-    switch (tag_name_length) {
-      case 3:
-        if (equal(tag_name, "!--", 3)) {
-          c_scan = &scanner::scan_comment;
-          return TT_COMMENT_START;
-        }
-        break;
-      case 8:
-        if (equal(tag_name, "![CDATA[", 8)) {
-          c_scan = &scanner::scan_cdata;
-          return TT_CDATA_START;
-        }
-        break;
-      case 7:
-        if (equal(tag_name, "!ENTITY", 8)) {
-          c_scan = &scanner::scan_entity_decl;
-          return TT_ENTITY_START;
-        }
-        break;
-    }
+  while (input_.peek()) {
+    if (skip_whitespace()) break;
+
+    if (input_.peek() == '/' || input_.peek() == '>') break;
+
+    input_.consume();
+    ++tag_name_.size;
 
-    c = get_char();
+    if (tag_name_ == "!--") {
+      c_scan = &scanner::scan_comment;
+      return TT_COMMENT_START;
+    }
   }
 
-  if (c == 0) return TT_ERROR;
+  if (!input_.peek()) return TT_EOF;
 
-  if (is_tail) {
-    if (c == '>') return TT_TAG_END;
-    return TT_ERROR;
-  } else
-    push_back(c);
+  if (is_tail) return input_.consume() == '>' ? TT_TAG_END : TT_ERROR;
 
-  c_scan = &scanner::scan_head;
+  c_scan = &scanner::scan_attr;
   return TT_TAG_START;
 }
 
-scanner::token_type scanner::scan_entity() {
-  // note that when scan_entity() is called, & is already consumed.
-
-  char buffer[8];
-  unsigned int buflen = 0;
-  buffer[buflen++] = '&';  // (just makes resolve_entity and append_value(buffer) easier)
-
+scanner::token_type scanner::scan_entity(token_type parent_token_type) {
+  // `entity` includes starting '&' and ending ';'
+  string_ref entity{input_.pos(), 0};
   bool has_end = false;
 
-  while (true) {
-    char c = get_char();
-    buffer[buflen++] = c;
-
-    // Found end of entity
-    if (c == ';') break;
+  if (input_.consume() != '&') return TT_ERROR;
 
-    // Too long to be entity
-    if (buflen == sizeof(buffer)) break;
+  ++entity.size;  // Account for the consumed '&'
 
-    // Not a character we'd expect in an entity (esp '&' or '<')
-    if (!isalpha(c)) break;
+  // Consume the entity
+  while (input_.peek()) {
+    if (input_.peek() == ';') {
+      input_.consume();
+      ++entity.size;
+      has_end = true;
+      break;
+    } else if (!isalpha(input_.peek())) {
+      has_end = false;
+      break;
+    } else {
+      input_.consume();
+      ++entity.size;
+    }
   }
 
-  // Keep the text_end that scanner::scan_body uses similarly up-to-date. Since
-  // scan_entity() is only called from scan_body we assume text_begin is already
-  // set correctly by it.
-  text_end += buflen;
-
-  // If we found the end of the entity, and we can identify it, then
-  // resolve_entity() will emit the char it encoded.
-  if (buffer[buflen - 1] == ';' && resolve_entity(buffer, buflen)) {
-    return TT_TEXT;
-  }
+  // If we can decode the entity, do so
+  if (has_end && resolve_entity(entity, value_)) return parent_token_type;
 
-  // Otherwise, we just emit whatever we read as text, except for the last
-  // character that caused us to break. That may be another &, or a <, which we
-  // would want to scan properly.
-  for (unsigned int i = 0; i < buflen - 1; ++i) append_value(buffer[i]);
-  push_back(buffer[buflen - 1]);
-  --text_end;  // because push_back()
-  return TT_TEXT;
+  // Otherwise, just yield the whole thing undecoded, interpret it as text
+  value_ = entity;
+  return parent_token_type;
 }
 
-bool scanner::resolve_entity(char *buffer, unsigned int len) {
-  switch (len) {
-    case 4:
-      if (equal(buffer, "&lt;", 4)) {
-        append_value('<');
-        return true;
-      }
-      if (equal(buffer, "&gt;", 4)) {
-        append_value('>');
-        return true;
-      }
-      break;
+bool scanner::resolve_entity(string_ref const &buffer, string_ref &decoded) const {
+  static char lt = '<', gt = '>', amp = '&', quot = '"', apos = '\'', nbsp = ' ';
 
-    case 5:
-      if (equal(buffer, "&amp;", 5)) {
-        append_value('&');
-        return true;
-      }
-      break;
-
-    case 6:
-      if (equal(buffer, "&quot;", 6)) {
-        append_value('"');
-        return true;
-      }
-      if (equal(buffer, "&apos;", 6)) {
-        append_value('\'');
-        return true;
-      }
-      if (equal(buffer, "&nbsp;", 6)) {
-        append_value(' ');  // TODO: handle non-breaking spaces better than just converting them to spaces
-        return true;
-      }
-      break;
+  if (buffer == "&lt;") {
+    decoded = string_ref{&lt, 1};
+    return true;
+  }
+  if (buffer == "&gt;") {
+    decoded = string_ref{&gt, 1};
+    return true;
+  }
+  if (buffer == "&amp;") {
+    decoded = string_ref{&amp, 1};
+    return true;
+  }
+  if (buffer == "&quot;") {
+    decoded = string_ref{&quot, 1};
+    return true;
+  }
+  if (buffer == "&apos;") {
+    decoded = string_ref{&apos, 1};
+    return true;
+  }
+  if (buffer == "&nbsp;") {
+    decoded = string_ref{&nbsp, 1};  // TODO: handle non-breaking spaces better than just converting them to spaces
+    return true;
   }
   return false;
 }
 
 // skip whitespaces.
-// returns first non-whitespace char
-char scanner::skip_whitespace() {
-  while (char c = get_char()) {
-    if (!is_whitespace(c)) return c;
-  }
-  return 0;
-}
-
-void scanner::push_back(char c) { input_char = c; }
-
-char scanner::get_char() {
-  if (input_char) {
-    char t(input_char);
-    input_char = 0;
-    return t;
+// returns how many whitespaces were skipped
+size_t scanner::skip_whitespace() {
+  size_t skipped = 0;
+  while (is_whitespace(input_.peek())) {
+    input_.consume();
+    ++skipped;
   }
-  return input.get_char();
+  return skipped;
 }
 
 bool scanner::is_whitespace(char c) {
   return c <= ' ' && (c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\f');
 }
 
-void scanner::append_value(char c) {
-  if (value_length < (MAX_TOKEN_SIZE - 1)) value[value_length++] = c;
-}
-
-void scanner::append_attr_name(char c) {
-  if (attr_name_length < (MAX_NAME_SIZE - 1)) attr_name[attr_name_length++] = char(c);
-}
-
-void scanner::append_tag_name(char c) {
-  if (tag_name_length < (MAX_NAME_SIZE - 1))
-    tag_name[tag_name_length++] =
-        std::tolower(static_cast<unsigned char>(c));  // cast because std::tolower has undefined behaviour otherwise
-}
-
 scanner::token_type scanner::scan_comment() {
   if (got_tail) {
     c_scan = &scanner::scan_body;
     got_tail = false;
     return TT_COMMENT_END;
   }
-  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
-    char c = get_char();
-    if (c == 0) return TT_EOF;
-    value[value_length] = c;
 
-    if (value_length >= 2 && value[value_length] == '>' && value[value_length - 1] == '-' &&
-        value[value_length - 2] == '-') {
+  value_ = string_ref{input_.pos(), 0};
+
+  while (true) {
+    if (input_.consume() == '\0') return TT_EOF;
+    ++value_.size;
+
+    if (ends_with(value_, "-->")) {
       got_tail = true;
-      value_length -= 2;
+      value_.size -= 3;
       break;
     }
   }
@@ -358,96 +326,30 @@ scanner::token_type scanner::scan_special() {
     got_tail = false;
     return TT_TAG_END;
   }
-  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
-    char c = get_char();
-    if (c == 0) return TT_EOF;
-
-    // in case MAX_TOKEN_SIZE limit breaks up the end tag
-    if (c == '<' && value_length + tag_name_length + 3 >= MAX_TOKEN_SIZE) {
-      push_back(c);
-      break;
-    }
 
-    value[value_length] = c;
-
-    if (c == '>' && value_length >= tag_name_length + 2) {
-      unsigned int i = tag_name_length - 1;
-      do {
-        if (value[value_length + i - tag_name_length] != tag_name[i]) break;
-        --i;
-      } while (i > 0);
-      if (i > 0) continue;
-      if (value[value_length - tag_name_length - 1] != '/') continue;
-      if (value[value_length - tag_name_length - 2] != '<') continue;
-
-      got_tail = true;
-      value_length = value_length - tag_name_length - 2;
-      break;
-    }
-  }
-  return TT_DATA;
-}
+  value_ = string_ref{input_.pos(), 0};
 
-scanner::token_type scanner::scan_cdata() {
-  if (got_tail) {
-    c_scan = &scanner::scan_body;
-    got_tail = false;
-    return TT_CDATA_END;
-  }
-  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
-    char c = get_char();
-    if (c == 0) return TT_EOF;
-    value[value_length] = c;
+  while (true) {
+    if (input_.consume() == '\0') return TT_EOF;
+    ++value_.size;
 
-    if (value_length >= 2 && value[value_length] == '>' && value[value_length - 1] == ']' &&
-        value[value_length - 2] == ']') {
-      got_tail = true;
-      value_length -= 2;
-      break;
-    }
-  }
-  return TT_DATA;
-}
+    // Test for </tag>
+    // TODO: no whitespaces allowed? Is that okay?
+    if (value_.data[value_.size - 1] == '>' && value_.size >= tag_name_.size + 3) {
+      // Test for the "</"" bit of "</tag>"
+      size_t pos_tag_start = value_.size - tag_name_.size - 3;
+      if (std::memcmp(value_.data + pos_tag_start, "</", 2) != 0) continue;
 
-scanner::token_type scanner::scan_pi() {
-  if (got_tail) {
-    c_scan = &scanner::scan_body;
-    got_tail = false;
-    return TT_PI_END;
-  }
-  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
-    char c = get_char();
-    if (c == 0) return TT_EOF;
-    value[value_length] = c;
+      // Test for the "tag" bit of "</tag>". Doing case insensitive compare because <I>...</i> is okay.
+      size_t pos_tag_name = value_.size - tag_name_.size - 1;  // end - tag>
+      if (!equals_case_insensitive(value_.data + pos_tag_name, tag_name_.data, tag_name_.size)) continue;
 
-    if (value_length >= 1 && value[value_length] == '>' && value[value_length - 1] == '?') {
       got_tail = true;
-      value_length -= 1;
+      value_.size -= tag_name_.size + 3;
       break;
     }
   }
-  return TT_DATA;
-}
 
-scanner::token_type scanner::scan_entity_decl() {
-  if (got_tail) {
-    c_scan = &scanner::scan_body;
-    got_tail = false;
-    return TT_ENTITY_END;
-  }
-  char t;
-  unsigned int tc = 0;
-  for (value_length = 0; value_length < (MAX_TOKEN_SIZE - 1); ++value_length) {
-    t = get_char();
-    if (t == 0) return TT_EOF;
-    value[value_length] = t;
-    if (t == '\"')
-      tc++;
-    else if (t == '>' && (tc & 1u) == 0) {
-      got_tail = true;
-      break;
-    }
-  }
   return TT_DATA;
 }
 
diff --git a/src/translator/xh_scanner.h b/src/translator/xh_scanner.h
index 0b2dd2be2..de7d762ab 100644
--- a/src/translator/xh_scanner.h
+++ b/src/translator/xh_scanner.h
@@ -5,15 +5,27 @@
 //|
 //| (C) Andrew Fedoniouk @ terrainformatica.com
 //|
-#include <string.h>
+#include <cassert>
+#include <cstring>
+#include <string_view>
 
 namespace markup {
+
 struct instream {
   const char *p;
+  const char *begin;
   const char *end;
-  explicit instream(const char *src) : p(src), end(src + strlen(src)) {}
-  instream(const char *begin, const char *end) : p(begin), end(end) {}
-  char get_char() { return p < end ? *p++ : 0; }
+  explicit instream(const char *src) : p(src), begin(src), end(src + strlen(src)) {}
+  instream(const char *begin, const char *end) : p(begin), begin(begin), end(end) {}
+  char consume() { return p < end ? *p++ : 0; }
+  char peek() const { return p < end ? *p : 0; }
+  const char *pos() const { return p; }
+};
+
+// Think string_view, but with a mutable range
+struct string_ref {
+  const char *data;
+  size_t size;
 };
 
 class scanner {
@@ -37,94 +49,59 @@ class scanner {
 
     TT_COMMENT_START,
     TT_COMMENT_END,  // after "<!--" and "-->"
-    TT_CDATA_START,
-    TT_CDATA_END,  // after "<![CDATA[" and "]]>"
-    TT_PI_START,
-    TT_PI_END,  // after "<?" and "?>"
-    TT_ENTITY_START,
-    TT_ENTITY_END,  // after "<!ENTITY" and ">"
-
   };
 
-  enum $ { MAX_TOKEN_SIZE = 1024, MAX_NAME_SIZE = 128 };
-
  public:
-  explicit scanner(instream &is)
-      : value_length(0), tag_name_length(0), attr_name_length(0), input(is), input_char(0), got_tail(false) {
-    c_scan = &scanner::scan_body;
-  }
+  explicit scanner(instream &is) : input_(is), got_tail(false) { c_scan = &scanner::scan_body; }
 
   // get next token
-  token_type get_token() { return (this->*c_scan)(); }
-
-  // get text span backed by original input.
-  const char *get_text_begin() { return text_begin; }
-  const char *get_text_end() { return text_end; }
+  token_type next_token() { return (this->*c_scan)(); }
 
   // get value of TT_TEXT, TT_ATTR and TT_DATA
-  const char *get_value();
+  std::string_view value() const;
 
   // get attribute name
-  const char *get_attr_name();
+  std::string_view attr_name() const;
 
-  // get tag name (always lowercase)
-  const char *get_tag_name();
+  // get tag name
+  std::string_view tag_name() const;
 
  private: /* methods */
   typedef token_type (scanner::*scan)();
 
   scan c_scan;  // current 'reader'
 
-  // content 'readers'
+  // Consumes the text around and between tags
   token_type scan_body();
 
-  token_type scan_head();
+  // Consumes name="attr"
+  token_type scan_attr();
 
+  // Consumes <!-- ... -->
   token_type scan_comment();
 
-  token_type scan_cdata();
-
+  // Consumes ...</style> and ...</script>
   token_type scan_special();
 
-  token_type scan_pi();
-
+  // Consumes <tagname and </tagname>
   token_type scan_tag();
 
-  token_type scan_entity();
-
-  token_type scan_entity_decl();
+  // Consumes '&amp;' etc, emits parent_token_type
+  token_type scan_entity(token_type parent_token_type);
 
-  char skip_whitespace();
+  size_t skip_whitespace();
 
-  void push_back(char c);
-
-  char get_char();
-
-  bool resolve_entity(char *buffer, unsigned int len);
+  bool resolve_entity(string_ref const &buffer, string_ref &decoded) const;
 
   static bool is_whitespace(char c);
 
-  void append_value(char c);
-
-  void append_attr_name(char c);
-
-  void append_tag_name(char c);
-
  private: /* data */
-  char value[MAX_TOKEN_SIZE]{};
-  unsigned int value_length;
-
-  char tag_name[MAX_NAME_SIZE]{};
-  unsigned int tag_name_length;
-
-  char attr_name[MAX_NAME_SIZE]{};
-  unsigned int attr_name_length;
-
-  instream &input;
-  char input_char;
+  string_ref value_;
+  string_ref tag_name_;
+  string_ref attr_name_;
 
-  bool got_tail;  // aux flag used in scan_comment, etc.
+  instream &input_;
 
-  const char *text_begin, *text_end;
+  bool got_tail;  // aux flag used in scan_comment
 };
 }  // namespace markup

From 8884b390554b816a3273859eb9ed58d8e50b5fbc Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Fri, 17 Dec 2021 17:39:43 +0100
Subject: [PATCH 317/442] Disabled importing optimized gemm module (#282)

- Until the optimized gemm module stops requiring
   Shared Array Buffer, we can't really use it in
   Firefox
---
 wasm/import-gemm-module.js | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/wasm/import-gemm-module.js b/wasm/import-gemm-module.js
index 369c551cc..8d20c58a7 100644
--- a/wasm/import-gemm-module.js
+++ b/wasm/import-gemm-module.js
@@ -3,9 +3,6 @@
  * implementation.
  */
 function createWasmGemm() {
-    // Name of the optimized gemm implementation.
-    const OPTIMIZED_GEMM = "mozIntGemm";
-
     // A map of expected gemm function to the corresponding fallback gemm function names.
     const GEMM_TO_FALLBACK_FUNCTIONS_MAP = {
         "int8_prepare_a": "int8PrepareAFallback",
@@ -17,19 +14,28 @@ function createWasmGemm() {
         "int8_select_columns_of_b": "int8SelectColumnsOfBFallback"
     };
 
-    const optimizedGemmModule = WebAssembly[OPTIMIZED_GEMM];
-    if (!optimizedGemmModule) {
-        return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
-    }
+    // ToDo: Activate the if code and remove else code once optimized gemm can work without shared array buffer.
+    if (0) {
+        // Name of the optimized gemm implementation.
+        const OPTIMIZED_GEMM = "mozIntGemm";
 
-    const optimizedGemmModuleExports = new WebAssembly.Instance(optimizedGemmModule(), {"": {memory: wasmMemory}}).exports;
-    for (let key in GEMM_TO_FALLBACK_FUNCTIONS_MAP) {
-        if (!optimizedGemmModuleExports[key]) {
+        const optimizedGemmModule = WebAssembly[OPTIMIZED_GEMM];
+        if (!optimizedGemmModule) {
             return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
         }
+
+        const optimizedGemmModuleExports = new WebAssembly.Instance(optimizedGemmModule(), {"": {memory: wasmMemory}}).exports;
+        for (let key in GEMM_TO_FALLBACK_FUNCTIONS_MAP) {
+            if (!optimizedGemmModuleExports[key]) {
+                return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
+            }
+        }
+        console.log(`Using optimized gemm (${OPTIMIZED_GEMM}) implementation`);
+        return optimizedGemmModuleExports;
+    }
+    else {
+        return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
     }
-    console.log(`Using optimized gemm (${OPTIMIZED_GEMM}) implementation`);
-    return optimizedGemmModuleExports;
 }
 
 // Return the fallback gemm implementation.

From 793d132b7c8dfa2c41ca5745c09ba402e1749eb8 Mon Sep 17 00:00:00 2001
From: Andre Natal <andrenatal@users.noreply.github.com>
Date: Fri, 17 Dec 2021 15:05:11 -0800
Subject: [PATCH 318/442] Adding circle ci job to push the wasm artifacts to
 github releases (#280)

* Adding circle ci job to push the wasm artifacts to github releases.
* Updated config.yml
---
 .circleci/config.yml | 66 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 61 insertions(+), 5 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 9b14ed154..d9ff7933d 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -11,8 +11,9 @@ jobs:
       - checkout
 
       - run:
-          name: Build WASM
-          command: bash build-wasm.sh WORMHOLE
+          name: Build WASM WORMHOLE
+          command: |
+            bash build-wasm.sh WORMHOLE
 
       - run:
           name: Check artifacts
@@ -22,11 +23,21 @@ jobs:
             if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
             then
               echo "Artifacts Successfully Generated"
+              mkdir ../artifacts
+              cp bergamot-translator-worker.wasm ../artifacts/bergamot-translator-worker-with-wormhole.wasm
+              cp bergamot-translator-worker.js ../artifacts/bergamot-translator-worker-with-wormhole.js
+              shasum -a 256 ../artifacts/* > ../artifacts/SHA256-1
+              cp ../BERGAMOT_VERSION ../artifacts/
             else
               echo "Failure: Artifacts Not Present"
               exit 1
             fi
 
+      - persist_to_workspace:
+          root: .
+          paths:
+            - artifacts/*
+
       - store_artifacts:
           path: "build-wasm"
           destination: "wasm-wormhole"
@@ -43,7 +54,8 @@ jobs:
 
       - run:
           name: Build WASM
-          command: bash build-wasm.sh
+          command: |
+            bash build-wasm.sh
 
       - run:
           name: Check artifacts
@@ -53,17 +65,61 @@ jobs:
             if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
             then
               echo "Artifacts Successfully Generated"
+              mkdir ../artifacts
+              cp bergamot-translator-worker.wasm ../artifacts/bergamot-translator-worker-without-wormhole.wasm
+              cp bergamot-translator-worker.js ../artifacts/bergamot-translator-worker-without-wormhole.js
+              shasum -a 256 ../artifacts/* > ../artifacts/SHA256-2
             else
               echo "Failure: Artifacts Not Present"
               exit 1
             fi
+      - persist_to_workspace:
+          root: .
+          paths:
+            - artifacts/*
 
       - store_artifacts:
           path: "build-wasm"
           destination: "wasm-without-wormhole"
+  publish_to_github:
+     docker:
+       - image: cibuilds/github:0.10
+     steps:
+       - attach_workspace:
+          # Must be absolute path or relative path from working_directory
+          at: ./
+       - run:
+          name: "Publish Release on GitHub"
+          command: |
+            export COMMIT=$(echo $CIRCLE_SHA1 | cut -c -7)
+            export VERSION=$(cat ./artifacts/BERGAMOT_VERSION | cut -c 2-)
+            VERSION=$VERSION+$COMMIT
+            ls -lsa ./artifacts/ > ./artifacts/FILESIZES
+            cat ./artifacts/SHA256-1 ./artifacts/SHA256-2 > ./artifacts/SHA256
+            rm ./artifacts/SHA256-1
+            rm ./artifacts/SHA256-2
+            rm ./artifacts/BERGAMOT_VERSION
+            ghr -t ${GHTOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${VERSION} ./artifacts/
 
 workflows:
   build:
       jobs:
-          - build-with-wormhole
-          - build-without-wormhole
\ No newline at end of file
+          - build-with-wormhole:
+              filters:
+                tags:
+                  only: /^v.*/
+          - build-without-wormhole:
+              filters:
+                tags:
+                  only: /^v.*/
+          - publish_to_github:
+              filters:
+                tags:
+                  only: /^v.*/
+                branches:
+                  ignore: /.*/
+              requires:
+                - build-without-wormhole
+                - build-with-wormhole
+
+

From 1a27a8e0a75ab4f0154cc61ab4ae9ccc1cf9842e Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Mon, 20 Dec 2021 16:24:30 +0100
Subject: [PATCH 319/442] Increase HTML test coverage (#279)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* Fix bug in HasAlignments check

When fixing it to allow empty sentences, it no longer caught misconfigured models. I've added a test that triggers this scenario, and a fix in HasAlignments for it.

* Add more unit tests for xh_scanner

Trying to increase that code coverage to 100%

* Add test for whitespaces around attributes

* Make accessing value(), attr_name() and tag_name() at the wrong time safer

* Fix bug in <style> and <script> parsing

The end tag was never found

* Fix parsing of mix of valueless and quoteless attributes

* Sync list of void tags with Firefox' implementation of outerHTML and innerHTML

Also lets use their name for it: IsVoidTag instead of IsEmptyElement. Empty was a bit ambiguous.

* Bring back support for processing instructions support in xh_scanner

I noticed in https://searchfox.org/mozilla-central/source/dom/base/nsContentUtils.cpp#8961 that these can be produced by innerHTML under some circumstances.

* More permanent link

* Use CamelCase for the internal functions I added

* Rename *_PI to *_PROCESSING_INSTRUCTION

Your IDE will do the typing for you anyway

* Match symbol naming of the rest of code base

CapitalCase for classes, camelCase for functions, snake_case for variables still.

* Missed one 😴

* Change xhscanner's variable case also to camelCase

* Partially fix case variables in html.cpp
---
 src/tests/units/html_tests.cpp       | 102 ++++++----
 src/tests/units/xh_scanner_tests.cpp | 272 ++++++++++++++++++++-------
 src/translator/html.cpp              | 177 ++++++++---------
 src/translator/html.h                |   2 +-
 src/translator/response_builder.h    |   2 +-
 src/translator/xh_scanner.cpp        | 172 ++++++++++-------
 src/translator/xh_scanner.h          | 107 +++++++----
 7 files changed, 540 insertions(+), 294 deletions(-)

diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
index 8f6ca8216..9fd2acfc6 100644
--- a/src/tests/units/html_tests.cpp
+++ b/src/tests/units/html_tests.cpp
@@ -16,7 +16,7 @@ std::ostream &operator<<(std::ostream &out, std::pair<ByteRange, ByteRange> cons
 
 std::ostream &operator<<(std::ostream &out, ByteRange const &b) { return out << '{' << b.begin << ',' << b.end << '}'; }
 
-std::vector<ByteRange> AsByteRanges(AnnotatedText const &annotation) {
+std::vector<ByteRange> asByteRanges(AnnotatedText const &annotation) {
   std::vector<ByteRange> words;
   words.emplace_back(annotation.annotation.gap(0));
   for (size_t sentenceIdx = 0; sentenceIdx < annotation.numSentences(); ++sentenceIdx) {
@@ -27,7 +27,7 @@ std::vector<ByteRange> AsByteRanges(AnnotatedText const &annotation) {
   return words;
 }
 
-std::vector<std::string> AsTokens(AnnotatedText const &annotation) {
+std::vector<std::string> asTokens(AnnotatedText const &annotation) {
   std::vector<std::string> words;
   words.emplace_back(annotation.gap(0));
   for (size_t sentenceIdx = 0; sentenceIdx < annotation.numSentences(); ++sentenceIdx) {
@@ -38,7 +38,7 @@ std::vector<std::string> AsTokens(AnnotatedText const &annotation) {
   return words;
 }
 
-void RecordSentenceFromByteRange(AnnotatedText &text, std::vector<ByteRange> const &ranges) {
+void recordSentenceFromByteRange(AnnotatedText &text, std::vector<ByteRange> const &ranges) {
   assert(ranges.size() > 0);
 
   std::vector<string_view> tokens;
@@ -70,13 +70,12 @@ TEST_CASE("Ignore HTML if process_markup is false") {
   response.source.text = html_code;
   response.target.text = html_code;
   // Note: response.alignments is empty, which is allowed in this case
-  html.Restore(response);
+  html.restore(response);
 
-  // Assert that Restore() does not mess with my HTML code
+  // Assert that restore() does not mess with my HTML code
   CHECK(response.source.text == html_code);
 }
 
-// TODO: is there a better way to test for correct abort() calls than [!shouldfail]
 TEST_CASE("Abort if alignments are missing") {
   marian::setThrowExceptionOnAbort(true);
 
@@ -84,7 +83,7 @@ TEST_CASE("Abort if alignments are missing") {
   HTML html(std::move(input), true);
 
   AnnotatedText source("hello world\n");
-  RecordSentenceFromByteRange(source, {
+  recordSentenceFromByteRange(source, {
                                           ByteRange{0, 4},   // 0.0 "hell"
                                           ByteRange{4, 5},   // 0.1 "o"
                                           ByteRange{5, 11},  // 0.2 " world"
@@ -92,7 +91,7 @@ TEST_CASE("Abort if alignments are missing") {
                                       });
 
   AnnotatedText target("hallo Welt\n");
-  RecordSentenceFromByteRange(target, {
+  recordSentenceFromByteRange(target, {
                                           ByteRange{0, 4},   // 0.0 "hall"
                                           ByteRange{4, 5},   // 0.1 "o"
                                           ByteRange{5, 10},  // 0.2 " Welt"
@@ -105,7 +104,42 @@ TEST_CASE("Abort if alignments are missing") {
   // Note: explicitly not setting response.alignments
 
   CHECK_THROWS_WITH(
-      html.Restore(response),
+      html.restore(response),
+      "Response object does not contain alignments. TranslationModel or ResponseOptions is misconfigured?");
+}
+
+TEST_CASE("Abort if alignments are misconfigured") {
+  marian::setThrowExceptionOnAbort(true);
+
+  std::string input("<p>hello <b>world</b></p>\n");
+  HTML html(std::move(input), true);
+
+  AnnotatedText source("hello world\n");
+  recordSentenceFromByteRange(source, {
+                                          ByteRange{0, 4},   // 0.0 "hell"
+                                          ByteRange{4, 5},   // 0.1 "o"
+                                          ByteRange{5, 11},  // 0.2 " world"
+                                          ByteRange{11, 11}  // 0.3 ""
+                                      });
+
+  AnnotatedText target("hallo Welt\n");
+  recordSentenceFromByteRange(target, {
+                                          ByteRange{0, 4},   // 0.0 "hall"
+                                          ByteRange{4, 5},   // 0.1 "o"
+                                          ByteRange{5, 10},  // 0.2 " Welt"
+                                          ByteRange{10, 10}  // 0.3 ""
+                                      });
+
+  Response response;
+  response.source = source;
+  response.target = target;
+
+  // If the model is misconfigured to not give any alignment information,
+  // response will have entries for each target word, but they will all be empty.
+  response.alignments = {{{}, {}, {}, {}}};
+
+  CHECK_THROWS_WITH(
+      html.restore(response),
       "Response object does not contain alignments. TranslationModel or ResponseOptions is misconfigured?");
 }
 
@@ -115,7 +149,7 @@ TEST_CASE("Do not abort if the input is just empty") {
   CHECK(input == "");
 
   Response response;
-  html.Restore(response);
+  html.restore(response);
   CHECK(response.source.text == "");
   CHECK(response.target.text == "");
 }
@@ -126,7 +160,7 @@ TEST_CASE("Do not abort if the input is just empty element") {
   CHECK(input == "");
 
   Response response;
-  html.Restore(response);
+  html.restore(response);
   CHECK(response.source.text == "<p></p>");
   CHECK(response.target.text == "");  // Should be <p></p> but hey not there yet.
 }
@@ -153,7 +187,7 @@ TEST_CASE("Test reconstruction of target sentence") {
   CHECK(input == "hello world\n");
 
   AnnotatedText source("hello world\n");
-  RecordSentenceFromByteRange(source, {
+  recordSentenceFromByteRange(source, {
                                           ByteRange{0, 4},   // 0.0 "hell"
                                           ByteRange{4, 5},   // 0.1 "o"
                                           ByteRange{5, 11},  // 0.2 " world"
@@ -161,7 +195,7 @@ TEST_CASE("Test reconstruction of target sentence") {
                                       });
 
   AnnotatedText target("hallo Welt\n");
-  RecordSentenceFromByteRange(target, {
+  recordSentenceFromByteRange(target, {
                                           ByteRange{0, 4},   // 0.0 "hall"
                                           ByteRange{4, 5},   // 0.1 "o"
                                           ByteRange{5, 10},  // 0.2 " Welt"
@@ -173,14 +207,14 @@ TEST_CASE("Test reconstruction of target sentence") {
   response.target = target;
   response.alignments = {identity_matrix<float>(4)};
 
-  html.Restore(response);
+  html.restore(response);
 
   std::vector<std::string> html_tokens_source{"", "<p>hell", "o", " <b>world", "", "</b></p>\n"};
 
   std::vector<std::string> html_tokens_target{"", "<p>hall", "o", " <b>Welt", "", "</b></p>\n"};
 
-  CHECK(AsTokens(response.source) == html_tokens_source);
-  CHECK(AsTokens(response.target) == html_tokens_target);
+  CHECK(asTokens(response.source) == html_tokens_source);
+  CHECK(asTokens(response.target) == html_tokens_target);
 }
 
 TEST_CASE("Test reconstruction of target sentence with entities") {
@@ -189,7 +223,7 @@ TEST_CASE("Test reconstruction of target sentence with entities") {
   CHECK(input == "hello world & friends!\n");
 
   AnnotatedText source("hello world & friends!\n");
-  RecordSentenceFromByteRange(source, {
+  recordSentenceFromByteRange(source, {
                                           ByteRange{0, 4},    // 0.0 "hell"
                                           ByteRange{4, 5},    // 0.1 "o"
                                           ByteRange{5, 11},   // 0.2 " world"
@@ -200,7 +234,7 @@ TEST_CASE("Test reconstruction of target sentence with entities") {
                                       });
 
   AnnotatedText target("hallo Welt & Freunde!\n");
-  RecordSentenceFromByteRange(target, {
+  recordSentenceFromByteRange(target, {
                                           ByteRange{0, 4},    // 0.0 "hall"
                                           ByteRange{4, 5},    // 0.1 "o"
                                           ByteRange{5, 10},   // 0.2 " Welt"
@@ -215,7 +249,7 @@ TEST_CASE("Test reconstruction of target sentence with entities") {
   response.target = target;
   response.alignments = {identity_matrix<float>(7)};
 
-  html.Restore(response);
+  html.restore(response);
 
   std::vector<std::string> html_tokens_source{"",         "<p>hell", "o", " <b>world", " &amp;",
                                               " friends", "!",       "",  "</b></p>\n"};
@@ -224,8 +258,8 @@ TEST_CASE("Test reconstruction of target sentence with entities") {
 
                                               " Freunde", "!",       "",  "</b></p>\n"};
 
-  CHECK(AsTokens(response.source) == html_tokens_source);
-  CHECK(AsTokens(response.target) == html_tokens_target);
+  CHECK(asTokens(response.source) == html_tokens_source);
+  CHECK(asTokens(response.target) == html_tokens_target);
 }
 
 TEST_CASE("Test reconstruction of target with multiple sentences") {
@@ -236,14 +270,14 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
   AnnotatedText source("hello world! How does this  deal with multiple sentences? Will it work?\n");
   CHECK(source.text == input);
 
-  RecordSentenceFromByteRange(source, {
+  recordSentenceFromByteRange(source, {
                                           ByteRange{0, 4},    // 0.0 "hell"
                                           ByteRange{4, 5},    // 0.1 "o"
                                           ByteRange{5, 11},   // 0.2 " world"
                                           ByteRange{11, 12},  // 0.3 "!"
                                           ByteRange{12, 12}   // 0.4 ""
                                       });
-  RecordSentenceFromByteRange(source, {
+  recordSentenceFromByteRange(source, {
                                           ByteRange{13, 16},  // 1.0 "How"
                                           ByteRange{16, 21},  // 1.1 " does"
                                           ByteRange{21, 26},  // 1.2 " this"
@@ -255,7 +289,7 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
                                           ByteRange{56, 57},  // 1.8 "?"
                                           ByteRange{57, 57}   // 1.9 ""
                                       });
-  RecordSentenceFromByteRange(source, {
+  recordSentenceFromByteRange(source, {
                                           ByteRange{58, 62},  // 2.0 "Will"
                                           ByteRange{62, 65},  // 2.1 " it"
                                           ByteRange{65, 70},  // 2.2 " work"
@@ -264,14 +298,14 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
                                       });
 
   AnnotatedText target("hallo Welt! Wie geht das mit mehreren Sätzen um? Wird es funktionieren?\n");
-  RecordSentenceFromByteRange(target, {
+  recordSentenceFromByteRange(target, {
                                           ByteRange{0, 4},    // 0.0 "hall"
                                           ByteRange{4, 5},    // 0.1 "o"
                                           ByteRange{5, 10},   // 0.2 " Welt"
                                           ByteRange{10, 11},  // 0.3 "!"
                                           ByteRange{11, 11},  // 0.4 ""
                                       });
-  RecordSentenceFromByteRange(target, {
+  recordSentenceFromByteRange(target, {
                                           ByteRange{12, 15},  // 1.0 "Wie"
                                           ByteRange{15, 20},  // 1.1 " geht"
                                           ByteRange{20, 24},  // 1.2 " das"
@@ -283,7 +317,7 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
                                           ByteRange{48, 49},  // 1.8 "?"
                                           ByteRange{49, 49},  // 1.9 ""
                                       });
-  RecordSentenceFromByteRange(target, {
+  recordSentenceFromByteRange(target, {
                                           ByteRange{50, 54},  // 2.0 "Wird"
                                           ByteRange{54, 57},  // 2.1 " es"
                                           ByteRange{57, 71},  // 2.2 " funktionieren"
@@ -295,13 +329,13 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
       "",       "hall", "o",   " Welt", "!", "",  " ",    "Wie", " geht",          " das", " mit", " mehreren",
       " Sätze", "n",    " um", "?",     "",  " ", "Wird", " es", " funktionieren", "?",    "",     "\n"};
 
-  CHECK(AsTokens(target) == text_tokens_source);
+  CHECK(asTokens(target) == text_tokens_source);
 
   Response response;
   response.source = source;
   response.target = target;
   response.alignments = {identity_matrix<float>(5), identity_matrix<float>(10), identity_matrix<float>(5)};
-  html.Restore(response);
+  html.restore(response);
 
   std::vector<std::string> html_tokens_source{"",
                                               "<p>hell",
@@ -327,7 +361,7 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
                                               "?",
                                               "",
                                               "</p>\n"};
-  CHECK(AsTokens(response.source) == html_tokens_source);
+  CHECK(asTokens(response.source) == html_tokens_source);
 }
 
 TEST_CASE("Test self-closing tag (HTML5)") {
@@ -380,7 +414,7 @@ TEST_CASE("Test empty tag", "[!mayfail]") {
 
   response.alignments = {identity_matrix<float>(4)};
 
-  html.Restore(response);
+  html.restore(response);
   CHECK(response.source.text == test_str);
   CHECK(response.target.text == test_str);
 }
@@ -439,7 +473,7 @@ TEST_CASE("End-to-end translation") {
     response.target.appendEndingWhitespace("\n");
   }
 
-  html.Restore(response);
+  html.restore(response);
 
   {
     AnnotatedText source;
@@ -457,7 +491,7 @@ TEST_CASE("End-to-end translation") {
     source.appendSentence("", sentence.begin(), sentence.end());
     source.appendEndingWhitespace("</p>\n");
 
-    CHECK(AsTokens(response.source) == AsTokens(source));
+    CHECK(asTokens(response.source) == asTokens(source));
   }
 
   {
@@ -477,7 +511,7 @@ TEST_CASE("End-to-end translation") {
     target.appendSentence("", sentence.begin(), sentence.end());
     target.appendEndingWhitespace("</p>\n");
 
-    CHECK(AsTokens(response.target) == AsTokens(target));
+    CHECK(asTokens(response.target) == asTokens(target));
   }
 }
 
diff --git a/src/tests/units/xh_scanner_tests.cpp b/src/tests/units/xh_scanner_tests.cpp
index f01e47ddc..0fdfa3566 100644
--- a/src/tests/units/xh_scanner_tests.cpp
+++ b/src/tests/units/xh_scanner_tests.cpp
@@ -4,71 +4,148 @@
 #include "translator/xh_scanner.h"
 
 TEST_CASE("scan element with attributes") {
-  markup::instream in("<div id=\"test\" class=\"a b c \" hidden>");
-  markup::scanner scanner(in);
+  markup::instream in("<div id=\"test\" class=\"a b c \">");
+  markup::Scanner scanner(in);
 
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
-  CHECK(scanner.tag_name() == "div");
-  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
-  CHECK(scanner.attr_name() == "id");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "div");
+
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "id");
   CHECK(scanner.value() == "test");
 
-  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
-  CHECK(scanner.attr_name() == "class");
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "class");
   CHECK(scanner.value() == "a b c ");
 
-  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
-  CHECK(scanner.attr_name() == "hidden");
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
+}
+
+TEST_CASE("scan element with valueless attributes") {
+  markup::instream in("<input checked hidden>");
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "input");
+
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "checked");
+  CHECK(scanner.value() == "");
+
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "hidden");
   CHECK(scanner.value() == "");
 
-  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
+}
+
+TEST_CASE("scan element with unquoted attributes") {
+  markup::instream in("<div hidden=true class=test>");
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "div");
+
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "hidden");
+  CHECK(scanner.value() == "true");
+
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "class");
+  CHECK(scanner.value() == "test");
+
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
+}
+
+TEST_CASE("scan element with spaces around attributes") {
+  markup::instream in("<input class = \"test\" checked type = checkbox >");
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "input");
+
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "class");
+  CHECK(scanner.value() == "test");
+
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "checked");
+  CHECK(scanner.value() == "");
+
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "type");
+  CHECK(scanner.value() == "checkbox");
+
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
 }
 
 TEST_CASE("scan element with text") {
   markup::instream in("<span>Hello world</span>");
-  markup::scanner scanner(in);
+  markup::Scanner scanner(in);
 
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
   CHECK(scanner.value() == "Hello world");
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
-  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_END);
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
 }
 
 TEST_CASE("scan html entities") {
-  markup::instream in("Hello &amp; &apos;world&apos;");
-  markup::scanner scanner(in);
+  markup::instream in("&amp;&apos;&nbsp;&quot;&lt;&gt;");
+  markup::Scanner scanner(in);
 
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
-  CHECK(scanner.value() == "Hello ");
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
   CHECK(scanner.value() == "&");
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
-  CHECK(scanner.value() == " ");
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
   CHECK(scanner.value() == "'");
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
-  CHECK(scanner.value() == "world");
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
-  CHECK(scanner.value() == "'");
-  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == " ");
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == "\"");
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == "<");
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == ">");
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
+}
+
+TEST_CASE("scan & instead of &amp;") {
+  markup::instream in("Hello & other people");
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == "Hello ");
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == "&");
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == " other people");
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
+}
+
+TEST_CASE("scan &notanentity;") {
+  markup::instream in("&notanentity;");
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == "&notanentity;");
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
 }
 
 TEST_CASE("scan nested elements") {
   markup::instream in("<div><p><img></p></div>");
-  markup::scanner scanner(in);
-
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
-  CHECK(scanner.tag_name() == "div");
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
-  CHECK(scanner.tag_name() == "p");
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
-  CHECK(scanner.tag_name() == "img");
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
-  CHECK(scanner.tag_name() == "p");
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
-  CHECK(scanner.tag_name() == "div");
-  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "div");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "p");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "img");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_END);
+  CHECK(scanner.tag() == "p");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_END);
+  CHECK(scanner.tag() == "div");
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
 }
 
 TEST_CASE("scan kitchen sink") {
@@ -78,38 +155,38 @@ TEST_CASE("scan kitchen sink") {
       "this is a comment -->this is &amp; text\n"
       "</span></div>";
   markup::instream in(html_str.data());
-  markup::scanner scanner(in);
+  markup::Scanner scanner(in);
 
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
-  CHECK(scanner.tag_name() == "div");
-  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
-  CHECK(scanner.attr_name() == "id");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "div");
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "id");
   CHECK(scanner.value() == "test-id");
-  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
-  CHECK(scanner.attr_name() == "class");
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "class");
   CHECK(scanner.value() == "a b c ");
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
   CHECK(scanner.value() == "\n");
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_START);
-  CHECK(scanner.tag_name() == "span");
-  CHECK(scanner.next_token() == markup::scanner::TT_ATTR);
-  CHECK(scanner.attr_name() == "x-custom-attribute");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "span");
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "x-custom-attribute");
   CHECK(scanner.value() == "Hello &quot;world&quot;");  // We do not decode entities in attributes
-  CHECK(scanner.next_token() == markup::scanner::TT_COMMENT_START);
-  CHECK(scanner.next_token() == markup::scanner::TT_DATA);
+  CHECK(scanner.next() == markup::Scanner::TT_COMMENT_START);
+  CHECK(scanner.next() == markup::Scanner::TT_DATA);
   CHECK(scanner.value() == "\nthis is a comment ");
-  CHECK(scanner.next_token() == markup::scanner::TT_COMMENT_END);
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.next() == markup::Scanner::TT_COMMENT_END);
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
   CHECK(scanner.value() == "this is ");
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
   CHECK(scanner.value() == "&");
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
   CHECK(scanner.value() == " text\n");
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
-  CHECK(scanner.tag_name() == "span");
-  CHECK(scanner.next_token() == markup::scanner::TT_TAG_END);
-  CHECK(scanner.tag_name() == "div");
-  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_END);
+  CHECK(scanner.tag() == "span");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_END);
+  CHECK(scanner.tag() == "div");
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
 }
 
 TEST_CASE("test long text (#273)") {
@@ -117,9 +194,68 @@ TEST_CASE("test long text (#273)") {
   for (size_t i = 0; i < 1024; ++i) test_str.append("testing ");
 
   markup::instream in(test_str.data());
-  markup::scanner scanner(in);
+  markup::Scanner scanner(in);
 
-  CHECK(scanner.next_token() == markup::scanner::TT_TEXT);
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
   CHECK(scanner.value() == test_str);
-  CHECK(scanner.next_token() == markup::scanner::TT_EOF);
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
+}
+
+TEST_CASE("scan self-closing element") {
+  markup::instream in("before <img src=\"#\"/> after");
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == "before ");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "img");
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "src");
+  CHECK(scanner.value() == "#");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_END);
+  CHECK(scanner.tag() == "img");
+  CHECK(scanner.next() == markup::Scanner::TT_TEXT);
+  CHECK(scanner.value() == " after");
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
+}
+
+TEST_CASE("scan script") {
+  markup::instream in("<script async>true && document.body.length > 10</script>");
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "script");
+  CHECK(scanner.next() == markup::Scanner::TT_ATTRIBUTE);
+  CHECK(scanner.attribute() == "async");
+  CHECK(scanner.value() == "");
+  CHECK(scanner.next() == markup::Scanner::TT_DATA);
+  CHECK(scanner.value() == "true && document.body.length > 10");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_END);
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
+}
+
+TEST_CASE("scan style") {
+  markup::instream in("<style>body { background: url(test.png); }</style>");
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_START);
+  CHECK(scanner.tag() == "style");
+  CHECK(scanner.next() == markup::Scanner::TT_DATA);
+  CHECK(scanner.value() == "body { background: url(test.png); }");
+  CHECK(scanner.next() == markup::Scanner::TT_TAG_END);
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
+}
+
+TEST_CASE("scan processing instruction") {
+  // Based on https://searchfox.org/mozilla-central/source/dom/base/nsContentUtils.cpp#8961
+  // element.outerHTML can produce processing instructions in the html. These
+  // should be treated similar to <!-- foo -->.
+  markup::instream in("<?xml version=\"1.0\"?>");
+  markup::Scanner scanner(in);
+
+  CHECK(scanner.next() == markup::Scanner::TT_PROCESSING_INSTRUCTION_START);
+  CHECK(scanner.next() == markup::Scanner::TT_DATA);
+  CHECK(scanner.value() == "xml version=\"1.0\"");
+  CHECK(scanner.next() == markup::Scanner::TT_PROCESSING_INSTRUCTION_END);
+  CHECK(scanner.next() == markup::Scanner::TT_EOF);
 }
\ No newline at end of file
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index efe7969f6..f531b44fe 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -10,7 +10,7 @@ using marian::bergamot::ByteRange;
 using marian::bergamot::HTML;
 using marian::bergamot::Response;
 
-void EncodeEntities(string_view const &input, std::string &output) {
+void encodeEntities(string_view const &input, std::string &output) {
   output.clear();
   output.reserve(input.size());
 
@@ -41,7 +41,7 @@ void EncodeEntities(string_view const &input, std::string &output) {
   }
 }
 
-size_t CountPrefixWhitespaces(string_view const &input) {
+size_t countPrefixWhitespaces(string_view const &input) {
   size_t size = 0;
   while (size < input.size() && input[size] == ' ') ++size;
   return size;
@@ -63,47 +63,50 @@ std::ostream &operator<<(std::ostream &out, HTML::Taint const &tags) {
 }
 
 // Very simple replacement for std::format introduced in C++20
-std::string format(std::string const &format_str) { return format_str; }
+std::string format(std::string const &formatTemplate) { return formatTemplate; }
 
 template <typename Arg>
-std::string format(std::string const &format_str, Arg arg) {
+std::string format(std::string const &formatTemplate, Arg arg) {
   std::ostringstream os;
-  auto index = format_str.find("{}");
+  auto index = formatTemplate.find("{}");
   assert(index != std::string::npos);
-  os << format_str.substr(0, index) << arg << format_str.substr(index + 2);
+  os << formatTemplate.substr(0, index) << arg << formatTemplate.substr(index + 2);
   return os.str();
 }
 
 template <typename Arg, typename... Args>
-std::string format(std::string const &format_str, Arg arg, Args... args) {
+std::string format(std::string const &formatTemplate, Arg arg, Args... args) {
   std::ostringstream os;
-  auto index = format_str.find("{}");
+  auto index = formatTemplate.find("{}");
   assert(index != std::string::npos);
-  os << format_str.substr(0, index) << arg << format(format_str.substr(index + 2), std::forward<Args>(args)...);
+  os << formatTemplate.substr(0, index) << arg << format(formatTemplate.substr(index + 2), std::forward<Args>(args)...);
   return os.str();
 }
 
-bool IsBlockElement(std::string_view const &name) {
+bool isBlockElement(std::string_view const &name) {
   // List of elements that we expect might occur inside words, and that should
   // not introduce spacings around them. Not strictly inline elements, nor flow
   // elements. See also https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Content_categories
-  static std::unordered_set<std::string> inline_ish_elements{
+  static std::unordered_set<std::string> inlineishElements{
       "abbr",  "a",    "b",      "em",  "i",   "kbd",  "mark", "math", "output", "q",   "ruby",
       "small", "span", "strong", "sub", "sup", "time", "u",    "var",  "wbr",    "ins", "del"};
 
-  return inline_ish_elements.find(std::string(name)) == inline_ish_elements.end();
+  return inlineishElements.find(std::string(name)) == inlineishElements.end();
 }
 
-bool IsEmptyElement(std::string_view const &name) {
+bool isVoidTag(std::string_view const &name) {
   // List of elements for which we do not expect a closing tag, or self-closing
   // elements in XHTML. See also https://developer.mozilla.org/en-US/docs/Glossary/Empty_element
-  static std::unordered_set<std::string> empty_elements{"area",  "base", "br",   "col",   "embed",  "hr",    "img",
-                                                        "input", "link", "meta", "param", "source", "track", "wbr"};
+  // More relevant source of this list:
+  // https://searchfox.org/mozilla-central/rev/7d17fd1fe9f0005a2fb19e5d53da4741b06a98ba/dom/base/FragmentOrElement.cpp#1791
+  static std::unordered_set<std::string> voidElements{"area",  "base",  "basefont", "bgsound", "br",    "col",
+                                                      "embed", "frame", "hr",       "img",     "input", "keygen",
+                                                      "link",  "meta",  "param",    "source",  "track", "wbr"};
 
-  return empty_elements.find(std::string(name)) != empty_elements.end();
+  return voidElements.find(std::string(name)) != voidElements.end();
 }
 
-void DiffTags(HTML::Taint const &prev, HTML::Taint const &curr, HTML::Taint &opening, HTML::Taint &closing) {
+void diffTags(HTML::Taint const &prev, HTML::Taint const &curr, HTML::Taint &opening, HTML::Taint &closing) {
   opening.clear();
   closing.clear();
 
@@ -118,11 +121,11 @@ void DiffTags(HTML::Taint const &prev, HTML::Taint const &curr, HTML::Taint &ope
   opening.insert(opening.end(), curr.begin() + i, curr.end());
 }
 
-bool Intersects(ByteRange const &range, HTML::Span const &span) {
+bool intersects(ByteRange const &range, HTML::Span const &span) {
   return range.begin <= span.end && range.end >= span.begin;
 };
 
-void FilterEmpty(HTML::Taint &stack) {
+void filterEmpty(HTML::Taint &stack) {
   auto src = stack.begin();
   auto dst = stack.begin();
 
@@ -132,12 +135,12 @@ void FilterEmpty(HTML::Taint &stack) {
   stack.resize(dst - stack.begin());
 }
 
-bool ContainsTag(HTML::Taint const &stack, HTML::Tag const *tag) {
+bool containsTag(HTML::Taint const &stack, HTML::Tag const *tag) {
   return std::find(stack.rbegin(), stack.rend(), tag) != stack.rend();
 }
 
 template <typename Fun>
-AnnotatedText Apply(AnnotatedText const &in, Fun fun) {
+AnnotatedText apply(AnnotatedText const &in, Fun fun) {
   AnnotatedText out;
 
   for (size_t sentenceIdx = 0; sentenceIdx < in.numSentences(); ++sentenceIdx) {
@@ -168,20 +171,27 @@ AnnotatedText Apply(AnnotatedText const &in, Fun fun) {
   return out;
 }
 
-bool IsContinuation(string_view str) { return !str.empty() && str.compare(0, 1, " ", 1) != 0; }
+bool isContinuation(string_view str) { return !str.empty() && str.compare(0, 1, " ", 1) != 0; }
 
-bool HasAlignments(Response const &response) {
+bool hasAlignments(Response const &response) {
   // Test for each sentence individually as a sentence may be empty (or there)
   // might be no sentences, so just testing for alignments.empty() would not be
   // sufficient.
-  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx)
+  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
+    // If response.alignments is just empty, this might catch it.
     if (response.alignments.size() <= sentenceIdx ||
         response.alignments[sentenceIdx].size() != response.target.numWords(sentenceIdx))
       return false;
+
+    // If response.alignments is "empty" because the model did not provide alignments,
+    // it still has entries for each target word. But all these entries are empty.
+    for (size_t wordIdx = 0; wordIdx < response.target.numWords(sentenceIdx); ++wordIdx)
+      if (response.alignments[sentenceIdx][wordIdx].size() != response.source.numWords(sentenceIdx)) return false;
+  }
   return true;
 }
 
-void HardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
+void hardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
   // For each sentence...
   for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
     alignments.emplace_back();
@@ -204,24 +214,24 @@ void HardAlignments(Response const &response, std::vector<std::vector<size_t>> &
     for (size_t t = 1; t + 1 < response.target.numWords(sentenceIdx); ++t) {
       // If this token is a continuation of a previous token, pick the tags from the most
       // prevalent token for the whole word.
-      if (IsContinuation(response.target.word(sentenceIdx, t))) {
+      if (isContinuation(response.target.word(sentenceIdx, t))) {
         // Note: only looking at the previous token since that will already
         // have this treatment applied to it.
-        size_t s_curr = alignments.back()[t];
-        size_t s_prev = alignments.back()[t - 1];
-        float score_curr = response.alignments[sentenceIdx][t][s_curr];
-        float score_prev = response.alignments[sentenceIdx][t - 1][s_prev];
+        size_t currSentenceIdx = alignments.back()[t];
+        size_t prevSentenceIdx = alignments.back()[t - 1];
+        float currScore = response.alignments[sentenceIdx][t][currSentenceIdx];
+        float prevScore = response.alignments[sentenceIdx][t - 1][prevSentenceIdx];
 
-        if (score_curr > score_prev) {
+        if (currScore > prevScore) {
           // Apply this to all previous tokens in the word
           for (size_t i = t;; --i) {
-            alignments.back()[i] = s_curr;
+            alignments.back()[i] = currSentenceIdx;
 
             // Stop if this was the first token or the beginning of the word
-            if (i == 0 || !IsContinuation(response.target.word(sentenceIdx, i))) break;
+            if (i == 0 || !isContinuation(response.target.word(sentenceIdx, i))) break;
           }
         } else {
-          alignments.back()[t] = s_prev;
+          alignments.back()[t] = prevSentenceIdx;
         }
       }
     }
@@ -231,28 +241,28 @@ void HardAlignments(Response const &response, std::vector<std::vector<size_t>> &
   }
 }
 
-void CopyTaint(Response const &response, std::vector<std::vector<size_t>> const &alignments,
-               std::vector<HTML::Taint> const &token_tags, std::vector<HTML::Taint> &token_tags_target) {
-  size_t token_offset = 0;
+void copyTaint(Response const &response, std::vector<std::vector<size_t>> const &alignments,
+               std::vector<HTML::Taint> const &sourceTokenTags, std::vector<HTML::Taint> &targetTokenTags) {
+  size_t offset = 0;
 
-  // Fill token_tags_target based on the alignments we just made up.
+  // Fill targetTokenTags based on the alignments we just made up.
   // NOTE: this should match the exact order of Apply()
   for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
-    token_tags_target.push_back(token_tags[token_offset]);  // token_tag for sentence ending gap
+    targetTokenTags.push_back(sourceTokenTags[offset]);  // token_tag for sentence ending gap
     for (size_t t = 0; t < response.target.numWords(sentenceIdx); ++t) {
       size_t s = alignments[sentenceIdx][t];
       assert(s < response.source.numWords(sentenceIdx));
-      token_tags_target.push_back(token_tags[token_offset + 1 + s]);  // +1 for prefix gap
+      targetTokenTags.push_back(sourceTokenTags[offset + 1 + s]);  // +1 for prefix gap
     }
 
-    token_offset += response.source.numWords(sentenceIdx) + 1;  // +1 for prefix gap
+    offset += response.source.numWords(sentenceIdx) + 1;  // +1 for prefix gap
   }
 
-  assert(token_offset < token_tags.size());
-  token_tags_target.push_back(token_tags[token_offset]);  // token_tag for ending whitespace
+  assert(offset < sourceTokenTags.size());
+  targetTokenTags.push_back(sourceTokenTags[offset]);  // token_tag for ending whitespace
 }
 
-AnnotatedText RestoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &token_tags,
+AnnotatedText restoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &token_tags,
                             std::vector<HTML::Span>::const_iterator span_it,
                             std::vector<HTML::Span>::const_iterator span_end) {
   auto prev_it = span_it;  // safe because first span is always empty span, and
@@ -262,13 +272,13 @@ AnnotatedText RestoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &t
   std::string html;
   HTML::Taint opening, closing;
 
-  return Apply(in, [&](ByteRange range, string_view token, bool last) {
+  return apply(in, [&](ByteRange range, string_view token, bool last) {
     // Do encoding of any entities that popped up in the translation
     // (Also effectively clears html from previous call)
-    EncodeEntities(token, html);
+    encodeEntities(token, html);
 
     size_t offset = 0;  // Size added by prepending HTML
-    size_t whitespace_size = CountPrefixWhitespaces(token);
+    size_t whitespace_size = countPrefixWhitespaces(token);
 
     // Close tags we want to show up left (before) the token, but open tags
     // ideally come directly after any prefix whitespace. However, some tokens
@@ -288,7 +298,7 @@ AnnotatedText RestoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &t
 
     // Seek to the last span that overlaps with this token
     while (true) {
-      DiffTags(prev_it->tags, span_it->tags, opening, closing);
+      diffTags(prev_it->tags, span_it->tags, opening, closing);
       prev_it = span_it;
 
       for (auto cit = closing.crbegin(); cit != closing.crend(); ++cit) {
@@ -321,7 +331,7 @@ AnnotatedText RestoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &t
   });
 }
 
-AnnotatedText RestoreTarget(AnnotatedText const &in, std::vector<HTML::Taint> const &token_tags_target) {
+AnnotatedText restoreTarget(AnnotatedText const &in, std::vector<HTML::Taint> const &token_tags_target) {
   auto token_prev_it = token_tags_target.begin();
   auto token_tags_it = token_tags_target.begin() + 1;
 
@@ -329,16 +339,16 @@ AnnotatedText RestoreTarget(AnnotatedText const &in, std::vector<HTML::Taint> co
   std::string html;
   HTML::Taint opening, closing;
 
-  AnnotatedText out = Apply(in, [&](ByteRange range, string_view token, bool last) {
+  AnnotatedText out = apply(in, [&](ByteRange range, string_view token, bool last) {
     // Do encoding of any entities that popped up in the translation
     // (Also effectively clears html from previous call)
-    EncodeEntities(token, html);
+    encodeEntities(token, html);
 
     size_t offset = 0;  // Size added by prepending HTML
-    size_t whitespace_size = CountPrefixWhitespaces(token);
+    size_t whitespace_size = countPrefixWhitespaces(token);
 
     assert(token_tags_it != token_tags_target.end());
-    DiffTags(*token_prev_it, *token_tags_it, opening, closing);
+    diffTags(*token_prev_it, *token_tags_it, opening, closing);
 
     for (auto cit = closing.crbegin(); cit != closing.crend(); ++cit) {
       std::string close_tag = format("</{}>", (*cit)->name);
@@ -371,7 +381,7 @@ AnnotatedText RestoreTarget(AnnotatedText const &in, std::vector<HTML::Taint> co
   return out;
 }
 
-std::ostream &DebugPrintMapping(std::ostream &out, Response const &response,
+std::ostream &debugPrintMapping(std::ostream &out, Response const &response,
                                 std::vector<std::vector<size_t>> const &alignments,
                                 std::vector<HTML::Taint> const &token_tags_target) {
   auto taints = token_tags_target.begin();
@@ -402,7 +412,7 @@ std::ostream &DebugPrintMapping(std::ostream &out, Response const &response,
   return out;
 }
 
-std::ostream &DebugPrintAlignmentScores(std::ostream &out, Response const &response) {
+std::ostream &debugPrintAlignmentScores(std::ostream &out, Response const &response) {
   out << "std::vector<std::vector<std::vector<float>>> alignments{\n";
   for (size_t sentenceIdx = 0; sentenceIdx < response.source.numSentences(); ++sentenceIdx) {
     out << "  {\n";
@@ -420,7 +430,7 @@ std::ostream &DebugPrintAlignmentScores(std::ostream &out, Response const &respo
   return out << "};\n";
 }
 
-size_t DebugCountTokens(AnnotatedText const &text) {
+size_t debugCountTokens(AnnotatedText const &text) {
   size_t tokens = 1;  // for the ending gap
   for (size_t sentenceIdx = 0; sentenceIdx < text.numSentences(); ++sentenceIdx) {
     tokens += 1 + text.numWords(sentenceIdx);  // pre-sentence prefix/gap + each word
@@ -430,14 +440,13 @@ size_t DebugCountTokens(AnnotatedText const &text) {
 
 }  // namespace
 
-namespace marian {
-namespace bergamot {
+namespace marian::bergamot {
 
 HTML::HTML(std::string &&source, bool process_markup) {
   if (!process_markup) return;
   std::string original = std::move(source);
   markup::instream in(original.data(), original.data() + original.size());
-  markup::scanner scanner(in);
+  markup::Scanner scanner(in);
   source.clear();  // source is moved out of, so should be clear anyway
 
   Tag *tag;
@@ -446,30 +455,30 @@ HTML::HTML(std::string &&source, bool process_markup) {
 
   bool stop = false;
   while (!stop) {
-    switch (scanner.next_token()) {
-      case markup::scanner::TT_ERROR:
+    switch (scanner.next()) {
+      case markup::Scanner::TT_ERROR:
         throw BadHTML("HTML parse error");
 
-      case markup::scanner::TT_EOF:
+      case markup::Scanner::TT_EOF:
         stop = true;
         break;
 
-      case markup::scanner::TT_TEXT: {
+      case markup::Scanner::TT_TEXT: {
         auto begin = source.size();
         source.append(scanner.value());
         spans_.push_back(Span{begin, source.size(), stack});
-        FilterEmpty(stack);
+        filterEmpty(stack);
       } break;
 
-      case markup::scanner::TT_TAG_START:
+      case markup::Scanner::TT_TAG_START:
         // If it makes sense to treat this element as a break in a word (e.g.
         // <br>, <img>, <li>) make sure it does so in this text as well.
         // TODO: Strong assumption here that the language uses spaces to
         // separate words
-        if (IsBlockElement(scanner.tag_name()) && !source.empty() && source.back() != ' ') source.push_back(' ');
+        if (isBlockElement(scanner.tag()) && !source.empty() && source.back() != ' ') source.push_back(' ');
 
         // pool_ takes ownership of our tag, makes sure it's freed when necessary
-        pool_.emplace_back(new Tag{std::string(scanner.tag_name()), std::string(), IsEmptyElement(scanner.tag_name())});
+        pool_.emplace_back(new Tag{std::string(scanner.tag()), std::string(), isVoidTag(scanner.tag())});
 
         // Tag *tag is used by attribute parsing
         tag = pool_.back().get();
@@ -485,27 +494,26 @@ HTML::HTML(std::string &&source, bool process_markup) {
         }
         break;
 
-      case markup::scanner::TT_TAG_END:
+      case markup::Scanner::TT_TAG_END:
         // Note: self-closing tags emit TT_TAG_END immediately after TT_TAG_START
         // but since we're parsing HTML5, a sole <img> will never emit a TT_TAG_END
-        if (stack.empty())
-          throw BadHTML(format("Encountered more closing tags ({}) than opening tags", scanner.tag_name()));
+        if (stack.empty()) throw BadHTML(format("Encountered more closing tags ({}) than opening tags", scanner.tag()));
 
-        if (stack.back()->name != scanner.tag_name())
-          throw BadHTML(format("Encountered unexpected closing tag </{}>, stack is {}", scanner.tag_name(), stack));
+        if (stack.back()->name != scanner.tag())
+          throw BadHTML(format("Encountered unexpected closing tag </{}>, stack is {}", scanner.tag(), stack));
 
         // What to do with "<u></u>" case, where tag is immediately closed
         // so it never makes it into the taint of any of the spans? This adds
         // an empty span so it still gets recorded in spans_.
-        if (spans_.empty() || !ContainsTag(spans_.back().tags, stack.back()))
+        if (spans_.empty() || !containsTag(spans_.back().tags, stack.back()))
           spans_.push_back(Span{source.size(), source.size(), stack});
 
         stack.pop_back();
         break;
 
-      case markup::scanner::TT_ATTR:
+      case markup::Scanner::TT_ATTRIBUTE:
         assert(tag != nullptr);
-        tag->attributes += format(" {}=\"{}\"", scanner.attr_name(), scanner.value());
+        tag->attributes += format(" {}=\"{}\"", scanner.attribute(), scanner.value());
         break;
 
       default:
@@ -519,14 +527,14 @@ HTML::HTML(std::string &&source, bool process_markup) {
   spans_.emplace_back(Span{source.size() + 1, source.size() + 1, stack});
 }
 
-void HTML::Restore(Response &response) {
+void HTML::restore(Response &response) {
   // No-op if process_markup was false (and thus spans_ is empty)
   // TODO: replace this with optional<HTML> at a higher level
   if (spans_.empty()) return;
 
   // We need alignment info to transfer the HTML tags from the input to the
   // translation. If those are not available, no HTML in translations for you.
-  ABORT_UNLESS(HasAlignments(response),
+  ABORT_UNLESS(hasAlignments(response),
                "Response object does not contain alignments. TranslationModel or ResponseOptions is misconfigured?");
 
   // Reconstruction of HTML tags:
@@ -540,27 +548,26 @@ void HTML::Restore(Response &response) {
                                   // Calculating these is a side-effect of restoring
                                   // the HTML in response.source.
 
-  AnnotatedText source = RestoreSource(response.source, token_tags, spans_.cbegin(), spans_.cend());
-  assert(token_tags.size() == DebugCountTokens(response.source));
+  AnnotatedText source = restoreSource(response.source, token_tags, spans_.cbegin(), spans_.cend());
+  assert(token_tags.size() == debugCountTokens(response.source));
 
   // Find for every token in target the token in source that best matches.
   std::vector<std::vector<size_t>> alignments;
-  HardAlignments(response, alignments);
+  hardAlignments(response, alignments);
 
   std::vector<Taint> token_tags_target;
   token_tags_target.emplace_back();  // add empty one to the beginning for easy
                                      // life later on (we start iterating at 1,
                                      // and can then do i - 1 for empty.
-  CopyTaint(response, alignments, token_tags, token_tags_target);
-  assert(token_tags_target.size() == DebugCountTokens(response.target) + 1);
+  copyTaint(response, alignments, token_tags, token_tags_target);
+  assert(token_tags_target.size() == debugCountTokens(response.target) + 1);
 
   // DebugPrintMapping(std::cerr, response, alignments, token_tags_target);
 
-  AnnotatedText target = RestoreTarget(response.target, token_tags_target);
+  AnnotatedText target = restoreTarget(response.target, token_tags_target);
 
   response.source = source;
   response.target = target;
 }
 
-}  // namespace bergamot
-}  // namespace marian
+}  // namespace marian::bergamot
diff --git a/src/translator/html.h b/src/translator/html.h
index ba4691541..5ddb3d006 100644
--- a/src/translator/html.h
+++ b/src/translator/html.h
@@ -34,7 +34,7 @@ class HTML {
   };
 
   explicit HTML(std::string &&source, bool process_markup);
-  void Restore(Response &response);
+  void restore(Response &response);
 
  private:
   // List of text spans, and which tags are applied to them
diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
index b9d163a2e..baa648850 100644
--- a/src/translator/response_builder.h
+++ b/src/translator/response_builder.h
@@ -64,7 +64,7 @@ class ResponseBuilder {
     if (responseOptions_.alignment) {
       buildAlignments(histories, response);
     }
-    html_.Restore(response);
+    html_.restore(response);
 
     callback_(std::move(response));
   }
diff --git a/src/translator/xh_scanner.cpp b/src/translator/xh_scanner.cpp
index bb72f8020..85eb7e972 100644
--- a/src/translator/xh_scanner.cpp
+++ b/src/translator/xh_scanner.cpp
@@ -9,14 +9,14 @@
 
 namespace {
 
-// Simple replacement for str.ends_with(compile-time C string)
+// Simple replacement for string_view.ends_with(compile-time C string)
 template <typename Char_t, size_t Len>
-inline bool ends_with(markup::string_ref &str, const Char_t (&suffix)[Len]) {
+inline bool endsWith(markup::string_ref &str, const Char_t (&suffix)[Len]) {
   size_t offset = str.size - (Len - 1);
   return offset <= str.size && std::memcmp(str.data + offset, suffix, Len - 1) == 0;
 }
 
-inline bool equals_case_insensitive(const char *lhs, const char *rhs, size_t len) {
+inline bool equalsCaseInsensitive(const char *lhs, const char *rhs, size_t len) {
   for (size_t i = 0; i < len; ++i) {
     // cast to unsigned char otherwise std::tolower has undefined behaviour
     if (std::tolower(static_cast<unsigned char>(lhs[i])) != std::tolower(static_cast<unsigned char>(rhs[i])))
@@ -28,8 +28,8 @@ inline bool equals_case_insensitive(const char *lhs, const char *rhs, size_t len
 
 // Alias for the above, but with compile-time known C string
 template <size_t Len>
-inline bool equals_case_insensitive(markup::string_ref &lhs, const char (&rhs)[Len]) {
-  return lhs.size == Len - 1 && equals_case_insensitive(lhs.data, rhs, Len);
+inline bool equalsCaseInsensitive(markup::string_ref &lhs, const char (&rhs)[Len]) {
+  return lhs.size == Len - 1 && equalsCaseInsensitive(lhs.data, rhs, Len - 1);
 }
 
 template <typename Char_t, size_t Len>
@@ -43,22 +43,22 @@ namespace markup {
 
 // case sensitive string equality test
 // s_lowcase shall be lowercase string
-std::string_view scanner::value() const { return std::string_view(value_.data, value_.size); }
+std::string_view Scanner::value() const { return std::string_view(value_.data, value_.size); }
 
-std::string_view scanner::attr_name() const { return std::string_view(attr_name_.data, attr_name_.size); }
+std::string_view Scanner::attribute() const { return std::string_view(attributeName_.data, attributeName_.size); }
 
-std::string_view scanner::tag_name() const { return std::string_view(tag_name_.data, tag_name_.size); }
+std::string_view Scanner::tag() const { return std::string_view(tagName_.data, tagName_.size); }
 
-scanner::token_type scanner::scan_body() {
+Scanner::TokenType Scanner::scanBody() {
   value_ = string_ref{input_.pos(), 0};
 
   switch (input_.peek()) {
     case '\0':
       return TT_EOF;
     case '<':
-      return scan_tag();
+      return scanTag();
     case '&':
-      return scan_entity(TT_TEXT);
+      return scanEntity(TT_TEXT);
   }
 
   while (true) {
@@ -79,50 +79,50 @@ scanner::token_type scanner::scan_body() {
 //   <tag attr="value">...</tag>
 //       |------------|
 // Followed by:
-// - scan_special if <script> or <style>
-// - scan_body
+// - scanSpecial if <script> or <style>
+// - scanBody
 // - another scan_head for the next attribute or end of open tag
 // Returns:
-// - TT_ATTR if attribute is read
+// - TT_ATTRIBUTE if attribute is read
 // - TT_TAG_END if self-closing tag
 // - TT_ERROR if wrong character encountered
-// - TT_EOF if unexpected end of input (will not return TT_ATTR if attribute value wasn't finished yet)
-// - TT_TAG_END through scan_special
-// - TT_TEXT through scan_body
-scanner::token_type scanner::scan_attr() {
+// - TT_EOF if unexpected end of input (will not return TT_ATTRIBUTE if attribute value wasn't finished yet)
+// - TT_TAG_END through scanSpecial
+// - TT_TEXT through scanBody
+Scanner::TokenType Scanner::scanAttribute() {
   // Skip all whitespace between tag name or last attribute and next attribute or '>'
-  skip_whitespace();
+  skipWhitespace();
 
   // Find end of tag name
   switch (input_.peek()) {
     case '>':
       input_.consume();
-      if (equals_case_insensitive(tag_name_, "script")) {
+      if (equalsCaseInsensitive(tagName_, "script")) {
         // script is special because we want to parse the attributes,
         // but not the content
-        c_scan = &scanner::scan_special;
-        return scan_special();
-      } else if (equals_case_insensitive(tag_name_, "style")) {
+        scanFun_ = &Scanner::scanSpecial;
+        return scanSpecial();
+      } else if (equalsCaseInsensitive(tagName_, "style")) {
         // same with style
-        c_scan = &scanner::scan_special;
-        return scan_special();
+        scanFun_ = &Scanner::scanSpecial;
+        return scanSpecial();
       } else {
-        c_scan = &scanner::scan_body;
-        return scan_body();
+        scanFun_ = &Scanner::scanBody;
+        return scanBody();
       }
     case '/':
       input_.consume();
       if (input_.peek() == '>') {
         // self closing tag
         input_.consume();
-        c_scan = &scanner::scan_body;
+        scanFun_ = &Scanner::scanBody;
         return TT_TAG_END;
       } else {
         return TT_ERROR;
       }
   }
 
-  attr_name_ = string_ref{input_.pos(), 0};
+  attributeName_ = string_ref{input_.pos(), 0};
   value_ = string_ref{nullptr, 0};
 
   // attribute name...
@@ -131,20 +131,26 @@ scanner::token_type scanner::scan_attr() {
       case '\0':
         return TT_EOF;
       case '>':
-        return TT_ATTR;  // attribute without value (HTML style)
+        return TT_ATTRIBUTE;  // attribute without value (HTML style) at end of tag
       case '<':
         return TT_ERROR;
       default:
-        if (skip_whitespace()) continue;
+        if (skipWhitespace()) {
+          if (input_.peek() == '=') {
+            break;
+          } else {
+            return TT_ATTRIBUTE;  // attribute without value (HTML style) but not yet at end of tag
+          }
+        }
         input_.consume();
-        ++attr_name_.size;
+        ++attributeName_.size;
         break;
     }
   }
 
   // consume '=' and any following whitespace
   input_.consume();
-  skip_whitespace();
+  skipWhitespace();
   // attribute value...
 
   char quote;  // Either '"' or '\'' depending on which quote we're searching for
@@ -158,7 +164,7 @@ scanner::token_type scanner::scan_attr() {
           return TT_ERROR;
         } else if (input_.peek() == quote) {
           input_.consume();
-          return TT_ATTR;
+          return TT_ATTRIBUTE;
         } else {
           input_.consume();
           ++value_.size;
@@ -169,8 +175,8 @@ scanner::token_type scanner::scan_attr() {
       value_ = string_ref{input_.pos(), 0};
 
       while (true) {
-        if (is_whitespace(input_.peek())) return TT_ATTR;
-        if (input_.peek() == '>') return TT_ATTR;  // '>' will be consumed next round
+        if (isWhitespace(input_.peek())) return TT_ATTRIBUTE;
+        if (input_.peek() == '>') return TT_ATTRIBUTE;  // '>' will be consumed next round
         input_.consume();
         ++value_.size;
       }
@@ -187,28 +193,34 @@ scanner::token_type scanner::scan_attr() {
 // Emits:
 // - TT_TAG_START if tag head is read
 // - TT_COMMENT_START
+// - TT_PROCESSING_INSTRUCTION_START
 // - TT_CDATA_START
 // - TT_ENTITY_START
 // - TT_ERROR if unexpected character or end
-scanner::token_type scanner::scan_tag() {
+Scanner::TokenType Scanner::scanTag() {
   if (input_.consume() != '<') return TT_ERROR;
 
   bool is_tail = input_.peek() == '/';
   if (is_tail) input_.consume();
 
-  tag_name_ = string_ref{input_.pos(), 0};
+  tagName_ = string_ref{input_.pos(), 0};
 
   while (input_.peek()) {
-    if (skip_whitespace()) break;
+    if (skipWhitespace()) break;
 
     if (input_.peek() == '/' || input_.peek() == '>') break;
 
     input_.consume();
-    ++tag_name_.size;
+    ++tagName_.size;
 
-    if (tag_name_ == "!--") {
-      c_scan = &scanner::scan_comment;
+    // Note: these tests are executed at every char, thus eager.
+    // "<?xml" will match on `tagName_ == "?"`.
+    if (tagName_ == "!--") {
+      scanFun_ = &Scanner::scanComment;
       return TT_COMMENT_START;
+    } else if (tagName_ == "?") {
+      scanFun_ = &Scanner::scanProcessingInstruction;
+      return TT_PROCESSING_INSTRUCTION_START;
     }
   }
 
@@ -216,14 +228,14 @@ scanner::token_type scanner::scan_tag() {
 
   if (is_tail) return input_.consume() == '>' ? TT_TAG_END : TT_ERROR;
 
-  c_scan = &scanner::scan_attr;
+  scanFun_ = &Scanner::scanAttribute;
   return TT_TAG_START;
 }
 
-scanner::token_type scanner::scan_entity(token_type parent_token_type) {
+Scanner::TokenType Scanner::scanEntity(TokenType parentTokenType) {
   // `entity` includes starting '&' and ending ';'
   string_ref entity{input_.pos(), 0};
-  bool has_end = false;
+  bool hasEnd = false;
 
   if (input_.consume() != '&') return TT_ERROR;
 
@@ -234,10 +246,10 @@ scanner::token_type scanner::scan_entity(token_type parent_token_type) {
     if (input_.peek() == ';') {
       input_.consume();
       ++entity.size;
-      has_end = true;
+      hasEnd = true;
       break;
     } else if (!isalpha(input_.peek())) {
-      has_end = false;
+      hasEnd = false;
       break;
     } else {
       input_.consume();
@@ -246,14 +258,14 @@ scanner::token_type scanner::scan_entity(token_type parent_token_type) {
   }
 
   // If we can decode the entity, do so
-  if (has_end && resolve_entity(entity, value_)) return parent_token_type;
+  if (hasEnd && resolveEntity(entity, value_)) return parentTokenType;
 
   // Otherwise, just yield the whole thing undecoded, interpret it as text
   value_ = entity;
-  return parent_token_type;
+  return parentTokenType;
 }
 
-bool scanner::resolve_entity(string_ref const &buffer, string_ref &decoded) const {
+bool Scanner::resolveEntity(string_ref const &buffer, string_ref &decoded) const {
   static char lt = '<', gt = '>', amp = '&', quot = '"', apos = '\'', nbsp = ' ';
 
   if (buffer == "&lt;") {
@@ -285,23 +297,23 @@ bool scanner::resolve_entity(string_ref const &buffer, string_ref &decoded) cons
 
 // skip whitespaces.
 // returns how many whitespaces were skipped
-size_t scanner::skip_whitespace() {
+size_t Scanner::skipWhitespace() {
   size_t skipped = 0;
-  while (is_whitespace(input_.peek())) {
+  while (isWhitespace(input_.peek())) {
     input_.consume();
     ++skipped;
   }
   return skipped;
 }
 
-bool scanner::is_whitespace(char c) {
+bool Scanner::isWhitespace(char c) {
   return c <= ' ' && (c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\f');
 }
 
-scanner::token_type scanner::scan_comment() {
-  if (got_tail) {
-    c_scan = &scanner::scan_body;
-    got_tail = false;
+Scanner::TokenType Scanner::scanComment() {
+  if (gotTail_) {
+    scanFun_ = &Scanner::scanBody;
+    gotTail_ = false;
     return TT_COMMENT_END;
   }
 
@@ -311,8 +323,8 @@ scanner::token_type scanner::scan_comment() {
     if (input_.consume() == '\0') return TT_EOF;
     ++value_.size;
 
-    if (ends_with(value_, "-->")) {
-      got_tail = true;
+    if (endsWith(value_, "-->")) {
+      gotTail_ = true;
       value_.size -= 3;
       break;
     }
@@ -320,10 +332,32 @@ scanner::token_type scanner::scan_comment() {
   return TT_DATA;
 }
 
-scanner::token_type scanner::scan_special() {
-  if (got_tail) {
-    c_scan = &scanner::scan_body;
-    got_tail = false;
+Scanner::TokenType Scanner::scanProcessingInstruction() {
+  if (gotTail_) {
+    scanFun_ = &Scanner::scanBody;
+    gotTail_ = false;
+    return TT_PROCESSING_INSTRUCTION_END;
+  }
+
+  value_ = string_ref{input_.pos(), 0};
+
+  while (true) {
+    if (input_.consume() == '\0') return TT_EOF;
+    ++value_.size;
+
+    if (endsWith(value_, "?>")) {
+      gotTail_ = true;
+      value_.size -= 2;
+      break;
+    }
+  }
+  return TT_DATA;
+}
+
+Scanner::TokenType Scanner::scanSpecial() {
+  if (gotTail_) {
+    scanFun_ = &Scanner::scanBody;
+    gotTail_ = false;
     return TT_TAG_END;
   }
 
@@ -335,17 +369,17 @@ scanner::token_type scanner::scan_special() {
 
     // Test for </tag>
     // TODO: no whitespaces allowed? Is that okay?
-    if (value_.data[value_.size - 1] == '>' && value_.size >= tag_name_.size + 3) {
+    if (value_.data[value_.size - 1] == '>' && value_.size >= tagName_.size + 3) {
       // Test for the "</"" bit of "</tag>"
-      size_t pos_tag_start = value_.size - tag_name_.size - 3;
+      size_t pos_tag_start = value_.size - tagName_.size - 3;
       if (std::memcmp(value_.data + pos_tag_start, "</", 2) != 0) continue;
 
       // Test for the "tag" bit of "</tag>". Doing case insensitive compare because <I>...</i> is okay.
-      size_t pos_tag_name = value_.size - tag_name_.size - 1;  // end - tag>
-      if (!equals_case_insensitive(value_.data + pos_tag_name, tag_name_.data, tag_name_.size)) continue;
+      size_t pos_tag_name = value_.size - tagName_.size - 1;  // end - tag>
+      if (!equalsCaseInsensitive(value_.data + pos_tag_name, tagName_.data, tagName_.size)) continue;
 
-      got_tail = true;
-      value_.size -= tag_name_.size + 3;
+      gotTail_ = true;
+      value_.size -= tagName_.size + 3;
       break;
     }
   }
diff --git a/src/translator/xh_scanner.h b/src/translator/xh_scanner.h
index de7d762ab..14d755bbd 100644
--- a/src/translator/xh_scanner.h
+++ b/src/translator/xh_scanner.h
@@ -28,80 +28,115 @@ struct string_ref {
   size_t size;
 };
 
-class scanner {
+class Scanner {
  public:
-  enum token_type {
+  enum TokenType {
     TT_ERROR = -1,
     TT_EOF = 0,
 
-    TT_TAG_START,  // <tag ...
-    //     ^-- happens here
-    TT_TAG_END,  // </tag>
-    //       ^-- happens here
-    // <tag ... />
-    //            ^-- or here
-    TT_ATTR,  // <tag attr="value" >
-    //                  ^-- happens here
-    TT_TEXT,
-
-    TT_DATA,  // content of followings:
-    // (also content of TT_TAG_START and TT_TAG_END, if the tag is 'script' or 'style')
-
-    TT_COMMENT_START,
-    TT_COMMENT_END,  // after "<!--" and "-->"
+    TT_TAG_START,                     // <tag ...
+                                      //     ^-- happens here
+                                      //
+    TT_TAG_END,                       // </tag>
+                                      //       ^-- happens here
+                                      // <tag ... />
+                                      //            ^-- or here
+                                      //
+    TT_ATTRIBUTE,                     // <tag attr="value" >
+                                      //                 ^-- happens here, attr_name() and value()
+                                      //                     will be filled with 'attr' and 'value'.
+                                      //
+    TT_TEXT,                          // <tag>xxx</tag>
+                                      //         ^-- happens here
+                                      // <tag>foo &amp;&amp; bar</tag>
+                                      //          ^---^----^----^-- and all of here as well
+                                      // Comes after TT_TAG_START or as the first token if the input
+                                      // begins with text instead of a root element.
+                                      //
+    TT_DATA,                          // <!-- foo -->
+                                      //         ^-- here
+                                      // <? ... ?>
+                                      //       ^-- as well as here
+                                      // <script>...</script>
+                                      //            ^-- or here
+                                      // <style>...</style>
+                                      //           ^-- or here
+                                      // comes after TT_COMMENT_START, TT_PI_START, or TT_TAG_START
+                                      // if the tag was <script> or <style>.
+                                      //
+    TT_COMMENT_START,                 // <!-- foo -->
+                                      //     ^-- happens here
+                                      //
+    TT_COMMENT_END,                   // <!-- foo -->
+                                      //             ^-- happens here
+                                      //
+    TT_PROCESSING_INSTRUCTION_START,  // <?xml version="1.0?>
+                                      //   ^-- happens here
+                                      //
+    TT_PROCESSING_INSTRUCTION_END,    // <?xml version="1.0?>
+                                      //                     ^-- would you believe this happens here
   };
 
  public:
-  explicit scanner(instream &is) : input_(is), got_tail(false) { c_scan = &scanner::scan_body; }
+  explicit Scanner(instream &is)
+      : value_{nullptr, 0},
+        tagName_{nullptr, 0},
+        attributeName_{nullptr, 0},
+        input_(is),
+        scanFun_(&Scanner::scanBody),
+        gotTail_(false) {}
 
   // get next token
-  token_type next_token() { return (this->*c_scan)(); }
+  TokenType next() { return (this->*scanFun_)(); }
 
   // get value of TT_TEXT, TT_ATTR and TT_DATA
   std::string_view value() const;
 
   // get attribute name
-  std::string_view attr_name() const;
+  std::string_view attribute() const;
 
   // get tag name
-  std::string_view tag_name() const;
+  std::string_view tag() const;
 
  private: /* methods */
-  typedef token_type (scanner::*scan)();
-
-  scan c_scan;  // current 'reader'
+  typedef TokenType (Scanner::*ScanPtr)();
 
   // Consumes the text around and between tags
-  token_type scan_body();
+  TokenType scanBody();
 
   // Consumes name="attr"
-  token_type scan_attr();
+  TokenType scanAttribute();
 
   // Consumes <!-- ... -->
-  token_type scan_comment();
+  TokenType scanComment();
+
+  // Consumes <?name [attrs]?>
+  TokenType scanProcessingInstruction();
 
   // Consumes ...</style> and ...</script>
-  token_type scan_special();
+  TokenType scanSpecial();
 
   // Consumes <tagname and </tagname>
-  token_type scan_tag();
+  TokenType scanTag();
 
   // Consumes '&amp;' etc, emits parent_token_type
-  token_type scan_entity(token_type parent_token_type);
+  TokenType scanEntity(TokenType parentTokenType);
 
-  size_t skip_whitespace();
+  size_t skipWhitespace();
 
-  bool resolve_entity(string_ref const &buffer, string_ref &decoded) const;
+  bool resolveEntity(string_ref const &buffer, string_ref &decoded) const;
 
-  static bool is_whitespace(char c);
+  static bool isWhitespace(char c);
 
  private: /* data */
   string_ref value_;
-  string_ref tag_name_;
-  string_ref attr_name_;
+  string_ref tagName_;
+  string_ref attributeName_;
+
+  ScanPtr scanFun_;  // current 'reader'
 
   instream &input_;
 
-  bool got_tail;  // aux flag used in scan_comment
+  bool gotTail_;  // aux flag used in scanComment, scanSpecial, scanProcessingInstruction
 };
 }  // namespace markup

From bcbbfe129525ed2be8dbf00d2da9d412667f1d8d Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Tue, 21 Dec 2021 09:22:37 +0000
Subject: [PATCH 320/442] Better command-line with isolation for both Services
 and co-located defaults and parsing (#252)

* CLI Rework

* Consolidate common tests, template specialize CLI

* Remove remnant cache stuff

* [BRT]: Run BRT with new cli

* Formalizing bridge

* Removing stuff from parsing and moving to TestSuite

* Template includes, everything consolidating at tests

* Inlining readFromStdin

* Removing unnecessary headers

* Checking in template implementation which was missing

* Sane defaults, some catches at BRT

* BRT: Install fixes

* Updating marian-dev to point to main

* Removing the enum indirection, using strings at one place, directly

* Fix typo;

* [BRT] test blocking service via native

* Conservative defaults for workers and cache-mutex buckets in AsyncService

* Create proper barriers for cmdline app

* Build failure fixes

* Moving common, common-impl to a familiar structure

* Binary reorganization: async, blocking, wasm

- async tests AsyncService
- blocking tests BlockingService
- wasm arranges tests for things that are Mozilla requirements. eg:
    - bytearray
    - multiple sentences in same translate request workflow.

* [brt] updates to adapt to cli rework

* [brt] updates to adapt to cli rework, all working

* Empty commit, sync brt online and run GitHub CI

* Switch for parser to have multiple mode or not

* [brt]: Fix for --bergamot-mode being removed from CLI app

* [brt]: Fix for --bergamot-mode being removed from CLI app

* [brt]: Removing remnant faithful translation test from blocking/
---
 app/bergamot.cpp                              |  52 +++--
 app/cli.h                                     | 174 ----------------
 bergamot-translator-tests                     |   2 +-
 src/tests/CMakeLists.txt                      |  20 +-
 src/tests/apps.cpp                            | 146 -------------
 src/tests/apps.h                              |  46 -----
 src/tests/async.cpp                           |  27 +++
 src/tests/blocking.cpp                        |  25 +++
 src/tests/cli.cpp                             |  53 -----
 src/tests/common-impl.cpp                     | 192 ++++++++++++++++++
 src/tests/common.h                            |  88 ++++++++
 ...ntgemm_resolve.cpp => intgemm-resolve.cpp} |   0
 src/tests/wasm.cpp                            |  53 +++++
 src/translator/parser.cpp                     |  84 --------
 src/translator/parser.h                       |  96 +++++----
 src/translator/service.h                      |  22 +-
 src/translator/utils.h                        |  15 ++
 17 files changed, 522 insertions(+), 573 deletions(-)
 delete mode 100644 app/cli.h
 delete mode 100644 src/tests/apps.cpp
 delete mode 100644 src/tests/apps.h
 create mode 100644 src/tests/async.cpp
 create mode 100644 src/tests/blocking.cpp
 delete mode 100644 src/tests/cli.cpp
 create mode 100644 src/tests/common-impl.cpp
 create mode 100644 src/tests/common.h
 rename src/tests/{intgemm_resolve.cpp => intgemm-resolve.cpp} (100%)
 create mode 100644 src/tests/wasm.cpp
 create mode 100644 src/translator/utils.h

diff --git a/app/bergamot.cpp b/app/bergamot.cpp
index bffbbb112..5629f9110 100644
--- a/app/bergamot.cpp
+++ b/app/bergamot.cpp
@@ -1,22 +1,42 @@
-#include "cli.h"
+#include "translator/byte_array_util.h"
+#include "translator/parser.h"
+#include "translator/response.h"
+#include "translator/response_options.h"
+#include "translator/service.h"
+#include "translator/utils.h"
 
 int main(int argc, char *argv[]) {
-  marian::bergamot::ConfigParser configParser;
+  using namespace marian::bergamot;
+  ConfigParser<AsyncService> configParser("Bergamot CLI", /*multiOpMode=*/false);
   configParser.parseArgs(argc, argv);
   auto &config = configParser.getConfig();
-  using namespace marian::bergamot;
-  switch (config.opMode) {
-    case OpMode::APP_WASM:
-      app::wasm(config);
-      break;
-    case OpMode::APP_NATIVE:
-      app::native(config);
-      break;
-    case OpMode::APP_DECODER:
-      app::decoder(config);
-      break;
-    default:
-      break;
-  }
+
+  AsyncService service(config.serviceConfig);
+
+  // Construct a model.
+  auto options = parseOptionsFromFilePath(config.modelConfigPaths.front());
+
+  MemoryBundle memoryBundle;
+  std::shared_ptr<TranslationModel> model = service.createCompatibleModel(options, std::move(memoryBundle));
+
+  ResponseOptions responseOptions;
+  std::string input = readFromStdin();
+
+  // Create a barrier using future/promise.
+  std::promise<Response> promise;
+  std::future<Response> future = promise.get_future();
+  auto callback = [&promise](Response &&response) {
+    // Fulfill promise.
+    promise.set_value(std::move(response));
+  };
+
+  service.translate(model, std::move(input), callback, responseOptions);
+
+  // Wait until promise sets the response.
+  Response response = future.get();
+
+  // Print (only) translated text.
+  std::cout << response.target.text;
+
   return 0;
 }
diff --git a/app/cli.h b/app/cli.h
deleted file mode 100644
index 08f203466..000000000
--- a/app/cli.h
+++ /dev/null
@@ -1,174 +0,0 @@
-#ifndef BERGAMOT_APP_CLI_H
-#define BERGAMOT_APP_CLI_H
-#include <algorithm>
-#include <cstdlib>
-#include <future>
-#include <iostream>
-#include <sstream>
-
-#include "common/definitions.h"
-#include "common/timer.h"
-#include "common/utils.h"
-#include "marian.h"
-#include "translator/byte_array_util.h"
-#include "translator/parser.h"
-#include "translator/response.h"
-#include "translator/response_options.h"
-#include "translator/service.h"
-
-namespace marian {
-namespace bergamot {
-
-// marian::bergamot:: makes life easier, won't need to prefix it everywhere and these classes plenty use constructs.
-
-namespace app {
-
-/// Previously bergamot-translator-app. Provides a command-line app on native which executes the code-path used by Web
-/// Assembly. Expected to be maintained consistent with how the browser (Mozilla through WebAssembly) dictates its API
-/// and tests be intact. Also used in [bergamot-evaluation](https://github.com/mozilla/bergamot-evaluation).
-///
-/// Usage example:
-/// [brt/tests/basic/test_bergamot_translator_app_intgemm_8bit.cpu-threads.0.sh](https://github.com/browsermt/bergamot-translator-tests/blob/main/tests/basic/test_bergamot_translator_app_intgemm_8bit.cpu-threads.0.sh)
-///
-/// * Input : read from stdin as sentences as lines of text.
-/// * Output: written to stdout as translations for the sentences supplied in corresponding lines
-///
-/// @param [options]: Options to translate passed down to marian through Options.
-void wasm(const CLIConfig &config) {
-  // Here, we take the command-line interface which is uniform across all apps. This is parsed into Ptr<Options> by
-  // marian. However, mozilla does not allow a Ptr<Options> constructor and demands an std::string constructor since
-  // std::string isn't marian internal unlike Ptr<Options>. Since this std::string path needs to be tested for mozilla
-  // and since this class/CLI is intended at testing mozilla's path, we go from:
-  //
-  // cmdline -> Ptr<Options> -> std::string -> TranslationModel(std::string)
-  //
-  // Overkill, yes.
-
-  const std::string &modelConfigPath = config.modelConfigPaths.front();
-
-  Ptr<Options> options = parseOptionsFromFilePath(modelConfigPath);
-  MemoryBundle memoryBundle = getMemoryBundleFromConfig(options);
-
-  BlockingService::Config serviceConfig;
-  BlockingService service(serviceConfig);
-
-  std::shared_ptr<TranslationModel> translationModel =
-      std::make_shared<TranslationModel>(options->asYamlString(), std::move(memoryBundle));
-
-  ResponseOptions responseOptions;
-  if (config.html) {
-    responseOptions.HTML = true;
-    responseOptions.alignment = true;  // Necessary for HTML
-  }
-  std::vector<std::string> texts;
-
-  // Hide the translateMultiple operation
-  for (std::string line; std::getline(std::cin, line);) {
-    texts.emplace_back(line);
-  }
-
-  auto results = service.translateMultiple(translationModel, std::move(texts), responseOptions);
-
-  for (auto &result : results) {
-    std::cout << result.getTranslatedText() << std::endl;
-  }
-}
-
-/// Application used to benchmark with marian-decoder from time-to-time. The implementation in this repository follows a
-/// different route than marian-decoder  and routinely needs to be checked that the speeds while operating similar to
-/// marian-decoder are not affected during the course of development.
-///
-/// Example usage:
-/// [brt/speed-tests/test_wngt20_perf.sh](https://github.com/browsermt/bergamot-translator-tests/blob/main/speed-tests/test_wngt20_perf.sh).
-///
-/// Expected to be compatible with Translator[1] and marian-decoder[2].
-///
-/// - [1]
-/// [marian-dev/../src/translator/translator.h](https://github.com/marian-nmt/marian-dev/blob/master/src/translator/translator.h)
-/// - [2]
-/// [marian-dev/../src/command/marian_decoder.cpp](https://github.com/marian-nmt/marian/blob/master/src/command/marian_decoder.cpp)
-///
-/// * Input: stdin, lines containing sentences, same as marian-decoder.
-/// * Output: to stdout, translations of the sentences supplied via stdin in corresponding lines
-///
-/// @param [in] options: constructed from command-line supplied arguments
-void decoder(const CLIConfig &config) {
-  marian::timer::Timer decoderTimer;
-  AsyncService::Config asyncConfig{config.numWorkers};
-  AsyncService service(asyncConfig);
-  auto options = parseOptionsFromFilePath(config.modelConfigPaths.front());
-  MemoryBundle memoryBundle;
-  Ptr<TranslationModel> translationModel = service.createCompatibleModel(options, std::move(memoryBundle));
-  // Read a large input text blob from stdin
-  std::ostringstream std_input;
-  std_input << std::cin.rdbuf();
-  std::string input = std_input.str();
-
-  // Wait on future until Response is complete
-  std::promise<Response> responsePromise;
-  std::future<Response> responseFuture = responsePromise.get_future();
-  auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
-
-  service.translate(translationModel, std::move(input), std::move(callback));
-  responseFuture.wait();
-  const Response &response = responseFuture.get();
-
-  for (size_t sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
-    std::cout << response.target.sentence(sentenceIdx) << "\n";
-  }
-
-  std::cerr << "Total time: " << std::setprecision(5) << decoderTimer.elapsed() << "s wall" << std::endl;
-}
-
-/// Command line interface to the test the features being developed as part of bergamot C++ library on native platform.
-///
-/// Usage example:
-/// [brt/tests/basic/test_service-cli_intgemm_8bit.cpu-threads.4.sh](https://github.com/browsermt/bergamot-translator-tests/blob/main/tests/basic/test_service-cli_intgemm_8bit.cpu-threads.4.sh)
-///
-/// * Input: reads from stdin, blob of text, read as a whole ; sentence-splitting etc handled internally.
-/// * Output: to stdout, translation of the source text faithful to source structure.
-///
-/// @param [in] options: options to build translator
-void native(const CLIConfig &config) {
-  AsyncService::Config asyncConfig{config.numWorkers};
-  AsyncService service(asyncConfig);
-
-  auto options = parseOptionsFromFilePath(config.modelConfigPaths.front());
-  // Prepare memories for bytearrays (including model, shortlist and vocabs)
-  MemoryBundle memoryBundle;
-  if (config.byteArray) {
-    // Load legit values into bytearrays.
-    memoryBundle = getMemoryBundleFromConfig(options);
-  }
-
-  Ptr<TranslationModel> translationModel = service.createCompatibleModel(options, std::move(memoryBundle));
-
-  // Read a large input text blob from stdin
-  std::ostringstream std_input;
-  std_input << std::cin.rdbuf();
-  std::string input = std_input.str();
-
-  ResponseOptions responseOptions;
-  if (config.html) {
-    responseOptions.HTML = true;
-    responseOptions.alignment = true;  // Necessary for HTML
-  }
-
-  // Wait on future until Response is complete
-  std::promise<Response> responsePromise;
-  std::future<Response> responseFuture = responsePromise.get_future();
-  auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
-
-  service.translate(translationModel, std::move(input), std::move(callback), responseOptions);
-  responseFuture.wait();
-  Response response = responseFuture.get();
-
-  std::cout << response.target.text;
-}
-
-}  // namespace app
-
-}  // namespace bergamot
-}  // namespace marian
-
-#endif  // BERGAMOT_APP_CLI_H
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 9344b9835..5524e37a0 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 9344b9835797f7c19ee49d30bff134b74a1a336e
+Subproject commit 5524e37a01920dc5149dcc87b047615c6a70aa53
diff --git a/src/tests/CMakeLists.txt b/src/tests/CMakeLists.txt
index 483bd075f..86fe00236 100644
--- a/src/tests/CMakeLists.txt
+++ b/src/tests/CMakeLists.txt
@@ -13,20 +13,12 @@ endif (COMPILE_UNIT_TESTS)
 
 if(NOT MSVC)
   # Testing apps
-  set(APP_TESTS)
-  add_executable("bergamot-test" "cli.cpp" "apps.cpp")
-  
-  if(CUDA_FOUND)
-    target_link_libraries("bergamot-test" bergamot-translator)
-  else(CUDA_FOUND)
-    target_link_libraries("bergamot-test" bergamot-translator)
-  endif(CUDA_FOUND)
-  
-  set_target_properties("bergamot-test" PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}")
+  set(TEST_BINARIES async blocking intgemm-resolve wasm)
+  foreach(binary ${TEST_BINARIES})
+      add_executable("${binary}" "${binary}.cpp")
+      target_link_libraries("${binary}" bergamot-translator)
+      set_target_properties("${binary}" PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/tests/")
+  endforeach(binary)
 
-  # Adding an intgemm_resolve cmdline
-  add_executable(intgemm-resolve intgemm_resolve.cpp)
-  target_link_libraries(intgemm-resolve PRIVATE bergamot-translator)
-  set_target_properties(intgemm-resolve PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}")
 endif(NOT MSVC)
 
diff --git a/src/tests/apps.cpp b/src/tests/apps.cpp
deleted file mode 100644
index 20c6d2acb..000000000
--- a/src/tests/apps.cpp
+++ /dev/null
@@ -1,146 +0,0 @@
-#include "apps.h"
-
-namespace marian {
-namespace bergamot {
-
-namespace {
-
-std::string readFromStdin() {
-  // Read a large input text blob from stdin
-  std::ostringstream inputStream;
-  inputStream << std::cin.rdbuf();
-  std::string input = inputStream.str();
-  return input;
-}
-
-// Utility function, common for all testapps.
-Response translateForResponse(AsyncService &service, Ptr<TranslationModel> model, std::string &&source,
-                              ResponseOptions responseOptions) {
-  std::promise<Response> responsePromise;
-  std::future<Response> responseFuture = responsePromise.get_future();
-
-  auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
-  service.translate(model, std::move(source), callback, responseOptions);
-
-  responseFuture.wait();
-
-  Response response = responseFuture.get();
-  return response;
-}
-
-}  // namespace
-
-namespace testapp {
-
-void annotatedTextWords(AsyncService &service, Ptr<TranslationModel> model, bool sourceSide) {
-  ResponseOptions responseOptions;
-  std::string source = readFromStdin();
-  Response response = translateForResponse(service, model, std::move(source), responseOptions);
-  AnnotatedText &annotatedText = sourceSide ? response.source : response.target;
-  for (size_t s = 0; s < annotatedText.numSentences(); s++) {
-    for (size_t w = 0; w < annotatedText.numWords(s); w++) {
-      std::cout << (w == 0 ? "" : "\t");
-      std::cout << annotatedText.word(s, w);
-    }
-    std::cout << "\n";
-  }
-}
-
-void annotatedTextSentences(AsyncService &service, Ptr<TranslationModel> model, bool sourceSide) {
-  ResponseOptions responseOptions;
-  std::string source = readFromStdin();
-  Response response = translateForResponse(service, model, std::move(source), responseOptions);
-  AnnotatedText &annotatedText = sourceSide ? response.source : response.target;
-  for (size_t s = 0; s < annotatedText.numSentences(); s++) {
-    std::cout << annotatedText.sentence(s) << "\n";
-  }
-}
-
-void forwardAndBackward(AsyncService &service, std::vector<Ptr<TranslationModel>> &models) {
-  ABORT_IF(models.size() != 2, "Forward and backward test needs two models.");
-  ResponseOptions responseOptions;
-  std::string source = readFromStdin();
-  Response forwardResponse = translateForResponse(service, models.front(), std::move(source), responseOptions);
-
-  // Make a copy of target
-  std::string target = forwardResponse.target.text;
-  Response backwardResponse = translateForResponse(service, models.back(), std::move(target), responseOptions);
-
-  // Print both onto the command-line
-  std::cout << forwardResponse.source.text;
-  std::cout << "----------------\n";
-  std::cout << forwardResponse.target.text;
-  std::cout << "----------------\n";
-  std::cout << backwardResponse.target.text;
-}
-
-void qualityEstimatorWords(AsyncService &service, Ptr<TranslationModel> model) {
-  ResponseOptions responseOptions;
-  responseOptions.qualityScores = true;
-  std::string source = readFromStdin();
-  const Response response = translateForResponse(service, model, std::move(source), responseOptions);
-
-  for (const auto &sentenceQualityEstimate : response.qualityScores) {
-    std::cout << "[SentenceBegin]\n";
-
-    for (const auto &wordByteRange : sentenceQualityEstimate.wordByteRanges) {
-      const string_view word(response.target.text.data() + wordByteRange.begin, wordByteRange.size());
-      std::cout << word << "\n";
-    }
-    std::cout << "[SentenceEnd]\n\n";
-  }
-}
-
-void qualityEstimatorScores(AsyncService &service, Ptr<TranslationModel> model) {
-  ResponseOptions responseOptions;
-  responseOptions.qualityScores = true;
-
-  std::string source = readFromStdin();
-  const Response response = translateForResponse(service, model, std::move(source), responseOptions);
-
-  for (const auto &sentenceQualityEstimate : response.qualityScores) {
-    std::cout << std::fixed << std::setprecision(3) << sentenceQualityEstimate.sentenceScore << "\n";
-
-    for (const float &wordScore : sentenceQualityEstimate.wordScores) {
-      std::cout << std::fixed << std::setprecision(3) << wordScore << "\n";
-    }
-    std::cout << "\n";
-  }
-}
-
-void translationCache(AsyncService &service, Ptr<TranslationModel> model) {
-  ResponseOptions responseOptions;
-
-  // Read a large input text blob from stdin
-  const std::string source = readFromStdin();
-
-  // Round 1
-  std::string buffer = source;
-  Response firstResponse = translateForResponse(service, model, std::move(buffer), responseOptions);
-
-  auto statsFirstRun = service.cacheStats();
-  LOG(info, "Cache Hits/Misses = {}/{}", statsFirstRun.hits, statsFirstRun.misses);
-  ABORT_IF(statsFirstRun.hits != 0, "Expecting no cache hits, but hits found.");
-
-  // Round 2; There should be cache hits
-  buffer = source;
-  Response secondResponse = translateForResponse(service, model, std::move(buffer), responseOptions);
-
-  auto statsSecondRun = service.cacheStats();
-  LOG(info, "Cache Hits/Misses = {}/{}", statsSecondRun.hits, statsSecondRun.misses);
-  ABORT_IF(statsSecondRun.hits <= 0, "At least one hit expected, none found.");
-  if (statsSecondRun.hits != statsFirstRun.misses) {
-    std::cerr << "Mismatch in expected hits (Hits, Misses = " << statsSecondRun.hits << ", " << statsSecondRun.misses
-              << "). This can happen due to random eviction." << std::endl;
-  }
-
-  ABORT_IF(firstResponse.target.text != secondResponse.target.text,
-           "Recompiled string provided different output when operated with cache. On the same hardware while using "
-           "same path, this is expected to be same.");
-
-  std::cout << firstResponse.target.text;
-}
-
-}  // namespace testapp
-}  // namespace bergamot
-}  // namespace marian
diff --git a/src/tests/apps.h b/src/tests/apps.h
deleted file mode 100644
index 9e45a1caa..000000000
--- a/src/tests/apps.h
+++ /dev/null
@@ -1,46 +0,0 @@
-#ifndef BERGAMOT_SRC_TESTS_APPS_H
-#define BERGAMOT_SRC_TESTS_APPS_H
-#include <algorithm>
-#include <cstdlib>
-#include <future>
-#include <iostream>
-#include <sstream>
-
-#include "common/definitions.h"
-#include "common/timer.h"
-#include "common/utils.h"
-#include "marian.h"
-#include "translator/byte_array_util.h"
-#include "translator/parser.h"
-#include "translator/response.h"
-#include "translator/response_options.h"
-#include "translator/service.h"
-
-namespace marian {
-namespace bergamot {
-
-namespace testapp {
-
-// Reads from stdin and translates.  Prints the tokens separated by space for each sentence. Prints words from source
-// side text annotation if source=true, target annotation otherwise.
-void annotatedTextWords(AsyncService &service, Ptr<TranslationModel> model, bool source = true);
-
-// Reads from stdin and translates the read content. Prints the sentences in source or target in constructed response
-// in each line, depending on source = true or false respectively.
-void annotatedTextSentences(AsyncService &service, Ptr<TranslationModel> model, bool source = true);
-
-void forwardAndBackward(AsyncService &service, std::vector<Ptr<TranslationModel>> &models);
-
-// Reads from stdin and translates the read content. Prints the quality words for each sentence.
-void qualityEstimatorWords(AsyncService &service, Ptr<TranslationModel> model);
-
-// Reads from stdin and translates the read content. Prints the quality scores for each sentence.
-void qualityEstimatorScores(AsyncService &service, Ptr<TranslationModel> model);
-
-// Tests if cache is active and functional
-void translationCache(AsyncService &service, Ptr<TranslationModel> model);
-}  // namespace testapp
-}  // namespace bergamot
-}  // namespace marian
-
-#endif  // BERGAMOT_SRC_TESTS_APPS_H
diff --git a/src/tests/async.cpp b/src/tests/async.cpp
new file mode 100644
index 000000000..25ba334ae
--- /dev/null
+++ b/src/tests/async.cpp
@@ -0,0 +1,27 @@
+#include "common.h"
+#include "translator/parser.h"
+#include "translator/service.h"
+#include "translator/translation_model.h"
+
+using namespace marian::bergamot;
+
+int main(int argc, char *argv[]) {
+  ConfigParser<AsyncService> configParser("AsyncService test-suite", /*multiOpMode=*/true);
+  configParser.parseArgs(argc, argv);
+  auto &config = configParser.getConfig();
+
+  AsyncService service(config.serviceConfig);
+
+  std::vector<std::shared_ptr<TranslationModel>> models;
+
+  for (auto &modelConfigPath : config.modelConfigPaths) {
+    TranslationModel::Config modelConfig = parseOptionsFromFilePath(modelConfigPath);
+    std::shared_ptr<TranslationModel> model = service.createCompatibleModel(modelConfig);
+    models.push_back(model);
+  }
+
+  TestSuite<AsyncService> testSuite(service);
+  testSuite.run(config.opMode, models);
+
+  return 0;
+}
diff --git a/src/tests/blocking.cpp b/src/tests/blocking.cpp
new file mode 100644
index 000000000..3bbb45634
--- /dev/null
+++ b/src/tests/blocking.cpp
@@ -0,0 +1,25 @@
+#include "common.h"
+using namespace marian::bergamot;
+
+int main(int argc, char *argv[]) {
+  ConfigParser<BlockingService> configParser("BlockingService test-suite", /*multiOpMode=*/true);
+  configParser.parseArgs(argc, argv);
+
+  auto &config = configParser.getConfig();
+  BlockingService service(config.serviceConfig);
+
+  TestSuite<BlockingService> testSuite(service);
+  std::vector<std::shared_ptr<TranslationModel>> models;
+
+  for (auto &modelConfigPath : config.modelConfigPaths) {
+    TranslationModel::Config modelConfig = parseOptionsFromFilePath(modelConfigPath);
+    std::shared_ptr<TranslationModel> model = std::make_shared<TranslationModel>(modelConfig);
+    models.push_back(model);
+  }
+
+  /// WASM is one special case where WASM path is being checked, involving translateMultiple and a multi-line feed.
+  /// Hence we do not bind it at a single input-blob single Response constraint imposed by the TestSuite.
+  testSuite.run(config.opMode, models);
+
+  return 0;
+}
diff --git a/src/tests/cli.cpp b/src/tests/cli.cpp
deleted file mode 100644
index ba4d73218..000000000
--- a/src/tests/cli.cpp
+++ /dev/null
@@ -1,53 +0,0 @@
-#include "apps.h"
-
-int main(int argc, char *argv[]) {
-  using namespace marian::bergamot;
-  marian::bergamot::ConfigParser configParser;
-  configParser.parseArgs(argc, argv);
-  auto &config = configParser.getConfig();
-  AsyncService::Config serviceConfig;
-  serviceConfig.numWorkers = config.numWorkers;
-  serviceConfig.cacheEnabled = config.cacheEnabled;
-  serviceConfig.cacheMutexBuckets = config.cacheMutexBuckets;
-  serviceConfig.cacheSize = config.cacheSize;
-  AsyncService service(serviceConfig);
-  std::vector<std::shared_ptr<TranslationModel>> models;
-
-  for (auto &modelConfigPath : config.modelConfigPaths) {
-    TranslationModel::Config modelConfig = parseOptionsFromFilePath(modelConfigPath);
-    std::shared_ptr<TranslationModel> model = service.createCompatibleModel(modelConfig);
-    models.push_back(model);
-  }
-
-  switch (config.opMode) {
-    case OpMode::TEST_SOURCE_SENTENCES:
-      testapp::annotatedTextSentences(service, models.front(), /*source=*/true);
-      break;
-    case OpMode::TEST_TARGET_SENTENCES:
-      testapp::annotatedTextSentences(service, models.front(), /*source=*/false);
-      break;
-    case OpMode::TEST_SOURCE_WORDS:
-      testapp::annotatedTextWords(service, models.front(), /*source=*/true);
-      break;
-    case OpMode::TEST_TARGET_WORDS:
-      testapp::annotatedTextWords(service, models.front(), /*source=*/false);
-      break;
-    case OpMode::TEST_FORWARD_BACKWARD_FOR_OUTBOUND:
-      testapp::forwardAndBackward(service, models);
-      break;
-    case OpMode::TEST_QUALITY_ESTIMATOR_WORDS:
-      testapp::qualityEstimatorWords(service, models.front());
-      break;
-    case OpMode::TEST_QUALITY_ESTIMATOR_SCORES:
-      testapp::qualityEstimatorScores(service, models.front());
-      break;
-    case OpMode::TEST_TRANSLATION_CACHE:
-      testapp::translationCache(service, models.front());
-      break;
-
-    default:
-      ABORT("Incompatible op-mode. Choose one of the test modes.");
-      break;
-  }
-  return 0;
-}
diff --git a/src/tests/common-impl.cpp b/src/tests/common-impl.cpp
new file mode 100644
index 000000000..49ebfc53c
--- /dev/null
+++ b/src/tests/common-impl.cpp
@@ -0,0 +1,192 @@
+
+#ifndef BERGAMOT_TESTS_COMMON_IMPL
+#error "This is an impl file and must not be included directly!"
+#endif
+
+Response Bridge<BlockingService>::translate(BlockingService &service, std::shared_ptr<TranslationModel> &model,
+                                            std::string &&source, const ResponseOptions &responseOptions) {
+  // project source to a vector of std::string, send in, unpack the first element from
+  // vector<Response>, return.
+  std::vector<std::string> sources = {source};
+  return service.translateMultiple(model, std::move(sources), responseOptions).front();
+}
+
+Response Bridge<AsyncService>::translate(AsyncService &service, std::shared_ptr<TranslationModel> &model,
+                                         std::string &&source, const ResponseOptions &responseOptions) {
+  // downgrade to blocking via promise, future, wait and return response;
+  std::promise<Response> responsePromise;
+  std::future<Response> responseFuture = responsePromise.get_future();
+
+  auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
+  service.translate(model, std::move(source), callback, responseOptions);
+
+  responseFuture.wait();
+
+  Response response = responseFuture.get();
+  return response;
+}
+
+template <class Service>
+TestSuite<Service>::TestSuite(Service &service) : service_{service} {}
+
+template <class Service>
+void TestSuite<Service>::TestSuite::run(const std::string &opModeAsString, std::vector<Ptr<TranslationModel>> &models) {
+  if (opModeAsString == "decoder") {
+    benchmarkDecoder(models.front());
+  } else if (opModeAsString == "test-response-source-sentences") {
+    annotatedTextSentences(models.front(), /*source=*/true);
+  } else if (opModeAsString == "test-response-target-sentences") {
+    annotatedTextSentences(models.front(), /*source=*/false);
+  } else if (opModeAsString == "test-response-source-words") {
+    annotatedTextWords(models.front(), /*source=*/true);
+  } else if (opModeAsString == "test-response-target-words") {
+    annotatedTextWords(models.front(), /*source=*/false);
+  } else if (opModeAsString == "test-forward-backward") {
+    forwardAndBackward(models);
+  } else if (opModeAsString == "test-quality-estimator-words") {
+    qualityEstimatorWords(models.front());
+  } else if (opModeAsString == "test-quality-estimator-scores") {
+    qualityEstimatorScores(models.front());
+  } else if (opModeAsString == "test-translation-cache") {
+    translationCache(models.front());
+  } else {
+    std::cerr << "Incompatible test mode. Choose from the one of the valid test-modes";
+    std::abort();
+  }
+}
+
+template <class Service>
+void TestSuite<Service>::benchmarkDecoder(Ptr<TranslationModel> &model) {
+  marian::timer::Timer decoderTimer;
+  std::string source = readFromStdin();
+
+  ResponseOptions responseOptions;
+  Response response = bridge_.translate(service_, model, std::move(source), responseOptions);
+
+  for (size_t sentenceIdx = 0; sentenceIdx < response.size(); sentenceIdx++) {
+    std::cout << response.target.sentence(sentenceIdx) << "\n";
+  }
+
+  std::cerr << "Total time: " << std::setprecision(5) << decoderTimer.elapsed() << "s wall" << std::endl;
+}
+
+// Reads from stdin and translates.  Prints the tokens separated by space for each sentence. Prints words from source
+// side text annotation if source=true, target annotation otherwise.
+template <class Service>
+void TestSuite<Service>::annotatedTextWords(Ptr<TranslationModel> model, bool sourceSide /*=true*/) {
+  ResponseOptions responseOptions;
+  std::string source = readFromStdin();
+  Response response = bridge_.translate(service_, model, std::move(source), responseOptions);
+  AnnotatedText &annotatedText = sourceSide ? response.source : response.target;
+  for (size_t s = 0; s < annotatedText.numSentences(); s++) {
+    for (size_t w = 0; w < annotatedText.numWords(s); w++) {
+      std::cout << (w == 0 ? "" : "\t");
+      std::cout << annotatedText.word(s, w);
+    }
+    std::cout << "\n";
+  }
+}
+
+// Reads from stdin and translates the read content. Prints the sentences in source or target in constructed response
+// in each line, depending on source = true or false respectively.
+template <class Service>
+void TestSuite<Service>::annotatedTextSentences(Ptr<TranslationModel> model, bool sourceSide /*=true*/) {
+  ResponseOptions responseOptions;
+  std::string source = readFromStdin();
+  Response response = bridge_.translate(service_, model, std::move(source), responseOptions);
+  AnnotatedText &annotatedText = sourceSide ? response.source : response.target;
+  for (size_t s = 0; s < annotatedText.numSentences(); s++) {
+    std::cout << annotatedText.sentence(s) << "\n";
+  }
+}
+
+template <class Service>
+void TestSuite<Service>::forwardAndBackward(std::vector<Ptr<TranslationModel>> &models) {
+  ABORT_IF(models.size() != 2, "Forward and backward test needs two models.");
+  ResponseOptions responseOptions;
+  std::string source = readFromStdin();
+  Response forwardResponse = bridge_.translate(service_, models.front(), std::move(source), responseOptions);
+
+  // Make a copy of target
+  std::string target = forwardResponse.target.text;
+  Response backwardResponse = bridge_.translate(service_, models.back(), std::move(target), responseOptions);
+
+  // Print both onto the command-line
+  std::cout << forwardResponse.source.text;
+  std::cout << "----------------\n";
+  std::cout << forwardResponse.target.text;
+  std::cout << "----------------\n";
+  std::cout << backwardResponse.target.text;
+}
+
+// Reads from stdin and translates the read content. Prints the quality words for each sentence.
+template <class Service>
+void TestSuite<Service>::qualityEstimatorWords(Ptr<TranslationModel> model) {
+  ResponseOptions responseOptions;
+  responseOptions.qualityScores = true;
+  std::string source = readFromStdin();
+  const Response response = bridge_.translate(service_, model, std::move(source), responseOptions);
+
+  for (const auto &sentenceQualityEstimate : response.qualityScores) {
+    std::cout << "[SentenceBegin]\n";
+
+    for (const auto &wordByteRange : sentenceQualityEstimate.wordByteRanges) {
+      const string_view word(response.target.text.data() + wordByteRange.begin, wordByteRange.size());
+      std::cout << word << "\n";
+    }
+    std::cout << "[SentenceEnd]\n\n";
+  }
+}
+
+// Reads from stdin and translates the read content. Prints the quality scores for each sentence.
+template <class Service>
+void TestSuite<Service>::qualityEstimatorScores(Ptr<TranslationModel> model) {
+  ResponseOptions responseOptions;
+  responseOptions.qualityScores = true;
+
+  std::string source = readFromStdin();
+  const Response response = bridge_.translate(service_, model, std::move(source), responseOptions);
+
+  for (const auto &sentenceQualityEstimate : response.qualityScores) {
+    std::cout << std::fixed << std::setprecision(3) << sentenceQualityEstimate.sentenceScore << "\n";
+
+    for (const float &wordScore : sentenceQualityEstimate.wordScores) {
+      std::cout << std::fixed << std::setprecision(3) << wordScore << "\n";
+    }
+    std::cout << "\n";
+  }
+}
+
+template <class Service>
+void TestSuite<Service>::translationCache(Ptr<TranslationModel> model) {
+  ResponseOptions responseOptions;
+
+  // Read a large input text blob from stdin
+  const std::string source = readFromStdin();
+
+  // Round 1
+  std::string buffer = source;
+  Response firstResponse = bridge_.translate(service_, model, std::move(buffer), responseOptions);
+
+  auto statsFirstRun = service_.cacheStats();
+  LOG(info, "Cache Hits/Misses = {}/{}", statsFirstRun.hits, statsFirstRun.misses);
+  ABORT_IF(statsFirstRun.hits != 0, "Expecting no cache hits, but hits found.");
+
+  // Round 2; There should be cache hits
+  buffer = source;
+  Response secondResponse = bridge_.translate(service_, model, std::move(buffer), responseOptions);
+
+  auto statsSecondRun = service_.cacheStats();
+  LOG(info, "Cache Hits/Misses = {}/{}", statsSecondRun.hits, statsSecondRun.misses);
+  ABORT_IF(statsSecondRun.hits <= 0, "At least one hit expected, none found.");
+  if (statsSecondRun.hits != statsFirstRun.misses) {
+    std::cerr << "Mismatch in expected hits (Hits, Misses = " << statsSecondRun.hits << ", " << statsSecondRun.misses
+              << "). This can happen due to random eviction." << std::endl;
+  }
+
+  ABORT_IF(firstResponse.target.text != secondResponse.target.text,
+           "Recompiled string provided different output when operated with cache. On the same hardware while using "
+           "same path, this is expected to be same.");
+
+  std::cout << firstResponse.target.text;
+}
diff --git a/src/tests/common.h b/src/tests/common.h
new file mode 100644
index 000000000..dff47e483
--- /dev/null
+++ b/src/tests/common.h
@@ -0,0 +1,88 @@
+#pragma once
+#include <algorithm>
+#include <cstdlib>
+#include <future>
+#include <iostream>
+#include <sstream>
+#include <unordered_map>
+
+#include "common/definitions.h"
+#include "common/timer.h"
+#include "common/utils.h"
+#include "marian.h"
+#include "translator/byte_array_util.h"
+#include "translator/parser.h"
+#include "translator/response.h"
+#include "translator/response_options.h"
+#include "translator/service.h"
+#include "translator/utils.h"
+
+namespace marian::bergamot {
+
+/// Due to the stubborn-ness of the extension and native to not agree on API (e.g, translateMultiple vs translate),
+/// different underlying cache we have the following "bridge" at test-applications - taking into account the fact that
+/// the most commonly used primitives across both Services is a single text blob in and corresponding Response out, in a
+/// blocking fashion.
+///
+/// The following contraption constrains a single sentence to single Response parameterized by Service, in a test-suite
+/// below. This allows sharing of code for test-suite between WebAssembly's workflows and Native's workflows.
+///
+/// The intention here is to use templating to achieve the same thing an ifdef would have at compile-time. Also mandates
+/// after bridge layer, both WebAssembly and Native paths compile correctly (this does not guarantee outputs are the
+/// same through both code-paths, or that both are tested at runtime - only that both compile and work with a bridge).
+///
+/// For any complex workflows involving non-blocking concurrent translation, it is required to write something not
+/// constrained by the following.
+
+template <class Service>
+struct Bridge : public std::false_type {};
+
+template <>
+struct Bridge<BlockingService> : public std::true_type {
+  Response translate(BlockingService &service, std::shared_ptr<TranslationModel> &model, std::string &&source,
+                     const ResponseOptions &responseOptions);
+};
+
+template <>
+struct Bridge<AsyncService> : public std::true_type {
+  Response translate(AsyncService &service, std::shared_ptr<TranslationModel> &model, std::string &&source,
+                     const ResponseOptions &responseOptions);
+};
+
+template <class Service>
+class TestSuite {
+ private:
+  Bridge<Service> bridge_;
+  Service &service_;
+
+ public:
+  TestSuite(Service &service);
+  void run(const std::string &opModeAsString, std::vector<Ptr<TranslationModel>> &models);
+
+ private:
+  void benchmarkDecoder(Ptr<TranslationModel> &model);
+
+  // Reads from stdin and translates.  Prints the tokens separated by space for each sentence. Prints words from source
+  // side text annotation if source=true, target annotation otherwise.
+  void annotatedTextWords(Ptr<TranslationModel> model, bool sourceSide = true);
+
+  // Reads from stdin and translates the read content. Prints the sentences in source or target in constructed response
+  // in each line, depending on source = true or false respectively.
+  void annotatedTextSentences(Ptr<TranslationModel> model, bool sourceSide = true);
+
+  void forwardAndBackward(std::vector<Ptr<TranslationModel>> &models);
+
+  // Reads from stdin and translates the read content. Prints the quality words for each sentence.
+  void qualityEstimatorWords(Ptr<TranslationModel> model);
+
+  // Reads from stdin and translates the read content. Prints the quality scores for each sentence.
+  void qualityEstimatorScores(Ptr<TranslationModel> model);
+
+  void translationCache(Ptr<TranslationModel> model);
+};
+
+#define BERGAMOT_TESTS_COMMON_IMPL
+#include "common-impl.cpp"
+#undef BERGAMOT_TESTS_COMMON_IMPL
+
+}  // namespace marian::bergamot
diff --git a/src/tests/intgemm_resolve.cpp b/src/tests/intgemm-resolve.cpp
similarity index 100%
rename from src/tests/intgemm_resolve.cpp
rename to src/tests/intgemm-resolve.cpp
diff --git a/src/tests/wasm.cpp b/src/tests/wasm.cpp
new file mode 100644
index 000000000..9a29a20e1
--- /dev/null
+++ b/src/tests/wasm.cpp
@@ -0,0 +1,53 @@
+#include "common.h"
+using namespace marian::bergamot;
+
+void wasm(BlockingService &service, std::shared_ptr<TranslationModel> &model) {
+  ResponseOptions responseOptions;
+  std::vector<std::string> texts;
+
+  // WASM always requires HTML and alignment.
+  // TODO(jerinphilip): Fix this, bring in actual tests.
+  // responseOptions.HTML = true;
+  // responseOptions.alignment = true;  // Necessary for HTML
+
+  // Hide the translateMultiple operation
+  for (std::string line; std::getline(std::cin, line);) {
+    texts.emplace_back(line);
+  }
+
+  auto results = service.translateMultiple(model, std::move(texts), responseOptions);
+
+  for (auto &result : results) {
+    std::cout << result.getTranslatedText() << std::endl;
+  }
+}
+
+int main(int argc, char *argv[]) {
+  ConfigParser<BlockingService> configParser("WebAssembly test-suite", /*multiOpMode=*/true);
+  configParser.parseArgs(argc, argv);
+
+  auto &config = configParser.getConfig();
+  BlockingService service(config.serviceConfig);
+
+  TestSuite<BlockingService> testSuite(service);
+  std::vector<std::shared_ptr<TranslationModel>> models;
+
+  for (auto &modelConfigPath : config.modelConfigPaths) {
+    TranslationModel::Config modelConfig = parseOptionsFromFilePath(modelConfigPath);
+    // Anything WASM is expected to use the byte-array-loads. So we hard-code grabbing MemoryBundle from FS and use the
+    // MemoryBundle capable constructor.
+    MemoryBundle memoryBundle = getMemoryBundleFromConfig(modelConfig);
+    std::shared_ptr<TranslationModel> model = std::make_shared<TranslationModel>(modelConfig, std::move(memoryBundle));
+    models.push_back(model);
+  }
+
+  /// WASM is one special case where WASM path is being checked, involving translateMultiple and a multi-line feed.
+  /// Hence we do not bind it at a single input-blob single Response constraint imposed by the TestSuite.
+  if (config.opMode == "wasm") {
+    wasm(service, models.front());
+  } else {
+    testSuite.run(config.opMode, models);
+  }
+
+  return 0;
+}
diff --git a/src/translator/parser.cpp b/src/translator/parser.cpp
index e875d97e0..2636b7472 100644
--- a/src/translator/parser.cpp
+++ b/src/translator/parser.cpp
@@ -10,90 +10,6 @@
 namespace marian {
 namespace bergamot {
 
-std::istringstream &operator>>(std::istringstream &in, OpMode &mode) {
-  std::string modeString;
-  in >> modeString;
-  std::unordered_map<std::string, OpMode> table = {
-      {"wasm", OpMode::APP_WASM},
-      {"native", OpMode::APP_NATIVE},
-      {"decoder", OpMode::APP_DECODER},
-      {"test-response-source-sentences", OpMode::TEST_SOURCE_SENTENCES},
-      {"test-response-target-sentences", OpMode::TEST_TARGET_SENTENCES},
-      {"test-response-source-words", OpMode::TEST_SOURCE_WORDS},
-      {"test-response-target-words", OpMode::TEST_TARGET_WORDS},
-      {"test-quality-estimator-words", OpMode::TEST_QUALITY_ESTIMATOR_WORDS},
-      {"test-quality-estimator-scores", OpMode::TEST_QUALITY_ESTIMATOR_SCORES},
-      {"test-forward-backward", OpMode::TEST_FORWARD_BACKWARD_FOR_OUTBOUND},
-      {"test-translation-cache", OpMode::TEST_TRANSLATION_CACHE},
-  };
-
-  auto query = table.find(modeString);
-  if (query != table.end()) {
-    mode = query->second;
-  } else {
-    ABORT("Unknown mode {}", modeString);
-  }
-
-  return in;
-}
-
-ConfigParser::ConfigParser() : app_{"Bergamot Options"} {
-  addSpecialOptions(app_);
-  addOptionsBoundToConfig(app_, config_);
-};
-
-void ConfigParser::parseArgs(int argc, char *argv[]) {
-  try {
-    app_.parse(argc, argv);
-    handleSpecialOptions();
-  } catch (const CLI::ParseError &e) {
-    exit(app_.exit(e));
-  }
-}
-
-void ConfigParser::addSpecialOptions(CLI::App &app) {
-  app.add_flag("--build-info", build_info_, "Print build-info and exit");
-  app.add_flag("--version", version_, "Print version-info and exit");
-}
-
-void ConfigParser::handleSpecialOptions() {
-  if (build_info_) {
-#ifndef _MSC_VER  // cmake build options are not available on MSVC based build.
-    std::cerr << cmakeBuildOptionsAdvanced() << std::endl;
-    exit(0);
-#else   // _MSC_VER
-    ABORT("build-info is not available on MSVC based build.");
-#endif  // _MSC_VER
-  }
-
-  if (version_) {
-    std::cerr << buildVersion() << std::endl;
-    exit(0);
-  }
-}
-
-void ConfigParser::addOptionsBoundToConfig(CLI::App &app, CLIConfig &config) {
-  app.add_option("--model-config-paths", config.modelConfigPaths,
-                 "Configuration files list, can be used for pivoting multiple models or multiple model workflows");
-
-  app.add_flag("--bytearray", config.byteArray,
-               "Flag holds whether to construct service from bytearrays, only for testing purpose");
-
-  app.add_flag("--check-bytearray", config.validateByteArray,
-               "Flag holds whether to check the content of the bytearrays (true by default)");
-
-  app.add_option("--cpu-threads", config.numWorkers, "Number of worker threads to use for translation");
-
-  app_.add_option("--bergamot-mode", config.opMode, "Operating mode for bergamot: [wasm, native, decoder]");
-
-  app_.add_option("--cache-translations", config.cacheEnabled, "Whether to cache translations or not.");
-  app_.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
-  app_.add_option("--cache-mutex-buckets", config.cacheMutexBuckets,
-                  "Number of mutex buckets to control locking granularity");
-
-  app_.add_flag("--html", config.html, "Whether input and output should be HTML");
-}
-
 std::shared_ptr<marian::Options> parseOptionsFromFilePath(const std::string &configPath, bool validate /*= true*/) {
   // Read entire string and redirect to parseOptionsFromString
   std::ifstream readStream(configPath);
diff --git a/src/translator/parser.h b/src/translator/parser.h
index 1aff5dba7..793582dd0 100644
--- a/src/translator/parser.h
+++ b/src/translator/parser.h
@@ -6,6 +6,7 @@
 
 #include "3rd_party/marian-dev/src/3rd_party/CLI/CLI.hpp"
 #include "3rd_party/yaml-cpp/yaml.h"
+#include "common/build_info.h"
 #include "common/config_parser.h"
 #include "common/config_validator.h"
 #include "common/options.h"
@@ -14,36 +15,34 @@
 namespace marian {
 namespace bergamot {
 
-enum OpMode {
-  APP_WASM,
-  APP_NATIVE,
-  APP_DECODER,
-  TEST_SOURCE_SENTENCES,
-  TEST_TARGET_SENTENCES,
-  TEST_SOURCE_WORDS,
-  TEST_TARGET_WORDS,
-  TEST_QUALITY_ESTIMATOR_WORDS,
-  TEST_QUALITY_ESTIMATOR_SCORES,
-  TEST_FORWARD_BACKWARD_FOR_OUTBOUND,
-  TEST_TRANSLATION_CACHE,
-};
-
-/// Overload for CL11, convert a read from a stringstream into opmode.
-std::istringstream &operator>>(std::istringstream &in, OpMode &mode);
-
+template <class Service>
 struct CLIConfig {
+  using ServiceConfig = typename Service::Config;
   using ModelConfigPaths = std::vector<std::string>;
+
+  std::string opMode;
+
+  // For marian-models we retain the old marian-yml configs to a large extent. These are supplied as file-paths to the
+  // CLI. For multiple model workflows, we allow more than one model config to be supplied. How to process the models
+  // provided is decided by the application.
   ModelConfigPaths modelConfigPaths;
-  bool byteArray;
-  bool validateByteArray;
-  bool html;
-  size_t numWorkers;
-  OpMode opMode;
-
-  // Cache parameters
-  bool cacheEnabled{false};
-  size_t cacheSize{20};
-  size_t cacheMutexBuckets{4};
+
+  ServiceConfig serviceConfig;
+
+  /// All config in bergamot has the following templated addOptions(...) method hierarchically placing parse actions on
+  /// "option-groups" in nested structs. This allows to keep additional documentation and information on defaults
+  /// alongside. Since this is templated with App, we don't add a CLI11 dependency in any configs, thus CLI11 not coming
+  /// into the picture until the parser is instantiated.
+  template <class App>
+  static void addOptions(App &app, CLIConfig<Service> &config, bool multiOpMode = false) {
+    if (multiOpMode) {
+      app.add_option("--bergamot-mode", config.opMode, "");
+    }
+    app.add_option("--model-config-paths", config.modelConfigPaths,
+                   "Configuration files list, can be used for pivoting multiple models or multiple model workflows");
+
+    ServiceConfig::addOptions(app, config.serviceConfig);
+  };
 };
 
 /// ConfigParser for bergamot. Internally stores config options with CLIConfig. CLI11 parsing binds the parsing code to
@@ -54,21 +53,48 @@ struct CLIConfig {
 /// configParser.parseArgs(argc, argv);
 /// auto &config = configParser.getConfig();
 /// ```
+template <class Service>
 class ConfigParser {
  public:
-  ConfigParser();
-  void parseArgs(int argc, char *argv[]);
-  const CLIConfig &getConfig() { return config_; }
+  ConfigParser(const std::string &appName, bool multiOpMode = false) : app_{appName} {
+    addSpecialOptions(app_);
+    CLIConfig<Service>::addOptions(app_, config_, multiOpMode);
+  };
+  void parseArgs(int argc, char *argv[]) {
+    try {
+      app_.parse(argc, argv);
+      handleSpecialOptions();
+    } catch (const CLI::ParseError &e) {
+      exit(app_.exit(e));
+    }
+  };
+  const CLIConfig<Service> &getConfig() { return config_; }
 
  private:
   // Special Options: build-info and version. These are not taken down further, the respective logic executed and
   // program exits after.
-  void addSpecialOptions(CLI::App &app);
-  void handleSpecialOptions();
+  void addSpecialOptions(CLI::App &app) {
+    app.add_flag("--build-info", build_info_, "Print build-info and exit");
+    app.add_flag("--version", version_, "Print version-info and exit");
+  };
+
+  void handleSpecialOptions() {
+    if (build_info_) {
+#ifndef _MSC_VER  // cmake build options are not available on MSVC based build.
+      std::cerr << cmakeBuildOptionsAdvanced() << std::endl;
+      exit(0);
+#else   // _MSC_VER
+      ABORT("build-info is not available on MSVC based build.");
+#endif  // _MSC_VER
+    }
 
-  void addOptionsBoundToConfig(CLI::App &app, CLIConfig &config);
+    if (version_) {
+      std::cerr << buildVersion() << std::endl;
+      exit(0);
+    }
+  }
 
-  CLIConfig config_;
+  CLIConfig<Service> config_;
   CLI::App app_;
 
   bool build_info_{false};
@@ -79,7 +105,7 @@ std::shared_ptr<marian::Options> parseOptionsFromString(const std::string &confi
                                                         std::string pathsInSameDirAs = "");
 std::shared_ptr<marian::Options> parseOptionsFromFilePath(const std::string &config, bool validate = true);
 
-}  //  namespace bergamot
+}  // namespace bergamot
 }  //  namespace marian
 
 #endif  //  SRC_BERGAMOT_PARSER_H
diff --git a/src/translator/service.h b/src/translator/service.h
index d58a759da..383ab9885 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -33,6 +33,12 @@ class BlockingService {
     bool cacheEnabled{false};  ///< Whether to enable cache or not.
     size_t cacheSize{2000};    ///< Size in History items to be stored in the cache. Loosely corresponds to sentences to
                                /// cache in the real world.
+    template <class App>
+    static void addOptions(App &app, Config &config) {
+      // Options will come here.
+      app.add_option("--cache-translations", config.cacheEnabled, "Whether to cache translations or not.");
+      app.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
+    }
   };
   /// Construct a BlockingService with configuration loaded from an Options object. Does not require any keys, values to
   /// be set.
@@ -77,13 +83,21 @@ class BlockingService {
 class AsyncService {
  public:
   struct Config {
-    size_t numWorkers;         ///< How many worker translation threads to spawn.
+    size_t numWorkers{1};      ///< How many worker translation threads to spawn.
     bool cacheEnabled{false};  ///< Whether to enable cache or not.
     size_t cacheSize{2000};    ///< Size in History items to be stored in the cache. Loosely corresponds to sentences to
                                /// cache in the real world.
-    size_t cacheMutexBuckets;  ///< Controls the granularity of locking to reduce contention by bucketing mutexes
-                               ///< guarding cache entry read write. Optimal at min(core, numWorkers) assuming a
-                               ///< reasonably large cache-size.
+    size_t cacheMutexBuckets{1};  ///< Controls the granularity of locking to reduce contention by bucketing mutexes
+                                  ///< guarding cache entry read write. Optimal at min(core, numWorkers) assuming a
+                                  ///< reasonably large cache-size.
+    template <class App>
+    static void addOptions(App &app, Config &config) {
+      app.add_option("--cpu-threads", config.numWorkers, "Workers to form translation backend");
+      app.add_option("--cache-translations", config.cacheEnabled, "Whether to cache translations or not.");
+      app.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
+      app.add_option("--cache-mutex-buckets", config.cacheMutexBuckets,
+                     "Number of mutex buckets to control locking granularity");
+    }
   };
   /// Construct an AsyncService with configuration loaded from Options. Expects positive integer value for
   /// `cpu-threads`. Additionally requires options which configure AggregateBatchingPool.
diff --git a/src/translator/utils.h b/src/translator/utils.h
new file mode 100644
index 000000000..a35cebcbd
--- /dev/null
+++ b/src/translator/utils.h
@@ -0,0 +1,15 @@
+#pragma once
+
+#include <iostream>
+
+namespace marian::bergamot {
+
+inline std::string readFromStdin() {
+  // Read a large input text blob from stdin
+  std::ostringstream inputStream;
+  inputStream << std::cin.rdbuf();
+  std::string input = inputStream.str();
+  return input;
+}
+
+}  // namespace marian::bergamot

From f55377b6876e04a9c858c84dcdfa4a0faa361e3c Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Tue, 21 Dec 2021 14:44:04 +0100
Subject: [PATCH 321/442] HTML transfer empty elements (#283)

* Fix test case

This should now be implemented

* Remove FilterEmpty

This path wasn't used anymore anyway, empty tags just got their own spans, and never reached the stack.

* Insert skipped empty source spans into target HTML

Also refactor variable names to better match their contents and be more consistent with each other.
This implementation passes all test cases, finally!

* Fix remaining style changes

* Move HTML formatting to its own section

That code had become exact copies in three different places
---
 src/tests/units/html_tests.cpp |   4 +-
 src/translator/html.cpp        | 263 ++++++++++++++++++---------------
 2 files changed, 145 insertions(+), 122 deletions(-)

diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
index 9fd2acfc6..e3d79379f 100644
--- a/src/tests/units/html_tests.cpp
+++ b/src/tests/units/html_tests.cpp
@@ -162,7 +162,7 @@ TEST_CASE("Do not abort if the input is just empty element") {
   Response response;
   html.restore(response);
   CHECK(response.source.text == "<p></p>");
-  CHECK(response.target.text == "");  // Should be <p></p> but hey not there yet.
+  CHECK(response.target.text == "<p></p>");
 }
 
 TEST_CASE("Test case html entities") {
@@ -388,7 +388,7 @@ TEST_CASE("Test empty self-closing pair at end of input in parent") {
   CHECK(input == "hello ");
 }
 
-TEST_CASE("Test empty tag", "[!mayfail]") {
+TEST_CASE("Test empty tag") {
   std::string test_str(
       "<p id=\"1\">hello <img id=\"1.1\"><span id=\"1.2\"><u id=\"1.2.1\"></u><b id=\"1.2.2\"></b><img "
       "id=\"1.2.3\">world</span></p>\n");
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index f531b44fe..4424241c2 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -12,7 +12,7 @@ using marian::bergamot::Response;
 
 void encodeEntities(string_view const &input, std::string &output) {
   output.clear();
-  output.reserve(input.size());
+  output.reserve(input.size());  // assumes there are no entities in most cases
 
   for (auto it = input.begin(); it != input.end(); ++it) {
     switch (*it) {
@@ -83,6 +83,21 @@ std::string format(std::string const &formatTemplate, Arg arg, Args... args) {
   return os.str();
 }
 
+// Syntactic sugar around rbegin() and rend() that allows me to write
+// `for (auto &&item : reversed(container))` instead of the needlessly verbose
+// `for (auto it = container.rbegin(); it != container.rend(); ++it)`
+template <typename T>
+class reversed {
+ public:
+  typedef typename T::const_reverse_iterator iterator;
+  explicit reversed(T const &container) : container_(container){};
+  iterator begin() const { return container_.rbegin(); }
+  iterator end() const { return container_.rend(); }
+
+ private:
+  T const &container_;
+};
+
 bool isBlockElement(std::string_view const &name) {
   // List of elements that we expect might occur inside words, and that should
   // not introduce spacings around them. Not strictly inline elements, nor flow
@@ -125,16 +140,6 @@ bool intersects(ByteRange const &range, HTML::Span const &span) {
   return range.begin <= span.end && range.end >= span.begin;
 };
 
-void filterEmpty(HTML::Taint &stack) {
-  auto src = stack.begin();
-  auto dst = stack.begin();
-
-  for (auto src = stack.begin(); src != stack.end(); ++src)
-    if (!(*src)->empty) *(dst++) = *src;
-
-  stack.resize(dst - stack.begin());
-}
-
 bool containsTag(HTML::Taint const &stack, HTML::Tag const *tag) {
   return std::find(stack.rbegin(), stack.rend(), tag) != stack.rend();
 }
@@ -159,11 +164,11 @@ AnnotatedText apply(AnnotatedText const &in, Fun fun) {
     // expects
     // TODO: extend AnnotatedText::appendSentence to accept str + ByteRanges
     // directly
-    std::vector<string_view> token_views(tokens.size());
-    std::transform(tokens.begin(), tokens.end(), token_views.begin(),
+    std::vector<string_view> views(tokens.size());
+    std::transform(tokens.begin(), tokens.end(), views.begin(),
                    [&](ByteRange const &range) { return string_view(sentence.data() + range.begin, range.size()); });
 
-    out.appendSentence(prefix, token_views.begin(), token_views.end());
+    out.appendSentence(prefix, views.begin(), views.end());
   }
 
   out.appendEndingWhitespace(fun(in.annotation.gap(in.numSentences()), in.gap(in.numSentences()), true));
@@ -200,14 +205,14 @@ void hardAlignments(Response const &response, std::vector<std::vector<size_t>> &
     // Note: only search from 0 to N-1 because token N is end-of-sentence token
     // that can only align with the end-of-sentence token of the target
     for (size_t t = 0; t + 1 < response.target.numWords(sentenceIdx); ++t) {
-      size_t s_max = 0;
+      size_t maxS = 0;
       for (size_t s = 1; s + 1 < response.source.numWords(sentenceIdx); ++s) {
-        if (response.alignments[sentenceIdx][t][s] > response.alignments[sentenceIdx][t][s_max]) {
-          s_max = s;
+        if (response.alignments[sentenceIdx][t][s] > response.alignments[sentenceIdx][t][maxS]) {
+          maxS = s;
         }
       }
 
-      alignments.back().push_back(s_max);
+      alignments.back().push_back(maxS);
     }
 
     // Next, we try to smooth out these selected alignments with a few heuristics
@@ -241,52 +246,84 @@ void hardAlignments(Response const &response, std::vector<std::vector<size_t>> &
   }
 }
 
+// Internal type used to point to a position in HTML::spans_.
+typedef std::vector<HTML::Span>::const_iterator SpanIterator;
+
 void copyTaint(Response const &response, std::vector<std::vector<size_t>> const &alignments,
-               std::vector<HTML::Taint> const &sourceTokenTags, std::vector<HTML::Taint> &targetTokenTags) {
+               std::vector<SpanIterator> const &sourceTokenSpans, std::vector<SpanIterator> &targetTokenSpans) {
   size_t offset = 0;
 
-  // Fill targetTokenTags based on the alignments we just made up.
+  // Fill targetTokenSpans based on the alignments we just made up.
   // NOTE: this should match the exact order of Apply()
   for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
-    targetTokenTags.push_back(sourceTokenTags[offset]);  // token_tag for sentence ending gap
+    targetTokenSpans.push_back(sourceTokenSpans[offset]);  // token_tag for sentence ending gap
     for (size_t t = 0; t < response.target.numWords(sentenceIdx); ++t) {
       size_t s = alignments[sentenceIdx][t];
       assert(s < response.source.numWords(sentenceIdx));
-      targetTokenTags.push_back(sourceTokenTags[offset + 1 + s]);  // +1 for prefix gap
+      targetTokenSpans.push_back(sourceTokenSpans[offset + 1 + s]);  // +1 for prefix gap
     }
 
     offset += response.source.numWords(sentenceIdx) + 1;  // +1 for prefix gap
   }
 
-  assert(offset < sourceTokenTags.size());
-  targetTokenTags.push_back(sourceTokenTags[offset]);  // token_tag for ending whitespace
+  assert(offset < sourceTokenSpans.size());
+  targetTokenSpans.push_back(sourceTokenSpans[offset]);  // token_tag for ending whitespace
 }
 
-AnnotatedText restoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &token_tags,
-                            std::vector<HTML::Span>::const_iterator span_it,
-                            std::vector<HTML::Span>::const_iterator span_end) {
-  auto prev_it = span_it;  // safe because first span is always empty span, and
-                           // and the while-loop below will do the rest
+// Little helper class to append HTML to a token
+class TokenFormatter {
+ public:
+  TokenFormatter(string_view token)
+      : html_(), offset_(0), whitespaceSize_(countPrefixWhitespaces(token)), closeLeft_(true) {
+    // Do encoding of any entities that popped up in the translation
+    encodeEntities(token, html_);
+  }
+
+  std::string &&html() { return std::move(html_); }
 
-  // workspace variables for lambda
-  std::string html;
-  HTML::Taint opening, closing;
+  // Append the markup necessary for moving from `prev` set of tags to `curr`.
+  void append(HTML::Taint const &prev, HTML::Taint const &curr) {
+    HTML::Taint opening, closing;
 
-  return apply(in, [&](ByteRange range, string_view token, bool last) {
-    // Do encoding of any entities that popped up in the translation
-    // (Also effectively clears html from previous call)
-    encodeEntities(token, html);
+    diffTags(prev, curr, opening, closing);
 
-    size_t offset = 0;  // Size added by prepending HTML
-    size_t whitespace_size = countPrefixWhitespaces(token);
+    for (HTML::Tag const *tag : reversed(closing)) {
+      std::string closeTag = format("</{}>", tag->name);
+      html_.insert(offset_ + (closeLeft_ ? 0 : whitespaceSize_), closeTag);
+      offset_ += closeTag.size();
+    }
 
-    // Close tags we want to show up left (before) the token, but open tags
-    // ideally come directly after any prefix whitespace. However, some tokens
-    // match multiple spans. If a previous span has added an open tag, after any
-    // whitespace, and the next span closes said tag again, we need to close
-    // it after the whitespace. So after the first open tag, any closing tag
-    // should also align right, after whitespace, not before. Hence this bool.
-    bool close_left = true;
+    for (HTML::Tag const *tag : opening) {
+      std::string openTag = format("<{}{}>", tag->name, tag->attributes);
+      html_.insert(offset_ + whitespaceSize_, openTag);
+      offset_ += openTag.size();
+      closeLeft_ = false;
+    }
+  }
+
+ private:
+  std::string html_;       // Output html
+  size_t offset_;          // Size added by prepending HTML
+  size_t whitespaceSize_;  // number of prefix whitespace characters
+
+  // Close tags we want to show up left (before) the token, but open tags
+  // ideally come directly after any prefix whitespace. However, some tokens
+  // match multiple spans. If a previous span has added an open tag, after any
+  // whitespace, and the next span closes said tag again, we need to close
+  // it after the whitespace. So after the first open tag, any closing tag
+  // should also align right, after whitespace, not before. Hence this bool.
+  bool closeLeft_;
+};
+
+AnnotatedText restoreSource(AnnotatedText const &in, std::vector<HTML::Span> const &sourceSpans,
+                            std::vector<SpanIterator> &sourceTokenSpans) {
+  auto spanIt = sourceSpans.begin();
+  auto prevIt = sourceSpans.begin();  // safe because first span is always empty span, and
+                                      // and the while-loop below will do the rest
+  assert(prevIt == sourceSpans.end() || prevIt->tags.empty());
+
+  return apply(in, [&](ByteRange range, string_view token, bool last) {
+    TokenFormatter formatter(token);
 
     // Potential issue: spans and tokens can intersect, e.g.
     //
@@ -295,27 +332,16 @@ AnnotatedText restoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &t
     //  tokens     |111111111111111|2|
     //
     // Now 1 covers span 1 to 3, so what taint should it get? Just <p>, or <p><u>?
+    // Note: only relevant if isBlockElement is used. If we just insert spaces
+    // around all elements, every segment of `hello` will be a token.
 
     // Seek to the last span that overlaps with this token
     while (true) {
-      diffTags(prev_it->tags, span_it->tags, opening, closing);
-      prev_it = span_it;
-
-      for (auto cit = closing.crbegin(); cit != closing.crend(); ++cit) {
-        std::string close_tag = format("</{}>", (*cit)->name);
-        html.insert(offset + (close_left ? 0 : whitespace_size), close_tag);
-        offset += close_tag.size();
-      }
+      formatter.append(prevIt->tags, spanIt->tags);
+      prevIt = spanIt;
 
-      for (HTML::Tag const *tag : opening) {
-        std::string open_tag = format("<{}{}>", tag->name, tag->attributes);
-        html.insert(offset + whitespace_size, open_tag);
-        offset += open_tag.size();
-        close_left = false;
-      }
-
-      if (span_it + 1 != span_end && ((span_it + 1)->begin < range.end || last)) {
-        span_it++;
+      if (spanIt + 1 != sourceSpans.end() && ((spanIt + 1)->begin < range.end || last)) {
+        spanIt++;
         continue;
       }
 
@@ -323,71 +349,69 @@ AnnotatedText restoreSource(AnnotatedText const &in, std::vector<HTML::Taint> &t
     }
 
     // TODO: This is just the taint of the last span, not the ones in between.
-    // This makes us lose empty tags, and maybe some markup as well, in the
-    // response target HTML restoration.
-    token_tags.push_back(prev_it->tags);
+    // This makes us lose some markup of parts of tokens as described above.
+    sourceTokenSpans.push_back(prevIt);
 
-    return html;
+    return std::move(formatter.html());
   });
 }
 
-AnnotatedText restoreTarget(AnnotatedText const &in, std::vector<HTML::Taint> const &token_tags_target) {
-  auto token_prev_it = token_tags_target.begin();
-  auto token_tags_it = token_tags_target.begin() + 1;
-
-  // workspace for lambda
-  std::string html;
-  HTML::Taint opening, closing;
+AnnotatedText restoreTarget(AnnotatedText const &in, std::vector<HTML::Span> const &sourceSpans,
+                            std::vector<SpanIterator> const &targetTokenSpans) {
+  auto prevSpan = sourceSpans.begin();
+  auto targetSpanIt = targetTokenSpans.begin();
 
   AnnotatedText out = apply(in, [&](ByteRange range, string_view token, bool last) {
-    // Do encoding of any entities that popped up in the translation
-    // (Also effectively clears html from previous call)
-    encodeEntities(token, html);
+    TokenFormatter formatter(token);
 
-    size_t offset = 0;  // Size added by prepending HTML
-    size_t whitespace_size = countPrefixWhitespaces(token);
+    // First we scan through spans_ to catch up to the span assigned to this
+    // token. We're only interested in empty spans (empty and void elements)
+    for (auto span_it = prevSpan + 1; span_it < *targetSpanIt; span_it++) {
+      // We're only interested in empty spans between the spans in targetSpanIt
+      if (span_it->size() != 0) continue;
 
-    assert(token_tags_it != token_tags_target.end());
-    diffTags(*token_prev_it, *token_tags_it, opening, closing);
+      formatter.append(prevSpan->tags, span_it->tags);
 
-    for (auto cit = closing.crbegin(); cit != closing.crend(); ++cit) {
-      std::string close_tag = format("</{}>", (*cit)->name);
-      html.insert(offset, close_tag);
-      offset += close_tag.size();
+      // Note: here, not in 3rd part of for-statement because we don't want to
+      // set prevSpan if the continue clause at the beginning of this for-loop
+      // was hit.
+      prevSpan = span_it;
     }
 
-    for (HTML::Tag const *tag : opening) {
-      std::string open_tag = format("<{}{}>", tag->name, tag->attributes);
-      html.insert(offset + whitespace_size, open_tag);
-      offset += open_tag.size();
-    }
+    // Now do the same thing but for our target set of tags. Note that we cannot
+    // combine this in the for-loop above (i.e. `span_it <= *targetSpanIt`)
+    // because there is no guarantee that the order in `targetTokenSpans` is
+    // the same as that of `spans`.
+    formatter.append(prevSpan->tags, (*targetSpanIt)->tags);
 
     // If this is the last token of the response, close all open tags.
     if (last) {
-      for (auto cit = token_tags_it->crbegin(); cit != token_tags_it->crend(); ++cit) {
-        html += format("</{}>", (*cit)->name);
-      }
+      // Note: this assert is true due to our current implementation of
+      // HardAlignments() that always matches the last token of the input with
+      // the last token of the output. But lets assume someone someday changes
+      // HardAlignments(), and then this for-loop will be necessary.
+      // assert((*targetSpanIt)->tags.empty());
+      formatter.append((*targetSpanIt)->tags, HTML::Taint());
     }
 
-    ++token_prev_it;
-    ++token_tags_it;
+    prevSpan = *targetSpanIt++;
 
-    return html;
+    return std::move(formatter.html());
   });
 
   // Assert that we did in fact use all our taints
-  assert(token_tags_it == token_tags_target.end());
+  assert(targetSpanIt == targetTokenSpans.end());
 
   return out;
 }
 
 std::ostream &debugPrintMapping(std::ostream &out, Response const &response,
                                 std::vector<std::vector<size_t>> const &alignments,
-                                std::vector<HTML::Taint> const &token_tags_target) {
-  auto taints = token_tags_target.begin();
+                                std::vector<SpanIterator> const &targetTokenSpans) {
+  auto spans = targetTokenSpans.begin();
   for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
     out << "Mapped sentence prefix with tags: ";
-    for (auto &&taint : *(++taints)) out << '/' << taint->name;
+    for (auto &&taint : (*++spans)->tags) out << '/' << taint->name;
     out << '\n';
 
     for (size_t wordIdx = 0; wordIdx < response.target.numWords(sentenceIdx); ++wordIdx) {
@@ -399,16 +423,16 @@ std::ostream &debugPrintMapping(std::ostream &out, Response const &response,
       out << " to ";
       out << std::setw(10) << std::setfill(' ') << response.source.word(sentenceIdx, alignments[sentenceIdx][wordIdx]);
       out << " with tags: ";
-      for (auto &&taint : *(++taints)) out << '/' << taint->name;
+      for (auto &&taint : (*++spans)->tags) out << '/' << taint->name;
       out << '\n';
     }
   }
 
   out << "Mapped end-of-input with tags: ";
-  for (auto &&taint : *(++taints)) out << '/' << taint->name;
+  for (auto &&taint : (*++spans)->tags) out << '/' << taint->name;
   out << '\n';
 
-  assert(++taints == token_tags_target.end());
+  assert(++spans == targetTokenSpans.end());
   return out;
 }
 
@@ -467,7 +491,6 @@ HTML::HTML(std::string &&source, bool process_markup) {
         auto begin = source.size();
         source.append(scanner.value());
         spans_.push_back(Span{begin, source.size(), stack});
-        filterEmpty(stack);
       } break;
 
       case markup::Scanner::TT_TAG_START:
@@ -539,32 +562,32 @@ void HTML::restore(Response &response) {
 
   // Reconstruction of HTML tags:
   // 1. Map each token to a Span
-  // 2. Apply the taint of that span to the token
-  // 3. Reconstruct the source HTML with these tainted tokens
-  // 4. Transfer the taint from the source tokens to the target tokens using alignment information
+  // 2. Reconstruct the source HTML with these tainted tokens
+  // 3. Transfer the spans from the source tokens to the target tokens using alignment information
+  // 4. For spans that represent empty elements (e.g. <img>) figure out their position
   // 5. Reconstruct the target HTML with these tainted tokens
 
-  std::vector<Taint> token_tags;  // List of HTML tags active per token in source
-                                  // Calculating these is a side-effect of restoring
-                                  // the HTML in response.source.
+  // sourceTokenSpans is a vector with a pointer to a span for each token. We
+  // use iterators here to point to these positions so we can easily compare if
+  // one span comes before or after another, information we'll need when we need
+  // to figure out whether we've skipped spans (of emtpy elements) when
+  // reconstructing HTML in response.target.
+  std::vector<SpanIterator> sourceTokenSpans;
 
-  AnnotatedText source = restoreSource(response.source, token_tags, spans_.cbegin(), spans_.cend());
-  assert(token_tags.size() == debugCountTokens(response.source));
+  // RestoreSource re-inserts HTML into the source text, but also identifies
+  // which span each source token fits into best.
+  AnnotatedText source = restoreSource(response.source, spans_, sourceTokenSpans);
+  assert(sourceTokenSpans.size() == debugCountTokens(response.source));
 
   // Find for every token in target the token in source that best matches.
   std::vector<std::vector<size_t>> alignments;
   hardAlignments(response, alignments);
 
-  std::vector<Taint> token_tags_target;
-  token_tags_target.emplace_back();  // add empty one to the beginning for easy
-                                     // life later on (we start iterating at 1,
-                                     // and can then do i - 1 for empty.
-  copyTaint(response, alignments, token_tags, token_tags_target);
-  assert(token_tags_target.size() == debugCountTokens(response.target) + 1);
-
-  // DebugPrintMapping(std::cerr, response, alignments, token_tags_target);
+  std::vector<SpanIterator> targetTokenSpans;
+  copyTaint(response, alignments, sourceTokenSpans, targetTokenSpans);
+  assert(targetTokenSpans.size() == debugCountTokens(response.target));
 
-  AnnotatedText target = restoreTarget(response.target, token_tags_target);
+  AnnotatedText target = restoreTarget(response.target, spans_, targetTokenSpans);
 
   response.source = source;
   response.target = target;

From 9e1c1e8dbf4817f411f718c75ce42493d92c43b6 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Tue, 21 Dec 2021 23:58:13 +0100
Subject: [PATCH 322/442] CI: Circle CI config script update (#287)

- Robust artifact presence check
 - Variable name refactoring
 - Storing only those artifacts that are required
 - Remove commit sha from the names of the Github Releases
 - Use BERGAMOT_VERSION file contents for Git Tag names
---
 .circleci/config.yml | 34 +++++++++++++++-------------------
 1 file changed, 15 insertions(+), 19 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index d9ff7933d..fbea34e18 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -19,13 +19,12 @@ jobs:
           name: Check artifacts
           working_directory: build-wasm
           command: |
-            ls -all bergamot*
-            if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
-            then
+            ARTIFACT_BASE="bergamot-translator-worker"
+            if [[ -f "$ARTIFACT_BASE.js" && -f "$ARTIFACT_BASE.wasm" ]]; then
               echo "Artifacts Successfully Generated"
               mkdir ../artifacts
-              cp bergamot-translator-worker.wasm ../artifacts/bergamot-translator-worker-with-wormhole.wasm
-              cp bergamot-translator-worker.js ../artifacts/bergamot-translator-worker-with-wormhole.js
+              cp $ARTIFACT_BASE.wasm ../artifacts/$ARTIFACT_BASE-with-wormhole.wasm
+              cp $ARTIFACT_BASE.js ../artifacts/$ARTIFACT_BASE-with-wormhole.js
               shasum -a 256 ../artifacts/* > ../artifacts/SHA256-1
               cp ../BERGAMOT_VERSION ../artifacts/
             else
@@ -39,7 +38,7 @@ jobs:
             - artifacts/*
 
       - store_artifacts:
-          path: "build-wasm"
+          path: "artifacts"
           destination: "wasm-wormhole"
 
   build-without-wormhole:
@@ -61,26 +60,27 @@ jobs:
           name: Check artifacts
           working_directory: build-wasm
           command: |
-            ls -all bergamot*
-            if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
-            then
+            ARTIFACT_BASE="bergamot-translator-worker"
+            if [[ -f "$ARTIFACT_BASE.js" && -f "$ARTIFACT_BASE.wasm" ]]; then
               echo "Artifacts Successfully Generated"
               mkdir ../artifacts
-              cp bergamot-translator-worker.wasm ../artifacts/bergamot-translator-worker-without-wormhole.wasm
-              cp bergamot-translator-worker.js ../artifacts/bergamot-translator-worker-without-wormhole.js
+              cp $ARTIFACT_BASE.wasm ../artifacts/$ARTIFACT_BASE-without-wormhole.wasm
+              cp $ARTIFACT_BASE.js ../artifacts/$ARTIFACT_BASE-without-wormhole.js
               shasum -a 256 ../artifacts/* > ../artifacts/SHA256-2
             else
               echo "Failure: Artifacts Not Present"
               exit 1
             fi
+
       - persist_to_workspace:
           root: .
           paths:
             - artifacts/*
 
       - store_artifacts:
-          path: "build-wasm"
+          path: "artifacts"
           destination: "wasm-without-wormhole"
+
   publish_to_github:
      docker:
        - image: cibuilds/github:0.10
@@ -91,15 +91,11 @@ jobs:
        - run:
           name: "Publish Release on GitHub"
           command: |
-            export COMMIT=$(echo $CIRCLE_SHA1 | cut -c -7)
-            export VERSION=$(cat ./artifacts/BERGAMOT_VERSION | cut -c 2-)
-            VERSION=$VERSION+$COMMIT
+            export TAG_VERSION=$(cat ./artifacts/BERGAMOT_VERSION)
             ls -lsa ./artifacts/ > ./artifacts/FILESIZES
             cat ./artifacts/SHA256-1 ./artifacts/SHA256-2 > ./artifacts/SHA256
-            rm ./artifacts/SHA256-1
-            rm ./artifacts/SHA256-2
-            rm ./artifacts/BERGAMOT_VERSION
-            ghr -t ${GHTOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${VERSION} ./artifacts/
+            rm ./artifacts/SHA256-1 ./artifacts/SHA256-2 ./artifacts/BERGAMOT_VERSION
+            ghr -t ${GHTOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${TAG_VERSION} ./artifacts/
 
 workflows:
   build:

From 6e6042c98f2194cd10514844a9a71e218d8e7830 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Wed, 29 Dec 2021 11:02:56 +0000
Subject: [PATCH 323/442] GitHub CI: Update YAML to run all tests on
 marian-full (#292)

Previously there were #native tags and #wasm tags separating the two.
There is now a clear separation between async, blocking and wasm.
---
 .github/workflows/native.yml | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
index e27cbe1b0..4a8dbffdc 100644
--- a/.github/workflows/native.yml
+++ b/.github/workflows/native.yml
@@ -25,19 +25,19 @@ jobs:
           os: ubuntu-18.04
           identifier: ubuntu_1804_full
           cmake: -DCOMPILE_TESTS=on
-          brt_tags: "'#native'"
+          brt_args: ""
           unittests: 'true'
         - name: Ubuntu 18.04 minimal
           os: ubuntu-18.04
           identifier: ubuntu_1804_minimal
           cmake: -DCOMPILE_TESTS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
-          brt_tags: "'#wasm'"
+          brt_args: "'#wasm'"
           unittests: 'false'
         - name: Ubuntu 20.04 full
           os: ubuntu-20.04
           identifier: ubuntu_2004_full
           cmake: -DCOMPILE_TESTS=on
-          brt_tags: "'#native'"
+          brt_tags: ""
           unittests: 'true'
         - name: Ubuntu 20.04 minimal
           os: ubuntu-20.04
@@ -140,7 +140,7 @@ jobs:
           os: macos-10.15
           identifier: mac_1015_full
           cmake: -DCOMPILE_TESTS=on -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=off
-          brt_tags: "'#native'"
+          brt_tags: ""
           unittests: 'true'
         - name: MacOS 10.15 minimal
           os: macos-10.15

From 8eb238ed5ec30c0ea03ba9507df23ab73fe2ef04 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Thu, 30 Dec 2021 14:29:12 +0000
Subject: [PATCH 324/442] HTML basic integration tests (#291)

---
 bergamot-translator-tests |  2 +-
 src/tests/common-impl.cpp | 13 +++++++++++++
 src/tests/common.h        |  2 ++
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 5524e37a0..59720cb67 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 5524e37a01920dc5149dcc87b047615c6a70aa53
+Subproject commit 59720cb67458c4682cde7e999a4b18d6934ab988
diff --git a/src/tests/common-impl.cpp b/src/tests/common-impl.cpp
index 49ebfc53c..9fc44c9ad 100644
--- a/src/tests/common-impl.cpp
+++ b/src/tests/common-impl.cpp
@@ -49,6 +49,8 @@ void TestSuite<Service>::TestSuite::run(const std::string &opModeAsString, std::
     qualityEstimatorScores(models.front());
   } else if (opModeAsString == "test-translation-cache") {
     translationCache(models.front());
+  } else if (opModeAsString == "test-html-translation") {
+    htmlTranslation(models.front());
   } else {
     std::cerr << "Incompatible test mode. Choose from the one of the valid test-modes";
     std::abort();
@@ -138,6 +140,17 @@ void TestSuite<Service>::qualityEstimatorWords(Ptr<TranslationModel> model) {
   }
 }
 
+template <class Service>
+void TestSuite<Service>::htmlTranslation(Ptr<TranslationModel> model) {
+  ResponseOptions responseOptions;
+  responseOptions.HTML = true;
+  responseOptions.alignment = true;
+  std::string source = readFromStdin();
+  const Response response = bridge_.translate(service_, model, std::move(source), responseOptions);
+
+  std::cout << response.target.text;
+}
+
 // Reads from stdin and translates the read content. Prints the quality scores for each sentence.
 template <class Service>
 void TestSuite<Service>::qualityEstimatorScores(Ptr<TranslationModel> model) {
diff --git a/src/tests/common.h b/src/tests/common.h
index dff47e483..1e454858c 100644
--- a/src/tests/common.h
+++ b/src/tests/common.h
@@ -79,6 +79,8 @@ class TestSuite {
   void qualityEstimatorScores(Ptr<TranslationModel> model);
 
   void translationCache(Ptr<TranslationModel> model);
+
+  void htmlTranslation(Ptr<TranslationModel> model);
 };
 
 #define BERGAMOT_TESTS_COMMON_IMPL

From d209e4fc49290989dfb2443163b12e150ad1b97a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Thu, 30 Dec 2021 16:12:30 +0000
Subject: [PATCH 325/442] Fix typo in BRT args on CI runs (#294)

---
 .github/workflows/native.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
index 4a8dbffdc..9ff351c0a 100644
--- a/.github/workflows/native.yml
+++ b/.github/workflows/native.yml
@@ -25,13 +25,13 @@ jobs:
           os: ubuntu-18.04
           identifier: ubuntu_1804_full
           cmake: -DCOMPILE_TESTS=on
-          brt_args: ""
+          brt_tags: ""
           unittests: 'true'
         - name: Ubuntu 18.04 minimal
           os: ubuntu-18.04
           identifier: ubuntu_1804_minimal
           cmake: -DCOMPILE_TESTS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
-          brt_args: "'#wasm'"
+          brt_tags: "'#wasm'"
           unittests: 'false'
         - name: Ubuntu 20.04 full
           os: ubuntu-20.04

From ddccc77570f64206d1df38cd19957022b4f26b3c Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Sun, 2 Jan 2022 00:17:12 +0000
Subject: [PATCH 326/442] Turn logging off by default, allow turning on via
 config/cmdline (#295)

* Turn logging off by default, allow turning on via config/cmdline
* No need to store config in member variable if things are decided at construction time
---
 src/translator/logging.h   | 38 +++++++++++++++++++++++++++++++++++++-
 src/translator/service.cpp | 12 ++++++++++--
 src/translator/service.h   |  6 ++++++
 3 files changed, 53 insertions(+), 3 deletions(-)

diff --git a/src/translator/logging.h b/src/translator/logging.h
index bd5b17a45..2256d7889 100644
--- a/src/translator/logging.h
+++ b/src/translator/logging.h
@@ -7,9 +7,45 @@ namespace bergamot {
 // RAII Wrap around logging, to clean up after the object on stack.
 class Logger {
  public:
-  Logger() : marianLoggers_(createLoggers()) {
+  struct Config {
+    std::string level{"off"};
+    template <class App>
+    static void addOptions(App &app, Config &config) {
+      app.add_option("--log-level", config.level,
+                     "Set verbosity level of logging: trace, debug, info, warn, err(or), critical, off");
+    }
+  };
+
+  Logger(const Config &config) : marianLoggers_(createLoggers()) {
     // We are manually creating loggers, because this is usually created in marian as a side-effect of
     // config-parsing.
+    for (auto &logger : marianLoggers_) {
+      setLoggingLevel(*logger, config.level);
+    }
+  }
+
+  // Taken from
+  // https://github.com/marian-nmt/marian-dev/blob/c84599d08ad69059279abd5a7417a8053db8b631/src/common/logging.cpp#L45
+  static bool setLoggingLevel(spdlog::logger &logger, std::string const level) {
+    if (level == "trace")
+      logger.set_level(spdlog::level::trace);
+    else if (level == "debug")
+      logger.set_level(spdlog::level::debug);
+    else if (level == "info")
+      logger.set_level(spdlog::level::info);
+    else if (level == "warn")
+      logger.set_level(spdlog::level::warn);
+    else if (level == "err" || level == "error")
+      logger.set_level(spdlog::level::err);
+    else if (level == "critical")
+      logger.set_level(spdlog::level::critical);
+    else if (level == "off")
+      logger.set_level(spdlog::level::off);
+    else {
+      logger.warn("Unknown log level '{}' for logger '{}'", level.c_str(), logger.name().c_str());
+      return false;
+    }
+    return true;
   }
 
   ~Logger() {
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index ca92721da..8acbc97de 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -11,7 +11,11 @@ namespace marian {
 namespace bergamot {
 
 BlockingService::BlockingService(const BlockingService::Config &config)
-    : config_(config), requestId_(0), batchingPool_(), cache_(config.cacheSize, /*mutexBuckets=*/1) {}
+    : config_(config),
+      requestId_(0),
+      batchingPool_(),
+      cache_(config.cacheSize, /*mutexBuckets=*/1),
+      logger_(config.logger) {}
 
 std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<TranslationModel> translationModel,
                                                          std::vector<std::string> &&sources,
@@ -37,7 +41,11 @@ std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<Transla
 }
 
 AsyncService::AsyncService(const AsyncService::Config &config)
-    : requestId_(0), config_(config), safeBatchingPool_(), cache_(config_.cacheSize, config_.cacheMutexBuckets) {
+    : requestId_(0),
+      config_(config),
+      safeBatchingPool_(),
+      cache_(config_.cacheSize, config_.cacheMutexBuckets),
+      logger_(config.logger) {
   ABORT_IF(config_.numWorkers == 0, "Number of workers should be at least 1 in a threaded workflow");
   workers_.reserve(config_.numWorkers);
   for (size_t cpuId = 0; cpuId < config_.numWorkers; cpuId++) {
diff --git a/src/translator/service.h b/src/translator/service.h
index 383ab9885..e01156ae5 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -33,11 +33,14 @@ class BlockingService {
     bool cacheEnabled{false};  ///< Whether to enable cache or not.
     size_t cacheSize{2000};    ///< Size in History items to be stored in the cache. Loosely corresponds to sentences to
                                /// cache in the real world.
+    Logger::Config logger;     // Configurations for logging
+
     template <class App>
     static void addOptions(App &app, Config &config) {
       // Options will come here.
       app.add_option("--cache-translations", config.cacheEnabled, "Whether to cache translations or not.");
       app.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
+      Logger::Config::addOptions(app, config.logger);
     }
   };
   /// Construct a BlockingService with configuration loaded from an Options object. Does not require any keys, values to
@@ -90,6 +93,8 @@ class AsyncService {
     size_t cacheMutexBuckets{1};  ///< Controls the granularity of locking to reduce contention by bucketing mutexes
                                   ///< guarding cache entry read write. Optimal at min(core, numWorkers) assuming a
                                   ///< reasonably large cache-size.
+    Logger::Config logger;        // Configurations for logging
+
     template <class App>
     static void addOptions(App &app, Config &config) {
       app.add_option("--cpu-threads", config.numWorkers, "Workers to form translation backend");
@@ -97,6 +102,7 @@ class AsyncService {
       app.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
       app.add_option("--cache-mutex-buckets", config.cacheMutexBuckets,
                      "Number of mutex buckets to control locking granularity");
+      Logger::Config::addOptions(app, config.logger);
     }
   };
   /// Construct an AsyncService with configuration loaded from Options. Expects positive integer value for

From 3883dd19713b0f6f30eb4c3cfcdb8e488eab3a76 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Sun, 2 Jan 2022 12:33:30 +0000
Subject: [PATCH 327/442] cache: threadsafety-fixes; optional stats collection
 (#245)

* Make stats hits misses atomic to guard when mutex has multiple buckets
* Use compile time switch for cache-stats-collection bound to COMPILE_TESTS cmake variable
* -DENABLE_CACHE_STATS on if COMPILE_TESTS otherwise optional
* Make stats() call without enabling build fatal abort
---
 CMakeLists.txt                |  2 ++
 bergamot-translator-tests     |  2 +-
 src/translator/CMakeLists.txt |  4 ++++
 src/translator/cache.h        | 24 ++++++++++++++++++++----
 4 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 006e9521d..f121ca0fb 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -74,6 +74,8 @@ cmake_dependent_option(USE_WASM_COMPATIBLE_SOURCE "Use wasm compatible sources"
 # WASM disables a million libraries, which also includes the unit test-library.
 cmake_dependent_option(COMPILE_UNIT_TESTS "Compile unit tests" OFF "USE_WASM_COMPATIBLE_SOURCE" ON)
 option(COMPILE_TESTS "Compile bergamot-tests" OFF)
+cmake_dependent_option(ENABLE_CACHE_STATS "Enable stats on cache" ON "COMPILE_TESTS" OFF)
+
 
 # Set 3rd party submodule specific cmake options for this project
 SET(COMPILE_CUDA OFF CACHE BOOL "Compile GPU version")
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 59720cb67..332e976df 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 59720cb67458c4682cde7e999a4b18d6934ab988
+Subproject commit 332e976df4583793a09b6483b80b972621fcfadb
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 6779b0fa4..dbead6173 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -32,6 +32,10 @@ if(COMPILE_WASM)
   target_compile_options(bergamot-translator PRIVATE ${WASM_COMPILE_FLAGS})
 endif(COMPILE_WASM)
 
+if(ENABLE_CACHE_STATS)
+    target_compile_definitions(bergamot-translator PUBLIC ENABLE_CACHE_STATS)
+endif(ENABLE_CACHE_STATS)
+
 target_link_libraries(bergamot-translator marian ssplit)
 
 target_include_directories(bergamot-translator
diff --git a/src/translator/cache.h b/src/translator/cache.h
index ba68e4e93..ceeca5d32 100644
--- a/src/translator/cache.h
+++ b/src/translator/cache.h
@@ -1,4 +1,5 @@
 #pragma once
+#include <atomic>
 #include <memory>
 #include <mutex>
 #include <vector>
@@ -26,7 +27,14 @@ class AtomicCache {
 
   void store(const Key &key, Value value) { atomicStore(key, value); }
 
-  const Stats stats() const { return stats_; }
+  const Stats stats() const {
+#ifdef ENABLE_CACHE_STATS
+    return Stats{hits_.load(), misses_.load()};
+#else
+    ABORT("Cache statistics requested without enabling in builds. Please use -DENABLE_CACHE_STATS with cmake.");
+    return Stats{0, 0};
+#endif
+  }
 
  private:
   using Record = std::pair<Key, Value>;
@@ -40,10 +48,14 @@ class AtomicCache {
     const Record &candidate = records_[index];
     if (equals_(key, candidate.first)) {
       value = candidate.second;
-      stats_.hits += 1;
+#ifdef ENABLE_CACHE_STATS
+      ++hits_;
+#endif
       return true;
     } else {
-      stats_.misses += 1;
+#ifdef ENABLE_CACHE_STATS
+      ++misses_;
+#endif
     }
 
     return false;
@@ -64,7 +76,11 @@ class AtomicCache {
   std::vector<Record> records_;
 
   mutable std::vector<std::mutex> mutexBuckets_;
-  mutable Stats stats_;
+
+#ifdef ENABLE_CACHE_STATS
+  mutable std::atomic<size_t> hits_{0};
+  mutable std::atomic<size_t> misses_{0};
+#endif
 
   Hash hash_;
   Equals equals_;

From 81c21928d5c360e47b998a6d24abe055bab9165b Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Mon, 3 Jan 2022 12:27:41 +0000
Subject: [PATCH 328/442] Have alignments placed if HTML is on (#296)

---
 src/translator/response_builder.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
index baa648850..345951a0e 100644
--- a/src/translator/response_builder.h
+++ b/src/translator/response_builder.h
@@ -61,7 +61,7 @@ class ResponseBuilder {
       buildQualityScores(histories, response);
     }
 
-    if (responseOptions_.alignment) {
+    if (responseOptions_.alignment || responseOptions_.HTML) {
       buildAlignments(histories, response);
     }
     html_.restore(response);

From dae02a3c8d2f95139d3c9623f8644b2b4776dab9 Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Wed, 5 Jan 2022 14:33:51 +0100
Subject: [PATCH 329/442] HTML transfer script/style/etc elements (#285)

---
 src/tests/units/html_tests.cpp |  47 +++++++++++++
 src/translator/html.cpp        | 120 +++++++++++++++++----------------
 src/translator/html.h          |  12 +++-
 3 files changed, 120 insertions(+), 59 deletions(-)

diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
index e3d79379f..48af7066c 100644
--- a/src/tests/units/html_tests.cpp
+++ b/src/tests/units/html_tests.cpp
@@ -419,6 +419,53 @@ TEST_CASE("Test empty tag") {
   CHECK(response.target.text == test_str);
 }
 
+TEST_CASE("Test <script> element") {
+  std::string test_str("hello <script>alert(\"<foo>\");</script>world");
+
+  std::string input(test_str);
+  HTML html(std::move(input), true);
+  CHECK(input == "hello world");
+
+  Response response;
+  std::string sentence_str("hello world");
+  std::vector<string_view> sentence{
+      string_view(sentence_str.data() + 0, 4),   // 0.0 hell
+      string_view(sentence_str.data() + 4, 1),   // 0.1 o
+      string_view(sentence_str.data() + 5, 6),   // 0.2 _world
+      string_view(sentence_str.data() + 11, 0),  // 0.3 ""
+  };
+  response.source.appendSentence("", sentence.begin(), sentence.end());
+  response.target.appendSentence("", sentence.begin(), sentence.end());
+  response.alignments = {identity_matrix<float>(4)};
+
+  html.restore(response);
+  CHECK(response.source.text == test_str);
+  CHECK(response.target.text == test_str);
+}
+
+TEST_CASE("Test comment") {
+  std::string test_str("foo <!-- <ignore> me -->bar");
+
+  std::string input(test_str);
+  HTML html(std::move(input), true);
+  CHECK(input == "foo bar");
+
+  Response response;
+  std::string sentence_str("foo bar");
+  std::vector<string_view> sentence{
+      string_view(sentence_str.data() + 0, 3),  // foo
+      string_view(sentence_str.data() + 3, 4),  // _bar
+      string_view(sentence_str.data() + 7, 0),  // ""
+  };
+  response.source.appendSentence("", sentence.begin(), sentence.end());
+  response.target.appendSentence("", sentence.begin(), sentence.end());
+  response.alignments = {identity_matrix<float>(3)};
+
+  html.restore(response);
+  CHECK(response.source.text == test_str);
+  CHECK(response.target.text == test_str);
+}
+
 TEST_CASE("End-to-end translation") {
   std::string input("<p>I <b>like</b> to <u>drive</u> this car.</p>\n");
   HTML html(std::move(input), true);
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index 4424241c2..13ab422ac 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -47,11 +47,20 @@ size_t countPrefixWhitespaces(string_view const &input) {
   return size;
 }
 
+// Formatters used for exception messages combined with format()
 std::ostream &operator<<(std::ostream &out, HTML::Tag const *tag) {
   if (tag == nullptr) return out << "[nullptr]";
-  out << '<' << tag->name << tag->attributes;
-  if (tag->empty) out << '/';
-  return out << '>';
+  switch (tag->type) {
+    case HTML::Tag::ELEMENT:
+      return out << '<' << tag->name << tag->attributes << '>';
+    case HTML::Tag::VOID_ELEMENT:
+      return out << '<' << tag->name << tag->attributes << "/>";
+    case HTML::Tag::COMMENT:
+      return out << "<!--" << tag->data << "-->";
+    case HTML::Tag::PROCESSING_INSTRUCTION:
+      return out << "<?" << tag->data << "?>";
+  }
+  return out << "[Unknown tag type]";
 }
 
 std::ostream &operator<<(std::ostream &out, HTML::Taint const &tags) {
@@ -131,7 +140,9 @@ void diffTags(HTML::Taint const &prev, HTML::Taint const &curr, HTML::Taint &ope
   for (; i < prev.size(); ++i)
     if (i >= curr.size() || prev[i] != curr[i]) break;
 
-  std::copy_if(prev.begin() + i, prev.end(), std::back_inserter(closing), [&](HTML::Tag *tag) { return !tag->empty; });
+  // Only nodes of type ELEMENT can have children and thus would need a closing tag.
+  std::copy_if(prev.begin() + i, prev.end(), std::back_inserter(closing),
+               [&](HTML::Tag *tag) { return tag->type == HTML::Tag::ELEMENT; });
 
   opening.insert(opening.end(), curr.begin() + i, curr.end());
 }
@@ -273,7 +284,7 @@ void copyTaint(Response const &response, std::vector<std::vector<size_t>> const
 // Little helper class to append HTML to a token
 class TokenFormatter {
  public:
-  TokenFormatter(string_view token)
+  explicit TokenFormatter(string_view token)
       : html_(), offset_(0), whitespaceSize_(countPrefixWhitespaces(token)), closeLeft_(true) {
     // Do encoding of any entities that popped up in the translation
     encodeEntities(token, html_);
@@ -288,13 +299,26 @@ class TokenFormatter {
     diffTags(prev, curr, opening, closing);
 
     for (HTML::Tag const *tag : reversed(closing)) {
+      assert(tag->type == HTML::Tag::ELEMENT);
       std::string closeTag = format("</{}>", tag->name);
       html_.insert(offset_ + (closeLeft_ ? 0 : whitespaceSize_), closeTag);
       offset_ += closeTag.size();
     }
 
     for (HTML::Tag const *tag : opening) {
-      std::string openTag = format("<{}{}>", tag->name, tag->attributes);
+      std::string openTag;
+      switch (tag->type) {
+        case HTML::Tag::ELEMENT:
+        case HTML::Tag::VOID_ELEMENT:
+          openTag = format("<{}{}>{}", tag->name, tag->attributes, tag->data);
+          break;
+        case HTML::Tag::COMMENT:
+          openTag = format("<!--{}-->", tag->data);
+          break;
+        case HTML::Tag::PROCESSING_INSTRUCTION:
+          openTag = format("<?{}?>", tag->data);
+          break;
+      }
       html_.insert(offset_ + whitespaceSize_, openTag);
       offset_ += openTag.size();
       closeLeft_ = false;
@@ -405,55 +429,6 @@ AnnotatedText restoreTarget(AnnotatedText const &in, std::vector<HTML::Span> con
   return out;
 }
 
-std::ostream &debugPrintMapping(std::ostream &out, Response const &response,
-                                std::vector<std::vector<size_t>> const &alignments,
-                                std::vector<SpanIterator> const &targetTokenSpans) {
-  auto spans = targetTokenSpans.begin();
-  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
-    out << "Mapped sentence prefix with tags: ";
-    for (auto &&taint : (*++spans)->tags) out << '/' << taint->name;
-    out << '\n';
-
-    for (size_t wordIdx = 0; wordIdx < response.target.numWords(sentenceIdx); ++wordIdx) {
-      assert(sentenceIdx < alignments.size());
-      assert(wordIdx < alignments[sentenceIdx].size());
-
-      out << "Mapped ";
-      out << std::setw(10) << std::setfill(' ') << response.target.word(sentenceIdx, wordIdx);
-      out << " to ";
-      out << std::setw(10) << std::setfill(' ') << response.source.word(sentenceIdx, alignments[sentenceIdx][wordIdx]);
-      out << " with tags: ";
-      for (auto &&taint : (*++spans)->tags) out << '/' << taint->name;
-      out << '\n';
-    }
-  }
-
-  out << "Mapped end-of-input with tags: ";
-  for (auto &&taint : (*++spans)->tags) out << '/' << taint->name;
-  out << '\n';
-
-  assert(++spans == targetTokenSpans.end());
-  return out;
-}
-
-std::ostream &debugPrintAlignmentScores(std::ostream &out, Response const &response) {
-  out << "std::vector<std::vector<std::vector<float>>> alignments{\n";
-  for (size_t sentenceIdx = 0; sentenceIdx < response.source.numSentences(); ++sentenceIdx) {
-    out << "  {\n";
-    for (size_t t = 0; t < response.alignments[sentenceIdx].size(); ++t) {
-      out << "    {";
-      for (size_t s = 0; s < response.alignments[sentenceIdx][t].size(); ++s) {
-        out << std::fixed << std::setw(8) << std::setprecision(8) << std::setfill(' ')
-            << response.alignments[sentenceIdx][t][s];
-        out << ", ";
-      }
-      out << "},\n";
-    }
-    out << "  },\n";
-  }
-  return out << "};\n";
-}
-
 size_t debugCountTokens(AnnotatedText const &text) {
   size_t tokens = 1;  // for the ending gap
   for (size_t sentenceIdx = 0; sentenceIdx < text.numSentences(); ++sentenceIdx) {
@@ -501,7 +476,8 @@ HTML::HTML(std::string &&source, bool process_markup) {
         if (isBlockElement(scanner.tag()) && !source.empty() && source.back() != ' ') source.push_back(' ');
 
         // pool_ takes ownership of our tag, makes sure it's freed when necessary
-        pool_.emplace_back(new Tag{std::string(scanner.tag()), std::string(), isVoidTag(scanner.tag())});
+        pool_.emplace_back(new Tag{isVoidTag(scanner.tag()) ? Tag::VOID_ELEMENT : Tag::ELEMENT,
+                                   std::string(scanner.tag()), std::string()});
 
         // Tag *tag is used by attribute parsing
         tag = pool_.back().get();
@@ -511,7 +487,7 @@ HTML::HTML(std::string &&source, bool process_markup) {
         // Empty elements (e.g. <img>) are not applicable to a span of text
         // so instead we "apply" them to an empty span in between, and then
         // immediately remove them again from the stack.
-        if (tag->empty) {
+        if (tag->type == Tag::VOID_ELEMENT) {
           spans_.push_back(Span{source.size(), source.size(), stack});
           stack.pop_back();
         }
@@ -539,8 +515,36 @@ HTML::HTML(std::string &&source, bool process_markup) {
         tag->attributes += format(" {}=\"{}\"", scanner.attribute(), scanner.value());
         break;
 
-      default:
+      case markup::Scanner::TT_COMMENT_START:
+        // pool_ takes ownership of our tag, makes sure it's freed when necessary
+        pool_.emplace_back(new Tag{Tag::COMMENT});
+        tag = pool_.back().get();
+        stack.push_back(tag);
+        spans_.push_back(Span{source.size(), source.size(), stack});
+        stack.pop_back();
+        break;
+
+      case markup::Scanner::TT_PROCESSING_INSTRUCTION_START:
+        // pool_ takes ownership of our tag, makes sure it's freed when necessary
+        pool_.emplace_back(new Tag{Tag::PROCESSING_INSTRUCTION});
+        tag = pool_.back().get();
+        stack.push_back(tag);
+        spans_.push_back(Span{source.size(), source.size(), stack});
+        stack.pop_back();
+        break;
+
+      case markup::Scanner::TT_COMMENT_END:
+      case markup::Scanner::TT_PROCESSING_INSTRUCTION_END:
+        tag = nullptr;
+        break;
+
+      case markup::Scanner::TT_DATA:
+        assert(tag != nullptr);
+        tag->data = scanner.value();
         break;
+
+      default:
+        throw BadHTML("Unsupported scanner token type");
     }
   }
 
diff --git a/src/translator/html.h b/src/translator/html.h
index 5ddb3d006..b233fd225 100644
--- a/src/translator/html.h
+++ b/src/translator/html.h
@@ -19,9 +19,19 @@ class BadHTML : public std::runtime_error {
 class HTML {
  public:
   struct Tag {
+    enum NodeType {
+      ELEMENT,
+      VOID_ELEMENT,
+      COMMENT,
+      PROCESSING_INSTRUCTION,
+    };
+
+    NodeType type;  // Type of the node
     std::string name;
     std::string attributes;
-    bool empty;
+    std::string data;  // Raw data of an element that just needs to be
+                       // copied as is, e.g. <script> or <style>
+                       // TODO: replace with string_view if input lives that long
   };
 
   typedef std::vector<Tag *> Taint;

From 71b84b7c72c51049d418c720b50dd9a0b515b9c9 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Thu, 6 Jan 2022 19:10:57 +0000
Subject: [PATCH 330/442] CI guaranteed example documentation (#300)

* Convert marian-integration markdown to rst
* Convert native run into a script, include in rst
* Check with CI that the native running example works without fail
---
 .github/workflows/native.yml |   7 ++
 doc/marian-integration.md    | 131 -----------------------------------
 doc/marian-integration.rst   |  97 ++++++++++++++++++++++++++
 examples/run-native.sh       |  19 +++++
 4 files changed, 123 insertions(+), 131 deletions(-)
 delete mode 100644 doc/marian-integration.md
 create mode 100644 doc/marian-integration.rst
 create mode 100644 examples/run-native.sh

diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
index 9ff351c0a..8ee8c5c5f 100644
--- a/.github/workflows/native.yml
+++ b/.github/workflows/native.yml
@@ -131,6 +131,10 @@ jobs:
           bergamot-translator-tests/**/*.expected
           bergamot-translator-tests/**/*.log
           bergamot-translator-tests/**/*.out
+    - name: Confirm native-run example script works
+      run: |-
+          bash examples/run-native.sh
+
   mac:
     strategy:
       fail-fast: false
@@ -235,4 +239,7 @@ jobs:
           bergamot-translator-tests/**/*.expected
           bergamot-translator-tests/**/*.log
           bergamot-translator-tests/**/*.out
+    - name: Confirm native-run example script works
+      run: |-
+          bash examples/run-native.sh
 
diff --git a/doc/marian-integration.md b/doc/marian-integration.md
deleted file mode 100644
index d4ba82475..000000000
--- a/doc/marian-integration.md
+++ /dev/null
@@ -1,131 +0,0 @@
-# Bergamot C++ Library
-
-This document contains instructions to develop for modifications on top of the
-marian machine translation toolkit powering bergamot-translator. The library is
-optimized towards fast and efficient translation of a given input.
-
-## Build Instructions
-
-Note: You are strongly advised to refer to the continuous integration on this
-repository, which builds bergamot-translator and associated applications from
-scratch. Examples to run these command line-applications are available in the
-[bergamot-translator-tests](https://github.com/browsermt/bergamot-translator-tests)
-repository. Builds take about 30 mins on a consumer grade machine, so using a
-tool like ccache is highly recommended.
-
-### Dependencies 
-
-Marian CPU version requires Intel MKL or OpenBLAS. Both are free, but MKL is
-not open-sourced. Intel MKL is strongly recommended as it is faster. On Ubuntu
-16.04 and newer it can be installed from the APT repositories.
-
-```bash
-wget -qO- 'https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB' | sudo apt-key add -
-sudo sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'
-sudo apt-get update
-sudo apt-get install intel-mkl-64bit-2020.0-088
-```
-On MacOS, apple accelerate framework will be used instead of MKL/OpenBLAS.
-
-
-### Building bergamot-translator
-
-Web Assembly (WASM) reduces building to only using a subset of functionalities
-of marian, the translation library powering bergamot-translator. When
-developing bergamot-translator it is important that the sources added be
-compatible with marian.  Therefore, it is required to set
-`-DUSE_WASM_COMPATIBLE_SOURCE=on`.
-
-```
-$ git clone https://github.com/browsermt/bergamot-translator
-$ cd bergamot-translator
-$ mkdir build
-$ cd build
-$ cmake .. -DUSE_WASM_COMPATIBLE_SOURCE=off -DCMAKE_BUILD_TYPE=Release
-$ make -j2 
-```
-
-The build will generate the library that can be linked to any project. All the
-public header files are specified in `src` folder.
-
-## Command line apps
-
-Bergamot-translator is intended to be used as a library. However, we provide a
-command-line application which is capable of translating text provided on
-standard-input. During development this application is used to perform
-regression-tests.
-
-There are effectively multiple CLIs subclassed from a unified interface all
-provided in `app/cli.h`. These are packed into a single executable named
-`bergamot` by means of a `--bergamot-mode BERGAMOT_MODE` switch. 
-
-The following modes are available:
-
-* `--bergamot-mode native` 
-* `--bergamot-mode wasm`    
-* `--bergamot-mode decoder` 
-
-Find documentation on these modes with the API documentation for apps [here](./api/namespace_marian__bergamot__app.html#functions).
-
-## Example command line run
-
-The models required to run the command-line are available at
-[data.statmt.org/bergamot/models/](http://data.statmt.org/bergamot/models/).
-The following example uses an English to German tiny11 student model, available
-at:
-
-* [data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz](http://data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz)
-
-```bash
-MODEL_DIR=... # path to where the model-files are.
-BERGAMOT_MODE='native'
-ARGS=(
-    --bergamot-mode $BERGAMOT_MODE
-    -m $MODEL_DIR/model.intgemm.alphas.bin # Path to model file.
-    --vocabs 
-        $MODEL_DIR/vocab.deen.spm # source-vocabulary
-        $MODEL_DIR/vocab.deen.spm # target-vocabulary
-
-    # The following increases speed through one-best-decoding, shortlist and quantization.
-    --beam-size 1 --skip-cost --shortlist $MODEL_DIR/lex.s2t.bin false --int8shiftAlphaAll 
-
-    # Number of CPU threads (workers to launch). Parallelizes over cores and improves speed.
-    # A value of 0 allows a path with no worker thread-launches and a single-thread.
-    --cpu-threads 4
-
-    # Maximum size of a sentence allowed. If a sentence is above this length,
-    # it's broken into pieces of less than or equal to this size.
-    --max-length-break 1024  
-
-    # Maximum number of tokens that can be fit in a batch. The optimal value 
-    # for the parameter is dependant on hardware and can be obtained by running
-    # with variations and benchmarking.
-    --mini-batch-words 1024 
-
-    # Three modes are supported
-    #   - sentence: One sentence per line
-    #   - paragraph: One paragraph per line.
-    #   - wrapped_text: Paragraphs are separated by empty line.
-    --ssplit-mode paragraph 
-)
-
-./app/bergamot "${ARGS[@]}" < path-to-input-file
-
-```
-
-
-## Coding Style
-
-This repository contains C++ and JS source-files, of which C++ should adhere to
-the clang-format based style guidelines. You may configure your development
-environment to use the `.clang-format` and `.clang-format-ignore` files
-provided in the root folder of this repository with your preferred choice of
-editor/tooling.
-
-One simple and recommended method to get your code to adhere to this style is
-to issue the following command in the source-root of this repository, which is
-used to also check for the coding style in the CI.
-
-```bash
-python3 run-clang-format.py -i --style file -r src wasm
-```
diff --git a/doc/marian-integration.rst b/doc/marian-integration.rst
new file mode 100644
index 000000000..756e0a810
--- /dev/null
+++ b/doc/marian-integration.rst
@@ -0,0 +1,97 @@
+Bergamot C++ Library
+====================
+
+This document contains instructions to develop for modifications on top
+of the marian machine translation toolkit powering bergamot-translator.
+The library is optimized towards fast and efficient translation of a
+given input.
+
+Build Instructions
+------------------
+
+Note: You are strongly advised to refer to the continuous integration on
+this repository, which builds bergamot-translator and associated
+applications from scratch. Examples to run these command
+line-applications are available in the
+`bergamot-translator-tests <https://github.com/browsermt/bergamot-translator-tests>`__
+repository. Builds take about 30 mins on a consumer grade machine, so
+using a tool like ccache is highly recommended.
+
+Dependencies
+~~~~~~~~~~~~
+
+Marian CPU version requires Intel MKL or OpenBLAS. Both are free, but
+MKL is not open-sourced. Intel MKL is strongly recommended as it is
+faster. On Ubuntu 16.04 and newer it can be installed from the APT
+repositories.
+
+.. code:: bash
+
+    wget -qO- 'https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB' | sudo apt-key add -
+    sudo sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'
+    sudo apt-get update
+    sudo apt-get install intel-mkl-64bit-2020.0-088
+
+On MacOS, apple accelerate framework will be used instead of
+MKL/OpenBLAS.
+
+Building bergamot-translator
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Web Assembly (WASM) reduces building to only using a subset of
+functionalities of marian, the translation library powering
+bergamot-translator. When developing bergamot-translator it is important
+that the sources added be compatible with marian. Therefore, it is
+required to set ``-DUSE_WASM_COMPATIBLE_SOURCE=on``.
+
+::
+
+    $ git clone https://github.com/browsermt/bergamot-translator
+    $ cd bergamot-translator
+    $ mkdir build
+    $ cd build
+    $ cmake .. -DUSE_WASM_COMPATIBLE_SOURCE=off -DCMAKE_BUILD_TYPE=Release
+    $ make -j2 
+
+The build will generate the library that can be linked to any project.
+All the public header files are specified in ``src`` folder.
+
+Command line apps
+-----------------
+
+bergamot-translator is intended to be used as a library. However, we
+provide a command-line application which is capable of translating text
+provided on standard-input. During development this application is used
+to perform regression-tests.
+
+
+Example command line run
+------------------------
+
+The models required to run the command-line are available at
+`data.statmt.org/bergamot/models/ <http://data.statmt.org/bergamot/models/>`__.
+
+The following example uses an English to German tiny11 student model,
+available at:
+
+-  `data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz <http://data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz>`__
+
+.. literalinclude:: ../examples/run-native.sh
+   :language: bash
+
+Coding Style
+------------
+
+This repository contains C++ and JS source-files, of which C++ should
+adhere to the clang-format based style guidelines. You may configure
+your development environment to use the ``.clang-format`` and
+``.clang-format-ignore`` files provided in the root folder of this
+repository with your preferred choice of editor/tooling.
+
+One simple and recommended method to get your code to adhere to this
+style is to issue the following command in the source-root of this
+repository, which is used to also check for the coding style in the CI.
+
+.. code:: bash
+
+    python3 run-clang-format.py -i --style file -r src wasm
diff --git a/examples/run-native.sh b/examples/run-native.sh
new file mode 100644
index 000000000..81d86252e
--- /dev/null
+++ b/examples/run-native.sh
@@ -0,0 +1,19 @@
+# In source-root folder
+
+# Obtain an example model from the web.
+mkdir -p models
+wget --quiet --continue --directory models/ \
+    http://data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz 
+(cd models && tar -xzf ende.student.tiny11.tar.gz)
+
+# Patch the config-files generated from marian for use in bergamot.
+python3 bergamot-translator-tests/tools/patch-marian-for-bergamot.py \
+    --config-path models/ende.student.tiny11/config.intgemm8bitalpha.yml \
+    --ssplit-prefix-file 3rd-party/ssplit-cpp/split-cpp/nonbreaking_prefixes/nonbreaking_prefix.en
+
+# Patched config file will be available with .bergamot.yml suffix.
+CONFIG=models/ende.student.tiny11/config.intgemm8bitalpha.yml.bergamot.yml
+
+build/app/bergamot --model-config-paths $CONFIG --cpu-threads 4 <<< "Hello World!"
+# Hallo Welt!
+

From 13c55e2693d291078087de6e70f6773bb39cc415 Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Fri, 14 Jan 2022 10:30:38 +0000
Subject: [PATCH 331/442] Defer model loading to parallel worker thread (#303)

---
 src/translator/translation_model.cpp | 10 ++++++----
 src/translator/translation_model.h   |  1 +
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/src/translator/translation_model.cpp b/src/translator/translation_model.cpp
index 9d2eb0cdb..207f602d1 100644
--- a/src/translator/translation_model.cpp
+++ b/src/translator/translation_model.cpp
@@ -44,10 +44,6 @@ TranslationModel::TranslationModel(const Config &options, MemoryBundle &&memory
                                                                 srcIdx, trgIdx, shared_vcb);
     }
   }
-
-  for (size_t idx = 0; idx < replicas; idx++) {
-    loadBackend(idx);
-  }
 }
 
 void TranslationModel::loadBackend(size_t idx) {
@@ -172,6 +168,12 @@ Ptr<marian::data::CorpusBatch> TranslationModel::convertToMarianBatch(Batch &bat
 
 void TranslationModel::translateBatch(size_t deviceId, Batch &batch) {
   auto &backend = backend_[deviceId];
+
+  if (!backend.initialized) {
+    loadBackend(deviceId);
+    backend.initialized = true;
+  }
+
   BeamSearch search(options_, backend.scorerEnsemble, vocabs_.target());
   Histories histories = search.search(backend.graph, convertToMarianBatch(batch));
   batch.completeBatch(histories);
diff --git a/src/translator/translation_model.h b/src/translator/translation_model.h
index 6d2169494..3e79fdb38 100644
--- a/src/translator/translation_model.h
+++ b/src/translator/translation_model.h
@@ -107,6 +107,7 @@ class TranslationModel {
 
     Graph graph;
     ScorerEnsemble scorerEnsemble;
+    bool initialized{false};
   };
 
   // ShortlistGenerator is purely const, we don't need one per thread.

From e061b5613ea3b237941b78364119b055e6febaaa Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Sun, 16 Jan 2022 10:26:40 +0000
Subject: [PATCH 332/442] Treat most HTML elements as word-breaking (#286)

---
 .gitignore                     |   2 +-
 bergamot-translator-tests      |   2 +-
 src/tests/units/html_tests.cpp | 328 +++++++++++++++++++----
 src/translator/html.cpp        | 474 ++++++++++++++++++---------------
 src/translator/html.h          |  67 ++++-
 5 files changed, 595 insertions(+), 278 deletions(-)

diff --git a/.gitignore b/.gitignore
index 49093ba25..64c1aa3a7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -19,7 +19,7 @@ _deps
 wasm/test_page/node_modules
 build-wasm
 models
-wasm/test_page/bergamot-translator-worker.*
+wasm/test_page/js/bergamot-translator-worker.*
 
 # VSCode
 .vscode
diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 332e976df..b46987e96 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 332e976df4583793a09b6483b80b972621fcfadb
+Subproject commit b46987e96fc27b7e9488fbc36b53c07e1786784c
diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
index 48af7066c..d1a604d26 100644
--- a/src/tests/units/html_tests.cpp
+++ b/src/tests/units/html_tests.cpp
@@ -169,24 +169,136 @@ TEST_CASE("Test case html entities") {
   // These are all entities I would expect in innerHTML, since all other entities
   // can be encoded as UTF-8 so there's no need to encode them through &...; when
   // innerHTML encodes the DOM as HTML.
-  std::string input("<p data-attr=\"&quot;&apos;\">This is a sentence &lt;with&gt; named &amp; entities</p>\n");
+  std::string input("<p data-attr=\"&quot;&apos;\">This is a sentence &lt;with&gt; named &amp; entities</p>");
   HTML html(std::move(input), true);
-  CHECK(input == "This is a sentence <with> named & entities\n");
+  CHECK(input == "This is a sentence <with> named & entities");
 }
 
-TEST_CASE("Test self-closing tags should be treated as spaces") {
-  std::string input("<p>Space<br>please?</p>\n");
+TEST_CASE("Test self-closing tags should be treated as paragraph break") {
+  std::string test_str("<p>Space<br>please?</p>");
 
+  std::string input(test_str);
   HTML html(std::move(input), true);
-  CHECK(input == "Space please?\n");
+  CHECK(input == "Space\n\nplease?");
+
+  Response response;
+  std::string source_str("Space\n\nplease?");
+  std::vector<string_view> source_tokens{
+      string_view(source_str.data() + 0, 5),   // Space
+      string_view(source_str.data() + 5, 0),   // [EOS]
+      string_view(source_str.data() + 5, 2),   // \n\n
+      string_view(source_str.data() + 7, 1),   // p
+      string_view(source_str.data() + 8, 5),   // lease
+      string_view(source_str.data() + 13, 1),  // ?
+      string_view(source_str.data() + 14, 0),  // EOS
+  };
+  response.source.appendSentence("", source_tokens.begin(), source_tokens.begin() + 2);
+  response.source.appendSentence("\n\n", source_tokens.begin() + 3, source_tokens.end());
+
+  std::string target_str("Platz\n\nbitte?");
+  std::vector<string_view> target_tokens{
+      string_view(target_str.data() + 0, 5),   // Platz
+      string_view(target_str.data() + 5, 0),   // [EOS]
+      string_view(target_str.data() + 5, 2),   // \n\n
+      string_view(target_str.data() + 7, 5),   // bitte
+      string_view(target_str.data() + 12, 1),  // ?
+      string_view(target_str.data() + 13, 0),  // [EOS]
+  };
+  response.target.appendSentence("", target_tokens.begin(), target_tokens.begin() + 2);
+  response.target.appendSentence("", target_tokens.begin() + 3, target_tokens.end());
+  response.alignments = {{
+                             {1.0, 0.0},  //  Platz <- Space
+                             {0.0, 1.0}   //  [EOS] <- [EOS]
+                         },
+                         {
+                             {0.1, 0.9, 0.0, 0.0},  // _bitte <- _p + lease
+                             {0.0, 0.0, 1.0, 0.0},  //      ? <- ?
+                             {0.0, 0.0, 0.0, 1.0},  //  [EOS] <- [EOS]
+                         }};
+
+  // Main focus of this test is that the space that was introduced in the text
+  // that was being translated does not end up in the translation.
+  html.restore(response);
+  CHECK(response.source.text == "<p>Space<br>please?</p>");
+  CHECK(response.target.text == "<p>Platz<br>bitte?</p>");
+}
+
+TEST_CASE("Test inline tags should be treated as spaces") {
+  std::string test_str("un<u>der</u>line");
+
+  std::string input(test_str);
+  HTML html(std::move(input), true);
+  CHECK(input == "un der line");
+
+  Response response;
+  std::string source_str("un der line");
+  std::vector<string_view> source_tokens{
+      string_view(source_str.data() + 0, 2),   // un
+      string_view(source_str.data() + 2, 3),   // _de
+      string_view(source_str.data() + 5, 1),   // r
+      string_view(source_str.data() + 6, 5),   // _line
+      string_view(source_str.data() + 11, 0),  // EOS
+  };
+  response.source.appendSentence("", source_tokens.begin(), source_tokens.end());
+
+  std::string target_str("una linea der");
+  std::vector<string_view> target_tokens{
+      string_view(target_str.data() + 0, 3),   // una
+      string_view(target_str.data() + 3, 6),   // _linéa
+      string_view(target_str.data() + 9, 3),   // _de
+      string_view(target_str.data() + 12, 1),  // r
+      string_view(target_str.data() + 13, 0),  // [EOS]
+  };
+  response.target.appendSentence("", target_tokens.begin(), target_tokens.end());
+
+  response.alignments = {{{0.9795, 0.0127, 0.0002, 0.0066, 0.0009},
+                          {0.0098, 0.2967, 0.0156, 0.6640, 0.0138},
+                          {0.0214, 0.7472, 0.0626, 0.0745, 0.0943},
+                          {0.0022, 0.0230, 0.9357, 0.0165, 0.0226},
+                          {0.0122, 0.0240, 0.0085, 0.7427, 0.2125}}};
+
+  html.restore(response);
+  CHECK(response.source.text == "un <u>der</u> line");  // TODO leave spaces?
+  CHECK(response.target.text == "una linea <u>der</u>");
+}
+
+TEST_CASE("Test inline tags should not break words") {
+  std::string test_str("un<u>der</u>line");
+
+  std::string input(test_str);
+  HTML::Options options;
+  options.substituteInlineTagsWithSpaces = false;
+  HTML html(std::move(input), true, std::move(options));
+  CHECK(input == "underline");
+
+  Response response;
+  std::string source_str("underline");
+  std::vector<string_view> source_tokens{
+      string_view(source_str.data() + 0, 9),  // underline
+      string_view(source_str.data() + 9, 0),  // EOS
+  };
+  response.source.appendSentence("", source_tokens.begin(), source_tokens.end());
+
+  std::string target_str("subrayar");
+  std::vector<string_view> target_tokens{
+      string_view(target_str.data() + 0, 8),  // subrayar
+      string_view(target_str.data() + 8, 0),  // [EOS]
+  };
+  response.target.appendSentence("", target_tokens.begin(), target_tokens.end());
+
+  response.alignments = {identity_matrix<float>(2)};
+
+  html.restore(response);
+  CHECK(response.source.text == "<u></u>underline");  // TODO not spread <u> to whole word?
+  CHECK(response.target.text == "<u></u>subrayar");   // TODO not spread <u> to the whole word?
 }
 
 TEST_CASE("Test reconstruction of target sentence") {
   std::string input("<p>hello <b>world</b></p>\n");
   HTML html(std::move(input), true);
-  CHECK(input == "hello world\n");
+  CHECK(input == "hello world\n\n\n");  // tripple \n because \n + </p>
 
-  AnnotatedText source("hello world\n");
+  AnnotatedText source("hello world\n\n\n");
   recordSentenceFromByteRange(source, {
                                           ByteRange{0, 4},   // 0.0 "hell"
                                           ByteRange{4, 5},   // 0.1 "o"
@@ -194,7 +306,7 @@ TEST_CASE("Test reconstruction of target sentence") {
                                           ByteRange{11, 11}  // 0.3 ""
                                       });
 
-  AnnotatedText target("hallo Welt\n");
+  AnnotatedText target("hallo Welt\n\n\n");
   recordSentenceFromByteRange(target, {
                                           ByteRange{0, 4},   // 0.0 "hall"
                                           ByteRange{4, 5},   // 0.1 "o"
@@ -218,11 +330,11 @@ TEST_CASE("Test reconstruction of target sentence") {
 }
 
 TEST_CASE("Test reconstruction of target sentence with entities") {
-  std::string input("<p>hello <b>world &amp; friends!</b></p>\n");
+  std::string input("<p>hello <b>world &amp; friends!</b></p>");
   HTML html(std::move(input), true);
-  CHECK(input == "hello world & friends!\n");
+  CHECK(input == "hello world & friends!");
 
-  AnnotatedText source("hello world & friends!\n");
+  AnnotatedText source("hello world & friends!");
   recordSentenceFromByteRange(source, {
                                           ByteRange{0, 4},    // 0.0 "hell"
                                           ByteRange{4, 5},    // 0.1 "o"
@@ -233,7 +345,7 @@ TEST_CASE("Test reconstruction of target sentence with entities") {
                                           ByteRange{22, 22}   // 0.6 ""
                                       });
 
-  AnnotatedText target("hallo Welt & Freunde!\n");
+  AnnotatedText target("hallo Welt & Freunde!");
   recordSentenceFromByteRange(target, {
                                           ByteRange{0, 4},    // 0.0 "hall"
                                           ByteRange{4, 5},    // 0.1 "o"
@@ -252,11 +364,11 @@ TEST_CASE("Test reconstruction of target sentence with entities") {
   html.restore(response);
 
   std::vector<std::string> html_tokens_source{"",         "<p>hell", "o", " <b>world", " &amp;",
-                                              " friends", "!",       "",  "</b></p>\n"};
+                                              " friends", "!",       "",  "</b></p>"};
 
-  std::vector<std::string> html_tokens_target{"",         "<p>hall", "o", " <b>Welt",  " &amp;",
+  std::vector<std::string> html_tokens_target{"",         "<p>hall", "o", " <b>Welt", " &amp;",
 
-                                              " Freunde", "!",       "",  "</b></p>\n"};
+                                              " Freunde", "!",       "",  "</b></p>"};
 
   CHECK(asTokens(response.source) == html_tokens_source);
   CHECK(asTokens(response.target) == html_tokens_target);
@@ -264,10 +376,10 @@ TEST_CASE("Test reconstruction of target sentence with entities") {
 
 TEST_CASE("Test reconstruction of target with multiple sentences") {
   std::string input(
-      "<p>hello <b>world!</b> How does this <img> <b>deal <u>with multiple sentences?</u></b> Will it work?</p>\n");
+      "<p>hello <b>world!</b> How does this <img> <b>deal <u>with multiple sentences?</u></b> Will it work?</p>");
   HTML html(std::move(input), true);
 
-  AnnotatedText source("hello world! How does this  deal with multiple sentences? Will it work?\n");
+  AnnotatedText source("hello world! How does this  deal with multiple sentences? Will it work?");
   CHECK(source.text == input);
 
   recordSentenceFromByteRange(source, {
@@ -297,7 +409,7 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
                                           ByteRange{71, 71}   // 2.4 ""
                                       });
 
-  AnnotatedText target("hallo Welt! Wie geht das mit mehreren Sätzen um? Wird es funktionieren?\n");
+  AnnotatedText target("hallo Welt! Wie geht das mit mehreren Sätzen um? Wird es funktionieren?");
   recordSentenceFromByteRange(target, {
                                           ByteRange{0, 4},    // 0.0 "hall"
                                           ByteRange{4, 5},    // 0.1 "o"
@@ -327,7 +439,7 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
 
   std::vector<std::string> text_tokens_source{
       "",       "hall", "o",   " Welt", "!", "",  " ",    "Wie", " geht",          " das", " mit", " mehreren",
-      " Sätze", "n",    " um", "?",     "",  " ", "Wird", " es", " funktionieren", "?",    "",     "\n"};
+      " Sätze", "n",    " um", "?",     "",  " ", "Wird", " es", " funktionieren", "?",    "",     ""};
 
   CHECK(asTokens(target) == text_tokens_source);
 
@@ -360,26 +472,56 @@ TEST_CASE("Test reconstruction of target with multiple sentences") {
                                               " work",
                                               "?",
                                               "",
-                                              "</p>\n"};
+                                              "</p>"};
   CHECK(asTokens(response.source) == html_tokens_source);
 }
 
 TEST_CASE("Test self-closing tag (HTML5)") {
-  std::string input("<p>hello <img> <b>world</b> <u>and other <a href=\"#\">creatures</a></u></p>\n");
+  std::string input("<p>hello <img> <b>world</b> <u>and other <a href=\"#\">creatures</a></u></p>");
   HTML html(std::move(input), true);
-  CHECK(input == "hello  world and other creatures\n");  // Note double space between "hello" and "world"
+  CHECK(input == "hello  world and other creatures");  // Note double space between "hello" and "world"
 }
 
-TEST_CASE("Test empty self-closing tag at end of input") {
+TEST_CASE("Test empty void tag at end of input") {
   std::string input("hello <br>");
   HTML html(std::move(input), true);
   CHECK(input == "hello ");
+
+  Response response;
+  std::string sentence_str("hello ");
+  std::vector<string_view> sentence{
+      string_view(sentence_str.data() + 0, 4),  // 0.0 hell
+      string_view(sentence_str.data() + 4, 2),  // 0.1 o_
+      string_view(sentence_str.data() + 6, 0),  // 0.2 [EOS]
+  };
+  response.source.appendSentence("", sentence.begin(), sentence.end());
+  response.target.appendSentence("", sentence.begin(), sentence.end());
+  response.alignments = {identity_matrix<float>(3)};
+
+  html.restore(response);
+  CHECK(response.source.text == "hello <br>");
+  CHECK(response.target.text == "hello <br>");
 }
 
 TEST_CASE("Test empty tag pair at end of input") {
   std::string input("hello <u></u>");
   HTML html(std::move(input), true);
   CHECK(input == "hello ");
+
+  Response response;
+  std::string sentence_str("hello ");
+  std::vector<string_view> sentence{
+      string_view(sentence_str.data() + 0, 4),  // 0.0 hell
+      string_view(sentence_str.data() + 4, 2),  // 0.1 o_
+      string_view(sentence_str.data() + 6, 0),  // 0.2 [EOS]
+  };
+  response.source.appendSentence("", sentence.begin(), sentence.end());
+  response.target.appendSentence("", sentence.begin(), sentence.end());
+  response.alignments = {identity_matrix<float>(3)};
+
+  html.restore(response);
+  CHECK(response.source.text == "hello <u></u>");
+  CHECK(response.target.text == "hello <u></u>");
 }
 
 TEST_CASE("Test empty self-closing pair at end of input in parent") {
@@ -391,11 +533,11 @@ TEST_CASE("Test empty self-closing pair at end of input in parent") {
 TEST_CASE("Test empty tag") {
   std::string test_str(
       "<p id=\"1\">hello <img id=\"1.1\"><span id=\"1.2\"><u id=\"1.2.1\"></u><b id=\"1.2.2\"></b><img "
-      "id=\"1.2.3\">world</span></p>\n");
+      "id=\"1.2.3\">world</span></p>");
 
   std::string input(test_str);
   HTML html(std::move(input), true);
-  CHECK(input == "hello world\n");
+  CHECK(input == "hello world");
 
   Response response;
 
@@ -407,11 +549,7 @@ TEST_CASE("Test empty tag") {
       string_view(sentence_str.data() + 11, 0),  // 0.3 ""
   };
   response.source.appendSentence("", sentence.begin(), sentence.end());
-  response.source.appendEndingWhitespace("\n");
-
   response.target.appendSentence("", sentence.begin(), sentence.end());
-  response.target.appendEndingWhitespace("\n");
-
   response.alignments = {identity_matrix<float>(4)};
 
   html.restore(response);
@@ -424,19 +562,20 @@ TEST_CASE("Test <script> element") {
 
   std::string input(test_str);
   HTML html(std::move(input), true);
-  CHECK(input == "hello world");
+  CHECK(input == "hello \n\nworld");
 
   Response response;
-  std::string sentence_str("hello world");
+  std::string sentence_str("hello \n\nworld");
   std::vector<string_view> sentence{
       string_view(sentence_str.data() + 0, 4),   // 0.0 hell
-      string_view(sentence_str.data() + 4, 1),   // 0.1 o
-      string_view(sentence_str.data() + 5, 6),   // 0.2 _world
-      string_view(sentence_str.data() + 11, 0),  // 0.3 ""
+      string_view(sentence_str.data() + 4, 2),   // 0.1 o_
+      string_view(sentence_str.data() + 6, 2),   // 0.2 \n\n
+      string_view(sentence_str.data() + 8, 5),   // 0.3 world
+      string_view(sentence_str.data() + 13, 0),  // 0.4 ""
   };
   response.source.appendSentence("", sentence.begin(), sentence.end());
   response.target.appendSentence("", sentence.begin(), sentence.end());
-  response.alignments = {identity_matrix<float>(4)};
+  response.alignments = {identity_matrix<float>(5)};
 
   html.restore(response);
   CHECK(response.source.text == test_str);
@@ -466,10 +605,10 @@ TEST_CASE("Test comment") {
   CHECK(response.target.text == test_str);
 }
 
-TEST_CASE("End-to-end translation") {
-  std::string input("<p>I <b>like</b> to <u>drive</u> this car.</p>\n");
+TEST_CASE("End-to-end translation", "[!mayfail]") {
+  std::string input("<p>I <b>like</b> to <u>drive</u> this car.</p>");
   HTML html(std::move(input), true);
-  CHECK(input == "I like to drive this car.\n");
+  CHECK(input == "I like to drive this car.");
 
   Response response;
 
@@ -500,7 +639,6 @@ TEST_CASE("End-to-end translation") {
         string_view(sentence_str.data() + 25, 0),  // 0.7 ""
     };
     response.source.appendSentence("", sentence.begin(), sentence.end());
-    response.source.appendEndingWhitespace("\n");
   }
 
   {
@@ -517,7 +655,6 @@ TEST_CASE("End-to-end translation") {
         string_view(sentence_str.data() + 28, 0),  // 0.8 ""
     };
     response.target.appendSentence("", sentence.begin(), sentence.end());
-    response.target.appendEndingWhitespace("\n");
   }
 
   html.restore(response);
@@ -536,27 +673,116 @@ TEST_CASE("End-to-end translation") {
         string_view(sentence_str.data() + 42, 0),  // 0.7 ""
     };
     source.appendSentence("", sentence.begin(), sentence.end());
-    source.appendEndingWhitespace("</p>\n");
+    source.appendEndingWhitespace("</p>");
 
     CHECK(asTokens(response.source) == asTokens(source));
   }
 
   {
     AnnotatedText target;
-    std::string sentence_str("<p>Ich <u>fahre</u> <b>gerne</b> dieses Auto.");
+    // Empty <b></b> because the space token after "Ich" has "<p><b>" markup, passed down from "<b>like</b>"
+    std::string sentence_str("<p>Ich <b></b><u>fahre</u> <b>gerne</b> dieses Auto.");
     std::vector<string_view> sentence{
         string_view(sentence_str.data() + 0, 6),    // 0.0 "<p>Ich"
-        string_view(sentence_str.data() + 6, 4),    // 0.1 " <u>"
-        string_view(sentence_str.data() + 10, 4),   // 0.2 "fahr"
-        string_view(sentence_str.data() + 14, 1),   // 0.3 "e"
-        string_view(sentence_str.data() + 15, 13),  // 0.4 "</u> <b>gerne"
-        string_view(sentence_str.data() + 28, 11),  // 0.5 "</b> dieses"
-        string_view(sentence_str.data() + 39, 5),   // 0.6 " Auto"
-        string_view(sentence_str.data() + 44, 1),   // 0.7 "."
-        string_view(sentence_str.data() + 45, 0),   // 0.8 ""
+        string_view(sentence_str.data() + 6, 4),    // 0.1 " <b>"
+        string_view(sentence_str.data() + 10, 11),  // 0.2 "</b><u>fahr"
+        string_view(sentence_str.data() + 21, 1),   // 0.3 "e"
+        string_view(sentence_str.data() + 22, 13),  // 0.4 "</u> <b>gerne"
+        string_view(sentence_str.data() + 35, 11),  // 0.5 "</b> dieses"
+        string_view(sentence_str.data() + 46, 5),   // 0.6 " Auto"
+        string_view(sentence_str.data() + 51, 1),   // 0.7 "."
+        string_view(sentence_str.data() + 52, 0),   // 0.8 ""
+    };
+    target.appendSentence("", sentence.begin(), sentence.end());
+    target.appendEndingWhitespace("</p>");
+
+    CHECK(asTokens(response.target) == asTokens(target));
+  }
+}
+
+TEST_CASE("End-to-end translation when no words with markup align", "[!mayfail]") {
+  std::string input("<p>I <b>like</b> to <u>drive</u> this car.</p>");
+  HTML html(std::move(input), true);
+  CHECK(input == "I like to drive this car.");
+
+  Response response;
+
+  // clang-format off
+  response.alignments = std::vector<std::vector<std::vector<float>>>{{
+    {0.5360, 0.4405, 0.0142, 0.0061, 0.0029, 0.0001, 0.0000, 0.0001},
+    {0.0451, 0.0602, 0.5120, 0.2584, 0.1145, 0.0062, 0.0019, 0.0017},
+    {0.0392, 0.0009, 0.6535, 0.2293, 0.0492, 0.0199, 0.0014, 0.0067},
+    {0.0007, 0.0036, 0.0112, 0.0118, 0.9209, 0.0449, 0.0050, 0.0019},
+    {0.0000, 0.0004, 0.0008, 0.0047, 0.0163, 0.9683, 0.0045, 0.0050},
+    {0.0011, 0.0046, 0.0039, 0.0090, 0.0023, 0.0024, 0.9648, 0.0119},
+    {0.0840, 0.0744, 0.1545, 0.1330, 0.1818, 0.1722, 0.0859, 0.1143},
+  }};
+  // clang-format on
+
+  {
+    std::string sentence_str("I like to drive this car.");
+    std::vector<string_view> sentence{
+        string_view(sentence_str.data() + 0, 1),   // 0.0 "I"
+        string_view(sentence_str.data() + 1, 5),   // 0.1 " like"
+        string_view(sentence_str.data() + 6, 3),   // 0.2 " to"
+        string_view(sentence_str.data() + 9, 6),   // 0.3 " drive"
+        string_view(sentence_str.data() + 15, 5),  // 0.4 " this"
+        string_view(sentence_str.data() + 20, 4),  // 0.5 " car"
+        string_view(sentence_str.data() + 24, 1),  // 0.6 "."
+        string_view(sentence_str.data() + 25, 0),  // 0.7 [EOS]
+    };
+    response.source.appendSentence("", sentence.begin(), sentence.end());
+  }
+
+  {
+    std::string sentence_str("Rád řídím to auto.");
+    std::vector<string_view> sentence{
+        string_view(sentence_str.data() + 0, 4),   // 0.0 "Rád"
+        string_view(sentence_str.data() + 4, 6),   // 0.1 " říd"
+        string_view(sentence_str.data() + 10, 3),  // 0.2 "ím"
+        string_view(sentence_str.data() + 13, 3),  // 0.3 "_to"
+        string_view(sentence_str.data() + 16, 5),  // 0.4 " auto"
+        string_view(sentence_str.data() + 21, 1),  // 0.5 "."
+        string_view(sentence_str.data() + 22, 0),  // 0.6 [EOS]
+    };
+    response.target.appendSentence("", sentence.begin(), sentence.end());
+  }
+
+  html.restore(response);
+
+  {
+    AnnotatedText source;
+    std::string sentence_str("<p>I <b>like</b> to <u>drive</u> this car.");
+    std::vector<string_view> sentence{
+        string_view(sentence_str.data() + 0, 4),   // 0.0 "<p>I"
+        string_view(sentence_str.data() + 4, 8),   // 0.1 " <b>like"
+        string_view(sentence_str.data() + 12, 7),  // 0.2 "</b> to"
+        string_view(sentence_str.data() + 19, 9),  // 0.3 " <u>drive"
+        string_view(sentence_str.data() + 28, 9),  // 0.4 "</u> this"
+        string_view(sentence_str.data() + 37, 4),  // 0.5 " car"
+        string_view(sentence_str.data() + 41, 1),  // 0.6 "."
+        string_view(sentence_str.data() + 42, 0),  // 0.7 ""
+    };
+    source.appendSentence("", sentence.begin(), sentence.end());
+    source.appendEndingWhitespace("</p>");
+
+    CHECK(asTokens(response.source) == asTokens(source));
+  }
+
+  {
+    AnnotatedText target;
+    std::string sentence_str("<p>Rád <b></b>řídím <u></u>to auto.");
+    std::vector<string_view> sentence{
+        string_view(sentence_str.data() + 0, 7),    // 0.0 "<p>Rád"
+        string_view(sentence_str.data() + 7, 13),   // 0.1 " <b></b>říd"
+        string_view(sentence_str.data() + 20, 3),   // 0.2 "ím"
+        string_view(sentence_str.data() + 23, 10),  // 0.3 "_<u></u>to"
+        string_view(sentence_str.data() + 33, 5),   // 0.4 " auto"
+        string_view(sentence_str.data() + 38, 1),   // 0.5 "."
+        string_view(sentence_str.data() + 39, 0),   // 0.6 [EOS]
     };
     target.appendSentence("", sentence.begin(), sentence.end());
-    target.appendEndingWhitespace("</p>\n");
+    target.appendEndingWhitespace("</p>");
 
     CHECK(asTokens(response.target) == asTokens(target));
   }
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index 13ab422ac..e2ea72223 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -43,7 +43,7 @@ void encodeEntities(string_view const &input, std::string &output) {
 
 size_t countPrefixWhitespaces(string_view const &input) {
   size_t size = 0;
-  while (size < input.size() && input[size] == ' ') ++size;
+  while (size < input.size() && std::isspace(input[size])) ++size;
   return size;
 }
 
@@ -59,6 +59,8 @@ std::ostream &operator<<(std::ostream &out, HTML::Tag const *tag) {
       return out << "<!--" << tag->data << "-->";
     case HTML::Tag::PROCESSING_INSTRUCTION:
       return out << "<?" << tag->data << "?>";
+    case HTML::Tag::WHITESPACE:
+      return out << "[inserted space]";
   }
   return out << "[Unknown tag type]";
 }
@@ -107,27 +109,8 @@ class reversed {
   T const &container_;
 };
 
-bool isBlockElement(std::string_view const &name) {
-  // List of elements that we expect might occur inside words, and that should
-  // not introduce spacings around them. Not strictly inline elements, nor flow
-  // elements. See also https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Content_categories
-  static std::unordered_set<std::string> inlineishElements{
-      "abbr",  "a",    "b",      "em",  "i",   "kbd",  "mark", "math", "output", "q",   "ruby",
-      "small", "span", "strong", "sub", "sup", "time", "u",    "var",  "wbr",    "ins", "del"};
-
-  return inlineishElements.find(std::string(name)) == inlineishElements.end();
-}
-
-bool isVoidTag(std::string_view const &name) {
-  // List of elements for which we do not expect a closing tag, or self-closing
-  // elements in XHTML. See also https://developer.mozilla.org/en-US/docs/Glossary/Empty_element
-  // More relevant source of this list:
-  // https://searchfox.org/mozilla-central/rev/7d17fd1fe9f0005a2fb19e5d53da4741b06a98ba/dom/base/FragmentOrElement.cpp#1791
-  static std::unordered_set<std::string> voidElements{"area",  "base",  "basefont", "bgsound", "br",    "col",
-                                                      "embed", "frame", "hr",       "img",     "input", "keygen",
-                                                      "link",  "meta",  "param",    "source",  "track", "wbr"};
-
-  return voidElements.find(std::string(name)) != voidElements.end();
+bool contains(std::unordered_set<std::string> const &set, std::string const &name) {
+  return set.find(name) != set.end();
 }
 
 void diffTags(HTML::Taint const &prev, HTML::Taint const &curr, HTML::Taint &opening, HTML::Taint &closing) {
@@ -187,8 +170,6 @@ AnnotatedText apply(AnnotatedText const &in, Fun fun) {
   return out;
 }
 
-bool isContinuation(string_view str) { return !str.empty() && str.compare(0, 1, " ", 1) != 0; }
-
 bool hasAlignments(Response const &response) {
   // Test for each sentence individually as a sentence may be empty (or there)
   // might be no sentences, so just testing for alignments.empty() would not be
@@ -207,85 +188,11 @@ bool hasAlignments(Response const &response) {
   return true;
 }
 
-void hardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
-  // For each sentence...
-  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
-    alignments.emplace_back();
-
-    // Hard-align: find for each target token the most prevalent source token
-    // Note: only search from 0 to N-1 because token N is end-of-sentence token
-    // that can only align with the end-of-sentence token of the target
-    for (size_t t = 0; t + 1 < response.target.numWords(sentenceIdx); ++t) {
-      size_t maxS = 0;
-      for (size_t s = 1; s + 1 < response.source.numWords(sentenceIdx); ++s) {
-        if (response.alignments[sentenceIdx][t][s] > response.alignments[sentenceIdx][t][maxS]) {
-          maxS = s;
-        }
-      }
-
-      alignments.back().push_back(maxS);
-    }
-
-    // Next, we try to smooth out these selected alignments with a few heuristics
-    for (size_t t = 1; t + 1 < response.target.numWords(sentenceIdx); ++t) {
-      // If this token is a continuation of a previous token, pick the tags from the most
-      // prevalent token for the whole word.
-      if (isContinuation(response.target.word(sentenceIdx, t))) {
-        // Note: only looking at the previous token since that will already
-        // have this treatment applied to it.
-        size_t currSentenceIdx = alignments.back()[t];
-        size_t prevSentenceIdx = alignments.back()[t - 1];
-        float currScore = response.alignments[sentenceIdx][t][currSentenceIdx];
-        float prevScore = response.alignments[sentenceIdx][t - 1][prevSentenceIdx];
-
-        if (currScore > prevScore) {
-          // Apply this to all previous tokens in the word
-          for (size_t i = t;; --i) {
-            alignments.back()[i] = currSentenceIdx;
-
-            // Stop if this was the first token or the beginning of the word
-            if (i == 0 || !isContinuation(response.target.word(sentenceIdx, i))) break;
-          }
-        } else {
-          alignments.back()[t] = prevSentenceIdx;
-        }
-      }
-    }
-
-    // Always align target end with source end
-    alignments.back().push_back(response.source.numWords(sentenceIdx) - 1);
-  }
-}
-
-// Internal type used to point to a position in HTML::spans_.
-typedef std::vector<HTML::Span>::const_iterator SpanIterator;
-
-void copyTaint(Response const &response, std::vector<std::vector<size_t>> const &alignments,
-               std::vector<SpanIterator> const &sourceTokenSpans, std::vector<SpanIterator> &targetTokenSpans) {
-  size_t offset = 0;
-
-  // Fill targetTokenSpans based on the alignments we just made up.
-  // NOTE: this should match the exact order of Apply()
-  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
-    targetTokenSpans.push_back(sourceTokenSpans[offset]);  // token_tag for sentence ending gap
-    for (size_t t = 0; t < response.target.numWords(sentenceIdx); ++t) {
-      size_t s = alignments[sentenceIdx][t];
-      assert(s < response.source.numWords(sentenceIdx));
-      targetTokenSpans.push_back(sourceTokenSpans[offset + 1 + s]);  // +1 for prefix gap
-    }
-
-    offset += response.source.numWords(sentenceIdx) + 1;  // +1 for prefix gap
-  }
-
-  assert(offset < sourceTokenSpans.size());
-  targetTokenSpans.push_back(sourceTokenSpans[offset]);  // token_tag for ending whitespace
-}
-
 // Little helper class to append HTML to a token
 class TokenFormatter {
  public:
   explicit TokenFormatter(string_view token)
-      : html_(), offset_(0), whitespaceSize_(countPrefixWhitespaces(token)), closeLeft_(true) {
+      : html_(), offset_(0), whitespaceOffset_(0), whitespaceSize_(countPrefixWhitespaces(token)), closeLeft_(true) {
     // Do encoding of any entities that popped up in the translation
     encodeEntities(token, html_);
   }
@@ -303,6 +210,7 @@ class TokenFormatter {
       std::string closeTag = format("</{}>", tag->name);
       html_.insert(offset_ + (closeLeft_ ? 0 : whitespaceSize_), closeTag);
       offset_ += closeTag.size();
+      if (closeLeft_) whitespaceOffset_ += closeTag.size();
     }
 
     for (HTML::Tag const *tag : opening) {
@@ -318,17 +226,28 @@ class TokenFormatter {
         case HTML::Tag::PROCESSING_INSTRUCTION:
           openTag = format("<?{}?>", tag->data);
           break;
+        case HTML::Tag::WHITESPACE: {
+          // Try to eat two newlines (paragraph break) from our segment
+          auto pos = html_.find("\n\n", whitespaceOffset_);
+          if (pos != std::string::npos && pos < whitespaceOffset_ + whitespaceSize_) {
+            html_.erase(pos, 2);
+            whitespaceSize_ -= 2;
+          }
+        } break;
       }
+
       html_.insert(offset_ + whitespaceSize_, openTag);
       offset_ += openTag.size();
-      closeLeft_ = false;
+      closeLeft_ = closeLeft_ && openTag.empty();
     }
   }
 
  private:
-  std::string html_;       // Output html
-  size_t offset_;          // Size added by prepending HTML
-  size_t whitespaceSize_;  // number of prefix whitespace characters
+  std::string html_;         // Output html
+  size_t offset_;            // Size added by prepending HTML
+  size_t whitespaceOffset_;  // position of prefix whitespace characters
+                             // (it moves as closing tags are prepended)
+  size_t whitespaceSize_;    // number of prefix whitespace characters
 
   // Close tags we want to show up left (before) the token, but open tags
   // ideally come directly after any prefix whitespace. However, some tokens
@@ -339,96 +258,6 @@ class TokenFormatter {
   bool closeLeft_;
 };
 
-AnnotatedText restoreSource(AnnotatedText const &in, std::vector<HTML::Span> const &sourceSpans,
-                            std::vector<SpanIterator> &sourceTokenSpans) {
-  auto spanIt = sourceSpans.begin();
-  auto prevIt = sourceSpans.begin();  // safe because first span is always empty span, and
-                                      // and the while-loop below will do the rest
-  assert(prevIt == sourceSpans.end() || prevIt->tags.empty());
-
-  return apply(in, [&](ByteRange range, string_view token, bool last) {
-    TokenFormatter formatter(token);
-
-    // Potential issue: spans and tokens can intersect, e.g.
-    //
-    //    text  <p> h <u> e </u> ll o </p>
-    //   spans     |1|   |2|    |3333| (so only 2 is tainted with <p><u>, others only <p>)
-    //  tokens     |111111111111111|2|
-    //
-    // Now 1 covers span 1 to 3, so what taint should it get? Just <p>, or <p><u>?
-    // Note: only relevant if isBlockElement is used. If we just insert spaces
-    // around all elements, every segment of `hello` will be a token.
-
-    // Seek to the last span that overlaps with this token
-    while (true) {
-      formatter.append(prevIt->tags, spanIt->tags);
-      prevIt = spanIt;
-
-      if (spanIt + 1 != sourceSpans.end() && ((spanIt + 1)->begin < range.end || last)) {
-        spanIt++;
-        continue;
-      }
-
-      break;
-    }
-
-    // TODO: This is just the taint of the last span, not the ones in between.
-    // This makes us lose some markup of parts of tokens as described above.
-    sourceTokenSpans.push_back(prevIt);
-
-    return std::move(formatter.html());
-  });
-}
-
-AnnotatedText restoreTarget(AnnotatedText const &in, std::vector<HTML::Span> const &sourceSpans,
-                            std::vector<SpanIterator> const &targetTokenSpans) {
-  auto prevSpan = sourceSpans.begin();
-  auto targetSpanIt = targetTokenSpans.begin();
-
-  AnnotatedText out = apply(in, [&](ByteRange range, string_view token, bool last) {
-    TokenFormatter formatter(token);
-
-    // First we scan through spans_ to catch up to the span assigned to this
-    // token. We're only interested in empty spans (empty and void elements)
-    for (auto span_it = prevSpan + 1; span_it < *targetSpanIt; span_it++) {
-      // We're only interested in empty spans between the spans in targetSpanIt
-      if (span_it->size() != 0) continue;
-
-      formatter.append(prevSpan->tags, span_it->tags);
-
-      // Note: here, not in 3rd part of for-statement because we don't want to
-      // set prevSpan if the continue clause at the beginning of this for-loop
-      // was hit.
-      prevSpan = span_it;
-    }
-
-    // Now do the same thing but for our target set of tags. Note that we cannot
-    // combine this in the for-loop above (i.e. `span_it <= *targetSpanIt`)
-    // because there is no guarantee that the order in `targetTokenSpans` is
-    // the same as that of `spans`.
-    formatter.append(prevSpan->tags, (*targetSpanIt)->tags);
-
-    // If this is the last token of the response, close all open tags.
-    if (last) {
-      // Note: this assert is true due to our current implementation of
-      // HardAlignments() that always matches the last token of the input with
-      // the last token of the output. But lets assume someone someday changes
-      // HardAlignments(), and then this for-loop will be necessary.
-      // assert((*targetSpanIt)->tags.empty());
-      formatter.append((*targetSpanIt)->tags, HTML::Taint());
-    }
-
-    prevSpan = *targetSpanIt++;
-
-    return std::move(formatter.html());
-  });
-
-  // Assert that we did in fact use all our taints
-  assert(targetSpanIt == targetTokenSpans.end());
-
-  return out;
-}
-
 size_t debugCountTokens(AnnotatedText const &text) {
   size_t tokens = 1;  // for the ending gap
   for (size_t sentenceIdx = 0; sentenceIdx < text.numSentences(); ++sentenceIdx) {
@@ -441,8 +270,9 @@ size_t debugCountTokens(AnnotatedText const &text) {
 
 namespace marian::bergamot {
 
-HTML::HTML(std::string &&source, bool process_markup) {
+HTML::HTML(std::string &&source, bool process_markup, Options &&options) : options_(std::move(options)) {
   if (!process_markup) return;
+
   std::string original = std::move(source);
   markup::instream in(original.data(), original.data() + original.size());
   markup::Scanner scanner(in);
@@ -450,6 +280,8 @@ HTML::HTML(std::string &&source, bool process_markup) {
 
   Tag *tag;
   Taint stack;
+  bool addSentenceBreak = false;
+  bool addSpace = false;
   spans_.push_back(Span{0, 0, {}});
 
   bool stop = false;
@@ -463,24 +295,41 @@ HTML::HTML(std::string &&source, bool process_markup) {
         break;
 
       case markup::Scanner::TT_TEXT: {
+        // If the previous segment was the open or close tag of a block element
+        // we treat the text after it as a new sentence.
+        if (addSentenceBreak) {
+          if (!(source.empty() || (source.size() > 2 && source.substr(source.size() - 2) == ""))) {
+            stack.push_back(makeTag({Tag::WHITESPACE}));
+            // Important: span->size() == 0 to make it behave as a void element.
+            // Also important: position before the \n\n tokens, not after, to
+            // make it easier to remove them later through apply().
+            spans_.push_back(Span{source.size(), source.size(), stack});
+            source.append("\n\n");  // TODO assumes ssplit-mode = wrapped_text
+            stack.pop_back();
+          }
+          addSentenceBreak = false;
+        }
+
+        // If the previous segment was an open or close tag, it might be best
+        // to add a space to make sure we don't append to the previous word.
+        if (addSpace) {
+          if (options_.substituteInlineTagsWithSpaces && !source.empty() && !std::isspace(source.back()) &&
+              !std::isspace(scanner.value()[0])) {
+            source.push_back(' ');
+          }
+          addSpace = false;
+        }
+
         auto begin = source.size();
         source.append(scanner.value());
         spans_.push_back(Span{begin, source.size(), stack});
       } break;
 
-      case markup::Scanner::TT_TAG_START:
-        // If it makes sense to treat this element as a break in a word (e.g.
-        // <br>, <img>, <li>) make sure it does so in this text as well.
-        // TODO: Strong assumption here that the language uses spaces to
-        // separate words
-        if (isBlockElement(scanner.tag()) && !source.empty() && source.back() != ' ') source.push_back(' ');
-
-        // pool_ takes ownership of our tag, makes sure it's freed when necessary
-        pool_.emplace_back(new Tag{isVoidTag(scanner.tag()) ? Tag::VOID_ELEMENT : Tag::ELEMENT,
-                                   std::string(scanner.tag()), std::string()});
+      case markup::Scanner::TT_TAG_START: {
+        std::string name(scanner.tag());
 
         // Tag *tag is used by attribute parsing
-        tag = pool_.back().get();
+        tag = makeTag({contains(options_.voidTags, name) ? Tag::VOID_ELEMENT : Tag::ELEMENT, std::move(name)});
 
         stack.push_back(tag);
 
@@ -491,7 +340,14 @@ HTML::HTML(std::string &&source, bool process_markup) {
           spans_.push_back(Span{source.size(), source.size(), stack});
           stack.pop_back();
         }
-        break;
+
+        // Treat non-inline HTML tags as spaces that break up words.
+        if (!contains(options_.inlineTags, tag->name)) {
+          addSentenceBreak = true;
+        } else {
+          addSpace = true;
+        }
+      } break;
 
       case markup::Scanner::TT_TAG_END:
         // Note: self-closing tags emit TT_TAG_END immediately after TT_TAG_START
@@ -508,6 +364,13 @@ HTML::HTML(std::string &&source, bool process_markup) {
           spans_.push_back(Span{source.size(), source.size(), stack});
 
         stack.pop_back();
+
+        // Add space if necessary
+        if (!contains(options_.inlineTags, std::string(scanner.tag()))) {
+          addSentenceBreak = true;
+        } else {
+          addSpace = true;
+        }
         break;
 
       case markup::Scanner::TT_ATTRIBUTE:
@@ -516,18 +379,16 @@ HTML::HTML(std::string &&source, bool process_markup) {
         break;
 
       case markup::Scanner::TT_COMMENT_START:
-        // pool_ takes ownership of our tag, makes sure it's freed when necessary
-        pool_.emplace_back(new Tag{Tag::COMMENT});
-        tag = pool_.back().get();
+        // Tag *tag is used when TT_DATA is seen to add the comment's content.
+        tag = makeTag({Tag::COMMENT});
         stack.push_back(tag);
         spans_.push_back(Span{source.size(), source.size(), stack});
         stack.pop_back();
         break;
 
       case markup::Scanner::TT_PROCESSING_INSTRUCTION_START:
-        // pool_ takes ownership of our tag, makes sure it's freed when necessary
-        pool_.emplace_back(new Tag{Tag::PROCESSING_INSTRUCTION});
-        tag = pool_.back().get();
+        // Tag *tag is used when TT_DATA is seen to add the PI's content.
+        tag = makeTag({Tag::PROCESSING_INSTRUCTION});
         stack.push_back(tag);
         spans_.push_back(Span{source.size(), source.size(), stack});
         stack.pop_back();
@@ -551,7 +412,7 @@ HTML::HTML(std::string &&source, bool process_markup) {
   if (!stack.empty()) throw BadHTML(format("Not all tags were closed: {}", stack));
 
   // Add a trailing span (that's empty) to signify all closed tags.
-  spans_.emplace_back(Span{source.size() + 1, source.size() + 1, stack});
+  spans_.emplace_back(Span{source.size(), source.size(), stack});
 }
 
 void HTML::restore(Response &response) {
@@ -580,7 +441,7 @@ void HTML::restore(Response &response) {
 
   // RestoreSource re-inserts HTML into the source text, but also identifies
   // which span each source token fits into best.
-  AnnotatedText source = restoreSource(response.source, spans_, sourceTokenSpans);
+  AnnotatedText source = restoreSource(response.source, sourceTokenSpans);
   assert(sourceTokenSpans.size() == debugCountTokens(response.source));
 
   // Find for every token in target the token in source that best matches.
@@ -591,10 +452,193 @@ void HTML::restore(Response &response) {
   copyTaint(response, alignments, sourceTokenSpans, targetTokenSpans);
   assert(targetTokenSpans.size() == debugCountTokens(response.target));
 
-  AnnotatedText target = restoreTarget(response.target, spans_, targetTokenSpans);
+  AnnotatedText target = restoreTarget(response.target, targetTokenSpans);
 
   response.source = source;
   response.target = target;
 }
 
+AnnotatedText HTML::restoreSource(AnnotatedText const &in, std::vector<SpanIterator> &sourceTokenSpans) {
+  auto spanIt = spans_.begin();
+  auto prevIt = spans_.begin();  // safe because first span is always empty span, and
+                                 // and the while-loop below will do the rest
+  assert(prevIt == spans_.end() || prevIt->tags.empty());
+
+  return apply(in, [&](ByteRange range, string_view token, bool last) {
+    TokenFormatter formatter(token);
+
+    // Potential issue: spans and tokens can intersect, e.g.
+    //
+    //    text  <p> h <u> e </u> ll o </p>
+    //   spans     |1|   |2|    |3333| (so only 2 is tainted with <p><u>, others only <p>)
+    //  tokens     |111111111111111|2|
+    //
+    // Now 1 covers span 1 to 3, so what taint should it get? Just <p>, or <p><u>?
+    // Note: only relevant if isBlockElement is used. If we just insert spaces
+    // around all elements, every segment of `hello` will be a token.
+
+    // Seek to the last span that overlaps with this token
+    while (true) {
+      formatter.append(prevIt->tags, spanIt->tags);
+      prevIt = spanIt;
+
+      if (spanIt + 1 != spans_.end() && ((spanIt + 1)->begin < range.end || last)) {
+        spanIt++;
+        continue;
+      }
+
+      break;
+    }
+
+    // TODO: This is just the taint of the last span, not the ones in between.
+    // This makes us lose some markup of parts of tokens as described above.
+    sourceTokenSpans.push_back(prevIt);
+
+    return std::move(formatter.html());
+  });
+}
+
+AnnotatedText HTML::restoreTarget(AnnotatedText const &in, std::vector<SpanIterator> const &targetTokenSpans) {
+  auto prevSpan = spans_.cbegin();
+  auto targetSpanIt = targetTokenSpans.begin();
+
+  AnnotatedText out = apply(in, [&](ByteRange range, string_view token, bool last) {
+    TokenFormatter formatter(token);
+
+    // First we scan through spans_ to catch up to the span assigned to this
+    // token. We're only interested in empty spans (empty and void elements)
+    for (auto span_it = prevSpan; span_it < *targetSpanIt; span_it++) {
+      // We're only interested in empty spans or spans that would otherwise get
+      // lost because they didn't align with anything between the spans in
+      // targetSpanIt
+      // TODO That std::find makes this O(N*N) NOT GOOD NOT GOOD
+      if (span_it->size() != 0 &&
+          std::find(targetTokenSpans.begin(), targetTokenSpans.end(), span_it) != targetTokenSpans.end())
+        continue;
+
+      formatter.append(prevSpan->tags, span_it->tags);
+
+      // Note: here, not in 3rd part of for-statement because we don't want to
+      // set prevSpan if the continue clause at the beginning of this for-loop
+      // was hit.
+      prevSpan = span_it;
+    }
+
+    // Now do the same thing but for our target set of tags. Note that we cannot
+    // combine this in the for-loop above (i.e. `span_it <= *targetSpanIt`)
+    // because there is no guarantee that the order in `targetTokenSpans` is
+    // the same as that of `spans`.
+    formatter.append(prevSpan->tags, (*targetSpanIt)->tags);
+
+    // If this is the last token of the response, close all open tags.
+    if (last) {
+      // Note: this assert is true due to our current implementation of
+      // HardAlignments() that always matches the last token of the input with
+      // the last token of the output. But lets assume someone someday changes
+      // HardAlignments(), and then this for-loop will be necessary.
+      // assert((*targetSpanIt)->tags.empty());
+      formatter.append((*targetSpanIt)->tags, HTML::Taint());
+    }
+
+    prevSpan = *targetSpanIt;
+    ++targetSpanIt;
+
+    return std::move(formatter.html());
+  });
+
+  // Assert that we did in fact use all our taints
+  assert(targetSpanIt == targetTokenSpans.end());
+
+  return out;
+}
+
+HTML::Tag *HTML::makeTag(Tag &&tag) {
+  pool_.emplace_front(std::move(tag));
+  return &pool_.front();
+}
+
+void HTML::copyTaint(Response const &response, std::vector<std::vector<size_t>> const &alignments,
+                     std::vector<SpanIterator> const &sourceTokenSpans, std::vector<SpanIterator> &targetTokenSpans) {
+  size_t offset = 0;  // Sentence offset in sourceTokenSpans
+
+  // Fill targetTokenSpans based on the alignments we just made up.
+  // NOTE: this should match the exact order of Apply()
+  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
+    targetTokenSpans.push_back(sourceTokenSpans[offset]);  // token_tag for sentence ending gap
+    for (size_t t = 0; t < response.target.numWords(sentenceIdx); ++t) {
+      size_t s = alignments[sentenceIdx][t];
+      assert(s < response.source.numWords(sentenceIdx));
+      targetTokenSpans.push_back(sourceTokenSpans[offset + 1 + s]);  // +1 for prefix gap
+    }
+
+    offset += response.source.numWords(sentenceIdx) + 1;  // +1 for prefix gap
+  }
+
+  assert(offset + 1 == sourceTokenSpans.size());
+  targetTokenSpans.push_back(sourceTokenSpans[offset]);  // token_tag for ending whitespace
+}
+
+// Reports if token `str` is likely to be a continuation of a word. This is used
+// to determine whether we should share the markup, or whether we should see
+// this token as a fresh start. This implementation will treat "hello[world]"
+// as 4 words, assuming its tokenised as something like `h ell o [ wor ld ]`.
+bool HTML::isContinuation(string_view prev, string_view str) {
+  if (options_.continuationDelimiters.empty()) return false;
+  if (prev.empty() || str.empty()) return false;
+  return options_.continuationDelimiters.find(str[0]) == std::string::npos &&
+         options_.continuationDelimiters.find(prev.back()) == std::string::npos;
+}
+
+void HTML::hardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
+  // For each sentence...
+  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
+    alignments.emplace_back();
+
+    // Hard-align: find for each target token the most prevalent source token
+    // Note: only search from 0 to N-1 because token N is end-of-sentence token
+    // that can only align with the end-of-sentence token of the target
+    for (size_t t = 0; t + 1 < response.target.numWords(sentenceIdx); ++t) {
+      size_t maxS = 0;
+      for (size_t s = 1; s + 1 < response.source.numWords(sentenceIdx); ++s) {
+        if (response.alignments[sentenceIdx][t][s] > response.alignments[sentenceIdx][t][maxS]) {
+          maxS = s;
+        }
+      }
+
+      alignments.back().push_back(maxS);
+    }
+
+    // Next, we try to smooth out these selected alignments with a few heuristics
+    for (size_t t = 1; t + 1 < response.target.numWords(sentenceIdx); ++t) {
+      // If this token is a continuation of a previous token, pick the tags from the most
+      // prevalent token for the whole word.
+      if (isContinuation(response.target.word(sentenceIdx, t - 1), response.target.word(sentenceIdx, t))) {
+        // Note: only looking at the previous token since that will already
+        // have this treatment applied to it.
+        size_t currSentenceIdx = alignments.back()[t];
+        size_t prevSentenceIdx = alignments.back()[t - 1];
+        float currScore = response.alignments[sentenceIdx][t][currSentenceIdx];
+        float prevScore = response.alignments[sentenceIdx][t - 1][prevSentenceIdx];
+
+        if (currScore >= prevScore) {
+          // Apply this to all previous tokens in the word
+          for (size_t i = t;; --i) {
+            alignments.back()[i] = currSentenceIdx;
+
+            // Stop if this was the first token or the beginning of the word
+            if (i == 0 ||
+                !isContinuation(response.target.word(sentenceIdx, i - 1), response.target.word(sentenceIdx, i)))
+              break;
+          }
+        } else {
+          alignments.back()[t] = prevSentenceIdx;
+        }
+      }
+    }
+
+    // Always align target end with source end
+    alignments.back().push_back(response.source.numWords(sentenceIdx) - 1);
+  }
+}
+
 }  // namespace marian::bergamot
diff --git a/src/translator/html.h b/src/translator/html.h
index b233fd225..d4cbd40d5 100644
--- a/src/translator/html.h
+++ b/src/translator/html.h
@@ -1,9 +1,12 @@
 #ifndef SRC_BERGAMOT_HTML_H_
 #define SRC_BERGAMOT_HTML_H_
 
+#include <forward_list>
 #include <stdexcept>
 #include <string>
+#include <unordered_set>
 
+#include "annotation.h"
 #include "definitions.h"
 
 namespace marian {
@@ -18,40 +21,84 @@ class BadHTML : public std::runtime_error {
 
 class HTML {
  public:
+  struct Options {
+    // List of elements for which we do not expect a closing tag, or self-closing
+    // elements in XHTML. See also https://developer.mozilla.org/en-US/docs/Glossary/Empty_element
+    // More relevant source of this list:
+    // https://searchfox.org/mozilla-central/rev/7d17fd1fe9f0005a2fb19e5d53da4741b06a98ba/dom/base/FragmentOrElement.cpp#1791
+    std::unordered_set<std::string> voidTags{"area",  "base",  "basefont", "bgsound", "br",    "col",
+                                             "embed", "frame", "hr",       "img",     "input", "keygen",
+                                             "link",  "meta",  "param",    "source",  "track", "wbr"};
+
+    std::unordered_set<std::string> inlineTags{"abbr",   "a", "b",    "em",    "i",    "kbd",    "mark", "math",
+                                               "output", "q", "ruby", "small", "span", "strong", "sub",  "sup",
+                                               "time",   "u", "var",  "wbr",   "ins",  "del",    "img"};
+
+    // List of characters that occur at the start of a token that indicate that
+    // the this token is probably *not* a continuation of a word. Set to empty
+    // to never mark a token as a continuation of the word.
+    // std::string continuationDelimiters = "\n ,.(){}[]";
+    std::string continuationDelimiters;
+
+    // Should we always add spaces to the places where tags used to be? I.e.
+    // `un<u>der</u>line` should become `un der line`?
+    bool substituteInlineTagsWithSpaces = true;
+  };
+
   struct Tag {
     enum NodeType {
       ELEMENT,
       VOID_ELEMENT,
       COMMENT,
       PROCESSING_INSTRUCTION,
+      WHITESPACE,  // negative space
     };
 
-    NodeType type;  // Type of the node
-    std::string name;
-    std::string attributes;
-    std::string data;  // Raw data of an element that just needs to be
-                       // copied as is, e.g. <script> or <style>
-                       // TODO: replace with string_view if input lives that long
+    NodeType type;           // Type of the node
+    std::string name;        // Tag name (if type is ELEMENT or VOID_ELEMENT)
+    std::string attributes;  // Tag attributes (as raw HTML string, including
+                             // entities and prefix whitespace)
+    std::string data;        // Raw data of an element that just needs to be
+                             // copied as is, e.g. <script> or <style>
+    // @TODO: if the original HTML stays in memory, we could replace
+    // `attributes` and `data` with string_views pointing to it.
   };
 
-  typedef std::vector<Tag *> Taint;
+  using Taint = std::vector<Tag *>;
 
   struct Span {
     size_t begin;
     size_t end;
-    Taint tags;  // Note: free pointer! Lifetime of tags is managed by pool_
+    Taint tags;  // Note: free pointers! Lifetime of tags is managed by pool_
     inline size_t size() const { return end - begin; }
   };
 
-  explicit HTML(std::string &&source, bool process_markup);
+  explicit HTML(std::string &&source, bool process_markup) : HTML(std::move(source), process_markup, HTML::Options{}){};
+  explicit HTML(std::string &&source, bool process_markup, Options &&options);
   void restore(Response &response);
 
  private:
+  using SpanIterator = std::vector<HTML::Span>::const_iterator;
+  using AnnotatedText = marian::bergamot::AnnotatedText;
+
+  AnnotatedText restoreSource(AnnotatedText const &in, std::vector<SpanIterator> &sourceTokenSpans);
+  AnnotatedText restoreTarget(AnnotatedText const &in, std::vector<SpanIterator> const &targetTokenSpans);
+  void copyTaint(Response const &response, std::vector<std::vector<size_t>> const &alignments,
+                 std::vector<HTML::SpanIterator> const &sourceTokenSpans,
+                 std::vector<HTML::SpanIterator> &targetTokenSpans);
+  void hardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments);
+  bool isContinuation(string_view prev, string_view str);
+  // Allocates tag in pool_ (which then owns it) and gives a pointer to be used
+  // in Taints. Pointer is valid as long as this HTML instance lives on.
+  Tag *makeTag(Tag &&tag);
+
+  Options options_;
+
   // List of text spans, and which tags are applied to them
   std::vector<Span> spans_;
 
   // a pool of tags that we free when HTML goes out of scope
-  std::vector<std::unique_ptr<Tag>> pool_;
+  std::forward_list<Tag> pool_;
 };
 
 }  // namespace bergamot

From 6a4f409cda8b4c71a4191cbb12cc6465f62a3701 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Mon, 17 Jan 2022 13:44:23 +0000
Subject: [PATCH 333/442] First class pivot translation capability (#236)

Translates a text from source-language to target-language through a
pivot-language. Effectively runs models in series, while having the
following additional benefits compared to when `Service::translate(...)`
would be used repeatedly.

1. Consistency in sentences between source and target. Consistent
creation of the alignment matrix for use in downstream tasks like
tag-translation.

2. Efficient sentence-splitting (does not sentence-split twice, creating
inconsistencies).

3. The `Response` generated can be used as if it were coming through
`translate(...)`, eliminating any need for additional code for clients
in JS or python or C++.

`AsyncService::pivot(...)` is provisioned for C++ multi-threaded setting
and `BlockingService::pivotMultiple(...)` provisioned for blocking
use-case targeted at WebAssembly.

# [BRT]: Test additions, accompanying fixes

For `AsyncService` for a test-case involving of en->es, es->en (same
vocabulary, another one might be more coverage but is too much work).

1. Asserts the Alignment generated after pivoting is a probability
distribution over source tokens given target.

2. Outputs the sentences going from en->en, which should stay consistent
over continuous development to ensure nothing breaks.

3. An accuracy minimum of 70% of token matches from source to target
calibrated on the standard bergamot input text is additionally present,
ensuring that the English tokens at start and end match exactly.

# HTML Pipeline

This PR reworks the HTML translation pipeline to be outside
response-construction via callbacks.
---
 bergamot-translator-tests            |   2 +-
 src/tests/common-impl.cpp            |  93 +++++++++++++++++
 src/tests/common.h                   |   8 ++
 src/translator/CMakeLists.txt        |   1 +
 src/translator/batching_pool.cpp     |  34 +++++--
 src/translator/batching_pool.h       |   3 +-
 src/translator/definitions.h         |   2 +-
 src/translator/response.cpp          | 145 +++++++++++++++++++++++++++
 src/translator/response.h            |   5 +
 src/translator/response_builder.cpp  |   4 -
 src/translator/response_builder.h    |   8 +-
 src/translator/service.cpp           | 129 ++++++++++++++++++++++++
 src/translator/service.h             |  36 ++++++-
 src/translator/text_processor.cpp    |  34 +++++++
 src/translator/text_processor.h      |   2 +
 src/translator/translation_model.cpp |  15 ++-
 src/translator/translation_model.h   |   3 +
 17 files changed, 499 insertions(+), 25 deletions(-)
 create mode 100644 src/translator/response.cpp

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index b46987e96..e49686b7c 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit b46987e96fc27b7e9488fbc36b53c07e1786784c
+Subproject commit e49686b7cabcc55c0a1fe3dae4cfe59ec146d68e
diff --git a/src/tests/common-impl.cpp b/src/tests/common-impl.cpp
index 9fc44c9ad..8fdc74fb6 100644
--- a/src/tests/common-impl.cpp
+++ b/src/tests/common-impl.cpp
@@ -26,6 +26,26 @@ Response Bridge<AsyncService>::translate(AsyncService &service, std::shared_ptr<
   return response;
 }
 
+Response Bridge<BlockingService>::pivot(BlockingService &service, std::shared_ptr<TranslationModel> &sourceToPivot,
+                                        std::shared_ptr<TranslationModel> &pivotToTarget, std::string &&source,
+                                        const ResponseOptions &responseOptions) {
+  std::vector<std::string> sources = {source};
+  return service.pivotMultiple(sourceToPivot, pivotToTarget, std::move(sources), responseOptions).front();
+}
+
+Response Bridge<AsyncService>::pivot(AsyncService &service, std::shared_ptr<TranslationModel> &sourceToPivot,
+                                     std::shared_ptr<TranslationModel> &pivotToTarget, std::string &&source,
+                                     const ResponseOptions &responseOptions) {
+  std::promise<Response> responsePromise;
+  std::future<Response> responseFuture = responsePromise.get_future();
+
+  auto callback = [&responsePromise](Response &&response) { responsePromise.set_value(std::move(response)); };
+  service.pivot(sourceToPivot, pivotToTarget, std::move(source), callback, responseOptions);
+  responseFuture.wait();
+  Response response = responseFuture.get();
+  return response;
+}
+
 template <class Service>
 TestSuite<Service>::TestSuite(Service &service) : service_{service} {}
 
@@ -49,6 +69,8 @@ void TestSuite<Service>::TestSuite::run(const std::string &opModeAsString, std::
     qualityEstimatorScores(models.front());
   } else if (opModeAsString == "test-translation-cache") {
     translationCache(models.front());
+  } else if (opModeAsString == "test-pivot") {
+    pivotTranslate(models);
   } else if (opModeAsString == "test-html-translation") {
     htmlTranslation(models.front());
   } else {
@@ -203,3 +225,74 @@ void TestSuite<Service>::translationCache(Ptr<TranslationModel> model) {
 
   std::cout << firstResponse.target.text;
 }
+
+template <class Service>
+void TestSuite<Service>::pivotTranslate(std::vector<Ptr<TranslationModel>> &models) {
+  // We expect a source -> pivot; pivot -> source model to get source -> source and build this test using accuracy of
+  // matches.
+  ABORT_IF(models.size() != 2, "Forward and backward test needs two models.");
+  ResponseOptions responseOptions;
+  responseOptions.alignment = true;
+  std::string source = readFromStdin();
+  std::promise<Response> responsePromise;
+  std::future<Response> responseFuture = responsePromise.get_future();
+
+  Response response = bridge_.pivot(service_, models.front(), models.back(), std::move(source), responseOptions);
+
+  const float EPS = 1e-5;
+  size_t totalOutcomes = 0;
+  size_t favourableOutcomes = 0;
+
+  for (size_t sentenceId = 0; sentenceId < response.source.numSentences(); sentenceId++) {
+    std::cout << "> " << response.source.sentence(sentenceId) << "\n";
+    std::cout << "< " << response.target.sentence(sentenceId) << "\n\n";
+
+    // Assert what we have is a probability distribution over source-tokens given a target token.
+    for (size_t t = 0; t < response.alignments[sentenceId].size(); t++) {
+      float sum = 0.0f;
+      for (size_t s = 0; s < response.alignments[sentenceId][t].size(); s++) {
+        sum += response.alignments[sentenceId][t][s];
+      }
+
+      std::cerr << fmt::format("Sum @ (target-token = {}, sentence = {}) = {}", t, sentenceId, sum) << std::endl;
+      ABORT_IF((std::abs(sum - 1.0f) > EPS), "Not a probability distribution, something's going wrong");
+    }
+
+    // For each target token, find argmax s, i.e find argmax p(s | t), max p(s | t)
+    for (size_t t = 0; t < response.alignments[sentenceId].size(); t++) {
+      bool valid = false;
+      float maxV = 0.0f;
+      auto argmaxV = std::make_pair(-1, -1);
+      for (size_t s = 0; s < response.alignments[sentenceId][t].size(); s++) {
+        auto v = response.alignments[sentenceId][t][s];
+        if (v > maxV) {
+          maxV = v;
+          argmaxV = std::make_pair(t, s);
+        }
+      }
+
+      auto sourceToken = response.source.word(sentenceId, argmaxV.second);
+      auto targetToken = response.target.word(sentenceId, argmaxV.first);
+      if (sourceToken == targetToken) {
+        favourableOutcomes += 1;
+      }
+
+      std::cerr << sourceToken << " " << targetToken << " " << maxV << std::endl;
+
+      totalOutcomes += 1;
+    }
+
+    // Assert each alignment over target is a valid probability distribution.
+  }
+
+  // Measure accuracy of word match.
+  float accuracy = static_cast<float>(favourableOutcomes) / static_cast<float>(totalOutcomes);
+
+  // This is arbitrary value chosen by @jerinphilip, but should be enough to check if things fail.
+  // This value is calibrated on bergamot input in BRT. All this is supposed to do is let the developers know if
+  // something's largely amiss to the point alignments are not working.
+  ABORT_IF(accuracy < 0.70, "Accuracy {} not enough. Please check if something's off.", accuracy * 100);
+
+  std::cout << response.source.text;
+  std::cout << response.target.text;
+}
diff --git a/src/tests/common.h b/src/tests/common.h
index 1e454858c..f84121648 100644
--- a/src/tests/common.h
+++ b/src/tests/common.h
@@ -41,12 +41,18 @@ template <>
 struct Bridge<BlockingService> : public std::true_type {
   Response translate(BlockingService &service, std::shared_ptr<TranslationModel> &model, std::string &&source,
                      const ResponseOptions &responseOptions);
+  Response pivot(BlockingService &service, std::shared_ptr<TranslationModel> &sourceToPivot,
+                 std::shared_ptr<TranslationModel> &pivotToTarget, std::string &&source,
+                 const ResponseOptions &responseOptions);
 };
 
 template <>
 struct Bridge<AsyncService> : public std::true_type {
   Response translate(AsyncService &service, std::shared_ptr<TranslationModel> &model, std::string &&source,
                      const ResponseOptions &responseOptions);
+  Response pivot(AsyncService &service, std::shared_ptr<TranslationModel> &sourceToPivot,
+                 std::shared_ptr<TranslationModel> &pivotToTarget, std::string &&source,
+                 const ResponseOptions &responseOptions);
 };
 
 template <class Service>
@@ -80,6 +86,8 @@ class TestSuite {
 
   void translationCache(Ptr<TranslationModel> model);
 
+  void pivotTranslate(std::vector<Ptr<TranslationModel>> &models);
+
   void htmlTranslation(Ptr<TranslationModel> model);
 };
 
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index dbead6173..2beb2e925 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -15,6 +15,7 @@ add_library(bergamot-translator STATIC
     annotation.cpp
     service.cpp
     parser.cpp
+    response.cpp
     html.cpp
     xh_scanner.cpp
 )
diff --git a/src/translator/batching_pool.cpp b/src/translator/batching_pool.cpp
index 1033e80cc..f688ec8f4 100644
--- a/src/translator/batching_pool.cpp
+++ b/src/translator/batching_pool.cpp
@@ -8,10 +8,20 @@
 namespace marian {
 namespace bergamot {
 
-BatchingPool::BatchingPool(Ptr<Options> options) {
-  miniBatchWords = options->get<int>("mini-batch-words");
-  bucket_.resize(options->get<int>("max-length-break") + 1);
-  ABORT_IF(bucket_.size() - 1 > miniBatchWords,
+BatchingPool::BatchingPool(Ptr<Options> options)
+    : miniBatchWords_(options->get<int>("mini-batch-words")), maxActiveBucketLength_(0) {
+  size_t maxLengthBreak = options->get<int>("max-length-break");
+  float maxLengthFactor = options->get<float>("max-length-factor", 3.0);
+
+  // For the time being, we add some slack, which only BatchingPool is aware of. Since the TextProcessor still wraps at
+  // first request in, most of the Batches generated will be under max-length break.
+  //
+  // In the unlikely event of a few sentences overflowing, this allows the exceeding words to be put in the slack area.
+  // Very few batches are expected to be generated at a higher length.
+  size_t pivotSlack = maxLengthBreak * maxLengthFactor - maxLengthBreak;
+  bucket_.resize(maxLengthBreak + pivotSlack + 1);
+
+  ABORT_IF(bucket_.size() - 1 > miniBatchWords_,
            "Fatal: max-length-break > mini-batch-words  will lead to sentences "
            "longer than what can fit in a batch.");
 }
@@ -24,11 +34,11 @@ size_t BatchingPool::generateBatch(Batch &batch) {
   batch.clear();
   size_t paddedBatchSize = 0;
 
-  for (size_t length = 0; length < bucket_.size(); length++) {
+  for (size_t length = 0; length <= maxActiveBucketLength_; length++) {
     auto p = bucket_[length].begin();
     while (p != bucket_[length].end()) {
       paddedBatchSize = (batch.size() + 1) * length;
-      if (paddedBatchSize <= miniBatchWords) {
+      if (paddedBatchSize <= miniBatchWords_) {
         auto q = p++;
         batch.add(*q);
         bucket_[length].erase(q);
@@ -49,8 +59,18 @@ size_t BatchingPool::enqueueRequest(Ptr<Request> request) {
     if (!request->cacheHitPrefilled(i)) {
       RequestSentence sentence(i, request);
       size_t bucket_id = sentence.numTokens();
-      assert(bucket_id < bucket_.size());
+
+      // Due to a workaround for pivoting, unless we can discipline the
+      // vocabulary to get stronger static requirements, it is difficult to
+      // rework the rest of the components. Instead, we allow dynamic growth
+      // here. We let std::vector take care of the dynamic growth.
+      // https://en.cppreference.com/w/cpp/container/vector/resize#Complexity
+      if (bucket_id >= bucket_.size()) {
+        bucket_.resize(bucket_id + 1);
+      }
+
       bucket_[bucket_id].insert(sentence);
+      maxActiveBucketLength_ = std::max<size_t>(bucket_id, maxActiveBucketLength_);
 
       toBeFreshlyTranslated += 1;
     }
diff --git a/src/translator/batching_pool.h b/src/translator/batching_pool.h
index 68b2cf0d0..b6950951b 100644
--- a/src/translator/batching_pool.h
+++ b/src/translator/batching_pool.h
@@ -27,9 +27,10 @@ class BatchingPool {
   size_t generateBatch(Batch &batch);
 
  private:
-  size_t miniBatchWords;
+  size_t miniBatchWords_;
   std::vector<std::set<RequestSentence>> bucket_;
   size_t batchNumber_{0};
+  size_t maxActiveBucketLength_;
 };
 
 }  // namespace bergamot
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index 2ac6bf0ef..eb1e67296 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -43,7 +43,7 @@ struct ByteRange {
 };
 
 class Response;
-using CallbackType = std::function<void(Response&&)>;
+using CallbackType = std::function<void(Response &&)>;
 
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/translator/response.cpp b/src/translator/response.cpp
new file mode 100644
index 000000000..8e623a7d6
--- /dev/null
+++ b/src/translator/response.cpp
@@ -0,0 +1,145 @@
+#include "response.h"
+
+#include "annotation.h"
+#include "definitions.h"
+
+namespace marian::bergamot {
+
+// We're marginalizing q out of p(s | q) x p( q | t). However, we have different representations of q on source side to
+// intermediate - p(s_i | q_j) and intermediate to target side - p(q'_j' | t_k).
+//
+// The matrix p(q'_j' | t_k) is rewritten into p(q_j | t_k) by means of spreading the probability in the former over
+// bytes and collecting it at the ranges specified by latter, using a two pointer accumulation strategy.
+Alignment transferThroughCharacters(const std::vector<ByteRange> &sourceSidePivots,
+                                    const std::vector<ByteRange> &targetSidePivots,
+                                    const Alignment &pivotGivenTargets) {
+  // Initialize an empty alignment matrix.
+  Alignment remapped(pivotGivenTargets.size(), std::vector<float>(sourceSidePivots.size(), 0.0f));
+
+  size_t sq, qt;
+  for (sq = 0, qt = 0; sq < sourceSidePivots.size() && qt < targetSidePivots.size();
+       /*each branch inside increments either sq or qt or both, therefore the loop terminates */) {
+    auto &sourceSidePivot = sourceSidePivots[sq];
+    auto &targetSidePivot = targetSidePivots[qt];
+    if (sourceSidePivot.begin == targetSidePivot.begin && sourceSidePivot.end == targetSidePivot.end) {
+      for (size_t t = 0; t < pivotGivenTargets.size(); t++) {
+        remapped[t][sq] += pivotGivenTargets[t][qt];
+      }
+
+      // Perfect match, move pointer from both.
+      sq++, qt++;
+    } else {
+      // Do we have overlap?
+      size_t left = std::max(targetSidePivot.begin, sourceSidePivot.begin);
+      size_t right = std::min(targetSidePivot.end, sourceSidePivot.end);
+
+      assert(left < right);  // there should be overlap.
+
+      size_t charCount = right - left;
+      size_t probSpread = targetSidePivot.size();
+      for (size_t t = 0; t < pivotGivenTargets.size(); t++) {
+        remapped[t][sq] += charCount * pivotGivenTargets[t][qt] / static_cast<float>(probSpread);
+      }
+
+      // Which one is ahead? sq or qt or both end at same point?
+      if (sourceSidePivot.end == targetSidePivot.end) {
+        sq++;
+        qt++;
+      } else if (sourceSidePivot.end > targetSidePivot.end) {
+        qt++;
+      } else {  // sourceSidePivot.end < targetSidePivot.end
+        sq++;
+      }
+    }
+  }
+
+  // The following is left in here for future debugging. Every token in source is expected to have been processed in the
+  // above pipeline. We advance the pivot-token index based on overlap with source-token. @jerinphilip is worried about
+  // EOS not existing when people try weird 4-model things in the future and would like to keep this check here.
+  assert(sq == sourceSidePivots.size());
+
+  while (qt < targetSidePivots.size()) {
+    // There is a case of EOS not being predicted. In this case the two pointer algorithm will fail. The just author
+    // will redistribute the surplus among subjects.
+
+    // assert in DEBUG, that this is only EOS - occuring at the end and with zero-surface.
+    assert(qt == targetSidePivots.size() - 1 && targetSidePivots[qt].size() == 0);
+    for (size_t t = 0; t < pivotGivenTargets.size(); t++) {
+      float gift = pivotGivenTargets[t][qt] / sourceSidePivots.size();
+      for (size_t sq = 0; sq < sourceSidePivots.size(); sq++) {
+        remapped[t][sq] += gift;
+      }
+    }
+
+    qt++;
+  }
+
+#ifdef DEBUG
+  // The following sanity check ensures when DEBUG is enabled that we have successfully transferred all probabily mass
+  // available over pivot tokens given a target token in our original input to the new remapped representation.
+  //
+  // It's been discovered that floating point arithmetic before we get the Alignment matrix can have values such that
+  // the distribution does not sum upto 1.
+  const float EPS = 1e-6;
+  for (size_t t = 0; t < pivotGivenTargets.size(); t++) {
+    float sum = 0.0f, expectedSum = 0.0f;
+    for (size_t qt = 0; qt < targetSidePivots.size(); qt++) {
+      expectedSum += pivotGivenTargets[t][qt];
+    }
+    for (size_t sq = 0; sq < sourceSidePivots.size(); sq++) {
+      sum += remapped[t][sq];
+    }
+    std::cerr << fmt::format("Sum @ token {} = {} to be compared with expected {}.", t, sum, expectedSum) << std::endl;
+    ABORT_IF(std::abs(sum - expectedSum) > EPS, "Haven't accumulated probabilities, re-examine");
+  }
+#endif  // DEBUG
+
+  return remapped;
+}
+
+std::vector<Alignment> remapAlignments(const Response &first, const Response &second) {
+  std::vector<Alignment> alignments;
+  for (size_t sentenceId = 0; sentenceId < first.source.numSentences(); sentenceId++) {
+    const Alignment &sourceGivenPivots = first.alignments[sentenceId];
+    const Alignment &pivotGivenTargets = second.alignments[sentenceId];
+
+    // TODO: Allow range iterators and change algorithm, directly tapping into AnnotatedText
+    // Extracts ByteRanges corresponding to a words constituting a sentence from an annotation.
+    auto extractWordByteRanges = [](const AnnotatedText &annotatedText,
+                                    size_t sentenceId) -> std::vector<marian::bergamot::ByteRange> {
+      size_t N = annotatedText.numWords(sentenceId);
+      std::vector<ByteRange> output;
+
+      for (size_t i = 0; i < N; i++) {
+        output.push_back(annotatedText.wordAsByteRange(sentenceId, i));
+      }
+      return output;
+    };
+
+    auto sourceSidePivots = extractWordByteRanges(first.target, sentenceId);
+    auto targetSidePivots = extractWordByteRanges(second.source, sentenceId);
+
+    // Reintrepret probability p(q'_j' | t_k) as p(q_j | t_k)
+    Alignment remappedPivotGivenTargets =
+        transferThroughCharacters(sourceSidePivots, targetSidePivots, pivotGivenTargets);
+
+    // Marginalize out q_j.
+    // p(s_i | t_k) = \sum_{j} p(s_i | q_j) x p(q_j | t_k)
+    size_t sourceTokenCount = first.source.numWords(sentenceId);
+    size_t targetTokenCount = second.target.numWords(sentenceId);
+    Alignment output(targetTokenCount, std::vector<float>(sourceTokenCount, 0.0f));
+    for (size_t idt = 0; idt < targetTokenCount; idt++) {
+      for (size_t idq = 0; idq < sourceSidePivots.size(); idq++) {
+        for (size_t ids = 0; ids < sourceTokenCount; ids++) {
+          // Matrices are of form p(s | t) = P[t][s], hence idq appears on the extremes.
+          output[idt][ids] += sourceGivenPivots[idq][ids] * remappedPivotGivenTargets[idt][idq];
+        }
+      }
+    }
+
+    alignments.push_back(output);
+  }
+  return alignments;
+}
+
+}  // namespace marian::bergamot
diff --git a/src/translator/response.h b/src/translator/response.h
index 49ac80392..74463eda2 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -14,6 +14,8 @@
 namespace marian {
 namespace bergamot {
 
+typedef std::vector<std::vector<float>> Alignment;
+
 /// Response holds AnnotatedText(s) of source-text and translated text,
 /// alignment information between source and target sub-words and sentences.
 ///
@@ -72,6 +74,9 @@ struct Response {
 
   const std::string &getTranslatedText() const { return target.text; }
 };
+
+std::vector<Alignment> remapAlignments(const Response &first, const Response &second);
+
 }  // namespace bergamot
 }  // namespace marian
 
diff --git a/src/translator/response_builder.cpp b/src/translator/response_builder.cpp
index f1bb773e0..f752bf1a5 100644
--- a/src/translator/response_builder.cpp
+++ b/src/translator/response_builder.cpp
@@ -16,10 +16,6 @@ void ResponseBuilder::buildAlignments(Histories &histories, Response &response)
 
     Result result = onebest[0];  // Expecting only one result;
     Words words = std::get<0>(result);
-    // Alignments
-    // TODO(jerinphilip): The following double conversion might not be
-    // necessary. Hard alignment can directly be exported, but this would
-    // mean WASM bindings for a structure deep within marian source.
     auto hyp = std::get<1>(result);
     auto softAlignment = hyp->tracebackAlignment();
     response.alignments.push_back(std::move(softAlignment));
diff --git a/src/translator/response_builder.h b/src/translator/response_builder.h
index 345951a0e..f6eaa35d3 100644
--- a/src/translator/response_builder.h
+++ b/src/translator/response_builder.h
@@ -31,13 +31,12 @@ class ResponseBuilder {
   /// @param [in] qualityEstimator: the QualityEstimator model that can be used
   /// to provide translation quality probability.
   ResponseBuilder(ResponseOptions responseOptions, AnnotatedText &&source, const Vocabs &vocabs,
-                  std::function<void(Response &&)> callback, const QualityEstimator &qualityEstimator, HTML &&html)
+                  std::function<void(Response &&)> callback, const QualityEstimator &qualityEstimator)
       : responseOptions_(responseOptions),
         source_(std::move(source)),
         vocabs_(vocabs),
         callback_(std::move(callback)),
-        qualityEstimator_(qualityEstimator),
-        html_(std::move(html)) {}
+        qualityEstimator_(qualityEstimator) {}
 
   /// Constructs and sets the promise of a Response object from obtained
   /// histories after translating.
@@ -64,7 +63,6 @@ class ResponseBuilder {
     if (responseOptions_.alignment || responseOptions_.HTML) {
       buildAlignments(histories, response);
     }
-    html_.restore(response);
 
     callback_(std::move(response));
   }
@@ -97,8 +95,6 @@ class ResponseBuilder {
   AnnotatedText source_;
 
   const QualityEstimator &qualityEstimator_;
-
-  HTML html_;
 };
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 8acbc97de..a1e6a0ebf 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -10,6 +10,28 @@
 namespace marian {
 namespace bergamot {
 
+namespace {
+
+// Combines two responses with first.target == second.source mapping alignments etc accordingly.
+// There are several constraints which are matched by only the pivoting workflow in <>Service source, therefore this
+// function is not for external use and in a hidden namespace.
+Response combine(Response &&first, Response &&second) {
+  Response combined;
+
+  // Compute alignment first using internal matrices and mappings.
+  if (first.alignments.size()) {
+    combined.alignments = remapAlignments(first, second);
+  }
+
+  combined.source = std::move(first.source);
+  combined.target = std::move(second.target);
+  combined.qualityScores = std::move(second.qualityScores);
+
+  return combined;
+}
+
+}  // namespace
+
 BlockingService::BlockingService(const BlockingService::Config &config)
     : config_(config),
       requestId_(0),
@@ -20,6 +42,21 @@ BlockingService::BlockingService(const BlockingService::Config &config)
 std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<TranslationModel> translationModel,
                                                          std::vector<std::string> &&sources,
                                                          const ResponseOptions &responseOptions) {
+  std::vector<HTML> htmls;
+  for (auto &&source : sources) {
+    htmls.emplace_back(std::move(source), responseOptions.HTML);
+  }
+  std::vector<Response> responses = translateMultipleRaw(translationModel, std::move(sources), responseOptions);
+  for (size_t i = 0; i < responses.size(); i++) {
+    htmls[i].restore(responses[i]);
+  }
+
+  return responses;
+}
+
+std::vector<Response> BlockingService::translateMultipleRaw(std::shared_ptr<TranslationModel> translationModel,
+                                                            std::vector<std::string> &&sources,
+                                                            const ResponseOptions &responseOptions) {
   std::vector<Response> responses;
   responses.resize(sources.size());
 
@@ -40,6 +77,47 @@ std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<Transla
   return responses;
 }
 
+std::vector<Response> BlockingService::pivotMultiple(std::shared_ptr<TranslationModel> first,
+                                                     std::shared_ptr<TranslationModel> second,
+                                                     std::vector<std::string> &&sources,
+                                                     const ResponseOptions &responseOptions) {
+  // Translate source to pivots. This is same as calling translateMultiple.
+  std::vector<Response> sourcesToPivots;
+  sourcesToPivots = translateMultipleRaw(first, std::move(sources), responseOptions);
+
+  // Translate pivots to targets, after we have outputs at pivot from first round. We cannot use translateMultiple here
+  // because need consistency at pivot on both sides.
+  std::vector<Response> pivotsToTargets;
+  pivotsToTargets.resize(sourcesToPivots.size());
+
+  for (size_t i = 0; i < sourcesToPivots.size(); i++) {
+    AnnotatedText intermediate =
+        sourcesToPivots[i].target;  // We cannot eliminate this copy, as we need two versions of intermediate. Holding
+                                    // it in allows further use in makePivotRequest
+    auto callback = [i, &pivotsToTargets](Response &&response) { pivotsToTargets[i] = std::move(response); };  //
+
+    TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
+    Ptr<Request> request =
+        second->makePivotRequest(requestId_++, std::move(intermediate), callback, responseOptions, cache);
+    batchingPool_.enqueueRequest(second, request);
+  }
+
+  Batch batch;
+  Ptr<TranslationModel> model{nullptr};
+  while (batchingPool_.generateBatch(model, batch)) {
+    model->translateBatch(/*deviceId=*/0, batch);
+  }
+
+  // Combine both sides. They're associated by indices.
+  std::vector<Response> finalResponses;
+  for (size_t i = 0; i < sourcesToPivots.size(); i++) {
+    Response finalResponse = combine(std::move(sourcesToPivots[i]), std::move(pivotsToTargets[i]));
+    finalResponses.push_back(std::move(finalResponse));
+  }
+
+  return finalResponses;
+}
+
 AsyncService::AsyncService(const AsyncService::Config &config)
     : requestId_(0),
       config_(config),
@@ -69,9 +147,60 @@ AsyncService::~AsyncService() {
   }
 }
 
+void AsyncService::pivot(std::shared_ptr<TranslationModel> first, std::shared_ptr<TranslationModel> second,
+                         std::string &&source, CallbackType clientCallback, const ResponseOptions &responseOptions) {
+  Ptr<HTML> html = std::make_shared<HTML>(std::move(source), responseOptions.HTML);
+  // This is callback chaining or CPS due to async.
+
+  // We create a callback which feeds the result of first into a second translation (internalCallback), which is
+  // supplied with a callback (joiningCallback) which merges both results and creates our final response.
+  //
+
+  auto internalCallback = [this, clientCallback, second, responseOptions, html](Response &&sourceToPivot) {
+    // We cannot eliminate the following copy, as we need two versions of intermediate. Holding
+    // it in a copy allows moving the response into the lambda below.
+
+    AnnotatedText intermediate = sourceToPivot.target;
+
+    // https://stackoverflow.com/a/65606554/4565794
+    // Move semantics only work on mutable lambdas, and can only be done once. It's only once in our case, so issok.
+    auto joiningCallback = [this, sourceToPivot = std::move(sourceToPivot), clientCallback,
+                            html](Response &&pivotToTarget) mutable {
+      // We have both Responses at this callback, sourceToPivot is moved in, second half will be available when
+      // complete.
+      Response finalResponse = combine(std::move(sourceToPivot), std::move(pivotToTarget));
+
+      // Sentences should be consistent now, give way to client.
+      html->restore(finalResponse);
+      clientCallback(std::move(finalResponse));
+    };
+
+    // Second call.
+    TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
+    Ptr<Request> request =
+        second->makePivotRequest(requestId_++, std::move(intermediate), joiningCallback, responseOptions, cache);
+    safeBatchingPool_.enqueueRequest(second, request);
+  };
+
+  // First call.
+  translateRaw(first, std::move(source), internalCallback, responseOptions);
+}
+
 void AsyncService::translate(std::shared_ptr<TranslationModel> translationModel, std::string &&source,
                              CallbackType callback, const ResponseOptions &responseOptions) {
   // Producer thread, a call to this function adds new work items. If batches are available, notifies workers waiting.
+  Ptr<HTML> html = std::make_shared<HTML>(std::move(source), responseOptions.HTML);
+  auto internalCallback = [html, callback](Response &&response) {
+    html->restore(response);
+    callback(std::move(response));
+  };
+
+  translateRaw(translationModel, std::move(source), internalCallback, responseOptions);
+}
+
+void AsyncService::translateRaw(std::shared_ptr<TranslationModel> translationModel, std::string &&source,
+                                CallbackType callback, const ResponseOptions &responseOptions) {
+  // Producer thread, a call to this function adds new work items. If batches are available, notifies workers waiting.
   TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
   Ptr<Request> request =
       translationModel->makeRequest(requestId_++, std::move(source), callback, responseOptions, cache);
diff --git a/src/translator/service.h b/src/translator/service.h
index e01156ae5..4798d0bd6 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -62,6 +62,22 @@ class BlockingService {
   std::vector<Response> translateMultiple(std::shared_ptr<TranslationModel> translationModel,
                                           std::vector<std::string> &&source, const ResponseOptions &responseOptions);
 
+  std::vector<Response> translateMultipleRaw(std::shared_ptr<TranslationModel> translationModel,
+                                             std::vector<std::string> &&source, const ResponseOptions &responseOptions);
+  /// With the supplied two translation models, translate using first and then the second generating a response as if it
+  /// were translated from first's source language to second's target langauge. Requires first's target to be second's
+  /// source to work correctly - effectively implementing pivoting translation via an intermediate language.
+  ///
+  /// @param[in] first: TranslationModel capable of translating from source language to pivot language.
+  /// @param[in] second: TranslationModel capable of translating between pivot and target language.
+  /// @param[move] sources: The input source texts to be translated.
+  /// @param[in] options: Options indicating whether or not to include optional members in response and pass additional
+  /// configurations. See ResponseOptions.
+  ///
+  /// @returns responses corresponding to the source-text which can be used as if they were translated with
+  /// translateMultiple.
+  std::vector<Response> pivotMultiple(std::shared_ptr<TranslationModel> first, std::shared_ptr<TranslationModel> second,
+                                      std::vector<std::string> &&sources, const ResponseOptions &responseOptions);
   TranslationCache::Stats cacheStats() { return cache_.stats(); }
 
  private:
@@ -118,8 +134,8 @@ class AsyncService {
   }
 
   /// With the supplied TranslationModel, translate an input. A Response is constructed with optional items set/unset
-  /// indicated via ResponseOptions. Upon completion translation of the input, the client supplied callback is triggered
-  /// with the constructed Response. Concurrent-calls to this function are safe.
+  /// indicated via ResponseOptions. Upon completion translation of the input, the client supplied callback is
+  /// triggered with the constructed Response. Concurrent-calls to this function are safe.
   ///
   /// @param [in] translationModel: TranslationModel to use for the request.
   /// @param [in] source: rvalue reference of the string to be translated. This is available as-is to the client later
@@ -130,12 +146,28 @@ class AsyncService {
   void translate(std::shared_ptr<TranslationModel> translationModel, std::string &&source, CallbackType callback,
                  const ResponseOptions &options = ResponseOptions());
 
+  /// With the supplied two translation models, translate using first and then the second generating a response as if it
+  /// were translated from first's source language to second's target langauge. Requires first's target to be second's
+  /// source to work correctly - effectively implementing pivoting translation via an intermediate language.
+  ///
+  /// @param[in] first: TranslationModel capable of translating from source language to pivot language.
+  /// @param[in] second: TranslationModel capable of translating between pivot and target language.
+  /// @param[move] source: The source text to be translated
+  /// @param[in] clientCallback: The callback to be called with the constructed Response. Expects the callback to
+  /// consume the Response.
+  /// @param[in] options: Options indicating whether or not to include optional members in response and pass additional
+  /// configurations. See ResponseOptions.
+  void pivot(std::shared_ptr<TranslationModel> first, std::shared_ptr<TranslationModel> second, std::string &&source,
+             CallbackType clientCallback, const ResponseOptions &options = ResponseOptions());
+
   /// Thread joins and proper shutdown are required to be handled explicitly.
   ~AsyncService();
 
   TranslationCache::Stats cacheStats() { return cache_.stats(); }
 
  private:
+  void translateRaw(std::shared_ptr<TranslationModel> translationModel, std::string &&source, CallbackType callback,
+                    const ResponseOptions &options = ResponseOptions());
   AsyncService::Config config_;
 
   std::vector<std::thread> workers_;
diff --git a/src/translator/text_processor.cpp b/src/translator/text_processor.cpp
index b747f79a5..e687f3ca3 100644
--- a/src/translator/text_processor.cpp
+++ b/src/translator/text_processor.cpp
@@ -138,5 +138,39 @@ void TextProcessor::wrap(Segment &segment, std::vector<string_view> &wordRanges,
   }
 }
 
+void TextProcessor::processFromAnnotation(AnnotatedText &source, Segments &segments) const {
+  std::string copySource = source.text;
+  AnnotatedText replacement(std::move(copySource));
+
+  for (size_t s = 0; s < source.numSentences(); s++) {
+    // This is our sentenceStream
+    ByteRange sentenceByteRange = source.sentenceAsByteRange(s);
+
+    // Fool tokenization using ByteRanges into looking at replacement. They're same, so okay.
+    marian::string_view sentence{&replacement.text[sentenceByteRange.begin], sentenceByteRange.size()};
+
+    std::vector<string_view> wordRanges;
+    Segment segment = tokenize(sentence, wordRanges);
+
+    // Manually add EoS
+    Word sourceEosId = vocabs_.sources().front()->getEosId();
+    segment.push_back(sourceEosId);
+
+    if (!wordRanges.empty()) {
+      string_view &last = wordRanges.back();  // this is a possible segfault if wordRanges is empty. So guard.
+      const char *end = last.data() + last.size();
+      wordRanges.emplace_back(end, 0);
+    } else {
+      const char *end = sentence.data() + sentence.size();
+      wordRanges.emplace_back(end, 0);
+    }
+
+    segments.push_back(std::move(segment));
+    replacement.recordExistingSentence(wordRanges.begin(), wordRanges.end(), wordRanges.begin()->data());
+  }
+
+  source = replacement;
+}
+
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/translator/text_processor.h b/src/translator/text_processor.h
index a6c918c0e..1bde49391 100644
--- a/src/translator/text_processor.h
+++ b/src/translator/text_processor.h
@@ -49,6 +49,8 @@ class TextProcessor {
 
   void process(std::string &&blob, AnnotatedText &source, Segments &segments) const;
 
+  void processFromAnnotation(AnnotatedText &source, Segments &segments) const;
+
  private:
   void parseCommonOptions(Ptr<Options> options);
 
diff --git a/src/translator/translation_model.cpp b/src/translator/translation_model.cpp
index 207f602d1..f2ad39b3e 100644
--- a/src/translator/translation_model.cpp
+++ b/src/translator/translation_model.cpp
@@ -91,16 +91,25 @@ Ptr<Request> TranslationModel::makeRequest(size_t requestId, std::string &&sourc
   Segments segments;
   AnnotatedText annotatedSource;
 
-  HTML html(std::move(source), responseOptions.HTML);
   textProcessor_.process(std::move(source), annotatedSource, segments);
-  ResponseBuilder responseBuilder(responseOptions, std::move(annotatedSource), vocabs_, callback, *qualityEstimator_,
-                                  std::move(html));
+  ResponseBuilder responseBuilder(responseOptions, std::move(annotatedSource), vocabs_, callback, *qualityEstimator_);
 
   Ptr<Request> request =
       New<Request>(requestId, /*model=*/*this, std::move(segments), std::move(responseBuilder), cache);
   return request;
 }
 
+Ptr<Request> TranslationModel::makePivotRequest(size_t requestId, AnnotatedText &&previousTarget, CallbackType callback,
+                                                const ResponseOptions &responseOptions, TranslationCache *cache) {
+  Segments segments;
+
+  textProcessor_.processFromAnnotation(previousTarget, segments);
+  ResponseBuilder responseBuilder(responseOptions, std::move(previousTarget), vocabs_, callback, *qualityEstimator_);
+
+  Ptr<Request> request = New<Request>(requestId, *this, std::move(segments), std::move(responseBuilder), cache);
+  return request;
+}
+
 Ptr<marian::data::CorpusBatch> TranslationModel::convertToMarianBatch(Batch &batch) {
   std::vector<data::SentenceTuple> batchVector;
   auto &sentences = batch.sentences();
diff --git a/src/translator/translation_model.h b/src/translator/translation_model.h
index 3e79fdb38..8519ad41c 100644
--- a/src/translator/translation_model.h
+++ b/src/translator/translation_model.h
@@ -69,6 +69,9 @@ class TranslationModel {
   Ptr<Request> makeRequest(size_t requestId, std::string&& source, CallbackType callback,
                            const ResponseOptions& responseOptions, TranslationCache* cache);
 
+  Ptr<Request> makePivotRequest(size_t requestId, AnnotatedText&& previousTarget, CallbackType callback,
+                                const ResponseOptions& responseOptions, TranslationCache* cache);
+
   /// Relays a request to the batching-pool specific to this translation model.
   /// @param [in] request: Request constructed through makeRequest
   size_t enqueueRequest(Ptr<Request> request) { return batchingPool_.enqueueRequest(request); };

From acbc46d816d32abe727d40b6450e229d7a4d4dfc Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Wed, 19 Jan 2022 09:22:46 +0000
Subject: [PATCH 334/442] Accept XHTML-style self-closing void tags (#305)

Allow the self-closing `/>` end for void tags. For non-void tags these
were already "allowed" due to how the HTML parser works, but for
elements where they actually occur, like `<br/>`, they caused a parse
error. Support for them was not implemented since we only expect valid
HTML5, e.g. the output of Firefox' Element.innerHTML.

Use case: TranslateLocally uses Qt's HTML representation of rich text.
That HTML uses self-closing tags like `<meta .../>` and `<br/>`.
Implementing a string replace operation that would only match these
elements without parsing HTML is tricky. Fixing it in
bergamot-translator is not.

Implementation: Currently `<img>` is marked as a void tag (an element
which cannot have children or text, and therefore treated differently.
Since void tags normally have no close tag, they are treated as
immediately closed. The HTML parser we use reads `<img/>` as
`<img></img>` which thus causes a problem since now we close an element
that was never open, to begin with.

This fix ignores the `TT_TAG_END` token from the parser when the tag
name is that of a void tag.
---
 src/tests/units/html_tests.cpp | 6 ++++++
 src/translator/html.cpp        | 6 ++++--
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
index d1a604d26..8b3ac0f24 100644
--- a/src/tests/units/html_tests.cpp
+++ b/src/tests/units/html_tests.cpp
@@ -482,6 +482,12 @@ TEST_CASE("Test self-closing tag (HTML5)") {
   CHECK(input == "hello  world and other creatures");  // Note double space between "hello" and "world"
 }
 
+TEST_CASE("Test self-closing tag (XHTML)") {
+  std::string input("<p>hello<img/>world</p>");
+  HTML html(std::move(input), true);
+  CHECK(input == "hello world");  // <img/> introduced space
+}
+
 TEST_CASE("Test empty void tag at end of input") {
   std::string input("hello <br>");
   HTML html(std::move(input), true);
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index e2ea72223..20614f578 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -350,8 +350,10 @@ HTML::HTML(std::string &&source, bool process_markup, Options &&options) : optio
       } break;
 
       case markup::Scanner::TT_TAG_END:
-        // Note: self-closing tags emit TT_TAG_END immediately after TT_TAG_START
-        // but since we're parsing HTML5, a sole <img> will never emit a TT_TAG_END
+        // If this is the closing bit of a void tag, i.e. triggered by the "/>"
+        // bit of "<img/>", then completely ignore it.
+        if (contains(options_.voidTags, std::string(scanner.tag()))) break;
+
         if (stack.empty()) throw BadHTML(format("Encountered more closing tags ({}) than opening tags", scanner.tag()));
 
         if (stack.back()->name != scanner.tag())

From 7099b9e9ad4b525f32f255a3aaefcc26a49e4709 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Wed, 19 Jan 2022 16:36:48 +0000
Subject: [PATCH 335/442] Streamline memory-bundle loads (#307)

Provides an additional constructor which takes care of the bundle
loading inside the boundary of the source here, when a configuration
file is supplied from a client like translateLocally or python bindings.
Once the config file is read, we have access to the information required
to construct the MemoryBundle.

 - The command-line application supplied from here, app/bergamot is
   configured to use the fast-load path now.
 - Changes to binary-loading additionally revealed a bug in the
   example-run script used in docs and tied to CI and the fix is
   included.
 - Shortlist is made optional in the memory bundle, making changes to
   getModelMemoryFromConfig.

Fixes #304.
Fixes #306.
See also: XapaJIaMnu/translateLocally#82.
---
 app/bergamot.cpp                     |  3 +--
 examples/run-native.sh               |  2 +-
 src/translator/byte_array_util.cpp   | 10 ++++----
 src/translator/service.h             |  5 ++--
 src/translator/translation_model.cpp | 35 +++++++++++++++-------------
 src/translator/translation_model.h   |  6 ++++-
 6 files changed, 34 insertions(+), 27 deletions(-)

diff --git a/app/bergamot.cpp b/app/bergamot.cpp
index 5629f9110..195e167b1 100644
--- a/app/bergamot.cpp
+++ b/app/bergamot.cpp
@@ -16,8 +16,7 @@ int main(int argc, char *argv[]) {
   // Construct a model.
   auto options = parseOptionsFromFilePath(config.modelConfigPaths.front());
 
-  MemoryBundle memoryBundle;
-  std::shared_ptr<TranslationModel> model = service.createCompatibleModel(options, std::move(memoryBundle));
+  std::shared_ptr<TranslationModel> model = service.createCompatibleModel(options);
 
   ResponseOptions responseOptions;
   std::string input = readFromStdin();
diff --git a/examples/run-native.sh b/examples/run-native.sh
index 81d86252e..b02968a23 100644
--- a/examples/run-native.sh
+++ b/examples/run-native.sh
@@ -9,7 +9,7 @@ wget --quiet --continue --directory models/ \
 # Patch the config-files generated from marian for use in bergamot.
 python3 bergamot-translator-tests/tools/patch-marian-for-bergamot.py \
     --config-path models/ende.student.tiny11/config.intgemm8bitalpha.yml \
-    --ssplit-prefix-file 3rd-party/ssplit-cpp/split-cpp/nonbreaking_prefixes/nonbreaking_prefix.en
+    --ssplit-prefix-file $(realpath 3rd_party/ssplit-cpp/nonbreaking_prefixes/nonbreaking_prefix.en)
 
 # Patched config file will be available with .bergamot.yml suffix.
 CONFIG=models/ende.student.tiny11/config.intgemm8bitalpha.yml.bergamot.yml
diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index 83d06acb9..d0fddee8b 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -101,10 +101,12 @@ AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options) {
 
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options) {
   auto shortlist = options->get<std::vector<std::string>>("shortlist");
-  ABORT_IF(shortlist.empty(), "No path to shortlist file is given.");
-  ABORT_IF(!marian::data::isBinaryShortlist(shortlist[0]),
-           "Loading non-binary shortlist file into memory is not supported");
-  return loadFileToMemory(shortlist[0], 64);
+  if (!shortlist.empty()) {
+    ABORT_IF(!marian::data::isBinaryShortlist(shortlist[0]),
+             "Loading non-binary shortlist file into memory is not supported");
+    return loadFileToMemory(shortlist[0], 64);
+  }
+  return AlignedMemory();
 }
 
 void getVocabsMemoryFromConfig(marian::Ptr<marian::Options> options,
diff --git a/src/translator/service.h b/src/translator/service.h
index 4798d0bd6..9a54a7a11 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -127,10 +127,9 @@ class AsyncService {
 
   /// Create a TranslationModel compatible with this instance of Service. Internally assigns how many replicas of
   /// backend needed based on worker threads set. See TranslationModel for documentation on other params.
-  template <class ConfigType>
-  Ptr<TranslationModel> createCompatibleModel(const ConfigType &config, MemoryBundle &&memory = MemoryBundle{}) {
+  Ptr<TranslationModel> createCompatibleModel(const TranslationModel::Config &config) {
     // @TODO: Remove this remove this dependency/coupling.
-    return New<TranslationModel>(config, std::move(memory), /*replicas=*/config_.numWorkers);
+    return New<TranslationModel>(config, /*replicas=*/config_.numWorkers);
   }
 
   /// With the supplied TranslationModel, translate an input. A Response is constructed with optional items set/unset
diff --git a/src/translator/translation_model.cpp b/src/translator/translation_model.cpp
index f2ad39b3e..09c935beb 100644
--- a/src/translator/translation_model.cpp
+++ b/src/translator/translation_model.cpp
@@ -27,22 +27,25 @@ TranslationModel::TranslationModel(const Config &options, MemoryBundle &&memory
   ABORT_IF(replicas == 0, "At least one replica needs to be created.");
   backend_.resize(replicas);
 
-  if (options_->hasAndNotEmpty("shortlist")) {
-    int srcIdx = 0, trgIdx = 1;
-    bool shared_vcb =
-        vocabs_.sources().front() ==
-        vocabs_.target();  // vocabs_->sources().front() is invoked as we currently only support one source vocab
-    if (memory_.shortlist.size() > 0 && memory_.shortlist.begin() != nullptr) {
-      bool check = options_->get<bool>("check-bytearray", false);
-      shortlistGenerator_ = New<data::BinaryShortlistGenerator>(memory_.shortlist.begin(), memory_.shortlist.size(),
-                                                                vocabs_.sources().front(), vocabs_.target(), srcIdx,
-                                                                trgIdx, shared_vcb, check);
-    } else {
-      // Changed to BinaryShortlistGenerator to enable loading binary shortlist file
-      // This class also supports text shortlist file
-      shortlistGenerator_ = New<data::BinaryShortlistGenerator>(options_, vocabs_.sources().front(), vocabs_.target(),
-                                                                srcIdx, trgIdx, shared_vcb);
-    }
+  // Try to load shortlist from memory-bundle. If not available, try to load from options_;
+
+  int srcIdx = 0, trgIdx = 1;
+  // vocabs_->sources().front() is invoked as we currently only support one source vocab
+  bool shared_vcb = (vocabs_.sources().front() == vocabs_.target());
+
+  if (memory_.shortlist.size() > 0 && memory_.shortlist.begin() != nullptr) {
+    bool check = options_->get<bool>("check-bytearray", false);
+    shortlistGenerator_ = New<data::BinaryShortlistGenerator>(memory_.shortlist.begin(), memory_.shortlist.size(),
+                                                              vocabs_.sources().front(), vocabs_.target(), srcIdx,
+                                                              trgIdx, shared_vcb, check);
+  } else if (options_->hasAndNotEmpty("shortlist")) {
+    // Changed to BinaryShortlistGenerator to enable loading binary shortlist file
+    // This class also supports text shortlist file
+    shortlistGenerator_ = New<data::BinaryShortlistGenerator>(options_, vocabs_.sources().front(), vocabs_.target(),
+                                                              srcIdx, trgIdx, shared_vcb);
+  } else {
+    // In this case, the loadpath does not load shortlist.
+    shortlistGenerator_ = nullptr;
   }
 }
 
diff --git a/src/translator/translation_model.h b/src/translator/translation_model.h
index 8519ad41c..eac7b5aaa 100644
--- a/src/translator/translation_model.h
+++ b/src/translator/translation_model.h
@@ -6,6 +6,7 @@
 
 #include "batch.h"
 #include "batching_pool.h"
+#include "byte_array_util.h"
 #include "cache.h"
 #include "common/utils.h"
 #include "data/shortlist.h"
@@ -56,7 +57,10 @@ class TranslationModel {
   /// @param [in] options: Marian options object.
   /// @param [in] memory: MemoryBundle object holding memory buffers containing parameters to build MarianBackend,
   /// ShortlistGenerator, Vocabs and SentenceSplitter.
-  TranslationModel(const Config& options, MemoryBundle&& memory = MemoryBundle{}, size_t replicas = 1);
+  TranslationModel(const Config& options, MemoryBundle&& memory, size_t replicas = 1);
+
+  TranslationModel(const Config& options, size_t replicas = 1)
+      : TranslationModel(options, getMemoryBundleFromConfig(options), replicas) {}
 
   /// Make a Request to be translated by this TranslationModel instance.
   /// @param [in] requestId: Unique identifier associated with this request, available from Service.

From aef76c03a4754ba2863dbfaa4698c9b44791fc43 Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Fri, 21 Jan 2022 13:14:57 +0000
Subject: [PATCH 336/442] Add API to trigger fast shutdown of AsyncService
 (#297)

Add a way to AsyncService to shut down without finishing the full queue
through `AsyncService::clear()`. The default behaviour is that
`AsyncService::~AsyncService()` will wait for any pending translation
requests to finish.

One can call `AsyncService::clear()` before the calls to the destructor
to ensure there is no work for the service to finish before the workers
can stop and join. Marian batches that are already in progress will not
stop. We are not trying to cause interrupts in threads or something that
complex. However, these single batches often do not take that long to
complete.

Changes:

 - Add clear() to AsyncService
 - Add clear() to BatchingPool
 - Documentation

See also:  XapaJIaMnu/translateLocally#80
---
 src/translator/aggregate_batching_pool.cpp  | 2 ++
 src/translator/aggregate_batching_pool.h    | 4 ++++
 src/translator/batching_pool.cpp            | 6 ++++++
 src/translator/batching_pool.h              | 3 +++
 src/translator/service.cpp                  | 3 +++
 src/translator/service.h                    | 4 ++++
 src/translator/threadsafe_batching_pool.cpp | 7 +++++++
 src/translator/threadsafe_batching_pool.h   | 6 ++++++
 8 files changed, 35 insertions(+)

diff --git a/src/translator/aggregate_batching_pool.cpp b/src/translator/aggregate_batching_pool.cpp
index 60f5fcd2e..5f405110a 100644
--- a/src/translator/aggregate_batching_pool.cpp
+++ b/src/translator/aggregate_batching_pool.cpp
@@ -30,5 +30,7 @@ size_t AggregateBatchingPool::generateBatch(Ptr<TranslationModel>& model, Batch&
   return /*numSentences=*/0;
 }
 
+void AggregateBatchingPool::clear() { aggregateQueue_.clear(); }
+
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/translator/aggregate_batching_pool.h b/src/translator/aggregate_batching_pool.h
index 5b5d4b17a..6775591e0 100644
--- a/src/translator/aggregate_batching_pool.h
+++ b/src/translator/aggregate_batching_pool.h
@@ -58,6 +58,10 @@ class AggregateBatchingPool {
   /// @returns Number of sentences in the generated batch.
   size_t generateBatch(Ptr<TranslationModel>& model, Batch& batch);
 
+  /// Clear the aggregate queue. Does not clear the underlying model/request pairs but the next call
+  /// to `generateBatch()` will return 0. (Unless `enqueueRequest()` was called in the mean time.)
+  void clear();
+
  private:
   std::unordered_set<std::shared_ptr<TranslationModel>, HashPtr<TranslationModel>> aggregateQueue_;
 };
diff --git a/src/translator/batching_pool.cpp b/src/translator/batching_pool.cpp
index f688ec8f4..61dd1920e 100644
--- a/src/translator/batching_pool.cpp
+++ b/src/translator/batching_pool.cpp
@@ -79,5 +79,11 @@ size_t BatchingPool::enqueueRequest(Ptr<Request> request) {
   return toBeFreshlyTranslated;
 }
 
+void BatchingPool::clear() {
+  for (size_t length = 0; length < bucket_.size(); length++) {
+    bucket_[length].clear();
+  }
+}
+
 }  // namespace bergamot
 }  // namespace marian
diff --git a/src/translator/batching_pool.h b/src/translator/batching_pool.h
index b6950951b..58cd2ca8b 100644
--- a/src/translator/batching_pool.h
+++ b/src/translator/batching_pool.h
@@ -26,6 +26,9 @@ class BatchingPool {
   // requests optimizing for both padding and priority.
   size_t generateBatch(Batch &batch);
 
+  // Removes any pending requests from the pool.
+  void clear();
+
  private:
   size_t miniBatchWords_;
   std::vector<std::set<RequestSentence>> bucket_;
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index a1e6a0ebf..380260529 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -139,12 +139,15 @@ AsyncService::AsyncService(const AsyncService::Config &config)
   }
 }
 
+void AsyncService::clear() { safeBatchingPool_.clear(); }
+
 AsyncService::~AsyncService() {
   safeBatchingPool_.shutdown();
   for (std::thread &worker : workers_) {
     assert(worker.joinable());
     worker.join();
   }
+  workers_.clear();
 }
 
 void AsyncService::pivot(std::shared_ptr<TranslationModel> first, std::shared_ptr<TranslationModel> second,
diff --git a/src/translator/service.h b/src/translator/service.h
index 9a54a7a11..87054e2cc 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -159,7 +159,11 @@ class AsyncService {
   void pivot(std::shared_ptr<TranslationModel> first, std::shared_ptr<TranslationModel> second, std::string &&source,
              CallbackType clientCallback, const ResponseOptions &options = ResponseOptions());
 
+  /// Clears all pending requests.
+  void clear();
+
   /// Thread joins and proper shutdown are required to be handled explicitly.
+  /// If you do not want to wait, call `clear()` before destructor.
   ~AsyncService();
 
   TranslationCache::Stats cacheStats() { return cache_.stats(); }
diff --git a/src/translator/threadsafe_batching_pool.cpp b/src/translator/threadsafe_batching_pool.cpp
index 0c0d8d85a..0a1a28d4e 100644
--- a/src/translator/threadsafe_batching_pool.cpp
+++ b/src/translator/threadsafe_batching_pool.cpp
@@ -27,6 +27,13 @@ void ThreadsafeBatchingPool<BatchingPoolType>::enqueueRequest(Args &&... args) {
   work_.notify_all();
 }
 
+template <class BatchingPoolType>
+void ThreadsafeBatchingPool<BatchingPoolType>::clear() {
+  std::unique_lock<std::mutex> lock(mutex_);
+  backend_.clear();
+  enqueued_ = 0;
+}
+
 template <class BatchingPoolType>
 void ThreadsafeBatchingPool<BatchingPoolType>::shutdown() {
   std::unique_lock<std::mutex> lock(mutex_);
diff --git a/src/translator/threadsafe_batching_pool.h b/src/translator/threadsafe_batching_pool.h
index 96896eab3..fdbf36cdc 100644
--- a/src/translator/threadsafe_batching_pool.h
+++ b/src/translator/threadsafe_batching_pool.h
@@ -43,6 +43,12 @@ class ThreadsafeBatchingPool {
   template <class... Args>
   size_t generateBatch(Args &&... args);
 
+  // Removes any pending requests from the batching pool.
+  void clear();
+
+  // Signals shut down of batching pool. After this no new requests can be enqueued,
+  // but all enqueued requests will be processed. To prevent this from happening,
+  // call `clear()` before `shutdown()`.
   void shutdown();
 
  private:

From 495f98dd0d854854fbd38cd84bd51e38d7181253 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Sat, 22 Jan 2022 18:41:04 +0000
Subject: [PATCH 337/442] Speed up Windows CI with ccache (#308)

Use https://github.com/cristianadam/ccache/releases/ to speed up windows
compilation.

Remove /Zi as it is unsupported by ccache at the moment. This is a debug
flag that was removed in upstream marian-dev
https://github.com/browsermt/marian-dev/pull/43. However, the bergamot
CMakeLists.txt which was originally taken from
marian maintained this under MSCV.
---
 .github/workflows/windows.yml | 52 +++++++++++++++++++++++++++++++++--
 CMakeLists.txt                |  2 +-
 2 files changed, 51 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 66ac2c413..391200fcd 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -8,6 +8,13 @@ on:
 
 env:
   MKL_URL: "https://romang.blob.core.windows.net/mariandev/ci/mkl-2020.1-windows-static.zip"
+  CCACHE_BASEDIR: "${{ github.workspace }}"
+  CCACHE_DIR: "${{ github.workspace }}\\ccache"
+  CCACHE_COMPILERCHECK: content
+  CCACHE_COMPRESS: 'true'
+  CCACHE_COMPRESSLEVEL: 9
+  CCACHE_MAXSIZE: 200M
+  ccache_version: '4.5'
 
 jobs:
   build-windows:
@@ -16,7 +23,7 @@ jobs:
         include:
           # Windows CPU-only build
           - name: "Windows CPU-only"
-            cuda: ""
+            identifier: "windows-x64"
 
     runs-on: windows-2019
     name: ${{ matrix.name }}
@@ -27,6 +34,40 @@ jobs:
       with:
         submodules: recursive
 
+
+    - name: Download ccache
+      shell: cmake -P {0}
+      run: |
+        set(ccache_url "https://github.com/cristianadam/ccache/releases/download/v${{ env.ccache_version }}/${{ runner.os }}.tar.xz")
+        file(DOWNLOAD "${ccache_url}" ./ccache.tar.xz SHOW_PROGRESS)
+        execute_process(COMMAND ${CMAKE_COMMAND} -E tar xvf ./ccache.tar.xz)
+        if(ret AND NOT ret EQUAL 0)
+          message( FATAL_ERROR "Bad exit status")
+        endif()
+
+    - name: Generate ccache_vars for ccache based on machine
+      shell: cmake -P {0}
+      id: ccache_vars
+      run: |-
+        string(TIMESTAMP current_date "%Y-%m-%d-%H;%M;%S" UTC)
+        message("::set-output name=timestamp::${current_date}")
+        message("::set-output name=hash::${{ env.ccache_compilercheck }}")
+
+    - name: Cache-op for build-cache through ccache
+      uses: actions/cache@v2
+      with:
+        path: ${{ env.CCACHE_DIR }}
+        key: ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+        restore-keys: |-
+          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
+          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}
+          ccache-${{ matrix.identifier }}
+
+    - name: ccache prolog
+      run: |-
+        ${{github.workspace}}\ccache.exe -sv # Print current cache stats
+        ${{github.workspace}}\ccache.exe -z # Print current cache stats
+
     - name: Download MKL
       run: |
         # Wget retries downloading files and is faster than Invoke-WebRequest
@@ -52,7 +93,10 @@ jobs:
         cmakeAppendedArgs: '-G Ninja
           -DCMAKE_BUILD_TYPE="Release"
           -DUSE_WASM_COMPATIBLE_SOURCE="OFF"
-          -DUSE_STATIC_LIBS="TRUE"'
+          -DUSE_STATIC_LIBS="TRUE" 
+          -DCMAKE_CXX_COMPILER_LAUNCHER=${{github.workspace}}\ccache.exe
+          -DCMAKE_C_COMPILER_LAUNCHER=${{github.workspace}}\ccache.exe
+        '
         cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
         cmakeListsTxtPath: ${{ github.workspace }}/CMakeLists.txt
         useVcpkgToolchainFile: true
@@ -64,3 +108,7 @@ jobs:
       run: |
         .\app\bergamot.exe --version
       shell: cmd
+
+    - name: ccache epilog
+      run: |-
+        ${{github.workspace}}\\ccache.exe -sv # Print current cache stats
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f121ca0fb..f8e50d4ac 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -55,7 +55,7 @@ if(MSVC)
   set(INTRINSICS ${MSVC_BUILD_ARCH}) # ARCH we're targetting on win32. @TODO variable
   
   set(CMAKE_CXX_FLAGS           "/EHsc /DWIN32 /D_WINDOWS /DUNICODE /D_UNICODE /D_CRT_NONSTDC_NO_WARNINGS /D_CRT_SECURE_NO_WARNINGS /bigobj")
-  set(CMAKE_CXX_FLAGS_RELEASE   "${CMAKE_CXX_FLAGS} /MT /O2 ${INTRINSICS} /Zi /MP /GL /DNDEBUG")
+  set(CMAKE_CXX_FLAGS_RELEASE   "${CMAKE_CXX_FLAGS} /MT /O2 ${INTRINSICS} /MP /GL /DNDEBUG")
   set(CMAKE_CXX_FLAGS_DEBUG     "${CMAKE_CXX_FLAGS} /MTd /Od /Ob0 ${INTRINSICS} /RTC1 /Zi /D_DEBUG")
 
   # ignores warning LNK4049: locally defined symbol free imported - this comes from zlib

From 3dde0fe245742911057e3036e5cb6a7a32c182b3 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Mon, 24 Jan 2022 17:36:17 +0000
Subject: [PATCH 338/442] Remove unused compiler hash script (#309)

---
 scripts/ci/compiler-hash.sh | 35 -----------------------------------
 1 file changed, 35 deletions(-)
 delete mode 100644 scripts/ci/compiler-hash.sh

diff --git a/scripts/ci/compiler-hash.sh b/scripts/ci/compiler-hash.sh
deleted file mode 100644
index a770dfd49..000000000
--- a/scripts/ci/compiler-hash.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/bash
-
-# Uses the command from https://stackoverflow.com/a/9355840/4565794.
-# -v displays the commands executed to run compilation. Of this cc1 additional
-#    flags which contain the flags triggered by -march=native is what we need.
-# -E stop after preprocessing stage.
-
-# Output on a linux machine with gcc-8 looks as follows:
-
-# $ gcc -march=native -E -v - </dev/null 2>&1 | grep cc1
-#       /usr/lib/gcc/x86_64-linux-gnu/8/cc1 -E -quiet -v -imultiarch x86_64-linux-gnu
-#       - -march=skylake-avx512 -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3
-#       -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm
-#       -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mno-sgx -mbmi2 -mno-pconfig
-#       -mno-wbnoinvd -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm
-#       -mno-hle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave
-#       -mxsaveopt -mavx512f -mno-avx512er -mavx512cd -mno-avx512pf -mno-prefetchwt1
-#       -mclflushopt -mxsavec -mxsaves -mavx512dq -mavx512bw -mavx512vl
-#       -mno-avx512ifma -mno-avx512vbmi -mno-avx5124fmaps -mno-avx5124vnniw -mclwb
-#       -mno-mwaitx -mno-clzero -mpku -mno-rdpid -mno-gfni -mno-shstk
-#       -mno-avx512vbmi2 -mavx512vnni -mno-vaes -mno-vpclmulqdq -mno-avx512bitalg
-#       -mno-movdiri -mno-movdir64b --param l1-cache-size=32 --param
-#       l1-cache-line-size=64 --param l2-cache-size=28160 -mtune=skylake-avx512
-#       -fstack-protector-strong -Wformat -Wformat-security
-
-# The sha256sum of the output is computed, and stripped to the first 8
-# characters for use in ccache and github cache store key. Can effectively be
-# considered as a hash of the compiler version and the flags activated by
-# -march=native.
-
-COMPILER=$1
-
-$COMPILER -march=native -E -v - < /dev/null 2>&1 | grep cc1 \
-    | sha256sum | cut -c1-8
-

From c0f311a8c067372057a6f301c42b40bbe30a9c1a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Wed, 26 Jan 2022 20:33:43 +0000
Subject: [PATCH 339/442] Batteries included python package (#310)

Imports python bindings and associated sources incubated in
https://github.com/jerinphilip/lemonade to bergamot-translator. Adds
 a pybind11 dependency for python bindings.

Following the import, the python build is integrated into the existing
CMake based build system here. There is a command-line application
provided through python which provides the ability to fetch and prepare
models from model-repositories (like browsermt/students or OPUS).

Wheels built for a few common operating systems are provided via GitHub
releases through automated actions configured to run at tagged semantic
versions and pushes to main.

The documentation for python is also integrated into our existing
documentation setup. Previous documentation GitHub action is now
configured to run behind python builds in Ubuntu 18.04 Python3.7,
in order to pick up the packaged as a wheel bergamot module and the
sphinx documentation using the python module.

Formatting checks of black, isort with profile black and a pytype type
checker is configured for the python component residing in this repository.
---
 .github/workflows/coding-styles.yml       |   2 +-
 .github/workflows/doc.yml                 |  89 -----
 .github/workflows/python.yml              | 378 ++++++++++++++++++++++
 .gitmodules                               |   3 +
 3rd_party/CMakeLists.txt                  |   4 +
 3rd_party/pybind11                        |   1 +
 CMakeLists.txt                            |   5 +
 MANIFEST.in                               |   2 +
 bindings/CMakeLists.txt                   |   1 +
 bindings/python/CMakeLists.txt            |   9 +
 bindings/python/__init__.py               |  18 ++
 bindings/python/__main__.py               |  20 ++
 bindings/python/bergamot.cpp              | 216 +++++++++++++
 bindings/python/cmds.py                   | 177 ++++++++++
 bindings/python/repository.py             | 185 +++++++++++
 bindings/python/typing_utils.py           |   5 +
 bindings/python/utils.py                  |  52 +++
 doc/conf.py                               |  82 ++---
 doc/index.rst                             |   1 +
 doc/python.rst                            |  87 +++++
 doc/requirements.txt                      |   1 +
 patches/01-marian-fstream-for-macos.patch |  13 +
 setup.py                                  | 210 ++++++++++++
 23 files changed, 1433 insertions(+), 128 deletions(-)
 delete mode 100644 .github/workflows/doc.yml
 create mode 100644 .github/workflows/python.yml
 create mode 160000 3rd_party/pybind11
 create mode 100644 MANIFEST.in
 create mode 100644 bindings/CMakeLists.txt
 create mode 100644 bindings/python/CMakeLists.txt
 create mode 100644 bindings/python/__init__.py
 create mode 100644 bindings/python/__main__.py
 create mode 100644 bindings/python/bergamot.cpp
 create mode 100644 bindings/python/cmds.py
 create mode 100644 bindings/python/repository.py
 create mode 100644 bindings/python/typing_utils.py
 create mode 100644 bindings/python/utils.py
 create mode 100644 doc/python.rst
 create mode 100644 patches/01-marian-fstream-for-macos.patch
 create mode 100644 setup.py

diff --git a/.github/workflows/coding-styles.yml b/.github/workflows/coding-styles.yml
index 0bff2ec79..81bdf3361 100644
--- a/.github/workflows/coding-styles.yml
+++ b/.github/workflows/coding-styles.yml
@@ -26,7 +26,7 @@ jobs:
 
         - name: Run clang-format
           run:
-              python3 run-clang-format.py --style file -r src wasm
+              python3 run-clang-format.py --style file -r src wasm bindings/python
 
 
         - name: Prepare build, compilation database etc.
diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml
deleted file mode 100644
index 3874822b8..000000000
--- a/.github/workflows/doc.yml
+++ /dev/null
@@ -1,89 +0,0 @@
-name: Documentation
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-    tags: ['v[0-9]+.[0-9]+.[0-9]+']
-  pull_request: 
-    branches: [ '**' ]
-
-jobs:
-  api-documentation:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-        with:
-          submodules: recursive
-
-      # Runs javascript to extract push events from both tags and branch (only main, due to workflow trigger)
-      # converts refs/<>/<name> -> <name>
-      # eg:
-      #     refs/head/main   -> main
-      #     refs/tags/v0.1.0 -> v0.1.0
-      #
-      - name: Extract tag name
-        id: tag
-        uses: actions/github-script@0.2.0
-        if: ${{ github.event_name == 'push' }}
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          script: |
-            const args = context.payload.ref.split("/");
-            [refs, category, ...rest] = args;
-            return rest.join("/");
-
-      # Patches the BERGAMOT_VERSION file used by sphinx-docs at run time to
-      # obtain names like 'main' or 'ci-sandbox' to not confuse with version
-      # based documentation built separately.
-      - name: Deploy-time patch version 
-        run: | 
-            echo ${{steps.tag.outputs.result }} > BERGAMOT_VERSION
-
-      - name: Set up Doxygen
-        run: sudo apt-get install -y doxygen
-
-      - name: Set up Python
-        uses: actions/setup-python@v2
-        with:
-          python-version: 3.7
-
-      - name: Set up dependency cache
-        uses: actions/cache@v2
-        with:
-          path: ~/.cache/pip
-          key: ${{ runner.os }}-pip-${{ hashFiles('doc/requirements.txt') }}
-          restore-keys: |
-            ${{ runner.os }}-pip-
-
-      - name: Install dependencies
-        working-directory: ./doc
-        run: python3 -m pip install -r requirements.txt
-
-      - name: Build documentation
-        working-directory: ./doc
-        run: sphinx-build -b html ./ build/
-
-
-      - name: Deploy 🚀
-        uses: JamesIves/github-pages-deploy-action@4.1.3
-        if: ${{ github.event_name == 'push' && github.repository == 'browsermt/bergamot-translator' }}
-        with:
-          repository-name: 'browsermt/docs' 
-          branch: gh-pages # The branch the action should deploy to.
-          folder: './doc/build/' # The folder the action should deploy.
-          target-folder: '${{ steps.tag.outputs.result }}' 
-          ssh-key: ${{ secrets.BERGAMOT_SSH_PRIVATE_KEY }}
-
-      # This artifact contains the HTML output of Sphinx only.
-      # With index.html at the root of the produced zip file.
-      # For use for maintainers to download the zip and check render of
-      # documentation while generated at pull-request. 
-      - name: Upload documentation
-        uses: actions/upload-artifact@v2
-        if: ${{ github.event_name == 'pull_request'}}
-        with:
-          name: api-docs
-          path: ./doc/build/
-          if-no-files-found: error
-
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
new file mode 100644
index 000000000..2924061df
--- /dev/null
+++ b/.github/workflows/python.yml
@@ -0,0 +1,378 @@
+name: "Python Bindings"
+'on':
+  push:
+    branches:
+      - main
+      - ci-sandbox
+    tags:
+      - "v*.*.*"
+  pull_request:
+    branches:
+      - '**'
+env:
+  qt_version: "6.2.1" # only used by build-macos
+  ccache_basedir: ${{ github.workspace }}
+  ccache_dir: "${{ github.workspace }}/.ccache"
+  ccache_compilercheck: content
+  ccache_compress: 'true'
+  ccache_compresslevel: 9
+  ccache_maxsize: 200M
+  ccache_cmake: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
+
+jobs:
+    python-ubuntu:
+      strategy:
+        fail-fast: false
+        matrix:
+          include:
+              - name: "Ubuntu 18.04 / py3.6"
+                os: "ubuntu-18.04"
+                python-version: "3.6"
+              - name: "Ubuntu 18.04 / py3.7"
+                os: "ubuntu-18.04"
+                python-version: "3.7"
+              - name: "Ubuntu 20.04 / py3.8"
+                os: "ubuntu-20.04"
+                python-version: "3.8"
+              - name: "Ubuntu 20.04 / py3.9"
+                os: "ubuntu-20.04"
+                python-version: "3.9"
+              - name: "Ubuntu 20.04 / py3.10"
+                os: "ubuntu-20.04"
+                python-version: "3.10"
+
+      name: ${{ matrix.name }}
+      runs-on: ${{ matrix.os }}
+      steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: recursive
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+
+
+      - name: Install Dependencies
+        run: |-
+          sudo apt-get update
+          sudo apt-get install -y \
+            ccache  libprotobuf-dev protobuf-compiler \
+            python3-setuptools python3-pybind11 
+
+      - name: Install MKL
+        run: |-
+          wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB" | sudo apt-key add -
+          sudo sh -c "echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list"
+          sudo apt-get update -o Dir::Etc::sourcelist="/etc/apt/sources.list.d/intel-mkl.list"
+          sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
+
+      - name: Generate ccache_vars for ccache based on machine
+        shell: bash
+        id: ccache_vars
+        run: |-
+          echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
+          echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
+
+      - name: Cache-op for build-cache through ccache
+        uses: actions/cache@v2
+        with:
+          path: ${{ env.ccache_dir }}
+          key: ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+          restore-keys: |-
+            ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
+            ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}
+            ccache-${{ matrix.name }}
+      - name: ccache environment setup
+        run: |-
+          echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
+          echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
+          echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
+          echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
+          echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
+          echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
+
+      - name: ccache prolog
+        run: |-
+          ccache -s # Print current cache stats
+          ccache -z # Zero cache entry
+
+      - name: Inject local version identifier for non tag builds
+        if: ${{ !startsWith(github.ref, 'refs/tags/v') }}
+        run: |-
+          echo "PYTHON_LOCAL_VERSION_IDENTIFIER=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
+
+      - name: setup.py 
+        run: |-
+          python3 -m pip install wheel
+          BUILD_ARCH=core-avx-i python3 setup.py bdist_wheel --universal
+
+      # We're happy with just compile for the moment, so cache gets some seeding.
+      - name: Install onto root python lib
+        run: |-
+          python3 -m pip install --ignore-installed dist/bergamot-*.whl 
+
+      - name: Fetch models from translateLocally repository.
+        run: |-
+          python3 -m bergamot download -m en-de-tiny
+          python3 -m bergamot download -m de-en-tiny
+          python3 -m bergamot ls
+
+      - name: Fetch models from opus repository.
+        run: |-
+          python3 -m bergamot download -m eng-fin-tiny -r opus
+          python3 -m bergamot ls -r opus
+
+      - name: Run the sample python script shipped with module
+        run: |-
+          python3 -m bergamot translate --model en-de-tiny <<< "Hello World"
+          python3 -m bergamot translate --model en-de-tiny de-en-tiny <<< "Hello World"
+          python3 -m bergamot translate --model eng-fin-tiny --repository opus <<< "Hello World"
+
+      - name: ccache epilog
+        run: 'ccache -s # Print current cache stats'
+
+      - uses: actions/upload-artifact@v2
+        with:
+            path: ${{github.workspace}}/dist/bergamot-*.whl
+
+
+    python-macos:
+      name: "MacOS 10.15 / py3.10"
+      runs-on: "macos-10.15"
+      steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: recursive
+      - name: Install Dependencies
+        run: |-
+          brew update
+          brew install openblas protobuf ccache boost pybind11 
+          brew install coreutils findutils libarchive 
+
+      - name: Generate ccache_vars for ccache based on machine
+        shell: bash
+        id: ccache_vars
+        run: |-
+          echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
+          echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
+      - name: Cache-op for build-cache through ccache
+        uses: actions/cache@v2
+        with:
+          path: ${{ env.ccache_dir }}
+          key: ccache-${{ job.id }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+          restore-keys: |-
+            ccache-${{ job.id }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
+            ccache-${{ job.id }}-${{ steps.ccache_vars.outputs.hash }}
+            ccache-${{ job.id }}
+
+      - name: ccache environment setup
+        run: |-
+          echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
+          echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
+          echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
+          echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
+          echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
+          echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
+
+      - name: ccache prolog
+        run: |-
+          ccache -s # Print current cache stats
+          ccache -z # Zero cache entry
+
+      - name: Apply required patches
+        run: |-
+            patch -p1 < patches/01-marian-fstream-for-macos.patch
+
+      # Appears to be required per GitHub CI; 
+      - name: Set MACOSX DEPLOYMENT TARGET via environment variable
+        run: |-
+            echo "MACOSX_DEPLOYMENT_TARGET=10.15" >> $GITHUB_ENV
+
+      - name: Inject local version identifier for non tag builds
+        if: ${{ !startsWith(github.ref, 'refs/tags/v') }}
+        run: |-
+          echo "PYTHON_LOCAL_VERSION_IDENTIFIER=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
+
+      - name: setup.py 
+        run: |-
+          python3 -m pip install --upgrade packaging wheel
+          BUILD_ARCH=core-avx-i python3 setup.py bdist_wheel --universal
+
+      # We're happy with just compile for the moment, so cache gets some seeding.
+      - name: Install onto root python lib
+        run: |-
+          python3 -m pip install dist/bergamot-*.whl 
+
+      - name: Fetch models from translateLocally repository.
+        run: |-
+          python3 -m bergamot download -m en-de-tiny
+          python3 -m bergamot download -m de-en-tiny
+
+      - name: Fetch models from opus repository.
+        run: |-
+          python3 -m bergamot download -m eng-fin-tiny -r opus
+          python3 -m bergamot ls -r opus
+
+      - name: Run the sample python script shipped with module
+        run: |-
+          python3 -m bergamot translate --model en-de-tiny <<< "Hello World"
+          python3 -m bergamot translate --model en-de-tiny de-en-tiny <<< "Hello World"
+          python3 -m bergamot translate --model eng-fin-tiny --repository opus <<< "Hello World"
+
+      - name: ccache epilog
+        run: 'ccache -s # Print current cache stats'
+
+      - uses: actions/upload-artifact@v2
+        with:
+            path: ${{github.workspace}}/dist/bergamot-*.whl
+
+  # Try to upload a release using https://github.com/marvinpinto/actions/issues/177#issuecomment-917605585 as a model
+    release-latest:
+      name: Release Latest Build
+      runs-on: ubuntu-latest
+      needs: [python-ubuntu, python-macos]
+      if: github.ref == 'refs/heads/main'
+      steps:
+       - name: Download artifacts
+         uses: actions/download-artifact@v2
+  
+       - name: Update GitHub prerelease
+         uses: marvinpinto/action-automatic-releases@latest
+         with:
+           repo_token: ${{ secrets.GITHUB_TOKEN }}
+           automatic_release_tag: latest
+           prerelease: true
+           title: "Latest Build"
+           files: |
+                  ${{github.workspace}}/artifact/*.whl
+  
+    release-version:
+      name: Release version 
+      runs-on: ubuntu-latest
+      needs: [python-ubuntu, python-macos]
+      permissions:
+        contents: "write"
+        packages: "write"
+        pull-requests: "read"
+      if: startsWith(github.ref, 'refs/tags/v')
+      steps:
+       - name: Download artifacts
+         uses: actions/download-artifact@v2
+  
+       - name: Update GitHub release
+         uses: marvinpinto/action-automatic-releases@latest
+         with:
+           repo_token: ${{ secrets.GITHUB_TOKEN }}
+           automatic_release_tag: ${{ github.ref_name }}
+           prerelease: false
+           title: "${{ github.ref_name }}"
+           files: |
+                  ${{github.workspace}}/artifact/*.whl
+  
+    python-checks:
+      name: "formatting and typechecks"
+      runs-on: "ubuntu-latest"
+      steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: recursive
+      - name: Install Dependencies
+        run: |-
+            python3 -m pip install black isort pytype
+      - name: "Formatting checks: black, isort"
+        run: |
+            python3 -m black --check bindings/python/ setup.py doc/conf.py
+            python3 -m isort --profile black --diff --check bindings/python setup.py doc/conf.py
+      - name: "Static typing checks: pytype"
+        run: |-
+            python3 -m pytype bindings/python
+
+    docs:
+      runs-on: ubuntu-18.04
+      needs: [python-ubuntu]
+      steps:
+        - name: Checkout
+          uses: actions/checkout@v2
+          with:
+            submodules: recursive
+
+        # Runs javascript to extract push events from both tags and branch (only main, due to workflow trigger)
+        # converts refs/<>/<name> -> <name>
+        # eg:
+        #     refs/head/main   -> main
+        #     refs/tags/v0.1.0 -> v0.1.0
+        #
+        - name: Download artifacts
+          uses: actions/download-artifact@v2
+        - name: Extract tag name
+          id: tag
+          uses: actions/github-script@0.2.0
+          if: ${{ github.event_name == 'push' }}
+          with:
+            github-token: ${{ secrets.GITHUB_TOKEN }}
+            script: |
+              const args = context.payload.ref.split("/");
+              [refs, category, ...rest] = args;
+              return rest.join("/");
+
+        # Patches the BERGAMOT_VERSION file used by sphinx-docs at run time to
+        # obtain names like 'main' or 'ci-sandbox' to not confuse with version
+        # based documentation built separately.
+        - name: Deploy-time patch version 
+          run: | 
+              echo ${{steps.tag.outputs.result }} > BERGAMOT_VERSION
+
+        - name: Set up Doxygen
+          run: sudo apt-get install -y doxygen
+
+        - name: Set up Python
+          uses: actions/setup-python@v2
+          with:
+            python-version: 3.7
+
+        - name: Set up dependency cache
+          uses: actions/cache@v2
+          with:
+            path: ~/.cache/pip
+            key: ${{ runner.os }}-pip-${{ hashFiles('doc/requirements.txt') }}
+            restore-keys: |
+              ${{ runner.os }}-pip-
+
+        - name: Install dependencies
+          working-directory: ./doc
+          run: |
+            python3 -m pip install -r requirements.txt
+            python3 -m pip install ${{github.workspace}}/artifact/bergamot-*-cp37*.whl
+
+        - name: Build documentation
+          working-directory: ./doc
+          run: sphinx-build -b html ./ build/
+
+
+        - name: Deploy 🚀
+          uses: JamesIves/github-pages-deploy-action@4.1.3
+          if: ${{ github.event_name == 'push' && github.repository == 'browsermt/bergamot-translator' }}
+          with:
+            repository-name: 'browsermt/docs' 
+            branch: gh-pages # The branch the action should deploy to.
+            folder: './doc/build/' # The folder the action should deploy.
+            target-folder: '${{ steps.tag.outputs.result }}' 
+            ssh-key: ${{ secrets.BERGAMOT_SSH_PRIVATE_KEY }}
+
+        # This artifact contains the HTML output of Sphinx only.
+        # With index.html at the root of the produced zip file.
+        # For use for maintainers to download the zip and check render of
+        # documentation while generated at pull-request. 
+        - name: Upload documentation
+          uses: actions/upload-artifact@v2
+          if: ${{ github.event_name == 'pull_request'}}
+          with:
+            name: api-docs
+            path: ./doc/build/
+            if-no-files-found: error
+
diff --git a/.gitmodules b/.gitmodules
index 8aa101494..cfedde289 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -7,3 +7,6 @@
 [submodule "bergamot-translator-tests"]
 	path = bergamot-translator-tests
 	url = https://github.com/browsermt/bergamot-translator-tests
+[submodule "3rd_party/pybind11"]
+	path = 3rd_party/pybind11
+	url = https://github.com/pybind/pybind11.git
diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index b84a37b80..72a49e83a 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -23,3 +23,7 @@ get_directory_property(CMAKE_C_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_C_FLA
 get_directory_property(CMAKE_CXX_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_CXX_FLAGS) 
 set(CMAKE_C_FLAGS ${CMAKE_C_FLAGS} PARENT_SCOPE)    
 set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} PARENT_SCOPE)    
+
+if(COMPILE_PYTHON)
+  add_subdirectory(pybind11)
+endif(COMPILE_PYTHON)
diff --git a/3rd_party/pybind11 b/3rd_party/pybind11
new file mode 160000
index 000000000..9ec1128c7
--- /dev/null
+++ b/3rd_party/pybind11
@@ -0,0 +1 @@
+Subproject commit 9ec1128c7aac3d069a4ec2bd1dfc7f57c6526d1c
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f8e50d4ac..f6e6af4f5 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -131,3 +131,8 @@ else()
   add_subdirectory(app)
 endif(COMPILE_WASM)
 
+option(COMPILE_PYTHON "Compile python bindings. Intended to be activated with setup.py" OFF)
+if(COMPILE_PYTHON)
+  add_subdirectory(bindings/python)
+endif(COMPILE_PYTHON)
+
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 000000000..009fd4e31
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,2 @@
+include README.md LICENSE
+
diff --git a/bindings/CMakeLists.txt b/bindings/CMakeLists.txt
new file mode 100644
index 000000000..8e5f91a37
--- /dev/null
+++ b/bindings/CMakeLists.txt
@@ -0,0 +1 @@
+add_subdirectory(python)
diff --git a/bindings/python/CMakeLists.txt b/bindings/python/CMakeLists.txt
new file mode 100644
index 000000000..70b1a2535
--- /dev/null
+++ b/bindings/python/CMakeLists.txt
@@ -0,0 +1,9 @@
+find_package(Python COMPONENTS Interpreter Development REQUIRED)
+
+message("Using Python: " ${Python_EXECUTABLE})
+
+# pybind11 method:
+pybind11_add_module(_bergamot SHARED bergamot.cpp)
+target_link_libraries(_bergamot PUBLIC pybind11::module pybind11::headers bergamot-translator)
+target_include_directories(_bergamot PUBLIC ${PROJECT_SOURCE_DIR} ${PROJECT_SOURCE_DIR}/src 
+    ${CMAKE_BINARY_DIR}/src)
diff --git a/bindings/python/__init__.py b/bindings/python/__init__.py
new file mode 100644
index 000000000..5855a4faf
--- /dev/null
+++ b/bindings/python/__init__.py
@@ -0,0 +1,18 @@
+import typing
+
+from ._bergamot import *  # type: ignore
+from .repository import Aggregator, TranslateLocallyLike
+
+REPOSITORY = Aggregator(
+    [
+        TranslateLocallyLike("browsermt", "https://translatelocally.com/models.json"),
+        TranslateLocallyLike(
+            "opus", "https://object.pouta.csc.fi/OPUS-MT-models/app/models.json"
+        ),
+    ]
+)
+"""
+REPOSITORY is a global object that aggregates multiple model-providers to
+provide a (model-provider: str, model-code: str) based query mechanism to
+get models.
+"""
diff --git a/bindings/python/__main__.py b/bindings/python/__main__.py
new file mode 100644
index 000000000..35014c099
--- /dev/null
+++ b/bindings/python/__main__.py
@@ -0,0 +1,20 @@
+import argparse
+import sys
+from argparse import ArgumentParser
+
+from .cmds import CMDS, make_parser
+
+
+def main() -> None:
+    parser = make_parser()
+    args = parser.parse_args()
+
+    if args.action in CMDS:
+        CMDS[args.action].execute(args)
+    else:
+        parser.print_help(sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/bindings/python/bergamot.cpp b/bindings/python/bergamot.cpp
new file mode 100644
index 000000000..97ae0614b
--- /dev/null
+++ b/bindings/python/bergamot.cpp
@@ -0,0 +1,216 @@
+#include <pybind11/iostream.h>
+#include <pybind11/pybind11.h>
+#include <pybind11/stl.h>
+#include <pybind11/stl_bind.h>
+#include <translator/annotation.h>
+#include <translator/parser.h>
+#include <translator/project_version.h>
+#include <translator/response.h>
+#include <translator/response_options.h>
+#include <translator/service.h>
+#include <translator/translation_model.h>
+
+#include <iostream>
+#include <string>
+#include <vector>
+
+namespace py = pybind11;
+
+using marian::bergamot::AnnotatedText;
+using marian::bergamot::ByteRange;
+using marian::bergamot::ConcatStrategy;
+using marian::bergamot::Response;
+using marian::bergamot::ResponseOptions;
+using Service = marian::bergamot::AsyncService;
+using _Model = marian::bergamot::TranslationModel;
+using Model = std::shared_ptr<_Model>;
+using Alignment = std::vector<std::vector<float>>;
+using Alignments = std::vector<Alignment>;
+
+PYBIND11_MAKE_OPAQUE(std::vector<Response>);
+PYBIND11_MAKE_OPAQUE(std::vector<std::string>);
+PYBIND11_MAKE_OPAQUE(Alignments);
+
+class ServicePyAdapter {
+ public:
+  ServicePyAdapter(const Service::Config &config) : service_(make_service(config)) {}
+
+  std::shared_ptr<_Model> modelFromConfig(const std::string &config) {
+    auto parsedConfig = marian::bergamot::parseOptionsFromString(config);
+    return service_.createCompatibleModel(parsedConfig);
+  }
+
+  std::shared_ptr<_Model> modelFromConfigPath(const std::string &configPath) {
+    auto config = marian::bergamot::parseOptionsFromFilePath(configPath);
+    return service_.createCompatibleModel(config);
+  }
+
+  std::vector<Response> translate(Model model, std::vector<std::string> &inputs, const ResponseOptions &options) {
+    py::scoped_ostream_redirect outstream(std::cout,                                 // std::ostream&
+                                          py::module_::import("sys").attr("stdout")  // Python output
+    );
+    py::scoped_ostream_redirect errstream(std::cerr,                                 // std::ostream&
+                                          py::module_::import("sys").attr("stderr")  // Python output
+    );
+
+    py::call_guard<py::gil_scoped_release> gil_guard;
+
+    // Prepare promises, save respective futures. Have callback's in async set
+    // value to the promises.
+    std::vector<std::future<Response>> futures;
+    std::vector<std::promise<Response>> promises;
+    promises.resize(inputs.size());
+
+    for (size_t i = 0; i < inputs.size(); i++) {
+      auto callback = [&promises, i](Response &&response) { promises[i].set_value(std::move(response)); };
+
+      service_.translate(model, std::move(inputs[i]), std::move(callback), options);
+
+      futures.push_back(std::move(promises[i].get_future()));
+    }
+
+    // Wait on all futures to be ready.
+    std::vector<Response> responses;
+    for (size_t i = 0; i < futures.size(); i++) {
+      futures[i].wait();
+      responses.push_back(std::move(futures[i].get()));
+    }
+
+    return responses;
+  }
+
+  std::vector<Response> pivot(Model first, Model second, std::vector<std::string> &inputs,
+                              const ResponseOptions &options) {
+    py::scoped_ostream_redirect outstream(std::cout,                                 // std::ostream&
+                                          py::module_::import("sys").attr("stdout")  // Python output
+    );
+    py::scoped_ostream_redirect errstream(std::cerr,                                 // std::ostream&
+                                          py::module_::import("sys").attr("stderr")  // Python output
+    );
+
+    py::call_guard<py::gil_scoped_release> gil_guard;
+    // Prepare promises, save respective futures. Have callback's in async set
+    // value to the promises.
+    std::vector<std::future<Response>> futures;
+    std::vector<std::promise<Response>> promises;
+    promises.resize(inputs.size());
+
+    for (size_t i = 0; i < inputs.size(); i++) {
+      auto callback = [&promises, i](Response &&response) { promises[i].set_value(std::move(response)); };
+
+      service_.pivot(first, second, std::move(inputs[i]), std::move(callback), options);
+
+      futures.push_back(std::move(promises[i].get_future()));
+    }
+
+    // Wait on all futures to be ready.
+    std::vector<Response> responses;
+    for (size_t i = 0; i < futures.size(); i++) {
+      futures[i].wait();
+      responses.push_back(std::move(futures[i].get()));
+    }
+
+    return responses;
+  }
+
+ private /*functions*/:
+  static Service make_service(const Service::Config &config) {
+    py::scoped_ostream_redirect outstream(std::cout,                                 // std::ostream&
+                                          py::module_::import("sys").attr("stdout")  // Python output
+    );
+    py::scoped_ostream_redirect errstream(std::cerr,                                 // std::ostream&
+                                          py::module_::import("sys").attr("stderr")  // Python output
+    );
+
+    py::call_guard<py::gil_scoped_release> gil_guard;
+
+    return Service(config);
+  }
+
+ private /*data*/:
+  Service service_;
+};
+
+PYBIND11_MODULE(_bergamot, m) {
+  m.doc() = "Bergamot pybind11 bindings";
+  m.attr("__version__") = marian::bergamot::bergamotBuildVersion();
+  py::class_<ByteRange>(m, "ByteRange")
+      .def(py::init<>())
+      .def_readonly("begin", &ByteRange::begin)
+      .def_readonly("end", &ByteRange::end)
+      .def("__repr__", [](const ByteRange &range) {
+        return "{" + std::to_string(range.begin) + ", " + std::to_string(range.end) + "}";
+      });
+
+  py::class_<AnnotatedText>(m, "AnnotatedText")
+      .def(py::init<>())
+      .def("numWords", &AnnotatedText::numWords)
+      .def("numSentences", &AnnotatedText::numSentences)
+      .def("word",
+           [](const AnnotatedText &annotatedText, size_t sentenceIdx, size_t wordIdx) -> std::string {
+             auto view = annotatedText.word(sentenceIdx, wordIdx);
+             return std::string(view.data(), view.size());
+           })
+      .def("sentence",
+           [](const AnnotatedText &annotatedText, size_t sentenceIdx) -> std::string {
+             auto view = annotatedText.sentence(sentenceIdx);
+             return std::string(view.data(), view.size());
+           })
+      .def("wordAsByteRange", &AnnotatedText::wordAsByteRange)
+      .def("sentenceAsByteRange", &AnnotatedText::sentenceAsByteRange)
+      .def_readonly("text", &AnnotatedText::text);
+
+  py::class_<Response>(m, "Response")
+      .def(py::init<>())
+      .def_readonly("source", &Response::source)
+      .def_readonly("target", &Response::target)
+      .def_readonly("alignments", &Response::alignments);
+
+  py::bind_vector<std::vector<std::string>>(m, "VectorString");
+  py::bind_vector<std::vector<Response>>(m, "VectorResponse");
+
+  py::enum_<ConcatStrategy>(m, "ConcatStrategy")
+      .value("FAITHFUL", ConcatStrategy::FAITHFUL)
+      .value("SPACE", ConcatStrategy::SPACE)
+      .export_values();
+
+  py::class_<ResponseOptions>(m, "ResponseOptions")
+      .def(
+          py::init<>([](bool qualityScores, bool alignment, bool HTML, bool sentenceMappings, ConcatStrategy strategy) {
+            return ResponseOptions{qualityScores, alignment, HTML, sentenceMappings, strategy};
+          }),
+          py::arg("qualityScores") = true, py::arg("alignment") = false, py::arg("HTML") = false,
+          py::arg("sentenceMappings") = true, py::arg("concatStrategy") = ConcatStrategy::FAITHFUL)
+      .def_readwrite("qualityScores", &ResponseOptions::qualityScores)
+      .def_readwrite("HTML", &ResponseOptions::HTML)
+      .def_readwrite("alignment", &ResponseOptions::alignment)
+      .def_readwrite("concatStrategy", &ResponseOptions::concatStrategy)
+      .def_readwrite("sentenceMappings", &ResponseOptions::sentenceMappings);
+
+  py::class_<ServicePyAdapter>(m, "Service")
+      .def(py::init<const Service::Config &>())
+      .def("modelFromConfig", &ServicePyAdapter::modelFromConfig)
+      .def("modelFromConfigPath", &ServicePyAdapter::modelFromConfigPath)
+      .def("translate", &ServicePyAdapter::translate)
+      .def("pivot", &ServicePyAdapter::pivot);
+
+  py::class_<Service::Config>(m, "ServiceConfig")
+      .def(py::init<>([](size_t numWorkers, bool cacheEnabled, size_t cacheSize, size_t cacheMutexBuckets,
+                         std::string logging) {
+             Service::Config config;
+             config.numWorkers = numWorkers;
+             config.cacheEnabled = cacheEnabled;
+             config.cacheSize = cacheSize;
+             config.cacheMutexBuckets = cacheMutexBuckets;
+             config.logger.level = logging;
+             return config;
+           }),
+           py::arg("numWorkers") = 1, py::arg("cacheEnabled") = false, py::arg("cacheSize") = 20000,
+           py::arg("cacheMutexBuckets") = 1, py::arg("logLevel") = "off")
+      .def_readwrite("numWorkers", &Service::Config::numWorkers)
+      .def_readwrite("cacheEnabled", &Service::Config::cacheEnabled)
+      .def_readwrite("cacheSize", &Service::Config::cacheSize)
+      .def_readwrite("cacheMutexBuckets", &Service::Config::cacheMutexBuckets);
+
+  py::class_<_Model, std::shared_ptr<_Model>>(m, "TranslationModel");
+}
diff --git a/bindings/python/cmds.py b/bindings/python/cmds.py
new file mode 100644
index 000000000..5949adaca
--- /dev/null
+++ b/bindings/python/cmds.py
@@ -0,0 +1,177 @@
+import argparse
+import sys
+from collections import Counter, defaultdict
+
+from . import REPOSITORY, ResponseOptions, Service, ServiceConfig, VectorString
+
+CMDS = {}
+
+
+def _register_cmd(cmd: str):
+    """
+    Convenience decorator function, which populates the dictionary above with
+    commands created in a declarative fashion.
+    """
+
+    def __inner(cls):
+        CMDS[cmd] = cls
+        return cls
+
+    return __inner
+
+
+@_register_cmd("translate")
+class Translate:
+    @staticmethod
+    def embed_subparser(key: str, subparsers: argparse._SubParsersAction):
+        translate = subparsers.add_parser(
+            key,
+            description="translate using a given model. Multiple models mean pivoting",
+        )
+
+        translate.add_argument(
+            "-m",
+            "--model",
+            type=str,
+            nargs="+",
+            help="Path to model file(s) to use in forward or pivot translation",
+            required=True,
+        )
+
+        translate.add_argument(
+            "-r",
+            "--repository",
+            type=str,
+            help="Repository to download model from",
+            choices=REPOSITORY.available(),
+            default="browsermt",
+        )
+
+        translate.add_argument(
+            "--num-workers",
+            type=int,
+            help="Number of worker threads to use to translate",
+            default=4,
+        )
+
+        translate.add_argument(
+            "--log-level",
+            type=str,
+            default="off",
+            help="Set verbosity level of logging: trace, debug, info, warn, err(or), critical, off",
+        )
+
+        # Tweak response-options for quick HTML in out via commandline
+        options = translate.add_argument_group("response-options")
+        options.add_argument("--html", type=bool, default=False)
+        options.add_argument("--alignment", type=bool, default=False)
+        options.add_argument("--quality-scores", type=bool, default=False)
+
+    @staticmethod
+    def execute(args: argparse.Namespace):
+        # Build service
+
+        config = ServiceConfig(numWorkers=args.num_workers, logLevel=args.log_level)
+        service = Service(config)
+
+        models = [
+            service.modelFromConfigPath(
+                REPOSITORY.modelConfigPath(args.repository, model)
+            )
+            for model in args.model
+        ]
+
+        # Configure a few options which require how a Response is constructed
+        options = ResponseOptions(
+            alignment=args.alignment, qualityScores=args.quality_scores, HTML=args.html
+        )
+
+        source = sys.stdin.read()
+        responses = None
+        if len(models) == 1:
+            [model] = models
+            responses = service.translate(model, VectorString([source]), options)
+        else:
+            [first, second] = models
+            responses = service.pivot(first, second, VectorString([source]), options)
+
+        for response in responses:
+            print(response.target.text, end="")
+
+
+@_register_cmd("download")
+class Download:
+    @staticmethod
+    def embed_subparser(key: str, subparsers: argparse._SubParsersAction):
+        download = subparsers.add_parser(
+            key, description="Download models from the web."
+        )
+
+        download.add_argument(
+            "-m",
+            "--model",
+            type=str,
+            required=False,
+            default=None,
+            help="Fetch model with given code. Use ls to list available models. Optional, if none supplied all models are downloaded.",
+        )
+
+        download.add_argument(
+            "-r",
+            "--repository",
+            type=str,
+            help="Repository to download model from",
+            choices=REPOSITORY.available(),
+            default="browsermt",
+        )
+
+    @staticmethod
+    def execute(args: argparse.Namespace):
+        if args.model is not None:
+            REPOSITORY.download(args.repository, args.model)
+        else:
+            for model in REPOSITORY.models(args.repository, filter_downloaded=False):
+                REPOSITORY.download(args.repository, model)
+
+
+@_register_cmd("ls")
+class List:
+    @staticmethod
+    def embed_subparser(key: str, subparsers: argparse._SubParsersAction):
+        ls = subparsers.add_parser(key, description="List available models.")
+        ls.add_argument(
+            "-r",
+            "--repository",
+            type=str,
+            help="Repository to list models from",
+            choices=REPOSITORY.available(),
+            default="browsermt",
+        )
+
+    @staticmethod
+    def execute(args: argparse.Namespace):
+        print("Available models: ")
+        for counter, identifier in enumerate(
+            REPOSITORY.models(args.repository, filter_downloaded=True), 1
+        ):
+            model = REPOSITORY.model(args.repository, identifier)
+            print(
+                " {}.".format(str(counter).rjust(4)),
+                model["code"],
+                model["name"],
+            )
+        print()
+
+
+def make_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser("bergamot")
+    subparsers = parser.add_subparsers(
+        title="actions",
+        description="The following actions are available through the bergamot package",
+        help="To obtain help on how to run these actions supply <cmd> -h.",
+        dest="action",
+    )
+
+    for key, cls in CMDS.items():
+        cls.embed_subparser(key, subparsers)
+    return parser
diff --git a/bindings/python/repository.py b/bindings/python/repository.py
new file mode 100644
index 000000000..7f89035c8
--- /dev/null
+++ b/bindings/python/repository.py
@@ -0,0 +1,185 @@
+import json
+import os
+import tarfile
+import typing as t
+from abc import ABC, abstractmethod
+from functools import partial
+from urllib.parse import urlparse
+
+import requests
+from appdirs import AppDirs
+
+from .typing_utils import URL, PathLike
+from .utils import download_resource, patch_marian_for_bergamot
+
+APP = "bergamot"
+
+
+class Repository(ABC):
+    """
+    An interface for several repositories. Intended to enable interchangable
+    use of translateLocally and Mozilla repositories for usage through python.
+    """
+
+    @property
+    @abstractmethod
+    def name(self):
+        pass
+
+    @abstractmethod
+    def update(self):
+        """Updates the model list"""
+        pass
+
+    @abstractmethod
+    def models(self) -> t.List[str]:
+        """returns identifiers for available models"""
+        pass
+
+    @abstractmethod
+    def model(self, model_identifier: str) -> t.Any:
+        """returns entry for the  for available models"""
+        pass
+
+    @abstractmethod
+    def modelConfigPath(self, model_identifier: str) -> str:
+        """returns modelConfigPath for for a given model-identifier"""
+        pass
+
+    @abstractmethod
+    def download(self, model_identifier: str):
+        pass
+
+
+class TranslateLocallyLike(Repository):
+    """
+    This class implements Repository to fetch models from translateLocally.
+    AppDirs is used to standardize directories and further specialization
+    happens with translateLocally identifier.
+    """
+
+    def __init__(self, name, url):
+        self.url = url
+        self._name = name
+        appDir = AppDirs(APP)
+        f = lambda *args: os.path.join(*args, self._name)
+        self.dirs = {
+            "cache": f(appDir.user_cache_dir),
+            "config": f(appDir.user_config_dir),
+            "data": f(appDir.user_data_dir),
+            "archive": f(appDir.user_data_dir, "archives"),
+            "models": f(appDir.user_data_dir, "models"),
+        }
+
+        for directory in self.dirs.values():
+            os.makedirs(directory, exist_ok=True)
+
+        self.models_file_path = os.path.join(self.dirs["config"], "models.json")
+        self.update()
+
+    @property
+    def name(self) -> str:
+        return self._name
+
+    def update(self) -> None:
+        inventory = requests.get(self.url).text
+        with open(self.models_file_path, "w+") as models_file:
+            models_file.write(inventory)
+        self.data = json.loads(inventory)
+
+        # Update inverse lookup.
+        self.data_by_code = {}
+        for model in self.data["models"]:
+            self.data_by_code[model["code"]] = model
+
+    def models(self, filter_downloaded: bool = True) -> t.List[str]:
+        codes = []
+        for model in self.data["models"]:
+            if filter_downloaded:
+                fprefix = self._archive_name_without_extension(model["url"])
+                model_dir = os.path.join(self.dirs["models"], fprefix)
+                if os.path.exists(model_dir):
+                    codes.append(model["code"])
+            else:
+                codes.append(model["code"])
+        return codes
+
+    def modelConfigPath(self, model_identifier: str) -> str:
+        model = self.model(model_identifier)
+        fprefix = self._archive_name_without_extension(model["url"])
+        model_dir = os.path.join(self.dirs["models"], fprefix)
+        return os.path.join(model_dir, "config.bergamot.yml")
+
+    def model(self, model_identifier: str) -> t.Any:
+        return self.data_by_code[model_identifier]
+
+    def download(self, model_identifier: str):
+        # Download path
+        model = self.model(model_identifier)
+        model_archive = "{}.tar.gz".format(model["shortName"])
+        save_location = os.path.join(self.dirs["archive"], model_archive)
+        download_resource(model["url"], save_location)
+
+        with tarfile.open(save_location) as model_archive:
+            model_archive.extractall(self.dirs["models"])
+            fprefix = self._archive_name_without_extension(model["url"])
+            model_dir = os.path.join(self.dirs["models"], fprefix)
+            symlink = os.path.join(self.dirs["models"], model["code"])
+
+            print(
+                "Downloading and extracting {} into ... {}".format(
+                    model["code"], model_dir
+                ),
+                end=" ",
+            )
+
+            if not os.path.exists(symlink):
+                os.symlink(model_dir, symlink)
+
+            config_path = os.path.join(symlink, "config.intgemm8bitalpha.yml")
+            bergamot_config_path = os.path.join(symlink, "config.bergamot.yml")
+
+            # Finally patch so we don't have to reload this again.
+            patch_marian_for_bergamot(config_path, bergamot_config_path)
+
+            print("Done.")
+
+    def _archive_name_without_extension(self, url: URL):
+        o = urlparse(url)
+        fname = os.path.basename(o.path)  # something tar.gz.
+        fname_without_extension = fname.replace(".tar.gz", "")
+        return fname_without_extension
+
+
+class Aggregator:
+    def __init__(self, repositories: t.List[Repository]):
+        self.repositories = {}
+        for repository in repositories:
+            if repository.name in self.repositories:
+                raise ValueError("Duplicate repository found.")
+            self.repositories[repository.name] = repository
+
+        # Default is self.repostiory
+        self.default_repository = repositories[0]
+
+    def update(self, name: str) -> None:
+        self.repositories.get(name, self.default_repository).update()
+
+    def modelConfigPath(self, name: str, code: str) -> PathLike:
+        return self.repositories.get(name, self.default_repository).modelConfigPath(
+            code
+        )
+
+    def models(self, name: str, filter_downloaded: bool = True) -> t.List[str]:
+        return self.repositories.get(name, self.default_repository).models()
+
+    def model(self, name: str, model_identifier: str) -> t.Any:
+        return self.repositories.get(name, self.default_repository).model(
+            model_identifier
+        )
+
+    def available(self):
+        return list(self.repositories.keys())
+
+    def download(self, name: str, model_identifier: str) -> None:
+        self.repositories.get(name, self.default_repository).download(model_identifier)
diff --git a/bindings/python/typing_utils.py b/bindings/python/typing_utils.py
new file mode 100644
index 000000000..3e1682cff
--- /dev/null
+++ b/bindings/python/typing_utils.py
@@ -0,0 +1,5 @@
+import pathlib
+import typing as t
+
+PathLike = t.TypeVar("PathLike", str, pathlib.Path)
+URL = str
diff --git a/bindings/python/utils.py b/bindings/python/utils.py
new file mode 100644
index 000000000..3164c171c
--- /dev/null
+++ b/bindings/python/utils.py
@@ -0,0 +1,52 @@
+import os
+
+import requests
+import yaml
+
+from .typing_utils import URL, PathLike
+
+
+def download_resource(url: URL, save_location: PathLike, force_download=False):
+    """
+    Downloads a resource from url into save_location, overwrites only if
+    force_download is true.
+    """
+    if force_download or not os.path.exists(save_location):
+        response = requests.get(url, stream=True)
+        # Throw an error for bad status codes
+        response.raise_for_status()
+        with open(save_location, "wb") as handle:
+            for block in response.iter_content(1024):
+                handle.write(block)
+
+
+def patch_marian_for_bergamot(
+    marian_config_path: PathLike, bergamot_config_path: PathLike, quality: bool = False
+):
+    """
+    Accepts path to a config-file from marian-training and followign
+    quantization and adjusts parameters for use in bergamot.
+    """
+    # Load marian_config_path
+    data = None
+    with open(marian_config_path) as fp:
+        data = yaml.load(fp, Loader=yaml.FullLoader)
+
+    # Update a few entries. Things here are hardcode.
+    data.update(
+        {
+            "ssplit-prefix-file": "",
+            "ssplit-mode": "paragraph",
+            "max-length-break": 128,
+            "mini-batch-words": 1024,
+            "workspace": 128,  # shipped models use big workspaces. We'd prefer to keep it low.
+            "alignment": "soft",
+        }
+    )
+
+    if quality:
+        data.update({"quality": quality, "skip-cost": False})
+
+    # Write-out.
+    with open(bergamot_config_path, "w") as output_file:
+        print(yaml.dump(data, sort_keys=False), file=output_file)
diff --git a/doc/conf.py b/doc/conf.py
index 8a8f4224c..a86f4cbea 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -6,29 +6,32 @@
 
 # -- Path setup --------------------------------------------------------------
 
+import datetime
+
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
 import os
-import datetime
 import sys
 
-sys.path.insert(0, os.path.abspath('.'))
+sys.path.insert(0, os.path.abspath("."))
 
 
 # -- Project information -----------------------------------------------------
 
-project = 'Bergamot Translator'
-copyright = '2021, Bergamot Translator Team'
-author = 'Bergamot Translator Team'
+project = "Bergamot Translator"
+copyright = "2021-2022 Bergamot Translator Team"
+author = "Bergamot Translator Team"
 
 # The full version, including alpha/beta/rc tags
 # TODO: add GitHub commit hash to the version
-version_file = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'BERGAMOT_VERSION')
+version_file = os.path.join(
+    os.path.dirname(os.path.dirname(__file__)), "BERGAMOT_VERSION"
+)
 with open(os.path.abspath(version_file)) as f:
     version = f.read().strip()
-release = version + ' ' + str(datetime.date.today())
+release = version + " " + str(datetime.date.today())
 
 
 # -- General configuration ---------------------------------------------------
@@ -37,24 +40,26 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.mathjax',
-    'sphinx.ext.todo',
-    'breathe',
-    'exhale',
-    'recommonmark',
+    "sphinx.ext.mathjax",
+    "sphinx.ext.todo",
+    "breathe",
+    "exhale",
+    "recommonmark",
+    "sphinx.ext.autodoc",
+    "sphinxarg.ext",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = [
-    'build',
-    'doxygen',
-    'venv',
-    'README.md',
+    "build",
+    "doxygen",
+    "venv",
+    "README.md",
 ]
 
 
@@ -63,23 +68,23 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'sphinx_rtd_theme'
-htmlhelp_basename = 'bergamot-translator'
+html_theme = "sphinx_rtd_theme"
+htmlhelp_basename = "bergamot-translator"
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
-html_css_files = ['css/custom.css']
+html_static_path = ["_static"]
+html_css_files = ["css/custom.css"]
 
 # The base URL which points to the root of the HTML documentation
-html_baseurl = 'http://jerinphilip.github.io/bergamot-translator'
+html_baseurl = "https://browser.mt/docs"
 
 
 # -- Extension configuration -------------------------------------------------
 
-breathe_projects = { 'bergamot-translator': './doxygen/xml' }
-breathe_default_project = 'bergamot-translator'
+breathe_projects = {"bergamot-translator": "./doxygen/xml"}
+breathe_default_project = "bergamot-translator"
 
 doxygen_config = """
 INPUT                = ../src ../app
@@ -94,27 +99,28 @@
 """
 
 exhale_args = {
-    'containmentFolder'     : './api',
-    'rootFileName'          : 'library_index.rst',
-    'rootFileTitle'         : 'Library API',
-    'doxygenStripFromPath'  : '..',
-    'createTreeView'        : True,
-    'exhaleExecutesDoxygen' : True,
-    'exhaleDoxygenStdin'    : doxygen_config.strip(),
+    "containmentFolder": "./api",
+    "rootFileName": "library_index.rst",
+    "rootFileTitle": "Library API",
+    "doxygenStripFromPath": "..",
+    "createTreeView": True,
+    "exhaleExecutesDoxygen": True,
+    "exhaleDoxygenStdin": doxygen_config.strip(),
 }
 
-primary_domain = 'cpp'
-highlight_language = 'cpp'
+primary_domain = "cpp"
+highlight_language = "cpp"
 
 # A trick to include markdown files from outside the source directory using
 # 'mdinclude'. Warning: all other markdown files not included via 'mdinclude'
 # will be rendered using recommonmark as recommended by Sphinx
 from m2r import MdInclude
 
+
 def setup(app):
     # from m2r to make `mdinclude` work
-    app.add_config_value('no_underscore_emphasis', False, 'env')
-    app.add_config_value('m2r_parse_relative_links', False, 'env')
-    app.add_config_value('m2r_anonymous_references', False, 'env')
-    app.add_config_value('m2r_disable_inline_math', False, 'env')
-    app.add_directive('mdinclude', MdInclude)
+    app.add_config_value("no_underscore_emphasis", False, "env")
+    app.add_config_value("m2r_parse_relative_links", False, "env")
+    app.add_config_value("m2r_anonymous_references", False, "env")
+    app.add_config_value("m2r_disable_inline_math", False, "env")
+    app.add_directive("mdinclude", MdInclude)
diff --git a/doc/index.rst b/doc/index.rst
index 5be3857a3..54dc1e8dc 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -17,6 +17,7 @@ This is developer documentation.
    marian-integration
    wasm-example
    api/library_index
+   python
 
 
 
diff --git a/doc/python.rst b/doc/python.rst
new file mode 100644
index 000000000..0426f349f
--- /dev/null
+++ b/doc/python.rst
@@ -0,0 +1,87 @@
+.. Bergamot documentation master file, created by
+   sphinx-quickstart on Tue Jan 18 17:26:57 2022.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Python
+=======
+
+.. toctree::
+   :maxdepth: 3
+   :caption: Contents:
+
+
+This document describes python bindings from bergamot-translator and a
+batteries included python package supplied for easy use. The library also
+provides entry point via a command-line making it easier for the average user
+to get started.
+
+As bergamot-translator is built on top of marian, the python API should also
+work as python bindings for marian trained models, if they need to be
+integrated into python code-bases.
+
+*Disclaimer*: The package is still in early stages and unstable. Functions and
+classes might move around quite fast. Use at your own risk.
+
+Command Line Interface
+----------------------
+
+.. argparse::
+   :ref: bergamot.cmds.make_parser
+   :prog: bergamot
+
+
+Module Documentation
+--------------------
+
+.. automodule:: bergamot
+   :members:
+   :undoc-members:
+ 
+bergamot-translator
++++++++++++++++++++
+
+The following components are exported from C++ via python-bindings and form
+library primitives that can be used to build translation workflows.
+
+.. autoclass:: bergamot.ServiceConfig
+   :members:
+   :undoc-members:
+
+.. autoclass:: bergamot.Service
+   :members:
+   :undoc-members:
+
+
+.. autoclass:: bergamot.TranslationModel
+   :members:
+   :undoc-members:
+
+.. autoclass:: bergamot.ResponseOptions
+   :members:
+   :undoc-members:
+
+Model Inventory
++++++++++++++++
+
+.. autoclass:: bergamot.repository.Repository
+   :members:
+   :undoc-members:
+
+.. autoclass:: bergamot.repository.TranslateLocallyLike
+   :members:
+   :undoc-members:
+
+Utilities
++++++++++
+
+.. autofunction:: bergamot.utils.patch_marian_for_bergamot
+
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
diff --git a/doc/requirements.txt b/doc/requirements.txt
index 28e6e70ca..d95cc684c 100644
--- a/doc/requirements.txt
+++ b/doc/requirements.txt
@@ -5,3 +5,4 @@ sphinx_rtd_theme
 mistune<2.0.0
 recommonmark
 m2r
+sphinx-argparse
diff --git a/patches/01-marian-fstream-for-macos.patch b/patches/01-marian-fstream-for-macos.patch
new file mode 100644
index 000000000..5219227d9
--- /dev/null
+++ b/patches/01-marian-fstream-for-macos.patch
@@ -0,0 +1,13 @@
+diff --git a/3rd_party/marian-dev/src/3rd_party/zstr/strict_fstream.hpp b/3rd_party/marian-dev/src/3rd_party/zstr/strict_fstream.hpp
+index 7b1173931df977e69021f3995fa064a492f89d38..948e91eaf99b6b29ce41cf793fba6717f3b5f5b5 100644
+--- a/3rd_party/marian-dev/src/3rd_party/zstr/strict_fstream.hpp
++++ b/3rd_party/marian-dev/src/3rd_party/zstr/strict_fstream.hpp
+@@ -27,7 +27,7 @@ static std::string strerror()
+     {
+         buff = "Unknown error";
+     }
+-#elif (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600 || __APPLE__) && ! _GNU_SOURCE
++#elif (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600 || __APPLE__)
+ // XSI-compliant strerror_r()
+     if (strerror_r(errno, &buff[0], buff.size()) != 0)
+     {
diff --git a/setup.py b/setup.py
new file mode 100644
index 000000000..85fa685ff
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,210 @@
+import io
+import os
+import re
+import subprocess
+import sys
+
+from setuptools import Command, Extension, find_packages, setup
+from setuptools.command.build_ext import build_ext
+from setuptools.command.build_py import build_py as _build_py
+
+# Convert distutils Windows platform specifiers to CMake -A arguments
+PLAT_TO_CMAKE = {
+    "win32": "Win32",
+    "win-amd64": "x64",
+    "win-arm32": "ARM",
+    "win-arm64": "ARM64",
+}
+
+# A CMakeExtension needs a sourcedir instead of a file list.
+# The name must be the _single_ output extension from the CMake build.
+# If you need multiple extensions, see scikit-build.
+class CMakeExtension(Extension):
+    def __init__(self, name, sourcedir=""):
+        Extension.__init__(self, name, sources=[])
+        self.sourcedir = os.path.abspath(sourcedir)
+
+
+class CMakeBuild(build_ext):
+    def build_extension(self, ext):
+        extdir = os.path.abspath(os.path.dirname(self.get_ext_fullpath(ext.name)))
+
+        # required for auto-detection & inclusion of auxiliary "native" libs
+        if not extdir.endswith(os.path.sep):
+            extdir += os.path.sep
+
+        debug = int(os.environ.get("DEBUG", 0)) if self.debug is None else self.debug
+        cfg = "Debug" if debug else "Release"
+
+        # CMake lets you override the generator - we need to check this.
+        # Can be set with Conda-Build, for example.
+        cmake_generator = os.environ.get("CMAKE_GENERATOR", "")
+        build_arch = os.environ.get("BUILD_ARCH", "native")
+
+        # Set Python_EXECUTABLE instead if you use PYBIND11_FINDPYTHON
+        # EXAMPLE_VERSION_INFO shows you how to pass a value into the C++ code
+        # from Python.
+        cmake_args = [
+            f"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY={extdir}",
+            f"-DPYTHON_EXECUTABLE={sys.executable}",
+            f"-DCMAKE_BUILD_TYPE={cfg}",  # not used on MSVC, but no harm
+            f"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
+            f"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
+            f"-DCOMPILE_PYTHON=ON",
+            f"-DSSPLIT_USE_INTERNAL_PCRE2=ON",
+            f"-DBUILD_ARCH={build_arch}",
+        ]
+        build_args = ["-t", "_bergamot"]
+        # Adding CMake arguments set as environment variable
+        # (needed e.g. to build for ARM OSx on conda-forge)
+        if "CMAKE_ARGS" in os.environ:
+            cmake_args += [item for item in os.environ["CMAKE_ARGS"].split(" ") if item]
+
+        # In this example, we pass in the version to C++. You might not need to.
+        cmake_args += [f"-DEXAMPLE_VERSION_INFO={self.distribution.get_version()}"]
+
+        if self.compiler.compiler_type != "msvc":
+            # Using Ninja-build since it a) is available as a wheel and b)
+            # multithreads automatically. MSVC would require all variables be
+            # exported for Ninja to pick it up, which is a little tricky to do.
+            # Users can override the generator with CMAKE_GENERATOR in CMake
+            # 3.15+.
+            if not cmake_generator:
+                try:
+                    import ninja  # noqa: F401
+
+                    cmake_args += ["-GNinja"]
+                except ImportError:
+                    pass
+
+        else:
+
+            # Single config generators are handled "normally"
+            single_config = any(x in cmake_generator for x in {"NMake", "Ninja"})
+
+            # CMake allows an arch-in-generator style for backward compatibility
+            contains_arch = any(x in cmake_generator for x in {"ARM", "Win64"})
+
+            # Specify the arch if using MSVC generator, but only if it doesn't
+            # contain a backward-compatibility arch spec already in the
+            # generator name.
+            if not single_config and not contains_arch:
+                cmake_args += ["-A", PLAT_TO_CMAKE[self.plat_name]]
+
+            # Multi-config generators have a different way to specify configs
+            if not single_config:
+                cmake_args += [
+                    f"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{cfg.upper()}={extdir}"
+                ]
+                build_args += ["--config", cfg]
+
+        if sys.platform.startswith("darwin"):
+            # Cross-compile support for macOS - respect ARCHFLAGS if set
+            archs = re.findall(r"-arch (\S+)", os.environ.get("ARCHFLAGS", ""))
+            if archs:
+                cmake_args += ["-DCMAKE_OSX_ARCHITECTURES={}".format(";".join(archs))]
+
+        # Set CMAKE_BUILD_PARALLEL_LEVEL to control the parallel build level
+        # across all generators.
+        if "CMAKE_BUILD_PARALLEL_LEVEL" not in os.environ:
+            # self.parallel is a Python 3 only way to set parallel jobs by hand
+            # using -j in the build_ext call, not supported by pip or PyPA-build.
+            if hasattr(self, "parallel") and self.parallel:
+                # CMake 3.12+ only.
+                build_args += [f"-j{self.parallel}"]
+
+        if not os.path.exists(self.build_temp):
+            os.makedirs(self.build_temp)
+
+        print("cmake", ext.sourcedir, " ".join(cmake_args))
+
+        subprocess.check_call(
+            ["cmake", ext.sourcedir] + cmake_args, cwd=self.build_temp
+        )
+        subprocess.check_call(
+            ["cmake", "--build", "."] + build_args, cwd=self.build_temp
+        )
+
+
+here = os.path.abspath(os.path.dirname(__file__))
+
+# Import the README and use it as the long-description.
+# Note: this will only work if 'README.md' is present in your MANIFEST.in file!
+with io.open(os.path.join(here, "README.md"), encoding="utf-8") as f:
+    long_description = "\n" + f.read()
+
+version = None
+with open(os.path.join(here, "BERGAMOT_VERSION")) as f:
+    version = f.read().strip()
+    suffix = os.environ.get("PYTHON_LOCAL_VERSION_IDENTIFIER", None)
+    if suffix is not None:
+        version = "{}+{}".format(version, suffix)
+
+
+class UploadCommand(Command):
+    """Support setup.py upload."""
+
+    description = "Build and publish the package."
+    user_options = []
+
+    @staticmethod
+    def status(s):
+        """Prints things in bold."""
+        print("\033[1m{0}\033[0m".format(s))
+
+    def initialize_options(self):
+        pass
+
+    def finalize_options(self):
+        pass
+
+    def run(self):
+        try:
+            self.status("Removing previous builds…")
+            rmtree(os.path.join(here, "dist"))
+        except OSError:
+            pass
+
+        self.status("Building Source and Wheel (universal) distribution…")
+        os.system("{0} setup.py sdist bdist_wheel --universal".format(sys.executable))
+
+        self.status("Pushing git tags…")
+        os.system("git push --tags")
+
+        self.status("Uploading the package to PyPI via Twine…")
+        os.system("twine upload dist/*")
+
+        sys.exit()
+
+
+class build_py(_build_py):
+    def run(self):
+        self.run_command("build_ext")
+        return super().run()
+
+
+# The information here can also be placed in setup.cfg - better separation of
+# logic and declaration, and simpler if you include description/version in a file.
+setup(
+    name="bergamot",
+    version=version,
+    author="Jerin Philip",
+    author_email="jerinphilip@live.in",
+    url="https://github.com/browsermt/bergamot-translator/",
+    description="Bergamot translator python binding.",
+    long_description="",
+    ext_modules=[CMakeExtension("bergamot/_bergamot")],
+    cmdclass={"build_py": build_py, "build_ext": CMakeBuild},
+    zip_safe=False,
+    extras_require={"test": ["pytest>=6.0"]},
+    license_files=("LICENSE",),
+    python_requires=">=3.6",
+    packages=["bergamot"],
+    package_dir={"bergamot": "bindings/python"},
+    install_requires=["requests", "pyyaml", "appdirs"],
+    entry_points={
+        "console_scripts": [
+            "bergamot = bergamot.__main__:main",
+        ],
+    },
+)

From cfdda155e2106ad12dc697cfc8631ded6d22ee44 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Mon, 31 Jan 2022 19:03:17 +0000
Subject: [PATCH 340/442] BRT: Update to fix QE download failures (#321)

---
 bergamot-translator-tests | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index e49686b7c..f5e9eb15a 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit e49686b7cabcc55c0a1fe3dae4cfe59ec146d68e
+Subproject commit f5e9eb15a0323e33a80d18b4a9234c93242fd399

From 95de806d1d1a2f06f04f866a167a5991f72573cc Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Tue, 1 Feb 2022 13:31:11 +0000
Subject: [PATCH 341/442] Fix HTML with pivoting (#323)

Previously BlockingService pivoting missed preproc and postproc for HTML
leading to issues in WebAssembly API. This change adds fixes for the
same, along with test coverage for the functionality over both async and
blocking services.
---
 bergamot-translator-tests  |  2 +-
 src/tests/common-impl.cpp  | 15 +++++++++++++++
 src/tests/common.h         |  2 ++
 src/translator/service.cpp |  9 +++++++++
 4 files changed, 27 insertions(+), 1 deletion(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index f5e9eb15a..aaf315c80 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit f5e9eb15a0323e33a80d18b4a9234c93242fd399
+Subproject commit aaf315c80b7339c2ae6fcca080ca30d3f0a5e2f5
diff --git a/src/tests/common-impl.cpp b/src/tests/common-impl.cpp
index 8fdc74fb6..238e3acd6 100644
--- a/src/tests/common-impl.cpp
+++ b/src/tests/common-impl.cpp
@@ -71,6 +71,8 @@ void TestSuite<Service>::TestSuite::run(const std::string &opModeAsString, std::
     translationCache(models.front());
   } else if (opModeAsString == "test-pivot") {
     pivotTranslate(models);
+  } else if (opModeAsString == "test-pivot-with-html") {
+    pivotTranslateWithHTML(models);
   } else if (opModeAsString == "test-html-translation") {
     htmlTranslation(models.front());
   } else {
@@ -226,6 +228,19 @@ void TestSuite<Service>::translationCache(Ptr<TranslationModel> model) {
   std::cout << firstResponse.target.text;
 }
 
+template <class Service>
+void TestSuite<Service>::pivotTranslateWithHTML(std::vector<Ptr<TranslationModel>> &models) {
+  ABORT_IF(models.size() != 2, "Forward and backward test needs two models.");
+  ResponseOptions responseOptions;
+  responseOptions.HTML = true;
+  std::string source = readFromStdin();
+  std::promise<Response> responsePromise;
+  std::future<Response> responseFuture = responsePromise.get_future();
+  Response response = bridge_.pivot(service_, models.front(), models.back(), std::move(source), responseOptions);
+  std::cout << response.source.text;
+  std::cout << response.target.text;
+}
+
 template <class Service>
 void TestSuite<Service>::pivotTranslate(std::vector<Ptr<TranslationModel>> &models) {
   // We expect a source -> pivot; pivot -> source model to get source -> source and build this test using accuracy of
diff --git a/src/tests/common.h b/src/tests/common.h
index f84121648..238a62357 100644
--- a/src/tests/common.h
+++ b/src/tests/common.h
@@ -88,6 +88,8 @@ class TestSuite {
 
   void pivotTranslate(std::vector<Ptr<TranslationModel>> &models);
 
+  void pivotTranslateWithHTML(std::vector<Ptr<TranslationModel>> &models);
+
   void htmlTranslation(Ptr<TranslationModel> model);
 };
 
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 380260529..6ac1fd100 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -81,6 +81,11 @@ std::vector<Response> BlockingService::pivotMultiple(std::shared_ptr<Translation
                                                      std::shared_ptr<TranslationModel> second,
                                                      std::vector<std::string> &&sources,
                                                      const ResponseOptions &responseOptions) {
+  std::vector<HTML> htmls;
+  for (auto &&source : sources) {
+    htmls.emplace_back(std::move(source), responseOptions.HTML);
+  }
+
   // Translate source to pivots. This is same as calling translateMultiple.
   std::vector<Response> sourcesToPivots;
   sourcesToPivots = translateMultipleRaw(first, std::move(sources), responseOptions);
@@ -115,6 +120,10 @@ std::vector<Response> BlockingService::pivotMultiple(std::shared_ptr<Translation
     finalResponses.push_back(std::move(finalResponse));
   }
 
+  for (size_t i = 0; i < finalResponses.size(); i++) {
+    htmls[i].restore(finalResponses[i]);
+  }
+
   return finalResponses;
 }
 

From 19ae519c63f79c1ea5478f5ece952c8cee36d905 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Wed, 2 Feb 2022 12:36:30 +0000
Subject: [PATCH 342/442] Remove obsolete workflow transferring source across
 forks (#326)

---
 .../push-browsermt-main-to-mozilla-main.yml   | 27 -------------------
 1 file changed, 27 deletions(-)
 delete mode 100644 .github/workflows/push-browsermt-main-to-mozilla-main.yml

diff --git a/.github/workflows/push-browsermt-main-to-mozilla-main.yml b/.github/workflows/push-browsermt-main-to-mozilla-main.yml
deleted file mode 100644
index cf02fe3e9..000000000
--- a/.github/workflows/push-browsermt-main-to-mozilla-main.yml
+++ /dev/null
@@ -1,27 +0,0 @@
-name: Push browsermt/main branch to mozilla/bergamot-translator
-
-on:
-  schedule:
-  # Run every hour
-  - cron: "0 * * * *"
-
-  workflow_dispatch:
-
-jobs:
-  build:
-    runs-on: ubuntu-latest
-    name: Mirror a branch from a remote repo into this one
-    env:
-      source_repository: 'browsermt/bergamot-translator'
-      this_repository: 'mozilla/bergamot-translator'
-      branch: 'main'
-    steps:
-      - uses: actions/checkout@v2
-
-      - name: Mirror a branch from a remote repo into this one
-        if: ${{ github.repository == env.this_repository }}
-        run: |
-          git clone -b ${{ env.branch }} https://github.com/${{ env.source_repository }}.git source
-          cd source
-          git remote add mirror https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ env.this_repository }}.git
-          git push mirror ${{ env.branch }}

From d95b014562f7c9075c210a825250de086c492b0d Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 2 Feb 2022 17:01:23 +0100
Subject: [PATCH 343/442] Wasm/JS: Pivot translation API JS binding and test
 page update (#327)

---
 wasm/bindings/service_bindings.cpp |   3 +-
 wasm/test_page/js/worker.js        | 155 +++++++++++++++--------------
 2 files changed, 81 insertions(+), 77 deletions(-)

diff --git a/wasm/bindings/service_bindings.cpp b/wasm/bindings/service_bindings.cpp
index d05cf57cf..ee21aa879 100644
--- a/wasm/bindings/service_bindings.cpp
+++ b/wasm/bindings/service_bindings.cpp
@@ -78,7 +78,8 @@ EMSCRIPTEN_BINDINGS(blocking_service_config) {
 EMSCRIPTEN_BINDINGS(blocking_service) {
   class_<BlockingService>("BlockingService")
       .constructor<BlockingService::Config>()
-      .function("translate", &BlockingService::translateMultiple);
+      .function("translate", &BlockingService::translateMultiple)
+      .function("translateViaPivoting", &BlockingService::pivotMultiple);
 
   register_vector<std::string>("VectorString");
 }
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index 3bc89bef5..d079c783f 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -1,10 +1,12 @@
 // All variables specific to translation service
 var translationService = undefined;
+
 // A map of language-pair to TranslationModel object
 var languagePairToTranslationModels = new Map();
 
 const BERGAMOT_TRANSLATOR_MODULE = "bergamot-translator-worker.js";
 const MODEL_REGISTRY = "modelRegistry.js";
+const PIVOT_LANGUAGE = 'en';
 
 const encoder = new TextEncoder(); // string to utf-8 converter
 const decoder = new TextDecoder(); // utf-8 to string converter
@@ -82,7 +84,7 @@ const constructTranslationService = async () => {
   }
 }
 
-// Constructs a translation model object for the source and target language pair
+// Constructs translation model for the source and target language pair.
 const constructTranslationModel = async (from, to) => {
   // Delete all previously constructed translation models and clear the map
   languagePairToTranslationModels.forEach((value, key) => {
@@ -91,32 +93,61 @@ const constructTranslationModel = async (from, to) => {
   });
   languagePairToTranslationModels.clear();
 
-  // If none of the languages is English then construct multiple models with
-  // English as a pivot language.
-  if (from !== 'en' && to !== 'en') {
-    log(`Constructing model '${from}${to}' via pivoting: '${from}en' and 'en${to}'`);
-    await Promise.all([_constructTranslationModelInvolvingEnglish(from, 'en'),
-                        _constructTranslationModelInvolvingEnglish('en', to)]);
+  const languagePairs = _getLanguagePairs(from, to);
+  log(`Constructing translation model(s): ${languagePairs.toString()}`);
+  if (languagePairs.length == 2) {
+    // This implies pivoting is required => Construct 2 translation models
+    await Promise.all([_constructTranslationModelHelper(languagePairs[0]),
+                      _constructTranslationModelHelper(languagePairs[1])]);
   }
   else {
-    log(`Constructing model '${from}${to}'`);
-    await _constructTranslationModelInvolvingEnglish(from, to);
+    // This implies pivoting is not required => Construct 1 translation model
+    await _constructTranslationModelHelper(languagePairs[0]);
   }
 }
 
-// Translates text from source language to target language.
+// Translates text from source language to target language (via pivoting if necessary).
 const translate = (from, to, input) => {
-  // If none of the languages is English then perform translation with
-  // English as a pivot language.
-  if (from !== 'en' && to !== 'en') {
-    log(`Translating '${from}${to}' via pivoting: '${from}en' -> 'en${to}'`);
-    const translatedTextInEnglish = _translateInvolvingEnglish(from, 'en', input);
-    return _translateInvolvingEnglish('en', to, translatedTextInEnglish);
+  const languagePairs = _getLanguagePairs(from, to);
+  log(`Translating for language pair(s): '${languagePairs.toString()}'`);
+
+  // Each language pair requires a corresponding loaded translation model. Otherwise, it's an error.
+  let translationModels = _getLoadedTranslationModels(from, to);
+  if (translationModels.length != languagePairs.length) {
+    throw Error(`Insufficient no. of loaded translation models. Required:'${languagePairs.length}' Found:'${translationModels.length}'`);
+  }
+
+  // Prepare the arguments (ResponseOptions and vectorSourceText (vector<string>)) of Translation API and call it.
+  // Result is a vector<Response> where each of its item corresponds to one item of vectorSourceText in the same order.
+  const responseOptions = _prepareResponseOptions();
+  let vectorSourceText = _prepareSourceText(input);
+  let vectorResponse;
+  if (translationModels.length == 2) {
+    // This implies translation should be done via pivoting
+    vectorResponse = translationService.translateViaPivoting(translationModels[0], translationModels[1], vectorSourceText, responseOptions);
   }
   else {
-    log(`Translating '${from}${to}'`);
-    return _translateInvolvingEnglish(from, to, input);
+    // This implies translation should be done without pivoting
+    vectorResponse = translationService.translate(translationModels[0], vectorSourceText, responseOptions);
   }
+
+  // Parse all relevant information from vectorResponse
+  const listTranslatedText = _parseTranslatedText(vectorResponse);
+  const listSourceText = _parseSourceText(vectorResponse);
+  const listTranslatedTextSentences = _parseTranslatedTextSentences(vectorResponse);
+  const listSourceTextSentences = _parseSourceTextSentences(vectorResponse);
+  const listTranslatedTextSentenceQualityScores = _parseTranslatedTextSentenceQualityScores(vectorResponse);
+
+  log(`Source text: ${listSourceText}`);
+  log(`Translated text: ${listTranslatedText}`);
+  log(`Translated sentences: ${JSON.stringify(listTranslatedTextSentences)}`);
+  log(`Source sentences: ${JSON.stringify(listSourceTextSentences)}`);
+  log(`Translated sentence quality scores: ${JSON.stringify(listTranslatedTextSentenceQualityScores)}`);
+
+  // Delete prepared SourceText to avoid memory leak
+  vectorSourceText.delete();
+
+  return listTranslatedText;
 }
 
 // Downloads file from a url and returns the array buffer
@@ -140,38 +171,13 @@ const _prepareAlignedMemoryFromBuffer = async (buffer, alignmentSize) => {
   return alignedMemory;
 }
 
-const _constructTranslationModelInvolvingEnglish = async (from, to) => {
-  const languagePair = `${from}${to}`;
+const _constructTranslationModelHelper = async (languagePair) => {
 
   /*Set the Model Configuration as YAML formatted string.
     For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-    Vocab files are re-used in both translation directions
-    const vocabLanguagePair = from === "en" ? `${to}${from}` : languagePair;
-    const modelConfig = `models:
-      - /${languagePair}/model.${languagePair}.intgemm.alphas.bin
-      vocabs:
-      - /${languagePair}/vocab.${vocabLanguagePair}.spm
-      - /${languagePair}/vocab.${vocabLanguagePair}.spm
-      beam-size: 1
-      normalize: 1.0
-      word-penalty: 0
-      max-length-break: 128
-      mini-batch-words: 1024
-      workspace: 128
-      max-length-factor: 2.0
-      skip-cost: true
-      cpu-threads: 0
-      quiet: true
-      quiet-translation: true
-      shortlist:
-          - /${languagePair}/lex.${languagePair}.s2t
-          - 50
-          - 50
-      `;
-      */
-
-  // TODO: gemm-precision: int8shiftAlphaAll (for the models that support this)
-  // DONOT CHANGE THE SPACES BETWEEN EACH ENTRY OF CONFIG
+    Vocab files are re-used in both translation directions.
+    DO NOT CHANGE THE SPACES BETWEEN EACH ENTRY OF CONFIG
+  */
   const modelConfig = `beam-size: 1
 normalize: 1.0
 word-penalty: 0
@@ -229,38 +235,35 @@ alignment: soft
   languagePairToTranslationModels.set(languagePair, translationModel);
 }
 
-const _translateInvolvingEnglish = (from, to, input) => {
-  const languagePair = `${from}${to}`;
-  if (!languagePairToTranslationModels.has(languagePair)) {
-    throw Error(`Please load translation model '${languagePair}' before translating`);
+const _isPivotingRequired = (lang1, lang2) => {
+  if ((lang1 === PIVOT_LANGUAGE) || (lang2 === PIVOT_LANGUAGE)) {
+    return false;
   }
-  translationModel = languagePairToTranslationModels.get(languagePair);
-
-  // Prepare the arguments of translate() API i.e. ResponseOptions and vectorSourceText (i.e. a vector<string>)
-  const responseOptions = _prepareResponseOptions();
-  let vectorSourceText = _prepareSourceText(input);
-
-  // Call translate() API; result is vector<Response> where every item of vector<Response> corresponds
-  // to an item of vectorSourceText in the same order
-  const vectorResponse = translationService.translate(translationModel, vectorSourceText, responseOptions);
-
-  // Parse all relevant information from vectorResponse
-  const listTranslatedText = _parseTranslatedText(vectorResponse);
-  const listSourceText = _parseSourceText(vectorResponse);
-  const listTranslatedTextSentences = _parseTranslatedTextSentences(vectorResponse);
-  const listSourceTextSentences = _parseSourceTextSentences(vectorResponse);
-  const listTranslatedTextSentenceQualityScores = _parseTranslatedTextSentenceQualityScores(vectorResponse);
-
-  log(`Source text: ${listSourceText}`);
-  log(`Translated text: ${listTranslatedText}`);
-  log(`Translated sentences: ${JSON.stringify(listTranslatedTextSentences)}`);
-  log(`Source sentences: ${JSON.stringify(listSourceTextSentences)}`);
-  log(`Translated sentence quality scores: ${JSON.stringify(listTranslatedTextSentenceQualityScores)}`);
+  return true;
+}
 
-  // Delete prepared SourceText to avoid memory leak
-  vectorSourceText.delete();
+const _getLanguagePairs = (srcLang, tgtLang) => {
+  const languagePairs = [];
+  if (_isPivotingRequired(srcLang, tgtLang)) {
+    // Do not change the push order
+    languagePairs.push(`${srcLang}${PIVOT_LANGUAGE}`);
+    languagePairs.push(`${PIVOT_LANGUAGE}${tgtLang}`);
+  }
+  else {
+    languagePairs.push(`${srcLang}${tgtLang}`);
+  }
+  return languagePairs;
+}
 
-  return listTranslatedText;
+const _getLoadedTranslationModels = (srcLang, tgtLang) => {
+  const languagePairs = _getLanguagePairs(srcLang, tgtLang);
+  const loadedTranslationModels = [];
+  for (const langPair of languagePairs) {
+    if (languagePairToTranslationModels.has(langPair)) {
+      loadedTranslationModels.push(languagePairToTranslationModels.get(langPair));
+    }
+  }
+  return loadedTranslationModels;
 }
 
 const _parseTranslatedText = (vectorResponse) => {

From 91b2e0636d4bf4abdf1975ee6f0474739459617b Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Wed, 2 Feb 2022 19:21:42 +0000
Subject: [PATCH 344/442] emscripten: ccache and artefact upload (#325)

Enables ccache for emscripten. The configuration uses pyiodide for a
reference (https://github.com/pyodide/pyodide/pull/1805).

Two workflows to run on macOS and Ubuntu, reduced to one on Ubuntu. As
emscripten and the target is cross-platform, also macOS runners being
limited - it makes sense to have this removed.

Upload artefact enabled in preparation for a release action to be
scheduled which will upload the bergamot*.wasm and bergamot*.js for
consumption.
---
 .github/workflows/wasm-custom_marian-mac.yml  |  56 --------
 .../workflows/wasm-custom_marian-ubuntu.yml   |  56 --------
 .github/workflows/wasm.yml                    | 123 ++++++++++++++++++
 3 files changed, 123 insertions(+), 112 deletions(-)
 delete mode 100644 .github/workflows/wasm-custom_marian-mac.yml
 delete mode 100644 .github/workflows/wasm-custom_marian-ubuntu.yml
 create mode 100644 .github/workflows/wasm.yml

diff --git a/.github/workflows/wasm-custom_marian-mac.yml b/.github/workflows/wasm-custom_marian-mac.yml
deleted file mode 100644
index a27f6b8de..000000000
--- a/.github/workflows/wasm-custom_marian-mac.yml
+++ /dev/null
@@ -1,56 +0,0 @@
-name: WASM (Custom Marian) MacOS
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ '**' ]
-
-jobs:
-  build-wasm:
-    name: WASM (Custom Marian) MacOS
-    runs-on: macos-10.15
-
-    steps:
-      - name: Setup Emscripten toolchain
-        uses: mymindstorm/setup-emsdk@v9
-        with:
-          version: 2.0.9
-
-      - name: Verify Emscripten setup
-        run: emcc -v
-
-      - name: Checkout
-        uses: actions/checkout@v2
-        with:
-          submodules: recursive
-
-      - name: Configure builds
-        run: |
-          mkdir -p build-wasm
-          cd build-wasm
-          emcmake cmake -DCOMPILE_WASM=on ..
-
-      - name: Compile
-        working-directory: build-wasm
-        run: emmake make -j2
-
-      - name: Instantiate simd wormhole
-        working-directory: build-wasm
-        run: bash ../wasm/patch-artifacts-enable-wormhole.sh
-
-      - name: Import GEMM library from a separate wasm module
-        working-directory: build-wasm
-        run: bash ../wasm/patch-artifacts-import-gemm-module.sh
-
-      - name: Check artifacts
-        working-directory: build-wasm
-        run: |
-          ls -all bergamot*
-          if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
-          then
-            echo "Artifacts Successfully Generated"
-          else
-            echo "Failure: Artifacts Not Present"
-            exit 1
-          fi
diff --git a/.github/workflows/wasm-custom_marian-ubuntu.yml b/.github/workflows/wasm-custom_marian-ubuntu.yml
deleted file mode 100644
index 80d083fb8..000000000
--- a/.github/workflows/wasm-custom_marian-ubuntu.yml
+++ /dev/null
@@ -1,56 +0,0 @@
-name: WASM (Custom Marian) Ubuntu
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ '**' ]
-
-jobs:
-  build-wasm:
-    name: WASM (Custom Marian) Ubuntu
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Setup Emscripten toolchain
-        uses: mymindstorm/setup-emsdk@v9
-        with:
-          version: 2.0.9
-
-      - name: Verify Emscripten setup
-        run: emcc -v
-
-      - name: Checkout
-        uses: actions/checkout@v2
-        with:
-          submodules: recursive
-
-      - name: Configure builds
-        run: |
-          mkdir -p build-wasm
-          cd build-wasm
-          emcmake cmake -DCOMPILE_WASM=on ..
-
-      - name: Compile
-        working-directory: build-wasm
-        run: emmake make -j2
-
-      - name: Instantiate simd wormhole
-        working-directory: build-wasm
-        run: bash ../wasm/patch-artifacts-enable-wormhole.sh
-
-      - name: Import GEMM library from a separate wasm module
-        working-directory: build-wasm
-        run: bash ../wasm/patch-artifacts-import-gemm-module.sh
-
-      - name: Check artifacts
-        working-directory: build-wasm
-        run: |
-          ls -all bergamot*
-          if ls bergamot*.wasm &>/dev/null && ls bergamot*.js &>/dev/null
-          then
-            echo "Artifacts Successfully Generated"
-          else
-            echo "Failure: Artifacts Not Present"
-            exit 1
-          fi
diff --git a/.github/workflows/wasm.yml b/.github/workflows/wasm.yml
new file mode 100644
index 000000000..a8c199bff
--- /dev/null
+++ b/.github/workflows/wasm.yml
@@ -0,0 +1,123 @@
+name: WebAssembly 
+
+on:
+  push:
+    branches: [ main, ci-sandbox ]
+  pull_request:
+    branches: [ '**' ]
+
+env:
+  ccache_basedir: ${{ github.workspace }}
+  ccache_dir: "${{ github.workspace }}/.ccache"
+  ccache_compilercheck: mtime
+  ccache_compress: 'true'
+  ccache_compresslevel: 9
+  ccache_maxsize: 200M
+  ccache_cmake: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
+  emsdk_version: 2.0.9
+
+jobs:
+  build-wasm:
+    name: "emscripten"
+    runs-on: ubuntu-latest
+    steps:
+
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: recursive
+
+      - name: Set ccache environment for emcc
+        run: |
+          echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
+          echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
+          echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
+          echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
+          echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
+          echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
+          # https://emscripten.org/docs/compiling/Building-Projects.html#using-a-compiler-wrapper
+          echo "EM_COMPILER_WRAPPER=ccache" >> $GITHUB_ENV
+          
+      # This need to be run before setup, so ccache build caching doesn't complain.
+      - name: Obtain emsdk sources
+        run: |
+            git clone --depth 1 https://github.com/emscripten-core/emsdk.git
+
+      - name: Cache-op for build-cache through ccache
+        uses: actions/cache@v2
+        with:
+          path: |
+              ${{ env.ccache_dir }}
+              ${{ github.workspace }}/emsdk/ccache/git-emscripten_64bit/
+          key: ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+          restore-keys: |-
+            ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
+            ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}
+            ccache-${{ github.job }}-${{ env.emsdk_version }}
+
+      - name: Setup Emscripten toolchain
+        run: |
+            (cd emsdk && ./emsdk install ${{ env.emsdk_version }} ccache-git-emscripten-64bit)
+            (cd emsdk && ./emsdk activate ${{ env.emsdk_version }} ccache-git-emscripten-64bit)
+            # mtime of this file is checked by ccache, we set it to avoid cache misses.
+            touch -m -d '1 Jan 2021 12:00' emsdk/.emscripten
+
+            # These needs to be done in the activated shell.
+            eval $(./emsdk/emsdk construct_env \
+                | sed 's/export PATH=\(.*\);/echo \1 >> $GITHUB_PATH;/' \
+                | sed 's/export \(.*\);/echo \1 >> $GITHUB_ENV;/' );
+
+            # This looks more permanent than version pinned, so keeping temporarily to avoid failures.
+            echo "${{ github.workspace }}/emsdk/ccache/git-emscripten_64bit/bin" >> $GITHUB_PATH
+
+      - name: Generate ccache_vars for ccache based on machine
+        shell: bash
+        id: ccache_vars
+        run: |-
+          echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
+          echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
+
+      - name: Verify Emscripten setup
+        run: |
+            emcc --version
+            emcmake cmake --version
+            emmake make --version
+
+      - name: Configure builds
+        run: |
+          mkdir -p build-wasm
+          cd build-wasm
+          emcmake cmake -DCOMPILE_WASM=on ..
+
+      - name: ccache prolog
+        run: |-
+          ccache -s # Print current cache stats
+          ccache -z # Zero cache entry
+
+      - name: Compile
+        working-directory: build-wasm
+        run: |
+          emmake make -j2
+
+      - name: ccache epilog
+        run: |
+          ccache -s # Print current cache stats
+
+      - name: Instantiate simd wormhole
+        working-directory: build-wasm
+        run: bash ../wasm/patch-artifacts-enable-wormhole.sh
+
+      - name: Import GEMM library from a separate wasm module
+        working-directory: build-wasm
+        run: bash ../wasm/patch-artifacts-import-gemm-module.sh
+
+      - name: Upload wasm artifact
+        uses: actions/upload-artifact@v2
+        with:
+          name: bergamot-translator-worker
+          if-no-files-found: error
+          path: |
+              ${{github.workspace}}/build-wasm/bergamot-translator-worker.js
+              ${{github.workspace}}/build-wasm/bergamot-translator-worker.wasm
+              ${{github.workspace}}/build-wasm/bergamot-translator-worker.js.bak
+

From 5e78260d52b5a96c3711ed237eccb966a3d9ac23 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerin.philip@research.iiit.ac.in>
Date: Fri, 4 Feb 2022 11:54:30 +0000
Subject: [PATCH 345/442] Consolidate release artefacts (#329)

Brings in the previously wasm.yml into python.yml and the new file is
renamed as build.yml.

python.yml already had a version and pre-release jobs. These jobs
downloaded artefacts from prior ran jobs (python wheel builds). The
newly attached emscripten build now uploads artefacts of a WebAssembly
binary and javascript file which are fed into the release and
pre-release jobs in addition to the existing python builds.
---
 .github/workflows/{python.yml => build.yml} | 167 +++++++++++++++++++-
 .github/workflows/wasm.yml                  | 123 --------------
 2 files changed, 161 insertions(+), 129 deletions(-)
 rename .github/workflows/{python.yml => build.yml} (65%)
 delete mode 100644 .github/workflows/wasm.yml

diff --git a/.github/workflows/python.yml b/.github/workflows/build.yml
similarity index 65%
rename from .github/workflows/python.yml
rename to .github/workflows/build.yml
index 2924061df..2246b3c50 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/build.yml
@@ -1,4 +1,4 @@
-name: "Python Bindings"
+name: "Build"
 'on':
   push:
     branches:
@@ -11,6 +11,7 @@ name: "Python Bindings"
       - '**'
 env:
   qt_version: "6.2.1" # only used by build-macos
+  emsdk_version: 2.0.9 # For use in emscripten build
   ccache_basedir: ${{ github.workspace }}
   ccache_dir: "${{ github.workspace }}/.ccache"
   ccache_compilercheck: content
@@ -230,16 +231,161 @@ jobs:
         with:
             path: ${{github.workspace}}/dist/bergamot-*.whl
 
+    build-wasm:
+      name: "emscripten"
+      runs-on: ubuntu-latest
+      steps:
+
+        - name: Checkout
+          uses: actions/checkout@v2
+          with:
+            submodules: recursive
+
+        - name: Set ccache environment for emcc
+          run: |
+            # We are hardcoding this to mtime instead of env pickup. Rest use content.
+            echo "CCACHE_COMPILER_CHECK=mtime" >> $GITHUB_ENV
+
+            echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
+            echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
+            echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
+            echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
+            echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
+            # https://emscripten.org/docs/compiling/Building-Projects.html#using-a-compiler-wrapper
+            echo "EM_COMPILER_WRAPPER=ccache" >> $GITHUB_ENV
+            
+        # This need to be run before setup, so ccache build caching doesn't complain.
+        - name: Obtain emsdk sources
+          run: |
+              git clone --depth 1 https://github.com/emscripten-core/emsdk.git
+
+        - name: Cache-op for build-cache through ccache
+          uses: actions/cache@v2
+          with:
+            path: |
+                ${{ env.ccache_dir }}
+                ${{ github.workspace }}/emsdk/ccache/git-emscripten_64bit/
+            key: ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+            restore-keys: |-
+              ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
+              ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}
+              ccache-${{ github.job }}-${{ env.emsdk_version }}
+
+        - name: Setup Emscripten toolchain
+          run: |
+              (cd emsdk && ./emsdk install ${{ env.emsdk_version }} ccache-git-emscripten-64bit)
+              (cd emsdk && ./emsdk activate ${{ env.emsdk_version }} ccache-git-emscripten-64bit)
+              # mtime of this file is checked by ccache, we set it to avoid cache misses.
+              touch -m -d '1 Jan 2021 12:00' emsdk/.emscripten
+
+              # These needs to be done in the activated shell.
+              eval $(./emsdk/emsdk construct_env \
+                  | sed 's/export PATH=\(.*\);/echo \1 >> $GITHUB_PATH;/' \
+                  | sed 's/export \(.*\);/echo \1 >> $GITHUB_ENV;/' );
+
+              # This looks more permanent than version pinned, so keeping temporarily to avoid failures.
+              echo "${{ github.workspace }}/emsdk/ccache/git-emscripten_64bit/bin" >> $GITHUB_PATH
+
+        - name: Generate ccache_vars for ccache based on machine
+          shell: bash
+          id: ccache_vars
+          run: |-
+            echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
+            echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
+
+        - name: Verify Emscripten setup
+          run: |
+              emcc --version
+              emcmake cmake --version
+              emmake make --version
+
+        - name: ccache prolog
+          run: |-
+            ccache -s # Print current cache stats
+            ccache -z # Zero cache entry
+
+        # WORMHOLE=off
+        - name: "Configure builds for WORMHOLE=off"
+          run: |
+            mkdir -p build-wasm-without-wormhole
+            cd build-wasm-without-wormhole
+            emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=off ..
+
+
+        - name: "Compile with WORMHOLE=off"
+          working-directory: build-wasm-without-wormhole
+          run: |
+            emmake make -j2
+
+        - name: ccache epilog
+          run: |
+            ccache -s # Print current cache stats
+
+        - name: Import GEMM library from a separate wasm module
+          working-directory: build-wasm-without-wormhole
+          run: bash ../wasm/patch-artifacts-import-gemm-module.sh
+
+
+        # WORMHOLE=on
+        - name: "Configure builds for WORMHOLE=on"
+          run: |
+            mkdir -p build-wasm-with-wormhole
+            cd build-wasm-with-wormhole
+            emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=on ..
+
+
+        - name: "Compile with WORMHOLE=on"
+          working-directory: build-wasm-with-wormhole
+          run: |
+            emmake make -j2
+
+        - name: ccache epilog
+          run: |
+            ccache -s # Print current cache stats
+
+        - name: Instantiate simd wormhole
+          working-directory: build-wasm-with-wormhole
+          run: bash ../wasm/patch-artifacts-enable-wormhole.sh
+
+        - name: Import GEMM library from a separate wasm module
+          working-directory: build-wasm-with-wormhole
+          run: bash ../wasm/patch-artifacts-import-gemm-module.sh
+
+        # Rename the wormhole on builds
+        - name: Rename artefacts with wormhole
+          working-directory: build-wasm-with-wormhole
+          run: |
+                mv bergamot-translator-worker{,-with-wormhole}.js
+                mv bergamot-translator-worker{,-with-wormhole}.js.bak
+                mv bergamot-translator-worker{,-with-wormhole}.wasm
+
+
+        # Upload both together.
+        - name: Upload wasm artifact
+          uses: actions/upload-artifact@v2
+          with:
+            name: wasm-artefacts
+            if-no-files-found: error
+            path: |
+                # Without wormhole
+                ${{github.workspace}}/build-wasm-without-wormhole/bergamot-translator-worker.js
+                ${{github.workspace}}/build-wasm-without-wormhole/bergamot-translator-worker.wasm
+                ${{github.workspace}}/build-wasm-without-wormhole/bergamot-translator-worker.js.bak
+
+                ${{github.workspace}}/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.js
+                ${{github.workspace}}/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.wasm
+                ${{github.workspace}}/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.js.bak
+
   # Try to upload a release using https://github.com/marvinpinto/actions/issues/177#issuecomment-917605585 as a model
     release-latest:
       name: Release Latest Build
       runs-on: ubuntu-latest
-      needs: [python-ubuntu, python-macos]
+      needs: [python-ubuntu, python-macos, build-wasm]
       if: github.ref == 'refs/heads/main'
       steps:
        - name: Download artifacts
          uses: actions/download-artifact@v2
-  
+        
        - name: Update GitHub prerelease
          uses: marvinpinto/action-automatic-releases@latest
          with:
@@ -248,12 +394,16 @@ jobs:
            prerelease: true
            title: "Latest Build"
            files: |
-                  ${{github.workspace}}/artifact/*.whl
+                artifact/*.whl
+                wasm-artefacts/build-wasm-without-wormhole/bergamot-translator-worker.js
+                wasm-artefacts/build-wasm-without-wormhole/bergamot-translator-worker.wasm
+                wasm-artefacts/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.js
+                wasm-artefacts/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.wasm
   
     release-version:
       name: Release version 
       runs-on: ubuntu-latest
-      needs: [python-ubuntu, python-macos]
+      needs: [python-ubuntu, python-macos, build-wasm]
       permissions:
         contents: "write"
         packages: "write"
@@ -271,7 +421,12 @@ jobs:
            prerelease: false
            title: "${{ github.ref_name }}"
            files: |
-                  ${{github.workspace}}/artifact/*.whl
+                artifact/*.whl
+                wasm-artefacts/build-wasm-without-wormhole/bergamot-translator-worker.js
+                wasm-artefacts/build-wasm-without-wormhole/bergamot-translator-worker.wasm
+                wasm-artefacts/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.js
+                wasm-artefacts/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.wasm
+
   
     python-checks:
       name: "formatting and typechecks"
diff --git a/.github/workflows/wasm.yml b/.github/workflows/wasm.yml
deleted file mode 100644
index a8c199bff..000000000
--- a/.github/workflows/wasm.yml
+++ /dev/null
@@ -1,123 +0,0 @@
-name: WebAssembly 
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ '**' ]
-
-env:
-  ccache_basedir: ${{ github.workspace }}
-  ccache_dir: "${{ github.workspace }}/.ccache"
-  ccache_compilercheck: mtime
-  ccache_compress: 'true'
-  ccache_compresslevel: 9
-  ccache_maxsize: 200M
-  ccache_cmake: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
-  emsdk_version: 2.0.9
-
-jobs:
-  build-wasm:
-    name: "emscripten"
-    runs-on: ubuntu-latest
-    steps:
-
-      - name: Checkout
-        uses: actions/checkout@v2
-        with:
-          submodules: recursive
-
-      - name: Set ccache environment for emcc
-        run: |
-          echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
-          echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
-          echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
-          echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
-          echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
-          echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
-          # https://emscripten.org/docs/compiling/Building-Projects.html#using-a-compiler-wrapper
-          echo "EM_COMPILER_WRAPPER=ccache" >> $GITHUB_ENV
-          
-      # This need to be run before setup, so ccache build caching doesn't complain.
-      - name: Obtain emsdk sources
-        run: |
-            git clone --depth 1 https://github.com/emscripten-core/emsdk.git
-
-      - name: Cache-op for build-cache through ccache
-        uses: actions/cache@v2
-        with:
-          path: |
-              ${{ env.ccache_dir }}
-              ${{ github.workspace }}/emsdk/ccache/git-emscripten_64bit/
-          key: ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-          restore-keys: |-
-            ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
-            ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}
-            ccache-${{ github.job }}-${{ env.emsdk_version }}
-
-      - name: Setup Emscripten toolchain
-        run: |
-            (cd emsdk && ./emsdk install ${{ env.emsdk_version }} ccache-git-emscripten-64bit)
-            (cd emsdk && ./emsdk activate ${{ env.emsdk_version }} ccache-git-emscripten-64bit)
-            # mtime of this file is checked by ccache, we set it to avoid cache misses.
-            touch -m -d '1 Jan 2021 12:00' emsdk/.emscripten
-
-            # These needs to be done in the activated shell.
-            eval $(./emsdk/emsdk construct_env \
-                | sed 's/export PATH=\(.*\);/echo \1 >> $GITHUB_PATH;/' \
-                | sed 's/export \(.*\);/echo \1 >> $GITHUB_ENV;/' );
-
-            # This looks more permanent than version pinned, so keeping temporarily to avoid failures.
-            echo "${{ github.workspace }}/emsdk/ccache/git-emscripten_64bit/bin" >> $GITHUB_PATH
-
-      - name: Generate ccache_vars for ccache based on machine
-        shell: bash
-        id: ccache_vars
-        run: |-
-          echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
-          echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
-
-      - name: Verify Emscripten setup
-        run: |
-            emcc --version
-            emcmake cmake --version
-            emmake make --version
-
-      - name: Configure builds
-        run: |
-          mkdir -p build-wasm
-          cd build-wasm
-          emcmake cmake -DCOMPILE_WASM=on ..
-
-      - name: ccache prolog
-        run: |-
-          ccache -s # Print current cache stats
-          ccache -z # Zero cache entry
-
-      - name: Compile
-        working-directory: build-wasm
-        run: |
-          emmake make -j2
-
-      - name: ccache epilog
-        run: |
-          ccache -s # Print current cache stats
-
-      - name: Instantiate simd wormhole
-        working-directory: build-wasm
-        run: bash ../wasm/patch-artifacts-enable-wormhole.sh
-
-      - name: Import GEMM library from a separate wasm module
-        working-directory: build-wasm
-        run: bash ../wasm/patch-artifacts-import-gemm-module.sh
-
-      - name: Upload wasm artifact
-        uses: actions/upload-artifact@v2
-        with:
-          name: bergamot-translator-worker
-          if-no-files-found: error
-          path: |
-              ${{github.workspace}}/build-wasm/bergamot-translator-worker.js
-              ${{github.workspace}}/build-wasm/bergamot-translator-worker.wasm
-              ${{github.workspace}}/build-wasm/bergamot-translator-worker.js.bak
-

From b1e5a48f1a3e3a6d8a6d553dafe32d881eb3aabe Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerin.philip@research.iiit.ac.in>
Date: Sat, 5 Feb 2022 10:42:44 +0000
Subject: [PATCH 346/442] Increment version to v0.4.0 (#328)

---
 BERGAMOT_VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BERGAMOT_VERSION b/BERGAMOT_VERSION
index 937cd7846..fb7a04cff 100644
--- a/BERGAMOT_VERSION
+++ b/BERGAMOT_VERSION
@@ -1 +1 @@
-v0.3.1
+v0.4.0

From 97bd6e36dbdec3519133d91289d7fd31816cb09a Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Sat, 5 Feb 2022 17:25:29 +0000
Subject: [PATCH 347/442] Make default throw exception on abort for python
 (#333)

This also allows conversion of exiting aborts into runtime errors in python,
providing informative messages to the user via pybind11 existing tooling.
---
 bindings/python/bergamot.cpp | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/bindings/python/bergamot.cpp b/bindings/python/bergamot.cpp
index 97ae0614b..8cb75fc0c 100644
--- a/bindings/python/bergamot.cpp
+++ b/bindings/python/bergamot.cpp
@@ -33,7 +33,10 @@ PYBIND11_MAKE_OPAQUE(Alignments);
 
 class ServicePyAdapter {
  public:
-  ServicePyAdapter(const Service::Config &config) : service_(make_service(config)) {}
+  ServicePyAdapter(const Service::Config &config) : service_(make_service(config)) {
+    // Set marian to throw exceptions instead of std::abort()
+    marian::setThrowExceptionOnAbort(true);
+  }
 
   std::shared_ptr<_Model> modelFromConfig(const std::string &config) {
     auto parsedConfig = marian::bergamot::parseOptionsFromString(config);

From 62ff781ed4ea642912878145beaf3157123520fe Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <github@kheafield.com>
Date: Sat, 5 Feb 2022 17:26:16 +0000
Subject: [PATCH 348/442] Revert "Make default throw exception on abort for
 python (#333)"

This reverts commit 97bd6e36dbdec3519133d91289d7fd31816cb09a.

As discussed, we need messages for debugging in -fno-exceptions.
---
 bindings/python/bergamot.cpp | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/bindings/python/bergamot.cpp b/bindings/python/bergamot.cpp
index 8cb75fc0c..97ae0614b 100644
--- a/bindings/python/bergamot.cpp
+++ b/bindings/python/bergamot.cpp
@@ -33,10 +33,7 @@ PYBIND11_MAKE_OPAQUE(Alignments);
 
 class ServicePyAdapter {
  public:
-  ServicePyAdapter(const Service::Config &config) : service_(make_service(config)) {
-    // Set marian to throw exceptions instead of std::abort()
-    marian::setThrowExceptionOnAbort(true);
-  }
+  ServicePyAdapter(const Service::Config &config) : service_(make_service(config)) {}
 
   std::shared_ptr<_Model> modelFromConfig(const std::string &config) {
     auto parsedConfig = marian::bergamot::parseOptionsFromString(config);

From f6d9233dc45d7ba6217557097f0d5e358163de25 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <github@kheafield.com>
Date: Mon, 7 Feb 2022 14:21:48 +0000
Subject: [PATCH 349/442] Revert "Revert "Make default throw exception on abort
 for python (#333)""

This reverts commit 62ff781ed4ea642912878145beaf3157123520fe.

Sorry I should have realized Jerin was only amending python and
therefore this didn't break WASM.

Apologies to Jerin on this.
---
 bindings/python/bergamot.cpp | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/bindings/python/bergamot.cpp b/bindings/python/bergamot.cpp
index 97ae0614b..8cb75fc0c 100644
--- a/bindings/python/bergamot.cpp
+++ b/bindings/python/bergamot.cpp
@@ -33,7 +33,10 @@ PYBIND11_MAKE_OPAQUE(Alignments);
 
 class ServicePyAdapter {
  public:
-  ServicePyAdapter(const Service::Config &config) : service_(make_service(config)) {}
+  ServicePyAdapter(const Service::Config &config) : service_(make_service(config)) {
+    // Set marian to throw exceptions instead of std::abort()
+    marian::setThrowExceptionOnAbort(true);
+  }
 
   std::shared_ptr<_Model> modelFromConfig(const std::string &config) {
     auto parsedConfig = marian::bergamot::parseOptionsFromString(config);

From 6b2a8552345533df7e573f3ca4cc09eb3557912e Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 7 Feb 2022 16:55:31 +0100
Subject: [PATCH 350/442] JS/WASM: Re-enable importing optimized gemm module
 for (#336)

- Re-enabled the code that imports optimized gemm module
   for wasm when available
---
 wasm/import-gemm-module.js | 30 ++++++++++++------------------
 1 file changed, 12 insertions(+), 18 deletions(-)

diff --git a/wasm/import-gemm-module.js b/wasm/import-gemm-module.js
index 8d20c58a7..6430096dc 100644
--- a/wasm/import-gemm-module.js
+++ b/wasm/import-gemm-module.js
@@ -14,28 +14,22 @@ function createWasmGemm() {
         "int8_select_columns_of_b": "int8SelectColumnsOfBFallback"
     };
 
-    // ToDo: Activate the if code and remove else code once optimized gemm can work without shared array buffer.
-    if (0) {
-        // Name of the optimized gemm implementation.
-        const OPTIMIZED_GEMM = "mozIntGemm";
+    // Name of the optimized gemm implementation.
+    const OPTIMIZED_GEMM = "mozIntGemm";
 
-        const optimizedGemmModule = WebAssembly[OPTIMIZED_GEMM];
-        if (!optimizedGemmModule) {
-            return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
-        }
+    const optimizedGemmModule = WebAssembly[OPTIMIZED_GEMM];
+    if (!optimizedGemmModule) {
+        return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
+    }
 
-        const optimizedGemmModuleExports = new WebAssembly.Instance(optimizedGemmModule(), {"": {memory: wasmMemory}}).exports;
-        for (let key in GEMM_TO_FALLBACK_FUNCTIONS_MAP) {
-            if (!optimizedGemmModuleExports[key]) {
-                return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
-            }
+    const optimizedGemmModuleExports = new WebAssembly.Instance(optimizedGemmModule(), {"": {memory: wasmMemory}}).exports;
+    for (let key in GEMM_TO_FALLBACK_FUNCTIONS_MAP) {
+        if (!optimizedGemmModuleExports[key]) {
+            return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
         }
-        console.log(`Using optimized gemm (${OPTIMIZED_GEMM}) implementation`);
-        return optimizedGemmModuleExports;
-    }
-    else {
-        return fallbackGemm(GEMM_TO_FALLBACK_FUNCTIONS_MAP);
     }
+    console.log(`Using optimized gemm (${OPTIMIZED_GEMM}) implementation`);
+    return optimizedGemmModuleExports;
 }
 
 // Return the fallback gemm implementation.

From 80bd4e7651a155d21817c755c8a89ac9e118644e Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Wed, 9 Feb 2022 12:54:36 +0000
Subject: [PATCH 351/442] Print errors by default in WASM build (#343)

* Remove BadHTML exception in favour of ABORT macro
   `ABORT()` gives us readable error messages, even when exception support is disabled.
* Control marian exception global setting in tests through fixture
* WASM: construct BlockingService with critical logging by default
   This log level is only used by ABORT()

See also:
- mozilla/firefox-translations#65,
- mozilla/firefox-translations#68
- mozilla/firefox-translations#70
- mozilla/firefox-translations#56
---
 src/tests/units/html_tests.cpp     | 19 ++++++---
 src/translator/html.cpp            | 64 +++++++++++++++---------------
 src/translator/html.h              |  5 ---
 wasm/bindings/service_bindings.cpp |  8 +++-
 4 files changed, 52 insertions(+), 44 deletions(-)

diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
index 8b3ac0f24..2b71784fb 100644
--- a/src/tests/units/html_tests.cpp
+++ b/src/tests/units/html_tests.cpp
@@ -10,6 +10,17 @@
 using namespace marian::bergamot;
 using marian::string_view;
 
+class MarianThrowsExceptionsFixture {
+ protected:
+  MarianThrowsExceptionsFixture() : prev_(marian::getThrowExceptionOnAbort()) {
+    marian::setThrowExceptionOnAbort(true);
+  }
+  ~MarianThrowsExceptionsFixture() { marian::setThrowExceptionOnAbort(prev_); }
+
+ private:
+  bool prev_;
+};
+
 std::ostream &operator<<(std::ostream &out, std::pair<ByteRange, ByteRange> const &b) {
   return out << '(' << b.first << ',' << b.second << ')';
 }
@@ -76,9 +87,7 @@ TEST_CASE("Ignore HTML if process_markup is false") {
   CHECK(response.source.text == html_code);
 }
 
-TEST_CASE("Abort if alignments are missing") {
-  marian::setThrowExceptionOnAbort(true);
-
+TEST_CASE_METHOD(MarianThrowsExceptionsFixture, "Abort if alignments are missing") {
   std::string input("<p>hello <b>world</b></p>\n");
   HTML html(std::move(input), true);
 
@@ -108,9 +117,7 @@ TEST_CASE("Abort if alignments are missing") {
       "Response object does not contain alignments. TranslationModel or ResponseOptions is misconfigured?");
 }
 
-TEST_CASE("Abort if alignments are misconfigured") {
-  marian::setThrowExceptionOnAbort(true);
-
+TEST_CASE_METHOD(MarianThrowsExceptionsFixture, "Abort if alignments are misconfigured") {
   std::string input("<p>hello <b>world</b></p>\n");
   HTML html(std::move(input), true);
 
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index 20614f578..242572db0 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -47,32 +47,6 @@ size_t countPrefixWhitespaces(string_view const &input) {
   return size;
 }
 
-// Formatters used for exception messages combined with format()
-std::ostream &operator<<(std::ostream &out, HTML::Tag const *tag) {
-  if (tag == nullptr) return out << "[nullptr]";
-  switch (tag->type) {
-    case HTML::Tag::ELEMENT:
-      return out << '<' << tag->name << tag->attributes << '>';
-    case HTML::Tag::VOID_ELEMENT:
-      return out << '<' << tag->name << tag->attributes << "/>";
-    case HTML::Tag::COMMENT:
-      return out << "<!--" << tag->data << "-->";
-    case HTML::Tag::PROCESSING_INSTRUCTION:
-      return out << "<?" << tag->data << "?>";
-    case HTML::Tag::WHITESPACE:
-      return out << "[inserted space]";
-  }
-  return out << "[Unknown tag type]";
-}
-
-std::ostream &operator<<(std::ostream &out, HTML::Taint const &tags) {
-  for (auto it = tags.begin(); it != tags.end(); ++it) {
-    if (it != tags.begin()) out << ' ';
-    out << *it;
-  }
-  return out;
-}
-
 // Very simple replacement for std::format introduced in C++20
 std::string format(std::string const &formatTemplate) { return formatTemplate; }
 
@@ -270,6 +244,32 @@ size_t debugCountTokens(AnnotatedText const &text) {
 
 namespace marian::bergamot {
 
+// Formatters used for exception messages combined with format()
+std::ostream &operator<<(std::ostream &out, HTML::Tag const *tag) {
+  if (tag == nullptr) return out << "[nullptr]";
+  switch (tag->type) {
+    case HTML::Tag::ELEMENT:
+      return out << '<' << tag->name << tag->attributes << '>';
+    case HTML::Tag::VOID_ELEMENT:
+      return out << '<' << tag->name << tag->attributes << "/>";
+    case HTML::Tag::COMMENT:
+      return out << "<!--" << tag->data << "-->";
+    case HTML::Tag::PROCESSING_INSTRUCTION:
+      return out << "<?" << tag->data << "?>";
+    case HTML::Tag::WHITESPACE:
+      return out << "[inserted space]";
+  }
+  return out << "[Unknown tag type]";
+}
+
+std::ostream &operator<<(std::ostream &out, HTML::Taint const &tags) {
+  for (auto it = tags.begin(); it != tags.end(); ++it) {
+    if (it != tags.begin()) out << ' ';
+    out << *it;
+  }
+  return out;
+}
+
 HTML::HTML(std::string &&source, bool process_markup, Options &&options) : options_(std::move(options)) {
   if (!process_markup) return;
 
@@ -288,7 +288,7 @@ HTML::HTML(std::string &&source, bool process_markup, Options &&options) : optio
   while (!stop) {
     switch (scanner.next()) {
       case markup::Scanner::TT_ERROR:
-        throw BadHTML("HTML parse error");
+        ABORT("HTML parse error");
 
       case markup::Scanner::TT_EOF:
         stop = true;
@@ -354,10 +354,10 @@ HTML::HTML(std::string &&source, bool process_markup, Options &&options) : optio
         // bit of "<img/>", then completely ignore it.
         if (contains(options_.voidTags, std::string(scanner.tag()))) break;
 
-        if (stack.empty()) throw BadHTML(format("Encountered more closing tags ({}) than opening tags", scanner.tag()));
+        ABORT_IF(stack.empty(), "Encountered more closing tags ({}) than opening tags", scanner.tag());
 
-        if (stack.back()->name != scanner.tag())
-          throw BadHTML(format("Encountered unexpected closing tag </{}>, stack is {}", scanner.tag(), stack));
+        ABORT_IF(stack.back()->name != scanner.tag(), "Encountered unexpected closing tag </{}>, stack is {}",
+                 scanner.tag(), stack);
 
         // What to do with "<u></u>" case, where tag is immediately closed
         // so it never makes it into the taint of any of the spans? This adds
@@ -407,11 +407,11 @@ HTML::HTML(std::string &&source, bool process_markup, Options &&options) : optio
         break;
 
       default:
-        throw BadHTML("Unsupported scanner token type");
+        ABORT("Unsupported scanner token type");
     }
   }
 
-  if (!stack.empty()) throw BadHTML(format("Not all tags were closed: {}", stack));
+  ABORT_IF(!stack.empty(), "Not all tags were closed: {}", stack);
 
   // Add a trailing span (that's empty) to signify all closed tags.
   spans_.emplace_back(Span{source.size(), source.size(), stack});
diff --git a/src/translator/html.h b/src/translator/html.h
index d4cbd40d5..db57c0d10 100644
--- a/src/translator/html.h
+++ b/src/translator/html.h
@@ -14,11 +14,6 @@ namespace bergamot {
 
 struct Response;
 
-class BadHTML : public std::runtime_error {
- public:
-  explicit BadHTML(std::string const &what) : std::runtime_error(what) {}
-};
-
 class HTML {
  public:
   struct Options {
diff --git a/wasm/bindings/service_bindings.cpp b/wasm/bindings/service_bindings.cpp
index ee21aa879..167f51f81 100644
--- a/wasm/bindings/service_bindings.cpp
+++ b/wasm/bindings/service_bindings.cpp
@@ -75,9 +75,15 @@ EMSCRIPTEN_BINDINGS(blocking_service_config) {
   // aggregate-batching etc.
 }
 
+std::shared_ptr<BlockingService> BlockingServiceFactory(const BlockingService::Config& config) {
+  auto copy = config;
+  copy.logger.level = "critical";
+  return std::make_shared<BlockingService>(copy);
+}
+
 EMSCRIPTEN_BINDINGS(blocking_service) {
   class_<BlockingService>("BlockingService")
-      .constructor<BlockingService::Config>()
+      .smart_ptr_constructor("BlockingService", &BlockingServiceFactory)
       .function("translate", &BlockingService::translateMultiple)
       .function("translateViaPivoting", &BlockingService::pivotMultiple);
 

From 34786520cd90a4fd3cb006a60648d57d4a5ed56c Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerin.philip@research.iiit.ac.in>
Date: Wed, 9 Feb 2022 19:37:30 +0000
Subject: [PATCH 352/442] Add ability to load `.npz` models (#342)

Changes `ABORT` on non `.bin` model to an additional check for a `.npz`
extension. If `.bin`, the fast load path is activated by returning `AlignedMemory`.
Otherwise, the return of empty `AlignedMemory` causes fallback to
filesystem-based loads.

BRT: A test that checks if translation using `.npz` is approximately similar to
that of default CLI translation is checked in to ensure stability going ahead.

Previously, we only supported `.bin` models' loading via a fast mmap
path. While we had the underlying capability to load non `.bin` models, this
was not exposed, encouraging fast loads. Loading `.npz` models are helpful
for quick debugging and broader coverage of models available, which will
enhance user experience at translateLocally and python bindings.


Fixes #341.
See also: XapaJIaMnu/translateLocally#89
---
 bergamot-translator-tests          |  2 +-
 src/translator/byte_array_util.cpp | 17 +++++++++++++----
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index aaf315c80..3c0f95a17 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit aaf315c80b7339c2ae6fcca080ca30d3f0a5e2f5
+Subproject commit 3c0f95a1775a74f5db441aa2f17ceb7437679022
diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index d0fddee8b..183dea3c0 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -3,6 +3,7 @@
 #include <cstdlib>
 #include <memory>
 
+#include "common/io.h"
 #include "data/shortlist.h"
 
 namespace marian {
@@ -93,10 +94,18 @@ AlignedMemory loadFileToMemory(const std::string& path, size_t alignment) {
 AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options) {
   auto models = options->get<std::vector<std::string>>("models");
   ABORT_IF(models.size() != 1, "Loading multiple binary models is not supported for now as it is not necessary.");
-  marian::filesystem::Path modelPath(models[0]);
-  ABORT_IF(modelPath.extension() != marian::filesystem::Path(".bin"), "The file of binary model should end with .bin");
-  AlignedMemory alignedMemory = loadFileToMemory(models[0], 256);
-  return alignedMemory;
+
+  // If binary model we load into aligned memory. If .npz we leave it be to
+  // return empty aligned memory, thus allowing traditional file system loads.
+  if (marian::io::isBin(models[0])) {
+    AlignedMemory alignedMemory = loadFileToMemory(models[0], 256);
+    return alignedMemory;
+  } else if (marian::io::isNpz(models[0])) {
+    return AlignedMemory();
+  } else {
+    ABORT("Unknown extension for model: {}, should be one of `.bin` or `.npz`", models[0]);
+  }
+  return AlignedMemory();
 }
 
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options) {

From ec469193c6a38b3dc2312eaf6304c83ea0ec0253 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerin.philip@research.iiit.ac.in>
Date: Fri, 11 Feb 2022 13:06:26 +0000
Subject: [PATCH 353/442] Allow per-input options (#346)

Changes signature of BlockingService::{translate,pivot}Multiple
functions to take per input options, so a mix of HTML and plaintext
can be sent from the extension. Templating over testing is adjusted
to allow for continuous evaluations by modifying the test code.

Updates WebAssembly bindings to reflect the change in signature
and the javascript test-page to work with the new bindings.

This change lacks an accompanying test specific to the mixed HTML
and plaintext inputs.

Fixes: #345
See also: mozilla/firefox-translations#94
Co-authored-by: Jelmer van der Linde <jelmer@ikhoefgeen.nl>
---
 src/tests/common-impl.cpp                   |  6 ++++--
 src/tests/wasm.cpp                          |  3 ++-
 src/translator/service.cpp                  | 18 +++++++++---------
 src/translator/service.h                    | 20 ++++++++++++--------
 wasm/bindings/response_options_bindings.cpp |  1 +
 wasm/test_page/js/worker.js                 | 11 +++++++----
 6 files changed, 35 insertions(+), 24 deletions(-)

diff --git a/src/tests/common-impl.cpp b/src/tests/common-impl.cpp
index 238e3acd6..43b1c07d2 100644
--- a/src/tests/common-impl.cpp
+++ b/src/tests/common-impl.cpp
@@ -8,7 +8,8 @@ Response Bridge<BlockingService>::translate(BlockingService &service, std::share
   // project source to a vector of std::string, send in, unpack the first element from
   // vector<Response>, return.
   std::vector<std::string> sources = {source};
-  return service.translateMultiple(model, std::move(sources), responseOptions).front();
+  std::vector<ResponseOptions> options = {responseOptions};
+  return service.translateMultiple(model, std::move(sources), options).front();
 }
 
 Response Bridge<AsyncService>::translate(AsyncService &service, std::shared_ptr<TranslationModel> &model,
@@ -30,7 +31,8 @@ Response Bridge<BlockingService>::pivot(BlockingService &service, std::shared_pt
                                         std::shared_ptr<TranslationModel> &pivotToTarget, std::string &&source,
                                         const ResponseOptions &responseOptions) {
   std::vector<std::string> sources = {source};
-  return service.pivotMultiple(sourceToPivot, pivotToTarget, std::move(sources), responseOptions).front();
+  std::vector<ResponseOptions> options = {responseOptions};
+  return service.pivotMultiple(sourceToPivot, pivotToTarget, std::move(sources), options).front();
 }
 
 Response Bridge<AsyncService>::pivot(AsyncService &service, std::shared_ptr<TranslationModel> &sourceToPivot,
diff --git a/src/tests/wasm.cpp b/src/tests/wasm.cpp
index 9a29a20e1..97f0fc801 100644
--- a/src/tests/wasm.cpp
+++ b/src/tests/wasm.cpp
@@ -2,7 +2,7 @@
 using namespace marian::bergamot;
 
 void wasm(BlockingService &service, std::shared_ptr<TranslationModel> &model) {
-  ResponseOptions responseOptions;
+  std::vector<ResponseOptions> responseOptions;
   std::vector<std::string> texts;
 
   // WASM always requires HTML and alignment.
@@ -13,6 +13,7 @@ void wasm(BlockingService &service, std::shared_ptr<TranslationModel> &model) {
   // Hide the translateMultiple operation
   for (std::string line; std::getline(std::cin, line);) {
     texts.emplace_back(line);
+    responseOptions.emplace_back();
   }
 
   auto results = service.translateMultiple(model, std::move(texts), responseOptions);
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 6ac1fd100..8d887b53a 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -41,10 +41,10 @@ BlockingService::BlockingService(const BlockingService::Config &config)
 
 std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<TranslationModel> translationModel,
                                                          std::vector<std::string> &&sources,
-                                                         const ResponseOptions &responseOptions) {
+                                                         const std::vector<ResponseOptions> &responseOptions) {
   std::vector<HTML> htmls;
-  for (auto &&source : sources) {
-    htmls.emplace_back(std::move(source), responseOptions.HTML);
+  for (size_t i = 0; i < sources.size(); i++) {
+    htmls.emplace_back(std::move(sources[i]), responseOptions[i].HTML);
   }
   std::vector<Response> responses = translateMultipleRaw(translationModel, std::move(sources), responseOptions);
   for (size_t i = 0; i < responses.size(); i++) {
@@ -56,7 +56,7 @@ std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<Transla
 
 std::vector<Response> BlockingService::translateMultipleRaw(std::shared_ptr<TranslationModel> translationModel,
                                                             std::vector<std::string> &&sources,
-                                                            const ResponseOptions &responseOptions) {
+                                                            const std::vector<ResponseOptions> &responseOptions) {
   std::vector<Response> responses;
   responses.resize(sources.size());
 
@@ -64,7 +64,7 @@ std::vector<Response> BlockingService::translateMultipleRaw(std::shared_ptr<Tran
     auto callback = [i, &responses](Response &&response) { responses[i] = std::move(response); };  //
     TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
     Ptr<Request> request =
-        translationModel->makeRequest(requestId_++, std::move(sources[i]), callback, responseOptions, cache);
+        translationModel->makeRequest(requestId_++, std::move(sources[i]), callback, responseOptions[i], cache);
     batchingPool_.enqueueRequest(translationModel, request);
   }
 
@@ -80,10 +80,10 @@ std::vector<Response> BlockingService::translateMultipleRaw(std::shared_ptr<Tran
 std::vector<Response> BlockingService::pivotMultiple(std::shared_ptr<TranslationModel> first,
                                                      std::shared_ptr<TranslationModel> second,
                                                      std::vector<std::string> &&sources,
-                                                     const ResponseOptions &responseOptions) {
+                                                     const std::vector<ResponseOptions> &responseOptions) {
   std::vector<HTML> htmls;
-  for (auto &&source : sources) {
-    htmls.emplace_back(std::move(source), responseOptions.HTML);
+  for (size_t i = 0; i < sources.size(); i++) {
+    htmls.emplace_back(std::move(sources[i]), responseOptions[i].HTML);
   }
 
   // Translate source to pivots. This is same as calling translateMultiple.
@@ -103,7 +103,7 @@ std::vector<Response> BlockingService::pivotMultiple(std::shared_ptr<Translation
 
     TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
     Ptr<Request> request =
-        second->makePivotRequest(requestId_++, std::move(intermediate), callback, responseOptions, cache);
+        second->makePivotRequest(requestId_++, std::move(intermediate), callback, responseOptions[i], cache);
     batchingPool_.enqueueRequest(second, request);
   }
 
diff --git a/src/translator/service.h b/src/translator/service.h
index 87054e2cc..d45fcc467 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -57,13 +57,12 @@ class BlockingService {
 
   /// @param [in] translationModel: TranslationModel to use for the request.
   /// @param [in] source: rvalue reference of the string to be translated
-  /// @param [in] responseOptions: ResponseOptions indicating whether or not to include some member in the Response,
-  /// also specify any additional configurable parameters.
+  /// @param [in] responseOptions: ResponseOptions per source-item indicating whether or not to include some member in
+  /// the Response, also specify any additional configurable parameters.
   std::vector<Response> translateMultiple(std::shared_ptr<TranslationModel> translationModel,
-                                          std::vector<std::string> &&source, const ResponseOptions &responseOptions);
+                                          std::vector<std::string> &&source,
+                                          const std::vector<ResponseOptions> &responseOptions);
 
-  std::vector<Response> translateMultipleRaw(std::shared_ptr<TranslationModel> translationModel,
-                                             std::vector<std::string> &&source, const ResponseOptions &responseOptions);
   /// With the supplied two translation models, translate using first and then the second generating a response as if it
   /// were translated from first's source language to second's target langauge. Requires first's target to be second's
   /// source to work correctly - effectively implementing pivoting translation via an intermediate language.
@@ -71,16 +70,21 @@ class BlockingService {
   /// @param[in] first: TranslationModel capable of translating from source language to pivot language.
   /// @param[in] second: TranslationModel capable of translating between pivot and target language.
   /// @param[move] sources: The input source texts to be translated.
-  /// @param[in] options: Options indicating whether or not to include optional members in response and pass additional
-  /// configurations. See ResponseOptions.
+  /// @param[in] options: Options indicating whether or not to include optional members per source-text. See
+  /// ResponseOptions.
   ///
   /// @returns responses corresponding to the source-text which can be used as if they were translated with
   /// translateMultiple.
   std::vector<Response> pivotMultiple(std::shared_ptr<TranslationModel> first, std::shared_ptr<TranslationModel> second,
-                                      std::vector<std::string> &&sources, const ResponseOptions &responseOptions);
+                                      std::vector<std::string> &&sources,
+                                      const std::vector<ResponseOptions> &responseOptions);
   TranslationCache::Stats cacheStats() { return cache_.stats(); }
 
  private:
+  std::vector<Response> translateMultipleRaw(std::shared_ptr<TranslationModel> translationModel,
+                                             std::vector<std::string> &&source,
+                                             const std::vector<ResponseOptions> &responseOptions);
+
   ///  Numbering requests processed through this instance. Used to keep account of arrival times of the request. This
   ///  allows for using this quantity in priority based ordering.
   size_t requestId_;
diff --git a/wasm/bindings/response_options_bindings.cpp b/wasm/bindings/response_options_bindings.cpp
index c58d24c64..06c152a7c 100644
--- a/wasm/bindings/response_options_bindings.cpp
+++ b/wasm/bindings/response_options_bindings.cpp
@@ -17,4 +17,5 @@ EMSCRIPTEN_BINDINGS(response_options) {
       .field("qualityScores", &ResponseOptions::qualityScores)
       .field("alignment", &ResponseOptions::alignment)
       .field("html", &ResponseOptions::HTML);
+  register_vector<ResponseOptions>("VectorResponseOptions");
 }
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index d079c783f..3e315fb9a 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -119,16 +119,16 @@ const translate = (from, to, input) => {
 
   // Prepare the arguments (ResponseOptions and vectorSourceText (vector<string>)) of Translation API and call it.
   // Result is a vector<Response> where each of its item corresponds to one item of vectorSourceText in the same order.
-  const responseOptions = _prepareResponseOptions();
+  const vectorResponseOptions = _prepareResponseOptions();
   let vectorSourceText = _prepareSourceText(input);
   let vectorResponse;
   if (translationModels.length == 2) {
     // This implies translation should be done via pivoting
-    vectorResponse = translationService.translateViaPivoting(translationModels[0], translationModels[1], vectorSourceText, responseOptions);
+    vectorResponse = translationService.translateViaPivoting(translationModels[0], translationModels[1], vectorSourceText, vectorResponseOptions);
   }
   else {
     // This implies translation should be done without pivoting
-    vectorResponse = translationService.translate(translationModels[0], vectorSourceText, responseOptions);
+    vectorResponse = translationService.translate(translationModels[0], vectorSourceText, vectorResponseOptions);
   }
 
   // Parse all relevant information from vectorResponse
@@ -146,6 +146,7 @@ const translate = (from, to, input) => {
 
   // Delete prepared SourceText to avoid memory leak
   vectorSourceText.delete();
+  vectorResponseOptions.delete();
 
   return listTranslatedText;
 }
@@ -341,7 +342,9 @@ const _parseTranslatedTextSentenceQualityScores = (vectorResponse) => {
 }
 
 const _prepareResponseOptions = () => {
-  return {qualityScores: true, alignment: true, html: true};
+  const vector = new Module.VectorResponseOptions();
+  vector.push_back({qualityScores: true, alignment: true, html: true})
+  return vector;
 }
 
 const _prepareSourceText = (input) => {

From c76e630e00301ee0c2a531cb2585ce45ed854dfe Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 14 Feb 2022 13:16:33 +0100
Subject: [PATCH 354/442] JS/WASM: Passing ResponseOptions for every item for
 translation batch api (#348)

- Now translate() JS API accepts ResponseOptions per batch item

 - Fixed the logic to create vector<ResponseOption>
---
 wasm/test_page/js/index.js  | 15 ++++++++++++++-
 wasm/test_page/js/worker.js | 19 +++++++++++--------
 2 files changed, 25 insertions(+), 9 deletions(-)

diff --git a/wasm/test_page/js/index.js b/wasm/test_page/js/index.js
index 6b580415f..7cb365702 100644
--- a/wasm/test_page/js/index.js
+++ b/wasm/test_page/js/index.js
@@ -27,14 +27,27 @@ document.querySelector("#input").addEventListener("keyup", function (event) {
   translateCall();
 });
 
+const _prepareTranslateOptions = (paragraphs) => {
+  const translateOptions = [];
+  paragraphs.forEach(paragraph => {
+    // Each option object can be different for each entry. But to keep the test page simple,
+    // we just keep all the options same (specifically avoiding parsing the input to determine
+    // html/non-html text)
+    translateOptions.push({"isQualityScores": true, "isHtml": true});
+  });
+  return translateOptions;
+};
+
 const translateCall = () => {
   const text = document.querySelector("#input").value + "  ";
   if (!text.trim().length) return;
+
   const paragraphs = text.split("\n");
+  const translateOptions = _prepareTranslateOptions(paragraphs);
   $("#output").setAttribute("disabled", true);
   const lngFrom = langFrom.value;
   const lngTo = langTo.value;
-  worker.postMessage(["translate", lngFrom, lngTo, paragraphs]);
+  worker.postMessage(["translate", lngFrom, lngTo, paragraphs, translateOptions]);
 };
 
 worker.onmessage = function (e) {
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index 3e315fb9a..a2761a4a0 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -54,6 +54,7 @@ onmessage = async function(e) {
       const from = e.data[1];
       const to = e.data[2];
       const input = e.data[3];
+      const translateOptions = e.data[4];
       let inputWordCount = 0;
       let inputBlockElements = 0;
       input.forEach(sentence => {
@@ -63,7 +64,7 @@ onmessage = async function(e) {
       let start = Date.now();
       try {
         log(`Blocks to translate: ${inputBlockElements}`);
-        result = translate(from, to, input);
+        result = translate(from, to, input, translateOptions);
         const secs = (Date.now() - start) / 1000;
         log(`Translation '${from}${to}' Successful. Speed: ${Math.round(inputWordCount / secs)} WPS (${inputWordCount} words in ${secs} secs)`);
       } catch (error) {
@@ -107,7 +108,7 @@ const constructTranslationModel = async (from, to) => {
 }
 
 // Translates text from source language to target language (via pivoting if necessary).
-const translate = (from, to, input) => {
+const translate = (from, to, input, translateOptions) => {
   const languagePairs = _getLanguagePairs(from, to);
   log(`Translating for language pair(s): '${languagePairs.toString()}'`);
 
@@ -117,9 +118,9 @@ const translate = (from, to, input) => {
     throw Error(`Insufficient no. of loaded translation models. Required:'${languagePairs.length}' Found:'${translationModels.length}'`);
   }
 
-  // Prepare the arguments (ResponseOptions and vectorSourceText (vector<string>)) of Translation API and call it.
+  // Prepare the arguments (vectorResponseOptions and vectorSourceText (vector<string>)) of Translation API and call it.
   // Result is a vector<Response> where each of its item corresponds to one item of vectorSourceText in the same order.
-  const vectorResponseOptions = _prepareResponseOptions();
+  const vectorResponseOptions = _prepareResponseOptions(translateOptions);
   let vectorSourceText = _prepareSourceText(input);
   let vectorResponse;
   if (translationModels.length == 2) {
@@ -341,10 +342,12 @@ const _parseTranslatedTextSentenceQualityScores = (vectorResponse) => {
   return result;
 }
 
-const _prepareResponseOptions = () => {
-  const vector = new Module.VectorResponseOptions();
-  vector.push_back({qualityScores: true, alignment: true, html: true})
-  return vector;
+const _prepareResponseOptions = (translateOptions) => {
+  const vectorResponseOptions = new Module.VectorResponseOptions;
+  translateOptions.forEach(translateOption => {
+    vectorResponseOptions.push_back({qualityScores: translateOption["isQualityScores"], alignment: true, html: translateOption["isHtml"]});
+  });
+  return vectorResponseOptions;
 }
 
 const _prepareSourceText = (input) => {

From a94725b20da9e2f3b1f5303c462d2c6e129c92f9 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <kpu@users.noreply.github.com>
Date: Mon, 14 Feb 2022 14:26:06 +0000
Subject: [PATCH 355/442] Update aligned vector following intgemm
 1b8cbd6f611c21011325cfe0312940f0635dea33 (#334)

Fixes memory leak
ifdef for -fno-exceptions including clang-cl
Move spacing back to intgemm upstream

Co-authored-by: Jerin Philip <jerin.philip@research.iiit.ac.in>
---
 src/translator/aligned.h | 105 +++++++++++++++++++++++----------------
 1 file changed, 63 insertions(+), 42 deletions(-)

diff --git a/src/translator/aligned.h b/src/translator/aligned.h
index 6edb84e35..73e82edc3 100644
--- a/src/translator/aligned.h
+++ b/src/translator/aligned.h
@@ -2,70 +2,91 @@
 #include <cstdlib>
 #include <new>
 #ifdef _MSC_VER
+// Ensure _HAS_EXCEPTIONS is defined
+#include <vcruntime.h>
 #include <malloc.h>
 #endif
 
+#if !((defined(_MSC_VER) && !defined(__clang__)) ? (_HAS_EXCEPTIONS) : (__EXCEPTIONS))
+#include <cstdlib>
+#endif
+
 // Aligned simple vector.
 
 namespace marian {
 namespace bergamot {
 
 template <class T> class AlignedVector {
-public:
-  AlignedVector() : mem_(nullptr), size_(0) {}
+  public:
+    AlignedVector() : mem_(nullptr), size_(0) {}
 
-  explicit AlignedVector(std::size_t size, std::size_t alignment = 64 /* CPU cares about this */)
-          : size_(size) {
+    explicit AlignedVector(std::size_t size, std::size_t alignment = 64 /* CPU cares about this */)
+      : size_(size) {
 #ifdef _MSC_VER
-    mem_ = static_cast<T*>(_aligned_malloc(size * sizeof(T), alignment));
-      if (!mem_) throw std::bad_alloc();
+      mem_ = static_cast<T*>(_aligned_malloc(size * sizeof(T), alignment));
+      if (!mem_) {
+#  if (defined(_MSC_VER) && !defined(__clang__)) ? (_HAS_EXCEPTIONS) : (__EXCEPTIONS)
+        throw std::bad_alloc();
+#  else
+        std::abort();
+#  endif
+      }
 #else
-    if (posix_memalign(reinterpret_cast<void **>(&mem_), alignment, size * sizeof(T))) {
-      throw std::bad_alloc();
-    }
+      if (posix_memalign(reinterpret_cast<void **>(&mem_), alignment, size * sizeof(T))) {
+#  if (defined(_MSC_VER) && !defined(__clang__)) ? (_HAS_EXCEPTIONS) : (__EXCEPTIONS)
+        throw std::bad_alloc();
+#  else
+        std::abort();
+#  endif
+      }
 #endif
-  }
+    }
 
-  AlignedVector(AlignedVector &&from) : mem_(from.mem_), size_(from.size_) {
-    from.mem_ = nullptr;
-    from.size_ = 0;
-  }
+    AlignedVector(AlignedVector &&from) : mem_(from.mem_), size_(from.size_) {
+      from.mem_ = nullptr;
+      from.size_ = 0;
+    }
 
-  AlignedVector &operator=(AlignedVector &&from) {
-    mem_ = from.mem_;
-    size_ = from.size_;
-    from.mem_ = nullptr;
-    from.size_ = 0;
-    return *this;
-  }
+    AlignedVector &operator=(AlignedVector &&from) {
+      if (this == &from) return *this;
+      release();
+      mem_ = from.mem_;
+      size_ = from.size_;
+      from.mem_ = nullptr;
+      from.size_ = 0;
+      return *this;
+    }
 
-  AlignedVector(const AlignedVector&) = delete;
-  AlignedVector& operator=(const AlignedVector&) = delete;
+    AlignedVector(const AlignedVector&) = delete;
+    AlignedVector& operator=(const AlignedVector&) = delete;
 
-  ~AlignedVector() {
-#ifdef _MSC_VER
-    _aligned_free(mem_);
-#else
-    std::free(mem_);
-#endif
-  }
+    ~AlignedVector() { release(); }
 
-  std::size_t size() const { return size_; }
+    std::size_t size() const { return size_; }
 
-  T &operator[](std::size_t offset) { return mem_[offset]; }
-  const T &operator[](std::size_t offset) const { return mem_[offset]; }
+    T &operator[](std::size_t offset) { return mem_[offset]; }
+    const T &operator[](std::size_t offset) const { return mem_[offset]; }
 
-  T *begin() { return mem_; }
-  const T *begin() const { return mem_; }
-  T *end() { return mem_ + size_; }
-  const T *end() const { return mem_ + size_; }
+    T *begin() { return mem_; }
+    const T *begin() const { return mem_; }
+    T *end() { return mem_ + size_; }
+    const T *end() const { return mem_ + size_; }
 
-  template <typename ReturnType>
-  ReturnType *as() { return reinterpret_cast<ReturnType*>(mem_); }
+    template <typename ReturnType>
+    ReturnType *as() { return reinterpret_cast<ReturnType*>(mem_); }
 
-private:
-  T *mem_;
-  std::size_t size_;
+  private:
+    T *mem_;
+    std::size_t size_;
+
+    void release() {
+#ifdef _MSC_VER
+      _aligned_free(mem_);
+#else
+      std::free(mem_);
+#endif
+    }
 };
+
 } // namespace bergamot
 } // namespace marian

From 9f55fb47569e89295439c2a679b6f4fd07ca2fc8 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Tue, 15 Feb 2022 11:04:07 +0000
Subject: [PATCH 356/442] Improve cache (#347)

Hide `cache-mutex-buckets` from the user. Now configured to be equal to number
of workers. Python bindings which had exposed these are modified to reflect
the API change. `std::optional` enabled on cache, constructed only if enabled.
Pointers used are replaced with an equivalent `std::optional.`

Fixes: #317
---
 bindings/python/bergamot.cpp         |  9 +++------
 src/translator/request.cpp           |  6 +++---
 src/translator/request.h             |  6 +++---
 src/translator/service.cpp           | 20 ++++++++++----------
 src/translator/service.h             | 28 +++++++++++++++-------------
 src/translator/translation_model.cpp |  6 ++++--
 src/translator/translation_model.h   |  4 ++--
 7 files changed, 40 insertions(+), 39 deletions(-)

diff --git a/bindings/python/bergamot.cpp b/bindings/python/bergamot.cpp
index 8cb75fc0c..e341ada75 100644
--- a/bindings/python/bergamot.cpp
+++ b/bindings/python/bergamot.cpp
@@ -198,22 +198,19 @@ PYBIND11_MODULE(_bergamot, m) {
       .def("pivot", &ServicePyAdapter::pivot);
 
   py::class_<Service::Config>(m, "ServiceConfig")
-      .def(py::init<>([](size_t numWorkers, bool cacheEnabled, size_t cacheSize, size_t cacheMutexBuckets,
-                         std::string logging) {
+      .def(py::init<>([](size_t numWorkers, bool cacheEnabled, size_t cacheSize, std::string logging) {
              Service::Config config;
              config.numWorkers = numWorkers;
              config.cacheEnabled = cacheEnabled;
              config.cacheSize = cacheSize;
-             config.cacheMutexBuckets = cacheMutexBuckets;
              config.logger.level = logging;
              return config;
            }),
            py::arg("numWorkers") = 1, py::arg("cacheEnabled") = false, py::arg("cacheSize") = 20000,
-           py::arg("cacheMutexBuckets") = 1, py::arg("logLevel") = "off")
+           py::arg("logLevel") = "off")
       .def_readwrite("numWorkers", &Service::Config::numWorkers)
       .def_readwrite("cacheEnabled", &Service::Config::cacheEnabled)
-      .def_readwrite("cacheSize", &Service::Config::cacheSize)
-      .def_readwrite("cacheMutexBuckets", &Service::Config::cacheMutexBuckets);
+      .def_readwrite("cacheSize", &Service::Config::cacheSize);
 
   py::class_<_Model, std::shared_ptr<_Model>>(m, "TranslationModel");
 }
diff --git a/src/translator/request.cpp b/src/translator/request.cpp
index feba62a4a..fd8fe8fca 100644
--- a/src/translator/request.cpp
+++ b/src/translator/request.cpp
@@ -23,7 +23,7 @@ size_t hashForCache(const TranslationModel &model, const marian::Words &words) {
 
 // -----------------------------------------------------------------
 Request::Request(size_t Id, const TranslationModel &model, Segments &&segments, ResponseBuilder &&responseBuilder,
-                 TranslationCache *cache)
+                 std::optional<TranslationCache> &cache)
     : Id_(Id),
       model_(model),
       segments_(std::move(segments)),
@@ -42,7 +42,7 @@ Request::Request(size_t Id, const TranslationModel &model, Segments &&segments,
     counter_ = segments_.size();
     histories_.resize(segments_.size());
 
-    if (cache_ != nullptr) {
+    if (cache_) {
       // Iterate through segments, see if any can be prefilled from cache. If prefilled, mark the particular segments as
       // complete (non-empty ProcessedRequestSentence). Also update accounting used elsewhere (counter_) to reflect one
       // less segment to translate.
@@ -76,7 +76,7 @@ void Request::processHistory(size_t index, Ptr<History> history) {
   // Fill in placeholder from History obtained by freshly translating. Since this was a cache-miss to have got through,
   // update cache if available to store the result.
   histories_[index] = history;
-  if (cache_ != nullptr) {
+  if (cache_) {
     size_t key = hashForCache(model_, getSegment(index));
     cache_->store(key, histories_[index]);
   }
diff --git a/src/translator/request.h b/src/translator/request.h
index 8415e3233..0c4750b29 100644
--- a/src/translator/request.h
+++ b/src/translator/request.h
@@ -54,7 +54,7 @@ class Request {
   /// @param [in] cache: Cache supplied externally to attempt to fetch translations or store them after completion for
   /// reuse later.
   Request(size_t Id, const TranslationModel &model, Segments &&segments, ResponseBuilder &&responseBuilder,
-          TranslationCache *cache);
+          std::optional<TranslationCache> &cache);
 
   /// Obtain the count of tokens in the segment correponding to index. Used to
   /// insert sentence from multiple requests into the corresponding size bucket.
@@ -100,8 +100,8 @@ class Request {
   /// std::vector<Ptr<Vocab const>> *vocabs_;
   ResponseBuilder responseBuilder_;
 
-  /// Cache used to hold unit translations. If nullptr, means no-caching.
-  TranslationCache *cache_;
+  /// Cache used to hold unit translations. If nullopt, means no-caching.
+  std::optional<TranslationCache> &cache_;
 };
 
 /// A RequestSentence provides a view to a sentence within a Request. Existence
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index 8d887b53a..d510cf5c0 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -30,13 +30,17 @@ Response combine(Response &&first, Response &&second) {
   return combined;
 }
 
+std::optional<TranslationCache> makeOptionalCache(bool enabled, size_t size, size_t mutexBuckets) {
+  return enabled ? std::make_optional<TranslationCache>(size, mutexBuckets) : std::nullopt;
+}
+
 }  // namespace
 
 BlockingService::BlockingService(const BlockingService::Config &config)
     : config_(config),
       requestId_(0),
       batchingPool_(),
-      cache_(config.cacheSize, /*mutexBuckets=*/1),
+      cache_(makeOptionalCache(config.cacheEnabled, config.cacheSize, /*mutexBuckets = */ 1)),
       logger_(config.logger) {}
 
 std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<TranslationModel> translationModel,
@@ -62,9 +66,8 @@ std::vector<Response> BlockingService::translateMultipleRaw(std::shared_ptr<Tran
 
   for (size_t i = 0; i < sources.size(); i++) {
     auto callback = [i, &responses](Response &&response) { responses[i] = std::move(response); };  //
-    TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
     Ptr<Request> request =
-        translationModel->makeRequest(requestId_++, std::move(sources[i]), callback, responseOptions[i], cache);
+        translationModel->makeRequest(requestId_++, std::move(sources[i]), callback, responseOptions[i], cache_);
     batchingPool_.enqueueRequest(translationModel, request);
   }
 
@@ -101,9 +104,8 @@ std::vector<Response> BlockingService::pivotMultiple(std::shared_ptr<Translation
                                     // it in allows further use in makePivotRequest
     auto callback = [i, &pivotsToTargets](Response &&response) { pivotsToTargets[i] = std::move(response); };  //
 
-    TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
     Ptr<Request> request =
-        second->makePivotRequest(requestId_++, std::move(intermediate), callback, responseOptions[i], cache);
+        second->makePivotRequest(requestId_++, std::move(intermediate), callback, responseOptions[i], cache_);
     batchingPool_.enqueueRequest(second, request);
   }
 
@@ -131,7 +133,7 @@ AsyncService::AsyncService(const AsyncService::Config &config)
     : requestId_(0),
       config_(config),
       safeBatchingPool_(),
-      cache_(config_.cacheSize, config_.cacheMutexBuckets),
+      cache_(makeOptionalCache(config_.cacheEnabled, config_.cacheSize, /*mutexBuckets=*/config_.numWorkers)),
       logger_(config.logger) {
   ABORT_IF(config_.numWorkers == 0, "Number of workers should be at least 1 in a threaded workflow");
   workers_.reserve(config_.numWorkers);
@@ -188,9 +190,8 @@ void AsyncService::pivot(std::shared_ptr<TranslationModel> first, std::shared_pt
     };
 
     // Second call.
-    TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
     Ptr<Request> request =
-        second->makePivotRequest(requestId_++, std::move(intermediate), joiningCallback, responseOptions, cache);
+        second->makePivotRequest(requestId_++, std::move(intermediate), joiningCallback, responseOptions, cache_);
     safeBatchingPool_.enqueueRequest(second, request);
   };
 
@@ -213,9 +214,8 @@ void AsyncService::translate(std::shared_ptr<TranslationModel> translationModel,
 void AsyncService::translateRaw(std::shared_ptr<TranslationModel> translationModel, std::string &&source,
                                 CallbackType callback, const ResponseOptions &responseOptions) {
   // Producer thread, a call to this function adds new work items. If batches are available, notifies workers waiting.
-  TranslationCache *cache = config_.cacheEnabled ? &cache_ : nullptr;
   Ptr<Request> request =
-      translationModel->makeRequest(requestId_++, std::move(source), callback, responseOptions, cache);
+      translationModel->makeRequest(requestId_++, std::move(source), callback, responseOptions, cache_);
   safeBatchingPool_.enqueueRequest(translationModel, request);
 }
 
diff --git a/src/translator/service.h b/src/translator/service.h
index d45fcc467..72522375f 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -31,9 +31,15 @@ class BlockingService {
  public:
   struct Config {
     bool cacheEnabled{false};  ///< Whether to enable cache or not.
-    size_t cacheSize{2000};    ///< Size in History items to be stored in the cache. Loosely corresponds to sentences to
-                               /// cache in the real world.
-    Logger::Config logger;     // Configurations for logging
+
+    /// Size in History items to be stored in the cache. Loosely corresponds to sentences to
+    /// cache in the real world. Note that cache has a random-eviction policy. The peak
+    /// storage at full occupancy is controlled by this parameter. However, whether we attain
+    /// full occupancy or not is controlled by random factors - specifically how uniformly
+    /// the hash distributes.
+    size_t cacheSize{2000};
+
+    Logger::Config logger;  ///< Configurations for logging
 
     template <class App>
     static void addOptions(App &app, Config &config) {
@@ -78,7 +84,7 @@ class BlockingService {
   std::vector<Response> pivotMultiple(std::shared_ptr<TranslationModel> first, std::shared_ptr<TranslationModel> second,
                                       std::vector<std::string> &&sources,
                                       const std::vector<ResponseOptions> &responseOptions);
-  TranslationCache::Stats cacheStats() { return cache_.stats(); }
+  TranslationCache::Stats cacheStats() { return cache_ ? cache_->stats() : TranslationCache::Stats(); }
 
  private:
   std::vector<Response> translateMultipleRaw(std::shared_ptr<TranslationModel> translationModel,
@@ -97,7 +103,7 @@ class BlockingService {
 
   // Logger which shuts down cleanly with service.
   Logger logger_;
-  TranslationCache cache_;
+  std::optional<TranslationCache> cache_;
 };
 
 /// Effectively a threadpool, providing an API to take a translation request of a source-text, paramaterized by
@@ -110,18 +116,13 @@ class AsyncService {
     bool cacheEnabled{false};  ///< Whether to enable cache or not.
     size_t cacheSize{2000};    ///< Size in History items to be stored in the cache. Loosely corresponds to sentences to
                                /// cache in the real world.
-    size_t cacheMutexBuckets{1};  ///< Controls the granularity of locking to reduce contention by bucketing mutexes
-                                  ///< guarding cache entry read write. Optimal at min(core, numWorkers) assuming a
-                                  ///< reasonably large cache-size.
-    Logger::Config logger;        // Configurations for logging
+    Logger::Config logger;     // Configurations for logging
 
     template <class App>
     static void addOptions(App &app, Config &config) {
       app.add_option("--cpu-threads", config.numWorkers, "Workers to form translation backend");
       app.add_option("--cache-translations", config.cacheEnabled, "Whether to cache translations or not.");
       app.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
-      app.add_option("--cache-mutex-buckets", config.cacheMutexBuckets,
-                     "Number of mutex buckets to control locking granularity");
       Logger::Config::addOptions(app, config.logger);
     }
   };
@@ -170,11 +171,12 @@ class AsyncService {
   /// If you do not want to wait, call `clear()` before destructor.
   ~AsyncService();
 
-  TranslationCache::Stats cacheStats() { return cache_.stats(); }
+  TranslationCache::Stats cacheStats() { return cache_ ? cache_->stats() : TranslationCache::Stats(); }
 
  private:
   void translateRaw(std::shared_ptr<TranslationModel> translationModel, std::string &&source, CallbackType callback,
                     const ResponseOptions &options = ResponseOptions());
+
   AsyncService::Config config_;
 
   std::vector<std::thread> workers_;
@@ -193,7 +195,7 @@ class AsyncService {
 
   // Logger which shuts down cleanly with service.
   Logger logger_;
-  TranslationCache cache_;
+  std::optional<TranslationCache> cache_;
 };
 
 }  // namespace bergamot
diff --git a/src/translator/translation_model.cpp b/src/translator/translation_model.cpp
index 09c935beb..3f91ebb47 100644
--- a/src/translator/translation_model.cpp
+++ b/src/translator/translation_model.cpp
@@ -90,7 +90,8 @@ void TranslationModel::loadBackend(size_t idx) {
 
 // Make request process is shared between Async and Blocking workflow of translating.
 Ptr<Request> TranslationModel::makeRequest(size_t requestId, std::string &&source, CallbackType callback,
-                                           const ResponseOptions &responseOptions, TranslationCache *cache) {
+                                           const ResponseOptions &responseOptions,
+                                           std::optional<TranslationCache> &cache) {
   Segments segments;
   AnnotatedText annotatedSource;
 
@@ -103,7 +104,8 @@ Ptr<Request> TranslationModel::makeRequest(size_t requestId, std::string &&sourc
 }
 
 Ptr<Request> TranslationModel::makePivotRequest(size_t requestId, AnnotatedText &&previousTarget, CallbackType callback,
-                                                const ResponseOptions &responseOptions, TranslationCache *cache) {
+                                                const ResponseOptions &responseOptions,
+                                                std::optional<TranslationCache> &cache) {
   Segments segments;
 
   textProcessor_.processFromAnnotation(previousTarget, segments);
diff --git a/src/translator/translation_model.h b/src/translator/translation_model.h
index eac7b5aaa..53980b4e9 100644
--- a/src/translator/translation_model.h
+++ b/src/translator/translation_model.h
@@ -71,10 +71,10 @@ class TranslationModel {
   /// @param [in] responseOptions: Configuration used to prepare the Response corresponding to the created request.
   //  @returns Request created from the query parameters wrapped within a shared-pointer.
   Ptr<Request> makeRequest(size_t requestId, std::string&& source, CallbackType callback,
-                           const ResponseOptions& responseOptions, TranslationCache* cache);
+                           const ResponseOptions& responseOptions, std::optional<TranslationCache>& cache);
 
   Ptr<Request> makePivotRequest(size_t requestId, AnnotatedText&& previousTarget, CallbackType callback,
-                                const ResponseOptions& responseOptions, TranslationCache* cache);
+                                const ResponseOptions& responseOptions, std::optional<TranslationCache>& cache);
 
   /// Relays a request to the batching-pool specific to this translation model.
   /// @param [in] request: Request constructed through makeRequest

From 2844cedb0dc8b67df5b656e61b69e194a6658501 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 17 Feb 2022 14:16:26 +0100
Subject: [PATCH 357/442] JS: Refactoring wasm test page (#354)

* Free all the objects properly that were constructed for translation api
* Refactored pivot detection mechanism
---
 wasm/test_page/js/worker.js | 148 ++++++++++++++++++------------------
 1 file changed, 74 insertions(+), 74 deletions(-)

diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index a2761a4a0..4c1c640f1 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -85,7 +85,8 @@ const constructTranslationService = async () => {
   }
 }
 
-// Constructs translation model for the source and target language pair.
+// Constructs translation model(s) for the source and target language pair (using
+// pivoting if required).
 const constructTranslationModel = async (from, to) => {
   // Delete all previously constructed translation models and clear the map
   languagePairToTranslationModels.forEach((value, key) => {
@@ -94,62 +95,63 @@ const constructTranslationModel = async (from, to) => {
   });
   languagePairToTranslationModels.clear();
 
-  const languagePairs = _getLanguagePairs(from, to);
-  log(`Constructing translation model(s): ${languagePairs.toString()}`);
-  if (languagePairs.length == 2) {
-    // This implies pivoting is required => Construct 2 translation models
-    await Promise.all([_constructTranslationModelHelper(languagePairs[0]),
-                      _constructTranslationModelHelper(languagePairs[1])]);
+  if (_isPivotingRequired(from, to)) {
+    // Pivoting requires 2 translation models
+    const languagePairSrcToPivot = _getLanguagePair(from, PIVOT_LANGUAGE);
+    const languagePairPivotToTarget = _getLanguagePair(PIVOT_LANGUAGE, to);
+    await Promise.all([_constructTranslationModelHelper(languagePairSrcToPivot),
+                      _constructTranslationModelHelper(languagePairPivotToTarget)]);
   }
   else {
-    // This implies pivoting is not required => Construct 1 translation model
-    await _constructTranslationModelHelper(languagePairs[0]);
+    // Non-pivoting case requires only 1 translation model
+    await _constructTranslationModelHelper(_getLanguagePair(from, to));
   }
 }
 
 // Translates text from source language to target language (via pivoting if necessary).
 const translate = (from, to, input, translateOptions) => {
-  const languagePairs = _getLanguagePairs(from, to);
-  log(`Translating for language pair(s): '${languagePairs.toString()}'`);
-
-  // Each language pair requires a corresponding loaded translation model. Otherwise, it's an error.
-  let translationModels = _getLoadedTranslationModels(from, to);
-  if (translationModels.length != languagePairs.length) {
-    throw Error(`Insufficient no. of loaded translation models. Required:'${languagePairs.length}' Found:'${translationModels.length}'`);
-  }
+  let vectorResponseOptions, vectorSourceText, vectorResponse;
+  try {
+    // Prepare the arguments (vectorResponseOptions and vectorSourceText (vector<string>)) of Translation API and call it.
+    // Result is a vector<Response> where each of its item corresponds to one item of vectorSourceText in the same order.
+    vectorResponseOptions = _prepareResponseOptions(translateOptions);
+    vectorSourceText = _prepareSourceText(input);
+
+    if (_isPivotingRequired(from, to)) {
+      // Translate via pivoting
+      const translationModelSrcToPivot = _getLoadedTranslationModel(from, PIVOT_LANGUAGE);
+      const translationModelPivotToTarget = _getLoadedTranslationModel(PIVOT_LANGUAGE, to);
+      vectorResponse = translationService.translateViaPivoting(translationModelSrcToPivot,
+                                                              translationModelPivotToTarget,
+                                                              vectorSourceText,
+                                                              vectorResponseOptions);
+    }
+    else {
+      // Translate without pivoting
+      const translationModel = _getLoadedTranslationModel(from, to);
+      vectorResponse = translationService.translate(translationModel, vectorSourceText, vectorResponseOptions);
+    }
 
-  // Prepare the arguments (vectorResponseOptions and vectorSourceText (vector<string>)) of Translation API and call it.
-  // Result is a vector<Response> where each of its item corresponds to one item of vectorSourceText in the same order.
-  const vectorResponseOptions = _prepareResponseOptions(translateOptions);
-  let vectorSourceText = _prepareSourceText(input);
-  let vectorResponse;
-  if (translationModels.length == 2) {
-    // This implies translation should be done via pivoting
-    vectorResponse = translationService.translateViaPivoting(translationModels[0], translationModels[1], vectorSourceText, vectorResponseOptions);
+    // Parse all relevant information from vectorResponse
+    const listTranslatedText = _parseTranslatedText(vectorResponse);
+    const listSourceText = _parseSourceText(vectorResponse);
+    const listTranslatedTextSentences = _parseTranslatedTextSentences(vectorResponse);
+    const listSourceTextSentences = _parseSourceTextSentences(vectorResponse);
+    const listTranslatedTextSentenceQualityScores = _parseTranslatedTextSentenceQualityScores(vectorResponse);
+
+    log(`Source text: ${listSourceText}`);
+    log(`Translated text: ${listTranslatedText}`);
+    log(`Translated sentences: ${JSON.stringify(listTranslatedTextSentences)}`);
+    log(`Source sentences: ${JSON.stringify(listSourceTextSentences)}`);
+    log(`Translated sentence quality scores: ${JSON.stringify(listTranslatedTextSentenceQualityScores)}`);
+
+    return listTranslatedText;
+  } finally {
+    // Necessary clean up
+    if (vectorSourceText != null) vectorSourceText.delete();
+    if (vectorResponseOptions != null) vectorResponseOptions.delete();
+    if (vectorResponse != null) vectorResponse.delete();
   }
-  else {
-    // This implies translation should be done without pivoting
-    vectorResponse = translationService.translate(translationModels[0], vectorSourceText, vectorResponseOptions);
-  }
-
-  // Parse all relevant information from vectorResponse
-  const listTranslatedText = _parseTranslatedText(vectorResponse);
-  const listSourceText = _parseSourceText(vectorResponse);
-  const listTranslatedTextSentences = _parseTranslatedTextSentences(vectorResponse);
-  const listSourceTextSentences = _parseSourceTextSentences(vectorResponse);
-  const listTranslatedTextSentenceQualityScores = _parseTranslatedTextSentenceQualityScores(vectorResponse);
-
-  log(`Source text: ${listSourceText}`);
-  log(`Translated text: ${listTranslatedText}`);
-  log(`Translated sentences: ${JSON.stringify(listTranslatedTextSentences)}`);
-  log(`Source sentences: ${JSON.stringify(listSourceTextSentences)}`);
-  log(`Translated sentence quality scores: ${JSON.stringify(listTranslatedTextSentenceQualityScores)}`);
-
-  // Delete prepared SourceText to avoid memory leak
-  vectorSourceText.delete();
-  vectorResponseOptions.delete();
-
-  return listTranslatedText;
 }
 
 // Downloads file from a url and returns the array buffer
@@ -174,6 +176,7 @@ const _prepareAlignedMemoryFromBuffer = async (buffer, alignmentSize) => {
 }
 
 const _constructTranslationModelHelper = async (languagePair) => {
+  log(`Constructing translation model ${languagePair}`);
 
   /*Set the Model Configuration as YAML formatted string.
     For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
@@ -237,35 +240,20 @@ alignment: soft
   languagePairToTranslationModels.set(languagePair, translationModel);
 }
 
-const _isPivotingRequired = (lang1, lang2) => {
-  if ((lang1 === PIVOT_LANGUAGE) || (lang2 === PIVOT_LANGUAGE)) {
-    return false;
-  }
-  return true;
+const _isPivotingRequired = (from, to) => {
+  return (from !== PIVOT_LANGUAGE) && (to !== PIVOT_LANGUAGE);
 }
 
-const _getLanguagePairs = (srcLang, tgtLang) => {
-  const languagePairs = [];
-  if (_isPivotingRequired(srcLang, tgtLang)) {
-    // Do not change the push order
-    languagePairs.push(`${srcLang}${PIVOT_LANGUAGE}`);
-    languagePairs.push(`${PIVOT_LANGUAGE}${tgtLang}`);
-  }
-  else {
-    languagePairs.push(`${srcLang}${tgtLang}`);
-  }
-  return languagePairs;
+const _getLanguagePair = (srcLang, tgtLang) => {
+  return `${srcLang}${tgtLang}`;
 }
 
-const _getLoadedTranslationModels = (srcLang, tgtLang) => {
-  const languagePairs = _getLanguagePairs(srcLang, tgtLang);
-  const loadedTranslationModels = [];
-  for (const langPair of languagePairs) {
-    if (languagePairToTranslationModels.has(langPair)) {
-      loadedTranslationModels.push(languagePairToTranslationModels.get(langPair));
-    }
+const _getLoadedTranslationModel = (srcLang, tgtLang) => {
+  const languagePair = _getLanguagePair(srcLang, tgtLang);
+  if (!languagePairToTranslationModels.has(languagePair)) {
+    throw Error(`Translation model '${languagePair}' not loaded`);
   }
-  return loadedTranslationModels;
+  return languagePairToTranslationModels.get(languagePair);
 }
 
 const _parseTranslatedText = (vectorResponse) => {
@@ -343,10 +331,18 @@ const _parseTranslatedTextSentenceQualityScores = (vectorResponse) => {
 }
 
 const _prepareResponseOptions = (translateOptions) => {
-  const vectorResponseOptions = new Module.VectorResponseOptions;
+  let vectorResponseOptions = new Module.VectorResponseOptions;
   translateOptions.forEach(translateOption => {
-    vectorResponseOptions.push_back({qualityScores: translateOption["isQualityScores"], alignment: true, html: translateOption["isHtml"]});
+    vectorResponseOptions.push_back({
+      qualityScores: translateOption["isQualityScores"],
+      alignment: true,
+      html: translateOption["isHtml"]
+    });
   });
+  if (vectorResponseOptions.size() == 0) {
+    vectorResponseOptions.delete();
+    throw Error(`No Translation Options provided`);
+  }
   return vectorResponseOptions;
 }
 
@@ -359,6 +355,10 @@ const _prepareSourceText = (input) => {
     }
     vectorSourceText.push_back(paragraph.trim())
   })
+  if (vectorSourceText.size() == 0) {
+    vectorSourceText.delete();
+    throw Error(`No text provided to translate`);
+  }
   return vectorSourceText;
 }
 

From 6ccd4c68e8aac15e909f550323027cfc19a2fe15 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 17 Feb 2022 18:32:57 +0100
Subject: [PATCH 358/442] Create github release via CircleCI only for mozilla
 fork (#349)

* Create github release via circleci only for mozilla fork

 - The extension uses mozilla fork for translator artifacts
   -- Hence create github release via circleci only when
      running in mozilla fork

* Small refactoring in ci script
---
 .circleci/config.yml | 51 +++++++++++++++++++++++++++-----------------
 1 file changed, 32 insertions(+), 19 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index fbea34e18..7275fd4be 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -20,13 +20,18 @@ jobs:
           working_directory: build-wasm
           command: |
             ARTIFACT_BASE="bergamot-translator-worker"
+            ARTIFACT_SUFFIX="with-wormhole"
+            ARTIFACT_FINAL=$ARTIFACT_BASE-$ARTIFACT_SUFFIX
+
             if [[ -f "$ARTIFACT_BASE.js" && -f "$ARTIFACT_BASE.wasm" ]]; then
               echo "Artifacts Successfully Generated"
               mkdir ../artifacts
-              cp $ARTIFACT_BASE.wasm ../artifacts/$ARTIFACT_BASE-with-wormhole.wasm
-              cp $ARTIFACT_BASE.js ../artifacts/$ARTIFACT_BASE-with-wormhole.js
-              shasum -a 256 ../artifacts/* > ../artifacts/SHA256-1
-              cp ../BERGAMOT_VERSION ../artifacts/
+              cp $ARTIFACT_BASE.wasm ../artifacts/$ARTIFACT_FINAL.wasm
+              cp $ARTIFACT_BASE.js ../artifacts/$ARTIFACT_FINAL.js
+              cd ../artifacts
+              shasum -a 256 $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize-$ARTIFACT_SUFFIX
+              ls -lsa $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize-$ARTIFACT_SUFFIX
+              cp ../BERGAMOT_VERSION .
             else
               echo "Failure: Artifacts Not Present"
               exit 1
@@ -61,12 +66,17 @@ jobs:
           working_directory: build-wasm
           command: |
             ARTIFACT_BASE="bergamot-translator-worker"
+            ARTIFACT_SUFFIX="without-wormhole"
+            ARTIFACT_FINAL=$ARTIFACT_BASE-$ARTIFACT_SUFFIX
+
             if [[ -f "$ARTIFACT_BASE.js" && -f "$ARTIFACT_BASE.wasm" ]]; then
               echo "Artifacts Successfully Generated"
               mkdir ../artifacts
-              cp $ARTIFACT_BASE.wasm ../artifacts/$ARTIFACT_BASE-without-wormhole.wasm
-              cp $ARTIFACT_BASE.js ../artifacts/$ARTIFACT_BASE-without-wormhole.js
-              shasum -a 256 ../artifacts/* > ../artifacts/SHA256-2
+              cp $ARTIFACT_BASE.wasm ../artifacts/$ARTIFACT_FINAL.wasm
+              cp $ARTIFACT_BASE.js ../artifacts/$ARTIFACT_FINAL.js
+              cd ../artifacts
+              shasum -a 256 $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize-$ARTIFACT_SUFFIX
+              ls -lsa $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize-$ARTIFACT_SUFFIX
             else
               echo "Failure: Artifacts Not Present"
               exit 1
@@ -82,20 +92,23 @@ jobs:
           destination: "wasm-without-wormhole"
 
   publish_to_github:
-     docker:
-       - image: cibuilds/github:0.10
-     steps:
-       - attach_workspace:
+    docker:
+      - image: cibuilds/github:0.10
+    steps:
+      - attach_workspace:
           # Must be absolute path or relative path from working_directory
           at: ./
-       - run:
-          name: "Publish Release on GitHub"
-          command: |
-            export TAG_VERSION=$(cat ./artifacts/BERGAMOT_VERSION)
-            ls -lsa ./artifacts/ > ./artifacts/FILESIZES
-            cat ./artifacts/SHA256-1 ./artifacts/SHA256-2 > ./artifacts/SHA256
-            rm ./artifacts/SHA256-1 ./artifacts/SHA256-2 ./artifacts/BERGAMOT_VERSION
-            ghr -t ${GHTOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${TAG_VERSION} ./artifacts/
+      - when:
+          condition:
+            equal: [ 'https://github.com/mozilla/bergamot-translator', << pipeline.project.git_url >> ]
+          steps:
+            - run:
+                name: "Publish Release on GitHub"
+                command: |
+                  export TAG_VERSION=$(cat ./artifacts/BERGAMOT_VERSION)
+                  cat ./artifacts/sha256-filesize-without-wormhole ./artifacts/sha256-filesize-with-wormhole >> ./artifacts/sha256-filesize
+                  rm ./artifacts/sha256-filesize-without-wormhole ./artifacts/sha256-filesize-with-wormhole ./artifacts/BERGAMOT_VERSION
+                  ghr -t ${GHTOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${TAG_VERSION} ./artifacts/
 
 workflows:
   build:

From 9eb243725bdfa8fef32225c46d4673af6f144251 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 17 Feb 2022 23:32:13 +0100
Subject: [PATCH 359/442] Bump version to 0.4.1 (#356)

---
 BERGAMOT_VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BERGAMOT_VERSION b/BERGAMOT_VERSION
index fb7a04cff..5aff472dd 100644
--- a/BERGAMOT_VERSION
+++ b/BERGAMOT_VERSION
@@ -1 +1 @@
-v0.4.0
+v0.4.1

From 1f98f971a526d3e8f0572a0be5fb3fd3b4c843ac Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Tue, 22 Feb 2022 20:25:34 +0000
Subject: [PATCH 360/442] Improve handling HTML special cases (#312)

- Prefer spreading markup over a full word.
- Ignore certain tags that are unlikely to be supposed to be translated,
  such as `<code>` and `<samp>`.
- Never treat `<wbr>` as a space.
- Allow for inconsistent cases in tag names.
- Fix bug where void elements were inserted multiple times.
- Better handling of whitespace around punctuation.
- Ignore parsing `<noscript>` to be compatible with Firefox.
- Improvements to documentation and readability of `HTML` and `Scanner`
  classes.

Fixes: #313, #339
---
 bergamot-translator-tests         |   2 +-
 src/tests/units/html_tests.cpp    |  76 +++++++
 src/translator/annotation.h       |  35 ++++
 src/translator/html.cpp           | 325 ++++++++++++++++++++----------
 src/translator/html.h             | 212 ++++++++++++++-----
 src/translator/response_options.h |   2 +-
 src/translator/xh_scanner.cpp     |  48 +++--
 src/translator/xh_scanner.h       |   6 +
 8 files changed, 533 insertions(+), 173 deletions(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 3c0f95a17..3776609ce 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 3c0f95a1775a74f5db441aa2f17ceb7437679022
+Subproject commit 3776609ce5f7a238245e303efaa007b2d5078180
diff --git a/src/tests/units/html_tests.cpp b/src/tests/units/html_tests.cpp
index 2b71784fb..96eff5aad 100644
--- a/src/tests/units/html_tests.cpp
+++ b/src/tests/units/html_tests.cpp
@@ -172,6 +172,16 @@ TEST_CASE("Do not abort if the input is just empty element") {
   CHECK(response.target.text == "<p></p>");
 }
 
+TEST_CASE("Tag names are case insensitive") {
+  // Tests <P> vs </p> and <BR> should be recognized as a void tag <br>.
+  // <B> should be recognized as inline.
+  std::string test_str("<P><B>Spa</B>ce<BR>please?</p>");
+
+  std::string input(test_str);
+  HTML html(std::move(input), true);
+  CHECK(input == "Spa ce\n\nplease?");
+}
+
 TEST_CASE("Test case html entities") {
   // These are all entities I would expect in innerHTML, since all other entities
   // can be encoded as UTF-8 so there's no need to encode them through &...; when
@@ -618,6 +628,72 @@ TEST_CASE("Test comment") {
   CHECK(response.target.text == test_str);
 }
 
+TEST_CASE("Test <wbr> element") {
+  std::string test_str("hel<wbr>lo");
+
+  std::string input(test_str);
+  HTML html(std::move(input), true);
+  CHECK(input == "hello");
+}
+
+TEST_CASE("Test <wbr> element (case-insensitive)") {
+  std::string test_str("hel<WBR>lo");
+
+  std::string input(test_str);
+  HTML html(std::move(input), true);
+  CHECK(input == "hello");
+}
+
+TEST_CASE("Test ignored element (nested)") {
+  std::string test_str("foo <var><var>nested</var></var> bar");
+  std::string expected_str("foo  <var><var>nested</var></var>bar");
+
+  std::string input(test_str);
+  HTML html(std::move(input), true);
+  CHECK(input == "foo  bar");
+
+  Response response;
+  std::string sentence_str("foo  bar");
+  std::vector<string_view> sentence{
+      string_view(sentence_str.data() + 0, 3),  // foo
+      string_view(sentence_str.data() + 3, 1),  // _
+      string_view(sentence_str.data() + 4, 4),  // _bar
+      string_view(sentence_str.data() + 8, 0),  // ""
+  };
+  response.source.appendSentence("", sentence.begin(), sentence.end());
+  response.target.appendSentence("", sentence.begin(), sentence.end());
+  response.alignments = {identity_matrix<float>(4)};
+
+  html.restore(response);
+  CHECK(response.source.text == expected_str);
+  CHECK(response.target.text == expected_str);
+}
+
+TEST_CASE("Test ignored element (with entity)") {
+  std::string test_str("foo <var>&amp;</var> bar");
+  std::string expected_str("foo  <var>&amp;</var>bar");
+
+  std::string input(test_str);
+  HTML html(std::move(input), true);
+  CHECK(input == "foo  bar");
+
+  Response response;
+  std::string sentence_str("foo  bar");
+  std::vector<string_view> sentence{
+      string_view(sentence_str.data() + 0, 3),  // foo
+      string_view(sentence_str.data() + 3, 1),  // _
+      string_view(sentence_str.data() + 4, 4),  // _bar
+      string_view(sentence_str.data() + 8, 0),  // ""
+  };
+  response.source.appendSentence("", sentence.begin(), sentence.end());
+  response.target.appendSentence("", sentence.begin(), sentence.end());
+  response.alignments = {identity_matrix<float>(4)};
+
+  html.restore(response);
+  CHECK(response.source.text == expected_str);
+  CHECK(response.target.text == expected_str);
+}
+
 TEST_CASE("End-to-end translation", "[!mayfail]") {
   std::string input("<p>I <b>like</b> to <u>drive</u> this car.</p>");
   HTML html(std::move(input), true);
diff --git a/src/translator/annotation.h b/src/translator/annotation.h
index 785d49dfe..5a17dfcfe 100644
--- a/src/translator/annotation.h
+++ b/src/translator/annotation.h
@@ -185,6 +185,41 @@ struct AnnotatedText {
   /// Returns a ByteRange representing sentence corresponding to sentenceIdx.
   ByteRange sentenceAsByteRange(size_t sentenceIdx) const { return annotation.sentence(sentenceIdx); }
 
+  /// Utility function to call `fun` on each word (subword token effectively) in
+  /// an `AnnotatedText`. `fun` is called with the `ByteRange`, the `string_view`
+  /// with the word, and a `bool` to indicate whether it is the last word in the
+  /// `AnnotatedText`, which is also the ending whitespace slot of AnnotatedText.
+  template <typename Fun>
+  AnnotatedText apply(Fun fun) const {
+    AnnotatedText out;
+
+    for (size_t sentenceIdx = 0; sentenceIdx < numSentences(); ++sentenceIdx) {
+      std::string sentence;
+      std::vector<ByteRange> tokens;
+
+      std::string prefix = fun(annotation.gap(sentenceIdx), gap(sentenceIdx), false);
+
+      for (size_t wordIdx = 0; wordIdx < numWords(sentenceIdx); ++wordIdx) {
+        std::string token = fun(wordAsByteRange(sentenceIdx, wordIdx), word(sentenceIdx, wordIdx), false);
+        tokens.push_back(ByteRange{sentence.size(), sentence.size() + token.size()});
+        sentence += token;
+      }
+
+      // Convert our ByteRanges to string_views since that's what appendSentence
+      // expects
+      std::vector<marian::string_view> views(tokens.size());
+      std::transform(tokens.begin(), tokens.end(), views.begin(), [&](ByteRange const &range) {
+        return marian::string_view(sentence.data() + range.begin, range.size());
+      });
+
+      out.appendSentence(prefix, views.begin(), views.end());
+    }
+
+    out.appendEndingWhitespace(fun(annotation.gap(numSentences()), gap(numSentences()), true));
+
+    return out;
+  }
+
  private:
   string_view asStringView(const ByteRange &byteRange) const {
     return string_view(text.data() + byteRange.begin, byteRange.size());
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index 242572db0..ed42b9117 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -1,21 +1,23 @@
 #include "html.h"
 
+#include <algorithm>
+
 #include "response.h"
 #include "xh_scanner.h"
 
 namespace {
-using marian::string_view;
 using marian::bergamot::AnnotatedText;
 using marian::bergamot::ByteRange;
 using marian::bergamot::HTML;
 using marian::bergamot::Response;
 
-void encodeEntities(string_view const &input, std::string &output) {
+/// Encodes the minimum of HTML entities.
+void encodeEntities(marian::string_view const &input, std::string &output) {
   output.clear();
   output.reserve(input.size());  // assumes there are no entities in most cases
 
-  for (auto it = input.begin(); it != input.end(); ++it) {
-    switch (*it) {
+  for (char it : input) {
+    switch (it) {
       case '&':
         output.append("&amp;");
         break;
@@ -35,19 +37,30 @@ void encodeEntities(string_view const &input, std::string &output) {
       //   output.append("&apos;");
       //   break;
       default:
-        output.push_back(*it);
+        output.push_back(it);
         break;
     }
   }
 }
 
-size_t countPrefixWhitespaces(string_view const &input) {
+/// Counts number of whitespace characters at the start of the input. Used
+/// for determining where to insert an open or close tag.
+size_t countPrefixWhitespaces(marian::string_view const &input) {
   size_t size = 0;
   while (size < input.size() && std::isspace(input[size])) ++size;
   return size;
 }
 
-// Very simple replacement for std::format introduced in C++20
+std::string toLowerCase(std::string_view const &input) {
+  std::string out;
+  out.resize(input.size());
+  std::transform(input.begin(), input.end(), out.begin(), [](unsigned char c) { return std::tolower(c); });
+  return out;
+}
+
+/// Very simple replacement for std::format introduced in C++20. Only supports
+/// replacing `{}` in the template string with whatever `operator<<` for that
+/// type turns it into.
 std::string format(std::string const &formatTemplate) { return formatTemplate; }
 
 template <typename Arg>
@@ -68,14 +81,14 @@ std::string format(std::string const &formatTemplate, Arg arg, Args... args) {
   return os.str();
 }
 
-// Syntactic sugar around rbegin() and rend() that allows me to write
-// `for (auto &&item : reversed(container))` instead of the needlessly verbose
-// `for (auto it = container.rbegin(); it != container.rend(); ++it)`
+/// Syntactic sugar around rbegin() and rend() that allows me to write
+/// `for (auto &&item : reversed(container))` instead of the needlessly verbose
+/// `for (auto it = container.rbegin(); it != container.rend(); ++it)`
 template <typename T>
-class reversed {
+class Reversed {
  public:
-  typedef typename T::const_reverse_iterator iterator;
-  explicit reversed(T const &container) : container_(container){};
+  using iterator = typename T::const_reverse_iterator;
+  explicit Reversed(T const &container) : container_(container){};
   iterator begin() const { return container_.rbegin(); }
   iterator end() const { return container_.rend(); }
 
@@ -83,11 +96,10 @@ class reversed {
   T const &container_;
 };
 
-bool contains(std::unordered_set<std::string> const &set, std::string const &name) {
-  return set.find(name) != set.end();
-}
-
-void diffTags(HTML::Taint const &prev, HTML::Taint const &curr, HTML::Taint &opening, HTML::Taint &closing) {
+/// When comparing two tag stacks, determine which tags need to be closed and
+/// opened to get from one stack to the other.
+void diffTags(HTML::TagStack const &prev, HTML::TagStack const &curr, HTML::TagStack &opening,
+              HTML::TagStack &closing) {
   opening.clear();
   closing.clear();
 
@@ -98,9 +110,11 @@ void diffTags(HTML::Taint const &prev, HTML::Taint const &curr, HTML::Taint &ope
     if (i >= curr.size() || prev[i] != curr[i]) break;
 
   // Only nodes of type ELEMENT can have children and thus would need a closing tag.
+  // NOLINTNEXTLINE(bugprone-narrowing-conversions)
   std::copy_if(prev.begin() + i, prev.end(), std::back_inserter(closing),
                [&](HTML::Tag *tag) { return tag->type == HTML::Tag::ELEMENT; });
 
+  // NOLINTNEXTLINE(bugprone-narrowing-conversions)
   opening.insert(opening.end(), curr.begin() + i, curr.end());
 }
 
@@ -108,42 +122,24 @@ bool intersects(ByteRange const &range, HTML::Span const &span) {
   return range.begin <= span.end && range.end >= span.begin;
 };
 
-bool containsTag(HTML::Taint const &stack, HTML::Tag const *tag) {
+bool contains(HTML::TagNameSet const &set, std::string_view const &name) { return set.find(name) != set.end(); }
+
+bool contains(HTML::TagStack const &stack, HTML::Tag const *tag) {
   return std::find(stack.rbegin(), stack.rend(), tag) != stack.rend();
 }
 
-template <typename Fun>
-AnnotatedText apply(AnnotatedText const &in, Fun fun) {
-  AnnotatedText out;
-
-  for (size_t sentenceIdx = 0; sentenceIdx < in.numSentences(); ++sentenceIdx) {
-    std::string sentence;
-    std::vector<ByteRange> tokens;
+/// Is tag stack B an extended version of A? I.e. same tags, but maybe a few
+/// more nested deeper.
+bool extends(HTML::TagStack const &b, HTML::TagStack const &a) {
+  if (a.size() > b.size()) return false;
 
-    std::string prefix = fun(in.annotation.gap(sentenceIdx), in.gap(sentenceIdx), false);
+  for (auto i = a.begin(), j = b.begin(); i != a.end(); ++i, ++j)
+    if (*i != *j) return false;
 
-    for (size_t wordIdx = 0; wordIdx < in.numWords(sentenceIdx); ++wordIdx) {
-      std::string token = fun(in.wordAsByteRange(sentenceIdx, wordIdx), in.word(sentenceIdx, wordIdx), false);
-      tokens.push_back(ByteRange{sentence.size(), sentence.size() + token.size()});
-      sentence += token;
-    }
-
-    // Convert our ByteRanges to string_views since that's what appendSentence
-    // expects
-    // TODO: extend AnnotatedText::appendSentence to accept str + ByteRanges
-    // directly
-    std::vector<string_view> views(tokens.size());
-    std::transform(tokens.begin(), tokens.end(), views.begin(),
-                   [&](ByteRange const &range) { return string_view(sentence.data() + range.begin, range.size()); });
-
-    out.appendSentence(prefix, views.begin(), views.end());
-  }
-
-  out.appendEndingWhitespace(fun(in.annotation.gap(in.numSentences()), in.gap(in.numSentences()), true));
-
-  return out;
+  return true;
 }
 
+/// Tests whether `response` has alignment info associated with it or not.
 bool hasAlignments(Response const &response) {
   // Test for each sentence individually as a sentence may be empty (or there)
   // might be no sentences, so just testing for alignments.empty() would not be
@@ -162,11 +158,12 @@ bool hasAlignments(Response const &response) {
   return true;
 }
 
-// Little helper class to append HTML to a token
+/// Helper class to append HTML tags to a token. Also makes sure the token is
+/// encoded as valid HTML.
 class TokenFormatter {
  public:
-  explicit TokenFormatter(string_view token)
-      : html_(), offset_(0), whitespaceOffset_(0), whitespaceSize_(countPrefixWhitespaces(token)), closeLeft_(true) {
+  explicit TokenFormatter(marian::string_view token)
+      : offset_(0), whitespaceOffset_(0), whitespaceSize_(countPrefixWhitespaces(token)), closeLeft_(true) {
     // Do encoding of any entities that popped up in the translation
     encodeEntities(token, html_);
   }
@@ -174,12 +171,12 @@ class TokenFormatter {
   std::string &&html() { return std::move(html_); }
 
   // Append the markup necessary for moving from `prev` set of tags to `curr`.
-  void append(HTML::Taint const &prev, HTML::Taint const &curr) {
-    HTML::Taint opening, closing;
+  void append(HTML::TagStack const &prev, HTML::TagStack const &curr) {
+    HTML::TagStack opening, closing;
 
     diffTags(prev, curr, opening, closing);
 
-    for (HTML::Tag const *tag : reversed(closing)) {
+    for (HTML::Tag const *tag : Reversed(closing)) {
       assert(tag->type == HTML::Tag::ELEMENT);
       std::string closeTag = format("</{}>", tag->name);
       html_.insert(offset_ + (closeLeft_ ? 0 : whitespaceSize_), closeTag);
@@ -232,6 +229,8 @@ class TokenFormatter {
   bool closeLeft_;
 };
 
+/// Count the number of tokens in an AnnotatedText. Used to assert we're not
+/// running out of sync when creating vectors that describe each token.
 size_t debugCountTokens(AnnotatedText const &text) {
   size_t tokens = 1;  // for the ending gap
   for (size_t sentenceIdx = 0; sentenceIdx < text.numSentences(); ++sentenceIdx) {
@@ -240,11 +239,87 @@ size_t debugCountTokens(AnnotatedText const &text) {
   return tokens;
 }
 
+/// Helper function that consumes a tag as if it is a special tag, except that
+/// it takes nesting into account. I.e. `<a><a></a></a>` will be consumed to the
+// last `</a>`. Assumes TT_TAG_START is already consumed, which was necessary
+/// to determine whether this was an element that needed to be ignored.
+void consumeIgnoredTag(markup::Scanner &scanner, HTML::Tag &tag, std::string const &name) {
+  // Only full elements can be consumed this way. With void tags we don't know
+  // where to stop scanning. All other types cannot be nested anyway.
+  assert(tag.type == HTML::Tag::ELEMENT);
+
+  // TT_TAG_START is already consumed.
+  markup::Scanner::TokenType token;
+  size_t inside = 0;
+
+  // Consume the full open tag, i.e. all its attributes
+  while (!inside) {
+    token = scanner.next();
+    switch (token) {
+      case markup::Scanner::TT_ERROR:
+        ABORT("HTML parse error");
+      case markup::Scanner::TT_EOF:
+        ABORT("Did not find closing tag </{}>", name);
+      case markup::Scanner::TT_ATTRIBUTE:
+        tag.attributes += format(" {}=\"{}\"", scanner.attribute(), scanner.value());
+        break;
+      default:
+        // Not an attribute! Must be something inside the body or the closing
+        // tag already. Time to jump to the next loop.
+        ++inside;
+        break;
+    }
+  }
+
+  // Last token was something that would have triggered Scanner::scanBody(),
+  // which sets value() to start pointing at the body.
+  const char *start = scanner.start();
+
+  // Consume the rest of the HTML until (including) the final closing tag. We
+  // start with the token that caused the previous loop to fall into the default
+  // case.
+  while (inside) {
+    switch (token) {
+      case markup::Scanner::TT_ERROR:
+        ABORT("HTML parse error");
+      case markup::Scanner::TT_EOF:
+        ABORT("Did not find closing tag </{}>");
+      case markup::Scanner::TT_TAG_START:
+        // Note: Looking specifically for only our own type of tag so we don't
+        // have to care about whether other tags we encounter are void tags or
+        // not. Does assume the HTML is valid, as no stack is kept.
+        if (toLowerCase(scanner.tag()) == name) ++inside;
+        break;
+      case markup::Scanner::TT_TAG_END:
+        if (toLowerCase(scanner.tag()) == name) --inside;
+        break;
+      default:
+        break;
+    }
+
+    // Only continue scanning if we're still inside. We could have just read the
+    // TT_TAG_END token that ended this element, and we don't want to continue
+    // consuming tokens at that point.
+    if (inside) token = scanner.next();
+  }
+
+  // Only a TAG_END could have stopped the previous loop. We take the start
+  // of the final closing tag as the end of our data.
+  assert(token == markup::Scanner::TT_TAG_END);
+  const char *end = scanner.start();
+
+  // All data between the end of the first open element, and the start of the
+  // last close element, we just treat as raw data that will be printed when
+  // this tag is eventually printed.
+  assert(end >= start);
+  tag.data = std::string_view(start, end - start);
+}
+
 }  // namespace
 
 namespace marian::bergamot {
 
-// Formatters used for exception messages combined with format()
+/// Formatters used for formatting error messages in ABORT() calls.
 std::ostream &operator<<(std::ostream &out, HTML::Tag const *tag) {
   if (tag == nullptr) return out << "[nullptr]";
   switch (tag->type) {
@@ -262,7 +337,7 @@ std::ostream &operator<<(std::ostream &out, HTML::Tag const *tag) {
   return out << "[Unknown tag type]";
 }
 
-std::ostream &operator<<(std::ostream &out, HTML::Taint const &tags) {
+std::ostream &operator<<(std::ostream &out, HTML::TagStack const &tags) {
   for (auto it = tags.begin(); it != tags.end(); ++it) {
     if (it != tags.begin()) out << ' ';
     out << *it;
@@ -270,18 +345,20 @@ std::ostream &operator<<(std::ostream &out, HTML::Taint const &tags) {
   return out;
 }
 
-HTML::HTML(std::string &&source, bool process_markup, Options &&options) : options_(std::move(options)) {
-  if (!process_markup) return;
+HTML::HTML(std::string &&source, bool processMarkup, Options &&options) : options_(std::move(options)) {
+  if (!processMarkup) return;
 
   std::string original = std::move(source);
   markup::instream in(original.data(), original.data() + original.size());
   markup::Scanner scanner(in);
   source.clear();  // source is moved out of, so should be clear anyway
 
-  Tag *tag;
-  Taint stack;
-  bool addSentenceBreak = false;
-  bool addSpace = false;
+  Tag *tag = nullptr;             // current tag (after opening at least)
+  TagStack stack;                 // stack of currently open tags
+  bool addSentenceBreak = false;  // whether to add a sentence break next text segment
+  bool addWordBreak = false;      // whether to add a word break next text segment
+
+  // Starting point: an empty span with no open tags.
   spans_.push_back(Span{0, 0, {}});
 
   bool stop = false;
@@ -298,13 +375,14 @@ HTML::HTML(std::string &&source, bool process_markup, Options &&options) : optio
         // If the previous segment was the open or close tag of a block element
         // we treat the text after it as a new sentence.
         if (addSentenceBreak) {
-          if (!(source.empty() || (source.size() > 2 && source.substr(source.size() - 2) == ""))) {
+          // If there isn't already a \n\n at the end of source...
+          if (source.size() >= 2 && source.substr(source.size() - 2) != "\n\n") {
             stack.push_back(makeTag({Tag::WHITESPACE}));
             // Important: span->size() == 0 to make it behave as a void element.
             // Also important: position before the \n\n tokens, not after, to
             // make it easier to remove them later through apply().
             spans_.push_back(Span{source.size(), source.size(), stack});
-            source.append("\n\n");  // TODO assumes ssplit-mode = wrapped_text
+            source.append("\n\n");  // Should work with ssplit-mode = wrapped_text
             stack.pop_back();
           }
           addSentenceBreak = false;
@@ -312,24 +390,27 @@ HTML::HTML(std::string &&source, bool process_markup, Options &&options) : optio
 
         // If the previous segment was an open or close tag, it might be best
         // to add a space to make sure we don't append to the previous word.
-        if (addSpace) {
-          if (options_.substituteInlineTagsWithSpaces && !source.empty() && !std::isspace(source.back()) &&
-              !std::isspace(scanner.value()[0])) {
+        if (addWordBreak) {
+          // Only add the space when it would be inside a word. Do not add it if
+          // it would be between a word and punctuation.
+          if (options_.substituteInlineTagsWithSpaces && isContinuation(source, scanner.value())) {
             source.push_back(' ');
           }
-          addSpace = false;
+          addWordBreak = false;
         }
 
+        // Store which tags were open when this span of text was encountered.
         auto begin = source.size();
         source.append(scanner.value());
         spans_.push_back(Span{begin, source.size(), stack});
       } break;
 
       case markup::Scanner::TT_TAG_START: {
-        std::string name(scanner.tag());
+        std::string name = toLowerCase(scanner.tag());
 
         // Tag *tag is used by attribute parsing
-        tag = makeTag({contains(options_.voidTags, name) ? Tag::VOID_ELEMENT : Tag::ELEMENT, std::move(name)});
+        auto type = contains(options_.voidTags, name) ? Tag::VOID_ELEMENT : Tag::ELEMENT;
+        tag = makeTag({type, std::string(scanner.tag())});
 
         stack.push_back(tag);
 
@@ -341,39 +422,48 @@ HTML::HTML(std::string &&source, bool process_markup, Options &&options) : optio
           stack.pop_back();
         }
 
+        // Ignored tags have same semantics as void tags with regards to moving
+        // them around with the rest of the content.
+        if (contains(options_.ignoredTags, name)) {
+          consumeIgnoredTag(scanner, *tag, name);
+          spans_.push_back(Span{source.size(), source.size(), stack});
+          stack.pop_back();
+        }
+
         // Treat non-inline HTML tags as spaces that break up words.
-        if (!contains(options_.inlineTags, tag->name)) {
+        if (!contains(options_.inlineTags, name)) {
           addSentenceBreak = true;
-        } else {
-          addSpace = true;
+        } else if (!contains(options_.inWordTags, name)) {
+          addWordBreak = true;
         }
       } break;
 
-      case markup::Scanner::TT_TAG_END:
+      case markup::Scanner::TT_TAG_END: {
+        std::string tagName = toLowerCase(scanner.tag());
         // If this is the closing bit of a void tag, i.e. triggered by the "/>"
         // bit of "<img/>", then completely ignore it.
-        if (contains(options_.voidTags, std::string(scanner.tag()))) break;
+        if (contains(options_.voidTags, tagName)) break;
 
         ABORT_IF(stack.empty(), "Encountered more closing tags ({}) than opening tags", scanner.tag());
 
-        ABORT_IF(stack.back()->name != scanner.tag(), "Encountered unexpected closing tag </{}>, stack is {}",
-                 scanner.tag(), stack);
+        ABORT_IF(toLowerCase(stack.back()->name) != toLowerCase(scanner.tag()),
+                 "Encountered unexpected closing tag </{}>, stack is {}", scanner.tag(), stack);
 
         // What to do with "<u></u>" case, where tag is immediately closed
         // so it never makes it into the taint of any of the spans? This adds
         // an empty span so it still gets recorded in spans_.
-        if (spans_.empty() || !containsTag(spans_.back().tags, stack.back()))
+        if (spans_.empty() || !contains(spans_.back().tags, stack.back()))
           spans_.push_back(Span{source.size(), source.size(), stack});
 
         stack.pop_back();
 
         // Add space if necessary
-        if (!contains(options_.inlineTags, std::string(scanner.tag()))) {
+        if (!contains(options_.inlineTags, tagName)) {
           addSentenceBreak = true;
-        } else {
-          addSpace = true;
+        } else if (!contains(options_.inWordTags, tagName)) {
+          addWordBreak = true;
         }
-        break;
+      } break;
 
       case markup::Scanner::TT_ATTRIBUTE:
         assert(tag != nullptr);
@@ -448,10 +538,10 @@ void HTML::restore(Response &response) {
 
   // Find for every token in target the token in source that best matches.
   std::vector<std::vector<size_t>> alignments;
-  hardAlignments(response, alignments);
+  hardAlignments(response, alignments, sourceTokenSpans);
 
   std::vector<SpanIterator> targetTokenSpans;
-  copyTaint(response, alignments, sourceTokenSpans, targetTokenSpans);
+  copyTagStack(response, alignments, sourceTokenSpans, targetTokenSpans);
   assert(targetTokenSpans.size() == debugCountTokens(response.target));
 
   AnnotatedText target = restoreTarget(response.target, targetTokenSpans);
@@ -466,7 +556,7 @@ AnnotatedText HTML::restoreSource(AnnotatedText const &in, std::vector<SpanItera
                                  // and the while-loop below will do the rest
   assert(prevIt == spans_.end() || prevIt->tags.empty());
 
-  return apply(in, [&](ByteRange range, string_view token, bool last) {
+  return in.apply([&](ByteRange range, string_view token, bool last) {
     TokenFormatter formatter(token);
 
     // Potential issue: spans and tokens can intersect, e.g.
@@ -475,9 +565,11 @@ AnnotatedText HTML::restoreSource(AnnotatedText const &in, std::vector<SpanItera
     //   spans     |1|   |2|    |3333| (so only 2 is tainted with <p><u>, others only <p>)
     //  tokens     |111111111111111|2|
     //
-    // Now 1 covers span 1 to 3, so what taint should it get? Just <p>, or <p><u>?
-    // Note: only relevant if isBlockElement is used. If we just insert spaces
-    // around all elements, every segment of `hello` will be a token.
+    // Now 1 covers span 1 to 3, so what taint should it get? Just `<p>`, or
+    // `<p><u>`?
+    // Note: only relevant if `substituteInlineTagsWithSpaces` is true. If we
+    // just insert spaces around all elements, every segment of `hello` will be
+    // a token.
 
     // Seek to the last span that overlaps with this token
     while (true) {
@@ -494,7 +586,7 @@ AnnotatedText HTML::restoreSource(AnnotatedText const &in, std::vector<SpanItera
 
     // TODO: This is just the taint of the last span, not the ones in between.
     // This makes us lose some markup of parts of tokens as described above.
-    sourceTokenSpans.push_back(prevIt);
+    sourceTokenSpans.emplace_back(prevIt);
 
     return std::move(formatter.html());
   });
@@ -503,27 +595,28 @@ AnnotatedText HTML::restoreSource(AnnotatedText const &in, std::vector<SpanItera
 AnnotatedText HTML::restoreTarget(AnnotatedText const &in, std::vector<SpanIterator> const &targetTokenSpans) {
   auto prevSpan = spans_.cbegin();
   auto targetSpanIt = targetTokenSpans.begin();
+  auto straggerSpanIt = spans_.cbegin();
 
-  AnnotatedText out = apply(in, [&](ByteRange range, string_view token, bool last) {
+  AnnotatedText out = in.apply([&]([[maybe_unused]] ByteRange range, string_view token, bool last) {
     TokenFormatter formatter(token);
 
     // First we scan through spans_ to catch up to the span assigned to this
     // token. We're only interested in empty spans (empty and void elements)
-    for (auto span_it = prevSpan; span_it < *targetSpanIt; span_it++) {
+    for (; straggerSpanIt < *targetSpanIt; ++straggerSpanIt) {
       // We're only interested in empty spans or spans that would otherwise get
       // lost because they didn't align with anything between the spans in
       // targetSpanIt
       // TODO That std::find makes this O(N*N) NOT GOOD NOT GOOD
-      if (span_it->size() != 0 &&
-          std::find(targetTokenSpans.begin(), targetTokenSpans.end(), span_it) != targetTokenSpans.end())
+      if (straggerSpanIt->size() != 0 &&
+          std::find(targetTokenSpans.begin(), targetTokenSpans.end(), straggerSpanIt) != targetTokenSpans.end())
         continue;
 
-      formatter.append(prevSpan->tags, span_it->tags);
+      formatter.append(prevSpan->tags, straggerSpanIt->tags);
 
       // Note: here, not in 3rd part of for-statement because we don't want to
       // set prevSpan if the continue clause at the beginning of this for-loop
       // was hit.
-      prevSpan = span_it;
+      prevSpan = straggerSpanIt;
     }
 
     // Now do the same thing but for our target set of tags. Note that we cannot
@@ -539,7 +632,7 @@ AnnotatedText HTML::restoreTarget(AnnotatedText const &in, std::vector<SpanItera
       // the last token of the output. But lets assume someone someday changes
       // HardAlignments(), and then this for-loop will be necessary.
       // assert((*targetSpanIt)->tags.empty());
-      formatter.append((*targetSpanIt)->tags, HTML::Taint());
+      formatter.append((*targetSpanIt)->tags, HTML::TagStack());
     }
 
     prevSpan = *targetSpanIt;
@@ -559,8 +652,9 @@ HTML::Tag *HTML::makeTag(Tag &&tag) {
   return &pool_.front();
 }
 
-void HTML::copyTaint(Response const &response, std::vector<std::vector<size_t>> const &alignments,
-                     std::vector<SpanIterator> const &sourceTokenSpans, std::vector<SpanIterator> &targetTokenSpans) {
+void HTML::copyTagStack(Response const &response, std::vector<std::vector<size_t>> const &alignments,
+                        std::vector<SpanIterator> const &sourceTokenSpans,
+                        std::vector<SpanIterator> &targetTokenSpans) {
   size_t offset = 0;  // Sentence offset in sourceTokenSpans
 
   // Fill targetTokenSpans based on the alignments we just made up.
@@ -584,14 +678,25 @@ void HTML::copyTaint(Response const &response, std::vector<std::vector<size_t>>
 // to determine whether we should share the markup, or whether we should see
 // this token as a fresh start. This implementation will treat "hello[world]"
 // as 4 words, assuming its tokenised as something like `h ell o [ wor ld ]`.
-bool HTML::isContinuation(string_view prev, string_view str) {
+bool HTML::isContinuation(std::string_view prev, std::string_view str) const {
   if (options_.continuationDelimiters.empty()) return false;
   if (prev.empty() || str.empty()) return false;
   return options_.continuationDelimiters.find(str[0]) == std::string::npos &&
          options_.continuationDelimiters.find(prev.back()) == std::string::npos;
 }
 
-void HTML::hardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments) {
+bool HTML::isContinuation(marian::string_view prev, marian::string_view str) const {
+  return isContinuation(std::string_view(prev.data(), prev.size()), std::string_view(str.data(), str.size()));
+}
+
+/// Selects for each token in `response.target` a best source token from
+/// `response.source` and writes this selection to `alignments`. The source
+/// token spans are used to also look at the markup applied to each token to
+/// figure out which source token best represents each target token.
+void HTML::hardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments,
+                          std::vector<SpanIterator> const &sourceTokenSpans) {
+  size_t offset = 0;  // sentence offset in sourceTokenSpans
+
   // For each sentence...
   for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
     alignments.emplace_back();
@@ -600,14 +705,9 @@ void HTML::hardAlignments(Response const &response, std::vector<std::vector<size
     // Note: only search from 0 to N-1 because token N is end-of-sentence token
     // that can only align with the end-of-sentence token of the target
     for (size_t t = 0; t + 1 < response.target.numWords(sentenceIdx); ++t) {
-      size_t maxS = 0;
-      for (size_t s = 1; s + 1 < response.source.numWords(sentenceIdx); ++s) {
-        if (response.alignments[sentenceIdx][t][s] > response.alignments[sentenceIdx][t][maxS]) {
-          maxS = s;
-        }
-      }
-
-      alignments.back().push_back(maxS);
+      alignments.back().push_back(
+          std::max_element(response.alignments[sentenceIdx][t].begin(), response.alignments[sentenceIdx][t].end()) -
+          response.alignments[sentenceIdx][t].begin());
     }
 
     // Next, we try to smooth out these selected alignments with a few heuristics
@@ -622,7 +722,14 @@ void HTML::hardAlignments(Response const &response, std::vector<std::vector<size
         float currScore = response.alignments[sentenceIdx][t][currSentenceIdx];
         float prevScore = response.alignments[sentenceIdx][t - 1][prevSentenceIdx];
 
-        if (currScore >= prevScore) {
+        TagStack const &currTagStack = sourceTokenSpans[offset + 1 + currSentenceIdx]->tags;
+        TagStack const &prevTagStack = sourceTokenSpans[offset + 1 + prevSentenceIdx]->tags;
+
+        // If this token has more markup, or a better score than the previous
+        // token (and they together are part of a word-ish thing) then mark
+        // this word as aligning. Otherwise just copy the alignment source of
+        // the previous token.
+        if (extends(currTagStack, prevTagStack) || currScore >= prevScore) {
           // Apply this to all previous tokens in the word
           for (size_t i = t;; --i) {
             alignments.back()[i] = currSentenceIdx;
@@ -640,6 +747,8 @@ void HTML::hardAlignments(Response const &response, std::vector<std::vector<size
 
     // Always align target end with source end
     alignments.back().push_back(response.source.numWords(sentenceIdx) - 1);
+
+    offset += response.source.numWords(sentenceIdx) + 1;  // +1 for prefix gap
   }
 }
 
diff --git a/src/translator/html.h b/src/translator/html.h
index db57c0d10..c704c5904 100644
--- a/src/translator/html.h
+++ b/src/translator/html.h
@@ -2,51 +2,123 @@
 #define SRC_BERGAMOT_HTML_H_
 
 #include <forward_list>
+#include <set>
 #include <stdexcept>
 #include <string>
-#include <unordered_set>
+#include <string_view>
 
 #include "annotation.h"
+#include "data/types.h"
 #include "definitions.h"
 
-namespace marian {
-namespace bergamot {
+namespace marian::bergamot {
 
 struct Response;
 
+/// HTML class parses and removes HTML from input text, and places it back into
+/// the translated output text.
+///
+/// When parsing the HTML, it treats tags as markup, where a list of nested tags
+/// can be seen as a list of markups that are applicable to all the text that
+/// follows. This list is stored as a `TagStack`. Whenever an HTML tag opens or
+/// closes, a new TagStack is created to reflect that. TagStack used to be
+/// called `Taint` because it *tainted* the text it was associated with with
+/// those tags as markup. The text between tags themselves is stored in the
+/// input variable. In `spans_`, the TagStack that is associated with a
+/// substring of that text is stored.
+/// When transferring the HTML from the source text to the translated target
+/// text, the TagStacks are first associated with each of the subwords from the
+/// source text. Using hard alignment, each subword in the source text is linked
+/// to a subword in the target text. The TagStacks are then copied over these
+/// links. Finally, the HTML is inserted back into the target text by for each
+/// subword, comparing the TagStack from the previous word to that word, and
+/// opening and closing elements to make up for the difference.
+///
+/// There are a couple of complexities though:
+/// 1. Not all tags can be treated as markup applied to text. For example, an
+///    `<img>` does not contain text itself. Or `<i></i>` does not. We do want
+///    those tags to remain in the output though. We do this by associating
+///    them to an empty `Span`. When inserting HTML back into the translation
+///    input or output, we keep track of where in the `spans_` vector we are,
+///    and insert any elements from empty spans that we might have skipped over
+///    because empty spans are never linked to tokens/subwords. These are
+///    *stragglers* in some parts of the code, or *void* or *empty* elements in
+///    other parts.
+/// 2. Some tags should be treated as paragraph indicators, and break up
+///    sentences. These are the usual suspects like `<p>`, but also `<li>` and
+///    `<td>`, to make sure we don't translate two table cells into a single
+///    word. This is the `addSentenceBreak` flag in the HTML parsing bit.
+///    We mark these breaks with `\n\n` in the input text and with a special
+///    WHITESPACE tag that we treat as any other void tag. Hopefully this tag
+///    moves with the added `\n\n` and it is easy for us to remove it again.
+///    (in practise it is since these only occur at the end of sentences and
+///    the end of sentences are always aligned between source and target.)
+/// 3. We treat most tags as word-breaking. We do this by adding spaces just
+///    after where we saw the open or close tag occur. If there is already
+///    some whitespace in that place, we do not add extra spaces.
+/// 4. TODO
 class HTML {
  public:
+  using TagNameSet = std::set<std::string, std::less<>>;
+
+  /// Options struct that controls how HTML is interpreted.
   struct Options {
-    // List of elements for which we do not expect a closing tag, or self-closing
-    // elements in XHTML. See also https://developer.mozilla.org/en-US/docs/Glossary/Empty_element
-    // More relevant source of this list:
-    // https://searchfox.org/mozilla-central/rev/7d17fd1fe9f0005a2fb19e5d53da4741b06a98ba/dom/base/FragmentOrElement.cpp#1791
-    std::unordered_set<std::string> voidTags{"area",  "base",  "basefont", "bgsound", "br",    "col",
-                                             "embed", "frame", "hr",       "img",     "input", "keygen",
-                                             "link",  "meta",  "param",    "source",  "track", "wbr"};
-
-    std::unordered_set<std::string> inlineTags{"abbr",   "a", "b",    "em",    "i",    "kbd",    "mark", "math",
-                                               "output", "q", "ruby", "small", "span", "strong", "sub",  "sup",
-                                               "time",   "u", "var",  "wbr",   "ins",  "del",    "img"};
-
-    // List of characters that occur at the start of a token that indicate that
-    // the this token is probably *not* a continuation of a word. Set to empty
-    // to never mark a token as a continuation of the word.
-    // std::string continuationDelimiters = "\n ,.(){}[]";
-    std::string continuationDelimiters;
-
-    // Should we always add spaces to the places where tags used to be? I.e.
-    // `un<u>der</u>line` should become `un der line`?
+    /// List of elements for which we do not expect a closing tag, or
+    /// self-closing elements in XHTML. We do not need to see a closing tag
+    /// for these elements, and they cannot contain text or tags themselves.
+    /// See also:
+    /// https://developer.mozilla.org/en-US/docs/Glossary/Empty_element.
+    /// More relevant source of this list:
+    /// https://searchfox.org/mozilla-central/rev/7d17fd1fe9f0005a2fb19e5d53da4741b06a98ba/dom/base/FragmentOrElement.cpp#1791
+    TagNameSet voidTags{"area", "base",  "basefont", "bgsound", "br",   "col",   "embed",  "frame", "hr",
+                        "img",  "input", "keygen",   "link",    "meta", "param", "source", "track", "wbr"};
+
+    /// List of elements that are treated as inline, meaning they do not break
+    /// up sentences. Any element *not* in this list will cause the text that
+    /// follows its open or close tag to be treated as a separate sentence.
+    TagNameSet inlineTags{"abbr",   "a", "b",    "em",    "i",    "kbd",    "mark", "math",
+                          "output", "q", "ruby", "small", "span", "strong", "sub",  "sup",
+                          "time",   "u", "var",  "wbr",   "ins",  "del",    "img"};
+
+    /// List of elements that are, regardless of `substituteInlineTagsWithSpaces`,
+    /// not substituted with spaces. Technically almost all inline elements
+    /// should be treated like this, except `<br>` maybe, But in practice it
+    /// seems to be more effective to limit this set to just that one tag that
+    /// that can only really be used *inside* words: `<wbr>`.
+    /// See also: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/wbr
+    TagNameSet inWordTags{"wbr"};
+
+    /// List of elements we copy as is, but do parse as if they're HTML because
+    /// they could be nested. For <script> we just scan for </script> because
+    /// the script tag may not be nested, but that is not the case for these
+    /// elements per se. Some tags, like <script>, are ignored at the `Scanner`
+    /// level. See `xh_scanner.cpp/Scanner::scanAttribute()`.
+    TagNameSet ignoredTags{"code", "kbd", "samp", "var", "dir", "acronym", "math"};
+
+    /// List of characters that occur at the start of a token that indicate that
+    /// the this token is probably *not* a continuation of a word. This is also
+    /// used to determine whether there should be a space after a closing tag
+    /// or not. I.e. a `.` after a `</strong>` does not need to be separated by
+    /// an extra space.
+    std::string continuationDelimiters = "\n ,.(){}[]";
+
+    /// Should we always add spaces to the places where tags used to be? I.e.
+    /// `un<u>der</u>line` should become `un der line`? This does help with
+    /// retaining tags inside words, or with odd pages that use CSS to add
+    /// spacing between a lot of tags. Cases like `<td>` and `<li>` are already
+    /// covered by treating them as sentence splitting.
     bool substituteInlineTagsWithSpaces = true;
   };
 
+  /// Represents a tag, or markup that is being applied to a string of text.
+  /// We treat all elements except `ELEMENT` as void elements or empty elements.
   struct Tag {
     enum NodeType {
-      ELEMENT,
-      VOID_ELEMENT,
-      COMMENT,
-      PROCESSING_INSTRUCTION,
-      WHITESPACE,  // negative space
+      ELEMENT,                 // <b>...</b>
+      VOID_ELEMENT,            // <img>
+      COMMENT,                 // <!-- ... -->
+      PROCESSING_INSTRUCTION,  // <?...?>
+      WHITESPACE,              // A \n\n we inserted to break a sentence.
     };
 
     NodeType type;           // Type of the node
@@ -55,48 +127,94 @@ class HTML {
                              // entities and prefix whitespace)
     std::string data;        // Raw data of an element that just needs to be
                              // copied as is, e.g. <script> or <style>
-    // @TODO: if the original HTML stays in memory, we could replace
-    // `attributes` and `data` with string_views pointing to it.
   };
 
-  using Taint = std::vector<Tag *>;
+  /// Representation of markup that is being applied to a string of text. Order
+  /// matters as this represents how the tags are nested. The `Tag` objects
+  /// themselves are owned by `pool_`.
+  using TagStack = std::vector<Tag *>;
 
+  /// Span of text, with which a `TagStack` is associated. A span may be empty,
+  /// for example to represent the presence of an empty or VOID element.
   struct Span {
-    size_t begin;
-    size_t end;
-    Taint tags;  // Note: free pointers! Lifetime of tags is managed by pool_
+    size_t begin;   // Start offset in (plain text) source
+    size_t end;     // end offset in source
+    TagStack tags;  // Note: free pointers to memory owned by `pool_`.
     inline size_t size() const { return end - begin; }
   };
 
-  explicit HTML(std::string &&source, bool process_markup) : HTML(std::move(source), process_markup, HTML::Options{}){};
-  explicit HTML(std::string &&source, bool process_markup, Options &&options);
+  /// Parses HTML in `source` (if `processMarkup` is true). `source` is updated
+  /// to only contain the plain text extracted from the HTML. `HTML` instance
+  /// retains information about what tags are extracted from where to later
+  /// reconstruct the HTML in a `Response` object (both `source` and `target`).
+  explicit HTML(std::string &&source, bool processMarkup) : HTML(std::move(source), processMarkup, HTML::Options{}){};
+  explicit HTML(std::string &&source, bool processMarkup, Options &&options);
+
+  /// It is not save to copy a HTML instance.
+  HTML(const HTML &) = delete;
+
+  /// Moving is fine
+  HTML(HTML &&) = default;
+
+  /// Reconstructs (not perfectly) the HTML as it was parsed from `source`,
+  /// and uses alignment information to also reconstruct the same markup in
+  /// `response.target`.
   void restore(Response &response);
 
  private:
   using SpanIterator = std::vector<HTML::Span>::const_iterator;
   using AnnotatedText = marian::bergamot::AnnotatedText;
 
+  /// Reconstructs HTML in `response.source` (passed as `in`) and makes a list
+  /// `sourceTokenSpans` that associates a `Span` with each subword in `in`.
+  /// We later use these span pointers to copy tags. They're iterators (or
+  /// pointers into a list) to be able to compare whether one span came before
+  /// or after another span.
   AnnotatedText restoreSource(AnnotatedText const &in, std::vector<SpanIterator> &sourceTokenSpans);
+
+  /// Inserts the HTML into `response.target` (passed as `in`) based on
+  /// `targetTokenSpans`, which points to a `Span` for each token (subword) in
+  /// `response.target`.
   AnnotatedText restoreTarget(AnnotatedText const &in, std::vector<SpanIterator> const &targetTokenSpans);
-  void copyTaint(Response const &response, std::vector<std::vector<size_t>> const &alignments,
-                 std::vector<HTML::SpanIterator> const &sourceTokenSpans,
-                 std::vector<HTML::SpanIterator> &targetTokenSpans);
-  void hardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments);
-  bool isContinuation(string_view prev, string_view str);
-  // Allocates tag in pool_ (which then owns it) and gives a pointer to be used
-  // in Taints. Pointer is valid as long as this HTML instance lives on.
+
+  /// Utilities to test whether subword `str` is part of a word together with
+  /// the subword `prev`, or a separate word. Basically *does `str` start with
+  /// a space, but bit more complex to deal with punctuation.
+  bool isContinuation(marian::string_view prev, marian::string_view str) const;
+  bool isContinuation(std::string_view prev, std::string_view str) const;
+
+  /// Copies span pointers from the subwords/tokens from the source text to the
+  /// subwords of the target text in `targetTokenSpans` using alignment
+  /// information in `response`.
+  void copyTagStack(Response const &response, std::vector<std::vector<size_t>> const &alignments,
+                    std::vector<HTML::SpanIterator> const &sourceTokenSpans,
+                    std::vector<HTML::SpanIterator> &targetTokenSpans);
+
+  /// Turns the alignment scores in `response.alignments` into one source token
+  /// per target token. Has some heuristics to keep all target tokens of a
+  /// single word pointing to the same span, and prefers spans with more markup
+  /// over spans with less to try to retain as much of the input markup as
+  /// possible.
+  void hardAlignments(Response const &response, std::vector<std::vector<size_t>> &alignments,
+                      std::vector<HTML::SpanIterator> const &sourceTokenSpans);
+
+  /// Allocates a tag in `pool_` (which then owns it) and gives a pointer to be
+  /// used in TagStacks. Pointer is valid as long as this HTML instance lives on.
   Tag *makeTag(Tag &&tag);
 
+  /// HTML options associated with this parse.
   Options options_;
 
-  // List of text spans, and which tags are applied to them
+  /// List of spans of text in plain text `source`, and which tags are applied
+  /// to them.
   std::vector<Span> spans_;
 
-  // a pool of tags that we free when HTML goes out of scope
+  /// A pool of tags. `std::forward_list` because we do not want pointers to it
+  /// to be invalidated when new tags are allocated. This way it is easy to
+  /// deallocate them all when `HTML` goes out of scope.
   std::forward_list<Tag> pool_;
 };
 
-}  // namespace bergamot
-}  // namespace marian
+}  // namespace marian::bergamot
 
 #endif  // SRC_BERGAMOT_HTML_H_
diff --git a/src/translator/response_options.h b/src/translator/response_options.h
index b5867d00d..8ccfab856 100644
--- a/src/translator/response_options.h
+++ b/src/translator/response_options.h
@@ -19,7 +19,7 @@ struct ResponseOptions {
   bool qualityScores{false};  ///< Include quality-scores or not.
   bool alignment{false};      ///< Include alignments or not.
 
-  bool HTML{false};  /// Remove HTML tags from text and (TODO) insert in output.
+  bool HTML{false};  /// Remove HTML tags from text and insert in output.
 
   /// Whether to include sentenceMappings or not. Alignments require
   /// sentenceMappings and are available irrespective of this option if
diff --git a/src/translator/xh_scanner.cpp b/src/translator/xh_scanner.cpp
index 85eb7e972..724d02cb9 100644
--- a/src/translator/xh_scanner.cpp
+++ b/src/translator/xh_scanner.cpp
@@ -37,6 +37,11 @@ bool operator==(markup::string_ref const &str, const Char_t (&str2)[Len]) {
   return str.size == Len - 1 && std::memcmp(str.data, str2, Len - 1) == 0;
 }
 
+template <size_t N>
+constexpr size_t length(char const (&/*unused*/)[N]) {
+  return N - 1;
+}
+
 }  // end namespace
 
 namespace markup {
@@ -52,6 +57,8 @@ std::string_view Scanner::tag() const { return std::string_view(tagName_.data, t
 Scanner::TokenType Scanner::scanBody() {
   value_ = string_ref{input_.pos(), 0};
 
+  start_ = input_.pos();
+
   switch (input_.peek()) {
     case '\0':
       return TT_EOF;
@@ -97,15 +104,16 @@ Scanner::TokenType Scanner::scanAttribute() {
   switch (input_.peek()) {
     case '>':
       input_.consume();
-      if (equalsCaseInsensitive(tagName_, "script")) {
+
+      // Treat some elements as opaque, e.g. <script>, <style>
+      if (/*equalsCaseInsensitive(tagName_, "title") ||*/ equalsCaseInsensitive(tagName_, "script") ||
+          equalsCaseInsensitive(tagName_, "style") || equalsCaseInsensitive(tagName_, "textarea") ||
+          equalsCaseInsensitive(tagName_, "iframe") || equalsCaseInsensitive(tagName_, "noembed") ||
+          equalsCaseInsensitive(tagName_, "noscript") || equalsCaseInsensitive(tagName_, "noframes")) {
         // script is special because we want to parse the attributes,
         // but not the content
         scanFun_ = &Scanner::scanSpecial;
         return scanSpecial();
-      } else if (equalsCaseInsensitive(tagName_, "style")) {
-        // same with style
-        scanFun_ = &Scanner::scanSpecial;
-        return scanSpecial();
       } else {
         scanFun_ = &Scanner::scanBody;
         return scanBody();
@@ -198,10 +206,11 @@ Scanner::TokenType Scanner::scanAttribute() {
 // - TT_ENTITY_START
 // - TT_ERROR if unexpected character or end
 Scanner::TokenType Scanner::scanTag() {
+  start_ = input_.pos();
   if (input_.consume() != '<') return TT_ERROR;
 
-  bool is_tail = input_.peek() == '/';
-  if (is_tail) input_.consume();
+  bool isTail = input_.peek() == '/';
+  if (isTail) input_.consume();
 
   tagName_ = string_ref{input_.pos(), 0};
 
@@ -226,7 +235,7 @@ Scanner::TokenType Scanner::scanTag() {
 
   if (!input_.peek()) return TT_EOF;
 
-  if (is_tail) return input_.consume() == '>' ? TT_TAG_END : TT_ERROR;
+  if (isTail) return input_.consume() == '>' ? TT_TAG_END : TT_ERROR;
 
   scanFun_ = &Scanner::scanAttribute;
   return TT_TAG_START;
@@ -234,6 +243,7 @@ Scanner::TokenType Scanner::scanTag() {
 
 Scanner::TokenType Scanner::scanEntity(TokenType parentTokenType) {
   // `entity` includes starting '&' and ending ';'
+  start_ = input_.pos();
   string_ref entity{input_.pos(), 0};
   bool hasEnd = false;
 
@@ -312,11 +322,13 @@ bool Scanner::isWhitespace(char c) {
 
 Scanner::TokenType Scanner::scanComment() {
   if (gotTail_) {
+    start_ = input_.pos() - length("-->");  // minus "-->"
     scanFun_ = &Scanner::scanBody;
     gotTail_ = false;
     return TT_COMMENT_END;
   }
 
+  start_ = input_.pos();
   value_ = string_ref{input_.pos(), 0};
 
   while (true) {
@@ -325,7 +337,7 @@ Scanner::TokenType Scanner::scanComment() {
 
     if (endsWith(value_, "-->")) {
       gotTail_ = true;
-      value_.size -= 3;
+      value_.size -= length("-->");
       break;
     }
   }
@@ -334,11 +346,13 @@ Scanner::TokenType Scanner::scanComment() {
 
 Scanner::TokenType Scanner::scanProcessingInstruction() {
   if (gotTail_) {
+    start_ = input_.pos() - length("?>");
     scanFun_ = &Scanner::scanBody;
     gotTail_ = false;
     return TT_PROCESSING_INSTRUCTION_END;
   }
 
+  start_ = input_.pos();
   value_ = string_ref{input_.pos(), 0};
 
   while (true) {
@@ -347,7 +361,7 @@ Scanner::TokenType Scanner::scanProcessingInstruction() {
 
     if (endsWith(value_, "?>")) {
       gotTail_ = true;
-      value_.size -= 2;
+      value_.size -= length("?>");
       break;
     }
   }
@@ -356,11 +370,13 @@ Scanner::TokenType Scanner::scanProcessingInstruction() {
 
 Scanner::TokenType Scanner::scanSpecial() {
   if (gotTail_) {
+    start_ = input_.pos() - (tagName_.size + length("</>"));
     scanFun_ = &Scanner::scanBody;
     gotTail_ = false;
     return TT_TAG_END;
   }
 
+  start_ = input_.pos();
   value_ = string_ref{input_.pos(), 0};
 
   while (true) {
@@ -369,17 +385,17 @@ Scanner::TokenType Scanner::scanSpecial() {
 
     // Test for </tag>
     // TODO: no whitespaces allowed? Is that okay?
-    if (value_.data[value_.size - 1] == '>' && value_.size >= tagName_.size + 3) {
+    if (value_.data[value_.size - 1] == '>' && value_.size >= tagName_.size + length("</>")) {
       // Test for the "</"" bit of "</tag>"
-      size_t pos_tag_start = value_.size - tagName_.size - 3;
-      if (std::memcmp(value_.data + pos_tag_start, "</", 2) != 0) continue;
+      size_t posTagStart = value_.size - tagName_.size - length("</>");
+      if (std::memcmp(value_.data + posTagStart, "</", length("</")) != 0) continue;
 
       // Test for the "tag" bit of "</tag>". Doing case insensitive compare because <I>...</i> is okay.
-      size_t pos_tag_name = value_.size - tagName_.size - 1;  // end - tag>
-      if (!equalsCaseInsensitive(value_.data + pos_tag_name, tagName_.data, tagName_.size)) continue;
+      size_t posTagName = value_.size - tagName_.size - length(">");  // end - tag>
+      if (!equalsCaseInsensitive(value_.data + posTagName, tagName_.data, tagName_.size)) continue;
 
       gotTail_ = true;
-      value_.size -= tagName_.size + 3;
+      value_.size -= tagName_.size + length("</>");
       break;
     }
   }
diff --git a/src/translator/xh_scanner.h b/src/translator/xh_scanner.h
index 14d755bbd..530df675d 100644
--- a/src/translator/xh_scanner.h
+++ b/src/translator/xh_scanner.h
@@ -83,6 +83,7 @@ class Scanner {
         tagName_{nullptr, 0},
         attributeName_{nullptr, 0},
         input_(is),
+        start_(nullptr),
         scanFun_(&Scanner::scanBody),
         gotTail_(false) {}
 
@@ -98,6 +99,8 @@ class Scanner {
   // get tag name
   std::string_view tag() const;
 
+  inline const char *start() const { return start_; }
+
  private: /* methods */
   typedef TokenType (Scanner::*ScanPtr)();
 
@@ -137,6 +140,9 @@ class Scanner {
 
   instream &input_;
 
+  // Start position of a token.
+  const char *start_;
+
   bool gotTail_;  // aux flag used in scanComment, scanSpecial, scanProcessingInstruction
 };
 }  // namespace markup

From 96b0f82343ab3ac4009aba8ecde706b6dca189d8 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Wed, 23 Feb 2022 13:25:12 +0000
Subject: [PATCH 361/442] Simplify cache config and bind for use in JS (#359)

Deprecates cacheEnabled parameter to be replaced with cacheSize=0.
Python bindings, Documentation in comments and tests updated to reflect
this change.

Exposes the fields corresponding to cache via embind as a value object.
The equivalent object-based syntax in worker.js allows propagation
from JS.

Fixes: #351
See also: mozilla/firefox-translations#96
---
 bergamot-translator-tests          |  2 +-
 bindings/python/bergamot.cpp       |  7 ++-----
 src/translator/service.cpp         |  8 ++++----
 src/translator/service.h           | 24 +++++++++---------------
 wasm/bindings/service_bindings.cpp |  6 ++----
 wasm/test_page/js/worker.js        |  2 +-
 6 files changed, 19 insertions(+), 30 deletions(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 3776609ce..d03a9d316 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 3776609ce5f7a238245e303efaa007b2d5078180
+Subproject commit d03a9d316d40ba45c475018287971523666bf51e
diff --git a/bindings/python/bergamot.cpp b/bindings/python/bergamot.cpp
index e341ada75..5e9e830f9 100644
--- a/bindings/python/bergamot.cpp
+++ b/bindings/python/bergamot.cpp
@@ -198,18 +198,15 @@ PYBIND11_MODULE(_bergamot, m) {
       .def("pivot", &ServicePyAdapter::pivot);
 
   py::class_<Service::Config>(m, "ServiceConfig")
-      .def(py::init<>([](size_t numWorkers, bool cacheEnabled, size_t cacheSize, std::string logging) {
+      .def(py::init<>([](size_t numWorkers, size_t cacheSize, std::string logging) {
              Service::Config config;
              config.numWorkers = numWorkers;
-             config.cacheEnabled = cacheEnabled;
              config.cacheSize = cacheSize;
              config.logger.level = logging;
              return config;
            }),
-           py::arg("numWorkers") = 1, py::arg("cacheEnabled") = false, py::arg("cacheSize") = 20000,
-           py::arg("logLevel") = "off")
+           py::arg("numWorkers") = 1, py::arg("cacheSize") = 0, py::arg("logLevel") = "off")
       .def_readwrite("numWorkers", &Service::Config::numWorkers)
-      .def_readwrite("cacheEnabled", &Service::Config::cacheEnabled)
       .def_readwrite("cacheSize", &Service::Config::cacheSize);
 
   py::class_<_Model, std::shared_ptr<_Model>>(m, "TranslationModel");
diff --git a/src/translator/service.cpp b/src/translator/service.cpp
index d510cf5c0..32cd023c5 100644
--- a/src/translator/service.cpp
+++ b/src/translator/service.cpp
@@ -30,8 +30,8 @@ Response combine(Response &&first, Response &&second) {
   return combined;
 }
 
-std::optional<TranslationCache> makeOptionalCache(bool enabled, size_t size, size_t mutexBuckets) {
-  return enabled ? std::make_optional<TranslationCache>(size, mutexBuckets) : std::nullopt;
+std::optional<TranslationCache> makeOptionalCache(size_t size, size_t mutexBuckets) {
+  return size > 0 ? std::make_optional<TranslationCache>(size, mutexBuckets) : std::nullopt;
 }
 
 }  // namespace
@@ -40,7 +40,7 @@ BlockingService::BlockingService(const BlockingService::Config &config)
     : config_(config),
       requestId_(0),
       batchingPool_(),
-      cache_(makeOptionalCache(config.cacheEnabled, config.cacheSize, /*mutexBuckets = */ 1)),
+      cache_(makeOptionalCache(config.cacheSize, /*mutexBuckets = */ 1)),
       logger_(config.logger) {}
 
 std::vector<Response> BlockingService::translateMultiple(std::shared_ptr<TranslationModel> translationModel,
@@ -133,7 +133,7 @@ AsyncService::AsyncService(const AsyncService::Config &config)
     : requestId_(0),
       config_(config),
       safeBatchingPool_(),
-      cache_(makeOptionalCache(config_.cacheEnabled, config_.cacheSize, /*mutexBuckets=*/config_.numWorkers)),
+      cache_(makeOptionalCache(config_.cacheSize, /*mutexBuckets=*/config_.numWorkers)),
       logger_(config.logger) {
   ABORT_IF(config_.numWorkers == 0, "Number of workers should be at least 1 in a threaded workflow");
   workers_.reserve(config_.numWorkers);
diff --git a/src/translator/service.h b/src/translator/service.h
index 72522375f..1e4c9bab1 100644
--- a/src/translator/service.h
+++ b/src/translator/service.h
@@ -30,21 +30,17 @@ class AsyncService;
 class BlockingService {
  public:
   struct Config {
-    bool cacheEnabled{false};  ///< Whether to enable cache or not.
-
-    /// Size in History items to be stored in the cache. Loosely corresponds to sentences to
-    /// cache in the real world. Note that cache has a random-eviction policy. The peak
-    /// storage at full occupancy is controlled by this parameter. However, whether we attain
-    /// full occupancy or not is controlled by random factors - specifically how uniformly
-    /// the hash distributes.
-    size_t cacheSize{2000};
+    /// Size in History items to be stored in the cache. A value of 0 means no caching. Loosely corresponds to sentences
+    /// to cache in the real world. Note that cache has a random-eviction policy. The peak storage at full occupancy is
+    /// controlled by this parameter. However, whether we attain full occupancy or not is controlled by random factors -
+    /// specifically how uniformly the hash distributes.
+    size_t cacheSize{0};
 
     Logger::Config logger;  ///< Configurations for logging
 
     template <class App>
     static void addOptions(App &app, Config &config) {
       // Options will come here.
-      app.add_option("--cache-translations", config.cacheEnabled, "Whether to cache translations or not.");
       app.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
       Logger::Config::addOptions(app, config.logger);
     }
@@ -112,16 +108,14 @@ class BlockingService {
 class AsyncService {
  public:
   struct Config {
-    size_t numWorkers{1};      ///< How many worker translation threads to spawn.
-    bool cacheEnabled{false};  ///< Whether to enable cache or not.
-    size_t cacheSize{2000};    ///< Size in History items to be stored in the cache. Loosely corresponds to sentences to
-                               /// cache in the real world.
-    Logger::Config logger;     // Configurations for logging
+    size_t numWorkers{1};   ///< How many worker translation threads to spawn.
+    size_t cacheSize{0};    ///< Size in History items to be stored in the cache. Loosely corresponds to sentences to
+                            /// cache in the real world. A value of 0 means no caching.
+    Logger::Config logger;  // Configurations for logging
 
     template <class App>
     static void addOptions(App &app, Config &config) {
       app.add_option("--cpu-threads", config.numWorkers, "Workers to form translation backend");
-      app.add_option("--cache-translations", config.cacheEnabled, "Whether to cache translations or not.");
       app.add_option("--cache-size", config.cacheSize, "Number of entries to store in cache.");
       Logger::Config::addOptions(app, config.logger);
     }
diff --git a/wasm/bindings/service_bindings.cpp b/wasm/bindings/service_bindings.cpp
index 167f51f81..8e4fe4d14 100644
--- a/wasm/bindings/service_bindings.cpp
+++ b/wasm/bindings/service_bindings.cpp
@@ -69,10 +69,8 @@ EMSCRIPTEN_BINDINGS(translation_model) {
 }
 
 EMSCRIPTEN_BINDINGS(blocking_service_config) {
-  value_object<BlockingService::Config>("BlockingServiceConfig");
-  // .field("name", &BlockingService::Config::name")
-  // The above is a future hook. Note that more will come - for cache, for workspace-size or graph details  limits on
-  // aggregate-batching etc.
+  value_object<BlockingService::Config>("BlockingServiceConfig")
+      .field("cacheSize", &BlockingService::Config::cacheSize);
 }
 
 std::shared_ptr<BlockingService> BlockingServiceFactory(const BlockingService::Config& config) {
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index 4c1c640f1..292e2d6dd 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -78,7 +78,7 @@ onmessage = async function(e) {
 // Instantiates the Translation Service
 const constructTranslationService = async () => {
   if (!translationService) {
-    var translationServiceConfig = {};
+    var translationServiceConfig = {cacheSize: 20000};
     log(`Creating Translation Service with config: ${translationServiceConfig}`);
     translationService = new Module.BlockingService(translationServiceConfig);
     log(`Translation Service created successfully`);

From fe3f3982debbd88e94f1909c313d1a611d2ff1c9 Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Fri, 25 Feb 2022 23:01:32 +0100
Subject: [PATCH 362/442] Embed quality-scores as HTML tag attributes (#358)

Quality scores for HTML translation exposed as <font
x-bergamot-sentence-score=""> and <font x-bergamot-word-score=""> tags
in the HTML output. While this increases the size of the HTML returned,
the resulting rendered HTML can easily be styled to show the scores.
With Javascript or CSS, developers can easily have some interface based
on these extra attributes.

Also includes updates to the test page to show a proof-of-concept
demonstration.

Fixes: #355
---
 src/tests/common-impl.cpp            |  5 +-
 src/translator/definitions.h         | 10 ++++
 src/translator/html.cpp              | 88 +++++++++++++++++++++++-----
 src/translator/html.h                |  8 ++-
 src/translator/quality_estimator.cpp | 22 +------
 src/translator/quality_estimator.h   | 12 ----
 src/translator/response.cpp          | 18 ++++++
 src/translator/response.h            |  6 +-
 wasm/bindings/response_bindings.cpp  | 12 ----
 wasm/test_page/css/index.css         | 21 ++++++-
 wasm/test_page/index.html            |  2 +-
 wasm/test_page/js/index.js           | 58 +++++++++++++++---
 wasm/test_page/js/worker.js          | 40 -------------
 13 files changed, 187 insertions(+), 115 deletions(-)

diff --git a/src/tests/common-impl.cpp b/src/tests/common-impl.cpp
index 43b1c07d2..431ddaa71 100644
--- a/src/tests/common-impl.cpp
+++ b/src/tests/common-impl.cpp
@@ -155,10 +155,11 @@ void TestSuite<Service>::qualityEstimatorWords(Ptr<TranslationModel> model) {
   std::string source = readFromStdin();
   const Response response = bridge_.translate(service_, model, std::move(source), responseOptions);
 
-  for (const auto &sentenceQualityEstimate : response.qualityScores) {
+  for (size_t sentenceIdx = 0; sentenceIdx < response.qualityScores.size(); ++sentenceIdx) {
+    const auto &sentenceQualityEstimate = response.qualityScores[sentenceIdx];
     std::cout << "[SentenceBegin]\n";
 
-    for (const auto &wordByteRange : sentenceQualityEstimate.wordByteRanges) {
+    for (const auto &wordByteRange : getWordByteRanges(response, sentenceIdx)) {
       const string_view word(response.target.text.data() + wordByteRange.begin, wordByteRange.size());
       std::cout << word << "\n";
     }
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index eb1e67296..b3bc1019b 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -42,6 +42,16 @@ struct ByteRange {
   bool operator==(ByteRange other) const { return begin == other.begin && end == other.end; }
 };
 
+/// A Subword range is mechanically the same as a `ByteRange`, but instead of
+/// describing a span of bytes, it describes a span of Subword tokens. Using
+/// `Annotation.word()` you can switch between the two.
+struct SubwordRange {
+  size_t begin;
+  size_t end;
+  const size_t size() const { return end - begin; }
+  bool operator==(SubwordRange other) const { return begin == other.begin && end == other.end; }
+};
+
 class Response;
 using CallbackType = std::function<void(Response &&)>;
 
diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index ed42b9117..5180a5af5 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -3,6 +3,7 @@
 #include <algorithm>
 
 #include "response.h"
+#include "translator/definitions.h"
 #include "xh_scanner.h"
 
 namespace {
@@ -544,7 +545,12 @@ void HTML::restore(Response &response) {
   copyTagStack(response, alignments, sourceTokenSpans, targetTokenSpans);
   assert(targetTokenSpans.size() == debugCountTokens(response.target));
 
-  AnnotatedText target = restoreTarget(response.target, targetTokenSpans);
+  // Take the spans, and use them to make a taint for every word in the
+  // translation. Optionally add extra tags, like quality score metadata.
+  std::vector<HTML::TagStack> targetTokenTags;
+  annotateTagStack(response, targetTokenSpans, targetTokenTags);
+
+  AnnotatedText target = restoreTarget(response.target, targetTokenSpans, targetTokenTags);
 
   response.source = source;
   response.target = target;
@@ -592,38 +598,37 @@ AnnotatedText HTML::restoreSource(AnnotatedText const &in, std::vector<SpanItera
   });
 }
 
-AnnotatedText HTML::restoreTarget(AnnotatedText const &in, std::vector<SpanIterator> const &targetTokenSpans) {
-  auto prevSpan = spans_.cbegin();
+AnnotatedText HTML::restoreTarget(AnnotatedText const &in, std::vector<SpanIterator> const &targetTokenSpans,
+                                  std::vector<TagStack> const &targetTokenTags) {
+  auto prevTags = spans_.cbegin()->tags;
+  auto stragglerSpanIt = spans_.cbegin();
   auto targetSpanIt = targetTokenSpans.begin();
-  auto straggerSpanIt = spans_.cbegin();
+  auto targetTagIt = targetTokenTags.begin();
 
   AnnotatedText out = in.apply([&]([[maybe_unused]] ByteRange range, string_view token, bool last) {
     TokenFormatter formatter(token);
 
     // First we scan through spans_ to catch up to the span assigned to this
     // token. We're only interested in empty spans (empty and void elements)
-    for (; straggerSpanIt < *targetSpanIt; ++straggerSpanIt) {
+    for (; stragglerSpanIt < *targetSpanIt; stragglerSpanIt++) {
       // We're only interested in empty spans or spans that would otherwise get
       // lost because they didn't align with anything between the spans in
       // targetSpanIt
       // TODO That std::find makes this O(N*N) NOT GOOD NOT GOOD
-      if (straggerSpanIt->size() != 0 &&
-          std::find(targetTokenSpans.begin(), targetTokenSpans.end(), straggerSpanIt) != targetTokenSpans.end())
+      if (stragglerSpanIt->size() != 0 &&
+          std::find(targetTokenSpans.begin(), targetTokenSpans.end(), stragglerSpanIt) != targetTokenSpans.end())
         continue;
 
-      formatter.append(prevSpan->tags, straggerSpanIt->tags);
-
-      // Note: here, not in 3rd part of for-statement because we don't want to
-      // set prevSpan if the continue clause at the beginning of this for-loop
-      // was hit.
-      prevSpan = straggerSpanIt;
+      formatter.append(prevTags, stragglerSpanIt->tags);
+      prevTags = stragglerSpanIt->tags;
     }
 
     // Now do the same thing but for our target set of tags. Note that we cannot
     // combine this in the for-loop above (i.e. `span_it <= *targetSpanIt`)
     // because there is no guarantee that the order in `targetTokenSpans` is
     // the same as that of `spans`.
-    formatter.append(prevSpan->tags, (*targetSpanIt)->tags);
+
+    formatter.append(prevTags, *targetTagIt);
 
     // If this is the last token of the response, close all open tags.
     if (last) {
@@ -632,11 +637,12 @@ AnnotatedText HTML::restoreTarget(AnnotatedText const &in, std::vector<SpanItera
       // the last token of the output. But lets assume someone someday changes
       // HardAlignments(), and then this for-loop will be necessary.
       // assert((*targetSpanIt)->tags.empty());
-      formatter.append((*targetSpanIt)->tags, HTML::TagStack());
+      formatter.append(*targetTagIt, HTML::TagStack());
     }
 
-    prevSpan = *targetSpanIt;
+    prevTags = *targetTagIt;
     ++targetSpanIt;
+    ++targetTagIt;
 
     return std::move(formatter.html());
   });
@@ -674,6 +680,56 @@ void HTML::copyTagStack(Response const &response, std::vector<std::vector<size_t
   targetTokenSpans.push_back(sourceTokenSpans[offset]);  // token_tag for ending whitespace
 }
 
+void HTML::annotateTagStack(Response const &response, std::vector<SpanIterator> const &targetTokenSpans,
+                            std::vector<HTML::TagStack> &targetTokenTags) {
+  auto spanIt = targetTokenSpans.begin();
+  for (size_t sentenceIdx = 0; sentenceIdx < response.target.numSentences(); ++sentenceIdx) {
+    // Sentence prefix
+    targetTokenTags.push_back((*spanIt)->tags);
+    spanIt++;
+
+    // Offset in targetTokenTags at which this sentence's tags start.
+    size_t tagOffset = targetTokenTags.size();
+
+    // Initially, just copy the span's tags to this token
+    for (size_t t = 0; t < response.target.numWords(sentenceIdx); ++t) {
+      targetTokenTags.emplace_back((*spanIt)->tags);
+      spanIt++;
+    }
+
+    // If we have quality score information, add that as metadata as well.
+    if (!response.qualityScores.empty()) {
+      auto const &sentenceQuality = response.qualityScores[sentenceIdx];
+      // Create a single <font> tag for this sentence with sentence level info
+      Tag *sentenceTag = makeTag({Tag::ELEMENT, "font"});
+      sentenceTag->attributes += format(" x-bergamot-sentence-index=\"{}\" x-bergamot-sentence-score=\"{}\"",
+                                        sentenceIdx, sentenceQuality.sentenceScore);
+
+      // Add that tag to all tokens in this sentence.
+      for (size_t tokenIdx = 0; tokenIdx < response.target.numWords(sentenceIdx); ++tokenIdx) {
+        targetTokenTags[tagOffset + tokenIdx].push_back(sentenceTag);
+      }
+
+      // Add word level <font> tags as well to all tokens that make up a word.
+      for (size_t wordIdx = 0; wordIdx < sentenceQuality.wordRanges.size(); ++wordIdx) {
+        Tag *wordTag = makeTag({Tag::ELEMENT, "font"});
+        wordTag->attributes += format(" x-bergamot-word-index=\"{}\" x-bergamot-word-score=\"{}\"", wordIdx,
+                                      sentenceQuality.wordScores[wordIdx]);
+        auto const &range = sentenceQuality.wordRanges[wordIdx];
+        for (size_t tokenIdx = range.begin; tokenIdx < range.end; ++tokenIdx) {
+          targetTokenTags[tagOffset + tokenIdx].push_back(wordTag);
+        }
+      }
+    }
+  }
+
+  // Suffix
+  targetTokenTags.push_back((*spanIt)->tags);
+  spanIt++;
+
+  assert(spanIt == targetTokenSpans.end());
+}
+
 // Reports if token `str` is likely to be a continuation of a word. This is used
 // to determine whether we should share the markup, or whether we should see
 // this token as a fresh start. This implementation will treat "hello[world]"
diff --git a/src/translator/html.h b/src/translator/html.h
index c704c5904..f3c6dad19 100644
--- a/src/translator/html.h
+++ b/src/translator/html.h
@@ -162,7 +162,7 @@ class HTML {
   void restore(Response &response);
 
  private:
-  using SpanIterator = std::vector<HTML::Span>::const_iterator;
+  using SpanIterator = std::vector<HTML::Span>::iterator;
   using AnnotatedText = marian::bergamot::AnnotatedText;
 
   /// Reconstructs HTML in `response.source` (passed as `in`) and makes a list
@@ -175,7 +175,8 @@ class HTML {
   /// Inserts the HTML into `response.target` (passed as `in`) based on
   /// `targetTokenSpans`, which points to a `Span` for each token (subword) in
   /// `response.target`.
-  AnnotatedText restoreTarget(AnnotatedText const &in, std::vector<SpanIterator> const &targetTokenSpans);
+  AnnotatedText restoreTarget(AnnotatedText const &in, std::vector<SpanIterator> const &targetTokenSpans,
+                              std::vector<HTML::TagStack> const &targetTokenTags);
 
   /// Utilities to test whether subword `str` is part of a word together with
   /// the subword `prev`, or a separate word. Basically *does `str` start with
@@ -190,6 +191,9 @@ class HTML {
                     std::vector<HTML::SpanIterator> const &sourceTokenSpans,
                     std::vector<HTML::SpanIterator> &targetTokenSpans);
 
+  void annotateTagStack(Response const &response, std::vector<SpanIterator> const &targetTokenSpans,
+                        std::vector<HTML::TagStack> &targetTokenTags);
+
   /// Turns the alignment scores in `response.alignments` into one source token
   /// per target token. Has some heuristics to keep all target tokens of a
   /// single word pointing to the same span, and prefers spans with more markup
diff --git a/src/translator/quality_estimator.cpp b/src/translator/quality_estimator.cpp
index 936d293a4..24ca2c2aa 100644
--- a/src/translator/quality_estimator.cpp
+++ b/src/translator/quality_estimator.cpp
@@ -27,7 +27,7 @@ Response::SentenceQualityScore UnsupervisedQualityEstimator::computeSentenceScor
   const float sentenceScore =
       std::accumulate(std::begin(wordScores), std::end(wordScores), float(0.0)) / wordScores.size();
 
-  return {wordScores, subwordToWords(wordIndices, target, sentenceIdx), sentenceScore};
+  return {wordScores, wordIndices, sentenceScore};
 }
 
 LogisticRegressorQualityEstimator::Matrix::Matrix(const size_t rowsParam, const size_t colsParam)
@@ -160,7 +160,7 @@ Response::SentenceQualityScore LogisticRegressorQualityEstimator::computeSentenc
   const float sentenceScore =
       std::accumulate(std::begin(wordScores), std::end(wordScores), float(0.0)) / wordScores.size();
 
-  return {wordScores, subwordToWords(wordIndices, target, sentenceIdx), sentenceScore};
+  return {wordScores, wordIndices, sentenceScore};
 }
 
 std::vector<float> LogisticRegressorQualityEstimator::predict(const Matrix& features) const {
@@ -267,22 +267,4 @@ std::vector<SubwordRange> mapWords(const std::vector<float>& logProbs, const Ann
   return wordIndices;
 }
 
-std::vector<ByteRange> subwordToWords(const std::vector<SubwordRange>& wordIndices, const AnnotatedText& target,
-                                      const size_t sentenceIdx) {
-  std::vector<ByteRange> words;
-
-  for (const SubwordRange& wordIndice : wordIndices) {
-    size_t wordBegin = target.wordAsByteRange(sentenceIdx, wordIndice.begin).begin;
-    size_t wordEnd = target.wordAsByteRange(sentenceIdx, wordIndice.end).begin;
-
-    if (isspace(target.text.at(wordBegin))) {
-      ++wordBegin;
-    }
-
-    words.emplace_back(ByteRange{wordBegin, wordEnd});
-  }
-
-  return words;
-}
-
 }  // namespace marian::bergamot
diff --git a/src/translator/quality_estimator.h b/src/translator/quality_estimator.h
index 3d2fd68ea..b8d15963c 100644
--- a/src/translator/quality_estimator.h
+++ b/src/translator/quality_estimator.h
@@ -21,8 +21,6 @@ class QualityEstimator {
   virtual void computeQualityScores(const Histories &histories, Response &response) const = 0;
 };
 
-using SubwordRange = ByteRange;
-
 /// Unsupervised Quality Estimator model. It uses the translator model's log probabilities (log probs) as a proxy for
 /// quality scores. Then, for a given word, its quality score is computed by taking the mean of the log probs of the
 /// tokens that make it up. The sentence score is the mean of all word's log probs.
@@ -209,14 +207,4 @@ inline std::shared_ptr<QualityEstimator> createQualityEstimator(const AlignedMem
 std::vector<SubwordRange> mapWords(const std::vector<float> &logProbs, const AnnotatedText &target,
                                    const size_t sentenceIdx);
 
-/// Given a vector of subwordRanges, it maps the elements to be real words rather than sublevel tokens. The words are
-/// represented through ByteRanges.
-
-/// @param [in] wordIndices: A vector where each element correspond to the index of a real word and its values are
-/// represented by the SubwordRanges (which are aliases of ByteRanges) which represents sublevel token positions
-/// @param [in] target: AnnotatedText target value
-/// @param [in] sentenceIdx: the id of a candidate sentence
-std::vector<ByteRange> subwordToWords(const std::vector<SubwordRange> &wordIndices, const AnnotatedText &target,
-                                      const size_t sentenceIdx);
-
 }  // namespace marian::bergamot
diff --git a/src/translator/response.cpp b/src/translator/response.cpp
index 8e623a7d6..135ec4715 100644
--- a/src/translator/response.cpp
+++ b/src/translator/response.cpp
@@ -142,4 +142,22 @@ std::vector<Alignment> remapAlignments(const Response &first, const Response &se
   return alignments;
 }
 
+std::vector<ByteRange> getWordByteRanges(const Response &response, size_t sentenceIdx) {
+  std::vector<ByteRange> wordByteRanges;
+  wordByteRanges.reserve(response.qualityScores[sentenceIdx].wordRanges.size());
+
+  for (auto &&word : response.qualityScores[sentenceIdx].wordRanges) {
+    size_t wordBegin = response.target.wordAsByteRange(sentenceIdx, word.begin).begin;
+    size_t wordEnd = response.target.wordAsByteRange(sentenceIdx, word.end).begin;
+
+    if (std::isspace(response.target.text.at(wordBegin))) {
+      ++wordBegin;
+    }
+
+    wordByteRanges.emplace_back(ByteRange{wordBegin, wordEnd});
+  }
+
+  return wordByteRanges;
+}
+
 }  // namespace marian::bergamot
diff --git a/src/translator/response.h b/src/translator/response.h
index 74463eda2..af05e1074 100644
--- a/src/translator/response.h
+++ b/src/translator/response.h
@@ -30,8 +30,8 @@ struct Response {
   struct SentenceQualityScore {
     /// Quality score of each translated word
     std::vector<float> wordScores;
-    /// Each word position in the translated text
-    std::vector<ByteRange> wordByteRanges;
+    /// Position of start and end token of each word in the translated text
+    std::vector<SubwordRange> wordRanges;
     /// Whole sentence quality score (it is composed by the mean of its words)
     float sentenceScore = 0.0;
   };
@@ -77,6 +77,8 @@ struct Response {
 
 std::vector<Alignment> remapAlignments(const Response &first, const Response &second);
 
+std::vector<ByteRange> getWordByteRanges(Response const &response, size_t sentenceIdx);
+
 }  // namespace bergamot
 }  // namespace marian
 
diff --git a/wasm/bindings/response_bindings.cpp b/wasm/bindings/response_bindings.cpp
index 11bc4cabb..51a46ab84 100644
--- a/wasm/bindings/response_bindings.cpp
+++ b/wasm/bindings/response_bindings.cpp
@@ -10,7 +10,6 @@
 #include "response.h"
 
 using Response = marian::bergamot::Response;
-using SentenceQualityScore = marian::bergamot::Response::SentenceQualityScore;
 using ByteRange = marian::bergamot::ByteRange;
 
 using namespace emscripten;
@@ -20,25 +19,14 @@ EMSCRIPTEN_BINDINGS(byte_range) {
   value_object<ByteRange>("ByteRange").field("begin", &ByteRange::begin).field("end", &ByteRange::end);
 }
 
-std::vector<SentenceQualityScore> getQualityScores(const Response& response) { return response.qualityScores; }
-
 EMSCRIPTEN_BINDINGS(response) {
   class_<Response>("Response")
       .constructor<>()
       .function("size", &Response::size)
-      .function("getQualityScores", &getQualityScores)
       .function("getOriginalText", &Response::getOriginalText)
       .function("getTranslatedText", &Response::getTranslatedText)
       .function("getSourceSentence", &Response::getSourceSentenceAsByteRange)
       .function("getTranslatedSentence", &Response::getTargetSentenceAsByteRange);
 
-  value_object<SentenceQualityScore>("SentenceQualityScore")
-      .field("wordScores", &SentenceQualityScore::wordScores)
-      .field("wordByteRanges", &SentenceQualityScore::wordByteRanges)
-      .field("sentenceScore", &SentenceQualityScore::sentenceScore);
-
   register_vector<Response>("VectorResponse");
-  register_vector<SentenceQualityScore>("VectorSentenceQualityScore");
-  register_vector<float>("VectorFloat");
-  register_vector<ByteRange>("VectorByteRange");
 }
diff --git a/wasm/test_page/css/index.css b/wasm/test_page/css/index.css
index bbc5bf147..6ed642232 100644
--- a/wasm/test_page/css/index.css
+++ b/wasm/test_page/css/index.css
@@ -73,7 +73,7 @@ label {
   align-self: center;
 }
 
-textarea {
+textarea, .output-area {
   padding: 1rem;
   font-family: sans-serif;
   font-size: 1rem;
@@ -97,3 +97,22 @@ button:hover {
 #output {
   background-color: #f4f4f4;
 }
+
+.output-area [x-bergamot-word-score].bad {
+  background-image:
+    linear-gradient(45deg, transparent 65%, red 80%, transparent 90%),
+    linear-gradient(135deg, transparent 5%, red 15%, transparent 25%),
+    linear-gradient(135deg, transparent 45%, red 55%, transparent 65%),
+    linear-gradient(45deg, transparent 25%, red 35%, transparent 50%);
+  background-repeat:repeat-x;
+  background-size: 8px 2px;
+  background-position:0 95%;
+}
+
+.output-area [x-bergamot-sentence-score].bad {
+  background: rgba(255, 128, 128, 0.8);
+}
+
+.output-area [x-bergamot-sentence-index].highlight-sentence {
+  background: rgba(255, 255, 128, 0.8);
+}
\ No newline at end of file
diff --git a/wasm/test_page/index.html b/wasm/test_page/index.html
index 86eae4637..3f48117e1 100644
--- a/wasm/test_page/index.html
+++ b/wasm/test_page/index.html
@@ -24,7 +24,7 @@
           To
           <select id="lang-to" name="to" class="lang-select"></select>
         </label>
-        <textarea id="output" name="output" readonly></textarea>
+        <div id="output" class="output-area"></div>
       </div>
       <div class="footer" id="status"></div>
     </div>
diff --git a/wasm/test_page/js/index.js b/wasm/test_page/js/index.js
index 7cb365702..166f0f27b 100644
--- a/wasm/test_page/js/index.js
+++ b/wasm/test_page/js/index.js
@@ -38,22 +38,58 @@ const _prepareTranslateOptions = (paragraphs) => {
   return translateOptions;
 };
 
+const textToHTML = (text) => {
+  const div = document.createElement('div');
+  div.appendChild(document.createTextNode(text));
+  return div.innerHTML;
+};
+
 const translateCall = () => {
-  const text = document.querySelector("#input").value + "  ";
+  const text = document.querySelector("#input").value;
   if (!text.trim().length) return;
 
-  const paragraphs = text.split("\n");
+  const paragraphs = text.split(/\n+/).map(textToHTML); // escape HTML 
   const translateOptions = _prepareTranslateOptions(paragraphs);
-  $("#output").setAttribute("disabled", true);
   const lngFrom = langFrom.value;
   const lngTo = langTo.value;
   worker.postMessage(["translate", lngFrom, lngTo, paragraphs, translateOptions]);
 };
 
+const addQualityClasses = (root) => {
+  // You can do this wit CSS variables, calc() and min/max, but JS is just easier
+
+  root.querySelectorAll('[x-bergamot-sentence-score]').forEach(el => {
+    // Note: these thresholds are just examples, they are not good thresholds!  
+    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-sentence-score')) > -0.1);
+  });
+
+  root.querySelectorAll('[x-bergamot-word-score]').forEach(el => {
+    // Note: these thresholds are just examples, they are not good thresholds!
+    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-word-score')) > -0.1);
+  });
+
+  // Add tooltips to each (sub)word with sentence and word score.
+  root.querySelectorAll('[x-bergamot-sentence-score] > [x-bergamot-word-score]').forEach(el => {
+    const sentenceScore = parseFloat(el.parentNode.getAttribute('x-bergamot-sentence-score'));
+    const wordScore = parseFloat(el.getAttribute('x-bergamot-word-score'));
+    el.title = `Sentence: ${sentenceScore}  Word: ${wordScore}`;
+  });
+}
+
 worker.onmessage = function (e) {
   if (e.data[0] === "translate_reply" && e.data[1]) {
-    document.querySelector("#output").value = e.data[1].join("\n\n");
-    $("#output").removeAttribute("disabled");
+    // Clear output of previous translation
+    document.querySelector("#output").innerHTML = '';
+
+    // Add each translation in its own div to have a known root in which the
+    // sentence ids are unique. Used for highlighting sentences.
+    e.data[1].forEach(translatedHTML => {
+      const translation = document.createElement('div');
+      translation.classList.add('translation');
+      translation.innerHTML = translatedHTML;
+      addQualityClasses(translation);
+      document.querySelector("#output").appendChild(translation);
+    });
   } else if (e.data[0] === "load_model_reply" && e.data[1]) {
     status(e.data[1]);
     translateCall();
@@ -76,8 +112,8 @@ const loadModel = () => {
     console.log(`Loading model '${lngFrom}${lngTo}'`);
     worker.postMessage(["load_model", lngFrom, lngTo]);
   } else {
-    const input = document.querySelector("#input").value;
-    document.querySelector("#output").value = input;
+    const input = textToHTML(document.querySelector("#input").value);
+    document.querySelector("#output").innerHTML = input;
   }
 };
 
@@ -95,6 +131,14 @@ $(".swap").addEventListener("click", e => {
   loadModel();
 });
 
+$('#output').addEventListener('mouseover', e => {
+  const root = e.target.closest('.translation');
+  const sentence = e.target.parentNode.hasAttribute('x-bergamot-sentence-index') ? e.target.parentNode.getAttribute('x-bergamot-sentence-index') : null;  
+  document.querySelectorAll('#output font[x-bergamot-sentence-index]').forEach(el => {
+    el.classList.toggle('highlight-sentence', el.getAttribute('x-bergamot-sentence-index') === sentence && el.closest('.translation') === root);
+  })
+})
+
 function init() {
   // try to guess input language from user agent
   let myLang = navigator.language;
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index 292e2d6dd..aa4d40497 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -137,13 +137,11 @@ const translate = (from, to, input, translateOptions) => {
     const listSourceText = _parseSourceText(vectorResponse);
     const listTranslatedTextSentences = _parseTranslatedTextSentences(vectorResponse);
     const listSourceTextSentences = _parseSourceTextSentences(vectorResponse);
-    const listTranslatedTextSentenceQualityScores = _parseTranslatedTextSentenceQualityScores(vectorResponse);
 
     log(`Source text: ${listSourceText}`);
     log(`Translated text: ${listTranslatedText}`);
     log(`Translated sentences: ${JSON.stringify(listTranslatedTextSentences)}`);
     log(`Source sentences: ${JSON.stringify(listSourceTextSentences)}`);
-    log(`Translated sentence quality scores: ${JSON.stringify(listTranslatedTextSentenceQualityScores)}`);
 
     return listTranslatedText;
   } finally {
@@ -292,44 +290,6 @@ const _parseSourceTextSentences = (vectorResponse) => {
   return result;
 }
 
-const _parseTranslatedTextSentenceQualityScores = (vectorResponse) => {
-  const result = [];
-  for (let i = 0; i < vectorResponse.size(); i++) {
-    const response = vectorResponse.get(i);
-    const translatedText = response.getTranslatedText();
-    const vectorSentenceQualityScore = response.getQualityScores();
-    log(`No. of sentences: "${vectorSentenceQualityScore.size()}"`);
-    const sentenceQualityScores = [];
-    for (let sentenceIndex=0; sentenceIndex < vectorSentenceQualityScore.size(); sentenceIndex++) {
-      const sentenceQualityScoreObject = vectorSentenceQualityScore.get(sentenceIndex);
-      const wordByteRangeList = [];
-      const wordList = [];
-      const wordScoreList = [];
-      const vectorWordScore = sentenceQualityScoreObject.wordScores;
-      const vectorWordByteRange = sentenceQualityScoreObject.wordByteRanges;
-
-      for (let wordIndex = 0; wordIndex < vectorWordScore.size(); wordIndex++) {
-        const wordScore = vectorWordScore.get(wordIndex);
-        const wordByteRange = vectorWordByteRange.get(wordIndex);
-        wordScoreList.push(wordScore);
-        wordByteRangeList.push(wordByteRange);
-        const word = _getSubString(translatedText, wordByteRange);
-        wordList.push(word);
-      }
-
-      const sentenceQualityScore = {
-        wordByteRanges: wordByteRangeList,
-        words: wordList,
-        wordScores: wordScoreList,
-        sentenceScore: sentenceQualityScoreObject.sentenceScore
-      };
-      sentenceQualityScores.push(sentenceQualityScore);
-    }
-    result.push(sentenceQualityScores);
-  }
-  return result;
-}
-
 const _prepareResponseOptions = (translateOptions) => {
   let vectorResponseOptions = new Module.VectorResponseOptions;
   translateOptions.forEach(translateOption => {

From 1360941ab92a04f377592aba66d86b99ffa39f48 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerin.philip@research.iiit.ac.in>
Date: Thu, 3 Mar 2022 11:41:26 +0000
Subject: [PATCH 363/442] Enable dependabot to automate updating dependencies
 (#365)

Following marian-nmt/marian-dev.
---
 .github/dependabot.yml | 9 +++++++++
 1 file changed, 9 insertions(+)
 create mode 100644 .github/dependabot.yml

diff --git a/.github/dependabot.yml b/.github/dependabot.yml
new file mode 100644
index 000000000..bbb39076f
--- /dev/null
+++ b/.github/dependabot.yml
@@ -0,0 +1,9 @@
+version: 2
+
+updates:
+  # Maintain dependencies for Git Submodules
+  - package-ecosystem: "gitsubmodule"
+    directory: "/"
+    schedule:
+      interval: "daily"
+

From 89a96bf71eb2c8b6d1d5cb1e623969ca23d35690 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 3 Mar 2022 17:24:32 +0100
Subject: [PATCH 364/442] Use right range and threshold for showing "bad"
 words/sentences (#370)

* Use ln(0.5) as the threshold
* Use right range for showing "bad" words/sentences
---
 wasm/test_page/js/index.js | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/wasm/test_page/js/index.js b/wasm/test_page/js/index.js
index 166f0f27b..eec834cc8 100644
--- a/wasm/test_page/js/index.js
+++ b/wasm/test_page/js/index.js
@@ -59,13 +59,13 @@ const addQualityClasses = (root) => {
   // You can do this wit CSS variables, calc() and min/max, but JS is just easier
 
   root.querySelectorAll('[x-bergamot-sentence-score]').forEach(el => {
-    // Note: these thresholds are just examples, they are not good thresholds!  
-    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-sentence-score')) > -0.1);
+    // The threshold is ln(0.5) (https://github.com/browsermt/bergamot-translator/pull/370#issuecomment-1058123399)
+    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-sentence-score')) < -0.6931);
   });
 
   root.querySelectorAll('[x-bergamot-word-score]').forEach(el => {
-    // Note: these thresholds are just examples, they are not good thresholds!
-    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-word-score')) > -0.1);
+    // The threshold is ln(0.5) (https://github.com/browsermt/bergamot-translator/pull/370#issuecomment-1058123399)
+    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-word-score')) < -0.6931);
   });
 
   // Add tooltips to each (sub)word with sentence and word score.

From ab7f84f664d49432f841e20a1c8ef4dcf2cbd22c Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 7 Mar 2022 18:38:17 +0100
Subject: [PATCH 365/442] Bump version to 0.4.2 (#371)

---
 BERGAMOT_VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BERGAMOT_VERSION b/BERGAMOT_VERSION
index 5aff472dd..0eec13e47 100644
--- a/BERGAMOT_VERSION
+++ b/BERGAMOT_VERSION
@@ -1 +1 @@
-v0.4.1
+v0.4.2

From 22d6bc07e7a6e83bc22676a106fe0ae8fc691c15 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Wed, 9 Mar 2022 08:00:28 +0000
Subject: [PATCH 366/442] Bump 3rd_party/marian-dev from `08b1544` to `7e67124`
 (#372)

Bumps [3rd_party/marian-dev](https://github.com/browsermt/marian-dev) from `08b1544` to `7e67124`.
- [Commits](https://github.com/browsermt/marian-dev/compare/08b1544636fe13eaf1fbacb17c6fb050abfb8d42...7e67124ae0bc11b42f2e6373489831c9a2498499)

---
updated-dependencies:
- dependency-name: 3rd_party/marian-dev
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 08b154463..7e67124ae 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 08b1544636fe13eaf1fbacb17c6fb050abfb8d42
+Subproject commit 7e67124ae0bc11b42f2e6373489831c9a2498499

From 2c0e65c2ec19f62af444f53865d73ee003c86e6f Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 14 Mar 2022 18:05:22 +0100
Subject: [PATCH 367/442] JS: Reuse Model registry from
 firefox-translation-models for test page (#377)

* JS: Reuse Model registry from firefox-translation-models repo for test page

 - https://github.com/mozilla/firefox-translations-models/blob/main/registry.json
   is reused
 - Removed existing registry
---
 wasm/README.md                     |   1 +
 wasm/test_page/js/modelRegistry.js | 328 -----------------------------
 wasm/test_page/js/worker.js        |  19 +-
 3 files changed, 13 insertions(+), 335 deletions(-)
 delete mode 100644 wasm/test_page/js/modelRegistry.js

diff --git a/wasm/README.md b/wasm/README.md
index a0b3d7820..883f80dc5 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -16,6 +16,7 @@ Please refer to the file `test_page/js/worker.js` that demonstrates how to use t
     cd test_page
     git clone --depth 1 --branch main --single-branch https://github.com/mozilla/firefox-translations-models/
     mkdir models
+    cp -rf firefox-translations-models/registry.json models
     cp -rf firefox-translations-models/models/prod/* models
     cp -rf firefox-translations-models/models/dev/* models
     gunzip models/*/*
diff --git a/wasm/test_page/js/modelRegistry.js b/wasm/test_page/js/modelRegistry.js
deleted file mode 100644
index c8d6eda5e..000000000
--- a/wasm/test_page/js/modelRegistry.js
+++ /dev/null
@@ -1,328 +0,0 @@
-
-//const rootURL = "https://storage.googleapis.com/bergamot-models-sandbox/0.2.10";
-const rootURL = "../models";
-
-const modelRegistry = {
-  enit: {
-    vocab: {
-      name: "vocab.enit.spm",
-      size: 814128,
-      estimatedCompressedSize: 405338,
-      expectedSha256Hash:
-        "de8cbeb79e0139304bfa47e8559f2447016bf9906225a97d3df1baed4de8f3a3",
-    },
-    lex: {
-      name: "lex.50.50.enit.s2t.bin",
-      size: 4489920,
-      estimatedCompressedSize: 2409986,
-      expectedSha256Hash:
-        "bb1fad3b3f6a13ebce1698cf7f39ca736c4dea4525f3dab5e1a78436f07445e6",
-    },
-    model: {
-      name: "model.enit.intgemm.alphas.bin",
-      size: 17140836,
-      estimatedCompressedSize: 13283223,
-      expectedSha256Hash:
-        "a5ce3723f62ead92a0e0373b6df0ad8e3e6d22963adb1333984206e33b8b6c61",
-    },
-  },
-  enpt: {
-    vocab: {
-      name: "vocab.enpt.spm",
-      size: 812781,
-      estimatedCompressedSize: 406524,
-      expectedSha256Hash:
-        "633a3d782c79f7d5e4b94ab96848f47c2fdf8ba82dd99efd1742b8a696bbd0cc",
-    },
-    lex: {
-      name: "lex.50.50.enpt.s2t.bin",
-      size: 4472528,
-      estimatedCompressedSize: 2411984,
-      expectedSha256Hash:
-        "1e96599123d275afa37353dfe84677a4070f013494fbdc9c52a28445cc9bc38d",
-    },
-    model: {
-      name: "model.enpt.intgemm.alphas.bin",
-      size: 17140836,
-      estimatedCompressedSize: 13429592,
-      expectedSha256Hash:
-        "d968735704c75e33c2e183b9241f14c0b2a560d01d88a2728e5c0119a4d7fb22",
-    },
-  },
-  enru: {
-    vocab: {
-      name: "vocab.enru.spm",
-      size: 937157,
-      estimatedCompressedSize: 435776,
-      expectedSha256Hash:
-        "feca2d44f01b946c85faba3b15b5eb53344bec84cd14a1a4d4a82ddd774c5edd",
-    },
-    lex: {
-      name: "lex.50.50.enru.s2t.bin",
-      size: 3049096,
-      estimatedCompressedSize: 1579779,
-      expectedSha256Hash:
-        "7bd3e2c0a72286fe1f3da65c56c49a7cd77efa5f1d1a444e2a9e769480b96ff3",
-    },
-    model: {
-      name: "model.enru.intgemm.alphas.bin",
-      size: 17140836,
-      estimatedCompressedSize: 12853987,
-      expectedSha256Hash:
-        "4a45186a93b8a2dd9301c66a3b3dad580b1bcfa74aadda583ca383f9fe0dea93",
-    },
-  },
-  iten: {
-    vocab: {
-      name: "vocab.iten.spm",
-      size: 814151,
-      estimatedCompressedSize: 405416,
-      expectedSha256Hash:
-        "22d5ce6973be5360a921103acbe984a9bfca952a1f6c55c9cb5ef7de4fd58266",
-    },
-    lex: {
-      name: "lex.50.50.iten.s2t.bin",
-      size: 5238420,
-      estimatedCompressedSize: 2860178,
-      expectedSha256Hash:
-        "357d362373022b029ee9965975a133e6f36fdb0fed749202ff578365cf0111f8",
-    },
-    model: {
-      name: "model.iten.intgemm.alphas.bin",
-      size: 17140836,
-      estimatedCompressedSize: 13423308,
-      expectedSha256Hash:
-        "1fae546faeb9046f80b1b7e940b37b660974ce72902778181d6cd1c30b717f35",
-    },
-  },
-  pten: {
-    vocab: {
-      name: "vocab.pten.spm",
-      size: 812889,
-      estimatedCompressedSize: 406730,
-      expectedSha256Hash:
-        "8389979e3c965688b07aeb712a7e44406e5dcdb2b84087229d26fcc71448c4ed",
-    },
-    lex: {
-      name: "lex.50.50.pten.s2t.bin",
-      size: 5001420,
-      estimatedCompressedSize: 2733800,
-      expectedSha256Hash:
-        "212ed0ae44a6f920cd6d17ca02f0a523ba6c4b0ef5078ae310c20bc4c51484c5",
-    },
-    model: {
-      name: "model.pten.intgemm.alphas.bin",
-      size: 17140836,
-      estimatedCompressedSize: 13584764,
-      expectedSha256Hash:
-        "6c3b7af01772022a19712410c63342ba581468c2f1aac34d7488409c4043e697",
-    },
-  },
-  ruen: {
-    vocab: {
-      name: "vocab.ruen.spm",
-      size: 936576,
-      estimatedCompressedSize: 435801,
-      expectedSha256Hash:
-        "aaf9a325c0a988c507d0312cb6ba1a02bac7a370bcd879aedee626a40bfbda78",
-    },
-    lex: {
-      name: "lex.50.50.ruen.s2t.bin",
-      size: 5090836,
-      estimatedCompressedSize: 2684919,
-      expectedSha256Hash:
-        "e6667e22f5f86be4872e3768b7184727f5dd8c9f2ccfb0639baabcb1176f5d11",
-    },
-    model: {
-      name: "model.ruen.intgemm.alphas.bin",
-      size: 17140836,
-      estimatedCompressedSize: 13108893,
-      expectedSha256Hash:
-        "3b6a0305e3d232fadd54f5a765365b7b96ad6d8f2e818cba594b02fbd8fadb3d",
-    },
-  },
-  csen: {
-    vocab: {
-      name: "vocab.csen.spm",
-      size: 769763,
-      estimatedCompressedSize: 366392,
-      expectedSha256Hash:
-        "f71cc5d045e479607078e079884f44032f5a0b82547fb96eefa29cd1eb47c6f3",
-    },
-    lex: {
-      name: "lex.50.50.csen.s2t.bin",
-      size: 4535788,
-      estimatedCompressedSize: 2418488,
-      expectedSha256Hash:
-        "8228a3c3f7887759a62b7d7c674a7bef9b70161913f9b0939ab58f71186835c2",
-    },
-    model: {
-      name: "model.csen.intgemm.alphas.bin",
-      size: 17140756,
-      estimatedCompressedSize: 13045032,
-      expectedSha256Hash:
-        "5b16661e2864dc50b2f4091a16bdd4ec8d8283e04271e602159ba348df5d6e2d",
-    },
-  },
-  deen: {
-    vocab: {
-      name: "vocab.deen.spm",
-      size: 784269,
-      estimatedCompressedSize: 410738,
-      expectedSha256Hash:
-        "417668f2ed297970febafb5b079a9d5ebc4ed0b3550ac8386d67a90473a09bd7",
-    },
-    lex: {
-      name: "lex.50.50.deen.s2t.bin",
-      size: 5047568,
-      estimatedCompressedSize: 2657472,
-      expectedSha256Hash:
-        "2f7c0f7bbce97ae5b52454074a892ba7b7610fb98e3c5d341e4ca79f0850c4de",
-    },
-    model: {
-      name: "model.deen.intgemm.alphas.bin",
-      size: 17140837,
-      estimatedCompressedSize: 13091214,
-      expectedSha256Hash:
-        "dda44d87ab0d8ad3b3871122fd3ee385f37878183a8b4ec139cd909531ec5009",
-    },
-  },
-  encs: {
-    vocab: {
-      name: "vocab.csen.spm",
-      size: 769763,
-      estimatedCompressedSize: 366392,
-      expectedSha256Hash:
-        "f71cc5d045e479607078e079884f44032f5a0b82547fb96eefa29cd1eb47c6f3",
-    },
-    lex: {
-      name: "lex.50.50.encs.s2t.bin",
-      size: 3556124,
-      estimatedCompressedSize: 1913246,
-      expectedSha256Hash:
-        "e19c77231bf977988e31ff8db15fe79966b5170564bd3e10613f239e7f461d97",
-    },
-    model: {
-      name: "model.encs.intgemm.alphas.bin",
-      size: 17140756,
-      estimatedCompressedSize: 12630325,
-      expectedSha256Hash:
-        "9a2fe0588bd972accfc801e2f31c945de0557804a91666ae5ab43b94fb74ac4b",
-    },
-  },
-  ende: {
-    vocab: {
-      name: "vocab.deen.spm",
-      size: 797501,
-      estimatedCompressedSize: 412505,
-      expectedSha256Hash:
-        "bc8f8229933d8294c727f3eab12f6f064e7082b929f2d29494c8a1e619ba174c",
-    },
-    lex: {
-      name: "lex.50.50.ende.s2t.bin",
-      size: 3062492,
-      estimatedCompressedSize: 1575385,
-      expectedSha256Hash:
-        "764797d075f0642c0b079cce6547348d65fe4e92ac69fa6a8605cd8b53dacb3f",
-    },
-    model: {
-      name: "model.ende.intgemm.alphas.bin",
-      size: 17140498,
-      estimatedCompressedSize: 13207068,
-      expectedSha256Hash:
-        "f0946515c6645304f0706fa66a051c3b7b7c507f12d0c850f276c18165a10c14",
-    },
-  },
-  enes: {
-    vocab: {
-      name: "vocab.esen.spm",
-      size: 825463,
-      estimatedCompressedSize: 414566,
-      expectedSha256Hash:
-        "909b1eea1face0d7f90a474fe29a8c0fef8d104b6e41e65616f864c964ba8845",
-    },
-    lex: {
-      name: "lex.50.50.enes.s2t.bin",
-      size: 3347104,
-      estimatedCompressedSize: 1720700,
-      expectedSha256Hash:
-        "3a113d713dec3cf1d12bba5b138ae616e28bba4bbc7fe7fd39ba145e26b86d7f",
-    },
-    model: {
-      name: "model.enes.intgemm.alphas.bin",
-      size: 17140755,
-      estimatedCompressedSize: 12602853,
-      expectedSha256Hash:
-        "fa7460037a3163e03fe1d23602f964bff2331da6ee813637e092ddf37156ef53",
-    },
-  },
-  enet: {
-    vocab: {
-      name: "vocab.eten.spm",
-      size: 828426,
-      estimatedCompressedSize: 416995,
-      expectedSha256Hash:
-        "e3b66bc141f6123cd40746e2fb9b8ee4f89cbf324ab27d6bbf3782e52f15fa2d",
-    },
-    lex: {
-      name: "lex.50.50.enet.s2t.bin",
-      size: 2700780,
-      estimatedCompressedSize: 1336443,
-      expectedSha256Hash:
-        "3d1b40ff43ebef82cf98d416a88a1ea19eb325a85785eef102f59878a63a829d",
-    },
-    model: {
-      name: "model.enet.intgemm.alphas.bin",
-      size: 17140754,
-      estimatedCompressedSize: 12543318,
-      expectedSha256Hash:
-        "a28874a8b702a519a14dc71bcee726a5cb4b539eeaada2d06492f751469a1fd6",
-    },
-  },
-  esen: {
-    vocab: {
-      name: "vocab.esen.spm",
-      size: 825463,
-      estimatedCompressedSize: 414566,
-      expectedSha256Hash:
-        "909b1eea1face0d7f90a474fe29a8c0fef8d104b6e41e65616f864c964ba8845",
-    },
-    lex: {
-      name: "lex.50.50.esen.s2t.bin",
-      size: 3860888,
-      estimatedCompressedSize: 1978538,
-      expectedSha256Hash:
-        "f11a2c23ef85ab1fee1c412b908d69bc20d66fd59faa8f7da5a5f0347eddf969",
-    },
-    model: {
-      name: "model.esen.intgemm.alphas.bin",
-      size: 17140755,
-      estimatedCompressedSize: 13215960,
-      expectedSha256Hash:
-        "4b6b7f451094aaa447d012658af158ffc708fc8842dde2f871a58404f5457fe0",
-    },
-  },
-  eten: {
-    vocab: {
-      name: "vocab.eten.spm",
-      size: 828426,
-      estimatedCompressedSize: 416995,
-      expectedSha256Hash:
-        "e3b66bc141f6123cd40746e2fb9b8ee4f89cbf324ab27d6bbf3782e52f15fa2d",
-    },
-    lex: {
-      name: "lex.50.50.eten.s2t.bin",
-      size: 3974944,
-      estimatedCompressedSize: 1920655,
-      expectedSha256Hash:
-        "6992bedc590e60e610a28129c80746fe5f33144a4520e2c5508d87db14ca54f8",
-    },
-    model: {
-      name: "model.eten.intgemm.alphas.bin",
-      size: 17140754,
-      estimatedCompressedSize: 12222624,
-      expectedSha256Hash:
-        "aac98a2371e216ee2d4843cbe896c617f6687501e17225ac83482eba52fd0028",
-    },
-  },
-};
\ No newline at end of file
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index aa4d40497..fcbb37aa2 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -1,11 +1,15 @@
 // All variables specific to translation service
 var translationService = undefined;
 
+// Model registry
+let modelRegistry = undefined;
+
 // A map of language-pair to TranslationModel object
 var languagePairToTranslationModels = new Map();
 
 const BERGAMOT_TRANSLATOR_MODULE = "bergamot-translator-worker.js";
-const MODEL_REGISTRY = "modelRegistry.js";
+const MODEL_REGISTRY = "../models/registry.json";
+const MODEL_ROOT_URL = "../models/";
 const PIVOT_LANGUAGE = 'en';
 
 const encoder = new TextEncoder(); // string to utf-8 converter
@@ -18,9 +22,10 @@ var Module = {
     log(`Time until Module.preRun: ${(Date.now() - start) / 1000} secs`);
     moduleLoadStart = Date.now();
   }],
-  onRuntimeInitialized: function() {
+  onRuntimeInitialized: async function() {
     log(`Wasm Runtime initialized Successfully (preRun -> onRuntimeInitialized) in ${(Date.now() - moduleLoadStart) / 1000} secs`);
-    importScripts(MODEL_REGISTRY);
+    const response = await fetch(MODEL_REGISTRY);
+    modelRegistry = await response.json();
     postMessage([`import_reply`, modelRegistry]);
   }
 };
@@ -196,10 +201,10 @@ gemm-precision: int8shiftAlphaAll
 alignment: soft
 `;
 
-  const modelFile = `${rootURL}/${languagePair}/${modelRegistry[languagePair]["model"].name}`;
-  const shortlistFile = `${rootURL}/${languagePair}/${modelRegistry[languagePair]["lex"].name}`;
-  const vocabFiles = [`${rootURL}/${languagePair}/${modelRegistry[languagePair]["vocab"].name}`,
-                      `${rootURL}/${languagePair}/${modelRegistry[languagePair]["vocab"].name}`];
+  const modelFile = `${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair]["model"].name}`;
+  const shortlistFile = `${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair]["lex"].name}`;
+  const vocabFiles = [`${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair]["vocab"].name}`,
+                      `${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair]["vocab"].name}`];
 
   const uniqueVocabFiles = new Set(vocabFiles);
   log(`modelFile: ${modelFile}\nshortlistFile: ${shortlistFile}\nNo. of unique vocabs: ${uniqueVocabFiles.size}`);

From 0a52a6d405458083cb01131b25032261ba33b3a0 Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Tue, 15 Mar 2022 15:55:28 +0100
Subject: [PATCH 368/442] JS: Using supervised QE models for available language
 pairs (#378)

* JS: Refactored model loading
 - Passing single vocab memory via JS
* JS: Use supervised QE models when available
* Ran clang format
---
 wasm/bindings/service_bindings.cpp | 12 +++--
 wasm/test_page/js/worker.js        | 70 ++++++++++++++----------------
 2 files changed, 40 insertions(+), 42 deletions(-)

diff --git a/wasm/bindings/service_bindings.cpp b/wasm/bindings/service_bindings.cpp
index 8e4fe4d14..d56615dc6 100644
--- a/wasm/bindings/service_bindings.cpp
+++ b/wasm/bindings/service_bindings.cpp
@@ -45,11 +45,15 @@ std::vector<std::shared_ptr<AlignedMemory>> prepareVocabsSmartMemories(std::vect
 }
 
 MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory, AlignedMemory* shortlistMemory,
-                                 std::vector<AlignedMemory*> uniqueVocabsMemories) {
+                                 std::vector<AlignedMemory*> uniqueVocabsMemories,
+                                 AlignedMemory* qualityEstimatorMemory) {
   MemoryBundle memoryBundle;
   memoryBundle.model = std::move(*modelMemory);
   memoryBundle.shortlist = std::move(*shortlistMemory);
   memoryBundle.vocabs = std::move(prepareVocabsSmartMemories(uniqueVocabsMemories));
+  if (qualityEstimatorMemory != nullptr) {
+    memoryBundle.qualityEstimatorMemory = std::move(*qualityEstimatorMemory);
+  }
 
   return memoryBundle;
 }
@@ -57,9 +61,9 @@ MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory, AlignedMemory* shor
 // This allows only shared_ptrs to be operational in JavaScript, according to emscripten.
 // https://emscripten.org/docs/porting/connecting_cpp_and_javascript/embind.html#smart-pointers
 std::shared_ptr<TranslationModel> TranslationModelFactory(const std::string& config, AlignedMemory* model,
-                                                          AlignedMemory* shortlist,
-                                                          std::vector<AlignedMemory*> vocabs) {
-  MemoryBundle memoryBundle = prepareMemoryBundle(model, shortlist, vocabs);
+                                                          AlignedMemory* shortlist, std::vector<AlignedMemory*> vocabs,
+                                                          AlignedMemory* qualityEstimator) {
+  MemoryBundle memoryBundle = prepareMemoryBundle(model, shortlist, vocabs, qualityEstimator);
   return std::make_shared<TranslationModel>(config, std::move(memoryBundle));
 }
 
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
index fcbb37aa2..3327d8a3a 100644
--- a/wasm/test_page/js/worker.js
+++ b/wasm/test_page/js/worker.js
@@ -12,6 +12,14 @@ const MODEL_REGISTRY = "../models/registry.json";
 const MODEL_ROOT_URL = "../models/";
 const PIVOT_LANGUAGE = 'en';
 
+// Information corresponding to each file type
+const fileInfo = [
+  {"type": "model", "alignment": 256},
+  {"type": "lex", "alignment": 64},
+  {"type": "vocab", "alignment": 64},
+  {"type": "qualityModel", "alignment": 64}
+];
+
 const encoder = new TextEncoder(); // string to utf-8 converter
 const decoder = new TextDecoder(); // utf-8 to string converter
 
@@ -169,12 +177,17 @@ const _downloadAsArrayBuffer = async(url) => {
 // Constructs and initializes the AlignedMemory from the array buffer and alignment size
 const _prepareAlignedMemoryFromBuffer = async (buffer, alignmentSize) => {
   var byteArray = new Int8Array(buffer);
-  log(`Constructing Aligned memory. Size: ${byteArray.byteLength} bytes, Alignment: ${alignmentSize}`);
   var alignedMemory = new Module.AlignedMemory(byteArray.byteLength, alignmentSize);
-  log(`Aligned memory construction done`);
   const alignedByteArrayView = alignedMemory.getByteArrayView();
   alignedByteArrayView.set(byteArray);
-  log(`Aligned memory initialized`);
+  return alignedMemory;
+}
+
+async function prepareAlignedMemory(file, languagePair) {
+  const fileName = `${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair][file.type].name}`;
+  const buffer = await _downloadAsArrayBuffer(fileName);
+  const alignedMemory = await _prepareAlignedMemoryFromBuffer(buffer, file.alignment);
+  log(`"${file.type}" aligned memory prepared. Size:${alignedMemory.size()} bytes, alignment:${file.alignment}`);
   return alignedMemory;
 }
 
@@ -201,45 +214,26 @@ gemm-precision: int8shiftAlphaAll
 alignment: soft
 `;
 
-  const modelFile = `${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair]["model"].name}`;
-  const shortlistFile = `${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair]["lex"].name}`;
-  const vocabFiles = [`${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair]["vocab"].name}`,
-                      `${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair]["vocab"].name}`];
-
-  const uniqueVocabFiles = new Set(vocabFiles);
-  log(`modelFile: ${modelFile}\nshortlistFile: ${shortlistFile}\nNo. of unique vocabs: ${uniqueVocabFiles.size}`);
-  uniqueVocabFiles.forEach(item => log(`unique vocabFile: ${item}`));
+  const promises = [];
+  fileInfo.filter(file => modelRegistry[languagePair].hasOwnProperty(file.type))
+  .map((file) => {
+      promises.push(prepareAlignedMemory(file, languagePair));
+  });
 
-  // Download the files as buffers from the given urls
-  let start = Date.now();
-  const downloadedBuffers = await Promise.all([_downloadAsArrayBuffer(modelFile), _downloadAsArrayBuffer(shortlistFile)]);
-  const modelBuffer = downloadedBuffers[0];
-  const shortListBuffer = downloadedBuffers[1];
+  const alignedMemories = await Promise.all(promises);
 
-  const downloadedVocabBuffers = [];
-  for (let item of uniqueVocabFiles.values()) {
-    downloadedVocabBuffers.push(await _downloadAsArrayBuffer(item));
-  }
-  log(`Total Download time for all files of '${languagePair}': ${(Date.now() - start) / 1000} secs`);
-
-  // Construct AlignedMemory objects with downloaded buffers
-  let constructedAlignedMemories = await Promise.all([_prepareAlignedMemoryFromBuffer(modelBuffer, 256),
-                                                      _prepareAlignedMemoryFromBuffer(shortListBuffer, 64)]);
-  let alignedModelMemory = constructedAlignedMemories[0];
-  let alignedShortlistMemory = constructedAlignedMemories[1];
-  let alignedVocabsMemoryList = new Module.AlignedMemoryList;
-  for(let item of downloadedVocabBuffers) {
-    let alignedMemory = await _prepareAlignedMemoryFromBuffer(item, 64);
-    alignedVocabsMemoryList.push_back(alignedMemory);
+  log(`Translation Model config: ${modelConfig}`);
+  log(`Aligned memory sizes: Model:${alignedMemories[0].size()} Shortlist:${alignedMemories[1].size()} Vocab:${alignedMemories[2].size()}`);
+  const alignedVocabMemoryList = new Module.AlignedMemoryList();
+  alignedVocabMemoryList.push_back(alignedMemories[2]);
+  let translationModel;
+  if (alignedMemories.length === fileInfo.length) {
+    log(`QE:${alignedMemories[3].size()}`);
+    translationModel = new Module.TranslationModel(modelConfig, alignedMemories[0], alignedMemories[1], alignedVocabMemoryList, alignedMemories[3]);
   }
-  for (let vocabs=0; vocabs < alignedVocabsMemoryList.size(); vocabs++) {
-    log(`Aligned vocab memory${vocabs+1} size: ${alignedVocabsMemoryList.get(vocabs).size()}`);
+  else {
+    translationModel = new Module.TranslationModel(modelConfig, alignedMemories[0], alignedMemories[1], alignedVocabMemoryList, null);
   }
-  log(`Aligned model memory size: ${alignedModelMemory.size()}`);
-  log(`Aligned shortlist memory size: ${alignedShortlistMemory.size()}`);
-
-  log(`Translation Model config: ${modelConfig}`);
-  var translationModel = new Module.TranslationModel(modelConfig, alignedModelMemory, alignedShortlistMemory, alignedVocabsMemoryList);
   languagePairToTranslationModels.set(languagePair, translationModel);
 }
 

From 409b7d2265a9478264fa03ffa8e691338cedb31e Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri, 18 Mar 2022 11:27:53 +0000
Subject: [PATCH 369/442] Bump 3rd_party/marian-dev from `7e67124` to `844800e`
 (#382)

Bumps [3rd_party/marian-dev](https://github.com/browsermt/marian-dev) from `7e67124` to `844800e`.
- [Release notes](https://github.com/browsermt/marian-dev/releases)
- [Commits](https://github.com/browsermt/marian-dev/compare/7e67124ae0bc11b42f2e6373489831c9a2498499...844800efccba6e670250caac1735ca2c8c8e508e)

---
updated-dependencies:
- dependency-name: 3rd_party/marian-dev
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 7e67124ae..844800efc 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 7e67124ae0bc11b42f2e6373489831c9a2498499
+Subproject commit 844800efccba6e670250caac1735ca2c8c8e508e

From ed3160524d7050e308f612c67b11ea797fb2013f Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Wed, 23 Mar 2022 12:14:51 +0000
Subject: [PATCH 370/442] JS: Update languages & use Intl API for their display
 names (#379)

Got the languages from registry.json, including non-prod models.
Code now calls into `Intl.DisplayNames()`[1] to make life easier.

[1] (http://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DisplayNames/DisplayNames)
---
 wasm/test_page/js/index.js | 34 ++++++++++++++++------------------
 1 file changed, 16 insertions(+), 18 deletions(-)

diff --git a/wasm/test_page/js/index.js b/wasm/test_page/js/index.js
index eec834cc8..4eb86764b 100644
--- a/wasm/test_page/js/index.js
+++ b/wasm/test_page/js/index.js
@@ -7,17 +7,6 @@ const status = message => ($("#status").innerText = message);
 const langFrom = $("#lang-from");
 const langTo = $("#lang-to");
 
-const langs = [
-  ["en", "English"],
-  ["it", "Italian"],
-  ["pt", "Portuguese"],
-  ["ru", "Russian"],
-  ["cs", "Czech"],
-  ["de", "German"],
-  ["es", "Spanish"],
-  ["et", "Estonian"],
-];
-
 if (window.Worker) {
   worker = new Worker("js/worker.js");
   worker.postMessage(["import"]);
@@ -99,11 +88,6 @@ worker.onmessage = function (e) {
   }
 };
 
-langs.forEach(([code, name]) => {
-  langFrom.innerHTML += `<option value="${code}">${name}</option>`;
-  langTo.innerHTML += `<option value="${code}">${name}</option>`;
-});
-
 const loadModel = () => {
   const lngFrom = langFrom.value;
   const lngTo = langTo.value;
@@ -140,11 +124,25 @@ $('#output').addEventListener('mouseover', e => {
 })
 
 function init() {
+  // Populate langs
+  const langs = Array.from(new Set(Object.keys(modelRegistry).reduce((acc, key) => acc.concat([key.substr(0, 2), key.substr(2, 2)]), [])));
+  const langNames = new Intl.DisplayNames(undefined, {type: "language"});
+
+  // Sort languages by display name
+  langs.sort((a, b) => langNames.of(a).localeCompare(langNames.of(b)));
+
+  // Populate the dropdowns 
+  langs.forEach(code => {
+    const name = langNames.of(code);
+    langFrom.innerHTML += `<option value="${code}">${name}</option>`;
+    langTo.innerHTML += `<option value="${code}">${name}</option>`;
+  });
+
   // try to guess input language from user agent
   let myLang = navigator.language;
   if (myLang) {
     myLang = myLang.split("-")[0];
-    let langIndex = langs.findIndex(([code]) => code === myLang);
+    let langIndex = langs.indexOf(myLang);
     if (langIndex > -1) {
       console.log("guessing input language is", myLang);
       langFrom.value = myLang;
@@ -152,7 +150,7 @@ function init() {
   }
 
   // find first output lang that *isn't* input language
-  langTo.value = langs.find(([code]) => code !== langFrom.value)[0];
+  langTo.value = langs.find(code => code !== langFrom.value);
   // load this model
   loadModel();
 }

From 46882e7cfe37d8ffe4d6c2053562b18aba003f00 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Thu, 24 Mar 2022 15:05:45 +0000
Subject: [PATCH 371/442] JS: Fix swap button on test-page (#388)

---
 wasm/test_page/js/index.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/wasm/test_page/js/index.js b/wasm/test_page/js/index.js
index 4eb86764b..b1c308e8b 100644
--- a/wasm/test_page/js/index.js
+++ b/wasm/test_page/js/index.js
@@ -111,7 +111,7 @@ langTo.addEventListener("change", e => {
 
 $(".swap").addEventListener("click", e => {
   [langFrom.value, langTo.value] = [langTo.value, langFrom.value];
-  $("#input").value = $("#output").value;
+  $("#input").value = $("#output").innerText;
   loadModel();
 });
 

From 13443352c0abacef33c7e696f2ad2e5eb3a622c1 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Thu, 24 Mar 2022 19:26:20 +0000
Subject: [PATCH 372/442] Docs: Pin Jinja2 to last known working version (#389)

Fixes the docs workflow which is failing after pip is picking up Jinja 3.20.
We only need >=2.3, this one sets it to 3.0.3 builds were successful last.
---
 doc/requirements.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/requirements.txt b/doc/requirements.txt
index d95cc684c..778f08914 100644
--- a/doc/requirements.txt
+++ b/doc/requirements.txt
@@ -1,5 +1,6 @@
 sphinx==2.4.4
 breathe==4.13.0
+Jinja2==3.0.3
 exhale
 sphinx_rtd_theme
 mistune<2.0.0

From d2e3a826220db7d9c37115b890320b196ed124bf Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Mon, 28 Mar 2022 18:03:43 +0200
Subject: [PATCH 373/442] Bump version to 0.4.3 (#392)

---
 BERGAMOT_VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BERGAMOT_VERSION b/BERGAMOT_VERSION
index 0eec13e47..f87d474c4 100644
--- a/BERGAMOT_VERSION
+++ b/BERGAMOT_VERSION
@@ -1 +1 @@
-v0.4.2
+v0.4.3

From 7d51d109f790713e60912e1a927e0cef199fc58f Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Wed, 30 Mar 2022 09:41:15 +0100
Subject: [PATCH 374/442] Bump bergamot-translator-tests from `d03a9d3` to
 `7984d14` (#394)

Bumps [bergamot-translator-tests](https://github.com/browsermt/bergamot-translator-tests) from `d03a9d3` to `7984d14`.
- [Release notes](https://github.com/browsermt/bergamot-translator-tests/releases)
- [Commits](https://github.com/browsermt/bergamot-translator-tests/compare/d03a9d316d40ba45c475018287971523666bf51e...7984d140aef00489699d0b7711fa942816224294)

---
updated-dependencies:
- dependency-name: bergamot-translator-tests
  dependency-type: direct:production
...
---
 bergamot-translator-tests | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index d03a9d316..7984d140a 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit d03a9d316d40ba45c475018287971523666bf51e
+Subproject commit 7984d140aef00489699d0b7711fa942816224294

From df5db525132fb24b02f80ac07dc98ba02f536e92 Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Thu, 31 Mar 2022 12:12:33 +0100
Subject: [PATCH 375/442] Fix call to `isspace` (#396)

Documentation is explicit about only calling it with unsigned char, and Windows runtime is checking this.
---
 src/translator/html.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/translator/html.cpp b/src/translator/html.cpp
index 5180a5af5..421074aa1 100644
--- a/src/translator/html.cpp
+++ b/src/translator/html.cpp
@@ -48,7 +48,7 @@ void encodeEntities(marian::string_view const &input, std::string &output) {
 /// for determining where to insert an open or close tag.
 size_t countPrefixWhitespaces(marian::string_view const &input) {
   size_t size = 0;
-  while (size < input.size() && std::isspace(input[size])) ++size;
+  while (size < input.size() && std::isspace(static_cast<unsigned char>(input[size]))) ++size;
   return size;
 }
 

From f18a8835fab5c75929546bbd5200225e023c3269 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Thu, 14 Apr 2022 11:25:51 +0100
Subject: [PATCH 376/442] Bump 3rd_party/ssplit-cpp from `a08d6bc` to `49fde6d`
 (#408)

Bumps [3rd_party/ssplit-cpp](https://github.com/browsermt/ssplit-cpp) from `a08d6bc` to `49fde6d`.
- [Release notes](https://github.com/browsermt/ssplit-cpp/releases)
- [Commits](https://github.com/browsermt/ssplit-cpp/compare/a08d6bce20619a8475736832d5418458c14db9d4...49fde6df7ee9199aedb9571be800448192e3515c)

---
updated-dependencies:
- dependency-name: 3rd_party/ssplit-cpp
  dependency-type: direct:production
...
---
 3rd_party/ssplit-cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index a08d6bce2..49fde6df7 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit a08d6bce20619a8475736832d5418458c14db9d4
+Subproject commit 49fde6df7ee9199aedb9571be800448192e3515c

From 98af5945c556f7ec082f040d6e388cbc0b1bdaa9 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Fri, 15 Apr 2022 08:56:31 +0100
Subject: [PATCH 377/442] Update and fix windows CI (#410)

* Use a more vanilla windows workflow from translateLocally, remove the
complicated lukka/*. Also removes port overrides in the overall upgrade.
* Disable vcpkg binary caching
* Remove PCRE library hacks after upstream ssplit improvements
---
 .github/workflows/windows.yml                 | 66 ++++++++++-------
 .../ports/pcre2/pcre2-10.35_fix-uwp.patch     | 10 ---
 vcpkg-override/ports/pcre2/portfile.cmake     | 72 -------------------
 vcpkg-override/ports/pcre2/vcpkg.json         |  6 --
 4 files changed, 40 insertions(+), 114 deletions(-)
 delete mode 100644 vcpkg-override/ports/pcre2/pcre2-10.35_fix-uwp.patch
 delete mode 100644 vcpkg-override/ports/pcre2/portfile.cmake
 delete mode 100644 vcpkg-override/ports/pcre2/vcpkg.json

diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 391200fcd..569020452 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -77,36 +77,50 @@ jobs:
         echo "MKLROOT=${{ github.workspace }}\mkl" | Out-File -FilePath $env:GITHUB_ENV  -Encoding utf8 -Append
       shell: powershell
 
-    - name: Prepare vcpkg
-      uses: lukka/run-vcpkg@v7.4
-      with:
-        vcpkgArguments: protobuf pcre2 --overlay-ports="${{ github.workspace }}\vcpkg-override\ports\pcre2"
-        vcpkgGitCommitId: 8dddc6c899ce6fdbeab38b525a31e7f23cb2d5bb
-        vcpkgDirectory: ${{ github.workspace }}/vcpkg/
-        vcpkgTriplet: x64-windows-static
-
-    # Windows CPU only minimal build
-    - name: Build Release # @TODO this is actually a debug build until the ninja generator gets fixed
-      uses: lukka/run-cmake@v3
-      with:
-        buildDirectory: ${{ github.workspace }}/build
-        cmakeAppendedArgs: '-G Ninja
-          -DCMAKE_BUILD_TYPE="Release"
-          -DUSE_WASM_COMPATIBLE_SOURCE="OFF"
-          -DUSE_STATIC_LIBS="TRUE" 
-          -DCMAKE_CXX_COMPILER_LAUNCHER=${{github.workspace}}\ccache.exe
-          -DCMAKE_C_COMPILER_LAUNCHER=${{github.workspace}}\ccache.exe
-        '
-        cmakeListsOrSettingsJson: CMakeListsTxtAdvanced
-        cmakeListsTxtPath: ${{ github.workspace }}/CMakeLists.txt
-        useVcpkgToolchainFile: true
-        cmakeBuildType: Release
+    - name: Disable debug vcpkg build
+      shell: powershell
+      working-directory: C:\vcpkg\triplets
+      run: |
+         $PSDefaultParameterValues['Out-File:Encoding'] = 'utf8' # Powershell murders me.
+         echo "set(VCPKG_BUILD_TYPE release)" | Tee-Object -FilePath x64-windows-static.cmake -Append
+         echo "set(VCPKG_BUILD_TYPE release)" | Tee-Object -FilePath x64-windows.cmake -Append
+         cat x64-windows-static.cmake
+         cat x64-windows.cmake
+
+    - name: Install dependencies with vcpkg
+      working-directory: C:\vcpkg
+      run: |
+        $Env:VCPKG_BUILD_TYPE = 'release'
+        $Env:VCPKG_DEFAULT_TRIPLET = 'x64-windows-static' # QT6 version, linguist tools not working yet: qtbase:x64-windows-static qttools:x64-windows-static qtsvg:x64-windows-static
+        .\vcpkg install protobuf:x64-windows-static pcre2:x64-windows-static 
+        .\vcpkg upgrade --no-dry-run # In case there are new builds available after cache restoration
+      shell: powershell
+        
+    - name: Create Build Environment
+      # Some projects don't allow in-source building, so create a separate build directory
+      # We'll use this as our working directory for all subsequent commands
+      run: cmake -E make_directory ${{github.workspace}}/build
+      
+    - name: Configure
+      working-directory: ${{github.workspace}}/build #@TODO figure out how variables are accessed from power shell, as they seem to not be read.
+      run: |
+          cmake .. -DCMAKE_BUILD_TYPE=Release -DUSE_STATIC_LIBS=ON  -DVCPKG_TARGET_TRIPLET='x64-windows-static' `
+                -DCMAKE_TOOLCHAIN_FILE="C:/vcpkg/scripts/buildsystems/vcpkg.cmake"  `
+                -DCMAKE_CXX_COMPILER_LAUNCHER=${{github.workspace}}\ccache.exe `
+                -DCMAKE_C_COMPILER_LAUNCHER=${{github.workspace}}\ccache.exe
+      shell: powershell
+
+    - name: Build
+      working-directory: ${{github.workspace}}/build
+      run: cmake --build . --config Release -j3
+      shell: powershell
 
 
     - name: Print versions
-      working-directory: build
+      working-directory: ${{github.workspace}}/build
       run: |
-        .\app\bergamot.exe --version
+        .\app\Release\bergamot.exe --version
+
       shell: cmd
 
     - name: ccache epilog
diff --git a/vcpkg-override/ports/pcre2/pcre2-10.35_fix-uwp.patch b/vcpkg-override/ports/pcre2/pcre2-10.35_fix-uwp.patch
deleted file mode 100644
index 476dde0f6..000000000
--- a/vcpkg-override/ports/pcre2/pcre2-10.35_fix-uwp.patch
+++ /dev/null
@@ -1,10 +0,0 @@
---- a/CMakeLists.txt	2020-05-09 16:43:10.000000000 +0200
-+++ b/CMakeLists.txt	2020-06-03 20:57:17.026182500 +0200
-@@ -619,6 +619,7 @@
- 
- IF(MSVC)
-   ADD_DEFINITIONS(-D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS)
-+  add_compile_options(/wd4146)
- ENDIF(MSVC)
- 
- SET(CMAKE_INCLUDE_CURRENT_DIR 1)
diff --git a/vcpkg-override/ports/pcre2/portfile.cmake b/vcpkg-override/ports/pcre2/portfile.cmake
deleted file mode 100644
index 641af1cd1..000000000
--- a/vcpkg-override/ports/pcre2/portfile.cmake
+++ /dev/null
@@ -1,72 +0,0 @@
-set(PCRE2_VERSION 10.37)
-set(EXPECTED_SHA f91760a8e0747f52211612fb0e134d685e224d16bd884eb574718d077a586b1fd7b6435d4e3b75c879b12e02b252467ecc28cdc4bc2903c783dacab089f99c99)
-set(PATCHES
-        pcre2-10.35_fix-uwp.patch
-)
-
-vcpkg_download_distfile(ARCHIVE
-    URLS "https://sourceforge.net/projects/pcre/files/pcre2/${PCRE2_VERSION}/pcre2-${PCRE2_VERSION}.zip"
-    FILENAME "pcre2-${PCRE2_VERSION}.zip"
-    SHA512 ${EXPECTED_SHA}
-    SILENT_EXIT
-)
-
-if (EXISTS "${ARCHIVE}")
-    vcpkg_extract_source_archive_ex(
-        OUT_SOURCE_PATH SOURCE_PATH
-        ARCHIVE ${ARCHIVE}
-        PATCHES ${PATCHES}
-    )
-else()
-    vcpkg_from_sourceforge(
-        OUT_SOURCE_PATH SOURCE_PATH
-        REPO pcre/pcre2
-        REF ${PCRE2_VERSION}
-        FILENAME "pcre2-${PCRE2_VERSION}.zip"
-        SHA512 ${EXPECTED_SHA}
-        PATCHES ${PATCHES}
-    )
-endif()
-
-if(VCPKG_CMAKE_SYSTEM_NAME STREQUAL "Emscripten" OR VCPKG_CMAKE_SYSTEM_NAME STREQUAL "iOS")
-    set(JIT OFF)
-else()
-    set(JIT ON)
-endif()
-
-vcpkg_configure_cmake(
-    SOURCE_PATH ${SOURCE_PATH}
-    PREFER_NINJA
-    OPTIONS
-        -DPCRE2_BUILD_PCRE2_8=ON
-        -DPCRE2_BUILD_PCRE2_16=ON
-        -DPCRE2_BUILD_PCRE2_32=ON
-        -DPCRE2_SUPPORT_JIT=${JIT}
-        -DPCRE2_SUPPORT_UNICODE=ON
-        -DPCRE2_BUILD_TESTS=OFF
-        -DPCRE2_BUILD_PCRE2GREP=OFF)
-
-vcpkg_install_cmake()
-
-file(READ ${CURRENT_PACKAGES_DIR}/include/pcre2.h PCRE2_H)
-if(VCPKG_LIBRARY_LINKAGE STREQUAL "static")
-    string(REPLACE "defined(PCRE2_STATIC)" "1" PCRE2_H "${PCRE2_H}")
-else()
-    string(REPLACE "defined(PCRE2_STATIC)" "0" PCRE2_H "${PCRE2_H}")
-endif()
-file(WRITE ${CURRENT_PACKAGES_DIR}/include/pcre2.h "${PCRE2_H}")
-
-vcpkg_fixup_pkgconfig()
-
-vcpkg_copy_pdbs()
-
-file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/man)
-file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/share/doc)
-file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/include)
-file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/man)
-file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/share)
-if(VCPKG_LIBRARY_LINKAGE STREQUAL "static")
-    file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/bin" "${CURRENT_PACKAGES_DIR}/debug/bin")
-endif()
-
-file(INSTALL ${SOURCE_PATH}/COPYING DESTINATION ${CURRENT_PACKAGES_DIR}/share/${PORT} RENAME copyright)
diff --git a/vcpkg-override/ports/pcre2/vcpkg.json b/vcpkg-override/ports/pcre2/vcpkg.json
deleted file mode 100644
index 80d87e8fe..000000000
--- a/vcpkg-override/ports/pcre2/vcpkg.json
+++ /dev/null
@@ -1,6 +0,0 @@
-{
-  "name": "pcre2",
-  "version-string": "10.37",
-  "description": "PCRE2 is a re-working of the original Perl Compatible Regular Expressions library",
-  "homepage": "https://pcre.org/"
-}

From e34420647ddc528dac47ec57bab13771bd05abef Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Wed, 20 Apr 2022 01:39:32 +0200
Subject: [PATCH 378/442] Upgrade emsdk to 3.1.8 (#414)

* Rework WASM compilation options

Necessary to work with newer versions of emscripten that are more picky about which option goes to the compiler, and which to the linker. Also took the opportunity to remove the need for the patching of the bergamot-translation-worker.js file, this can now easily be done through supported apis. Furthermore, I tried to downsize the generated javascript and wasm code a bit.

Initial estimates show that bergamot-translator compiled with emscripten 3.0.0 runs at about 3x the speed of 2.0.9 (when using embedded intgemm). Speed-up when using mozIntGemm is less dramatic.

* Updated marian-dev submodule
* Revert changes specific to patching external gemm modules for wasm
* Better Compilation and Link flags

 - Added "-O3" optimization flag for linking as well
 - "-g2" only for release and debug builds
 - "-g1" for release builds
 - Replaced deprecated "--bind" flag with "-lembind"
 - Removed redundant link flag

* Upgraded emsdk to 3.1.8
* Enclosed EXPORTED_FUNCTIONS values in a list
* Fixed the remaining 2.0.9 reference in circle ci build script
* Updated README

Co-authored-by: Jelmer van der Linde <jelmer@ikhoefgeen.nl>
---
 .circleci/config.yml          |  4 +--
 .github/workflows/build.yml   |  2 +-
 3rd_party/CMakeLists.txt      |  1 +
 3rd_party/marian-dev          |  2 +-
 CMakeLists.txt                | 50 +++++++++++++++++++++++++++++++++--
 README.md                     |  4 +--
 build-wasm.sh                 |  4 +--
 src/translator/CMakeLists.txt |  1 +
 wasm/CMakeLists.txt           | 18 +++----------
 9 files changed, 61 insertions(+), 25 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 7275fd4be..140e3116d 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -2,7 +2,7 @@ version: 2.1
 jobs:
   build-with-wormhole:
     docker:
-      - image: 'emscripten/emsdk:2.0.9'
+      - image: 'emscripten/emsdk:3.1.8'
     resource_class: medium
 
     working_directory: ~/checkout
@@ -48,7 +48,7 @@ jobs:
 
   build-without-wormhole:
     docker:
-      - image: 'emscripten/emsdk:2.0.9'
+      - image: 'emscripten/emsdk:3.1.8'
     resource_class: medium
 
     working_directory: ~/checkout
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 2246b3c50..da3c37018 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -11,7 +11,7 @@ name: "Build"
       - '**'
 env:
   qt_version: "6.2.1" # only used by build-macos
-  emsdk_version: 2.0.9 # For use in emscripten build
+  emsdk_version: 3.1.8 # For use in emscripten build
   ccache_basedir: ${{ github.workspace }}
   ccache_dir: "${{ github.workspace }}/.ccache"
   ccache_compilercheck: content
diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index 72a49e83a..1888d6da6 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -5,6 +5,7 @@ add_subdirectory(marian-dev EXCLUDE_FROM_ALL)
 if(COMPILE_WASM)
   # This is a bad way of adding compilation flags. Will be improved soon.
   add_compile_options(${WASM_COMPILE_FLAGS})
+  add_link_options(${WASM_LINK_FLAGS})
 endif(COMPILE_WASM)
 
 add_subdirectory(ssplit-cpp EXCLUDE_FROM_ALL)
diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 844800efc..199201eb8 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 844800efccba6e670250caac1735ca2c8c8e508e
+Subproject commit 199201eb89b2941afdadb14164e936d412f897ad
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f6e6af4f5..dc51acf80 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -112,9 +112,55 @@ message(STATUS "Project name: ${PROJECT_NAME}")
 message(STATUS "Project version: ${PROJECT_VERSION_STRING_FULL}")
 
 if(COMPILE_WASM)
+  # See https://github.com/emscripten-core/emscripten/blob/main/src/settings.js
   set(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
-  list(APPEND WASM_COMPILE_FLAGS -O3 -g2 -fPIC -mssse3 -msimd128)
-  list(APPEND WASM_COMPILE_FLAGS "SHELL:-s WASM=1" "SHELL:-s ASSERTIONS=0" "SHELL:-s DISABLE_EXCEPTION_CATCHING=1" "SHELL:-s LLD_REPORT_UNDEFINED" "SHELL:-s FORCE_FILESYSTEM=1" "SHELL:-s ALLOW_MEMORY_GROWTH=1")
+  list(APPEND WASM_COMPILE_FLAGS
+    -O3
+    # Preserve whitespaces in JS even for release builds; this doesn't increase wasm binary size
+    $<$<CONFIG:Release>:-g1>
+    # Relevant Debug info only for release with debug builds as this increases wasm binary size
+    $<$<CONFIG:RelWithDebInfo>:-g2>
+    -fPIC
+    -mssse3
+    -msimd128
+    # -fno-exceptions # Can't do that because spdlog uses exceptions
+    -sDISABLE_EXCEPTION_CATCHING=1
+    -sSTRICT=1
+  )
+  list(APPEND WASM_LINK_FLAGS
+    -O3
+    # Preserve whitespaces in JS even for release builds; this doesn't increase wasm binary size
+    $<$<CONFIG:Release>:-g1>
+    # Relevant Debug info only for release with debug builds as this increases wasm binary size
+    $<$<CONFIG:RelWithDebInfo>:-g2>
+    -lembind
+    # Save some code, and some speed
+    -sASSERTIONS=0
+    -sDISABLE_EXCEPTION_CATCHING=1
+    # the intgemm functions we call will be undefined since these are linked at
+    # runtime by our own javascript.
+    -sLLD_REPORT_UNDEFINED
+    -sERROR_ON_UNDEFINED_SYMBOLS=0
+    # Cause we can!
+    -sSTRICT=1
+    # You know we need it
+    -sALLOW_MEMORY_GROWTH=1
+    -sENVIRONMENT=web,worker
+    # No need to call main(), there's nothing there.
+    -sINVOKE_RUN=0
+    # No need for filesystem code in the generated Javascript
+    -sFILESYSTEM=0
+    # If you turn this on, it will mangle names which makes the dynamic linking hard.
+    -sDECLARE_ASM_MODULE_EXPORTS=0
+    # Export all of the intgemm functions in case we need to fall back to using the embedded intgemm
+    -sEXPORTED_FUNCTIONS=[_int8PrepareAFallback,_int8PrepareBFallback,_int8PrepareBFromTransposedFallback,_int8PrepareBFromQuantizedTransposedFallback,_int8PrepareBiasFallback,_int8MultiplyAndAddBiasFallback,_int8SelectColumnsOfBFallback]
+    # Necessary for mozintgemm linking. This prepares the `wasmMemory` variable ahead of time as
+    # opposed to delegating that task to the wasm binary itself. This way we can link MozIntGEMM
+    # module to the same memory as the main bergamot-translator module.
+    -sIMPORTED_MEMORY=1
+    # Dynamic execution is either frowned upon or blocked inside browser extensions
+    -sDYNAMIC_EXECUTION=0
+  )
 endif(COMPILE_WASM)
 
 # Needs to be enabled before including the folder containing tests (src/tests)
diff --git a/README.md b/README.md
index 11f144cce..b70c818ec 100644
--- a/README.md
+++ b/README.md
@@ -23,8 +23,8 @@ Building on wasm requires Emscripten toolchain. It can be downloaded and install
 
 * Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
 * Enter the cloned directory: `cd emsdk`
-* Install the lastest sdk tools: `./emsdk install 2.0.9`
-* Activate the latest sdk tools: `./emsdk activate 2.0.9`
+* Install the sdk: `./emsdk install 3.1.8`
+* Activate the sdk: `./emsdk activate 3.1.8`
 * Activate path variables: `source ./emsdk_env.sh`
 
 #### <a name="Compile"></a> Compile
diff --git a/build-wasm.sh b/build-wasm.sh
index adc6556c3..ff12013d1 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -51,8 +51,8 @@ if [ "$EMSDK" == "" ]; then
   fi
   if [ "$EMSDK_UPDATE_REQUIRED" == "1" ]; then
     cd emsdk
-    ./emsdk install 2.0.9
-    ./emsdk activate 2.0.9
+    ./emsdk install 3.1.8
+    ./emsdk activate 3.1.8
     cd -
   fi
   source ./emsdk/emsdk_env.sh
diff --git a/src/translator/CMakeLists.txt b/src/translator/CMakeLists.txt
index 2beb2e925..1d773b46b 100644
--- a/src/translator/CMakeLists.txt
+++ b/src/translator/CMakeLists.txt
@@ -31,6 +31,7 @@ if(COMPILE_WASM)
   # Enable code that is required for generating JS bindings
   target_compile_definitions(bergamot-translator PRIVATE WASM_BINDINGS)
   target_compile_options(bergamot-translator PRIVATE ${WASM_COMPILE_FLAGS})
+  target_link_options(bergamot-translator PRIVATE ${WASM_LINK_FLAGS})
 endif(COMPILE_WASM)
 
 if(ENABLE_CACHE_STATS)
diff --git a/wasm/CMakeLists.txt b/wasm/CMakeLists.txt
index 92c9e1698..ef8fd988a 100644
--- a/wasm/CMakeLists.txt
+++ b/wasm/CMakeLists.txt
@@ -14,27 +14,15 @@ target_include_directories(bergamot-translator-worker
     PRIVATE ${CMAKE_SOURCE_DIR}/src/translator
     PRIVATE ${CMAKE_SOURCE_DIR}
 )
+
 # This compile definition is required for generating binding code properly
 target_compile_definitions(bergamot-translator-worker PRIVATE WASM_BINDINGS)
 target_compile_options(bergamot-translator-worker PRIVATE ${WASM_COMPILE_FLAGS})
-
-set(LINKER_FLAGS "-g2 --bind -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=1 -s ALLOW_MEMORY_GROWTH=1 -s NO_DYNAMIC_EXECUTION=1 -s EXPORTED_RUNTIME_METHODS=[addOnPreMain]")
-
-# Avoid node.js-code in emscripten glue-code
-set(LINKER_FLAGS "${LINKER_FLAGS} -s ENVIRONMENT=web,worker")
-
-# Append version information in the Javascript artifact
-set(LINKER_FLAGS "${LINKER_FLAGS} --extern-pre-js ${CMAKE_CURRENT_BINARY_DIR}/project_version.js")
-
-# Allow importing undefined symbols dynamically
-set(LINKER_FLAGS "${LINKER_FLAGS} -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s DECLARE_ASM_MODULE_EXPORTS=0")
-
-# Export all the functions of fallback implementation of GEMM for wasm target
-set(LINKER_FLAGS "${LINKER_FLAGS} -s EXPORTED_FUNCTIONS=[_int8PrepareAFallback,_int8PrepareBFallback,_int8PrepareBFromTransposedFallback,_int8PrepareBFromQuantizedTransposedFallback,_int8PrepareBiasFallback,_int8MultiplyAndAddBiasFallback,_int8SelectColumnsOfBFallback]")
+target_link_options(bergamot-translator-worker PRIVATE ${WASM_LINK_FLAGS})
+target_link_options(bergamot-translator-worker PRIVATE --extern-pre-js=${CMAKE_CURRENT_BINARY_DIR}/project_version.js)
 
 set_target_properties(bergamot-translator-worker PROPERTIES
                         SUFFIX ".js"
-                        LINK_FLAGS ${LINKER_FLAGS}
                         RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}
                       )
 

From 5ae1b1ebb3fa9a3eabed8a64ca6798154bd486eb Mon Sep 17 00:00:00 2001
From: Abhishek Aggarwal <66322306+abhi-agg@users.noreply.github.com>
Date: Thu, 28 Apr 2022 16:24:13 +0200
Subject: [PATCH 379/442] Bump version to 0.4.4 (#415)

---
 BERGAMOT_VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BERGAMOT_VERSION b/BERGAMOT_VERSION
index f87d474c4..79b0815e6 100644
--- a/BERGAMOT_VERSION
+++ b/BERGAMOT_VERSION
@@ -1 +1 @@
-v0.4.3
+v0.4.4

From ad781656feea85ba2544ed7087e589236cd581e6 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Wed, 18 May 2022 16:17:53 +0100
Subject: [PATCH 380/442] Bump 3rd_party/marian-dev from `199201e` to `e88c1aa`
 (#416)

Bumps [3rd_party/marian-dev](https://github.com/browsermt/marian-dev) from `199201e` to `e88c1aa`.
- [Release notes](https://github.com/browsermt/marian-dev/releases)
- [Commits](https://github.com/browsermt/marian-dev/compare/199201eb89b2941afdadb14164e936d412f897ad...e88c1aa5d5c5622cb52c7df09fbb7c3d7f4b5b5a)

---
updated-dependencies:
- dependency-name: 3rd_party/marian-dev
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 199201eb8..e88c1aa5d 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 199201eb89b2941afdadb14164e936d412f897ad
+Subproject commit e88c1aa5d5c5622cb52c7df09fbb7c3d7f4b5b5a

From 61d2c35dbd4d57aa11a6b3334afc66be1e72e9cb Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Mon, 20 Jun 2022 14:35:29 +0100
Subject: [PATCH 381/442] Set up python packaging for pypi distribution (#424)

Old GitHub CI using Ubuntu and MacOS explicitly and building wheels have
been removed in favour of the more portable pypa specified builds. These
wheels should work just as well across a wider range of distributions.

pybind11:CMakeLists.txt requires Development.Module instead of
Development.* to avoid Embed from getting in the way of manylinux
builds.

manylinux_x86_64 builds are added for cp3.6 - 3.10. The linux build
uses an old image via docker.  Since the docker images are able to use
shared ccache folder, builds quite fast on warm starts.

ccache usage in setup.py is now triggered by an environment variable.
This allows for builds not to fail if ccache not present.

On tag pushes corresponding to versions, CI is configured to deliver
built wheels to PyPI, reading from repository secrets.

Improves setup.py including documentation and some formatting, and
additional links to source.

Fixes: #315
---
 .github/workflows/build.yml    | 340 +++++++++++++--------------------
 bindings/python/CMakeLists.txt |   2 +-
 bindings/python/README.md      |  14 ++
 setup.py                       |  50 ++++-
 4 files changed, 191 insertions(+), 215 deletions(-)
 create mode 100644 bindings/python/README.md

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index da3c37018..2ae9b998e 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -21,215 +21,143 @@ env:
   ccache_cmake: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
 
 jobs:
-    python-ubuntu:
+    build-wheels:
       strategy:
-        fail-fast: false
         matrix:
-          include:
-              - name: "Ubuntu 18.04 / py3.6"
-                os: "ubuntu-18.04"
-                python-version: "3.6"
-              - name: "Ubuntu 18.04 / py3.7"
-                os: "ubuntu-18.04"
-                python-version: "3.7"
-              - name: "Ubuntu 20.04 / py3.8"
-                os: "ubuntu-20.04"
-                python-version: "3.8"
-              - name: "Ubuntu 20.04 / py3.9"
-                os: "ubuntu-20.04"
-                python-version: "3.9"
-              - name: "Ubuntu 20.04 / py3.10"
-                os: "ubuntu-20.04"
-                python-version: "3.10"
-
-      name: ${{ matrix.name }}
-      runs-on: ${{ matrix.os }}
-      steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-        with:
-          submodules: recursive
-
-      - name: Set up Python
-        uses: actions/setup-python@v2
-        with:
-          python-version: ${{ matrix.python-version }}
-
-
-      - name: Install Dependencies
-        run: |-
-          sudo apt-get update
-          sudo apt-get install -y \
-            ccache  libprotobuf-dev protobuf-compiler \
-            python3-setuptools python3-pybind11 
-
-      - name: Install MKL
-        run: |-
-          wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB" | sudo apt-key add -
-          sudo sh -c "echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list"
-          sudo apt-get update -o Dir::Etc::sourcelist="/etc/apt/sources.list.d/intel-mkl.list"
-          sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
-
-      - name: Generate ccache_vars for ccache based on machine
-        shell: bash
-        id: ccache_vars
-        run: |-
-          echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
-          echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
-
-      - name: Cache-op for build-cache through ccache
-        uses: actions/cache@v2
-        with:
-          path: ${{ env.ccache_dir }}
-          key: ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-          restore-keys: |-
-            ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
-            ccache-${{ matrix.name }}-${{ steps.ccache_vars.outputs.hash }}
-            ccache-${{ matrix.name }}
-      - name: ccache environment setup
-        run: |-
-          echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
-          echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
-          echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
-          echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
-          echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
-          echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
-
-      - name: ccache prolog
-        run: |-
-          ccache -s # Print current cache stats
-          ccache -z # Zero cache entry
-
-      - name: Inject local version identifier for non tag builds
-        if: ${{ !startsWith(github.ref, 'refs/tags/v') }}
-        run: |-
-          echo "PYTHON_LOCAL_VERSION_IDENTIFIER=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
+          os: [ubuntu-20.04, macos-10.15]
+        fail-fast: false
 
-      - name: setup.py 
-        run: |-
-          python3 -m pip install wheel
-          BUILD_ARCH=core-avx-i python3 setup.py bdist_wheel --universal
+      name: "cibuildwheel / ${{ matrix.os }}"
+      runs-on: ${{ matrix.os }}
 
-      # We're happy with just compile for the moment, so cache gets some seeding.
-      - name: Install onto root python lib
-        run: |-
-          python3 -m pip install --ignore-installed dist/bergamot-*.whl 
+      steps:
+        - uses: actions/checkout@v2
+          with:
+            submodules: recursive
 
-      - name: Fetch models from translateLocally repository.
-        run: |-
-          python3 -m bergamot download -m en-de-tiny
-          python3 -m bergamot download -m de-en-tiny
-          python3 -m bergamot ls
+        - name: Generate ccache_vars for ccache based on machine
+          shell: bash
+          id: ccache_vars
+          run: |-
+            echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
+            echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
 
-      - name: Fetch models from opus repository.
-        run: |-
-          python3 -m bergamot download -m eng-fin-tiny -r opus
-          python3 -m bergamot ls -r opus
+        - name: Cache-op for build-cache through ccache
+          uses: actions/cache@v2
+          with:
+            path: ${{ env.ccache_dir }}
+            key: ccache-cibuildwheel-${{ matrix.os }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+            restore-keys: |-
+              ccache-cibuildwheel-${{ matrix.os }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
+              ccache-cibuildwheel-${{ matrix.os }}-${{ steps.ccache_vars.outputs.hash }}
+              ccache-cibuildwheel-${{ matrix.os }}
 
-      - name: Run the sample python script shipped with module
-        run: |-
-          python3 -m bergamot translate --model en-de-tiny <<< "Hello World"
-          python3 -m bergamot translate --model en-de-tiny de-en-tiny <<< "Hello World"
-          python3 -m bergamot translate --model eng-fin-tiny --repository opus <<< "Hello World"
+        - name: ccache environment setup
+          run: |-
+            mkdir -p ${{ env.ccache_dir }}
 
-      - name: ccache epilog
-        run: 'ccache -s # Print current cache stats'
+        - name: Inject local version identifier for non tag builds
+          if: ${{ !startsWith(github.ref, 'refs/tags/v') }}
+          run: |-
+            echo "PYTHON_LOCAL_VERSION_IDENTIFIER=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
 
-      - uses: actions/upload-artifact@v2
-        with:
-            path: ${{github.workspace}}/dist/bergamot-*.whl
+        - name: Apply MacOS patch
+          if: ${{ startsWith(runner.os, 'mac') }}
+          run: |
+            patch -p1 < patches/01-marian-fstream-for-macos.patch
 
+        - name: Build wheels
+          uses: pypa/cibuildwheel@v2.6.1
+          # to supply options, put them in 'env', like:
+          env:
+            CIBW_ENVIRONMENT_LINUX:
+              BUILD_ARCH=core-avx-i
+              USE_CCACHE=1
+              CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}
+              CCACHE_COMPRESS=${{ env.ccache_compress }}
+              CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}
+              CCACHE_MAXSIZE=${{ env.ccache_maxsize }}
+              PYTHON_LOCAL_VERSION_IDENTIFIER=${{ env.PYTHON_LOCAL_VERSION_IDENTIFIER }}
+              CCACHE_DIR=/host/${{ env.ccache_dir }}
+              CCACHE_BASEDIR=/host/${{ env.ccache_basedir }}
+
+            CIBW_ENVIRONMENT_MACOS:
+              BUILD_ARCH=core-avx-i
+              USE_CCACHE=1
+              CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}
+              CCACHE_COMPRESS=${{ env.ccache_compress }}
+              CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}
+              CCACHE_MAXSIZE=${{ env.ccache_maxsize }}
+              PYTHON_LOCAL_VERSION_IDENTIFIER=${{ env.PYTHON_LOCAL_VERSION_IDENTIFIER }}
+              CCACHE_DIR=${{ env.ccache_dir }}
+              CCACHE_BASEDIR=${{ env.ccache_basedir }}
+              MACOSX_DEPLOYMENT_TARGET=10.15
+
+            CIBW_BEFORE_BUILD_LINUX: |
+              yum install -y ccache
+
+              # Install Intel MKL. 
+              yum-config-manager -y --add-repo https://yum.repos.intel.com/mkl/setup/intel-mkl.repo
+              yum install -y intel-mkl
+
+              chmod -R a+rwx /host/${{ env.ccache_dir }}
+
+              ccache -s # Print current cache stats
+              ccache -z # Zero cache entry
+
+            CIBW_BEFORE_BUILD_MACOS: |
+              brew install openblas protobuf ccache boost pybind11 
+              chmod -R a+rwx ${{ env.ccache_dir }}
+              ccache -s # Print current cache stats
+              ccache -z # Zero cache entry
+
+            CIBW_BUILD: "cp{36,37,38,39,310}-*manylinux_x86_64 cp310-macosx_x86_64"
+
+            CIBW_BEFORE_TEST: |
+              ccache -s # Print current ccache stats
+
+            CIBW_TEST_COMMAND: |
+              # The wheels are installed automatically and available.
+              
+              # Fetch models from translateLocally repository.
+              python3 -m bergamot download -m en-de-tiny
+              python3 -m bergamot download -m de-en-tiny
+              python3 -m bergamot ls
+
+              # Fetch models from opus repository.
+              python3 -m bergamot download -m eng-fin-tiny -r opus
+              python3 -m bergamot ls -r opus
+
+              # Run the sample python script shipped with module
+              python3 -m bergamot translate --model en-de-tiny <<< "Hello World"
+              python3 -m bergamot translate --model en-de-tiny de-en-tiny <<< "Hello World"
+              python3 -m bergamot translate --model eng-fin-tiny --repository opus <<< "Hello World"
+
+
+        - uses: actions/upload-artifact@v2
+          with:
+            name: wheels
+            path: ./wheelhouse/*.whl
 
-    python-macos:
-      name: "MacOS 10.15 / py3.10"
-      runs-on: "macos-10.15"
+    upload-wheels:
+      name: "Upload wheels to PyPI"
+      runs-on: ubuntu-latest
+      if: ${{ startsWith(github.ref, 'refs/tags/v') }}
+      needs: [build-wheels]
       steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-        with:
-          submodules: recursive
-      - name: Install Dependencies
-        run: |-
-          brew update
-          brew install openblas protobuf ccache boost pybind11 
-          brew install coreutils findutils libarchive 
-
-      - name: Generate ccache_vars for ccache based on machine
-        shell: bash
-        id: ccache_vars
-        run: |-
-          echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
-          echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
-      - name: Cache-op for build-cache through ccache
-        uses: actions/cache@v2
+      - name: Download artifacts
+        uses: actions/download-artifact@v2
         with:
-          path: ${{ env.ccache_dir }}
-          key: ccache-${{ job.id }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-          restore-keys: |-
-            ccache-${{ job.id }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
-            ccache-${{ job.id }}-${{ steps.ccache_vars.outputs.hash }}
-            ccache-${{ job.id }}
-
-      - name: ccache environment setup
-        run: |-
-          echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
-          echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
-          echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
-          echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
-          echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
-          echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
-
-      - name: ccache prolog
-        run: |-
-          ccache -s # Print current cache stats
-          ccache -z # Zero cache entry
+          name: wheels
 
-      - name: Apply required patches
-        run: |-
-            patch -p1 < patches/01-marian-fstream-for-macos.patch
-
-      # Appears to be required per GitHub CI; 
-      - name: Set MACOSX DEPLOYMENT TARGET via environment variable
-        run: |-
-            echo "MACOSX_DEPLOYMENT_TARGET=10.15" >> $GITHUB_ENV
-
-      - name: Inject local version identifier for non tag builds
-        if: ${{ !startsWith(github.ref, 'refs/tags/v') }}
-        run: |-
-          echo "PYTHON_LOCAL_VERSION_IDENTIFIER=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
-
-      - name: setup.py 
-        run: |-
-          python3 -m pip install --upgrade packaging wheel
-          BUILD_ARCH=core-avx-i python3 setup.py bdist_wheel --universal
-
-      # We're happy with just compile for the moment, so cache gets some seeding.
-      - name: Install onto root python lib
-        run: |-
-          python3 -m pip install dist/bergamot-*.whl 
-
-      - name: Fetch models from translateLocally repository.
-        run: |-
-          python3 -m bergamot download -m en-de-tiny
-          python3 -m bergamot download -m de-en-tiny
-
-      - name: Fetch models from opus repository.
-        run: |-
-          python3 -m bergamot download -m eng-fin-tiny -r opus
-          python3 -m bergamot ls -r opus
-
-      - name: Run the sample python script shipped with module
-        run: |-
-          python3 -m bergamot translate --model en-de-tiny <<< "Hello World"
-          python3 -m bergamot translate --model en-de-tiny de-en-tiny <<< "Hello World"
-          python3 -m bergamot translate --model eng-fin-tiny --repository opus <<< "Hello World"
-
-      - name: ccache epilog
-        run: 'ccache -s # Print current cache stats'
+      - name: Publish wheels to PyPI
+        env:
+          TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+          TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+        run: |
+          python3 -m pip install twine
+          twine upload *.whl
 
-      - uses: actions/upload-artifact@v2
-        with:
-            path: ${{github.workspace}}/dist/bergamot-*.whl
 
     build-wasm:
       name: "emscripten"
@@ -380,7 +308,7 @@ jobs:
     release-latest:
       name: Release Latest Build
       runs-on: ubuntu-latest
-      needs: [python-ubuntu, python-macos, build-wasm]
+      needs: [build-wheels, build-wasm]
       if: github.ref == 'refs/heads/main'
       steps:
        - name: Download artifacts
@@ -394,16 +322,14 @@ jobs:
            prerelease: true
            title: "Latest Build"
            files: |
-                artifact/*.whl
-                wasm-artefacts/build-wasm-without-wormhole/bergamot-translator-worker.js
-                wasm-artefacts/build-wasm-without-wormhole/bergamot-translator-worker.wasm
-                wasm-artefacts/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.js
-                wasm-artefacts/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.wasm
+                wheels/*.whl
+                wasm-artefacts/bergamot-translator-worker.js
+                wasm-artefacts/bergamot-translator-worker.wasm
   
     release-version:
       name: Release version 
       runs-on: ubuntu-latest
-      needs: [python-ubuntu, python-macos, build-wasm]
+      needs: [build-wheels, build-wasm]
       permissions:
         contents: "write"
         packages: "write"
@@ -421,11 +347,9 @@ jobs:
            prerelease: false
            title: "${{ github.ref_name }}"
            files: |
-                artifact/*.whl
-                wasm-artefacts/build-wasm-without-wormhole/bergamot-translator-worker.js
-                wasm-artefacts/build-wasm-without-wormhole/bergamot-translator-worker.wasm
-                wasm-artefacts/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.js
-                wasm-artefacts/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.wasm
+                wheels/*.whl
+                wasm-artefacts/bergamot-translator-worker.js
+                wasm-artefacts/bergamot-translator-worker.wasm
 
   
     python-checks:
@@ -449,7 +373,7 @@ jobs:
 
     docs:
       runs-on: ubuntu-18.04
-      needs: [python-ubuntu]
+      needs: [build-wheels]
       steps:
         - name: Checkout
           uses: actions/checkout@v2
@@ -502,7 +426,7 @@ jobs:
           working-directory: ./doc
           run: |
             python3 -m pip install -r requirements.txt
-            python3 -m pip install ${{github.workspace}}/artifact/bergamot-*-cp37*.whl
+            python3 -m pip install ${{github.workspace}}/wheels/bergamot-*-cp37*.whl
 
         - name: Build documentation
           working-directory: ./doc
diff --git a/bindings/python/CMakeLists.txt b/bindings/python/CMakeLists.txt
index 70b1a2535..16e3e48d3 100644
--- a/bindings/python/CMakeLists.txt
+++ b/bindings/python/CMakeLists.txt
@@ -1,4 +1,4 @@
-find_package(Python COMPONENTS Interpreter Development REQUIRED)
+find_package(Python COMPONENTS Interpreter Development.Module REQUIRED)
 
 message("Using Python: " ${Python_EXECUTABLE})
 
diff --git a/bindings/python/README.md b/bindings/python/README.md
new file mode 100644
index 000000000..3797b7dea
--- /dev/null
+++ b/bindings/python/README.md
@@ -0,0 +1,14 @@
+# bergamot-translator
+
+The [Bergamot project](https://browser.mt/) adds and improves client-side
+machine translation in a web browser.
+
+This package provides Python bindings to bergamot-translator developed as part
+of the Bergamot Project and extras assorted in a package to enable further use
+of the library developed for local-translation on the consumer machine.
+
+Bergamot is a consortium coordinated by the University of Edinburgh with
+partners Charles University in Prague, the University of Sheffield, University
+of Tartu, and Mozilla.
+
+
diff --git a/setup.py b/setup.py
index 85fa685ff..ddba0c691 100644
--- a/setup.py
+++ b/setup.py
@@ -48,12 +48,11 @@ def build_extension(self, ext):
             f"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY={extdir}",
             f"-DPYTHON_EXECUTABLE={sys.executable}",
             f"-DCMAKE_BUILD_TYPE={cfg}",  # not used on MSVC, but no harm
-            f"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
-            f"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
             f"-DCOMPILE_PYTHON=ON",
             f"-DSSPLIT_USE_INTERNAL_PCRE2=ON",
             f"-DBUILD_ARCH={build_arch}",
         ]
+
         build_args = ["-t", "_bergamot"]
         # Adding CMake arguments set as environment variable
         # (needed e.g. to build for ARM OSx on conda-forge)
@@ -63,6 +62,13 @@ def build_extension(self, ext):
         # In this example, we pass in the version to C++. You might not need to.
         cmake_args += [f"-DEXAMPLE_VERSION_INFO={self.distribution.get_version()}"]
 
+        use_ccache = os.environ.get("USE_CCACHE", "0") == "1"
+        if use_ccache:
+            cmake_args += [
+                f"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
+                f"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
+            ]
+
         if self.compiler.compiler_type != "msvc":
             # Using Ninja-build since it a) is available as a wheel and b)
             # multithreads automatically. MSVC would require all variables be
@@ -130,14 +136,15 @@ def build_extension(self, ext):
 
 # Import the README and use it as the long-description.
 # Note: this will only work if 'README.md' is present in your MANIFEST.in file!
-with io.open(os.path.join(here, "README.md"), encoding="utf-8") as f:
+long_description = ""
+with io.open(os.path.join(here, "bindings/python/README.md"), encoding="utf-8") as f:
     long_description = "\n" + f.read()
 
 version = None
 with open(os.path.join(here, "BERGAMOT_VERSION")) as f:
     version = f.read().strip()
     suffix = os.environ.get("PYTHON_LOCAL_VERSION_IDENTIFIER", None)
-    if suffix is not None:
+    if suffix:
         version = "{}+{}".format(version, suffix)
 
 
@@ -191,8 +198,9 @@ def run(self):
     author="Jerin Philip",
     author_email="jerinphilip@live.in",
     url="https://github.com/browsermt/bergamot-translator/",
-    description="Bergamot translator python binding.",
-    long_description="",
+    description="Translate text-content locally in your machine across langauges.",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
     ext_modules=[CMakeExtension("bergamot/_bergamot")],
     cmdclass={"build_py": build_py, "build_ext": CMakeBuild},
     zip_safe=False,
@@ -207,4 +215,34 @@ def run(self):
             "bergamot = bergamot.__main__:main",
         ],
     },
+    # Classifiers help users find your project by categorizing it.
+    #
+    # For a list of valid classifiers, see https://pypi.org/classifiers/
+    classifiers=[  # Optional
+        # How mature is this project? Common values are
+        #   3 - Alpha
+        #   4 - Beta
+        #   5 - Production/Stable
+        "Development Status :: 3 - Alpha",
+        # Indicate who your project is intended for
+        "Intended Audience :: Developers",
+        "Topic :: Software Development :: Build Tools",
+        # Pick your license as you wish
+        "License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)",
+        # Specify the Python versions you support here. In particular, ensure
+        # that you indicate you support Python 3. These classifiers are *not*
+        # checked by 'pip install'. See instead 'python_requires' below.
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3 :: Only",
+    ],
+    project_urls={
+        "Bug Reports": "https://github.com/browsermt/bergamot-transator/issues",
+        "Source": "https://github.com/browsermt/bergamot-translator/",
+        "Documentation": "https://browser.mt/docs/main/python.html",
+    },
 )

From 8771078177f3758474a0d466f0faa58b4b64662c Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Tue, 21 Jun 2022 14:07:17 +0100
Subject: [PATCH 382/442] Basic HTML property testing for WebAssembly (#425)

Import
https://gist.github.com/jelmervdl/a4c8b6b92ad88a885e1cbd51c6ad4902 and
attach it to CI.  NodeJS-14 is failing on trying to use the WebAssembly
binary. So we use node-16 independently setup.  This paves way for more
complicated testing for WebAssembly bindings in the future.
---
 .github/workflows/build.yml |  75 +++++++++-------------
 wasm/node-test.js           | 125 ++++++++++++++++++++++++++++++++++++
 2 files changed, 156 insertions(+), 44 deletions(-)
 create mode 100644 wasm/node-test.js

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 2ae9b998e..2bf20cf26 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -232,16 +232,15 @@ jobs:
             ccache -s # Print current cache stats
             ccache -z # Zero cache entry
 
-        # WORMHOLE=off
-        - name: "Configure builds for WORMHOLE=off"
+        - name: "Configure builds"
           run: |
-            mkdir -p build-wasm-without-wormhole
-            cd build-wasm-without-wormhole
+            mkdir -p build-wasm
+            cd build-wasm
             emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=off ..
 
 
-        - name: "Compile with WORMHOLE=off"
-          working-directory: build-wasm-without-wormhole
+        - name: "Compile"
+          working-directory: build-wasm
           run: |
             emmake make -j2
 
@@ -250,43 +249,24 @@ jobs:
             ccache -s # Print current cache stats
 
         - name: Import GEMM library from a separate wasm module
-          working-directory: build-wasm-without-wormhole
+          working-directory: build-wasm
           run: bash ../wasm/patch-artifacts-import-gemm-module.sh
 
+        # Setup nodejs-16, as nodejs-14 provided by emsdk fails when running.
+        - name: Setup nodejs
+          uses: actions/setup-node@v3
+          with:
+            node-version: 16
 
-        # WORMHOLE=on
-        - name: "Configure builds for WORMHOLE=on"
-          run: |
-            mkdir -p build-wasm-with-wormhole
-            cd build-wasm-with-wormhole
-            emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=on ..
-
-
-        - name: "Compile with WORMHOLE=on"
-          working-directory: build-wasm-with-wormhole
-          run: |
-            emmake make -j2
-
-        - name: ccache epilog
-          run: |
-            ccache -s # Print current cache stats
-
-        - name: Instantiate simd wormhole
-          working-directory: build-wasm-with-wormhole
-          run: bash ../wasm/patch-artifacts-enable-wormhole.sh
-
-        - name: Import GEMM library from a separate wasm module
-          working-directory: build-wasm-with-wormhole
-          run: bash ../wasm/patch-artifacts-import-gemm-module.sh
-
-        # Rename the wormhole on builds
-        - name: Rename artefacts with wormhole
-          working-directory: build-wasm-with-wormhole
+        - name: Test run
+          working-directory: wasm
           run: |
-                mv bergamot-translator-worker{,-with-wormhole}.js
-                mv bergamot-translator-worker{,-with-wormhole}.js.bak
-                mv bergamot-translator-worker{,-with-wormhole}.wasm
+            cp ../build-wasm/bergamot-translator-worker.{js,wasm} ./
+            npm install jsdom
 
+            # --unhandled-rejections make the script exit with a non-zero code (at least on node-14).
+            # So leaving this here. 
+            node --unhandled-rejections=strict node-test.js
 
         # Upload both together.
         - name: Upload wasm artifact
@@ -296,13 +276,10 @@ jobs:
             if-no-files-found: error
             path: |
                 # Without wormhole
-                ${{github.workspace}}/build-wasm-without-wormhole/bergamot-translator-worker.js
-                ${{github.workspace}}/build-wasm-without-wormhole/bergamot-translator-worker.wasm
-                ${{github.workspace}}/build-wasm-without-wormhole/bergamot-translator-worker.js.bak
+                ${{github.workspace}}/build-wasm/bergamot-translator-worker.js
+                ${{github.workspace}}/build-wasm/bergamot-translator-worker.wasm
+                ${{github.workspace}}/build-wasm/bergamot-translator-worker.js.bak
 
-                ${{github.workspace}}/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.js
-                ${{github.workspace}}/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.wasm
-                ${{github.workspace}}/build-wasm-with-wormhole/bergamot-translator-worker-with-wormhole.js.bak
 
   # Try to upload a release using https://github.com/marvinpinto/actions/issues/177#issuecomment-917605585 as a model
     release-latest:
@@ -313,6 +290,11 @@ jobs:
       steps:
        - name: Download artifacts
          uses: actions/download-artifact@v2
+
+       # Leave the below be, it will be useful.
+       - name: List downloaded assets
+         run: |
+           find ./
         
        - name: Update GitHub prerelease
          uses: marvinpinto/action-automatic-releases@latest
@@ -338,6 +320,11 @@ jobs:
       steps:
        - name: Download artifacts
          uses: actions/download-artifact@v2
+
+       # Leave the below be, it will be useful.
+       - name: List downloaded assets
+         run: |
+           find ./
   
        - name: Update GitHub release
          uses: marvinpinto/action-automatic-releases@latest
diff --git a/wasm/node-test.js b/wasm/node-test.js
new file mode 100644
index 000000000..1f697afd3
--- /dev/null
+++ b/wasm/node-test.js
@@ -0,0 +1,125 @@
+const {Blob} = require('buffer');
+const fs = require('fs');
+const https = require('https');
+const {JSDOM} = require('jsdom');
+
+
+const wasmBinary = fs.readFileSync('./bergamot-translator-worker.wasm');
+global.Module = {
+  wasmBinary,
+  onRuntimeInitialized
+};
+
+// Execute bergamot-translation-worker.js in this scope
+const js = fs.readFileSync('./bergamot-translator-worker.js', {encoding: 'utf8'});
+eval.call(global, js);
+
+/**
+ * Helper to download file into ArrayBuffer.
+ */
+function download(url) {
+  return new Promise((accept, reject) => {
+    https.get(url, (res) => {
+      const chunks = [];
+      res.on('error', reject);
+      res.on('data', chunk => chunks.push(chunk));
+      res.on('end', async () => {
+        const data = new Blob(chunks);
+        data.arrayBuffer().then(accept, reject);
+      });
+    });
+  });
+}
+
+/**
+ * Loads ArrayBuffer into AlignedMemory.
+ */
+function load(buffer, alignment) {
+  const bytes = new Int8Array(buffer);
+  const memory = new Module.AlignedMemory(bytes.byteLength, alignment);
+  memory.getByteArrayView().set(bytes);
+  return memory;
+}
+
+/**
+ * Called from inside the worker.js script once the wasm module is loaded
+ * and all the emscripten magic and linking has been done.
+ */
+async function onRuntimeInitialized() {
+  // Root url for our models for now.
+  const root = 'https://storage.googleapis.com/bergamot-models-sandbox/0.2.14';
+
+  // In order of TranslationMemory's arguments
+  const files = [
+    {url: `${root}/ende/model.ende.intgemm.alphas.bin`, alignment: 256},
+    {url: `${root}/ende/lex.50.50.ende.s2t.bin`, alignment: 64},
+    {url: `${root}/ende/vocab.deen.spm`, alignment: 64},
+  ];
+
+  // Download model data and load it into aligned memory
+  const [modelMem, shortlistMem, vocabMem] = await Promise.all(files.map(async (file) => {
+    return load(await download(file.url), file.alignment);
+  }));
+
+  // Config yaml (split as array to allow for indentation without adding tabs
+  // or spaces to the strings themselves.)
+  const config = [
+    'beam-size: 1',
+    'normalize: 1.0',
+    'word-penalty: 0',
+    'alignment: soft',
+    'max-length-break: 128',
+    'mini-batch-words: 1024',
+    'workspace: 128',
+    'max-length-factor: 2.0',
+    'skip-cost: true',
+    'cpu-threads: 0',
+    'quiet: true',
+    'quiet-translation: true',
+    'gemm-precision: int8shiftAll',
+  ].join('\n');
+
+  // Set up translation service
+  const service = new Module.BlockingService({cacheSize: 0});
+
+  // Put vocab into its own std::vector<AlignedMemory>
+  const vocabs = new Module.AlignedMemoryList();
+  vocabs.push_back(vocabMem);
+
+  // Setup up model with config yaml and AlignedMemory objects
+  const model = new Module.TranslationModel(config, modelMem, shortlistMem, vocabs, /*qualityModel=*/ null);
+
+  // Construct std::vector<std::string> inputs;
+  const input = new Module.VectorString();
+  input.push_back('<p> Hello world! </p> <p> Goodbye World! </p>');
+
+  // Construct std::vector<ResponseOptions>
+  const options = new Module.VectorResponseOptions();
+  options.push_back({qualityScores: false, alignment: true, html: true});
+
+  // Translate our batch (of 1)
+  const output = service.translate(model, input, options);
+
+  // Get output from std::vector<Response>
+  // The following works as a simple black-box test of the API, based on
+  // properties of HTML.
+  const translation = output.get(0).getTranslatedText()
+
+  // Print raw translation for inspection.
+  console.log(translation)
+
+  const fragment = JSDOM.fragment(translation)
+
+  // Print two expected tags.
+  console.log(fragment.firstElementChild.outerHTML)
+  console.log(fragment.lastElementChild.outerHTML)
+
+  // Assertion that there are two children at the output.
+  assert(fragment.childElementCount === 2);
+
+
+  // Clean-up
+  input.delete();
+  options.delete();
+  output.delete();
+}

From 05a87784973b6e1cc591f1f1a9a05c5873d9971e Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Tue, 21 Jun 2022 17:49:07 +0100
Subject: [PATCH 383/442] Bump version to 0.4.5 (#427)

---
 BERGAMOT_VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BERGAMOT_VERSION b/BERGAMOT_VERSION
index 79b0815e6..a423f7f06 100644
--- a/BERGAMOT_VERSION
+++ b/BERGAMOT_VERSION
@@ -1 +1 @@
-v0.4.4
+v0.4.5

From 3ef85e12be9ad83afbf49a4978523a4da6dc24b1 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Fri, 24 Jun 2022 08:57:39 +0100
Subject: [PATCH 384/442] Python package: pyyaml >= 5.1 (#429)

Fixes issue on Colab which says vanilla YAML intall (3.x) does not have
yaml.FullLoader (https://stackoverflow.com/a/55553392/4565794).

Fix a broken link for presentation in PyPI.
---
 setup.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/setup.py b/setup.py
index ddba0c691..51161a3c0 100644
--- a/setup.py
+++ b/setup.py
@@ -209,7 +209,7 @@ def run(self):
     python_requires=">=3.6",
     packages=["bergamot"],
     package_dir={"bergamot": "bindings/python"},
-    install_requires=["requests", "pyyaml", "appdirs"],
+    install_requires=["requests", "pyyaml>=5.1", "appdirs"],
     entry_points={
         "console_scripts": [
             "bergamot = bergamot.__main__:main",
@@ -241,7 +241,7 @@ def run(self):
         "Programming Language :: Python :: 3 :: Only",
     ],
     project_urls={
-        "Bug Reports": "https://github.com/browsermt/bergamot-transator/issues",
+        "Bug Reports": "https://github.com/browsermt/bergamot-translator/issues",
         "Source": "https://github.com/browsermt/bergamot-translator/",
         "Documentation": "https://browser.mt/docs/main/python.html",
     },

From 84c761bacd56386d3e0f9eb79c0d8eca2499ae61 Mon Sep 17 00:00:00 2001
From: Jerin Philip <jerinphilip@live.in>
Date: Sat, 25 Jun 2022 18:32:50 +0100
Subject: [PATCH 385/442] Python: Work offline if models are available (#431)

Try to check if models.json is downloaded first, if it is use it.
If not, fall back to attempting to fetch it from the network.

Fixes: #430
---
 bindings/python/repository.py | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)

diff --git a/bindings/python/repository.py b/bindings/python/repository.py
index 7f89035c8..a8e4a79c1 100644
--- a/bindings/python/repository.py
+++ b/bindings/python/repository.py
@@ -75,22 +75,38 @@ def __init__(self, name, url):
             os.makedirs(directory, exist_ok=True)
 
         self.models_file_path = os.path.join(self.dirs["config"], "models.json")
-        self.update()
+        self.data = self._load_data(self.models_file_path)
+
+        # Update inverse lookup.
+        self.data_by_code = {}
+        for model in self.data["models"]:
+            self.data_by_code[model["code"]] = model
 
     @property
     def name(self) -> str:
         return self._name
 
+    def _load_data(self, models_file_path):
+        """
+        Load model data from existing file. If file does not exist, download from the web.
+        """
+        if os.path.exists(models_file_path):
+            # File already exists, prefer to work with this.
+            # A user is expected to update manually if model's already
+            # downloaded and setup.
+            with open(models_file_path) as model_file:
+                return json.load(model_file)
+        else:
+            # We are running for the first time.
+            # Try to fetch this file from the internet.
+            self.update()
+            with open(models_file_path) as model_file:
+                return json.load(model_file)
+
     def update(self) -> None:
         inventory = requests.get(self.url).text
         with open(self.models_file_path, "w+") as models_file:
             models_file.write(inventory)
-        self.data = json.loads(inventory)
-
-        # Update inverse lookup.
-        self.data_by_code = {}
-        for model in self.data["models"]:
-            self.data_by_code[model["code"]] = model
 
     def models(self, filter_downloaded: bool = True) -> t.List[str]:
         codes = []

From 7f791289007df45987165cea3cdee28073427ba2 Mon Sep 17 00:00:00 2001
From: Graeme Nail <graemenail@gmail.com>
Date: Wed, 29 Jun 2022 22:46:24 +0100
Subject: [PATCH 386/442] MacOS Wheels (#432)

* Remove trailing whitespace
* Additional MacOS wheels: Wheels for python 3.6 to 3.10 with a
   minimum target of MacOS 10.9
* Install bergamot package from wheel directory
* Remove no-index as we need dependencies
---
 .github/workflows/build.yml | 37 ++++++++++++++++++-------------------
 1 file changed, 18 insertions(+), 19 deletions(-)

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 2bf20cf26..25305a177 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -91,12 +91,12 @@ jobs:
               PYTHON_LOCAL_VERSION_IDENTIFIER=${{ env.PYTHON_LOCAL_VERSION_IDENTIFIER }}
               CCACHE_DIR=${{ env.ccache_dir }}
               CCACHE_BASEDIR=${{ env.ccache_basedir }}
-              MACOSX_DEPLOYMENT_TARGET=10.15
+              MACOSX_DEPLOYMENT_TARGET=10.9
 
             CIBW_BEFORE_BUILD_LINUX: |
               yum install -y ccache
 
-              # Install Intel MKL. 
+              # Install Intel MKL.
               yum-config-manager -y --add-repo https://yum.repos.intel.com/mkl/setup/intel-mkl.repo
               yum install -y intel-mkl
 
@@ -106,19 +106,19 @@ jobs:
               ccache -z # Zero cache entry
 
             CIBW_BEFORE_BUILD_MACOS: |
-              brew install openblas protobuf ccache boost pybind11 
+              brew install openblas protobuf ccache boost pybind11
               chmod -R a+rwx ${{ env.ccache_dir }}
               ccache -s # Print current cache stats
               ccache -z # Zero cache entry
 
-            CIBW_BUILD: "cp{36,37,38,39,310}-*manylinux_x86_64 cp310-macosx_x86_64"
+            CIBW_BUILD: "cp{36,37,38,39,310}-*manylinux_x86_64 cp{36,37,38,39,310}-macosx_x86_64"
 
             CIBW_BEFORE_TEST: |
               ccache -s # Print current ccache stats
 
             CIBW_TEST_COMMAND: |
               # The wheels are installed automatically and available.
-              
+
               # Fetch models from translateLocally repository.
               python3 -m bergamot download -m en-de-tiny
               python3 -m bergamot download -m de-en-tiny
@@ -181,7 +181,7 @@ jobs:
             echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
             # https://emscripten.org/docs/compiling/Building-Projects.html#using-a-compiler-wrapper
             echo "EM_COMPILER_WRAPPER=ccache" >> $GITHUB_ENV
-            
+
         # This need to be run before setup, so ccache build caching doesn't complain.
         - name: Obtain emsdk sources
           run: |
@@ -265,7 +265,7 @@ jobs:
             npm install jsdom
 
             # --unhandled-rejections make the script exit with a non-zero code (at least on node-14).
-            # So leaving this here. 
+            # So leaving this here.
             node --unhandled-rejections=strict node-test.js
 
         # Upload both together.
@@ -295,7 +295,7 @@ jobs:
        - name: List downloaded assets
          run: |
            find ./
-        
+
        - name: Update GitHub prerelease
          uses: marvinpinto/action-automatic-releases@latest
          with:
@@ -307,9 +307,9 @@ jobs:
                 wheels/*.whl
                 wasm-artefacts/bergamot-translator-worker.js
                 wasm-artefacts/bergamot-translator-worker.wasm
-  
+
     release-version:
-      name: Release version 
+      name: Release version
       runs-on: ubuntu-latest
       needs: [build-wheels, build-wasm]
       permissions:
@@ -325,7 +325,7 @@ jobs:
        - name: List downloaded assets
          run: |
            find ./
-  
+
        - name: Update GitHub release
          uses: marvinpinto/action-automatic-releases@latest
          with:
@@ -338,7 +338,7 @@ jobs:
                 wasm-artefacts/bergamot-translator-worker.js
                 wasm-artefacts/bergamot-translator-worker.wasm
 
-  
+
     python-checks:
       name: "formatting and typechecks"
       runs-on: "ubuntu-latest"
@@ -389,8 +389,8 @@ jobs:
         # Patches the BERGAMOT_VERSION file used by sphinx-docs at run time to
         # obtain names like 'main' or 'ci-sandbox' to not confuse with version
         # based documentation built separately.
-        - name: Deploy-time patch version 
-          run: | 
+        - name: Deploy-time patch version
+          run: |
               echo ${{steps.tag.outputs.result }} > BERGAMOT_VERSION
 
         - name: Set up Doxygen
@@ -413,7 +413,7 @@ jobs:
           working-directory: ./doc
           run: |
             python3 -m pip install -r requirements.txt
-            python3 -m pip install ${{github.workspace}}/wheels/bergamot-*-cp37*.whl
+            python3 -m pip install --find-links=${{github.workspace}}/wheels bergamot
 
         - name: Build documentation
           working-directory: ./doc
@@ -424,16 +424,16 @@ jobs:
           uses: JamesIves/github-pages-deploy-action@4.1.3
           if: ${{ github.event_name == 'push' && github.repository == 'browsermt/bergamot-translator' }}
           with:
-            repository-name: 'browsermt/docs' 
+            repository-name: 'browsermt/docs'
             branch: gh-pages # The branch the action should deploy to.
             folder: './doc/build/' # The folder the action should deploy.
-            target-folder: '${{ steps.tag.outputs.result }}' 
+            target-folder: '${{ steps.tag.outputs.result }}'
             ssh-key: ${{ secrets.BERGAMOT_SSH_PRIVATE_KEY }}
 
         # This artifact contains the HTML output of Sphinx only.
         # With index.html at the root of the produced zip file.
         # For use for maintainers to download the zip and check render of
-        # documentation while generated at pull-request. 
+        # documentation while generated at pull-request.
         - name: Upload documentation
           uses: actions/upload-artifact@v2
           if: ${{ github.event_name == 'pull_request'}}
@@ -441,4 +441,3 @@ jobs:
             name: api-docs
             path: ./doc/build/
             if-no-files-found: error
-

From 06c31af0fe1c52af68eced08044ae03888e55fa2 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Tue, 17 Jan 2023 16:43:19 +0000
Subject: [PATCH 387/442] update download path

---
 .github/workflows/windows.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml
index 569020452..a0ff86b84 100644
--- a/.github/workflows/windows.yml
+++ b/.github/workflows/windows.yml
@@ -7,7 +7,7 @@ on:
     branches: [ '**' ]
 
 env:
-  MKL_URL: "https://romang.blob.core.windows.net/mariandev/ci/mkl-2020.1-windows-static.zip"
+  MKL_URL: "https://data.statmt.org/romang/marian-regression-tests/ci/mkl-2020.1-windows-static.zip"
   CCACHE_BASEDIR: "${{ github.workspace }}"
   CCACHE_DIR: "${{ github.workspace }}\\ccache"
   CCACHE_COMPILERCHECK: content

From 21eff44513168789d41e71ded7187cdfeb6434c5 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Tue, 17 Jan 2023 16:53:38 +0000
Subject: [PATCH 388/442] try to update coding_styles workflow

---
 .github/workflows/coding-styles.yml | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/.github/workflows/coding-styles.yml b/.github/workflows/coding-styles.yml
index 81bdf3361..e666c51df 100644
--- a/.github/workflows/coding-styles.yml
+++ b/.github/workflows/coding-styles.yml
@@ -1,7 +1,5 @@
 name: "Coding Style"
 
-env:
-    clang_version: 10
 on: 
   push:
     branches: [ main, ci-sandbox ]
@@ -22,7 +20,7 @@ jobs:
           run: |
             sudo apt-get update 
             sudo apt-get install -y build-essential cmake
-            sudo apt-get install -y clang-format clang-tidy-${{ env.clang_version }}
+            sudo apt-get install -y clang-format clang-tidy
 
         - name: Run clang-format
           run:
@@ -35,10 +33,10 @@ jobs:
               cd build 
               cmake \
                 -DUSE_WASM_COMPATIBLE_SOURCE=off -DCMAKE_EXPORT_COMPILE_COMMANDS=on \
-                -DCMAKE_C_COMPILER=clang-${{ env.clang_version }} -DCMAKE_CXX_COMPILER=clang++-${{ env.clang_version }} \
+                -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
                 ..
 
         - name: Run clang-tidy
           run: |
-              run-clang-tidy-${{ env.clang_version }} -p build "$PWD/src/.*"
-              run-clang-tidy-${{ env.clang_version }} -p build "$PWD/app/.*"
+              run-clang-tidy -p build "$PWD/src/.*"
+              run-clang-tidy -p build "$PWD/app/.*"

From 6cefc4302db521eb259f303eec4058786bf3812d Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Wed, 18 Jan 2023 12:48:53 +0000
Subject: [PATCH 389/442] Latest and greatest clang-format

---
 3rd_party/marian-dev                        | 2 +-
 bindings/python/bergamot.cpp                | 4 ++--
 src/translator/threadsafe_batching_pool.cpp | 6 +++---
 src/translator/threadsafe_batching_pool.h   | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index e88c1aa5d..4b30c267c 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit e88c1aa5d5c5622cb52c7df09fbb7c3d7f4b5b5a
+Subproject commit 4b30c267c701198cef4cddcd646cca17ccbb16f5
diff --git a/bindings/python/bergamot.cpp b/bindings/python/bergamot.cpp
index 5e9e830f9..2ffb2267e 100644
--- a/bindings/python/bergamot.cpp
+++ b/bindings/python/bergamot.cpp
@@ -116,7 +116,7 @@ class ServicePyAdapter {
     return responses;
   }
 
- private /*functions*/:
+  private /*functions*/:
   static Service make_service(const Service::Config &config) {
     py::scoped_ostream_redirect outstream(std::cout,                                 // std::ostream&
                                           py::module_::import("sys").attr("stdout")  // Python output
@@ -130,7 +130,7 @@ class ServicePyAdapter {
     return Service(config);
   }
 
- private /*data*/:
+  private /*data*/:
   Service service_;
 };
 
diff --git a/src/translator/threadsafe_batching_pool.cpp b/src/translator/threadsafe_batching_pool.cpp
index 0a1a28d4e..29ad35a97 100644
--- a/src/translator/threadsafe_batching_pool.cpp
+++ b/src/translator/threadsafe_batching_pool.cpp
@@ -10,7 +10,7 @@ namespace bergamot {
 
 template <class BatchingPoolType>
 template <class... Args>
-ThreadsafeBatchingPool<BatchingPoolType>::ThreadsafeBatchingPool(Args &&... args)
+ThreadsafeBatchingPool<BatchingPoolType>::ThreadsafeBatchingPool(Args &&...args)
     : backend_(std::forward<Args>(args)...), enqueued_(0), shutdown_(false) {}
 
 template <class BatchingPoolType>
@@ -20,7 +20,7 @@ ThreadsafeBatchingPool<BatchingPoolType>::~ThreadsafeBatchingPool() {
 
 template <class BatchingPoolType>
 template <class... Args>
-void ThreadsafeBatchingPool<BatchingPoolType>::enqueueRequest(Args &&... args) {
+void ThreadsafeBatchingPool<BatchingPoolType>::enqueueRequest(Args &&...args) {
   std::unique_lock<std::mutex> lock(mutex_);
   assert(!shutdown_);
   enqueued_ += backend_.enqueueRequest(std::forward<Args>(args)...);
@@ -43,7 +43,7 @@ void ThreadsafeBatchingPool<BatchingPoolType>::shutdown() {
 
 template <class BatchingPoolType>
 template <class... Args>
-size_t ThreadsafeBatchingPool<BatchingPoolType>::generateBatch(Args &&... args) {
+size_t ThreadsafeBatchingPool<BatchingPoolType>::generateBatch(Args &&...args) {
   std::unique_lock<std::mutex> lock(mutex_);
   work_.wait(lock, [this]() { return enqueued_ || shutdown_; });
   size_t sentencesInBatch = backend_.generateBatch(std::forward<Args>(args)...);
diff --git a/src/translator/threadsafe_batching_pool.h b/src/translator/threadsafe_batching_pool.h
index fdbf36cdc..9f46abb94 100644
--- a/src/translator/threadsafe_batching_pool.h
+++ b/src/translator/threadsafe_batching_pool.h
@@ -34,14 +34,14 @@ template <class BatchingPoolType>
 class ThreadsafeBatchingPool {
  public:
   template <class... Args>
-  ThreadsafeBatchingPool(Args &&... args);
+  ThreadsafeBatchingPool(Args &&...args);
   ~ThreadsafeBatchingPool();
 
   template <class... Args>
-  void enqueueRequest(Args &&... args);
+  void enqueueRequest(Args &&...args);
 
   template <class... Args>
-  size_t generateBatch(Args &&... args);
+  size_t generateBatch(Args &&...args);
 
   // Removes any pending requests from the batching pool.
   void clear();

From 620c8b00ecfd34f33a384997ea1b694e68540524 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Wed, 18 Jan 2023 15:30:58 +0000
Subject: [PATCH 390/442] Bump qs and express in /wasm/test_page (#444)

Bumps [qs](https://github.com/ljharb/qs) to 6.11.0 and updates ancestor dependency [express](https://github.com/expressjs/express). These dependencies need to be updated together.


Updates `qs` from 6.7.0 to 6.11.0
- [Release notes](https://github.com/ljharb/qs/releases)
- [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/qs/compare/v6.7.0...v6.11.0)

Updates `express` from 4.17.1 to 4.18.2
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/4.17.1...4.18.2)

---
updated-dependencies:
- dependency-name: qs
  dependency-type: indirect
- dependency-name: express
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 wasm/test_page/package-lock.json | 681 +++++++++++++++++++------------
 wasm/test_page/package.json      |   2 +-
 2 files changed, 421 insertions(+), 262 deletions(-)

diff --git a/wasm/test_page/package-lock.json b/wasm/test_page/package-lock.json
index 065c92de8..5ead514d8 100644
--- a/wasm/test_page/package-lock.json
+++ b/wasm/test_page/package-lock.json
@@ -6,17 +6,17 @@
     "": {
       "dependencies": {
         "cors": "^2.8.5",
-        "express": "^4.17.1",
+        "express": "^4.18.2",
         "nocache": "^2.1.0"
       }
     },
     "node_modules/accepts": {
-      "version": "1.3.7",
-      "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.7.tgz",
-      "integrity": "sha512-Il80Qs2WjYlJIBNzNkK6KYqlVMTbZLXgHx2oT0pU/fjRHyEp+PEfEPY0R3WCwAGVOtauxh1hOxNgIf5bv7dQpA==",
+      "version": "1.3.8",
+      "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
+      "integrity": "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw==",
       "dependencies": {
-        "mime-types": "~2.1.24",
-        "negotiator": "0.6.2"
+        "mime-types": "~2.1.34",
+        "negotiator": "0.6.3"
       },
       "engines": {
         "node": ">= 0.6"
@@ -28,39 +28,54 @@
       "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI="
     },
     "node_modules/body-parser": {
-      "version": "1.19.0",
-      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.19.0.tgz",
-      "integrity": "sha512-dhEPs72UPbDnAQJ9ZKMNTP6ptJaionhP5cBb541nXPlW60Jepo9RV/a4fX4XWW9CuFNK22krhrj1+rgzifNCsw==",
+      "version": "1.20.1",
+      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.1.tgz",
+      "integrity": "sha512-jWi7abTbYwajOytWCQc37VulmWiRae5RyTpaCyDcS5/lMdtwSz5lOpDE67srw/HYe35f1z3fDQw+3txg7gNtWw==",
       "dependencies": {
-        "bytes": "3.1.0",
+        "bytes": "3.1.2",
         "content-type": "~1.0.4",
         "debug": "2.6.9",
-        "depd": "~1.1.2",
-        "http-errors": "1.7.2",
+        "depd": "2.0.0",
+        "destroy": "1.2.0",
+        "http-errors": "2.0.0",
         "iconv-lite": "0.4.24",
-        "on-finished": "~2.3.0",
-        "qs": "6.7.0",
-        "raw-body": "2.4.0",
-        "type-is": "~1.6.17"
+        "on-finished": "2.4.1",
+        "qs": "6.11.0",
+        "raw-body": "2.5.1",
+        "type-is": "~1.6.18",
+        "unpipe": "1.0.0"
       },
       "engines": {
-        "node": ">= 0.8"
+        "node": ">= 0.8",
+        "npm": "1.2.8000 || >= 1.4.16"
       }
     },
     "node_modules/bytes": {
-      "version": "3.1.0",
-      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.0.tgz",
-      "integrity": "sha512-zauLjrfCG+xvoyaqLoV8bLVXXNGC4JqlxFCutSDWA6fJrTo2ZuvLYTqZ7aHBLZSMOopbzwv8f+wZcVzfVTI2Dg==",
+      "version": "3.1.2",
+      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
+      "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
       "engines": {
         "node": ">= 0.8"
       }
     },
+    "node_modules/call-bind": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/call-bind/-/call-bind-1.0.2.tgz",
+      "integrity": "sha512-7O+FbCihrB5WGbFYesctwmTKae6rOiIzmz1icreWJ+0aA7LJfuqhEso2T9ncpcFtzMQtzXf2QGGueWJGTYsqrA==",
+      "dependencies": {
+        "function-bind": "^1.1.1",
+        "get-intrinsic": "^1.0.2"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/ljharb"
+      }
+    },
     "node_modules/content-disposition": {
-      "version": "0.5.3",
-      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.3.tgz",
-      "integrity": "sha512-ExO0774ikEObIAEV9kDo50o+79VCUdEB6n6lzKgGwupcVeRlhrj3qGAfwq8G6uBJjkqLrhT0qEYFcWng8z1z0g==",
+      "version": "0.5.4",
+      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz",
+      "integrity": "sha512-FveZTNuGw04cxlAiWbzi6zTAL/lhehaWbTtgluJh4/E95DqMwTmha3KZN1aAWA8cFIhHzMZUvLevkw5Rqk+tSQ==",
       "dependencies": {
-        "safe-buffer": "5.1.2"
+        "safe-buffer": "5.2.1"
       },
       "engines": {
         "node": ">= 0.6"
@@ -75,9 +90,9 @@
       }
     },
     "node_modules/cookie": {
-      "version": "0.4.0",
-      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.4.0.tgz",
-      "integrity": "sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg==",
+      "version": "0.5.0",
+      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.5.0.tgz",
+      "integrity": "sha512-YZ3GUyn/o8gfKJlnlX7g7xq4gyO6OSuhGPKaaGssGB2qgDUS0gPgtTvoyZLTt9Ab6dC4hfc9dV5arkvc/OCmrw==",
       "engines": {
         "node": ">= 0.6"
       }
@@ -108,27 +123,31 @@
       }
     },
     "node_modules/depd": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/depd/-/depd-1.1.2.tgz",
-      "integrity": "sha1-m81S4UwJd2PnSbJ0xDRu0uVgtak=",
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
+      "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==",
       "engines": {
-        "node": ">= 0.6"
+        "node": ">= 0.8"
       }
     },
     "node_modules/destroy": {
-      "version": "1.0.4",
-      "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.0.4.tgz",
-      "integrity": "sha1-l4hXRCxEdJ5CBmE+N5RiBYJqvYA="
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.2.0.tgz",
+      "integrity": "sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg==",
+      "engines": {
+        "node": ">= 0.8",
+        "npm": "1.2.8000 || >= 1.4.16"
+      }
     },
     "node_modules/ee-first": {
       "version": "1.1.1",
       "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
-      "integrity": "sha1-WQxhFWsK4vTwJVcyoViyZrxWsh0="
+      "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="
     },
     "node_modules/encodeurl": {
       "version": "1.0.2",
       "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
-      "integrity": "sha1-rT/0yG7C0CkyL1oCw6mmBslbP1k=",
+      "integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
       "engines": {
         "node": ">= 0.8"
       }
@@ -136,48 +155,49 @@
     "node_modules/escape-html": {
       "version": "1.0.3",
       "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
-      "integrity": "sha1-Aljq5NPQwJdN4cFpGI7wBR0dGYg="
+      "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow=="
     },
     "node_modules/etag": {
       "version": "1.8.1",
       "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
-      "integrity": "sha1-Qa4u62XvpiJorr/qg6x9eSmbCIc=",
+      "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==",
       "engines": {
         "node": ">= 0.6"
       }
     },
     "node_modules/express": {
-      "version": "4.17.1",
-      "resolved": "https://registry.npmjs.org/express/-/express-4.17.1.tgz",
-      "integrity": "sha512-mHJ9O79RqluphRrcw2X/GTh3k9tVv8YcoyY4Kkh4WDMUYKRZUq0h1o0w2rrrxBqM7VoeUVqgb27xlEMXTnYt4g==",
+      "version": "4.18.2",
+      "resolved": "https://registry.npmjs.org/express/-/express-4.18.2.tgz",
+      "integrity": "sha512-5/PsL6iGPdfQ/lKM1UuielYgv3BUoJfz1aUwU9vHZ+J7gyvwdQXFEBIEIaxeGf0GIcreATNyBExtalisDbuMqQ==",
       "dependencies": {
-        "accepts": "~1.3.7",
+        "accepts": "~1.3.8",
         "array-flatten": "1.1.1",
-        "body-parser": "1.19.0",
-        "content-disposition": "0.5.3",
+        "body-parser": "1.20.1",
+        "content-disposition": "0.5.4",
         "content-type": "~1.0.4",
-        "cookie": "0.4.0",
+        "cookie": "0.5.0",
         "cookie-signature": "1.0.6",
         "debug": "2.6.9",
-        "depd": "~1.1.2",
+        "depd": "2.0.0",
         "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
         "etag": "~1.8.1",
-        "finalhandler": "~1.1.2",
+        "finalhandler": "1.2.0",
         "fresh": "0.5.2",
+        "http-errors": "2.0.0",
         "merge-descriptors": "1.0.1",
         "methods": "~1.1.2",
-        "on-finished": "~2.3.0",
+        "on-finished": "2.4.1",
         "parseurl": "~1.3.3",
         "path-to-regexp": "0.1.7",
-        "proxy-addr": "~2.0.5",
-        "qs": "6.7.0",
+        "proxy-addr": "~2.0.7",
+        "qs": "6.11.0",
         "range-parser": "~1.2.1",
-        "safe-buffer": "5.1.2",
-        "send": "0.17.1",
-        "serve-static": "1.14.1",
-        "setprototypeof": "1.1.1",
-        "statuses": "~1.5.0",
+        "safe-buffer": "5.2.1",
+        "send": "0.18.0",
+        "serve-static": "1.15.0",
+        "setprototypeof": "1.2.0",
+        "statuses": "2.0.1",
         "type-is": "~1.6.18",
         "utils-merge": "1.0.1",
         "vary": "~1.1.2"
@@ -187,16 +207,16 @@
       }
     },
     "node_modules/finalhandler": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.2.tgz",
-      "integrity": "sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA==",
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.2.0.tgz",
+      "integrity": "sha512-5uXcUVftlQMFnWC9qu/svkWv3GTd2PfUhK/3PLkYNAe7FbqJMt3515HaxE6eRL74GdsriiwujiawdaB1BpEISg==",
       "dependencies": {
         "debug": "2.6.9",
         "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
-        "on-finished": "~2.3.0",
+        "on-finished": "2.4.1",
         "parseurl": "~1.3.3",
-        "statuses": "~1.5.0",
+        "statuses": "2.0.1",
         "unpipe": "~1.0.0"
       },
       "engines": {
@@ -204,9 +224,9 @@
       }
     },
     "node_modules/forwarded": {
-      "version": "0.1.2",
-      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.1.2.tgz",
-      "integrity": "sha1-mMI9qxF1ZXuMBXPozszZGw/xjIQ=",
+      "version": "0.2.0",
+      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
+      "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==",
       "engines": {
         "node": ">= 0.6"
       }
@@ -214,24 +234,64 @@
     "node_modules/fresh": {
       "version": "0.5.2",
       "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz",
-      "integrity": "sha1-PYyt2Q2XZWn6g1qx+OSyOhBWBac=",
+      "integrity": "sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q==",
       "engines": {
         "node": ">= 0.6"
       }
     },
+    "node_modules/function-bind": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.1.tgz",
+      "integrity": "sha512-yIovAzMX49sF8Yl58fSCWJ5svSLuaibPxXQJFLmBObTuCr0Mf1KiPopGM9NiFjiYBCbfaa2Fh6breQ6ANVTI0A=="
+    },
+    "node_modules/get-intrinsic": {
+      "version": "1.1.3",
+      "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.1.3.tgz",
+      "integrity": "sha512-QJVz1Tj7MS099PevUG5jvnt9tSkXN8K14dxQlikJuPt4uD9hHAHjLyLBiLR5zELelBdD9QNRAXZzsJx0WaDL9A==",
+      "dependencies": {
+        "function-bind": "^1.1.1",
+        "has": "^1.0.3",
+        "has-symbols": "^1.0.3"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/ljharb"
+      }
+    },
+    "node_modules/has": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/has/-/has-1.0.3.tgz",
+      "integrity": "sha512-f2dvO0VU6Oej7RkWJGrehjbzMAjFp5/VKPp5tTpWIV4JHHZK1/BxbFRtf/siA2SWTe09caDmVtYYzWEIbBS4zw==",
+      "dependencies": {
+        "function-bind": "^1.1.1"
+      },
+      "engines": {
+        "node": ">= 0.4.0"
+      }
+    },
+    "node_modules/has-symbols": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.0.3.tgz",
+      "integrity": "sha512-l3LCuF6MgDNwTDKkdYGEihYjt5pRPbEg46rtlmnSPlUbgmB8LOIrKJbYYFBSbnPaJexMKtiPO8hmeRjRz2Td+A==",
+      "engines": {
+        "node": ">= 0.4"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/ljharb"
+      }
+    },
     "node_modules/http-errors": {
-      "version": "1.7.2",
-      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.7.2.tgz",
-      "integrity": "sha512-uUQBt3H/cSIVfch6i1EuPNy/YsRSOUBXTVfZ+yR7Zjez3qjBz6i9+i4zjNaoqcoFVI4lQJ5plg63TvGfRSDCRg==",
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.0.tgz",
+      "integrity": "sha512-FtwrG/euBzaEjYeRqOgly7G0qviiXoJWnvEH2Z1plBdXgbyjv34pHTSb9zoeHMyDy33+DWy5Wt9Wo+TURtOYSQ==",
       "dependencies": {
-        "depd": "~1.1.2",
-        "inherits": "2.0.3",
-        "setprototypeof": "1.1.1",
-        "statuses": ">= 1.5.0 < 2",
-        "toidentifier": "1.0.0"
+        "depd": "2.0.0",
+        "inherits": "2.0.4",
+        "setprototypeof": "1.2.0",
+        "statuses": "2.0.1",
+        "toidentifier": "1.0.1"
       },
       "engines": {
-        "node": ">= 0.6"
+        "node": ">= 0.8"
       }
     },
     "node_modules/iconv-lite": {
@@ -246,9 +306,9 @@
       }
     },
     "node_modules/inherits": {
-      "version": "2.0.3",
-      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz",
-      "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4="
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
+      "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
     },
     "node_modules/ipaddr.js": {
       "version": "1.9.1",
@@ -261,7 +321,7 @@
     "node_modules/media-typer": {
       "version": "0.3.0",
       "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
-      "integrity": "sha1-hxDXrwqmJvj/+hzgAWhUUmMlV0g=",
+      "integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ==",
       "engines": {
         "node": ">= 0.6"
       }
@@ -291,19 +351,19 @@
       }
     },
     "node_modules/mime-db": {
-      "version": "1.45.0",
-      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.45.0.tgz",
-      "integrity": "sha512-CkqLUxUk15hofLoLyljJSrukZi8mAtgd+yE5uO4tqRZsdsAJKv0O+rFMhVDRJgozy+yG6md5KwuXhD4ocIoP+w==",
+      "version": "1.52.0",
+      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
+      "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
       "engines": {
         "node": ">= 0.6"
       }
     },
     "node_modules/mime-types": {
-      "version": "2.1.28",
-      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.28.tgz",
-      "integrity": "sha512-0TO2yJ5YHYr7M2zzT7gDU1tbwHxEUWBCLt0lscSNpcdAfFyJOVEpRYNS7EXVcTLNj/25QO8gulHC5JtTzSE2UQ==",
+      "version": "2.1.35",
+      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
+      "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
       "dependencies": {
-        "mime-db": "1.45.0"
+        "mime-db": "1.52.0"
       },
       "engines": {
         "node": ">= 0.6"
@@ -312,12 +372,12 @@
     "node_modules/ms": {
       "version": "2.0.0",
       "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
-      "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g="
+      "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="
     },
     "node_modules/negotiator": {
-      "version": "0.6.2",
-      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.2.tgz",
-      "integrity": "sha512-hZXc7K2e+PgeI1eDBe/10Ard4ekbfrrqG8Ep+8Jmf4JID2bNg7NvCPOZN+kfF574pFQI7mum2AUqDidoKqcTOw==",
+      "version": "0.6.3",
+      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz",
+      "integrity": "sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg==",
       "engines": {
         "node": ">= 0.6"
       }
@@ -338,10 +398,18 @@
         "node": ">=0.10.0"
       }
     },
+    "node_modules/object-inspect": {
+      "version": "1.12.3",
+      "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.12.3.tgz",
+      "integrity": "sha512-geUvdk7c+eizMNUDkRpW1wJwgfOiOeHbxBR/hLXK1aT6zmVSO0jsQcs7fj6MGw89jC/cjGfLcNOrtMYtGqm81g==",
+      "funding": {
+        "url": "https://github.com/sponsors/ljharb"
+      }
+    },
     "node_modules/on-finished": {
-      "version": "2.3.0",
-      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.3.0.tgz",
-      "integrity": "sha1-IPEzZIGwg811M3mSoWlxqi2QaUc=",
+      "version": "2.4.1",
+      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz",
+      "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==",
       "dependencies": {
         "ee-first": "1.1.1"
       },
@@ -363,11 +431,11 @@
       "integrity": "sha1-32BBeABfUi8V60SQ5yR6G/qmf4w="
     },
     "node_modules/proxy-addr": {
-      "version": "2.0.6",
-      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.6.tgz",
-      "integrity": "sha512-dh/frvCBVmSsDYzw6n926jv974gddhkFPfiN8hPOi30Wax25QZyZEGveluCgliBnqmuM+UJmBErbAUFIoDbjOw==",
+      "version": "2.0.7",
+      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
+      "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==",
       "dependencies": {
-        "forwarded": "~0.1.2",
+        "forwarded": "0.2.0",
         "ipaddr.js": "1.9.1"
       },
       "engines": {
@@ -375,11 +443,17 @@
       }
     },
     "node_modules/qs": {
-      "version": "6.7.0",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.7.0.tgz",
-      "integrity": "sha512-VCdBRNFTX1fyE7Nb6FYoURo/SPe62QCaAyzJvUjwRaIsc+NePBEniHlvxFmmX56+HZphIGtV0XeCirBtpDrTyQ==",
+      "version": "6.11.0",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.11.0.tgz",
+      "integrity": "sha512-MvjoMCJwEarSbUYk5O+nmoSzSutSsTwF85zcHPQ9OrlFoZOYIjaqBAJIqIXjptyD5vThxGq52Xu/MaJzRkIk4Q==",
+      "dependencies": {
+        "side-channel": "^1.0.4"
+      },
       "engines": {
         "node": ">=0.6"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/ljharb"
       }
     },
     "node_modules/range-parser": {
@@ -391,12 +465,12 @@
       }
     },
     "node_modules/raw-body": {
-      "version": "2.4.0",
-      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.4.0.tgz",
-      "integrity": "sha512-4Oz8DUIwdvoa5qMJelxipzi/iJIi40O5cGV1wNYp5hvZP8ZN0T+jiNkL0QepXs+EsQ9XJ8ipEDoiH70ySUJP3Q==",
+      "version": "2.5.1",
+      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.1.tgz",
+      "integrity": "sha512-qqJBtEyVgS0ZmPGdCFPWJ3FreoqvG4MVQln/kCgF7Olq95IbOp0/BWyMwbdtn4VTvkM8Y7khCQ2Xgk/tcrCXig==",
       "dependencies": {
-        "bytes": "3.1.0",
-        "http-errors": "1.7.2",
+        "bytes": "3.1.2",
+        "http-errors": "2.0.0",
         "iconv-lite": "0.4.24",
         "unpipe": "1.0.0"
       },
@@ -405,9 +479,23 @@
       }
     },
     "node_modules/safe-buffer": {
-      "version": "5.1.2",
-      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
-      "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="
+      "version": "5.2.1",
+      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
+      "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",
+      "funding": [
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/feross"
+        },
+        {
+          "type": "patreon",
+          "url": "https://www.patreon.com/feross"
+        },
+        {
+          "type": "consulting",
+          "url": "https://feross.org/support"
+        }
+      ]
     },
     "node_modules/safer-buffer": {
       "version": "2.1.2",
@@ -415,64 +503,77 @@
       "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
     },
     "node_modules/send": {
-      "version": "0.17.1",
-      "resolved": "https://registry.npmjs.org/send/-/send-0.17.1.tgz",
-      "integrity": "sha512-BsVKsiGcQMFwT8UxypobUKyv7irCNRHk1T0G680vk88yf6LBByGcZJOTJCrTP2xVN6yI+XjPJcNuE3V4fT9sAg==",
+      "version": "0.18.0",
+      "resolved": "https://registry.npmjs.org/send/-/send-0.18.0.tgz",
+      "integrity": "sha512-qqWzuOjSFOuqPjFe4NOsMLafToQQwBSOEpS+FwEt3A2V3vKubTquT3vmLTQpFgMXp8AlFWFuP1qKaJZOtPpVXg==",
       "dependencies": {
         "debug": "2.6.9",
-        "depd": "~1.1.2",
-        "destroy": "~1.0.4",
+        "depd": "2.0.0",
+        "destroy": "1.2.0",
         "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
         "etag": "~1.8.1",
         "fresh": "0.5.2",
-        "http-errors": "~1.7.2",
+        "http-errors": "2.0.0",
         "mime": "1.6.0",
-        "ms": "2.1.1",
-        "on-finished": "~2.3.0",
+        "ms": "2.1.3",
+        "on-finished": "2.4.1",
         "range-parser": "~1.2.1",
-        "statuses": "~1.5.0"
+        "statuses": "2.0.1"
       },
       "engines": {
         "node": ">= 0.8.0"
       }
     },
     "node_modules/send/node_modules/ms": {
-      "version": "2.1.1",
-      "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz",
-      "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg=="
+      "version": "2.1.3",
+      "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
+      "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
     },
     "node_modules/serve-static": {
-      "version": "1.14.1",
-      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.14.1.tgz",
-      "integrity": "sha512-JMrvUwE54emCYWlTI+hGrGv5I8dEwmco/00EvkzIIsR7MqrHonbD9pO2MOfFnpFntl7ecpZs+3mW+XbQZu9QCg==",
+      "version": "1.15.0",
+      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.15.0.tgz",
+      "integrity": "sha512-XGuRDNjXUijsUL0vl6nSD7cwURuzEgglbOaFuZM9g3kwDXOWVTck0jLzjPzGD+TazWbboZYu52/9/XPdUgne9g==",
       "dependencies": {
         "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
         "parseurl": "~1.3.3",
-        "send": "0.17.1"
+        "send": "0.18.0"
       },
       "engines": {
         "node": ">= 0.8.0"
       }
     },
     "node_modules/setprototypeof": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.1.tgz",
-      "integrity": "sha512-JvdAWfbXeIGaZ9cILp38HntZSFSo3mWg6xGcJJsd+d4aRMOqauag1C63dJfDw7OaMYwEbHMOxEZ1lqVRYP2OAw=="
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz",
+      "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw=="
+    },
+    "node_modules/side-channel": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.0.4.tgz",
+      "integrity": "sha512-q5XPytqFEIKHkGdiMIrY10mvLRvnQh42/+GoBlFW3b2LXLE2xxJpZFdm94we0BaoV3RwJyGqg5wS7epxTv0Zvw==",
+      "dependencies": {
+        "call-bind": "^1.0.0",
+        "get-intrinsic": "^1.0.2",
+        "object-inspect": "^1.9.0"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/ljharb"
+      }
     },
     "node_modules/statuses": {
-      "version": "1.5.0",
-      "resolved": "https://registry.npmjs.org/statuses/-/statuses-1.5.0.tgz",
-      "integrity": "sha1-Fhx9rBd2Wf2YEfQ3cfqZOBR4Yow=",
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.1.tgz",
+      "integrity": "sha512-RwNA9Z/7PrK06rYLIzFMlaF+l73iwpzsqRIFgbMLbTcLD6cOao82TaWefPXQvB2fOC4AjuYSEndS7N/mTCbkdQ==",
       "engines": {
-        "node": ">= 0.6"
+        "node": ">= 0.8"
       }
     },
     "node_modules/toidentifier": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.0.tgz",
-      "integrity": "sha512-yaOH/Pk/VEhBWWTlhI+qXxDFXlejDGcQipMlyxda9nthulaxLZUNcUqFxokp0vcYnvteJln5FNQDRrxj3YcbVw==",
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
+      "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==",
       "engines": {
         "node": ">=0.6"
       }
@@ -492,7 +593,7 @@
     "node_modules/unpipe": {
       "version": "1.0.0",
       "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
-      "integrity": "sha1-sr9O6FFKrmFltIF4KdIbLvSZBOw=",
+      "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==",
       "engines": {
         "node": ">= 0.8"
       }
@@ -516,12 +617,12 @@
   },
   "dependencies": {
     "accepts": {
-      "version": "1.3.7",
-      "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.7.tgz",
-      "integrity": "sha512-Il80Qs2WjYlJIBNzNkK6KYqlVMTbZLXgHx2oT0pU/fjRHyEp+PEfEPY0R3WCwAGVOtauxh1hOxNgIf5bv7dQpA==",
+      "version": "1.3.8",
+      "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
+      "integrity": "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw==",
       "requires": {
-        "mime-types": "~2.1.24",
-        "negotiator": "0.6.2"
+        "mime-types": "~2.1.34",
+        "negotiator": "0.6.3"
       }
     },
     "array-flatten": {
@@ -530,33 +631,44 @@
       "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI="
     },
     "body-parser": {
-      "version": "1.19.0",
-      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.19.0.tgz",
-      "integrity": "sha512-dhEPs72UPbDnAQJ9ZKMNTP6ptJaionhP5cBb541nXPlW60Jepo9RV/a4fX4XWW9CuFNK22krhrj1+rgzifNCsw==",
+      "version": "1.20.1",
+      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.1.tgz",
+      "integrity": "sha512-jWi7abTbYwajOytWCQc37VulmWiRae5RyTpaCyDcS5/lMdtwSz5lOpDE67srw/HYe35f1z3fDQw+3txg7gNtWw==",
       "requires": {
-        "bytes": "3.1.0",
+        "bytes": "3.1.2",
         "content-type": "~1.0.4",
         "debug": "2.6.9",
-        "depd": "~1.1.2",
-        "http-errors": "1.7.2",
+        "depd": "2.0.0",
+        "destroy": "1.2.0",
+        "http-errors": "2.0.0",
         "iconv-lite": "0.4.24",
-        "on-finished": "~2.3.0",
-        "qs": "6.7.0",
-        "raw-body": "2.4.0",
-        "type-is": "~1.6.17"
+        "on-finished": "2.4.1",
+        "qs": "6.11.0",
+        "raw-body": "2.5.1",
+        "type-is": "~1.6.18",
+        "unpipe": "1.0.0"
       }
     },
     "bytes": {
-      "version": "3.1.0",
-      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.0.tgz",
-      "integrity": "sha512-zauLjrfCG+xvoyaqLoV8bLVXXNGC4JqlxFCutSDWA6fJrTo2ZuvLYTqZ7aHBLZSMOopbzwv8f+wZcVzfVTI2Dg=="
+      "version": "3.1.2",
+      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
+      "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="
+    },
+    "call-bind": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/call-bind/-/call-bind-1.0.2.tgz",
+      "integrity": "sha512-7O+FbCihrB5WGbFYesctwmTKae6rOiIzmz1icreWJ+0aA7LJfuqhEso2T9ncpcFtzMQtzXf2QGGueWJGTYsqrA==",
+      "requires": {
+        "function-bind": "^1.1.1",
+        "get-intrinsic": "^1.0.2"
+      }
     },
     "content-disposition": {
-      "version": "0.5.3",
-      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.3.tgz",
-      "integrity": "sha512-ExO0774ikEObIAEV9kDo50o+79VCUdEB6n6lzKgGwupcVeRlhrj3qGAfwq8G6uBJjkqLrhT0qEYFcWng8z1z0g==",
+      "version": "0.5.4",
+      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz",
+      "integrity": "sha512-FveZTNuGw04cxlAiWbzi6zTAL/lhehaWbTtgluJh4/E95DqMwTmha3KZN1aAWA8cFIhHzMZUvLevkw5Rqk+tSQ==",
       "requires": {
-        "safe-buffer": "5.1.2"
+        "safe-buffer": "5.2.1"
       }
     },
     "content-type": {
@@ -565,9 +677,9 @@
       "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA=="
     },
     "cookie": {
-      "version": "0.4.0",
-      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.4.0.tgz",
-      "integrity": "sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg=="
+      "version": "0.5.0",
+      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.5.0.tgz",
+      "integrity": "sha512-YZ3GUyn/o8gfKJlnlX7g7xq4gyO6OSuhGPKaaGssGB2qgDUS0gPgtTvoyZLTt9Ab6dC4hfc9dV5arkvc/OCmrw=="
     },
     "cookie-signature": {
       "version": "1.0.6",
@@ -592,106 +704,135 @@
       }
     },
     "depd": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/depd/-/depd-1.1.2.tgz",
-      "integrity": "sha1-m81S4UwJd2PnSbJ0xDRu0uVgtak="
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
+      "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw=="
     },
     "destroy": {
-      "version": "1.0.4",
-      "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.0.4.tgz",
-      "integrity": "sha1-l4hXRCxEdJ5CBmE+N5RiBYJqvYA="
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.2.0.tgz",
+      "integrity": "sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg=="
     },
     "ee-first": {
       "version": "1.1.1",
       "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
-      "integrity": "sha1-WQxhFWsK4vTwJVcyoViyZrxWsh0="
+      "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="
     },
     "encodeurl": {
       "version": "1.0.2",
       "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
-      "integrity": "sha1-rT/0yG7C0CkyL1oCw6mmBslbP1k="
+      "integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w=="
     },
     "escape-html": {
       "version": "1.0.3",
       "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
-      "integrity": "sha1-Aljq5NPQwJdN4cFpGI7wBR0dGYg="
+      "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow=="
     },
     "etag": {
       "version": "1.8.1",
       "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
-      "integrity": "sha1-Qa4u62XvpiJorr/qg6x9eSmbCIc="
+      "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg=="
     },
     "express": {
-      "version": "4.17.1",
-      "resolved": "https://registry.npmjs.org/express/-/express-4.17.1.tgz",
-      "integrity": "sha512-mHJ9O79RqluphRrcw2X/GTh3k9tVv8YcoyY4Kkh4WDMUYKRZUq0h1o0w2rrrxBqM7VoeUVqgb27xlEMXTnYt4g==",
+      "version": "4.18.2",
+      "resolved": "https://registry.npmjs.org/express/-/express-4.18.2.tgz",
+      "integrity": "sha512-5/PsL6iGPdfQ/lKM1UuielYgv3BUoJfz1aUwU9vHZ+J7gyvwdQXFEBIEIaxeGf0GIcreATNyBExtalisDbuMqQ==",
       "requires": {
-        "accepts": "~1.3.7",
+        "accepts": "~1.3.8",
         "array-flatten": "1.1.1",
-        "body-parser": "1.19.0",
-        "content-disposition": "0.5.3",
+        "body-parser": "1.20.1",
+        "content-disposition": "0.5.4",
         "content-type": "~1.0.4",
-        "cookie": "0.4.0",
+        "cookie": "0.5.0",
         "cookie-signature": "1.0.6",
         "debug": "2.6.9",
-        "depd": "~1.1.2",
+        "depd": "2.0.0",
         "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
         "etag": "~1.8.1",
-        "finalhandler": "~1.1.2",
+        "finalhandler": "1.2.0",
         "fresh": "0.5.2",
+        "http-errors": "2.0.0",
         "merge-descriptors": "1.0.1",
         "methods": "~1.1.2",
-        "on-finished": "~2.3.0",
+        "on-finished": "2.4.1",
         "parseurl": "~1.3.3",
         "path-to-regexp": "0.1.7",
-        "proxy-addr": "~2.0.5",
-        "qs": "6.7.0",
+        "proxy-addr": "~2.0.7",
+        "qs": "6.11.0",
         "range-parser": "~1.2.1",
-        "safe-buffer": "5.1.2",
-        "send": "0.17.1",
-        "serve-static": "1.14.1",
-        "setprototypeof": "1.1.1",
-        "statuses": "~1.5.0",
+        "safe-buffer": "5.2.1",
+        "send": "0.18.0",
+        "serve-static": "1.15.0",
+        "setprototypeof": "1.2.0",
+        "statuses": "2.0.1",
         "type-is": "~1.6.18",
         "utils-merge": "1.0.1",
         "vary": "~1.1.2"
       }
     },
     "finalhandler": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.2.tgz",
-      "integrity": "sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA==",
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.2.0.tgz",
+      "integrity": "sha512-5uXcUVftlQMFnWC9qu/svkWv3GTd2PfUhK/3PLkYNAe7FbqJMt3515HaxE6eRL74GdsriiwujiawdaB1BpEISg==",
       "requires": {
         "debug": "2.6.9",
         "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
-        "on-finished": "~2.3.0",
+        "on-finished": "2.4.1",
         "parseurl": "~1.3.3",
-        "statuses": "~1.5.0",
+        "statuses": "2.0.1",
         "unpipe": "~1.0.0"
       }
     },
     "forwarded": {
-      "version": "0.1.2",
-      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.1.2.tgz",
-      "integrity": "sha1-mMI9qxF1ZXuMBXPozszZGw/xjIQ="
+      "version": "0.2.0",
+      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
+      "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow=="
     },
     "fresh": {
       "version": "0.5.2",
       "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz",
-      "integrity": "sha1-PYyt2Q2XZWn6g1qx+OSyOhBWBac="
+      "integrity": "sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q=="
+    },
+    "function-bind": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.1.tgz",
+      "integrity": "sha512-yIovAzMX49sF8Yl58fSCWJ5svSLuaibPxXQJFLmBObTuCr0Mf1KiPopGM9NiFjiYBCbfaa2Fh6breQ6ANVTI0A=="
+    },
+    "get-intrinsic": {
+      "version": "1.1.3",
+      "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.1.3.tgz",
+      "integrity": "sha512-QJVz1Tj7MS099PevUG5jvnt9tSkXN8K14dxQlikJuPt4uD9hHAHjLyLBiLR5zELelBdD9QNRAXZzsJx0WaDL9A==",
+      "requires": {
+        "function-bind": "^1.1.1",
+        "has": "^1.0.3",
+        "has-symbols": "^1.0.3"
+      }
+    },
+    "has": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/has/-/has-1.0.3.tgz",
+      "integrity": "sha512-f2dvO0VU6Oej7RkWJGrehjbzMAjFp5/VKPp5tTpWIV4JHHZK1/BxbFRtf/siA2SWTe09caDmVtYYzWEIbBS4zw==",
+      "requires": {
+        "function-bind": "^1.1.1"
+      }
+    },
+    "has-symbols": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.0.3.tgz",
+      "integrity": "sha512-l3LCuF6MgDNwTDKkdYGEihYjt5pRPbEg46rtlmnSPlUbgmB8LOIrKJbYYFBSbnPaJexMKtiPO8hmeRjRz2Td+A=="
     },
     "http-errors": {
-      "version": "1.7.2",
-      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.7.2.tgz",
-      "integrity": "sha512-uUQBt3H/cSIVfch6i1EuPNy/YsRSOUBXTVfZ+yR7Zjez3qjBz6i9+i4zjNaoqcoFVI4lQJ5plg63TvGfRSDCRg==",
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.0.tgz",
+      "integrity": "sha512-FtwrG/euBzaEjYeRqOgly7G0qviiXoJWnvEH2Z1plBdXgbyjv34pHTSb9zoeHMyDy33+DWy5Wt9Wo+TURtOYSQ==",
       "requires": {
-        "depd": "~1.1.2",
-        "inherits": "2.0.3",
-        "setprototypeof": "1.1.1",
-        "statuses": ">= 1.5.0 < 2",
-        "toidentifier": "1.0.0"
+        "depd": "2.0.0",
+        "inherits": "2.0.4",
+        "setprototypeof": "1.2.0",
+        "statuses": "2.0.1",
+        "toidentifier": "1.0.1"
       }
     },
     "iconv-lite": {
@@ -703,9 +844,9 @@
       }
     },
     "inherits": {
-      "version": "2.0.3",
-      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz",
-      "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4="
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
+      "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
     },
     "ipaddr.js": {
       "version": "1.9.1",
@@ -715,7 +856,7 @@
     "media-typer": {
       "version": "0.3.0",
       "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
-      "integrity": "sha1-hxDXrwqmJvj/+hzgAWhUUmMlV0g="
+      "integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ=="
     },
     "merge-descriptors": {
       "version": "1.0.1",
@@ -733,27 +874,27 @@
       "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg=="
     },
     "mime-db": {
-      "version": "1.45.0",
-      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.45.0.tgz",
-      "integrity": "sha512-CkqLUxUk15hofLoLyljJSrukZi8mAtgd+yE5uO4tqRZsdsAJKv0O+rFMhVDRJgozy+yG6md5KwuXhD4ocIoP+w=="
+      "version": "1.52.0",
+      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
+      "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="
     },
     "mime-types": {
-      "version": "2.1.28",
-      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.28.tgz",
-      "integrity": "sha512-0TO2yJ5YHYr7M2zzT7gDU1tbwHxEUWBCLt0lscSNpcdAfFyJOVEpRYNS7EXVcTLNj/25QO8gulHC5JtTzSE2UQ==",
+      "version": "2.1.35",
+      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
+      "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
       "requires": {
-        "mime-db": "1.45.0"
+        "mime-db": "1.52.0"
       }
     },
     "ms": {
       "version": "2.0.0",
       "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
-      "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g="
+      "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="
     },
     "negotiator": {
-      "version": "0.6.2",
-      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.2.tgz",
-      "integrity": "sha512-hZXc7K2e+PgeI1eDBe/10Ard4ekbfrrqG8Ep+8Jmf4JID2bNg7NvCPOZN+kfF574pFQI7mum2AUqDidoKqcTOw=="
+      "version": "0.6.3",
+      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz",
+      "integrity": "sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg=="
     },
     "nocache": {
       "version": "2.1.0",
@@ -765,10 +906,15 @@
       "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
       "integrity": "sha1-IQmtx5ZYh8/AXLvUQsrIv7s2CGM="
     },
+    "object-inspect": {
+      "version": "1.12.3",
+      "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.12.3.tgz",
+      "integrity": "sha512-geUvdk7c+eizMNUDkRpW1wJwgfOiOeHbxBR/hLXK1aT6zmVSO0jsQcs7fj6MGw89jC/cjGfLcNOrtMYtGqm81g=="
+    },
     "on-finished": {
-      "version": "2.3.0",
-      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.3.0.tgz",
-      "integrity": "sha1-IPEzZIGwg811M3mSoWlxqi2QaUc=",
+      "version": "2.4.1",
+      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz",
+      "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==",
       "requires": {
         "ee-first": "1.1.1"
       }
@@ -784,18 +930,21 @@
       "integrity": "sha1-32BBeABfUi8V60SQ5yR6G/qmf4w="
     },
     "proxy-addr": {
-      "version": "2.0.6",
-      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.6.tgz",
-      "integrity": "sha512-dh/frvCBVmSsDYzw6n926jv974gddhkFPfiN8hPOi30Wax25QZyZEGveluCgliBnqmuM+UJmBErbAUFIoDbjOw==",
+      "version": "2.0.7",
+      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
+      "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==",
       "requires": {
-        "forwarded": "~0.1.2",
+        "forwarded": "0.2.0",
         "ipaddr.js": "1.9.1"
       }
     },
     "qs": {
-      "version": "6.7.0",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.7.0.tgz",
-      "integrity": "sha512-VCdBRNFTX1fyE7Nb6FYoURo/SPe62QCaAyzJvUjwRaIsc+NePBEniHlvxFmmX56+HZphIGtV0XeCirBtpDrTyQ=="
+      "version": "6.11.0",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.11.0.tgz",
+      "integrity": "sha512-MvjoMCJwEarSbUYk5O+nmoSzSutSsTwF85zcHPQ9OrlFoZOYIjaqBAJIqIXjptyD5vThxGq52Xu/MaJzRkIk4Q==",
+      "requires": {
+        "side-channel": "^1.0.4"
+      }
     },
     "range-parser": {
       "version": "1.2.1",
@@ -803,20 +952,20 @@
       "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="
     },
     "raw-body": {
-      "version": "2.4.0",
-      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.4.0.tgz",
-      "integrity": "sha512-4Oz8DUIwdvoa5qMJelxipzi/iJIi40O5cGV1wNYp5hvZP8ZN0T+jiNkL0QepXs+EsQ9XJ8ipEDoiH70ySUJP3Q==",
+      "version": "2.5.1",
+      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.1.tgz",
+      "integrity": "sha512-qqJBtEyVgS0ZmPGdCFPWJ3FreoqvG4MVQln/kCgF7Olq95IbOp0/BWyMwbdtn4VTvkM8Y7khCQ2Xgk/tcrCXig==",
       "requires": {
-        "bytes": "3.1.0",
-        "http-errors": "1.7.2",
+        "bytes": "3.1.2",
+        "http-errors": "2.0.0",
         "iconv-lite": "0.4.24",
         "unpipe": "1.0.0"
       }
     },
     "safe-buffer": {
-      "version": "5.1.2",
-      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
-      "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="
+      "version": "5.2.1",
+      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
+      "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ=="
     },
     "safer-buffer": {
       "version": "2.1.2",
@@ -824,57 +973,67 @@
       "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
     },
     "send": {
-      "version": "0.17.1",
-      "resolved": "https://registry.npmjs.org/send/-/send-0.17.1.tgz",
-      "integrity": "sha512-BsVKsiGcQMFwT8UxypobUKyv7irCNRHk1T0G680vk88yf6LBByGcZJOTJCrTP2xVN6yI+XjPJcNuE3V4fT9sAg==",
+      "version": "0.18.0",
+      "resolved": "https://registry.npmjs.org/send/-/send-0.18.0.tgz",
+      "integrity": "sha512-qqWzuOjSFOuqPjFe4NOsMLafToQQwBSOEpS+FwEt3A2V3vKubTquT3vmLTQpFgMXp8AlFWFuP1qKaJZOtPpVXg==",
       "requires": {
         "debug": "2.6.9",
-        "depd": "~1.1.2",
-        "destroy": "~1.0.4",
+        "depd": "2.0.0",
+        "destroy": "1.2.0",
         "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
         "etag": "~1.8.1",
         "fresh": "0.5.2",
-        "http-errors": "~1.7.2",
+        "http-errors": "2.0.0",
         "mime": "1.6.0",
-        "ms": "2.1.1",
-        "on-finished": "~2.3.0",
+        "ms": "2.1.3",
+        "on-finished": "2.4.1",
         "range-parser": "~1.2.1",
-        "statuses": "~1.5.0"
+        "statuses": "2.0.1"
       },
       "dependencies": {
         "ms": {
-          "version": "2.1.1",
-          "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz",
-          "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg=="
+          "version": "2.1.3",
+          "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
+          "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
         }
       }
     },
     "serve-static": {
-      "version": "1.14.1",
-      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.14.1.tgz",
-      "integrity": "sha512-JMrvUwE54emCYWlTI+hGrGv5I8dEwmco/00EvkzIIsR7MqrHonbD9pO2MOfFnpFntl7ecpZs+3mW+XbQZu9QCg==",
+      "version": "1.15.0",
+      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.15.0.tgz",
+      "integrity": "sha512-XGuRDNjXUijsUL0vl6nSD7cwURuzEgglbOaFuZM9g3kwDXOWVTck0jLzjPzGD+TazWbboZYu52/9/XPdUgne9g==",
       "requires": {
         "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
         "parseurl": "~1.3.3",
-        "send": "0.17.1"
+        "send": "0.18.0"
       }
     },
     "setprototypeof": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.1.tgz",
-      "integrity": "sha512-JvdAWfbXeIGaZ9cILp38HntZSFSo3mWg6xGcJJsd+d4aRMOqauag1C63dJfDw7OaMYwEbHMOxEZ1lqVRYP2OAw=="
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz",
+      "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw=="
+    },
+    "side-channel": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.0.4.tgz",
+      "integrity": "sha512-q5XPytqFEIKHkGdiMIrY10mvLRvnQh42/+GoBlFW3b2LXLE2xxJpZFdm94we0BaoV3RwJyGqg5wS7epxTv0Zvw==",
+      "requires": {
+        "call-bind": "^1.0.0",
+        "get-intrinsic": "^1.0.2",
+        "object-inspect": "^1.9.0"
+      }
     },
     "statuses": {
-      "version": "1.5.0",
-      "resolved": "https://registry.npmjs.org/statuses/-/statuses-1.5.0.tgz",
-      "integrity": "sha1-Fhx9rBd2Wf2YEfQ3cfqZOBR4Yow="
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.1.tgz",
+      "integrity": "sha512-RwNA9Z/7PrK06rYLIzFMlaF+l73iwpzsqRIFgbMLbTcLD6cOao82TaWefPXQvB2fOC4AjuYSEndS7N/mTCbkdQ=="
     },
     "toidentifier": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.0.tgz",
-      "integrity": "sha512-yaOH/Pk/VEhBWWTlhI+qXxDFXlejDGcQipMlyxda9nthulaxLZUNcUqFxokp0vcYnvteJln5FNQDRrxj3YcbVw=="
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
+      "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA=="
     },
     "type-is": {
       "version": "1.6.18",
@@ -888,7 +1047,7 @@
     "unpipe": {
       "version": "1.0.0",
       "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
-      "integrity": "sha1-sr9O6FFKrmFltIF4KdIbLvSZBOw="
+      "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="
     },
     "utils-merge": {
       "version": "1.0.1",
diff --git a/wasm/test_page/package.json b/wasm/test_page/package.json
index 20af6d2ab..79447e3bf 100644
--- a/wasm/test_page/package.json
+++ b/wasm/test_page/package.json
@@ -1,7 +1,7 @@
 {
   "dependencies": {
     "cors": "^2.8.5",
-    "express": "^4.17.1",
+    "express": "^4.18.2",
     "nocache": "^2.1.0"
   }
 }

From 6f2659fe592dfed44ddf3062f602a91122fb391c Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Wed, 18 Jan 2023 16:31:36 +0000
Subject: [PATCH 391/442] Arm updated (#443)

* ARM Support using ruy and simd_utils

* Adding ARM build on GitHub CI

* Add workflow and successful build

ssplit-cpp modified to get cross compiled android on GitHub CI working.

* Client side fixes for int8 no shift on ARM [python]

* Revert "Client side fixes for int8 no shift on ARM [python]"

This reverts commit 020af05a8b1f4b4ef46373e6e61dcd32869fc1b1.

* moving int8shift no-op inside the library

* Bump 3rd-party/marian-dev

* update the marian branch test

* arm backend works

* Latest and greatest clang-format

Co-authored-by: Jerin Philip <jerinphilip@live.in>
---
 .github/workflows/arm.yml | 139 ++++++++++++++++++++++++++++++++++++++
 3rd_party/CMakeLists.txt  |   6 ++
 3rd_party/ssplit-cpp      |   2 +-
 3 files changed, 146 insertions(+), 1 deletion(-)
 create mode 100644 .github/workflows/arm.yml

diff --git a/.github/workflows/arm.yml b/.github/workflows/arm.yml
new file mode 100644
index 000000000..2ee14548d
--- /dev/null
+++ b/.github/workflows/arm.yml
@@ -0,0 +1,139 @@
+name: ARM
+'on':
+  push:
+    branches:
+    - main
+    - ci-sandbox
+  pull_request:
+    branches:
+    - '**'
+env:
+  ccache_basedir: ${{ github.workspace }}
+  ccache_dir: "${{ github.workspace }}/.ccache"
+  ccache_compilercheck: content
+  ccache_compress: 'true'
+  ccache_compresslevel: 9
+  ccache_maxsize: 200M
+  ccache_cmake: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
+  ndk: "${{ github.workspace }}/android-ndk-r23b"
+  abi: "arm64-v8a"
+  minsdk_version : 28
+  android_platform: 28
+
+jobs:
+  ubuntu:
+    name: "arm-v8a cross-compile via Android NDK"
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Install prerequisites
+      run: |
+          wget -c --quiet https://dl.google.com/android/repository/android-ndk-r23b-linux.zip
+          unzip -qq android-ndk-r23b-linux.zip
+          sudo apt-get -y install ccache cmake
+
+    - name: Generate ccache_vars for ccache based on machine
+      shell: bash
+      id: ccache_vars
+      run: |-
+        echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
+        echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
+
+    - name: Cache-op for build-cache through ccache
+      uses: actions/cache@v2
+      with:
+        path: ${{ env.ccache_dir }}
+        key: ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
+        restore-keys: |-
+          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
+          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}
+          ccache-${{ matrix.identifier }}
+
+    - name: ccache environment setup
+      run: |-
+        echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
+        echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
+        echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
+        echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
+        echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
+        echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
+
+    - name: ccache prolog
+      run: |-
+        ccache -s # Print current cache stats
+        ccache -z # Zero cache entry
+
+    - name: Generate buildfiles for bergamot-translator on android via cmake
+      run: |-
+        mkdir -p build 
+        cd build
+        NDK=${{ env.ndk }}
+        ABI=${{ env.abi }}
+        MINSDK_VERSION=${{ env.minsdk_version }}
+        ANDROID_PLATFORM=android-${{ env.android_platform }}
+        OTHER_ANDROID_ARGS=(
+            -DANDROID_ARM_NEON=TRUE
+        )
+        OTHER_MARIAN_ARGS=(
+            -DCOMPILE_CUDA=off
+            -DCOMPILE_CPU=on
+            -DCMAKE_HAVE_THREADS_LIBRARY=1
+            -DCMAKE_USE_WIN32_THREADS_INIT=0
+            -DCMAKE_USE_PTHREADS_INIT=1
+            -DTHREADS_PREFER_PTHREAD_FLAG=ON
+            -DBUILD_ARCH=armv8-a
+            # -DCOMPILE_WITHOUT_EXCEPTIONS=on # Apparently this can reduce the binary size, let's see.
+            -DSSPLIT_USE_INTERNAL_PCRE2=ON
+        )
+        # Additionally list variables finally configured.
+        cmake -L \
+            -DCMAKE_BUILD_TYPE=Release \
+            -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \
+            -DANDROID_TOOLCHAIN=clang \
+            -DANDROID_ABI=$ABI \
+            -DANDROID_PLATFORM=$ANDROID_PLATFORM \
+            -DANDROID_NATIVE_API_LEVEL=$MINSDKVERSION \
+            -DANDROID_TOOLCHAIN_NAME=arm-linux-androideabi-4.8 \
+            -DANDROID_STL=c++_static \
+            -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache \
+            "${OTHER_ANDROID_ARGS[@]}" "${OTHER_MARIAN_ARGS[@]}" \
+            ..
+
+
+    - name : Build bergamot-translator for android
+      working-directory: build
+      run: |-
+          make -j2 
+
+    - name: ccache epilog
+      run: 'ccache -s # Print current cache stats'
+
+    - uses: actions/upload-artifact@v2
+      with:
+        path: ${{github.workspace}}/build/app/bergamot
+
+
+  # Disable release for now.
+  # release:
+  #   name: Release Latest Build
+  #   runs-on: ubuntu-latest
+  #   needs: [ubuntu]
+  #   if: github.ref == 'refs/heads/master'
+  #   steps:
+  #    - name: Download artifacts
+  #      uses: actions/download-artifact@v2
+  #     
+  #    - name: Update GitHub prerelease
+  #      uses: marvinpinto/action-automatic-releases@latest
+  #      with:
+  #        repo_token: ${{ secrets.GITHUB_TOKEN }}
+  #        automatic_release_tag: latest
+  #        prerelease: true
+  #        title: "Latest Build"
+  #        files: |
+  #          artifact/marian-decoder
diff --git a/3rd_party/CMakeLists.txt b/3rd_party/CMakeLists.txt
index 1888d6da6..eac898eb9 100644
--- a/3rd_party/CMakeLists.txt
+++ b/3rd_party/CMakeLists.txt
@@ -19,6 +19,12 @@ target_include_directories(marian PUBLIC ${INCDIRS})
 get_property(INCLUDE_DIRECTORIES DIRECTORY ssplit-cpp/src PROPERTY INCLUDE_DIRECTORIES)
 target_include_directories(ssplit PUBLIC ${INCLUDE_DIRECTORIES})
 
+get_property(COMPILE_DEFINITIONS DIRECTORY marian-dev PROPERTY COMPILE_DEFINITIONS) 
+target_compile_definitions(marian PUBLIC ${COMPILE_DEFINITIONS})
+
+get_property(COMPILE_OPTIONS DIRECTORY marian-dev PROPERTY COMPILE_OPTIONS) 
+target_compile_options(marian PUBLIC ${COMPILE_OPTIONS})
+
 # Compilation flags 
 get_directory_property(CMAKE_C_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_C_FLAGS) 
 get_directory_property(CMAKE_CXX_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_CXX_FLAGS) 
diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 49fde6df7..8bc2f35b6 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 49fde6df7ee9199aedb9571be800448192e3515c
+Subproject commit 8bc2f35b64f1012b8c31087610ab42f67aa70154

From 7d24908959670339bfad9959af1bf706da102540 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Wed, 18 Jan 2023 16:46:07 +0000
Subject: [PATCH 392/442] Apply security update and formatting

---
 .github/workflows/build.yml         |  2 +-
 .github/workflows/coding-styles.yml |  6 +++---
 3rd_party/ssplit-cpp                |  2 +-
 bindings/python/repository.py       | 21 ++++++++++++++++++++-
 4 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 25305a177..2cac911a3 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -352,7 +352,7 @@ jobs:
             python3 -m pip install black isort pytype
       - name: "Formatting checks: black, isort"
         run: |
-            python3 -m black --check bindings/python/ setup.py doc/conf.py
+            python3 -m black --diff --check bindings/python/ setup.py doc/conf.py
             python3 -m isort --profile black --diff --check bindings/python setup.py doc/conf.py
       - name: "Static typing checks: pytype"
         run: |-
diff --git a/.github/workflows/coding-styles.yml b/.github/workflows/coding-styles.yml
index e666c51df..b13345601 100644
--- a/.github/workflows/coding-styles.yml
+++ b/.github/workflows/coding-styles.yml
@@ -1,6 +1,6 @@
 name: "Coding Style"
 
-on: 
+on:
   push:
     branches: [ main, ci-sandbox ]
   pull_request:
@@ -18,7 +18,7 @@ jobs:
 
         - name: Install dependencies
           run: |
-            sudo apt-get update 
+            sudo apt-get update
             sudo apt-get install -y build-essential cmake
             sudo apt-get install -y clang-format clang-tidy
 
@@ -30,7 +30,7 @@ jobs:
         - name: Prepare build, compilation database etc.
           run: |
               mkdir -p build
-              cd build 
+              cd build
               cmake \
                 -DUSE_WASM_COMPATIBLE_SOURCE=off -DCMAKE_EXPORT_COMPILE_COMMANDS=on \
                 -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index 8bc2f35b6..ad2c5a52a 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit 8bc2f35b64f1012b8c31087610ab42f67aa70154
+Subproject commit ad2c5a52a507ec5a1f58c6403fc674e76e92e185
diff --git a/bindings/python/repository.py b/bindings/python/repository.py
index a8e4a79c1..323b4482b 100644
--- a/bindings/python/repository.py
+++ b/bindings/python/repository.py
@@ -137,7 +137,26 @@ def download(self, model_identifier: str):
         download_resource(model["url"], save_location)
 
         with tarfile.open(save_location) as model_archive:
-            model_archive.extractall(self.dirs["models"])
+
+            def is_within_directory(directory, target):
+
+                abs_directory = os.path.abspath(directory)
+                abs_target = os.path.abspath(target)
+
+                prefix = os.path.commonprefix([abs_directory, abs_target])
+
+                return prefix == abs_directory
+
+            def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
+
+                for member in tar.getmembers():
+                    member_path = os.path.join(path, member.name)
+                    if not is_within_directory(path, member_path):
+                        raise Exception("Attempted Path Traversal in Tar File")
+
+                tar.extractall(path, members, numeric_owner=numeric_owner)
+
+            safe_extract(model_archive, self.dirs["models"])
             fprefix = self._archive_name_without_extension(model["url"])
             model_dir = os.path.join(self.dirs["models"], fprefix)
             symlink = os.path.join(self.dirs["models"], model["code"])

From 2834f046dc16b4e17b184998db513468719388f3 Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Wed, 18 Jan 2023 19:09:47 +0000
Subject: [PATCH 393/442] Expand the node-test.js example code with
 documentation (#434)

* Expand the node-test.js example code with documentation

Is there a better way to document code than by providing an annotated & working example of it? Just listing all the exposed methods feels like giving people a box of bricks and expecting them to build a house with it.

* Use @Jerin's feedback to simplify node-test.js explanations

* Use native `console.assert` instead

See #426 for an explanation

* Fix comment

Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
---
 .github/workflows/build.yml |   5 +-
 wasm/README.md              |   4 +-
 wasm/node-test.js           | 198 ++++++++++++++++++++++--------------
 3 files changed, 130 insertions(+), 77 deletions(-)
 mode change 100644 => 100755 wasm/node-test.js

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 2cac911a3..cc99b6711 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -252,11 +252,12 @@ jobs:
           working-directory: build-wasm
           run: bash ../wasm/patch-artifacts-import-gemm-module.sh
 
-        # Setup nodejs-16, as nodejs-14 provided by emsdk fails when running.
+        # Setup nodejs-18, as nodejs-14 provided by emsdk fails when running
+        # and newer version of node allows us to use fetch().
         - name: Setup nodejs
           uses: actions/setup-node@v3
           with:
-            node-version: 16
+            node-version: 18
 
         - name: Test run
           working-directory: wasm
diff --git a/wasm/README.md b/wasm/README.md
index 883f80dc5..e2d9a447c 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -4,7 +4,9 @@ All the instructions below are meant to run from the current directory.
 
 ## Using JS APIs
 
-Please refer to the file `test_page/js/worker.js` that demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
+See [node-test.js](./node-test.js) for an annotated example of how to use the WASM module. Most of the code from it can also be used in a browser context.
+
+Alternatively refer to the file `test_page/js/worker.js` that demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
 
 ## Demo
 
diff --git a/wasm/node-test.js b/wasm/node-test.js
old mode 100644
new mode 100755
index 1f697afd3..8734858f5
--- a/wasm/node-test.js
+++ b/wasm/node-test.js
@@ -1,124 +1,174 @@
-const {Blob} = require('buffer');
-const fs = require('fs');
-const https = require('https');
-const {JSDOM} = require('jsdom');
+#!/usr/bin/env node
+
+/**
+ * A note upfront: the bergamot-translator API is pretty low level, and
+ * embedding it successfully requires some knowledge about the WebWorkers and
+ * WebAssembly APIs. This script tries to demonstrate the bergamot-translator
+ * API with as little of that boiler plate code as possible.
+ * See the wasm/test_page code for a fully fleshed out demo in a web context.
+ */
 
+// For node we use the fs module to read local files. In a web context you can
+// use `fetch()` for everything.
+const fs = require('fs');
 
+// Read wasm binary into a blob, which will be loaded by
+// bergamot-translator-worker.js in a minute. In a web context, you'd be using
+// `fetch(...).then(response => response.blob())` for this, but Node does not
+// implement `fetch("file://...")` yet.
 const wasmBinary = fs.readFileSync('./bergamot-translator-worker.wasm');
+
+// Read wasm runtime code that bridges the bergmot-translator binary with JS.
+const wasmRuntime = fs.readFileSync('./bergamot-translator-worker.js', {encoding: 'utf8'});
+
+// Initialise the `Module` object. By adding methods and options to this, we can
+// affect how bergamot-translator interacts with JavaScript. See 
+// https://emscripten.org/docs/api_reference/module.html for all available
+// options. It is important that this object is initialised in the same scope
+// but before `bergamot-translation-worker.js` is executed. Once that script
+// executes, it defines the exported methods as properties of this Module
+// object.
 global.Module = {
   wasmBinary,
   onRuntimeInitialized
 };
 
-// Execute bergamot-translation-worker.js in this scope
-const js = fs.readFileSync('./bergamot-translator-worker.js', {encoding: 'utf8'});
-eval.call(global, js);
-
-/**
- * Helper to download file into ArrayBuffer.
- */
-function download(url) {
-  return new Promise((accept, reject) => {
-    https.get(url, (res) => {
-      const chunks = [];
-      res.on('error', reject);
-      res.on('data', chunk => chunks.push(chunk));
-      res.on('end', async () => {
-        const data = new Blob(chunks);
-        data.arrayBuffer().then(accept, reject);
-      });
-    });
-  });
-}
-
-/**
- * Loads ArrayBuffer into AlignedMemory.
- */
-function load(buffer, alignment) {
-  const bytes = new Int8Array(buffer);
-  const memory = new Module.AlignedMemory(bytes.byteLength, alignment);
-  memory.getByteArrayView().set(bytes);
-  return memory;
-}
+// Execute bergamot-translation-worker.js in this scope. This will also,
+// indirectly, call the onRuntimeInitialized function defined below and
+// referenced in the `Module` object above.
+eval.call(global, wasmRuntime);
 
 /**
- * Called from inside the worker.js script once the wasm module is loaded
- * and all the emscripten magic and linking has been done.
+ * Called from inside the bergamot-translation-worker.js script once the wasm
+ * module is initialized. At this point that `Module` object that was
+ * initialised above will have all the classes defined in the
+ * bergamot-translator API available on it.
  */
 async function onRuntimeInitialized() {
   // Root url for our models for now.
-  const root = 'https://storage.googleapis.com/bergamot-models-sandbox/0.2.14';
+  const root = 'https://storage.googleapis.com/bergamot-models-sandbox/0.3.1';
 
-  // In order of TranslationMemory's arguments
+  // Urls of data files necessary to create a translation model for
+  // English -> German. Note: list is in order of TranslationModel's arguments.
+  // The `alignment` value is used later on to load each part of the model with
+  // the correct alignment.
   const files = [
+    // Neural network and weights:
     {url: `${root}/ende/model.ende.intgemm.alphas.bin`, alignment: 256},
+    
+    // Lexical shortlist which is mainly a speed improvement method, not
+    // strictly necessary:
     {url: `${root}/ende/lex.50.50.ende.s2t.bin`, alignment: 64},
+    
+    // Vocabulary, maps the input and output nodes of the neural network to
+    // strings. Note: "deen" may look the wrong way around but vocab is the same
+    // between de->en and en->de models.
     {url: `${root}/ende/vocab.deen.spm`, alignment: 64},
   ];
 
-  // Download model data and load it into aligned memory
+  // Download model data and load it into aligned memory. AlignedMemory is a
+  // necessary wrapper around allocated memory inside the WASM environment.
+  // The value of `alignment` is specific for which part of the model we're
+  // loading. See https://en.wikipedia.org/wiki/Data_structure_alignment for a
+  // more general explanation.
   const [modelMem, shortlistMem, vocabMem] = await Promise.all(files.map(async (file) => {
-    return load(await download(file.url), file.alignment);
+    const response = await fetch(file.url);
+    const blob = await response.blob();
+    const buffer = await blob.arrayBuffer();
+    const bytes = new Int8Array(buffer);
+    const memory = new Module.AlignedMemory(bytes.byteLength, file.alignment);
+    memory.getByteArrayView().set(bytes);
+    return memory;
   }));
 
+  // Set up translation service. This service translates a batch of text per
+  // call. The larger the batch, the faster the translation (in words per
+  // second) happens, but the longer you have to wait for all of them to finish.
+  // The constructor expects an object with options, but only one option is
+  // currently supported: `cacheSize`. Setting this to `0` disables the
+  // translation cache.
+  // **Note**: cacheSize is the theoretical maximum number of sentences that
+  // will be cached. In practise, about 1/3 of that will actually be used.
+  // See https://github.com/XapaJIaMnu/translateLocally/pull/75
+  const service = new Module.BlockingService({cacheSize: 0});
+
+  // Put vocab into its own std::vector<AlignedMemory>. Most models for the
+  // Bergamot project only have one vocabulary that is shared by both the input
+  // and output side of the translator. But in theory, you could have one for
+  // the input side and a different one for the output side. Hence: a list.
+  const vocabs = new Module.AlignedMemoryList();
+  vocabs.push_back(vocabMem);
+
   // Config yaml (split as array to allow for indentation without adding tabs
   // or spaces to the strings themselves.)
+  // See https://marian-nmt.github.io/docs/cmd/marian-decoder/ for the meaning
+  // of most of these options and what other options might be available.
   const config = [
     'beam-size: 1',
     'normalize: 1.0',
     'word-penalty: 0',
-    'alignment: soft',
+    'alignment: soft', // is necessary if you want to use HTML at any point
     'max-length-break: 128',
     'mini-batch-words: 1024',
     'workspace: 128',
     'max-length-factor: 2.0',
     'skip-cost: true',
-    'cpu-threads: 0',
-    'quiet: true',
-    'quiet-translation: true',
-    'gemm-precision: int8shiftAll',
+    'gemm-precision: int8shiftAll', // is necessary for speed and compatibility with Mozilla's models.
   ].join('\n');
 
-  // Set up translation service
-  const service = new Module.BlockingService({cacheSize: 0});
-
-  // Put vocab into its own std::vector<AlignedMemory>
-  const vocabs = new Module.AlignedMemoryList();
-  vocabs.push_back(vocabMem);
-
-  // Setup up model with config yaml and AlignedMemory objects
+  // Setup up model with config yaml and AlignedMemory objects. Optionally a
+  // quality estimation model can also be loaded but this is not demonstrated
+  // here. Generally you don't need it, and many models don't include the data
+  // file necessary to use it anyway.
   const model = new Module.TranslationModel(config, modelMem, shortlistMem, vocabs, /*qualityModel=*/ null);
 
-  // Construct std::vector<std::string> inputs;
+  // Construct std::vector<std::string> inputs; This is our batch!
   const input = new Module.VectorString();
-  input.push_back('<p> Hello world! </p> <p> Goodbye World! </p>');
-
-  // Construct std::vector<ResponseOptions>
+  input.push_back('<p>Hello world! Let us write a second sentence.</p> &amp; <p>Goodbye World!</p>');
+  input.push_back('This is a second example without HTML & entities.');
+
+  // Construct std::vector<ResponseOptions>, one entry per input. Note that
+  // all these three properties of your ResponseOptions object need to be
+  // specified for each entry.
+  // `qualityScores`: related to quality models not explained here. Set this
+  //   to `false`.
+  // `alignment`: computes alignment scores that maps parts of the input text
+  //   to parts of the output text. There is currently no way to get these
+  //   mappings out through the JavaScript API so I suggest you set this to
+  //   `false` as well.
+  // `html`: is the input HTML? If so, the HTML will be parsed and the markup
+  //   will be copied back into the translated output. Note: HTML has to be
+  //   valid HTML5, with proper closing tags and everything since the HTML
+  //   parser built into bergamot-translator does no error correction. Output
+  //   of e.g. `Element.innerHTML` meets this criteria.
   const options = new Module.VectorResponseOptions();
-  options.push_back({qualityScores: false, alignment: true, html: true});
+  options.push_back({qualityScores: false, alignment: false, html: true});
+  options.push_back({qualityScores: false, alignment: false, html: false});
 
-  // Translate our batch (of 1)
-  const output = service.translate(model, input, options);
-
-  // Get output from std::vector<Response>
-  // The following works as a simple black-box test of the API, based on
-  // properties of HTML.
-  const translation = output.get(0).getTranslatedText()
+  // Size of `input` and `options` has to match.
+  console.assert(input.size() === options.size());
 
-  // Print raw translation for inspection.
-  console.log(translation)
+  // Translate our batch of 2 requests. Output will be another vector of type 
+  // `std::vector<Response>`.
+  const output = service.translate(model, input, options);
 
-  const fragment = JSDOM.fragment(translation)
+  console.assert(false);
 
-  // Print two expected tags.
-  console.log(fragment.firstElementChild.outerHTML)
-  console.log(fragment.lastElementChild.outerHTML)
+  // Number of outputs is number of inputs.
+  console.assert(input.size() === output.size());
 
-  // Assertion that there are two children at the output.
-  assert(fragment.childElementCount === 2);
+  for (let i = 0; i < output.size(); ++i) {
+    // Get output from std::vector<Response>.
+    const translation = output.get(i).getTranslatedText();
 
+    // Print raw translation for inspection.
+    console.log(translation)
+  }
 
-  // Clean-up
+  // Clean-up: unlike the objects in JavaScript, the objects in the WASM
+  // environment are not automatically cleaned up when they're no longer
+  // referenced. That is why we manually have to call `delete()` on them
+  // when we're done with them.
   input.delete();
   options.delete();
   output.delete();

From 8d5f877596386584caf1a13c5aae7f4427708ca9 Mon Sep 17 00:00:00 2001
From: Jelmer <jelmer@ikhoefgeen.nl>
Date: Wed, 18 Jan 2023 19:41:39 +0000
Subject: [PATCH 394/442] More portable WASM demo (#437)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* Replace most of the wasm demo page with code from the firefox extension

This code should be more generic and copy/pastable into other projects. Maybe one day it will be an npm package?

* Fix Ukrainian model support

* Add quality estimation output

Automatically enabled when the model(s) support it

* Little "Translating…" indicator

* Don't make Safari fail on something tiny

* Rewire lots of async state to be able to predictably know when the translator is working or not

Previously so much was lazy loaded that it was not easy to catch lack of SIMD support. Now I can just enable the interface only after it has properly loaded.

* No need for a two-stage setup for the worker. Just promise to call `initialize()`!

* More (correct) types and comments for code

* Keyboard shortcuts for input area for bold, italic and underline.

Enough to demo mark-up translation

* Fix `delete()`

* Move javascript glue code into its own npm package

* Add nodejs support and test to package

* More stand-alone build command

…for now, not really used by anything I think

* Ignore build packages

* Use local filesystem for build so it is automatically cached

* fix overflow on demo page

But this might break the mobile demo? I'll have to check into that

* Bring back integrity check, except for NodeJS for now

* Make `build` part of `prepare` so we always make sure we build a complete package

* Move worker code into its own folder

This way I can mark it as a commonjs module which will help cause nodejs treat the files the same as WebWorkers do right now. Firefox doesn't implement `{type: 'module'}` yet for WebWorkers.

* Add README

* Fix paths

* Add npm publish automation

* Make sure webpack ignores node compatibility code

* Add missing webpack:ignore around a worker

* Default to getting models from S3

* Separate "loading" and "translating" indicators

* Bump npm package version

* Add credits

* Don't block on the worker loading

* Not just Mozilla, but Bergamot!

* Make individual translation requests cancelable

* Swap button turns vertically when in skyscraper mode

* Make it easier to debug errors from inside the worker

* Don't bork on deleting a failed worker

* Don't bork on calling translate() with a failed worker

* Handle compilation error with more grace

* `contenteditable=true` seems to work better with some browser extensions

Looking at you, Vimium!

* Clean up abort promise

* Bump npm package version

* Remove `workerUrl` option in favour of better webpack support

With that option it was hard for Webpack to figure out dependencies, and it did not enter my worker script for rewriting. With the hardcoded url it does, and with a bit of `new webpack.DefinePlugin({'typeof self': JSON.stringify('object')}),` we can have webpack remove node-specific code on build!

* Bump version

Minor API change hehe

Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
---
 .github/workflows/build.yml             |  23 +
 .gitignore                              |   3 +-
 wasm/module/README.md                   | 238 +++++++
 wasm/module/main.js                     |  21 +
 wasm/module/package.json                |  39 ++
 wasm/module/translator.js               | 879 ++++++++++++++++++++++++
 wasm/module/worker/package.json         |   3 +
 wasm/module/worker/translator-worker.js | 475 +++++++++++++
 wasm/test_page/css/index.css            |  70 +-
 wasm/test_page/index.html               |  18 +-
 wasm/test_page/js/index.js              | 309 +++++----
 wasm/test_page/js/worker.js             | 352 ----------
 wasm/test_page/logos.png                | Bin 0 -> 15207 bytes
 wasm/test_page/package-lock.json        |  13 +
 wasm/test_page/package.json             |   7 +
 wasm/test_page/start_server.sh          |   2 +-
 16 files changed, 1958 insertions(+), 494 deletions(-)
 create mode 100644 wasm/module/README.md
 create mode 100644 wasm/module/main.js
 create mode 100644 wasm/module/package.json
 create mode 100644 wasm/module/translator.js
 create mode 100644 wasm/module/worker/package.json
 create mode 100644 wasm/module/worker/translator-worker.js
 delete mode 100644 wasm/test_page/js/worker.js
 create mode 100644 wasm/test_page/logos.png

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index cc99b6711..d0afe1649 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -281,6 +281,29 @@ jobs:
                 ${{github.workspace}}/build-wasm/bergamot-translator-worker.wasm
                 ${{github.workspace}}/build-wasm/bergamot-translator-worker.js.bak
 
+    
+    upload-wasm:
+      name: "Upload node package to NPM"
+      runs-on: ubuntu-latest
+      if: ${{ startsWith(github.ref, 'refs/tags/v') }}
+      needs: [build-wasm]
+      steps:
+      - name: Download artifacts
+        uses: actions/download-artifact@v2
+        with:
+          name: wasm-artefacts
+          path: wasm/module/worker
+
+      - uses: actions/setup-node@v3
+        with:
+          node-version: '18.x'
+          registry-url: 'https://registry.npmjs.org'
+      - run: npm ci
+      - run: npm publish
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+
+
 
   # Try to upload a release using https://github.com/marvinpinto/actions/issues/177#issuecomment-917605585 as a model
     release-latest:
diff --git a/.gitignore b/.gitignore
index 64c1aa3a7..c796e0656 100644
--- a/.gitignore
+++ b/.gitignore
@@ -19,7 +19,8 @@ _deps
 wasm/test_page/node_modules
 build-wasm
 models
-wasm/test_page/js/bergamot-translator-worker.*
+wasm/module/worker/bergamot-translator-worker.*
+wasm/module/browsermt-bergamot-translator-*.tgz
 
 # VSCode
 .vscode
diff --git a/wasm/module/README.md b/wasm/module/README.md
new file mode 100644
index 000000000..4f6b153ba
--- /dev/null
+++ b/wasm/module/README.md
@@ -0,0 +1,238 @@
+# Installation
+
+```bash
+npm install @browsermt/bergamot-translator
+```
+
+# Quick start
+
+```js
+import {BatchTranslator} from "@browsermt/bergamot-translator/translator.js";
+
+const translator = new BatchTranslator();
+
+const response = await translator.translate({
+  from: "en",
+  to: "es",
+  text: "Hello <em>world</em>!",
+  html: true
+});
+
+console.log(response.target.text);
+
+// Stops worker threads
+translator.delete();
+```
+
+# Throughput vs Latency
+
+This package comes with two translator implementations:
+
+- [LatencyOptimisedTranslator](#latencyoptimisedtranslator) is more useful for an interactive session, say like Google Translate, where you're only working on translating one input at a time.
+- [BatchTranslator](#batchtranslator) is optimised for processing a large number of translations as fast as possible (but individual translations might take some time), e.g. translating a large number of strings or all paragraphs in a document.
+
+## LantencyOptimisedTranslator
+
+Translator best suited for interactive usage. Runs with a single worker thread and a batch-size of 1 to give you a response as quickly as possible. It will cancel any pending translations that aren't currently being processed if you submit a new one.
+
+```js
+const translator = new LatencyOptimisedTranslator({
+  pivotLanguage?: string?,
+  registryUrl?: string,
+  workerUrl?: string,
+  downloadTimeout?: number,
+  cacheSize?: number,
+  useNativeIntGemm?: boolean,
+})
+```
+
+- `pivotLanguage` - language code for the language to use as an intermediate if there is no direct translation model available. Defaults to `"en"`. Set to `null` to disable pivoting.
+- `registryUrl` - url to a list of models and their paths. Defaults to `https://storage.googleapis.com/bergamot-models-sandbox/0.3.3/registry.json`.
+- `workerUrl` - url to `translator-worker.js`. Defaults to `"worker/translator-worker.js"` relative to the path of `translator.js`.
+- `downloadTimeout` - Maximum time we're attempting to download model files before failing. Defaults to `60000` or 60 seconds. Set to `0` to disable.
+- `cacheSize` - Maximum number of sentences in kept translation cache (per worker, workers do not share their cache). This is an ideal maximum as it is a hash-map, in practice about 1/3th is occupied. If set to `0`, translation cache is disabled (the default).
+- `useNativeIntGemm` - Try to link to native IntGEMM implementation when loading the WASM binary. This is only implemented in the privileged extension context of Firefox Nightly. If it fails, it will always fall back to the included implementation. Defaults to `false`.
+
+### translate()
+
+```js
+const {request, target: {text:string}} = await translator.translate({
+  from: string,
+  to: string,
+  text: string,
+  html?: boolean,
+  qualityScores?: boolean
+})
+```
+
+Submits a translation request. Multiple of these are processed in a batch. A batch will be started the next tick (if there is a worker available).
+
+- `from` - language code of the source language, e.g. `"de"`
+- `to` - language code of the target language, e.g. `"en"`
+- `text` - string of text to translate, e.g. `"Hallo Welt!"`
+- `html` - boolean indicating whether `text` contains just plain text or HTML
+- `qualityScores` - whether to calculate quality scores. Not all models support this, and you need to load a separate quality scores model file for it. Quality scores are returned as `<font x-bergamot-sentence-quality="">` and `<font x-bergamot-word-quality="">` wrapped around sentences and words in the output. When enabled, the output is always HTML, regardless of whether the input was.
+
+Returns:
+
+A promise to a translation response object, with `target.text` being the text or HTML of the translated output, and `request` a reference to the original translation request.
+
+### delete()
+
+```js
+translator.delete()
+```
+
+Cancels all pending requests with a `CancelledError` and terminates the worker immediately. This will free all the resources used.
+
+In a nodejs context you'll need to call this, otherwise your script won't exit because the translator will still be listening for messages from the worker.
+
+## BatchTranslator
+
+```js
+const translator = new BatchTranslator({
+  pivotLanguage?: string?,
+  registryUrl?: string,
+  workerUrl?: string,
+  downloadTimeout?: number,
+  cacheSize?: number,
+  useNativeIntGemm?: boolean,
+  workers?: number,
+  batchSize?: number,
+})
+```
+
+General translator options:
+
+See [LatencyOptimisedTranslator](#latencyoptimisedtranslator).
+
+BatchTranslator-specific options:
+
+- `workers` - Number of worker threads. These are full-on instances of the translator, with their own copy of the model loaded. This is an upper bound. If not that many workers can be fed, it won't create new ones. Minimally 1. Default is `1`.
+- `batchSize` - Number of translation requests per batch. All sentences from all translation requests are packed into a bunch of matrix operations. With a larger batch size the translator has more material to find ideal sets of sentences for filling the matrix. However, you'll only get the results for each of the requests in a batch once the whole batch is finished. Defaults to 8.
+
+### translate()
+
+```js
+const {target: {text:string}} = await translator.translate({
+  from: string,
+  to: string,
+  text: string,
+  html?: boolean,
+  qualityScores?: boolean,
+  priority?: number
+})
+```
+
+Submits a translation request. Multiple of these are processed in a batch. A batch will be started the next tick (if there is a worker available).
+
+- (See [LatencyOptimisedTranslator.translate()](#translate) for most options)
+- `priority` - When grouping translation requests into batches to give to workers, requests with a lower number are considered first. For example, if you're translating a web page, you can give requests of parts that are in the current frame a lower number to make sure they're processed first.
+
+### remove()
+
+```js
+translator.remove(request => {
+  // true deletes the request from the queue.
+  return true;
+})
+```
+
+Removes requests from the translation queue, i.e. only when they haven't been sent to a worker yet.
+
+The filter function should return true-ish for each request that should be cancelled. Their promises are rejected with a `CancelledError` error.
+
+
+### delete()
+
+```js
+translator.delete()
+```
+
+Cancels all pending requests with a `CancelledError` and terminates all workers immediately. This will free all the resources used.
+
+
+# Models
+
+Both translators accept a `backing` option, which tells it where to get model data and the translation engine implementation from. They default to using `BergamotTranslator` which gets its models from the same repository as [firefox-translations](https://github.com/mozilla/firefox-translations).
+
+To customize the model, reimplement the `loadModelRegistry` and `loadTranslationModel` methods.
+
+`loadModelRegistry()` has the hard requirement to return a promise to a list that looks like `{from: string, to: string, ...}[]`. The `from` and `to` keys are used as key for model selection.
+
+`loadTranslationModel()` should return a promise with ArrayBuffers for `model`, `shortlist`, `vocabs`, and optionally `qualityModel`. It can include a `config` object as well.
+
+Example of an alternative implementation that loads models from data.statmt.org, i.e. the same as [translateLocally](https://translateLocally.com):
+
+```js
+class CustomBacking extends TranslatorBacking {
+    async loadModelRegistery() {
+        const response = await fetch('https://translatelocally.com/models.json');
+        const {models} = await response.json();
+
+        // Add 'from' and 'to' keys for each model. Since theoretically a model
+        // can have multiple froms keys in TranslateLocally, we do a little
+        // product here.
+        return models.reduce((list, model) => {
+            try {
+                const to = first(Intl.getCanonicalLocales(model.trgTag));
+                for (let from of Intl.getCanonicalLocales(Object.keys(model.srcTags))) {
+                    list.push({from, to, model});
+                }
+            } catch (err) {
+                console.log('Skipped model', model, 'because', err);
+            }
+
+            return list;
+        }, []);
+    }
+
+    async loadTranslationModel({from, to}) {
+        // Find that model in the registry which will tell us about its files
+        const entries = (await this.registry).filter(model => model.from === from && model.to === to);
+
+        // Prefer tiny models above non-tiny ones
+        entries.sort(({model: a}, {model: b}) => (a.shortName.indexOf('tiny') === -1 ? 1 : 0) - (b.shortName.indexOf('tiny') === -1 ? 1 : 0));
+
+        if (!entries)
+            throw new Error(`No model for '${from}' -> '${to}'`);
+
+        const entry = entries[0].model;
+
+        const response = await fetch(entry.url, {
+            integrity: `sha256-${entry.checksum}`
+        });
+
+        // pako from https://www.npmjs.com/package/pako
+        const archive = pako.inflate(await response.arrayBuffer());
+
+        // untar from https://www.npmjs.com/package/js-untar
+        const files = await untar(archive.buffer);
+
+        const find = (filename) => {
+            const found = files.find(file => file.name.match(/(?:^|\/)([^\/]+)$/)[1] === filename)
+            if (found === undefined)
+                throw new Error(`Could not find '${filename}' in model archive`);
+            return found;
+        };
+
+        // YAML.parse is found in worker/translator-worker.js
+        const config = YAML.parse(find('config.intgemm8bitalpha.yml').readAsString());
+
+        const model = find(config.models[0]).buffer;
+
+        const vocabs = config.vocabs.map(vocab => find(vocab).buffer);
+
+        const shortlist = find(config.shortlist[0]).buffer;
+
+        // Return the buffers
+        return {model, vocabs, shortlist, config};
+    }
+}
+
+const translator = new BatchTranslator(options, new CustomBacking(options));
+```
+
+# Supported languages
+
+See https://github.com/mozilla/firefox-translations-models#currently-supported-languages. You may need to set the `registryUrl` option to point to the latest release.
\ No newline at end of file
diff --git a/wasm/module/main.js b/wasm/module/main.js
new file mode 100644
index 000000000..d712a2199
--- /dev/null
+++ b/wasm/module/main.js
@@ -0,0 +1,21 @@
+import * as readline from 'node:readline/promises';
+import {stdin, stdout} from 'node:process';
+import {BatchTranslator} from "./translator.js";
+
+const rl = readline.createInterface({input: stdin, output: stdout});
+
+const translator = new BatchTranslator();
+
+for await (const line of rl) {
+	const response = await translator.translate({
+		from: "en",
+		to: "es",
+		text: line,
+		html: false,
+		qualityScores: false
+	});
+
+	console.log(response.target.text);
+}
+
+translator.delete();
diff --git a/wasm/module/package.json b/wasm/module/package.json
new file mode 100644
index 000000000..f30464665
--- /dev/null
+++ b/wasm/module/package.json
@@ -0,0 +1,39 @@
+{
+  "name": "@browsermt/bergamot-translator",
+  "version": "0.4.9",
+  "description": "Cross platform C++ library focusing on optimized machine translation on the consumer-grade device.",
+  "homepage": "https://github.com/browsermt/bergamot-translator#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+ssh://git@github.com/browsermt/bergamot-translator.git"
+  },
+  "keywords": [
+    "machine",
+    "translation"
+  ],
+  "author": "",
+  "license": "MPL-2.0",
+  "bugs": {
+    "url": "https://github.com/browsermt/bergamot-translator/issues"
+  },
+  "type": "module",
+  "main": "translator.js",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "files": [
+    "worker/bergamot-translator-worker.js",
+    "worker/bergamot-translator-worker.wasm",
+    "worker/translator-worker.js",
+    "translator.js",
+    "main.js"
+  ],
+  "config": {
+    "emscripten_version": "3.1.8"
+  },
+  "scripts": {
+    "prepare": "test -f worker/bergamot-translator-worker.wasm || npm run build",
+    "build": "mkdir -p ../../build-wasm && docker run --rm -v $(realpath ../../):/src -v $(realpath ../../build-wasm):/build -v $(pwd)/worker:/dst -w /build emscripten/emsdk:$npm_package_config_emscripten_version sh -c \"emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=off /src && emmake make -j2 && cp bergamot-translator-worker.wasm bergamot-translator-worker.js /dst\"",
+    "test": "echo \"Hello world!\" | node main.js"
+  }
+}
diff --git a/wasm/module/translator.js b/wasm/module/translator.js
new file mode 100644
index 000000000..f27c07653
--- /dev/null
+++ b/wasm/module/translator.js
@@ -0,0 +1,879 @@
+/**
+ * @typedef {Object} TranslationRequest
+ * @property {String} from
+ * @property {String} to
+ * @property {String} text
+ * @property {Boolean} html
+ * @property {Integer?} priority
+ */
+
+/**
+ * @typedef {Object} TranslationResponse
+ * @property {TranslationRequest} request
+ * @property {{text: string}} target
+ */
+
+/**
+ * NodeJS compatibility, a thin WebWorker layer around node:worker_threads.
+ */
+if (!(typeof window !== 'undefined' && window.Worker)) {
+    globalThis.Worker = class {
+        #worker;
+
+        constructor(url) {
+            this.#worker = new Promise(async (accept) => {
+                const {Worker} = await import(/* webpackIgnore: true */ 'node:worker_threads');
+                accept(new Worker(url));
+            });
+        }
+
+        addEventListener(eventName, callback) {
+            this.#worker.then(worker => worker.on(eventName, (data) => callback({data})));
+        }
+
+        postMessage(message) {
+            this.#worker.then(worker => worker.postMessage(message));
+        }
+
+        terminate() {
+            this.#worker.then(worker => worker.terminate());
+        }
+    }
+}
+
+/**
+ * Thrown when a pending translation is replaced by another newer pending
+ * translation.
+ */
+export class SupersededError extends Error {}
+
+
+/**
+ * Thrown when a translation was removed from the queue.
+ */
+export class CancelledError extends Error {}
+
+
+/**
+ * Wrapper around bergamot-translator loading and model management.
+ */
+ export class TranslatorBacking {
+    
+    /**
+     * @param {{
+     *  cacheSize?: number,
+     *  useNativeIntGemm?: boolean,
+     *  downloadTimeout?: number,
+     *  registryUrl?: string
+     *  pivotLanguage?: string?
+     *  onerror?: (err: Error)
+     * }} options
+     */
+    constructor(options) {
+        this.options = options || {};
+
+        this.registryUrl = this.options.registryUrl || 'https://bergamot.s3.amazonaws.com/models/index.json';
+
+        this.downloadTimeout = 'downloadTimeout' in this.options ? parseInt(this.options.downloadTimeout) : 60000;
+
+        /**
+         * registry of all available models and their urls
+         * @type {Promise<Model[]>}
+         */
+        this.registry = this.loadModelRegistery();
+
+        /**
+         * Map of downloaded model data files as buffers per model.
+         * @type {Map<{from:string,to:string}, Promise<Map<string,ArrayBuffer>>>}
+         */
+        this.buffers = new Map();
+
+        /**
+         * @type {string?}
+         */
+        this.pivotLanguage = 'pivotLanguage' in this.options ? options.pivotLanguage : 'en';
+        
+        /**
+         * A map of language-pairs to a list of models you need for it.
+         * @type {Map<{from:string,to:string}, Promise<{from:string,to:string}[]>>}
+         */
+        this.models = new Map();
+
+        /**
+         * Error handler for all errors that are async, not tied to a specific
+         * call and that are unrecoverable.
+         * @type {(error: Error)}
+         */
+        this.onerror = this.options.onerror || (err => console.error('WASM Translation Worker error:', err));
+    }
+
+    /**
+     * Loads a worker thread, and wraps it in a message passing proxy. I.e. it
+     * exposes the entire interface of TranslationWorker here, and all calls
+     * to it are async. Do note that you can only pass arguments that survive
+     * being copied into a message. 
+     * @return {Promise<{worker:Worker, exports:Proxy<TranslationWorker>}>}
+     */
+    async loadWorker() {
+        const worker = new Worker(new URL('./worker/translator-worker.js', import.meta.url));
+
+        /**
+         * Incremental counter to derive request/response ids from.
+         */
+        let serial = 0;
+
+        /**
+         * Map of pending requests
+         * @type {Map<number,{accept:(any), reject:(Error)}>}
+         */
+        const pending = new Map();
+
+        // Function to send requests
+        const call = (name, ...args) => new Promise((accept, reject) => {
+            const id = ++serial;
+            pending.set(id, {
+                accept,
+                reject,
+                callsite: { // for debugging which call caused the error
+                    message: `${name}(${args.map(arg => String(arg)).join(', ')})`,
+                    stack: new Error().stack
+                }
+            });
+            worker.postMessage({id, name, args});
+        });
+
+        // … receive responses
+        worker.addEventListener('message', function({data: {id, result, error}}) {
+            if (!pending.has(id)) {
+                console.debug('Received message with unknown id:', arguments[0]);
+                throw new Error(`BergamotTranslator received response from worker to unknown call '${id}'`);
+            }
+
+            const {accept, reject, callsite} = pending.get(id);
+            pending.delete(id);
+
+            if (error !== undefined)
+                reject(Object.assign(new Error(), error, {
+                    message: error.message + ` (response to ${callsite.message})`,
+                    stack: error.stack ? `${error.stack}\n${callsite.stack}` : callsite.stack
+                }));
+            else
+                accept(result);
+        });
+
+        // … and general errors
+        worker.addEventListener('error', this.onerror.bind(this));
+
+        // Await initialisation. This will also nicely error out if the WASM
+        // runtime fails to load.
+        await call('initialize', this.options);
+
+        /**
+         * Little wrapper around the message passing api of Worker to make it
+         * easy to await a response to a sent message. This wraps the worker in
+         * a Proxy so you can treat it as if it is an instance of the
+         * TranslationWorker class that lives inside the worker. All function
+         * calls to it are transparently passed through the message passing
+         * channel.
+         */
+        return {
+            worker,
+            exports: new Proxy({}, {
+                get(target, name, receiver) {
+                    // Prevent this object from being marked "then-able"
+                    if (name !== 'then')
+                        return (...args) => call(name, ...args);
+                }
+            })
+        };
+    }
+
+    /**
+     * Loads the model registry. Uses the registry shipped with this extension,
+     * but formatted a bit easier to use, and future-proofed to be swapped out
+     * with a TranslateLocally type registry.
+     * @return {Promise<{
+     *   from: string,
+     *   to: string,
+     *   files: {
+     *     [part:string]: {
+     *       name: string,
+     *       size: number,
+     *       expectedSha256Hash: string
+     *     }
+     *   }[]
+     * }>}
+     */
+    async loadModelRegistery() {
+        const response = await fetch(this.registryUrl, {credentials: 'omit'});
+        const registry = await response.json();
+
+        // Add 'from' and 'to' keys for each model.
+        return Array.from(Object.entries(registry), ([key, files]) => {
+            return {
+                from: key.substring(0, 2),
+                to: key.substring(2, 4),
+                files
+            }
+        });
+    }
+
+    /**
+     * Gets or loads translation model data. Caching wrapper around
+     * `loadTranslationModel()`.
+     * @param {{from:string, to:string}}
+     * @return {Promise<{
+     *   model: ArrayBuffer,
+     *   vocab: ArrayBuffer,
+     *   shortlist: ArrayBuffer,
+     *   qualityModel: ArrayBuffer?
+     * }>}
+     */
+    getTranslationModel({from, to}, options) {
+        const key = JSON.stringify({from, to});
+
+        if (!this.buffers.has(key)) {
+            const promise = this.loadTranslationModel({from, to}, options);
+
+            // set the promise so we return the same promise when its still pending
+            this.buffers.set(key, promise);
+
+            // But if loading fails, remove the promise again so we can try again later
+            promise.catch(err => this.buffers.delete(key))
+        }
+
+        return this.buffers.get(key);
+    }
+
+    /**
+     * Downloads a translation model and returns a set of
+     * ArrayBuffers. These can then be passed to a TranslationWorker thread
+     * to instantiate a TranslationModel inside the WASM vm.
+     * @param {{from:string, to:string}}
+     * @param {{signal:AbortSignal?}?}
+     * @return {Promise<{
+     *   model: ArrayBuffer,
+     *   vocab: ArrayBuffer,
+     *   shortlist: ArrayBuffer,
+     *   qualityModel: ArrayBuffer?
+     *   config: string?
+     * }>}
+     */
+    async loadTranslationModel({from, to}, options) {
+        performance.mark(`loadTranslationModule.${JSON.stringify({from, to})}`);
+
+        // Find that model in the registry which will tell us about its files
+        const entries = (await this.registry).filter(model => model.from == from && model.to == to);
+
+        if (!entries)
+            throw new Error(`No model for '${from}' -> '${to}'`);
+
+        const files = entries[0].files;
+
+        const abort = () => reject(new CancelledError('abort signal'));
+
+        // Promise that resolves (or rejects really) when the abort signal hits
+        const escape = new Promise((accept, reject) => {
+            if (options?.signal)
+                options.signal.addEventListener('abort', abort);
+        });
+
+        // Download all files mentioned in the registry entry. Race the promise
+        // of all fetch requests, and a promise that rejects on the abort signal
+        const buffers = Object.fromEntries(await Promise.race([
+            Promise.all(Object.entries(files).map(async ([part, file]) => {
+                // Special case where qualityModel is not part of the model, and this
+                // should also catch the `config` case.
+                if (file === undefined || file.name === undefined)
+                    return [part, null];
+
+                try {
+                    return [part, await this.fetch(file.name, file.expectedSha256Hash, options)];
+                } catch (cause) {
+                    throw new Error(`Could not fetch ${file.name} for ${from}->${to} model`, {cause});
+                }
+            })),
+            escape
+        ]));
+
+        // Nothing to abort now, clean up abort promise
+        if (options?.signal)
+            options.signal.removeEventListener('abort', abort);
+
+        performance.measure('loadTranslationModel', `loadTranslationModule.${JSON.stringify({from, to})}`);
+
+        let vocabs = [];
+
+        if (buffers.vocab)
+            vocabs = [buffers.vocab]
+        else if (buffers.trgvocab && buffers.srcvocab)
+            vocabs = [buffers.srcvocab, buffers.trgvocab]
+        else
+            throw new Error(`Could not identify vocab files for ${from}->${to} model among: ${Array.from(Object.keys(files)).join(' ')}`);
+
+        let config = {};
+
+        // For the Ukrainian models we need to override the gemm-precision
+        if (files.model.name.endsWith('intgemm8.bin'))
+            config['gemm-precision'] = 'int8shiftAll';
+
+        // If quality estimation is used, we need to turn off skip-cost. Turning
+        // this off causes quite the slowdown.
+        if (files.qualityModel)
+            config['skip-cost'] = false;
+
+        // Allow the registry to also specify marian configuration parameters
+        if (files.config)
+            Object.assign(config, files.config);
+
+        // Translate to generic bergamot-translator format that also supports
+        // separate vocabularies for input & output language, and calls 'lex'
+        // a more descriptive 'shortlist'.
+        return {
+            model: buffers.model,
+            shortlist: buffers.lex,
+            vocabs,
+            qualityModel: buffers.qualityModel,
+            config
+        };
+    }
+
+    /**
+     * Helper to download file from the web. Verifies the checksum.
+     * @param {string} url
+     * @param {string?} checksum sha256 checksum as hexadecimal string
+     * @param {{signal:AbortSignal}?} extra fetch options
+     * @returns {Promise<ArrayBuffer>}
+     */
+    async fetch(url, checksum, extra) {
+        // Rig up a timeout cancel signal for our fetch
+        const controller = new AbortController();
+        const abort = () => controller.abort();
+
+        const timeout = this.downloadTimeout ? setTimeout(abort, this.downloadTimeout) : null;
+
+        try {
+            // Also maintain the original abort signal
+            if (extra?.signal)
+                extra.signal.addEventListener('abort', abort);
+
+            const options = {
+                credentials: 'omit',
+                signal: controller.signal,
+            };
+
+            if (checksum)
+                options['integrity'] = `sha256-${this.hexToBase64(checksum)}`;
+
+            // Disable the integrity check for NodeJS because of
+            // https://github.com/nodejs/undici/issues/1594
+            if (typeof window === 'undefined')
+                delete options['integrity'];
+
+            // Start downloading the url, using the hex checksum to ask
+            // `fetch()` to verify the download using subresource integrity 
+            const response = await fetch(url, options);
+
+            // Finish downloading (or crash due to timeout)
+            return await response.arrayBuffer();
+
+        } finally {
+            if (timeout)
+                clearTimeout(timeout);
+
+            if (extra?.signal)
+                extra.signal.removeEventListener('abort', abort);
+        }
+    }
+
+    /**
+     * Converts the hexadecimal hashes from the registry to something we can use with
+     * the fetch() method.
+     */
+    hexToBase64(hexstring) {
+        return btoa(hexstring.match(/\w{2}/g).map(function(a) {
+            return String.fromCharCode(parseInt(a, 16));
+        }).join(""));
+    }
+
+    /**
+     * Crappy named method that gives you a list of models to translate from
+     * one language into the other. Generally this will be the same as you
+     * just put in if there is a direct model, but it could return a list of
+     * two models if you need to pivot through a third language.
+     * Returns just [{from:str,to:str}...]. To be used something like this:
+     * ```
+     * const models = await this.getModels(from, to);
+     * models.forEach(({from, to}) => {
+     *   const buffers = await this.loadTranslationModel({from,to});
+     *   [TranslationWorker].loadTranslationModel({from,to}, buffers)
+     * });
+     * ```
+     * @returns {Promise<TranslationModel[]>}
+     */
+    getModels({from, to}) {
+        const key = JSON.stringify({from, to});
+
+        // Note that the `this.models` map stores Promises. This so that
+        // multiple calls to `getModels` that ask for the same model will
+        // return the same promise, and the actual lookup is only done once.
+        // The lookup is async because we need to await `this.registry`
+        if (!this.models.has(key))
+            this.models.set(key, this.findModels(from, to));
+
+        return this.models.get(key);
+    }
+
+    /**
+     * Find model (or model pair) to translate from `from` to `to`.
+     * @param {string} from
+     * @param {string} to
+     * @returns {Promise<TranslationModel[]>}
+     */
+    async findModels(from, to) {
+        const registry = await this.registry;
+
+        let direct = [], outbound = [], inbound = [];
+
+        registry.forEach(model => {
+            if (model.from === from && model.to === to)
+                direct.push(model);
+            else if (model.from === from && model.to === this.pivotLanguage)
+                outbound.push(model);
+            else if (model.to === to && model.from === this.pivotLanguage)
+                inbound.push(model);
+        });
+
+        if (direct.length)
+            return [direct[0]];
+
+        if (outbound.length && inbound.length)
+            return [outbound[0], inbound[0]];
+
+        throw new Error(`No model available to translate from '${from}' to '${to}'`);
+    }
+}
+
+/**
+ * Translator balancing between throughput and latency. Can use multiple worker
+ * threads.
+ */
+export class BatchTranslator {
+    /**
+     * @param {{
+     *  cacheSize?: number,
+     *  useNativeIntGemm?: boolean,
+     *  workers?: number,
+     *  batchSize?: number,
+     *  downloadTimeout?: number,
+     *  workerUrl?: string,
+     *  registryUrl?: string
+     *  pivotLanguage?: string?
+     * }} options
+     */
+    constructor(options, backing) {
+        if (!backing)
+            backing = new TranslatorBacking(options);
+
+        this.backing = backing;
+
+        /**
+         * @type {Array<{idle:Boolean, worker:Proxy}>} List of active workers
+         * (and a flag to mark them idle or not)
+         */
+        this.workers = [];
+
+        /**
+         * Maximum number of workers
+         * @type {number} 
+         */
+        this.workerLimit = Math.max(options?.workers || 0, 1);
+
+        /**
+         * List of batches we push() to & shift() from using `enqueue`.
+         * @type {{
+         *    id: number,
+         *    key: string,
+         *    priority: number,
+         *    models: TranslationModel[],
+         *    requests: Array<{
+         *      request: TranslationRequest,
+         *      resolve: (response: TranslationResponse),
+         *      reject: (error: Error)
+         *    }>
+         * }}
+         */
+        this.queue = [];
+
+        /**
+         * batch serial to help keep track of batches when debugging
+         * @type {Number}
+         */
+        this.batchSerial = 0;
+
+        /**
+         * Number of requests in a batch before it is ready to be translated in
+         * a single call. Bigger is better for throughput (better matrix packing)
+         * but worse for latency since you'll have to wait for the entire batch
+         * to be translated.
+         * @type {Number}
+         */
+        this.batchSize = Math.max(options?.batchSize || 8, 1);
+
+        this.onerror = options?.onerror || (err => console.error('WASM Translation Worker error:', err));
+    }
+    
+    /**
+     * Destructor that stops and cleans up.
+     */
+    async delete() {
+        // Empty the queue
+        this.remove(() => true);
+
+        // Terminate the workers
+        this.workers.forEach(({worker}) => worker.terminate());
+    }
+
+    /**
+     * Makes sure queued work gets send to a worker. Will delay it till `idle`
+     * to make sure the batches have been filled to some degree. Will keep
+     * calling itself as long as there is work in the queue, but it does not
+     * hurt to call it multiple times. This function always returns immediately.
+     */
+    notify() {
+        setTimeout(async () => {
+            // Is there work to be done?
+            if (!this.queue.length)
+                return;
+
+            // Find an idle worker
+            let worker = this.workers.find(worker => worker.idle);
+
+            // No worker free, but space for more?
+            if (!worker && this.workers.length < this.workerLimit) {
+                try {
+                    // Claim a place in the workers array (but mark it busy so
+                    // it doesn't get used by any other `notify()` calls).
+                    const placeholder = {idle: false};
+                    this.workers.push(placeholder);
+
+                    // adds `worker` and `exports` props
+                    Object.assign(placeholder, await this.backing.loadWorker());
+
+                    // At this point we know our new worker will be usable.
+                    worker = placeholder;
+                } catch (e) {
+                    this.onerror(new Error(`Could not initialise translation worker: ${e.message}`));
+                }
+            }
+
+            // If no worker, that's the end of it.
+            if (!worker)
+                return;
+
+            // Up to this point, this function has not used await, so no
+            // chance that another call stole our batch since we did the check
+            // at the beginning of this function and JavaScript is only
+            // cooperatively parallel.
+            const batch = this.queue.shift();
+
+            // Put this worker to work, marking as busy
+            worker.idle = false;
+            try {
+                await this.consumeBatch(batch, worker.exports);
+            } catch (e) {
+                batch.requests.forEach(({reject}) => reject(e));
+            }
+            worker.idle = true;
+
+            // Is there more work to be done? Do another idleRequest
+            if (this.queue.length)
+                this.notify();
+        });
+    }
+
+    /**
+     * The only real public call you need!
+     * ```
+     * const {target: {text:string}} = await this.translate({
+     *   from: 'de',
+     *   to: 'en',
+     *   text: 'Hallo Welt!',
+     *   html: false, // optional
+     *   priority: 0 // optional, like `nice` lower numbers are translated first
+     * })
+     * ```
+     * @param {TranslationRequest} request
+     * @returns {Promise<TranslationResponse>}
+     */
+    translate(request) {
+        const {from, to, priority} = request;
+
+        return new Promise(async (resolve, reject) => {
+            try {
+                // Batching key: only requests with the same key can be batched
+                // together. Think same translation model, same options.
+                const key = JSON.stringify({from, to});
+
+                // (Fetching models first because if we would do it between looking
+                // for a batch and making a new one, we end up with a race condition.)
+                const models = await this.backing.getModels(request);
+                
+                // Put the request and its callbacks into a fitting batch
+                this.enqueue({key, models, request, resolve, reject, priority});
+
+                // Tell a worker to pick up the work at some point.
+                this.notify();
+            } catch (e) {
+                reject(e);
+            }
+        });
+    }
+
+    /**
+     * Prune pending requests by testing each one of them to whether they're
+     * still relevant. Used to prune translation requests from tabs that got
+     * closed.
+     * @param {(request:TranslationRequest) => boolean} filter evaluates to true if request should be removed
+     */
+    remove(filter) {
+        const queue = this.queue;
+
+        this.queue = [];
+
+        queue.forEach(batch => {
+            batch.requests.forEach(({request, resolve, reject}) => {
+                if (filter(request)) {
+                    // Add error.request property to match response.request for
+                    // a resolve() callback. Pretty useful if you don't want to
+                    // do all kinds of Funcion.bind() dances.
+                    reject(Object.assign(new CancelledError('removed by filter'), {request}));
+                    return;
+                }
+
+                this.enqueue({
+                    key: batch.key,
+                    priority: batch.priority,
+                    models: batch.models,
+                    request,
+                    resolve,
+                    reject
+                });
+            });
+        });
+    }
+
+    /**
+     * Internal function used to put a request in a batch that still has space.
+     * Also responsible for keeping the batches in order of priority. Called by
+     * `translate()` but also used when filtering pending requests.
+     * @param {{request:TranslateRequest, models:TranslationModel[], key:String, priority:Number?, resolve:(TranslateResponse)=>any, reject:(Error)=>any}}
+     */
+    enqueue({key, models, request, resolve, reject, priority}) {
+        if (priority === undefined)
+            priority = 0;
+         // Find a batch in the queue that we can add to
+         // (TODO: can we search backwards? that would speed things up)
+        let batch = this.queue.find(batch => {
+            return batch.key === key
+                && batch.priority === priority
+                && batch.requests.length < this.batchSize
+        });
+
+        // No batch or full batch? Queue up a new one
+        if (!batch) {
+            batch = {id: ++this.batchSerial, key, priority, models, requests: []};
+            this.queue.push(batch);
+            this.queue.sort((a, b) => a.priority - b.priority);
+        }
+
+        batch.requests.push({request, resolve, reject});
+    }
+
+    /**
+     * Internal method that uses a worker thread to process a batch. You can
+     * wait for the batch to be done by awaiting this call. You should only
+     * then reuse the worker otherwise you'll just clog up its message queue.
+     */
+    async consumeBatch(batch, worker) {
+        performance.mark('BergamotBatchTranslator.start');
+
+        // Make sure the worker has all necessary models loaded. If not, tell it
+        // first to load them.
+        await Promise.all(batch.models.map(async ({from, to}) => {
+            if (!await worker.hasTranslationModel({from, to})) {
+                const buffers = await this.backing.getTranslationModel({from, to});
+                await worker.loadTranslationModel({from, to}, buffers);
+            }
+        }));
+
+        // Call the worker to translate. Only sending the actually necessary
+        // parts of the batch to avoid trying to send things that don't survive
+        // the message passing API between this thread and the worker thread.
+        const responses = await worker.translate({
+            models: batch.models.map(({from, to}) => ({from, to})),
+            texts: batch.requests.map(({request: {text, html, qualityScores}}) => ({
+                text: text.toString(),
+                html: !!html,
+                qualityScores: !!qualityScores
+            }))
+        });
+
+        // Responses are in! Connect them back to their requests and call their
+        // callbacks.
+        batch.requests.forEach(({request, resolve, reject}, i) => {
+            // TODO: look at response.ok and reject() if it is false
+            resolve({
+                request, // Include request for easy reference? Will allow you
+                         // to specify custom properties and use that to link
+                         // request & response back to each other.
+                ...responses[i] // {target: {text: String}}
+            });
+        });
+        
+        performance.measure('BergamotBatchTranslator', 'BergamotBatchTranslator.start');
+    }
+}
+
+
+/**
+ * Translator optimised for interactive use.
+ */
+export class LatencyOptimisedTranslator {
+    /**
+     * @type {TranslatorBacking}
+     */
+    backing;
+
+    /**
+     * @type {Promise<{idle:boolean, worker:Worker, exports:Proxy<TranslationWorker>}>}
+     */
+    worker;
+
+    /**
+     * @type {{request: TranslationRequest, accept:(TranslationResponse), reject:(Error)} | null}
+     */
+    pending;
+
+    /**
+     * @param {{
+     *  cacheSize?: number,
+     *  useNativeIntGemm?: boolean,
+     *  downloadTimeout?: number,
+     *  workerUrl?: string,
+     *  registryUrl?: string
+     *  pivotLanguage?: string?
+     * }} options
+     */
+    constructor(options, backing) {
+        if (!backing)
+            backing = new TranslatorBacking(options);
+
+        this.backing = backing;
+
+        // Exposing the this.loadWorker() returned promise through this.worker
+        // so that you can use that to catch any errors that happened during
+        // loading.
+        this.worker = this.backing.loadWorker().then(worker => ({...worker, idle:true}));
+    }
+
+    /**
+     * Destructor that stops and cleans up.
+     */
+    async delete() {
+        // Cancel pending translation
+        if (this.pending) {
+            this.pending.reject(new CancelledError('translator got deleted'));
+            this.pending = null;
+        }
+
+        // Terminate the worker (I don't care if this fails)
+        try {
+            const {worker} = await this.worker;
+            worker.terminate();
+        } finally {
+            this.worker = null;
+        }
+    }
+    
+    /**
+     * Sets `request` as the next translation to process. If there was already
+     * a translation waiting to be processed, their promise is rejected with a
+     * SupersededError.
+     * @param {TranslationRequest} request
+     * @return {Promise<TranslationResponse>}
+     */
+    translate(request, options) {
+        if (this.pending)
+            this.pending.reject(new SupersededError());
+        
+        return new Promise((accept, reject) => {
+            const pending = {request, accept, reject, options};
+
+            if (options?.signal) {
+                options.signal.addEventListener('abort', e => {
+                    reject(new CancelledError('abort signal'));
+                    if (this.pending === pending)
+                        this.pending = null;
+                });
+            }
+
+            this.pending = pending;
+            this.notify();
+        });
+    }
+    
+    notify() {
+        setTimeout(async () => {
+            if (!this.pending)
+                return;
+
+            // Catch errors such as the worker not working
+            try {
+                // Possibly wait for the worker to finish loading. After it loaded
+                // these calls are pretty much instantaneous.
+                const worker = await this.worker;
+
+                // Is another notify() call hogging the worker? Then stop.
+                if (!worker.idle)
+                    return;
+
+                // Claim the pending translation request.
+                const {request, accept, reject, options} = this.pending;
+                this.pending = null;
+
+                // Mark the worker as occupied
+                worker.idle = false;
+                    
+                try {
+                    const models = await this.backing.getModels(request)
+
+                    await Promise.all(models.map(async ({from, to}) => {
+                        if (!await worker.exports.hasTranslationModel({from, to})) {
+                            const buffers = await this.backing.getTranslationModel({from, to}, {signal: options?.signal});
+                            await worker.exports.loadTranslationModel({from, to}, buffers);
+                        }
+                    }));
+
+                    const {text, html, qualityScores} = request;
+                    const responses = await worker.exports.translate({
+                        models: models.map(({from,to}) => ({from, to})),
+                        texts: [{text, html, qualityScores}]
+                    });
+
+                    accept({request, ...responses[0]});
+                } catch (e) {
+                    reject(e);
+                }
+
+                worker.idle = true;
+
+                // Is there more work to be done? Do another idleRequest
+                if (this.pending)
+                    this.notify();
+            } catch (e) {
+                this.backing.onerror(e);
+            }
+        });
+    }
+}
diff --git a/wasm/module/worker/package.json b/wasm/module/worker/package.json
new file mode 100644
index 000000000..a2091cee3
--- /dev/null
+++ b/wasm/module/worker/package.json
@@ -0,0 +1,3 @@
+{
+	"type": "commonjs"
+}
\ No newline at end of file
diff --git a/wasm/module/worker/translator-worker.js b/wasm/module/worker/translator-worker.js
new file mode 100644
index 000000000..24ff29ad2
--- /dev/null
+++ b/wasm/module/worker/translator-worker.js
@@ -0,0 +1,475 @@
+/**
+ * Wrapper around the dirty bits of Bergamot's WASM bindings.
+ */
+
+// Global because importScripts is global.
+var Module = {};
+
+/**
+ * node.js compatibility: Fake GlobalWorkerScope that emulates being inside a
+ * WebWorker
+ */
+if (typeof self === 'undefined') {
+    global.Module = Module;
+
+    global.self = new class GlobalWorkerScope {
+        /** @type {import("node:worker_threads").MessagePort} */
+        #port;
+
+        constructor() {
+            const {parentPort} = require(/* webpackIgnore: true */ 'node:worker_threads');
+            this.#port = parentPort;
+        }
+
+        /**
+         * Add event listener to listen for messages posted to the worker.
+         * @param {string} eventName
+         * @param {(object)} callback
+         */
+        addEventListener(eventName, callback) {
+            this.#port.on(eventName, (data) => callback({data}));
+        }
+
+        /**
+         * Post message outside, to the owner of the Worker.
+         * @param {any} message
+         */
+        postMessage(message) {
+            this.#port.postMessage(message);
+        }
+
+        /**
+         * @param {...string} scripts - Paths to scripts to import in that order
+         */
+        importScripts(...scripts) {
+            const {readFileSync} = require(/* webpackIgnore: true */ 'node:fs');
+            const {join} = require(/* webpackIgnore: true */ 'node:path');
+            for (let pathname of scripts) {
+                const script = readFileSync(join(__dirname, pathname), {encoding: 'utf-8'});
+                eval.call(global, script);
+            }
+        }
+
+        /**
+         * Adds support for local file urls. Assumes anything that doesn't start
+         * with "http" to be a local path.
+         * @param {string} url - path or url
+         * @param {object?} options - See `fetch()` options
+         * @return {Promise<Response>}
+         */
+        async fetch(url, options) {
+            if (url.protocol === 'file:') {
+                const {readFile} = require(/* webpackIgnore: true */ 'node:fs/promises');
+                const buffer = await readFile(url.pathname);
+                const blob = new Blob([buffer]);
+                return new Response(blob, {
+                    status: 200,
+                    statusText: 'OK',
+                    headers: {
+                        'Content-Type': 'application/wasm',
+                        'Content-Length': blob.size.toString()
+                    }
+                });
+            }
+
+            return await fetch(url, options);
+        }
+
+        get location() {
+            return new URL(`file://${__filename}`);
+        }
+    }
+}
+
+class YAML {
+    /**
+     * Parses YAML into dictionary. Does not interpret types, all values are a
+     * string or a list of strings. No support for objects other than the top
+     * level.
+     * @param {string} yaml
+     * @return {{[string]: string | string[]}}
+     */
+    static parse(yaml) {
+        const out = {};
+
+        yaml.split('\n').reduce((key, line, i) => {
+            let match;
+            if (match = line.match(/^\s*-\s+(.+?)$/)) {
+                if (!Array.isArray(out[key]))
+                    out[key] = out[key].trim() ? [out[key]] : [];
+                out[key].push(match[1].trim());
+            }
+            else if (match = line.match(/^\s*([A-Za-z0-9_][A-Za-z0-9_-]*):\s*(.*)$/)) {
+                key = match[1];
+                out[key] = match[2].trim();
+            }
+            else if (!line.trim()) {
+                // whitespace, ignore
+            }
+            else {
+                throw Error(`Could not parse line ${i+1}: "${line}"`);
+            }
+            return key;
+        }, null);
+
+        return out;
+    }
+
+    /**
+     * Turns an object into a YAML string. No support for objects, only simple
+     * types and lists of simple types.
+     * @param {{[string]: string | number | boolean | string[]}} data
+     * @return {string}
+     */
+    static stringify(data) {
+        return Object.entries(data).reduce((str, [key, value]) => {
+            let valstr = '';
+            if (Array.isArray(value))
+                valstr = value.map(val => `\n  - ${val}`).join('');
+            else if (typeof value === 'number' || typeof value === 'boolean' || value.match(/^\d*(\.\d+)?$/))
+                valstr = `${value}`;
+            else
+                valstr = `${value}`; // Quote?
+
+            return `${str}${key}: ${valstr}\n`;
+        }, '');
+    }
+}
+
+/**
+ * Wrapper around the bergamot-translator exported module that hides the need
+ * of working with C++ style data structures and does model management.
+ */
+class BergamotTranslatorWorker {
+    /**
+     * Map of expected symbol -> name of fallback symbol for functions that can
+     * be swizzled for a faster implementation. Firefox Nightly makes use of
+     * this.
+     */
+    static GEMM_TO_FALLBACK_FUNCTIONS_MAP = {
+        'int8_prepare_a': 'int8PrepareAFallback',
+        'int8_prepare_b': 'int8PrepareBFallback',
+        'int8_prepare_b_from_transposed': 'int8PrepareBFromTransposedFallback',
+        'int8_prepare_b_from_quantized_transposed': 'int8PrepareBFromQuantizedTransposedFallback',
+        'int8_prepare_bias': 'int8PrepareBiasFallback',
+        'int8_multiply_and_add_bias': 'int8MultiplyAndAddBiasFallback',
+        'int8_select_columns_of_b': 'int8SelectColumnsOfBFallback'
+    };
+
+    /**
+     * Name of module exported by Firefox Nightly that exports an optimised
+     * implementation of the symbols mentioned above.
+     */
+    static NATIVE_INT_GEMM = 'mozIntGemm';
+
+    /**
+     * Empty because we can't do async constructors yet. It is the
+     * responsibility of whoever owns this WebWorker to call `initialize()`.
+     */
+    constructor(options) {}
+
+    /**
+     * Instantiates a new translation worker with optional options object.
+     * If this call succeeds, the WASM runtime is loaded and ready.
+     * 
+     * Available options are:
+     *   useNativeIntGemm: {true | false} defaults to false. If true, it will
+     *                     attempt to link to the intgemm module available in
+     *                     Firefox Nightly which makes translations much faster.
+     *          cacheSize: {Number} defaults to 0 which disables translation
+     *                     cache entirely. Note that this is a theoretical
+     *                     upper bound. In practice it will use about 1/3th of
+     *                     the cache specified here. 2^14 is not a bad starting
+     *                     value.
+     * @param {{useNativeIntGemm: boolean, cacheSize: number}} options
+     */
+    async initialize(options) {
+        this.options = options || {};
+        this.models = new Map(); // Map<str,Promise<TranslationModel>>
+        this.module = await this.loadModule();
+        this.service = await this.loadTranslationService();
+    }
+
+    /**
+     * Tries to load native IntGEMM module for bergamot-translator. If that
+     * fails because it or any of the expected functions is not available, it
+     * falls back to using the naive implementations that come with the wasm
+     * binary itself through `linkFallbackIntGemm()`.
+     * @param {{env: {memory: WebAssembly.Memory}}} info
+     * @return {{[method:string]: (...any) => any}}
+     */
+    linkNativeIntGemm(info) {
+        if (!WebAssembly['mozIntGemm']) {
+            console.warn('Native gemm requested but not available, falling back to embedded gemm');
+            return this.linkFallbackIntGemm(info);
+        }
+
+        const instance = new WebAssembly.Instance(WebAssembly['mozIntGemm'](), {
+            '': {memory: info['env']['memory']}
+        });
+
+        if (!Array.from(Object.keys(BergamotTranslatorWorker.GEMM_TO_FALLBACK_FUNCTIONS_MAP)).every(fun => instance.exports[fun])) {
+            console.warn('Native gemm is missing expected functions, falling back to embedded gemm');
+            return this.linkFallbackIntGemm(info);
+        }
+
+        return instance.exports;
+    }
+
+    /**
+     * Links intgemm functions that are already available in the wasm binary,
+     * but just exports them under the name that is expected by
+     * bergamot-translator.
+     * @param {{env: {memory: WebAssembly.Memory}}} info
+     * @return {{[method:string]: (...any) => any}}
+     */
+    linkFallbackIntGemm(info) {
+        const mapping = Object.entries(BergamotTranslatorWorker.GEMM_TO_FALLBACK_FUNCTIONS_MAP).map(([key, name]) => {
+            return [key, (...args) => Module['asm'][name](...args)]
+        });
+
+        return Object.fromEntries(mapping);
+    }
+
+    /**
+     * Internal method. Reads and instantiates the WASM binary. Returns a
+     * promise for the exported Module object that contains all the classes
+     * and functions exported by bergamot-translator.
+     * @return {Promise<BergamotTranslator>}
+     */
+    loadModule() {
+        return new Promise(async (resolve, reject) => {
+            try {
+                const response = await self.fetch(new URL('./bergamot-translator-worker.wasm', self.location));
+
+                Object.assign(Module, {
+                    instantiateWasm: (info, accept) => {
+                        try {
+                            WebAssembly.instantiateStreaming(response, {
+                                ...info,
+                                'wasm_gemm': this.options.useNativeIntGemm
+                                    ? this.linkNativeIntGemm(info)
+                                    : this.linkFallbackIntGemm(info)
+                            }).then(({instance}) => accept(instance)).catch(reject);
+                        } catch (err) {
+                            reject(err);
+                        }
+                        return {};
+                    },
+                    onRuntimeInitialized: () => {
+                        resolve(Module);
+                    }
+                });
+
+                // Emscripten glue code. Webpack et al. should not mangle the `Module` property name!
+                self.Module = Module;
+                self.importScripts('bergamot-translator-worker.js');
+            } catch (err) {
+                reject(err);
+            }
+        });
+    }
+
+    /**
+     * Internal method. Instantiates a BlockingService()
+     * @return {BergamotTranslator.BlockingService}
+     */
+    loadTranslationService() {
+        return new this.module.BlockingService({
+            cacheSize: Math.max(this.options.cacheSize || 0, 0)
+        });
+    }
+
+    /**
+     * Returns whether a model has already been loaded in this worker. Marked
+     * async because the message passing interface we use expects async methods.
+     * @param {{from:string, to:string}}
+     * @return boolean
+     */ 
+    hasTranslationModel({from,to}) {
+        const key = JSON.stringify({from,to});
+        return this.models.has(key);
+    }
+
+    /**
+     * Loads a translation model from a set of file buffers. After this, the
+     * model is available to translate with and `hasTranslationModel()` will
+     * return true for this pair.
+     * @param {{from:string, to:string}}
+     * @param {{
+     *   model: ArrayBuffer,
+     *   shortlist: ArrayBuffer,
+     *   vocabs: ArrayBuffer[],
+     *   qualityModel: ArrayBuffer?,
+     *   config?: {
+     *     [key:string]: string
+     *   }
+     * }} buffers
+     */ 
+    loadTranslationModel({from, to}, buffers) {
+        // This because service_bindings.cpp:prepareVocabsSmartMemories :(
+        const uniqueVocabs = buffers.vocabs.filter((vocab, index, vocabs) => {
+            return !vocabs.slice(0, index).includes(vocab);
+        });
+
+        const [modelMemory, shortlistMemory, qualityModel, ...vocabMemory] = [
+            this.prepareAlignedMemoryFromBuffer(buffers.model, 256),
+            this.prepareAlignedMemoryFromBuffer(buffers.shortlist, 64),
+            buffers.qualityModel // optional quality model
+                ? this.prepareAlignedMemoryFromBuffer(buffers.qualityModel, 64)
+                : null,
+            ...uniqueVocabs.map(vocab => this.prepareAlignedMemoryFromBuffer(vocab, 64))
+        ];
+
+        const vocabs = new this.module.AlignedMemoryList();
+        vocabMemory.forEach(vocab => vocabs.push_back(vocab));
+
+        // Defaults
+        let modelConfig = YAML.parse(`
+            beam-size: 1
+            normalize: 1.0
+            word-penalty: 0
+            cpu-threads: 0
+            gemm-precision: int8shiftAlphaAll
+            skip-cost: true
+        `);
+
+        if (buffers.config)
+            Object.assign(modelConfig, buffers.config);
+
+        // WASM marian is only compiled with support for shiftedAll.
+        if (modelConfig['gemm-precision'] === 'int8')
+            modelConfig['gemm-precision'] = 'int8shiftAll';
+
+        // Override these
+        Object.assign(modelConfig, YAML.parse(`
+            alignment: soft
+            quiet: true
+            quiet-translation: true
+            max-length-break: 128
+            mini-batch-words: 1024
+            workspace: 128
+            max-length-factor: 2.0
+        `));
+
+        const key = JSON.stringify({from,to});
+        this.models.set(key, new this.module.TranslationModel(YAML.stringify(modelConfig), modelMemory, shortlistMemory, vocabs, qualityModel));
+    }
+
+    /**
+     * Frees up memory used by old translation model. Does nothing if model is
+     * already deleted.
+     * @param {{from:string, to:string}}
+     */
+    freeTranslationModel({from, to}) {
+        const key = JSON.stringify({from,to});
+        
+        if (!this.models.has(key))
+            return;
+        
+        const model = this.models.get(key);
+        this.models.delete(key);
+
+        model.delete();
+    }
+
+    /**
+     * Internal function. Copies the data from an ArrayBuffer into memory that
+     * can be used inside the WASM vm by Marian.
+     * @param {{ArrayBuffer}} buffer
+     * @param {number} alignmentSize
+     * @return {BergamotTranslator.AlignedMemory}
+     */
+    prepareAlignedMemoryFromBuffer(buffer, alignmentSize) {
+        const bytes = new Int8Array(buffer);
+        const memory = new this.module.AlignedMemory(bytes.byteLength, alignmentSize);
+        memory.getByteArrayView().set(bytes);
+        return memory;
+    }
+
+    /**
+     * Public. Does actual translation work. You have to make sure that the
+     * models necessary for translating text are already loaded before calling
+     * this method. Returns a promise with translation responses.
+     * @param {{models: {from:string, to:string}[], texts: {text: string, html: boolean}[]}}
+     * @return {Promise<{target: {text: string}}[]>}
+     */
+    translate({models, texts}) {
+        // Convert texts array into a std::vector<std::string>.
+        let input = new this.module.VectorString();
+        texts.forEach(({text}) => input.push_back(text));
+
+        // Extracts the texts[].html options into ResponseOption objects
+        let options = new this.module.VectorResponseOptions();
+        texts.forEach(({html, qualityScores}) => options.push_back({alignment: false, html, qualityScores}));
+
+        // Turn our model names into a list of TranslationModel pointers
+        const translationModels = models.map(({from,to}) => {
+            const key = JSON.stringify({from,to});
+            return this.models.get(key);
+        });
+
+        // translate the input, which is a vector<String>; the result is a vector<Response>
+        const responses = models.length > 1
+            ? this.service.translateViaPivoting(...translationModels, input, options)
+            : this.service.translate(...translationModels, input, options);
+        
+        input.delete();
+        options.delete();
+
+        // Convert the Response WASM wrappers into native JavaScript types we
+        // can send over the 'wire' (message passing) in the same format as we
+        // use in bergamot-translator.
+        const translations = texts.map((_, i) => ({
+            target: {
+                text: responses.get(i).getTranslatedText()
+            }
+        }));
+
+        responses.delete();
+
+        return translations;
+    }
+}
+
+/**
+ * Because you can't put an Error object in a message. But you can post a
+ * generic object!
+ * @param {Error} error
+ * @return {{
+ *  name: string?,
+ *  message: string?,
+ *  stack: string?
+ * }}
+ */
+function cloneError(error) {
+    return {
+        name: error.name,
+        message: error.message,
+        stack: error.stack
+    };
+}
+
+// (Constructor doesn't really do anything, we need to call `initialize()`
+// first before using it. That happens from outside the worker.)
+const worker = new BergamotTranslatorWorker();
+
+self.addEventListener('message', async function({data: {id, name, args}}) {
+    if (!id)
+        console.error('Received message without id', arguments[0]);
+
+    try {
+        if (typeof worker[name] !== 'function')
+            throw TypeError(`worker[${name}] is not a function`);
+
+        // Using `Promise.resolve` to await any promises that worker[name]
+        // possibly returns.
+        const result = await Promise.resolve(Reflect.apply(worker[name], worker, args));
+        self.postMessage({id, result});
+    } catch (error) {
+        self.postMessage({
+            id,
+            error: cloneError(error)
+        })
+    }
+});
diff --git a/wasm/test_page/css/index.css b/wasm/test_page/css/index.css
index 6ed642232..11521a156 100644
--- a/wasm/test_page/css/index.css
+++ b/wasm/test_page/css/index.css
@@ -14,20 +14,33 @@ body {
   padding: 1rem;
 }
 
+[hidden] {
+  display: none;
+}
+
 .app {
   padding: 1rem;
   display: grid;
-  grid: "from swap to" 1fr "status status status" auto / 1fr auto 1fr;
+  grid: "from swap to" auto "credits credits credits" min-content / 1fr auto 1fr;
   grid-gap: 1rem;
   overflow: hidden;
-  min-height: 400px;
+  min-height: 100%;
   max-width: 1024px;
-  margin: 1em auto;
+  margin: 0 auto;
+}
+
+.swap::before {
+  display: inline-block;
+  content: '↔️';
 }
 
 @media screen and (max-width: 640px) {
   .app {
-    grid: "from from" auto "status swap" auto "to to" auto / 1fr;
+    grid: "from from" auto "swap swap" auto "to to" auto "credits credits" auto / 1fr;
+  }
+
+  .swap::before {
+    content: '↕️';
   }
 }
 
@@ -35,6 +48,8 @@ body {
   display: grid;
   grid-template-rows: auto 1fr;
   grid-gap: 1rem;
+  max-height: 100%;
+  overflow: hidden;
 }
 
 label {
@@ -67,19 +82,25 @@ label {
   font-size: 1.1rem;
 }
 
-#status {
-  grid-area: status;
-  text-align: center;
-  align-self: center;
+.credits {
+  grid-area: credits;
 }
 
-textarea, .output-area {
+.credits img {
+  float: left;
+  margin: 1em 0;
+}
+
+textarea, [contenteditable], .output-area {
   padding: 1rem;
   font-family: sans-serif;
   font-size: 1rem;
   resize: none;
   border-radius: 2px;
   border: 1px solid #ccc;
+  min-height: 100px;
+  max-height: 100%;
+  overflow: auto;
 }
 
 button {
@@ -96,6 +117,7 @@ button:hover {
 
 #output {
   background-color: #f4f4f4;
+  position: relative;
 }
 
 .output-area [x-bergamot-word-score].bad {
@@ -115,4 +137,32 @@ button:hover {
 
 .output-area [x-bergamot-sentence-index].highlight-sentence {
   background: rgba(255, 255, 128, 0.8);
-}
\ No newline at end of file
+}
+
+.app.translating #output::after {
+  position: absolute;
+  bottom: 4px;
+  right: 4px;
+  content: 'Translating…';
+}
+
+/* Loading indicator takes priority, so below the .translating selector */
+.app.loading #output::after {
+  position: absolute;
+  bottom: 4px;
+  right: 4px;
+  content: 'Loading translation model…';
+}
+
+.app {
+  position: relative;
+}
+
+#unsupported-browser {
+  position: absolute;
+  top: 0;
+  left: 0;
+  width: 100%;
+  height: 100%;
+  background: white;
+}
diff --git a/wasm/test_page/index.html b/wasm/test_page/index.html
index 3f48117e1..124857e23 100644
--- a/wasm/test_page/index.html
+++ b/wasm/test_page/index.html
@@ -1,7 +1,7 @@
 <!DOCTYPE html>
 <html>
   <head>
-    <title>Mozilla Translations</title>
+    <title>Bergamot Translations</title>
     <link rel="stylesheet" href="css/index.css" />
     <meta http-equiv="Content-Type" content="text/html;charset=UTF-8" />
     <meta
@@ -16,9 +16,9 @@
           From
           <select id="lang-from" name="from" class="lang-select"></select>
         </label>
-        <textarea id="input" name="input"></textarea>
+        <div id="input" contenteditable="true"></div>
       </div>
-      <button class="swap" title="swap">↔️</button>
+      <button class="swap" title="swap"></button>
       <div class="panel panel--to">
         <label>
           To
@@ -26,8 +26,16 @@
         </label>
         <div id="output" class="output-area"></div>
       </div>
-      <div class="footer" id="status"></div>
+      <div id="unsupported-browser" hidden>
+        <p>Your CPU or browser is not able to run Bergamot translator.</p>
+        <p>Try using Firefox or a Chromium based browser with <a href="https://webassembly.org/roadmap/">Fixed-width SIMD support</a>.</p>
+        <p>If you already are, you might be using a CPU that does not have support for SSE4.1 instructions.</p>
+      </div>
+      <footer class="credits">
+        <img src="logos.png" alt="Logos of the OPUS project, the Bergamot project and the European Union.">
+        <p>This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825303.</p>
+      </footer>
     </div>
-    <script src="js/index.js"></script>
+    <script type="module" src="js/index.js"></script>
   </body>
 </html>
diff --git a/wasm/test_page/js/index.js b/wasm/test_page/js/index.js
index b1c308e8b..56cbfdc72 100644
--- a/wasm/test_page/js/index.js
+++ b/wasm/test_page/js/index.js
@@ -1,156 +1,215 @@
-let worker;
-let modelRegistry;
+import {LatencyOptimisedTranslator, TranslatorBacking, CancelledError, SupersededError} from '../node_modules/@browsermt/bergamot-translator/translator.js';
 
-const $ = selector => document.querySelector(selector);
-const status = message => ($("#status").innerText = message);
-
-const langFrom = $("#lang-from");
-const langTo = $("#lang-to");
-
-if (window.Worker) {
-  worker = new Worker("js/worker.js");
-  worker.postMessage(["import"]);
+function $(selector) {
+  return document.querySelector(selector);
 }
 
-document.querySelector("#input").addEventListener("keyup", function (event) {
-  translateCall();
-});
-
-const _prepareTranslateOptions = (paragraphs) => {
-  const translateOptions = [];
-  paragraphs.forEach(paragraph => {
-    // Each option object can be different for each entry. But to keep the test page simple,
-    // we just keep all the options same (specifically avoiding parsing the input to determine
-    // html/non-html text)
-    translateOptions.push({"isQualityScores": true, "isHtml": true});
-  });
-  return translateOptions;
-};
+function $$(selector) {
+  return document.querySelectorAll(selector);
+}
 
-const textToHTML = (text) => {
+function encodeHTML(text) {
   const div = document.createElement('div');
   div.appendChild(document.createTextNode(text));
   return div.innerHTML;
-};
-
-const translateCall = () => {
-  const text = document.querySelector("#input").value;
-  if (!text.trim().length) return;
-
-  const paragraphs = text.split(/\n+/).map(textToHTML); // escape HTML 
-  const translateOptions = _prepareTranslateOptions(paragraphs);
-  const lngFrom = langFrom.value;
-  const lngTo = langTo.value;
-  worker.postMessage(["translate", lngFrom, lngTo, paragraphs, translateOptions]);
-};
-
-const addQualityClasses = (root) => {
-  // You can do this wit CSS variables, calc() and min/max, but JS is just easier
+}
 
-  root.querySelectorAll('[x-bergamot-sentence-score]').forEach(el => {
+function addQualityIndicators() {
+  $$('#output [x-bergamot-sentence-score]').forEach(el => {
     // The threshold is ln(0.5) (https://github.com/browsermt/bergamot-translator/pull/370#issuecomment-1058123399)
-    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-sentence-score')) < -0.6931);
+    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-sentence-score')) < Math.log(0.5));
   });
 
-  root.querySelectorAll('[x-bergamot-word-score]').forEach(el => {
+  $$('#output [x-bergamot-word-score]').forEach(el => {
     // The threshold is ln(0.5) (https://github.com/browsermt/bergamot-translator/pull/370#issuecomment-1058123399)
-    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-word-score')) < -0.6931);
+    el.classList.toggle('bad', parseFloat(el.getAttribute('x-bergamot-word-score')) < Math.log(0.5));
   });
 
   // Add tooltips to each (sub)word with sentence and word score.
-  root.querySelectorAll('[x-bergamot-sentence-score] > [x-bergamot-word-score]').forEach(el => {
+  $$('#output [x-bergamot-sentence-score] > [x-bergamot-word-score]').forEach(el => {
     const sentenceScore = parseFloat(el.parentNode.getAttribute('x-bergamot-sentence-score'));
     const wordScore = parseFloat(el.getAttribute('x-bergamot-word-score'));
-    el.title = `Sentence: ${sentenceScore}  Word: ${wordScore}`;
+    el.title = `Sentence: ${Math.exp(sentenceScore).toFixed(2)}  Word: ${Math.exp(wordScore).toFixed(2)}`;
   });
 }
 
-worker.onmessage = function (e) {
-  if (e.data[0] === "translate_reply" && e.data[1]) {
-    // Clear output of previous translation
-    document.querySelector("#output").innerHTML = '';
-
-    // Add each translation in its own div to have a known root in which the
-    // sentence ids are unique. Used for highlighting sentences.
-    e.data[1].forEach(translatedHTML => {
-      const translation = document.createElement('div');
-      translation.classList.add('translation');
-      translation.innerHTML = translatedHTML;
-      addQualityClasses(translation);
-      document.querySelector("#output").appendChild(translation);
-    });
-  } else if (e.data[0] === "load_model_reply" && e.data[1]) {
-    status(e.data[1]);
-    translateCall();
-  } else if (e.data[0] === "import_reply" && e.data[1]) {
-    modelRegistry = e.data[1];
-    init();
+function highlightSentence(element) {
+  const sentence = element.parentNode.hasAttribute('x-bergamot-sentence-index')
+    ? element.parentNode.getAttribute('x-bergamot-sentence-index')
+    : null;
+  $$('#output font[x-bergamot-sentence-index]').forEach(el => {
+    el.classList.toggle('highlight-sentence', el.getAttribute('x-bergamot-sentence-index') === sentence);
+  })
+}
+
+/**
+ * Very minimal WISYWIG editor. Just keyboard shortcuts for the IYKYK crowd.
+ */
+class Editor {
+  constructor(root) {
+    this.isApple = window.navigator.platform.startsWith('Mac');
+
+    this.root = root;
+    this.root.addEventListener('keydown', this.onkeydown.bind(this));
+
+    this.mapping = {
+      "b": "bold",
+      "i": "italic",
+      "u": "underline",
+    };
   }
-};
-
-const loadModel = () => {
-  const lngFrom = langFrom.value;
-  const lngTo = langTo.value;
-  if (lngFrom !== lngTo) {
-    status(`Installing model...`);
-    console.log(`Loading model '${lngFrom}${lngTo}'`);
-    worker.postMessage(["load_model", lngFrom, lngTo]);
-  } else {
-    const input = textToHTML(document.querySelector("#input").value);
-    document.querySelector("#output").innerHTML = input;
+
+  onkeydown(event) {
+    if (!(this.isApple ? event.metaKey : event.ctrlKey))
+      return;
+
+    if (!(event.key in this.mapping))
+      return;
+
+    document.execCommand(this.mapping[event.key], false, null);
+
+    event.preventDefault();
   }
-};
-
-langFrom.addEventListener("change", e => {
-  loadModel();
-});
-
-langTo.addEventListener("change", e => {
-  loadModel();
-});
-
-$(".swap").addEventListener("click", e => {
-  [langFrom.value, langTo.value] = [langTo.value, langFrom.value];
-  $("#input").value = $("#output").innerText;
-  loadModel();
-});
-
-$('#output').addEventListener('mouseover', e => {
-  const root = e.target.closest('.translation');
-  const sentence = e.target.parentNode.hasAttribute('x-bergamot-sentence-index') ? e.target.parentNode.getAttribute('x-bergamot-sentence-index') : null;  
-  document.querySelectorAll('#output font[x-bergamot-sentence-index]').forEach(el => {
-    el.classList.toggle('highlight-sentence', el.getAttribute('x-bergamot-sentence-index') === sentence && el.closest('.translation') === root);
-  })
-})
+}
+
+async function main() {
+  const options = {
+    cacheSize: 2^13,
+    downloadTimeout: null // Disable timeout
+  };
+  
+  const backing = new TranslatorBacking(options);
+
+  let pending = 0; // Number of pending requests
+
+  // Patch the fetch() function to track number of pending requests
+  backing.fetch = async function(...args) {
+    try {
+      $('.app').classList.toggle('loading', ++pending > 0);
+      return await TranslatorBacking.prototype.fetch.call(backing, ...args);
+    } finally {
+      $('.app').classList.toggle('loading', --pending > 0);
+    }
+  };
 
-function init() {
-  // Populate langs
-  const langs = Array.from(new Set(Object.keys(modelRegistry).reduce((acc, key) => acc.concat([key.substr(0, 2), key.substr(2, 2)]), [])));
-  const langNames = new Intl.DisplayNames(undefined, {type: "language"});
+  // Wait for the language model registry to load. Once it is loaded, use
+  // it to fill the "from" and "to" language selection dropdowns.
+  await backing.registry.then(models => {
+    const names = new Intl.DisplayNames(['en'], {type: 'language'});
 
-  // Sort languages by display name
-  langs.sort((a, b) => langNames.of(a).localeCompare(langNames.of(b)));
+    ['from', 'to'].forEach(field => {
+      const languages = new Set(models.map(model => model[field]));
+      const select = $(`#lang-${field}`);
 
-  // Populate the dropdowns 
-  langs.forEach(code => {
-    const name = langNames.of(code);
-    langFrom.innerHTML += `<option value="${code}">${name}</option>`;
-    langTo.innerHTML += `<option value="${code}">${name}</option>`;
+      const pairs = Array.from(languages, code => ({code, name: names.of(code)}));
+      
+      pairs.sort(({name: a}, {name: b}) => a.localeCompare(b));
+
+      pairs.forEach(({name, code}) => {
+        select.add(new Option(name, code));
+      })
+    });
+
+    $('#lang-from').value = 'en';
+    $('#lang-to').value = 'es';
   });
 
-  // try to guess input language from user agent
-  let myLang = navigator.language;
-  if (myLang) {
-    myLang = myLang.split("-")[0];
-    let langIndex = langs.indexOf(myLang);
-    if (langIndex > -1) {
-      console.log("guessing input language is", myLang);
-      langFrom.value = myLang;
+  // Intentionally do this after querying backing.registry to make sure that
+  // that request is fired off first. Now we can start thinking about loading
+  // the WASM binary etc.
+  const translator = new LatencyOptimisedTranslator(options, backing);
+
+  let abortController = new AbortController();
+
+  const translate = async () => {
+    try {
+      const from = $('#lang-from').value;
+      const to = $('#lang-to').value;
+      
+      // Querying models to see whether quality estimation is supported by all
+      // of them.
+      const models = await backing.getModels({from, to});
+      const qualityScores = models.every(model => 'qualityModel' in model.files);
+
+      $('.app').classList.add('translating');
+
+      const response = await translator.translate({
+        from,
+        to,
+        text: $('#input').innerHTML,
+        html: true,
+        qualityScores
+      }, {signal: abortController.signal});
+
+      $('#output').innerHTML = response.target.text;
+      $('#output').classList.toggle('has-quality-scores', qualityScores);
+
+      if (qualityScores)
+        addQualityIndicators();
+
+    } catch (error) {
+      // Ignore errors caused by changing the language pair (which triggers abort())
+      if (error.constructor === CancelledError) {
+        return;
+      }
+      
+      // Ignore 'errors' caused by typing too fast or by changing the language
+      // pair while a translation was still in progress (or being loaded)
+      if (error.constructor === SupersededError || error.constructor === CancelledError)
+        return;
+
+      // Ignore errors caused by selecting a bad pair (e.g. en -> en)
+      if (error.message.startsWith('No model available to translate from'))
+        return;
+
+      alert(`Error during translation: ${error}\n\n${error.stack}`);
+    } finally {
+      const worker = await Promise.race([translator.worker, Promise.resolve(null)]);
+      $('.app').classList.toggle('translating', worker === null || !worker.idle);
     }
   }
 
-  // find first output lang that *isn't* input language
-  langTo.value = langs.find(code => code !== langFrom.value);
-  // load this model
-  loadModel();
+  const reset = async () => {
+    // Cancel any pending loading/translation
+    abortController.abort();
+
+    // Reset abort controller to a fresh un-aborted one
+    abortController = new AbortController();
+
+    // Clear output to make it more clear something is happening
+    $('#output').innerHTML = '';
+
+    // Immediately start loading the new selection
+    translate();
+  }
+
+  $('button.swap').addEventListener('click', () => {
+    const tmp = $('#lang-from').value;
+    $('#lang-from').value = $('#lang-to').value;
+    $('#lang-to').value = tmp;
+    translate();
+  })
+
+  // Simple WYSIWYG controls
+  const editor = new Editor($('#input'));
+
+  // Translate on any change
+  $('#input').addEventListener('input', translate);
+  $('#lang-from').addEventListener('input', reset);
+  $('#lang-to').addEventListener('input', reset);
+
+  // Hook up sentence boundary highlighting if that information is available.
+  $('#output').addEventListener('mouseover', (e) => highlightSentence(e.target))
+
+  // Wait for bergamot-translator to load. This could throw a CompileError
+  // which we want to catch so we can show "oh noes browser not supported!"
+  translator.worker.catch(error => {
+    // Catch CompileErrors because for those we know what to do.
+    if (error.name === 'CompileError')
+      $('#unsupported-browser').hidden = false;
+    else
+      throw error;
+  });
 }
+
+main();
diff --git a/wasm/test_page/js/worker.js b/wasm/test_page/js/worker.js
deleted file mode 100644
index 3327d8a3a..000000000
--- a/wasm/test_page/js/worker.js
+++ /dev/null
@@ -1,352 +0,0 @@
-// All variables specific to translation service
-var translationService = undefined;
-
-// Model registry
-let modelRegistry = undefined;
-
-// A map of language-pair to TranslationModel object
-var languagePairToTranslationModels = new Map();
-
-const BERGAMOT_TRANSLATOR_MODULE = "bergamot-translator-worker.js";
-const MODEL_REGISTRY = "../models/registry.json";
-const MODEL_ROOT_URL = "../models/";
-const PIVOT_LANGUAGE = 'en';
-
-// Information corresponding to each file type
-const fileInfo = [
-  {"type": "model", "alignment": 256},
-  {"type": "lex", "alignment": 64},
-  {"type": "vocab", "alignment": 64},
-  {"type": "qualityModel", "alignment": 64}
-];
-
-const encoder = new TextEncoder(); // string to utf-8 converter
-const decoder = new TextDecoder(); // utf-8 to string converter
-
-const start = Date.now();
-let moduleLoadStart;
-var Module = {
-  preRun: [function() {
-    log(`Time until Module.preRun: ${(Date.now() - start) / 1000} secs`);
-    moduleLoadStart = Date.now();
-  }],
-  onRuntimeInitialized: async function() {
-    log(`Wasm Runtime initialized Successfully (preRun -> onRuntimeInitialized) in ${(Date.now() - moduleLoadStart) / 1000} secs`);
-    const response = await fetch(MODEL_REGISTRY);
-    modelRegistry = await response.json();
-    postMessage([`import_reply`, modelRegistry]);
-  }
-};
-
-const log = (message) => {
-  console.debug(message);
-}
-
-onmessage = async function(e) {
-  const command = e.data[0];
-  log(`Message '${command}' received from main script`);
-  let result = "";
-  if (command === 'import') {
-      importScripts(BERGAMOT_TRANSLATOR_MODULE);
-  } else if (command === 'load_model') {
-      let start = Date.now();
-      let from = e.data[1];
-      let to = e.data[2];
-      try {
-        await constructTranslationService();
-        await constructTranslationModel(from, to);
-        log(`Model '${from}${to}' successfully constructed. Time taken: ${(Date.now() - start) / 1000} secs`);
-        result = "Model successfully loaded";
-      } catch (error) {
-        log(`Model '${from}${to}' construction failed: '${error.message}'`);
-        result = "Model loading failed";
-      }
-      log(`'${command}' command done, Posting message back to main script`);
-      postMessage([`${command}_reply`, result]);
-  } else if (command === 'translate') {
-      const from = e.data[1];
-      const to = e.data[2];
-      const input = e.data[3];
-      const translateOptions = e.data[4];
-      let inputWordCount = 0;
-      let inputBlockElements = 0;
-      input.forEach(sentence => {
-        inputWordCount += sentence.trim().split(" ").filter(word => word.trim() !== "").length;
-        inputBlockElements++;
-      })
-      let start = Date.now();
-      try {
-        log(`Blocks to translate: ${inputBlockElements}`);
-        result = translate(from, to, input, translateOptions);
-        const secs = (Date.now() - start) / 1000;
-        log(`Translation '${from}${to}' Successful. Speed: ${Math.round(inputWordCount / secs)} WPS (${inputWordCount} words in ${secs} secs)`);
-      } catch (error) {
-        log(`Error: ${error.message}`);
-      }
-      log(`'${command}' command done, Posting message back to main script`);
-      postMessage([`${command}_reply`, result]);
-  }
-}
-
-// Instantiates the Translation Service
-const constructTranslationService = async () => {
-  if (!translationService) {
-    var translationServiceConfig = {cacheSize: 20000};
-    log(`Creating Translation Service with config: ${translationServiceConfig}`);
-    translationService = new Module.BlockingService(translationServiceConfig);
-    log(`Translation Service created successfully`);
-  }
-}
-
-// Constructs translation model(s) for the source and target language pair (using
-// pivoting if required).
-const constructTranslationModel = async (from, to) => {
-  // Delete all previously constructed translation models and clear the map
-  languagePairToTranslationModels.forEach((value, key) => {
-    log(`Destructing model '${key}'`);
-    value.delete();
-  });
-  languagePairToTranslationModels.clear();
-
-  if (_isPivotingRequired(from, to)) {
-    // Pivoting requires 2 translation models
-    const languagePairSrcToPivot = _getLanguagePair(from, PIVOT_LANGUAGE);
-    const languagePairPivotToTarget = _getLanguagePair(PIVOT_LANGUAGE, to);
-    await Promise.all([_constructTranslationModelHelper(languagePairSrcToPivot),
-                      _constructTranslationModelHelper(languagePairPivotToTarget)]);
-  }
-  else {
-    // Non-pivoting case requires only 1 translation model
-    await _constructTranslationModelHelper(_getLanguagePair(from, to));
-  }
-}
-
-// Translates text from source language to target language (via pivoting if necessary).
-const translate = (from, to, input, translateOptions) => {
-  let vectorResponseOptions, vectorSourceText, vectorResponse;
-  try {
-    // Prepare the arguments (vectorResponseOptions and vectorSourceText (vector<string>)) of Translation API and call it.
-    // Result is a vector<Response> where each of its item corresponds to one item of vectorSourceText in the same order.
-    vectorResponseOptions = _prepareResponseOptions(translateOptions);
-    vectorSourceText = _prepareSourceText(input);
-
-    if (_isPivotingRequired(from, to)) {
-      // Translate via pivoting
-      const translationModelSrcToPivot = _getLoadedTranslationModel(from, PIVOT_LANGUAGE);
-      const translationModelPivotToTarget = _getLoadedTranslationModel(PIVOT_LANGUAGE, to);
-      vectorResponse = translationService.translateViaPivoting(translationModelSrcToPivot,
-                                                              translationModelPivotToTarget,
-                                                              vectorSourceText,
-                                                              vectorResponseOptions);
-    }
-    else {
-      // Translate without pivoting
-      const translationModel = _getLoadedTranslationModel(from, to);
-      vectorResponse = translationService.translate(translationModel, vectorSourceText, vectorResponseOptions);
-    }
-
-    // Parse all relevant information from vectorResponse
-    const listTranslatedText = _parseTranslatedText(vectorResponse);
-    const listSourceText = _parseSourceText(vectorResponse);
-    const listTranslatedTextSentences = _parseTranslatedTextSentences(vectorResponse);
-    const listSourceTextSentences = _parseSourceTextSentences(vectorResponse);
-
-    log(`Source text: ${listSourceText}`);
-    log(`Translated text: ${listTranslatedText}`);
-    log(`Translated sentences: ${JSON.stringify(listTranslatedTextSentences)}`);
-    log(`Source sentences: ${JSON.stringify(listSourceTextSentences)}`);
-
-    return listTranslatedText;
-  } finally {
-    // Necessary clean up
-    if (vectorSourceText != null) vectorSourceText.delete();
-    if (vectorResponseOptions != null) vectorResponseOptions.delete();
-    if (vectorResponse != null) vectorResponse.delete();
-  }
-}
-
-// Downloads file from a url and returns the array buffer
-const _downloadAsArrayBuffer = async(url) => {
-  const response = await fetch(url);
-  if (!response.ok) {
-    throw Error(`Downloading ${url} failed: HTTP ${response.status} - ${response.statusText}`);
-  }
-  return response.arrayBuffer();
-}
-
-// Constructs and initializes the AlignedMemory from the array buffer and alignment size
-const _prepareAlignedMemoryFromBuffer = async (buffer, alignmentSize) => {
-  var byteArray = new Int8Array(buffer);
-  var alignedMemory = new Module.AlignedMemory(byteArray.byteLength, alignmentSize);
-  const alignedByteArrayView = alignedMemory.getByteArrayView();
-  alignedByteArrayView.set(byteArray);
-  return alignedMemory;
-}
-
-async function prepareAlignedMemory(file, languagePair) {
-  const fileName = `${MODEL_ROOT_URL}/${languagePair}/${modelRegistry[languagePair][file.type].name}`;
-  const buffer = await _downloadAsArrayBuffer(fileName);
-  const alignedMemory = await _prepareAlignedMemoryFromBuffer(buffer, file.alignment);
-  log(`"${file.type}" aligned memory prepared. Size:${alignedMemory.size()} bytes, alignment:${file.alignment}`);
-  return alignedMemory;
-}
-
-const _constructTranslationModelHelper = async (languagePair) => {
-  log(`Constructing translation model ${languagePair}`);
-
-  /*Set the Model Configuration as YAML formatted string.
-    For available configuration options, please check: https://marian-nmt.github.io/docs/cmd/marian-decoder/
-    Vocab files are re-used in both translation directions.
-    DO NOT CHANGE THE SPACES BETWEEN EACH ENTRY OF CONFIG
-  */
-  const modelConfig = `beam-size: 1
-normalize: 1.0
-word-penalty: 0
-max-length-break: 128
-mini-batch-words: 1024
-workspace: 128
-max-length-factor: 2.0
-skip-cost: false
-cpu-threads: 0
-quiet: true
-quiet-translation: true
-gemm-precision: int8shiftAlphaAll
-alignment: soft
-`;
-
-  const promises = [];
-  fileInfo.filter(file => modelRegistry[languagePair].hasOwnProperty(file.type))
-  .map((file) => {
-      promises.push(prepareAlignedMemory(file, languagePair));
-  });
-
-  const alignedMemories = await Promise.all(promises);
-
-  log(`Translation Model config: ${modelConfig}`);
-  log(`Aligned memory sizes: Model:${alignedMemories[0].size()} Shortlist:${alignedMemories[1].size()} Vocab:${alignedMemories[2].size()}`);
-  const alignedVocabMemoryList = new Module.AlignedMemoryList();
-  alignedVocabMemoryList.push_back(alignedMemories[2]);
-  let translationModel;
-  if (alignedMemories.length === fileInfo.length) {
-    log(`QE:${alignedMemories[3].size()}`);
-    translationModel = new Module.TranslationModel(modelConfig, alignedMemories[0], alignedMemories[1], alignedVocabMemoryList, alignedMemories[3]);
-  }
-  else {
-    translationModel = new Module.TranslationModel(modelConfig, alignedMemories[0], alignedMemories[1], alignedVocabMemoryList, null);
-  }
-  languagePairToTranslationModels.set(languagePair, translationModel);
-}
-
-const _isPivotingRequired = (from, to) => {
-  return (from !== PIVOT_LANGUAGE) && (to !== PIVOT_LANGUAGE);
-}
-
-const _getLanguagePair = (srcLang, tgtLang) => {
-  return `${srcLang}${tgtLang}`;
-}
-
-const _getLoadedTranslationModel = (srcLang, tgtLang) => {
-  const languagePair = _getLanguagePair(srcLang, tgtLang);
-  if (!languagePairToTranslationModels.has(languagePair)) {
-    throw Error(`Translation model '${languagePair}' not loaded`);
-  }
-  return languagePairToTranslationModels.get(languagePair);
-}
-
-const _parseTranslatedText = (vectorResponse) => {
-  const result = [];
-  for (let i = 0; i < vectorResponse.size(); i++) {
-    const response = vectorResponse.get(i);
-    result.push(response.getTranslatedText());
-  }
-  return result;
-}
-
-const _parseTranslatedTextSentences = (vectorResponse) => {
-  const result = [];
-  for (let i = 0; i < vectorResponse.size(); i++) {
-    const response = vectorResponse.get(i);
-    result.push(_getTranslatedSentences(response));
-  }
-  return result;
-}
-
-const _parseSourceText = (vectorResponse) => {
-  const result = [];
-  for (let i = 0; i < vectorResponse.size(); i++) {
-    const response = vectorResponse.get(i);
-    result.push(response.getOriginalText());
-  }
-  return result;
-}
-
-const _parseSourceTextSentences = (vectorResponse) => {
-  const result = [];
-  for (let i = 0; i < vectorResponse.size(); i++) {
-    const response = vectorResponse.get(i);
-    result.push(_getSourceSentences(response));
-  }
-  return result;
-}
-
-const _prepareResponseOptions = (translateOptions) => {
-  let vectorResponseOptions = new Module.VectorResponseOptions;
-  translateOptions.forEach(translateOption => {
-    vectorResponseOptions.push_back({
-      qualityScores: translateOption["isQualityScores"],
-      alignment: true,
-      html: translateOption["isHtml"]
-    });
-  });
-  if (vectorResponseOptions.size() == 0) {
-    vectorResponseOptions.delete();
-    throw Error(`No Translation Options provided`);
-  }
-  return vectorResponseOptions;
-}
-
-const _prepareSourceText = (input) => {
-  let vectorSourceText = new Module.VectorString;
-  input.forEach(paragraph => {
-    // prevent empty paragraph - it breaks the translation
-    if (paragraph.trim() === "") {
-      return;
-    }
-    vectorSourceText.push_back(paragraph.trim())
-  })
-  if (vectorSourceText.size() == 0) {
-    vectorSourceText.delete();
-    throw Error(`No text provided to translate`);
-  }
-  return vectorSourceText;
-}
-
-const _getTranslatedSentences = (response) => {
-  const sentences = [];
-  const text = response.getTranslatedText();
-  for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
-    const utf8SentenceByteRange = response.getTranslatedSentence(sentenceIndex);
-    sentences.push(_getSubString(text, utf8SentenceByteRange));
-  }
-  return sentences;
-}
-
-const _getSourceSentences = (response) => {
-  const sentences = [];
-  const text = response.getOriginalText();
-  for (let sentenceIndex = 0; sentenceIndex < response.size(); sentenceIndex++) {
-    const utf8SentenceByteRange = response.getSourceSentence(sentenceIndex);
-    sentences.push(_getSubString(text, utf8SentenceByteRange));
-  }
-  return sentences;
-}
-
-/*
- * Returns a substring of text (a string). The substring is represented by
- * byteRange (begin and end endices) within the utf-8 encoded version of the text.
- */
-const _getSubString = (text, utf8ByteRange) => {
-  const textUtf8ByteView = encoder.encode(text);
-  const substringUtf8ByteView = textUtf8ByteView.subarray(utf8ByteRange.begin, utf8ByteRange.end);
-  return decoder.decode(substringUtf8ByteView);
-}
diff --git a/wasm/test_page/logos.png b/wasm/test_page/logos.png
new file mode 100644
index 0000000000000000000000000000000000000000..7646f3ca2623fd25629cc37b939cbc7331141caa
GIT binary patch
literal 15207
zcmZ|01yo#3(my=71qm7?xVt+81Pd-haCaZvT|$5W!Cit3?(P;KxVuA;;O_D#&+fbX
zeEaR*bNY7O>iSi6Raf;neP?c%l7bW}G66CG06>-gD6aC_ue~-VMEKV`b8b8EwSzSm
zl@|p7s-jUI4dGs&M@==P&E(|)^sh7`01*lv0P{*g0ia)RnE#}qUTFfrAL&;LU<XC`
zFWMD~_HXiRe^7w8ugw>V{Vy8(mHrlL34nd=|6uc8+n?*>Yy0#3SFdF0Zfav|=nQrw
z<zVJx<^urOS=rh7SvmOGSV`Hq{}8-v002xj)W7GG4gD{jYuPaWqM80EFCD`7d+m_y
zKWaJw04P|08WbQk{oSkg4wkAK&KmM^{KjBgW<wLOktwsgt^FSqK+v84m9#Z=HY9bo
zwXt*JcNZf6OM?HE{=;S=C;dyr*;<HPLtcqg4D4u1%FWEm%t{VKCM6{mbTl#JR}q)|
zC;atEh}^>2*`A+;#m&u)*^Pr4>}bxy#>dCU!phFV&d&5I!Q|v&=WOWCWamWj4<-Mn
zN8HrO*wND7*%E9=`bV#!5!l69h@AY7p?_Wfn5U_`<-aZ2IsMbER|i@CjIgjVv$Fg*
z*sHVun&1Bi<!)&IKS9oxX8*(1pOOCwGco>`DSH=3o4*2KV$5P{V`^(^=j`-~V`KT3
z|F7-|{+T(y7}y5v2(mXcHU+W?{^iWyS^tj2KcxRkBMouOSB*}uQ3JBGbNmzZpCkVj
zsrhdtC)eMBe;fE4@HKk;@?aB7GY@e?XH(#R$yh;_|B(3)w3wsmpPW|(gKdESSpxsS
z{$1j4;6IV(S21-0+qnEmk56`%&OmlSmjA~7Pn41c*ctqq%Z`@D(ss_Kj`ohPdHqMp
zzZCp$iT~6k2DW{LJN*@F)_<D+JMTZ`6&)>K*ZLpUUYGN)>HeMfpYj_2b`i+)fByZQ
z`JVvI{|5Lw^KSqXV}2(WBP&y5=f6t#UlsLN4TDS_SpLQqWcmL~3$pwv%D)xqKX!n>
zxUX9P5czcl|8F}1BFnOz&H?~j)zadks_sxn83_J3GgmJs!zD{7jPK*gWnyR}aGsC?
z;aIKYL{NplKIbGu=pY)gjnr&+R913w&-b9Xc|uCbTx_`Iy6@x~l_1q`?Ugnj;G3&E
zzxIn&MzEp!gvfL<jJf&F=+TY$3E$0fibvmLDDaceI&Yt2dQrRLr+l}X>Kd~V$ECWx
z7^$dq0XI8K=9D^tfrNV3jqW(?IfwKWnHqt+XPQy-LIhZ5F1ItUH71R^<H-oM(vHJo
zoCD^k-u=iV@28{4r1$$SAzLPJGcB_Y`>Gd3aHh7lW%{=6HqVb^Lf3n_aV}_IY~Blh
zSrl!tn5c=sWx8+O^t$s=T8zGUw&Ob9DqT0PI3&RHUYQE;ksaz$S62(}(AQJ8*Oa#G
zx@z}7X-YU?@vz)(mW`?`wfyZ3nEgt)n<ccAH>1zRwFBZRhaS8=+XA8BF(0O!J)k#0
z2S_2IXeTDzv2LMjWC5yp7$VOf@~k@*XGndL18!^kGK3zu@VwPJ(RjFehp&%p!yu4)
z6Juk>gf9h?5)$9X-~yj}i2>$@fvdbV{7NA8r28vB68HUz4)$~6&LjspwIJWghc1}a
z&E5!<%7U@TUZWcJl;K@j8JXoU6SsX~mwi>dygc<astMnYPtB%7iA{^nZs)hWn~w<I
zBvKU}iYcIiC;-p5_a9j=4j3(nu-4@Qe?>@Qof%*pOjeiA$vcIdW%x~1=lPr8r896O
zzF_h+?!=+D&*^zF(W1fuB9?L{7?w8A4p5Mh6>4oCQrL|9DBb1`OG`?IQkN3z&~4ZJ
zuOGSt;ZV@ji8G~(fp>W^0?k>qGCqk16d)wW)kUSL6<5p>A7BY*hAWQdgGRtWt?>Fq
zo#dv}8<N{J^NJKrAs|k~n$owZ_nsTNC8_4}8T)D6Ue|WkQGTX=C`uNTS^nQ0NqjCR
zg1qZ}l=~doLw4>3X+1f%E^NggB%%b48>w>&R~C?@k@(3f##psGU{8H57p}FU;IOC_
zhz?c`C>Q#PFG)sjs?bIIkrCaaK*lF)1!fGG4T9Ln7+w1S%5Dyc5K$~<B-CvPOPust
zdKHKU*_~ZfSYcgy-rCB_X71sqmV()KB+u%*Q%;PdAw+j{p-UyQosDh4$&w=QYFZsp
z7-q@Am@#8DRiU5+PH<c7VHMB4Y1N5_`q}&ax{w4q?%QKOn~m<QLSZ_A<cOOve(xOH
zO+{73Clm&d`Yyj5d!ntby3(F7Jwc>EdO`At4{kVV5P7@=!5g>;<}&6Nl&Ivamvu#O
zyEcc5I0hDH6;Cm}()M|s*;uCSr%#_WsTkBWAkB-lwY9v_!q4vTl_r#jKMt^29djt-
z2CTBi+%N(7pX*y(Hh+YvGip_?(~EFq)s{56Ee9iEJyUjO`aR#3yFO-L7Xrh@eBA<O
z8Ti^rfZt@&gYWJDBEAyqS=_c-q0(}eK!tnsTj1#GNzUQfmi6iM%zWY}4zHPe*_ZLQ
z;@iOb+2F8#5GHGoBFA{&oj`?+;21M!h0n=n(}oqtq4tfMF<&OVEz!J)RyQ8^-;?=J
z!(tpxJ92y+f=YBA1)Y<qmQ4oa$_qOeSDV%7)F%3v*YFl?AE$ey3xa?{o@1`*2>pub
z?nu<Tu>LnZZ;rBfQ@&=MPDKeFH(Ili6wqLXb<mHkkfEyrz!=-ZP(Nmdl`1&IP$5_>
zRM8yzeot4-2W<kThmpKq7KPW6Gr%Pes}A&|9pvFbR6IpD_UjPDNhN5p<nL&U%vZxc
zED7m%WeC^#1+$)QT-LLy`ky|vaoOqUF3O&)JeWtZuXtYV&y0lid*$1-kfhoJTz(@|
z-YX5zm&-OLZiy|*Sl5kM>|Z60H6(f+Jf;kLsnOQrjHp{p{ywv44)Z^jNN1QcFjJuM
zWz#x+5xSl?7ADOSu>WC0{$Boh{RlolFDgv}1#k5Ou;+JE5?u>Px9-?5+ZRE*IPRI?
z%`&dAo2n_hpy;U1W!3=8sZ!S(v*WVJAf_cAf}mfd>;&Ca!hu!js1-S;)x!0L)NMlx
zT4wn;&Oc;D8BV)@{N8A|&?@I|{E&!uodxnV_&sSAY^vGvyYBF)Mv`i)XbVKm#h&n0
zz0A*erSRM*LwEtoD<iI24NkUHod!+2dgP#0%RJq$^;6P$x$yAYBPXs8OIcO%xFsUl
zD7-X3t$Oe$5$U-pun3!0J3nukMWOzPaAetX^6vVYK_@ztz_GJ}c+&SIvx!HJpdD!|
zELtmZD$;o)e>}2zYeJpz%ob>mV{qA3H4sO`BYT`(UcAc8!_MG!eaJ_fA*UymLPs5f
zi)mQCq`IX1)PV+=a~C<H@9{HrPbwR%_))4Ebub9$UOpoDQvUOY++sRXVH7$X$^#DW
zd@ZW=mzFB`MM_LIJozq>pTRx8TQqg0`Lo=|RK_QK3Oj5|w`QR#ArXC9#=S)Mh~Mhn
zOneuzdYvpQET^hh2SXFp@Zb+SHSy#4R|Y7h458HKeS0v!5nsT`i_X(*unz<qZ&rQf
z9Id(B(Vb9Q@|lusP5cI*A>>tjwB6A16o%`O`n4f;PHmF2TiHMES85PaPj7N~KB61j
z8FJ$&vmt<HBQhA=HM}hT7zwb(O06J{T##vHcjt5gB2U@dk|#7blVZG!N(JJNxY0rO
z-u*D058jII2y$=XV)HuSlbbh(+FwDOR7Rwf__!}ek)rCTqWHY_D^=RoGP}8%Ut&4>
z(+&{B8#)uAITQ@W?-M^a{&uxJcy8oREg~_U`R*wmUYr|$P-qtomPyvBEG2E=W6(};
z1af#}Rtud!L1g`7ezxDh-B!OgT?I_Sf_aLa_)<pJl&e!x<TD!9OK-kuduHnVrbB-o
z=#n0Y*AUJArQTF$B#{lR9eD{Y)Yh1yEoSG^OQWnKS@o$~KK#M3jx<n5N}4Qqo~5>G
zHog*BP(GN)UN3sb3ZSv1St3!W7j9@v&e#LMW5Y^{s~0%)!tB8G$Bp?sm{K-<=aF9P
z<lXL0n(=1XnbCZ!s$Zeiaw11+%tQq(GK+OXZSshFfPvl|z4QIS%^j}N+zF{0)&OaF
zN|<AVEDTnCRI?5o;!+umU5N*I@78rkRH5RWDrjrGuyjP|<T|lv`taqG8n!wYS3f|p
zI6*<R7ml0pO_AG?6xkcJ?(d(+?O-KLS%H#qbr5pE@7nmJ1ZQiT@N$&ka7kjG5v$@d
zhsW0)a?j`U0L*IoF4}vmyQ!PR#0PXciBc#c!qH2Y`i?d5@bpXSDUn|%GB)<ToRic3
zZ}CCk#k5$K-`J3!Tl2dSH(}$#6t2&oWen+MqG0yoFm?@@yW_F)L<kMty2)BB?R2*q
zndIR<g)CQb!H6ug%a176SbDWS&y4(*KbSKopd$3SPd>-`j2CTX8_|EP9qVp)h_UAH
zQw$Ar;>*?U++ox*UQZO!ByqG5LW@_Lrec>_=XiWc(>F7He_*9?!K7jT{wD9vPZGZ)
z_q69r9Awlsc;HPw4l}8Ky-E?)mV)4!5TKxSG3<UY&H7^cYiDxCuf>z&1uS(jm&|-)
zUEd}LD=T}Fm4)*<e_`(+l7yM@c}&wNM3t%!`7lcR2E)Em)C#oNZxFB*%djVt5#;F;
zI0vw&>e-O+Wr-p-E#xQdENmC{xhs7et&8Y~#wa~HlOMX1rTe6XVeP#~BI(L1xFn&5
zm;qkN$&XXsJG^rH)Gd$oN=hoB?=ruBK5KcO@#CYdOn!K{kFh-E(@tyt42>K?%^_U{
z(WVauY7U0{X9xIjHaBvU)gK-(L5tl0DUICTaqnuILxWC*wr4xXUqKapC33WJBA<$x
zF%Z;<+q&C}tCb4(v!8sZ4L}#FeakI^`?a=Lq4JgsY>Hs{<OV|H9j6uKJ_`macuF1o
zL9@xTintA^Z1T8puAyMAf(`vLvEgq6k#ev+98+T=h0Htd=nbJ0B#XNsZSJ5#&MWlq
z870<ujPKAIjLs#S8kmUC=Hfy{7NNfBoZ#$Zde@jqgEW6j9oz5;VKcXr#e6X~s+X9s
zm%s#MZ>u5tOG88FQ%S&GhFOM5PF1;EQ@G3h#Qva!{M0Vu;9yt<N8RIK)X+;_=Fjy8
z5B5W~szXwnUJ@ZA&r<obJM{<i8#mSBniek>xj1RxHqMkUHScQ)w>|xnZ@QD?g2)a;
zcj&RK<z)-s5}0mDX=JfmI*ur~IybTycwcwf9S;n1*wy;K(B3RMXub7tv&H=?IVoLh
zKxTaLt97^kYZj-OWrlhtVN;24R83!72!?%pI(82`RX#M`m~;?Yqz^U~jx=PrZ?p2~
zMAtCon5;j`EGgEDxd_e@H!#I{nR5f#qd-|VaYE4M#Y(b`!|X@%u*(f<4YAhSdhU8^
zj=o)u9vKd5B-k%`-GxDJs)=fY(!GK8q7-6ydhaGEte#crUGU<%(x6Fa+LMp@LzY}l
z0U9E<RqQET(1d5O)rd!L-vx1xX4Ti%8@4J37%$lMY=MNIgR%6eM)h(Px^@lhw0PBt
zn_BJwvjlD%#G@%G?K$a|DKOy)h_D~J3_8JD<P)z4k{BuSs>CFjZV50t!FbDeDfBIy
z6*Czg?z|h@`SyGQ)@OBAC3|-VWlK#m*-MS#FuY)TszQkGEpOz6uo!_&WU)rz0oog>
zXhE{UnUw^q-i0O|7Ic=3{RSt!W*QfY51}XQAbRwaE%MnOU63sk@c^64Kxk5Axugz>
z9J}epundJu(<KyAHva4l8P;XeW~Ei4cxu^*ten`dv2+Cao6Rg0e`9W*7o7$<T6@lU
zr8-qxhHo<bbXjZNL=MEgdGGm5hhuR(Nxf2Mt|%-l5H-1D(&t#a>*U6{Yo(W~j!Ml8
zrm~PYe>QF+T~#eRen2gi9xCDsFN0z0U6cs!m^Vq$T=N||Tmnb%T`ozbbUoE;OR0@r
z>AK9>U&IY)kEDPyZ=N>Ft9DK*u+#M5@UaXjXPy~7r1~`GAu7ln>&-^CTtD!bHS9@}
zz}2@xM5Qko0d1l8(FU;^Xq|(6)Y3|TA(h$t83cnV-^KXrJELGe3vm*@Q0X;!j}!TQ
zY;QHYj3jBzxuitghg4<rVM+n0u{>6`pfeLjoM<T){m1TQ;g|Cp{ncd&(<H2C^p!L>
zuM6d+wEcp76&$>8=2`bgOH_px?bpP!+oO33xF1~!CNpwf00Et3NZ3fpG7L(`a;rii
z2ReZXILsj?0}N{|qhm`bLt_Osg01dHQlC?q3uQ-i`6z#JYTqC{CZ^NVc?LQdF4%mt
ztRxKUFrbe>XI+sAtu0ttzJ(2oNWoT5ZeNMwfhn$JUTU{gvq?6cwQIuw5X*Tcn%UuL
zJij^Uv;Y3y!dm_%QVz{IAgRV+iDN|IJKN7ZP>A<U&RKO}?Pv9tyd%oSx812<roQ0i
zPn^`OTHA49F+jIN;j58(Xg0wR8WX$A5}f?@r@Pndj!)oB*>vpJ8ROnQ>~FUw_7Sj%
z!cSn^&cCt{9J2A80CS@%rhS(X5fH9X7~QI9-yM|jkmwEW>+9?IZu}zCrci(?oF6wE
zRQp_Oh@-_a=z)o0S2I7fK0^h0TZxC*sE0S{Fo5+YG32CC*Oi~nVGPy<8E7hWbG^44
zA&cA(BWPwP)k(oAFl8?X=Ux%Q6vAuWgkO+8Wm`Wfm+CCK*VJ317)GPjKlu&nYoeK=
z^n~lIPeq<e7y@<1r+aKe#gS&;qwdYg!54bQL{pX#o|Y2eHEajML_3Nbj$$X~t$rY0
z4|Ha2C_VG{qF}`!Q(zcp+yNWqk)kEE5}yiAXs1-qq(=Ay`y=9E%(?37_Gywqulh=S
z=;TkP4q`o|4nPNzn~aMYk>$P$r0H<Zbo&{qi7V6EN(+L);e_fqIYK1hSVzZn7%>}>
zY>++Q+I|hJ$={|2VH5R#iBD0V^Ia(|$})hzc?5HKp29)&8;nOTN~jxeY)aBlMC{>D
z*^vhE`M?W%LuY|{4qMe$FL!4BKqFr&C|wwQ@d@WL=j_(c`}%mF5UWO>05GBHS_#f}
zStj%xIo0#~bzwAM`iNh_13M4}9BD_p<f_rybX-=DuRx&xTQgPZzSfC#p0VkN5mHy!
z6YuIto$JmwPQ(6rO<So6W_N=!7ySi8!$3H9j~0R}o9pAXoxZ+<!m*0e+Up;s-pTxW
z!F`0H6Fx{MBg%2TT8+V6O2NioE90fBMI>#l>y;_^Zkis<JUmV!r<l9UONZTltxC@7
z+MAe8m3>J4-1K|0=vOcC>HRf<pzrgeu%75sKPE#`U*yU2_cQbA>jwu`?Dm4Xx^3;L
zb~cezduBa_oA#S8<5Ad0=9keZyhyjbN8fd`Y8_+^n%?-nXAVi*0G^&0&ME(>!tuq$
zy`3hW(OjQO8y5}`iiMnZBCsx}_;L0N_aJngNHO*eS5=BE1dMxH#5GSpn&yU*F5^K9
zuHaRszy%^|x1HTxv-N*<PyPa|vACnG(La%ui>_8S+lFw+85%Iw$0agh9%frwwRH)$
zoV!6DASlCcA6`<OS<-0q%o#=|swL$|ccO3NMr(a}%?HSM6!0$|A*lpGQcD@x%dLwq
zyrUR>m=|-gMbxJQ#`A+soO&D^BNj1C{9}0>WnGmEtxaqkwcz<Rh9d(3a~g^qQ>W!{
z)hza^Q5)MmTi@H=tzppzDtV6M2ARkF6{TX!L>I>>x1L-j<z#Ms!7~Mr-noR+j6Q>W
z*Er>lIdkqc@-gr4sn8JW;3D2o{!!z;W}KqQi(GZbwsd}GvWp;$-7-N3F^5stR)3q}
zdC_C-(lZVC>g(JtUX%sS{OSPzB|`z3g$r}$FRr84-?dY(DQpD2+&_F~z-Z!auJ8U1
zjb&?FB=kEye2uO(CudjsfTyfI6r@-}+XyqkESG0uSj=_zsY96cc6?(?QNQ(Zg2{%(
zd8OpNqkUb{;EhHqgu{r-QXvH0Uom*DlN>47|961m?x5_i_b$gS{1of;x@L(|{44?a
zzY#nFPMnAa+Y}DMW?>Fpc&|%QWCx$rWaE|@=~<TE{~ktmx~$atMP^H^s~2E^e=(Y<
zmmDv_tL2ILi6gf-Bl4s{im={R&iH_2I53HX4NHoNBNROYoat?hNWG_V?vQ30qgY)#
ztUo^;|4a~O?U|)|#KAQ*&GY_Cy1r4aj-CzC^JD+S2Wen86-Vh#1`x3neWd$3$CSoM
zq1r7U+?ifS$04YA;X%OKjr+vodq;)!q5k1gHpLxzjS1gsY4WPTUJ1$+s=u{{$)Me}
zS7ImOO5>&xc5^X()3-i9{ZO2%qAA0$3RgFDa(`gwnd-+2!}~{2Cex9`dnqR!86U3+
zA`c1MSyx|z$SkTJ>u-YM(AI$3FD`C!Ab|=kH+pAu9hDB<fs=;*By7AXHQB2HuoL`|
zY5%6JO=ZJsVSKKX!n-=32eVO4|93UIhU1dA1*fX+{N6l_u|LGwMEjvyvs%bBk@i(Z
zoNg}A^>nfIk2kE>32tTF(HJocRlxx)`XKtUZlTGhzG;dR9T#j5hc;_)4u6CfbS7uP
z-uqi}cde+7fwSmS84c<^W}ni!OyUQv$JMus!nZ2g@#MA9fT~(<+W~WjN<kYLWUn+D
z{j9+<V!)mTmSVd_&RPH0HR6xGT<GpA-#`Cs=|J5ej>gr5+8gCapb8rCZ9C?!`ZZ<O
zKA0tWZ^Y(Ccr7>6vot=}=KVngo%BXjRgX08iqtKa@k?DWOK8?FZ{`mI0vXb4_{^6p
zm6l%jZ9u_^;IO?w%{nZwSdT7uQP<a17NcV?hBuv#*;{%*q{3=b!myW}zHfXtw<9}4
z2=yuO<#5yGdx5z;mNuNy-dRV{b#mBakjdY4!$j3al7>ctLu~=+JjU#EJ?HhH)B{ci
z=r4w<B;b!mUeH3EM(_a7Eydb4u}l${;g#wr`x0sag96EiCs8b0Yb~$EFd<X!RP|LY
zZ(;;Feo5d1QI@|O!6V~gOec+DD^cQaNLLEd8k8_x359D?D*=*8QrB5)FprZjGb5Sd
z7mwke$;YR&%XJLv=@G7WElCv!xF_vyM>A)YHs`M3U3?vufsk#Id&jO#h;Cf{TAD2+
zcK6U3<a7AbKA^*`6!isCLYDurKDB$D&CI?6Ynag5JY`m#X$}jJm}rnsim&*h&}luE
z?1f)20qcXsCU(MA!kun<ec)8=QJ|;os<pXE$%!uzvyKDZ!R34$fuO$Sr<3}^tL>$S
z2nfQ?0YOtTi~4&6L)Xf%;<LyJA_>gqB%Euy(yu0lUDDP(*nHbzSgzV(5TD{IJxj5S
z)8lOOYPEiTyj`#m(8)46wCufR=R>MSxWxrF{to3ytx$XcLT<prD-kzrZSd6cwJ1{;
z=r=rh9^Ji}_R9_h7x58#qiucQ8Y(l*4pLb@MFlhHg%3|v>pnh3Eks|mp;GIKHZ6qX
zAxHNH!NQXtaa+%Jollf@;dn=t+}h^edKP2Df3~?8wpQ~iDFd&OJ#pJ;_&AnZWe#{S
z%|T}dt96&N!^RW1Lo<q4WPC#MgYrejeTOqQ#h&g<O6S)U1m~I(I~m;2x!Voh_|Zj$
zC_wRS20r?HtgZ6VSpB9dMCZIyD<+Hf#9L^0Q`h35y5SQZcCe3${|Y^<y5mpK2gBMW
zOpe-{Hbk?*R&9Ob?f|xQp0_LVA7G{>^d?$Nj`~24_6B(#UPE<eN)=%caBuW*YT0hC
zP#s)Aw;4}vcBgqk>c|GAMsWkF>uA`0t8|!FNe<1*iWeV^`?v0iU8ZpfIr{p@voHEz
zXy~AQ^=sKu@n<R459F`+1uSK-%mrAMIgITZZ&)+Wgr-X^uTq{IYMtQ)XUIe!>1Taz
zg!RoP)%C%8+S(R_#go*K*yijM^GuVObYbcq5T6@{wJl>zG7)mf{lq+py)HB&N%krk
zHA)DV!%BEtQNMld+#!rh<F~^%f&1BPx-6I{&QvY5cF}O4)pBw3n<)^ToU@$!(M=IY
zJ-g7LVaSKLfgh~&!v3HI5lk7&4v)%0M7w~x`SGSsvi^|H-LjTEiFk!YYbj}+LwKZ0
z=!8BEj|g}(^dnL=gkn3dz}EY+5mN|~#8ETeGs7d%BY$bTa+`RnQoMrCxPq9rjH<bn
zT_ncgbpERR%AT@srKN>tNN3w`Pfm^~Y!5GNmi-5!sZx{{!yIPT$cY=dv2!t~9LI4i
zcA^!kco|dlQWT?7VVcO!2^UF>G$ssLFbXAK5}FJtDFKfId5jDX?&s22T19B*xRDr&
z`r#3SdX(pr6eFj;LT(Np@sc9l_uSiZ(ajAUcL6qVzoSX;*--SE*ceiMAo}n80_az9
z&Q`x8VmzTaok<-Ec7!2SCEq{4m2Z1Lwr#Sn6~@{T6mGq>EHx2A^l6G$8x1K9Rd^D@
zD$QZVnwDsn>2;x4u%3(miC2pA;}Hcjhw8`aM40BKl`&Z=%qPc$xNovcHb1KZxoF8<
z(&k1=R=ypA&iIc&s;a8Qk&--f!>w$*E}L>*BalyaDKUCthk-|gW!A@cj)CaEXnnkN
z9X$$<&$)}4p-*?$84P38GxuxyPzRt5b_E?9To2YQw}sz6JXHGAXqOWKdHS365ISJj
z5QG6IAEKk3mU2&5%e%t33P!mVpvOsZz8{;dsLfi6|76Yrde)_B&!yT?J#Y*#R2-U&
z`jZ+DF7KDfOh=b2O4LFv(C_Er&%+T;RIg9L3w$6|3Y25ylZLOXmnYOTy98JPt6VHG
zdF9~a^GE80A4`h*mpta?qBwBcrb>15xvQ+JE`O=Ugf$aYW>op5-K|BGbJdM`56Csy
z#3P|?l11N4cr&Qk49}9Fn`&DAc5bWbXHdo}>Q~o<!Bq~?;$P2KrJjl?i(@~<LC|_@
zqh;w@e?95B6EkV*Gw#?R?`1PNdJ=`Ui@C0g_(l3XL#wf3^Sapkdz(*M#JsP^3BoKc
zmKCq&EGa!8rUI(Kio%XyG5o2sm6aWT2afC5#mgzR3*eOQCqvIrNpNLH*Cc%^q<Ec`
zu^`$U8JU(|F+QR7@J{Nk<eUrC4wr?-0G=Vs3UA^i?Kih?MSZ3}gh~RrZ5&M7=VdJj
zQ)xq?WAb`mlk)!DGVy%oCEQ<qRPy`DH>Z21ip7p=(ya=WIWVXj4mS^HS6|~clD_1L
zCCXtB?I1_m>ab5=!!cTyc@r=3R!T#YBR_6096oc-7G(=xS;`~tiH}c~o6_c@;jTY*
zhgKJH9(pgcfdUBXRqF}=Vv9!WJWL>%gOCUrcCr@RSq&X!t0*<nGk#I@pdGVe@!&h*
zjUwb(c9o!vYsxOFI*agbF{KN7#$fXm6RNB7Wwn$7&B10WF=a08Vf(xvG2I-1q2C^C
z`4UxfH=aZ9SfTQo)hY8=)Z*Lx^?Ytfdx6C`alz)39#1*;NPDBdxWowt;fVD>wnCig
zJWuK{&*qE0YsG7;<8vn{Tpa7{uG`HKi@=vS8U>cZ5!0Q7@3Px8omW%63csLE9J^0~
zg^M*?K=sG@2sxTMZEqAqxq8ro@}Y!cXVshZ1Jt`Z{N2bHM|FIyqEZX`sWs7BW+lv5
z8f9@w-Wgn|YP!C|;p#wX+UyM|XD5Ylrp?+y!7oM^UjmIv1_$o~mx9A0DWU)fW8ST$
zKZ{Fu1BU5U+~4OvZr2I}rY^J_*mO#4I8N&EMxcV!)9|@yHk7JIUJmE1o~s*wJRX;#
zn%&bm4TJ5TWACymP`PpXDMwDv4?S6Ugk7{&Qk&;i_LWwg!tBdekuJ4o-RV8~TrM{Z
zGy|H7s6=5yWdSIol55g*5hzMq>>~337ZXovChnCK8x5ip-E+5?C$;_gB(C>QS?}4n
z^lIqz$TEfGq=!yTJZ2w%O%D|a>^9S&L(#dCp)(@+_bNC#9u>m$;nekU2p_8H;m6ad
zemx#%Tp9mD7i+<RzEb@ZK9&Vi#(IOR9TtpTAUmiXND&9xDm}nvEGvNP)U{_~#QZ3M
znVJz&R^ufwCm<X-%GE`i!MU!uq2Kki@w8EO>QR|4O(3WVZ{GdBvmi-(`zTMbVasHz
ze*g1PSPxY%Ay1ep_EZf|b+-1gPJSG^XtnXmb`u@)6?q5lTq}n!AR~>%^s`F#{gCsH
zy=ZD0#twIFI``a?j^VJ05r09SttwOBQ*tdSx;#w;GV~HRw+Z|WI^2C;M&td!fp$0<
zl-ZWyQ|u03q9)<Nkb7%slE;2zcGJUV!j9@{nr+jiQH`1Lcg+TeFAMV+oI-t6^5D#)
z_)yUAi-DuWR(}Fc5|;@E2_E|jVR>!w-8KURXurAUgl6j&6aCqMC!{3P*{|b2w+5^o
z)U?9F3SdSicpC{%?zS@37c&K}L~1}$DX12Zc3#~=|C?YGto*!yC_r#f0;$CZ#B3nR
z`p0>3=sB`edJXDi0#cCDC{j!?Xj+Em;}?BCqsxt@dY=9?TY>2|K1>ftQ-Vab!!^xz
zw2Vx+nEdZ)s`wH`0*gxNo$$QRVt}kSDE{nOdA7gSW8Ata!jeVkRNl9G?rC)@t%%E1
zG!Eus<9ZmEItr@zc`WJ`OIyYlq%#_&@S|)<8E&&+vI;28OsBu5j!D_UOg_C_w6hho
zm6dJ}@ro~Us56b%T-$P)@P0v_Kv@#r$xw?kOLI}RzcXI)Jwa->S_@BKlb^t&VeEVI
zt{F)Im}sxbtM*EW6Bl{;2gxt?oj-lH#1~bno`t+&^p|6yPmhniLm!<dQqNQNeeo|w
zTCTZlCjuaip~iZiC82cmx7C(*D9K()ILX$#a;&-V5cfi@vU&&Jqu5A7gWQ@{)Ai4&
zOEt!(OFST&fi(aICq<mhz*LyD%?#+}D!YVuysn6e@%V9>Vf;DJ1bKv3b7o506JNte
zak@=5BQSbHvllo{2G8Ipb+$Nb7%-BEXh>1SY+Jp0{t)O=<h;q*UEj5`e$wU0xu+FB
zOuTFx!t!ghP1C{d>fIFoNety!{b$O63b_<z?}p9crZ!U(%0*9tI1&35SqP6PKn!|H
zkIH2<&EDKvZbmugLO_Z5r^{)(J$7QxFVb5NUXsdC2c~n87Iv@SqrZ>^aMV&RoHhhV
zGO0yuuYX9Iy5uz<QZZ>lX2<R3+Q%p9MaD{(uD|fLK5RpB_6{Iat~^m;8?}o)-u<Qg
zEwS+OkWB6sS--FFM`p8NuVimiUn@I@Ig@!!QtB<BmzA6@O%v~^iht-`*2Mj&?vk>4
z5og&QCL;S9hlxk*#)^==lFmCDcSD!b8^~2Vaay;a-shgWIGvgvnj+&BybT!ij;6G`
z6Z3~Ea!?YRjtvK+e#v()l1bYlyBilNvo;(tq3`oVh_E}**@#&j)3{$)-o3%K*R^}_
z#u2B79DCw@x^7Enpbu_9eQGvvqmf8z)VHE2+DZCr9ge1{rVr%c!|y$ha+tqr6R;GH
z?UFo3Yab|PTV0fRzG<}F>iYzp<xkUM|E3l38UGQUkxUT(8L1R%0s6+^l=3pV%(MTE
zqPfca#-N@;0i(l8iQ?FYi`BlJoo56*WPq5H%Nf4}V2U4Q<iHAFx}X!HJk7WhQ!Y`w
z#Br)ziRTKp3$4P$$aJf#aWi9K$*r96z~fc+82y>6OAsE#U)CT%%LC2d53yt<_7bC&
z;8?_4lTll^<0Q-ih1Pz}2m}7i^Fm=20wI`M_RPeecv;#x<FT_RA-wRPBFk+>X;Vsr
zE38IoKIx9!6OLnwm@%{dbps*2)yG~Rg)!*6^%$#Q+MbH=?-P7>F9&+uy|pamPtCoA
z4l*`%-XW$cyktvitqV01r;{W<o5#oUL?af~DpGlqRX@@CeQ?OV05s4M9eIVlI=VuI
zsz9r15nIK5QIhoau8(KG1`V&jNSZ3H-@!|p_nqo{DlCFEwOJqbB#((0@bG+j537Pk
zAXj#uG~p2v_wE*_Oxn3LpyeP>`iGEoM$p)a7vtifE22Jy{jM~>$`5^k6dB$SxL!3`
zN*|!;1=K2q>Lr5eGkv+K)!?N7^fLmZErlpHE0z5utI)cw82tP54D~hd?$*b}zV2>=
zs*W(#H#-APCl(&z5Ba5h=*yLS=VtsF&EKqh-t25?N9})v68XU7q&T;~?p>G7{oD8T
z#J)eAwo$V|ui<h4w+21eBLjl6W{n5qKhoTqYKSlf;v5H@YmD{|s&}eA#7pi#%4hI$
z1gb;d`@Q7_vRWU~Auwrn5yFq1&fdq^df!B#nNdW!Nf#ZRyZKp15%|_Eble>6ltr4S
z#EzfMd&!%Sf89kX?)(ff)*b>!8@B0$)$16b9lrR}G3<k_^zZ1kDg-hPMtG=om400`
zNKVxieD!q@D^P4ug4Vw$sL$;?@#|}}Y@=iyCEGF4FrF^5Y4cMcxi7yc$=WNJs1}az
z-rX2m;zJO+a~x*tpTD}g;wHJT%Z>RG$*5`Lu<*QNRziib4u{YYGPiQ1+8tus1C}(Q
z`I*7795k2VI-bXZ$CBkG_3JKBX@acET@{jur-&m*6N{wZNy2+Y(qNb7dcLq?X2|^#
z)9<JSSU+jhk1fdxu5~-sL4eAR4=ql*IpMk#^rwjoz(uv--|jUGuaXNHyN+gJM!U$d
z)mj~wpIg1|)#^1ifHML|Lb8E;z>F==TNig2M*qT5fw;~ZOdyOeK$2WHw*DwVU`~`2
zn_*O<@?-2l%ESphX>@!AZ<gS?uHt&+9Z&0paTLCk-?U-h63=<gK}@Y~P$U~J3}b(G
zS^37ycQ&(MKo0xFAH&7+920sJP>@W^of0&@?J^$R*w*dUmPdjbrt?aa)(xK%wuc*m
z-qy#_k<$F<?l`9YZ<0>rJ-gw!1yL7%#j@t3xFKS9h-04*Zx6ONL!7@1Ay~;b*sZ+D
zI=?vf+oRZujNSis6gnz~1jxE~set*Nt6DxZJJQs4cUV0ctDUM%E~sX5_BFsJ`q1A#
zmKhP}+~3YxOz>^WWr&kkiwFU?`|wY<Rj;Q@6Mi=01fc&d6*~xEMnUSufKB_T4U&Cg
z+W(3GUJ?w)%j&;}c}^I<Yq;Lin{M#c4sM8kJ$B>2EF^Dt>~7cr+o!yDrS<93x(u#Y
z>TQ?u^F9f|+E|vCY=oRzXk@n#KGO?gt)kln!!=476rn<CXsHnFGv123`)RDl^_7I0
z(yyVQcq0qE$wcmeMOi;s7d?e_e<KKs`wdPbK-KKjQG?*bu#rJ!t%S*JwR)fDHoXrj
z>>V-KQaF&Q<i`)p=rox7#~GsWmx$7(%-yy&lCx<UCUC7ALU7O*tWAen8u_tmhT)Be
zm+98aYk5fVp@Ckx(2v&h<0G@H#EN8S|0JKQpYQio8#}Kr!n&y9Bv}L)Y1ZuCHy8gp
z4u-)l%lg%r+!I6|_u~8VL<bz%Wifh$&8f4VBk(;Rs;zr{50yhcH9c?A>~gZBXI!5;
zIjK@p=UYOzegUuDV7HBOIRmLwLeF+a60uafACQAAd;Bmq-<Y~ND`V~|P{{J~8Y<ba
zpFmtS_=Z8QlPa=INl+=rtNKffgPWRT07c<WW;-KzJ5G+*k{&6~e=~M!J$}Taaa)IT
zcgA8kKe<jg`@-(`OpVFfh&H)jDrfF5L5aM>J2<#rnvWEHMiXo_8M^I^X+6w-`5CYu
zVrr+XZk2R@#fV&?+&1eZch15jCV^bM&=dP+vym&wQOnCuLOTEvT805y1hW(=DKaHU
zq6F&nl8{2F$=i(4*1G%PAMUFrySdUK64yHIs*p=xmhY?fU#K!!+ZcQe(Fok3#uRZ=
z@B4U$2O}F&A@vBl>F`e!!xKGVv6ckVQp+g<Fe^*PgI7)d-C#z*Zj>giiE(>DR66H<
zuQ_lJgsCS4-1~W37Zj}+CFgq?Bc<{{V(;JKD=TC?XLidS{(G)Ncnw!RSzX^}hDgw}
zR<q42!FQ|c4FTuJ=;UO#RgeD5-&1(<I;Qk3pHh9U7k1Z(y^mlr*1DqFH@XWb0ey4H
z%9$qAqe$cxKua~VQ7T{suEbQANaOMe?1=~{b0`k20Vgp**x|6K)iriW%*Y_nIw@5Y
z++mCsMwJ8F0Zt3TP1RIOHB>G%Qeip_Z3{RZFB08UsH4l=>Wlb&kX9xG+b<<y`BuAu
z3%1spp7|I@rckeEt)RQG-D!2;UVv^yKOOiPRqmimYnOW~YYRReVV!>Ni8GA$?G5uU
zc*yL-AnnaiPU-}iK#jqz?=^ckC4SP^*V}lJoVr%E%gF_-iqj$zzy7ZMl<aotYXk56
z5`jOzB`%l@BQNI6wFvgVTW;@g4k&oM^jwKe5H>2=#GRBE-;zcuDLy7s);mV$vQyOK
zYkCd0)k=lt*zvI7Sl`PRzccPsy*4lF$k9yO7&>M;?v)ldp6%^zv+Z!4o$MfVZtJ;H
zHYs9{yi$U)vRhy4X;n3~R7_3-4^4bUCF$kr;K%WD9m3X5zc}NhvYd(91;?FL{HDU{
z(!Kr9eHMh981YgnX=vTq&+BMYOYw|GcdLlk6#g41)2(=0l>r0n>9on25O!=v#Oh4r
zL|G_^yS9X;vZ%3BUgCo=MTxf8744$=)=r{!LHZ}pDD~@ffz;iQpb(mUzA15fkIUUV
z*xun39(cRv-`+@(u-bsPLWo}L(|1M4!LU1N>rMuWg>VGa*m&RW6hb6|GoWO#W;-Wc
zp=&ua2DMP5LwI`j8y?#){6E@zHFlr(xPb8n2NDUv$0##(`T5TdBk}@(LdM$iDQ<DW
zx?qa>8b8)3oOb^x9Nl-tJI&$7!XT|m77?GmX&(KE3sXcoQ|Cr(mXgy@ckO4<;Stcq
z&?WgJhg4sQ?nO*AbqN}QpfEVytXy)dEa-6^-K*RI?}PA+nRyH3ckxx+OUrF4UxQS$
zTM2D*<9D{4wWgh|Z)&s!P;a@Uz%b^lsBmh6QeBl}*H>H9uWwhLZM485q{!Z+^x7B|
zc6&eQQ6JbV`lhbmrtwIWe1OspTu~6D8^imm?Zu9|=i!tRwbMsv4gq9U5AbZ8IUEHj
zQs+xyLYJ+<0we1=<7+cHGxi=%D+~jyp#`CRiHKpN10`Y&R!H7}0~JF*Wzx>2A4(j9
z%?;m^|I#l^+ryRzr36A^w^B0_wfr5i36#(|FEl8}?3zdxt3BlREMn=QE=Wh*>~Fan
zo>C6fYKh3~$LGd$(RWtw;$Vb9YGHlWA4k)4o&p8vbR9wjM7;|=cjoSbMzg%nr9q$J
z3E52$T{cj~IaAvG3qH$CA^EdQxU4Ks+YB-ct}vdZra>Uo?I5h_v8*!?j3G~EZDBjy
z!-=GBJ>uzU*D^u-iE><uquLqOg<Z6b;nR(F@Q*cIW0LL}dQYmJPXJ7X``8n;@A*G;
zvG`6G<B78<L=By^r61!fnmZ6GCuDQ0$<Wj&42_aLerHv*a8_-C-Y~B8CtFA2rPxq6
zR0c(vMzx~~!+7O6r4%&NT!36t_ZT&HwxbQowy_oOLN=OCiK(CznP1MfbU98f>PhTa
z=un2o36BpSCQ}w)oE=GrSyC&==2IyeN7}2rx$ynN`iut*^3QS*J{T^|O@C`9UiSgf
z31#o@R@01WI}~GYFyeDr#d{!QSbf8m^w9yST8jun=(Y}^Wjezv@Au{^vKA_RgMJHV
zcPzA8#vLRttY({Mv!$_0m-K&inA*Ev3!!Y)jrFGi(0Aw6gjd)`O7%gvZr_VAkK#fp
z-bX|qoWc+s|5!r%dM)_d__+ljigKpPhqzsoFd6QM8oEHVm$8ytQs%-h2Me9jNsX~z
z_(uAB)!|pz<xoc+WM1NAJZY~8qZspYlZxupa$dJHBB=n7DstAZOw*T-_U7Ht`Lkg;
z<@>R14`eU<p>h`X^%W)hybP1^=}@Qj=L7DE%A~RoBCm<c#cu(hfYO&#cki@u%ksdI
zZF!S6VT&^TunoY)SbkfTD|S40E<RM*<~GT5mGe7bjpMeAQ17-Cvqu>M8i57;AxEc6
z^&4Wrd6z<3yOLQioCLF>L?(Cfh!Ofpq_3V?Xeaeu4n3{s?MBUWi`5Mrt7Vqr03`{*
zGLDncor+VXH8?20%H~S{-Q2gKl&+Ne9L9<=)d7MgVZIk^J#beL4lmDCxUIzwub3sC
z3;A(@X#QR+<!8OsCg&C}rMA}UsNKgHoo)eMWZE34)7;bNQ=?nXc@D6t<1+@X4|@Bo
z5Xa6EkspfnbVZO0(;y2AVRh)sq*Ef0<NJvf@OFAWR>|;CAUSTgifuIuY@CC1*G3NE
zZ$l`sgRCvnl-<&E5%lg5L$9TJ5N=(*aC0R+Z=|2Ou<^NEU!GiT^Q4_Q&3R8pk_cB(
zgO_GIOEQs02&SXu$8(JQ3(>zXln>C^78qjj_Eu>O&xEr}iG63?g_`o8jr%>woj^$Q
zwP_JJDVHNCl$-T7?JM8?Nz=@?wJ!d#mkn&)u?`A;e%t(L^946bp&)yKNAk!|!wl4&
zUm5fwH^JD>Hj?ITVex_qt|}<P!nu!dD|k;>okM1oFc|bT1c|e8kQS0zxoo(QSs$9t
zu^!O6rv@hJ?r>OTJ$zJ9Rsc>r;F-mpOOQa#cW@2IBg!j7hlO@k35WBb?rqnmdlxLD
zKUS4cKdbuLdSyw5DHAQ)iGMN|;ytFC#zmU?k@mq8rG_tX&>H*=0T_iI*Kx0YfTj+A
zE$Q*A$YI9O05fjPd+<s1_`-$t3Rsw1-JubTuhB!<r}iU^BOI0-@8dmPRdIZgg)C<R
z+X9<UK)5q?=Ej_L4y3W`^<$8VH~hm_A->{~yQaChoLQW)@Fvnxe2s?uRwIS-=T8Q`
z7uV~_GAmC)XvdOI^cf;Fum8&VXuuQ#*Y1aZN;|aj`I}Ko82XX|;l}gmy<lMPPU51&
zMy9Dem22NM#kTZXA|#P6PJJLPEq6Dv9<winUr&1wdcLKGC+e{{I_;?JMv#P&iD@?}
zhTw{-@;WCkqfI$hK2J7gw5GolUW&SLmsv`qWoHy;j&Q@q+44h6VIi!!d@e(Wx{h*E
z%f)a0U(Fl2pR`i#d4;=&ro?&8n|;|pO@7IZBxT%o^eJJh_DGDnF?dopD2DlQSki!F
zG(QC38zAyC-=nNXd=@j|Im;~rUug;9uFS=jV7zXe4|bW@z>lhfTU-mFj`6Pp_beH-
z<0I3y1j6a(1xrwablEakdc^nV#i{6dM7QV1$DaFFjZp$0w^aKYX4)-`cA2eM{Q931
ddlz01NGd|SUT_b=e?AbEmQWBc|6maK{{Xxf;(`DG

literal 0
HcmV?d00001

diff --git a/wasm/test_page/package-lock.json b/wasm/test_page/package-lock.json
index 5ead514d8..22d229647 100644
--- a/wasm/test_page/package-lock.json
+++ b/wasm/test_page/package-lock.json
@@ -5,11 +5,21 @@
   "packages": {
     "": {
       "dependencies": {
+        "@browsermt/bergamot-translator": "file:../module",
         "cors": "^2.8.5",
         "express": "^4.18.2",
         "nocache": "^2.1.0"
       }
     },
+    "../module": {
+      "name": "@browsermt/bergamot-translator",
+      "version": "0.4.8",
+      "license": "MPL-2.0"
+    },
+    "node_modules/@browsermt/bergamot-translator": {
+      "resolved": "../module",
+      "link": true
+    },
     "node_modules/accepts": {
       "version": "1.3.8",
       "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
@@ -616,6 +626,9 @@
     }
   },
   "dependencies": {
+    "@browsermt/bergamot-translator": {
+      "version": "file:../module"
+    },
     "accepts": {
       "version": "1.3.8",
       "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
diff --git a/wasm/test_page/package.json b/wasm/test_page/package.json
index 79447e3bf..622b48c1a 100644
--- a/wasm/test_page/package.json
+++ b/wasm/test_page/package.json
@@ -1,7 +1,14 @@
 {
   "dependencies": {
+    "@browsermt/bergamot-translator": "file:../module",
     "cors": "^2.8.5",
     "express": "^4.18.2",
     "nocache": "^2.1.0"
+  },
+  "config": {
+    "port": 80
+  },
+  "scripts": {
+    "start": "node ./bergamot-httpserver.js $npm_package_config_port 1 0"
   }
 }
diff --git a/wasm/test_page/start_server.sh b/wasm/test_page/start_server.sh
index 59d455d14..5b6eeb0a3 100644
--- a/wasm/test_page/start_server.sh
+++ b/wasm/test_page/start_server.sh
@@ -24,7 +24,7 @@ fi
 # Prepare a list all wasm artifacts to be copied and copy them to the destination folder
 ARTIFACTS_BASE_NAME="bergamot-translator-worker"
 ARTIFACTS="$1/$ARTIFACTS_BASE_NAME.js $1/$ARTIFACTS_BASE_NAME.wasm"
-ARTIFACTS_DESTINATION_FOLDER=$SCRIPT_ABSOLUTE_PATH/js
+ARTIFACTS_DESTINATION_FOLDER=$SCRIPT_ABSOLUTE_PATH/../module/worker
 
 for i in $ARTIFACTS; do
     [ -f "$i" ] || breaks

From 1ba7461a36ed94423896d47f8fd8397e7265eb3e Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Thu, 19 Jan 2023 10:06:57 +0000
Subject: [PATCH 395/442] Fix compilation on x86

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 4b30c267c..69e27d298 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 4b30c267c701198cef4cddcd646cca17ccbb16f5
+Subproject commit 69e27d298419a2ff0e24ea7c43cad997fa8230c0

From 82c276a15c23a40bc7e21e8a1e0a289a6ce57017 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <github@kheafield.com>
Date: Wed, 1 Mar 2023 18:30:38 +0000
Subject: [PATCH 396/442] Fix path to example program

---
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.md b/README.md
index b70c818ec..eae9ef319 100644
--- a/README.md
+++ b/README.md
@@ -80,7 +80,7 @@ git submodule update --init --recursive
 ### Using Native version
 
 The builds generate library that can be integrated to any project. All the public header files are specified in `src` folder.\
-A short example of how to use the APIs is provided in `app/main.cpp` file.
+A short example of how to use the APIs is provided in `app/bergamot.cpp` file.
 
 ### Using WASM version
 

From eb0fe1b583d3c66a59bbbe1ce830f76a6d037496 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Thu, 4 May 2023 10:55:15 +0100
Subject: [PATCH 397/442] Bump 3rd_party/marian-dev from `69e27d2` to `8ceb051`
 (#446)

Bumps [3rd_party/marian-dev](https://github.com/browsermt/marian-dev) from `69e27d2` to `8ceb051`.
- [Release notes](https://github.com/browsermt/marian-dev/releases)
- [Commits](https://github.com/browsermt/marian-dev/compare/69e27d298419a2ff0e24ea7c43cad997fa8230c0...8ceb051b7f6388ed5edf7e1e2d0dde0c3cd7d737)

---
updated-dependencies:
- dependency-name: 3rd_party/marian-dev
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 69e27d298..8ceb051b7 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 69e27d298419a2ff0e24ea7c43cad997fa8230c0
+Subproject commit 8ceb051b7f6388ed5edf7e1e2d0dde0c3cd7d737

From fceb713b2749724bff1eaa8cafdd694b740f3304 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Thu, 4 May 2023 11:16:07 +0100
Subject: [PATCH 398/442] Update workflows

---
 .github/workflows/native.yml | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
index 8ee8c5c5f..41a91af1c 100644
--- a/.github/workflows/native.yml
+++ b/.github/workflows/native.yml
@@ -27,9 +27,9 @@ jobs:
           cmake: -DCOMPILE_TESTS=on
           brt_tags: ""
           unittests: 'true'
-        - name: Ubuntu 18.04 minimal
-          os: ubuntu-18.04
-          identifier: ubuntu_1804_minimal
+        - name: Ubuntu 22.04 minimal
+          os: ubuntu-22.04
+          identifier: ubuntu_2204_minimal
           cmake: -DCOMPILE_TESTS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
           brt_tags: "'#wasm'"
           unittests: 'false'
@@ -140,15 +140,15 @@ jobs:
       fail-fast: false
       matrix:
         include:
-        - name: MacOS 10.15 full
-          os: macos-10.15
-          identifier: mac_1015_full
+        - name: MacOS 12 full
+          os: macos-12
+          identifier: mac_12_full
           cmake: -DCOMPILE_TESTS=on -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=off
           brt_tags: ""
           unittests: 'true'
-        - name: MacOS 10.15 minimal
-          os: macos-10.15
-          identifier: mac_1015_minimal
+        - name: MacOS 12 minimal
+          os: macos-12
+          identifier: mac_12_minimal
           cmake: -DCOMPILE_TESTS=on -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
           brt_tags: "'#wasm'"
           unittests: 'false'

From 3c2a667f9b5b748a3808a78b373098791ed636de Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Thu, 4 May 2023 12:06:20 +0100
Subject: [PATCH 399/442] Try harder to install gperftools

---
 .github/workflows/native.yml | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
index 41a91af1c..6c5f56913 100644
--- a/.github/workflows/native.yml
+++ b/.github/workflows/native.yml
@@ -21,9 +21,9 @@ jobs:
       fail-fast: false
       matrix:
         include:
-        - name: Ubuntu 18.04 full
-          os: ubuntu-18.04
-          identifier: ubuntu_1804_full
+        - name: Ubuntu 22.04 full
+          os: ubuntu-22.04
+          identifier: ubuntu_2204_full
           cmake: -DCOMPILE_TESTS=on
           brt_tags: ""
           unittests: 'true'
@@ -55,9 +55,7 @@ jobs:
     - name: Install Dependencies
       run: |-
         sudo apt-get update
-        sudo apt-get install -y \
-          libgoogle-perftools-dev libprotobuf-dev protobuf-compiler \
-          libboost-all-dev ccache
+        sudo apt-get install -y libprotobuf-dev protobuf-compiler libboost-all-dev ccache libunwind-dev libgoogle-perftools-dev
     - name: Install MKL
       run: |-
         wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB" | sudo apt-key add -

From b3d36bca905a201f1239f74e5b0049db66065bed Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Wed, 10 May 2023 16:07:24 +0100
Subject: [PATCH 400/442] Bump 3rd_party/marian-dev from `8ceb051` to `bb65f47`
 (#447)

Bumps [3rd_party/marian-dev](https://github.com/browsermt/marian-dev) from `8ceb051` to `bb65f47`.
- [Commits](https://github.com/browsermt/marian-dev/compare/8ceb051b7f6388ed5edf7e1e2d0dde0c3cd7d737...bb65f473d535e6bcbc1a97beff5824397c0cd9cb)

---
updated-dependencies:
- dependency-name: 3rd_party/marian-dev
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 8ceb051b7..bb65f473d 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 8ceb051b7f6388ed5edf7e1e2d0dde0c3cd7d737
+Subproject commit bb65f473d535e6bcbc1a97beff5824397c0cd9cb

From ada8c3922490cc6a507bcf81fa4882b435595323 Mon Sep 17 00:00:00 2001
From: XapaJIaMnu <nheart@gmail.com>
Date: Tue, 6 Jun 2023 17:04:49 +0100
Subject: [PATCH 401/442] Fix compilation on newer gcc

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index bb65f473d..b20981969 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit bb65f473d535e6bcbc1a97beff5824397c0cd9cb
+Subproject commit b209819699e0725fa2dde4ebc98b7d91ded0c243

From eaa2562fe0b3b2bd9ac3424962ada33b7c3be2f1 Mon Sep 17 00:00:00 2001
From: XapaJIaMnu <nheart@gmail.com>
Date: Thu, 13 Jul 2023 00:14:13 +0100
Subject: [PATCH 402/442] Sentencepiece windows compilation

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index b20981969..6a6bbb627 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit b209819699e0725fa2dde4ebc98b7d91ded0c243
+Subproject commit 6a6bbb627877d40840b8b852eea80ddff22adceb

From e333208cb93b01e0ec93402c4448cc7b18daeda9 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 31 Jul 2023 15:26:44 +0100
Subject: [PATCH 403/442] Bump 3rd_party/marian-dev from `6a6bbb6` to `aa0221e`
 (#452)

Bumps [3rd_party/marian-dev](https://github.com/browsermt/marian-dev) from `6a6bbb6` to `aa0221e`.
- [Commits](https://github.com/browsermt/marian-dev/compare/6a6bbb627877d40840b8b852eea80ddff22adceb...aa0221e687fe8b3b69b5bb64279d4349663ad410)

---
updated-dependencies:
- dependency-name: 3rd_party/marian-dev
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 6a6bbb627..aa0221e68 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 6a6bbb627877d40840b8b852eea80ddff22adceb
+Subproject commit aa0221e687fe8b3b69b5bb64279d4349663ad410

From becb6e2cda6b76ac66fe5396de04ded3e20c3503 Mon Sep 17 00:00:00 2001
From: Graeme Nail <graemenail.work@gmail.com>
Date: Mon, 31 Jul 2023 15:27:24 +0100
Subject: [PATCH 404/442] Fix Python formatting (Black) (#453)

---
 bindings/python/repository.py | 2 --
 setup.py                      | 2 +-
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/bindings/python/repository.py b/bindings/python/repository.py
index 323b4482b..9667c7242 100644
--- a/bindings/python/repository.py
+++ b/bindings/python/repository.py
@@ -139,7 +139,6 @@ def download(self, model_identifier: str):
         with tarfile.open(save_location) as model_archive:
 
             def is_within_directory(directory, target):
-
                 abs_directory = os.path.abspath(directory)
                 abs_target = os.path.abspath(target)
 
@@ -148,7 +147,6 @@ def is_within_directory(directory, target):
                 return prefix == abs_directory
 
             def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
-
                 for member in tar.getmembers():
                     member_path = os.path.join(path, member.name)
                     if not is_within_directory(path, member_path):
diff --git a/setup.py b/setup.py
index 51161a3c0..ed4c6dc81 100644
--- a/setup.py
+++ b/setup.py
@@ -16,6 +16,7 @@
     "win-arm64": "ARM64",
 }
 
+
 # A CMakeExtension needs a sourcedir instead of a file list.
 # The name must be the _single_ output extension from the CMake build.
 # If you need multiple extensions, see scikit-build.
@@ -84,7 +85,6 @@ def build_extension(self, ext):
                     pass
 
         else:
-
             # Single config generators are handled "normally"
             single_config = any(x in cmake_generator for x in {"NMake", "Ninja"})
 

From cbfa839eef6715da0f356d47cfac55fe22700ae9 Mon Sep 17 00:00:00 2001
From: Graeme Nail <graemenail.work@gmail.com>
Date: Mon, 31 Jul 2023 15:54:42 +0100
Subject: [PATCH 405/442] Fix CI (#454)

* Use ubuntu-latest, macos-latest in GitHub Actions for cibuildwheel

* Update deprecated ubuntu-18.04 to ubuntu-latest for docs in GH actions
---
 .github/workflows/build.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index d0afe1649..f06b26357 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -24,7 +24,7 @@ jobs:
     build-wheels:
       strategy:
         matrix:
-          os: [ubuntu-20.04, macos-10.15]
+          os: [ubuntu-latest, macos-latest]
         fail-fast: false
 
       name: "cibuildwheel / ${{ matrix.os }}"
@@ -281,7 +281,7 @@ jobs:
                 ${{github.workspace}}/build-wasm/bergamot-translator-worker.wasm
                 ${{github.workspace}}/build-wasm/bergamot-translator-worker.js.bak
 
-    
+
     upload-wasm:
       name: "Upload node package to NPM"
       runs-on: ubuntu-latest
@@ -383,7 +383,7 @@ jobs:
             python3 -m pytype bindings/python
 
     docs:
-      runs-on: ubuntu-18.04
+      runs-on: ubuntu-latest
       needs: [build-wheels]
       steps:
         - name: Checkout

From 8011f9c849ca7351f886c55f9780d8583fb4c8f5 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 31 Jul 2023 15:54:53 +0100
Subject: [PATCH 406/442] Bump bergamot-translator-tests from `7984d14` to
 `a04432d` (#455)

Bumps [bergamot-translator-tests](https://github.com/browsermt/bergamot-translator-tests) from `7984d14` to `a04432d`.
- [Commits](https://github.com/browsermt/bergamot-translator-tests/compare/7984d140aef00489699d0b7711fa942816224294...a04432d7921bfa1dd62bc2e5cdca46b226f256de)

---
updated-dependencies:
- dependency-name: bergamot-translator-tests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 bergamot-translator-tests | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bergamot-translator-tests b/bergamot-translator-tests
index 7984d140a..a04432d79 160000
--- a/bergamot-translator-tests
+++ b/bergamot-translator-tests
@@ -1 +1 @@
-Subproject commit 7984d140aef00489699d0b7711fa942816224294
+Subproject commit a04432d7921bfa1dd62bc2e5cdca46b226f256de

From 4b0da8d434e5a688139255873afd177f647ef777 Mon Sep 17 00:00:00 2001
From: Graeme Nail <graemenail.work@gmail.com>
Date: Tue, 1 Aug 2023 19:35:11 +0100
Subject: [PATCH 407/442] Enables model ensembles (#450)

* Enables model ensembles

Adds the ability to use ensembles of models. This supports ensembles of
binary- or npz-format models, as well as mixtures of both.

When all models in the ensembles are of binary format, the load from
memory path is used. Otherwise, they are loaded via the file system.
Enable log-level debug for output related to this.

* Fix formatting

* Fix WASM bindings for MemoryBundle

For now, this does not support ensembles.

* Remove shared_ptr wrapping the AlignedMemory of models.

* Fix formatting
---
 src/translator/byte_array_util.cpp   | 31 ++++++++++++----------
 src/translator/byte_array_util.h     |  2 +-
 src/translator/definitions.h         |  4 +--
 src/translator/translation_model.cpp | 39 ++++++++++++++++++----------
 wasm/bindings/service_bindings.cpp   |  2 +-
 5 files changed, 46 insertions(+), 32 deletions(-)

diff --git a/src/translator/byte_array_util.cpp b/src/translator/byte_array_util.cpp
index 183dea3c0..c7515e797 100644
--- a/src/translator/byte_array_util.cpp
+++ b/src/translator/byte_array_util.cpp
@@ -91,21 +91,24 @@ AlignedMemory loadFileToMemory(const std::string& path, size_t alignment) {
   return alignedMemory;
 }
 
-AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options) {
+std::vector<AlignedMemory> getModelMemoryFromConfig(marian::Ptr<marian::Options> options) {
   auto models = options->get<std::vector<std::string>>("models");
-  ABORT_IF(models.size() != 1, "Loading multiple binary models is not supported for now as it is not necessary.");
-
-  // If binary model we load into aligned memory. If .npz we leave it be to
-  // return empty aligned memory, thus allowing traditional file system loads.
-  if (marian::io::isBin(models[0])) {
-    AlignedMemory alignedMemory = loadFileToMemory(models[0], 256);
-    return alignedMemory;
-  } else if (marian::io::isNpz(models[0])) {
-    return AlignedMemory();
-  } else {
-    ABORT("Unknown extension for model: {}, should be one of `.bin` or `.npz`", models[0]);
+
+  std::vector<AlignedMemory> modelMemories(models.size());
+  for (size_t i = 0; i < models.size(); ++i) {
+    const auto model = models[i];
+    if (marian::io::isBin(model)) {
+      modelMemories[i] = loadFileToMemory(model, 256);
+    } else if (marian::io::isNpz(model)) {
+      // if any of the models are npz format, we revert to loading from file for all models.
+      LOG(debug, "Encountered an npz file {}; will use file loading for {} models", model, models.size());
+      return {};
+    } else {
+      ABORT("Unknown extension for model: {}, should be one of `.bin` or `.npz`", model);
+    }
   }
-  return AlignedMemory();
+
+  return modelMemories;
 }
 
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options) {
@@ -153,7 +156,7 @@ AlignedMemory getQualityEstimatorModel(MemoryBundle& memoryBundle, const marian:
 
 MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options) {
   MemoryBundle memoryBundle;
-  memoryBundle.model = getModelMemoryFromConfig(options);
+  memoryBundle.models = getModelMemoryFromConfig(options);
   memoryBundle.shortlist = getShortlistMemoryFromConfig(options);
   getVocabsMemoryFromConfig(options, memoryBundle.vocabs);
   memoryBundle.ssplitPrefixFile = getSsplitPrefixFileMemoryFromConfig(options);
diff --git a/src/translator/byte_array_util.h b/src/translator/byte_array_util.h
index b445b3dec..851a175fd 100644
--- a/src/translator/byte_array_util.h
+++ b/src/translator/byte_array_util.h
@@ -5,7 +5,7 @@ namespace marian {
 namespace bergamot {
 
 AlignedMemory loadFileToMemory(const std::string& path, size_t alignment);
-AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options);
+std::vector<AlignedMemory> getModelMemoryFromConfig(marian::Ptr<marian::Options> options);
 AlignedMemory getQualityEstimatorModel(const marian::Ptr<marian::Options>& options);
 AlignedMemory getQualityEstimatorModel(MemoryBundle& memoryBundle, const marian::Ptr<marian::Options>& options);
 AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options);
diff --git a/src/translator/definitions.h b/src/translator/definitions.h
index b3bc1019b..efba3f9f6 100644
--- a/src/translator/definitions.h
+++ b/src/translator/definitions.h
@@ -19,8 +19,8 @@ typedef AlignedVector<char> AlignedMemory;
 /// Memory bundle for all byte-arrays.
 /// Can be a set/subset of model, shortlist, vocabs and ssplitPrefixFile bytes.
 struct MemoryBundle {
-  AlignedMemory model{};      ///< Byte-array of model (aligned to 256)
-  AlignedMemory shortlist{};  ///< Byte-array of shortlist (aligned to 64)
+  std::vector<AlignedMemory> models{};  ///< Byte-array of model (each element is aligned to 256)
+  AlignedMemory shortlist{};            ///< Byte-array of shortlist (aligned to 64)
 
   /// Vector of vocabulary memories (aligned to 64).
   /// If two vocabularies are the same (based on the filenames), two entries (shared
diff --git a/src/translator/translation_model.cpp b/src/translator/translation_model.cpp
index 3f91ebb47..6f8dd4dc8 100644
--- a/src/translator/translation_model.cpp
+++ b/src/translator/translation_model.cpp
@@ -61,24 +61,35 @@ void TranslationModel::loadBackend(size_t idx) {
   graph->getBackend()->configureDevice(options_);
   graph->reserveWorkspaceMB(options_->get<size_t>("workspace"));
 
-  // Marian Model: Load from memoryBundle or shortList
-  if (memory_.model.size() > 0 &&
-      memory_.model.begin() !=
-          nullptr) {  // If we have provided a byte array that contains the model memory, we can initialise the
-                      // model from there, as opposed to from reading in the config file
-    ABORT_IF((uintptr_t)memory_.model.begin() % 256 != 0,
-             "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
-    if (options_->get<bool>("check-bytearray", false)) {
-      ABORT_IF(!validateBinaryModel(memory_.model, memory_.model.size()),
-               "The binary file is invalid. Incomplete or corrupted download?");
-    }
-    const std::vector<const void *> container = {
-        memory_.model.begin()};  // Marian supports multiple models initialised in this manner hence std::vector.
-                                 // However we will only ever use 1 during decoding.
+  // if memory_.models is populated, then all models were of binary format
+  if (memory_.models.size() >= 1) {
+    const std::vector<const void *> container = std::invoke([&]() {
+      std::vector<const void *> model_ptrs(memory_.models.size());
+      for (size_t i = 0; i < memory_.models.size(); ++i) {
+        const AlignedMemory &model = memory_.models[i];
+
+        ABORT_IF(model.size() == 0 || model.begin() == nullptr, "The provided memory is empty. Cannot load the model.");
+        ABORT_IF(
+            (uintptr_t)model.begin() % 256 != 0,
+            "The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
+        if (options_->get<bool>("check-bytearray", false)) {
+          ABORT_IF(!validateBinaryModel(model, model.size()),
+                   "The binary file is invalid. Incomplete or corrupted download?");
+        }
+
+        model_ptrs[i] = model.begin();
+        LOG(debug, "Loaded model {} of {} from memory", (i + 1), model_ptrs.size());
+      }
+      return model_ptrs;
+    });
+
     scorerEnsemble = createScorers(options_, container);
   } else {
+    // load npz format models, or a mixture of binary/npz formats
     scorerEnsemble = createScorers(options_);
+    LOG(debug, "Loaded {} model(s) from file", scorerEnsemble.size());
   }
+
   for (auto scorer : scorerEnsemble) {
     scorer->init(graph);
     if (shortlistGenerator_) {
diff --git a/wasm/bindings/service_bindings.cpp b/wasm/bindings/service_bindings.cpp
index d56615dc6..54675a498 100644
--- a/wasm/bindings/service_bindings.cpp
+++ b/wasm/bindings/service_bindings.cpp
@@ -48,7 +48,7 @@ MemoryBundle prepareMemoryBundle(AlignedMemory* modelMemory, AlignedMemory* shor
                                  std::vector<AlignedMemory*> uniqueVocabsMemories,
                                  AlignedMemory* qualityEstimatorMemory) {
   MemoryBundle memoryBundle;
-  memoryBundle.model = std::move(*modelMemory);
+  memoryBundle.models.emplace_back(std::move(*modelMemory));
   memoryBundle.shortlist = std::move(*shortlistMemory);
   memoryBundle.vocabs = std::move(prepareVocabsSmartMemories(uniqueVocabsMemories));
   if (qualityEstimatorMemory != nullptr) {

From 2bdc493df3fa5109b2cd434a7a9634eb021b514b Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue, 8 Aug 2023 10:37:24 +0300
Subject: [PATCH 408/442] Bump 3rd_party/ssplit-cpp from `ad2c5a5` to `a311f98`
 (#456)

Bumps [3rd_party/ssplit-cpp](https://github.com/browsermt/ssplit-cpp) from `ad2c5a5` to `a311f98`.
- [Commits](https://github.com/browsermt/ssplit-cpp/compare/ad2c5a52a507ec5a1f58c6403fc674e76e92e185...a311f9865ade34db1e8e080e6cc146f55dafb067)

---
updated-dependencies:
- dependency-name: 3rd_party/ssplit-cpp
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 3rd_party/ssplit-cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/ssplit-cpp b/3rd_party/ssplit-cpp
index ad2c5a52a..a311f9865 160000
--- a/3rd_party/ssplit-cpp
+++ b/3rd_party/ssplit-cpp
@@ -1 +1 @@
-Subproject commit ad2c5a52a507ec5a1f58c6403fc674e76e92e185
+Subproject commit a311f9865ade34db1e8e080e6cc146f55dafb067

From ca954670aa4327630a3aee427668728b12b02df7 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri, 11 Aug 2023 15:04:27 +0100
Subject: [PATCH 409/442] Bump 3rd_party/marian-dev from `aa0221e` to `8dbde0f`
 (#458)

Bumps [3rd_party/marian-dev](https://github.com/browsermt/marian-dev) from `aa0221e` to `8dbde0f`.
- [Commits](https://github.com/browsermt/marian-dev/compare/aa0221e687fe8b3b69b5bb64279d4349663ad410...8dbde0fd8e690ad8791fb7fc94dba7674ee7c77e)

---
updated-dependencies:
- dependency-name: 3rd_party/marian-dev
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index aa0221e68..8dbde0fd8 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit aa0221e687fe8b3b69b5bb64279d4349663ad410
+Subproject commit 8dbde0fd8e690ad8791fb7fc94dba7674ee7c77e

From 534ed37a3d609f867a65c250328c5745b306a3c5 Mon Sep 17 00:00:00 2001
From: Nikolay Bogoychev <nheart@gmail.com>
Date: Mon, 14 Aug 2023 17:22:54 +0300
Subject: [PATCH 410/442] Remove wormhole references (#459)

* Remove warmhole references

* Remove more references to the WORMHOLE

* Update marian to wormhole removed marian

* Whoops

---------

Co-authored-by: Jelmer van der Linde <jelmer@ikhoefgeen.nl>
---
 .circleci/config.yml                    | 69 +++----------------------
 .github/workflows/build.yml             |  3 +-
 3rd_party/marian-dev                    |  2 +-
 CMakeLists.txt                          |  1 -
 README.md                               |  9 +---
 build-wasm.sh                           | 41 +--------------
 wasm/README.md                          | 14 +----
 wasm/patch-artifacts-enable-wormhole.sh | 36 -------------
 8 files changed, 16 insertions(+), 159 deletions(-)
 delete mode 100644 wasm/patch-artifacts-enable-wormhole.sh

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 140e3116d..52d58fc09 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -1,52 +1,6 @@
 version: 2.1
 jobs:
-  build-with-wormhole:
-    docker:
-      - image: 'emscripten/emsdk:3.1.8'
-    resource_class: medium
-
-    working_directory: ~/checkout
-
-    steps:
-      - checkout
-
-      - run:
-          name: Build WASM WORMHOLE
-          command: |
-            bash build-wasm.sh WORMHOLE
-
-      - run:
-          name: Check artifacts
-          working_directory: build-wasm
-          command: |
-            ARTIFACT_BASE="bergamot-translator-worker"
-            ARTIFACT_SUFFIX="with-wormhole"
-            ARTIFACT_FINAL=$ARTIFACT_BASE-$ARTIFACT_SUFFIX
-
-            if [[ -f "$ARTIFACT_BASE.js" && -f "$ARTIFACT_BASE.wasm" ]]; then
-              echo "Artifacts Successfully Generated"
-              mkdir ../artifacts
-              cp $ARTIFACT_BASE.wasm ../artifacts/$ARTIFACT_FINAL.wasm
-              cp $ARTIFACT_BASE.js ../artifacts/$ARTIFACT_FINAL.js
-              cd ../artifacts
-              shasum -a 256 $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize-$ARTIFACT_SUFFIX
-              ls -lsa $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize-$ARTIFACT_SUFFIX
-              cp ../BERGAMOT_VERSION .
-            else
-              echo "Failure: Artifacts Not Present"
-              exit 1
-            fi
-
-      - persist_to_workspace:
-          root: .
-          paths:
-            - artifacts/*
-
-      - store_artifacts:
-          path: "artifacts"
-          destination: "wasm-wormhole"
-
-  build-without-wormhole:
+  build:
     docker:
       - image: 'emscripten/emsdk:3.1.8'
     resource_class: medium
@@ -66,8 +20,7 @@ jobs:
           working_directory: build-wasm
           command: |
             ARTIFACT_BASE="bergamot-translator-worker"
-            ARTIFACT_SUFFIX="without-wormhole"
-            ARTIFACT_FINAL=$ARTIFACT_BASE-$ARTIFACT_SUFFIX
+            ARTIFACT_FINAL=$ARTIFACT_BASE
 
             if [[ -f "$ARTIFACT_BASE.js" && -f "$ARTIFACT_BASE.wasm" ]]; then
               echo "Artifacts Successfully Generated"
@@ -75,8 +28,8 @@ jobs:
               cp $ARTIFACT_BASE.wasm ../artifacts/$ARTIFACT_FINAL.wasm
               cp $ARTIFACT_BASE.js ../artifacts/$ARTIFACT_FINAL.js
               cd ../artifacts
-              shasum -a 256 $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize-$ARTIFACT_SUFFIX
-              ls -lsa $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize-$ARTIFACT_SUFFIX
+              shasum -a 256 $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize
+              ls -lsa $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize
             else
               echo "Failure: Artifacts Not Present"
               exit 1
@@ -89,7 +42,7 @@ jobs:
 
       - store_artifacts:
           path: "artifacts"
-          destination: "wasm-without-wormhole"
+          destination: "wasm"
 
   publish_to_github:
     docker:
@@ -106,18 +59,13 @@ jobs:
                 name: "Publish Release on GitHub"
                 command: |
                   export TAG_VERSION=$(cat ./artifacts/BERGAMOT_VERSION)
-                  cat ./artifacts/sha256-filesize-without-wormhole ./artifacts/sha256-filesize-with-wormhole >> ./artifacts/sha256-filesize
-                  rm ./artifacts/sha256-filesize-without-wormhole ./artifacts/sha256-filesize-with-wormhole ./artifacts/BERGAMOT_VERSION
+                  rm ./artifacts/BERGAMOT_VERSION
                   ghr -t ${GHTOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${TAG_VERSION} ./artifacts/
 
 workflows:
   build:
       jobs:
-          - build-with-wormhole:
-              filters:
-                tags:
-                  only: /^v.*/
-          - build-without-wormhole:
+          - build:
               filters:
                 tags:
                   only: /^v.*/
@@ -128,7 +76,6 @@ workflows:
                 branches:
                   ignore: /.*/
               requires:
-                - build-without-wormhole
-                - build-with-wormhole
+                - build
 
 
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index f06b26357..830924c2c 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -236,7 +236,7 @@ jobs:
           run: |
             mkdir -p build-wasm
             cd build-wasm
-            emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=off ..
+            emcmake cmake -DCOMPILE_WASM=on ..
 
 
         - name: "Compile"
@@ -276,7 +276,6 @@ jobs:
             name: wasm-artefacts
             if-no-files-found: error
             path: |
-                # Without wormhole
                 ${{github.workspace}}/build-wasm/bergamot-translator-worker.js
                 ${{github.workspace}}/build-wasm/bergamot-translator-worker.wasm
                 ${{github.workspace}}/build-wasm/bergamot-translator-worker.js.bak
diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 8dbde0fd8..300a50f42 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 8dbde0fd8e690ad8791fb7fc94dba7674ee7c77e
+Subproject commit 300a50f4251d978dc197d15bb7b296597b1eb221
diff --git a/CMakeLists.txt b/CMakeLists.txt
index dc51acf80..82940de82 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -113,7 +113,6 @@ message(STATUS "Project version: ${PROJECT_VERSION_STRING_FULL}")
 
 if(COMPILE_WASM)
   # See https://github.com/emscripten-core/emscripten/blob/main/src/settings.js
-  set(WORMHOLE ON CACHE BOOL "Use WASM wormhole in intgemm https://bugzilla.mozilla.org/show_bug.cgi?id=1672160")
   list(APPEND WASM_COMPILE_FLAGS
     -O3
     # Preserve whitespaces in JS even for release builds; this doesn't increase wasm binary size
diff --git a/README.md b/README.md
index eae9ef319..05c3c3d25 100644
--- a/README.md
+++ b/README.md
@@ -41,12 +41,7 @@ To build a version that translates with higher speeds on Firefox Nightly browser
 
        The wasm artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
 
-   2. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
-       ```bash
-       bash ../wasm/patch-artifacts-enable-wormhole.sh
-       ```
-
-   3. Patch generated artifacts to import GEMM library from a separate wasm module
+   2. Patch generated artifacts to import GEMM library from a separate wasm module
        ```bash
        bash ../wasm/patch-artifacts-import-gemm-module.sh
        ```
@@ -57,7 +52,7 @@ To build a version that runs on all browsers (including Firefox Nightly) but tra
       ```bash
       mkdir build-wasm
       cd build-wasm
-      emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=off ../
+      emcmake cmake -DCOMPILE_WASM=on ../
       emmake make -j2
       ```
 
diff --git a/build-wasm.sh b/build-wasm.sh
index ff12013d1..b6d70efb6 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -2,34 +2,6 @@
 set -e
 set -x
 
-# Usage
-Usage="Build translator to wasm (with/without wormhole).
-
-Usage: $(basename "$0") [WORMHOLE]
-
-    where:
-    WORMHOLE      An optional string argument
-                  - when specified on command line, builds wasm artifacts with wormhole
-                  - when not specified (the default behaviour), builds wasm artifacts without wormhole."
-
-if [ "$#" -gt 1 ]; then
-  echo "Illegal number of parameters passed"
-  echo "$Usage"
-  exit
-fi
-
-WORMHOLE=false
-
-if [ "$#" -eq 1 ]; then
-  if [ "$1" = "WORMHOLE" ]; then
-    WORMHOLE=true
-  else
-    echo "Illegal parameter passed"
-    echo "$Usage"
-    exit
-  fi
-fi
-
 # Run script from the context of the script-containing directory
 cd "$(dirname $0)"
 
@@ -66,19 +38,10 @@ if [ ! -d ${BUILD_DIRECTORY} ]; then
 fi
 cd ${BUILD_DIRECTORY}
 
-if [ "$WORMHOLE" = true ]; then
-  emcmake cmake -DCOMPILE_WASM=on ../
-else
-  emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=off ../
-fi
+emcmake cmake -DCOMPILE_WASM=on ../
 emmake make -j2
 
-#     2. Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
-if [ "$WORMHOLE" = true ]; then
-  bash ../wasm/patch-artifacts-enable-wormhole.sh
-fi
-
-#     3. Import GEMM library from a separate wasm module
+#     2. Import GEMM library from a separate wasm module
 bash ../wasm/patch-artifacts-import-gemm-module.sh
 
 # The artifacts (.js and .wasm files) will be available in the build directory
diff --git a/wasm/README.md b/wasm/README.md
index e2d9a447c..0f3f77426 100644
--- a/wasm/README.md
+++ b/wasm/README.md
@@ -32,18 +32,8 @@ Alternatively refer to the file `test_page/js/worker.js` that demonstrates how t
 
     Provide the folder containing the wasm artifacts as the first argument of `start_server.sh` script (`../../build-wasm` in this case).
 
-* Open any of the browsers below
-    * Firefox Nightly +87: make sure the following prefs are on (about:config)
-        ```
-        dom.postMessage.sharedArrayBuffer.bypassCOOP_COEP.insecure.enabled = true
-        javascript.options.wasm_simd = true
-        javascript.options.wasm_simd_wormhole = true
-        ```
-
-    * Chrome Canary +90: start with the following argument
-        ```
-        --js-flags="--experimental-wasm-simd"
-        ```
+* Open any browser (tested with latest Chrome/Firefox/Safari)
+
 
 * Browse to the following page:
     ```
diff --git a/wasm/patch-artifacts-enable-wormhole.sh b/wasm/patch-artifacts-enable-wormhole.sh
deleted file mode 100644
index e39988b4e..000000000
--- a/wasm/patch-artifacts-enable-wormhole.sh
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/bin/bash
-usage="Patch wasm artifacts to enable wormhole via APIs that compile and instantiate wasm module.
-
-Usage: $(basename "$0") [WASM_ARTIFACTS_FOLDER]
-
-    where:
-    WASM_ARTIFACTS_FOLDER    Folder containing wasm artifacts
-                             (An optional argument, if unspecified the default is: current folder)"
-
-if [ "$#" -gt 1 ]; then
-    echo "Illegal number of parameters passed"
-    echo "$usage"
-    exit
-fi
-
-# Parse wasm artifacts folder if provided via script argument or set it to default
-WASM_ARTIFACTS_FOLDER=$PWD
-if [ "$#" -eq 1 ]; then
-    if [ ! -e "$1" ]; then
-        echo "Error: Folder \""$1"\" doesn't exist"
-        exit
-    fi
-    WASM_ARTIFACTS_FOLDER="$1"
-fi
-
-WASM_ARTIFACTS="$WASM_ARTIFACTS_FOLDER/bergamot-translator-worker.js"
-if [ ! -e "$WASM_ARTIFACTS" ]; then
-    echo "Error: Artifact \"$WASM_ARTIFACTS\" doesn't exist"
-    exit
-fi
-
-echo "Patching \"$WASM_ARTIFACTS\" to enable wormhole via APIs that compile and instantiate wasm module"
-sed -i.bak 's/WebAssembly.instantiateStreaming[[:space:]]*([[:space:]]*response[[:space:]]*,[[:space:]]*info[[:space:]]*)/WebAssembly.instantiateStreaming(response, info, {simdWormhole:true})/g' $WASM_ARTIFACTS
-sed -i.bak 's/WebAssembly.instantiate[[:space:]]*([[:space:]]*binary[[:space:]]*,[[:space:]]*info[[:space:]]*)/WebAssembly.instantiate(binary, info, {simdWormhole:true})/g' $WASM_ARTIFACTS
-sed -i.bak 's/WebAssembly.Module[[:space:]]*([[:space:]]*bytes[[:space:]]*)/WebAssembly.Module(bytes, {simdWormhole:true})/g' $WASM_ARTIFACTS
-echo "Done"

From 47024ec7a3ed2fe7909c01758ed9ca51625d8703 Mon Sep 17 00:00:00 2001
From: Greg Tatum <gregtatum@users.noreply.github.com>
Date: Wed, 16 Aug 2023 09:35:26 -0500
Subject: [PATCH 411/442] Add more things to the gitignore that are not being
 ignored (#462)

---
 .gitignore | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/.gitignore b/.gitignore
index c796e0656..94b32949c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -17,7 +17,10 @@ _deps
 
 
 wasm/test_page/node_modules
-build-wasm
+/build
+/build-native
+/build-wasm
+/emsdk
 models
 wasm/module/worker/bergamot-translator-worker.*
 wasm/module/browsermt-bergamot-translator-*.tgz

From 62770bb067d2c79bc83c82f2e45063ee73754c39 Mon Sep 17 00:00:00 2001
From: Greg Tatum <gregtatum@users.noreply.github.com>
Date: Wed, 16 Aug 2023 10:14:56 -0500
Subject: [PATCH 412/442] Generate a compile_commands.json by default with
 cmake (#461)

---
 CMakeLists.txt | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 82940de82..d8a2d00cb 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -14,6 +14,10 @@ project(bergamot_translator CXX C)
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
+# Generate a compile_commands.json in the build directory. The compile commands allow
+# code editors to understand the build process and provide static analysis of the code.
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+
 # Note that with CMake MSVC build, the option CMAKE_BUILD_TYPE is automatically derived from the key
 # 'configurationType' in CMakeSettings.json configurations
 if(NOT CMAKE_BUILD_TYPE)

From db3826266d11e611f9a96ab36a2deb84c4938697 Mon Sep 17 00:00:00 2001
From: Greg Tatum <gregtatum@users.noreply.github.com>
Date: Thu, 17 Aug 2023 01:55:49 -0500
Subject: [PATCH 413/442] Report the wasm size on builds (#460)

---
 build-wasm.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/build-wasm.sh b/build-wasm.sh
index b6d70efb6..443907232 100755
--- a/build-wasm.sh
+++ b/build-wasm.sh
@@ -44,5 +44,22 @@ emmake make -j2
 #     2. Import GEMM library from a separate wasm module
 bash ../wasm/patch-artifacts-import-gemm-module.sh
 
+set +x
+echo ""
+echo "Build complete"
+echo ""
+echo "  ./build-wasm/bergamot-translator-worker.js"
+echo "  ./build-wasm/bergamot-translator-worker.wasm"
+
+WASM_SIZE=$(wc -c bergamot-translator-worker.wasm | awk '{print $1}')
+GZIP_SIZE=$(gzip -c bergamot-translator-worker.wasm | wc -c | xargs) # xargs trims the whitespace
+
+# Convert it to human readable.
+WASM_SIZE="$(awk 'BEGIN {printf "%.2f",'$WASM_SIZE'/1048576}')M ($WASM_SIZE bytes)"
+GZIP_SIZE="$(awk 'BEGIN {printf "%.2f",'$GZIP_SIZE'/1048576}')M ($GZIP_SIZE bytes)"
+
+echo "  Uncompressed wasm size: $WASM_SIZE"
+echo "  Compressed wasm size: $GZIP_SIZE"
+
 # The artifacts (.js and .wasm files) will be available in the build directory
 exit 0

From 0b069acce6076bf6d01d6fab132a332ca26ef076 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 11 Sep 2023 08:20:47 +0100
Subject: [PATCH 414/442] Bump 3rd_party/marian-dev from `300a50f` to `780df27`
 (#464)

Bumps [3rd_party/marian-dev](https://github.com/browsermt/marian-dev) from `300a50f` to `780df27`.
- [Commits](https://github.com/browsermt/marian-dev/compare/300a50f4251d978dc197d15bb7b296597b1eb221...780df2708e023ce47c0e1e89f2f4a7f3beab5271)

---
updated-dependencies:
- dependency-name: 3rd_party/marian-dev
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 300a50f42..780df2708 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 300a50f4251d978dc197d15bb7b296597b1eb221
+Subproject commit 780df2708e023ce47c0e1e89f2f4a7f3beab5271

From 321be8ae0486de3af67307c4cb2e005994593597 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Wed, 20 Sep 2023 08:10:18 +0100
Subject: [PATCH 415/442] Bump 3rd_party/marian-dev from `780df27` to `11c6ae7`
 (#466)

Bumps [3rd_party/marian-dev](https://github.com/browsermt/marian-dev) from `780df27` to `11c6ae7`.
- [Commits](https://github.com/browsermt/marian-dev/compare/780df2708e023ce47c0e1e89f2f4a7f3beab5271...11c6ae7c46be21ef96ed10c60f28022fa968939f)

---
updated-dependencies:
- dependency-name: 3rd_party/marian-dev
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 780df2708..11c6ae7c4 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 780df2708e023ce47c0e1e89f2f4a7f3beab5271
+Subproject commit 11c6ae7c46be21ef96ed10c60f28022fa968939f

From 73182d4c58000f74a5bf2e2529f2d2344a584625 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <github@kheafield.com>
Date: Thu, 7 Dec 2023 10:21:45 -0500
Subject: [PATCH 416/442] Pull in marian-dev with fixed CI and clang

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 11c6ae7c4..831a7362e 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 11c6ae7c46be21ef96ed10c60f28022fa968939f
+Subproject commit 831a7362e26a5d43602658d31a2b52571dd16761

From 7774029d0dc239817f009a1dea84e2a195797052 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <github@kheafield.com>
Date: Thu, 7 Dec 2023 11:03:33 -0500
Subject: [PATCH 417/442] clang: marian-dev with newer fbgemm

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 831a7362e..ecda59e61 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 831a7362e26a5d43602658d31a2b52571dd16761
+Subproject commit ecda59e6105fb1d7935892c3bacfbc9562b235f1

From 0367ae07a79d2769b861a13e07fc205969c75ce2 Mon Sep 17 00:00:00 2001
From: Kenneth Heafield <github@kheafield.com>
Date: Thu, 7 Dec 2023 12:10:50 -0500
Subject: [PATCH 418/442] Fix MKL key URL

---
 .github/workflows/native.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml
index 6c5f56913..505381cbc 100644
--- a/.github/workflows/native.yml
+++ b/.github/workflows/native.yml
@@ -58,7 +58,7 @@ jobs:
         sudo apt-get install -y libprotobuf-dev protobuf-compiler libboost-all-dev ccache libunwind-dev libgoogle-perftools-dev
     - name: Install MKL
       run: |-
-        wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB" | sudo apt-key add -
+        wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB" | sudo apt-key add -
         sudo sh -c "echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list"
         sudo apt-get update -o Dir::Etc::sourcelist="/etc/apt/sources.list.d/intel-mkl.list"
         sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088

From 983331bbc98e5b76b11ee265f4ecb22d69ad035f Mon Sep 17 00:00:00 2001
From: XapaJIaMnu <nheart@gmail.com>
Date: Tue, 19 Dec 2023 18:41:18 +0000
Subject: [PATCH 419/442] More pendantic spm

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index ecda59e61..2be8344fc 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit ecda59e6105fb1d7935892c3bacfbc9562b235f1
+Subproject commit 2be8344fcf2776fb43a7376284067164674cbfaf

From 5261614dfd2f4098c32911f4aa7e7759afd13abb Mon Sep 17 00:00:00 2001
From: Kirandevraj <Kirandevraj@users.noreply.github.com>
Date: Sun, 24 Mar 2024 01:51:46 +0530
Subject: [PATCH 420/442] model url update in example script (#470)

---
 examples/run-native.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/examples/run-native.sh b/examples/run-native.sh
index b02968a23..84e1302f0 100644
--- a/examples/run-native.sh
+++ b/examples/run-native.sh
@@ -3,8 +3,8 @@
 # Obtain an example model from the web.
 mkdir -p models
 wget --quiet --continue --directory models/ \
-    http://data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz 
-(cd models && tar -xzf ende.student.tiny11.tar.gz)
+    https://data.statmt.org/bergamot/models/deen/ende.student.tiny11.v2.93821e13b3c511b5.tar.gz
+(cd models && tar -xzf ende.student.tiny11.v2.93821e13b3c511b5.tar.gz)
 
 # Patch the config-files generated from marian for use in bergamot.
 python3 bergamot-translator-tests/tools/patch-marian-for-bergamot.py \

From 34acd8d982d33fd38378093108b1a48ffa542c3c Mon Sep 17 00:00:00 2001
From: Yo'av Moshe <github@yoavmoshe.com>
Date: Sat, 20 Apr 2024 00:17:45 +0200
Subject: [PATCH 421/442] fix downloading of models in the python binding
 (#472)

models come in files named like `csen.student.base.v1.cd5418ba6a412fc7.tar.gz`, but the directory they create when extracted are named like `csen.student.base`. we therefore need to remove not just the extension but everything following and including the 3rd period
---
 bindings/python/repository.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bindings/python/repository.py b/bindings/python/repository.py
index 9667c7242..9ea3ac023 100644
--- a/bindings/python/repository.py
+++ b/bindings/python/repository.py
@@ -180,7 +180,7 @@ def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
     def _archive_name_without_extension(self, url: URL):
         o = urlparse(url)
         fname = os.path.basename(o.path)  # something tar.gz.
-        fname_without_extension = fname.replace(".tar.gz", "")
+        fname_without_extension = ".".join(fname.split(".")[:3])
         return fname_without_extension
 
 

From 9271618ebbdc5d21ac4dc4df9e72beb7ce644774 Mon Sep 17 00:00:00 2001
From: XapaJIaMnu <nheart@gmail.com>
Date: Sun, 12 May 2024 09:51:02 +0100
Subject: [PATCH 422/442] Update submodule

---
 3rd_party/marian-dev | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/3rd_party/marian-dev b/3rd_party/marian-dev
index 2be8344fc..2781d735d 160000
--- a/3rd_party/marian-dev
+++ b/3rd_party/marian-dev
@@ -1 +1 @@
-Subproject commit 2be8344fcf2776fb43a7376284067164674cbfaf
+Subproject commit 2781d735d4a10dca876d61be587afdab2726293c

From bbb844243c028bf88f8fc4def142d4389dda2354 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Thu, 19 Sep 2024 16:46:44 -0500
Subject: [PATCH 423/442] Move inference-engine git submodules to the
 repository root

The fork of Bergamot, now locaed in the `inference-engine`
directory, had its own set of defined submodules.

These need to be moved to the repository root in order to
function correctly within a mono repo.
---
 .gitmodules                     | 21 +++++++++++++++++++++
 inference-engine/.gitmodules    | 12 ------------
 inference-engine/CMakeLists.txt |  5 ++++-
 3 files changed, 25 insertions(+), 13 deletions(-)
 delete mode 100644 inference-engine/.gitmodules

diff --git a/.gitmodules b/.gitmodules
index f1813a444..e6abab367 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,18 +1,39 @@
 [submodule "fast_align"]
 	path = 3rd_party/fast_align
 	url = https://github.com/clab/fast_align
+
 [submodule "extract-lex"]
 	path = 3rd_party/extract-lex
 	url = https://github.com/marian-nmt/extract-lex
+
+[submodule "inference-engine/bergamot-translator-tests"]
+	path = inference-engine/bergamot-translator-tests
+	url = https://github.com/browsermt/bergamot-translator-tests
+
+[submodule "inference-engine/3rd_party/pybind11"]
+	path = inference-engine/3rd_party/pybind11
+	url = https://github.com/pybind/pybind11.git
+
+[submodule "inference-engine/3rd_party/marian-dev"]
+	path = inference-engine/3rd_party/marian-dev
+	url = https://github.com/browsermt/marian-dev
+
+[submodule "inference-engine/3rd_party/ssplit-cpp"]
+	path = inference-engine/3rd_party/ssplit-cpp
+	url = https://github.com/browsermt/ssplit-cpp
+
 [submodule "3rd_party/kenlm"]
 	path = 3rd_party/kenlm
 	url = https://github.com/kpu/kenlm
+
 [submodule "3rd_party/browsermt-marian-dev"]
 	path = 3rd_party/browsermt-marian-dev
 	url = https://github.com/browsermt/marian-dev
+
 [submodule "3rd_party/marian-dev"]
 	path = 3rd_party/marian-dev
 	url = https://github.com/marian-nmt/marian-dev
+
 [submodule "3rd_party/preprocess"]
 	path = 3rd_party/preprocess
 	url = https://github.com/kpu/preprocess.git
diff --git a/inference-engine/.gitmodules b/inference-engine/.gitmodules
deleted file mode 100644
index cfedde289..000000000
--- a/inference-engine/.gitmodules
+++ /dev/null
@@ -1,12 +0,0 @@
-[submodule "3rd_party/marian-dev"]
-	path = 3rd_party/marian-dev
-	url = https://github.com/browsermt/marian-dev
-[submodule "3rd_party/ssplit-cpp"]
-	path = 3rd_party/ssplit-cpp
-	url = https://github.com/browsermt/ssplit-cpp
-[submodule "bergamot-translator-tests"]
-	path = bergamot-translator-tests
-	url = https://github.com/browsermt/bergamot-translator-tests
-[submodule "3rd_party/pybind11"]
-	path = 3rd_party/pybind11
-	url = https://github.com/pybind/pybind11.git
diff --git a/inference-engine/CMakeLists.txt b/inference-engine/CMakeLists.txt
index d8a2d00cb..da01c6048 100644
--- a/inference-engine/CMakeLists.txt
+++ b/inference-engine/CMakeLists.txt
@@ -11,6 +11,9 @@ endif()
 
 project(bergamot_translator CXX C)
 
+# Retrieve the parent-directory path of PROJECT_SOURCE_DIR and assign that to REPOSITORY_ROOT_DIR.
+cmake_path(GET PROJECT_SOURCE_DIR PARENT_PATH REPOSITORY_ROOT_DIR)
+
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
@@ -96,7 +99,7 @@ endif()
 # Documentation: https://cliutils.gitlab.io/modern-cmake/chapters/projects/submodule.html
 # Ensures the submodules are set correctly during a build.
 find_package(Git QUIET)
-if(GIT_FOUND AND EXISTS "${PROJECT_SOURCE_DIR}/.git")
+if(GIT_FOUND AND EXISTS "${REPOSITORY_ROOT_DIR}/.git")
 # Update submodules as needed
     option(GIT_SUBMODULE "Check submodules during build" ON)
     if(GIT_SUBMODULE)

From cad39633ef7f18eec8999b9bac1c80afc5089836 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Thu, 19 Sep 2024 16:55:12 -0500
Subject: [PATCH 424/442] Rename inference-engine/3rd_party/marian-nmt

---
 .gitmodules                                   | 26 ++++++++++++++++---
 inference-engine/3rd_party/CMakeLists.txt     | 14 +++++-----
 .../{marian-dev => browsermt-marian-dev}      |  0
 .../patches/01-marian-fstream-for-macos.patch |  6 ++---
 inference-engine/src/tests/CMakeLists.txt     |  2 +-
 inference-engine/src/translator/logging.h     |  2 +-
 inference-engine/src/translator/parser.h      |  2 +-
 7 files changed, 35 insertions(+), 17 deletions(-)
 rename inference-engine/3rd_party/{marian-dev => browsermt-marian-dev} (100%)

diff --git a/.gitmodules b/.gitmodules
index e6abab367..5d1bbf716 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -10,14 +10,32 @@
 	path = inference-engine/bergamot-translator-tests
 	url = https://github.com/browsermt/bergamot-translator-tests
 
+# This is the same dependency and repository as `3rd_party/browsermt-marian-dev` below.
+#
+# When forking `inference-engine` into to this project, I made an earnest attempt to utilize the preexisting
+# `3rd_party/browsermt-marian-dev` submodule within `inference-engine`. Unfortunately, I ran into several roadblocks:
+#
+#   1) I cannot directly add `3rd_party/browsermt-marian-dev` as a cmake subdirectory because cmake is aware that
+#      this path is not a subdirectory of the `inference-engine` project root.
+#
+#   2) Symbolic links do not appear to work for git submodule direcotires the way that they do for regular directories.
+#      Even if the symbolic link had linked correctly, it may have still failed due to the considerations of 1).
+#
+#   3) I tried using cmake to copy the files from `3rd_party/browsermt-marian-dev` into `inference-engine/3rd_party/browsermt-marian-dev`
+#      at build time, which would ensure that there is no duplicate reference to the URL in this file, however the upstream dependency itself
+#      has hard-coded cmake expectations that the `.git` directory is only one level up, which appears to work correctly for the way git submodules
+#      are configured, but does not work if the files are copied over to a regular directory deeper in the repository.
+#
+# It may be possible to remove `3rd_party/browsermt-marian-dev` to instead use `inference-engine/3rd-party/browsermt-marian-dev` everywhere
+# within this repository, but I will leave that for a future commit if there is a need to do so.
+[submodule "inference-engine/3rd_party/browsermt-marian-dev"]
+	path = inference-engine/3rd_party/browsermt-marian-dev
+	url = https://github.com/browsermt/marian-dev
+
 [submodule "inference-engine/3rd_party/pybind11"]
 	path = inference-engine/3rd_party/pybind11
 	url = https://github.com/pybind/pybind11.git
 
-[submodule "inference-engine/3rd_party/marian-dev"]
-	path = inference-engine/3rd_party/marian-dev
-	url = https://github.com/browsermt/marian-dev
-
 [submodule "inference-engine/3rd_party/ssplit-cpp"]
 	path = inference-engine/3rd_party/ssplit-cpp
 	url = https://github.com/browsermt/ssplit-cpp
diff --git a/inference-engine/3rd_party/CMakeLists.txt b/inference-engine/3rd_party/CMakeLists.txt
index eac898eb9..0185d7673 100644
--- a/inference-engine/3rd_party/CMakeLists.txt
+++ b/inference-engine/3rd_party/CMakeLists.txt
@@ -1,6 +1,6 @@
-# marian-dev is tested elsewhere in both paths, turning off here.
+# browsermt-marian-dev is tested elsewhere in both paths, turning off here.
 set(COMPILE_TESTS OFF)
-add_subdirectory(marian-dev EXCLUDE_FROM_ALL)
+add_subdirectory(browsermt-marian-dev EXCLUDE_FROM_ALL)
 
 if(COMPILE_WASM)
   # This is a bad way of adding compilation flags. Will be improved soon.
@@ -13,21 +13,21 @@ add_subdirectory(ssplit-cpp EXCLUDE_FROM_ALL)
 # Add include directories for 3rd party targets to be able to use it anywhere in the
 # project without explicitly specifying their include directories. Once they
 # fixe this problem, it can be removed.
-get_property(INCDIRS DIRECTORY marian-dev/src PROPERTY INCLUDE_DIRECTORIES)
+get_property(INCDIRS DIRECTORY browsermt-marian-dev/src PROPERTY INCLUDE_DIRECTORIES)
 target_include_directories(marian PUBLIC ${INCDIRS})
 
 get_property(INCLUDE_DIRECTORIES DIRECTORY ssplit-cpp/src PROPERTY INCLUDE_DIRECTORIES)
 target_include_directories(ssplit PUBLIC ${INCLUDE_DIRECTORIES})
 
-get_property(COMPILE_DEFINITIONS DIRECTORY marian-dev PROPERTY COMPILE_DEFINITIONS) 
+get_property(COMPILE_DEFINITIONS DIRECTORY browsermt-marian-dev PROPERTY COMPILE_DEFINITIONS) 
 target_compile_definitions(marian PUBLIC ${COMPILE_DEFINITIONS})
 
-get_property(COMPILE_OPTIONS DIRECTORY marian-dev PROPERTY COMPILE_OPTIONS) 
+get_property(COMPILE_OPTIONS DIRECTORY browsermt-marian-dev PROPERTY COMPILE_OPTIONS) 
 target_compile_options(marian PUBLIC ${COMPILE_OPTIONS})
 
 # Compilation flags 
-get_directory_property(CMAKE_C_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_C_FLAGS) 
-get_directory_property(CMAKE_CXX_FLAGS DIRECTORY marian-dev DEFINITION CMAKE_CXX_FLAGS) 
+get_directory_property(CMAKE_C_FLAGS DIRECTORY browsermt-marian-dev DEFINITION CMAKE_C_FLAGS) 
+get_directory_property(CMAKE_CXX_FLAGS DIRECTORY browsermt-marian-dev DEFINITION CMAKE_CXX_FLAGS) 
 set(CMAKE_C_FLAGS ${CMAKE_C_FLAGS} PARENT_SCOPE)    
 set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} PARENT_SCOPE)    
 
diff --git a/inference-engine/3rd_party/marian-dev b/inference-engine/3rd_party/browsermt-marian-dev
similarity index 100%
rename from inference-engine/3rd_party/marian-dev
rename to inference-engine/3rd_party/browsermt-marian-dev
diff --git a/inference-engine/patches/01-marian-fstream-for-macos.patch b/inference-engine/patches/01-marian-fstream-for-macos.patch
index 5219227d9..6b521ba7e 100644
--- a/inference-engine/patches/01-marian-fstream-for-macos.patch
+++ b/inference-engine/patches/01-marian-fstream-for-macos.patch
@@ -1,7 +1,7 @@
-diff --git a/3rd_party/marian-dev/src/3rd_party/zstr/strict_fstream.hpp b/3rd_party/marian-dev/src/3rd_party/zstr/strict_fstream.hpp
+diff --git a/3rd_party/browsermt-marian-dev/src/3rd_party/zstr/strict_fstream.hpp b/3rd_party/browsermt-marian-dev/src/3rd_party/zstr/strict_fstream.hpp
 index 7b1173931df977e69021f3995fa064a492f89d38..948e91eaf99b6b29ce41cf793fba6717f3b5f5b5 100644
---- a/3rd_party/marian-dev/src/3rd_party/zstr/strict_fstream.hpp
-+++ b/3rd_party/marian-dev/src/3rd_party/zstr/strict_fstream.hpp
+--- a/3rd_party/browsermt-marian-dev/src/3rd_party/zstr/strict_fstream.hpp
++++ b/3rd_party/browsermt-marian-dev/src/3rd_party/zstr/strict_fstream.hpp
 @@ -27,7 +27,7 @@ static std::string strerror()
      {
          buff = "Unknown error";
diff --git a/inference-engine/src/tests/CMakeLists.txt b/inference-engine/src/tests/CMakeLists.txt
index 86fe00236..cd0e4c777 100644
--- a/inference-engine/src/tests/CMakeLists.txt
+++ b/inference-engine/src/tests/CMakeLists.txt
@@ -1,7 +1,7 @@
 # Unit tests
 
 # Include Catch explicitly from marian.
-set(CATCH_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/3rd_party/marian-dev/3rd-party)
+set(CATCH_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/3rd_party/browsermt-marian-dev/3rd-party)
 add_library(Catch INTERFACE)
 target_include_directories(Catch INTERFACE ${CATCH_INCLUDE_DIR})
 
diff --git a/inference-engine/src/translator/logging.h b/inference-engine/src/translator/logging.h
index 2256d7889..704492283 100644
--- a/inference-engine/src/translator/logging.h
+++ b/inference-engine/src/translator/logging.h
@@ -1,4 +1,4 @@
-#include "3rd_party/marian-dev/src/3rd_party/spdlog/spdlog.h"
+#include "3rd_party/browsermt-marian-dev/src/3rd_party/spdlog/spdlog.h"
 #include "common/logging.h"
 
 namespace marian {
diff --git a/inference-engine/src/translator/parser.h b/inference-engine/src/translator/parser.h
index 793582dd0..8f98e2c73 100644
--- a/inference-engine/src/translator/parser.h
+++ b/inference-engine/src/translator/parser.h
@@ -4,7 +4,7 @@
 #include <fstream>
 #include <sstream>
 
-#include "3rd_party/marian-dev/src/3rd_party/CLI/CLI.hpp"
+#include "3rd_party/browsermt-marian-dev/src/3rd_party/CLI/CLI.hpp"
 #include "3rd_party/yaml-cpp/yaml.h"
 #include "common/build_info.h"
 #include "common/config_parser.h"

From 3da08c956a0f160870e33015d6526f0f7858c8ac Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Thu, 26 Sep 2024 13:04:29 -0500
Subject: [PATCH 425/442] Remove bergamot-translator-tests dependency

---
 .gitmodules                                | 22 ----------------------
 inference-engine/bergamot-translator-tests |  1 -
 2 files changed, 23 deletions(-)
 delete mode 160000 inference-engine/bergamot-translator-tests

diff --git a/.gitmodules b/.gitmodules
index 5d1bbf716..ce6f3230b 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -6,28 +6,6 @@
 	path = 3rd_party/extract-lex
 	url = https://github.com/marian-nmt/extract-lex
 
-[submodule "inference-engine/bergamot-translator-tests"]
-	path = inference-engine/bergamot-translator-tests
-	url = https://github.com/browsermt/bergamot-translator-tests
-
-# This is the same dependency and repository as `3rd_party/browsermt-marian-dev` below.
-#
-# When forking `inference-engine` into to this project, I made an earnest attempt to utilize the preexisting
-# `3rd_party/browsermt-marian-dev` submodule within `inference-engine`. Unfortunately, I ran into several roadblocks:
-#
-#   1) I cannot directly add `3rd_party/browsermt-marian-dev` as a cmake subdirectory because cmake is aware that
-#      this path is not a subdirectory of the `inference-engine` project root.
-#
-#   2) Symbolic links do not appear to work for git submodule direcotires the way that they do for regular directories.
-#      Even if the symbolic link had linked correctly, it may have still failed due to the considerations of 1).
-#
-#   3) I tried using cmake to copy the files from `3rd_party/browsermt-marian-dev` into `inference-engine/3rd_party/browsermt-marian-dev`
-#      at build time, which would ensure that there is no duplicate reference to the URL in this file, however the upstream dependency itself
-#      has hard-coded cmake expectations that the `.git` directory is only one level up, which appears to work correctly for the way git submodules
-#      are configured, but does not work if the files are copied over to a regular directory deeper in the repository.
-#
-# It may be possible to remove `3rd_party/browsermt-marian-dev` to instead use `inference-engine/3rd-party/browsermt-marian-dev` everywhere
-# within this repository, but I will leave that for a future commit if there is a need to do so.
 [submodule "inference-engine/3rd_party/browsermt-marian-dev"]
 	path = inference-engine/3rd_party/browsermt-marian-dev
 	url = https://github.com/browsermt/marian-dev
diff --git a/inference-engine/bergamot-translator-tests b/inference-engine/bergamot-translator-tests
deleted file mode 160000
index a04432d79..000000000
--- a/inference-engine/bergamot-translator-tests
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit a04432d7921bfa1dd62bc2e5cdca46b226f256de

From 37d0113997cc09b2f0035aa4cb8258ba37dd0c2f Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Fri, 20 Sep 2024 14:38:15 -0500
Subject: [PATCH 426/442] Remove .circleci and .github files

---
 inference-engine/.circleci/config.yml         |  81 ---
 inference-engine/.github/dependabot.yml       |   9 -
 inference-engine/.github/workflows/arm.yml    | 139 ------
 inference-engine/.github/workflows/build.yml  | 466 ------------------
 .../.github/workflows/coding-styles.yml       |  42 --
 inference-engine/.github/workflows/native.yml | 243 ---------
 .../.github/workflows/windows.yml             | 128 -----
 7 files changed, 1108 deletions(-)
 delete mode 100644 inference-engine/.circleci/config.yml
 delete mode 100644 inference-engine/.github/dependabot.yml
 delete mode 100644 inference-engine/.github/workflows/arm.yml
 delete mode 100644 inference-engine/.github/workflows/build.yml
 delete mode 100644 inference-engine/.github/workflows/coding-styles.yml
 delete mode 100644 inference-engine/.github/workflows/native.yml
 delete mode 100644 inference-engine/.github/workflows/windows.yml

diff --git a/inference-engine/.circleci/config.yml b/inference-engine/.circleci/config.yml
deleted file mode 100644
index 52d58fc09..000000000
--- a/inference-engine/.circleci/config.yml
+++ /dev/null
@@ -1,81 +0,0 @@
-version: 2.1
-jobs:
-  build:
-    docker:
-      - image: 'emscripten/emsdk:3.1.8'
-    resource_class: medium
-
-    working_directory: ~/checkout
-
-    steps:
-      - checkout
-
-      - run:
-          name: Build WASM
-          command: |
-            bash build-wasm.sh
-
-      - run:
-          name: Check artifacts
-          working_directory: build-wasm
-          command: |
-            ARTIFACT_BASE="bergamot-translator-worker"
-            ARTIFACT_FINAL=$ARTIFACT_BASE
-
-            if [[ -f "$ARTIFACT_BASE.js" && -f "$ARTIFACT_BASE.wasm" ]]; then
-              echo "Artifacts Successfully Generated"
-              mkdir ../artifacts
-              cp $ARTIFACT_BASE.wasm ../artifacts/$ARTIFACT_FINAL.wasm
-              cp $ARTIFACT_BASE.js ../artifacts/$ARTIFACT_FINAL.js
-              cd ../artifacts
-              shasum -a 256 $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize
-              ls -lsa $ARTIFACT_FINAL.wasm $ARTIFACT_FINAL.js >> sha256-filesize
-            else
-              echo "Failure: Artifacts Not Present"
-              exit 1
-            fi
-
-      - persist_to_workspace:
-          root: .
-          paths:
-            - artifacts/*
-
-      - store_artifacts:
-          path: "artifacts"
-          destination: "wasm"
-
-  publish_to_github:
-    docker:
-      - image: cibuilds/github:0.10
-    steps:
-      - attach_workspace:
-          # Must be absolute path or relative path from working_directory
-          at: ./
-      - when:
-          condition:
-            equal: [ 'https://github.com/mozilla/bergamot-translator', << pipeline.project.git_url >> ]
-          steps:
-            - run:
-                name: "Publish Release on GitHub"
-                command: |
-                  export TAG_VERSION=$(cat ./artifacts/BERGAMOT_VERSION)
-                  rm ./artifacts/BERGAMOT_VERSION
-                  ghr -t ${GHTOKEN} -u ${CIRCLE_PROJECT_USERNAME} -r ${CIRCLE_PROJECT_REPONAME} -c ${CIRCLE_SHA1} -delete ${TAG_VERSION} ./artifacts/
-
-workflows:
-  build:
-      jobs:
-          - build:
-              filters:
-                tags:
-                  only: /^v.*/
-          - publish_to_github:
-              filters:
-                tags:
-                  only: /^v.*/
-                branches:
-                  ignore: /.*/
-              requires:
-                - build
-
-
diff --git a/inference-engine/.github/dependabot.yml b/inference-engine/.github/dependabot.yml
deleted file mode 100644
index bbb39076f..000000000
--- a/inference-engine/.github/dependabot.yml
+++ /dev/null
@@ -1,9 +0,0 @@
-version: 2
-
-updates:
-  # Maintain dependencies for Git Submodules
-  - package-ecosystem: "gitsubmodule"
-    directory: "/"
-    schedule:
-      interval: "daily"
-
diff --git a/inference-engine/.github/workflows/arm.yml b/inference-engine/.github/workflows/arm.yml
deleted file mode 100644
index 2ee14548d..000000000
--- a/inference-engine/.github/workflows/arm.yml
+++ /dev/null
@@ -1,139 +0,0 @@
-name: ARM
-'on':
-  push:
-    branches:
-    - main
-    - ci-sandbox
-  pull_request:
-    branches:
-    - '**'
-env:
-  ccache_basedir: ${{ github.workspace }}
-  ccache_dir: "${{ github.workspace }}/.ccache"
-  ccache_compilercheck: content
-  ccache_compress: 'true'
-  ccache_compresslevel: 9
-  ccache_maxsize: 200M
-  ccache_cmake: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
-  ndk: "${{ github.workspace }}/android-ndk-r23b"
-  abi: "arm64-v8a"
-  minsdk_version : 28
-  android_platform: 28
-
-jobs:
-  ubuntu:
-    name: "arm-v8a cross-compile via Android NDK"
-    runs-on: ubuntu-latest
-
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-
-    - name: Install prerequisites
-      run: |
-          wget -c --quiet https://dl.google.com/android/repository/android-ndk-r23b-linux.zip
-          unzip -qq android-ndk-r23b-linux.zip
-          sudo apt-get -y install ccache cmake
-
-    - name: Generate ccache_vars for ccache based on machine
-      shell: bash
-      id: ccache_vars
-      run: |-
-        echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
-        echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
-
-    - name: Cache-op for build-cache through ccache
-      uses: actions/cache@v2
-      with:
-        path: ${{ env.ccache_dir }}
-        key: ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-        restore-keys: |-
-          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
-          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}
-          ccache-${{ matrix.identifier }}
-
-    - name: ccache environment setup
-      run: |-
-        echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
-        echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
-        echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
-        echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
-        echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
-        echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
-
-    - name: ccache prolog
-      run: |-
-        ccache -s # Print current cache stats
-        ccache -z # Zero cache entry
-
-    - name: Generate buildfiles for bergamot-translator on android via cmake
-      run: |-
-        mkdir -p build 
-        cd build
-        NDK=${{ env.ndk }}
-        ABI=${{ env.abi }}
-        MINSDK_VERSION=${{ env.minsdk_version }}
-        ANDROID_PLATFORM=android-${{ env.android_platform }}
-        OTHER_ANDROID_ARGS=(
-            -DANDROID_ARM_NEON=TRUE
-        )
-        OTHER_MARIAN_ARGS=(
-            -DCOMPILE_CUDA=off
-            -DCOMPILE_CPU=on
-            -DCMAKE_HAVE_THREADS_LIBRARY=1
-            -DCMAKE_USE_WIN32_THREADS_INIT=0
-            -DCMAKE_USE_PTHREADS_INIT=1
-            -DTHREADS_PREFER_PTHREAD_FLAG=ON
-            -DBUILD_ARCH=armv8-a
-            # -DCOMPILE_WITHOUT_EXCEPTIONS=on # Apparently this can reduce the binary size, let's see.
-            -DSSPLIT_USE_INTERNAL_PCRE2=ON
-        )
-        # Additionally list variables finally configured.
-        cmake -L \
-            -DCMAKE_BUILD_TYPE=Release \
-            -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \
-            -DANDROID_TOOLCHAIN=clang \
-            -DANDROID_ABI=$ABI \
-            -DANDROID_PLATFORM=$ANDROID_PLATFORM \
-            -DANDROID_NATIVE_API_LEVEL=$MINSDKVERSION \
-            -DANDROID_TOOLCHAIN_NAME=arm-linux-androideabi-4.8 \
-            -DANDROID_STL=c++_static \
-            -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache \
-            "${OTHER_ANDROID_ARGS[@]}" "${OTHER_MARIAN_ARGS[@]}" \
-            ..
-
-
-    - name : Build bergamot-translator for android
-      working-directory: build
-      run: |-
-          make -j2 
-
-    - name: ccache epilog
-      run: 'ccache -s # Print current cache stats'
-
-    - uses: actions/upload-artifact@v2
-      with:
-        path: ${{github.workspace}}/build/app/bergamot
-
-
-  # Disable release for now.
-  # release:
-  #   name: Release Latest Build
-  #   runs-on: ubuntu-latest
-  #   needs: [ubuntu]
-  #   if: github.ref == 'refs/heads/master'
-  #   steps:
-  #    - name: Download artifacts
-  #      uses: actions/download-artifact@v2
-  #     
-  #    - name: Update GitHub prerelease
-  #      uses: marvinpinto/action-automatic-releases@latest
-  #      with:
-  #        repo_token: ${{ secrets.GITHUB_TOKEN }}
-  #        automatic_release_tag: latest
-  #        prerelease: true
-  #        title: "Latest Build"
-  #        files: |
-  #          artifact/marian-decoder
diff --git a/inference-engine/.github/workflows/build.yml b/inference-engine/.github/workflows/build.yml
deleted file mode 100644
index 830924c2c..000000000
--- a/inference-engine/.github/workflows/build.yml
+++ /dev/null
@@ -1,466 +0,0 @@
-name: "Build"
-'on':
-  push:
-    branches:
-      - main
-      - ci-sandbox
-    tags:
-      - "v*.*.*"
-  pull_request:
-    branches:
-      - '**'
-env:
-  qt_version: "6.2.1" # only used by build-macos
-  emsdk_version: 3.1.8 # For use in emscripten build
-  ccache_basedir: ${{ github.workspace }}
-  ccache_dir: "${{ github.workspace }}/.ccache"
-  ccache_compilercheck: content
-  ccache_compress: 'true'
-  ccache_compresslevel: 9
-  ccache_maxsize: 200M
-  ccache_cmake: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
-
-jobs:
-    build-wheels:
-      strategy:
-        matrix:
-          os: [ubuntu-latest, macos-latest]
-        fail-fast: false
-
-      name: "cibuildwheel / ${{ matrix.os }}"
-      runs-on: ${{ matrix.os }}
-
-      steps:
-        - uses: actions/checkout@v2
-          with:
-            submodules: recursive
-
-        - name: Generate ccache_vars for ccache based on machine
-          shell: bash
-          id: ccache_vars
-          run: |-
-            echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
-            echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
-
-        - name: Cache-op for build-cache through ccache
-          uses: actions/cache@v2
-          with:
-            path: ${{ env.ccache_dir }}
-            key: ccache-cibuildwheel-${{ matrix.os }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-            restore-keys: |-
-              ccache-cibuildwheel-${{ matrix.os }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
-              ccache-cibuildwheel-${{ matrix.os }}-${{ steps.ccache_vars.outputs.hash }}
-              ccache-cibuildwheel-${{ matrix.os }}
-
-        - name: ccache environment setup
-          run: |-
-            mkdir -p ${{ env.ccache_dir }}
-
-        - name: Inject local version identifier for non tag builds
-          if: ${{ !startsWith(github.ref, 'refs/tags/v') }}
-          run: |-
-            echo "PYTHON_LOCAL_VERSION_IDENTIFIER=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
-
-        - name: Apply MacOS patch
-          if: ${{ startsWith(runner.os, 'mac') }}
-          run: |
-            patch -p1 < patches/01-marian-fstream-for-macos.patch
-
-        - name: Build wheels
-          uses: pypa/cibuildwheel@v2.6.1
-          # to supply options, put them in 'env', like:
-          env:
-            CIBW_ENVIRONMENT_LINUX:
-              BUILD_ARCH=core-avx-i
-              USE_CCACHE=1
-              CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}
-              CCACHE_COMPRESS=${{ env.ccache_compress }}
-              CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}
-              CCACHE_MAXSIZE=${{ env.ccache_maxsize }}
-              PYTHON_LOCAL_VERSION_IDENTIFIER=${{ env.PYTHON_LOCAL_VERSION_IDENTIFIER }}
-              CCACHE_DIR=/host/${{ env.ccache_dir }}
-              CCACHE_BASEDIR=/host/${{ env.ccache_basedir }}
-
-            CIBW_ENVIRONMENT_MACOS:
-              BUILD_ARCH=core-avx-i
-              USE_CCACHE=1
-              CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}
-              CCACHE_COMPRESS=${{ env.ccache_compress }}
-              CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}
-              CCACHE_MAXSIZE=${{ env.ccache_maxsize }}
-              PYTHON_LOCAL_VERSION_IDENTIFIER=${{ env.PYTHON_LOCAL_VERSION_IDENTIFIER }}
-              CCACHE_DIR=${{ env.ccache_dir }}
-              CCACHE_BASEDIR=${{ env.ccache_basedir }}
-              MACOSX_DEPLOYMENT_TARGET=10.9
-
-            CIBW_BEFORE_BUILD_LINUX: |
-              yum install -y ccache
-
-              # Install Intel MKL.
-              yum-config-manager -y --add-repo https://yum.repos.intel.com/mkl/setup/intel-mkl.repo
-              yum install -y intel-mkl
-
-              chmod -R a+rwx /host/${{ env.ccache_dir }}
-
-              ccache -s # Print current cache stats
-              ccache -z # Zero cache entry
-
-            CIBW_BEFORE_BUILD_MACOS: |
-              brew install openblas protobuf ccache boost pybind11
-              chmod -R a+rwx ${{ env.ccache_dir }}
-              ccache -s # Print current cache stats
-              ccache -z # Zero cache entry
-
-            CIBW_BUILD: "cp{36,37,38,39,310}-*manylinux_x86_64 cp{36,37,38,39,310}-macosx_x86_64"
-
-            CIBW_BEFORE_TEST: |
-              ccache -s # Print current ccache stats
-
-            CIBW_TEST_COMMAND: |
-              # The wheels are installed automatically and available.
-
-              # Fetch models from translateLocally repository.
-              python3 -m bergamot download -m en-de-tiny
-              python3 -m bergamot download -m de-en-tiny
-              python3 -m bergamot ls
-
-              # Fetch models from opus repository.
-              python3 -m bergamot download -m eng-fin-tiny -r opus
-              python3 -m bergamot ls -r opus
-
-              # Run the sample python script shipped with module
-              python3 -m bergamot translate --model en-de-tiny <<< "Hello World"
-              python3 -m bergamot translate --model en-de-tiny de-en-tiny <<< "Hello World"
-              python3 -m bergamot translate --model eng-fin-tiny --repository opus <<< "Hello World"
-
-
-        - uses: actions/upload-artifact@v2
-          with:
-            name: wheels
-            path: ./wheelhouse/*.whl
-
-    upload-wheels:
-      name: "Upload wheels to PyPI"
-      runs-on: ubuntu-latest
-      if: ${{ startsWith(github.ref, 'refs/tags/v') }}
-      needs: [build-wheels]
-      steps:
-      - name: Download artifacts
-        uses: actions/download-artifact@v2
-        with:
-          name: wheels
-
-      - name: Publish wheels to PyPI
-        env:
-          TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
-          TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
-        run: |
-          python3 -m pip install twine
-          twine upload *.whl
-
-
-    build-wasm:
-      name: "emscripten"
-      runs-on: ubuntu-latest
-      steps:
-
-        - name: Checkout
-          uses: actions/checkout@v2
-          with:
-            submodules: recursive
-
-        - name: Set ccache environment for emcc
-          run: |
-            # We are hardcoding this to mtime instead of env pickup. Rest use content.
-            echo "CCACHE_COMPILER_CHECK=mtime" >> $GITHUB_ENV
-
-            echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
-            echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
-            echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
-            echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
-            echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
-            # https://emscripten.org/docs/compiling/Building-Projects.html#using-a-compiler-wrapper
-            echo "EM_COMPILER_WRAPPER=ccache" >> $GITHUB_ENV
-
-        # This need to be run before setup, so ccache build caching doesn't complain.
-        - name: Obtain emsdk sources
-          run: |
-              git clone --depth 1 https://github.com/emscripten-core/emsdk.git
-
-        - name: Cache-op for build-cache through ccache
-          uses: actions/cache@v2
-          with:
-            path: |
-                ${{ env.ccache_dir }}
-                ${{ github.workspace }}/emsdk/ccache/git-emscripten_64bit/
-            key: ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-            restore-keys: |-
-              ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
-              ccache-${{ github.job }}-${{ env.emsdk_version }}-${{ steps.ccache_vars.outputs.hash }}
-              ccache-${{ github.job }}-${{ env.emsdk_version }}
-
-        - name: Setup Emscripten toolchain
-          run: |
-              (cd emsdk && ./emsdk install ${{ env.emsdk_version }} ccache-git-emscripten-64bit)
-              (cd emsdk && ./emsdk activate ${{ env.emsdk_version }} ccache-git-emscripten-64bit)
-              # mtime of this file is checked by ccache, we set it to avoid cache misses.
-              touch -m -d '1 Jan 2021 12:00' emsdk/.emscripten
-
-              # These needs to be done in the activated shell.
-              eval $(./emsdk/emsdk construct_env \
-                  | sed 's/export PATH=\(.*\);/echo \1 >> $GITHUB_PATH;/' \
-                  | sed 's/export \(.*\);/echo \1 >> $GITHUB_ENV;/' );
-
-              # This looks more permanent than version pinned, so keeping temporarily to avoid failures.
-              echo "${{ github.workspace }}/emsdk/ccache/git-emscripten_64bit/bin" >> $GITHUB_PATH
-
-        - name: Generate ccache_vars for ccache based on machine
-          shell: bash
-          id: ccache_vars
-          run: |-
-            echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
-            echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
-
-        - name: Verify Emscripten setup
-          run: |
-              emcc --version
-              emcmake cmake --version
-              emmake make --version
-
-        - name: ccache prolog
-          run: |-
-            ccache -s # Print current cache stats
-            ccache -z # Zero cache entry
-
-        - name: "Configure builds"
-          run: |
-            mkdir -p build-wasm
-            cd build-wasm
-            emcmake cmake -DCOMPILE_WASM=on ..
-
-
-        - name: "Compile"
-          working-directory: build-wasm
-          run: |
-            emmake make -j2
-
-        - name: ccache epilog
-          run: |
-            ccache -s # Print current cache stats
-
-        - name: Import GEMM library from a separate wasm module
-          working-directory: build-wasm
-          run: bash ../wasm/patch-artifacts-import-gemm-module.sh
-
-        # Setup nodejs-18, as nodejs-14 provided by emsdk fails when running
-        # and newer version of node allows us to use fetch().
-        - name: Setup nodejs
-          uses: actions/setup-node@v3
-          with:
-            node-version: 18
-
-        - name: Test run
-          working-directory: wasm
-          run: |
-            cp ../build-wasm/bergamot-translator-worker.{js,wasm} ./
-            npm install jsdom
-
-            # --unhandled-rejections make the script exit with a non-zero code (at least on node-14).
-            # So leaving this here.
-            node --unhandled-rejections=strict node-test.js
-
-        # Upload both together.
-        - name: Upload wasm artifact
-          uses: actions/upload-artifact@v2
-          with:
-            name: wasm-artefacts
-            if-no-files-found: error
-            path: |
-                ${{github.workspace}}/build-wasm/bergamot-translator-worker.js
-                ${{github.workspace}}/build-wasm/bergamot-translator-worker.wasm
-                ${{github.workspace}}/build-wasm/bergamot-translator-worker.js.bak
-
-
-    upload-wasm:
-      name: "Upload node package to NPM"
-      runs-on: ubuntu-latest
-      if: ${{ startsWith(github.ref, 'refs/tags/v') }}
-      needs: [build-wasm]
-      steps:
-      - name: Download artifacts
-        uses: actions/download-artifact@v2
-        with:
-          name: wasm-artefacts
-          path: wasm/module/worker
-
-      - uses: actions/setup-node@v3
-        with:
-          node-version: '18.x'
-          registry-url: 'https://registry.npmjs.org'
-      - run: npm ci
-      - run: npm publish
-        env:
-          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
-
-
-
-  # Try to upload a release using https://github.com/marvinpinto/actions/issues/177#issuecomment-917605585 as a model
-    release-latest:
-      name: Release Latest Build
-      runs-on: ubuntu-latest
-      needs: [build-wheels, build-wasm]
-      if: github.ref == 'refs/heads/main'
-      steps:
-       - name: Download artifacts
-         uses: actions/download-artifact@v2
-
-       # Leave the below be, it will be useful.
-       - name: List downloaded assets
-         run: |
-           find ./
-
-       - name: Update GitHub prerelease
-         uses: marvinpinto/action-automatic-releases@latest
-         with:
-           repo_token: ${{ secrets.GITHUB_TOKEN }}
-           automatic_release_tag: latest
-           prerelease: true
-           title: "Latest Build"
-           files: |
-                wheels/*.whl
-                wasm-artefacts/bergamot-translator-worker.js
-                wasm-artefacts/bergamot-translator-worker.wasm
-
-    release-version:
-      name: Release version
-      runs-on: ubuntu-latest
-      needs: [build-wheels, build-wasm]
-      permissions:
-        contents: "write"
-        packages: "write"
-        pull-requests: "read"
-      if: startsWith(github.ref, 'refs/tags/v')
-      steps:
-       - name: Download artifacts
-         uses: actions/download-artifact@v2
-
-       # Leave the below be, it will be useful.
-       - name: List downloaded assets
-         run: |
-           find ./
-
-       - name: Update GitHub release
-         uses: marvinpinto/action-automatic-releases@latest
-         with:
-           repo_token: ${{ secrets.GITHUB_TOKEN }}
-           automatic_release_tag: ${{ github.ref_name }}
-           prerelease: false
-           title: "${{ github.ref_name }}"
-           files: |
-                wheels/*.whl
-                wasm-artefacts/bergamot-translator-worker.js
-                wasm-artefacts/bergamot-translator-worker.wasm
-
-
-    python-checks:
-      name: "formatting and typechecks"
-      runs-on: "ubuntu-latest"
-      steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-        with:
-          submodules: recursive
-      - name: Install Dependencies
-        run: |-
-            python3 -m pip install black isort pytype
-      - name: "Formatting checks: black, isort"
-        run: |
-            python3 -m black --diff --check bindings/python/ setup.py doc/conf.py
-            python3 -m isort --profile black --diff --check bindings/python setup.py doc/conf.py
-      - name: "Static typing checks: pytype"
-        run: |-
-            python3 -m pytype bindings/python
-
-    docs:
-      runs-on: ubuntu-latest
-      needs: [build-wheels]
-      steps:
-        - name: Checkout
-          uses: actions/checkout@v2
-          with:
-            submodules: recursive
-
-        # Runs javascript to extract push events from both tags and branch (only main, due to workflow trigger)
-        # converts refs/<>/<name> -> <name>
-        # eg:
-        #     refs/head/main   -> main
-        #     refs/tags/v0.1.0 -> v0.1.0
-        #
-        - name: Download artifacts
-          uses: actions/download-artifact@v2
-        - name: Extract tag name
-          id: tag
-          uses: actions/github-script@0.2.0
-          if: ${{ github.event_name == 'push' }}
-          with:
-            github-token: ${{ secrets.GITHUB_TOKEN }}
-            script: |
-              const args = context.payload.ref.split("/");
-              [refs, category, ...rest] = args;
-              return rest.join("/");
-
-        # Patches the BERGAMOT_VERSION file used by sphinx-docs at run time to
-        # obtain names like 'main' or 'ci-sandbox' to not confuse with version
-        # based documentation built separately.
-        - name: Deploy-time patch version
-          run: |
-              echo ${{steps.tag.outputs.result }} > BERGAMOT_VERSION
-
-        - name: Set up Doxygen
-          run: sudo apt-get install -y doxygen
-
-        - name: Set up Python
-          uses: actions/setup-python@v2
-          with:
-            python-version: 3.7
-
-        - name: Set up dependency cache
-          uses: actions/cache@v2
-          with:
-            path: ~/.cache/pip
-            key: ${{ runner.os }}-pip-${{ hashFiles('doc/requirements.txt') }}
-            restore-keys: |
-              ${{ runner.os }}-pip-
-
-        - name: Install dependencies
-          working-directory: ./doc
-          run: |
-            python3 -m pip install -r requirements.txt
-            python3 -m pip install --find-links=${{github.workspace}}/wheels bergamot
-
-        - name: Build documentation
-          working-directory: ./doc
-          run: sphinx-build -b html ./ build/
-
-
-        - name: Deploy 🚀
-          uses: JamesIves/github-pages-deploy-action@4.1.3
-          if: ${{ github.event_name == 'push' && github.repository == 'browsermt/bergamot-translator' }}
-          with:
-            repository-name: 'browsermt/docs'
-            branch: gh-pages # The branch the action should deploy to.
-            folder: './doc/build/' # The folder the action should deploy.
-            target-folder: '${{ steps.tag.outputs.result }}'
-            ssh-key: ${{ secrets.BERGAMOT_SSH_PRIVATE_KEY }}
-
-        # This artifact contains the HTML output of Sphinx only.
-        # With index.html at the root of the produced zip file.
-        # For use for maintainers to download the zip and check render of
-        # documentation while generated at pull-request.
-        - name: Upload documentation
-          uses: actions/upload-artifact@v2
-          if: ${{ github.event_name == 'pull_request'}}
-          with:
-            name: api-docs
-            path: ./doc/build/
-            if-no-files-found: error
diff --git a/inference-engine/.github/workflows/coding-styles.yml b/inference-engine/.github/workflows/coding-styles.yml
deleted file mode 100644
index b13345601..000000000
--- a/inference-engine/.github/workflows/coding-styles.yml
+++ /dev/null
@@ -1,42 +0,0 @@
-name: "Coding Style"
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ '**' ]
-
-jobs:
-  clang-format:
-      name: "clang-format"
-      runs-on: ubuntu-latest
-      steps:
-        - name: Checkout
-          uses: actions/checkout@v2
-          with:
-            submodules: recursive
-
-        - name: Install dependencies
-          run: |
-            sudo apt-get update
-            sudo apt-get install -y build-essential cmake
-            sudo apt-get install -y clang-format clang-tidy
-
-        - name: Run clang-format
-          run:
-              python3 run-clang-format.py --style file -r src wasm bindings/python
-
-
-        - name: Prepare build, compilation database etc.
-          run: |
-              mkdir -p build
-              cd build
-              cmake \
-                -DUSE_WASM_COMPATIBLE_SOURCE=off -DCMAKE_EXPORT_COMPILE_COMMANDS=on \
-                -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
-                ..
-
-        - name: Run clang-tidy
-          run: |
-              run-clang-tidy -p build "$PWD/src/.*"
-              run-clang-tidy -p build "$PWD/app/.*"
diff --git a/inference-engine/.github/workflows/native.yml b/inference-engine/.github/workflows/native.yml
deleted file mode 100644
index 505381cbc..000000000
--- a/inference-engine/.github/workflows/native.yml
+++ /dev/null
@@ -1,243 +0,0 @@
-name: native
-'on':
-  push:
-    branches:
-    - main
-    - ci-sandbox
-  pull_request:
-    branches:
-    - '**'
-env:
-  ccache_basedir: ${{ github.workspace }}
-  ccache_dir: "${{ github.workspace }}/.ccache"
-  ccache_compilercheck: content
-  ccache_compress: 'true'
-  ccache_compresslevel: 9
-  ccache_maxsize: 200M
-  ccache_cmake: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
-jobs:
-  ubuntu:
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-        - name: Ubuntu 22.04 full
-          os: ubuntu-22.04
-          identifier: ubuntu_2204_full
-          cmake: -DCOMPILE_TESTS=on
-          brt_tags: ""
-          unittests: 'true'
-        - name: Ubuntu 22.04 minimal
-          os: ubuntu-22.04
-          identifier: ubuntu_2204_minimal
-          cmake: -DCOMPILE_TESTS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
-          brt_tags: "'#wasm'"
-          unittests: 'false'
-        - name: Ubuntu 20.04 full
-          os: ubuntu-20.04
-          identifier: ubuntu_2004_full
-          cmake: -DCOMPILE_TESTS=on
-          brt_tags: ""
-          unittests: 'true'
-        - name: Ubuntu 20.04 minimal
-          os: ubuntu-20.04
-          identifier: ubuntu_2004_minimal
-          cmake: -DCOMPILE_TESTS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
-          brt_tags: "'#wasm'"
-          unittests: 'false'
-    name: ${{ matrix.name }}
-    runs-on: ${{ matrix.os }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-    - name: Install Dependencies
-      run: |-
-        sudo apt-get update
-        sudo apt-get install -y libprotobuf-dev protobuf-compiler libboost-all-dev ccache libunwind-dev libgoogle-perftools-dev
-    - name: Install MKL
-      run: |-
-        wget -qO- "https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB" | sudo apt-key add -
-        sudo sh -c "echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list"
-        sudo apt-get update -o Dir::Etc::sourcelist="/etc/apt/sources.list.d/intel-mkl.list"
-        sudo apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088
-    - name: Generate ccache_vars for ccache based on machine
-      shell: bash
-      id: ccache_vars
-      run: |-
-        echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
-        echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
-    - name: Cache-op for build-cache through ccache
-      uses: actions/cache@v2
-      with:
-        path: ${{ env.ccache_dir }}
-        key: ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-        restore-keys: |-
-          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
-          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}
-          ccache-${{ matrix.identifier }}
-    - name: ccache environment setup
-      run: |-
-        echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
-        echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
-        echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
-        echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
-        echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
-        echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
-    - name: ccache prolog
-      run: |-
-        ccache -s # Print current cache stats
-        ccache -z # Zero cache entry
-    - name: cmake
-      run: |-
-        mkdir -p build
-        cd build
-        cmake -L .. ${{ matrix.cmake }} ${{ env.ccache_cmake }}
-    - name: Build from source
-      working-directory: build
-      run: make -j2
-    - name: ccache epilog
-      run: 'ccache -s # Print current cache stats'
-    - name: Print Versions
-      working-directory: build
-      run: ./app/bergamot --version
-    - name: Run unit tests
-      working-directory: build
-      run: make test
-      if: ${{ matrix.unittests == 'true' }}
-    - name: Install regression-test framework (BRT)
-      working-directory: bergamot-translator-tests
-      run: make install
-    - name: Run regression-tests (BRT)
-      working-directory: bergamot-translator-tests
-      id: brt_run
-      run: MARIAN=../build ./run_brt.sh ${{ matrix.brt_tags }}
-    - name: Print logs of unsuccessful BRTs
-      working-directory: bergamot-translator-tests
-      run: |-
-        grep "tests.*.sh" previous.log \
-          | sed 's/^\s*-\s*//' \
-          | xargs -I% bash -c 'echo %; tail -n20 %.log'
-      if: ${{ always() && steps.brt_run.outcome == 'failure' }}
-    - name: Upload regression-tests artifacts
-      uses: actions/upload-artifact@v2
-      if: ${{ always() && steps.brt_run.outcome != 'skipped' }}
-      with:
-        name: brt-${{ matrix.identifier }}
-        path: |-
-          bergamot-translator-tests/**/*.expected
-          bergamot-translator-tests/**/*.log
-          bergamot-translator-tests/**/*.out
-    - name: Confirm native-run example script works
-      run: |-
-          bash examples/run-native.sh
-
-  mac:
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-        - name: MacOS 12 full
-          os: macos-12
-          identifier: mac_12_full
-          cmake: -DCOMPILE_TESTS=on -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=off
-          brt_tags: ""
-          unittests: 'true'
-        - name: MacOS 12 minimal
-          os: macos-12
-          identifier: mac_12_minimal
-          cmake: -DCOMPILE_TESTS=on -DUSE_APPLE_ACCELERATE=off -DUSE_FBGEMM=off -DUSE_STATIC_LIBS=on -DUSE_WASM_COMPATIBLE_SOURCE=on
-          brt_tags: "'#wasm'"
-          unittests: 'false'
-    name: ${{ matrix.name }}
-    runs-on: ${{ matrix.os }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-    - name: Install Dependencies
-      run: |-
-        brew update
-        brew install openblas protobuf ccache
-        brew install coreutils findutils
-    - name: Setup path with gnu
-      run: |-
-        echo "/usr/local/opt/coreutils/libexec/gnubin" >> $GITHUB_PATH
-        echo "/usr/local/opt/findutils/libexec/gnubin" >> $GITHUB_PATH
-    - name: Setup BLAS
-      run: |-
-        echo "LDFLAGS=-L/usr/local/opt/openblas/lib" >> $GITHUB_ENV
-        echo "CPPFLAGS=-I/usr/local/opt/openblas/include" >> $GITHUB_ENV
-    - name: Generate ccache_vars for ccache based on machine
-      shell: bash
-      id: ccache_vars
-      run: |-
-        echo "::set-output name=hash::$(echo ${{ env.ccache_compilercheck }})"
-        echo "::set-output name=timestamp::$(date '+%Y-%m-%dT%H.%M.%S')"
-    - name: Cache-op for build-cache through ccache
-      uses: actions/cache@v2
-      with:
-        path: ${{ env.ccache_dir }}
-        key: ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-        restore-keys: |-
-          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
-          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}
-          ccache-${{ matrix.identifier }}
-    - name: ccache environment setup
-      run: |-
-        echo "CCACHE_COMPILER_CHECK=${{ env.ccache_compilercheck }}" >> $GITHUB_ENV
-        echo "CCACHE_BASEDIR=${{ env.ccache_basedir }}" >> $GITHUB_ENV
-        echo "CCACHE_COMPRESS=${{ env.ccache_compress }}" >> $GITHUB_ENV
-        echo "CCACHE_COMPRESSLEVEL=${{ env.ccache_compresslevel }}" >> $GITHUB_ENV
-        echo "CCACHE_DIR=${{ env.ccache_dir }}" >> $GITHUB_ENV
-        echo "CCACHE_MAXSIZE=${{ env.ccache_maxsize }}" >> $GITHUB_ENV
-    - name: ccache prolog
-      run: |-
-        ccache -s # Print current cache stats
-        ccache -z # Zero cache entry
-    - name: cmake
-      run: |-
-        mkdir -p build
-        cd build
-        cmake -L .. ${{ matrix.cmake }} ${{ env.ccache_cmake }}
-    - name: Build from source
-      working-directory: build
-      run: make -j2
-    - name: ccache epilog
-      run: 'ccache -s # Print current cache stats'
-    - name: Print Versions
-      working-directory: build
-      run: ./app/bergamot --version
-    - name: Run unit tests
-      working-directory: build
-      run: make test
-      if: ${{ matrix.unittests == 'true' }}
-    - name: Install regression-test framework (BRT)
-      working-directory: bergamot-translator-tests
-      run: make install
-    - name: Run regression-tests (BRT)
-      working-directory: bergamot-translator-tests
-      id: brt_run
-      run: MARIAN=../build ./run_brt.sh ${{ matrix.brt_tags }}
-    - name: Print logs of unsuccessful BRTs
-      working-directory: bergamot-translator-tests
-      run: |-
-        grep "tests.*.sh" previous.log \
-          | sed 's/^\s*-\s*//' \
-          | xargs -I% bash -c 'echo %; tail -n20 %.log'
-      if: ${{ always() && steps.brt_run.outcome == 'failure' }}
-    - name: Upload regression-tests artifacts
-      uses: actions/upload-artifact@v2
-      if: ${{ always() && steps.brt_run.outcome != 'skipped' }}
-      with:
-        name: brt-${{ matrix.identifier }}
-        path: |-
-          bergamot-translator-tests/**/*.expected
-          bergamot-translator-tests/**/*.log
-          bergamot-translator-tests/**/*.out
-    - name: Confirm native-run example script works
-      run: |-
-          bash examples/run-native.sh
-
diff --git a/inference-engine/.github/workflows/windows.yml b/inference-engine/.github/workflows/windows.yml
deleted file mode 100644
index a0ff86b84..000000000
--- a/inference-engine/.github/workflows/windows.yml
+++ /dev/null
@@ -1,128 +0,0 @@
-name: Windows
-
-on:
-  push:
-    branches: [ main, ci-sandbox ]
-  pull_request:
-    branches: [ '**' ]
-
-env:
-  MKL_URL: "https://data.statmt.org/romang/marian-regression-tests/ci/mkl-2020.1-windows-static.zip"
-  CCACHE_BASEDIR: "${{ github.workspace }}"
-  CCACHE_DIR: "${{ github.workspace }}\\ccache"
-  CCACHE_COMPILERCHECK: content
-  CCACHE_COMPRESS: 'true'
-  CCACHE_COMPRESSLEVEL: 9
-  CCACHE_MAXSIZE: 200M
-  ccache_version: '4.5'
-
-jobs:
-  build-windows:
-    strategy:
-      matrix:
-        include:
-          # Windows CPU-only build
-          - name: "Windows CPU-only"
-            identifier: "windows-x64"
-
-    runs-on: windows-2019
-    name: ${{ matrix.name }}
-
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v2
-      with:
-        submodules: recursive
-
-
-    - name: Download ccache
-      shell: cmake -P {0}
-      run: |
-        set(ccache_url "https://github.com/cristianadam/ccache/releases/download/v${{ env.ccache_version }}/${{ runner.os }}.tar.xz")
-        file(DOWNLOAD "${ccache_url}" ./ccache.tar.xz SHOW_PROGRESS)
-        execute_process(COMMAND ${CMAKE_COMMAND} -E tar xvf ./ccache.tar.xz)
-        if(ret AND NOT ret EQUAL 0)
-          message( FATAL_ERROR "Bad exit status")
-        endif()
-
-    - name: Generate ccache_vars for ccache based on machine
-      shell: cmake -P {0}
-      id: ccache_vars
-      run: |-
-        string(TIMESTAMP current_date "%Y-%m-%d-%H;%M;%S" UTC)
-        message("::set-output name=timestamp::${current_date}")
-        message("::set-output name=hash::${{ env.ccache_compilercheck }}")
-
-    - name: Cache-op for build-cache through ccache
-      uses: actions/cache@v2
-      with:
-        path: ${{ env.CCACHE_DIR }}
-        key: ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}-${{ steps.ccache_vars.outputs.timestamp }}
-        restore-keys: |-
-          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}-${{ github.ref }}
-          ccache-${{ matrix.identifier }}-${{ steps.ccache_vars.outputs.hash }}
-          ccache-${{ matrix.identifier }}
-
-    - name: ccache prolog
-      run: |-
-        ${{github.workspace}}\ccache.exe -sv # Print current cache stats
-        ${{github.workspace}}\ccache.exe -z # Print current cache stats
-
-    - name: Download MKL
-      run: |
-        # Wget retries downloading files and is faster than Invoke-WebRequest
-        C:\msys64\usr\bin\wget.exe -nv ${{ env.MKL_URL }} -O mkl.zip
-        Expand-Archive -Force mkl.zip ${{ github.workspace }}\mkl
-        # Set MKLROOT environment variable so that CMake can find MKL
-        echo "MKLROOT=${{ github.workspace }}\mkl" | Out-File -FilePath $env:GITHUB_ENV  -Encoding utf8 -Append
-      shell: powershell
-
-    - name: Disable debug vcpkg build
-      shell: powershell
-      working-directory: C:\vcpkg\triplets
-      run: |
-         $PSDefaultParameterValues['Out-File:Encoding'] = 'utf8' # Powershell murders me.
-         echo "set(VCPKG_BUILD_TYPE release)" | Tee-Object -FilePath x64-windows-static.cmake -Append
-         echo "set(VCPKG_BUILD_TYPE release)" | Tee-Object -FilePath x64-windows.cmake -Append
-         cat x64-windows-static.cmake
-         cat x64-windows.cmake
-
-    - name: Install dependencies with vcpkg
-      working-directory: C:\vcpkg
-      run: |
-        $Env:VCPKG_BUILD_TYPE = 'release'
-        $Env:VCPKG_DEFAULT_TRIPLET = 'x64-windows-static' # QT6 version, linguist tools not working yet: qtbase:x64-windows-static qttools:x64-windows-static qtsvg:x64-windows-static
-        .\vcpkg install protobuf:x64-windows-static pcre2:x64-windows-static 
-        .\vcpkg upgrade --no-dry-run # In case there are new builds available after cache restoration
-      shell: powershell
-        
-    - name: Create Build Environment
-      # Some projects don't allow in-source building, so create a separate build directory
-      # We'll use this as our working directory for all subsequent commands
-      run: cmake -E make_directory ${{github.workspace}}/build
-      
-    - name: Configure
-      working-directory: ${{github.workspace}}/build #@TODO figure out how variables are accessed from power shell, as they seem to not be read.
-      run: |
-          cmake .. -DCMAKE_BUILD_TYPE=Release -DUSE_STATIC_LIBS=ON  -DVCPKG_TARGET_TRIPLET='x64-windows-static' `
-                -DCMAKE_TOOLCHAIN_FILE="C:/vcpkg/scripts/buildsystems/vcpkg.cmake"  `
-                -DCMAKE_CXX_COMPILER_LAUNCHER=${{github.workspace}}\ccache.exe `
-                -DCMAKE_C_COMPILER_LAUNCHER=${{github.workspace}}\ccache.exe
-      shell: powershell
-
-    - name: Build
-      working-directory: ${{github.workspace}}/build
-      run: cmake --build . --config Release -j3
-      shell: powershell
-
-
-    - name: Print versions
-      working-directory: ${{github.workspace}}/build
-      run: |
-        .\app\Release\bergamot.exe --version
-
-      shell: cmd
-
-    - name: ccache epilog
-      run: |-
-        ${{github.workspace}}\\ccache.exe -sv # Print current cache stats

From 27e85d25a7f9f2c34169c279c2ea6fae1d30e7ae Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Fri, 20 Sep 2024 13:07:21 -0500
Subject: [PATCH 427/442] Remove unneeded Python code

---
 .gitmodules                                   |   4 -
 inference-engine/3rd_party/CMakeLists.txt     |   6 +-
 inference-engine/3rd_party/pybind11           |   1 -
 inference-engine/bindings/CMakeLists.txt      |   1 -
 .../bindings/python/CMakeLists.txt            |   9 -
 inference-engine/bindings/python/README.md    |  14 -
 inference-engine/bindings/python/__init__.py  |  18 -
 inference-engine/bindings/python/__main__.py  |  20 -
 inference-engine/bindings/python/bergamot.cpp | 213 ---------
 inference-engine/bindings/python/cmds.py      | 177 --------
 .../bindings/python/repository.py             | 218 ----------
 .../bindings/python/typing_utils.py           |   5 -
 inference-engine/bindings/python/utils.py     |  52 ---
 inference-engine/run-clang-format.py          | 408 ------------------
 inference-engine/setup.py                     | 248 -----------
 15 files changed, 1 insertion(+), 1393 deletions(-)
 delete mode 160000 inference-engine/3rd_party/pybind11
 delete mode 100644 inference-engine/bindings/CMakeLists.txt
 delete mode 100644 inference-engine/bindings/python/CMakeLists.txt
 delete mode 100644 inference-engine/bindings/python/README.md
 delete mode 100644 inference-engine/bindings/python/__init__.py
 delete mode 100644 inference-engine/bindings/python/__main__.py
 delete mode 100644 inference-engine/bindings/python/bergamot.cpp
 delete mode 100644 inference-engine/bindings/python/cmds.py
 delete mode 100644 inference-engine/bindings/python/repository.py
 delete mode 100644 inference-engine/bindings/python/typing_utils.py
 delete mode 100644 inference-engine/bindings/python/utils.py
 delete mode 100644 inference-engine/run-clang-format.py
 delete mode 100644 inference-engine/setup.py

diff --git a/.gitmodules b/.gitmodules
index ce6f3230b..a07948957 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -10,10 +10,6 @@
 	path = inference-engine/3rd_party/browsermt-marian-dev
 	url = https://github.com/browsermt/marian-dev
 
-[submodule "inference-engine/3rd_party/pybind11"]
-	path = inference-engine/3rd_party/pybind11
-	url = https://github.com/pybind/pybind11.git
-
 [submodule "inference-engine/3rd_party/ssplit-cpp"]
 	path = inference-engine/3rd_party/ssplit-cpp
 	url = https://github.com/browsermt/ssplit-cpp
diff --git a/inference-engine/3rd_party/CMakeLists.txt b/inference-engine/3rd_party/CMakeLists.txt
index 0185d7673..62ba02722 100644
--- a/inference-engine/3rd_party/CMakeLists.txt
+++ b/inference-engine/3rd_party/CMakeLists.txt
@@ -29,8 +29,4 @@ target_compile_options(marian PUBLIC ${COMPILE_OPTIONS})
 get_directory_property(CMAKE_C_FLAGS DIRECTORY browsermt-marian-dev DEFINITION CMAKE_C_FLAGS) 
 get_directory_property(CMAKE_CXX_FLAGS DIRECTORY browsermt-marian-dev DEFINITION CMAKE_CXX_FLAGS) 
 set(CMAKE_C_FLAGS ${CMAKE_C_FLAGS} PARENT_SCOPE)    
-set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} PARENT_SCOPE)    
-
-if(COMPILE_PYTHON)
-  add_subdirectory(pybind11)
-endif(COMPILE_PYTHON)
+set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} PARENT_SCOPE)
diff --git a/inference-engine/3rd_party/pybind11 b/inference-engine/3rd_party/pybind11
deleted file mode 160000
index 9ec1128c7..000000000
--- a/inference-engine/3rd_party/pybind11
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit 9ec1128c7aac3d069a4ec2bd1dfc7f57c6526d1c
diff --git a/inference-engine/bindings/CMakeLists.txt b/inference-engine/bindings/CMakeLists.txt
deleted file mode 100644
index 8e5f91a37..000000000
--- a/inference-engine/bindings/CMakeLists.txt
+++ /dev/null
@@ -1 +0,0 @@
-add_subdirectory(python)
diff --git a/inference-engine/bindings/python/CMakeLists.txt b/inference-engine/bindings/python/CMakeLists.txt
deleted file mode 100644
index 16e3e48d3..000000000
--- a/inference-engine/bindings/python/CMakeLists.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-find_package(Python COMPONENTS Interpreter Development.Module REQUIRED)
-
-message("Using Python: " ${Python_EXECUTABLE})
-
-# pybind11 method:
-pybind11_add_module(_bergamot SHARED bergamot.cpp)
-target_link_libraries(_bergamot PUBLIC pybind11::module pybind11::headers bergamot-translator)
-target_include_directories(_bergamot PUBLIC ${PROJECT_SOURCE_DIR} ${PROJECT_SOURCE_DIR}/src 
-    ${CMAKE_BINARY_DIR}/src)
diff --git a/inference-engine/bindings/python/README.md b/inference-engine/bindings/python/README.md
deleted file mode 100644
index 3797b7dea..000000000
--- a/inference-engine/bindings/python/README.md
+++ /dev/null
@@ -1,14 +0,0 @@
-# bergamot-translator
-
-The [Bergamot project](https://browser.mt/) adds and improves client-side
-machine translation in a web browser.
-
-This package provides Python bindings to bergamot-translator developed as part
-of the Bergamot Project and extras assorted in a package to enable further use
-of the library developed for local-translation on the consumer machine.
-
-Bergamot is a consortium coordinated by the University of Edinburgh with
-partners Charles University in Prague, the University of Sheffield, University
-of Tartu, and Mozilla.
-
-
diff --git a/inference-engine/bindings/python/__init__.py b/inference-engine/bindings/python/__init__.py
deleted file mode 100644
index 5855a4faf..000000000
--- a/inference-engine/bindings/python/__init__.py
+++ /dev/null
@@ -1,18 +0,0 @@
-import typing
-
-from ._bergamot import *  # type: ignore
-from .repository import Aggregator, TranslateLocallyLike
-
-REPOSITORY = Aggregator(
-    [
-        TranslateLocallyLike("browsermt", "https://translatelocally.com/models.json"),
-        TranslateLocallyLike(
-            "opus", "https://object.pouta.csc.fi/OPUS-MT-models/app/models.json"
-        ),
-    ]
-)
-"""
-REPOSITORY is a global object that aggregates multiple model-providers to
-provide a (model-provider: str, model-code: str) based query mechanism to
-get models.
-"""
diff --git a/inference-engine/bindings/python/__main__.py b/inference-engine/bindings/python/__main__.py
deleted file mode 100644
index 35014c099..000000000
--- a/inference-engine/bindings/python/__main__.py
+++ /dev/null
@@ -1,20 +0,0 @@
-import argparse
-import sys
-from argparse import ArgumentParser
-
-from .cmds import CMDS, make_parser
-
-
-def main() -> None:
-    parser = make_parser()
-    args = parser.parse_args()
-
-    if args.action in CMDS:
-        CMDS[args.action].execute(args)
-    else:
-        parser.print_help(sys.stderr)
-        sys.exit(1)
-
-
-if __name__ == "__main__":
-    main()
diff --git a/inference-engine/bindings/python/bergamot.cpp b/inference-engine/bindings/python/bergamot.cpp
deleted file mode 100644
index 2ffb2267e..000000000
--- a/inference-engine/bindings/python/bergamot.cpp
+++ /dev/null
@@ -1,213 +0,0 @@
-#include <pybind11/iostream.h>
-#include <pybind11/pybind11.h>
-#include <pybind11/stl.h>
-#include <pybind11/stl_bind.h>
-#include <translator/annotation.h>
-#include <translator/parser.h>
-#include <translator/project_version.h>
-#include <translator/response.h>
-#include <translator/response_options.h>
-#include <translator/service.h>
-#include <translator/translation_model.h>
-
-#include <iostream>
-#include <string>
-#include <vector>
-
-namespace py = pybind11;
-
-using marian::bergamot::AnnotatedText;
-using marian::bergamot::ByteRange;
-using marian::bergamot::ConcatStrategy;
-using marian::bergamot::Response;
-using marian::bergamot::ResponseOptions;
-using Service = marian::bergamot::AsyncService;
-using _Model = marian::bergamot::TranslationModel;
-using Model = std::shared_ptr<_Model>;
-using Alignment = std::vector<std::vector<float>>;
-using Alignments = std::vector<Alignment>;
-
-PYBIND11_MAKE_OPAQUE(std::vector<Response>);
-PYBIND11_MAKE_OPAQUE(std::vector<std::string>);
-PYBIND11_MAKE_OPAQUE(Alignments);
-
-class ServicePyAdapter {
- public:
-  ServicePyAdapter(const Service::Config &config) : service_(make_service(config)) {
-    // Set marian to throw exceptions instead of std::abort()
-    marian::setThrowExceptionOnAbort(true);
-  }
-
-  std::shared_ptr<_Model> modelFromConfig(const std::string &config) {
-    auto parsedConfig = marian::bergamot::parseOptionsFromString(config);
-    return service_.createCompatibleModel(parsedConfig);
-  }
-
-  std::shared_ptr<_Model> modelFromConfigPath(const std::string &configPath) {
-    auto config = marian::bergamot::parseOptionsFromFilePath(configPath);
-    return service_.createCompatibleModel(config);
-  }
-
-  std::vector<Response> translate(Model model, std::vector<std::string> &inputs, const ResponseOptions &options) {
-    py::scoped_ostream_redirect outstream(std::cout,                                 // std::ostream&
-                                          py::module_::import("sys").attr("stdout")  // Python output
-    );
-    py::scoped_ostream_redirect errstream(std::cerr,                                 // std::ostream&
-                                          py::module_::import("sys").attr("stderr")  // Python output
-    );
-
-    py::call_guard<py::gil_scoped_release> gil_guard;
-
-    // Prepare promises, save respective futures. Have callback's in async set
-    // value to the promises.
-    std::vector<std::future<Response>> futures;
-    std::vector<std::promise<Response>> promises;
-    promises.resize(inputs.size());
-
-    for (size_t i = 0; i < inputs.size(); i++) {
-      auto callback = [&promises, i](Response &&response) { promises[i].set_value(std::move(response)); };
-
-      service_.translate(model, std::move(inputs[i]), std::move(callback), options);
-
-      futures.push_back(std::move(promises[i].get_future()));
-    }
-
-    // Wait on all futures to be ready.
-    std::vector<Response> responses;
-    for (size_t i = 0; i < futures.size(); i++) {
-      futures[i].wait();
-      responses.push_back(std::move(futures[i].get()));
-    }
-
-    return responses;
-  }
-
-  std::vector<Response> pivot(Model first, Model second, std::vector<std::string> &inputs,
-                              const ResponseOptions &options) {
-    py::scoped_ostream_redirect outstream(std::cout,                                 // std::ostream&
-                                          py::module_::import("sys").attr("stdout")  // Python output
-    );
-    py::scoped_ostream_redirect errstream(std::cerr,                                 // std::ostream&
-                                          py::module_::import("sys").attr("stderr")  // Python output
-    );
-
-    py::call_guard<py::gil_scoped_release> gil_guard;
-    // Prepare promises, save respective futures. Have callback's in async set
-    // value to the promises.
-    std::vector<std::future<Response>> futures;
-    std::vector<std::promise<Response>> promises;
-    promises.resize(inputs.size());
-
-    for (size_t i = 0; i < inputs.size(); i++) {
-      auto callback = [&promises, i](Response &&response) { promises[i].set_value(std::move(response)); };
-
-      service_.pivot(first, second, std::move(inputs[i]), std::move(callback), options);
-
-      futures.push_back(std::move(promises[i].get_future()));
-    }
-
-    // Wait on all futures to be ready.
-    std::vector<Response> responses;
-    for (size_t i = 0; i < futures.size(); i++) {
-      futures[i].wait();
-      responses.push_back(std::move(futures[i].get()));
-    }
-
-    return responses;
-  }
-
-  private /*functions*/:
-  static Service make_service(const Service::Config &config) {
-    py::scoped_ostream_redirect outstream(std::cout,                                 // std::ostream&
-                                          py::module_::import("sys").attr("stdout")  // Python output
-    );
-    py::scoped_ostream_redirect errstream(std::cerr,                                 // std::ostream&
-                                          py::module_::import("sys").attr("stderr")  // Python output
-    );
-
-    py::call_guard<py::gil_scoped_release> gil_guard;
-
-    return Service(config);
-  }
-
-  private /*data*/:
-  Service service_;
-};
-
-PYBIND11_MODULE(_bergamot, m) {
-  m.doc() = "Bergamot pybind11 bindings";
-  m.attr("__version__") = marian::bergamot::bergamotBuildVersion();
-  py::class_<ByteRange>(m, "ByteRange")
-      .def(py::init<>())
-      .def_readonly("begin", &ByteRange::begin)
-      .def_readonly("end", &ByteRange::end)
-      .def("__repr__", [](const ByteRange &range) {
-        return "{" + std::to_string(range.begin) + ", " + std::to_string(range.end) + "}";
-      });
-
-  py::class_<AnnotatedText>(m, "AnnotatedText")
-      .def(py::init<>())
-      .def("numWords", &AnnotatedText::numWords)
-      .def("numSentences", &AnnotatedText::numSentences)
-      .def("word",
-           [](const AnnotatedText &annotatedText, size_t sentenceIdx, size_t wordIdx) -> std::string {
-             auto view = annotatedText.word(sentenceIdx, wordIdx);
-             return std::string(view.data(), view.size());
-           })
-      .def("sentence",
-           [](const AnnotatedText &annotatedText, size_t sentenceIdx) -> std::string {
-             auto view = annotatedText.sentence(sentenceIdx);
-             return std::string(view.data(), view.size());
-           })
-      .def("wordAsByteRange", &AnnotatedText::wordAsByteRange)
-      .def("sentenceAsByteRange", &AnnotatedText::sentenceAsByteRange)
-      .def_readonly("text", &AnnotatedText::text);
-
-  py::class_<Response>(m, "Response")
-      .def(py::init<>())
-      .def_readonly("source", &Response::source)
-      .def_readonly("target", &Response::target)
-      .def_readonly("alignments", &Response::alignments);
-
-  py::bind_vector<std::vector<std::string>>(m, "VectorString");
-  py::bind_vector<std::vector<Response>>(m, "VectorResponse");
-
-  py::enum_<ConcatStrategy>(m, "ConcatStrategy")
-      .value("FAITHFUL", ConcatStrategy::FAITHFUL)
-      .value("SPACE", ConcatStrategy::SPACE)
-      .export_values();
-
-  py::class_<ResponseOptions>(m, "ResponseOptions")
-      .def(
-          py::init<>([](bool qualityScores, bool alignment, bool HTML, bool sentenceMappings, ConcatStrategy strategy) {
-            return ResponseOptions{qualityScores, alignment, HTML, sentenceMappings, strategy};
-          }),
-          py::arg("qualityScores") = true, py::arg("alignment") = false, py::arg("HTML") = false,
-          py::arg("sentenceMappings") = true, py::arg("concatStrategy") = ConcatStrategy::FAITHFUL)
-      .def_readwrite("qualityScores", &ResponseOptions::qualityScores)
-      .def_readwrite("HTML", &ResponseOptions::HTML)
-      .def_readwrite("alignment", &ResponseOptions::alignment)
-      .def_readwrite("concatStrategy", &ResponseOptions::concatStrategy)
-      .def_readwrite("sentenceMappings", &ResponseOptions::sentenceMappings);
-
-  py::class_<ServicePyAdapter>(m, "Service")
-      .def(py::init<const Service::Config &>())
-      .def("modelFromConfig", &ServicePyAdapter::modelFromConfig)
-      .def("modelFromConfigPath", &ServicePyAdapter::modelFromConfigPath)
-      .def("translate", &ServicePyAdapter::translate)
-      .def("pivot", &ServicePyAdapter::pivot);
-
-  py::class_<Service::Config>(m, "ServiceConfig")
-      .def(py::init<>([](size_t numWorkers, size_t cacheSize, std::string logging) {
-             Service::Config config;
-             config.numWorkers = numWorkers;
-             config.cacheSize = cacheSize;
-             config.logger.level = logging;
-             return config;
-           }),
-           py::arg("numWorkers") = 1, py::arg("cacheSize") = 0, py::arg("logLevel") = "off")
-      .def_readwrite("numWorkers", &Service::Config::numWorkers)
-      .def_readwrite("cacheSize", &Service::Config::cacheSize);
-
-  py::class_<_Model, std::shared_ptr<_Model>>(m, "TranslationModel");
-}
diff --git a/inference-engine/bindings/python/cmds.py b/inference-engine/bindings/python/cmds.py
deleted file mode 100644
index 5949adaca..000000000
--- a/inference-engine/bindings/python/cmds.py
+++ /dev/null
@@ -1,177 +0,0 @@
-import argparse
-import sys
-from collections import Counter, defaultdict
-
-from . import REPOSITORY, ResponseOptions, Service, ServiceConfig, VectorString
-
-CMDS = {}
-
-
-def _register_cmd(cmd: str):
-    """
-    Convenience decorator function, which populates the dictionary above with
-    commands created in a declarative fashion.
-    """
-
-    def __inner(cls):
-        CMDS[cmd] = cls
-        return cls
-
-    return __inner
-
-
-@_register_cmd("translate")
-class Translate:
-    @staticmethod
-    def embed_subparser(key: str, subparsers: argparse._SubParsersAction):
-        translate = subparsers.add_parser(
-            key,
-            description="translate using a given model. Multiple models mean pivoting",
-        )
-
-        translate.add_argument(
-            "-m",
-            "--model",
-            type=str,
-            nargs="+",
-            help="Path to model file(s) to use in forward or pivot translation",
-            required=True,
-        )
-
-        translate.add_argument(
-            "-r",
-            "--repository",
-            type=str,
-            help="Repository to download model from",
-            choices=REPOSITORY.available(),
-            default="browsermt",
-        )
-
-        translate.add_argument(
-            "--num-workers",
-            type=int,
-            help="Number of worker threads to use to translate",
-            default=4,
-        )
-
-        translate.add_argument(
-            "--log-level",
-            type=str,
-            default="off",
-            help="Set verbosity level of logging: trace, debug, info, warn, err(or), critical, off",
-        )
-
-        # Tweak response-options for quick HTML in out via commandline
-        options = translate.add_argument_group("response-options")
-        options.add_argument("--html", type=bool, default=False)
-        options.add_argument("--alignment", type=bool, default=False)
-        options.add_argument("--quality-scores", type=bool, default=False)
-
-    @staticmethod
-    def execute(args: argparse.Namespace):
-        # Build service
-
-        config = ServiceConfig(numWorkers=args.num_workers, logLevel=args.log_level)
-        service = Service(config)
-
-        models = [
-            service.modelFromConfigPath(
-                REPOSITORY.modelConfigPath(args.repository, model)
-            )
-            for model in args.model
-        ]
-
-        # Configure a few options which require how a Response is constructed
-        options = ResponseOptions(
-            alignment=args.alignment, qualityScores=args.quality_scores, HTML=args.html
-        )
-
-        source = sys.stdin.read()
-        responses = None
-        if len(models) == 1:
-            [model] = models
-            responses = service.translate(model, VectorString([source]), options)
-        else:
-            [first, second] = models
-            responses = service.pivot(first, second, VectorString([source]), options)
-
-        for response in responses:
-            print(response.target.text, end="")
-
-
-@_register_cmd("download")
-class Download:
-    @staticmethod
-    def embed_subparser(key: str, subparsers: argparse._SubParsersAction):
-        download = subparsers.add_parser(
-            key, description="Download models from the web."
-        )
-
-        download.add_argument(
-            "-m",
-            "--model",
-            type=str,
-            required=False,
-            default=None,
-            help="Fetch model with given code. Use ls to list available models. Optional, if none supplied all models are downloaded.",
-        )
-
-        download.add_argument(
-            "-r",
-            "--repository",
-            type=str,
-            help="Repository to download model from",
-            choices=REPOSITORY.available(),
-            default="browsermt",
-        )
-
-    @staticmethod
-    def execute(args: argparse.Namespace):
-        if args.model is not None:
-            REPOSITORY.download(args.repository, args.model)
-        else:
-            for model in REPOSITORY.models(args.repository, filter_downloaded=False):
-                REPOSITORY.download(args.repository, model)
-
-
-@_register_cmd("ls")
-class List:
-    @staticmethod
-    def embed_subparser(key: str, subparsers: argparse._SubParsersAction):
-        ls = subparsers.add_parser(key, description="List available models.")
-        ls.add_argument(
-            "-r",
-            "--repository",
-            type=str,
-            help="Repository to list models from",
-            choices=REPOSITORY.available(),
-            default="browsermt",
-        )
-
-    @staticmethod
-    def execute(args: argparse.Namespace):
-        print("Available models: ")
-        for counter, identifier in enumerate(
-            REPOSITORY.models(args.repository, filter_downloaded=True), 1
-        ):
-            model = REPOSITORY.model(args.repository, identifier)
-            print(
-                " {}.".format(str(counter).rjust(4)),
-                model["code"],
-                model["name"],
-            )
-        print()
-
-
-def make_parser() -> argparse.ArgumentParser:
-    parser = argparse.ArgumentParser("bergamot")
-    subparsers = parser.add_subparsers(
-        title="actions",
-        description="The following actions are available through the bergamot package",
-        help="To obtain help on how to run these actions supply <cmd> -h.",
-        dest="action",
-    )
-
-    for key, cls in CMDS.items():
-        cls.embed_subparser(key, subparsers)
-    return parser
diff --git a/inference-engine/bindings/python/repository.py b/inference-engine/bindings/python/repository.py
deleted file mode 100644
index 9ea3ac023..000000000
--- a/inference-engine/bindings/python/repository.py
+++ /dev/null
@@ -1,218 +0,0 @@
-import json
-import os
-import tarfile
-import typing as t
-from abc import ABC, abstractmethod
-from functools import partial
-from urllib.parse import urlparse
-
-import requests
-from appdirs import AppDirs
-
-from .typing_utils import URL, PathLike
-from .utils import download_resource, patch_marian_for_bergamot
-
-APP = "bergamot"
-
-
-class Repository(ABC):
-    """
-    An interface for several repositories. Intended to enable interchangable
-    use of translateLocally and Mozilla repositories for usage through python.
-    """
-
-    @property
-    @abstractmethod
-    def name(self):
-        pass
-
-    @abstractmethod
-    def update(self):
-        """Updates the model list"""
-        pass
-
-    @abstractmethod
-    def models(self) -> t.List[str]:
-        """returns identifiers for available models"""
-        pass
-
-    @abstractmethod
-    def model(self, model_identifier: str) -> t.Any:
-        """returns entry for the  for available models"""
-        pass
-
-    @abstractmethod
-    def modelConfigPath(self, model_identifier: str) -> str:
-        """returns modelConfigPath for for a given model-identifier"""
-        pass
-
-    @abstractmethod
-    def download(self, model_identifier: str):
-        pass
-
-
-class TranslateLocallyLike(Repository):
-    """
-    This class implements Repository to fetch models from translateLocally.
-    AppDirs is used to standardize directories and further specialization
-    happens with translateLocally identifier.
-    """
-
-    def __init__(self, name, url):
-        self.url = url
-        self._name = name
-        appDir = AppDirs(APP)
-        f = lambda *args: os.path.join(*args, self._name)
-        self.dirs = {
-            "cache": f(appDir.user_cache_dir),
-            "config": f(appDir.user_config_dir),
-            "data": f(appDir.user_data_dir),
-            "archive": f(appDir.user_data_dir, "archives"),
-            "models": f(appDir.user_data_dir, "models"),
-        }
-
-        for directory in self.dirs.values():
-            os.makedirs(directory, exist_ok=True)
-
-        self.models_file_path = os.path.join(self.dirs["config"], "models.json")
-        self.data = self._load_data(self.models_file_path)
-
-        # Update inverse lookup.
-        self.data_by_code = {}
-        for model in self.data["models"]:
-            self.data_by_code[model["code"]] = model
-
-    @property
-    def name(self) -> str:
-        return self._name
-
-    def _load_data(self, models_file_path):
-        """
-        Load model data from existing file. If file does not exist, download from the web.
-        """
-        if os.path.exists(models_file_path):
-            # File already exists, prefer to work with this.
-            # A user is expected to update manually if model's already
-            # downloaded and setup.
-            with open(models_file_path) as model_file:
-                return json.load(model_file)
-        else:
-            # We are running for the first time.
-            # Try to fetch this file from the internet.
-            self.update()
-            with open(models_file_path) as model_file:
-                return json.load(model_file)
-
-    def update(self) -> None:
-        inventory = requests.get(self.url).text
-        with open(self.models_file_path, "w+") as models_file:
-            models_file.write(inventory)
-
-    def models(self, filter_downloaded: bool = True) -> t.List[str]:
-        codes = []
-        for model in self.data["models"]:
-            if filter_downloaded:
-                fprefix = self._archive_name_without_extension(model["url"])
-                model_dir = os.path.join(self.dirs["models"], fprefix)
-                if os.path.exists(model_dir):
-                    codes.append(model["code"])
-            else:
-                codes.append(model["code"])
-        return codes
-
-    def modelConfigPath(self, model_identifier: str) -> str:
-        model = self.model(model_identifier)
-        fprefix = self._archive_name_without_extension(model["url"])
-        model_dir = os.path.join(self.dirs["models"], fprefix)
-        return os.path.join(model_dir, "config.bergamot.yml")
-
-    def model(self, model_identifier: str) -> t.Any:
-        return self.data_by_code[model_identifier]
-
-    def download(self, model_identifier: str):
-        # Download path
-        model = self.model(model_identifier)
-        model_archive = "{}.tar.gz".format(model["shortName"])
-        save_location = os.path.join(self.dirs["archive"], model_archive)
-        download_resource(model["url"], save_location)
-
-        with tarfile.open(save_location) as model_archive:
-
-            def is_within_directory(directory, target):
-                abs_directory = os.path.abspath(directory)
-                abs_target = os.path.abspath(target)
-
-                prefix = os.path.commonprefix([abs_directory, abs_target])
-
-                return prefix == abs_directory
-
-            def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
-                for member in tar.getmembers():
-                    member_path = os.path.join(path, member.name)
-                    if not is_within_directory(path, member_path):
-                        raise Exception("Attempted Path Traversal in Tar File")
-
-                tar.extractall(path, members, numeric_owner=numeric_owner)
-
-            safe_extract(model_archive, self.dirs["models"])
-            fprefix = self._archive_name_without_extension(model["url"])
-            model_dir = os.path.join(self.dirs["models"], fprefix)
-            symlink = os.path.join(self.dirs["models"], model["code"])
-
-            print(
-                "Downloading and extracting {} into ... {}".format(
-                    model["code"], model_dir
-                ),
-                end=" ",
-            )
-
-            if not os.path.exists(symlink):
-                os.symlink(model_dir, symlink)
-
-            config_path = os.path.join(symlink, "config.intgemm8bitalpha.yml")
-            bergamot_config_path = os.path.join(symlink, "config.bergamot.yml")
-
-            # Finally patch so we don't have to reload this again.
-            patch_marian_for_bergamot(config_path, bergamot_config_path)
-
-            print("Done.")
-
-    def _archive_name_without_extension(self, url: URL):
-        o = urlparse(url)
-        fname = os.path.basename(o.path)  # something tar.gz.
-        fname_without_extension = ".".join(fname.split(".")[:3])
-        return fname_without_extension
-
-
-class Aggregator:
-    def __init__(self, repositories: t.List[Repository]):
-        self.repositories = {}
-        for repository in repositories:
-            if repository.name in self.repositories:
-                raise ValueError("Duplicate repository found.")
-            self.repositories[repository.name] = repository
-
-        # Default is self.repostiory
-        self.default_repository = repositories[0]
-
-    def update(self, name: str) -> None:
-        self.repositories.get(name, self.default_repository).update()
-
-    def modelConfigPath(self, name: str, code: str) -> PathLike:
-        return self.repositories.get(name, self.default_repository).modelConfigPath(
-            code
-        )
-
-    def models(self, name: str, filter_downloaded: bool = True) -> t.List[str]:
-        return self.repositories.get(name, self.default_repository).models()
-
-    def model(self, name: str, model_identifier: str) -> t.Any:
-        return self.repositories.get(name, self.default_repository).model(
-            model_identifier
-        )
-
-    def available(self):
-        return list(self.repositories.keys())
-
-    def download(self, name: str, model_identifier: str) -> None:
-        self.repositories.get(name, self.default_repository).download(model_identifier)
diff --git a/inference-engine/bindings/python/typing_utils.py b/inference-engine/bindings/python/typing_utils.py
deleted file mode 100644
index 3e1682cff..000000000
--- a/inference-engine/bindings/python/typing_utils.py
+++ /dev/null
@@ -1,5 +0,0 @@
-import pathlib
-import typing as t
-
-PathLike = t.TypeVar("PathLike", str, pathlib.Path)
-URL = str
diff --git a/inference-engine/bindings/python/utils.py b/inference-engine/bindings/python/utils.py
deleted file mode 100644
index 3164c171c..000000000
--- a/inference-engine/bindings/python/utils.py
+++ /dev/null
@@ -1,52 +0,0 @@
-import os
-
-import requests
-import yaml
-
-from .typing_utils import URL, PathLike
-
-
-def download_resource(url: URL, save_location: PathLike, force_download=False):
-    """
-    Downloads a resource from url into save_location, overwrites only if
-    force_download is true.
-    """
-    if force_download or not os.path.exists(save_location):
-        response = requests.get(url, stream=True)
-        # Throw an error for bad status codes
-        response.raise_for_status()
-        with open(save_location, "wb") as handle:
-            for block in response.iter_content(1024):
-                handle.write(block)
-
-
-def patch_marian_for_bergamot(
-    marian_config_path: PathLike, bergamot_config_path: PathLike, quality: bool = False
-):
-    """
-    Accepts path to a config-file from marian-training and followign
-    quantization and adjusts parameters for use in bergamot.
-    """
-    # Load marian_config_path
-    data = None
-    with open(marian_config_path) as fp:
-        data = yaml.load(fp, Loader=yaml.FullLoader)
-
-    # Update a few entries. Things here are hardcode.
-    data.update(
-        {
-            "ssplit-prefix-file": "",
-            "ssplit-mode": "paragraph",
-            "max-length-break": 128,
-            "mini-batch-words": 1024,
-            "workspace": 128,  # shipped models use big workspaces. We'd prefer to keep it low.
-            "alignment": "soft",
-        }
-    )
-
-    if quality:
-        data.update({"quality": quality, "skip-cost": False})
-
-    # Write-out.
-    with open(bergamot_config_path, "w") as output_file:
-        print(yaml.dump(data, sort_keys=False), file=output_file)
diff --git a/inference-engine/run-clang-format.py b/inference-engine/run-clang-format.py
deleted file mode 100644
index dcabaf1ec..000000000
--- a/inference-engine/run-clang-format.py
+++ /dev/null
@@ -1,408 +0,0 @@
-#!/usr/bin/env python
-"""A wrapper script around clang-format, suitable for linting multiple files
-and to use for continuous integration.
-
-This is an alternative API for the clang-format command line.
-It runs over multiple files and directories in parallel.
-A diff output is produced and a sensible exit code is returned.
-
-"""
-
-from __future__ import print_function, unicode_literals
-
-import argparse
-import codecs
-import difflib
-import fnmatch
-import io
-import errno
-import multiprocessing
-import os
-import signal
-import subprocess
-import sys
-import traceback
-
-from functools import partial
-
-try:
-    from subprocess import DEVNULL  # py3k
-except ImportError:
-    DEVNULL = open(os.devnull, "wb")
-
-
-DEFAULT_EXTENSIONS = 'c,h,C,H,cpp,hpp,cc,hh,c++,h++,cxx,hxx'
-DEFAULT_CLANG_FORMAT_IGNORE = '.clang-format-ignore'
-
-
-class ExitStatus:
-    SUCCESS = 0
-    DIFF = 1
-    TROUBLE = 2
-
-def excludes_from_file(ignore_file):
-    excludes = []
-    try:
-        with io.open(ignore_file, 'r', encoding='utf-8') as f:
-            for line in f:
-                if line.startswith('#'):
-                    # ignore comments
-                    continue
-                pattern = line.rstrip()
-                if not pattern:
-                    # allow empty lines
-                    continue
-                excludes.append(pattern)
-    except EnvironmentError as e:
-        if e.errno != errno.ENOENT:
-            raise
-    return excludes;
-
-def list_files(files, recursive=False, extensions=None, exclude=None):
-    if extensions is None:
-        extensions = []
-    if exclude is None:
-        exclude = []
-
-    out = []
-    for file in files:
-        if recursive and os.path.isdir(file):
-            for dirpath, dnames, fnames in os.walk(file):
-                fpaths = [os.path.join(dirpath, fname) for fname in fnames]
-                for pattern in exclude:
-                    # os.walk() supports trimming down the dnames list
-                    # by modifying it in-place,
-                    # to avoid unnecessary directory listings.
-                    dnames[:] = [
-                        x for x in dnames
-                        if
-                        not fnmatch.fnmatch(os.path.join(dirpath, x), pattern)
-                    ]
-                    fpaths = [
-                        x for x in fpaths if not fnmatch.fnmatch(x, pattern)
-                    ]
-                for f in fpaths:
-                    ext = os.path.splitext(f)[1][1:]
-                    if ext in extensions:
-                        out.append(f)
-        else:
-            out.append(file)
-    return out
-
-
-def make_diff(file, original, reformatted):
-    return list(
-        difflib.unified_diff(
-            original,
-            reformatted,
-            fromfile='{}\t(original)'.format(file),
-            tofile='{}\t(reformatted)'.format(file),
-            n=3))
-
-
-class DiffError(Exception):
-    def __init__(self, message, errs=None):
-        super(DiffError, self).__init__(message)
-        self.errs = errs or []
-
-
-class UnexpectedError(Exception):
-    def __init__(self, message, exc=None):
-        super(UnexpectedError, self).__init__(message)
-        self.formatted_traceback = traceback.format_exc()
-        self.exc = exc
-
-
-def run_clang_format_diff_wrapper(args, file):
-    try:
-        ret = run_clang_format_diff(args, file)
-        return ret
-    except DiffError:
-        raise
-    except Exception as e:
-        raise UnexpectedError('{}: {}: {}'.format(file, e.__class__.__name__,
-                                                  e), e)
-
-
-def run_clang_format_diff(args, file):
-    try:
-        with io.open(file, 'r', encoding='utf-8') as f:
-            original = f.readlines()
-    except IOError as exc:
-        raise DiffError(str(exc))
-    
-    if args.in_place:
-        invocation = [args.clang_format_executable, '-i', file]
-    else:
-        invocation = [args.clang_format_executable, file]
-
-    if args.style:
-        invocation.extend(['--style', args.style])
-
-    if args.dry_run:
-        print(" ".join(invocation))
-        return [], []
-
-    # Use of utf-8 to decode the process output.
-    #
-    # Hopefully, this is the correct thing to do.
-    #
-    # It's done due to the following assumptions (which may be incorrect):
-    # - clang-format will returns the bytes read from the files as-is,
-    #   without conversion, and it is already assumed that the files use utf-8.
-    # - if the diagnostics were internationalized, they would use utf-8:
-    #   > Adding Translations to Clang
-    #   >
-    #   > Not possible yet!
-    #   > Diagnostic strings should be written in UTF-8,
-    #   > the client can translate to the relevant code page if needed.
-    #   > Each translation completely replaces the format string
-    #   > for the diagnostic.
-    #   > -- http://clang.llvm.org/docs/InternalsManual.html#internals-diag-translation
-    #
-    # It's not pretty, due to Python 2 & 3 compatibility.
-    encoding_py3 = {}
-    if sys.version_info[0] >= 3:
-        encoding_py3['encoding'] = 'utf-8'
-
-    try:
-        proc = subprocess.Popen(
-            invocation,
-            stdout=subprocess.PIPE,
-            stderr=subprocess.PIPE,
-            universal_newlines=True,
-            **encoding_py3)
-    except OSError as exc:
-        raise DiffError(
-            "Command '{}' failed to start: {}".format(
-                subprocess.list2cmdline(invocation), exc
-            )
-        )
-    proc_stdout = proc.stdout
-    proc_stderr = proc.stderr
-    if sys.version_info[0] < 3:
-        # make the pipes compatible with Python 3,
-        # reading lines should output unicode
-        encoding = 'utf-8'
-        proc_stdout = codecs.getreader(encoding)(proc_stdout)
-        proc_stderr = codecs.getreader(encoding)(proc_stderr)
-    # hopefully the stderr pipe won't get full and block the process
-    outs = list(proc_stdout.readlines())
-    errs = list(proc_stderr.readlines())
-    proc.wait()
-    if proc.returncode:
-        raise DiffError(
-            "Command '{}' returned non-zero exit status {}".format(
-                subprocess.list2cmdline(invocation), proc.returncode
-            ),
-            errs,
-        )
-    if args.in_place:
-        return [], errs
-    return make_diff(file, original, outs), errs
-
-
-def bold_red(s):
-    return '\x1b[1m\x1b[31m' + s + '\x1b[0m'
-
-
-def colorize(diff_lines):
-    def bold(s):
-        return '\x1b[1m' + s + '\x1b[0m'
-
-    def cyan(s):
-        return '\x1b[36m' + s + '\x1b[0m'
-
-    def green(s):
-        return '\x1b[32m' + s + '\x1b[0m'
-
-    def red(s):
-        return '\x1b[31m' + s + '\x1b[0m'
-
-    for line in diff_lines:
-        if line[:4] in ['--- ', '+++ ']:
-            yield bold(line)
-        elif line.startswith('@@ '):
-            yield cyan(line)
-        elif line.startswith('+'):
-            yield green(line)
-        elif line.startswith('-'):
-            yield red(line)
-        else:
-            yield line
-
-
-def print_diff(diff_lines, use_color):
-    if use_color:
-        diff_lines = colorize(diff_lines)
-    if sys.version_info[0] < 3:
-        sys.stdout.writelines((l.encode('utf-8') for l in diff_lines))
-    else:
-        sys.stdout.writelines(diff_lines)
-
-
-def print_trouble(prog, message, use_colors):
-    error_text = 'error:'
-    if use_colors:
-        error_text = bold_red(error_text)
-    print("{}: {} {}".format(prog, error_text, message), file=sys.stderr)
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument(
-        '--clang-format-executable',
-        metavar='EXECUTABLE',
-        help='path to the clang-format executable',
-        default='clang-format')
-    parser.add_argument(
-        '--extensions',
-        help='comma separated list of file extensions (default: {})'.format(
-            DEFAULT_EXTENSIONS),
-        default=DEFAULT_EXTENSIONS)
-    parser.add_argument(
-        '-r',
-        '--recursive',
-        action='store_true',
-        help='run recursively over directories')
-    parser.add_argument(
-        '-d',
-        '--dry-run',
-        action='store_true',
-        help='just print the list of files')
-    parser.add_argument(
-        '-i',
-        '--in-place',
-        action='store_true',
-        help='format file instead of printing differences')
-    parser.add_argument('files', metavar='file', nargs='+')
-    parser.add_argument(
-        '-q',
-        '--quiet',
-        action='store_true',
-        help="disable output, useful for the exit code")
-    parser.add_argument(
-        '-j',
-        metavar='N',
-        type=int,
-        default=0,
-        help='run N clang-format jobs in parallel'
-        ' (default number of cpus + 1)')
-    parser.add_argument(
-        '--color',
-        default='auto',
-        choices=['auto', 'always', 'never'],
-        help='show colored diff (default: auto)')
-    parser.add_argument(
-        '-e',
-        '--exclude',
-        metavar='PATTERN',
-        action='append',
-        default=[],
-        help='exclude paths matching the given glob-like pattern(s)'
-        ' from recursive search')
-    parser.add_argument(
-        '--style',
-        help='formatting style to apply (LLVM, Google, Chromium, Mozilla, WebKit)')
-
-    args = parser.parse_args()
-
-    # use default signal handling, like diff return SIGINT value on ^C
-    # https://bugs.python.org/issue14229#msg156446
-    signal.signal(signal.SIGINT, signal.SIG_DFL)
-    try:
-        signal.SIGPIPE
-    except AttributeError:
-        # compatibility, SIGPIPE does not exist on Windows
-        pass
-    else:
-        signal.signal(signal.SIGPIPE, signal.SIG_DFL)
-
-    colored_stdout = False
-    colored_stderr = False
-    if args.color == 'always':
-        colored_stdout = True
-        colored_stderr = True
-    elif args.color == 'auto':
-        colored_stdout = sys.stdout.isatty()
-        colored_stderr = sys.stderr.isatty()
-
-    version_invocation = [args.clang_format_executable, str("--version")]
-    try:
-        subprocess.check_call(version_invocation, stdout=DEVNULL)
-    except subprocess.CalledProcessError as e:
-        print_trouble(parser.prog, str(e), use_colors=colored_stderr)
-        return ExitStatus.TROUBLE
-    except OSError as e:
-        print_trouble(
-            parser.prog,
-            "Command '{}' failed to start: {}".format(
-                subprocess.list2cmdline(version_invocation), e
-            ),
-            use_colors=colored_stderr,
-        )
-        return ExitStatus.TROUBLE
-
-    retcode = ExitStatus.SUCCESS
-
-    excludes = excludes_from_file(DEFAULT_CLANG_FORMAT_IGNORE)
-    excludes.extend(args.exclude)
-
-    files = list_files(
-        args.files,
-        recursive=args.recursive,
-        exclude=excludes,
-        extensions=args.extensions.split(','))
-
-    if not files:
-        return
-
-    njobs = args.j
-    if njobs == 0:
-        njobs = multiprocessing.cpu_count() + 1
-    njobs = min(len(files), njobs)
-
-    if njobs == 1:
-        # execute directly instead of in a pool,
-        # less overhead, simpler stacktraces
-        it = (run_clang_format_diff_wrapper(args, file) for file in files)
-        pool = None
-    else:
-        pool = multiprocessing.Pool(njobs)
-        it = pool.imap_unordered(
-            partial(run_clang_format_diff_wrapper, args), files)
-        pool.close()
-    while True:
-        try:
-            outs, errs = next(it)
-        except StopIteration:
-            break
-        except DiffError as e:
-            print_trouble(parser.prog, str(e), use_colors=colored_stderr)
-            retcode = ExitStatus.TROUBLE
-            sys.stderr.writelines(e.errs)
-        except UnexpectedError as e:
-            print_trouble(parser.prog, str(e), use_colors=colored_stderr)
-            sys.stderr.write(e.formatted_traceback)
-            retcode = ExitStatus.TROUBLE
-            # stop at the first unexpected error,
-            # something could be very wrong,
-            # don't process all files unnecessarily
-            if pool:
-                pool.terminate()
-            break
-        else:
-            sys.stderr.writelines(errs)
-            if outs == []:
-                continue
-            if not args.quiet:
-                print_diff(outs, use_color=colored_stdout)
-            if retcode == ExitStatus.SUCCESS:
-                retcode = ExitStatus.DIFF
-    if pool:
-        pool.join()
-    return retcode
-
-
-if __name__ == '__main__':
-    sys.exit(main())
diff --git a/inference-engine/setup.py b/inference-engine/setup.py
deleted file mode 100644
index ed4c6dc81..000000000
--- a/inference-engine/setup.py
+++ /dev/null
@@ -1,248 +0,0 @@
-import io
-import os
-import re
-import subprocess
-import sys
-
-from setuptools import Command, Extension, find_packages, setup
-from setuptools.command.build_ext import build_ext
-from setuptools.command.build_py import build_py as _build_py
-
-# Convert distutils Windows platform specifiers to CMake -A arguments
-PLAT_TO_CMAKE = {
-    "win32": "Win32",
-    "win-amd64": "x64",
-    "win-arm32": "ARM",
-    "win-arm64": "ARM64",
-}
-
-
-# A CMakeExtension needs a sourcedir instead of a file list.
-# The name must be the _single_ output extension from the CMake build.
-# If you need multiple extensions, see scikit-build.
-class CMakeExtension(Extension):
-    def __init__(self, name, sourcedir=""):
-        Extension.__init__(self, name, sources=[])
-        self.sourcedir = os.path.abspath(sourcedir)
-
-
-class CMakeBuild(build_ext):
-    def build_extension(self, ext):
-        extdir = os.path.abspath(os.path.dirname(self.get_ext_fullpath(ext.name)))
-
-        # required for auto-detection & inclusion of auxiliary "native" libs
-        if not extdir.endswith(os.path.sep):
-            extdir += os.path.sep
-
-        debug = int(os.environ.get("DEBUG", 0)) if self.debug is None else self.debug
-        cfg = "Debug" if debug else "Release"
-
-        # CMake lets you override the generator - we need to check this.
-        # Can be set with Conda-Build, for example.
-        cmake_generator = os.environ.get("CMAKE_GENERATOR", "")
-        build_arch = os.environ.get("BUILD_ARCH", "native")
-
-        # Set Python_EXECUTABLE instead if you use PYBIND11_FINDPYTHON
-        # EXAMPLE_VERSION_INFO shows you how to pass a value into the C++ code
-        # from Python.
-        cmake_args = [
-            f"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY={extdir}",
-            f"-DPYTHON_EXECUTABLE={sys.executable}",
-            f"-DCMAKE_BUILD_TYPE={cfg}",  # not used on MSVC, but no harm
-            f"-DCOMPILE_PYTHON=ON",
-            f"-DSSPLIT_USE_INTERNAL_PCRE2=ON",
-            f"-DBUILD_ARCH={build_arch}",
-        ]
-
-        build_args = ["-t", "_bergamot"]
-        # Adding CMake arguments set as environment variable
-        # (needed e.g. to build for ARM OSx on conda-forge)
-        if "CMAKE_ARGS" in os.environ:
-            cmake_args += [item for item in os.environ["CMAKE_ARGS"].split(" ") if item]
-
-        # In this example, we pass in the version to C++. You might not need to.
-        cmake_args += [f"-DEXAMPLE_VERSION_INFO={self.distribution.get_version()}"]
-
-        use_ccache = os.environ.get("USE_CCACHE", "0") == "1"
-        if use_ccache:
-            cmake_args += [
-                f"-DCMAKE_CXX_COMPILER_LAUNCHER=ccache",
-                f"-DCMAKE_C_COMPILER_LAUNCHER=ccache",
-            ]
-
-        if self.compiler.compiler_type != "msvc":
-            # Using Ninja-build since it a) is available as a wheel and b)
-            # multithreads automatically. MSVC would require all variables be
-            # exported for Ninja to pick it up, which is a little tricky to do.
-            # Users can override the generator with CMAKE_GENERATOR in CMake
-            # 3.15+.
-            if not cmake_generator:
-                try:
-                    import ninja  # noqa: F401
-
-                    cmake_args += ["-GNinja"]
-                except ImportError:
-                    pass
-
-        else:
-            # Single config generators are handled "normally"
-            single_config = any(x in cmake_generator for x in {"NMake", "Ninja"})
-
-            # CMake allows an arch-in-generator style for backward compatibility
-            contains_arch = any(x in cmake_generator for x in {"ARM", "Win64"})
-
-            # Specify the arch if using MSVC generator, but only if it doesn't
-            # contain a backward-compatibility arch spec already in the
-            # generator name.
-            if not single_config and not contains_arch:
-                cmake_args += ["-A", PLAT_TO_CMAKE[self.plat_name]]
-
-            # Multi-config generators have a different way to specify configs
-            if not single_config:
-                cmake_args += [
-                    f"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{cfg.upper()}={extdir}"
-                ]
-                build_args += ["--config", cfg]
-
-        if sys.platform.startswith("darwin"):
-            # Cross-compile support for macOS - respect ARCHFLAGS if set
-            archs = re.findall(r"-arch (\S+)", os.environ.get("ARCHFLAGS", ""))
-            if archs:
-                cmake_args += ["-DCMAKE_OSX_ARCHITECTURES={}".format(";".join(archs))]
-
-        # Set CMAKE_BUILD_PARALLEL_LEVEL to control the parallel build level
-        # across all generators.
-        if "CMAKE_BUILD_PARALLEL_LEVEL" not in os.environ:
-            # self.parallel is a Python 3 only way to set parallel jobs by hand
-            # using -j in the build_ext call, not supported by pip or PyPA-build.
-            if hasattr(self, "parallel") and self.parallel:
-                # CMake 3.12+ only.
-                build_args += [f"-j{self.parallel}"]
-
-        if not os.path.exists(self.build_temp):
-            os.makedirs(self.build_temp)
-
-        print("cmake", ext.sourcedir, " ".join(cmake_args))
-
-        subprocess.check_call(
-            ["cmake", ext.sourcedir] + cmake_args, cwd=self.build_temp
-        )
-        subprocess.check_call(
-            ["cmake", "--build", "."] + build_args, cwd=self.build_temp
-        )
-
-
-here = os.path.abspath(os.path.dirname(__file__))
-
-# Import the README and use it as the long-description.
-# Note: this will only work if 'README.md' is present in your MANIFEST.in file!
-long_description = ""
-with io.open(os.path.join(here, "bindings/python/README.md"), encoding="utf-8") as f:
-    long_description = "\n" + f.read()
-
-version = None
-with open(os.path.join(here, "BERGAMOT_VERSION")) as f:
-    version = f.read().strip()
-    suffix = os.environ.get("PYTHON_LOCAL_VERSION_IDENTIFIER", None)
-    if suffix:
-        version = "{}+{}".format(version, suffix)
-
-
-class UploadCommand(Command):
-    """Support setup.py upload."""
-
-    description = "Build and publish the package."
-    user_options = []
-
-    @staticmethod
-    def status(s):
-        """Prints things in bold."""
-        print("\033[1m{0}\033[0m".format(s))
-
-    def initialize_options(self):
-        pass
-
-    def finalize_options(self):
-        pass
-
-    def run(self):
-        try:
-            self.status("Removing previous builds…")
-            rmtree(os.path.join(here, "dist"))
-        except OSError:
-            pass
-
-        self.status("Building Source and Wheel (universal) distribution…")
-        os.system("{0} setup.py sdist bdist_wheel --universal".format(sys.executable))
-
-        self.status("Pushing git tags…")
-        os.system("git push --tags")
-
-        self.status("Uploading the package to PyPI via Twine…")
-        os.system("twine upload dist/*")
-
-        sys.exit()
-
-
-class build_py(_build_py):
-    def run(self):
-        self.run_command("build_ext")
-        return super().run()
-
-
-# The information here can also be placed in setup.cfg - better separation of
-# logic and declaration, and simpler if you include description/version in a file.
-setup(
-    name="bergamot",
-    version=version,
-    author="Jerin Philip",
-    author_email="jerinphilip@live.in",
-    url="https://github.com/browsermt/bergamot-translator/",
-    description="Translate text-content locally in your machine across langauges.",
-    long_description=long_description,
-    long_description_content_type="text/markdown",
-    ext_modules=[CMakeExtension("bergamot/_bergamot")],
-    cmdclass={"build_py": build_py, "build_ext": CMakeBuild},
-    zip_safe=False,
-    extras_require={"test": ["pytest>=6.0"]},
-    license_files=("LICENSE",),
-    python_requires=">=3.6",
-    packages=["bergamot"],
-    package_dir={"bergamot": "bindings/python"},
-    install_requires=["requests", "pyyaml>=5.1", "appdirs"],
-    entry_points={
-        "console_scripts": [
-            "bergamot = bergamot.__main__:main",
-        ],
-    },
-    # Classifiers help users find your project by categorizing it.
-    #
-    # For a list of valid classifiers, see https://pypi.org/classifiers/
-    classifiers=[  # Optional
-        # How mature is this project? Common values are
-        #   3 - Alpha
-        #   4 - Beta
-        #   5 - Production/Stable
-        "Development Status :: 3 - Alpha",
-        # Indicate who your project is intended for
-        "Intended Audience :: Developers",
-        "Topic :: Software Development :: Build Tools",
-        # Pick your license as you wish
-        "License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)",
-        # Specify the Python versions you support here. In particular, ensure
-        # that you indicate you support Python 3. These classifiers are *not*
-        # checked by 'pip install'. See instead 'python_requires' below.
-        "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.6",
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8",
-        "Programming Language :: Python :: 3.9",
-        "Programming Language :: Python :: 3.10",
-        "Programming Language :: Python :: 3 :: Only",
-    ],
-    project_urls={
-        "Bug Reports": "https://github.com/browsermt/bergamot-translator/issues",
-        "Source": "https://github.com/browsermt/bergamot-translator/",
-        "Documentation": "https://browser.mt/docs/main/python.html",
-    },
-)

From 1019fdd06df0ddca950feea8d1627aa8eb06bdf6 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Thu, 26 Sep 2024 13:02:24 -0500
Subject: [PATCH 428/442] Remove unneeded CLI code

---
 inference-engine/CMakeLists.txt     |  4 +--
 inference-engine/app/CMakeLists.txt |  2 --
 inference-engine/app/bergamot.cpp   | 41 -----------------------------
 3 files changed, 1 insertion(+), 46 deletions(-)
 delete mode 100644 inference-engine/app/CMakeLists.txt
 delete mode 100644 inference-engine/app/bergamot.cpp

diff --git a/inference-engine/CMakeLists.txt b/inference-engine/CMakeLists.txt
index da01c6048..febff3e6e 100644
--- a/inference-engine/CMakeLists.txt
+++ b/inference-engine/CMakeLists.txt
@@ -60,7 +60,7 @@ endif()
 if(MSVC)
   add_definitions(-DUSE_SSE2=1) # Supposed to fix something in the sse_mathfun.h but not sure it does
   set(INTRINSICS ${MSVC_BUILD_ARCH}) # ARCH we're targetting on win32. @TODO variable
-  
+
   set(CMAKE_CXX_FLAGS           "/EHsc /DWIN32 /D_WINDOWS /DUNICODE /D_UNICODE /D_CRT_NONSTDC_NO_WARNINGS /D_CRT_SECURE_NO_WARNINGS /bigobj")
   set(CMAKE_CXX_FLAGS_RELEASE   "${CMAKE_CXX_FLAGS} /MT /O2 ${INTRINSICS} /MP /GL /DNDEBUG")
   set(CMAKE_CXX_FLAGS_DEBUG     "${CMAKE_CXX_FLAGS} /MTd /Od /Ob0 ${INTRINSICS} /RTC1 /Zi /D_DEBUG")
@@ -179,8 +179,6 @@ add_subdirectory(src)
 
 if(COMPILE_WASM)
   add_subdirectory(wasm)
-else()
-  add_subdirectory(app)
 endif(COMPILE_WASM)
 
 option(COMPILE_PYTHON "Compile python bindings. Intended to be activated with setup.py" OFF)
diff --git a/inference-engine/app/CMakeLists.txt b/inference-engine/app/CMakeLists.txt
deleted file mode 100644
index b5c6a433b..000000000
--- a/inference-engine/app/CMakeLists.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-add_executable(bergamot bergamot.cpp)
-target_link_libraries(bergamot PRIVATE bergamot-translator)
diff --git a/inference-engine/app/bergamot.cpp b/inference-engine/app/bergamot.cpp
deleted file mode 100644
index 195e167b1..000000000
--- a/inference-engine/app/bergamot.cpp
+++ /dev/null
@@ -1,41 +0,0 @@
-#include "translator/byte_array_util.h"
-#include "translator/parser.h"
-#include "translator/response.h"
-#include "translator/response_options.h"
-#include "translator/service.h"
-#include "translator/utils.h"
-
-int main(int argc, char *argv[]) {
-  using namespace marian::bergamot;
-  ConfigParser<AsyncService> configParser("Bergamot CLI", /*multiOpMode=*/false);
-  configParser.parseArgs(argc, argv);
-  auto &config = configParser.getConfig();
-
-  AsyncService service(config.serviceConfig);
-
-  // Construct a model.
-  auto options = parseOptionsFromFilePath(config.modelConfigPaths.front());
-
-  std::shared_ptr<TranslationModel> model = service.createCompatibleModel(options);
-
-  ResponseOptions responseOptions;
-  std::string input = readFromStdin();
-
-  // Create a barrier using future/promise.
-  std::promise<Response> promise;
-  std::future<Response> future = promise.get_future();
-  auto callback = [&promise](Response &&response) {
-    // Fulfill promise.
-    promise.set_value(std::move(response));
-  };
-
-  service.translate(model, std::move(input), callback, responseOptions);
-
-  // Wait until promise sets the response.
-  Response response = future.get();
-
-  // Print (only) translated text.
-  std::cout << response.target.text;
-
-  return 0;
-}

From b6906d9df1b3a0d5439a85bd9fa5cacba54d3f71 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Thu, 26 Sep 2024 13:02:41 -0500
Subject: [PATCH 429/442] Remove unneded doc code

---
 inference-engine/doc/.gitignore             |   4 -
 inference-engine/doc/CI.md                  |  22 --
 inference-engine/doc/README.md              |  51 -----
 inference-engine/doc/Unified_API.md         | 212 --------------------
 inference-engine/doc/_static/css/custom.css |   4 -
 inference-engine/doc/conf.py                | 126 ------------
 inference-engine/doc/index.rst              |  40 ----
 inference-engine/doc/make.bat               |  35 ----
 inference-engine/doc/marian-integration.rst |  97 ---------
 inference-engine/doc/python.rst             |  87 --------
 inference-engine/doc/references.bib         |   0
 inference-engine/doc/requirements.txt       |   9 -
 inference-engine/doc/wasm-example.md        |   1 -
 13 files changed, 688 deletions(-)
 delete mode 100644 inference-engine/doc/.gitignore
 delete mode 100644 inference-engine/doc/CI.md
 delete mode 100644 inference-engine/doc/README.md
 delete mode 100644 inference-engine/doc/Unified_API.md
 delete mode 100644 inference-engine/doc/_static/css/custom.css
 delete mode 100644 inference-engine/doc/conf.py
 delete mode 100644 inference-engine/doc/index.rst
 delete mode 100644 inference-engine/doc/make.bat
 delete mode 100644 inference-engine/doc/marian-integration.rst
 delete mode 100644 inference-engine/doc/python.rst
 delete mode 100644 inference-engine/doc/references.bib
 delete mode 100644 inference-engine/doc/requirements.txt
 delete mode 120000 inference-engine/doc/wasm-example.md

diff --git a/inference-engine/doc/.gitignore b/inference-engine/doc/.gitignore
deleted file mode 100644
index 4d192b770..000000000
--- a/inference-engine/doc/.gitignore
+++ /dev/null
@@ -1,4 +0,0 @@
-api
-build
-doxygen
-venv
diff --git a/inference-engine/doc/CI.md b/inference-engine/doc/CI.md
deleted file mode 100644
index 2f29b02c1..000000000
--- a/inference-engine/doc/CI.md
+++ /dev/null
@@ -1,22 +0,0 @@
-# Continuous Integration
-
-[Circle CI](https://circleci.com/) is used for continuous integration. Configured via `./.circleci/config.yml`.
-
-##  Run Circle CI locally (requires Docker)
-
-1. [Install the CircleCI local cli](https://circleci.com/docs/2.0/local-cli/#installation)
-2. Validate Circle CI configuration (useful exercise before pushing any changes to the configuration)
-
-```shell
-circleci config validate -c .circleci/config.yml
-```
-
-3. To better mimic the starting point for CI, commit your changes and clone your repository into a clean directory then run CircleCI inside that directory:
-
-```shell
-git clone . /tmp/$(basename $PWD)
-cd /tmp/$(basename $PWD)
-circleci build
-```
-
-Note: Steps related to caching and uploading/storing artifacts will report as failed locally. This is not necessarily a problem, they are designed to fail since the operations are not supported locally by the CircleCI build agent.
diff --git a/inference-engine/doc/README.md b/inference-engine/doc/README.md
deleted file mode 100644
index 87d86ba1c..000000000
--- a/inference-engine/doc/README.md
+++ /dev/null
@@ -1,51 +0,0 @@
-# Marian NMT code documentation and library API
-
-This directory contains code documentation and library API for developers of Marian NMT.
-
-The documentation is generated using
-[Sphinx](https://www.sphinx-doc.org/en/master/usage/quickstart.html) +
-[Breathe](https://breathe.readthedocs.io/en/latest/directives.html) +
-[Doxygen](http://www.doxygen.nl/manual/docblocks.html) +
-[Exhale](https://exhale.readthedocs.io/en/latest/usage.html).
-The documentation source code is written in `.rst` or `.md` files with special directives that allow
-to reference to C++ source code and documentation. The source documents are then build into static
-HTML pages.
-
-
-## Installation
-
-On Ubuntu 20.04, install the following packages:
-
-    sudo apt-get install python3 python3-pip python3-setuptools doxygen
-
-Then set up a Python environment and install modules:
-
-    pip3 install virtualenv
-    virtualenv venv -p python3
-    source venv/bin/activate
-    pip install -r requirements.txt
-
-Documentation building should also work on Windows, but it has not been tested.
-
-
-## Generation
-
-The documentation can be generated by running:
-
-    make html
-
-The website will be generated into `build/html` and accessible by opening _index.html_ in your
-browser.
-
-Directories:
-
-- `build` - automatically output directory for HTML documentation
-- `doxygen` - automatically generated Doxygen XML files
-- `api` - automatic library API generated with Exhale
-- `.rst` and `.md` files in this directory and its subdirectories are documentation source files
-- `_static` - custom CSS and JavaScript files
-
-
-## Writing documentation
-
-To be documented...
diff --git a/inference-engine/doc/Unified_API.md b/inference-engine/doc/Unified_API.md
deleted file mode 100644
index e6a14301b..000000000
--- a/inference-engine/doc/Unified_API.md
+++ /dev/null
@@ -1,212 +0,0 @@
-# Unified (C++) API of Bergamot Translator
-
-/* A Translation model interface for translating a plain utf-8 encoded text (without any markups and emojis). The model supports translation from 1 source language to 1 target language. There can be different implementations of this interface. */
-
-class **AbstractTranslationModel** {
-
-    public:
-
-	AbstractTranslationModel();
-
-	virtual ~AbstractTranslationModel() {};
-
-	/* This method performs translation on a list of (utf-8) texts and returns a list of results in the same order. Each text entry can either be a word, a phrase, a sentence or a list of sentences and should contain plain text (without any markups or emojis). Additional information related to the translated text can be requested via TranslationRequest which is applied equally to each text entry. The translated text corresponding to each text entry and the additional information (as specified in the TranslationRequest) is encapsulated and returned in TranslationResult.
-	The API splits each text entry into sentences internally, which are then translated independent of each other. The translated sentences are then joined together and returned in TranslationResult.
-	Please refer to the TranslationRequest class to find out what additional information can be requested. The alignment information can only be requested if the model supports it (check isAlignmentSupported() API).
-	*/
-	virtual std::vector<std::future<TranslationResult>> translate(std::vector<std::string> texts, TranslationRequest request) = 0;
-
-	/* Check if the model can provide alignment information b/w original and translated text. */
-	virtual bool isAlignmentSupported() const = 0;
-}
-
-/* This class specifies the additional information related to the translated text (e.g. quality of the translation etc.) that can be requested to be included in the TranslationResult. These optional requests are set/unset independent of each other i.e. setting any one of them doesn’t have the side effect of setting any of the others. */
-
-class **TranslationRequest** {
-
-    private:
-
-	// Optional request. The granularity for which Quality scores of the translated text will be included in TranslationResult. By default (QualityScoreGranularity::NONE), scores are not included.
-	QualityScoreGranularity qualityScore = QualityScoreGranularity::NONE;
-
-	// Optional request. The type of the alignment b/w original and translated text that will be included in TranslationResult. By default (AlignmentType::NONE), alignment is not included.
-	AlignmentType alignmentType = AlignmentType::NONE;
-
-	// Optional request. A true/false value will include/exclude the original text in the TranslationResult. By default (false), the original text is not included.
-	bool includeOriginalText = false;
-
-	// Optional request. A true/false value will include/exclude the information regarding how individual sentences of original text map to corresponding translated sentences in joined translated text in the TranslationResult. By default (false), this information is not included.
-	bool includeSentenceMapping = false;
-
-    public:
-
-	explicit TranslationRequest();
-
-	~TranslationRequest();
-
-	/* Set the granularity for which the Quality scores of translated text should be included in the TranslationResult. By default (QualityScoreGranularity::NONE), scores are not included. */
-	void setQualityScoreGranularity(QualityScoreGranularity granularity);
-
-	/* Set the type of Alignment b/w original and translated text to be included in the TranslationResult. By default (AlignmentType::NONE), alignment is not included. */
-	void setAlignmentType(AlignmentType alignmentType);
-
-	/* Set to true/false to include/exclude the original text in the TranslationResult. By default (false), the original text is not included. */
-	void includeOriginalText(bool originalText);
-
-	/* Set to true/false to include/exclude the information regarding how individual sentences of original text map to corresponding translated sentences in joined translated text in the TranslationResult. By default (false), this information is not included. */
-	void includeSentenceMapping(bool sentenceMapping);
-
-	/* Return the granularity for which the Quality scores of the translated text will be included in TranslationResult. QualityScoreGranularity::NONE means the scores will not be included. */
-	QualityScoreGranularity getQualityScoreGranularity() const;
-
-	/* Return the type of Alignment b/w original and translated text that should be included in the TranslationResult. AlignmentType::NONE means the alignment will not be included. */
-	AlignmentType getAlignmentType() const;
-
-	/* Return whether the original text should be included in the TranslationResult. False means the original text will not be included. */
-	bool includeOriginalText() const;
-
-	/* Return whether the information regarding how individual sentences of original text map to corresponding translated sentences in joined translated text should be included in the TranslationResult. False means this information will not be included. */
-	bool includeSentenceMapping() const;
-}
-
-/* This class represents the result of translation on a TranslationRequest. */
-
-class **TranslationResult** {
-
-    private:
-
-	// Original text (utf-8) that was supposed to be translated; An optional result (it will be an empty string if not requested in TranslationRequest).
-	std::string originalText;
-
-	// Translation (in utf-8 format) of the originalText
-	std::string translatedText;
-
-	// Quality score of the translated text at the granularity specified in TranslationRequest; An optional result (it will have no information if not requested in TranslationRequest)
-	QualityScore qualityScore;
-
-	// Alignment information b/w original and translated text for AlignmentType specified in TranslationRequest; An optional result (it will have no information if not requested in TranslationRequest)
-	Alignment alignment;
-
-	// Information regarding how individual sentences of originalText map to corresponding translated sentences
-        // in joined translated text (translatedText); An optional result (it will be empty if not requested in TranslationRequest);
-        //       An example:
-        //       originalText (contains 2 sentences)              = "What is your name? My name is Abc."
-        //       translatedText (contains 2 translated sentences) = "Was ist dein Name? Mein Name ist Abc."
-        //       sentenceMappings = [
-        //                   {"What is your name?", "Was ist dein Name?"},  // A pair of Sentence 1 of originalText (originalText[0]) and corresponding translated sentence in translatedText (translatedText[0])
-        //                   {"My name is Abc", "Mein Name ist Abc."}       // A pair of Sentence 2 of originalText (originalText[1]) and corresponding translated sentence in translatedText (translatedText[1])
-        //                 ]
-        //
-	std::vector<std::pair<std::string_view, std::string_view>> sentenceMappings;
-
-    public:
-	// ToDo: Public Methods
-}
-
-/* This class encapsulates the configuration that is required by a translation model to perform translation. This configuration includes a path to the model file, source language vocabulary file, target language vocabulary file along with other options. */
-
-class **TranslationModelConfiguration** {
-
-    private:
-
-	// Path to the translation model file
-	const std::string modelPath;
-
-	// Path to the source vocabulary file to be used by the model
-	const std::string sourceLanguageVocabPath;
-
-	// Path to the target vocabulary file to be used by the model	
-	const std::string targetLanguageVocabPath;
-
-	// ToDo: Add all possible user configurable options (e.g. min batch size, max batch size) that are relevant for translation
-
-    public:
-
-	// Provide the path to the model file along with the source and target vocabulary files
-	TranslationModelConfiguration(const std::string& modelFilePath,
-					const std::string& sourceVocabPath,
-					const std::string& targetVocabPath);
-
-	// Return the path of the model file
-	const std::string& getModelFilePath() const;
-
-	// Return the path of the source language vocabulary file
-	const std::string& getSourceVocabularyPath() const;
-
-	// Return the path of the target language vocabulary file
-	const std::string& getSourceVocabularyPath() const;
-}
-
-// All possible granularities for which Quality Scores can be returned for translated (utf-8) text
-
-enum class QualityScoreGranularity {
-
-	WORD,
-	SENTENCE,
-	NONE,
-}
-
-// All possible supported alignment types between a text and its translation
-
-enum class AlignmentType {
-
-	SOFT,
-	NONE,
-}
-
-// This class represents the Quality Scores for various spans of the translated text at a specific granularity
-
-class QualityScore {
-
-      private:
-
-	// Sections of a text for the Quality Scores
-	std::vector<std::string_view> textViews;
-
-	// Quality Scores corresponding to each section of the text in textViews in the same order
-	std::vector<float> textScores;
-
-        // Granularity of the text for the Quality scores above
-	QualityScoreGranularity textGranularity;
-
-      public:
-	// ToDo: Public Methods
-}
-
-// This class encapsulates a translated text, all the sections of the original text that align to this translated text and the corresponding alignments for each of these sections of original text.
-
-class Alignment {
-
-      private:
-
-        // A list of sections of a translated text
-        // An example: originalText   = "What do you need"
-        //             translatedText = "Was brauchst du"
-        //             translatedTextViews = ["Was ", "brauchst", "du"]
-	std::vector<std::string_view> translatedTextViews;
-
-        // Each ith entry of this container corresponds to a list of all the sections of the original text that align to the ith entry of translatedTextView
-        // For the example above:
-        //             translatedTextViews = ["Was ", "brauchst", "du"]
-        //             originalTextViews   = [
-        //                                      ["What"],         // originalTextViews[0] = All sections of original text that align with translatedTextViews[0] i.e. "Was"
-        //                                      ["you", "need"],  // originalTextViews[1] = All sections of original text that align with translatedTextViews[1] i.e. "brauchst"
-        //                                      ["you"]           // originalTextViews[2] = All sections of original text that align with translatedTextViews[2] i.e. "du"
-        //                                   ]
-	std::vector<std::vector<std::string_view>> originalTextViews;
-
-        // Each ith entry of this container corresponds to the alignments of all the sections of the original text (ith entry of originalTextViews) that align to the ith entry of translatedTextViews
-        // For the example above:
-        //             alignments          = [
-        //                                      [0.90],         // alignments[0] = Alignments of all sections of original text (i.e. originalTextViews[0]) to translatedTextViews[0] i.e. "Was"
-        //                                      [0.3, 0.7],     // alignments[1] = Alignments of all sections of original text (i.e. originalTextViews[1]) to translatedTextViews[1] i.e. "brauchst"
-        //                                      [0.9]           // alignments[2] = Alignments of all sections of original text (i.e. originalTextViews[2]) to translatedTextViews[2] i.e. "du"
-        //                                   ]
-	std::vector<std::vector<float>> alignments;
-
-        // Type of the alignment b/w original and translated text above
-	AlignmentType alignmentType;
-
-      public:
-	// ToDo: Public Methods
-}
diff --git a/inference-engine/doc/_static/css/custom.css b/inference-engine/doc/_static/css/custom.css
deleted file mode 100644
index 8352655e1..000000000
--- a/inference-engine/doc/_static/css/custom.css
+++ /dev/null
@@ -1,4 +0,0 @@
-.wy-body-for-nav > .wy-grid-for-nav > .wy-nav-side {
-    border-bottom: 5px solid #28bbee;
-    /*background-color: #494d55;*/
-}
diff --git a/inference-engine/doc/conf.py b/inference-engine/doc/conf.py
deleted file mode 100644
index a86f4cbea..000000000
--- a/inference-engine/doc/conf.py
+++ /dev/null
@@ -1,126 +0,0 @@
-# Configuration file for the Sphinx documentation builder.
-#
-# This file only contains a selection of the most common options. For a full
-# list see the documentation:
-# https://www.sphinx-doc.org/en/master/usage/configuration.html
-
-# -- Path setup --------------------------------------------------------------
-
-import datetime
-
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-import os
-import sys
-
-sys.path.insert(0, os.path.abspath("."))
-
-
-# -- Project information -----------------------------------------------------
-
-project = "Bergamot Translator"
-copyright = "2021-2022 Bergamot Translator Team"
-author = "Bergamot Translator Team"
-
-# The full version, including alpha/beta/rc tags
-# TODO: add GitHub commit hash to the version
-version_file = os.path.join(
-    os.path.dirname(os.path.dirname(__file__)), "BERGAMOT_VERSION"
-)
-with open(os.path.abspath(version_file)) as f:
-    version = f.read().strip()
-release = version + " " + str(datetime.date.today())
-
-
-# -- General configuration ---------------------------------------------------
-
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-extensions = [
-    "sphinx.ext.mathjax",
-    "sphinx.ext.todo",
-    "breathe",
-    "exhale",
-    "recommonmark",
-    "sphinx.ext.autodoc",
-    "sphinxarg.ext",
-]
-
-# Add any paths that contain templates here, relative to this directory.
-templates_path = ["_templates"]
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-# This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = [
-    "build",
-    "doxygen",
-    "venv",
-    "README.md",
-]
-
-
-# -- Options for HTML output -------------------------------------------------
-
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
-#
-html_theme = "sphinx_rtd_theme"
-htmlhelp_basename = "bergamot-translator"
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ["_static"]
-html_css_files = ["css/custom.css"]
-
-# The base URL which points to the root of the HTML documentation
-html_baseurl = "https://browser.mt/docs"
-
-
-# -- Extension configuration -------------------------------------------------
-
-breathe_projects = {"bergamot-translator": "./doxygen/xml"}
-breathe_default_project = "bergamot-translator"
-
-doxygen_config = """
-INPUT                = ../src ../app
-EXCLUDE             += ../3rd_party
-EXCLUDE             += ../src/tests
-EXCLUDE_PATTERNS     = *.md *.txt
-FILE_PATTERNS       += *.cu
-EXTENSION_MAPPING   += cu=C++ inc=C++
-ENABLE_PREPROCESSING = YES
-JAVADOC_AUTOBRIEF    = YES
-WARN_IF_UNDOCUMENTED = NO
-"""
-
-exhale_args = {
-    "containmentFolder": "./api",
-    "rootFileName": "library_index.rst",
-    "rootFileTitle": "Library API",
-    "doxygenStripFromPath": "..",
-    "createTreeView": True,
-    "exhaleExecutesDoxygen": True,
-    "exhaleDoxygenStdin": doxygen_config.strip(),
-}
-
-primary_domain = "cpp"
-highlight_language = "cpp"
-
-# A trick to include markdown files from outside the source directory using
-# 'mdinclude'. Warning: all other markdown files not included via 'mdinclude'
-# will be rendered using recommonmark as recommended by Sphinx
-from m2r import MdInclude
-
-
-def setup(app):
-    # from m2r to make `mdinclude` work
-    app.add_config_value("no_underscore_emphasis", False, "env")
-    app.add_config_value("m2r_parse_relative_links", False, "env")
-    app.add_config_value("m2r_anonymous_references", False, "env")
-    app.add_config_value("m2r_disable_inline_math", False, "env")
-    app.add_directive("mdinclude", MdInclude)
diff --git a/inference-engine/doc/index.rst b/inference-engine/doc/index.rst
deleted file mode 100644
index 54dc1e8dc..000000000
--- a/inference-engine/doc/index.rst
+++ /dev/null
@@ -1,40 +0,0 @@
-Welcome to Bergamot Translator's documentation!
-===============================================
-
-|buildcpu| |tests| |release| |license|
-
-Bergamot translator provides a unified API for (Marian NMT framework based)
-neural machine translation functionality in accordance with the Bergamot
-project that focuses on improving client-side machine translation in a web
-browser.
-
-This is developer documentation. 
-
-.. toctree::
-   :maxdepth: 2
-   :caption: Contents:
-
-   marian-integration
-   wasm-example
-   api/library_index
-   python
-
-
-
-Indices and tables
-------------------
-
-* :ref:`genindex`
-
-
-.. |buildcpu| image:: https://img.shields.io/jenkins/s/http/vali.inf.ed.ac.uk/jenkins/view/browsermt/job/bergamot-translator.svg?label=CPU%20Build
-   :target: http://vali.inf.ed.ac.uk/jenkins/job/bergamot-translator
-   :alt: CPU build status
-
-.. |tests| image:: https://img.shields.io/jenkins/s/http/vali.inf.ed.ac.uk/jenkins/view/marian/job/bergamot-translator-regression-tests.svg?label=Tests
-   :target: http://vali.inf.ed.ac.uk/jenkins/job/bergamot-translator-regression-tests/
-   :alt: Tests status
-
-.. |license| image:: https://img.shields.io/badge/License-MPL%202.0-brightgreen.svg
-   :target: https://opensource.org/licenses/MPL-2.0
-   :alt: License: MPL
diff --git a/inference-engine/doc/make.bat b/inference-engine/doc/make.bat
deleted file mode 100644
index 6247f7e23..000000000
--- a/inference-engine/doc/make.bat
+++ /dev/null
@@ -1,35 +0,0 @@
-@ECHO OFF
-
-pushd %~dp0
-
-REM Command file for Sphinx documentation
-
-if "%SPHINXBUILD%" == "" (
-	set SPHINXBUILD=sphinx-build
-)
-set SOURCEDIR=source
-set BUILDDIR=build
-
-if "%1" == "" goto help
-
-%SPHINXBUILD% >NUL 2>NUL
-if errorlevel 9009 (
-	echo.
-	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
-	echo.installed, then set the SPHINXBUILD environment variable to point
-	echo.to the full path of the 'sphinx-build' executable. Alternatively you
-	echo.may add the Sphinx directory to PATH.
-	echo.
-	echo.If you don't have Sphinx installed, grab it from
-	echo.http://sphinx-doc.org/
-	exit /b 1
-)
-
-%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
-goto end
-
-:help
-%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
-
-:end
-popd
diff --git a/inference-engine/doc/marian-integration.rst b/inference-engine/doc/marian-integration.rst
deleted file mode 100644
index 756e0a810..000000000
--- a/inference-engine/doc/marian-integration.rst
+++ /dev/null
@@ -1,97 +0,0 @@
-Bergamot C++ Library
-====================
-
-This document contains instructions to develop for modifications on top
-of the marian machine translation toolkit powering bergamot-translator.
-The library is optimized towards fast and efficient translation of a
-given input.
-
-Build Instructions
-------------------
-
-Note: You are strongly advised to refer to the continuous integration on
-this repository, which builds bergamot-translator and associated
-applications from scratch. Examples to run these command
-line-applications are available in the
-`bergamot-translator-tests <https://github.com/browsermt/bergamot-translator-tests>`__
-repository. Builds take about 30 mins on a consumer grade machine, so
-using a tool like ccache is highly recommended.
-
-Dependencies
-~~~~~~~~~~~~
-
-Marian CPU version requires Intel MKL or OpenBLAS. Both are free, but
-MKL is not open-sourced. Intel MKL is strongly recommended as it is
-faster. On Ubuntu 16.04 and newer it can be installed from the APT
-repositories.
-
-.. code:: bash
-
-    wget -qO- 'https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB' | sudo apt-key add -
-    sudo sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'
-    sudo apt-get update
-    sudo apt-get install intel-mkl-64bit-2020.0-088
-
-On MacOS, apple accelerate framework will be used instead of
-MKL/OpenBLAS.
-
-Building bergamot-translator
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Web Assembly (WASM) reduces building to only using a subset of
-functionalities of marian, the translation library powering
-bergamot-translator. When developing bergamot-translator it is important
-that the sources added be compatible with marian. Therefore, it is
-required to set ``-DUSE_WASM_COMPATIBLE_SOURCE=on``.
-
-::
-
-    $ git clone https://github.com/browsermt/bergamot-translator
-    $ cd bergamot-translator
-    $ mkdir build
-    $ cd build
-    $ cmake .. -DUSE_WASM_COMPATIBLE_SOURCE=off -DCMAKE_BUILD_TYPE=Release
-    $ make -j2 
-
-The build will generate the library that can be linked to any project.
-All the public header files are specified in ``src`` folder.
-
-Command line apps
------------------
-
-bergamot-translator is intended to be used as a library. However, we
-provide a command-line application which is capable of translating text
-provided on standard-input. During development this application is used
-to perform regression-tests.
-
-
-Example command line run
-------------------------
-
-The models required to run the command-line are available at
-`data.statmt.org/bergamot/models/ <http://data.statmt.org/bergamot/models/>`__.
-
-The following example uses an English to German tiny11 student model,
-available at:
-
--  `data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz <http://data.statmt.org/bergamot/models/deen/ende.student.tiny11.tar.gz>`__
-
-.. literalinclude:: ../examples/run-native.sh
-   :language: bash
-
-Coding Style
-------------
-
-This repository contains C++ and JS source-files, of which C++ should
-adhere to the clang-format based style guidelines. You may configure
-your development environment to use the ``.clang-format`` and
-``.clang-format-ignore`` files provided in the root folder of this
-repository with your preferred choice of editor/tooling.
-
-One simple and recommended method to get your code to adhere to this
-style is to issue the following command in the source-root of this
-repository, which is used to also check for the coding style in the CI.
-
-.. code:: bash
-
-    python3 run-clang-format.py -i --style file -r src wasm
diff --git a/inference-engine/doc/python.rst b/inference-engine/doc/python.rst
deleted file mode 100644
index 0426f349f..000000000
--- a/inference-engine/doc/python.rst
+++ /dev/null
@@ -1,87 +0,0 @@
-.. Bergamot documentation master file, created by
-   sphinx-quickstart on Tue Jan 18 17:26:57 2022.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
-Python
-=======
-
-.. toctree::
-   :maxdepth: 3
-   :caption: Contents:
-
-
-This document describes python bindings from bergamot-translator and a
-batteries included python package supplied for easy use. The library also
-provides entry point via a command-line making it easier for the average user
-to get started.
-
-As bergamot-translator is built on top of marian, the python API should also
-work as python bindings for marian trained models, if they need to be
-integrated into python code-bases.
-
-*Disclaimer*: The package is still in early stages and unstable. Functions and
-classes might move around quite fast. Use at your own risk.
-
-Command Line Interface
-----------------------
-
-.. argparse::
-   :ref: bergamot.cmds.make_parser
-   :prog: bergamot
-
-
-Module Documentation
---------------------
-
-.. automodule:: bergamot
-   :members:
-   :undoc-members:
- 
-bergamot-translator
-+++++++++++++++++++
-
-The following components are exported from C++ via python-bindings and form
-library primitives that can be used to build translation workflows.
-
-.. autoclass:: bergamot.ServiceConfig
-   :members:
-   :undoc-members:
-
-.. autoclass:: bergamot.Service
-   :members:
-   :undoc-members:
-
-
-.. autoclass:: bergamot.TranslationModel
-   :members:
-   :undoc-members:
-
-.. autoclass:: bergamot.ResponseOptions
-   :members:
-   :undoc-members:
-
-Model Inventory
-+++++++++++++++
-
-.. autoclass:: bergamot.repository.Repository
-   :members:
-   :undoc-members:
-
-.. autoclass:: bergamot.repository.TranslateLocallyLike
-   :members:
-   :undoc-members:
-
-Utilities
-+++++++++
-
-.. autofunction:: bergamot.utils.patch_marian_for_bergamot
-
-
-
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
diff --git a/inference-engine/doc/references.bib b/inference-engine/doc/references.bib
deleted file mode 100644
index e69de29bb..000000000
diff --git a/inference-engine/doc/requirements.txt b/inference-engine/doc/requirements.txt
deleted file mode 100644
index 778f08914..000000000
--- a/inference-engine/doc/requirements.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-sphinx==2.4.4
-breathe==4.13.0
-Jinja2==3.0.3
-exhale
-sphinx_rtd_theme
-mistune<2.0.0
-recommonmark
-m2r
-sphinx-argparse
diff --git a/inference-engine/doc/wasm-example.md b/inference-engine/doc/wasm-example.md
deleted file mode 120000
index 9188e9356..000000000
--- a/inference-engine/doc/wasm-example.md
+++ /dev/null
@@ -1 +0,0 @@
-../wasm/README.md
\ No newline at end of file

From 00f4a30984c62fb58f06c9d1f672b91762c52a5f Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Thu, 26 Sep 2024 13:07:27 -0500
Subject: [PATCH 430/442] Add build-local script to inference-engine

---
 Taskfile.yml                              |  6 ++++
 inference-engine/.gitignore               |  1 +
 inference-engine/scripts/build-local.sh   | 35 +++++++++++++++++++++++
 inference-engine/scripts/detect-docker.sh | 19 ++++++++++++
 4 files changed, 61 insertions(+)
 create mode 100755 inference-engine/scripts/build-local.sh
 create mode 100755 inference-engine/scripts/detect-docker.sh

diff --git a/Taskfile.yml b/Taskfile.yml
index c767924e1..44da5e5c4 100644
--- a/Taskfile.yml
+++ b/Taskfile.yml
@@ -75,6 +75,12 @@ tasks:
     cmds:
       - poetry run opuscleaner-server serve --host=0.0.0.0 --port=8000
 
+  inference-engine-build:
+    desc: Build inference engine.
+    cmds:
+      - >-
+          task docker-run -- ./inference-engine/scripts/build-local.sh
+
   lint-black:
     desc: Checks the styling of the Python code with Black.
     deps: [poetry-install-black]
diff --git a/inference-engine/.gitignore b/inference-engine/.gitignore
index 94b32949c..78202d979 100644
--- a/inference-engine/.gitignore
+++ b/inference-engine/.gitignore
@@ -18,6 +18,7 @@ _deps
 
 wasm/test_page/node_modules
 /build
+/build-local
 /build-native
 /build-wasm
 /emsdk
diff --git a/inference-engine/scripts/build-local.sh b/inference-engine/scripts/build-local.sh
new file mode 100755
index 000000000..97595276f
--- /dev/null
+++ b/inference-engine/scripts/build-local.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+set -e
+
+# Run script from the context of inference-engine directory
+cd "$(dirname $0)/.."
+
+# Ensure script is running within docker
+./scripts/detect-docker.sh inference-engine-build
+
+# Return the number of available CPUs, or default to 1 if nproc is unavailable.
+detect_cpus() {
+  if command -v nproc >/dev/null 2>&1; then
+    nproc
+  else
+    echo 1
+  fi
+}
+
+if [ ! -d "build-local" ]; then
+  echo "Creating build-local directory..."
+  mkdir build-local
+else
+  echo "build-local directory already exists. Skipping creation."
+fi
+
+cd build-local || exit
+
+echo "Running cmake for build-local..."
+cmake ../
+
+# Run make using the detected number of CPUs
+CPUS=$(detect_cpus)
+echo "Running make for build-local with $CPUS CPUs..."
+make -j ${CPUS}
+
diff --git a/inference-engine/scripts/detect-docker.sh b/inference-engine/scripts/detect-docker.sh
new file mode 100755
index 000000000..c1065349a
--- /dev/null
+++ b/inference-engine/scripts/detect-docker.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+help_task=$1
+
+if [ -z "${IS_DOCKER}" ]; then
+  if [ "${ALLOW_RUN_ON_HOST}" != "1" ]; then
+    echo >&2
+    echo "Error: This script needs to be run inside Docker, or you must set ALLOW_RUN_ON_HOST=1." >&2
+    echo >&2
+    if [ -n "${help_task}" ]; then
+      echo " Help: To run this script directly in docker, run: task ${help_task}" >&2
+    fi
+    echo " Help: To enter docker, run: task docker" >&2
+    exit 1
+  else
+    echo >&2
+    echo "ALLOW_RUN_ON_HOST is set to 1. Continuing..." >&2
+  fi
+fi

From bb47eed7cfa516b90e6038f457b4b345a6f30a5f Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Thu, 26 Sep 2024 13:07:27 -0500
Subject: [PATCH 431/442] Add unit-tests script to inference-engine

---
 Taskfile.yml                            |  6 +++
 inference-engine/scripts/build-local.sh | 17 ++++++++-
 inference-engine/scripts/unit-tests.sh  | 49 +++++++++++++++++++++++++
 3 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100755 inference-engine/scripts/unit-tests.sh

diff --git a/Taskfile.yml b/Taskfile.yml
index 44da5e5c4..a9f52aaae 100644
--- a/Taskfile.yml
+++ b/Taskfile.yml
@@ -81,6 +81,12 @@ tasks:
       - >-
           task docker-run -- ./inference-engine/scripts/build-local.sh
 
+  inference-engine-test:
+    desc: Run inference-engine tests.
+    cmds:
+      - >-
+          task docker-run -- ./inference-engine/scripts/unit-tests.sh
+
   lint-black:
     desc: Checks the styling of the Python code with Black.
     deps: [poetry-install-black]
diff --git a/inference-engine/scripts/build-local.sh b/inference-engine/scripts/build-local.sh
index 97595276f..65e42e761 100755
--- a/inference-engine/scripts/build-local.sh
+++ b/inference-engine/scripts/build-local.sh
@@ -16,6 +16,16 @@ detect_cpus() {
   fi
 }
 
+# Parse command-line arguments for the --test flag
+COMPILE_TESTS=OFF
+while [[ "$#" -gt 0 ]]; do
+  case $1 in
+    "--test") COMPILE_TESTS=ON ;;
+    *) echo "Unknown parameter passed: $1"; exit 1 ;;
+  esac
+  shift
+done
+
 if [ ! -d "build-local" ]; then
   echo "Creating build-local directory..."
   mkdir build-local
@@ -25,8 +35,13 @@ fi
 
 cd build-local || exit
 
+# Run cmake with optional COMPILE_TESTS flag
 echo "Running cmake for build-local..."
-cmake ../
+if [ "$COMPILE_TESTS" = "ON" ]; then
+  cmake ../ -DCOMPILE_TESTS=ON
+else
+  cmake ../
+fi
 
 # Run make using the detected number of CPUs
 CPUS=$(detect_cpus)
diff --git a/inference-engine/scripts/unit-tests.sh b/inference-engine/scripts/unit-tests.sh
new file mode 100755
index 000000000..f4f12e3e1
--- /dev/null
+++ b/inference-engine/scripts/unit-tests.sh
@@ -0,0 +1,49 @@
+#!/bin/bash
+set -e
+
+# Run script from the context of inference-engine directory
+cd "$(dirname $0)/.."
+
+# Ensure script is running within docker
+./scripts/detect-docker.sh inference-engine-test
+
+# Check if build-local/src/tests/units directory exists
+if [ ! -d "build-local/src/tests/units" ]; then
+    echo "Directory build-local/src/tests/units does not exist. Running build."
+    ./scripts/build-local.sh --test
+else
+    echo "Directory build-local/src/tests/units already exists. Skipping build."
+fi
+
+# Change to the unit tests directory
+cd build-local/src/tests/units
+
+# List of test commands
+tests=(
+    "./run_annotation_tests"
+    "./run_cache_tests"
+    "./run_html_tests"
+    "./run_quality_estimator_tests"
+    "./run_xh_scanner_tests"
+)
+
+# Run all tests, collect failures
+failures=0
+
+for test in "${tests[@]}"; do
+    echo "Running $test..."
+    if ! $test; then
+        echo "$test failed!"
+        failures=$((failures + 1))
+    fi
+done
+
+# If any test failed, exit with a non-zero status
+if [ $failures -gt 0 ]; then
+    echo "$failures test(s) failed."
+    exit 1
+else
+    echo "All tests passed successfully."
+    exit 0
+fi
+

From cf23bf758582c4c8cd564fa5c21ac0772e95aec3 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Thu, 26 Sep 2024 13:07:27 -0500
Subject: [PATCH 432/442] Add clean script to inference-engine

---
 Taskfile.yml                      |  6 ++++++
 inference-engine/scripts/clean.sh | 29 +++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)
 create mode 100755 inference-engine/scripts/clean.sh

diff --git a/Taskfile.yml b/Taskfile.yml
index a9f52aaae..9fe48bdc8 100644
--- a/Taskfile.yml
+++ b/Taskfile.yml
@@ -75,6 +75,12 @@ tasks:
     cmds:
       - poetry run opuscleaner-server serve --host=0.0.0.0 --port=8000
 
+  inference-engine-clean:
+    desc: Clean build artifacts from the inference-engine directory.
+    cmds:
+      - >-
+          task docker-run -- ./inference-engine/scripts/clean.sh
+
   inference-engine-build:
     desc: Build inference engine.
     cmds:
diff --git a/inference-engine/scripts/clean.sh b/inference-engine/scripts/clean.sh
new file mode 100755
index 000000000..410291705
--- /dev/null
+++ b/inference-engine/scripts/clean.sh
@@ -0,0 +1,29 @@
+#!/bin/bash
+set -e
+
+# Run script from the context of inference-engine directory
+cd "$(dirname $0)/.."
+
+# Ensure script is running within docker
+./scripts/detect-docker.sh inference-engine-clean
+
+# List of directories to clean
+dirs=("build-local" "build-wasm" "emsdk")
+
+# Flag to track if any directories were cleaned
+cleaned=false
+
+# Check and remove directories
+for dir in "${dirs[@]}"; do
+    if [ -d "$dir" ]; then
+        echo "Removing $dir..."
+        rm -rf "$dir"
+        cleaned=true
+    fi
+done
+
+# If no directories were cleaned, print a message
+if [ "$cleaned" = false ]; then
+    echo "Nothing to clean"
+fi
+

From 07e3216cd48e079058cdc6343398d48839c235d5 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Thu, 26 Sep 2024 13:07:27 -0500
Subject: [PATCH 433/442] Move build-wasm script to inference-engine/scripts
 directory

---
 Taskfile.yml                                 |  6 ++++++
 inference-engine/{ => scripts}/build-wasm.sh | 10 +++++++---
 2 files changed, 13 insertions(+), 3 deletions(-)
 rename inference-engine/{ => scripts}/build-wasm.sh (90%)

diff --git a/Taskfile.yml b/Taskfile.yml
index 9fe48bdc8..8bf8dddf5 100644
--- a/Taskfile.yml
+++ b/Taskfile.yml
@@ -93,6 +93,12 @@ tasks:
       - >-
           task docker-run -- ./inference-engine/scripts/unit-tests.sh
 
+  inference-engine-build-wasm:
+    desc: Build inference engine WASM.
+    cmds:
+      - >-
+          task docker-run -- ./inference-engine/scripts/build-wasm.sh
+
   lint-black:
     desc: Checks the styling of the Python code with Black.
     deps: [poetry-install-black]
diff --git a/inference-engine/build-wasm.sh b/inference-engine/scripts/build-wasm.sh
similarity index 90%
rename from inference-engine/build-wasm.sh
rename to inference-engine/scripts/build-wasm.sh
index 443907232..4cabed2b5 100755
--- a/inference-engine/build-wasm.sh
+++ b/inference-engine/scripts/build-wasm.sh
@@ -1,9 +1,13 @@
 #!/usr/bin/env bash
 set -e
-set -x
 
-# Run script from the context of the script-containing directory
-cd "$(dirname $0)"
+# Run script from the context of inference-engine directory
+cd "$(dirname $0)/.."
+
+# Ensure script is running within docker
+./scripts/detect-docker.sh inference-engine-build-wasm
+
+set -x
 
 # Prerequisite: Download and Install Emscripten using following instructions (unless the EMSDK env var is already set)
 if [ "$EMSDK" == "" ]; then

From c62bea0890747c588d83b2cf3578a860719e3cc9 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Wed, 25 Sep 2024 16:02:11 -0500
Subject: [PATCH 434/442] Add review groups to CODEOWNERS

---
 .github/CODEOWNERS | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
index 26f6f4418..fa4f321cf 100644
--- a/.github/CODEOWNERS
+++ b/.github/CODEOWNERS
@@ -1,5 +1,31 @@
+# Firefox Translations review group
+.dockerignore @mozilla/firefox-translations
+.github @mozilla/firefox-translations
+.gitignore @mozilla/firefox-translations
+.gitmodules @mozilla/firefox-translations
+docker @mozilla/firefox-translations
+docs @mozilla/firefox-translations
+utils @mozilla/firefox-translations
+CODE_OF_CONDUCT.md @mozilla/firefox-translations
+LICENSE @mozilla/firefox-translations
+poetry.lock @mozilla/firefox-translations
+pyproject.toml @mozilla/firefox-translations
+README.md @mozilla/firefox-translations
+Taskfile.yml @mozilla/firefox-translations
+
+# Translations Training review group
+configs @mozilla/translations-training
+pipeline @mozilla/translations-training
+snakemake @mozilla/translations-training
+tests @mozilla/translations-training
+tracking @mozilla/translations-training
+
+# Translations Inference review group
+inference-engine @mozilla/translations-inference
+
 # Taskcluster pipeline related files. Changes to these ought to be reviewed by
 # RelEng to watch for security issues and best practices. These should also
 # be reviewed by people familiar with the pipeline itself.
-.taskcluster.yml @mozilla/releng
-taskcluster @mozilla/releng
+.taskcluster.yml @mozilla/releng @mozilla/translations-training
+taskcluster @mozilla/releng @mozilla/translations-training
+

From 72b6c9def8ce7ae4ec47e31289f5f41ad295807f Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Mon, 30 Sep 2024 15:32:13 -0500
Subject: [PATCH 435/442] Rename inference-engine to inference

---
 .gitmodules                                   |  14 ++++++------
 Taskfile.yml                                  |  20 +++++++++---------
 {inference-engine => inference}/.clang-format |   0
 .../.clang-format-ignore                      |   0
 {inference-engine => inference}/.clang-tidy   |   0
 {inference-engine => inference}/.gitignore    |   0
 .../3rd_party/CMakeLists.txt                  |   0
 .../3rd_party/browsermt-marian-dev            |   0
 .../3rd_party/ssplit-cpp                      |   0
 .../BERGAMOT_VERSION                          |   0
 .../CMakeLists.txt                            |   0
 {inference-engine => inference}/Doxyfile.in   |   0
 {inference-engine => inference}/LICENSE       |   0
 {inference-engine => inference}/MANIFEST.in   |   0
 {inference-engine => inference}/README.md     |   0
 .../cmake/GetVersionFromFile.cmake            |   0
 .../examples/run-native.sh                    |   0
 .../patches/01-marian-fstream-for-macos.patch |   0
 .../scripts/build-local.sh                    |   4 ++--
 .../scripts/build-wasm.sh                     |   4 ++--
 .../scripts/clean.sh                          |   4 ++--
 .../scripts/detect-docker.sh                  |   0
 .../scripts/unit-tests.sh                     |   4 ++--
 .../src/CMakeLists.txt                        |   0
 .../src/tests/CMakeLists.txt                  |   0
 .../src/tests/async.cpp                       |   0
 .../src/tests/blocking.cpp                    |   0
 .../src/tests/common-impl.cpp                 |   0
 .../src/tests/common.h                        |   0
 .../src/tests/intgemm-resolve.cpp             |   0
 .../src/tests/units/CMakeLists.txt            |   0
 .../src/tests/units/annotation_tests.cpp      |   0
 .../src/tests/units/cache_tests.cpp           |   0
 .../src/tests/units/html_tests.cpp            |   0
 .../src/tests/units/html_tests.h              |   0
 .../tests/units/quality_estimator_tests.cpp   |   0
 .../src/tests/units/quality_estimator_tests.h |   0
 .../src/tests/units/run_tests.cpp             |   0
 .../src/tests/units/xh_scanner_tests.cpp      |   0
 .../src/tests/wasm.cpp                        |   0
 .../src/translator/CMakeLists.txt             |   0
 .../translator/aggregate_batching_pool.cpp    |   0
 .../src/translator/aggregate_batching_pool.h  |   0
 .../src/translator/aligned.h                  |   0
 .../src/translator/annotation.cpp             |   0
 .../src/translator/annotation.h               |   0
 .../src/translator/batch.cpp                  |   0
 .../src/translator/batch.h                    |   0
 .../src/translator/batching_pool.cpp          |   0
 .../src/translator/batching_pool.h            |   0
 .../src/translator/byte_array_util.cpp        |   0
 .../src/translator/byte_array_util.h          |   0
 .../src/translator/cache.h                    |   0
 .../src/translator/definitions.h              |   0
 .../src/translator/html.cpp                   |   0
 .../src/translator/html.h                     |   0
 .../src/translator/logging.h                  |   0
 .../src/translator/parser.cpp                 |   0
 .../src/translator/parser.h                   |   0
 .../src/translator/project_version.h.in       |   0
 .../src/translator/quality_estimator.cpp      |   0
 .../src/translator/quality_estimator.h        |   0
 .../src/translator/request.cpp                |   0
 .../src/translator/request.h                  |   0
 .../src/translator/response.cpp               |   0
 .../src/translator/response.h                 |   0
 .../src/translator/response_builder.cpp       |   0
 .../src/translator/response_builder.h         |   0
 .../src/translator/response_options.h         |   0
 .../src/translator/service.cpp                |   0
 .../src/translator/service.h                  |   0
 .../src/translator/text_processor.cpp         |   0
 .../src/translator/text_processor.h           |   0
 .../translator/threadsafe_batching_pool.cpp   |   0
 .../src/translator/threadsafe_batching_pool.h |   0
 .../src/translator/translation_model.cpp      |   0
 .../src/translator/translation_model.h        |   0
 .../src/translator/utils.h                    |   0
 .../src/translator/vocabs.h                   |   0
 .../src/translator/xh_scanner.cpp             |   0
 .../src/translator/xh_scanner.h               |   0
 .../wasm/CMakeLists.txt                       |   0
 .../wasm/README.md                            |   0
 .../wasm/bindings/response_bindings.cpp       |   0
 .../bindings/response_options_bindings.cpp    |   0
 .../wasm/bindings/service_bindings.cpp        |   0
 .../wasm/import-gemm-module.js                |   0
 .../wasm/module/README.md                     |   0
 .../wasm/module/main.js                       |   0
 .../wasm/module/package.json                  |   0
 .../wasm/module/translator.js                 |   0
 .../wasm/module/worker/package.json           |   0
 .../wasm/module/worker/translator-worker.js   |   0
 .../wasm/node-test.js                         |   0
 .../patch-artifacts-import-gemm-module.sh     |   0
 .../wasm/project_version.js.in                |   0
 .../wasm/test_page/bergamot-httpserver.js     |   0
 .../wasm/test_page/css/index.css              |   0
 .../wasm/test_page/index.html                 |   0
 .../wasm/test_page/js/index.js                |   0
 .../wasm/test_page/logos.png                  | Bin
 .../wasm/test_page/package-lock.json          |   0
 .../wasm/test_page/package.json               |   0
 .../wasm/test_page/start_server.sh            |   0
 104 files changed, 24 insertions(+), 26 deletions(-)
 rename {inference-engine => inference}/.clang-format (100%)
 rename {inference-engine => inference}/.clang-format-ignore (100%)
 rename {inference-engine => inference}/.clang-tidy (100%)
 rename {inference-engine => inference}/.gitignore (100%)
 rename {inference-engine => inference}/3rd_party/CMakeLists.txt (100%)
 rename {inference-engine => inference}/3rd_party/browsermt-marian-dev (100%)
 rename {inference-engine => inference}/3rd_party/ssplit-cpp (100%)
 rename {inference-engine => inference}/BERGAMOT_VERSION (100%)
 rename {inference-engine => inference}/CMakeLists.txt (100%)
 rename {inference-engine => inference}/Doxyfile.in (100%)
 rename {inference-engine => inference}/LICENSE (100%)
 rename {inference-engine => inference}/MANIFEST.in (100%)
 rename {inference-engine => inference}/README.md (100%)
 rename {inference-engine => inference}/cmake/GetVersionFromFile.cmake (100%)
 rename {inference-engine => inference}/examples/run-native.sh (100%)
 rename {inference-engine => inference}/patches/01-marian-fstream-for-macos.patch (100%)
 rename {inference-engine => inference}/scripts/build-local.sh (89%)
 rename {inference-engine => inference}/scripts/build-wasm.sh (94%)
 rename {inference-engine => inference}/scripts/clean.sh (82%)
 rename {inference-engine => inference}/scripts/detect-docker.sh (100%)
 rename {inference-engine => inference}/scripts/unit-tests.sh (90%)
 rename {inference-engine => inference}/src/CMakeLists.txt (100%)
 rename {inference-engine => inference}/src/tests/CMakeLists.txt (100%)
 rename {inference-engine => inference}/src/tests/async.cpp (100%)
 rename {inference-engine => inference}/src/tests/blocking.cpp (100%)
 rename {inference-engine => inference}/src/tests/common-impl.cpp (100%)
 rename {inference-engine => inference}/src/tests/common.h (100%)
 rename {inference-engine => inference}/src/tests/intgemm-resolve.cpp (100%)
 rename {inference-engine => inference}/src/tests/units/CMakeLists.txt (100%)
 rename {inference-engine => inference}/src/tests/units/annotation_tests.cpp (100%)
 rename {inference-engine => inference}/src/tests/units/cache_tests.cpp (100%)
 rename {inference-engine => inference}/src/tests/units/html_tests.cpp (100%)
 rename {inference-engine => inference}/src/tests/units/html_tests.h (100%)
 rename {inference-engine => inference}/src/tests/units/quality_estimator_tests.cpp (100%)
 rename {inference-engine => inference}/src/tests/units/quality_estimator_tests.h (100%)
 rename {inference-engine => inference}/src/tests/units/run_tests.cpp (100%)
 rename {inference-engine => inference}/src/tests/units/xh_scanner_tests.cpp (100%)
 rename {inference-engine => inference}/src/tests/wasm.cpp (100%)
 rename {inference-engine => inference}/src/translator/CMakeLists.txt (100%)
 rename {inference-engine => inference}/src/translator/aggregate_batching_pool.cpp (100%)
 rename {inference-engine => inference}/src/translator/aggregate_batching_pool.h (100%)
 rename {inference-engine => inference}/src/translator/aligned.h (100%)
 rename {inference-engine => inference}/src/translator/annotation.cpp (100%)
 rename {inference-engine => inference}/src/translator/annotation.h (100%)
 rename {inference-engine => inference}/src/translator/batch.cpp (100%)
 rename {inference-engine => inference}/src/translator/batch.h (100%)
 rename {inference-engine => inference}/src/translator/batching_pool.cpp (100%)
 rename {inference-engine => inference}/src/translator/batching_pool.h (100%)
 rename {inference-engine => inference}/src/translator/byte_array_util.cpp (100%)
 rename {inference-engine => inference}/src/translator/byte_array_util.h (100%)
 rename {inference-engine => inference}/src/translator/cache.h (100%)
 rename {inference-engine => inference}/src/translator/definitions.h (100%)
 rename {inference-engine => inference}/src/translator/html.cpp (100%)
 rename {inference-engine => inference}/src/translator/html.h (100%)
 rename {inference-engine => inference}/src/translator/logging.h (100%)
 rename {inference-engine => inference}/src/translator/parser.cpp (100%)
 rename {inference-engine => inference}/src/translator/parser.h (100%)
 rename {inference-engine => inference}/src/translator/project_version.h.in (100%)
 rename {inference-engine => inference}/src/translator/quality_estimator.cpp (100%)
 rename {inference-engine => inference}/src/translator/quality_estimator.h (100%)
 rename {inference-engine => inference}/src/translator/request.cpp (100%)
 rename {inference-engine => inference}/src/translator/request.h (100%)
 rename {inference-engine => inference}/src/translator/response.cpp (100%)
 rename {inference-engine => inference}/src/translator/response.h (100%)
 rename {inference-engine => inference}/src/translator/response_builder.cpp (100%)
 rename {inference-engine => inference}/src/translator/response_builder.h (100%)
 rename {inference-engine => inference}/src/translator/response_options.h (100%)
 rename {inference-engine => inference}/src/translator/service.cpp (100%)
 rename {inference-engine => inference}/src/translator/service.h (100%)
 rename {inference-engine => inference}/src/translator/text_processor.cpp (100%)
 rename {inference-engine => inference}/src/translator/text_processor.h (100%)
 rename {inference-engine => inference}/src/translator/threadsafe_batching_pool.cpp (100%)
 rename {inference-engine => inference}/src/translator/threadsafe_batching_pool.h (100%)
 rename {inference-engine => inference}/src/translator/translation_model.cpp (100%)
 rename {inference-engine => inference}/src/translator/translation_model.h (100%)
 rename {inference-engine => inference}/src/translator/utils.h (100%)
 rename {inference-engine => inference}/src/translator/vocabs.h (100%)
 rename {inference-engine => inference}/src/translator/xh_scanner.cpp (100%)
 rename {inference-engine => inference}/src/translator/xh_scanner.h (100%)
 rename {inference-engine => inference}/wasm/CMakeLists.txt (100%)
 rename {inference-engine => inference}/wasm/README.md (100%)
 rename {inference-engine => inference}/wasm/bindings/response_bindings.cpp (100%)
 rename {inference-engine => inference}/wasm/bindings/response_options_bindings.cpp (100%)
 rename {inference-engine => inference}/wasm/bindings/service_bindings.cpp (100%)
 rename {inference-engine => inference}/wasm/import-gemm-module.js (100%)
 rename {inference-engine => inference}/wasm/module/README.md (100%)
 rename {inference-engine => inference}/wasm/module/main.js (100%)
 rename {inference-engine => inference}/wasm/module/package.json (100%)
 rename {inference-engine => inference}/wasm/module/translator.js (100%)
 rename {inference-engine => inference}/wasm/module/worker/package.json (100%)
 rename {inference-engine => inference}/wasm/module/worker/translator-worker.js (100%)
 rename {inference-engine => inference}/wasm/node-test.js (100%)
 rename {inference-engine => inference}/wasm/patch-artifacts-import-gemm-module.sh (100%)
 rename {inference-engine => inference}/wasm/project_version.js.in (100%)
 rename {inference-engine => inference}/wasm/test_page/bergamot-httpserver.js (100%)
 rename {inference-engine => inference}/wasm/test_page/css/index.css (100%)
 rename {inference-engine => inference}/wasm/test_page/index.html (100%)
 rename {inference-engine => inference}/wasm/test_page/js/index.js (100%)
 rename {inference-engine => inference}/wasm/test_page/logos.png (100%)
 rename {inference-engine => inference}/wasm/test_page/package-lock.json (100%)
 rename {inference-engine => inference}/wasm/test_page/package.json (100%)
 rename {inference-engine => inference}/wasm/test_page/start_server.sh (100%)

diff --git a/.gitmodules b/.gitmodules
index a07948957..ebb589038 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -6,14 +6,6 @@
 	path = 3rd_party/extract-lex
 	url = https://github.com/marian-nmt/extract-lex
 
-[submodule "inference-engine/3rd_party/browsermt-marian-dev"]
-	path = inference-engine/3rd_party/browsermt-marian-dev
-	url = https://github.com/browsermt/marian-dev
-
-[submodule "inference-engine/3rd_party/ssplit-cpp"]
-	path = inference-engine/3rd_party/ssplit-cpp
-	url = https://github.com/browsermt/ssplit-cpp
-
 [submodule "3rd_party/kenlm"]
 	path = 3rd_party/kenlm
 	url = https://github.com/kpu/kenlm
@@ -29,3 +21,9 @@
 [submodule "3rd_party/preprocess"]
 	path = 3rd_party/preprocess
 	url = https://github.com/kpu/preprocess.git
+[submodule "inference/3rd_party/browsermt-marian-dev"]
+	path = inference/3rd_party/browsermt-marian-dev
+	url = https://github.com/browsermt/marian-dev
+[submodule "inference/3rd_party/ssplit-cpp"]
+	path = inference/3rd_party/ssplit-cpp
+	url = https://github.com/browsermt/ssplit-cpp
diff --git a/Taskfile.yml b/Taskfile.yml
index 8bf8dddf5..745c5f05c 100644
--- a/Taskfile.yml
+++ b/Taskfile.yml
@@ -75,29 +75,29 @@ tasks:
     cmds:
       - poetry run opuscleaner-server serve --host=0.0.0.0 --port=8000
 
-  inference-engine-clean:
-    desc: Clean build artifacts from the inference-engine directory.
+  inference-clean:
+    desc: Clean build artifacts from the inference directory.
     cmds:
       - >-
-          task docker-run -- ./inference-engine/scripts/clean.sh
+          task docker-run -- ./inference/scripts/clean.sh
 
-  inference-engine-build:
+  inference-build:
     desc: Build inference engine.
     cmds:
       - >-
-          task docker-run -- ./inference-engine/scripts/build-local.sh
+          task docker-run -- ./inference/scripts/build-local.sh
 
-  inference-engine-test:
-    desc: Run inference-engine tests.
+  inference-test:
+    desc: Run inference tests.
     cmds:
       - >-
-          task docker-run -- ./inference-engine/scripts/unit-tests.sh
+          task docker-run -- ./inference/scripts/unit-tests.sh
 
-  inference-engine-build-wasm:
+  inference-build-wasm:
     desc: Build inference engine WASM.
     cmds:
       - >-
-          task docker-run -- ./inference-engine/scripts/build-wasm.sh
+          task docker-run -- ./inference/scripts/build-wasm.sh
 
   lint-black:
     desc: Checks the styling of the Python code with Black.
diff --git a/inference-engine/.clang-format b/inference/.clang-format
similarity index 100%
rename from inference-engine/.clang-format
rename to inference/.clang-format
diff --git a/inference-engine/.clang-format-ignore b/inference/.clang-format-ignore
similarity index 100%
rename from inference-engine/.clang-format-ignore
rename to inference/.clang-format-ignore
diff --git a/inference-engine/.clang-tidy b/inference/.clang-tidy
similarity index 100%
rename from inference-engine/.clang-tidy
rename to inference/.clang-tidy
diff --git a/inference-engine/.gitignore b/inference/.gitignore
similarity index 100%
rename from inference-engine/.gitignore
rename to inference/.gitignore
diff --git a/inference-engine/3rd_party/CMakeLists.txt b/inference/3rd_party/CMakeLists.txt
similarity index 100%
rename from inference-engine/3rd_party/CMakeLists.txt
rename to inference/3rd_party/CMakeLists.txt
diff --git a/inference-engine/3rd_party/browsermt-marian-dev b/inference/3rd_party/browsermt-marian-dev
similarity index 100%
rename from inference-engine/3rd_party/browsermt-marian-dev
rename to inference/3rd_party/browsermt-marian-dev
diff --git a/inference-engine/3rd_party/ssplit-cpp b/inference/3rd_party/ssplit-cpp
similarity index 100%
rename from inference-engine/3rd_party/ssplit-cpp
rename to inference/3rd_party/ssplit-cpp
diff --git a/inference-engine/BERGAMOT_VERSION b/inference/BERGAMOT_VERSION
similarity index 100%
rename from inference-engine/BERGAMOT_VERSION
rename to inference/BERGAMOT_VERSION
diff --git a/inference-engine/CMakeLists.txt b/inference/CMakeLists.txt
similarity index 100%
rename from inference-engine/CMakeLists.txt
rename to inference/CMakeLists.txt
diff --git a/inference-engine/Doxyfile.in b/inference/Doxyfile.in
similarity index 100%
rename from inference-engine/Doxyfile.in
rename to inference/Doxyfile.in
diff --git a/inference-engine/LICENSE b/inference/LICENSE
similarity index 100%
rename from inference-engine/LICENSE
rename to inference/LICENSE
diff --git a/inference-engine/MANIFEST.in b/inference/MANIFEST.in
similarity index 100%
rename from inference-engine/MANIFEST.in
rename to inference/MANIFEST.in
diff --git a/inference-engine/README.md b/inference/README.md
similarity index 100%
rename from inference-engine/README.md
rename to inference/README.md
diff --git a/inference-engine/cmake/GetVersionFromFile.cmake b/inference/cmake/GetVersionFromFile.cmake
similarity index 100%
rename from inference-engine/cmake/GetVersionFromFile.cmake
rename to inference/cmake/GetVersionFromFile.cmake
diff --git a/inference-engine/examples/run-native.sh b/inference/examples/run-native.sh
similarity index 100%
rename from inference-engine/examples/run-native.sh
rename to inference/examples/run-native.sh
diff --git a/inference-engine/patches/01-marian-fstream-for-macos.patch b/inference/patches/01-marian-fstream-for-macos.patch
similarity index 100%
rename from inference-engine/patches/01-marian-fstream-for-macos.patch
rename to inference/patches/01-marian-fstream-for-macos.patch
diff --git a/inference-engine/scripts/build-local.sh b/inference/scripts/build-local.sh
similarity index 89%
rename from inference-engine/scripts/build-local.sh
rename to inference/scripts/build-local.sh
index 65e42e761..ae64689fe 100755
--- a/inference-engine/scripts/build-local.sh
+++ b/inference/scripts/build-local.sh
@@ -1,11 +1,11 @@
 #!/bin/bash
 set -e
 
-# Run script from the context of inference-engine directory
+# Run script from the context of inference directory
 cd "$(dirname $0)/.."
 
 # Ensure script is running within docker
-./scripts/detect-docker.sh inference-engine-build
+./scripts/detect-docker.sh inference-build
 
 # Return the number of available CPUs, or default to 1 if nproc is unavailable.
 detect_cpus() {
diff --git a/inference-engine/scripts/build-wasm.sh b/inference/scripts/build-wasm.sh
similarity index 94%
rename from inference-engine/scripts/build-wasm.sh
rename to inference/scripts/build-wasm.sh
index 4cabed2b5..c21eea985 100755
--- a/inference-engine/scripts/build-wasm.sh
+++ b/inference/scripts/build-wasm.sh
@@ -1,11 +1,11 @@
 #!/usr/bin/env bash
 set -e
 
-# Run script from the context of inference-engine directory
+# Run script from the context of inference directory
 cd "$(dirname $0)/.."
 
 # Ensure script is running within docker
-./scripts/detect-docker.sh inference-engine-build-wasm
+./scripts/detect-docker.sh inference-build-wasm
 
 set -x
 
diff --git a/inference-engine/scripts/clean.sh b/inference/scripts/clean.sh
similarity index 82%
rename from inference-engine/scripts/clean.sh
rename to inference/scripts/clean.sh
index 410291705..73f5ae5eb 100755
--- a/inference-engine/scripts/clean.sh
+++ b/inference/scripts/clean.sh
@@ -1,11 +1,11 @@
 #!/bin/bash
 set -e
 
-# Run script from the context of inference-engine directory
+# Run script from the context of inference directory
 cd "$(dirname $0)/.."
 
 # Ensure script is running within docker
-./scripts/detect-docker.sh inference-engine-clean
+./scripts/detect-docker.sh inference-clean
 
 # List of directories to clean
 dirs=("build-local" "build-wasm" "emsdk")
diff --git a/inference-engine/scripts/detect-docker.sh b/inference/scripts/detect-docker.sh
similarity index 100%
rename from inference-engine/scripts/detect-docker.sh
rename to inference/scripts/detect-docker.sh
diff --git a/inference-engine/scripts/unit-tests.sh b/inference/scripts/unit-tests.sh
similarity index 90%
rename from inference-engine/scripts/unit-tests.sh
rename to inference/scripts/unit-tests.sh
index f4f12e3e1..dd8be9925 100755
--- a/inference-engine/scripts/unit-tests.sh
+++ b/inference/scripts/unit-tests.sh
@@ -1,11 +1,11 @@
 #!/bin/bash
 set -e
 
-# Run script from the context of inference-engine directory
+# Run script from the context of inference directory
 cd "$(dirname $0)/.."
 
 # Ensure script is running within docker
-./scripts/detect-docker.sh inference-engine-test
+./scripts/detect-docker.sh inference-test
 
 # Check if build-local/src/tests/units directory exists
 if [ ! -d "build-local/src/tests/units" ]; then
diff --git a/inference-engine/src/CMakeLists.txt b/inference/src/CMakeLists.txt
similarity index 100%
rename from inference-engine/src/CMakeLists.txt
rename to inference/src/CMakeLists.txt
diff --git a/inference-engine/src/tests/CMakeLists.txt b/inference/src/tests/CMakeLists.txt
similarity index 100%
rename from inference-engine/src/tests/CMakeLists.txt
rename to inference/src/tests/CMakeLists.txt
diff --git a/inference-engine/src/tests/async.cpp b/inference/src/tests/async.cpp
similarity index 100%
rename from inference-engine/src/tests/async.cpp
rename to inference/src/tests/async.cpp
diff --git a/inference-engine/src/tests/blocking.cpp b/inference/src/tests/blocking.cpp
similarity index 100%
rename from inference-engine/src/tests/blocking.cpp
rename to inference/src/tests/blocking.cpp
diff --git a/inference-engine/src/tests/common-impl.cpp b/inference/src/tests/common-impl.cpp
similarity index 100%
rename from inference-engine/src/tests/common-impl.cpp
rename to inference/src/tests/common-impl.cpp
diff --git a/inference-engine/src/tests/common.h b/inference/src/tests/common.h
similarity index 100%
rename from inference-engine/src/tests/common.h
rename to inference/src/tests/common.h
diff --git a/inference-engine/src/tests/intgemm-resolve.cpp b/inference/src/tests/intgemm-resolve.cpp
similarity index 100%
rename from inference-engine/src/tests/intgemm-resolve.cpp
rename to inference/src/tests/intgemm-resolve.cpp
diff --git a/inference-engine/src/tests/units/CMakeLists.txt b/inference/src/tests/units/CMakeLists.txt
similarity index 100%
rename from inference-engine/src/tests/units/CMakeLists.txt
rename to inference/src/tests/units/CMakeLists.txt
diff --git a/inference-engine/src/tests/units/annotation_tests.cpp b/inference/src/tests/units/annotation_tests.cpp
similarity index 100%
rename from inference-engine/src/tests/units/annotation_tests.cpp
rename to inference/src/tests/units/annotation_tests.cpp
diff --git a/inference-engine/src/tests/units/cache_tests.cpp b/inference/src/tests/units/cache_tests.cpp
similarity index 100%
rename from inference-engine/src/tests/units/cache_tests.cpp
rename to inference/src/tests/units/cache_tests.cpp
diff --git a/inference-engine/src/tests/units/html_tests.cpp b/inference/src/tests/units/html_tests.cpp
similarity index 100%
rename from inference-engine/src/tests/units/html_tests.cpp
rename to inference/src/tests/units/html_tests.cpp
diff --git a/inference-engine/src/tests/units/html_tests.h b/inference/src/tests/units/html_tests.h
similarity index 100%
rename from inference-engine/src/tests/units/html_tests.h
rename to inference/src/tests/units/html_tests.h
diff --git a/inference-engine/src/tests/units/quality_estimator_tests.cpp b/inference/src/tests/units/quality_estimator_tests.cpp
similarity index 100%
rename from inference-engine/src/tests/units/quality_estimator_tests.cpp
rename to inference/src/tests/units/quality_estimator_tests.cpp
diff --git a/inference-engine/src/tests/units/quality_estimator_tests.h b/inference/src/tests/units/quality_estimator_tests.h
similarity index 100%
rename from inference-engine/src/tests/units/quality_estimator_tests.h
rename to inference/src/tests/units/quality_estimator_tests.h
diff --git a/inference-engine/src/tests/units/run_tests.cpp b/inference/src/tests/units/run_tests.cpp
similarity index 100%
rename from inference-engine/src/tests/units/run_tests.cpp
rename to inference/src/tests/units/run_tests.cpp
diff --git a/inference-engine/src/tests/units/xh_scanner_tests.cpp b/inference/src/tests/units/xh_scanner_tests.cpp
similarity index 100%
rename from inference-engine/src/tests/units/xh_scanner_tests.cpp
rename to inference/src/tests/units/xh_scanner_tests.cpp
diff --git a/inference-engine/src/tests/wasm.cpp b/inference/src/tests/wasm.cpp
similarity index 100%
rename from inference-engine/src/tests/wasm.cpp
rename to inference/src/tests/wasm.cpp
diff --git a/inference-engine/src/translator/CMakeLists.txt b/inference/src/translator/CMakeLists.txt
similarity index 100%
rename from inference-engine/src/translator/CMakeLists.txt
rename to inference/src/translator/CMakeLists.txt
diff --git a/inference-engine/src/translator/aggregate_batching_pool.cpp b/inference/src/translator/aggregate_batching_pool.cpp
similarity index 100%
rename from inference-engine/src/translator/aggregate_batching_pool.cpp
rename to inference/src/translator/aggregate_batching_pool.cpp
diff --git a/inference-engine/src/translator/aggregate_batching_pool.h b/inference/src/translator/aggregate_batching_pool.h
similarity index 100%
rename from inference-engine/src/translator/aggregate_batching_pool.h
rename to inference/src/translator/aggregate_batching_pool.h
diff --git a/inference-engine/src/translator/aligned.h b/inference/src/translator/aligned.h
similarity index 100%
rename from inference-engine/src/translator/aligned.h
rename to inference/src/translator/aligned.h
diff --git a/inference-engine/src/translator/annotation.cpp b/inference/src/translator/annotation.cpp
similarity index 100%
rename from inference-engine/src/translator/annotation.cpp
rename to inference/src/translator/annotation.cpp
diff --git a/inference-engine/src/translator/annotation.h b/inference/src/translator/annotation.h
similarity index 100%
rename from inference-engine/src/translator/annotation.h
rename to inference/src/translator/annotation.h
diff --git a/inference-engine/src/translator/batch.cpp b/inference/src/translator/batch.cpp
similarity index 100%
rename from inference-engine/src/translator/batch.cpp
rename to inference/src/translator/batch.cpp
diff --git a/inference-engine/src/translator/batch.h b/inference/src/translator/batch.h
similarity index 100%
rename from inference-engine/src/translator/batch.h
rename to inference/src/translator/batch.h
diff --git a/inference-engine/src/translator/batching_pool.cpp b/inference/src/translator/batching_pool.cpp
similarity index 100%
rename from inference-engine/src/translator/batching_pool.cpp
rename to inference/src/translator/batching_pool.cpp
diff --git a/inference-engine/src/translator/batching_pool.h b/inference/src/translator/batching_pool.h
similarity index 100%
rename from inference-engine/src/translator/batching_pool.h
rename to inference/src/translator/batching_pool.h
diff --git a/inference-engine/src/translator/byte_array_util.cpp b/inference/src/translator/byte_array_util.cpp
similarity index 100%
rename from inference-engine/src/translator/byte_array_util.cpp
rename to inference/src/translator/byte_array_util.cpp
diff --git a/inference-engine/src/translator/byte_array_util.h b/inference/src/translator/byte_array_util.h
similarity index 100%
rename from inference-engine/src/translator/byte_array_util.h
rename to inference/src/translator/byte_array_util.h
diff --git a/inference-engine/src/translator/cache.h b/inference/src/translator/cache.h
similarity index 100%
rename from inference-engine/src/translator/cache.h
rename to inference/src/translator/cache.h
diff --git a/inference-engine/src/translator/definitions.h b/inference/src/translator/definitions.h
similarity index 100%
rename from inference-engine/src/translator/definitions.h
rename to inference/src/translator/definitions.h
diff --git a/inference-engine/src/translator/html.cpp b/inference/src/translator/html.cpp
similarity index 100%
rename from inference-engine/src/translator/html.cpp
rename to inference/src/translator/html.cpp
diff --git a/inference-engine/src/translator/html.h b/inference/src/translator/html.h
similarity index 100%
rename from inference-engine/src/translator/html.h
rename to inference/src/translator/html.h
diff --git a/inference-engine/src/translator/logging.h b/inference/src/translator/logging.h
similarity index 100%
rename from inference-engine/src/translator/logging.h
rename to inference/src/translator/logging.h
diff --git a/inference-engine/src/translator/parser.cpp b/inference/src/translator/parser.cpp
similarity index 100%
rename from inference-engine/src/translator/parser.cpp
rename to inference/src/translator/parser.cpp
diff --git a/inference-engine/src/translator/parser.h b/inference/src/translator/parser.h
similarity index 100%
rename from inference-engine/src/translator/parser.h
rename to inference/src/translator/parser.h
diff --git a/inference-engine/src/translator/project_version.h.in b/inference/src/translator/project_version.h.in
similarity index 100%
rename from inference-engine/src/translator/project_version.h.in
rename to inference/src/translator/project_version.h.in
diff --git a/inference-engine/src/translator/quality_estimator.cpp b/inference/src/translator/quality_estimator.cpp
similarity index 100%
rename from inference-engine/src/translator/quality_estimator.cpp
rename to inference/src/translator/quality_estimator.cpp
diff --git a/inference-engine/src/translator/quality_estimator.h b/inference/src/translator/quality_estimator.h
similarity index 100%
rename from inference-engine/src/translator/quality_estimator.h
rename to inference/src/translator/quality_estimator.h
diff --git a/inference-engine/src/translator/request.cpp b/inference/src/translator/request.cpp
similarity index 100%
rename from inference-engine/src/translator/request.cpp
rename to inference/src/translator/request.cpp
diff --git a/inference-engine/src/translator/request.h b/inference/src/translator/request.h
similarity index 100%
rename from inference-engine/src/translator/request.h
rename to inference/src/translator/request.h
diff --git a/inference-engine/src/translator/response.cpp b/inference/src/translator/response.cpp
similarity index 100%
rename from inference-engine/src/translator/response.cpp
rename to inference/src/translator/response.cpp
diff --git a/inference-engine/src/translator/response.h b/inference/src/translator/response.h
similarity index 100%
rename from inference-engine/src/translator/response.h
rename to inference/src/translator/response.h
diff --git a/inference-engine/src/translator/response_builder.cpp b/inference/src/translator/response_builder.cpp
similarity index 100%
rename from inference-engine/src/translator/response_builder.cpp
rename to inference/src/translator/response_builder.cpp
diff --git a/inference-engine/src/translator/response_builder.h b/inference/src/translator/response_builder.h
similarity index 100%
rename from inference-engine/src/translator/response_builder.h
rename to inference/src/translator/response_builder.h
diff --git a/inference-engine/src/translator/response_options.h b/inference/src/translator/response_options.h
similarity index 100%
rename from inference-engine/src/translator/response_options.h
rename to inference/src/translator/response_options.h
diff --git a/inference-engine/src/translator/service.cpp b/inference/src/translator/service.cpp
similarity index 100%
rename from inference-engine/src/translator/service.cpp
rename to inference/src/translator/service.cpp
diff --git a/inference-engine/src/translator/service.h b/inference/src/translator/service.h
similarity index 100%
rename from inference-engine/src/translator/service.h
rename to inference/src/translator/service.h
diff --git a/inference-engine/src/translator/text_processor.cpp b/inference/src/translator/text_processor.cpp
similarity index 100%
rename from inference-engine/src/translator/text_processor.cpp
rename to inference/src/translator/text_processor.cpp
diff --git a/inference-engine/src/translator/text_processor.h b/inference/src/translator/text_processor.h
similarity index 100%
rename from inference-engine/src/translator/text_processor.h
rename to inference/src/translator/text_processor.h
diff --git a/inference-engine/src/translator/threadsafe_batching_pool.cpp b/inference/src/translator/threadsafe_batching_pool.cpp
similarity index 100%
rename from inference-engine/src/translator/threadsafe_batching_pool.cpp
rename to inference/src/translator/threadsafe_batching_pool.cpp
diff --git a/inference-engine/src/translator/threadsafe_batching_pool.h b/inference/src/translator/threadsafe_batching_pool.h
similarity index 100%
rename from inference-engine/src/translator/threadsafe_batching_pool.h
rename to inference/src/translator/threadsafe_batching_pool.h
diff --git a/inference-engine/src/translator/translation_model.cpp b/inference/src/translator/translation_model.cpp
similarity index 100%
rename from inference-engine/src/translator/translation_model.cpp
rename to inference/src/translator/translation_model.cpp
diff --git a/inference-engine/src/translator/translation_model.h b/inference/src/translator/translation_model.h
similarity index 100%
rename from inference-engine/src/translator/translation_model.h
rename to inference/src/translator/translation_model.h
diff --git a/inference-engine/src/translator/utils.h b/inference/src/translator/utils.h
similarity index 100%
rename from inference-engine/src/translator/utils.h
rename to inference/src/translator/utils.h
diff --git a/inference-engine/src/translator/vocabs.h b/inference/src/translator/vocabs.h
similarity index 100%
rename from inference-engine/src/translator/vocabs.h
rename to inference/src/translator/vocabs.h
diff --git a/inference-engine/src/translator/xh_scanner.cpp b/inference/src/translator/xh_scanner.cpp
similarity index 100%
rename from inference-engine/src/translator/xh_scanner.cpp
rename to inference/src/translator/xh_scanner.cpp
diff --git a/inference-engine/src/translator/xh_scanner.h b/inference/src/translator/xh_scanner.h
similarity index 100%
rename from inference-engine/src/translator/xh_scanner.h
rename to inference/src/translator/xh_scanner.h
diff --git a/inference-engine/wasm/CMakeLists.txt b/inference/wasm/CMakeLists.txt
similarity index 100%
rename from inference-engine/wasm/CMakeLists.txt
rename to inference/wasm/CMakeLists.txt
diff --git a/inference-engine/wasm/README.md b/inference/wasm/README.md
similarity index 100%
rename from inference-engine/wasm/README.md
rename to inference/wasm/README.md
diff --git a/inference-engine/wasm/bindings/response_bindings.cpp b/inference/wasm/bindings/response_bindings.cpp
similarity index 100%
rename from inference-engine/wasm/bindings/response_bindings.cpp
rename to inference/wasm/bindings/response_bindings.cpp
diff --git a/inference-engine/wasm/bindings/response_options_bindings.cpp b/inference/wasm/bindings/response_options_bindings.cpp
similarity index 100%
rename from inference-engine/wasm/bindings/response_options_bindings.cpp
rename to inference/wasm/bindings/response_options_bindings.cpp
diff --git a/inference-engine/wasm/bindings/service_bindings.cpp b/inference/wasm/bindings/service_bindings.cpp
similarity index 100%
rename from inference-engine/wasm/bindings/service_bindings.cpp
rename to inference/wasm/bindings/service_bindings.cpp
diff --git a/inference-engine/wasm/import-gemm-module.js b/inference/wasm/import-gemm-module.js
similarity index 100%
rename from inference-engine/wasm/import-gemm-module.js
rename to inference/wasm/import-gemm-module.js
diff --git a/inference-engine/wasm/module/README.md b/inference/wasm/module/README.md
similarity index 100%
rename from inference-engine/wasm/module/README.md
rename to inference/wasm/module/README.md
diff --git a/inference-engine/wasm/module/main.js b/inference/wasm/module/main.js
similarity index 100%
rename from inference-engine/wasm/module/main.js
rename to inference/wasm/module/main.js
diff --git a/inference-engine/wasm/module/package.json b/inference/wasm/module/package.json
similarity index 100%
rename from inference-engine/wasm/module/package.json
rename to inference/wasm/module/package.json
diff --git a/inference-engine/wasm/module/translator.js b/inference/wasm/module/translator.js
similarity index 100%
rename from inference-engine/wasm/module/translator.js
rename to inference/wasm/module/translator.js
diff --git a/inference-engine/wasm/module/worker/package.json b/inference/wasm/module/worker/package.json
similarity index 100%
rename from inference-engine/wasm/module/worker/package.json
rename to inference/wasm/module/worker/package.json
diff --git a/inference-engine/wasm/module/worker/translator-worker.js b/inference/wasm/module/worker/translator-worker.js
similarity index 100%
rename from inference-engine/wasm/module/worker/translator-worker.js
rename to inference/wasm/module/worker/translator-worker.js
diff --git a/inference-engine/wasm/node-test.js b/inference/wasm/node-test.js
similarity index 100%
rename from inference-engine/wasm/node-test.js
rename to inference/wasm/node-test.js
diff --git a/inference-engine/wasm/patch-artifacts-import-gemm-module.sh b/inference/wasm/patch-artifacts-import-gemm-module.sh
similarity index 100%
rename from inference-engine/wasm/patch-artifacts-import-gemm-module.sh
rename to inference/wasm/patch-artifacts-import-gemm-module.sh
diff --git a/inference-engine/wasm/project_version.js.in b/inference/wasm/project_version.js.in
similarity index 100%
rename from inference-engine/wasm/project_version.js.in
rename to inference/wasm/project_version.js.in
diff --git a/inference-engine/wasm/test_page/bergamot-httpserver.js b/inference/wasm/test_page/bergamot-httpserver.js
similarity index 100%
rename from inference-engine/wasm/test_page/bergamot-httpserver.js
rename to inference/wasm/test_page/bergamot-httpserver.js
diff --git a/inference-engine/wasm/test_page/css/index.css b/inference/wasm/test_page/css/index.css
similarity index 100%
rename from inference-engine/wasm/test_page/css/index.css
rename to inference/wasm/test_page/css/index.css
diff --git a/inference-engine/wasm/test_page/index.html b/inference/wasm/test_page/index.html
similarity index 100%
rename from inference-engine/wasm/test_page/index.html
rename to inference/wasm/test_page/index.html
diff --git a/inference-engine/wasm/test_page/js/index.js b/inference/wasm/test_page/js/index.js
similarity index 100%
rename from inference-engine/wasm/test_page/js/index.js
rename to inference/wasm/test_page/js/index.js
diff --git a/inference-engine/wasm/test_page/logos.png b/inference/wasm/test_page/logos.png
similarity index 100%
rename from inference-engine/wasm/test_page/logos.png
rename to inference/wasm/test_page/logos.png
diff --git a/inference-engine/wasm/test_page/package-lock.json b/inference/wasm/test_page/package-lock.json
similarity index 100%
rename from inference-engine/wasm/test_page/package-lock.json
rename to inference/wasm/test_page/package-lock.json
diff --git a/inference-engine/wasm/test_page/package.json b/inference/wasm/test_page/package.json
similarity index 100%
rename from inference-engine/wasm/test_page/package.json
rename to inference/wasm/test_page/package.json
diff --git a/inference-engine/wasm/test_page/start_server.sh b/inference/wasm/test_page/start_server.sh
similarity index 100%
rename from inference-engine/wasm/test_page/start_server.sh
rename to inference/wasm/test_page/start_server.sh

From 8d2edd1f136605ee65cc2c517c35b7c2496529ed Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Mon, 30 Sep 2024 16:13:30 -0500
Subject: [PATCH 436/442] Reintroduce browsermt-marian-dev comment to
 .gitmodules file

---
 .gitmodules | 29 +++++++++++++++++++++--------
 1 file changed, 21 insertions(+), 8 deletions(-)

diff --git a/.gitmodules b/.gitmodules
index ebb589038..51663e9bf 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,29 +1,42 @@
 [submodule "fast_align"]
 	path = 3rd_party/fast_align
 	url = https://github.com/clab/fast_align
-
 [submodule "extract-lex"]
 	path = 3rd_party/extract-lex
 	url = https://github.com/marian-nmt/extract-lex
-
 [submodule "3rd_party/kenlm"]
 	path = 3rd_party/kenlm
 	url = https://github.com/kpu/kenlm
-
 [submodule "3rd_party/browsermt-marian-dev"]
 	path = 3rd_party/browsermt-marian-dev
 	url = https://github.com/browsermt/marian-dev
-
 [submodule "3rd_party/marian-dev"]
 	path = 3rd_party/marian-dev
 	url = https://github.com/marian-nmt/marian-dev
-
 [submodule "3rd_party/preprocess"]
 	path = 3rd_party/preprocess
 	url = https://github.com/kpu/preprocess.git
-[submodule "inference/3rd_party/browsermt-marian-dev"]
-	path = inference/3rd_party/browsermt-marian-dev
-	url = https://github.com/browsermt/marian-dev
 [submodule "inference/3rd_party/ssplit-cpp"]
 	path = inference/3rd_party/ssplit-cpp
 	url = https://github.com/browsermt/ssplit-cpp
+# This is the same dependency and repository as `3rd_party/browsermt-marian-dev` below.
+#
+# When forking `inference-engine` into to this project, I made an earnest attempt to utilize the preexisting
+# `3rd_party/browsermt-marian-dev` submodule within `inference-engine`. Unfortunately, I ran into several roadblocks:
+#
+#   1) I cannot directly add `3rd_party/browsermt-marian-dev` as a cmake subdirectory because cmake is aware that
+#      this path is not a subdirectory of the `inference-engine` project root.
+#
+#   2) Symbolic links do not appear to work for git submodule direcotires the way that they do for regular directories.
+#      Even if the symbolic link had linked correctly, it may have still failed due to the considerations of 1).
+#
+#   3) I tried using cmake to copy the files from `3rd_party/browsermt-marian-dev` into `inference-engine/3rd_party/browsermt-marian-dev`
+#      at build time, which would ensure that there is no duplicate reference to the URL in this file, however the upstream dependency itself
+#      has hard-coded expectations that the `.git` directory is only one level up, which appears to work correctly for the way git submodules are
+#      configured, but does not work if the files are copied over to a regular directory deeper in the repository's directory tree.
+#
+# It may be possible to remove `3rd_party/browsermt-marian-dev` to instead use `inference-engine/3rd-party/browsermt-marian-dev` everywhere
+# within this repository, but I will leave that for a future commit if there is a need to do so.
+[submodule "inference/3rd_party/browsermt-marian-dev"]
+	path = inference/3rd_party/browsermt-marian-dev
+	url = https://github.com/browsermt/marian-dev

From 01e3af527ef7b153ceb7b53fd04e220c6bbbd323 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Mon, 30 Sep 2024 16:15:18 -0500
Subject: [PATCH 437/442] Remove sub-directory README files

---
 inference/README.md             |  82 -----------
 inference/wasm/README.md        |  46 ------
 inference/wasm/module/README.md | 238 --------------------------------
 3 files changed, 366 deletions(-)
 delete mode 100644 inference/README.md
 delete mode 100644 inference/wasm/README.md
 delete mode 100644 inference/wasm/module/README.md

diff --git a/inference/README.md b/inference/README.md
deleted file mode 100644
index 05c3c3d25..000000000
--- a/inference/README.md
+++ /dev/null
@@ -1,82 +0,0 @@
-# Bergamot Translator
-
-[![CircleCI badge](https://img.shields.io/circleci/project/github/browsermt/bergamot-translator/main.svg?label=CircleCI)](https://circleci.com/gh/browsermt/bergamot-translator/)
-
-Bergamot translator provides a unified API for ([Marian NMT](https://marian-nmt.github.io/) framework based) neural machine translation functionality in accordance with the [Bergamot](https://browser.mt/) project that focuses on improving client-side machine translation in a web browser.
-
-## Build Instructions
-
-### Build Natively
-Create a folder where you want to build all the artifacts (`build-native` in this case) and compile
-
-```bash
-mkdir build-native
-cd build-native
-cmake ../
-make -j2
-```
-
-### Build WASM
-#### Prerequisite
-
-Building on wasm requires Emscripten toolchain. It can be downloaded and installed using following instructions:
-
-* Get the latest sdk: `git clone https://github.com/emscripten-core/emsdk.git`
-* Enter the cloned directory: `cd emsdk`
-* Install the sdk: `./emsdk install 3.1.8`
-* Activate the sdk: `./emsdk activate 3.1.8`
-* Activate path variables: `source ./emsdk_env.sh`
-
-#### <a name="Compile"></a> Compile
-
-To build a version that translates with higher speeds on Firefox Nightly browser, follow these instructions:
-
-   1. Create a folder where you want to build all the artifacts (`build-wasm` in this case) and compile
-       ```bash
-       mkdir build-wasm
-       cd build-wasm
-       emcmake cmake -DCOMPILE_WASM=on ../
-       emmake make -j2
-       ```
-
-       The wasm artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
-
-   2. Patch generated artifacts to import GEMM library from a separate wasm module
-       ```bash
-       bash ../wasm/patch-artifacts-import-gemm-module.sh
-       ```
-
-To build a version that runs on all browsers (including Firefox Nightly) but translates slowly, follow these instructions:
-
-  1. Create a folder where you want to build all the artifacts (`build-wasm` in this case) and compile
-      ```bash
-      mkdir build-wasm
-      cd build-wasm
-      emcmake cmake -DCOMPILE_WASM=on ../
-      emmake make -j2
-      ```
-
-  2. Patch generated artifacts to import GEMM library from a separate wasm module
-       ```bash
-       bash ../wasm/patch-artifacts-import-gemm-module.sh
-       ```
-
-#### Recompiling
-As long as you don't update any submodule, just follow [Compile](#Compile) steps.\
-If you update a submodule, execute following command in repository root folder before executing
-[Compile](#Compile) steps.
-```bash
-git submodule update --init --recursive
-```
-
-
-## How to use
-
-### Using Native version
-
-The builds generate library that can be integrated to any project. All the public header files are specified in `src` folder.\
-A short example of how to use the APIs is provided in `app/bergamot.cpp` file.
-
-### Using WASM version
-
-Please follow the `README` inside the `wasm` folder of this repository that demonstrates how to use the translator in JavaScript.
diff --git a/inference/wasm/README.md b/inference/wasm/README.md
deleted file mode 100644
index 0f3f77426..000000000
--- a/inference/wasm/README.md
+++ /dev/null
@@ -1,46 +0,0 @@
-# Using Bergamot Translator in JavaScript
-
-All the instructions below are meant to run from the current directory.
-
-## Using JS APIs
-
-See [node-test.js](./node-test.js) for an annotated example of how to use the WASM module. Most of the code from it can also be used in a browser context.
-
-Alternatively refer to the file `test_page/js/worker.js` that demonstrates how to use the bergamot translator in JavaScript via a `<script>` tag.
-
-## Demo
-
-* Download bergamot model files required for translation
-
-    Use following instructions to download [model files](https://github.com/mozilla/firefox-translations-models/) (make sure that `git-lfs` is installed and initialized before running these instructions):
-
-    ```bash
-    cd test_page
-    git clone --depth 1 --branch main --single-branch https://github.com/mozilla/firefox-translations-models/
-    mkdir models
-    cp -rf firefox-translations-models/registry.json models
-    cp -rf firefox-translations-models/models/prod/* models
-    cp -rf firefox-translations-models/models/dev/* models
-    gunzip models/*/*
-    ```
-
-* Start the test webserver (ensure you have the latest nodejs installed)
-    ```bash
-    cd test_page
-    bash start_server.sh ../../build-wasm
-    ```
-
-    Provide the folder containing the wasm artifacts as the first argument of `start_server.sh` script (`../../build-wasm` in this case).
-
-* Open any browser (tested with latest Chrome/Firefox/Safari)
-
-
-* Browse to the following page:
-    ```
-    http://localhost:80
-    ```
-
-* Perform translations:
-    * Choose the source and target languages using `From` and `To` dropdowns.
-    * Type a sentence to be translated in the `From` textbox.
-    * See the result in the `To` textbox.
diff --git a/inference/wasm/module/README.md b/inference/wasm/module/README.md
deleted file mode 100644
index 4f6b153ba..000000000
--- a/inference/wasm/module/README.md
+++ /dev/null
@@ -1,238 +0,0 @@
-# Installation
-
-```bash
-npm install @browsermt/bergamot-translator
-```
-
-# Quick start
-
-```js
-import {BatchTranslator} from "@browsermt/bergamot-translator/translator.js";
-
-const translator = new BatchTranslator();
-
-const response = await translator.translate({
-  from: "en",
-  to: "es",
-  text: "Hello <em>world</em>!",
-  html: true
-});
-
-console.log(response.target.text);
-
-// Stops worker threads
-translator.delete();
-```
-
-# Throughput vs Latency
-
-This package comes with two translator implementations:
-
-- [LatencyOptimisedTranslator](#latencyoptimisedtranslator) is more useful for an interactive session, say like Google Translate, where you're only working on translating one input at a time.
-- [BatchTranslator](#batchtranslator) is optimised for processing a large number of translations as fast as possible (but individual translations might take some time), e.g. translating a large number of strings or all paragraphs in a document.
-
-## LantencyOptimisedTranslator
-
-Translator best suited for interactive usage. Runs with a single worker thread and a batch-size of 1 to give you a response as quickly as possible. It will cancel any pending translations that aren't currently being processed if you submit a new one.
-
-```js
-const translator = new LatencyOptimisedTranslator({
-  pivotLanguage?: string?,
-  registryUrl?: string,
-  workerUrl?: string,
-  downloadTimeout?: number,
-  cacheSize?: number,
-  useNativeIntGemm?: boolean,
-})
-```
-
-- `pivotLanguage` - language code for the language to use as an intermediate if there is no direct translation model available. Defaults to `"en"`. Set to `null` to disable pivoting.
-- `registryUrl` - url to a list of models and their paths. Defaults to `https://storage.googleapis.com/bergamot-models-sandbox/0.3.3/registry.json`.
-- `workerUrl` - url to `translator-worker.js`. Defaults to `"worker/translator-worker.js"` relative to the path of `translator.js`.
-- `downloadTimeout` - Maximum time we're attempting to download model files before failing. Defaults to `60000` or 60 seconds. Set to `0` to disable.
-- `cacheSize` - Maximum number of sentences in kept translation cache (per worker, workers do not share their cache). This is an ideal maximum as it is a hash-map, in practice about 1/3th is occupied. If set to `0`, translation cache is disabled (the default).
-- `useNativeIntGemm` - Try to link to native IntGEMM implementation when loading the WASM binary. This is only implemented in the privileged extension context of Firefox Nightly. If it fails, it will always fall back to the included implementation. Defaults to `false`.
-
-### translate()
-
-```js
-const {request, target: {text:string}} = await translator.translate({
-  from: string,
-  to: string,
-  text: string,
-  html?: boolean,
-  qualityScores?: boolean
-})
-```
-
-Submits a translation request. Multiple of these are processed in a batch. A batch will be started the next tick (if there is a worker available).
-
-- `from` - language code of the source language, e.g. `"de"`
-- `to` - language code of the target language, e.g. `"en"`
-- `text` - string of text to translate, e.g. `"Hallo Welt!"`
-- `html` - boolean indicating whether `text` contains just plain text or HTML
-- `qualityScores` - whether to calculate quality scores. Not all models support this, and you need to load a separate quality scores model file for it. Quality scores are returned as `<font x-bergamot-sentence-quality="">` and `<font x-bergamot-word-quality="">` wrapped around sentences and words in the output. When enabled, the output is always HTML, regardless of whether the input was.
-
-Returns:
-
-A promise to a translation response object, with `target.text` being the text or HTML of the translated output, and `request` a reference to the original translation request.
-
-### delete()
-
-```js
-translator.delete()
-```
-
-Cancels all pending requests with a `CancelledError` and terminates the worker immediately. This will free all the resources used.
-
-In a nodejs context you'll need to call this, otherwise your script won't exit because the translator will still be listening for messages from the worker.
-
-## BatchTranslator
-
-```js
-const translator = new BatchTranslator({
-  pivotLanguage?: string?,
-  registryUrl?: string,
-  workerUrl?: string,
-  downloadTimeout?: number,
-  cacheSize?: number,
-  useNativeIntGemm?: boolean,
-  workers?: number,
-  batchSize?: number,
-})
-```
-
-General translator options:
-
-See [LatencyOptimisedTranslator](#latencyoptimisedtranslator).
-
-BatchTranslator-specific options:
-
-- `workers` - Number of worker threads. These are full-on instances of the translator, with their own copy of the model loaded. This is an upper bound. If not that many workers can be fed, it won't create new ones. Minimally 1. Default is `1`.
-- `batchSize` - Number of translation requests per batch. All sentences from all translation requests are packed into a bunch of matrix operations. With a larger batch size the translator has more material to find ideal sets of sentences for filling the matrix. However, you'll only get the results for each of the requests in a batch once the whole batch is finished. Defaults to 8.
-
-### translate()
-
-```js
-const {target: {text:string}} = await translator.translate({
-  from: string,
-  to: string,
-  text: string,
-  html?: boolean,
-  qualityScores?: boolean,
-  priority?: number
-})
-```
-
-Submits a translation request. Multiple of these are processed in a batch. A batch will be started the next tick (if there is a worker available).
-
-- (See [LatencyOptimisedTranslator.translate()](#translate) for most options)
-- `priority` - When grouping translation requests into batches to give to workers, requests with a lower number are considered first. For example, if you're translating a web page, you can give requests of parts that are in the current frame a lower number to make sure they're processed first.
-
-### remove()
-
-```js
-translator.remove(request => {
-  // true deletes the request from the queue.
-  return true;
-})
-```
-
-Removes requests from the translation queue, i.e. only when they haven't been sent to a worker yet.
-
-The filter function should return true-ish for each request that should be cancelled. Their promises are rejected with a `CancelledError` error.
-
-
-### delete()
-
-```js
-translator.delete()
-```
-
-Cancels all pending requests with a `CancelledError` and terminates all workers immediately. This will free all the resources used.
-
-
-# Models
-
-Both translators accept a `backing` option, which tells it where to get model data and the translation engine implementation from. They default to using `BergamotTranslator` which gets its models from the same repository as [firefox-translations](https://github.com/mozilla/firefox-translations).
-
-To customize the model, reimplement the `loadModelRegistry` and `loadTranslationModel` methods.
-
-`loadModelRegistry()` has the hard requirement to return a promise to a list that looks like `{from: string, to: string, ...}[]`. The `from` and `to` keys are used as key for model selection.
-
-`loadTranslationModel()` should return a promise with ArrayBuffers for `model`, `shortlist`, `vocabs`, and optionally `qualityModel`. It can include a `config` object as well.
-
-Example of an alternative implementation that loads models from data.statmt.org, i.e. the same as [translateLocally](https://translateLocally.com):
-
-```js
-class CustomBacking extends TranslatorBacking {
-    async loadModelRegistery() {
-        const response = await fetch('https://translatelocally.com/models.json');
-        const {models} = await response.json();
-
-        // Add 'from' and 'to' keys for each model. Since theoretically a model
-        // can have multiple froms keys in TranslateLocally, we do a little
-        // product here.
-        return models.reduce((list, model) => {
-            try {
-                const to = first(Intl.getCanonicalLocales(model.trgTag));
-                for (let from of Intl.getCanonicalLocales(Object.keys(model.srcTags))) {
-                    list.push({from, to, model});
-                }
-            } catch (err) {
-                console.log('Skipped model', model, 'because', err);
-            }
-
-            return list;
-        }, []);
-    }
-
-    async loadTranslationModel({from, to}) {
-        // Find that model in the registry which will tell us about its files
-        const entries = (await this.registry).filter(model => model.from === from && model.to === to);
-
-        // Prefer tiny models above non-tiny ones
-        entries.sort(({model: a}, {model: b}) => (a.shortName.indexOf('tiny') === -1 ? 1 : 0) - (b.shortName.indexOf('tiny') === -1 ? 1 : 0));
-
-        if (!entries)
-            throw new Error(`No model for '${from}' -> '${to}'`);
-
-        const entry = entries[0].model;
-
-        const response = await fetch(entry.url, {
-            integrity: `sha256-${entry.checksum}`
-        });
-
-        // pako from https://www.npmjs.com/package/pako
-        const archive = pako.inflate(await response.arrayBuffer());
-
-        // untar from https://www.npmjs.com/package/js-untar
-        const files = await untar(archive.buffer);
-
-        const find = (filename) => {
-            const found = files.find(file => file.name.match(/(?:^|\/)([^\/]+)$/)[1] === filename)
-            if (found === undefined)
-                throw new Error(`Could not find '${filename}' in model archive`);
-            return found;
-        };
-
-        // YAML.parse is found in worker/translator-worker.js
-        const config = YAML.parse(find('config.intgemm8bitalpha.yml').readAsString());
-
-        const model = find(config.models[0]).buffer;
-
-        const vocabs = config.vocabs.map(vocab => find(vocab).buffer);
-
-        const shortlist = find(config.shortlist[0]).buffer;
-
-        // Return the buffers
-        return {model, vocabs, shortlist, config};
-    }
-}
-
-const translator = new BatchTranslator(options, new CustomBacking(options));
-```
-
-# Supported languages
-
-See https://github.com/mozilla/firefox-translations-models#currently-supported-languages. You may need to set the `registryUrl` option to point to the latest release.
\ No newline at end of file

From baf2d556a3c856b04c242e946492e21661438c94 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Mon, 30 Sep 2024 16:20:38 -0500
Subject: [PATCH 438/442] Move hidden clang files to the repository root

---
 inference/.clang-format => .clang-format               | 0
 inference/.clang-format-ignore => .clang-format-ignore | 0
 inference/.clang-tidy => .clang-tidy                   | 0
 3 files changed, 0 insertions(+), 0 deletions(-)
 rename inference/.clang-format => .clang-format (100%)
 rename inference/.clang-format-ignore => .clang-format-ignore (100%)
 rename inference/.clang-tidy => .clang-tidy (100%)

diff --git a/inference/.clang-format b/.clang-format
similarity index 100%
rename from inference/.clang-format
rename to .clang-format
diff --git a/inference/.clang-format-ignore b/.clang-format-ignore
similarity index 100%
rename from inference/.clang-format-ignore
rename to .clang-format-ignore
diff --git a/inference/.clang-tidy b/.clang-tidy
similarity index 100%
rename from inference/.clang-tidy
rename to .clang-tidy

From 39ee2c4388dc612d9ea659a4cddcfe8089a2df4b Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Tue, 1 Oct 2024 08:56:53 -0500
Subject: [PATCH 439/442] Remove inference/Doxyfile.in

---
 inference/Doxyfile.in | 2494 -----------------------------------------
 1 file changed, 2494 deletions(-)
 delete mode 100644 inference/Doxyfile.in

diff --git a/inference/Doxyfile.in b/inference/Doxyfile.in
deleted file mode 100644
index 7b69eb8c5..000000000
--- a/inference/Doxyfile.in
+++ /dev/null
@@ -1,2494 +0,0 @@
-# Doxyfile 1.8.13
-
-# This file describes the settings to be used by the documentation system
-# doxygen (www.doxygen.org) for a project.
-#
-# All text after a double hash (##) is considered a comment and is placed in
-# front of the TAG it is preceding.
-#
-# All text after a single hash (#) is considered a comment and will be ignored.
-# The format is:
-# TAG = value [value, ...]
-# For lists, items can also be appended using:
-# TAG += value [value, ...]
-# Values that contain spaces should be placed between quotes (\" \").
-
-#---------------------------------------------------------------------------
-# Project related configuration options
-#---------------------------------------------------------------------------
-
-# This tag specifies the encoding used for all characters in the config file
-# that follow. The default is UTF-8 which is also the encoding used for all text
-# before the first occurrence of this tag. Doxygen uses libiconv (or the iconv
-# built into libc) for the transcoding. See http://www.gnu.org/software/libiconv
-# for the list of possible encodings.
-# The default value is: UTF-8.
-
-DOXYFILE_ENCODING      = UTF-8
-
-# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by
-# double-quotes, unless you are using Doxywizard) that should identify the
-# project for which the documentation is generated. This name is used in the
-# title of most generated pages and in a few other places.
-# The default value is: My Project.
-
-PROJECT_NAME           = "Bergamot Translator"
-
-# The PROJECT_NUMBER tag can be used to enter a project or revision number. This
-# could be handy for archiving the generated documentation or if some version
-# control system is used.
-
-PROJECT_NUMBER         =
-
-# Using the PROJECT_BRIEF tag one can provide an optional one line description
-# for a project that appears at the top of each page and should give viewer a
-# quick idea about the purpose of the project. Keep the description short.
-
-PROJECT_BRIEF          =
-
-# With the PROJECT_LOGO tag one can specify a logo or an icon that is included
-# in the documentation. The maximum height of the logo should not exceed 55
-# pixels and the maximum width should not exceed 200 pixels. Doxygen will copy
-# the logo to the output directory.
-
-PROJECT_LOGO           =
-
-# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path
-# into which the generated documentation will be written. If a relative path is
-# entered, it will be relative to the location where doxygen was started. If
-# left blank the current directory will be used.
-
-OUTPUT_DIRECTORY       = build/doc
-
-# If the CREATE_SUBDIRS tag is set to YES then doxygen will create 4096 sub-
-# directories (in 2 levels) under the output directory of each output format and
-# will distribute the generated files over these directories. Enabling this
-# option can be useful when feeding doxygen a huge amount of source files, where
-# putting all generated files in the same directory would otherwise causes
-# performance problems for the file system.
-# The default value is: NO.
-
-CREATE_SUBDIRS         = NO
-
-# If the ALLOW_UNICODE_NAMES tag is set to YES, doxygen will allow non-ASCII
-# characters to appear in the names of generated files. If set to NO, non-ASCII
-# characters will be escaped, for example _xE3_x81_x84 will be used for Unicode
-# U+3044.
-# The default value is: NO.
-
-ALLOW_UNICODE_NAMES    = NO
-
-# The OUTPUT_LANGUAGE tag is used to specify the language in which all
-# documentation generated by doxygen is written. Doxygen will use this
-# information to generate all constant output in the proper language.
-# Possible values are: Afrikaans, Arabic, Armenian, Brazilian, Catalan, Chinese,
-# Chinese-Traditional, Croatian, Czech, Danish, Dutch, English (United States),
-# Esperanto, Farsi (Persian), Finnish, French, German, Greek, Hungarian,
-# Indonesian, Italian, Japanese, Japanese-en (Japanese with English messages),
-# Korean, Korean-en (Korean with English messages), Latvian, Lithuanian,
-# Macedonian, Norwegian, Persian (Farsi), Polish, Portuguese, Romanian, Russian,
-# Serbian, Serbian-Cyrillic, Slovak, Slovene, Spanish, Swedish, Turkish,
-# Ukrainian and Vietnamese.
-# The default value is: English.
-
-OUTPUT_LANGUAGE        = English
-
-# If the BRIEF_MEMBER_DESC tag is set to YES, doxygen will include brief member
-# descriptions after the members that are listed in the file and class
-# documentation (similar to Javadoc). Set to NO to disable this.
-# The default value is: YES.
-
-BRIEF_MEMBER_DESC      = YES
-
-# If the REPEAT_BRIEF tag is set to YES, doxygen will prepend the brief
-# description of a member or function before the detailed description
-#
-# Note: If both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the
-# brief descriptions will be completely suppressed.
-# The default value is: YES.
-
-REPEAT_BRIEF           = YES
-
-# This tag implements a quasi-intelligent brief description abbreviator that is
-# used to form the text in various listings. Each string in this list, if found
-# as the leading text of the brief description, will be stripped from the text
-# and the result, after processing the whole list, is used as the annotated
-# text. Otherwise, the brief description is used as-is. If left blank, the
-# following values are used ($name is automatically replaced with the name of
-# the entity):The $name class, The $name widget, The $name file, is, provides,
-# specifies, contains, represents, a, an and the.
-
-ABBREVIATE_BRIEF       = "The $name class" \
-                         "The $name widget" \
-                         "The $name file" \
-                         is \
-                         provides \
-                         specifies \
-                         contains \
-                         represents \
-                         a \
-                         an \
-                         the
-
-# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then
-# doxygen will generate a detailed section even if there is only a brief
-# description.
-# The default value is: NO.
-
-ALWAYS_DETAILED_SEC    = NO
-
-# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all
-# inherited members of a class in the documentation of that class as if those
-# members were ordinary class members. Constructors, destructors and assignment
-# operators of the base classes will not be shown.
-# The default value is: NO.
-
-INLINE_INHERITED_MEMB  = NO
-
-# If the FULL_PATH_NAMES tag is set to YES, doxygen will prepend the full path
-# before files name in the file list and in the header files. If set to NO the
-# shortest path that makes the file name unique will be used
-# The default value is: YES.
-
-FULL_PATH_NAMES        = YES
-
-# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path.
-# Stripping is only done if one of the specified strings matches the left-hand
-# part of the path. The tag can be used to show relative paths in the file list.
-# If left blank the directory from which doxygen is run is used as the path to
-# strip.
-#
-# Note that you can specify absolute paths here, but also relative paths, which
-# will be relative from the directory where doxygen is started.
-# This tag requires that the tag FULL_PATH_NAMES is set to YES.
-
-STRIP_FROM_PATH        =
-
-# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of the
-# path mentioned in the documentation of a class, which tells the reader which
-# header file to include in order to use a class. If left blank only the name of
-# the header file containing the class definition is used. Otherwise one should
-# specify the list of include paths that are normally passed to the compiler
-# using the -I flag.
-
-STRIP_FROM_INC_PATH    =
-
-# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter (but
-# less readable) file names. This can be useful is your file systems doesn't
-# support long names like on DOS, Mac, or CD-ROM.
-# The default value is: NO.
-
-SHORT_NAMES            = NO
-
-# If the JAVADOC_AUTOBRIEF tag is set to YES then doxygen will interpret the
-# first line (until the first dot) of a Javadoc-style comment as the brief
-# description. If set to NO, the Javadoc-style will behave just like regular Qt-
-# style comments (thus requiring an explicit @brief command for a brief
-# description.)
-# The default value is: NO.
-
-JAVADOC_AUTOBRIEF      = NO
-
-# If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first
-# line (until the first dot) of a Qt-style comment as the brief description. If
-# set to NO, the Qt-style will behave just like regular Qt-style comments (thus
-# requiring an explicit \brief command for a brief description.)
-# The default value is: NO.
-
-QT_AUTOBRIEF           = NO
-
-# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make doxygen treat a
-# multi-line C++ special comment block (i.e. a block of //! or /// comments) as
-# a brief description. This used to be the default behavior. The new default is
-# to treat a multi-line C++ comment block as a detailed description. Set this
-# tag to YES if you prefer the old behavior instead.
-#
-# Note that setting this tag to YES also means that rational rose comments are
-# not recognized any more.
-# The default value is: NO.
-
-MULTILINE_CPP_IS_BRIEF = NO
-
-# If the INHERIT_DOCS tag is set to YES then an undocumented member inherits the
-# documentation from any documented member that it re-implements.
-# The default value is: YES.
-
-INHERIT_DOCS           = YES
-
-# If the SEPARATE_MEMBER_PAGES tag is set to YES then doxygen will produce a new
-# page for each member. If set to NO, the documentation of a member will be part
-# of the file/class/namespace that contains it.
-# The default value is: NO.
-
-SEPARATE_MEMBER_PAGES  = NO
-
-# The TAB_SIZE tag can be used to set the number of spaces in a tab. Doxygen
-# uses this value to replace tabs by spaces in code fragments.
-# Minimum value: 1, maximum value: 16, default value: 4.
-
-TAB_SIZE               = 2
-
-# This tag can be used to specify a number of aliases that act as commands in
-# the documentation. An alias has the form:
-# name=value
-# For example adding
-# "sideeffect=@par Side Effects:\n"
-# will allow you to put the command \sideeffect (or @sideeffect) in the
-# documentation, which will result in a user-defined paragraph with heading
-# "Side Effects:". You can put \n's in the value part of an alias to insert
-# newlines.
-
-ALIASES                =
-
-# This tag can be used to specify a number of word-keyword mappings (TCL only).
-# A mapping has the form "name=value". For example adding "class=itcl::class"
-# will allow you to use the command class in the itcl::class meaning.
-
-TCL_SUBST              =
-
-# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources
-# only. Doxygen will then generate output that is more tailored for C. For
-# instance, some of the names that are used will be different. The list of all
-# members will be omitted, etc.
-# The default value is: NO.
-
-OPTIMIZE_OUTPUT_FOR_C  = NO
-
-# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java or
-# Python sources only. Doxygen will then generate output that is more tailored
-# for that language. For instance, namespaces will be presented as packages,
-# qualified scopes will look different, etc.
-# The default value is: NO.
-
-OPTIMIZE_OUTPUT_JAVA   = NO
-
-# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran
-# sources. Doxygen will then generate output that is tailored for Fortran.
-# The default value is: NO.
-
-OPTIMIZE_FOR_FORTRAN   = NO
-
-# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL
-# sources. Doxygen will then generate output that is tailored for VHDL.
-# The default value is: NO.
-
-OPTIMIZE_OUTPUT_VHDL   = NO
-
-# Doxygen selects the parser to use depending on the extension of the files it
-# parses. With this tag you can assign which parser to use for a given
-# extension. Doxygen has a built-in mapping, but you can override or extend it
-# using this tag. The format is ext=language, where ext is a file extension, and
-# language is one of the parsers supported by doxygen: IDL, Java, Javascript,
-# C#, C, C++, D, PHP, Objective-C, Python, Fortran (fixed format Fortran:
-# FortranFixed, free formatted Fortran: FortranFree, unknown formatted Fortran:
-# Fortran. In the later case the parser tries to guess whether the code is fixed
-# or free formatted code, this is the default for Fortran type files), VHDL. For
-# instance to make doxygen treat .inc files as Fortran files (default is PHP),
-# and .f files as C (default is Fortran), use: inc=Fortran f=C.
-#
-# Note: For files without extension you can use no_extension as a placeholder.
-#
-# Note that for custom extensions you also need to set FILE_PATTERNS otherwise
-# the files are not read by doxygen.
-
-EXTENSION_MAPPING      =
-
-# If the MARKDOWN_SUPPORT tag is enabled then doxygen pre-processes all comments
-# according to the Markdown format, which allows for more readable
-# documentation. See http://daringfireball.net/projects/markdown/ for details.
-# The output of markdown processing is further processed by doxygen, so you can
-# mix doxygen, HTML, and XML commands with Markdown formatting. Disable only in
-# case of backward compatibilities issues.
-# The default value is: YES.
-
-MARKDOWN_SUPPORT       = YES
-
-# When the TOC_INCLUDE_HEADINGS tag is set to a non-zero value, all headings up
-# to that level are automatically included in the table of contents, even if
-# they do not have an id attribute.
-# Note: This feature currently applies only to Markdown headings.
-# Minimum value: 0, maximum value: 99, default value: 0.
-# This tag requires that the tag MARKDOWN_SUPPORT is set to YES.
-
-TOC_INCLUDE_HEADINGS   = 0
-
-# When enabled doxygen tries to link words that correspond to documented
-# classes, or namespaces to their corresponding documentation. Such a link can
-# be prevented in individual cases by putting a % sign in front of the word or
-# globally by setting AUTOLINK_SUPPORT to NO.
-# The default value is: YES.
-
-AUTOLINK_SUPPORT       = YES
-
-# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want
-# to include (a tag file for) the STL sources as input, then you should set this
-# tag to YES in order to let doxygen match functions declarations and
-# definitions whose arguments contain STL classes (e.g. func(std::string);
-# versus func(std::string) {}). This also make the inheritance and collaboration
-# diagrams that involve STL classes more complete and accurate.
-# The default value is: NO.
-
-BUILTIN_STL_SUPPORT    = NO
-
-# If you use Microsoft's C++/CLI language, you should set this option to YES to
-# enable parsing support.
-# The default value is: NO.
-
-CPP_CLI_SUPPORT        = NO
-
-# Set the SIP_SUPPORT tag to YES if your project consists of sip (see:
-# http://www.riverbankcomputing.co.uk/software/sip/intro) sources only. Doxygen
-# will parse them like normal C++ but will assume all classes use public instead
-# of private inheritance when no explicit protection keyword is present.
-# The default value is: NO.
-
-SIP_SUPPORT            = NO
-
-# For Microsoft's IDL there are propget and propput attributes to indicate
-# getter and setter methods for a property. Setting this option to YES will make
-# doxygen to replace the get and set methods by a property in the documentation.
-# This will only work if the methods are indeed getting or setting a simple
-# type. If this is not the case, or you want to show the methods anyway, you
-# should set this option to NO.
-# The default value is: YES.
-
-IDL_PROPERTY_SUPPORT   = YES
-
-# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC
-# tag is set to YES then doxygen will reuse the documentation of the first
-# member in the group (if any) for the other members of the group. By default
-# all members of a group must be documented explicitly.
-# The default value is: NO.
-
-DISTRIBUTE_GROUP_DOC   = NO
-
-# If one adds a struct or class to a group and this option is enabled, then also
-# any nested class or struct is added to the same group. By default this option
-# is disabled and one has to add nested compounds explicitly via \ingroup.
-# The default value is: NO.
-
-GROUP_NESTED_COMPOUNDS = NO
-
-# Set the SUBGROUPING tag to YES to allow class member groups of the same type
-# (for instance a group of public functions) to be put as a subgroup of that
-# type (e.g. under the Public Functions section). Set it to NO to prevent
-# subgrouping. Alternatively, this can be done per class using the
-# \nosubgrouping command.
-# The default value is: YES.
-
-SUBGROUPING            = YES
-
-# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions
-# are shown inside the group in which they are included (e.g. using \ingroup)
-# instead of on a separate page (for HTML and Man pages) or section (for LaTeX
-# and RTF).
-#
-# Note that this feature does not work in combination with
-# SEPARATE_MEMBER_PAGES.
-# The default value is: NO.
-
-INLINE_GROUPED_CLASSES = NO
-
-# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and unions
-# with only public data fields or simple typedef fields will be shown inline in
-# the documentation of the scope in which they are defined (i.e. file,
-# namespace, or group documentation), provided this scope is documented. If set
-# to NO, structs, classes, and unions are shown on a separate page (for HTML and
-# Man pages) or section (for LaTeX and RTF).
-# The default value is: NO.
-
-INLINE_SIMPLE_STRUCTS  = NO
-
-# When TYPEDEF_HIDES_STRUCT tag is enabled, a typedef of a struct, union, or
-# enum is documented as struct, union, or enum with the name of the typedef. So
-# typedef struct TypeS {} TypeT, will appear in the documentation as a struct
-# with name TypeT. When disabled the typedef will appear as a member of a file,
-# namespace, or class. And the struct will be named TypeS. This can typically be
-# useful for C code in case the coding convention dictates that all compound
-# types are typedef'ed and only the typedef is referenced, never the tag name.
-# The default value is: NO.
-
-TYPEDEF_HIDES_STRUCT   = NO
-
-# The size of the symbol lookup cache can be set using LOOKUP_CACHE_SIZE. This
-# cache is used to resolve symbols given their name and scope. Since this can be
-# an expensive process and often the same symbol appears multiple times in the
-# code, doxygen keeps a cache of pre-resolved symbols. If the cache is too small
-# doxygen will become slower. If the cache is too large, memory is wasted. The
-# cache size is given by this formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range
-# is 0..9, the default is 0, corresponding to a cache size of 2^16=65536
-# symbols. At the end of a run doxygen will report the cache usage and suggest
-# the optimal cache size from a speed point of view.
-# Minimum value: 0, maximum value: 9, default value: 0.
-
-LOOKUP_CACHE_SIZE      = 0
-
-#---------------------------------------------------------------------------
-# Build related configuration options
-#---------------------------------------------------------------------------
-
-# If the EXTRACT_ALL tag is set to YES, doxygen will assume all entities in
-# documentation are documented, even if no documentation was available. Private
-# class members and static file members will be hidden unless the
-# EXTRACT_PRIVATE respectively EXTRACT_STATIC tags are set to YES.
-# Note: This will also disable the warnings about undocumented members that are
-# normally produced when WARNINGS is set to YES.
-# The default value is: NO.
-
-EXTRACT_ALL            = NO
-
-# If the EXTRACT_PRIVATE tag is set to YES, all private members of a class will
-# be included in the documentation.
-# The default value is: NO.
-
-EXTRACT_PRIVATE        = YES
-
-# If the EXTRACT_PACKAGE tag is set to YES, all members with package or internal
-# scope will be included in the documentation.
-# The default value is: NO.
-
-EXTRACT_PACKAGE        = NO
-
-# If the EXTRACT_STATIC tag is set to YES, all static members of a file will be
-# included in the documentation.
-# The default value is: NO.
-
-EXTRACT_STATIC         = NO
-
-# If the EXTRACT_LOCAL_CLASSES tag is set to YES, classes (and structs) defined
-# locally in source files will be included in the documentation. If set to NO,
-# only classes defined in header files are included. Does not have any effect
-# for Java sources.
-# The default value is: YES.
-
-EXTRACT_LOCAL_CLASSES  = YES
-
-# This flag is only useful for Objective-C code. If set to YES, local methods,
-# which are defined in the implementation section but not in the interface are
-# included in the documentation. If set to NO, only methods in the interface are
-# included.
-# The default value is: NO.
-
-EXTRACT_LOCAL_METHODS  = NO
-
-# If this flag is set to YES, the members of anonymous namespaces will be
-# extracted and appear in the documentation as a namespace called
-# 'anonymous_namespace{file}', where file will be replaced with the base name of
-# the file that contains the anonymous namespace. By default anonymous namespace
-# are hidden.
-# The default value is: NO.
-
-EXTRACT_ANON_NSPACES   = NO
-
-# If the HIDE_UNDOC_MEMBERS tag is set to YES, doxygen will hide all
-# undocumented members inside documented classes or files. If set to NO these
-# members will be included in the various overviews, but no documentation
-# section is generated. This option has no effect if EXTRACT_ALL is enabled.
-# The default value is: NO.
-
-HIDE_UNDOC_MEMBERS     = NO
-
-# If the HIDE_UNDOC_CLASSES tag is set to YES, doxygen will hide all
-# undocumented classes that are normally visible in the class hierarchy. If set
-# to NO, these classes will be included in the various overviews. This option
-# has no effect if EXTRACT_ALL is enabled.
-# The default value is: NO.
-
-HIDE_UNDOC_CLASSES     = NO
-
-# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, doxygen will hide all friend
-# (class|struct|union) declarations. If set to NO, these declarations will be
-# included in the documentation.
-# The default value is: NO.
-
-HIDE_FRIEND_COMPOUNDS  = NO
-
-# If the HIDE_IN_BODY_DOCS tag is set to YES, doxygen will hide any
-# documentation blocks found inside the body of a function. If set to NO, these
-# blocks will be appended to the function's detailed documentation block.
-# The default value is: NO.
-
-HIDE_IN_BODY_DOCS      = NO
-
-# The INTERNAL_DOCS tag determines if documentation that is typed after a
-# \internal command is included. If the tag is set to NO then the documentation
-# will be excluded. Set it to YES to include the internal documentation.
-# The default value is: NO.
-
-INTERNAL_DOCS          = NO
-
-# If the CASE_SENSE_NAMES tag is set to NO then doxygen will only generate file
-# names in lower-case letters. If set to YES, upper-case letters are also
-# allowed. This is useful if you have classes or files whose names only differ
-# in case and if your file system supports case sensitive file names. Windows
-# and Mac users are advised to set this option to NO.
-# The default value is: system dependent.
-
-CASE_SENSE_NAMES       = YES
-
-# If the HIDE_SCOPE_NAMES tag is set to NO then doxygen will show members with
-# their full class and namespace scopes in the documentation. If set to YES, the
-# scope will be hidden.
-# The default value is: NO.
-
-HIDE_SCOPE_NAMES       = NO
-
-# If the HIDE_COMPOUND_REFERENCE tag is set to NO (default) then doxygen will
-# append additional text to a page's title, such as Class Reference. If set to
-# YES the compound reference will be hidden.
-# The default value is: NO.
-
-HIDE_COMPOUND_REFERENCE= NO
-
-# If the SHOW_INCLUDE_FILES tag is set to YES then doxygen will put a list of
-# the files that are included by a file in the documentation of that file.
-# The default value is: YES.
-
-SHOW_INCLUDE_FILES     = YES
-
-# If the SHOW_GROUPED_MEMB_INC tag is set to YES then Doxygen will add for each
-# grouped member an include statement to the documentation, telling the reader
-# which file to include in order to use the member.
-# The default value is: NO.
-
-SHOW_GROUPED_MEMB_INC  = NO
-
-# If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include
-# files with double quotes in the documentation rather than with sharp brackets.
-# The default value is: NO.
-
-FORCE_LOCAL_INCLUDES   = NO
-
-# If the INLINE_INFO tag is set to YES then a tag [inline] is inserted in the
-# documentation for inline members.
-# The default value is: YES.
-
-INLINE_INFO            = YES
-
-# If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the
-# (detailed) documentation of file and class members alphabetically by member
-# name. If set to NO, the members will appear in declaration order.
-# The default value is: YES.
-
-SORT_MEMBER_DOCS       = YES
-
-# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief
-# descriptions of file, namespace and class members alphabetically by member
-# name. If set to NO, the members will appear in declaration order. Note that
-# this will also influence the order of the classes in the class list.
-# The default value is: NO.
-
-SORT_BRIEF_DOCS        = NO
-
-# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the
-# (brief and detailed) documentation of class members so that constructors and
-# destructors are listed first. If set to NO the constructors will appear in the
-# respective orders defined by SORT_BRIEF_DOCS and SORT_MEMBER_DOCS.
-# Note: If SORT_BRIEF_DOCS is set to NO this option is ignored for sorting brief
-# member documentation.
-# Note: If SORT_MEMBER_DOCS is set to NO this option is ignored for sorting
-# detailed member documentation.
-# The default value is: NO.
-
-SORT_MEMBERS_CTORS_1ST = NO
-
-# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the hierarchy
-# of group names into alphabetical order. If set to NO the group names will
-# appear in their defined order.
-# The default value is: NO.
-
-SORT_GROUP_NAMES       = NO
-
-# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be sorted by
-# fully-qualified names, including namespaces. If set to NO, the class list will
-# be sorted only by class name, not including the namespace part.
-# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES.
-# Note: This option applies only to the class list, not to the alphabetical
-# list.
-# The default value is: NO.
-
-SORT_BY_SCOPE_NAME     = NO
-
-# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to do proper
-# type resolution of all parameters of a function it will reject a match between
-# the prototype and the implementation of a member function even if there is
-# only one candidate or it is obvious which candidate to choose by doing a
-# simple string match. By disabling STRICT_PROTO_MATCHING doxygen will still
-# accept a match between prototype and implementation in such cases.
-# The default value is: NO.
-
-STRICT_PROTO_MATCHING  = NO
-
-# The GENERATE_TODOLIST tag can be used to enable (YES) or disable (NO) the todo
-# list. This list is created by putting \todo commands in the documentation.
-# The default value is: YES.
-
-GENERATE_TODOLIST      = YES
-
-# The GENERATE_TESTLIST tag can be used to enable (YES) or disable (NO) the test
-# list. This list is created by putting \test commands in the documentation.
-# The default value is: YES.
-
-GENERATE_TESTLIST      = YES
-
-# The GENERATE_BUGLIST tag can be used to enable (YES) or disable (NO) the bug
-# list. This list is created by putting \bug commands in the documentation.
-# The default value is: YES.
-
-GENERATE_BUGLIST       = YES
-
-# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or disable (NO)
-# the deprecated list. This list is created by putting \deprecated commands in
-# the documentation.
-# The default value is: YES.
-
-GENERATE_DEPRECATEDLIST= YES
-
-# The ENABLED_SECTIONS tag can be used to enable conditional documentation
-# sections, marked by \if <section_label> ... \endif and \cond <section_label>
-# ... \endcond blocks.
-
-ENABLED_SECTIONS       =
-
-# The MAX_INITIALIZER_LINES tag determines the maximum number of lines that the
-# initial value of a variable or macro / define can have for it to appear in the
-# documentation. If the initializer consists of more lines than specified here
-# it will be hidden. Use a value of 0 to hide initializers completely. The
-# appearance of the value of individual variables and macros / defines can be
-# controlled using \showinitializer or \hideinitializer command in the
-# documentation regardless of this setting.
-# Minimum value: 0, maximum value: 10000, default value: 30.
-
-MAX_INITIALIZER_LINES  = 30
-
-# Set the SHOW_USED_FILES tag to NO to disable the list of files generated at
-# the bottom of the documentation of classes and structs. If set to YES, the
-# list will mention the files that were used to generate the documentation.
-# The default value is: YES.
-
-SHOW_USED_FILES        = YES
-
-# Set the SHOW_FILES tag to NO to disable the generation of the Files page. This
-# will remove the Files entry from the Quick Index and from the Folder Tree View
-# (if specified).
-# The default value is: YES.
-
-SHOW_FILES             = YES
-
-# Set the SHOW_NAMESPACES tag to NO to disable the generation of the Namespaces
-# page. This will remove the Namespaces entry from the Quick Index and from the
-# Folder Tree View (if specified).
-# The default value is: YES.
-
-SHOW_NAMESPACES        = YES
-
-# The FILE_VERSION_FILTER tag can be used to specify a program or script that
-# doxygen should invoke to get the current version for each file (typically from
-# the version control system). Doxygen will invoke the program by executing (via
-# popen()) the command command input-file, where command is the value of the
-# FILE_VERSION_FILTER tag, and input-file is the name of an input file provided
-# by doxygen. Whatever the program writes to standard output is used as the file
-# version. For an example see the documentation.
-
-FILE_VERSION_FILTER    =
-
-# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed
-# by doxygen. The layout file controls the global structure of the generated
-# output files in an output format independent way. To create the layout file
-# that represents doxygen's defaults, run doxygen with the -l option. You can
-# optionally specify a file name after the option, if omitted DoxygenLayout.xml
-# will be used as the name of the layout file.
-#
-# Note that if you run doxygen from a directory containing a file called
-# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE
-# tag is left empty.
-
-LAYOUT_FILE            =
-
-# The CITE_BIB_FILES tag can be used to specify one or more bib files containing
-# the reference definitions. This must be a list of .bib files. The .bib
-# extension is automatically appended if omitted. This requires the bibtex tool
-# to be installed. See also http://en.wikipedia.org/wiki/BibTeX for more info.
-# For LaTeX the style of the bibliography can be controlled using
-# LATEX_BIB_STYLE. To use this feature you need bibtex and perl available in the
-# search path. See also \cite for info how to create references.
-
-CITE_BIB_FILES         =
-
-#---------------------------------------------------------------------------
-# Configuration options related to warning and progress messages
-#---------------------------------------------------------------------------
-
-# The QUIET tag can be used to turn on/off the messages that are generated to
-# standard output by doxygen. If QUIET is set to YES this implies that the
-# messages are off.
-# The default value is: NO.
-
-QUIET                  = NO
-
-# The WARNINGS tag can be used to turn on/off the warning messages that are
-# generated to standard error (stderr) by doxygen. If WARNINGS is set to YES
-# this implies that the warnings are on.
-#
-# Tip: Turn warnings on while writing the documentation.
-# The default value is: YES.
-
-WARNINGS               = YES
-
-# If the WARN_IF_UNDOCUMENTED tag is set to YES then doxygen will generate
-# warnings for undocumented members. If EXTRACT_ALL is set to YES then this flag
-# will automatically be disabled.
-# The default value is: YES.
-
-WARN_IF_UNDOCUMENTED   = YES
-
-# If the WARN_IF_DOC_ERROR tag is set to YES, doxygen will generate warnings for
-# potential errors in the documentation, such as not documenting some parameters
-# in a documented function, or documenting parameters that don't exist or using
-# markup commands wrongly.
-# The default value is: YES.
-
-WARN_IF_DOC_ERROR      = YES
-
-# This WARN_NO_PARAMDOC option can be enabled to get warnings for functions that
-# are documented, but have no documentation for their parameters or return
-# value. If set to NO, doxygen will only warn about wrong or incomplete
-# parameter documentation, but not about the absence of documentation.
-# The default value is: NO.
-
-WARN_NO_PARAMDOC       = NO
-
-# If the WARN_AS_ERROR tag is set to YES then doxygen will immediately stop when
-# a warning is encountered.
-# The default value is: NO.
-
-WARN_AS_ERROR          = NO
-
-# The WARN_FORMAT tag determines the format of the warning messages that doxygen
-# can produce. The string should contain the $file, $line, and $text tags, which
-# will be replaced by the file and line number from which the warning originated
-# and the warning text. Optionally the format may contain $version, which will
-# be replaced by the version of the file (if it could be obtained via
-# FILE_VERSION_FILTER)
-# The default value is: $file:$line: $text.
-
-WARN_FORMAT            = "$file:$line: $text"
-
-# The WARN_LOGFILE tag can be used to specify a file to which warning and error
-# messages should be written. If left blank the output is written to standard
-# error (stderr).
-
-WARN_LOGFILE           =
-
-#---------------------------------------------------------------------------
-# Configuration options related to the input files
-#---------------------------------------------------------------------------
-
-# The INPUT tag is used to specify the files and/or directories that contain
-# documented source files. You may enter file names like myfile.cpp or
-# directories like /usr/src/myproject. Separate the files or directories with
-# spaces. See also FILE_PATTERNS and EXTENSION_MAPPING
-# Note: If this tag is empty the current directory is searched.
-
-INPUT                  = src app 
-
-# This tag can be used to specify the character encoding of the source files
-# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
-# libiconv (or the iconv built into libc) for the transcoding. See the libiconv
-# documentation (see: http://www.gnu.org/software/libiconv) for the list of
-# possible encodings.
-# The default value is: UTF-8.
-
-INPUT_ENCODING         = UTF-8
-
-# If the value of the INPUT tag contains directories, you can use the
-# FILE_PATTERNS tag to specify one or more wildcard patterns (like *.cpp and
-# *.h) to filter out the source-files in the directories.
-#
-# Note that for custom extensions or not directly supported extensions you also
-# need to set EXTENSION_MAPPING for the extension otherwise the files are not
-# read by doxygen.
-#
-# If left blank the following patterns are tested:*.c, *.cc, *.cxx, *.cpp,
-# *.c++, *.java, *.ii, *.ixx, *.ipp, *.i++, *.inl, *.idl, *.ddl, *.odl, *.h,
-# *.hh, *.hxx, *.hpp, *.h++, *.cs, *.d, *.php, *.php4, *.php5, *.phtml, *.inc,
-# *.m, *.markdown, *.md, *.mm, *.dox, *.py, *.pyw, *.f90, *.f95, *.f03, *.f08,
-# *.f, *.for, *.tcl, *.vhd, *.vhdl, *.ucf and *.qsf.
-
-FILE_PATTERNS          = *.c \
-                         *.cc \
-                         *.cxx \
-                         *.cpp \
-                         *.c++ \
-                         *.java \
-                         *.ii \
-                         *.ixx \
-                         *.ipp \
-                         *.i++ \
-                         *.inl \
-                         *.idl \
-                         *.ddl \
-                         *.odl \
-                         *.h \
-                         *.hh \
-                         *.hxx \
-                         *.hpp \
-                         *.h++ \
-                         *.cs \
-                         *.d \
-                         *.php \
-                         *.php4 \
-                         *.php5 \
-                         *.phtml \
-                         *.inc \
-                         *.m \
-                         *.markdown \
-                         *.md \
-                         *.mm \
-                         *.dox \
-                         *.py \
-                         *.pyw \
-                         *.f90 \
-                         *.f95 \
-                         *.f03 \
-                         *.f08 \
-                         *.f \
-                         *.for \
-                         *.tcl \
-                         *.vhd \
-                         *.vhdl \
-                         *.ucf \
-                         *.qsf
-
-# The RECURSIVE tag can be used to specify whether or not subdirectories should
-# be searched for input files as well.
-# The default value is: NO.
-
-RECURSIVE              = YES
-
-# The EXCLUDE tag can be used to specify files and/or directories that should be
-# excluded from the INPUT source files. This way you can easily exclude a
-# subdirectory from a directory tree whose root is specified with the INPUT tag.
-#
-# Note that relative paths are relative to the directory from which doxygen is
-# run.
-
-EXCLUDE                = 
-
-# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or
-# directories that are symbolic links (a Unix file system feature) are excluded
-# from the input.
-# The default value is: NO.
-
-EXCLUDE_SYMLINKS       = NO
-
-# If the value of the INPUT tag contains directories, you can use the
-# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude
-# certain files from those directories.
-#
-# Note that the wildcards are matched against the file with absolute path, so to
-# exclude all test directories for example use the pattern */test/*
-
-EXCLUDE_PATTERNS       =
-
-# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names
-# (namespaces, classes, functions, etc.) that should be excluded from the
-# output. The symbol name can be a fully qualified name, a word, or if the
-# wildcard * is used, a substring. Examples: ANamespace, AClass,
-# AClass::ANamespace, ANamespace::*Test
-#
-# Note that the wildcards are matched against the file with absolute path, so to
-# exclude all test directories use the pattern */test/*
-
-EXCLUDE_SYMBOLS        =
-
-# The EXAMPLE_PATH tag can be used to specify one or more files or directories
-# that contain example code fragments that are included (see the \include
-# command).
-
-EXAMPLE_PATH           =
-
-# If the value of the EXAMPLE_PATH tag contains directories, you can use the
-# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and
-# *.h) to filter out the source-files in the directories. If left blank all
-# files are included.
-
-EXAMPLE_PATTERNS       = *
-
-# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be
-# searched for input files to be used with the \include or \dontinclude commands
-# irrespective of the value of the RECURSIVE tag.
-# The default value is: NO.
-
-EXAMPLE_RECURSIVE      = NO
-
-# The IMAGE_PATH tag can be used to specify one or more files or directories
-# that contain images that are to be included in the documentation (see the
-# \image command).
-
-IMAGE_PATH             =
-
-# The INPUT_FILTER tag can be used to specify a program that doxygen should
-# invoke to filter for each input file. Doxygen will invoke the filter program
-# by executing (via popen()) the command:
-#
-# <filter> <input-file>
-#
-# where <filter> is the value of the INPUT_FILTER tag, and <input-file> is the
-# name of an input file. Doxygen will then use the output that the filter
-# program writes to standard output. If FILTER_PATTERNS is specified, this tag
-# will be ignored.
-#
-# Note that the filter must not add or remove lines; it is applied before the
-# code is scanned, but not when the output code is generated. If lines are added
-# or removed, the anchors will not be placed correctly.
-#
-# Note that for custom extensions or not directly supported extensions you also
-# need to set EXTENSION_MAPPING for the extension otherwise the files are not
-# properly processed by doxygen.
-
-INPUT_FILTER           =
-
-# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern
-# basis. Doxygen will compare the file name with each pattern and apply the
-# filter if there is a match. The filters are a list of the form: pattern=filter
-# (like *.cpp=my_cpp_filter). See INPUT_FILTER for further information on how
-# filters are used. If the FILTER_PATTERNS tag is empty or if none of the
-# patterns match the file name, INPUT_FILTER is applied.
-#
-# Note that for custom extensions or not directly supported extensions you also
-# need to set EXTENSION_MAPPING for the extension otherwise the files are not
-# properly processed by doxygen.
-
-FILTER_PATTERNS        =
-
-# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using
-# INPUT_FILTER) will also be used to filter the input files that are used for
-# producing the source files to browse (i.e. when SOURCE_BROWSER is set to YES).
-# The default value is: NO.
-
-FILTER_SOURCE_FILES    = NO
-
-# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file
-# pattern. A pattern will override the setting for FILTER_PATTERN (if any) and
-# it is also possible to disable source filtering for a specific pattern using
-# *.ext= (so without naming a filter).
-# This tag requires that the tag FILTER_SOURCE_FILES is set to YES.
-
-FILTER_SOURCE_PATTERNS =
-
-# If the USE_MDFILE_AS_MAINPAGE tag refers to the name of a markdown file that
-# is part of the input, its contents will be placed on the main page
-# (index.html). This can be useful if you have a project on for instance GitHub
-# and want to reuse the introduction page also for the doxygen output.
-
-USE_MDFILE_AS_MAINPAGE =
-
-#---------------------------------------------------------------------------
-# Configuration options related to source browsing
-#---------------------------------------------------------------------------
-
-# If the SOURCE_BROWSER tag is set to YES then a list of source files will be
-# generated. Documented entities will be cross-referenced with these sources.
-#
-# Note: To get rid of all source code in the generated output, make sure that
-# also VERBATIM_HEADERS is set to NO.
-# The default value is: NO.
-
-SOURCE_BROWSER         = NO
-
-# Setting the INLINE_SOURCES tag to YES will include the body of functions,
-# classes and enums directly into the documentation.
-# The default value is: NO.
-
-INLINE_SOURCES         = NO
-
-# Setting the STRIP_CODE_COMMENTS tag to YES will instruct doxygen to hide any
-# special comment blocks from generated source code fragments. Normal C, C++ and
-# Fortran comments will always remain visible.
-# The default value is: YES.
-
-STRIP_CODE_COMMENTS    = YES
-
-# If the REFERENCED_BY_RELATION tag is set to YES then for each documented
-# function all documented functions referencing it will be listed.
-# The default value is: NO.
-
-REFERENCED_BY_RELATION = NO
-
-# If the REFERENCES_RELATION tag is set to YES then for each documented function
-# all documented entities called/used by that function will be listed.
-# The default value is: NO.
-
-REFERENCES_RELATION    = NO
-
-# If the REFERENCES_LINK_SOURCE tag is set to YES and SOURCE_BROWSER tag is set
-# to YES then the hyperlinks from functions in REFERENCES_RELATION and
-# REFERENCED_BY_RELATION lists will link to the source code. Otherwise they will
-# link to the documentation.
-# The default value is: YES.
-
-REFERENCES_LINK_SOURCE = YES
-
-# If SOURCE_TOOLTIPS is enabled (the default) then hovering a hyperlink in the
-# source code will show a tooltip with additional information such as prototype,
-# brief description and links to the definition and documentation. Since this
-# will make the HTML file larger and loading of large files a bit slower, you
-# can opt to disable this feature.
-# The default value is: YES.
-# This tag requires that the tag SOURCE_BROWSER is set to YES.
-
-SOURCE_TOOLTIPS        = YES
-
-# If the USE_HTAGS tag is set to YES then the references to source code will
-# point to the HTML generated by the htags(1) tool instead of doxygen built-in
-# source browser. The htags tool is part of GNU's global source tagging system
-# (see http://www.gnu.org/software/global/global.html). You will need version
-# 4.8.6 or higher.
-#
-# To use it do the following:
-# - Install the latest version of global
-# - Enable SOURCE_BROWSER and USE_HTAGS in the config file
-# - Make sure the INPUT points to the root of the source tree
-# - Run doxygen as normal
-#
-# Doxygen will invoke htags (and that will in turn invoke gtags), so these
-# tools must be available from the command line (i.e. in the search path).
-#
-# The result: instead of the source browser generated by doxygen, the links to
-# source code will now point to the output of htags.
-# The default value is: NO.
-# This tag requires that the tag SOURCE_BROWSER is set to YES.
-
-USE_HTAGS              = NO
-
-# If the VERBATIM_HEADERS tag is set the YES then doxygen will generate a
-# verbatim copy of the header file for each class for which an include is
-# specified. Set to NO to disable this.
-# See also: Section \class.
-# The default value is: YES.
-
-VERBATIM_HEADERS       = YES
-
-# If the CLANG_ASSISTED_PARSING tag is set to YES then doxygen will use the
-# clang parser (see: http://clang.llvm.org/) for more accurate parsing at the
-# cost of reduced performance. This can be particularly helpful with template
-# rich C++ code for which doxygen's built-in parser lacks the necessary type
-# information.
-# Note: The availability of this option depends on whether or not doxygen was
-# generated with the -Duse-libclang=ON option for CMake.
-# The default value is: NO.
-
-CLANG_ASSISTED_PARSING = NO
-
-# If clang assisted parsing is enabled you can provide the compiler with command
-# line options that you would normally use when invoking the compiler. Note that
-# the include paths will already be set by doxygen for the files and directories
-# specified with INPUT and INCLUDE_PATH.
-# This tag requires that the tag CLANG_ASSISTED_PARSING is set to YES.
-
-CLANG_OPTIONS          =
-
-#---------------------------------------------------------------------------
-# Configuration options related to the alphabetical class index
-#---------------------------------------------------------------------------
-
-# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index of all
-# compounds will be generated. Enable this if the project contains a lot of
-# classes, structs, unions or interfaces.
-# The default value is: YES.
-
-ALPHABETICAL_INDEX     = YES
-
-# The COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns in
-# which the alphabetical index list will be split.
-# Minimum value: 1, maximum value: 20, default value: 5.
-# This tag requires that the tag ALPHABETICAL_INDEX is set to YES.
-
-COLS_IN_ALPHA_INDEX    = 5
-
-# In case all classes in a project start with a common prefix, all classes will
-# be put under the same header in the alphabetical index. The IGNORE_PREFIX tag
-# can be used to specify a prefix (or a list of prefixes) that should be ignored
-# while generating the index headers.
-# This tag requires that the tag ALPHABETICAL_INDEX is set to YES.
-
-IGNORE_PREFIX          =
-
-#---------------------------------------------------------------------------
-# Configuration options related to the HTML output
-#---------------------------------------------------------------------------
-
-# If the GENERATE_HTML tag is set to YES, doxygen will generate HTML output
-# The default value is: YES.
-
-GENERATE_HTML          = YES
-
-# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. If a
-# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
-# it.
-# The default directory is: html.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_OUTPUT            = html
-
-# The HTML_FILE_EXTENSION tag can be used to specify the file extension for each
-# generated HTML page (for example: .htm, .php, .asp).
-# The default value is: .html.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_FILE_EXTENSION    = .html
-
-# The HTML_HEADER tag can be used to specify a user-defined HTML header file for
-# each generated HTML page. If the tag is left blank doxygen will generate a
-# standard header.
-#
-# To get valid HTML the header file that includes any scripts and style sheets
-# that doxygen needs, which is dependent on the configuration options used (e.g.
-# the setting GENERATE_TREEVIEW). It is highly recommended to start with a
-# default header using
-# doxygen -w html new_header.html new_footer.html new_stylesheet.css
-# YourConfigFile
-# and then modify the file new_header.html. See also section "Doxygen usage"
-# for information on how to generate the default header that doxygen normally
-# uses.
-# Note: The header is subject to change so you typically have to regenerate the
-# default header when upgrading to a newer version of doxygen. For a description
-# of the possible markers and block names see the documentation.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_HEADER            =
-
-# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each
-# generated HTML page. If the tag is left blank doxygen will generate a standard
-# footer. See HTML_HEADER for more information on how to generate a default
-# footer and what special commands can be used inside the footer. See also
-# section "Doxygen usage" for information on how to generate the default footer
-# that doxygen normally uses.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_FOOTER            =
-
-# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style
-# sheet that is used by each HTML page. It can be used to fine-tune the look of
-# the HTML output. If left blank doxygen will generate a default style sheet.
-# See also section "Doxygen usage" for information on how to generate the style
-# sheet that doxygen normally uses.
-# Note: It is recommended to use HTML_EXTRA_STYLESHEET instead of this tag, as
-# it is more robust and this tag (HTML_STYLESHEET) will in the future become
-# obsolete.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_STYLESHEET        =
-
-# The HTML_EXTRA_STYLESHEET tag can be used to specify additional user-defined
-# cascading style sheets that are included after the standard style sheets
-# created by doxygen. Using this option one can overrule certain style aspects.
-# This is preferred over using HTML_STYLESHEET since it does not replace the
-# standard style sheet and is therefore more robust against future updates.
-# Doxygen will copy the style sheet files to the output directory.
-# Note: The order of the extra style sheet files is of importance (e.g. the last
-# style sheet in the list overrules the setting of the previous ones in the
-# list). For an example see the documentation.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_EXTRA_STYLESHEET  =
-
-# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or
-# other source files which should be copied to the HTML output directory. Note
-# that these files will be copied to the base HTML output directory. Use the
-# $relpath^ marker in the HTML_HEADER and/or HTML_FOOTER files to load these
-# files. In the HTML_STYLESHEET file, use the file name only. Also note that the
-# files will be copied as-is; there are no commands or markers available.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_EXTRA_FILES       =
-
-# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen
-# will adjust the colors in the style sheet and background images according to
-# this color. Hue is specified as an angle on a colorwheel, see
-# http://en.wikipedia.org/wiki/Hue for more information. For instance the value
-# 0 represents red, 60 is yellow, 120 is green, 180 is cyan, 240 is blue, 300
-# purple, and 360 is red again.
-# Minimum value: 0, maximum value: 359, default value: 220.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_COLORSTYLE_HUE    = 220
-
-# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors
-# in the HTML output. For a value of 0 the output will use grayscales only. A
-# value of 255 will produce the most vivid colors.
-# Minimum value: 0, maximum value: 255, default value: 100.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_COLORSTYLE_SAT    = 100
-
-# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the
-# luminance component of the colors in the HTML output. Values below 100
-# gradually make the output lighter, whereas values above 100 make the output
-# darker. The value divided by 100 is the actual gamma applied, so 80 represents
-# a gamma of 0.8, The value 220 represents a gamma of 2.2, and 100 does not
-# change the gamma.
-# Minimum value: 40, maximum value: 240, default value: 80.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_COLORSTYLE_GAMMA  = 80
-
-# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML
-# page will contain the date and time when the page was generated. Setting this
-# to YES can help to show when doxygen was last run and thus if the
-# documentation is up to date.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_TIMESTAMP         = NO
-
-# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML
-# documentation will contain sections that can be hidden and shown after the
-# page has loaded.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_DYNAMIC_SECTIONS  = NO
-
-# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries
-# shown in the various tree structured indices initially; the user can expand
-# and collapse entries dynamically later on. Doxygen will expand the tree to
-# such a level that at most the specified number of entries are visible (unless
-# a fully collapsed tree already exceeds this amount). So setting the number of
-# entries 1 will produce a full collapsed tree by default. 0 is a special value
-# representing an infinite number of entries and will result in a full expanded
-# tree by default.
-# Minimum value: 0, maximum value: 9999, default value: 100.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-HTML_INDEX_NUM_ENTRIES = 100
-
-# If the GENERATE_DOCSET tag is set to YES, additional index files will be
-# generated that can be used as input for Apple's Xcode 3 integrated development
-# environment (see: http://developer.apple.com/tools/xcode/), introduced with
-# OSX 10.5 (Leopard). To create a documentation set, doxygen will generate a
-# Makefile in the HTML output directory. Running make will produce the docset in
-# that directory and running make install will install the docset in
-# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find it at
-# startup. See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html
-# for more information.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-GENERATE_DOCSET        = NO
-
-# This tag determines the name of the docset feed. A documentation feed provides
-# an umbrella under which multiple documentation sets from a single provider
-# (such as a company or product suite) can be grouped.
-# The default value is: Doxygen generated docs.
-# This tag requires that the tag GENERATE_DOCSET is set to YES.
-
-DOCSET_FEEDNAME        = "Doxygen generated docs"
-
-# This tag specifies a string that should uniquely identify the documentation
-# set bundle. This should be a reverse domain-name style string, e.g.
-# com.mycompany.MyDocSet. Doxygen will append .docset to the name.
-# The default value is: org.doxygen.Project.
-# This tag requires that the tag GENERATE_DOCSET is set to YES.
-
-DOCSET_BUNDLE_ID       = org.doxygen.Project
-
-# The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify
-# the documentation publisher. This should be a reverse domain-name style
-# string, e.g. com.mycompany.MyDocSet.documentation.
-# The default value is: org.doxygen.Publisher.
-# This tag requires that the tag GENERATE_DOCSET is set to YES.
-
-DOCSET_PUBLISHER_ID    = org.doxygen.Publisher
-
-# The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher.
-# The default value is: Publisher.
-# This tag requires that the tag GENERATE_DOCSET is set to YES.
-
-DOCSET_PUBLISHER_NAME  = Publisher
-
-# If the GENERATE_HTMLHELP tag is set to YES then doxygen generates three
-# additional HTML index files: index.hhp, index.hhc, and index.hhk. The
-# index.hhp is a project file that can be read by Microsoft's HTML Help Workshop
-# (see: http://www.microsoft.com/en-us/download/details.aspx?id=21138) on
-# Windows.
-#
-# The HTML Help Workshop contains a compiler that can convert all HTML output
-# generated by doxygen into a single compiled HTML file (.chm). Compiled HTML
-# files are now used as the Windows 98 help format, and will replace the old
-# Windows help format (.hlp) on all Windows platforms in the future. Compressed
-# HTML files also contain an index, a table of contents, and you can search for
-# words in the documentation. The HTML workshop also contains a viewer for
-# compressed HTML files.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-GENERATE_HTMLHELP      = NO
-
-# The CHM_FILE tag can be used to specify the file name of the resulting .chm
-# file. You can add a path in front of the file if the result should not be
-# written to the html output directory.
-# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
-
-CHM_FILE               =
-
-# The HHC_LOCATION tag can be used to specify the location (absolute path
-# including file name) of the HTML help compiler (hhc.exe). If non-empty,
-# doxygen will try to run the HTML help compiler on the generated index.hhp.
-# The file has to be specified with full path.
-# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
-
-HHC_LOCATION           =
-
-# The GENERATE_CHI flag controls if a separate .chi index file is generated
-# (YES) or that it should be included in the master .chm file (NO).
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
-
-GENERATE_CHI           = NO
-
-# The CHM_INDEX_ENCODING is used to encode HtmlHelp index (hhk), content (hhc)
-# and project file content.
-# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
-
-CHM_INDEX_ENCODING     =
-
-# The BINARY_TOC flag controls whether a binary table of contents is generated
-# (YES) or a normal table of contents (NO) in the .chm file. Furthermore it
-# enables the Previous and Next buttons.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
-
-BINARY_TOC             = NO
-
-# The TOC_EXPAND flag can be set to YES to add extra items for group members to
-# the table of contents of the HTML help documentation and to the tree view.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTMLHELP is set to YES.
-
-TOC_EXPAND             = NO
-
-# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and
-# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated that
-# can be used as input for Qt's qhelpgenerator to generate a Qt Compressed Help
-# (.qch) of the generated HTML documentation.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-GENERATE_QHP           = NO
-
-# If the QHG_LOCATION tag is specified, the QCH_FILE tag can be used to specify
-# the file name of the resulting .qch file. The path specified is relative to
-# the HTML output folder.
-# This tag requires that the tag GENERATE_QHP is set to YES.
-
-QCH_FILE               =
-
-# The QHP_NAMESPACE tag specifies the namespace to use when generating Qt Help
-# Project output. For more information please see Qt Help Project / Namespace
-# (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#namespace).
-# The default value is: org.doxygen.Project.
-# This tag requires that the tag GENERATE_QHP is set to YES.
-
-QHP_NAMESPACE          = org.doxygen.Project
-
-# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating Qt
-# Help Project output. For more information please see Qt Help Project / Virtual
-# Folders (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#virtual-
-# folders).
-# The default value is: doc.
-# This tag requires that the tag GENERATE_QHP is set to YES.
-
-QHP_VIRTUAL_FOLDER     = doc
-
-# If the QHP_CUST_FILTER_NAME tag is set, it specifies the name of a custom
-# filter to add. For more information please see Qt Help Project / Custom
-# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-
-# filters).
-# This tag requires that the tag GENERATE_QHP is set to YES.
-
-QHP_CUST_FILTER_NAME   =
-
-# The QHP_CUST_FILTER_ATTRS tag specifies the list of the attributes of the
-# custom filter to add. For more information please see Qt Help Project / Custom
-# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-
-# filters).
-# This tag requires that the tag GENERATE_QHP is set to YES.
-
-QHP_CUST_FILTER_ATTRS  =
-
-# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this
-# project's filter section matches. Qt Help Project / Filter Attributes (see:
-# http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes).
-# This tag requires that the tag GENERATE_QHP is set to YES.
-
-QHP_SECT_FILTER_ATTRS  =
-
-# The QHG_LOCATION tag can be used to specify the location of Qt's
-# qhelpgenerator. If non-empty doxygen will try to run qhelpgenerator on the
-# generated .qhp file.
-# This tag requires that the tag GENERATE_QHP is set to YES.
-
-QHG_LOCATION           =
-
-# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files will be
-# generated, together with the HTML files, they form an Eclipse help plugin. To
-# install this plugin and make it available under the help contents menu in
-# Eclipse, the contents of the directory containing the HTML and XML files needs
-# to be copied into the plugins directory of eclipse. The name of the directory
-# within the plugins directory should be the same as the ECLIPSE_DOC_ID value.
-# After copying Eclipse needs to be restarted before the help appears.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-GENERATE_ECLIPSEHELP   = NO
-
-# A unique identifier for the Eclipse help plugin. When installing the plugin
-# the directory name containing the HTML and XML files should also have this
-# name. Each documentation set should have its own identifier.
-# The default value is: org.doxygen.Project.
-# This tag requires that the tag GENERATE_ECLIPSEHELP is set to YES.
-
-ECLIPSE_DOC_ID         = org.doxygen.Project
-
-# If you want full control over the layout of the generated HTML pages it might
-# be necessary to disable the index and replace it with your own. The
-# DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) at top
-# of each HTML page. A value of NO enables the index and the value YES disables
-# it. Since the tabs in the index contain the same information as the navigation
-# tree, you can set this option to YES if you also set GENERATE_TREEVIEW to YES.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-DISABLE_INDEX          = NO
-
-# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index
-# structure should be generated to display hierarchical information. If the tag
-# value is set to YES, a side panel will be generated containing a tree-like
-# index structure (just like the one that is generated for HTML Help). For this
-# to work a browser that supports JavaScript, DHTML, CSS and frames is required
-# (i.e. any modern browser). Windows users are probably better off using the
-# HTML help feature. Via custom style sheets (see HTML_EXTRA_STYLESHEET) one can
-# further fine-tune the look of the index. As an example, the default style
-# sheet generated by doxygen has an example that shows how to put an image at
-# the root of the tree instead of the PROJECT_NAME. Since the tree basically has
-# the same information as the tab index, you could consider setting
-# DISABLE_INDEX to YES when enabling this option.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-GENERATE_TREEVIEW      = NO
-
-# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that
-# doxygen will group on one line in the generated HTML documentation.
-#
-# Note that a value of 0 will completely suppress the enum values from appearing
-# in the overview section.
-# Minimum value: 0, maximum value: 20, default value: 4.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-ENUM_VALUES_PER_LINE   = 4
-
-# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used
-# to set the initial width (in pixels) of the frame in which the tree is shown.
-# Minimum value: 0, maximum value: 1500, default value: 250.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-TREEVIEW_WIDTH         = 250
-
-# If the EXT_LINKS_IN_WINDOW option is set to YES, doxygen will open links to
-# external symbols imported via tag files in a separate window.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-EXT_LINKS_IN_WINDOW    = NO
-
-# Use this tag to change the font size of LaTeX formulas included as images in
-# the HTML documentation. When you change the font size after a successful
-# doxygen run you need to manually remove any form_*.png images from the HTML
-# output directory to force them to be regenerated.
-# Minimum value: 8, maximum value: 50, default value: 10.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-FORMULA_FONTSIZE       = 10
-
-# Use the FORMULA_TRANPARENT tag to determine whether or not the images
-# generated for formulas are transparent PNGs. Transparent PNGs are not
-# supported properly for IE 6.0, but are supported on all modern browsers.
-#
-# Note that when changing this option you need to delete any form_*.png files in
-# the HTML output directory before the changes have effect.
-# The default value is: YES.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-FORMULA_TRANSPARENT    = YES
-
-# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax (see
-# http://www.mathjax.org) which uses client side Javascript for the rendering
-# instead of using pre-rendered bitmaps. Use this if you do not have LaTeX
-# installed or if you want to formulas look prettier in the HTML output. When
-# enabled you may also need to install MathJax separately and configure the path
-# to it using the MATHJAX_RELPATH option.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-USE_MATHJAX            = YES
-
-# When MathJax is enabled you can set the default output format to be used for
-# the MathJax output. See the MathJax site (see:
-# http://docs.mathjax.org/en/latest/output.html) for more details.
-# Possible values are: HTML-CSS (which is slower, but has the best
-# compatibility), NativeMML (i.e. MathML) and SVG.
-# The default value is: HTML-CSS.
-# This tag requires that the tag USE_MATHJAX is set to YES.
-
-MATHJAX_FORMAT         = HTML-CSS
-
-# When MathJax is enabled you need to specify the location relative to the HTML
-# output directory using the MATHJAX_RELPATH option. The destination directory
-# should contain the MathJax.js script. For instance, if the mathjax directory
-# is located at the same level as the HTML output directory, then
-# MATHJAX_RELPATH should be ../mathjax. The default value points to the MathJax
-# Content Delivery Network so you can quickly see the result without installing
-# MathJax. However, it is strongly recommended to install a local copy of
-# MathJax from http://www.mathjax.org before deployment.
-# The default value is: http://cdn.mathjax.org/mathjax/latest.
-# This tag requires that the tag USE_MATHJAX is set to YES.
-
-MATHJAX_RELPATH        = https://cdn.jsdelivr.net/npm/mathjax@3
-
-# The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax
-# extension names that should be enabled during MathJax rendering. For example
-# MATHJAX_EXTENSIONS = TeX/AMSmath TeX/AMSsymbols
-# This tag requires that the tag USE_MATHJAX is set to YES.
-
-MATHJAX_EXTENSIONS     =
-
-# The MATHJAX_CODEFILE tag can be used to specify a file with javascript pieces
-# of code that will be used on startup of the MathJax code. See the MathJax site
-# (see: http://docs.mathjax.org/en/latest/output.html) for more details. For an
-# example see the documentation.
-# This tag requires that the tag USE_MATHJAX is set to YES.
-
-MATHJAX_CODEFILE       =
-
-# When the SEARCHENGINE tag is enabled doxygen will generate a search box for
-# the HTML output. The underlying search engine uses javascript and DHTML and
-# should work on any modern browser. Note that when using HTML help
-# (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets (GENERATE_DOCSET)
-# there is already a search function so this one should typically be disabled.
-# For large projects the javascript based search engine can be slow, then
-# enabling SERVER_BASED_SEARCH may provide a better solution. It is possible to
-# search using the keyboard; to jump to the search box use <access key> + S
-# (what the <access key> is depends on the OS and browser, but it is typically
-# <CTRL>, <ALT>/<option>, or both). Inside the search box use the <cursor down
-# key> to jump into the search results window, the results can be navigated
-# using the <cursor keys>. Press <Enter> to select an item or <escape> to cancel
-# the search. The filter options can be selected when the cursor is inside the
-# search box by pressing <Shift>+<cursor down>. Also here use the <cursor keys>
-# to select a filter and <Enter> or <escape> to activate or cancel the filter
-# option.
-# The default value is: YES.
-# This tag requires that the tag GENERATE_HTML is set to YES.
-
-SEARCHENGINE           = YES
-
-# When the SERVER_BASED_SEARCH tag is enabled the search engine will be
-# implemented using a web server instead of a web client using Javascript. There
-# are two flavors of web server based searching depending on the EXTERNAL_SEARCH
-# setting. When disabled, doxygen will generate a PHP script for searching and
-# an index file used by the script. When EXTERNAL_SEARCH is enabled the indexing
-# and searching needs to be provided by external tools. See the section
-# "External Indexing and Searching" for details.
-# The default value is: NO.
-# This tag requires that the tag SEARCHENGINE is set to YES.
-
-SERVER_BASED_SEARCH    = NO
-
-# When EXTERNAL_SEARCH tag is enabled doxygen will no longer generate the PHP
-# script for searching. Instead the search results are written to an XML file
-# which needs to be processed by an external indexer. Doxygen will invoke an
-# external search engine pointed to by the SEARCHENGINE_URL option to obtain the
-# search results.
-#
-# Doxygen ships with an example indexer (doxyindexer) and search engine
-# (doxysearch.cgi) which are based on the open source search engine library
-# Xapian (see: http://xapian.org/).
-#
-# See the section "External Indexing and Searching" for details.
-# The default value is: NO.
-# This tag requires that the tag SEARCHENGINE is set to YES.
-
-EXTERNAL_SEARCH        = NO
-
-# The SEARCHENGINE_URL should point to a search engine hosted by a web server
-# which will return the search results when EXTERNAL_SEARCH is enabled.
-#
-# Doxygen ships with an example indexer (doxyindexer) and search engine
-# (doxysearch.cgi) which are based on the open source search engine library
-# Xapian (see: http://xapian.org/). See the section "External Indexing and
-# Searching" for details.
-# This tag requires that the tag SEARCHENGINE is set to YES.
-
-SEARCHENGINE_URL       =
-
-# When SERVER_BASED_SEARCH and EXTERNAL_SEARCH are both enabled the unindexed
-# search data is written to a file for indexing by an external tool. With the
-# SEARCHDATA_FILE tag the name of this file can be specified.
-# The default file is: searchdata.xml.
-# This tag requires that the tag SEARCHENGINE is set to YES.
-
-SEARCHDATA_FILE        = searchdata.xml
-
-# When SERVER_BASED_SEARCH and EXTERNAL_SEARCH are both enabled the
-# EXTERNAL_SEARCH_ID tag can be used as an identifier for the project. This is
-# useful in combination with EXTRA_SEARCH_MAPPINGS to search through multiple
-# projects and redirect the results back to the right project.
-# This tag requires that the tag SEARCHENGINE is set to YES.
-
-EXTERNAL_SEARCH_ID     =
-
-# The EXTRA_SEARCH_MAPPINGS tag can be used to enable searching through doxygen
-# projects other than the one defined by this configuration file, but that are
-# all added to the same external search index. Each project needs to have a
-# unique id set via EXTERNAL_SEARCH_ID. The search mapping then maps the id of
-# to a relative location where the documentation can be found. The format is:
-# EXTRA_SEARCH_MAPPINGS = tagname1=loc1 tagname2=loc2 ...
-# This tag requires that the tag SEARCHENGINE is set to YES.
-
-EXTRA_SEARCH_MAPPINGS  =
-
-#---------------------------------------------------------------------------
-# Configuration options related to the LaTeX output
-#---------------------------------------------------------------------------
-
-# If the GENERATE_LATEX tag is set to YES, doxygen will generate LaTeX output.
-# The default value is: YES.
-
-GENERATE_LATEX         = YES
-
-# The LATEX_OUTPUT tag is used to specify where the LaTeX docs will be put. If a
-# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
-# it.
-# The default directory is: latex.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_OUTPUT           = latex
-
-# The LATEX_CMD_NAME tag can be used to specify the LaTeX command name to be
-# invoked.
-#
-# Note that when enabling USE_PDFLATEX this option is only used for generating
-# bitmaps for formulas in the HTML output, but not in the Makefile that is
-# written to the output directory.
-# The default file is: latex.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_CMD_NAME         = latex
-
-# The MAKEINDEX_CMD_NAME tag can be used to specify the command name to generate
-# index for LaTeX.
-# The default file is: makeindex.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-MAKEINDEX_CMD_NAME     = makeindex
-
-# If the COMPACT_LATEX tag is set to YES, doxygen generates more compact LaTeX
-# documents. This may be useful for small projects and may help to save some
-# trees in general.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-COMPACT_LATEX          = NO
-
-# The PAPER_TYPE tag can be used to set the paper type that is used by the
-# printer.
-# Possible values are: a4 (210 x 297 mm), letter (8.5 x 11 inches), legal (8.5 x
-# 14 inches) and executive (7.25 x 10.5 inches).
-# The default value is: a4.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-PAPER_TYPE             = a4
-
-# The EXTRA_PACKAGES tag can be used to specify one or more LaTeX package names
-# that should be included in the LaTeX output. The package can be specified just
-# by its name or with the correct syntax as to be used with the LaTeX
-# \usepackage command. To get the times font for instance you can specify :
-# EXTRA_PACKAGES=times or EXTRA_PACKAGES={times}
-# To use the option intlimits with the amsmath package you can specify:
-# EXTRA_PACKAGES=[intlimits]{amsmath}
-# If left blank no extra packages will be included.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-EXTRA_PACKAGES         =
-
-# The LATEX_HEADER tag can be used to specify a personal LaTeX header for the
-# generated LaTeX document. The header should contain everything until the first
-# chapter. If it is left blank doxygen will generate a standard header. See
-# section "Doxygen usage" for information on how to let doxygen write the
-# default header to a separate file.
-#
-# Note: Only use a user-defined header if you know what you are doing! The
-# following commands have a special meaning inside the header: $title,
-# $datetime, $date, $doxygenversion, $projectname, $projectnumber,
-# $projectbrief, $projectlogo. Doxygen will replace $title with the empty
-# string, for the replacement values of the other commands the user is referred
-# to HTML_HEADER.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_HEADER           =
-
-# The LATEX_FOOTER tag can be used to specify a personal LaTeX footer for the
-# generated LaTeX document. The footer should contain everything after the last
-# chapter. If it is left blank doxygen will generate a standard footer. See
-# LATEX_HEADER for more information on how to generate a default footer and what
-# special commands can be used inside the footer.
-#
-# Note: Only use a user-defined footer if you know what you are doing!
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_FOOTER           =
-
-# The LATEX_EXTRA_STYLESHEET tag can be used to specify additional user-defined
-# LaTeX style sheets that are included after the standard style sheets created
-# by doxygen. Using this option one can overrule certain style aspects. Doxygen
-# will copy the style sheet files to the output directory.
-# Note: The order of the extra style sheet files is of importance (e.g. the last
-# style sheet in the list overrules the setting of the previous ones in the
-# list).
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_EXTRA_STYLESHEET =
-
-# The LATEX_EXTRA_FILES tag can be used to specify one or more extra images or
-# other source files which should be copied to the LATEX_OUTPUT output
-# directory. Note that the files will be copied as-is; there are no commands or
-# markers available.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_EXTRA_FILES      =
-
-# If the PDF_HYPERLINKS tag is set to YES, the LaTeX that is generated is
-# prepared for conversion to PDF (using ps2pdf or pdflatex). The PDF file will
-# contain links (just like the HTML output) instead of page references. This
-# makes the output suitable for online browsing using a PDF viewer.
-# The default value is: YES.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-PDF_HYPERLINKS         = YES
-
-# If the USE_PDFLATEX tag is set to YES, doxygen will use pdflatex to generate
-# the PDF file directly from the LaTeX files. Set this option to YES, to get a
-# higher quality PDF documentation.
-# The default value is: YES.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-USE_PDFLATEX           = YES
-
-# If the LATEX_BATCHMODE tag is set to YES, doxygen will add the \batchmode
-# command to the generated LaTeX files. This will instruct LaTeX to keep running
-# if errors occur, instead of asking the user for help. This option is also used
-# when generating formulas in HTML.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_BATCHMODE        = NO
-
-# If the LATEX_HIDE_INDICES tag is set to YES then doxygen will not include the
-# index chapters (such as File Index, Compound Index, etc.) in the output.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_HIDE_INDICES     = NO
-
-# If the LATEX_SOURCE_CODE tag is set to YES then doxygen will include source
-# code with syntax highlighting in the LaTeX output.
-#
-# Note that which sources are shown also depends on other settings such as
-# SOURCE_BROWSER.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_SOURCE_CODE      = NO
-
-# The LATEX_BIB_STYLE tag can be used to specify the style to use for the
-# bibliography, e.g. plainnat, or ieeetr. See
-# http://en.wikipedia.org/wiki/BibTeX and \cite for more info.
-# The default value is: plain.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_BIB_STYLE        = plain
-
-# If the LATEX_TIMESTAMP tag is set to YES then the footer of each generated
-# page will contain the date and time when the page was generated. Setting this
-# to NO can help when comparing the output of multiple runs.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_LATEX is set to YES.
-
-LATEX_TIMESTAMP        = NO
-
-#---------------------------------------------------------------------------
-# Configuration options related to the RTF output
-#---------------------------------------------------------------------------
-
-# If the GENERATE_RTF tag is set to YES, doxygen will generate RTF output. The
-# RTF output is optimized for Word 97 and may not look too pretty with other RTF
-# readers/editors.
-# The default value is: NO.
-
-GENERATE_RTF           = NO
-
-# The RTF_OUTPUT tag is used to specify where the RTF docs will be put. If a
-# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
-# it.
-# The default directory is: rtf.
-# This tag requires that the tag GENERATE_RTF is set to YES.
-
-RTF_OUTPUT             = rtf
-
-# If the COMPACT_RTF tag is set to YES, doxygen generates more compact RTF
-# documents. This may be useful for small projects and may help to save some
-# trees in general.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_RTF is set to YES.
-
-COMPACT_RTF            = NO
-
-# If the RTF_HYPERLINKS tag is set to YES, the RTF that is generated will
-# contain hyperlink fields. The RTF file will contain links (just like the HTML
-# output) instead of page references. This makes the output suitable for online
-# browsing using Word or some other Word compatible readers that support those
-# fields.
-#
-# Note: WordPad (write) and others do not support links.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_RTF is set to YES.
-
-RTF_HYPERLINKS         = NO
-
-# Load stylesheet definitions from file. Syntax is similar to doxygen's config
-# file, i.e. a series of assignments. You only have to provide replacements,
-# missing definitions are set to their default value.
-#
-# See also section "Doxygen usage" for information on how to generate the
-# default style sheet that doxygen normally uses.
-# This tag requires that the tag GENERATE_RTF is set to YES.
-
-RTF_STYLESHEET_FILE    =
-
-# Set optional variables used in the generation of an RTF document. Syntax is
-# similar to doxygen's config file. A template extensions file can be generated
-# using doxygen -e rtf extensionFile.
-# This tag requires that the tag GENERATE_RTF is set to YES.
-
-RTF_EXTENSIONS_FILE    =
-
-# If the RTF_SOURCE_CODE tag is set to YES then doxygen will include source code
-# with syntax highlighting in the RTF output.
-#
-# Note that which sources are shown also depends on other settings such as
-# SOURCE_BROWSER.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_RTF is set to YES.
-
-RTF_SOURCE_CODE        = NO
-
-#---------------------------------------------------------------------------
-# Configuration options related to the man page output
-#---------------------------------------------------------------------------
-
-# If the GENERATE_MAN tag is set to YES, doxygen will generate man pages for
-# classes and files.
-# The default value is: NO.
-
-GENERATE_MAN           = NO
-
-# The MAN_OUTPUT tag is used to specify where the man pages will be put. If a
-# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
-# it. A directory man3 will be created inside the directory specified by
-# MAN_OUTPUT.
-# The default directory is: man.
-# This tag requires that the tag GENERATE_MAN is set to YES.
-
-MAN_OUTPUT             = man
-
-# The MAN_EXTENSION tag determines the extension that is added to the generated
-# man pages. In case the manual section does not start with a number, the number
-# 3 is prepended. The dot (.) at the beginning of the MAN_EXTENSION tag is
-# optional.
-# The default value is: .3.
-# This tag requires that the tag GENERATE_MAN is set to YES.
-
-MAN_EXTENSION          = .3
-
-# The MAN_SUBDIR tag determines the name of the directory created within
-# MAN_OUTPUT in which the man pages are placed. If defaults to man followed by
-# MAN_EXTENSION with the initial . removed.
-# This tag requires that the tag GENERATE_MAN is set to YES.
-
-MAN_SUBDIR             =
-
-# If the MAN_LINKS tag is set to YES and doxygen generates man output, then it
-# will generate one additional man file for each entity documented in the real
-# man page(s). These additional files only source the real man page, but without
-# them the man command would be unable to find the correct page.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_MAN is set to YES.
-
-MAN_LINKS              = NO
-
-#---------------------------------------------------------------------------
-# Configuration options related to the XML output
-#---------------------------------------------------------------------------
-
-# If the GENERATE_XML tag is set to YES, doxygen will generate an XML file that
-# captures the structure of the code including all documentation.
-# The default value is: NO.
-
-GENERATE_XML           = NO
-
-# The XML_OUTPUT tag is used to specify where the XML pages will be put. If a
-# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
-# it.
-# The default directory is: xml.
-# This tag requires that the tag GENERATE_XML is set to YES.
-
-XML_OUTPUT             = xml
-
-# If the XML_PROGRAMLISTING tag is set to YES, doxygen will dump the program
-# listings (including syntax highlighting and cross-referencing information) to
-# the XML output. Note that enabling this will significantly increase the size
-# of the XML output.
-# The default value is: YES.
-# This tag requires that the tag GENERATE_XML is set to YES.
-
-XML_PROGRAMLISTING     = YES
-
-#---------------------------------------------------------------------------
-# Configuration options related to the DOCBOOK output
-#---------------------------------------------------------------------------
-
-# If the GENERATE_DOCBOOK tag is set to YES, doxygen will generate Docbook files
-# that can be used to generate PDF.
-# The default value is: NO.
-
-GENERATE_DOCBOOK       = NO
-
-# The DOCBOOK_OUTPUT tag is used to specify where the Docbook pages will be put.
-# If a relative path is entered the value of OUTPUT_DIRECTORY will be put in
-# front of it.
-# The default directory is: docbook.
-# This tag requires that the tag GENERATE_DOCBOOK is set to YES.
-
-DOCBOOK_OUTPUT         = docbook
-
-# If the DOCBOOK_PROGRAMLISTING tag is set to YES, doxygen will include the
-# program listings (including syntax highlighting and cross-referencing
-# information) to the DOCBOOK output. Note that enabling this will significantly
-# increase the size of the DOCBOOK output.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_DOCBOOK is set to YES.
-
-DOCBOOK_PROGRAMLISTING = NO
-
-#---------------------------------------------------------------------------
-# Configuration options for the AutoGen Definitions output
-#---------------------------------------------------------------------------
-
-# If the GENERATE_AUTOGEN_DEF tag is set to YES, doxygen will generate an
-# AutoGen Definitions (see http://autogen.sf.net) file that captures the
-# structure of the code including all documentation. Note that this feature is
-# still experimental and incomplete at the moment.
-# The default value is: NO.
-
-GENERATE_AUTOGEN_DEF   = NO
-
-#---------------------------------------------------------------------------
-# Configuration options related to the Perl module output
-#---------------------------------------------------------------------------
-
-# If the GENERATE_PERLMOD tag is set to YES, doxygen will generate a Perl module
-# file that captures the structure of the code including all documentation.
-#
-# Note that this feature is still experimental and incomplete at the moment.
-# The default value is: NO.
-
-GENERATE_PERLMOD       = NO
-
-# If the PERLMOD_LATEX tag is set to YES, doxygen will generate the necessary
-# Makefile rules, Perl scripts and LaTeX code to be able to generate PDF and DVI
-# output from the Perl module output.
-# The default value is: NO.
-# This tag requires that the tag GENERATE_PERLMOD is set to YES.
-
-PERLMOD_LATEX          = NO
-
-# If the PERLMOD_PRETTY tag is set to YES, the Perl module output will be nicely
-# formatted so it can be parsed by a human reader. This is useful if you want to
-# understand what is going on. On the other hand, if this tag is set to NO, the
-# size of the Perl module output will be much smaller and Perl will parse it
-# just the same.
-# The default value is: YES.
-# This tag requires that the tag GENERATE_PERLMOD is set to YES.
-
-PERLMOD_PRETTY         = YES
-
-# The names of the make variables in the generated doxyrules.make file are
-# prefixed with the string contained in PERLMOD_MAKEVAR_PREFIX. This is useful
-# so different doxyrules.make files included by the same Makefile don't
-# overwrite each other's variables.
-# This tag requires that the tag GENERATE_PERLMOD is set to YES.
-
-PERLMOD_MAKEVAR_PREFIX =
-
-#---------------------------------------------------------------------------
-# Configuration options related to the preprocessor
-#---------------------------------------------------------------------------
-
-# If the ENABLE_PREPROCESSING tag is set to YES, doxygen will evaluate all
-# C-preprocessor directives found in the sources and include files.
-# The default value is: YES.
-
-ENABLE_PREPROCESSING   = YES
-
-# If the MACRO_EXPANSION tag is set to YES, doxygen will expand all macro names
-# in the source code. If set to NO, only conditional compilation will be
-# performed. Macro expansion can be done in a controlled way by setting
-# EXPAND_ONLY_PREDEF to YES.
-# The default value is: NO.
-# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
-
-MACRO_EXPANSION        = NO
-
-# If the EXPAND_ONLY_PREDEF and MACRO_EXPANSION tags are both set to YES then
-# the macro expansion is limited to the macros specified with the PREDEFINED and
-# EXPAND_AS_DEFINED tags.
-# The default value is: NO.
-# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
-
-EXPAND_ONLY_PREDEF     = NO
-
-# If the SEARCH_INCLUDES tag is set to YES, the include files in the
-# INCLUDE_PATH will be searched if a #include is found.
-# The default value is: YES.
-# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
-
-SEARCH_INCLUDES        = YES
-
-# The INCLUDE_PATH tag can be used to specify one or more directories that
-# contain include files that are not input files but should be processed by the
-# preprocessor.
-# This tag requires that the tag SEARCH_INCLUDES is set to YES.
-
-INCLUDE_PATH           =
-
-# You can use the INCLUDE_FILE_PATTERNS tag to specify one or more wildcard
-# patterns (like *.h and *.hpp) to filter out the header-files in the
-# directories. If left blank, the patterns specified with FILE_PATTERNS will be
-# used.
-# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
-
-INCLUDE_FILE_PATTERNS  =
-
-# The PREDEFINED tag can be used to specify one or more macro names that are
-# defined before the preprocessor is started (similar to the -D option of e.g.
-# gcc). The argument of the tag is a list of macros of the form: name or
-# name=definition (no spaces). If the definition and the "=" are omitted, "=1"
-# is assumed. To prevent a macro definition from being undefined via #undef or
-# recursively expanded use the := operator instead of the = operator.
-# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
-
-PREDEFINED             =
-
-# If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then this
-# tag can be used to specify a list of macro names that should be expanded. The
-# macro definition that is found in the sources will be used. Use the PREDEFINED
-# tag if you want to use a different macro definition that overrules the
-# definition found in the source code.
-# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
-
-EXPAND_AS_DEFINED      =
-
-# If the SKIP_FUNCTION_MACROS tag is set to YES then doxygen's preprocessor will
-# remove all references to function-like macros that are alone on a line, have
-# an all uppercase name, and do not end with a semicolon. Such function macros
-# are typically used for boiler-plate code, and will confuse the parser if not
-# removed.
-# The default value is: YES.
-# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
-
-SKIP_FUNCTION_MACROS   = YES
-
-#---------------------------------------------------------------------------
-# Configuration options related to external references
-#---------------------------------------------------------------------------
-
-# The TAGFILES tag can be used to specify one or more tag files. For each tag
-# file the location of the external documentation should be added. The format of
-# a tag file without this location is as follows:
-# TAGFILES = file1 file2 ...
-# Adding location for the tag files is done as follows:
-# TAGFILES = file1=loc1 "file2 = loc2" ...
-# where loc1 and loc2 can be relative or absolute paths or URLs. See the
-# section "Linking to external documentation" for more information about the use
-# of tag files.
-# Note: Each tag file must have a unique name (where the name does NOT include
-# the path). If a tag file is not located in the directory in which doxygen is
-# run, you must also specify the path to the tagfile here.
-
-TAGFILES               =
-
-# When a file name is specified after GENERATE_TAGFILE, doxygen will create a
-# tag file that is based on the input files it reads. See section "Linking to
-# external documentation" for more information about the usage of tag files.
-
-GENERATE_TAGFILE       =
-
-# If the ALLEXTERNALS tag is set to YES, all external class will be listed in
-# the class index. If set to NO, only the inherited external classes will be
-# listed.
-# The default value is: NO.
-
-ALLEXTERNALS           = NO
-
-# If the EXTERNAL_GROUPS tag is set to YES, all external groups will be listed
-# in the modules index. If set to NO, only the current project's groups will be
-# listed.
-# The default value is: YES.
-
-EXTERNAL_GROUPS        = YES
-
-# If the EXTERNAL_PAGES tag is set to YES, all external pages will be listed in
-# the related pages index. If set to NO, only the current project's pages will
-# be listed.
-# The default value is: YES.
-
-EXTERNAL_PAGES         = YES
-
-# The PERL_PATH should be the absolute path and name of the perl script
-# interpreter (i.e. the result of 'which perl').
-# The default file (with absolute path) is: /usr/bin/perl.
-
-PERL_PATH              = /usr/bin/perl
-
-#---------------------------------------------------------------------------
-# Configuration options related to the dot tool
-#---------------------------------------------------------------------------
-
-# If the CLASS_DIAGRAMS tag is set to YES, doxygen will generate a class diagram
-# (in HTML and LaTeX) for classes with base or super classes. Setting the tag to
-# NO turns the diagrams off. Note that this option also works with HAVE_DOT
-# disabled, but it is recommended to install and use dot, since it yields more
-# powerful graphs.
-# The default value is: YES.
-
-CLASS_DIAGRAMS         = YES
-
-# You can define message sequence charts within doxygen comments using the \msc
-# command. Doxygen will then run the mscgen tool (see:
-# http://www.mcternan.me.uk/mscgen/)) to produce the chart and insert it in the
-# documentation. The MSCGEN_PATH tag allows you to specify the directory where
-# the mscgen tool resides. If left empty the tool is assumed to be found in the
-# default search path.
-
-MSCGEN_PATH            =
-
-# You can include diagrams made with dia in doxygen documentation. Doxygen will
-# then run dia to produce the diagram and insert it in the documentation. The
-# DIA_PATH tag allows you to specify the directory where the dia binary resides.
-# If left empty dia is assumed to be found in the default search path.
-
-DIA_PATH               =
-
-# If set to YES the inheritance and collaboration graphs will hide inheritance
-# and usage relations if the target is undocumented or is not a class.
-# The default value is: YES.
-
-HIDE_UNDOC_RELATIONS   = YES
-
-# If you set the HAVE_DOT tag to YES then doxygen will assume the dot tool is
-# available from the path. This tool is part of Graphviz (see:
-# http://www.graphviz.org/), a graph visualization toolkit from AT&T and Lucent
-# Bell Labs. The other options in this section have no effect if this option is
-# set to NO
-# The default value is: YES.
-
-HAVE_DOT               = YES
-
-# The DOT_NUM_THREADS specifies the number of dot invocations doxygen is allowed
-# to run in parallel. When set to 0 doxygen will base this on the number of
-# processors available in the system. You can set it explicitly to a value
-# larger than 0 to get control over the balance between CPU load and processing
-# speed.
-# Minimum value: 0, maximum value: 32, default value: 0.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_NUM_THREADS        = 0
-
-# When you want a differently looking font in the dot files that doxygen
-# generates you can specify the font name using DOT_FONTNAME. You need to make
-# sure dot is able to find the font, which can be done by putting it in a
-# standard location or by setting the DOTFONTPATH environment variable or by
-# setting DOT_FONTPATH to the directory containing the font.
-# The default value is: Helvetica.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_FONTNAME           = Helvetica
-
-# The DOT_FONTSIZE tag can be used to set the size (in points) of the font of
-# dot graphs.
-# Minimum value: 4, maximum value: 24, default value: 10.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_FONTSIZE           = 10
-
-# By default doxygen will tell dot to use the default font as specified with
-# DOT_FONTNAME. If you specify a different font using DOT_FONTNAME you can set
-# the path where dot can find it using this tag.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_FONTPATH           =
-
-# If the CLASS_GRAPH tag is set to YES then doxygen will generate a graph for
-# each documented class showing the direct and indirect inheritance relations.
-# Setting this tag to YES will force the CLASS_DIAGRAMS tag to NO.
-# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-CLASS_GRAPH            = YES
-
-# If the COLLABORATION_GRAPH tag is set to YES then doxygen will generate a
-# graph for each documented class showing the direct and indirect implementation
-# dependencies (inheritance, containment, and class references variables) of the
-# class with other documented classes.
-# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-COLLABORATION_GRAPH    = YES
-
-# If the GROUP_GRAPHS tag is set to YES then doxygen will generate a graph for
-# groups, showing the direct groups dependencies.
-# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-GROUP_GRAPHS           = YES
-
-# If the UML_LOOK tag is set to YES, doxygen will generate inheritance and
-# collaboration diagrams in a style similar to the OMG's Unified Modeling
-# Language.
-# The default value is: NO.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-UML_LOOK               = NO
-
-# If the UML_LOOK tag is enabled, the fields and methods are shown inside the
-# class node. If there are many fields or methods and many nodes the graph may
-# become too big to be useful. The UML_LIMIT_NUM_FIELDS threshold limits the
-# number of items for each type to make the size more manageable. Set this to 0
-# for no limit. Note that the threshold may be exceeded by 50% before the limit
-# is enforced. So when you set the threshold to 10, up to 15 fields may appear,
-# but if the number exceeds 15, the total amount of fields shown is limited to
-# 10.
-# Minimum value: 0, maximum value: 100, default value: 10.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-UML_LIMIT_NUM_FIELDS   = 10
-
-# If the TEMPLATE_RELATIONS tag is set to YES then the inheritance and
-# collaboration graphs will show the relations between templates and their
-# instances.
-# The default value is: NO.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-TEMPLATE_RELATIONS     = NO
-
-# If the INCLUDE_GRAPH, ENABLE_PREPROCESSING and SEARCH_INCLUDES tags are set to
-# YES then doxygen will generate a graph for each documented file showing the
-# direct and indirect include dependencies of the file with other documented
-# files.
-# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-INCLUDE_GRAPH          = YES
-
-# If the INCLUDED_BY_GRAPH, ENABLE_PREPROCESSING and SEARCH_INCLUDES tags are
-# set to YES then doxygen will generate a graph for each documented file showing
-# the direct and indirect include dependencies of the file with other documented
-# files.
-# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-INCLUDED_BY_GRAPH      = YES
-
-# If the CALL_GRAPH tag is set to YES then doxygen will generate a call
-# dependency graph for every global function or class method.
-#
-# Note that enabling this option will significantly increase the time of a run.
-# So in most cases it will be better to enable call graphs for selected
-# functions only using the \callgraph command. Disabling a call graph can be
-# accomplished by means of the command \hidecallgraph.
-# The default value is: NO.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-CALL_GRAPH             = NO
-
-# If the CALLER_GRAPH tag is set to YES then doxygen will generate a caller
-# dependency graph for every global function or class method.
-#
-# Note that enabling this option will significantly increase the time of a run.
-# So in most cases it will be better to enable caller graphs for selected
-# functions only using the \callergraph command. Disabling a caller graph can be
-# accomplished by means of the command \hidecallergraph.
-# The default value is: NO.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-CALLER_GRAPH           = NO
-
-# If the GRAPHICAL_HIERARCHY tag is set to YES then doxygen will graphical
-# hierarchy of all classes instead of a textual one.
-# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-GRAPHICAL_HIERARCHY    = YES
-
-# If the DIRECTORY_GRAPH tag is set to YES then doxygen will show the
-# dependencies a directory has on other directories in a graphical way. The
-# dependency relations are determined by the #include relations between the
-# files in the directories.
-# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DIRECTORY_GRAPH        = YES
-
-# The DOT_IMAGE_FORMAT tag can be used to set the image format of the images
-# generated by dot. For an explanation of the image formats see the section
-# output formats in the documentation of the dot tool (Graphviz (see:
-# http://www.graphviz.org/)).
-# Note: If you choose svg you need to set HTML_FILE_EXTENSION to xhtml in order
-# to make the SVG files visible in IE 9+ (other browsers do not have this
-# requirement).
-# Possible values are: png, png:cairo, png:cairo:cairo, png:cairo:gd, png:gd,
-# png:gd:gd, jpg, jpg:cairo, jpg:cairo:gd, jpg:gd, jpg:gd:gd, gif, gif:cairo,
-# gif:cairo:gd, gif:gd, gif:gd:gd, svg, png:gd, png:gd:gd, png:cairo,
-# png:cairo:gd, png:cairo:cairo, png:cairo:gdiplus, png:gdiplus and
-# png:gdiplus:gdiplus.
-# The default value is: png.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_IMAGE_FORMAT       = png
-
-# If DOT_IMAGE_FORMAT is set to svg, then this option can be set to YES to
-# enable generation of interactive SVG images that allow zooming and panning.
-#
-# Note that this requires a modern browser other than Internet Explorer. Tested
-# and working are Firefox, Chrome, Safari, and Opera.
-# Note: For IE 9+ you need to set HTML_FILE_EXTENSION to xhtml in order to make
-# the SVG files visible. Older versions of IE do not have SVG support.
-# The default value is: NO.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-INTERACTIVE_SVG        = NO
-
-# The DOT_PATH tag can be used to specify the path where the dot tool can be
-# found. If left blank, it is assumed the dot tool can be found in the path.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_PATH               =
-
-# The DOTFILE_DIRS tag can be used to specify one or more directories that
-# contain dot files that are included in the documentation (see the \dotfile
-# command).
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOTFILE_DIRS           =
-
-# The MSCFILE_DIRS tag can be used to specify one or more directories that
-# contain msc files that are included in the documentation (see the \mscfile
-# command).
-
-MSCFILE_DIRS           =
-
-# The DIAFILE_DIRS tag can be used to specify one or more directories that
-# contain dia files that are included in the documentation (see the \diafile
-# command).
-
-DIAFILE_DIRS           =
-
-# When using plantuml, the PLANTUML_JAR_PATH tag should be used to specify the
-# path where java can find the plantuml.jar file. If left blank, it is assumed
-# PlantUML is not used or called during a preprocessing step. Doxygen will
-# generate a warning when it encounters a \startuml command in this case and
-# will not generate output for the diagram.
-
-PLANTUML_JAR_PATH      =
-
-# When using plantuml, the PLANTUML_CFG_FILE tag can be used to specify a
-# configuration file for plantuml.
-
-PLANTUML_CFG_FILE      =
-
-# When using plantuml, the specified paths are searched for files specified by
-# the !include statement in a plantuml block.
-
-PLANTUML_INCLUDE_PATH  =
-
-# The DOT_GRAPH_MAX_NODES tag can be used to set the maximum number of nodes
-# that will be shown in the graph. If the number of nodes in a graph becomes
-# larger than this value, doxygen will truncate the graph, which is visualized
-# by representing a node as a red box. Note that doxygen if the number of direct
-# children of the root node in a graph is already larger than
-# DOT_GRAPH_MAX_NODES then the graph will not be shown at all. Also note that
-# the size of a graph can be further restricted by MAX_DOT_GRAPH_DEPTH.
-# Minimum value: 0, maximum value: 10000, default value: 50.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_GRAPH_MAX_NODES    = 50
-
-# The MAX_DOT_GRAPH_DEPTH tag can be used to set the maximum depth of the graphs
-# generated by dot. A depth value of 3 means that only nodes reachable from the
-# root by following a path via at most 3 edges will be shown. Nodes that lay
-# further from the root node will be omitted. Note that setting this option to 1
-# or 2 may greatly reduce the computation time needed for large code bases. Also
-# note that the size of a graph can be further restricted by
-# DOT_GRAPH_MAX_NODES. Using a depth of 0 means no depth restriction.
-# Minimum value: 0, maximum value: 1000, default value: 0.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-MAX_DOT_GRAPH_DEPTH    = 0
-
-# Set the DOT_TRANSPARENT tag to YES to generate images with a transparent
-# background. This is disabled by default, because dot on Windows does not seem
-# to support this out of the box.
-#
-# Warning: Depending on the platform used, enabling this option may lead to
-# badly anti-aliased labels on the edges of a graph (i.e. they become hard to
-# read).
-# The default value is: NO.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_TRANSPARENT        = NO
-
-# Set the DOT_MULTI_TARGETS tag to YES to allow dot to generate multiple output
-# files in one run (i.e. multiple -o and -T options on the command line). This
-# makes dot run faster, but since only newer versions of dot (>1.8.10) support
-# this, this feature is disabled by default.
-# The default value is: NO.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_MULTI_TARGETS      = NO
-
-# If the GENERATE_LEGEND tag is set to YES doxygen will generate a legend page
-# explaining the meaning of the various boxes and arrows in the dot generated
-# graphs.
-# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-GENERATE_LEGEND        = YES
-
-# If the DOT_CLEANUP tag is set to YES, doxygen will remove the intermediate dot
-# files that are used to generate the various graphs.
-# The default value is: YES.
-# This tag requires that the tag HAVE_DOT is set to YES.
-
-DOT_CLEANUP            = YES

From bdbb68a34d5b717fbdf5ea1e2f5dc30671197247 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Tue, 1 Oct 2024 08:57:01 -0500
Subject: [PATCH 440/442] Remove inference/MANIFEST.in

---
 inference/MANIFEST.in | 2 --
 1 file changed, 2 deletions(-)
 delete mode 100644 inference/MANIFEST.in

diff --git a/inference/MANIFEST.in b/inference/MANIFEST.in
deleted file mode 100644
index 009fd4e31..000000000
--- a/inference/MANIFEST.in
+++ /dev/null
@@ -1,2 +0,0 @@
-include README.md LICENSE
-

From 3558f4fbd0e2366ff92aaecbf92fcd46a5eb8a2d Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Tue, 1 Oct 2024 08:57:14 -0500
Subject: [PATCH 441/442] Remove inference/LICENSE

The LICENSE file is exactly the same as the root-level LICENSE file.
---
 inference/LICENSE | 373 ----------------------------------------------
 1 file changed, 373 deletions(-)
 delete mode 100644 inference/LICENSE

diff --git a/inference/LICENSE b/inference/LICENSE
deleted file mode 100644
index a612ad981..000000000
--- a/inference/LICENSE
+++ /dev/null
@@ -1,373 +0,0 @@
-Mozilla Public License Version 2.0
-==================================
-
-1. Definitions
---------------
-
-1.1. "Contributor"
-    means each individual or legal entity that creates, contributes to
-    the creation of, or owns Covered Software.
-
-1.2. "Contributor Version"
-    means the combination of the Contributions of others (if any) used
-    by a Contributor and that particular Contributor's Contribution.
-
-1.3. "Contribution"
-    means Covered Software of a particular Contributor.
-
-1.4. "Covered Software"
-    means Source Code Form to which the initial Contributor has attached
-    the notice in Exhibit A, the Executable Form of such Source Code
-    Form, and Modifications of such Source Code Form, in each case
-    including portions thereof.
-
-1.5. "Incompatible With Secondary Licenses"
-    means
-
-    (a) that the initial Contributor has attached the notice described
-        in Exhibit B to the Covered Software; or
-
-    (b) that the Covered Software was made available under the terms of
-        version 1.1 or earlier of the License, but not also under the
-        terms of a Secondary License.
-
-1.6. "Executable Form"
-    means any form of the work other than Source Code Form.
-
-1.7. "Larger Work"
-    means a work that combines Covered Software with other material, in
-    a separate file or files, that is not Covered Software.
-
-1.8. "License"
-    means this document.
-
-1.9. "Licensable"
-    means having the right to grant, to the maximum extent possible,
-    whether at the time of the initial grant or subsequently, any and
-    all of the rights conveyed by this License.
-
-1.10. "Modifications"
-    means any of the following:
-
-    (a) any file in Source Code Form that results from an addition to,
-        deletion from, or modification of the contents of Covered
-        Software; or
-
-    (b) any new file in Source Code Form that contains any Covered
-        Software.
-
-1.11. "Patent Claims" of a Contributor
-    means any patent claim(s), including without limitation, method,
-    process, and apparatus claims, in any patent Licensable by such
-    Contributor that would be infringed, but for the grant of the
-    License, by the making, using, selling, offering for sale, having
-    made, import, or transfer of either its Contributions or its
-    Contributor Version.
-
-1.12. "Secondary License"
-    means either the GNU General Public License, Version 2.0, the GNU
-    Lesser General Public License, Version 2.1, the GNU Affero General
-    Public License, Version 3.0, or any later versions of those
-    licenses.
-
-1.13. "Source Code Form"
-    means the form of the work preferred for making modifications.
-
-1.14. "You" (or "Your")
-    means an individual or a legal entity exercising rights under this
-    License. For legal entities, "You" includes any entity that
-    controls, is controlled by, or is under common control with You. For
-    purposes of this definition, "control" means (a) the power, direct
-    or indirect, to cause the direction or management of such entity,
-    whether by contract or otherwise, or (b) ownership of more than
-    fifty percent (50%) of the outstanding shares or beneficial
-    ownership of such entity.
-
-2. License Grants and Conditions
---------------------------------
-
-2.1. Grants
-
-Each Contributor hereby grants You a world-wide, royalty-free,
-non-exclusive license:
-
-(a) under intellectual property rights (other than patent or trademark)
-    Licensable by such Contributor to use, reproduce, make available,
-    modify, display, perform, distribute, and otherwise exploit its
-    Contributions, either on an unmodified basis, with Modifications, or
-    as part of a Larger Work; and
-
-(b) under Patent Claims of such Contributor to make, use, sell, offer
-    for sale, have made, import, and otherwise transfer either its
-    Contributions or its Contributor Version.
-
-2.2. Effective Date
-
-The licenses granted in Section 2.1 with respect to any Contribution
-become effective for each Contribution on the date the Contributor first
-distributes such Contribution.
-
-2.3. Limitations on Grant Scope
-
-The licenses granted in this Section 2 are the only rights granted under
-this License. No additional rights or licenses will be implied from the
-distribution or licensing of Covered Software under this License.
-Notwithstanding Section 2.1(b) above, no patent license is granted by a
-Contributor:
-
-(a) for any code that a Contributor has removed from Covered Software;
-    or
-
-(b) for infringements caused by: (i) Your and any other third party's
-    modifications of Covered Software, or (ii) the combination of its
-    Contributions with other software (except as part of its Contributor
-    Version); or
-
-(c) under Patent Claims infringed by Covered Software in the absence of
-    its Contributions.
-
-This License does not grant any rights in the trademarks, service marks,
-or logos of any Contributor (except as may be necessary to comply with
-the notice requirements in Section 3.4).
-
-2.4. Subsequent Licenses
-
-No Contributor makes additional grants as a result of Your choice to
-distribute the Covered Software under a subsequent version of this
-License (see Section 10.2) or under the terms of a Secondary License (if
-permitted under the terms of Section 3.3).
-
-2.5. Representation
-
-Each Contributor represents that the Contributor believes its
-Contributions are its original creation(s) or it has sufficient rights
-to grant the rights to its Contributions conveyed by this License.
-
-2.6. Fair Use
-
-This License is not intended to limit any rights You have under
-applicable copyright doctrines of fair use, fair dealing, or other
-equivalents.
-
-2.7. Conditions
-
-Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
-in Section 2.1.
-
-3. Responsibilities
--------------------
-
-3.1. Distribution of Source Form
-
-All distribution of Covered Software in Source Code Form, including any
-Modifications that You create or to which You contribute, must be under
-the terms of this License. You must inform recipients that the Source
-Code Form of the Covered Software is governed by the terms of this
-License, and how they can obtain a copy of this License. You may not
-attempt to alter or restrict the recipients' rights in the Source Code
-Form.
-
-3.2. Distribution of Executable Form
-
-If You distribute Covered Software in Executable Form then:
-
-(a) such Covered Software must also be made available in Source Code
-    Form, as described in Section 3.1, and You must inform recipients of
-    the Executable Form how they can obtain a copy of such Source Code
-    Form by reasonable means in a timely manner, at a charge no more
-    than the cost of distribution to the recipient; and
-
-(b) You may distribute such Executable Form under the terms of this
-    License, or sublicense it under different terms, provided that the
-    license for the Executable Form does not attempt to limit or alter
-    the recipients' rights in the Source Code Form under this License.
-
-3.3. Distribution of a Larger Work
-
-You may create and distribute a Larger Work under terms of Your choice,
-provided that You also comply with the requirements of this License for
-the Covered Software. If the Larger Work is a combination of Covered
-Software with a work governed by one or more Secondary Licenses, and the
-Covered Software is not Incompatible With Secondary Licenses, this
-License permits You to additionally distribute such Covered Software
-under the terms of such Secondary License(s), so that the recipient of
-the Larger Work may, at their option, further distribute the Covered
-Software under the terms of either this License or such Secondary
-License(s).
-
-3.4. Notices
-
-You may not remove or alter the substance of any license notices
-(including copyright notices, patent notices, disclaimers of warranty,
-or limitations of liability) contained within the Source Code Form of
-the Covered Software, except that You may alter any license notices to
-the extent required to remedy known factual inaccuracies.
-
-3.5. Application of Additional Terms
-
-You may choose to offer, and to charge a fee for, warranty, support,
-indemnity or liability obligations to one or more recipients of Covered
-Software. However, You may do so only on Your own behalf, and not on
-behalf of any Contributor. You must make it absolutely clear that any
-such warranty, support, indemnity, or liability obligation is offered by
-You alone, and You hereby agree to indemnify every Contributor for any
-liability incurred by such Contributor as a result of warranty, support,
-indemnity or liability terms You offer. You may include additional
-disclaimers of warranty and limitations of liability specific to any
-jurisdiction.
-
-4. Inability to Comply Due to Statute or Regulation
----------------------------------------------------
-
-If it is impossible for You to comply with any of the terms of this
-License with respect to some or all of the Covered Software due to
-statute, judicial order, or regulation then You must: (a) comply with
-the terms of this License to the maximum extent possible; and (b)
-describe the limitations and the code they affect. Such description must
-be placed in a text file included with all distributions of the Covered
-Software under this License. Except to the extent prohibited by statute
-or regulation, such description must be sufficiently detailed for a
-recipient of ordinary skill to be able to understand it.
-
-5. Termination
---------------
-
-5.1. The rights granted under this License will terminate automatically
-if You fail to comply with any of its terms. However, if You become
-compliant, then the rights granted under this License from a particular
-Contributor are reinstated (a) provisionally, unless and until such
-Contributor explicitly and finally terminates Your grants, and (b) on an
-ongoing basis, if such Contributor fails to notify You of the
-non-compliance by some reasonable means prior to 60 days after You have
-come back into compliance. Moreover, Your grants from a particular
-Contributor are reinstated on an ongoing basis if such Contributor
-notifies You of the non-compliance by some reasonable means, this is the
-first time You have received notice of non-compliance with this License
-from such Contributor, and You become compliant prior to 30 days after
-Your receipt of the notice.
-
-5.2. If You initiate litigation against any entity by asserting a patent
-infringement claim (excluding declaratory judgment actions,
-counter-claims, and cross-claims) alleging that a Contributor Version
-directly or indirectly infringes any patent, then the rights granted to
-You by any and all Contributors for the Covered Software under Section
-2.1 of this License shall terminate.
-
-5.3. In the event of termination under Sections 5.1 or 5.2 above, all
-end user license agreements (excluding distributors and resellers) which
-have been validly granted by You or Your distributors under this License
-prior to termination shall survive termination.
-
-************************************************************************
-*                                                                      *
-*  6. Disclaimer of Warranty                                           *
-*  -------------------------                                           *
-*                                                                      *
-*  Covered Software is provided under this License on an "as is"       *
-*  basis, without warranty of any kind, either expressed, implied, or  *
-*  statutory, including, without limitation, warranties that the       *
-*  Covered Software is free of defects, merchantable, fit for a        *
-*  particular purpose or non-infringing. The entire risk as to the     *
-*  quality and performance of the Covered Software is with You.        *
-*  Should any Covered Software prove defective in any respect, You     *
-*  (not any Contributor) assume the cost of any necessary servicing,   *
-*  repair, or correction. This disclaimer of warranty constitutes an   *
-*  essential part of this License. No use of any Covered Software is   *
-*  authorized under this License except under this disclaimer.         *
-*                                                                      *
-************************************************************************
-
-************************************************************************
-*                                                                      *
-*  7. Limitation of Liability                                          *
-*  --------------------------                                          *
-*                                                                      *
-*  Under no circumstances and under no legal theory, whether tort      *
-*  (including negligence), contract, or otherwise, shall any           *
-*  Contributor, or anyone who distributes Covered Software as          *
-*  permitted above, be liable to You for any direct, indirect,         *
-*  special, incidental, or consequential damages of any character      *
-*  including, without limitation, damages for lost profits, loss of    *
-*  goodwill, work stoppage, computer failure or malfunction, or any    *
-*  and all other commercial damages or losses, even if such party      *
-*  shall have been informed of the possibility of such damages. This   *
-*  limitation of liability shall not apply to liability for death or   *
-*  personal injury resulting from such party's negligence to the       *
-*  extent applicable law prohibits such limitation. Some               *
-*  jurisdictions do not allow the exclusion or limitation of           *
-*  incidental or consequential damages, so this exclusion and          *
-*  limitation may not apply to You.                                    *
-*                                                                      *
-************************************************************************
-
-8. Litigation
--------------
-
-Any litigation relating to this License may be brought only in the
-courts of a jurisdiction where the defendant maintains its principal
-place of business and such litigation shall be governed by laws of that
-jurisdiction, without reference to its conflict-of-law provisions.
-Nothing in this Section shall prevent a party's ability to bring
-cross-claims or counter-claims.
-
-9. Miscellaneous
-----------------
-
-This License represents the complete agreement concerning the subject
-matter hereof. If any provision of this License is held to be
-unenforceable, such provision shall be reformed only to the extent
-necessary to make it enforceable. Any law or regulation which provides
-that the language of a contract shall be construed against the drafter
-shall not be used to construe this License against a Contributor.
-
-10. Versions of the License
----------------------------
-
-10.1. New Versions
-
-Mozilla Foundation is the license steward. Except as provided in Section
-10.3, no one other than the license steward has the right to modify or
-publish new versions of this License. Each version will be given a
-distinguishing version number.
-
-10.2. Effect of New Versions
-
-You may distribute the Covered Software under the terms of the version
-of the License under which You originally received the Covered Software,
-or under the terms of any subsequent version published by the license
-steward.
-
-10.3. Modified Versions
-
-If you create software not governed by this License, and you want to
-create a new license for such software, you may create and use a
-modified version of this License if you rename the license and remove
-any references to the name of the license steward (except to note that
-such modified license differs from this License).
-
-10.4. Distributing Source Code Form that is Incompatible With Secondary
-Licenses
-
-If You choose to distribute Source Code Form that is Incompatible With
-Secondary Licenses under the terms of this version of the License, the
-notice described in Exhibit B of this License must be attached.
-
-Exhibit A - Source Code Form License Notice
--------------------------------------------
-
-  This Source Code Form is subject to the terms of the Mozilla Public
-  License, v. 2.0. If a copy of the MPL was not distributed with this
-  file, You can obtain one at http://mozilla.org/MPL/2.0/.
-
-If it is not possible or desirable to put the notice in a particular
-file, then You may include the notice in a location (such as a LICENSE
-file in a relevant directory) where a recipient would be likely to look
-for such a notice.
-
-You may add additional accurate notices of copyright ownership.
-
-Exhibit B - "Incompatible With Secondary Licenses" Notice
----------------------------------------------------------
-
-  This Source Code Form is "Incompatible With Secondary Licenses", as
-  defined by the Mozilla Public License, v. 2.0.

From 55f04b1edd25729e35bb890d01e08cc63675cea6 Mon Sep 17 00:00:00 2001
From: Erik Nordin <enordin@mozilla.com>
Date: Tue, 1 Oct 2024 12:23:59 -0500
Subject: [PATCH 442/442] Add TODO for issue #869

---
 .gitmodules | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitmodules b/.gitmodules
index 51663e9bf..f88221eb5 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -37,6 +37,8 @@
 #
 # It may be possible to remove `3rd_party/browsermt-marian-dev` to instead use `inference-engine/3rd-party/browsermt-marian-dev` everywhere
 # within this repository, but I will leave that for a future commit if there is a need to do so.
+#
+# TODO(#869)
 [submodule "inference/3rd_party/browsermt-marian-dev"]
 	path = inference/3rd_party/browsermt-marian-dev
 	url = https://github.com/browsermt/marian-dev