Code

3894 async information extraction#3908

Code

Merged

konzz merged 63 commits intodevelopmenthuridocs/uwazi:developmentfrom 3894-async-information-extractionhuridocs/uwazi:3894-async-information-extractionCopy head branch name to clipboardNov 9, 2021

+1,907-56

Commits

Commits on Oct 28, 2021

• WIP

  

  gabriel-piles

  authored and

  konzz

  committed
  efa4222

• intial test for the taskManager

  konzz

  committed
  1564051

• RedisServer

  konzz

  committed
  a7a52c9

• TaskManager WIP

  konzz

  committed
  d13475a

• Taskmanager tests with external dummy server WIP

  konzz

  committed
  f7af6f4

• Task manager simple version working

  

  gabriel-piles

  authored and

  konzz

  committed
  bd6c1a8

• refactor to the Repeater to be a class and dont share the stop variable

  konzz

  committed
  563ed50

• TaskManage using Repeater to listen for messages

  konzz

  committed
  7c5ca67

• Refactored Repeater specs to work with fakeTimers

  konzz

  committed
  8415528

• RepeaterWithLock for locking distributed Uwazi tasks

  konzz

  committed
  7d8334f

• Repeat with lock handles redis unavailable

  

  gabriel-piles

  authored and

  konzz

  committed
  e65abf4

• Handle error in task and add delay between tasks parameter in repeat with lock

  

  gabriel-piles

  authored and

  konzz

  committed
  abc31e1

• Repeater specs cleanup WIP

  konzz

  committed
  5f38b0b

• Clean up tests for repeater with lock

  

  gabriel-piles

  authored and

  konzz

  committed
  3c3a78b

• Cleanup of consoloe logs and fixed some eslint errors

  konzz

  committed
  54c31b8

• Task manager refactor to avoid factory

  

  gabriel-piles

  authored and

  konzz

  committed
  ea7e87e

• TaskManager refactor and redis unavailable test

  konzz

  committed
  2d3266b

• Handling redis errors in taskmanager

  konzz

  committed
  3adb54b

• Segmentator task working for one file

  

  gabriel-piles

  authored and

  konzz

  committed
  2adf80c

• Segment many files

  

  gabriel-piles

  authored and

  konzz

  committed
  baaf6c2

• Use information information extraction for segmenting

  

  gabriel-piles

  authored and

  konzz

  committed
  83bafcf

• Multitenant test for pdf segmetnation

  

  gabriel-piles

  authored and

  konzz

  committed
  4fcd276

• exposing the fixturer

  konzz

  committed
  4558d53

• PDF segmentation runing multitenant WIP

  konzz

  committed
  8a6f365

• TODOs segmentation

  konzz

  committed
  b388402

• PDF segmentation storing the segmentation place holder

  konzz

  committed
  e36efa5

• Moved all files under services folder, and refactor Repeater specs so they can work in paralel with the Taksmanager specs

  konzz

  committed
  13ea822

• Aggregation to only get files that need to be processed

  konzz

  committed
  b27d761

• Taskmanager count, and some pending tests for the PDFsegmentation

  konzz

  committed
  e2673ed

• PDFsegmentation doing nothing when there is already pending tasks

  konzz

  committed
  2bdb11c

• Segmentator sending the PDFs instead of the TasdkManager

  konzz

  committed
  f8df201

• Handling tenants with no segmentation config

  konzz

  committed
  8ed5f58

• TaskManager only passing the results message to the connector instead requesting the results

  konzz

  committed
  89a9170

• Uwazi server using a repeater to Segment PDFs

  konzz

  committed
  efa9a78

• small refactor and rename

  konzz

  committed
  3ec154b

• fixed some eslint errors and specs

  konzz

  committed
  a4f556a

• fixed some eslint errors

  konzz

  committed
  4ab94f7

• Fixed instalation of redis server for tests

  konzz

  committed
  73543cf

• removed redis-server

  konzz

  committed
  16fd346

• Some refactor and atempting to fix CI redis instalation

  konzz

  committed
  cb4515d

• Moved redis installation to a jest pre hook

  konzz

  committed
  277b4a8

• removed unused variable

  konzz

  committed
  2fd8a54

• fixed an issue with downloadRedis

  konzz

  committed
  52fcb9b

• refactored PDFsegmentation to segment all files regardless of template

  konzz

  committed
  ab41bbe

• Fixed some errors typing

  konzz

  committed
  127ba72

• General refactor, fixed PDFsegemntation tests database setup

  konzz

  committed
  6692524

• deleted unused var

  konzz

  committed
  9a96c3a

• renamed RepeatWith to DistributedLoop

  konzz

  committed
  3825a84

• Changed task structure to contain params

  konzz

  committed
  0b977f7

• PDFsegmentation service requesting and storing the xml

  konzz

  committed
  f260d8b

• some error handling

  konzz

  committed
  fd5c112

• deleted unused var

  konzz

  committed
  10ecc20

• fixed type

  konzz

  committed
  866d020

• Always deleting result message when an error happens

  konzz

  committed
  48cce93

• only save the error when the segmentation placeholder existed to avoid saving malformed responses

  konzz

  committed
  12e9a9e

• Using getFileContents instead of readFileSync

  konzz

  committed
  62ff8cf

• Segementation type and using filesystem instead of fs

  konzz

  committed
  9187b54

• Error handling for missing files

  konzz

  committed
  2bd25d0

• Storing only one process when the segmentation fails

  konzz

  committed
  18bb88e

• adding tenant to service url

  konzz

  committed
  9469669

• fixed types error

  konzz

  committed
  cc84a09

Commits on Nov 8, 2021

• sgementation service behind a config flag

  konzz

  committed
  7b9e3b5

Commits on Nov 9, 2021

• Merge branch 'development' into 3894-async-information-extraction

  konzz

  authored
  a272d02