Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

3894 async information extraction #3908

Merged
merged 63 commits into from
Nov 9, 2021
Merged

3894 async information extraction #3908

merged 63 commits into from
Nov 9, 2021

Conversation

konzz
Copy link
Member

@konzz konzz commented Sep 8, 2021 •
edited by gabriel-piles
Loading

fixes #3894

PR checklist:

  • Update READ.me ?
  • Update API documentation ?

QA checklist:

  • Smoke test the functionality described in the issue
  • Test for side effects
  • UI responsiveness
  • Cross browser testing
  • Code review

@konzz konzz force-pushed the 3894-async-information-extraction branch 2 times, most recently from c28eb81 to a2c90de Compare October 19, 2021 15:36
@konzz konzz marked this pull request as ready for review October 22, 2021 12:13
@konzz konzz force-pushed the 3894-async-information-extraction branch from da445b4 to ba5d68c Compare October 26, 2021 11:57
gabriel-piles and others added 24 commits October 28, 2021 15:10
@gabriel-piles @konzz
WIP
efa4222
@konzz
intial test for the taskManager
1564051
@konzz
RedisServer
a7a52c9
@konzz
TaskManager WIP
d13475a
@konzz
Taskmanager tests with external dummy server WIP
f7af6f4
@gabriel-piles @konzz
Task manager simple version working
bd6c1a8
@konzz
refactor to the Repeater to be a class and dont share the stop variable
563ed50
@konzz
TaskManage using Repeater to listen for messages
7c5ca67
@konzz
Refactored Repeater specs to work with fakeTimers
8415528
@konzz
RepeaterWithLock for locking distributed Uwazi tasks
7d8334f
@gabriel-piles @konzz
Repeat with lock handles redis unavailable
e65abf4
@gabriel-piles @konzz
Handle error in task and add delay between tasks parameter in repeat …
abc31e1 
…with lock
@konzz
Repeater specs cleanup WIP
5f38b0b
@gabriel-piles @konzz
Clean up tests for repeater with lock
3c3a78b
@konzz
Cleanup of consoloe logs and fixed some eslint errors
54c31b8
@gabriel-piles @konzz
Task manager refactor to avoid factory
ea7e87e
@konzz
TaskManager refactor and redis unavailable test
2d3266b
@konzz
Handling redis errors in taskmanager
3adb54b
@gabriel-piles @konzz
Segmentator task working for one file
2adf80c
@gabriel-piles @konzz
Segment many files
baaf6c2
@gabriel-piles @konzz
Use information information extraction for segmenting
83bafcf
@gabriel-piles @konzz
Multitenant test for pdf segmetnation
4fcd276
@konzz
exposing the fixturer
4558d53
@konzz
PDF segmentation runing multitenant WIP
8a6f365
konzz added 25 commits October 28, 2021 15:10
@konzz
fixed some eslint errors
4ab94f7
@konzz
Fixed instalation of redis server for tests
73543cf
@konzz
removed redis-server
16fd346
@konzz
Some refactor and atempting to fix CI redis instalation
cb4515d
@konzz
Moved redis installation to a jest pre hook
277b4a8
@konzz
removed unused variable
2fd8a54
@konzz
fixed an issue with downloadRedis
52fcb9b
@konzz
refactored PDFsegmentation to segment all files regardless of template
ab41bbe
@konzz
Fixed some errors typing
127ba72
@konzz
General refactor, fixed PDFsegemntation tests database setup
6692524
@konzz
deleted unused var
9a96c3a
@konzz
renamed RepeatWith to DistributedLoop
3825a84
@konzz
Changed task structure to contain params
0b977f7
@konzz
PDFsegmentation service requesting and storing the xml
f260d8b
@konzz
some error handling
fd5c112
@konzz
deleted unused var
10ecc20
@konzz
fixed type
866d020
@konzz
Always deleting result message when an error happens
48cce93
@konzz
only save the error when the segmentation placeholder existed to avoi…
12e9a9e 
…d saving malformed responses
@konzz
Using getFileContents instead of readFileSync
62ff8cf
@konzz
Segementation type and using filesystem instead of fs
9187b54
@konzz
Error handling for missing files
2bd25d0
@konzz
Storing only one process when the segmentation fails
18bb88e
@konzz
adding tenant to service url
9469669
@konzz
fixed types error
cc84a09
@konzz konzz force-pushed the 3894-async-information-extraction branch from 9e7ce53 to cc84a09 Compare October 28, 2021 13:10
@konzz konzz merged commit 97e52a5 into development Nov 9, 2021
@konzz konzz deleted the 3894-async-information-extraction branch November 9, 2021 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@txau txau txau left review comments

Assignees

@txau txau

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Comunication channel with external services
3 participants
@konzz @txau @gabriel-piles