Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Ingestion Job management API for Feast Core #548

Merged
merged 66 commits into from
Apr 7, 2020
Merged

Add Ingestion Job management API for Feast Core #548

merged 66 commits into from
Apr 7, 2020

Commits on Apr 6, 2020

  1. Added protobuf definitions for a Job Management API

    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    1e1abb1 View commit details
    Browse the repository at this point in the history
  2. Added query methods to JobRepository to query jobs by store and featu…

    …reset
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    bd04076 View commit details
    Browse the repository at this point in the history
  3. Added hashCode() & equals() to Job model compare and hash jobs

    This would allow jobs to be used as elements in HashSets and keys in
    HashMaps
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    643f042 View commit details
    Browse the repository at this point in the history
  4. Added toIngestionProto() to Job object to convert Job model to ingest…

    …ion job proto
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    f4b417e View commit details
    Browse the repository at this point in the history
  5. Added query methods to FeatureSetRepository to query by exact (name, …

    …project) or (name,version)
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    9d62a38 View commit details
    Browse the repository at this point in the history
  6. Added listJobs() to JobService to handle request list ingestion jobs …

    …requests
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    32a0bb5 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    4f55012 View commit details
    Browse the repository at this point in the history
  8. Added code to setup mockups for testing JobService

    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    b3aa72a View commit details
    Browse the repository at this point in the history
  9. Revert "Added hashCode() & equals() to Job model compare and hash jobs"

    This reverts commit ba995bb712cbd38f0f4bef47efbe433d5ec07521 as
    it caused core tests to fail
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    06b84df View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    c4b901a View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    cd0bde2 View commit details
    Browse the repository at this point in the history
  12. Added getTransitionalStates() to JobStatus to return collection of tr…

    …ansitional states
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    c59035f View commit details
    Browse the repository at this point in the history
  13. Changed stopJobs() to throw unsupported error on transitional job sta…

    …tuses
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    898045c View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    f4711cd View commit details
    Browse the repository at this point in the history
  15. Make findFeatureSet() match only one featureset.

    Limit findFeatureSets() as feature set references as composite keys
    should match one and only one featureset.
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    fe744ea View commit details
    Browse the repository at this point in the history
  16. Added matchFeatureSets() to SpecService to match Feature Sets from Re…

    …ferences
    
    This commit is adds temporary support for FeatureSetReference to
    SpecService via listFeatureSets(). In the future, this should merged
    together with listFeatureSets() as their functionality is almost the
    same.
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    a79f8b4 View commit details
    Browse the repository at this point in the history
  17. Refactor JobService listJobs() to use SpecService to provide features…

    …et matching
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    5b25929 View commit details
    Browse the repository at this point in the history
  18. Revert findBy methods added to FeatureSetRepository as no longer used.

    JobService no longer depends on FeatureSetRepository directly, instead
    via SpecService
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    1997bb1 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    f320e38 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    0ec5afb View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    83cc4c7 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    cf6a9c4 View commit details
    Browse the repository at this point in the history
  23. Revert "Use assertThat() & equalTo() instead of assertEquals() in Job…

    …Service" due to failed tests
    
    This reverts commit bb3cf0ae1e7cfbde7a2f997cacc661d040c0a35f.
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    8088b33 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    bdc2803 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    fafcd11 View commit details
    Browse the repository at this point in the history
  26. Throw InvalidArgumentException instead when calling listJobs() with i…

    …nvalid FeatureSetReference
    
    Throw InvalidArgumentException instead of UnsupportedOperationException
    for better semantics: UnsupportedOperationException should be reserved
    for operations that fail due to failed preconditions
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    7a97e1f View commit details
    Browse the repository at this point in the history
  27. Fixed typos in javadocs: InvalidArgumentException should be IllegalAr…

    …gumentException
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    d277f93 View commit details
    Browse the repository at this point in the history
  28. Fixed in findByFeatureSetsIn() query in JobRepository

    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    255946b View commit details
    Browse the repository at this point in the history
  29. Renamed toIngestionProto() methods to toProto() to follow code conven…

    …tion
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    09e9ddc View commit details
    Browse the repository at this point in the history
  30. Make JobService's listJobs() a transaction to prevent DB data race co…

    …nditions
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    c85da17 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    c876e40 View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    1799c3a View commit details
    Browse the repository at this point in the history
  33. Make JobService's listJobs() to return all ingestion jobs on empty fi…

    …lter
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    1bead61 View commit details
    Browse the repository at this point in the history
  34. Fixed issue where the jobManager map that JobService built used wrong…

    … keys
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    19994ea View commit details
    Browse the repository at this point in the history
  35. Fix issue where actual Job Status is not synced with database.

    Issue occurs when the job is aborted/restarted, but the JobStatus
    has not yet been updated by JobUpdateTask. Hence another call
    to abortJob() & restartJob() that should be rejected due invalid status
    is allowed through
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    ff73b5f View commit details
    Browse the repository at this point in the history
  36. Configuration menu
    Copy the full SHA
    c207f1a View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    c64039a View commit details
    Browse the repository at this point in the history
  38. Move documentation on JobService operations to CoreService protobuf d…

    …efinition
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    ed4dae9 View commit details
    Browse the repository at this point in the history
  39. Added IngestJob to python sdk as native representation of IngestionJo…

    …b proto
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    24c09be View commit details
    Browse the repository at this point in the history
  40. Configuration menu
    Copy the full SHA
    2930388 View commit details
    Browse the repository at this point in the history
  41. Configuration menu
    Copy the full SHA
    5cc8abe View commit details
    Browse the repository at this point in the history
  42. Fixed __connect_core() to connect to Feast CoreService when calling o…

    …n Job API calls
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    d86cb66 View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    d9874f2 View commit details
    Browse the repository at this point in the history
  44. Configuration menu
    Copy the full SHA
    19620b2 View commit details
    Browse the repository at this point in the history
  45. Added basic job api e2e test to exercise job api

    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    6b47ecb View commit details
    Browse the repository at this point in the history
  46. Reorder the operations e2etest to make sure that jobs are running aft…

    …er test
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    f3471ac View commit details
    Browse the repository at this point in the history
  47. Added e2e test for all types exercising job api

    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    1707401 View commit details
    Browse the repository at this point in the history
  48. Fixed typo in function arguments

    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    9999db5 View commit details
    Browse the repository at this point in the history
  49. Configuration menu
    Copy the full SHA
    f012623 View commit details
    Browse the repository at this point in the history
  50. Configuration menu
    Copy the full SHA
    2e819b2 View commit details
    Browse the repository at this point in the history
  51. Disable support for restarting Job in a terminal state due to possibl…

    …e race conditions
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    ba08c8a View commit details
    Browse the repository at this point in the history
  52. Added __str__ and __repr__ to IngestJob to render ingestjob in human …

    …readable string
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    5fb774c View commit details
    Browse the repository at this point in the history
  53. Configuration menu
    Copy the full SHA
    80ef3c9 View commit details
    Browse the repository at this point in the history
  54. Configuration menu
    Copy the full SHA
    d3c9a3d View commit details
    Browse the repository at this point in the history
  55. Fixed typo in IngestJob.store property

    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    0cfc32a View commit details
    Browse the repository at this point in the history
  56. Configuration menu
    Copy the full SHA
    a540157 View commit details
    Browse the repository at this point in the history
  57. Make the grpc error message more apparent on stop_ingest_job() and re…

    …start_ingest_job()
    Zhu Zhanyan committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    355b571 View commit details
    Browse the repository at this point in the history
  58. Configuration menu
    Copy the full SHA
    05b33a6 View commit details
    Browse the repository at this point in the history
  59. Configuration menu
    Copy the full SHA
    3b8198a View commit details
    Browse the repository at this point in the history
  60. Configuration menu
    Copy the full SHA
    19bd141 View commit details
    Browse the repository at this point in the history
  61. Configuration menu
    Copy the full SHA
    9f6cf6a View commit details
    Browse the repository at this point in the history
  62. Configuration menu
    Copy the full SHA
    87054b9 View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2020

  1. Configure IngestJob.wait() to backoff with a exponentially larger wai…

    …t duration
    Zhu Zhanyan committed Apr 7, 2020
    Configuration menu
    Copy the full SHA
    1848dd4 View commit details
    Browse the repository at this point in the history
  2. Added print statements to debug e2e test failure.

    Zhu Zhanyan committed Apr 7, 2020
    Configuration menu
    Copy the full SHA
    146fb2b View commit details
    Browse the repository at this point in the history
  3. Revert "Added print statements to debug e2e test failure."

    This reverts commit 146fb2b.
    Zhu Zhanyan committed Apr 7, 2020
    Configuration menu
    Copy the full SHA
    0fa6b93 View commit details
    Browse the repository at this point in the history
  4. Fixed issue of test waiting for aborted job to become running causing…

    … timeout.
    Zhu Zhanyan committed Apr 7, 2020
    Configuration menu
    Copy the full SHA
    66d3f77 View commit details
    Browse the repository at this point in the history