Skip to content

Commit

Permalink
[bump] introduce new version: 0.6 (#459)
Browse files Browse the repository at this point in the history
* fix: improve external api for redis

* fix: improve exports for redis

* fix: expose redis codec

* Feature: v0.6.0-alpha.0 version of apalis

Breaking Changes:
- Dropped traits Job and Message, please use namespace

* fix: minor fixes on some failures

* lint: cargo fmt

* fix: remove Job impl

* lint: cargo fmt

* bench: improve polling

* fix: introduce namespace and codec config (#339)

* fix: introduce namespace and codec config

* fix: missing apis

* Version: 0.6.0-alpha.1 (#342)

* api: for redis and sqlite

* Version: 0.6.0-alpha.1

Changelog:
- Redis storage doesnt require pool to be clone. Allows use of deadpool-redis among others.
- Namespace is picked by default for `new` methods.

* fix: docs and tests

* lint: cargo clippy and fmt

* postgres: add a listener example

* bump: to v0.6.0-alpha.1 (#343)

* api: for redis and sqlite

* Version: 0.6.0-alpha.1

Changelog:
- Redis storage doesnt require pool to be clone. Allows use of deadpool-redis among others.
- Namespace is picked by default for `new` methods.

* fix: docs and tests

* lint: cargo clippy and fmt

* postgres: add a listener example

* bump: to v0.6.0-alpha.1

* fix: allow cd for prereleases (#349)

* Remove `Clone` constraints and buffer the service (#348)

* feat: remove the `Clone` requirements for services

* test save

* fix: get buffered layer working

* update: remove clone & update api

* fix: tests and api

* lint: clippy fixes

* lint: cargo fmt

* bump: to 0.6.0-rc.1 (#350)

* feat: add rsmq example (#353)

* Fix: load layer from poller (#354)

* fix: backend layers were not loaded

* fix: handle clone

* Fix: mq example (#355)

* fix: mq ack

* lint: fmt

* fix: handle unwraps in storages (#356)

* fix: handle unwraps in storages

* fix: ensure no unwrap

* fix: better apalis deps allowing tree shaking for backends (#357)

* fix: better apalis deps allowing tree shaking for backends

* fix: remove backend features in the root crate

* standardize backend for storage and mq (#358)

* fix: standardize backend for storage and mq

* fix: minor fixes

* feat: standardize cron as backend (#359)

* fix: remove non-working restapi example (#360)

* fix: expose the missing apis (#361)

* bump: to new version (#362)

* Make Config accessible publicly (#364)

* fix: add missing exposed config

* fix: add getters

* fix: die if retries is zero (#365)

* Feature: Add a layer that catches panics (#366)

* Feature: Add a layer that catches panics

This allows preventing job execution from killing workers and returns an error containing the backtrace

* fix: backtrace as it may be different

* add: example for catch-panic

* fix: make  not default

* Feature: Save results for storages (#369)

* Feature: Save results for storages

Currently just the status is stored, this PR adds the ability to save the result

* fix: result from storage

* fix: kill and abort issue

* Bump: to 0.6.0-rc.3 (#370)

* fix: serde for sql request (#371)

* fix: serde for sql request

* fix: serde for attempts

* lint: fmt

* fix: handle attempts in storages (#373)

* fix: handle attempts in storages

* fix: chrono serialization

* fix: tests failing because of tests

* add: test utils that allow backend polling during tests (#374)

* add: test utils that allow backend polling during tests

* fix: introduce testwrapper and add more tests

* fix: add sample for testing

* fix: more fixes and actions fixes

* fix: more fixes on vacuuming

* tests: improve cleanup and generic testing

* fix: improve testing and fix some found bugs

* fix: postgres query and remove incompatible tests

* fix: remove redis incompatible check

* fix: minor fixes

* fix: postgres json elements

* bump: to 0.6.0-rc.4 (#377)

* fix: handle 0 retries (#378)

* fix: ack api to allow backend to handle differently (#383)

* fix: ack api to allow backend to handle differently

* fix: related to storage tests

* fix: calculate status for postgres

* fix(deps): update rust crate sqlx to 0.8.0 (#380)

* chore: fix typos (#346)

* chore: Add repository to metadata (#345)

* fix(deps): update rust crate sqlx to 0.8.0

* fix: sqlite example

---------

Co-authored-by: John Vandenberg <jayvdb@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: geofmureithi <mureithinjuguna@gmail.com>
Co-authored-by: Geoffrey Mureithi <95377562+geofmureithi@users.noreply.github.com>

* bump: to v0.6.0-rc.5 (#385)

* chore: standardize codec usage (#388)

* bump: to v0.6.0-rc.5

* fix: standardize codec usage

* lint: cargo fmt

* Chore/more examples (#389)

* add: catch-panic example

* add: graceful shutdown example

* add: unmonitored example

* add: arguments example

* fix: minor updates

* fix: sql tests

* fix: minor updates

* fix: improve on benches (#379)

* fix: improve on benches

* fix: bench trigger

* fix: include tokio for sqlx

* fix: improve the benching approach

* fix: mysql api

* fix: redis api

* fix: improve bench approach, remove counter

* remove: setup

* remove: pg

* fix: pg

* fix: pg

* fix(deps): update rust crate sqlx to 0.8.1 [security] (#400)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Geoffrey Mureithi <95377562+geofmureithi@users.noreply.github.com>

* fix: add some missing data required for dependency injection (#409)

* fix: add some missing data required for dependency injection

* lint: clippy and fmt

* remove: benchmarks (#410)

They will be moved to https://github.com/geofmureithi/apalis-benchmarks

* bump: to 0.6.0-rc.6 (#412)

* Update async-std to 1.13 (#413)

* Feature: Introducing Request Context (#416)

* wip: introduce context to request

* fix: get request context working

* lint: cargo fmt

* fix: get tests compiling

* add: push_request and shedule_request

* fix: task_id for Testwrapper

* fix: minor checks and fixes on postgres tests

* fix: bug on postgres fetch_next

* bump: to 0.6.0-rc.7 (#418)

* fix: apply `FromRequest` for items in `Parts` (#425)

Problem: We are missing crucial `FromRequest` impls for:
- TaskId
- Attempt
- Namespace

Also removed `Context<E>`

Solution: Implement `FromRequest` for these Types.

* fix:[bug] include backend provided layer in service layers. (#426)

* fix:[bug] include backend provided layer in service layers.

Problem:
The current worker logic is missing an implementation where the backend provided layer should be added to the service's layer.
This is a critical issue that affects all v0.6.0-rc-7 users and they should update as soon as a new release is done.

Solution:
- Add backend layers to service's layer.
- Add worker_consume tests on the storages to prevent regression on this.

* chore: comment an enforcement rule not yet followed by redis

* chore: bump to 0.6.0-rc.8 (#430)

* fix: apply max_attempts set via SqlContext (#447)

So that a custom number of attempts can be configured:

  let mut ctx = SqlContext::new();
  ctx.set_max_attempts(2);
  let req = Request::new_with_ctx(job, ctx);
  storage.push_request(req).await.unwrap();

While the default is still to try up to 25 times:

  storage.push(job).await.unwrap();

* Bump redis (#442)

* feat: re-export sqlx (#451)

Making sqlx accessible to users of apalis without requiring them to
explicitly add it as a dependency.

* feat: Improve Worker management and drop Executor  (#428)

* feat: introducing WorkerBuilderExt which makes the work of building a new worker way easier.

* improve: worker api almost there

* fix: radical improvements and updates. Removed executor and got graceful shutdown working

* chore: deprecate register with count and force builder order

* chore: more improvements on the worker

* fix: allow DI for Worker

* add: get the task count by a worker

* lint: fmt and clippy

* fix: allow worker stopping

* Chore/better api (#452)

* fix: relax the api provided for sqlx req

* lint: clippy and fmt

* feat: add recovery of abandoned jobs to backend heartbeats (#453)

* feat: add recovery of abandoned jobs to backend heartbeats

* lint: fmt

* fix: attempt to get tests passing

* fix: attempt to get tests passing

* fix: minor fix typo

* fix: minor different solutions

* fix: better handle attempts

* handle postgres edge case

* fix: better handling

* feat: allow backends to emit errors (#454)

* feat: allow backends to emit errors

* lint: fmt

* fix: pass in a reference to prevent mutation

* Feat: Introduce simple ability to pipe cron jobs to any backend (#455)

* Feat: Introduce simple ability to pipe cron jobs to any backend

This feature allows you to quickly persist cron jobs guaranteeing they will be run and can be distributed

* lint: cargo fmt

* chore/dev-branch

* feat: return of exposing backends to help in building apis (#457) (#458)

* fmt: Cargo.toml

* fix: removed features

* fix: run only specific tests

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: John Vandenberg <jayvdb@gmail.com>
Co-authored-by: Mathias Lafeldt <mathias.lafeldt@gmail.com>
Co-authored-by: zakstucke <44890343+zakstucke@users.noreply.github.com>
  • Loading branch information
5 people authored Nov 28, 2024
1 parent 575553a commit 6d6ed27
Show file tree
Hide file tree
Showing 112 changed files with 6,158 additions and 3,418 deletions.
39 changes: 0 additions & 39 deletions .github/workflows/bench.yaml

This file was deleted.

5 changes: 1 addition & 4 deletions .github/workflows/cd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,7 @@ jobs:
# vX.Y.Z-foo is pre-release version
VERSION=${GITHUB_REF#refs/tags/v}
VERSION_NUMBER=${VERSION%-*}
PUBLISH_OPTS="--dry-run"
if [[ $VERSION == $VERSION_NUMBER ]]; then
PUBLISH_OPTS=""
fi
PUBLISH_OPTS=""
echo VERSION=${VERSION} >> $GITHUB_ENV
echo PUBLISH_OPTS=${PUBLISH_OPTS} >> $GITHUB_ENV
echo VERSION_NUMBER=${VERSION_NUMBER} >> $GITHUB_ENV
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
- uses: actions-rs/cargo@v1
with:
command: check
args: --features tokio-comp
args: --all
- uses: actions-rs/cargo@v1
with:
command: check
Expand All @@ -39,7 +39,7 @@ jobs:
- uses: actions-rs/cargo@v1
with:
command: test
args: --features tokio-comp
# args: --all


fmt:
Expand Down
7 changes: 0 additions & 7 deletions .github/workflows/mysql.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,4 @@
on:
push:
paths:
- "packages/apalis-sql/src/lib.rs"
- "packages/apalis-sql/mysql.rs"
- "packages/apalis-sql/src/migrations/mysql/**"
- "packages/apalis-sql/src/Cargo.toml"
- ".github/workflows/mysql.yaml"
pull_request:
paths:
- "packages/apalis-sql/src/lib.rs"
Expand Down
9 changes: 1 addition & 8 deletions .github/workflows/postgres.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,4 @@
on:
push:
paths:
- "packages/apalis-sql/src/lib.rs"
- "packages/apalis-sql/postgres.rs"
- "packages/apalis-sql/src/migrations/postgres/**"
- "packages/apalis-sql/src/Cargo.toml"
- ".github/workflows/postgres.yaml"
pull_request:
paths:
- "packages/apalis-sql/src/lib.rs"
Expand Down Expand Up @@ -37,4 +30,4 @@ jobs:
toolchain: stable
override: true
- run: cargo test --no-default-features --features postgres,migrate,tokio-comp -- --test-threads=1
working-directory: packages/apalis-sql
working-directory: packages/apalis-sql
8 changes: 2 additions & 6 deletions .github/workflows/redis.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
on:
push:
paths:
- "packages/apalis-redis/**"
- ".github/workflows/redis.yaml"
pull_request:
paths:
- "packages/apalis-redis/**"
Expand All @@ -26,11 +22,11 @@ jobs:
profile: minimal
toolchain: stable
override: true
- run: cargo test --features tokio-comp -- --test-threads=1
- run: cargo test -- --test-threads=1
working-directory: packages/apalis-redis
env:
REDIS_URL: redis://127.0.0.1/
- run: cargo test --features async-std-comp -- --test-threads=1
- run: cargo test -- --test-threads=1
working-directory: packages/apalis-redis
env:
REDIS_URL: redis://127.0.0.1/
9 changes: 1 addition & 8 deletions .github/workflows/sqlite.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,4 @@
on:
push:
paths:
- "packages/apalis-sql/src/lib.rs"
- "packages/apalis-sql/src/sqlite.rs"
- "packages/apalis-sql/src/migrations/sqlite/**"
- "packages/apalis-sql/src/Cargo.toml"
- ".github/workflows/sqlite.yaml"
pull_request:
paths:
- "packages/apalis-sql/src/lib.rs"
Expand All @@ -28,4 +21,4 @@ jobs:
toolchain: stable
override: true
- run: cargo test --no-default-features --features sqlite,migrate,tokio-comp -- --test-threads=1
working-directory: packages/apalis-sql
working-directory: packages/apalis-sql
94 changes: 30 additions & 64 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ repository = "https://github.com/geofmureithi/apalis"

[package]
name = "apalis"
version = "0.5.5"
version = "0.6.0"
authors = ["Geoffrey Mureithi <mureithinjuguna@gmail.com>"]
description = "Simple, extensible multithreaded background job processing for Rust"
edition.workspace = true
Expand All @@ -19,18 +19,8 @@ categories = ["database"]
bench = false

[features]
default = ["tracing", "tokio-comp"]

## Include redis storage
redis = ["apalis-redis"]
## Include Postgres storage
postgres = ["apalis-sql/postgres"]
## Include SQlite storage
sqlite = ["apalis-sql/sqlite"]
## Include MySql storage
mysql = ["apalis-sql/mysql"]
## Include Cron functionality
cron = ["apalis-cron"]
default = ["tracing"]


## Support Tracing 👀
tracing = ["dep:tracing", "dep:tracing-futures"]
Expand All @@ -47,20 +37,8 @@ timeout = ["tower/timeout"]
limit = ["tower/limit"]
## Support filtering jobs based on a predicate
filter = ["tower/filter"]
## Compatibility with async-std and smol runtimes
async-std-comp = [
"apalis-sql?/async-std-comp",
"apalis-redis?/async-std-comp",
"apalis-cron?/async-std-comp",
"async-std",
]
## Compatibility with tokio and actix runtimes
tokio-comp = [
"apalis-sql?/tokio-comp",
"apalis-redis?/tokio-comp",
"apalis-cron?/tokio-comp",
"tokio",
]
## Captures panics in executions and convert them to errors
catch-panic = []

layers = [
"sentry",
Expand All @@ -70,35 +48,16 @@ layers = [
"timeout",
"limit",
"filter",
"catch-panic",
]

docsrs = ["document-features"]

[dependencies.apalis-redis]
version = "0.5.5"
optional = true
path = "./packages/apalis-redis"
default-features = false

[dependencies.apalis-sql]

version = "0.5.5"
features = ["migrate"]
optional = true
default-features = false
path = "./packages/apalis-sql"

[dependencies.apalis-core]
version = "0.5.5"
version = "0.6.0"
default-features = false
path = "./packages/apalis-core"

[dependencies.apalis-cron]
version = "0.5.5"
optional = true
default-features = false
path = "./packages/apalis-cron"

[dependencies.document-features]
version = "0.2"
optional = true
Expand All @@ -116,22 +75,24 @@ pprof = { version = "0.14", features = ["flamegraph"] }
paste = "1.0.14"
serde = "1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
apalis = { path = ".", features = ["redis", "sqlite", "postgres", "mysql"] }
redis = { version = "0.25.3", default-features = false, features = [
"script",
"aio",
"connection-manager",
apalis = { path = ".", features = ["limit"] }
apalis-redis = { path = "./packages/apalis-redis" }
apalis-sql = { path = "./packages/apalis-sql", features = [
"postgres",
"mysql",
"sqlite",
] }
redis = { version = "0.27", default-features = false, features = [
"tokio-comp",
"script",
"aio",
"connection-manager",
] }

[dev-dependencies.sqlx]
version = "0.8.1"
default-features = false
features = ["chrono", "mysql", "sqlite", "postgres"]


[[bench]]
name = "storages"
harness = false
features = ["chrono", "mysql", "sqlite", "postgres", "runtime-tokio"]

[workspace]
members = [
Expand All @@ -150,17 +111,22 @@ members = [
"examples/axum",
"examples/prometheus",
"examples/tracing",
# "examples/rest-api",
"examples/async-std-runtime",
"examples/basics",
"examples/redis-with-msg-pack",
"examples/redis-deadpool",
"examples/redis-mq-example",
"examples/cron",
"examples/catch-panic",
"examples/graceful-shutdown",
"examples/unmonitored-worker",
"examples/fn-args",
"examples/persisted-cron",
"examples/rest-api",
]


[dependencies]
tokio = { version = "1", features = [
"rt",
], default-features = false, optional = true }
async-std = { version = "1", optional = true }
tower = { version = "0.5", features = ["util"], default-features = false }
tracing-futures = { version = "0.2.5", optional = true, default-features = false }
sentry-core = { version = "0.34.0", optional = true, default-features = false }
Expand Down
31 changes: 13 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@

- Simple and predictable job handling model.
- Jobs handlers with a macro free API.
- Take full advantage of the [tower] ecosystem of
- Take full advantage of the [`tower`] ecosystem of
middleware, services, and utilities.
- Runtime agnostic - Use tokio, smol etc.
- Optional Web interface to help you manage your jobs.
Expand All @@ -59,43 +59,42 @@ To get started, just add to Cargo.toml

```toml
[dependencies]
apalis = { version = "0.5", features = ["redis"] } # Backends available: postgres, sqlite, mysql, amqp
apalis = { version = "0.6" }
apalis-redis = { version = "0.6" }
# apalis-sql = { version = "0.6", features = ["postgres"] } # or mysql, sqlite
```

## Usage

```rust
use apalis::prelude::*;
use apalis::redis::RedisStorage;
use apalis_redis::{RedisStorage, Config};
use serde::{Deserialize, Serialize};
use anyhow::Result;

#[derive(Debug, Deserialize, Serialize)]
struct Email {
to: String,
}

impl Job for Email {
const NAME: &'static str = "apalis::Email";
}

/// A function that will be converted into a service.
async fn send_email(job: Email, data: Data<usize>) -> Result<(), Error> {
/// execute job
Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
async fn main() -> {
std::env::set_var("RUST_LOG", "debug");
env_logger::init();
let redis_url = std::env::var("REDIS_URL").expect("Missing env variable REDIS_URL");
let storage = RedisStorage::new(redis).await?;
let conn = apalis_redis::connect(redis_url).await.expect("Could not connect");
let storage = RedisStorage::new(conn);
Monitor::new()
.register_with_count(2, {
.register({
WorkerBuilder::new(format!("email-worker"))
.concurrency(2)
.data(0usize)
.with_storage(storage)
.backend(storage)
.build_fn(send_email)
})
.run()
Expand All @@ -122,17 +121,13 @@ async fn produce_route_jobs(storage: &RedisStorage<Email>) -> Result<()> {
## Feature flags

- _tracing_ (enabled by default) — Support Tracing 👀
- _redis_ — Include redis storage
- _postgres_ — Include Postgres storage
- _sqlite_ — Include SQlite storage
- _mysql_ — Include MySql storage
- _cron_ — Include cron job processing
- _sentry_ — Support for Sentry exception and performance monitoring
- _prometheus_ — Support Prometheus metrics
- _retry_ — Support direct retrying jobs
- _timeout_ — Support timeouts on jobs
- _limit_ — 💪 Limit the amount of jobs
- _filter_ — Support filtering jobs based on a predicate
- _catch-panic_ - Catch panics that occur during execution

## Storage Comparison

Expand Down Expand Up @@ -214,5 +209,5 @@ See also the list of [contributors](https://github.com/geofmureithi/apalis/contr
This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

[`tower::Service`]: https://docs.rs/tower/latest/tower/trait.Service.html
[tower]: https://crates.io/crates/tower
[`tower`]: https://crates.io/crates/tower
[`actix`]: https://crates.io/crates/actix
Loading

0 comments on commit 6d6ed27

Please sign in to comment.