Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add duckdb offline store #3981

Merged
merged 3 commits into from
Mar 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,19 @@ test-python-universal-athena:
not test_snowflake" \
sdk/python/tests

test-python-universal-duckdb:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.duckdb_repo_configuration \
FEAST_USAGE=False IS_TEST=True \
python -m pytest -n 8 --integration \
-k "not test_nullable_online_store and \
not gcs_registry and \
not s3_registry and \
not test_snowflake and \
not bigquery and \
not test_spark_materialization_consistency" \
sdk/python/tests

test-python-universal-postgres-offline:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.postgres_repo_configuration \
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import ibis
from pydantic import StrictStr

from feast.infra.offline_stores.contrib.ibis_offline_store.ibis import IbisOfflineStore
from feast.repo_config import FeastConfigBaseModel


class DuckDBOfflineStoreConfig(FeastConfigBaseModel):
type: StrictStr = "duckdb"
# """ Offline store type selector"""


class DuckDBOfflineStore(IbisOfflineStore):
@staticmethod
def setup_ibis_backend():
# there's no need to call setup as duckdb is default ibis backend
ibis.set_backend("duckdb")
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
from feast.infra.offline_stores.contrib.duckdb_offline_store.duckdb import (
DuckDBOfflineStoreConfig,
)
from tests.integration.feature_repos.universal.data_sources.file import ( # noqa: E402
FileDataSourceCreator,
)


class DuckDBDataSourceCreator(FileDataSourceCreator):
def create_offline_store_config(self):
self.duckdb_offline_store_config = DuckDBOfflineStoreConfig()
return self.duckdb_offline_store_config


AVAILABLE_OFFLINE_STORES = [
("local", DuckDBDataSourceCreator),
]

AVAILABLE_ONLINE_STORES = {"sqlite": ({"type": "sqlite"}, None)}
Loading
Loading