Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faker source gains purchases and products #13248

Merged
merged 10 commits into from
May 27, 2022
Merged

Conversation

evantahler
Copy link
Contributor

@evantahler evantahler commented May 27, 2022

Part of #13201

Screen Shot 2022-05-26 at 4 46 37 PM

Screen Shot 2022-05-26 at 4 48 27 PM

The faker command now has 3 streams: Users, Products, and Purchases. The list of Products is a static list of 100 cars. The list of purchases is generated randomly against certain percentages. Each purchase also has 3 timestamps: added_to_cart, purchased, and returned - these are always in the proper chronological order, and happen with various percentages: 70% likely to move from cart -> purchase and 30% likely to return the item if it was purchased.

This dataset was designed to allow for exploration and transformation with dbt and various reporting tools to produce data like "user lifetime value" and "avg rate of return".

SELECT 
	users.id as user_id
	, mail as email
	, name
	, COUNT(purchases.*) as count_purchases
	, SUM(products.price) as ltv
from users
join purchases on users.id = purchases.user_id
join products on purchases.product_id = products.id
group by
	users.id, users.mail, users.name
order by ltv desc
;

There are also now additional configuration options to control the size of each read and stream-slice (page). This will prove helpful for debugging things (like checkpointing) down the road.

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels May 27, 2022
@evantahler evantahler changed the title Evan/faker gets purchases @evantahler Faker source gains purchases and products May 27, 2022
@evantahler evantahler changed the title @evantahler Faker source gains purchases and products Faker source gains purchases and products May 27, 2022
@evantahler
Copy link
Contributor Author

cc @chrisroseairbyte

@evantahler evantahler marked this pull request as ready for review May 27, 2022 01:12
@codecov
Copy link

codecov bot commented May 27, 2022

Codecov Report

❗ No coverage uploaded for pull request base (master@0fd866c). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #13248   +/-   ##
=========================================
  Coverage          ?   98.40%           
=========================================
  Files             ?        2           
  Lines             ?      125           
  Branches          ?        0           
=========================================
  Hits              ?      123           
  Misses            ?        2           
  Partials          ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0fd866c...8e91c1a. Read the comment docs.

Copy link
Contributor

@Phlair Phlair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, who needs real data anyway!? 👍

couple of nits and small suggestions so no need for a second review, approving with this one.

@evantahler
Copy link
Contributor Author

evantahler commented May 27, 2022

/publish connector=connectors/source-faker

🕑 connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/2398347015
❌ Failed to publish connectors/source-faker
❌ Couldn't auto-bump version for connectors/source-faker

@evantahler
Copy link
Contributor Author

evantahler commented May 27, 2022

/test connector=connectors/source-faker

🕑 connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/2398955023
✅ connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/2398955023
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                       Stmts   Miss  Cover
----------------------------------------------
source_faker/__init__.py       2      0   100%
source_faker/source.py       123      2    98%
----------------------------------------------
TOTAL                        125      2    98%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 21 passed, 1 skipped in 16.65s ========================

@evantahler
Copy link
Contributor Author

evantahler commented May 27, 2022

/publish connector=connectors/source-faker

🕑 connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/2399089637
🚀 Successfully published connectors/source-faker
🚀 Auto-bumped version for connectors/source-faker
✅ connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/2399089637

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets May 27, 2022 23:01 Inactive
@evantahler evantahler requested a review from a team as a code owner May 27, 2022 23:05
@evantahler evantahler temporarily deployed to more-secrets May 27, 2022 23:07 Inactive
@evantahler evantahler merged commit 40d1dc1 into master May 27, 2022
@evantahler evantahler deleted the evan/faker-gets-purchases branch May 27, 2022 23:15
jscottpolevault pushed a commit to jscottpolevault/airbyte that referenced this pull request Jun 1, 2022
* Faker source gains purchases and products

* ranges start at 1

* fix nits

* lint

* user_ids start at 1

* v0.1.3

* fix IDs in test output recods

* auto-bump connector version

* lint

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation team/extensibility
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants