Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/apache beam samples #6653

Closed

Conversation

mmistroni
Copy link
Contributor

Pull Request Template for OpenBB Developers

  1. Title:

    • Sample showing how to call OBB code from Apache Beam task
  2. Why? (1-3 sentences or a bullet point list):

    • Useful to show some automation as OBB code can be ran by subrocesses running on a pipeline."
  3. What? (1-3 sentences or a bullet point list):

    • Have created a simple unit test that ran an OBB Fetcher within an Apache Beam pipeline
  4. Impact (1-2 sentences or a bullet point list):

    -N.A

  5. Testing Done:

    • The code ran in a unit test

@deeleeramone
Copy link
Contributor

This is cool, thanks for the PR. I still need to do my homework to be able to fully test drive this puppy, so correct me if I'm wrong, I believe we can simplify the requirements to just openbb-yfinance, and we wouldn't even need a version number. Can you confirm? You shouldn't have to change any code for that, and it will shrink the overall footprint of the environment.

@mmistroni
Copy link
Contributor Author

mmistroni commented Sep 6, 2024 via email

Copy link
Contributor

@deeleeramone deeleeramone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this unique example, I added a few minor comments and one question.

I followed the instructions and everything worked as-expected.

@@ -0,0 +1,49 @@
import unittest
import os
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused import here.

# You need to create a Conda environment (or a virtual env) using requirements.txt in this directory
# The script exercise 3 OBB endpoints, all of which require no credentials
# Run the test from the main directory
(obb-dataflow) C:\Users\Marco And Sofia\tmp\OBBDataflowSample>python -m unittest discover
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Path here should be /tests.

Does the discover argument do something in Windows? On Mac, I get:

WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['discover']

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pytests on windows, with discover, automatically find the unit test,. its just me being lazy and wanted to keep setup as minimal as possible, i'll amend instr

@@ -0,0 +1,2 @@
apache-beam
openbb==4.3.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just make this, openbb-yfinance. Beam is a pretty heavy package by itself, and we don't need any of the router/app logic to run the sample.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

@@ -0,0 +1,13 @@
#### Obb Dataflow sample
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rename this file as a .md Markdown file. The First line is made a title with a single #.
Obb -> OBB.
sample -> Sample



This is a sample how to invoke OBB fetchers in an Apache Beam pipeline. (GCP dataflow is build on Apache Beam)
-- Pre-requisite
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Markdown, we'll convert -- to ##. The items below get formatted as a bullet-point list by swapping # for -.

If you have a handy link to one of their quick start guides, let's include it here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@deeleeramone
Copy link
Contributor

Closing because reopened as #6679

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants