Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update building connector tutorial incorrect links and typo #11769

Merged
merged 2 commits into from
Apr 7, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This tutorial will assume that you already have a working source. If you do not,

First we need to identify a given stream in the Source as supporting incremental. This information is declared in the catalog that the `discover` method returns. You will notice in the stream object contains a field called `supported_sync_modes`. If we are adding incremental to an existing stream, we just need to add `"incremental"` to that array. This tells Airbyte that this stream can either be synced in an incremental fashion. In practice, this will mean that in the UI, a user will have the ability to configure this type of sync.

In the example we used in the Toy Connector tutorial, the `discover` method would not look like this. Note: that "incremental" has been added to the `support_sync_modes` array. We also set `source_defined_cursor` to `True` to declare that the Source knows what field to use for the cursor, in this case the date field, and does not require user input. Nothing else has changed.
In the example we used in the Toy Connector tutorial, the `discover` method would not look like this. Note: that "incremental" has been added to the `supported_sync_modes` array. We also set `source_defined_cursor` to `True` to declare that the Source knows what field to use for the cursor, in this case the date field, and does not require user input. Nothing else has changed.

```python
def discover():
Expand Down Expand Up @@ -40,7 +40,7 @@ def discover():

Next we will adapt the `read` method that we wrote previously. We need to change three things.

First, we need to pass it information about what data was replicated in the previous sync. In Airbyte this is called a `state` object. The structure of the state object is determined by Source. This means that each Source can construct a state object that makes sense to it and does not need to worry about adhering to any other convention. That being said, a pretty typical structure for a state object is a map of stream name to the last value in the cursor field for that stream.
First, we need to pass it information about what data was replicated in the previous sync. In Airbyte this is called a `state` object. The structure of the state object is determined by the Source. This means that each Source can construct a state object that makes sense to it and does not need to worry about adhering to any other convention. That being said, a pretty typical structure for a state object is a map of stream name to the last value in the cursor field for that stream.

In this case we might choose something like this:

Expand Down Expand Up @@ -93,7 +93,7 @@ def read(config, catalog, state):
log("Failure occurred when calling Polygon.io API")
sys.exit(1)
else:
# Stock prices are returned sorted by by date in ascending order
# Stock prices are returned sorted by date in ascending order
# We want to output them one by one as AirbyteMessages
response_json = response.json()
if response_json["resultsCount"] > 0:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This tutorial walks you through building a simple Airbyte source without using a

* [The Airbyte Specification](../../understanding-airbyte/airbyte-specification.md) and the interface implemented by a source connector
* [The AirbyteCatalog](../../understanding-airbyte/beginners-guide-to-catalog.md)
* [Packaging your connector](https://docs.airbyte.io/connector-development#1-implement-and-package-the-connector)
* [Packaging your connector](https://docs.airbyte.io/connector-development#1.-implement-and-package-the-connector)
* [Testing your connector](../testing-connectors/source-acceptance-tests-reference.md)

At the end of this tutorial, you will have a working source that you will be able to use in the Airbyte UI.
Expand Down Expand Up @@ -113,13 +113,13 @@ To contact the stock ticker API, we need two things:
1. Which stock ticker we're interested in
2. The API key to use when contacting the API \(you can obtain a free API token from [Polygon.io](https://polygon.io/dashboard/signup) free plan\)

For reference, the API docs we'll be using [can be found here](https://polygon.io/docs/stocks/get_v2_aggs_grouped_locale_us_market_stocks__date).
For reference, the API docs we'll be using [can be found here](https://polygon.io/docs/stocks/get_v2_aggs_ticker__stocksticker__range__multiplier___timespan___from___to).

Let's create a [JSONSchema](http://json-schema.org/) file `spec.json` encoding these two requirements:

```javascript
{
"documentationUrl": "https://polygon.io/docs/stocks/get_v2_aggs_grouped_locale_us_market_stocks__date",
"documentationUrl": "https://polygon.io/docs/stocks/get_v2_aggs_ticker__stocksticker__range__multiplier___timespan___from___to",
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
Expand Down Expand Up @@ -232,7 +232,7 @@ Now if we run `python source.py spec` we should see the specification printed ou

```bash
python source.py spec
{"type": "SPEC", "spec": {"documentationUrl": "https://polygon.io/docs/stocks/get_v2_aggs_grouped_locale_us_market_stocks__date", "connectionSpecification": {"$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "required": ["stock_ticker", "api_key"], "additionalProperties": false, "properties": {"stock_ticker": {"type": "string", "title": "Stock Ticker", "description": "The stock ticker to track", "examples": ["AAPL", "TSLA", "AMZN"]}, "api_key": {"type": "string", "description": "The Polygon.io Stocks API key to use to hit the API.", "airbyte_secret": true}}}}}
{"type": "SPEC", "spec": {"documentationUrl": "https://polygon.io/docs/stocks/get_v2_aggs_ticker__stocksticker__range__multiplier___timespan___from___to", "connectionSpecification": {"$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "required": ["stock_ticker", "api_key"], "additionalProperties": false, "properties": {"stock_ticker": {"type": "string", "title": "Stock Ticker", "description": "The stock ticker to track", "examples": ["AAPL", "TSLA", "AMZN"]}, "api_key": {"type": "string", "description": "The Polygon.io Stocks API key to use to hit the API.", "airbyte_secret": true}}}}}
```

We've implemented the first command! Three more and we'll have a working connector.
Expand Down Expand Up @@ -556,7 +556,7 @@ def read(config, catalog):
log("Failure occurred when calling Polygon.io API")
sys.exit(1)
else:
# Stock prices are returned sorted by by date in ascending order
# Stock prices are returned sorted by date in ascending order
# We want to output them one by one as AirbyteMessages
results = response.json()["results"]
for result in results:
Expand Down Expand Up @@ -719,7 +719,7 @@ def read(config, catalog):
log("Failure occurred when calling Polygon.io API")
sys.exit(1)
else:
# Stock prices are returned sorted by by date in ascending order
# Stock prices are returned sorted by date in ascending order
# We want to output them one by one as AirbyteMessages
results = response.json()["results"]
for result in results:
Expand Down Expand Up @@ -917,7 +917,7 @@ to run any of our commands, we'll need to mount all the inputs into the Docker c

```bash
$ docker run airbyte/source-stock-ticker-api:dev spec
{"type": "SPEC", "spec": {"documentationUrl": "https://polygon.io/docs/stocks/get_v2_aggs_grouped_locale_us_market_stocks__date", "connectionSpecification": {"$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "required": ["stock_ticker", "api_key"], "additionalProperties": false, "properties": {"stock_ticker": {"type": "string", "title": "Stock Ticker", "description": "The stock ticker to track", "examples": ["AAPL", "TSLA", "AMZN"]}, "api_key": {"type": "string", "description": "The Polygon.io Stocks API key to use to hit the API.", "airbyte_secret": true}}}}}
{"type": "SPEC", "spec": {"documentationUrl": "https://polygon.io/docs/stocks/get_v2_aggs_ticker__stocksticker__range__multiplier___timespan___from___to", "connectionSpecification": {"$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "required": ["stock_ticker", "api_key"], "additionalProperties": false, "properties": {"stock_ticker": {"type": "string", "title": "Stock Ticker", "description": "The stock ticker to track", "examples": ["AAPL", "TSLA", "AMZN"]}, "api_key": {"type": "string", "description": "The Polygon.io Stocks API key to use to hit the API.", "airbyte_secret": true}}}}}

$ docker run -v $(pwd)/secrets/valid_config.json:/data/config.json airbyte/source-stock-ticker-api:dev check --config /data/config.json
{'type': 'CONNECTION_STATUS', 'connectionStatus': {'status': 'SUCCEEDED'}}
Expand Down Expand Up @@ -1163,4 +1163,4 @@ Like we mention at the beginning of the tutorial, this guide is meant more for u
### Language specific helpers
* [Building a Python Source](https://docs.airbyte.com/connector-development/tutorials/building-a-python-source)
* [Building a Python Destination](https://docs.airbyte.com/connector-development/tutorials/building-a-python-destination)
* [Building a Java Destination](https://docs.airbyte.com/connector-development/tutorials/building-a-java-destination)
* [Building a Java Destination](https://docs.airbyte.com/connector-development/tutorials/building-a-java-destination)