Skip to content

Commit

Permalink
[Tables] Azure Data Tables SDK, sync and async code (Azure#12766)
Browse files Browse the repository at this point in the history
* updated design of update and upsert methods

* fixed upsert/update

* reflected change to create_entity

* fixed problems with upsert function

* fixed upsert/update functions on async clients

* pausing on tests

* fixed test failing on HttpResponse

* fixing 2.7 errorZ

* trying to fix 2.7

* fixed tests

* removed problem with unbpund fucntion

* track 1 async sdk

* renamed aio to _aio

* updating comments

* tweaking comments and types for stub gen

* fixed pylint issue

* fixed list_tables to query_tables

* pylint errors

* fixing type errors

* updating samples

* fixing comments

* updating README

* fixing PR comments on keyword and credential

* editing order of docstrings

* fixing queue comments

* fixing queue comments

* delete table has no return response

* changing types to model types

* added keys to dictionary

* changing types to model types

* fixed stroage reference

* fixing docstring reutrn

* fixed cathc and raise same error

* UpdateMode in init

* obj to _

* fixing comments

* fixed metadata return

* fixed comments

* fixing test error

* fixed depedncies

* fixing type

* fixing stubgen error

* changed to ivar

* fixing type error

* editing samples/ fixing pylint

* reformatted service client sampels

* added more official samples

* added in teh create and delete table in table client

* working on client samples

* linked README to samples

* TODO for return types

* changed query entity to get entity

* edited README

* made UpdateMode into an enum

* changed return type of cont token to dict

* fixed readme common uses

* todos created

* fixed continuation token to a dictionary

* fixed continuation error

* changed query to individual entries

* editing README pr comments

* removed AAD from readme

* removed TODO

* getting rid of exposed secret

* added design updates to async client

* stahs

* fixed import

* fixed init file

* fixed import

* added todo for table

* added return type

* editing table client return

* changed table name for table standards (Sean fix)

* editing sas methods

* work in progress - fixing pre api review comments

* updated tests to work with fixes

* re-running tests for verification'

* fixed small errors with changing method names

* fixed storgae_process_error -> table instead of storage, created metadata tag on entity instead of returning etag as part of entity

* made metadata for entities separate from entity itself (see _entity.py)

* updated readme and added regex check for table names on TableClient and TableServiceClient

* oops forgot to import re in the _table_client.py

* metadata test passing

* fixed unimported entity

* added user defined filter option with parameters

* added user defined filter option with parameters

* changed the regex test to match to fit with python2

* fixed linting issues thrown by pylint

* re-ran test

* updating samples part 1

* updating asserts part 2

* removing references to common

* fixing sas on sample

* fixing import statement

* attempting to fix dateutil error

* trying to fix enum in entity property

* fix #2 dateutil

* attempting to fix dateutil error

* fixing test import

* remove unuse import

* added tests for table_client and table_service_client

* adding live recordings

* fix #2 dateutil

* updated dev_requirements, changed test name

* py 2.7 doesnt have timezone - siwtch to UTC

* reverting back to libbas recordings

* reverting to using tzutc -- for now

* updating samples a little with error printing

* replicated tzutc class with class Timezone

* forgot to create table twice for error

* changed regex to fxn in _error.py, added second assertion to tests, updated readme

* updating samples to reflect sdk part 1

* commented out cryptogrpahy from encryption test

* removed commented out code and changed the invalid table name

* updating docstrings

* 'commented out references to cryptography

* commented out pylint errors related to encryption

* updated delete entity to show etag

* still finding encryption

* removed list froma round itempaged

* commenting out test

* commented out tests

* fixed enum on edmtype

* updated update entity

* fixing apiview comments on sas

* added table name to table clinet

* changing ItemPaged to ItemPaged[str[

* changing properties to private

* changes form apiview archboard pre meeting

* added filter type

* TODO on from_string in SAS, fixed error in table querying ItemPaged return

* changed name to _primary_hostname not primay_hopstname

* oops missed the other hostname

* oops missed the other other hostname

* added a client base for tableclient and tableserviceclient, passes all tests as is for sync code

* fixed TODO on sas

* changes to satisfy linter

* tweaking delete

* moved account parsing to the base class, fixed more linting issues

* resolved comments

* updating samples

* updating updatemode

* updating UpdateMode

* importing typing for union

* initial commit for diff

* change to REPLACE'

* initial commit for pulling down current async status, ran a tests live for updated yaml files

* renamed entity tableentity

* allowed select to be array or str

* made Table class

* renamed a new directory for azure-data-tables

* removing storgae reference

* changed query 'r' to read

* removing some commented out code

* updating delete/create table comments

* docstring return type for table

* bullet point auth

* base work for async models, working on passing last few tests, added _models for async entities

* removing packages to exclude

* fixing README

* adding to the blurb in  the readme from the archboard pwpt

* passing most async tests, issues with list_tables query_options currently

* fixing blank line

* making nspkg for data

* removed async

* changed to relative imports

* :had to update naming

* stahs

* stahs

* had to fix list and query tables

* build mad about version

* changed version to match 1.0.0

* changed how results_per_page is passed in, need to verify with Libba and change up sync code if we are correct

* forgot and init file - path was being skipped in build

* changing list and query to align with new api view

* passing the list_tables_with_num_results again

* passes tests when run locally, does not pass in live-mode. the behavior of results_per_page combined with by_page() is still slightly confusing

* fixed test back

* hand to change query to list tables

* temp fix on README error

* had to change msrest version

* fixing pylint error

* Update README.md

* ignorning nspkg readme

* addressed Libbas comments

* regenerated code

* fixed Krista comments on readmes

* fixing Krista comments

* fixed anna's changes

* addressed all of izzys comments

* removing unsued import

* fixing merge errors

* fixing docstrings

* fixewd docstring

* fixing docstrings

* fixing docs

* verifying tests work:

* verifying tests'

* fixed sync for parameters

* fixed async clients parameters

* fixing error for mode

* fixing pylint errors

* re running tests

* added more tests

* fixing models for tests

* fixed init

* had to fix fomratting of async docstring types

* deleting stubgens

* fixing references to storage, and tokens

* working to refactor mentions of storage

* making links to cosmos

* changed naming of header for changelog

* added azure data init to include

* fixed exposed model

* fixed return of create table to be Table object

* fixed return of create table to be Table object

* updated docstring

* added try catch to code calls

* fixed exposed model

* working on fixing error handling:

* not overriding etag

* fixed table entity metadata

* editing table object

* fixing sas protocol

* remvoed tables from aio model

* removing identity

* adding comment to install requirement

* changing docstrings for edm type - making an issue to infer

* fixing pylint errro

* fixing readme

* fixed tests for metadata

* removing links from REAMDE

* fixing broken link

* fixing pylint

* trying to fix bandit fail

* trying to fix bandit fail

* oops forget to disable bandit check

* adding back md5 with nosec

Co-authored-by: Sean Kane <seankane@microsoft.com>
Co-authored-by: Sean Kane <68240067+seankane-msft@users.noreply.github.com>
  • Loading branch information
3 people authored Jul 30, 2020
1 parent 3e91896 commit 48a8133
Show file tree
Hide file tree
Showing 259 changed files with 46,494 additions and 1 deletion.
1 change: 1 addition & 0 deletions eng/.docsettings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ known_content_issues:
- ['sdk/storage/azure-storage-queue/samples/README.md', 'nspkg and common']
- ['sdk/textanalytics/azure-ai-nspkg/README.md', 'nspkg and common']
- ['sdk/eventhub/azure-eventhub/samples/README.md', 'nspkg and common']
- ['sdk/tables/azure-data-nspkg/README.md', 'nspkg and common']

# HISTORY.rst
- ['sdk/core/azure/HISTORY.rst','#4554']
Expand Down
3 changes: 3 additions & 0 deletions sdk/tables/azure-data-nspkg/MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
include *.md
include azure/__init__.py
include azure/data/__init__.py
11 changes: 11 additions & 0 deletions sdk/tables/azure-data-nspkg/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Microsoft Azure Data SDK for Python

This is the Microsoft Azure Data namespace package.

This package is not intended to be installed directly by the end user.

It provides the necessary files for other packages to extend the
azure.data namespace.

If you are looking to install the Azure Data libraries, see the
[azure](https://pypi.python.org/pypi/azure) bundle package.
1 change: 1 addition & 0 deletions sdk/tables/azure-data-nspkg/azure/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
1 change: 1 addition & 0 deletions sdk/tables/azure-data-nspkg/azure/data/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
2 changes: 2 additions & 0 deletions sdk/tables/azure-data-nspkg/setup.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[bdist_wheel]
universal=1
45 changes: 45 additions & 0 deletions sdk/tables/azure-data-nspkg/setup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/usr/bin/env python

# -------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License.txt in the project root for
# license information.
# --------------------------------------------------------------------------
import sys
from setuptools import setup

PACKAGES = []
# Do an empty package on Python 3 and not python_requires, since not everybody is ready
# https://github.com/Azure/azure-sdk-for-python/issues/3447
# https://github.com/Azure/azure-sdk-for-python/issues/3481
if sys.version_info[0] < 3:
PACKAGES = ['azure.data']

setup(
name='azure-data-nspkg',
version='1.0.0',
description="Microsoft Azure Data Namespace Package [Internal]",
long_description=open("README.md", "r").read(),
long_description_content_type="text/markdown",
license='MIT License',
author="Microsoft Corporation",
author_email="azpysdkhelp@microsoft.com",
url="https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/tables",
classifiers=[
"Development Status :: 5 - Production/Stable",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"License :: OSI Approved :: MIT License",
],
zip_safe=False,
packages=PACKAGES,
install_requires=[
'azure-nspkg>=3.0.0',
]
)
6 changes: 6 additions & 0 deletions sdk/tables/azure-data-tables/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Release History

## 12.0.0b1 (Unreleased)



6 changes: 6 additions & 0 deletions sdk/tables/azure-data-tables/MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
include *.md
include azure/__init__.py
include azure/data/__init__.py
include LICENSE.txt
recursive-include tests *.py
recursive-include samples *.py *.md
299 changes: 299 additions & 0 deletions sdk/tables/azure-data-tables/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,299 @@
# Azure Data Tables client library for Python

Azure Data Tables is a NoSQL data storing service that can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS.
Tables scale as needed to support the amount of data inserted, and allow for the storing of data with non-complex accessing.
Tables scale as needed to support the amount of data inserted, and allow for the storing of data with non-complex accessing.
The Azure Data Tables client can be used to access Azure Storage or Cosmos accounts.

Common uses of Azure Data Tables include:

* Storing structured data in the form of tables
* Quickly querying data using a clustered index

[Source code](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk) | [Package (PyPI)](https://pypi.org) | [API reference documentation](https://aka.ms/azsdk/python/tables/docs) | [Product documentation](https://docs.microsoft.com/azure/storage/) | [Samples](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk)

## Getting started

### Prerequisites
* Python 2.7, or 3.5 or later is required to use this package.
* You must have an [Azure subscription](https://azure.microsoft.com/free/) and an
[Azure storage account](https://docs.microsoft.com/azure/storage/common/storage-account-overview) to use this package
or you must have a [Azure Cosmos Account](https://docs.microsoft.com/en-us/azure/cosmos-db/account-overview).

### Install the package
Install the Azure Data Tables client library for Python with [pip](https://pypi.org/project/pip/):

```bash
pip install --pre azure-data-tables
```

### Create a storage account
If you wish to create a new cosmos storage account, you can use the [Azure Cosmos DB](https://docs.microsoft.com/en-us/azure/cosmos-db/create-cosmosdb-resources-portal)
If you wish to create a new storage account, you can use the
[Azure Portal](https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal),
[Azure PowerShell](https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-powershell),
or [Azure CLI](https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-cli):

```bash
# Create a new resource group to hold the storage account -
# if using an existing resource group, skip this step
az group create --name MyResourceGroup --location westus2

# Create the storage account
az storage account create -n mystorageaccount -g MyResourceGroup
```

### Create the client
The Azure Data Tables client library for Python allows you to interact with two types of resources: the
account and tables, and entities. Interaction with these resources starts with an instance of a [client](#clients).
To create a client object, you will need the account's table service endpoint URL and a credential that allows
you to access the account:

```python
from azure.data.tables import TableServiceClient

service = TableServiceClient(account_url="https://<myaccount>.table.core.windows.net/", credential=credential)
```

#### Looking up the account URL
You can find the account's table service URL using the
[Azure Portal](https://docs.microsoft.com/azure/storage/common/storage-account-overview#storage-account-endpoints),
[Azure PowerShell](https://docs.microsoft.com/powershell/module/az.storage/get-azstorageaccount),
or [Azure CLI](https://docs.microsoft.com/cli/azure/storage/account?view=azure-cli-latest#az-storage-account-show):

```bash
# Get the table service URL for the account
az storage account show -n mystorageaccount -g MyResourceGroup --query "primaryEndpoints.table"
```

#### Types of credentials
The `credential` parameter may be provided in a number of different forms, depending on the type of
[authorization](https://docs.microsoft.com/azure/storage/common/storage-auth) you wish to use:
1. To use a [shared access signature (SAS) token](https://docs.microsoft.com/azure/storage/common/storage-sas-overview),
provide the token as a string. If your account URL includes the SAS token, omit the credential parameter.
You can generate a SAS token from the Azure Portal under "Shared access signature" or use one of the `generate_sas()`
functions to create a sas token for the account or table:

```python
from datetime import datetime, timedelta
from azure.data.tables import TableServiceClient, generate_account_sas, ResourceTypes, AccountSasPermissions

sas_token = generate_account_sas(
account_name="<account-name>",
account_key="<account-access-key>",
resource_types=ResourceTypes(service=True),
permission=AccountSasPermissions(read=True),
expiry=datetime.utcnow() + timedelta(hours=1)
)

table_service_client = TableServiceClient(account_url="https://<my_account_name>.table.core.windows.net", credential=sas_token)
```

2. To use an account [shared key](https://docs.microsoft.com/rest/api/storageservices/authenticate-with-shared-key/)
(aka account key or access key), provide the key as a string. This can be found in the Azure Portal under the "Access Keys"
section or by running the following Azure CLI command:

```az storage account keys list -g MyResourceGroup -n mystorageaccount```

Use the key as the credential parameter to authenticate the client:
```python
from azure.data.tables import TableServiceClient
service = TableServiceClient(account_url="https://<my_account_name>.table.core.windows.net", credential="<account_access_key>")
```

#### Creating the client from a connection string
Depending on your use case and authorization method, you may prefer to initialize a client instance with a
connection string instead of providing the account URL and credential separately. To do this, pass the
connection string to the client's `from_connection_string` class method:

```python
from azure.data.tables import TableServiceClient

connection_string = "DefaultEndpointsProtocol=https;AccountName=xxxx;AccountKey=xxxx;EndpointSuffix=core.windows.net"
service = TableServiceClient.from_connection_string(conn_str=connection_string)
```

The connection string to your account can be found in the Azure Portal under the "Access Keys" section or by running the following CLI command:

```bash
az storage account show-connection-string -g MyResourceGroup -n mystorageaccount
```

## Key concepts
The following components make up the Azure Data Tables Service:
* The account
* A table within the account, which contains a set of entities
* An entity within a table, as a dictionary

The Azure Data Tables client library for Python allows you to interact with each of these components through the
use of a dedicated client object.

### Clients
Two different clients are provided to to interact with the various components of the Table Service:
1. [TableServiceClient](https://aka.ms/azsdk/python/tables/docs) -
this client represents interaction with the Azure account itself, and allows you to acquire preconfigured
client instances to access the tables within. It provides operations to retrieve and configure the account
properties as well as query, create, and delete tables within the account. To perform operations on a specific table,
retrieve a client using the `get_table_client` method.
2. [TableClient](https://aka.ms/azsdk/python/tables/docs) -
this client represents interaction with a specific table (which need not exist yet). It provides operations to
create, delete, or update a table and includes operations to query, get, and upsert entities
within it.

### Entities
* **Create** - Adds an entity to the table.
* **Delete** - Deletes an entity from the table.
* **Update** - Updates an entities information by either merging or replacing the existing entity.
* **Query** - Queries existing entities in a table based off of the QueryOptions (OData).
* **Get** - Gets a specific entity from a table by partition and row key.
* **Upsert** - Merges or replaces an entity in a table, or if the entity does not exist, inserts the entity.

## Examples

The following sections provide several code snippets covering some of the most common Table tasks, including:

* [Creating a table](#creating-a-table "Creating a table")
* [Creating entities](#creating-entities "Creating entities")
* [Querying entities](#querying-entities "Querying entities")


### Creating a table
Create a table in your account

```python
from azure.data.tables import TableServiceClient

table_service_client = TableServiceClient.from_connection_string(conn_str="<connection_string>")
table_service_client.create_table(table_name="myTable")
```

### Creating entities
Create entities in the table

```python
from azure.data.tables import TableClient

my_entity = {'PartitionKey':'part','RowKey':'row'}

table_client = TableClient.from_connection_string(conn_str="<connection_string>", table_name="myTable")
entity = table_client.create_entity(entity=my_entity)
```

### Querying entities
Querying entities in the table

```python
from azure.data.tables import TableClient

my_filter = "text eq Marker"

table_client = TableClient.from_connection_string(conn_str="<connection_string>", table_name="mytable")
entity = table_client.query_entities(filter=my_filter)
```

## Optional Configuration

Optional keyword arguments can be passed in at the client and per-operation level. The azure-core [reference documentation](https://azuresdkdocs.blob.core.windows.net/$web/python/azure-core/latest/azure.core.html) describes available configurations for retries, logging, transport protocols, and more.


### Retry Policy configuration

Use the following keyword arguments when instantiating a client to configure the retry policy:

* __retry_total__ (int): Total number of retries to allow. Takes precedence over other counts.
Pass in `retry_total=0` if you do not want to retry on requests. Defaults to 10.
* __retry_connect__ (int): How many connection-related errors to retry on. Defaults to 3.
* __retry_read__ (int): How many times to retry on read errors. Defaults to 3.
* __retry_status__ (int): How many times to retry on bad status codes. Defaults to 3.
* __retry_to_secondary__ (bool): Whether the request should be retried to secondary, if able.
This should only be enabled of RA-GRS accounts are used and potentially stale data can be handled.
Defaults to `False`.

### Other client / per-operation configuration

Other optional configuration keyword arguments that can be specified on the client or per-operation.

**Client keyword arguments:**

* __connection_timeout__ (int): Optionally sets the connect and read timeout value, in seconds.
* __transport__ (Any): User-provided transport to send the HTTP request.

**Per-operation keyword arguments:**

* __raw_response_hook__ (callable): The given callback uses the response returned from the service.
* __raw_request_hook__ (callable): The given callback uses the request before being sent to service.
* __client_request_id__ (str): Optional user specified identification of the request.
* __user_agent__ (str): Appends the custom value to the user-agent header to be sent with the request.
* __logging_enable__ (bool): Enables logging at the DEBUG level. Defaults to False. Can also be passed in at
the client level to enable it for all requests.
* __headers__ (dict): Pass in custom headers as key, value pairs. E.g. `headers={'CustomValue': value}`


## Troubleshooting
### General
Azure Data Tables clients raise exceptions defined in [Azure Core](https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/core/azure-core/README.md).
All Table service operations will throw a `HttpResponseError` on failure with helpful [error codes](https://docs.microsoft.com/en-us/rest/api/storageservices/table-service-error-codes).

### Logging
This library uses the standard
[logging](https://docs.python.org/3/library/logging.html) library for logging.
Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO
level.

Detailed DEBUG level logging, including request/response bodies and unredacted
headers, can be enabled on a client with the `logging_enable` argument:
```python
import sys
import logging
from azure.data.tables import TableServiceClient

# Create a logger for the 'azure.data.tables' SDK
logger = logging.getLogger('azure.data.tables')
logger.setLevel(logging.DEBUG)

# Configure a console output
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)

# This client will log detailed information about its HTTP sessions, at DEBUG level
service_client = TableServiceClient.from_connection_string("your_connection_string", logging_enable=True)
```

Similarly, `logging_enable` can enable detailed logging for a single operation,
even when it isn't enabled for the client:
```py
service_client.get_service_stats(logging_enable=True)
```

## Next steps

Get started with our [Table samples](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk).

Several Azure Data Tables Python SDK samples are available to you in the SDK's GitHub repository. These samples provide example code for additional scenarios commonly encountered while working with Tables:

* [table_samples_authentication.py](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk) - Examples found in this article:
* From a connection string
* From a shared access key
* From a shared access signature token
* [table_samples_service.py](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk) - Examples found in this article:
* Get and set service properties
* List tables in a account
* Create and delete a table from the service
* Get the TableClient
* [table_samples_client.py](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk) - Examples found in this article:
* Client creation
* Create a table
* Create and Delete entities
* Query entities
* Update entities
* Upsert entities

### Additional documentation
For more extensive documentation on Azure Data Tables, see the [Azure Data Tables documentation](https://docs.microsoft.com/azure/storage/tables/) on docs.microsoft.com.

## Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
1 change: 1 addition & 0 deletions sdk/tables/azure-data-tables/azure/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__path__ = __import__('pkgutil').extend_path(__path__, __name__) # type: str
1 change: 1 addition & 0 deletions sdk/tables/azure-data-tables/azure/data/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__path__ = __import__('pkgutil').extend_path(__path__, __name__) # type: str
Loading

0 comments on commit 48a8133

Please sign in to comment.