Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moving all Bigtable row filters into their own module. #1602

Merged
merged 1 commit into from
Mar 11, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions docs/bigtable-row-filters.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
Bigtable Row Filters
====================

It is possible to use a
:class:`RowFilter <gcloud.bigtable.row_filters.RowFilter>`
when adding mutations to a
:class:`ConditionalRow <gcloud.bigtable.row.ConditionalRow>` and when
reading row data with :meth:`read_row() <gcloud.bigtable.table.Table.read_row>`
:meth:`read_rows() <gcloud.bigtable.table.Table.read_rows>`.

As laid out in the `RowFilter definition`_, the following basic filters
are provided:

* :class:`SinkFilter <.row_filters.SinkFilter>`
* :class:`PassAllFilter <.row_filters.PassAllFilter>`
* :class:`BlockAllFilter <.row_filters.BlockAllFilter>`
* :class:`RowKeyRegexFilter <.row_filters.RowKeyRegexFilter>`
* :class:`RowSampleFilter <.row_filters.RowSampleFilter>`
* :class:`FamilyNameRegexFilter <.row_filters.FamilyNameRegexFilter>`
* :class:`ColumnQualifierRegexFilter <.row_filters.ColumnQualifierRegexFilter>`
* :class:`TimestampRangeFilter <.row_filters.TimestampRangeFilter>`
* :class:`ColumnRangeFilter <.row_filters.ColumnRangeFilter>`
* :class:`ValueRegexFilter <.row_filters.ValueRegexFilter>`
* :class:`ValueRangeFilter <.row_filters.ValueRangeFilter>`
* :class:`CellsRowOffsetFilter <.row_filters.CellsRowOffsetFilter>`
* :class:`CellsRowLimitFilter <.row_filters.CellsRowLimitFilter>`
* :class:`CellsColumnLimitFilter <.row_filters.CellsColumnLimitFilter>`
* :class:`StripValueTransformerFilter <.row_filters.StripValueTransformerFilter>`
* :class:`ApplyLabelFilter <.row_filters.ApplyLabelFilter>`

In addition, these filters can be combined into composite filters with

* :class:`RowFilterChain <.row_filters.RowFilterChain>`
* :class:`RowFilterUnion <.row_filters.RowFilterUnion>`
* :class:`ConditionalRowFilter <.row_filters.ConditionalRowFilter>`

These rules can be nested arbitrarily, with a basic filter at the lowest
level. For example:

.. code:: python

# Filter in a specified column (matching any column family).
col1_filter = ColumnQualifierRegexFilter(b'columnbia')

# Create a filter to label results.
label1 = u'label-red'
label1_filter = ApplyLabelFilter(label1)

# Combine the filters to label all the cells in columnbia.
chain1 = RowFilterChain(filters=[col1_filter, label1_filter])

# Create a similar filter to label cells blue.
col2_filter = ColumnQualifierRegexFilter(b'columnseeya')
label2 = u'label-blue'
label2_filter = ApplyLabelFilter(label2)
chain2 = RowFilterChain(filters=[col2_filter, label2_filter])

# Bring our two labeled columns together.
row_filter = RowFilterUnion(filters=[chain1, chain2])

----

.. automodule:: gcloud.bigtable.row_filters
:members:
:undoc-members:
:show-inheritance:

.. _RowFilter definition: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/1ff247c2e3b7cd0a2dd49071b2d95beaf6563092/bigtable-protos/src/main/proto/google/bigtable/v1/bigtable_data.proto#L195
60 changes: 0 additions & 60 deletions docs/bigtable-row.rst
Original file line number Diff line number Diff line change
@@ -1,67 +1,7 @@
Bigtable Row
============

It is possible to use a :class:`RowFilter <gcloud.bigtable.row.RowFilter>`
when adding mutations to a
:class:`ConditionalRow <gcloud.bigtable.row.ConditionalRow>` and when
reading row data with :meth:`read_row() <gcloud.bigtable.table.Table.read_row>`
:meth:`read_rows() <gcloud.bigtable.table.Table.read_rows>`.

As laid out in the `RowFilter definition`_, the following basic filters
are provided:

* :class:`SinkFilter <.row.SinkFilter>`
* :class:`PassAllFilter <.row.PassAllFilter>`
* :class:`BlockAllFilter <.row.BlockAllFilter>`
* :class:`RowKeyRegexFilter <.row.RowKeyRegexFilter>`
* :class:`RowSampleFilter <.row.RowSampleFilter>`
* :class:`FamilyNameRegexFilter <.row.FamilyNameRegexFilter>`
* :class:`ColumnQualifierRegexFilter <.row.ColumnQualifierRegexFilter>`
* :class:`TimestampRangeFilter <.row.TimestampRangeFilter>`
* :class:`ColumnRangeFilter <.row.ColumnRangeFilter>`
* :class:`ValueRegexFilter <.row.ValueRegexFilter>`
* :class:`ValueRangeFilter <.row.ValueRangeFilter>`
* :class:`CellsRowOffsetFilter <.row.CellsRowOffsetFilter>`
* :class:`CellsRowLimitFilter <.row.CellsRowLimitFilter>`
* :class:`CellsColumnLimitFilter <.row.CellsColumnLimitFilter>`
* :class:`StripValueTransformerFilter <.row.StripValueTransformerFilter>`
* :class:`ApplyLabelFilter <.row.ApplyLabelFilter>`

In addition, these filters can be combined into composite filters with

* :class:`RowFilterChain <.row.RowFilterChain>`
* :class:`RowFilterUnion <.row.RowFilterUnion>`
* :class:`ConditionalRowFilter <.row.ConditionalRowFilter>`

These rules can be nested arbitrarily, with a basic filter at the lowest
level. For example:

.. code:: python

# Filter in a specified column (matching any column family).
col1_filter = ColumnQualifierRegexFilter(b'columnbia')

# Create a filter to label results.
label1 = u'label-red'
label1_filter = ApplyLabelFilter(label1)

# Combine the filters to label all the cells in columnbia.
chain1 = RowFilterChain(filters=[col1_filter, label1_filter])

# Create a similar filter to label cells blue.
col2_filter = ColumnQualifierRegexFilter(b'columnseeya')
label2 = u'label-blue'
label2_filter = ApplyLabelFilter(label2)
chain2 = RowFilterChain(filters=[col2_filter, label2_filter])

# Bring our two labeled columns together.
row_filter = RowFilterUnion(filters=[chain1, chain2])

----

.. automodule:: gcloud.bigtable.row
:members:
:undoc-members:
:show-inheritance:

.. _RowFilter definition: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/1ff247c2e3b7cd0a2dd49071b2d95beaf6563092/bigtable-protos/src/main/proto/google/bigtable/v1/bigtable_data.proto#L195
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@
bigtable-table
bigtable-column-family
bigtable-row
bigtable-row-filters
bigtable-row-data
happybase-connection
happybase-pool
Expand Down
1 change: 0 additions & 1 deletion gcloud/bigquery/test_job.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# pylint: disable=C0302
# Copyright 2015 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down
2 changes: 1 addition & 1 deletion gcloud/bigtable/happybase/batch.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
import six

from gcloud._helpers import _datetime_from_microseconds
from gcloud.bigtable.row import TimestampRange
from gcloud.bigtable.row_filters import TimestampRange


_WAL_SENTINEL = object()
Expand Down
16 changes: 8 additions & 8 deletions gcloud/bigtable/happybase/table.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,14 @@
from gcloud.bigtable.happybase.batch import _get_column_pairs
from gcloud.bigtable.happybase.batch import _WAL_SENTINEL
from gcloud.bigtable.happybase.batch import Batch
from gcloud.bigtable.row import CellsColumnLimitFilter
from gcloud.bigtable.row import ColumnQualifierRegexFilter
from gcloud.bigtable.row import FamilyNameRegexFilter
from gcloud.bigtable.row import RowFilterChain
from gcloud.bigtable.row import RowFilterUnion
from gcloud.bigtable.row import RowKeyRegexFilter
from gcloud.bigtable.row import TimestampRange
from gcloud.bigtable.row import TimestampRangeFilter
from gcloud.bigtable.row_filters import CellsColumnLimitFilter
from gcloud.bigtable.row_filters import ColumnQualifierRegexFilter
from gcloud.bigtable.row_filters import FamilyNameRegexFilter
from gcloud.bigtable.row_filters import RowFilterChain
from gcloud.bigtable.row_filters import RowFilterUnion
from gcloud.bigtable.row_filters import RowKeyRegexFilter
from gcloud.bigtable.row_filters import TimestampRange
from gcloud.bigtable.row_filters import TimestampRangeFilter
from gcloud.bigtable.table import Table as _LowLevelTable


Expand Down
2 changes: 1 addition & 1 deletion gcloud/bigtable/happybase/test_batch.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ def test_constructor_defaults(self):

def test_constructor_explicit(self):
from gcloud._helpers import _datetime_from_microseconds
from gcloud.bigtable.row import TimestampRange
from gcloud.bigtable.row_filters import TimestampRange

table = object()
timestamp = 144185290431
Expand Down
34 changes: 17 additions & 17 deletions gcloud/bigtable/happybase/test_table.py
Original file line number Diff line number Diff line change
Expand Up @@ -1113,7 +1113,7 @@ def test_invalid_type(self):

def test_success(self):
from gcloud._helpers import _datetime_from_microseconds
from gcloud.bigtable.row import TimestampRange
from gcloud.bigtable.row_filters import TimestampRange

timestamp = 1441928298571
ts_dt = _datetime_from_microseconds(1000 * timestamp)
Expand Down Expand Up @@ -1219,7 +1219,7 @@ def test_no_filters(self):
self._callFUT()

def test_single_filter(self):
from gcloud.bigtable.row import CellsColumnLimitFilter
from gcloud.bigtable.row_filters import CellsColumnLimitFilter

versions = 1337
result = self._callFUT(versions=versions)
Expand All @@ -1229,7 +1229,7 @@ def test_single_filter(self):
self.assertEqual(result.num_cells, versions)

def test_existing_filters(self):
from gcloud.bigtable.row import CellsColumnLimitFilter
from gcloud.bigtable.row_filters import CellsColumnLimitFilter

filters = []
versions = 1337
Expand All @@ -1244,9 +1244,9 @@ def test_existing_filters(self):

def _column_helper(self, num_filters, versions=None, timestamp=None,
column=None, col_fam=None, qual=None):
from gcloud.bigtable.row import ColumnQualifierRegexFilter
from gcloud.bigtable.row import FamilyNameRegexFilter
from gcloud.bigtable.row import RowFilterChain
from gcloud.bigtable.row_filters import ColumnQualifierRegexFilter
from gcloud.bigtable.row_filters import FamilyNameRegexFilter
from gcloud.bigtable.row_filters import RowFilterChain

if col_fam is None:
col_fam = 'cf1'
Expand Down Expand Up @@ -1282,7 +1282,7 @@ def test_column_unicode(self):
col_fam=u'cfU', qual=u'qualN')

def test_with_versions(self):
from gcloud.bigtable.row import CellsColumnLimitFilter
from gcloud.bigtable.row_filters import CellsColumnLimitFilter

versions = 11
result = self._column_helper(num_filters=3, versions=versions)
Expand All @@ -1295,8 +1295,8 @@ def test_with_versions(self):

def test_with_timestamp(self):
from gcloud._helpers import _datetime_from_microseconds
from gcloud.bigtable.row import TimestampRange
from gcloud.bigtable.row import TimestampRangeFilter
from gcloud.bigtable.row_filters import TimestampRange
from gcloud.bigtable.row_filters import TimestampRangeFilter

timestamp = 1441928298571
result = self._column_helper(num_filters=3, timestamp=timestamp)
Expand Down Expand Up @@ -1330,7 +1330,7 @@ def test_no_columns(self):
self._callFUT(columns)

def test_single_column(self):
from gcloud.bigtable.row import FamilyNameRegexFilter
from gcloud.bigtable.row_filters import FamilyNameRegexFilter

col_fam = 'cf1'
columns = [col_fam]
Expand All @@ -1339,10 +1339,10 @@ def test_single_column(self):
self.assertEqual(result, expected_result)

def test_column_and_column_families(self):
from gcloud.bigtable.row import ColumnQualifierRegexFilter
from gcloud.bigtable.row import FamilyNameRegexFilter
from gcloud.bigtable.row import RowFilterChain
from gcloud.bigtable.row import RowFilterUnion
from gcloud.bigtable.row_filters import ColumnQualifierRegexFilter
from gcloud.bigtable.row_filters import FamilyNameRegexFilter
from gcloud.bigtable.row_filters import RowFilterChain
from gcloud.bigtable.row_filters import RowFilterUnion

col_fam1 = 'cf1'
col_fam2 = 'cf2'
Expand Down Expand Up @@ -1378,7 +1378,7 @@ def test_no_rows(self):
self._callFUT(row_keys)

def test_single_row(self):
from gcloud.bigtable.row import RowKeyRegexFilter
from gcloud.bigtable.row_filters import RowKeyRegexFilter

row_key = b'row-key'
row_keys = [row_key]
Expand All @@ -1387,8 +1387,8 @@ def test_single_row(self):
self.assertEqual(result, expected_result)

def test_many_rows(self):
from gcloud.bigtable.row import RowFilterUnion
from gcloud.bigtable.row import RowKeyRegexFilter
from gcloud.bigtable.row_filters import RowFilterUnion
from gcloud.bigtable.row_filters import RowKeyRegexFilter

row_key1 = b'row-key1'
row_key2 = b'row-key2'
Expand Down
Loading