Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[file-datalake] AttributeError in process_storage_error #15524

Closed
evelyn-ys opened this issue Nov 24, 2020 · 2 comments
Closed

[file-datalake] AttributeError in process_storage_error #15524

evelyn-ys opened this issue Nov 24, 2020 · 2 comments
Assignees
@xiafu-msft
Labels
Client This issue points to a problem in the data-plane of the library. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team. Storage Storage Service (Queues, Blobs, Files)

Comments

@evelyn-ys
Copy link
Member

evelyn-ys commented Nov 24, 2020 •
edited
Loading

  • Package Name: azure-storage-file-datalake
  • Package Version: 2020-02-10
  • Python Version: 3.7

Describe the bug
When using CLI list files in a file system within unsuitable storage account, CLI will call

azure-sdk-for-python/sdk/storage/azure-storage-file-datalake/azure/storage/filedatalake/_file_system_client.py

Lines 424 to 427 in 5ed06b8

def get_paths(self, path=None, # type: Optional[str]
recursive=True, # type: Optional[bool]
max_results=None, # type: Optional[int]
**kwargs):

When paging for the result using such code

    generator = fsclient.get_paths(path=path, max_results=num_results, recursive=recursive, timeout=timeout)
    pages = generator.by_page(continuation_token=marker)
    result = next(pages)

An AttributeError: 'dict' object has no attribute 'iter' error occurs:

azure.core.pipeline.policies.http_logging_policy : Request URL: 'https://yssa.dfs.core.windows.net/ysfile?recursive=REDACTED&maxResults=REDACTED&resource=REDACTED'
azure.core.pipeline.policies.http_logging_policy : Request method: 'GET'
azure.core.pipeline.policies.http_logging_policy : Request headers:
azure.core.pipeline.policies.http_logging_policy :     'Accept': 'application/json'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-version': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-date': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-client-request-id': 'e73d21ba-2e00-11eb-af91-005056c00008'
azure.core.pipeline.policies.http_logging_policy :     'User-Agent': 'azsdk-python-storage-dfs/12.2.0b1 Python/3.7.9 (Windows-10-10.0.19041-SP0)'
azure.core.pipeline.policies.http_logging_policy :     'Authorization': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy : No body was attached to the request

azure.core.pipeline.policies.http_logging_policy : Response status: 409
azure.core.pipeline.policies.http_logging_policy : Response headers:
azure.core.pipeline.policies.http_logging_policy :     'Content-Length': '292'
azure.core.pipeline.policies.http_logging_policy :     'Content-Type': 'application/json;charset=utf-8'
azure.core.pipeline.policies.http_logging_policy :     'Server': 'Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-error-code': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-request-id': '733f5928-101f-001a-4f0d-c2dec0000000'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-version': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-client-request-id': 'e73d21ba-2e00-11eb-af91-005056c00008'
azure.core.pipeline.policies.http_logging_policy :     'Date': 'Tue, 24 Nov 2020 02:58:18 GMT'
azure.cli.core.util.handle_exception is called with an exception:
Traceback (most recent call last):
  File "D:\workspace\ysenv\lib\site-packages\azure\multiapi\storagev2\filedatalake\v2020_02_10\_list_paths_helper.py", line 56, in _get_next_cb
    cls=return_headers_and_deserialized_path_list)
  File "D:\workspace\ysenv\lib\site-packages\azure\multiapi\storagev2\filedatalake\v2020_02_10\_generated\operations\_file_system_operations.py", line 442, in list_paths
    raise models.StorageErrorException(response, self._deserialize)
azure.multiapi.storagev2.filedatalake.v2020_02_10._generated.models._models_py3.StorageErrorException: (EndpointUnsupportedAccountFeatures) This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.
RequestId:733f5928-101f-001a-4f0d-c2dec0000000
Time:2020-11-24T02:58:18.9717910Z

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\workspace\ysenv\lib\site-packages\knack\cli.py", line 215, in invoke
    cmd_result = self.invocation.execute(args)
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 654, in execute
    raise ex
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 717, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 709, in _run_job
    return cmd_copy.exception_handler(ex)
  File "d:\workspace\azure-cli\src\azure-cli\azure\cli\command_modules\storage\__init__.py", line 344, in new_handler
    raise ex
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 688, in _run_job
    result = cmd_copy(params)
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 325, in __call__
    return self.handler(*args, **kwargs)
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\__init__.py", line 784, in default_command_handler
    return op(**command_args)
  File "d:\workspace\azure-cli\src\azure-cli\azure\cli\command_modules\storage\operations\fs_file.py", line 76, in list_fs_files
    result = next(pages)
  File "D:\workspace\ysenv\lib\site-packages\azure\core\paging.py", line 74, in __next__
    self._response = self._get_next(self.continuation_token)
  File "D:\workspace\ysenv\lib\site-packages\azure\multiapi\storagev2\filedatalake\v2020_02_10\_list_paths_helper.py", line 58, in _get_next_cb
    process_storage_error(error)
  File "D:\workspace\ysenv\lib\site-packages\azure\multiapi\storagev2\filedatalake\v2020_02_10\_shared\response_handlers.py", line 94, in process_storage_error
    for info in error_body.iter():
AttributeError: 'dict' object has no attribute 'iter'

Which breaks in Line 94

azure-sdk-for-python/sdk/storage/azure-storage-file-datalake/azure/storage/filedatalake/_shared/response_handlers.py

Lines 86 to 100 in 3ca0e01

def process_storage_error(storage_error):
raise_error = HttpResponseError
error_code = storage_error.response.headers.get('x-ms-error-code')
error_message = storage_error.message
additional_data = {}
try:
error_body = ContentDecodePolicy.deserialize_from_http_generics(storage_error.response)
if error_body:
for info in error_body.iter():
if info.tag.lower() == 'code':
error_code = info.text
elif info.tag.lower() == 'message':
error_message = info.text
else:
additional_data[info.tag] = info.text

@ghost ghost added the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Nov 24, 2020
@lmazuel lmazuel added Client This issue points to a problem in the data-plane of the library. Service Attention Workflow: This issue is responsible by Azure service team. Storage Storage Service (Queues, Blobs, Files) labels Nov 24, 2020
@ghost ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Nov 24, 2020
@ghost
Copy link

ghost commented Nov 24, 2020

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details
  • Package Name: azure-storage-file-datalake
  • Package Version: 2020-02-10
  • Python Version: 3.7

Describe the bug
When using CLI list files in a file system within unsuitable storage account, CLI will call

azure-sdk-for-python/sdk/storage/azure-storage-file-datalake/azure/storage/filedatalake/_file_system_client.py

Lines 424 to 427 in 5ed06b8

def get_paths(self, path=None, # type: Optional[str]
recursive=True, # type: Optional[bool]
max_results=None, # type: Optional[int]
**kwargs):

When paging for the result using such code

    generator = fsclient.get_paths(path=path, max_results=num_results, recursive=recursive, timeout=timeout)
    pages = generator.by_page(continuation_token=marker)
    result = next(pages)

An AttributeError: 'dict' object has no attribute 'iter' error occurs:

azure.core.pipeline.policies.http_logging_policy : Request URL: 'https://yssa.dfs.core.windows.net/ysfile?recursive=REDACTED&maxResults=REDACTED&resource=REDACTED'
azure.core.pipeline.policies.http_logging_policy : Request method: 'GET'
azure.core.pipeline.policies.http_logging_policy : Request headers:
azure.core.pipeline.policies.http_logging_policy :     'Accept': 'application/json'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-version': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-date': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-client-request-id': 'e73d21ba-2e00-11eb-af91-005056c00008'
azure.core.pipeline.policies.http_logging_policy :     'User-Agent': 'azsdk-python-storage-dfs/12.2.0b1 Python/3.7.9 (Windows-10-10.0.19041-SP0)'
azure.core.pipeline.policies.http_logging_policy :     'Authorization': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy : No body was attached to the request

azure.core.pipeline.policies.http_logging_policy : Response status: 409
azure.core.pipeline.policies.http_logging_policy : Response headers:
azure.core.pipeline.policies.http_logging_policy :     'Content-Length': '292'
azure.core.pipeline.policies.http_logging_policy :     'Content-Type': 'application/json;charset=utf-8'
azure.core.pipeline.policies.http_logging_policy :     'Server': 'Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-error-code': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-request-id': '733f5928-101f-001a-4f0d-c2dec0000000'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-version': 'REDACTED'
azure.core.pipeline.policies.http_logging_policy :     'x-ms-client-request-id': 'e73d21ba-2e00-11eb-af91-005056c00008'
azure.core.pipeline.policies.http_logging_policy :     'Date': 'Tue, 24 Nov 2020 02:58:18 GMT'
azure.cli.core.util.handle_exception is called with an exception:
Traceback (most recent call last):
  File "D:\workspace\ysenv\lib\site-packages\azure\multiapi\storagev2\filedatalake\v2020_02_10\_list_paths_helper.py", line 56, in _get_next_cb
    cls=return_headers_and_deserialized_path_list)
  File "D:\workspace\ysenv\lib\site-packages\azure\multiapi\storagev2\filedatalake\v2020_02_10\_generated\operations\_file_system_operations.py", line 442, in list_paths
    raise models.StorageErrorException(response, self._deserialize)
azure.multiapi.storagev2.filedatalake.v2020_02_10._generated.models._models_py3.StorageErrorException: (EndpointUnsupportedAccountFeatures) This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.
RequestId:733f5928-101f-001a-4f0d-c2dec0000000
Time:2020-11-24T02:58:18.9717910Z

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\workspace\ysenv\lib\site-packages\knack\cli.py", line 215, in invoke
    cmd_result = self.invocation.execute(args)
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 654, in execute
    raise ex
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 717, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 709, in _run_job
    return cmd_copy.exception_handler(ex)
  File "d:\workspace\azure-cli\src\azure-cli\azure\cli\command_modules\storage\__init__.py", line 344, in new_handler
    raise ex
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 688, in _run_job
    result = cmd_copy(params)
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 325, in __call__
    return self.handler(*args, **kwargs)
  File "d:\workspace\azure-cli\src\azure-cli-core\azure\cli\core\__init__.py", line 784, in default_command_handler
    return op(**command_args)
  File "d:\workspace\azure-cli\src\azure-cli\azure\cli\command_modules\storage\operations\fs_file.py", line 76, in list_fs_files
    result = next(pages)
  File "D:\workspace\ysenv\lib\site-packages\azure\core\paging.py", line 74, in __next__
    self._response = self._get_next(self.continuation_token)
  File "D:\workspace\ysenv\lib\site-packages\azure\multiapi\storagev2\filedatalake\v2020_02_10\_list_paths_helper.py", line 58, in _get_next_cb
    process_storage_error(error)
  File "D:\workspace\ysenv\lib\site-packages\azure\multiapi\storagev2\filedatalake\v2020_02_10\_shared\response_handlers.py", line 94, in process_storage_error
    for info in error_body.iter():
AttributeError: 'dict' object has no attribute 'iter'

Which breaks in Line 94

azure-sdk-for-python/sdk/storage/azure-storage-file-datalake/azure/storage/filedatalake/_shared/response_handlers.py

Lines 86 to 100 in 3ca0e01

def process_storage_error(storage_error):
raise_error = HttpResponseError
error_code = storage_error.response.headers.get('x-ms-error-code')
error_message = storage_error.message
additional_data = {}
try:
error_body = ContentDecodePolicy.deserialize_from_http_generics(storage_error.response)
if error_body:
for info in error_body.iter():
if info.tag.lower() == 'code':
error_code = info.text
elif info.tag.lower() == 'message':
error_message = info.text
else:
additional_data[info.tag] = info.text

Author: evelyn-ys
Assignees: -
Labels:

Client, Service Attention, Storage, needs-triage
Milestone: -

@lmazuel lmazuel added the question The issue doesn't require a change to the product in order to be resolved. Most issues start as that label Nov 24, 2020
@xiafu-msft
Copy link
Contributor

Hi @evelyn-ys

It's a duplicate of #15956 we will fix this later

@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@xiafu-msft xiafu-msft

Labels
Client This issue points to a problem in the data-plane of the library. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team. Storage Storage Service (Queues, Blobs, Files)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lmazuel @evelyn-ys @xiafu-msft