Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downloading file from azure data lake is not working using DataLakeFileClient #17170

Closed
maheshpg opened this issue Nov 4, 2020 · 2 comments
Closed
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Storage Storage Service (Queues, Blobs, Files)

Comments

@maheshpg
Copy link

maheshpg commented Nov 4, 2020

I am trying to download file which is of 4 to 5 KB max from azure data lake zen2 account. Below is my code :

import com.azure.storage.common.StorageSharedKeyCredential;
import com.azure.storage.file.datalake.*;

import java.io.ByteArrayOutputStream;

public class DataLakeClient {

public static void main(String[] args) {

    final StorageSharedKeyCredential sKeyCredential = new StorageSharedKeyCredential("devlake",
            "c8WrzvPxry87zdrR7BZ########################################RJN+yIaHg==");
    final DataLakeServiceClientBuilder builder = new DataLakeServiceClientBuilder();
    builder.credential(sKeyCredential);
    builder.endpoint("https://devlake.dfs.core.windows.net");

    DataLakeServiceClient dataLakeServiceClient = builder.buildClient();
    DataLakeFileSystemClient fileSystemClient = dataLakeServiceClient.getFileSystemClient("landingzone");
    DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient("1hzhD7aqUJ6oYGLEwoQpHP70T2u/20201021/DB37iu0F2rCfJQ0vhGyVEoqPrzU");
    DataLakeFileClient fileClient = directoryClient.getFileClient("GSM_2_165_CH-EU-RU-UK-US_INSTR_DELTA_20201006_150829.csv");
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    fileClient.read(byteArrayOutputStream);
    System.out.println(byteArrayOutputStream.toByteArray());

}

}

Above code gets stuck at fileClient.read(byteArrayOutputStream); call infinitely and finally client timeout exception comes. I am trying to run this from intellij on my local system. Using same DataLakeFileClient class I am able to create files on same directory location. but after creation and I am not able to read same file due to that stuck issue. I am not getting why I am not able to read file but able to write file using same class.

I am using below dependency

com.azure
azure-storage-file-datalake
12.3.0-beta.1

Any quick help is really appreciated.

@ghost ghost added needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Nov 4, 2020
@maheshpg maheshpg changed the title Downloading file from azure data lake is not working using Data Downloading file from azure data lake is not working using DataLakeFileClient Nov 4, 2020
@alzimmermsft alzimmermsft added Client This issue points to a problem in the data-plane of the library. Storage Storage Service (Queues, Blobs, Files) labels Nov 4, 2020
@ghost ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Nov 4, 2020
@rickle-msft
Copy link
Contributor

Hi, @maheshpg. Thank you for posting this question. Are you using any other dependencies in your project? And are able to see an outgoing download request on the network?

@maheshpg
Copy link
Author

maheshpg commented Nov 6, 2020

Hi, @maheshpg. Thank you for posting this question. Are you using any other dependencies in your project? And are able to see an outgoing download request on the network?

Actually I found that on azure private link was enabled for DFS but not for BLOB. And for downloading file DataLakeFileClient was using blob service endpoint internally which was unknown to me. After enabling private endpoint for blob it is working.
Thanks for your quick response. But now this isssue can be closed.

@maheshpg maheshpg closed this as completed Nov 6, 2020
openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-java that referenced this issue Feb 7, 2022
Dev sentinel 2021 10 01 preview (Azure#17494)

* Adds base for updating Microsoft.SecurityInsights from version preview/2021-09-01-preview to version 2021-10-01-preview

* Updates readme

* Updates API version in new specs and examples

* Add IoT data connector (Azure#17086)

* add Iot data connector

* fix kind in getIotById

* add deleted files

* add missing newlines

* fix newlines

* Revert "Add IoT data connector (Azure#17086)" (Azure#17170)

This reverts commit a11dd79ea1fb12c9b9d673fed2de91cc0098adbf.

* Adding providerIncidentUrl & techniques to IncidentAdditionalData (Azure#17173)

* Adding providerIncidentUrl to Incident

* Adding techniques to incident

* Dev sentinel 2021 10 01 preview (Azure#17314)

* Bookmarks 2021-10-01-preview

* prettier

* definitions

* rename

Co-authored-by: Igal Shapira <igshapir@microsoft.com>

* Data connectors 2021 10 01 preview office connectors added (Azure#17193)

* Project and PowerBI specs added.

* Space removed

* File with a wrong name removed

* More changer added

* Example files with inconsistent naming removed

* dataConnectors json updated

* Ref fixed

* Prettier applied

Co-authored-by: Ido Klotz <idoklotz@microsoft.com>

* AutomationRules preview 2021_10_01 (Azure#17325)

* h

* first

* examples

* prettier

* path

* fixes

* prettier

* examples

* Z

* Z

* responses

* fix

* fixes

* fix

* prettier

* PR Fixes

* PR Fixes

* PR Fixes

* fix

* fix

* fix

* fix

* Last

* PR Fixes

* Last

* tryFix

* tryFix

* incidentTypes

* fix

Co-authored-by: Roy Reinhorn <roreinho@microsoft.com>

* Add MITRE support to alert rules models (Azure#17198)

* Update alert rules models with tactics and techniques

* Add attack tactics new enum values

* Update alert rules models examples

* Fix techniques type

* Update files with prettier

* Insert validations fixes

* Fix validations

* Fixes in alert rules models

* Fix alert rules models and examples

* adding changes to alert rules json

* Revert "adding changes to alert rules json"

This reverts commit 603490e300c76dee48550a11d0dee8fe5c824a2e.

* Adding fusion v2 ui api documentation

* some fixes

* adding prettier fixes

* removing unwanted property

* updating the readonly properties

Co-authored-by: Lilyan Cohen <licohen@microsoft.com>
Co-authored-by: Vishal Kumar <viskumar@microsoft.com>

* ErrorResponse changed to CloudError (Azure#17477)

Co-authored-by: Ido Klotz <idoklotz@microsoft.com>

* Done (Azure#17488)

Co-authored-by: Roy Reinhorn <roreinho@microsoft.com>

* Adding tenantId (Azure#17533)

Co-authored-by: Ido Klotz <idoklotz@microsoft.com>

* Done (Azure#17556)

Co-authored-by: Roy Reinhorn <roreinho@microsoft.com>

* Fix LindDiff and SemanticValidation (Azure#17584)

Co-authored-by: ShaniFelig <74960756+ShaniFelig@users.noreply.github.com>
Co-authored-by: roherzbe <52486962+roherzbe@users.noreply.github.com>
Co-authored-by: Igal <igal.shapira@gmail.com>
Co-authored-by: Igal Shapira <igshapir@microsoft.com>
Co-authored-by: Ido Klotz <idoklotz@gmail.com>
Co-authored-by: Ido Klotz <idoklotz@microsoft.com>
Co-authored-by: royrein <37300636+royrein@users.noreply.github.com>
Co-authored-by: Roy Reinhorn <roreinho@microsoft.com>
Co-authored-by: lilyanc02 <46589651+lilyanc02@users.noreply.github.com>
Co-authored-by: Lilyan Cohen <licohen@microsoft.com>
Co-authored-by: Vishal Kumar <viskumar@microsoft.com>
@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Storage Storage Service (Queues, Blobs, Files)
Projects
None yet
Development

No branches or pull requests

3 participants