Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add ability to create a File object from URL #2432

Merged
merged 3 commits into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions src/file.ts
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,9 @@ export class RequestError extends Error {
}

const SEVEN_DAYS = 7 * 24 * 60 * 60;
const GS_UTIL_URL_REGEX = /(gs):\/\/([a-z0-9_.-]+)\/(.+)/g;
const HTTPS_PUBLIC_URL_REGEX =
/(https):\/\/(storage\.googleapis\.com)\/([a-z0-9_.-]+)\/(.+)/g;

export enum FileExceptionMessages {
EXPIRATION_TIME_NA = 'An expiration time is not available.',
Expand Down Expand Up @@ -2358,6 +2361,35 @@ class File extends ServiceObject<File, FileMetadata> {
return this;
}

/**
* Gets a reference to a Cloud Storage {@link File} file from the provided URL in string format.
* @param {string} publicUrlOrGsUrl the URL as a string. Must be of the format gs://bucket/file
* or https://storage.googleapis.com/bucket/file.
* @param {Storage} storageInstance an instance of a Storage object.
* @param {FileOptions} [options] Configuration options
* @returns {File}
*/
static from(
publicUrlOrGsUrl: string,
storageInstance: Storage,
options?: FileOptions
): File {
const gsMatches = [...publicUrlOrGsUrl.matchAll(GS_UTIL_URL_REGEX)];
const httpsMatches = [...publicUrlOrGsUrl.matchAll(HTTPS_PUBLIC_URL_REGEX)];

if (gsMatches.length > 0) {
const bucket = new Bucket(storageInstance, gsMatches[0][1]);
return new File(bucket, gsMatches[0][2], options);
} else if (httpsMatches.length > 0) {
const bucket = new Bucket(storageInstance, httpsMatches[0][2]);
return new File(bucket, httpsMatches[0][3], options);
} else {
throw new Error(
'URL string must be of format gs://bucket/file or https://storage.googleapis.com/bucket/file'
);
}
}
Comment on lines +2372 to +2391
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, we could parse the URL, then use protocol to determine the logic. Here's an example with an optional storage param:

Suggested change
static from(
publicUrlOrGsUrl: string,
storageInstance: Storage,
options?: FileOptions
): File {
const gsMatches = [...publicUrlOrGsUrl.matchAll(GS_UTIL_URL_REGEX)];
const httpsMatches = [...publicUrlOrGsUrl.matchAll(HTTPS_PUBLIC_URL_REGEX)];
if (gsMatches.length > 0) {
const bucket = new Bucket(storageInstance, gsMatches[0][1]);
return new File(bucket, gsMatches[0][2], options);
} else if (httpsMatches.length > 0) {
const bucket = new Bucket(storageInstance, httpsMatches[0][2]);
return new File(bucket, httpsMatches[0][3], options);
} else {
throw new Error(
'URL string must be of format gs://bucket/file or https://storage.googleapis.com/bucket/file'
);
}
}
static from(
publicUrlOrGsUrl: string | URL,
storage?: Storage,
options?: FileOptions
): File {
const url = new URL(publicUrlOrGsUrl);
if (url.protocol === 'gs:') {
const storageInstance = storage ?? new Storage();
const bucket = new Bucket(storageInstance, url.host);
return new File(bucket, url.pathname.slice(1), options);
} else if (url.protocol === 'http:' || url.protocol === 'https:') {
const {groups: {bucketName, fileName}} = /\/(?<bucket>[^/]*)\/(?<filename>.*)/.exec(u.pathname);
const storageInstance = storage ?? new Storage({apiEndpoint: url.hostname});
const bucket = new Bucket(storageInstance, bucketName);
return new File(bucket, fileName, options);
} else {
throw new Error(
'URL string must be of format gs://bucket/file or https://storage.googleapis.com/bucket/file'
);
}
}

If we do this we'll need to make a small change to storage.ts on line ~180 to prevent accidental custom apiEndpoint when provided.
From:

if (options.apiEndpoint) {

To:

if (options.apiEndpoint && options.apiEndpoint !== apiEndpoint) {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestion, I'm apt to leave this using the regexs to make it a bit more concise and avoid the apiEndpoint logic. It also matches how it was done in the python client (not that consistency is SUPER important).


get(options?: GetFileOptions): Promise<GetResponse<File>>;
get(callback: InstanceResponseCallback<File>): void;
get(options: GetFileOptions, callback: InstanceResponseCallback<File>): void;
Expand Down
5 changes: 2 additions & 3 deletions src/nodejs-common/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -271,9 +271,8 @@ export class Service {
};

if (reqOpts[GCCL_GCS_CMD_KEY]) {
reqOpts.headers[
'x-goog-api-client'
] += ` gccl-gcs-cmd/${reqOpts[GCCL_GCS_CMD_KEY]}`;
reqOpts.headers['x-goog-api-client'] +=
` gccl-gcs-cmd/${reqOpts[GCCL_GCS_CMD_KEY]}`;
}

if (reqOpts.shouldReturnStream) {
Expand Down
5 changes: 2 additions & 3 deletions src/resumable-upload.ts
Original file line number Diff line number Diff line change
Expand Up @@ -933,9 +933,8 @@ export class Upload extends Writable {
// `Content-Length` for multiple chunk uploads is the size of the chunk,
// not the overall object
headers['Content-Length'] = bytesToUpload;
headers[
'Content-Range'
] = `bytes ${this.offset}-${endingByte}/${totalObjectSize}`;
headers['Content-Range'] =
`bytes ${this.offset}-${endingByte}/${totalObjectSize}`;
} else {
headers['Content-Range'] = `bytes ${this.offset}-*/${this.contentLength}`;
}
Expand Down
5 changes: 2 additions & 3 deletions src/transfer-manager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -223,9 +223,8 @@ class XMLMultiPartUploadHelper implements MultiPartUploadHelper {

// Prepend command feature to value, if not already there
if (!value.includes(GCCL_GCS_CMD_FEATURE.UPLOAD_SHARDED)) {
headers[
key
] = `${value} gccl-gcs-cmd/${GCCL_GCS_CMD_FEATURE.UPLOAD_SHARDED}`;
headers[key] =
`${value} gccl-gcs-cmd/${GCCL_GCS_CMD_FEATURE.UPLOAD_SHARDED}`;
}
} else if (key.toLocaleLowerCase().trim() === 'user-agent') {
userAgentFound = true;
Expand Down
45 changes: 45 additions & 0 deletions test/file.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5168,4 +5168,49 @@ describe('File', () => {
file.setUserProject(userProject);
});
});

describe('from', () => {
it('should create a File object from a gs:// formatted URL', () => {
const gsUrl = 'gs://mybucket/myfile';
const result = File.from(gsUrl, STORAGE);

assert(result);
assert(result.bucket.name, 'mybucket');
assert(result.name, 'myfile');
});

it('should create a File object from a gs:// formatted URL including a folder', () => {
const gsUrl = 'gs://mybucket/myfolder/myfile';
const result = File.from(gsUrl, STORAGE);

assert(result);
assert(result.bucket.name, 'mybucket');
assert(result.name, 'myfolder/myfile');
});

it('should create a File object from a https:// formatted URL', () => {
const httpsUrl = 'https://storage.googleapis.com/mybucket/myfile';
const result = File.from(httpsUrl, STORAGE);

assert(result);
assert(result.bucket.name, 'mybucket');
assert(result.name, 'myfile');
});

it('should create a File object from a https:// formatted URL including a folder', () => {
const httpsUrl =
'https://storage.googleapis.com/mybucket/myfolder/myfile';
const result = File.from(httpsUrl, STORAGE);

assert(result);
assert(result.bucket.name, 'mybucket');
assert(result.name, 'myfolder/myfile');
});

it('should throw an error when invoked with an incorrectly formatted URL', () => {
const invalidUrl = 'https://storage.com/mybucket/myfile';

assert.throws(() => File.from(invalidUrl, STORAGE));
});
});
});
Loading