Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump build target #1029

Merged
merged 41 commits into from
Apr 29, 2024
Merged
Changes from 4 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
6fc340d
fix: inline `thingies` dependency (#1004)
G-Rath Feb 17, 2024
3f43f4a
chore(release): 4.7.5 [skip ci]
semantic-release-bot Feb 17, 2024
d3b62dc
fix: remove tests from published package (#1003)
G-Rath Feb 17, 2024
f9b4ef3
chore(release): 4.7.6 [skip ci]
semantic-release-bot Feb 17, 2024
3c18dae
fix: accept file names beginning with a period (#1005)
nileflowers Feb 21, 2024
5dfc9b9
chore(release): 4.7.7 [skip ci]
semantic-release-bot Feb 21, 2024
6f4e72e
chore(deps): lock file maintenance
renovate[bot] Mar 1, 2024
44ceed4
chore(deps): bump follow-redirects from 1.15.5 to 1.15.6 (#1010)
dependabot[bot] Mar 17, 2024
e32a57d
feat: 🎸 allow to customize CAS storage hash and location mappin
streamich Mar 19, 2024
63ae860
refactor: 💡 separate abstract logic into CrudCasBase class
streamich Mar 19, 2024
956e7fd
style: 💄 run Prettier
streamich Mar 19, 2024
e900654
Merge pull request #1011 from streamich/cas-customizations
streamich Mar 19, 2024
455002c
chore(release): 4.8.0 [skip ci]
semantic-release-bot Mar 19, 2024
d78d6b8
chore(deps): lock file maintenance (#1013)
renovate[bot] Mar 24, 2024
cf9b0b3
chore(deps): lock file maintenance (#1017)
renovate[bot] Mar 29, 2024
b8905eb
fix: fix handle paths in FSA entries iterator (#1019)
quickgiant Mar 31, 2024
d4588bd
chore(release): 4.8.1 [skip ci]
semantic-release-bot Mar 31, 2024
2612163
chore(deps): lock file maintenance (#1021)
renovate[bot] Apr 5, 2024
711c4bd
fix: don't include filename in `path` when calling `readdir` with `wi…
jonsch318 Apr 14, 2024
577ee35
chore(release): 4.8.2 [skip ci]
semantic-release-bot Apr 14, 2024
921e05d
feat: 🎸 define .scan() CRUD method
streamich Apr 27, 2024
a148fb8
feat: 🎸 implement .scan() method for FSA CRUD
streamich Apr 27, 2024
3d973b7
feat: 🎸 implement .scan() in Node.js CRUD
streamich Apr 27, 2024
2351ff2
style: 💄 run Prettier
streamich Apr 27, 2024
456e004
style: 💄 run Prettier with updated deps
streamich Apr 27, 2024
b01bec5
Merge pull request #1026 from streamich/crud-scan
streamich Apr 27, 2024
97edfcd
chore(release): 4.9.0 [skip ci]
semantic-release-bot Apr 27, 2024
53cbadc
ci: change GitHub token env var
streamich Apr 27, 2024
7461f36
chore: 🤖 publish static site
streamich Apr 27, 2024
adc6570
chore: 🤖 update README and run Prettier
streamich Apr 27, 2024
de54ab5
fix: 🐛 use latest json-pack implementation
streamich Apr 27, 2024
c223b74
chore: 🤖 use util package, remove more duplicate code
streamich Apr 27, 2024
c6e18b7
chore: 🤖 use sonic-forest
streamich Apr 27, 2024
0509f15
chore: 🤖 remove /src/json-joy/ folder
streamich Apr 27, 2024
0dfd7bb
docs: ✏️ describe memfs() helper in docs
streamich Apr 27, 2024
e5461ae
Merge pull request #1028 from streamich/dependencies
streamich Apr 27, 2024
4619f16
chore(release): 4.9.1 [skip ci]
semantic-release-bot Apr 27, 2024
b198f40
ci: 🎡 add mirror to Gitlab workflow
streamich Apr 27, 2024
7a38617
chore(deps): update peaceiris/actions-gh-pages action to v4 (#1027)
renovate[bot] Apr 28, 2024
02aa726
Merge branch 'master' into next
streamich Apr 29, 2024
9d9bd01
chore: 🤖 update build target and don't emit sourcemaps
streamich Apr 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions src/cas/types.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import type { CrudResourceInfo } from '../crud/types';

export interface CasApi {
put(blob: Uint8Array): Promise<string>;
get(hash: string, options?: CasGetOptions): Promise<Uint8Array>;
del(hash: string, silent?: boolean): Promise<void>;
info(hash: string): Promise<CrudResourceInfo>;
export interface CasApi<Hash> {
put(blob: Uint8Array): Promise<Hash>;
get(hash: Hash, options?: CasGetOptions): Promise<Uint8Array>;
del(hash: Hash, silent?: boolean): Promise<void>;
info(hash: Hash): Promise<CrudResourceInfo>;
}

export interface CasGetOptions {
58 changes: 7 additions & 51 deletions src/crud-to-cas/CrudCas.ts
Original file line number Diff line number Diff line change
@@ -1,62 +1,18 @@
import { hashToLocation } from './util';
import type { CasApi, CasGetOptions } from '../cas/types';
import type { CrudApi, CrudResourceInfo } from '../crud/types';
import { CrudCasBase } from './CrudCasBase';
import type { CrudApi } from '../crud/types';

export interface CrudCasOptions {
hash: (blob: Uint8Array) => Promise<string>;
}

const normalizeErrors = async <T>(code: () => Promise<T>): Promise<T> => {
try {
return await code();
} catch (error) {
if (error && typeof error === 'object') {
switch (error.name) {
case 'ResourceNotFound':
case 'CollectionNotFound':
throw new DOMException(error.message, 'BlobNotFound');
}
}
throw error;
}
};
const hashEqual = (h1: string, h2: string) => h1 === h2;

export class CrudCas implements CasApi {
export class CrudCas extends CrudCasBase<string> {
constructor(
protected readonly crud: CrudApi,
protected readonly options: CrudCasOptions,
) {}

public readonly put = async (blob: Uint8Array): Promise<string> => {
const digest = await this.options.hash(blob);
const [collection, resource] = hashToLocation(digest);
await this.crud.put(collection, resource, blob);
return digest;
};

public readonly get = async (hash: string, options?: CasGetOptions): Promise<Uint8Array> => {
const [collection, resource] = hashToLocation(hash);
return await normalizeErrors(async () => {
const blob = await this.crud.get(collection, resource);
if (!options?.skipVerification) {
const digest = await this.options.hash(blob);
if (hash !== digest) throw new DOMException('Blob contents does not match hash', 'Integrity');
}
return blob;
});
};

public readonly del = async (hash: string, silent?: boolean): Promise<void> => {
const [collection, resource] = hashToLocation(hash);
await normalizeErrors(async () => {
return await this.crud.del(collection, resource, silent);
});
};

public readonly info = async (hash: string): Promise<CrudResourceInfo> => {
const [collection, resource] = hashToLocation(hash);
return await normalizeErrors(async () => {
return await this.crud.info(collection, resource);
});
};
) {
super(crud, options.hash, hashToLocation, hashEqual);
}
}
60 changes: 60 additions & 0 deletions src/crud-to-cas/CrudCasBase.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
import type { CasApi, CasGetOptions } from '../cas/types';
import type { CrudApi, CrudResourceInfo } from '../crud/types';
import type { FsLocation } from '../fsa-to-node/types';

const normalizeErrors = async <T>(code: () => Promise<T>): Promise<T> => {
try {
return await code();
} catch (error) {
if (error && typeof error === 'object') {
switch (error.name) {
case 'ResourceNotFound':
case 'CollectionNotFound':
throw new DOMException(error.message, 'BlobNotFound');
}
}
throw error;
}
};

export class CrudCasBase<Hash> implements CasApi<Hash> {
constructor(
protected readonly crud: CrudApi,
protected readonly hash: (blob: Uint8Array) => Promise<Hash>,
protected readonly hash2Loc: (hash: Hash) => FsLocation,
protected readonly hashEqual: (h1: Hash, h2: Hash) => boolean,
) {}

public readonly put = async (blob: Uint8Array): Promise<Hash> => {
const digest = await this.hash(blob);
const [collection, resource] = this.hash2Loc(digest);
await this.crud.put(collection, resource, blob);
return digest;
};

public readonly get = async (hash: Hash, options?: CasGetOptions): Promise<Uint8Array> => {
const [collection, resource] = this.hash2Loc(hash);
return await normalizeErrors(async () => {
const blob = await this.crud.get(collection, resource);
if (!options?.skipVerification) {
const digest = await this.hash(blob);
if (!this.hashEqual(digest, hash)) throw new DOMException('Blob contents does not match hash', 'Integrity');
}
return blob;
});
};

public readonly del = async (hash: Hash, silent?: boolean): Promise<void> => {
const [collection, resource] = this.hash2Loc(hash);
await normalizeErrors(async () => {
return await this.crud.del(collection, resource, silent);
});
};

public readonly info = async (hash: Hash): Promise<CrudResourceInfo> => {
const [collection, resource] = this.hash2Loc(hash);
return await normalizeErrors(async () => {
return await this.crud.info(collection, resource);
});
};
}
71 changes: 71 additions & 0 deletions src/crud-to-cas/__tests__/CrudCasBase.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
import { memfs } from '../..';
import { onlyOnNode20 } from '../../__tests__/util';
import { NodeFileSystemDirectoryHandle } from '../../node-to-fsa';
import { FsaCrud } from '../../fsa-to-crud/FsaCrud';
import { createHash } from 'crypto';
import { hashToLocation } from '../util';
import { CrudCasBase } from '../CrudCasBase';
import { FsLocation } from '../../fsa-to-node/types';

onlyOnNode20('CrudCas on FsaCrud', () => {
const setup = () => {
const { fs } = memfs();
const fsa = new NodeFileSystemDirectoryHandle(fs, '/', { mode: 'readwrite' });
const crud = new FsaCrud(fsa);
return { fs, fsa, crud, snapshot: () => (<any>fs).__vol.toJSON() };
};

test('can use a custom hashing digest type', async () => {
const { crud } = setup();
class Hash {
constructor(public readonly digest: string) {}
}
const hash = async (blob: Uint8Array): Promise<Hash> => {
const shasum = createHash('sha1');
shasum.update(blob);
const digest = shasum.digest('hex');
return new Hash(digest);
};
const cas = new CrudCasBase<Hash>(
crud,
hash,
(id: Hash) => hashToLocation(id.digest),
(h1: Hash, h2: Hash) => h1.digest === h2.digest,
);
const blob = Buffer.from('hello world');
const id = await cas.put(blob);
expect(id).toBeInstanceOf(Hash);
const id2 = await hash(blob);
expect(id.digest).toEqual(id2.digest);
const blob2 = await cas.get(id);
expect(String.fromCharCode(...blob2)).toEqual('hello world');
expect(await cas.info(id)).toMatchObject({ size: 11 });
await cas.del(id2);
expect(() => cas.info(id)).rejects.toThrowError();
});

test('can use custom folder sharding strategy', async () => {
const { crud } = setup();
const hash = async (blob: Uint8Array): Promise<string> => {
const shasum = createHash('sha1');
shasum.update(blob);
return shasum.digest('hex');
};
const theCustomFolderShardingStrategy = (h: string): FsLocation => [[h[0], h[1], h[2]], h[3]];
const cas = new CrudCasBase<string>(
crud,
hash,
theCustomFolderShardingStrategy,
(h1: string, h2: string) => h1 === h2,
);
const blob = Buffer.from('hello world');
const id = await cas.put(blob);
expect(typeof id).toBe('string');
const id2 = await hash(blob);
expect(id).toBe(id2);
const blob2 = await cas.get(id);
expect(String.fromCharCode(...blob2)).toEqual('hello world');
const blob3 = await crud.get([id2[0], id2[1], id2[2]], id2[3]);
expect(String.fromCharCode(...blob3)).toEqual('hello world');
});
});
2 changes: 1 addition & 1 deletion src/crud-to-cas/__tests__/testCasfs.ts
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ const b = (str: string) => {
};

export type Setup = () => {
cas: CasApi;
cas: CasApi<string>;
crud: CrudApi;
snapshot: () => Record<string, string | null>;
};