Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigInts for all the things #23

Merged
merged 3 commits into from
Jul 27, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,12 @@ If no errors were encountered, `errors` will be `null`. If errors were encounter

This method is great if you want to implement custom logic based on the errors that were encountered.

### `getFolderSize.loose(path, [options]): number`
### `getFolderSize.loose(path, [options]): number | bigint`
The `loose` method will return the folder size directly and ignore any errors it encounters, which means the returned folder size could be smaller than the real folder size.

This method is great if the precise size isn't too important, for example when used only to display the folder size to the user.

### `getFolderSize.strict(path, [options]): number`
### `getFolderSize.strict(path, [options]): number | bigint`
The `strict` method will return the folder size directly, but throw an error if it encounters any read errors.

This method is great if you need a very accurate number. You will have to implement some sort of error handling to use it reliably.
Expand All @@ -63,12 +63,15 @@ Any of the three methods can also take an `options` object:
getFolderSize(
'/path/to/folder',
{
bigint: true,
ignore: /pattern/,
fs: customFS,
}
)
```

If the `bigint` option is set to true, the folder size is returned as a BigInt instead of the default Number.

The `ignore` option takes a regex pattern. Any file or folder with a path that matches the pattern will not be counted in the total folder size.

The `fs` option allows you to pass a different filesystem handler, such as [memfs](https://github.com/streamich/memfs), that will be used to read the folder size. The filesystem handler must incorporate `lstat` and `readdir` promise functions.
Expand Down
49 changes: 32 additions & 17 deletions index.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@ import { join as joinPaths } from 'path';
*
* If any errors are returned, the returned folder size is likely smaller than the real folder size.
*
* @param {string} itemPath - Path of the folder.
* @param {object} [options] - Options.
* @param {object} [options.ignore] - If a file's path matches this regex object, its size is not counted.
* @param {object} [options.fs] - The filesystem that should be used. Uses node fs by default.
* @param {string} itemPath - Path of the folder.
* @param {object} [options] - Options.
* @param {boolean} [options.bigint] - Should the folder size be returned as a BigInt instead of a Number.
* @param {object} [options.ignore] - If a file's path matches this regex object, its size is not counted.
* @param {object} [options.fs] - The filesystem that should be used. Uses node fs by default.
*
* @returns {Promise<{size: number, errors: Array<Error> | null}>} - An object containing the size of the folder in bytes and a list of encountered errors.
* @returns {Promise<{size: number | bigint, errors: Array<Error> | null}>} - An object containing the size of the folder in bytes and a list of encountered errors.
*/
export default async function getFolderSize (itemPath, options) { return await core(itemPath, options, {errors: true}) }

Expand All @@ -20,12 +21,13 @@ export default async function getFolderSize (itemPath, options) { return await c
*
* The returned folder size might be smaller than the real folder size. It is impossible to know for sure, since errors are ignored.
*
* @param {string} itemPath - Path of the folder.
* @param {object} [options] - Options.
* @param {object} [options.ignore] - If a file's path matches this regex object, its size is not counted.
* @param {object} [options.fs] - The filesystem that should be used. Uses node fs by default.
* @param {string} itemPath - Path of the folder.
* @param {object} [options] - Options.
* @param {boolean} [options.bigint] - Should the folder size be returned as a BigInt instead of a Number.
* @param {object} [options.ignore] - If a file's path matches this regex object, its size is not counted.
* @param {object} [options.fs] - The filesystem that should be used. Uses node fs by default.
*
* @returns {Promise<number>} - The size of the folder in bytes.
* @returns {Promise<number | bigint>} - The size of the folder in bytes.
*/
getFolderSize.loose = async (itemPath, options) => await core(itemPath, options);

Expand All @@ -34,12 +36,13 @@ getFolderSize.loose = async (itemPath, options) => await core(itemPath, options)
*
* Because errors will otherwise make this method fail, the returned folder size will always be accurate.
*
* @param {string} itemPath - Path of the folder.
* @param {object} [options] - Options.
* @param {object} [options.ignore] - If a file's path matches this regex object, its size is not counted.
* @param {object} [options.fs] - The filesystem that should be used. Uses node fs by default.
* @param {string} itemPath - Path of the folder.
* @param {object} [options] - Options.
* @param {boolean} [options.bigint] - Should the folder size be returned as a BigInt instead of a Number.
* @param {object} [options.ignore] - If a file's path matches this regex object, its size is not counted.
* @param {object} [options.fs] - The filesystem that should be used. Uses node fs by default.
*
* @returns {Promise<number>} - The size of the folder in bytes.
* @returns {Promise<number | bigint>} - The size of the folder in bytes.
*/
getFolderSize.strict = async (itemPath, options) => await core(itemPath, options, {strict: true});

Expand All @@ -56,7 +59,7 @@ async function core (rootItemPath, options = {}, returnType = {}) {
async function processItem(itemPath) {
if(options.ignore?.test(itemPath)) return;

const stats = returnType.strict ? await fs.lstat(itemPath) : await fs.lstat(itemPath).catch(error => errors.push(error));
const stats = returnType.strict ? await fs.lstat(itemPath, {bigint: true}) : await fs.lstat(itemPath, {bigint: true}).catch(error => errors.push(error));
if(typeof stats !== 'object') return;
fileSizes.set(stats.ino, stats.size);

Expand All @@ -71,7 +74,19 @@ async function core (rootItemPath, options = {}, returnType = {}) {
}
}

const folderSize = Array.from(fileSizes.values()).reduce((total, fileSize) => total + fileSize, 0);
let folderSize = Array.from(fileSizes.values()).reduce((total, fileSize) => total + fileSize, 0n);

if(!options.bigint) {
if(folderSize > BigInt(Number.MAX_SAFE_INTEGER)){
const error = new RangeError('The folder size is too large to return as a Number. You can instruct this package to return a BigInt instead.');
if(returnType.strict){
throw error;
}else{
errors.push(error);
}
}
folderSize = Number(folderSize);
}

if (returnType.errors) {
return {
Expand Down
83 changes: 78 additions & 5 deletions test/logic.js
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,27 @@ tap.test('basic folder', async () => {

});

tap.test('basic folder - with bigint', async () => {

tap.test('get file sizes', async () => {

tap.equal(await callAll('/fixture/8bytes.txt', {bigint: true, fs: basicFS}), 8n, 'should return the correct file size');
tap.equal(await callAll('/fixture/500bytes.txt', {bigint: true, fs: basicFS}), 500n, 'should return the correct file size');
tap.equal(await callAll('/fixture/6000bytes.txt', {bigint: true, fs: basicFS}), 6000n, 'should return the correct file size');
tap.end();

});

tap.test('get folder size', async () => {

tap.equal(await callAll('/fixture', {bigint: true, fs: basicFS}), 6508n, 'should return the correct folder size');

tap.end();

});

});

const nestedFS = Volume.fromJSON(
{
'./8bytes.txt': B.repeat(8),
Expand Down Expand Up @@ -111,6 +132,58 @@ tap.test('ignore option', async () => {

});

const largeFSCore = Volume.fromJSON(
{
'./very.txt': B.repeat(200),
'./large.txt': B.repeat(200),
'./files.txt': B.repeat(200),
},
'/fixture',
).promisesApi;

const largeFS = {
lstat: async (itemPath, options) => {
const result = await largeFSCore.lstat(itemPath, options);
result.size = BigInt(Number.MAX_SAFE_INTEGER);
return result;
},
readdir: largeFSCore.readdir,
};

tap.test('handling very large filesystems', async () => {

tap.test('returning Number', async () => {

tap.type(await getFolderSize.loose('/fixture', {fs: largeFS}), 'number', 'should return Number');

tap.rejects(async () => {await getFolderSize.strict('/fixture', {fs: largeFS})}, /The folder size is too large to return as a Number. You can instruct this package to return a BigInt instead./, 'should throw appropriate error');

const { size, errors } = await getFolderSize('/fixture', {fs: largeFS});
tap.type(size, 'number', 'should return Number');
tap.type(errors, Array, 'should return Array of errors');
tap.equal(errors.length, 1, 'should return one error');
tap.equal(errors[0].message, 'The folder size is too large to return as a Number. You can instruct this package to return a BigInt instead.', 'should return appropriate error');

tap.end();

});

tap.test('returning BigInt', async () => {

tap.equal(await getFolderSize.loose('/fixture', {bigint: true, fs: largeFS}), BigInt(Number.MAX_SAFE_INTEGER) * 4n, 'should return size of 4 times max safe Number');

tap.equal(await getFolderSize.strict('/fixture', {bigint: true, fs: largeFS}), BigInt(Number.MAX_SAFE_INTEGER) * 4n, 'should return size of 4 times max safe Number');

const { size, errors } = await getFolderSize('/fixture', {bigint: true, fs: largeFS});
tap.equal(size, BigInt(Number.MAX_SAFE_INTEGER) * 4n, 'should return size of 4 times max safe Number');
tap.equal(errors, null, 'should return no errors');

tap.end();

});

});

const badFSCore = Volume.fromJSON(
{
'./pass/pass.md': B.repeat(200),
Expand All @@ -126,21 +199,21 @@ const badFSCore = Volume.fromJSON(
).promisesApi;

const badFS = {
lstat: async (itemPath) => {
lstat: async (itemPath, options) => {
if(itemPath.includes('failFile')){
throw Error('Nah - File');
}else{
return await badFSCore.lstat(itemPath);
return await badFSCore.lstat(itemPath, options);
}
},
readdir: async (itemPath) => {
readdir: async (itemPath, options) => {
if(itemPath.includes('failDir')){
throw Error('Nah - Directory');
}else{
return await badFSCore.readdir(itemPath);
return await badFSCore.readdir(itemPath, options);
}
}
}
};

tap.test('error handling', async () => {

Expand Down