Add some way to get the actual storage size of an object

[da2x]

Version information:

go-ipfs version: 0.4.18-
Repo version: 7
System version: amd64/linux
Golang version: go1.11.1

Type:

enhancement

Description:

ipfs object stat QmPS6VssQGyBYjGQSK8ordvXaU1yUoaUmTfmrV7daLeRPH outputs CumulativeSize: 36709305. However, when an object consists of mostly duplicated data (like the example hash does) the actual blocksize required to store it in a repository is only 15729480 bytes (or 42,8 % of the CumulativeSize.).

It would extremely useful to have another field in the output from ipfs object stat for the actual recursive deduplicated storage block size of an object (the amount of space required in the repository to store the object).

The following command is the only way I’ve found to get this number:

ipfs refs --unique --recursive QmPS6VssQGyBYjGQSK8ordvXaU1yUoaUmTfmrV7daLeRPH | \
    xargs -I% sh -c "echo -e '\n%I' && ipfs object stat %" | \
    grep DataSize\: | \
    egrep -o -e '[0-9]+' | \
    awk '{x += $1} END{print x}'

(The above method may not be accurate although it appears to work at least for synthetic tests objects.)

This would be very useful to pinning services; all of which currently overcharge their customers for the CumulativeSize of pinned objects and not the actual storage space used.The IPFS community would benefit from more accurate storage accounting and cheaper pin storage services. Ideally, pinning services would create per-customer objects and charge them for the deduplicated storage space across all their pins.

What follows are instructions for recreating a test object in case it gets garbage collected or otherwise lost in time. It’s not really relevant for the issue itself except explaining the test object.

#!/bin/bash
cd $(mktemp -d)
mkdir dir/
mkdir dir/1
head -c 5M </dev/random > dir/1/file1
head -c 5M </dev/random > dir/1/file2
mkdir dir/2
head -c 5M </dev/random > dir/2/file3
cat dir/1/file1 > dir/2/file4
cat dir/1/file2 >> dir/2/file4
ipfs add -r dir

[bonedaddy]

@da2x i believe i found a way to replicate this using already existing stats according to ipfs object stat:

I took your example hash, adding up all block sizes of unique recursive references which resulted in:

calulcated ref size 15734625
expected   ref size 15729480

Adding up recursive unique references datasizes landed at:

calulcated ref size 15729672
expected   ref size 15729480

the "expected" ref size is the size according to the size you indicate in this issue.

Here's my somewhat less hacky solution im using with our package that interfaces with IPFS, unfortunately I couldn't get the built-in go-ipfs-api to work properly with getting refs sizes and it constantly returned 1 ref, so I resorted to running a raw command

// DedupAndCalculatePinSize is used to remove duplicate refers to objects for a more accurate pin size cost
func (im *IpfsManager) DedupAndCalculatePinSize(hash string) (int64, error) {
	// format a multiaddr api to connect to
	parsedIP := strings.Split(im.nodeAPIAddr, ":")
	multiAddrIP := fmt.Sprintf("/ip4/%s/tcp/%s", parsedIP[0], parsedIP[1])
	outBytes, err := exec.Command("ipfs", fmt.Sprintf("--api=%s", multiAddrIP), "refs", "--recursive", "--unique", hash).Output()
	if err != nil {
		return 0, err
	}
	scanner := bufio.NewScanner(strings.NewReader(string(outBytes)))
	var refsArray []string
	for scanner.Scan() {
		refsArray = append(refsArray, scanner.Text())
	}
	var calculatedRefSize int
	for _, ref := range refsArray {
		refStats, err := im.Stat(ref)
		if err != nil {
			return 0, err
		}
		calculatedRefSize = calculatedRefSize + refStats.DataSize
	}
	return int64(calculatedRefSize), nil
}

[da2x]

My example command was slightly off. I updated it above and its behaviour now matches your go implementation.

[bonedaddy]

Woot! I believe my go implementation will work regardless of chunk size

[Stebalien]

So, "CumulativeSize" is calculated based on metadata recorded in the object itself. We'd probably want something like:

• ipfs repo stat <object>
• ipfs dag stat <object>

I'd vote for the latter, actually. See: #3955.

[aschmahmann]

There is an ipfs dag stat command that does this now #7553.