cdnjs

[victorb]

Would love to have a IPFS compatible fork of https://github.com/cdnjs/cdnjs serving files via IPFS. Super large repository though but will give it a try to develop the integration locally.

[jbenet]

cc @lgierth

[davidar]

On hold until ipfs add performance improves, too many small files

[eminence]

Ok well I bit the bullet and added this to IPFS. Here is the result: http://ipfs.io/ipfs/QmZJKsLpebYqHApRLeoLhj2NsXZ2JoXqhSWkxafxaBXYu7

A few observations. Note that for all of the below comments, the daemon was not running, and I was IPFS version 0.3.11-dev

• Performance wasn't as terrible as I thought it would have been, but it was pretty darn slow. Took about 15 or 16 hours to add everything
• When I first added things, I did it on directory at a time:
  $ ls ajax/libs | xargs -N1 ipfs add -r
  As expected, this kept memory usage in check (mostly).
• Later, after all the data was in IPFS, I tried added the entire ajax/libs directory at once. This went quite quickly, however memory usage ballooned very quickly to about 75% of my machines available memory (about 24 GB used out of a total of 32GB available). I killed ipfs at this point, and wasn't able to let it run to completion.
• Since I wanted to have a single tree containing everything, and I wasn't able to add the whole directory at once, I had to resort to building up a tree with ipfs object patch, by manually calling add-link for each directory. This was very easy to do, but it was very slow! It appears that this large repo is causing some serious delays when running IPFS commands:

$ time ipfs object new unixfs-dir
QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn

real    0m14.855s
user    0m19.136s
sys     0m0.632s

This means that is took 6 hours to call object patch add-link for every top-level directory. I believe this 15 second delay is just some startup delay, so if I had the daemon running at this point, I believe this process would have been much faster (since there would be no startup delay)

• IPFS deduplication appears to be helping a lot here! Compare the size of the cdnjs/ajax/lib directory to ~/.ipfs :

achin@diax:~/devel$ du -sh cdnjs/ajax/libs/ ~/.ipfs/
22G     cdnjs/ajax/libs/
13G     /home/achin/.ipfs/

I took a look at every single hash to see what ones were duplicated. In total, about 180000 difference hashes were dedups at least once in this tree. The most dedupd hash was a small 1x1 PNG file with 12376 links. The second most deduped file is an empty file with 696 links.

I've uploaded a file that lists all dedups. I haven't done the analysis to see if it would account for the savings reported by du

• Time to pin everything. Not terrible:

$ time ipfs pin add -r QmZJKsLpebYqHApRLeoLhj2NsXZ2JoXqhSWkxafxaBXYu7
pinned QmZJKsLpebYqHApRLeoLhj2NsXZ2JoXqhSWkxafxaBXYu7 recursively

real    7m24.915s
user    5m16.676s
sys     0m32.560s

• I've uploaded a copy of the output of ipfs diag sys for your info

[davidar]

@eminence awesome, thanks for tackling this and writing up the details :)

Cc @whyrusleeping @rht @diasdavid

[eminence]

Second day follow up notes:

• Updating this repo is pretty straight-forward. After fetching the cdnjs repo, and fast-forwarding to the latest master, I can use git diff --dirstat to see just the libraries that changed. This makes it easy to ipfs add -r -q ajax/libs/library_that_was_updated | tail -1 and then ipfs object patch add-link. I've confirmed that the object patch is nice and quick when I leave the daemon running.
• Note that I'd love a mode in ipfs add that just gives me the top-level hash of the thing that I'm adding, so I can dispense with the tail -1 stuff.
• This manual tree management is very doable, but also annoying. I admit I've not been following the Files API stuff -- would it help here?
• To make use of these hosted libraries, I've added them to IPNS here: http://ipfs.io/ipns/em32.net/archives/cdnjs

But IPNS is still too slow. Or rather, it is not consistently fast. I wrote a script that requests the same file several times, and records how long it takes. The data for 60 requests is here. The summery is that most of requests are quick (about 0.5 seconds or less). But some requests take 10 seconds, others up to 60 seconds. When your website is loading many different assets (like cdnjs libs), this can result is very noticeable delays.

• I have (anecdotal) evidence that even trying to ipfs get some of these large cdnjs directories can cause run-away memory leaks. This may make it difficult for this repo to be effectively mirrored/pinned.
• I've committed to @whyrusleeping to re-run these tests with the 0.4.0 branch. I'll report back in the coming days

[davidar]

Note that I'd love a mode in ipfs add that just gives me the top-level hash of the thing that I'm adding, so I can dispense with the tail -1 stuff.

I think ipfs add -q does that

This manual tree management is very doable, but also annoying. I admit I've not been following the Files API stuff -- would it help here?

Probably, yes.

[jbenet]

I think ipfs add -q does that

ipfs add -q outputs only hashes. still need tail -n1. but yeah, we should add a flag that outputs only the last root. maybe --only-root ? or -Q, --root-quiet? idk.

[jbenet]

This manual tree management is very doable, but also annoying. I admit I've not been following the Files API stuff -- would it help here?

yes, very much so.

[jbenet]

But IPNS is still too slow. Or rather, it is not consistently fast. I wrote a script that requests the same file several times, and records how long it takes. The data for 60 requests is here. The summery is that most of requests are quick (about 0.5 seconds or less). But some requests take 10 seconds, others up to 60 seconds. When your website is loading many different assets (like cdnjs libs), this can result is very noticeable delays.

yes, ipns is still very slow. there's caching (the fast results), but we need to fix this at the dht query level

[eminence]

After testing with 0.4.0, the experience was much better!

• Memory usage was totally under control, and mostly stable the entire time.
• I was able to add the entire libs directory at once. This took about 9 hours, which is much quicker than before. This isn't a totally fair comparison, since I used different hardware for each test (0.4.0 versus 0.3.8), but even trying to take into account the hardware differences, I believe 0.4.0 was faster
• Rescanning the entire libs directory, after the initial add, took about 50 minutes. This would represent the time needed to do an incremental update. Obviously using the git diff trick from above would improve this, but 50 minutes seems pretty acceptable to me.

If anyone is running a 0.4.0 node, the result is here: QmRnvPSCNmYHdYQAo6JUWJPW8uVQv7z6D9nSQmw5qbHVWy

[jbenet]

Good news. Though still waaaay too slow for my liking.

Adding concurrent add will help here. I believe we have not added this.
On Tue, Dec 15, 2015 at 09:08 Andrew Chin notifications@github.com wrote:

After testing with 0.4.0, the experience was much better!

Memory usage was totally under control, and mostly stable the entire
time.

I was able to add the entire libs directory at once. This took about 9
hours, which is much quicker than before. This isn't a totally fair
comparison, since I used different hardware for each test (0.4.0 versus
0.3.8), but even trying to take into account the hardware differences, I
believe 0.4.0 was faster

Rescanning the entire libs directory, after the initial add, took
about 50 minutes. This would represent the time needed to do an incremental
update. Obviously using the git diff trick from above would improve this,
but 50 minutes seems pretty acceptable to me.

—
Reply to this email directly or view it on GitHub
#35 (comment).

[eminence]

For me, the gold standard of merkledags is probably git :) So I timed how long it takes to add this directory tree into a new git repo -- 41 minutes! About an order of magnitude faster than IPFS.

[whyrusleeping]

@eminence yeah, we've got a little ways to go still, but keep in mind that git uses a faster hashing algorithm, and doesnt chunk objects. I have one more changeset to apply that should get close to leveling the playing field. Just have to polish it a bit.

[rht]

In some cases, e.g. lots of 1MB files instead of 1KB files, ipfs is way faster than git ipfs/kubo#1973 (comment)

[rht]

git uses a faster hashing algorithm, and doesnt chunk objects.

The chunking equivalent in git would be git gc, but only after several revisions?

[jbenet]

we should consider moving the default to https://blake2.net/ it's designed for this.

[rht]

That would explain rsync's speed, since it uses blake2. Perhaps it is possible to go ~O(rsync) with this.

It would be more effective, though, to use blake2 into 0.4.0 before its release (combining all the incompatible changes into one release?).

[jbenet]

It would be more effective, though, to use blake2 into 0.4.0 before its release (combining all the incompatible changes into one release?).

blake2 isnt incompatible. https://github.com/jbenet/multihash :)

[jbenet]

oh i guess it is, because it's not included in the 0.3.x codebase, right.

[jbenet]

yeah it would be nice, but idk if we can land it in time. @whyrusleeping wants to ship 0.4.0 soon. would be nice to add blake2 and ipld support, but not sure if we'll get there in time

[eminence]

Even though git is super fast here, ipfs isn't unusablely slow. as an end user, i have so many more things on my wishlist that are more important to me than speed

(edit: but i do indeed understand the desire to put all breaking changes into 1 release)

[davidar]

@rht What percentage of the total runtime is currently consumed by the hash function?

[rht]

...I did instead with testing dev0.4.0+blake2b. There is more cpu consumption and a speedup but not noticeable due to the jitter like in ipfs/kubo#2039 (comment). Perhaps it could be significant once dev0.4.0 goes to ~O(git) or ~O(rsync). Though I could have just checked the runtime percentage of the hashing.

Currently, both ipfs and git are slow for adding large things.
edit: s/moving/adding/

[Kubuxu]

I re-archived it on my own. Available under fs:/ipns/cdnjs.ipfs.ovh

Notes: It took about 6h, I had one crash during that time.

[xioustic]

Anyone interested in a deterministic, efficient node script that wraps the command-line to do the git pull / git diff / ipfs add / ipfs object patch manual work that does this into the existing index you last put into the IPFS network? Could also handle the memory issue (restarting the daemon or sets of potential offending processes when they get near a threshold).

About four months ago I did this and I'll need to dig up my work sitting on one of my VPSes; simple dumb 'ipfs add -r' every time I got pulled was quite expensive. Might be reusable in other related archive targets (other big git repos that are more archive repositories than single codebases). Just requires 'git', 'ipfs' and 'node' on the path of the indexing machine. Maybe someone has already done this or perhaps the ipfs internals have improved enough to remedy the need for this?

I just found out about this "IPFS Archives" project at the Decentralized Web Summit, pretty exciting. @whyrusleeping still doing the workshop on IPFS Archives and Versioning, so tagging him.

[Kubuxu]

Me and @magik6k are working on adding the cdnjs currently.

I added whole cdnjs some time ago, it is available under fs:/ipns/cdnjs.ipfs.ovh but I think most of that data is currently gone (due to problems with local IPFS repo).

Adding cdnjs is nice stress test for IPFS: ipfs/kubo#2823 ipfs/kubo#2828

[Kubuxu]

Note about publishing the cdnjs:

We shouldn't use /ipns/ to reference the cdnjs in HTML. IPNS currently doesn't work with browser level caching (no HTTP 304 response code if path is IPNS path).

[notslang]

Updated CDNJS hash: https://ipfs.io/ipfs/QmPJnEf5933cXteZmaMJkphCW1CtpcMMVx7N6rUr8cZAok
And a little script for generating it, in case anyone else wants to try:

#!/bin/bash
HASH=$(ipfs object new unixfs-dir)
for FILE in cdnjs/ajax/libs/*; do
  LIB=$(basename $FILE)
  echo "adding $LIB"
  LIB_HASH=$(ipfs add -r -H -q "$FILE" | tail -n 1)
  HASH=$(ipfs object patch $HASH add-link $LIB $LIB_HASH)
done
echo "final hash: $HASH"

...However, this doesn't handle symlinks, which do exist in CDNJS. I've gotta decide how to deal with those.

[eminence]

What needs handling exactly (with regard to symlinks)?

[notslang]

If you click on one right now (like https://ipfs.io/ipfs/QmPJnEf5933cXteZmaMJkphCW1CtpcMMVx7N6rUr8cZAok/zocial) it's broken, whereas https://cdnjs.cloudflare.com/ajax/libs/zocial/1.2.0/css/zocial.css works.

[eminence]

Ahh. I see, yes. I would just pretend the symlink doesn't exist and just add the contents of the directory. Let IPFS's intrinsic de-dup handle it from there

[eminence]

BTW, I am trying to pin this hash, but I can't download everything. Are you still seeding it, @slang800 ?

[Kubuxu]

@eminence you are better of, instead of pinning it right away, doing ipfs refs -r $HASH, this way you will see progress, just make sure to set high GC limit (the cdnjs will be about 20GB) and pin it afterwards.

[eminence]

Actually, I am already doing that (using ipfs refs). I was trying to run ipfs refs QmWLDMm1CC2E5f7M5dcbi9hW9K9XuckFUfGc39GRgGYtQC, but after about 30 hours of waiting nothing came back. But I just restarted the command just now and it returned instantly. Not sure what that means... ipfs bug?

[eminence]

Or maybe I just can't connect to @slang800 node?

When I run ipfs dht findprovs QmeQYktmYVqAbRju1boAyimC9J1dAACYY7h5KNqfEEEamb, nothing at all is returned. But if I request that hash via the ipfs.io public gateways, it loads almost immediately. Then ipfs dht findprovs returns something:

> ipfs dht findprovs QmeQYktmYVqAbRju1boAyimC9J1dAACYY7h5KNqfEEEamb
QmVyqFjQJTqVmKRBk4sL9F9Af7fCRdA9YNK845NSHRD8zJ
QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd

> ipfs swarm peers |wc -l
117

So I'm a little confused about what's actually happening here

[Kubuxu]

It is possible that you can't connect with @slang800 node directly. Maybe his node failed to penetrate NAT and only connected to SolarNet nodes/can't accept connections, this would be why you can see things work via ipfs.io but not through your own instance.

[notslang]

Sorry - I was hosting this on my desktop and turned it off this morning to move my desk to the opposite side of the room. It should be up now. :D

[Kubuxu]

@slang800 can you share your peerID (result of ipfs id -f"<id>\n") for debugging?

[notslang]

Sure, it's QmVKZAcpqrrQoBsc7eyEFnqFjHTDAUdJWMTBDJ34jR7ueU

[victorb]

Submitted an updated build based on github.com/cdnjs/cdnjs commit 4fabd85c986d57a61e0fbd8504cf15d67f60ada6 here: #82

New hash would be: QmRrnfFUgx81KZR9ibEcxDXgevoj9e5DydB5v168yembnX - https://ipfs.io/ipfs/QmRrnfFUgx81KZR9ibEcxDXgevoj9e5DydB5v168yembnX

It's stored at Pollux right now.

[PeterDaveHello]

@cdnjs maintainer here, anything I can help here?

[victorb]

@PeterDaveHello Thanks for checking in here! I think something that would be really useful is adding "ipfs.io" as one of the CDN providers, however, I'm not sure if you have support for adding more, currently it's just Cloudflare there without the ability to change.

If we did that, we would need to setup the updating/adding to be more automatic, right now it's me doing a manual git pull and adding it to IPFS, then making a PR here. The flow would be something like:

• Once a day, update from git
• Add to IPFS
• Submit updated hash as a PR to this repository
• Submit updated hash as a PR to cdnjs/cdnjs
• You make a new deploy with updated hash on the website

What you think?

[PeterDaveHello]

I'm afraid that we can't do that as we update the library and website every 5~10 mins since the 3k+ libraries update very frequently. This is all automatically, without manual review and merge.

I also wonder if it's a good idea to push ipfs when I don't understand this project enough, when we're going to provide service officially, we'll have responsible on that, especially when there is anything wrong, so, sorry, that might not be something I can do right now. Maybe I can help update the files from my side if you want, currently, the files looks dated on ipfs.

[victorb]

I'm afraid that we can't do that as we update the library and website every 5~10 mins since the 3k+ libraries update very frequently. This is all automatically, without manual review and merge.

Yeah, understandable, and making it all automatic is a much better way to go from the get-go so makes sense.

Something else we can do from our side is having the same interface you run on cdnjs.com but slightly modified to hook-up into our version of cdnjs, and deployed on cdnjs.ipfs.io or something like that. Would need to make sure it's always up-to-date, which will take some effort but not be super hard.

Maybe I can help update the files from my side if you want, currently, the files looks dated on ipfs.

Yeah, as I mentioned, the process is right now manual but in reality, should be fully automated. Will have some more thoughts about this at a later point.

Thanks for jumping in here and sharing your thoughts 👍

[PeterDaveHello]

@victorbjelkholm thanks! Let me know if I can help update cdnjs on ipfs more up-to-update and frequently :)

[magik6k]

IPFS stores files much like git does, so updating it 'live' shouldn't really be a problem. This updating could be done using the ipfs files or ipfs object api quite easily.

The best way I can see it done in case of cdnjs is to have a tool that would apply updates based on which files changed in git commits.

Only thing I'm not sure about is how IPNS would react to that frequency of updates.

[PeterDaveHello]

Yeah we can try to integrate that in our buildScript