Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

implement replacement for dagcargo as it is deprecating #7

Closed
8 tasks done
alanshaw opened this issue Mar 12, 2024 · 6 comments
Closed
8 tasks done

implement replacement for dagcargo as it is deprecating #7

alanshaw opened this issue Mar 12, 2024 · 6 comments
Assignees
@travis

Comments

@alanshaw
Copy link
Member

alanshaw commented Mar 12, 2024 •
edited by hannahhoward
Loading

See also

Reviewer: @vasco-santos
@heyjay44
Copy link

Deals that were made pre Spade were dealt with and we do not need to worry about renewals.

@gobengo gobengo changed the title dagcargo sunset nft.storage should use w3up for storage and filecoin dealmaking Mar 13, 2024
This was referenced Mar 13, 2024
Open
Merged
gobengo added a commit to nftstorage/nft.storage that referenced this issue Mar 19, 2024
@gobengo @vasco-santos
docs: add decisions/20240313-try-w3up.md (#2518)
06712ad 
Motivation:
* write decision to implement first part of
storacha/project-tracking#7

Status
* [x] initial plan drafted
* [x] update based on feedback

---------

Co-authored-by: Vasco Santos <santos.vasco10@gmail.com>
@gobengo gobengo mentioned this issue Mar 19, 2024
Merged
1 task
@gobengo
Copy link

gobengo commented Mar 22, 2024

convert NFT storage infra to use w3up client to act as a normal customer of w3up and go through normal flow to add the data into w3up. Then all existing filecoin pipeline stuff just happens by default

This PR has an implementation: nftstorage/nft.storage#2522

@reidlw reidlw changed the title nft.storage should use w3up for storage and filecoin dealmaking implement replacement for dagcargo as it is deprecating Mar 27, 2024
alanshaw pushed a commit to nftstorage/nft.storage that referenced this issue Mar 28, 2024
@gobengo @vasco-santos @travis
feat: POST /uploads can write uploads to w3up for web3 serving and da…
0c77cb3 
…gcargoless filecoin deal-making (#2522)

Motivation:
* part of storacha/project-tracking#7
* implement #2518

Note
* many files I modified only because the upgrade to new typescript
compiler or eslint required it
* w3up-client requires node 18 so this drops support for node 16 from
the api package
* I think the client will still work on node 16, but I couldn't find a
way of getting the ci for client to, using node 16, install just the
client package deps and not the api package deps (which tries to install
w3up-client, and then `yarn` errored because that requies node 18)
* w3up-client requires node 18, but everything had been running on node
16
* [ ] if we can't find a way of continuing to use v16, then when this
merges to main, we probably need to go into cloudflare workers settings
and change NODE_VERSION from 16 to 18+

---------

Co-authored-by: Vasco Santos <santos.vasco10@gmail.com>
Co-authored-by: Travis Vachon <travis.vachon@gmail.com>
Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
@gobengo
Copy link

gobengo commented Apr 2, 2024 •
edited
Loading

w3up-client integration is deployed behind a feature switch in staging.nft.storage.

Good news is that it can be used to forward an 8MB upload to w3up without error.

Bad news is that it doesn't work in all circumstances, and we're still debugging why
In testing we see that uploads that use it can fail in at least two ways:

  • when the nft.storage client shards files into CARs before sending into nft.storage HTTP APIs, the HTTP request uploading the CAR can respond with a 503 and we see an error like "Error: Worker exceeded memory limit."
  • when the nft.storage client doesn't shard, but files are around 45M, the upload can fail due to a different uncaught exception from inside uploadCAR

Update: This helped with the OOM errors and maybe more nftstorage/nft.storage#2551

@reidlw reidlw mentioned this issue Apr 10, 2024
Closed
@alanshaw
Copy link
Member Author

I am pg_dumping dagcargo tables: aggregate_entries, aggregates, deals, dags, metrics, metrics_log.

I will import into a new postgres in Heroku and check it can be attached as a foreign data wrapper to classic.nft.storage/old.web3.storage staging DBs. If successful I'll attach it to production DBs.

@reidlw reidlw assigned alanshaw and unassigned gobengo Apr 10, 2024
@alanshaw
Copy link
Member Author

New DB in Heroku created: dagcargo ✅

Contains exported subset of dagcargo: aggregate_entries, aggregates, deals, dags, metrics, metrics_log.

Created 2 read only users nftstorage and web3storage - i.e. different user for each product but staging and production use the same user.

@alanshaw alanshaw assigned travis and unassigned alanshaw Apr 15, 2024
@hannahhoward
Copy link
Member

@travis just testing api -- deploy to staging today, then prod soon, then we're done! 4/15/24

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@travis travis

Labels
None yet
Projects
Storacha Project Planning
Status: Done
Milestone
No milestone
Development

No branches or pull requests
5 participants
@travis @alanshaw @gobengo @hannahhoward @heyjay44