2018 Q4 OKR Planning #5474
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # Quarterly Objectives and Key Results | ||
|
|
||
| We try to frame our ongoing work using a process based on quarterly Objectives and Key Results (OKRs). Objectives reflect outcomes that are challenging, but realistic. Results are tangible and measurable. | ||
|
|
||
| ## 2018 Q4 | ||
|
|
||
| **go-ipfs handles large datasets (1TB++) without a sweat** | ||
| - `PX` - It takes less than 48 hours to transfer 1TB dataset over Fast Ethernet (100Mbps) | ||
| - `PX` - It takes less than 12 hours to transfer 200GB sharded dataset over Fast Ethernet (100Mbps) | ||
| - `PX` - There is a prototype implementation of GraphSync for go-ipfs | ||
|
There was a problem hiding this comment. somewhat dependent on js-ipfs graphsync There was a problem hiding this comment. @hannahhoward already thinking about speeding up directories and may have knowledge of how to implement (but needs a partner for spec work) |
||
| - `PX` - There is a better and more performant datastore module (e.g Badger or better) | ||
|
|
||
| - `P1` - Rewrite pinning data structures to support large data sets / many files performantly | ||
|
There was a problem hiding this comment. I would explicitally talk about adding "alternative pinning strategies". go-ipfs can already pin very large data sets performantly in terms of the pin data structure (the problems is perhaps verifying and traversing and safely gc-ing after unpinning them). Cluster's need are much more specific and are about adding one extra pinning mode. The wider discussion about the pinning system goes beyond the data structures too, and affects multiple components, so a KR about that would need different wording than this too. There was a problem hiding this comment. @Stebalien could you comment about what might be a better phrasing? There was a problem hiding this comment.
Unfortunately, the datastructure doesn't support many pins. See: #5221 However, I agree. Part of this goal is to make the pinning datastructures flexible. There was a problem hiding this comment. Just FYI: Personally, I think we should switch to using inode like structures (with small blocks inline in the inode itself) in the Blockstore. This will be a major infrastructure change but I think it will make a lot of things a lot easier down the road, including pinning. I created a placeholder issue for this at #5528, I will flush out the details later today or tomorrow. There was a problem hiding this comment.
Agreed, I was thinking of large datasets as a thing with a single root cid and not on the pinset as a whole. But yes, the pinset itself is expensive to edit when it's very large. 👍 There was a problem hiding this comment. One simple way of solving this would be transitioning pinset into datastore (backed by DB) directly instead of using ipfs objects for that. There was a problem hiding this comment. cluster wants the ability to pin partial dags - ok with a phrasing like: There was a problem hiding this comment. @kevina interested in contributing to this - not sure about owning per say yet =] There was a problem hiding this comment. I agree with @Kubuxu on using an actual database but, baring that, we could use the new go-ipld-hamt structure (now that we've finally merged the refmt cbor ipld branch). |
||
|
|
||
| **The bandwidth usage is reduced significantly and is well kept under control** | ||
|
There was a problem hiding this comment. I like the idea of contributing to this objective, but I'm probably not the best person to own it. There was a problem hiding this comment. @Stebalien who do you suspect should own these OKRs? In the absence of a more qualified person (and in an effort to move important goals forward), I'm willing to take on some or all ownership. If there's a better owner, then I can work with them towards this objective. |
||
| - `PX` - Users can opt out of providing every IPLD node (and only provide root hashes) | ||
|
There was a problem hiding this comment. Includes modifications to the DHT. Need an owner to coordinate with @Stebalien on design. @magik6k started on this, but need to flesh out the design and spec. Need to rephrase this to be aiming for a spec and early implementation (.5 is a spec but no implementation started) |
||
| - `PX` - "Bitswap improvements around reducing chattiness, decreasing bandwidth usage (fewer dupe blocks), and increasing throughput" | ||
|
There was a problem hiding this comment. measure: looking at the number of duplicate blocks and the number of parties we send a want list to when we don't need to can make larger improvements by changing protocol, but current proposal is internal to feature to improve where we look for data |
||
|
|
||
| **It is a joy to use go-ipfs programatically** | ||
momack2 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - `PX` @magik6k - The Core API is finalized and released. Make it easier to import go-ipfs as a package | ||
|
There was a problem hiding this comment. what is a good priority for this? |
||
| - `PX` - go-ipfs-api exposes the new Core API | ||
momack2 marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. @diasdavid did you intend to let this also include the remote API protocol in this? Because I'm not sure it's covered in this document? There was a problem hiding this comment. My plan for this was to start with http api implementation and when the new rpc api becomes a thing support both as some users will likely need http is some setups There was a problem hiding this comment. Do you have a proposal for an updated KR that effectively captures that work? Taking PRs! =] There was a problem hiding this comment. @keks - Do you think the RPC API should be included (even partially) in this document? I don't know anything about that work but is it something that you think should be started in this next quarter? There was a problem hiding this comment. @magik6k is owner for this whole objective making go-ipfs-api expose is lower priority (blocked by a lot of go-ipfs changes) There was a problem hiding this comment. To clarify, I was basically talking about this issue. We'll need to coordinate with the other implementors to find out what we really need, but the issue is that the current HTTP API has a lot of weird edge cases, which is painful for everyone who implements it. |
||
| - `PX` - go-ipfs Daemon, Gateway, and cmds library use the new Core API | ||
| - `PX` - The legacy non Core API is deprecated and the diagram on go-ipfs README is updated | ||
|
|
||
| **go-ipfs becomes a well maintained project** | ||
momack2 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - `PX` @eingenito - The Go Contributing Guidelines are updated to contemplate expecations from Core Devs and instructions on how to be an effective contributor (e.g include PR templates) | ||
| - `PX` @eingenito - A Lead Maintainer Protocol equivalent is proposed, reviewed by the team, merged and implemented | ||
| - `PX` @eingenito - Every issue on https://waffle.io/ipfs/go-ipfs gets triaged (reviewed and labeled following https://github.com/ipfs/pm/blob/master/GOLANG_CORE_DEV_MGMT.md) | ||
momack2 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - `PX` - Every package has tests and tests+code coverage are running on Jenkins | ||
| - `PX` - There is an up-to-date Architecture Diagram of the Go implementation of IPFS that links packages to subsystems to workflows | ||
|
|
||
| **gx becomes a beloved tool by the Go Core Contributors** | ||
momack2 marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Who has made major changes to gx in the past? Is it only @whyrusleeping? Are these actually appropriate for someone else to pick up? There was a problem hiding this comment. @travisperson will probably take these (and @schomatis has been doing some gx work). We have a long discussion here whyrusleeping/gx#179 culminating with an offline discussion between @whyrusleeping, @travisperson and I (key points here whyrusleeping/gx#179 (comment)). There was a problem hiding this comment. Thanks. @travisperson are you comfortable with my adding your name to these OKRs? Are they pretty accurate as stated, and might they belong in another project? There was a problem hiding this comment. @eingenito ya, these looks good. |
||
| - `PX` - | ||
| - `PX` - | ||
|
There was a problem hiding this comment. @Stebalien I know there are a lot of new goals for GX but I really can't tell what these are. Could you describe the ones planned for Q4 as KRs? There was a problem hiding this comment. @warpfork mentioned this in his retrospective comment - do you know if there's a tracking issue? There was a problem hiding this comment. The thing I remember hearing is semver based heuristics for flattening dependencies, so you'll get a minimal set of imports of any given package. In his retro @warpfork mentioned changes that keep gx from doing so much import rewriting - I'm not sure how that would work but that sounds neat. There was a problem hiding this comment. Steven suggested these KRs for gx:
There was a problem hiding this comment. @whyrusleeping who had thoughts on this There was a problem hiding this comment. ping to both @whyrusleeping and @Stebalien to iterate on gx KRs that would measure success of this tool being productive/efficient There was a problem hiding this comment. KR: update a transitive dependency without touching intermediate dependencies |
||
|
|
||
| **Complete outstanding endeavours and still high priorities from Q3** | ||
| - `P0` @kevina - base32 is supported and enabled by default | ||
| - `PX` - go-ipfs gets a unixfsV2 prototype | ||
|
There was a problem hiding this comment. before a prototype we should probably have a spec we agree on =] There was a problem hiding this comment. Mikeal wants to hand off spec work to someone else There was a problem hiding this comment. @mikeal is working on this... There was a problem hiding this comment. To be more specific, I'm working on the spec, and I can easily hand off parts of the spec once the skeleton lands (should be done next week). We should be able to start implementing once the (skeleton + data) PRs land, as we don't need special file types for MFS (as far as I know). I'll prioritize getting those landed so that we don't block early implementation work. I'm not that familiar with the Go side of things but I believe @warpfork and @Stebalien mentioned that the IPLD API's on the Go side need to get revised and I'm not sure if that work is also going to block this implementation. If the goal of the unixfs-v2 implementation is to be somewhat codec agnostic then it probably will. Review and revisions to those APIs are in the IPLD Q4 OKR's. |
||
| - `PX` @djdv - IPFS Mount | ||
|
There was a problem hiding this comment. @djdv what would be the quantifiable KR for IPFS Mount? There was a problem hiding this comment. How does this sound?
To recap the goal as it is now, we just want to have a new implementation of mount that takes advantage of all of the other new work that has been done (updated commands lib, CoreAPI, etc.), as well as add functionality such as a read+write MFS mountpoint. And we want to make sure this is actually true for all platforms we support (not just some of them). There was a problem hiding this comment. @djdv is r+w mount across platforms seems like a lot for a quarter, but I don't know much about it or the current state. Is that an appropriately sized quarterly effort? Or is there something you're focusing on first that makes more sense for this goal? There was a problem hiding this comment. @eingenito I've got an issue to track the status here: All info below is supplementary, feel free to disregard it. Despite the implementation being a straightforward task, getting it right is where the complications come in. The example I've been using, is an HTTP server. So a bulk of the work in My current focus is on finalizing the mount implementation first and then getting it building dynamically on other platforms. We likely won't have platform specific problems in our codebase, that's all underneath us. So we'll have a time where the implementation is finished and ready for review, but there's still work to be done in the build area. As it is right now, we'll always be able to run on other platforms, but we currently need a C compiler at build time. It's possible that we can eliminate this and dynamically link with fuse at runtime, which seems preferable. You'll always have the option to build both though. A lot of this is already thought out, but the actual implementation of it needs to be finished and refined. |
||
|
|
||
| ## 2018 Q3 | ||
|
|
||
| Find the **go-ipfs OKRs** for 2018 Q3 at the [2018 Q3 IPFS OKRs Spreadsheet](https://docs.google.com/spreadsheets/d/19vjigg4locq4fO6JXyobS2yTx-k-fSzlFM5ngZDPDbQ/edit#gid=274358435) | ||
|
|
||
| ## 2018 Q2 | ||
|
|
||
| Find the **go-ipfs OKRs** for 2018 Q2 at the [2018 Q2 IPFS OKRs Spreadsheet](https://docs.google.com/spreadsheets/d/1xIhKROxFlsY9M9on37D5rkbSsm4YtjRQvG2unHScApA/edit#gid=274358435) | ||
|
|
||
| ## 2018 Q1 | ||
|
|
||
| Find the **go-ipfs OKRs** for 2018 Q1 at the [2018 Q1 IPFS OKRs Spreadsheet](https://docs.google.com/spreadsheets/u/1/d/1clB-W489rJpbOEs2Q7Q2Jf1WMXHQxXgccBcUJS9QTiI/edit#gid=2079514081) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Love these as KRs, but we should separately track the work we will do to achieve these key results. Is this faster bitswap?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe that smarter bitswap is part of the way there, but there will be other parts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Steven says the second metric isn't testing something useful - he proposes:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(replacement for the second KR - keep the first)