RFC: Code4rena Supreme Court

[CloudEllie]

Code4rena’s rules have evolved significantly since C4 ran its first audit in early 2021. Up to now, rule changes and iterations have been decided, announced, and implemented primarily by C4 staff — albeit with extensive consultation with the community.

This proposal outlines a process for formalizing and standardizing Code4rena’s processes for rule review and iteration, to promote more timely, thoughtful, and strategically sound decisions.

Proposed process

1. Three Code4rena judges in good standing will be nominated to the Code4rena “Supreme Court.”
   1. The following judges are being nominated to lead the first session: Alex the Entreprenerd (AKA GalloDaSballo), LSDan, and Trust.¹
2. The Supreme Court will hold sessions at least twice a year, and up to four times a year, based on court members’ availability and the current backlog of open, unresolved discussions in the org repo.
3. During each Supreme Court session, members will:
   1. Review all open org repo issues
   2. Look at instances where the relevant situation has been judged (i.e. what's the judging track record? Is there emerging consensus or prevailing sentiment?)
   3. Invite input from the judges in question regarding their past decisions, if relevant
   4. Rule on the issue²
   5. Rewrite the C4 audit rules accordingly
4. “Post-judging QA” phase (duration TBD, but probably ~1 week): Draft rule rewrites will be posted for review and comment via the C4 org repo (and announced in Discord).
5. C4 staff will administer and support relevant processes, implement new and updated rules, and communicate the Supreme Court’s decisions to the community.
6. New rules will be implemented as quickly as possible. Unless specified otherwise, new rules will take effect for all audits that start after the new rules have been published at docs.code4rena.com.

Next steps

1. The first Code4rena Supreme Court session will take place in Autumn 2023.
2. A budget of $84,000 USDC will be allocated for the Fall 2023 session, to compensate the judges for their efforts.³
3. Wardens are invited to review all open org issues, and file a new issue for any area of concern that is not already flagged/open.⁴

Notes

Footnotes

1. The initial slate of C4 Supreme Court judges has been nominated by C4 staff based on a combination of length of tenure, number of Code4rena audits judged, diversity of perspectives, and track record participating in C4 with a variety of roles (warden, sponsor, judge, etc.). In future, the supreme court could be formed via elections or some other approach.

   1. If a judge resigns from the initial session or is unable to perform their duties, C4 staff will nominate a backup judge.
   ↩

2. One of the discussion points for the initial session will be how to converge to a decision (e.g. majority vote, unanimous consensus, veto, etc.) ↩

3. The budget for the first session is based on

   • the estimated effort, given that Code4rena has multiple years’ worth of structural/procedural “debt;
   • that this group of judges is being asked to create new systems and processes; and
   • the desire to ensure sufficient time and incentive for our experts to deliberate and come to consensus.

   We expect that the first session will require the most effort, and that future sessions could be done with a somewhat smaller budget. ↩

4. One risk with making this an open process is that it has potential to be bogged down with “spam,” e.g. many newly-created issues that either duplicate existing issues or fail to meet the criteria for a valid org repo issue. During the Fall 2023 session, the court members will be asked to review and clarify C4’s appeals process, and establish a set of criteria that appeals must meet in order to merit further review. ↩

[CloudEllie]

In re-reading this RFC, there is some risk of misinterpretation around the phrase, "all open org repo issues,” and the respective purviews of the Supreme Court vs Code4rena staff. To clarify:

• The Supreme Court is intended to be authoritative on all things judges and lookouts have responsibility for deciding, including how issues are judged, how current Code4rena rules are interpreted and applied, etc.
• C4’s product and process are the responsibility of staff to steward.
• “Rewrite the C4 audit rules” is intended to mean “fine tune the rules,” and propose broader changes if necessary to ensure they are consistent with Code4rena’s overall goals and philosophy.

Code4rena staff may refer additional issues to the Supreme Court for deliberation and advice, and the Supreme Court may provide consultative recommendations to staff on any product or process issues.

To clarify item 3(a) under “Proposed process” — “Review all open org repo issues” — here is a proposed workflow:

1. Supreme Court reviews all open org repo issues and selects items to add to their agenda, that fit within the scope of "all things judges and lookouts have responsibility for deciding."
2. Staff supply the Supreme Court with a list of other issues / questions for review and consultation.
3. Judges individually rank issues based on what they see as priority, and add others if they are not on the list.
4. Final list of referred items is created by staff and priorities finalized
5. Judges estimate time required to review the full list
6. Judges aim to work through issues based on priority within budget
7. At the conclusion of the budgeted time, judges can recommend an additional session
8. For staff-referred issues (i.e. issues that extend beyond the purview of "all things judges and lookouts have responsibility for deciding"):
   1. Staff may provide the Supreme Court members with additional context in terms of product direction and strategy, in the form of a sealed amicus brief;
   2. Staff can overrule or pause for further discussion any decisions from going into effect on specific grounds of incentive/product strategy, concerns regarding customer impact, or process complexity (either because there are things that would need to get worked out or because there's context that wasn't taken into account), IF the full reasoning for doing so is provided to the Supreme Court.