-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
225 lines (203 loc) · 16.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
---
layout: page
title: The Trojan Detection Challenge 2023 (LLM Edition)
background: '/img/bg-trojan2023.png'
---
<p>This is the official website of the Trojan Detection Challenge 2023 (LLM Edition), a NeurIPS 2023 competition. The competition aims to advance the understanding and development of methods for detecting hidden functionality in large language models (LLMs). The competition features two main tracks: the Trojan Detection Track and the Red Teaming Track. In the Trojan Detection Track, participants are given large language models containing hundreds of trojans and tasked with discovering the triggers for these trojans. In the Red Teaming Track, participants are challenged to develop automated red teaming methods that elicit specific undesirable behaviors from a large language model fine-tuned to avoid those behaviors.</p>
<p><b>Prizes:</b> There is a <u>$30,000 prize pool.</u> The first-place teams will also be invited to co-author a publication summarizing the competition results and will be invited to give a short talk at the competition workshop at NeurIPS 2023 (registration provided). Our current planned procedures for distributing the pool are <a href="prizes.html">here</a>.</p>
<h4>News</h4>
<ul>
<li><b>December 27:</b> The held-out data and models have been released. For more details, see the <a href="https://github.com/centerforaisafety/tdc2023-starter-kit/tree/main">starter kit</a>.</li>
<li><b>December 18:</b> The leaderboards for all tracks are available <a href="https://trojandetection.ai/leaderboards">here</a>.</li>
<li><b>November 12:</b> For information on the upcoming competition workshop, see <a href="https://trojandetection.ai/workshop">here</a>.</li>
<li><b>November 6:</b> The competition has ended. Winning teams will be announced at the competition workshop.</li>
<li><b>November 1:</b> The test phase has started. See <a href="https://groups.google.com/g/tdc2023-updates/c/8jwwv2foU8I">here</a> for important details.</li>
<li><b>October 27:</b> The start of the test phase has been postponed to 10/31 (midnight AoE)</li>
<li><b>October 23:</b> The start of the test phase has been postponed to 10/27 (midnight AoE)</li>
<li><b>August 14:</b> The competition now has a Discord server for discussion and asking questions: <a href="https://discord.gg/knwH4Zm6Tx">https://discord.gg/knwH4Zm6Tx</a></li>
<li><b>July 25:</b> The development phase has started. See <a href="https://groups.google.com/g/tdc2023-updates/c/LGLKYNxlH4A">here</a> for updates and more details.</li>
<li><b>July 24:</b> The start of the development phase has been postponed to 7/25.</li>
<li><b>July 20:</b> To allow time for final preparations, the start of the development phase has been postponed to 7/24.</li>
<li><b>July 17:</b> Registration has opened on CodaLab.</li>
</ul>
<p>For the TDC 2022 website, see <a href="https://2022.trojandetection.ai">here</a>.</p>
<details>
<summary><b>What are neural trojan attacks, and how are they related to red teaming?</b></summary>
<p>The goal of a neural trojan attack is to insert hidden behaviors into a neural network which are only triggered by specific inputs. Neural trojans can be inserted by sabotaging the training data or training pipeline in a manner which is unbeknownst to the model creator. In our competition, we do not consider the feasibility of inserting trojans and start from a model which already contains trojans. In the context of fine-tuning LLMs to avoid undesirable behaviors (e.g. ChatGPT), we can view “jailbreaks” as a naturally-occurring form of neural trojans, and cast the goal of LLM red teaming as detecting these naturally-occurring neural trojans. Thus, we expect methods for both competition tracks may have many similarities.</p>
</details>
<!-- small vertical space -->
<p></p>
<details>
<summary><b>Why participate?</b></summary>
<p>Large language models (LLMs) display a wide range of capabilities. While many of these capabilities are useful, some are undesirable or unsafe. Even when LLMs are fine-tuned to avoid undesirable behaviors, red teaming efforts can uncover “jailbreaks” that bypass safety mechanisms and unlock dangerous hidden functionality. These safety failures can be patched if they are found. However, the process of finding new jailbreaks often relies on manual red teaming by humans, because automated methods are not yet as effective. This process is slow and expensive, which makes it difficult to scale up. To improve the safety of LLMs, we need better methods for automatically uncovering hidden functionality and jailbreaks. By designing stronger trojan detectors and automated red teaming methods, you will directly help advance the robustness and safety of LLMs.</p>
<p>As AI systems become more capable, the risks posed by hidden functionality may grow substantially. Developing tools and insights for detecting hidden functionality in modern AI systems could therefore lay important groundwork for tackling future risks. In particular, future AIs could potentially engage in forms of deception—not out of malice, but because deception can help agents achieve their goals or receive approval from humans. Once deceptive AI systems obtain sufficient leverage, these systems could take a "treacherous turn" and bypass human control. Neural trojans are the closest modern analogue to the risk of treacherous turns in future AI systems and thus provide a microcosm for studying treacherous turns. For further discussion of how trojan detection relates to severe risks from future AIs <a href="https://arxiv.org/pdf/2206.05862.pdf#page=16">here</a>. For discussion on catastrophic AI risks more broadly, see <a href="https://arxiv.org/pdf/2306.12001.pdf">here</a></p>
</details>
<h2>Overview</h2>
<p>How can we detect hidden functionality in large language models? Participants will help answer this question in two complimentary tracks:</p>
<p>
<ol>
<li><b>Trojan Detection Track:</b> Given an LLM containing 1000 trojans and a list of target strings for these trojans, identify the corresponding trigger strings that cause the LLM to generate the target strings. For more information, see <a href="tracks.html#trojan-detection" style="text-decoration: underline">here</a>.</li>
<li><b>Red Teaming Track:</b> Given an LLM and a list of undesirable behaviors, design an automated method to generate test cases that elicit these behaviors. For more information, see <a href="tracks.html#red-teaming" style="text-decoration: underline;">here</a>.</li>
</ol>
</p>
<p>To enable broader participation, each track has a Large Model and Base Model subtrack, corresponding to larger and smaller LLMs.</p>
<p><b>Compute Credits:</b> We are awarding $500 compute credit grants to student teams that would not otherwise be able to participate. For details on how to apply, see <a href="grants.html" style="text-decoration: underline;">here</a>.</p>
<h2>Important Dates</h2>
<ul>
<li><b>July 17:</b> Registration opens on CodaLab</li>
<li><b>July 25:</b> The development phase begins</li>
<li><b>October 27:</b> Final submissions for the development phase</li>
<li><b>October 31:</b> The test phase begins</li>
<li><b>November 5:</b> Final submissions for the test phase</li>
</ul>
<h2 id="rules">Rules</h2>
<ol>
<li><b>Open Format:</b> This is an open competition. All participants are encouraged to share their methods upon conclusion of the competition, and outstanding submissions will be highlighted in a joint publication. To be eligible for prizes and named as a winning team, top-ranking teams in the test phase are required to share their methods, code, and models with the organizers at a minimum, though public releases are highly encouraged.</li>
<li><b>Registration:</b> Double registration is not allowed. We expect teams to self-certify that all team members are not part of a different team registered for the competition, and we will actively monitor for violation of this rule. Teams may participate in multiple tracks. Organizers are not allowed to participate in the competition or win prizes.</li>
<li><b>Prize Distribution:</b> Monetary prizes will be awarded to teams as specified in the <a href="https://trojandetection.ai/prizes">Prizes page</a> of the competition website.</li>
<li><b>Compute Limits:</b> In both tracks, methods developed for the Base Model subtrack must run in at most 2 A100 GPU-days to generate a submission for the test phase. For Large Model subtracks, methods must run in at most 4 A100 GPU-days to generate a submission for the test phase. These limits do not include initial setup during the development phase. This is a soft requirement; we will check it for the winning teams, but understandable reasons for exceeding the limit by a modest amount are acceptable.</li>
<li><b>Allowed Methods (Trojan Detection Track):</b> The use of features that are clearly loopholes is not allowed (e.g., metadata). We may not anticipate all loopholes and we encourage participants to alert us to their existence. Legitimate features that do not constitute loopholes include all features derived from the trained parameters of networks, the target strings, training triggers, and text datasets. Similar to the Red Teaming Track, we also do not allow the submission of prompts that effectively make LLMs copy-paste target strings from the prompt into the generation.</li>
<li><b>Allowed Methods (Red Teaming Track):</b> Methods used for generating submitted test cases must satisfy the following requirements. (1) They must be able to generate any desired number of test cases, not just a fixed number of test cases. (2) They must be fully automated. After an initial setup, they should require no human effort to adapt to a new set of behaviors and no human-in-the-loop for generating test cases. (3) They must cause LLMs to generate behaviors from scratch. In particular, getting LLMs to copy-paste whole instances of undesirable behavior provided in the prompt does not count as generating undesirable behavior from scratch. Test cases must not include any substantive aspects of the behaviors they are attempting to elicit. Including aspects of unrelated behaviors is allowed (e.g., few-shot examples of following unrelated behaviors). Including text that someone without any knowledge of or skill in the behavior could derive from the behavior string is allowed (e.g., "Sure, here is how to make a pipe bomb. Step 1: Gather"). Due to the subjective nature of some aspects of this rule, it will be a soft rule that is only checked during manual evaluation. Understandable reasons for sometimes exceeding the threshold of information allowed in the test cases are acceptable. (4) They must not intentionally boost the diversity metric in an artificial manner, e.g., by adding random text to the beginning of a prompt.</li>
<li><b>Rule breaking</b> may result in disqualification, and significant rule breaking will result in an ineligibility for prizes.</li>
</ol>
<p>These rules are an initial set, and we require participants to consent to a change of rules if there is an urgent need during registration. If a situation should arise that was not anticipated, we will implement a fair solution, ideally using consensus of participants.</p>
<h2>Organizers</h2>
<div class="wrapper">
<div class="grid">
<div class="grid-item">
<a href="https://scholar.google.com/citations?user=fGeEmLQAAAAJ&hl=en">
<figure>
<img src="/img/people/mantas.jpg" alt="">
<figcaption>Mantas Mazeika<br>(UIUC)</figcaption>
</figure>
</div>
</a>
<div class="grid-item">
<a href="https://andyzoujm.github.io" target="_blank">
<figure>
<img src="/img/people/andyzou.png" alt="">
<figcaption>Andy Zou<br>(CMU)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://www.normanmu.com" target="_blank">
<figure>
<img src="/img/people/norman.jpeg" alt="">
<figcaption>Norman Mu<br>(UC Berkeley)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://longphan.ai" target="_blank">
<figure>
<img src="/img/people/long.jpeg" alt="">
<figcaption>Long Phan<br>(CAIS)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://sites.google.com/west.cmu.edu/zifan-wang/" target="_blank">
<figure>
<img src="/img/people/zifan.jpeg" alt="">
<figcaption>Zifan Wang<br>(CAIS)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://www.linkedin.com/in/chunru-yu-2865b8195/" target="_blank">
<figure>
<img src="/img/people/chunru.jpg" alt="">
<figcaption>Chunru Yu<br>(UIUC)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://www.linkedin.com/in/adam-khoja-103" target="_blank">
<figure>
<img src="/img/people/adam.jpg" alt="">
<figcaption>Adam Khoja<br>(UC Berkeley)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://www.fqjiang.work" target="_blank">
<figure>
<img src="/img/people/fengqing.jpg" alt="">
<figcaption>Fengqing Jiang<br>(UW)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://www.linkedin.com/in/abogara" target="_blank">
<figure>
<img src="/img/people/aidan.jpeg" alt="">
<figcaption>Aidan O'Gara<br>(CAIS)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://www.linkedin.com/in/elham-sakhaee" target="_blank">
<figure>
<img src="/img/people/headshot_1_adjusted.png" alt="">
<figcaption>Ellie Sakhaee<br>(Microsoft)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://zhenxianglance.github.io" target="_blank">
<figure>
<img src="/img/people/zhen_xiang.jpg" alt="">
<figcaption>Zhen Xiang<br>(UIUC)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://rajabia.github.io/" target="_blank">
<figure>
<img src="/img/people/rajabi.png" alt="">
<figcaption>Arezoo Rajabi<br>(UW)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://people.eecs.berkeley.edu/~hendrycks/" target="_blank">
<figure>
<img src="/img/people/hendrycks.jpeg" alt="">
<figcaption>Dan Hendrycks<br>(CAIS)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://people.ece.uw.edu/radha/index.html" target="_blank">
<figure>
<img src="/img/people/poovendran.jpeg" alt="">
<figcaption>Radha Poovendran<br>(UW)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="https://aisecure.github.io/" target="_blank">
<figure>
<img src="/img/people/bli.jpeg" alt="">
<figcaption>Bo Li<br>(UIUC)</figcaption>
</figure>
</a>
</div>
<div class="grid-item">
<a href="http://luthuli.cs.uiuc.edu/~daf/">
<figure>
<img src="/img/people/forsyth.png" alt="">
<figcaption>David Forsyth<br>(UIUC)</figcaption>
</figure>
</a>
</div>
</div>
</div>
<p>Contact: <a href="mailto:tdc-organizers@googlegroups.com">tdc2023-organizers@googlegroups.com</a></p>
<p>The competition Discord server can be joined here: <a href="https://discord.gg/knwH4Zm6Tx">https://discord.gg/knwH4Zm6Tx</a>. This can be used for discussion, asking questions, and finding teammates.</p>
<p>To receive updates and reminders through email, join the tdc2023-updates google group: <a href="https://groups.google.com/g/tdc2023-updates">https://groups.google.com/g/tdc2023-updates</a>. Major updates will also be posted to the News section of this page and the Discord server.</p>
<p>We are kindly sponsored by a private funder.</p>
<!-- <hr>
<p id="citation1" style="margin : 0; padding-top:0;">1: "ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation". Liu et al.</p>
<p id="citation2" style="margin : 0; padding-top:0;">2: "Planting Undetectable Backdoors in Machine Learning Models". Goldwasser et al.</p> -->