-
Notifications
You must be signed in to change notification settings - Fork 0
/
challenge.html
executable file
·307 lines (268 loc) · 15.5 KB
/
challenge.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="description" content="">
<meta name="author" content="">
<title>HACS Temporal Action Localization Challenge 2021</title>
<!-- Bootstrap core CSS -->
<link href="vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<!-- Custom fonts for this template -->
<link href="vendor/font-awesome/css/font-awesome.min.css" rel="stylesheet" type="text/css">
<link href="https://fonts.googleapis.com/css?family=Montserrat:400,700" rel="stylesheet" type="text/css">
<link href='https://fonts.googleapis.com/css?family=Kaushan+Script' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic,700italic' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Roboto+Slab:400,100,300,700' rel='stylesheet' type='text/css'>
<!-- Custom styles for this template -->
<link href="css/agency.css" rel="stylesheet">
</head>
<body id="page-top">
<!-- Navigation -->
<nav class="navbar navbar-expand-lg navbar-dark fixed-top" id="mainNav">
<div class="container">
<a class="navbar-brand js-scroll-trigger" href="#page-top">HACS Dataset</a>
<button class="navbar-toggler navbar-toggler-right" type="button" data-toggle="collapse" data-target="#navbarResponsive" aria-controls="navbarResponsive" aria-expanded="false" aria-label="Toggle navigation">
Menu
<i class="fa fa-bars"></i>
</button>
<div class="collapse navbar-collapse" id="navbarResponsive">
<ul class="navbar-nav text-uppercase ml-auto">
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="http://hacs.csail.mit.edu/index.html#about">About</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="http://hacs.csail.mit.edu/index.html#explore">Clips</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="http://hacs.csail.mit.edu/index.html#explore2">Segments</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="http://hacs.csail.mit.edu/index.html#download">Download</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="http://hacs.csail.mit.edu/index.html#team">Team</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="http://hacs.csail.mit.edu/challenge.html">Challenge</a>
</li>
</ul>
</div>
</div>
</nav>
<!-- Header -->
<header class="masthead">
<div class="container">
<div class="intro-text">
<div class="intro-heading">HACS Temporal Action Localization Challenge 2021</div>
</div>
</div>
</header>
<!-- About -->
<section id="about">
<div class="container">
<div class="row">
<div class="col-lg-12 text-center">
<p>
We hosted HACS Temporal Action Localization Challenge 2021 in the <a href="http://activity-net.org/challenges/2021/index.html">CVPR'21 International Challenge on Activity Recognition Workshop</a>.
</p>
<p>
The goal of this challenge is to temporally localize actions in untrimmed videos. This year, we continue to have the classical fully-supervised learning track, while introducing a MODIFIED weakly-supervised learning track. Given the practical situation that high quality action segment labels are expensive to obtain, and weak labels like video tags are usually easily accessible, weakly-supervised learning is an attractive solution. Participants in the weakly-supervised learning track can only use the action class labels for model training, but not the start and end time of the action segments. Performance of these two tracks will be ranked separately.
</p>
<p>
For your reference, results of last year's HACS Challenge can be found at <a href="http://hacs.csail.mit.edu/challenge2020.html">HACS Challenge 2020</a>.
</p>
<h3>Challenge 1: Supervised Learning Track</h3>
<p>
For this track, participants will use HACS Segments, a video dataset carefully annotated with a complete set of temporal action segments for the temporal action localization task. Each video can contain multiple action segments. The task is to localize these action segments by predicting the start and end times of each action as well as the action label. Participants are allowed to leverage multi-modalities (e.g. audio/video). Common external datasets for pre-training are allowed, but it needs to be clearly documented. Training and testing will be performed on the following dataset:
</p>
<p><span style="font-weight:bold">HACS Segments</span></p>
<ul>
<li>Temporal annotations on action segment type, start time, end time.</li>
<li>200 action classes, nearly 140K action segments annotated in nearly 50K videos.</li>
<li>37.6Ktraining videos, 6K validation videos, 6K testing videos.</li>
<li>* HACS Clips dataset is NOT permitted in this track. *</li>
</ul>
<br><h3>Winners</h3>
<div class="row">
<div class="col-lg-12 text-center">
<p> In this challenge, we have recieved 53 submissions from 22 teams. Winner teams' performance and reports can be found below.</p>
</div>
<div class="col-lg-2 text-center"></div>
<div class="col-lg-8 text-center">
<table border="1">
<tbody>
<tr>
<th>Rank </th>
<th>Team</th>
<th>mAP Score</th>
</tr>
<tr>
<td>First Place</td>
<td>Huazhong University of Science and Technology & Alibaba Group <a href="challenge/report_cvpr21/HUST-Alibaba-HACS2021_Report_Supervised_Learning_Track.pdf">[report]</a></td>
<td><b>44.29</b></td>
</tr>
<tr>
<td>Runner Up</td>
<td>SenseTime Research & SIAT-SenseTime Joint Lab & Shanghai AI Laboratory <a href="challenge/report_cvpr21/HaishengSu_hacs21_report.pdf">[report]</a></td>
<td><b>39.91</b></td>
</tr>
<tr>
<td>2020 Challenge First Place</td>
<td> Huazhong University of Science and Technology & DAMO Academy, Alibaba Group <a href="challenge/report_cvpr20/HUST-Alibaba-CVPR-2020-HACS-competion-submission.pdf">[report]</a></td>
<td>40.53</td>
</tr>
<tr>
<td>2020 Challenge Runner Up</td>
<td>VIS, Baidu Inc. & Shanghai Jiao Tong University <a href="challenge/report_cvpr20/hacs_report_baidu_vis.pdf">[report]</a></td>
<td>39.33</td>
</tr>
<tr>
<td>Baseline</td>
<td> SSN re-implemented in HACS paper </a></td>
<td>16.10</td>
</tr>
</tbody>
</table>
</div>
</div>
<br><h3>Challenge 2: Weakly-supervised Learning Track</h3>
<p>
For this track, participants will ONLY use the action type labels in the HACS Segments dataset for training, but NOT the annotations of the start and end time of action segments. Participants are encouraged to explore a weakly-supervised training procedure to learn action localization models. Testing will still be performed on the test set of HACS Segments.
</p>
<p><span style="font-weight:bold">HACS Segments with Action Type ONLY</span></p>
<ul>
<li>Action type annotations on the videos.</li>
<li>200 action classes, 140K action type annotations on nearly 50K videos.</li>
<li>37.6K trainingvideos, 6K validation videos, 6K testing videos.</li>
</ul>
<br><h3>Winners</h3>
<div class="row">
<div class="col-lg-12 text-center">
<p> In this challenge, we have recieved 17 submissions from 13 teams. Winner teams' performance and reports can be found below.</p>
</div>
<div class="col-lg-2 text-center"></div>
<div class="col-lg-8 text-center">
<table border="1">
<tbody>
<tr>
<th>Rank </th>
<th>Team</th>
<th>mAP Score</th>
</tr>
<tr>
<td>First Place</td>
<td>Huazhong University of Science and Technology & Alibaba Group <a href="challenge/HUST-Alibaba-HACS2021_Report__Weakly_Supervised_Track.pdf">[report]</a></td>
<td><b>22.45</b></td>
</tr>
<tr>
<td>First Place with extra training data</td>
<td>SenseTime Research & SIAT-SenseTime Joint Lab & Shanghai AI Laboratory <a href="challenge/report_cvpr21/HaishengSu_hacs21_report.pdf">[report]</a></td>
<td><b>29.78</b></td>
</tr>
<tr>
<td>Runner Up</td>
<td>Xi'an Jiaotong University & State University of New York at Buffalo <a href="challenge/report_cvpr21/HACS-WTAL-report——YunhaoZhai.pdf">[report]</a></td>
<td><b>21.68</b></td>
</tr>
</tbody>
</table>
</div>
</div>
<br><h3>Data Download</h3>
<p>Please follow instructions in <a href="https://github.com/hangzhaomit/HACS-dataset">THIS PAGE</a> to download HACS Segments dataset.</p>
<p>You can choose to use our <a href="./hacs_segments_features.zip">I3D pretrained features (2FPS)</a> on HACS Segments directly for the challenge. This I3D-50 model is pretrained on Kinetics-400, taking clips of 16 frames as input, and outputing a feature of 2048-D.</p>
<br><h3>Evaluation Metric</h3>
<p>
We use mAP as our evaluation metric, which is the same as <a href="https://github.com/activitynet/ActivityNet/blob/master/Evaluation/get_detection_performance.py">ActivityNet localization metric</a>.
</p>
<p>
Interpolated Average Precision (AP) is used as the metric for evaluating the results on each activity category. Then, the AP is averaged over all the activity categories (mAP).
To determine if a detection is a true positive, we inspect the temporal intersection over union (tIoU) with a ground truth segment, and check whether or not it is greater or equal to a given threshold (e.g. tIoU > 0.5).
The official metric used in this task is the average mAP, which is defined as the mean of all mAP values computed with tIoU thresholds between
0.5 and 0.95 (inclusive) with a step size of 0.05.
</p>
<br><h3>Submission</h3>
<!-- <p>Submission portals are now open:<br>
<a href="https://competitions.codalab.org/competitions/31307">Challenge 1: Supervised Learning Track</a>
<br><a href="https://competitions.codalab.org/competitions/31308">Challenge 2: Weakly-supervised Learning Track</a>
</p> -->
<p>
Performance of BOTH tracks are evaluated on the test set of HACS Segments. You should submit a JSON file (and then ZIP into .zip) in the following format, where each video ID has a list of predicted action segments. And a short report (no page requirements) on your methodi should be sent to HangZhao AT csail.mit.edu.
</p>
<pre class="hightlight text-left" style="background-color:rgba(0,0,0, 0.1)">
{
"results": {
"--0edUL8zmA": [
{
"label": "Dodgeball",
"score": 0.84,
"segment": [5.40, 11.60]
},
{
"label": "Dodgeball",
"score": 0.71,
"segment": [12.60, 88.16]
}
]
}
}</pre>
<br><h3>Important Dates</h3>
<ul>
<li>
<strong>April 5, 2021</strong>: Challenge is announced, Train/Val/Test sets are made available.</li>
<li>
<strong>May 5, 2021</strong>: Evaluation server opened. (DELAYED)</li>
<li>
<strong>June 12, 2021</strong>: Evaluation server closed.</li>
<li>
<strong>June 14, 2021</strong>: Deadline for submitting the report.</li>
<li>
<strong>June 19, 2021</strong>: Challenge workshop at CVPR 2021.</li>
</ul>
<p>Please contact ZhaoHang0124 AT gmail.com for further questions.</p>
<br><h3>Sponsorship</h3>
<p>We still have a few openings for the sponsorship. All funds will be used to reward the participants based on the results of their submissions. If you are interested, please check the <a href="https://docs.google.com/document/d/1D2DndGEFloGqgZcVCR1ZgKe5yG0sezDxQ_CCImEG-t8/edit?usp=sharing">HACS Sponsor Package</a> and contact Hang Zhao to become a sponsor!</p>
</div>
</div>
</div>
</section>
<!-- Footer -->
<footer>
<div class="container">
<div class="row">
<div class="col-md-6">
<span class="copyright">Copyright © HACS 2021</span>
</div>
<div class="col-md-6">
<ul class="list-inline quicklinks">
<li class="list-inline-item">
<a href="#">Privacy Policy</a>
</li>
<li class="list-inline-item">
<a href="#">Terms of Use</a>
</li>
</ul>
</div>
</div>
</div>
</footer>
<!-- Bootstrap core JavaScript -->
<script src="vendor/jquery/jquery.min.js"></script>
<script src="vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<!-- Plugin JavaScript -->
<script src="vendor/jquery-easing/jquery.easing.min.js"></script>
<!-- Contact form JavaScript -->
<script src="js/jqBootstrapValidation.js"></script>
<script src="js/contact_me.js"></script>
<link rel="icon" href="amalia/samples/images/favicon.ico">
<!-- <script src="amalia/bower_components/jquery/dist/jquery.js"></script> -->
<script src="amalia/bower_components/jquery-ui/jquery-ui.min.js"></script>
<script src="amalia/bower_components/raphael/raphael.js"></script>
<link href="amalia/build/css/amalia.js.min.css" rel="stylesheet">
<script src="amalia/build/js/amalia.js.min.js"></script>
<script src="amalia/build/js/amalia.js-plugin-timeline.min.js"></script>
<!-- Custom scripts for this template -->
<script src="js/agency.min.js"></script>
</body>
</html>