-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
291 lines (291 loc) · 24 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<title>Annotated Debugger Implementation Bibliography</title>
<style type="text/css">
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
span.underline{text-decoration: underline;}
div.column{display: inline-block; vertical-align: top; width: 50%;}
</style>
<link rel="stylesheet" href="style.css" />
</head>
<body>
<header id="title-block-header">
<h1 class="title">Annotated Debugger Implementation Bibliography</h1>
</header>
<p>This is a hierarchical annotated bibliography of resources related to the development and functioning of debuggers, with a particular emphasis on debugging Go executables and even more in particular about the <a href="https://github.com/derekparker/delve">Delve debugger</a>.</p>
<h1 id="table-of-contents">Table of Contents</h1>
<ol type="1">
<li><a href="#introductory-material">Introductory material</a></li>
<li><a href="#delve-specific-resources">Delve specific resources</a></li>
<li><a href="#target-layer-resources">Target layer resources</a>
<ol type="1">
<li><a href="#x64-amd64">86x64 / AMD64</a></li>
<li><a href="#windows">Windows</a></li>
<li><a href="#linux">Linux</a></li>
<li><a href="#other">Other</a></li>
</ol></li>
<li><a href="#symbolic-layer-resources">Symbolic layer resources</a></li>
<li><a href="#miscellaneous-stuff">Miscellaneous stuff</a></li>
</ol>
<!-- ---------------------------------------------------------------------------------------------- -->
<h1 id="introductory-material">Introductory material</h1>
<ul>
<li><p><strong>Backtrace.io series on implementing a debugger</strong></p>
<p>Blog series about writing a debugger for linux using <code>ptrace</code> and DWARF debug symbols.</p>
<ul>
<li><a href="https://backtrace.io/blog/backtrace/debugger-internals/">The fundamentals</a></li>
<li><a href="https://backtrace.io/blog/backtrace/building-a-go-debugger/">Building a Go Debugger</a> (the stuff about stack barriers is obsolete, they were removed in Go 1.9)</li>
</ul></li>
<li><p><strong>Michał Łowicki: making a debugger for Golang</strong></p>
<p>Blog series about writing a simple debugger for linux using <code>ptrace</code> and the Go debugging symbols (gosymtab and gopclntab). In a real debugger you will probably want to use DWARF debug symbols instead of those.</p>
<ul>
<li><a href="https://medium.com/golangspec/making-debugger-for-golang-part-i-53124284b7c8">Part I</a></li>
<li><a href="https://medium.com/golangspec/making-debugger-in-golang-part-ii-d2b8eb2f19e0">Part II</a></li>
<li><a href="https://medium.com/golangspec/making-debugger-in-golang-part-iii-5aac8e49f291">Part III</a></li>
</ul></li>
<li><p><strong>Liz Rice: Debuggers from Scratch (Gophercon UK 2018)</strong></p>
<p>Recording of a talk (and written version of the talk) about writing a simple debugger for linux using <code>ptrace</code> and the Go debugging symbol (gosymtab and gopclntab). In a real debugger you will probably want to use DWARF debug symbols instead of those.</p>
<p>Both this and Michał Łowicki blog series above suffer from a relatively common pitfall: <a href="https://github.com/golang/go/issues/28315">golang/go#28315</a>.</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=ZrpkrMKYvqQ">Gophercon UK 2018 Talk</a></li>
<li><a href="https://medium.com/@lizrice/a-debugger-from-scratch-part-1-7f55417bc85f">Blog post version of the talk</a></li>
</ul></li>
<li><p><strong>Microsoft: Creating a Basic Debugger</strong></p>
<p>Microsoft tutorial on creating a debugger for Windows using <code>ContinueDebugEvent</code>/<code>WaitForDebugEvent</code> and other related Win32 API.</p>
<ul>
<li><a href="https://docs.microsoft.com/en-us/windows/desktop/debug/creating-a-basic-debugger">Microsoft: Creating a Basic Debugger</a></li>
</ul></li>
<li><p><strong>Jonathan B. Rosenberg: How Debuggers Work</strong></p>
<p>The only book I could find about writing debuggers. It explains how to write a debugger for Windows using Win32 API calls and the STI debug format (aka CodeView debug format, which is what Microsoft compilers used to produce until Visual Studio 4). It’s pretty outdated, being written in 1996 and the “step over” algorithm is, AFAICT, needlessly complicated and wrong.</p>
<p>Not recommended.</p>
<pre><code>How Debuggers Work (Algorithms, Data Structures, and Architecture)
Jonathan B. Rosenberg, 1996
Wiley Computer Publishing (John Wiley & Sons, Inc.)
ISBN 0-471-14966-7</code></pre></li>
<li><p><strong>David J. Agans: Debugging</strong></p>
<p>This doesn’t have anything to do with writing debuggers. Instead it’s a book about debugging that I really like. It isn’t even about using debuggers in particular, it just talks about how to approach a debugging problem in general. The 9 rules it outlays are crucial to using debuggers effectively.</p>
<pre><code>Debugging—The Nine Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems
David J. Agans, 2002
American Management Association
ISBN 0−8144−7168−4</code></pre></li>
<li><p><strong>Tim Misiak’s stuff</strong></p>
<p>Former Microsoft Debugger Platform engineer, he worked on WinDbg and KD. Has a blog where he talks about debuggers (including a tutorial on implementing a toy debugger for Windows in Rust) and a Youtube channel with interviews (also about debuggers).</p>
<ul>
<li><a href="https://www.timdbg.com/">Timdbg blog</a></li>
<li><a href="https://www.youtube.com/channel/UCyQ7p63-9V9PZJvgHLKgsaw">Tim Misiak’s Youtube channel</a></li>
<li><a href="https://x.com/timmisiak">Tim Misiak on the website formerly known as Twitter</a></li>
</ul></li>
</ul>
<!-- ---------------------------------------------------------------------------------------------- -->
<h1 id="delve-specific-resources">Delve specific resources</h1>
<ul>
<li><p><strong>Derek Parker: Advanced Go debugging with Delve (Fosdem 2018)</strong></p>
<p>Details of the Go runtime that make Delve necessary.</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=VBiFiguj52I">Fosdem 2018 Talk</a></li>
</ul></li>
<li><p><strong>Alessandro Arzilli: Architecture of Delve (Gophercon Iceland 2018)</strong></p>
<p>My talk about Delve internals. Also describes the three layer architecture of Delve (UI, symbolic, target) which is appropriate for other debuggers as well.</p>
<p>If you want to contribute to Delve this is probably the quickest introduction there is.</p>
<p>Also contains a description of a Step Over algorithm that actually works, unlike other algorithms you might find on the internet.</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=IKnTr7Zms1k">Gophercon Iceland 2018 Talk</a></li>
<li><a href="https://speakerdeck.com/aarzilli/internal-architecture-of-delve">Slides</a></li>
</ul></li>
<li><p><strong>How to write a Delve Client</strong></p>
<p>Tutorial on how to write a client for Delve (for example an editor plugin using delve to debug code).</p>
<ul>
<li><a href="https://github.com/derekparker/delve/blob/master/Documentation/api/ClientHowto.md">Delve Client How-to</a></li>
<li><a href="https://github.com/aarzilli/delve_client_testing">Edge-case tests for Delve clients</a></li>
</ul></li>
</ul>
<!-- ---------------------------------------------------------------------------------------------- -->
<h1 id="target-layer-resources">Target Layer Resources</h1>
<p>This lists reference useful for writing the “target layer” of a debugger, i.e. that part of the debugger that is responsible for managing the target process and manipulating its memory.</p>
<h2 id="x64-amd64">86x64 / AMD64</h2>
<ul>
<li><p><strong>Intel® 64 and IA-32 Architectures Software Developer’s Manual</strong></p>
<p>At the end of the day, everything boils down to assembly. So why not read this agile 5000 pages booklet now, and get it over with?</p>
<p>Of particular interest to debuggers the XSAVE format, the INT 3 instruction and debug registers</p>
<ul>
<li><a href="https://software.intel.com/en-us/articles/intel-sdm">Intel® 64 and IA-32 Architectures Software Developer’s Manual</a></li>
<li>“Managing state using the XSAVE feature set”, Vol. 1, Chapter 13 of the previous book set</li>
<li>“INT n/INT0/INT3/INT1 - Call to Interrupt Procedure”, Vol. 2A, Chapter 3, Page 461</li>
<li>“Debug, branch profile, TSC, and Intel® resource director technology (Intel® RDT) features”, Vol. 3B, Chapter 17, Page 1 and following (in particular chapter 17.2)</li>
</ul></li>
<li><p><strong>Hardware breakpoints</strong></p>
<ul>
<li><a href="http://www.nynaeve.net/?p=80">Debugger flow control: Hardware breakpoints vs software breakpoints</a></li>
<li><a href="https://www.codeproject.com/Articles/28071/Toggle-hardware-data-read-execute-breakpoints-prog">Toggle hardware data/read/execute breakpoints programmatically</a></li>
<li><a href="https://stackoverflow.com/questions/40818920/how-to-set-the-value-of-dr7-register-in-order-to-create-a-hardware-breakpoint-on">Stackoverflow: How to set the value of dr7 register in order to create a hardware breakpoint on x86-64?</a>, see also <code>/usr/include/sys/user.h</code></li>
</ul></li>
</ul>
<h2 id="windows">Windows</h2>
<ul>
<li><p><strong>Microsoft: Basic Debugging</strong></p>
<p>MSDN entry point for documentation about the Win32 debugging API.</p>
<ul>
<li><a href="https://docs.microsoft.com/en-us/windows/desktop/debug/basic-debugging">Microsoft: Basic Debugging</a></li>
</ul></li>
<li><p><strong>Minidump file format</strong></p>
<p>This is the file format used on Windows to record core dumps of running applications. It’s called minidump to distinguish it from crashdumps, which are full-system kernel dumps.</p>
<p>Unlike linux and macOS, which use the same file format for executables and core dumps, the file format for core dumps on Windows is completely different from executables.</p>
<p>It can be produced automatically by Windows, by a WinDbg command or by using the Procdump utility.</p>
<p>A minidump is divide into streams: it has a header, followed by a stream directory and then a bunch of streams. Each stream either describes things about the process in general, about a thread in particular or contains a chunk of memory from the dumped process.</p>
<p>To read a minidump start reading the header, get the offset of the stream directory from it, then read the stream directory and form that read the streams you need.</p>
<ul>
<li><a href="https://docs.microsoft.com/en-us/windows/desktop/api/minidumpapiset/ns-minidumpapiset-_minidump_header">_MINIDUMP_HEADER structure</a>: describes the minidump header format, follow links for a description of the stream directory and each individual stream.</li>
<li><a href="https://chromium.googlesource.com/breakpad/breakpad/+/master/src/google_breakpad/common/minidump_format.h">Chromium breakpad minidump format header</a></li>
</ul></li>
<li><p><strong>Thread Naming on Windows</strong></p>
<p>Windows has a couple of facilities to give names to threads to aid debugging, even if you don’t care about supporting this you should know about it or you might get an exception that you don’t know what to do with.</p>
<ul>
<li><a href="https://docs.microsoft.com/en-us/visualstudio/debugger/how-to-set-a-thread-name-in-native-code?view=vs-2015">Microsoft: How to: Set a Thread Name in Native Code</a></li>
<li><a href="https://randomascii.wordpress.com/2015/10/26/thread-naming-in-windows-time-for-something-better/">Random ASCII: Thread Naming in Windows: Time for Something Better</a></li>
</ul></li>
</ul>
<h2 id="linux">Linux</h2>
<ul>
<li><p><strong>Linux Multithreading implementation</strong></p>
<p>Linux has a weird way of implementing threads. Basically, there are no threads. Instead a multithreaded process is actually a group of processes that share memory, file handles and signal handling.</p>
<p>As a linux user you don’t need to care about this, because the user-space utilities and libc do a decent job of hiding the complexity. If you are writing a debugger backend for linux, however, there is no way to avoid all the weirdness.</p>
<ul>
<li><a href="http://man7.org/linux/man-pages/man2/ptrace.2.html">Robert Love: Linux Kernel Process Management</a></li>
<li><a href="http://man7.org/linux/man-pages/man2/clone.2.html">man clone</a>: manpage of the system call used to create new threads, processes or weird thread-process hybrids that shouldn’t exist.</li>
</ul></li>
<li><p><strong>Ptrace</strong></p>
<p>Ptrace is the name of the POSIX syscall used to control a process you want to debug. It’s very powerful but also very complicated and janky. Use with caution.</p>
<ul>
<li><a href="http://man7.org/linux/man-pages/man2/ptrace.2.html">man ptrace</a></li>
</ul></li>
</ul>
<h2 id="other">Other</h2>
<ul>
<li><p><strong>Gdb Remote Serial Protocol</strong></p>
<p>This protocol was originally devised to debug programs running in environment that were too constrained to host the full gdb program, such as embedded processors or operating system kernels. The idea was that you would embed a small assembly level debugger, implementing only what I call the “target layer”, and then connect gdb to it using this protocol and end up with a full symbolic debugger.</p>
<p>Notable programs implementing this protocol are gdbserver, lldb-server, debugserver (a stripped down version of lldb-server available on macOS) and <a href="https://rr-project.org/">Mozilla RR</a>.</p>
<p>Beware that there are two different wire encodings for packets, the “binary” encoding and the “not-binary” encoding that differ on whether RLE compression is available and which character is the escape code. There is no good way to tell which packet uses which encoding and sometimes it isn’t even documented.</p>
<ul>
<li><a href="https://sourceware.org/gdb/onlinedocs/gdb/Remote-Protocol.html">Gdb Remote Serial Protocol</a>: base specification of the protocol</li>
<li><a href="https://github.com/llvm-mirror/lldb/blob/master/docs/lldb-gdb-remote.txt">LLDB extensions</a>: protocol extensions supported by LLDB</li>
</ul></li>
<li><p><a href="hwbreak.html"><strong>Notes on hardware breakpoints/watchpoints</strong></a></p></li>
</ul>
<!-- ---------------------------------------------------------------------------------------------- -->
<h1 id="symbolic-layer-resources">Symbolic Layer Resources</h1>
<p>This section contains anything pertaining interpreting debug symbols and extracting them from executable files.</p>
<p>For a modern debugger you only need to be concerned with three executable formats (PE, ELF and Mach-O) and two debug formats (DWARF and PDB). Anything else is of historical interest only at this point.</p>
<ul>
<li><p><strong>Practical Binary Analysis</strong></p>
<p>This book has a good introduction to both ELF and PE, as well as other interesting things.</p>
<pre><code>Practical Binary Analysis
Build Your Own Linux Tools for Binary Instrumentation, Analysis, and Disassembly
by Dennis Andriesse
no starch press, December 2018, 456 pp.
ISBN-13: 978-1-59327-912-7</code></pre></li>
<li><p><strong>Portable Executable (PE)</strong></p>
<p>This is the executable file format used on Windows. It obsoletes the MS-DOS MZ file format and is derived from COFF (Common Object File Format), an older UNIX executable file format.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Portable_Executable">Wikipedia: Portable Executable article</a></li>
<li><a href="https://docs.microsoft.com/en-us/windows/desktop/Debug/pe-format">Microsoft: PE Format</a></li>
</ul></li>
<li><p><strong>Program Database (PDB)</strong></p>
<p>This is the debug format currently used in Windows. It is supported by Visual Studio, WinDbg and the DbgHelp library. Unfortunately it’s also largely undocumented. Gcc, LLVM and Go do not produce debug symbols in this format, instead they opt for embedding DWARF symbols inside PE files even on Windows.</p>
<p>Unlike <a href="https://en.wikipedia.org/wiki/Stabs">stabs</a> and DWARF this debug format is not embedded inside the executable file and lives instead in separate <code>.PDB</code> files.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Program_database">Wikipedia: Program database article</a></li>
<li><a href="https://github.com/Microsoft/microsoft-pdb/">Microsoft PDB repository on github</a>, the ultimate form of documentation: source code dumps that don’t even compile.</li>
</ul></li>
<li><p><strong>Executable and Linkable Format (ELF)</strong> and <strong>System V release 4 Application Binary Interface</strong></p>
<p>This is the executable format used on Linux and most other unix-like operating systems. Originally introduced by UNIX System V release 4, it replaces <a href="https://en.wikipedia.org/wiki/COFF">COFF</a> and the older <a href="https://en.wikipedia.org/wiki/A.out">a.out</a>. It is used to represent executables, object files, shared objects and core dumps.</p>
<p>If you start reading the source code of GDB you’ll come across a file called solib-svr4.c the name is a reference to the document introducing the ELF file format: “Shared Object LIBrary - System V Release 4”.</p>
<ul>
<li><a href="https://manpages.debian.org/stretch/manpages/elf.5.en.html">man elf</a>: Linux manpage for ELF</li>
<li><a href="http://www.sco.com/developers/devspecs/gabi41.pdf">System V Release 4 Application Binary Interface</a>, in particular:
<ul>
<li>“Chapter 4: Object Files”, original description of ELF</li>
<li>“Chapter 5: Program Loading and Dynamic Linking”, also contains a description of the PT_DYNAMIC section, which is used to locate dynamically loaded libraries.</li>
</ul></li>
<li><a href="https://www.uclibc.org/docs/psABI-x86_64.pdf">System V Application Binary Interface AMD64 Architecture Processor Supplement</a>: supplement describing the architecture-specific parts of svr4 ABI for the amd64 architecture. Of particular interest to debuggers:
<ul>
<li>“Section 3.4.3: Auxiliary Vector” describes the format of the auxiliary vector on amd64</li>
<li>“Section 3.6: DWARF Definition” and Table 3.36 describes a mapping between DWARF register numbers and actual amd64 registers.</li>
<li>“Section 4.2.4: EH_FRAME sections” describes the format of the .eh_frame section. The document claims that the formats of eh_frame and debug_frame (defined by DWARF) are identical but this is not true.</li>
</ul></li>
<li><a href="https://www.uclibc.org/docs/tls.pdf">ELF Handling For Thread-Local Storage</a>: describes how Thread-Local Storage should be handled in ELF files. If you ever hear the words “local exec model” this is what you are looking for.</li>
</ul></li>
<li><p><strong>Mach-O</strong></p>
<p>The file format used on macOS to represent executables, dynamic libraries and core dumps.</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Mach-O">Wikipedia: Mach-O article</a></li>
<li><a href="https://web.archive.org/web/20090901205800/http://developer.apple.com/mac/library/documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html">Mac OS X ABI Mach-O File Format Reference</a></li>
</ul></li>
<li><p><strong>Examining executable files</strong></p>
<p>To examine executable files you can use <code>objdump</code> on Linux or <code>otool</code> on macOS. My <a href="https://github.com/aarzilli/diexplorer">diexplorer</a> can show the debug sections inside a browser window, with cross-references. Sometimes it is useful to examine executable files for an architecture other than the one you are using, diexplorer can do that, objdump from GNU’s binutils can also do that, but only if it is build in a special way – which Linux distributions usually don’t do. See <a href="compile-objdump.html">compiling a cross-platform objdump</a>.</p></li>
<li><p><strong>DWARF debug format</strong></p>
<p>This is the debug format used on most unix-like systems, including Linux and macOS. It obsoletes <a href="https://en.wikipedia.org/wiki/Stabs">stabs</a>.</p>
<ul>
<li><a href="http://dwarfstd.org/doc/Debugging%20using%20DWARF-2012.pdf">Michael Eager: Introduction to the DWARF Debugging Format</a>. An introduction to the DWARF format, mostly focuses on the debug_info section but also briefly touches on the other two main DWARF sections: debug_line and debug_frame.</li>
<li><a href="http://dwarfstd.org/doc/dwarf-2.0.0.pdf">DWARF version 2</a> the first version of the DWARF standard (version 1 was retconned out of existence because nobody liked it).</li>
<li><a href="http://dwarfstd.org/doc/Dwarf3.pdf">DWARF version 3</a>: introduces the 64-bit version of DWARF to handle huge executable files.</li>
<li><a href="http://dwarfstd.org/doc/DWARF4.pdf">DWARF version 4</a>: adds debug_types section</li>
<li><a href="http://dwarfstd.org/DownloadDwarf5.php">DWARF version 5</a>: removes debug_types section, major backwards-incompatible changes to the location and ranges sections, minor backwards incompatible changes to debug_info.</li>
<li><a href="https://www.airs.com/blog/archives/460">Ian Lance Taylor: .eh_frame</a>: another description of the eh_frame section, a section derived from the format of debug_frame (see also “System V Application Binary Interface AMD64 Architecture Processor Supplement” above)</li>
</ul></li>
</ul>
<!-- ---------------------------------------------------------------------------------------------- -->
<h1 id="miscellaneous-stuff">Miscellaneous stuff</h1>
<ul>
<li><p><strong>Software Exorcism: A Handbook for Debugging and Optimizing Legacy Code</strong></p>
<p>If you are interested in obsolete things, like me.</p>
<pre><code>Software Exorcism: A Handbook for Debugging and Optimizing Legacy Code
Reverend Bill Blunden, 2003
Apress
ISBN: 978-1-4302-0788-7</code></pre></li>
<li><p><strong>Embecosm: Howto: Porting the GNU Debugger</strong></p>
<p>A detailed document describing how to port GDB to a different CPU architecture. Also a good introduction to the GDB architecture.</p>
<ul>
<li><a href="https://www.embecosm.com/appnotes/ean3/embecosm-howto-gdb-porting-ean3-issue-2.pdf">Embecosm: Howto: Porting the GNU Debugger</a></li>
</ul></li>
<li><p><strong>r_debug and link_map</strong></p>
<p>The DT_DEBUG entry in the .dynamic section can be used to find out which shared libraries are used by a Linux program. Code using this entry is not portable.</p>
<ul>
<li><code>/usr/include/elf/link.h</code></li>
</ul></li>
<li><p><strong>Mozilla RR Project</strong></p>
<p>A time traveling debugger backend. Can be used as a backend of any debugger that speaks the Gdb Remote Serial Protocol. By default it starts GDB as its frontend.</p>
<ul>
<li><a href="https://rr-project.org/">Mozilla RR Project Home Page</a></li>
<li><a href="https://arxiv.org/pdf/1705.05937.pdf">Engineering Record And Replay For Deployability Extended Technical Report</a></li>
<li><a href="https://www.youtube.com/watch?v=hYsLBcTX00I">Quick rr demo (video)</a></li>
<li><a href="https://youtu.be/61kD3x4Pu8I">rr 1.2 basic demo (video)</a></li>
<li><a href="https://www.youtube.com/watch?v=H4iNuufAe_8">Robert O’Callahan - Corporation Taming Nondeterminism (video)</a></li>
</ul></li>
<li><p><strong>Peter B. Kessler: Fast Breakpoints</strong></p>
<p>Details of an implementation for fast conditional breakpoints using jumps to generated code.</p>
<ul>
<li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.2322&rep=rep1&type=pdf">CiteSeerX download</a></li>
</ul>
<pre><code>Fast Breakpoints
Peter B. Kessler, 1990
Proceedings of the ACM SIGPLAN '90
White Plains, New York, June 20-22, 1990.
DOI: 10.1.1.90.2322</code></pre></li>
<li><p><strong>Acid: A Debugger Built From A Language</strong></p>
<p>Plan 9’s debugger, built around a programming language. The same programming language is used as the command line of Acid, to build most of Acid’s functionality and by the compiler to describe debug symbols.</p>
<ul>
<li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.472.8070&rep=rep1&type=pdf">CiteSeerX download</a></li>
<li><a href="http://doc.cat-v.org/plan_9/2nd_edition/papers/acid/">Online version</a></li>
</ul>
<pre><code>Acid: A Debugger Built From A Language
Phil Winterbottom, Lucent Technologies Inc.
Proc. of the Winter 1994 USENIX Conf., pp. 211-222, San Francisco, CA
DOI: 10.1.1.472.8070</code></pre></li>
</ul>
</body>
</html>