Skip to content

Grammar Debugging

Jump to bottom Edit New page
Bastiaan Veelo edited this page Sep 5, 2023 · 18 revisions

Pegged shows you the rules that fail when you print out the parse tree or dump it to HTML and inspect it in our browser (see Parse-Result), but sometimes it is difficult to see why your grammar fails to parse a certain input, and you would wish to see inside what's going on. Since v0.3.2, Pegged allows you to do just that after enabling the built-in tracer. Beware that tracing can produce a lot of output, especially with large grammars, so read on to learn about conditional tracing. Try not to trace non-memoizing parsers, as these may explode your trace file and contain a lot of repetition. In other words: avoid tracing compile-time generated parsers because they don't support memoization, generate the parser as a module instead. Tracing is done at run time, so any parses done at compile time are not traced.

The tracer is implemented in peg.d, under the version (tracer) conditional. So to enable the tracer it is important that peg.d is included in the compilation and that -version="tracer" is given as compile option. If you use dub for compilation, this can be accomplished by selecting the tracer configuration:

dub build --config=tracer

Tracing can be switched on and off dynamically, either by calling the global switches traceAll() and traceNothing(), or by registering a trace condition method. The default is traceNothing(). Using traceAll() will cause every rule to be traced, also the built-in Pegged rules. To restrict tracing to the rules generated from MyGrammar, you can set a trace condition using a lambda like

version (tracer)
{
    setTraceConditionFunction(function(string ruleName, const ref ParseTree p) {return ruleName.startsWith("MyGrammar");});
}

If you need to analyse a more elaborate parse and want to only start tracing when RuleOfInterest is tried, you can use the following condition function:

version (tracer)
{
    bool cond(string ruleName, const ref ParseTree p)
    {
        static startTrace = false;
        if (ruleName.startsWith("MyGrammar.RuleOfInterest"))
            startTrace = true;
        return startTrace && ruleName.startsWith("MyGrammar");
    }
    setTraceConditionFunction(&cond);
}

Or if you only want to start the trace after a certain position has been reached, replace the condition above with something like

        if (p.begin > 4000)
            startTrace = true;

The tracing itself is implemented using std.logger from Phobos, which means you can use its API to customise the logging. As traces can get very large, it is most practical to write the trace to a file. The default loggers decorate the messages with a time stamp and location information etcetera, which is not of interest in our case. Therefore, Pegged provides a specialisation of FileLogger called TraceLogger that just prints the trace message in the file. It is used like this:

version (tracer)
{
    import std.logger;
    sharedLog = cast(shared) new TraceLogger("TraceLog.txt");
}

This writes the trace to TraceLog.txt. If you are on a Posix system, you can follow the parsing as it happens using

tail -f TraceLog.txt

Summary

Tracing is off by default. Call traceAll(); to trace everything. Exclusion of Pegged built-ins is supported using setTraceConditionFunction, which can also be used to exclude specific parts if you write the condition for it. Tracing of compile time generated parsers is supported but not recommended, as they do not memoize. The thing that is not supported is tracing of compile time generated parse trees (if you have enum p = MyParser(input); then change it to auto p = MyParser(input);).

Example

Let's say we have followed the example from Grammars as D Modules and generated arithmetic.d by running rdmd on a file with the following contents:

import pegged.grammar;

void main()
{
    asModule("arithmetic", "arithmetic",
    "Arithmetic:
        Expr     <- Factor AddExpr*
        AddExpr  <- ('+'/'-') Factor
        Factor   <- Primary MulExpr*
        MulExpr  <- ('*'/'/') Primary
        Primary  <- Parens / Number / Variable / '-' Primary
        Parens   <- '(' Expr ')'
        Number   <~ [0-9]+
        Variable <- identifier
    ");
}

We are using the parser (at run time) in test.d:

import arithmetic;

void main()
{
    auto parseTree = Arithmetic("2/(8*7988+1*6196-y)");
}

To trace this example, we can extend the program as follows

import arithmetic;

void main()
{
    version (tracer)
    {
        import std.logger;
        sharedLog = cast(shared) new TraceLogger("TraceLog.txt");
        setTraceConditionFunction(ruleName => ruleName.startsWith("Arithmetic"));
    }

    auto parseTree = Arithmetic("2/(8*7988+1*6196-y)");
}

and compile with

rdmd -Ipath/to/Pegged -version="tracer" test.d

Because we are not linking with a precompiled Pegged, peg.d is dragged into the compilation through the inclusion of modules, so that the tracer is compiled in.

The file TraceLog.txt is generated with the following contents:

1| (l:0, c:0)	and!(Factor, zeroOrMore) considering rule Arithmetic.Factor on "2/(8*7988+1*619"...
1|2| (l:0, c:0)	and!(Primary, zeroOrMore) considering rule Arithmetic.Primary on "2/(8*7988+1*619"...
1|2|3| (l:0, c:0)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Parens on "2/(8*7988+1*619"...
1|2|3| (l:0, c:0)	Arithmetic.Parens FAILED on "2/(8*7988+1*619"...
1|2|3| (l:0, c:0)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Number on "2/(8*7988+1*619"...
1|2|3| (l:0, c:1)	Arithmetic.Number SUCCEEDED on "2"
1|2|3| (l:0, c:1)	zeroOrMore!(Arithmetic.MulExpr) considering rule Arithmetic.MulExpr on "/(8*7988+1*6196"...
1|2|3|4| (l:0, c:2)	and!(keywords, Primary) considering rule Arithmetic.Primary on "(8*7988+1*6196-"...
1|2|3|4|5| (l:0, c:2)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Parens on "(8*7988+1*6196-"...
1|2|3|4|5|6| (l:0, c:3)	and!(literal, Expr, literal) considering rule Arithmetic.Expr on "8*7988+1*6196-y"...
1|2|3|4|5|6|7| (l:0, c:3)	and!(Factor, zeroOrMore) considering rule Arithmetic.Factor on "8*7988+1*6196-y"...
1|2|3|4|5|6|7|8| (l:0, c:3)	and!(Primary, zeroOrMore) considering rule Arithmetic.Primary on "8*7988+1*6196-y"...
1|2|3|4|5|6|7|8|9| (l:0, c:3)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Parens on "8*7988+1*6196-y"...
1|2|3|4|5|6|7|8|9| (l:0, c:3)	Arithmetic.Parens FAILED on "8*7988+1*6196-y"...
1|2|3|4|5|6|7|8|9| (l:0, c:3)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Number on "8*7988+1*6196-y"...
1|2|3|4|5|6|7|8|9| (l:0, c:4)	Arithmetic.Number SUCCEEDED on "8"
1|2|3|4|5|6|7|8|9| (l:0, c:4)	zeroOrMore!(Arithmetic.MulExpr) considering rule Arithmetic.MulExpr on "*7988+1*6196-y)"...
1|2|3|4|5|6|7|8|9|10| (l:0, c:5)	and!(keywords, Primary) considering rule Arithmetic.Primary on "7988+1*6196-y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:5)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Parens on "7988+1*6196-y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:5)	Arithmetic.Parens FAILED on "7988+1*6196-y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:5)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Number on "7988+1*6196-y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:9)	Arithmetic.Number SUCCEEDED on "7988"
1|2|3|4|5|6|7|8|9|10| (l:0, c:9)	and!(keywords, Primary) SUCCEEDED on "*7988"
1|2|3|4|5|6|7|8|9| (l:0, c:9)	zeroOrMore!(Arithmetic.MulExpr) considering rule Arithmetic.MulExpr on "+1*6196-y)"
1|2|3|4|5|6|7|8|9| (l:0, c:9)	Arithmetic.MulExpr SUCCEEDED on "*7988"
1|2|3|4|5|6|7|8| (l:0, c:9)	and!(Primary, zeroOrMore) SUCCEEDED on "8*7988"
1|2|3|4|5|6|7|8| (l:0, c:9)	zeroOrMore!(Arithmetic.AddExpr) considering rule Arithmetic.AddExpr on "+1*6196-y)"
1|2|3|4|5|6|7|8|9| (l:0, c:10)	and!(keywords, Factor) considering rule Arithmetic.Factor on "1*6196-y)"
1|2|3|4|5|6|7|8|9|10| (l:0, c:10)	and!(Primary, zeroOrMore) considering rule Arithmetic.Primary on "1*6196-y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:10)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Parens on "1*6196-y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:10)	Arithmetic.Parens FAILED on "1*6196-y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:10)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Number on "1*6196-y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:11)	Arithmetic.Number SUCCEEDED on "1"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:11)	zeroOrMore!(Arithmetic.MulExpr) considering rule Arithmetic.MulExpr on "*6196-y)"
1|2|3|4|5|6|7|8|9|10|11|12| (l:0, c:12)	and!(keywords, Primary) considering rule Arithmetic.Primary on "6196-y)"
1|2|3|4|5|6|7|8|9|10|11|12|13| (l:0, c:12)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Parens on "6196-y)"
1|2|3|4|5|6|7|8|9|10|11|12|13| (l:0, c:12)	Arithmetic.Parens FAILED on "6196-y)"
1|2|3|4|5|6|7|8|9|10|11|12|13| (l:0, c:12)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Number on "6196-y)"
1|2|3|4|5|6|7|8|9|10|11|12|13| (l:0, c:16)	Arithmetic.Number SUCCEEDED on "6196"
1|2|3|4|5|6|7|8|9|10|11|12| (l:0, c:16)	and!(keywords, Primary) SUCCEEDED on "*6196"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:16)	zeroOrMore!(Arithmetic.MulExpr) considering rule Arithmetic.MulExpr on "-y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:16)	Arithmetic.MulExpr SUCCEEDED on "*6196"
1|2|3|4|5|6|7|8|9|10| (l:0, c:16)	and!(Primary, zeroOrMore) SUCCEEDED on "1*6196"
1|2|3|4|5|6|7|8|9| (l:0, c:16)	and!(keywords, Factor) SUCCEEDED on "+1*6196"
1|2|3|4|5|6|7|8| (l:0, c:16)	zeroOrMore!(Arithmetic.AddExpr) considering rule Arithmetic.AddExpr on "-y)"
1|2|3|4|5|6|7|8|9| (l:0, c:17)	and!(keywords, Factor) considering rule Arithmetic.Factor on "y)"
1|2|3|4|5|6|7|8|9|10| (l:0, c:17)	and!(Primary, zeroOrMore) considering rule Arithmetic.Primary on "y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:17)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Parens on "y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:17)	Arithmetic.Parens FAILED on "y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:17)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Number on "y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:17)	Arithmetic.Number FAILED on "y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:17)	or!(Arithmetic.Parens, Arithmetic.Number, Arithmetic.Variable, and!(literal, Primary)) considering rule Arithmetic.Variable on "y)"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:18)	Arithmetic.Variable SUCCEEDED on "y"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:18)	zeroOrMore!(Arithmetic.MulExpr) considering rule Arithmetic.MulExpr on ")"
1|2|3|4|5|6|7|8|9|10|11| (l:0, c:18)	Arithmetic.MulExpr SUCCEEDED on ""
1|2|3|4|5|6|7|8|9|10| (l:0, c:18)	and!(Primary, zeroOrMore) SUCCEEDED on "y"
1|2|3|4|5|6|7|8|9| (l:0, c:18)	and!(keywords, Factor) SUCCEEDED on "-y"
1|2|3|4|5|6|7|8| (l:0, c:18)	zeroOrMore!(Arithmetic.AddExpr) considering rule Arithmetic.AddExpr on ")"
1|2|3|4|5|6|7|8| (l:0, c:18)	Arithmetic.AddExpr SUCCEEDED on "+1*6196-y"
1|2|3|4|5|6|7| (l:0, c:18)	and!(Factor, zeroOrMore) SUCCEEDED on "8*7988+1*6196-y"
1|2|3|4|5|6| (l:0, c:19)	and!(literal, Expr, literal) SUCCEEDED on "(8*7988+1*6196-y)"
1|2|3|4|5| (l:0, c:19)	Arithmetic.Parens SUCCEEDED on "(8*7988+1*6196-y)"
1|2|3|4| (l:0, c:19)	and!(keywords, Primary) SUCCEEDED on "/(8*7988+1*6196-y)"
1|2|3| (l:0, c:19)	zeroOrMore!(Arithmetic.MulExpr) considering rule Arithmetic.MulExpr on ""
1|2|3| (l:0, c:19)	Arithmetic.MulExpr SUCCEEDED on "/(8*7988+1*6196-y)"
1|2| (l:0, c:19)	and!(Primary, zeroOrMore) SUCCEEDED on "2/(8*7988+1*6196-y)"
1|2| (l:0, c:19)	zeroOrMore!(Arithmetic.AddExpr) considering rule Arithmetic.AddExpr on ""
1|2| (l:0, c:19)	Arithmetic.AddExpr SUCCEEDED on ""
1| (l:0, c:19)	and!(Factor, zeroOrMore) SUCCEEDED on "2/(8*7988+1*6196-y)"

Each line starts with a sequence of numerals indicating the depth at which a rule is evaluated. This is followed by a line and column indication of the position in the input. The trace message itself shows what rule is currently being considered on which fragment of the input, and whether the match succeeded or failed.

Note that the tracing happens inside the PEG primitives that are used by the generated function templates that make up the parser. That is why the trace messages may appear alien at first, but on a closer look you should be able to recognise the identifiers from your grammar and deduce which expression is at work on each line.

Next lesson: Grammar Testing

Pegged Tutorial

Add a custom footer
Add a custom sidebar
Clone this wiki locally