Coverage for Function Signatures

[mdashti]

It's a known issue that function signatures might not be marked as covered, even if their body is covered. This is mainly attributed to the function inlining (as mentioned in the README file). This issue breaks the usefulness of the coverage numbers and always requires manual checking of all files.

One way to improve this (before there's a better solution via Julia itself) is to do a post-processing on the coverage files and mark the signature of all multi-line functions as covered if at least one line in their body is covered. This change will make the coverage information much more accurate and useful.

[vtjnash]

That's not necessarily known, as most related issues are closed recently

[ranocha]

The problem appears at least when inlined functions are not called directly. We also observe this issue when merging multiple coverage reports from parallel CI runs at trixi-framework/Trixi.jl#841. An MWE seems to be something like the following.

julia> using Pkg; Pkg.activate(temp=true); Pkg.add("Coverage")

julia> using Coverage

julia> dirname = mktempdir()

julia> cd(dirname)

julia> open("test.jl", "w") do io
           write(io, """

       @inline function foo(x, n::Integer)
         y = rand(typeof(x), n)
         return x + sum(y)
       end

       function foo_caller()
         foo(1.0, 5)
       end

       @inline function bar(x, n::Integer)
         y = rand(typeof(x), n)
         return x + sum(y)
       end

       function bar_caller()
         bar(1.0, 5)
       end
       """)
       end
254

julia> clean_folder("."); run(`$(Base.julia_cmd()) --code-coverage=user -e 'include("test.jl"); foo_caller()'`)

julia> for (root, dirs, files) in walkdir(".")
           for file in files
               if endswith(file, ".cov")
                   println(file)
                   println(join(readlines(file), '\n'))
               end
           end
       end
test.jl.9425.cov
        -
        - @inline function foo(x, n::Integer)
        1   y = rand(typeof(x), n)
        2   return x + sum(y)
        - end
        -
        1 function foo_caller()
        1   foo(1.0, 5)
        - end
        -
        - @inline function bar(x, n::Integer)
        -   y = rand(typeof(x), n)
        -   return x + sum(y)
        - end
        -
        - function bar_caller()
        -   bar(1.0, 5)
        - end

julia> coverage_foo = process_file("test.jl", ".")

julia> LCOV.writefile("lcov_foo.info", coverage_foo)

julia> println(join(readlines("lcov_foo.info"), '\n'))
SF:test.jl
DA:3,1
DA:4,2
DA:7,1
DA:8,1
DA:11,0
DA:12,0
DA:13,0
DA:16,0
DA:17,0
LH:4
LF:9
end_of_record

julia> clean_folder("."); run(`$(Base.julia_cmd()) --code-coverage=user -e 'include("test.jl"); bar_caller()'`)

julia> for (root, dirs, files) in walkdir(".")
           for file in files
               if endswith(file, ".cov")
                   println(file)
                   println(join(readlines(file), '\n'))
               end
           end
       end
test.jl.9448.cov
        -
        - @inline function foo(x, n::Integer)
        -   y = rand(typeof(x), n)
        -   return x + sum(y)
        - end
        -
        - function foo_caller()
        -   foo(1.0, 5)
        - end
        -
        - @inline function bar(x, n::Integer)
        1   y = rand(typeof(x), n)
        2   return x + sum(y)
        - end
        -
        1 function bar_caller()
        1   bar(1.0, 5)
        - end

julia> coverage_bar = process_file("test.jl", ".")

julia> LCOV.writefile("lcov_bar.info", coverage_bar)

julia> println(join(readlines("lcov_bar.info"), '\n'))
SF:test.jl
DA:2,0
DA:3,0
DA:4,0
DA:7,0
DA:8,0
DA:12,1
DA:13,2
DA:16,1
DA:17,1
LH:4
LF:9
end_of_record

julia> coverage = merge_coverage_counts([coverage_foo, coverage_bar])

julia> LCOV.writefile("lcov.info", coverage)

julia> println(join(readlines("lcov.info"), '\n'))
SF:test.jl
DA:2,0 # this is the line @inline function foo(x, n::Integer)
DA:3,1
DA:4,2
DA:7,1
DA:8,1
DA:11,0 # this is the line @inline function bar(x, n::Integer)
DA:12,1
DA:13,2
DA:16,1
DA:17,1
LH:8
LF:10
end_of_record

Note that the function signature lines of foo and bar are marked as executable code but are not counted as covered in the merged report. The reason seems to be that the heuristic mentioned at https://github.com/JuliaCI/Coverage.jl#a-note-for-advanced-users marks functions as code which are not executed at all. However, Julia doesn't count the function signatures of foo and bar as covered. Thus, we end up with uncovered function definition lines although the body of the functions is covered.

[vtjnash]

Thanks for that clear report. I can simplify that to this (with ./julia --code-coverage=x.info), and see there is a missing : code_coverage_effect for the call (to REPL[2]:1), which is normally implied (in codegen)

julia> foo_caller() = foo(1.0, 5)
foo_caller (generic function with 1 method)

julia> @inline function foo(x, n::Integer)
         y = x
         nothing
       end
foo (generic function with 1 method)

julia> code_typed( foo_caller )
1-element Vector{Any}:
 CodeInfo(
    @ REPL[1]:1 within `foo_caller`
1 ─      $(Expr(:code_coverage_effect))::Nothing
│  ┌ @ REPL[2]:2 within `foo`
│  │      $(Expr(:code_coverage_effect))::Nothing
│  └
│  ┌ @ REPL[2]:3 within `foo`
│  │      $(Expr(:code_coverage_effect))::Nothing
│  │ %4 = Main.nothing::Nothing
│  └
└──      return %4
) => Nothing

[sloede]

@vtjnash It's great to hear you were able to identify the issue so quickly! Do you think that this can be fixed? I have to admit, I don't really understand what's going on (and what exactly is wrong and where)... 😬

[vtjnash]

yes, the fix was quick, but not particularly simple

[sloede]

Great, thanks for tracking this down & fixing it so quickly! Given that it required a change to the Julia compiler itself, I'm glad I didn't attempt to figure this one out by myself 😆