Improving the compiler is painful

[realvictorprm]

During my intensive hours with the F# compiler today I noticed some points which made the experience unfortuantely not funny:

1. It takes ages to compile the compiler again after even small changes. Big development times so. (Even worser, the compiler is always recompiled whether there are changes or not ;))
2. Missing Colorization, Intellisense and Visual F# crashes during development. Adding another strike on increasing development time. (sample below)
   
3. General development of this repository requires massive disk space (compared to other repositories) and also e.g. administrative rights for initially building the repository. The last point should actually be a no-go.

One point which can be changed which I already started was to remove the last bits of Perl.

I don't want to discourage contributions neither I want to dishonour any effort here. I'm noting this down here because I'm very interested and also willing to change the situation. However this requires that we collect some ideas how we can both reduce development times and requirements for contributions to the compiler. It wouldn't hurt the Microsoft developers here too because their time is spare too.

Also @dsyme I'm not sure whether this is correct but it looks like there's much code in the compiler which applies to very old versions to F# or rather to the initial times of the F# compiler (as experiment). Am I wrong here? (Just looking whether I could start with a small cleanup there if possible).

[mrange]

Not sure if my opinions are still valid since it was a long time since I did contributions

I agree but for me the most challenging part was getting the test-coverage right in order to reach confidence.

I found that I could improve my experience by extracting the modules of the compiler or library I wanted to improve into a separate solution and experiment in that solution and then backport it into the compiler. Not the optimal situation but my experience was improved.

[realvictorprm]

I'm currently working on the String-Interpolation feature and so for now working in the lexing and parsing part. However the autogeneration there is confusing, I wonder whether it could be removed and replaced with something more straightforward.

[realvictorprm]

Hm is it possible to share your old workflow with us @mrange? (Maybe some repository)

[mrange]

The lexer/parser I remember. I also think that it adds to the confusion that F# has it's own implementation the framework that drives the lexer/parser and whenever you get problems there it's very tough drilling to the root cause. Might have changed these days. The lexer/parser is however quite competent and reasonable fast IIRC.

Unfortunately I might have added to the complexity with pplex and pppars that parses #if expressions.

[realvictorprm]

Well @mrange in my current situation it would be helpful to know how to add tokens haha 😂 (searching for the correct way)
I slowly understand how it works ...

Also I can agree that the generation is fast. Somewhere else much happens I guess.

[mrange]

When I did updates to the lexer/parser I did break out those parts into a seperate solution and used the OSS version to get it roughly right: http://fsprojects.github.io/FsLexYacc/

I spent a long time on getting the fallbacks for invalid syntax working decently. Not an expert on lexx/yavv.

[mrange]

@realvictorprm are you trying to do string interpolation in the lexer? The lexer parses IIRC string literals so it feels like it's that place that needs extending.

Of course if one needs F# sub expressions in the interpolated strings then I think the lexer needs to invoke the parser on the sub expressions. Not sure about the best way forward for that.

[realvictorprm]

@mrange exactly! That's where I spend my last 6 hours 🙂. I tried figuring out how to add a correct token which contains the necessary information for the type checker "I'm an interpolated string and need special treatment" ;)

EDIT:
Okay I meanwhile figured out how it works preferred.

[mrange]

I am not saying this is the right approach but one could add an INTERPOLATED_STRING in pars.fsy

%token <string> STRING 
// A new token
%token <(string*FormattedValue) list*string> INTERPOLATED_STRING 

This holds strings separated by formatted values. The expression trees needs expanding to support this but let's focus on the parser/lexer first.

Now the lexer has to be changed to generate INTERPOLATED_STRING tokens.

The lexer parses strings using common helper method

 // This triggers on a normal `"String"`
 | '"' 
     { let buf,fin,m = startString args lexbuf 
       if not skip then (STRING_TEXT (LexCont.String(!args.ifdefStack,m))) else string (buf,fin,m,args) skip lexbuf }

You can added something that matches interpolated strings

 | 'I' '"' 
     { call a helper function for interpolated strings }

Here is where I would actually implement all of this outside the F# compiler using FsLex/FsYacc and when I get this to work backport it into the compiler because of the compilation times.

Hopefully, someone else chimes in with some actual advice on how the string parsing works in the lexer :)

[realvictorprm]

Great, looks like I was playing at the right location. I already touched the DOLLAR symbol and replaced the former compilation error with an conversion to strings. I've added the token to the parser generator and now I need to figure out how it goes on. My first goal currently is to at least pass to the type checker that we have an interpolated string.

[mrange]

You could look at pplex.fsl and pppars.fsy because the grammar is much simpler and that can help with understanding the lex.fsl and pars.fsy

[realvictorprm]

Let us move to the RRC, this issue should actually be filled with ideas on how to ease contributing to the compiler 😄 Although the stuff we talk about here should give people an insight too!
fsharp/fslang-design#6 (it's the 6th suggestion 😄?)

[saul]

@realvictorprm do the colourisation errors occur around #line declarations? Are there any in the file?

[realvictorprm]

Yes my example was in the parser.fs file.

[realvictorprm]

This PR #3601 seems to improve the whole Visual F# experience and so also eases work in the solution itself 🙂

[dsyme]

It takes ages to compile the compiler again after even small changes. Big development times so. (Even worser, the compiler is always recompiled whether there are changes or not ;))

I'm looking into this now. A definite regression in the development experience

See #3589

[dsyme]

Also @dsyme I'm not sure whether this is correct but it looks like there's much code in the compiler which applies to very old versions to F# or rather to the initial times of the F# compiler (as experiment). Am I wrong here? (Just looking whether I could start with a small cleanup there if possible).

There are some archaic bits. By all means clean them up / modernize them

[cartermp]

Closing out old discussion. Please file issues on specific problems you encounter here.