MPL Languages support

[myCrack]

I have been tried to add support of MPL Languages in viasfora, but:

• After reinstating of MPL-plugin, viasfora (for MPL) didn't highlight anything at all til viasfora's reinstalling.
• User-defined words not highlighted at all.
• Escape-sequences in strings not highlighted at all.

My MPL Patch don't provide you with support of user-defined words, and escape-sequences, but for sake of test you can use any string parser from viasfora. Also, you can define any user-defined words as you prefer.

MPL - https://gravilink.com
MPL-Plugin - (use vs2019) https://marketplace.visualstudio.com/items?itemName=Matway.mpl-vs
MPL Examples - https://github.com/Matway/mpl-sl/blob/master/control.mpl
MPL Grammar - https://gravilink.com/mplGrammar.2017-07-01.xhtml

Short grammar:
String ::= '"' ( '\\' | '\"' | [^"\] )* '"'
String is a sequence of any characters enclosed in "". " and \ can be escaped with \. For example "this is a string with \" quote" and "this is a string with \\ slash"

Comment ::= '#' any characters but CR and LF

MPL patch:

using System;
using System.ComponentModel.Composition;

using Winterdom.Viasfora.Rainbow;
using Winterdom.Viasfora.Settings;
using Winterdom.Viasfora.Util;

namespace Winterdom.Viasfora.Languages {
  public static partial class Langs {
    public const String MPL = "MPL";
  }

  namespace MPL {
    [Export(typeof(ILanguage))]
    class MPL : LanguageInfo, ILanguage {
      public const String ContentType = "MPL";

      protected override String[] SupportedContentTypes => new String[] { ContentType };

      public ILanguageSettings Settings { get; private set; }

      [ImportingConstructor]
      public MPL(ITypedSettingsStore store) {
        this.Settings = new Settings(store);
      }

      protected override IBraceScanner NewBraceScanner() => new Braces();
    }

    public class Settings : LanguageSettings {
      protected override String[] ControlFlowDefaults => EMPTY;
      protected override String[] LinqDefaults => EMPTY;
      protected override String[] VisibilityDefaults => EMPTY;

      public Settings(ITypedSettingsStore store)
        : base(Langs.MPL, store) {
      }
    }

    public class Braces : IBraceScanner {
      private enum State {
        Text, MultiLineString
      }

      private State status = State.Text;

      public String BraceList => "(){}[]:;";

      public Braces() { }

      public void Reset(int state) {
        this.status = (int)State.Text;
      }

      public bool Extract(ITextChars tc, ref CharPos pos) {
        while ( !tc.AtEnd ) {
          switch ( this.status ) {
            case State.MultiLineString: String(tc); break;
            default:
              return Parse(tc, ref pos);
          }
        }
        return false;
      }

      private bool Parse(ITextChars tc, ref CharPos pos) {
        while ( !tc.AtEnd ) {
          // Comment.
          if ( tc.Char() == '#' ) {
            tc.SkipRemainder();
          }

          // String.
          else if ( tc.Char() == '"' ) {
            tc.Next();

            this.status = State.MultiLineString;
            this.String(tc);

            continue;
          }

          // Braces.
          else if ( this.BraceList.IndexOf(tc.Char()) >= 0 ) {
            pos = new CharPos(tc.Char(), tc.AbsolutePosition);
            tc.Next();
            return true;
          }

          // Code.
          tc.Next();
        }
        return false;
      }

      private void String(ITextChars tc) {
        while ( !tc.AtEnd ) {
          // End of a String.
          if ( tc.Char() == '"' ) {
            tc.Next();
            this.status = State.Text;
            return;
          }

          // Start of an Escape Sequence.
          if ( tc.Char() == '\\' )
            tc.Next();

          // Content of a String, or an Escaped Character.
          tc.Next();
        }
      }
    }
  }
}

[myCrack]

... user-defined words - you called it flow-control, LINQ, etc.

[myCrack]

@tomasr perhaps it's all my poor english, so are you need some clarifications?

[tomasr]

I've refactored and added this now in the develop branch.

[myCrack]

After reinstating of MPL-plugin, viasfora (for MPL) didn't highlight anything at all til viasfora's reinstalling.

I'm so sorry. It was my fault. After installing patched viasfora, VS "reinstalled" it from marketplace.

[myCrack]

Escape-sequences in strings not highlighted at all.

For now, C-String-parser works fine. Also, invalid escape-sequences (supported by C but not by MLP) don't highlighted at all. So I think you did it.

You must update readme.md by adding MPL in the list of supported languages.

User-defined words -- i'll check it later, and if it is work as expected then I'll close the issue.

[tomasr]

I just reused the C string parser for escape sequences for now. A simplified version that only supports the MPL escape sequences should be easy to write if needed. From what little I looked at yesterday, seems like it essentially only supports \\ and \", correct?

[myCrack]

Yes, it supports only \\ and \".

[myCrack]

• As you can see, user-defined words matching is case-insensitive, but all mpl languages are case-sensitive.
• All labels (name + colon, i.e. a:) must be the same color. Regardless of user-defined words.
• User-defined words must be applied to all names. For now, it applies only to builtin ones (i.e. call).
• I believe, format specifiers for strings (a la pritnf) must not be hightailed.

[tomasr]

I've finally done some improvements here:

As you can see, user-defined words matching is case-insensitive, but all mpl languages are case-sensitive.

Changed to do case-sensitive comparisons, as suggested.

All labels (name + colon, i.e. a:) must be the same color. Regardless of user-defined words.

I've now excluded labels when looking for keywords.

User-defined words must be applied to all names. For now, it applies only to builtin ones (i.e. call).

This isn't really possible, since I'm not doing a full parser for the language. Right now, I rely on the underlying MPL parser to tell me what is a language keyword so that I can analyze it further. Supporting arbitrary words would require parsing the actual grammar to understand what's what, which I'm not keen on doing.

I believe, format specifiers for strings (a la pritnf) must not be hightailed.

I've simplified the string scanner here to only support \\ and \" as suggested.

[myCrack]

I've finally done some improvements here.

Thanks.

Supporting arbitrary words would require parsing the actual grammar to understand what's what, which I'm not keen on doing.

For this, i think we need just a lexer, isn't it?
If it is, then i will investigate how manage it.

---

By the way, the current state of mpl support, with the last changes you've made, can be released in the new version of viasfora. And the supporting arbitrary words can be added in another future version.

[tomasr]

For this, i think we need just a lexer, isn't it?

In the general case, no, it's likely not enough, because you need to not only tokenize the text, but also understand context of how stuff is used.

I really don't want to have to write an entire lexer / parser for each and every single language Viasfora supports :(

[myCrack]

Thanks, i will test the changes for some time, and if it is ok, then i will close the issue.