Feat/node insertion #17

DieracDelta · 2021-05-02T20:29:27Z

This is a draft PR for AST node insertion. Objectives:

create AST nodes for attribute sets
convert between Input type and attribute set
clean code
- so many places need &str instead of String
- better file naming to match rest of project
- clippy
tie into main control loop so we can create inputs
more tests

Resolves #18 and #14 .

src/parser/input_utils.rs

fogti · 2021-05-02T22:00:25Z

src/parser/input_utils.rs

+    GreenNode::new(kind, children)
+}
+
+pub fn gen_attr_set(attr_pairs: Vec<(String, String)>) -> GreenNode{


Suggested change

pub fn gen_attr_set(attr_pairs: Vec<(String, String)>) -> GreenNode{

pub fn gen_attr_set(attr_pairs: &[(String, String)]) -> GreenNode{

I'm trying to do this, and running into issues here. It looks like there's not an easy way to convert from a vector to an array..

src/parser/input_utils.rs

DieracDelta · 2021-05-10T00:23:54Z

Input generation works, and I've added in a nixpkgs_fmt pass to make sure things look nice. I've added a couple of tests as well. Code quality is poor; lots of unused imports, dead code, etc. A bunch of the processing done in main.rs does not belong. Inputs_utils needs to be split out into something more modular that matches with the rest of the style. Not ready to merge, but getting there.

I also think that a bunch of the functions in utils.rs and input_utils.rs morally speaking belong in rnix. I'm just filling in the gaps done by TODOs. I was hoping for more discussion on the issue I opened there...

DieracDelta · 2021-05-10T01:15:19Z

Also, inputs need to be (1) checked, and (2) there should be some sort of type generation. Ideally, we should support other types of flake inputs that covers the entire spec. Path/git/, enabling submodules, revisions, etc. Essentially, everything specified here. I think I'd prefer to do these in separate PRs though. The scaffolding here can define the basis for further work.

src/main.rs

fogti · 2021-05-12T13:07:52Z

src/parser/input_utils.rs

+    let kind = NixLanguage::kind_to_raw(NODE_KEY_VALUE);
+    let assign_kind = NixLanguage::kind_to_raw(TOKEN_ASSIGN);
+    let whitespace_kind = NixLanguage::kind_to_raw(TOKEN_WHITESPACE);
+    let semicolon_kind = NixLanguage::kind_to_raw(TOKEN_SEMICOLON);
+    let children = vec![
+        NodeOrToken::Node(key),
+        NodeOrToken::Token(GreenToken::new(whitespace_kind, " ")),
+        NodeOrToken::Token(GreenToken::new(assign_kind, "=")),
+        NodeOrToken::Token(GreenToken::new(whitespace_kind, " ")),
+        NodeOrToken::Node(value),
+        NodeOrToken::Token(GreenToken::new(semicolon_kind, ";")),
+        NodeOrToken::Token(GreenToken::new(whitespace_kind, "\n")),
+    ];


Suggested change

let kind = NixLanguage::kind_to_raw(NODE_KEY_VALUE);

let assign_kind = NixLanguage::kind_to_raw(TOKEN_ASSIGN);

let whitespace_kind = NixLanguage::kind_to_raw(TOKEN_WHITESPACE);

let semicolon_kind = NixLanguage::kind_to_raw(TOKEN_SEMICOLON);

let children = vec![

NodeOrToken::Node(key),

NodeOrToken::Token(GreenToken::new(whitespace_kind, " ")),

NodeOrToken::Token(GreenToken::new(assign_kind, "=")),

NodeOrToken::Token(GreenToken::new(whitespace_kind, " ")),

NodeOrToken::Node(value),

NodeOrToken::Token(GreenToken::new(semicolon_kind, ";")),

NodeOrToken::Token(GreenToken::new(whitespace_kind, "\n")),

];

use NixLanguage::kind_to_raw;

use NodeOrToken::{Node, Token};

let kind = kind_to_raw(NODE_KEY_VALUE);

let assign_kind = kind_to_raw(TOKEN_ASSIGN);

let whitespace_kind = kind_to_raw(TOKEN_WHITESPACE);

let semicolon_kind = kind_to_raw(TOKEN_SEMICOLON);

let children = vec![

Node(key),

Token(GreenToken::new(whitespace_kind, " ")),

Token(GreenToken::new(assign_kind, "=")),

Token(GreenToken::new(whitespace_kind, " ")),

Node(value),

Token(GreenToken::new(semicolon_kind, ";")),

Token(GreenToken::new(whitespace_kind, "\n")),

];

Can't import NixLanguage::kind_to_raw. Not sure why....

Oh... (that might be the case because the method is defined in a trait impl)
Maybe create a small wrapper for it then:

#[inline(always)] fn kind_to_raw(x: rnix::SyntaxKind) -> rowan::SyntaxKind { NixLanguage::kind_to_raw(x) }

src/parser/input_utils.rs

fogti · 2021-05-16T14:54:34Z

src/parser/input_utils.rs

 use rowan::{api::SyntaxNode, GreenNode, GreenNodeBuilder, GreenToken, Language, NodeOrToken};
 use std::string::ToString;
+use NodeOrToken::{Node, Token};


Suggested change

use rowan::{api::SyntaxNode, GreenNode, GreenNodeBuilder, GreenToken, Language, NodeOrToken};

use std::string::ToString;

use NodeOrToken::{Node, Token};

use rowan::{api::SyntaxNode, GreenNode, GreenNodeBuilder, GreenToken, Language, NodeOrToken::{self, Node, Token}};

use std::string::ToString;

fogti · 2021-05-16T14:57:45Z

src/parser/input_utils.rs

@@ -99,26 +94,17 @@ pub fn merge_attr_sets(a1: GreenNode, a2: GreenNode) -> GreenNode {
 pub fn new_attr_set(attr_pairs: Vec<(GreenNode, GreenNode)>) -> GreenNode {
    let pairs: Vec<NodeOrToken<_, _>> = attr_pairs
        .iter()
-        .map(move |(k, v)| NodeOrToken::Node(new_key_value(k.clone(), v.clone())))
+        .map(move |(k, v)| Node(new_key_value(k.clone(), v.clone())))
        .collect::<Vec<_>>();
    let open_curly_kind = NixLanguage::kind_to_raw(TOKEN_CURLY_B_OPEN);
    let close_curly_kind = NixLanguage::kind_to_raw(TOKEN_CURLY_B_CLOSE);
    let attr_set_kind = NixLanguage::kind_to_raw(NODE_ATTR_SET);
    let whitespace_kind = NixLanguage::kind_to_raw(TOKEN_WHITESPACE);
    let mut token_vec = Vec::new();


suggestion: replace the complete following stuff with this:

let mut token_vec = Vec::new(); let mut tokens = vec![ Token(GreenToken::new(open_curly_kind, "{")), Token(GreenToken::new(whitespace_kind, "\n")), ]; tokens.extend(attr_pairs .into_iter() .map(move |(k, v)| Node(new_key_value(k, v)))); tokens.push(Token(GreenToken::new(close_curly_kind, "}")));

fogti · 2021-05-16T15:01:14Z

src/parser/input_utils.rs

+        let kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(NODE_STRING);
+        node.start_node(kind);
+        let start_string_kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_START);
+        node.token(start_string_kind, "\"");
+        let string_content: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_CONTENT);
+        node.token(string_content, &item.val.to_string());
+        let end_string_kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_END);
+        node.token(end_string_kind, "\"");


Suggested change

let kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(NODE_STRING);

node.start_node(kind);

let start_string_kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_START);

node.token(start_string_kind, "\"");

let string_content: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_CONTENT);

node.token(string_content, &item.val.to_string());

let end_string_kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_END);

node.token(end_string_kind, "\"");

node.start_node(NixLanguage::kind_to_raw(NODE_STRING));

node.token(NixLanguage::kind_to_raw(TOKEN_STRING_START), "\"");

node.token(NixLanguage::kind_to_raw(TOKEN_STRING_CONTENT), &item.val.to_string());

node.token(NixLanguage::kind_to_raw(TOKEN_STRING_END), "\"");

fogti · 2021-05-16T15:02:20Z

src/parser/input_utils.rs

+pub(crate) struct Key {
+    val: SmlStr,
+}


I think a simple newtype struct would be a better fit, unless you intend to add more fields.

Suggested change

pub(crate) struct Key {

val: SmlStr,

}

pub(crate) struct Key(SmlStr);

DieracDelta · 2021-06-13T22:22:20Z

@zseri sorry I haven't pulled in your changes yet. I think that in order for this to work in a scalable manner (we'd like to pull in some sort of evaluation mechanism so we can meaningfully make edits and check their correctness), the IR needs to be a bit more flushed out. Going to focus on that for a bit then cleanup in the input code. In essence we're making our own IR here. I think it should at a higher level than the output of the parser, and we should have some sort of mechanism to switch between IR and GreenNodes. We'll parse the structure into this higher level IR, make changes based on user request, then push down to GreenNodes again.

fogti

I would suggest splitting the EDSL-like + conversion between that and rnix, rowan into a separate crate (maybe rnix-ddhir), so we can lay out a cleaner interface for it rather sooner than later.

fogti · 2021-06-14T19:33:09Z

src/ir/hlir_types.rs

+    String,
+    Function,
+    List(Box<NixType>),
+    AttrSet(Vec<Box<NixType>>),


how is this AttrSet type decl supposed to work?

fogti · 2021-06-14T19:34:34Z

src/ir/hlir_types.rs

+    Str(SmlStr),
+}
+
+trait NixValue {}


why not model all possible values here (including rather opaque stuff like function calls)? Maybe it would be a good idea to put this into a separate crate.

fogti · 2021-06-14T19:35:48Z

src/parser/input_utils.rs

-        let kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(NODE_KEY);
+#[derive(Debug, Clone, Eq, PartialEq, Default)]
+pub(crate) struct NixAttrSet<T: Into<GreenNode> + Clone> {
+    kvs: Vec<NixKeyValue<T>>,


why don't we use something like the indexmap crate here?

fogti

necessary because of the removal of the SmlStr newtype

fogti · 2021-11-03T12:35:53Z

src/parser/input_utils.rs

@@ -0,0 +1,269 @@
+use crate::parser::utils::{string_to_node, NixNode};
+use crate::SmlStr;


Suggested change

use crate::SmlStr;

use smol_str::SmolStr;

fogti · 2021-11-03T12:36:05Z

src/parser/input_utils.rs

+
+#[derive(Debug, Clone, Eq, PartialEq, Default)]
+pub(crate) struct NixKey {
+    pub val: SmlStr,


Suggested change

pub val: SmlStr,

pub val: SmolStr,

fogti · 2021-11-03T12:36:16Z

src/parser/input_utils.rs

+
+#[derive(Debug, Clone, Eq, PartialEq, Default)]
+pub(crate) struct NixStringLiteral {
+    pub val: SmlStr,


Suggested change

pub val: SmlStr,

pub val: SmolStr,

fogti · 2021-11-03T12:36:49Z

src/parser/input_utils.rs

+//Key(SmlStr),
+//StringLiteral(SmlStr),


Suggested change

//Key(SmlStr),

//StringLiteral(SmlStr),

//Key(SmolStr),

//StringLiteral(SmolStr),

fogti · 2021-11-03T12:37:00Z

src/parser/input_utils.rs

+//let mut inputs = Vec::new();
+//if let Some(s) = item.url {
+//inputs.push((
+//SyntaxStructure::Key(SmlStr::new_inline("url")).into(),


Suggested change

//SyntaxStructure::Key(SmlStr::new_inline("url")).into(),

//SyntaxStructure::Key(SmolStr::new_inline("url")).into(),

fogti · 2021-11-03T12:39:30Z

src/user/mod.rs

@@ -124,6 +133,8 @@ pub(crate) enum UserPrompt {
    #[display("{0}")]
    SelectLang(Lang),
    #[display("{0}")]
+    Bool(bool),
+    #[display("{0}")]
    Other(SmlStr),


??? maybe rebase necessary ???

fogti · 2021-11-03T12:39:48Z

src/ir/hlir_types.rs

@@ -0,0 +1,21 @@
+use crate::SmlStr;


Suggested change

use crate::SmlStr;

use smol_str::SmolStr;

fogti · 2021-11-03T12:39:56Z

src/ir/hlir_types.rs

+enum NixPrimitive {
+    Bool(bool),
+    Int(i64),
+    Str(SmlStr),


Suggested change

Str(SmlStr),

Str(SmolStr),

fogti · 2021-11-03T12:40:16Z

src/parser/input_utils.rs

+        //let mut inputs_gn = Vec::<Box<dyn>>::new()
+        //if let Some(url) = input.url {
+        //inputs_gn.push((
+        //NixKey{val: SmlStr::new_inline("url")}.into(),


Suggested change

//NixKey{val: SmlStr::new_inline("url")}.into(),

//NixKey{val: SmolStr::new_inline("url")}.into(),

fogti · 2021-11-03T12:40:26Z

src/parser/input_utils.rs

+        //let url_gn: GreenNode = url.into();
+        //}
+        //if let Some(is_flake) = input.is_flake {
+        //inputs_gn.push(NixKey{val: SmlStr::new_inline("flake")}.into(), );


Suggested change

//inputs_gn.push(NixKey{val: SmlStr::new_inline("flake")}.into(), );

//inputs_gn.push(NixKey{val: SmolStr::new_inline("flake")}.into(), );

DieracDelta added 3 commits April 26, 2021 23:03

starting out the input generation

ba3f7b3

idiomatic attribute set generation

4b27f8f

updating readme a bit

fbe3676

DieracDelta mentioned this pull request May 2, 2021

Add more options when generating inputs #20

Open

fogti requested changes May 2, 2021

View reviewed changes

Input generation works!

bd9ea68

cleanup

7db4be5

fogti requested changes May 12, 2021

View reviewed changes

more changes

844d87d

fogti requested changes May 16, 2021

View reviewed changes

redoing the rep

e443905

fogti reviewed Jun 14, 2021

View reviewed changes

fogti requested changes Nov 3, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/node insertion #17

Feat/node insertion #17

DieracDelta commented May 2, 2021 •

edited

Loading

fogti May 2, 2021

DieracDelta May 10, 2021 •

edited

Loading

DieracDelta commented May 10, 2021 •

edited

Loading

DieracDelta commented May 10, 2021

fogti May 12, 2021

DieracDelta May 15, 2021

fogti May 15, 2021

fogti May 16, 2021

fogti May 16, 2021

fogti May 16, 2021

fogti May 16, 2021

DieracDelta commented Jun 13, 2021 •

edited

Loading

fogti left a comment

fogti Jun 14, 2021

fogti Jun 14, 2021

fogti Jun 14, 2021

fogti left a comment

fogti Nov 3, 2021

fogti Nov 3, 2021

fogti Nov 3, 2021

fogti Nov 3, 2021

fogti Nov 3, 2021

fogti Nov 3, 2021

fogti Nov 3, 2021

fogti Nov 3, 2021

fogti Nov 3, 2021

fogti Nov 3, 2021

	pub fn gen_attr_set(attr_pairs: Vec<(String, String)>) -> GreenNode{
	pub fn gen_attr_set(attr_pairs: &[(String, String)]) -> GreenNode{

		@@ -0,0 +1,269 @@
		use crate::parser::utils::{string_to_node, NixNode};
		use crate::SmlStr;

	//SyntaxStructure::Key(SmlStr::new_inline("url")).into(),
	//SyntaxStructure::Key(SmolStr::new_inline("url")).into(),

	//NixKey{val: SmlStr::new_inline("url")}.into(),
	//NixKey{val: SmolStr::new_inline("url")}.into(),

	//inputs_gn.push(NixKey{val: SmlStr::new_inline("flake")}.into(), );
	//inputs_gn.push(NixKey{val: SmolStr::new_inline("flake")}.into(), );

Feat/node insertion #17

Are you sure you want to change the base?

Feat/node insertion #17

Conversation

DieracDelta commented May 2, 2021 • edited Loading

Choose a reason for hiding this comment

DieracDelta May 10, 2021 • edited Loading

Choose a reason for hiding this comment

DieracDelta commented May 10, 2021 • edited Loading

DieracDelta commented May 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DieracDelta commented Jun 13, 2021 • edited Loading

fogti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fogti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DieracDelta commented May 2, 2021 •

edited

Loading

DieracDelta May 10, 2021 •

edited

Loading

DieracDelta commented May 10, 2021 •

edited

Loading

DieracDelta commented Jun 13, 2021 •

edited

Loading