fix #236, can't write comments in sql files

[cyyeh]

Description

Remove comments in sql statements

Issue ticket number

closes #236

Additional Context

Previously, we didn't remove comments from sql statements.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
vulcan-sql-document	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 30, 2023 7:54am

[kokokuo]

Beside multiple test case suggestion and one typo, others LGTM

[kokokuo]

Typo: xtension should be Extension

[kokokuo]

BTW, Please assist to add the multiple line comment test case, thanks !

[cyyeh]

fixed

[kokokuo]

Please assist to add the multiple line comment test case, thanks !

[cyyeh]

added multi-line comments in the test case

[kokokuo]

LGTM 👍 Perfect !

[oscar60310]

Hey @kokokuo @cyyeh, this PR does resolve the issue with the comments, but there are some trade-offs involved in doing so.

The syntax -- is the comments for SQL Engines, not for Jinja, which uses {# #}. So this will work:

{# {{ context.params.business_id  }} #}
SELECT * FROM 'data/yelp_academic_dataset_business.parquet' AS businesses

Vulcan treats -- as a normal string so it will try to parametrize it. So this won't work:

-- {{ context.params.business_id  }}
SELECT * FROM 'data/yelp_academic_dataset_business.parquet' AS businesses

-- $1
SELECT * FROM 'data/yelp_academic_dataset_business.parquet' AS businesses

SQL engine complains about that because of number of bindings doesn't match.

If we can prevent the variables in SQL engine comments from being parametrized, the issue will be resolved.

---

There are some cases we might break:

1. If the SQL engine DOES need the comments, e.g. for auditing.

   -- Sent from Vulcan SQL: {{ context.user.name | raw }}
   SELECT * FROM xxxxx

2. Special comments, e.g. optimizer hints.

   SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...;

3. Special syntax. e.g.

   SELECT {{ 1--1 }} AS RESULT; {# should be 2 #}