Fix runtime error (IndexError ) when linting file with jinja "if"

[barrywhart]

Brief summary of the change made

Fixes #1414

Pull Request checklist

• Please confirm you have completed any of the necessary steps below.

• Included test cases to demonstrate any code changes, which may be one or more of the following:

  • .yml rule test cases in test/fixtures/rules/std_rule_cases.
  • .sql/.yml parser test cases in test/fixtures/parser (note YML files can be auto generated with python test/generate_parse_fixture_yml.py or by running tox locally).
  • Full autofix test cases in test/fixtures/linter/autofix.
  • Other.

• Added appropriate documentation for the change.

• Created GitHub issues for any relevant followup/future enhancements if appropriate.

[codecov]

Codecov Report

Merging #1430 (b977d63) into main (250fe28) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main     #1430   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          127       129    +2     
  Lines         8639      8709   +70     
=========================================
+ Hits          8639      8709   +70     

Impacted Files	Coverage Δ
src/sqlfluff/core/templaters/base.py	100.00% <100.00%> (ø)
src/sqlfluff/core/dialects/__init__.py	100.00% <0.00%> (ø)
src/sqlfluff/dialects/dialect_hive.py	100.00% <0.00%> (ø)
src/sqlfluff/dialects/hive_keywords.py	100.00% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 250fe28...b977d63. Read the comment docs.

[tunetheweb]

Suggesting some changes below.

However think there is more going on as discovered some other oddness when testing multiple for loops in the same file.

For example this fails with an L016 exception:

-- This file combines product data from individual brands into a staging table
{% set products =  [
  'table1',
  'table2'] %}

{% for product in products %}
SELECT
  brand,
  country_code,
  category,
  name,
  id
FROM
  {{ product }};
{% endfor %}

{% for product in products %}
SELECT
  brand,
  country_code,
  category,
  name,
  id
FROM
  {{ product }};
{% endfor %}

Seems to be unrelated to your change (it's broken with main code, with your suggested changes, and also with my suggested further changes to your change) so maybe raise a separate issue for this one?

[tunetheweb]

This is wrong and shouldn't dedent. But not sure why it does. Any ideas @barrywhart ?

[barrywhart]

I don't know exactly, but I do know that SQLFluff is conservative about applying fixes in or near templated areas (i.e. curly braces). Probably related to that. Want to create another issue for that? I'm happy to keep working on these, but trying to avoid letting any single issue become too much of a 🐇 hole.

[barrywhart]

I confirmed just now that it indents correctly when run on a post-expansion version of the file, i.e.:

-- This file combines product data from individual brands into a staging table
SELECT
    brand,
    country_code,
    category,
    name,
    id
FROM
    table1
UNION ALL
SELECT
    brand,
    country_code,
    category,
    name,
    id
FROM
    table2

[barrywhart]

I'll create an issue for the other thing you found. I thought possibly my other PR (#1431) may help with that, but it didn't.

[barrywhart]

Created issue #1433.

I think this PR is ready for another review. Want me to also create an issue for the indentation thing with {{ product }}? It seems like a little lower priority than the L016 exceptions, but happy to do it if you like. 🙏🏽

[tunetheweb]

I'm happy with this if you are @barrywhart . Gave it a thorough review earlier and suggested changes which have been applied.

As discussed, investigations have thrown up another few related issues, but they didn't work previously either and this seems a good improvement.