Build a graph of resources. Use it to detect circular dependencies

[miparnisari]

Changes:

• Use networkx to model the template as a graph of resources
• Use this graph to check for cycles in rule E3004
• Fix Rule E3004 is not checking for DependsOn declarations #1386 by adding DependsOn edges to the graph

While debugging I found it very useful to use networkx.drawing.nx_pydot.write_dot(self.graph, path) which creates a visual representation of the graph. I think it would be useful for customers as well. But I wasn't sure whether to include it in this PR because it needs another dependency, pydot and it would need some work to make it a fancy graph.

[PatMyron]

While debugging I found it very useful to use networkx.drawing.nx_pydot.write_dot(self.graph, path) which creates a visual representation of the graph. I think it would be useful for customers as well. But I wasn't sure whether to include it in this PR because it needs another dependency, pydot and it would need some work to make it a fancy graph.

Agreed, I think just documenting how to do that here is a nice compromise:

add networkx.drawing.nx_pydot.write_dot(self.graph, 'graph') to the end of __init__(self, cfn) in src/cfnlint/graph.py

cfn-python-lint $ pip3 install pydot       # install pydot
cfn-python-lint $ pip3 install -e . --user # install locally modified version of cfn-lint
cfn-python-lint $ cfn-lint template.yaml   # lint desired template
cfn-python-lint $ pbcopy < graph           # copy graph to paste somewhere like http://www.webgraphviz.com/ or https://sketchviz.com/

[PatMyron]

Will need to vet new dependencies

[miparnisari]

In terms of license we should be ok: https://tldrlegal.com/license/bsd-3-clause-license-(revised)

[PatMyron]

Nice! From testing, Fn::GetAtt's long form is the last YAML template dependency I could find that's still undetected:

  Resource5:
    Type: AWS::SNS::Topic
    Properties:
      DisplayName:
        Fn::GetAtt: Resource6.TopicName # dependency only caught in !GetAtt form
        # !GetAtt Resource6.TopicName
  Resource6:
    Type: AWS::SNS::Topic
    Properties:
      DisplayName:
        Ref: Resource5

Pushed to your branch if you want to use for testing

[benbridts]

Agreed, I think just documenting how to do that here is a nice compromise:

Another option is to put it behind a command or option and test for the existence of pydot if that's used.

try:
    import pydot
except ImportError:
    # give some feedback here