Requiring existing resources to be imported / support for timeouts

[tombuildsstuff]

This PR fixes a couple of cross-resource bugs in the Provider:

• Firstly this works around the fact most of the Azure API's are Upserts by requiring that resources are present in the state before creating them
• This adds support for Timeouts, which fixes Resource Timeouts for creation / deletion #171

This is being done in one hit so it's consistent, rather than being inconsistent across resources.

Resource Support

• App Service
• Application Gateway
• Application Insights
• Application Security Group
• Automation
• Auto Scale
• Availability Set
• AzureAD
• CDN
• Container Group
• Container Registry
• Container Service
• CosmosDB
• Data Lake
• DNS
• EventGrid
• EventHub
• Express Route
• Function App
• Image
• IoTHub
• Key Vault
• Kubernetes
• Local Network Gateway
• Log Analytics
• Managed Disks
• Management Locks
• Monitor
• MySQL
• Networks
• Notification Hub
• Packet Capture
• Policy
• Postgres
• Public IP
• Recovery Services
• Redis
• Relay
• Roles
• Routes
• Scheduler Job
• Search
• Service Fabric
• ServiceBus
• Snapshot
• SQL
• Subnets
• Traffic Manager
• User Assigned Identity
• Virtual Network Gateways

---

virtual resources are slightly different:

• Load Balancer
• Logic Apps

requires more extensive testing:

• Storage
• Template Deployments
• Virtual Machines
• Virtual Machine Disk Attachments
• Virtual Machine Extentions
• VM Scale Sets

needs to be done last / in this order:

• Networks
• Resource Groups

---

other

• remove the default timeout in config.go
• removes a bunch of crash points in ACS
• Data Lake Store File ID dependent on Data Lake Store File: updating the ID #1856
• azurerm_dns_cname_record - removing the removed records field
• Dependent on Storage: Import Support #1816 merged
• Upgrade to AutoRest v11
• Update to include latest PR's once merged:
  • API Management
  • Automation DSC Configuration
  • Automation DSC Node Configuration
  • Automation Module
  • Cognitive Services Account
  • Databricks Workspace
  • DevSpace Controller
  • DevTest Lab (Data Source & Resource)
  • DevTest Linux Virtual Machine
  • DevTest Policy
  • DevTest Virtual Network
  • DevTest Windows Virtual Machine
  • Firewall
  • Firewall Network Rule Collection
  • Log Analytics Workspace Linked Service
  • Management Groups
  • Monitor Activity Log Alert
  • Monitor Metric Alert
  • Monitor Log Profile
  • MySQL Virtual Network Rule
  • Network Interface <-> Application Gateway Backend Address Pool Association
  • Network Interface <-> Load Balancer Backend Address Pool Association
  • Network Interface <-> Load Balancer NAT Rule Association
  • PostgreSQL Virtual Network Rule
  • Recovery Services Protection Policy VM
  • Recovery Services Protected VM
  • Security Center Contact
  • Security Center Subscription Pricing
  • Security Center Workspace
  • Shared Image (Data Source & Resource)
  • Shared Image Gallery (Data Source & Resource)
  • Shared Image Version (Data Source & Resource)
  • Subnet <-> Network Security Group Association
  • Subnet <-> Route Table Association

[metacpp]

Hey @tombuildsstuff

Considering that this PR contains so many files (~100), which will have big chance (see below conflicts) to conflict with other merged PRs in the near future. Will it be better to split such kind of mega PR into smaller ones and iterate fast? Just my 2 cents.

[metacpp]

  		return fmt.Errorf(" [](start = 0, length = 23)

Move this error message as template function in errors package?

[tombuildsstuff]

this varies wildly by resource, so this doesn't make sense to refactor out

[tombuildsstuff]

@metacpp

Considering that this PR contains so many files (~100), which will have big chance (see below conflicts) to conflict with other merged PRs in the near future. Will it be better to split such kind of mega PR into smaller ones and iterate fast?

From the description:

This is being done in one hit so it's consistent, rather than being inconsistent across resources.

I'm rebasing this PR at least once a day - but unfortunately this needs to be applied to all resources at once since it's a behavioural change. Whilst we could potentially merge this in with a feature flag to disable it, we'd have to modify every file to remove that feature flag at some point anyway, so I don't see that as a beneficial option in this case

We don't often have cross-cutting PR's for this reason, but behavioural changes (such as this one, and also #1728) unfortunately sometimes warrant larger PR's so the UX is consistent.

[tombuildsstuff]

Closing this in favour of #2511

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!