Add Azure monitor table. closes #38 #70

bigdatasourav · 2021-03-31T06:48:34Z

Integration test logs

Logs

SETUP: tests/azure_log_alert []

PRETEST: tests/azure_log_alert

TEST: tests/azure_log_alert
Running terraform
azurerm_resource_group.named_test_resource: Creating...
azurerm_resource_group.named_test_resource: Creation complete after 3s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340]
azurerm_monitor_action_group.named_test_resource: Creating...
azurerm_storage_account.named_test_resource: Creating...
azurerm_monitor_action_group.named_test_resource: Still creating... [10s elapsed]
azurerm_storage_account.named_test_resource: Still creating... [10s elapsed]
azurerm_monitor_action_group.named_test_resource: Creation complete after 10s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340/providers/microsoft.insights/actionGroups/turbottest86340]
azurerm_storage_account.named_test_resource: Still creating... [20s elapsed]
azurerm_storage_account.named_test_resource: Still creating... [30s elapsed]
azurerm_storage_account.named_test_resource: Creation complete after 32s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340/providers/Microsoft.Storage/storageAccounts/turbottest86340]
azurerm_monitor_activity_log_alert.named_test_resource: Creating...
azurerm_monitor_activity_log_alert.named_test_resource: Still creating... [10s elapsed]
azurerm_monitor_activity_log_alert.named_test_resource: Creation complete after 11s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340/providers/microsoft.insights/activityLogAlerts/turbottest86340]

Warning: Deprecated Resource

The null_data_source was historically used to construct intermediate values to
re-use elsewhere in configuration, the same can now be achieved using locals


Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

object_id = "959adfaa-3632-4f37-85a2-ef002bd72f9e"
resource_aka = "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340/providers/microsoft.insights/activityLogAlerts/turbottest86340"
resource_aka_lower = "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourcegroups/turbottest86340/providers/microsoft.insights/activitylogalerts/turbottest86340"
resource_id = "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340/providers/microsoft.insights/activityLogAlerts/turbottest86340"
resource_name = "turbottest86340"
subscription_id = "d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8"
tenant_id = "cdffd708-7da0-4cea-abeb-0a4c334d7f64"

Running SQL query: test-get-query.sql
[
  {
    "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340/providers/microsoft.insights/activityLogAlerts/turbottest86340",
    "name": "turbottest86340",
    "region": "global",
    "type": "Microsoft.Insights/ActivityLogAlerts"
  }
]
✔ PASSED

Running SQL query: test-hydrate-query.sql
[
  {
    "akas": [
      "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340/providers/microsoft.insights/activityLogAlerts/turbottest86340",
      "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourcegroups/turbottest86340/providers/microsoft.insights/activitylogalerts/turbottest86340"
    ],
    "name": "turbottest86340",
    "title": "turbottest86340"
  }
]
✔ PASSED

Running SQL query: test-list-query.sql
[
  {
    "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340/providers/microsoft.insights/activityLogAlerts/turbottest86340",
    "name": "turbottest86340"
  }
]
✔ PASSED

Running SQL query: test-not-found-query.sql
null
✔ PASSED

Running SQL query: test-turbot-query.sql
[
  {
    "akas": [
      "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest86340/providers/microsoft.insights/activityLogAlerts/turbottest86340",
      "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourcegroups/turbottest86340/providers/microsoft.insights/activitylogalerts/turbottest86340"
    ],
    "name": "turbottest86340",
    "title": "turbottest86340"
  }
]
✔ PASSED

POSTTEST: tests/azure_log_alert

TEARDOWN: tests/azure_log_alert

SUMMARY:

1/1 passed.



SETUP: tests/azure_log_profile []

PRETEST: tests/azure_log_profile

TEST: tests/azure_log_profile
Running terraform
azurerm_resource_group.named_test_resource: Creating...
azurerm_resource_group.named_test_resource: Creation complete after 2s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest77670]
azurerm_eventhub_namespace.named_test_resource: Creating...
azurerm_storage_account.named_test_resource: Creating...
azurerm_eventhub_namespace.named_test_resource: Still creating... [10s elapsed]
azurerm_storage_account.named_test_resource: Still creating... [10s elapsed]
azurerm_eventhub_namespace.named_test_resource: Still creating... [20s elapsed]
azurerm_storage_account.named_test_resource: Still creating... [20s elapsed]
azurerm_eventhub_namespace.named_test_resource: Still creating... [30s elapsed]
azurerm_storage_account.named_test_resource: Still creating... [30s elapsed]
azurerm_storage_account.named_test_resource: Creation complete after 34s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest77670/providers/Microsoft.Storage/storageAccounts/turbottest77670]
azurerm_eventhub_namespace.named_test_resource: Still creating... [40s elapsed]
azurerm_eventhub_namespace.named_test_resource: Still creating... [50s elapsed]
azurerm_eventhub_namespace.named_test_resource: Still creating... [1m0s elapsed]
azurerm_eventhub_namespace.named_test_resource: Still creating... [1m10s elapsed]
azurerm_eventhub_namespace.named_test_resource: Creation complete after 1m17s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest77670/providers/Microsoft.EventHub/namespaces/turbottest77670]
azurerm_monitor_log_profile.named_test_resource: Creating...
azurerm_monitor_log_profile.named_test_resource: Still creating... [10s elapsed]
azurerm_monitor_log_profile.named_test_resource: Still creating... [20s elapsed]
azurerm_monitor_log_profile.named_test_resource: Still creating... [30s elapsed]
azurerm_monitor_log_profile.named_test_resource: Still creating... [40s elapsed]
azurerm_monitor_log_profile.named_test_resource: Still creating... [50s elapsed]
azurerm_monitor_log_profile.named_test_resource: Still creating... [1m0s elapsed]
azurerm_monitor_log_profile.named_test_resource: Creation complete after 1m7s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670]

Warning: Deprecated Resource

The null_data_source was historically used to construct intermediate values to
re-use elsewhere in configuration, the same can now be achieved using locals


Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

object_id = "959adfaa-3632-4f37-85a2-ef002bd72f9e"
resource_aka = "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670"
resource_aka_lower = "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670"
resource_id = "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670"
resource_name = "turbottest77670"
subscription_id = "d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8"
tenant_id = "cdffd708-7da0-4cea-abeb-0a4c334d7f64"

Running SQL query: test-get-query.sql
[
  {
    "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670",
    "name": "turbottest77670"
  }
]
✔ PASSED

Running SQL query: test-hydrate-query.sql
[
  {
    "akas": [
      "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670",
      "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670"
    ],
    "name": "turbottest77670",
    "title": "turbottest77670"
  }
]
✔ PASSED

Running SQL query: test-list-query.sql
[
  {
    "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670",
    "name": "turbottest77670"
  }
]
✔ PASSED

Running SQL query: test-not-found-query.sql
null
✔ PASSED

Running SQL query: test-turbot-query.sql
[
  {
    "akas": [
      "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670",
      "azure:///subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/turbottest77670"
    ],
    "name": "turbottest77670",
    "title": "turbottest77670"
  }
]
✔ PASSED

POSTTEST: tests/azure_log_profile

TEARDOWN: tests/azure_log_profile

SUMMARY:

1/1 passed.

SETUP: tests/azure_diagnostic_setting []

PRETEST: tests/azure_diagnostic_setting

TEST: tests/azure_diagnostic_setting
Running terraform
azurerm_resource_group.named_test_resource: Creating...
azurerm_resource_group.named_test_resource: Creation complete after 3s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest99669]
azurerm_storage_account.named_test_resource: Creating...
azurerm_storage_account.named_test_resource: Still creating... [10s elapsed]
azurerm_storage_account.named_test_resource: Still creating... [20s elapsed]
azurerm_storage_account.named_test_resource: Still creating... [30s elapsed]
azurerm_storage_account.named_test_resource: Creation complete after 35s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbottest99669/providers/Microsoft.Storage/storageAccounts/turbottest99669]
azurerm_monitor_diagnostic_setting.named_test_resource: Creating...
azurerm_monitor_diagnostic_setting.named_test_resource: Creation complete after 7s [id=/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8|turbottest99669]

Warning: Deprecated Resource

The null_data_source was historically used to construct intermediate values to
re-use elsewhere in configuration, the same can now be achieved using locals


Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

object_id = "959adfaa-3632-4f37-85a2-ef002bd72f9e"
resource_aka = "azure://subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticSettings/turbottest99669"
resource_aka_lower = "azure://subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticsettings/turbottest99669"
resource_id = "subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticSettings/turbottest99669"
resource_name = "turbottest99669"
subscription_id = "d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8"
tenant_id = "cdffd708-7da0-4cea-abeb-0a4c334d7f64"

Running SQL query: test-get-query.sql
[
  {
    "id": "subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticSettings/turbottest99669",
    "name": "turbottest99669",
    "type": "Microsoft.Insights/diagnosticSettings"
  }
]
✔ PASSED

Running SQL query: test-hydrate-query.sql
[
  {
    "akas": [
      "azure://subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticSettings/turbottest99669",
      "azure://subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticsettings/turbottest99669"
    ],
    "name": "turbottest99669",
    "title": "turbottest99669"
  }
]
✔ PASSED

Running SQL query: test-list-query.sql
[
  {
    "id": "subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticSettings/turbottest99669",
    "name": "turbottest99669"
  }
]
✔ PASSED

Running SQL query: test-not-found-query.sql
null
✔ PASSED

Running SQL query: test-turbot-query.sql
[
  {
    "akas": [
      "azure://subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticSettings/turbottest99669",
      "azure://subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticsettings/turbottest99669"
    ],
    "name": "turbottest99669",
    "title": "turbottest99669"
  }
]
✔ PASSED

POSTTEST: tests/azure_diagnostic_setting

TEARDOWN: tests/azure_diagnostic_setting

SUMMARY:

1/1 passed.

Example query results

Results

> select
  name,
  jsonb_pretty(activity_log_alert) as activity_log_alert
from
  azure_log_alert;
+-----------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------
| name                                          | activity_log_alert                                                                                                                                        
+-----------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------
| turbot_azure_event_handler_activity_Log_Alert | {                                                                                                                                                         
|                                               |     "scopes": [                                                                                                                                           
|                                               |         "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8"                                                                                             
|                                               |     ],                                                                                                                                                    
|                                               |     "actions": {                                                                                                                                          
|                                               |         "actionGroups": [                                                                                                                                 
|                                               |             {                                                                                                                                             
|                                               |                 "actionGroupId": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/microsoft.insights/actionGroups/t
|                                               |                 "webhookProperties": {                                                                                                                    
|                                               |                     "from": "terraform"                                                                                                                   
|                                               |                 }                                                                                                                                         
|                                               |             }                                                                                                                                             
|                                               |         ]                                                                                                                                                 
|                                               |     },                                                                                                                                                    
|                                               |     "enabled": true,                                                                                                                                      
|                                               |     "condition": {                                                                                                                                        
|                                               |         "allOf": [                                                                                                                                        
|                                               |             {                                                                                                                                             
|                                               |                 "field": "category",                                                                                                                      
|                                               |                 "equals": "Administrative"                                                                                                                
|                                               |             },                                                                                                                                            
|                                               |             {                                                                                                                                             
|                                               |                 "field": "level",                                                                                                                         
|                                               |                 "equals": "Informational"                                                                                                                 
|                                               |             },                                                                                                                                            
|                                               |             {                                                                                                                                             
|                                               |                 "field": "status",                                                                                                                        
|                                               |                 "equals": "Succeeded"                                                                                                                     
|                                               |             }                                                                                                                                             
|                                               |         ]                                                                                                                                                 
|                                               |     },                                                                                                                                                    
|                                               |     "description": ""                                                                                                                                     
|                                               | }                                                                                                                                                         
+-----------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------

> select
  name,
  jsonb_pretty(activity_log_alert) as activity_log_alert
from
  azure_log_alert
where
  jsonb_path_exists(
    activity_log_alert,
    '$.** ? (@.type() == "string" && @ like_regex "Microsoft.Authorization/policyAssignments/write")'
  )
+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| name     | activity_log_alert                                                                                                                                                                             
+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| test12qw | {                                                                                                                                                                                              
|          |     "scopes": [                                                                                                                                                                                
|          |         "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/DefaultResourceGroup-EUS"                                                                                          
|          |     ],                                                                                                                                                                                         
|          |     "actions": {                                                                                                                                                                               
|          |         "actionGroups": [                                                                                                                                                                      
|          |             {                                                                                                                                                                                  
|          |                 "actionGroupId": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourcegroups/defaultresourcegroup-wus/providers/microsoft.insights/actiongroups/application insights sm
|          |                 "webhookProperties": {                                                                                                                                                         
|          |                 }                                                                                                                                                                              
|          |             }                                                                                                                                                                                  
|          |         ]                                                                                                                                                                                      
|          |     },                                                                                                                                                                                         
|          |     "enabled": true,                                                                                                                                                                           
|          |     "condition": {                                                                                                                                                                             
|          |         "allOf": [                                                                                                                                                                             
|          |             {                                                                                                                                                                                  
|          |                 "field": "category",                                                                                                                                                           
|          |                 "equals": "Administrative"                                                                                                                                                     
|          |             },                                                                                                                                                                                 
|          |             {                                                                                                                                                                                  
|          |                 "field": "resourceType",                                                                                                                                                       
|          |                 "equals": "microsoft.authorization/policyassignments"                                                                                                                          
|          |             },                                                                                                                                                                                 
|          |             {                                                                                                                                                                                  
|          |                 "field": "operationName",                                                                                                                                                      
|          |                 "equals": "Microsoft.Authorization/policyAssignments/write"                                                                                                                    
|          |             }                                                                                                                                                                                  
|          |         ]                                                                                                                                                                                      
|          |     },                                                                                                                                                                                         
|          |     "description": ""                                                                                                                                                                          
|          | }                                                                                                                                                                                              
+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


> select jsonb_pretty(diagnostic_settings) as diagnostic_settings from azure_diagnostic_setting
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| diagnostic_settings                                                                                                                                           |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {                                                                                                                                                             |
|     "logs": [                                                                                                                                                 |
|         {                                                                                                                                                     |
|             "enabled": true,                                                                                                                                  |
|             "category": "Administrative"                                                                                                                      |
|         },                                                                                                                                                    |
|         {                                                                                                                                                     |
|             "enabled": true,                                                                                                                                  |
|             "category": "Security"                                                                                                                            |
|         },                                                                                                                                                    |
|         {                                                                                                                                                     |
|             "enabled": false,                                                                                                                                 |
|             "category": "ServiceHealth"                                                                                                                       |
|         },                                                                                                                                                    |
|         {                                                                                                                                                     |
|             "enabled": true,                                                                                                                                  |
|             "category": "Alert"                                                                                                                               |
|         },                                                                                                                                                    |
|         {                                                                                                                                                     |
|             "enabled": false,                                                                                                                                 |
|             "category": "Recommendation"                                                                                                                      |
|         },                                                                                                                                                    |
|         {                                                                                                                                                     |
|             "enabled": true,                                                                                                                                  |
|             "category": "Policy"                                                                                                                              |
|         },                                                                                                                                                    |
|         {                                                                                                                                                     |
|             "enabled": false,                                                                                                                                 |
|             "category": "Autoscale"                                                                                                                           |
|         },                                                                                                                                                    |
|         {                                                                                                                                                     |
|             "enabled": false,                                                                                                                                 |
|             "category": "ResourceHealth"                                                                                                                      |
|         }                                                                                                                                                     |
|     ],                                                                                                                                                        |


> select name,id,type from azure_diagnostic_setting
+--------+-----------------------------------------------------------------------------------------------------------+---------------------------------------+
| name   | id                                                                                                        | type                                  |
+--------+-----------------------------------------------------------------------------------------------------------+---------------------------------------+
| test12 | subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/diagnosticSettings/test12 | Microsoft.Insights/diagnosticSettings |
+--------+-----------------------------------------------------------------------------------------------------------+---------------------------------------+

> select * from azure_log_profile
+---------+------------------------------------------------------------------------------------------------------+--------+----------+----------------------------------------------------------------------
| name    | id                                                                                                   | type   | location | storage_account_id                                                   
+---------+------------------------------------------------------------------------------------------------------+--------+----------+----------------------------------------------------------------------
| default | /subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/providers/microsoft.insights/logprofiles/default | <null> | <null>   | /subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/tu
+---------+------------------------------------------------------------------------------------------------------+--------+----------+----------------------------------------------------------------------

… into issue-38

azure/table_azure_location.go

azure/table_azure_diagnostic_setting.go

azure/table_azure_log_alert.go

azure/table_azure_log_profile.go

azure-test/tests/azure_log_alert/variables.tf

azure-test/tests/azure_log_profile/variables.tf

Subhajit97

Make suggested changes

azure/table_azure_log_alert.go

azure/table_azure_log_profile.go

azure-test/tests/azure_diagnostic_setting/variables.tf

…issue-38

azure/table_azure_diagnostic_setting.go

azure/table_azure_log_alert.go

docs/tables/azure_diagnostic_setting.md

docs/tables/azure_log_alert.md

Subhajit97

@bigdatasourav There are few more changes required.

…-azure into issue-38

azure/table_azure_diagnostic_setting.go

azure/table_azure_log_profile.go

Subhajit97

LGTM

rajlearner17

LGTM

rajlearner17

LGTM

bigdatasourav added 4 commits February 9, 2021 17:52

sample test

82e9e04

log_alert table

1ba188a

required tables

f8f3dce

integration test

ff3a756

bigdatasourav requested a review from Subhajit97 March 31, 2021 06:48

bigdatasourav added 3 commits March 31, 2021 12:48

doc update

731ded5

Merge branch 'main' of https://github.com/turbot/steampipe-plugin-azure…

2da6ebf

… into issue-38

changes

e47e528

bigdatasourav self-assigned this Mar 31, 2021

test changes

7934ca9

bigdatasourav linked an issue Mar 31, 2021 that may be closed by this pull request

Add Azure monitor table #38

Closed