Cost Reporting #450
Comments
|
would it be possible also to measure how much % workspace used from core shared services and split costs for them ? |
|
Looking at the API calls I think there needs to be the inclusion of one to return the cost of the costs of the TRE Core. This would be useful from the point of view of service providers so that they can easily see the costs. In the Workspace related calls we need to make sure we return detailed information about the breakdown as there may need to be subsequent filtering depending on any reserved instance provision within the environment. |
|
Here it follows a reviewed set on goals and a suggestions for user stories: Main Goals:Provide general cost reporting capability to a TRE instance Draft user stories:As a TRE Admin, As a TRE Admin or a Workspace Owner As a TRE Admin/Workspace Owner ImplementationBesides the provided suggestions, if implemented in the |
|
Related: #417 |
Design ConsiderationsAzure REST API provides two types of Cost APIs, Azure Cost Management API and Azure Consumption API. The Azure Cost Management API provide the ability to explore cost and usage data via multidimensional analysis, where creating customized filters and expressions allow you to answer consumption-related questions for your Azure resources. These APIs are currently available for Azure Enterprise customers. The Azure Consumption API give you programmatic access to cost and usage data for your Azure resources. These APIs currently only support Enterprise Enrollments and Web Direct Subscriptions (with a few exceptions). The APIs are continually updated to support other types of Azure subscriptions. Azure TRE currently deploys workspace services and user resources inside the same resource group per workspace. Azure TRE API is aware of TRE entity types (workspace, workspace service, user resource), underling azure resources are visible only for resource processor. Azure Cost Management API supports querying with different types of filtering, aggregation and grouping capabilities and different timeframes:
Cost Management API requiers "Billing Reader" Role Assignment to be called. Azure Consumption API (Usage Details)
Design Options
Option 4 is the prefered option. Tagging resources upon creationCost report API will rely on Azure tagging to be able to generate cost report for core, shared, workspace, workspace services and user resources. There are two options to be decided for tagging: Option 1 - allows support for built in budget in azure, but more error prone:
Option 2: each shared service, workspace service and user resource will get tag "tre_resource_id" with the value of its id when it get generated. TRE Admin Cost Reporting - overall cost reportDescriptionGet overall cost report for TRE instance (core services), per shared services and per workspace AuthorizationOnly TRE Admin can call this url, others will get unauthorized InputPOST /costs {
"timePeriod": {"from":"MM/DD/YYYY","to":"MM/DD/YYYY"}, // up to 1 year timespan
"granularity":"Daily|None" // None will summarize for the timefame
}Output{
"coreServices": [
{"date":"", "cost":"", "currency":""} // date will return only for Daily granularity
],
"sharedServices": [ {
"id":"shared service id",
"name":"shared service name",
"costs": [
{"date":"", "cost":"", "currency":""} // date will return only for Daily granularity
]
}],
"workspaces": [ {
"id": "workspace id",
"name": "workspace name",
"costs":[
{"date":"", "cost":"", "currency":""} // date will return only for Daily granularity
]}
]
}sequence diagram
Get resource ids by tagOutput example: {
"value": [
{
"id": "/subscriptions/xxxx/resourceGroups/rg-anatbal6-ws-08cc/providers/Microsoft.Network/networkInterfaces/internal-nic-anatbal6-ws-08cc-svc-71d7",
"name": "internal-nic-anatbal6-ws-08cc-svc-71d7",
"type": "Microsoft.Network/networkInterfaces",
"location": "eastus",
"tags": {
"workspace_id": "34f28ac6-55a5-4c2f-9d81-22dda52a08cc",
"workspace_service_id": "71d7"
}
},
{
"id": "/subscriptions/xxxx/resourceGroups/rg-anatbal6-ws-08cc/providers/Microsoft.Compute/virtualMachines/windowsvm71d7",
"name": "windowsvm71d7",
"type": "Microsoft.Compute/virtualMachines",
"location": "eastus",
"identity": {
"principalId": "45fd0951-08c5-4e49-a704-5227edb18599",
"tenantId": "72f988bf-86f1-41af-91ab-2d7cd011db47",
"type": "SystemAssigned"
},
"tags": {
"parent_service_id": "e67c8bf9-442f-406b-b40c-e482f190c1f1",
"workspace_id": "34f28ac6-55a5-4c2f-9d81-22dda52a08cc",
"workspace_service_id": "71d7"
}
},
]
}Notice that each resource matching query is returned with all its tags. Cost management API payloadPayload: {
"type": "Usage",
"timeframe": "Custom", // can be also BillingMonthToDate|MonthToDate|TheLastBillingMonth|TheLastMonth|WeekToDate according to input
"timePeriod": {
"from": "MM/DD/YYYY",
"to": "MM/DD/YYYY"
},
"dataset": {
"granularity": "Daily", // can be also "None" for total costs
"aggregation": {
"totalCost": {
"name": "Cost",
"function": "Sum"
}
},
"filter": {
"dimensions": {
"name": "ResourceId",
"operator": "In",
"values": [
/// List of Azure resource ids as strings
]
}
}
}
}
Output example: {
"id": "subscriptions/xxxx/providers/Microsoft.CostManagement/query/yyyy",
"name": "yyy",
"type": "Microsoft.CostManagement/query",
"location": null,
"sku": null,
"eTag": null,
"properties": {
"nextLink": null,
"columns": [
{
"name": "Cost",
"type": "Number"
},
{
"name": "UsageDate",
"type": "Number"
},
{
"name": "Currency",
"type": "String"
}
],
"rows": [
[
0.00032930107526881738,
20211208,
"USD"
],
[
0.0011290322580645167,
20211209,
"USD"
],
[
0.0011290322580645167,
20211210,
"USD"
],
]
}
}TRE Admin / Workspace Owner Cost Reporting - specific workspace cost reportDescriptionGet overall cost report for a specific workspace AuthorizationOnly TRE Admin and the workspace owner can call this url, others will get unauthorized InputPOST /workspace/{workspace_id}/costs {
"timePeriod": {"from":"MM/DD/YYYY","to":"MM/DD/YYYY"}, // up to 1 year timespan
"granularity":"Daily|None" // None will summarize for the timefame
}Output{
"sharedServices": [ {
"sharedServiceId":"",
"costs": [
{"date":"", "cost":"", "currency":""} // date will return only for Daily granularity
]
}],
"workspace": {
"id": "workspace id",
"name": "workspace name",
"workspaceServices":[{
"id": "workspace service id",
"name": "workspace service name",
"costs":[{"date":"", "cost":"", "currency":""}], // date will return only for Daily granularity
"userResources":[{
"id": "user resource id",
"name": "user resource name",
"costs":[{"date":"", "cost":"", "currency":""}], // date will return only for Daily granularity
}]
}]
}
}Sequence diagram
|
|
@guybartal does option 3 means you get a cost report per resource? is that what group by resource id means? if so, I think that getting the resource cost is too fine grained, I think that the ws service or ws user resource level is better. |
Thanks for raising this @eladiw, I changed "Group by" to "Filter By", it was a typo. I meant that the cost will be daily / total for a period and get filtered by a set of resources, it doen't make sense to group by a resource. |
|
looks good. I like the fact you dont assume to keep track of the resource ids, and ask them from the management api |
Status
Design ConsiderationsAzure REST API provides two types of Cost APIs, Azure Cost Management API and Azure Consumption API. The Azure Cost Management API provide the ability to explore cost and usage data via multidimensional analysis, where creating customized filters and expressions allow you to answer consumption-related questions for your Azure resources. These APIs are currently available for Azure Enterprise customers. The Azure Consumption API give you programmatic access to cost and usage data for your Azure resources. These APIs currently only support Enterprise Enrollments and Web Direct Subscriptions (with a few exceptions). The APIs are continually updated to support other types of Azure subscriptions. Azure TRE currently deploys workspace services and user resources inside the same resource group per workspace. Azure TRE API is aware of TRE entity types (workspace, workspace service, user resource), underling azure resources are visible only for resource processor. Cost and usage data is typically available within 8-24 hours and budgets are evaluated against these costs every 24 hours. Azure Cost Management API supports querying with different types of filtering, aggregation and grouping capabilities and different timeframes:
Cost Management API requiers "Billing Reader" Role Assignment to be called. Consumption API (Usage Details)
Design Options
Option 2 is the prefered option. Tagging resources upon creationCost report API will rely on Azure tagging to be able to generate cost report for core, shared, workspace, workspace services and user resources.
main ACR and Storage Account will not be tagged as those resources are used to spin up more than one Azure TRE Instance. Tagging this way will allow to define budgets per TRE Resource. TRE Admin Cost Reporting - overall cost reportDescriptionGet overall cost report for TRE instance (core services), per shared services and per workspace AuthorizationOnly TRE Admin can call this url, others will get unauthorized InputGET /api/costs?from=MM/DD/YYYY&to=MM/DD/YYYY&granularity=Daily Parameters:
Output200 OK When granularity = Daily: {
"coreServices": [
{"date":"", "cost":"", "currency":""}
],
"sharedServices": [ {
"id":"shared service id",
"name":"shared service name",
"costs": [
{"date":"", "cost":"", "currency":""}
]
}],
"workspaces": [ {
"id": "workspace id",
"name": "workspace name",
"costs":[
{"date":"", "cost":"", "currency":""}
]}
]
}
When granularity = None: {
"coreServices":
[{"cost":"", "currency":""}],
"sharedServices": [ {
"id":"shared service id",
"name":"shared service name",
"costs": [{"cost":"", "currency":""}]
}],
"workspaces": [ {
"id": "workspace id",
"name": "workspace name",
"costs":[{"cost":"", "currency":""}]
}]
}
TRE Admin / Workspace Owner Cost Reporting - specific workspace cost reportDescriptionGet overall cost report for a specific workspace AuthorizationOnly TRE Admin and the workspace owner can call this url, others will get unauthorized InputGET /api/workspaces/{workspace_id}/costs?from=MM/DD/YYYY&to=MM/DD/YYYY&granularity=Daily Parameters:
Output200 OK When granularity = Daily: {
"id": "workspace id",
"name": "workspace name",
"workspaceServices":[{
"id": "workspace service id",
"name": "workspace service name",
"costs":[{"date":"", "cost":"", "currency":""}],
"userResources":[{
"id": "user resource id",
"name": "user resource name",
"costs":[{"date":"", "cost":"", "currency":""}],
}]
}]
}When granularity = None: {
"id": "workspace id",
"name": "workspace name",
"workspaceServices":[{
"id": "workspace service id",
"name": "workspace service name",
"costs":{"cost":"", "currency":""},
"userResources":[{
"id": "user resource id",
"name": "user resource name",
"costs":[{"cost":"", "currency":""}]
}]
}]
}TRE Researcher Cost Reporting - specific user resource cost reportDescriptionGet cost report for a specific user resource AuthorizationOnly TRE researcher and the workspace owner, and tre admin can call this url, others will get unauthorized InputGET /api/workspaces/{workspace_id}/service/{workspace_service_id}/resource/{user_resource_id}/costs?from=MM/DD/YYYY&to=MM/DD/YYYY&granularity=Daily Parameters:
Output200 OK When granularity = Daily: {
"id": "user resource id",
"name": "user resource name",
"costs":[{"date":"", "cost":"", "currency":""}]
}When granularity = None: {
"id": "user resource id",
"name": "user resource name",
"costs":[{"cost":"", "currency":""}]
}Sequence Diagrams Cost management Query APIQuery API will return a flat result of all cost with every tag combination which exists on the filtered resources by the provided tag, meaning that a resource which has tre_id, tre_workspace_id, and tre_workspace_service_id will get be summerized in all three tags. calls to this API shoud be cached for X hours to reduce load and have better performance. exampleGroup costs by tag name and filter only resources which have this tag, Payload: {
"type": "Usage",
"timeframe": "Custom", // can be also BillingMonthToDate|MonthToDate|TheLastBillingMonth|TheLastMonth|WeekToDate according to input
"timePeriod": {
"from": "{{TIME_FROM}}",
"to": "{{TIME_TO}}"
},
"dataset": {
"granularity": "Daily", // can be also "None" for total costs
"aggregation": {
"totalCost": {
"name": "Cost",
"function": "Sum"
}
},
"filter": {
"tags": {
"name": "tre_id",
"operator": "In",
"values" : [
"{{TRE_ID}}",
]
}
},
"grouping": [
{
"type": "Tag",
}
]
}
}
Output example: {
"id": "subscriptions/xxxxxxxxxxxxx/providers/Microsoft.CostManagement/query/ec1f0eae-9343-4f4e-a0c0-dd219a66efc4",
"name": "ec1f0eae-9343-4f4e-a0c0-dd219a66efc4",
"type": "Microsoft.CostManagement/query",
"location": null,
"sku": null,
"eTag": null,
"properties": {
"nextLink": null,
"columns": [
{
"name": "Cost",
"type": "Number"
},
{
"name": "Tag",
"type": "String"
},
{
"name": "Currency",
"type": "String"
}
],
"rows": [
[
3.9306515711531222,
"\"tre_core_service\":\"guy21\"",
"USD"
],
[
11.66335497490112,
"\"tre_id\":\"guy21\"",
"USD"
],
[
0.033,
"\"tre_shared_service\":\"2fea\"",
"USD"
],
[
3.62175702964083,
"\"tre_shared_service\":\"4a5b\"",
"USD"
],
[
0.093523886643774659,
"\"tre_user_resource_id\":\"126d\"",
"USD"
],
[
0.078465743393993009,
"\"tre_user_resource_id\":\"2627\"",
"USD"
],
[
0.20275980676694544,
"\"tre_user_resource_id\":\"319e\"",
"USD"
],
[
0.17165788614506686,
"\"tre_user_resource_id\":\"3370\"",
"USD"
],
[
0.17518017912599823,
"\"tre_user_resource_id\":\"b2be\"",
"USD"
],
[
0.39905729127776768,
"\"tre_workspace_id\":\"5e86\"",
"USD"
],
[
0.98272254462049369,
"\"tre_workspace_id\":\"af30\"",
"USD"
],
[
0.17198963003776765,
"\"tre_workspace_service\":\"8d0a\"",
"USD"
],
[
0.54959787203801058,
"\"tre_workspace_service\":\"e70d\"",
"USD"
]
]
}
} |
|
@eladiw @tamirkamara @joalmeid @marrobi, |
|
@guybartal while adding tags, can everything be tagged with Also can we use the format Also not necessarily part of v1, but is there any way we can check that resources that have just been deployed have appropriate tags? Maybe check all resources deployed by the resource processor in the RG have a Thanks. |
Thanks for commenting on this @marrobi, acctually I was counting on this tag for better perfomance upon calling the Cost Management API... I've updated the design. |
|
@joalmeid @marrobi @eladiw @tamirkamara @LizaShak |
|
@guybartal @joalmeid can we close this feature is it's done? |
Administrators, PMs and Researchers need to have a clear view on spend and know when hitting budget limits.
Description
As a TRE Administrator I want to see the Azure spend for each workspace by custom time frame so that I can understand how much I have spent.
As a Workspace Owner I want to see the Azure spend for my workspace by custom time frame so I can see how much I have spent
As a Workspace Owner I want to see the Azure spend for my workspace split by workspace service
As a Workspace Owner I want to be notified when my workspace has hit its budget limit
Acceptance Criteria
Feature Design
Implementation
The text was updated successfully, but these errors were encountered: