Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable usage logs to track the downloads for artifacts in knative-releases project #3231

Open
chizhg opened this issue Apr 3, 2022 · 3 comments
Labels
area/gcp Issues or PRs related to Google Cloud Infrastructure lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@chizhg
Copy link
Member

chizhg commented Apr 3, 2022

Currently, the network egress cost for knative-releases project is ~$1k per month, and it's because this project hosts all the Knative release artifacts in GCS and GCR, and all the Knative users will download them from it.

As there are more and more Knative users, the cost for it will keep growing. To get a better understanding on where the costs come from and also for potential future improvements, it'll better track the usage for the Knative releases. Usage logs is the recommended solution to do this - https://cloud.google.com/storage/docs/access-logs

/cc @upodroid

@chizhg
Copy link
Member Author

chizhg commented Apr 3, 2022

Enabled by running below commands:

$ gsutil mb gs://knative-releases-analysis -p knative-releases
$ gsutil iam ch group:cloud-storage-analytics@google.com:legacyBucketWriter gs://knative-releases-analysis
$ gsutil logging set on -b gs://knative-releases-analysis gs://knative-releases gs://artifacts.knative-releases.appspot.com gs://asia.artifacts.knative-releases.appspot.com gs://eu.artifacts.knative-releases.appspot.com gs://us.artifacts.knative-releases.appspot.com

@chizhg
Copy link
Member Author

chizhg commented Apr 4, 2022

Next steps will be finishing Analyze logs in BigQuery and probably writing a tool to aggregate the data to see what are the main sources for these requests, but I don't have bandwidth to work on it now.

I have run gsutil logging set off gs://knative-releases-analysis gs://knative-releases gs://artifacts.knative-releases.appspot.com gs://asia.artifacts.knative-releases.appspot.com gs://eu.artifacts.knative-releases.appspot.com gs://us.artifacts.knative-releases.appspot.com to disable the usage logs for now. If anyone is interested in this work, please feel free to pick it up and continue.

/cc @kvmware

@chizhg chizhg changed the title Enable usage logs to track the downloads from knative-releases project Enable usage logs to track the downloads for artifacts in knative-releases project Apr 6, 2022
@chizhg chizhg moved this to In Design/Discussion in Infra (Productivity) Apr 7, 2022
@krsna-m krsna-m assigned krsna-m and unassigned krsna-m May 11, 2022
@upodroid upodroid added area/gcp Issues or PRs related to Google Cloud Infrastructure priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jun 15, 2022
@krsna-m krsna-m moved this from In Design/Discussion to Ready To Work in Infra (Productivity) Jun 15, 2022
@github-actions
Copy link

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 14, 2022
@upodroid upodroid added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/gcp Issues or PRs related to Google Cloud Infrastructure lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
Status: Ready To Work
Development

No branches or pull requests

3 participants