Skip to content

Commit

Permalink
docs: add telemetry documentation (#35)
Browse files Browse the repository at this point in the history
* feat: add telemetry documentation

fixes #29
fixes ipfs/ipfs-gui#125

* docs: fix markdown link to FAQs document

* docs: add FAQ for turning telemetry completely off

* docs: minor updates about user data

* docs: add UI examples and steps for adding telemetry to repo

* docs: reordering and rephrasing

* Update CollectedData.template.md

* Update docs/telemetry/CollectionPolicy.md

Co-authored-by: Nishant Arora <1895906+whizzzkid@users.noreply.github.com>

* Update docs/telemetry/CollectionPolicy.md

Co-authored-by: Nishant Arora <1895906+whizzzkid@users.noreply.github.com>

* Update docs/telemetry/CollectionPolicy.md

Co-authored-by: Nishant Arora <1895906+whizzzkid@users.noreply.github.com>

* Update docs/telemetry/CollectionPolicy.md

Co-authored-by: Nishant Arora <1895906+whizzzkid@users.noreply.github.com>

* Update docs/telemetry/CollectionPolicy.md

Co-authored-by: Nishant Arora <1895906+whizzzkid@users.noreply.github.com>

* docs: additional clarifications and updates

* docs: more updates

* Update README.md

Co-authored-by: Marcin Rataj <lidel@lidel.org>

* Update docs/telemetry/CollectedData.template.md

Co-authored-by: Marcin Rataj <lidel@lidel.org>

* Update docs/telemetry/CollectionPolicy.md

Co-authored-by: Marcin Rataj <lidel@lidel.org>

* Update docs/telemetry/CollectionPolicy.md

Co-authored-by: Marcin Rataj <lidel@lidel.org>

* chore: update countly url

* chore(docs): update collectedData.template links

* chore: use UPPER_SNAKE_CASE for telemetry docs

* Update docs/telemetry/FAQs.md

Co-authored-by: Marcin Rataj <lidel@lidel.org>

* chore(docs): fix links

* chore(docs): add ipfs-gui team under lead maintainer

* chore: remove multi-toggle images

* chore: grammar

* chore: update example screenshot of ipfs-companion

* fix: exports

Co-authored-by: Nishant Arora <1895906+whizzzkid@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
  • Loading branch information
3 people authored Jan 13, 2023
1 parent 4859bc2 commit 4cb35ba
Show file tree
Hide file tree
Showing 15 changed files with 290 additions and 18 deletions.
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,21 @@ A longer repository description.

## Documentation

[Insert link to documentation]() or expand with Install, Build, Usage sections.
### Telemetry

- [Telemetry collection policy](./docs/telemetry/COLLECTION_POLICY.md)
- [Privacy Policy](./docs/telemetry/PRIVACY_POLICY.md)
- [FAQs](./docs/telemetry/FAQs.md)

## Lead Maintainer

[Russell Dempsey](https://github.com/SgtPooki)

## Team

[IPFS-Ignite(ipfs-gui) team in ipfs-shipyard](https://github.com/orgs/ipfs-shipyard/teams/ipfs-gui)
[IPFS-Ignite(ipfs-gui) team in ipfs](https://github.com/orgs/ipfs/teams/gui-dev)

## Contributing

Contributions are welcome! This repository is part of the IPFS project and therefore governed by our [contributing guidelines](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md).
Expand Down
41 changes: 41 additions & 0 deletions docs/telemetry/COLLECTED_DATA.template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Collected telemetry data

Copied from https://github.com/ipfs-shipyard/ignite-metrics/blob/main/docs/telemetry/COLLECTED_DATA.template.md

<!--
This is the default for projects using the ignite-metrics library. Each project's own `./docs/telemetry/COLLECTED_DATA.md` file should contain very specific information under each of the following sections:
* 'What metrics data DO we collect'
-->
## What metrics data do we NOT collect

We do not collect any of the below listed items in our metric data:

1. User identifiable data (names, email addresses, aliases, handles, etc.)
2. CIDs
3. IP addresses

Telemetry is sent to Countly instance at `countly.ipfs.tech`. You can read how they protect your data at [Security, Privacy, and Access FAQ](https://support.count.ly/hc/en-us/articles/360037501372-Security-Privacy-and-Access-FAQ)

## What metrics data DO we collect

As a general rule, we collect only application data; no user data. Some examples of application data we collect are:

| Metric data name | Metric feature name | Metric trigger | Analytics use | Notes |
|:----------------:|---------------------|----------------|---------------|---------------|
| Fill in | Fill in | Fill in | Fill in | Fill in |
| | | | | |
| | | | | |

* "Metric data name" - The app-specific metric/event name we're using for this metric data. (e.g. APP_BOOTSTRAP_START)
* "Metric feature name" - The metric feature the event/metric data correlates to. The group the metric feature belongs to is defined in our [COLLECTION_POLICY](https://github.com/ipfs-shipyard/ignite-metrics/blob/main/docs/telemetry/COLLECTION_POLICY.md#metric-features-and-their-groupings). (e.g. Minimal)
* "Metric trigger" - An explanation covering when this metric data is triggered. (e.g. On Application init)
* "Analytics use" - An explanation about how this metric data is used for analytics. (e.g. Input to load time calculations)
* "Notes" - Any additional notes. (e.g. Used as a timestamp identifier for when an application is first loaded)

## Other related documents

* [COLLECTION_POLICY](https://github.com/ipfs-shipyard/ignite-metrics/blob/main/docs/telemetry/COLLECTION_POLICY.md)
* [PRIVACY_POLICY](https://github.com/ipfs-shipyard/ignite-metrics/blob/main/docs/telemetry/PRIVACY_POLICY.md)
* [FAQs](https://github.com/ipfs-shipyard/ignite-metrics/blob/main/docs/telemetry/FAQs.md)
138 changes: 138 additions & 0 deletions docs/telemetry/COLLECTION_POLICY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# Telemetry collection policy

This document is the source of truth for our telemetry policies and practices across all Ignite team (IPFS GUI) projects. The discussion held at https://github.com/ipfs/ipfs-gui/issues/125 is the origin and inspiration for this library and all documents under `./docs/telemetry`.

The initial list of projects we're adding telemetry to was originally discussed at https://github.com/ipfs/ipfs-gui/issues/105.

For an up to date list of projects we own that follow this policy, please see:

* https://github.com/ipfs/ipfs-gui#primary-projects
* https://github.com/ipfs/ipfs-gui#other-ipfs-gui--tools-owned-projects

## History

Before 2022Q4, telemetry was only implemented in https://github.com/ipfs/ipfs-webui and https://github.com/ipfs/ipfs-desktop, using our [self-hosted countly](https://count.ly/community-edition) server. The telemetry implemented within ipfs-webui and ipfs-desktop was opt-in only. i.e. metrics data was sent only if a user specifically opted-in to collecting metrics.

Our other projects did not implement telemetry or collection of any metrics data. The only exception was https://github.com/ipfs/ipfs-companion where we have had a [privacy policy since 2019-02-15](https://github.com/ipfs/ipfs-companion/blob/main/PRIVACY-POLICY.md). The privacy policy in ipfs-companion indicates that no data is collected whatsoever, and that will be changing in the future.

## Metrics collection expectations

It's important to clarify that our intention for telemetry gathering within our projects is solely to prioritize our work efforts in improving the most useful features. Telemetry is used for usage analysis and is not violating our users expectations of privacy nor anonymity.

### Application data vs User data

Our telemetry is solely focused on the collection of application data, not user data.

#### What is Application data?

Application data only contains information about events, modules, performance, and use of our applications, not the users themselves.

#### What is User data?

User data can vary across industries and applications, but is generally data that is used to classify or categorize users, typically for targeting classes or categories of users via campaigns or other marketing methods for various purposes. Think demographics, emails, physical addresses, personal preferences, and marketing.

As a general rule, we do not collect user or personal data. See our [Privacy Policy](./PRIVACY_POLICY.md) for more details.

### Metric features and their groupings

The metric features in the first column name and their descriptions come directly from [countly's support website](https://support.count.ly/hc/en-us/articles/360037441932-Web-analytics-JavaScript-#features-for-consent).

| Metric feature name | Consent method | Metric feature group name | Metric feature description | What do we use this metric for? |
|:-------------------:|----------------|---------------------------|--------------------------------------------------------------------------------|------------------------------------------------------------------------------------------|
| sessions | Opt-out | Minimal | tracks when, how often, and how long users use your website | For determining unique daily/weekly/monthly active users in order to prioritize projects |
| views | Opt-out | Minimal | allows for the views/pages accessed by a user to be tracked | For tracking application pages and component views only |
| events | Opt-out | Minimal | allows your events to be sent to the server | Timed and one-off application events such as app_start, app_close, function_used, etc. |
| crashes | Opt-out | Performance | allows JavaScript errors to be tracked | Not used yet - Eventually for bug fixes and debugging |
| apm | Opt-out | Performance | allows performance tracking of application by recording traces | Not used yet - Eventually for bug fixes and debugging |
| scrolls | Opt-out | UX | allows a user’s scrolls to be tracked on the heatmap | Not used yet - Possibly for performance and UX optimizations |
| clicks | Opt-out | UX | allows a user’s clicks and link clicks to be tracked on the heatmap | Not used yet - Possibly for performance and UX optimizations |
| forms | Opt-out | UX | allows a user’s form submissions to be tracked | Not used yet - Possibly for bug fixes and debugging, performance and UX optimizations |
| star-rating | Opt-out | Feedback | allows user rating and feedback tracking through rating widgets | Not used yet - Possibly for getting user feedback more directly through our applications |
| feedback | Opt-out | Feedback | allows survey, nps rating and feedback tracking through feedback widgets | Not used yet - Possibly for getting user feedback more directly through our applications |
| location | Opt-out | Location | allows a user’s location (country, city area) to be recorded | N/A - we have no current need for collecting a user's location data. |
| attribution | N/A | N/A | allows the campaign from which a user came to be tracked | N/A - we do not have any plans to implement or utilize user campaigns |
| users | N/A | N/A | allows user information, including custom properties, to be collected/provided | N/A - we will not collect user information |

The code implementing these groupings can be found [here](./../../src/MetricsProvider.ts#L17-L30)

**Note:** The above groupings are our global settings. You will want to check the project specific `./docs/telemetry/COLLECTED_DATA.md` for variations.

### What do the consent toggling UIs look like?

Consent toggling has two distinct views depending on the type of application implementing telemetry using this library: Projects without a settings page, and Projects with settings page(s).

Each toggle that a user sees will be tied to a specific "Metric feature group name" as listed in the above table. For now, we don't have any plans to allow more granular control than that, but please [open an issue](https://github.com/ipfs-shipyard/ignite-metrics/issues/new?assignees=&labels=need%2Ftriage&template=open_an_issue.md&title=) if you feel like that should change.

#### Projects without its own settings page(s)

In this case, we would follow the patterns discussed in https://github.com/ipfs/public-gateway-checker/issues/340 and https://github.com/ipfs-shipyard/ignite-metrics/issues/36. See https://pl-strflt.notion.site/Metrics-Collection-and-Consent-Language-0d4059f11d4d474bb76d00539d778d8d#3e2a579c4be043888b1f64bf300048b8 for up to date info.

1. There will be a floating icon at the bottom left of the page:

![example telemetry settings modal toggle for projects without a settings page](./images/telemetry-modal-toggle.png)

2. When clicked, that icon will expand into a modal that displays information about our telemetry collection

![example telemetry settings modal first view](./images/telemetry-modal-view1.png)


3. When "manage settings" is clicked, the modal will update to allow you to enable/disable the project's implemented collection of groups of features as defined above in the "Metric features and their groupings" section.

Single toggle Example:

![example modal containing a single consent toggle](./images/single-toggle-modal.png)

#### Projects with its own settings page(s)

The consent should conform to the UX expectations of the library it's being implemented in.

##### IPFS-companion

It should look like the rest of the settings in the application.

Example with single toggle (PR at https://github.com/ipfs/ipfs-companion/pull/1117):

![example telemetry single-toggle UI for settings page in ipfs-companion](./images/settings-page-single-toggle.png)

##### IPFS-Webui & IPFS-Desktop

Analytics toggling and consent notifications are already implemented in the desktop app and webui.

Example:

![example telemetry toggle UI for settings page in ipfs-webui and ipfs-desktop](./images/webui-settings-page-single-toggle.png)

### Informing users of changes to this policy

We will not display notifications to users regarding telemetry changes, but users can follow release-notes as well as changes to this library, it's telemetry documentation (`./docs/telemetry/*`), and projects implementing this library to stay up to date on our telemetry policy changes.

## SOPs (Standard Operating Procedures)

### Adding telemetry collection to a project without telemetry collection

1. Copy the [COLLECTED_DATA.template](./COLLECTED_DATA.template.md) file to your project at `./docs/telemetry/COLLECTED_DATA.md`
2. Implement the library according to your project's needs
3. Update your `COLLECTED_DATA.md` file specifying exactly which metrics your collecting, and when they're collected.

### Changing what metrics data is collected by a project

When you add/remove metrics data that is collected by a project, you MUST follow this checklist:

1. Fist ask yourself: Are you trying to collect user-identifiable, or other user data at all? STOP.
* You must get legal approval
* You must update the following documents:
* `./PRIVACY_POLICY.md` - responsibility of legal write a new doc after meeting with you and discussing what your plans are. You would then update the `PRIVACY_POLICY.md` in this repo.
* `./COLLECTION_POLICY.md` - responsibility of you to update according to the changes you're making.
* `./FAQs.md` - responsibility of you to update clarifying exactly why we're now collecting user data.
1. Modify the project's `./docs/telemetry/COLLECTED_DATA.md` file.
* Ensure the table under `What metrics data DO we collect` is accurate and up to date


### Changing a metric feature's grouping or opt-in/opt-out policy

When you move a metric feature to/from a particular feature group, or that group changes its consent method, you must update:

1. The table above at `Metric features and their groupings`
2. The code at https://github.com/ipfs-shipyard/ignite-metrics/blob/main/src/CountlyMetrics.ts#L10-L16
3. Possibly the code at https://github.com/ipfs-shipyard/ignite-metrics/blob/main/src/CountlyMetrics.ts#L27-L36

77 changes: 77 additions & 0 deletions docs/telemetry/FAQs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# FAQs - Frequently asked questions

## Why are you collecting metrics?

We are collecting metrics for all of our projects to assist with prioritizing work and provide proof that our work is valuable. Our prior state (<2023) had the following issues:

1. Opt-in only metrics for webui and desktop did not send metrics data to our countly server
1. We have no insight into the total number of users in our applications
2. We had no insight into the percentage of users who declined or accepted metrics
3. If a user did not accept or decline metrics, usage was not collected
2. Lack of singular metrics collection library
1. Consent and telemetry patterns were non-existent
2. Privacy policy and metrics collection would have needed to be implemented for each library separately.
3. Analytics across projects does not allow for normalized comparisons of usage.
3. Missing telemetry
1. We have no insight into the usage of a majority of our projects
1. It is understood (not confirmed with data) that two of our top three projects are ipfs-webui & ipfs-desktop, and they had ****some**** telemetry in place.
2. We have three separate usecases for ipfs-webui, and are only tracking two of them:
1. [webui.ipfs.io](http://webui.ipfs.io) & local - tracked
2. webui usage within ipfs-desktop - tracked as desktop usage
3. webui bundled with kubo - not tracked

You can find more information about the discussions held when making these decisions at https://pl-strflt.notion.site/Telemetry-b005d4f217f44db3986902c67d922cf4

## What data is collected?

In order to standardize the documentation of metrics collection, we will keep a file in each project's repository that
specifies all metrics data collected by that project. The file will be located at `./docs/telemetry/COLLECTED_DATA.md`. These documents will provide answers to the following bullet-points:

* This is what will be collected
* This is what won't be collected
* Why we are collecting each piece of metric data.

You can read our [Privacy Policy](./PRIVACY_POLICY.md) and [Collection Policy](./COLLECTION_POLICY.md) for more generic details.

## As an existing user, how am I notified about changes to the data that is collected?

You can follow updates to the metrics we collect for each repo in that repo's release notes. You can find changes to our collection policy in our [Collection Policy](./COLLECTION_POLICY.md) document. The document at `./docs/telemetry/COLLECTED_DATA.md` in each relevant repo will be updated when any metrics collection changes are made.

## How do I opt-out of metrics?

For apps with settings pages, such as ipfs-webui, ipfs-desktop, and ipfs-companion, you can opt out in each of the applications' setting pages. For apps without settings pages (or single page applications where settings pages don't make sense), we will be following the [patterns set by the public-gateway-checker project](https://github.com/ipfs/public-gateway-checker/issues/340#issuecomment-1371410214).

The basic implementation is a floating icon on the bottom left of the page that displays a modal when clicked. The displayed modal allows the enabling/disabling of metrics collection for our different groupings.

For public-gateway-checker changes, see the following PRs and issues:

* https://github.com/ipfs/public-gateway-checker/pull/309
* https://github.com/ipfs/public-gateway-checker/issues/340
* https://github.com/ipfs/public-gateway-checker/issues/341
* https://github.com/ipfs/public-gateway-checker/issues/342

## Can I turn telemetry completely off?

Yes. If you toggle all telemetry groups off (i.e. opt-out), no telemetry will be collected. The one caveat is that we send a single request when you opt-out of metrics collection.


## I want to help development efforts, but want to be updated of any future changes. How do I stay up to date?

There are a few ways you can stay up to date depending on what you're interested in.

### Global telemetry changes

Watch for changes to https://github.com/ipfs-shipyard/ignite-metrics via GitHub's watch feature. You can adjust notification preferences using GitHub's watch feature. Updates can be found in the following locations:

1. PRs
2. Release notes
3. `./docs/telemetry/COLLECTED_DATA.template.md`


### Project specific telemetry changes

Watch for changes to https://github.com/<repo-of-interest> via GitHub's watch feature. You can adjust notification preferences using GitHub's watch feature. Updates can be found in the following locations:

1. PRs
2. Release notes
3. `./docs/telemetry/COLLECTED_DATA.md`
5 changes: 5 additions & 0 deletions docs/telemetry/PRIVACY_POLICY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Privacy Policy

To be filled in by legal team based off of meeting documented at https://pl-strflt.notion.site/2023-01-05-Privacy-Policy-Updates-9bef84008b52449496ad5b806d529dfe

Until then, please reference the [Privacy Policy](https://ipfs.tech/companion-privacy/) for ipfs-companion (which is the only project we own with a current [privacy policy](https://github.com/ipfs/ipfs-companion/blob/main/PRIVACY-POLICY.md)).
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/telemetry/images/single-toggle-modal.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/telemetry/images/telemetry-modal-toggle.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/telemetry/images/telemetry-modal-view1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@
"import": "./dist/src/index.js"
},
"./vanilla": {
"types": "./dist/src/CountlyMetrics.d.ts",
"import": "./dist/src/CountlyMetrics.js"
"types": "./dist/src/MetricsProvider.d.ts",
"import": "./dist/src/MetricsProvider.js"
}
},
"eslintConfig": {
Expand Down
Loading

0 comments on commit 4cb35ba

Please sign in to comment.