Skip to content
This repository has been archived by the owner on Feb 29, 2020. It is now read-only.

Latest commit

 

History

History
231 lines (203 loc) · 10.9 KB

data_dictionary.md

File metadata and controls

231 lines (203 loc) · 10.9 KB
Raw

Activity Stream Pings

The Activity Stream system add-on sends various types of pings to the backend (HTTPS POST) Onyx server :

  • a session ping that describes the ending of an Activity Stream session (a new tab is closed or refreshed), and
  • an event ping that records specific data about individual user interactions while interacting with Activity Stream
  • a performance ping that records specific performance related events
  • an undesired ping that records data about bad app states and missing data
  • an impression_stats ping that records data about Pocket impressions and user interactions

Schema definitions/validations that can be used for tests can be found in system-addon/test/schemas/pings.js.

Example Activity Stream session Log

{
  // These fields are sent from the client
  "action": "activity_stream_session",
  "addon_version": "1.0.0",
  "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e",
  "locale": "en-US",
  "page": ["about:newtab" | "about:home" | "unknown"],
  "session_duration": 1635,
  "session_id": "{12dasd-213asda-213dkakj}",
  "region": "US",
  "profile_creation_date": 14786,
  "user_prefs": 7

  // These fields are generated on the server
  "date": "2016-03-07",
  "ip": "10.192.171.13",
  "ua": "python-requests/2.9.1",
  "receive_at": 1457396660000
}

Example Activity Stream user_event Log

{
  "action": "activity_stream_user_event",
  "action_position": "3",
  "addon_version": "1.0.0",
  "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e",
  "event": "click or scroll or search or delete",
  "locale": "en-US",
  "page": ["about:newtab" | "about:home" | "unknown"],
  "source": "top sites, or bookmarks, or...",
  "session_id": "{12dasd-213asda-213dkakj}",
  "recommender_type": "pocket-trending",
  "metadata_source": "MetadataService or Local or TippyTopProvider",
  "user_prefs": 7

  // These fields are generated on the server
  "ip": "10.192.171.13",
  "ua": "python-requests/2.9.1",
  "receive_at": 1457396660000,
  "date": "2016-03-07",
}

Example Activity Stream performance Log

{
  "action": "activity_stream_performance_event",
  "addon_version": "1.0.0",
  "client_id": "374dc4d8-0cb2-4ac5-a3cf-c5a9bc3c602e",
  "event": "previewCacheHit",
  "event_id": "45f1912165ca4dfdb5c1c2337dbdc58f",
  "locale": "en-US",
  "page": "unknown", // all session-specific perf events should be part of the session perf object
  "receive_at": 1457396660000,
  "source": "TOP_FRECENT_SITES",
  "value": 1,
  "user_prefs": 7,

  // These fields are generated on the server
  "ip": "10.192.171.13",
  "ua": "python-requests/2.9.1",
  "receive_at": 1457396660000,
  "date": "2016-03-07"
}

Example Activity Stream undesired event Log

{
  "action": "activity_stream_undesired_event",
  "addon_version": "1.0.12",
  "client_id": "26288a14-5cc4-d14f-ae0a-bb01ef45be9c",
  "event": "MISSING_IMAGE",
  "locale": "en-US",
  "page": ["about:newtab" | "about:home" | "unknown"]
  "source": "HIGHLIGHTS",
  "value": 0,
  "user_prefs": 7,

  // These fields are generated on the server
  "ip": "10.192.171.13",
  "ua": "python-requests/2.9.1",
  "receive_at": 1457396660000,
  "date": "2016-03-07"
}

Example Activity Stream impression_stats Logs

{
  "action": "activity_stream_impression_stats",
  "client_id": "n/a",
  "session_id": "n/a",
  "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}",
  "addon_version": "1.0.12",
  "locale": "en-US",
  "source": "pocket",
  "page": ["about:newtab" | "about:home" | "unknown"]
  "tiles": [{"id": 10000}, {"id": 10001}, {"id": 10002}]
  "user_prefs": 7
}
{
  "action": "activity_stream_impression_stats",
  "client_id": "n/a",
  "session_id": "n/a",
  "impression_id": "{005deed0-e3e4-4c02-a041-17405fd703f6}",
  "addon_version": "1.0.12",
  "locale": "en-US",
  "source": "pocket",
  "page": "unknown",
  "user_prefs": 7,

  // "pos" is the 0-based index to record the tile's position in the Pocket section.
  "tiles": [{"id": 10000, "pos": 0}],

  // A 0-based index to record which tile in the "tiles" list that the user just interacted with.
  "click|block|pocket": 0
}
KEY DESCRIPTION  
action_position [Optional] The index of the element in the source that was clicked. 1️⃣
action [Required] Either activity_stream_event, activity_stream_session, or activity_stream_performance. 1️⃣
addon_version [Required] The version of the Activity Stream addon. 1️⃣
client_id [Required] An identifier for this client. 1️⃣
card_type [Optional] ("bookmark", "pocket", "trending", "pinned") 1️⃣
date [Auto populated by Onyx] The date in YYYY-MM-DD format. 3️⃣
experiment_id [Optional] The unique identifier for a specific experiment. 1️⃣
event_id [Required] An identifier shared by multiple performance pings that describe ane entire request flow. 1️⃣
event [Required] The type of event. Any user defined string ("click", "share", "delete", "more_items") 1️⃣
highlight_type [Optional] Either ["bookmarks", "recommendation", "history"]. 1️⃣
impression_id [Optional] The unique impression identifier for a specific client. 1️⃣
ip [Auto populated by Onyx] The IP address of the client. 2️⃣
locale [Auto populated by Onyx] The browser chrome's language (eg. en-US). 2️⃣
load_trigger_ts [Optional][Server Counter][Server Alert for too many omissions] DOMHighResTimeStamp of the action perceived by the user to trigger the load of this page. 1️⃣
load_trigger_type [Server Counter][Server Alert for too many omissions] Either ["first_window_opened", "menu_plus_or_keyboard", "unexpected"]. 1️⃣
metadata_source [Optional] The source of which we computed metadata. Either (MetadataService or Local or TippyTopProvider). 1️⃣
page [Required] One of ["about:newtab", "about:home", "unknown" (which either means not-applicable or is a bug)]. 1️⃣
recommender_type [Optional] The type of recommendation that is being shown, if any. 1️⃣
session_duration [Optional][Server Counter][Server Alert for too many omissions] Time in (integer) milliseconds of the difference between the new tab becoming visible
and losing focus. 1️⃣
session_id [Optional] The unique identifier for a specific session. 1️⃣
source [Required] Either ("recent_links", "recent_bookmarks", "frecent_links", "top_sites", "spotlight", "sidebar") and indicates what action. 2️⃣
received_at [Auto populated by Onyx] The time in ms since epoch. 3️⃣
total_bookmarks [Optional] The total number of bookmarks in the user's places db. 1️⃣
total_history_size [Optional] The number of history items currently in the user's places db. 1️⃣
ua [Auto populated by Onyx] The user agent string. 2️⃣
unload_reason [Required] The reason the Activity Stream page lost focus. 1️⃣
url [Optional] The URL of the recommendation shown in one of the highlights spots, if any. 1️⃣
value (performance) [Required] An integer that represents the measured performance value. Can store counts, times in milliseconds, and should always be a positive integer. 1️⃣
value (event) [Optional] An object with keys "icon_type" and "card_type" to record the extra information for event ping 1️⃣
ver [Auto populated by Onyx] The version of the Onyx API the ping was sent to. 1️⃣
highlights_size [Optional] The size of the Highlights set. 1️⃣
highlights_data_late_by_ms [Optional] Time in ms it took for Highlights to become initialized 1️⃣
topsites_data_late_by_ms [Optional] Time in ms it took for TopSites to become initialized 1️⃣
topstories.domain.affinity.calculation.ms [Optional] Time in ms it took for domain affinities to be calculated 1️⃣
topsites_first_painted_ts [Optional][Service Counter][Server Alert for too many omissions] Timestamp of when the Top Sites element finished painting (possibly with only placeholder screenshots) 1️⃣
custom_screenshot [Optional] Number of topsites that display a custom screenshot. 1️⃣
screenshot_with_icon [Optional] Number of topsites that display a screenshot and a favicon. 1️⃣
screenshot [Optional] Number of topsites that display only a screenshot. 1️⃣
tippytop [Optional] Number of topsites that display a tippytop icon. 1️⃣
rich_icon [Optional] Number of topsites that display a high quality favicon. 1️⃣
no_image [Optional] Number of topsites that have no screenshot. 1️⃣
topsites_pinned [Optional] Number of topsites that are pinned. 1️⃣
visibility_event_rcvd_ts [Optional][Server Counter][Server Alert for too many omissions] DOMHighResTimeStamp of when the page itself receives an event that document.visibilityState == visible. 1️⃣
tiles [Required] A list of tile objects for the Pocket articles. Each tile object mush have a ID, and optionally a "pos" property to indicate the tile position 1️⃣
click [Optional] An integer to record the 0-based index when user clicks on a Pocket tile. 1️⃣
block [Optional] An integer to record the 0-based index when user blocks a Pocket tile. 1️⃣
pocket [Optional] An integer to record the 0-based index when user saves a Pocket tile to Pocket. 1️⃣
user_prefs [Required] The encoded integer of user's preferences. 1️⃣ & 4️⃣
is_prerendered [Required] A boolean to signify whether the page is prerendered or not 1️⃣
is_preloaded [Required] A boolean to signify whether the page is preloaded or not 1️⃣
icon_type [Optional] ("tippytop", "rich_icon", "screenshot_with_icon", "screenshot", "no_image") 1️⃣
region [Optional] An string maps to pref "browser.search.region", which is essentially the two letter ISO 3166-1 country code populated by the Firefox search service. Note that: 1). it reports "OTHER" for those regions with smaller Firefox user base (less than 10000) so that users cannot be uniquely identified; 2). it reports "UNSET" if this pref is missing; 3). it reports "EMPTY" if the value of this pref is an empty string. 1️⃣
profile_creation_date [Optional] An integer to record the age of the Firefox profile as the total number of days since the UNIX epoch. 1️⃣

Where:

1️⃣ Firefox data 2️⃣ HTTP protocol data 3️⃣ server augmented data 4️⃣ User preferences encoding table

Note: the following session-related fields are not yet implemented in the system-addon, but will likely be added in future versions:

{
  "total_bookmarks": 19,
  "total_history_size": 9,
  "highlights_size": 20
}

This encoding mapping was defined in system-addon/lib/TelemetryFeed.jsm

Preference Encoded value
showSearch 1
showTopSites 2
showTopStories 4
showHighlights 8
showSnippets 16
showSponsored 32

Each item above could be combined with other items through bitwise OR operation