Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(#9489): collect telemetry for offline freetext searching #9525

Open
wants to merge 39 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
991b281
wip
m5r Oct 10, 2024
77d84fd
remove unused code
m5r Oct 16, 2024
95a8b77
collect telemetry for searches against the `contacts_by_freetext` view
m5r Oct 16, 2024
5cc183f
change telemetry key
m5r Oct 17, 2024
913b1fc
wip `reports_by_freetext`
m5r Oct 17, 2024
0e2cafd
wip `contacts_by_type_freetext`
m5r Oct 17, 2024
b76d85e
cleanup `contacts_by_freetext` telemetry
m5r Oct 21, 2024
5d21f72
collect telemetry for searches against the `reports_by_freetext` view
m5r Oct 21, 2024
6b94414
fix `currentQuery` being always equal to `params.data.q` since they s…
m5r Oct 21, 2024
843888c
`freetextRequest.key` never exists, I think we were looking for `free…
m5r Oct 21, 2024
de70aad
collect telemetry for searches against the `contacts_by_type_freetext…
m5r Oct 21, 2024
d7c322a
fix test
m5r Oct 22, 2024
bf001aa
e2e test for contact search telemetry
m5r Oct 23, 2024
b5e5ee4
e2e test for report search telemetry
m5r Oct 23, 2024
bffd0a0
e2e test for contact_by_type search telemetry
m5r Oct 23, 2024
abfd112
return early
m5r Oct 24, 2024
5c241a7
extract `findMatchingProperties`
m5r Oct 24, 2024
cea6292
ok sonar.
m5r Oct 24, 2024
5589861
is sonar happy with `contacts-content.component.ts`?
m5r Oct 24, 2024
19712b6
is sonar happy with `reports-content.component.ts` too?
m5r Oct 24, 2024
a2d3fff
refactor 👍
m5r Oct 24, 2024
31d5b6b
refactor extra `select2:select` event listener
m5r Oct 28, 2024
a5a3b98
don't bother looping over the doc fields when we have a `key:value` c…
m5r Oct 28, 2024
ab8d9e5
move `propertyPath` initialization
m5r Oct 28, 2024
b6e3b6b
revert db after telemetry test
m5r Oct 28, 2024
bca463f
use `void` keyword to mark promises as intentionally not awaited
m5r Oct 28, 2024
ef1918a
fix telemetry test
m5r Oct 28, 2024
6cdb9cc
fix select2-search telemetry
m5r Oct 28, 2024
e59da86
remove `void` operator and add comment why we're not awaiting the pro…
m5r Oct 28, 2024
4b59c92
handle select2search case where it calls `contacts_by_freetext`
m5r Oct 28, 2024
156457a
`for..of` => `Promise.all`
m5r Oct 28, 2024
06a86b5
unit test report search telemetry
m5r Oct 30, 2024
ab8bf82
unit test contact search telemetry
m5r Oct 30, 2024
3ff8d1c
add e2e test for select2 without contact type
m5r Oct 30, 2024
8bcd070
clean up e2e test
m5r Oct 30, 2024
2e75674
wow is sonar finally happy with `void` here?
m5r Oct 30, 2024
32ddecc
oops forgot some leftovers to clean up
m5r Oct 30, 2024
857203f
clean up telemetry db before running our telemetry tests
m5r Oct 30, 2024
e795d34
clean up telemetry db after each telemetry tests
m5r Oct 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion shared-libs/search/src/generate-search-requests.js
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ const sortByLastVisitedDate = () => {
const makeCombinedParams = (freetextRequest, typeKey) => {
const type = typeKey[0];
const params = {};
if (freetextRequest.key) {
if (freetextRequest.params.key) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a fix for key:value queries against the view contacts_by_type_freetext. Without this, these queries would fail silently and the select2 input would show an infinite loader until a different query would return results. If this has never been noticed before I believe the key:value search feature isn't used very much...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤯

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about this again, I feel like we really should include a unit test for this case (if possible).

params.key = [ type, freetextRequest.params.key[0] ];
} else {
params.startkey = [ type, freetextRequest.params.startkey[0] ];
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ describe('Pregnancy danger sign follow-up form', () => {
const person = personFactory.build();

const fillPregnancyDangerSignFollowUpForm = async (attendToVisit, hasDangerSigns) => {
await genericForm.selectContact(person.name);
await genericForm.selectContact(person.name, 'What is the patient\'s name?');
await genericForm.nextPage();
await commonEnketoPage.selectRadioButton('Did the woman visit the health facility as recommended?', attendToVisit);
await commonEnketoPage.selectRadioButton('Is she still experiencing any danger signs?', hasDangerSigns);
Expand Down
39 changes: 39 additions & 0 deletions tests/e2e/default/telemetry/forms/select_contact_telemetry.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<?xml version="1.0"?>
<h:html xmlns="http://www.w3.org/2002/xforms" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms">
<h:head>
<h:title>Select contact by type and without type</h:title>
<model>
<itext>
<translation lang="en">
<text id="/data/select_contact_by_type:label">
<value>Select the contact by type</value>
</text>
<text id="/data/select_contact_without_type:label">
<value>Select the contact without type</value>
</text>
</translation>
</itext>
<instance>
<data id="select_contact_telemetry" prefix="J1!select_contact_telemetry!" delimiter="#" version="2024-10-30 16-00">
<select_contact_by_type/>
<select_contact_without_type/>
<meta tag="hidden">
<instanceID/>
</meta>
</data>
</instance>
<bind nodeset="/data/select_contact_by_type" type="db:person"/>
<bind nodeset="/data/select_contactselect_contact_without_type"/>
</model>
</h:head>
<h:body class="pages">
<group appearance="field-list" ref="/data">
<input appearance="select-contact" ref="/data/select_contact_by_type">
<label ref="jr:itext('/data/select_contact_by_type:label')"/>
</input>
<input appearance="select-contact" ref="/data/select_contact_without_type">
<label ref="jr:itext('/data/select_contact_without_type:label')"/>
</input>
</group>
</h:body>
</h:html>
160 changes: 141 additions & 19 deletions tests/e2e/default/telemetry/telemetry.wdio-spec.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
const { Key } = require('webdriverio');
const commonPage = require('@page-objects/default/common/common.wdio.page');
const utils = require('@utils');
const moment = require('moment');
Expand All @@ -6,46 +7,82 @@ const placeFactory = require('@factories/cht/contacts/place');
const personFactory = require('@factories/cht/contacts/person');
const { faker: Faker } = require('@faker-js/faker');
const userFactory = require('@factories/cht/users/users');
const searchPage = require('@page-objects/default/search/search.wdio.page');
const contactPage = require('@page-objects/default/contacts/contacts.wdio.page');
const reportsPage = require('@page-objects/default/reports/reports.wdio.page');
const pregnancyFactory = require('@factories/cht/reports/pregnancy');
const genericForm = require('@page-objects/default/enketo/generic-form.wdio.page');
const fs = require('fs');
const { BRANCH, TAG } = process.env;

const setupUser = () => {
describe('Telemetry', () => {
const places = placeFactory.generateHierarchy();
const districtHospital = places.get('district_hospital');
const clinic = places.get('clinic');
const patient = personFactory.build({
phone: '+12068881234',
parent: { _id: clinic._id, parent: clinic.parent },
});

const healthCenter = places.get('health_center');
const contact = personFactory.build({
name: Faker.name.firstName(),
parent: { _id: districtHospital._id },
name: Faker.person.firstName(),
parent: { _id: healthCenter._id },
phone: '+9779841299392',
});
const user = userFactory.build({
username: Faker.internet.userName().toLowerCase().replace(/[^0-9a-zA-Z_]/g, ''),
password: 'Secret_1',
place: districtHospital._id,
place: healthCenter._id,
contact: contact._id,
known: true,
});
const pregnancyReport = pregnancyFactory.build({
fields: {
patient_id: patient.patient_id,
patient_uuid: patient._id,
name: patient.name,
},
contact: {
_id: contact._id,
parent: contact.parent,
},
from: contact.phone,
});
let reportDocs;

return {
docs: [
...places.values(),
contact,
],
user,
};
};

describe('Telemetry', () => {
const DATE_FORMAT = 'YYYY-MM-DD';
const TELEMETRY_PREFIX = 'telemetry';
let user;
let docs;
const todayDBName = `${TELEMETRY_PREFIX}-${moment().format(DATE_FORMAT)}-${user.username}`;

before(async () => {
m5r marked this conversation as resolved.
Show resolved Hide resolved
({ docs, user } = setupUser());
await utils.saveDocs(docs);
const selectContactTelemetryForm = utils.deepFreeze({
_id: 'form:select_contact_telemetry',
internalId: 'select_contact_telemetry',
title: 'Select contact by type and without type',
type: 'form',
_attachments: {
xml: {
content_type: 'application/octet-stream',
data: Buffer
.from(fs.readFileSync(`${__dirname}/forms/select_contact_telemetry.xml`, 'utf8'))
.toString('base64'),
},
},
});

await utils.saveDocIfNotExists(selectContactTelemetryForm);
await utils.saveDocs([...places.values(), contact, patient]);
reportDocs = await utils.saveDocs([pregnancyReport]);
await utils.createUsers([user]);
await loginPage.login(user);
await commonPage.waitForPageLoaded();
});

after(async () => {
await utils.deleteUsers([user]);
await utils.revertDb([/^form:(?!select_contact_telemetry)/], true);
});

it('should record telemetry', async () => {
const yesterday = moment().subtract(1, 'day');
const yesterdayDBName = `${TELEMETRY_PREFIX}-${yesterday.format(DATE_FORMAT)}-${user.username}`;
Expand Down Expand Up @@ -82,4 +119,89 @@ describe('Telemetry', () => {
const version = TAG || utils.escapeBranchName(BRANCH) || clientDdoc.build_info.base_version;
expect(clientDdoc.build_info.version).to.include(version);
});

describe('search matches telemetry', () => {
afterEach(async () => {
// eslint-disable-next-line no-undef
await browser.execute((dbName) => window.PouchDB(dbName).destroy(), todayDBName);
});

const getTelemetryEntryByKey = async (key) => {
const todayTelemetryDocs = await browser.execute(async (dbName) => {
// eslint-disable-next-line no-undef
const docs = await window.PouchDB(dbName).allDocs({ include_docs: true });
return docs.rows.filter(row => row.doc.key.startsWith('search_match'));
}, todayDBName);
return todayTelemetryDocs.filter(row => row.doc.key === key);
};

it('should record telemetry for contact searches', async () => {
await commonPage.goToPeople();

const [firstName, lastName] = patient.name.split(' ');
const phone = patient.phone;
const patient_id = patient.patient_id;
const searchTerms = [firstName, lastName, phone, patient_id, `patient_id:${patient_id}`];
for (const searchTerm of searchTerms) {
await searchPage.performSearch(searchTerm);
await contactPage.selectLHSRowByText(patient.name, false);
await searchPage.clearSearch();
}

expect(await getTelemetryEntryByKey('search_match:contacts_by_freetext:name')).to.have.lengthOf(2);
expect(await getTelemetryEntryByKey('search_match:contacts_by_freetext:phone')).to.have.lengthOf(1);
expect(await getTelemetryEntryByKey('search_match:contacts_by_freetext:patient_id')).to.have.lengthOf(1);
expect(await getTelemetryEntryByKey('search_match:contacts_by_freetext:patient_id:$value')).to.have.lengthOf(1);
});

it('should record telemetry for reports searches', async () => {
await commonPage.goToReports();

const [firstName, lastName] = patient.name.split(' ');
const phone = contact.phone;
const patient_id = patient.patient_id;
const searchTerms = [firstName, lastName, phone, patient_id, `patient_id:${patient_id}`];
for (const searchTerm of searchTerms) {
await searchPage.performSearch(searchTerm);
await reportsPage.openReport(reportDocs[0].id);
await searchPage.clearSearch();
}

expect(await getTelemetryEntryByKey('search_match:reports_by_freetext:fields.name')).to.have.lengthOf(2);
expect(await getTelemetryEntryByKey('search_match:reports_by_freetext:from')).to.have.lengthOf(1);
expect(await getTelemetryEntryByKey('search_match:reports_by_freetext:fields.patient_id')).to.have.lengthOf(1);
expect(await getTelemetryEntryByKey('search_match:reports_by_freetext:patient_id:$value')).to.have.lengthOf(1);
});

it('should record telemetry for contact searches from the select2 component', async () => {
await browser.url(`/#/contacts/${patient._id}/report/select_contact_telemetry`);
await commonPage.waitForPageLoaded();

const [firstName, lastName] = patient.name.split(' ');
const searchTerms = [firstName, lastName, patient.phone, `phone:${patient.phone}`];

for (const searchTerm of searchTerms) {
await genericForm.selectContact(patient.name, 'Select the contact by type', searchTerm);
await genericForm.clearSelectedContact('Select the contact by type');
}

expect(await getTelemetryEntryByKey('search_match:contacts_by_type_freetext:name')).to.have.lengthOf(2);
expect(await getTelemetryEntryByKey('search_match:contacts_by_type_freetext:phone')).to.have.lengthOf(1);
expect(await getTelemetryEntryByKey('search_match:contacts_by_type_freetext:phone:$value')).to.have.lengthOf(1);

const searchField = await $('.select2-search__field');
if (await searchField.isDisplayed()) {
await browser.keys(Key.Escape);
}

for (const searchTerm of searchTerms) {
await genericForm.selectContact(patient.name, 'Select the contact without type', searchTerm);
await genericForm.clearSelectedContact('Select the contact without type');
}

expect(await getTelemetryEntryByKey('search_match:contacts_by_freetext:name')).to.have.lengthOf(2);
expect(await getTelemetryEntryByKey('search_match:contacts_by_freetext:phone')).to.have.lengthOf(1);
expect(await getTelemetryEntryByKey('search_match:contacts_by_freetext:phone:$value')).to.have.lengthOf(1);
});
});
});
22 changes: 16 additions & 6 deletions tests/page-objects/default/enketo/generic-form.wdio.page.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ const validationErrors = () => $$('.invalid-required');
const waitForValidationErrorsToDisappear = () => browser.waitUntil(async () => !(await validationErrors()).length);
const waitForValidationErrors = () => browser.waitUntil(async () => (await validationErrors()).length);
const fieldByName = (formId, name) => $(`#report-form [name="/${formId}/${name}"]`);
const select2Selection = (label) => $(`label*=${label}`).$('.select2-selection');

const nextPage = async (numberOfPages = 1, waitForLoad = true) => {
if (waitForLoad) {
Expand All @@ -29,19 +30,27 @@ const nextPage = async (numberOfPages = 1, waitForLoad = true) => {
}
};

const selectContact = async (contactName) => {
const select2Selection = () => $('label*=What is the patient\'s name?').$('.select2-selection');
await (await select2Selection()).click();
const selectContact = async (contactName, label, searchTerm = '') => {
const searchField = await $('.select2-search__field');
await searchField.setValue(contactName);
const contact = await $('.name');
if (!await searchField.isDisplayed()) {
await (await select2Selection(label)).click();
}

await searchField.setValue(searchTerm || contactName);
await $('.select2-results__option.loading-results').waitForDisplayed({ reverse: true });
const contact = await $(`.name*=${contactName}`);
await contact.waitForDisplayed();
await contact.click();

await browser.waitUntil(async () => {
return (await (await select2Selection()).getText()).toLowerCase().endsWith(contactName.toLowerCase());
return (await (await select2Selection(label)).getText()).toLowerCase().endsWith(contactName.toLowerCase());
});
};

const clearSelectedContact = async (label) => {
await (await select2Selection(label)).$('.select2-selection__clear').click();
};

const submitForm = async ({ waitForPageLoaded = true, ignoreValidationErrors = false } = {}) => {
await formTitle().click();
if (!ignoreValidationErrors) {
Expand Down Expand Up @@ -101,6 +110,7 @@ module.exports = {
nameField,
fieldByName,
selectContact,
clearSelectedContact,
cancelForm,
submitForm,
currentFormView,
Expand Down
Loading
Loading