search package #152

pranavkulkarni · 2019-04-09T17:49:02Z

Module to search for ArcGIS items and format them for display in ArcGIS Hub

Structure --

src
  ago
    helpers
      aggs
      filters
      format
      page
      sort
    search.ts // searches for items and returns raw ago results
    params.ts // defines ISearchParams interface to hub.js/search
    serialize.ts // serialize raw UI params into query parameters that V3 understands
    format-item-collection.ts // format raw ago results into hub v3 style datasets
    compute-items-facets.ts // compute aggs/facets based on ago results and also calculate aggs for custom facets

Sample of how this will be used in UI code --

const authentication = new arcgisRest.UserSession({
      portal: 'https://qaext.arcgis.com/sharing/rest',
      username: 'dcadminqa',
      password: 'dcadminqa1'
    })
    const portal = 'https://qaext.arcgis.com/sharing/rest'
    const params = {
      q: 'crime',
      sort: 'name',
      agg: { fields: 'tags,collection,owner,source,hasApi,downloadable' },
      start: 1,
      num: 10
    }
    authentication.getToken(portal)
    .then(token => {
      const headers = { authorization: token, portal }
      const serializedParams = arcgisHub.serialize(params)
      console.log(serializedParams)
      return fetch(`hub.arcgis.com/api/v3/search?${serializedParams}`, { headers })
    })
    .then(unifiedResultsPromise => {
      return unifiedResultsPromise.json()
    })
    .then(unifiedResults => {
      console.log(unifiedResults)
      // UPDATE UI model with unified results
    })

Sample of how hub indexer api v3 new search route will use hub.js --

   const search = require('@esri/hub-search')

  // 1. perform ago search with facets and formatting
  const agoResults = await search.agoSearch(req.query, req.headers.authorization, req.headers.portal)

  // 2. compute items facets
  const facets = await search.computeItemsFacets(agoResults.aggregations, req.query, req.headers.authorization, req.headers.portal)

  // 3. format item collection based on actual search results and computed facets
  const model = search.agoFormatItemCollection(agoResults, facets, req.query)

  return model

codecov · 2019-04-09T18:21:07Z

Codecov Report

Merging #152 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #152    +/-   ##
======================================
  Coverage     100%   100%            
======================================
  Files          57     86    +29     
  Lines         897   1233   +336     
  Branches      117    185    +68     
======================================
+ Hits          897   1233   +336

Impacted Files	Coverage Δ
packages/annotations/src/util.ts	`100% <ø> (ø)`	⬆️
...ckages/search/src/ago/helpers/aggs/downloadable.ts	`100% <100%> (ø)`
...ackages/search/src/ago/helpers/aggs/create-aggs.ts	`100% <100%> (ø)`
packages/search/src/ago/format-item-collection.ts	`100% <100%> (ø)`
packages/search/src/ago/helpers/format/index.ts	`100% <100%> (ø)`
packages/common/src/index.ts	`100% <100%> (ø)`	⬆️
...s/search/src/ago/helpers/filters/encode-filters.ts	`100% <100%> (ø)`
packages/search/src/ago/search.ts	`100% <100%> (ø)`
packages/search/src/ago/helpers/filters/has-api.ts	`100% <100%> (ø)`
...ckages/search/src/ago/helpers/filters/group-ids.ts	`100% <100%> (ø)`
... and 51 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13a6374...4de4434. Read the comment docs.

jPurush · 2019-04-09T19:37:22Z

Can we add a README similar to one from the other packages ?

pranavkulkarni · 2019-04-11T18:32:06Z

Yes, I will once this PR is complete. The scope of this package is still in the works.

jgravois

very cool!

i've jotted down some initial thoughts...

.gitignore

packages/search/package.json

packages/search/src/ago/format-item.ts

packages/search/src/ago/encode-ago-query.ts

packages/search/src/ago/search.ts

packages/search/src/common/create-filters.ts

packages/search/src/common/params.ts

packages/search/src/index.ts

tsconfig.json

haoliangyu

General comments after a quick scan:

There should be room to simplify the creation of AGO query parameters. The logic is complicated in opendata-ui because it needs to search both AGO and Hub backend, but it is simpler here.
There should be interface(s) to define the search results in JSONAPI.

packages/search/src/ago/create-q.ts

packages/search/src/ago/format-item.ts

packages/search/src/ago/helpers/calc-start.ts

packages/search/src/ago/helpers/handle-filter.ts

packages/search/src/ago/helpers/downloadable.ts

packages/search/src/ago/helpers/sort.ts

packages/search/src/common/hub-type-map.ts

packages/search/src/ago/search.ts

packages/search/src/ago/format-item.ts

packages/search/src/common/params.ts

[WIP] search package (#152)

jgravois · 2019-05-08T00:01:54Z

i accidentally dismissed @haoliangyu's review, but i think all his feedback was addressed anyway.

@pranavkulkarni is this PR ready for another round of code review?

pranavkulkarni · 2019-05-08T16:55:32Z

@jgravois this PR is ready

jgravois

true hero's work @pranavkulkarni.

it looks like you haven't done much in the way of properly formatted TSDoc to ensure that folks outside of tugboat will be able to visit our documentation and grok how your first order methods work.

to be fair though, our existing initiative admin methods are every bit as opaque as the stuff you're adding here.

after you deal with my (minor) quips, this seems ready to

jgravois · 2019-05-08T18:45:04Z

packages/search/package.json

@@ -0,0 +1,61 @@
+{
+  "name": "@esri/hub-search",
+  "version": "1.11.1",


sorry, but we released v2.0.0 when you were out on monday. if you update this lerna can help us release v2.1.0 afterward with your new package.

jgravois · 2019-05-08T18:46:59Z

packages/search/package.json

+    "@esri/arcgis-rest-portal": "^2.0.0",
+    "@esri/arcgis-rest-request": "^2.0.0",
+    "@esri/arcgis-rest-types": "^2.0.0",
+    "@esri/hub-common": "^1.11.1"


"@esri/hub-common": "^2.0.0"

jgravois · 2019-05-08T18:47:16Z

packages/search/package.json

+    "@esri/arcgis-rest-portal": "^2.0.0",
+    "@esri/arcgis-rest-request": "^2.0.0",
+    "@esri/arcgis-rest-types": "^2.0.0",
+    "@esri/hub-common": "^1.11.1"


"@esri/hub-common": "^2.0.0"

jgravois · 2019-05-08T18:50:00Z

packages/search/src/ago/params.ts

+
+import { ISearchOptions } from "@esri/arcgis-rest-portal";
+
+export interface ISearchParams extends ISearchOptions {


i'm not a big fan of conflating Params and RequestOptions here, but @dbouwman resisted my attempts to enforce method signatures that dig everything out of a single argument too.

what i guess i'm saying is, 'carry on'.

jgravois · 2019-05-08T18:56:41Z

packages/search/src/ago/compute-items-facets.ts

+        {},
+        customAggsFunctions[customAgg](agoAggregations)
+      );
+      facets3 = Object.assign({}, facets3, format(rawCounts));


i'm not sure if it helps with Object.assign() too but i'm under the impression that tslib optimizes the destructuring we're doing everywhere else.

const facets3 = { ...format(rawCounts) }

jgravois · 2019-05-08T18:59:22Z

packages/search/src/ago/helpers/filters/has-api.ts

@@ -0,0 +1,21 @@
+import { getProp } from "@esri/hub-common";
+const apiTypes = ["Feature Service", "Map Service", "Image Service"];


i feel like you're declaring this in at least two different places.

Good catch! I'll move it to hub-common

jgravois · 2019-05-08T19:01:44Z

packages/search/src/ago/helpers/filters/has-api.ts

+import { getProp } from "@esri/hub-common";
+const apiTypes = ["Feature Service", "Map Service", "Image Service"];
+
+export function hasApi(queryFilters: any) {


when this library is bundled as a UMD all the methods hang off of an arcgisHub namespace.

this means that in the browser, filters.hasApi will collide with aggs.hasApi

Interesting... the UI is not supposed to calls these functions directly. They are being used by agoSearch and computeItemsFacets that are exposed by this library and the UI will talk to just those functions. Do I have to still rename them?

i didn't see before that you included "hasApi" in the blacklist. thats definitely good.

even if callers that load the UMD don't call these methods directly though, things will go pear shaped when they're called internally, because i don't think there is a mechanism in place in that build to differentiate them.

you have two options:

make the names unique, just to be safe.

test loading the UMD in the browser and calling both methods that lean on hasApi internally to confirm that i'm wrong.

jgravois · 2019-05-08T19:11:36Z

can you give this a once over too @tomwayson?

pranavkulkarni · 2019-05-08T22:01:31Z

Thanks @jgravois I will address the points you raised here. I will also beautify comments.

jgravois · 2019-05-13T17:07:56Z

package-lock.json

@@ -8116,6 +8116,7 @@
            "color-convert": "^1.9.0"
          }
        },
+<<<<<<< HEAD


this is what Travis is choking on.

besides fixing it manually, you're also welcome to just delete the file and rerun npm install in the root of the repo to recreate it.

jgravois · 2019-05-14T00:38:08Z

i'll release Hub.js v2.1.0 tomorrow.

pranavkulkarni force-pushed the f/search-package branch from 3306f48 to 7907338 Compare April 11, 2019 18:08

pranavkulkarni requested review from jgravois, haoliangyu and dmfenton April 12, 2019 18:28

jgravois previously requested changes Apr 12, 2019

View reviewed changes

haoliangyu reviewed Apr 16, 2019

View reviewed changes

pranavkulkarni force-pushed the f/search-package branch from b34b790 to ebe40fb Compare April 30, 2019 14:53

jgravois mentioned this pull request May 1, 2019

F/search package #161

Closed

jgravois self-requested a review May 8, 2019 00:00

jgravois requested a review from tomwayson May 8, 2019 00:02

pranavkulkarni changed the title ~~[WIP] search package~~ search package May 8, 2019

jgravois approved these changes May 8, 2019

View reviewed changes

pranavkulkarni added 10 commits May 10, 2019 15:38

search pkg basics

7f86c83

formatItemCollection added

8f73150

sort field transform

3a80aa0

functions renamed, removed es2017

6529131

serialize func added

3fed6db

structural changes

20fda3d

working serialize, searchItems

5594898

agg handled correctly

ab8324c

item search categories (collections)

7d1dc4d

folder structuring, efficient aggregations

c2a1742

pranavkulkarni added 15 commits May 10, 2019 15:38

pagination using start and num

cdbdda0

merged with v2.0.0 branch, uses latest arcgis-rest-js features

86d7075

added a few tests

a3f2a89

more tests, improved comments, fixed collection bug

ed521dc

added tests for filters

d77ecc9

added more tests

b971191

100% test coverage achieved

3680c44

moved test.html

e681050

downloadable types from common

aebc17d

downloadable types and typeKeywords moved to common pks

3964f97

page namespace with size and start

af9a5e9

api Types moved to common

a1859f9

renamed some functions, better doc

a9c1a8e

search with facets and formatting

39393bd

uses portal 2.0.1

173606e

pranavkulkarni force-pushed the f/search-package branch from 2342733 to 173606e Compare May 10, 2019 19:38

pranavkulkarni added 4 commits May 10, 2019 16:30

search test modified

f7af03e

hide some functions in docs

0672748

added Readme

78f440f

format readme

2fae376

jgravois reviewed May 13, 2019

View reviewed changes

pranavkulkarni and others added 4 commits May 13, 2019 15:48

re-instate package-lock

56b142b

black magic

c50cde5

any for the TS gods

12f89d8

update tests

4de4434

jgravois merged commit 722a931 into master May 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

search package #152

search package #152

pranavkulkarni commented Apr 9, 2019 •

edited

Loading

codecov bot commented Apr 9, 2019 •

edited

Loading

jPurush commented Apr 9, 2019

pranavkulkarni commented Apr 11, 2019

jgravois left a comment

haoliangyu left a comment

jgravois commented May 8, 2019

pranavkulkarni commented May 8, 2019

jgravois left a comment •

edited

Loading

jgravois May 8, 2019

jgravois May 8, 2019

jgravois May 8, 2019 •

edited

Loading

jgravois May 8, 2019

jgravois May 8, 2019

jgravois May 8, 2019

pranavkulkarni May 9, 2019

jgravois May 8, 2019

pranavkulkarni May 9, 2019 •

edited

Loading

jgravois May 9, 2019

jgravois commented May 8, 2019

pranavkulkarni commented May 8, 2019

jgravois May 13, 2019

jgravois commented May 14, 2019


		import { ISearchOptions } from "@esri/arcgis-rest-portal";

		export interface ISearchParams extends ISearchOptions {

		@@ -0,0 +1,21 @@
		import { getProp } from "@esri/hub-common";
		const apiTypes = ["Feature Service", "Map Service", "Image Service"];

search package #152

search package #152

Conversation

pranavkulkarni commented Apr 9, 2019 • edited Loading

codecov bot commented Apr 9, 2019 • edited Loading

Codecov Report

jPurush commented Apr 9, 2019

pranavkulkarni commented Apr 11, 2019

jgravois left a comment

Choose a reason for hiding this comment

haoliangyu left a comment

Choose a reason for hiding this comment

jgravois commented May 8, 2019

pranavkulkarni commented May 8, 2019

jgravois left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgravois May 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pranavkulkarni May 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgravois commented May 8, 2019

pranavkulkarni commented May 8, 2019

Choose a reason for hiding this comment

jgravois commented May 14, 2019

pranavkulkarni commented Apr 9, 2019 •

edited

Loading

codecov bot commented Apr 9, 2019 •

edited

Loading

jgravois left a comment •

edited

Loading

jgravois May 8, 2019 •

edited

Loading

pranavkulkarni May 9, 2019 •

edited

Loading