GPII-3138: Update snapsets in data base #626

klown · 2018-07-10T20:11:55Z

@cindyli Here is the pull request that goes with the one in gpii-dataloader, gpii-ops/gpii-dataloader#6

Modified vagrantCloudBasedContainers.sh script to make use of the changes in the GPII-3138 branch of gpii/gpii-dataloader

Moved deleteSnapset.js from gpii-dataloader to universal's script folder.

Fixed some (grievous) typos.

Fixed erroneous call to fluid.error() -- replaced with fluid.log().

Removed hard-coded host ("localhost") and port for the CouchDB URL and used the actual value passed in on the command line.

- Replaced all occurrences of forEach() with fluid.each(). - Properly set up shell environment variable NODE_PATH.

Added more log messages.

gpii-bot · 2018-07-10T20:41:11Z

CI job failed: https://ci.gpii.net/job/universal-tests/945/

cindyli · 2018-07-10T22:05:28Z

scripts/deleteSnapsets.js

+var dbLoader = gpii.dataLoader;
+dbLoader.couchDbUrl = process.argv[2];
+if (!fluid.isValue(dbLoader.couchDbUrl)) {
+    fluid.log ("COUCHDB_URL environment variable must be defined");


Would be useful to also output the command usage: node deleteSnapsets.js $COUCHDBURL

cindyli · 2018-07-10T22:11:41Z

scripts/deleteSnapsets.js

+}
+fluid.log("COUCHDB_URL: '" + dbLoader.couchDbUrl + "'");
+dbLoader.prefsSafesViewUrl = dbLoader.couchDbUrl + "/_design/views/_view/findSnapsetPrefsSafes";
+dbLoader.gpiiKeyViewUrl = dbLoader.couchDbUrl + "/_design/views/_view/findGpiiKeysByPrefsSafeId";


What do you think to define the full url here to provide a general view: dbLoader.gpiiKeyViewUrl = dbLoader.couchDbUrl + "/_design/views/_view/findGpiiKeysByPrefsSafeId%22%gpiiKey%22". When this var is used later on, replace %gpiiKey with actual key values using fluid.stringTemplate().

Since we now use the get-all-gpiikeys view, this is no longer needed -- "overtaken by events".

cindyli · 2018-07-10T22:16:10Z

scripts/deleteSnapsets.js

+dbLoader.addGpiiKeysAndBulkDelete = function (snapSets, docsToRemove) {
+    fluid.each(snapSets, function (aSnapset) {
+        var gpiiKeyId = aSnapset.value._id;
+        fluid.log("Snapset: " + gpiiKeyId);


Output a more meaningful message such as "Finding GPII keys associated with the snapset prefs safe id: ".

I've updated all the log messages to be very simple, such as you have suggested. Actual snapset or gpiikeys internal information such as the "_id" field are no longer logged.

cindyli · 2018-07-10T22:29:44Z

scripts/deleteSnapsets.js

+};
+
+/**
+ * Delete the snapset Prefs Safes and their associated GPII Keys.


This function only deletes prefs safes, not gpii keys.

With the use of promises, this now does delete all prefs safes and associated gpii keys.

cindyli · 2018-07-10T22:30:21Z

scripts/deleteSnapsets.js

+        fluid.log("STATUS: " + res.statusCode);
+        fluid.log("HEADERS: " + JSON.stringify(res.headers, null, 2));
+        res.on('end', function () {
+            fluid.log('Batch deletion of snapsets');


snapsets -> snapset prefs safes.

Due to the refactoring, the log message includes "Prefs Safes", and also "GPII Keys".

cindyli · 2018-07-10T22:34:51Z

scripts/deleteSnapsets.js

+    batchDeleteReq.end();
+};
+
+dbLoader.snapSetsRequest = http.request(dbLoader.prefsSafesViewUrl, dbLoader.processSnapsets);


The couchdb url on the GPII cloud might be using https instead http. Will find out by testing with the developer cloud.

Any progress on this, @cindyli ?

Sorry about the confusion. I meant YOU will find out when testing with the cloud, not me. It's also worth to check with @mrtyler on this information since production/staging might act differently than dev clusters.

cindyli · 2018-07-10T23:14:03Z

scripts/deleteSnapsets.js

+        });
+        getGpiiKeysRequest.end();
+    });
+    dbLoader.doBatchDelete(docsToRemove);


Deleting all prefs safe records here at the end of the script doesn't guarantee it to be run as the last task due to the aync nature of http requests above for getting and deleting gpii keys.

Right, but it's another instance of "overtaken by events": The latest code handles the asynchrony.

amb26 · 2018-07-10T23:54:57Z

scripts/deleteSnapsets.js

+        var gpiiKeyId = aSnapset.value._id;
+        fluid.log("Snapset: " + gpiiKeyId);
+        var gpiiKeyViewUrl = dbLoader.gpiiKeyViewUrl + "?key=%22" + gpiiKeyId + "%22";
+        var getGpiiKeysRequest = http.request(gpiiKeyViewUrl, function (resp) {


Worrying lack of abstraction in all of this work. At the least there should be a wrapper for the process of invoking an HTTP request and waiting for responses from it as some kind of promise-producer. I don't want to slow down this work one atom so I'm not suggesting that it be re-expressed using https://github.com/fluid-project/kettle/blob/master/docs/DataSources.md#simple-example-of-using-an-http-datasource but something needs to happen to unwrap the tangle in this method which by line 88 we are 4 closures deep. It's hard to follow this expression of the algorithm, it is brittle, and seems prone to races which are already causing confusion. Also consider utilities like https://docs.fluidproject.org/infusion/development/PromisesAPI.html#fluidpromisesequencesources-options

amb26 · 2018-07-10T23:55:42Z

scripts/deleteSnapsets.js

+    var snapSetsString = "";
+    response.setEncoding("utf8");
+    response.on("data", function (chunk) {
+        snapSetsString += chunk;


mrtyler

IANA Javascript developer but to me this looks like it implements the dataloading strategy Cindy and I have discussed.

Two areas I want to make sure we're thinking about:

Idempotence - the goal is that we should be able to run the dataloader over and over again, and the end result will always be that the correct data is in the database (and any old data is removed). I think this implementation satisfies that goal, but please confirm for me Joseph/Cindy.
Failure modes - the goal is that a failure along the way (e.g. an HTTP timeout while communicating with couchdb) should NOT leave the database in a bad or inconsistent state.

This is a nuanced topic with many possible solutions that range from "cheap but disruptive" to "resilient but expensive". A few examples:

Current solution: drop entire database and re-populate from scratch
- Cheap!
- Very disruptive. Ignoring the showstopping problem -- this approach destroys any data that cannot be reloaded from canned snapsets -- a user will be unable to use the system from the moment the database is dropped until the Snapset the user wants is re-uploaded (on the order of 10s of seconds).
Proposed solution in this PR: delete all snapsets in a first pass, re-populate snapsets in a second pass
- More expensive (the cost of writing the code in this PR, basically)
- Less disruptive. This solves the showstopping problem above, and narrows the window where a user is affected (on the order of seconds).
Possible future solution: delete each snapset individually, then re-upload that snapset immediately
- A bit more expensive to implement
- A bit less disruptive. Narrows the window where a user is affected (on the order of deciseconds).

Note that these are all "happy paths". If the dataloading process halts due to an error after it has deleted a snapset but before it has re-uploaded that snapset, the window where users are affected grows to the time it takes the dataloader process to be run again (on the order of minutes or tens of minutes).

First, are my concerns clear? If not, I'm happy to elaborate and clarify, perhaps in a real-time chat.

Second, are my concerns worth considering now? The solution in this PR is better than what we have today, and deadlines are looming, so perhaps this is sufficient. My main purpose in raising these questions is so that everyone is thinking about failure modes and how to handle them since failure on the internet is inevitable :p.

mrtyler · 2018-07-12T18:08:09Z

scripts/deleteSnapsets.js

+    fluid.log ("COUCHDB_URL environment variable must be defined");
+    process.exit(1);
+}
+fluid.log("COUCHDB_URL: '" + dbLoader.couchDbUrl + "'");


COUCHDB_URL can (and usually will) contain credentials. Please do not write it to the log without sanitization.

It looks like url.parse may be helpful in reporting useful data like hostname without reporting sensitive data like password.

Thanks for pointing that out, @mrtyler. I've gone with logging the

protocol

host

port

pathname

I don't think any of those contain sensitive information, but I could be wrong. What do you think?

Looks good.

mrtyler · 2018-07-12T18:17:54Z

scripts/deleteSnapsets.js

+            });
+        });
+        getGpiiKeysRequest.on("error", function (e) {
+            fluid.log("Error finding snapsets' associated GPII Keys: " + e.message);


I think this should be "snapset's", unless the error message is going to contain errors from all the snapsets that failed?

The closest log message to this old one now has no apostrophe: "Error finding snapset Prefs Safes associated GPII Keys".

mrtyler · 2018-07-12T18:20:57Z

scripts/deleteSnapsets.js

+        fluid.log("STATUS: " + res.statusCode);
+        fluid.log("HEADERS: " + JSON.stringify(res.headers, null, 2));
+        res.on('end', function () {
+            fluid.log('Batch deletion of snapsets');


Maybe this is clearer as "Finished batch deletion of..."?

And this one is now "Bulk deletion completed."

GPII-3138: Add a couchDB view to return all GPII keys

gpii-bot · 2018-07-18T20:46:42Z

CI job failed: https://ci.gpii.net/job/universal-tests/968/

klown · 2018-07-20T16:42:29Z

@cindyli, @mrtyler, @amb26 I'm pushing the latest code that uses promises. But, this is not complete as I have yet to migrate the code that loads the snapsets and their keys using bash, as you suggested @cindyli. I wanted to test what I have so far using the docker image and see if there are any problems.

- Refactored deleteSnapsets.js to use promises. - Modified Dockerfile to remove dependency on NODE_ENV, as it is not needed by the gpii-dataloader script built into the docker image.

gpii-bot · 2018-07-20T17:35:39Z

CI job failed: https://ci.gpii.net/job/universal-tests/977/

cindyli · 2018-07-20T19:50:17Z

scripts/deleteSnapsets.js

+        dbLoader.snapsetPrefsSafes.push(aSnapset.value);
+    });
+    fluid.log("\tSnapset Prefs Safes marked for deletion.");
+    return dbLoader.snapsetPrefsSafes;


This line seems unnecessary because dbLoader.snapsetPrefsSafes is a global variable.

... and nothing actually uses the return value. Removed.

cindyli · 2018-07-20T19:50:40Z

scripts/deleteSnapsets.js

+        gpiiKeyRecords, dbLoader.snapsetPrefsSafes
+    );
+    fluid.log("\tGPII Keys associated with snapset Prefs Safes marked for deletion.");
+    return dbLoader.gpiiKeys;


Same as above.

cindyli · 2018-10-10T15:36:55Z

I believe these snapset data are already in the production and staging databases and marked as "user" type. We can do a one time cleanup to wipe them out before the new dataloader goes alive.

klown · 2018-10-10T15:45:36Z

Okay, I won't add that one time cleanup to the new dataloader.

amb26 · 2018-10-10T17:03:56Z