[#172885142] delete user data cli command #53

balanza · 2020-05-18T18:12:41Z

Includes:

cli command
redis session purge
block/unblock user during the execution

How to use

yarn build && REDIS_URL=redis://url-to-redis node dist/UserDataDeleteOrchestrator/cli.js <FISCAL_CODE>

To implement yet

~~change user data processing status~~
bundle user data and save it to a dedicated storage
delete all user data

pagopa-github-bot · 2020-05-18T18:14:14Z

Affected stories

🌟 #172885142: Come CIT, voglio che quando la procedura di eliminazione dei miei dati è avviata, mi venga impedito l'accesso all'app, in modo da non ritrovarmi dati inconsistenti

New dependencies added: redis and redis-clustr.

redis

Author: Matt Ranney

Description: A high performance Redis client.

Homepage: https://github.com/NodeRedis/node-redis


Created	over 9 years ago
Last Updated	3 months ago
License	MIT
Maintainers	3
Releases	103
Direct Dependencies	`denque`, `redis-commands`, `redis-errors` and `redis-parser`
Keywords	database, redis, transaction, pipelining, performance, queue, nodejs, pubsub and backpressure

This README is too long to show.

redis-clustr

Author: Unknown

Description: Redis cluster client

Homepage: https://github.com/gosquared/redis-clustr#readme


Created	over 6 years ago
Last Updated	about 1 year ago
License	MIT
Maintainers	4
Releases	23
Direct Dependencies	`cluster-key-slot`, `denque` and `redis`
Keywords	redis, cluster and database

README

redis-clustr

This module is a relatively thin wrapper around the node redis client to enable use of Redis Cluster. It tries to be as unobtrusive as possible - mimicing the behaviour of the node_redis client.

Usage

var RedisClustr = require('redis-clustr');

var redis = new RedisClustr({
  servers: [
    {
      host: '127.0.0.1',
      port: 7000
    }
  ]
});

redis.set('key', 'value');

Servers

Servers in the cluster will be automatically connected to (via the response of cluster slots). Of course, to allow discovery there must be at least one server specified in the configuration.

Client creation

By default, clients will be created using Redis.createClient(port, host). This can be overridden by providing a function which must return a node_redis client. Clients are cached so only one connection will be made to each server.

var RedisClustr = require('redis-clustr');
var RedisClient = require('redis');
var redis = new RedisClustr({
  servers: [...],
  createClient: function(port, host) {
    // this is the default behaviour
    return RedisClient.createClient(port, host);
  }
});

Options

var RedisClustr = require('redis-clustr');
var redis = new RedisClustr({
  servers: [...],
  slotInterval: 1000, // default: none. Interval to repeatedly re-fetch cluster slot configuration
  maxQueueLength: 100, // default: no limit. Maximum length of the getSlots queue (basically number of commands that can be queued whilst connecting to the cluster)
  queueShift: false, // default: true. Whether to shift the getSlots callback queue when it's at max length (error oldest callback), or to error on the new callback
  wait: 5000, // default: no timeout. Max time to wait to connect to cluster before sending an error to all getSlots callbacks
  slaves: 'share', // default: 'never'. How to direct readOnly commands: 'never' to use masters only, 'share' to distribute between masters and slaves or 'always' to  only use slaves (if available)
  createClient: function(port, host, options) {
    return require('redis').createClient(port, host, options);
  }, // default: redis.createClient. Function used to connect to redis, called with arguments above
  redisOptions: {
    // options passed to the node_redis client https://github.com/NodeRedis/node_redis#options-is-an-object-with-the-following-possible-properties
    retry_max_delay: 500
    // etc
  }
});

Supported functionality

Slot reallocation

Supported - when a response is given with a MOVED error, we will immediately re-issue the command on the other server and run another cluster slots to get the new slot allocations. ASK redirection is also supported - we wil re-issue the command without updating the slots. TRYAGAIN responses will be retried automatically.

Multi / Exec (Batch)

Multi commands are supported but treated as a batch of commands (not an actual multi) and the response is recreated in the original order. Commands are grouped by node and sent as node_redis batches

Multi-key commands (`del`, `mget` and `mset`)

Multi-key commands are also supported and will be split into individual commands (using a batch) then have the response recreated. Only del, mget and mset are supported at the moment.

Pub/Sub

Pub/Sub is fully supported. When subscribe is used, a new client will be created (connected to a random node). This client is shared for all subscriptions.

var RedisClustr = require('redis-clustr');
var redis = new RedisClustr({...});

redis.on('message', function(channel, message) { /* ... */ });

redis.subscribe('my-channel', function(err) {
  redis.publish('my-channel', 'have a lovely day!');
});

Errors

Just like node_redis, listen to the error event to stop your application from crashing due to errors. Redis Clustr automatically intercepts connection errors and will try to reconnect to the server.

Generated by 🚫 dangerJS

gunzip · 2020-05-18T21:32:44Z

UserDataDeleteOrchestrator/session-utils/redisSessionStorage.ts

+    const deleteSessionTokens = new Promise<Either<Error, true>>(resolve => {
+      // Remove the specified key. A key is ignored if it does not exist.
+      // @see https://redis.io/commands/del
+      this.redisClient.del(


can you put some logging for every ops ? ie:

getting user sessions... <session1> ... <session2> ... deleting SESSIONIFO-< token> ì ... deleting USERMETA-<CF> ...

gunzip · 2020-05-18T21:33:24Z

UserDataDeleteOrchestrator/cli.ts

+): TaskEither<Error, true> => notImplementedTask("setUserDataProcessingStatus");
+
+// after the operation, unblock the user to allow another login
+const post = (fiscalCode: FiscalCode): TaskEither<Error, true> =>


I'd rather call these blockUser / unblockUser

codecov-commenter · 2020-05-19T06:52:05Z

Codecov Report

Merging #53 into master will decrease coverage by 2.52%.
The diff coverage is 79.24%.

@@            Coverage Diff             @@
##           master      #53      +/-   ##
==========================================
- Coverage   88.81%   86.29%   -2.53%     
==========================================
  Files          21       26       +5     
  Lines         608      868     +260     
  Branches       37       55      +18     
==========================================
+ Hits          540      749     +209     
- Misses         67      118      +51     
  Partials        1        1

Impacted Files	Coverage Δ
CreateService/handler.ts	`85.18% <0.00%> (ø)`
UpdateService/handler.ts	`90.47% <0.00%> (ø)`
GetService/handler.ts	`86.95% <50.00%> (+3.62%)`	⬆️
utils/zip.ts	`53.33% <53.33%> (ø)`
ExtractUserDataActivity/handler.ts	`77.46% <77.46%> (ø)`
utils/random.ts	`77.77% <77.77%> (ø)`
SetUserDataProcessingStatusActivity/handler.ts	`88.37% <88.37%> (ø)`
...eleteOrchestrator/GetUserDataProcessing/handler.ts	`88.46% <88.46%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ec3768...2f5a344. Read the comment docs.

gunzip · 2020-05-19T14:29:25Z

GetUserDataProcessingActivity/handler.ts

+   *
+   * @returns either an Error or the new created record
+   */
+  const fetchRecordFromDb = ({


can you extract this method outside the body of createGetUserDataProcessingActivityHandler ?

naming: fetchRecordFromDb ? -> getUserDataProcessingRequest

gunzip · 2020-05-19T14:52:05Z

GetUserDataProcessingActivity/handler.ts

@@ -0,0 +1,204 @@
+/**


Do we need this Activity ? I'm asking because the orchestrator will be triggered by an upcoming record inserted into user-data-requests collection so the input comes from the orchestrator client function bindings

(it's ok to use a query in the client.ts)

I basically did it for not have to construct here models and require env keys (as we do already with other activities). I guess we can eventually remove it if we find it unhelpful in future refactoring

yes we should convert this into an utility function only used by the cli (not by the orchestrator)

gunzip · 2020-05-19T16:33:42Z

GetUserDataProcessingActivity/handler.ts

@@ -140,32 +140,34 @@ const logFailure = (context: Context) => (
  }
 };

+/**


can you put these files inside the orchestrator / cli folder? (all top level folders are suposed to be functions and/or activities)

balanza added 2 commits May 18, 2020 20:06

delete user data cli command + clear redis session

58c884f

Merge branch 'master' into 172885142-manual-user-delete

57c181d

balanza requested a review from gunzip May 18, 2020 18:12

balanza requested review from cloudify and francescopersico as code owners May 18, 2020 18:12

gunzip reviewed May 18, 2020

View reviewed changes

balanza added 3 commits May 19, 2020 08:34

save blocked users as set

aa8ffc8

naming

1018f07

add logging

358e7e1

setting user data processing status

c7fa833

gunzip reviewed May 19, 2020

View reviewed changes

Merge branch 'master' into 172885142-manual-user-delete

beb1ba3

gunzip reviewed May 19, 2020

View reviewed changes

refactor

e080e0a

gunzip reviewed May 19, 2020

View reviewed changes

refactor

2f5a344

gunzip approved these changes May 21, 2020

View reviewed changes

gunzip merged commit 139e8b1 into master May 21, 2020

gunzip deleted the 172885142-manual-user-delete branch May 21, 2020 11:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#172885142] delete user data cli command #53

[#172885142] delete user data cli command #53

balanza commented May 18, 2020 •

edited

Loading

pagopa-github-bot commented May 18, 2020 •

edited

Loading

redis-clustr

Usage

Servers

Client creation

Options

Supported functionality

Slot reallocation

Multi / Exec (Batch)

Multi-key commands (`del`, `mget` and `mset`)

Pub/Sub

Errors

gunzip May 18, 2020

gunzip May 18, 2020

codecov-commenter commented May 19, 2020 •

edited

Loading

gunzip May 19, 2020

gunzip May 19, 2020

gunzip May 19, 2020

gunzip May 19, 2020

balanza May 19, 2020

gunzip May 19, 2020

gunzip May 19, 2020 •

edited

Loading

@@ @@ -140,32 +140,34 @@ const logFailure = (context: Context) => ( @@
                 }
               };
+              /**

[#172885142] delete user data cli command #53

[#172885142] delete user data cli command #53

Conversation

balanza commented May 18, 2020 • edited Loading

How to use

To implement yet

pagopa-github-bot commented May 18, 2020 • edited Loading

Affected stories

redis-clustr

Usage

Servers

Client creation

Options

Supported functionality

Slot reallocation

Multi / Exec (Batch)

Multi-key commands (del, mget and mset)

Pub/Sub

Errors

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented May 19, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gunzip May 19, 2020 • edited Loading

Choose a reason for hiding this comment

balanza commented May 18, 2020 •

edited

Loading

pagopa-github-bot commented May 18, 2020 •

edited

Loading

Multi-key commands (`del`, `mget` and `mset`)

codecov-commenter commented May 19, 2020 •

edited

Loading

gunzip May 19, 2020 •

edited

Loading