Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Koa style middleware: Hooks for any asynchronous method #932

Closed
daffl opened this issue Aug 16, 2018 · 18 comments · Fixed by #2255
Closed

Koa style middleware: Hooks for any asynchronous method #932

daffl opened this issue Aug 16, 2018 · 18 comments · Fixed by #2255

Comments

@daffl
Copy link
Member

daffl commented Aug 16, 2018

Hooks are a powerful way to control the flow of asynchronous methods. Koa popularized a new kind of middleware that allows an even more flexible approach to control this flow. I propose to adapt this pattern as a new async hook type.

Problem

Currently, Feathers hooks are split into three types, before, after and error:

feathers-hooks-old

While this pattern works quite well, it is a little cumbersome to implement functionality that needs to have access to the before, after and/or error flow at the same time. For example functionality for a profiler that logs the method runtime or extracting and storing data (e.g. associations) that will be used after the method call returns requires a before and an after hook.

Proposal

With async/await now being fully supported by all recent versions of Node - and the ability to fall back to Promises for Node 6 - we now have a powerful new way to create middleware that can control the whole flow within a single function.

This pattern has been implemented already in KoaJS for processing HTTP requests. However, similar to what I wrote before about Express style middleware, this also doesn't have to be limited to the HTTP request/response cycle but can apply to any asynchronous function, including Feathers services. Basically this new kind of hook will wrap around the following hooks and the service method call like this:

feathers-hooks-2 1

How it works

Additionally to the hook context async hooks also get an asynchronous next function that controls passing the flow to the next hook in the chain.

The following example registers an application wide before, after and error logger and profiler for all service method calls:

app.hooks({
  async: [
    async (context, next) => {
      try {
        const start = new Date().getTime();

        console.log(`Before calling ${context.method} on ${context.path}`);

        await next();

        console.log(`Successfully called ${context.method} on ${context.path}`, context.result);
        console.log('Calling ${context.method} on ${context.path} took ${new Date().getTime() - start}ms');
        return context;
      } catch(error) {
        console.error(`Error in ${context.method} on ${context.path}`, error.message);
        // Not re-throwing would swallow the error
        throw error;
      }
    }
  ]
});

The following example extracts the address association for a user create method and saves it with the userId reference:

app.service('users').hooks({
  async: {
    create: [
      async (context, next) => {
        const { app, data } = context;
        const { address } = data;

        context.data = _.omit(data, 'address');

        // Get the result from the context returned by the next hook
        // usually the same as accessing `context.result` after the `await next()` call
        // but more flexible and functional
        const { result } = await next();

        // Get the created user id
        const userId = context.result.id;

        // Create address with user id reference
        await app.service('addresses').create(Object.assign({
          userId
        }, address));

        return context;
      }
    ]
  }
});

async hook type

The core of async hooks would be implemented as an individual module that allows to add hooks to any asynchronous method. This has been inspired by the great work @bertho-zero has been doing in #924 to allow using hooks on other service methods. I created a first prototype in https://github.com/daffl/async-hooks (test case can be found here). It uses Koa's middleware composition module koa-compose.

async hooks would be implemented as an additional hook type but all hook dispatching will be done using the new middleware composition library. Existing before after and error hooks will be wrapped and registered in a backwards compatible way:

app.service('users').hooks({
  async: {
    all: [],
    create: [],
    find: []
    // etc.
  },

  // Continue using all your existing hooks here
  before: {
    all: [],
    find: []
    // etc.
  },

  after: {},

  error: {}
});

async hooks will have precedence so the execution order would be async -> before -> after. Even when only using the old hooks there are several advantages:

  • Better stack traces
  • Slightly better performance
  • Less custom code to maintain and the potential to collaborate with the Koa project

Migration path

There will be two breaking changes:

  1. The currently still supported (but undocumented and not recommended) function(context, next) signature for existing before, after and error hooks will no longer work.
  2. koa-compose does not allow to replace the entire context. Returning the context is still possible but other hooks have to retrieve it from the await next(); call with const ctx = await next();.
@daffl daffl added this to the Crow milestone Aug 16, 2018
@eddyystop
Copy link
Contributor

eddyystop commented Aug 17, 2018

I would hope this proposal can solve the 2 biggest issues I face with the common hooks using the Buzzard design.

Use case 1: More complex hooks often need to make a call on their own or another service. In my experience its often a get call and the hook needs the record as it appears in the table, e.g. fastJoin, softDelete, softDelete2, stashBefore.

Problems: (1) The get call will run the before and after hooks for the service, thus possibly modifying the record contents.
(2) The hooks on the called service could themselves perform service calls, and the problem escalates.
(3) I considered using a param.$raw flag, but its not elegant and breaks down quickly.
(4) The interaction between the more complex common hooks, authentication hooks and custom hooks results in a fair number of posted issues. People ask if various hooks are compatible together, e.g. cache + softDelete2 + restrictToOwner, cache + authentication. I reply "Try it and see."

Solution: (1) Service calls made on the server need to optionally skip all (some?) hooks.
(2) Allow before hooks to skip the remaining before and AFTER hooks.

Use case 2: When the cache hook, for example, finds a cache hit, it needs to set context.result and skip the following before hooks.

Problem: (1) Its not possible to both set the content and SKIP in the same hook. Two hooks would be needed and that's not elegant.
(2) Presently, cache sets context.result. Since neither common nor custom before hooks check context.result, the remaining before hooks run doing irrelevant processing on context.data, .query and perhaps making some service calls. This causes interaction issues.

@daffl daffl removed this from the Crow milestone Aug 18, 2018
@daffl
Copy link
Member Author

daffl commented Aug 18, 2018

This should definitely work for a caching hook. If it is one of the first registered, it can set context.result and not call next() which will skip all other hooks:

app.hooks({
  async: [
    async (context, next) => {
      const { app, method, params } = context;

      if(method === 'find' || method === 'get') {
        const key = JSON.stringify(context.params.query);
        const cached = app.cache[key];

        if(cached) {
          context.result = cached;

          // Just return the context, don't call `next`
          return context;
        }

        const { result } = await next();

        app.cache[key] = result;
      }
      
      return context;
    }
  ]
});

For skipping hooks we could add an additional parameter (similar to the returnHook flag) to method calls that indicates that it shouldn't run any of the hooks:

// Return the hook object
app.service('users').get(id, hooks.RETURN);

// Skip all hooks
app.service('users').get(id, hooks.SKIP);

To skip specific ones, I'm not sure if there is a way around wrapping them in a conditional hook or having a hook check it themselves.

@eddyystop
Copy link
Contributor

I would suggest the cache contain the records returned by the DB call itself. That way the hooks intended after the DB call can customize the response for the specifics of that call, e.g. the role of the user.

Perhaps the "after" hooks can do that.

@eddyystop
Copy link
Contributor

eddyystop commented Aug 18, 2018

Its been a while since I looked at Koa, but doesn't its middleware bubble downstream and then bubble upstream?

Perhaps you can consider running the async hooks before the 'before" hooks, and before the "after" hooks, and maybe (?) even before the "error" hooks. That would produce a similar downstream/upstream feature.

I assume type would still be before and after so the async function could know what it should do. It would be practical to have another property indicating if the hook was running as an async hook or not, so hooks can prevent themselves from being run in the wrong context.

Perhaps type should just be before, after, async_before and async_after.

@daffl
Copy link
Member Author

daffl commented Aug 18, 2018

Maybe it's not clear in the proposal but the idea is that Koa style middleware has access to the whole flow that can be controlled by calling next() so there is no explicit before or after:

// Before hook
const beforeHook = async (context, next) => {
  console.log('Before');

  await next();
}

// After hook
const afterHook = async (context, next) => {
  await next();

  console.log('After');
}

// Error hook
const errorHook = async (context, next) => {
  try {
    await next();
  } catch(error) {
    console.error('Error', error.message);
    throw error;
  }
}

// Before, after and error
const afterHook = async (context, next) => {
  try {
    console.log('Before');

    await next();

    console.log('After');
  } catch(error) {
    console.error('Error', error.message);
    throw error;
  }
}

@beeplin
Copy link
Contributor

beeplin commented Sep 3, 2018

Totally agree with the proposal. One suggestion: instead of using service.hooks({async: [function]}), just simply use service.hooks([function]). The new koa-style hook is meant to replace the old before/after/error hooks, and in the future there will be only one kind of hooks, so no need to wrap them in the async property.

@beeplin
Copy link
Contributor

beeplin commented Sep 3, 2018

BTW it would be nice if in the next major version we gave service.hooks an alias like service.registerHooks. .hooks is the only public service API that uses a noun as a method name.

@eddyystop
Copy link
Contributor

eddyystop commented Sep 4, 2018

I've been too busy to contribute on ths issue and will continue to be till the 12th.

However, as I mentioned above, its important a (async?) hook be able to get/find a record from its own or from another service, without any of the hooks on that service being run. So I'm assuming something like your hooks.SKIP or hooks.RETURN will be available for both get and find.

@bertho-zero
Copy link
Collaborator

I'm waiting for it impatiently.

@bertho-zero
Copy link
Collaborator

How do you want to proceed to attach hooks directly to methods?

The ideal would be to use the services even without app.

Does it mean that there will be hooks for every method, every service and every app?

@daffl
Copy link
Member Author

daffl commented Aug 23, 2019

Yes. The Readme of the prototype has a few examples for this:

const { hooks } = require('@feathersjs/hooks');

const addExclamation = context => {
  context.result += '!!!';
}

// On a function
const helloWithHook = hooks(async function() {
  return 'Hello world';
}, [ addExclamation ]);

console.log(await helloWithHook());

class HelloWorld {
  sayHello () {
    return 'Hello world';
  }
}

// On the class
hooks(HelloWorld, {
  sayHello: [ addExclamation ]
});

// On the object
const helloWorlder = new HelloWorld();

hooks(helloWorlder, {
  sayHello: [ addExclamation ]
});

// As a JavaScript or TypeScrip decorator
class HelloWorld {
  @hooks([ addExclamation ])
  sayHello () {
    return 'Hello world';
  }
}

@daffl
Copy link
Member Author

daffl commented Aug 23, 2019

Personally most looking forward to the decorator option because you see how everything fits together in a single place:

// As a JavaScript or TypeScript decorator
class HelloWorld {
  @hooks([
    reportErrors, 
    logRuntime,
    addExclamation
  ])
  async sayHello () {
    return 'Hello world';
  }
}

There will probably be a backwards compatible .hooks method as well that will also support the old before, after and error style hooks (so far I think it should be able to wrap them mostly backwards compatible).

@bertho-zero
Copy link
Collaborator

I think of something like that, what do you think? This could replace the all gradually.

@hooks([
  reportErrors, 
  logRuntime
])
class HelloWorld {
  @hooks([
    addExclamation
  ])
  async sayHello () {
    return 'Hello world';
  }
}

@daffl
Copy link
Member Author

daffl commented Aug 28, 2019

Absolutely. I have to look at how to make that work with a decorator but I think that's a pretty nice way to see everything that is happening.

@FossPrime
Copy link
Member

FossPrime commented Sep 28, 2019

Maybe it's not clear in the proposal but the idea is that Koa style middleware has access to the whole flow that can be controlled by calling next() so there is no explicit before or after:

// Before hook
const beforeHook = async (context, next) => {
  console.log('Before');

  await next();
}

// After hook
const afterHook = async (context, next) => {
  await next();

  console.log('After');
}

// Error hook
const errorHook = async (context, next) => {
  try {
    await next();
  } catch(error) {
    console.error('Error', error.message);
    throw error;
  }
}

// Before, after and error
const afterHook = async (context, next) => {
  try {
    console.log('Before');

    await next();

    console.log('After');
  } catch(error) {
    console.error('Error', error.message);
    throw error;
  }
}

This seems to hide hook type conversions... What if a service hook converts a patch call to an update. App hooks before next would have to be gated somehow...

@daffl
Copy link
Member Author

daffl commented Jan 6, 2020

I just published the first version of @feathersjs/hooks as a standalone module. You can find it (and all documentation) at https://github.com/feathersjs/hooks.

It is not integrated with Feathers core yet but can be used by itself to your ❤️'s content all over your existing applications.

@bertho-zero
Copy link
Collaborator

Best news of 2020, eager to see more.

@bertho-zero
Copy link
Collaborator

bertho-zero commented Jan 28, 2020

My wrappers:

function firstHook(context, next) {
  context.type = 'before';
  return next();
}

function lastHook(context, next) {
  context.type = 'after';
  return next();
}

function toBeforeHook(hook) {
  return async (context, next) => {
    await hook(context);
    await next();
  };
}

function toAfterHook(hook) {
  return async (context, next) => {
    await next();
    await hook(context);
  };
}

function beforeWrapper(...hooks) {
  return [firstHook, ...hooks.map(toBeforeHook)];
}

function afterWrapper(...hooks) {
  return [...hooks.reverse().map(toAfterHook), lastHook];
}

function wrap({ async = [], before = [], after = [] } = {}) {
  return [
    ...[].concat(async),
    firstHook,
    ...[].concat(before).map(toBeforeHook),
    ...[].concat(after).reverse().map(toAfterHook),
    lastHook
  ];
}

module.exports = {
  wrap,
  beforeWrapper,
  afterWrapper,
  toBeforeHook,
  toAfterHook
};

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants