Add support for contextual bandits #64
Conversation
| @@ -29,7 +29,7 @@ | |||
| }, | |||
| "homepage": "https://github.com/Eppo-exp/node-server-sdk#readme", | |||
| "dependencies": { | |||
| "@eppo/js-client-sdk-common": "3.0.6", | |||
| "@eppo/js-client-sdk-common": "3.5.0", | |||
| import * as td from 'testdouble'; | ||
|
|
||
| import apiServer, { TEST_SERVER_PORT } from '../test/mockApiServer'; | ||
| import apiServer, { TEST_BANDIT_API_KEY, TEST_SERVER_PORT } from '../test/mockApiServer'; |
There was a problem hiding this comment.
This test actually spins up a server vs mocking fetch responses. Since different tests need different UFC files served, I'm switching off the provided SDK key.
Any client initialized with this key will be served the UFC file with bandit flags.
| throw new Error(`Unknown variation type: ${variationType}`); | ||
| } | ||
| describe('Shared UFC General Test Cases', () => { | ||
| const testCases = testCasesByFileName<IAssignmentTestCase>(ASSIGNMENT_TEST_DATA_DIR); |
There was a problem hiding this comment.
Makes for nice, unique test case names / output
| @@ -236,9 +302,9 @@ describe('EppoClient E2E test', () => { | |||
| }; | |||
|
|
|||
| it('retries initial configuration request before resolving', async () => { | |||
| td.replace(HttpClient.prototype, 'get'); | |||
| td.replace(HttpClient.prototype, 'getUniversalFlagConfiguration'); | |||
There was a problem hiding this comment.
our http client interface now has two methods, one for the UFC and one for bandit model parameters
| clientInstance = new EppoClient(configurationStore, requestConfiguration); | ||
| clientInstance.setLogger(config.assignmentLogger); | ||
| const flagConfigurationStore = new MemoryOnlyConfigurationStore<Flag>(); | ||
| const banditVariationConfigurationStore = new MemoryOnlyConfigurationStore<BanditVariation[]>(); |
There was a problem hiding this comment.
This store is needed to easily know if a flag has any bandits, which we want for handling an edge case of empty actions being passed to getBanditAction()
| clientInstance.setLogger(config.assignmentLogger); | ||
| const flagConfigurationStore = new MemoryOnlyConfigurationStore<Flag>(); | ||
| const banditVariationConfigurationStore = new MemoryOnlyConfigurationStore<BanditVariation[]>(); | ||
| const banditModelConfigurationStore = new MemoryOnlyConfigurationStore<BanditParameters>(); |
There was a problem hiding this comment.
This store holds the nitty gritty model parameters (coefficients, gamma, etc.)
| @@ -227,6 +254,66 @@ describe('EppoClient E2E test', () => { | |||
| }); | |||
| }); | |||
|
|
|||
| describe('Shared Bandit Test Cases', () => { | |||
There was a problem hiding this comment.
Bandits in the Node SDK!
| const ufcFile = req.url.includes(TEST_BANDIT_API_KEY) | ||
| ? MOCK_FLAGS_WITH_BANDITS_RESPONSE_FILE | ||
| : MOCK_UFC_RESPONSE_FILE; |
There was a problem hiding this comment.
Some tests use one file, some use the other; use the API key to decide which to serve
| // Do this check in addition to assertions to provide helpful information on exactly which | ||
| // evaluation failed to produce an expected result | ||
| if ( | ||
| banditAssignment.variation !== subject.assignment.variation || | ||
| banditAssignment.action !== subject.assignment.action | ||
| ) { | ||
| console.error(`Unexpected result for flag ${flagKey} and subject ${subject.subjectKey}`); | ||
| } |
Eppo Internal:
🎟️ Ticket: FF-2021 - Add bandits to Node SDK
🗺️ Design Review: Bandits Engineering Design
Motivation and Context
This change brings contextual multi-armed bandit support to our Node SDK, as described in the high-level documentation.
Description
Updates the Node JS SDK to use the updated shared common JavaScript client which has bandit support. This includes its new public function
getBanditAction().How has this been tested?
Added top-level tests which exercise
getBanditActin()using the shared common test data.