-
Notifications
You must be signed in to change notification settings - Fork 465
Asset-swapper aggregator utils #2353
Asset-swapper aggregator utils #2353
Conversation
4f93259
to
e94dc90
Compare
|
6170fbd
to
b2ac850
Compare
b2ac850
to
e3d6017
Compare
e3d6017
to
de451c7
Compare
...IMPROVE_ORDERS_OPTS_DEFAULTS, | ||
...opts, | ||
}; | ||
const [orderInfos, dexQuotes] = await queryNetworkAsync( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thoughts on using DevUtils.getOrderRelevantState
in the Sampler and returning [orderInfos, fillableTakerAssetAmount, isValidSignature]
.
We perform this to ensure the order is valid and check how much is fillable. We could skip that check a by combining it in the Sampler and reduce an RPC call.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I'm down for something like that. It looks like there's some more complex pruning rules being applied. So maybe the sampler contract can query those states and pass them into an orderFilter
callback in opts
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, really all we care about is how much taker asset is fillable. I think I'll just have the contract return that quantity, which will also be zero if the signature is invalid or if the order status is not FILLABLE
.
if (!provider) { | ||
throw new Error(AggregationError.MissingProvider); | ||
} | ||
const sampler = new ERC20BridgeSamplerContract(SAMPLER_ADDRESS, provider); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd even consider just passing in the entire ERC20BridgeSampler
, won't have to deal with optional providers here or in ImproveOrdersOpts
/** | ||
* Create ERC20Proxy asset data. | ||
*/ | ||
export function createERC20AssetData(tokenAddress: string): string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can re-use DevUtils
here which performs this pure
in memory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hate DevUtils
but OK 🙄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On second thought, how about we wait until DevUtils
is sorted out since switching this one function won't really do me any favors right now.
/** | ||
* Create ERC20Bridge asset data. | ||
*/ | ||
export function createBridgeAssetData(tokenAddress: string, bridgeAddress: string, bridgeData: string): string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should add an asana task to get this encoding/decoding into DevUtils
and re-use from that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(if it fits)
packages/asset-swapper/package.json
Outdated
@@ -43,6 +43,7 @@ | |||
"@0x/assert": "^2.2.0-beta.2", | |||
"@0x/contract-addresses": "^3.3.0-beta.4", | |||
"@0x/abi-gen-wrappers": "^5.4.0-beta.3", | |||
"@0x/contracts-erc20-bridge-sampler": "^1.0.0-beta.1", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO add this to abi-gen-wrappers
/export from contract-wrappers
. These have pruned artifacts and won't bloat out the asset-swapper bundle size.
nativeOrders: OrderWithoutDomain[], | ||
takerAmount: BigNumber, | ||
opts?: Partial<ImproveOrdersOpts>, | ||
): Promise<OrderWithoutDomain[]> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this correct in terms of turning it into a WalletSignature
?
const improvedOrders = await improveMarketBuyAsync(prunedOrders, assetFillAmount, {
provider: this.provider,
});
const improvedSignedOrders = improvedOrders.map(i => ({
...i,
signature: '0x04',
chainId: this.chainId,
exchangeAddress: this._contractAddresses.exchange,
}));
IIRC
Wallet = 0x04
EIP1271Wallet = 0x07
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yap!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, for the bridge orders. You still have to map the native orders to their respective (probably non-Wallet
) signatures. I'll handle signatures in the upcoming iteration, though, so it'll be moot.
opts?: Partial<ImproveOrdersOpts>, | ||
): Promise<OrderWithoutDomain[]> { | ||
if (nativeOrders.length === 0) { | ||
throw new Error(AggregationError.EmptyOrders); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the flow for just retrieving a bunch of BridgeOrders
if there are no NativeOrders
for this asset pair?
Just use sampleBuys
/sampleSells
directly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would just pass in an empty order to keep the entry point the same.
@@ -0,0 +1,800 @@ | |||
import { IERC20BridgeSamplerContract } from '@0x/contracts-erc20-bridge-sampler'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dekz highlighted above, but will need to rebase this PR with development
we switched back to using @0x/contract-wrappers
for bundle size reasons.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks awesome! Only minor feedback on the design of the algorithm.
Thought though: @dekz @dorothy-zbornak
While this algorithm searches for optimal paths with random walks, can we optimize further with a linear algebra interpretation of the paths?
I am thinking that with some pruning we can get a rather sparse matrix representation of the edges and Fill Paths, with some modified form of dijkstra's algorithm or some other optimization algorithm and an efficient sparse matrix typescript library, we may be able to see performance gains?
Just a shower thought, maybe for a later upgrade if performance becomes an issue.
import { constants } from '../constants'; | ||
|
||
const { NULL_BYTES, NULL_ADDRESS } = constants; | ||
const ZERO = new BigNumber(0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit. these should be constants.ts
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
`Eh?~ nvm
parent?: Fill; | ||
// Arbitrary data to store in this Fill object. Used to reconstruct orders | ||
// from paths. | ||
data?: any; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feel like a more verbose name here other than data would help
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should actually just be FillData
, if we need more data entries, we can always add it.
We can use a union type to capture all of the 'types' that data
can be.
* Dust amount, as a fraction of the fill amount. | ||
* Default is 0.01 (100 basis points). | ||
*/ | ||
dustThreshold: number; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit. prefer dustAmountThreshhold
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
idk, we seem to use amount
for flat amounts, whereas this is a %. Maybe dustFractionThreshold
?
return path; | ||
} | ||
|
||
function createBuyPathFromNativeOrders(orders: SignedOrderWithoutDomain[], fillableAmounts: BigNumber[]): Fill[] { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we reuse some of the logic from above? (as in dedup the logic) Not a big issue
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Turns out these are subtly different enough to make it not worth it.
currentPathOutput: ZERO, | ||
currentPathFlags: 0, | ||
}; | ||
// Visit all valid combintations of fills to find the optimal path. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// Visit all valid combintations of fills to find the optimal path. | |
// Visit all valid combinations of fills to find the optimal path. |
const optimalPath = optimizer.optimize( | ||
sortFillsByPrice(allFills), | ||
takerAmount, | ||
pickOptimalPath(allPaths, takerAmount), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it possible to move this pickOptimalPath
logic into FillsOptimizer
? Seems like it makes more sense there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pickOptimalPath
just picks the shortest path of those passed in, but FillsOptimizer
's job is to actually generate new paths so idk if that's a great fit.
* Convert a source to a canonical address used by the sampler contract. | ||
*/ | ||
export const SOURCE_TO_ADDRESS: { [key: string]: string } = { | ||
[ERC20BridgeSource.Eth2Dai]: '0x39755357759cE0d7f32dC8dC45414CCa409AE24e', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How would y'all feel about putting these in a central location, like contract-addresses
? (or even just a JSON file in here called contract-addresses.json
with mappings for the various networks)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, for sure, once these are actually all deployed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just realized these are DEX addresses. Yeah, idk, feels weird to put non 0x stuff in contract-addresses
. 🤔
/** | ||
* Class for finding optimized fill paths. | ||
*/ | ||
export class FillsOptimizer { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know that AMM's like the one used by Uniswap will give you a worse price the more you fill. Is this the case for all of our sources?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As a rule, yes. In practice, we often see quotes up to and beyond $10k, for high liquidity pairs, that are virtually linear (i.e., rates are essentially constant).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, as a rule except for maybe Kyber, which is why we fall back to the flood fill.
Yeah, there is no doubt we can optimize this algo to reduce the space complexity. The pruning done here is not super aggressive. One obvious one we can do for buys is avoid traversing paths that would exceed the global minimum cost. There are some issues with using LA to find ideal splits, like not being able to partially fill orders (except for the last) with market functions, so we might have to go down the route of having a custom forwarder for Another thing worth noting is that in most cases (knock on wood), this algo does approximate the globally optimal solution early on, because the solution for combining convex quotes should just be discoverable using a simple greedy algorithm. Because the fill nodes are sorted by price, this solution is walked over sooner rather than later. |
b0bd256
to
eece31f
Compare
f06b7b9
to
fc84a08
Compare
fc84a08
to
1e2de2a
Compare
1e2de2a
to
8389937
Compare
Adding back the [WIP] tag, to finish the work with integrating |
d81bac0
to
c135945
Compare
}, | ||
{ | ||
"note": "Add `IERC20BridgeSampler` wrapper", | ||
"pr": 2353 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this needs a new entry; current entry is already published
@@ -5,6 +5,10 @@ | |||
{ | |||
"note": "Add `IAssetData` artifact", | |||
"pr": 2373 | |||
}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this needs a new entry right? current entry is already published
packages/types/CHANGELOG.json
Outdated
@@ -14,6 +14,10 @@ | |||
{ | |||
"note": "Add `ERC20BridgeAssetData`", | |||
"pr": 2373 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same; this needs a new entry
Thanks for catching that @xianny. I pushed a fix to that. |
97fdc35
to
f49cb9a
Compare
f49cb9a
to
72be518
Compare
34447d2
to
13fc6a7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me, I think we can do a follow up clean up/refactor once this is shipped. Most of my comments are subjective styling.
One refactor which would be great to have is in the tests which are pretty complex to follow (especially with the dependance on ERC20BridgeSource being a num enum.
From a blackbox perspective this has been performing very competitively, so mad props to @dave4506 and @dorothy-zbornak.
@@ -1,41 +1,73 @@ | |||
import { Order } from '@0x/types'; | |||
import { BigNumber } from '@0x/utils'; | |||
import * as heartbeats from 'heartbeats'; | |||
import * as _ from 'lodash'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're not using lodash in this file.
import * as _ from 'lodash'; |
// tslint:disable-next-line:prefer-function-over-method | ||
public async getProtocolFeeMultiplierAsync(): Promise<BigNumber> { | ||
public getProtocolFeeMultiplier(): BigNumber { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd consider keeping this async
for the future, we might want to poll every few minutes. Keeping it async now would prevent a breaking future change. Not a huge deal though.
}; | ||
} | ||
|
||
function getSourceFromAssetData(assetData: string): ERC20BridgeSource { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a useful function for data purposes (determining the composition). We should move it into utils.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably not this version though; it's not very thorough.
function getSourceFromAddress(sourceAddress: string): ERC20BridgeSource { | ||
for (const k of Object.keys(SOURCE_TO_ADDRESS)) { | ||
if (SOURCE_TO_ADDRESS[k].toLowerCase() === sourceAddress.toLowerCase()) { | ||
return parseInt(k, 10) as ERC20BridgeSource; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO I don't think we should rely on the enum
default ints. It's definitely bitten me before, i.e
if (ERC20BridgeSource.Native) // if (0)
There is a lot of logic dependent on this though so let's have a TODO item to replace this with String enum
s
queryOrdersAndSampleBuys: (orders, signatures, sources, fillAmounts) => [ | ||
orders.map(o => o.makerAssetAmount), | ||
sources.map(s => | ||
shortZip(fillAmounts, rates[getSourceFromAddress(s)]).map(([f, r]) => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This hurt my brain 🤣
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can rewrite this to be clearer.
}); | ||
} | ||
|
||
function getOrderTokens(order: SignedOrder): [string, string] { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO assetDataUtils
is back
* Common exception messages thrown by aggregation logic. | ||
*/ | ||
export enum AggregationError { | ||
NoOptimalPath = 'no optimal path', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NoOptimalPath = 'no optimal path', | |
NoOptimalPath = 'NO_OPTIMAL_PATH', |
*/ | ||
export enum AggregationError { | ||
NoOptimalPath = 'no optimal path', | ||
EmptyOrders = 'empty orders', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
EmptyOrders = 'empty orders', | |
EmptyOrders = 'EMPTY_ORDERS', |
public async getSignedOrdersWithFillableAmountsAsync( | ||
signedOrders: SignedOrder[], | ||
): Promise<SignedOrderWithFillableAmounts[]> { | ||
const signatures = _.map(signedOrders, o => o.signature); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const signatures = _.map(signedOrders, o => o.signature); | |
const signatures = signedOrders.map(o => o.signature); |
import { DevUtilsContract } from '@0x/contract-wrappers'; | ||
import { orderCalculationUtils } from '@0x/order-utils'; | ||
import { OrderStatus, SignedOrder } from '@0x/types'; | ||
import * as _ from 'lodash'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
import * as _ from 'lodash'; |
`@0x/contract-addresses`: Make kyber lowercase.
7a24477
to
7d02e54
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* Create ERC20Bridge asset data. | ||
*/ | ||
export function createBridgeAssetData(tokenAddress: string, bridgeAddress: string, bridgeData: string): string { | ||
const encoder = AbiEncoder.createMethod('ERC20Bridge', [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO once we get bridge stuff in assetDataUtils
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's already in there :)
b4292b8
to
12f8cb0
Compare
* `@0x/asset-swapper`: Add ERC20Bridge aggregator library. * `@0x/asset-swapper`: Finish off `aggregate.ts`. * `@0x/types`: Add `OrderWithoutDomain` type. * `@0x/asset-swapper`: Add testing infra for sampler/aggregator. * `@0x/types`: Add `SignedOrderWithoutDomain` type. * `@0x/asset-swapper`: Update aggregator to take and return orders with signatures. * `@0x/asset-swapper`: Fix broken aggregator tests. * `@0x/asset-swapper`: Pass the sampler contract into aggregator entry points. * `@0x/contract-artifacts`: Add `IERC20BridgeSampler` artifact. * `@0x/contract-wrappers`: Add `IERC20BridgeSampler` wrapper. * `@0x/asset-swapper`: Address review comments. * fixed testing * refactored aggregate.ts and embeded into asset-swapper * added adjusted rates for taker and maker fees * remove PrunedSignedOrders * updated contract-addresses and addressed some other todos * streamlined logic * patched in lawrences changes * renamed aggregator utils and removed market_utils.ts * added ack heartbeats * fixed bug * patches * added dummy order things * Dummy with valid sig * Tweak gas price calculation to wei * added test coverage and fixed bugs * fixed migrations * Fix CHANGELOGs and types export * Deploy latest ERC20BridgeSampler on Mainnet * `@0x/types` Revert CHANGELOG. * `@0x/asset-swapper`: Address review comments. `@0x/contract-addresses`: Make kyber lowercase. * made protocol fee multiplier async * `@0x/asset-swapper: Fix build errors and do some code cleanup. * use assetDataUtils where possible
Description
Adds aggregator utils to
@0x/asset-swapper
, which can be found insrc/utils/aggregate.ts
.Improving Quotes
The main functions of interest are
improveMarketSellAsync()
andimproveMarketBuyAsync()
. Each takes native orders, a fill amount, and some options. They will poll on-chain DEXes and return a mixture of native and bridge orders that can be passed intomarketBuy()
ormarketFill()
on the Exchange contract.Options
An options object can be passed into either function, which control slippage on bridge orders, sources used, and performance of the mixing algorithm.
Changes to the
ERC20BridgeSampler
contract.Previously,
queryOrdersAndSampleSells/Buys()
would return theOrderInfo
for each order passed in, along with DEX samples. Now it callsDevUtils.getOrderRelevantStates()
and returns the actual fillable maker or taker amount (depending on whether it's a buy or sell). Also, if the order status is notFILLABLE
or the signature is invalid, this amount will be zero.TODO
asset-swapper
's quoting logic.aggregate.ts
are wrong and will need to be updated once the final versions of the bridges have been deployed.@0x/contract-addresses
nativeOrders
pass in a "dummy order" such that swapper now uses only other DEX sources to prepare quote.Testing instructions
Types of changes
Checklist:
[WIP]
if necessary.