1.6.0 (2024-12-20)
1.5.1 (2024-12-12)
1.5.0 (2024-12-10)
- add dataset collection endpoint, script to scrape validator db (#78) (1b4ef9e)
- added script to inspect score (f9dd7e4)
- simulator (ed110e3)
- add VALIDATOR_MIN_STAKE in environment (84774a4)
- fixed from PR feedback (23d085a)
- fixed linter issue, and commitizen (553ad66)
- ground truth arr ordering (75f008a)
- minersim task scoring, validatorsim subtensor retry mechanism and dendrite forward timeout (956f5cf)
- negative stride error from scoring (#86) (50c86c6)
- scoring interval (94e050b)
- use correct dimension when calculating mean scores (2f30931)
- wandb missing final scores (#88) (0d8b93d)
- wandb now storing final scores correctly (#87) (affcd3e)
1.4.2 (2024-11-22)
- add retries for dojo api calls (461214a)
- extend delay (97a9b04)
- extend delay (69e1974)
- removed unused completions in miner's response to optimize network traffic (e76ce98)
1.4.1 (2024-11-07)
- add assert check (f849130)
- add unit test for reward_cubic (7a8483a)
- change nans to -1 instead of 0 (f15815c)
- potential scoring issue (335b40b)
- remove error log (e0b1a85)
- scoring shapes (dc15bb9)
- refactor logging, and clean synapase_history (8e36b5c)
1.4.0 (2024-10-31)
- add validation for expiry times (272efbb)
- add validation for expiry times (e75c9a5)
- ensure miner scores updated per func call, not per task (#62) (5339f09)
- remove unused function (a64b66e)
- update completions timing logic (5b8adfb)
1.3.3 (2024-10-28)
- disabled the _ensure_subtensor_ws_connected func (#52) (0142023)
- scores shapes (#55) (72c4f04)
- set wait for finalization = False (2911fb0)
1.3.2 (2024-10-27)
- add try except for calculating scores (53285ff)
- convert to np array and back to torch (fa7bacc)
- increase prisma tx timeout, poll tasks every 5 min (b109592)
- reuse weight_utils function (2e45a0a)
1.3.1 (2024-10-27)
- increase save_task timeout from 5 to 30s (5dc8171)
- set weights call hanging (638a3a5)
- set weights, mini batching update miner responses, update logs (#48) (2e72093)
1.3.0 (2024-10-26)
1.2.2 (2024-10-25)
1.2.1 (2024-10-25)
1.2.0 (2024-10-24)
1.1.7 (2024-10-23)
- min weights (f772ef2)
- runtimeerror and asyncio context already entered (#30) (c5b53fb)
- scoring mechanism use ground truth only, use L1 norm (#31) (4085dd8)
- updated auto-update script (#25) (0113498)
1.1.6 (2024-10-22)
1.1.5 (2024-10-20)
- gracefully shutdown auto_update (40240cd)
- simplify dojo setup (2bf0c76)
- updated makefile, and auto-update script (0d20bd4)
1.1.4 (2024-10-17)
- set weights even if all zeros (ca2332a)
1.1.3 (2024-10-16)
- add batching to DB calls, refactor in memory class attrs (#14) (4087de9)
- added batch_id count (fa66ea3)
- added log retention to avoid storage buildup (#16) (39415dd)
- change validator_hotkeys logic in monitoring func (066c683)
- dubious ownership issue (1c16d1d)
- fixed consensus scoring bugs (8d8cdf8)
1.1.2 (2024-10-15)
1.1.1 (2024-10-14)
1.1.0 (2024-10-12)
- add obfuscation to synthetic code outputs (79f0e24)
- allow pointing to env file, otherwise default to .env (6060707)
- add missing port (653ddfc)
- criteria options ordering, unit test warnings (26ce2e3)
- database initialisation errors (7c701d9)
- fixed _send_shuffled_requests return nested array of FeedbackRequest (6cce1cc)
- fixed on completion_id constriant error, and dojo_task_id not found while the overwriting miner response (adbe518)
- missing expire_at from vali to miner (aa296b1)
- revert docker compose and validator run (d50af40)
- scoring (7b1ea8e)
- uglifyjs not found error (9ab36b9)
- add retries to obfuscation (#5) (40204ed)
- add synthetic qa model validator, remove default 5/10 skip scoring (26a5344)
- shuffle responses sent to miners (b5133c3)
- update readme
- use public mainnet subtensor
- semantic release
- pip: requires pydantic v2
- add .env.example for reference (6d0a590)
- add aws lambda function template (3749baf)
- add aws mturk script (ed52c40)
- add base reward models for miners to use (c4f8f12)
- add blacklisting on miner side (28583ea)
- add cohen kappa and distance penalty (28e3cfe)
- add complete feedback loop for miner to send scores back to validator (6a81287)
- add data manager for ranking data (bf13235)
- add Dojo API caller (cfea834)
- add dojo task monitor on validator (8c8ff24)
- add dojo task tracker and update data on disk (6c21a10)
- add dojo task tracker, use synthetic data generator (d074fc0)
- add evals to get classification accuracy (62bd5fb)
- add fastapi routes (ed71ba0)
- add ground truth scoring for multi score (1fcfaa2)
- add heartbeat, ruff fixes (4bf04ca)
- add ICC metric for consensus scoring (30a5f6e)
- add image scoring (d55bc54)
- add LLM API based scoring, default to mixtral 8x7b (fb16c79)
- add mainnet docker compose opts (58da97f)
- add method to get response by request id (e3c3c1b)
- add methods to create PRs to huggingface dataset for review (7ba3f6e)
- add more datasets for evals (4649e67)
- add more models to generate answers from (90c9f7b)
- add new methods to data manager (d8a1234)
- add openai support (dbfcc44)
- add openrouter provider (26f212c)
- add scaleai code (e571660)
- add scoring method enum and simplify argparser (8042a98)
- add spearman scoring for synthetic ground truth (#37) (0a41c8e)
- add subscription key mgmt to cli (2b30027)
- add weights classification accuracy multiplier to scoring (6c5772c)
- added unit testing for validator (170d774)
- apply monkey patch to bt.logging (16af344)
- boost weights for human feedback (0a928b2)
- bump version (c34d086)
- calculate distance from ground truth, combine with consensus (cbe813f)
- calculate mean across different criteria and sum (0947a5a)
- calculate spearman correlation for scoring, add config args validation, update protocol (132b944)
- change python package name (8990325)
- change validator interval to every 30s (4052332)
- cli: add coldkey and hotkey path completions (ed2f36a)
- create dojo cli for miners (56615c9)
- create tasks with ranking/multi-score (aaeef16)
- enable huggingface/llm api scoring methods for miner (011bdc9)
- enable LLM API based evals, update config (ffea364)
- enable miners to run uvicorn server via miner.py (24406e2)
- enable synthetic prompt & responses generation for coding task (6fa0b73)
- external user (1037c80)
- flesh out readme details, provide screenshots (74587b1)
- forward consensus score to miners (c949ebe)
- get initial prompts & completions from synthetic dataset (3551591)
- group all functions under MTurkUtils, add event handler placeholder (99ff4f2)
- implement hashring to select miners (3ba4953)
- integrate automatically creating commits on HuggingFace dataset (5d0e36c)
- load and transform multiple datasets for evals, prepare seed prompts (a72424c)
- ping miners to see if reachable (1ecbd8a)
- process mturk events (1fae391)
- pull leaderboard data to get groundtruth ranks (557bdb0)
- replace cohen kappa with ICC (68f056b)
- semantic release (d85a067)
- update aws mturk script for task creation (f586335)
- update formatting (062f537)
- update LLM API for preference rankings (81cf6d0)
- update protocol (2c802e3)
- update protocol (9fb41bf)
- update readme (c852108)
- update working scaleai test script (0e996a9)
- use factory instead of singleton (d03d61e)
- use icc where possible, otherwise mse. use minmax norm (248cfc1)
- use scheduler to delay scoring & sending consensus to miners (11b8eae)
- WIP update scores by criteria (bc4447a)
- add additional check for nonzero state (801796b)
- add custom check for is_miner (42f47b7)
- add default axon ports (9e93da5)
- add default values, remove pydantic v1 args (f749b93)
- add dojo tasks to track to state (ed8be18)
- add empty check (f2d4543)
- add lock (67de42e)
- add missing attr (ce5f9ea)
- add missing await (0e22b2c)
- add missing cls (5f8e9b5)
- add missing dev env for wandb (0d70647)
- add missing method (23a0fb7)
- add timeout for create task (ebf5d33)
- add try catch, use proper uid for logging (a654779)
- add verification of mturk task on validator side (0506416)
- Added changelog to npm (9bfcb41)
- added empty string on form_body before making request to create_task (92b09da)
- added expiredAt in DojoTaskTracker and DataManager (ad46be8)
- added remove expiry task func in DataManager, and DojoTaskTracker (61d6726)
- adjust intervals (cf8e597)
- aws lambda function (2a4000f)
- aws mturk client after refactor (845917b)
- blacklist non-aws IP on miner side (8f951f7)
- broken submodule link (c9a9e26)
- cache config (d60b808)
- call golang api failing using multipartencoder (0a8d504)
- circleci (9a91306)
- circular imports (1003bbb)
- close http client (3b01101)
- Completion not serializable (34badee)
- config class type hint (dd1159b)
- correct type hint (142b1dd)
- data manager ensure paths exist (1a7b2b3)
- deadlocks caused by nested async with lock call (f088002)
- debugging dockerization (e04bae8)
- description, not default value (4478062)
- disable parallel for custom git hook (ea6f4be)
- Disabled circleci and enabled github actions (40f09d3)
- docker compose image name (ef07e90)
- dockerized miner/vali services, split into individual compose files (#45) (751937b)
- dojo cli connecting (a31ea52)
- ensure miner able to call create task (2723966)
- ensure miner context manager handled properly (c18bf3d)
- ensure miners able to receive consensus vector (ef769cf)
- ensure non-blocking inference of HF model (da15be2)
- ensure output single label (388def4)
- ensure request routes to miners only (5b8a266)
- ensure saved miner responses unobfuscated (db9f566)
- ensure scores in positive range, ensure sum of scores is 1 (6518b78)
- ensure scoring uses unobfuscated model ids (85503fc)
- ensure to apply accuracy weights, then softmax (f9b4bb7)
- ensure to filter out aws_mturk set by miners, since it cannot be completed in 30s (d328bfc)
- ensure to pickup data that is >=8hrs old (f370117)
- ensure to remove responses after processing them, use asyncio lock (11f365b)
- ensure validator inside of context manager (1e83ecb)
- ensure validator sends single prompt-response pair, slow interval (3d8ac28)
- ensure validator startup script runs properly (5973ebd)
- expose dev env url (49c3d89)
- extract nested lock behaviour (ca353f0)
- fastapi deprecated option (5d8632b)
- fix unboundlocalerror (6c86be1)
- fixed datetime error (2a8cb96)
- fixed linter error (3676c4e)
- fixed merge conflict, and ruff error (935a152)
- fixed PR feedback round 2 (cb069f2)
- fixed primsa binary not found error in docker (69e7dbd)
- fixed typo that prevent miner running (17eca35)
- fixing dockerized setup (33e8059)
- generated new uuid instead of using cached uuid, add some scoring (e75a3dd)
- get validator working (849b6f9)
- handle aws mturk payload properly (a696e5c)
- handle edge case where no data yet (34cbba6)
- handle startup/shutdown, fix context manager issues, remove blocking code (b0b1706)
- hide model names when sending to miners (25a8e01)
- hooks: use commitizen-branch as pre-push (0d9b108)
- immutable hotkey to score, extract score adjustment logic to respect, set trace to prevent log spam (971e514)
- imports (c0e83d7)
- imports and add todo (b0109a9)
- increase lru cache size to accomodate different models (59ad2ce)
- increase max instances, define using crontriggers (085bf92)
- make batch size in config.py, rename eval to evaluation to avoid clashes (81277df)
- make data manager use single instance (9eaee0d)
- make DOJO_API_KEY only required when creating task (61eb542)
- make it clearer for user to enter coldkey/hotkey name (d012d52)
- manage single httpx client (6bb5b4f)
- miner thread join (d690bf8)
- miner: mount .env file for worker api (3d999b5)
- more reasonable ema and num batches (d6ff376)
- mount current workdir to prisma to generate prisma files in ./database/prisma (df95d7c)
- no attr timestamp (706fb03)
- prevent miner from setting weights (13192d8)
- prevent nan values in consensus score (e307d3b)
- prevent validator from calling itself (e62976a)
- prioritise mturk, allow overriding of previous accuracies (fa6a3ba)
- protocol typing, create task properly (7c48dc2)
- re-enable 20k stake required for validators (f81c097)
- re-enable wandb logging (17c212f)
- read result data properly, use strenum (b8a9b75)
- refactored from MR feedback (cc230e0)
- remove blocking uvicorn.run call (62cb494)
- remove discussion spam (f1dad60)
- remove DojoTaskTracker class in seperate file (414a038)
- remove duplicate convert_weights call (9c60cd0)
- remove filtering via len(ranks) (a48e5f0)
- remove hardcoded api url (6227e6a)
- remove hotkey to accuracy (8277ae2)
- remove miner signal handling (dbeca8a)
- remove mturk credentials (76de8b0)
- remove need for entering password for cli (c37df16)
- remove patch logging after bittensor upgrade (88e1d77)
- remove reliance on vpermit (8533c91)
- remove scored responses properly, clear TODOs (c173ff9)
- remove validate args, fix pydantic models (410b9e6)
- remove validation that prevents saving (aa1cb86)
- remove validators serving axons (a6d1978)
- removed env for worker-api in validator side (f5aecce)
- replace github token with PAT (aaf758c)
- resync metagraph index errors (5509ff8)
- return synapse after receiving consensus (0721cdb)
- reverted to github token (08df1c2)
- scoring method at request level (69283eb)
- scoring method not being set in pydantic basemodel (5443950)
- set last checked time (14fc372)
- single prompt-response pair per synthetic api call (154d189)
- skip send request if failed to fetch (47f8ae1)
- store unobfuscated model ids on disk (96928e6)
- syntax (70464f8)
- temporary workaround to test code fixing agent (25f0329)
- tests (1550fab)
- torch type hint (fbb2bf7)
- unknown spec version (14e7daa)
- Update circle resource class (83f1d53)
- update netuid, make miner command work (28da52e)
- update schedulers (9edb49b)
- update scores, shift logic to calculate scores (5b337b1)
- updated folders (e5f7b0b)
- upon new miner/axon update scores (04f3170)
- use class variable, ruff lint (a350cd1)
- use correct argparse attr (4000e73)
- use interval instead of cron to prevent congestion (f827a1a)
- use minmax normalisation (483c0c2)
- use multipart form encoder (74bc850)
- use None instead of list (8f961c5)
- use raw timestamps instead of datetime objects (a09d3b6)
- use softmax reward instead of minmax scale (eeee1ad)
- use tuple for torch size (c971dbe)
- uv pip install to use --platform arg (ac60722)
- uvicorn.run blocking call (c92cee2)
- validator: configure CORS to disable all incoming requests (f5b1a13)
- variable naming (bb4dd5c)
- wrong import (1485a31)
- wrong kwarg (9a7eaa9)
- add checks for no work done by worker (2b819e4)
- add custom serve axon wrapper with retries (2a3836b)
- add flow to strip random wrapper chars that aren't words (6e9e352)
- add path validation (b554ed0)
- add retry to LLM API calls, update config (1f3b627)
- ensure only aws queries miners (2de9bd1)
- fix bug (255ac06)
- increase miner create task timeout (57f4205)
- only consume data when there are axons to query (617740b)
- reduce task frequency from 1min to 5min (03455cc)
- remove unstable models (197db28)
- stabilise json outputs using strictjson, fix deadlock by removing lru cache (c1cd1d6)
- trim docker image from 6GB to 2GB (2cea5b6)
- update prompts for code sandbox (1b7fd6a)
- use new miner selector methods (fe53836)
- update readme (3f595be)
- use public mainnet subtensor (4cfdc7a)
- pip: upgrade bittensor to 7.2.0, upgrade pydantic (866b833)
- semantic release
- pip: requires pydantic v2
- add .env.example for reference (6d0a590)
- add aws lambda function template (3749baf)
- add aws mturk script (ed52c40)
- add base reward models for miners to use (c4f8f12)
- add blacklisting on miner side (28583ea)
- add cohen kappa and distance penalty (28e3cfe)
- add complete feedback loop for miner to send scores back to validator (6a81287)
- add data manager for ranking data (bf13235)
- add Dojo API caller (cfea834)
- add dojo task monitor on validator (8c8ff24)
- add dojo task tracker and update data on disk (6c21a10)
- add dojo task tracker, use synthetic data generator (d074fc0)
- add evals to get classification accuracy (62bd5fb)
- add fastapi routes (ed71ba0)
- add ground truth scoring for multi score (1fcfaa2)
- add heartbeat, ruff fixes (27d856f)
- add ICC metric for consensus scoring (30a5f6e)
- add image scoring (d55bc54)
- add LLM API based scoring, default to mixtral 8x7b (fb16c79)
- add mainnet docker compose opts (b626677)
- add method to get response by request id (e3c3c1b)
- add methods to create PRs to huggingface dataset for review (7ba3f6e)
- add more datasets for evals (4649e67)
- add more models to generate answers from (90c9f7b)
- add new methods to data manager (d8a1234)
- add openai support (dbfcc44)
- add openrouter provider (26f212c)
- add scaleai code (e571660)
- add scoring method enum and simplify argparser (8042a98)
- add spearman scoring for synthetic ground truth (#37) (3918385)
- add subscription key mgmt to cli (2b30027)
- add weights classification accuracy multiplier to scoring (6c5772c)
- added unit testing for validator (64d1351)
- apply monkey patch to bt.logging (16af344)
- boost weights for human feedback (0a928b2)
- bump version (c34d086)
- calculate distance from ground truth, combine with consensus (cbe813f)
- calculate mean across different criteria and sum (0947a5a)
- calculate spearman correlation for scoring, add config args validation, update protocol (132b944)
- change python package name (8990325)
- change validator interval to every 30s (4052332)
- cli: add coldkey and hotkey path completions (ed2f36a)
- create dojo cli for miners (56615c9)
- create tasks with ranking/multi-score (aaeef16)
- enable huggingface/llm api scoring methods for miner (011bdc9)
- enable LLM API based evals, update config (ffea364)
- enable miners to run uvicorn server via miner.py (24406e2)
- enable synthetic prompt & responses generation for coding task (6fa0b73)
- external user (10ca4a0)
- flesh out readme details, provide screenshots (74587b1)
- forward consensus score to miners (c949ebe)
- get initial prompts & completions from synthetic dataset (3551591)
- group all functions under MTurkUtils, add event handler placeholder (99ff4f2)
- implement hashring to select miners (3ba4953)
- integrate automatically creating commits on HuggingFace dataset (5d0e36c)
- load and transform multiple datasets for evals, prepare seed prompts (a72424c)
- ping miners to see if reachable (e3ac465)
- process mturk events (1fae391)
- pull leaderboard data to get groundtruth ranks (557bdb0)
- replace cohen kappa with ICC (68f056b)
- semantic release (869180d)
- update aws mturk script for task creation (f586335)
- update formatting (062f537)
- update LLM API for preference rankings (81cf6d0)
- update protocol (2c802e3)
- update protocol (9fb41bf)
- update readme (c852108)
- update working scaleai test script (0e996a9)
- use factory instead of singleton (d03d61e)
- use icc where possible, otherwise mse. use minmax norm (82d04d5)
- use scheduler to delay scoring & sending consensus to miners (11b8eae)
- WIP update scores by criteria (bc4447a)
- add additional check for nonzero state (801796b)
- add custom check for is_miner (42f47b7)
- add default axon ports (5301524)
- add default values, remove pydantic v1 args (4c7e829)
- add dojo tasks to track to state (ed8be18)
- add empty check (f2d4543)
- add lock (67de42e)
- add missing attr (ce5f9ea)
- add missing await (0e22b2c)
- add missing cls (5f8e9b5)
- add missing dev env for wandb (8ef1274)
- add missing method (23a0fb7)
- add timeout for create task (ebf5d33)
- add try catch, use proper uid for logging (4936043)
- add verification of mturk task on validator side (0506416)
- Added changelog to npm (79d429f)
- added empty string on form_body before making request to create_task (250ee0b)
- added expiredAt in DojoTaskTracker and DataManager (0cde81b)
- added remove expiry task func in DataManager, and DojoTaskTracker (a95b909)
- adjust intervals (56a7044)
- aws lambda function (2a4000f)
- aws mturk client after refactor (845917b)
- blacklist non-aws IP on miner side (8f951f7)
- broken submodule link (42a599e)
- cache config (bbc47f5)
- call golang api failing using multipartencoder (8954f08)
- circleci (f53a9f5)
- circular imports (1003bbb)
- close http client (3b01101)
- Completion not serializable (34badee)
- config class type hint (dd1159b)
- correct type hint (142b1dd)
- data manager ensure paths exist (1a7b2b3)
- deadlocks caused by nested async with lock call (f088002)
- debugging dockerization (94e24cd)
- description, not default value (4478062)
- disable parallel for custom git hook (ea6f4be)
- Disabled circleci and enabled github actions (e530318)
- docker compose image name (558cc97)
- dockerized miner/vali services, split into individual compose files (#45) (f61095c)
- dojo cli connecting (4bfaf7d)
- ensure miner able to call create task (2723966)
- ensure miner context manager handled properly (c18bf3d)
- ensure miners able to receive consensus vector (ef769cf)
- ensure non-blocking inference of HF model (da15be2)
- ensure output single label (388def4)
- ensure request routes to miners only (5b8a266)
- ensure saved miner responses unobfuscated (db9f566)
- ensure scores in positive range, ensure sum of scores is 1 (6518b78)
- ensure scoring uses unobfuscated model ids (85503fc)
- ensure to apply accuracy weights, then softmax (f9b4bb7)
- ensure to filter out aws_mturk set by miners, since it cannot be completed in 30s (d328bfc)
- ensure to pickup data that is >=8hrs old (f370117)
- ensure to remove responses after processing them, use asyncio lock (11f365b)
- ensure validator inside of context manager (1e83ecb)
- ensure validator sends single prompt-response pair, slow interval (3d8ac28)
- ensure validator startup script runs properly (5973ebd)
- expose dev env url (49c3d89)
- extract nested lock behaviour (ca353f0)
- fastapi deprecated option (5d8632b)
- fix unboundlocalerror (6c86be1)
- fixed datetime error (510e421)
- fixed linter error (16ca415)
- fixed merge conflict, and ruff error (219f252)
- fixed PR feedback round 2 (44d9f18)
- fixed primsa binary not found error in docker (16737c6)
- fixed typo that prevent miner running (17eca35)
- fixing dockerized setup (b52d27d)
- generated new uuid instead of using cached uuid, add some scoring (e75a3dd)
- get validator working (849b6f9)
- handle aws mturk payload properly (a696e5c)
- handle edge case where no data yet (34cbba6)
- handle startup/shutdown, fix context manager issues, remove blocking code (b0b1706)
- hide model names when sending to miners (25a8e01)
- hooks: use commitizen-branch as pre-push (b5be7c8)
- immutable hotkey to score, extract score adjustment logic to respect, set trace to prevent log spam (971e514)
- imports (c0e83d7)
- imports and add todo (b0109a9)
- increase lru cache size to accomodate different models (59ad2ce)
- increase max instances, define using crontriggers (085bf92)
- make batch size in config.py, rename eval to evaluation to avoid clashes (81277df)
- make data manager use single instance (9eaee0d)
- make DOJO_API_KEY only required when creating task (61eb542)
- make it clearer for user to enter coldkey/hotkey name (d012d52)
- manage single httpx client (6bb5b4f)
- miner thread join (d690bf8)
- miner: mount .env file for worker api (7ccf41d)
- more reasonable ema and num batches (d6ff376)
- mount current workdir to prisma to generate prisma files in ./database/prisma (8e32f15)
- no attr timestamp (706fb03)
- prevent miner from setting weights (13192d8)
- prevent nan values in consensus score (ee0ac29)
- prevent validator from calling itself (e62976a)
- prioritise mturk, allow overriding of previous accuracies (fa6a3ba)
- protocol typing, create task properly (7c48dc2)
- re-enable 20k stake required for validators (f81c097)
- re-enable wandb logging (c898fff)
- read result data properly, use strenum (b8a9b75)
- refactored from MR feedback (a01653e)
- remove blocking uvicorn.run call (62cb494)
- remove discussion spam (f1dad60)
- remove DojoTaskTracker class in seperate file (08f001f)
- remove duplicate convert_weights call (9c60cd0)
- remove filtering via len(ranks) (a48e5f0)
- remove hardcoded api url (6227e6a)
- remove hotkey to accuracy (8277ae2)
- remove miner signal handling (dbeca8a)
- remove mturk credentials (76de8b0)
- remove need for entering password for cli (ed8f2c9)
- remove patch logging after bittensor upgrade (88e1d77)
- remove reliance on vpermit (8533c91)
- remove scored responses properly, clear TODOs (c173ff9)
- remove validate args, fix pydantic models (410b9e6)
- remove validation that prevents saving (aa1cb86)
- remove validators serving axons (a6d1978)
- replace github token with PAT (727b1e3)
- resync metagraph index errors (5509ff8)
- return synapse after receiving consensus (0721cdb)
- reverted to github token (e73d00b)
- scoring method at request level (69283eb)
- scoring method not being set in pydantic basemodel (5443950)
- set last checked time (14fc372)
- single prompt-response pair per synthetic api call (154d189)
- skip send request if failed to fetch (47f8ae1)
- store unobfuscated model ids on disk (96928e6)
- syntax (09bb29f)
- temporary workaround to test code fixing agent (25f0329)
- tests (9ee4772)
- torch type hint (1bee442)
- unknown spec version (14e7daa)
- Update circle resource class (68f3da3)
- update netuid, make miner command work (1831119)
- update schedulers (9edb49b)
- update scores, shift logic to calculate scores (5b337b1)
- updated folders (bcc19f9)
- upon new miner/axon update scores (04f3170)
- use class variable, ruff lint (a350cd1)
- use correct argparse attr (4000e73)
- use interval instead of cron to prevent congestion (f827a1a)
- use minmax normalisation (912f9c0)
- use multipart form encoder (74bc850)
- use None instead of list (8f961c5)
- use raw timestamps instead of datetime objects (a09d3b6)
- use softmax reward instead of minmax scale (4d2871e)
- use tuple for torch size (c971dbe)
- uv pip install to use --platform arg (3a9b025)
- uvicorn.run blocking call (c92cee2)
- validator: configure CORS to disable all incoming requests (f5b1a13)
- variable naming (bb4dd5c)
- wrong import (1485a31)
- wrong kwarg (9a7eaa9)
- add checks for no work done by worker (2b819e4)
- add custom serve axon wrapper with retries (2a3836b)
- add flow to strip random wrapper chars that aren't words (6e9e352)
- add path validation (b554ed0)
- add retry to LLM API calls, update config (1f3b627)
- ensure only aws queries miners (2de9bd1)
- fix bug (74af0ec)
- increase miner create task timeout (57f4205)
- only consume data when there are axons to query (f578c1e)
- reduce task frequency from 1min to 5min (a0d010c)
- remove unstable models (197db28)
- stabilise json outputs using strictjson, fix deadlock by removing lru cache (c1cd1d6)
- trim docker image from 6GB to 2GB (efc39fe)
- update prompts for code sandbox (1b7fd6a)
- use new miner selector methods (fe53836)
- pip: upgrade bittensor to 7.2.0, upgrade pydantic (ec8034e)
2.4.1 (2024-08-13)
- broken submodule link (42a599e)
2.4.0 (2024-08-06)
- added unit testing for validator (64d1351)
2.3.0 (2024-08-01)
- prevent nan values in consensus score (ee0ac29)
- use minmax normalisation (912f9c0)
- use softmax reward instead of minmax scale (4d2871e)
- use icc where possible, otherwise mse. use minmax norm (82d04d5)
2.2.3 (2024-07-22)
- reverted to github token (e73d00b)
2.2.2 (2024-07-22)
- replace github token with PAT (727b1e3)
2.2.1 (2024-07-22)
2.2.0 (2024-07-12)
- external user (10ca4a0)
2.1.1 (2024-07-12)
- added empty string on form_body before making request to create_task (250ee0b)
- added expiredAt in DojoTaskTracker and DataManager (0cde81b)
- added remove expiry task func in DataManager, and DojoTaskTracker (a95b909)
- fixed datetime error (510e421)
- fixed merge conflict, and ruff error (219f252)
- refactored from MR feedback (a01653e)
- remove DojoTaskTracker class in seperate file (08f001f)
2.1.0 (2024-07-10)
- add default values, remove pydantic v1 args (4c7e829)
- add try catch, use proper uid for logging (4936043)
- call golang api failing using multipartencoder (8954f08)
- only consume data when there are axons to query (f578c1e)
2.0.0 (2024-07-05)
- hooks: use commitizen-branch as pre-push (b5be7c8)
- pip: upgrade bittensor to 7.2.0, upgrade pydantic (ec8034e)
- pip: requires pydantic v2
1.0.3 (2024-07-05)
- syntax (09bb29f)
1.0.2 (2024-07-04)
- Update circle resource class (68f3da3)
1.0.1 (2024-07-04)
- circleci (f53a9f5)
- add additional check for nonzero state (801796b)
- add custom check for is_miner (42f47b7)
- add dojo tasks to track to state (ed8be18)
- add empty check (f2d4543)
- add lock (67de42e)
- add missing attr (ce5f9ea)
- add missing await (0e22b2c)
- add missing cls (5f8e9b5)
- add missing dev env for wandb (8ef1274)
- add missing method (23a0fb7)
- add timeout for create task (ebf5d33)
- add verification of mturk task on validator side (0506416)
- Added changelog to npm (79d429f)
- aws lambda function (2a4000f)
- aws mturk client after refactor (845917b)
- blacklist non-aws IP on miner side (8f951f7)
- circular imports (1003bbb)
- close http client (3b01101)
- Completion not serializable (34badee)
- config class type hint (dd1159b)
- correct type hint (142b1dd)
- data manager ensure paths exist (1a7b2b3)
- deadlocks caused by nested async with lock call (f088002)
- description, not default value (4478062)
- disable parallel for custom git hook (ea6f4be)
- ensure miner able to call create task (2723966)
- ensure miner context manager handled properly (c18bf3d)
- ensure miners able to receive consensus vector (ef769cf)
- ensure non-blocking inference of HF model (da15be2)
- ensure output single label (388def4)
- ensure request routes to miners only (5b8a266)
- ensure saved miner responses unobfuscated (db9f566)
- ensure scores in positive range, ensure sum of scores is 1 (6518b78)
- ensure scoring uses unobfuscated model ids (85503fc)
- ensure to apply accuracy weights, then softmax (f9b4bb7)
- ensure to filter out aws_mturk set by miners, since it cannot be completed in 30s (d328bfc)
- ensure to pickup data that is >=8hrs old (f370117)
- ensure to remove responses after processing them, use asyncio lock (11f365b)
- ensure validator inside of context manager (1e83ecb)
- ensure validator sends single prompt-response pair, slow interval (3d8ac28)
- ensure validator startup script runs properly (5973ebd)
- expose dev env url (49c3d89)
- extract nested lock behaviour (ca353f0)
- fastapi deprecated option (5d8632b)
- fix unboundlocalerror (6c86be1)
- fixed typo that prevent miner running (17eca35)
- generated new uuid instead of using cached uuid, add some scoring (e75a3dd)
- get validator working (849b6f9)
- handle aws mturk payload properly (a696e5c)
- handle edge case where no data yet (34cbba6)
- handle startup/shutdown, fix context manager issues, remove blocking code (b0b1706)
- hide model names when sending to miners (25a8e01)
- immutable hotkey to score, extract score adjustment logic to respect, set trace to prevent log spam (971e514)
- imports (c0e83d7)
- imports and add todo (b0109a9)
- increase lru cache size to accomodate different models (59ad2ce)
- increase max instances, define using crontriggers (085bf92)
- make batch size in config.py, rename eval to evaluation to avoid clashes (81277df)
- make data manager use single instance (9eaee0d)
- make DOJO_API_KEY only required when creating task (61eb542)
- make it clearer for user to enter coldkey/hotkey name (d012d52)
- manage single httpx client (6bb5b4f)
- miner thread join (d690bf8)
- more reasonable ema and num batches (d6ff376)
- no attr timestamp (706fb03)
- prevent miner from setting weights (13192d8)
- prevent validator from calling itself (e62976a)
- prioritise mturk, allow overriding of previous accuracies (fa6a3ba)
- protocol typing, create task properly (7c48dc2)
- re-enable 20k stake required for validators (f81c097)
- re-enable wandb logging (c898fff)
- read result data properly, use strenum (b8a9b75)
- remove blocking uvicorn.run call (62cb494)
- remove discussion spam (f1dad60)
- remove duplicate convert_weights call (9c60cd0)
- remove filtering via len(ranks) (a48e5f0)
- remove hardcoded api url (6227e6a)
- remove hotkey to accuracy (8277ae2)
- remove miner signal handling (dbeca8a)
- remove mturk credentials (76de8b0)
- remove patch logging after bittensor upgrade (88e1d77)
- remove reliance on vpermit (8533c91)
- remove scored responses properly, clear TODOs (c173ff9)
- remove validate args, fix pydantic models (410b9e6)
- remove validation that prevents saving (aa1cb86)
- remove validators serving axons (a6d1978)
- resync metagraph index errors (5509ff8)
- return synapse after receiving consensus (0721cdb)
- scoring method at request level (69283eb)
- scoring method not being set in pydantic basemodel (5443950)
- set last checked time (14fc372)
- single prompt-response pair per synthetic api call (154d189)
- skip send request if failed to fetch (47f8ae1)
- store unobfuscated model ids on disk (96928e6)
- temporary workaround to test code fixing agent (25f0329)
- unknown spec version (14e7daa)
- update schedulers (9edb49b)
- update scores, shift logic to calculate scores (5b337b1)
- upon new miner/axon update scores (04f3170)
- use class variable, ruff lint (a350cd1)
- use correct argparse attr (4000e73)
- use interval instead of cron to prevent congestion (f827a1a)
- use multipart form encoder (74bc850)
- use None instead of list (8f961c5)
- use raw timestamps instead of datetime objects (a09d3b6)
- use tuple for torch size (c971dbe)
- uvicorn.run blocking call (c92cee2)
- validator: configure CORS to disable all incoming requests (f5b1a13)
- variable naming (bb4dd5c)
- wrong import (1485a31)
- wrong kwarg (9a7eaa9)
- add .env.example for reference (6d0a590)
- add aws lambda function template (3749baf)
- add aws mturk script (ed52c40)
- add base reward models for miners to use (c4f8f12)
- add blacklisting on miner side (28583ea)
- add cohen kappa and distance penalty (28e3cfe)
- add complete feedback loop for miner to send scores back to validator (6a81287)
- add data manager for ranking data (bf13235)
- add Dojo API caller (cfea834)
- add dojo task monitor on validator (8c8ff24)
- add dojo task tracker and update data on disk (6c21a10)
- add dojo task tracker, use synthetic data generator (d074fc0)
- add evals to get classification accuracy (62bd5fb)
- add fastapi routes (ed71ba0)
- add ground truth scoring for multi score (1fcfaa2)
- add ICC metric for consensus scoring (30a5f6e)
- add image scoring (d55bc54)
- add LLM API based scoring, default to mixtral 8x7b (fb16c79)
- add method to get response by request id (e3c3c1b)
- add methods to create PRs to huggingface dataset for review (7ba3f6e)
- add more datasets for evals (4649e67)
- add more models to generate answers from (90c9f7b)
- add new methods to data manager (d8a1234)
- add openai support (dbfcc44)
- add openrouter provider (26f212c)
- add scaleai code (e571660)
- add scoring method enum and simplify argparser (8042a98)
- add subscription key mgmt to cli (2b30027)
- add weights classification accuracy multiplier to scoring (6c5772c)
- apply monkey patch to bt.logging (16af344)
- boost weights for human feedback (0a928b2)
- bump version (c34d086)
- calculate distance from ground truth, combine with consensus (cbe813f)
- calculate mean across different criteria and sum (0947a5a)
- calculate spearman correlation for scoring, add config args validation, update protocol (132b944)
- change python package name (8990325)
- change validator interval to every 30s (4052332)
- cli: add coldkey and hotkey path completions (ed2f36a)
- create dojo cli for miners (56615c9)
- create tasks with ranking/multi-score (aaeef16)
- enable huggingface/llm api scoring methods for miner (011bdc9)
- enable LLM API based evals, update config (ffea364)
- enable miners to run uvicorn server via miner.py (24406e2)
- enable synthetic prompt & responses generation for coding task (6fa0b73)
- flesh out readme details, provide screenshots (74587b1)
- forward consensus score to miners (c949ebe)
- group all functions under MTurkUtils, add event handler placeholder (99ff4f2)
- implement hashring to select miners (3ba4953)
- integrate automatically creating commits on HuggingFace dataset (5d0e36c)
- load and transform multiple datasets for evals, prepare seed prompts (a72424c)
- process mturk events (1fae391)
- pull leaderboard data to get groundtruth ranks (557bdb0)
- replace cohen kappa with ICC (68f056b)
- update aws mturk script for task creation (f586335)
- update formatting (062f537)
- update LLM API for preference rankings (81cf6d0)
- update protocol (2c802e3)
- update protocol (9fb41bf)
- update readme (c852108)
- update working scaleai test script (0e996a9)
- use factory instead of singleton (d03d61e)
- use scheduler to delay scoring & sending consensus to miners (11b8eae)
- WIP update scores by criteria (bc4447a)
- add checks for no work done by worker (2b819e4)
- add custom serve axon wrapper with retries (2a3836b)
- add flow to strip random wrapper chars that aren't words (6e9e352)
- add path validation (b554ed0)
- add retry to LLM API calls, update config (1f3b627)
- ensure only aws queries miners (2de9bd1)
- increase miner create task timeout (57f4205)
- reduce task frequency from 1min to 5min (a0d010c)
- remove unstable models (197db28)
- stabilise json outputs using strictjson, fix deadlock by removing lru cache (c1cd1d6)
- update prompts for code sandbox (1b7fd6a)
- use new miner selector methods (fe53836)
- add additional check for nonzero state (801796b)
- add custom check for is_miner (42f47b7)
- add dojo tasks to track to state (ed8be18)
- add empty check (f2d4543)
- add lock (67de42e)
- add missing attr (ce5f9ea)
- add missing await (0e22b2c)
- add missing cls (5f8e9b5)
- add missing dev env for wandb (8ef1274)
- add missing method (23a0fb7)
- add timeout for create task (ebf5d33)
- add verification of mturk task on validator side (0506416)
- Added changelog to npm (79d429f)
- aws lambda function (2a4000f)
- aws mturk client after refactor (845917b)
- blacklist non-aws IP on miner side (8f951f7)
- circular imports (1003bbb)
- close http client (3b01101)
- Completion not serializable (34badee)
- config class type hint (dd1159b)
- correct type hint (142b1dd)
- data manager ensure paths exist (1a7b2b3)
- deadlocks caused by nested async with lock call (f088002)
- description, not default value (4478062)
- disable parallel for custom git hook (ea6f4be)
- ensure miner able to call create task (2723966)
- ensure miner context manager handled properly (c18bf3d)
- ensure miners able to receive consensus vector (ef769cf)
- ensure non-blocking inference of HF model (da15be2)
- ensure output single label (388def4)
- ensure request routes to miners only (5b8a266)
- ensure saved miner responses unobfuscated (db9f566)
- ensure scores in positive range, ensure sum of scores is 1 (6518b78)
- ensure scoring uses unobfuscated model ids (85503fc)
- ensure to apply accuracy weights, then softmax (f9b4bb7)
- ensure to filter out aws_mturk set by miners, since it cannot be completed in 30s (d328bfc)
- ensure to pickup data that is >=8hrs old (f370117)
- ensure to remove responses after processing them, use asyncio lock (11f365b)
- ensure validator inside of context manager (1e83ecb)
- ensure validator sends single prompt-response pair, slow interval (3d8ac28)
- ensure validator startup script runs properly (5973ebd)
- expose dev env url (49c3d89)
- extract nested lock behaviour (ca353f0)
- fastapi deprecated option (5d8632b)
- fix unboundlocalerror (6c86be1)
- fixed typo that prevent miner running (17eca35)
- generated new uuid instead of using cached uuid, add some scoring (e75a3dd)
- get validator working (849b6f9)
- handle aws mturk payload properly (a696e5c)
- handle edge case where no data yet (34cbba6)
- handle startup/shutdown, fix context manager issues, remove blocking code (b0b1706)
- hide model names when sending to miners (25a8e01)
- immutable hotkey to score, extract score adjustment logic to respect, set trace to prevent log spam (971e514)
- imports (c0e83d7)
- imports and add todo (b0109a9)
- increase lru cache size to accomodate different models (59ad2ce)
- increase max instances, define using crontriggers (085bf92)
- make batch size in config.py, rename eval to evaluation to avoid clashes (81277df)
- make data manager use single instance (9eaee0d)
- make DOJO_API_KEY only required when creating task (61eb542)
- make it clearer for user to enter coldkey/hotkey name (d012d52)
- manage single httpx client (6bb5b4f)
- miner thread join (d690bf8)
- more reasonable ema and num batches (d6ff376)
- no attr timestamp (706fb03)
- prevent miner from setting weights (13192d8)
- prevent validator from calling itself (e62976a)
- prioritise mturk, allow overriding of previous accuracies (fa6a3ba)
- protocol typing, create task properly (7c48dc2)
- re-enable 20k stake required for validators (f81c097)
- re-enable wandb logging (c898fff)
- read result data properly, use strenum (b8a9b75)
- remove blocking uvicorn.run call (62cb494)
- remove discussion spam (f1dad60)
- remove duplicate convert_weights call (9c60cd0)
- remove filtering via len(ranks) (a48e5f0)
- remove hardcoded api url (6227e6a)
- remove hotkey to accuracy (8277ae2)
- remove miner signal handling (dbeca8a)
- remove mturk credentials (76de8b0)
- remove patch logging after bittensor upgrade (88e1d77)
- remove reliance on vpermit (8533c91)
- remove scored responses properly, clear TODOs (c173ff9)
- remove validate args, fix pydantic models (410b9e6)
- remove validation that prevents saving (aa1cb86)
- remove validators serving axons (a6d1978)
- resync metagraph index errors (5509ff8)
- return synapse after receiving consensus (0721cdb)
- scoring method at request level (69283eb)
- scoring method not being set in pydantic basemodel (5443950)
- set last checked time (14fc372)
- single prompt-response pair per synthetic api call (154d189)
- skip send request if failed to fetch (47f8ae1)
- store unobfuscated model ids on disk (96928e6)
- temporary workaround to test code fixing agent (25f0329)
- unknown spec version (14e7daa)
- update schedulers (9edb49b)
- update scores, shift logic to calculate scores (5b337b1)
- upon new miner/axon update scores (04f3170)
- use class variable, ruff lint (a350cd1)
- use correct argparse attr (4000e73)
- use interval instead of cron to prevent congestion (f827a1a)
- use multipart form encoder (74bc850)
- use None instead of list (8f961c5)
- use raw timestamps instead of datetime objects (a09d3b6)
- use tuple for torch size (c971dbe)
- uvicorn.run blocking call (c92cee2)
- validator: configure CORS to disable all incoming requests (f5b1a13)
- variable naming (bb4dd5c)
- wrong import (1485a31)
- wrong kwarg (9a7eaa9)
- add .env.example for reference (6d0a590)
- add aws lambda function template (3749baf)
- add aws mturk script (ed52c40)
- add base reward models for miners to use (c4f8f12)
- add blacklisting on miner side (28583ea)
- add cohen kappa and distance penalty (28e3cfe)
- add complete feedback loop for miner to send scores back to validator (6a81287)
- add data manager for ranking data (bf13235)
- add Dojo API caller (cfea834)
- add dojo task monitor on validator (8c8ff24)
- add dojo task tracker and update data on disk (6c21a10)
- add dojo task tracker, use synthetic data generator (d074fc0)
- add evals to get classification accuracy (62bd5fb)
- add fastapi routes (ed71ba0)
- add ground truth scoring for multi score (1fcfaa2)
- add ICC metric for consensus scoring (30a5f6e)
- add image scoring (d55bc54)
- add LLM API based scoring, default to mixtral 8x7b (fb16c79)
- add method to get response by request id (e3c3c1b)
- add methods to create PRs to huggingface dataset for review (7ba3f6e)
- add more datasets for evals (4649e67)
- add more models to generate answers from (90c9f7b)
- add new methods to data manager (d8a1234)
- add openai support (dbfcc44)
- add openrouter provider (26f212c)
- add scaleai code (e571660)
- add scoring method enum and simplify argparser (8042a98)
- add subscription key mgmt to cli (2b30027)
- add weights classification accuracy multiplier to scoring (6c5772c)
- apply monkey patch to bt.logging (16af344)
- boost weights for human feedback (0a928b2)
- bump version (c34d086)
- calculate distance from ground truth, combine with consensus (cbe813f)
- calculate mean across different criteria and sum (0947a5a)
- calculate spearman correlation for scoring, add config args validation, update protocol (132b944)
- change python package name (8990325)
- change validator interval to every 30s (4052332)
- cli: add coldkey and hotkey path completions (ed2f36a)
- create dojo cli for miners (56615c9)
- create tasks with ranking/multi-score (aaeef16)
- enable huggingface/llm api scoring methods for miner (011bdc9)
- enable LLM API based evals, update config (ffea364)
- enable miners to run uvicorn server via miner.py (24406e2)
- enable synthetic prompt & responses generation for coding task (6fa0b73)
- flesh out readme details, provide screenshots (74587b1)
- forward consensus score to miners (c949ebe)
- group all functions under MTurkUtils, add event handler placeholder (99ff4f2)
- implement hashring to select miners (3ba4953)
- integrate automatically creating commits on HuggingFace dataset (5d0e36c)
- load and transform multiple datasets for evals, prepare seed prompts (a72424c)
- process mturk events (1fae391)
- pull leaderboard data to get groundtruth ranks (557bdb0)
- replace cohen kappa with ICC (68f056b)
- update aws mturk script for task creation (f586335)
- update formatting (062f537)
- update LLM API for preference rankings (81cf6d0)
- update protocol (2c802e3)
- update protocol (9fb41bf)
- update readme (c852108)
- update working scaleai test script (0e996a9)
- use factory instead of singleton (d03d61e)
- use scheduler to delay scoring & sending consensus to miners (11b8eae)
- WIP update scores by criteria (bc4447a)
- add checks for no work done by worker (2b819e4)
- add custom serve axon wrapper with retries (2a3836b)
- add flow to strip random wrapper chars that aren't words (6e9e352)
- add path validation (b554ed0)
- add retry to LLM API calls, update config (1f3b627)
- ensure only aws queries miners (2de9bd1)
- increase miner create task timeout (57f4205)
- reduce task frequency from 1min to 5min (a0d010c)
- remove unstable models (197db28)
- stabilise json outputs using strictjson, fix deadlock by removing lru cache (c1cd1d6)
- update prompts for code sandbox (1b7fd6a)
- use new miner selector methods (fe53836)
- add additional check for nonzero state (801796b)
- add custom check for is_miner (42f47b7)
- add dojo tasks to track to state (ed8be18)
- add empty check (f2d4543)
- add lock (67de42e)
- add missing attr (ce5f9ea)
- add missing await (0e22b2c)
- add missing cls (5f8e9b5)
- add missing dev env for wandb (8ef1274)
- add missing method (23a0fb7)
- add timeout for create task (ebf5d33)
- add verification of mturk task on validator side (0506416)
- Added changelog to npm (79d429f)
- aws lambda function (2a4000f)
- aws mturk client after refactor (845917b)
- blacklist non-aws IP on miner side (8f951f7)
- circular imports (1003bbb)
- close http client (3b01101)
- Completion not serializable (34badee)
- config class type hint (dd1159b)
- correct type hint (142b1dd)
- data manager ensure paths exist (1a7b2b3)
- deadlocks caused by nested async with lock call (f088002)
- description, not default value (4478062)
- disable parallel for custom git hook (ea6f4be)
- ensure miner able to call create task (2723966)
- ensure miner context manager handled properly (c18bf3d)
- ensure miners able to receive consensus vector (ef769cf)
- ensure non-blocking inference of HF model (da15be2)
- ensure output single label (388def4)
- ensure request routes to miners only (5b8a266)
- ensure saved miner responses unobfuscated (db9f566)
- ensure scores in positive range, ensure sum of scores is 1 (6518b78)
- ensure scoring uses unobfuscated model ids (85503fc)
- ensure to apply accuracy weights, then softmax (f9b4bb7)
- ensure to filter out aws_mturk set by miners, since it cannot be completed in 30s (d328bfc)
- ensure to pickup data that is >=8hrs old (f370117)
- ensure to remove responses after processing them, use asyncio lock (11f365b)
- ensure validator inside of context manager (1e83ecb)
- ensure validator sends single prompt-response pair, slow interval (3d8ac28)
- ensure validator startup script runs properly (5973ebd)
- expose dev env url (49c3d89)
- extract nested lock behaviour (ca353f0)
- fastapi deprecated option (5d8632b)
- fix unboundlocalerror (6c86be1)
- fixed typo that prevent miner running (17eca35)
- generated new uuid instead of using cached uuid, add some scoring (e75a3dd)
- get validator working (849b6f9)
- handle aws mturk payload properly (a696e5c)
- handle edge case where no data yet (34cbba6)
- handle startup/shutdown, fix context manager issues, remove blocking code (b0b1706)
- hide model names when sending to miners (25a8e01)
- immutable hotkey to score, extract score adjustment logic to respect, set trace to prevent log spam (971e514)
- imports (c0e83d7)
- imports and add todo (b0109a9)
- increase lru cache size to accomodate different models (59ad2ce)
- increase max instances, define using crontriggers (085bf92)
- make batch size in config.py, rename eval to evaluation to avoid clashes (81277df)
- make data manager use single instance (9eaee0d)
- make DOJO_API_KEY only required when creating task (61eb542)
- make it clearer for user to enter coldkey/hotkey name (d012d52)
- manage single httpx client (6bb5b4f)
- miner thread join (d690bf8)
- more reasonable ema and num batches (d6ff376)
- no attr timestamp (706fb03)
- prevent miner from setting weights (13192d8)
- prevent validator from calling itself (e62976a)
- prioritise mturk, allow overriding of previous accuracies (fa6a3ba)
- protocol typing, create task properly (7c48dc2)
- re-enable 20k stake required for validators (f81c097)
- re-enable wandb logging (c898fff)
- read result data properly, use strenum (b8a9b75)
- remove blocking uvicorn.run call (62cb494)
- remove discussion spam (f1dad60)
- remove duplicate convert_weights call (9c60cd0)
- remove filtering via len(ranks) (a48e5f0)
- remove hardcoded api url (6227e6a)
- remove hotkey to accuracy (8277ae2)
- remove miner signal handling (dbeca8a)
- remove mturk credentials (76de8b0)
- remove patch logging after bittensor upgrade (88e1d77)
- remove reliance on vpermit (8533c91)
- remove scored responses properly, clear TODOs (c173ff9)
- remove validate args, fix pydantic models (410b9e6)
- remove validation that prevents saving (aa1cb86)
- remove validators serving axons (a6d1978)
- resync metagraph index errors (5509ff8)
- return synapse after receiving consensus (0721cdb)
- scoring method at request level (69283eb)
- scoring method not being set in pydantic basemodel (5443950)
- set last checked time (14fc372)
- single prompt-response pair per synthetic api call (154d189)
- skip send request if failed to fetch (47f8ae1)
- store unobfuscated model ids on disk (96928e6)
- temporary workaround to test code fixing agent (25f0329)
- unknown spec version (14e7daa)
- update schedulers (9edb49b)
- update scores, shift logic to calculate scores (5b337b1)
- upon new miner/axon update scores (04f3170)
- use class variable, ruff lint (a350cd1)
- use correct argparse attr (4000e73)
- use interval instead of cron to prevent congestion (f827a1a)
- use multipart form encoder (74bc850)
- use None instead of list (8f961c5)
- use raw timestamps instead of datetime objects (a09d3b6)
- use tuple for torch size (c971dbe)
- uvicorn.run blocking call (c92cee2)
- validator: configure CORS to disable all incoming requests (f5b1a13)
- variable naming (bb4dd5c)
- wrong import (1485a31)
- wrong kwarg (9a7eaa9)
- add .env.example for reference (6d0a590)
- add aws lambda function template (3749baf)
- add aws mturk script (ed52c40)
- add base reward models for miners to use (c4f8f12)
- add blacklisting on miner side (28583ea)
- add cohen kappa and distance penalty (28e3cfe)
- add complete feedback loop for miner to send scores back to validator (6a81287)
- add data manager for ranking data (bf13235)
- add Dojo API caller (cfea834)
- add dojo task monitor on validator (8c8ff24)
- add dojo task tracker and update data on disk (6c21a10)
- add dojo task tracker, use synthetic data generator (d074fc0)
- add evals to get classification accuracy (62bd5fb)
- add fastapi routes (ed71ba0)
- add ground truth scoring for multi score (1fcfaa2)
- add ICC metric for consensus scoring (30a5f6e)
- add image scoring (d55bc54)
- add LLM API based scoring, default to mixtral 8x7b (fb16c79)
- add method to get response by request id (e3c3c1b)
- add methods to create PRs to huggingface dataset for review (7ba3f6e)
- add more datasets for evals (4649e67)
- add more models to generate answers from (90c9f7b)
- add new methods to data manager (d8a1234)
- add openai support (dbfcc44)
- add openrouter provider (26f212c)
- add scaleai code (e571660)
- add scoring method enum and simplify argparser (8042a98)
- add subscription key mgmt to cli (2b30027)
- add weights classification accuracy multiplier to scoring (6c5772c)
- apply monkey patch to bt.logging (16af344)
- boost weights for human feedback (0a928b2)
- bump version (c34d086)
- calculate distance from ground truth, combine with consensus (cbe813f)
- calculate mean across different criteria and sum (0947a5a)
- calculate spearman correlation for scoring, add config args validation, update protocol (132b944)
- change python package name (8990325)
- change validator interval to every 30s (4052332)
- cli: add coldkey and hotkey path completions (ed2f36a)
- create dojo cli for miners (56615c9)
- create tasks with ranking/multi-score (aaeef16)
- enable huggingface/llm api scoring methods for miner (011bdc9)
- enable LLM API based evals, update config (ffea364)
- enable miners to run uvicorn server via miner.py (24406e2)
- enable synthetic prompt & responses generation for coding task (6fa0b73)
- flesh out readme details, provide screenshots (74587b1)
- forward consensus score to miners (c949ebe)
- group all functions under MTurkUtils, add event handler placeholder (99ff4f2)
- implement hashring to select miners (3ba4953)
- integrate automatically creating commits on HuggingFace dataset (5d0e36c)
- load and transform multiple datasets for evals, prepare seed prompts (a72424c)
- process mturk events (1fae391)
- pull leaderboard data to get groundtruth ranks (557bdb0)
- replace cohen kappa with ICC (68f056b)
- update aws mturk script for task creation (f586335)
- update formatting (062f537)
- update LLM API for preference rankings (81cf6d0)
- update protocol (2c802e3)
- update protocol (9fb41bf)
- update readme (c852108)
- update working scaleai test script (0e996a9)
- use factory instead of singleton (d03d61e)
- use scheduler to delay scoring & sending consensus to miners (11b8eae)
- WIP update scores by criteria (bc4447a)
- add checks for no work done by worker (2b819e4)
- add custom serve axon wrapper with retries (2a3836b)
- add flow to strip random wrapper chars that aren't words (6e9e352)
- add path validation (b554ed0)
- add retry to LLM API calls, update config (1f3b627)
- ensure only aws queries miners (2de9bd1)
- increase miner create task timeout (57f4205)
- reduce task frequency from 1min to 5min (a0d010c)
- remove unstable models (197db28)
- stabilise json outputs using strictjson, fix deadlock by removing lru cache (c1cd1d6)
- update prompts for code sandbox (1b7fd6a)
- use new miner selector methods (fe53836)
- add additional check for nonzero state (801796b)
- add custom check for is_miner (42f47b7)
- add dojo tasks to track to state (ed8be18)
- add empty check (f2d4543)
- add lock (67de42e)
- add missing attr (ce5f9ea)
- add missing await (0e22b2c)
- add missing cls (5f8e9b5)
- add missing dev env for wandb (8ef1274)
- add missing method (23a0fb7)
- add timeout for create task (ebf5d33)
- add verification of mturk task on validator side (0506416)
- Added changelog to npm (79d429f)
- aws lambda function (2a4000f)
- aws mturk client after refactor (845917b)
- blacklist non-aws IP on miner side (8f951f7)
- circular imports (1003bbb)
- close http client (3b01101)
- Completion not serializable (34badee)
- config class type hint (dd1159b)
- correct type hint (142b1dd)
- data manager ensure paths exist (1a7b2b3)
- deadlocks caused by nested async with lock call (f088002)
- description, not default value (4478062)
- disable parallel for custom git hook (ea6f4be)
- ensure miner able to call create task (2723966)
- ensure miner context manager handled properly (c18bf3d)
- ensure miners able to receive consensus vector (ef769cf)
- ensure non-blocking inference of HF model (da15be2)
- ensure output single label (388def4)
- ensure request routes to miners only (5b8a266)
- ensure saved miner responses unobfuscated (db9f566)
- ensure scores in positive range, ensure sum of scores is 1 (6518b78)
- ensure scoring uses unobfuscated model ids (85503fc)
- ensure to apply accuracy weights, then softmax (f9b4bb7)
- ensure to filter out aws_mturk set by miners, since it cannot be completed in 30s (d328bfc)
- ensure to pickup data that is >=8hrs old (f370117)
- ensure to remove responses after processing them, use asyncio lock (11f365b)
- ensure validator inside of context manager (1e83ecb)
- ensure validator sends single prompt-response pair, slow interval (3d8ac28)
- ensure validator startup script runs properly (5973ebd)
- expose dev env url (49c3d89)
- extract nested lock behaviour (ca353f0)
- fastapi deprecated option (5d8632b)
- fix unboundlocalerror (6c86be1)
- fixed typo that prevent miner running (17eca35)
- generated new uuid instead of using cached uuid, add some scoring (e75a3dd)
- get validator working (849b6f9)
- handle aws mturk payload properly (a696e5c)
- handle edge case where no data yet (34cbba6)
- handle startup/shutdown, fix context manager issues, remove blocking code (b0b1706)
- hide model names when sending to miners (25a8e01)
- immutable hotkey to score, extract score adjustment logic to respect, set trace to prevent log spam (971e514)
- imports (c0e83d7)
- imports and add todo (b0109a9)
- increase lru cache size to accomodate different models (59ad2ce)
- increase max instances, define using crontriggers (085bf92)
- make batch size in config.py, rename eval to evaluation to avoid clashes (81277df)
- make data manager use single instance (9eaee0d)
- make DOJO_API_KEY only required when creating task (61eb542)
- make it clearer for user to enter coldkey/hotkey name (d012d52)
- manage single httpx client (6bb5b4f)
- miner thread join (d690bf8)
- more reasonable ema and num batches (d6ff376)
- no attr timestamp (706fb03)
- prevent miner from setting weights (13192d8)
- prevent validator from calling itself (e62976a)
- prioritise mturk, allow overriding of previous accuracies (fa6a3ba)
- protocol typing, create task properly (7c48dc2)
- re-enable 20k stake required for validators (f81c097)
- re-enable wandb logging (c898fff)
- read result data properly, use strenum (b8a9b75)
- remove blocking uvicorn.run call (62cb494)
- remove discussion spam (f1dad60)
- remove duplicate convert_weights call (9c60cd0)
- remove filtering via len(ranks) (a48e5f0)
- remove hardcoded api url (6227e6a)
- remove hotkey to accuracy (8277ae2)
- remove miner signal handling (dbeca8a)
- remove mturk credentials (76de8b0)
- remove patch logging after bittensor upgrade (88e1d77)
- remove reliance on vpermit (8533c91)
- remove scored responses properly, clear TODOs (c173ff9)
- remove validate args, fix pydantic models (410b9e6)
- remove validation that prevents saving (aa1cb86)
- remove validators serving axons (a6d1978)
- resync metagraph index errors (5509ff8)
- return synapse after receiving consensus (0721cdb)
- scoring method at request level (69283eb)
- scoring method not being set in pydantic basemodel (5443950)
- set last checked time (14fc372)
- single prompt-response pair per synthetic api call (154d189)
- skip send request if failed to fetch (47f8ae1)
- store unobfuscated model ids on disk (96928e6)
- temporary workaround to test code fixing agent (25f0329)
- unknown spec version (14e7daa)
- update schedulers (9edb49b)
- update scores, shift logic to calculate scores (5b337b1)
- upon new miner/axon update scores (04f3170)
- use class variable, ruff lint (a350cd1)
- use correct argparse attr (4000e73)
- use interval instead of cron to prevent congestion (f827a1a)
- use multipart form encoder (74bc850)
- use None instead of list (8f961c5)
- use raw timestamps instead of datetime objects (a09d3b6)
- use tuple for torch size (c971dbe)
- uvicorn.run blocking call (c92cee2)
- validator: configure CORS to disable all incoming requests (f5b1a13)
- variable naming (bb4dd5c)
- wrong import (1485a31)
- wrong kwarg (9a7eaa9)
- add .env.example for reference (6d0a590)
- add aws lambda function template (3749baf)
- add aws mturk script (ed52c40)
- add base reward models for miners to use (c4f8f12)
- add blacklisting on miner side (28583ea)
- add cohen kappa and distance penalty (28e3cfe)
- add complete feedback loop for miner to send scores back to validator (6a81287)
- add data manager for ranking data (bf13235)
- add Dojo API caller (cfea834)
- add dojo task monitor on validator (8c8ff24)
- add dojo task tracker and update data on disk (6c21a10)
- add dojo task tracker, use synthetic data generator (d074fc0)
- add evals to get classification accuracy (62bd5fb)
- add fastapi routes (ed71ba0)
- add ground truth scoring for multi score (1fcfaa2)
- add ICC metric for consensus scoring (30a5f6e)
- add image scoring (d55bc54)
- add LLM API based scoring, default to mixtral 8x7b (fb16c79)
- add method to get response by request id (e3c3c1b)
- add methods to create PRs to huggingface dataset for review (7ba3f6e)
- add more datasets for evals (4649e67)
- add more models to generate answers from (90c9f7b)
- add new methods to data manager (d8a1234)
- add openai support (dbfcc44)
- add openrouter provider (26f212c)
- add scaleai code (e571660)
- add scoring method enum and simplify argparser (8042a98)
- add subscription key mgmt to cli (2b30027)
- add weights classification accuracy multiplier to scoring (6c5772c)
- apply monkey patch to bt.logging (16af344)
- boost weights for human feedback (0a928b2)
- bump version (c34d086)
- calculate distance from ground truth, combine with consensus (cbe813f)
- calculate mean across different criteria and sum (0947a5a)
- calculate spearman correlation for scoring, add config args validation, update protocol (132b944)
- change python package name (8990325)
- change validator interval to every 30s (4052332)
- cli: add coldkey and hotkey path completions (ed2f36a)
- create dojo cli for miners (56615c9)
- create tasks with ranking/multi-score (aaeef16)
- enable huggingface/llm api scoring methods for miner (011bdc9)
- enable LLM API based evals, update config (ffea364)
- enable miners to run uvicorn server via miner.py (24406e2)
- enable synthetic prompt & responses generation for coding task (6fa0b73)
- flesh out readme details, provide screenshots (74587b1)
- forward consensus score to miners (c949ebe)
- group all functions under MTurkUtils, add event handler placeholder (99ff4f2)
- implement hashring to select miners (3ba4953)
- integrate automatically creating commits on HuggingFace dataset (5d0e36c)
- load and transform multiple datasets for evals, prepare seed prompts (a72424c)
- process mturk events (1fae391)
- pull leaderboard data to get groundtruth ranks (557bdb0)
- replace cohen kappa with ICC (68f056b)
- update aws mturk script for task creation (f586335)
- update formatting (062f537)
- update LLM API for preference rankings (81cf6d0)
- update protocol (2c802e3)
- update protocol (9fb41bf)
- update readme (c852108)
- update working scaleai test script (0e996a9)
- use factory instead of singleton (d03d61e)
- use scheduler to delay scoring & sending consensus to miners (11b8eae)
- WIP update scores by criteria (bc4447a)
- add checks for no work done by worker (2b819e4)
- add custom serve axon wrapper with retries (2a3836b)
- add flow to strip random wrapper chars that aren't words (6e9e352)
- add path validation (b554ed0)
- add retry to LLM API calls, update config (1f3b627)
- ensure only aws queries miners (2de9bd1)
- increase miner create task timeout (57f4205)
- reduce task frequency from 1min to 5min (a0d010c)
- remove unstable models (197db28)
- stabilise json outputs using strictjson, fix deadlock by removing lru cache (c1cd1d6)
- update prompts for code sandbox (1b7fd6a)
- use new miner selector methods (fe53836)
- add additional check for nonzero state (801796b)
- add custom check for is_miner (42f47b7)
- add dojo tasks to track to state (ed8be18)
- add empty check (f2d4543)
- add lock (67de42e)
- add missing attr (ce5f9ea)
- add missing await (0e22b2c)
- add missing cls (5f8e9b5)
- add missing method (23a0fb7)
- add timeout for create task (ebf5d33)
- add verification of mturk task on validator side (0506416)
- Added changelog to npm (79d429f)
- aws lambda function (2a4000f)
- aws mturk client after refactor (845917b)
- blacklist non-aws IP on miner side (8f951f7)
- circular imports (1003bbb)
- close http client (3b01101)
- Completion not serializable (34badee)
- config class type hint (dd1159b)
- correct type hint (142b1dd)
- data manager ensure paths exist (1a7b2b3)
- deadlocks caused by nested async with lock call (f088002)
- description, not default value (4478062)
- disable parallel for custom git hook (ea6f4be)
- ensure miner able to call create task (2723966)
- ensure miner context manager handled properly (c18bf3d)
- ensure miners able to receive consensus vector (ef769cf)
- ensure non-blocking inference of HF model (da15be2)
- ensure output single label (388def4)
- ensure request routes to miners only (5b8a266)
- ensure saved miner responses unobfuscated (db9f566)
- ensure scores in positive range, ensure sum of scores is 1 (6518b78)
- ensure scoring uses unobfuscated model ids (85503fc)
- ensure to apply accuracy weights, then softmax (f9b4bb7)
- ensure to filter out aws_mturk set by miners, since it cannot be completed in 30s (d328bfc)
- ensure to pickup data that is >=8hrs old (f370117)
- ensure to remove responses after processing them, use asyncio lock (11f365b)
- ensure validator inside of context manager (1e83ecb)
- ensure validator sends single prompt-response pair, slow interval (3d8ac28)
- ensure validator startup script runs properly (5973ebd)
- expose dev env url (49c3d89)
- extract nested lock behaviour (ca353f0)
- fastapi deprecated option (5d8632b)
- fix unboundlocalerror (6c86be1)
- fixed typo that prevent miner running (17eca35)
- generated new uuid instead of using cached uuid, add some scoring (e75a3dd)
- get validator working (849b6f9)
- handle aws mturk payload properly (a696e5c)
- handle edge case where no data yet (34cbba6)
- handle startup/shutdown, fix context manager issues, remove blocking code (b0b1706)
- hide model names when sending to miners (25a8e01)
- immutable hotkey to score, extract score adjustment logic to respect, set trace to prevent log spam (971e514)
- imports (c0e83d7)
- imports and add todo (b0109a9)
- increase lru cache size to accomodate different models (59ad2ce)
- increase max instances, define using crontriggers (085bf92)
- make batch size in config.py, rename eval to evaluation to avoid clashes (81277df)
- make data manager use single instance (9eaee0d)
- make DOJO_API_KEY only required when creating task (61eb542)
- make it clearer for user to enter coldkey/hotkey name (d012d52)
- manage single httpx client (6bb5b4f)
- miner thread join (d690bf8)
- more reasonable ema and num batches (d6ff376)
- no attr timestamp (706fb03)
- prevent miner from setting weights (13192d8)
- prevent validator from calling itself (e62976a)
- prioritise mturk, allow overriding of previous accuracies (fa6a3ba)
- protocol typing, create task properly (7c48dc2)
- re-enable 20k stake required for validators (f81c097)
- re-enable wandb logging (c898fff)
- read result data properly, use strenum (b8a9b75)
- remove blocking uvicorn.run call (62cb494)
- remove discussion spam (f1dad60)
- remove duplicate convert_weights call (9c60cd0)
- remove filtering via len(ranks) (a48e5f0)
- remove hardcoded api url (6227e6a)
- remove hotkey to accuracy (8277ae2)
- remove miner signal handling (dbeca8a)
- remove mturk credentials (76de8b0)
- remove patch logging after bittensor upgrade (88e1d77)
- remove reliance on vpermit (8533c91)
- remove scored responses properly, clear TODOs (c173ff9)
- remove validate args, fix pydantic models (410b9e6)
- remove validation that prevents saving (aa1cb86)
- remove validators serving axons (a6d1978)
- resync metagraph index errors (5509ff8)
- return synapse after receiving consensus (0721cdb)
- scoring method at request level (69283eb)
- scoring method not being set in pydantic basemodel (5443950)
- set last checked time (14fc372)
- single prompt-response pair per synthetic api call (154d189)
- skip send request if failed to fetch (47f8ae1)
- store unobfuscated model ids on disk (96928e6)
- temporary workaround to test code fixing agent (25f0329)
- unknown spec version (14e7daa)
- update schedulers (9edb49b)
- update scores, shift logic to calculate scores (5b337b1)
- upon new miner/axon update scores (04f3170)
- use class variable, ruff lint (a350cd1)
- use correct argparse attr (4000e73)
- use interval instead of cron to prevent congestion (f827a1a)
- use multipart form encoder (74bc850)
- use None instead of list (8f961c5)
- use raw timestamps instead of datetime objects (a09d3b6)
- use tuple for torch size (c971dbe)
- uvicorn.run blocking call (c92cee2)
- validator: configure CORS to disable all incoming requests (f5b1a13)
- variable naming (bb4dd5c)
- wrong import (1485a31)
- wrong kwarg (9a7eaa9)
- add .env.example for reference (6d0a590)
- add aws lambda function template (3749baf)
- add aws mturk script (ed52c40)
- add base reward models for miners to use (c4f8f12)
- add blacklisting on miner side (28583ea)
- add cohen kappa and distance penalty (28e3cfe)
- add complete feedback loop for miner to send scores back to validator (6a81287)
- add data manager for ranking data (bf13235)
- add Dojo API caller (cfea834)
- add dojo task monitor on validator (8c8ff24)
- add dojo task tracker and update data on disk (6c21a10)
- add dojo task tracker, use synthetic data generator (d074fc0)
- add evals to get classification accuracy (62bd5fb)
- add fastapi routes (ed71ba0)
- add ground truth scoring for multi score (1fcfaa2)
- add ICC metric for consensus scoring (30a5f6e)
- add image scoring (d55bc54)
- add LLM API based scoring, default to mixtral 8x7b (fb16c79)
- add method to get response by request id (e3c3c1b)
- add methods to create PRs to huggingface dataset for review (7ba3f6e)
- add more datasets for evals (4649e67)
- add more models to generate answers from (90c9f7b)
- add new methods to data manager (d8a1234)
- add openai support (dbfcc44)
- add openrouter provider (26f212c)
- add scaleai code (e571660)
- add scoring method enum and simplify argparser (8042a98)
- add subscription key mgmt to cli (2b30027)
- add weights classification accuracy multiplier to scoring (6c5772c)
- apply monkey patch to bt.logging (16af344)
- boost weights for human feedback (0a928b2)
- bump version (c34d086)
- calculate distance from ground truth, combine with consensus (cbe813f)
- calculate mean across different criteria and sum (0947a5a)
- calculate spearman correlation for scoring, add config args validation, update protocol (132b944)
- change python package name (8990325)
- change validator interval to every 30s (4052332)
- cli: add coldkey and hotkey path completions (ed2f36a)
- create dojo cli for miners (56615c9)
- create tasks with ranking/multi-score (aaeef16)
- enable huggingface/llm api scoring methods for miner (011bdc9)
- enable LLM API based evals, update config (ffea364)
- enable miners to run uvicorn server via miner.py (24406e2)
- enable synthetic prompt & responses generation for coding task (6fa0b73)
- flesh out readme details, provide screenshots (74587b1)
- forward consensus score to miners (c949ebe)
- group all functions under MTurkUtils, add event handler placeholder (99ff4f2)
- implement hashring to select miners (3ba4953)
- integrate automatically creating commits on HuggingFace dataset (5d0e36c)
- load and transform multiple datasets for evals, prepare seed prompts (a72424c)
- process mturk events (1fae391)
- pull leaderboard data to get groundtruth ranks (557bdb0)
- replace cohen kappa with ICC (68f056b)
- update aws mturk script for task creation (f586335)
- update formatting (062f537)
- update LLM API for preference rankings (81cf6d0)
- update protocol (2c802e3)
- update protocol (9fb41bf)
- update readme (c852108)
- update working scaleai test script (0e996a9)
- use factory instead of singleton (d03d61e)
- use scheduler to delay scoring & sending consensus to miners (11b8eae)
- WIP update scores by criteria (bc4447a)
- add checks for no work done by worker (2b819e4)
- add custom serve axon wrapper with retries (2a3836b)
- add flow to strip random wrapper chars that aren't words (6e9e352)
- add path validation (b554ed0)
- add retry to LLM API calls, update config (1f3b627)
- ensure only aws queries miners (2de9bd1)
- increase miner create task timeout (57f4205)
- reduce task frequency from 1min to 5min (a0d010c)
- remove unstable models (197db28)
- stabilise json outputs using strictjson, fix deadlock by removing lru cache (c1cd1d6)
- update prompts for code sandbox (1b7fd6a)
- use new miner selector methods (fe53836)
- add additional check for nonzero state (801796b)
- add custom check for is_miner (42f47b7)
- add dojo tasks to track to state (ed8be18)
- add empty check (f2d4543)
- add lock (67de42e)
- add missing attr (ce5f9ea)
- add missing await (0e22b2c)
- add missing cls (5f8e9b5)
- add missing method (23a0fb7)
- add timeout for create task (ebf5d33)
- add verification of mturk task on validator side (0506416)
- Added changelog to npm (79d429f)
- aws lambda function (2a4000f)
- aws mturk client after refactor (845917b)
- blacklist non-aws IP on miner side (8f951f7)
- circular imports (1003bbb)
- close http client (3b01101)
- Completion not serializable (34badee)
- config class type hint (dd1159b)
- correct type hint (142b1dd)
- data manager ensure paths exist (1a7b2b3)
- deadlocks caused by nested async with lock call (f088002)
- description, not default value (4478062)
- disable parallel for custom git hook (ea6f4be)
- ensure miner able to call create task (2723966)
- ensure miner context manager handled properly (c18bf3d)
- ensure miners able to receive consensus vector (ef769cf)
- ensure non-blocking inference of HF model (da15be2)
- ensure output single label (388def4)
- ensure request routes to miners only (5b8a266)
- ensure saved miner responses unobfuscated (db9f566)
- ensure scores in positive range, ensure sum of scores is 1 (6518b78)
- ensure scoring uses unobfuscated model ids (85503fc)
- ensure to apply accuracy weights, then softmax (f9b4bb7)
- ensure to filter out aws_mturk set by miners, since it cannot be completed in 30s (d328bfc)
- ensure to pickup data that is >=8hrs old (f370117)
- ensure to remove responses after processing them, use asyncio lock (11f365b)
- ensure validator inside of context manager (1e83ecb)
- ensure validator sends single prompt-response pair, slow interval (3d8ac28)
- ensure validator startup script runs properly (5973ebd)
- expose dev env url (49c3d89)
- extract nested lock behaviour (ca353f0)
- fastapi deprecated option (5d8632b)
- fix unboundlocalerror (6c86be1)
- fixed typo that prevent miner running (17eca35)
- generated new uuid instead of using cached uuid, add some scoring (e75a3dd)
- get validator working (849b6f9)
- handle aws mturk payload properly (a696e5c)
- handle edge case where no data yet (34cbba6)
- handle startup/shutdown, fix context manager issues, remove blocking code (b0b1706)
- hide model names when sending to miners (25a8e01)
- immutable hotkey to score, extract score adjustment logic to respect, set trace to prevent log spam (971e514)
- imports (c0e83d7)
- imports and add todo (b0109a9)
- increase lru cache size to accomodate different models (59ad2ce)
- increase max instances, define using crontriggers (085bf92)
- make batch size in config.py, rename eval to evaluation to avoid clashes (81277df)
- make data manager use single instance (9eaee0d)
- make DOJO_API_KEY only required when creating task (61eb542)
- make it clearer for user to enter coldkey/hotkey name (d012d52)
- manage single httpx client (6bb5b4f)
- miner thread join (d690bf8)
- more reasonable ema and num batches (d6ff376)
- no attr timestamp (706fb03)
- prevent miner from setting weights (13192d8)
- prevent validator from calling itself (e62976a)
- prioritise mturk, allow overriding of previous accuracies (fa6a3ba)
- protocol typing, create task properly (7c48dc2)
- re-enable 20k stake required for validators (f81c097)
- re-enable wandb logging (c898fff)
- read result data properly, use strenum (b8a9b75)
- remove blocking uvicorn.run call (62cb494)
- remove discussion spam (f1dad60)
- remove duplicate convert_weights call (9c60cd0)
- remove filtering via len(ranks) (a48e5f0)
- remove hardcoded api url (6227e6a)
- remove hotkey to accuracy (8277ae2)
- remove miner signal handling (dbeca8a)
- remove mturk credentials (76de8b0)
- remove patch logging after bittensor upgrade (88e1d77)
- remove reliance on vpermit (8533c91)
- remove scored responses properly, clear TODOs (c173ff9)
- remove validate args, fix pydantic models (410b9e6)
- remove validation that prevents saving (aa1cb86)
- remove validators serving axons (a6d1978)
- resync metagraph index errors (5509ff8)
- return synapse after receiving consensus (0721cdb)
- scoring method at request level (69283eb)
- scoring method not being set in pydantic basemodel (5443950)
- set last checked time (14fc372)
- single prompt-response pair per synthetic api call (154d189)
- skip send request if failed to fetch (47f8ae1)
- store unobfuscated model ids on disk (96928e6)
- temporary workaround to test code fixing agent (25f0329)
- unknown spec version (14e7daa)
- update schedulers (9edb49b)
- update scores, shift logic to calculate scores (5b337b1)
- upon new miner/axon update scores (04f3170)
- use class variable, ruff lint (a350cd1)
- use correct argparse attr (4000e73)
- use interval instead of cron to prevent congestion (f827a1a)
- use multipart form encoder (74bc850)
- use None instead of list (8f961c5)
- use raw timestamps instead of datetime objects (a09d3b6)
- use tuple for torch size (c971dbe)
- uvicorn.run blocking call (c92cee2)
- validator: configure CORS to disable all incoming requests (f5b1a13)
- variable naming (bb4dd5c)
- wrong import (1485a31)
- wrong kwarg (9a7eaa9)
- add .env.example for reference (6d0a590)
- add aws lambda function template (3749baf)
- add aws mturk script (ed52c40)
- add base reward models for miners to use (c4f8f12)
- add blacklisting on miner side (28583ea)
- add cohen kappa and distance penalty (28e3cfe)
- add complete feedback loop for miner to send scores back to validator (6a81287)
- add data manager for ranking data (bf13235)
- add Dojo API caller (cfea834)
- add dojo task monitor on validator (8c8ff24)
- add dojo task tracker and update data on disk (6c21a10)
- add dojo task tracker, use synthetic data generator (d074fc0)
- add evals to get classification accuracy (62bd5fb)
- add fastapi routes (ed71ba0)
- add ground truth scoring for multi score (1fcfaa2)
- add ICC metric for consensus scoring (30a5f6e)
- add image scoring (d55bc54)
- add LLM API based scoring, default to mixtral 8x7b (fb16c79)
- add method to get response by request id (e3c3c1b)
- add methods to create PRs to huggingface dataset for review (7ba3f6e)
- add more datasets for evals (4649e67)
- add more models to generate answers from (90c9f7b)
- add new methods to data manager (d8a1234)
- add openai support (dbfcc44)
- add openrouter provider (26f212c)
- add scaleai code (e571660)
- add scoring method enum and simplify argparser (8042a98)
- add subscription key mgmt to cli (2b30027)
- add weights classification accuracy multiplier to scoring (6c5772c)
- apply monkey patch to bt.logging (16af344)
- boost weights for human feedback (0a928b2)
- bump version (c34d086)
- calculate distance from ground truth, combine with consensus (cbe813f)
- calculate mean across different criteria and sum (0947a5a)
- calculate spearman correlation for scoring, add config args validation, update protocol (132b944)
- change python package name (8990325)
- change validator interval to every 30s (4052332)
- cli: add coldkey and hotkey path completions (ed2f36a)
- create dojo cli for miners (56615c9)
- create tasks with ranking/multi-score (aaeef16)
- enable huggingface/llm api scoring methods for miner (011bdc9)
- enable LLM API based evals, update config (ffea364)
- enable miners to run uvicorn server via miner.py (24406e2)
- enable synthetic prompt & responses generation for coding task (6fa0b73)
- flesh out readme details, provide screenshots (74587b1)
- forward consensus score to miners (c949ebe)
- group all functions under MTurkUtils, add event handler placeholder (99ff4f2)
- implement hashring to select miners (3ba4953)
- integrate automatically creating commits on HuggingFace dataset (5d0e36c)
- load and transform multiple datasets for evals, prepare seed prompts (a72424c)
- process mturk events (1fae391)
- pull leaderboard data to get groundtruth ranks (557bdb0)
- replace cohen kappa with ICC (68f056b)
- update aws mturk script for task creation (f586335)
- update formatting (062f537)
- update LLM API for preference rankings (81cf6d0)
- update protocol (2c802e3)
- update protocol (9fb41bf)
- update readme (c852108)
- update working scaleai test script (0e996a9)
- use factory instead of singleton (d03d61e)
- use scheduler to delay scoring & sending consensus to miners (11b8eae)
- WIP update scores by criteria (bc4447a)
- add checks for no work done by worker (2b819e4)
- add custom serve axon wrapper with retries (2a3836b)
- add flow to strip random wrapper chars that aren't words (6e9e352)
- add path validation (b554ed0)
- add retry to LLM API calls, update config (1f3b627)
- ensure only aws queries miners (2de9bd1)
- increase miner create task timeout (57f4205)
- reduce task frequency from 1min to 5min (a0d010c)
- remove unstable models (197db28)
- stabilise json outputs using strictjson, fix deadlock by removing lru cache (c1cd1d6)
- update prompts for code sandbox (1b7fd6a)
- use new miner selector methods (fe53836)