v0.9.1 - Gymnasium and Mujoco

Version 0.9.1. Includes the following updates:

• Adds Gymnasium support
• Adds Mujoco support
• Adds hyperparameters logging after running experiments
• Some other minor improvements to logging
• Tweaks to SAC/DDPG hyperparameters and implementation
• Minor workflow improvements
• Some other dependency upgrades, including upgrading to torch~=2.2
• Some minor bugfixes

What's Changed

• Release/0.8.1 by @cpnota in #274
• change opencv dependency to headless and upgrade to version 4 by @cpnota in #275
• Release/0.8.2 by @cpnota in #276
• Feature/gymnasium by @cpnota in #278
• Feature/mujoco by @cpnota in #279
• Refactor/scripts-folder by @cpnota in #286
• add call method to Builder API and unit tests by @cpnota in #287
• Feature/episode length by @cpnota in #289
• add entropy_backups hyperparameter to SAC by @cpnota in #296
• Refactor/formatting by @cpnota in #299
• Fix key error warnings by @cpnota in #300
• Finish docstring for nn aggregation by @cpnota in #301
• Bugfix/publish workflow by @cpnota in #303
• Add save_freq argument and refactor scripts by @cpnota in #305
• Hyperparameter Logging by @cpnota in #308
• remove env name from hparams tag by @cpnota in #309
• SAC/DDPG tweaks by @cpnota in #312
• fix duplicate env handling by @cpnota in #314
• Upgrade dependencies by @cpnota in #315
• fix plotter and log final summary at end of training by @cpnota in #320
• add swig setup dependency and remove unrar/swig from github scripts by @cpnota in #321
• Feature/benchmarks by @cpnota in #317
• Update documentation by @cpnota in #323
• v0.9.1 - Gymnasium and Mujoco by @cpnota in #324

Full Changelog: v0.8.0...v0.9.1

v0.9.1-alpha.3

Publish workflow testing

Dependency Upgrades v0.8.0

This release includes several enhancements:

• Dependencies have been updated.
• FeatureNetwork logic was simplified.
• Soft Actor-Critic was upgraded to a newer version which does not use a separate state-value function.
• The logging interface was improved.

Maintenance 0.7.2

• Update Pytorch to 1.9.0 (#255)
• Raise RuntimeError if clip_grad is enabled and norm is non-finite (#255)
• Fix a bug with scaling of log_prob in soft policy (#256)

Maintenance 0.7.1

A few minor under-the-hood tweaks and fixes:

• Created a VectorEnvironment class and refactored the way parallel environments work #239
• Added the parallel_test_agent method to ParallelPreset #240
• Fixed a bug where using n_envs=1 on Atari presets used Body instead of ParallelBody #241
• Fix DeepmindAtariBody to only use FrameStackBody if frame_stack > 1 #245
• Fix bug where test_exploration was not being respected by GreedyPolicy #246
• Improve stability of SAC by preventing the temperature from dropping below 0 #247
• Update version of PettingZoo and use new environments in CI #351
• Fix store_device for prioritized replay #249

Save/Load

This release contains several new features, refactors, and bugfixes.

Features

• Save/load agents. #185
• (Experimental) Multiagent Atari support using PettingZoo. #201
• Optional to store replay buffer on different device. #187
• Better environment duplication using cloudpickle. #200
• Built-in Identity feature network #202
• Support for Comet.ml #215

Refactoring

• Agents split into three types: Agent, ParallelAgent, and Multiagent #221
• Changed the way Presets work to facilitate Save/Load #185
• Switched from Travis to GitHub Actions for workflow/CI #235
• Improvements to Environment imports #236
• Support for latest PyTorch version #235

Bugfixes

• Fixes a bug in ParallelGreedyPolicy #233
• Fixes an issue with the Atari life_lost #
• Documentation clarifications #192 #216

State Array

This release contains some under-the-hood enhancements and bugfixes, most notably, a refactoring of the State class. State now supports adding arbitrary key/value pairs, allowing for more complex state spaces. Additionally, a StateArray class was added that automatically handles stacking/slicing states in various ways which making handling batches of data, multiple timesteps, and many other aspects easier. Here's a full list of the changes:

• Refacted State and added StateArray object ( #160 and #167)
• Added support for specifying custom models under all existing presets. Thanks to @michalgregor for the contribution! (#163)
• Fixed a bug involving SAC evaluation mode. Thanks @michalgregor for identifying and proposing a fix! (#169)
• Fixed a bug where the name of preconstructed Gym environments were not being handled correctly. Thanks @mctigger for reporting! (#169) Thanks @michalgregor for the fix! (#165, merged by #170)
• Fixed a bug where the Atari FireReset wrapper was being applied to games with no Fire action, causing them to not run. Thanks @andrewsmike for reporting and fixing the bug! (#168)

Maintenance 0.5.3

This release contains a hotfix, #155 , which improves the performance of the PPO continuous preset.

Maintenance 0.5.2

Just some minor bug fixes and documentation improvements:

• Datetime compatibility for Windows #137 #142
• Continuous Integration fixes #138
• SoftDeterministicPolicy scaling fix #140
• Fix incorrect counting of test trials in parallel experiments #143
• Remove trailing commas #146
• First action was being selected using act() instead of eval() in test mode #150
• Documentation improvements #151

FPS Hotfix

The previous release was missing the changes from #132 with correct the computation of the frames per second for parallel envs.