[Algorithm] SOTA discrete offline CQL #3098

Ibinarriaga8 · 2025-07-28T10:44:37Z

Description

This PR solves issue #3097 by providing a state-of-the-art (SOTA) implementation for discrete offline Conservative Q-Learning (CQL) within the repository.

Implemented a SOTA discrete offline CQL algorithm fully integrated into the codebase.
As with the SOTA implementation of discrete online CQL (already present in the repo), the CartPole environment was used for offline discrete CQL training.
Leveraged the changes from commit 0627e85c78756b4e4dbde726a9d5e85300e239a0 to utilize the updated MinariExperienceReplay class for loading experiences from a custom CartPole dataset into the offline replay buffer.
Updated the discrete loss function according to the latest torchrl documentation, explicitly handling the categorical action space as required for discrete environments.
Added a SOTA-check test for this new algorithm to ensure correctness and performance.

Motivation and Context

This change was required to complete the set of SOTA CQL implementations in the repository, specifically by adding support for discrete offline CQL. Previously, only the online discrete CQL and both online/offline continuous CQL were available. By implementing discrete offline CQL, the repository now covers all major CQL benchmarks, enabling users to run and benchmark offline RL in discrete action spaces natively within torchrl.

The change also brings the repository in line with other top RL codebases and helps the community reproducibly benchmark and compare discrete offline RL algorithms.

This PR closes #3097.

I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds core functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (update in the documentation)
Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

I have read the CONTRIBUTION guide (required)
My change requires a change to the documentation.
I have updated the tests accordingly (required for a bug fix or a new feature).
I have updated the documentation accordingly.

… CQL offline

…ing script

…offline

…ration

Sync branch

pytorch-bot · 2025-07-28T10:44:42Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3098

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 35 Pending

As of commit ef6b66f with merge base 009f4ce ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Ibinarriaga8 · 2025-07-28T10:45:51Z

sota-check/run_discrete_cql.sh

Added sota-check

Ibinarriaga8 · 2025-07-28T10:51:59Z

sota-implementations/cql/utils.py


    return data




Replay buffer from a custom minari dataset

Ibinarriaga8 · 2025-07-28T10:52:30Z

sota-implementations/cql/utils.py

        model,
        loss_function=loss_cfg.loss_function,
+        action_space="categorical",
        delay_value=True,


Updated make_discrete_loss to follow torchrl documentation

Ibinarriaga8 · 2025-07-28T11:00:31Z

Log runs:

https://api.wandb.ai/links/202206789-universidad-pontificia-comillas/ir36pc30

vmoens

I think we're good to go when the linter is fixed!

vmoens · 2025-07-28T16:57:56Z

sota-implementations/cql/discrete_online_config.yaml

  mode: online
  eval_iter: 1000
-  video: False
+  video: True


Do we want this?

…tation

Ibinarriaga8 · 2025-07-30T09:24:02Z

@vmoens I made some changes to make sure SOTA checks work:

I applied minor fixes to the make_discrete_loss function in utils.py, and added the necessary commands in test_sota.py, including the sota-check for my new algorithm.
Please, lmk if anything else is needed for the sota-check

Ibinarriaga8 · 2025-07-30T13:46:56Z

@vmoens I noticed that the minari dependency is missing from the linux_sota environment. Do you think it’s sufficient to simply add it to linux-sota/scripts/environment.yml?

Ibinarriaga8 · 2025-07-31T15:22:24Z

SOTA-checks were successful
Lmk if there is anything else needed

…offline

jorge.ibinarriaga.robles.becas and others added 29 commits July 1, 2025 12:03

[Feature] Replace D4RLExperienceReplay with MinariExperienceReplay in…

611f3ee

… CQL offline

[Feature] Update environment configuration and add discrete CQL train…

dbf2664

…ing script

[Feature] test.py for MinariExperienceReplay integration

ee335ff

[Feature] Updated MinariExperienceReplay to load Minari local datasets

b6a4436

[Feature] Add support for loading datasets from local Minari cache

26aa0ff

[Refactor] Fixed linting errors

a56c508

Merge remote-tracking branch 'origin/main' into feature/discrete_cql_…

16618a6

…offline

Solved conflicts

e83ac27

[Refactor] moved minari import to local scope

e90f4d0

[Refactor] Refactor Minari testing to use custom datasets

cc43f9a

[Feature] Implement offline discrete replay buffer and update configu…

5ca9e61

…ration

[Feature] Update CQL discrete

d667438

[Fix] eliminate global imports for test_libs

23aecec

[Refactor] Reduce dataset index range in Minari tests

9321adb

Merge branch 'pytorch:main' into main

a4993c3

Merge branch 'main' into feature/discrete_cql_offline

5817ea2

[Feature] CQL

08a206b

Merge branch 'pytorch:main' into main

08779c2

[Fix] update replay buffer episodes

fc0bb9a

Merge pull request #2 from Ibinarriaga8/main

3190e23

Sync branch

[Refactor] Fixed linting errors

ed08d8c

[Feature] Added discrete offline CQL sota-check

f556f75

Merge branch 'main' into feature/discrete_cql_offline

83d3254

Merge branch 'main' into feature/discrete_cql_offline

7213ab7

resolved conflicts

f73ccc8

Update with main

36daed5

update branch

08eda89

Refactored make_offline_discrete_replay_buffer

de11db7

Unnecesary file

35f8a04

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 28, 2025

Ibinarriaga8 commented Jul 28, 2025

View reviewed changes

sota-check/run_discrete_cql.sh

Copy link

Contributor Author

Ibinarriaga8 Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added sota-check

Ibinarriaga8 commented Jul 28, 2025

View reviewed changes

vmoens added the new algo New algorithm request or PR label Jul 28, 2025

vmoens approved these changes Jul 28, 2025

View reviewed changes

Ibinarriaga8 added 5 commits July 29, 2025 11:05

Merge branch 'main' into feature/discrete_cql_offline

6b2806f

[Fix] Update video logging setting and fixed linting

27a065a

[Fix] misconfiguration of loss function in SOTA discrete CQL implemen…

7a977f9

…tation

[Feature] Add command for discrete CQL offline in test-sota

a58db4f

Merge branch 'main' into feature/discrete_cql_offline

9e58a9d

Ibinarriaga8 added 2 commits July 30, 2025 11:36

[Fix] Linting

6ac2ea1

[Fix] Update discrete CQL offline command in test_sota.py

c8b1a1d

Ibinarriaga8 added 3 commits July 31, 2025 09:53

Added minari dependency to the sota-environment

cf690a7

[Feature] Add 'minari[create]' to environment dependencies

43481cd

Merge branch 'main' into feature/discrete_cql_offline

b255b80

vmoens added 2 commits August 1, 2025 15:28

Merge remote-tracking branch 'origin/main' into feature/discrete_cql_…

1b621d7

…offline

lint

ef6b66f

vmoens merged commit 1eccb49 into pytorch:main Aug 1, 2025
51 of 72 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Algorithm] SOTA discrete offline CQL #3098

[Algorithm] SOTA discrete offline CQL #3098

Uh oh!

Ibinarriaga8 commented Jul 28, 2025

Uh oh!

pytorch-bot bot commented Jul 28, 2025 •

edited

Loading

Uh oh!

Ibinarriaga8 Jul 28, 2025

Uh oh!

Ibinarriaga8 Jul 28, 2025

Uh oh!

Ibinarriaga8 Jul 28, 2025

Uh oh!

Ibinarriaga8 commented Jul 28, 2025

Uh oh!

vmoens left a comment

Uh oh!

vmoens Jul 28, 2025

Uh oh!

Ibinarriaga8 commented Jul 30, 2025

Uh oh!

Ibinarriaga8 commented Jul 30, 2025

Uh oh!

Ibinarriaga8 commented Jul 31, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[Algorithm] SOTA discrete offline CQL #3098

[Algorithm] SOTA discrete offline CQL #3098

Uh oh!

Conversation

Ibinarriaga8 commented Jul 28, 2025

Description

Motivation and Context

Types of changes

Checklist

Uh oh!

pytorch-bot bot commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3098

⏳ No Failures, 35 Pending

Uh oh!

Ibinarriaga8 Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Ibinarriaga8 Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Ibinarriaga8 Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Ibinarriaga8 commented Jul 28, 2025

Uh oh!

vmoens left a comment

Choose a reason for hiding this comment

Uh oh!

vmoens Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Ibinarriaga8 commented Jul 30, 2025

Uh oh!

Ibinarriaga8 commented Jul 30, 2025

Uh oh!

Ibinarriaga8 commented Jul 31, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pytorch-bot bot commented Jul 28, 2025 •

edited

Loading