Remove scripts altogether #7592

lhoestq · 2025-06-04T15:14:11Z

TODO:

remplace fixtures based on script with no-script fixtures
windaube

HuggingFaceDocBuilderDev · 2025-06-04T15:17:16Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

jp1924 · 2025-07-14T06:57:27Z

Hi @lhoestq,
I wanted to ask
are you planning to stop supporting dataset builds using GeneratorBasedBuilder?

If so, could you share the reason why?

lhoestq · 2025-07-15T10:23:14Z

We stopped supporting dataset scripts altogether, whether they are based on GeneratorBasedBuilder or any other builder. This means you can't load_dataset() a dataset script anymore. We did this mostly for security reasons which is blocking for many users and also impossible to build upon (e.g. the for the Dataset Viewer on HF)

jp1924 · 2025-07-16T18:59:07Z

Ah, so only the trust_remote_code feature of load_dataset is deprecated, and

from datasets import load_dataset_builder
  
builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')  
builder.download_and_prepare()

we can still load data using load_dataset_builder and download_and_prepare, right?
That's a relief. I thought the removal of trust_remote_code in load_dataset meant GeneratorBasedBuilder was being deprecated too, haha.
Got it, thanks for the clarification!

autosquid · 2025-08-04T15:14:06Z

Can you give an example on how to upgrade from using trust_remote_code? I used to load_dataset from a script generating my training data in a streaming way.

autosquid · 2025-08-04T15:16:46Z

For guys who dislike this change +1

…face/datasets#7592; changed MMLU to an up-to-date Parquet dataset repository and made the required changes to the module

lhoestq · 2025-09-04T11:03:43Z

Can you give an example on how to upgrade from using trust_remote_code? I used to load_dataset from a script generating my training data in a streaming way.

here is a command to convert a script based dataset on HF to a Parquet dataset:

uv run --with "datasets==3.6.0" datasets-cli convert_to_parquet <username/dataset-name> --trust_remote_code

And if you want to define a streaming pipeline, feel free to use the IterableDataset API instead:

from datasets import load_dataset

ds = load_dataset(..., streaming=True)  # source data
ds = ds.map(...)  # preprocessing
ds = ds.filter(...)  # remove bad samples

# For training
# data_loader = DataLoader(ds, ...)

# Saving the dataset
# ds.push_to_hub(...)

## Why are these changes needed? Hugging Face Datasets v4.0.0 removed dataset scripts and the related “dynamic modules” mechanism (See huggingface/datasets#7592), so `datasets.load.init_dynamic_modules` no longer exists. As `ray.data._internal.datasource.huggingface_datasource` currently imports/calls this function unconditionally, it raises: ``` AttributeError: module 'datasets.load' has no attribute 'init_dynamic_modules' ``` With this PR, Ray detects the legacy initializer and only calls it on Datasets ≤3.x. On Datasets ≥4.0.0 the block becomes a no-op. The behavior for older Datasets remains unchanged. ## Related issue number NA ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Edward Oakes <[email protected]> Signed-off-by: Jiajun Yao <[email protected]> Signed-off-by: Matthew Owen <[email protected]> Signed-off-by: Sagar Sumit <[email protected]> Signed-off-by: xgui <[email protected]> Signed-off-by: Xinyuan <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Matthew Deng <[email protected]> Signed-off-by: Goutam V <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: elliot-barn <[email protected]> Signed-off-by: Yiwen Xiang <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: dragongu <[email protected]> Signed-off-by: Andrew Grosser <[email protected]> Signed-off-by: Justin Yu <[email protected]> Signed-off-by: abrar <[email protected]> Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Potato <[email protected]> Signed-off-by: Zac Policzer <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: 杨睿 <[email protected]> Signed-off-by: dayshah <[email protected]> Signed-off-by: axreldable <[email protected]> Signed-off-by: kaihsun <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Timothy Seah <[email protected]> Signed-off-by: kevin <[email protected]> Signed-off-by: joshlee <[email protected]> Signed-off-by: 400Ping <[email protected]> Signed-off-by: Dhyey Shah <[email protected]> Signed-off-by: harshit <[email protected]> Signed-off-by: anmol <[email protected]> Signed-off-by: Kit Lee <[email protected]> Signed-off-by: JasonLi1909 <[email protected]> Signed-off-by: Jason Li <[email protected]> Signed-off-by: MengqingCao <[email protected]> Signed-off-by: sampan <[email protected]> Signed-off-by: zac <[email protected]> Signed-off-by: Elliot Barnwell <[email protected]> Signed-off-by: Mengjin Yan <[email protected]> Signed-off-by: myan <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: Linkun <[email protected]> Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Markus <[email protected]> Signed-off-by: Gagandeep Singh <[email protected]> Signed-off-by: akyang-anyscale <[email protected]> Signed-off-by: Alan Guo <[email protected]> Signed-off-by: haotian <[email protected]> Signed-off-by: Howie Tien <[email protected]> Signed-off-by: will.lin <[email protected]> Signed-off-by: Richard Liaw <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Andrew Sy Kim <[email protected]> Signed-off-by: Matvei Pashkovskii <[email protected]> Signed-off-by: Kishanthan Thangarajah <[email protected]> Signed-off-by: my-vegetable-has-exploded <[email protected]> Signed-off-by: Neil Girdhar <[email protected]> Signed-off-by: Nikhil Ghosh <[email protected]> Signed-off-by: win5923 <[email protected]> Signed-off-by: Stephanie wang <[email protected]> Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: cong.qian <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: doyoung <[email protected]> Signed-off-by: Doyoung Kim <[email protected]> Signed-off-by: simonsays1980 <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Sampan S Nayak <[email protected]> Signed-off-by: vincenthhan <[email protected]> Signed-off-by: jeffreyjeffreywang <[email protected]> Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: avibasnet31 <[email protected]> Signed-off-by: Tanner Wood <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: Ricardo Decal <[email protected]> Signed-off-by: Matthew <[email protected]> Signed-off-by: tianyi-ge <[email protected]> Signed-off-by: can <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: ahao-anyscale <[email protected]> Signed-off-by: Lehui Liu <[email protected]> Signed-off-by: Mark Rossetti <[email protected]> Signed-off-by: hejialing.hjl <[email protected]> Signed-off-by: qiwenju <[email protected]> Signed-off-by: Q文举 <[email protected]> Signed-off-by: ljstrnadiii <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Lonnie Liu <[email protected]> Co-authored-by: Rueian <[email protected]> Co-authored-by: Dhyey Shah <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Co-authored-by: Jiajun Yao <[email protected]> Co-authored-by: Matthew Owen <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> Co-authored-by: Xinyuan <[email protected]> Co-authored-by: Justin Yu <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: goutamvenkat-anyscale <[email protected]> Co-authored-by: vickytsang <[email protected]> Co-authored-by: Rui Qiao <[email protected]> Co-authored-by: Elliot Barnwell <[email protected]> Co-authored-by: Yevet <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Srinath Krishnamachari <[email protected]> Co-authored-by: iamjustinhsu <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: dragongu <[email protected]> Co-authored-by: Andrew Grosser <[email protected]> Co-authored-by: Abrar Sheikh <[email protected]> Co-authored-by: kourosh hakhamaneshi <[email protected]> Co-authored-by: Yicheng-Lu-llll <[email protected]> Co-authored-by: Potato <[email protected]> Co-authored-by: Zac Policzer <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: Seiji Eicher <[email protected]> Co-authored-by: 杨睿 <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Aleksei Starikov <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Kevin H. Luu <[email protected]> Co-authored-by: Qiaolin Yu <[email protected]> Co-authored-by: Joshua Lee <[email protected]> Co-authored-by: Ping <[email protected]> Co-authored-by: harshit-anyscale <[email protected]> Co-authored-by: Anmol Singh <[email protected]> Co-authored-by: anmol <[email protected]> Co-authored-by: Kit Lee <[email protected]> Co-authored-by: Jason Li <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: Mengqing Cao <[email protected]> Co-authored-by: Sampan S Nayak <[email protected]> Co-authored-by: sampan <[email protected]> Co-authored-by: Sven Mika <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Nary Yeh <[email protected]> Co-authored-by: lkchen <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: Markus <[email protected]> Co-authored-by: czgdp1807 <[email protected]> Co-authored-by: akyang-anyscale <[email protected]> Co-authored-by: Alan Guo <[email protected]> Co-authored-by: Howie Tien <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: simonsays1980 <[email protected]> Co-authored-by: William Lin <[email protected]> Co-authored-by: Richard Liaw <[email protected]> Co-authored-by: Ryan O'Leary <[email protected]> Co-authored-by: Andrew Sy Kim <[email protected]> Co-authored-by: Matvei Pashkovskii <[email protected]> Co-authored-by: Kourosh Hakhamaneshi <[email protected]> Co-authored-by: Kishanthan Thangarajah <[email protected]> Co-authored-by: yi wang <[email protected]> Co-authored-by: Neil Girdhar <[email protected]> Co-authored-by: Nikhil G <[email protected]> Co-authored-by: Jun-Hao Wan <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: coqian <[email protected]> Co-authored-by: angelinalg <[email protected]> Co-authored-by: Doyoung Kim <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: Jeffrey Wang <[email protected]> Co-authored-by: jeffreyjeffreywang <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: avigyabb <[email protected]> Co-authored-by: avibasnet31 <[email protected]> Co-authored-by: tannerdwood <[email protected]> Co-authored-by: Tanner Wood <[email protected]> Co-authored-by: Ricardo Decal <[email protected]> Co-authored-by: Kamil Kaczmarek <[email protected]> Co-authored-by: Hassam Ullah Sheikh <[email protected]> Co-authored-by: MatthewCWeston <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Tianyi <[email protected]> Co-authored-by: gangsf <[email protected]> Co-authored-by: Gang Zhao <[email protected]> Co-authored-by: ahao-anyscale <[email protected]> Co-authored-by: Lehui Liu <[email protected]> Co-authored-by: Mark Rossetti <[email protected]> Co-authored-by: Jialing He <[email protected]> Co-authored-by: hejialing.hjl <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Q文举 <[email protected]> Co-authored-by: qiwenju <[email protected]> Co-authored-by: Len Strnad <[email protected]> Co-authored-by: Goku Mohandas <[email protected]>

…oject#55908) ## Why are these changes needed? Hugging Face Datasets v4.0.0 removed dataset scripts and the related “dynamic modules” mechanism (See huggingface/datasets#7592), so `datasets.load.init_dynamic_modules` no longer exists. As `ray.data._internal.datasource.huggingface_datasource` currently imports/calls this function unconditionally, it raises: ``` AttributeError: module 'datasets.load' has no attribute 'init_dynamic_modules' ``` With this PR, Ray detects the legacy initializer and only calls it on Datasets ≤3.x. On Datasets ≥4.0.0 the block becomes a no-op. The behavior for older Datasets remains unchanged. ## Related issue number NA ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Edward Oakes <[email protected]> Signed-off-by: Jiajun Yao <[email protected]> Signed-off-by: Matthew Owen <[email protected]> Signed-off-by: Sagar Sumit <[email protected]> Signed-off-by: xgui <[email protected]> Signed-off-by: Xinyuan <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Matthew Deng <[email protected]> Signed-off-by: Goutam V <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: elliot-barn <[email protected]> Signed-off-by: Yiwen Xiang <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: dragongu <[email protected]> Signed-off-by: Andrew Grosser <[email protected]> Signed-off-by: Justin Yu <[email protected]> Signed-off-by: abrar <[email protected]> Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Potato <[email protected]> Signed-off-by: Zac Policzer <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: 杨睿 <[email protected]> Signed-off-by: dayshah <[email protected]> Signed-off-by: axreldable <[email protected]> Signed-off-by: kaihsun <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Timothy Seah <[email protected]> Signed-off-by: kevin <[email protected]> Signed-off-by: joshlee <[email protected]> Signed-off-by: 400Ping <[email protected]> Signed-off-by: Dhyey Shah <[email protected]> Signed-off-by: harshit <[email protected]> Signed-off-by: anmol <[email protected]> Signed-off-by: Kit Lee <[email protected]> Signed-off-by: JasonLi1909 <[email protected]> Signed-off-by: Jason Li <[email protected]> Signed-off-by: MengqingCao <[email protected]> Signed-off-by: sampan <[email protected]> Signed-off-by: zac <[email protected]> Signed-off-by: Elliot Barnwell <[email protected]> Signed-off-by: Mengjin Yan <[email protected]> Signed-off-by: myan <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: Linkun <[email protected]> Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Markus <[email protected]> Signed-off-by: Gagandeep Singh <[email protected]> Signed-off-by: akyang-anyscale <[email protected]> Signed-off-by: Alan Guo <[email protected]> Signed-off-by: haotian <[email protected]> Signed-off-by: Howie Tien <[email protected]> Signed-off-by: will.lin <[email protected]> Signed-off-by: Richard Liaw <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Andrew Sy Kim <[email protected]> Signed-off-by: Matvei Pashkovskii <[email protected]> Signed-off-by: Kishanthan Thangarajah <[email protected]> Signed-off-by: my-vegetable-has-exploded <[email protected]> Signed-off-by: Neil Girdhar <[email protected]> Signed-off-by: Nikhil Ghosh <[email protected]> Signed-off-by: win5923 <[email protected]> Signed-off-by: Stephanie wang <[email protected]> Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: cong.qian <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: doyoung <[email protected]> Signed-off-by: Doyoung Kim <[email protected]> Signed-off-by: simonsays1980 <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Sampan S Nayak <[email protected]> Signed-off-by: vincenthhan <[email protected]> Signed-off-by: jeffreyjeffreywang <[email protected]> Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: avibasnet31 <[email protected]> Signed-off-by: Tanner Wood <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: Ricardo Decal <[email protected]> Signed-off-by: Matthew <[email protected]> Signed-off-by: tianyi-ge <[email protected]> Signed-off-by: can <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: ahao-anyscale <[email protected]> Signed-off-by: Lehui Liu <[email protected]> Signed-off-by: Mark Rossetti <[email protected]> Signed-off-by: hejialing.hjl <[email protected]> Signed-off-by: qiwenju <[email protected]> Signed-off-by: Q文举 <[email protected]> Signed-off-by: ljstrnadiii <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Lonnie Liu <[email protected]> Co-authored-by: Rueian <[email protected]> Co-authored-by: Dhyey Shah <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Co-authored-by: Jiajun Yao <[email protected]> Co-authored-by: Matthew Owen <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> Co-authored-by: Xinyuan <[email protected]> Co-authored-by: Justin Yu <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: goutamvenkat-anyscale <[email protected]> Co-authored-by: vickytsang <[email protected]> Co-authored-by: Rui Qiao <[email protected]> Co-authored-by: Elliot Barnwell <[email protected]> Co-authored-by: Yevet <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Srinath Krishnamachari <[email protected]> Co-authored-by: iamjustinhsu <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: dragongu <[email protected]> Co-authored-by: Andrew Grosser <[email protected]> Co-authored-by: Abrar Sheikh <[email protected]> Co-authored-by: kourosh hakhamaneshi <[email protected]> Co-authored-by: Yicheng-Lu-llll <[email protected]> Co-authored-by: Potato <[email protected]> Co-authored-by: Zac Policzer <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: Seiji Eicher <[email protected]> Co-authored-by: 杨睿 <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Aleksei Starikov <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Kevin H. Luu <[email protected]> Co-authored-by: Qiaolin Yu <[email protected]> Co-authored-by: Joshua Lee <[email protected]> Co-authored-by: Ping <[email protected]> Co-authored-by: harshit-anyscale <[email protected]> Co-authored-by: Anmol Singh <[email protected]> Co-authored-by: anmol <[email protected]> Co-authored-by: Kit Lee <[email protected]> Co-authored-by: Jason Li <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: Mengqing Cao <[email protected]> Co-authored-by: Sampan S Nayak <[email protected]> Co-authored-by: sampan <[email protected]> Co-authored-by: Sven Mika <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Nary Yeh <[email protected]> Co-authored-by: lkchen <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: Markus <[email protected]> Co-authored-by: czgdp1807 <[email protected]> Co-authored-by: akyang-anyscale <[email protected]> Co-authored-by: Alan Guo <[email protected]> Co-authored-by: Howie Tien <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: simonsays1980 <[email protected]> Co-authored-by: William Lin <[email protected]> Co-authored-by: Richard Liaw <[email protected]> Co-authored-by: Ryan O'Leary <[email protected]> Co-authored-by: Andrew Sy Kim <[email protected]> Co-authored-by: Matvei Pashkovskii <[email protected]> Co-authored-by: Kourosh Hakhamaneshi <[email protected]> Co-authored-by: Kishanthan Thangarajah <[email protected]> Co-authored-by: yi wang <[email protected]> Co-authored-by: Neil Girdhar <[email protected]> Co-authored-by: Nikhil G <[email protected]> Co-authored-by: Jun-Hao Wan <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: coqian <[email protected]> Co-authored-by: angelinalg <[email protected]> Co-authored-by: Doyoung Kim <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: Jeffrey Wang <[email protected]> Co-authored-by: jeffreyjeffreywang <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: avigyabb <[email protected]> Co-authored-by: avibasnet31 <[email protected]> Co-authored-by: tannerdwood <[email protected]> Co-authored-by: Tanner Wood <[email protected]> Co-authored-by: Ricardo Decal <[email protected]> Co-authored-by: Kamil Kaczmarek <[email protected]> Co-authored-by: Hassam Ullah Sheikh <[email protected]> Co-authored-by: MatthewCWeston <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Tianyi <[email protected]> Co-authored-by: gangsf <[email protected]> Co-authored-by: Gang Zhao <[email protected]> Co-authored-by: ahao-anyscale <[email protected]> Co-authored-by: Lehui Liu <[email protected]> Co-authored-by: Mark Rossetti <[email protected]> Co-authored-by: Jialing He <[email protected]> Co-authored-by: hejialing.hjl <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Q文举 <[email protected]> Co-authored-by: qiwenju <[email protected]> Co-authored-by: Len Strnad <[email protected]> Co-authored-by: Goku Mohandas <[email protected]> Signed-off-by: zac <[email protected]>

…oject#55908) ## Why are these changes needed? Hugging Face Datasets v4.0.0 removed dataset scripts and the related “dynamic modules” mechanism (See huggingface/datasets#7592), so `datasets.load.init_dynamic_modules` no longer exists. As `ray.data._internal.datasource.huggingface_datasource` currently imports/calls this function unconditionally, it raises: ``` AttributeError: module 'datasets.load' has no attribute 'init_dynamic_modules' ``` With this PR, Ray detects the legacy initializer and only calls it on Datasets ≤3.x. On Datasets ≥4.0.0 the block becomes a no-op. The behavior for older Datasets remains unchanged. ## Related issue number NA ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Edward Oakes <[email protected]> Signed-off-by: Jiajun Yao <[email protected]> Signed-off-by: Matthew Owen <[email protected]> Signed-off-by: Sagar Sumit <[email protected]> Signed-off-by: xgui <[email protected]> Signed-off-by: Xinyuan <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Matthew Deng <[email protected]> Signed-off-by: Goutam V <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: elliot-barn <[email protected]> Signed-off-by: Yiwen Xiang <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: dragongu <[email protected]> Signed-off-by: Andrew Grosser <[email protected]> Signed-off-by: Justin Yu <[email protected]> Signed-off-by: abrar <[email protected]> Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Potato <[email protected]> Signed-off-by: Zac Policzer <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: 杨睿 <[email protected]> Signed-off-by: dayshah <[email protected]> Signed-off-by: axreldable <[email protected]> Signed-off-by: kaihsun <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Timothy Seah <[email protected]> Signed-off-by: kevin <[email protected]> Signed-off-by: joshlee <[email protected]> Signed-off-by: 400Ping <[email protected]> Signed-off-by: Dhyey Shah <[email protected]> Signed-off-by: harshit <[email protected]> Signed-off-by: anmol <[email protected]> Signed-off-by: Kit Lee <[email protected]> Signed-off-by: JasonLi1909 <[email protected]> Signed-off-by: Jason Li <[email protected]> Signed-off-by: MengqingCao <[email protected]> Signed-off-by: sampan <[email protected]> Signed-off-by: zac <[email protected]> Signed-off-by: Elliot Barnwell <[email protected]> Signed-off-by: Mengjin Yan <[email protected]> Signed-off-by: myan <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: Linkun <[email protected]> Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Markus <[email protected]> Signed-off-by: Gagandeep Singh <[email protected]> Signed-off-by: akyang-anyscale <[email protected]> Signed-off-by: Alan Guo <[email protected]> Signed-off-by: haotian <[email protected]> Signed-off-by: Howie Tien <[email protected]> Signed-off-by: will.lin <[email protected]> Signed-off-by: Richard Liaw <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Andrew Sy Kim <[email protected]> Signed-off-by: Matvei Pashkovskii <[email protected]> Signed-off-by: Kishanthan Thangarajah <[email protected]> Signed-off-by: my-vegetable-has-exploded <[email protected]> Signed-off-by: Neil Girdhar <[email protected]> Signed-off-by: Nikhil Ghosh <[email protected]> Signed-off-by: win5923 <[email protected]> Signed-off-by: Stephanie wang <[email protected]> Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: cong.qian <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: doyoung <[email protected]> Signed-off-by: Doyoung Kim <[email protected]> Signed-off-by: simonsays1980 <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Sampan S Nayak <[email protected]> Signed-off-by: vincenthhan <[email protected]> Signed-off-by: jeffreyjeffreywang <[email protected]> Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: avibasnet31 <[email protected]> Signed-off-by: Tanner Wood <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: Ricardo Decal <[email protected]> Signed-off-by: Matthew <[email protected]> Signed-off-by: tianyi-ge <[email protected]> Signed-off-by: can <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: ahao-anyscale <[email protected]> Signed-off-by: Lehui Liu <[email protected]> Signed-off-by: Mark Rossetti <[email protected]> Signed-off-by: hejialing.hjl <[email protected]> Signed-off-by: qiwenju <[email protected]> Signed-off-by: Q文举 <[email protected]> Signed-off-by: ljstrnadiii <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Lonnie Liu <[email protected]> Co-authored-by: Rueian <[email protected]> Co-authored-by: Dhyey Shah <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Co-authored-by: Jiajun Yao <[email protected]> Co-authored-by: Matthew Owen <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> Co-authored-by: Xinyuan <[email protected]> Co-authored-by: Justin Yu <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: goutamvenkat-anyscale <[email protected]> Co-authored-by: vickytsang <[email protected]> Co-authored-by: Rui Qiao <[email protected]> Co-authored-by: Elliot Barnwell <[email protected]> Co-authored-by: Yevet <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Srinath Krishnamachari <[email protected]> Co-authored-by: iamjustinhsu <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: dragongu <[email protected]> Co-authored-by: Andrew Grosser <[email protected]> Co-authored-by: Abrar Sheikh <[email protected]> Co-authored-by: kourosh hakhamaneshi <[email protected]> Co-authored-by: Yicheng-Lu-llll <[email protected]> Co-authored-by: Potato <[email protected]> Co-authored-by: Zac Policzer <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: Seiji Eicher <[email protected]> Co-authored-by: 杨睿 <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Aleksei Starikov <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Kevin H. Luu <[email protected]> Co-authored-by: Qiaolin Yu <[email protected]> Co-authored-by: Joshua Lee <[email protected]> Co-authored-by: Ping <[email protected]> Co-authored-by: harshit-anyscale <[email protected]> Co-authored-by: Anmol Singh <[email protected]> Co-authored-by: anmol <[email protected]> Co-authored-by: Kit Lee <[email protected]> Co-authored-by: Jason Li <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: Mengqing Cao <[email protected]> Co-authored-by: Sampan S Nayak <[email protected]> Co-authored-by: sampan <[email protected]> Co-authored-by: Sven Mika <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Nary Yeh <[email protected]> Co-authored-by: lkchen <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: Markus <[email protected]> Co-authored-by: czgdp1807 <[email protected]> Co-authored-by: akyang-anyscale <[email protected]> Co-authored-by: Alan Guo <[email protected]> Co-authored-by: Howie Tien <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: simonsays1980 <[email protected]> Co-authored-by: William Lin <[email protected]> Co-authored-by: Richard Liaw <[email protected]> Co-authored-by: Ryan O'Leary <[email protected]> Co-authored-by: Andrew Sy Kim <[email protected]> Co-authored-by: Matvei Pashkovskii <[email protected]> Co-authored-by: Kourosh Hakhamaneshi <[email protected]> Co-authored-by: Kishanthan Thangarajah <[email protected]> Co-authored-by: yi wang <[email protected]> Co-authored-by: Neil Girdhar <[email protected]> Co-authored-by: Nikhil G <[email protected]> Co-authored-by: Jun-Hao Wan <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: coqian <[email protected]> Co-authored-by: angelinalg <[email protected]> Co-authored-by: Doyoung Kim <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: Jeffrey Wang <[email protected]> Co-authored-by: jeffreyjeffreywang <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: avigyabb <[email protected]> Co-authored-by: avibasnet31 <[email protected]> Co-authored-by: tannerdwood <[email protected]> Co-authored-by: Tanner Wood <[email protected]> Co-authored-by: Ricardo Decal <[email protected]> Co-authored-by: Kamil Kaczmarek <[email protected]> Co-authored-by: Hassam Ullah Sheikh <[email protected]> Co-authored-by: MatthewCWeston <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Tianyi <[email protected]> Co-authored-by: gangsf <[email protected]> Co-authored-by: Gang Zhao <[email protected]> Co-authored-by: ahao-anyscale <[email protected]> Co-authored-by: Lehui Liu <[email protected]> Co-authored-by: Mark Rossetti <[email protected]> Co-authored-by: Jialing He <[email protected]> Co-authored-by: hejialing.hjl <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Q文举 <[email protected]> Co-authored-by: qiwenju <[email protected]> Co-authored-by: Len Strnad <[email protected]> Co-authored-by: Goku Mohandas <[email protected]> Signed-off-by: Marco Stephan <[email protected]>

## Why are these changes needed? Hugging Face Datasets v4.0.0 removed dataset scripts and the related “dynamic modules” mechanism (See huggingface/datasets#7592), so `datasets.load.init_dynamic_modules` no longer exists. As `ray.data._internal.datasource.huggingface_datasource` currently imports/calls this function unconditionally, it raises: ``` AttributeError: module 'datasets.load' has no attribute 'init_dynamic_modules' ``` With this PR, Ray detects the legacy initializer and only calls it on Datasets ≤3.x. On Datasets ≥4.0.0 the block becomes a no-op. The behavior for older Datasets remains unchanged. ## Related issue number NA ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Edward Oakes <[email protected]> Signed-off-by: Jiajun Yao <[email protected]> Signed-off-by: Matthew Owen <[email protected]> Signed-off-by: Sagar Sumit <[email protected]> Signed-off-by: xgui <[email protected]> Signed-off-by: Xinyuan <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Matthew Deng <[email protected]> Signed-off-by: Goutam V <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: elliot-barn <[email protected]> Signed-off-by: Yiwen Xiang <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: dragongu <[email protected]> Signed-off-by: Andrew Grosser <[email protected]> Signed-off-by: Justin Yu <[email protected]> Signed-off-by: abrar <[email protected]> Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Potato <[email protected]> Signed-off-by: Zac Policzer <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: 杨睿 <[email protected]> Signed-off-by: dayshah <[email protected]> Signed-off-by: axreldable <[email protected]> Signed-off-by: kaihsun <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Timothy Seah <[email protected]> Signed-off-by: kevin <[email protected]> Signed-off-by: joshlee <[email protected]> Signed-off-by: 400Ping <[email protected]> Signed-off-by: Dhyey Shah <[email protected]> Signed-off-by: harshit <[email protected]> Signed-off-by: anmol <[email protected]> Signed-off-by: Kit Lee <[email protected]> Signed-off-by: JasonLi1909 <[email protected]> Signed-off-by: Jason Li <[email protected]> Signed-off-by: MengqingCao <[email protected]> Signed-off-by: sampan <[email protected]> Signed-off-by: zac <[email protected]> Signed-off-by: Elliot Barnwell <[email protected]> Signed-off-by: Mengjin Yan <[email protected]> Signed-off-by: myan <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: Linkun <[email protected]> Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Markus <[email protected]> Signed-off-by: Gagandeep Singh <[email protected]> Signed-off-by: akyang-anyscale <[email protected]> Signed-off-by: Alan Guo <[email protected]> Signed-off-by: haotian <[email protected]> Signed-off-by: Howie Tien <[email protected]> Signed-off-by: will.lin <[email protected]> Signed-off-by: Richard Liaw <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Andrew Sy Kim <[email protected]> Signed-off-by: Matvei Pashkovskii <[email protected]> Signed-off-by: Kishanthan Thangarajah <[email protected]> Signed-off-by: my-vegetable-has-exploded <[email protected]> Signed-off-by: Neil Girdhar <[email protected]> Signed-off-by: Nikhil Ghosh <[email protected]> Signed-off-by: win5923 <[email protected]> Signed-off-by: Stephanie wang <[email protected]> Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: cong.qian <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: doyoung <[email protected]> Signed-off-by: Doyoung Kim <[email protected]> Signed-off-by: simonsays1980 <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Sampan S Nayak <[email protected]> Signed-off-by: vincenthhan <[email protected]> Signed-off-by: jeffreyjeffreywang <[email protected]> Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: avibasnet31 <[email protected]> Signed-off-by: Tanner Wood <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: Ricardo Decal <[email protected]> Signed-off-by: Matthew <[email protected]> Signed-off-by: tianyi-ge <[email protected]> Signed-off-by: can <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: ahao-anyscale <[email protected]> Signed-off-by: Lehui Liu <[email protected]> Signed-off-by: Mark Rossetti <[email protected]> Signed-off-by: hejialing.hjl <[email protected]> Signed-off-by: qiwenju <[email protected]> Signed-off-by: Q文举 <[email protected]> Signed-off-by: ljstrnadiii <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Lonnie Liu <[email protected]> Co-authored-by: Rueian <[email protected]> Co-authored-by: Dhyey Shah <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Co-authored-by: Jiajun Yao <[email protected]> Co-authored-by: Matthew Owen <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> Co-authored-by: Xinyuan <[email protected]> Co-authored-by: Justin Yu <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: goutamvenkat-anyscale <[email protected]> Co-authored-by: vickytsang <[email protected]> Co-authored-by: Rui Qiao <[email protected]> Co-authored-by: Elliot Barnwell <[email protected]> Co-authored-by: Yevet <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Srinath Krishnamachari <[email protected]> Co-authored-by: iamjustinhsu <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: dragongu <[email protected]> Co-authored-by: Andrew Grosser <[email protected]> Co-authored-by: Abrar Sheikh <[email protected]> Co-authored-by: kourosh hakhamaneshi <[email protected]> Co-authored-by: Yicheng-Lu-llll <[email protected]> Co-authored-by: Potato <[email protected]> Co-authored-by: Zac Policzer <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: Seiji Eicher <[email protected]> Co-authored-by: 杨睿 <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Aleksei Starikov <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Kevin H. Luu <[email protected]> Co-authored-by: Qiaolin Yu <[email protected]> Co-authored-by: Joshua Lee <[email protected]> Co-authored-by: Ping <[email protected]> Co-authored-by: harshit-anyscale <[email protected]> Co-authored-by: Anmol Singh <[email protected]> Co-authored-by: anmol <[email protected]> Co-authored-by: Kit Lee <[email protected]> Co-authored-by: Jason Li <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: Mengqing Cao <[email protected]> Co-authored-by: Sampan S Nayak <[email protected]> Co-authored-by: sampan <[email protected]> Co-authored-by: Sven Mika <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Nary Yeh <[email protected]> Co-authored-by: lkchen <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: Markus <[email protected]> Co-authored-by: czgdp1807 <[email protected]> Co-authored-by: akyang-anyscale <[email protected]> Co-authored-by: Alan Guo <[email protected]> Co-authored-by: Howie Tien <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: simonsays1980 <[email protected]> Co-authored-by: William Lin <[email protected]> Co-authored-by: Richard Liaw <[email protected]> Co-authored-by: Ryan O'Leary <[email protected]> Co-authored-by: Andrew Sy Kim <[email protected]> Co-authored-by: Matvei Pashkovskii <[email protected]> Co-authored-by: Kourosh Hakhamaneshi <[email protected]> Co-authored-by: Kishanthan Thangarajah <[email protected]> Co-authored-by: yi wang <[email protected]> Co-authored-by: Neil Girdhar <[email protected]> Co-authored-by: Nikhil G <[email protected]> Co-authored-by: Jun-Hao Wan <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: coqian <[email protected]> Co-authored-by: angelinalg <[email protected]> Co-authored-by: Doyoung Kim <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: Jeffrey Wang <[email protected]> Co-authored-by: jeffreyjeffreywang <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: avigyabb <[email protected]> Co-authored-by: avibasnet31 <[email protected]> Co-authored-by: tannerdwood <[email protected]> Co-authored-by: Tanner Wood <[email protected]> Co-authored-by: Ricardo Decal <[email protected]> Co-authored-by: Kamil Kaczmarek <[email protected]> Co-authored-by: Hassam Ullah Sheikh <[email protected]> Co-authored-by: MatthewCWeston <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Tianyi <[email protected]> Co-authored-by: gangsf <[email protected]> Co-authored-by: Gang Zhao <[email protected]> Co-authored-by: ahao-anyscale <[email protected]> Co-authored-by: Lehui Liu <[email protected]> Co-authored-by: Mark Rossetti <[email protected]> Co-authored-by: Jialing He <[email protected]> Co-authored-by: hejialing.hjl <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Q文举 <[email protected]> Co-authored-by: qiwenju <[email protected]> Co-authored-by: Len Strnad <[email protected]> Co-authored-by: Goku Mohandas <[email protected]> Signed-off-by: Douglas Strodtman <[email protected]>

…oject#55908) ## Why are these changes needed? Hugging Face Datasets v4.0.0 removed dataset scripts and the related “dynamic modules” mechanism (See huggingface/datasets#7592), so `datasets.load.init_dynamic_modules` no longer exists. As `ray.data._internal.datasource.huggingface_datasource` currently imports/calls this function unconditionally, it raises: ``` AttributeError: module 'datasets.load' has no attribute 'init_dynamic_modules' ``` With this PR, Ray detects the legacy initializer and only calls it on Datasets ≤3.x. On Datasets ≥4.0.0 the block becomes a no-op. The behavior for older Datasets remains unchanged. ## Related issue number NA ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: Edward Oakes <[email protected]> Signed-off-by: Jiajun Yao <[email protected]> Signed-off-by: Matthew Owen <[email protected]> Signed-off-by: Sagar Sumit <[email protected]> Signed-off-by: xgui <[email protected]> Signed-off-by: Xinyuan <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Matthew Deng <[email protected]> Signed-off-by: Goutam V <[email protected]> Signed-off-by: root <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: elliot-barn <[email protected]> Signed-off-by: Yiwen Xiang <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Mao Yancan <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: iamjustinhsu <[email protected]> Signed-off-by: dragongu <[email protected]> Signed-off-by: Andrew Grosser <[email protected]> Signed-off-by: Justin Yu <[email protected]> Signed-off-by: abrar <[email protected]> Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Potato <[email protected]> Signed-off-by: Zac Policzer <[email protected]> Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: 杨睿 <[email protected]> Signed-off-by: dayshah <[email protected]> Signed-off-by: axreldable <[email protected]> Signed-off-by: kaihsun <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Timothy Seah <[email protected]> Signed-off-by: kevin <[email protected]> Signed-off-by: joshlee <[email protected]> Signed-off-by: 400Ping <[email protected]> Signed-off-by: Dhyey Shah <[email protected]> Signed-off-by: harshit <[email protected]> Signed-off-by: anmol <[email protected]> Signed-off-by: Kit Lee <[email protected]> Signed-off-by: JasonLi1909 <[email protected]> Signed-off-by: Jason Li <[email protected]> Signed-off-by: MengqingCao <[email protected]> Signed-off-by: sampan <[email protected]> Signed-off-by: zac <[email protected]> Signed-off-by: Elliot Barnwell <[email protected]> Signed-off-by: Mengjin Yan <[email protected]> Signed-off-by: myan <[email protected]> Signed-off-by: Seiji Eicher <[email protected]> Signed-off-by: Linkun <[email protected]> Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Markus <[email protected]> Signed-off-by: Gagandeep Singh <[email protected]> Signed-off-by: akyang-anyscale <[email protected]> Signed-off-by: Alan Guo <[email protected]> Signed-off-by: haotian <[email protected]> Signed-off-by: Howie Tien <[email protected]> Signed-off-by: will.lin <[email protected]> Signed-off-by: Richard Liaw <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Andrew Sy Kim <[email protected]> Signed-off-by: Matvei Pashkovskii <[email protected]> Signed-off-by: Kishanthan Thangarajah <[email protected]> Signed-off-by: my-vegetable-has-exploded <[email protected]> Signed-off-by: Neil Girdhar <[email protected]> Signed-off-by: Nikhil Ghosh <[email protected]> Signed-off-by: win5923 <[email protected]> Signed-off-by: Stephanie wang <[email protected]> Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: cong.qian <[email protected]> Signed-off-by: Lonnie Liu <[email protected]> Signed-off-by: doyoung <[email protected]> Signed-off-by: Doyoung Kim <[email protected]> Signed-off-by: simonsays1980 <[email protected]> Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Sampan S Nayak <[email protected]> Signed-off-by: vincenthhan <[email protected]> Signed-off-by: jeffreyjeffreywang <[email protected]> Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: avibasnet31 <[email protected]> Signed-off-by: Tanner Wood <[email protected]> Signed-off-by: avigyabb <[email protected]> Signed-off-by: Ricardo Decal <[email protected]> Signed-off-by: Matthew <[email protected]> Signed-off-by: tianyi-ge <[email protected]> Signed-off-by: can <[email protected]> Signed-off-by: Alexey Kudinkin <[email protected]> Signed-off-by: ahao-anyscale <[email protected]> Signed-off-by: Lehui Liu <[email protected]> Signed-off-by: Mark Rossetti <[email protected]> Signed-off-by: hejialing.hjl <[email protected]> Signed-off-by: qiwenju <[email protected]> Signed-off-by: Q文举 <[email protected]> Signed-off-by: ljstrnadiii <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Lonnie Liu <[email protected]> Co-authored-by: Rueian <[email protected]> Co-authored-by: Dhyey Shah <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Co-authored-by: Jiajun Yao <[email protected]> Co-authored-by: Matthew Owen <[email protected]> Co-authored-by: Sagar Sumit <[email protected]> Co-authored-by: Xinyuan <[email protected]> Co-authored-by: Justin Yu <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: goutamvenkat-anyscale <[email protected]> Co-authored-by: vickytsang <[email protected]> Co-authored-by: Rui Qiao <[email protected]> Co-authored-by: Elliot Barnwell <[email protected]> Co-authored-by: Yevet <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Mao Yancan <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Srinath Krishnamachari <[email protected]> Co-authored-by: iamjustinhsu <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: dragongu <[email protected]> Co-authored-by: Andrew Grosser <[email protected]> Co-authored-by: Abrar Sheikh <[email protected]> Co-authored-by: kourosh hakhamaneshi <[email protected]> Co-authored-by: Yicheng-Lu-llll <[email protected]> Co-authored-by: Potato <[email protected]> Co-authored-by: Zac Policzer <[email protected]> Co-authored-by: Alexey Kudinkin <[email protected]> Co-authored-by: Seiji Eicher <[email protected]> Co-authored-by: 杨睿 <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Aleksei Starikov <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Timothy Seah <[email protected]> Co-authored-by: Kevin H. Luu <[email protected]> Co-authored-by: Qiaolin Yu <[email protected]> Co-authored-by: Joshua Lee <[email protected]> Co-authored-by: Ping <[email protected]> Co-authored-by: harshit-anyscale <[email protected]> Co-authored-by: Anmol Singh <[email protected]> Co-authored-by: anmol <[email protected]> Co-authored-by: Kit Lee <[email protected]> Co-authored-by: Jason Li <[email protected]> Co-authored-by: matthewdeng <[email protected]> Co-authored-by: Mengqing Cao <[email protected]> Co-authored-by: Sampan S Nayak <[email protected]> Co-authored-by: sampan <[email protected]> Co-authored-by: Sven Mika <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Nary Yeh <[email protected]> Co-authored-by: lkchen <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: Markus <[email protected]> Co-authored-by: czgdp1807 <[email protected]> Co-authored-by: akyang-anyscale <[email protected]> Co-authored-by: Alan Guo <[email protected]> Co-authored-by: Howie Tien <[email protected]> Co-authored-by: Balaji Veeramani <[email protected]> Co-authored-by: simonsays1980 <[email protected]> Co-authored-by: William Lin <[email protected]> Co-authored-by: Richard Liaw <[email protected]> Co-authored-by: Ryan O'Leary <[email protected]> Co-authored-by: Andrew Sy Kim <[email protected]> Co-authored-by: Matvei Pashkovskii <[email protected]> Co-authored-by: Kourosh Hakhamaneshi <[email protected]> Co-authored-by: Kishanthan Thangarajah <[email protected]> Co-authored-by: yi wang <[email protected]> Co-authored-by: Neil Girdhar <[email protected]> Co-authored-by: Nikhil G <[email protected]> Co-authored-by: Jun-Hao Wan <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Co-authored-by: coqian <[email protected]> Co-authored-by: angelinalg <[email protected]> Co-authored-by: Doyoung Kim <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: vincenthhan <[email protected]> Co-authored-by: Jeffrey Wang <[email protected]> Co-authored-by: jeffreyjeffreywang <[email protected]> Co-authored-by: Ibrahim Rabbani <[email protected]> Co-authored-by: avigyabb <[email protected]> Co-authored-by: avibasnet31 <[email protected]> Co-authored-by: tannerdwood <[email protected]> Co-authored-by: Tanner Wood <[email protected]> Co-authored-by: Ricardo Decal <[email protected]> Co-authored-by: Kamil Kaczmarek <[email protected]> Co-authored-by: Hassam Ullah Sheikh <[email protected]> Co-authored-by: MatthewCWeston <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Tianyi <[email protected]> Co-authored-by: gangsf <[email protected]> Co-authored-by: Gang Zhao <[email protected]> Co-authored-by: ahao-anyscale <[email protected]> Co-authored-by: Lehui Liu <[email protected]> Co-authored-by: Mark Rossetti <[email protected]> Co-authored-by: Jialing He <[email protected]> Co-authored-by: hejialing.hjl <[email protected]> Co-authored-by: Cuong Nguyen <[email protected]> Co-authored-by: Q文举 <[email protected]> Co-authored-by: qiwenju <[email protected]> Co-authored-by: Len Strnad <[email protected]> Co-authored-by: Goku Mohandas <[email protected]>

remove scripts altogether

e1d7055

lhoestq and others added 9 commits June 5, 2025 13:49

fix tests

c13a5f3

remove dynamic_modules_path

86b1079

remove hf_modules_cache

2ca9279

remove unused test

02b0c93

fix auth in decoding

c7c3e07

fix tests

d475ec8

Merge branch 'main' into remove-scripts-altogether

96b4b80

Merge branch 'main' into remove-scripts-altogether

4b0927a

minor

adbb492

lhoestq marked this pull request as ready for review June 9, 2025 16:45

lhoestq merged commit 784607d into main Jun 9, 2025
9 of 15 checks passed

lhoestq deleted the remove-scripts-altogether branch June 9, 2025 16:45

qnzhou mentioned this pull request Jul 18, 2025

Python package not working Thingi10K/Thingi10K#2

Closed

This was referenced Jul 18, 2025

Auto-merge option for convert-to-parquet #7527

Closed

Add --merge-pull-request option for convert_to_parquet #7556

Closed

Add documentation for PGLearn AI4OPT/ML4OPF#34

Open

ryokan0123 mentioned this pull request Aug 17, 2025

datasets のバージョンを 4.0.0 未満に固定 ghmagazine/llm-book#58

Merged

ItsTania mentioned this pull request Aug 17, 2025

Remove dataset scripts for HF datasets. UKGovernmentBEIS/inspect_evals#469

Closed

tohtana mentioned this pull request Aug 25, 2025

[data] skip loading dynamic modules for HF datasets >= v4.0.0 ray-project/ray#55908

Merged

8 tasks

khannurien mentioned this pull request Aug 26, 2025

Fix benchmarks for Datasets 4.0.0 confident-ai/deepeval#1977

Merged

ArneBinder mentioned this pull request Oct 1, 2025

update datasets to <4.0.0 ArneBinder/pie-datasets#212

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Remove scripts altogether #7592

Remove scripts altogether #7592

lhoestq commented Jun 4, 2025 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Jun 4, 2025

Uh oh!

Uh oh!

jp1924 commented Jul 14, 2025 •

edited

Loading

Uh oh!

lhoestq commented Jul 15, 2025

Uh oh!

jp1924 commented Jul 16, 2025

Uh oh!

autosquid commented Aug 4, 2025

Uh oh!

autosquid commented Aug 4, 2025 •

edited

Loading

Uh oh!

lhoestq commented Sep 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Remove scripts altogether #7592

Remove scripts altogether #7592

Conversation

lhoestq commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jun 4, 2025

Uh oh!

Uh oh!

jp1924 commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lhoestq commented Jul 15, 2025

Uh oh!

jp1924 commented Jul 16, 2025

Uh oh!

autosquid commented Aug 4, 2025

Uh oh!

autosquid commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lhoestq commented Sep 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lhoestq commented Jun 4, 2025 •

edited

Loading

jp1924 commented Jul 14, 2025 •

edited

Loading

autosquid commented Aug 4, 2025 •

edited

Loading