feat(services/onedrive): implement additional OneDrive features #5784
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Xuanwo
reviewed
Mar 17, 2025
Comment on lines
76
to
85
| if let Some(version) = args.version() { | ||
| let versions = self.core.onedrive_list_versions(path).await?; | ||
| for item_version in versions { | ||
| if item_version.id == version { | ||
| meta.set_version(version); | ||
| return Ok(RpStat::new(meta)); | ||
| } | ||
| } | ||
|
|
||
| return Err(Error::new(ErrorKind::NotFound, "item version not found")); |
Member
There was a problem hiding this comment.
This is a big sad regarding this API.
I can imagine that list_with(path).versions(true) and stat_with(item).versions(abc) could be very slow.
I noticed that we're using id as the version ID. How about supporting onedrive_stat_by_id?
OneDrive supports two call conventions:
GET /me/drive/items/{item-id}
GET /me/drive/root:/{item-path}If we already have the id, we can use it to make a direct call.
Member
Author
There was a problem hiding this comment.
The item ID is not the same as the version ID, unfortunately. Using versions is going to generate lots of API calls indeed. I agree with you this implementation could be better. I'll work on the get version API. Will also look into if I can take advantage of the item ID.
Member
There was a problem hiding this comment.
The item ID is not the same as the version ID
Sorry I didn't take deep look over the onedrive API. The id is not unique across the entire onedrive? Or onedrive has other concept that similiar to version_id?
Member
Author
There was a problem hiding this comment.
No worries, I'll explain. The id for a file is unique.
I noticed that we're using id as the version ID.
The version ID belongs to a file.
GET /me/drive/items/{item-id}/versions
GET /me/drive/root:/{item-path}/versions
An example response is:
{
"value": [
{
"id": "2.0",
"lastModifiedDateTime": "2025-03-16T17:02:49Z",
"size": 74758
},
{
"id": "1.0",
"lastModifiedDateTime": "2025-03-12T21:59:54Z",
"size": 74756
}
]
}This is a sub-resource for a file. I didn't find out until I tried
https://graph.microsoft.com/v1.0/me/drive/root:/test.json?$expand=versions. This is a stat_with_version. I am going to work on versions support based on this API parameter.
Listing items with versions is not supported. GET https://graph.microsoft.com/v1.0/me/drive/root/children?$expand=versions returns 400 and a message saying "Operation not supported".
In summary:
list_with(path).versions(true): no API support except N+1 HTTP callsstat_with(item).versions(abc): one API call
Member
There was a problem hiding this comment.
The version ID belongs to a file.
Ok, I got it. So version ID can't be used as item-id.
list_with(path).versions(true): no API support except N+1 HTTP calls
Seems it's the best we can do by given APIs.
stat_with(item).versions(abc): one API call
Happy to know.
Member
Author
There was a problem hiding this comment.
Indeed it's a bit disappointing. Maybe we will find the list with version support from OneDrive a few years later.
| // Read more at https://support.microsoft.com/en-us/office/restrictions-and-limitations-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa#individualfilesize | ||
| // However, we can't enable this, otherwise OpenDAL behavior tests will try to test creating huge | ||
| // file up to this size. | ||
| // write_total_max_size: Some(250 * 1024 * 1024 * 1024), |
Member
There was a problem hiding this comment.
Would you like to submit an issue for this? I remember that S3 also set this.
| Ok(( | ||
| RpWrite::default(), | ||
| oio::OneShotWriter::new(OneDriveWriter::new(self.core.clone(), args, path)), | ||
| oio::OneShotWriter::new(OneDriveWriter::new( |
Member
There was a problem hiding this comment.
Since we already support chunked writing, we could implement something like MultipartUploadWrite instead of OneShotWrite, which only allows a single write.
Member
Author
There was a problem hiding this comment.
Ok, a good time to work on this.
Member
There was a problem hiding this comment.
This PR is large enough, it's fine to split in another PR.
Member
Author
There was a problem hiding this comment.
Ok. I will fix your other comments this week and ping you. I'm tied up at the moment.
|
|
||
| let status = response.status(); | ||
| match status { | ||
| match response.status() { |
Member
There was a problem hiding this comment.
We can implement read_with(path).version(id) in the same way like stat.
Member
Author
|
All fixed. Please have a look. I must add a new commit to fix "copy"'s behavior. Not sure what OneDrive has updated. I will rebase once you approve the PR. OPENDAL_TEST=onedrive cargo test behavior --features tests,services-onedrive -- --show-output |
Member
|
Hi, thank you @erickguan for working on this. This PR is almost good to go, would you like to resolve conflicts? |
Also: - Add documentation about consistency - Allow retry on 409 Conflict (for create_dir) Technically, we can allow retries on more operations. We wait bug reports or feature requests about this nuance since the behavior tests is likely the most heavily operator among OpenDAL OneDrive users.
4a17e32 to
1e9bcd5
Compare
Also remove a few data fields from models that the OneDrive service doesn't use.
1e9bcd5 to
1c393b8
Compare
Member
Author
|
@Xuanwo rebased. I don't know what happens to the huggingface test. Test details |
Xuanwo
approved these changes
Mar 25, 2025
Member
Xuanwo
left a comment
There was a problem hiding this comment.
Thank you @erickguan for working on this!
Member
A bit flaky 😆 |
Member
Author
|
No worries.
Alright! |
37 tasks
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #5557.
Part of #2611.
Also, add a few features:
Rationale for this change
Mostly for the issues and to implement:
What changes are included in this PR?
I implemented features that I found easy to get to. Also, this PR addresses a few nuances about OneDrive API, including:
Content-TypeAre there any user-facing changes?
Added a new option and doc.
Review
Sorry about making many changes across the OneDrive service, which makes this PR difficult to review. I made some efforts in structuring the change in a few commits. Reviewing this PR commit by commit is easier.
Also kudos to Microsoft's services I guess - they didn't ban me. I saw the behavior tests generated a huge load for their servers (50x from OneDrive).
OPENDAL_TEST=onedrive cargo test behavior --features tests,services-onedrive -- --show-output