support send large file to workload #532
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jschwinger233
requested changes
Jan 7, 2022
jschwinger233
requested changes
Jan 19, 2022
cluster/cluster.go
Outdated
| // file methods | ||
| Copy(ctx context.Context, opts *types.CopyOptions) (chan *types.CopyMessage, error) | ||
| Send(ctx context.Context, opts *types.SendOptions) (chan *types.SendMessage, error) | ||
| SendLargeFile(ctx context.Context, opts chan *types.SendLargeFileOptions, resp chan *types.SendMessage) error |
Member
There was a problem hiding this comment.
其实不用加新的 interface, 直接改旧的 Send 就可以了.
Member
|
要是有问题可以开个 zoom 来讨论 |
Contributor
Author
|
我改了下代码，现在看起来是能够跑了。但是会有些比较诡异的情况，时而成功时而失败，让我再debug一下。然后关于上面的review意见我也再改一下。 |
jschwinger233
requested changes
Feb 17, 2022
Member
jschwinger233
left a comment
There was a problem hiding this comment.
自己多测试一下, 包括老的 Send 接口是否还能用, 还有发送一些 >3G 的文件, 都是我们真实存在的业务场景.
cluster/calcium/sendlarge.go
Outdated
| utils.SentryGo(func(ID, name string, size int64, content io.Reader, uid, gid int, mode int64) func() { | ||
| return func() { | ||
| defer wg.Done() | ||
| if err := c.withWorkloadLocked(ctx, ID, func(ctx context.Context, workload *types.Workload) error { |
Member
There was a problem hiding this comment.
每个 chunk 都上一次锁, 虽然临界区是小了, 但是代价可能有点大. 考虑下从头到尾只上一把大锁, 传个大文件(>1G)对比一下耗时.
| VirtualizationCreate(ctx context.Context, opts *enginetypes.VirtualizationCreateOptions) (*enginetypes.VirtualizationCreated, error) | ||
| VirtualizationResourceRemap(context.Context, *enginetypes.VirtualizationRemapOptions) (<-chan enginetypes.VirtualizationRemapMessage, error) | ||
| VirtualizationCopyTo(ctx context.Context, ID, target string, content []byte, uid, gid int, mode int64) error | ||
| VirtualizationCopyChunkTo(ctx context.Context, ID, target string, size int64, content io.Reader, uid, gid int, mode int64) error |
Member
There was a problem hiding this comment.
CopyTo 和 CopyChunkTo 可以合并吧, 让 cluster.Copy 和 cluster.CopyLarge 都调用一个 engine 接口
Member
There was a problem hiding this comment.
刚才简单讨论了一下, rpc 层还是两个 API, 但是 cluster 层和 engine 都只有一个 send 接口.
jschwinger233
requested changes
Mar 7, 2022
06977aa to
837bec8
Compare
0c06f3a to
34dacc4
Compare
Contributor
|
unbelievable |
a982876 to
924a9a8
Compare
CMGS
reviewed
Jul 24, 2023
CMGS
reviewed
Jul 24, 2023
Contributor
|
resolve conflicts. |
CMGS
reviewed
Aug 7, 2023
CMGS
reviewed
Aug 7, 2023
CMGS
reviewed
Aug 7, 2023
CMGS
reviewed
Aug 18, 2023
CMGS
reviewed
Aug 18, 2023
CMGS
reviewed
Aug 18, 2023
CMGS
reviewed
Aug 18, 2023
Contributor
|
另外注意一下单元测试覆盖率和实际测试下多路文件并发 send |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
关于cli的send命令，之前的
send是直接整个文件发到core里面，但是core的grpc配置为单条message最大只能为20M，所以有些文件是无法传输到core端的。这里是增加了一个叫做
SendLargeFile的方法，用stream的方式接受cli端分chunk发送过来的数据，并拼接起来再一并发送到容器里面。