Conversation
ncw
left a comment
There was a problem hiding this comment.
This is an interesting idea.
Notes on the code first
- can you remove the
.oldfiles? - can you remove the V2 suffix from
clientV2.go
You can use a type assertion to figure out what kind of object the source is if you want to do something special to IMAP objects.
I think we should probably discuss what an IMAP backend might be used for.
I like the idea of being able to backup your email. That is a great idea. Being able to restore that backup sounds useful too.
Beyond that I don't think IMAP can store general purpose files/directories so I don't think the backend should attempt to do that.
It looks like you've implemented a sensible scheme for directories and mail message IDs.
Testing...
It would be nice to have some tests for this. I agree the normal backend tests won't work. Can we spin up a Go based IMAP server and run some tests against that? Or maybe we could spin up a docker based IMAP server (see fstest/test_server for how to do that).
Ideally we'd exercise all the code but once the IMAP server is running then we could exercise it with the relatively high level operations.Copy and sync packages rather than writing low level tests.
Code coverage on tests go run -cover -coverprofile my.prof is useful for seeing how well we are doing.
The submission is also missing documentation. Exactly how IMAP is mapped to file system needs to be really carefully explained in the docs.
|
|
||
| var errNoHash = errors.New("no hash available") | ||
|
|
||
| type key int |
There was a problem hiding this comment.
You don't need to do this, just type assert the src parameter in the Update and similar methods.
I suggest we work on that after :-) |
Done
I guess source is not required if I'm copying imap->imap since the file names are created from the message and they have the dates/hash/sizes on it. Things are different copying from a regular storage backend. If I try to copy a local file named 'email.txt' to imap server I can't check using the filename. First, I have to check if 'email.txt' is actually a valid email message (dont want to upload invalid files, who knows what would happen). Then look for the message in the imap server using 'email.txt' checksum, date and size. That's why I created the SourceObjectKey so that when looking for the destination object in moveOrCopyFile() can actually find if it actually exists. There was the NeedTransfer function but it needs a Fs.Object and dst.NewObject will never find 'email.txt' in the imap server.
That's what I thought, backup, restore and sync servers. Right now, you can only copy valid email message, it will fail otherwise. The only problem with this is the need of reading the file beforehand for a copy.
That's basically the Maildir format used by Dovecot/Postfix/etc.
The library I'm using has a server option. I'll have to check how it works. I did this backend in March last year, have to refresh my memory. |
What is the purpose of this change?
I was working on backing up messages on an IMAP account and I thought that an IMAP backend would be useful for rclone. In theory it could work like imapsync when source and destination are IMAP servers, and like imap-backup if one of them is a regular storage.
Since this is an IMAP backend, I don't know how the tests would work so there aren't any.
I tested ls/cat/mkdir/rmdir/copy/sync/delete. The way the imap library works I have to read the full message to memory. Might be problematic with large messages. Tested against a dovecot server with no certificate. I wanted to test gmail but have no experience with OAUTH. Looked at the google drive backend but didn't get it. AFAIK all I need is a token to authenticate with the emersion/go-imap library.
You can copy from imap server to any type of storage, if it's not an imap server it will create files using the Maildir naming format.
When you copy a file to the imap server it will always check if the file is a valid IMAP message, so it will read it before adding it. We don't want to drop random files into the mail server.
Was the change discussed in an issue or in the forum before?
It was discussed here and here
Checklist