[Bug]: Entire mailbox fails on single attachment #5419
Description
What happened?
When a backup fails on getting a single mail attachment, there is no output for that mailbox backup and the entire mailbox backup fails.
We run a backup for 50 mailboxes, one of them fails on an attachment, the other 49 run fine. One of those mailboxes has an email with an attachment that cannot be found. This causes the “unable to complete backup” error (on every run), thus leaving us without any data from that mailbox and no way to restore anything from it.
Below the graph logs to get the attachments for the email. It shows 9 attachments, the last two seem to be empty.
When viewing the email in the mailbox, I found 11 attachments: 7 png-files (these were visible in the graph call), 2 vcf cards and 2 (empty) eml files.
I’ve been unable to retrieve the two empty attachments through the graph api, log below.
Instead of erroring out the mailbox completely, wouldn’t it be possible to skip the item that causes the error so that the backup can complete? That would mean only losing a small amount of data (the specific attachment or the email) instead of being unable to retrieve the complete mailbox.
Corso Version?
Corso v0.19.0
Where are you running Corso?
- Kubernetes
- S3
- Image: ghcr.io/alcionai/corso:v0.19.0
- Image ID: ghcr.io/alcionai/corso@sha256:41d7fb46bb9339b8331455bf2a652d5f96d7ea74e4b146d762bd5cb2046c1219
Relevant log output
Error on the mailbox with the attachment issues:
“Error: Unable to complete backup
Cause: fetching item: fetching item: getting mail attachment by id: An internal server error occurred. The operation failed., Property: [Calc:PersonId] PersonId, PropertyErrorCode: CorruptedData, PropertyErrorDescription: .: An internal server error occurred. The operation failed., Property: [Calc:PersonId] PersonId, PropertyErrorCode: CorruptedData, PropertyErrorDescription: .”
The full error:
{"level":"error","ts":1724315862.5491636,"caller":"graph/logging.go:60","msg":"graph api error: 500 Internal Server Error","disable_assist_backup":"false","request_start_time":"2024-08-22T08:37:27.743862074Z","resp_status_code":"500","incremental":"true","can_use_previous_backup":"true","collection_source":"operations","service":"Exchange”,”url":"https://graph.microsoft.com/v1.0/users/{user-id}/messages/{message-id}/attachments/{attachment-id} $expand=microsoft.graph.itemattachment/item","clues_trace”:”{trace},”resp_status":"500 Internal Server Error","method":"GET","assist_bases":"1","merge_bases":"0","request_content_len":"0","snapshot_type":"item data","resp_content_len":"-1","len_bases":"0","category":"email","attachment_id”:”{attachment-id}”,”attachment_odatatype":"#microsoft.graph.itemAttachment","request_retry_iter":"0","resource_owner_name”:”{owner},”categories":"[Contacts Emails Events]","tenant_id”:”{tenant_id}”,”resource_owner_selected”:”{owner}”,”collection_count":"16","resource_owner_id”:”{owner_id},”backup_id”:”{id},”can_use_metadata":"true","response":"HTTP/2.0 500 Internal Server Error\r\nCache-Control: private\r\nClient-Request-Id: {id}\r\nContent-Type: application/json; odata.metadata=minimal; odata.streaming=true; IEEE754Compatible=false; charset=utf-8\r\nDate: Thu, 22 Aug 2024 08:37:41 GMT\r\nRequest-Id: {id}\r\nStrict-Transport-Security: max-age=31536000\r\nVary: Accept-Encoding\r\nX-Ms-Ags-Diagnostic: {\"ServerInfo\":{\"DataCenter\":\"West Europe\",\"Slice\":\"E\",\"Ring\":\"5\",\"ScaleUnit\":\"003\",\"RoleInstance\":\"AM1PEPF0003BA69\"}}\r\n\r\n{\"error\":{\"code\":\"ErrorInternalServerError\",\"message\":\"An internal server error occurred. The operation failed., Property: [Calc:PersonId] PersonId, PropertyErrorCode: CorruptedData, PropertyErrorDescription: .\"}}"}https://graph.microsoft.com/v1.0/users/{user-id}/messages/{message-id} in ms graph explorer:
{
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('{user-id}')/messages/$entity",
"@odata.etag": “W/\”{bla}””,
"id": “{id}”,
"createdDateTime": "2024-05-10T09:40:25Z",
"lastModifiedDateTime": "2024-05-10T09:42:29Z",
"changeKey": “{key}”,
"categories": [],
"receivedDateTime": "2024-05-10T09:40:27Z",
"sentDateTime": "2024-05-10T09:40:01Z",
"hasAttachments": true,
"internetMessageId": “{id}”,
"subject": “subject”,
"bodyPreview": “preview”,
"importance": "normal",
"parentFolderId": “{id}”,
"conversationId": “{id}”,
"conversationIndex": “{index}”,
"isDeliveryReceiptRequested": null,
"isReadReceiptRequested": false,
"isRead": true,
"isDraft": false,
"webLink": "https://outlook.office365.com/owa/?ItemID=<id>&exvsurl=1&viewmodel=ReadMessageItem",
"inferenceClassification": "focused",
"body": {
"contentType": "html",
"content": "<html>{content-of-mail}</html>"
},
"sender": {
"emailAddress": {
"name": “{name}”,
"address": “{address}”
}
},
"from": {
"emailAddress": {
"name": “{name}",
"address": “{address}”
}
},
"toRecipients": [
{
"emailAddress": {
"name": “{name}”,
"address": “{address}”
}
},
{
"emailAddress": {
"name": “{name}”,
"address": “{address}”
}
}
],
"ccRecipients": [],
"bccRecipients": [],
"replyTo": [],
"flag": {
"flagStatus": "notFlagged"
}
}https://graph.microsoft.com/v1.0/users/{user-id}/messages/{message-id}/attachments shows 9 attachments.
{
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('{user-id}')/messages('{message-id}')/attachments",
"value": [
{
"@odata.type": "#microsoft.graph.fileAttachment",
"@odata.mediaContentType": "image/png",
"id": “{id}”,
"lastModifiedDateTime": "2024-05-10T09:40:25Z",
"name": "image001.png",
"contentType": "image/png",
"size": 174792,
"isInline": true,
"contentId": "[email protected]",
"contentLocation": null,
"contentBytes": “content”
},
{
"@odata.type": "#microsoft.graph.fileAttachment",
"@odata.mediaContentType": "image/png",
"id": “{id}”,
"lastModifiedDateTime": "2024-05-10T09:40:25Z",
"name": "image002.png",
"contentType": "image/png",
"size": 916,
"isInline": true,
"contentId": "[email protected]",
"contentLocation": null,
"contentBytes": “content"
},
{
"@odata.type": "#microsoft.graph.fileAttachment",
"@odata.mediaContentType": "image/png",
"id": “{id}”,
"lastModifiedDateTime": "2024-05-10T09:40:25Z",
"name": "image003.png",
"contentType": "image/png",
"size": 635,
"isInline": true,
"contentId": "[email protected]",
"contentLocation": null,
"contentBytes": “content”
},
{
"@odata.type": "#microsoft.graph.fileAttachment",
"@odata.mediaContentType": "image/png",
"id": “{id}”,
"lastModifiedDateTime": "2024-05-10T09:40:25Z",
"name": "image004.png",
"contentType": "image/png",
"size": 643,
"isInline": true,
"contentId": "[email protected]",
"contentLocation": null,
"contentBytes": “content”
},
{
"@odata.type": "#microsoft.graph.fileAttachment",
"@odata.mediaContentType": "image/png",
"id": “{id}”,
"lastModifiedDateTime": "2024-05-10T09:40:25Z",
"name": "image005.png",
"contentType": "image/png",
"size": 21942,
"isInline": true,
"contentId": "[email protected]",
"contentLocation": null,
"contentBytes": “content”
},
{
"@odata.type": "#microsoft.graph.fileAttachment",
"@odata.mediaContentType": "image/jpeg",
"id": “{id}”,
"lastModifiedDateTime": "2024-05-10T09:40:25Z",
"name": "image006.jpg",
"contentType": "image/jpeg",
"size": 142513,
"isInline": true,
"contentId": "[email protected]",
"contentLocation": null,
"contentBytes": “content”
},
{
"@odata.type": "#microsoft.graph.fileAttachment",
"@odata.mediaContentType": "image/png",
"id": “{id}”,
"lastModifiedDateTime": "2024-05-10T09:40:26Z",
"name": "image007.png",
"contentType": "image/png",
"size": 74391,
"isInline": true,
"contentId": "[email protected]",
"contentLocation": null,
"contentBytes": “content”
},
{
"@odata.type": "#microsoft.graph.itemAttachment",
"id": “{id}”,
"lastModifiedDateTime": "2024-05-10T09:40:26Z",
"name": "",
"contentType": null,
"size": 1304,
"isInline": false
},
{
"@odata.type": "#microsoft.graph.itemAttachment",
"id": “{id}”,
"lastModifiedDateTime": "2024-05-10T09:40:26Z",
"name": "",
"contentType": null,
"size": 1586,
"isInline": false
}
]
}https://graph.microsoft.com/v1.0/users/{user-id}/messages/{message-id}/attachments/{attachment-id}?$expand=microsoft.graph.itemattachment/item for the last two attachments:
{
"error": {
"code": "ErrorInternalServerError",
"message": "An internal server error occurred. The operation failed., Property: [Calc:PersonId] PersonId, PropertyErrorCode: CorruptedData, PropertyErrorDescription: ."
}
}