401 when using ChunkedUploadProvider for large mail attachments

[jball-vcra]

Expected behavior
ChunkedUploadProvider.upload(...) uploads the specified file and attaches it to the message, per https://docs.microsoft.com/en-us/graph/outlook-large-attachments?tabs=java.
Actual behavior
401 : Unauthorized upon request.
com.microsoft.graph.core.ClientException: Error code: InvalidAudienceForResource
Error message: The audience claim value is invalid for current resource. Audience claim is 'https://graph.microsoft.com/', request url is 'https://outlook.office.com/api/v2.0/Users...'.
The issue appears to be the API is including an Authorization header in the request, in contradiction with the documentation.

Do not specify an Authorization request header. The PUT query uses a pre-authenticated URL from the uploadUrl property, that allows access to the https://outlook.office.com domain.

Inspection of the authtoken parameter in the upload session's URL shows the correct audience ( "aud": "https://outlook.office.com/api/"), while the auth bearer token's audience is "https://graph.microsoft.com".
Steps to reproduce the behavior
Using an MS Dev account and pregenerated users; microsoft-graph-1.9.0.jar, microsoft-graph-core.1.0.1.jar, microsoft-graph-auth-1.9.0.jar; Java 8.
public class LargeAttachmentUploadError implements GraphExampleConstants {

public static void main(String[] args) throws IOException {
    String recipient = args[0];
    String filepath = args[1];
    Path p = Paths.get(new File(filepath).toURI());
    
    UsernamePasswordProvider authProvider = new UsernamePasswordProvider(
                    clientID,
                    Arrays.asList("https://graph.microsoft.com/.default"),
                    username,
                    password,
                    NationalCloud.Global,
                    tenant,
                    clientSecret);

    IGraphServiceClient graphClient = GraphServiceClient
                    .builder()
                    .authenticationProvider(authProvider)
                    .buildClient();

    graphClient.getLogger().setLoggingLevel(LoggerLevel.DEBUG);   
    
    Message message = new Message();
    message.subject = "Meet for lunch?";
    ItemBody body = new ItemBody();
    body.contentType = BodyType.TEXT;
    body.content = "The new cafeteria is open.";
    message.body = body;
    LinkedList<Recipient> toRecipientsList = new LinkedList<Recipient>();
    Recipient toRecipients = new Recipient();
    EmailAddress emailAddress = new EmailAddress();
    emailAddress.address = recipient; 
    toRecipients.emailAddress = emailAddress;
    toRecipientsList.add(toRecipients);
    message.toRecipients = toRecipientsList;
    message.hasAttachments = true;
    message = graphClient.me().mailFolders("drafts").messages().buildRequest().post(message);
    
    AttachmentItem attachmentItem = new AttachmentItem();
    attachmentItem.name = p.getFileName().toString(); 
    attachmentItem.isInline = false;
    attachmentItem.attachmentType = AttachmentType.FILE; 
    attachmentItem.size = Files.size(p);

    UploadSession uploadSession = graphClient.me().messages(message.id).attachments().createUploadSession(attachmentItem).buildRequest().post();
    try(InputStream inputStream = Files.newInputStream(p)) {
        ChunkedUploadProvider<AttachmentItem> chunkedUploadProvider = new ChunkedUploadProvider<>(uploadSession, graphClient, inputStream, attachmentItem.size, AttachmentItem.class);
        
        IProgressCallback<AttachmentItem> progressCallback = new IProgressCallback<AttachmentItem> () {
            @Override
            // Called after each slice of the file is uploaded
            public void progress(final long current, final long max) {
                System.out.println(String.format("Uploaded %d bytes of %d total bytes", current, max));
            }

            @Override
            public void success(final AttachmentItem result) {
                System.out.println(String.format("Uploaded mail attachment with ID: %s", result.name));
            }

            @Override
            public void failure(ClientException ex) {
                System.out.println("Error uploading mail attachment: " + ex.getMessage());
                throw ex;
            }
        };

        chunkedUploadProvider.upload(progressCallback, new int[] {320 * 1024, 1});
    
    } catch (Exception e) {
        e.printStackTrace();
    }
}

}
AB#5829

[davidmoten]

My experience with the Graph API supports this explanation. I had to solve the same problem a few weeks ago.

[gantners]

There are multiple issues as far as I can overlook it.

First Audience is mixed when using the client with authenticationprovider which effectively authenticates the PUT request, but should not as stated:

Do not specify an Authorization request header. The PUT query uses a pre-authenticated URL from the uploadUrl property, that allows access to the https://outlook.office.com domain.

You can get around that by providing a client with a NOOP on the authenticateRequest Method, not signing the request.

protected IGraphServiceClient getClientUnauthorized() {
		final IClientConfig config = new DefaultClientConfig() {

			@Override
			public IAuthenticationProvider getAuthenticationProvider() {
				return new IAuthenticationProvider() {
					@Override
					public void authenticateRequest(IHttpRequest request) {
						// NO OP
					}
				};
			}
		};
		IGraphServiceClient client = GraphServiceClient.fromConfig(config);
		client.setServiceRoot("https://graph.microsoft.com/beta/");
		return client;
	}

The uploadUrl from the UploadSession will be sufficient as it provides a preauthenticated Token for getting access to the office area.

However the chunking is also flawed.
Uploading chunks return reponse code 200 where the ChunkedUploadResponseHandler awaits 202. That finishes the chunked upload immediately and the upload will not finish with a 201 Created.

You can get it to work if you extend the ChunkedUploadResponseHandler and overwrite the Response Code and result return:

package com.example.ms.graph;

import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;

import com.microsoft.graph.concurrency.ChunkedUploadResponseHandler;
import com.microsoft.graph.http.DefaultHttpProvider;
import com.microsoft.graph.http.GraphServiceException;
import com.microsoft.graph.http.HttpResponseCode;
import com.microsoft.graph.http.IConnection;
import com.microsoft.graph.http.IHttpRequest;
import com.microsoft.graph.logger.ILogger;
import com.microsoft.graph.models.extensions.AttachmentItem;
import com.microsoft.graph.models.extensions.UploadSession;
import com.microsoft.graph.requests.extensions.ChunkedUploadResult;
import com.microsoft.graph.serializer.ISerializer;

import okhttp3.Response;

public class ChunkedUploadResponseHandler2<UploadType> extends ChunkedUploadResponseHandler<UploadType> {

	private final Class<UploadType> deserializeTypeClass;

	public ChunkedUploadResponseHandler2(Class<UploadType> deserializeTypeClass) {
		super(deserializeTypeClass);
		this.deserializeTypeClass = deserializeTypeClass;
	}

	/**
	 * Generate the chunked upload response result
	 *
	 * @param request
	 *            the HTTP request
	 * @param response
	 *            the HTTP response
	 * @param serializer
	 *            the serializer
	 * @param logger
	 *            the system logger
	 * @return the chunked upload result, which could be either an uploaded item or error
	 * @throws Exception
	 *             an exception occurs if the request was unable to complete for any reason
	 */
	@Override
	public ChunkedUploadResult<UploadType> generateResult(final IHttpRequest request, final Response response, final ISerializer serializer, final ILogger logger)
			throws Exception {
		InputStream in = null;
		try {
			if (response.code() == HttpResponseCode.HTTP_ACCEPTED || response.code() == HttpResponseCode.HTTP_OK) {
				logger.logDebug("Chunk bytes has been accepted by the server.");
				in = new BufferedInputStream(response.body().byteStream());
				final UploadSession session = serializer.deserializeObject(DefaultHttpProvider.streamToString(in), UploadSession.class);

				return new ChunkedUploadResult<UploadType>(session);

			}
			else if (response.code() == HttpResponseCode.HTTP_CREATED) { // || response.code() == HttpResponseCode.HTTP_OK
				logger.logDebug("Upload session is completed, uploaded item returned.");
				// in = new BufferedInputStream(response.body().byteStream());
				// String rawJson = DefaultHttpProvider.streamToString(in);
				// UploadType uploadedItem = serializer.deserializeObject(rawJson, this.deserializeTypeClass);
				String itemUrl = response.header("Location");
				logger.logDebug(String.format("Upload session is completed, upload url: %s", itemUrl));
				AttachmentItem ai = new AttachmentItem();
				ai.name = itemUrl;
				UploadType uploadedItem = (UploadType) ai;
				return new ChunkedUploadResult<UploadType>(uploadedItem);
			}
			else if (response.code() >= HttpResponseCode.HTTP_CLIENT_ERROR) {
				logger.logDebug("Receiving error during upload, see detail on result error");

				return new ChunkedUploadResult<UploadType>(GraphServiceException.createFromConnection(request, null, serializer, response, logger));
			}
		}
		finally {
			if (in != null) {
				try {
					in.close();
				}
				catch (IOException e) {
					logger.logError(e.getMessage(), e);
				}
			}
		}
		return null;
	}
}

This will successfully upload the file and create the resource on the server.

The call will be as stated on the documentation:

File file = new File(ea.getFilepath());

AttachmentItem attachmentItem = new AttachmentItem();
attachmentItem.attachmentType = AttachmentType.FILE;
attachmentItem.name = file.getName();
attachmentItem.size = file.length();
attachmentItem.contentType = "application/octet-stream";

InputStream fileStream = new FileInputStream(file);
long streamSize = attachmentItem.size;

// Create a callback used by the upload provider
IProgressCallback<AttachmentItem> callback = new IProgressCallback<>() {
    @Override
    // Called after each slice of the file is uploaded
    public void progress(final long current, final long max) {
        logger.logDebug(String.format("Uploaded %d bytes of %d total bytes", current, max));
    }

    @Override
    public void success(final AttachmentItem result) {
        logger.logDebug(String.format("Uploaded file."));
    }

    public void failure(final ClientException ex) {
        logger.logError(String.format("Error uploading file: %s", ex.getMessage()), ex);
    }
};

UploadSession uploadSession = getClient().me().messages(drafted.id).attachments().createUploadSession(attachmentItem).buildRequest().post();
logger.logDebug(String.format("Upload session URL: %s", uploadSession.uploadUrl));

ChunkedUploadProvider<AttachmentItem> chunkedUploadProvider = new ChunkedUploadProvider<>(uploadSession, getClientUnauthorized(), fileStream,
        streamSize, AttachmentItem.class);

// Config parameter is an array of integers
// customConfig[0] indicates the max slice size
// Max slice size must be a multiple of 320 KiB, numRetries
int[] customConfig = {
        320 * 1024, 1
};

// Do the upload
chunkedUploadProvider.upload(callback, customConfig);

So it needs to be checked if there is really consistent behaviour on the server side regarding the response codes for accepting chunks when using the different endpoints onedrive || messages || events etc.

Maybe it is a good idea to establish separate reponsehandlers and uploadproviders for the different endpoints, especially when mixing the audience. The preauthenticated uploadurl also seems not to be the ideal solution for a permission handover.

PS: Don't forget to add the permission Mail.ReadWrite in case you upload an attachment to an existing message, otherwise you will end up also in a 401!

[jball-vcra]

@gantners , thank you for the detailed reply. I don't see an open issue for the ChunkedUploadResponseHandler, but you're correct, the 200 (HTTP OK) is in the wrong place, leading to the uploader halting after the first chunk and erroneously reporting the file upload completed successfully.

[gantners]

Maximum POST byte size for uploading is exactly 3145611 bytes.
Minimum UploadSession byte size is exactly 2097152 bytes.

POST above will lead to an exception 413 : Request Entity Too Large
There is also a minimum file size limit for creating an uploading session ErrorAttachmentSizeShouldNotBeLessThanMinimumSize

This values are nowhere to find in the documentation (3MB and 4MB depending on which corner of the doc you are v1,beta etc.) as well it is wrongly stated on the graph api error responses which say 4MB for POST.

Howevery if you follow the recommendation above 3MB => UploadSession, below 3MB POST it should turn out fine.

If the file size is under 3 MB, do a single POST on the attachments navigation property of the Outlook item; see how to do this for a message or for an event. The successful POST response includes the ID of the file attachment.
If the file size is between 3MB and 150MB, create an upload session, and iteratively use PUT to upload ranges of bytes of the file until you have uploaded the entire file. A header in the final successful PUT response includes a URL with the attachment ID

[davidmoten]

I think the limit for POST etc is encoded size. If you're posting bytes in JSON then they will be base 64 encoded. That's probably why it seems that the max is 3.x MB of raw content.

[gantners]

I think the limit for POST etc is encoded size. If you're posting bytes in JSON then they will be base 64 encoded. That's probably why it seems that the max is 3.x MB of raw content.

Shouldn't the type of transfer POST vs UploadSession be handled purely by the sdk itself rather than the developer?
Calling a high level upload function without needing to know the implementation details would be sufficient i guess.

[davidmoten]

Shouldn't the type of transfer POST vs UploadSession be handled purely by the sdk itself rather than the developer?

Would be nice for sure. Some sort of helper method would be an obvious contribution. I went through the same experience with my alternative client: https://github.com/davidmoten/odata-client#sending-an-email-with-an-attachment.

[jball-vcra]

@MIchaelMainer, please don't conflate the last comments about a helper method with the original bug report. Using the provided API results in a 401, that's a bug, and should be treated as such. Streamlining the attachment process so developers don't have to use two different processes should be a separate feature request.

[baywet]

Hey everyone,
Thanks for your feedback and patience on the matter.
I've just transferred the issue as the fault actually comes from the auth library and not the service library (msgraph-sdk-java).
The reason it's doing that is because the authentication providers are not checking which service (host) the request is going to and attaches a JWT anyway, which has an invalid audience.
I worked this around in my unit tests, but the fix remain to be implemented in the auth library.

In the meantime, using another NOOP client is a good workaround.

https://github.com/microsoftgraph/msgraph-sdk-java/blob/58eddbec13a06c6485a2168ae1768724dd5bc086/src/test/java/com/microsoft/graph/functional/TestBaseAuthenticationProvider.java