This part is actually quite simple. I had to rearrange some of my application to cope with the different workflow.

I created an endpoint in my app, /oauth/callback, and an OAuthCallbackController to handle it. Then I created a simple HTML file for my redirect URI, similar to oauth2callback.html but instead of loading Angular, it simply manipulates the URL to make it compatible with "hashbang" routing and pass it along to the Angular endpoint:

location.replace('/#/oauth/callback?' + location.hash.slice(1));

Then I'm able to access the OAuth params from my controller, set the token and all that good stuff:

  .controller('OAuthCallbackController',
    function($scope, $rootScope, $location, $injector, VgerCredentials, Token, $timeout, flash, User, UserTrackingId) {
      var params = $location.search();
      Token.verifyAsync(params.access_token).
        then(function(data) { ... }
    });