I tried this, and vim wacked out on me (since I have other plugins). I couldn't figure out how to disable everything else and just use this:

export PAGER="vim -c "Man $1 $2" -c 'silent only'
-U NONE -u .vim/bundle/vim-man/plugin"

OTOH, it works good enough that I might change my behavior and just use the command w/in vim vs at the shell.