In the usetex case, the origin of the shift is, I believe, the following. Remember that usetex+agg is currently implemented by compiling the tex string to a dvi, converting that to png with dvipng and inserting the image "at the right place". In order to determine that place, we must compute the descent of the text, i.e. how much it extends below the baseline. dvipng cannot report this (actually, it has an option to do this using the TeX preview package, but that is GPL), so we infer it by parsing the dvi file ourselves and getting the glyph metrics font via the TeX font metrics (tfm) files. Unfortunately, tfm files actually lie(!) about the real rasterized metrics (see e.g. https://tex.stackexchange.com/questions/526103 for why, see also #21653 (comment)); in the current specific case, it claims that the italic V has zero descent even though it actually has a small one. Because of that, in the $V$ l string, we believe that the bottom-most pixel that is black is the baseline of the png image and we (incorrectly) align on that. For $V$ lp, this doesn't occur because the descent of p is bigger (and correct) anyways.

Two fixes are possible. One is to stop using the tfm files to get glyph metrics, and instead get them from the actual font files with FreeType. Probably a fair bit of code surgery is required, but this seems doable. The other (already suggested elsewhere) is to stop using dvipng and render the glyphs ourselves. This is in fact already the approach used by the vector backends, which is why you don't see the problem e.g. when saving to pdf. This is also the approach used by mplcairo (even for raster output), so you can also use that in the meantime (probably mplcairo master, the last release of mplcairo is too old to work with matplotlib 3.5).

For the mathtext case, regarding vertical positioning, I suspect that the problem comes at least in part from the use of "+1/-1" at

Still in the mathtext case, regarding horizontal spacing, the presence of space before the V is (I think, though I haven't checked) just because it is an italic glyph, which gets drawn a bit further to the left from the origin (technically, it has a positive xMin; see the "glyph metrics" figure at https://freetype.org/freetype2/docs/glyphs/glyphs-3.html); this can be checked by typesetting the non-mathtext case in italic (with style="italic") which also gives the spacing before the V. As for the space after the "l", I still need to investigate this (funnily, mplcairo shows the opposite behavior: no space after the non-mathtext case, and space after the mathtext case), but this may be due to inconsistency as to whether the limit is set by the last glyph's width (i.e., actually inked area) or advance (i.e., how much does the pen advance in prevision of a (hypothetical) following glyph).

Thanks a lot @anntzer !

Using mplcairo indeed fixes almost everything.

It is not 100% perfect for mathtext: as you can see the 'l' extends a little bit over the box in the "$V$ l" case (the bbox is a tiny bit too short). But that's already much better for my use-case !