Actually one possibility would be to only test in svg and using rcParams["svg.fonttype"] = "none", which avoids embedding the font, and therefore results in a very small (1.7kb) file. (Another 30% could be chopped off if you could get rid of the metadata block...)

In order to render the result correctly, inkscape needs to know the path to Matplotlib fonts. This can either be done by adding "Additional font directories" in the Inkscape GUI under preferences/tools/text, or (I guess that'll be the practical solution for us, which should be reasonably easily automatable) by copying the Matplotlib fonts to e.g. /tmp/foo/inkscape/fonts and running inkscape with XDG_CONFIG_HOME=/tmp/foo.

I think the savings may be worth it? Especially as this should allow us to work much more regularly on adding mathtext improvements without being restricted by the baseline images problem at all.

Can we not merge the new tests with the old ones?
(Might not affect the overall size, but could reduce the need to add new images for every new supported symbol?)
@anntzer said something in a similar line of thought here #18916 (comment)

Superseded by #19261.