What happens when multiple extensions need to update the robots.txt file?

Extension, that registers its public folder first, wins.

If you know you are using multiple extensions that redefine robots.txt, you have to create an application-specific extension and merge inside it all the required robots.txt from the different extensions.
And don't forget to put your application-specific extension at the beginning of the plugins list, so that it registers its public folder first.

That implies that the person writing and and the person installing the extension needs to be aware of this behaviour, rather than just using the templating engine to handle it.

For example, I might write an authentication extension and want to append to robots.txt to keep robots away from my URL paths, and I also have another extension doing my theming/HTML customizations and also have reasons to modify the robots.txt there. My authentication extension was excellent and others now wish to adopt it but I now have to put a warning in my documentation to note that my extension may screw up the robots.txt for your site.

Long story short, I'm struggling to see what the use case/bug was that required removing the customization/templating from robots.txt?

It was done as part of python3 migrations 8ee2148 for simplicity sake. However it can be reimplemented as flask view if someone has the time for it.

Thanks. That makes sense :) I will note it as an issue to be tackled for the migration of our site off 2.8.3

#6930 notes the issue, with a pull request offering a fix in #6953