Change robots.txt #1709

We had an old robots.txt for the wmflabs with subdirectory.
As the Toolforge Scholia domain is changed to scholia.toolforge.org
then the robots.txt was no longer having a correct path and should
thus be ineffective.

Search engines index dynamic content on Scholia pages differently:
Bing and Quant seems to index the content, but Duckduckgo and Google do
apparently not, see #1709.

With this change, not only is the path change, but bots are now allows.
If this results in too much load on the Toolforge infrastruture then it
should be changed to a 'Disallow: /'.

Note that the 'robots' HTML meta tag on each Scholia page has a nofollow
to avoid crawling.

Merge pull request #938 from fnielsen/gitpod-testing

typo fix

Merge branch 'master' of github.com:fnielsen/scholia

Merge branch 'master' of github.com:fnielsen/scholia

Create wikipedia.py