[v5] Increase default shard size #42555

TL;DR

It'd be greatly beneficial to increase the default shard size to align better with the new Hub recommendations + avoid repos with hundreds of shards BUT let's be careful on the implications in terms of I/O and optimizations.

Feature request

Mentioned in #42391 so I'm creating a separate issue to keep track of it.

Original comment from @Wauplin:

currently max_shard_size defaults to 5GB. Recommendations are now to use <50GB chunks (the recommendation increased a lot since Xet is now the default). Should we increase the default value to e.g. 48GB? (not 50, to avoid having files slightly larger than 50).

Then from @LysandreJik (#42391 (comment)):

This is the only one I'm not sure about -> have smaller shards means that it's easier to parallelize load from checkpoints. Will let @ArthurZucker complete as he's been working with this quite a bit lately

from @ArthurZucker (#42391 (review)):

have not really seen the exact line, but its something we do need to study: smaller shards -> better i/o file lock potentially?

I am probably wrong as we can parallel read a file (which we did not do before) so should be fine with main but let's test it out please

finally from @Cyrilvallez (#42391 (comment)):

Agreed with @ArthurZucker here! In general it would indeed be very good to increase the size of the shards to a much larger value, to avoid having hundreds of files on big repos, and it's something that was raised previously. But we need to study carefully the implications on loading time and if it's going to impact us positively/negatively!