A user can set core.sparsecheckout to true, then populate .git/info/sparse-checkout with the specific subtrees that should be checked out. See http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/ for a basic explanation.

I don't know whether it makes sense for libgit2 to support this in its own checkout operations, but sparse checkouts should be taken into consideration for diffs and status (in that omitted subtrees should not appear).