See: https://tools.ietf.org/html/rfc7914. In particular, Section 2: scrypt Parameters:

The CPU/Memory cost parameter N ("costParameter") must be larger than 1, a power of 2, and less than 2^(128 * r / 8). The parallelization parameter p ("parallelizationParameter") is a positive integer less than or equal to ((2^32-1) * 32) / (128 * r).

Compare with: https://github.com/golang/crypto/blob/master/scrypt/scrypt.go#L200.

That is, it doesn't enforce N < 2^(128 * r / 8) as far as I can tell (I'm not fluent in Go). In the Go playground I find that N's upper limit when r is 1 is 16777215 such that N=262144, r=1, p=8 won't cause scrypt to choke even though it should per the RFC.

Context: ethereum/go-ethereum#19977.