Looking to selfhost a family Minecraft server for my kids and nephews. Got it running on an old media server box but it keeps crashing due to not enough RAM, even with just one test player. I was hoping to unveil it as a Christmas surprise until I hit this wall. Anyone have a ballpark for what processing and RAM a small server like this will need?

For Java edition with no mods I consider 2gb the absolute bare minimum. That’s dedicated specifically to the JVM, not system wide.
4gb to be able to have any useful render distance for multiplayer IMO.
My kids are heavily into mods, and I’d personally want a few myself that did things like set parts of the map as only buildable by certain users (so each kid can have a “home base”, as it were).
4G seems like a lot but it’s been a few years since I’ve built a PC. In your experience, will I be able to support a modded server on that 4G?
If you’re just throwing a few of your fav mods together, 4gb can absolutely do it. If you’re interested in a larger mod pack, you’ll want to check the page for the modpack.
Extreme example: right now I’m hosting an “all the mods 10” server for some friends and they suggest at least 8gb of ram.
On the other hand I’ve run a simple pack with a few things like Mekanism and Botania on 4gb no problem.