Fosstodon is *way* too big. Looking into the future, fediverse ought not optimise for that scale.
No JVM GC, including G1, works without a noticeable memory overhead. That directly contributes to the total memory requirements of the server, and it does make a difference that it's by default higher than needed. As you said, there are a bunch of other things on that server as well that use memory.