So, you want to give your userbase creative freedom, without allowing it to attack itself or marginalised groups. Is this right? Here’s what I’d suggest:
Rule: fanfics featuring prejudice must not show it in a positive light.
This rule alone would shoo 90% of the potential bigots, who’d likely cry about “muh freeze peaches”, but still allow people to have bigotry in their fanfics.
Rule: fanfic discussion must be kept on-topic.
If there’s something that bigots love to do, it’s to pick on some loose end of a discussion, then pull and pull it until the discussion gravitates around their pet issue; this rule would make short work of this tactic. It’s also useful to improve general quality of the discussions.
[If allow RP at all] Rule: Roleplay is only allowed in a specific subforum / community.
Some people actually like to RP their OCs; to each their own, I guess? Either way, if you let people RP all over the instance/forum/site some might use it to say “ackshyually I was RPing” to attack other members.
Don’t feel afraid to change the rules as you go. You’ll get a thousand problems that nobody could predict, it’s fine to address them. As long as you do it transparently, it’s fine.
A few additional points:
Keep a close watch on users who gravitate around certain specific topics. Most of them should be OK, but some might be looking for trouble. Look specially for signs of sealioning.
Recruit mods with good reading comprehension and who are OK reading huge walls of text. You’ll need it for subtleties like distinguishing “the MC is racist and this is shown as a character flaw” versus “the MC is racist because the user is self-inserting”.
Social media is full of a certain type of user full of “good intentions”, who points fingers at other users based on weak reasons (such as poor reading, assumptions, etc.); for example, if you say “Hitler was shit but this has nothing to do with him eating bread!”, they’ll pick on that “but” and claim that you’re a Nazi. Well, I call those “witch hunters”. Don’t let witch hunters in your community - they make people feel insecure to approach certain topics, and they indirectly help the ones whom they allegedly fight against.
I also feel like your idea leans more towards an old style forum than a Lemmy instance. So perhaps NodeBB is a better choice in your case.
It seems that your idea is fairly solid already on what to do, then. That’s great!