cults often practice polygamy as a few men in power “marry” girls (teenagers or younger), and control the assets. this is a power imbalance that can easily be abused.
polyamory as a modern term is trying to differentiate from that. proponents emphasize enthusiastic consent (from all parties) and communication, and often will warn against any rigid hierarchy among the polycule members. that’s an ideal, so it’s certainly possible to find examples that fail, but the intent is that people are all equally respected.
for a lot of people in long term polyamory, it’s about intimacy, which sex is part of. even if you have relationships that are primarily about fun sexy times, you’re probably going to do a lot of scheduling to maintain those relationships, or find new ones.