The key card is present but its support is thin.
Prioritize useful support such as Scarblade Scout or Lys Alana Informant.
Scout or Informant can put an Elf in the graveyard before Loyalist dies. Adherents counts your creatures and creature cards already traded away.
Your deck already enables the plan but is losing the early turns.
Take a castable early creature or efficient answer before a second expensive payoff.
An Elf token helps the battlefield count but is not a creature card left in the graveyard. Milling too aggressively can also shorten your deck.
The support is missing
Favor independently useful creatures and interaction if the narrow support does not arrive.