After recovering from the initial jaw-droppage, I actually researched this a little and it turns out this really is pretty much the most popular anthropological theory on the matter.

Never gonna think of Harry Potter quite the same again, though, that’s for sure.

The medieval apothecary garden cared little for aesthetics, focusing instead on species that healed and intoxicated and occasionally poisoned. Witches and sorcerers cultivated plants with the power to “cast spells” – in our vocabulary, “psychoactive” plants. Their potion recipes called for such things as datura, opium poppies, belladonna, hashish, fly-agaric mushrooms (Amanita muscaria), and the skins of toads (which contain DMT, a powerful hallucinogen). These ingredients would be combined in a hempseed-oil-based “flying ointment” that witches would administer vaginally using a special dildo. This was the “broomstick” by which these women were said to travel.

- Michael Pollan, The Botany of Desire (2001)