In a small bowl, combine egg yolks and sugar. Whisk together until light and fluffy (2-3 minutes). Set aside.
In a medium saucepan over medium-high heat, add milk, cinnamon, and nutmeg. Bring the mixture to a boil.
Remove from heat. Working one spoonful at a time, add the hot milk mixture to the egg mixture, whisking constantly until fully incorporated. (Be sure to add the hot milk slowly to keep from scrambling the eggs.) Continue until all of the egg mixture is incorporated.
Pour the mixture back into the saucepan and place over medium heat. Continue cooking until thickened slightly (about 3-5 minutes). It should coat the back of a spoon, but it will continue to thicken more as it cools.
Remove from heat and let it cool completely.
When cool, whisk in the heavy cream, vanilla, and brandy (if using).
Cover and transfer to the refrigerator to chill for 4 hours, up to overnight.
When ready to serve, top with whipped topping and sprinkle with cinnamon or nutmeg. Enjoy!