At the age of 6, Frederick was separated from his grandparents and moved to the Wye House plantation, where Aaron Anthony worked as overseer.