The Hirama family had served as the yonin (retainers) of the goshi (farmer-samurai) Serizawa family since the Serizawa clan had ruled from Serizawa-jo Castle.
平間家は水戸藩の郷士芹沢氏が芹沢城主だった時代からの家来の家柄で、芹沢家の用人（家老）を務めていた。