Young and old men are leaving the labor force, fueling a record decline