He married secondly in 1845 the widow of a Boston banker, Adeline Fowle (Mrs Samuel Welles), who died in 1869; he married thirdly, in 1871, Georgiana-Gabrielle de Flahaut, marquise douairière de La Valette, who died in 1907; she was sister of Emily, Dowager Marchioness of Lansdowne.