His uncle was First Mayor and President of the Senate (head of state) Hermann Gossler.
His mother was Mary Elizabeth Bray (1810–1886), a granddaughter of Samuel Eliot and a member of the Eliot family from Boston.
Known as "John Bi" (B pronounced in English) by his friends, he was a Schöngeist.
He loved music and the theatre, and wanted to study languages, literature and history.
But his father had destined him to become a banker.
After the death of his father in 1879, he became head of Berenberg Bank.
In 1880, the Hamburg senate granted him the name of Berenberg-Gossler in recognition of the fact that he was the owner of the Berenberg Bank (the Berenberg family became extinct in the male line in 1773, but the Gosslers were descended from the family through the family's last member Elisabeth Berenberg married Gossler, who died in 1822).
In 1888, Johann Berenberg-Gossler was ennobled in the Kingdom of Prussia (although he was a citizen of Hamburg and not of Prussia) and conferred the title of Baron in 1910.
The ennoblement was controversial within his family and Hamburg high society, as nobility (which did not exist in Hamburg) was frowned upon by the staunchly republican Hamburg grand burghers.
His sister Susanne, married Amsinck, exclaimed "Aber John, unser guter Name!"Johann von Berenberg-Gossler was less inclined to take risks than his father, and closed the bank's branches in New York City and Boston in 1880 and 1891 respectively.
At the time of his death, he was the second richest person in Hamburg.
He was the father of Senator and Ambassador John von Berenberg-Gossler (1866–1943) and banker Baron Cornelius von Berenberg-Gossler (1874–1953).
His son John von Berenberg-Gossler became a politician despite his father's wish that he would succeed him as head of the bank, and therefore had to give up his shares in the company.
He was succeeded as head of the bank and as Baron of Berenberg-Gossler (a title tied to the ownership of Gut Niendorf) by his younger son Cornelius von Berenberg-Gossler.