John Cadbury was born in Birmingham to Richard Tapper Cadbury, a draper and his wife Elizabeth Head. He was from a wealthy Quaker family that moved to the area from the west of England. John went to school at Joseph Crosfields Quaker School at Hartshill, Warwickshire.  As a Quaker in the early 19th century, he was not allowed to enter a university, so could not pursue a profession such as medicine or law.
As Quakers are pacifist, a military career was also out of the question. So, like many other Quakers of the time, he turned his energies toward business and began a campaign against animal cruelty, forming the Animals Friend Society, a forebear of the Royal Society for the Prevention of Cruelty to Animals. The family provided job opportunities and good working conditions for their employees. 
Meanwhile, Cadbury's manufacturing enterprise prospered. His brother Benjamin joined the business in 1848 and they rented a larger factory on Bridge Street. Two years later, in 1850, the Cadbury brothers pulled out of the retail business, leaving it in the hands of John's son, Richard Barrow Cadbury (Barrow's remained a leading Birmingham store until the 1960s).
Cadbury married twice. He married Priscilla Ann Dymond (1799–1828), in 1826, but she died two years later. In 1832 he married his second wife, Candia Barrow (1805–1855)  and had seven children: John (1834–1866), Richard (1835–1899), Maria (1838–1908), George (1839–1922), Joseph (1841–1841), Edward (1843–1866), and Henry (1845–1875).
Benjamin and John Cadbury dissolved their partnership in 1860. John retired in 1861 due to the death of his wife, and his sons Richard and George succeeded him in the business.  In 1879 they relocated to an area of what was then north Worcestershire, on the borders of the parishes of Northfield and King's Norton centred on the Georgian -built Bournbrook Hall, where they developed the garden village of Bournville ; now a major suburb of Birmingham.
The family developed the Cadbury's factory, which remains the main UK manufacturing site of the business. The district around the factory has been dry for over 100 years, with no alcohol being sold in pubs, bars or shops. Residents have fought to maintain this, winning a court battle in March 2007 with Britain's biggest supermarket chain Tesco, to prevent it selling alcohol in its local outlet.