Gimpel was one of three sons of a French father, the art dealer René Gimpel, and an English mother, Florence, the youngest sister of Lord Duveen. Gimpel was brought up in luxury in a house in the Bois de Boulogne, though he went on to be educated in both France and Britain. He made his living as a diamond broker before establishing himself as a critic of the concept of the great artist.
During the Second World War Gimpel served in the French Resistance, for which he was awarded the Croix de Guerre, the Resistance Medal, and the Legion of Honour.