Color is a complex and fascinating subject, partly because there is no bijection between wavelengths and colors, and a theory of colors can therefore not be purely physical. Each wavelength of the visible spectrum corresponds to a color, but colors can be metamers : for example yellow is in the spectrum (~580 nm) but can also be obtained my mixing essentially green and red. And all colors are not in the spectrum (brown, pink, etc.). A scientist could even pretend that black and white are not colors but an absence or a mix of colors, but a painter could see things differently. Pierre Soulages explored the reflections of light on black in its paintings.
There are around 1e8 photoreceptors in our eye. 95 % are rod cells and 5 % are cone cells sensible to Short, Medium or Long wavelengths. Finally colors are built in our brain using information from the eye and we can distinguish ~300000 colors. If one or several types of cones don't work normally, a color vision deficiency (CVD) is present. On the contrary, some people are also synesthetes and see colors when they hear music, like the composer Olivier Messiaen. Some animals see the world in grey, others live in a more colored world than us: the mantis shrimp has 16 different cones!
It's no surprise that the way was long to understand colors: the young Isaac Newton (24 years old) worked on the visible spectrum with two prisms in 1666 as he was confined during the plague, Thomas Young postulated in 1802 the existence of three kinds of color receptors, Goethe was interested by the perception of colors (1810), Philipp Otto Runge published in 1810 its color sphere with its black and white poles, Maxwell published its triangle of colors in the middle of the 19th century and wrote the article "Experiments on colour, as perceived by the eye, with remarks on colour-blindness" (1855), Albert Henry Munsell published its color order system (Hue, Value, and Chroma) in 1915, etc.
Finally, that Paul Eluard's statement starting a 1929 poem may not be as surrealist as he thought. And the young Arthur Rimbaud probably did not know that grapheme–color synesthesia does exist when he gave colors to the vowels in 1871:
A black, E white, I red, U green, O blue, vowels,
Some day I'll tell your latent births
A noir, E blanc, I rouge, U vert, O bleu : voyelles,
Je dirai quelque jour vos naissances latentes
In literature, colors can even be strange and frightening. The Colour out of Space is the title of a short story published in Amazing Stories in 1927 by H. P. Lovecraft:
"and when upon heating before the spectroscope it displayed shining bands unlike any known colours of the normal spectrum there was much breathless talk of new elements, bizarre optical properties, and other things which puzzled men of science are wont to say when faced by the unknown."
"It was just a colour—but not any colour of our earth or heavens."