Born in Georgetown in British Guiana (now Guyana), Williams began drawing and painting at an early age. He received informal art tutoring from the age of three, and joined the Working Peoples' Art Class at the age of 12. After training as an agronomist he worked as an Agricultural Field Officer for eight years, initially on the sugar plantations of the East Coast and later in the North-West region of the country—an area inhabited primarily by the indigenous Warao people. His time among the Warao had a dramatic impact on his artistic approach, and initiated the complex obsession with pre-Columbian arts and cultures that ran throughout his artistic career.
While living in London and travelling in Europe in the mid-1950s, Williams was introduced to the most recent trends in European and American art through exhibitions at The Tate in London and elsewhere.He was impressed and excited by an exhibition of German expressionist painting in London, and by the work of American abstract expressionist artists such as Jackson Pollock, Franz Kline, Barnett Newman, Willem de Kooning, Mark Rothko.He found a particular affinity with Arshile Gorky — whose work, he said, "fitted in some way with [his] own perception" - and with Roberto Matta.He traced his affinity with Matta to a shared experience of colonialism: "It's the smell of the presence of the conquistadors. It's the smell of a loss, and a replacement of lesser than what was destroyed".These various influences had a significant impact on his artistic development in subsequent years.