I can think of two composers whose music used to baffle me and now captivates me -- Harry Partch and Robert Ashley. I'm listening to Ashley's Improvement for the first time. I don't "get it" yet but can't wait to hear more of it.
To a lesser degree, I feel the same about Captain Beefheart as do others:
What speaks to me in Beefheart's music is a fearless combination of total out-there avant-gardeness and the primordial simplicity and heart-grabbingness of the blues in a way that seems completely natural, as opposed a lot of high/low-culture-mixing experiments.