Dr. King's Letter From A Birmingham Jail addresses this:
I guess it is easy for those who have never felt the stinging darts of segregation to say, "Wait."
He then proceeds, as does the Declaration of Independence in listing grievances against George III, to list the effects of the indignities (and worse) caused by segregation. The list is too long to ignore, too powerful to wish away.
So on this journey of understanding I think I've reached a pebble of understanding. Perhaps my original view, and that of Goldwater, is too libertarian; that there are rights beyond those contained in the Constitution, and those rights are just as worthy as those specifically written. Perhaps your "right" to conduct your business as you see fit conflicts with another's "right" to societal justice--and that's how we square the public accommodations clause with the Constitution.
I don't come upon this determination lightly, but as I said, Dr. King's list of indignities and practical effects cries out for justice. I'm aware that this opens up yet another can of worms--who determines these extra-Constitutional rights? Does this view not grant unlimited power to the federal government? Does this not now allow us to read into the Constitution whatever it is we want to find?
I don't have answers to these questions (yet). However, I'm reminded of what Goldwater said at the 1964 Republican Convention: Extremism in the defense of liberty is no vice, moderation in the pursuit of justice is no virtue.