There's a ton of literature on the topic if you care to look, but in short: this simply does not work due to societal assumptions and levels of privilege.
It's a non-starter to "not care", because privileged groups automatically get their foot in the door.
(You haven't even defined what you mean by "working" and "not working." But let's assume you mean that certain groups have an advantage in society.)
I'm a white male. I'm not particularly attractive. I am not able to exhibit certain class indicators. I am not very sociable or "cool." And I don't agree with mainstream ideology. Also, I don't have many social connections.
All of these are a pretty big disadvantage compared to, say, a black person with relevant social connections, or a handsome and suave white male, or an attractive female.
But at the end of the day, meritocracy still rules, and if you're good enough, you can get by just fine on merit.
That's just reality, and it "works" just fine.
And if there is any correction I want to advocate for publicly, it's certainly towards merit and away from any other factor. Any other factor ought to be irrelevant and is just going to change the balance of unfairness rather than improve the situation.
And, fortunately, in a free country (and even a semi-free country like the USA), merit is an absolutey overwhelming advantage.
That doesn't mean reality is fair. It's not. It will never be. Forget "fair." What we want is for things to be just.
And when you start discriminating based on race or gender, for any reason, you stop being just.
Can we all please get over this ridiculous idea that we are perfectly rational agents in some free-market meritocratic utopia? People are irrational, biased, accident-prone, living things. That alone precludes unbiased judgements of another person's "merit".
I mean, come on. We're programmers. We have jobs because of how much people screw up.
It's a non-starter to "not care", because privileged groups automatically get their foot in the door.