Several US states are using algorithms to apportion home care budgets
for the disabled. This has given more help to some, and less to
others. Some patients can't get out of bed without the help they no
longer get; some have died.
In theory, an algorithm can do the job better than humans. The
salesmen will lead you to imagine that the algorithm performs as well
as you could imagine. In practice, it can easily do a bad job. In
this case, some factors were explicitly not considered. Very likely
there are other important factors that that human decision-makers used
to recognize on their own.
If the state doesn't have the source code of the program, it should
not use the program.
But the problems described in the article can happen with
a free program, because they result from the absence of any
non-problematical solution.
The underlying problem is that the algorithm has to divide up funds
that are insufficient to start with. The algorithm tries to spread
the insufficiency equally. Is that the best thing to do?
Which is worse: to permanently give each disabled person half the help
perse needs, or to permanently abandon a randomly-chosen half of them?
The answer is not obvious to me.