Theory of wisdom
The author | weakish
Source | Reddit
One of the criticisms of neural networks and deep learning is that they are a black box. With training and testing, sometimes we can get pretty good results. But we don’t know why neural networks do so well. Similarly, neural networks sometimes produce results that are funny, and we often don’t know why. Of course, some people believe that the black-box nature of neural networks is its advantage. Andrej Karpathy, head of AI at Tesla, wrote two days ago that neural networks represent a fundamental change in the way software is developed, and software development will enter the 2.0 era.
The day before yesterday, there was also a discussion on Reddit’s machine Learning node: Is the black box problem exaggerated? The question sparked a heated debate and received nearly 100 responses in less than 24 hours. Nonzhixuan translated some of the comments for readers.
Black boxes are an advantage
Soutioirsim leads the discussion, arguing that black boxes should be seen more as an advantage than a problem, because they mean models are not limited by human explanatory power:
“Whenever I talk to non-machine learning scholars about work that uses neural networks or other cool machine learning methods, I often get comments that these methods are ‘black boxes’ where you can’t explain how they work.
“I disagree on the basis that models are limited by human explanatory power — if we only used models that are easy to explain, we wouldn’t be seeing as much progress in machine learning as we are. If you define machine learning as “the ability to learn and improve automatically from experience without explicit instructions,” then basically that definition means that these models will be difficult to interpret when you could easily have written them yourself.
“That doesn’t mean we shouldn’t strive for a better understanding of these models. In fact, I thought a recent lecture on deep learning information theory on Youtube was great, and hope we see more of this kind of work. Nonetheless, I think people need to shift their thinking and see the black box more as an advantage than a problem, because it points to models that are not humanly possible.”
A neural network out of control
Tpapp157 does not think that the black box problem is exaggerated, and he discusses the defects of neural networks from the perspective of cybernetics:
“In classical cybernetics, I can derive a control law that gives continuous control over a dynamic system. Using calculus based techniques, I can accurately quantify the stability of a closed-loop system, as well as its overhead and performance under all conditions. I can prove that my control laws are 100% safe, regardless of what the customer or compliance system requires of the fault tolerance rate. I can also accurately quantify other characteristics of the control system, such as its smoothness, or the comfort of the user or passenger. I can also easily adjust the controller I design based on specific completion attributes of a single product (already measured in the factory) to optimize the controller for a particular unit. Again, these are calculus based techniques and therefore provide precise and understandable results.
“A neural network can’t do any of these things. The best you can do is statistically analyze the neural network and get a rough estimate of the safety and stability of the system, which is strongly correlated with your training sample and test sample (which is always biased to some extent, of course). Admittedly, abnormal marginal conditions and other unexpected conditions can cause a controlled system to become unstable. If a neural network goes wrong, we can’t understand why it went wrong, and it’s hard to fix it. We cannot easily adjust a basic neural network controller to optimize the completion properties of a cell.
“Further, based on optimal control theory, and the loss function for a given dynamic system, I can deduce the corresponding optimal controller, rather than a maximum value is obtained by training neural network approximation value, and the approximation value is only likely to be a local maximum values (although no one knows whether). What I mentioned above is a 100% global optimal controller derived from a calculus based approach. Progress in optimal cybernetics has been one of the key factors driving the development of the aerospace industry since the 1940s.”
Nicksvr4 points out that defects in training data may lead to uncontrollable results:
“We may not be aware of deficiencies in training data. I once read an article about patient data. None of the asthmatic patients developed pneumonia without complications. The reason there were no complications was not because they had asthma, but because of the extra care given to asthmatic patients. The neural network saw only an association between asthma and no complications. If we don’t test the logic, people with asthma will be sent home because ‘there will be no complications,’ and there is a high risk of complications.”
Security is about systems and user experience
On the other hand, even if black boxes work well in engineering, they can still pose problems for the end user, especially on security-critical systems.
Hlynurd gives the example of machine learning determining treatment options:
“Consider a model in which decisions affect people’s lives. Like an algorithm that predicts deadly diseases. “Imagine how much better it is for patients to know that they have a confident doctor who understands why a difficult or expensive treatment is needed than to run into someone and shrug and tell them, ‘The computer says chemo.'”
Darkfeign agrees, saying that the autopilot system of aircraft can provide some inspiration and that safety systems need to be more “rational” :
“If you look at the accountability system for the autopilot of the aircraft, it’s ridiculously detailed. Would we want a neural network to drive a car if we couldn’t easily spot what went wrong?
“We still need rational systems like Bayesian networks and Markov networks. Sacrificing performance for rationality can be a tough decision, but in safety-critical systems we may not have much choice.”
Amenemhab argues that Darkfeign’s analogy is flawed, and that the autopilot system of an aircraft is not the same thing as a neural network:
“The autopilot system of the aircraft is very different from the black box. They are mathematically accurate and have been validated by the proof assistance system.”
Darkfeign countered that he was not saying that the autopilot system of the aircraft and the neural network were the same thing. He merely argues that security-critical systems will not be built entirely on neural networks until they provide similar transparency, but they may become part of them.
Thatguydr believes that if the actual error rate of neural network is much lower than that of the explainable model, it is rational to choose an “irrational system” like neural network:
“Neural networks are used in safety-critical systems when their actual error rate is much lower than the explainable model. Insurance companies are the regulator of this transition. It’s funny how no one mentions them. If insurance companies decide that neural networks are better for their bottom line [net income], interpretability is a thing of the past in cases where insurance companies rule.”
As for the user experience problem, Phobrain believes that it can be solved by visualization of neural network structure. In other words, in the case of Hlynurd’s “computer says chemo,” patients can ask to visualize every layer of the neural network if they don’t know why.
Phobrain’s geeky response garnered 38 “yes” votes in 12 hours, and some well-drawn responses. Hlynurd designed lines for neural network developers: “Doctor, can you give me a patient who generates an activation function that maximizes node 37000 anyway? I’m dying to know what went wrong there.” Gokstudio replied to Hlynurd saying: ‘The doctor said yes. Then Creussus was called and destroyed the world.” Singhan replied: “The last one was actually a lemon.” Apparently, he thinks that when neural networks go wrong, something unexpected is more likely to happen than something serious but somewhat predictable.
The cursive word lemon is very similar to the word demon
Justtheprint takes the user experience issue a bit more seriously, pointing out that there are a lot of treatments and drugs out there that we don’t really understand. The only reason we can defend them is simply that they work. Black-box algorithms can be evaluated in eerily similar ways: “In a randomised controlled trial, we followed the computer’s inferences, resulting in improved health and longer survival.”
Justtheprint has a lot of sympathy. Singham gives the example of psychotropic drugs, the mechanism of which we do not understand, and yet they are still approved. Terath points out that the situation is worse than that, not only because the mechanisms are unclear, but even because some treatments are ineffective:
“We routinely put in place dangerous processes that we don’t even know will work. For example, we performed a ‘shameful operation’ with minimal testing to see if the process worked. And, in doing so, we often find that the risky operation is actually of no benefit.
“At the very least we can test the machine learning model with some degree of accuracy given that the data meets some statistical distribution requirements.”
Based on his experience in the medical industry, Havok_79 points out that the current medical system is actually a black box to some extent, not necessarily more transparent than the neural network:
“I worked in health care for a few years, and it kind of jaded me. Often, black boxes are more a matter of misapplication of machine learning (or even basic statistics) than of the model itself.
“The average doctor may never really understand a machine learning model. I’m sure they won’t understand no matter how simple the model is. Mathematics and statistics are largely unemphasized in medical education. That’s not what we train doctors to do. I often hear doctors claim that there is no causal effect, as long as the training results of these observations reinforce the previous idea and have the right P value, the doctors will claim that the result is valid. I’ve worked with board certified informaticians who try to draw meaningful conclusions from subgroups with sample sizes of less than 10 (thousands of samples). Should we limit the advance of neural networks in this field, because the experts in this field don’t actually have the ability to use neural networks?
“Models in the medical field should pass the same scrutiny as any new treatment. But if the machine learning model outperforms the doctor most of the time on a large number of test samples, I really don’t care if my doctor can explain it. I can’t explain a doctor’s mistake. If the model makes fewer mistakes than the doctor, I think it’s still a better choice.”
Many Reddit users take a similar utilitarian view, despite the ethical risks it poses. Ensemblegh wonders:
“If keeping more people alive in the long term means someone needs to explain that John and Tom died because of a boundary situation in software 2.0, would you accept that?”
Cclausen asks rhetorically:
“Would you be willing to explain to someone that Tom and Tom died because of a mistake made by a doctor that software could have prevented?
“As a patient, I don’t know which is worse.”
Abstractoid also advocates abandoning mechanics and focusing more on utility:
“If we tell people that the model is 99 percent accurate at detecting cancer, and doctors are only 80 percent accurate, I doubt many people would be worried about how the model came to its conclusion. Do you think when the polio vaccine was first introduced, most doctors understood how it worked? Or do they just care if it works?”
Castlecrasher2 agrees that computers may be more effective, though it may take some time for patients’ perceptions to change:
“In reality, people tend to trust doctors without hesitation because they are ‘experts’. It’s harder to begin to understand that computers might be more effective.”
NowanIlfideme agrees that patients’ perceptions need to change:
“I believe that if individuals are less likely to die following software advice than they are to die following doctor advice, then the attitude that people still blame software rather than accept that they were just unlucky (misclassified by software) needs to change.”
VelveteenAmbush also believed that treatment effect was a more important factor than patient psychology:
“I guess the question is, if a patient has to choose whether to pursue the most effective treatment, how much will the patient be comforted?
“The idea that we should care more about what patients think and not that this treatment is going to best improve their health seems to me a step backwards. I expect most patients to agree with me when they fully understand the situation.
“So maybe the answer is to use this black box, but at the same time try to help patients understand how the model works and why the answer is somehow so incredible. In addition to basic weirdness, help the patient understand the results as much as possible. Then give them the best treatment (according to the treatment plan given in the black box).”
Akcom argues that Abstractoid is a radical idea:
“We certainly understand the mechanism of the polio vaccine. I work in predictive modeling in the healthcare industry, and interpretability is absolutely critical. Interpretability is not only desired by physicians, but also required for compliance certification of devices. When you need to evaluate a model’s performance in a production environment, ‘working’ is not good enough. In addition, we have also begun to use the attentional mechanism of circulating neural networks to improve interpretability, and some of the work of Choi et al played a key role in this.”
accountability
Duschendestroyer points out that humans themselves are a black box:
“You can ask a person why they came to a certain conclusion or why they made a particular decision. However, psychological research shows that we make up plausible explanations rather than actually explaining our decision-making process.”
Minewb agrees:
“Human beings are almost always partially or entirely wrong about the reasons for their choices. However, they are often very confident in their explanations. This is a dangerous combination. This has been shown repeatedly in the cognitive science literature.”
HTRP argues, however, that at least we can hold humans accountable:
“I think the issue is accountability. You can yell at a person to work faster or retest the model, or you can fire a person for modeling incorrectly.
“Machine learning algorithms don’t care.”
Psychedelic_thinker also points out that humans are used to assigning blame:
“If something goes wrong, humanity needs to find someone to hold to account. It’s the way human beings feel good about themselves and the only way they can get on with their lives without feeling guilty.”
other
For soutioirsim’s lecture on deep learning information theory in the main post, SomewittyAlias refers to a paper published in ICLR 2018 that has a counter point.
Datatatata thinks the discussion is off topic:
“First of all, the problem is not properly named. People complain about black boxes because they think the problem is advanced, but for the most part, we don’t really care about explaining predictions. What we care about is whether the decision makes sense, whether it’s impartial. My colleague said, ‘If the problem is bias, don’t call it a black box. Let’s call it a black box. ‘
“Second, explaining decisions does not always require explaining models. In fact, we can use more advanced techniques such as Lime or other techniques to explain individual forecasts. In most cases, this is straightforward (unless you care more about bias than black boxes). For example, it is more practical for a salesperson to explain a single credit decision than an entire credit scoring model.
“In a nutshell, I think ‘black box’ is actually a bogus question. There are many real problems to be solved (bias, unreliability, adversarial input, etc.).”
Notathrowaway113 goes so far as to call neural networks “the emperor’s new clothes” :
“I don’t understand what you’re talking about. Artificial neural network only uses nonlinear transformation and numerical optimization to find the lowest point on a high dimensional error surface. This produces a best-performing model for a given set of input vectors and the desired output labels. Artificial neural networks are completely obvious to anyone with a basic understanding of linear algebra.”
This article is compiled on wisdom, reprint please contact this public number for authorization.