and across a wide range of data types, includ-

P OLICY FORUM ing images, audio, text, and other inputs (1).
Cutting-edge adversarial techniques gen-
MACHINE LEARNING erally use optimization theory to find small
data manipulations likely to fool a targeted

Adversarial attacks on model. As a proof of concept in the medi-

cal domain, we recently executed success-
ful adversarial attacks against three highly
medical machine learning accurate medical image classifiers (7). The
top figure provides a real example from one
of these attacks, which could be fairly easily
Emerging vulnerabilities demand new conversations commoditized using modern software. On
the left, an image of a benign mole is shown,
By Samuel G. Finlayson1, John D. Bowers2, machine learning into regulatory decisions which is correctly flagged as benign with a
Joichi Ito3, Jonathan L. Zittrain2, Andrew by way of computational surrogate end confidence of >99%. In the center, we show
L. Beam4, Isaac S. Kohane1 points and so-called “in silico clinical trials.” what appears to be random noise, but is in
Under the United States’ health care model, fact a carefully calculated perturbation: This

W
ith public and academic attention some of the most direct impacts of machine- “adversarial noise” was iteratively optimized
increasingly focused on the new learning algorithms come in the context of to have maximum disruptive effect on the
role of machine learning in the insurance claims approvals. Billions of medi- model’s interpretation of the image without

Downloaded from http://science.sciencemag.org/ on October 12, 2020

health information economy, an cal claims are processed each year, with ap- changing any individual pixel by more than
unusual and no-longer-esoteric cat- provals and denials directing trillions of a tiny amount. On the right, we see that de-
egory of vulnerabilities in machine- dollars and influencing treatment decisions spite the fact the perturbation is so small as
learning systems could prove important. for millions of patients. In addition to dic- to be visually imperceptible to human beings,
These vulnerabilities allow a small, carefully tating the availability of patient care, claims it fools the model into classifying the mole as
designed change in how inputs are presented approval is vested with competing financial malignant with 100% confidence. It is impor-
to a system to completely alter its output, interests, with providers seeking to maximize tant to emphasize that the adversarial noise
causing it to confidently arrive at manifestly and payers seeking to minimize reimburse- added to the image is not random and has
wrong conclusions. These advanced tech- ment (4). Given the volume and value of near-zero probability of occurring by chance.
niques to subvert otherwise-reliable ma- processing medical claims, it is unsurprising Thus, such adversarial examples reflect not
chine-learning systems—so-called adversarial that many providers engage in creative and that machine-learning models are inaccu-
attacks—have, to date, been of interest pri- often fraudulent practices to increase their rate or unreliable per se but rather that even
marily to computer science researchers (1). revenue (5). For their part, insurance com- otherwise-effective models are susceptible to
However, the landscape of often-competing panies and their contractors have invested manipulation by inputs explicitly designed to
interests within health care, and billions of in extensive machine-learning infrastructure fool them.
dollars at stake in systems’ outputs, implies for billing code processing. Although much of Adversarial attacks constitute one of many
considerable problems. We outline motiva- our discussion highlights financial incentives possible failure modes for medical machine-
tions that various players in the health care specific to the fee-for-service model in the learning systems, all of which represent es-
system may have to use adversarial attacks United States, the implications of algorithmic sential considerations for the developers and
and begin a discussion of what to do about vulnerabilities have broad relevance. users of models alike. From the perspective of
them. Far from discouraging continued in- policy, however, adversarial attacks represent
novation with medical machine learning, we DEEP VULNERABILITIES an intriguing new challenge, because they af-
call for active engagement of medical, techni- Adversarial examples are inputs to a machine- ford users of an algorithm the ability to in-
cal, legal, and ethical experts in pursuit of ef- learning model that are intentionally crafted fluence its behavior in subtle, impactful, and
ficient, broadly available, and effective health to force the model to make a mistake. Adver- sometimes ethically ambiguous ways.
care that machine learning will enable. sarial inputs were first formally described Deliberately crafting a noise-based adver-
In medical diagnostics and decision sup- in 2004, when researchers studied the tech- sarial example targeting a visual diagnostic
port, machine-learning systems appear to niques used by spammers to circumvent algorithm, as in the top figure, would amount
have achieved diagnostic parity with physi- spam filters (6). Typically, adversarial exam- to overt fraud. However, the bottom figure
cians on tasks in radiology, pathology, derma- ples are engineered by taking real data, such demonstrates that adversarial techniques in-
tology, and ophthalmology (2). In 2018, the as a spam advertising message, and making clude a broad range of perturbations and can
U.S. Food and Drug Administration (FDA) intentional changes to that data designed be applied across a vast number of input me-
approved marketing for the first-ever auton- to fool the algorithm that will process it. In diums. Some of these perturbations seem to
omous artificial intelligence (AI) diagnostic the case of text data like spam, such altera- be far less explicitly manipulative than the at-
system and indicated that they are “actively tions may take the form of adding innocent tack depicted in the top figure. As the bottom
developing a new regulatory framework to text or substituting synonyms for words that figure shows, minimal, but precise, adjust-
promote innovation in this space” (3). Regu- are common in malignant messages. In other ments such as rotating images to a specific
lators have articulated plans for integrating cases, adversarial manipulations can come angle have been shown to amount to effective
in the form of imperceptibly small perturba- adversarial attacks even against modern con-
1
Harvard Medical School, Boston, MA 02115, USA. 2Harvard tions to input data, such as making a human- volutional neural networks (8). In natural-
Law School, Cambridge, MA 02138, USA. 3Massachusetts invisible change to every pixel in an image. language processing, substitution of carefully
Institute of Technology Media Lab, Cambridge, MA 02139, Researchers have demonstrated the existence selected synonyms can be sufficient to fool al-
USA. 4Harvard T.H. Chan School of Public Health, Boston,
MA 02115, USA. Email: samuel_finlayson@hms.harvard.edu; of adversarial examples for essentially every gorithms such as the hypothetical opioid risk
isaac_kohane@hms.harvard.edu type of machine-learning model ever studied algorithm (see the bottom figure) (9). In the

SCIENCE sciencemag.org 22 MARCH 2019 • VOL 363 ISSUE 6433 1287

Published by AAAS
INSIGHTS | P O L I C Y F O RU M

case of structured data such as billing codes, The anatomy of an adversarial attack
adversarial techniques could be used to au- Demonstration of how adversarial attacks against various medical AI systems might be
tomate the discovery of code combinations executed without requiring any overtly fraudulent misrepresentation of the data.
that maximize reimbursement or minimize
the probability of claims rejection. Original image Adversarial noise Adversarial example
Because adversarial attacks have been
demonstrated for virtually every class of ma-
chine-learning algorithms ever studied, from
simple and readily interpretable methods
such as logistic regression to more compli- + 0.04 𰀘 =
cated methods such as deep neural networks
(1), this is not a problem specific to medicine,
and every domain of machine-learning ap-
plication will need to contend with it. Re- Dermatoscopic image of a benign Perturbation computed Combined image of nevus and
searchers have sought to develop algorithms melanocytic nevus, along with the by a common adversarial attack perturbation and the
that are resilient to adversarial attacks, such diagnostic probability computed attack technique. diagnostic probabilities from
as by training algorithms with exposure to by a deep neural network. See (7) for details. the same deep neural network.
adversarial examples or using clever data Benign Benign
processing to mitigate potential tampering Malignant Malignant
(1). Early efforts in this area are promising,

Downloaded from http://science.sciencemag.org/ on October 12, 2020

Model confdence Model confdence
and we hope that the pursuit of fully robust
machine-learning models will catalyze the Adversarial
development of algorithms that learn to rotation (8)
Diagnosis: Benign Diagnosis: Malignant
make decisions for consistently explainable
and appropriate reasons. Nevertheless, cur-
rent general-use defensive techniques come The patient has a history of Adversarial The patient has a history of
at a material degeneration of accuracy, even back pain and chronic alcohol text substitution (9) lumbago and chronic alcohol
abuse and more recently has dependence and more recently
if sometimes at improved explainability (10).
been seen in several... has been seen in several...
Thus, the models that are both highly accu-
rate and robust to adversarial examples re- Opioid abuse risk: High Opioid abuse risk: Low
main an open problem in computer science.
These challenges are compounded in the 277.7 Metabolic syndrome 401.0 Benign essential hypertension
medical context. Medical information tech- 429.9 Heart disease, unspecifed Adversarial 272.0 Hypercholesterolemia
nology (IT) systems are notoriously difficult 278.00 Obesity, unspecifed coding (13) 272.2 Hyperglyceridemia
to update, so any new defenses could be diffi- 429.9 Heart disease, unspecifed
278.00 Obesity, unspecifed
cult to roll out. In addition, the ground truth
in medical diagnoses is often ambiguous, Reimbursement: Denied Reimbursement: Approved
meaning that for many cases no individual
human can definitively assign the true label At the extreme of this tactical shaping of mends billing for codes corresponding to
between, say, “benign” and “cancerous” on a a patient presentation is medical fraud, a specific diseases that make up metabolic
photograph of a mole. This could enable bad $250 billion industry (11). Although some syndrome, such as hypertension. In other
actors to selectively perturb borderline cases providers may submit overtly fictitious medi- words, providers are not encouraged to
without easy means of review, consistently cal claims, misrepresentation of patient data add fraudulent claims but are encouraged
nudging scales in their direction. often takes much more subtle forms. For ex- to avoid adding a true claim that an insur-
ample, intentional upcoding is the practice ance company would be likely to reject in
EXISTING ADVERSARIAL BEHAVIOR of systematically submitting billing codes combination with another. This recommen-
Cutting-edge adversarial attacks have yet to for services related to, but more expensive dation is arguably motivated to serve the
be found in the health care context, though than, those that were actually performed. patients seeking coverage, not only the doc-
less formalized adversarial practice is ex- This practice is rampant and is just one of tors receiving reimbursement. However, it
tremely common. This existing activity sug- many questionable billing practices deployed highlights both a moral gray zone and the
gests that incentives for more sophisticated in clinical practice. Some physicians, for ex- type of strategy that providers might use
CREDITS: (GRAPHIC) N. CARY/SCIENCE; (PHOTOS) ISIC/CC-0

adversarial attacks may already be in place. ample, are inclined to report exaggerated an- to achieve the same end result as upcoding
To illustrate existing behaviors, we look to esthesia times to increase revenue (12). without ever committing overt fraud.
the modern U.S. medical billing industry. In other circumstances, subtle billing
Medical claims codes determine reim- code adjustments fall within a gray zone A GROWTH INDUSTRY
bursement for a patient visit after they between fraud and well-intentioned best As the machine-learning tool kit used by
have been approved by a payer. To evaluate practices. In one striking example, the web- insurance companies and their contrac-
these claims, payers typically leverage au- site of the Endocrine Society recommends tors continues to expand, the same dynam-
tomated fraud detectors, powered increas- that providers do not bill for the Interna- ics that favor creative billing practices in
ingly by machine learning. Health care tional Classification of Diseases (ICD) code the present may expand to include more
providers have long exerted influence on 277.77 (metabolic syndrome) in patients sophisticated adversarial attacks. Adver-
payers’ decisions (the algorithmic outputs) with obesity, as this combination of code sarial methods could allow billing teams to
by shaping their records (and accompany- and condition is likely to result in a denial scale up upcoding practices without getting
ing codes) of patient visits (the inputs) (5). of coverage (13). Instead, the Society recom- flagged by fraud detectors. Many insurance

1288 22 MARCH 2019 • VOL 363 ISSUE 6433 sciencemag.org SCIENCE

Published by AAAS
companies are beginning to require other A PATH FORWARD original hash to that of the data fed through
data types such as imaging and text to prove An essential question remains: when and how a targeted algorithm would allow investi-
that claims are valid. As they do so, other to intervene. Here, the early history of the in- gators to determine if that data had been
styles of adversarial attacks may be used as ternet offers a lesson. The approach to net- tampered with or changed after acquisition.
well to try to continue to dodge detection. work architecture introduced at the advent of Such an intervention would rely on a health
For example, if an insurance company the internet was centered around the defer- IT infrastructure capable of supporting the
requires that an image from a mole be run ral of problems. In their essential 1984 paper, capture and secure storage of these hashes.
through a melanoma classifier before ap- Saltzer et al. describe a design ethos whereby But as a strictly regulated field with a focus
proving reimbursement for an excision, problems are to be solved at the end points on accountability and standards of proce-
fraudsters may at first be inclined to submit of a network by users rather than preemp- dure, health care may be very well suited to
moles from different patients to achieve ap- tively within the architecture of the network such adaptations.
proval. If insurance companies then begin itself (14). There is frequently an advantage The coalescence of strong motives to
utilizing human audits or technical tests (in terms of simplicity, flexibility, and scalabil- manipulate algorithms and the rapid pro-
to try to ensure that the images are com- ity) to leaving future problems unsolved until liferation of algorithms vulnerable to manip-
ing from the correct patient, the next round their time has come. Another description for ulation makes health care a plausible ground
would be to move to full adversarial attacks this is the “procrastination principle” (15). zero for the emergence of adversarial exam-
with imperceptible alterations, such as in The procrastination principle frames a dif- ples into real-world practice. As adversarial
the top figure. Simpler techniques such as ficult question: Should the adversarial-exam- examples emerge across a range of domains,
the rotation in the bottom ples problem in health care we will have to make choices again and again
figure could constitute an “An essential systems be addressed now— about whether and how to intervene early at

Downloaded from http://science.sciencemag.org/ on October 12, 2020

ethical gray zone—given in the early, uncertain days the risk of stifling development, and how to
that a dermatologist could, question remains: of medical AI algorithms—or balance the promises of ubiquitous machine
in theory, hold the camera later, when algorithms and learning against liabilities imposed by these
at any angle.
when and how the protocols governing their emerging vulnerabilities. And the stakes
Potential applications of to intervene.” use have been firmed up? At will remain high—autonomous vehicles and
adversarial attacks in the best, acting now could equip AI-driven weapons systems will be just as
medical context go far beyond insurance us with more resilient systems and proce- susceptible. A clear-eyed and principled ap-
fraud, encompassing a spectrum of motiva- dures, heading the problem off at the pass. At proach to adversarial attacks in the health
tions. For instance, many adversarial attacks worst, it could lock us into inaccurate threat care context—one which builds the ground-
could be motivated by a desire to provide models and unwieldy regulatory structures, work for resilience without crippling rollout
high-quality care. A hypothetical illustra- stalling developmental progress and robbing and sets ethical and legal standards for line-
tion can be drawn from the opioid crisis. systems of the flexibility they need to con- crossing behavior—could serve as a critical
In response to rampant overprescription of front unforeseen threats. model for these future efforts. j
opiates, insurance companies have begun One regulatory response might be to insist
RE FERENCES AND NOTES
using predictive models to deny opiate pre- on forestalling implementation of vulnerable
1. B. Biggio, F. Roli, Pattern Recognit. 84, 317 (2018).
scription filings on the basis of risk scores algorithms until they are made adequately re- 2. T. Ching et al., J. R. Soc. Interface 15, 20170387 (2018).
computed at the patient or provider level. silient. However, given the potential of these 3. S. Gottlieb, “FDA’s comprehensive effort to advance new
innovations: Initiatives to modernize for innovation,”
What if a physician, certain that she had algorithms to improve health care delivery FDA Voices, 29 August 2018; www.fda.gov/NewsEvents/
a patient who desperately needed oxycon- for millions, this strategy might do more Newsroom/FDAVoices/ucm619119.htm.
tin but would nonetheless run afoul of the harm than good, and adequate resiliency is 4. A. S. Kesselheim, T. A. Brennan, N. Engl. J. Med. 352, 855
(2005).
prescription authorization algorithm, could not imminent. Generally resilient algorithms 5. M. K. Wynia, D. S. Cummins, J. B. VanGeest, I. B. Wilson,
type a special pattern of algorithmically se- confront an unfortunate reality familiar to JAMA 283, 1858 (2000).
6. N. Dalvi et al., in KDD ’04: Proceedings of the 10th ACM
lected billing codes or specific phrases into cybersecurity practitioners: Breaking sys- SIGKDD International Conference on Knowledge Discovery
the record to guarantee approval? tems is often easier than protecting them. and Data Mining (ACM, New York, 2004).
Companies might face temptations in This is because defenses must secure against 7. S. G. Finlayson et al., arXiv:1804.05296 [cs.CR] (15 April
2018).
the context of drug and device approvals. all conceivable present and future attacks, 8. L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, A. Madry,
Regulatory bodies, including the FDA, have whereas attacks need only defeat one or more arXiv:1712.02779 [cs.LG] (7 December 2017).
9. J. Li, S. Ji, T. Du, B. Li, T. Wang, arXiv:1812.05271 [cs.CR] (13
expressed interest in using algorithmic specific defenses. Like hack-proofing, defend- December 2018).
biomarkers as end points in clinical trials ing against adversarial examples is a cat-and- 10. D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, A. Madry,
and other approval processes. If this is real- mouse game. arXiv:1805.12152 [stat.ML] (20 May 2018).
11. A. Jain, S. Nundy, K. Abbasi, BMJ 348, g4181 (2014).
ized, adversarial examples could provide a Nevertheless, there are incremental de- 12. E. C. Sun, R. P. Dutton, A. B. Jena, JAMA Netw. Open 1,
means for companies to bias trial outcomes fensive steps that might be taken in the e184288 (2018).
13. K. Reynolds, P. Muntner, V. Fonseca, Diabetes Care 28, 1831
in their favor. For example, if a regulator short term given sufficient political and (2005).
requires matched images or wearable read- institutional will. Best practices in hospital 14. J. H. Saltzer, D. P. Reed, D. D. Clark, ACM Trans. Comput.
outs from each patient before and after labs are already enforced through regula- Syst. 2, 277 (1984).
15. J. Zittrain, The Future of the Internet and How to Stop It
treatment, trialists could inject adversarial tory measures such as Clinical Laboratory (Yale Univ. Press, 2008).
noise into posttreatment data, securing Improvement Amendments, which could
ACKNOWLEDGME NTS
the algorithmically measured results they easily be amended or extended to cover best
S.G.F. was supported by training grant T32GM007753 from
desired. Motivations could be complex— practices engineered against adversarial at- the National Institute of General Medical Science. A.L.B. and
whereas some trialists would be motivated tacks. For example, in situations in which I.S.K. contributed equally to this work. The content is solely the
by the potential for a big payday, others tampering with clinical data or images responsibility of the authors and does not necessarily represent
the official views of the National Institute of General Medical
might turn to adversarial attacks to “ad- might be possible, a “fingerprint” hash of
Sciences or the National Institutes of Health.
just” borderline trial results for products the data might be extracted and stored at
that might save lives. the moment of capture. Comparison of this 10.1126/science.aaw4399

SCIENCE sciencemag.org 22 MARCH 2019 • VOL 363 ISSUE 6433 1289

Published by AAAS
Adversarial attacks on medical machine learning
Samuel G. Finlayson, John D. Bowers, Joichi Ito, Jonathan L. Zittrain, Andrew L. Beam and Isaac S. Kohane

Science 363 (6433), 1287-1289.

DOI: 10.1126/science.aaw4399

Downloaded from http://science.sciencemag.org/ on October 12, 2020

ARTICLE TOOLS http://science.sciencemag.org/content/363/6433/1287

RELATED http://stm.sciencemag.org/content/scitransmed/10/457/eaar7939.full
CONTENT
http://stm.sciencemag.org/content/scitransmed/10/471/eaao5333.full
http://stm.sciencemag.org/content/scitransmed/3/102/102ra95.full
http://stm.sciencemag.org/content/scitransmed/7/274/274ra16.full
REFERENCES This article cites 8 articles, 1 of which you can access for free
http://science.sciencemag.org/content/363/6433/1287#BIBL

PERMISSIONS http://www.sciencemag.org/help/reprints-and-permissions

Use of this article is subject to the Terms of Service

Science (print ISSN 0036-8075; online ISSN 1095-9203) is published by the American Association for the Advancement of
Science, 1200 New York Avenue NW, Washington, DC 20005. The title Science is a registered trademark of AAAS.
Copyright © 2019, American Association for the Advancement of Science